Building the back-end of the Tour of California “Tour Tracker” using Flex Data Services

If you hadn’t heard about it, the Tour of California ended on Sunday. To allow cycling enthusiasts around the world to follow the race in real time, the Experience Design (XD) team at Adobe built the “Tour Tracker”, an inspiring application built with Flex (in four weeks) and combining live video streaming, real time GPS tracking, commentary, race statistics, and geo-positioned photos and video clips uploaded during the course of each stage.

The “Tour Tracker” has been blogged about in many other places, but since I (was?) volunteered to build the back-end of the application using FDS, I figured I would share some details about the experience.

Because this application was about providing race information in real time, the Flex Data Services — and specifically the Flex Message Service — had a key role to play: push all the race events (GPS feed, VeloNews commentary, metadata about uploaded photos and videos, etc.) in real time to a large number of clients with virtually no latency.

You can still access the application here. However, the real excitement came from the live coverage of the race when video and all the real time information came together in a really slick user interface.

The Infrastructure

Overall FDS Infrastructure

One challenge in setting up the infrastructure is that we didn’t really know how many people would tune in. To be on the safe side, we decided to set up a cluster of eight servers. For logistic reasons, the servers were hosted externally on “virtual servers”. Each server was running FDS 2.0.1 on top of Tomcat 5.5.20.

It turned out we had about 1,500 concurrent users per server at peak time. The FDS servers were handling the load comfortably at a very low CPU utilization rate. We could definitely have handled the load with fewer servers.

Real-Time Protocol

Because of the number of users and the “immediate notification” requirement, RTMP was used as the default/preferred protocol. Clients behind a proxy or firewall that prevented RTMP connections automatically fell back to a polling channel.

Load Balancing and Failover

Because the application was stateless (no client state was maintained at the server side) and the servers didn’t have to exchange information with each other (each server was connecting directly to the different data providers), I implemented a simple load balancing and failover strategy: The client application created a ChannelSet with two Channels: one RTMP channel, and one fallback AMF/Polling channel to support clients unable to connect using the RTMP protocol. For each channel, a randomization algorithm was used to select a primary server to connect to and build the list of failover URIs. (The list of servers in the cluster was dynamically provided to the application using flashvars in the HTML wrapper).

This simple strategy worked well and the load was distributed very evenly between servers. The Flex Data Services Clustering solution based on JGroups can be used to handle more complex requirements.

The Application

The back-end of the TOC application consisted of a set of server-side components deployed in Tomcat that obtained data from different sources and used the Flex Message Service Java API to push the data to the clients.

The main components of the application at the server-side included: a GPS Listener, a Flickr Listener, and a set of servlets that allowed third parties to post data that had to be pushed to the clients (i.e. VeloNews commentary).

GPS Listener

CSC had set up an infrastructure to track the top riders and official vehicles with GPS devices. These devices transmitted their data to CSC servers using GPRS technology. CSC then published the near real-time GPS data stream through a socket server.

At the FDS side, a “GPS Listener” component running in its own thread listened to the socket server and published the GPS data it received to the clients. The GPS Listener had some built-in intelligence to deliver data more efficiently. For example, it aggregated the data at the server side for a configurable amount of time before publishing the consolidated data to the clients (sending two messages to clients for the same GPS device within a second wouldn’t make sense because it would probably not even move the dot on the map). However, since the design of the client application allowed you to zoom almost indefinitely, I eventually set up the gps.publish.delay parameter to three seconds to provide a smooth viewing experience.

This part of the application was actually very similar to real time market data streaming use cases I have been working on recently.

Flickr Listener

Mike Shimahara of BikeZen was taking geotagged photos that were uploaded to Flickr during the course of each stage. 10,000 users aggressively polling Flickr to check whether new photos had been uploaded would not have been a great strategy. Instead, a Flickr Listener component inside the FDS servers was polling Flickr on behalf of the clients. When the Flickr Listener found new photos, it sent a message to the clients including the metadata required to build the Flickr URLs pointing to the newly uploaded photos.

Play-by-Play and Geotagged Videos

VeloNews was providing the play-by-play commentary during each stage. VeloNews comments were posted to a servlet running on one server in the cluster. That servlet used a set of worker threads to dispatch the information in a non blocking manner to all the servers in the cluster, and each server instantly pushed the information to its connected clients. No polling required.

The same system was used to publish metadata about geotagged video clips, that like photos appeared geo-positioned on top of the map in the user interface.

That’s it. It was definitely a great experience to see FDS delivering all that real time information to a large audience in a high visibility and really cool application. Let me know if you have any question. I can probably share some of the server-side code too.

32 Responses to Building the back-end of the Tour of California “Tour Tracker” using Flex Data Services

  1. salman February 27, 2007 at 11:35 am #

    thanks for this detailed overview of the back-end of the TOC application. is it possible for me to have the code related to the GPS listener and its implementation?

  2. Michael Krisher February 27, 2007 at 1:52 pm #

    As a cyclist, and someone that has been using Flash since version 3, I really appreciated this app. Thanks! Excellent work.

  3. Kevin Kazmierczak February 27, 2007 at 7:52 pm #

    This was a great app, thanks for sharing the back-end details!

    (from another cyclist)

  4. Joe Rinehart February 28, 2007 at 6:19 am #

    As a cycling Flex developer, this app was a great thing to see!

    For the mapping side, did you wrap the Yahoo! Maps component in a .swf (as many of us are stuck with), or was there something else (maybe an early Flex 2.0 version?) at play here?

  5. Allan Padgett March 1, 2007 at 11:02 am #

    Christophe, great job! Allan

  6. Gear Fisher March 2, 2007 at 8:54 am #

    Watching ToC through your app was absolutely an incredible experience for those of us that could not attend in person. Far and away the coolest blend of technology on the web.. maybe ever. This shows the power of things to come! We’re working to transition our entire application to Flex, so it was exciting to see it’s potential. Great job!

  7. Lou Yanjun March 2, 2007 at 12:10 pm #

    Assuming you have a web server or more web server on separate boxes and the Tomcat/FDS servers are on separate box. How do you configure the RTMP to push messages through app box to web box, eventaully reach the flashplayer? We have been stuggling with RTMP in this configuration for quite some time.

  8. Lou Yanjun March 2, 2007 at 12:23 pm #

    Again, you did not put all the TomCat/FDS servers out side the firewall, right? For a really solid app, don’t know if we ever be able to do that.

    What if our app servers/FDS are inside the firewall, and use gaining access through web server portal outside the firewall. How does the RTMP work in this situation? Could you please shed some light on the configuration?

  9. Softwarecat March 8, 2007 at 9:59 pm #

    Wow – very inspiring and testament to Flex capabilities

  10. najier March 14, 2007 at 6:28 am #

    Nice work. I am interested in how u used GPS and Yahoo maps. Would u mind sharing some of this code?

    Much appreciated.

  11. cedric March 24, 2007 at 5:59 am #

    Really excellent and great work ! I would also be really interested in having a lot at the sources. Would you mind sharing ?

    Thanks again

  12. Mario Pintaric May 1, 2007 at 8:40 pm #

    When will the sources to the Tour Tracker app be released? The prospect of this release has been pending for some time now.

    Given that this is likely to be one of the most comprehensive examples of Flex usage to date, being able to peer under the hood would provide many insights.

    Just a reminder, in case you folks thought “we” thought the announced code release had anything to do with April 1st. We didn’t. Were we mistaken?

  13. Tom Schober May 10, 2007 at 7:18 am #

    I’m curious about what kind of partnership agreement with Flickr was required. I vaguely remember only being able to use the public Flickr API for limited use.

  14. California Team Building August 14, 2007 at 10:38 am #

    This is sick. Unbelievable what a little team work can accomplish. Awesome job. Hope you guys will share some of the behind the scenes code at some point.

  15. Murray Leslie January 20, 2008 at 12:10 am #

    Very good demo of Flex services; plus a great prototype of a real-time geolocation event. Would be interested in v.2.0 — sometime near Feb 2008?

  16. Craig Taylor January 20, 2008 at 4:25 pm #

    Can you or will you share any of the source for this application? I would like to build something similar and I’d love to poke around too see how you have handled things. BTW:this app rocks Cheers-Craig

  17. sohbet August 22, 2008 at 6:49 am #

    This was a great app, thanks for sharing the back-end details!

  18. chat August 22, 2008 at 6:49 am #

    Very good demo of Flex services; plus a great prototype of a real-time geolocation event. Would be interested in v.2.0 — sometime near Feb 2008?

  19. radyo dinle July 19, 2009 at 10:10 am #

    This was a great app, thanks for sharing the back-end details!

  20. yuvaraj March 24, 2010 at 5:57 am #

    how gprs generated
    ckt give me sir

  21. yuvaraj March 24, 2010 at 5:58 am #

    how many type of gprs having sir

  22. FatLoss4 diotsMenu July 26, 2010 at 5:06 pm #

    This was a great app, thanks for sharing the back-end details!

  23. matbaa August 8, 2010 at 8:16 am #

    This was a great app, thanks for sharing the back-end details!

  24. chat May 22, 2011 at 8:13 am #

    very good, thanks for sharing the back-end details

  25. Yemek Tarifleri July 24, 2013 at 3:23 am #

    very very thanks man

  26. Water Damage Restoration Houston November 13, 2013 at 4:19 pm #

    Very interesting work, thanks for the explanations and graphs as well, cool website.

  27. Bahçelievler Satılık Daire January 12, 2015 at 3:54 pm #

    Really, It is great … keep writing


  1. Tour de Georgia Tour Tracker - Velochimp: Astrochimp on Cycling - April 13, 2007

    […] Details on the backend of the Tour of California Toru Tracker […]

  2. No Time to Blog - The Tour Tracker is Back - September 17, 2007

    […] Building the back-end of the Tour of California “Tour Tracker” using Flex Data Services […]

  3. Comunidade Portuguesa de Rich Internet Applications» Blog Archive » Os 10 melhores posts do mês de Fevereiro ‘07 - October 23, 2007

    […] Ver […]

  4. Tour of California 2009 with Lance Armstrong, Flex and LiveCycle Data Services : Christophe Coenraets - February 13, 2009

    […] wrote about the initial implementation of the LCDS backend back in 2007. This write-up is now a little dated […]

  5. Real Time Web Analytics with Node.js and Socket.IO - October 10, 2012

    […] always enjoyed building real-time applications. A few years ago, I built the Java back-end for the Tour of California “Tour Tracker”, and over the years, I have also been […]

Leave a Reply