Building the back-end of the Tour of California “Tour Tracker” using Flex Data Services

If you hadn’t heard about it, the Tour of California ended on Sunday. To allow cycling enthusiasts around the world to follow the race in real time, the Experience Design (XD) team at Adobe built the “Tour Tracker”, an inspiring application built with Flex (in four weeks) and combining live video streaming, real time GPS tracking, commentary, race statistics, and geo-positioned photos and video clips uploaded during the course of each stage.

The “Tour Tracker” has been blogged about in many other places, but since I (was?) volunteered to build the back-end of the application using FDS, I figured I would share some details about the experience.

Because this application was about providing race information in real time, the Flex Data Services — and specifically the Flex Message Service — had a key role to play: push all the race events (GPS feed, VeloNews commentary, metadata about uploaded photos and videos, etc.) in real time to a large number of clients with virtually no latency.

You can still access the application here. However, the real excitement came from the live coverage of the race when video and all the real time information came together in a really slick user interface.

The Infrastructure

Overall FDS Infrastructure

One challenge in setting up the infrastructure is that we didn’t really know how many people would tune in. To be on the safe side, we decided to set up a cluster of eight servers. For logistic reasons, the servers were hosted externally on “virtual servers”. Each server was running FDS 2.0.1 on top of Tomcat 5.5.20.

It turned out we had about 1,500 concurrent users per server at peak time. The FDS servers were handling the load comfortably at a very low CPU utilization rate. We could definitely have handled the load with fewer servers.

Real-Time Protocol

Because of the number of users and the “immediate notification” requirement, RTMP was used as the default/preferred protocol. Clients behind a proxy or firewall that prevented RTMP connections automatically fell back to a polling channel.

Load Balancing and Failover

Because the application was stateless (no client state was maintained at the server side) and the servers didn’t have to exchange information with each other (each server was connecting directly to the different data providers), I implemented a simple load balancing and failover strategy: The client application created a ChannelSet with two Channels: one RTMP channel, and one fallback AMF/Polling channel to support clients unable to connect using the RTMP protocol. For each channel, a randomization algorithm was used to select a primary server to connect to and build the list of failover URIs. (The list of servers in the cluster was dynamically provided to the application using flashvars in the HTML wrapper).

This simple strategy worked well and the load was distributed very evenly between servers. The Flex Data Services Clustering solution based on JGroups can be used to handle more complex requirements.

The Application

The back-end of the TOC application consisted of a set of server-side components deployed in Tomcat that obtained data from different sources and used the Flex Message Service Java API to push the data to the clients.

The main components of the application at the server-side included: a GPS Listener, a Flickr Listener, and a set of servlets that allowed third parties to post data that had to be pushed to the clients (i.e. VeloNews commentary).

GPS Listener

CSC had set up an infrastructure to track the top riders and official vehicles with GPS devices. These devices transmitted their data to CSC servers using GPRS technology. CSC then published the near real-time GPS data stream through a socket server.

At the FDS side, a “GPS Listener” component running in its own thread listened to the socket server and published the GPS data it received to the clients. The GPS Listener had some built-in intelligence to deliver data more efficiently. For example, it aggregated the data at the server side for a configurable amount of time before publishing the consolidated data to the clients (sending two messages to clients for the same GPS device within a second wouldn’t make sense because it would probably not even move the dot on the map). However, since the design of the client application allowed you to zoom almost indefinitely, I eventually set up the gps.publish.delay parameter to three seconds to provide a smooth viewing experience.

This part of the application was actually very similar to real time market data streaming use cases I have been working on recently.

Flickr Listener

Mike Shimahara of BikeZen was taking geotagged photos that were uploaded to Flickr during the course of each stage. 10,000 users aggressively polling Flickr to check whether new photos had been uploaded would not have been a great strategy. Instead, a Flickr Listener component inside the FDS servers was polling Flickr on behalf of the clients. When the Flickr Listener found new photos, it sent a message to the clients including the metadata required to build the Flickr URLs pointing to the newly uploaded photos.

Play-by-Play and Geotagged Videos

VeloNews was providing the play-by-play commentary during each stage. VeloNews comments were posted to a servlet running on one server in the cluster. That servlet used a set of worker threads to dispatch the information in a non blocking manner to all the servers in the cluster, and each server instantly pushed the information to its connected clients. No polling required.

The same system was used to publish metadata about geotagged video clips, that like photos appeared geo-positioned on top of the map in the user interface.

That’s it. It was definitely a great experience to see FDS delivering all that real time information to a large audience in a high visibility and really cool application. Let me know if you have any question. I can probably share some of the server-side code too.

  • salman

    thanks for this detailed overview of the back-end of the TOC application. is it possible for me to have the code related to the GPS listener and its implementation?

  • As a cyclist, and someone that has been using Flash since version 3, I really appreciated this app. Thanks! Excellent work.

  • This was a great app, thanks for sharing the back-end details!

    (from another cyclist)

  • As a cycling Flex developer, this app was a great thing to see!

    For the mapping side, did you wrap the Yahoo! Maps component in a .swf (as many of us are stuck with), or was there something else (maybe an early Flex 2.0 version?) at play here?

  • Christophe, great job! Allan

  • Watching ToC through your app was absolutely an incredible experience for those of us that could not attend in person. Far and away the coolest blend of technology on the web.. maybe ever. This shows the power of things to come! We’re working to transition our entire application to Flex, so it was exciting to see it’s potential. Great job!

  • Lou Yanjun

    Assuming you have a web server or more web server on separate boxes and the Tomcat/FDS servers are on separate box. How do you configure the RTMP to push messages through app box to web box, eventaully reach the flashplayer? We have been stuggling with RTMP in this configuration for quite some time.

  • Lou Yanjun

    Again, you did not put all the TomCat/FDS servers out side the firewall, right? For a really solid app, don’t know if we ever be able to do that.

    What if our app servers/FDS are inside the firewall, and use gaining access through web server portal outside the firewall. How does the RTMP work in this situation? Could you please shed some light on the configuration?

  • Softwarecat

    Wow – very inspiring and testament to Flex capabilities

  • Nice work. I am interested in how u used GPS and Yahoo maps. Would u mind sharing some of this code?

    Much appreciated.

  • cedric

    Really excellent and great work ! I would also be really interested in having a lot at the sources. Would you mind sharing ?

    Thanks again

  • Pingback: Tour de Georgia Tour Tracker - Velochimp: Astrochimp on Cycling()

  • When will the sources to the Tour Tracker app be released? The prospect of this release has been pending for some time now.

    Given that this is likely to be one of the most comprehensive examples of Flex usage to date, being able to peer under the hood would provide many insights.

    Just a reminder, in case you folks thought “we” thought the announced code release had anything to do with April 1st. We didn’t. Were we mistaken?

  • I’m curious about what kind of partnership agreement with Flickr was required. I vaguely remember only being able to use the public Flickr API for limited use.

  • This is sick. Unbelievable what a little team work can accomplish. Awesome job. Hope you guys will share some of the behind the scenes code at some point.

  • Pingback: No Time to Blog - The Tour Tracker is Back()

  • Pingback: Comunidade Portuguesa de Rich Internet Applications» Blog Archive » Os 10 melhores posts do mês de Fevereiro ‘07()

  • Very good demo of Flex services; plus a great prototype of a real-time geolocation event. Would be interested in v.2.0 — sometime near Feb 2008?

  • Craig Taylor

    Can you or will you share any of the source for this application? I would like to build something similar and I’d love to poke around too see how you have handled things. BTW:this app rocks Cheers-Craig

  • This was a great app, thanks for sharing the back-end details!

  • Very good demo of Flex services; plus a great prototype of a real-time geolocation event. Would be interested in v.2.0 — sometime near Feb 2008?

  • Pingback: Tour of California 2009 with Lance Armstrong, Flex and LiveCycle Data Services : Christophe Coenraets()

  • This was a great app, thanks for sharing the back-end details!

  • yuvaraj

    how gprs generated
    ckt give me sir

  • yuvaraj

    how many type of gprs having sir

  • This was a great app, thanks for sharing the back-end details!

  • This was a great app, thanks for sharing the back-end details!

  • very good, thanks for sharing the back-end details

  • Pingback: Real Time Web Analytics with Node.js and Socket.IO()

  • very very thanks man

  • Very interesting work, thanks for the explanations and graphs as well, cool website.

  • Really, It is great … keep writing

  • A few years ago, I built the Java back-end for the Tour of California “Tour Tracker”, and over the years, I have also been

  • The “Tour Tracker” has been blogged about in many other places, but since I (was?)

  • profesyonel ürünler.kalitede

  • Afaq Husain

    Such a great information for blogger. Sportsmatik

  • TargbaBenjamin