Building the back-end of the Tour of California “Tour Tracker” using Flex Data Services

If you hadn’t heard about it, the Tour of California ended on Sunday. To allow cycling enthusiasts around the world to follow the race in real time, the Experience Design (XD) team at Adobe built the “Tour Tracker”, an inspiring application built with Flex (in four weeks) and combining live video streaming, real time GPS tracking, commentary, race statistics, and geo-positioned photos and video clips uploaded during the course of each stage.

The “Tour Tracker” has been blogged about in many other places, but since I (was?) volunteered to build the back-end of the application using FDS, I figured I would share some details about the experience.

Because this application was about providing race information in real time, the Flex Data Services — and specifically the Flex Message Service — had a key role to play: push all the race events (GPS feed, VeloNews commentary, metadata about uploaded photos and videos, etc.) in real time to a large number of clients with virtually no latency.

You can still access the application here. However, the real excitement came from the live coverage of the race when video and all the real time information came together in a really slick user interface.

The Infrastructure

Overall FDS Infrastructure

One challenge in setting up the infrastructure is that we didn’t really know how many people would tune in. To be on the safe side, we decided to set up a cluster of eight servers. For logistic reasons, the servers were hosted externally on “virtual servers”. Each server was running FDS 2.0.1 on top of Tomcat 5.5.20.

It turned out we had about 1,500 concurrent users per server at peak time. The FDS servers were handling the load comfortably at a very low CPU utilization rate. We could definitely have handled the load with fewer servers.

Real-Time Protocol

Because of the number of users and the “immediate notification” requirement, RTMP was used as the default/preferred protocol. Clients behind a proxy or firewall that prevented RTMP connections automatically fell back to a polling channel.

Load Balancing and Failover

Because the application was stateless (no client state was maintained at the server side) and the servers didn’t have to exchange information with each other (each server was connecting directly to the different data providers), I implemented a simple load balancing and failover strategy: The client application created a ChannelSet with two Channels: one RTMP channel, and one fallback AMF/Polling channel to support clients unable to connect using the RTMP protocol. For each channel, a randomization algorithm was used to select a primary server to connect to and build the list of failover URIs. (The list of servers in the cluster was dynamically provided to the application using flashvars in the HTML wrapper).

This simple strategy worked well and the load was distributed very evenly between servers. The Flex Data Services Clustering solution based on JGroups can be used to handle more complex requirements.

The Application

The back-end of the TOC application consisted of a set of server-side components deployed in Tomcat that obtained data from different sources and used the Flex Message Service Java API to push the data to the clients.

The main components of the application at the server-side included: a GPS Listener, a Flickr Listener, and a set of servlets that allowed third parties to post data that had to be pushed to the clients (i.e. VeloNews commentary).

GPS Listener

CSC had set up an infrastructure to track the top riders and official vehicles with GPS devices. These devices transmitted their data to CSC servers using GPRS technology. CSC then published the near real-time GPS data stream through a socket server.

At the FDS side, a “GPS Listener” component running in its own thread listened to the socket server and published the GPS data it received to the clients. The GPS Listener had some built-in intelligence to deliver data more efficiently. For example, it aggregated the data at the server side for a configurable amount of time before publishing the consolidated data to the clients (sending two messages to clients for the same GPS device within a second wouldn’t make sense because it would probably not even move the dot on the map). However, since the design of the client application allowed you to zoom almost indefinitely, I eventually set up the gps.publish.delay parameter to three seconds to provide a smooth viewing experience.

This part of the application was actually very similar to real time market data streaming use cases I have been working on recently.

Flickr Listener

Mike Shimahara of BikeZen was taking geotagged photos that were uploaded to Flickr during the course of each stage. 10,000 users aggressively polling Flickr to check whether new photos had been uploaded would not have been a great strategy. Instead, a Flickr Listener component inside the FDS servers was polling Flickr on behalf of the clients. When the Flickr Listener found new photos, it sent a message to the clients including the metadata required to build the Flickr URLs pointing to the newly uploaded photos.

Play-by-Play and Geotagged Videos

VeloNews was providing the play-by-play commentary during each stage. VeloNews comments were posted to a servlet running on one server in the cluster. That servlet used a set of worker threads to dispatch the information in a non blocking manner to all the servers in the cluster, and each server instantly pushed the information to its connected clients. No polling required.

The same system was used to publish metadata about geotagged video clips, that like photos appeared geo-positioned on top of the map in the user interface.

That’s it. It was definitely a great experience to see FDS delivering all that real time information to a large audience in a high visibility and really cool application. Let me know if you have any question. I can probably share some of the server-side code too.