Performance Tuning: Real Time Trader Desktop with Flex and LCDS

In the last few months, I’ve seen an explosion in the number of Financial Services firms interested in building Real Time Trader Desktop applications using Flex and LiveCycle Data Services.

The new Quality of Service features available in LCDS 3.x (guaranteed message delivery, message throttling, etc) have contributed to this intensified interest. But for many real time applications, performance is what matters most. So, in addition to adding and improving features, we also spent a lot of time running benchmarks and optimizing performance. Damon has been blogging about this recently. The tests he describes are mostly “server-focused”. They answer the question: How many messages per second can the server push to how many clients with what latency? In one of the scenarios tested, the answer was a total of 400,000 messages per second spread over 500 clients with an average latency of 15 milliseconds.

Another more “client-focused” question customers have asked me is: How many messages per second can one client consume and render with what latency? To answer that question, I built my own Performance Console and feed generator.

The Performance Console allows you to configure the throughput of the server-side feed generator as well as the client subscription details, and then measure the overall performance and health of the system.

Server Throughput

The number of symbols you select doesn’t have a direct impact on the total number of messages pushed. It only impacts the frequency (defined here as the number of messages pushed per subtopic, in other words, per symbol in this specific application). For example, if the feed generator generates updates for 1000 symbols using one thread and sleeping for 1ms between messages, it will generate a total of approximately 1000 messages per second, and each symbol will get one update per second. If you change the number of symbols to 500, the generator will still push a total of 1000 messages, but each symbol will now get two updates per second.

Client Subscription

You can specify the channel you want to use: RTMP, Streaming, or NIO Streaming. I used the Parsley framework to externalize the channel configuration in the channels-config.xml file. You could add additional channels in channels-config.xml if you wanted to test additional options. (The channels you add must be defined at the server-side in channels-config.xml and must be registered with the “f” destination in messaging-config.xml)

You can also enable throttling using the Max Frequency input field which defines the maximum number of messages per second you want the client to get per subscription (in other words, per symbol in this specific application). Setting Max Frequency to 0 (the default) keeps throttling disabled: the client will be sent all the messages processed by the server. If you set the value to 1, the client will receive a maximum of one message per second per symbol. A conflation policy is configured at the server side for the destination. The options are IGNORE, BUFFER, and MERGE. You can also define your own conflation policy by creating a Custom Outbound Queue.

Finally, you can set the Frame Rate of the application. The Frame Rate is not a parameter of the client subscription per se, but it can significantly impact the overall performance and behavior of this type of application, especially if you are pushing a very large number of messages to the client. This is why the performance console allows you to tweak it here, so you can find the best value for your own application.

Message Latency

The one indicator I found particularly useful to determine the maximum capabilities of the client is the message latency. The message latency is defined as the time it takes for a message to be handled by the client after it is published/pushed by the server. If the client can’t keep the latency low and fairly constant, you have probably reached the limits of the system, and many other things will start to go wrong: the application may become unresponsive, the messages will be queued and the latency may grow out of control, etc.

In this test suite, I calculate the message latency as follows:

• At the server-side, the feed generator adds a timestamp to the message when it is published.
• At the client-side, in the client-side message handler, the message latency is determined by calculating the difference between the current time and the timestamp added to the message at the server-side. Since both the client and the server are running on my machine, I don’t have to worry about clock synchronization in this case.

The Performance Console displays the latency of each incoming message (“Current Latency” field), providing a real time indicator of the overall health of the system. Before I started tracking and displaying this indicator in real time, it was a lot harder to figure out exactly when and why the client was starting to get overloaded.

Running Benchmarks

If you click the Run Benchmark button, the application will monitor the incoming messages for one minute and report the total number of messages received as well as the average latency in the Performance DataGrid.

Here are some of the tests I ran on my machine with the AIR version of the Performance Console. The first eight columns provide the test parameters. The last three columns provide the actual results of the test.

So for example, the second test shows that using the NIO streaming channel, the client processed 59,611 messages in one minute (993.5 messages per second) with an average latency of 0.3 milliseconds. The last test shows that using the RTMP channel, the client processed 116,136 messages in one minute (1,935.6 messages per second) with an average latency of 3.4 milliseconds.

These tests are theoretical. In a real life application, you’d probably not want to send 2,000 individual messages per second to your clients. Instead, you’d typically throttle messages using the policy (ignore, buffer, merge, or custom) that is appropriate for your application. Regardless, it is good to be able to identify the limits of the system.

NOTE: Because of a dependency that currently exists between Frame Rate and RTMP message processing, I increased the Frame Rate to handle large number of messages when using the RTMP channel.

Installing the Performance Console and Feed Generator.

Installing the Web application

1. Download LCDS 3.1 with Tomcat here
2. Run the installer
3. Make a copy the lcds web application in lcds/webapps, and call the new web application traderdesktop
4. Download traderdesktop.zip and overlay the content on top of lcds/webapps/traderdesktop
5. Start the server

Installing and running the AIR console

Click here to download and install the AIR application. This AIR file was built with the publicly available AIR 2 beta 2.

Running the Browser console

Access http://localhost:8400/traderdesktop/TraderDesktopWebPerfConsole/TraderDesktopWebPerfConsole.html

As always, your feedback is appreciated. If you are building this type of application I would definitely love to hear from you.

Disclaimer: I make this tool available to help you get started and run your own tests. This is not an official LCDS performance benchmark. There are many variables that affect the performance of real time systems. I put this tool together to help developers experiment with some of these variables, and not to produce the absolute best number Flex and LCDS can achieve.

UPDATE: Source Code

A number of people also asked for the source code of the application. You can download traderdesktop-projects.zip projects here.

Importing the projects in Flash Builder 4:

1. In Flash Builder 4, click File > Import > General > Existing Projects into Workspace.
2. Specify traderdesktop-projects as the root directory and click finish.
3. Explore the projects: TraderDesktop-lib is the library project where most of the source code lives. TraderDesktopPerfConsole is the project for the AIR version of the console. TraderDesktopWebPerfConsole is the project for the Browser version.

Fast DataGrid

The DataGrid available out-of-the-box in the Flex SDK is a general-purpose component that was built to support a wide variety of use cases and is therefore not specifically optimized to handle a trader desktop price grid with a large amount of real time updates. The good news is that it is easy to optimize the DataGrid for this use case. The basic idea is to disable the DataGrid’s collectionChange event handler and let the itemRenderers watch (and handle) changes to their respective data items: check out the FastDatagrid.as class and the applications item renderers for the details. Also read Tom Sugden’s recent blog post on this topic.