Real Time Web Analytics with Node.js and Socket.IO

I’ve always enjoyed building real-time applications. A few years ago, I built the Java back-end for the Tour of California “Tour Tracker”, and over the years, I have also been involved in several real-time Trader Desktop projects that I documented in this blog. So I thought it would be nice to explore the process of building a real-time application with Node.js and Socket.IO.

As an example, we’ll build a simple web analytics dashboard that shows pages being accessed on a web site in real-time. Kind of like the real-time view in Google Analytics.

The basic flow of the app is as follows:

1. The instrumented web application sends a message to the server every time there is a page change within the app.
2. Upon reception, the server processes the message and…
3. … pushes it to the clients who listen for “pageview” messages.

Testing the Hosted Version of the Application

A live version of the application is hosted on Heroku.

1. Open a browser and access http://nodecellar.coenraets.org/dashboard.html
2. Open a different browser window and start the main Node Cellar application: http://nodecellar.coenraets.org
3. Navigate through the application: the dashboard should display each page view.

In the remaining of this article we will go through the steps of building the application from scratch.

First Iteration: Basic Implementation

Node.js and Socket.IO make it easy to implement this type of real-time messaging system. Here is a simple (yet fully functional) implementation of the application:

The Server

In this first implementation, Node.js doesn’t serve the application’s HTML pages. Pages loaded from any domain (or even from the file system) can connect to our message server to send or listen to “pageview” messages.

The Instrumented Web App

Here is an example of an instrumented page with a script that sends a message to the server every time the page is loaded.

As mentioned above, this page can be loaded from any domain or from the file system.

The Real Time Dashboard

Here is a bare-bones implementation of the real-time dashboard application that receives “pageview” messages from the server.

That’s basically all you need to start building a sophisticated user interface with real-time charts, counters, maps, etc.

Testing the Application

1. Make sure you have Node.js and Socket.IO installed
2. Start the Node.js server

   node server

3. Double-click dashboard.html in Finder (Mac) or Explorer (Windows) to open the page in your default browser.
4. Double-click simplepage.html in Finder or Explorer to open the page in your default browser (in a separate window).
5. Refresh simplepage.html repeatedly: the dashboard should display each page view.

Second Iteration: Adding Authorization

The solution described above works fine, but there are some potential security issues we need to address:

1. Any application (from any domain) can send a message to our server.
2. Any application (from any domain) can listen to the messages pushed by our server.

Let’s refine our solution to make sure that only authorized clients can connect to our server. Socket.IO provides two different authorization methods: global authorization and namespace authorization (more info). In both cases, the handshakeData object provides information (headers, IP address, xdomain, etc) that is useful to implement your authorization logic and determine whether or not you want to allow the client to connect. In this application, we will use global authorization and reject all the cross-domain connection attempts:

There are many other ways you can secure your connections, but that discussion is beyond the scope of this article.

Rejecting cross-domain connections means that the application pages now also have to be served by the Node.js server. To put our new authorization policy into practice, let’s add the real-time web analytics capability to the Node Cellar application that I shared in my previous post.

The Server

The server now serves the HTML pages and hosts Socket.IO.

The Instrumented Web App

The following script has been added to index.html to send a “pageview” message to the server every time the page changes.

Notes:

• We actually don’t need to use Socket.IO for this part of the application. The client sends a “pageview” message to the server and doesn’t listen for messages from the server. In other words, this is a traditional client-to-server one-way communication that could be implemented with a traditional Ajax request.
• The onhashchange event is used because Node Cellar is a “Single Page Application”. The entire application runs within a single page (index.html): views are created dynamically at the client-side and injected into or removed from the DOM as you navigate through the application. Only the hash part of the URL changes (without a page refresh) to reflect the current state of the application.

The Real Time Dashboard

The final version of the dashboard is shown in the screenshot at the top of this post. The Socket.IO code didn’t change much since our initial implementation. The final code for dashboard.html is available here.

Testing the Application

1. Make sure you have Node.js and Socket.IO installed
2. Download the source code
3. Uncomment the Socket.IO related script in the head of public/index.html
4. Start the Node.js server

   node serverwithanalytics

5. Open a browser and access http://localhost:3000/dashboard.html
6. Open a different browser window and start the main Node Cellar application : http://localhost:3000
7. Navigate through the application: the dashboard should display each page view.

Source Code

I added the real-time analytics feature to the existing nodecellar repository.

• To start the regular server (without analytics), use:

  node server

• To start the server with analytics, uncomment the Socket.IO related script in the head of public/index.html, and use:

  node serverwithanalytics

Disclaimer

This is a sample application, not a production application. Some trade-offs were made to keep the code generic, simple and readable.