Real Time Web Analytics with Node.js and Socket.IO

I’ve always enjoyed building real-time applications. A few years ago, I built the Java back-end for the Tour of California “Tour Tracker”, and over the years, I have also been involved in several real-time Trader Desktop projects that I documented in this blog. So I thought it would be nice to explore the process of building a real-time application with Node.js and Socket.IO.

As an example, we’ll build a simple web analytics dashboard that shows pages being accessed on a web site in real-time. Kind of like the real-time view in Google Analytics.

The basic flow of the app is as follows:


  1. The instrumented web application sends a message to the server every time there is a page change within the app.
  2. Upon reception, the server processes the message and…
  3. … pushes it to the clients who listen for “pageview” messages.


Testing the Hosted Version of the Application

A live version of the application is hosted on Heroku.

  1. Open a browser and access http://nodecellar.coenraets.org/dashboard.html
  2. Open a different browser window and start the main Node Cellar application: http://nodecellar.coenraets.org
  3. Navigate through the application: the dashboard should display each page view.

In the remaining of this article we will go through the steps of building the application from scratch.

First Iteration: Basic Implementation

Node.js and Socket.IO make it easy to implement this type of real-time messaging system. Here is a simple (yet fully functional) implementation of the application:

The Server


In this first implementation, Node.js doesn’t serve the application’s HTML pages. Pages loaded from any domain (or even from the file system) can connect to our message server to send or listen to “pageview” messages.

The Instrumented Web App

Here is an example of an instrumented page with a script that sends a message to the server every time the page is loaded.

As mentioned above, this page can be loaded from any domain or from the file system.

The Real Time Dashboard

Here is a bare-bones implementation of the real-time dashboard application that receives “pageview” messages from the server.

That’s basically all you need to start building a sophisticated user interface with real-time charts, counters, maps, etc.

Testing the Application

  1. Make sure you have Node.js and Socket.IO installed
  2. Start the Node.js server

    node server

  3. Double-click dashboard.html in Finder (Mac) or Explorer (Windows) to open the page in your default browser.
  4. Double-click simplepage.html in Finder or Explorer to open the page in your default browser (in a separate window).
  5. Refresh simplepage.html repeatedly: the dashboard should display each page view.


Second Iteration: Adding Authorization

The solution described above works fine, but there are some potential security issues we need to address:

  1. Any application (from any domain) can send a message to our server.
  2. Any application (from any domain) can listen to the messages pushed by our server.

Let’s refine our solution to make sure that only authorized clients can connect to our server. Socket.IO provides two different authorization methods: global authorization and namespace authorization (more info). In both cases, the handshakeData object provides information (headers, IP address, xdomain, etc) that is useful to implement your authorization logic and determine whether or not you want to allow the client to connect. In this application, we will use global authorization and reject all the cross-domain connection attempts:

There are many other ways you can secure your connections, but that discussion is beyond the scope of this article.

Rejecting cross-domain connections means that the application pages now also have to be served by the Node.js server. To put our new authorization policy into practice, let’s add the real-time web analytics capability to the Node Cellar application that I shared in my previous post.

The Server

The server now serves the HTML pages and hosts Socket.IO.

The Instrumented Web App

The following script has been added to index.html to send a “pageview” message to the server every time the page changes.

Notes:

  • We actually don’t need to use Socket.IO for this part of the application. The client sends a “pageview” message to the server and doesn’t listen for messages from the server. In other words, this is a traditional client-to-server one-way communication that could be implemented with a traditional Ajax request.
  • The onhashchange event is used because Node Cellar is a “Single Page Application”. The entire application runs within a single page (index.html): views are created dynamically at the client-side and injected into or removed from the DOM as you navigate through the application. Only the hash part of the URL changes (without a page refresh) to reflect the current state of the application.

The Real Time Dashboard

The final version of the dashboard is shown in the screenshot at the top of this post. The Socket.IO code didn’t change much since our initial implementation. The final code for dashboard.html is available here.

Testing the Application

  1. Make sure you have Node.js and Socket.IO installed
  2. Download the source code
  3. Uncomment the Socket.IO related script in the head of public/index.html
  4. Start the Node.js server

    node serverwithanalytics

  5. Open a browser and access http://localhost:3000/dashboard.html
  6. Open a different browser window and start the main Node Cellar application : http://localhost:3000
  7. Navigate through the application: the dashboard should display each page view.

Source Code

I added the real-time analytics feature to the existing nodecellar repository.

  • To start the regular server (without analytics), use:

    node server

  • To start the server with analytics, uncomment the Socket.IO related script in the head of public/index.html, and use:

    node serverwithanalytics

Disclaimer

This is a sample application, not a production application. Some trade-offs were made to keep the code generic, simple and readable.

41 Responses to Real Time Web Analytics with Node.js and Socket.IO

  1. misaeljuvenal October 12, 2012 at 9:31 pm #

    hola.. este blog tiene bastante información útil.. gracias por los tutoriales…..;)

  2. Nicolas Froidure October 22, 2012 at 12:15 pm #

    Nice tutorial. It could be useful to add a listener to catch exits something like this :

    window.onclick = function (event) {
    if(event.target.nodeName==’A’
    &&event.target.getAttribute(‘href’).indexOf(‘http’)===0
    &&event.target.getAttribute(‘href’).indexOf(document.location.protocol
    +://’+document.location.host)!==0)
    socket.send(event.target.getAttribute(‘href’));
    }

    • rinku das November 5, 2012 at 9:45 am #

      @Nicolas Froidure: can you explain a bit little plz..seems an interesting concept..

      • Nicolas Froidure November 14, 2012 at 1:32 pm #

        The code is explicit, it save the url of the external links the user clicks tracking its exit.

  3. rinku das October 26, 2012 at 11:19 am #

    when we are refreshing the dashboard, there is no pageview record listed..even my client pages are opened..how to solve this problem, so that after refreshing the dashboard we can get previous result as well as updated result?plz ignore my english and programming knowledge..

  4. Bonjour Tristesse November 30, 2012 at 11:46 am #

    This was really simple to follow for a beginner like me. Thanks for putting it together.

  5. Tony December 5, 2012 at 8:01 pm #

    How would one go about extracting this live data into exportable reports (in .csv format)?

  6. sapardi December 6, 2012 at 6:56 am #

    Good article, i like it….How did you create project structure..? using comand :

    $> express blabla

    or create it manually..?

  7. sonu sindhu December 13, 2012 at 6:53 am #

    thanks for this
    very nice and helpful

  8. Fred Haegele December 20, 2012 at 4:20 pm #

    This tutorial is excellent!

    I’m surprised how little code it takes to get something together with this framework. This is coming from someone who has suffered through developing with RoR, as well as a few of the countless unnamed frameworks for PHP.

    I now have an excellent starting point for building a REAL real-time app with Node.JS. I’m surprised how easy it is….

    Regards-

    -A php/mysql guy.

  9. sohbet odalari December 20, 2012 at 11:21 pm #

    good blok thanks

  10. Candra January 14, 2013 at 5:36 am #

    Hi, coenraets its my first time leave comment to your tuts.

    I really like read your blog especially this articles.

    this is what i search for so long.

    thx

  11. Deva Karthik January 31, 2013 at 1:09 pm #

    Hi,

    I’m designing a site with PHP Zend Framework, plugged node.js into it to support instant notifications on the messaging module,
    to share the session centrally kept the PHP Session in redis and use it for authentication at node.js end.
    Included the node.js client javascript on all web pages to get notification bubble across the site.
    Issue : If I navigate from a page to another, the socket connection disconnects and connects again.
    any suggestions to overcome this.

  12. daniel May 28, 2013 at 7:59 pm #

    Very nice addon!!

    I try to host a nodejs application on heroku,

    how can i modify the socket.io path (http://localthost),
    that i works an the heroku instance?

    Kind regards
    daniel

  13. Beraki June 16, 2013 at 3:20 am #

    Nice Tutorial but I wanted to know which free nodejs server I can get to taste my product before testing.

  14. Create Web application June 26, 2013 at 6:31 am #

    Introducing web application development with Node.js, this book walks you through developing of a full-fledged chat system built with socket.io and introducing all the concepts of socket.io and its usage in the process.

  15. Jibran August 19, 2013 at 10:11 am #

    Great blog post. I have a question about some general dedicated server implementation. If I want to use a server instead of “localhost” how would I go about doing that. I have looked online for a general solution and most people suggest placing the URL into the area that reads “io.connect “. In short I replaced the socket variable to var socket = io.connect(‘http://mydoamin.com’);”. However this does not work, the browser throws an error saying that it cant find socket.io library. Any Ideas?

  16. Below Poverty Line September 2, 2013 at 1:18 am #

    Very nice for here the complete tutorial.
    Thanks

  17. fifa 14 Crack September 13, 2013 at 1:35 pm #

    of course like your website however you have to take a look at the spelling on several of your posts.
    Many of them are rife with spelling problems and I to find it very bothersome to tell the truth
    nevertheless I will definitely come back again.

  18. Ayaz September 30, 2013 at 5:11 am #

    Nice article, its really very simple. Thanks!!

  19. Sam November 5, 2013 at 7:09 am #

    Firstly, thanks for the demo.

    I think there is a bug somewhere though. There was 5 pageviews (4+1) and it says 14 active visitors. That can’t be right. See screenshot:

    http://screencast.com/t/vji8Rjfn

    Sam

  20. demirdöküm soba servisi November 6, 2013 at 6:37 am #

    Really nice App. Thank you. I am a beginner and it will help to start learning backbone and Bootstrap. I have some ideas I will try to learn and do my webapp on my own. I am wondering why you didn’t commercialize this. Small companies can buy this right and is very useful for them just like any time tracking app.

  21. demirdöküm kombi servisi November 6, 2013 at 6:38 am #

    Need small Info: Based on backbone+Bootstrap, are there any open srcd/example templates where a simple app that has support for user login/signups. I want to create an app that allows user to login and enter some details for app to work. One cant access others data. Any pointers how to go about ?? Any one else can also help me.

  22. demirdöküm servis November 6, 2013 at 6:38 am #

    Great information. I’ve been searching a long time for something like this.
    I should start now learning the aplication with the tutorial and your “Backbone Directory”, i’m new…hahaha
    Thanks a million!

  23. demirdöküm November 6, 2013 at 6:38 am #

    Great work Christophe. I’ve been getting to grips with Twitter Bootstrap 2.0 for an internal work tool the last couple of weeks and this sample is well timed.

    If you’re stuck for your next sample a backbone.js and Twitter Bootstrap CRUD app would be great ; )

  24. farmville 2 gratuit April 12, 2014 at 11:22 am #

    Every weekend i used to pay a visit this site, as i want enjoyment, for the reason that this this site conations actually good funny information too.

  25. Dusty May 23, 2014 at 12:58 am #

    I’m impressed, І must say. Rarely dо Ι come acrοss a blog that’s equally educative
    and entertaining, ɑnd let me tell үοu, you have hit the nail on the
    head. The issue iis ѕomething that too feա
    people aree speaking intelligently аbout. Now і’m very happy Ӏ stumbled
    ɑcross tis in mү search fօr something relating to this.

  26. acousticlazines43.blog.com May 26, 2014 at 6:59 pm #

    I have been browsing online more tthan 2 hours today, yet I never
    found any interestging article like yours. It’s pretty
    worth enough for me. In my view, if all webmasters and
    bloggers made good content as you did, the net
    will be much more userul than ever before.

  27. facetime for pc app May 30, 2014 at 8:50 am #

    Many claim that it is too soon for another updated and upgraded version
    of the i – Pad. Protecting your investment is important regardless of what you buy;
    it is a good idea to get a good support and ticket system even with
    the Apple Mac – Book Pro. Afloat also offers many other additional little niceties,
    like making windows transparent with a mouse gesture or multi-touch trackpad.

  28. fifa points gratuit May 31, 2014 at 10:45 pm #

    Great site. Lots of helpful info here. I’m sending it to several buddies ans also sharing in delicious.
    And naturally, thank you in your effort!

  29. wholesale cheap Dolphins jerseys June 20, 2014 at 2:52 am #

    cheap Red Wings jersey Review China

  30. Madie June 25, 2014 at 12:28 pm #

    Its not my first time to go to see this web site,
    i am browsing this web page dailly and take nice information from here all the time.

  31. kevinxu July 24, 2014 at 2:28 am #

    did same thing before, instead made a web server as websocket server but hook the websocket server into a active process which process tcp connection. Lot of fun :)

  32. Mobile websites July 28, 2014 at 11:54 pm #

    create a mobile friendly version, create your fully customized mobile site, start selling websites.
    Just for follow the link : http://todayone.com.au/

  33. DominO August 11, 2014 at 7:24 am #

    Dude. I just wanted to say: Thank you! Thank you for publishing this. Kudo`s!

  34. Vardan September 17, 2014 at 8:13 am #

    Awesome explanation.

  35. budi October 16, 2014 at 3:39 am #

    And yes this the best example ever i found on the internet….

  36. sofian November 30, 2014 at 6:13 am #

    this is great. Open my mind and the simplest guide to make a real time system.
    Thank you.

Trackbacks/Pingbacks

  1. How the Real-Time Web Will Change Everything | /danleslie » /tech /design /travel /photography - November 27, 2012

    […] Real Time Web Analytics with Node.js and Socket.IO (coenraets.org) […]

  2. Real Time Web Analytics with Node.js and Socket.IO « rg443blog - January 11, 2013

    […] Real Time Web Analytics with Node.js and Socket.IO […]

  3. Ra Puke Moana - April 28, 2014

    […] Real Time Web Analytics with Node.js and Socket.IO | Christophe Coenraets […]

Leave a Reply

css.php