What are real-time web applications and what's required to create them in Django

Problem

You want to understand the design implications of real-time web applications and what you need to use in order to enable Django for this type of design.

Solution

Understand the principles and benefits of non-blocking & asynchronous technology. Go deeper and study software concepts like callbacks, event loops, event driven programming, the reactor pattern, the observer pattern and closures to really grasp what makes real-time web applications 'tick'. Finally, learn about a server side event driven platform like Node.js or Tornado and how to integrate it with your Django web applications.

How it works

By default, web applications are stateless due to the underlying HTTP protocol on which they work. The term stateless means the connection between a web application client and server is ephemeral (i.e. it's short lived). If you pause to think about this approach, you'll realize that for many web applications this is convenient for both clients and servers. If a client (e.g. browser) accesses 10 web applications in a successive manner, it's more efficient to quickly open and close connections to fetch information than to keep active connections, because the latter takes up client resources unnecessarily (e.g. bandwidth, memory). Similarly, if a server receives requests from 1,000 clients, it's also more efficient to quickly open and close connections than to keep active connections, because the latter also takes up limited server resources (e.g. threads, memory).

To enhance the level of performance in this world of stateless communication, clients generally rely on things like cookies -- to identify themselves across multiple requests -- and caches to keep local copies of a web application's parts (e.g. CSS, JavaScript). Similarly, servers rely on databases -- to keep client information indefineltly -- as well as caches to maintain state on a user's behalf for a given time. In this same context of stateless communication, let's explore some of the inherent problems of using this same approach to create real-time web applications.

The clients and servers in real-time web applications by definition must be responsive to changes made by either party. For example, if you're reading the action of a live event (e.g. baseball game or political debate) you want to get updates as they are happening, similarly if you're chatting on line (e.g. with a customer service agent or friend) it's important to get updates as quickly as someone starts typing or if you're using a social application (e.g. Facebook or Twitter) you want to be able to get updates as soon as something of your interest is published. The crux of the matter is that to enable any type of real-time functionality using technologies that are designed for stateless communication, generally results in extraordinary efforts and sub-standard solutions. Let's walk through a greenfield Django application to exemplify this.

Your Django application starts with the basics of most modern web applications: you set up a database with a couple of dozen models, add authentication support through social platforms (e.g. Facebook, Twitter, Google), create a series of responsive templates to deliver content to browsers across multiple devices (e.g. phones, tablets, PCs), as well as design a few dozen REST services to function as an API that updates regular content and serves as the data end points for the iPhone and Android apps of the same web application. Then you get a call telling you that 'soon' the web application has to support publishing "daily specials" and 'soon after' a chat like feature for customers.

To solve the first problem you decide to add a large button to the application that says "Click to get the daily specials!", the button is hooked-up so that when a user clicks on it, a call is made to a REST service that fetches the "daily specials" and updates the user's screen, well that was easy! Except you then get another call telling you why is it necessary for users to click on a button to get updates, when Twitter or Facebook just present updates automatically without requiring any user action. After a few meetings and talking to the head of "daily specials", you're informed specials are only offered in the mornings. Given that you already have a REST service that publishes "daily specials", you decide for maximum re-usability and add a little JavaScript snippet that automatically (i.e. behind the scenes) contacts the REST service in the mornings to get "daily specials" without the user doing anything, well that was also relatively easy! But what can go wrong ?

The problem with this type of pseudo real-time solution is that it's fragile and puts unnecessary load on both clients and servers. What happens if certain days "daily specials" don't happen anymore or if the admin one day publishes the "daily specials" until the afternoon ? You can prepare for missed updates and heavy request loads on servers for "daily specials" that don't exist anymore. Then there's the question of over-burdening a client's resources, does the client even want "daily specials" and what is a good interval to get updates ? 5 minutes or 30 minutes ? There are no easy answers to these questions. And have you thought about how to solve the chat feature that also has a similar but more complex real-time pattern as this problem ? Here again there are no straightforward answers.

As fragile as this pseudo real-time solution appears, you'd be surprised at how many medium to large web applications function this way and continue to operate by throwing more hardware at the problem and dealing with constant code updates to keep everything up and running. The reality is that these sub-standard solutions that require extraordinary efforts to maintain come about by trying to build something (i.e. real-time functionality) with technology (i.e. a stateless medium) that isn't designed for it.

So what technology is desirable for real-time web applications ? Non-blocking & asynchronous technology.

Note The evolution of real-time web applications: A blast from the past to today

Web applications and real-time functionality up until a few years ago had always been done with plug-ins -- on the client side -- and expensive proprietary server software on the server side. In the 1990's support for real-time functionality on the web started through applets -- a Java technology -- that would later be overtaken by Flash -- from Adobe. During this same time, mainstream web application technology also underwent its own evolution: on the client side with various HTML & CSS versions, JavaScript AJAX and a broadening of client devices (e.g. phones, tablets); on the server side the first Perl CGI scripts morphed into Java servlets & PHP apps, REST services came to light and a series of storage technologies emerged over the classic RDBMS (e.g. document databases, key-value databases).

In 2008 the release of the first HTML5 draft marked an important milestone because it was the first time a mainstream web technology converged to support real-time related features (e.g. video and audio). Although it would take HTML5 almost 7 years, in 2015 major content providers (e.g. YouTube, Netflix and Facebook) phased out Flash -- a favorite technology for real-time solutions -- in favor of HTML5, a trend that now appears to be irreversible.

Also starting in 2008 JavaScript saw an explosion of frameworks pushing web application MVC functionality -- once reserved to the server side -- to the client, leading to yet another convergence in real-time related functionality. With logic & data once required to run on the server now run on the client, it lead to much quicker & real-time like functionality on clients. The JavaScript innovation cycle continued not only with MVC functionality on the client, but also spilled onto the server side where it's now feasible to run JavaScript or a combination of other classical server side languages to achieve real-time functionality that once required expensive proprietary server software.

Non-blocking & asynchronous technology

Non-blocking & asynchronous refer to a type of software design, that can be either already built in to third party software (e.g. library or web server) or must be incorporated into your own software (e.g. call sequence or function). Since these concepts are key to understanding a now relatively large software ecosystem, let's break them down.

Non-blocking means something doesn't have to wait, which for most web applications doesn't hold true. When a web application enters a logical sequence (e.g. fetches info from a database or computes business logic) the overall flow blocks until the sequence is finished, just as a web-server that accepts an incoming request has to keep a resource (e.g. thread) blocked until it's ready to send a response, before re-using the resource for another request. This type of blocking is often undetectable in web applications because it lasts milliseconds, but it can grow into seconds in certain circumstances which of course goes against the very premise of real-time behavior.

Asynchronous means something can be deferred without influencing the flow of the calling party, which again for most web application's doesn't hold true. When a call is made for a web application page, a browser acts synchronously and doesn't do anything else until it receives a response. The same thing happens when a call is made to a database, the caller acts synchronously and waits until the database returns a result. One asynchronous behavior you've probably used or at least heard of in web applications is: AJAX. AJAX happens when a web application on the client side requests additional data from the server side to complement its UI, while the request takes place a user can keep interacting with the UI without being affected by this additional work being done in the background.

The terms non-blocking and asynchronous are often used together, mainly because most non-blocking design is asynchronous and vice versa. Although there are subtle differences between the two terms, I won't dive into a semantic discussion as there are already many on line resources that do this pretty well. But now that you do know the definition of these two terms and why they're desirable for real-time web applications, the next step is to understand how to incorporate them into web applications.

The bad news is that most likely all of the Django projects you've done and the Django references you've read on line (e.g. documentation, advice & code snippets) contain 90% to 95% of blocking and synchronous design. The good news is that by learning a few techniques and leveraging a series of tools you can easily incorporate non-blocking and asynchronous design into Django applications. Next, I'll describe some of the techniques required to achieve non-blocking and asynchronous design: callbacks & event loops.

Callbacks and event loops

A callback is a piece of logic -- generally contained in a function -- that is designed to run after a certain event takes place. In more mundane terms, a callback works just like those buzzers restaurants give out to notify you when an order or table is ready, so you can leave the restaurant (i.e. non-blocking) and go do whatever you want without interference (i.e. asynchronously) and get a callback when the order or table is ready. The most common usage of callbacks in web applications is in client side design through AJAX: when an event (e.g. click) triggers the execution of a piece of logic (e.g. UI update) without interfering with the overall flow (i.e. it happens in the background).

Now think about callbacks in the context of the server side. The usefulness of callbacks on the server side is the same as on the client side, it avoids blocking which contributes to real-time behavior or the perceived 'snappiness' of a process. Listing 1 illustrates the pseudo-code for a typical server side sequential (i.e. blocking, non-asynchronous) design.

Listing 1 - Pseudo-code with sequential design


database_logic() { 
  # DB logic
}
 
business_logic() { 
  # Business logic
}
 
template_rendering() { 
  # Template logic
}

# Sequential flow
main(user) {
   db_data = database_logic(user)
   bl_data = business_logic(db_data)
   response = template_rendering(bl_data)
   ps = "Sorry for the wait, that may have taken longer than expected"
   return response + ps
}                                         

Listing 1 illustrates how inside the main method a call is made to multiple methods where each method accepts as input the results of the prior method, this is a typical design for most server side logic. The issue with the pseudo-code in listing 1 is the main method won't return a result until all of its internal logic (i.e. the three methods) run, so if the database is having a 'bad day', the business logic is intensive or the template is complex, the execution of the main method will vary widely to the upside, which is bad in terms of real-time behavior.

Now let's take a look at listing 2 which shows the equivalent pseudo-code from listing 1 but this time using callbacks.

Listing 2 - Pseudo-code with callback design


# Callback flow
main(user) { 
   database_logic(user, function(db_data) { 
           # DB logic
     business_logic(db_data, function(bl_data) { 
             # Business logic
        template_rendering(bl_data, function(response) { 
                 # Template logic
                 return end_response
        })
     }) 
   });
   ps = "We're preparing the results, look at this in the meantime..."
   return ps
}

Notice in listing 2 how the main method immediately returns a result but at the top initiates the logic for the same three methods in listing 1. The methods are nested as callbacks, so that when the database_logic method finishes it uses its result to call the business_logic method and then when the business_logic method finishes it uses its result to call the template_rendering method. The advantage of this design is the caller of the main method can go on doing something else while the sequence finishes.

Between listing 1 and listing 2, which one do you feel is simpler to write ? If you said listing 2, congratulations! You're a natural for non-blocking & asynchronous design. If you said listing 1 don't feel too bad, you're in my camp and what is likely a majority. Even though listing 2 delivers on increased performance and real-time behavior, it does so at the expense of readability, testing and maintenance, characteristics that are golden once a piece of software starts to grow. In fact, the use of callbacks can get so bad there's even a dedicated site called callbackhell.com that describes how to avoid some callback pitfalls and incorporate callback best practices -- note this last site is JavaScript specific, but offers a good conceptual guide.

Although listing 2 represents an extreme example of callbacks, it's fairly easy to end up with this type of design if you're doing it with a sequential mindset and don't familiarize yourself with callback good practices and common pitfalls. So yes, callbacks can give you increased performance and real-time behavior, just be careful not to do it while sacrificing other equally important software traits like readability, testing and maintenance.

An event loop is a construct that listens for events -- sometimes called messages -- and is charged with coordinating the logic associated with each event. Event loops in most circumstances are just there and abstracted away from direct use (i.e. you won't typically access an event loop). Event loops are most popular in UI designs, where the bulk of their logic is driven by events (e.g. mouse clicks, keyboard strokes, screen taps) that trigger certain actions. In web applications, event loops are practically a daily affair of client side design, if you've ever created logic triggered by JavaScript events like onclick, onmouseover or onkeydown then you've relied on JavaScript's event loop to manage these actions.

Now think about event loops in the context of the server side. The usefulness of event loops on the server side is the same as on the client side, it avoids blocking where by an event loop listens for new events (e.g. data in, data out) and reacts based on the event, instead of having resources (e.g. threads) wait idly until a sequence finishes. So similar to callbacks, event loops contribute to real-time behavior or the perceived 'snappiness' of a process. One of the papers that set forth the foundations of this design was the C10K problem by Dan Kegel (1999).

Because event loops are more abstract and there are many on line papers that explain this concept alone -- some as large as 100+ pages -- I won't even try to explain the technical differences between this type of design and regular server side processing (e.g. Apache web server). The C10K paper mentioned previously offers one of the best and shortest introductions into server side event based processing if you're interested in more technical details.

Fortunately, the task of creating a server side event loop is something that's already been solved and you don't need to write one from scratch yourself. Nevertheless, just like the use of callbacks, you'll need to understand how to work with this type of technology to use it effectively. But unlike callbacks which are a concept you can apply across programming languages, event loops on the server-side do require you select a "run-time" to provide the core event loop and other functionalites (e.g. package management, API). In the next section I'll describe two server side event driven "run-times" that I believe are the most natural choices to use Django applications.

Note Are callbacks and event loops all you need to know about real-time web technology ?

No, but callbacks and event loops are core concepts on almost anything associated with real-time web technology. Now a days for real-time design, you'll rarely read more than a few paragraphs without encountering some use or mention of callbacks. As for event loops, while the term is not as broadly cited, it's what underpins the server side "run-times" that are widely referenced as a solution for real-time web applications, so it's a key concept.

If you want explore additional real-time web technology concepts I would suggest you read up on: event driven programming, the reactor pattern, the observer pattern and closures.

Node.js and Tornado

Node.js is a server side JavaScript run-time based on the same V8 engine used by Google's Chrome browser. If this is the first time you read about Node.js, it may surprise you to see why with an abundance of server side programming languages (e.g. Python, PHP, Java), someone thought it was a good idea to run JavaScript -- a pillar of client side web development -- on the server side.

The inspiration for Node.js came when its creator saw a file upload task and realized the browser had to contact to server side constantly to get a real-time update on how much of the file had been uploaded -- a process that's inline with the unresponsive behavior I've referenced from the start. The problem in this case was solved by designing a server side engine capable of providing asynchronous & non-blocking behavior combining the V8 JavaScript engine, an event loop and a low-level I/O API. Thus Node.js was born in 2009 and to say it has gained in popularity since then would be an understatement.

Node.js is really many things rolled-up into one which can make it downright confusing to understand if you try to compare it with other technologies you already know and understand. Node.js is a run-time environment for server side applications, this means it's like the Apache web server, it just uses an event loop design which makes it different, so ✔ web server. Node.js has it own API for things like HTTP operations, processing buffers and DNS queries, among other things, this means it's framework like Django or jQuery, so ✔ framework. Node.js also comes with a package manager called npm -- just like Python's pip -- this means a lot of modules/libraries/frameworks get written for Node.js just like they are for Python, PHP or Ruby, so ✔ platform.

As you can see Node.js functionalities cut across many areas -- which I believe is one of its main appeals -- making it a one stop solution that functions as a web server, framework and platform with over 250,000 modules/packages/frameworks to choose from. But let's keeps things into perspective, as appealing as Node.js is, using all of its features requires you to re-learn a lot of things in a Node.js specific manner. In the upcoming recipes I'll show you how it's perfectly reasonable to use certain parts of Node.js and leverage them with Django.

I chose to expand on Node.js mainly because of its broad appeal, however, this doesn't mean Node.js is the only platform that provides these sets of features on the server side. In the Python world, there's a project named Tornado that aims to provide similar features to Node.js. Just like Node.js, Tornado is a web server, a framework with its own API, as well as a platform with some modules/libraries designed just for Tornado. However, the Tornado developer community and its modules/libraries are only a fraction of what Node.js has to offer, making Tornado a weaker choice in terms of a one stop solution for real-time web applications.

Nevertheless, the appeal of Tornado resides in that fact it's written in Python. If you're already invested in Django, the integration of certain parts of Tornado with Django should come much easier since you're using the same language across the board. In the upcoming recipes I'll also show you how it's perfectly reasonable to use certain parts of Tornado and leverage them with Django.