You want to understand the design implications of real-time web applications and what you need to use in order to enable Django for this type of design.
Understand the principles and benefits of non-blocking & asynchronous technology. Go deeper and study software concepts like callbacks, event loops, event driven programming, the reactor pattern, the observer pattern and closures to really grasp what makes real-time web applications 'tick'. Finally, learn about a server side event driven platform like Node.js or Tornado and how to integrate it with your Django web applications.
By default, web applications are stateless due to the underlying HTTP protocol on which they work. The term stateless means the connection between a web application client and server is ephemeral (i.e. it's short lived). If you pause to think about this approach, you'll realize that for many web applications this is convenient for both clients and servers. If a client (e.g. browser) accesses 10 web applications in a successive manner, it's more efficient to quickly open and close connections to fetch information than to keep active connections, because the latter takes up client resources unnecessarily (e.g. bandwidth, memory). Similarly, if a server receives requests from 1,000 clients, it's also more efficient to quickly open and close connections than to keep active connections, because the latter also takes up limited server resources (e.g. threads, memory).
The clients and servers in real-time web applications by definition must be responsive to changes made by either party. For example, if you're reading the action of a live event (e.g. baseball game or political debate) you want to get updates as they are happening, similarly if you're chatting on line (e.g. with a customer service agent or friend) it's important to get updates as quickly as someone starts typing or if you're using a social application (e.g. Facebook or Twitter) you want to be able to get updates as soon as something of your interest is published. The crux of the matter is that to enable any type of real-time functionality using technologies that are designed for stateless communication, generally results in extraordinary efforts and sub-standard solutions. Let's walk through a greenfield Django application to exemplify this.
Your Django application starts with the basics of most modern web applications: you set up a database with a couple of dozen models, add authentication support through social platforms (e.g. Facebook, Twitter, Google), create a series of responsive templates to deliver content to browsers across multiple devices (e.g. phones, tablets, PCs), as well as design a few dozen REST services to function as an API that updates regular content and serves as the data end points for the iPhone and Android apps of the same web application. Then you get a call telling you that 'soon' the web application has to support publishing "daily specials" and 'soon after' a chat like feature for customers.
The problem with this type of pseudo real-time solution is that it's fragile and puts unnecessary load on both clients and servers. What happens if certain days "daily specials" don't happen anymore or if the admin one day publishes the "daily specials" until the afternoon ? You can prepare for missed updates and heavy request loads on servers for "daily specials" that don't exist anymore. Then there's the question of over-burdening a client's resources, does the client even want "daily specials" and what is a good interval to get updates ? 5 minutes or 30 minutes ? There are no easy answers to these questions. And have you thought about how to solve the chat feature that also has a similar but more complex real-time pattern as this problem ? Here again there are no straightforward answers.
As fragile as this pseudo real-time solution appears, you'd be surprised at how many medium to large web applications function this way and continue to operate by throwing more hardware at the problem and dealing with constant code updates to keep everything up and running. The reality is that these sub-standard solutions that require extraordinary efforts to maintain come about by trying to build something (i.e. real-time functionality) with technology (i.e. a stateless medium) that isn't designed for it.
So what technology is desirable for real-time web applications ? Non-blocking & asynchronous technology.
|The evolution of real-time web applications: A blast from the past to today|
In 2008 the release of the first HTML5 draft marked an important milestone because it was the first time a mainstream web technology converged to support real-time related features (e.g. video and audio). Although it would take HTML5 almost 7 years, in 2015 major content providers (e.g. YouTube, Netflix and Facebook) phased out Flash -- a favorite technology for real-time solutions -- in favor of HTML5, a trend that now appears to be irreversible.
Non-blocking & asynchronous refer to a type of software design, that can be either already built in to third party software (e.g. library or web server) or must be incorporated into your own software (e.g. call sequence or function). Since these concepts are key to understanding a now relatively large software ecosystem, let's break them down.
Non-blocking means something doesn't have to wait, which for most web applications doesn't hold true. When a web application enters a logical sequence (e.g. fetches info from a database or computes business logic) the overall flow blocks until the sequence is finished, just as a web-server that accepts an incoming request has to keep a resource (e.g. thread) blocked until it's ready to send a response, before re-using the resource for another request. This type of blocking is often undetectable in web applications because it lasts milliseconds, but it can grow into seconds in certain circumstances which of course goes against the very premise of real-time behavior.
Asynchronous means something can be deferred without influencing the flow of the calling party, which again for most web application's doesn't hold true. When a call is made for a web application page, a browser acts synchronously and doesn't do anything else until it receives a response. The same thing happens when a call is made to a database, the caller acts synchronously and waits until the database returns a result. One asynchronous behavior you've probably used or at least heard of in web applications is: AJAX. AJAX happens when a web application on the client side requests additional data from the server side to complement its UI, while the request takes place a user can keep interacting with the UI without being affected by this additional work being done in the background.
The terms non-blocking and asynchronous are often used together, mainly because most non-blocking design is asynchronous and vice versa. Although there are subtle differences between the two terms, I won't dive into a semantic discussion as there are already many on line resources that do this pretty well. But now that you do know the definition of these two terms and why they're desirable for real-time web applications, the next step is to understand how to incorporate them into web applications.
The bad news is that most likely all of the Django projects you've done and the Django references you've read on line (e.g. documentation, advice & code snippets) contain 90% to 95% of blocking and synchronous design. The good news is that by learning a few techniques and leveraging a series of tools you can easily incorporate non-blocking and asynchronous design into Django applications. Next, I'll describe some of the techniques required to achieve non-blocking and asynchronous design: callbacks & event loops.
A callback is a piece of logic -- generally contained in a function -- that is designed to run after a certain event takes place. In more mundane terms, a callback works just like those buzzers restaurants give out to notify you when an order or table is ready, so you can leave the restaurant (i.e. non-blocking) and go do whatever you want without interference (i.e. asynchronously) and get a callback when the order or table is ready. The most common usage of callbacks in web applications is in client side design through AJAX: when an event (e.g. click) triggers the execution of a piece of logic (e.g. UI update) without interfering with the overall flow (i.e. it happens in the background).
Now think about callbacks in the context of the server side. The usefulness of callbacks on the server side is the same as on the client side, it avoids blocking which contributes to real-time behavior or the perceived 'snappiness' of a process. Listing 1 illustrates the pseudo-code for a typical server side sequential (i.e. blocking, non-asynchronous) design.
Listing 1 illustrates how inside the
main method a call is made to multiple methods where each method accepts as input the results of the prior method, this is a typical design for most server side logic. The issue with the pseudo-code in listing 1 is the
main method won't return a result until all of its internal logic (i.e. the three methods) run, so if the database is having a 'bad day', the business logic is intensive or the template is complex, the execution of the
main method will vary widely to the upside, which is bad in terms of real-time behavior.
Now let's take a look at listing 2 which shows the equivalent pseudo-code from listing 1 but this time using callbacks.
Notice in listing 2 how the
main method immediately returns a result but at the top initiates the logic for the same three methods in listing 1. The methods are nested as callbacks, so that when the
database_logic method finishes it uses its result to call the
business_logic method and then when the
business_logic method finishes it uses its result to call the
template_rendering method. The advantage of this design is the caller of the
main method can go on doing something else while the sequence finishes.
Although listing 2 represents an extreme example of callbacks, it's fairly easy to end up with this type of design if you're doing it with a sequential mindset and don't familiarize yourself with callback good practices and common pitfalls. So yes, callbacks can give you increased performance and real-time behavior, just be careful not to do it while sacrificing other equally important software traits like readability, testing and maintenance.
Now think about event loops in the context of the server side. The usefulness of event loops on the server side is the same as on the client side, it avoids blocking where by an event loop listens for new events (e.g. data in, data out) and reacts based on the event, instead of having resources (e.g. threads) wait idly until a sequence finishes. So similar to callbacks, event loops contribute to real-time behavior or the perceived 'snappiness' of a process. One of the papers that set forth the foundations of this design was the C10K problem by Dan Kegel (1999).
Because event loops are more abstract and there are many on line papers that explain this concept alone -- some as large as 100+ pages -- I won't even try to explain the technical differences between this type of design and regular server side processing (e.g. Apache web server). The C10K paper mentioned previously offers one of the best and shortest introductions into server side event based processing if you're interested in more technical details.
Fortunately, the task of creating a server side event loop is something that's already been solved and you don't need to write one from scratch yourself. Nevertheless, just like the use of callbacks, you'll need to understand how to work with this type of technology to use it effectively. But unlike callbacks which are a concept you can apply across programming languages, event loops on the server-side do require you select a "run-time" to provide the core event loop and other functionalites (e.g. package management, API). In the next section I'll describe two server side event driven "run-times" that I believe are the most natural choices to use Django applications.
|Are callbacks and event loops all you need to know about real-time web technology ?|
No, but callbacks and event loops are core concepts on almost anything associated with real-time web technology. Now a days for real-time design, you'll rarely read more than a few paragraphs without encountering some use or mention of callbacks. As for event loops, while the term is not as broadly cited, it's what underpins the server side "run-times" that are widely referenced as a solution for real-time web applications, so it's a key concept.
If you want explore additional real-time web technology concepts I would suggest you read up on: event driven programming, the reactor pattern, the observer pattern and closures.
Node.js is really many things rolled-up into one which can make it downright confusing to understand if you try to compare it with other technologies you already know and understand. Node.js is a run-time environment for server side applications, this means it's like the Apache web server, it just uses an event loop design which makes it different, so ✔ web server. Node.js has it own API for things like HTTP operations, processing buffers and DNS queries, among other things, this means it's framework like Django or jQuery, so ✔ framework. Node.js also comes with a package manager called
npm -- just like Python's
pip -- this means a lot of modules/libraries/frameworks get written for Node.js just like they are for Python, PHP or Ruby, so ✔ platform.
As you can see Node.js functionalities cut across many areas -- which I believe is one of its main appeals -- making it a one stop solution that functions as a web server, framework and platform with over 250,000 modules/packages/frameworks to choose from. But let's keeps things into perspective, as appealing as Node.js is, using all of its features requires you to re-learn a lot of things in a Node.js specific manner. In the upcoming recipes I'll show you how it's perfectly reasonable to use certain parts of Node.js and leverage them with Django.
I chose to expand on Node.js mainly because of its broad appeal, however, this doesn't mean Node.js is the only platform that provides these sets of features on the server side. In the Python world, there's a project named Tornado that aims to provide similar features to Node.js. Just like Node.js, Tornado is a web server, a framework with its own API, as well as a platform with some modules/libraries designed just for Tornado. However, the Tornado developer community and its modules/libraries are only a fraction of what Node.js has to offer, making Tornado a weaker choice in terms of a one stop solution for real-time web applications.
Nevertheless, the appeal of Tornado resides in that fact it's written in Python. If you're already invested in Django, the integration of certain parts of Tornado with Django should come much easier since you're using the same language across the board. In the upcoming recipes I'll also show you how it's perfectly reasonable to use certain parts of Tornado and leverage them with Django.