Django models and multiple databases

Back in Chapter 1, when you set up a database for a Django project, you used the DATABASES variable in settings.py to define the type and connection parameters to perform all database related operations in a Django application. Since this variable is plural, it means you can declare multiple databases in a Django project, as illustrated in listing 7-44.

Listing 7-44. Django multiple DATABASES definitions in settings.py

DATABASES = {
   'default': {
        ...
    },
   'devops': {
        ...
    },
   'analytics': {
        ...
    }
   'warehouse': {
        ...
    }
}

Listing 7-44 shows four different databases for a Django project, where the ... notation is the position in which each database's connection parameters are declared (e.g. ENGINE, NAME, and all those other described in table 1-3).

The most important aspect in listing 7-44 is the default key, which represents the database on which all Django project database related operations are made, unless otherwise specified. In the upcoming sections, I'll describe how to tell Django to perform data operations on a different databases than the default.

Now that you know how to declare multiple databases in a Django project and the importance of the default database, let's explore how to execute instructions on multiple databases from within a Django project.

Multiple databases for Django models: using

All Django model operations described in this chapter and the upcoming chapter are designed to run against the database defined in the default handle in DATABASES, or if provided, the DEFAULT_DB_ALIAS database handle. However, with multiple databases it's possible to selectively execute operations against different databases, overriding the default database.

Django models support the using keyword to indicate against which database to execute operations. There are two variations of the using keyword:

• The using option.- Django model methods such as save() and delete() support the using option to indicate against which database to save or delete records. For example, store.save(using='analytics') tells Django to save the store records to the analytics database handle, overriding the default database value.
• The using method.- The default Django model manager objects supports the using() method to indicate against which database to perform a query. For example, Item.objects.using('warehouse').all() tells Django to get all Item records from the warehouse database handle, overriding the default database value.

Multiple databases for Django tools: --database

In addition to Django models having the ability to selectively execute operations against different databases, Django tools that perform database related tasks also have the ability to specify a different database than the default database handle.

For example, manage.py commands such as migrate, dumpdata, loaddata and others described in this chapter support the --database flag to execute their logic against a different database than the default database handle. For example, mange.py migrate --database devops tells Django to perform the migration process on the devops database, overriding the default database value.

Multiple database routers: DATABASE_ROUTERS setting

Although the using and --database options offer a solution to work with multiple databases in a Django project, they both require spreading out the multi-database logic to different parts of an application (e.g. models and scripts). Django database routers offer the ability to centralize multi-database logic, so that based on the model or type of database operation, the logic is executed against a different database than the default.

Django database routers are standard Python classes that implement up to four different methods illustrated in table 7-6.

Table 7-6. Django database router methods

**hints Are additional information provided to each router method to further determine which database to use each routing case, in addition to the other input parameters.

Armed with this basic knowledge of methods used by Django database routers, let's create a custom router that stores Django models associated with core Django apps (e.g. users, admin, sessions) in one database and other Django project models in another database. Listing 7-45 illustrates a Django database router that performs this logic.

Listing 7-45. Django database router to store core app models in devops database and all other models in default database.

class DatabaseForDevOps(object):

    def db_for_read(self, model, **hints):
        if model._meta.app_label in ['auth','admin','sessions','contenttypes']:
            return 'devops'
        # Returning None is no opinion, defer to other routers or default database
        return None
    def db_for_write(self, model, **hints):
        if model._meta.app_label in ['auth','admin','sessions','contenttypes']:
            return 'devops'
         # Returning None is no opinion, defer to other routers or default database
        return None
    def allow_relation(self, obj1, obj2, **hints):
        # Allow relations between two models that are both Django core app models
        if obj1._meta.app_label in ['auth','admin','sessions','contenttypes'] and
	                    obj2._meta.app_label in ['auth','admin','sessions','contenttypes']:
            return True
        # If neither object is in a Django core app model (defer to other routers or default database)
        elif obj1._meta.app_label not in ['auth','admin','sessions','contenttypes'] or
	                 obj2._meta.app_label not in ['auth','admin','sessions','contenttypes']:
            return None
        return None
    def allow_migrate(self, db, app_label, model_name=None, **hints):
        if db == 'devops':
            # Migrate Django core app models if current database is devops
            if app_label in ['auth','admin','sessions','contenttypes']:
                return True            
            else:
                # Non Django core app models should not be migrated if database is devops
                return False
        # Other database should not migrate Django core app models            
        elif app_label in ['auth','admin','sessions','contenttypes']:
            return False
        # Otherwise no opinion, defer to other routers or default database
        return None

The first two methods in listing 7-45 -- db_for_read() and db_for_write() -- tell Django how to proceed with the read and write operations of models. In this case, both methods check the model app_name, if the app_name is either auth, admin, sessions or contenttypes -- all of which are Django core apps -- the methods return the 'devops' value which represents a database handle, as described in listing 7-44. This logic tells Django to perform all read and write operations for these type of models on the devops database. If a model does not match any of the cited app_name values, notice both methods return None, which is an indication the router class has no opinion on how to handle the model and passes control to another router class or the default database to determine where to perform a model's read/write operations.

Here it's worth mentioning you can use multiple Django database routers, with their configuration order defining their precedence. If one Django database router cannot determine or does not define a specific database on which to execute an operation or model, it moves on to the next database router, this process continues until all database routers are run out and the operation or model defaults to using the default database. This behavior has the advantage that database routers don't have to declare database routing paths for every single model and operation in a project, it's sufficient to declare routing rules for certain models or operations, and let Django use the default database as a back stop.

Turning our attention back to listing 7-45, the allow_relation() method is used to determine how to route model operations when a model contains related models. For example, if a model contains a ForeignKey field or ManyToManyField field, this relationship can cross over into another app, which in turn influences where a database table can reside. In the case of listing 7-45, the allow_relation() method indicates to allow relationship operations if the related model objects belong to either the auth, admin, sessions or contenttypes apps.

Finally, the allow_migrate() in listing 7-45 is used to define against which database migrations operations are run. In this case, it indicates that if a migrate operation is done against the devops database, it only migrate models belonging to either the auth, admin, sessions or contenttypes apps and it should ignore all other model migrations (i.e. the devops database should only contain model tables related to the auth, admin, sessions or contenttypes apps). The second part in allow_migrate() indicates that for any other database that's not devops, it should ignore model migrations for the auth, admin, sessions or contenttypes apps (i.e. the default database will not contain tables related to the auth, admin, sessions or contenttypes apps).

Once you have a database router class like the one in listing 7-45, you must declare it as part of the DATABASE_ROUTERS value list in settings.py. Assuming the database router class in listing 7-45 is located in the folder /coffeehouse/common/ in the routers.py file, then DATABASE_ROUTERS = ['coffeehouse.common.routers.DatabaseForDevOps']

As I already mentioned, the DATABASE_ROUTERS value can be a list of database routers for which all models pass through to determine on which database to perform their operations, if the DATABASE_ROUTERS list is exhausted, then the the default database is used.

Once you finish setting up the database router configuration, you must run the migrate operation on both you project databases. Once this is done, all read and write operations related to Django core app models will be done on the devops database and all other project models will be done on the default database.