Set up initial data for Django models with migrations

Problem

After you create a Django model and make its first migration, the database only has the corresponding database table in an empty state. You want to load a set of pre-defined data records on this database table.

Solution

Create an empty migration file with python manage.py makemigrations --empty <app_name>. You can hard-code the initial data set inside the migration file as Django model instances and use the RunPython method to load the data into the database. You can also have the initial data as standard SQL statements in an SQL script and use the RunSQL method to load the data into the database. Finally, you can also have the initial data in a fixture file -- Django's import/export format -- and use the RunPython method to load the data into the database.

How it works

On many occasions it can be helpful or necessary to load a set of pre-defined data records on a Django model. For example, you may already have a complete set of records for models that handle a lot of data or have data that changes very little (e.g. a Store model or a Menu model). Django allows you to load pre-defined records by either hard-coding them in Python, using a standard SQL script with SQL statements or using a fixture file which is a Django export/import format that supports either the JSON, XML or YAML formats.

The first step to load a set of pre-defined data records is to generate an empty migration file. If you're unfamiliar with the concept of a migration file in Django, see the prior recipe 'Set up Django models and understand the migrations workflow'. Listing 1 illustrates how to generate an empty migration file.

Listing 1 - Create empty Django migration file to load initial data for Django model

[user@coffeehouse ~]$ python manage.py makemigrations --empty stores
Migrations for 'stores':
  0002_auto_20150124_0507.py:

As you can see in listing 1, we generate an empty migration file for the stores app and Django creates the migration file 0002_auto_20150124_0507.py. Note the output in listing 1 assumes this is the second migration on the stores app -- hence the 0002 prefix -- but it could be any other number if you make other Django model operations after the initial 0001 migration.

Once you have an empty Django migration file like the one in listing 1, we can move forward and explore the different ways to modify this file to set up initial data for a Django model.

Hard-code predefined records in Python migration file

The simplest approach you can take is to hard-code the set of pre-defined data records and make it part of the migration file itself. Lisitng 2 illustrates a modified migration file with hard-coded model objects to load into a database.

Listing 2 - Load initial data with hard-coded data in Django migration file

# -*- coding: utf-8 -*-
from __future__ import unicode_literals

from django.db import models, migrations

def load_stores(apps, schema_editor):
    Store = apps.get_model("stores", "Store")
    store_corporate = Store(id=0,name='Corporate',address='624 Broadway',city='San Diego',state='CA',email='corporate@coffeehouse.com')
    store_corporate.save()
    store_downtown = Store(id=1,name='Downtown',address='Horton Plaza',city='San Diego',state='CA',email='downtown@coffeehouse.com')
    store_downtown.save()
    store_uptown = Store(id=2,name='Uptown',address='1240 University Ave',city='San Diego',state='CA',email='uptown@coffeehouse.com')
    store_uptown.save()
    store_midtown = Store(id=3,name='Midtown',address='784 W Washington St',city='San Diego',state='CA',email='midtown@coffeehouse.com')
    store_midtown.save()

class Migration(migrations.Migration):

    dependencies = [
        ('stores', '0001_initial'),
    ]

    operations = [
        migrations.RunPython(load_stores),
    ]

The first thing that's added to the empty migration file is the migrations.RunPython(load_stores) line in the operations[] list. The operations[] list defines a list of migration operations. And the RunPython methods runs Python code, which in this case corresponds to the load_stores method.

The load_stores method toward the top of listing 2 contains the hard-coded data records. This method first gets a reference of the Store model and then creates three different instances which are then saved to the database.

Once you make the additions illustrated in listing 2 to an empty migration file, you simply need to trigger the migration with the migrate command for the data to be loaded into the database.

SQL script with SQL statements

On other occasions you may already have a set of predefined records in an SQL script to populate a database table. Listing 3 illustrates a sample SQL script to populate the table associated with the Store model.

Listing 3 - SQL script with SQL statements

INSERT INTO stores_store (id,name,address,city,state,email) VALUES (0,'Corporate','624 Broadway','San Diego','CA','corporate@coffeehouse.com');
INSERT INTO stores_store (id,name,address,city,state,email) VALUES (1,'Downtown','Horton Plaza','San Diego','CA','downtown@coffeehouse.com');
INSERT INTO stores_store (id,name,address,city,state,email) VALUES (2,'Uptown','1240 University Ave','San Diego','CA','uptown@coffeehouse.com');
INSERT INTO stores_store (id,name,address,city,state,email) VALUES (3,'Midtown','784 W Washington St','San Diego','CA','midtown@coffeehouse.com');

By convention, it's a recommended practice in Django to name SQL scripts associated with a Django model with the model name and place them in a sub-folder named sql inside the app name. For example, the contents of listing 3 which are for the Store model in the stores app would be placed in the project folder stores/sql/store.sql. Once you have an SQL script inside a Django project's directory structure, you can take the next step to set it up as the initial data for a Django model.

Listing 4 illustrates a modified migration file to load data from the SQL script in listing 3 into a database.

Listing 4 - Load initial data with SQL script in Django migration file

# -*- coding: utf-8 -*-
from __future__ import unicode_literals

from django.db import models, migrations


def load_stores_from_sql():
    from coffeehouse.settings import PROJECT_DIR
    import os
    sql_statements = open(os.path.join(PROJECT_DIR,'stores/sql/store.sql'), 'r').read()
    return sql_statements

class Migration(migrations.Migration):

    dependencies = [
        ('stores', '0001_initial'),
    ]

    operations = [
        migrations.RunSQL(load_stores_from_sql()),
    ]
Note Loading SQL scripts requires sqlparse package

The RunSQL method used to load SQL statements relies on the sqlparse package.

So if you plan to use this functionality you need install this package (e.g. pip install sqlparse).

The first thing that's added to the empty migration file is the migrations.RunSQL(load_stores_from_sql()) line in the operations[] list. The operations[] list defines a list of migration operations. And the RunSQL methods runs SQL statements, which in this case are generated by a call to the load_stores_from_sql method.

The load_stores_from_sql method toward the top of listing 2 reads the contents of the SQL script from a relative path in the main project directory at stores/sql/store.sql and returns the SQL statements in the file. In this case, the relative path is provided by the PROJECT_DIR variable defined in Django's settings.py file and the SQL script is read using Python's standard open method. If you are unfamiliar with setting up relative paths in Django for this purpose, see the recipe Django settings.py for the real world, section 'Use dynamic absolute paths'.

Once you make the additions illustrated in listing 4 to an empty migration file, you simply need to trigger the migration with the migrate command for the data to be loaded into the database.

Django fixture file

Finally, another alternative to load initial data in a Django model is through a fixture file. A fixture file is a Django specific format used to manage the data export/import of Django models. Listing 5 illustrates a JSON fixture file to populate the table associated with the Store model.

Listing 5 - Django fixture file with JSON structure

[{
  "fields": {
    "city": "San Diego",
    "state": "CA",
    "email": "corporate@coffeehouse.com",
    "name": "Corporate",
    "address": "624 Broadway"
  },
  "model": "stores.store",
  "pk": 0
},
{
  "fields": {
    "city": "San Diego",
    "state": "CA",
    "email": "downtown@coffeehouse.com",
    "name": "Downtown",
    "address": "Horton Plaza"
  },
  "model": "stores.store",
  "pk": 1
},
{
  "fields": {
    "city": "San Diego",
    "state": "CA",
    "email": "uptown@coffeehouse.com",
    "name": "Uptown",
    "address": "1240 University Ave"
  },
  "model": "stores.store",
  "pk": 2
},
{
  "fields": {
    "city": "San Diego",
    "state": "CA",
    "email": "midtown@coffeehouse.com",
    "name": "Midtown",
    "address": "784 W Washington St"
  },
  "model": "stores.store",
  "pk": 3
}]
Note Use dumpdata to generate fixture files

Fixture files can look intimidating because they really aren't designed for a lot of editing. They are the data import/export format for Django models. So if you already added records to a database connected to a Django model and wish to make these records an initial data set, you can export the data using the dumpdata command to generate a fixture file.

For example, to generate a fixture file with all Django model records in a database you can run python manage.py dumpdata --indent=2. And to restrict a fixture file to a given model you can add the argument <app_name>.<model_name> (e.g. python manage.py dumpdata --indent=2 stores.store to limit the fixture file to instances of the Store model in the stores app)

By default, the dumpdata command outputs JSON. However, you can add the --format flag to output either XML or YAML (e.g. --format xml, --format yaml). Note that to output YAML you need to install the PyYAML package (e.g.pip install PyYAML).

By convention, it's a recommended practice in Django to name fixture files associated with a Django model with the model name and place them in a sub-folder named fixtures inside the app name. For example, the contents of listing 5 which are for the Store model in the stores app would be placed in the project folder stores/fixtures/store.json. Once you have an fixture file inside a Django project's directory structure, you can take the next step to set it up as the initial data for a Django model.

Listing 6 illustrates a modified migration file to load data from the fixture file in listing 5 into a database.

Listing 6 - Load initial data from Django fixture file in Django migration file

# -*- coding: utf-8 -*-
from __future__ import unicode_literals

from django.db import models, migrations


def load_stores_from_fixture(apps, schema_editor):
    from django.core.management import call_command
    call_command("loaddata", "stores")

class Migration(migrations.Migration):

    dependencies = [
        ('stores', '0001_initial'),
    ]

    operations = [
        migrations.RunPython(load_stores_from_fixture),
    ]

The first thing that's added to the empty migration file is the migrations.RunPython(load_stores_from_fixture) line in the operations[] list. The operations[] list defines a list of migration operations. And the RunPython methods runs Python code, which in this case corresponds to the load_stores_from_fixture method.

The load_stores_from_fixture method toward the top of listing 2 uses the call_command method to load the fixture file by simulating the command line execution of the manage.py loaddata command. The loaddata command requires an additional argument to search for a fixture file. In this case, the argument store tells Django to look for a fixture file named store in all the fixtures sub-directories for all apps.

Once you make the additions illustrated in listing 6 to an empty migration file, you simply need to trigger the migration with the migrate command for the data to be loaded into the database.