Migrations in Django

In this article, we'll craft a foundational app model and delve into Django migrations in Python. Migration in Django translates model changes into database schema updates. Django generates migration files within a designated folder, mapping each table to its corresponding model to establish the table schema efficiently.

The Commands

Django offers several commands that are useful for operations related to migration. We can employ these commands once a model has been created.

• migrate :
  It builds tables by the migration file's given schema.
• makemigrations :
  Django makemigrations is used to produce a migration file with code for a model's tabular schema.
• sqlmigrate :
  It is utilized to display the applied migration's raw SQL query.
• showmigrations :
  Each migration is listed along with its current status.

Backend Support

All Django-shipped backends and any third-party backends that have built-in support for schema alteration are compatible with migrations (done via the SchemaEditor class).

Workflow

Django can make your migrations. Add a field and remove a model, for example, then run Django makemigrations after making changes to your models :

A fresh set of Django migrations will be created after a scan of your models, and a comparison with the versions currently present in your migration files. Read the output carefully to find out what django makemigrations believe you have changed. It could be better and might not be picking up on complex changes, as you would hope.

To ensure that your new django migrations files function as expected, you could perhaps apply them to your database after receiving them :

When the migration has been implemented, add the migrations along with the updated models to your version control system as a single commit. This will ensure that other developers (or your production servers) who check out the code will receive both the updated models and the corresponding django migration at the same time.

Use the Django makemigrations --name option to replace the generated name for the migration(s) with a meaningful one :

Version Control

Because Django migrations are maintained in version control, it is possible for two developers to concurrently commit to a migration to the same app, resulting in two migrations with the same number.

The numbers are merely for developers' convenience, Django is only concerned, so each migration has a unique name. When there are two new django migrations for the same app that aren't ordered, it can be identified because they specify in the file which other migrations they depend on, including earlier django migrations within the same app.

When this occurs, Django will alert you and present you with a few options. If it is secure enough, this should automatically linearize the two migrations for you. . If not, you'll need to modify the Django migrations manually, don't worry, this is a simple process covered in more detail in the section below on migration files.

Transactions

All migration operations will, by default, take place within a single transaction on databases that support DDL transactions (SQLite and PostgreSQL). In comparison, if a database (such as MySQL or Oracle) does not support DDL transactions, then all operations will proceed without a trade.

The atomic attribute can be set to False to stop migration from taking place in a transaction.

Consider this :

Additionally, you can use atomic() or the atomic=True argument to RunPython to execute specific migration steps as part of a transaction. For more information, see Non-atomic migrations.

Dependencies

Even though Django migrations are per-app, the tables and relationships that your models imply are too complex to be created for a single app at a time. If you make a Django migration that depends on another Django migration to function, such as when you add a ForeignKey from your books app to your authors app, the resulting migration will be dependent on the migration in authors.

According to the aforementioned, the migration that generates the table that the ForeignKey references, the authors migration, runs first when the migrations are executed, and then the migration that creates the ForeignKey column runs second and creates the constraint. If this didn't occur, the migration would attempt to make the ForeignKey column without the table it is referencing already existing, and your database would throw an error.

Often these Django migrations operations where you are restricted to a single app are affected by this dependency behavior. Limiting to one app (in either Django makemigrations or migrate) is a best-efforts promise rather than a guarantee, any additional apps required to get dependencies will be used accurately.

Applications without Django migrations cannot be related to them through ForeignKey, ManyToManyField, etc. Although it is not supported, it may occasionally work.

Migration Files

The term "migration files" refers to the on-disk format in which migrations are stored. The above files are regular Python files written in such a declarative manner with a predetermined object layout.

An example of a simple Django migration file would be :

When loading a migration file (as a Python module), Django searches for a subclass of django.db.migrations. Migration is known as Migration. This object is then examined for four attributes, just two of which are typically used :

Dependencies are a rundown of the migrations that this one tends to depend on.

Operations are a collection of classes that define the actions that this migration takes.

The operations are crucial, which are a set of declarative commands that specify what schema changes should be made. Django scans them, creates an in-memory depiction of every schema change made to every app, and then uses it to create the SQL that makes the changes to the schema.

Django establishes the state of your models at the time you last executed Django makemigrations by iterating through each change on a set of models in memory in order. This in-memory structure is also used to determine the differences between your models as well as the current state of your migrations. The models in these models are then compared to those in your models.py files to determine what has been changed.

Although manually writing migration files should be rare, if ever necessary, it is entirely possible to do so if necessary. Be bold and edit them if necessary because some of the more complicated operations really aren't auto-detectable and can only be obtained through a hand-written migration.

Custom Fields

Changing the number of positional arguments in a custom field that has already been migrated will cause a TypeError. The modified __init__ method will be called by the old migration with the old signature. Therefore, if a new argument is required, consider creating a keyword argument as well as adding something such as assert, and argument_name in the constructor's kwargs.

Model Managers

In RunPython operations, managers are available and, therefore, can alternatively be serialized into other migrations. The use of the manager class in the django migrations attribute is intended to accomplish the following :

To make a manager class dynamically generated by the from_queryset() function importable, you must be descended from the generated class :

Initial Migrations

Migration.initial :

The migrations that produce an app's initial set of tables are referred to as "initial migrations". An app typically only has one initial migration, but it might have two or more in certain situations with complicated model dependencies.

A class attribute identifies initial django migrations on the migration class with the value initial = True. In the absence of an initial class attribute, a migration will be regarded as "initial" if this is the first migration with in-app (For example, if it is independent of any other migration within the same application)

These initial django migrations receive special treatment when the migrate —fake-initial option is used. Django verifies that all tables created during an initial migration before continuing (the CreateModel operation) are already present in the database and fake-applies the migration if they do. Similar to this, Django checks that all relevant columns already exist in the database before applying an initial migration that adds one or even more fields (AddField operation), and if they do, it fake-applies the migration. Initial migrations are handled the same as any other migration without —fake-initial.

History Consistency

When two development branches have been combined, manually linearise migrations might be necessary. You might unintentionally create a historical state in which a django migration has indeed been applied while some of its dependencies haven't if you're editing django migrations dependencies. Django won't allow migrations to run or new migrations to be made until the dependencies are fixed because this is a clear sign that they are incorrect. The allow_migrate() method of database routers can be used to manage which databases Django makemigration checks for consistent history when multiple databases are being used.

Adding Migrations to Apps

Folks could indeed add migrations by running Django makemigrations after making some changes because new apps are already set up to accept migrations.

You must make your app use Django migrations if it already has models as well as database tables but doesn't yet have migrations (for example, because you built it using an earlier version of Django). To do this, run the following command :

Your app will generate a new preliminary Django migration as a result. Now, when you run python manage.py migrate —fake-initial, Django will recognize that you have an initial migration as well as the tables it desires to create by now exist, marking the migration as applied. Since the tables it wants to create already exist, the command would fail without the migrate —fake-initial flag.

Keep in mind that this only functions under the following two conditions :

• Since you created their tables, you have not altered your models. Making the initial migration first is necessary for migrations to function because Django compares changes to migration files rather than the database.
• Django won't be able to tell that your database doesn't match your models because you haven't manually edited it; instead, you'll get errors when migrations try to change those tables.

Reversing Migrations

With migrate, migrations can be undone by passing the previous migration's number. For instance, books.0003 reverse migration

Just use the name zero to undo all django migrations that have been applied to an app.

If a Django migration includes any irreversible operations, it is irreversible. Reversing such migrations will cause IrreversibleError to be raised.

Historical Models

When you use migrations, Django uses older versions of your models that are stored within migration files. Use specific historical model versions instead of directly importing them if your database routers have allow_migrate methods or if you write Python code that uses the RunPython operation.

These historical models won't have any particular methods that you've defined because it's impossible to serialize arbitrary Python code. However, they will share the same fields, relationships, managers (limited to those with use_in_migrations = True), and Meta options (also versioned so that they may be different from your current ones).

The functions and classes must be kept around for as long as a migration references them because references to them are serialized in migrations. References to functions are made in field options like upload_to and limit_choices_to as well as model manager declarations with managers having use_in_migrations = True. As migrations directly import these, custom model fields must also be preserved.

As a result of this and the fact that the model's concrete base classes are stored as pointers, base classes must always be kept around if a migration contains a reference to them. Plus, since these base classes' methods and managers naturally inherit, you can choose to move them into your superclass if you absolutely must have access to them.

Considerations While Removing Model Fields

If custom model fields used in previous django migrations are removed from your project or third-party app, a similar problem will occur, as was the case with the "references to historical functions" considerations discussed in the previous section.

Django offers a few model field attributes to help with model field deprecation that use the system checks framework to address this issue.

To your model field, add the system_check_deprecated_details attribute as shown below :

Data Migration

If you want, you can use migrations and change the database schema to modify the data within the database.

Data migrations are commonly referred to as "migrations", and it is best to write them as separate migrations that sit alongside your schema migrations.

Unlike schema migrations, which Django can generate for you automatically, data migrations are relatively simple to write. RunPython is the primary operation you use for data migrations, and migration files in Django are composed of operations.

Create an empty Django migrations file that you can use as a starting point (Django will add dependencies, the proper location for the file, and a name for you) :

Then, open the file and it should appear as follows :

The last step is to create a new function and instruct "RunPython" to use it. The callable that "RunPython" expects as its argument accepts two arguments : a SchemaEditor to modify database schema manually and an app registry with historical variants among all your models loaded into it to match where in your history the Django migrations sit.

Let's create a django migration that adds first name and last name values together to our new name field (now that we've come to our senses and realized that not all people have last_name and first_names). All that is required is that we iterate over rows using the historical model :

The data migration will then take place in place alongside other django migrations when we use python manage.py migrate as usual.

When migrating backward, the logic you want to run can be passed as a second callable to RunPython. Migration back will raise an exception if this callable is skipped.

Accessing Models from Other Apps

The dependencies attribute of the migration should contain the most recent django migrations of every app that is involved if you're composing a RunPython function which seems to be using models from applications other than those in which the migration is located, otherwise, you might encounter an error like this: When attempting to fetch the model within RunPython function using apps.get_model(), a LookupError: No installed app with label 'myappname' is returned.

In the example below, a Django migration in app1 requires the use of models in app2. Other than the fact that move m1 will be required to access models both from apps, we are not concerned with move_m1's specific implementation. As a result, we've added a dependency that details app2's most recent migration :

More Advanced Migrations

Squashing Migrations

The migration code is optimized to handle hundreds of them at once without much slowdown, so you are encouraged to perform Django migrations without restriction and not worry about how many you have. Squashing comes into play when you eventually want to reduce your migration count from several hundred to just a few.

Squshing is the process of condensing an existing set of many migrations to a single migration (or occasionally a few Django migrations) that still represents the same changes.

Django accomplishes this by taking all of your current Django migrations, extracting their Operations, and ordering them all in a list. The list is then run through an optimizer to try and shorten it. For instance, it is aware that the operations CreateModel and DeleteModel cancel out each other and that AddField can be incorporated into CreateModel.

The amount of operation sequence that can be reduced depends on how intertwined your models are and whether you have any RunSQL or RunPython operations (which cannot be optimized unless they are marked as elidable). After that, Django will write the operation sequence back into a new set of migration files.

These files are labeled to indicate that they replace previously-squashed Django migrations, allowing them to coexist with older migration files. Django will intelligently switch between them based on your location in time. The set migrations you squashed will continue to be used if you are only partially through them until they reach their conclusion, at which point it will switch to the squashed history. New installations, however, will use the new squashed migration and omit the older ones.

This makes it possible to remove outdated systems that are currently in use without causing any harm. It is advised to squash, keep the old files, commit, and release, then wait until all systems have been updated with the new release (or, if you're working on a third-party project, make sure your users upgrade releases in the correct order without skipping any), remove the old files, commit, and perform a second release.

This is supported by the command squashmigrations, run it with the app label as well as django migrations name you would like to squash up to, and it will get to work :

If you prefer a custom name for the squashed migration to one that is generated automatically, use the squashmigrations —squashed-name option.

Be aware that Django's model interdependencies can become very intricate, and squashing may cause migrations to fail, either because they were improperly optimized (in whose case you can try again with —no-optimize, though you should furthermore report an issue) or because they encountered a CircularDependencyError, which you can manually fix.

Each of the ForeignKeys within the circular dependency loop should be broken out into a separate migration, and the dependency of the other app should be moved along with it, to manually fix a CircularDependencyError.When asked to create brand-new migrations from your models, see how django makemigrations handle the issue if you're unsure. Squashmigrations will indeed be updated to try to fix these issues on its own in a future Django release.

Once your migration has been squashed, you should commit it along with the migrations it replaces. Distribute this change to every instance of the applications you are currently running, and help ensure they all run migrate so that the change is stored in their databases.

The squashed Django migrations must then be converted to a regular migration by :

• Deleting every file that it replaces during migration.
• Update all migrations so that they rely on the squashed migration rather than the deleted migrations.
• Deleting the replaces attribute from the migration's squashed migration's Migration class (this is how Django tells that it is a squashed migration).

Serializing Values

Django must serialize your models' current state into a file to write migrations, which are Python file formats usually contain the old definitions of their models.

There isn't any Python standard for how a value can be converted back into code (repr() only tends to work for basic values and doesn't specify import paths), so even though Django can serialize the majority of things, there are some things we simply can't serialize out into a valid Python representation.

The following can be serialized by Django :

• int, float, bool, str, bytes, None, NoneType
• list, set, tuple, dict, range.
• datetime.date, datetime.time, and datetime.datetime instances (include those that are timezone-aware)
• decimal.Decimal instances
• enum.Enum instances
• uuid.UUID instances
• functools.partial() and functools.partialmethod instances which have serializable func, args, and keywords values.
• Pure and concrete path objects from pathlib. Concrete paths are converted to their pure path equivalent, e.g. pathlib.PosixPath to pathlib.PurePosixPath.
• os.PathLike instances, e.g. os.DirEntry, which are converted to str or bytes using os.fspath().
• LazyObject instances which wrap a serializable value.
• Enumeration types (e.g. TextChoices or IntegerChoices) instances.
• Any Django field
• Any function or method reference (e.g. datetime.datetime.today) (must be in the module’s top-level scope)
• Unbound methods used from within the class body
• Any class reference (must be in the module’s top-level scope)
• Anything with a custom deconstruct() method (see below)
• Django cannot serialize :
  • Nested classes
  • Arbitrary class instances (e.g. MyClass(4.3, 5.7))
  • Lambdas

Custom Serializers

By creating a unique serializer, you can serialize other types. For instance, if Django didn't serialize Decimal by default, you could do this :

The first argument of MigrationWriter.register_serializer() is A type or iterable of types that ought to use the serializer.

Your serializers serialise() method must return both a string indicating how the value could appear in migrations and a list of any imports required for the migration.

Adding a deconstruct() Method

Adding a deconstruct() method to the class enables Django to serialize instances of your custom classes. It should return a tuple of three items (path, args, and kwargs) and requires no arguments.

• Path should be the class's Python path, now with the class name added as the final component (for instance, myapp.custom_things.MyClass). Your class cannot be serializable if it is not present at the module's top level.
• For the __init__ method of your class, args ought to be a list of positional arguments. Everything on this list ought to be serializable in and of itself.
• To pass to the __init__ method of your class, kwargs ought to be a dict of keyword arguments. Each value should be serializable in and of itself.

Supporting Multiple Django Versions

It may be necessary to ship migrations that support various Django versions when you are the maintainer of such a third-party app with models. In this scenario, you should always select the lowest Django version you want to support when running django makemigrations.

The backward compatibility of the django migrations system will be maintained by the same guidelines as the remainder of Django, so migration files created for Django X.Y should continue to function correctly in Django X.Y+1. However, forward compatibility is not guaranteed by the migrations system. Migration files created with more recent Django versions might need to be fixed with older ones, and new features might be added