Some Things About Manually Rolling Back Migrations

Owned by Adam Blackwell (Deactivated)
Last updated: Jan 31, 2022
1 min read

Overview

This page is a guide for SREs and engineers who are undoing strange things. Please update it.

Some steps in this guide assume you have access to the root user of a database

Workflows

Testing Locally

  1. Before you do anything in Prod you should test in stage or devstack

 

Running Migrations via manage.py Manually

Note: Be careful about what version of the code you have since you can’t migrate back without the migration files (except via GoCD)

Note also: you’ll need access to edx-secure to run migrate. (One can run showmigrations as documented below without such permissions.) Passwords (e.g. the ones here) are necessary to have access to a database user with sufficient permissions.

TBD



Example Flow:

ssh <sre_user>@<ip_from_aws_console> sudo su - www-data -s /bin/bash cd /edx/app/<service>/ . venvs/<service>/bin/activate . <service>_env cd <service>/ (<service>) www-data@ip-10-3-117-24:/edx/app/ecommerce/ecommerce$ python manage.py showmigrations <model> communication [X] 0001_initial

Dropping Rows from The django_migrations Table

  1. Only do this if you’re sure you can’t roll back any other way and that the currently deployed code does not rely on the changes from the migration.

  2. Check the table

    SELECT * FROM django_migrations;

  3. Pick your ID:

    SELECT * FROM django_migrations WHERE id = <id>;

  4. Check what the migration was doing

    1. Shell in to a box

    2. Run something like this:

    3. Look at what tables are being touched.

  5. Look to see if anything is writing to the tables;

  6. Count rows in the related tables
    Example:

  7. Delete the record of the migration

  8. Drop the table

    1. Note, you don’t usually need to truncate.

  9. Check New Relic or Performance Insights to see that things look ok.

  10. Profit