Overview

This page is a guide for SREs and engineers who are undoing strange things. Please update it.

note

Some steps in this guide assume you have access to the root user of a database

Some steps in this guide assume you have access to the root user of a database

Workflows

Testing Locally

  1. Before you do anything in Prod you should test in stage or devstack

Running Migrations via manage.py Manually

Note: Be careful about what version of the code you have since you can’t migrate back without the migration files (except via GoCD)

Note also: you’ll need access to edx-secure to run migrate. (One can run showmigrations as documented below without such permissions.) Passwords (e.g. the ones here) are necessary to have access to a database user with sufficient permissions.

TBD



Example Flow:

ssh <sre_user>@<ip_from_aws_console>
sudo su - www-data -s /bin/bash
cd /edx/app/<service>/
. venvs/<service>/bin/activate
. <service>_env
cd <service>/

(<service>) www-data@ip-10-3-117-24:/edx/app/ecommerce/ecommerce$ python manage.py showmigrations <model>
communication
 [X] 0001_initial

Dropping Rows from The django_migrations Table

  1. Only do this if you’re sure you can’t roll back any other way and that the currently deployed code does not rely on the changes from the migration.

  2. Check the table

    SELECT * FROM django_migrations;

  3. Pick your ID:

    SELECT * FROM django_migrations WHERE id = <id>;

  4. Check what the migration was doing

    1. Shell in to a box

    2. Run something like this:

      python manage.py sqlmigrate catalog 0028 --backwards
python manage.py sqlmigrate catalog 0028

    3. Look at what tables are being touched.

  5. Look to see if anything is writing to the tables;

    show processlist;

  6. Count rows in the related tables
    Example:

    select count(*) from catalog_contentmetadatatoqueries;

  7. Delete the record of the migration

    DELETE * FROM django_migrations WHERE id = <id>;

  8. Drop the table

    DROP TABLE truncate catalog_contentmetadatatoqueries;

    1. Note, you don’t usually need to truncate.

  9. Check New Relic or Performance Insights to see that things look ok.

  10. Profit