Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 2 Next »

See PLAT-2437. This is a runbook for running gh-ost to alter the id field of courseware_studenthistory in edx-platform as safely and with as little downtime as possible.


Per-environment runbook (in progress)

  1. Use the gh-ost cheat sheet for "Connect to replica, migrate on master": https://github.com/github/gh-ost/blob/master/doc/cheatsheet.md#a-connect-to-replica-migrate-on-master
    1. Confirm the environment is set up correctly
      1. Is using Statement Based Replication
      2. Replica we want to use is configured with binary logs enabled (log_bin, log_slave_updates)
      3. binlog_format=ROW (gh-ost can apply the latter for you)
    2. Confirm the RDS environment has enough resources for two copies of the table!
    3. Make sure we have dropped the FK from courseware_studentmodulehistory
  2. Perform a no-op run to make sure everything is set up correctly
    1. The "alter" flag we want to run is: "change id bigint unsigned not null auto_increment"

  3. Run the migration on a replica only
    1. Monitor and see if the throttling variables need to be tweaked
    2. Confirm that the data all looks good and makes sense
  4. Run the migration on the master
    1. Monitor and tweak throttling as needed
    2. We should use the cut over flag file to make sure we are in the office and ready when it comes time to flip the switch: https://github.com/github/gh-ost/blob/master/doc/command-line-flags.md#postpone-cut-over-flag-file
    3. Confirm that the data looks good
  5. Cut over to the new table and monitor
  6. Trickle delete the rows from the old table, wait some time, then drop it: https://github.com/github/gh-ost/issues/307
    1. We will likely want to create a custom script / management command for the deletes
  • No labels