See PLAT-2437. This is a runbook for running gh-ost to alter the id
field of courseware_studenthistory
in edx-platform as safely and with as little downtime as possible.
Per-environment runbook (in progress)
- Use the gh-ost cheat sheet for "Connect to replica, migrate on master": https://github.com/github/gh-ost/blob/master/doc/cheatsheet.md#a-connect-to-replica-migrate-on-master
- Confirm the environment is set up correctly
- Is using Statement Based Replication
- Replica we want to use is configured with binary logs enabled (
log_bin
,log_slave_updates
) binlog_format=ROW
(gh-ost
can apply the latter for you)
- Confirm the RDS environment has enough resources for two copies of the table!
- Make sure we have dropped the FK from
courseware_studentmodulehistory
- Confirm the environment is set up correctly
- Perform a no-op run to make sure everything is set up correctly
The "alter" flag we want to run is: "
change id bigint unsigned not null auto_increment"
- Run the migration on a replica only
- Monitor and see if the throttling variables need to be tweaked
- Confirm that the data all looks good and makes sense
- Run the migration on the master
- Monitor and tweak throttling as needed
- We should use the cut over flag file to make sure we are in the office and ready when it comes time to flip the switch: https://github.com/github/gh-ost/blob/master/doc/command-line-flags.md#postpone-cut-over-flag-file
- Confirm that the data looks good
- Cut over to the new table and monitor
- Trickle delete the rows from the old table, wait some time, then drop it: https://github.com/github/gh-ost/issues/307
- We will likely want to create a custom script / management command for the deletes