See PLAT-2437. This is a runbook for running gh-ost to alter the id
field of courseware_studenthistory
in edx-platform as safely and with as little downtime as possible.
Questions we need to answer before running this:
- What changes do we need to make to support both versions of CSMH?
- Removing the FKs will likely mean that we need to use the integer IDs directly instead of using StudentModule objects, for example.
- Also manually handling deletes if they're cascading right now.
- After we upgrade CSM we will also need to migrate both CSMH's to have an appropriately sized column for CSM fake foreign keys.
- What do we need to do to ensure that Django migrations continue to work throughout this process? Especially after we switch over to the new table?
- ? At what point do we do that for each of the tables that matter (courseware_studentmodule, courseware_studentmodulehistory, courseware_studentmodulehistoryextended)?
Per-environment runbook (in progress)
- Use the gh-ost cheat sheet for "Connect to replica, migrate on master": https://github.com/github/gh-ost/blob/master/doc/cheatsheet.md#a-connect-to-replica-migrate-on-master
- Confirm the environment is set up correctly
- Is using Statement Based Replication
- Replica we want to use is configured with binary logs enabled (
log_bin
, log_slave_updates
) binlog_format=ROW
(gh-ost
can apply the latter for you)
- has enough resources for two copies of the table!
- Have we pruned
courseware_studentmodulehistory
? Should we?
- Perform a no-op run to make sure everything is set up correctly
The "alter" flag we want to run is: "change id bigint unsigned not null auto_increment"
- Run the migration on a replica only
- Monitor and see if the throttling variables need to be tweaked
- Confirm that the data all looks good and makes sense
- Run the migration on the master
- Monitor and tweak throttling as needed
- We should use the cut over flag file to make sure we are in the office and ready when it comes time to flip the switch: https://github.com/github/gh-ost/blob/master/doc/command-line-flags.md#postpone-cut-over-flag-file
- Confirm that the data looks good
- Cut over to the new table and monitor
- Trickle delete the rows from the old table, wait some time, then drop it: https://github.com/github/gh-ost/issues/307
- We will likely want to create a custom script / management command for the deletes
- Do we need to fake a migration forĀ
courseware_studentmodule
here or will that break things? - Do all of this again to upgrade
courseware_studentmodulehistory
's studentmodule_id column as well - Do we need to fake a migration for
courseware_studentmodulehistory
here or will that break things?