...
One known risk is performance issues once we upgrade Prod, due to size of the DB there. Stage DB is much smaller than Prod DB, so scaling might be an issue which we would not be able to detect during review and testing in Sandbox or Stage.
Rollout and Contingency
TBD Adam Blackwell (Deactivated) Joe Mulloy (Deactivated) Can you fill this out? Something along the lines of:
Rollout Plan
Stage will be upgraded, then prod and edge
New instances on the new version are added to the cluster one at a time andthe data is synced
There is 30 seconds or less of downtime when the cluster cuts over to a primary node on the new version
No downtime notifications should be needed
Old version nodes are then removed from the cluster
Rollback plan
“Rollback” would be swapping back to older version as primary node if switching primary to new version causes problems.
Cluster will have nodes of two versions at the same time for a while and app servers will start using nodes running newer Mongo as soon as they complete syncing data
Rollback to old version can be done until we update the
FeatureCompatibilityVersion
variable in Mongo. After it goes forward we can’t go backward without dump and restore.
...
Though we need to complete this anytime before October 15, SRE teams wants to complete in August 2021 if possible.
Project Team
Name | Squad | Role |
---|---|---|
SRE | Service Owner | |
Open edX | Project Manager | |
SRE (Cambridge) | Lead SRE Engineer | |
SRE (Arbisoft) | Consulting SRE Engineer | |
TNL (owner of Mongo usage in edx-platform) | TNL Lead | |
TNL (owner of Mongo usage in edx-platform) | Review (edx-platform) | |
Infinity squad (IDA owner of cs-comments-service) | Review (cs-comments-service) | |
Open edX | Stakeholder (Infrastructure) | |
Open edX | Stakeholder (Open edX named release) | |
Open edX Community | Open edX Community | Stakeholder (Consumers of Maple named release) |
Communication Plan
Due to a small number of stakeholders we will mostly likely communicate ad hoc via Slack. Dedicated channel is #mongodb-4-4.
...