Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

One known risk is performance issues once we upgrade Prod, due to size of the DB there. Stage DB is much smaller than Prod DB, so scaling might be an issue which we would not be able to detect during review and testing in Sandbox or Stage.

Rollout and Contingency

TBD Adam Blackwell (Deactivated) Joe Mulloy (Deactivated) Can you fill this out? Something along the lines of:

  • Rollout Plan

    • Stage will be upgraded, then prod and edge

    • New instances on the new version are added to the cluster one at a time andthe data is synced

    • There is 30 seconds or less of downtime when the cluster cuts over to a primary node on the new version

      • No downtime notifications should be needed

    • Old version nodes are then removed from the cluster

  • Rollback plan

    • “Rollback” would be swapping back to older version as primary node if switching primary to new version causes problems.

    • Cluster will have nodes of two versions at the same time for a while and app servers will start using nodes running newer Mongo as soon as they complete syncing data

    • Rollback to old version can be done until we update the FeatureCompatibilityVersion
      variable in Mongo. After it goes forward we can’t go backward without dump and restore.

...

Though we need to complete this anytime before October 15, SRE teams wants to complete in August 2021 if possible.

Project Team

Name

Squad

Role

lcicchese (Deactivated)

SRE

Service Owner

Natalia Berdnikov (Deactivated)

Open edX

Project Manager

Joe Mulloy (Deactivated)

SRE (Cambridge)

Lead SRE Engineer

Syed Imran Hassan Abdi

SRE (Arbisoft)

Consulting SRE Engineer

Jeremy Ristau

TNL (owner of Mongo usage in edx-platform)

TNL Lead

Kyle McCormick

TNL (owner of Mongo usage in edx-platform)

Review (edx-platform)

Awais Jibran (Deactivated)

Infinity squad (IDA owner of cs-comments-service)

Review (cs-comments-service)

Nimisha Asthagiri (Deactivated)

Open edX

Stakeholder (Infrastructure)

Ned Batchelder (Deactivated)

Open edX

Stakeholder (Open edX named release)

Open edX Community

Open edX Community

Stakeholder (Consumers of Maple named release)

Communication Plan

Due to a small number of stakeholders we will mostly likely communicate ad hoc via Slack. Dedicated channel is #mongodb-4-4.

...