Safeguarding edx.org deploys

Principles:

• We want to protect edx.org from accidental problems caused by non-2U committers.

• We also want to protect edx.org from problems from ourselves!

• We want to protect edx.org from malicious actors.

• We want to protect edx.org from problems from third-party code. This is less of a concern due to broader use of third party code.

Approaches:

• Prevent bad merges

• Prevent bad deploys

• Detect bad deploys, limit damage

Possible solutions:

• edX review of all external commits

  • pro: most security

  • con: extremely high effort

  • con: wrong community stance

• Fork edx-platform into edx org, pull changes daily

  • pro: gives us an understood deploy window

  • pro: opportunity to review changes for malicious code

    • Note that this option doesn’t necessarily involve reviewing the changes being pulled.

  • con: extra automation and work, especially if reviewing all changes

  • con: two-directional merges adds friction and effort

  • con: doesn’t protect us from ourselves

• Release from a different branch than master

  • Need details on how this would make things better

• Automate community merges to occur during opportune times

  • pro: gives us an understood deploy window

  • pro: relatively low effort

  • con: doesn’t protect us from ourselves

  • con: doesn’t protect us from malicious actors

• Better automated testing

  • pro: protects us from ourselves

  • con: a lot of effort, ongoing

  • con: not clear how to really move the needle on testing

  • con: probably doesn’t protect us from malicious actors

• Canary deploys

  • pro: protects us from ourselves

  • pro: not a lot of effort

  • con: not a top priority for SRE

  • con: probably doesn’t protect us from malicious actors

Other considerations:

• How will security fixes be developed and deployed prior to public disclosure?

• Will SOX compliance complicate this even further?

• Who else in the community deploys from master?

• How far can open our monitoring to the community so they at least have visibility to problems?

• Do we need or want all projects to use the same solution?

  • Completely different solutions for various repos would likely be madness, but perhaps a consistent automated solution with a manual-review variant for sensitive repos could be workable