Safeguarding edx.org deploys

Owned by Ned Batchelder (Deactivated)
Last updated: May 04, 2022
2 min read

This is a public page.

Principles:

  • We want to protect edx.org from accidental problems caused by non-2U committers.

  • We also want to protect edx.org from problems from ourselves!

  • We want to protect edx.org from malicious actors.

  • We want to protect edx.org from problems from third-party code. This is less of a concern due to broader use of third party code.

Approaches:

  • Prevent bad merges

  • Prevent bad deploys

  • Detect bad deploys, limit damage

Possible solutions:

  • edX review of all external commits

    • pro: most security

    • con: extremely high effort

    • con: wrong community stance

  • Fork edx-platform into edx org, pull changes daily

    • pro: gives us an understood deploy window

    • pro: opportunity to review changes for malicious code

      • Note that this option doesn’t necessarily involve reviewing the changes being pulled.

    • con: extra automation and work, especially if reviewing all changes

    • con: two-directional merges adds friction and effort

    • con: doesn’t protect us from ourselves

  • Release from a different branch than master

    • Need details on how this would make things better

  • Automate community merges to occur during opportune times

    • pro: gives us an understood deploy window

    • pro: relatively low effort

    • con: doesn’t protect us from ourselves

    • con: doesn’t protect us from malicious actors

  • Better automated testing

    • pro: protects us from ourselves

    • con: a lot of effort, ongoing

    • con: not clear how to really move the needle on testing

    • con: probably doesn’t protect us from malicious actors

  • Canary deploys

    • pro: protects us from ourselves

    • pro: not a lot of effort

    • con: not a top priority for SRE

    • con: probably doesn’t protect us from malicious actors

Other considerations:

  • How will security fixes be developed and deployed prior to public disclosure?

  • Will SOX compliance complicate this even further?

  • Who else in the community deploys from master?

  • How far can open our monitoring to the community so they at least have visibility to problems?

  • Do we need or want all projects to use the same solution?

    • Completely different solutions for various repos would likely be madness, but perhaps a consistent automated solution with a manual-review variant for sensitive repos could be workable