Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 3 Next »

Meeting date: December 4, 2020
Recording: TK

Notes

Rationale

  • Open edX Architecture moving from a centralized model to a decentralized model.

    • Similarly for the edX SRE team

  • Many of our teams own backend and frontend components, within and outside of the monolith

    • Each may want to do things differently - their pace, their testing, their definition of done, etc.

  • Hence, our centralized Configuration repo is problematic and bottlenecked

  • Changes to repo

    • Need to work with last 2 old releases, multiple OSes, multiple environments (sandboxes, etc).

      • Only the latest community release is supported, but some internal edX uses use older releases.

    • In the past, have taken OSPR changes but haven’t tested ourselves, but has added to our maintenance burden

    • Also realized that any change in configuration needed a long deprecation process

  • Configuration state in production

    • Many teams just couldn’t figure out what the state of a configuration was in production or in other environments

    • It was difficult to parse this from the code since the settings were tangled in Ansible files, YAML files, etc, etc.

    • So… moved to detangling the settings (De-DRY)

Current State

  • Moved to using a single YAML file for each App

  • Asym-Crypto-YAML: https://github.com/edx/asym-crypto-yaml

    • Inline encryption of our YAML - so secrets can be encrypted inline

    • Allows devs to see the secrets inlined, but protected encrypted

  • Remote Config

    • Hermes: https://github.com/edx/hermes

      • Intended to speed up config deployments

      • Whenever its ETAG changes, it runs a python file

    • Config files are completely separate from our IDA deployments

      • downside today: can’t tell today exactly when the config is updated; could be fixed with further tooling at some point

    • Have been doing this for about a year perhaps

    • Made refactoring configuration a lot easier

  • Config Flow changes today

    • Configuration repo runs once in the “beginning”

      • We run Ansible to create the initial AMI

      • After that, we no longer use the configuration repo

    • Dev

      • declares the Python configuration setting in the App

      • set the values in remote config

        • only set in configuration if the setting is needed in NGinx

    • Remote config changes are merged

      • CI verifies decryption of secrets

      • CI verifies formatting

    • Hermes automatically deploys the config change

Future with Containers

  • Background

    • Remote config today

      • Remote config is Hermes with a few lines of python code

      • Hermes is currently configured with Ansible code

    • edX Secrets today

      • Stored in Vault

      • Using OS tech today from Hashicorp

  • Moving to decentralized model

    • Defaults are in the app

    • Dockerfile in each repo

      • Open edX version of the application, without any edX-specific

      • Example YAML config file as well

    • Dockerfile buildable by running docker build . → without any additional params, etc.

    • Just: Build the image and mount the YAML

    • Clearer delineation between Open edX and edX.org

      • So edX.org can run at a different pace and a different scale

  • edX.org: Docker images → using Helm → k8s

    • Would continue to support Blue/Green Deployments

    • Helmcharts are currently edX-specific in order to provide clearer directions to edX engineers

      • Could provide Open edX versions of HelmCharts and k8s Manifest as examples, if they will be useful

    • Private requirements for edX.org → would like to move this out of the public repos

      • So edX.org can dogfood this design pattern

      • You may notice new files with -newrelic tags - these are the start of these edX-specific files

Discussion

  • OEP-45 follow-up

    • NGinx configuration - what are the plans for this?

      • Would this be with sample Helm charts?

      • edX would be using nginx ingress

      • Possible Options

        • A: Provide an example chart

        • B: Provide an open chart that edX also consumes

  • Myth: Entire configuration for an IDA can be stored in a given repo

    • Example: Authentication is coupled across repos

      • operation was in the Ansible playbooks

    • Note: decentralized devstack has a similar issue

  • In addition to Helmcharts, you are also using Consul Templates to pull in secrets from Vault?

    • Yes, those go into as values in the Helmcharts.

  • Timeline for deprecation?

    • Let’s discuss this together and work together on accelerating and converging on this effort

  • Currently evaluating Tutor as a community-supported deployment mechanism

    • Usual questions: Testing, Devstack, Docker files, etc.

    • It seems similar problems underneath, no?

    • For example, should Tutor be the next Devstack?

      • Tutor - make it easy to install

        • Devstack is trying to address the same issue

    • Currently, not a hard blocker since we are pushing images in our current Devstack.

Open Questions

  • Sane defaults

  • No labels