Meeting Info
Date:
Recording: TK
Notes
Rationale
Open edX Architecture moving from a centralized model to a decentralized model.
Similarly for the edX SRE team
Many of our teams own backend and frontend components, within and outside of the monolith
Each may want to do things differently - their pace, their testing, their definition of done, etc.
Hence, our centralized Configuration repo is problematic and bottlenecked
Changes to repo
Need to work with last 2 old releases, multiple OSes, multiple environments (sandboxes, etc).
Only the latest community release is supported, but some internal edX uses use older releases.
In the past, have taken OSPR changes but haven’t tested ourselves, but has added to our maintenance burden
Also realized that any change in configuration needed a long deprecation process
Configuration state in production
Many teams just couldn’t figure out what the state of a configuration was in production or in other environments
It was difficult to parse this from the code since the settings were tangled in Ansible files, YAML files, etc, etc.
So… moved to detangling the settings (De-DRY)
Current State
Moved to using a single YAML file for each App
Asym-Crypto-YAML: https://github.com/edx/asym-crypto-yaml
Inline encryption of our YAML - so secrets can be encrypted inline
Allows devs to see the secrets inlined, but protected encrypted
Remote Config
Hermes: https://github.com/edx/hermes
Intended to speed up config deployments
Whenever its ETAG changes, it runs a python file
Config files are completely separate from our IDA deployments
downside today: can’t tell today exactly when the config is updated; could be fixed with further tooling at some point
Have been doing this for about a year perhaps
Made refactoring configuration a lot easier
Config Flow changes today
Configuration repo runs once in the “beginning”
We run Ansible to create the initial AMI
After that, we no longer use the configuration repo
Dev
declares the Python configuration setting in the App
set the values in remote config
only set in configuration if the setting is needed in NGinx
Remote config changes are merged
CI verifies decryption of secrets
CI verifies formatting
Hermes automatically deploys the config change
Future with Containers
Background
Remote config today
Remote config is Hermes with a few lines of python code
Hermes is currently configured with Ansible code
edX Secrets today
Stored in Vault
Using OS tech today from Hashicorp
Moving to decentralized model
Defaults are in the app
Dockerfile in each repo
Open edX version of the application, without any edX-specific
Example YAML config file as well
Dockerfile buildable by running
docker build .
→ without any additional params, etc.Just: Build the image and mount the YAML
Clearer delineation between Open edX and edX.org
So edX.org can run at a different pace and a different scale
edX.org: Docker images → using Helm → k8s
Would continue to support Blue/Green Deployments
Helmcharts are currently edX-specific in order to provide clearer directions to edX engineers
Could provide Open edX versions of HelmCharts and k8s Manifest as examples, if they will be useful
Private requirements for edX.org → would like to move this out of the public repos
So edX.org can dogfood this design pattern
You may notice new files with
-newrelic
tags - these are the start of these edX-specific files
Discussion
OEP-45 follow-up
NGinx configuration - what are the plans for this?
Would this be with sample Helm charts?
edX would be using
nginx ingress
Possible Options
A: Provide an example chart
B: Provide an open chart that edX also consumes
Myth: Entire configuration for an IDA can be stored in a given repo
Example: Authentication is coupled across repos
operation was in the Ansible playbooks
Note: decentralized devstack has a similar issue
In addition to Helmcharts, you are also using Consul Templates to pull in secrets from Vault?
Yes, those go into as values in the Helmcharts.
Timeline for deprecation?
Let’s discuss this together and work together on accelerating and converging on this effort
Currently evaluating
Tutor
as a community-supported deployment mechanismUsual questions: Testing, Devstack, Docker files, etc.
It seems similar problems underneath, no?
For example, should Tutor be the next Devstack?
Tutor - make it easy to install
Devstack is trying to address the same issue
Currently, not a hard blocker since we are pushing images in our current Devstack.
edX Architecture team is the newly designated “owner” for Devstack and plans to look into addressing short-term issues as well as long-term plans. Nim will ask the team to watch the recording from today and follow-up with this group in order to converge paths.