Development Environment Notes

Owned by Tim McCormack
Last updated: Mar 16, 2023 by Adam Blackwell (Deactivated)
3 min read

Goals

  • Primarily:

    • To consider possible replacements for devstack and Tutor for development environments.

  • Secondarily:

    • Find possible replacements for sandboxes (as shareable/preview environments), which currently are slow to stand up and aren’t easy to hack on once they’re running.

    • Better unify our development and deployment systems. For instance, could we use kubernetes for both?

Links

Findings

  • Two main kinds of viable options:

    • Cluster interfaces: Both Tilt and DevSpace allow you to deploy a set of services to a kubernetes cluster and provide a simplified interface to it. Tilt and DevSpace are extremely similar.

      • Tilt seems to have more comprehensive docs.

      • DevSpace also tries to cover deployment space a bit, supporting non-interactive deployment and even some monitoring and alerting. Might work for small Open edX deployments.

    • Workspaces: A more comprehensive option, Coder even sets up the cluster itself and can manage it via config. (Okteto might be in this category, but it’s hard to tell.)

  • Other contenders in this space that don’t seem like a good fit:

    • Gitpod and Codespaces only seem to allow launching a single service at a time, and don’t have a local option.

  • Patterns

    • Most options offer an open-source version and can run fully locally, although we’ll need a remote version for those with M1 Macs or resource-constrained computers.

    • Many options offer a choice between various Docker management approaches: kubectl, Helm, docker-compose, others. Some in the community are in favor of using k8s, and others want to use docker-compose. We might be able to allow both to coexist, but there would be duplication.

    • As with devstack, we’d have hot code-reloading with all of these; it seems to be table-stakes.

      • However, devstack already has the issue of not automatically applying migrations, reinstalling requirements when they change, or rebuilding static assets. We’d need to fix that whether we stick with devstack or move to something else; none of these will solve that problem for us.

  • Replacing/enhancing existing systems

    • There isn’t an obvious option that would replace sandboxes, although DevSpace would be the one to look into.

    • It’s unclear if any of these would be able to unify with how deployments are done, though DevSpace might be sufficient for some in the community?

 

Notes

  • How does a multi-service dev env know which local git repositories (as filesystem directories) correspond to which dependencies?

    • Tilt has you decide for yourself, but suggests that all repositories be checked out under the same parent directory, as we do with devstack.

    • Coder probably manages all of this for you (since I don’t think it even lets you check out the code locally – all editing seems to be done in the workspace)

  • We should check into whether any of these make it easier/harder to use pip install -e when working with libraries in an IDA.

  • Things that would improve devstack now, and that would still help us in any new system:

    • Removing Ansible from our image-building, because our approach to Ansible is currently complicated by our re-use strategy, and because our Ansible reads from a config repo as it runs—whereas base Docker images would be config-free and would work across dev, stage, and prod.

    • Better auto-reloading (migrations, static assets, requirements)

  • Tools not assessed: Ambassador Telepresence, Garden, Skaffold

Features and Facets

  • It was hard to evaluate the options for cloud dev env management without knowing exactly what we want to replace from our dev environment (or add to it)

  • We created a feature breakdown of Open edX dev environments at https://docs.google.com/spreadsheets/d/1XAipznjIcTaE2H454M74No2Ro_9VftbolfV4YVv-B9I/edit#gid=0 , and wrote https://github.com/edx/edx-arch-experiments/issues/203 for refining it

  • Once we had a list of features, we started finding categories of them; essentially different facets of a complete development environment:

    • Standardization - Things we can get just by standardizing across our repos

    • Selection - Things needed to specify a desired subset of services and make sure common development tasks operate on exactly that subset

    • Orchestration - Things that are closely tied to the container orchestration system (typically docker-compose or Kubernetes)

    • Deployment - Things that are closely tied to how/where an environment is deployed: Docker Desktop (local), EC2 (cloud), EKS, etc.

    • Integration - Things that serve as glue for features in the other categories: do X for every repo in a particular set, do X for every service currently running, etc.

  • Some of the tools are Kubernetes-only

  • Spitballing:

    • Selection - go with Tutor plugins

    • Orchestration - Kubernetes preferred, but support docker-compose if possible

    • Deployment - find the best thing to smooth the Kubernetes learning curve, make it an option for Tutor

    • Integration - Tutor CLI?

  • For comparison:

    • Tutor with problematic areas replaced/supplemented with another tool

    • Go with another tool from the ground up

      • Possibilities

        • Devspace

  •