It’s easy to lose track of the target end state of a project when lost in the details of the current state and incremental improvement efforts.  With that in mind, here’s a proposed vision for what the Open edX development environment could look like within a year if we coordinate and focus our efforts appropriately, while getting incremental developer experience value along the way.  Some aspects of this may need to change in the face of complexity, resource constraints, or hosting costs, but this represents one possible ideal state.  Most of this applies equally to developing both server-side services and Micro-Frontends and other client-side code. This document is still in review and being amended based on feedback, and doesn’t yet represent any organization’s official decision/position on this topic.

1. Overall Objectives

These are the things we hope to accomplish by having a good Open edX development environment.  Technology choices and other fine-grained decisions should be guided by their impact on these objectives.  Note that some of these objectives are in tension with each other, and a change that improves one may do so at the detriment of others; in such cases, the goal is to create a development environment that strikes a good balance among them.  I’ve tried to avoid focusing on specific relevant projects (like devstack, Tutor, and OEP-37) at this level to avoid assumptions based on their current state and/or roadmaps.

Enables fast verification of code changes

A developer (even if completely new to the code) should be able to quickly interact with the Open edX software as a user to verify that a change to the code had the desired effect.  Without a developer environment, this would require manually installing the software and all of its dependencies, any other services it requires, updating configuration settings, starting all the services, and populating appropriate test data (likely requiring days of effort, if not weeks).

Requires minimal system administration effort

Keeping a complex web application running smoothly over time through multiple software updates is a very difficult task requiring specialized skills and a lot of time.  To avoid complex periodic maintenance and manual updates over time to keep up with recent changes in the software, it should be easy to rapidly create a new, up-to-date environment whenever needed.

Optimized for development velocity

The development environment should come with and/or seamlessly integrate with state of the art software development tools.  Debuggers, Application Performance Monitoring (APM) services, linters, test runners, IDEs, vim/emacs-class text editors, and more should all be extremely easy to use when working in the environment.  Default tools are recommended, as is documentation for using them effectively.

Consistent across the Open edX ecosystem

The development environment should provide a consistent experience whichever service or library in the Open edX ecosystem is being worked on.  A developer shouldn’t have to use Tutor for LMS, devstack for course-discovery, Codespaces for credentials, Vagrant for Xqueue, etc. (not a real example, thankfully).  Experience in using the development environment should be transferable across different areas of the code, minimizing context switching and unnecessary learning curves.

Minimal developer system requirements

A developer should not be required to have the very latest laptop with a top-tier CPU and maxed-out RAM and storage to be able to run the developer environment.  The developer’s computer should not be rendered unusable for other purposes while the development environment is running.  It should also not prevent users favoring a specific reasonably common operating system from being able to develop Open edX software.

Scalable to different subsets of Open edX services

Just as different Open edX operators choose to run different subsets of the full suite of Open edX services, the development environment should be capable of running any combination of them.  Some combinations may not work very well, but even those can be useful for testing and improving failure modes in the face of unexpected outages.

Usable by all Open edX developers

The development environment can be used equally by developers at 2U, tCRIL, OpenCraft, Arbisoft, or any other organizations interested in running and/or contributing to Open edX.  Some of these organizations may extend it in ways that support proprietary extensions or internal services, but the core should be free to use by all.

Closely matches production behavior

The development environment should closely match a typical production deployment.  The services find each other in similar ways, similar configuration options are used, similar background process workers and queues are present, and a similar set of feature flags is active.  The default settings may not exactly match any one organization’s production environment, but allow straightforward customization to closely resemble each organization’s production choices.

Easy to understand

The way developer environments are created and configured is clear and reasonably intuitive (so it can be easily maintained, improved, and extended over time).  And when things break in a running development environment, it is relatively straightforward to figure out what went wrong and where to start looking in order to fix it. There is up-to-date documentation covering most common aspects of working with the development environment, written with appropriate reader personas in mind.

Easy to employ and extend

Using the development environment for a new service or other software repository should be reasonably simple and straightforward, with good templates and documentation.  It should be possible to add support for proprietary related services without making detailed information about them public.

Inexpensive to operate

Running an instance of the development environment is not expensive enough to cause needless hesitation in launching it or pressure to terminate it prematurely.  If launched using cloud computing resources, it uses them efficiently so as not to cost more than absolutely necessary.  It does not require expensive licenses for proprietary software or services.  It may provide the option to use more expensive, higher-powered hardware and software, but defaults to more frugal operation.

Actively maintained

When a bug or regression is identified in the development, it’s clear where to report it.  Such problems are promptly remedied.  New developments in the ecosystem which enable clear improvements with respect to the other objectives listed above are investigated and employed in a reasonable span of time.

2. High-Level Technical Decisions

In order to accomplish the objectives listed above, these are high-level technical characteristics that our development environment should have, given the current state of the overall web application development ecosystem.

Standardized, layered Docker images

Each service has an easily located Dockerfile defining an image that can be deployed in either Kubernetes or docker-compose.  With appropriate runtime configuration provided, this base layer can be used for edx.org stage & production, other Open edX production sites of any scale, and short-lived cloud instances (sandboxes) intended for demo or testing purposes.  An additional layer in the same Dockerfile generates a derived image providing all the additional tooling needed for development.  This larger image can be used for development environments, wherever they’re deployed (local or cloud).  Both images are as simple and small as practical while getting the job done effectively.

Addresses: Closely matches production behavior, Consistent across the Open edX ecosystem, Easy to employ and extend, Easy to understand

Full support for Kubernetes

The Docker images are functional in either Kubernetes or docker-compose (and perhaps other orchestration systems), and Kubernetes is supported at least as well as docker-compose in developer environments.  That is increasingly the production orchestration choice for 2U and other large Open edX installations, and development deployments should try to match each organization’s production deployment as closely as practical.  Automation is used to minimize the amount of Kubernetes knowledge required to get started, and good documentation is recommended for those who need to start digging a little deeper into how the orchestration works.

Addresses: Closely matches production behavior

Actively maintained data sets

Data appropriate for a wide variety of development tasks is actively maintained in each service’s repository and has automated tests to ensure it continues working over time.  As with other code in the repository, the owning team is ultimately responsible for maintaining this data but can seek (or receive unsolicited) assistance from other Open edX developers.  These data sets can be easily loaded into any Open edX environment.  The data sets are robust enough to handle different subsets of services in use, ranging from the bare minimum to have a functioning site all the way up to the full set of services used on edx.org.

Addresses: Enables fast verification of code changes, Requires minimal system administration effort

Cloud development option

Standard tooling exists to generate a Cloud-hosted Open edX development environment.  This environment can use either the base or development Docker image of each service.  It is configured to support loading the source code of each service in a local editor or IDE for ease of navigation and modification.  Any combination of Open edX services can be deployed, even ones that aren’t expected to work well (useful for testing and improving failure modes when a service becomes unavailable).

Addresses: Closely matches production behavior, Minimal developer system requirements, Requires minimal system administration effort

Local development option

It is still possible to run a development environment on a laptop via Docker Desktop, Minikube, or other suitable Kubernetes distribution (one is chosen as a recommended default).  The source code directories of each running service can be present on the local filesystem for easy navigation via an editor or IDE.

Addresses: Inexpensive to operate, Usable by all Open edX developers

Extensive, intuitive developer tooling

Development environments come pre-equipped with modern development tools: integration with at least the most popular IDEs and editors, Django Debug Toolbar, easy Python debugger integration, and a viable APM (Application Performance Monitoring) option.

Addresses: Optimized for development velocity

Clear ownership assignment

The entire development environment has clearly assigned ownership for maintenance, improvements, and support.  One team is responsible for the overall development experience, with subsystems delegated to other teams as necessary.  Each team owning some aspect of the development environment allocates an appropriate and consistent percentage of its time each sprint to these ownership responsibilities.

Addresses: Actively maintained

3. Next Steps

Several projects have been started to address current limitations in our development environment, and they’re not all mutually compatible (although most have yielded valuable learnings).  To minimize the amount of time and effort needed to yield a development environment that satisfies the objectives above, the following are the proposed next steps (which can all be pursued in parallel).  Other efforts are likely to continue in parallel (especially in cases where they are complementary and not competing for attention from the same teams), but these are arguably the most important ones to prioritize.

Pick a preferred tool for managing development environments

Since we switched from Vagrant to docker-compose as the basis for our development environment, there has been an explosion in tools for managing containerized development environments.  At least a few of these seem to enable several of our objectives with little custom coding or configuration.  A discovery ticket has been created with tentative selection guidelines and some initial candidates for consideration.  Before investing large amounts of additional development effort into either devstack or Tutor, we should complete this discovery work and determine what exactly we need to support with Open edX-specific code beyond what our chosen tool provides.

Addresses: Cloud development option, Default to Kubernetes, Local development option

Continue OEP-37 (Dev Data) implementation & refinement

OEP-37 was intended to enable most development tasks to be performed without needing to build up a custom data set of users, courses, enrollments, grades, etc. every time.  This will become even more important if we start defaulting to a model where we routinely create new development and preview environments without reusing an existing database.  Implementation of this OEP got pretty far last year but stalled at waiting for a team to be the first to create appropriate data for its own service.  We should resume this work, starting with replacement of existing devstack data provisioning and moving on to the data most useful for testing functionality where the most active development is expected in the near future.

Addresses: Actively maintained data sets

Dedicate people to addressing the most acute current pain points

Reaching our target state will take some time, during which most 2U Open edX developers are still using devstack and Tutor is the only officially supported installation method for named releases.  2U is hesitating to make major new investments in these tools until the discovery work outlined above is completed in order to guide it, but will continue making targeted smaller improvements to solve particularly acute problems or regressions with the current developer experience.  Additionally, many developers outside 2U have been making steady progress on improving Tutor as a development environment, and plan to continue doing so for the foreseeable future. We should have different people working on these incremental improvements than the other two next steps, to minimize tension between making urgently needed fixes and planning out projects with potentially large long-term impact.

Addresses: Clear ownership assignment