The Arch -BOM squad sent a survey to edX and Arbisoft engineers in October 2020 to gain insight into devstack usage and pain points. This document captures summarizes the results of the survey.
Table of Contents | ||||
---|---|---|---|---|
|
Who responded (theme and expertise)?
Note |
---|
TODO: Summarize insights from this section. |
...
Info |
---|
✨ 40 people responded, of which 38 classified into a theme. |
...
Info |
---|
✨ There is a normal distribution (bell-curve) of self-reported expertise with devstack. |
...
Info |
---|
✨ Few people know how to add test data (but many think they could figure it out). |
...
Info |
---|
How to change configuration is a mystery to about ⅓ of respondents. |
...
How often are developers impacted?
Note |
---|
TODO: Summarize insights from this section. |
...
Info |
---|
Most folks use devstack often (few times a week or more). |
...
Info |
---|
About ⅔ of respondents run into devstack issues at least multiple times a month. |
...
Info |
---|
People don’t use “make destroy” very often (a few times a year or never). |
...
Info |
---|
✨ Most people don’t re-provision very often. |
...
How long do things take?
Info |
---|
The following chart is ordered from slowest to fastest. ✨ While All respondents consider ✨ Addressing It is still difficult to ❓ Now that we have a clear owner of devstack, perhaps |
...
Free-form Comments
Note |
---|
TODO: Review and categorize common concerns |
...
title | Comments on timing or level of knowledge |
---|
Some things like `make dev.pull` are quite slow, but I very rarely have to do them so it's not that big of a deal
...
Commands Know-How
Info |
---|
There is uncertainty around which commands to run when. We have big hammers (like reprovisioning all of devstack), but not clear regularly-needed hammers. |
Expand | ||
---|---|---|
| ||
|
Out-of-the-box Consistent Experience
Info |
---|
Desire to have services work together smoothly out of the box. |
Expand | ||
---|---|---|
I don't know when "make dev.provision" is needed It's unclear when refreshing devstack if a set of commands needs to be run,
I know how to change test data via discovery but nothing else. For website we only ever really need discovery so it works ok. For local test data: definitely depends on which service I need to dust off and remember about. There are a lot of common devstack problems that once you've encountered enough times you can easily work through by habit. Inherently, it's not very friendly for beginners. I often need example courses. It would be really great if I could run a management command to create a course that has all the correct data so that I can then use it with an enterprise front end. A significant problem is network. Any dev.pull takes about an hour for me. Because I am now an infrequent user I have to pretty much always re-provision whenever I need to do a devstack task, which is often closer to a 4 hr ordeal. The structure of Django projects is new to me and very different from microfrontends. I’m new-ish and most of my tickets have been in Prospectus; I haven’t touched devstack in months and didn’t get much time with it in before that. My workflow involves very rarely updating the containers. I mostly use stop and up. I mostly use LMS and a local dependency to a library I am working on. If I need newer edx-platform code I usually run git pull and then run whatever paver commands I need, which is often none, and sometimes just updating requirements and more rarely updating db. | ||
Expand | ||
| ||
This is more specific to enterprise - having to manage a bunch of different services at one time - catalog, license manager, the LMS, and then 1 or 2 MFEs. When not on enterprise teams, my biggest pain point was library development. Pulling images takes too long, LMS has too many dependencies (chrome and firefox? really?), Studio/LMS commands take way too long because they are all wrapped in Paver
|
Troubleshooting
Info |
---|
Folks are not empowered to solve their local problems. They resort to big hammers and are blocked by external help. |
Expand | ||
---|---|---|
| ||
|
Data Provisioning
Info |
---|
Developers need the ability to provision their devstacks with “basic” test data for their most popular workflows. |
Expand | ||
---|---|---|
| ||
|
Service Dependencies & Resource Consumption
Info |
---|
Developers seek decoupled services with consistent inter-service communications, and smaller footprints. |
Expand | ||
---|---|---|
| ||
title | Other feedback
not knowing what the correct way to update devstack is (e.g. do I just pull one repo master, or all of them each time, and do I pull images every time I update the git repos for one or more repoes?). Also not knowing what to do post-updates such as I do I have re-run make requirements, migrations, etc etc. Perhaps we can have some consistent post-repo-pull or post-image-pull steps that always run to keep everyone up to date. Database state and code state could go out of date, so remembering to do migrations, static asset update etc could in the least be indicated in the log output? so one can remember to do those (if not automated) Have had docker use all of available memory and lock up my computer, Docker quickly fills up on old images and requires a prune, breaking changes/incompatibilities across repos are relatively frequent and poorly communicated, relatively hard to set up mock data/services for development. Sometimes it can be confusing when it's right to work on a repository/package/service within devstack, or directly on your local machine (ie, outside of devstack, in a venv).
Having to run migrations or rebuild requirements in LMS is really annoying Services I don't care about that spin up to 100% CPU forever, that I just shut down instead of debugging what's wrong with them. The thing about devstack frustration is that it's a slightly different broken thing every month
I often run into inscrutable front end errors building static assets that I'm not sure how to troubleshoot. make commands not listedwhen you just type `make`. I'd love if there could just be a single location with all commands so I don't have to go `grep`ping | |
Expand | ||
Expand | ||
---|---|---|
| ||
We really need to integrate frontend apps into the future of devstack; more precisely, I want a consistent framework for managing a complete, working, development environment for edX projects independent of the technologies used
Devstack itself is too sprawling and complex, so very few engineers know how to jump in and make good changes to it I think we should try hard to simplify our tooling and get back to standard docker commands for the containers, python/django commands as much as possible for backend, and standard npm/node scripts for frontend. It feels like we have too many layers, and it makes it difficult to know what the most efficient way to perform a task is at any given time. The layers are ostensibly there to make things easier/more push button, but I think they're actually just masking what's really going on, and trying to do too much at the expense of making developers just wait. Paver, for instance, feels very, very, very heavy. I'm also not really clear on why we need Makefiles. That feels very... old school. Until I came here, I hadn't used Makefiles since C/C++ classes in college. I would just love to see the results of this survey. I'm especially curious about the numbers surrounding how often people are provisioning and starting from scratch.
|
Devstack Speed
Info |
---|
Developers avoid |
Expand | ||
---|---|---|
| ||
|
Devstack Workflow
Info |
---|
Devstack usage patterns vary significantly across squads and developer seniority. |
Expand | ||
---|---|---|
| ||
|
Devstack Complexity
Info |
---|
Varying layers of technology add to devstack’s complexity. |
Expand | ||
---|---|---|
| ||
|
Decentralized Devstack: Mixed Feelings
Info |
---|
Decentralized devstack can be a viable solution, but underlying problems like provisioning and complexity still remain. |
Expand | ||
---|---|---|
| ||
Keep iterating and evolving! I often test changes in sandboxes, more driven by the inability to integrate with external systems from a development environment than by dissatisfaction with devstack. I am, however, often surprised at how positive the experience is in contrast to devstack |
Other Feedback
Expand | ||
---|---|---|
| ||
|