Arch-BOM sent a survey to edX and Arbisoft engineers in October 2020 to gain insight into devstack usage and pain points. This document captures summarizes the results of the survey.
Table of Contents | ||||
---|---|---|---|---|
|
Who responded (theme and expertise)?
Note |
---|
TODO: Summarize insights from this section. |
...
Info |
---|
✨ 40 people responded, of which 38 classified into a theme. |
...
Info |
---|
✨ There is a normal distribution (bell-curve) of self-reported expertise with devstack. |
...
Info |
---|
✨ Few people know how to add test data (but many think they could figure it out). |
...
Info |
---|
How to change configuration is a mystery to about ⅓ of respondents. |
...
How often are developers impacted?
Note |
---|
TODO: Summarize insights from this section. |
...
Info |
---|
Most folks use devstack often (few times a week or more). |
...
Info |
---|
About ⅔ of respondents run into devstack issues at least multiple times a month. |
...
Info |
---|
People don’t use “make destroy” very often (a few times a year or never). |
...
Info |
---|
✨ Most people don’t re-provision very often. |
...
How long do things take?
Info |
---|
The following chart is ordered from slowest to fastest. ✨ While ✨ Addressing It is still difficult to ❓ Now that we have a clear owner of devstack, perhaps |
...
Free-form Comments
Note |
---|
TODO: Review and categorize common concerns |
...
title | Comments on timing or level of knowledge |
---|
Some things like `make dev.pull` are quite slow, but I very rarely have to do them so it's not that big of a deal
...
Commands Know-How
Info |
---|
There is uncertainty around which commands to run when. We have big hammers (like reprovisioning all of devstack), but not clear regularly-needed hammers. |
Expand | ||
---|---|---|
| ||
So with the timing of different things, yes, some of them take a while. But they should be done rarely. I provision my devstack maybe 1 or 2 times per year. Same with making static. For the vast majority of the time, I will individually update my images where its pulling git, and then if it says the container is missing a package, I go in and make requirements. I think a lot of people struggle with solutions like that and think that if anything is wrong, it requires a reset when 99 times out of 100, it can be fixed much more easily.
|
Out-of-the-box Consistent Experience
Info |
---|
Desire to have services work together smoothly out of the box. |
Expand | ||
---|---|---|
I've been using devstack for many years and while I know it's a constant work in progress it has really come a long way since the "old days" I know how to change test data via discovery but nothing else. For website we only ever really need discovery so it works ok. For local test data: definitely depends on which service I need to dust off and remember about. There are a lot of common devstack problems that once you've encountered enough times you can easily work through by habit. Inherently, it's not very friendly for beginners. I often need example courses. It would be really great if I could run a management command to create a course that has all the correct data so that I can then use it with an enterprise front end. A significant problem is network. Any dev.pull takes about an hour for me. Because I am now an infrequent user I have to pretty much always re-provision whenever I need to do a devstack task, which is often closer to a 4 hr ordeal. The structure of Django projects is new to me and very different from microfrontends. I’m new-ish and most of my tickets have been in Prospectus; I haven’t touched devstack in months and didn’t get much time with it in before that. My workflow involves very rarely updating the containers. I mostly use stop and up. I mostly use LMS and a local dependency to a library I am working on. If I need newer edx-platform code I usually run git pull and then run whatever paver commands I need, which is often none, and sometimes just updating requirements and more rarely updating db. | ||
Expand | ||
| ||
This is more specific to enterprise - having to manage a bunch of different services at one time - catalog, license manager, the LMS, and then 1 or 2 MFEs. When not on enterprise teams, my biggest pain point was library development. Pulling images takes too long, LMS has too many dependencies (chrome and firefox? really?), Studio/LMS commands take way too long because they are all wrapped in Paver
|
Troubleshooting
Info |
---|
Folks are not empowered to solve their local problems. They resort to big hammers and are blocked by external help. |
Expand | ||
---|---|---|
| ||
|
Data Provisioning
Info |
---|
Developers need the ability to provision their devstacks with “basic” test data for their most popular workflows. |
Expand | ||
---|---|---|
| ||
|
Service Dependencies & Resource Consumption
Info |
---|
Developers seek decoupled services with consistent inter-service communications, and smaller footprints. |
Expand | ||
---|---|---|
| ||
Have had docker use all of available memory and lock up my computer, Docker quickly fills up on old images and requires a prune, breaking changes/incompatibilities across repos are relatively frequent and poorly communicated, relatively hard to set up mock data/services for development. Sometimes it can be confusing when it's right to work on a repository/package/service within devstack, or directly on your local machine (ie, outside of devstack, in a venv).
Having to run migrations or rebuild requirements in LMS is really annoying Services I don't care about that spin up to 100% CPU forever, that I just shut down instead of debugging what's wrong with them. The thing about devstack frustration is that it's a slightly different broken thing every month. Maybe there's some npm install weirdness one time
I often run into inscrutable front end errors building static assets that I'm not sure how to troubleshoot. make commands not listedwhen you just type `make`. I'd love if there could just be a single location with all commands so I don't have to go `grep`ping | ||
Expand | ||
| ||
Expand | ||
---|---|---|
| ||
We really need to integrate frontend apps into the future of devstack; more precisely, I want a consistent framework for managing a complete, working, development environment for edX projects independent of the technologies used
Devstack itself is too sprawling and complex, so very few engineers know how to jump in and make good changes to it I think we should try hard to simplify our tooling and get back to standard docker commands for the containers, python/django commands as much as possible for backend, and standard npm/node scripts for frontend. It feels like we have too many layers, and it makes it difficult to know what the most efficient way to perform a task is at any given time. The layers are ostensibly there to make things easier/more push button, but I think they're actually just masking what's really going on, and trying to do too much at the expense of making developers just wait. Paver, for instance, feels very, very, very heavy. I'm also not really clear on why we need Makefiles. That feels very... old school. Until I came here, I hadn't used Makefiles since C/C++ classes in college. I would just love to see the results of this survey. I'm especially curious about the numbers surrounding how often people are provisioning and starting from scratch.
|
Devstack Speed
Info |
---|
Developers avoid |
Expand | ||
---|---|---|
| ||
|
Devstack Workflow
Info |
---|
Devstack usage patterns vary significantly across squads and developer seniority. |
Expand | ||
---|---|---|
| ||
|
Devstack Complexity
Info |
---|
Varying layers of technology add to devstack’s complexity. |
Expand | ||
---|---|---|
| ||
|
Decentralized Devstack: Mixed Feelings
Info |
---|
Decentralized devstack can be a viable solution, but underlying problems like provisioning and complexity still remain. |
Expand | ||
---|---|---|
| ||
Keep iterating and evolving! I often test changes in sandboxes, more driven by the inability to integrate with external systems from a development environment than by dissatisfaction with devstack. I am, however, often surprised at how positive the experience is in contrast to devstack |
Other Feedback
Expand | ||
---|---|---|
| ||
|