Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

tArch-BOM sent a survey to edX and Arbisoft engineers in October 2020 to gain insight into devstack usage and pain points. This document captures the results from the survey.

Who responded (theme and expertise)?

TODO: Summarize insights from this section.


How often are developers impacted?

TODO: Summarize insights from this section.


How long do things take?

The following chart is ordered from slowest to fastest.

✨ While dev.provision is super slow, devs do not run it as often (per chart above). On the other hand, we may have big wins in optimizing dev.pull.

✨ Addressing determine commands to run may also be a big win.

(question) Now that we have a clear owner of devstack, perhaps get help will now be faster.


Free-form Comments

TODO: Review and categorize common concerns

 Comments on timing or level of knowledge

Some things like `make dev.pull` are quite slow, but I very rarely have to do them so it's not that big of a deal

Our layering of configurations is very confusing, and it's not clear what layers are for what (defaults, fallbacks, devstack values, prod, etc.)

I don't think I've gotten `make dev.static` to work locally. Also, can never remember the difference between `make dev.down` vs `make dev.stop` but I always do the wrong one.

So with the timing of different things, yes, some of them take a while. But they should be done rarely. I provision my devstack maybe 1 or 2 times per year. Same with making static. For the vast majority of the time, I will individually update my images where its pulling git, and then if it says the container is missing a package, I go in and make requirements. I think a lot of people struggle with solutions like that and think that if anything is wrong, it requires a reset when 99 times out of 100, it can be fixed much more easily.

"figuring out the right combination of commands to run" and "fixing an error", particularly one in a part of the system I may depend on but don't directly work on, are some of the biggest issues.

I usually use the docker commands over the make commands (pull / up) so I'm less likely to pull all the stuff I don't need to.

I am still relatively new but as I tough more areas I get confused sometimes which service belongs in devstack and which I have to manually setup. Ideally each repo has a similar command like make dev.up, and 'it just works' :)

I don't know when "make dev.provision" is needed

It's unclear when refreshing devstack if a set of commands needs to be run,

I've been using devstack for many years and while I know it's a constant work in progress it has really come a long way since the "old days"

I know how to change test data via discovery but nothing else. For website we only ever really need discovery so it works ok.

For local test data: definitely depends on which service I need to dust off and remember about.

There are a lot of common devstack problems that once you've encountered enough times you can easily work through by habit. Inherently, it's not very friendly for beginners.

I often need example courses. It would be really great if I could run a management command to create a course that has all the correct data so that I can then use it with an enterprise front end.

A significant problem is network. Any dev.pull takes about an hour for me. Because I am now an infrequent user I have to pretty much always re-provision whenever I need to do a devstack task, which is often closer to a 4 hr ordeal.

The structure of Django projects is new to me and very different from microfrontends.

I’m new-ish and most of my tickets have been in Prospectus; I haven’t touched devstack in months and didn’t get much time with it in before that.

My workflow involves very rarely updating the containers. I mostly use stop and up. I mostly use LMS and a local dependency to a library I am working on. If I need newer edx-platform code I usually run git pull and then run whatever paver commands I need, which is often none, and sometimes just updating requirements and more rarely updating db.

 Other issues, problems, delays, and frustrations

This is more specific to enterprise - having to manage a bunch of different services at one time - catalog, license manager, the LMS, and then 1 or 2 MFEs. When not on enterprise teams, my biggest pain point was library development.

Pulling images takes too long, LMS has too many dependencies (chrome and firefox? really?), Studio/LMS commands take way too long because they are all wrapped in Paver

Would be nice if devstack came out of the box with all the data/setup needed to complete the most popular workflows in the UI (e.g. checkout a course, search for a course/program, create a course, etc.) Also would be nice if services were either more decoupled or kept more in sync in local environments - if I only really work on ecommerce, I'm probably not going to go out of my way to keep the forums repo up to date, but I do expect to be able to navigate around the course pages locally and not see random errors on the discussions tab. And apparently `make dev.down` is not the opposite of `make dev.up`.

Thank god for slack :) That's my greatest resources when I hit something weird in devstack

The biggest issue with devstack in my opinion is that it is not always easy setting up a devstack to accurately represent a production environment. This comes in many forms, but some examples are setting up data for scale testing (sure, it works with my 3 courses locally, but will it be fine with 15,000 on prod?), setting up "weird" content (like special exams or proctored exams), my note below about having a verified learner, and setting up flags to try and mirror the state for edX. I'm sure there are others, but these are just the ones that first popped into my head. THERE IS NO VERIFIED USER. Yes, I know we have verified@example.com, BUT that user isn't even enrolled in the Demo course as a verified learner. In order to set up a verified user in a course, I have to create a course, go into ecommerce, reduce the price to 0, and then purchase a free verified seat. It's a bit crazy to me how difficult it is to do anything but audit a course locally when most of the features we develop have different experiences based on audit/paid. The time it takes for code to update is sometimes pretty slow (as in Watchman triggering the reload for your application). Also running tests if reuse-db is not properly set up/used. Ah, another frustration is when a redirect might go to something like edx.devstack.lms (which doesn't work) so then you have to manually go in and switch it to localhost:18000. We just need to fix all uses that are broken so our app can just work.

Finding answers when issues come up; searching and asking in Slack doesn't even really scale internally. Also, we need to work out instructions for setting up a case-sensitive volume for devstack on macOS. Ecommerce now has dependencies that are confused by macOS's case-insensitive default ("No module named 'Crypto'"), and git behaves quite badly when there are case-only differentiated files.

Every service requiring LMS and LMS being broken. And not always "broken" broken, just things like master getting pulled, then the code not matching the requirements, etc. I'd like there to be a stable LMS for when I need to develop other services and LMS doesn't matter to me other than auth. Dealing with static assets is a nightmare in LMS, particularly if you need to test theming.

not knowing what the correct way to update devstack is (e.g. do I just pull one repo master, or all of them each time, and do I pull images every time I update the git repos for one or more repoes?). Also not knowing what to do post-updates such as I do I have re-run make requirements, migrations, etc etc. Perhaps we can have some consistent post-repo-pull or post-image-pull steps that always run to keep everyone up to date. Database state and code state could go out of date, so remembering to do migrations, static asset update etc could in the least be indicated in the log output? so one can remember to do those (if not automated)

Have had docker use all of available memory and lock up my computer, Docker quickly fills up on old images and requires a prune, breaking changes/incompatibilities across repos are relatively frequent and poorly communicated, relatively hard to set up mock data/services for development.

Sometimes it can be confusing when it's right to work on a repository/package/service within devstack, or directly on your local machine (ie, outside of devstack, in a venv).

Needing to use the lms for login when I want to work in publisher or discovery. I rarely work in the lms itself, so can easily get stuck if the lms container refuses to come up for any reason.

Having to run migrations or rebuild requirements in LMS is really annoying

Services I don't care about that spin up to 100% CPU forever, that I just shut down instead of debugging what's wrong with them. The thing about devstack frustration is that it's a slightly different broken thing every month. Maybe there's some npm install weirdness one time. The next time it'll be that MongoDB doesn't start up correctly unless you run the upgrade script every time, etc.

Missing configurations between services.

Getting different services to talk to each other is a problem. I'm always having to figure out how services are talking to each other and what needs to have permissions for what, etc. (CORS permissions, worker permissions, etc.) Getting help is slow, and the help I get doesn't usually empower me to solve my own problems in the future.

I often run into inscrutable front end errors building static assets that I'm not sure how to troubleshoot.

make commands not listedwhen you just type `make`. I'd love if there could just be a single location with all commands so I don't have to go `grep`ping

 Other feedback

How much memory and compute resources devstack uses on my laptop, I think this is going to continue to get worse until we get machines with 32GB of RAM, or take some measures to use fewer resources with our MySQL and LMS containers. Maybe every service can actually use the same mysql container?

Devstack itself is too sprawling and complex, so very few engineers know how to jump in and make good changes to it

I think we should try hard to simplify our tooling and get back to standard docker commands for the containers, python/django commands as much as possible for backend, and standard npm/node scripts for frontend. It feels like we have too many layers, and it makes it difficult to know what the most efficient way to perform a task is at any given time. The layers are ostensibly there to make things easier/more push button, but I think they're actually just masking what's really going on, and trying to do too much at the expense of making developers just wait. Paver, for instance, feels very, very, very heavy. I'm also not really clear on why we need Makefiles. That feels very... old school. Until I came here, I hadn't used Makefiles since C/C++ classes in college.

I would just love to see the results of this survey. I'm especially curious about the numbers surrounding how often people are provisioning and starting from scratch.

We really need to integrate frontend apps into the future of devstack; more precisely, I want a consistent framework for managing a complete, working, development environment for edX projects independent of the technologies used.

I'm very on board with DD, but I still don't fully understand how test data gets propagated and where we store things (scripts, sql dumps) for things like "run this to have all the data populated for a full purchase flow" or "run this to test Masters stuff".

love the decentralized work! have not started using it yet but happy to try when I can

Keep iterating and evolving!

I often test changes in sandboxes, more driven by the inability to integrate with external systems from a development environment than by dissatisfaction with devstack. I am, however, often surprised at how positive the experience is in contrast to devstack.

I've been working here for since June and I've lost *at least* a month's worth of work due to problems with devstack. I touch a lot of different repos and services and I want them to work together smoothly and out of the box.

There seems to be a focus on 'decentralizing' devstack? or running services seperately? tbh i don't care about this and it just makes me think I'm gonna have to relearn how everything works.

  • No labels