Devstack Issue Tracking

Devstack issues are now being collected as GitHub issues on the Devstack repository. Please add new issues as they come in straight into GitHub. https://github.com/openedx/devstack/issues . We will be moving over the issues from this page as we determine if they are still relevant.

A space to collaboratively track issues and solutions to Docker Devstack problems. This was moved over here from our internal engineering wiki. It is self-maintained by edX.org engineers who rely on Devstack. It is not meant to be an authoritative list of open issues.

Devstack documentation in github is now published to Readthedocs. Please update the docs where and when it makes sense.

There’s a previous version of something very similar to this: https://openedx.atlassian.net/wiki/spaces/OXA/overview?homepageId=157690905

Open Issues

Problem

People who’ve experienced it

Impacted Teams & Dates

Solution

Notes

Ticket / PR for fix (if applicable)

Problem

People who’ve experienced it

Impacted Teams & Dates

Solution

Notes

Ticket / PR for fix (if applicable)

Ecommerce logout leads to edx.devstack.lms instead of the localhost address failing logout attempts (it looks like things hang)

@Binod Pant (Deactivated)

Everyone who uses ecom locally

Login to ecom and try to logout: http://localhost:18130/enterprise/coupons/

 

https://openedx.atlassian.net/browse/ENT-3964

Lost connection to docker containers (all requests were hanging, as were make lms-logs and make lms-shell

@Lael Birch (Deactivated)

Quokkas

make down then make up
If that errors, restart docker entirely

I had to restart docker to fix this. It has not recurred.

 

ImportError: No module named 'safe_lxml'

@Lael Birch (Deactivated)

Quokkas

  • cd /edx/src/edx-enterprise

  • rm -rf venv/enterprise

  • exit container, do docker-compose restart lms

  • make lms-shell

  • In the shell, cd /edx/src/edx-enterprise

  • virtualenv venv/enterprise

  • source venv/enterprise/bin/activate

  • do which pip and which python to make sure we’re using the enterprise venv now

  • make requirements

This was likely caused by not properly managing the edx-enterprise venv.

Proposed solution:
I would love to add something to the container that automatically activates the edx-enterprise virtualenv when you move into the edx/src/edx-enterprise folder.

 

CSS/JS not loading (aka assets missing). Particularly, login page has no username and password field.

@Talia Rhodes (Deactivated) @Lael Birch (Deactivated)

Quokkas

make lms-static (or in an lms shell paver update_assets)

(if that errors, go into the lms shell and run rm -rf ./.prereqs_cache and rerun make lms-static)

If that fails, your container and/or edx-platform is in a real bad state. The next option depends on which of these is more annoying to you (may depend on how much uncommitted work you have in edx-platform)

a) Try make dev.down, docker image rm $(docker image ls -q) make dev.pull then make dev.up, rebuild assets again after that.

b) delete your edx-platform checkout directory and reclone it. Some weird stuff is getting cached there. (I tried A. a bunch, also deleting volumes and ultimately this option is what fixed it)

There were no errors except for 404s when this happens, so it’s somewhat difficult to understand what’s going on.

 

ImportError: cannot import name 'reset_problem_s' after running make dev.up

@Alexander Sheehan

quokkas

Ended up re-pulling master for edx-platform repo itself as none of us saw a reference to that code. Unclear what underlying issue was.

 

 

“An unexpected error has occurred.” on LMS (localhost:18000) or on Django Admin login: Forbidden (403) page with “CSRF verification failed. Request aborted.”

@Talia Rhodes (Deactivated) @Jansen Kantor @Adam Butterworth (Deactivated)

Quokkas, Masters

This appears to be due to Google starting to enforce Same Site security.

The latest edx-platform code has a fix.

 

TR: In my case it started occurring after I had to forcibly shutdown Chrome as it had frozen. Clearing cookie/cache did nothing. All LMS/Devstack side changes did nothing.

 

 

A new or modified package-lock.json shows up in various repos after some devstack operations, making those repos dirty

@Tim McCormack

 

 

 

 

makemigrations sometimes makes migrations in apps you’re not editing.

 

 

Don’t run makemigrations outside the app you are modifying. eg. run python manage.py lms makemigrations <app_name> where app_name is the name of the django app for which you change the model. If you didn’t change any models, don’t run makemigrations.

Some apps may not have migrations created on purpose because they are doing a expand/contract operation to remove fields or because an entire django app is going to go away. See examples in this section of the https://openedx.atlassian.net/wiki/spaces/AC/pages/23003228 page.

 

A change in credentials broke provisioning in devstack – but no one noticed until it blocked an unrelated devstack PR

@Tim McCormack

Arch-BOM, 2021-04-28

Fix provision-credentials.sh script in devstack to match the change in credentials repo

We don’t have alerting on devstack master tests, and they only run on merge. Maybe we should run daily and to get a full integration test – and having someone be alerted. (May need to stabilize the tests more first.)

Fixed Issues

Studio redirects to edx.devstack.lms after sign-in instead of localhost:18010. The result is a DNS failure. Studio login appears broken to the user.

Kyle McCormick Justin Lapierre

Potentially all

Log in to LMS first (localhost:18000/login) and then load Studio (localhost:18010).

I believe this is due to the fact that we use edx.devstack.lms for certain devstack “base URL” settings, for JWT-related reasons. When we end up serving this as part of an LMS redirect URL (instead of localhost:18000), though, it breaks.

Set size changed during iteration error

@Lael Birch (Deactivated) @Binod Pant (Deactivated) @Talia Rhodes (Deactivated)

Quokkas

Sometimes works, sometimes doesn’t
run make studio-update-db (or whatever service is relevant)
make lms-restart

 

This problem has come up multiple times for multiple people. The solution has worked intermittently.