Task 1: Set up Tutor for local development

Goal

• Using a branch B, which is based on openedx/edx-platform:master, I want to:

  • Run LMS and CMS with local code

  • Run static analysis & tests

  • Run a management command

Relevant Tutor Docs

• “Open edX Development”: https://docs.tutor.overhang.io/dev.html

  • Particularly, this section: https://docs.tutor.overhang.io/dev.html#setting-up-a-development-environment-for-edx-platform

• “Running Open edX on the master branch (“nightly”)”: https://docs.tutor.overhang.io/tutorials/nightly.html

Issues Encountered

• When working out of a repo generated by tutor dev bindmount:

  • git state was confusing as a newcomer

    • no branch information

    • patches applied on top of master - wasn’t clear where they came from at first

    • copied static assets and custom settings files made git state dirty

  • git remote protocol needed to be switched from https to ssh to allow push

  • git had to be configured to pull relevant branches

• When using tutor dev start ... or tutor dev exec ... :

  • settings weren’t set correctly? Regis mentioned that exec doesn’t set DJANGO_SETTINGS_MODULE properly…

• When setting COMMON_OPENEDX_VERSION to my feature branch and trying tutor images build openedx :

  • If branch has updates, needed to use --no-cache, which took a very long time.

  • Had to override openedx-dockerfile-git-patches-default to be empty, otherwise cherry-picking would conflict with the state of my branch (because said commits are already on master, it seems?)

• Volume mounting requires full path; ~ for home didn’t work.

• Had to start from scratch due to < insert requirements problem that carlos had >

Successful Approach

Suggested Improvements

This was just a first pass at suggested improvements. We’ve been iterating these ideas in some GitHub issues:

• https://github.com/overhangio/2u-tutor-adoption/issues/43

Task 2: Set up Tutor in a local Kubernetes (k8s) cluster

Goal

Run LMS, CMS and the Learning MFE in a k8s cluster on my local machine using Tutor

Relevant Tutor Docs

Notes

Stream of consciousness… follow-up items are denoted with

• Docs recommend Minikube for trying things out. Great.

  • Minikube setup was easy.

• tutor k8s start: Failing. Unable to connect to MySQL.

  • Tried tutor k8s exec 'mysql mysql --username=... --password=...', could not connect.

  • Also, tutor k8s exec command arg1 arg2: this doesn't work. Needs to be tutor k8s exec 'command arg1 arg2'.

    • Wrote up

  • Went back to dev mode, confirmed that I can connect to MySQL.

  • Brain: These are two different databases, so I need run quickstart again.

• tutor k8s quickstart:

  • Hung when it got to discovery.

  • Disabled discovery for now.

  • Without discovery plugin enabled, quickstart succeeds 👍🏻

• How do I view LMS in the browser?

  • local.overhang.io : unable to connect.

  • docs tell me to look at caddy's external IP and configure my DNS server with it.

  • kubectl --namespace openedx get services/caddy says that EXTERNAL_IP is <pending>

  • StackOverflow tells me that I need to run minikube tunnel in order for the load balancer to work within minikube

  • Now, EXTERNAL_IP shows up.

  • Putting that IP address in the browser hangs in an encouraging way but doesn't show anything.

  • Caddy's logs show that I am successfully making requests to Caddy.

  • I'm looking at the Caddyfile. Caddyfile is defined in terms of the hostnames I have from config.yaml. That is, local.overhang.io.

  • Brain: Caddy is expecting URLs to be in the form of *.local.overhang.io/*

  • Edit /etc/hosts, pointing local.overhang.io at the value of EXTERNAL_IP

  • Go to https://local.overhang.io in the browser... hangs still

  • Brain: Right, I don't have TLS set up

  • Go to http://local.overhang.io: LMS loads!

  • Takeaway: Would be good to note minikube tunnel and /etc/hosts changes that were necessary for this to work locally.

• Trying to log in...

  • Need to make a superuser

  • tutor k8s exec lms bash ... ./manage.py lms manage_user ... --superuser

  • Could not load django_debug_toolabr

  • DJANGO_SETTINGS_MODULE=lms.envs.production ./manage.py lms manage_user ... --superuser

    • TODO: File an issue for DJANGO_SETTINGS_MODULE being mis-set?

  • Log in works!

• Trying to enroll...

  • Whoops, no courses.

  • tutor k8s importdemocourse.

  • That worked! Enrolled.

• Trying to start the learning mfe...

  • tutor plugins enable mfe

  • tutor config save

  • tutor k8s init --limit=mfe -> exited successfully

  • MFE app isn't running, logs say Error from server (BadRequest): container "mfe" in pod "mfe-7757f78d77-qcpmr" is waiting to start: image can't be pulled

    • according to deployments.yml, the assigned image is docker.io/overhangio/openedx-mfe:13.0.2

    • docker pull docker.io/overhangio/openedx-mfe:13.0.2 yields manifest for overhangio/openedx-mfe:13.0.2 not found: manifest unknown: manifest unknown

    • Went to openedx-mfe on DockerHub, latest image is 12.01.

    • Brain: Oh, maybe I'm supposed to build this myself?

    • tutor images build mfe

      • Was going to file an issue about the missing 13.x dockerhub image, but then I found

      • Image built successfully!

      • Tutor still wants to use the remote image, which doesn’t exist…

      • Pushed custom-built image to https://hub.docker.com/r/kdmccormick96/openedx-mfe, and set MFE_DOCKER_IMAGE to point to it.

      • We should either:

        • address the lack of dynamic config overrides for MFEs (on frontend-wg’s radar with , not currently being worked on, though)

        • add docs explaining how to work around the MFE issue by building a custom image, and somehow remove the messaging about the MFE image being missing from dockerhub… which is technically correct but doesn’t lead the user to the solution.

  • tutor k8s stop mfe && tutor k8s start mfe -> The Deployment "mfe" is invalid.

    • I think it's failing because there are old pods running still. See 1: Old pods aren't getting destroyed

    • Went to dashboard (minikube dashboard) and selected the openedx namespace.

    • There are still pods running, but their instance ID is different than the instance ID that tutor k8s stop is using.

    • Find the instance ID from the dashboard.

    • Run kubectl delete --namespace openedx --selector=app.kubernetes.io/instance=openedx-INSTANCE_ID_FROM_DASHBOARD,app.kubernetes.io/component!=loadbalancer deployments,services,configmaps,jobs. Resources are deleted.

    • TODO: what was happening here?

  • Run tutor k8s start again.

    • local.overhang.io hangs. Ran tutor k8s logs lms. Get django.db.utils.OperationalError: (1045, "Access denied for user 'openedx'@'172.18.0.1' (using password: YES)").

    • Ran tutor k8s quickstart again.

    • Job failed; access denied still.

    • Stop containers: tutor k8s stop.

    • Ran tutor k8s quickstart again. Still access denied.

    • Brain:

      • This must be an issue with the mysql database itself. Where is mysql data stored?

      • checking volumes.yml...

      • it's stored using the default storage class

      • kubectl get storageclass -> minikube hostPath is the default -> it's somewhere in the minikube container.

      • maybe restarting minikube will start me off with a fresh database?

      • minikube stop && minikube start && tutor k8s quickstart -> same error as before

      • ugg, let's try minikube delete && minikube start && tutor k8s quickstart

      • TODO: what was happening here?

  • After destroying and starting over…

    • Minor issue: External IP had changed. Had to update /etc/hosts

    • Another issue: Celery 5 seems to break lms-worker and cms-worker. This isn’t specific to k8s, and seems to have come up in the past week.

    • With the Celery fix in place, Studio and LMS work!

    • and with my custom image pushed to DockerHub, the course outline in the Learning MFE works!

    • Unfortunately, courseware in the Learning MFE shows “An error has occurred” with no other explanation. No JS error logs in the console or 5XXs/4XXs in the network log.

      • Workaround: Toggle on courseware.use_legacy_frontendwaffle flag in order to use legacy courseware

      • TODO: figure out why this didn’t work

    • Now, using legacy, courseware works.

References

1: Old pods aren't getting destroyed