|Table of Contents|
Task 1: Set up Tutor for local development
Stream of consciousness… follow-up items are denoted with 🔔
Docs recommend Minikube for trying things out. Great.
Minikube setup was easy.
tutor k8s start: Failing. Unable to connect to MySQL.
tutor k8s exec 'mysql mysql --username=... --password=...', could not connect.
tutor k8s exec command arg1 arg2: this doesn't work. Needs to be
tutor k8s exec 'command arg1 arg2'.
Went back to
devmode, confirmed that I can connect to MySQL.
Brain: These are two different databases, so I need run
tutor k8s quickstart:
🔔 Hung when it got to discovery.
Disabled discovery for now.
Without discovery plugin enabled,
How do I view LMS in the browser?
local.overhang.io : unable to connect.
docs tell me to look at caddy's external IP and configure my DNS server with it.
kubectl --namespace openedx get services/caddysays that EXTERNAL_IP is
StackOverflow tells me that I need to run
minikube tunnelin order for the load balancer to work within minikube
Now, EXTERNAL_IP shows up.
Putting that IP address in the browser hangs in an encouraging way but doesn't show anything.
Caddy's logs show that I am successfully making requests to Caddy.
I'm looking at the Caddyfile. Caddyfile is defined in terms of the hostnames I have from config.yaml. That is, local.overhang.io.
Brain: Caddy is expecting URLs to be in the form of
Edit /etc/hosts, pointing local.overhang.io at the value of EXTERNAL_IP
Go to https://local.overhang.io in the browser... hangs still
Brain: Right, I don't have TLS set up
Go to http://local.overhang.io: LMS loads!
🔔 Takeaway: Would be good to note
/etc/hostschanges that were necessary for this to work locally.
Trying to log in...
Need to make a superuser
tutor k8s exec lms bash...
./manage.py lms manage_user ... --superuser
Could not load django_debug_toolabr
DJANGO_SETTINGS_MODULE=lms.envs.production ./manage.py lms manage_user ... --superuser
🔔 TODO: File an issue for DJANGO_SETTINGS_MODULE being mis-set?
Log in works!
Trying to enroll...
Whoops, no courses.
tutor k8s importdemocourse.
That worked! Enrolled.
Trying to start the learning mfe...
tutor plugins enable mfe
tutor config save
tutor k8s init --limit=mfe-> exited successfully
MFE app isn't running, logs say
Error from server (BadRequest): container "mfe" in pod "mfe-7757f78d77-qcpmr" is waiting to start: image can't be pulled
according to deployments.yml, the assigned image is docker.io/overhangio/openedx-mfe:13.0.2
docker pull docker.io/overhangio/openedx-mfe:13.0.2yields
manifest for overhangio/openedx-mfe:13.0.2 not found: manifest unknown: manifest unknown
Went to openedx-mfe on DockerHub, latest image is 12.01.
Brain: Oh, maybe I'm supposed to build this myself?
tutor images build mfe
Was going to file an issue about the missing 13.x dockerhub image, but then I found https://discuss.overhang.io/t/why-is-the-mfe-image-not-pre-built/2260/2
Image built successfully!
Tutor still wants to use the remote image, which doesn’t exist…
Pushed custom-built image to https://hub.docker.com/r/kdmccormick96/openedx-mfe, and set
MFE_DOCKER_IMAGEto point to it.
🔔 We should either:
address the lack of dynamic config overrides for MFEs (on frontend-wg’s radar with https://github.com/openedx/frontend-wg/milestone/2 , not currently being worked on, though)
add docs explaining how to work around the MFE issue by building a custom image, and somehow remove the messaging about the MFE image being missing from dockerhub… which is technically correct but doesn’t lead the user to the solution.
tutor k8s stop mfe && tutor k8s start mfe->
The Deployment "mfe" is invalid.
I think it's failing because there are old pods running still. See 1: Old pods aren't getting destroyed
Went to dashboard (
minikube dashboard) and selected the
There are still pods running, but their instance ID is different than the instance ID that
tutor k8s stopis using.
Find the instance ID from the dashboard.
kubectl delete --namespace openedx --selector=app.kubernetes.io/instance=openedx-INSTANCE_ID_FROM_DASHBOARD,app.kubernetes.io/component!=loadbalancer deployments,services,configmaps,jobs. Resources are deleted.
🔔 TODO: what was happening here?
tutor k8s startagain.
local.overhang.io hangs. Ran
tutor k8s logs lms. Get
django.db.utils.OperationalError: (1045, "Access denied for user 'openedx'@'172.18.0.1' (using password: YES)").
tutor k8s quickstartagain.
Job failed; access denied still.
tutor k8s stop.
tutor k8s quickstartagain. Still access denied.
This must be an issue with the mysql database itself. Where is mysql data stored?
it's stored using the default storage class
kubectl get storageclass-> minikube hostPath is the default -> it's somewhere in the minikube container.
maybe restarting minikube will start me off with a fresh database?
minikube stop && minikube start && tutor k8s quickstart-> same error as before
ugg, let's try
minikube delete && minikube start && tutor k8s quickstart
🔔 TODO: what was happening here?
After destroying and starting over…
Minor issue: External IP had changed. Had to update /etc/hosts
Another issue: Celery 5 seems to break lms-worker and cms-worker. This isn’t specific to k8s, and seems to have come up in the past week.
With the Celery fix in place, Studio and LMS work!
and with my custom image pushed to DockerHub, the course outline in the Learning MFE works!
Unfortunately, courseware in the Learning MFE shows “An error has occurred” with no other explanation. No JS error logs in the console or 5XXs/4XXs in the network log.
Workaround: Toggle on
courseware.use_legacy_frontendwaffle flag in order to use legacy courseware 😞
🔔 TODO: figure out why this didn’t work
Now, using legacy, courseware works.