Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 12 Current »

This is not an official guide! This is just a log of my (Kyle McCormick's) experiences running Tutor locally, both for my own reference, and for the community’s reference as we work to improve Tutor and it’s official docs via the Tutor Adoption Initiative.

Task 1: Set up Tutor for local development

Goal

  • Using a branch B, which is based on openedx/edx-platform:master, I want to:

    • Run LMS and CMS with local code

    • Run static analysis & tests

    • Run a management command

Relevant Tutor Docs

Issues Encountered

  • When working out of a repo generated by tutor dev bindmount:

    • git state was confusing as a newcomer

      • no branch information

      • patches applied on top of master - wasn’t clear where they came from at first

      • copied static assets and custom settings files made git state dirty

    • git remote protocol needed to be switched from https to ssh to allow push

    • git had to be configured to pull relevant branches

  • When using tutor dev start ... or tutor dev exec ... :

    • settings weren’t set correctly? Regis mentioned that exec doesn’t set DJANGO_SETTINGS_MODULE properly…

  • When setting COMMON_OPENEDX_VERSION to my feature branch and trying tutor images build openedx :

    • If branch has updates, needed to use --no-cache, which took a very long time.

    • Had to override openedx-dockerfile-git-patches-default to be empty, otherwise cherry-picking would conflict with the state of my branch (because said commits are already on master, it seems?)

  • Volume mounting requires full path; ~ for home didn’t work.

  • Had to start from scratch due to < insert requirements problem that carlos had >

Successful Approach

# Clone edx-platform and switch to my branch.
$ git clone git@github.com:openedx/edx-platform
$ (cd edx-platform && git checkout $B)

# Install Tutor Nightly.
$ virtualenv tutor-venv
$ source tutor-venv/bin/activate
$ git clone --branch nightly git@github.com:overhangio/tutor
$ cd tutor
$ pip install -r requirements/dev.txt
$ pip install -e .

# Provision Tutor.
$ tutor local quickstart
$ tutor local stop

# Copy edx-platform virtual environment to host.
$ tutor dev start lms
$ tutor dev bindmount lms /openedx/venv
$ tutor dev stop lms
$ tutor dev dc rm

# Configure Tutor to mount your local edx-platform and virtual environment
# by writing a env/dev/docker-compose.override.yml file to your Tutor env.
# The syntax for each mount is $HOST_PATH:$DOCKER_PATH.
# Be sure to substitute in the appropriate $HOST_PATH for each mount.
$ cat > $(tutor config printroot)/env/dev/docker-compose.override.yml <<- EOF
version: "3.7"
services:
  lms:
    volumes:
      - /home/kyle/openedx/edx-platform:/openedx/edx-platform
      - /home/kyle/.local/share/tutor-nightly/volumes/venv:/openedx/venv
  cms:
    volumes:
      - /home/kyle/openedx/edx-platform:/openedx/edx-platform
      - /home/kyle/.local/share/tutor-nightly/volumes/venv:/openedx/venv
  lms-worker:
    volumes:
      - /home/kyle/openedx/edx-platform:/openedx/edx-platform
      - /home/kyle/.local/share/tutor-nightly/volumes/venv:/openedx/venv
  cms-worker:
    volumes:
      - /home/kyle/openedx/edx-platform:/openedx/edx-platform
      - /home/kyle/.local/share/tutor-nightly/volumes/venv:/openedx/venv
EOF

# Install requirements, provision demo course, admin user and static assets.
$ tutor dev run lms make requirements
$ tutor dev run lms npm install
$ tutor dev run lms openedx-assets build --env=dev
$ tutor dev createuser admin admin@example.com --password admin --staff --superuser
$ tutor dev importdemocourse

# (NOT REQUIRED - JUST EXAMPLES) Run tests, linting, and a management command.
$ tutor dev run lms pytest path/to/some/code
$ tutor dev run lms pylint path/to/some/code
$ tutor dev run lms ./manage.py lms run_some_management_command
                            
# Run LMS and CMS.
$ tutor dev start -d lms cms

Suggested Improvements

This was just a first pass at suggested improvements. We’ve been iterating these ideas in some GitHub issues:

# Clone edx-platform and switch to my branch.
$ git clone git@github.com:openedx/edx-platform
$ (cd edx-platform && git checkout $B)

# Install Tutor Nightly.
$ virtualenv tutor-venv
$ source tutor-venv/bin/activate
# Either:
 $ git clone --branch nightly git@github.com:overhangio/tutor
 $ cd tutor
 $ make requirements
# OR:
 # tutor-nightly could be a metapackage depending on the latest nightly (`N.dev`) tutor release
 $ pip install tutor-nightly

# Configure mounting:
#   * from my edx-platform to the container's /openedx/edx-platform, and
#   * from the default location in tutor-nightly config to /openedx/venv.
$ tutor config save --set OPENEDX_MOUNTS=/home/kyle/openedx/edx-platfrom:/openedx/edx-platform,/openedx/venv

# Provision Tutor, to include a default user as well as static assets.
$ tutor dev quickinit

# Run tests, linting, and a management command.
$ tutor dev run bash
app@lms$ pytest path/to/some/code
app@lms$ pylint path/to/some/code
app@lms$ ./manage.py lms run_some_management_command
app@lms$ exit
                            
# Run LMS, with virtual environment and application code from host.
$ tutor dev runserver -d lms cms

Task 2: Set up Tutor in a local Kubernetes (k8s) cluster

Goal

Run LMS, CMS and the Learning MFE in a k8s cluster on my local machine using Tutor

Relevant Tutor Docs

https://docs.tutor.overhang.io/k8s.html

Notes

Stream of consciousness…

  • Docs recommend Minikube for trying things out. Great.

    • Minikube setup was easy.

  • tutor k8s start: Failing. Unable to connect to MySQL.

    • Tried tutor k8s exec 'mysql mysql --username=... --password=...', could not connect.

    • Also, tutor k8s exec command arg1 arg2: this doesn't work. Needs to be tutor k8s exec 'command arg1 arg2'.

    • Went back to dev mode, confirmed that I can connect to MySQL.

    • Brain: These are two different databases, so I need run quickstart again.

  • tutor k8s quickstart:

    • 🔔 Hung when it got to discovery.

    • Disabled discovery for now.

    • Without discovery plugin enabled, quickstart succeeds 👍🏻

  • How do I view LMS in the browser?

    • local.overhang.io : unable to connect.

    • docs tell me to look at caddy's external IP and configure my DNS server with it.

    • kubectl --namespace openedx get services/caddy says that EXTERNAL_IP is <pending>

    • StackOverflow tells me that I need to run minikube tunnel in order for the load balancer to work within minikube

    • Now, EXTERNAL_IP shows up.

    • Putting that IP address in the browser hangs in an encouraging way but doesn't show anything.

    • Caddy's logs show that I am successfully making requests to Caddy.

    • I'm looking at the Caddyfile. Caddyfile is defined in terms of the hostnames I have from config.yaml. That is, local.overhang.io.

    • Brain: Caddy is expecting URLs to be in the form of *.local.overhang.io/*

    • Edit /etc/hosts, pointing local.overhang.io at the value of EXTERNAL_IP

    • Go to https://local.overhang.io in the browser... hangs still

    • Brain: Right, I don't have TLS set up

    • Go to http://local.overhang.io: LMS loads!

    • 🔔 Takeaway: Would be good to note minikube tunnel and /etc/hosts changes that were necessary for this to work locally.

  • Trying to log in...

    • Need to make a superuser

    • tutor k8s exec lms bash ... ./manage.py lms manage_user ... --superuser

    • Could not load django_debug_toolabr

    • DJANGO_SETTINGS_MODULE=lms.envs.production ./manage.py lms manage_user ... --superuser

      • 🔔 TODO: File an issue for DJANGO_SETTINGS_MODULE being mis-set?

    • Log in works!

  • Trying to enroll...

    • Whoops, no courses.

    • tutor k8s importdemocourse.

    • That worked! Enrolled.

  • Trying to start the learning mfe...

    • tutor plugins enable mfe

    • tutor config save

    • tutor k8s init --limit=mfe -> exited successfully

    • MFE app isn't running, logs say Error from server (BadRequest): container "mfe" in pod "mfe-7757f78d77-qcpmr" is waiting to start: image can't be pulled

      • according to deployments.yml, the assigned image is docker.io/overhangio/openedx-mfe:13.0.2

      • docker pull docker.io/overhangio/openedx-mfe:13.0.2 yields manifest for overhangio/openedx-mfe:13.0.2 not found: manifest unknown: manifest unknown

      • Went to openedx-mfe on DockerHub, latest image is 12.01.

      • Brain: Oh, maybe I'm supposed to build this myself?

      • tutor images build mfe

    • tutor k8s stop mfe && tutor k8s start mfe -> The Deployment "mfe" is invalid.

      • I think it's failing because there are old pods running still. See 1: Old pods aren't getting destroyed

      • Went to dashboard (minikube dashboard) and selected the openedx namespace.

      • There are still pods running, but their instance ID is different than the instance ID that tutor k8s stop is using.

      • Find the instance ID from the dashboard.

      • Run kubectl delete --namespace openedx --selector=app.kubernetes.io/instance=openedx-INSTANCE_ID_FROM_DASHBOARD,app.kubernetes.io/component!=loadbalancer deployments,services,configmaps,jobs. Resources are deleted.

      • 🔔 TODO: what was happening here?

    • Run tutor k8s start again.

      • local.overhang.io hangs. Ran tutor k8s logs lms. Get django.db.utils.OperationalError: (1045, "Access denied for user 'openedx'@'172.18.0.1' (using password: YES)").

      • Ran tutor k8s quickstart again.

      • Job failed; access denied still.

      • Stop containers: tutor k8s stop.

      • Ran tutor k8s quickstart again. Still access denied.

      • Brain:

        • This must be an issue with the mysql database itself. Where is mysql data stored?

        • checking volumes.yml...

        • it's stored using the default storage class

        • kubectl get storageclass -> minikube hostPath is the default -> it's somewhere in the minikube container.

        • maybe restarting minikube will start me off with a fresh database?

        • minikube stop && minikube start && tutor k8s quickstart -> same error as before

        • ugg, let's try minikube delete && minikube start && tutor k8s quickstart

        • 🔔 TODO: what was happening here?

    • After destroying and starting over…

      • Minor issue: External IP had changed. Had to update /etc/hosts

      • Another issue: Celery 5 seems to break lms-worker and cms-worker. This isn’t specific to k8s, and seems to have come up in the past week.

      • With the Celery fix in place, Studio and LMS work!

      • and with my custom image pushed to DockerHub, the course outline in the Learning MFE works!

      • Unfortunately, courseware in the Learning MFE shows “An error has occurred” with no other explanation. No JS error logs in the console or 5XXs/4XXs in the network log.

        • Workaround: Toggle on courseware.use_legacy_frontendwaffle flag in order to use legacy courseware 😞

        • 🔔 TODO: figure out why this didn’t work

      • Now, using legacy, courseware works.

References

1: Old pods aren't getting destroyed

$  tutor k8s start mfe
kubectl get namespaces openedx
NAME      STATUS   AGE
openedx   Active   12h
Namespace already exists: skipping creation.
kubectl apply --kustomize /home/kyle/.local/share/tutor-nightly/env --selector app.kubernetes.io/name=mfe
The Deployment "mfe" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"openedx-KXV0satwwOtuGRz7lwgslwU7", "app.kubernetes.io/managed-by":"tutor", "app.kubernetes.io/name":"mfe", "app.kubernetes.io/part-of":"openedx"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable
Error: Command failed with status 1: kubectl apply --kustomize /home/kyle/.local/share/tutor-nightly/env --selector app.kubernetes.io/name=mfe
$ tutor k8s stop
kubectl delete --namespace openedx --selector=app.kubernetes.io/instance=openedx-KXV0satwwOtuGRz7lwgslwU7,app.kubernetes.io/component!=loadbalancer deployments,services,configmaps,jobs
service "cms" deleted
service "elasticsearch" deleted
service "lms" deleted
service "mfe" deleted
service "mongodb" deleted
service "mysql" deleted
service "redis" deleted
service "smtp" deleted
configmap "caddy-config-b9gk6kf847" deleted
configmap "openedx-config-kkhkt28hth" deleted
configmap "openedx-settings-cms-bkft5k2b4h" deleted
configmap "openedx-settings-lms-h25dtdkdh2" deleted
configmap "redis-config-fccm65mh4m" deleted
$ kubectl --namespace=openedx get pods
NAME                            READY   STATUS             RESTARTS      AGE
caddy-85589d6669-dnf65          1/1     Running            1 (11m ago)   52m
cms-7dd67c5f55-b4rvb            1/1     Running            1 (11m ago)   52m
cms-job-20220303113051-lrzh5    0/1     Completed          0             51m
cms-worker-67846c7d7b-t2djr     1/1     Running            1 (11m ago)   52m
elasticsearch-d8b6859f7-7rfmr   1/1     Running            1 (11m ago)   52m
lms-788dfd9669-spgdw            1/1     Running            1 (11m ago)   52m
lms-job-20220303113217-gwj82    0/1     Completed          0             50m
lms-worker-794866f475-zhzzb     1/1     Running            1 (11m ago)   52m
mfe-7757f78d77-qcpmr            0/1     ImagePullBackOff   0             49m
mongodb-f4f6dc446-jn7ck         1/1     Running            1 (11m ago)   52m
mysql-7cb5f98d7-tlf76           1/1     Running            1 (11m ago)   52m
redis-7ddff4c6dd-hcxb6          1/1     Running            1 (11m ago)   52m
smtp-7454bf587b-6cl9l           1/1     Running            1 (11m ago)   52m
$ tutor k8s stop
kubectl delete --namespace openedx --selector=app.kubernetes.io/instance=openedx-KXV0satwwOtuGRz7lwgslwU7,app.kubernetes.io/component!=loadbalancer deployments,services,configmaps,jobs
No resources found
$
  • No labels