Charter

It’s become clear that there is an acute need to support new Tutor users and plugin developers. It’s also clear that Tutor maintainers benefit from hearing feedback from people who are using the tool. 

Please join our users’ group for sharing help, tips, and feedback on using Tutor and developing Tutor plugins. Bring your questions, and get ready for some live debugging! Expert or newbie, all are welcome.

Announcements will be posted to Slack in #tutor-maintenance

Recordings

https://drive.google.com/drive/folders/1jHyN3DMxwfNEgPxiUJOJS3Q3mb0Z0uMx

Agenda and Notes

We meet every other week. Our schedule and video conferencing link is on the Working Group Calendar. The meeting is usually hosted by @Kyle McCormick .

You can add topics below ahead of time, or add them during the meeting.

2024-07-29

• Moises: https://docs.docker.com/reference/dockerfile/#run---mounttypessh I wanted to use the RUN --mount=type=ssh instruction to install private packages. This command needs an additional flag to the build command that I can add via the DOCKER_BUILD_COMMAND filter. The problem is that the --ssh=default=$SSH_AUTH_SOCK flag gets escaped by shlex. As a workaround is possible to use os.environ.get("SSH_AUTH_SOCK") but some people may not want to print those values to the stdout.

2024-07-15

• Glib: Tutor Plugins -

  • Discussing with marketing wg the idea of having a marketplace catalog

    • Making plugins available for operators

    • Particularly plugins with one-click installation process

    • ^ … that’s tutor plugins!

      • particularly plugins in the tutor index

    • Question: Many plugins require post-actiosn

      • Rebuild image

      • Run init

      • Configure something

    • Idea: a flag that runs all those necessary extra steps automatically, to enable the one-click installation

    • Idea: always rebuild and run init

      • Max mentioned that he wouldn’t want Tutor to automatically run any actions, since many operators have customizations that make it so they wouldn’t want to do every step necessarily

      • in other words – some operators will want to be able to run steps manually, others will want it all to happen automagically

    • Dave: Could Tutor have a GUI interface ?

      • Dawoud: This has been discussed, there is interest

        • there is the webui plugin - https://github.com/overhangio/tutor-webui

        • but it’s the same CLI, served for a web browser to access

    • Kyle: What is the UI for the marketplace to be?

      • Glib: First phase, just a catalog

        • Description, installation instructions

        • Manually maintained

      • Glib: Integration into tutor would be a second phase

    • Dawoud: We are already re-building and re-initting when we run tutor ... launch, so not sure if we want a flag on the plugin command

      • Ghassan: we also have tutor ... do init --limit=pluginname

      • Glib: We’re thinking about command that must be run after plugin is installed and enabled

        • Example: Jupyter plugin

      • Idea:

        • tutor plugins install jupyter --enable --rebuild --run-init --configure JUPYTER_VERSION=5

      • Glib: Currently it requires several commands

        • tutor plugins install blah tutor plugins enable blah tutor config save --set ... tutor images rebuild openedx tutor local do init --limit=blah

      • One option:

        • tutor plugins install blah && \ tutor plugins enable blah && \ tutor config save --set ... && \ tutor images rebuild openedx && \ tutor local do init --limit=blah

      • A compromise:

    • Moises: If launch had a flag to rebuild, would that cut it?

    • Dawoud: install can take multiple plugins

    • Kyle/Glib: --enable would enable all provided commands

    • Glib: When you’re installing a plugin from outside the indexes, it’s pip install -e ., right?

      • `tutor plugins install ./path/to/mylocal/plugin`

        • this works

      • $(tutor plugins printroot)/myplugin

        • putting plugins here will install them automatically

    • Three changes, Kyle will make issues, looking for volunteers to implement

      1. An --enable flag on tutor plugins install

      2. A --configure flag on tutor plugins install , just like tutor config save --set ...

      3. A --limit-init flag on tutor ... launch

• Moises: Build time for installing Python packages:

  • Glib: Brian M says he was experimenting with pip-tools replacement in rust, but hit an issue where it doesn’t support egg files

  • Dawoud:

• Michael: being able to limit init for launch might solve another problem: Shared mysql server, don’t want to provide admin user or password, so mysql init fails

  •  

2024-06-03

• MFE build time (below)

• Configuring open edx services with tutor

  • Moises and Maksim work on large instances group

  • Some of tutor’s default were meant to simplify setup

  • For large instances, they need to undo tutor’s defaults

  • example: celery queues

    • tutor uses one celery queue

    • at scale, the queue can fill up

      • using patches, large instances has been undoing tutor’s configuration changes

  • edx-platform is complex to follow, with all the indirection

    • related: OEP-45 simplification

  • could be better to have a blank slate

  • edunext had to replicate ansible installation settings, using tutor

  • they are using settings patches

    • yaml settings patch

    • common settings

    • production patches

    • some files don’t have patches

• Having plugins expose configuration interfaces

  • tutor-contrib-pod-autoscaling

  • rather than add more config settings, the plugin

•  

2024-05-20

• MFE build time

  • 1hr+ to build MFEs

  • Number of MFEs has not changed since Quince – 10 ( )

  • Custom npm registries

    • Local deployment of an npm registry

      • vs. shared npm registry

    • devstack had devpi for python reqs

      • devpi is pretty easy to set up

    • install vs build - which is the problem?

      • both

    • building all MFEs (except Learner Dash) without cache on AWS spot fleet m5.xlarge

      • parallelism=4

      • Duration: 43 minutes 4 seconds

  • Max number of simultaneous connections to npm registry by IP address

• why does the network connection break when we have too much parallelism in the build?

  • this seems different than the npm connection limitation. seems that the host’s network connection is failing

    • saturation of bandwidth?

  • to build mfes in sandbox, had to explicitly set max_parallelism=2

• Solutions

  • Sharing dependencies across MFEs

    • Hinted at in

    • @Kyle McCormick

  • “No New MFEs”

    • @Kyle McCormick

  • Remote registry

    • Axim-provided registry? network of registries, different providers in different regions

    • @Kyle McCormick

  • Local registry

    • This can definitely have multiple versions

    • @Maksim Sokolskiy

  • Local cache. Two potential issues:

    • 1. ensure that it’s actually caching anything at all:

    • 2. version conflicts across MFEs may invalidate a shared cache

    • @Dawoud Sheraz

• Other solutions (not currently following up on)

  • Pre-built image with some deps

    • Max has tried this

    • Install one mfe’s deps using previous mfe as a base

    • Used pnpm – can handle multiple different versions of package

      • pnpm didn’t work out of the box

    • pnpm , workspace for each mfe

2024-05-06

…

2024-04-22

Present: several people

2024-04-08

Meeting link: https://meet.google.com/rhs-veey-rrp

Present: @Chintan Joshi , @Danyal Faheem, @Dave Ormsbee (Axim) , @Dawoud Sheraz , @Deimer Morales ,@Maksim Sokolski, @Maria Grimaldi, Qasim Gulzar , @Régis Behmo

• Integrate Tutor with IntelliJ IDEA/Pycharm (by Qasim Gulzar@edly)

  • Recording:

  • The tutorial instructions will be published soon-ish in a dedicated space.

  • In the meantime, we can reproduce the instructions with the video recording.

  • Instructions are still hackish and a little brittle, we can definitely work to improve them.

• How to track the eclipse with Tutor (by @Régis Behmo if time permits) (it did not)

Notes:

• MongoDb 7 upgrade: scheduled for Redwood

• utf8mb3 upgrade. We reached the following decisions for Redwood:

  • upgrade the connection string and the database default charset to utf8mb4 for all platforms

  • For existing platforms, upgrade the default database charset (with ALTER DATABASE databasename CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;).

  • For existing platforms, convert tables in a conservative manner, using instructions similar to:

    ALTER TABLE tablename CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;. The list of candidate tables is available here.

  • Implementation of these decisions can be followed in this GitHub issue:

• MySQL high memory issue on Arch Linux:

  • It was suggested that Arch users manually create a docker-compose.override.yml file.

  • We should update the troubleshooting instructions for Arch Linux users. @Deimer Morales can you please open a pull request to update the Tutor docs?

2024-03-25

• Auto-mounted directories

  • Can we add mount keywords in our own plugin? → yes, or open a PR to upstream plugin

  • Do we list the keywords anywhere? example, platform-plugin-

    • The exact list is not documented anywhere

• MFE mounts

  • Tried to mount custom mfe – couldn’t do it

    • used command line mount

    • did not see the mfe on the port

    • seems like folder was mounted, but MFE filter wasn’t added to, so the tutor-mfe plugin didn’t know to build/run the mfe

    • How Zach did it:

• Let’s talk about MFE image building… why we (usually) don’t need to re-build, and figuring out this Docker build cache madness. (@Régis Behmo )

  • Theoretically, rebuild should only happen if (1) we change an MFE and (2) we’re pushing it to production

  • Errors when installing node_modules

    • npm error, socket reset, registry hangup

  • Docker build cache size needs to be set large enough so that it doesn’t get garbage collected between mfe builds

    • need to change a config item in $HOME/.docker/daemon.json

    • If you think you rebuild too often, try upping docker build cache max size

• Talked about - kyle will continue to work on

• Production vs dev flow in docs - docs nudge you towards production flow

• Remote image building - anyone?

  • edunext has - they build tutor-nightly images on an ec2 instance

    • 20-30 minutes for uncustomized mfe image

  • custom multiarch builder - regis builds macos images even though he doesn’t have a mac

Older Notes

Paste old notes here.