Large Instance Meeting notes - 2023-04-04

Video recording

https://drive.google.com/file/d/1HINaCdMaLUiTIQuh4KlHhcEJ3uAvKHDu/view?usp=share_link

Introductions

• Diana - DevOps engineer, been in 2U for a while, interested in spinning up dev environment, and curious how this ties with kubernetes - not interested in the deployment

• Xavier - OpenCraft founder, interested in ensuring we collaborate on instances deployments, instead of each reinventing the wheel in our corner

• Jeremy, manager at 2U, using kubernetes for dev environment, keeping things coordinated

• Glenn, with adjuster pro, online insurer training, new to the community, want to see what the community is up to to support there

• Thomas also with adjuster pro, devops engineer, learn how to setup things on Kubernetes

• Maksim, CTO Raccoon Gang, developing their own kubernetes solution, and see what the community is doing and help

• Braden, CTO OpenCraft, co-chair with Felipe of the working group and the Harmony project

• Gabor, devops OpenCraft, to contribute to the project and align OpenCraft with the project

Harmony project update

Braden: many of us are deploying Kubernetes, at OpenCraft we are moving there, have some instances there. Because there was no standard everyone was .Harmony allows to setup multiple instances of Open edX Kubernetes cluster, also to scale up. New, not production ready yet, but trying to get there as far as we can

Jhony gave a talk at the conference about deploying on Kubernetes.

Pull requests reviews

• Monitoring: in review

• Implementation of Karpenter by Lawrence: the code is ready and just have to update the commit messages

• Renaming of the project to Harmony: PR already approved by braden, ready to go

Grove?

Thomas: Grove and Harmony - differences?

Braden: Grove is mostly used by OpenCraft, built around gitlab CI. It is oriented towards our needs. In the process of making it compatible with Harmony, to use Grove for the deployment.

Xavier: open to being used and discussed here!

Glenn: interested in using it. Liked grove's approach.

Braden: We discussed at the conference

Xavier: will be happy to get feedback, questions and contributions

Issues

• Autoscaling: PRs in progress

• Karpenter: have the PR from Lawrence

• Cloud-based dev environment: Diana, currently on the side; Jeremy: will be piecing a plan for this soon. Devops toolkit.

• Renaming the project: almost done, PR in progress

• Getting ready for the conference: Google doc listing what projects people want, brainstorming from the community. Nothing in terms of the current project, but nice devops initiatives

• Deploying opensearch: pending currently, unassigned - Maksim volunteered to take it on!

• Add the plugin to the contrib plugin index: waiting on having the prod version ready

• Add verified ssl support: currently unassigned, not particularly urgent

• Adding permission to team members: gabor and jhony being made core contributors, Maksim will ask for right extensions

Steps to get to production ready

• edunext and opencraft have both looked at the list, and some of the steps are done. Will probably be decide to make it supported from Palm. Make an official release of the helm charts.

• Gabor: agree with what Moises has written, with some additional items that are added (node affinity, volume attachments, periodic/cron jobs that tutor isn't supporting but necessary for kubernetes, some level of monitoring tools, chart testing/support, readme doc, upgrading with multiple Open edX installations)

  • Braden: need to match the Open edX version to tutor's version, not in scope

  • Glenn: edunext had a lightning talk about a version manager, could that be incorporated?

  • Braden: Not prescribing how the images should be created.

  • Xavier: why not agreeing on a standard for image?

  • Braden: low hanging fruit, same common ground for MVP

  • Xavier: sounds good! Then in that spirit, how can we limit things to the minimum for the MVP?

  • Gabor: from my comments, only cron job and monitoring are really needed for that step

  • Maksim: About the tutor plugin - what do we do about having support for multiple versions?

  • Braden: We might have to support multiple versions. We are not planning to support before palm, but yes we will have to be able to handle it

Supporting 2U's needs

Follow-up on Adam's discussions, it looks unlikely that Harmony will be used by 2U

• Xavier: what is missing for 2U to use Harmony?

• Jeremy: we have some helm charts internally, but next step would be to open source them. Held up by legal, and by the availability of the devops team, to reconcile the internal work with the harmony project and the rest of the community

• Braden: those are more for other services, not open edx right?

• Jeremy: 2U starting to work on getting edx platform and studio supported, but no bandwidth to work with the community on it yet

• Xavier: Is working with the rest of the community on this the plan on the long term? Any hope to reconcile the differences?

• Jeremy: yes, but tricky

• Diana: 2U had a big incident where some of the old infra code caused issues, need to get better tooling around infrastructure - but not being part of the SRE team has limited visibility on the plans

• Jeremy: Project to switch to Kubernetes has momentum within 2U, switched a few services

• Jeremy: Adam is trying to join more regularly - will try to ask for a more regular representation of 2U in this working group

Testing and adoption of Harmony

• Glenn: Was useful to see how it aligns with the other providers initiatives. Knowing the approach for defining the MVP, who the actors of the project are. 

• Thomas: trying to dive and start contributing - in the "how does this work" stage

• Braden & Diana: glad that you are here! Ask questions!

• Thomas: Any documentation?

• Xavier: should we do a ticket? assign it?

• Braden: sure, I'll take it

Chat log