Large Instances Meeting Notes 2023-10-31 🎃

Owned by Braden MacDonald
Last updated: Nov 01, 2023 by Xavier Antoviaque

Video recording: https://drive.google.com/file/d/1DotHV8haNSbueA4S1pipGz1Rl6IQ06I2/view?usp=sharing

Transcript: https://drive.google.com/file/d/1B-9hjA_pp0BTl7hr-ItFbsrw_VPY4nPO/view?usp=sharing

Summary (automated)

Overview

The meeting was attended by representatives from various organizations. Updates were shared from each organization, including plans for experimenting with Kubernetes-based development environments, transferring deployments, and progress on Edx Plot and sandboxes. The team also discussed the implementation of a sandbox feature for a GitHub request and evaluated the status and issues of their shared projects.

Next Steps

  • Emad to review the documentation on shared elastic search or open search.

  • Johnny to work on the monitoring services for the Harmony Project.

Full Recap

Introduction and Updates Meeting

Braden, Johnny, Emad, Gábor, Max, and Jeremy introduced themselves in the meeting. Emad introduced himself as a software engineer from Edisbridge and Jeremy from the same organization. Gábor, a developer from Opencraft, also introduced himself. The meeting started with updates from each organization and was to be started by Jeremy.

Kubernetes Experimentation and Deployment Plans

Jeremy mentioned that their farm is starting to experiment with Kubernetes-based development environments and is also working on improving container raising for their production uses. Max shared that they are planning to transfer all their deployments to a new installation and may contribute to tutor enhancement. Max also mentioned that they are considering investigating current solutions from OpenCraft to work with Kubernetes. Emad shared that they have been using Kubernetes since launch and were the ones who introduced AKS support to tutor. They are also preparing for the upcoming release of Kubernetes 1.4.

Edx Plot Progress and Authentication Issues

Gábor discussed the progress and challenges encountered in the completion of the Edx Plot and sandboxes for the platform. He highlighted the complexity of the process and mentioned that the past two weeks were largely spent on this task. Gábor also mentioned an issue with the authentication of the Doctor Biad, which was running into rate limits due to the lack of proper authentication for API calls. He stated that there was a plan to address this issue and it would require further discussion with Regis and Bit adorable.

Sandbox Feature Implementation for GitHub Request

Gábor, Braden, Max, and jhony discussed the implementation of a sandbox feature for a GitHub request. They discussed the potential issues of hitting rate limits and the solution to use open-source resources. Gábor explained that each instance has its own Lms, Cms, and workers, and Max asked about the possibility of resource duplication. They discussed the shared resources and database servers, and Braden suggested the use of an empty database to speed up the first deployment. The team also addressed the problem of long provisioning times and plans to resolve it. Emad briefly participated in the discussion, but it's unclear what his specific input was.

Project Status and Issues Evaluation

Gábor, Braden, Emad, Max, and Johnny evaluated the status and issues of their shared projects. They discussed the use of helm charts for the Harmony Project and a lack of configuration in the Twitter Project. Emad asked about documentation on shared elastic or open search, which Braden confirmed was being worked on. Johnny provided updates on testing with Coj and a memory leak issue they were investigating. The team also discussed scaling different services and considered implementing auto scaling. Gábor questioned the issue with the Code jh service, which Johnny explained was consuming a lot of resources and needed to be restarted. Johnny also mentioned plans to write a post on implementing auto scaling for different services. Gábor also reviewed the code for the Harmony Project's monitoring services and needed to verify a detail before approval. Braden expressed optimism for increased activity on Harmony soon, noting Opencraft's work on team coordination for a migration to Harmony. The Front End Plugability Summit was also discussed, with Braden expressing optimism for progress resulting from the meeting. The team agreed to reconvene later.

Agenda

  1. Assign meeting lead and note taker.

  2. Greetings & introductions as needed.

Welcome to Emad Rad, the new attendant to the meeting, a software engineer from edSPIRIT.

Updates from each org on the call - 2U, eduNEXT, OpenCraft, Raccoon Gang. What's new with your deployment(s)?

2U:

  • Testing devspace as a development environment for Kubernetes.

  • Making progress on containerizing all of our apps

  • Working on extracting Codejail to an external service.

Raccoon Gang:

  • Migration of all operations to K8S Tutor-based installations

  • Getting more involved in the Tutor project to contribute to its Core/plugins.

edSPIRIT

  • Focus on improving operations on EKS OpenedX installations

  • Migrating the platform from Olive to Palm.

OpenCraft

  • The creation of Sandbox installations when creating PRs in edx-platform and MFE’s repos is available (for specific users).

  • There’s an open issue related to a rate-limit problem when building MFE images with unauthenticated users.

Edunext

  • Experiencing some leak memory/CPU issues when running the codejail service at a large scale. Trying to figure out the root cause of such behavior.

  • Will open a discussion in relation to the right way to add HPA to services in OpenedX.

Harmony project updates: Review the list of PRs and issues, and assign anything un-assigned.

@Gábor Boros is checking the Karpenter PR

Open discussion/questions, if any.

Chat log

00:04:02 Braden MacDonald: https://openedx.atlassian.net/wiki/spaces/COMM/pages/edit-v2/3910926346
00:14:25 Gábor Boros:
00:21:29 Braden MacDonald: includes the helm chart and tutor plugin
00:21:30 Max: