Large Instances Meeting Notes 2024-08-06

Assign meeting lead and note taker.

Greetings & introductions as needed.

• We welcomed @Moncef Abboud and Christian as new members.

Updates from each org on the call - 2U, eduNEXT, OpenCraft, Raccoon Gang. What's new with your deployment(s)?

eduNext

• Preparing the open source version of K6 testing suite – hopefully done by the next meeting

• Last meeting: Celery and Sentry plugins for Tutor

OpenCraft

• OpenCraft is now using almost all parts of Harmony in production. The deployment of the shared ElasticSearch cluster has been a huge win, saving a lot of RAM because each instance was using 3GB RAM for elasticsearch which is now saved by the efficiency of the shared cluster.

Harmony project updates: Review list of PRs and issues, and assign anything un-assigned.

• Recent updates were merged to support Redwood https://github.com/openedx/openedx-k8s-harmony/pull/75 and fix issues with shared ElasticSearch https://github.com/openedx/openedx-k8s-harmony/pull/73

• @Gábor Boros recently created https://github.com/openedx/openedx-k8s-harmony/issues/66 to track an potential issue with Elasticsearch data filling up storage over time. We’ve already started to notice this issue at OpenCraft.

• @Moisés González is working on

Open discussion/questions, if any.

(1) @Gábor Boros is proposing that we have Terraform scripts in Harmony, not just as an example but actually maintained, to provision the cluster itself at a cloud provider(s). This would be for the step that comes before installing Harmony - i.e. first one would use these proposed Terraform scripts to provision a K8S cluster from a cloud provider, then they’d use the Harmony Helm chart to install the shared resources for multi-tenancy onto the cluster, then they’d use Tutor to deploy individual instances onto the cluster.

• OpenCraft has already developed scripts like this and would be happy to clean them up and contribute them into the project

• @Jhony Avella mentioned that eduNEXT has similar Terraform scripts and requirements as well.

• @Jhony Avella proposed we create an issue for this, and work together to define a plan for it.

(2) @Gábor Boros asked to clarify the name of our working group. @Braden MacDonald said “Large Instances Working Group” seems to be the name.

(3) @Jhony Avella asked about creating a list of Tutor plugins that are recommended for large instances, e.g. celery, Sentry, New Relic, CloudFront. The group said that made sense, so @Jhony Avella will create an issue for it.

(4) @Jhony Avella proposed splitting the Harmony chart into sub-charts according to their functionality, e.g. one sub-chart for nginx+cert-monitor, one sub-chart for monitoring tools, etc.

• @Gábor Boros says it’s probably a good idea, but unlikely to make a big difference for OpenCraft. He linked to which is an example of how OpenCraft uses Terraform to configure and deploy the main chart with configuration variables to enable/disable various features.