Large Instances Meeting Notes 2024-08-06
Assign meeting lead and note taker.
Greetings & introductions as needed.
We welcomed @Moncef Abboud and Christian as new members.
Updates from each org on the call - 2U, eduNEXT, OpenCraft, Raccoon Gang. What's new with your deployment(s)?
eduNext
Preparing the open source version of K6 testing suite – hopefully done by the next meeting
Last meeting: Celery and Sentry plugins for Tutor
OpenCraft
OpenCraft is now using almost all parts of Harmony in production. The deployment of the shared ElasticSearch cluster has been a huge win, saving a lot of RAM because each instance was using 3GB RAM for elasticsearch which is now saved by the efficiency of the shared cluster.
Harmony project updates: Review list of PRs and issues, and assign anything un-assigned.
Recent updates were merged to support Redwood chore: bump tutor plugin dependency version for redwood support by tecoholic · Pull Request #75 · openedx/openedx-k8s-harmony and fix issues with shared ElasticSearch fix: ensure ES/OS cluster setup for shared usage by gabor-boros · Pull Request #73 · openedx/openedx-k8s-harmony
@Gábor Boros recently created Create cleanup policy for OpenSearch and/or Elasicsearch · Issue #66 · openedx/openedx-k8s-harmony to track an potential issue with Elasticsearch data filling up storage over time. We’ve already started to notice this issue at OpenCraft.
@Moisés González is working on Log collection and aggregation · Issue #32 · openedx/openedx-k8s-harmony
Open discussion/questions, if any.
(1) @Gábor Boros is proposing that we have Terraform scripts in Harmony, not just as an example but actually maintained, to provision the cluster itself at a cloud provider(s). This would be for the step that comes before installing Harmony - i.e. first one would use these proposed Terraform scripts to provision a K8S cluster from a cloud provider, then they’d use the Harmony Helm chart to install the shared resources for multi-tenancy onto the cluster, then they’d use Tutor to deploy individual instances onto the cluster.
OpenCraft has already developed scripts like this and would be happy to clean them up and contribute them into the project
@Jhony Avella mentioned that eduNEXT has similar Terraform scripts and requirements as well.
@Jhony Avella proposed we create an issue for this, and work together to define a plan for it.
(2) @Gábor Boros asked to clarify the name of our working group. @Braden MacDonald said “Large Instances Working Group” seems to be the name.
(3) @Jhony Avella asked about creating a list of Tutor plugins that are recommended for large instances, e.g. celery, Sentry, New Relic, CloudFront. The group said that made sense, so @Jhony Avella will create an issue for it.
(4) @Jhony Avella proposed splitting the Harmony chart into sub-charts according to their functionality, e.g. one sub-chart for nginx+cert-monitor, one sub-chart for monitoring tools, etc.
@Gábor Boros says it’s probably a good idea, but unlikely to make a big difference for OpenCraft. He linked to provider-modules/harmony/main.tf · main · opencraft / dev / Grove · GitLab which is an example of how OpenCraft uses Terraform to configure and deploy the main chart with configuration variables to enable/disable various features.