Large Instances Meeting Notes 2023-11-14

Owned by Braden MacDonald
Last updated: Nov 14, 2023 by Xavier Antoviaque
2 min read

Recording

Video: https://drive.google.com/file/d/1oeYR0P5oM_asoPySbqH9Z05CCYiAnsXN/view?usp=drive_link
Transcript: https://drive.google.com/file/d/1egFt13BDut9ikVL8yGRd4L3IdTn-EWN_/view?usp=drive_link
Chat log: https://drive.google.com/file/d/1rsb5xkHKKjINFzsZoju_wWu_9lkq-AW9/view?usp=drive_link

Agenda

  1. Assign meeting lead and note taker.

  2. Greetings & introductions as needed.

  3. Updates from each org on the call - 2U, eduNEXT, OpenCraft, Raccoon Gang. What's new with your deployment(s)?

    1. @Gábor Boros / OpenCraft - was on sick leave so not much progress, but did approve @Jhony Avella 's PR. Did run into an unrelated issue though:

    2. @Moisés González / eduNEXT - Last week the codejail issues were mentioned. I’ve been exploring that and noticed the current config doesn’t terminate the python process after a certain amount of time. So if I have an expensive python instruction, it will linger and eventually consume the whole CPU; eventually the whole CPU on the pod until the pod becomes unresponsive and the process eventually dies. Found that in ansible there is a 3s timeout. Secondly, and HPA issue:

    3. @Jeremy Bowman (Deactivated) / 2U - still working on MySQL 8 upgrade, having issues with the size of some edx-platform tables like CSM (4.5 TB).

  4. Harmony project updates: Review list of PRs and issues, and assign anything un-assigned.

    1. https://github.com/openedx/openedx-k8s-harmony/pull/41

    2. New issue:

  5. Open discussion/questions, if any.

    1. Q: What should we do next for Harmony?

      1. @Moisés González One issue is that initially it was focused on hosting multiple instances on a single cluster. But in practice, we find more common is a single installation on a single cluster.

      2. @Felipe Montoya Even if we focus on one instance per cluster, there’s a lot we can do to make it easy to set up autoscaling, databases, message bus, Aspects, etc. The easier we can make that the better, regardless of whether it’s multi instance or single instance. We’re sharing our collective knowledge.

      3. @Felipe Montoya We were doing multi-tenancy at the application layer for so long, and were excited about using Harmony as a way of moving the multi-tenancy at a lower level (k8s). But we’re limited by customer requests and customer requirements mean we need dedicated clusters for now.

      4. @Jeremy Bowman (Deactivated) What about virtual clusters? Would it help if they seem like separate clusters but aren’t? @Braden MacDonald Often the requirement is that each customer wants the resources on their own AWS account for example, which wouldn’t let us do that.

      5. Conclusion: @Felipe Montoya will open an issue so we can collect feedback around use cases and decide to adjust the project description/goals.

    2. @Felipe Montoya Is anyone else interested in hosting Aspects using Harmony in the near term?

      1. @Braden MacDonald Not a priority for us now, compared to converting our hosting platform to be Harmony-based. But definitely interested in it later on, and we’re involved in its development.

      2. @Jeremy Bowman (Deactivated) Not hearing much about it at 2U, but we are interested in replacing Insights at some point.