Large Instances Meeting Notes 2024-06-25

Assign meeting lead and note taker. @Felipe Montoya and @Braden MacDonald respectively.

Greetings & introductions as needed.

Updates from each org on the call

What's new with your deployment(s)?

eduNEXT - @Felipe Montoya : interested in discussing Reaching 50k concurrent users on Open edX with Oracle Cloud with stakeholders at the conference to better understand the numbers.

@Moisés González : Exploring using ingress-nginx for serving static assets in on-premises (non-cloud) installations, because it can cache some files in memory and reduce traffic to the LMS pods (e.g. uwsgi which normally serves static assets). It seems to be performing well, reducing load on the LMS pods, though not as effectively as using CloudFront. If this continues to seem promising, we should investigate adding it to Harmony.

@Jhony Avella In the last meeting, I mentioned 502 errors we were seeing. However, we stopped seeing the issue. At a future hackathon, we are planning to measure how various configuration policies (number of workers, pod size, etc) affect the instances and the error rate.

We have a client that doesn’t allow pods to run with the root user, but Tutor currently requires this for some pods like minio, caddy, and tutor-mfe. Planning to mention this to the tutor users group.

Regarding the celery workers: when using just one queue for LMS/CMS services, it’s very difficult to predict the workload when there’s a mix of long-running and short-running tasks. We want to separate the async load into different queues. We’re still testing this, but it’s looking promising.

OpenCraft - no major updates

Harmony project updates

Review list of PRs and issues, and assign anything un-assigned.

@Maksim Sokolskiy finished testing the Elasticsearch PR - thanks! Though we have a dependency on another PR for tutor-forum, so we’ll try to nudge the maintainer to get that dependency merged.

@Moisés González : For logging with Aspects, we use a helm chart to deploy Vector as a demonset, which uses a privileged container to capture and transform logs. The other approach is to use a sidecar container which pushes the logs to Vector. We’ve been using the demonset approach, where you can configure which pods (e.g. only ones with a specific annotation), and then it transforms the logs into a format that is compatible with aspects. Considering making a tutor and/or Harmony plugin for this approach.

@Moisés González : We should update all the dependencies in our Helm chart to make sure we’re up to date for security patches, etc. Also, one big helm chart is getting messy. Maybe we can split it up into several separate helm charts that operators can pick and choose.

Open discussion/questions, if any.

@Braden MacDonald : Do you guys at eduNEXT use edx-enterprise? @Felipe Montoya : Yes, but for one customer that has a dedicated team. So they forked enterprise and they maintain it, but outside of that team we don’t have any experience with edx-enterprise. We then discussed having a simpler version of enterprise and/or replacing its current platform integration with one based on hooks/filters.