Large Instances Meeting Notes 2024-01-23

Assign meeting lead and note taker.

Greetings & introductions as needed.

Transcript & summary (automated, via Otter)

Updates from each org on the call

What's new with your deployment(s)?

eduNEXT - working on Harmony project.

@Felipe Montoya and @Moisés González were contacted by Tim from 2U to see how their codejail implementation could align with eduNEXT’s. eduNEXT helped 2U solve their issue, and now they’re using a similar approach but not the same code. e.g. eduNEXT uses Flask but 2U is using Django. Now that this is unblocked, they may be containerizing LMS/CMS soon at 2U.

Deployed Aspects and have it connected to an Olive installation, and it seems to be running well. This is a good sign for use of Aspects in production in the future.

Q from @Jhony Avella : did you perform load testing on codejail? A: not sure if 2U did or not. Can ask them once the dust settles from recent round of layoffs.

Racoon Gang - Our team will be looking at issues and PRs on the Harmony project. In particular, interested in the Velero PR - Add optional Velero chart by gabor-boros · Pull Request #58 · openedx/openedx-k8s-harmony .

Re Aspects: We’ve integrated Aspects into one of our client installation and will be testing it on stage and hopefully soon in prod. So far everything seems great. We’ll monitor performance and operation. One strange thing: in the LMS we have an xAPI forwarder and there is another piece that pulls logs from k8s pods, but I can’t understand what this is doing. It seems to scrape the stdout logs from all containers but not be used anywhere. Wondering if this is from legacy architecture?

Investigating speeding up MFE build using Bun or pnpm instead of npm, and having a base build with light MFE overrides rather than rebuilding the whole thing.

Discussion: @Braden MacDonald mentioned on a previous call that you can build an MFE using random values for the config and then make it use ‘sed’ to replace those values in the compiled .js bundle with environment values at container startup. @Moisés González shared that they used this approach a bit in GitHub - eduNEXT/tutor-contrib-mfe-extensions: Experimental Tutor Plugin for CDN support and extras .

OpenCraft - @Braden MacDonald - just continuing slowly to move pieces of internal “Grove” hosting management tool over to Harmony.

Harmony project updates: Review list of PRs and issues, and assign anything un-assigned.

Open discussion/questions, if any.

ElasticSearch/OpenSearch - it would be good to investigate this and standardize on one or the other for Harmony, so we don’t have to do twice as much work. @Braden MacDonald from a dev perspective, the API is largely compatible so the edx-search/platform code should work with one or the other. So it’s more of an operations question. @Moisés González points out that they may differ in terms of multitenancy support, and that some orgs have a license/legal requirement to use OpenSearch only. Maybe we can continue to support ES for now since it’s so common, but if edx-search is changed to only support OpenSearch or to use some OpenSearch APIs, we’ll drop support for ES in Harmony and/or make OpenSearch the default. TODO: need to research this more.

Implementing Redis Cluster - @Maksim Sokolskiy wonders if anyone has experience using AWS ElastiCache Redis as a cluster (in “serverless” mode). (Using it as a cluster requires changes to the configuration and driver used in edx-platform.) So far if they switch from hosted Redis to AWS cluster, it simply doesn’t work. They’re seeing the error “redis.exceptions.ResponseError: CROSSSLOT Keys in request don't hash to the same slot”. Does anyone have experience with this?

@Moisés González is also wondering if anyone has experience with Redis filling up and causing errors.