/
Core Arch Working Group

Core Arch Working Group

A weekly meeting to discuss the code and data which underpins the platform’s teaching & learning capabilities, centered around the openedx-core repository.

Resources:

Current projects:

  • CBE

  • Pathways

  • MongoDB deprecation

Past projects:

Agendas and notes:

Aug 11, 2026

  • [Jesper] How can we avoid or handle deadlocks in competency status evaluation?

    • Notes on kyle’s understanding

      • don’t build new signal handler

      • there is an existing signal handler which invokes a celery task, subsection_grade_changed_v.. (need exact name)

        • we will add to this signal handler so that, in the same atomic transaction, we’re checking if a competencycriteria needs updating, and updating it if so

    • why the deadlock?

      • we might have multiple grade changes happening simultaneously

      • there is a potential race condition where two grade updates happen to nodes in the same criteria tree

      • to avoid the race condition, we’d put a lock on the group nodes

      • worried about deadlock given ^ . cases where it could deadlock:

        • tbd

      • in the case of deadlock, we’d expect that the celery task would timeout and then retry

      • what about locking on a user ?

    • A subsection grade change involves a re-evaluation of the student competency status tree. ADR 0004 (proposed) allows for concurrency while adding a lock to a group node (non-leaf tree node). Deadlocks can a) involve a grade change locking one group and
      b) involve a grade change locking multiple groups, which happens when one subsection manifests as multiple leaves in the same tree.

    • Avoiding single-leaf deadlocks?

    • Avoiding deadlocks on grade updates that touch multiple leaves (if that’s possible)?

    • if we’re locking a competency group, how long does that lock last? → For the whole transaction, hence the risk of deadlock.

    • when might be concerned about deadlocks causing performance issues? exams

    • (much more discussion …. )

    •  

  • [kyle] Is there need for the Usage table for any of the CBE features in Willow?

    • the CompetencyCriteria table

      • will point to the ObjectTag

        • will have the UsageKey

          • Braden and Kyle agree about wanting to change this to Usage, but it is not blocking CBE in Willow

      • will point at the CourseRun (not CourseKey)

    • sounds like we may not need the Usage table for Willow

  • [kyle] MongoDB-free static assets - how does it dovetail withhttps://github.com/openedx/openedx-core/issues/499 ?

    • yes it relates

Attendees: Kyle McCormick, Braden MacDonald, Jesper Hodge, Mary Gwozdz (Dave Ormsbee on PTO)

Notes generated with Claude by combining human notes, transcript, and chat log

1. Deadlocks in competency status evaluation (Jesper)

Correcting the framing

Kyle's summary was close but assumed a new signal handler. In fact, per Dave's review comment on the ADR, no new handler is being built: an existing Django signal already invokes a celery task that updates the subsection grade, and the competency status re-evaluation will happen inside that same task, in the same atomic transaction, synchronously — no secondary async hop.

The problem

A subsection grade change triggers re-evaluation of the learner's competency status tree. ADR 0004 (proposed) allows these tasks to run concurrently and takes a lock on group (non-leaf) nodes. Kyle's worked example clarified why the lock exists: if two criteria under the same parent flip to demonstrated at nearly the same moment, each child's task reads stale sibling state and can leave the parent marked not-demonstrated (or partially demonstrated) when it should be demonstrated. Locking the parent makes the second writer wait its turn.

Two deadlock shapes were identified:

  • Two threads contending for the same group node — solvable by ordering.

  • A grade change that manifests as multiple leaves in one tree (multiple assignment versions), where each thread holds one group and waits for another. Braden pointed out that ordering doesn't save you here: task 1 locks A+C, task 2 locks B+C, and they still deadlock over C.

Mary confirmed from Claude in chat that a row lock is held for the entire transaction, not just the statement that took it — which is where the exposure comes from.

How bad is it?

A real deadlock would make celery error and retry; because it's one transaction, the grade rolls back automatically with no manual cleanup. Braden noted that celery-only impact is bounded — fixed worker pool, tasks just queue — and the dangerous scenario is anything that touches LMS workers, which can spike resources and take a site down. Jesper was still concerned: this is the task that evaluates subsection grades, and stalling it during timed exams (hundreds or thousands of submissions in a narrow window) is time-sensitive, plus he's seen celery task explosions cause bad downstream effects at edX. Kyle and Braden agreed rare race-condition bugs are miserable to diagnose from learner reports, so it's worth solving up front rather than as a later bugfix.

Three candidate approaches

Approach

Assessment

Approach

Assessment

Coarser per-learner (or deployment-wide) lock

Already in the ADR's rejected alternatives — overkill, less performant, and adds lock-lifecycle machinery over a huge key space. Braden floated it as a fallback if deadlocks turn out to be common.

No lock; read-after-write

Commit the group node, then re-read the children (likely needing a second transaction) and fix if anything changed. Django's READ COMMITTED on MySQL guarantees a fresh snapshot. Cascading repeats up the tree are possible but rare.

Dirty flag + delayed sweep

Leaf and dirty flag written in the subsection transaction; a delayed or scheduled task walks dirty trees. Braden noted openedx-completion-aggregator already does exactly this — leaf updated immediately, tree flagged dirty, hourly rollup, with real-time computation for individual learners so they never see stale data. Caveat: there's one course tree per learner there, but many competency trees per learner here.

Outcome

Consensus to drop locking. Mary, Braden, and Kyle all leaned toward read-after-write for its simplicity, with Braden's caveat that he can't yet fully reason through every interleaving and be certain it's correct. Kyle framed the deciding principle as preferring an approach the team understands, even at the cost of twice the celery tasks or slightly stale data, over one where a known deadlock risk isn't fully characterized.

Jesper will write both non-locking options into the ADR as alternatives so it can merge (unblocking other work), with the final pick deferred to Dave when he's back from PTO next week — Braden expects Dave to recommend against locking generally. Jesper also suggested exhaustive test cases enumerating every race-condition constellation, partly as documentation. Kyle asked for logging around whatever mechanism lands so operators can see it, plus buffer in the implementation issue to scrutinize the code. Implementation is expected around end of September.

2. Is a Usage table needed for CBE in Willow? (Kyle)

Conclusion: no.

  • CompetencyCriteria points at ObjectTag, which carries the UsageKey — already implemented, so nothing is blocked.

  • CompetencyCriteria will point at CourseRun rather than CourseKey.

  • Mary and Kyle confirmed in chat that user_id is already a proper FK to User, not a usage key.

  • Braden and Kyle both want ObjectTag to eventually hold real foreign keys to the objects it tags, but that needs courses and content in openedx-core first (learning context and openedx-content), so it's a future item.

This spun into a side discussion about ADR churn: these ADRs keep needing edits as models firm up. Options raised were putting less implementation detail in ADRs, moving class/table specifics into Confluence or a separate design doc that can be updated freely while the ADR stays the durable source of truth, or waiting until the models from #613 land and then simplifying. Left for Mary and Jesper to settle.

3. MongoDB-free static assets + external asset management (Kyle, Braden)

Braden's estimates are waiting on Chris's review and should go to Kyle same-day. WGU has separately asked about asset management as a first-class citizen in Open edX and may be able to put a couple of engineers on it; Kyle asked whether that overlaps with getting assets out of MongoDB. Braden thinks yes, and it relates to openedx-core#499.

Braden's sketch (shared on screen): a new top-level asset entity in libraries, alongside collections/units/subsections/sections. One asset may bundle multiple files — video renditions, streaming formats, subtitle tracks — kept grouped rather than scattered through the library, and referenceable from components. Longer term the same model replaces files-and-uploads in courses, including assets scoped to a single component so they don't need separate management. Crucially, if asset is a top-level entity it doesn't have to be backed by openedx-content: it could point at an external DAM with metadata pulled in.

On versioning, Braden was firm that nothing should change under anything — the asset would carry its own asset versions, with a sync process minting new version metadata when the external system publishes a new version. Open questions: how to discover and sync external assets and their updates, and whether to surface an entire external catalog or only a chosen subset. Note that Dave has previously said we don't want to reimplement a DAM inside Open edX, only support external ones.

Jesper recalls http://edX.org overriding the MongoDB-backed asset store with an S3 plugin and thinks some of this may already exist in open source (S3 configuration thread), and will dig up the actual code tomorrow rather than rely on memory. Braden believes the S3 settings don't cover the files-and-uploads page specifically, and pointed to platform PR #38950, which documents how asset storage works today — he'd like corrections folded in there. Jesper also flagged that whatever changes land must not break http://edX.org 's existing S3 overrides. Migration will be course-by-course with both systems runnable in parallel.

Action items

  • Jesper — revise ADR 0004 (PR #713): remove the locking approach, present read-after-write and dirty-flag/cleanup-job as alternatives, note that deadlocks need handling and add logging; get the PR into a shape that can merge regardless of which is chosen.

  • Kyle / Jesper — put the two options to Dave when he's back from PTO next week for a recommendation.

  • Braden — send static-asset estimates to Kyle once Chris reviews.

  • Jesper — investigate http://edX.org 's S3 asset store overrides, send code and findings to Braden and comment on PR #38950.

  • Mary / Jesper — decide on an approach to reduce ADR churn (design doc vs. genericizing vs. waiting on #613).

  • Implementation of the CBE competency-status mechanism: ~end of September.

Aug 4, 2026

  • [Mary, Jesper] How to deal with overly massive historic competency attempt data?

    • Work-in-progress CBE ADR 0005 deals with storing learner subsection attempts (and other scored content) that demonstrate competency mastery. If competencies should be very widely adopted, as we hope, the largest OpenEdx instances could eventually have as many rows as the largest historic platform tables for learner attempts, possibly tens of billions of rows. As the new history table would scale with learner attempts, it can grow very quickly. Truncating the table after 2 years of historic data is accepted.

    • Question: do we need the history table at all? https://github.com/openedx/openedx-core/pull/657#issuecomment-5170123588

      • If we avoid it, we’ll need to change a couple of architectural things on the competency criteria tables

    • If we do want the history separately:

      • Just using clickhouse / aspects analytics and avoid the MySQL table?

      • Or: compressing data older than 1 month, maybe offloading to S3?

    • If we want the table currently discussed as storing all attempts separately in MySQL without compressing, what non-functional constraints do we need to adhere to, and how?

    • Discussion

      • Braden: can we reconstruct the data using student response history (CSMH) and historic CBE criteria info?

        • Jesper: That’s one of the questions I’m asking, it would be good to avoid the need to store all this data if possible.

        • Dave: problem is CSMH is not kept around for too long. Maybe a month or so on edx.org?

      • Is the history ever student facing? Yes, in the future.

        • Aspects is currently a staff facing thing only.

      • Is that history… we put 2 years as a rough marker in which we cand elete the history. on teh courseo r student module historys ide, we keep the history around for a very limited amount of time (1 month) because this is primarily for a debugging tool or a remote grader. The history is only useful when the course is actively running, so we kill it afterwards. Is tehre a point wehre the user has achieved htis thing so the history of how they’ve achieved it doesn’t matter anymore?

      • note: this is not for the willow MVP. but might affect data modelling decisions in MVP.

      • dave: CSMH was hastily built and has been operationally very challenging ever since. would want to wait until we have a solid use case and solid understanding of how we’ll query it before implementing another history table.

      • is CSMH truncation built in, or decided by operator?

        • decided by operator

          • thus, we can probably reconstruct mastery history

      • is there a point in time event where the user has achieved something, and thus the history no longer matters?

        • few different use cases

          • big one: learner dispute - they believe they should have earned mastery but system didn’t give it to them. a month or two?

          • for learners to understand how they did, knowing what they should go back and study for their summatives.

          • legal compliance--records need to be around for some amount time. 2+ years?

            • why is this different than CSMH?

            • probably isn’t. might need to persist CSMH for longer, too.

            • is a business decision of that organization.

            • could also be compressed and stored in a cheap (S3 glacier) way. but this could be complicated.

            • aspects is OK for this maybe?

          • for instructional designers to understand what course materials are/aren’t working well. but, aggregated data / aspects could work for that.

            • agreed that this should be in aspects.

          •  

      • Needs for Willow

        • Disputes ?

          • We actually do want to store history in willow, but just for admins to reverse engineer if necessary ?

          • What is the confidence level of dispute resolution being a Willow MVP requirement

          • Would like product documentation on this requirement

        • Working with EDL, disputes came up very frequently

          • Would resolving disputes require looking closely at history?

            • For EDL, they had to ask themselves “was this the learner’s fault or ours?” So being able to look back at attempt history was important.

        • We already store grades--this is about attempts, not grades

        • Aspects probably won’t have this for Willow

        • If we don’t do it for willow, then we wouldn’t have that data in Willow

        •  

      • Need for subsection-level history table?

      • Note that subsection-level history would involve one row per component attempt per competency.

      • Will ask Jenna & Tammy tmrw about needs for history

      •  

Jul 27, 2026

  • [Mary, Jesper] Talk through ADRs to move them along

  • [Jesper] Is there a good pattern for database reads in openedx-core?

    • edx-platform and openedx-core share the same database, while the dependency is one-way: edx-platform depends on openedx-core. This makes it awkward when openedx-core needs to read from models that may be defined in edx-platform.

    • Sometimes it may not be practical or clean to move the whole model from edx-platform to openedx-core if it’s just for a database read and we think the model would belong in edx-platform otherwise.

    • Can we find a good pattern that everyone is architecturally happy with?

    • Abstract Base Classes in openedx-core with minimal interface that the edx-platform model can inherit?

    • Some kind of bridge or adapter pattern?

    • Some kind of Python API?

    • Pass a reference to the model from edx-platform to the openedx-core library? (def register_model(self, django_model))

    • Example: we’ll need to read the grade of a subsection (currently stored in platform) in order to determine mastery (stored in core)

      • Generally, we want to avoid this. Would want to bring grades into core.

      • For example, we brought Course and CourseRun models into Core

    • Core should not even understand platform models

    • When complex stuff can’t be pulled into Core, then build the things that depend on it in Platform. e.g., the part of Content Libraries that works with the XBlock runtime, since the runtime is in Platform

      • CBE could have this same split.

    • Historical context: We’d been trying to extract large systems from openedx-platform. It’s very very hard, notably bc the dependencies are pointing all over the place. Want to keep them all pointing one direction.

    • Jesper: Sometimes it won’t work. We could pass individual callbacks in, like a get_grade callback

    • Kyle: aiming for openedx-core as self-contained data packged which can be pushed into. do not want to emulate xblock service system, which has very leaky callbacks

    • The “push to core” model basically always works

  • [Kyle] MongoDB-free courseware – what to target for Willow?

Jul 20, 2026

  • [Dave]: We're soon going to be hitting a point where we'll want to store user state against content in openedx_learning, and I would very much like to avoid writing usage keys everywhere. A large site like edx.org wastes terabytes of space because we don't normalize those references. Which means we should have a scalable, flexible data model representation for "this piece of content in this course" and a way to join that to student state.

    • Background

      • Right now there’s no database table for individual components in a course

      • When you add an xblock instance (usage), there’s no model instance representing that. Just a usage key (block-v1:MyOrg+MyCourse+MyRun+block@abcd+type@problem)

      • Every bit of state, including an answer to a problem, uses that key string

      • For every user in the course (10s of thousands)

      • Summary: We need primary keys for xblock usages

      • 99% of the time when we talk about student state, we’re talking about interactions between a user and an xblock. Things like “they attempte the problem by answering C” or “they stopped the video at 0:39”

      • We have one gigantic table called CoursewareStudentModule. It’s verrry large and very inefficient. Uses string keys, it’s denormalized, and stores student state as a student string.

      • Why?

        • Yes, primary key will be shorter than a usage key string

        • Can make the other columns more efficient – JSONField

        • We’ll be able to have FK constraints related to Usages and LearningContexts, which we couldn’t before

    • Migration

      • Will try to migrate many existing UsageKey strings over to the new tables

      • But will probably not be able to do it for the huge tables (CSM, CSMH)

      •  

    • Model sketch

      • LearningContext

        • Do we want to store student state directly against a library in the LMS?

          • Maybe yes - imagine

          • Does MIT use the library embed endpoint? Used for physics problems?

            • Two embed endpoints: one is for LTI, one is for iframe embed

        • There’s talk of things like micro-courses, “duolingo” type courses. Wouldn’t it make more sense to store that student state against Pathways (or some other learning context) instead of libraries?

        • Libraries probably shouldn’t have state stored against them. They’re a repository of libraries, not an instantiation.

          • LMS shouldn’t be aware of libraries? On principal, at least

      • LearningContextItemType ???

        • It’s only Course and Library now, but will probably include Learning Pathways soon, or possibly many types of Learning Pathways.

      • Usage

        • This means: Content Instantiation. Practically speaking in 2026, XBlock instantiation

        • I had a lot of kind of “out there” thoughts on Usage, but I think the simplest one is what I’d put forth as a proposal:

          • fkey to LearningContext, so a Usage is owned one and only one LearningContext.

          • nullable fkey to PublishableEntity, for a couple of reasons:

            • this allows us to bring over modulestore-backed content Usages for some uses (e.g. CBE), even before those courses get moved into openedx_content data models.

            • it guards against PublishableEntity deletion, e.g. deleting content won’t cause a cascade that wipes out student grades.

          • Usage has its own way to define namespace and identifier. This can often be redundant with LearningPackages, which also define their own namespace. But those do it from an authoring perspective. Usages have to be unique within a LearningContext, and Learning Pathways may want to use content that exists in different Libraries. Also, this would let us model something like a CCX, where Usages are M:1 on backing PublishableEntities.

        • Question: Will this data apply to Containers too, or just Components?

          • For instance, the Subsection container remembers the index of the Unit you were on. This is student state.

            • One option is for the LearningContext to remember it

          • Will we have usage keys for Containers, going forward?

            • We definitely need a key for the Container

            • But will we use them as UsageKeys going forward?

          • Jesper: Could have a lookup table which maps Container FKs to usage keys

            • But the system will probably need string keys for containers anyway

          • Braden: Could have ComponentUsage and ContainerUsage. Plugins will probably want to store data against containers, esp. Units

          • Mary: People will want analytics off of containers

          • Also grades, and completion.

          • What if we decouple string usage keys from the tables, use FKs as much as possible?

            • yes, we will just want to use the human-readable keys at the boundaries of the system

        • We’ll want to congizent of the extra lookup to translate to/from human-reaedable keys

          • Particularly we’ll want to not lock these tables, mostly used in read transactions

          • Think carefully about when the tables are written

          • e.g., constantly writing analytics info to these tables

        • Relevant:https://docs.openedx.org/projects/openedx-proposals/en/latest/best-practices/oep-0068-bp-content-identifiers.html

        • We do have mechanisms for storing student state against libraries. Was used for labxchange, but isn’t used now.

          • in CSM under the library key

          • no way to it from the UI. public embed URL

        • In studio, we store temporary state against components for purpose of testing content

    • How to store student state in a Pathway

      • e.g., you have 3 courses, and a standalone final exam in a pathway. is each course a learningcontext, and the final exam is too? is the entire pathway a learning context?

      • One option: make courses more lightweight, so the subsection could be a final exam.

        • This would break many assumptions baked into courseware. Would loosening those assumption be tractable as compared to building the standalone final exam as a separate kind of LC?

          • Braden: we could have backcompat wrappers for code which assumes the 3-tier structure. Like a ghost section

          • Mary: CC->Openedx conversion requires creating ghost units

        •  

  • [all] Any outstanding ADRs that would be good to hash out in person?

  • Question: any strategies when we want to access the database from openedx-core?

Jul 14, 2026

  • Conflict for Axim – sync up in Slack

Jul 7, 2026

Jun 16, 2026

  • keep fixing libraries bugs

  • keep merging in openedx-core docs

Jun 9, 2026

May 5, 2026

  • Next steps between now and verawood.1

    • [Dave] Course import into a library is painfully slow. The current pattern is to add and publish components one by one, meaning that even if events are processed on a PublishLog basis, we’re doing hundreds to thousands of them per import. Can we draft all the changes first and then publish all at once?

      • [Braden] PR: https://github.com/openedx/openedx-platform/pull/38508 Does that speed it up?

        • [Dave] It speeds up the actual database part of it, but it’s the meilisearch re-indexing that’s glacial. It looks like each block gets its own task. I guess ideally, we’d want to fire off the search updates in bulk based on the DraftChangeLog/PublishLog events.

          • [Braden] Thought that the modulestore side was slower than the meili side

          • [Dave] Not sure

        • [Dave] Also, this reminds me that we should have a .drafts property on DraftChangeLog so we can do something like publish_from_drafts(lp.id, change_log.drafts). “Publish the DraftChangeLog I just made” has to be a really common use case. Maybe its own function entirely.

        • [Dave] This is the top priority bugfix