Competency Mastery Concurrency ADR - #657
Conversation
|
Thanks for the pull request, @jesperhodge! This repository is currently maintained by Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review. 🔘 Get product approvalIf you haven't already, check this list to see if your contribution needs to go through the product review process.
🔘 Provide contextTo help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:
🔘 Get a green buildIf one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green. DetailsWhere can I find more information?If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources: When can I expect my changes to be merged?Our goal is to get community contributions seen and reviewed as efficiently as possible. However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:
💡 As a result it may take up to several weeks or months to complete a review and merge your PR. |
Rejected-alternative items 1 and 2 nested Pros/Cons bullet lists directly under a continuation paragraph at a different indent with no blank line separator, and item 3's closing paragraph was indented to neither the list body nor its sub-bullets. docutils flagged these as errors under -W, failing the readthedocs build.
mgwozdz-unicon
left a comment
There was a problem hiding this comment.
Following up on feedback I'd shared earlier outside this PR favoring the batch-lock approach over keyed partitioning, given the Kafka/Redis dependency and the accuracy-over-latency tradeoff for this use case: this revision reflects that direction, and the Kafka/Redis coupling concern is resolved.
I'm wondering if we should also consider a per-learner lock instead of the single deployment-wide lock. A DB advisory lock keyed on a hash of user_id would give the same same-learner serialization the chosen approach relies on, but would let different learners' batches run in parallel, which the current design gives up entirely. It also sidesteps everything Alternative 1 was rejected for: no event-transport dependency, no partition-key contract with openedx-platform, no reconciliation backstop.
Rejected Alternative 2's argument doesn't quite cover this: it treats "per-learner isolation" as equivalent to "keyed partitioning" ("Making parallel evaluation correct requires... per-learner isolation, i.e. the keyed-partitioning alternative above"), but a per-learner DB lock gets you per-learner isolation without touching the transport at all. If there's a reason this doesn't hold up here (lock-management overhead across many concurrent per-learner locks, contention patterns, something else), that reasoning is worth adding to the doc.
Requesting changes to get this addressed, either by adding the analysis to Rejected Alternatives or by folding it into the Decision.
| - **Same-learner correctness.** Grade-change events arrive asynchronously and can be delivered out | ||
| of order and processed on more than one worker. Because writes are append-only, two evaluations | ||
| for the same learner that overlap can each read a stale snapshot of the sibling leaf statuses and | ||
| each append a derived roll-up computed from an incomplete picture (a write-skew). Leaf rows are | ||
| always correct, since each leaf is a pure function of its own grade; only the derived | ||
| group/competency rows can be left wrong. Nothing crashes and no constraint is violated, but a | ||
| learner's stored competency status can be silently incorrect. |
There was a problem hiding this comment.
Maybe this is a silly question, but can't we mitigate this by making sure to commit the transaction of the leaf statuses, and then re-read the latest commited state of the leaves before doing the roll-up?
There was a problem hiding this comment.
Another thing we can do (if it's necessary) is to arrange it so that the roll-ups are a log that have an incrementing version number, and create a composite unique index that would cause a conflict that would reject concurrent writes. The losing write would have to catch the error, increment the version number again, and re-read the database state.
There was a problem hiding this comment.
I think this question is not relevant anymore with the current version, right? Since now we only process on one worker at a time.
@mgwozdz-unicon I'll improve the description in the ADR. Here's why this solution is not good: It fights the batching that the performance depends on. The chosen design's throughput comes from batching across learners: one bulk read of current statuses, evaluate the whole batch in memory, one bulk write — a few round-trips regardless of how many learners/events are in the batch. A per-learner lock is naturally per-learner (in the earlier design, per-event): acquire lock → read that learner → evaluate → write → release, one learner at a time. That reintroduces per-learner (or per-event) transaction/commit overhead plus lock acquire/release churn, which is exactly the overhead batching was collapsing. Under bursty grading across many learners, that per-unit overhead dominates. Lock-lifecycle machinery would be multiplied across millions of keys. One deployment-wide lock has exactly one lifecycle to run: acquisition, timeout, stale-lock recovery on a crashed worker. A per-learner lock needs a per-learner lock table (or keyed advisory locks) and that same lifecycle replicated across potentially millions of learner keys, held and waited on concurrently, plus a connection+worker tied up per contended lock. |
|
@mgwozdz-unicon remind me why we even need the history tables and not only keep the active tables? |
We need people to be able to audit student mastery statuses to understand why a learner did or didn't achieve mastery of a particular competency or any of the associated "measurement instruments" (gradeable subsections). |
| request thread. After that task writes the subsection grade, it calls a public openedx-core function | ||
| within the same transaction; this function does the monotone merge and the upward roll-up. This should be generalized as needed to other places that trigger a competency status update. | ||
|
|
||
| **4. ACTIVE writes and roll-ups commit atomically with the grade, or the whole task rolls back and retries.** |
There was a problem hiding this comment.
Either it's not clear to me what this is trying to say or it contradicts point 3 above it. I think my confusion might stem from the phrase "with the grade" because that is implying to me that writes to our competency status tables roll up atomically with the subsection grade table. But I think this is trying to say that writes for the competency criteria hierarchy status tables commit atomically with each other. Am I confused or does this need to be reworded for clarity, or both?
There was a problem hiding this comment.
Nevermind, I reread point 3, and I see that they're not contradictory now. It's possible that this could still be made clearer though since I did have to reread to understand.
| Rejected Alternatives | ||
| --------------------- | ||
|
|
||
| 1. Prevent concurrent writes with a deployment-wide lock. |
There was a problem hiding this comment.
I think it would be nice if the Rejected Alternatives each had brief Pros and Cons.
There was a problem hiding this comment.
Or a rationale sentence or two like in ADR 5 below.
| Once every write is a monotone merge and each | ||
| parent recompute is serialized only against concurrent writers of that same node by a brief row lock, correctness holds without needing a larger lock. | ||
|
|
||
| 4. Recompute derived levels on read instead of materializing them. |
There was a problem hiding this comment.
This numbering is off
|
@jesperhodge The inline comments are from my completed review. I had Claude review as well. These are the things that came up from that which I found to be relevant:
I'm pretty sure that this was because those tables are smaller, but I agree that this should be made clearer.
I would prefer at least one retry before logging and ignoring. If there's an accepted pattern in openedx for this though, then I defer to that.
|
|
Django's design assumes In [3]: from django.db import connection
...:
...: with connection.cursor() as cursor:
...: cursor.execute("SELECT @@transaction_isolation;")
...: print(cursor.fetchone()[0])
...:
READ-COMMITTEDRunning in higher isolation levels is not supported by our platform. |
|
@mgwozdz-unicon @ormsbee I updated the ADRs based on your comments. |
mgwozdz-unicon
left a comment
There was a problem hiding this comment.
Thanks for getting those changes in! Approving, though Claude flagged one more thing: the db_constraint=False decision itself has no stated rationale anywhere, it's the one bullet in ADR 0005 without any, while everything else in that doc (64-bit PKs, the dedicated HISTORY alias, read-replica scoping) explains why. This is the same field Braden raised a deletion-protection concern about on #661: that's resolved since his approach keys off the competency_criteria_id FK, not user_id, but that resolution isn't written down yet either. What's the actual reasoning for dropping the constraint on user_id here, and is it established for this table specifically, or assumed by analogy to StudentModule/PersistentSubsectionGrade? Not blocking on this, just think it might be good to add.
|
The concurrency ADR looks good to me. I have just a couple of concerns with the storage one. Will write up in the morning. |
ormsbee
left a comment
There was a problem hiding this comment.
These concerns are over the data storage portion. Please forgive me if I've made these comments earlier. (I thought I had already commented on the 64-bit ints before, but maybe I never properly submitted that review.
| For each of these, we also need to persist history, because we need an audit trail to understand | ||
| why a learner did or didn't achieve mastery of a particular competency or any of the associated "measurement instruments" | ||
| (gradeable subsections). |
There was a problem hiding this comment.
It does not block this ADR, but let's talk with Product/UX folks and explore just how much of an audit history we really need. History storage is the most dangerously unbounded thing in this system. Moving it to a separate system and potentially pushing it to another database entirely doesn't really solve it.
For instance, say we determine that we only need history showing when they cross the aggregate threshold for the first time. We could potentially save billions of rows of storage for a large site that would otherwise be storing a new row for every time someone clicked "submit" on a problem somewhere.
|
|
||
| That scale is not, on its own, what makes a relational database struggle. A point lookup against a | ||
| billion-row table backed by the right composite index is a logarithmic-time index seek regardless | ||
| of the table's size; the dashboard reads this feature performs are exactly such point lookups. What |
There was a problem hiding this comment.
| of the table's size; the dashboard reads this feature performs are exactly such point lookups. What | |
| of the table's size; the dashboard that reads this performs exactly such point lookups. What |
Is this what was meant?
| They do not use physical partitioning, but work by using 64-bit primary keys, | ||
| enabling optional read-replica offloads, and splitting out a history table that | ||
| uses a dedicated database router. The decisions below are designed to mirror these | ||
| existing patterns. |
There was a problem hiding this comment.
Okay, the database router thing was a giant, ugly hack, because I threw together the original StudentModuleHistory model in a half-day hackathon project to help out one of the first CS courses on the platform, and I neglected to override the primary key defaults to make it 64-bit. The result was that we were running out of primary keys, and some devops folks (Kevin I think?) retrofitted a new table in there
Also, the courseware_studentmodulehistory table stays somewhat manageable for large instances because they truncate it from time to time, because this history is used for course staff debugging purposes, not auditing. So this week's history is really important when a student is asking, "I submitted the right answer, why is the grader marking me wrong?", but that's not a thing we need to look up six months from now.
| **64-bit primary keys from the start.** The leaf ACTIVE and HISTORY tables use a 64-bit | ||
| ``BigAutoField`` primary key, chosen up front, mirroring edx-platform's | ||
| ``UnsignedBigIntAutoField`` on ``PersistentSubsectionGrade`` ("primary key will need to be | ||
| large for this table"). Changing a primary-key type on a billion-row table later is | ||
| prohibitively expensive. |
There was a problem hiding this comment.
This shouldn't need to be justified, as BigAutoField should be the default practice per OEP-38.
Also, please don't put up anything to imply that UnsignedBigIntAutoField is a good idea. It was an overkill reaction to having 32-bit primary keys run out on key tables, but it's more headache than it's worth. It theoretically gives us 2X the space of BigAutoField, but (a) the existing space of BigAutoField is so big that we'll never realistically reach it; (b) reaching it would break the database in a bunch of other ways anyway; and (c) declaring a custom field type like that introduces maintenance headaches like... (d) painful compatibility issues with PostgreSQL, which does not support unsigned ints.
| **A dedicated database alias and router for the leaf HISTORY table, baked in from the start.** | ||
| This only applies to the `StudentCompetencyCriteriaStatusHistory` table. | ||
| The leaf HISTORY table (``StudentCompetencyCriteriaStatusHistory``), the dominant table in this | ||
| model, is the one learner-status table assigned to a dedicated Django database alias through a | ||
| database router, mirroring edx-platform's courseware-history router | ||
| (``StudentModuleHistoryExtended``), which is likewise a history table. The alias defaults to the | ||
| main database. |
There was a problem hiding this comment.
Please don't do this. See my comment above about why this routing exists. Let's keep it simple and in the same database so that transactions just work.
| ``StudentModule``. This follows the app-boundary decision in :ref:`openedx-learning-adr-0001`, which keeps | ||
| learner-status models decoupled from the concrete user model. Furthermore, |
There was a problem hiding this comment.
I'm not clear as to what part of the referenced ADR this is drawing from, but our convention in this repo has been to have real foreign keys to the User model—it's just that we respect Django's ability for projects to define custom user models by grabbing it from settings.AUTH_USER_MODEL, instead of assuming that it's always django.contrib.auth.models.User.
There was a problem hiding this comment.
Maybe my misunderstanding.
| a hot user row would see extra lock contention under concurrent writes. This | ||
| follows the app-boundary decision in :ref:`openedx-learning-adr-0001`, which keeps | ||
| learner-status models decoupled from the concrete user model. |
There was a problem hiding this comment.
So in theory, very little should write to the User model, but it's true that we have seen reports of concurrency issues that we don't understand very well at this point:
- feat!: rm user foreign key in db completion#443
- feat: bookmarks rm user foreign key db constraint openedx-platform#38815
On that note, if there is competency information that makes sense to attach in a 1:1 way per student, e.g. a CompetencyStudent model that would go 1:1 with User, it might make sense to make various competency models foreign key to that instead of directly to User.
In either case though, I think we can remove this decision from the ADR.
| **Enable read-replica offload for heavy reads for the leaf tables.** | ||
| This only applies to the ACTIVE `StudentCompetencyCriteriaStatus` and HISTORY | ||
| `StudentCompetencyCriteriaStatusHistory` tables. Offload is scoped to the leaf level because that is | ||
| where row count and read volume concentrate (the ~200x per-course leaf multiplier above); the group | ||
| and top-level status tables are orders of magnitude smaller, so their dashboard reads are cheap point | ||
| lookups that do not need to leave the primary and are not worth the replica-lag complexity. | ||
| Prior art: ``edx_django_utils``'s ``read_replica_or_default()``. |
There was a problem hiding this comment.
This is always a possibility for the future, but one we'd want to punt until we had verified that it was a problem worth tackling and testing for. StudentModule load is really high because it's used in many, many places, the payload can be enormous, and the lookup pattern happens via a very expensive/large composite index. My hope would be that even on sites that make heavy use of CBE, the load from it should be much lower.
| **Advance-only banking, monotonic.** Once a node reaches ``Demonstrated`` its ACTIVE row is retained | ||
| ("banked"): the recorder never automatically regresses it, not on a later downward grade correction | ||
| and not on a criteria change. This applies at every level, including the leaf. A genuine downward |
There was a problem hiding this comment.
I strongly suspect that this will end up changing in the future, but I'm okay with it as a starting position.
| **Retroactive criteria changes are monotonic for the learner.** A retroactive edit can newly grant | ||
| or preserve mastery, but it never silently revokes it, and it never rewrites a learner's recorded | ||
| leaf mastery downward. |
There was a problem hiding this comment.
Again, I take the point that we shouldn't silently do this by default, but I expect people will have errors in their content settings, and this will need to be adjustable at some point. But also again, I'm not going to block this as a starting position.
Also, we've had mention of use cases in the past where mastery has a shelf-life, e.g. Continuing Medical Education (CME) credits.
UPDATE: ADRs in this PR have been completely rewritten as of July 17, 2026.
Strong involvement from Claude (AI).
I haven't fully reviewed changes to diagrams and ADRs 2 and 3 yet, made with Claude, need my own review still.
Summary
These are some significant changes to the data model and approach. They will warrant a good amount of discussion.
A main concern is that the leaf node table and leaf node history table could possibly contain billions of rows, at the scale of the
PersistentSubsectionGradetable of edx-platform. The current solution accepts this, but is this acceptable for OpenEdx? If not, there is also a rejected alternative to not store leaf nodes and just compute them at read time, but that would make it harder to keep subsection statuses frozen so that later competency criteria changes don't change the status.ADR 4: How learner competency mastery is recorded
correctly under concurrent, out-of-order grade-change events without
paying a per-event serialization cost at scale.
ADR 5: How competency criteria statuses for learners are stored at scale, given the tables
can be massive (billions of rows).