This proposal has been prepared by OpenCraft, on behalf of one of our clients.
Goals of this proposal are:
To provide the data required to allow learners to see how much of the course they have completed (which XBlocks they have completed).
To provide a scalable mechanism for computing and reporting completion for all users and courses on both small and huge (edx.org scale) Open edX instances. This includes generating completion reports for instructors/partners that provide per-student completion breakdowns within a course.
To provide an API that can be used [on mobile and desktop] to retrieve a user’s completion of a course, even when courses are modified or are different per learner (adaptive content, cohorts, etc.).
Completion stats must be consistent between mobile and desktop.
To support offline interaction with [supported] XBlocks on mobile (e.g. watching videos offline), and include an API that native mobile apps can use to later synchronize the user’s completion with the Open edX server.
To support leaderboards within courses or cohorts (allow computing in which learners have completed most of the course). This is most likely only useful for small courses where there is enough diversity in the ranking.
To dovetail with the new Open edX persistent grades architecture
To allow future platform improvements like locked content that unlocks based on completion/consumption of ungraded content. (e.g. lock most of the courseware until the user watches a video).
Example: The conditional module (new Studio UI).
Example: the "Entrance Exam" feature which works at the section level. The learners must complete one section before being able to access the following sections.
To have a solution that can be implemented and deployed quickly.
Non-goals of this proposal are:
To propose, plan, or implement any UI in Open edX for displaying completion (that will come as a second phase, but OpenCraft has no plans so it's likely up to the community or edX to take that on).
Some of these are open questions - comments are welcome!
What is “completion”?
"Completion" means how much of the course content the learner has completed (as a percentage and also a breakdown of specifically which parts are completed so far).
We mean whether or not a learner has completed a course component (not merely viewed the component).
e.g. for videos you must watch (at least part of) the video, or for problems you must submit an answer. (n.b. Insights defines “complete” as having watched up to “either 30 seconds from the end or at the 95% complete mark, whichever means that more of the video has elapsed.”)
Only for presentation/ungraded XBlocks like “HTML” does “completed” mean “viewed”, since there is no action to take to “complete” the XBlock. (n.b. Per Shelby, edX “previously considered either just accessed, or viewed for ~X number of seconds (particularly for units that are text only - to account for just navigating quickly through a subsection).”)
e.g. for activities that require a file upload, you must upload a document.
If a user submits a problem that allows only one attempt, and they get a failing score (0%), is that problem considered “complete”? Yes.
If a user submits a problem that allows unlimited attempts, and they get a failing score, is that problem considered “complete”? Yes.
Does anything in Open edX currently track completion?
Not really - see “Prior Art” below.
We can’t use the presence of a grade, because some XBlocks like videos or HTML blocks do not emit grades.
We can’t use the presence of StudentModule etc. because those can be created for actions other than using/completing an XBlock (like opening the progress page) and reflect viewing an XBlock, not completing one.
Is there any value in storing “viewed” state separately from “completed”?
Probably not. (Note that specific XBlocks like video will still do their own tracking of how much of the video is viewed, but this proposal does not affect that nor generalize it)
Should completion be binary?
For future flexibility, we will represent completion for each completable block as a fractional float value in the range [0.0, 1.0], but the current implementation will treat completion at the completable block level as binary (only submitting values of 0.0 and 1.0). The aggregation tables will sum the values provided by the completable blocks.
Does completion ever decrease? (Do users ever “uncomplete” a block?)
Yes, we need to support decreasing completion (“uncompleting” a block) for these reasons:
Blocks can have their attempts reset and would need to be marked as incomplete again.
To implement something like the DoneXBlock (“Check this box if you’ve completed the offline activity”) which must be checked to be counted as "completed": we need to “undo” the completed state if the user unchecks the box.
For any XBlock with a file upload feature, we would need to mark it as incomplete if the user deletes the file
We should handle the situation where an XBlock is deleted/hidden from students, and that should update the completion appropriately.
What Blocks should be excluded from completion calculations?
Inline Discussion XBlocks
Blocks meant only for instructors, hidden course sections, etc.
Blocks should be able to be marked as excluded from completion calculations (e.g. child XBlocks of problem builder are often non-visual and must be excluded); our client here also has this requirement for other block types.
Should completion be cached at the subsection, section, and/or course levels?
Yes - this is necessary for scalable leaderboards, which is an explicit goal here.
Open edX does not currently track completion. It has the concept of a course grade, where
Grade = completion × score
(summed over all XBlocks, then divided by the number of blocks; we are ignoring weighting here)
e.g. in a course with two assignments, if a user completes one with a score of 75% and has not even attempted the second one yet, their “grade” is
Grade = ([1 × 75%] + [0 × 0%]) / 2 = 37.5%
Contrast this with how grades work in traditional schoolroom classes, where while the course is progressing, a traditional grade would not include the zero scores from assignments that have not yet been given to the student (so in the example above, the student’s interim grade would be “75%” not “37.5%”)
There is some minimal completion-tracking code in Open edX but it is currently unused for the most part. Marco explained: “We used to have fractional progress [completion] on problem pages. The sequence bar showed a sort of visual bar underneath the icon. This broke at one point and instead of fixing it we stripped it out.”.
ll the details. Is distracting to read a whole section and then get to a point where you later state that the section is irrelevant “Progress” object which represents a fraction like “3 out of 4” is defined in common/lib/xmodule/xmodule/progress.py
All XBlocks inherit a get_progress() method which returns None by default
The CAPA XModule implements get_progress() which returns a Progress object. CAPA also uses this internally.
A few “container” XBlocks implement get_progress() with identical code which simply returns the combined get_progress() results from each child block: VerticalBlock, SequenceModule, and SplitTestModule.
This custom plugin for Open edX called progress-edx-platform-extensions (or just “progress”), together with corresponding features in the Solutions REST API, provides an implementation of completion that meets some of the goals in this document.
User completion per-XBlock is stored in a model called CourseModuleCompletion.
Note for people checking out the code: the "stage" column now seems to be unused.
Another model called StudentProgress summarizes the student’s progress through the whole course (as a single number)
Beyond simple tracking of “completed” / “incomplete”, this progress plugin can:
Ignore certain block types.
It has some drawbacks:
It is designed for server-to-server access, so any authorized API client can read/write the progress of any user on the system.
Not known to be scalable (or possibly known not to be scalable)
It is not part of the core Open edX platform
Completion can be written by POST to /api/server/courses/:course_id/completions/ but only by a trusted server. We are not aware of anyone/anything that actually uses this method to update completion, however.
seq1
will be 100% (2/2), the completion value for seq2
will be 50% (1/2), and the completion value for the chapter will be 75% (3/4): chapter
/ \
seq1 seq2
/ \ / \
prob1 prob2 prob3
Displaying subsection completion status on the course outline/navigation page
Including “completion” reports in grade reports (e.g. download a CSV showing the completion/progress and grades of all users in a course)
Displaying course completion on the dashboard (e.g. saying “30% complete” below each course)
Course-level leaderboards.
Fields: user, course_key, usage_key, block_type, completed (float [0.0, 1.0]) + fields from TimeStampedModel.
A record with "completed" == 0.0 is equivalent to no record that exists.
It should have appropriate indexes for fast lookups.
This is roughly based on the CourseModuleCompletion model from progress-edx-platform-extensions but with some changes. The “get_actual_completions” code and the "stage" column would not be used.
The LMS should be modified to mark an XBlock as “completed” using this logic:
If the block type indicates that it is not completable (completion_method=EXCLUDED or completion_method=AGGREGATOR), then the block is ignored.
If the XBlock indicates that it is aware of the Completion API, and has a custom implementation of completion (“has_custom_completion” is true): Wait for the XBlock to emit a “complete” event with a value in the range [0.0, 1.0] (also support the deprecated name “progress” for this event, which is how it currently works for progress-edx-platform-extensions), and then mark it as completed.
Otherwise, if the XBlock indicates that it is gradable (“has_score” is true): when the XBlock emits a grade, have a signal handler emit a completion event.
Otherwise: when the XBlock is first viewed by the student (student_view is loaded and rendered), mark it as completed.
The code to emit the event for default blocks will live in the "vertical" XBlock JS, so that when a unit becomes visible, it emits the "complete" event for every descendent that does not match the criteria above (that is, when completion_method == COMPLETABLE && has_custom_completion == False && has_score == False).
A block will be marked complete when it has been visible in the browser window for N seconds, or has had focus for N seconds (to account for users whose interface does not contain a view window).
Whenever the "completed" value of an XBlock change, the LMS will call an asynchronous task to update the corresponding AggregateCompletion table rows. The asynchronous task request will contain a modified timestamp of the completed block and will check the modified timestamp on those rows to verify they are out of date before performing an update. (This task must not be processed until after the completable block update is committed).
If a user completes blocks while offline (this could be fairly common in a phone app, but is also possible in a browser), they will need to see their completion updated when they come back online. This is being explored further in a separate proposal for Offline XBlock support. Some draft ideas:
There could be an API endpoint that can accept a mapping of block IDs to completion ratios, and then the app can mark blocks as completed for the current user when they come back online.
Alternative: Allow both JavaScript XBlock runtimes and native iOS/Android XBlock views to emit events (“complete”, “submit”, etc.) somewhat like current handler calls, but asynchronous and offline-compatible. When the app is online again, submit all the timestamped events to the LMS server. Any XBlocks that received a “complete” event will be marked as complete. (This allows the definition of completion to be controlled by the XBlock, e.g. in offline webviews, rather than requiring the native app to code its own definition of complete for each XBlock).
A RESTful API:
In the course blocks API:
Notes about this approach:
The following format would be used for API requests:
GET /api/completion/v1/courses/course-v1:McKA+BBV+Fall2017/?requested_fields=sequential,chapter |
{ "course_key": "course-v1:McKA+BBV+Fall2017", "completion": { "earned": 3.0, "possible": 9.0, "percent": 0.33333333 }, "chapter": [ { "usage_key": "block-v1:McKA+BBV+Fall2017+type@chapter+block@week-1", "completion": { "earned": 3.0, "possible": 9.0, "percent": 0.33333333 } } ], "sequential": [ { "usage_key": "block-v1:McKA+BBV+Fall2017+type@sequential+block@week-1-a", "completion": { "earned": 3.0, "possible": 3.0, "percent": 1.0, }, }, { "usage_key": "block-v1:McKA+BBV+Fall2017+type@sequential+block@week-1-b", "completion": { "earned": 0.0, "possible": 6.0, "percent": 0.0, }, } ] } |
GET /api/completion/v1/courses/course-v1:McKA+BBV+Fall2017/ |
{ "course_key": "course-v1:McKA+BBV+Fall2017", "completion": { "earned": 3.0, "possible": 9.0, "percent": 0.33333333 } } |
GET /api/completion/v1/courses/?requested_fields=sequential,mean |
{ "pagination": {"values": "TBD"}, "data": [ { "course_key": "course-v1:McKA+BBV+Fall2017", "completion": { "earned": 3.0, "possible": 9.0, "percent": 0.33333333 }, "mean": 0.6, "sequential": [ { "usage_key": "block-v1:McKA+BBV+Fall2017+type@sequential+block@week-1", "completion": { "earned": 3.0, "possible": 3.0, "percent": 1.0, }, }, { "usage_key": "block-v1:McKA+BBV+Fall2017+type@sequential+block@week-2", "completion": { "earned": 0.0, "possible": 6.0, "percent": 0.0, }, } ] }, { "course_key": "course-v1:McKA+Drive+Fall2017", "completion": { "earned": 0.0, "possible": 2.0, "percent": 0.0 }, "mean": 0.95, "sequential": [ "usage_key": "block-v1:McKA+Drive+Fall2017+type@sequential+block@thewholething", "completion": { "earned": 0.0, "possible": 2.0, "percent": 0.0 } ] } ] } |
GET /api/completion/v1/courses/ |
{ "pagination": {"values": "TBD"}, "data": [ { "course_key": "course-v1:McKA+BBV+Fall2017", "completion": { "earned": 3.0, "possible": 9.0, "percent": 0.33333333 }, }, { "course_key": "course-v1:McKA+Drive+Fall2017", "completion": { "earned": 0.0, "possible": 2.0, "percent": 0.0 } } ] } |
GET /api/completion/v1/leaders/course-v1:McKA+BBV+Fall2017/ |
{ "results": [ {"username": "akiko", "rank": 1, "completion": {"earned": 100.0, "possible": 100.0, "ratio": 1.0}}, {"username": "benazir", "rank": 2, "completion": {"earned": 98.0, "possible": 100.0, "percent": 0.98}}, {"username": "clyde", "rank": 2, "completion": {"earned": 98.0, "possible": 100.0, "percent": 0.98}}, {"username": "d'artagnan", "rank": 4, "completion": {"earned": 92.0, "possible": 100.0, "percent": 0.92222222}} ] } |