Configurable grading method for problems with multiple attempts
Overview
The Uni Digital initiative (Spanish universities consortium) is interested and motivated to contribute the ability for problems with multiple attempts to support alternative grading methods.
In problems that allow multiple submissions, if the learner is allowed to submit more than one attempt, the current scoring function only takes the score of the last attempt.
With this modification, course creators will be able to select the desired grading method for the problem.
The alternative methods would include:
Last score (this is what the platform currently does)
First score
Highest score
Average score
Problem
The current single strategy of grading the last attempt could in some cases discourage learners from exploring a problem in more depth or punish them from trying. It could also make it easier for learners to use the multiple attempts to guess the correct answer without any negative consequence
A number of universities currently using Open edX for MOOCs and interested in leveraging the platform also for in-campus purposes believe this would help their course creators provide a better range of options when building course gradable components
Other LMS systems have this as an option by default with their features: Moodle, Canvas and Sakai for example.
Use cases:
As an instructor, I want learners to engage with problems risk-free, that is to be able to submit an answer and use up to the last available attempt without fearing they will get a score that is lower than the one they already have. This would apply when it is particularly difficult to provide a response, perhaps because multiple input boxes need to be filled, I want the learner to know that if they provided some response that got, for example, a partial credit, they can try again and try to provide a better one without risking the loss of the score they already obtained. In this case, the best grading method would be “highest score”
As an instructor, I want learners to engage with problems and reward a learner that gets the answer right in their first attempt, as opposed to one learner that needs to try 4 times to get it right. In this case my grading method of choice would be “Average score”
As an instructor, I want learners to engage with problems the way they always have. In this case my grading method of choice would be the default “last score”
As an instructor, I want to measure the learner's ability to get the response right in one attempt, but I also want them to engage with problems multiple times, and be able to explore other alternative answers, see the feedback, etc, basically use the problem as a learning activity after they submitted the first response and received a grade. In this case my grading method of choice would be “first score”.
Proposed solution:
The proposal is to add the “Grading method” as a new setting to the ProblemXblock and change the behaviour of the submit_problem ↗ method to accommodate the proposed grading methods.
This change will give course creators some more flexibility when creating and configuring graded components (problems), it will level up the capabilities of the platform against other widely used open source LMS Solutions, and most importantly it will provide improved ways for Learners in Open edX to engage with the problem components in a way that supports their learning.
This feature will require to store as part of the problem state the series of scores obtain by the learner in all its attemps, so that the average can always be computed and the grading strategy can be changed at some point in the future and rescore the problem correctly.
This new option would be appropriately included in the course author documentation. We have started a concept document to draft the information that would be included in the official documentation and where it would go.
Proposed visuals:
| proposed |
|---|---|
New problem editor |  
|
Legacy settings in Studio | |
Student view |
|
There is a roadmap issue for this feature here: https://github.com/openedx/platform-roadmap/issues/261
Other approaches considered:
We have also considered forking the Xblock and building a new version with these changes, but since they are minor improvements and additions to the capability of the current component, it feels like a sub-optimal approach not to contribute the changes to the official component.
Competitive research:
How do Canvas/Moodle/Coursera solve this problem?
Moodle includes this option:
sakai has this option too:
Proposed plan for any relevant usability/UX testing
TBD
Plan for long-term ownership/maintainership
edunext is committed to building and contributing this work as part of the Uni Digital (Spanish government) project. As part of that commitment, edunext would commit to maintain the feature for a minimum of 2 years and after that, either find a suitable maintainer to hand it over to, or to follow the deprecation procedure in case the feature has any inconveniences or its maintenance is a burden that no one can carry.
Open questions for rollout/releases
TBD