Canceling and resetting ORA responses

Owned by Eric Fischer (Deactivated)
Last updated: May 11, 2023 by Nathan Sprenkle
4 min read

There are 2 different actions, both only available to staff, that can be taken to remove or cancel ORA responses and assessments.

  1. Remove Submission From Peer Grading is available under Manage Individual Learner tools in the openassessment XBlock. Remove From Peer Grading is used to flag a submission as inappropriate, prevent others from seeing it, give a grade of zero, and not allow the student to re-attempt the problem.

  2. Delete Student State is available in 2 locations in the LMS - the staff debug viewer on the problem page, or on the Student Admin tab of the Instructor Dashboard. Delete Student State is intended to be used to remove test data, or other extraordinary circumstances that would require the student "starting fresh", as if they had never even attempted the problem.

Continue reading for more in-depth information on how these actions work.

Key players in this interaction and the data they store

Data that is affected by these actions lives in 3 locations: the LMS, the openassessment XBlock, and the edx-submissions module. Here's the relevant data for each:

  • LMS - StudentModule

    • submission_uuid (for the currently logged in learner)

    • student_id

    • course_id

    • problem_id

  • openassessment

    • Workflows (state of the student, both overall and for individual steps in ORA)

    • Assessments of the learner's response

    • note that both workflows and assessments are keyed on submission_uuid

  • submissions

    • Submission, the learner's response. Has a uuid, and is linked to a student_item (triplet of student_id, course_id, problem_id)

    • Score, keyed on student_item. Reports 'highest' and 'latest' scores for a given student_item

Removing from Peer Grading in depth:

When a staff user decides to remove a user's submission from peer grading, all the data stored in the LMS and submissions is entirely unaffected. Internal to openassessment, a special CancellationWorkflow is created, which is used to pull the user's submission from any potential grading pools (peer and staff both use this pooled approach). In addition, the student's overall Workflow is marked as cancelled, and automatically set to have a score of zero. The student cannot see their response, but is informed of their cancelled status when they visit the page. Staff can view the response with staff tools if they desire.

Deleting Student State in depth:

When a staff user deletes student state, the goal is to have the student appear as if they have never attempted the problem. This is done by changing data in the LMS and submissions, but does not touch any data in the openassessment XBlock. The functionality to do this is in LMS code as a method reset_student_attempts (https://github.com/edx/edx-platform/blob/master/lms/djangoapps/instructor/enrollment.py).

First, the LMS will contact submissions and issue a 'reset score'. This has the effect of 'resetting' the score associated with the relevant (student, course, problem) triplet to be as if there were no scores ever given. Cool, that's hunky-dory.

Then, the LMS will delete the student module, effectively 'losing' the uuid key to the submission. Since the openassessment XBlock uses submission_uuid as a key for everything, a user returning to the problem without one saved will be shown the "make a response" dialog so they can begin the problem. Thanks to this approach, data is never deleted (so it can be kept for analytics purposes), and the student gets to re-attempt the problem as they'd like. Success on all fronts!

Hold up a minute though, here's where we start to poke holes in this approach. You see, the openassessment XBlock was never informed of this 'delete student state' operation, so it doesn't know that it should be hiding the relevant Workflows from grading pools. Because of this, staff and peer graders may very well be shown a response to grade after its author's state has been reset!

In addition, in submissions, the learner's Submission was never deleted or unlinked from the relevant (student, course, problem) triplet. This is fine, as long as we assume that "the current user's context" is the only way to access such a thing, and a nonexistent submission_uuid in that context means no submission should be fetched. Previously, this was a reasonable assumption. However, we've now added staff tools to the mix. Using the staff's "Manage Individual Learner's" button, a staff user can ask submissions to see all data related to a specified (student, course, problem) triplet. This is very confusing to staff users who may use this tool to check and see if their 'Delete State' request went through.

My proposal to fix delete state has 3 main parts:

  • The LMS should contact the openassessment XBlock directly, rather than going around it to submissions

  • The openassessment XBlock should issue a cancellation for the relevant workflows, so that the response is no longer present in grading pools

  • The Submission stored in submissions needs to have its student_item key removed or broken, so that the 'deleted' response is not accessible regardless of context.

Follow along on the above linked ticket for resolution, I'll update this page after any fixes have merged.