Page Comparison

Actions that can be taken to get rid of existing ORA responses and assessments

There are 2 different actions, both only available to staff, that can be taken to remove or cancel ORA responses and assessments. The first, '

1. Remove Submission From Peer Grading

...

1. is available under

...

1. Manage Individual Learner

...

1. tools in the openassessment

...

I'll expand on each in more detail shortly, but the tl;dr of each is:

1. XBlock. Remove From Peer Grading is used to flag a submission as inappropriate, prevent others from seeing it, give a grade of zero, and not allow the student to re-attempt the problem.

2. Delete Student State is available in 2 locations in the LMS - the staff debug viewer on the problem page, or on the Student Admin tab of the Instructor Dashboard. Delete Student State is intended to be used to remove test data, or other extraordinary circumstances that would require the student "starting fresh", as if they had never even attempted the problem.

Continue reading for more in-depth information on how these actions work.

Key players in this interaction and the data they store

Data that is affected by these actions lives in 3 locations - : the LMS, the openassessment xblockXBlock, and the edx-submissions module. Here's the relevant data for each:

• LMS - StudentModule

  • submission_uuid (for the currently logged in learner)

  • student_id

  • course_id

  • problem_id

• openassessment

  • Workflows (state of the student, both overall and for individual steps in ORA)

  • Assessments of the learner's response

  • note that both workflows and assessments are keyed on submission_uuid

• submissions

  • Submission, the learner's response. Has a uuid, and is linked to a student_item (triplet of student_id, course_id, problem_id)

  • Score, keyed on student_item. Reports 'highest' and 'latest' scores for a given student_item

Removing from Peer Grading in depth:

When a staff user decides to remove a user's submission from peer grading, all the data stored in the LMS and submissions is entirely unaffected. Internal to openassessment, a special 'CancellationWorkflow' is created, which is used to pull the user's submission from any potential grading pools (peer and staff both use this pooled approach). In addition, the student's overall Workflow is marked as cancelled, and automatically set to have a score of zero. The student cannot see their response, but is informed of their cancelled status when they visit the page. Staff can view the response with staff tools if they desire.

Deleting Student State in depth:

...

:

...

...

When a staff user deletes student state, the goal is to have the student appear as if they have never attempted the problem. This is done by changing data in the LMS and submissions, but does not touch any data in the openassessment xblock XBlock. The functionality to do this is in LMS code as a method reset_student_attempts (https://github.com/edx/edx-platform/blob/master/lms/djangoapps/instructor/enrollment.py).

...

Then, the LMS will delete the student module, effectively 'losing' the uuid key to the submission. Since the openassessment xblock XBlock uses submission_uuid as a key for everything, a user returning to the problem without one saved will be shown the "make a response" dialog so they can begin the problem. Thanks to this approach, data is never deleted (so it can be kept for analytics purposes), and the student gets to re-attempt the problem as they'd like. Success on all fronts!

Hold up a minute though, here's where we start to poke holes in this approach. You see, the openassessment xblock XBlock was never informed of this 'delete student state' operation, so it doesn't know that it should be hiding the relevant Workflows from grading pools. Because of this, staff and peer graders may very well be shown a response to grade after its author's state has been reset!

...

My proposal to fix delete state has 3 main parts:

• The LMS should contact the openassessment

  xblock

  XBlock directly, rather than going around it to submissions

• The openassessment

  xblock

  XBlock should issue a cancellation for the relevant workflows, so that the response is no longer present in grading pools

• The Submission stored in submissions needs to have its student_item key removed or broken, so that the 'deleted' response is not accessible regardless of context.

Follow along on the above linked ticket for resolution, I'll update this page after any fixes have merged.