ORA Assessment Step Requirements

Owned by Eric Fischer (Deactivated)
Last updated: May 11, 2023 by Nathan Sprenkle
4 min read

Usage

Various assessment steps can be enabled for ORA problems, and each defines its requirements slightly differently. A step's requirements are used to determine when a step has been completed, both:

  1. by the learner (submitter_is_finished in code), "has the learner completed all their requirements for this step?) and

  2. for the learner (assessment_is_finished in code), "has everything been done to give the user their grade"?).

Different steps may make use of one, none, or both of these definitions of "completed". These methods are defined in openassessment/assessment/api/XXX.py, where XXX is the name of the step in question. These methods are called when the workflow is trying to update, as seen in update_from_assessments in https://github.com/edx/edx-ora2/blob/master/openassessment/workflow/models.py.

Here are all of the possible steps and the requirement values they define:

  • Example-based (AI) grading

    • Defines submitter_is_finished as always True, since the submitter should never have to do anything for this step.

    • Defines assessment_is_finished as True if an ai-graded assessment has been made. This step is decrecated(-ish), and should not be used going forward, but it exists.

    • If you're reading this page and need to know more about AI-grading, you're probably redefining how that works anyways.

  • Learner Training: This step, when enabled, will force learners to complete a training step teaching them how to assess peers before they actually assess peers. This is done by having the staff user, at time of problem authoring, define one or more examples and the 'correct' assessments of those answers. The requirements for this step are calculated dynamically by getting the number of examples specified in the problem definition (https://github.com/edx/edx-ora2/blob/master/openassessment/xblock/workflow_mixin.py#L87), and are passed around the backend as a dictionary containing a key 'num_required' and an integer value.

    • Defines submitter_is_finished by checking for completed examples, and checking if that number >= the num_required value.

    • Does not define assessment_is_finished at all, but that ends up being a moot point due to our step ordering rules forcing peer and staff (the only steps for which assessment_is_finished is out of the learner's control) to always be after training.

  • Peer Grading: This step, the "classic" use case for ORA, is used to have peers grade each other.

    • Peer grading takes it's requirements as a dictionary with 2 keys: must_grade and must_be_graded_by. Both values are integers. As one would think, must_grade is used in submitter_is_finished (has the learner graded #must_grade# peers yet?), and must_be_graded_by is used in assessment_is_finished (have #must_be_graded_by# other students graded the learner, so we can calculate and show a grade?)

  • Staff Grading: Staff grading can occur in 2 forms - as an override on any problem, and as a required step if enabled by the course author. I'll highlight the features of each type below. Keep in mind that the only entry in a staff step's requirements (if it exists) will have required as a key, and a boolean value. Also, the learner will never have to do anything for this step, so submitter_is_finished is always True.

    • Both required and override: Both steps, once completed, will serve as the “final” grade for a learner, regardless of any other types of assessments made before or after the staff assessment (a later staff assessment will override a pre-existing one).  Because of this finality, either type of staff grade will make assessment_is_finished return True for all steps, as a grade exists and nothing can change that fact. Note, however, that submitter_is_finished is totally separate, and a learner may still have to complete their requirements to receive their grade.

    • Override: As an override is always possible, staff assessment will always be an enabled step. assessment_is_finished will only return True if the requirements dictionary for this step is present, said dictionary defines 'required' to be True, and a staff assessment s present. If any of those 3 clauses is False, then the function returns False. Note that if staff grading is not required (no step requirements OR 'required': False), this will not prevent the user from finishing the problem and receiving a grade.

    • Required: If the course author defined a problem to be staff graded, then the requirements dictionary for this step will be {'required': True}. In this case, assessment_is_finished will return True if a staff assessment is present for the learner, and otherwise prevent the user from finishing the problem and receiving a grade.

Definition

During execution, the requirements are all stored together in a single dictionary, with each step's specific requirements stored as a value entry keyed on the step's name. These steps are not held in-memory, but rather fetched from the assessment modules every time the workflow for a problem is updated (see workflow_requirements() in https://github.com/edx/edx-ora2/blob/master/openassessment/xblock/workflow_mixin.py). The reasoning for this is that a course author may want to change a step's requirements, and have all learners see the effect of this. (Example - a peer grading problem defines must_grade and must_be_graded_by to both be 4, but only 3 learners ever complete the problem. The course author changes this number to be 2 for both values, and the learners see their now-completed grades upon refreshing the problem page.)

Thanks to this "fetch from the assessments" nature, the requirements are defined by the course author when setting up the problem in studio, and are then serialized into the problem's XML, to be loaded by the LMS. For an example, see https://github.com/edx/edx-ora2/blob/master/openassessment/xblock/test/data/peer_only_scenario.xml#L90.