Usage
Various assessment steps can be enabled for ORA problems, and each defines its requirements slightly differently. A step's requirements are used to determine when a step has been completed, both both:
by the learner (
submitter_is_finished
in code), "has the learner completed all their requirements for this step?)
...
and
for the learner (
assessment_is_finished
in code), "has everything been done to give the user their grade"?).
Different steps may make use of one, none, or both of these definitions of "completed". These methods are defined in openassessment/assessment/api/XXX.py
, where XXX
is the name of the step in question. These methods are called when the workflow is trying to update, as seen in update_from_assessments
in https://github.com/edx/edx-ora2/blob/master/openassessment/workflow/models.py.
Here are all of the possible steps and the requirement values they define:
Example-based (AI) grading
:Defines
submitter_is_finished
as always True, since the submitter should never have to do anything for this step.Defines
assessment_is_finished
as True if an ai-graded assessment has been made. This step is decrecated(-ish), and should not be used going forward, but it exists.If you're reading this page and need to know more about AI-grading, you're probably redefining how that works anyways.
Learner Training: This step, when enabled, will force learners to complete a training step teaching them how to assess peers before they actually assess peers. This is done by having the staff user, at time of problem authoring, define one or more examples and the 'correct' assessments of those answers. The requirements for this step are calculated dynamically by getting the number of examples specified in the problem definition (https://github.com/edx/edx-ora2/blob/master/openassessment/xblock/workflow_mixin.py#L87), and are passed around the backend as a dictionary containing a key 'num_required' and an integer value.
Defines
submitter_is_finished
by checking for completed examples, and checking if that number >= the
num_required
value.
Does not define
assessment_is_finished
at all, but that ends up being a moot point due to our step ordering rules forcing peer and staff (the only steps for whichassessment_is_finished
is out of the learner's control) to always be after training.
Peer Grading: This step, the "classic" use case for ORA, is used to have peers grade each other.
Peer grading takes it's requirements as a dictionary with 2 keys:
must_grade
and
must_be_graded_by
. Both values are integers. As one would think,
must_grade
is used in
submitter_is_finished
(has the learner graded #must_grade# peers yet?), and
must_be_graded_by
is used in
assessment_is_finished
(have #must_be_graded_by# other students graded the learner, so we can calculate and show a grade?)
Staff Grading: Staff grading can occur in 2 forms - as an override on any problem, and as a required step if enabled by the course author. I'll highlight the features of each type below. Keep in mind that the only entry in a staff step's requirements (if it exists) will have
'
'required
as a key, and a boolean value. Also, the learner will never have to do anything for this step, so
submitter_is_finished
is always True.Both required and override: Both steps, once completed, will serve as the
'final'“final” grade for a learner, regardless of any other types of assessments made before or after the staff assessment (a later staff assessment will override a pre-existing one). Because of this finality, either type of staff grade will make
assessment_is_finished
return True for all steps, as a grade exists and nothing can change that fact. Note, however, thatsubmitter_is_finished
is totally separate, and a learner may still have to complete their requirements to receive their grade.Override: As an override is always possible, staff assessment will always be an enabled step.
assessment_is_finished
will only return True if the requirements dictionary for this step is present, said dictionary defines 'required' to be True, and a staff assessment s present. If any of those 3 clauses is False, then the function returns False. Note that if staff grading is not required (no step requirements OR 'required': False), this will not prevent the user from finishing the problem and receiving a grade.Required: If the course author defined a problem to be staff graded, then the requirements dictionary for this step will be {'required': True}. In this case,
assessment_is_finished
will return True if a staff assessment is present for the learner, and otherwise prevent the user from finishing the problem and receiving a grade.
Definition
During execution, the requirements are all stored together in a single dictionary, with each step's specific requirements stored as a value entry keyed on the step's name. These steps are not held in-memory, but rather fetched from the assessment modules every time the workflow for a problem is updated (see workflow_requirements()
in https://github.com/edx/edx-ora2/blob/master/openassessment/xblock/workflow_mixin.py). The reasoning for this is that a course author may want to change a step's requirements, and have all learners see the effect of this. (Example - a peer grading problem defines 'must_grade
' and 'must_be_graded_by
' to both be 4, but only 3 learners ever complete the problem. The course author changes this number to be 2 for both values, and the learners see their now-completed grades upon refreshing the problem page.)
...