Configurable grading strategy for peer assessments

Owned by Juan Camilo Montoya
Last updated: Oct 21, 2023
3 min read

The current calculation method for peer assessments is the median. This project aims to allow additional calculation methods such as the average.

Problem

The peer asessment step currently calculates the step score for each criterion by taking the median of the scores given by each peer. This decision may have been rooted in preventing outliers (extremely low scores in particular) from dragging the score down.

Alternative strategies such as the “Average all” or “Average drop low“ may be better suited for some educators or specific contexts.

Use cases:

  • As an instructor/learner, I need to be able to calculate the peer step grade as the average of all peer scores received, instead of the median.

    • Supporting market data: This has been requested by the consortium of spanish universities who use open edX intensively both in the mooc and in campus contexts.

Proposed solution:

The proposed solution is to add a setting to the peer step where course creators will be able to select the preferred grading strategy.

  • Include any UX/UI designs

This requires a simple intervention in the instructor UI for configuring an ora problem. In the modified UI, the peer configuration step would have this extra setting:

and then, in the learner UI, after the peer assessment step has been completed, the system tells what the calculation strategy was, so the interface will be adjusted accordingly:



 

Other approaches considered:

  • What other approaches did you consider and why won’t they work?

TBD

Competitive research:

  • How do Canvas/Moodle/Coursera solve this problem?

Moodle features a lot of flexibility for the calculation of final scores in a “Workshop“

The final grade can be split into 2 main weighted components:

  1. Grade for submission. The score a student gets for their submitted work

  2. Grade for assessment. The score a student gets for having reviewed other’s submitted work

Grade for submission

The final grade for every submission is calculated as weighted mean of particular assessment grades given by all reviewers of this submission. This includes the assessments given by peers and also the assessment given by the submitter, if allowed. The value is rounded to a number of decimal places set in the Workshop settings form.

The teacher can influence the grade in two ways:

  • by providing their own assessment, possibly with a higher weight than usual peer reviewers have

  • by overriding the grade to a fixed value

Grade for assessment

The grade for assessment tries to estimate the quality of assessments that the participant gave to the peers. This grade (also known as grading grade) is calculated by the artificial intelligence hidden within the Workshop module as it tries to do a typical teacher's job.

 

 

Proposed plan for any relevant usability/UX testing

TBD

Plan for long-term ownership/maintainership

edunext is commited to build and contribute this work as part of the unidigital (spanish government) project. As part of that commitment, edunext would commit to maintain the feature for a minimum of 2 years and after that, either find a suitable maintainer to hand it over to, or to follow a the deprecation procedure in case the feature has any inconvenience or its maintenance is a burden that no one can carry.

Open questions for rollout/releases

TBD