Open Response Assessment - Educator & Learner Updates

TL;DR: Over the past six months we have landed several improvements to the Open Response Assessment experience through a blended development project spanning both T&L & Programs Learner Experience Squads. These changes are detailed in separate linked documents but an overview is provided below.

TL;UseEmoji:

Engineering Lead: @Justin Lapierre (Deactivated) , @Mat Carter (Deactivated), (previously also Dave St. Germain )
Product Lead: @Sapana Thomas (Deactivated) , Marco Morales

What is it?
This project was aimed at improving Open Responses Assessments (ORA), the 4th most commonly used core course content block after Video, Text, and Problems. 31% of the edX course catalog uses ORA components in their course content. Various limitations of this tool impact educators, especially when using staff or peer grading workflows. These use cases span basic educator moderation and usability issues, limitations on peer grading ORA in self-paced courses, and other triage/moderation issues, especially in higher-touch or credit focused courses.

These changes are primarily focused on helping reduce the operational and instructional pain of courses that rely on peer and staff grading in particular. A recap of the work delivered between July and today is summarized below. Unless otherwise specified, all of the changes described in this section are now live on edX.org and edge.edx.org. A few of these changes also have pending authoring documentation updates we expect to publish soon as well.
 

1. Authoring Moderation & Reporting

   1. Added username details to ORA report download

   2. Added problem name and location to ORA report download

   3. Added ORA zipped file download for submission text + attached files

   4. (Still In Development ~4-6weeks away) Updated “Waiting” Open Responses Dashboard with Override Grade Option(s)

   5. Support Flexible Peer Grading Averaging for Learners delayed / in Peer Grading step

2. Authoring Ease & Usability

   1. Open response assessment problems as a component category

   2. Separate assessment steps & schedule authoring areas

3. Learner Facing Updates

   1. Grading status message

   2. Allow viewing ORA steps after peer review

   3. (Still In Development ~1-3 weeks away) Updated ORA Prompt Submission Editor

Key talking points for customers:

Why work on this area of the platform?
Open Response Assessments were flagged as a top 20 Teaching and Learning platform gap in the last platform map rating exercise about a year ago. Commonly cited as a time-intensive instructor process when this component uses staff and peer assessments, this was also called out as a reason why some course teams haven’t switched over to using self-paced courses. This component can provide rich learner feedback through its rubric-based grading and more broadly open-ended assignment submissions can drive higher quality courses beyond our other basic problem submission options.

Who will notice the change, and where?
We have published a partner portal post letting people know about the changes since our last post in 2020, and any educator authoring ORA components will be able to see the improvements.

What impact will it have on course development teams?
We hope course teams can make more use of ORA components and that courses with ORA components spend less instructor time managing/moderating these problem types.

Results
We are currently gathering feedback from the platform map ratings exercise both in quantitative & qualitative ratings to confirm how much of the common educator gaps have been addressed by this effort.

While several partners have referenced the value of these improvements, our original scope estimate was that about half the gaps in the ORA moderation experience would be covered by our efforts so far. Future improvements to staff grading we hope will close the bulk of the remaining commonly cited issues with the use of Open Response Assessments.

Credits / A Group Effort!

Thanks to the Programs Learner Experience & T&L Squads for all their efforts on this project. This effort has been run as a blended development project in parallel to other efforts in FY21.