ORA - Current State, Gaps, and Opportunities
- Matthew Harrington
Current State
While the ORA component is accessible and well-built, the process of configuring ORA activities is extremely complex and could be significantly more guided, particularly for non-academic authors unfamiliar with rubrics. In addition, the component itself is extremely "heavy", taking a great deal of screen real-estate and effort to implement and use, commonly leading authors who want to do things that ORA is fully capable of to use other tools such as the SGA XBlock.
In addition to this, staff-facing elements have numerous usability issues compared to the standard of the learner experience, and some elements do not conform to current UI standards (such as button styling).
Configuration
Constructing the ORA takes place in Studio, while management of running ORAs occurs in the LMS. This is mostly logical, but has a number of issues of note, including:
Staff are referred to the ORA documentation in multiple locations, but not told where to find the ORA documentation or what part of the ORA documentation they’re looking for.
Grading is fully dependent on the assignment having a robust rubric, which can only really be reasonably expected of academic staff with prior rubric crafting experience.
A simple flat grade entry is often desired by non-academic users who do not understand rubrics, or for more flexible grading from expert graders who want to grade according to a different system.
It is not simple to “seed” peer-assessed ORAs with additional responses, which means that the first learner to reach the ORA will regularly have nothing to grade.
The rubric cannot be reordered or restructured once entered - options must be deleted and reinserted
The default content present when an ORA is inserted requires a lot of deletion before the author can actually start creating their own ORA
Media content can be embedded in ORA in a player that does not match the rest of the platform, rendering it inaccessible and non-standard for users
Dates and deadlines are listed on all ORAs independently, meaning that anyone attempting to use ORA on a self-paced course must set these dates dramatically into the future to avoid learners from being locked out of the ORA.
This also means that their dates are handled differently to and separately from other due dates across the platform.
Generally, ORA usage expects a high level of competency from course staff users during configuration, making it daunting to use for less knowledgeable course authors.
Once an ORA is underway, grading and user management takes place in the LMS, via small buttons underneath each ORA.
Manage individual learners allows jumping to a specific learner’s assignment with the option to override their grade. View assignment statistics shows an overview of how many learners are at each step, and Grade Available Responses allows a staff user to “check out” a response for staff grading, locking other staff out of grading that response. These controls have the following issues:
While a learner’s response can be accessed and graded via Manage Individual Learners, the staff member must already know which learner they are looking for via the discussion or another source such as the gradebook. There is no way to browse the learners who have submitted, or generally see who has or hasn’t submitted their assignment.
Only being able to access these controls from the location of the assignment is an extremely poor user-experience, as the staff have to go and find the ORA every time, especially given the lack of notification that staff grading is necessary.
The Open Responses area of the instructor dashboard improves this, but not by much as you still have to use that dashboard to get to the ORA in course content and find it on the linked page.
These is no way for learners to flag malicious submissions or those suspected of cheating, other than by discussion posts, and as such no way for staff to easily review those flags
Community Opinions
Community feedback on ORAs includes a number of common suggestions, such as:
Greater control of the process by staff
For example, being able to move learners backwards or forwards in the process as needed, for example allowing for resubmission of a staff-graded assessment after submitting a failing submission.
In-line commenting and feedback on longer submissions, rather than being restricted to specific comment boxes at the end.
Commenting access without grading to allow staff to comment on submissions without overriding the entire grading process
Assignment searchability, browsing, and discovery improvements to allow staff to find submissions, rather than having to search by learner or simply be assigned assignments for grading
ORAs cannot be resubmitted or edited once submitted, leading to unfixable mistakes and frustration for learners.
There are no notifications of students awaiting grading for staff, or for learners that their submission has received a grade. Users are expected to check regularly for themselves.
The learner training step is overly complex and frustrating to users, leading to a common recommendation of “Never use this step” due to its overwhelmingly negative impact on course bounce rates and assessment completion rates. This runs entirely counter to the suggestion from research that training learners in effective peer- and self-assessment is important for the impact of these activities.
ORAs are clunky to use due to supporting everything to do with this sort of assessment. A more streamlined option for simple assignments is desired by many users in the community.
Learner responses do not auto-save, and require learners to manually hit save, resulting in many lost assignments.
Commonly, instructors are assigned groups of specific learners to grade. This is not possible through the normal grading interface, and instead requires finding individual learners, and it’s not easy to find out whether those learners have submitted anything yet. The only other way around this is to create a cohort, a content group, and multiple copies of the ORA. This is an extremely clunky way to configure this common activity.
Significant community survey work and research is ongoing at the time of writing that will add a higher level of detail than this summary.
Feature Improvement Recommendations
Though significant work in reviewing this is already underway, and will likely yield more significant feedback and areas of improvement than I can offer, the following improvements seem necessary:
Invest effort into making ORAs more usable and lighter-weight for both staff and learners
Ensure that ample, simple educational material is available on the teaching principles behind ORAs to drive their usage
Provide greater guidance in the UI around how to construct an effective ORA
Fundamentally rework the learner instruction step, or remove it entirely.
Continue to gather feedback and input from the community as to how ORAs can be improved and reworked, particularly from the non-academic, self-paced use case, where ORAs could see more use, but are often neglected.
Focus particularly on improving the staff user experience of ORAs