Allow Multiple Submissions for ORA Until Due Date
We've received a request from UT Austin Computer Science & Data Science, which heavily uses ORAs to allow multiple submissions for an ORA until the due date passes. Another way of putting this is: they would like to automate the process of "deleting a learner database history", which is the only way to allow resubmissions in an ORA, and is a very manual process.
Once a learner submits their assignment for an Open Response Assessment, they can not resubmit it unless a course team member manually deletes their learner database history, or state. This works for MOOCs, but for master's degrees, they are not reviewing the submissions until after the due date and would like students to be able to resubmit their homework before the due date for the ORA has passed, without requiring manual intervention from a course team member (they have 1,000 students and growing, so manual intervention is not scalable).
Here is what UT Austin CS said:
"edX’s peer grading was designed for continually-running MOOCs so when a student submitted, grading would start right away. In a for-credit learning environment, peer grading will not start until after the assignment due date has passed."
Allow students to resubmit their ORA submissions (e.g., file uploads) without manual intervention before the due date of the ORA has passed.
The thought is to essentially expose the “delete learner state” functionality to learners.
As a learner who has already submitted a response to an ORA, I want to be able to resubmit my response before that ORA’s deadline so that I can submit an improved version of my response that replaces my previous submission.
As a learner who has already completed the ORA, I want to know how many resubmit attempts I have.
As course staff, I want to be able to specify if an ORA should allow resubmission and how many resubmits learners are allowed.
Given I am a learner who has already submitted a response to an ORA and that ORA has been enabled for resubmissions, when I view that ORA, then I see how may submit attempts I have (if a max is set) and I see how I can clear my submission
Given I am a learner who has cleared my submission, when I navigate to that ORA, then I see how many resubmit attempts I have and am able to submit a response.
Given I am a learner who has reached the maximum allowed resubmissions (if there is a max set), when I navigate to that ORA, I am notified that I have no submission attempts left and I do not have the means to clear my submission.
Given I am course staff, when I am navigating the ORA settings during creation, then I am able to enable the ORA for resubmissions and if I choose, set the max attempts allowed.
Steps to Reproduce
Reason for Variance
User Impact Summary
Vicki sent over our q’s to UT Austin CS/DS, and provided our answers to his questions. Here are her comments:
In these use cases where you want learners to be able to make multiple submissions, which grading assessment workflows are enabled for these types of submissions? Are these peer-graded ORA, staff graded ORAs, self graded, or some combination?
runs the gamut - TA, peer, auto-graded.
For your ORAs that allows multiple submissions, would you say that grading should not start until the ORA due date has passed?
Some are auto-graded to check if the projects are wildly off. Students submit pieces of project to train the code. Running through XQueue.
Xqueue, Canvas submissions. Google Co Lab. Students submit their project, get autograded, get feedback, and then resubmit in the same area. They would like this to happen somehow in an ORA. 2 TAs for interviews have experience with this - NLP + Reinforcement Learning.
Secondly, they have this interesting use case where they want students to submit their coding projects multiple times so they can train their datasets & so that the student can find out if they are wildly off or not. And then just keep resubmitting. They are using every tool imaginable to do this right now, since ORA doesn't allow for resubmissions but they would love to streamline. I am not quite sure if this is an ORA option or not but wanted to mention and Aidan said 2 of the TAs who we will be interviewing can speak to this.
And I provided your update and Joe had the following comments: might be nice to keep the text but only delete the files. Really wants to know the feasibility of storing more than one solution (what would the scope for this?). He understands we'll be taking an iterative approach and might scope this out later, but it is something they are interested in.
no partner has requested a specific max # of attempts. I asked UT Austin CS/DS specifically and they said they would prefer to start with unlimited attempts and then add in limits later, maybe. So the two phases that Sapana outlined aligns really well with that.
This improvement makes sense but I have a QQ - Did UTA request a limit to the # of resubmissions? I’m curious how a max # of attempts benefits the learner or supports the assignment. If there’s no clear answer to this, then I suggest allowing learners to resubmit until the due date. CC
Hey Mat turmed this into an epic, with tickets to break out the work.
MVP (v1) will allow a partner to specify in the ORA settings if the ORA allows multiple submissions (doesn’t allow them to specify a limit), and allows a student to resubmit numerous times. We’ll release this first and get feedback from partners.
As of now, we have other tickets which can be part of a v2 to add the ability for partners to specify how many resubmission attempts a learner has for the ORA. We’ll evaluate if that is a valuable feature to add after we get partner feedback on the MVP.
We’ll groom the tickets for the MVP tomorrow and we’ll let you know once we have an idea for when it will be ready to release to production. We’ll need to stay in touch about proactive partner comms.
is going to let you and I know by then if there’s something we’d like you to run by them or ask. Thanks for the heads up!