/
Spec: ORA reminder notifications

Spec: ORA reminder notifications

  • Ready for review

    • This spec has been modified to include a v2 of potential solutions. Problem definition section is slightly modified but Data Analysis remains unchanged.

    Problem

    For ORAs that require self and/or peer grading, a significant share of learners never receive a final grade despite submitting their work. We can think of 3 potential reasons:

    1. Learners do not realize additional steps are required after submission.

    2. Learners defer the peer or self-assessment step and fail to return to complete it.

    3. Learners attempt peer grading before required number of peer submissions are available and forget to circle back.

    This has a negative impact on course completion and therefore learner progress and satisfaction.

    Data analysis

    We examined data from a large Open edX instance, covering tens of thousands of ORA submissions across hundreds of courses in a 30 day span. The findings highlight a significant grading gap, especially around peer review:

    • ~25% of submissions from this period remained ungraded 21 months later.

    • ~58% of ungraded submissions were for ORAs that require peer grading.

    • ~31% of submissions that require peer grading, remain ungraded.

    image-20250923-091939.png
    To-scale distribution of ORA types used in this analysis

     

    image-20250922-061455.png
    image-20250922-064039.png
    image-20250922-063336.png
    image-20250922-064300.png

    Proposed solution - v2 (new)

    Introduce a simple reminder system for ORAs that have required self and/or peer grading steps.

    1. For each ORA that a learner has made a submission for, if the required next step is peer review or self review and ORA due date or course end date has not passed, then

    2. Send up to X reminders, Y hours apart. The first reminder is sent Z hours after submission.

    3. Reminders stop when:

      • Due date for that ORA step or course end date has passed.

      • The learner does NOT have peer review or self review as pending next step.

      • The reminder count for that ORA reaches X. We don’t need to keep track of reminder count. Instead we can use condition:

        • Hours elapsed since submission > X*Y+Z

    Proposed language for peer and self review reminder notification is:

    • You have {pending step} to complete for assessment {ORA title}.

    Pending step could be “self review” or “peer reviews”.

    Optional improvement (if technically feasible): We only send peer review notification if total number of peer submissions available >= required number of peer reviews that are pending.

    The proposed solution uses the submission date as the reference point for the entire notification cycle.

    As a result, some learners may receive a reminder to complete peer review step immediately after finishing self review, or shortly after doing a peer review. This timing is not always ideal from a learner’s perspective.

    We are accepting this limitation in the initial implementation because it significantly simplifies the triggering logic and avoids deep coupling with ORA internal working.

    Later, we can improve this based on qualitative and quantitative feedback.

    A feedback suggested sending a reminder that points learners to the next required step immediately after they submit, to avoid the impression that the workflow is complete.

    We do not propose a new notification for this. Instead, we propose reusing the existing reminder mechanism by setting Z = 0, which causes the reminder to fire immediately after submission.

    This approach achieves the desired result without adding a new notification type that would need separate preferences, template, and maintenance.

    A feedback also suggested notifying learners who are unable to complete the peer review step due to insufficient peer submissions, once submissions become available.

    We agree this is a valid use case, but we plan to address it in a separate spec.

    This use case affects early submitters which is an important but relatively small subset of learners. Addressing it properly requires a deeper understanding of ORA peer review mechanics, including review limits per submission, prioritization logic etc.

    It is likely that this will require a dedicated notification with a relatively complex trigger logic.

    Proposed solutions - v1 (old)

    Solution A: One time reminder

    Each learner that is yet to complete the peer review step gets a one-time reminder, X hours/days after their submission or last review.

    Solution B: Periodic reminders with max count

    Each learner that is yet to complete the peer review step gets a periodic reminders, every X hours/days after their submission or last review. But total count of reminders is capped at Y.

    Solution C: Periodic reminders with unsubscribe option

    Each learner that is yet to complete the peer review step gets a periodic reminders, every X hours/days after their submission or last review. Reminders will continue until course end date or ORA due date. However, learners will have the ability to turn off reminders for each ORA xblock.

    Things to consider

    Things to account for:

    1. Reminders can annoy if:

      1. Received too soon (within mins/hours after submission or grading a peer).

      2. Received too frequently.

    2. Notification for new submission is received but submissions available to grade are less than what are required.

    3. Default preferences for this notification?

    Open questions

    1. Do we need to account for ORA due dates in the notifications logic?

    2. Should we send a reminder if the number of available submissions is less than the required number for completion?

    Other learnings from data

    1. Most learners tend to complete peer review the same day they’ve made their submission. Although some have completed even after months days!

    2. Some paid learners who make their submission but don’t have the required peer submissions available, don’t circle back later for ORAs with due dates.

    Comments

    1. As a learner, i submitted response to an ORA that requires peer grading. Required next step is to grade X peer submissions. Peer submissions available for review at the time of my submission:

    2. For each ORA that needs peer review as the next step:

    Comments