[Proposal] Graded Discussions (v1)

View the Github ticket for proposal status updates.

View the Github ticket for proposal status updates.

Overview

It should be possible to grade participation in discussions and factor this grading into a learner’s score for a course.

Problem

Currently, course teams and students use the Open edX Discussion Micro Frontend (MFE) to share ideas, views, and ask questions. However, there is no process for course admins to assign grades to student discussion responses.

The specific pain points include:

Course Admins cannot assign points to learner discussion responses. Enforcing a grade will encourage students to participate in discussions, swap ideas, and interact with their peers.
Course Admins cannot define the criteria of the discussion grade in the discussion settings. Defining the criteria of a discussion grade will encourage how often students participate, as well as the content of their participation.

Note: Course team members in Studio are not automatically granted discussion administration roles. Course Admins need to explicitly grant a team member a role in order to moderate or administer course discussions.

Note: Course team members in Studio are not automatically granted discussion administration roles. Course Admins need to explicitly grant a team member a role in order to moderate or administer course discussions.

Use Cases

As a course administrator, I want to be able to grant permissions to course staff in order for them to better manage graded discussions.
As an instructor, I want to make discussions gradable so that I can verify engagement.
As an instructor, I want to encourage students to participate regularly in thoughtful discussions.
As an instructor, I want to constrain graded discussions between date windows so that I can ensure learners are participating at the same time.
As a course staff member, I want a grading utility for discussions so that I can evaluate learners' contributions expeditiously.
As a learner, I want a rubric for graded discussions so that I know on which dimensions my contributions will be judged.

Proposed Solution

This document outlines a minimal but viable solution for graded discussions. Options for streamlining and automation are minimized, but can be added in subsequent updates.

Technical Approach

Currently, it is only possible to attach grades to XBlocks. We propose to extend grading to other features. This will require a significant reworking of the grading code. However, once completed, not only will graded discussions be possible, but other plugins will be able to leverage this capability to grade things outside of the normal course content paradigm.

Roles

The table below only takes graded discussion functionality in account. See the following links for a more extensive list of the team member discussion roles and permissions:

Administrators can:

Function	Admin	Discussion Admin*	Discussion Moderator*	TA

Function	Admin	Discussion Admin*	Discussion Moderator*	TA
Manage graded discussion posts as per the existing discussion roles	✔	✔	✔	✔
Create and manage graded discussion rubrics	✔	✔	✘	✘
Add / edit grades	✔	✔	✔	✘
Grant graded discussion permissions to existing team members	✔ Can grant permissions to: Staff, Discussion Admins, Discussion Moderators, Group Community TAs & Community TAs	✔ Can grant permissions to: Discussion Moderators, Group Community TAs & Community TAs	✘	✘

Note: Staff must be assigned a Discussion Admin, or Moderator role before being able to manage Discussions or Graded Discussions The Group Community TA has the same permissions as a Community TA, except the Group Community TA can only manage specific discussion groups assigned to them.

Note: Staff must be assigned a Discussion Admin, or Moderator role before being able to manage Discussions or Graded Discussions The Group Community TA has the same permissions as a Community TA, except the Group Community TA can only manage specific discussion groups assigned to them.

Students can:

✔ View the discussion’s possible points
✔ View the grading rubric (how will points be awarded)
✔ View discussion description (how many posts, responses, word count etc)
✔ View the due date
✔ Add a post to a graded discussion

With the above in mind, we need to design and build a feature that will allow non-technical users to manage graded discussions. Graded discussions can be added to:

Units
Course Wide Discussions

Graded Discussions in Studio

Once the Admin has installed the Discussions MFE and configured the “Discussion settings”, we suggest the following process in Studio:

Automatically assign Grading to Graded discussions according to the Subsection Grading Settings:

Note: Currently discussions are disabled on units in graded subsections, by default. A user can enable discussions on units on the discussion configuration page: For the MVP version we suggest keeping this functionality as is, and prompting users to toggle this switch on should they wish to include graded discussions in units.

Note: Currently discussions are disabled on units in graded subsections, by default. A user can enable discussions on units on the discussion configuration page: For the MVP version we suggest keeping this functionality as is, and prompting users to toggle this switch on should they wish to include graded discussions in units.

Once a user has confirmed their Subsection Grading Settings, they can select “Manage discussions” in the Unit’s 3-dot menu:

In a “Manage discussion” modal, or similar, allow users to:

Enable / Disable a discussion on a unit

If Enabled, the user has the option to add the following grading criteria:
(Note: for MVP, we’d like to leverage as much of the Open Response Assessment’s (ORA) UX and UI as possible)

Grading Description (ie. Prompt)
Schedule, eg:
- Discussion start date and time
- Discussion end date and time
Rubric
- This will follow the current ORA rubric setup:
  - Criterion name (eg: Grammar and spelling)
  - Criterion prompt (ie. Description of how to evaluate a discussion post based on this criterion)
  - Options (eg. “Fair”, “Good”, or “Excellent”)
    - Each option has a Name, Explanation and Point value

ORA also accounts for Rubric reuse. This could be leveraged in another phase of the project.

Managing Discussions on the Instructor Dashboard

Each course has its own Instructor Dashboard. We need to ensure there is a suitable space for administrators to assess graded discussions. Discussions will be manually graded for MVP.

Instructors will be able to:

View Dashboard (list view of student’s submissions)
- Include the following items in the list view:
  - Name/Username
  - Date submitted
  - Total Grade
  - Grading Status, as per ORA:
    - “Ungraded” when that submission has not yet received a grade.
    - “Currently being graded by someone else” when another staff member is actively grading that submission.
    - “You are currently grading this response” when you have claimed that submission for grading but have not yet submitted a grade.
    - “Grading Completed” when that submission has received a grade.
  - Checkboxes to allow for viewing and grading multiple submissions

ORA example

View student’s responses (non-editing mode)
Assign grades to student’s responses, eg UI concept:

From a competitor

The grading process should follow the ORA grading process.

The below information has been pulled directly from the edX docs:

When staff are ready to grade they can begin grading with the “Start grading” button. This will open an editable rubric which enables staff to select the score for a submission and provide comments.

The “Start grading” button will also “claim” the current submission for grading. This gives the staff member exclusive access to submit a grade for this submission, helping to avoid grading conflicts. A submission that has been claimed is still visible to all staff but cannot be graded by other staff until:

The grading staff member submits the grade,
The grading staff member releases the lock by clicking the “Stop grading” button, or
After 8 hours elapses (whichever comes first).

While the submission is claimed for grading, staff can select the rubric options for each grading criteria and provide any required feedback before clicking “Submit grade” to submit their grade. Alternatively, if the staff member no longer wishes to grade this response they can click the “Stop grading this response” button that now appears at the top of the page.

After a submission has received a grade, it is still possible to override that grade. Staff can navigate to a previously-graded submission and click “Override grade”. The workflow for submitting a grade override is similar to submitting an initial grade: select the appropriate rubric options and provide feedback before clicking “Submit override”. A modal will ask to confirm if this is the intended action before overriding the grade.

The below information has been pulled directly from the edX docs:

When staff are ready to grade they can begin grading with the “Start grading” button. This will open an editable rubric which enables staff to select the score for a submission and provide comments.

The “Start grading” button will also “claim” the current submission for grading. This gives the staff member exclusive access to submit a grade for this submission, helping to avoid grading conflicts. A submission that has been claimed is still visible to all staff but cannot be graded by other staff until:

The grading staff member submits the grade,
The grading staff member releases the lock by clicking the “Stop grading” button, or
After 8 hours elapses (whichever comes first).

While the submission is claimed for grading, staff can select the rubric options for each grading criteria and provide any required feedback before clicking “Submit grade” to submit their grade. Alternatively, if the staff member no longer wishes to grade this response they can click the “Stop grading this response” button that now appears at the top of the page.

After a submission has received a grade, it is still possible to override that grade. Staff can navigate to a previously-graded submission and click “Override grade”. The workflow for submitting a grade override is similar to submitting an initial grade: select the appropriate rubric options and provide feedback before clicking “Submit override”. A modal will ask to confirm if this is the intended action before overriding the grade.

Therefore Instructors should be able to:

View submissions
Assign grades to each grading criteria
Add comments to each criterion
Override grades
Actions
- Save as Draft
- Cancel
- Submit

Graded Discussions Student View

The Graded Discussions will leverage the current discussion UI in the LMS, but with subtle differences.

Graded discussions, like normal discussions, will be available on:

Units
Course Wide Discussions

On Course Discussions and Unit Pages, discussions should show:

Grading Description (ie. Prompt)
Total points possible
Grading rubric - how will points be awarded
Due date

Graded Discussions concept in a unit (note this is not a final UI and is merely for descriptive purposes)

Students will be able to:

Add / edit / delete their submissions
View / respond other student submissions as they’re posted
(a future phase could include only enabling viewing other submissions once a student has posted their initial requirements)

Graded Discussions remain “open” but we’ll indicate that the grading period has past.

On Course Discussions Page:

Search/filter on Discussions Page by Graded Discussions (in addition to the current functionality)

On the My Progress Page:

Discussions that are graded should show up as a line item in the “Grade Summary”.

Other approaches considered

A previous version of this product spec exists, but has been pared down considerably to this minimum viable plan.

One other approach on the technical side would be to not open up the grading code, and instead keep using XBlock-based grading. This was rejected after heavy consideration for the following reasons:

The discussion block is deprecated in favor of topics being auto-created. We would have to bring back a discussion block to track graded discussions, which is counter to the authoring experience we’re trying to build.
We COULD create some kind of hidden XBlock that is abstracted away. But then we’d need to carve out some special casing for XBlocks to allow them to be hidden from course authors. This seems like a distasteful approach, especially since no other block is provided such a privilege and it is an open question when else such a feature would (or should) be used.
Opening up the grading code has a lot of potential for extending the platform’s grading capabilities. Discussions are only the start-- other outside plugins could allow for grading without tying grades back to a particular XBlock. This solution offers a lot of potential in the future while avoiding the special casing in item 2.

Competitive Research

Yellowdig

From their documentation: Different actions and responses to your posted content in Yellowdig can earn you Yellowdig points if they are activated for your community. The Community Owner or Facilitators can configure rules for earning points. The rules can be seen at any time by clicking on the "Show details" section below the main points display (shown below).

Populi

Populi “attaches” Graded Discussions to an assignment. From their documentation:

As an assignment, they can be graded and factored into your students' final grade for your course.
As a discussion, they operate just like a regular discussion, but with additional options for grading student participation.

How do you create a Graded Discussion on Populi?

Add Assignment, and then:

Select “Discussion” under Type
Choose “New discussion” or “Existing (non-graded) discussion”
Choose a rubric for the discussion

Populi UI

When you first create a graded discussion in Populi:

It isn't published; it's in Draft mode
It doesn't have a topic
It doesn't have any grading requirements

Users need to go to the Discussion’s page to adjust its settings and add grading requirements:

Canvas

How do you create a Graded Discussion on Canvas?

A user needs to:

[1] Enter a topic title
[2] Discussion details
Then the user needs to select the Graded checkbox

After that the user sets the following criteria:
- [1] Enter the number of points possible
- [2] Select the Grading Type
- [3] Assignment Group
- [4] Assign Peer Reviews
Finally the user assigns students and sets the due dates
Once published this is what the student sees:

Blackboard

How do you create a Graded Discussion on Blackboard?

Choose a grade unit
Choose a grade category
Assign a rubric (Add existing or Create new)

Plan for long-term ownership/maintainership

We expect several teams to need this functionality and several core contributors from OpenCraft will be working on it. Once live, these core contributors will continue assisting with maintenance. If needed, one or more point persons can be assigned who will be the main point of contact for this code upon release.

Open questions for rollout/releases

Is there any reason why these features should not be enabled by default once available in a named release?
Previous solutions for ‘graded discussions’ exist by integrating other discussion providers, such as Yellowdig. Do we need to do anything to better incorporate the data from these existing solutions, such as finding a way to count their scores toward discussion progress on the dashboard, or are these safe to ignore?

Open edX Product Management