Course-Level Assessment Configuration

Open edX

The course-level assessment setup for Open edX courses is defined in three main areas of Open edX Studio:

• Settings -> Schedule & Details

• Settings -> Grading

• Settings -> Advanced Settings

Schedule & Details

In schedule & details, there are two (potentially three) configuration options that loosely relate to assessments.

Course End Date affects the behaviour of problems and grading, as graded content cannot be submitted after the course has ended.

Prerequisite Courses are not directly assessment-related, but are rather the result of an assessment, as learners cannot access a course without fulfilling a course milestone, which is typically reliant on earning a certificate in the prerequisite course.

When enabled, this area also holds configuration for enabling an Entrance Exam, which were proposed for deprecation, but the decision was made for them to continue to be supported due to the UI being low effort to maintain. Entrance Exams are graded subsections that a learner must earn a passing grade in prior to being able to access the course.

Grading

The grading area is the primary area where grading and assessments are configured.

Overall Grade Range

The Overall Grade Range feature provides a basic interface for managing grade cutoff points. Users can drag the grade boundaries representing a percentage overall grade, which dictates if a learner passes or fails. In addition to this, users can add letter grades from A-D with their own boundaries, with F being anything below a letter grade for granular passing grades.

Open

This interface is not immediately intuitive; there is no indicator of the fact you can drag the border between cutoffs, or remove cutoff points, or any indication of what will happen when you click the + button. In addition to this, the numbering is inaccurate - in the default configuration, a passing grade is set to 60-100, and a failing grade is set to 0-60. Meaning that a grade of 60 appears to be both a pass and a fail (when in reality it is a failing grade). These labels can be renamed by clicking next to each letter and typing, with the exception of F, which cannot be renamed for some reason.

Open

These custom names and letters no longer appear to be shown or supported in the Learner Progress page MFE, simply showing a flat Pass/Fail grade at the lowest non-F grade regardless of course settings.

Grading Rules & Policies

Despite its description, the Grading Rules & Policies section contains a single property - Grace Period on Deadline.

This sets a uniform grace period for graded content deadlines across the course, which is not something that can be set or overridden at the assignment-level. This grace period is in the HH:MM format (though that is not indicated in the UI); for example the period in the screenshot above is 5 hours.

No other properties exist in this area, suggesting that additional features around “deadlines, requirements, and logistics around grading student work” were once intended to exist in this area, but no such controls exist.

Assignment Types

The last, and perhaps most important, set of settings on this page relates to Assignment Types. Each assignment type has the following properties:

• Assignment Type Name - This is what the assignment type is called, such as “Exam” or “Quiz”, and is fully flexible (so can, for example, be called “Unit 1” or “Cardiology Content” arbitrarily).

• Abbreviation - What the assignment type name was previously abbreviated to on the learners’ progress pages. This no longer appears to be used in the new progress page MFE.

• Weight of Total Grade - How much of the final grade each assignment type is worth as a percentage (entered without the % symbol).

• Total Number - How many of the given assignment type the course is expecting to have.

• Number of Droppable - How many of the lowest-scoring assignments of this type can be ‘dropped’ (excluded from the final grade).

This is one of the areas of highest complexity for new course creators, due to the unintuitive nature of some controls and grading on the platform, for example:

• Weight of total grade requires a number to be used as a percentage, but only the text on page indicates that it is a percentage (there is no pre-filled % symbol or similar)

• Weight of total grade can exceed or be below 100% across all assignment types

  • A course that has assignments weighted below the passing grade is impossible to pass (for example 3 assignments worth 20% on a course with a passing grade of 70%)

• Per assignment-type grading can lead to unexpected grading behaviour, given problems themselves have weight.

  • For example, if two “quiz” assignments contain 10 questions that are displayed as being worth 1 point each, with one quiz worth 75% of the grade and one assignment worth 25% of the grade, each problem in the first quiz will be worth 3 times the amount in the final grade, despite being listed as 1 point.

• There is no requirement for completion of all assignments, only a pure grade.

  • For example, if the passing grade is 60%, and the assignment setup has 4 quizzes worth 20% each, plus a peer graded activity that is also worth 20%, it is possible to pass the course by achieving 100% score on 3 of the quizzes.

  • This is likely a relic of early Open edX having no way of tracking completion other than grade.

• Total number is a value that appears to no longer be used on the new progress MFE, and was never intuitive before it was removed.

• In general, the new progress MFE has been developed to better fit the needs of learners, but the tools available to course creators that inevitably connect to the progress page have simply not been updated.

Anecdotally, the way assignment types are configured typically results in a few models of grading, many of them less-than-ideal:

• As intended

  • Each “type” of assignment is given a weight

  • Example assignment name(s): Homework, Quiz, Final Exam

  • This is most commonly used by academic MOOC creators.

• Flat grading

  • All assignments are assigned to a single assignment type

  • Example assignment name(s): Graded Content

  • This is commonly used by non-academic course creators who don’t really care about grading or weights, and just want all graded content to be worth the same (with weights for more important content being controlled by problem weight).

• Participation grading

  • All gradable formative content throughout the course is assigned to a single assignment type, with individual assignments of higher value given their own assignment type.

  • Example assignment names(s): Participation, Module 1 Quiz, Module 2 Quiz, Final Exam

  • This is commonly used by instructors who want to reward a small number of points for completing formative activities to incentivise learners to complete them (so participation would have a low weight for the amount of content within).

• Granular grading

  • Each section or subsection is given its own assignment type, creating a huge list of assignment types

  • Example assignment name(s): 1.1 Introduction, 1.2 Setup and installation, 1.3 Creating your first project, 1.4…

  • This is commonly used by instructors on high-touch courses who want a more granular view of learner progress. Open edX lacks tools for tracking learner progress other than abuse of the grading system in this way.

Advanced Settings

The final area for course-level setup and configuration is in Advanced Settings, alongside a large laundry-list of semi-homeless and orphaned settings. Advanced Settings is home almost exclusively to features that have not got complete implementations or been properly productised, but have simply been added in an MVP-level state. The following settings in this area apply to grading and the creation of graded content:

• Advanced module list - This is where XBlocks are enabled, which can be graded pieces of content, such as SCORM packages, Peer Instruction content, LTI, or any other XBlock that can return a grade to the LMS.

  • This is configured in JSON, and is generally extremely poor and unintuitive. There are no instructions, and no suggestions of what tools are available on the site.

• Allow opting out of proctored exams - A true/false value that allows learners to bypass proctoring. This setting is always available, even if there is no proctoring integration set up on the platform, even if there are no proctored exams set up in the course. These settings are purely in alphabetical order, and there is no logic that dictates whether they display or not.

• Course Not Graded - A frankly baffling setting that can be true or false and disables all grading-related features in a very unclear way if set to true. It defaults to the double-negative of being not not graded.

• Create Zendesk tickets for suspicious proctored exam attempts - There is virtually no documentation I can find as to what a “suspicious” proctored exam attempt is, and this setting appears regardless of if a site has a Zendesk integration, a proctored exam integration, or anything else related to this feature. This may only even only apply to proctored exams that are sent to RPNow via SoftwareSecure.

• Disable Progress Graph - True or false, it removes the progress graph that no longer exists on the progress page for this course.

• Due dates - Described as the day by which problems are due, I honestly cannot figure out the order of precedence between this, the course end date, and individual subsection and problem due dates.

• Enable LaTeX Compiler - Enables the LaTeX compiler across the entire course so that it can be used on advanced problems and HTML components that use LaTeX.

• Enable proctored exams - Enables proctored exams on this course, despite the fact that proctored exams require a paid integration with proctoring services that have been set up to work with Open edX’s implementation of proctored exams - ProctorTrack and RPNow being the only two documented. Enabling proctored exams enables timed exams even if the setting to enable timed exams (which is found further down the page) is set to false, as proctored exams require timed exams.

• Enable Subsection Prerequisites - Enables course authors to “lock” content behind the completion of another subsection. This completion status can be gated behind component completion progress (the green checkmarks issued by the progress and completion API), grade (points earned in the subsection), or both.

• Enable Timed Exams - This setting allows timed exams to be created, which are subsections that act as distinct exams that can be reset as a single object, and have a time limit on their completion.

• LTI Passports - Provides a way to enter LTI passports at the course level in JSON format to allow the course to have access to LTI content in third-party tools. On other platforms, LTI passport details are commonly stored at the platform-level, but Open edX handles this differently due to the multi-org nature of the platform’s original MOOC use-case on edX.org. This allows for some flexibility at the cost of security (as these passports are contained within course exports).

• Matlab API Key - Provides a way to enter an API key specific to this course for use on maths problems. This feature is completely non-functional and no longer supported by Matlab, and has been accepted for deprecation.

• Maximum attempts - Sets the default maximum number of attempts that learners get on problems in the course, which can be overridden at a per-problem level

• Number of Relative Weeks Due By - Appears to be intended to allow dynamic due dates for self-paced courses, i.e. to set a due date of X weeks based on their self-enrollment. I regard this as having dubious value as a feature, at least at the course level, and currently this only appears to work on edX.org.

• Proctoring Provider - Appears to be intended as a way to select a proctoring provider, except there are only two accepted proctors, and both use the lti_external method. This setting is basically non-functional except as a barrier to using the supported proctoring providers.

• Randomization - Sets how often variables in Python problems should be identified. No explanation for what these values actually mean is displayed on the list. Doesn’t actually do anything if your course does not use randomised Python variables.

• Remote Gradebook - Enables use of a remote gradebook. Only usable when REMOTE_GRADEBOOK_URL is specified, which is not an available setting (likely a site-level configuration). No documentation was found further explaining this feature, and I do not know if it functions.
  
  

• Show Answer - Sets the default “show answer” property for problems. Possible values are  "always", "answered", "attempted", "closed", "finished", "past_due", "correct_or_past_due", "after_all_attempts", "after_all_attempts_or_correct", "attempted_no_past_due", and "never". No further information is given on what these values mean without finding this feature in the documentation.

• Show Calculator - Provides a fairly low-quality calculator tool for learners in the bottom right corner of the LMS.

• Show Reset Button for Learners - Sets the default value for showing the reset button in problems to learners.