Multiple Choice - Ideal Functionality

The ideal multiple choice problem isn’t far from what currently exists. The user experience of a multiple choice question is so fundamentally understood that on the learner side, we should ensure as much as possible that the problem functions no differently to how the user would expect, which is more or less how they function currently.

However, the ideal multiple choice has invisible features available for authors to create effective problems without the learner ever being aware of the existence of those extra features. These include:

• Answer shuffling - Presenting the learner with a randomised but consistent option order to avoid accidental bias from the instructor (such as always making option B the correct one, or having worse distractors towards the bottom of the list).

  • Answer fixing - Where answers are shuffled, some should be locked in place (for example “all of the above”, while not always good practice, should always appear at the bottom)

• Answer pools - Providing the learner with a subset of options from a list of distractors (so of the 10 answers available, the learner sees a subset of 3).

• Variable insertion and calculation - To enable mathematical and programming problems to be randomised

• Partial credit - Allowing learners to achieve a reduced number of points for certain partially-correct answers.

• Question-level and answer-level feedback - Providing learners with explanations of why answers are right or wrong, and optionally guiding them towards the correct answer on their next attempt, at both the overall question (explanations) and individual answer level (feedback).

• Hints - Providing learners with on-demand access to information that makes formative problems easier to answer ahead of answering.

• Multiple correct answers - Allowing staff flexibility in how their problems are structured, and inspire thought by forcing a learner to decide between multiple correct options.

These are all basic features currently supported in Open edX, but they are not all possible in the simple editor, and those that are are unintuitive to use at best. All of these features should be easily accessible through a UI, and be far more intuitive to implement than is currently possible.

Less easily defined are improvements that could exist to make the authoring of multiple choice questions as easy as possible, including:

• Consideration of multi- and single- select problems as a single MCQ entity - Currently, checkboxes/multi-select, dropdowns, and multiple choice/single-select problems are considered entirely separate, when in fact they are widely considered simply different forms of multiple choice. Whether the current consideration of these problems is suitable is extremely debatable.

• Improved tools for guiding course authors towards robust assessments - Creating assessments is a skill that not many master, and is easy to get wrong, which leads many less-skilled course creators to things like multiple choice quizzes with obvious distractors, as that is the easiest solution, but seldom the best. Technology can assist course authors with improving this, particularly with the advent of generative AI tools, though to what extent is currently unknown. Conceivably, however, the ideal multiple choice authoring tool could include:

  • Suggestions for plausible distractors - This would remove the difficulty of distractor construction from authors, making multiple choice problems far more robust.

  • Automatic question creation based on course content - This would allow formative assessments to be quickly generated, which is the best use of multiple choice questioning.

  • Suggestions on how to rework problems to use other assessment types - Where a multiple choice problem is not the best solution, suggest a better option and explain why.

  • Best practice suggestions - The documentation lists a wide range of known best practices for MCQs. Testing questions against some of the more measurable criteria in this list could enable authors to rework their problems using suggestions to improve their suitability, such as rewording to make answers more concise, additions of words to the question stem, and rephrasing of double negatives.

Overall, however, and on a more realistic and practical level, the biggest thing that needs to happen for better multiple choice questions is simply implementing all of the features available to multiple choice questions in the Open edX platform in the user interface of the simple problem editor in such a way as to much the process quick and simple. That’ll take us 80% of the way there. For a more full laundry list of what the ideal simple problem editor needs, see the Simple Problem Editor analysis.