Simple Problem Editor - Idealised Concept
- Matthew Harrington
Analytics
While extensive pedagogic and user research can and should probably be done into what the perfect problem editor is and has, from the above research and experience using the tools, the perfect problem editor has the following capabilities and features. Note that this is specifically defining a “problem editor” as a user interface where authors can construct problems through clickable UI elements and text, rather than a code or markup editor.
Problem Types
The ideal simple editor should support the following answer inputs for custom problem construction:
Multiple Choice
Additional features:
Option order randomisation & position fixing
Image options
Partial credit
Checkboxes
Additional features:
Option order randomisation & position fixing
Scoring mode: All-or-nothing vs Partial credit
Correctness display mode: Total (show overall correctness a single result) or Transparent (show which answers are correct or incorrect)
Image options
Dropdown
Additional features:
Item ordering control (alphabetical, random, fixed)
Partial credit
Numerical input
Additional features:
Basic variable setup, insertion and evaluation
For example: “A rectangle has a length of {a} and a width of {b}. What is its area?” with a numerical input answer of “{a}*{b}”
Tolerance
Precision (exact, significant digits, decimal places)
Short text input
Additional features:
Case sensitivity (default insensitive)
Fuzzy-matching/regexp for acceptable answers
Long text input - Not currently supported
The free text response XBlock provides a basic model for this. Mark correct if the text contains these key phrases, mark partially correct if it includes these key phrases.
Non-graded mode for basic self-reflection questions
Gap-Fill - Not currently supported
Allows the construction of cloze-style questions where the input is in-line with text, using other input types (dropdown, short-text and numerical inputs). This will be the most complex problem type to configure and design, but is extremely valuable pedagogically.
Non-Problem Elements
Additional content is often necessary to frame, contextualise, and explain problems, and for that reason the problem editor should also have the following:
WYSIWYG formatting options
Including insertion of elements such as images, icons, quotes, code blocks, and HTML snippets
Answer Recall
A way of pulling in an answer from a previously answered question by ID (Problem Builder XBlock has this, and it’s extremely useful, and prior answers-as-variables was a request from educator working group).
Listed as a subfeature of the WYSIWYG editor as it’d need to be insertable inline with other content to be usable.
Hints
Optionally available to learners on demand, with a gate on when learners can request hints available in settings.
Feedback
Adaptive to which options were selected by the learner on a per-input set basis.
Custom feedback labels.
Explanation
An overall explanation of why the answer is what it is.
Cross-problem Variables
Variables that can be used by all problems in a unit, for example a randomly generated variable that is used on multiple problems
Generic Settings
Problems should have the following configurable settings regardless of problem type. Defaults in bold. Where a default should be configurable in a separate interface at the course level, these are labelled Default.
Problem title/display name
Text input
Displayed/Hidden
Problem Auto-Numbering
Enabled/Disabled
Based on the order in which the problem appears on the page.
Maximum Attempts
Dropdown? Value, Unlimited, Automatic, Default (Unlimited)
Automatic value could be based on number of inputs -1, but is more of a nice-to-have than anything.
Making unlimited an explicit option is far clearer than current 0/null for unlimited
Penalty Factor
Enabled/Disabled/Default (Disabled)
Number input (%)/Default (10%)
Deduct from the available points for each incorrect submission
Reset Button
Enabled/Disabled/Default (Disabled)
Show Answer
Always, Attempted, Finished, Never, Default (Finished)
Suspect the other values of this are seldom used or useful (data required), so the list should essentially be refined down to those that are most intuitive and useful to avoid overwhelming users.
Show Hints
Always, Attempted, Never, Default
Show Associated Content
Always, Attempted, Never, Default
Associated Content
A unit-selector for linking content back to the unit in which the subject-matter was taught. Useful for both staff to ensure content has been taught, and learners for revision of low-stakes quizzes (see rough mockups below)
Usability and Configuration
It should remain possible to mix-and-match input types in a single editor problem, as this is a valuable differentiator for Open edx, while making it significantly simpler to insert inputs and configure their settings. In addition, the following configuration options would be conceptually useful:
Input set block insertion
Similar to unit components (a concept authors are already familiar with), problem inputs are inserted as blocks/objects, rather than as individual inputs defined by syntax or a single problem being restricted to a single input of a single type.
Per-input set configuration options
For example, option randomization associated with a specific multiple-choice input set rather than the problem as a whole, as well as per-input weighting as a percentage of the problem’s total weight.
General Settings
The only settings that should exist in general settings should be settings that can be applied to every problem type, regardless of inputs.
There should be no integration settings (such as Mathlab API keys) in individual problem settings of the problem editor. These should be kept in an overall course-level problem configuration area along with things like default setting values.
Convert to OLX
Provide access to the advanced XML/OLX editor from the Simple Editor
Allows problems to be started in the problem editor, then moved to the OLX editor.
Should not be labelled “advanced” or similar, but named appropriately as the OLX or XML editor - it is more than feasible to construct basic problems in OLX, and complex problems in the normal problem editor provided it’s fully-featured, and as such it makes no sense to name them simple vs advanced.
By explicitly labelling this feature “convert” it makes it much clearer that this is a conversion, not simply opening it in a different view.
Bi-directional conversion is not practical due to the flexibility of OLX, so while this would be nice, it’s not feasible.
Content library/Problem bank integration
It should be possible to easily push/pull questions from a bank of prebuilt problems specific to the institution to enable content reuse, standardised templates, and general problem consistency.
Rough Mockups
For illustrative purposes only, proper wireframe designs should be created and tested by a skilled UX designer. These are representative of conceptual functionality, not necessarily of a good user experience.
