Numerical Input & Formula Problem Comparator Research

Most platforms built for the academic use-case feature this functionality, but it is increasingly absent on platforms intended for the corporate use-case.

In a lot of LMSes, formula questions are also separated from numerical input problems. This does not seem to be led by a conscious design decision, but simply because numerical input problems were added to those platforms later, resulting in a new problem type being easier than extending existing problems.

Moodle

• Moodle has two main question types aimed at numerical input, Calculated and Numerical, with a few others that deal with mathematical questions that have similar configuration options, such as calculated multiple choice.

  • Numerical questions have the following settings of note:

    • Filters - Filters are how Moodle interprets markup to provide a different view to learners, for example converting mathematical notation using MathJax or TeX. The desired filter can be selected for numerical input questions to ensure that the correct interpretation is used and displayed to learners.

    • Units - Units of measurement can be specified for answers, For example, if you enter a unit of 'cm' , and the accepted answer is 15, then the answers '15cm' and '15' are both accepted as correct. You can also specify a multiplier. So, if your main answer was 5500 with unit W, you can also add the unit kW with a multiplier of 0.001. This means that the answers '5500', '5500W' or '5.5kW' would all be marked correct. Note that the accepted error is also multiplied, so an allowed error of 100W would become an error of 0.1kW.

  • Calculated questions support formulae, but otherwise feature the same settings as numerical questions, and as such have the following configuration settings in addition to those above:

    • Variables/wildcards - These are denoted using curled braces, for example {A}*{B}. 

      • Datasets - Each wildcard is assigned a dataset, which is used to generate the values available as wildcards. A dataset can be shared or private, which limits the variable to one specific question or to other questions in the section. This means that questions using a shared dataset can share the same variables across multiple questions. Each variable is configured separately, so a question can have a mixture of shared values and individual values.

      • Values - Each value in the dataset can be manually entered, or generated by Moodle. This allows for a fixed set of random values to pick between, or a fully randomly generated set within a range, with a choice of different distributions.

      • As each wildcard draws from a preset dataset that is limited to 100 items, on-the-fly randomization of these problems appears to be impossible. Instead, even when generating values with Moodle, staff have to pick a set of viable options to put into the dataset.

      • The editing experience for data sets is poor to the point of being noted in the documentation, as well as being discussed regularly in the community.

    • Functions - As the question involves a formula, a list of preset functions such as deg2rad, floor, and sin are defined for use within formulae.

Canvas

• Canvas has a similar divide between their Numeric and Formula questions.

  • Numeric questions have the following capabilities of note:

    • Requirements - The response must satisfy one of the following requirements:

      • Exact response - the response must match the answer exactly

      • Margin of error - the response must be within a +- range of the answer as a percentage or absolute value

      • Within a range - the response must be between two answer numbers

      • Precise response - the response must match the answer and be expressed to the correct number of significant digits or decimal places.

  • Formula questions have the following additional settings:

    • Variables - Similar to Moodle’s wildcards, the variables are defined before the problem formula is written. There is no complex dataset or variable sharing equivalent, you simply define a generation range and number of decimal places for  “X” and then use it in the formula.

    • Generate Possible Solutions - When a formula is given and variables are set, the platform is able to calculate a list of possible responses. This allows instructors to ensure their formula is correct, formatted properly, and is solvable. 

    • Helper Functions - Similarly to Moodle, Canvas has a number of “helper functions” designed to simplify the question creation process

Open

Learnosity

Learnosity is not one of our standard comparator platforms, but is noteworthy here as it is an assessment generation tool designed to integrate into other LMSes.

• Learnosity features a few maths-related question types (most notably Math Questions, structured as a form of Cloze question to insert text entry boxes into equations), but the most interesting tool they offer is the Math Question Generator which generates up to 50 questions for the standard Math question type in a multi-stage process

1. Parameters - these are variable used in the question to generate more questions

• These are added as a list (a comma-separated list of values), or a range (an upper and lower limit to generate between).

Open

2. Expression & Question - First, the author creates an expression to essentially evaluate learner inputs against to get the correct answer:

Then, the question stem itself is created in a WYSIWYG editor which has the option to insert the equation created above, as well as the parameters individually in order to form a question:

3. Correct Answer Format - The tool will then generate some possible answers, and present a variety of them in different formats, along with options for things like decimal places, significant digits, and other similar options based on the formula that was provided:
   
   

4. Generate variations - Variants of the question are generated and can be selected in order to save them for use:
   
   

This is a powerful tool that is not in itself a question type, but a “wizard”-style equation question creator that is unlike anything offered by the typical LMS offerings (Learnosity is an assessment engine platform designed to be integrated into LMSes, so it has a different use-case). While obviously biased, they have an interesting writeup of the requirements for math assessments that is worth reading when designing and constructing tools for maths education.