DRAFT Assessment Core Capability [Ideal Product Vision]
Context:
The Assessment Capability is one of the Core Capabilities that will be supported in the Open edX Core Product Strategy.
Note: Much of the input for this document is driven by feedback from the Educator’s Working Group. Raw data available here.
Value Proposition: What makes Open edX Core Assessment capabilities unique?
What comes with the Core Assessment Capability?
What comes with the Marketplace?
What comes with the DIY Environment?
Key areas of investment to get us to “ideal”
Value Proposition: What makes Open edX Core Assessment capabilities unique compared to other LMS’?
Grounded in pedagogical integrity and Interactive Pedagogy principles. The assessment capabilities in the core are designed to optimize best practices in online pedagogy, with instructional design principles, documentation and templates built into workflows for both course teams and learners.
The only LMS on the market that includes an all-in-one self-, peer-, and graded assessment tool for short- and long-form responses, with options for customizing for either MOOC or high-touch environments. Includes templates and documentation for learners and educators to get the most out of designing and participating in peer- and self-review.
Problem banks. Create your own, or use templates from a community-supported bank, which includes unique, subject-specific problems not available with any other LMS.
Raw problem editor. For DIYers, build your own custom problem types.
What comes with the Core Assessment Capability?
All the features and functionality you need to create, configure, deliver and grade formative and summative assessments and assignments.
Feature overview: [excel view here]
Feature | Description | Tablestakes OR Unique Value Proposition | Core Product Design Alignment | Needs investment? |
Single and multi-select problem type (auto-graded) | Auto-graded problems. Comprehensive set of “out of the box” question types and interactive tools that enable robust and comprehensive formative and summative assessment. Includes in-platform instructional design guidance and templates. All problem types have an intuitive WYSIWYG editing interface. | tablestakes | Core Product |
|
Numeric and text input (auto-graded) | Auto-graded problems. Comprehensive set of “out of the box” question types and interactive tools that enable robust and comprehensive formative and summative assessment. Includes in-platform instructional design guidance and templates. All problem types have an intuitive WYSIWYG editing interface. | tablestakes | Core Product |
|
Dropdown (auto-graded) | Auto-graded problems. Comprehensive set of “out of the box” question types and interactive tools that enable robust and comprehensive formative and summative assessment. Includes in-platform instructional design guidance and templates. All problem types have an intuitive WYSIWYG editing interface. | tablestakes | Core Product |
|
Dropdown modified for cloze tests (auto-graded) | Auto-graded problems. Comprehensive set of “out of the box” question types and interactive tools that enable robust and comprehensive formative and summative assessment. Includes in-platform instructional design guidance and templates. All problem types have an intuitive WYSIWYG editing interface. | tablestakes | Core Product | yes - needs enhancements |
Fill in the blank (auto-graded) | Auto-graded problems. Comprehensive set of “out of the box” question types and interactive tools that enable robust and comprehensive formative and summative assessment. Includes in-platform instructional design guidance and templates. All problem types have an intuitive WYSIWYG editing interface. | tablestakes | Core Product |
|
Drag and Drop (auto-graded) | Auto-graded problems. Comprehensive set of “out of the box” question types and interactive tools that enable robust and comprehensive formative and summative assessment. Includes in-platform instructional design guidance and templates. All problem types have an intuitive WYSIWYG editing interface. | tablestakes | Core Product | yes - needs enhancements |
Matching (auto-graded) | Auto-graded problems. Comprehensive set of “out of the box” question types and interactive tools that enable robust and comprehensive formative and summative assessment. Includes in-platform instructional design guidance and templates. All problem types have an intuitive WYSIWYG editing interface. | tablestakes | Core Product | yes - doesn't exist |
Ordering (auto-graded) | Auto-graded problems. Comprehensive set of “out of the box” question types and interactive tools that enable robust and comprehensive formative and summative assessment. Includes in-platform instructional design guidance and templates. All problem types have an intuitive WYSIWYG editing interface. | tablestakes | Core Product | yes - doesn't exist |
Image Mapping (auto-graded) | Auto-graded problems. Comprehensive set of “out of the box” question types and interactive tools that enable robust and comprehensive formative and summative assessment. Includes in-platform instructional design guidance and templates. All problem types have an intuitive WYSIWYG editing interface. | tablestakes | Core Product | yes - needs enhancements |
Flashcards (ungraded) | Ungraded interactives. Comprehensive set of “out of the box” question types and interactive tools that enable robust and comprehensive formative and summative assessment. Includes in-platform instructional design guidance and templates. All problem types have an intuitive WYSIWYG editing interface. | tablestakes | Core Product | yes - doesn't exist |
Polling (ungraded) | Ungraded interactives. Comprehensive set of “out of the box” question types and interactive tools that enable robust and comprehensive formative and summative assessment. Includes in-platform instructional design guidance and templates. All problem types have an intuitive WYSIWYG editing interface. | tablestakes | Core Product | yes - needs enhancements |
Problem editor (all WYSIWYG) | WSYIWYG Editing interface. | tablestakes | Core Product |
|
Problem bank (content library) | Problem bank creation. Use content libraries to create banks or collections of problems that can be tagged for subject, topic, skill, learning outcome, etc. | unique value proposition | Core Product | yes - needs investment |
Assignment/assessment creation (all WYSIWYG) | Assignment/assessment creation and configuration. Create and configure distinct assignment or assessment types, such as quizzes and exams. | tablestakes | Core Product | yes - needs investment/doesn't exist |
Answer randomization tool | Enhanced delivery presentation options to foster self-reflection and self-regulated learning, and to mitigate cheating. | unique value proposition | Core Product | yes - needs enhancements |
Content randomization tool | Enhanced delivery presentation options to foster self-reflection and self-regulated learning, and to mitigate cheating. | unique value proposition | Core Product | yes - needs enhancements |
Hints | Enhanced delivery presentation options to foster self-reflection and self-regulated learning, and to mitigate cheating. | tablestakes | Core Product |
|
Feedback | Enhanced delivery presentation options to foster self-reflection and self-regulated learning, and to mitigate cheating. | tablestakes | Core Product |
|
Explanation | Enhanced delivery presentation options to foster self-reflection and self-regulated learning, and to mitigate cheating. | tablestakes | Core Product |
|
Assocaited Content Links | Enhanced delivery presentation options to foster self-reflection and self-regulated learning, and to mitigate cheating. | tablestakes | Core Product | yes - doesn't exist |
Instructional Design Helper | In-platform instructional design feature to help course teams leverage best practices in instructional design in creating questions and assessments, covers both summative and formative assessment | unique value proposition | Core Product | yes - doesn't exist |
ORA Lite for MOOCs | All-in-one self-, peer-, and graded- assessment tools for short- and long-form responses (ORA). Includes templates and documentation for learners and educators to get the most out of designing and participating in peer- and self-review. | unique value proposition | Bundle | yes - doesn't exist |
ORA for Campus | All-in-one self-, peer-, and graded- assessment tools for short- and long-form responses (ORA). Includes templates and documentation for learners and educators to get the most out of designing and participating in peer- and self-review. | unique value proposition | Bundle | yes - needs enhancements |
Exam Proctoring | Supported integration points - Proctored exams LTIs (curated shortlist) | unique value proposition | Core Product/Marketplace | yes - doesn't exist |
Plagiarism Tools | Supported integration points - Plagiarism tools (curated shortlist) | unique value proposition | Core Product/Marketplace | yes - doesn't exist |
Community Problem Bank | A bank of specialized problem templates supporting a range of subject areas, particularly for teaching of advanced STEM concepts, from chemistry and circuits to quantum computing. These may require specialized knowledge (python, latex), but all come with well-documented templates. The idea is to start this collection with a curated selection of MIT templates and grow it into a community-supported, ever-growing bank of unique problem templates. Starter set of templates include: Chemical equation writer Circuit builder Conditional/dependency grader Python Grader Library Math expression input MathJax Problem written in Latex Wordcloud | unique value proposition | Marketplace | yes - doesn't exist |
Raw Problem Editor | Build your own problem types and contribute them to the community-supported problem bank. | unique value proposition | DIY |
|
Key areas of investment to get us to the “ideal”
Pedagogical improvements to the “out of the box” problem types, with a focus on fostering interactivity and better formative assessment
Improvements to current problem types
Enhancements to drag and drop
Partial credit grading
Use a draggable more than once
Improve/overhaul dropdown so it meets cloze question uses cases
Create new “out of the box” problem types
New question/assignment type for short answers (community request)
Matching (market parity)
Ordering (market parity)
Create UIs/”out of the box” experience for a subset of high-value “advanced” problem types, particularly ones that foster interactivity/formative assessment
Polling/Surveys (need to decide on one version and depr others)
Visual assessment like Peer instruction
Image mapped input
Feature-ize an in-platform instructional design tool for content authors that can provide quick tip guidance on when and how to use problems, delivery tools etc, links to templates for best practices.
Improvements to ORAs along three pathways:
Current ORA feature is used primarily in instructor-led courses, and most interest for enhancements are driven by high-touch needs. Rally around this use case and invest heavily in ORA for this use case. Laundry list of enhancements here.
Develop a lightweight ORA experience designed for MOOCs/asynch learning, focused on self-assessment, peer-assessment, and/or even AI assessment. Ex, short answer questions that can be used for 1-min self-reflections
Investments in teaching learners how to most effectively do self- and peer-assessment, why it’s important/what the learning outcome is, and in teaching educators how to most effectively utilize these
Feature-ize an assessment/assignment creator tool, with more obvious “create a quiz”, “create an exam” UX and all the requisite configuration/setting options in one place
Feature-ize the “bank of specialized problems” by creating templates and better documentation on 1) When to use these problems/which learning objectives they support and 2) How to use these problems.
Areas of investment for interactivity:
Graded Discussions as a means of fostering interactive, formative assessment environments
“Push the envelope” features to explore:
Short-response question type for instructor-less MOOCs graded by AI/some other auto-grading mechanism