Feature Plugins for edX Platform

This document is a work in progress and has not been approved.

Background

A major challenge with the edX platform today is that developers do not have a documented way to add new features to the web or mobile client. Worse still, there is no way to introduce such changes without modifying the edx-platform code base and then also updating the mobile clients. The only exception is for features that can be purely implemented as XBlocks as they can be installed into the platform with minimal code changes. See Extending edX Platform for the documented ways in which developers can extend the platform today.

This is also a major roadblock to the community providing new features. If a contributor has a feature that they wish to share, then they have to make a pull request against the edX platform. This needs to be carefully reviewed by edX across all of the dimensions that we are concerned with (i18n, accessibility, performance, security, theming, test coverage etc). This is often too high a hurdle for other organizations to want to take on.

Another challenge is that it is impossible for an operational team running Open edX to add or remove features, because each feature only works with a version of the platform which contains its UI. This requires that a fork of edX be created and the requisite platform changes carefully merged into it so that the set of desired features can all be used at once.

This document describes the steps required today to introduce new features, and then describes possible approaches to introduce a documented plug-in interface to satisfy a majority of such use cases. The goal of a plug-in interface is that features can be installed into a pre-existing edX platform and will start working with no code changes and minimal configuration. Then, if a feature is no longer needed it can simply be uninstalled and the platform will continue running without code changes. This would allow the Open edX community to develop and share Python packages that provide useful capabilities that edX does not necessarily want to incorporate into the platform. It also helps edX development too, as each developer can have experimental features that they install or remove without ending up with multiple forks of the platform.

Adding Features Today

There are five major ways to introduce a new feature to the edX platform

New XBlock
- Limitations:
  - XBlocks can currently only provide courseware components.
- Examples:
  - Mentoring XBlock: https://github.com/gsehub/xblock-mentoring
Update the platform directly
- Limitations:
  - The feature now becomes a built-in part of the platform, and must be reviewed as such.
- Examples:
  - Cohorted Courseware: PR 6299
New Django app in edx-platfom repo
- Limitations:
  - The feature must be reviewed to be merged into the platform, and currently most of the static assets cannot live in the app itself
  - Note: edX is moving towards having more of the assets live in the app. See: How to add a new feature to LMS or Studio
- Examples:
  - CCX: PR 6610
  - Student Notes: PR 6610
New Django app in its own Git repo
- Limitations:
  - The only UX integration available is XBlocks which limits the functionality to courseware components.
- Examples:
  - ORA 2: https://github.com/edx/edx-ora2
  - edX Notifications: https://github.com/edx/edx-notifications
New Independently deployable application (IDA)
- Limitations:
  - Most features today need a tighter integration with the edX platform
- Examples:
  - edX Analytics Dashboard: https://github.com/edx/edx-analytics-dashboard

However, for options 1-4, the new capabilities need to be added to hard-coded lists in the platform. For example, there is a list of installed XBlocks, a list of URLs, a list of tab types, a list of reports, a list of instructor dashboard components etc. This seems simple to update, but it means that the platform has to be changed to receive each new feature.

Use Cases

Here is a quick summary of how four recent features were implemented. For more details, see the Appendix: Recent Feature Audit.

1. CCX

PR 6636

CCX introduced some core platform changes along with a new CCX Django app:

A new CCX role was added to the platform
The instructor dashboard was updated to understand this new role

There were pieces that had to be hard-coded into the platform which would more naturally have lived in the Django app:

New configuration settings were added to lms/envs/common.py
New course settings were added to InheritanceMixin in common/lib/xmodule/xmodule/modulestore/inheritance.py
A new "CCX" tab was added to common/lib/xmodule/xmodule/tabs.py
- The URL for the new tab was added to lms/urls.py
- Mako templates for the new tab were added to lms/templates
- CSS for the new tab was added to lms/static/sass
- The JavaScript Backbone views were added to lms/static/js
Tests were added for the new UI
- Jasmine tests were added to lms/static/js/spec
- Jasmine HTML fixtures were added to lms/static/js/fixtures
- Bok Choy tests were added to common/tests/acceptance

2. Cohorted Courseware

PR 6299

Cohorted Courseware was a platform-wide PR so a substantial portion of the code was inevitably in the platform itself:

A new pluggable user partition scheme was added allowing any Django app to add new group types
- A cohorted scheme was added to support "Content Groups"
The new "Content Group" concept was added into the pre-existing "Group Configurations" view in Studio
Cohort information was added to the "Profile Information" CSV report

There were pieces that would have been cleaner if they had been implemented as plug-ins:

A new "Edit Visibility" modal was added to every xblock in Studio via a new mixin.
- This would have been better as a pluggable tab on the existing editor.
A new "Cohorts" tab was hard-coded into the Instructor Dashboard
- The URL for the new tab was added to lms/urls.py
- Mako templates for the new tab were added to lms/templates
- CSS for the new tab was added to lms/static/sass
Tests were added for the new UI
- The JavaScript Backbone views were added to lms/static/js
- Jasmine tests were added to lms/static/js/spec
- Bok Choy tests were added to common/tests/acceptance

3. ORA 2

https://github.com/edx/edx-ora2

ORA 2 plugs cleanly into the platform as an xblock:

This works seamlessly to introduce a new courseware component
Only very minor changes had to be made to edx-platform
- Referencing the current ORA 2 release commit hash in requirements/edx/github.txt
- Adding the xblock to the list of advanced problem types used by Studio

Living outside of the platform had challenges:

There is no way to extend the platform SASS, so instead independent SASS was written which is included in addition. This won't play well with theming.
XBlocks do not support RequireJS
- This made it hard to ensure that the block has the dependencies it needs
- It also required that a custom mechanism be used to modularize the code into smaller files
Studio authoring view is not truly integrated for xblocks
- ORA 2 had to fake out the Save and Cancel buttons as Studio can only save changes to xmodules today
The only option for admin views is through the Studio authoring view or through staff-level features added to the courseware block
- Adding admin views inside an xblock is powerful but constrained by the space provided
  - It could be better to have links from the xblock to a full-page admin view (maybe as a tab on the instructor dashboard)
- It would be useful to add ORA 2 reports to the "Data Downloads" tab
There was no easy way to write integration tests that get run with the platform release
- Separate sandbox was provisioned for automated tests but it stopped working and has been disabled

4. Student Notes

PR 6610

Student Notes was built as a mostly independent Django app living in the LMS:

The Django app lives here: https://github.com/edx/edx-platform/tree/master/lms/djangoapps/edxnotes
A new Notes Service was introduced that is independently deployable

There were many aspects that were not able to live in the edxnotes Django app:

New configuration settings were added to lms/envs/common.py
New course settings were added to InheritanceMixin in common/lib/xmodule/xmodule/modulestore/inheritance.py
The core HTML component was updated to render notes inline
The AnnotatorJS library was added as a vendor library to common/static/js/vendor
A new LMS "Notes" tab was added to common/lib/xmodule/xmodule/tabs.py
- The URL for the new tab was added to lms/urls.py
- Mako templates for the new tab were added to lms/templates
- CSS for the new tab was added to lms/static/sass
- The JavaScript Backbone views were added to lms/static/js
Tests were written for all the new UI
- Jasmine tests were added to lms/static/js/spec
- Jasmine HTML fixtures were added to lms/static/js/fixtures
- Bok Choy tests were added to common/tests/acceptance

5. Proctored Exams

Proctored exams were implemented as a standalone Python library along with some fairly substantial integration into edx-platform

The Python library lives here: https://github.com/edx/edx-proctoring

Some notes:

Proctored exams were not implemented as a new type of XBlock, but instead were hard-wired into the built-in sequential module
New fields were added directly to sequential module
- https://github.com/edx/edx-platform/blob/master/common/lib/xmodule/xmodule/seq_module.py#L70
The proctoring code was provided as an XBlock runtime service to be called out to from the sequential module
- https://github.com/edx/edx-platform/blob/master/common/lib/xmodule/xmodule/seq_module.py#L432
Conditional logic was added when rendering sequential modules to call out to the proctored exam rendering code
- https://github.com/edx/edx-platform/blob/master/common/lib/xmodule/xmodule/seq_module.py#L482
Course-scoped settings were added directly to the course module
- e.g. https://github.com/edx/edx-platform/blob/master/common/lib/xmodule/xmodule/course_module.py#L733
New configuration settings were added to both lms.env.json and lms.auth.json
Custom Studio UI was added to manage the new settings for sequentials
Studio was extended to include custom registration of proctored exams:
- https://github.com/edx/edx-platform/blob/master/cms/djangoapps/contentstore/proctoring.py
The course outline was updated to have custom rendering for proctored exams
- https://github.com/edx/edx-platform/blob/master/cms/templates/js/course-outline.underscore#L56
URLs were added directly to the LMS's urls.py file
- https://github.com/edx/edx-platform/blob/master/lms/urls.py#L928
All of the styling for proctored exams was added directly into the LMS's main courseware Sass file
- https://github.com/edx/edx-platform/blob/master/lms/static/sass/course/courseware/_courseware.scss#L171
A new tab for 'special exams' was added to the instructor dashboard
- https://github.com/edx/edx-platform/blob/master/lms/templates/instructor/instructor_dashboard_2/special_exams.html

6. Student Data Tab

Here is an upcoming use case from Braden MacDonald:

We have a custom XBlock used throughout certain Harvard courses, and their instructors want to export all the student data associated with those XBlocks (stored in edx-submissions). So we want to be able to add an instructor-only tab to the course in the LMS, where we can put a new UI for instructors to generate .CSV exports of that data.

Plug-in Architecture

There are a number of changes that were needed to support the ability for a new feature to be implemented purely within its own Django app:

Stevedore extension points should be provided for each of the common UX changes that need to be made
- See: How to add a new plugin entry point to edX platform
- Note: see the Audit below for a comprehensive list
The platform needs to be extended to support pluggable user interfacesDONE
- The Web Fragments library provides support for building view fragments in Django
- The edX platform has now started to use web fragments to support building course tabs
- OEP-12: Pluggable User Interfaces goes into more details
XBlocks should be extended to provide more capabilities
- Course-scoped fields for configuration settings
  - TNL-1804 - Getting issue details... STATUS
- Ability to add an xblock as a course tab
  - TNL-2319 - Getting issue details... STATUS
- Studio editing tabs for xblocks
  - TNL-850 - Getting issue details... STATUS
  - TNL-851 - Getting issue details... STATUS
- Admin views for both Studio and LMS
- Need built-in support for RequireJS or AMD compliant dependency management
  - PLAT-481 - Getting issue details... STATUS
The Django asset pipeline needs to be improved DONE
- Have the pipeline pick up assets from all installed Django apps (to support CDNs, minification etc)
  - A spike was done to prove the approach: PR 7610: Django extension spike
  - The work was completed as part of the Teams epic:
    - TNL-2242 - Getting issue details... STATUS
- RequireJS Optimizer support should be added to the LMS
  - TNL-1322 - Getting issue details... STATUS
  - See also RequireJS in the LMS

There are further complications if we want to move the Django app out to its own repo:

How do we write integration tests?
How can SASS build upon the base SASS from edx-platform?

Appendix: Recent Feature Audit

The following is an audit of four recent features (CCX, Cohorted Courseware, ORA 2 and Student Notes) which enumerates all of the ways in which they extend the platform. The hope is that this gives a fairly representative sample of the types of changes that might need to be pluggable.

Here are the various types of UX changes that were seen in the relevant features. Note that the "Mobile Support" column describes whether such a change will automatically be supported by mobile, or whether it will require additional changes to the mobile app.

UX Change	Description	Examples	Mobile Support
Feature Flag	Allow the feature to be enabled or disabled	`ENABLED_EDXNOTES` flag
Service	Introduce a new service that other parts of the platform can consume	edX Notes Service edX Submissions Service	(but clients need to be updated)
REST API	Introduce new REST APIs that allow new data models to be consumed by other parts of the platform	User API, edX Notes Service API	(but clients need to be updated)
Core behavior change	Change how core parts of the platform function	Cohorted content, CCX due dates
New course role	Introduce a new role for users within a course	CCX coach
New group type	Introduce a new kind of grouping of users	Cohorts, Content Groups
New LMS page	Add a new page to the top level of the LMS	Learner profile page
New Studio page	Add a new page to the top level of Studio	"Group Configurations" page	N/A
Update Studio course settings view	Update the "Settings" view to allow editing of new features	Course Licenses
Courseware component	Add a new component type that an author can add to their course	"Peer Assessment" component	(with some caveats)
LMS tab	There are two different classes of tabs: those that are shown per role, and those that are shown when a feature is enabled. Note: these are currently shown as tabs in the web version of the LMS, but this concept should be generalized to work for mobile and future non-tab implementations.	"Notes" tab, "CCX" tab
Instructor admin component	Add a new component that instructors can use to administer their course in the LMS	"Cohorts" tab
Instructor reports	Add a new report that instructors can download	"Download Profile Information"
Course feature flag	Allow the feature to be enabled per course (by author or instructor)	"Enable Student Notes", "Enable Cohorts"
Studio advanced setting	Add a new setting that authors can use to configure aspects of their course	"Student Notes Visibility"	N/A
Studio authoring view	Add a new view to allow authors to visually configure aspects of an xblock	"Edit Visibility", ORA 2 settings	N/A
Studio type	Introducing a new type requires providing an editor component	"Language" picker	N/A

Here are the types of implementation changes that had to be made to introduce the above features:

Change	Description	Update Required	Mobile Support	ORA 2
Configuration Settings	Allow the feature to be enabled or disabled	Update `lms/envs/common.py` (or cms)	Not picked up by mobile today
URLs	Most features require new LMS and/or Studio URLs	Update `lms/urls.py` (or cms)	Not picked up by mobile today
Django models	Add new persisted data models, or update existing ones	Add models to your Django app, or update existing ones in the platform
Modulestore logic	Changes needed in core modulestore logic	Update `xmodule/modulestore`
Courseware component	Add a new component type that an author can add to their course.	Implement an xblock (some caveats that still require xmodule). Update the Studio configuration to allow the user to create such a component.	(with some caveats)
Course-scoped fields	Add a new field that can be set per-course.	Add fields to `xmodule/course_module.py and sometimes xmodule/modulestore/inheritance.py`
User partition scheme	Provide a new way to partition users into groups/teams/cohorts etc	Provide a plug-in of type `openedx.user_partition.scheme`
Images	Images need to get into the pipeline to end up served through the CDN	Add new images to `lms/static/images` (or cms)
CSS	Introduce new styling for custom components	Add new SASS files to `lms/static/sass` (or cms)	(for web views)	(see ORA section below)
Mako Template	Include new HTML templates to be returned when serving a new page	Add new templates to `lms/templates` (or cms)
Third-party JS libraries	Add new libraries needed by the new JS UI, e.g. AnnotatorJS	Add the files to `common/static/js/vendor` Add the new dependencies to RequireJS and the Jasmine tests
JavaScript	Add new custom JS code to support more dynamic browser interactions	Add new JS files to `lms/static/js` (or cms). Add new dependencies to
Unit Tests	Add new Python or Jasmine code to test your new feature	Add new tests to your Django app Update `paver` to know about the new directories to run the tests, for coverage etc
Integration Tests	Add new Bok Choy tests that verify your feature works when running inside the platform	Add new tests to `common/test/acceptance` Note: ORA 2 has Bok Choy tests in its own repo, but no easy way to run them