| Note |
|---|
This document is a work in progress and has not been approved. |
| Table of Contents |
|---|
Background
A major challenge with the edX platform today is that developers do not have a documented way to add new features to the web or mobile client. Worse still, there is no way to introduce such changes without modifying the edx-platform code base and then also updating the mobile clients. The only primary exception is for features that can be purely implemented as XBlocks as they can be installed into the platform with minimal code changes. See Extending edX Platform for the documented ways in which developers can extend the platform today.
...
There are five major ways to introduce a new feature to the edX platform
...
- Limitations:
- XBlocks can currently only provide courseware components.
...
, in approximate priority order:
- Create a new XBlock in its own Git repository
- XBlocks allows you to define new types of components that can be used in courseware.
- For edX to consume your XBlock, it should just be added as a requirement to: https://github.com/gsehubedx/xblock-mentoring
- Update the platform directly
- Limitations:
- The feature now becomes a built-in part of the platform, and must be reviewed as such.
- Examples:
- Cohorted Courseware: PR 6299
- Limitations:
- New Django app in edx-platfom repo
- Limitations:
- The feature must be reviewed to be merged into the platform, and currently most of the static assets cannot live in the app itself
- Note: edX is moving towards having more of the assets live in the app. See: How to add a new feature to LMS or Studio
- Limitations:
- New Django app in its own Git repo
- Limitations:
- The only UX integration available is XBlocks which limits the functionality to courseware components.
- Examples:ORA 2edx-platform/blob/master/requirements/edx/github.txt
- Consider the following before choosing to create an XBlock:
- Testing of authoring views is challenging as there is no straightforward way to run with the block embedded in Studio
- There are aspects of XBlock integration into Studio that have not been completed, e.g.
- Examples of XBlocks:
- Recommended XBlock: https://github.com/pmitros/RecommenderXBlock
- Staff Graded Assignment XBlock: https://github.com/mitodl/edx-sga
- XBlock Poll: https://github.com/open-craft/xblock-poll
- Limitations:
- Create a new IDA (independently deployable application)
- If a feature doesn't need to be tightly integrated into Studio or the LMS then it can be built as an independent app
- It can make use of the edX Platform API to get any platform data that it requires
- IDAs are a good choice when:
- They need to scale independently of the edX platform
- They will be built, deployed and managed by a separate team
- LMS and Studio should continue unaffected even if the application goes down
- Consider the following before choosing to make a new IDA:
- There is more operational complexity for the community to deploy and manage your feature
- There are performance implications with introducing extra network calls
- There is more application complexity when working with related data from external APIs, e.g. joins and referential integrity must be implemented in the application layer
- Most features today need a tighter integration with the edX platform
- Examples of IDAs:
- edX Analytics Dashboard: https://github.com/edx/edx-analytics-dashboard
- edX eCommerce Service: https://github.com/edx/edx-ora2ecommerce
- edX NotificationsStudent Notes API: https://github.com/edx/edx-notifications
- New Independently deployable application (IDA)
- Limitations:
- Most features today need a tighter integration with the edX platform
- Examples:
- edX Analytics Dashboard: https://github.com/edx/edx-analytics-dashboard
- Limitations:
...
- Create a new Django app in its own Git repository
- New Django apps can be implemented in a separate repo and pip installed into the platform
- A Django app in its own repository is a good choice when:
- it provides new services that will be consumed by other parts of the platform
- if it requires new UI for LMS or Studio, it provides it via extensions and does not introduce new URLs
- See Web Fragments for the recommended way to add pluggable user interfaces
- Consider the following:
- If the feature has no UI then it may be better suited as an IDA if it doesn't require direct access to platform data models
- See Managing multiple repo development for descriptions of some of the challenges involved with this approach
- Examples of Django apps in their own repositories:
- ORA 2: https://github.com/edx/edx-ora2
- Submissions: https://github.com/edx/edx-submissions
- Milestones: https://github.com/edx/edx-milestones
- Add a new Django app to the edX platform
- If your feature cannot be built using the existing extension points, then it can be added directly to the edx-platform
- Consider the following:
- It would be better to introduce a platform extension point and then use it from a separate app
- It is best to build your new Django app in its own directory so that there is an option to move it into its own repo later
- Our current thinking is to add your new Django app to the directory
openedx/features
- Our current thinking is to add your new Django app to the directory
- See Web Fragments for the recommended way to add pluggable user interfaces
- Examples of Django apps in the edX platform:
- Update the platform directly
- This is the option of last resort
- When should you update the platform directly?
- When the feature you are implementing updates core functionality in fundamental ways
- When there are no extension points that you can use to build the functionality you need
- Consider the following before doing this:
- Features which update the platform directly require the most scrutiny in code review
- There is far more risk of unexpected fallout from your changes
- Consider whether it is possible to introduce a platform extension point and then use it
- The new code adds to the complexity of the monolith
- Be sure to consult with edX
- Examples:
However, even for options 2-4, the new capabilities need to be added to hard-coded lists in the platform. For example, there is a list of installed XBlocks, a list of URLs, a list of tab types, a list of reports, a list of instructor dashboard components etc. This seems simple to update, but it means that the platform has to be changed to receive each new feature.
...
Here is a quick summary of how four recent several large features were implemented. For more details, see the Appendix: Recent Feature Audit.
...
CCX introduced some core platform changes along with a new CCX Django app:
A new CCX role was added to the platform
The instructor dashboard was updated to understand this new role
There were pieces that had to be hard-coded into the platform which would more naturally have lived in the Django app:
New configuration settings were added to lms/envs/common.py
New course settings were added to InheritanceMixin in common/lib/xmodule/xmodule/modulestore/inheritance.py
A new "CCX" tab was added to common/lib/xmodule/xmodule/tabs.py
- The URL for the new tab was added to
lms/urls.py - Mako templates for the new tab were added to
lms/templates - CSS for the new tab was added to
lms/static/sass - The JavaScript Backbone views were added to
lms/static/js
Tests were added for the new UI
- Jasmine tests were added to lms/static/js/spec
- Jasmine HTML fixtures were added to lms/static/js/fixtures
- Bok Choy tests were added to
common/tests/acceptance
2. Cohorted Courseware
Cohorted Courseware was a platform-wide PR so a substantial portion of the code was inevitably in the platform itself:
A new pluggable user partition scheme was added allowing any Django app to add new group types
- A cohorted scheme was added to support "Content Groups"
The new "Content Group" concept was added into the pre-existing "Group Configurations" view in Studio
Cohort information was added to the "Profile Information" CSV report
There were pieces that would have been cleaner if they had been implemented as plug-ins:
A new "Edit Visibility" modal was added to every xblock in Studio via a new mixin.
- This would have been better as a pluggable tab on the existing editor.
A new "Cohorts" tab was hard-coded into the Instructor Dashboard
- The URL for the new tab was added to
lms/urls.py - Mako templates for the new tab were added to
lms/templates - CSS for the new tab was added to
lms/static/sass
Tests were added for the new UI
- The JavaScript Backbone views were added to
lms/static/js - Jasmine tests were added to lms/static/js/spec
- Bok Choy tests were added to
common/tests/acceptance
3. ORA 2
https://github.com/edx/edx-ora2
ORA 2 plugs cleanly into the platform as an xblock:
This works seamlessly to introduce a new courseware component
Only very minor changes had to be made to edx-platform
- Referencing the current ORA 2 release commit hash in
requirements/edx/github.txt - Adding the xblock to the list of advanced problem types used by Studio
Living outside of the platform had challenges:
There is no way to extend the platform SASS, so instead independent SASS was written which is included in addition. This won't play well with theming.
XBlocks do not support RequireJS
- This made it hard to ensure that the block has the dependencies it needs
- It also required that a custom mechanism be used to modularize the code into smaller files
Studio authoring view is not truly integrated for xblocks
- ORA 2 had to fake out the Save and Cancel buttons as Studio can only save changes to xmodules today
The only option for admin views is through the Studio authoring view or through staff-level features added to the courseware block
- Adding admin views inside an xblock is powerful but constrained by the space provided
- It could be better to have links from the xblock to a full-page admin view (maybe as a tab on the instructor dashboard)
- It would be useful to add ORA 2 reports to the "Data Downloads" tab
There was no easy way to write integration tests that get run with the platform release
- Separate sandbox was provisioned for automated tests but it stopped working and has been disabled
4. Student Notes
Student Notes was built as a mostly independent Django app living in the LMS:
The Django app lives here: https://github.com/edx/edx-platform/tree/master/lms/djangoapps/edxnotes
A new Notes Service was introduced that is independently deployable
There were many aspects that were not able to live in the edxnotes Django app:
New configuration settings were added to lms/envs/common.py
New course settings were added to InheritanceMixin in common/lib/xmodule/xmodule/modulestore/inheritance.py
The core HTML component was updated to render notes inline
The AnnotatorJS library was added as a vendor library to common/static/js/vendor
A new LMS "Notes" tab was added to common/lib/xmodule/xmodule/tabs.py
- The URL for the new tab was added to
lms/urls.py - Mako templates for the new tab were added to
lms/templates - CSS for the new tab was added to
lms/static/sass - The JavaScript Backbone views were added to
lms/static/js
Tests were written for all the new UI
- Jasmine tests were added to lms/static/js/spec
- Jasmine HTML fixtures were added to lms/static/js/fixtures
- Bok Choy tests were added to
common/tests/acceptance
5. Proctored Exams
Proctored exams were implemented as a standalone Python library along with some fairly substantial integration into edx-platform
The Python library lives here: https://github.com/edx/edx-proctoring
Some notes:
- Proctored exams were not implemented as a new type of XBlock, but instead were hard-wired into the built-in sequential module
- New fields were added directly to sequential module
- https://github.com/edx/edx-platform/blob/master/common/lib/xmodule/xmodule/seq_module.py#L70
- The proctoring code was provided as an XBlock runtime service to be called out to from the sequential module
- Conditional logic was added when rendering sequential modules to call out to the proctored exam rendering code
- Course-scoped settings were added directly to the course module
- New configuration settings were added to both
lms.env.jsonandlms.auth.json - Custom Studio UI was added to manage the new settings for sequentials
- Studio was extended to include custom registration of proctored exams:
- The course outline was updated to have custom rendering for proctored exams
- URLs were added directly to the LMS's urls.py file
- All of the styling for proctored exams was added directly into the LMS's main courseware Sass file
- A new tab for 'special exams' was added to the instructor dashboard
...
There are a number of changes that were needed to support the ability for a new feature to be implemented purely within its own Django app:
Stevedore extension points should be provided for each of the common UX changes that need to be made
- See: How to add a new plugin entry point to edX platform
- Note: see the Audit below for a comprehensive list
The platform needs to be extended to support pluggable user interfaces
| Status | ||||
|---|---|---|---|---|
|
- The Web Fragments library provides support for building view fragments in Django
- The edX platform has now started to use web fragments to support building course tabs
- OEP-12: Pluggable User Interfaces goes into more details
XBlocks should be extended to provide more capabilities
- Course-scoped fields for configuration settings
Jira Legacy server JIRA (openedx.atlassian.net) serverId 13fd1930-5608-3aac-a5dd-21b934d3a4b4 key TNL-1804
- Ability to add an xblock as a course tab
Jira Legacy server JIRA (openedx.atlassian.net) serverId 13fd1930-5608-3aac-a5dd-21b934d3a4b4 key TNL-2319
- Studio editing tabs for xblocks
Jira Legacy server JIRA (openedx.atlassian.net) serverId 13fd1930-5608-3aac-a5dd-21b934d3a4b4 key TNL-850 Jira Legacy server JIRA (openedx.atlassian.net) serverId 13fd1930-5608-3aac-a5dd-21b934d3a4b4 key TNL-851
- Admin views for both Studio and LMS
- Need built-in support for RequireJS or AMD compliant dependency management
Jira Legacy server JIRA (openedx.atlassian.net) serverId 13fd1930-5608-3aac-a5dd-21b934d3a4b4 key PLAT-481
The Django asset pipeline needs to be improved
| Status | ||||
|---|---|---|---|---|
|
- Have the pipeline pick up assets from all installed Django apps (to support CDNs, minification etc)
- A spike was done to prove the approach: PR 7610: Django extension spike
- The work was completed as part of the Teams epic:
Jira Legacy server JIRA (openedx.atlassian.net) serverId 13fd1930-5608-3aac-a5dd-21b934d3a4b4 key TNL-2242
- RequireJS Optimizer support should be added to the LMS
Jira Legacy server JIRA (openedx.atlassian.net) serverId 13fd1930-5608-3aac-a5dd-21b934d3a4b4 key TNL-1322 - See also RequireJS in the LMS
There are further complications if we want to move the Django app out to its own repo:
How do we write integration tests?
How can SASS build upon the base SASS from edx-platform?
| Anchor | ||||
|---|---|---|---|---|
|
...
| Change | Description | Update Required | Mobile Support | Cohorted Courseware | ORA 2 | Student Notes | CCX | Proctored Exams |
|---|---|---|---|---|---|---|---|---|
| Configuration Settings | Allow the feature to be enabled or disabled | Update lms/envs/common.py (or cms) | Not picked up by mobile today |  |  | | | |
| URLs | Most features require new LMS and/or Studio URLs | Update lms/urls.py (or cms) | Not picked up by mobile today | | | | | |
| Django models | Add new persisted data models, or update existing ones | Add models to your Django app, or update existing ones in the platform | | | | | | |
| Modulestore logic | Changes needed in core modulestore logic | Update xmodule/modulestore | | | | | | |
| Courseware component | Add a new component type that an author can add to their course. | Implement an xblock (some caveats that still require xmodule). Update the Studio configuration to allow the user to create such a component. | (with some caveats) | | | | | |
| Course-scoped fields | Add a new field that can be set per-course. | Add fields to xmodule/course_module.py and sometimes | | | | | | |
| User partition scheme | Provide a new way to partition users into groups/teams/cohorts etc | Provide a plug-in of type openedx.user_partition.scheme | | | | | | |
| Images | Images need to get into the pipeline to end up served through the CDN | Add new images to lms/static/images (or cms) | | | | | | |
| CSS | Introduce new styling for custom components | Add new SASS files to lms/static/sass (or cms) | (for web views) | |  (see ORA section below) | | | |
| Mako Template | Include new HTML templates to be returned when serving a new page | Add new templates to lms/templates (or cms) | | | | | | |
| Third-party JS libraries | Add new libraries needed by the new JS UI, e.g. AnnotatorJS | Add the files to Add the new dependencies to RequireJS and the Jasmine tests | | | | | | |
| JavaScript | Add new custom JS code to support more dynamic browser interactions | Add new JS files to lms/static/js (or cms). Add new dependencies to | | | | | | |
| Unit Tests | Add new Python or Jasmine code to test your new feature | Add new tests to your Django app Update | | | | | | |
| Integration Tests | Add new Bok Choy tests that verify your feature works when running inside the platform | Add new tests to Note: ORA 2 has Bok Choy tests in its own repo, but no easy way to run them | | | | | | |
See Also
PR 7610: Django extension spike
Thoughts on LMS front end architecture
How to add a new feature to LMS or Studio