This is a detailed design document on the Course Blocks API and how it enables xBlocks on Mobile.

Goal

To provide a REST interface for accessing course structure and course blocks for a user.

Background

The edX MOOC platform provides an online learning experience for students via scheduled and self-paced courses.  Educators create courses by either using edX's Studio (a GUI-based course-authoring tool) and/or directly manipulating XML files in edX's published OLX format.  Learners interface with the platform using an LMS (Learning Management System).  The LMS is also used by educators to administer and track live versions of courses.

Course

An edX Course encapsulates the various components that make up a learner's experience in a run of a course (session).  For example, it includes a discussion forum for students and educators to interact, course announcements to update students, a handouts corner for reference information, and other tabular components as customized by the course author.  In the context of this document, these are all peripheral components of a course and not considered further.  Instead, we focus on the courseware, which provides a structural learning pathway, and henceforth simply referred to as the course.

A Course structure is a hierarchical Recursive Composition of primitive and composite objects known as xBlocks. The course structure is a mutable directed acyclic graph (why?) with nodes representing navigational/structural elements (such as sections and subsections), dynamic user-targeted course pathways (such as split test modules), and pedagogical primitives (such as instructional content/videos and student assessments). The course structure is designed to be flexible to support an arbitrary depth of navigation elements and an arbitrary breadth of pathways.

Each edit of the course is uniquely versioned whenever a course author modifies the course structure or any of its components.  A production-level course would typically have anywhere from about 150 to 3000 xBlocks.  A typical MOOC-sized student enrollment could be anywhere from 1,000 to 100,000 students.

xBlock

An xBlock is the basic building block of any edX course.  It controls its own data structure (model), its own HTML rendering (view), and its own business logic (controller).  An xBlock executes in a runtime, which provides common services/utilities, user/request context, and storage.  While the course structure data is primarily stored in a database infrastucture called modulestore, an xBlock's persistence is configured on a field-by-field basis by designating the scope for each of its fields.  Some scopes support user-specific data, which results in different xBlock content for different target users.

An xBlock has a unique identifier known as block-id.  Its instantiation within a context of a course is identified by its usage-id.  A common class of xBlocks have the same defined type (or category) that indicate its common behavior and interface.  Some examples of xBlock types are: "course_info" (shared by Handouts and Announcements), "problem" (shared by all CAPA-based assessments), "video", "chapter" (a.k.a Section), and "sequential" (a.k.a. Subsection).

Access Control

Currently, access control checks are not governed by the xBlock, but by a higher-layer authorization framework in LMS.  A user's access to an xBlock varies depending on (a) their role, (b) their group membership, and (c) their individual settings and history.  Here are a few examples:

• Role-based Access
  
  • "Global staff" users can bypass any restrictions on an xBlock in any course.
  • "Course staff" users can bypass any restrictions on an xBlock within their own courses.
  • "Beta users" are exempted from the date restrictions on an xBlock.
  • "Students" cannot access xBlocks marked with visible_to_staff_only.
• Group Membership Pathways
  • Accessing an xBlock that is restricted to a cohort requires membership in that cohort.
  • Accessing a pathway that is restricted to an A/B test group requires membership in that group.
• Individual Differentials
  • A "student" that has been given an Individual Due Date Extension on an xBlock can bypass the due date requirement on the xBlock.
  • An xBlock that is gated by a milestone cannot be accessed by a "student" until the student fulfills the required milestone.

Views

Currently, an xBlock controls its own HTML rendering by returning HTML fragments in its implementations of "student_view", "studio_view", and/or "author_view" methods.  These HTML fragments are currently embedded inside LMS and Studio HTML pages and not independently viewed.

Requirements

Technical Requirements

Here are the basic technical requirements:

• FUTURE (for Course Navigation): Although the course structure supports an arbitrary navigational depth, the client UI should be able to enforce a maximum navigational depth.  This allows for a clear and consistent user experience through the course, if/when the UX chooses to do so.
• The payload needs to be user-specific, while honoring roles, group pathways, and individual differentials.

• It should be possible to delegate the rendering of an xBlock to the client by providing the xBlock's raw data (e.g., in JSON).  This allows us to support:

  • a user experience that is native to the device.

  • a different design of the xBlock's (student) view.
• It should be possible to render the student view of an individual xBlock without the envelope of the LMS chrome.
• APIs should follow the edX REST API Conventions.
• The response payload for a Course Blocks API should be consistent with the existing Course Structure API that was reviewed and approved by the Architecture Council.

Non-Requirements

• Generalizing the endpoint to allow the consumer to request the value of any arbitrary field of an xBlock.

Use Cases

1. Mobile Apps

The initial primary clients of these new API endpoints are the edX Mobile apps as we embark on supporting interactive assessments on iOS and Android devices.  Our initial goals are to evolve the mobile app's simple-to-use video player to provide context and interactivity by adding further course structural context and basic assessment support.

The currently proposed UI requires presenting the following elements to the user:

• • A course dashboard with links to various components of the course, including forums, announcements, handouts, and the courseware.
    
  • A course outline page listing the 2 top-level navigational components (Sections and Subsections) of the course structure.
  • A course subsection page showing the 3rd navigational component (Unit) and all primitive xBlocks within each Unit.
  • Aggregate information including (a) the total count of videos in a Section of the course and (b) whether a Section contains graded components.
  • A web-hosted rendering of an individual primitive xBlock through a mobile WebView runtime on the device.
  • A native rendering of all xBlocks of type video.
    

2. Custom LMS Front-End

We can also imagine an ecosystem with other/extended front-end LMS subsystems that are optimized for different user experiences.  For example:

• • An open source contributor creates a native implementation of an assessment type, which exploits the device's accelerometer to engage the user. 
  • An LMS view chooses to display the course blocks vertically in a Continuous Scrolling pattern.
  • An Adaptive Learning application that dynamically adjusts the student's navigation through the course.

3. Displaying Individual xBlocks

The capability to render an individual xBlock can enable many different features.  For example:

• • An Adaptive Learning tool that refreshes a student's memory by presenting the same assessment at a future date given aggregate analytical data on memory loss.
  • A Flashcard application that renders individual xBlocks of a new type called flashcard.

4. Reuse and Performance Optimization for other LMS Features

Assuming that the new endpoints have been optimized for performance, they can be reused for other features in the edX Platform.  For example, the Bookmarks feature may want to quickly query information about a user's perception of an xBlock.

Related APIs

• The Mobile Video Outlines API will be deprecated by this new API.  The current one is limited to returning only video modules in a course and was designed specifically for mobile's use.  Additionally, it intrusively retrieves the video module's data by directly accessing the video's fields.
  
  
• The existing Course Structure API is an instructor-facing API and not intended for use by a student-facing client.  It requires course staff access and returns all xBlocks in the course, disregarding any role-based, group-pathway, or individual differentiation.  This proposal keeps this endpoint as is, but proposes renaming it as described below.

API Design

This section describes the overall design approach for the following set of APIs:

1. A new generic Courses API as a residence for course-related endpoints.
2. A new Course Blocks API that provides necessary data for creating courseware navigation and interaction.  This API is parameterized to support both user-specific and not-user-specific responses.
3. A new xBlock URL that supports rendering of an individual xBlock. 

Generic APIs versus Mobile-specific APIs

First, a note on our overall approach for designing endpoints used by our mobile apps.  As we design APIs for mobile consumption, we try to abide by the following design constraints:

1. Authentication: The API must support OAuth2 as an authentication mechanism as the mobile apps use the Bearer authentication scheme to transmit the openedX-issued OAuth2 access token.
2. Authorization: The API must enforce server-side access control checks for all data access.
3. Client/server division of labor: In general, business logic should be centralized on the server thereby allowing thinner clients and future maintainability.
4. Performance:
   • Response Time: The 95% response time must be within 2 seconds (at most) for a healthy server and a responsive user experience.
   • Round Trips: A responsive global access to the API requires optimizing network latency by minimizing the number of API round trips.  For example, displaying a single screen on the mobile app should not require more than a single API call.  As a trade-off with payload size, pre-fetching and client-side caching of the data is also considered.
   • Payload Size: In addition to minimizing round trips, the size of the payload should be frugal to optimize network and cellular bandwidth.  So we design APIs that allow parameterizing which fields are returned.

While critical to the mobile experience, the above constraints can also be met by generic APIs that can be used by other (non-mobile) clients.  So going forward, we aim at creating generic APIs that can be consumed by both mobile and non-mobile clients.  However, if additional customized optimization is needed that is inherently mobile-specific, then we will implement a mobile-specific API as a layer above the generic one.

Courses API

 Having a multitude of endpoints and API resources for each type of data for a course can be both hard to maintain and difficult to understand for API consumers.  Instead, we have designed a top-level resource for courses, as follows.  The API is now documented on read-the-docs.

Note: The current Course Structure API would be renamed as below.

Block Rendering: Web, Responsive, and Native

As mentioned above, xBlocks currently manage the HTML rendering of their own data.  This work enhances that framework by introducing 2 additional viewing capabilities of an xBlock.  The API then provides freedom to the client on how to render the xBlock.

xBlock Views

View @supports(multi_device) decorator

Multi-device support is defined here as the functionality that provides (1) responsive web layouts, (2) support for touch-based inputs, and (3) interactive state management (hover, focus, etc) for various types and sized devices.

xBlock developers decorate the views that have multi-device support using a newly implemented @supports method decorator.  The existence of this decorator on the view indicate to the client whether the view's HTML has multi-device support.

class AnXBlockThatHasMultiDeviceSupport(XBlock):
	"""
	An xBlock whose student_view method returns HTML that is responsive to the user's device size.
	"""
	@XBlock.supports("multi_device")
	def student_view(self, context)
		"""
		This view's CSS/JS implementation has multi-device support, providing responsive and touch-sensitive HTML.
		"""
		pass

New student_view_data method

An xBlock that wants to enable other developers to implement their own view of the xBlock's data can implement a student_view_data method.  The semantics of the student_view_data method is analogous to the current student_view method on xBlocks, but it returns a JSON representation of the xBlock's data rather than a pre-formatted HTML rendering of the data.  This functionality supports native implementations of an xBlock.

class AnXBlockThatSupportsCustomizedStudentViews(XBlock):
	"""
	An xBlock that supports custom student views by implementing a student_view_json method.
	"""
	
	def student_view_data(self, context):
    	"""
    	Returns a JSON representation of the student_view of this block.
    	The contract of the JSON format should be published so it can be shared with the client.

		Arguments:
			context - is a JSON dict passed in by the client as request parameters to this block.
    	"""
		pass

Note: Consumers of this data will expect the same JSON representation from blocks of the same type.  So the following is recommended:

1. Block types should be namespaced so there is no naming conflict between classes of xBlocks.
2. JSON representation of a block type should be published and standardized, similar to the OLX standard, so there is a common standard for all xBlock developers and consumers.
3. A block's JSON representation should be properly versioned for future compatibility.

View Options in the API

With the above viewing enhancements for an xBlock, there are now various choices for viewing an xBlock within the context of a course.  These comprehensive choices are provided to consumers of the Course Blocks API for their discretionary use, in the following fields.

Option 1: student_view_data: Native

If the client requests the JSON data for an xBlock type and that xBlock implements the student_view_data method, then the entire JSON data returned by this method is provided.  The client can then choose to natively render the xBlock for a more enriching user experience.  

If native rendering is not supported by either the xBlock or the client, then the client can fallback to one of the other options below.

Option2: student_view_url + student_view_multi_device: Responsive Web View

The student_view_url field provides a URL to render the individual xBlock without the envelope of the web LMS chrome.  The client can choose to use the student_view_multi_device property of the xBlock to determine whether or not to direct its users to the student_view_url. 

For example, the upcoming version of the mobile app will present the individual xBlock by navigating to the student_view_url in a Web View but only if the xBlock has student_view_multi_device.  Otherwise, it requires the user to fallback to the lms_web_url option below.

An alternative implementation could allow users (after warning them) to access the student_view_url even if the block does not have student_view_multi_device.

Option 3: lms_web_url: Fallback Web Browser

If all else fails, the lms_web_url provides a direct link to the navigational container of the xBlock on the web LMS.  The client can choose to present this option to the user as a fallback experience.

Course Blocks API

The crux of this proposal is the introduction of the Course Blocks endpoint that supports student-facing data for presenting course block content.  This section provides design considerations and then details of the endpoint for a user-specific request.  The API is now documented at read-the-docs.

FUTURE: Why Course Navigation?

We will have FUTURE support for course navigation once it is designed.

This proposal decouples the navigation of a course from the rest of the blocks in the course by considering Navigation as a formatting strategy that is applied to the course blocks.  In future extensions, one can envision different navigational strategies including flattening or deepening of the course structure or supporting a choose-your-own-adventure experience.

Additionally, in order to support clients that present a limited N-tier hierarchy of the course structure (e.g., Section->Subsection->Unit), the Navigation strategy can provide this information.  The server-side provides the logic of collapsing the DAG into an N-tiered hierarchy, allowing the client to be a generic consumer of this strategy.

No Pagination?

We made a conscious decision to not paginate this endpoint (for now) although its data could be potentially large, for the following reasons:

• In order to compute aggregate data such as the total number of blocks within a container block, a full traversal of the course hierarchy is required anyway.
• Overall load on the server is not decreased by pagination.  In fact, unnecessary pagination increases the load on the server since there is an overhead for each request.  This is alleviated, however, if the response is cached on the server and subsequent page requests simply retrieve the previously cached data.

Mobile Considerations

Here are the points of consideration for each item listed in the mobile design constraints section above.

Authentication

The API requires OAuth2 or Session-based authentication to identify the user of the request.  Unauthenticated (guest) users are not allowed access. 

A 401 error is returned if authentication fails.

Authorization

The API makes the following access checks:

• Does the user have access to the course?  Specifically, the following states are verified and a 404 is returned if any of them fail.
  
  • user is enrolled in the course
  • mobile available flag is set on the course
  • course has started
  • there are no prerequisite courses or entrance exams that haven't been fulfilled by this user
  • or the user's privilege allows bypassing of any of the above
• For each Block traversed, the following is verified and the block is excluded when not accessible by the user.
  
  • Does the user have access to the Block?  Specifically, the following states are verified:
    • the Block is not visible_to_staff_only
    • the Block has been released
    • the cohort affiliation of the Block matches the requested user's cohort
    • there are no unfulfilled milestones for this Block for this user
    • or the user's privilege allows bypassing of any of the above
  • If the Block is an A/B Test type, only the assigned child for the requested user is traversed
  • The hide_from_toc field is honored when composing the list of descendants

When there's an access control failure, the response payload will contain information about the specific error as described in has_access interface design.  It includes an error_code, developer_message, and a user_message.

Client/Server Separation of Concerns

• The server retains the business logic of collapsing the navigational items according to the client's desired navigational depth.
• The client retains the freedom of choosing between the various rendering options of an xBlock.

Performance

Design

The API was designed with performance constraints in mind. 

• It is optimized for reducing the number of round trips by:
  • including all the blocks in the course in the payload
  • having an endpoint that combines the block and navigation information
• It allows managing the payload size by providing optional parameters to specify which fields to return.
• The response time is optimized as described in Course Block Transformers.

Resources

Block

Video Block

The following fields are associated with a block of type "video" and thus returned in the student_view_data value for a video block type.

Encoded Video

Endpoints

BLOCKS: /api/courses/v1/blocks/?root=<usage_id> 

FUTURE: NAVIGATION

Example of API call from Mobile Apps

Here is the expected API call that the mobile app makes on the blocks endpoint:

/api/courses/v1/blocks?root=<usage_id>/?user=<user_name>&block_counts=video&student_view_data=video&student_view_data.video.context=mobile_low&fields=graded,format,student_view_multi_device

XBlock URL

The following new xBlock URL supports HTML rendering of an individual xBlock without the chrome envelope of the LMS courseware.  This URL is the value for the student_view_url API field mentioned earlier.

/xblock/{usage_id}?view=student_view

Summary

• The objective is to design a generic set of APIs that can be used by various clients, including the mobile apps.  If further mobile-specific optimizations are later required, we can create a thin layer above for the mobile-specific ones.
• The generic Courses API sets a foundation as an envelope for course-related APIs.  It supports accessing both user-specific and system-wide course data.  The current CourseStructure API used by Insights will be renamed accordingly.
• The set of APIs proposed in this document provide the ability to create a custom student view of the courseware.  The edX mobile apps will be the first consumer of this functionality.
• FUTURE (for Course Navigation): We will have FUTURE support for course navigation once it is designed.  The decoupling of course blocks and course navigation allows for customization of navigational strategies.  Although the depth of a course hierarchy is arbitrary, the Navigation API allows consumers to create a simplified navigation UI by specifying a maximum navigation depth.
• The Blocks API, when a user is specified, provides individualized data of the course blocks and structure, taking into consideration the user's roles, group affiliations, and user-specific differentiations.
• The new xBlock URL allows a chromeless rendering of an individual xBlock.
• The proposed enhancements to xBlocks (student_view_data and the @supports("multi_device") decorator) allow additional student views for xBlocks.