Context

Programs are a structured way to group edX course offerings together, to serve learning objectives or instructional goals that span multiple courses.

Programs will be used as the technical foundation to deliver XSeries, an edx.org-specific product offering planned for early 2016. It is our intention to implement Programs as a generic and reusable capability in the edX Platform, so it can be leveraged in support of related use cases both by edx.org and the wider Open edX community. However, our main priority for Q1-Q2 2016 is to develop the functionality necessary to deliver the XSeries MVP.

This document focuses on a key new piece of infrastructure encapsulating the data, logic, and API upon which the new feature will be built, and how it will integrate with key services in the platform to support basic use cases.      

Architecture Overview

Program-related functionality will be implemented in a self-contained, independently-deployable web application. Related support will be implemented in the form of smaller changes to the LMS, Studio, and other participating platform applications.

Modularity is emphasized as a general principle; dependencies in related systems will be kept to a minimum and enabled/disabled via feature flags.  This is intended to maximize operational flexibility when choosing to introduce the Programs functionality within an existing installation.

The new application will serve as the "source of truth" in edx-platform for the Programs themselves, as well as Learners' associations with those Programs (aka Registrations). These data will be exposed as resources via RESTful HTTP APIs. In addition, the application will host client-side JavaScript applications as user-facing frontends to these APIs.

We plan to implement these features using Django / DRF / SQL backend and Backbone.js frontend stacks, keeping consistent with other platform applications.

Data Model

Entities

Program

The Program represents the collection of multiple Course Codes in an ordered sequence.  Each Course Code in turn collects Course Run Modes; learners must complete one course run mode for each course code in order to satisfy the completion requirement of the program. 

Attributes:

• name: a user-facing name for the Program, which must be unique.
• subtitle: a brief, descriptive subtitle for the Program (text only).
• category: a string which can be used to organize Programs for presentation / classification. For XSeries-based programs, the category will be 'xseries'.
• status: an enum indicating the current lifecycle state of the Program.  This is discussed further below in the "Program Lifecycle" section. 

Organization

This entity is a reference to an organization as known to the platform.  It maps to the first component in an LMS course_key, and exists in this model in order to enforce relational integrity at the physical (database) level.

Attributes:

• key: the "org" part occurring in course keys, for example, "ANUx" in "course-v1:ANUx+ASTRO2X+2B3T2015".
• display_name: a user-facing display name for the organization, for example "Australian National University"

CourseCode

This entity represents the notion of a certain course offering independent of run.  This concept does not exist in edx-platform, but can generally be inferred by comparing the first two components (org and course) in course keys; for example two course runs with course keys "course-v1:ANUx+ASTRO2X+2B3T2015" and "course-v1:ANUx+ASTRO2X+2B4T2015" would be associated with a single Course Code whose key is "ASTRO2X" and is associated with an Organization whose key is "ANUx".

• key: the "course" part occurring in course keys, for example, "ASTRO2X" in "course-v1:ANUx+ASTRO2X+2B3T2015".
• display_name: a user-facing display name for the course, for example "Astrophysics: Exploring Exoplanets"
• org: a reference to the Organization associated with this Course Code

ProgramOrganization

This represents the partner Organization that is publishing a given Program.

For the XSeries MVP, only one Organization per Program will be supported; this entity is modeled as many-to-many for future extensibility.

ProgramCourseCode

This represents an association between a Program and a CourseCode. The ProgramCourse has a 'position' attribute which is used to present CourseCodes within a Program in a user-defined order.

For the XSeries MVP, a CourseCode may only be associated with a single Program.  This is in order to simplify the learner dashboard implementation planned for MVP.  In a future iteration, a CourseCode and its runs / modes may appear in multiple Programs.

ProgramCourseRunMode

This represents a specific run of a course, and a "mode" of that course, linked to a Program (via ProgramCourseCode).  Runs and modes cannot be associated with Programs until they have been published to the participating LMS. (To avoid consistency problems, logic may be added to modify, remove, or invalidate Programs when a related run / mode stops being available in the LMS, though this is not part of XSeries MVP.)

ProgramUser

This represents a User's association with / registration into a Program, and captures any state relevant to that association. For the XSeries MVP, this state consists of a required 'status' enum attribute which will be used to track whether the User is "active", "completed", or "deleted" in that Program.  "Deleted" will be an internal-only status to support soft-deletion; API consumers will not see registrations with this status.

For the XSeries MVP, we are eschewing an explicit user interaction to make this association.  Instead, learners will be automatically and lazily associated with Programs:

• If a learner is enrolled in a course run that is part of an (active) Program, she will be considered to be associated with that Program.
• The following learner-specific events will trigger a call to the enrollment API to check in which course runs the learner is actively enrolled:
  • When calling the API to fetch a listing of Programs for that user.
  • When handling a notification that a certificate has been earned by that user.
• Based on the results of the enrollment check, the ProgramUser table will be updated (using a get-or-create semantic).
• This heavy usage of the Enrollment API will almost certainly necessitate performance optimizations.  Measurement and optimization of these interactions will be part of our implementation work.

Program Lifecycle

The following sequential lifecycle is envisioned for Programs, based on known/anticipated use cases:

1. unpublished - an initial status during which the Program is being constructed but should not yet be made available to Learners.  Unpublished Programs are invisible to non-admins and learners cannot be associated with them.
2. active – this status is set when the Author/Admin has finished building the Program and wants to make it available to Learners in the participating LMS.
   1. When a Program has been set to active, Course Runs cannot be removed from the Program.
   2. A Program can only be reverted from active to unpublished if no Users have yet been associated with the Program.
3. retired – this status is set when the Program should no longer be offered to new Learners. It may still be displayed as a historical item if that makes sense in a given context (e.g. viewing a Learner's past achievements).
4. deleted – used to support soft-deletion of Programs where needed. Programs with this status will not be visible to API consumers.

Population of Organizations

Organizations are pointers to related data for which the authoritative source is presently Studio (as part of the Certificate administration feature).  The existence, key and display name of each Organization has to be copied from Studio.  This can be done using a combination of strategies:

• A management command which copies the entire list of Organizations known to Studio and creates them using the Programs API.   This management command, and the API calls it uses, must be idempotent.  It will be used to initially populate the Organizations, and can be used in the future to manually re-sync the data if necessary to recover from an outage or interruption.
• A Django signal added to Studio, which listens for additions or changes to the list of Organizations in Studio, and makes Programs API calls to ensure the data is incrementally updated.
  • Organizations are not frequently changed.  A signal-triggered API call to update an Organization would ideally be performed asynchronously, but in practice this may be overkill provided we use short HTTP timeouts, handle failures gracefully, and monitor failures judiciously.
  • The signals should be disabled for bulk operations on the Organization list in Studio, which would present a more serious bottleneck.  If such a need arises, we would instead use the management command (as described above) for bulk resyncing.

Population of CourseCodes

CourseCodes have no equivalent entity in edx-platform.  They are instead derived from the set of course runs that are published into the LMS using Studio.  To generate this data in the Programs service, its API will expose and endpoint that accepts POST (or DELETE) requests containing information about a course run, and will generate CourseCode (and if needed, Organization) records.  

This endpoint will be used in a way similar to the way Organizations are synced:

• An idempotent management command which iterates over the list of course runs, POSTing each course key and title to the endpoint, for initial sync and manual re-sync as needed.
• A Django signal in the Studio publish operation (this signal already exists) which will incrementally update the data as it changes.
  • The signal handler which replicates individual changes to the Programs API should be performed asynchronously.

Internally, the API endpoint for course runs will parse the course_key into its component parts (org, course, run).  A get-or-create operation will ensure that records exist corresponding to the org (in Organization) and course (in CourseCode).  If the CourseCode is being created for the first time, the course run's title should be used for the display_name of the CourseCode.  If the Organization is being created for the first time (which should happen rarely, if ever, assuming Organizations are synced first), the org part will be used for both the key and the display_name.

Selection of Course Run Modes

Course runs themselves are not duplicated in the service.  Administrative tools used to build Programs will use our enrollment API to fetch listings of available runs and modes, filtered by an organization and course code selected using Admin interfaces.  (Support for these filters needs to be added to the enrollment API.)  When selections are made, the selected course key, mode (slug), and SKU (if used) will be used to create ProgramCourseRunMode records in the service. 

API

General notes

Unless otherwise specified in endpoint details,

• Response codes:
  
  • 200 indicates success for any operation other than a create operation
  • 201 indicates success for a create operation
  • 401 if the user is not authenticated

Roles and Permissions

MVP Permissions:

• see programs
• change programs (add/change/delete, add or remove orgs/courses/runs)
• publish program (i.e. set its status from "unpublished" to "active")
• see programs for a specific user (other than self)

MVP Roles (and their permissions):

• Learner (see programs)
• Author (see programs, change program)
• Admin (see programs, change programs, publish program, see programs for a specific user)

Role Mapping:

• Any authenticated user will have the Learner role
• Any global staff will have the Author and Admin roles

Future Expansion (speculative):

• add an Org-Author role
• make "change programs" specific to the org association(s) of that Program
• assign the Org-Author role to org-level staff in our system 

Programs

GET /programs/

List all Programs.

Optional query filters:

• category: list only Programs with the matching category.
• status: list only Programs with the given status ("unpublished", "active", or "retired").
• org: list only Programs associated with the given organization id.
• course: list only Programs associated with the given course id.
• run: list only Programs containing the given Course Run (i.e. course_key)
• username: only include Programs with which the given user is associated.
  
  

NOTE: querying / searching by Program name or description is out of scope for this MVP.

This endpoint handles access differently depending on whether it is accessed by a learner or staff:

• For learners,
  • when no username is passed, only active and/or retired programs will be returned.
  • when the username matches the requesting user's username, only programs with which this user is associated will be returned.
  • when a different username is passed, the service will send a 403 / Unauthorized response.
• For staff,
  • when no username is passed, unpublished programs will be included in the list along with active/retired.
  • when a username is passed, the list of programs will be filtered by association with that user.

Example Success Response:

HTTP/1.1 200 OK
Content-Type: application/json

[
  {
    "id": 1,	
    "name": "Astrophysics",
    "description": "A great astrophysics series",
    "category": "XSeries",
    "status": "unpublished",
    "organizations": [
      {
        "id": "ANUx",
        "display_name": "Australian National University"
      }
    ],
    "courses": [
      {
        "id": "ANUx/ANU-ASTRO2x",
        "organization": {
          "id": "ANUx",
          "display_name": "Australian National University"
        },
        "display_name": "Astrophysics: Exploring Exoplanets",
        "runs": [
          {
            "course_key": "ANUx/ANU-ASTRO2x/2B3T2015",
            "display_name": "Australian National University"
          },
          {
            "course_key": "ANUx/ANU-ASTRO2x/2B4T2015",
            "display_name": "Astrophysics: Exploring Exoplanets"
          }
        ]
      }
    ]
  }
]

POST /programs/

Create a Program.

Response codes:

• 201 when the resource was successfully created.
• 400 if the request contained invalid Program attributes, or an unrecognized organization, course, or course run.
  
• 403 if the user does not have permission to create a Program.

Example request body:

{
  "name": "Astrophysics",
  "description": "A great astrophysics series",
  "category": "XSeries",
  "organizations": [
    {
      "id": "ANUx"
    }
  ],
  "courses": [
    {
      "id": "ANUx/ANU-ASTRO2x",
      "organization": {
        "id": "ANUx",
      },
      "runs": []
    }
  ]
}

Example Response (success):

HTTP/1.1 201 Created
Content-Type: application/json

{
  "id": 1,	
  "name": "Astrophysics",
  "description": "A great astrophysics series",
  "category": "XSeries",
  "status": "unpublished",
  "organizations": [
    {
      "id": "ANUx",
      "display_name": "Australian National University"
    }
  ],
  "courses": [
    {
      "id": "ANUx/ANU-ASTRO2x",
      "organization": {
        "id": "ANUx",
        "display_name": "Australian National University"
      },
      "display_name": "Astrophysics: Exploring Exoplanets",
      "runs": []
    }
  ]
}

GET /programs/:program_id/

View full details of a Program, including Orgs, Courses, and Runs.

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "id": 1,	
  "name": "Astrophysics",
  "description": "A great astrophysics series",
  "category": "XSeries",
  "status": "unpublished",
  "organizations": [
    {
      "id": "ANUx",
      "display_name": "Australian National University"
    }
  ],
  "courses": [
    {
      "id": "ANUx/ANU-ASTRO2x",
      "organization": {
        "id": "ANUx",
        "display_name": "Australian National University"
      },
      "display_name": " Astrophysics: Exploring Exoplanets",
      "runs": [
        {
          "course_key": "ANUx/ANU-ASTRO2x/2B3T2015",
          "display_name": "Astrophysics: Exploring Exoplanets"
        },
        {
          "course_key": "ANUx/ANU-ASTRO2x/2B4T2015"
          "display_name": "Astrophysics: Exploring Exoplanets"
        }
      ]
    }
  ]
}

PATCH /programs/:program_id/

Partially update a Program.

Any of the following keys may be included, and will overwrite existing values: "name", "description", "status", "orgs", "courses". Values for any keys not included in the request will not be modified.

This is a merge-patch implementation, and callers must specify content-type "application/merge-patch+json".

Response codes:

• 200 when the resource was successfully patched.
• 400 if the request contained invalid Program attributes, unrecognized related entities, or an invalid status transition (e.g. active -> unpublished).
  
• 403 if the user does not have permission to change the Program.

After a successful patch, the entire (updated) resource is returned, same as for the GET response.

Example request body:

{
  "name": "Astrophysics",
  "description": "Learn contemporary astrophysics from the best",
  "status": "active"
}

Example response (success):

HTTP/1.1 200 OK
Content-Type: application/json

{
  "id": 1,	
  "name": "Astrophysics",
  "description": "Learn contemporary astrophysics from the best",
  "category": "XSeries",
  "status": "active",
  "organizations": [
    {
      "id": "ANUx",
      "display_name": "Australian National University"
    }
  ],
  "courses": [
    {
      "id": "ANUx/ANU-ASTRO2x",
      "organization": {
        "id": "ANUx",
        "display_name": "Australian National University"
      },
      "display_name": " Astrophysics: Exploring Exoplanets",
      "runs": [
        {
          "course_key": "ANUx/ANU-ASTRO2x/2B3T2015",
          "display_name": " Astrophysics: Exploring Exoplanets"
        },
        {
          "course_key": "ANUx/ANU-ASTRO2x/2B4T2015"
          "display_name": " Astrophysics: Exploring Exoplanets"
        }
      ]
    }
  ]
}

Course Runs

PUT /runs/:course_key/

Create or update a Course Run.

This will also automatically create an Org and a Course in the system derived from components of the course_key, if they do not already exist. When the Course is being created for the first time, the display_name attribute will be copied from the name of the Course Run. The display_name attribute of a newly-created Org will be initially empty.

Display names for the Org, Course, and Course Run may be subsequently customized via API calls.

The course_key part of the URL should be urlencoded.

DELETE /runs/:course_key/

Remove a Course Run. This will not remove any orphaned Org or Course records that may be left behind.

The course_key part of the URL should be urlencoded.

if any active Program would be affected by removing the Course Run, a 403 / Forbidden response will be returned.

Organizations

GET /organizations/

List all Organizations.

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

[
  {
    "id": "ANUx",
    "display_name": "Australian National University"
  }
]

Update the display_name of an Organization.  The entire (updated) resource is returned.

Example Request:

{
  "display_name": "The Australian National University"
}

Example Response:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "id": "ANUx",
  "display_name": "The Australian National University"
}

Courses

GET /courses/

List all Courses, with Course Runs inline.

Optional query filters:

• org: list only Course Runs that are related to the given organization id (e.g. "ANUx").

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

[
  {
    "id": "ANUx/ANU-ASTRO2x",
    "organization": {
      "id": "ANUx",
      "display_name": "Australian National University"
    },
    "display_name": " Astrophysics: Exploring Exoplanets",
    "runs": [
      {
        "course_key": "ANUx/ANU-ASTRO2x/2B3T2015",
        "display_name": " Astrophysics: Exploring Exoplanets"
      },
      {
        "course_key": "ANUx/ANU-ASTRO2x/2B4T2015",
        "display_name": " Astrophysics: Exploring Exoplanets"
      }
    ]
  }
]

GET /courses/:course_id/

View a single Course, with Course Runs inline. 

The course_id part of the URL should be urlencoded.

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "id": "ANUx/ANU-ASTRO2x",
  "organization": {
    "id": "ANUx",
    "display_name": "Australian National University"
  },
  "display_name": "Astrophysics: Exploring Exoplanets",
  "runs": [
    {
      "course_key": "ANUx/ANU-ASTRO2x/2B3T2015",
      "display_name": " Astrophysics: Exploring Exoplanets"
    },
    {
      "course_key": "ANUx/ANU-ASTRO2x/2B4T2015",
      "display_name": " Astrophysics: Exploring Exoplanets"
    }
  ]
}

PUT /courses/:course_id/

Update the display_name of a Course.  The entire (updated) resource is returned.

Example Request:

{
  "display_name": "Astrophysics: Exploring Exciting Exoplanets"
}

Example Response: (see GET response)

XSeries MVP Administration Flow (Otto / CAT)

The Program Admin views will be implemented as a JavaScript single-page application (SPA) which will deployed with the Programs service, and included within the Otto UX via script tags. This client application will use the Program HTTP APIs to perform CRUD on Programs and list related entities (Orgs, Courses, Runs).

edx-platform Integration

In MVP, learner-facing interfaces related to Programs will be exposed as links from the LMS dashboard.  Post-MVP, we may expose authoring views using JavaScript code embedded in Studio.

A new feature flag, ENABLE_PROGRAMS, will be introduced, which will toggle the availability of program-related functionality.

LMS Integration: Learner Registration / Dashboard

The ENABLE_PROGRAMS flag will toggle the inclusion of logic on the LMS dashboard that will call to the Programs API to determine which of their currently-enrolled course runs are associated with programs, and draw a link to that program inline with each.  The link will take the learner to a detailed view of that Program, including courses, course runs, and their completion status within each.

The program detail views will be implemented in JavaScript code which will be deployed with the Programs service, and included within the LMS UX via script tags.  This application will use the Program HTTP APIs and the Enrollment API together to indicate in which of the Program's Course Runs the Learner is presently enrolled.

Enrollment Integration

Under the scope of this proposal, the LMS remains the source of truth for Course [Run] Enrollments.  Program-related APIs do not require direct access to enrollment data per se; however, displaying enrollment-related information on a per-Learner basis will be relevant for the dashboard and similar use cases.  In the proposed implementation, it should be sufficient to retrieve enrollment information from the existing API and pass that data to the Program-specific views.

In order to provide controls for the Learner to enroll in a new Course Run from Program-related views, we will pass data/configuration to the views that is capable of redirecting the user to the appropriate endpoint when requesting new enrollments - either the ecommerce system (when participating/configured) or directly to the LMS's enrollment handlers. 

Progress, Completion, and Certification

For XSeries we need to implement behavior that will allow Learners to:

• View the Courses and Runs in a Program in which the Learner is registered, and indicate which ones they have in progress or completed.
• Create a certificate of completion when they have completed the entire Program (earned the required certificate in each Course), and make it available to the Learner.

In addition, administrators will need to be able to create and customize certificate templates for Programs.

In order to support these requirements we will (probably) need to enhance or create LMS functionality around tracking and communicating course completion, and both the LMS and the Programs service will need to collaborate in order to fulfill certificates.  We will address this work in detail in a follow-on discovery / design effort.  

Commerce Integration (tentative)

Integration with Otto (the external commerce service) will work along similar lines to the present integration for creating and fulfilling products based on seats in Course [Runs].

• An administration tool in Otto will read from the Program service to find available (active) Programs, and related Course information.
• New modules in Otto will define a product structure for XSeries encapsulating availability, pricing, upgrade, and refund logic.
• Otto will use the Program Users API to fulfill registrations in the Program, and continue to use the Enrollment API to fulfill Enrollments as it does presently.

This functionality / integration has been removed from the MVP and will very likely be revised in a successive design effort.

Mobile Integration

Learners' courseware experience on the Mobile app is not impacted by work being done for the XSeries MVP.  Purchasing / registering in XSeries is specifically out of scope within the MVP.  We do intend for all new APIs and UX's to be compatible with the Mobile implementations, to support future integrations post-MVP.

Analytics Integration

A bulk data export facility for consumption by downstream analytics will be developed, but this is not currently planned as part of the MVP.  edX also has specific reporting requirements around sales and registrations which will be driven in large part by data that will reside in our commerce service. 

XBlock Integration

It is not within the scope of the XSeries MVP to interact directly with Programs from XBlocks; however, implementing a runtime service client for the HTTP APIs or referencing the Javascript applications from HTML views should be straightforward and self-explanatory.