Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 23 Next »

Context

Programs are a structured way to group edX course offerings together, to serve learning objectives or instructional goals that span multiple courses.

Programs will be used as the technical foundation to deliver XSeries, an edx.org-specific product offering planned for early 2016. It is our intention to implement Programs as a generic and reusable capability in the edX Platform, so it can be leveraged in support of related use cases both by edx.org and the wider Open edX community. However, our main priority for Q1-Q2 2016 is to develop the functionality necessary to deliver the XSeries MVP.

This document focuses on a key new piece of infrastructure encapsulating the data, logic, and API upon which the new feature will be built, and how it will integrate with key services in the platform to support basic use cases.      

Architecture Overview

Program-related functionality will be implemented in a self-contained, independently-deployable web application. Related support will be implemented in the form of smaller changes to the LMS, Studio, and other participating platform applications.

Modularity is emphasized as a general principle; dependencies in related systems will be kept to a minimum and enabled/disabled via feature flags.  This is intended to maximize operational flexibility when choosing to introduce the Programs functionality within an existing installation.

The new application will serve as the "source of truth" in edx-platform for the Programs themselves, as well as Learners' associations with those Programs (aka Registrations). These data will be exposed as resources via RESTful HTTP APIs. In addition, the application will host client-side JavaScript applications as user-facing frontends to these APIs.

We plan to implement these features using Django / DRF / SQL backend and Backbone.js frontend stacks, keeping consistent with other platform applications.

Data Model




Diagram note: Entities drawn with dotted lines indicate references which are replicated from an external source of truth (i.e. LMS).

Entities

Program

The Program represents the collection of multiple Course Runs into a group.

Attributes:

  • name: a user-facing name for the Program, which must be unique.
  • description: an optional brief text string describing the Program.
  • category: an optional string which can be used to organize Programs for presentation / classification. For XSeries-based programs, the category will be 'XSeries'.
  • certificate_type: an optional string describing the type of certificate required to satisfy completion for Courses in the Program.  For XSeries-based programs, the certificate type will be 'verified'.
  • status: an enum indicating the current lifecycle state of the Program.  This is discussed further below in the "Program Lifecycle" section. 

ProgramOrg

This represents the partner Organization that is publishing a given Program.

For the XSeries MVP, only one Organization per Program will be supported; this entity is modeled as many-to-many for future extensibility.

ProgramCourse

This represents an association between a Program and a Course (independent of specific Runs of that Course). The ProgramCourse has a 'position' attribute which is used to present a Course's Programs in a user-defined order.

Not all Runs of a Course are automatically associated with a Program. Associating the Course provides the ability to select and link/unlink individual Runs of that Course with the Program.

For the XSeries MVP, a Course may only be associated with a single Program.  This is in order to simplify the learner dashboard implementation planned for MVP.  In a future iteration, a Course and its runs may appear in multiple Programs.

ProgramCourseRun

This represents a specific Course Run linked to a Program. A Course Run should not be linked to a Program until it has been published to the participating LMS. (To avoid consistency problems, logic may be added to modify, remove, or invalidate Programs when a related Course Run stops being available in the LMS, though this is not part of XSeries MVP.)

ProgramUser

This represents a User's association with / registration into a Program, and captures any state relevant to that association. For the XSeries MVP, this state consists of a required 'status' enum attribute which will be used to track whether the User is "active", "completed", or "deleted" in that Program.  "Deleted" will be an internal-only status to support soft-deletion; API consumers will not see registrations with this status.

For the XSeries MVP, we are eschewing an explicit user interaction to make this association.  Instead, learners will be automatically and lazily associated with Programs:

  • If a learner is enrolled in a course run that is part of an (active) Program, she will be considered to be associated with that Program.
  • The following learner-specific events will trigger a call to the enrollment API to check in which course runs the learner is actively enrolled:
    • When calling the API to fetch a listing of Programs for that user.
    • When handling a notification that a certificate has been earned by that user.
  • Based on the results of the enrollment check, the ProgramUser table will be updated (using a get-or-create semantic).
  • This heavy usage of the Enrollment API will almost certainly necessitate performance optimizations.  Measurement and optimization of these interactions will be part of our implementation work.

Program Lifecycle

The following sequential lifecycle is envisioned for Programs, based on known/anticipated use cases:

  1. unpublished - an initial status during which the Program is being constructed but should not yet be made available to Learners.  Unpublished Programs are invisible to non-admins and learners cannot be associated with them.
  2. active – this status is set when the Author/Admin has finished building the Program and wants to make it available to Learners in the participating LMS.
    1. When a Program has been set to active, Course Runs cannot be removed from the Program.
    2. A Program can only be reverted from active to unpublished if no Users have yet been associated with the Program.
  3. retired – this status is set when the Program should no longer be offered to new Learners. It may still be displayed as a historical item if that makes sense in a given context (e.g. viewing a Learner's past achievements).
  4. deleted – used to support soft-deletion of Programs where needed. Programs with this status will not be visible to API consumers.

Replication of LMS Entities

The Org, Course, and Course Run entities are pointers to entities replicated from the authoritative source for those data (i.e. the participating LMS). They are replicated to this service in order to be able to create foreign key constraints from Program-related tables.

Following an initial backpopulation of these tables, subsequent changes will be replicated to the Programs application using Django signals / event hooks installed in the LMS.

Presently, the LMS implements publish and delete signals for Course Runs in the modulestore. The planned behavior for the XSeries MVP is to fire asynchronous, idempotent requests to PUT or DELETE against a Course Run endpoint in the Programs service, triggered by these signals. That endpoint will, internally, use the opaque_keys shared library to derive Course and Org IDs from the Course Run IDs, and insert data in those tables if none yet exist.

With the recent deployment of edx-organizations (part of the automated certificates work), we now have an authoritative source of truth for Organizations hosted in the LMS, so our tools for backpopulation will use this source for Orgs. 

API

General notes

Unless otherwise specified in endpoint details,

  • Response codes:
    • 200 indicates success for any operation other than a create operation
    • 201 indicates success for a create operation
    • 401 if the user is not authenticated

Roles and Permissions

MVP Permissions:

  • see programs
  • change programs (add/change/delete, add or remove orgs/courses/runs)
  • publish program (i.e. set its status from "unpublished" to "active")
  • see programs for a specific user (other than self)

MVP Roles (and their permissions):

  • Learner (see programs)
  • Author (see programs, change program)
  • Admin (see programs, change programs, publish program, see programs for a specific user)

Role Mapping:

  • Any authenticated user will have the Learner role
  • Any global staff will have the Author and Admin roles

Future Expansion (speculative):

  • add an Org-Author role
  • make "change programs" specific to the org association(s) of that Program
  • assign the Org-Author role to org-level staff in our system 

Programs

GET /programs/
List all Programs.

Optional query filters:

  • category: list only Programs with the matching category.
  • status: list only Programs with the given status ("unpublished", "active", or "retired").
  • org: list only Programs associated with the given organization id.
  • course: list only Programs associated with the given course id.
  • run: list only Programs containing the given Course Run (i.e. course_key)
  • username: only include Programs with which the given user is associated.

NOTE: querying / searching by Program name or description is out of scope for this MVP.

This endpoint handles access differently depending on whether it is accessed by a learner or staff:

  • For learners,
    • when no username is passed, only active and/or retired programs will be returned.
    • when the username matches the requesting user's username, only programs with which this user is associated will be returned.
    • when a different username is passed, the service will send a 403 / Unauthorized response.
  • For staff,
    • when no username is passed, unpublished programs will be included in the list along with active/retired.
    • when a username is passed, the list of programs will be filtered by association with that user.

Example Success Response:

HTTP/1.1 200 OK
Content-Type: application/json

[
  {
    "id": 1,	
    "name": "Astrophysics",
    "description": "A great astrophysics series",
    "category": "XSeries",
    "status": "unpublished",
    "organizations": [
      {
        "id": "ANUx",
        "display_name": "Australian National University"
      }
    ],
    "courses": [
      {
        "id": "ANUx/ANU-ASTRO2x",
        "organization": {
          "id": "ANUx",
          "display_name": "Australian National University"
        },
        "display_name": "Astrophysics: Exploring Exoplanets",
        "runs": [
          {
            "course_key": "ANUx/ANU-ASTRO2x/2B3T2015",
            "display_name": "Australian National University"
          },
          {
            "course_key": "ANUx/ANU-ASTRO2x/2B4T2015",
            "display_name": "Astrophysics: Exploring Exoplanets"
          }
        ]
      }
    ]
  }
]
POST /programs/

Create a Program.

Response codes:

  • 201 when the resource was successfully created.
  • 400 if the request contained invalid Program attributes, or an unrecognized organization, course, or course run.
  • 403 if the user does not have permission to create a Program.

Example request body:

{
  "name": "Astrophysics",
  "description": "A great astrophysics series",
  "category": "XSeries",
  "organizations": [
    {
      "id": "ANUx"
    }
  ],
  "courses": [
    {
      "id": "ANUx/ANU-ASTRO2x",
      "organization": {
        "id": "ANUx",
      },
      "runs": []
    }
  ]
}

 

Example Response (success):

HTTP/1.1 201 Created
Content-Type: application/json

{
  "id": 1,	
  "name": "Astrophysics",
  "description": "A great astrophysics series",
  "category": "XSeries",
  "status": "unpublished",
  "organizations": [
    {
      "id": "ANUx",
      "display_name": "Australian National University"
    }
  ],
  "courses": [
    {
      "id": "ANUx/ANU-ASTRO2x",
      "organization": {
        "id": "ANUx",
        "display_name": "Australian National University"
      },
      "display_name": "Astrophysics: Exploring Exoplanets",
      "runs": []
    }
  ]
}

GET /programs/:program_id/

View full details of a Program, including Orgs, Courses, and Runs.

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "id": 1,	
  "name": "Astrophysics",
  "description": "A great astrophysics series",
  "category": "XSeries",
  "status": "unpublished",
  "organizations": [
    {
      "id": "ANUx",
      "display_name": "Australian National University"
    }
  ],
  "courses": [
    {
      "id": "ANUx/ANU-ASTRO2x",
      "organization": {
        "id": "ANUx",
        "display_name": "Australian National University"
      },
      "display_name": " Astrophysics: Exploring Exoplanets",
      "runs": [
        {
          "course_key": "ANUx/ANU-ASTRO2x/2B3T2015",
          "display_name": "Astrophysics: Exploring Exoplanets"
        },
        {
          "course_key": "ANUx/ANU-ASTRO2x/2B4T2015"
          "display_name": "Astrophysics: Exploring Exoplanets"
        }
      ]
    }
  ]
}

PATCH /programs/:program_id/

Partially update a Program.

Any of the following keys may be included, and will overwrite existing values: "name", "description", "status", "orgs", "courses". Values for any keys not included in the request will not be modified.

This is a merge-patch implementation, and callers must specify content-type "application/merge-patch+json".

Response codes:

  • 200 when the resource was successfully patched.
  • 400 if the request contained invalid Program attributes, unrecognized related entities, or an invalid status transition (e.g. active -> unpublished).
  • 403 if the user does not have permission to change the Program.

After a successful patch, the entire (updated) resource is returned, same as for the GET response.

Example request body:

{
  "name": "Astrophysics",
  "description": "Learn contemporary astrophysics from the best",
  "status": "active"
}


Example response (success):

HTTP/1.1 200 OK
Content-Type: application/json

{
  "id": 1,	
  "name": "Astrophysics",
  "description": "Learn contemporary astrophysics from the best",
  "category": "XSeries",
  "status": "active",
  "organizations": [
    {
      "id": "ANUx",
      "display_name": "Australian National University"
    }
  ],
  "courses": [
    {
      "id": "ANUx/ANU-ASTRO2x",
      "organization": {
        "id": "ANUx",
        "display_name": "Australian National University"
      },
      "display_name": " Astrophysics: Exploring Exoplanets",
      "runs": [
        {
          "course_key": "ANUx/ANU-ASTRO2x/2B3T2015",
          "display_name": " Astrophysics: Exploring Exoplanets"
        },
        {
          "course_key": "ANUx/ANU-ASTRO2x/2B4T2015"
          "display_name": " Astrophysics: Exploring Exoplanets"
        }
      ]
    }
  ]
}

 

Course Runs

PUT /runs/:course_key/

Create or update a Course Run.

This will also automatically create an Org and a Course in the system derived from components of the course_key, if they do not already exist. When the Course is being created for the first time, the display_name attribute will be copied from the name of the Course Run. The display_name attribute of a newly-created Org will be initially empty.

Display names for the Org, Course, and Course Run may be subsequently customized via API calls.

The course_key part of the URL should be urlencoded.

DELETE /runs/:course_key/

Remove a Course Run. This will not remove any orphaned Org or Course records that may be left behind.

The course_key part of the URL should be urlencoded.

if any active Program would be affected by removing the Course Run, a 403 / Forbidden response will be returned.

Organizations

GET /organizations/

List all Organizations.

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

[
  {
    "id": "ANUx",
    "display_name": "Australian National University"
  }
]
PUT /organizations/:organization_id/

Update the display_name of an Organization.  The entire (updated) resource is returned.

Example Request:

{
  "display_name": "The Australian National University"
}

Example Response:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "id": "ANUx",
  "display_name": "The Australian National University"
}

Courses

GET /courses/

List all Courses, with Course Runs inline.

Optional query filters:

  • org: list only Course Runs that are related to the given organization id (e.g. "ANUx").


Example response:

HTTP/1.1 200 OK
Content-Type: application/json

[
  {
    "id": "ANUx/ANU-ASTRO2x",
    "organization": {
      "id": "ANUx",
      "display_name": "Australian National University"
    },
    "display_name": " Astrophysics: Exploring Exoplanets",
    "runs": [
      {
        "course_key": "ANUx/ANU-ASTRO2x/2B3T2015",
        "display_name": " Astrophysics: Exploring Exoplanets"
      },
      {
        "course_key": "ANUx/ANU-ASTRO2x/2B4T2015",
        "display_name": " Astrophysics: Exploring Exoplanets"
      }
    ]
  }
]

GET /courses/:course_id/

View a single Course, with Course Runs inline. 

The course_id part of the URL should be urlencoded.

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "id": "ANUx/ANU-ASTRO2x",
  "organization": {
    "id": "ANUx",
    "display_name": "Australian National University"
  },
  "display_name": "Astrophysics: Exploring Exoplanets",
  "runs": [
    {
      "course_key": "ANUx/ANU-ASTRO2x/2B3T2015",
      "display_name": " Astrophysics: Exploring Exoplanets"
    },
    {
      "course_key": "ANUx/ANU-ASTRO2x/2B4T2015",
      "display_name": " Astrophysics: Exploring Exoplanets"
    }
  ]
}

PUT /courses/:course_id/

Update the display_name of a Course.  The entire (updated) resource is returned.

Example Request:

{
  "display_name": "Astrophysics: Exploring Exciting Exoplanets"
}

Example Response: (see GET response)

XSeries MVP Administration Flow (Otto / CAT)

The Program Admin views will be implemented as a JavaScript single-page application (SPA) which will deployed with the Programs service, and included within the Otto UX via script tags. This client application will use the Program HTTP APIs to perform CRUD on Programs and list related entities (Orgs, Courses, Runs).

edx-platform Integration

In MVP, learner-facing interfaces related to Programs will be exposed as links from the LMS dashboard.  Post-MVP, we may expose authoring views using JavaScript code embedded in Studio.

A new feature flag, ENABLE_PROGRAMS, will be introduced, which will toggle the availability of program-related functionality.

LMS Integration: Learner Registration / Dashboard

The ENABLE_PROGRAMS flag will toggle the inclusion of logic on the LMS dashboard that will call to the Programs API to determine which of their currently-enrolled course runs are associated with programs, and draw a link to that program inline with each.  The link will take the learner to a detailed view of that Program, including courses, course runs, and their completion status within each.

The program detail views will be implemented in JavaScript code which will be deployed with the Programs service, and included within the LMS UX via script tags.  This application will use the Program HTTP APIs and the Enrollment API together to indicate in which of the Program's Course Runs the Learner is presently enrolled.

Enrollment Integration

Under the scope of this proposal, the LMS remains the source of truth for Course [Run] Enrollments.  Program-related APIs do not require direct access to enrollment data per se; however, displaying enrollment-related information on a per-Learner basis will be relevant for the dashboard and similar use cases.  In the proposed implementation, it should be sufficient to retrieve enrollment information from the existing API and pass that data to the Program-specific views.

In order to provide controls for the Learner to enroll in a new Course Run from Program-related views, we will pass data/configuration to the views that is capable of redirecting the user to the appropriate endpoint when requesting new enrollments - either the ecommerce system (when participating/configured) or directly to the LMS's enrollment handlers. 

Progress, Completion, and Certification

For XSeries we need to implement behavior that will allow Learners to:

  • View the Courses and Runs in a Program in which the Learner is registered, and indicate which ones they have in progress or completed.
  • Create a certificate of completion when they have completed the entire Program (earned the required certificate in each Course), and make it available to the Learner.

In addition, administrators will need to be able to create and customize certificate templates for Programs.

In order to support these requirements we will (probably) need to enhance or create LMS functionality around tracking and communicating course completion, and both the LMS and the Programs service will need to collaborate in order to fulfill certificates.  We will address this work in detail in a follow-on discovery / design effort.  

Commerce Integration (tentative)

Integration with Otto (the external commerce service) will work along similar lines to the present integration for creating and fulfilling products based on seats in Course [Runs].

  • An administration tool in Otto will read from the Program service to find available (active) Programs, and related Course information.
  • New modules in Otto will define a product structure for XSeries encapsulating availability, pricing, upgrade, and refund logic.
  • Otto will use the Program Users API to fulfill registrations in the Program, and continue to use the Enrollment API to fulfill Enrollments as it does presently.

This functionality / integration has been removed from the MVP and will very likely be revised in a successive design effort.

Mobile Integration

Learners' courseware experience on the Mobile app is not impacted by work being done for the XSeries MVP.  Purchasing / registering in XSeries is specifically out of scope within the MVP.  We do intend for all new APIs and UX's to be compatible with the Mobile implementations, to support future integrations post-MVP.

Analytics Integration

A bulk data export facility for consumption by downstream analytics will be developed, but this is not currently planned as part of the MVP.  edX also has specific reporting requirements around sales and registrations which will be driven in large part by data that will reside in our commerce service. 

XBlock Integration

It is not within the scope of the XSeries MVP to interact directly with Programs from XBlocks; however, implementing a runtime service client for the HTTP APIs or referencing the Javascript applications from HTML views should be straightforward and self-explanatory.





  • No labels