edX REST API Conventions

Created by Nimisha Asthagiri (Deactivated)

Background

This document captures the conventions to be used for all edX REST APIs. 

Useful reading:

High Level Requirements

  • Consumer-Perspective. Design your API from the perspective of the consumer, NOT the perspective of your underlying implementation.  For example:

    • If the underlying implementation requires accessing multiple models or multiple apps/projects, this does not need to be reflected in a public interface.  From the perspective of the consumer, it's simply one thing they are requesting.

    • Keep CRUD operations together within its corresponding resource.  Why have the client go to one resource to read it and then another resource to write it?

  • Simple.  Keep the top-leveI resources clear and simple - focusing on what the client is looking to consume.  You can hide complexity within the parameters.

  • Separation of concerns.  Do not require the consumer to know any implementation details.  For example, an API shouldn't require the client to know the inter-dependencies of fields.  Rather, all such business rules should be owned by the server.  This allows the client to be lightweight and easily maintainable in the future.

  • Explicit Support Levels. APIs should make it clear whether they are experimental or more permanent as a part of their documentation.

  • Discoverability. Support HATEOAS where possible.  This allows us to change our URLs without needing to worry about updating the mobile apps if the app discovers its URLs from a base URL.

Conventions

1. URL Naming

  • Must: Keep your base URL simple and intuitive.  

    • Suggestion: Follow this basic URL structure if you are concerned about collisions within your service.  By "collisions", we mean TBD.

      1 /api/{API_NAME}/{VERSION}/...

      If collisions are not a concern (for example, when exposing APIs from context-specific IDAs), then you can eliminate the use of API_NAME.

      1 /api/{VERSION}/...

  • Must: Keep the API name flat.  Use two base URLs per resource: collection/identifier, e.g.

    1 /api/user/v1/accounts/{USERNAME}

  • It is not necessary that there is a one-to-one correspondence between an API and the Django app that provides it.  In fact, which Django app implements an API is an implementation detail and need not be bound to a public interface.

  • Resource name

    • Keep verbs out of the base URL

    • Must: Use plural rather than singular nouns

    • Must: Use concrete (e.g., blogs, videos, news) rather than abstract (e.g., items, assets) names

    • Must: Use Python conventions: use_underscores instead of CamelCase

  • Verbs

    • When no resource is involved, be direct and use Verbs instead of Nouns (p 19).  But only when absolutely necessary as we try to use nouns as much as possible.

    • /convert?from=EUR&to=CNY&amount=100

    • Make it clear in your API documentation that these “non-resource” scenarios are different.

See list of example APIs at the end of this document.

2. Identifying Resources

  • Nice: Do not expose database IDs where possible (Must for external APIs, per dev ops)

  • Must: edX resource identifiers

    • Users should be referenced by username.

    • Courses runs should be referenced via course run keys (eg. course-v1:edX+BlendedX+1T2018)

    • Catalog courses should be referenced via a catalog UUID.

    • Course blocks should be referenced via usage keys (eg. block-v1:org+course+run+type@sequential+block@2aa6fc9d8278)

  • Explicit Filters

    • Model system resource URL schemes as if all resources are available to all users

    • Include all necessary filters in the URL such that any user could theoretically access the resource

    • Separate filtering and authorization – ie, do not return different resource representations via the same URL based on the requesting user 

      • 1 Example 1:  Choose "/profiles/john_harvard" over "/profile"
      • 1 Example 2:  Choose "/courses?username=john_harvard" over "/courses"

    • See relevant conversation here for more context.

    • Remember, it's entirely possible that "/profiles/john_harvard" or "/courses?username=john_harvard" could be requested by the "administrator" user

    • Several benefits exist from an explicit filtering approach:

      • Ensures resources/results are individually-addressable

      • Enables discovery/sharing of resources, among other potential uses

      • Resource filtering mechanisms can be modified without impacting authorization mechanisms

      • Intermediate network gear can cache resource representations to improve performance (for open/unprotected resources)

  • Composite Resource (with multiple dimensions)

    • If an endpoint represents a relationship among multiple dimensions, the dimensions can be specified in the following ways:

    • As filter parameters:

      • 1 /api/enrollments/v1/enrollments/?user={username}&course={course_id}

    • As comma-delimited resource keys in the URL:

      • 1 /api/enrollments/v1/enrollments/{username},{course_id}

    • As a UUIDs to uniquely identify the relationship:

      • 1 /api/enrollments/v1/enrollments/{enrollment_id}

3. HTTP Verbs

  • Use HTTP verbs to operate on the collections and resources

    • "Safe" operations (Must not: modify any resource's state)

      • GET - Filter a collection down to a set of resources matching the provided criteria

        • /users

        • /users/:id

        • /users?last_name=Smith

    • State-modyfing operations

      • POST - Submit information about a resource to the service for processing

        • Typically utilized for resource creation workflows

        • POST /users   {user_data}

        • Other cases include establishing a link between two resources

        • POST /users/:id/groups  {group_id}

      • PATCH - Modify an existing resource

        • Return value must be 204 with no additional content. 

        • If implementing the simpler "merge patch" algorithm (https://tools.ietf.org/html/rfc7396), the content_type of "application/merge-patch+json" should be used to differentiate from the full JSON Patch specification (http://tools.ietf.org/html/rfc6902).  

        • See the note below for more information on PATCH.

      • PUT - Replace a resource in entirety.  

        • Most use cases can be solved using partial updates via PATCH

        • PUT can be used if a complete update of the resource, including all of its sub-resources, is desired. 

        • A typical PUT use case:  The resource is a singular primitive value addressable via URI, such as when updating the value for a particular User Preference.

      • DELETE - Remove a resource (or a relationship) from the system

        • Must return HTTP 204 No Content

        • DELETE /users/:id

        • DELETE /users/:id/groups/:group_id

A note on HTTP PATCH

"Plain" HTTP PATCH (RFC 5789) is neutral with respect to content type and only specifies that the request body provide instructions for how to update the resource in question, not the format of those instructions.  Several flavors of PATCH specific to JSON documents have more explicit definition, as noted above.  

In DRF, the default PATCH handling fairly closely resembles what is specified by merge patch, but it does not require (or understand) the "application/merge-patch+json" media type.  In edX REST APIs, if an implementation will use DRF's default PATCH handling, the implementation MUST recognize and accept the merge patch media type; API clients SHOULD use this media type, preferring it to the more typical "application/json" type.

The reason for this stricture is to explicitly leave room for supporting multiple PATCH styles in any given API, should this become desirable, without breaking existing clients.

4. URL Parameters

  • Must: Sweep complexity behind the ‘?’  (e.g., GET /dogs?color=red&state=running&location=park) (p 9)

  • fields parameter - Allow clients to specify/filter the fields in the response by supporting a fields parameter as a comma-delimited list. 
    "Partial Response allows you to give developers just the information they need." (p 16)

    1 2 /dogs?fields=name,color,location /dogs?fields=title,media:group(media:thumbnail)

  • expand parameter - Allow clients to request including data from other resources using the expand parameter as explained further in #expand. 

  • requested_fields parameter - Allow clients to request additional optional fields that will be added to the response.  (e.g. GET /comment?requested_fields=author_profile_image,endorse_profile_image)

  • text_search parameter - Allow clients to perform text-based search using this parameter.  (p 22)

    • global: /search?text_search=fluffy+fur

    • scoped: /owners/5678/dogs?text_search=fluffy+fur

    • formatted: /search.xml?text_search=fluffy+fur

5. Errors

  • HTTP status codes - Use the appropriate value amongst the following (or document in the code why an exception is needed)

    • 200 - OK

    • 201 - Created

    • 204 - OK, no content returned (use for PATCH)

    • 304 - Not Modified

    • 400 - Bad Request

    • 401 - Unauthorized (for unauthenticated users)

    • 403 - Forbidden (for authenticated users who do not have the right permissions)

    • 404 - Not Found

    • 415 - Unsupported Media Type (used for PATCH if implementation is "merge patch" algorithm, and caller did not specify "application/merge-patch+json" content_type).

    • 500 - Internal Server Error

  • Prevent information leakage

    • Must: Use 404 instead of 403 when the actual existence would be leaking information that we don't want - Dave O.

  • Error Description

    • Must: Be verbose and use plain language descriptions.

    • Must: Use i18n for user facing messages ("user_message" in example below)

    • Nice: Add as many hints as your API team can think of about what's causing an error.

    • Nice: Add a link in your description to more information, like Twilio does.

  • Error format:

    • developer_message (no i18n)

    • user_message(optional, but if provided, i18n as the expectation is that this may surface in a UI)

    • field_errors (if applicable)

    • error_code (for example: see /wiki/spaces/MS/pages/31687272)  Include description in your REST API Docs.

      1 2 3 4 5 6 7 8 9 10 11 { "error_code": "course_not_started" # a short string that the client can rely upon for handling different errors "developer_message" : "Verbose, plain language description of the problem for the app developer with hints about how to fix it.", "user_message":"Pass this message on to the app user if needed.", "field_errors": { "foo": { "developer_message": "", "user_message": "" } } }

6. Version

  • Location.

    • Put the version in the URL so the client can see it easily when handling the response logic.

    • Exception: Put it in the header only if it doesn't change the response handling logic, like the OAuth endpoint.

  • Numbering.

    • Must: Versions are major only ex. /v0/ /v1/ /v2/

    • Must: Major version should only be updated when changing the existing contract in a way that breaks backwards compatibility.  The previous major version should continue to be supported for old clients according to the service's deprecation policy.

    • Must: Additive changes to the contract of the API, or the results, should not bump the major version, unless those changes are non-optional, and break existing contracts with the API.

  • Future considerations:

    • (TODO) Responses should have a header containing minor / patch versions. 

      • X-API-Version (TODO look into established convention for header name)

    • Deprecation Process: varies from project to project based on product requirements and developer usage, client usage, etc. Should be well documented.

      • TODO: deeper dive into Deprecation.

7. Pagination

  • We use the conventions established by DRF, with some extra fields:

    • next: A URL to the next page, or null.

    • previous: A URL to the previous page, or null.

    • count: The total number of items.

    • num_pages: The total number of pages.

    • results: List of results.

  • The query parameters are as follows:

    • page - the page of results (zero indexed)

    • page_size - the number of results to return per page

    • sort_order - a string indicating the desired order (no direction modifiers)

8. Documentation

9. Discoverability

  • Use hyperlinks (with absolute URLs) to represent (primary/foreign) keys.

  • When there are links to other resources, we should have a hyperlink in the response.

10. Multiple Formats

  • JSON as the default format

  • Clients can request different formats by specifying the "Accept" header

  • The server should always include a "Content-Type" header which specifies the format of the data being returned

11. Authentication

By convention, our REST APIs support the following two authentication schemes:

  • OAuth2 - for mobile clients and micro-services

  • Session-based authentication - for mobile webviews and browser clients 

12. Serialization Conventions

  • Dates and Timestamps

    • Should be serialized to strings in the ISO 8601 standard format

    • Timestamps should include explicit timezone offsets

    • UTC timestamps are preferred

To Be Discussed

  • Consolidate all API requests under one API subdomain.  (p 23)

    • And also do Web redirects for common names

Expanding referenced objects

APIs can support expanding related objects, e.g. the team API optionally includes user profile information for each member.  This is preferable to requiring the client to make multiple subsequent AJAX calls.

For example, there can be three levels for a related object (using "username" as an example):

  • Returning just the id

1 "username": "andya"

  • Returning an object containing the id and a URL to get more details

1 2 3 4 "user": { "username": "andya", "url": "https://openedx.example.com/api/user/v1/accounts/andya" }

  • Returning an object containing the expanded details

1 2 3 4 5 6 7 8 9 10 11 12 "user": { "username": "andya", "country": "UK", "profile_image": { "has_image": True, "image_url_full": "http://my-image-storage.net/media/profile_images/123456789_500.jpg", "image_url_large": "http://my-image-storage.net/media/profile_images/123456789_120.jpg", "image_url_medium": "http://my-image-storage.net/media/profile_images/123456789_50.jpg", "image_url_small": "http://my-image-storage.net/media/profile_images/123456789_30.jpg", }, ... }

By default:

  • For objects that have their own APIs, use option (b) and return both an id and a URL

  • For other objects, use option (a) and just return the id

  • Support a query parameter indicating an optional list of objects to be expanded: e.g. ?expand=user,team

Review Process

APIs must go through a community review process and review by the architecture council before being accepted into the platform. See the Architecture Review Process for the process for gaining approval.

Example APIs

There are a number of existing APIs that you can crib from:

Here are some representative URLs

1 2 3 /api/course/v1/courses/ /api/enrollment/v1/enrollments/ /api/discussion/v1/threads/{thread_id}/