Using Existing Established Conventions

READ THIS FIRST: Apigee RESTful Conventions

• Quick read
• Simple conventions

We'll be using this document as our basis. This page will outline any deviation from the document in our own conventions.

Use two base URLs per resource

• See Page 5 of the Apigree RESTful Conventions
  • Ex. /dog/12345, where 'dog' is the object, '12345' is the identifier.
• The Identifier is up to the provider, and should be an opaque value. (This does not have to be an ID, but could be, say, the username)
  • /user/bob42
• Do not expose Database IDs as the Identifiers if it can be avoided.

RESTful API Naming Conventions

Make sure a developer subscribing to multiple edX APIs  does not have to re-learn a new vocabulary for each API by standardizing on our nouns and URL structure.

General Structure: /api/[application]/[version]/...

Example: /api/enrollment/v0/users/bob/courses/edX/Demo101/2014T1

Where:

• api is the base of the URL. 
  • Use the "api" base URL for all new APIs, in order to consolidate all API requests
• enrollment is the name of the Enrollment API 
• v0 is the current version 
• users and courses are variables used to obtain information from the API.

Versioning

• Use v0 until the API is ready for external use, or is subscribed to by other applications. (No API should exist at v0 for a long period of time)
• There is an expectation for backwards compatibility beyond v0, where old URL structures should be supported 
• Use major versions in the URL (v0, v1, v2, etc..)
• Minor / Patch versions may be used in the response header.

Versioning Stability

• v0 is designated as a beta release, and may change.
• Anything v1 and beyond should try to support backwards compatibility 
• The version should be increased any time:
  • The existing contract is changed in a way that breaks backward compatibility.
• Versions should not be increased by:
  • Additive changes to the API that does not break the current contract.

Common Variables

Please refer to this list, and add to it, as we define our API vocabulary.

• We use the plural for our variable names by convention. This is still true if the API can only support a single user or course per request. 

Conventions on GET, POST, and PATCH

Use the following basic guidelines when constructing your views to ensure consumers of our APIs can anticipate their behaviors.

• GET - Read-only operations to return complete representations of objects based on their path.
• PATCH - Update operation where we assume the given object is a partial representation of the new object. PUT may be used, but we prefer to use PATCH, since it will be more likely that our operations are going to perform partial updates to any resource.
• POST - Used for Create or Update operations where only a part of the information is provided, and the service is responsible for constructing the complete object.

Technologies

Common technologies will reduce complexity and encourage conventions. 

• Django REST Framework - Commonly being used in new Python based projects.

Collections: Pagination and Handling Large Results

All our APIs should support pagination on list-based views if they may return a large number of results. The Django REST Framework supports pagination if your views are tightly coupled with your models.  If your API takes on the separation of a views --> API --> Data Layer, you may not be able to use these features.  Leveraging the Paginator from Django, you can still return a paginated response.  Here is an example decorator for doing so.

Pagination

• List Based Views should return Pages. 
• Pages will contain a list of objects
• Pages should have the following information:
  • Current Page
  • URL to previous and next
  • Maximum pages
• API should have its own default page sizes.

Handling Large Results

Some APIs may return more results than are required for each consumer. Mobile, for example, may only want to get a subset of the results from an API to reduce the cost of each request. One option is to implement parameter filtering on the API, where only the specified attributes are returned. This gets a little complicated when the API supports PUT requests, where a consumer of the API may only be working with the partial state of an object. 

A simpler approach is to structure your return values into compartmentalized data structures, providing refined URL paths to each smaller object in the result. This can reduce the cost of each request.

Example from Enrollment API:

This URL will get you a user's enrollment in a course, as well as all the enrollment information specific to the course itself:

/api/enrollment/v1/courses/edX/DemoX/Demo_Course

{
    "created": "2014-10-08T21:18:56Z", 
    "mode": "honor", 
    "is_active": true, 
    "course_details": {
        "course_id": "edX/DemoX/Demo_Course", 
        "enrollment_end": null, 
        "course_modes": [
            {
                "slug": "honor", 
                "name": "Honor Code Certificate", 
                "min_price": 0, 
                "suggested_prices": [], 
                "currency": "usd", 
                "expiration_datetime": null, 
                "description": null
            }
        ], 
        "enrollment_start": null, 
        "invite_only": false
    }, 
    "user": "steve"
}

Note that the course_details are a separate dict in the response. Allow a URL to refine down to just get that information:

/api/enrollment/v1/courses/edX/DemoX/Demo_Course/course_details/

{
    "course_id": "edX/DemoX/Demo_Course", 
    "enrollment_end": null, 
    "course_modes": [
        {
            "slug": "honor", 
            "name": "Honor Code Certificate", 
            "min_price": 0, 
            "suggested_prices": [], 
            "currency": "usd", 
            "expiration_datetime": null, 
            "description": null
        }
    ], 
    "enrollment_start": null, 
    "invite_only": false
}

Exception Handling

• See page 10 of the Apigee Conventions

• Use HTTP Error Codes. 
  • If you use an uncommon error code, document it in the base.
• Use 404 (Not Found) instead of 401 (Unauthorized) when requests look for other users' information.
  • For example, if "Steve" tries to upgrade his own enrollment to a verified certificate, he should be "unauthorized" because he should not have permission to upgrade himself.
  • However, if "Steve" tries to look up the enrollment for "Bob" in a course, he should receive a 404 "Not found", otherwise "Steve" can still deduce that "Bob" is enrolled.
• Use the following JSON error format:
  • developer_message
  • user_message (optional)
  • field_errors (if applicable)
    
    

{
    "developer_message" : "Verbose, plain language description of the problem for the app developer with hints about how to fix it.",
    "user_message":"Pass this message on to the app user if needed.",
    "field_errors": {
        "foo": {
            "developer_message": "",
            "user_message": ""
        }
    }
}

Open Question: If you want to return additional data or information with the error, such as a URL, or a list of valid request parameters, how do you attach this to the returned exception?

Combined responses from multiple resources

• See Page 16 of the Apigee Documentation
• e.g. ?fields=title,media:group(media:thumbnail)
• DRF does not easily support this. We may need to build additional support to make this easier to leverage for new APIs build with DRF.

Pagination

• Use Django REST Framework Pagination conventions instead of what is outlined in the Apigee documentation.

Documentation

• Use Swagger. http://swagger.io/
• ReadTheDocs to supplement Swagger, with full documentation of the API
• Re-use docstrings for both Swagger and ReadTheDocs
  • See this Enrollment API PR to see the effort required to build re-usable strings https://github.com/edx/edx-platform/pull/6447/files

Multiple Response Formats

• See Page 21 of the Apigee Documentation
• We will not use a suffix to define the format.
• We will use the Request / Response Headers for "Content-Type" to define the output.
• JSON is the default format.

Naming Conventions

• See Page 21 of the Apigee Documentation
• We will use Python conventions, instead of CamelCase, use_underscores.
• Do not expose database IDs where possible.

Authentication

Authorization

Eventing and Analytics

Notes from Discussions

Small section capturing notes from API discussions and meetings.

Discussion w/ Dave O. Matt D. and Steve S., Dec 5 2014

These are notes captured while discussing how to make some conventions and expectations for APIs built for the edX platform going forward.

• Turn off browse-able Django REST view for prod environment
• Doc set via Mark - contact
• Potential to build an API client (similar to BOTO)
• Error conventions (Django REST framework conventions)
• Pagination / Filtering 
  • Always page? (Yes, by convention) (service to service)
  • Filtering? (Matt may have done something like this to reduce via serializers) DynamicModelSerializer (filter=field,field2,field3)

Arch Lunch discussion, Dec 11 2014

• Also need:
  • Authentication
  • Authorization
  • Monitoring: healthcheck endpoints
  • Events/analytics
  • How to handle collections?
  • POST vs PUT vs PATCH
  • When do we depart from pure ReST?
  • Versioning and stability, of endpoints and the data they return
  • What does it mean to use more than one API at once?
  • Client libraries?
    • Python/Java/Obj-C etc?
    • Can they automate parallelism?
  • How to debug remote services??
  • Exception handling: return http error codes, or error objects?
  • Can we support in-process calls as well as HTTP calls?  ZeroMQ?

Other subjects we would like to discuss / be opinionated about

• • How normalized/denormalized should data be in our responses?
  • Endpoint organization, what if two APIs both deal with courses?
  • How discoverable should our services be?
  • How do we expose documentation and help?
  • Throttling

API Conventions Meeting, Dec 23 2014

• Read and discussed multiple points regarding the Apigee conventions:
  • edX REST API Conventions
• Still need to cover a few significant areas, such as versioning. A second meeting will be held on Dec 29, 2014.