In Otto, order fullfillment currently works synchronously, triggered by a django signal that is fired upon the completion of an order. The following are motivations for making fulfillment work asynchronously instead:
To summarize, adding asynchronous fulfillment capability will give us a scalable capacity to service orders insulated from the impacts of transient exceptions (network outages, maintenance windows / service disruptions). It will also better equip us to withstand sudden spikes in request volume and isolate their impacts from end users, or other services within the integrated system.
Summary:
Changing the call/response interaction would have a drastic effect on existing tests and development workflows. This impact can be largely neutralized:
That said, it may be necessary to make changes to the acceptance tests we currently use to validate staging deployments.
In edx.org and "fullstack" deployments there is already queue and worker infrastructure in place to service celery tasks owned by other parts of the system. There's an optimistic assumption that ecommerce requirements could be met by adding configurations to this existing infrastructure.
One particular impact to be prepared for is the increased complexity of troubleshooting / debugging. It is an unfortunate consequence of asynchronous processing that bugs and misconfigurations are more difficult to track down, since there may be no user-facing interaction or feedback to help diagnose mysterious behavior. To mitigate, we must ensure that logging, instrumentation, and monitoring are sufficiently comprehensive and verbose.
Users' perceived response times involving interactions with Otto should be faster after this change, and Otto will be more available to service high volumes of requests.
On the downside, however, this change will introduce the possibility that a user may visit their dashboard after successfully completing a transaction via Otto but before their order (course enrollment) has been fulfilled. This is particularly likely with Otto checkouts involving honor-mode (free) seats, since the user will be instantly redirected to their dashboard on completion of the order, and there's a good chance the fulfillment task will not yet have successfully completed before that time.
To mitigate this problem, we should start to create enrollments with a new status of 'pending' when registration happens via the LMS, and have the enrollment api clear this status when the user's enrollment is made active during fulfillment. This will permit us to display the enrollment immediately on the dashboard, with a link into the course disabled and accompanied by an indication that the seat is not yet available. This will reassure the user that their request has been acknowledged by the system and is in progress. While the user is viewing their dashboard, a script on running in the browser would check the server frequently for updates to the pending status, which would generally be expected to become active within a second or two - the client could then refresh the display in-place and permit the user to the fully active enrollment.