...
The E-Commerce team is making plans to implement Asynchronous Order Fulfillment in order to achieve a more resilient integration between Otto and the LMS. In discussions about this with DevOps, we have recognized the opportunity to make some improvements to the way we build, deploy, and execute asynchronous tasks for our IDAs.
...
In the event of a task's interface changing, different versions of the same task can be distinguished using task names. For example, the function name of a fulfillment task might be fulfill_product, but its v0 and v1 implementations could be distinguished by task name: fulfill_product_v0 as opposed to fulfill_product_v1. The application then calls tasks by name using Celery's send_task(). Incrementing the version on a task should only be necessary when circumstances dictate that the function/task signature be changed. These situations should be rare, but when they inevitably do arise, we have a way to deal with them. We shouldn't need to increment a task's version when changing the version of an API consumed by the task, for reasons described above.
Performance
(This discussion leaves aside the performance impacts of migrating any particular synchronous platform feature to work asynchronously - these are complex and difficult to generalize as each workflow is different.)
The performance of individual tasks will vary entirely based on what they, and the APIs upon which they depend, are executing. One risk in particular is blocking a queue of small, high-frequency tasks behind one or a few heavier, long-running ones. This implies a need to classify those tasks into worker pools by their "size" so that we can progressively fine-tune the provisioning for each category of task. (This further implies that we provide some metadata associated with tasks that can be used to automate that classification, although that may not necessarily need to be developer-defined.) Teams managing their IDAs' worker pools will be expected to divide their tasks across queues as necessary to avoid starving tasks and ensure that tasks are prioritized appropriately.