...
The way we deploy and host worker processes in OpenEdX is one of the main changes introduced under this proposal.
Worker Application
Download example code: worker.tar.gz
A number of long-running processes are implemented in edx-platform as Celery tasks. Presently, these tasks are deployed as standalone worker processes, meaning that in addition to managing edx-platform deployed as a WSGI process on web servers, it is also running on separate hosts under a Celery process. This model does not scale well and makes it easy to exploit Celery anti-patterns, in particular the sharing of Django models between application and task code. As we decompose edx-platform into an increasing number of smaller services, we want to avoid the complexity introduced by models shared in this way by separating tasks from the rest of the application they service.
...