...
- Jobs will use
celery&/or remote services. The existing framework forinstructor_taskmay be a good starting point. Evaluating its fit will be part of the project plan. - We will develop a pattern of use which allows independent deployability, remoting, and upgrade in place. Not all of these must exist in the first version, just be possible.
- We will use celery & add-ons to do state introspection, high-availability
- We will define an interface for notifications & result delivery allowing multiple back-ends
- We will define an auto-notifier on process initiation and termination which jobs can disable (to prevent flooding for highly decomposed jobs)
- We will define a mechanism for handling fine-grained highly decomposed jobs so that they do not require proportional memory for tracking (e.g., bulk email w/ one job per recipient)
- We may define a thin interface over celery and its add-ons so that we can replace celery with something which can also handle remote services (FSM)
- We will need to come up with a versioning mechanism for job workers w/ some level of backward compatibility (capable of handling jobs for the most recent previous version as well as the current version). (could be later release but design should be aware of this future requirement)
- Storage: we will use
django-storages, but we need to figure out all of the above functional requirements on top of the storage. Ensuring proper authorization is crucial (can't have unauthorized users find and download results just due to url leakage or phishing). I don't see anything in django storage for handling auth.
...