Page Comparison

...

• Develop a base pattern & framework for asynchronous tasks to use
  
  • An asynchronous task is one which does not block the user while it runs. The user may navigate to other pages or log out without losing her/his ability to retrieve results. (H)
    • The pattern must provide a common way to report failures. (H)
    • The pattern must provide a common way for the user to find the job output--any resulting files (reports, tar balls, etc). (H)
    • The system should use the same notification mechanism to report status and results as the system uses for other asynchronous notifications. (H)
    • The user must be able to check or be notified about (pull v push) the status of the task. (M)
    • The pattern must handle massive fanout such as one job per student without falling over while still supporting restart, upgrade, failover, etc. (M)
    • The reusable pattern must provide a common way to declare the states and state transitions of jobs so that GUIs can show the user what steps will occur as well as which have occurred along with dependencies. (M, need at least pending, in progress, complete, and failed/canceled)
    • The pattern should provide a common way to handle retries or alternative processes on exceptions (L)
    • The system should provide user control for cancelling jobs: (L)
      
      • cancelling some processes may leave the course in a broken state upon cancellation depending on the modulestore. We'll need to decide whether such processes should refactor.
        • Canceling an import in split will work right but in old mongo it will have made some but not all changes.
  • The platform should not block on queued jobs and could even possibly remote the job (H except remoting)
  • Jobs should be able to output reports, zip files, or other assets using a common storage, naming, and auth system (H)
  • The platform should allow devops or developers to add or configure other storage backends without changing any asynchronous task code (H)
  • We should decide on a retention policy for historical job information and perhaps make it configurable (job submission data & the finite state machine transition records) (H)
  • The storage system for job output:
    • Job output should associate w/ the job information (H)
    • Course teams should be able to delete job outputs (H)
    • Only authorized users should be able to access job output (H)
    • Course team members should be able to find job output from other team members or past jobs (perhaps configurable per job or by job type?) (L)
    • Task definition developers should be able to indicate that the output is use-once and should be deleted upon download v kept until course ended or a specific date or indefinitely (L)
    • The storage system should allow append so that small subjobs can append results to a base file created by the parent process (M)
  • The user should be able to retrieve large results without streaming them through the app server (the app should use a storage system accessible to the webserver w/ appropriate auth) (M, piggyback on assets approach for storage only)
  • Asynchronous task definitions (the code defining the task) should be in separate repos without requiring all of edx-platform
    
    • Asynchronous task definitions should be separately deployable (deployable w/o shutting down the platform) (L)
    • There should be a sound method to upgrade task definitions without breaking queued or running jobs (H)
  • The system should include manual job invocation and status checking via command line (L)
• Implement an example using this pattern which others can easily copy to adapt to their use (_)
• Implement this pattern for import and export (H)

...