Celery Workers Configuration

Owned by Awais Jibran (Deactivated)
Apr 03, 2023
2 min read

This document will provide a basic understanding of celery workers in prod and their configurations and will also guide if you need to add a new worker what steps you need to take.

What?

There are certain celery workers in production that make asynchronous task management easy. Let’s celery be a task-queue that handles and executes the tasks on the workers.

We are not keeping celery configuration along with our code in edx-platform. so, this leads us to the next section where the celery configuration resides.

Where?

There few private GitHub repos which keep these settings which are listed below.

  • edx-internal - This repo keeps track of configuration of prod and stage.

  • edge-internal - This repo keeps track of configuration of edge only.

How?

There can be a bunch of questions that you may think of but not limited only to the following questions.

How many celery queues Prod - LMS use?

If you are looking for celery workers' information for prod-LMS, you would find it here >>CELERY_QUEUES >> lms.yml#L208. This list contains all of the queues LMS use. Similarly, there are settings for the stage in the stage folder.

How many celery workers are defined against my queue in Prod - LMS?

You can find all the worker's information against each queue in ansible/vars/edx.yml >> EDXAPP_CELERY_WORKERS file.

How can I add a new queue/worker in PROD - LMS?

You need to think about a few basic questions before you add a new worker.

  1. Do you need to create the queue/worker for the stage as well?

  2. Do you need to create the queue/worker for the edge as well?

  3. Do you need to create the queue/worker for the sandbox as well?

Once you know all the answers proceed by creating a PR against edx-internal. See this PR for reference (https://github.com/edx/edx-internal/pull/1667/files)

  1. Define a routing key in edx.yml file. This key is used with celery tasks to direct them to the queue.

    1. You can define a routing key as EDXAPP_SOFTWARE_SECURE_VERIFICATION_ROUTING_KEY: "edx.lms.core.software_secure_verification"

    2. Add the routing key in EDXAPP_CELERY_WORKERS list with all of the configurations.

    3. - queue: software_secure_verification service_variant: lms concurrency: 1 monitor: True max_tasks_per_child: 5000

       

    4. Define a new queue CELERY_QUEUES` in edx-remote-config/prod/lms.yml.

Once you have defined it, set up the same configuration for stage, edge. Feel free to edit and add new questions when you have.