Currently, LMS uses 3 Celery workers: lms_default_1, lms_high_1 and
lms_high_mem_1. Each Celery worker sends messages to a single queue:
edx.core.default, edx.core.high and edx.core.high_mem, respectively.
The number of child processes per Celery worker is set to 1. Due to
this configuration, any task in a queue blocks all other tasks.
Currently, the Celery check task submitted by the /heartbeat?extended
LMS HTTP API endpoint runs in the default queue. When some slow task
(eg course grades creation) is sent to the default queue, it will
block the Celery check task, which will expire and the heartbeat endpoint
will fail. This patch moves the task to another queue which has
only shorter tasks and in which this problem will not occur.
Use a Django setting for the Celery check task routing key so that
it can be overriden by individual OpenEDX instances via JSON
env files.
We may want to add additional heartbeat checks, particularly depending
on the context (e.g. different environments). This is meant to be a
start at enabling a more plug-in style heartbeat check that provides
more detailed diagnostics if desired.
