Distributed task queue · async jobs, scheduling, workflows · verified against Celery 5.6 (2026)

Celery cheat sheet

Celery runs Python work outside the request — async tasks, scheduled jobs, and multi-step workflows — across a pool of workers that pull from a broker (Redis/RabbitMQ). You define tasks with @app.task, call them with .delay(), and a worker executes them; results (optional) go to a result backend. This sheet targets Celery 5.6 (Python 3.9+). Common in ML/data for training jobs, batch inference, emails, ETL and anything slow that shouldn't block a web response.

app & tasks calling & results workers & beat canvas (workflows) config & ops gotcha ★ most common

Verified 2026-08-24 against the official docs at docs.celeryq.dev (Celery 5.6) and the celery/celery repo. Needs a broker (Redis or RabbitMQ); a result backend is optional but required to read return values.

Outline

Define a task → call .delay() → run a worker. Add a result backend to read outputs, beat to schedule, and canvas to compose multi-step pipelines.

Define

An app tied to a broker, and functions marked as tasks.

1Install & app5.6

pip install "celery[redis]" # or celery[amqp] for RabbitMQ★
Install the broker extra you use.
from celery import Celery app = Celery("proj", broker="redis://localhost:6379/0", backend="redis://localhost:6379/1")★
The app is your entry point. Broker transports messages; backend stores results (omit if you don't need return values).
app.config_from_object("celeryconfig") · app.autodiscover_tasks()
Load settings from a module; auto-find tasks.py in your apps (Django-style).

2Define tasks@app.task

@app.task def add(x, y): return x + y★
Any function becomes a task. Keep args/return JSON-serializable (the default serializer).
@app.task(bind=True, max_retries=3, name="proj.add") def add(self, x, y): ...★
bind=True gives self (for retries/state); set an explicit name so it's stable across refactors.
@app.task(acks_late=True, time_limit=300, rate_limit="10/m")
acks_late re-queues if a worker dies mid-task; time/rate limits protect the pool.

Call

Enqueue tasks and (optionally) collect results.

3delay / apply_asyncenqueue

add.delay(2, 3)★
The shortcut — enqueues and returns immediately with an AsyncResult. Non-blocking.
add.apply_async((2, 3), countdown=10, queue="heavy", priority=5)★
Full control: countdown/eta to delay, queue to route, expires, retry, headers.
add.apply_async((2,3), link=notify.s(), link_error=on_fail.s())
Attach success/error callbacks (task signatures).
add(2, 3) # calls the plain function synchronously — NOT queued
Calling a task directly runs it inline like a normal function — useful in tests, but it bypasses Celery entirely.

4Results & retriesoutcomes

res = add.delay(2, 3) res.id · res.status · res.ready() · res.get(timeout=10)★
get() blocks until done (needs a result backend). States: PENDING→STARTED→SUCCESS/FAILURE/RETRY.
from celery.result import AsyncResult AsyncResult(task_id, app=app).get()
Re-attach to a task by id from elsewhere (e.g. a web view polling status).
try: ... except Exc as e: raise self.retry(exc=e, countdown=5)★
Retry on failure (needs bind=True). Or use autoretry_for=(Exc,) + retry_backoff=True on the decorator.
res.get() inside a taskavoid
Blocking on a subtask's result from within a task can deadlock the pool — use a chord/callback instead.

Run

Start workers, and schedule periodic tasks.

5Workers & queuesexecute

celery -A proj worker --loglevel=INFO★
Start a worker for the proj app. -A points at the module holding app. Nothing runs until a worker is up.
celery -A proj worker -c 8 --pool=prefork # 8 processes★
-c = concurrency. Pools: prefork (CPU, default), gevent/eventlet (I/O-bound), threads, solo.
celery -A proj worker -Q heavy,default★
Consume specific queues — route slow tasks to dedicated workers so they don't starve fast ones.
celery -A proj inspect active · control revoke <id> · purge
Introspect/control running tasks; purge empties a queue (careful).

6Scheduling (beat)periodic tasks

app.conf.beat_schedule = { "nightly": {"task": "proj.cleanup", "schedule": crontab(hour=3, minute=0)}}★
Define periodic jobs; from celery.schedules import crontab. A float schedule = every N seconds.
celery -A proj beat --loglevel=INFO★
The beat process is the scheduler — it enqueues periodic tasks that workers then run. Run exactly one beat.
only ONE beat instancegotcha
Multiple beats = duplicate scheduled jobs. Use a single beat (or a lock/DB scheduler like django-celery-beat) in HA setups.

Compose (Canvas)

Build multi-step workflows out of task signatures.

7Signatures & chainspipelines

sig = add.s(2, 3) # a "signature": a task + its args, not yet run★
.s() packages a call. .si() = immutable (ignores the previous task's result).
from celery import chain chain(add.s(2, 3), mul.s(10))() # (2+3)*10 — output feeds next★
Chain = sequential pipeline; each task's result becomes the next's first arg. Also written add.s(2,3) | mul.s(10).
add.s(2, 3).set(queue="heavy", countdown=5)
Attach options to a signature before composing.

8Groups & chordsparallel + join

from celery import group group(add.s(i, i) for i in range(10))()★
Group = run many tasks in parallel; the result is a list of their results.
from celery import chord chord(group(add.s(i,i) for i in range(10)))(summarize.s())★
Chord = a group + a callback that runs once all finish, receiving the list of results. Map-reduce in Celery.
chain(group(...), chord(...), ...) # nest freely
Canvas primitives compose — build complex DAGs from chain/group/chord.

Operate

Production config and monitoring.

9Config & monitoringproduction

app.conf.update(task_serializer="json", accept_content=["json"], result_expires=3600, task_acks_late=True, worker_prefetch_multiplier=1)★
Sane defaults: JSON only (never pickle from untrusted sources), expire results, prefetch=1 for long tasks so they spread evenly.
task_routes = {"proj.heavy_*": {"queue": "heavy"}}★
Route tasks to queues by name pattern — the basis of isolating workloads.
pip install flower · celery -A proj flower★
Flower = web dashboard for tasks/workers/queues. Also celery events, Prometheus exporters, OpenTelemetry.
broker_connection_retry_on_startup = True
Common resilience knob. Idempotent tasks + acks_late is the recipe for at-least-once safety.

!Common gotchasread before shipping

pass ids, not objects
Task args are serialized (JSON). Pass a DB id and re-fetch inside the task — don't pass ORM objects/large payloads.
tasks should be idempotent
With acks_late/retries a task can run more than once. Design for at-least-once delivery (use a dedupe key / upsert).
no backend = no results
.get() hangs/errors without a result backend configured. Many tasks are fire-and-forget and don't need one.
never pickle from untrusted brokers
Stick to JSON serialization. Pickle can execute arbitrary code on the worker.
one beat, and mind prefetch
Duplicate beats duplicate jobs; high prefetch_multiplier makes one worker hog long tasks — set it to 1 for those.

Worth memorizing

broker moves messages, backend stores resultsRedis/RabbitMQ broker; backend optional

@app.task then .delay()nothing runs until a worker is up

.delay() vs calling directlydelay enqueues; add(2,3) runs inline, bypassing Celery

apply_async for controlcountdown/eta, queue, priority, link callbacks

bind=True for retriesself.retry(exc=e); or autoretry_for + retry_backoff

-Q to isolate workloadsroute heavy tasks to dedicated workers

beat schedules, workers executerun exactly one beat

canvas: chain / group / chordsequential / parallel / parallel-then-callback (map-reduce)

pass ids, stay idempotent, JSON onlyat-least-once delivery; never pickle untrusted