Quick Reference · the distributed backend for Dask

dask.distributed cheat sheet

The distributed scheduler runs Dask across many machines. A Client submits work to a central Scheduler, which farms tasks out to Workers that run them and keep results in their own memory. You hold lightweight Futures — pointers to data that lives on the cluster — and workers swap data peer-to-peer. The trick: move computation to the data, and gather back only when you must.

setup / connect cluster / deploy futures data movement task control coordination gotcha ★ most common

Distilled & cross-checked against: distributed.dask.org (client, api, worker, locality, resilience, manage-computation) · docs.dask.org (deploying) · dask-jobqueue / dask-kubernetes / dask-cloudprovider docs — API for distributed ≈ 2026.7 (CalVer)

The architecture · Client submits · Scheduler coordinates · Workers compute & hold data

01Setup & Connectget a Client

pip install "dask[distributed]"★
Scheduler, workers, dashboard.
from dask.distributed import Client★
The entry point to everything.
client = Client()★
Starts a local cluster + connects.
client = Client("tcp://scheduler:8786")
Connect to a running cluster.
client.dashboard_link★
Live diagnostics at :8787.
client.close()
Disconnect (closes a Client() cluster too).

★The Architecturefive roles

Client★
Your session — submits work, holds Futures.
Scheduler
The brain (:8786): task graph + who-has-what.
Worker★
Runs tasks; keeps results in local RAM.
Nanny
Watches a worker, restarts it if it dies.
Dashboard
Live task stream & memory at :8787.

03Local Clusterone machine

from dask.distributed import LocalCluster
Multi-process cluster on your box.
cluster = LocalCluster(n_workers=4, threads_per_worker=2)★
Shape workers × threads.
client = Client(cluster)★
Connect the client to it.
LocalCluster(memory_limit="4GB")
Per-worker memory cap (spills over it).
Client(processes=False)
Threads only, in-process — easy debugging.

04Deploy to a Clustermany machines

SSHCluster(["head", "w1", "w2"])
Spin up workers over SSH.
SLURMCluster(...) # PBS · LSF★
HPC job queues — dask-jobqueue.
KubeCluster(...)
Kubernetes — dask-kubernetes.
EC2Cluster() # Fargate · Azure · GCP
Cloud VMs — dask-cloudprovider.
dask scheduler · dask worker <addr>★
The CLI — wire it up by hand.

05Scale & Restartsize the cluster

cluster.scale(20)★
Fixed number of workers.
cluster.adapt(minimum=1, maximum=50)★
Auto-scale to the workload.
client.wait_for_workers(4)
Block until N workers are up.
client.restart()
Restart all workers, clear cluster state.
client.retire_workers([...])
Drain data off, then remove.
client.shutdown()
Stop the scheduler + all workers.

06Futures · Submitreal-time tasks

fut = client.submit(fn, x)★eager
Runs now; returns a Future immediately.
futs = client.map(fn, seq)★
One Future per item.
client.submit(fn, fut)
Chain on a Future — no gather needed.
fut.result()★
Block, copy the result to the client.
fut.status # .done()
pending · finished · error.
client.submit(f, x, pure=False)
For impure fns (e.g. random) — new key each call.

07Futures · Collectawait results

client.gather(futs)★
Bring many results back at once.
wait(futs)
Block until all are finished.
for f in as_completed(futs):★
Handle each as soon as it finishes.
fut.exception() # .traceback()
Inspect a failed task.
progress(futs)
Live progress bar (notebook).
del fut★
Drop the pointer → frees worker memory.

08Move Data In & Outclient ⇄ workers

f = client.scatter(data)★
Push local data onto workers → Future.
client.scatter(data, broadcast=True)
Copy to every worker (shared lookup).
client.gather(futs)★
Pull results back to the client.
x = x.persist()★
Compute a collection, keep it on workers.
client.compute(x)
Collection → Future(s), lazily started.

09Cluster Memorywhere data lives

df = df.persist()★
Materialize across distributed RAM.
client.replicate(futs, n=3)
Extra copies for resilience & speed.
client.rebalance()
Even out memory across workers.
client.who_has(fut)★
Which workers hold this data.
client.has_what()
Keys held per worker.
client.nbytes(summary=True)
Memory by key type.

10Control Placementsubmit/map kwargs

submit(fn, x, workers=["alice"])★
Pin a task to specific workers.
allow_other_workers=True
Make the pin a soft preference.
resources={"GPU": 1}★
Require a resource-tagged worker.
priority=10
Higher priority runs first.
retries=3
Auto-retry a flaky task.
key="step-1"
Name the task (controls dedup).

11Run & Manage Tasksside effects

client.run(setup_fn)★
Run on every worker (imports, config).
client.run_on_scheduler(fn)
Run once on the scheduler process.
client.upload_file("mymod.py")
Ship local code to all workers.
fire_and_forget(futs)
Run even after you drop the Future.
client.cancel(futs)
Stop scheduled/running tasks.
client.retry(futs)
Re-run failed futures.

12Tasks from Tasksnested submits

from dask.distributed import worker_client
Talk to the cluster from inside a task.
with worker_client() as c:★
Submit more work mid-task.
secede()★
Leave the thread pool while waiting — no deadlock.
rejoin()
Rejoin once ready to compute again.
get_worker() # get_client()
The worker/client running this task.

13Locks & Eventscoordinate workers

from dask.distributed import Lock, Semaphore
Cluster-wide, named primitives.
with Lock("resource"):★
Mutual exclusion across the cluster.
Semaphore(max_leases=2, name="db")★
Cap concurrent access (e.g. a DB).
Event("go").wait() # .set()
Signal across tasks/workers.
MultiLock(["a", "b"])
Acquire several locks at once.

14Share Stateacross clients

Queue("q").put(fut) # .get()★
Pass Futures between tasks/clients.
Variable("v").set(fut) # .get()
A shared, named slot on the scheduler.
client.publish_dataset(df, name="train")★
Publish a collection for others to use.
client.get_dataset("train")
Another client picks it up — no recompute.
client.list_datasets()
What's published on the scheduler.

15Diagnose & Profilesee the cluster

client.dashboard_link★
Task stream · memory · progress (:8787).
with performance_report(filename="r.html"):★
Save a shareable HTML report.
with get_task_stream() as ts:
Capture per-task timings.
client.processing()
Tasks currently running per worker.
client.get_worker_logs()
Pull logs off the workers.
client.profile(filename="p.html")
Statistical profiler of recent work.

16Gotchascommon traps

client.gather(giant)floods client
Keep big data on workers; gather summaries.
client.scatter(big_local_df)bottleneck
Load with workers (read_parquet) instead.
one long GIL-holding taskblocks worker
Stalls heartbeats — release the GIL or use processes.
last future goes out of scopedata freed
Hold the Future or the result is dropped.
Client() # at import timespawn
Guard under if __name__ == "__main__":.

Four ideas that make distributed click

Peer-to-peer data locality, futures as pointers, loading data the right way, and elastic scaling. Based on the official Dask.distributed docs.

data moves worker-to-worker

The scheduler says who needs what; workers then fetch dependencies directly from each other. Computation is sent to where the data already sits.

futures are pointers

A Future is a tiny token in your session. The real object sits in a worker's memory. gather() copies it back — so do it sparingly.

load with workers, don't scatter

Reading data locally then scattering routes everything through your client. Let workers read storage in parallel instead.

scale vs adapt

scale(n) fixes the worker count. adapt(min, max) grows the pool when tasks pile up and shrinks it when the cluster goes idle.

Worth memorizing

Client holds Futuresthe data itself stays on the workers

scatter ≠ submitprefer loading data with workers

gather sparinglypull results back only when you must

persistcompute a collection, keep it in cluster RAM

:8786 / :8787scheduler port / dashboard port

workers= / resources=pin tasks to specific hardware

scale vs adaptfixed worker count vs elastic to load

secede()free a thread-pool slot during long waits

del the futurereleases its result from worker memory

Nanny restartsa dead worker comes back automatically

Dask.distributed cheat sheet · synthesized from the official docs at distributed.dask.org (client, api, worker, data locality, resilience, managing computation) and the deployment projects (dask-jobqueue, dask-kubernetes, dask-cloudprovider).
API shown for distributed ≈ 2026.7 (CalVer YYYY.MM.x, versioned with Dask) · Python 3.10–3.13 · the dashboard needs bokeh · pairs with all Dask collections (array, dataframe, bag, delayed).