Quick Reference · parallel & distributed computing in Python

dask cheat sheet

Dask has three moving parts: a collection (a DataFrame, array, or bag split into many small pieces) records the operations you ask for into a lazy task graph, and a scheduler runs that graph in parallel. Nothing actually runs until you call .compute(). Learn that one loop and the whole API stops being a list to memorize.

setup / import create / load transform (lazy) execute / trigger cluster / scheduler gotcha ★ most common

Distilled & cross-checked against: docs.dask.org · distributed.dask.org · ml.dask.org · Dask cheat sheet (dask/dask) · Anaconda · Coiled · Saturn Cloud — API for Dask ≈ 2026.7 (CalVer)

The mental model · collection → lazy task graph → scheduler → result

01Setup & Importonce per environment

pip install "dask[complete]"★
Full install — array/dataframe/bag + distributed.
conda install dask -c conda-forge
Same, via conda-forge.
import dask.dataframe as dd★
pandas at scale.
import dask.array as da
NumPy at scale.
import dask.bag as db
Messy / semi-structured data.
import dask
For delayed & dask.compute.
dask.__version__
CalVer, e.g. '2026.7.1'.

★Pick a Collectionwhich API do I want?

dask.dataframe★
Bigger-than-memory / many pandas frames.
dask.array★
Large N-dimensional NumPy arrays.
dask.bag
JSON/log/text objects — pre-DataFrame ETL.
dask.delayed★
Wrap arbitrary Python into a custom graph.
distributed futures
Real-time, eager tasks (needs a Client).
dask_ml
Scalable scikit-learn-style ML.

03DataFrame · Load & Saveread with Dask

dd.read_csv("data/*.csv")★
Many files → one lazy DataFrame (globs OK).
dd.read_parquet("data/")★
Preferred format — columnar, splittable.
dd.read_parquet(p, columns=[...])
Column projection — reads far less.
dd.from_pandas(df, npartitions=8)
Split an in-memory pandas frame.
dd.read_sql_table(…)
Pull from a database in parallel.
ddf.to_parquet("out/")★
Write one Parquet file per partition.

04DataFrame · Transformpandas API, lazy

ddf[ddf.x > 0]★lazy
Boolean row filter — same as pandas.
ddf.groupby("k").y.mean()★
Split-apply-combine across partitions.
ddf.assign(z=ddf.x + ddf.y)
Add a derived column.
ddf.merge(other, on="id")
Join — a shuffle unless index-aligned.
ddf.x.value_counts()
Frequency table of a column.
ddf.map_partitions(func)★
Run a pandas function on each partition.

05Array · CreateNumPy in chunks

da.from_array(x, chunks=(1000,1000))★
Wrap a NumPy / HDF5 / Zarr array.
da.ones((10000,10000), chunks=1000)
ones/zeros/full/arange, chunked.
da.random.random((10000,10000))★
Chunked random data for testing.
da.from_zarr("data.zarr")
Cloud-friendly, chunk-aligned reads.
x.chunks
Inspect the chunk grid (tuple of tuples).
x.rechunk((2000,2000))
Re-tile — may move data between chunks.

06Array · OperateNumPy API, lazy

x + x.T★lazy
Elementwise math & broadcasting.
x.mean(axis=0)★
Reductions along axes (sum, std…).
x @ y
Matrix multiply / dot product.
da.linalg.svd(x)
Parallel linear algebra, QR, etc.
x.map_blocks(func)★
Apply a NumPy function per chunk.
da.map_overlap(f, x, depth=1)
Stencils — shares a halo of neighbor rows.

07Bag · Messy DataJSON, logs, objects

db.read_text("*.json")★
Lines of text → a bag of strings.
db.from_sequence(seq, npartitions=4)
Any Python iterable → a bag.
b.map(json.loads)★
Apply a function to every element.
b.filter(lambda d: d["ok"])
Keep elements passing a predicate.
b.pluck("name")
Grab one field from each dict.
b.foldby(key, binop)
Streaming group-and-reduce.
b.to_dataframe()★
Hand cleaned records to a DataFrame.

08Delayed · Any Codecustom task graphs

@dask.delayed★lazy
Decorate a function so calls become tasks.
dask.delayed(func)(a, b)
Wrap a single call without decorating.
total = dask.delayed(sum)(parts)
Chain delayed results into a graph.
total.compute()★
Run the whole graph in parallel.
# many delayed calls,
Rule: build lots of tasks, then compute once.

09Trigger Executionturn graph into data

x.compute()★
Run → concrete pandas / numpy / list.
dask.compute(a, b, c)★
Compute several at once — shares subresults.
x.persist()★
Run now, keep as a Dask object in memory.
x.compute(scheduler="processes")
Override the scheduler for this call.
ddf.head()
Small peek — runs just the first partition.

10Cluster & Schedulerthe execution engine

from dask.distributed import Client★
The "advanced" scheduler — local or remote.
client = Client()★
Spin up a local cluster + dashboard.
Client(n_workers=4, threads_per_worker=2)
Shape workers × threads explicitly.
Client("tcp://scheduler:8786")
Connect to an existing cluster.
client.dashboard_link★
Live diagnostics at :8787.
dask.config.set(scheduler="threads")
threads · processes · synchronous.
client.close()
Shut the cluster down.

11Futures · Real-timeeager · needs Client

fut = client.submit(func, x)★
Starts running immediately (not lazy).
futs = client.map(func, seq)
One future per item.
fut.result()
Block until done, gather one result.
client.gather(futs)★
Collect many results back to the client.
as_completed(futs)
Iterate results as each finishes.
client.scatter(data)
Pre-place shared data onto workers.

12Partitions & Chunksthe performance lever

ddf.npartitions★
How many pandas pieces make up the frame.
ddf.repartition(partition_size="100MB")★
Retile toward the ~100 MB sweet spot.
ddf.repartition(npartitions=10)
Fewer, fatter partitions after a big filter.
ddf.set_index("time")
Sorted index → fast loc & joins (expensive).
ddf.divisions
The index bounds of each partition.
x.rechunk("auto")
Let Dask pick array chunk sizes.

13Inspect & Diagnosesee what's happening

x.visualize()★
Draw the task graph (needs graphviz).
x.dask
The underlying HighLevelGraph.
from dask.diagnostics import ProgressBar
Progress bar for the local scheduler.
with ProgressBar(): x.compute()
Wrap a compute to watch it run.
client.dashboard_link★
Task stream, memory, workers — live.

14Gotchascommon anti-patterns

for i in …: x.compute()serial
Computes one at a time → dask.compute(*xs).
da.from_array(np.load(f))embeds
Loads locally into the graph — read with Dask.
huge / tiny partitionsOOM/overhead
Too big = crash; too small = slow. Aim ~100 MB.
ddf.set_index(…) # in a loopshuffle
A costly reshuffle — do it once, then persist.
ddf.iloc[5]unsupported
No positional row indexing across partitions.

Four ideas that make Dask click

The distinctions that trip people up most — laziness, partitioning, compute vs persist, and choosing a scheduler. Based on the official Dask docs diagrams.

lazy build, then one compute

Every op just adds a node to the graph. Work happens only at .compute() — so batch your work and call it once.

one Dask object = many small ones

A DataFrame is a stack of pandas partitions; an array is a grid of NumPy chunks. Dask coordinates the pieces.

compute vs persist

compute pulls one concrete result back to your client. persist runs the graph but leaves results spread across worker memory as a Dask object.

which scheduler?

Threads share memory (great for NumPy/pandas, which release the GIL). Processes suit text & dicts. The distributed Client scales to many machines — and works on one.

Worth memorizing

lazy by defaultnothing runs until .compute()

compute ≠ persistcompute → concrete on client; persist → Dask object in RAM

~100 MBtarget size of each partition / chunk

one computedask.compute(*xs) — never compute in a loop

threads vs processesthreads for numeric; processes for text/dicts

delayed ≠ futuresdelayed is lazy; futures are eager (need a Client)

read with Daskdd.read_csv — don't pandas-read then hand it over

set_indexexpensive shuffle — do it once, then persist

:8787the dashboard is your best debugging tool

start smallif pandas/NumPy already fit in RAM, skip Dask

Dask cheat sheet · synthesized from the official references at docs.dask.org, distributed.dask.org, ml.dask.org, the dask/dask cheat sheet, and community guides (Anaconda, Coiled, Saturn Cloud).
API shown for Dask ≈ 2026.7 (CalVer YYYY.MM.x) · Python 3.10–3.13 · the query-optimizing DataFrame (dask-expr) is now the default backend.