Lightweight pipelining: parallelism · caching · persistence · the engine under scikit-learn · verified against joblib 1.5.x (2026)

joblib cheat sheet

joblib does three things really well: run embarrassingly-parallel loops with Parallel(delayed(...)), cache expensive function results to disk with Memory, and persist Python objects — especially big NumPy arrays — fast with dump/load. Zero required dependencies, a vendored loky process pool that survives crashes, and memory-mapping so worker processes share arrays without copying. It's the parallel/persistence engine inside scikit-learn. Targets joblib 1.5.x, Python 3.9+.

parallel backends & control caching (Memory) persistence (dump/load) gotchas & ecosystem gotcha ★ most common

Verified 2026-08-31 against the official docs at joblib.readthedocs.io (joblib 1.5.3, released 2025-12-15; Python 3.9+). No mandatory dependencies (vendors loky & cloudpickle). Default parallel backend is loky (separate processes); use threading for I/O- or C-bound work that releases the GIL.

Outline

Parallel(delayed(f)(x) for x in xs) to parallelize; Memory to cache; dump/load to persist. Then pick the right backend and mind the pickling rules.

Parallel

Turn a Python loop into parallel jobs with two names: Parallel and delayed.

1Install & first parallelParallel + delayed

pip install joblib★
No required deps. Python 3.9+. (Often already present via scikit-learn.)
from joblib import Parallel, delayed Parallel(n_jobs=4)(delayed(f)(x) for x in range(10))★
The core idiom: delayed(f)(x) packages a call without running it; Parallel(n_jobs=) runs the whole generator across workers and returns a list in order.
import math Parallel(n_jobs=-1)(delayed(math.sqrt)(i**2) for i in range(10))★
n_jobs=-1 uses all cores; -2 = all but one. Results come back in the same order as the input.
# delayed(f(x)) -> WRONG: runs f now, in the main processgotcha
It's delayed(f)(x), not delayed(f(x)). The first defers the call; the second executes it immediately.

2Parallel patternscommon shapes

results = Parallel(n_jobs=4)( delayed(process)(item) for item in dataset)★
Map a function over a collection — the 90% use case. Each call must be a picklable, ideally pure function.
Parallel(n_jobs=4)(delayed(f)(x, y, key=z) for x,y,z in args)
Positional and keyword args pass straight through delayed(f)(...).
with Parallel(n_jobs=4) as parallel: a = parallel(delayed(f)(x) for x in xs) b = parallel(delayed(g)(y) for y in ys)★
Reuse one Parallel context across several batches to keep the worker pool warm (avoids re-spawning processes).
# nested loops: parallelize the OUTER loop only
Don't nest Parallel calls naively — joblib coordinates nesting, but usually parallelize the coarsest loop and keep inner work sequential.

Backends & Control

Choose processes vs threads, and watch progress or stream results.

3Backends & n_jobsprocesses vs threads

Parallel(n_jobs=4, backend="loky") # default: separate processes★
loky (default) uses robust worker processes — bypasses the GIL, best for CPU-bound pure-Python work. Survives worker crashes.
Parallel(n_jobs=4, backend="threading")★
threading shares memory (no pickling) — use for I/O-bound work or C code that releases the GIL (NumPy, pandas). Not for pure-Python CPU loops.
from joblib import parallel_config with parallel_config(backend="threading", n_jobs=8): Parallel()(delayed(f)(x) for x in xs)★
parallel_config sets the backend/n_jobs for a block — how you override a library's internal Parallel (e.g. scikit-learn's).
Parallel(n_jobs=4, prefer="threads") · require="sharedmem"
prefer is a soft hint; require="sharedmem" forces a shared-memory backend. Dask/Ray register as backends too.

4Progress & return_asobserve & stream

Parallel(n_jobs=4, verbose=10)(delayed(f)(x) for x in xs)★
verbose>0 prints progress/ETA to stderr. Higher = more frequent updates.
for r in Parallel(n_jobs=4, return_as="generator")( delayed(f)(x) for x in xs): print(r) # results as they arrive, in order★1.3+
return_as="generator" streams results (bounded memory over many tasks). "generator_unordered" yields whichever finishes first.
Parallel(n_jobs=4, batch_size="auto", pre_dispatch="2*n_jobs")
Tune task batching & how many tasks are dispatched ahead — raise batch_size for many tiny tasks; limit pre_dispatch to cap memory.
Parallel(n_jobs=4, timeout=60)
Per-task timeout raises TimeoutError. Combine with retries in your function for robustness.

Caching (Memory)

Memoize expensive functions to disk so you compute each result once.

5Memory cachingdisk memoization

from joblib import Memory memory = Memory("./cachedir", verbose=0)★
A Memory object caches results under a directory. Persists across runs — unlike functools.lru_cache (in-memory only).
@memory.cache def expensive(x, y): ... return result★
Decorate a function; the first call computes & stores, later calls with the same args load from disk. Great for data prep & feature engineering.
cached = memory.cache(expensive) # wrap without decorator cached(1, 2)
Same effect functionally. Keying is by the function source + hashed arguments (NumPy arrays are hashed by content).
# args must be hashable/picklable; result must be picklablegotcha
Cache invalidates if you change the function body or arg values. Don't cache functions with side effects or unhashable args.

6Cache managementcontrol it

memory.reduce_size(bytes_limit="500M")★
Evict least-recently-used entries to keep the cache under a size (or age_limit=, items_limit=).
memory.clear(warn=False) # wipe the whole cache
Delete everything. expensive.clear() clears just one cached function.
result, meta = cached.call(1, 2) # force recompute + metadata
.call() bypasses the cache and returns call metadata; .check_call_in_cache(1,2) tests for a hit.
@memory.cache(ignore=["verbose"])
ignore= excludes non-result-affecting args (loggers, flags) from the cache key.

Persistence (dump / load)

Fast, compressed serialization tuned for large NumPy arrays.

7dump / loadsave objects

from joblib import dump, load dump(model, "model.joblib") model = load("model.joblib")★
Pickle-based, but far faster & smaller than plain pickle for objects holding big NumPy arrays — the standard way to save a scikit-learn model.
dump(obj, "f.joblib") # also accepts a file object / BytesIO
dump returns the list of files written. load takes a path or file-like object.
# load() unpickles — NEVER load untrusted .joblib filessecurity
Deserialization can execute arbitrary code. Only load files you trust. Pin library versions — an object may not load under a different scikit-learn/NumPy.

8Compression & mmapsize & speed

dump(obj, "f.joblib", compress=3) # zlib level 0-9 dump(obj, "f.joblib", compress=("lz4", 3)) # pick codec★
compress shrinks files a lot (great for models over a network). Codecs: zlib, gzip, bz2, lzma/xz, lz4 (fastest). Extension can auto-select (f.joblib.gz).
big = load("arrays.joblib", mmap_mode="r")★
Memory-map arrays instead of loading them into RAM — multiple processes share one on-disk copy read-only. (Doesn't combine with compression.)
# joblib memmaps large args to Parallel workers automatically
In loky, big NumPy arrays passed to tasks are memory-mapped to workers automatically (max_nbytes threshold) — zero-copy sharing.

Gotchas & Ecosystem

The pitfalls that bite, and where joblib sits.

9Gotchasavoid these

# tiny fast tasks: parallel overhead > the workgotcha
Process startup & pickling cost real time. Parallelize coarse chunks; for millions of microtasks, batch them or stay sequential/vectorized.
# everything passed to loky workers must be picklablegotcha
Lambdas/closures/local functions can fail to pickle (joblib vendors cloudpickle to help). Define worker functions at module top level.
if __name__ == "__main__": Parallel(...)(...) # guard on Windows/spawntip
On Windows/macOS (spawn start method), wrap the top-level parallel call in a __main__ guard or workers re-import your script.
# don't over-subscribe: joblib n_jobs × BLAS threads = CPU thrash
If your function already uses multithreaded NumPy/BLAS, high n_jobs oversubscribes cores. joblib+loky mitigates via threadpoolctl, but cap threads if you see contention.

10Ecosystemwhere it fits

# scikit-learn uses joblib for n_jobs= and model persistence★
Every sklearn n_jobs= is joblib under the hood; joblib.dump(clf, ...) is the recommended way to save fitted estimators.
import joblib with joblib.parallel_config(backend="dask"): clf.fit(X, y) # distribute sklearn across a cluster
Register Dask or Ray as the backend to scale the same Parallel code from one machine to a cluster.
# vs alternatives
multiprocessing: lower-level, less robust. concurrent.futures: stdlib, no memmap/backends. Dask/Ray: full distributed frameworks. joblib is the sweet spot for single-machine science code.
joblib.hash(obj) · joblib.cpu_count()
Handy utilities: content hashing (used by Memory) and a safe CPU count that respects cgroup/affinity limits.

Worth memorizing

Parallel(n_jobs=)(delayed(f)(x) for x in xs)the core idiom

delayed(f)(x)NOT delayed(f(x)) — defer the call

n_jobs=-1all cores; -2 = all but one

backend loky vs threadingCPU (processes) vs I/O/GIL-free (threads)

parallel_config(...)override a library's backend/n_jobs

return_as="generator"stream results, bounded memory

Memory("./dir").cachedisk memoization across runs

reduce_size / clearbound / wipe the cache

dump / loadfast persistence for NumPy-heavy objects

compress= / mmap_mode="r"smaller files / zero-copy load

load = unpicklenever load untrusted files

coarse tasks + picklable fnsavoid overhead & pickle errors

joblib cheat sheet · v0 · content verified 2026-08-31 against joblib.readthedocs.io (joblib 1.5.3, released 2025-12-15, Python 3.9+).
Three tools in one: Parallel(delayed(f)(x) for ...) for embarrassingly-parallel loops (loky processes by default, threading for GIL-free work), Memory().cache for cross-run disk memoization, and dump/load for fast, compressible, memory-mappable persistence of NumPy-heavy objects (the standard way to save scikit-learn models). No mandatory dependencies; scales to Dask/Ray via pluggable backends.
Star (★) marks the most common operations. Never load() untrusted files, and parallelize coarse, picklable work.

Outline

Parallel

Backends & control

Caching

Persistence

Gotchas & ecosystem

Parallel

Backends & Control

Caching (Memory)

Persistence (dump / load)

Gotchas & Ecosystem

Worth memorizing