Hugging Face Accelerate Cheat Sheet

accelerate cheat sheet

Accelerate lets one PyTorch training script run unchanged on CPU, a single GPU, multi-GPU, multi-node, or TPU — with mixed precision (incl. fp8), FSDP and DeepSpeed a config away. You add ~4 lines (Accelerator(), prepare(), accelerator.backward(), drop manual .to(device)) and launch with accelerate launch. It's the distributed engine under 🌐 Transformers' Trainer, usable standalone. This sheet targets Accelerate 1.x.

the 4 code changes launch precision & accumulation distributed helpers scale & big models gotcha ★ most common

Verified 2026-08-24 against the official docs at huggingface.co/docs/accelerate (Accelerate 1.x) and the huggingface/accelerate repo. Wraps native PyTorch DDP/FSDP/DeepSpeed; your model/optimizer/loop stay ordinary PyTorch.

3config & launchCLI

accelerate config★
Interactive Q&A (num GPUs, mixed precision, FSDP/DeepSpeed) → writes default_config.yaml. accelerate config --help for non-interactive.
accelerate launch train.py --args★
The launcher — spawns the right number of processes. Replaces python train.py / torchrun.
accelerate launch --num_processes 4 --mixed_precision bf16 train.py
Override config on the command line for a one-off run.
accelerate test · accelerate estimate-memory <model>
Verify your setup works; estimate a model's memory footprint before launching.

4notebook_launcherJupyter / Colab

from accelerate import notebook_launcher notebook_launcher(training_fn, args=(...), num_processes=2)★
Run multi-GPU from inside a notebook (Colab TPUs/Kaggle) where you can't call accelerate launch.
no CUDA init before the launchergotcha
Don't touch CUDA (create tensors/models on GPU) in the notebook before notebook_launcher — it must fork clean processes. Put all model/loop code inside training_fn.

7Process control & loggingonce vs everywhere

accelerator.print("epoch done")★
Print from the main process only — avoids N duplicate lines across GPUs.
if accelerator.is_main_process: save_something()★
Guard one-time work (checkpoints, logging to W&B). is_local_main_process for per-node.
accelerator.wait_for_everyone()★
Barrier — sync all processes before/after a critical section (e.g. before loading a just-saved checkpoint).
with accelerator.main_process_first(): dataset = load_and_cache()
Let rank 0 do the download/cache first, then the others reuse it.

8Save / load / gathercheckpoints & metrics

all_preds = accelerator.gather_for_metrics(preds)★
Collect predictions from every process for evaluation — and it drops the duplicated tail samples the distributed sampler adds.
model = accelerator.unwrap_model(model) accelerator.save(model.state_dict(), "model.pt")★
Unwrap the DDP/FSDP wrapper before saving weights, and use accelerator.save (main-process-safe).
accelerator.save_state("ckpt/") · accelerator.load_state("ckpt/")★
Full training state (model, optimizer, scheduler, RNG, step) for exact resume. save_model for just the model.
accelerator.gather(t) · accelerator.reduce(t, "mean")
Low-level collectives when you need them.

9FSDP / DeepSpeedconfig, not code

accelerate config # choose FSDP or DeepSpeed here★
Enable sharding via config — your training script usually doesn't change. Shards params/grads/optimizer state across GPUs.
Accelerator(fsdp_plugin=FullyShardedDataParallelPlugin(...))
Or configure in code with the plugin (sharding strategy, wrap policy, offload).
DeepSpeedPlugin(zero_stage=3, offload_optimizer_device="cpu")
ZeRO-3 + CPU offload to fit models that don't fit in GPU memory.

10Big-model inferenceload huge weights

from accelerate import init_empty_weights with init_empty_weights(): model = MyModel(config)★
Build the model on the "meta" device with no memory for weights — then load onto real devices.
from accelerate import load_checkpoint_and_dispatch model = load_checkpoint_and_dispatch(model, ckpt, device_map="auto")★
Stream weights and shard across GPU/CPU/disk — this is what powers Transformers' device_map="auto".
infer_auto_device_map(model, max_memory={0:"20GiB", "cpu":"64GiB"})
Compute a custom placement map when the automatic one isn't what you want.

!Common gotchasread before shipping

prepare the dataloader too
If you don't pass the dataloader through prepare(), batches aren't distributed — every GPU sees the same data.
unwrap before saving
Saving a prepared (DDP/FSDP-wrapped) model bakes in the wrapper. accelerator.unwrap_model() first.
launch, don't python
Running python train.py gives one process. Use accelerate launch (or notebook_launcher) to actually go distributed.
gather_for_metrics for eval, not gather
Plain gather keeps the sampler's duplicate padding rows — use gather_for_metrics so your metrics aren't skewed.

Worth memorizing

4 editsAccelerator(), prepare(), accelerator.backward(), drop .to(device)

prepare the dataloaderor every GPU trains on the same batches

accelerate launch, not pythonthe launcher spawns the processes; notebook_launcher in Jupyter

mixed_precision handles the scalerset "bf16"/"fp16"/"fp8"; no manual GradScaler

with accelerator.accumulate(model)clean gradient accumulation, skips sync between boundaries

is_main_process / accelerator.printdo one-time work once; avoid N duplicate logs

unwrap_model before savestrip the DDP/FSDP wrapper; save_state for full resume

gather_for_metrics for evaldrops the distributed sampler's padding

FSDP/DeepSpeed = configenable sharding via accelerate config, not code changes

accelerate cheat sheet

Outline

Code

Run

Features

Multi-process

Scale

The Code Changes

Run It

Precision & Accumulation

Multi-process Control

Scale & Big Models

Worth memorizing