Quick Reference · distributed hyperparameter tuning · Ray 2.57

ray.tune cheat sheet

Define a search space, wrap your objective in a trainable, hand both to a Tuner. A search algorithm proposes configs, a scheduler kills weak trials early, and tuner.fit() runs many trials in parallel as Ray actors — then hands back a ResultGrid. Two config objects do the steering: TuneConfig (how to search) and RunConfig (where results go).

setup & Ray search space trainable & results Tuner & config search algs & schedulers resources · scale · analyze gotcha ★ most common

Distilled & verified against docs.ray.io/en/latest/tune (Key Concepts · Lifecycle · API, v2.57) · examples live-introspected from Ray 2.56.1 (Tuner API unchanged through 2.57). Verified 2026-08-27 against Ray 2.57.

Search space + trainable → Tuner → parallel trials → ResultGrid
THE PIPELINE search space {"lr": tune.loguniform(…)} trainable def objective(config): … Tuner TuneConfig — how to search RunConfig — where results go search_alg · scheduler .fit() trials run in parallel as Ray actors ×num_samples Trial 0 · config → report() Trial 1 · config → report() Trial 2 · ✂ stopped early Trial 3… RUNNING search_alg picks WHICH configs TPE / Bayes / random scheduler decides HOW LONG ASHA early-stops losers ResultGrid .get_best_result() .config · .metrics .get_dataframe() WHAT RUNS WHERE Driver process your script — calls tuner.fit() → ray.init() under the hood hosts search_alg + scheduler schedules Trainable actor 1 CPU · 0 GPU Trainable actor 1 CPU · 0.5 GPU Trainable actor tune.with_resources … on any node worker actors — one per trial, placed anywhere in the cluster storage_path checkpoints + results local · NFS · S3 shared for multi-node Tuner.restore(path) resume a crashed run
the whole thing, function APIuse tune.report inside the trainable
from ray import tune
from ray.tune.schedulers import ASHAScheduler

def objective(config):                          # the trainable (function API)
    for step in range(100):
        score = train_one_epoch(config["lr"])   # your code
        tune.report({"score": score})            # report each iteration

space = {"lr": tune.loguniform(1e-4, 1e-1), "bs": tune.choice([16, 32, 64])}

tuner = tune.Tuner(
    tune.with_resources(objective, {"cpu": 1, "gpu": 0}),
    param_space=space,
    tune_config=tune.TuneConfig(metric="score", mode="max", num_samples=20,
                              scheduler=ASHAScheduler()),
    run_config=tune.RunConfig(name="exp", storage_path="~/ray_results"),
)
results = tuner.fit()
print(results.get_best_result().config)
01Setup & Rayinstall & init
02The Workflowspace → Tuner → fit
03Search Space · Samplingtune.* domains
04Search Space · Advancedconditional & custom
05Trainable · Function APIrecommended
06Reporting Metricstune.report
07Trainable · Class APIstateful control
08Checkpointingsave · resume
09The Tunerthe entry point
10TuneConfighow to search
11RunConfigwhere results go
12Search Algorithmswhich configs to try
13Limiting the Searcherconcurrency & repeats
14Schedulershow long trials run
15Tuning ASHAthe halving knobs
16Resources per TrialCPUs · GPUs
17Passing Big Objectswith_parameters
18Results & Analysisthe ResultGrid
19Stopping & Resumingend early · restart
20Scale & Integrationscluster · loggers

Four ideas worth a picture

The two ways to write a trainable, how ASHA saves compute, why search algorithm and scheduler are different knobs, and the trial-count arithmetic that surprises everyone.

Function API vs Class API

Two ways to define the trainable. The function API is recommended — a plain def with a report loop. The class API gives explicit save/restore hooks for PBT.

FUNCTION API ★ recommended def objective(config): for step in range(N): s = train(config) tune.report({"s": s}) report drives the scheduler less code · most use-cases CLASS API class T(tune.Trainable): def setup(self, cfg) def step(self) def save_checkpoint() def load_checkpoint() explicit state control needed for PBT / pause-resume

How ASHA saves compute

Successive halving: start many trials cheap; at each rung keep only the top fraction and give survivors more budget. Weak configs die young.

trials budget → rung 1rung 2rung 3rung 4 ★ best promoted stopped (grace_period first)

search_alg vs scheduler

Two orthogonal knobs. The search algorithm picks which config each trial gets; the scheduler decides how long each trial is allowed to run.

search_alg → WHICH config (position in space) scheduler → HOW LONG tall = ran to full budget short = scheduler cut it

num_samples × grid = trial count

The count that surprises people: num_samples repeats the whole space, and every grid_search axis multiplies on top. Random domains are re-sampled each time.

num_samples=4 × grid_search([16,32,64]) = 12 trials bs=16 bs=32 bs=64 sample 1sample 2sample 3sample 4 every other tune.* domain is re-drawn for each of the 12

Worth memorizing

tune.report inside trainablesray.train.report/session.report are deprecated there
Tuner is the APIthe new entry point — replaces the old tune.run()
set metric + modein TuneConfig (or on the searcher/scheduler) — Tune must know
num_samples × gridgrid multiplies: N samples × |grid| = N·|grid| trials
default = randomBasicVariantGenerator; add a searcher for smart search
schedulers need reportsreport an intermediate metric every iteration to prune
with_resourcessets per-trial CPUs/GPUs — fractional GPUs OK ("gpu":0.5)
with_parameterspass big data via the object store, not a closure
storage_pathmust be shared (S3/NFS) for multi-node runs
checkpoint = resumereport(m, checkpoint=…) then tune.get_checkpoint()
fit() calls ray.init()driver process + one worker actor per trial
grace_periodstops ASHA from killing trials before they warm up