Quick Reference · hyperparameter optimization · v4.9

optuna cheat sheet

Write one objective(trial) function. Every suggest call both defines a hyperparameter and returns a value to try — that's define-by-run. A sampler proposes smart values from past trials, a pruner kills losers early, and the study remembers everything. study.optimize(objective, n_trials=N) runs the loop.

setup & study search space (suggest) optimize & results samplers pruners & advanced persist · viz · scale gotcha ★ most common

Distilled & verified against optuna.readthedocs.io (v4.9) · optuna.org · hub.optuna.org · the KDD 2019 paper (arXiv 1907.10902) · live-introspected from Optuna 4.9.0

One study = many trials · the optimize loop
THE LOOP create_study direction=… sampler · pruner study.optimize(objective, n_trials=N) — repeat per trial Sampler proposes values from past trials TPE (default) objective(trial) x = trial.suggest_*() build model · train define-by-run report(v, step) should_prune()? per epoch early-stop losers return value study records the trial ✂ raise TrialPruned() next trial — sampler learns from the history so far ×N best_params best_value best_trial trials_dataframe() WHAT A STUDY HOLDS Study an optimization run; a history of trials Trial 0COMPLETE lr=3e-4 · n=54 opt="adam" value = 0.42 Trial 1PRUNED lr=9e-2 · n=3 stopped @ step 2 (no final value) Trial 2COMPLETE lr=1e-3 · n=80 opt="sgd" value = 0.55 Trial 3 ★ best lr=8e-4 · n=64 opt="adam" value = 0.13 ◀ min Trial 4… RUNNING
the whole thing in 12 linesminimize by default
import optuna

def objective(trial):
    # each suggest_* both defines a param AND returns a value
    lr    = trial.suggest_float("lr", 1e-5, 1e-1, log=True)
    units = trial.suggest_int("units", 16, 256, log=True)
    opt   = trial.suggest_categorical("opt", ["adam", "sgd"])
    model = build_and_train(lr, units, opt)          # your code
    return validation_loss(model)                    # the number to minimize

study = optuna.create_study(direction="minimize")
study.optimize(objective, n_trials=100)
print(study.best_params, study.best_value)
01Setup & Importinstall
02The Core Loopstudy + objective
03Create & Load Studiescreate_study(…)
04Search Space · Floatsuggest_float
05Search Space · Int & Catthe other two
06Define-by-run Patternsdynamic spaces
07Running the Studystudy.optimize
08Results & Inspectionread the history
09Samplershow values are chosen
10Choosing & Tuning Samplersreproduce & auto
11Prunerskill losers early
12Pruning in the Objectivereport → should_prune
13Multi-objectivePareto fronts
14Ask-and-Tellyour own loop
15Warm-starts & Fixed Paramsseed known-goods
16Persistence & Storagesave · resume · share
17Parallel & Distributedscale out
18Visualizationoptuna.visualization
19Importance & Integrationsplug into your stack
20Dashboard & CLIno-code inspection

Four ideas worth a picture

The mental models behind the API — define-by-run search spaces, how pruning saves compute, what TPE actually does, and reading an importance plot.

define-by-run vs define-and-run

Other tools declare a fixed grid up front. Optuna builds the space live — a conditional means a param only exists on the branch that needs it.

DEFINE-AND-RUN (static grid) lr ∈ [1e-5 … 1e-1] units ∈ {16…256} svr_c ∈ […] max_depth ∈ […] all always present DEFINE-BY-RUN (live tree) suggest "clf" "SVR" "RandomForest" svr_c only max_depth only only the relevant branch is sampled

Pruning saves the compute budget

Report a metric each step; a trial trailing the median gets stopped before it wastes a full run. report() feeds the pruner, should_prune() asks it.

loss step → running median ✓ ✓ ✂ pruned @ step 3 raise TrialPruned() 12345

What TPE actually does

The default sampler splits finished trials into good and bad by their value, fits a density to each, and proposes points where good is likely and bad is not — l(x) / g(x).

param x → g(x) — bad trials l(x) — good trials next sample: max l(x)/g(x)

Reading param importances

After a study, ask which knobs moved the objective. Spend your next budget on the tall bars; freeze or narrow the short ones.

units 0.61 x 0.35 lr 0.03 opt 0.02 get_param_importances(study) · normalized to sum 1

Worth memorizing

minimize is defaultset direction="maximize" for accuracy/reward
log=Truefor learning rates & anything spanning magnitudes
report + should_pruneboth needed — and you raise TrialPruned() yourself
TPE is the defaultsampler; RandomSampler for an honest baseline
best_params vs best_trialssingle-objective vs the Pareto set (multi)
sqlite + load_if_exists= resumable, and lets many workers cooperate
suggest_uniformdeprecated → suggest_float(log=…, step=…)
n_jobs > 1 = threadsreal parallelism = many processes, one shared storage
seed the samplerTPESampler(seed=…), not just numpy, to reproduce
keep a param's range fixeddon't change low/high mid-study for the same name
catch=(...)optimize catches nothing by default — flaky trials crash
unsure which sampler?AutoSampler from OptunaHub picks one for you