Quick Reference · distributed hyperparameter optimization · v0.3

hyperopt cheat sheet

Hand fmin four things: an objective (config → loss), a search space written as probability distributions (hp.*), an algorithm (TPE by default choice), and a Trials database. It runs SMBO — TPE learns which regions look promising and samples there. One quirk to burn in: hp.choice returns an index, so call space_eval(space, best) to get real values back.

setup search space (hp.*) objective fmin driver algorithms & trials scale · persist · integrate gotcha ★ most common

Distilled & verified against hyperopt.github.io (docs · FMin wiki) · the SciPy 2013 paper (Bergstra et al.) · live-introspected from Hyperopt 0.3.0

objective + space + algo + trials → fmin → best → space_eval
THE fmin LOOP space (hp.*) distributions, not bounds objective(params) returns a loss to minimize fmin algo=tpe.suggest max_evals=N each of max_evals iterations TPE proposes a point from the space objective → loss {'loss', 'status'} Trials records it TPE learns splits past trials into good l(x) vs bad g(x), samples where l(x)/g(x) is highest warm-up: random first next trial ×N best argmin (indices!) {'opt': 1, 'x': 2.0} space_eval (space, best) → real values TWO WAYS TO WRITE THE OBJECTIVE SIMPLE — just a float def objective(p): return (p['x']-2)**2 quick · can't store extras RICH — a dict return {'loss': L, 'status': STATUS_OK, 'model': clf} # extras stores anything per trial Trials() — the history tid 0 · loss 12.4 · STATUS_OK · vals {x:5.1} tid 1 · loss 3.1 · STATUS_OK · vals {x:0.7} tid 2 · loss 0.0 · ★ best_trial · argmin trials.losses() · trials.best_trial · trials.results
the whole thingfmin minimizes — negate to maximize
import numpy as np
from hyperopt import fmin, tpe, hp, Trials, STATUS_OK, space_eval

space = {                                            # 1 · space as distributions
    "lr":  hp.loguniform("lr", np.log(1e-5), np.log(1e-1)),  # log-space bounds!
    "opt": hp.choice("opt", ["adam", "sgd"]),
}

def objective(p):                                    # 2 · config -> loss
    loss = train_and_eval(p["lr"], p["opt"])
    return {"loss": loss, "status": STATUS_OK}

trials = Trials()                                    # 3 · the database
best = fmin(objective, space, algo=tpe.suggest,      # 4 · search
            max_evals=100, trials=trials, rstate=np.random.default_rng(0))

print(space_eval(space, best))                     # decode choice indices -> values
01Setup & Importinstall
02The Workflowspace → fmin → decode
03Search Space · Corehp.* domains
04Search Space · Morethe rest of hp.*
05Conditional / Nested Spaceshyperopt's superpower
06The Objectiveconfig → loss
07Rich Return Dictstore more per trial
08fmin · The Driverruns the search
09fmin · More Optionsbudgets & warm starts
10Algorithmsthe algo= arg
11Tuning TPEpartial(...)
12Recovering the Bestthe #1 gotcha
13The Trials Objectthe history db
14Inspecting Trialsdig into results
15Early Stoppingstop when stuck
16Persistence & Resumesave the history
17Parallel · SparkTrialsscale on Spark
18Parallel · MongoTrialsasync cluster
19ML Integrationthe common pattern
20Gotchas & Tipswhat bites people

Four ideas worth a picture

Describing a space as distributions, the conditional-space tree that sets Hyperopt apart, how TPE actually chooses, and the choice-index trap that catches everyone.

A space is distributions, not bounds

Each hp.* encodes not just a range but where to look. Giving the search a shape is what lets TPE beat a plain grid.

uniform loguniform small values favored normal quniform choice — discrete, equal weight adam sgd rms pchoice adds custom weights lognormal for positive tails

Conditional search spaces

An hp.choice can branch into whole sub-spaces. Each branch's params exist only when that branch is picked — impossible to express as a flat grid.

hp.choice("model") "type": "svm" "type": "rf" C = loguniform depth = quniform only the chosen branch's params are sampled

How TPE chooses (born here)

TPE — first shipped in Hyperopt — splits finished trials into good and bad by loss, models each as a density, and proposes where l(x)/g(x) is largest.

param x → g(x) — bad l(x) — good next: max l(x)/g(x) gamma sets the good/bad cutoff · n_startup_jobs random first

The choice-index trap

The most common Hyperopt bug: best stores the position a category had in the list, not the category. space_eval is the only correct decoder.

hp.choice("opt", ["adam", "sgd", "rmsprop"]) idx 0idx 1idx 2 best = {"opt": 1} ✗ best["opt"] → 1 an int, not "sgd" — silently wrong downstream ✓ space_eval(space, best) → {"opt": "sgd"} the actual category

Worth memorizing

hp.choice → indexalways space_eval(space, best) to get real values
fmin minimizesreturn -metric to maximize
loguniform boundsare in LOG space — pass np.log(lo), np.log(hi)
unique labelsevery hp.* label must be unique across the whole space
pass a Trials()to keep the full history — best alone is just the argmin
tpe.suggestis the smart default; rand.suggest for a baseline
rich return{'loss':…, 'status': STATUS_OK} stores extras per trial
rstatenp.random.default_rng(seed) for reproducibility
early_stop_fnno_progress_loss(N) stops when improvement stalls
SparkTrials / MongoTrialsswap the backend to parallelize — fmin is unchanged
quniform is a floatcast int() for counts, depths, estimators
points_to_evaluatewarm-starts fmin with known-good configs