Hyperopt Cheat Sheet

01Setup & Importinstall

pip install hyperopt★
Core library (pulls in numpy, scipy, networkx, tqdm).
pip install "hyperopt[mongotrials]"
Extra for the Mongo-backed distributed backend.
from hyperopt import fmin, tpe, hp, Trials★
The four names you reach for every time.
from hyperopt import STATUS_OK, space_eval★
Return-status flag + the index decoder.
import numpy as np
Needed for log() bounds & rstate.

02The Workflowspace → fmin → decode

space = {"x": hp.uniform("x", -5, 5)}★
1 · describe the space with hp.*.
def objective(p): return (p["x"]-2)**2★
2 · map a config to a loss.
best = fmin(objective, space, algo=tpe.suggest, max_evals=100)★
3 · search — returns the argmin.
space_eval(space, best)★
4 · decode indices → real values (always do this).

03Search Space · Corehp.* domains

hp.uniform("x", low, high)
Continuous float, flat over the range.
hp.loguniform("lr", np.log(1e-4), np.log(1e-1))★
Log scale — bounds are in log space.
hp.quniform("n", 1, 100, 1)
Quantized float (returns a float — cast to int).
hp.choice("opt", ["adam", "sgd"])★
Categorical — best gives its index.
hp.randint("k", 10)
Integer in [0, 10).

04Search Space · Morethe rest of hp.*

hp.normal("m", mu, sigma)
Gaussian — unbounded, centered on mu.
hp.lognormal("s", mu, sigma)
Positive, heavy-tailed.
hp.uniformint("u", low, high)
Integer, uniform & inclusive.
hp.qloguniform · hp.qnormal · hp.qlognormal
Quantized log / normal variants.
hp.pchoice("c", [(0.3, "a"), (0.7, "b")])
Weighted categorical (custom prior probabilities).

05Conditional / Nested Spaceshyperopt's superpower

hp.choice("model", [svm_branch, rf_branch])★
Each branch is a dict with its own hp.* params.
{"type": "svm", "C": hp.loguniform("C", …)}
Sub-params only sampled when the branch is chosen.
hp.choice("a", [("c1", hp.uniform(…)), ("c2", …)])
Tuple branches work too — unpack in the objective.
# every hp label must be globally UNIQUErule
Reusing a label across branches silently breaks the space.

06The Objectiveconfig → loss

def objective(params): …★
One arg — the sampled point (dict / list / value).
return loss★
Simplest protocol: a single float to minimize.
return -accuracy
fmin only minimizes — negate to maximize.
params["lr"], params["model"]["C"]
Read hyperparameters (nested for conditional spaces).

07Rich Return Dictstore more per trial

return {"loss": L, "status": STATUS_OK}★
The two required keys of the rich protocol.
"status": STATUS_FAIL
Flag a bad config so it's ignored, not counted as best.
"loss_variance": v · "true_loss": t
Optional stats for noisy / held-out losses.
"attachments": {"model": pickled}
Stash big artefacts on the trial (retrieve later).

08fmin · The Driverruns the search

fmin(fn, space, algo=tpe.suggest, max_evals=100)★
The four essentials — returns best (the argmin).
trials=Trials()★
Pass one to keep the full evaluation history.
rstate=np.random.default_rng(42)
Seed for reproducible runs.
return_argmin=False
Return the best trial instead of the argmin dict.

09fmin · More Optionsbudgets & warm starts

10Algorithmsthe algo= arg

11Tuning TPEpartial(...)

from hyperopt import partial
Wrap the suggester to set its params.
algo=partial(tpe.suggest, n_startup_jobs=20)★
Random warm-up trials before TPE kicks in.
partial(tpe.suggest, gamma=0.25)
The good/bad split quantile.
partial(tpe.suggest, n_EI_candidates=24)
Candidates scored by expected improvement each step.

12Recovering the Bestthe #1 gotcha

space_eval(space, best)★
hp.choice puts an index in best — this decodes it.
trials.argmin
Same argmin dict fmin returns (still indices).
trials.best_trial["result"]["loss"]
The winning loss value.
space_eval(space, trials.argmin)
Full real-valued winning config.

13The Trials Objectthe history db

14Inspecting Trialsdig into results

for t in trials.trials: t["result"], t["misc"]["vals"]
Per-trial result + the values that produced it.
trials.statuses()
OK / FAIL per trial.
pd.DataFrame(trials.vals)
Turn the search history into a DataFrame.
trials.trial_attachments(trial)
Retrieve stashed artefacts (models, arrays).

15Early Stoppingstop when stuck

from hyperopt.early_stop import no_progress_loss★
The built-in patience stopper.
early_stop_fn=no_progress_loss(30)
Halt after 30 evals without improvement.
def stop(trials, *a): return cond, {}
Custom stopper — returns (stop?, state).
timeout=600 · loss_threshold=0.0
Or bound by wall-clock / target loss.

16Persistence & Resumesave the history

pickle.dump(trials, f)
A Trials object pickles cleanly.
trials_save_file="t.pkl"
Let fmin checkpoint the trials itself.
fmin(…, trials=old, max_evals=len(old)+50)★
Resume: reuse trials & raise the budget.
points_to_evaluate=prev_best
Carry known-good configs into a new run.

17Parallel · SparkTrialsscale on Spark

from hyperopt import SparkTrials★
Run trials across a Spark cluster.
spark_trials = SparkTrials(parallelism=8)
Up to 8 concurrent trials.
fmin(…, trials=spark_trials)★
Same fmin — just swap the trials backend.
# TPE is sequential; parallelism trades off info
Higher parallelism ≈ more random-like search.

18Parallel · MongoTrialsasync cluster

from hyperopt.mongoexp import MongoTrials
Async distributed search via a MongoDB queue.
MongoTrials("mongo://host:27017/db/jobs", exp_key="e1")
Point trials at a Mongo collection.
$ hyperopt-mongo-worker --mongo=host:27017/db
Launch workers that pull & evaluate jobs.
# objective must be importable, not a closurenote
Workers re-import your function by reference.

19ML Integrationthe common pattern

return -cross_val_score(clf, X, y).mean()★
Minimize negative CV score = maximize accuracy.
clf = SVC(C=p["C"], gamma=p["gamma"])
Build the estimator from the sampled config.
from hpsklearn import HyperoptEstimator
Auto-sklearn-style wrapper over Hyperopt.
# xgboost: cast quniform depth to int
int(p["max_depth"]) — quniform returns a float.

20Gotchas & Tipswhat bites people

hp.choice → index in best★
Always space_eval; never index the list yourself.
fmin minimizes only★
Return -metric to maximize.
loguniform bounds are log-space★
Pass np.log(lo), np.log(hi), not raw values.
labels must be globally unique
Duplicate hp.* labels corrupt the space.
quniform returns a float
Cast to int() for counts / depths.

hyperopt cheat sheet