Quick Reference · gradient-boosted decision trees (Python)

lightgbm cheat sheet

LightGBM grows trees leaf-wise (best-first) instead of level-wise, and buckets continuous features into histogram bins for speed. Almost every tuning knob works on one of three axes: how many leaves a tree gets, how much data each leaf needs, and how much of the data/features each tree sees.

data & I/O core training regularize / sample sklearn API & eval pitfall ★ most common

Distilled & cross-checked against: lightgbm.readthedocs.io — Python API · Parameters · Parameters Tuning · Features  ·  github.com/microsoft/LightGBM

Two ways in, one booster out
Raw Data numpy · pandas · csv · sparse X, y lgb.Dataset values → histogram bins categorical_feature=[…] lgb.train() leaf-wise boosting loop valid_sets, callbacks=[…] Booster ensemble of trees bst.predict(X) LGBMClassifier / LGBMRegressor / LGBMRanker . fit(X, y) wraps Dataset + train() into one scikit-learn-style call YOU BRING YOU GET
01Install & Importsetup
02Build a Datasetcore API · binning
03Train — Booster APIfunctional / low-level
04Train — scikit-learn APIhigh-level
05Core Boosting Paramsshape of every tree
06Regularize a Leaf-wise Treefight over-fitting
07Sub-sample Rows & Columnsspeed + variance control
08Categorical Featuresnative, no one-hot
09Objectiveswhat the tree optimizes
10Evaluation Metricswhat gets logged
11Callbacks & Early Stoppingcontrol the loop
12Cross-Validationlgb.cv
13Importance & Plotsinspect the model
14Save & Loadpersistence
★Common Pitfallshandle with care

How LightGBM grows & samples

The signature ideas behind LightGBM's speed: best-first (leaf-wise) tree growth instead of level-wise, and gradient-based sampling instead of plain row subsampling. Based on the official Features reference.

Leaf-wise growth (LightGBM default)

Always splits the leaf with the largest loss reduction, regardless of depth. Fewer leaves reach the same accuracy — but depth can run away, so pair num_leaves with min_data_in_leaf.

L1 L2 L3 L4 4 leaves · uneven depth (1–3)

Level-wise growth (contrast)

Splits every node at the current depth before going deeper — the strategy most other GBM libraries default to. Same leaf count, more balanced, but reaches it in more total splits.

L1 L2 L3 L4 4 leaves · uniform depth (2)

GOSS sampling

Gradient-based One-Side Sampling: always keeps the highest-gradient (under-trained) rows, and randomly samples a small slice of the rest instead of scanning every row every round.

top_rate 0.2 · kept other_rate 0.1 · sampled discarded this round rows sorted by |gradient|, descending → data_sample_strategy='goss'

Worth memorizing

num_leaves < 2^max_depthleaf-wise trees overfit fast if leaves aren't capped below full depth
leaf-wise ≠ level-wiseLightGBM's default growth strategy; converges faster, needs tighter regularization
min_data_in_leafthe single most effective anti-overfit knob for leaf-wise trees
categorical_featurepass native ints/category dtype — no one-hot, ~8× faster
reg_alpha / reg_lambdasklearn aliases for native lambda_l1 / lambda_l2
early stoppingrequires valid_sets + a metric; disabled under boosting_type='dart'
feature_fraction ≈ colsample_bytreebagging_fraction ≈ subsample — same knob, different library's naming
GOSS vs baggingGOSS keeps high-gradient rows deterministically; bagging samples uniformly at random