Quick Reference · gradient-boosted decision trees (Python) · v2

lightgbm cheat sheet v2

LightGBM grows trees leaf-wise (best-first) and buckets continuous features into histogram bins. Three innovations power its speed: histogram-based splits, GOSS (gradient-based row sampling), and EFB (bundling sparse mutually-exclusive features). Every tuning knob works on one of three axes: tree complexity (leaves/depth), data per leaf, and randomness (row/column sampling).

data & I/O core training regularize / sample sklearn API · eval · ecosystem pitfall ★ most common

Distilled & cross-checked against: lightgbm.readthedocs.io — Python API · Parameters · Parameters Tuning · Features · Distributed Learning Guide  ·  github.com/microsoft/LightGBM  ·  Ke et al. 2017 (NeurIPS paper)

Two ways in, one booster out
Raw Data numpy · pandas · arrow · sparse X, y · NaN = missing lgb.Dataset values → ≤ max_bin bins · EFB categorical_feature=[…] lgb.train() leaf-wise boosting · GOSS valid_sets, callbacks=[…] Booster ensemble of trees predict · refit · save_model LGBMClassifier / LGBMRegressor / LGBMRanker . fit(X, y) · DaskLGBM* for clusters wraps Dataset + train() into one scikit-learn-style call YOU BRING YOU GET
The four equations behind every parameter

1 · Boosting update

Each round adds one tree, shrunk by the learning rate. Small learning_rate + more num_iterations ⇒ better generalization, slower training.

Fm(x) = Fm−1(x) + η · fm(x) η = learning_rate · fm = tree fit to gradients · m = 1…num_iterations

2 · Optimal leaf value (Newton step)

Each leaf outputs the ratio of summed gradients to summed Hessians. lambda_l2 shrinks it; min_sum_hessian_in_leaf keeps the denominator healthy.

w* = − Σi∈leaf gi Σi∈leaf hi + λ λ = lambda_l2

3 · Split gain (what "best leaf" means)

The leaf with the largest gain is split next — that's leaf-wise growth. A split is kept only if gain ≥ min_gain_to_split.

Gain = ½ [ GL² HL+λ + GR² HR+λ − (GL+GR)² HL+HR+λ ] G = Σg, H = Σh over left / right candidate children · evaluated per histogram bin edge

4 · GOSS re-weighting

Keep the top a·N rows by |gradient|, sample b·N of the rest, then amplify the sampled rows so gain estimates stay unbiased.

multiplier = 1 − a b a = top_rate (0.2) · kept b = other_rate (0.1) · sampled
01Install & Importsetup
02Build a Datasetcore API · binning
03Train — Booster APIfunctional / low-level
04Train — scikit-learn APIhigh-level · pipelines
05Predict — All ModesBooster.predict flags
06Core Boosting Paramsshape of every tree
07Regularize a Leaf-wise Treefight over-fitting
08Sub-sample Rows & Columnsvariance + speed
09DART Modeboosting_type='dart'
10Categorical Featuresnative, no one-hot
11Missing Values & SparsityNaN · EFB
12Objectives — Regressionloss functions
13Objectives — Class & Rank+ imbalance
14Evaluation Metricswhat gets logged
15Callbacks & Early Stoppingcontrol the loop
16Cross-Validationlgb.cv
17Custom Objective & Metricbring your own loss
18Constraintsdomain knowledge in
19Linear Treespiece-wise linear leaves
20Speed & Hardwarecpu · gpu · cuda
21Distributed — Dasklightgbm.dask
22Importance & Interpretationinspect the model
23Persist & Productionsave · load · refit
24Reproducibility & Loggingdeterminism
25Official Tuning PlaybookParameters-Tuning docs
★Common Pitfallshandle with care

How LightGBM grows, bins & samples

The signature mechanics: best-first growth, histogram binning with the subtraction trick, gradient-based sampling, and exclusive feature bundling. Based on the official Features reference and the NeurIPS 2017 paper.

Leaf-wise growth (LightGBM default)

Always splits the leaf with the largest gain (eq. 3), regardless of depth. Fewer leaves reach the same loss — but depth can run away, so pair num_leaves with min_data_in_leaf.

L1 L2 L3 L4 4 leaves · uneven depth (1–3)

Level-wise growth (contrast)

Splits every node at the current depth before going deeper — most other GBM libraries' default. Same leaf count, more balanced, but spends splits on low-gain leaves.

L1 L2 L3 L4 4 leaves · uniform depth (2)

Histogram binning + subtraction trick

Feature values collapse into ≤ max_bin bins holding (Σg, Σh). A child's histogram = parent − sibling, so only the smaller child is ever built from data.

bins (Σg, Σh) · ≤ max_bin continuous values → uint8 bin ids parent − sibling = child free — no data scan

GOSS sampling

Keeps all high-|gradient| (under-trained) rows, samples a slice of the rest, and re-weights the sample by (1−a)/b (eq. 4) so estimated gains stay unbiased.

top_rate a=0.2 always kept b=0.1 sampled ×(1−a)/b discarded this round rows sorted by |gradient|, descending → data_sample_strategy='goss'

EFB — exclusive feature bundling

Sparse features that are (almost) never non-zero together get merged into one bundled feature with offset value ranges. Histogram cost drops from O(data×features) to O(data×bundles).

f1f2f3 bundle range 1 range 2 range 3 non-zeros never collide → merge with offsets O(data × features) → O(data × bundles) enable_bundle / is_enable_sparse (default on)

The tuning compass

Every important parameter pushes along one of three axes. Diagnose which axis is wrong (under-fit vs over-fit vs too slow) before touching anything.

fit complexity ↑ num_leaves · max_depth · max_bin data per leaf ↑ min_data_in_leaf · λ₁ λ₂ randomness ↑ bagging · feature_fraction

Worth memorizing

num_leaves < 2^max_depthleaf-wise trees overfit fast if leaves aren't capped below full depth
min_data_in_leaffirst knob to raise when valid loss diverges from train loss
bagging needs freqbagging_fraction does nothing until bagging_freq > 0
Dataset-time vs train-timemax_bin & binning freeze at Dataset build; boosting params don't
params dict winsin lgb.train(), params entries override same-named keyword arguments
refit ≠ init_modelrefit updates leaf values only; init_model adds brand-new trees
gain > splitprefer importance_type='gain' — split counts favor high-cardinality features
custom objective → raw scoresapply sigmoid/softmax yourself when predicting
early stopping needs≥1 valid set + ≥1 metric · saves best_iteration · dead under dart
reg_alpha / reg_lambdasklearn aliases of native lambda_l1 / lambda_l2 — same knobs
NaN = missingnever encode missing as 0 unless zero_as_missing=true
GOSS multipliersampled small-gradient rows scaled by (1−a)/b to stay unbiased