ML Handbook

Theme

Tree Models and Ensemble Learning

Quick overview How do decision trees split? Why do random forests reduce variance? Why does GBDT reduce bias? This article systematically covers the complete progression from a single decision tree to XGBoost and LightGBM, and provides practical code with feature importance and SHAP interpretation.

Tree Models and Ensemble Learning ​

In an era where deep learning dominates images, speech, and text, there is a counterintuitive fact: in Kaggle-style competitions focused on tabular data, the winning solutions are overwhelmingly powered by tree-based models — XGBoost, LightGBM, CatBoost. They don't look "intelligent" or "cutting-edge," yet they have become the de facto standard for tabular data in industry, thanks to five key advantages: fast training, strong stability, almost no need for feature scaling, high tolerance for hyperparameter tuning mistakes, and excellent performance. This judgment was foreshadowed in What is Machine Learning; this article expands on it in detail.

The narrative of this article follows a "three-refresher" arc (three lessons total): single decision trees (principles and weaknesses) → ensemble learning (Bagging for variance reduction, Boosting for bias reduction) → engineering (XGBoost/LightGBM scaling the algorithm to industrial scale). Finally, it provides runnable code and interpretability tools that go from "the prediction is right" to "explaining why."

1. Decision Trees: Turning "if-then" into a Model ​

1. A tree is a set of nested if-else statements ​

Decision Trees are among the simplest and most ancient machine learning models in terms of intuition: they recursively partition the feature space, and each leaf node outputs a prediction. Suppose we want to predict "whether it will rain," with features like "humidity" and "are there clouds":

                  Humidity < 70% ?
                  /           \
             Yes (go left)   No (go right)
                │                │
             "No rain"      Are there clouds?
                            /       \
                         "Rain"    "No rain"

Building a tree is an ongoing process of answering one question: which feature, and at what threshold, should we split on, so that the resulting partitions are "purer"? This "split" operation is called splitting, and the tree grows recursively. CART (Classification And Regression Tree) is the foundation of all modern tree models. It has two constraints: binary trees (one split at a time) and support for both classification and regression tasks.

2. Splitting criteria: Information gain and Gini coefficient ​

"Purer" needs a quantitative measure. Suppose node D has K classes of samples, with proportions p₁...p_K. Two of the most classic purity measures:

MeasureFormulaIntuitionUsed By
Information EntropyH(D) = −Σₖ pₖ·log₂ pₖUncertainty: pure nodes have entropy 0, uniform nodes have maximum entropyID3 / C4.5
Gini coefficientGini(D) = 1 − Σₖ pₖ²Probability of randomly drawing two samples with different classesCART

The benefit of splitting on feature A is called information gain (Information Gain):

IG(D, A) = H(D) − Σ_v  |D_v|/|D| · H(D_v)

where v iterates over each child node corresponding to each value of A. Information gain = "uncertainty before splitting" minus "weighted remaining uncertainty after splitting by sample count." The larger the gain, the better A can distinguish classes.

Example: A dataset has 14 samples, 9 positive and 5 negative. H(D) = −(9/14)log₂(9/14) − (5/14)log₂(5/14) ≈ 0.940. If we split on feature "wind strength": 8 samples have weak wind (6 positive, 2 negative), 6 have strong wind (3 positive, 3 negative), then the weighted remaining entropy = (8/14)·0.811 + (6/14)·1.000 ≈ 0.892, information gain ≈ 0.940 − 0.892 = 0.048. Calculate the information gain for all candidate features the same way, and split on the one with the maximum gain first.

Two details are worth noting. First, information gain favors features with many values (e.g., a feature like "ID number" where every value is unique will have an exploding gain). C4.5 therefore uses the gain ratio (information gain divided by intrinsic value) instead. CART simply uses binary splitting to mitigate this issue. Second, the Gini coefficient and entropy are highly consistent in their ranking, but Gini doesn't require computing logarithms, so it's faster — this is why modern implementations like XGBoost prefer it: a tiny difference in a single split, accumulated across millions of rows and dozens of iterations, becomes a huge speed gap.

3. Pruning: the first line of defense against overfitting ​

Overfitting in decision trees is structural: as long as you keep splitting, the tree can always achieve 100% accuracy on the training set (each leaf contains only one class). A tree that memorizes all the noise is completely isomorphic to the "model that memorizes answers" described in Overfitting and Regularization. The countermeasure is pruning, which comes in two flavors:

  • Pre-pruning (Pre-pruning): Stop early during growth. Typical tactics: limit max_depth (maximum depth), min_samples_leaf (minimum samples per leaf), min_samples_split (minimum samples required to split), or stop splitting if the gain falls below a threshold. Pre-pruning is fast, but "early stopping" may sacrifice splits that would have been beneficial later (the myopic problem).
  • Post-pruning (Post-pruning): First grow the full tree, then prune subtrees that don't benefit generalization, bottom-up. CART uses cost-complexity pruning: define the objective
R_α(T) = R(T) + α · |T_leaf|

where R(T) is the training error, |T_leaf| is the number of leaves, and α is the penalty coefficient. Increasing α trades "accuracy" for "tree size," and the optimal α is selected using a validation set. The ccp_alpha parameter in sklearn implements this mechanism.

A key intuition

A tree's depth and number of leaves are its regularization strength knobs. A depth-1 tree (a stump) is a "nearly underfit" weak model, while a depth-20 tree is a "nearly overfit" strong model. This continuous spectrum from underfitting to overfitting is the key to understanding ensemble learning later — Bagging ensembles many "overfitting-prone" trees, while Boosting approaches the "underfit" target one tree at a time.

4. Strengths and weaknesses of a single tree ​

First, the strengths — these are the seeds that later made tree models dominate tabular data:

StrengthExplanation
No feature scaling neededSplitting only depends on the relative order of feature values; monotonic transformations (log, square root) don't change the tree structure
Naturally handles nonlinearity and interactionsNo need to manually construct feature cross-terms like linear models do; trees automatically carve out high-order combinations like "age > 30 AND income < 50k"
InterpretableA path from root to leaf is a human-readable rule
Tolerant of missing values and outliersOutliers only affect the position of a single split point; missing values can fall on the majority side

But single trees have two fatal weaknesses that ultimately determined their historical fate:

  • High variance: Swap out the training set, and the tree structure may look completely different — because splitting is a "winner-takes-all" greedy choice, one decision at the root determines the direction of the entire tree. A single tree will almost certainly overfit (test error much higher than training error).
  • Expressive power limited by axis-aligned splits: Slanted decision boundaries need many small "stair steps" built from many small trees to approximate; a single tree gives a "jagged" approximation.

Before the mid-1990s, the way to deal with "high variance" was to prune trees very shallowly — at the cost of weak models. This problem was only thoroughly solved when Breiman and Freund each proposed two routes: Bagging in the next section and Boosting in the section after that.

2. Bagging and Random Forests: Turning "unstable" into "stable" ​

1. Where does variance come from? ​

A metaphor: the prediction error of a single deep tree = systematic "bias" + random "jitter." Pruning can reduce "bias," but the "jitter" (fluctuations sensitive to specific training samples) remains large. Statistically, the variance of a random variable fluctuates with samples, but the average of multiple independent random variables has significantly reduced variance — this is the entire idea behind Bagging.

2. Bagging: Bootstrap Aggregating ​

Bagging = Bootstrap Aggregating (Breiman, 1996). The process has only three steps:

Original dataset D (n samples)
   │
   ├─① Bootstrap sampling: Sample n times with replacement, producing T new datasets D₁…D_T
   │   (each D_t contains about 63.2% of the original samples on average; the rest are duplicates)
   ├─② Parallel training: Train a (deep) tree T_t independently on each D_t
   └─③ Aggregation: Voting for classification, averaging for regression

Note: in Bagging, each tree is deliberately not pruned — let them fully overfit their respective Bootstrap samples. Precisely because each tree "overfits in different ways" (different sampling), their errors cancel out when averaged, rather than amplifying.

3. Random Forests: one more layer of randomness ​

Bagging has a flaw: if one feature is particularly strong, all trees will prefer to split on the same feature, making the trees highly correlated — the premise of "averaging independent variables reduces variance" breaks. In his 2001 paper on Random Forests, Breiman added a crucial twist: at each split, only consider a randomly sampled subset of m features. For classification, m = √p by default; for regression, m = p/3.

Random Forest = Bagging (randomness in the sample dimension) + feature subsets (randomness in the feature dimension)

At split:  randomly draw m features from all p features, and find the best split only among these m

The significance of this twist is forced decorrelation: even if there is a "perfect feature," each tree can only use it in some splits; the rest must rely on other features, deliberately amplifying differences between trees. Looking at the formula from the previous section:

Variance of the average of T trees  =  ρ·σ²  +  (1−ρ)·σ²/T
                                      └──────────┬──────────┘
                           Decorrelated term (gets smaller)   Averaging term (gets smaller as T increases)

Even as T approaches infinity, the first term ρ·σ² still exists — so the ultimate error ceiling of random forests is determined by the inter-tree correlation ρ. Feature randomization and Bootstrap sampling both work to suppress ρ.

4. Out-of-bag error: free cross-validation ​

During Bootstrap sampling, each training set misses about 36.8% of the samples on average (1/e as n → ∞). Out-of-Bag (OOB) samples are "samples this tree hasn't seen." Using each tree's OOB samples to evaluate that tree, then aggregating all trees' evaluation results, gives a nearly free estimate of test error that doesn't require a separate validation set — and it aligns closely with holdout methods. Enable it in sklearn with oob_score=True. This is a hidden bonus of random forests: the validation set error is computed along with training.

5. Limits of random forests ​

Random forests suppress the "high variance of single trees," trading for robustness: almost no tuning needed, immune to noise, natively parallel (each tree is independent). But it has two bottlenecks:

  • Only reduces variance, not bias: If a single tree has high bias (e.g., using stumps), ensembling more trees won't help — "averaging a systematic bias" is still that bias.
  • Trees have an expressive ceiling: Each tree in a random forest is full-depth, but prediction accuracy is locked at a certain ceiling by the "averaging" mechanism.

To break through the ceiling, we need a different approach: not averaging, but relays. This is Boosting.

3. Boosting: Turning "weak" into many ​

Bagging is "three cobblers each doing their own thing, then voting"; Boosting is "a student repeatedly practicing wrong answers" — each round focuses on the samples the previous round got wrong, stringing weak models into a strong one.

1. AdaBoost: weighting the wrong answers ​

AdaBoost (Adaptive Boosting, Freund & Schapire, 1997) is the first widely used Boosting algorithm:

① Initialize sample weights wᵢ = 1/n
② for t = 1..T:
      a. Train a weak classifier h_t (usually a stump) on weighted samples
      b. Compute weighted error rate ε_t = Σ wᵢ·I(h_t(xᵢ)≠yᵢ)
      c. Compute voting weight for this classifier α_t = ½·ln((1−ε_t)/ε_t)
      d. Update sample weights: correct predictions × e^(−α_t), wrong × e^(α_t), then normalize
③ Output: H(x) = sign( Σ_t α_t · h_t(x) )

Two key intuitions: ① samples predicted wrong get their weights amplified each round, forcing the next tree to focus on "hard cases"; ② each weak classifier gets a voting weight α_t — more accurate trees get more say. Stumps (max_depth=1) have very weak predictive power (accuracy slightly above 50%), but AdaBoost relays through hundreds of stumps, and the error can decrease exponentially to arbitrarily small levels — a strong theoretical result that makes "weak learners can be boosted to strong learners" a foundational theorem of Boosting.

2. A statistical perspective: additive models and exponential loss ​

In 1999–2000, Friedman, Hastie, and Tibshirani proved in Additive Logistic Regression that: AdaBoost is fitting an additive model using forward stagewise additive modeling, with an exponential loss L(y, f) = e^(−y·f). In other words, Boosting isn't a collection of ungrounded heuristics — it's a greedy algorithm that does stepwise optimization in function space.

Unified Boosting perspective:

Objective: find F(x) = Σ_t α_t·h_t(x)  that minimizes  Σᵢ L(yᵢ, F(xᵢ))
Method: forward stagewise — each round only optimizes the newly added tree h_t; previously fixed trees stay unchanged

3. GBDT: fitting "residuals" with negative gradients ​

Exponential loss looks great for classification, but isn't friendly for regression and customized losses. Friedman (2001) generalized the above framework into Gradient Boosting Decision Trees (GBDT):

Each round does three things:
① Compute the "pseudo-residual" for sample i under the current model F_{t-1}
       rᵢ = −∂L(yᵢ, F(xᵢ)) / ∂F(xᵢ)      ← negative gradient of the loss w.r.t. model output
② Fit a regression tree to pseudo-residuals rᵢ (not to yᵢ!)
③ F_t(x) = F_{t-1}(x) + η · h_t(x)        ← add after scaling by learning rate η

Compare GBDT with the gradient descent you learned in Optimization and Gradient Descent: ordinary gradient descent updates parameters w along the negative gradient in parameter space; GBDT updates the function F itself along the negative gradient in function space — "each tree = one gradient step in function space." The pseudo-residual is what the current model "still owes," and the tree fits it to pay that debt.

Three must-know GBDT details:

  • Learning rate η (shrinkage): Each new tree contributes only η of amplitude (typical: 0.01~0.1). Smaller learning rate means less overfitting for the same number of total steps, but requires more trees. Learning rate and number of trees are a pair of parameters that must be tuned together.
  • Subsampling: Each round uses only a subset of samples to train the next tree, introducing randomness to prevent overfitting — this is "row sampling," complementary to random forests.
  • Pluggable loss functions: Squared loss (L2), absolute loss (L1), Huber, LogLoss, custom — all work. This lets GBDT elegantly handle regression, classification, and ranking (LambdaMART is a famous variant of GBDT for learning to rank).

4. Why Boosting reduces bias ​

Now let's compare the two schools side by side — this is the core landscape for understanding ensemble learning:

Bagging / Random ForestBoosting / GBDT
Ensemble methodParallel, voting/averagingSerial, stepwise accumulation
Goal of each treeEach fits its own Bootstrap sampleFits the "residual/gradient" of the previous round
Single tree configDeep trees, no pruningShallow trees, weak models (to prevent per-step overfitting)
Mainly reducesVariance (decorrelation + averaging)Bias (stepwise approaching target function)
Main riskLimited gains when inter-tree correlation is too highSensitive to noise (noise in residuals also gets fitted)

The reason Boosting reduces bias is intuitive: the expressive power of the additive model F(x) = Σ α_t·h_t(x) increases with T — each step adds a tree to "where we still owe." As long as the step size is small and there are many steps, we can approximate any complex target function. The cost is that it is no longer naturally noise-resistant: if a sample has a labeling error, the residual is all noise, and subsequent trees will desperately fit the noise. This explains two industrial experience rules: random forests are more robust on dirty data, GBDT has a higher accuracy ceiling on clean data.

Boosting is sensitive to noise

A rule of thumb: when a dataset has obvious noise or few samples, random forests often outperform GBDT; when the data is large, clean, and has strong feature signals, GBDT has a higher ceiling. So don't blindly trust "XGBoost is always best" — running a random forest baseline first is always the more stable first step.

4. XGBoost and LightGBM: Engineering Boosting ​

GBDT had excellent performance in the 2000s but was extremely slow to train — every round scanned all samples and computed gains at all candidate split points for all features. After 2014, XGBoost and LightGBM turned GBDT from "can run" to "train hundreds of millions of samples per day" through systematic engineering improvements. The competition between them is one of the most exciting chapters in ML engineering history.

1. XGBoost: four key improvements ​

XGBoost (arXiv:1603.02754), published by Chen and Guestrin in 2016, fully industrialized GBDT:

  • Second-order Taylor expansion + regularization term: The objective function is expanded to second order for each tree's output (using gradients gᵢ and Hessian values hᵢ), and an explicit regularization term γ·T + ½λ·‖w‖² is added (T = number of leaves, w = leaf weights). Second-order information makes split gain calculations more precise, and the regularization term directly writes "the cost of tree complexity" into the objective function — consistent with the ideas in Overfitting and Regularization.
  • Approximate splits and weighted quantile sketches: Instead of iterating over all values of a feature, candidate split points are selected based on weighted quantiles of the second-order gradient. Particularly effective for features with extremely uneven distributions (e.g., "click count").
  • Sparsity-aware splits: Explicitly learns "which direction to send missing values," eliminating the need to impute missing values during preprocessing.
  • Column blocks and cache optimization: Data is pre-sorted by feature, stored in compressed blocks, and scanned block-by-block in parallel during splits — this is the key source of XGBoost's training speed.

2. LightGBM: three bolder engineering moves ​

Microsoft's LightGBM (arXiv:1706.08374, 2017) goes further on the path of "fast":

  • Histogram-based algorithm: Discretizes continuous features into 256 bins, searching for split points only at bin boundaries. Training complexity drops from O(samples × features) to O(bin_count × features), and histograms can be accumulated and reused, reducing memory from "one float per sample" to "one byte per sample." The cost is split-point precision loss (no subdivision within bins) — in practice this is negligible when bin count is increased.
  • GOSS (Gradient-based One-Side Sampling): During training, keep only the samples with large gradients (top a%), and randomly sample b% from the small-gradient samples, scaling them by (1−a)/b to maintain the distribution. Intuition: large gradient = large residual = most useful for model improvement; small-gradient samples can be sampled. Reduces per-round sample size to about one-tenth while preserving accuracy.
  • EFB (Exclusive Feature Bundling): Bundles mutually exclusive features (features that are rarely nonzero at the same time, like columns from one-hot encoding) into a single histogram, reducing feature dimensions and specifically addressing computational waste on high-dimensional sparse data.
  • Leaf-wise growth: XGBoost uses level-wise (layer-by-layer) growth; LightGBM uses leaf-wise — always splitting the leaf with the maximum gain. For the same number of trees, accuracy is higher, but it's also more prone to overfitting, so LightGBM must be paired with max_depth or min_data_in_leaf constraints.

3. Selection comparison ​

DimensionXGBoostLightGBM
Split algorithmPre-sorting + exact/approximate quantileHistogram (default 256 bins)
Growth strategyLevel-wise (layer by layer)Leaf-wise (by gain)
Large data / high dimensionsLarge memory overhead, but more stable accuracySignificantly faster, more memory-efficient
Small samples / sparse dataOften more stableWatch out: set min_data_in_leaf to prevent overfitting
EcosystemAll platforms, early, full deployment toolsFast training, friendly early stopping / callbacks
OtherBuilt-in booster='dart', GPU supportGOSS/EFB, native categorical feature support

Practical advice: within hundreds of thousands of samples, pursuing stability, either works (XGBoost is slightly more stable); millions of rows+ with constrained compute, LightGBM is the default choice. Additionally, CatBoost (sequential target encoding for categorical features) is worth trying in scenarios with many categorical features. See Framework Comparison for more framework and ecosystem comparisons.

5. Why Tree Models Dominate Tabular Data ​

Tightening the logic chain: why specifically trees, rather than deep learning, rule tabular data? There are four core reasons.

① Tabular data signals are "sparse piecewise structures," and trees' inductive bias matches perfectly. Tabular data is often generated by discrete, piecewise, highly nonlinear rules ("lend only if age < 30 and credit score > 700"), which is exactly the structure that "axis-aligned splits" of trees excel at expressing. Deep learning's default bias is "smooth continuous functions + learnable feature hierarchies," which either wastes capacity on tabular data or requires extremely large amounts of data.

② No scaling, no complex preprocessing. Neural network gradient descent requires features in the same scale; trees only care about ordering. An empirical fact: for the same tabular data, the engineering effort to get "good results from raw data" with tree models is typically an order of magnitude smaller than deep learning. See Feature Engineering for discussions on preprocessing and normalization.

③ More robust in small-data, high-noise scenarios. Tabular data often has only thousands to hundreds of thousands of samples, while deep learning is "data-hungry" — it trades parameter count for expressive power, and more parameters need more data. Tree models with ensembles have relatively fewer parameters and more stable structures, making it easier to avoid overfitting when data volume is insufficient.

④ Maturity of engineering experience. XGBoost/LightGBM have built-in cross-validation, early stopping, missing value handling, and feature importance, ready to use out of the box; while training a decent tabular neural network requires solving a series of problems: feature embedding, normalization layers, learning rate scheduling, random seed stability, etc.

Comparison dimensionTree models (RF/GBDT)Deep learning (MLP/TabNet, etc.)
Feature scalingNot neededUsually required
Data volume neededSmall to mediumLarge
Categorical/missing value handlingBuilt-inManual
Training costLow~mediumHigh (especially hyperparameter tuning)
InterpretabilityStrong (importance/SHAP)Weak (requires specialized tools)
Unstructured data (graphs/text/audio)Basically unusableAbsolute home field

It's important to emphasize: this section has a clear boundary — tabular data. Once the data is images, text, speech, or graph structures, tree models stand no chance against deep learning — their "piecewise structure prior" fails on these data, while deep learning's hierarchical representations are the correct solution. The two paths each have their own domain. See Deep Learning Foundations and Linear Models (the tradeoff between linear models and trees). The trend in the 2020s is "trees + deep learning" hybrids: use tree models as tabular baselines, deep learning for unstructured features, then ensemble.

6. Practice: Complete RandomForest and LightGBM Workflow ​

Using sklearn's built-in breast cancer classification dataset as an example, demonstrating the complete loop from training to feature importance visualization. First, install dependencies: pip install scikit-learn lightgbm matplotlib pandas.

1. Data preparation and baseline ​

python
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
import pandas as pd

data = load_breast_cancer()
X = pd.DataFrame(data.data, columns=data.feature_names)
y = data.target

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
print(f"Training samples: {X_train.shape[0]}, Test samples: {X_test.shape[0]}, Feature count: {X_train.shape[1]}")

Note stratify=y to ensure class proportions are consistent across train/test sets — this is fundamental for classification tasks. See Model Evaluation and Validation.

2. Random Forest ​

python
rf = RandomForestClassifier(
    n_estimators=500,        # Number of trees
    max_features="sqrt",     # Random features per step m = √p (default for classification)
    min_samples_leaf=2,      # Pre-pruning: at least 2 samples per leaf
    n_jobs=-1,               # Parallel (Bagging is natively parallelizable)
    random_state=42,
    oob_score=True,          # Free out-of-bag error estimate
)
rf.fit(X_train, y_train)

print(f"Training set accuracy: {rf.score(X_train, y_train):.4f}")
print(f"Test set accuracy: {rf.score(X_test, y_test):.4f}")
print(f"Out-of-bag score (OOB): {rf.oob_score_:.4f}")   # ≈ test accuracy, no separate validation set needed

Training accuracy near 1 and test accuracy around 0.96 is a very normal random forest profile — not having 100% training accuracy is not a bad thing; the important thing is stable OOB and test scores. With this step, you can already grab feature importance and plot it directly:

python
import matplotlib.pyplot as plt

importances = pd.Series(rf.feature_importances_, index=X.columns)
importances.sort_values().tail(15).plot.barh(
    figsize=(9, 7), title="RandomForest Feature Importance (based on mean impurity decrease)"
)
plt.tight_layout()
plt.savefig("rf_importance.png", dpi=120)

3. LightGBM ​

python
import lightgbm as lgb

lgb_model = lgb.LGBMClassifier(
    n_estimators=2000,        # Paired with early_stopping, give it enough budget
    learning_rate=0.05,       # shrinkage: small step size to prevent overfitting
    num_leaves=31,            # Core complexity knob for leaf-wise (≈2^5)
    min_child_samples=20,     # Must set: overfitting insurance for leaf-wise growth
    subsample=0.8,            # Row sampling
    colsample_bytree=0.8,     # Column sampling
    random_state=42,
)
lgb_model.fit(
    X_train, y_train,
    eval_set=[(X_test, y_test)],
    callbacks=[lgb.early_stopping(50, verbose=False)],  # Stop if no improvement for 50 rounds
)
print(f"LightGBM test set accuracy: {lgb_model.best_score_['valid_0']['binary_logloss']:.4f}")

early_stopping is the most valuable engineering feature of GBDT models: the number of trees doesn't need manual specification — the model decides when to stop itself. LightGBM has two types of feature importance, which must be distinguished:

python
# ① "split": number of times the feature was used for splitting (sklearn default)
# ② "gain": average information gain brought by the feature (more reflective of "contribution")
gain_imp = pd.Series(
    lgb_model.booster_.feature_importance(importance_type="gain"),
    index=X.columns,
).sort_values()
gain_imp.tail(15).plot.barh(figsize=(9, 7), title="LightGBM Feature Importance (by gain)")
plt.tight_layout()
plt.savefig("lgb_gain_importance.png", dpi=120)

Hyperparameter starting point

A conservative starting point for beginners: learning_rate=0.05 + n_estimators with early_stopping + num_leaves≈31 + min_child_samples≈20. Get it running first, then talk about tuning. See Hyperparameter Tuning Practice for systematic tuning methods, and Common Pitfalls for common failure points.

4. Three types of feature importance ​

TypeDefinitionRisk
split countHow often the feature was selected for splittingFavors features with many values/categories
mean gainAverage information gain from splitsBiased toward high-frequency features used at shallow levels
permutationHow much error increases when the feature is shuffledComputationally expensive, and correlated features cancel each other out

None of these are perfect. Their common flaw is they only look at the marginal contribution of a single feature, ignoring interactions. For more honest attribution, we need SHAP in the next section.

7. Feature Importance and SHAP: Making Tree Models Speak ​

Tree model interpretability doesn't stop at "a single rule." In modern practice, SHAP (SHapley Additive exPlanations) is the de facto standard for explaining tabular models (Lundberg & Lee, 2017).

SHAP's core idea is to explain each prediction as a set of additive attributions:

f(x) = φ₀ + Σⱼ φⱼ        # prediction = baseline value + sum of each feature's contribution

where φⱼ is the SHAP value for feature j, originating from the Shapley value in game theory: in a cooperative game, treating "prediction" as the output of all features cooperating, each feature's "fair share" is determined by averaging its marginal contribution across all possible feature subsets. Fairness is embodied in three axioms — each feature's contribution is symmetric (equal contribution = equal share), additive (sum equals the prediction difference), and null player (irrelevant features contribute 0).

For tree models, there's a huge engineering dividend: TreeSHAP can compute all Shapley values exactly in polynomial time, without approximate sampling like black-box models require. This gives us four layers of explanation:

python
import shap

# Build TreeExplainer for trained LightGBM (exact solution for tree models)
explainer = shap.TreeExplainer(lgb_model)
shap_values = explainer.shap_values(X_test)   # Shape: (n_samples, n_features)

# ① Global view: bee swarm plot — each point is a sample, color indicates feature value
shap.summary_plot(shap_values, X_test, max_display=12)

# ② Single-sample view: force plot — explains "why this patient was classified as malignant"
shap.force_plot(explainer.expected_value, shap_values[0], X_test.iloc[0])

API version note

The newer shap (≥0.45) marks summary_plot / force_plot as deprecated, recommending shap.plots.beeswarm(shap_values) and shap.plots.force(...) instead. This article keeps the classic syntax for compatibility with most tutorials online.

SHAP can answer questions that feature importance can't:

  • Direction: Does a feature push the prediction "up" or "down"? (Random forest gain importance has no direction)
  • Nonlinearity: A feature pushes up at low values but pushes down in the middle range — the SHAP dependence plot makes this immediately obvious
  • Interactions: SHAP dependence plots colored by interaction features can reveal second-order structures like "income's effect is only significant when age > 40"

Combining SHAP values with coarse-grained importance from random forests/GBDT is the standard explanation combo for modern tabular modeling. For full methodology and fairness discussions, see Interpretability and Fairness; for terminology cross-references, see Glossary.

SHAP is not causality

SHAP answers "which features the model attributes its predictions to," not "changing this feature will change the result" — it's doing attribution within the model, not involving the true causal structure of the data. Causal inference requires dedicated experimental design; don't use SHAP values as causal effects.

8. Tradeoffs and Decision Points ​

Compressing the key decisions of this article into a checklist:

  • Single decision tree vs. ensemble: Unless you need an extremely small model or extremely fast single inference (e.g., embedded rules), always ensemble. A single tree is an understanding tool, not a production model.
  • Random forest vs GBDT: Dirty data, few samples, want stability → random forest. Large data, clean, want accuracy ceiling → GBDT family. Recommend running a random forest baseline first, then decide whether to upgrade.
  • XGBoost vs LightGBM: XGBoost is often more stable with small data; LightGBM is faster and more memory-efficient with large data. The accuracy difference is usually small; engineering constraints (deployment environment, categorical features, compute) are often the deciding factors.
  • Prediction accuracy vs. interpretability: The tree family's strength is that both can be had simultaneously — high enough accuracy, plus three layers of explanation: feature importance, SHAP, and path rules. This is why they've remained popular in highly regulated domains like risk control, healthcare, and credit.
  • Trees vs deep learning: Trees for tabular data; deep learning for unstructured data; when budget allows, ensemble both — it often squeezes out a few more percentage points. Avoid "mindlessly applying neural networks."
  • Overfitting protection priority: For Boosting, prioritize learning_rate + early_stopping + min_child_samples; for random forests, prioritize max_features and tree count. For either, one of these three must be present: OOB / early stopping / cross-validation to objectively report error.

One-line summary: The success of the tree family is not a "victory of one algorithm," but the combined result of a simple inductive bias (axis-aligned recursive splitting) + two complementary ensemble mechanisms (parallel variance reduction, serial bias reduction) + two implementations that pushed engineering to the extreme (XGBoost/LightGBM). Understanding this main thread means you understand half of tabular machine learning.

Further Reading ​

References ​