10 Decision Trees

10.1 Overview

Decision trees are nonparametric estimators that approximate an unknown conditional mean or conditional event probability by repeatedly splitting the predictor space and assigning a constant prediction within each resulting region. The classical reference is the Classification and Regression Trees (CART) monograph of Breiman et al. (1984). For econometricians, trees are useful when thresholds, interactions, and regime changes matter but are hard to specify in advance. A tree can learn rules such as “if the output gap is negative and credit spreads are high, predict lower gross domestic product (GDP) growth” without the researcher having to write down all interactions beforehand.

Single trees are interpretable, but that interpretability comes with three weaknesses: trees are discontinuous, unstable w.r.t. small changes in the data, and unable to extrapolate outside the support of the training data. These weaknesses motivate the next chapters: random forests average many randomized trees to stabilize predictions, while boosting builds trees sequentially to correct the current model’s errors.

10.2 Roadmap

  1. We start with regression trees and show how they approximate the conditional mean by partitioning the predictor space into leaves.
  2. We then explain recursive binary splitting, the greedy algorithm used to construct a tree.
  3. Next we turn to classification trees, measures of within-node class mixing called impurity, and the interpretation of leaf predictions as event probabilities.
  4. We then discuss overfitting, stopping rules, and cost-complexity pruning, which collapses weak branches under a penalty for tree size.
  5. Finally, we summarize when trees are useful for econometric work and where they fail.

10.3 Regression Trees

We begin with conditional-mean prediction because it makes the mechanics of a tree especially transparent. Suppose we observe a training sample \{(x_i, y_i)\}_{i=1}^N with x_i \in \mathbb{R}^p and scalar response y_i—in the book’s case convention, the realized values of random pairs (X_i, Y_i). The splitting algorithm operates entirely on these realized values; the random-pair view returns when we analyze bias and variance below. A regression tree with J leaves predicts

\hat f(x) = \sum_{j=1}^J \hat w_j \mathbf{1}\{x \in R_j\},

where the leaves R_1, \ldots, R_J form a partition of the predictor space, \hat w_j is a constant prediction (the leaf value) assigned to every x that falls in region R_j, and \mathbf{1}\{x \in R_j\} is the indicator that equals 1 when x \in R_j and 0 otherwise. A leaf is a terminal region of the tree, so the leaves are exactly the regions R_1, \ldots, R_J in which a prediction is made.

Definition: Regression Tree

A regression tree is a piecewise-constant estimator of the conditional mean function. Each leaf R_j corresponds to a region of the predictor space, and all observations in that region receive the same prediction \hat w_j.

Leaf Predictions Under Squared Loss

If the regions R_1,\ldots,R_J were fixed, the optimal leaf value in region R_j solves

\hat w_j = \arg\min_w \sum_{i:x_i \in R_j} (y_i - w)^2.

Taking the derivative with respect to w gives

-2\sum_{i:x_i \in R_j}(y_i - w) = 0 \quad \Longrightarrow \quad \hat w_j = \frac{1}{N_j}\sum_{i:x_i \in R_j} y_i,

where N_j = \sum_{i=1}^N \mathbf{1}\{x_i \in R_j\} is the number of observations in leaf j.

Once the partition is chosen, the fitted value in each leaf is the sample mean of the responses in that leaf. The remaining problem is choosing the partition.

Econometric Interpretation

A regression tree is a data-driven threshold model. Each split creates a regime such as output_gap <= 0.5 or credit_spread > 1.8, and the fitted value in that regime is a local average. In that sense, trees behave like automatically selected interacted dummy regressions with unknown thresholds.

There is a closer classical relative. A regressogram—the oldest nonparametric regression estimator—partitions the predictor space into fixed bins chosen in advance and reports the sample mean of y within each bin. Equation (10.1) is exactly that estimator: a weighted sum of bin indicators with the leaf means as coefficients. The tree’s contribution is that the bins are chosen adaptively from the data: both the splitting variable and threshold are selected to reduce within-bin heterogeneity. This chapter focuses on prediction and does not develop statistical inference for adaptively chosen tree partitions. For an inference-oriented treatment using honest sample splitting, see Wager and Athey (2018).

Recursive Binary Splitting

In principle we would like to choose regions R_1,\ldots,R_J to minimize the residual sum of squares (RSS)

\text{RSS}(R_1,\ldots,R_J) = \sum_{j=1}^J \sum_{i:x_i \in R_j} (y_i - \bar y_{R_j})^2,

where \bar y_{R_j} is the sample mean within region R_j.

Unlike a neural network, this objective is not optimized by differentiating a smooth parameter vector. The regions R_j change discretely when a split threshold moves past an observation, and the indicator \mathbf{1}\{x \in R_j\} is not differentiable. Trees are therefore grown by searching over candidate splits rather than trained by gradient descent.

A global search over all possible partitions is computationally infeasible. Decision trees therefore restrict attention to binary, axis-aligned splits. We use the term node for any subset of the training data reached by a sequence of splits: an internal node is split further into two child nodes, whereas a terminal node, also called a leaf, is never split again and corresponds to one of the final regions R_j where prediction occurs. At a given node containing observations I_m, the algorithm considers splits of the form

x_{ij} \leq s \qquad \text{versus} \qquad x_{ij} > s,

for feature j \in \{1,\ldots,p\} and threshold s, where x_{ij} denotes the value of predictor j for observation i. Define the left and right child nodes by

I_m^L(j,s) = \{i \in I_m : x_{ij} \leq s\}, \qquad I_m^R(j,s) = \{i \in I_m : x_{ij} > s\}.

The split criterion is the within-child residual sum of squares after replacing each child by its own sample mean. A good split is therefore one that creates two children whose outcomes are more homogeneous than the parent node:

Q_m(j,s) = \sum_{i \in I_m^L(j,s)} (y_i - \bar y_m^L)^2 + \sum_{i \in I_m^R(j,s)} (y_i - \bar y_m^R)^2,

where \bar y_m^L and \bar y_m^R are the sample means of y_i over the left and right child index sets I_m^L(j,s) and I_m^R(j,s). The algorithm chooses the pair (j,s) that minimizes Q_m(j,s). If several pairs attain the minimum, the tie is broken by a fixed deterministic rule, such as the smallest feature index j and then the smallest threshold s, so that the fitted tree is well-defined.

Why only midpoints matter

For a continuous predictor, candidate cutoffs only need to be checked at midpoints between consecutive distinct ordered values observed in the current node. Between two adjacent distinct values, the membership of the left and right child nodes does not change, so the objective is constant. A cutoff is skipped if either child would violate a stated minimum-observation or other admissibility constraint.

Figure 10.1 makes the criterion and this restriction visible at once. The top panel fits the two-leaf tree at three candidate thresholds; the bottom panel traces Q_m(j,s) across the whole range of s.

Two features of the bottom panel carry the argument. The criterion jumps only where an observation changes side, which is why the midpoints exhaust the candidates, and it is not monotone or single-peaked in s, which is why the algorithm evaluates every candidate rather than searching locally. The two non-minimizing fits in the top panel show what a worse split buys: each leaves one child that still mixes observations from both sides of the jump, so that child’s mean sits between the two regimes and fits neither.

After the best split is chosen, the same procedure is applied within each child node. The recursion stops when no admissible split exists or when the best impurity decrease falls below a chosen threshold, such as min_impurity_decrease. The scikit-learn default sets that threshold to zero and accepts a zero-decrease split. This convention can matter for a pure interaction: a first split may have no immediate gain yet reveal profitable child splits at the next depth. We can also limit the number of split levels or require a minimum number of observations in each leaf. This top-down greedy algorithm is called recursive binary splitting. It is computationally feasible, but each split is only myopically optimal given the splits already made; the resulting tree need not minimize RSS among all trees with the same number of leaves.

Figure 10.2 shows what the resulting estimator looks like in the simplest one-predictor case.

Show the code 
import numpy as np
import matplotlib.pyplot as plt
from sklearn.tree import DecisionTreeRegressor

rng = np.random.default_rng(42)
x = np.sort(rng.uniform(-3, 3, 120))
y = (
    1.8
    + 0.3 * x
    + 0.9 * (x > 0.5)
    - 0.6 * (x < -1.5)
    + rng.normal(scale=0.25, size=x.shape[0])
)

tree = DecisionTreeRegressor(max_depth=2, min_samples_leaf=12, random_state=42)
tree.fit(x.reshape(-1, 1), y)

x_grid = np.linspace(-3.2, 3.2, 500)
y_hat = tree.predict(x_grid.reshape(-1, 1))

fig, ax = plt.subplots(figsize=(8, 5))
ax.scatter(x, y, s=18, color="0.35", alpha=0.7, label="Training observations")
ax.plot(x_grid, y_hat, color="C3", linewidth=2.5, label="Tree fit")
ax.set_xlabel("Output gap")
ax.set_ylabel("Next-quarter inflation")
ax.legend(frameon=False)
ax.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()
Figure 10.2: Training observations (gray points) and the fitted depth-2 regression tree (red step function) for next-quarter inflation against the current output gap. Horizontal segments are leaf predictions, and vertical jumps mark split thresholds.

Figure 10.2 shows the defining feature of a regression tree: the fitted function is piecewise constant. This can be useful when the underlying relationship contains genuine threshold effects, but it also means that a tree cannot represent a smooth marginal effect anywhere in the input space. A tree can approximate a smooth relationship by adding more splits, but the fitted function itself is never smooth.

Two Representations of the Same Estimator

A fitted tree can be read in two equivalent ways: as the diagram of splits that gave it its name, and as the piecewise-constant function \hat f(x) = \sum_{j=1}^J \hat w_j \mathbf{1}\{x \in R_j\} introduced at the start of this section. It is worth seeing both for the same object, because the diagram is what makes a tree interpretable while the step function is what it actually predicts. Figure 10.3 does this for the smallest tree that still involves a recursion: two splits and three leaves, fitted to the same sample as Figure 10.2.

Show the code 
import numpy as np
import matplotlib.pyplot as plt
from sklearn.tree import DecisionTreeRegressor, plot_tree

rng = np.random.default_rng(42)
x = np.sort(rng.uniform(-3, 3, 120))
y = (
    1.8
    + 0.3 * x
    + 0.9 * (x > 0.5)
    - 0.6 * (x < -1.5)
    + rng.normal(scale=0.25, size=x.shape[0])
)

# max_leaf_nodes = 3 performs exactly two splits: it repeatedly splits the leaf
# with the largest reduction in RSS until three leaves remain.
small_tree = DecisionTreeRegressor(max_leaf_nodes=3, random_state=42)
small_tree.fit(x.reshape(-1, 1), y)

x_grid = np.linspace(-3.2, 3.2, 500)
y_hat = small_tree.predict(x_grid.reshape(-1, 1))
thresholds = small_tree.tree_.threshold[small_tree.tree_.children_left != -1]

fig, (top, bottom) = plt.subplots(2, 1, figsize=(9, 8.6))

plot_tree(
    small_tree,
    feature_names=["Output gap"],
    filled=True, rounded=True, impurity=False, precision=2, ax=top, fontsize=10,
)

bottom.scatter(x, y, s=18, color="0.35", alpha=0.7, label="Training observations")
bottom.plot(x_grid, y_hat, color="C3", linewidth=2.5, label="Tree fit")
for threshold in thresholds:
    bottom.axvline(threshold, color="0.55", linestyle=":", linewidth=1.5)
bottom.set_xlabel("Output gap")
bottom.set_ylabel("Next-quarter inflation")
bottom.legend(frameon=False)
bottom.grid(True, alpha=0.3)

plt.tight_layout()
plt.show()
Figure 10.3: The same fitted estimator in its two representations. Top: the tree diagram. The root splits on the output gap; only the left child is split again, so the tree has two internal nodes and three leaves, and each leaf reports its prediction (value) and its number of training observations (samples). Bottom: the piecewise-constant function this tree computes, over the same training sample, with dotted vertical lines at the two split thresholds. Each leaf of the diagram is one horizontal segment, and each internal node is one jump.

The two panels contain the same information. Reading the diagram from the root to a leaf gives the conditions that define one region R_j, and the value at that leaf is the \hat w_j predicted there. Reading the step function from left to right passes through those regions in order of the predictor. Note also that only one child of the root is split further: recursive binary splitting treats each node as a separate problem, so nothing forces the two sides of a split to be refined symmetrically, and a variable already used at the root may be split on again below it.

Axis-Aligned Partitions

With more than one predictor, recursive binary splitting produces rectangles in two dimensions and hyperrectangles in higher dimensions. Each additional split refines one existing region along a single coordinate.

Figure 10.4 makes this recursive geometry visible for two predictors.

Show the code 
import numpy as np
import matplotlib.pyplot as plt
from sklearn.tree import DecisionTreeRegressor

rng = np.random.default_rng(7)
n = 250
gap = rng.uniform(-2.5, 2.5, n)
spread = rng.uniform(0.0, 3.5, n)
mu = 0.8 + 0.7 * (gap > 0.0) + 0.5 * (spread > 1.8) + 0.6 * ((gap > 1.0) & (spread > 2.4))
y = mu + rng.normal(scale=0.15, size=n)

X = np.column_stack([gap, spread])
tree2 = DecisionTreeRegressor(max_depth=2, min_samples_leaf=20, random_state=7)
tree2.fit(X, y)

grid_gap = np.linspace(-2.5, 2.5, 250)
grid_spread = np.linspace(0.0, 3.5, 250)
xx, yy = np.meshgrid(grid_gap, grid_spread)
zz = tree2.predict(np.column_stack([xx.ravel(), yy.ravel()])).reshape(xx.shape)

fig, ax = plt.subplots(figsize=(8.2, 5.5))
vmin, vmax = min(y.min(), zz.min()), max(y.max(), zz.max())
ax.contourf(xx, yy, zz, levels=np.linspace(vmin, vmax, 12),
            cmap="viridis", alpha=0.35, vmin=vmin, vmax=vmax)
pts = ax.scatter(gap, spread, c=y, s=18, cmap="viridis",
                 edgecolor="none", alpha=0.8, vmin=vmin, vmax=vmax)
fig.colorbar(pts, ax=ax, label="Next-quarter inflation")
ax.set_xlabel("Output gap")
ax.set_ylabel("Credit spread")
ax.grid(True, alpha=0.2)
plt.tight_layout()
plt.show()
Figure 10.4: A depth-2 regression tree over the output gap (horizontal axis) and credit spread (vertical axis), fit to simulated data. Shaded background regions are the four leaves, colored by their fitted value; dots are the training observations, colored by their observed next-quarter inflation. Both use the same color scale, shown in the colorbar, so a dot that matches its background is well fit by its leaf. The abrupt vertical and horizontal color changes are the tree’s split thresholds.

This geometry makes trees easy to interpret, but it also reveals a limitation: a single split can only move along one variable at a time. If the true decision boundary is smooth or diagonal, a small tree may approximate it poorly.

Question for Reflection

Based on the tree geometry above, when is a discontinuous threshold more substantively plausible than a smooth conditional-mean relationship, and when is the step-function approximation a limitation?

A threshold rule is plausible when institutions create discontinuities, such as tax brackets, credit-score cutoffs, covenant violations, eligibility rules, or regulatory capital thresholds. A smooth marginal effect is more plausible for relationships such as income and consumption, interest rates and investment, or experience and wages, where small changes in the regressor should usually imply small changes in the conditional mean. A tree can approximate smooth effects, but it does so with step functions.

10.4 Classification Trees

Now suppose y_i \in \{0,1\} indicates an event such as recession, default, or policy intervention. A classification tree still partitions the predictor space into leaves, but the natural quantity inside each leaf is now the empirical event probability

\hat p_j = \frac{1}{N_j}\sum_{i:x_i \in R_j} y_i.

This leaf average is the estimated conditional probability of the event for observations landing in leaf R_j. It is also the Bernoulli maximum-likelihood estimator within that leaf.

Probabilities First, Decisions Second

A classification tree does three conceptually distinct things:

  1. It estimates a leaf probability \hat p_j.
  2. It converts that probability into a class label only after a decision threshold is chosen.
  3. The default threshold of 0.5 is optimal only under symmetric classification losses.

For econometric work, the probability forecast is often the main object of interest.

Impurity Measures

To grow a classification tree, we need a criterion that measures how mixed the classes are inside a node. For a node with positive-class share \hat p, two standard impurity measures are:

  • Entropy H(\hat p) = -\hat p \log \hat p - (1-\hat p)\log(1-\hat p)
  • Gini impurity G(\hat p) = 2\hat p(1-\hat p) (the Breiman et al. (1984) form; some software drops the factor of 2, which does not affect which splits are preferred).

Both are zero for pure nodes with \hat p \in \{0,1\} and are maximized at \hat p = 1/2. For the entropy this uses the standard continuous-extension convention 0 \log 0 = 0, which is the limiting value of p \log p as p \downarrow 0; without it the formula is undefined at a pure node.

For a candidate split of node m, the weighted impurity is

\mathcal{I}_m(j,s) = \frac{N_m^L}{N_m} \mathcal{I}(\hat p_m^L) + \frac{N_m^R}{N_m} \mathcal{I}(\hat p_m^R),

where \mathcal{I} is either entropy or Gini, N_m^L and N_m^R are the numbers of observations in the left and right children, N_m = N_m^L + N_m^R is the parent size, and \hat p_m^L and \hat p_m^R are the positive-class shares in the left and right children. The preferred split is the one that minimizes this weighted child impurity, or equivalently maximizes impurity reduction.

The size weights are not a normalization of convenience, and an unweighted average of the two child impurities would be a different and worse criterion. The regression case already uses the weighted form. The sum of squared deviations inside a child equals the number of observations in that child times that child’s mean squared deviation from its own mean, so dividing Q_m(j,s) by the parent size N_m gives exactly the expression above, with the child’s mean squared deviation in place of \mathcal{I}. In both cases the quantity being minimized is a per-observation loss totaled over the observations in the node, and dividing by N_m, which is fixed across all candidate splits of node m, puts that total on the same scale as the parent impurity \mathcal{I}(\hat p_m) so that the two can be differenced.

Dropping the weights breaks that comparison and rewards splitting off a handful of observations, because a child holding a single observation is pure by construction and contributes zero impurity no matter what the split has actually learned. Consider a node of 100 firms of which 50 default, so \hat p_m = 1/2 and the entropy is \log 2 \approx 0.693, and a candidate split that peels off one defaulting firm. The other child then has \hat p_m^R = 49/99 and an entropy of 0.693, indistinguishable from the parent’s at three decimals. The unweighted average \tfrac{1}{2}(0 + 0.693) \approx 0.347 records this as halving the impurity, whereas the weighted criterion returns \tfrac{1}{100}(0) + \tfrac{99}{100}(0.693) \approx 0.686, a reduction of about 0.007. The second number is the honest accounting for a split that has classified one firm and left the remaining 99 as mixed as before. The weights are also what make the comparison with the parent meaningful in general, because the parent share is itself the size-weighted average of the child shares, an identity we use below.

Because entropy is the Bernoulli entropy from the Information Theory chapter, the impurity reduction under entropy is exactly an information gain calculation, in the sense of the entropy and cross-entropy quantities defined in that chapter.

Figure 10.5 compares how entropy and Gini impurity vary with the event share inside a node.

Show the code 
import numpy as np
import matplotlib.pyplot as plt

p = np.linspace(0.001, 0.999, 500)
entropy = -(p * np.log(p) + (1 - p) * np.log(1 - p))
gini = 2 * p * (1 - p)
misclassification = np.minimum(p, 1 - p)

fig, ax = plt.subplots(figsize=(8, 5))
ax.plot(p, entropy, label="Entropy", linewidth=2)
ax.plot(p, gini, label="Gini impurity", linewidth=2)
ax.plot(p, misclassification, label="Misclassification error",
        linewidth=2, linestyle="--")
ax.set_xlabel("Leaf event probability")
ax.set_ylabel("Impurity")
ax.grid(True, alpha=0.3)
ax.legend(frameon=False)
plt.tight_layout()
plt.show()
Figure 10.5: Entropy (blue), Gini impurity (orange), and misclassification error (green, dashed) as functions of the leaf event probability on the horizontal axis. Entropy and Gini impurity are strictly concave; misclassification error is piecewise linear, with a kink at one half.

One could also use misclassification error, \min\{\hat p, 1-\hat p\}. It is a legitimate measure of node impurity, but a poor criterion for growing a tree, and the reason is its shape rather than its steepness: it is piecewise linear in \hat p, with a single kink at \hat p = 1/2, whereas entropy and Gini impurity are strictly concave.

Strict concavity is what allows a criterion to register partial progress. The parent share is the size-weighted average of the child shares,

\hat p_m = \frac{N_m^L}{N_m}\hat p_m^L + \frac{N_m^R}{N_m}\hat p_m^R ,

so any impurity that is linear over the range spanned by the two children passes straight through that average and yields an impurity reduction of exactly zero. Misclassification error is linear on each side of 1/2, so every split whose children fall on the same side of 1/2 registers no gain at all, however much it has separated the classes. For a strictly concave impurity, Jensen’s inequality makes the weighted child impurity strictly smaller than the parent’s whenever the two child shares differ.

For example, take a node with \hat p_m = 0.6 and split it into equal-sized children with \hat p_m^L = 0.65 and \hat p_m^R = 0.55. Misclassification error is 0.4 before the split and \tfrac{1}{2}(0.35) + \tfrac{1}{2}(0.45) = 0.4 after, an impurity reduction of zero, while entropy falls from 0.673 to 0.668 and Gini impurity from 0.480 to 0.475. For that reason, entropy and Gini impurity are commonly used for growing trees, while misclassification error can be used to assess a grown classification tree (Breiman et al. 1984).

Entropy vs Gini

In practice, entropy and Gini lead to similar trees, and both are the in-leaf value of a proper scoring rule evaluated at the leaf’s own fitted probability: entropy is the negative Bernoulli log-likelihood, and Gini is twice the Brier score, since the mean squared deviation of a 0-1 outcome from the leaf share is \hat p(1-\hat p). Gini is slightly simpler algebraically and is the default in many software packages, while entropy links directly to information theory.

Leaf Probabilities and Class Predictions

Once the tree has been grown, prediction is straightforward:

  • For a probability forecast, report the leaf average \hat p_j.
  • For a class prediction, compare \hat p_j with a decision threshold c.

Under symmetric 0-1 loss, the threshold is c=0.5. Under asymmetric losses, the threshold changes. If a false negative costs C_{FN} and a false positive costs C_{FP}, then predicting class 1 is optimal whenever

\hat p_j > \frac{C_{FP}}{C_{FP} + C_{FN}}.

The decision threshold matters in economics and finance because missed recessions, missed defaults, and false alarms rarely have the same cost.

10.5 Tree Size, Overfitting, and Pruning

A sufficiently deep tree can often interpolate the training sample by creating singleton or outcome-homogeneous leaves. Zero training error is not guaranteed, however: observations with identical predictor vectors but different outcomes cannot be separated by axis-aligned splits, and minimum-leaf-size or other split constraints may also prevent isolation. A close fit to training outcomes need not translate into accurate predictions for new observations.

For regression trees, additional splits can capture finer features of the conditional mean, but smaller leaves estimate their means from fewer observations. The partition itself also depends on the training sample: changing a split near the top changes the observations available to every split below it. Together, these mechanisms can make predictions from deep trees sensitive to the training sample. Pruning collapses branches, trading a coarser approximation for potentially more stable predictions.

Bias-Variance View

Fix a predictor value x_0 and draw a new response Y_0 from the conditional distribution at that point, independently of the training sample. Write f(x_0)=\mathbb{E}[Y_0] and \sigma^2(x_0)=\operatorname{Var}(Y_0). With finite second moments, expected squared prediction error decomposes as

\mathbb{E}\left[(Y_0-\hat f(x_0))^2\right] = \underbrace{\operatorname{Var}\left(\hat f(x_0)\right)}_{\text{estimation variance}} +\underbrace{\left(\mathbb{E}[\hat f(x_0)]-f(x_0)\right)^2}_{\text{squared bias}} +\underbrace{\sigma^2(x_0)}_{\text{irreducible noise}}.

The expectation on the left averages over both new responses and repeated training samples. Estimation variance measures how much the prediction at x_0 changes across training samples; bias is the difference between its average and f(x_0) and reflects the entire fitting procedure, including restrictions on tree size.

Pruning can improve prediction when the reduction in variance outweighs the increase in squared bias. The decomposition does not guarantee that every pruning step helps.

For time series, a later observation is not automatically independent of the training sample; the technical note below explains the relevant condition.

In the independent-draw setup, the mean-zero noise Y_0-f(x_0) is orthogonal to the fitted prediction, so their cross-term vanishes. For time series, use the forecast-origin information set \mathcal{F}_t, containing what is available at time t, as in the Cross-Validation chapter. Define the future innovation as \varepsilon_{t+1}=Y_{t+1}-\mathbb{E}[Y_{t+1}\mid\mathcal{F}_t]. A forecast \hat y_{t+1\mid t} issued at t for Y_{t+1} must be \mathcal{F}_t-measurable. With finite second moments,

\mathbb{E}[\varepsilon_{t+1}\hat y_{t+1\mid t}\mid\mathcal{F}_t] =\hat y_{t+1\mid t}\,\mathbb{E}[\varepsilon_{t+1}\mid\mathcal{F}_t] =0.

This conditional orthogonality supplies the corresponding cross-term cancellation in a forecast-error decomposition; independence of successive observations is unnecessary.

There are two common ways to control complexity.

Pre-pruning

The first approach is to stop the tree from growing too far in the first place. Common controls include:

  • max_depth: maximum number of split levels
  • min_samples_leaf: minimum number of observations allowed in a leaf
  • min_samples_split: minimum number of observations required before a node may be split
  • max_leaf_nodes: direct cap on the number of terminal leaves

These hyperparameters trade bias against variance. A shallow tree may miss important nonlinearities. A deep tree may fit noise and become unstable.

Cost-Complexity Pruning

The second approach is to first grow a large tree and then prune it back. A standard criterion is the cost-complexity objective

C_\alpha(T) = \sum_{i=1}^N (y_i - \hat f_T(x_i))^2 + \alpha |T|,

where |T| is the number of leaves in tree T and \alpha \geq 0 is a tuning parameter. The minimization runs over a specific, finite domain: the subtrees of the grown tree T_0 obtained by collapsing internal nodes, not over all conceivable trees. Restricting the search to these subtrees makes it feasible. A classical result of Breiman et al. (1984) makes the calculation efficient: as \alpha increases from zero, the minimizing subtrees form a finite nested sequence T_0 \supseteq T_1 \supseteq \cdots \supseteq \{\text{root}\}, obtained by successively collapsing the “weakest link”—the internal node whose branch contributes least fit improvement per leaf. One therefore computes the entire sequence once and only has to choose \alpha, which should be tuned with the time-aware validation schemes discussed in the warning below.

Larger \alpha penalizes bigger trees more heavily, so the selected subtree becomes smaller. This is directly analogous to complexity penalization in econometrics: we do not accept an extra parameter, or here an extra leaf, unless it reduces the in-sample fit criterion enough to justify the added flexibility.

Econometric Warning

For macroeconomic or financial time series, tree depth and pruning parameters must be tuned with an honest time-aware validation scheme. Randomly shuffled folds can leak future information and make a tree appear much more accurate than it would be in real time. The correct benchmark is the actual information set available at the forecast origin.

Link to the Next Chapters

Random forests are designed primarily to stabilize high-variance trees through averaging, whereas boosting often emphasizes sequential bias reduction by correcting the current fit. Both methods can change both bias and variance, and both keep the partitioning logic of trees while improving predictive performance.

10.6 Trees for Econometric Data: Strengths and Limits

Decision trees are useful when the relationship between predictors and outcomes is plausibly nonlinear and interaction-heavy. Typical examples include:

  • default prediction from balance-sheet ratios
  • recession prediction from financial indicators
  • household demand or credit response with threshold effects
  • heterogeneous policy targeting when prediction is the goal

Small trees are also attractive when communication matters, because the fitted rule can be read as a sequence of if-then statements.

But trees also have sharp limitations.

Strengths of decision trees

  • They automatically discover interactions and threshold effects.
  • They require little preprocessing and are not sensitive to variable scaling.
  • They can handle many predictors without manually writing basis expansions or interaction terms.
  • Small trees can be interpreted as explicit decision rules.

Limitations of decision trees

  • They produce discontinuous predictions.
  • They cannot extrapolate trends outside the support of the training data.
  • They are unstable: small sample changes can produce different trees.
  • On many held-out prediction tasks, a single unrestricted tree is less accurate than a tuned tree ensemble because averaging or sequential fitting can reduce its instability.
  • Split variables and thresholds are chosen adaptively for prediction, so they should not be given a causal interpretation.

For time-series econometrics, the extrapolation issue is especially important. If GDP growth, inflation, or asset volatility moves outside the historical range seen during training, a tree can only return the constant prediction from the outermost leaf. That is often too rigid for forecasting.

Figure 10.6 gives a concrete illustration using the quarterly Federal Reserve Economic Data collection (FRED-QD) (McCracken and Ng 2020). The target is annualized GDP growth in quarter t+1, and the predictors are five macro and financial variables dated quarter t: lagged GDP growth, inflation, the term spread between long- and short-maturity yields, which is a well-known leading indicator of business cycle turning points (Estrella and Mishkin 1998), the log volatility index (VIX), an option-implied measure of stock-market volatility, and the unemployment rate.1

The fitted depth-2 tree splits first on inflation, separating the 174 quarters with inflation below 9.57% from the 14 high-inflation quarters. Three features are worth noting. First, the two branches then split on different variables: the low-inflation branch splits on lagged GDP growth, isolating the 18 quarters in which output had contracted by more than half a percentage point at an annual rate and predicting 0.68 for them against 3.73 for the remaining 156, while the high-inflation branch splits on the log VIX and predicts -4.56 for the four quarters that combined high inflation with high volatility. Nothing forces the two sides of a split to continue with the same variable, because each split is chosen conditionally on the region reached so far. Second, only two of the five predictors appear in the fitted tree at all; with three splits to spend, the greedy search never reaches the term spread or the unemployment rate. Allowing one more level of depth does bring the term spread into the tree; the depth-2 fit simply has no room for it. Third, the piecewise-constant structure is visible in the leaf values. Regardless of how far predictors move during the test period, the tree can return only one of four constants, the most extreme of which is -4.56. This is a direct manifestation of the extrapolation limitation.

Show the code 
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.tree import DecisionTreeRegressor, plot_tree

raw = pd.read_csv("data/fred_qd_current.csv")
fred_qd = raw.iloc[2:].copy()
fred_qd["sasdate"] = pd.to_datetime(fred_qd["sasdate"])
for col in fred_qd.columns:
    if col != "sasdate":
        fred_qd[col] = pd.to_numeric(fred_qd[col], errors="coerce")
fred_qd = fred_qd.sort_values("sasdate").reset_index(drop=True)

gdp_growth  = 400 * np.log(fred_qd["GDPC1"]).diff()
inflation   = 400 * np.log(fred_qd["CPIAUCSL"]).diff()
term_spread = fred_qd["GS10"] - fred_qd["TB3MS"]
log_vix     = np.log(fred_qd["VIXCLSx"])

macro = pd.DataFrame({
    "y":           gdp_growth.shift(-1),
    "gdp_lag1":    gdp_growth,
    "infl_lag1":   inflation,
    "spread_lag1": term_spread,
    "vix_lag1":    log_vix,
    "unrate_lag1": fred_qd["UNRATE"],
}).dropna()

feature_cols = ["gdp_lag1", "infl_lag1", "spread_lag1", "vix_lag1", "unrate_lag1"]
split = int(len(macro) * 0.75)
X_train = macro.iloc[:split][feature_cols].values
y_train = macro.iloc[:split]["y"].values

tree = DecisionTreeRegressor(max_depth=2, random_state=42)
tree.fit(X_train, y_train)

fig, ax = plt.subplots(figsize=(11, 5.5))
plot_tree(
    tree,
    feature_names=["GDP growth", "Inflation", "Term spread", "Log VIX", "Unemployment"],
    filled=True, rounded=True, impurity=False, precision=2, ax=ax, fontsize=10,
)
plt.tight_layout()
plt.show()
Figure 10.6: Depth-2 regression tree fit to one-step-ahead GDP growth (annualized log-difference of GDPC1) using five FRED-QD predictors. Each internal node shows the split variable and threshold; leaf nodes show the mean prediction (value), measured in annualized percentage points, and the number of training observations (samples). Colors indicate the leaf prediction: darker shading corresponds to higher predicted growth. The tree is trained on the first 75% of available quarters.

The four displayed leaf constants make the extrapolation limit concrete: new observations can be routed to a different leaf, but the tree cannot continue a trend beyond the fitted leaf values.

10.7 Summary

Key Takeaways
  1. Decision trees partition predictor space into leaves that estimate conditional means or event probabilities.
  2. Regression trees select splits by squared-error reduction, while classification trees use impurity measures.
  3. Recursive splitting captures threshold effects and interactions without specifying them in advance.
  4. Depth limits, minimum leaf sizes and pruning control the complexity of a greedily fitted tree.
Common Pitfalls
  • Leaf averages and adaptively selected splits do not identify marginal or causal effects.
  • A classification threshold of 0.5 need not be appropriate when the two error types have different costs.
  • Tree predictions cannot extrapolate a smooth trend beyond the outermost splits.
  • Randomly shuffled validation can give misleading depth choices for dependent forecasting data.

10.8 Exercises

Exercise 10.1: Growing and Pruning a Regression Tree

Consider the following training sample with one predictor x and response y:

Observation x y
1 1 2.0
2 2 3.0
3 3 2.5
4 4 5.0
  1. Compute the root-node mean and the root-node residual sum of squares (RSS).
  2. Evaluate the candidate one-split regression trees with cutoffs s \in \{1.5, 2.5, 3.5\}. For each split, compute the leaf means and total RSS. Which split is chosen by recursive binary splitting?
  3. Apply recursive binary splitting once more, inside the left child of the split chosen in Part 2. The admissible cutoffs there are s\in\{1.5,\,2.5\}. Give the within-child RSS of each, the three leaf values of the resulting tree, and its total RSS. Stop after this second split, so the grown tree T_0 has three leaves.
  4. Consider the cost-complexity objective C_\alpha(T) = \sum_{i=1}^4 (y_i - \hat f_T(x_i))^2 + \alpha |T|, where |T| is the number of leaves and \alpha\geq0. Evaluate it for the two-split tree T_0, the one-split tree T_1 of Part 2, and the root-only tree (the no-split tree consisting of a single root leaf that predicts the full-sample mean). For which values of \alpha is each preferred? Check that the selected trees form a nested sequence as \alpha increases.
  5. Give the two-split tree’s prediction at x=2.7 and at x=5. What limitation of regression trees does this illustrate for forecasting trending macroeconomic variables?

Exam level: suitable as-is but they would be considered to be easy questions. Parts 1-4 test the mechanics of tree growing and pruning, while Part 5 checks understanding of extrapolation.

Only the three observations in the left child are involved. Each candidate child gets its own mean, not the mean of the left child as a whole.

The three trees have one, two and three leaves. Write all three penalized criteria explicitly as lines in \alpha and compare them pairwise; the tree preferred at a given \alpha is the one whose line is lowest there.

Part 1: Root node

The root-node mean is

\bar y = \frac{2 + 3 + 2.5 + 5}{4} = 3.125.

So the root-only tree’s RSS is

(2 - 3.125)^2 + (3 - 3.125)^2 + (2.5 - 3.125)^2 + (5 - 3.125)^2 = 5.1875.

Part 2: Candidate splits

For s=1.5:

  • Left leaf: \{2\} with mean 2
  • Right leaf: \{3, 2.5, 5\} with mean 3.5

Hence

\text{RSS}(1.5) = 0 + (3-3.5)^2 + (2.5-3.5)^2 + (5-3.5)^2 = 3.5.

For s=2.5:

  • Left leaf: \{2, 3\} with mean 2.5
  • Right leaf: \{2.5, 5\} with mean 3.75

Hence

\text{RSS}(2.5) = (2-2.5)^2 + (3-2.5)^2 + (2.5-3.75)^2 + (5-3.75)^2 = 3.625.

For s=3.5:

  • Left leaf: \{2, 3, 2.5\} with mean 2.5
  • Right leaf: \{5\} with mean 5

Hence

\text{RSS}(3.5) = (2-2.5)^2 + (3-2.5)^2 + (2.5-2.5)^2 + (5-5)^2 = 0.5.

Therefore the best one-split tree uses the cutoff s=3.5.

Part 3: The second split

The right child of the chosen split is the single observation \{5\}, which cannot be split. The recursion therefore continues in the left child, which contains observations 1, 2 and 3 with responses \{2, 3, 2.5\}, mean 2.5 and RSS 0.5.

For s=1.5:

  • Left leaf: \{2\} with mean 2
  • Right leaf: \{3, 2.5\} with mean 2.75

Q(1.5) = 0 + (3-2.75)^2 + (2.5-2.75)^2 = 0.125.

For s=2.5:

  • Left leaf: \{2, 3\} with mean 2.5
  • Right leaf: \{2.5\} with mean 2.5

Q(2.5) = (2-2.5)^2 + (3-2.5)^2 + 0 = 0.5.

The cutoff s=1.5 is chosen, and it reduces the left child’s RSS from 0.5 to 0.125. The cutoff s=2.5 achieves nothing at all: it separates observation 3, whose response happens to equal the mean of the node, so both children keep the parent’s mean.

The grown tree T_0 has three leaves,

\hat f_{T_0}(x)= \begin{cases} 2.00, & x \leq 1.5, \\ 2.75, & 1.5 < x \leq 3.5, \\ 5.00, & x > 3.5, \end{cases}

with total RSS 0.125 + 0 = 0.125. Note what a further split would do: the middle leaf holds two observations with different responses, so splitting it at s=2.5 would drive the training RSS to exactly zero with four leaves and four observations. Interpolation is one split away, which is why the next part is needed.

Part 4: The cost-complexity path

Collecting the three trees:

Tree Leaves RSS C_\alpha
T_0 (two splits) 3 0.125 0.125 + 3\alpha
T_1 (one split) 2 0.500 0.500 + 2\alpha
Root only 1 5.1875 5.1875 + \alpha

Comparing the first two,

0.125 + 3\alpha < 0.5 + 2\alpha \quad \Longleftrightarrow \quad \alpha < 0.375,

and comparing the second with the root,

0.5 + 2\alpha < 5.1875 + \alpha \quad \Longleftrightarrow \quad \alpha < 4.6875.

So T_0 is selected for 0\leq\alpha<0.375, T_1 for 0.375<\alpha<4.6875, and the root-only tree for \alpha>4.6875, with ties at the two breakpoints. The selected trees are nested, T_0 \supset T_1 \supset \{\text{root}\}, and each step removes one leaf.

The two breakpoints are exactly the two weakest-link prices. Collapsing the internal node at x=1.5 costs 0.5-0.125=0.375 in RSS and saves one leaf; collapsing the root then costs 5.1875-0.5=4.6875 and saves one more. The cheaper branch is pruned first, which is why the sequence is nested rather than jumping between unrelated subtrees. Note that comparing T_0 with the root directly would give a crossing at \alpha=2.53125, but that comparison never decides anything: T_1 is strictly better than both trees throughout that neighborhood.

Part 5: Predictions and extrapolation

From the three-leaf tree of Part 3, x=2.7 falls in the middle leaf and x=5 in the rightmost leaf, so

\hat f_{T_0}(2.7) = 2.75, \qquad \hat f_{T_0}(5) = 5.

The constant outer-leaf prediction illustrates the extrapolation problem: once x is above the largest cutoff, the tree cannot continue an upward trend. The tree simply returns the same value regardless of how far x moves beyond the training range.

Exercise 10.2: Classification Trees, Impurity, and Decision Thresholds

A bank uses a single predictor x (the debt-to-equity ratio) to predict firm default, where y=1 denotes default and y=0 denotes no default.

Firm x y
1 0.5 0
2 1.2 0
3 1.8 1
4 2.5 0
5 3.0 1
6 3.5 1

Use natural logarithms throughout, with \log 2 \approx 0.693, \log 3 \approx 1.099, and \log(4/3) \approx 0.288.

  1. Compute the root-node entropy and root-node Gini impurity.
  2. For the candidate split at x=2.15, compute the weighted child entropy and weighted child Gini impurity.
  3. Repeat Part 2 for the candidate split at x=1.5. Which split is preferred under entropy, and which under Gini? For the preferred split, report the default probability in each leaf and the implied class prediction under a 0.5 threshold.
  4. Suppose the cost of a false negative is C_{FN} and the cost of a false positive is C_{FP}. Show that predicting default minimizes expected cost when \hat p > \frac{C_{FP}}{C_{FP}+C_{FN}}. Evaluate the threshold for C_{FN}=4 and C_{FP}=1.
  5. Apply the cost-sensitive threshold from Part 4 to the leaves of both candidate splits from Parts 2 and 3. Report which leaves change their predicted class relative to the 0.5 rule. Explain why a leaf changes in one split but not the other, and state the general interval of leaf probabilities for which lowering the threshold changes the decision.

Exam level: the required logarithms are supplied, so all five parts are calculator-free. Parts 1–3 test the mechanics of impurity calculations, Part 4 derives the cost-sensitive threshold, and Part 5 applies it to the fitted leaves.

The split at x=2.15 places firms 1-3 on the left and firms 4-6 on the right. Compute the default share in each child, then take the weighted average of the child impurities.

Compare the expected loss from predicting class 1 with the expected loss from predicting class 0 when the leaf event probability is \hat p.

Part 1: Root impurity

The root node has default share

\hat p = \frac{3}{6} = \frac{1}{2}.

Hence the root entropy is

H\left(\frac{1}{2}\right) = -\frac{1}{2}\log\left(\frac{1}{2}\right) - \frac{1}{2}\log\left(\frac{1}{2}\right) = \log 2 \approx 0.693.

The root Gini impurity is

G\left(\frac{1}{2}\right) = 2 \cdot \frac{1}{2} \cdot \frac{1}{2} = 0.5.

Part 2: Split at x=2.15

Left leaf: firms 1, 2, 3 with labels (0,0,1), so \hat p_L = 1/3.

Right leaf: firms 4, 5, 6 with labels (0,1,1), so \hat p_R = 2/3.

Entropy in each child is the same:

H\left(\frac{1}{3}\right) = -\frac{1}{3}\log\left(\frac{1}{3}\right) - \frac{2}{3}\log\left(\frac{2}{3}\right) \approx 0.637.

Therefore the weighted child entropy is

\frac{3}{6} \cdot 0.637 + \frac{3}{6} \cdot 0.637 = 0.637.

The child Gini impurity is also the same in both leaves:

G\left(\frac{1}{3}\right) = 2 \cdot \frac{1}{3} \cdot \frac{2}{3} = \frac{4}{9} \approx 0.444.

So the weighted child Gini impurity is

\frac{3}{6}\cdot \frac{4}{9} + \frac{3}{6}\cdot \frac{4}{9} = \frac{4}{9} \approx 0.444.

Part 3: Split at x=1.5

Left leaf: firms 1 and 2 with labels (0,0), so \hat p_L = 0.

Right leaf: firms 3, 4, 5, 6 with labels (1,0,1,1), so \hat p_R = 3/4.

Left-leaf entropy is H(0)=0, and right-leaf entropy is

H\left(\frac{3}{4}\right) = -\frac{3}{4}\log\left(\frac{3}{4}\right) - \frac{1}{4}\log\left(\frac{1}{4}\right) \approx 0.562.

Hence the weighted child entropy is

\frac{2}{6}\cdot 0 + \frac{4}{6}\cdot 0.562 \approx 0.375.

Left-leaf Gini is G(0)=0, and right-leaf Gini is

G\left(\frac{3}{4}\right) = 2 \cdot \frac{3}{4}\cdot \frac{1}{4} = \frac{3}{8} = 0.375.

Hence the weighted child Gini impurity is

\frac{2}{6}\cdot 0 + \frac{4}{6}\cdot 0.375 = 0.25.

So the split at x=1.5 is preferred under both entropy and Gini because it produces the lower weighted child impurity. It is worth noting that the two criteria agree here; they need not in general, which is why the question asks for both.

For that preferred split the leaf default probabilities are \hat p_L = 0 and \hat p_R = 3/4, so under a 0.5 threshold the implied classifier predicts no default on the left leaf and default on the right leaf.

Part 4: Asymmetric decision threshold

If we predict default, the only mistake is a false positive, which occurs with probability 1-\hat p. The expected loss is therefore

L(1) = C_{FP}(1-\hat p).

If we predict no default, the only mistake is a false negative, which occurs with probability \hat p. The expected loss is

L(0) = C_{FN}\hat p.

Predicting default is optimal when L(1) < L(0):

C_{FP}(1-\hat p) < C_{FN}\hat p.

Rearranging gives

C_{FP} < \hat p(C_{FP}+C_{FN}) \quad \Longleftrightarrow \quad \hat p > \frac{C_{FP}}{C_{FP}+C_{FN}}.

If C_{FN}=4 and C_{FP}=1, the optimal threshold is

\frac{1}{1+4} = 0.2.

So with asymmetric losses, a bank should predict default for a firm whenever the estimated default probability exceeds 0.2, not 0.5.

Part 5: Applying the threshold

Applying the two thresholds to the leaves of both candidate splits:

Split Leaf \hat p 0.5 rule 0.2 rule
x=1.5 (preferred) left 0 no default no default
x=1.5 (preferred) right 3/4 default default
x=2.15 left 1/3 no default default
x=2.15 right 2/3 default default

Only one leaf changes: the left leaf of the split at x=2.15, whose default probability 1/3 lies between the two thresholds. That is the general principle—lowering the threshold from 0.5 to 0.2 reclassifies exactly those leaves with \hat p \in (0.2, 0.5], and no others. The leaves of the preferred split have \hat p = 0 and \hat p = 3/4, both far outside that interval, so its classifier is unchanged.

Two lessons follow. First, the direction of the change is what asymmetric costs demand: because a missed default costs four times a false alarm, the rule becomes more willing to declare default, and a leaf that was previously called safe on a one-in-three default rate is now flagged. Second, the preferred split’s classifications are unchanged for this particular threshold change from 0.5 to 0.2. A different cost ratio can move the threshold past a leaf’s probability and change its class even when that probability is close to zero or one. Reporting the leaf probabilities therefore shows which cost changes could alter the decisions.

10.9 References

Breiman, Leo, Jerome H. Friedman, Richard A. Olshen, and Charles J. Stone. 1984. Classification and Regression Trees. Belmont, CA: Wadsworth.
Estrella, Arturo, and Frederic S. Mishkin. 1998. “Predicting U.S. Recessions: Financial Variables as Leading Indicators.” Review of Economics and Statistics 80 (1): 45–61. https://doi.org/10.1162/003465398557320.
McCracken, Michael W., and Serena Ng. 2020. “FRED-QD: A Quarterly Database for Macroeconomic Research.” Working Paper 26872. National Bureau of Economic Research. https://doi.org/10.3386/w26872.
Wager, Stefan, and Susan Athey. 2018. “Estimation and Inference of Heterogeneous Treatment Effects Using Random Forests.” Journal of the American Statistical Association 113 (523): 1228–42. https://doi.org/10.1080/01621459.2017.1319839.

Footnotes

  1. This is a pedagogical final-vintage FRED-QD example. A full real-time GDP forecasting evaluation would also need to account for release calendars and data revisions.↩︎