Under active development. A stable version is expected in November 2026. Feedback is welcome by email or via GitHub issues.

11 Random Forests

11.1 Overview

Random forests address the main weakness of single decision trees: high variance. A single tree can change sharply when the sample changes slightly, especially when early splits are unstable. Random forests reduce this instability by averaging many trees grown on perturbed versions of the data and by injecting additional randomness into the split search. The original random-forest paper is Breiman (2001).

For econometricians, random forests are attractive when prediction is the goal and the conditional mean or event probability depends on nonlinearities and interactions of unknown form. They work especially well on tabular data such as firm-level balance-sheet variables, household characteristics, or macro-financial predictors. At the same time, they remain nonparametric prediction tools rather than structural models: they do not extrapolate well, and their split structure should not be read causally.

11.2 Roadmap

We begin with bagging and show why averaging unstable trees can reduce variance substantially.
We then explain random feature selection, which turns bagging into a random forest by limiting the candidate predictors at each split.
Next we study bootstrap sampling, out-of-bag evaluation, and the forest-as-weights interpretation.
We then discuss the main hyperparameters and the bias-variance trade-offs they control.
Finally, we summarize where random forests fit well in econometric work and where caution is needed.

11.3 From Bagging to Random Forests

Suppose we have a predictor \hat T(x) produced by a deep regression tree. If we grow many such trees on slightly different datasets and average them, we obtain the bagged predictor

\hat f_B(x) = \frac{1}{B}\sum_{b=1}^B \hat T_b(x),

where \hat T_b(x) is the prediction of tree b and B is the number of trees.

Bagging stands for bootstrap aggregating:

draw a bootstrap sample from the training data (with replacement, of the same size N as the training sample)
grow a tree on that sample
repeat many times
average the resulting predictions in regression, or average class probabilities or use majority voting in classification

A related variant, often called subagging (Bühlmann and Yu 2002), instead draws subsamples of size s<N, usually without replacement (we write s rather than L, which denotes a terminal leaf below). Both approaches create many perturbed training sets; the random forest then applies equal weights 1/B to the resulting tree predictions.

The idea works best when the individual model being averaged is unstable. Deep trees have exactly this property: small sample changes can alter early splits and therefore the whole fitted function. This high variance is a weakness for a single tree but an opportunity for averaging.

Variance of an Averaged Forest

The variance calculation must distinguish two sources of randomness. Let D denote the random training sample and let \Theta_b collect the independently generated bootstrap and feature-selection variables for tree b. At a fixed prediction point x, write

T_b(x)=T(D,\Theta_b;x).

The unhatted T_b(x) is the same tree prediction as \hat T_b(x) above, now viewed as a random variable through its dependence on (D,\Theta_b); we drop the hat exactly because the moments below are taken over that randomness.

Conditional on D, independent randomizations produce independent and identically distributed tree predictions. Define their conditional mean and variance by

m(D,x)=\mathbb{E}_{\Theta}[T(D,\Theta;x)\mid D], \qquad v(D,x)=\operatorname{Var}_{\Theta}(T(D,\Theta;x)\mid D).

For two distinct trees, the law of total covariance gives

\begin{aligned} \operatorname{Cov}(T_b(x),T_{b'}(x)) &= \mathbb{E}_D\!\left[\operatorname{Cov}(T_b(x),T_{b'}(x)\mid D)\right] \\ &\quad+ \operatorname{Cov}_D\!\left(\mathbb{E}[T_b(x)\mid D], \mathbb{E}[T_{b'}(x)\mid D]\right) \\ &= 0+\operatorname{Cov}_D(m(D,x),m(D,x)) \\ &= \operatorname{Var}_D(m(D,x)), \qquad b\neq b'. \end{aligned}

The second equality uses conditional independence and \mathbb{E}[T_b(x)\mid D]=m(D,x). The covariance can nevertheless remain nonzero unconditionally because both trees use the same training sample. For a single tree, the law of total variance similarly gives

\begin{aligned} \operatorname{Var}(T_b(x)) &= \operatorname{Var}_D\!\left(\mathbb{E}[T_b(x)\mid D]\right) + \mathbb{E}_D\!\left[\operatorname{Var}(T_b(x)\mid D)\right] \\ &= \operatorname{Var}_D(m(D,x)) + \mathbb{E}_D[v(D,x)]. \end{aligned}

Now consider the B-tree average \hat f_B(x)=B^{-1}\sum_{b=1}^B T_b(x). Its conditional mean is

\begin{aligned} \mathbb{E}[\hat f_B(x)\mid D] &= \frac{1}{B}\sum_{b=1}^B \mathbb{E}[T_b(x)\mid D] \\ &= m(D,x). \end{aligned}

Conditional independence also implies

\begin{aligned} \operatorname{Var}(\hat f_B(x)\mid D) &= \frac{1}{B^2}\sum_{b=1}^B \operatorname{Var}(T_b(x)\mid D) \\ &= \frac{v(D,x)}{B}. \end{aligned}

Applying the law of total variance to the forest average therefore gives

\begin{aligned} \operatorname{Var}[\hat f_B(x)] &= \operatorname{Var}_D\!\left(\mathbb{E}[\hat f_B(x)\mid D]\right) + \mathbb{E}_D\!\left[\operatorname{Var}(\hat f_B(x)\mid D)\right] \\ &= \underbrace{\operatorname{Var}_D(m(D,x))}_{\text{training-sample component}} + \underbrace{\frac{1}{B}\mathbb{E}_D[v(D,x)]}_{\text{finite-forest Monte Carlo component}}. \end{aligned}

Increasing B removes only the Monte Carlo component. The training-sample component remains because every tree depends on the same dataset. This decomposition also yields an equicorrelation representation, but its correlation must be interpreted unconditionally, over repeated training samples and tree randomizations. If \sigma^2(x)=\operatorname{Var}(T_b(x))>0, then

\rho(x) = \frac{\operatorname{Cov}(T_b(x),T_{b'}(x))}{\sigma^2(x)} = \frac{\operatorname{Var}_D(m(D,x))}{\operatorname{Var}(T_b(x))}, \qquad b\neq b',

so \rho(x)\geq 0 and

\operatorname{Var}_D(m(D,x))=\rho(x)\sigma^2(x).

Combining this identity with the single-tree variance decomposition gives

\begin{aligned} \mathbb{E}_D[v(D,x)] &= \sigma^2(x)-\operatorname{Var}_D(m(D,x)) \\ &= \sigma^2(x)[1-\rho(x)]. \end{aligned}

Substituting both components into the forest variance decomposition yields

\operatorname{Var}[\hat f_B(x)] = \sigma^2(x)\left[\rho(x)+\frac{1-\rho(x)}{B}\right].

Thus the nonvanishing term in the equicorrelation formula is shared training-sample variability, not positive conditional covariance among independently randomized trees with D fixed. In plain language, growing more trees averages away randomness in how the trees are constructed; it does not average away sensitivity to which training sample happened to be observed.

Why Bagging Alone Is Not Enough

If some predictors are very strong, many bootstrap trees keep splitting on the same variables near the root and can produce very similar fitted rules. At a fixed x, tightly concentrated tree predictions correspond to a small conditional variance v(D,x), so averaging then changes the prediction little relative to using a single randomized tree: \hat f_B(x) is already close to m(D,x). Any instability of m(D,x) across training samples remains because averaging does not remove the training-sample component \operatorname{Var}_D(m(D,x)).

Random forests address this directly. At each split, instead of considering all p predictors, the algorithm draws a random subset of size m and searches only within that subset. The best split among those m predictors is used.

Random feature selection, also called feature subsampling, does two things at once:

it diversifies the fitted rules by forcing trees to try different split variables
it may slightly weaken each individual tree because the globally best predictor is not always available

This additional randomization changes both parts of the variance decomposition. It can reduce \operatorname{Var}_D(m(D,x)) by making the conditional mean rule less sensitive to dominant predictors, while increasing \mathbb{E}_D[v(D,x)] because tree predictions are more dispersed conditional on the data. The second effect enters the forest variance only through the Monte Carlo term \mathbb{E}_D[v(D,x)]/B, so it is harmless once B is large—its practical meaning is that a strongly randomized forest needs more trees. Neither direction is guaranteed, and random feature selection can also change bias, which no amount of averaging removes. The actual performance of the forest depends on the relative magnitudes of these effects and must be assessed empirically.

The bias-variance trade-off is therefore explicit. Deep individual trees have low bias but high variance. Averaging many randomized trees lowers variance, while random feature selection and bootstrap perturbations can slightly increase bias because each tree is deliberately denied the strongest split it could use. Forests work well when the regression function contains nonlinearities and interactions that trees can capture and when variance reduction dominates this modest bias increase. They can underperform a well-specified simpler model when the true signal is close to linear, when extrapolation is needed, or when the sample is too small for flexible trees to be estimated reliably.

11.4 The Random-Forest Predictor

The variance argument above applies to numerical tree predictions. We now separate the two output types used in practice. For regression, the random-forest predictor is

\hat f_B(x) = \frac{1}{B}\sum_{b=1}^B \hat T_b(x).

For classification, each tree can produce either a class vote or an estimated leaf probability. In econometric applications, the probability forecast is usually more informative than the hard class label. A forest therefore often predicts

\hat p_B(x) = \frac{1}{B}\sum_{b=1}^B \hat p_b(x),

where \hat p_b(x) is the event probability from tree b.

Random forests are commonly applied to:

default prediction
recession probability forecasting
treatment assignment risk scoring
nonlinear forecasting of inflation, output, or firm sales

11.5 Forest Weights and Distributional Outlook

A random forest can also be viewed as a data-adaptive local averaging estimator. First consider trees grown by subsampling without replacement, and let S_b denote the subsample used to grow tree b. Let L_b(x) denote the terminal leaf containing the forecast point x in tree b, and let N_b(x) be the number of sampled observations in that leaf. The regression prediction can be written as

\hat f_B(x) = \sum_{i=1}^N w_i(x)y_i, \qquad w_i(x) = \frac{1}{B}\sum_{b=1}^B \frac{\mathbf{1}\{i \in S_b\}\,\mathbf{1}\{x_i \in L_b(x)\}}{N_b(x)}.

With bootstrap sampling, observation i can appear in tree b’s in-bag sample with multiplicity m_{ib}\in\{0,1,2,\ldots\}. Because the stored leaf mean counts duplicate draws, the exact weight is

w_i(x)=\frac{1}{B}\sum_{b=1}^B \frac{m_{ib}\,\mathbf{1}\{x_i\in L_b(x)\}}{N_b(x)},

where N_b(x) now counts bootstrap draws in the leaf. In either sampling design, the weights are nonnegative and sum to one.

The weight w_i(x) is large when observation i often lands in the same terminal leaf as the forecast point. The construction resembles a fixed-distance local weighting estimator, but the forest learns its neighborhoods through tree partitions instead of prescribing them with a distance formula.

This weighting view is also the bridge to more advanced forests. Quantile regression forests keep more than the leaf mean and use the same neighborhood weights to estimate a conditional distribution (Meinshausen 2006). Local-linear forests go in a different direction: they retain the forest neighborhood but fit a local linear model rather than a constant inside the neighborhood. The quantile regression forest section develops the distributional version.

Figure 11.1 makes the variance-stabilization effect of averaging visible in a one-predictor example.

Show the code

import numpy as np
import matplotlib.pyplot as plt
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import RandomForestRegressor

rng = np.random.default_rng(123)
x = np.sort(rng.uniform(-3, 3, 180))
y = np.sin(1.4 * x) + 0.25 * x + rng.normal(scale=0.35, size=x.shape[0])

tree = DecisionTreeRegressor(max_depth=None, min_samples_leaf=3, random_state=123)
forest = RandomForestRegressor(
    n_estimators=300,
    min_samples_leaf=3,
    max_features=1,
    random_state=123,
)

tree.fit(x.reshape(-1, 1), y)
forest.fit(x.reshape(-1, 1), y)

x_grid = np.linspace(-3.1, 3.1, 500)
tree_hat = tree.predict(x_grid.reshape(-1, 1))
forest_hat = forest.predict(x_grid.reshape(-1, 1))

fig, axes = plt.subplots(1, 2, figsize=(12, 4.8), sharey=True)

axes[0].scatter(x, y, s=16, color="0.35", alpha=0.7)
axes[0].plot(x_grid, tree_hat, color="C3", linewidth=2.3)
axes[0].set_title("Single Deep Tree")
axes[0].set_xlabel("Predictor")
axes[0].set_ylabel("Outcome")
axes[0].grid(True, alpha=0.3)

axes[1].scatter(x, y, s=16, color="0.35", alpha=0.7)
axes[1].plot(x_grid, forest_hat, color="C0", linewidth=2.3)
axes[1].set_title("Random Forest")
axes[1].set_xlabel("Predictor")
axes[1].grid(True, alpha=0.3)

plt.tight_layout()
plt.show()

Figure 11.1: Simulated outcomes drawn from Y=\sin(1.4X)+0.25X+\varepsilon (gray points), a single deep-tree fit (red line, left panel), and a 300-tree random-forest fit (blue line, right panel). Both models use the same 180 training observations.

The single tree reacts strongly to local sample noise. The forest remains nonlinear, but averaging across many trees stabilizes the fit.

11.6 Bootstrap Sampling and Out-of-Bag Evaluation

Each tree is grown on a bootstrap sample of size N drawn with replacement from the original sample. Because sampling is with replacement, some observations appear multiple times in a given tree’s training set and some observations are omitted.

For a particular observation i, the probability of being omitted from a bootstrap sample is

\left(1-\frac{1}{N}\right)^N \to e^{-1} \approx 0.368.

So roughly 36.8% of the trees leave a given observation out. Those trees form the out-of-bag (OOB) set for that observation.

Out-of-Bag Predictions

For each observation i, the OOB prediction is the average over trees that did not include i in their bootstrap sample:

\hat f^{\text{OOB}}(x_i) = \frac{1}{|\mathcal{B}_i^{\text{OOB}}|} \sum_{b \in \mathcal{B}_i^{\text{OOB}}} \hat T_b(x_i),

where \mathcal{B}_i^{\text{OOB}} = \{\, b : i \notin \text{the bootstrap sample used to grow tree } b \,\} is the set of trees whose bootstrap sample excludes observation i, and |\mathcal{B}_i^{\text{OOB}}| is its cardinality.

The OOB construction produces an approximately out-of-sample prediction for observation i, because the trees contributing to that prediction were not trained on i. (We avoid the word honest here: in the forest literature it has the specific technical meaning of separating the observations used to choose splits from those used to compute leaf values, a stronger condition that OOB averaging does not deliver.) The OOB error is the average loss of these OOB predictions over all observations; for regression it is

\text{OOB error} = \frac{1}{N}\sum_{i=1}^N \big(y_i - \hat f^{\text{OOB}}(x_i)\big)^2.

Under independent and identically distributed (i.i.d.) sampling, OOB error is a useful internal estimate of prediction error for new observations drawn from the same distribution; with enough trees, it is often close to a same-size holdout estimate (Breiman 2001). This argument relies on an exchangeability/random-holdout target. With temporal, spatial, or clustered dependence, standard row-wise bootstrap/OOB validation need not reproduce the dependence structure or information set relevant for forecasting. OOB error can therefore be misleading, and rolling, blocked, spatial, or cluster-level validation is usually more appropriate (Roberts et al. 2017).

OOB Inherits the Sampling Design

Out-of-bag error is convenient because it comes “for free” during estimation, but it inherits the sampling design built into the forest. If the data are serially dependent, clustered, revised over time, or otherwise non-i.i.d., standard OOB error can be misleading.

Question for Reflection

For a panel dataset of firms observed over time, which prediction target is standard OOB error closest to: new firms, new time periods, or new firm-time cells? What validation split would better match each target?

11.7 Hyperparameters and Trade-Offs

Although forests are often robust, they still require tuning. The most important hyperparameters are the following.

Number of Trees

The number of trees B mainly controls Monte Carlo error in the forest average.

More trees lower simulation noise and stabilize predictions.
More trees do not usually create overfitting in the same way as adding parameters to ordinary least squares (OLS).
After a point, gains are negligible and only computation rises.

Number of Candidate Features per Split

This hyperparameter is often called max_features or mtry.

Smaller max_features diversifies the fitted trees and can lower training-sample variability. It also makes individual trees noisier conditional on the data, a Monte Carlo cost that averaging removes at rate 1/B and that therefore mainly requires more trees. As discussed above, it can also increase bias: when informative predictors are unavailable at useful splits, trees must substitute weaker splits that may systematically miss part of the signal. Increasing B estimates the average of this randomized fitting rule more precisely; it does not correct bias in that average rule.
Larger max_features makes each tree stronger but more similar to the others.
If one predictor is much stronger than the others, reducing max_features can materially improve the ensemble by forcing other variables into the split search.

Common rules of thumb are to try roughly p/3 candidate features per split for regression and roughly \sqrt{p} for classification. These particular defaults come from the widely used randomForest implementation (Liaw and Wiener 2002); Breiman (2001) himself worked with considerably smaller values in his random-feature experiments, namely m=1 and m=\lfloor \log_2 p + 1 \rfloor. They are not econometric laws; they are starting values that should be checked with a validation design appropriate for the data structure.

Leaf Size and Tree Depth

Random forests typically use deep trees, but the leaf size still matters.

Smaller leaves reduce bias but increase the variance of individual trees.
Larger leaves smooth the prediction surface and can improve performance when the signal is weak or noisy.

Monotonicity Constraints in Some Implementations

Monotonicity constraints are not part of the classical random-forest theory developed in this chapter, but some modern implementations allow them. The idea is straightforward: if economic reasoning implies that a predictor should move the fitted outcome only upward or only downward, the ensemble can be restricted to respect that sign pattern. (The same device reappears for gradient boosting in a later chapter.) This can make a forest more credible when the unrestricted fit shows local reversals that are hard to defend economically.

Mechanically, the constraint is enforced inside each tree by rejecting any candidate split on the constrained feature that would produce children violating the sign pattern. Tree-growing algorithms additionally propagate bounds through the tree so that splits on other features cannot eventually create leaves that break the global monotonicity. Since every tree in the forest respects the constraint, the averaged prediction does too.

In scikit-learn. Since version 1.4, RandomForestRegressor accepts a monotonic_cst argument with one integer per feature: +1 for increasing, -1 for decreasing, and 0 for no constraint:

from sklearn.ensemble import RandomForestRegressor

model = RandomForestRegressor(
    monotonic_cst=[1, 0, 0],
    n_estimators=500,
    max_features="sqrt",
)

See the scikit-learn example Monotonic Constraints for a visual comparison of constrained and unconstrained fits.

The same warning applies as elsewhere: a monotonicity restriction is a predictive shape constraint, not a structural identification device. If the sign restriction is wrong or only locally valid, the forest can become systematically misspecified.

Practical Bias-Variance Summary

The forest design has two levers:

Make each tree strong.
Diversify the fitted rules so that averaging removes more of their sampling instability.

The best tuning balances those two goals rather than maximizing either one in isolation. Changing max_features can affect bias, training-sample variability, and finite-forest Monte Carlo variability at the same time.

11.8 Econometric Interpretation and Limits

Random forests are well suited to prediction with many candidate interactions. They can uncover nonlinear combinations of predictors without the econometrician specifying them manually, which is valuable in cross-sectional or panel-style prediction tasks with rich covariate sets.

But several limits remain.

No Extrapolation

Like single trees, forests average leaf values. That means they remain local averaging estimators. If the forecast point lies outside the historical support of the training data, the forest cannot extrapolate a trend; it returns a weighted average of observed outcomes from nearby leaves.

Interpretation Is Predictive, Not Structural

A variable that appears important for a forest is not necessarily a causal driver. The forest is optimized for prediction, not identification.

Standard OOB Evaluation Can Fail for Dependent Data

In macroeconomic and financial forecasting, three distinct questions are easy to run together.

The first concerns algorithmic randomization. Row bootstrap samples diversify the fitted trees; in that role they are part of the learning algorithm, not automatically an approximation to the sampling distribution of an estimator. Whether row sampling, subsampling, or contiguous-block sampling improves prediction depends on the data-generating process, feature construction, and forecast target.

The second concerns dependence-aware estimation and inference. If block resampling is meant to preserve a time-series sampling law, its validity requires a named scheme, assumptions on dependence, an appropriate block-length sequence or selector, and a result for the estimator and target at hand. A generic block bootstrap is therefore a design option, not an automatic remedy.

The third concerns evaluation, and it is decisive for real-time forecasting. An observation at date t is OOB for trees that did not draw it, but those trees can still be trained on dates after t. Its OOB prediction can therefore use information unavailable at the forecast origin. Changing the resampling scheme does not by itself fix an evaluation that ignores the arrow of time. The direction of the distortion is not guaranteed. With a structural break halfway through the sample, post-break observations can contaminate a pre-break OOB prediction and make the assessment pessimistic. At an early post-break origin, later observations from the new regime can make it optimistic. Rolling or expanding validation windows instead reproduce the actual information set at each origin (Roberts et al. 2017).

Econometric Warning

For forecasting problems with serial dependence, publication lags, and real-time data revisions, standard random-forest OOB error is not a valid real-time performance measure. Use rolling or expanding validation windows that replicate the actual information set at the forecast origin.

Variable Importance: Useful but Imperfect

Forests are often accompanied by variable-importance measures. The two most common versions are:

Impurity-based importance: sum the reduction in the split criterion, such as residual sum of squares (RSS) or Gini impurity (both defined in the chapter on decision trees), attributed to each predictor across all trees.
Permutation importance: shuffle one predictor in an out-of-bag or validation sample, recompute predictive performance, and use the resulting deterioration as the importance measure.

Both measures can be useful screening devices, but they have limitations (Strobl et al. 2007):

impurity-based importance can favor variables with many possible split points
importance does not measure causal effect size
correlated predictors can split the signal across variables and make each one look less important than it truly is

Permutation importance answers a different predictive perturbation question from impurity reduction: how much worse this fitted forest predicts when one column is broken. It does not identify the variable’s causal effect or its unique information contribution.

Two cautions matter especially for the dependent, correlated data of this book. First, permuting a column row by row destroys the predictor’s temporal ordering along with its relationship to the target, so in a time-series setting the deterioration mixes the loss of the variable’s signal with the destruction of its serial structure. Permuting contiguous blocks can serve as a sensitivity check because it preserves dependence within each block, but it still breaks dependence across block boundaries and its result depends on the chosen block length. Block permutation therefore reduces rather than eliminates the serial-structure problem; it does not isolate a clean signal contribution. Second, when predictors are correlated, row-wise permutation evaluates the forest at feature combinations that never occur in the data—a permuted term spread paired with an unpermuted unemployment rate may describe a macroeconomic state that has never existed. The resulting number measures how much the fitted function relies on that column once its joint relationship with the others is broken, which is not the same as how much predictive information the variable carries.

Figure 11.3 illustrates the stabilization argument directly using the quarterly Federal Reserve Economic Data collection (FRED-QD) (McCracken and Ng 2020). The target is annualized gross domestic product (GDP) growth in quarter t+1, while the predictors are dated quarter t.¹ The top panel shows test-period predictions from a single unrestricted decision tree; the bottom panel shows predictions from a 500-tree random forest trained on the same predictors. The single-tree forecast jumps between a small set of leaf constants. Averaging 500 step-function predictions produces a less jumpy forest path. The plot cannot by itself demonstrate in-sample overfitting or identify statistical bias and variance, which are repeated-sample concepts. It does show that neither model approaches the annualized GDP-growth rates of roughly -33\% and +30\% observed in 2020; the forest is especially flat at those extremes. A tree ensemble averages leaf constants estimated from historical data and therefore cannot forecast outside the range those constants span.

Show the code

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import RandomForestRegressor

raw = pd.read_csv("data/fred_qd_current.csv")
fred_qd = raw.iloc[2:].copy()
fred_qd["sasdate"] = pd.to_datetime(fred_qd["sasdate"])
for col in fred_qd.columns:
    if col != "sasdate":
        fred_qd[col] = pd.to_numeric(fred_qd[col], errors="coerce")
fred_qd = fred_qd.sort_values("sasdate").reset_index(drop=True)

gdp_growth  = 400 * np.log(fred_qd["GDPC1"]).diff()
inflation   = 400 * np.log(fred_qd["CPIAUCSL"]).diff()
term_spread = fred_qd["GS10"] - fred_qd["TB3MS"]
log_vix     = np.log(fred_qd["VIXCLSx"])

macro = pd.DataFrame({
    "date":        fred_qd["sasdate"],
    "y":           gdp_growth.shift(-1),
    "gdp_lag1":    gdp_growth,
    "infl_lag1":   inflation,
    "spread_lag1": term_spread,
    "vix_lag1":    log_vix,
    "unrate_lag1": fred_qd["UNRATE"],
}).dropna()

feature_cols = ["gdp_lag1", "infl_lag1", "spread_lag1", "vix_lag1", "unrate_lag1"]
split = int(len(macro) * 0.75)
X_train = macro.iloc[:split][feature_cols].values
y_train = macro.iloc[:split]["y"].values
X_test  = macro.iloc[split:][feature_cols].values
y_test  = macro.iloc[split:]["y"].values
test_dates = macro.iloc[split:]["date"]

single_tree = DecisionTreeRegressor(random_state=42)
forest = RandomForestRegressor(n_estimators=500, max_features="sqrt", random_state=42)
single_tree.fit(X_train, y_train)
forest.fit(X_train, y_train)

fig, axes = plt.subplots(2, 1, figsize=(11, 6), sharex=True)
for ax, pred, label, color in zip(
    axes,
    [single_tree.predict(X_test), forest.predict(X_test)],
    ["Single tree (unconstrained depth)", "Random forest (500 trees)"],
    ["C1", "C0"],
):
    ax.plot(test_dates, y_test, color="0.5", linewidth=1, label="Realized GDP growth")
    ax.plot(test_dates, pred,   color=color, linewidth=1.5, label=label)
    ax.axhline(0, color="0.8", linewidth=0.7, linestyle="--")
    ax.set_ylabel("pct. pts. ann.")
    ax.legend(frameon=False, loc="upper right")
    ax.grid(True, alpha=0.25)

axes[1].set_xlabel("Quarter")
plt.tight_layout()
plt.show()

Figure 11.3: Out-of-sample one-step-ahead GDP growth forecasts over the last 25% of the FRED-QD sample from a single deep decision tree (top) and a 500-tree random forest (bottom), each trained on five lagged predictors. The gray line is realized GDP growth, measured as the annualized log-difference of GDPC1.

11.9 Summary

Key Takeaways

Random forests average randomized trees to reduce the variability of a single-tree predictor.
Adding trees reduces finite-forest Monte Carlo variance at rate 1/B while leaving training-sample uncertainty.
Random feature subsets diversify split opportunities, with a bias-variance trade-off that must be validated.
Out-of-bag predictions aggregate trees whose bootstrap samples excluded the observation.

Common Pitfalls

OOB error does not reproduce historical forecasting when its trees use observations from later dates.
Variable importance is not a causal ranking, especially when predictors are correlated.
Forest averages cannot extrapolate a smooth trend beyond the training support.
Offering nearly all predictors at every split can leave trees highly correlated.

11.10 Exercises

Exercise 11.1: Variance Reduction in a Random Forest

Fix a point x_0. Let D be a random training sample and let tree b use an independent randomization \Theta_b, so T_b=T(D,\Theta_b;x_0). All moments in this exercise are unconditional, over repeated draws of both D and the tree randomizations. Suppose

\mathbb{E}[T_b] = \mu, \qquad \operatorname{Var}[T_b] = \sigma^2>0,

and for all b \neq b',

\operatorname{Corr}(T_b,T_{b'}) = \rho, \qquad 0 \le \rho \le 1.

Conditional on D, the independent randomizations make different tree predictions independent. The unconditional correlation \rho arises because all trees depend on the same random training sample. Under this construction, nonnegativity is automatic because \rho\sigma^2=\operatorname{Var}_D(m(D,x_0))\geq 0.

Define the forest predictor as

\bar T_B = \frac{1}{B}\sum_{b=1}^B T_b.

Show that \mathbb{E}[\bar T_B] = \mu.
Derive the variance formula \operatorname{Var}[\bar T_B] = \sigma^2\left[\rho + \frac{1-\rho}{B}\right].
Let \sigma^2 = 9, \rho = 0.2, and B=100. Compute the variance of a single tree and the variance of the forest. By what percentage is variance reduced?
What is the limit of \operatorname{Var}[\bar T_B] as B \to \infty? Relate this limit to the training-sample component \operatorname{Var}_D(m(D,x_0)) from the chapter and explain why increasing B cannot remove it. Then explain why the effect of changing max_features is an empirical bias-variance trade-off rather than a guaranteed reduction in one fixed correlation parameter.
How many trees are enough? For \rho>0, use Parts 2 and 4 to show that \frac{\operatorname{Var}[\bar T_B]}{\operatorname{Var}[\bar T_\infty]} = 1 + \frac{1-\rho}{\rho B}, so that the relative excess variance from using finitely many trees depends on \rho and B but not on \sigma^2. Find the smallest B for which this excess is at most 5% when \rho = 0.2, and repeat for \rho = 0.05.
Holding \sigma^2 and bias fixed, explain why the lower-\rho case needs more trees to approach its own variance limit even though that limit is lower. Relate this result to a forest using more aggressive random feature selection. Finally, treat the boundary \rho=0 separately: state \operatorname{Var}(\bar T_B) and explain why relative distance from the zero limiting variance is not defined.

Exam level: suitable as-is. Parts 1–2 derive forest variance, Part 3 applies it, Part 4 separates the two variance components, Part 5 determines the tree count, and Part 6 interprets the result and its boundary case.

Hint for Part 2

Expand the variance of the average into own-variance and covariance terms, and count how many of each there are. Use \operatorname{Cov}(T_b,T_{b'}) = \rho\sigma^2 for b \neq b'.

Hint for Part 3

First compute the bracketed term. Then compare with the single-tree variance 9.

Solution

Part 1: Expectation

By linearity of expectation,

\mathbb{E}[\bar T_B] = \mathbb{E}\left[\frac{1}{B}\sum_{b=1}^B T_b\right] = \frac{1}{B}\sum_{b=1}^B \mathbb{E}[T_b] = \frac{1}{B}\sum_{b=1}^B \mu = \mu.

So averaging does not change the mean prediction.

Part 2: Variance

Using the variance-of-a-sum formula,

\operatorname{Var}[\bar T_B] = \frac{1}{B^2} \left( \sum_{b=1}^B \operatorname{Var}[T_b] + \sum_{b \neq b'} \operatorname{Cov}(T_b,T_{b'}) \right).

There are B variance terms, each equal to \sigma^2, and B(B-1) covariance terms, each equal to \rho \sigma^2. Therefore

\operatorname{Var}[\bar T_B] = \frac{1}{B^2}\left(B\sigma^2 + B(B-1)\rho \sigma^2\right).

Factor out \sigma^2:

\operatorname{Var}[\bar T_B] = \sigma^2 \frac{B + B(B-1)\rho}{B^2} = \sigma^2\left(\frac{1}{B} + \frac{B-1}{B}\rho\right).

Rewriting gives

\operatorname{Var}[\bar T_B] = \sigma^2\left[\rho + \frac{1-\rho}{B}\right].

Part 3: Numerical example

For a single tree,

\operatorname{Var}[T_b] = 9.

For the forest,

\operatorname{Var}[\bar T_{100}] = 9\left[0.2 + \frac{0.8}{100}\right] = 9(0.208) = 1.872.

So the variance falls from 9 to 1.872.

The percentage reduction is

\frac{9 - 1.872}{9} \times 100\% = 79.2\%.

Part 4: Infinite-forest limit

As B \to \infty,

\operatorname{Var}[\bar T_B] \to \sigma^2 \rho.

Under the decomposition in the chapter, \sigma^2\rho=\operatorname{Var}_D(m(D,x_0)). Averaging can eliminate finite-forest Monte Carlo variability, but not the training-sample variability of the infinite-forest predictor. Changing max_features can alter this training-sample component by changing the fitted rule, but it can also change individual-tree variability and bias. Its net effect is therefore a tuning question, not a guaranteed consequence of lowering a fixed conditional tree correlation.

Part 5: How many trees are enough

Dividing the Part 2 formula by the Part 4 limit \sigma^2\rho,

\frac{\operatorname{Var}[\bar T_B]}{\operatorname{Var}[\bar T_\infty]} = \frac{\sigma^2\left[\rho + (1-\rho)/B\right]}{\sigma^2\rho} = 1 + \frac{1-\rho}{\rho B},

and \sigma^2 cancels: how close a finite forest sits to its own limit is a question about \rho and B alone, not about how noisy the individual trees are.

Requiring the excess to be at most 5% gives

\frac{1-\rho}{\rho B} \le 0.05 \quad\Longleftrightarrow\quad B \ge \frac{20(1-\rho)}{\rho}.

For \rho = 0.2 this is B \ge 20(0.8)/0.2 = 80 trees; for \rho = 0.05 it is B \ge 20(0.95)/0.05 = 380.

Part 6: Interpretation and the zero-correlation boundary

Holding \sigma^2 and bias fixed, the lower-\rho case needs almost five times as many trees to come within 5% of its own limit. There is no contradiction: lowering \rho lowers the variance floor \sigma^2\rho, but raises the ratio (1-\rho)/\rho that governs relative distance from that floor because more of the single-tree variance must be averaged away. Within this stylized model, more aggressive random feature selection represented by the lower \rho should therefore be paired with more trees. In an actual tuning exercise, changing max_features can change \sigma^2, \rho, and bias together, so validation must judge the complete predictor.

At \rho=0,

\operatorname{Var}(\bar T_B)=\frac{\sigma^2}{B}\longrightarrow0.

The finite-B variance is well defined and converges to zero, but its ratio to the zero limiting variance is not. An absolute tolerance, rather than a relative percentage above the limit, is needed at this boundary.

Exercise 11.2: Out-of-Bag Predictions and Dependent Data

Suppose a random forest is trained on a sample of size N, and each tree uses a bootstrap sample of size N drawn with replacement from the original data.

Show that the probability a fixed observation i is not selected in one bootstrap sample is q_N=\left(1-\frac{1}{N}\right)^N, and conclude that q_N\to e^{-1} as N \to \infty. If K_i is the number of trees whose bootstrap samples omit observation i, show that K_i\sim\operatorname{Binomial}(B,q_N) and find \mathbb{E}[K_i]. For B=500 and large N, approximately how many trees enter the OOB prediction?
To isolate finite-forest Monte Carlo variability, condition on the training sample D and on K_i=k>0. Suppose the OOB-eligible tree predictions are conditionally independent with common variance v_i. Compare the variance of their k-tree average with a hypothetical B-tree average made from B independent draws from this same OOB-eligible tree distribution.
Replace K_i by its expectation Bq_N and show that the approximate variance ratio from Part 2 is 1/q_N\to e. For v_i=9 and B=500, compare 9/184 with the same-distribution benchmark 9/500. Explain why the factor e is not an exact finite-N, finite-B identity, referring to both the randomness of K_i and the event K_i=0. Why is this hypothetical benchmark not necessarily the Monte Carlo variance of the usual full forest?
Explain why OOB prediction can serve as an internal validation device under i.i.d. sampling.
You are forecasting quarterly inflation with a long macroeconomic time series. Explain why standard OOB error is not a valid real-time evaluation method in this setting, why the direction of its distortion is not guaranteed, and suggest a more appropriate validation design.
Two highly correlated predictors contain nearly the same signal. A forest assigns each a small individual permutation importance. Explain why this result does not establish that either variable is uninformative, and propose one grouped perturbation that better measures their joint predictive role.

Exam level: suitable as-is. Part 1 derives the OOB count, Parts 2–3 analyze its Monte Carlo consequence, Part 4 explains the i.i.d. validation logic, Part 5 addresses real-time dependence, and Part 6 tests an importance diagnostic.

Hint for Part 1

In one draw, the chance of not selecting observation i is 1-1/N. The bootstrap sample contains N independent draws with replacement.

Hint for Parts 2–3

Conditional on K_i=k>0, the variance of an average of k independent eligible tree predictions is v_i/k. For the exact unconditional calculation, ask whether \mathbb{E}[1/K_i\mid K_i>0] equals 1/\mathbb{E}[K_i].

Hint for Part 5

Ask whether a bootstrap resample respects time order, publication lags, and real-time information sets. To determine why the sign is ambiguous, place a structural break halfway through the sample and consider pre-break and early post-break forecast origins separately.

Solution

Part 1: OOB probability

In a single bootstrap draw, the probability of not selecting observation i is

1 - \frac{1}{N}.

Because the bootstrap sample contains N draws with replacement, the probability that i is never selected is

q_N=\left(1-\frac{1}{N}\right)^N.

Using the standard limit,

\left(1-\frac{1}{N}\right)^N \to e^{-1} \approx 0.368 \qquad \text{as } N \to \infty.

The omission indicators are independent across independently randomized trees, so

K_i\sim\operatorname{Binomial}(B,q_N), \qquad \mathbb{E}[K_i]=Bq_N.

Thus roughly 36.8% of trees leave any given observation out in a large sample. With B=500 trees, the expected number entering the OOB prediction is approximately

500 \times e^{-1} \approx 500 \times 0.368 \approx 184.

Part 2: The OOB prediction averages fewer trees

Conditional on K_i=k>0, independence of the eligible randomized-tree draws gives

\operatorname{Var}_{\Theta}(\text{$k$-tree average}\mid K_i=k) =\frac{v_i}{k}, \qquad \operatorname{Var}_{\Theta}(\text{same-distribution $B$-tree average}) =\frac{v_i}{B}.

The second quantity is a hypothetical benchmark made from the same exclusion-conditional tree distribution, not the usual full forest.

Part 3: Approximation and finite-sample caveats

Replacing the random count K_i by its expectation Bq_N gives the approximation

\frac{v_i/(Bq_N)}{v_i/B} =\frac{1}{q_N} \longrightarrow e\approx 2.72.

For v_i=9 and B=500, the large-sample approximation gives

\frac{9}{184}\approx 0.0489 \qquad\text{and}\qquad \frac{9}{500}=0.0180.

The approximate OOB Monte Carlo variance is therefore about 2.72 times the same-distribution B-tree benchmark.

The factor e is therefore a useful large-N, large-B approximation, not an exact identity. Even after conditioning on an OOB prediction being available, the exact calculation involves

\mathbb{E}\!\left[\frac{1}{K_i}\,\middle|\,K_i>0\right],

which is not 1/\mathbb{E}[K_i], and K_i=0 occurs with probability (1-q_N)^B. The usual full forest includes trees trained on observation i. Inclusion can change both the conditional mean and variance of a tree prediction, so v_i/B need not describe its Monte Carlo variance. The calculation is best read as a controlled comparison within the OOB-eligible tree distribution: using only K_i eligible draws creates more simulation noise than using B draws from that same distribution.

Part 4: Why OOB works under i.i.d. sampling

For observation i, the OOB prediction averages only trees that were not trained on i. That makes the prediction approximately out-of-sample for that observation. Repeating this over all observations yields a validation-style error estimate without creating a separate holdout set.

Part 5: Why OOB can fail in macroeconomic forecasting

In a macroeconomic time series, standard bootstrap resampling does not preserve temporal ordering, dependence, publication delays, or real-time data vintages. A tree used in the OOB prediction for date t may still have been trained on observations from dates after t, which would not have been available in real time. This contaminates the forecast-origin information set, so standard OOB error does not estimate the intended real-time risk.

To see why the distortion can have either sign, suppose a structural break occurs halfway through the sample. For a pre-break date, OOB trees trained partly on post-break observations may predict worse than a valid real-time forest trained on the homogeneous pre-break history, making the OOB assessment pessimistic. For an early post-break date, OOB trees may use later observations from the new regime that were unavailable at the forecast origin, often making the assessment optimistic.

A more appropriate design uses validation based on rolling or expanding windows, where each training set contains only information available up to the forecast origin and the validation observation lies strictly in the future.

Part 6: Correlated predictors and importance

When the predictors contain nearly the same signal, either can substitute for the other after one column is permuted. Their individual permutation importances can therefore be small even though the pair is strongly predictive. One diagnostic is to permute the two columns jointly in the validation sample and recompute the loss. The resulting deterioration measures how much the fitted forest relies on breaking their joint contribution, although it remains a predictive perturbation measure rather than a causal effect.

11.11 References

Breiman, Leo. 2001. “Random Forests.” Machine Learning 45 (1): 5–32. https://doi.org/10.1023/A:1010933404324.

Bühlmann, Peter, and Bin Yu. 2002. “Analyzing Bagging.” The Annals of Statistics 30 (4): 927–61. https://doi.org/10.1214/aos/1031689014.

Liaw, Andy, and Matthew Wiener. 2002. “Classification and Regression by randomForest.” R News 2 (3): 18–22.

McCracken, Michael W., and Serena Ng. 2020. “FRED-QD: A Quarterly Database for Macroeconomic Research.” Working Paper 26872. National Bureau of Economic Research. https://doi.org/10.3386/w26872.

Meinshausen, Nicolai. 2006. “Quantile Regression Forests.” Journal of Machine Learning Research 7: 983–99.

Roberts, David R., Volker Bahn, Simone Ciuti, Mark S. Boyce, Jane Elith, Gurutzeta Guillera-Arroita, Severin Hauenstein, et al. 2017. “Cross-Validation Strategies for Data with Temporal, Spatial, Hierarchical, or Phylogenetic Structure.” Ecography 40 (8): 913–29. https://doi.org/10.1111/ecog.02881.

Strobl, Carolin, Anne-Laure Boulesteix, Achim Zeileis, and Torsten Hothorn. 2007. “Bias in Random Forest Variable Importance Measures: Illustrations, Sources and a Solution.” BMC Bioinformatics 8: 25. https://doi.org/10.1186/1471-2105-8-25.

Footnotes

This is a pedagogical final-vintage FRED-QD example. A full real-time GDP forecasting evaluation would also need to account for release calendars and data revisions.↩︎