Random forests address the main weakness of single decision trees: high variance. A single tree can change sharply when the sample changes slightly, especially when early splits are unstable. Random forests reduce this instability by averaging many trees grown on perturbed versions of the data and by injecting additional randomness into the split search. The original random-forest paper is Breiman (2001).
For econometricians, random forests are attractive when prediction is the goal and the conditional mean or event probability depends on nonlinearities and interactions of unknown form. They work especially well on tabular data such as firm-level balance-sheet variables, household characteristics, or macro-financial predictors, and published evidence covers each of these settings:
Asset pricing.Gu, Kelly, and Xiu (2020) predict monthly individual stock returns from a large set of firm characteristics and macroeconomic predictors. Tree ensembles, including random forests, and neural networks are the best-performing methods, and the authors trace their gains to nonlinear predictor interactions that linear models miss.
Macroeconomic forecasting.Medeiros et al. (2021) forecast US inflation from a large macroeconomic dataset. The random forest delivers the smallest forecast errors at most forecast horizons and in most subsamples, ahead of linear models using the same predictors, and the authors attribute this to both its implicit variable selection and nonlinearities.
Household credit.Fuster et al. (2022) predict mortgage default. A random forest is more accurate out of sample than logit models, although the gains are unevenly distributed across borrower groups.
At the same time, they remain nonparametric prediction tools rather than structural models: they do not extrapolate well, and their split structure should not be read causally.
11.2 Roadmap
We begin with bagging and show why averaging unstable trees can reduce variance substantially.
We then explain random feature selection, which turns bagging into a random forest by limiting the candidate predictors at each split.
Next we interpret forests as weighted averages of observed outcomes and study out-of-bag evaluation, a built-in form of cross-validation that uses observations omitted from each tree’s bootstrap sample.
We then discuss the main hyperparameters and the bias-variance trade-offs they control.
Finally, we summarize where random forests fit well in econometric work and where caution is needed.
11.3 From Bagging to Random Forests
Let (x_i,y_i), i=1,\ldots,N, denote the training observations, with p predictors in x_i. Suppose we have a predictor \hat T(x) produced by a deep regression tree. If we grow many such trees on slightly different datasets and average them, we obtain the bagged predictor
where \hat T_b(x) is the prediction of tree b and B is the number of trees. Because many trees make a forest, this is how the nomenclature of “random forests” arose: a forest is a collection of trees, and bagging is the averaging procedure that stabilizes them.
Bagging stands for bootstrap aggregating:
draw a bootstrap sample from the training data (with replacement, of the same size N as the training sample)
grow a tree on that sample
repeat many times
average the resulting predictions in regression, or average class probabilities or use majority voting in classification
A related variant, often called subagging(Bühlmann and Yu 2002), instead draws subsamples of size s<N, usually without replacement (we write s rather than L, which denotes a terminal leaf below). Both approaches create many perturbed training sets; the random forest then applies equal weights 1/B to the resulting tree predictions.
The idea works best when the individual model being averaged is unstable. Deep trees have exactly this property: small sample changes can alter early splits and therefore the whole fitted function. This high variance is a weakness for a single tree but an opportunity for averaging.
Figure 11.1 makes this visible in a one-predictor example.
Show the code
import numpy as npimport matplotlib.pyplot as pltfrom sklearn.ensemble import RandomForestRegressorrng = np.random.default_rng(123)x = np.sort(rng.uniform(-3, 3, 180))y = np.sin(1.4* x) +0.25* x + rng.normal(scale=0.35, size=x.shape[0])# With a single predictor, a random forest reduces to bagging.forest = RandomForestRegressor( n_estimators=300, min_samples_leaf=3, max_features=1, random_state=123,)forest.fit(x.reshape(-1, 1), y)x_grid = np.linspace(-3.1, 3.1, 500)# Predictions of the first three bootstrap trees and of the full averagetree_hats = [tree.predict(x_grid.reshape(-1, 1)) for tree in forest.estimators_[:3]]forest_hat = forest.predict(x_grid.reshape(-1, 1))fig, axes = plt.subplots(1, 2, figsize=(12, 4.8), sharey=True)axes[0].scatter(x, y, s=16, color="0.35", alpha=0.7)for tree_hat, color inzip(tree_hats, ["C3", "C1", "C2"]): axes[0].plot(x_grid, tree_hat, color=color, linewidth=1.6)axes[0].set_title("Three Bootstrap Trees")axes[0].set_xlabel("Predictor")axes[0].set_ylabel("Outcome")axes[0].grid(True, alpha=0.3)axes[1].scatter(x, y, s=16, color="0.35", alpha=0.7)axes[1].plot(x_grid, forest_hat, color="C0", linewidth=2.3)axes[1].set_title("Bagged Trees")axes[1].set_xlabel("Predictor")axes[1].grid(True, alpha=0.3)plt.tight_layout()plt.show()
Figure 11.1: Simulated outcomes drawn from Y=\sin(1.4X)+0.25X+\varepsilon (gray points) with fits from deep regression trees. Left panel: three individual trees, each grown on a different bootstrap sample (red, orange, and green lines). Right panel: the average of 300 such trees (blue line). All trees draw their bootstrap samples from the same 180 training observations.
Each bootstrap tree reacts strongly to local sample noise, and the three trees disagree wherever their bootstrap samples differ. This spread at a given x is variation of tree predictions for a fixed training sample, which the callout below formalizes as v(D,x). The bagged fit remains nonlinear, but averaging across many trees stabilizes it. With a single predictor, the random feature selection introduced below has nothing to choose among, so the figure isolates the averaging effect of bagging.
Variance of an Averaged Forest
There are two ways a forest prediction can vary: we can draw a different training sample, or grow new randomized trees using the same sample. Holding the training sample D and prediction point x fixed, independently randomizing each tree under the same fitting rule produces independent and identically distributed predictions over the algorithmic randomness. This conditional independence does not imply independence across repeated draws of the training sample.
The mean m(D,x) is the prediction obtained by averaging infinitely many trees grown from D; the variance v(D,x) measures how much individual tree predictions vary while D stays fixed.
An average of B such trees has conditional mean m(D,x) and conditional variance v(D,x)/B. The law of total variance therefore gives
\begin{aligned}
\operatorname{Var}[\hat f_B(x)]
&=
\operatorname{Var}_D\!\left(\mathbb{E}[\hat f_B(x)\mid D]\right)
+
\mathbb{E}_D\!\left[\operatorname{Var}(\hat f_B(x)\mid D)\right] \\
&=
\underbrace{\operatorname{Var}_D(m(D,x))}_{\text{variation across training samples}}
+
\underbrace{\frac{1}{B}\mathbb{E}_D[v(D,x)]}_{\text{variation from finitely many trees}}.
\end{aligned}
The second component, often called Monte Carlo variance, shrinks at rate 1/B. The first remains: even an infinite forest can give different predictions when trained on a different sample. Growing more trees therefore stabilizes the prediction for the observed dataset without eliminating its sensitivity to which dataset was observed. Exercise 11.1 connects this decomposition to the correlation between trees.
Why Bagging Alone Is Not Enough
If some predictors are very strong, many bootstrap trees keep splitting on the same variables near the root and can produce very similar fitted rules. At a fixed x, tightly concentrated tree predictions correspond to a small conditional variance v(D,x), so averaging then changes the prediction little relative to using a single randomized tree: \hat f_B(x) is already close to m(D,x). Any instability of m(D,x) across training samples remains because averaging does not remove the training-sample component \operatorname{Var}_D(m(D,x)).
Random forests address this directly. At each split, instead of considering all p predictors, the algorithm draws a random subset of size m and searches only within that subset. The best split among those m predictors is used. A fresh subset is drawn at each split, not just once per tree.
Random feature selection, also called feature subsampling, does two things at once:
it diversifies the fitted rules by forcing trees to try different split variables
it may slightly weaken each individual tree because the globally best predictor is not always available
With feature subsets also drawn independently for each tree, the variance decomposition still applies: conditional on D, the algorithmic randomness now includes both bootstrap sampling and feature selection. The additional randomization changes both parts of the decomposition. It can reduce \operatorname{Var}_D(m(D,x)) by making the conditional mean rule less sensitive to dominant predictors, while increasing \mathbb{E}_D[v(D,x)] because tree predictions are more dispersed conditional on the data; neither sign is guaranteed. The second effect enters the forest variance only through the Monte Carlo term \mathbb{E}_D[v(D,x)]/B, so it is harmless once B is large—its practical meaning is that a strongly randomized forest needs more trees. Random feature selection also changes the rule m(D,x) itself, and with it the bias of the forest. Because \mathbb{E}[\hat f_B(x)]=\mathbb{E}_D[m(D,x)] for every B, adding trees leaves that bias unchanged. The actual performance of the forest depends on the relative magnitudes of these effects and must be assessed empirically.
The bias-variance trade-off is therefore explicit. Deep individual trees have low bias but high variance. Averaging many randomized trees lowers variance, while random feature selection and bootstrap perturbations can slightly increase bias because each tree is deliberately denied the strongest split it could use. Forests work well when the regression function contains nonlinearities and interactions that trees can capture and when variance reduction dominates this modest bias increase. They can underperform a well-specified simpler model when the true signal is close to linear, when extrapolation is needed, or when the sample is too small for flexible trees to be estimated reliably.
11.4 The Random-Forest Predictor
The variance argument above applies to numerical tree predictions. We now separate the two output types used in practice. For regression, the random-forest predictor is
For classification, each tree can produce either a class vote or an estimated leaf probability. In econometric applications, the probability forecast is usually more informative than the hard class label. A forest therefore often predicts
where \hat p_b(x) is the event probability from tree b.
Random forests are commonly applied to:
default prediction
recession probability forecasting
treatment assignment risk scoring
nonlinear forecasting of inflation, output, or firm sales
11.5 Forest Weights as Local Averages
A random forest can also be viewed as a data-adaptive local averaging estimator. For making this point, we consider squared-error regression trees, whose leaf predictions are sample means. First consider trees grown by subsampling without replacement, and let S_b denote the subsample used to grow tree b. Let L_b(x) denote the terminal leaf containing the prediction point x in tree b, and let N_b(x) be the number of sampled observations in that leaf. The regression prediction can be written as
With bootstrap sampling, observation i can appear in tree b’s in-bag sample with multiplicity m_{ib}\in\{0,1,2,\ldots\}. Because the stored leaf mean counts duplicate draws, the exact weight is
where N_b(x) now counts bootstrap draws in the leaf. In either sampling design, the weights are nonnegative and sum to one.
The weight w_i(x) is large when observation i often lands in the same terminal leaf as the prediction point, especially when that leaf contains few observations. For example, suppose three observations have outcomes 2, 4, and 8. If the prediction point shares a leaf with the first two in one tree and with the last two in another, the two leaf means are 3 and 6. The forest prediction is (3+6)/2=4.5, equivalently \tfrac14(2)+\tfrac12(4)+\tfrac14(8): the middle observation receives weight from both trees.
The construction resembles kernel regression, which weights observations by their distance from the prediction point (the kernel smoothing primer introduces it), but the forest learns its neighborhoods through tree partitions instead of prescribing them with a distance formula. The forest weights generally depend on the training outcomes as well as the predictors, because outcomes determine which splits are selected.
This weighting view is also the bridge to more advanced forests. Quantile Regression Forests keep more than the leaf mean and use forest neighborhood weights to estimate a conditional distribution (Meinshausen 2006). Local Linear Forests go in a different direction: they retain the forest neighborhood but fit a local linear model rather than a constant inside it (Friedberg et al. 2021). Both are developed in Chapter 13, in the quantile regression forest section and the local linear forest section respectively.
11.6 Bootstrap Sampling and Out-of-Bag Evaluation
Each tree is grown on a bootstrap sample of size N drawn with replacement from the original sample. Because sampling is with replacement, some observations appear multiple times in a given tree’s training set and some observations are omitted.
For a particular observation i, the probability of being omitted from a bootstrap sample is
where \mathcal{B}_i^{\text{OOB}} = \{\, b : i \notin \text{the bootstrap sample used to grow tree } b \,\} is the set of trees whose bootstrap sample excludes observation i, and |\mathcal{B}_i^{\text{OOB}}| is its cardinality.
The prediction is defined only when |\mathcal{B}_i^{\text{OOB}}|>0. With few trees, some observations may have no OOB prediction; grow more trees and check coverage before computing the all-observation OOB error below.
The OOB construction produces an approximately out-of-sample prediction for observation i, because the trees contributing to that prediction were not trained on i. (We avoid the word honest here: in the forest literature it has the specific technical meaning of separating the observations used to choose splits from those used to compute leaf values, a stronger condition that OOB averaging does not deliver.) The OOB error is the average loss of these OOB predictions over all observations; for regression it is
OOB error is a form of cross-validation that bagging supplies without refitting. Like the pooled K-fold estimate \widehat{\mathrm{CV}}_K in the cross-validation chapter, it scores each observation with a predictor fitted without that observation and averages the held-out losses over all N observations. The closest analogue is leave-one-out cross-validation, the case K=N. Conditional on omitting observation i, a bootstrap sample of size N consists of N draws from the remaining N-1 observations. Every tree in \mathcal{B}_i^{\text{OOB}} is therefore grown by the forest’s usual rule on the leave-one-out sample (with N rather than N-1 draws), and \hat f^{\text{OOB}}(x_i) is a forest fitted without observation i(Hastie, Tibshirani, and Friedman 2009, sec. 15.3.1). It differs from leave-one-out cross-validation of the full forest mainly in size: it averages about e^{-1}B trees rather than B, which adds Monte Carlo variance that vanishes as B grows. Exercise 11.2 quantifies this effect.
Under independent and identically distributed (i.i.d.) sampling, OOB error is therefore a useful internal estimate of prediction error for new observations drawn from the same distribution (Breiman 2001). The caveats of cross-validation carry over. This interpretation requires that observation i’s outcome has not entered its OOB predictors through preprocessing or feature selection, which is the leakage principle of cross-validation. If OOB error is used to choose hyperparameters, the selected minimum is a tuning result; use a separate test sample for final evaluation. With temporal, spatial, or clustered dependence, row-wise OOB validation shares the weakness of shuffled K-fold cross-validation discussed for time series: it need not reproduce the dependence structure or information set relevant for forecasting. OOB error can therefore be misleading, and rolling, blocked, spatial, or cluster-level validation is usually more appropriate (Roberts et al. 2017).
OOB Inherits the Sampling Design
Out-of-bag error is convenient because it comes “for free” during estimation, but it inherits the sampling design built into the forest. If the data are serially dependent, clustered, revised over time, or otherwise non-i.i.d., standard OOB error can be misleading.
Question for Reflection
For a panel dataset of firms observed over time, which prediction target is standard OOB error closest to: new firms, new time periods, or new firm-time cells? What validation split would better match each target?
Suggested Answer
Standard OOB error is closest to validating randomly held-out firm-time rows, so it is most defensible for a target resembling new cells drawn from the same panel distribution. It is not a good proxy for predicting entirely new firms or future time periods if there is firm dependence or temporal dependence. New firms call for leave-firm-out validation, future periods call for forward splits that hold out future time blocks, and combined targets may require holding out both firms and time blocks.
Figure 11.2 compares OOB error with the error on an independent test sample in a setting where OOB error is justified, because the observations are i.i.d. by construction. The data are simulated afresh and are unrelated to Figure 11.1. There are N=400 training observations with eight independent standard normal predictors. Only the first four enter the conditional mean, through a smooth nonlinearity, a step, and an interaction; the other four are pure noise. A second sample of 400 observations drawn from the same design serves as the test set. For each number of trees B on the grid, we fit a forest with three candidate predictors per split and a minimum leaf size of 5. We then record two numbers: the OOB mean squared error (MSE) on the training observations and the MSE of the full forest on the test sample. The forests share a random seed, so each larger forest contains the trees of the smaller ones, and the curves trace a single forest as trees are added.
Figure 11.2: Out-of-bag (OOB) mean squared error (MSE) on the 400 training observations (blue) and MSE on 400 independently drawn test observations (orange), plotted against the number of trees B. The simulated data have eight independent standard normal predictors X_1,\ldots,X_8 and outcome Y=1.2\sin(X_1)+0.8\,\mathbf{1}\{X_2>0\}+0.6X_3X_4+\varepsilon, where \varepsilon\sim N(0,0.5^2) is independent of the predictors. Each forest offers three candidate predictors per split and uses a minimum leaf size of 5. The grid begins at 25 trees, when every training observation has at least one OOB prediction in this seeded simulation.
The OOB MSE falls most at the smaller tree counts, whereas the independent-test MSE is comparatively flat. The difference reflects the number of trees behind each prediction: at B=25, an OOB prediction averages about nine trees, whereas each test prediction averages all 25. Both curves stabilize as the finite-forest Monte Carlo component becomes small; further trees then mainly increase computation. The OOB curve lies below the test curve at every B, but this gap is not evidence that OOB error is optimistic. Each curve averages only 400 squared errors, and the test MSE alone has a standard error of about 0.09 here. Rerunning the simulation with other seeds produces gaps of either sign.
11.7 Hyperparameters and Trade-Offs
Although forests are often robust, they still require tuning. The most important hyperparameters are the following.
Number of Trees
The number of trees B mainly controls Monte Carlo error in the forest average.
More trees lower simulation noise and stabilize predictions.
More trees do not usually create overfitting in the same way as adding parameters to ordinary least squares (OLS).
After a point, gains are negligible and only computation rises.
Number of Candidate Features per Split
This hyperparameter is often called max_features or mtry.
Smaller max_features diversifies the fitted trees and can lower training-sample variability. It can also make individual trees noisier conditional on the data, a Monte Carlo cost that averaging removes at rate 1/B and that therefore mainly requires more trees. As discussed above, it can also increase bias: when informative predictors are unavailable at useful splits, trees must substitute weaker splits that may systematically miss part of the signal. Increasing B estimates the average of this randomized fitting rule more precisely; it does not correct bias in that average rule.
Larger max_features makes each tree stronger but more similar to the others.
If one predictor is much stronger than the others, reducing max_features can materially improve the ensemble by forcing other variables into the split search.
Common rules of thumb are to try roughly p/3 candidate features per split for regression and \sqrt{p} for classification. These particular defaults come from the widely used randomForest implementation (Liaw and Wiener 2002); Breiman (2001) himself worked with considerably smaller values in his random-feature experiments, for example \lfloor \log_2 p + 1 \rfloor. They are not econometric laws; they are starting values that should be checked with a validation design appropriate for the data structure.
Leaf Size and Tree Depth
Random forests typically use deep trees, but the leaf size still matters.
Smaller leaves reduce bias but increase the variance of individual trees.
Larger leaves smooth the prediction surface and can improve performance when the signal is weak or noisy.
Monotonicity Constraints in Some Implementations
Monotonicity constraints are not part of the classical random-forest theory developed in this chapter, but some modern implementations allow them. The idea is straightforward: if economic reasoning implies that a predictor should move the fitted outcome only upward or only downward, the ensemble can be restricted to respect that sign pattern. (The same device reappears for gradient boosting in Chapter 12, in the section on regularization in boosting.) This can make a forest more credible when the unrestricted fit shows local reversals that are hard to defend economically.
Mechanically, the constraint is enforced inside each tree by rejecting any candidate split on the constrained feature that would produce children violating the sign pattern. Tree-growing algorithms additionally propagate bounds through the tree so that splits on other features cannot eventually create leaves that break the global monotonicity. Since every tree in the forest respects the constraint, the averaged prediction does too.
In scikit-learn. Since version 1.4, RandomForestRegressor accepts a monotonic_cst argument with one integer per feature: +1 for increasing, -1 for decreasing, and 0 for no constraint:
See the scikit-learn example Monotonic Constraints for a visual comparison of constrained and unconstrained fits.
The same warning applies as elsewhere: a monotonicity restriction is a predictive shape constraint, not a structural identification device. If the sign restriction is wrong or only locally valid, the forest can become systematically misspecified.
11.8 Econometric Interpretation and Limits
Random forests are well suited to prediction with many candidate interactions. They can uncover nonlinear combinations of predictors without the econometrician specifying them manually, which is valuable in cross-sectional or panel-style prediction tasks with rich covariate sets.
But several limits remain.
No Extrapolation
Like single trees, forests average leaf values. That means they remain local averaging estimators. If the prediction point lies outside the historical support of the training data, the forest cannot extrapolate a trend; it returns a weighted average of observed outcomes from nearby leaves.
Figure 11.3 illustrates this limit, together with the stabilizing effect of averaging, using the quarterly Federal Reserve Economic Data collection (FRED-QD)(McCracken and Ng 2020). The target is annualized gross domestic product (GDP) growth in quarter t+1, while the predictors are dated quarter t: GDP growth, inflation, the term spread, the log of the volatility index (VIX), and the unemployment rate.1 The top panel shows test-period predictions from a single unrestricted decision tree; the bottom panel shows predictions from a 500-tree random forest trained on the same predictors. The single-tree forecast jumps between a small set of leaf constants. Averaging 500 step-function predictions produces a less jumpy forest path. The plot cannot by itself demonstrate in-sample overfitting or identify statistical bias and variance, which are repeated-sample concepts. It does show that neither model approaches the annualized GDP-growth rates of roughly -33\% and +30\% observed in 2020; the forest is especially flat at those extremes. A tree ensemble averages leaf constants estimated from historical data and therefore cannot forecast outside the range those constants span.
Show the code
import numpy as npimport pandas as pdimport matplotlib.pyplot as pltfrom sklearn.tree import DecisionTreeRegressorfrom sklearn.ensemble import RandomForestRegressorraw = pd.read_csv("data/fred_qd_current.csv")fred_qd = raw.iloc[2:].copy()fred_qd["sasdate"] = pd.to_datetime(fred_qd["sasdate"])for col in fred_qd.columns:if col !="sasdate": fred_qd[col] = pd.to_numeric(fred_qd[col], errors="coerce")fred_qd = fred_qd.sort_values("sasdate").reset_index(drop=True)gdp_growth =400* np.log(fred_qd["GDPC1"]).diff()inflation =400* np.log(fred_qd["CPIAUCSL"]).diff()term_spread = fred_qd["GS10"] - fred_qd["TB3MS"]log_vix = np.log(fred_qd["VIXCLSx"])macro = pd.DataFrame({# Row t contains predictors from t and the outcome and date from t+1."target_date": fred_qd["sasdate"].shift(-1),"y": gdp_growth.shift(-1),"gdp_lag1": gdp_growth,"infl_lag1": inflation,"spread_lag1": term_spread,"vix_lag1": log_vix,"unrate_lag1": fred_qd["UNRATE"],}).dropna()feature_cols = ["gdp_lag1", "infl_lag1", "spread_lag1", "vix_lag1", "unrate_lag1"]split =int(len(macro) *0.75)X_train = macro.iloc[:split][feature_cols].valuesy_train = macro.iloc[:split]["y"].valuesX_test = macro.iloc[split:][feature_cols].valuesy_test = macro.iloc[split:]["y"].valuestest_dates = macro.iloc[split:]["target_date"]single_tree = DecisionTreeRegressor(random_state=42)forest = RandomForestRegressor(n_estimators=500, max_features="sqrt", random_state=42)single_tree.fit(X_train, y_train)forest.fit(X_train, y_train)fig, axes = plt.subplots(2, 1, figsize=(11, 6), sharex=True)for ax, pred, label, color inzip( axes, [single_tree.predict(X_test), forest.predict(X_test)], ["Single tree (unconstrained depth)", "Random forest (500 trees)"], ["C1", "C0"],): ax.plot(test_dates, y_test, color="0.5", linewidth=1, label="Realized GDP growth") ax.plot(test_dates, pred, color=color, linewidth=1.5, label=label) ax.axhline(0, color="0.8", linewidth=0.7, linestyle="--") ax.set_ylabel("pct. pts. ann.") ax.legend(frameon=False, loc="upper right") ax.grid(True, alpha=0.25)axes[1].set_xlabel("Target quarter")plt.tight_layout()plt.show()
Figure 11.3: Out-of-sample one-step-ahead GDP growth forecasts over the last 25% of the usable FRED-QD sample from a single deep decision tree (top, orange) and a 500-tree random forest (bottom, blue). Both models are fitted once on the first 75% using five predictors dated one quarter before the target. The horizontal axis dates the target quarter; the gray line is realized GDP growth, measured as the annualized log-difference of GDPC1.
Interpretation Is Predictive, Not Structural
The forest is optimized for prediction, not identification, and this limits how its output can be read. Forest software usually reports a variable-importance score for each predictor. Impurity-based importance sums the reduction in the split criterion, such as the residual sum of squares (RSS) or Gini impurity (both defined in the chapter on decision trees), over all splits on that predictor in all trees. Permutation importance shuffles one predictor in an out-of-bag or validation sample and records how much the fitted forest’s loss increases.
Both scores measure how much this fitted forest relies on a predictor, not its causal effect or the information it carries on its own. Impurity-based importance can favor predictors with many possible split points, and correlated predictors can share credit so that each looks unimportant on its own (Strobl et al. 2007). With time series, row-wise shuffling also destroys the predictor’s serial structure, so the loss increase mixes the loss of signal with that distortion.
11.9 Summary
Key Takeaways
Random forests average randomized trees to reduce the variability of a single-tree predictor.
Adding trees reduces finite-forest Monte Carlo variance at rate 1/B while leaving training-sample uncertainty.
Random feature subsets diversify split opportunities, with a bias-variance trade-off that must be validated.
Out-of-bag error amounts to leave-one-out cross-validation of a forest with about e^{-1}B trees, obtained without refitting.
Common Pitfalls
OOB error does not reproduce historical forecasting when its trees use observations from later dates.
Forest averages cannot extrapolate a smooth trend beyond the training support.
Offering nearly all predictors at every split can leave trees highly correlated.
11.10 Exercises
Exercise 11.1: Variance Reduction in a Random Forest
Fix a point x_0 and write T_b=\hat T_b(x_0) for the prediction of tree b. Let D be a random training sample. Conditional on D, suppose the trees are identically distributed and independently randomized, as in the chapter’s variance decomposition. Unless conditioning on D is stated explicitly, all moments in this exercise are unconditional, over repeated draws of both D and the tree randomizations. Suppose
The unconditional correlation \rho arises because all trees depend on the same random training sample, even though their predictions are independent conditional on D.
Identify which component increasing B removes and explain why conditional independence of the trees does not imply that the infinite forest has zero unconditional variance.
How many trees are enough? Write \bar T_\infty=m(D,x_0). For \rho>0, use Parts 1 and 2 to derive
and find the smallest B for which the relative excess variance is at most 5% when \rho=0.2 and when \rho=0.05. Holding \sigma^2 fixed, explain why the lower-\rho case needs more trees to approach its lower variance limit. At \rho=0, state the finite-B variance and explain why an absolute tolerance is needed instead.
Comparing prediction risk. Let a new outcome Y_0 at x_0 have mean f(x_0) and finite variance \tau^2, independently of D and the tree randomizations. Two forest specifications satisfy the assumptions above, with:
Specification
Bias \mu-f(x_0)
Single-tree variance \sigma^2
Correlation \rho
Forest 1
0
9
0.2
Forest 2
0.5
12
0.1
Derive each forest’s mean squared prediction error \mathbb{E}[(Y_0-\bar T_B)^2] as a function of the common tree count B. Find the smallest integer B for which Forest 2 has lower prediction risk. Explain why choosing a feature-subsampling setting solely because it lowers \rho is not justified.
Exam level: suitable as-is. Part 1 combines the variance derivation with a numerical check, Part 2 separates the sources of uncertainty, Part 3 derives a tree-count requirement, and Part 4 compares complete prediction risks.
Hint for Part 1
Expand the variance of the average into own-variance and covariance terms, and count how many of each there are. Use \operatorname{Cov}(T_b,T_{b'})=\rho\sigma^2 for b\neq b'.
Hint for Part 2
For two distinct trees, split their unconditional covariance into the expected conditional covariance and the covariance of their conditional means. Use conditional independence for the first term and recall that both conditional means equal m(D,x_0).
Hint for Part 4
Write Y_0-\bar T_B as (Y_0-f(x_0))+(f(x_0)-\mu)+(\mu-\bar T_B). Identify which cross terms vanish before comparing the two risks.
so \mathbb{E}_D[v(D,x_0)]=(1-\rho)\sigma^2. The forest variance is therefore
\operatorname{Var}(\bar T_B)
=\underbrace{\rho\sigma^2}_{\text{training-sample variance}}
+\underbrace{\frac{(1-\rho)\sigma^2}{B}}_{\text{Monte Carlo variance}}.
Increasing B removes the second term. The limit \rho\sigma^2 remains because m(D,x_0) changes with the training sample. Conditional independence concerns fresh randomizations for a fixed D; it does not make the shared training sample nonrandom.
Part 3: How many trees are enough
Dividing the Part 1 variance by the Part 2 limit \sigma^2\rho,
and \sigma^2 cancels: how close a finite forest sits to its own limit is a question about \rho and B alone, not about how noisy the individual trees are.
Requiring the excess to be at most 5% gives
\frac{1-\rho}{\rho B} \le 0.05
\quad\Longleftrightarrow\quad
B \ge \frac{20(1-\rho)}{\rho}.
For \rho = 0.2 this is B \ge 20(0.8)/0.2 = 80 trees; for \rho = 0.05 it is B \ge 20(0.95)/0.05 = 380.
Holding \sigma^2 fixed, the lower-\rho case needs almost five times as many trees to come within 5% of its own limit. Lowering \rho lowers the variance floor \sigma^2\rho but raises the ratio (1-\rho)/\rho: a larger share of the single-tree variance must be averaged away, and the reference variance is smaller.
The finite-B variance converges to zero. Its ratio to the limiting variance is undefined because the limiting variance is zero, so closeness to the limit must be measured using an absolute tolerance.
Part 4: Comparing complete prediction risks
The new outcome is independent of the fitted forest, and both Y_0-f(x_0) and \bar T_B-\mu have mean zero. Expanding the squared prediction error therefore gives
The smallest integer is B=11. Forest 2 has a lower limiting risk but more Monte Carlo variance at each finite B; for B\leq10, that extra variance outweighs its advantage in limiting risk.
Changing max_features can alter bias, single-tree variance, and correlation together. Here, even though Forest 2 has lower correlation, it performs worse at small tree counts. A correlation comparison alone therefore cannot rank the prediction risks of two fitting rules.
Exercise 11.2: Out-of-Bag Predictions and Dependent Data
Suppose a random forest with B independently randomized trees is trained on a sample of size N\geq2, and each tree uses an independent bootstrap sample of size N drawn with replacement from the original data.
Show that the probability a fixed observation i is not selected in one bootstrap sample is
q_N=\left(1-\frac{1}{N}\right)^N,
and conclude that q_N\to e^{-1} as N \to \infty. If K_i is the number of trees whose bootstrap samples omit observation i, show that K_i\sim\operatorname{Binomial}(B,q_N) and find \mathbb{E}[K_i]. For B=500 and large N, approximately how many trees enter the OOB prediction?
Explain why OOB prediction can serve as an internal validation device under i.i.d. sampling when the fitting settings are fixed and the held-out outcome has not entered preprocessing or feature selection. Would minimizing OOB error over many settings leave the selected minimum suitable as a final performance estimate? Give the reason and an appropriate final evaluation design.
You are forecasting quarterly inflation with a long macroeconomic time series. Explain why standard OOB predictions do not reproduce real-time training information sets. Using a structural break halfway through the sample, give mechanisms by which the resulting error estimate could be pessimistic for pre-break dates and optimistic for early post-break dates. Specify a more appropriate validation design.
Exam level: suitable as-is. Part 1 derives the OOB count, Part 2 separates tuning from assessment, and Part 3 addresses real-time evaluation.
Hint for Part 3
Ask whether a bootstrap resample respects time order, publication lags, and real-time information sets. To determine why the sign is ambiguous, place a structural break halfway through the sample and consider pre-break and early post-break forecast origins separately.
Solution
Part 1: OOB probability
In a single bootstrap draw, the probability of not selecting observation i is
1 - \frac{1}{N}.
Because the bootstrap sample contains N draws with replacement, the probability that i is never selected is
Thus roughly 36.8% of trees leave any given observation out in a large sample. With B=500 trees, the expected number entering the OOB prediction is approximately
For observation i, the OOB prediction averages only trees that were not trained on i. That makes the prediction approximately out-of-sample for that observation. Under i.i.d. sampling, a held-out observation is distributed like a fresh draw from the same population, so the loss of its OOB prediction estimates the prediction error for new observations. Averaging these losses over all observations therefore yields a validation-style error estimate without creating a separate holdout set.
This argument requires the stated exclusion of the held-out outcome from preprocessing and feature selection as well as from tree fitting. If many settings are compared by OOB error, the selected minimum also reflects favorable noise in the tuning estimates. It is no longer an independent assessment of the selected fitting procedure. Use an untouched test sample, or an outer validation loop that repeats all OOB-based selection using only its training observations.
Part 3: Why OOB can fail in macroeconomic forecasting
In a macroeconomic time series, a tree used in the OOB prediction for date t may have been trained on observations from dates after t, which would not have been available in real time. Omitting row t does not exclude these future training observations or correct publication delays and data revisions. Standard OOB predictions therefore do not reproduce historical forecasting information sets, and their error need not estimate the intended real-time risk. A timing mismatch alone does not establish the magnitude or sign of any statistical bias.
To see why the distortion can have either sign, suppose a structural break occurs halfway through the sample. For a pre-break date, OOB trees trained partly on post-break observations may predict worse than a valid real-time forest trained on the homogeneous pre-break history, making the OOB assessment pessimistic. For an early post-break date, OOB trees may use later observations from the new regime that were unavailable at the forecast origin, often making the assessment optimistic.
A more appropriate design uses validation based on rolling or expanding windows, where each training set contains only information available up to the forecast origin and the validation observation lies strictly in the future.
Friedberg, Rina, Julie Tibshirani, Susan Athey, and Stefan Wager. 2021. “Local Linear Forests.”Journal of Computational and Graphical Statistics 30 (2): 503–17. https://doi.org/10.1080/10618600.2020.1831930.
Fuster, Andreas, Paul Goldsmith-Pinkham, Tarun Ramadorai, and Ansgar Walther. 2022. “Predictably Unequal? The Effects of Machine Learning on Credit Markets.”Journal of Finance 77 (1): 5–47. https://doi.org/10.1111/jofi.13090.
Gu, Shihao, Bryan Kelly, and Dacheng Xiu. 2020. “Empirical asset pricing via machine learning.”Review of Financial Studies 33 (5): 2223–73. https://doi.org/10.1093/rfs/hhaa009.
Hastie, Trevor, Robert Tibshirani, and Jerome Friedman. 2009. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. 2nd ed. New York: Springer. https://doi.org/10.1007/978-0-387-84858-7.
Liaw, Andy, and Matthew Wiener. 2002. “Classification and Regression by randomForest.”R News 2 (3): 18–22.
McCracken, Michael W., and Serena Ng. 2020. “FRED-QD: A Quarterly Database for Macroeconomic Research.” Working Paper 26872. National Bureau of Economic Research. https://doi.org/10.3386/w26872.
Medeiros, Marcelo C., Gabriel F. R. Vasconcelos, Álvaro Veiga, and Eduardo Zilberman. 2021. “Forecasting Inflation in a Data-Rich Environment: The Benefits of Machine Learning Methods.”Journal of Business & Economic Statistics 39 (1): 98–119. https://doi.org/10.1080/07350015.2019.1637745.
Meinshausen, Nicolai. 2006. “Quantile Regression Forests.”Journal of Machine Learning Research 7: 983–99.
Roberts, David R., Volker Bahn, Simone Ciuti, Mark S. Boyce, Jane Elith, Gurutzeta Guillera-Arroita, Severin Hauenstein, et al. 2017. “Cross-Validation Strategies for Data with Temporal, Spatial, Hierarchical, or Phylogenetic Structure.”Ecography 40 (8): 913–29. https://doi.org/10.1111/ecog.02881.
Strobl, Carolin, Anne-Laure Boulesteix, Achim Zeileis, and Torsten Hothorn. 2007. “Bias in Random Forest Variable Importance Measures: Illustrations, Sources and a Solution.”BMC Bioinformatics 8: 25. https://doi.org/10.1186/1471-2105-8-25.
Footnotes
This is a pedagogical final-vintage FRED-QD example. A full real-time GDP forecasting evaluation would also need to account for release calendars and data revisions.↩︎
