Under active development. A stable version is expected in November 2026. Feedback is welcome by email or via GitHub issues.

15 Conformal Prediction

15.1 Overview

Earlier chapters studied predictive uncertainty by modeling or evaluating conditional distributions, for example through likelihood-based neural networks, quantile methods, distributional forests, and proper scoring rules. Conformal prediction takes a different route. It starts from any predictive model and computes a nonconformity score, a number that is larger when an observation fits the model poorly, on held-out or out-of-fold data. It then uses those scores to construct prediction sets with a finite-sample marginal coverage guarantee under exchangeability (Vovk, Gammerman, and Shafer 2005; Lei et al. 2018).

For econometricians, this is useful because point forecasts alone are often not enough. A central bank may want an inflation fan chart, a risk manager may want a forecast interval for portfolio returns, and a macro forecaster may want uncertainty bands around gross domestic product (GDP) growth projections. In all of those settings, the question is not only “What is the predicted mean?” but also “How wide should a well-calibrated prediction band be?”

Conformal prediction is attractive because it is:

• model-agnostic: the underlying predictor can be ordinary least squares (OLS), a forest, boosting, or a neural network
• distribution-free under exchangeability: no Gaussian residual assumption is required
• finite-sample: the main guarantee is not asymptotic

The guarantee has limits. It is a marginal guarantee, not a full conditional one, and it depends on how calibration data are chosen. That makes conformal prediction especially relevant for econometric work with heteroskedasticity, regime change, and time dependence.

15.2 Roadmap

1. We begin by motivating conformal prediction as a calibration layer on top of a fitted model rather than a full distribution model.
2. We then develop split conformal prediction in its general score-based form and specialize it to regression intervals.
3. Next we define exchangeability, state the finite-sample coverage result, and sketch the rank argument behind it.
4. We then study what basic split conformal does and does not guarantee, including the distinction between marginal and conditional coverage.
5. After that we cover adaptive scores, which let interval width vary with predictors, and conformalized quantile regression (CQR), which calibrates an initially asymmetric quantile band.
6. We apply split conformal prediction to quarterly U.S. gross domestic product growth and interpret overall and high-volatility coverage diagnostics.
7. We then study data-efficient variants: jackknife+ uses leave-one-out predictions, while cross-validation+ (CV+) replaces them with omitted-fold predictions. We then turn to conformal prediction for time series.
8. We close with a practical workflow for econometric applications, a summary, and exercises.

15.3 Why Conformal Prediction

The central issue is visible in Figure 15.1. All three panels use the same conditional mean function, so a point-forecast chapter would treat them identically. But the uncertainty structure is different in each case: one panel is roughly homoskedastic, one is heteroskedastic, and one has asymmetric or multimodal-looking dispersion. A good interval method should react to those differences rather than merely attach one global standard-error-style band to all cases.

import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(15)
x = np.linspace(0.0, 1.0, 220)
m = 0.6 * np.sin(2 * np.pi * x) + 0.3 * x

e1 = rng.normal(0.0, 0.18, size=x.size)
e2 = rng.normal(0.0, 0.07 + 0.28 * x, size=x.size)
mix = rng.uniform(size=x.size) < 0.75
e3 = np.where(mix, rng.normal(-0.08, 0.11, size=x.size), rng.normal(0.24, 0.18, size=x.size))

y1 = m + e1
y2 = m + e2
y3 = m + e3

fig, axes = plt.subplots(1, 3, figsize=(11.6, 3.9), sharey=True)
titles = ["Constant Width", "Heteroskedastic", "Asymmetric / Mixture"]
ys = [y1, y2, y3]

for ax, y, title in zip(axes, ys, titles):
    ax.scatter(x, y, s=11, alpha=0.45, color="C0")
    ax.plot(x, m, color="black", linewidth=2)
    ax.set_title(title)
    ax.set_xlabel("Predictor")
    ax.grid(True, alpha=0.2)

axes[0].set_ylabel("Outcome")
fig.suptitle("Same Mean Prediction, Different Uncertainty", y=1.03, fontsize=13)
plt.tight_layout()
plt.show()

Figure 15.1: Three data-generating situations with the same conditional mean but different uncertainty. In all panels, the black curve is the shared conditional mean function. The left panel has roughly constant Gaussian noise, the middle panel has variance increasing with the predictor, and the right panel uses an asymmetric mixture disturbance. The figure illustrates why point predictions alone do not determine appropriate prediction intervals.

This chapter therefore fits naturally after Evaluating Predictive Distributions, which develops tools for assessing distribution forecasts, and the chapters on Distribution Modeling with Neural Networks and Advanced Tree-Based Methods, which construct conditional-distribution models. Conformal prediction instead calibrates a prediction set around a fitted model.

Formally, the object of interest is a prediction set C(x) such that for a new observation (X_{N+1},Y_{N+1}),

\mathbb{P}\{Y_{N+1} \in C(X_{N+1})\} \ge 1-\alpha.

Here \alpha\in(0,1) is the target miscoverage level, so 1-\alpha is the nominal coverage. The index N+1 in this overview is informal: it just denotes a generic future point, and the precise index is fixed in the split-conformal theorem below, where the calibration sample has size N_{\text{cal}} and the future point is indexed N_{\text{cal}}+1. We keep the conformal literature’s convention of writing the prediction set as an unhatted C, even though—like the quantile \hat q inside it—the set is constructed from the data.

The probability is taken over the joint randomness of the sample, the calibration split, and the future observation. The notation looks simple, but the requirement is strong: a finite-sample coverage guarantee without imposing a full parametric error model.

Prediction Distribution vs Conformal Calibration

Conformal prediction answers a different question from likelihood-based density models.

1. A distributional model tries to learn the full law of Y \mid X=x.
2. Conformal prediction calibrates a prediction set so that it has the right marginal coverage frequency.
3. The first approach can be more informative when correctly specified, but it is sensitive to distributional misspecification.
4. The second approach is more robust as a coverage device, but it usually provides less structural information than a full predictive density.

15.4 Split Conformal Prediction

The most widely used variant is split conformal prediction, also called inductive conformal prediction because a model is first induced from a proper training set and then kept fixed while a separate calibration set is scored. By contrast, full or transductive conformal prediction avoids a permanent training/calibration split by reconsidering the fitted procedure after augmenting the sample with each candidate test value. That can use the data more efficiently but is much more computationally demanding. This chapter develops the inductive construction first and later introduces CV+ and jackknife+ as other ways to recover data efficiency. The data are partitioned into:

• a training set used to fit the model or models
• a calibration set used only to measure nonconformity
• a test or future point for which we want a prediction set

The flow of information is summarized in Figure 15.2. The design principle is that the calibration residuals must be computed from predictions generated by a model that did not use the calibration outcomes for fitting.

import matplotlib.pyplot as plt
from matplotlib.patches import Rectangle

fig, ax = plt.subplots(figsize=(10.8, 3.1))
ax.set_xlim(0, 100)
ax.set_ylim(0, 18)
ax.axis("off")

blocks = [
    (6, 4.8, 26, 7.0, "#9ecae1", "Training sample\nfit model"),
    (39, 4.8, 22, 7.0, "#fdd49e", "Calibration sample\ncompute scores and quantile"),
    (70, 4.8, 18, 7.0, "#c7e9c0", "New point\nform interval"),
]

for x0, y0, w, h, color, text in blocks:
    rect = Rectangle((x0, y0), w, h, facecolor=color, edgecolor="0.25", linewidth=1.2)
    ax.add_patch(rect)
    ax.text(x0 + w / 2, y0 + h / 2, text, ha="center", va="center", fontsize=11)

ax.annotate("", xy=(39, 8.3), xytext=(32, 8.3), arrowprops={"arrowstyle": "->", "linewidth": 1.5})
ax.annotate("", xy=(70, 8.3), xytext=(61, 8.3), arrowprops={"arrowstyle": "->", "linewidth": 1.5})
ax.text(35.5, 10.4, "predict on calibration set", fontsize=10, ha="center")
ax.text(65.2, 10.4, "use calibration quantile", fontsize=10, ha="center")
ax.text(50, 15.7, "Split Conformal Prediction", fontsize=13, ha="center")

plt.tight_layout()
plt.show()

Figure 15.2: Split conformal workflow for regression. The left block is the training sample used to fit the base predictor or quantile models. The middle block is the calibration sample used only to compute nonconformity scores and the calibration quantile. The right block is the new test point, where the fitted model and calibration quantile are combined to form the final prediction interval. The direction of the arrows indicates that model fitting and calibration are separated.

The mathematically clean way to write split conformal is through a score function. Let D_{\text{train}} denote the training sample and let D_{\text{cal}}=\{(X_i,Y_i)\}_{i=1}^{N_{\text{cal}}} denote the calibration sample. Fit any model or collection of models using only D_{\text{train}}, and define a nonconformity score

S(x,y;D_{\text{train}}),

where larger values mean that the pair (x,y) looks less plausible under the fitted model. On the calibration sample, compute scores

R_i = S(X_i,Y_i;D_{\text{train}}), \qquad i=1,\dots,N_{\text{cal}}.

Sort these into ascending order statistics

R_{(1)} \le \cdots \le R_{(N_{\text{cal}})},

so R_{(k)} is the k-th smallest score, with ties broken arbitrarily (the effect of ties on coverage is discussed with the randomized-tie device below). Choose

k = \left\lceil (1-\alpha)(N_{\text{cal}}+1)\right\rceil, \qquad \hat q = R_{(k)}.

By convention, when \lceil (1-\alpha)(N_{\text{cal}}+1)\rceil > N_{\text{cal}}, no finite order statistic exists, so we set \hat q = +\infty and the prediction set becomes all of \mathbb{R}. This happens exactly when N_{\text{cal}} < (1-\alpha)/\alpha, so at \alpha=0.1 whenever N_{\text{cal}} \le 8. An infinite interval is uninformative but honest: with that few calibration points, no finite band can be guaranteed at the requested level. The implementation below follows this convention rather than silently capping the index at the largest observed score.

Then the conformal prediction set for a new predictor value x is

C(x) = \{y: S(x,y;D_{\text{train}})\le \hat q\}.

This general form matters because the validity theorem does not depend on one special residual definition. It applies to any score that is fixed after training.

Algorithm: Split Conformal Regression

For the standard symmetric regression interval:

1. Split the sample into training data and calibration data.
2. Fit a point predictor \hat \mu(x) on the training data only.
3. On the calibration sample, compute absolute residual scores R_i = |Y_i-\hat \mu(X_i)|.
4. Set \hat q = R_{(k)}, \qquad k=\left\lceil (1-\alpha)(N_{\text{cal}}+1)\right\rceil.
5. For a new point x, report C(x) = [\hat \mu(x)-\hat q,\hat \mu(x)+\hat q].

The width is constant because the same calibration quantile \hat q is used at every x.

Example: Split Conformal by Hand

Suppose the calibration residuals are

\{0.2,0.4,0.7,0.9,1.1\},

and the target miscoverage is \alpha=0.2. Then N_{\text{cal}}=5, so

k=\left\lceil (1-\alpha)(N_{\text{cal}}+1)\right\rceil = \lceil 0.8 \times 6\rceil = 5.

Hence the conformal quantile is

\hat q = 1.1.

If the fitted model predicts \hat \mu(x_{\text{new}})=3.4 at a new point, the split conformal interval is

[3.4-1.1,\;3.4+1.1]=[2.3,\;4.5].

This example is deliberately simple. The whole procedure is:

1. fit the model on one part of the data
2. measure typical forecast error on the calibration part
3. inflate the new point forecast by that calibration error quantile

This construction is the foundation for the adaptive, quantile-based, and jackknife-style conformal methods below.

The symmetric absolute-residual score is the simplest case:

S(x,y;D_{\text{train}})=|y-\hat \mu(x)|.

Substituting this into the general set representation gives

C(x) = \{y: |y-\hat \mu(x)|\le \hat q\} = [\hat \mu(x)-\hat q,\hat \mu(x)+\hat q].

This derivation is short algebraically, but conceptually important: the prediction interval is not produced by a Gaussian approximation. It is the inversion of a calibrated residual test.

15.5 Exchangeability and the Coverage Guarantee

The key assumption is exchangeability.

Definition: Exchangeability

A sequence (Z_1,\dots,Z_N) is exchangeable if its joint distribution is invariant to permutation. For every permutation \pi,

(Z_1,\dots,Z_N) \overset{d}{=} (Z_{\pi(1)},\dots,Z_{\pi(N)}).

Independent and identically distributed (i.i.d.) data are exchangeable, but exchangeability is strictly weaker because it does not require independence. For example, let Z_i=U+\varepsilon_i, where U and the \varepsilon_i are mutually independent Gaussian variables and the \varepsilon_i have the same distribution. The vector (Z_1,\ldots,Z_N) is exchangeable, yet its coordinates are dependent because \operatorname{Cov}(Z_i,Z_j)=\operatorname{Var}(U)>0 for i\ne j.

In conformal prediction, the objects being permuted are usually the pairs Z_i=(X_i,Y_i). Under exchangeability and a clean train/calibration split, the calibration scores and the future score are exchangeable once the score function has been fixed using the training part only.

Theorem: Split Conformal Coverage

Suppose an exchangeable sample has been split into a training part D_{\text{train}}, a calibration part (X_i,Y_i) for i=1,\dots,N_{\text{cal}}, and one future point (X_{N_{\text{cal}}+1},Y_{N_{\text{cal}}+1}). The split indices must be fixed in advance or selected independently of the observation values. Suppose the score function S(\cdot,\cdot;D_{\text{train}}) is fully determined by D_{\text{train}} and then kept fixed. Define the calibration scores

R_i^{\mathrm{cal}}=S(X_i,Y_i;D_{\text{train}}), \qquad i=1,\dots,N_{\text{cal}},

and let R_{(1)}^{\mathrm{cal}}\le\cdots\le R_{(N_{\text{cal}})}^{\mathrm{cal}} be their order statistics. Set

k=\left\lceil (1-\alpha)(N_{\text{cal}}+1)\right\rceil, \qquad \hat q = \begin{cases} R_{(k)}^{\mathrm{cal}}, & k\le N_{\text{cal}},\\ +\infty, & k=N_{\text{cal}}+1. \end{cases}

Define the future score separately as

R_{\mathrm{new}} =S(X_{N_{\text{cal}}+1},Y_{N_{\text{cal}}+1};D_{\text{train}}).

Then the split conformal set

C(x)=\{y:S(x,y;D_{\text{train}})\le \hat q\}

satisfies

\mathbb{P}\{Y_{N_{\text{cal}}+1}\in C(X_{N_{\text{cal}}+1})\}\ge 1-\alpha.

If the calibration and future scores are almost surely distinct, then

\mathbb{P}\{Y_{N_{\text{cal}}+1}\in C(X_{N_{\text{cal}}+1})\} = \frac{k}{N_{\text{cal}}+1},

so the actual coverage lies in the interval

1-\alpha \le \mathbb{P}\{Y_{N_{\text{cal}}+1}\in C(X_{N_{\text{cal}}+1})\} < 1-\alpha+\frac{1}{N_{\text{cal}}+1}.

This is the standard finite-sample split-conformal result (Vovk, Gammerman, and Shafer 2005; Lei et al. 2018).

The proof is a rank argument, but it is worth spelling out because the entire finite-sample result comes from it. Once the training subset has been chosen and the score function has been fitted on that subset, the training data are no longer random objects inside the proof. What remains random are the calibration points and the future point, and those still enter symmetrically. Conditional on the realized training data, the scores

R_1^{\mathrm{cal}},\dots,R_{N_{\text{cal}}}^{\mathrm{cal}},R_{\mathrm{new}}

are exchangeable. If ties are absent, the rank of R_{\mathrm{new}} among these N_{\text{cal}}+1 values is uniformly distributed on \{1,\dots,N_{\text{cal}}+1\}. Writing

\operatorname{rank}(R_{\mathrm{new}}) = 1+\sum_{i=1}^{N_{\text{cal}}}\mathbf{1}\{R_i^{\mathrm{cal}}<R_{\mathrm{new}}\},

coverage occurs when

R_{\mathrm{new}}\le \hat q,

which is equivalent to the event that the future score has rank at most k when k\le N_{\text{cal}}; under the +\infty convention, the event is automatic when k=N_{\text{cal}}+1. Therefore

\mathbb{P}\{Y_{N_{\text{cal}}+1}\in C(X_{N_{\text{cal}}+1})\} = \mathbb{P}\{\operatorname{rank}(R_{\mathrm{new}})\le k\} = \frac{k}{N_{\text{cal}}+1} \ge 1-\alpha.

The ceiling rule explains the conservativeness. Since

k=\left\lceil (1-\alpha)(N_{\text{cal}}+1)\right\rceil,

we have

k \ge (1-\alpha)(N_{\text{cal}}+1),

so

\frac{k}{N_{\text{cal}}+1}\ge 1-\alpha.

At the same time,

k < (1-\alpha)(N_{\text{cal}}+1)+1,

which yields the upper bound

\frac{k}{N_{\text{cal}}+1} < 1-\alpha+\frac{1}{N_{\text{cal}}+1}.

Two procedures, two coverage statements. It is worth separating the unmodified procedure from the randomized one, because they are not the same algorithm.

1. Unmodified split conformal (the procedure used in practice): the prediction set uses the calibration-only threshold \hat q defined above. When ties among scores have positive probability, the resulting marginal coverage is at least 1-\alpha but may strictly exceed it; the exchangeability argument delivers the lower bound but not equality.
2. Randomized split conformal (a theoretical device): attach independent V_i\sim\mathrm{Uniform}(0,1) variables to the scores and order the pairs (R_i,V_i) lexicographically. The auxiliary variable breaks only ties in R_i; it never reverses two unequal scores. For this modified randomized procedure, the future pair has a uniform strict rank on \{1,\dots,N_{\text{cal}}+1\}, and the coverage probability equals exactly \lceil(1-\alpha)(N_{\text{cal}}+1)\rceil/(N_{\text{cal}}+1).

The unmodified procedure in (1) inherits the coverage lower bound from (2) because the event R_{\mathrm{new}}\le\hat q contains the corresponding randomized-rank event. This is the core reason conformal prediction is finite-sample and distribution-free: the proof uses only exchangeability and ranks, not a parametric error law.

One useful mathematical consequence is that validity is separated from efficiency. If the score function is badly chosen, intervals can be unnecessarily wide, but marginal coverage still holds. Better base models and better scores mainly improve interval length rather than validity.

Question for Reflection

Why does the split conformal theorem not require the forecasting model to be correctly specified?

Suggested Answer

Because the proof is based on the rank of the future score among calibration and future scores. That rank argument uses exchangeability, not correctness of the model. A poor model can make intervals wide or uninformative, but it does not by itself destroy the marginal coverage guarantee.

15.6 What Basic Split Conformal Does and Does Not Guarantee

The theorem is strong, but it is important to read it correctly.

Marginal coverage. The guarantee applies on average over the full distribution of (X,Y). If we target 90% coverage, the average long-run inclusion frequency across all future points is at least 90%.

Not conditional coverage. The theorem does not imply

\mathbb{P}\{Y_{N_{\text{cal}}+1}\in C(X_{N_{\text{cal}}+1})\mid X_{N_{\text{cal}}+1}=x\}\ge 1-\alpha \qquad \text{for every } x.

This distinction matters for econometrics because uncertainty often differs sharply across subgroups, regimes, income levels, leverage levels, or business-cycle states. A method can achieve nominal marginal coverage while still undercovering in high-volatility regions and overcovering in quiet regions.

The gap is not an artifact of a clumsy method; it is fundamental. Distribution-free exact conditional coverage is impossible for any procedure producing nontrivial intervals: a method that guaranteed conditional coverage at every x, under every distribution, would be forced to produce intervals of infinite expected length at non-atomic covariate values (Vovk 2012; Lei and Wasserman 2014; Foygel Barber et al. 2021). Better nonconformity scores—the normalized and quantile-based scores introduced later in this chapter—improve approximate conditional behavior, but no choice of score can turn the marginal guarantee into a conditional one.

The marginal–conditional distinction is not merely verbal. For constant-width intervals, the failure of exact conditional coverage under heteroskedasticity can be shown directly.

Proposition: Constant-Width Coverage Cannot Be Conditionally Exact Under Heteroskedasticity

Suppose

Y=\mu(X)+\sigma(X)\varepsilon,

where \varepsilon is independent of X, satisfies \mathbb{E}|\varepsilon|<\infty and \mathbb{E}[\varepsilon]=0, has a density that is strictly positive on all of \mathbb{R}, and \sigma(X)>0 is not almost surely constant. The independence and zero-mean conditions imply \mu(X)=\mathbb{E}[Y\mid X]; symmetry is not required. Consider a constant-width interval centered at this conditional mean,

C_q(x)=[\mu(x)-q,\mu(x)+q], \qquad q>0.

Then

\mathbb{P}\{Y\in C_q(x)\mid X=x\} = \mathbb{P}\left\{|\varepsilon|\le \frac{q}{\sigma(x)}\right\},

so the conditional coverage depends on x whenever \sigma(x) varies with x. Therefore this constant-width interval centered at \mu(x) can match the target only on average; it cannot generally deliver the same exact conditional coverage level for all x.

To verify the display, note that

Y\in C_q(x) \iff |\mu(x)+\sigma(x)\varepsilon-\mu(x)|\le q \iff |\varepsilon|\le \frac{q}{\sigma(x)}.

Because \varepsilon has a strictly positive density on \mathbb{R}, the cumulative distribution function (CDF) of |\varepsilon| is strictly increasing on [0,\infty). Hence \mathbb{P}\{|\varepsilon| \le q/\sigma(x)\} is a strictly decreasing function of \sigma(x), so conditional coverage genuinely varies with x whenever \sigma(x) is not constant. This is the formal reason a constant-width split conformal interval struggles in heteroskedastic settings.

The positive-density condition is doing real work and is not a technicality. If \varepsilon instead had bounded support, say \varepsilon \sim \mathrm{Uniform}(-1,1) with \sigma(x) taking the two values 0.1 and 0.2, then q=1 would give q/\sigma(x) \in \{5,10\}, and \mathbb{P}\{|\varepsilon| \le q/\sigma(x)\} would equal one at both values. Conditional coverage would then be constant in x despite the heteroskedasticity, simply because the band is wide enough to cover the entire conditional support everywhere. The proposition therefore describes the interesting case, in which the band is not trivially wide.

That phenomenon is illustrated in Figure 15.3. The left panel shows a heteroskedastic data-generating process and a constant-width interval centered at the true conditional mean. The right panel then shows that local coverage varies with the predictor even though the average coverage is close to the target. The point is not that conformal fails. The point is that marginal validity is weaker than conditional validity.

import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(16)
x = rng.uniform(0.0, 1.0, 3000)
mu = 0.8 * np.sin(2.4 * np.pi * x)
sigma = 0.12 + 0.45 * x
y = mu + rng.normal(0.0, sigma)

q = np.quantile(np.abs(y - mu), 0.9)
covered = np.abs(y - mu) <= q

grid = np.linspace(0.08, 0.92, 16)
halfwidth = 0.07
local_cov = []
for g in grid:
    mask = np.abs(x - g) <= halfwidth
    local_cov.append(np.mean(covered[mask]))
local_cov = np.array(local_cov)

xline = np.linspace(0, 1, 250)
mline = 0.8 * np.sin(2.4 * np.pi * xline)

fig, axes = plt.subplots(1, 2, figsize=(11.5, 4.1))
axes[0].scatter(x, y, s=8, alpha=0.15, color="C0")
axes[0].plot(xline, mline, color="black", linewidth=2)
axes[0].fill_between(xline, mline - q, mline + q, color="C1", alpha=0.25)
axes[0].set_title("Constant-Width Band")
axes[0].set_xlabel("Predictor")
axes[0].set_ylabel("Outcome")
axes[0].grid(True, alpha=0.2)

axes[1].plot(grid, local_cov, marker="o", color="C3", linewidth=2)
axes[1].axhline(0.9, linestyle="--", color="0.3", linewidth=1.4)
axes[1].set_ylim(0.55, 1.02)
axes[1].set_title("Local Coverage by Predictor Region")
axes[1].set_xlabel("Predictor")
axes[1].set_ylabel("Estimated coverage")
axes[1].grid(True, alpha=0.2)

plt.tight_layout()
plt.show()

Figure 15.3: Marginal versus conditional coverage under heteroskedasticity. In the left panel, the black curve is the conditional mean and the shaded band is a constant-width interval calibrated to achieve about 90% coverage on average. The data are heteroskedastic, with dispersion increasing in the predictor. The right panel plots local coverage estimated in moving bins of the predictor. The dashed horizontal line marks nominal 90% coverage. Average coverage can be near target even when local coverage is too high in low-variance regions and too low in high-variance regions.

The need to assess interval width, sharpness, and subgroup behavior is one reason the Evaluating Predictive Distributions chapter remains relevant. Coverage is not the whole story. Conformal prediction gives a strong calibration guarantee, but it does not exempt us from checking whether the intervals are informative and where they may fail.

The other main limitation is data efficiency. In split conformal, the calibration observations are not used to fit the predictive model. If data are scarce, the base model can become worse and the resulting intervals can widen.

15.7 Adaptive and Asymmetric Conformal Intervals

The simplest split conformal interval has constant width. That is often unrealistic in economic data, where forecast uncertainty changes with leverage, income, business-cycle state, or liquidity conditions.

One way to fix this is to keep the split conformal logic but change the score.

Adaptive symmetric scores. Suppose we train both a point predictor \hat \mu(x) and a positive scale predictor \hat \sigma(x)>0 on the proper training set only. Fitting \hat \sigma on the training data only is essential: it keeps the score S(x,y;D_{\text{train}}) a fixed-after-training function, so the calibration and future scores remain exchangeable. We then define

S(x,y;D_{\text{train}}) = \frac{|y-\hat \mu(x)|}{\hat \sigma(x)}.

On the calibration sample, compute

R_i = \frac{|Y_i-\hat \mu(X_i)|}{\hat \sigma(X_i)},

take the conformal quantile \hat q, and report

C(x) = [\hat \mu(x)-\hat q \hat \sigma(x),\hat \mu(x)+\hat q \hat \sigma(x)].

The resulting interval is still symmetric around the point forecast, but its width now adapts with x through \hat \sigma(x).

The validity theorem applies unchanged: the rank argument requires only that S(x,y) be a measurable function of (x,y) that is fixed after training and that the calibration and future scores are exchangeable. Whether S is an absolute residual, a normalized residual, or a more elaborate score, the same rank argument applies. What changes is interval efficiency.

Conformalized Quantile Regression. Symmetry may still be too restrictive. In downside-risk problems, the lower and upper tails need not move together. Conformalized quantile regression (CQR) starts with two quantile predictors, a lower one \hat q_{\mathrm{low}}(x) and an upper one \hat q_{\mathrm{high}}(x), both fit on the proper training set only so that the resulting score stays a fixed-after-training function and the calibration and future scores remain exchangeable. For target coverage 1-\alpha, the conventional initial choices are the \alpha/2 and 1-\alpha/2 conditional quantiles. Validity does not require those levels to be estimated correctly: any fixed-after-training lower and upper functions can be conformalized by the rank argument. Their quality and chosen levels determine efficiency, however, because a poor initial band may need a large calibration correction. CQR then calibrates the two functions using the score (Romano, Patterson, and Candes 2019)

S(x,y;D_{\text{train}}) = \max\{\hat q_{\mathrm{low}}(x)-y,\; y-\hat q_{\mathrm{high}}(x)\}.

This score is negative when y already lies well inside the initial band and positive when y falls outside it.

Let \hat q_{\mathrm{cal}} be the conformal order statistic of these CQR scores, using the same ceiling rule as in split conformal. The subscript distinguishes this one scalar calibration correction from the two quantile functions. The conformal set is

C(x)=\{y: \max(\hat q_{\mathrm{low}}(x)-y,\; y-\hat q_{\mathrm{high}}(x))\le \hat q_{\mathrm{cal}}\}.

The set can be simplified algebraically:

\max(\hat q_{\mathrm{low}}(x)-y,\; y-\hat q_{\mathrm{high}}(x))\le \hat q_{\mathrm{cal}}

if and only if

\hat q_{\mathrm{low}}(x)-y \le \hat q_{\mathrm{cal}} \qquad \text{and} \qquad y-\hat q_{\mathrm{high}}(x)\le \hat q_{\mathrm{cal}},

which is equivalent to

\hat q_{\mathrm{low}}(x)-\hat q_{\mathrm{cal}} \le y \le \hat q_{\mathrm{high}}(x)+\hat q_{\mathrm{cal}}.

So the final CQR interval is

C(x)= [\hat q_{\mathrm{low}}(x)-\hat q_{\mathrm{cal}},\hat q_{\mathrm{high}}(x)+\hat q_{\mathrm{cal}}].

This interval is conceptually different from the basic absolute-residual split-conformal interval. Its width already varies with x before calibration, because the lower and upper quantile models can respond differently to heteroskedasticity and asymmetry.

The logic is visualized in Figure 15.4. The dashed band is the raw quantile-regression interval. The solid band is the conformalized version after applying the calibration correction \hat q_{\mathrm{cal}} to the lower and upper sides. In the figure the adjustment widens the band because \hat q_{\mathrm{cal}}>0. More generally, the conformal step can either widen or shrink the initial band depending on the calibration scores.

import numpy as np
import matplotlib.pyplot as plt

x = np.linspace(0.0, 1.0, 300)
mu = 0.5 * np.sin(2.3 * np.pi * x) + 0.2 * x
scale = 0.12 + 0.32 * x

q_low = mu - 1.15 * scale - 0.03 * np.cos(4 * np.pi * x)
q_high = mu + 1.05 * scale + 0.02 * np.sin(3 * np.pi * x)
qhat_cal = 0.12

fig, ax = plt.subplots(figsize=(9.2, 4.4))
ax.plot(x, mu, color="black", linewidth=2, label="Conditional center")
ax.plot(x, q_low, color="C1", linestyle="--", linewidth=2, label="Initial lower quantile")
ax.plot(x, q_high, color="C1", linestyle="--", linewidth=2, label="Initial upper quantile")
ax.plot(x, q_low - qhat_cal, color="C3", linewidth=2, label="Conformalized lower bound")
ax.plot(x, q_high + qhat_cal, color="C3", linewidth=2, label="Conformalized upper bound")
ax.fill_between(x, q_low - qhat_cal, q_high + qhat_cal, color="C3", alpha=0.14)
ax.set_xlabel("Predictor")
ax.set_ylabel("Outcome")
ax.set_title("CQR Recalibrates an Adaptive Quantile Band")
ax.grid(True, alpha=0.22)
ax.legend(frameon=False, ncol=2, loc="upper left")
plt.tight_layout()
plt.show()

Figure 15.4: Conformalized quantile regression in a heteroskedastic setting. The dashed curves are the initial lower and upper quantile predictions from a quantile model. The solid curves are the conformalized bounds after subtracting and adding the calibration correction term \hat q_{\mathrm{cal}}. The correction shifts each bound by the same amount at every value of x, so the vertical widening is uniform; the final band remains adaptive only because the initial quantile band already varies in width.

Question for Reflection

Holding the marginal coverage target fixed, when can conformalized quantile regression produce a more efficient or appropriately shaped interval than symmetric absolute-residual split conformal?

Suggested Answer

CQR can be more efficient when uncertainty is clearly heteroskedastic or asymmetric, so a constant-width symmetric band would be implausible. By starting from lower and upper quantile models, CQR can adapt interval width and asymmetry across the predictor space before conformal calibration corrects overall marginal coverage.

15.8 Real Data Example: Quarterly GDP Growth with FRED-QD

The chapter is easier to interpret once we see the methods on real macroeconomic data. We therefore analyze the quarterly U.S. macro panel in data/fred_qd_current.csv, the repository copy of the Federal Reserve Economic Data Quarterly Database (FRED-QD) release underlying these notes (McCracken and Ng 2020). The Data Appendix documents the file and variables. The target is next-quarter annualized real GDP growth, constructed from GDPC1. The predictors are current-quarter macro and financial indicators: GDP growth, consumer price index (CPI) inflation, the unemployment rate, the federal funds rate, the term spread, and the log of the Cboe Volatility Index (VIX), recorded as VIXCLSx.

This is a chronological pseudo-out-of-sample illustration, not a literal application of the split-conformal theorem and not a genuine real-time forecast evaluation. Quarterly macro data are time ordered and are not exchangeable. Moreover, the repository file is a single current-vintage panel: the code respects row dates but reconstructs neither historical revisions nor release delays. Current-quarter macro values are treated as if they were known at the end of their reference quarter, which is generally false in real time. A genuine forecast exercise would use historical vintages or lag each predictor to its actual release date. The point here is practical: to show how the chapter’s calibration logic can be organized chronologically and why overall coverage can differ from subgroup coverage.

To keep the empirical section transparent, we use only one method here: a split conformal interval around a random-forest point forecast. The sample is divided chronologically:

1. training sample through 1999Q4
2. calibration sample from 2000Q1 through 2011Q4
3. test sample from 2012Q1 onward

One data-handling detail matters for the code below: this FRED-QD file dates each quarter by the first day of its final month, so 1999Q4 appears as 1999-12-01. In each row dated t, the predictors refer to quarter t and y_next is GDP growth in quarter t+1. The displayed split dates therefore label predictor/reference quarters, which the plot calls pseudo-forecast origins, while each realized target occurs one quarter later. The cutoffs are placed after the month-start dates so every row lands in exactly one subsample.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_absolute_error

raw = pd.read_csv("data/fred_qd_current.csv")
fred_qd = raw.iloc[2:].copy()
fred_qd["sasdate"] = pd.to_datetime(fred_qd["sasdate"])

for column in fred_qd.columns:
    if column != "sasdate":
        fred_qd[column] = pd.to_numeric(fred_qd[column], errors="coerce")

fred_qd = fred_qd.sort_values("sasdate").reset_index(drop=True)

gdp_growth = 400 * np.log(fred_qd["GDPC1"]).diff()
inflation = 400 * np.log(fred_qd["CPIAUCSL"]).diff()
term_spread = fred_qd["GS10"] - fred_qd["TB3MS"]
log_vix = np.log(fred_qd["VIXCLSx"])

macro = pd.DataFrame(
    {
        "date": fred_qd["sasdate"],
        "y_next": gdp_growth.shift(-1),
        "gdp_growth_current": gdp_growth,
        "gdp_growth_lag1": gdp_growth.shift(1),
        "inflation_current": inflation,
        "inflation_lag1": inflation.shift(1),
        "unrate_current": fred_qd["UNRATE"],
        "fedfunds_current": fred_qd["FEDFUNDS"],
        "term_spread_current": term_spread,
        "vix_current": log_vix,
    }
).dropna().reset_index(drop=True)

train = macro[macro["date"] <= pd.Timestamp("1999-12-31")].copy()
cal = macro[
    (macro["date"] > pd.Timestamp("1999-12-31"))
    & (macro["date"] <= pd.Timestamp("2011-12-31"))
].copy()
test = macro[macro["date"] > pd.Timestamp("2011-12-31")].copy()

features = [column for column in macro.columns if column not in ["date", "y_next"]]

def conformal_quantile(scores, alpha):
    scores = np.asarray(scores)
    n = len(scores)
    k = int(np.ceil((1 - alpha) * (n + 1)))
    return np.inf if k > n else np.sort(scores)[k - 1]

alpha = 0.10
model = RandomForestRegressor(n_estimators=500, min_samples_leaf=5, random_state=42)
model.fit(train[features], train["y_next"])

cal_pred = model.predict(cal[features])
cal_scores = np.abs(cal["y_next"].to_numpy() - cal_pred)
qhat = conformal_quantile(cal_scores, alpha)

test = test.copy()
test["pred"] = model.predict(test[features])
test["lower"] = test["pred"] - qhat
test["upper"] = test["pred"] + qhat
test["covered"] = (test["y_next"] >= test["lower"]) & (test["y_next"] <= test["upper"])
test["width"] = test["upper"] - test["lower"]

vix_cutoff = test["vix_current"].quantile(0.80)
test["high_vix"] = test["vix_current"] > vix_cutoff

summary = pd.DataFrame(
    {
        "Statistic": [
            "Training observations",
            "Calibration observations",
            "Test observations",
            "Nominal coverage",
            "Test coverage",
            "Average interval width",
            "Point-forecast MAE",
            "High-VIX test coverage",
            "High-VIX test observations",
            "Conformal half-width",
        ],
        "Value": [
            len(train),
            len(cal),
            len(test),
            1 - alpha,
            test["covered"].mean(),
            test["width"].mean(),
            mean_absolute_error(test["y_next"], test["pred"]),
            test.loc[test["high_vix"], "covered"].mean(),
            int(test["high_vix"].sum()),
            qhat,
        ],
    }
)

display(summary.round(3))

Table 15.1: Split conformal results for one-quarter-ahead U.S. real GDP growth. The row dates label predictor/reference quarters: the base predictor is fit through 1999Q4, calibrated on rows from 2000Q1 through 2011Q4 at \alpha=0.10, and evaluated on rows from 2012Q1 onward. Coverage, width, and mean absolute error (MAE) are computed on the test sample; the high-VIX rows are restricted to the top quintile by current-row log VIX. Growth rates and interval widths are in annualized percentage points.

	Statistic	Value
0	Training observations	150.000
1	Calibration observations	48.000
2	Test observations	53.000
3	Nominal coverage	0.900
4	Test coverage	0.943
5	Average interval width	13.543
6	Point-forecast MAE	3.157
7	High-VIX test coverage	0.818
8	High-VIX test observations	11.000
9	Conformal half-width	6.771

The results are collected in Table 15.1. This split produces 150 training quarters, 48 calibration quarters, and 53 test quarters. Overall test coverage is about 94%, above the nominal 90% target. Under i.i.d. sampling and distinct scores, finite-sample discreteness alone would imply expected coverage 45/49\approx91.8\%, because k=\lceil0.9\times49\rceil=45. The gap to 94% can easily arise from a short evaluation window: as a rough i.i.d. benchmark, the binomial standard error of a coverage estimate near 0.9 with 53 observations is about 4 percentage points. Across i.i.d. continuous calibration samples, the conditional coverage associated with the k-th order statistic is Beta(k,N_{\text{cal}}+1-k) distributed, here also with a standard deviation of about 4 percentage points. These calculations are benchmarks, not valid time-series standard errors. Quarterly macro data are dependent and nonexchangeable, so the split-conformal theorem does not apply to this exercise; the reported coverage is descriptive. The average interval width is about 13.5 annualized growth points, and the point-forecast MAE is about 3.16 points.

The subgroup comparison is suggestive, but it must be read with more care than the overall figure. If we classify test rows in the top quintile of current-row VIX as high-volatility cases, coverage in that subgroup falls to about 82%. The quintile cutoff is estimated from the full test sample, so this is an ex post diagnostic, not a grouping rule available to a real-time forecaster. A constant-width band can undercover where dispersion is largest, but the evidence here is thin. The subgroup contains only 11 rows and exactly two misses, so the binomial standard error attached to the 82% figure is roughly 12 percentage points, and the gap to the 90% target is well under one standard error.

The composition of those two misses matters even more than their count. They occur at rows dated 2020Q1 and 2020Q2, whose targets are the 2020Q2 COVID collapse and 2020Q3 rebound, with realized annualized growth of -32.8% and +29.9%. If rows whose predictor/reference quarter lies in calendar 2020 are excluded, the remaining high-VIX subgroup contains seven cases and covers all seven. The historically estimated residual quantile used by this fitted constant-width band was far too small to contain the -32.8% realization. These two misses therefore mainly record an unprecedented shock rather than persuasive evidence of systematic conditional undercoverage.

The right conclusion is therefore methodological rather than empirical. The marginal-versus-conditional distinction is genuine, and it is established above by the proposition and by Figure 15.3, where the sample is large enough for the pattern to be visible. A 53-quarter evaluation window with an 11-quarter subgroup is simply too short to demonstrate it convincingly. That limitation is itself worth absorbing: subgroup coverage diagnostics are appealing in principle, but in quarterly macroeconomic samples the subgroups are usually far too small to distinguish a real conditional-coverage failure from a couple of unusual quarters.

Figure 15.5 makes the same point visually. The left panel plots the split-conformal band over the test period. The right panel compares overall coverage with the coverage in the high-VIX subgroup. The nominal 90% line is included so the interpretation stays tied to the calibration target rather than to a vague notion of “good” performance.

plot_df = test.copy().reset_index(drop=True)
coverage_df = pd.DataFrame(
    {
        "group": ["Overall test sample", "High-VIX test quintile"],
        "coverage": [
            plot_df["covered"].mean(),
            plot_df.loc[plot_df["high_vix"], "covered"].mean(),
        ],
    }
)

fig, axes = plt.subplots(1, 2, figsize=(12.2, 4.8))

axes[0].fill_between(
    plot_df["date"],
    plot_df["lower"],
    plot_df["upper"],
    color="C0",
    alpha=0.18,
    label="Split conformal interval",
)
axes[0].plot(plot_df["date"], plot_df["pred"], color="C0", linewidth=1.8, label="Point forecast")
axes[0].scatter(plot_df["date"], plot_df["y_next"], color="black", s=26, label="Realized growth", zorder=3)
miss_mask = ~plot_df["covered"].to_numpy()
axes[0].scatter(
    plot_df.loc[miss_mask, "date"],
    plot_df.loc[miss_mask, "y_next"],
    color="C3",
    s=34,
    label="Miss",
    zorder=4,
)
axes[0].set_title("FRED-QD Split Conformal Intervals")
axes[0].set_xlabel("Forecast origin")
axes[0].set_ylabel("Next-quarter annualized GDP growth")
axes[0].grid(True, alpha=0.22)
axes[0].legend(frameon=False, loc="upper left")

bars = axes[1].bar(coverage_df["group"], coverage_df["coverage"], color=["C0", "C1"], alpha=0.85)
axes[1].axhline(1 - alpha, color="0.35", linestyle="--", linewidth=1.6, label="Nominal coverage")
axes[1].set_ylim(0, 1.05)
axes[1].set_title("Overall Coverage vs High-Volatility Coverage")
axes[1].set_ylabel("Empirical coverage")
axes[1].grid(True, axis="y", alpha=0.22)
axes[1].legend(frameon=False, loc="upper right")
for bar in bars:
    height = bar.get_height()
    axes[1].text(
        bar.get_x() + bar.get_width() / 2,
        height + 0.03,
        f"{height:.3f}",
        ha="center",
        va="bottom",
        fontsize=10,
    )

plt.tight_layout()
plt.show()

Figure 15.5: Chronological current-vintage illustration using quarterly U.S. macro data from FRED-QD. Row dates on the horizontal axis label predictor/reference quarters, and each outcome is next-quarter annualized GDP growth. The left panel shows random-forest point forecasts and split-conformal intervals calibrated on rows dated 2000Q1-2011Q4; red markers indicate misses. The right panel compares empirical coverage over all test rows dated from 2012 onward with coverage in the highest-VIX quintile. The dashed line marks the nominal 90% target. Release delays and real-time data vintages are not reconstructed.

This example should not be overread. It is not a theorem check for dependent data, it is not a horse race among conformal variants, and as the subgroup discussion above shows, it is not strong evidence about conditional coverage either. Its purpose is narrower: to show how a manual conformal workflow looks in a realistic macro forecasting design, and to make concrete both how such a workflow is assembled and how little a short evaluation window can settle.

15.9 Improving Data Efficiency: CV+ and Jackknife+

Split conformal uses one part of the sample for fitting and another for calibration. This clean separation makes the theory easy, but it can waste data when the sample is small.

The key pedagogical point is that split conformal uses one global calibration quantile computed from a held-out set. Jackknife+ and CV+ use a different idea: they generate an out-of-sample residual for every observation, then transport each of those residuals to the new predictor value through the model that excluded that observation.

This comparison is shown in Figure 15.6. Split conformal pays for its clean theory by holding back a calibration sample. Jackknife+ instead fits many leave-one-out models so that every observation contributes one out-of-sample residual without being permanently removed from model fitting. CV+ keeps the same logic but replaces leave-one-out fitting by K-fold omitted blocks.

import matplotlib.pyplot as plt
from matplotlib.patches import Rectangle

fig, ax = plt.subplots(figsize=(11.2, 4.8))
ax.set_xlim(0, 100)
ax.set_ylim(0, 15)
ax.axis("off")

rows = [
    (11.2, "Split conformal", [(8, 44, "Training block", "#9ecae1"), (54, 24, "Calibration block", "#fdd49e")]),
    (7.1, "Jackknife+", [(8, 70, "Repeated leave-one-out fits and residuals for every observation", "#c7e9c0")]),
    (3.0, "CV+", [(8, 70, "Repeated K-fold omitted-block fits and out-of-fold residuals", "#dadaeb")]),
]

for y, label, blocks in rows:
    ax.text(1.2, y + 0.55, label, ha="left", va="center", fontsize=11, fontweight="bold")
    for x0, width, text, color in blocks:
        rect = Rectangle((x0, y), width, 1.0, facecolor=color, edgecolor="0.25", linewidth=1.0)
        ax.add_patch(rect)
        ax.text(x0 + width / 2, y + 0.5, text, ha="center", va="center", fontsize=10)

for xpos in [20, 36, 52, 68]:
    ax.plot([xpos, xpos], [6.8, 8.4], color="0.35", linestyle="--", linewidth=1.0)
    ax.plot([xpos, xpos], [2.7, 4.3], color="0.35", linestyle="--", linewidth=1.0)

ax.text(50, 14.1, "How More Data-Efficient Conformal Methods Reuse the Sample", ha="center", fontsize=13)
ax.text(43, 9.1, "Every observation contributes one out-of-sample residual", fontsize=10, ha="center")
ax.text(43, 4.9, "Folds replace leave-one-out refits when full jackknife+ is too expensive", fontsize=10, ha="center")

plt.tight_layout()
plt.show()

Figure 15.6: Data usage in split conformal, jackknife+, and CV+. The first row shows split conformal, where one block is reserved for calibration and never used for fitting the base model. The second row shows jackknife+, where each observation is omitted once, producing one out-of-sample residual per observation. The third row shows CV+, where folds rather than single observations are omitted. The figure is schematic: the repeated blue and orange blocks indicate many repeated refits, not one single partition.

Split Conformal vs Jackknife+

The easiest way to understand the difference is:

1. Split conformal fits one model and calibrates it on a held-out sample.
2. Jackknife+ fits many leave-one-out models and lets every observation contribute one out-of-sample residual.
3. CV+ keeps the same logic as jackknife+ but replaces leave-one-out refits by K-fold refits.
4. That is why jackknife+ and CV+ use many candidate lower and upper bounds, not one single residual quantile.

Jackknife+ is the cleanest bridge from split conformal to these more data-efficient methods, so it is best to understand that case first (Barber et al. 2021).

Suppose we have observations (X_i,Y_i) for i=1,\dots,N. For each i, fit a leave-one-out model \hat \mu^{(-i)} using all data except observation i. Then compute the leave-one-out residual

R_i = |Y_i-\hat \mu^{(-i)}(X_i)|.

For a new predictor value x, the omitted-observation-i model implies the candidate lower and upper bounds

L_i(x)=\hat \mu^{(-i)}(x)-R_i, \qquad U_i(x)=\hat \mu^{(-i)}(x)+R_i.

Sort these as

L_{(1)}(x)\le \cdots \le L_{(N)}(x), \qquad U_{(1)}(x)\le \cdots \le U_{(N)}(x).

The jackknife+ interval is obtained by taking a lower order statistic of the L_i(x) values and an upper order statistic of the U_i(x) values. For a generic ordered sample a_{(1)}\le\cdots\le a_{(m)}, extend the notation by setting a_{(0)}=-\infty and a_{(m+1)}=+\infty. The jackknife+ lower endpoint uses index \lfloor\alpha(m+1)\rfloor, and its upper endpoint uses index \lceil(1-\alpha)(m+1)\rceil. These extended-real conventions keep the finite-sample correction defined even when the requested coverage is too high for a finite endpoint at the available sample size. Concretely,

C_{J+}(x) = \left[ L_{(\lfloor \alpha(N+1)\rfloor)}(x),\; U_{(\lceil (1-\alpha)(N+1)\rceil)}(x) \right].

Thus L_{(0)}(x)=-\infty and U_{(N+1)}(x)=+\infty whenever the corresponding boundary index occurs.

Example: Jackknife+ by Hand

Suppose N=4, \alpha=0.2, and the four leave-one-out models give the following predictions at a new point x_{\text{new}}:

\hat \mu^{(-1)}(x_{\text{new}})=4.9,\; \hat \mu^{(-2)}(x_{\text{new}})=5.2,\; \hat \mu^{(-3)}(x_{\text{new}})=4.7,\; \hat \mu^{(-4)}(x_{\text{new}})=5.0.

Suppose the corresponding leave-one-out residuals are

R_1=0.4,\quad R_2=0.6,\quad R_3=0.5,\quad R_4=0.3.

Then the candidate lower bounds are

\{4.5,\;4.6,\;4.2,\;4.7\},

and the candidate upper bounds are

\{5.3,\;5.8,\;5.2,\;5.3\}.

After sorting,

L_{(1)}=4.2,\;L_{(2)}=4.5,\;L_{(3)}=4.6,\;L_{(4)}=4.7,

and

U_{(1)}=5.2,\;U_{(2)}=5.3,\;U_{(3)}=5.3,\;U_{(4)}=5.8.

Because

\lfloor \alpha(N+1)\rfloor=\lfloor 0.2 \times 5\rfloor = 1 \qquad\text{and}\qquad \lceil (1-\alpha)(N+1)\rceil=\lceil 0.8 \times 5\rceil = 4,

the jackknife+ interval is

[L_{(1)},U_{(4)}]=[4.2,5.8].

The construction is the key lesson. Each omitted-observation fit produces its own lower and upper bound at the new point, and the final interval comes from the empirical spread of those candidate bounds.

CV+ is now easy to interpret: it uses the same idea as jackknife+, but with omitted folds instead of omitted single observations.

Let the sample be partitioned into folds I_1,\dots,I_K. For each fold b, fit a model \hat \mu^{(-b)} on all observations not in I_b. For any observation i\in I_b, define the out-of-fold residual

R_i = |Y_i-\hat \mu^{(-b)}(X_i)|.

Every observation now receives a score computed from a model that did not train on it. For a new predictor value x, define the lower and upper candidate bounds

L_i(x)=\hat \mu^{(-b(i))}(x)-R_i, \qquad U_i(x)=\hat \mu^{(-b(i))}(x)+R_i,

where b(i) is the fold containing observation i. Sort these candidate bounds as

L_{(1)}(x)\le \cdots \le L_{(N)}(x), \qquad U_{(1)}(x)\le \cdots \le U_{(N)}(x).

Using the same extended-order-statistic convention, the displayed CV+ construction is

C_{CV+}(x) = \left[ L_{(\lfloor \alpha(N+1)\rfloor)}(x),\; U_{(\lceil (1-\alpha)(N+1)\rceil)}(x) \right].

Here too, L_{(0)}(x)=-\infty and U_{(N+1)}(x)=+\infty. Jackknife+ is the leave-one-out special case obtained by setting K=N.

So the operational difference is not conceptual but computational:

1. jackknife+ omits one observation at a time
2. CV+ omits one fold at a time
3. both methods propagate out-of-sample residuals into lower and upper candidate bounds at the new point

Because jackknife+ and CV+ train each base model on more observations than a single split does, they can produce shorter intervals when the fitting algorithm is stable and the additional training data improve prediction. Shorter intervals are not guaranteed. Jackknife+ does have a finite-sample marginal coverage result under exchangeability: for a fitting algorithm that is symmetric in the training observations, Barber et al. (2021) prove coverage of at least 1-2\alpha. Although the interval is indexed to target 1-\alpha, its general worst-case guarantee is therefore weaker. CV+ has a related finite-sample theorem, but its exact correction depends on the fold construction, fold sizes, N, and K. Rather than importing those additional conditions here, we use CV+ as a construction and refer to Barber et al. (2021) for the precise result. The practical tradeoff is computational cost, which can be substantial in econometric forecasting settings with expensive rolling validation.

15.10 Conformal Prediction for Time Series

Time series are where econometricians need to be most careful. The standard split conformal theorem assumes exchangeability, but time-ordered data are usually not exchangeable because of serial dependence, structural change, seasonality, and volatility clustering.

The failure is not technical nitpicking. If

Y_t = \rho Y_{t-1}+\varepsilon_t, \qquad |\rho|<1,

then the vector (Y_1,Y_2,Y_3) does not generally have the same distribution as (Y_1,Y_3,Y_2), because the covariance between adjacent observations is \rho \sigma_Y^2 while the covariance at lag two is \rho^2 \sigma_Y^2. Permuting the entries changes the covariance structure. So the rank argument from split conformal no longer applies automatically.

In practice, the right design is closer to the logic of Time Series Cross-Validation. Let \mathcal F_t denote the information available when the forecast is issued at origin t, and let X_t be an \mathcal F_t-measurable predictor vector used to forecast Y_{t+1}. Forecasts should use only \mathcal F_t, and calibration residuals should come from past out-of-sample forecast errors rather than random reshuffling. Known-ahead future covariates can be included in X_t only when their values are genuinely available at origin t.

A simple rolling conformal workflow is:

1. At each forecast origin t, fit the forecasting model \hat\mu_t using only \mathcal F_t.
2. Store recent out-of-sample absolute residuals R_s = |Y_{s+1}-\hat \mu_s(X_s)|, for past origins s<t, using only residuals whose outcomes have already been observed.
3. Form the interval for Y_{t+1} as C_t(X_t) = [\hat \mu_t(X_t)-\hat q_t,\hat \mu_t(X_t)+\hat q_t], where \hat q_t is the empirical (1-\alpha)-quantile of a recent residual window.

This procedure is no longer distribution-free in the same finite-sample sense as i.i.d. split conformal, but it is aligned with the information structure of forecasting. If the residual process is locally stable, the recent residual quantile can still serve as a useful calibration device.

A formal treatment of conformal prediction under distribution shift is given by Gibbs and Candès (2021), who develop adaptive conformal inference (ACI). Let C_t^{1-\alpha_t} be the prediction set issued at origin t for the outcome Y_{t+1} at working miscoverage level \alpha_t—the same indexing as the rolling workflow above—define the realized error

\operatorname{err}_{t+1} = \mathbf{1}\{Y_{t+1}\notin C_t^{1-\alpha_t}\},

and choose a step size \gamma>0. ACI updates

\alpha_{t+1} = \alpha_t+\gamma(\alpha-\operatorname{err}_{t+1}) = \alpha_t-\gamma(\operatorname{err}_{t+1}-\alpha).

Coverage therefore feeds back into the next set: a miss lowers \alpha_t and widens the next set, while a covered outcome raises \alpha_t and narrows it.

The flagship long-run calculation is deterministic. Summing the update from t=1 to T gives

\frac{1}{T}\sum_{t=1}^T \operatorname{err}_{t+1}-\alpha = \frac{\alpha_1-\alpha_{T+1}}{\gamma T}.

Hence the empirical miscoverage discrepancy is O(1/(\gamma T)) whenever the working levels remain uniformly bounded. The usual ACI convention guarantees this boundedness by saturating the underlying set family: a requested coverage level at or below zero returns the empty set, while a level at or above one returns the whole outcome space. With \alpha_1\in[0,1], this feedback keeps \alpha_t\in[-\gamma,1+\gamma] (Gibbs and Candès 2021). No stochastic assumption on the data-generating process is needed for this identity.

This result is about the average error frequency over a long horizon. It does not promise conditional coverage, does not promise coverage at each date, and does not by itself ensure useful interval width: a procedure that alternates between nearly empty and nearly universal sets can have the right average error rate while being operationally poor. The step size \gamma trades faster adaptation against more volatile working levels and interval widths.

A complementary route beyond exchangeability is to replace the unweighted calibration quantile by a weighted one that emphasizes observations judged more relevant to the current test point or forecast origin. Such methods can reduce coverage degradation under distribution drift, but their guarantees depend on the weighting construction; they do not restore the ordinary exchangeable-rank theorem merely by assigning larger weights to recent observations (Barber et al. 2023).

A separate time-series approach, ensemble batch prediction intervals (EnbPI), combines ensemble forecasts with time-ordered residual calibration (Xu and Xie 2021). Its operational spirit is similar to CV+:

• fit many bootstrap-style models on past data
• aggregate their predictions
• calibrate using out-of-sample residuals from the aggregate predictor

The time-series design is shown in Figure 15.7. Training, calibration, and forecasting move forward together over time. This is the right mental model for macroeconomic and financial forecasting. Conformal intervals should be built from historical forecast errors that would actually have been available in real time.

import matplotlib.pyplot as plt
from matplotlib.patches import Rectangle

fig, ax = plt.subplots(figsize=(10.8, 3.6))
ax.set_xlim(0, 100)
ax.set_ylim(0, 12)
ax.axis("off")

rows = [
    (7.0, "Origin t", [(8, 46, "Train up to t", "#9ecae1"), (56, 18, "Calibration residual window", "#fdd49e"), (78, 12, "Forecast t+1", "#c7e9c0")]),
    (3.5, "Origin t+1", [(12, 46, "Train up to t+1", "#9ecae1"), (60, 18, "Updated residual window", "#fdd49e"), (82, 8, "Forecast t+2", "#c7e9c0")]),
]

for y, label, blocks in rows:
    ax.text(1.5, y + 0.35, label, ha="left", va="center", fontsize=11, fontweight="bold")
    for x0, width, text, color in blocks:
        rect = Rectangle((x0, y), width, 0.8, facecolor=color, edgecolor="0.25", linewidth=1.1)
        ax.add_patch(rect)
        ax.text(x0 + width / 2, y + 0.4, text, ha="center", va="center", fontsize=10)

ax.annotate("time", xy=(92, 1.2), xytext=(8, 1.2), arrowprops={"arrowstyle": "->", "linewidth": 1.4})
ax.text(50, 10.8, "Conformal Prediction for Time Series Must Respect Forecast Origins", ha="center", fontsize=13)

plt.tight_layout()
plt.show()

Figure 15.7: Rolling conformal calibration for time-series forecasting. The blue block is the model-training history available at the current forecast origin. The orange block contains past out-of-sample forecast errors used as the calibration window. The green block is the next forecast target. As time moves forward, the training and calibration windows roll, so interval construction respects temporal order instead of permuting observations randomly.

Question for Reflection

Why is it not enough to keep a random calibration set separate from training when the data are a time series?

Suggested Answer

Because the problem is not only data reuse. A random calibration split can mix future and past observations, so the calibration residuals no longer mimic genuine forecast errors at a real forecast origin. Even if training and calibration are disjoint, the resulting conformal quantile can still be based on an invalid information structure.

15.11 Practical Workflow for Econometricians

For applied work, the right conformal design depends on the data structure and the research goal.

If the data are approximately exchangeable and the sample size is adequate. Basic split conformal is often a strong default. It is easy to explain, easy to compute, and its finite-sample guarantee is transparent.

If uncertainty clearly varies with covariates. Use an adaptive score or CQR rather than constant-width intervals. The guarantee is still marginal, but the intervals can become much sharper in quiet regions and more cautious in volatile ones.

If data are scarce and fitting the model is expensive but still feasible to repeat. Consider CV+ or jackknife+. The extra computation may be worthwhile if split conformal wastes too much training information.

If the data are time ordered. Do not use standard split conformal mechanically. Use a forecasting design that respects time order and calibrates on past out-of-sample errors; when appropriate, use a time-series conformal method such as EnbPI.

Evaluate more than raw coverage. Coverage should be checked together with interval width and stability across relevant subgroups or regimes. A method that attains 95% coverage by giving nearly uselessly wide intervals may satisfy the theorem and still fail the applied forecasting task.

This chapter therefore complements rather than replaces Evaluating Predictive Distributions. Proper scores and distributional evaluation remain useful when the researcher cares about the whole predictive distribution, not only coverage. Conformal prediction is a calibration layer, not a universal substitute for all distributional modeling.

15.12 Summary

Key Takeaways

1. Under exchangeability, split conformal uses held-out calibration scores to construct prediction sets with finite-sample marginal coverage.
2. The rank argument remains valid when the base forecasting model is misspecified.
3. Adaptive scores and conformalized quantile regression adjust interval width to predictor-dependent uncertainty.
4. CV+ and jackknife+ reuse data through out-of-fold fits and residuals.

Common Pitfalls

• Marginal coverage does not guarantee coverage conditional on predictors or within every subgroup.
• Keep split-conformal calibration observations out of model fitting and hyperparameter selection.
• Serial dependence or regime change can invalidate the exchangeability-based coverage guarantee.
• Assess interval width and subgroup behavior alongside overall coverage.

15.13 Exercises

Exercise 15.1: Split Conformal Coverage and Adaptive Updating

You use split conformal prediction with a target miscoverage rate of \alpha=0.1 to forecast quarterly GDP growth. The calibration sample has N_{\text{cal}}=19 observations. After fitting a point predictor \hat \mu on the training sample, you compute the absolute residual scores on the calibration sample. Their sorted values are

\{0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1.0,1.1,1.2,1.3,1.4,1.5,1.6,1.7,1.8,1.9\}.

1. Compute the conformal rank index k=\left\lceil (1-\alpha)(N_{\text{cal}}+1)\right\rceil and the conformal quantile \hat q. If \hat \mu(x_{\text{new}})=5.0, compute the 90% split conformal prediction interval.
2. Assume the calibration and future scores are almost surely distinct. Starting from the uniform rank on \{1,\ldots,N_{\text{cal}}+1\}, derive \mathbb{P}\{Y_{N_{\text{cal}}+1}\in C(X_{N_{\text{cal}}+1})\} =\frac{\lceil(1-\alpha)(N_{\text{cal}}+1)\rceil}{N_{\text{cal}}+1} and show that this probability lies in [1-\alpha,\,1-\alpha+1/(N_{\text{cal}}+1)). Evaluate it first for N_{\text{cal}}=19 and then for N_{\text{cal}}=18. Which case has strict ceiling-induced overcoverage?
3. Adaptive conformal inference uses \alpha_{t+1}=\alpha_t+\gamma(\alpha-\operatorname{err}_{t+1}), with \operatorname{err}_{t+1} the miss indicator for the outcome Y_{t+1}, as in the text. Starting from \alpha_1=0.10 with \gamma=0.05, compute \alpha_2,\ldots,\alpha_6 for the error sequence (\operatorname{err}_2,\ldots,\operatorname{err}_6)=(0,1,0,0,1).
4. Verify numerically that the sequence in Part 3 satisfies \frac{1}{5}\sum_{t=1}^5\operatorname{err}_{t+1}-\alpha =\frac{\alpha_1-\alpha_6}{5\gamma}. Explain why this identity does not by itself imply conditional coverage or informative interval widths.

Exam level. Part 1 checks the construction, Part 2 requires the finite-sample rank and discreteness derivation, and Parts 3-4 apply and interpret adaptive conformal inference.

Hint for Part 2

Count how many of the N_{\text{cal}}+1 possible ranks produce coverage, then apply the two defining inequalities for the ceiling function.

Solution

Part 1: Quantile index and interval

Here

k=\left\lceil (1-0.1)(19+1)\right\rceil =\lceil 0.9 \times 20\rceil =18.

So \hat q is the 18th smallest calibration score, which is

\hat q = 1.8.

The split conformal interval is

C(x_{\text{new}}) = [\hat \mu(x_{\text{new}})-\hat q,\hat \mu(x_{\text{new}})+\hat q].

Substituting the numbers gives

[5.0-1.8,\; 5.0+1.8]=[3.2,\; 6.8].

Part 2: Exact coverage and finite-sample discreteness

With distinct scores, the future score has a uniform rank on \{1,\ldots,N_{\text{cal}}+1\}. Exactly k of these ranks produce coverage, so

\mathbb{P}\{Y_{N_{\text{cal}}+1}\in C(X_{N_{\text{cal}}+1})\} =\frac{k}{N_{\text{cal}}+1} =\frac{\lceil(1-\alpha)(N_{\text{cal}}+1)\rceil}{N_{\text{cal}}+1}.

For any real u, u\le\lceil u\rceil<u+1. Taking u=(1-\alpha)(N_{\text{cal}}+1) and dividing by N_{\text{cal}}+1 gives

1-\alpha \le\frac{k}{N_{\text{cal}}+1} <1-\alpha+\frac{1}{N_{\text{cal}}+1}.

For N_{\text{cal}}=19, k=18 and coverage is 18/20=0.9, so the gap is zero. For N_{\text{cal}}=18, k=\lceil0.9\times19\rceil=18 and coverage is 18/19\approx0.947, about 4.7 percentage points above nominal. The second case has strict ceiling-induced overcoverage.

Part 3: Adaptive updates

Apply the update sequentially:

\begin{aligned} \alpha_2&=0.10+0.05(0.10-0)=0.105,\\ \alpha_3&=0.105+0.05(0.10-1)=0.060,\\ \alpha_4&=0.060+0.05(0.10-0)=0.065,\\ \alpha_5&=0.065+0.05(0.10-0)=0.070,\\ \alpha_6&=0.070+0.05(0.10-1)=0.025. \end{aligned}

Part 4: Telescoping identity and its scope

The average error is 2/5=0.4, so the left-hand side is 0.4-0.1=0.3. The right-hand side is

\frac{0.10-0.025}{5\times0.05}=\frac{0.075}{0.25}=0.3.

The identity controls a time average through the endpoint difference in working levels. It does not condition on the current covariates or regime, so it does not imply conditional coverage. Nor does it constrain interval width: a sequence of nearly empty and nearly universal sets could satisfy the same average-error identity while being operationally useless.

Exercise 15.2: Conformalized Quantile Regression and Calibration Direction

You use conformalized quantile regression with a target coverage of 80%, so \alpha=0.2, to build downside-risk intervals for firm-level returns. The initial models target the conventional 10th and 90th conditional quantiles. The calibration sample has four observations, and the lower-quantile prediction, upper-quantile prediction, and realized outcome are:

Observation	\hat q_{\mathrm{low}}(x_i)	\hat q_{\mathrm{high}}(x_i)	y_i
1	10	20	12
2	15	25	26
3	20	40	18
4	22	32	25

1. Compute the four realized CQR scores r_i = \max\{\hat q_{\mathrm{low}}(x_i)-y_i,\; y_i-\hat q_{\mathrm{high}}(x_i)\}, the realizations of the calibration scores R_i from the CQR section.
2. Compute the scalar calibration correction \hat q_{\mathrm{cal}}.
3. For a new point x_{\text{new}}, suppose \hat q_{\mathrm{low}}(x_{\text{new}})=30, \qquad \hat q_{\mathrm{high}}(x_{\text{new}})=50. Compute the final conformalized interval.
4. Show algebraically that the set \{y:\max(\hat q_{\mathrm{low}}(x)-y,\; y-\hat q_{\mathrm{high}}(x))\le \hat q_{\mathrm{cal}}\} is equal to [\hat q_{\mathrm{low}}(x)-\hat q_{\mathrm{cal}},\hat q_{\mathrm{high}}(x)+\hat q_{\mathrm{cal}}].
5. Compare the raw interval [30,50] with the conformalized interval from Part 3. Did the conformal step widen or shrink the band, and what does the sign of \hat q_{\mathrm{cal}} say about the quality of the raw quantile band on the calibration sample?
6. Suppose the method still undercovers for highly leveraged firms while hitting 80% overall coverage in the full sample. Explain why this does not contradict the chapter’s theory, and give one diagnostic or modification from the chapter to consider.

Exam level. Parts 1-4 cover the mechanics and algebra of CQR, while Parts 5-6 test whether students can interpret the conformal correction and understand subgroup undercoverage.

Hint for Part 1

The score is positive when the realized value lies outside the initial quantile band and negative when it lies comfortably inside.

Hint for Part 4

The inequality \max(a,b)\le c is equivalent to imposing both a\le c and b\le c.

Solution

Part 1: Calibration scores

The scores are

\begin{aligned} r_1 &= \max(10-12,\;12-20)=\max(-2,-8)=-2,\\ r_2 &= \max(15-26,\;26-25)=\max(-11,1)=1,\\ r_3 &= \max(20-18,\;18-40)=\max(2,-22)=2,\\ r_4 &= \max(22-25,\;25-32)=\max(-3,-7)=-3. \end{aligned}

So the sorted scores are

\{-3,-2,1,2\}.

Part 2: Conformal quantile

With N_{\text{cal}}=4 and \alpha=0.2,

k=\left\lceil (1-0.2)(4+1)\right\rceil =\lceil 0.8 \times 5\rceil =4.

So \hat q_{\mathrm{cal}} is the 4th smallest score:

\hat q_{\mathrm{cal}} = 2.

Part 3: Final interval

The CQR interval is

[\hat q_{\mathrm{low}}(x_{\text{new}})-\hat q_{\mathrm{cal}},\hat q_{\mathrm{high}}(x_{\text{new}})+\hat q_{\mathrm{cal}}] = [30-2,\;50+2] = [28,\;52].

Part 4: Algebraic simplification

We start from

\max(\hat q_{\mathrm{low}}(x)-y,\; y-\hat q_{\mathrm{high}}(x))\le \hat q_{\mathrm{cal}}.

The maximum inequality is equivalent to requiring both

\hat q_{\mathrm{low}}(x)-y\le \hat q_{\mathrm{cal}} \qquad \text{and} \qquad y-\hat q_{\mathrm{high}}(x)\le \hat q_{\mathrm{cal}}.

The first inequality gives

y\ge \hat q_{\mathrm{low}}(x)-\hat q_{\mathrm{cal}},

and the second gives

y\le \hat q_{\mathrm{high}}(x)+\hat q_{\mathrm{cal}}.

Together,

\hat q_{\mathrm{low}}(x)-\hat q_{\mathrm{cal}} \le y \le \hat q_{\mathrm{high}}(x)+\hat q_{\mathrm{cal}}.

So the conformalized quantile set is exactly

[\hat q_{\mathrm{low}}(x)-\hat q_{\mathrm{cal}},\hat q_{\mathrm{high}}(x)+\hat q_{\mathrm{cal}}].

Part 5: Direction of the conformal correction

The raw interval is

[30,50].

The conformalized interval from Part 3 is

[28,52].

So the conformal step widens the band. Because \hat q_{\mathrm{cal}}=2>0, the calibration sample indicates that the raw quantile band was too narrow to achieve the target coverage and needed outward adjustment.

Part 6: Subgroup undercoverage

This does not contradict the theory because the chapter’s conformal guarantee is marginal. It applies on average over the full population, not necessarily within the subgroup of highly leveraged firms. If leverage is associated with a different uncertainty structure, the overall 80% coverage can coexist with subgroup undercoverage.

A natural diagnostic is to check coverage separately by leverage group or by bins of a risk proxy. A natural methodological response is to use a more adaptive interval construction or to enrich the quantile model so that the lower and upper quantile predictors respond more strongly to state-dependent risk.

Exercise 15.3: Jackknife+ by Hand

Suppose you have N=5 observations and target miscoverage \alpha=0.2. For a new point x_{\text{new}}, the five leave-one-out models produce

\hat \mu^{(-1)}(x_{\text{new}})=2.1,\; \hat \mu^{(-2)}(x_{\text{new}})=1.7,\; \hat \mu^{(-3)}(x_{\text{new}})=2.4,\; \hat \mu^{(-4)}(x_{\text{new}})=2.0,\; \hat \mu^{(-5)}(x_{\text{new}})=2.6.

The corresponding realized leave-one-out residuals are

r_1=0.5,\quad r_2=0.8,\quad r_3=0.3,\quad r_4=0.6,\quad r_5=0.4.

1. Compute the five candidate lower bounds L_i(x_{\text{new}}) and upper bounds U_i(x_{\text{new}}), sort them, determine the required order-statistic indices (note that (1-\alpha)(N+1) is not an integer here), and compute the jackknife+ interval.
2. Suppose instead that all five leave-one-out predictions at x_{\text{new}} were equal to the same value m. Show that with N=5 and \alpha=0.2, the jackknife+ interval reduces to [m-\max_i r_i,\;m+\max_i r_i].
3. Explain why jackknife+ is more data-efficient than split conformal.
4. Choose a method for each application and justify the choice.
   • Application A has N=80 observations, each model refit takes 2 seconds, and holding out 20 observations materially worsens point predictions.
   • Application B has N=5{,}000 observations, each model refit takes 30 minutes, a 20% calibration split barely changes point accuracy, and the total fitting budget is 4 hours.
   Compare split conformal with jackknife+ in terms of training information and computation. Do not claim that either method is universally superior.

Exam level. Parts 1-2 require students to work through the jackknife+ construction formally, and Parts 3-4 test whether they understand the efficiency-computation tradeoff in an applied setting.

Hint for Part 2

If all leave-one-out predictions equal m, ask how sorting m-r_i reverses the ordering of the residuals, while sorting m+r_i preserves it.

Solution

Part 1: Candidate bounds and final interval

The candidate lower bounds are

\begin{aligned} L_1 &= 2.1-0.5 = 1.6,\\ L_2 &= 1.7-0.8 = 0.9,\\ L_3 &= 2.4-0.3 = 2.1,\\ L_4 &= 2.0-0.6 = 1.4,\\ L_5 &= 2.6-0.4 = 2.2. \end{aligned}

So the sorted lower bounds are

L_{(1)}=0.9,\quad L_{(2)}=1.4,\quad L_{(3)}=1.6,\quad L_{(4)}=2.1,\quad L_{(5)}=2.2.

The candidate upper bounds are

\begin{aligned} U_1 &= 2.1+0.5 = 2.6,\\ U_2 &= 1.7+0.8 = 2.5,\\ U_3 &= 2.4+0.3 = 2.7,\\ U_4 &= 2.0+0.6 = 2.6,\\ U_5 &= 2.6+0.4 = 3.0. \end{aligned}

So the sorted upper bounds are

U_{(1)}=2.5,\quad U_{(2)}=2.6,\quad U_{(3)}=2.6,\quad U_{(4)}=2.7,\quad U_{(5)}=3.0.

Because

\lfloor \alpha(N+1)\rfloor=\lfloor 0.2 \times 6\rfloor = 1 \qquad \text{and} \qquad \lceil (1-\alpha)(N+1)\rceil=\lceil 0.8 \times 6\rceil = \lceil 4.8\rceil = 5,

where the ceiling genuinely rounds up because 4.8 is not an integer, the jackknife+ interval is

[L_{(1)},U_{(5)}]=[0.9,\,3.0].

Part 2: Equal fitted values case

If all leave-one-out predictions are equal to m, then

L_i=m-r_i, \qquad U_i=m+r_i.

The sorted lower bounds run from the smallest value m-\max_i r_i upward, while the sorted upper bounds run from the smallest value m+\min_i r_i up to the largest value m+\max_i r_i.

With N=5 and \alpha=0.2, the lower index is \lfloor 0.2 \times 6\rfloor = 1 and the upper index is \lceil 0.8 \times 6\rceil = 5, so

L_{(1)}=m-\max_i r_i, \qquad U_{(5)}=m+\max_i r_i.

Therefore the jackknife+ interval becomes

[m-\max_i r_i,\;m+\max_i r_i].

Part 3: Why jackknife+ is more data-efficient

Split conformal permanently reserves a calibration sample and does not use those observations to fit the base model. Jackknife+ instead fits many leave-one-out models, so every observation contributes to model fitting in almost every refit and also contributes one out-of-sample residual. That is why it often uses scarce data more efficiently.

Part 4: Choosing under data and compute constraints

Application A favors jackknife+. With only 80 observations, the 20-observation calibration block would materially weaken the split-conformal base model, while 80 leave-one-out fits take only about 160 seconds. Jackknife+ lets each fitted model use 79 observations and gives every observation an out-of-sample residual.

Application B favors split conformal. A jackknife+ implementation would require 5,000 refits, far beyond the 4-hour budget, whereas holding out 1,000 observations has little effect on point accuracy. One split fit is therefore computationally feasible and sacrifices little predictive quality. The comparison depends on both the value of additional training observations and the cost of repeated fitting.

Exercise 15.4: Conformal Prediction with Dependent Data

Suppose (Y_t) follows a stationary first-order autoregressive (AR(1)) process

Y_t=\rho Y_{t-1}+\varepsilon_t, \qquad |\rho|<1, \qquad \{\varepsilon_t\}\text{ is i.i.d.},\qquad \mathbb{E}[\varepsilon_t]=0,\qquad \operatorname{Var}(\varepsilon_t)=\sigma_\varepsilon^2.

1. Show that the vector (Y_1,Y_2,Y_3) is generally not exchangeable when \rho \neq 0 by comparing the covariance between the first and second entries of (Y_1,Y_2,Y_3) with the covariance between the first and second entries of the permuted vector (Y_1,Y_3,Y_2).
2. Explain why this invalidates the usual finite-sample split conformal guarantee if one simply randomizes a calibration split from the time series.
3. You are forecasting monthly inflation. Describe a rolling conformal design that respects forecast origins and explain one reason why the calibration window should not be too short and one reason why it should not be too long.
4. Suppose inflation volatility rises sharply after an energy-price shock, but the conformal interval still uses a very long calibration window dominated by pre-shock residuals. Would you expect undercoverage or overcoverage after the shock, and why?

Exam level. Part 1 is a formal dependence argument, Part 2 links it to the conformal rank proof, and Parts 3-4 ask for time-series conformal designs grounded in econometric forecasting practice.

Hint for Part 1

For a stationary AR(1), \operatorname{Cov}(Y_t,Y_{t+h})=\rho^{|h|}\operatorname{Var}(Y_t).

Hint for Part 3

Draw a timeline and mark the information available when each forecast is issued. Use it to decide which realized forecast errors may enter the calibration window.

Hint for Part 4

Ask whether pre-shock residuals are likely to be smaller or larger than post-shock forecast errors when volatility rises abruptly.

Solution

Part 1: Why the process is not exchangeable

For a stationary AR(1),

\operatorname{Cov}(Y_t,Y_{t+h})=\rho^{|h|}\operatorname{Var}(Y_t).

In the vector (Y_1,Y_2,Y_3), the covariance between the first and second entries is

\operatorname{Cov}(Y_1,Y_2)=\rho \operatorname{Var}(Y_t).

In the permuted vector (Y_1,Y_3,Y_2), the covariance between the first and second entries is instead

\operatorname{Cov}(Y_1,Y_3)=\rho^2 \operatorname{Var}(Y_t).

These two covariances are equal only in special cases such as \rho=0. For generic \rho \neq 0, the covariance structure changes under permutation, so the joint distribution is not invariant to permutation. Hence the process is not exchangeable.

Part 2: Why naive split conformal breaks

The split conformal proof relies on the calibration scores and future score being exchangeable, so that the future score has a uniform rank among them. In a time series, random splitting does not restore exchangeability. The calibration residuals can mix observations from different times, different volatility states, and even future information relative to earlier forecast origins. Once the scores are no longer exchangeable, the uniform-rank argument fails, so the usual finite-sample coverage theorem no longer applies.

Part 3: Rolling conformal design

At each month t:

1. Fit the forecasting model using only data available up to month t.
2. Produce the forecast for month t+1.
3. Store the realized out-of-sample absolute forecast error once Y_{t+1} becomes available.
4. Form the conformal interval for month t+2 using the empirical (1-\alpha)-quantile of a recent window of past forecast errors.

The calibration window should not be too short because then the empirical quantile is unstable and sensitive to a few unusually large or small forecast errors. It should not be too long because old residuals may come from different policy regimes, volatility states, or structural relationships and therefore be unrepresentative of current forecast uncertainty.

Part 4: Regime change and long windows

If volatility rises sharply after the shock but the calibration window is dominated by pre-shock residuals, then the conformal quantile will typically be too small relative to current forecast uncertainty. That leads to undercoverage after the shock, because the interval is calibrated on a calmer historical regime than the one currently faced by the forecaster.

15.14 References

Barber, Rina Foygel, Emmanuel J. Candes, Aaditya Ramdas, and Ryan J. Tibshirani. 2021. “Predictive Inference with the Jackknife+.” Annals of Statistics 49 (1): 486–507. https://doi.org/10.1214/20-AOS1965.

Barber, Rina Foygel, Emmanuel J. Candès, Aaditya Ramdas, and Ryan J. Tibshirani. 2023. “Conformal Prediction Beyond Exchangeability.” The Annals of Statistics 51 (2): 816–45. https://doi.org/10.1214/23-AOS2276.

Foygel Barber, Rina, Emmanuel J. Candès, Aaditya Ramdas, and Ryan J. Tibshirani. 2021. “The Limits of Distribution-Free Conditional Predictive Inference.” Information and Inference: A Journal of the IMA 10 (2): 455–82. https://doi.org/10.1093/imaiai/iaaa017.

Gibbs, Isaac, and Emmanuel Candès. 2021. “Adaptive Conformal Inference Under Distribution Shift.” In Advances in Neural Information Processing Systems. Vol. 34.

Lei, Jing, Max G’Sell, Alessandro Rinaldo, Ryan J. Tibshirani, and Larry Wasserman. 2018. “Distribution-Free Predictive Inference for Regression.” Journal of the American Statistical Association 113 (523): 1094–1111. https://doi.org/10.1080/01621459.2017.1307116.

Lei, Jing, and Larry Wasserman. 2014. “Distribution-Free Prediction Bands for Non-Parametric Regression.” Journal of the Royal Statistical Society: Series B 76 (1): 71–96. https://doi.org/10.1111/rssb.12021.

McCracken, Michael W., and Serena Ng. 2020. “FRED-QD: A Quarterly Database for Macroeconomic Research.” Working Paper 26872. National Bureau of Economic Research. https://doi.org/10.3386/w26872.

Romano, Yaniv, Evan Patterson, and Emmanuel J. Candes. 2019. “Conformalized Quantile Regression.” In Advances in Neural Information Processing Systems 32, 3538–48.

Vovk, Vladimir. 2012. “Conditional Validity of Inductive Conformal Predictors.” In Proceedings of the Asian Conference on Machine Learning, 25:475–90. Proceedings of Machine Learning Research.

Vovk, Vladimir, Alexander Gammerman, and Glenn Shafer. 2005. Algorithmic Learning in a Random World. New York: Springer. https://doi.org/10.1007/b106715.

Xu, Chen, and Yao Xie. 2021. “Conformal Prediction Interval for Dynamic Tme-Series.” Proceedings of Machine Learning Research 139.