6 Recurrent Neural Networks
6.1 Overview
Feed-forward neural networks (FNNs) are flexible nonlinear maps from an input vector to an output, but they do not have a built-in state variable. If we want to use them for time-series forecasting, we usually have to create lagged inputs manually, such as the L most recent observations (y_t, y_{t-1}, \ldots, y_{t-L+1}). That approach is often useful, but it fixes the memory length L in advance and treats temporal dependence as a feature-engineering problem.
Recurrent neural networks (RNNs) instead maintain a hidden state that is updated as new observations arrive. This makes them closer in spirit to econometric models with recursively updated states. Examples include autoregressive moving-average (ARMA) models, generalized autoregressive conditional heteroskedasticity (GARCH) models, and state-space models, which represent dynamics through a state that is updated or filtered over time. The key difference is that the RNN update function is learned from data and can be nonlinear and high-dimensional.
This chapter introduces the basic RNN architecture in a forecasting setting, interprets it through an econometric state-variable lens, explains how training works when the data are one long time series, and then turns to the central difficulty of RNNs: vanishing and exploding gradients. The basic recurrent architecture is closely associated with Elman (1990), while training through an unrolled sequence uses the backpropagation ideas of Rumelhart, Hinton, and Williams (1986) and backpropagation through time as described by Werbos (1990). The vanishing-gradient difficulty is analyzed in Bengio, Simard, and Frasconi (1994). That difficulty is the main reason why the next chapter moves to Long Short-Term Memory (LSTM) networks.
6.2 Roadmap
- We first define the forecasting setup and the information-set problem that RNNs are meant to address.
- We contrast a lagged feed-forward network with a recurrent state update.
- We introduce the RNN equations, parameter sharing—the same weights reused at every date—and the unrolled computational graph, in which the recurrence is drawn out over time as a chain.
- We connect RNNs to econometric state-variable models such as GARCH.
- We discuss deterministic versus probabilistic outputs for forecasting applications.
- We derive backpropagation through time and analyze the vanishing and exploding gradient problems.
- We finish with training practice—how one long series becomes many training windows, why the backward gradient computation is cut off after a fixed number of steps, and why using the same parameters at every date amounts to assuming stable dynamics—and with the motivation for LSTMs.
6.3 Forecasting Setup and the Information Set
Suppose we want to forecast a scalar outcome Y_{t+1} using information available at time t. Recall the forecast-origin information set \mathcal{F}_t from the time-series cross-validation discussion: it contains the variables in the form in which they were actually available at the forecast origin. In a macroeconomic or financial application, \mathcal{F}_t may contain past values of the target variable, lagged predictors, realized returns, volatility measures, survey variables, text-based indicators, and other signals known at time t.
A lagged feed-forward network uses a fixed window of this information, for example
\hat{y}_{t+1\mid t} = f_\theta(y_t, y_{t-1}, \ldots, y_{t-L+1}, \mathbf{x}_t, \mathbf{x}_{t-1}, \ldots, \mathbf{x}_{t-L+1}).
Here \hat{y}_{t+1\mid t} denotes a forecast of Y_{t+1} formed at the origin t; the two-index subscript records both dates, and the forecast itself is built from the realized history, following the book’s case convention. This is a reasonable baseline. It is close to how econometricians build distributed-lag or autoregressive forecasting models. The limitation is that the window length L must be chosen before estimation, and the model does not update an internal state as new observations arrive.
From this point onward, \mathbf{x}_t denotes the complete per-period input to the RNN. It may include y_t together with any other predictors known at time t. With this convention, an RNN replaces the fixed lag window by a recursively updated summary:
\mathbf{h}_t = \text{summary}_\theta(\mathbf{x}_1, \ldots, \mathbf{x}_t).
The hidden state \mathbf{h}_t is not an observed economic variable, but it is not a latent stochastic state that must be filtered. Once the network parameters are fixed, \mathbf{h}_t is computed exactly from the input history. It is a learned vector summary of the past that is useful for prediction.
For our purposes, the useful analogy is: an RNN is a nonlinear state-update model. Like a state-space model or a GARCH recursion, it carries information forward through a state variable. Unlike those classical models, the state update is not chosen by theory in a low-dimensional parametric form; it is learned as part of a neural network.
6.4 From a Lagged FNN to a Recurrent State
The previous section described the hidden state abstractly, as a recursively updated summary of the past. We now make the recursion explicit. A feed-forward network with lagged inputs can approximate nonlinear autoregressive relationships, but it treats the chosen lag vector as the full input: if the relevant history spans many periods, or if the features of the history that matter change over time, the fixed window becomes the binding constraint. At the architectural level, a state-carrying RNN removes the fixed window: at each step, the network updates a hidden state from the current input and the previous state, so all earlier observations can enter the prediction through the current state. (The fixed-window training scheme introduced later in the chapter truncates this history in estimation.)
An RNN uses the same update rule at every time step. At time t, it takes the current input \mathbf{x}_t and the previous hidden state \mathbf{h}_{t-1} and produces a new hidden state \mathbf{h}_t. This creates a recursive representation of the past:
\mathbf{h}_t = g_\theta(\mathbf{h}_{t-1}, \mathbf{x}_t).
This state can then be used to produce a forecast, classification probability, or distributional parameter:
\hat{y}_{t+1\mid t} = q_\theta(\mathbf{h}_t).
The essential point is that the model learns both the update rule and the predictive mapping.
6.5 RNN Architecture and Notation
The recurrence can be visualized in two ways: a compact form with a self-loop, representing the recurrence, and an “unrolled” form, which shows the flow of information through time.
The left panel in Figure 6.1 shows the compact representation: a recurrent link feeds the previous hidden state back into the model. The right panel shows the same model unrolled through time. Unrolling is mainly a computational device for training; it does not mean that the model has different parameters at different dates.
Throughout this section, an output subscript records the forecast origin. Thus, in a scalar one-step-ahead application, \hat{\mathbf{y}}_t plays the role of \hat{y}_{t+1\mid t}, or gives parameters of the predictive distribution of Y_{t+1} given \mathcal{F}_t. As everywhere in the book, an unhatted lowercase y_s denotes the outcome realized at calendar time s, and uppercase Y_s the corresponding random variable.
At each time step t, define:
- \mathbf{x}_t \in \mathbb{R}^{p}: The input vector at the current time step (e.g., today’s stock returns and trading volume); p is the per-period input dimension, as in the feed-forward chapter.
- \mathbf{h}_{t-1} \in \mathbb{R}^{H}: The hidden state from the previous time step, which acts as the network’s memory of the past.
- \mathbf{h}_t \in \mathbb{R}^{H}: The new hidden state, calculated by combining the current input \mathbf{x}_t with the previous state \mathbf{h}_{t-1}.
- \hat{\mathbf{y}}_t \in \mathbb{R}^{O}: The output produced at forecast origin t (e.g., the prediction for the next period’s volatility), obtained from the current hidden state. The hat marks it as a model output rather than an observed quantity.
The update equations are:
\begin{aligned} \mathbf{h}_t &= g(\mathbf{W}_{hh}\mathbf{h}_{t-1} + \mathbf{W}_{xh}\mathbf{x}_t + \mathbf{b}_h) \quad &\text{(Update memory)} \\ \hat{\mathbf{y}}_t &= \mathbf{W}_{hy}\mathbf{h}_t + \mathbf{b}_y \quad &\text{(Produce output)} \end{aligned}
with \mathbf{h}_0 fixed by convention, typically \mathbf{h}_0 = \mathbf{0}, since the state recursion is otherwise undefined at t=1, and with the following parameters:
- \mathbf{W}_{hh} \in \mathbb{R}^{H \times H} (hidden-to-hidden weights)
- \mathbf{W}_{xh} \in \mathbb{R}^{H \times p} (input-to-hidden weights)
- \mathbf{W}_{hy} \in \mathbb{R}^{O \times H} (hidden-to-output weights)
- \mathbf{b}_h \in \mathbb{R}^{H}, \mathbf{b}_y \in \mathbb{R}^{O} (bias terms)
These are collectively \theta = \{\mathbf{W}_{hh}, \mathbf{W}_{xh}, \mathbf{W}_{hy}, \mathbf{b}_h, \mathbf{b}_y\}, the parameters that the generic maps f_\theta, g_\theta, and q_\theta introduced above stand for.
We use \tanh as the activation g. Its range is bounded between -1 and 1, and it is centered at zero.
Key Properties
- Parameter sharing: The same weight matrices (\mathbf{W}_{hh}, \mathbf{W}_{xh}, \mathbf{W}_{hy}) and biases are used at every time step. The total parameter count is therefore independent of sequence length, so the same network can be applied to sequences of different lengths.
- Memory: The hidden state \mathbf{h}_t carries information from past inputs forward, so observations at t-k for k\geq 1 can still influence \hat{y}_{t+1\mid t} through the state.
6.6 Econometric State-Variable Interpretation
An RNN can be viewed as a flexible nonlinear generalization of classical time-series models with recursively updated states. A useful benchmark is the GARCH model of Bollerslev (1986).
In the GARCH(1,1) model, the return at t is decomposed as R_t = \mu_t + \varepsilon_t, where \mu_t=\mathbb{E}[R_t\mid\mathcal{F}_{t-1}] and \varepsilon_t=\sigma_t Z_t. The innovations Z_t are independent and identically distributed, independent of \mathcal{F}_{t-1}, and normalized so that \mathbb{E}[Z_t]=0 and \mathbb{E}[Z_t^2]=1. Under these conditions, \sigma_t^2=\operatorname{Var}(\varepsilon_t\mid\mathcal{F}_{t-1}), and the conditional variance follows the recursion
\sigma_t^2 = \omega + \alpha \varepsilon_{t-1}^2 + \beta \sigma_{t-1}^2,
where \omega>0 and \alpha,\beta\geq 0. For a covariance-stationary innovation process with finite unconditional variance, the stated normalization requires \alpha+\beta<1. The state is \sigma_{t-1}^2, and the update is linear in the lagged squared innovation and the lagged variance.
An RNN-style volatility model could instead use
\begin{aligned} \mathbf{h}_t &= \tanh(\mathbf{W}_{hh}\mathbf{h}_{t-1} + \mathbf{W}_{xh}\mathbf{x}_t + \mathbf{b}_h) \\ \sigma_{t+1}^2 &= \text{softplus}(\mathbf{W}_{hy}\mathbf{h}_t + \mathbf{b}_y), \end{aligned}
where \mathbf{x}_t might contain lagged returns, squared returns, realized volatility measures, or macro-financial predictors known at time t. The hidden state \mathbf{h}_t is a high-dimensional vector that learns a nonlinear summary of the relevant history. The softplus transformation ensures a positive variance forecast.
The analogy has clear limits. A GARCH conditional variance is a filtered state: a deterministic function of past observations, updated as new data arrive—what Cox (1981) calls an observation-driven model—and it has a precise probabilistic role as the conditional variance of the innovation within a fully specified conditional distribution. An RNN hidden state is likewise observation-driven, a deterministic function of past inputs, but its components carry no assigned probabilistic role: the hidden state is a predictive representation. The RNN becomes a probabilistic econometric model once we specify how its output maps into a conditional distribution. A proper scoring rule then determines how the parameters of that model are estimated.
The hidden state \mathbf{h}_t plays a role similar to a state vector in a state-space model or a volatility state in GARCH. Like the GARCH variance—and unlike the latent state of a stochastic state-space model, which must be inferred by filtering—\mathbf{h}_t is exactly computable from past observations once the parameters are known. The difference is that its components are learned predictive features, without a direct one-to-one econometric interpretation.
6.7 Outputs and Loss Functions
The output layer determines what kind of forecasting object the RNN produces. For a conditional mean forecast, we can use a linear output
\hat{y}_{t+1\mid t} = \mathbf{W}_{hy}\mathbf{h}_t + \mathbf{b}_y
and train by minimizing squared loss. For a binary event, such as recession prediction, we can pass the output through a sigmoid and train with cross-entropy. For a probabilistic forecast, the output should parameterize a distribution and be trained with a proper scoring rule. For example, for demeaned returns \varepsilon_{t+1}:
\varepsilon_{t+1} \mid \mathcal{F}_t \sim \mathcal{N}(0, \sigma_{t+1}^2), \qquad \sigma_{t+1}^2 = \text{softplus}(\mathbf{W}_{hy}\mathbf{h}_t + \mathbf{b}_y).
This distinction matters in econometrics. A deterministic RNN trained by squared loss gives a point forecast. A distributional RNN gives a predictive distribution; negative log-likelihood is one familiar training score, but other differentiable proper scores, such as the continuous ranked probability score (CRPS), are also valid. The resulting forecasts can be evaluated with the scoring rules from the predictive-distribution chapter.
For time-series applications, \mathbf{x}_t and \mathbf{h}_t must only use information available at the forecast origin. The recurrent architecture does not remove the usual econometric concern about look-ahead bias, revised macroeconomic data, publication lags, or temporal leakage in validation.
If a macroeconomic predictor is published with a one-month delay and later revised, what should be included in \mathcal{F}_t when training an RNN for real-time forecasting?
\mathcal{F}_t should contain only the data vintage available at forecast origin t. With a one-month publication delay, the current-month value is not yet available, and revisions released after t are also unavailable. Each training row should therefore use the vintage that had been released by its own forecast origin, rather than values from the revised historical series available later.
6.8 Training Through Time
Training an RNN raises a problem absent from feed-forward networks: because the same weight matrices (\mathbf{W}_{hh}, \mathbf{W}_{xh}) are applied at every time step, the gradient with respect to each matrix is the sum of contributions from every time step. The algorithm that handles this is backpropagation through time (BPTT).
The core idea of BPTT is to unroll the network through time, as shown in Figure 6.1. This conceptually transforms the RNN into a deep feed-forward network where each time step becomes a layer, but with the critical constraint that the weights are shared across all layers. Once unrolled, standard backpropagation can be applied.
For clarity, the chapter specializes to a single terminal loss evaluated at the terminal step T, written in the forecasting convention: the step-T output is scored against the outcome it forecasts,
L_T = \ell(y_{T+1}, \hat{\mathbf{y}}_T),
with the observed target first and the network output second, as in the feed-forward networks chapter. In a sequence-to-sequence objective, the gradient is obtained by summing analogous contributions over the per-step losses \sum_{t=1}^{T}\ell(y_{t+1},\hat{\mathbf{y}}_t). For a loss evaluated only at the final time T, the gradient with respect to \mathbf{W}_{hh} can be written schematically as
\frac{\partial L_T}{\partial \mathbf{W}_{hh}}=\sum_{t=1}^{T}\frac{\partial L_T}{\partial \mathbf{h}_t}\frac{\partial \mathbf{h}_t}{\partial \mathbf{W}_{hh}}.
In this decomposition, \partial \mathbf{h}_t/\partial \mathbf{W}_{hh} denotes the local derivative of the step-t update with \mathbf{h}_{t-1} held fixed; the dependence of \mathbf{h}_{t-1} itself on \mathbf{W}_{hh} is not part of this factor—those paths are collected by the other summands. Without this convention the sum would double-count. The term \frac{\partial L_T}{\partial \mathbf{h}_t} captures how an adjustment to the hidden state at an early time t affects the final loss at time T. This requires propagating the gradient backward through every intermediate step, leading to a long product of Jacobian matrices:
\frac{\partial \mathbf{h}_t}{\partial \mathbf{h}_k} = \frac{\partial \mathbf{h}_t}{\partial \mathbf{h}_{t-1}}\, \frac{\partial \mathbf{h}_{t-1}}{\partial \mathbf{h}_{t-2}}\cdots \frac{\partial \mathbf{h}_{k+1}}{\partial \mathbf{h}_{k}}, \qquad \frac{\partial \mathbf{h}_i}{\partial \mathbf{h}_{i-1}} = \operatorname{diag}\big(g'(\mathbf{z}_i)\big)\,\mathbf{W}_{hh},
where g' is the derivative of the activation function (e.g., \tanh') and \mathbf{z}_i = \mathbf{W}_{hh}\mathbf{h}_{i-1} + \mathbf{W}_{xh}\mathbf{x}_i + \mathbf{b}_h is its pre-activation input at step i. Each one-step Jacobian follows from \mathbf{h}_i = g(\mathbf{z}_i) with \mathbf{z}_i affine in \mathbf{h}_{i-1}; note that the factors do not commute, so the product must be composed in the order shown, from step t down to step k+1. (Backpropagation code often works with the transposed form \mathbf{W}_{hh}^{\top}\operatorname{diag}(g'(\mathbf{z}_i))—that is the adjoint, used to propagate gradient vectors backward, not the Jacobian \partial \mathbf{h}_i/\partial \mathbf{h}_{i-1} itself.) The detailed derivations for a one-dimensional hidden state can be found in Exercise 6.1 at the end of the chapter.
6.9 Vanishing and Exploding Gradients
The long product of Jacobians is the source of the vanishing gradient problem. If each one-step Jacobian along the trajectory satisfies \lVert \operatorname{diag}(g'(\mathbf{z}_i))\,\mathbf{W}_{hh} \rVert \leq \kappa for some \kappa < 1, the norm of the product is at most \kappa^{T-t}: the gradient signal shrinks exponentially as it propagates backward through time. Consequently, contributions from early time steps (t \ll T) become so small that they are effectively zero. For econometric forecasting, this is the central limitation of a plain RNN. The model may have a recursive state, but training can fail when a loss must assign credit across a long time gap.
The same recurrent-state Jacobian product can also grow, in which case gradients explode rather than vanish. Since g' \leq 1 for the \tanh activation, each one-step Jacobian is bounded in norm by \lVert\mathbf{W}_{hh}\rVert, so \lVert\mathbf{W}_{hh}\rVert > 1 is necessary for this upper bound to permit exponential pathwise growth. It is not sufficient: whether the product actually grows depends on the entire trajectory of factors \operatorname{diag}(g'(\mathbf{z}_i))\,\mathbf{W}_{hh}. A per-factor bound below one forces decay, but individual factors with norms above one do not by themselves force growth. Nor does \lVert\mathbf{W}_{hh}\rVert\leq 1 uniformly bound every total parameter gradient, because contributions from many loss terms can still accumulate with sequence length.
In the scalar case, the pathwise picture is transparent: the product is \prod_{i=t+1}^{T} |W_{hh}|\,g'(z_i) and grows geometrically whenever the factors stay above some \gamma > 1 along the trajectory. Exploding gradients show up as numerical overflow or parameter updates that jump erratically between iterations, rather than as a silent failure to learn long-run dependence.
The standard remedy is gradient clipping (Pascanu, Mikolov, and Bengio 2013): if the norm of the stacked raw gradient vector \mathbf{g} exceeds a threshold c, replace \mathbf{g} by c\,\mathbf{g}/\lVert\mathbf{g}\rVert before the optimizer sees it. This transformation caps the raw gradient norm and preserves the raw gradient direction. Under plain stochastic gradient descent, it also caps the norm of the immediate parameter step, up to the learning rate; with momentum or adaptive optimizers, the eventual update also depends on stored optimizer state. Clipping stabilizes the exploding case but does nothing for vanishing gradients, whose problem is the loss of signal across a long credit-assignment path.
6.10 From One Series to Training Windows
The derivations above treat one sequence of length T with a single terminal loss. Estimation practice starts from something less tidy: one long time series of, say, N observations, from which the training sequences must first be constructed. Three choices—the window construction, the truncation of the gradient, and the decision to share parameters across time—deserve to be explicit, because each has an econometric interpretation and each reappears in the empirical chapter on networks for time series.
Windows as training observations. The standard construction cuts the series into overlapping fixed-length windows. With window length \tau, each forecast origin t \geq \tau contributes one training pair: the input sequence (\mathbf{x}_{t-\tau+1}, \ldots, \mathbf{x}_t) and the target y_{t+1}. The state is re-initialized at \mathbf{h}_0 = \mathbf{0} at the start of every window, and the recursion runs only within the window. Temporal order therefore matters within each window, but whole windows may be assigned to mini-batches in any order. Contiguity across batch elements becomes necessary only when the final state of one segment is carried into the next. One series of length N thus becomes roughly N - \tau training sequences. These windows overlap heavily—consecutive windows share \tau - 1 observations—so they are anything but independent draws: the caveats about mini-batch gradients under dependent data from the optimization chapter apply, and the chronological train–validation–test discipline of the cross-validation chapter applies at the level of forecast origins.
Fixed windows and truncated backpropagation. Re-initializing the state at each window has a second effect: both the state recursion and the gradients run only across the \tau steps inside a window. It is worth separating two horizons that this fixed-window training scheme happens to tie together. The gradient horizon is how far back credit assignment reaches. Cutting the backward derivative chain after a fixed number of steps is truncated backpropagation through time (Williams and Peng 1990), and it caps the Jacobian products of the previous sections at length \tau no matter how long the series is. The state horizon is how far back the forward recursion carries information. A state-carrying variant of truncated backpropagation feeds the final state of one segment in as the initial state of the next while still detaching the gradient computation between segments: the state horizon then extends beyond \tau even though the gradient horizon does not, so information from the distant past can still enter the forecast even though no learning signal flows back to it. Fixed-window training—the scheme used in the empirical chapter on networks for time series—truncates both horizons at \tau. Truncation reduces computation because the cost of a backward pass grows with sequence length. It may discard little additional usable gradient signal when contributions from earlier steps are already numerically negligible, but that negligibility is itself the vanishing-gradient problem rather than evidence that distant dependence is irrelevant.
The econometric consequence is most precise in information-set terms. Under fixed-window training, the forecast is a function of \sigma(\mathbf{x}_{t-\tau+1},\ldots,\mathbf{x}_t) and of any engineered features included in those inputs. It cannot use incremental predictive information contained only in observations before the window and not recoverable from the included inputs. This does not mean that every dependence at a horizon longer than \tau is unlearnable: a moving average, cumulative return, or other long-memory summary placed in \mathbf{x}_t can carry older information into the window. The window length therefore plays the same role as the lag-window length L of the lagged FNN at the start of the chapter—the recurrence removes the need to hand-pick which lags matter within the window, but the truncation horizon itself returns as a hyperparameter.
Parameter sharing imposes time-homogeneous dynamics. Finally, applying the same weights (\mathbf{W}_{hh}, \mathbf{W}_{xh}, \mathbf{W}_{hy}) at every time step is not a mere convenience: it imposes that the mapping from state and input to the next state is the same at every date. Whether that restriction is plausible is a question about the data, and it is the direct analogue of assuming a stable parametric structure when estimating an ARMA or GARCH model over a long sample. Stability of the relevant conditional relationships is what makes windows drawn from the 1990s and the 2010s informative about the same mapping—which is exactly what gives the network enough effective observations to estimate its many weights; note that the inputs themselves need not be stationary for shared parameters to be coherent, since time-invariance is a restriction on the update rule, not on the input distribution. Under structural breaks in the dynamics, the shared parameters are misspecified, and the usual econometric responses—subsample estimation, rolling estimation windows, break-aware predictors in \mathbf{x}_t—apply to RNNs just as they do to GARCH.
Plain RNNs are hard to train reliably whenever the forecast depends on information many recurrent steps in the past—precisely the long-run dependence case that motivates reaching for a recurrent model. Their value in this chapter is conceptual: they expose the structural problem that learning long-run dependence requires gradient signal across many time steps. LSTMs address this by modifying the state update with gates and a cell state that preserve information more reliably.
Where the implementation lives. This chapter deliberately stays at the conceptual level—recurrence, information sets, and gradient mechanics—without fitting a network in code. The empirical chapter on networks for time series provides the full Python workflow (sequence construction from a time series, chronological splits, training-only scaling, tuning, and evaluation) that applies directly to the recurrent models introduced here and in the next chapter.
6.11 Summary
- RNNs use shared parameters to update a hidden state that summarizes past inputs.
- Backpropagation through time applies the chain rule to the unrolled sequence of state updates.
- Repeated state transitions can attenuate or amplify gradients, making long-range learning difficult.
- Fixed windows limit the gradient horizon; forecasts can use earlier history only if the window’s inputs carry it.
- Hidden-state coordinates do not automatically correspond to identified economic states.
- A deterministic RNN output is not a full predictive distribution without a probabilistic output model.
- Keep observations ordered within windows and keep future targets out of the training split.
- Gradient clipping limits large gradients but does not repair vanishing gradients.
6.12 Exercises
In this exercise, we examine the vanishing gradient problem in RNNs. Consider a simple RNN with a one-dimensional hidden state h_t = \tanh(W_{hh} h_{t-1} + W_{xh} x_t + b_h)—the scalar case H = 1 of the chapter’s update equation, so that W_{hh}, W_{xh}, and b_h are scalars—and loss L_T at the final time step T.
Parts:
- Derive the gradient formula for backpropagation through time. Using the chain rule, derive the expression for \frac{\partial L_T}{\partial W_{hh}}. Show that it involves the term \frac{\partial h_T}{\partial h_t} for each time step t = 1, 2, \ldots, T and express it as a product of derivatives.
- Bound early-step gradient contributions. Show that \left|\frac{\partial h_T}{\partial h_t}\right| \leq |W_{hh}|^{T-t} and explain why, when |W_{hh}| < 1, the contribution associated with an earlier time t is bounded by a geometrically decaying factor.
- Analyze learning implications. Explain why vanishing gradients weaken parameter updates associated with early time steps. Then explain why the bound does not imply a universal maximum sequence length and identify what determines how quickly the contribution of a distant state fades.
- Derive data memory in an AR(1). Consider a stationary first-order autoregressive (AR(1)) process Y_t = \phi Y_{t-1} + U_t with |\phi| < 1 and independent and identically distributed (i.i.d.) mean-zero innovations U_t. Derive the impulse response \partial Y_{t+k} / \partial U_t and show that it decays geometrically in k, formally paralleling the bound from Part 2.
- Separate data memory from model memory. Explain which object’s memory is described by the AR(1) impulse response and which object’s memory is described by the bound from Part 2.
- Locate the learning signal. An RNN is trained to forecast the AR(1) process one step ahead, observing the current value as its input at each step (x_t = y_t), with a one-step-ahead loss at each forecast origin. Explain why the vanishing-gradient bound does not prevent this network from receiving a local learning signal about the AR(1) dynamics, identifying the time gap across which the loss-relevant signal about \phi must travel and whether W_{hh} must equal \phi.
- Trace a genuinely long-gap signal. Now let the data-generating process be Y_{t+1} = \psi X_{t-k} + U_{t+1} for a large delay k, where the network observes the realized scalar input sequence \{x_t\}, the intermediate inputs X_{t-k+1}, \ldots, X_t carry no information about Y_{t+1}, and training again uses a one-step-ahead loss at each origin. Explain why the only gradient path connecting the loss to the relevant input bridges k recurrence steps.
- Explain the lock-in mechanism. Suppose the one-step sensitivities along the relevant fitted path satisfy |W_{hh}\tanh'(z_j)|\leq\kappa<1 uniformly in j. Using the dual role of \partial h_T/\partial h_t as a measure of fitted-state sensitivity and as a factor in the parameter gradient, explain why the network can remain in a short-memory regime during training in the delayed-signal design of Part 7.
Exam level, deliberately long as a whole. Parts 1–3 form one exam question. Parts 4–8 form a second only if the decomposition and bound from Parts 1–2 are supplied with the question.
The weight W_{hh} enters the recursion at every time step, so it affects the loss through every hidden state; the gradient is a sum over these paths. For the product of derivatives, think about how h_T depends on h_t through the chain h_t \to h_{t+1} \to h_{t+2} \to \cdots \to h_T.
You need to bound the absolute value of a product of derivatives; use |ab| = |a||b|. Each derivative \frac{\partial h_k}{\partial h_{k-1}} involves both the weight W_{hh} and the derivative of the activation function. What is the maximum value of \tanh'(z)?
Substitute the AR(1) recursion forward k times to express Y_{t+k} in terms of Y_t and the innovations U_{t+1}, \ldots, U_{t+k}.
Trace the path from the current input x_t into h_t and then into the forecast. Ask how many recurrent transitions this path crosses after x_t enters the state.
Note that \partial h_T / \partial h_t appears twice in this chapter: as the fitted network’s sensitivity to an earlier state, and as a factor in the Part 1 gradient decomposition. Ask which parameter movement would extend the network’s memory, and which gradient carries the demand for that movement.
Part 1: Backpropagation through time
The goal is to find \frac{\partial L_T}{\partial W_{hh}}. The key challenge is that the weight W_{hh} is used at every time step, so its influence on the final loss L_T is the sum of its influence through each step.
The final loss L_T depends on the last hidden state h_T, which depends on h_{T-1} and W_{hh}; in turn, h_{T-1} depends on h_{T-2} and W_{hh}, and so on. Thus W_{hh} affects L_T through its repeated use in every hidden-state update.
Since W_{hh} contributes to the final loss through all these paths, we have by the multivariate chain rule:
\frac{\partial L_T}{\partial W_{hh}} = \sum_{t=1}^T \frac{\partial L_T}{\partial h_t} \frac{\partial h_t}{\partial W_{hh}}
Calculating the Direct Influence
The term \frac{\partial h_t}{\partial W_{hh}} represents the direct influence of W_{hh} on h_t at a single time step: the local derivative of the step-t update with h_{t-1} held fixed, sometimes written \left.\partial h_t/\partial W_{hh}\right|_{h_{t-1}\,\text{fixed}}. The dependence of h_{t-1} itself on W_{hh} is not included in this factor—those paths enter the decomposition through the other summands, and including them here would double-count. From the RNN equation h_t = \tanh(W_{hh} h_{t-1} + W_{xh} x_t + b_h), holding h_{t-1} fixed, this is straightforward:
\frac{\partial h_t}{\partial W_{hh}} = \tanh'(W_{hh} h_{t-1} + W_{xh} x_t + b_h) \cdot h_{t-1}
Calculating the Indirect Influence (The Chain Rule Through Time)
The term \frac{\partial L_T}{\partial h_t} is more complex. It measures how a change in an early hidden state h_t propagates all the way to the end to affect the final loss L_T. The state sensitivity is
\frac{\partial h_T}{\partial h_t} =\prod_{k=t+1}^T \frac{\partial h_k}{\partial h_{k-1}},
where the empty product equals one when t=T. Applying the chain rule through the sequence of hidden states therefore gives
\frac{\partial L_T}{\partial h_t} = \frac{\partial L_T}{\partial h_T} \frac{\partial h_T}{\partial h_{T-1}} \frac{\partial h_{T-1}}{\partial h_{T-2}} \cdots \frac{\partial h_{t+1}}{\partial h_t} = \frac{\partial L_T}{\partial h_T} \prod_{k=t+1}^T \frac{\partial h_k}{\partial h_{k-1}}
This long chain of derivatives is the core of backpropagation through time.
Now we can write the full expression for the gradient. Substituting our product form back into the sum at the beginning, we get
\frac{\partial L_T}{\partial W_{hh}} = \sum_{t=1}^T \underbrace{\left( \frac{\partial L_T}{\partial h_T} \prod_{k=t+1}^T \frac{\partial h_k}{\partial h_{k-1}} \right)}_{\frac{\partial L_T}{\partial h_t}} \underbrace{\left( \frac{\partial h_t}{\partial W_{hh}} \right)}_{\text{direct effect}}
This formula explicitly shows that the gradient contribution from each time step t involves a product of terms that bridges the time gap from t to T.
Part 2: Proving the vanishing-gradient bound
Each derivative in the product has the form:
\frac{\partial h_k}{\partial h_{k-1}} = \tanh'(W_{hh} h_{k-1} + W_{xh} x_k + b_h) \cdot W_{hh}
Since \tanh'(z) = 1 - \tanh^2(z) \leq 1, its value is always between 0 and 1. We can bound the absolute value of the derivative:
\left|\frac{\partial h_k}{\partial h_{k-1}}\right| = |\tanh'(\cdot)| \cdot |W_{hh}| \leq 1 \cdot |W_{hh}| = |W_{hh}|
The key term \frac{\partial h_T}{\partial h_t} for t < T represents the product of derivatives through time:
\left|\frac{\partial h_T}{\partial h_t}\right| = \left|\prod_{k=t+1}^T \frac{\partial h_k}{\partial h_{k-1}}\right| \leq |W_{hh}|^{T-t}
Exponential vanishing: When |W_{hh}| < 1, we have |W_{hh}|^{T-t} \to 0 geometrically as the time gap (T-t) increases. Provided the remaining factors are bounded, the contribution associated with an early time t is therefore bounded by a geometrically decaying factor. This statement concerns early-step contributions, not the sum of all contributions to the parameter gradient.
The bound above is the scalar H=1 case in which the recurrent weight W_{hh} is a single real number. For a multidimensional hidden state (H>1), W_{hh} is a square matrix and the per-step bound uses the spectral norm \|W_{hh}\|_2 (the largest singular value):
\left\|\frac{\partial \mathbf{h}_T}{\partial \mathbf{h}_t}\right\|_2 \le \prod_{k=t+1}^{T} \|\mathbf{W}_{hh}\|_2 \cdot \|\operatorname{diag}(g'(\mathbf{z}_k))\|_2 \le \|\mathbf{W}_{hh}\|_2^{T-t}.
For the linear recurrence \mathbf{h}_t = \mathbf{W}_{hh}\mathbf{h}_{t-1}—equivalently, when all activation derivatives equal one—the asymptotic decay rate is governed by the spectral radius \rho(\mathbf{W}_{hh}) = \max_i |\lambda_i(\mathbf{W}_{hh})|, which coincides with the spectral norm when \mathbf{W}_{hh} is normal (e.g., symmetric). For the nonlinear recursion, the time-varying factors \operatorname{diag}(g'(\mathbf{z}_k)) prevent such a clean characterization, and the spectral-norm bound above is what remains. The qualitative conclusion for the pathwise state sensitivity is the same as in the scalar case: uniformly contractive dynamics geometrically attenuate contributions that travel across long gaps.
Part 3: Learning implications
The exponential decay has three critical effects on learning:
Attenuated early-step contributions: Since \frac{\partial L_T}{\partial W_{hh}} = \sum_{t=1}^T \frac{\partial L_T}{\partial h_T} \frac{\partial h_T}{\partial h_t} \frac{\partial h_t}{\partial W_{hh}}, when \frac{\partial h_T}{\partial h_t} becomes geometrically small for early t, those early-step contributions are attenuated. If the remaining local factors are of comparable magnitude and do not cancel, the effective parameter update is then driven mainly by more recent time steps (large t).
Loss of long-term information: The network receives an exponentially attenuated learning signal for patterns in which information at time t influences the loss at time T with a large gap T-t.
No universal sequence cutoff: The bound does not imply a fixed maximum number of usable lags. The effective decay rate depends on the recurrent weight |W_{hh}| and on the activation derivatives \tanh'(z_k) encountered along the fitted trajectory. Values close to one can preserve signal for many steps; smaller factors make the same signal disappear much sooner. Plain RNNs are therefore unreliable for long-run dependence without architectural or training modifications, but the bound alone does not determine one universal horizon.
The fundamental issue is that learning requires non-negligible gradients to update parameters. Under the contractive conditions above, the signal transmitted across a long gap is geometrically attenuated, so distant dependencies can exert very little influence on parameter updates.
Part 4: Data memory in an AR(1)
Iterating the AR(1) recursion forward k times gives
Y_{t+k} = \phi^k Y_t + \sum_{j=0}^{k-1} \phi^j U_{t+k-j}.
Since Y_t = \phi Y_{t-1} + U_t implies \partial Y_t / \partial U_t = 1, the impulse response is
\frac{\partial Y_{t+k}}{\partial U_t} = \phi^k,
which decays geometrically in k because |\phi| < 1—formally the same geometric decay as the bound |W_{hh}|^{T-t} from Part 2.
Part 5: Data memory versus model memory
The impulse response \phi^k is a property of the data-generating process: it describes how a shock at time t propagates into future values of the observed series, and for \phi close to one the data are highly persistent. The bound from Part 2 concerns \partial h_T / \partial h_t, which is a property of the fitted model: it measures how sensitive the network’s state remains to what happened at time t. If the one-step factors satisfy |W_{hh}|\tanh'(z_k)\leq\kappa<1 uniformly along a path, the fitted network’s sensitivity along that path decays geometrically, whatever the persistence of the data.
Part 6: Why local learning suffices for the AR(1)
By Part 1, \partial h_T / \partial h_t is also the factor that carries the training signal from the loss back to early time steps—yet for the AR(1) this does not prevent the network from receiving a local learning signal about the dynamics. The one-step conditional mean is \mathbb{E}[Y_{t+1} \mid \mathcal{F}_t] = \phi Y_t: everything the network needs in order to forecast is contained in its current input, the realized value x_t = y_t. The relevant path is therefore x_t \to h_t \to \widehat y_{t+1\mid t}, which crosses no recurrent transition after x_t enters the state. The input and output weights can approximate the map y_t\mapsto\phi y_t over the relevant range, with \tanh operating near its linear region. The recurrent term is not required for this one-step target, so W_{hh}=0 is compatible with it; in particular, W_{hh} need not equal \phi. Adjacent-observation forecast errors thus provide a local learning signal without long-range credit assignment.
Part 7: The long-gap gradient path
In the delayed-signal design, the predictive information sits only at lag k and is, by assumption, not recoverable from the intermediate inputs. The only gradient path connecting the one-step-ahead loss to the relevant input x_{t-k} therefore runs through the state recursion across k steps and is scaled by the product from Part 2.
Part 8: The lock-in mechanism
If the one-step sensitivity satisfies |W_{hh}\tanh'(z_j)|\leq\kappa<1 uniformly along the relevant fitted path, the double role of the resulting product can create a lock-in mechanism. The contractive network attenuates the long-range signal that would be needed to push its parameters toward less contractive, longer-memory dynamics. Gradient-based training can therefore remain in a short-memory regime—not because the architecture cannot represent the dependence at other parameter values, but because learning receives little signal pushing it there. The vanishing-gradient problem is thus a failure of credit assignment for long-gap dependence, over and above the fitted model’s own short memory.
Consider the simple RNN shown in Figure 6.2: a single hidden unit processing a sequence of length 3, with hidden update h_t = \tanh(W_{xh} x_t + W_{hh} h_{t-1} + b_h) and linear output \hat{y}_t = W_{hy} h_t + b_y (no output activation).
The parameters are:
- Weight for input-to-hidden: W_{xh} = 0.5
- Weight for hidden-to-hidden: W_{hh} = 0.8
- Bias for hidden unit: b_h = 0.1
- Weight for hidden-to-output: W_{hy} = 1.2
- Bias for output: b_y = 0.0
The input sequence is (x_1, x_2, x_3) = (0.2, -0.5, 0.3) and the initial hidden state is h_0 = 0. Use the approximations \tanh(0.2) \approx 0.197, \tanh(0.008) \approx 0.008, \tanh(0.256) \approx 0.251, \tanh'(0.2) \approx 0.961, and \tanh'(0.008) \approx 1.000.
- Compute the forward pass. For t = 1, 2, 3, compute the hidden state h_t and the output \hat{y}_t.
- Separate value from sensitivity. Your forward pass shows that the second input x_2 = -0.5 is large and negative, yet h_2 \approx 0.008 is nearly zero. Show numerically that the memory term W_{hh} h_1 and the bias b_h almost exactly cancel the input contribution at t=2. Then compute the sensitivity \partial h_2/\partial x_1 and state what a near-zero value of the hidden state does and does not imply about the memory of x_1, connecting your answer to the geometric decay factor from Exercise 6.1.
- Compare two windowing schemes. Suppose this RNN is trained on sequences of length \tau, with one forecast at the end of each sequence. Compare (a) resetting the hidden state to zero for every overlapping window and (b) dividing the series into non-overlapping consecutive segments, carrying the final state from one segment into the next, and detaching it from the previous computational graph. For each scheme, state the state horizon and the gradient horizon. Under scheme (a), describe precisely what information before the window the forecast cannot use. Finally, explain why a structural break can still be a problem under either scheme even though the weights are estimated from many sequences.
Exam level. Part 1 checks the mechanics of the recursion; Part 2 tests the distinction between the value of a state and its sensitivity to past inputs; Part 3 checks the econometric meaning of state truncation, gradient truncation, and shared parameters.
For the sensitivity, apply the chain rule along x_1 \to h_1 \to h_2 and recall that \tanh'(z)=1-\tanh^2(z).
Part 1: Forward pass
At t = 1:
\begin{aligned} h_1 &= \tanh(0.5 \cdot 0.2 + 0.8 \cdot 0 + 0.1) = \tanh(0.2) \approx 0.197 \\ \hat{y}_1 &= 1.2 \cdot 0.197 \approx 0.236 \end{aligned}
At t = 2:
\begin{aligned} h_2 &= \tanh(0.5 \cdot (-0.5) + 0.8 \cdot 0.197 + 0.1) \\ &= \tanh(-0.25 + 0.158 + 0.1) = \tanh(0.008) \approx 0.008 \\ \hat{y}_2 &= 1.2 \cdot 0.008 \approx 0.010 \end{aligned}
At t = 3:
\begin{aligned} h_3 &= \tanh(0.5 \cdot 0.3 + 0.8 \cdot 0.008 + 0.1) \\ &= \tanh(0.15 + 0.006 + 0.1) = \tanh(0.256) \approx 0.251 \\ \hat{y}_3 &= 1.2 \cdot 0.251 \approx 0.301 \end{aligned}
Part 2: A near-zero state value is not the same as lost memory
At t = 2, the input contribution W_{xh} x_2 = 0.5 \times (-0.5) = -0.25 is almost exactly offset by the memory term W_{hh} h_1 = 0.8 \times 0.197 = 0.158 plus the bias b_h = 0.1, giving a pre-activation near zero and thus h_2 \approx 0. This is cancellation: two sizable contributions of opposite sign happen to sum to nearly zero.
Cancellation of the value does not mean the memory of x_1 is gone. Memory in the gradient sense is the sensitivity
\frac{\partial h_2}{\partial x_1} = \underbrace{\tanh'(z_2)\, W_{hh}}_{\partial h_2/\partial h_1} \cdot \underbrace{\tanh'(z_1)\, W_{xh}}_{\partial h_1/\partial x_1} \approx (1.000 \times 0.8) \times (0.961 \times 0.5) \approx 0.38,
using \tanh'(z) = 1 - \tanh^2(z) with z_1 = 0.2 and z_2 = 0.008. A sensitivity of 0.38 after one recurrence step is substantial—in fact, because z_2 \approx 0 places \tanh in its steepest region, h_2 is close to maximally sensitive to perturbations of h_1. Had x_1 been slightly different, h_2 would have been noticeably different: the state still remembers x_1 even though its value happens to sit near zero.
The genuine connection to Exercise 6.1 runs through the rate of decay, not the value of the state. Each additional recurrence step multiplies the sensitivity by at most W_{hh}\,\max_z \tanh'(z) = 0.8, so the influence of x_1 fades geometrically over many steps (0.8^{10} \approx 0.11, 0.8^{30} \approx 0.001). The two-part conclusion: after one step, the memory of x_1 is still strong despite the near-zero state value; over long horizons, it is the geometric factor from Exercise 6.1—not value cancellation—that makes plain RNNs forget.
Part 3: Fixed-window and state-carrying truncation
Under scheme (a), resetting the state gives both the forward state and the backward gradient a horizon of at most \tau steps. The forecast is a function of the inputs inside its window and cannot use predictive information contained only in earlier observations unless that information is summarized in an included feature. Under scheme (b), the carried state can transmit information back to the last reset—potentially the start of the series—although its sensitivity generally decays along the recurrent path. Detaching the state stops the derivative chain at each segment boundary, so the gradient horizon remains at most \tau steps.
Both schemes reuse the same parameters across windows and dates. A structural break changes the state-update or forecast relationship that those common parameters are supposed to describe. Pooling pre-break and post-break windows can then estimate a compromise that fits neither regime well. Chronological validation, rolling or subsample estimation, and break-aware inputs are possible responses; changing the state initialization alone does not remove the shared-parameter restriction.