7 Long Short-Term Memory (LSTM) Networks

7.1 Overview

The previous chapter showed the central weakness of a plain recurrent neural network (RNN): the hidden state is recursively updated, but the gradients used for learning can vanish as they are propagated backward through many time steps. Long Short-Term Memory (LSTM) networks were designed to address precisely this problem.

LSTMs add a dedicated cell state C_t, a memory path through time whose attenuation the network itself controls through gates—small neural-network components that decide what information to retain, what new information to add, and what part of the internal memory to expose as the hidden state.

For econometricians, LSTMs are useful when the relevant predictive state may evolve over time and depend on more than the most recent observations. Examples include asset pricing with time-varying macroeconomic states, macroeconomic forecasting with long predictor histories, and volatility or risk forecasting with persistent regimes.

For optional visual intuition, Christopher Olah’s illustrated LSTM tutorial gives a detailed informal discussion.

7.2 Roadmap

We first introduce the LSTM cell state, hidden state, and gates.
We then work through the LSTM update equations and notation.
Next, we explain why the cell state improves gradient flow relative to a plain RNN.
We discuss when the additional complexity of an LSTM is useful in econometric forecasting, and what changes—and what does not—when training one.
We present a research application in asset pricing.
We close with a look ahead from recurrent states to attention—a construction that lets a model consult earlier positions in a sequence directly instead of compressing the entire past into one state—plus key takeaways, common pitfalls, and manual forward-pass and gradient-flow exercises.

7.3 LSTM Architecture and Notation

LSTM networks, introduced by Hochreiter and Schmidhuber (1997), extend plain RNNs by adding a gating mechanism that controls long-term information flow; the forget gate, central to the modern form of the cell, was added later by Gers, Schmidhuber, and Cummins (2000). The notation below lists the quantities that the LSTM tracks at each time step.

At each time step t, an LSTM cell maintains the following quantities, where H is the number of hidden units and p the input dimension. To keep the gate equations light, we drop the boldface used for vectors in the RNN chapter: x_t, h_t, C_t, and the gates below are all vectors of the stated dimensions, and the weight matrices W_f, W_i, W_c, W_o are likewise set without boldface. The uppercase C_t is the LSTM literature’s standard symbol for the cell state; like every architecture quantity here, it is a deterministic function of the inputs and weights, so the capital letter does not mark a random variable.

Input: x_t \in \mathbb{R}^p, the current input vector
Hidden state: h_t \in \mathbb{R}^H, the output of the LSTM cell, similar to RNN hidden states
Cell state: C_t \in \mathbb{R}^H, the internal memory of the LSTM cell

The LSTM uses three gates and the candidate values (all in \mathbb{R}^H):

Forget gate: f_t, controls which information from the previous cell state to retain or discard
Input gate: i_t, controls what new information to store in the cell state
Candidate values: \tilde{C}_t, new candidate information that could be added to the cell state
Output gate: o_t, controls what parts of the cell state to output as the hidden state

The Gating Mechanism

A gate scales information by a learned multiplier between zero and one. LSTMs obtain these multipliers from the sigmoid function \sigma(z)=1/(1+e^{-z}), applied to each coordinate, and use \odot for elementwise (Hadamard) multiplication. The candidate and hidden-state transformations use the hyperbolic tangent \tanh, which maps each coordinate to (-1,1).

A gate value close to 0 lets almost nothing through that channel.
A gate value close to 1 lets almost everything through.
Intermediate values retain a fraction of the channel’s value. The sigmoid never attains either endpoint, and a gate cannot amplify the value it multiplies.

Figure 7.1 traces how these pieces fit together within a single time step. When reading it, note in particular how short the top path from C_{t-1} to C_t is compared with everything else in the cell.

Figure 7.1: Flow of information through one LSTM cell. The previous hidden state h_{t-1} and the current input x_t enter at the lower left and feed four parallel blocks: three sigmoid gates (\sigma, yellow) producing the forget gate f_t, the input gate i_t, and the output gate o_t, and one tanh block (green) producing the candidate \tilde{C}_t. Circles marked \odot and + apply elementwise multiplication and addition. Along the top, the previous cell state C_{t-1} is scaled by f_t, augmented by i_t \odot \tilde{C}_t, and passed on as C_t—the long-term memory highway. The new hidden state h_t (lower right) is the output gate applied to \tanh(C_t).

Figure 7.1 separates three operations: retain old memory, add selected new content, and expose part of the updated memory. Along the top, f_t scales C_{t-1} and the term i_t\odot\tilde C_t is added to obtain C_t. This top path is short because it involves one elementwise multiplication and one addition and no activation function; it is the route along which information, and later gradient signal, can travel across many periods, and the next section analyzes exactly this path. The downward branch applies \tanh and the output gate o_t to obtain h_t. The cell state therefore carries internal memory, while the hidden state is the representation passed to the output layer and to the next step’s gate computations. We first follow one numerical update, then write the full equations.

Example: Retained memory and new content

Consider one unit at one date, with previous cell state C_{t-1}=1. Suppose the gate and candidate computations have produced f_t=0.8, i_t=0.25, \tilde C_t=0.4, and o_t=0.5. These supplied values isolate the memory update from the calculation of the gates:

C_t = f_t C_{t-1}+i_t\tilde C_t = \underbrace{0.8\times 1}_{\text{retained memory}} +\underbrace{0.25\times 0.4}_{\text{new content}} = 0.9.

The cell retains 0.8 from its old state and adds 0.1 of new content. The hidden state is

h_t=o_t\tanh(C_t)=0.5\tanh(0.9)\approx 0.358.

The two state values differ because the output applies both a nonlinear transformation and a gate. A small hidden-state value need not mean that the cell stores little information.

Update Equations and Weight Matrices

We now compute the gates and candidate from the inputs and learned parameters. In each equation below, [h_{t-1}, x_t] stacks the previous hidden state and the current input into a single vector in \mathbb{R}^{H+p}, and \cdot denotes matrix-vector multiplication (the same symbol is also used for ordinary scalar multiplication). The complete update is:

\begin{aligned} f_t &= \sigma(W_f \cdot [h_{t-1}, x_t] + b_f) \quad \text{(forget gate)} \\ i_t &= \sigma(W_i \cdot [h_{t-1}, x_t] + b_i) \quad \text{(input gate)} \\ \tilde{C}_t &= \tanh(W_c \cdot [h_{t-1}, x_t] + b_c) \quad \text{(candidate values)} \\ C_t &= f_t \odot C_{t-1} + i_t \odot \tilde{C}_t \quad \text{(cell state update)} \\ o_t &= \sigma(W_o \cdot [h_{t-1}, x_t] + b_o) \quad \text{(output gate)} \\ h_t &= o_t \odot \tanh(C_t) \quad \text{(hidden state)} \end{aligned}

Each gate and the candidate values are computed using their own weight matrices and bias vectors. Since the input to each is the concatenation [h_{t-1}, x_t] \in \mathbb{R}^{H+p}, the dimensions are:

W_f, W_i, W_c, W_o \in \mathbb{R}^{H \times (H+p)}: weight matrices
b_f, b_i, b_c, b_o \in \mathbb{R}^H: bias vectors

Equivalently, partitioning each weight matrix into its state and input blocks—W_f = [W_{f,h} \; W_{f,x}] with W_{f,h} \in \mathbb{R}^{H \times H} and W_{f,x} \in \mathbb{R}^{H \times p}—gives W_f \cdot [h_{t-1}, x_t] = W_{f,h}\, h_{t-1} + W_{f,x}\, x_t: the concatenated form used here and the two-matrix form used for the plain RNN in the previous chapter describe the same model.

7.4 Gradient Flow in LSTMs

LSTMs address the vanishing-gradient problem of plain RNNs through the design of the cell state. Its update rule is

C_t = f_t \odot C_{t-1} + i_t \odot \tilde{C}_t

The update has an additive structure. The old state enters through f_t\odot C_{t-1}, while new information enters through i_t\odot\tilde C_t. To isolate the direct cell-state path, hold the gates f_t,i_t,o_t and the candidate \tilde C_t fixed. Because the update acts elementwise, its Jacobian is

\frac{\partial C_t}{\partial C_{t-1}}\bigg|_{\text{gates fixed}} = \operatorname{diag}(f_t),

so for coordinate j,

\frac{\partial C_{t,j}}{\partial C_{t-1,j}}\bigg|_{\text{gates fixed}} = f_{t,j},

and chaining over T-t steps gives

\frac{\partial C_{T,j}}{\partial C_{t,j}}\bigg|_{\text{gates fixed}} = \prod_{k=t+1}^T f_{k,j}.

For comparison, the plain RNN propagates state sensitivity through an ordered chain of matrices:

\frac{\partial h_T}{\partial h_t} = \frac{\partial h_T}{\partial h_{T-1}}\, \frac{\partial h_{T-1}}{\partial h_{T-2}}\cdots \frac{\partial h_{t+1}}{\partial h_t}.

The latest transition is on the left, as in the RNN chapter; these matrices generally do not commute. Each one-step factor is

\frac{\partial h_k}{\partial h_{k-1}} =\operatorname{diag}\!\bigl(\tanh'(W_{hh}h_{k-1}+W_{xh}x_k+b_h)\bigr)\,W_{hh}.

Every plain-RNN factor contains both a recurrent transformation and an activation derivative. The direct LSTM cell-state path instead bypasses \tanh, and its attenuation is controlled by the input-dependent forget gate f_t=\sigma(W_f\cdot[h_{t-1},x_t]+b_f) rather than by \|W_{hh}\|—a statement about this one path only, since the gates are themselves functions of W_f and the full parameter gradient still depends on the weights through them. A value near zero removes old information; a value near one retains it; and different coordinates can operate at different rates. The degree of retention depends on both the gate values and the horizon. For a constant forget gate f_{k,j}=f_j, the gates-fixed derivative is

\frac{\partial C_{T,j}}{\partial C_{t,j}}\bigg|_{\text{gates fixed}} =f_j^{\,T-t}.

Retention is near one only when (T-t)|\log f_j| is small. Each finite product over a positive number of steps is strictly below one; if the gates stay uniformly below some \bar f<1, the product shrinks at least geometrically with the horizon. The LSTM can therefore learn the rate of retention separately by coordinate and context. In a plain RNN, the same recurrent transformation helps determine both the state dynamics and the gradient path, while every step also contributes an activation derivative.

Unlike plain RNNs, where gradient flow is shaped by repeated multiplication with the recurrent weight matrix, LSTMs learn when to let gradients flow through the forget gates. This learned control is the mechanism by which an LSTM can learn long-run dependence more reliably—recall from Exercise 6.1 how repeated state transitions can attenuate the training signal across long time gaps.

The gates-fixed expression describes only the direct cell-state path. In a full backward pass, the gates themselves depend on h_{k-1}=o_{k-1}\odot\tanh(C_{k-1}). Parameter gradients therefore also travel through the hidden state and gate-producing networks, where products of sigmoid and tanh derivatives can attenuate as in a plain RNN. The cell state supplies a low-attenuation route for one component of the backward pass; it does not eliminate all vanishing gradients. Whether a fitted LSTM learns useful long-run dependence remains an empirical question to be judged by validation at deployment-relevant horizons.

7.5 LSTM vs. Plain RNN

The architectural and gradient-flow comparisons summarize the mechanisms introduced by Hochreiter and Schmidhuber (1997) and Gers, Schmidhuber, and Cummins (2000). The parameter counts follow directly from the displayed update equations. The computational comparison counts affine blocks per time step; it is not a claim that an LSTM takes exactly four times as long in every implementation.

Aspect	Plain RNN	LSTM
Memory capability	Can represent long-run dependence, but its long-gap training signal may vanish	Adds a controlled cell-state path that can retain long-run information more reliably
Gradient flow	Products of recurrent weight matrices and activation derivatives	Additive cell-state path with products of forget-gate values
Vanishing gradients	Highly susceptible along the recurrent-state path	Mitigated along the cell-state path, but not eliminated throughout the network
Parameters (recurrent cell)	H(H+p) + H	4H(H+p) + 4H (four weight blocks, approximately four times the RNN cell)
Computational cost	One recurrent affine block per time step	Four gate or candidate affine blocks per time step; wall-clock cost is implementation-dependent

When to use LSTM:

Forecasting tasks where information from distant observations plausibly matters, such as macroeconomic state dynamics or asset-pricing applications with persistent risk states.
Settings where validation performance justifies the extra parameters and computational cost.

When a plain RNN might suffice:

Problems with very short sequences for which long-term memory is not needed.
When computational resources are extremely limited.

Question for Reflection

Suppose an LSTM is trained on monthly data and the forget gate learns values close to 1 for one hidden dimension and close to 0 for another. Using the gates-fixed coordinatewise gradient expression \partial C_{T,j} / \partial C_{t,j} = \prod_{k=t+1}^{T} f_{k,j}, which coordinate attenuates information more slowly? Can the phrase “close to 1” alone justify saying that its gradient is essentially undamped over an arbitrary horizon?

7.6 Training an LSTM

The core estimation framework remains the same when moving from a plain RNN to an LSTM. Training minimizes a loss L over windows cut from the series, gradients are computed on the unrolled cell by backpropagation through time, and the fixed-window and truncated-backpropagation choices of the previous chapter still apply. Under fixed-window training—the scheme used in the empirical chapter—both the hidden state and the cell state are reset to h_0=0 and C_0=0 at the start of every window, so the forward state horizon and the backward gradient horizon both end at the window boundary. Under the state-carrying variant of truncated backpropagation (called stateful training in Keras), the final h and C of one segment are passed to the next contiguous segment while the gradient is detached at the boundary, so the forward state can carry information from before the current segment even though the gradient does not reach back that far. What changes relative to a plain RNN, under either scheme, is what flows through the backward pass: alongside the hidden-state path, the cell-state path contributes the gate-controlled products analyzed above.

Two practical differences deserve mention. First, the parameter count roughly quadruples relative to a plain RNN cell of the same width (4H(H+p)+4H against H(H+p)+H, as in the table above), so overfitting concerns and validation discipline bite correspondingly harder in the small samples typical of macroeconomic applications. Second, gradient clipping remains standard practice: the cell state tames the vanishing problem along one path, but gradients can still explode through the gate-producing networks, and the clipping safeguard from the RNN training discussion carries over unchanged. In Keras, finally, the switch is one line—an LSTM layer in place of a SimpleRNN layer—as the empirical chapter shows in its volatility application.

7.7 Research Application: Deep Learning in Asset Pricing

To see how LSTMs are used in current econometric research, consider two closely related lines of work in empirical asset pricing: the return-prediction approach of Gu, Kelly, and Xiu (2020) and the no-arbitrage approach of Chen, Pelger, and Zhu (2024), which embeds an LSTM at its core.

Two research objectives. Both papers ask whether neural networks can improve on linear specifications of the relation between firm characteristics, macroeconomic information, and stock returns, but they target different objects. Gu, Kelly, and Xiu (2020) predict returns, whereas Chen, Pelger, and Zhu (2024) estimate a no-arbitrage stochastic discount factor (SDF), the pricing kernel implied by the absence of arbitrage. Let \mathcal{I}_{i,t} denote the macroeconomic and firm-specific predictors available for stock i at time t, write R^e_{i,t+1} for the random next-month excess return of stock i and r^e_{i,t+1} for its realization, and define the conditional-mean target

m_{i,t}=\mathbb{E}[R^e_{i,t+1}\mid\mathcal{I}_{i,t}].

A linear characteristic model restricts this target to m_{i,t}=\beta_0+\sum_{k=1}^{K}\beta_k\,\text{characteristic}_{i,t,k}, whereas a neural network uses a flexible function m_{i,t}=f_\theta(\mathcal{I}_{i,t}). The latter permits nonlinearities and interactions that would otherwise have to be chosen manually. The two papers differ in whether that flexibility is disciplined by a prediction loss or by asset-pricing moment restrictions.

Return prediction with a feed-forward network. Gu, Kelly, and Xiu (2020) frame the task as a large-scale supervised prediction problem: a panel regression with a flexible functional form.

Inputs (\mathbf{x}_{i,t}): In the paper’s baseline, 94 firm-specific characteristics for stock i at time t (e.g., size, value, momentum, profitability), which are further interacted with eight aggregate time-series predictors and combined with industry indicators—over 900 baseline signals in total. The illustration below uses the 94 characteristics alone.
Target (r^e_{i,t+1}): The realized excess return of stock i in the next month.
Objective: Train a neural network f_\theta to minimize the mean squared error (MSE) between predicted and realized returns over a large dataset of all U.S. stocks from 1957 to 2016. \min_\theta \frac{1}{N \cdot T} \sum_{i,t} (r^e_{i,t+1} - f_\theta(\mathbf{x}_{i,t}))^2 The display is schematic: the U.S. stock panel is unbalanced, so the actual average runs over the observed stock-month pairs rather than all N \cdot T combinations.

The trained network is a return forecaster: economics enters primarily through the target, predictors, and portfolio evaluation, rather than through a no-arbitrage estimating equation. A natural architecture is a plain feed-forward network for the cross-sectional relationship; Figure 7.2 sketches its shape.

graph LR
    A["Input Layer<br/>(94 Characteristics)"] --> B["Hidden Layer 1<br/>(32 neurons, ReLU)"]
    B --> C["Hidden Layer 2<br/>(16 neurons, ReLU)"]
    C --> D["Hidden Layer 3<br/>(8 neurons, ReLU)"]
    D --> E["Output Layer<br/>(1 neuron, linear)"]

    style A fill:#e6f2ff,stroke:#333
    style B fill:#fff2e6,stroke:#333
    style C fill:#fff2e6,stroke:#333
    style D fill:#fff2e6,stroke:#333
    style E fill:#ffe6e6,stroke:#333

Figure 7.2: Feed-forward architecture for the Gu, Kelly, and Xiu (2020)-style return forecaster, simplified to the 94 firm characteristics (the paper’s full baseline design adds macro interactions and industry indicators). The characteristics enter an input layer, pass through three fully connected hidden layers of decreasing width (32, 16, and 8 units with rectified linear unit (ReLU) activations), and produce a single linear output: the return forecast.

The depth of the network allows it to learn a hierarchy of features. For example, the first layer might learn to combine basic accounting ratios into a value signal, and a deeper layer might then learn to model the interaction between this value signal and momentum signals.

From prediction to a no-arbitrage objective. Chen, Pelger, and Zhu (2024) take a deliberately different route. Rather than forecasting returns, they estimate M_{t+1} directly, and they replace the MSE objective with Euler-equation restrictions that characterize admissible SDFs:

\mathbb{E}\big[M_{t+1}\, R^e_{i,t+1}\, g(\mathcal{I}_t, \mathcal{I}_{i,t})\big] = 0 \quad \text{for all conditioning functions } g,

where R^e_{i,t+1} is the excess return on stock i as above, \mathcal{I}_t and \mathcal{I}_{i,t} collect the macroeconomic and firm-specific information available at time t, and g(\cdot) provides test assets, or instruments. In generalized method of moments (GMM), parameters are chosen to make sample analogues of model-implied moment restrictions as close to zero as possible. Conditional GMM allows those restrictions to be multiplied by functions of information available at time t; here, g creates those functions. Their sample covers U.S. stocks from 1967 to 2016 and includes 46 firm characteristics and 178 macroeconomic time series. The estimation principle is therefore conditional GMM, not least-squares prediction. The authors argue that this distinction carries real weight: in their comparisons, machine-learning models trained on a pure prediction objective can perform worse than linear models that impose no-arbitrage.

A recurrent macro state. Motivated by time-varying risk premia and a persistent macroeconomic state, Chen, Pelger, and Zhu (2024) use an LSTM to process the macroeconomic sequence and compress it into a small hidden state h_t. Feeding all 178 raw macro series into the SDF network each month would treat the latest releases as the complete state, whereas h_t is a learned, low-dimensional summary of macroeconomic history. This state vector enters the SDF network alongside the firm characteristics.

Figure 7.3: Architecture of the SDF model of Chen, Pelger, and Zhu (2024). The sequence of macroeconomic variables is compressed by an LSTM into a low-dimensional macro state vector h_t; this state is concatenated with the firm characteristics and passed through a feed-forward network whose outputs are the SDF portfolio weights \omega.

This architecture allows the model to learn a state-dependent SDF: the discount factor is built from portfolio weights \omega that depend on both firm-level and macro information,

M_{t+1} = 1 - \sum_{i=1}^{N_t} \omega\big(\text{characteristics}_{i,t},\, h_t\big)\, R^e_{i,t+1},

where N_t is the number of stocks observed at time t, the weight function \omega is a feed-forward network, and h_t is the LSTM’s macro state.

To see identification within this restricted class, collect the excess returns in a vector R^e_{t+1} and write M_{t+1}=1-\omega_t^{\top}R^e_{t+1}. The conditional Euler equation implies

\mathbb{E}_t[R^e_{t+1}R_{t+1}^{e\top}]\,\omega_t =\mathbb{E}_t[R^e_{t+1}].

If the conditional second-moment matrix is nonsingular, this equation identifies \omega_t within the affine tradable class. It does not identify a unique unrestricted SDF—in an incomplete market many pricing kernels satisfy the same Euler equations—and the neural-network parameters representing the same weight function need not themselves be unique. The adversarial, regularized empirical criterion selects a solution within the chosen function classes and instrument functions. Exact no-arbitrage also requires a strictly positive SDF; if strict positivity is not imposed, the fitted object is best understood as an empirical pricing-kernel candidate rather than as proof of an everywhere-positive SDF.

Adversarial instruments. The moment condition must hold for all conditioning functions g, which no finite procedure can impose directly. The authors’ solution is a generative adversarial network (GAN) framework: a second network chooses g—in effect, the test assets—to make the pricing errors of the current SDF as large as possible, while the SDF network is trained to shrink them. The adversary thus supplies the hardest-to-price moments in the conditional GMM problem; it is an estimation device for the moment restrictions, not an adjustment to a prediction loss. We do not cover GANs further in this book.

Empirical results. The headline numbers, all from Chen, Pelger, and Zhu (2024), summarize what this combination of economic structure and neural-network flexibility delivers:

Performance: The full model (GAN-SDF) achieves an annual out-of-sample Sharpe ratio of approximately 2.6, compared with about 1.7 for linear models and 0.8 for the Fama-French 5-factor model.
Pricing average returns: The model achieves a cross-sectional pricing R^2 above 90% for each of 46 characteristic sorts, evaluated on the average excess returns of the ten decile portfolios within each sort (460 portfolios in total; Table 3 and Figure 9 in Chen, Pelger, and Zhu (2024)). The statistic measures the fit of average returns across portfolios.
Main lesson: In the paper’s sample and benchmark design, combining economic structure (no-arbitrage moment conditions, a time-varying macro state) with machine-learning flexibility (LSTMs, adversarial estimation) performs better than the linear and prediction-only machine-learning alternatives considered. The comparison suggests that the economic restrictions do substantive work in the estimation objective.

The full-model comparison changes several components together: the recurrent macro state, the flexible pricing function, and the adversarial moment selection. Its performance therefore does not isolate the contribution of the LSTM.

Question for Reflection

Suppose the LSTM state h_t were replaced by the latest observation of each of the 178 macroeconomic series, with no additional lags or historical summaries. What information could this remove from the pricing model? Does the full model’s reported performance by itself identify the gain from using an LSTM?

7.8 From Recurrent States to Attention

The LSTM cell state C_t is a fixed-dimensional vector that summarizes the relevant past at each step. Its width H is chosen once, before training, and does not change with the length of the input sequence. Everything a plain LSTM uses at step t to forecast must pass through this bottleneck.

This design is efficient, but it has a structural cost. If the predictively relevant past at step t requires distinguishing fine information from many earlier positions, an H-dimensional vector has to compress all of it into the same width. For moderately long sequences or for tasks where position-specific information matters—a monetary-policy statement whose meaning depends on whether a given clause appears in paragraph two or paragraph eight, for instance—the bottleneck can become binding: it binds when the predictively relevant past does not admit an adequate H-dimensional summary.

Attention mechanisms use a different construction: instead of forcing all relevant history through a single recurrent state, they form position-specific weighted combinations of representations from earlier tokens or time points. The model can therefore access multiple earlier representations directly, although the attention weights and positional encodings—numerical features that tell the model where each token or time point occurs in the sequence—introduce their own modeling choices. A formal treatment lies beyond the scope of this chapter.

Where the implementation lives. As in the RNN chapter, we have not fit an LSTM in code here. The empirical chapter on networks for time series trains an LSTM on realized-volatility data with the complete Python pipeline—sequence construction, chronological splits, training-only scaling, tuning, and evaluation against a heterogeneous autoregressive (HAR) benchmark.

7.9 Summary

Key Takeaways

An LSTM maintains a cell state alongside its hidden output to represent persistent information.
Forget, input and output gates control what information is retained, added and revealed.
The additive cell-state path can preserve gradients when the relevant forget gates remain close to one, with retention depending on both the gate values and the horizon.
The additional gates require more parameters and computation than a plain RNN.

Common Pitfalls

The direct cell-state path does not guarantee that the full network learns long-range dependence.
A learned cell state is not a directly observed or identified economic state.
Small macroeconomic samples can overfit the added flexibility, so compare against simpler time-series benchmarks.
Random train-validation-test splits can violate the information available at a historical forecast origin.

7.10 Exercises

Exercise 7.1: Manual LSTM Forward Pass

Consider an LSTM cell with a single unit processing one time step. You will compute all gate values, the cell state, and the hidden state step by step.

Given parameters:

Forget gate: W_f = [0.3, 0.2], b_f = 0.1
Input gate: W_i = [0.4, 0.1], b_i = 0.0
Output gate: W_o = [0.2, 0.5], b_o = 0.2
Candidate value: W_c = [0.6, -0.3], b_c = 0.0

Initial conditions: - Current input: x_1 = 0.5 - Previous hidden state: h_0 = 0.3 - Previous cell state: C_0 = 0.4

Parts:

Compute the forward pass. Compute, in order, the forget gate f_1, the input gate i_1, the candidate value \tilde{C}_1, the updated cell state C_1, the output gate o_1, and the hidden state h_1.
Interpret the update. Decompose C_1 into its two terms, f_1 \odot C_0 and i_1 \odot \tilde{C}_1. Which term dominates, and what does this say about how much the cell relies on old memory versus new information at this step? Then solve for the input-gate value i^\ast that would make the two terms equal, holding f_1, C_0, and \tilde{C}_1 fixed, and explain what the result reveals about what a gate can and cannot do.
Translate a gate value into a retention horizon. Holding the gates and candidate fixed, derive \partial C_1/\partial C_0. Now suppose the same forget-gate value f=0.572 applies for d consecutive steps. Derive the retained fraction of an initial perturbation after d steps and solve for its half-life d_{1/2}, defined by f^{d_{1/2}}=1/2. Use \log(1/2)\approx-0.693 and \log(0.572)\approx-0.559. Explain why this calculation alone does not establish that a fitted LSTM has learned economically useful long memory.

Note: Use the sigmoid function \sigma(z) = \frac{1}{1+e^{-z}} for the gates and \tanh for the candidate values and final hidden-state computation. Use the approximations \sigma(0.29) \approx 0.572, \sigma(0.17) \approx 0.542, \sigma(0.51) \approx 0.625, \tanh(0.03) \approx 0.030, and \tanh(0.245) \approx 0.240.

Exam-level exercise: Part 1 is mechanical scaffolding; Part 2 interprets the competing state-update channels; Part 3 connects local sensitivity to a retention horizon and its limits.

Hint for Part 1

Concatenate the previous hidden state and current input: [h_{t-1}, x_t] = [h_0, x_1] = [0.3, 0.5]. Use this same concatenated vector in all four gate or candidate computations in Part 1.

Solution

The LSTM equations are:

\begin{aligned} f_t &= \sigma(W_f \cdot [h_{t-1}, x_t] + b_f) \quad \text{(forget gate)} \\ i_t &= \sigma(W_i \cdot [h_{t-1}, x_t] + b_i) \quad \text{(input gate)} \\ \tilde{C}_t &= \tanh(W_c \cdot [h_{t-1}, x_t] + b_c) \quad \text{(candidate value)} \\ C_t &= f_t \odot C_{t-1} + i_t \odot \tilde{C}_t \quad \text{(cell state)} \\ o_t &= \sigma(W_o \cdot [h_{t-1}, x_t] + b_o) \quad \text{(output gate)} \\ h_t &= o_t \odot \tanh(C_t) \quad \text{(hidden state)} \end{aligned}

where \sigma(z) = \frac{1}{1+e^{-z}} is the sigmoid function.

Given: x_1 = 0.5, h_0 = 0.3, C_0 = 0.4

The concatenated input vector is [h_0, x_1] = [0.3, 0.5].

Part 1: Forward pass

Forget gate.

\begin{aligned} f_1 &= \sigma(W_f \cdot [0.3, 0.5] + b_f) \\ &= \sigma(0.3 \cdot 0.3 + 0.2 \cdot 0.5 + 0.1) \\ &= \sigma(0.09 + 0.1 + 0.1) \\ &= \sigma(0.29) \approx 0.572 \end{aligned}

Input gate.

\begin{aligned} i_1 &= \sigma(W_i \cdot [0.3, 0.5] + b_i) \\ &= \sigma(0.4 \cdot 0.3 + 0.1 \cdot 0.5 + 0.0) \\ &= \sigma(0.12 + 0.05) \\ &= \sigma(0.17) \approx 0.542 \end{aligned}

Candidate value.

\begin{aligned} \tilde{C}_1 &= \tanh(W_c \cdot [0.3, 0.5] + b_c) \\ &= \tanh(0.6 \cdot 0.3 + (-0.3) \cdot 0.5 + 0.0) \\ &= \tanh(0.18 - 0.15) \\ &= \tanh(0.03) \approx 0.030 \end{aligned}

Cell state update.

\begin{aligned} C_1 &= f_1 \odot C_0 + i_1 \odot \tilde{C}_1 \\ &= 0.572 \cdot 0.4 + 0.542 \cdot 0.030 \\ &= 0.229 + 0.016 \\ &= 0.245 \end{aligned}

Output gate.

\begin{aligned} o_1 &= \sigma(W_o \cdot [0.3, 0.5] + b_o) \\ &= \sigma(0.2 \cdot 0.3 + 0.5 \cdot 0.5 + 0.2) \\ &= \sigma(0.06 + 0.25 + 0.2) \\ &= \sigma(0.51) \approx 0.625 \end{aligned}

Hidden state.

\begin{aligned} h_1 &= o_1 \odot \tanh(C_1) \\ &= 0.625 \cdot \tanh(0.245) \\ &= 0.625 \cdot 0.240 \\ &\approx 0.150 \end{aligned}

Final results for time step 1:

Forget gate: f_1 \approx 0.572
Input gate: i_1 \approx 0.542
Candidate value: \tilde{C}_1 \approx 0.030
Cell state: C_1 \approx 0.245
Output gate: o_1 \approx 0.625
Hidden state: h_1 \approx 0.150

Part 2: Old memory versus new information

The two terms of the update are f_1 \odot C_0 = 0.572 \times 0.4 \approx 0.229 and i_1 \odot \tilde{C}_1 = 0.542 \times 0.030 \approx 0.016. The memory term dominates by an order of magnitude: at this step the cell mostly carries forward old information and adds little from the current input. The reason is not the input gate—it is moderately open (i_1 \approx 0.542)—but the candidate itself: its pre-activation is only 0.03, and since \tanh(z) \approx z near zero, the candidate \tilde{C}_1 \approx 0.030 is itself close to zero, so there is little new content to add, however open the gate. The moderately open forget gate (f_1 \approx 0.572) passes on roughly half of the previous cell state.

For the two terms to be equal with f_1, C_0, and \tilde{C}_1 held fixed, the input gate would need i^\ast = 0.229 / 0.030 \approx 7.6—far outside the sigmoid’s range (0,1). No admissible input-gate value can make the new information dominate under this counterfactual. A gate rescales its channel by a factor between 0 and 1, so it can attenuate a signal but never amplify one. With the current carry-over term fixed, dominance would require a larger candidate magnitude. In the unrestricted cell update, a smaller forget-gate contribution f_1C_0 is the other route by which new information can dominate.

Part 3: Retention horizon

Holding the gates and candidate fixed gives

\frac{\partial C_1}{\partial C_0}=f_1=0.572.

If the same forget gate applies for d steps, the retained fraction is the product f^d=0.572^d. The half-life therefore solves

d_{1/2}=\frac{\log(1/2)}{\log(0.572)} \approx\frac{-0.693}{-0.559}\approx1.24\text{ steps}.

This coordinate would attenuate an initial perturbation by one half after only about 1.24 steps under the constant-gate approximation, so the computed value does not indicate long retention. More generally, one realized gate value does not describe the time-varying products along other observations, and even a near-one gate says nothing about whether the retained state improves forecasts; Exercise 7.2 takes up the validation evidence that such a claim requires.

Further remarks:

Selective output: The output gate (o_1 = 0.625) filters what portion of the cell state becomes the hidden state, demonstrating how LSTMs can store information internally without immediately exposing it.
Computational complexity: Even for a single time step with one unit, LSTMs require computing four different weight-input combinations, highlighting why they are computationally more expensive than plain RNNs.

Exercise 7.2: The Cell-State Gradient Highway

Consider a scalar LSTM cell with state update

C_k = f_k\, C_{k-1} + i_k\, \tilde{C}_k,

where the forget gate f_k \in (0,1), the input gate i_k \in (0,1), and the candidate \tilde{C}_k \in (-1,1) are computed from (h_{k-1}, x_k) as in the chapter. In Parts 1 and 2, use the gates-fixed approximation from the chapter: treat f_k, i_k, and \tilde{C}_k as fixed numbers that do not vary with the earlier cell state.

Derive
$\frac{\partial C_T}{\partial C_t} = \prod_{k=t+1}^{T} f_k$
under the gates-fixed approximation, starting from the one-step derivative \partial C_k / \partial C_{k-1}.
Compare this with the one-dimensional RNN sensitivity from Exercise 6.1, \partial h_T/\partial h_t = \prod_{k=t+1}^{T} W_{hh} \tanh'(z_k), whose magnitude is bounded by |W_{hh}|^{T-t}. Suppose W_{hh} = 0.8 (as in Exercise 6.2) while the LSTM has learned f_k = 0.99 for every k. For a horizon of T-t = 30 steps, evaluate the RNN upper bound and the LSTM cell-path magnitude (use 0.8^{30} \approx 0.0012 and 0.99^{30} \approx 0.74) and explain which two structural features of the cell-state update produce the difference: what replaces the repeated multiplication by W_{hh}, and why does no activation derivative appear in the LSTM product?
Part 1 holds the gates fixed. Identify the channels through which gradients flow in a full backward pass that this approximation ignores, and explain why the claim “LSTMs eliminate vanishing gradients” therefore does not follow from Part 1. Then explain why this limitation makes an out-of-sample comparison against a lagged feed-forward or HAR benchmark necessary before attributing forecast gains to learned long memory.

Exam level. Part 1 is a short chain-rule derivation; Part 2 rewards understanding of what structurally differs between the two recursions; Part 3 tests whether students can locate the limits of the gates-fixed result.

Hint for Part 3

Ask which quantities at step k+1 are computed from h_k, and what h_k itself is computed from.

Solution

Part 1: The gates-fixed derivation

Under the gates-fixed approximation, the only dependence of C_k on C_{k-1} is the explicit linear term:

\frac{\partial C_k}{\partial C_{k-1}} = f_k,

since i_k \tilde{C}_k is treated as a constant. By the chain rule along the cell-state path,

\frac{\partial C_T}{\partial C_t} = \prod_{k=t+1}^{T} \frac{\partial C_k}{\partial C_{k-1}} = \prod_{k=t+1}^{T} f_k.

Part 2: Comparison with the RNN recursion

For the RNN, |\partial h_T/\partial h_t| \le |W_{hh}|^{T-t} because |\tanh'(z)| \le 1; with W_{hh} = 0.8 and T-t = 30, the bound is 0.8^{30} \approx 0.0012—the gradient signal from step t is essentially extinguished. For the LSTM with f_k = 0.99, the cell-state path carries 0.99^{30} \approx 0.74—about three-quarters of the signal survives thirty steps.

Two structural features produce the difference. First, C_{k-1} enters the cell-state update linearly, multiplied only by a learned, input-dependent forget gate in (0,1)—a separate, adaptive retention channel. A scalar RNN recurrent weight can also be set near one, but the same weight must simultaneously implement the state dynamics, so it cannot be tuned for retention alone. Second, the direct cell-state path bypasses the activation function: C_{k-1} is not passed through a \tanh on its way into C_k, so no activation derivative enters this product, whereas every RNN step contributes a factor \tanh'(z_k)\le 1 that attenuates further whenever the unit saturates.

Part 3: What the gates-fixed approximation ignores

In a full backward pass, gradients also flow through channels that Part 1 freezes: the hidden state h_k = o_k \odot \tanh(C_k) feeds the gate computations at step k+1, so f_k, i_k, o_k, and \tilde{C}_k all depend on earlier cell states through h_{k-1}. Those channels involve products of sigmoid and tanh derivatives with weight matrices and can attenuate exactly as in a plain RNN. Every finite forget-gate product over a positive number of steps is below one, and the amount of attenuation depends on the gate values and the horizon. The direct cell-state path avoids repeated recurrent-matrix and activation-derivative factors; its attenuation is controlled by forget gates whose values still depend on learned weights. The result describes a mitigation of vanishing gradients along one path and does not establish their elimination throughout the network.

Practical implication: do not assume the architecture alone guarantees that long-run dependence is learned. Long-horizon LSTM forecasts still require enough data and out-of-sample validation evidence before performance is attributed to learned long memory. A comparison against a lagged feed-forward or HAR-type benchmark, as in the empirical chapter, is the natural test: these benchmarks use a fixed lag window by construction, so gains over them are evidence that the retained state contributes beyond recent lags.

7.11 References

Chen, Luyang, Markus Pelger, and Jason Zhu. 2024. “Deep Learning in Asset Pricing.” Management Science 70 (2): 714–50. https://doi.org/10.1287/mnsc.2023.4695.

Gers, Felix A., Jürgen Schmidhuber, and Fred Cummins. 2000. “Learning to Forget: Continual Prediction with LSTM.” Neural Computation 12 (10): 2451–71. https://doi.org/10.1162/089976600300015015.

Gu, Shihao, Bryan Kelly, and Dacheng Xiu. 2020. “Empirical asset pricing via machine learning.” Review of Financial Studies 33 (5): 2223–73. https://doi.org/10.1093/rfs/hhaa009.

Hochreiter, Sepp, and Jürgen Schmidhuber. 1997. “Long Short-Term Memory.” Neural Computation 9 (8): 1735–80. https://doi.org/10.1162/neco.1997.9.8.1735.