Data Sets Used in This Book

This reference appendix documents the file-backed datasets shipped with the book repository. Earlier chapters link here instead of re-describing the source each time. Synthetic datasets generated inside code chunks are not listed here; they are documented where they appear.

FRED-QD

The Federal Reserve Economic Data Quarterly Database (FRED-QD) is the quarterly U.S. macroeconomic database developed by McCracken and Ng (2020). The Federal Reserve Bank of St. Louis maintains the release files, documentation, and historical vintages. A repository copy of the release used in these notes is stored at data/fred_qd_current.csv.

  • Unit of observation: calendar quarter
  • Coverage: U.S. macro-financial series beginning in 1959Q1. In this repository snapshot, dated rows extend through 2025Q3, but availability differs by series; GDPC1 is populated through 2025Q2.
  • Frequency: quarterly
  • File layout: the CSV contains two metadata rows before the actual data. The first row stores factor-group identifiers and the second row stores the authors’ suggested transformation codes. In this file, sasdate is the first day of the quarter’s final month: for example, 12/1/1999 denotes 1999Q4. It is a quarter label, not a literal release date.

The chapters in the book use the subset of series listed below. Full variable documentation is available in the FRED-QD release notes.

Mnemonic Description
GDPC1 Real gross domestic product, chained dollars
CPIAUCSL Consumer price index, all urban consumers
UNRATE Civilian unemployment rate
FEDFUNDS Effective federal funds rate
GS10 10-year Treasury constant-maturity yield
TB3MS 3-month Treasury bill secondary-market rate
VIXCLSx CBOE volatility index (VIX), quarterly average

Figure 1.1 shows the series used in the book on a common standardized scale so that their timing can be compared across the sample. Magnitudes are not directly comparable on this scale.

Show the code 
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

raw = pd.read_csv("data/fred_qd_current.csv")
fred_qd = raw.iloc[2:].copy()
fred_qd["sasdate"] = pd.to_datetime(fred_qd["sasdate"])

for column in fred_qd.columns:
    if column != "sasdate":
        fred_qd[column] = pd.to_numeric(fred_qd[column], errors="coerce")

gdp_growth = 400 * np.log(fred_qd["GDPC1"]).diff()
inflation = 400 * np.log(fred_qd["CPIAUCSL"]).diff()
term_spread = fred_qd["GS10"] - fred_qd["TB3MS"]
log_vix = np.log(fred_qd["VIXCLSx"])

plot_df = pd.DataFrame(
    {
        "date": fred_qd["sasdate"],
        "GDP growth": gdp_growth,
        "Inflation": inflation,
        "Unemployment": fred_qd["UNRATE"],
        "Fed funds": fred_qd["FEDFUNDS"],
        "Term spread": term_spread,
        "Log VIX": log_vix,
    }
).dropna()

standardized = plot_df.copy()
for column in standardized.columns[1:]:
    standardized[column] = (
        standardized[column] - standardized[column].mean()
    ) / standardized[column].std()

fig, ax = plt.subplots(figsize=(11.4, 4.8))
for column in standardized.columns[1:]:
    ax.plot(standardized["date"], standardized[column], linewidth=1.5, label=column)

ax.set_xlabel("Quarter")
ax.set_ylabel("Standardized value")
ax.grid(True, alpha=0.2)
ax.legend(frameon=False, ncol=3, loc="upper left")
plt.tight_layout()
plt.show()
Figure 1.1: Selected FRED-QD series used in the book, standardized to mean zero and variance one. Real GDP growth is the annualized log difference of GDPC1; inflation is the annualized log difference of CPIAUCSL; the remaining series are shown as levels, with the term spread constructed as GS10 - TB3MS and log VIX as log(VIXCLSx).

Loading FRED-QD. The snippet below is the canonical way to load FRED-QD in this book. It skips the two metadata rows, parses sasdate as a date, and coerces the remaining columns to numeric.

Show the code 
import pandas as pd

raw = pd.read_csv("data/fred_qd_current.csv")
fred_qd = raw.iloc[2:].copy()
fred_qd["sasdate"] = pd.to_datetime(fred_qd["sasdate"])

for column in fred_qd.columns:
    if column != "sasdate":
        fred_qd[column] = pd.to_numeric(fred_qd[column], errors="coerce")

fred_qd = fred_qd.sort_values("sasdate").reset_index(drop=True)

The authors’ transformation codes in the second metadata row are not applied here. Individual chapters construct the transformations they need directly from the raw levels so that the forecast origin and information set are explicit.

Taiwan Credit-Card Default

The Taiwan credit-card default data contain 30,000 clients and were introduced by Yeh and Lien (2009). The original file is distributed by the UCI Machine Learning Repository under a Creative Commons Attribution 4.0 International license (Yeh 2009). The repository copy is stored at data/credit_card_default_taiwan.csv. It removes the two spreadsheet header rows, gives the columns descriptive names, and otherwise preserves the original integer values.

  • Unit of observation: credit-card client
  • Predictor window: April through September 2005
  • Target: default payment in the following month, coded as 1 for default and 0 otherwise
  • Sample size: 30,000 clients, of whom 6,636 defaulted
  • Currency: New Taiwan dollars (NTD)
  • File layout: id is a row identifier, not a predictor. The remaining columns contain 23 predictors and the binary target.

The variables used in the book are grouped below. A month placeholder such as {month} runs from apr through sep.

Variable Description
credit_limit_ntd Granted credit limit, including supplementary family credit
sex_code Original demographic code: 1 for male and 2 for female
education_code Original education code: 1 graduate school, 2 university, 3 high school, and 4 other; the raw data also contain undocumented codes 0, 5, and 6
marriage_code Original marital-status code: 1 married, 2 single, and 3 other; the raw data also contain undocumented code 0
age_years Age in years
repay_status_{month} Monthly repayment-status code. UCI documents -1 as paid duly and positive values as months of delay; the raw data also contain undocumented codes -2 and 0
bill_amount_{month}_ntd Monthly bill statement amount in NTD
payment_amount_{month}_ntd Amount paid in the month in NTD
default_next_month Binary target: 1 for default in the following month and 0 otherwise

UCI reports no missing values, but that statement should not be confused with complete coding documentation. The undocumented category values listed above remain in the repository copy and should be treated as labels of unknown meaning rather than silently assigned an economic interpretation. The client identifier should be excluded before fitting a prediction model.

Loading the credit-default data. The canonical loading snippet separates the target and removes the identifier. Individual chapters specify their own training-only transformations for the categorical and continuous predictors.

Show the code 
import pandas as pd

credit_default = pd.read_csv("data/credit_card_default_taiwan.csv")
y = credit_default["default_next_month"]
X = credit_default.drop(columns=["id", "default_next_month"])

SPY TAQ 5-Minute Realized Measures

The repository also contains scripts for constructing realized-volatility measures from the Wharton Research Data Services (WRDS) Trade and Quote (TAQ) database for the SPDR S&P 500 exchange-traded fund (SPY). The raw TAQ data and derived realized-measure files are not shipped with the public book because TAQ access is licensed through WRDS. The expected local paths are data/taq_spy/SPY_daily_measures.csv for the daily CSV used in the neural-network illustration and data/taq_spy/SPY_5min_daily_rv.parquet for the Parquet output produced by the 5-minute compilation script. The whole data/taq_spy/ directory is excluded from git.

  • Unit of observation: trading day
  • Underlying data: WRDS TAQ millisecond trades, queried from taqmsec.ctm_YYYYMMDD
  • Instrument: SPY
  • Frequency used for realized measures: 5-minute intraday prices
  • Coverage of the local examples: Both SPY_daily_measures.csv and SPY_5min_daily_rv.parquet run from 2015-01-02 through 2024-12-31.
  • Construction scripts: scripts/download_spy_taq_wrds.R downloads and cleans TAQ trades and writes both the per-day 5-minute bar files and the daily CSV; scripts/compile_spy_5min_rv_parquet.R compiles daily realized measures from the local 5-minute bar files. The repair script scripts/repair_spy_daily_measures_from_bars.R can reconstruct daily CSV rows from existing bar files without a new WRDS query.

The two local files have different column sets.

Daily CSV (SPY_daily_measures.csv)—used by the neural-network illustration chapter:

Variable Description
date Trading day
rv Realized variance, the sum of squared 5-minute log returns after multiplying returns by 100; its units are squared percentage points

Parquet file (SPY_5min_daily_rv.parquet)—output of the 5-minute compilation script:

Variable Description
date Trading day
rv_5min Realized variance, the sum of squared 5-minute log returns after multiplying returns by 100; its units are squared percentage points
bv_5min Bipower variation, computed from adjacent absolute 5-minute returns with the standard \pi/2 scaling, on the same squared-percentage-point scale as rv_5min
rq_5min Realized quarticity proxy based on fourth powers of 5-minute returns, a fourth-moment (percent-to-the-fourth) quantity on the corresponding scale
intraday_return Open-to-close log return in percent

The daily CSV used in the neural-network chapters can be loaded in Python as follows:

Show the code 
import pandas as pd

spy_daily = (
    pd.read_csv("data/taq_spy/SPY_daily_measures.csv", parse_dates=["date"])
    .sort_values("date")
    .reset_index(drop=True)
)

The richer Parquet file can be loaded in R as follows:

library(arrow)

spy_rv <- read_parquet("data/taq_spy/SPY_5min_daily_rv.parquet")

References

McCracken, Michael W., and Serena Ng. 2020. “FRED-QD: A Quarterly Database for Macroeconomic Research.” Working Paper 26872. National Bureau of Economic Research. https://doi.org/10.3386/w26872.
Yeh, I-Cheng. 2009. “Default of Credit Card Clients.” UCI Machine Learning Repository. https://doi.org/10.24432/C55S3H.
Yeh, I-Cheng, and Che-hui Lien. 2009. “The Comparisons of Data Mining Techniques for the Predictive Accuracy of Probability of Default of Credit Card Clients.” Expert Systems with Applications 36 (2): 2473–80. https://doi.org/10.1016/j.eswa.2007.12.020.