Suppose N=qK+r with integers q\geq1 and 0<r<K. A K-fold implementation therefore creates r validation folds of size q+1 and K-r validation folds of size q. Let
\bar L_k=\frac{1}{|D_k|}\sum_{i\in D_k} L\big(y_i,\hat f^{-k}(x_i)\big)
be the mean loss in fold k. Write \bar L_{+} for the average of \bar L_k over the r larger folds and \bar L_{-} for the average over the K-r smaller folds.
- Express the pooled per-observation estimator \widehat{\mathrm{CV}}_K and the unweighted mean of fold means \widetilde{\mathrm{CV}}_K in terms of q, r, K, \bar L_{+}, and \bar L_{-}.
- Show that \widehat{\mathrm{CV}}_K-\widetilde{\mathrm{CV}}_K =\frac{r(K-r)}{NK}\big(\bar L_{+}-\bar L_{-}\big). Determine which estimator is larger when the larger folds have the higher average loss, and explain why the pooled estimator gives the correct per-observation weighting.
- Now suppose the observations are i.i.d. and r=0. Let \hat f_m denote the output of the fitting procedure trained on an independent sample of size m. Derive the risk targeted by \widehat{\mathrm{CV}}_K in expectation, and contrast it with the conditional risk R(\hat f_N) of the model refitted on all N observations. Finally, state how the target changes when r>0.
Exam level. The exercise separates an implementation detail — weighting unequal folds — from the estimand question created by fitting each fold model on fewer than N observations.
Use N=qK+r to simplify the coefficient on each of the two group means.
Condition on a fold’s training sample first. Its held-out losses average to that fitted model’s risk; then average over training samples of the relevant size.
Part 1: The Two Weighting Rules
The pooled estimator weights each fold mean by its share of the N held-out observations:
\widehat{\mathrm{CV}}_K =\frac{r(q+1)}{N}\bar L_{+} +\frac{(K-r)q}{N}\bar L_{-}.
The unweighted mean gives every fold weight 1/K, so
\widetilde{\mathrm{CV}}_K =\frac{r}{K}\bar L_{+} +\frac{K-r}{K}\bar L_{-}.
Part 2: Difference and Interpretation
Subtracting the second expression from the first, the coefficient on \bar L_{+} is
r\left(\frac{q+1}{N}-\frac{1}{K}\right) =\frac{r(K-r)}{NK},
because N=qK+r. The coefficient on \bar L_{-} is
(K-r)\left(\frac{q}{N}-\frac{1}{K}\right) =-\frac{r(K-r)}{NK}.
Therefore
\widehat{\mathrm{CV}}_K-\widetilde{\mathrm{CV}}_K =\frac{r(K-r)}{NK}\big(\bar L_{+}-\bar L_{-}\big).
If \bar L_{+}>\bar L_{-}, the pooled estimator is larger. This direction is appropriate because each larger fold contains one more held-out observation: its mean must receive proportionally more weight if the target is the average loss over all N observations.
Part 3: The Training-Size Estimand
When r=0, every validation fold has size N/K and every fold model is trained on
m=N-\frac{N}{K}=\frac{N(K-1)}{K}
observations. Conditional on a fold’s fitted model, the expected held-out loss is R(\hat f_m). Averaging first over held-out observations and then over training samples gives
\mathbb E[\widehat{\mathrm{CV}}_K] =\mathbb E\big[R(\hat f_{N(K-1)/K})\big].
The right-hand side is the expected risk of the procedure at the fold-induced training size. It is not the conditional risk R(\hat f_N) of the particular full-sample fit, which conditions on the realized model and uses N training observations.
When r>0, a larger validation fold leaves N-q-1 training observations and a smaller validation fold leaves N-q. The pooled estimator targets the corresponding observation-weighted mixture of the procedure risks at these two training sizes.