Predictive Validation, Coordinate Selection, and Oracle Bounds

Conceptual map

1. II-13.01Chronological validation
2. II-13.02Separating selection and evaluation
3. II-13.03Finite-candidate guarantees
4. II-13.04Cross-fitting estimated geometry
5. II-13.05Prediction as a diagnostic

1. Finite validation dictionaries admit a uniform risk bound

Held-out prediction can compare a declared finite candidate set. Its guarantee depends on separation between training, selection, and evaluation and on the dependence structure of validation observations.

Let \(\mathcal{F}\) be a fixed set of \(M\) predictors. On an independent validation sample \(Z_{1},\ldots,Z_{n}\), let loss \(\ell(f,Z)\) lie in [0,1], population risk \(R(f)=\mathbb{E}[\ell(f,Z)]\), and empirical risk \(\hat{R}_{n}(f)=n^{-1}\sum \ell(f,Z_{i})\). Let \(\hat{f}\) minimize empirical risk over \(\mathcal{F}\).

2. The generic bound can be conservative

With \(M=10\), \(n=1000\), and \(\delta=0.05\), \(\varepsilon=\sqrt[\operatorname{log}(400)/2000]\approx0.05473\) and the excess-risk term \(2\varepsilon \approx0.10947\). This distribution-free number can exceed empirical differences between candidates. It remains a valid benchmark under the assumptions of Proposition 1; sharper bounds need variance, margin, stability, or complexity information.

Ties in empirical risk can be resolved by a prespecified rule. Every minimizer satisfies (1) on the same uniform event.

3. Time order changes both candidates and guarantees

In an archive or backlog problem, a random split can place later regimes, field definitions, or outcomes in the training set used to predict earlier observations. An out-of-time split trains only on dates available before the validation block. Hyperparameter and coordinate selection must occur inside the training period or a nested validation period.

Serially dependent validation losses do not satisfy the iid Hoeffding step. A blocked concentration result, martingale argument, or effective-sample analysis must replace it. The chronological gap between training and validation can reduce short-range dependence and also changes the target deployment date.

4. Cross-fitting controls specific reuse paths

If candidate functions are trained on data independent of the validation fold, Proposition 1 applies conditionally on the training data for a finite candidate list. If the same validation outcomes select coordinates, estimate geometry, tune hyperparameters, and report final loss, the candidate set is data dependent and the proof no longer applies directly.

Baseline choice is part of the claim. Report absolute loss, improvement relative to a named baseline, uncertainty in their paired difference, and all dates. At zero validation size the risk estimate is undefined; with a single observation its sampling uncertainty cannot be learned from the fold alone.

5. Implementation, exercises, and sources

Freeze the candidate dictionary and split dates before evaluation. Store preprocessing fit dates, coordinate selection, hyperparameters, baseline, loss units, fold sizes, dependence gap, and predictions. Use paired loss differences on identical validation observations and report uncertainty matched to their dependence.

Exercises

1. Reproduce the numerical bound in Section 2.
2. Draw a nested chronological split for training, tuning, and final evaluation.
3. Construct a leakage example in which a future-coded field improves random-split loss and fails out of time.

1. Vladimir Vapnik, Statistical Learning Theory.Empirical risk and finite-class uniform control.
2. Peter Bühlmann and Sara van de Geer, Statistics for High-Dimensional Data, chapters on oracle inequalities.Selection bounds and nuisance control.
3. Rob J. Hyndman and George Athanasopoulos, Forecasting: Principles and Practice, sections on time-series cross-validation.Chronological forecast evaluation.