Generated Responses, Estimated Operators, and Noisy Derivatives

Conceptual map

II-09.01Same-sample covariance
II-09.02Stable bridges
II-09.03Limits of pointwise derivatives
II-09.04Bias in noisy Gram matrices
II-09.05Moving coordinates and frames

1. Squaring an estimated derivative adds noise energy

A generated derivative carries first-stage variance into every Gram matrix, singular value, and bridge. Independent splits can remove a noise-square expectation while leaving sampling variance and range instability.

Let $D\in \mathbb{R}^{r\times p}$ be a fixed population derivative. Two estimates satisfy $\hat{D}_{1}=D+E_{1}$ and $\hat{D}_{2}=D+E_{2}$ , conditional on a training object. Assume E[ $E_{j}$ ]=0, finite second moments, and independence between the two errors.

Proposition 1 · Naive and split Gram expectations

The naive Gram has positive-semidefinite noise-square bias, while a symmetrized split Gram removes it:

\[\begin{aligned}E[\hat{D}_{1}^{\top }\hat{D}_{1}]=D^{\top }D+E[E_{1}^{\top }E_{1}], \\ E[(\hat{D}_{1}^{\top }\hat{D}_{2}+\hat{D}_{2}^{\top }\hat{D}_{1})/2]=D^{\top }D.\end{aligned}\](1)

Proof. Expand both products. Mean-zero errors remove the linear terms. The naive square retains E[ $E_{1}^{\top}E_{1}$ ]. Independence makes E[ $E_{1}^{\top}E_{2}$ ]=0. ∎

2. A two-coordinate example isolates the bias

Let $$D$$ =(1,2) and let each independent error equal $$v$$ =(1, $$-1$$ ) or $$-v$$ with equal probability. Then

\[\begin{aligned}D^{\top }D=\begin{bmatrix}1 & 2 \\ 2 & 4\end{bmatrix}, \\ E[\hat{D}^{\top }\hat{D}]=\begin{bmatrix}2 & 1 \\ 1 & 5\end{bmatrix}, \quad E[G_{split}]=\begin{bmatrix}1 & 2 \\ 2 & 4\end{bmatrix}.\end{aligned}\](2)

The split matrix can be indefinite in one realization even though its expectation is positive semidefinite. Projecting it onto the PSD cone changes its finite-sample expectation. That projection is a separate regularization step.

3. Exact range membership can be discontinuous

Let $A(\eta)$ =(1, $\eta ^{2}$ ) map $\mathbb{R}^{2}$ to $\mathbb{R}$ and consider query $c=(1,0)^{\top}$ . At $\eta=0$ , $$c$$ lies in $\operatorname{range}(A(0)^{\top})$ . For every $\eta \ne0$ , it lies outside the one-dimensional span of $(1,\eta ^{2})^{\top}$ . The distance to that span is $|\eta|^{2}/\sqrt{1+\eta ^{4}}$ , which tends to zero while exact membership fails throughout the punctured neighborhood.

A small range residual therefore supports approximate recovery under a declared latent norm. It does not establish a locally stable exact bridge. Stable exact membership needs a constant-rank or invariant-range condition plus compatible movement of the query.

4. Splitting addresses only declared dependence terms

Failure case · Cross-fitting treated as range stability

Independent folds remove the cross-error expectation in (1). They leave the realized matrix noisy and do not keep a query inside an estimated adjoint range as the nuisance changes.

If both derivative estimates use overlapping observations, their cross covariance appears in the split expectation. If a spectrum is expressed in an estimated tangent frame, frame estimation contributes an additional rotation term. Repeated singular values make individual singular vectors unidentified; projection onto the full tied subspace is the stable object.

5. Implementation, exercises, and sources

Record fold membership and overlap, compute naive and symmetrized split matrices, report asymmetry before symmetrization, PSD eigenvalues, any cone projection, and its size. For bridges, track rank, range residual, bridge norm, and residual sensitivity along nuisance perturbations.

Download the volume verification script →

Exercises

Reproduce every matrix in (2).
Enumerate the four independent error-sign pairs and average their split matrices.
Derive the distance from $(1,0)^{\top}$ to $\operatorname{span}\{(1,\eta ^{2})^{\top}\}$ .

Partial solutions

1. Add $v^{\top}v=\begin{bmatrix}1 & -1 \\ -1 & 1\end{bmatrix}$ to the true Gram. 2. The cross-error signs average to zero, leaving the true Gram. 3. Square the query norm minus its squared projection: $1-1/(1+\eta ^{4})=\eta ^{4}/(1+\eta ^{4})$ .

A. W. van der Vaart, Asymptotic Statistics, chapters on the delta method and estimated functions.Propagation of first-stage error.
Rajendra Bhatia, Matrix Analysis, chapters on matrix perturbation.Generated Gram matrices and spectral maps.
III-12 · Ill-Posed Inverse Problems and Regularization.Range conditions and approximate query recovery.

6. Audit checkpoint

State the population derivative, conditioning object, fold construction, error means and cross covariance, Gram orientation, PSD projection, nuisance path, rank and gap boundaries, frame convention, range residual, bridge norm, and latent norm for approximation.

7. Scope boundary

The chapter covers finite-dimensional generated Gram matrices and one range-instability example. General debiased learning and random-operator limits require additional conditions.