Representation Budgets, Singular Values, and Approximation Loss

Conceptual map

I-05.01Two loss functions
I-05.02Linear approximation frontiers
I-05.03Continuous nonlinear encoders
I-05.04Width lower bounds
I-05.05Finite-perturbation error

1. A rank budget approximates a target derivative

When exact representation exceeds the available dimension, a loss function determines which response directions should be retained. Operator loss and average squared loss produce related spectral solutions with different numerical summaries.

Let $D\in \mathbb{R}^{r\times n}$ be a target derivative after the state and target metrics have been whitened. A rank- $$q$$ representation followed by a linear decoder produces an approximation $$L$$ with $\operatorname{rank}(L)\le q$ . Write the singular value decomposition $D=U\operatorname{diag}(\sigma _{1},\ldots,\sigma _{p})V^{\top}$ , with $\sigma _{1}\ge\cdots\ge\sigma _{p}\ge0$ .

Proposition 1 · Two optimal spectral truncations

For $0\le q<p$ , the truncated SVD $D_{q}$ minimizes both the spectral and Frobenius errors:

\[\operatorname{min}_{\operatorname{rank}(L)\le q}\lVert D-L\rVert _{2}=\sigma _{q+1}, \quad \operatorname{min} \lVert D-L\rVert _{F}^{2}=\sum _{j>q}\sigma _{j}^{2}.\](1)

Proof sketch. $D_{q}$ attains the displayed errors by orthogonality of the singular components. Any rank- $$q$$ map has a null direction in the span of the first $$q+1$$ right singular vectors; evaluating the error on a unit vector in that null direction gives the spectral lower bound $\sigma _{q+1}$ . The Frobenius result follows by the same orthogonal decomposition and the variational characterization of singular values. ∎

2. Norms encode different experiments

The spectral norm controls the largest response error over unit state directions. If a whitened perturbation $$v$$ has mean zero and covariance $$I$$ , then $\mathbb{E}[\lVert(D-L)v\rVert^{2}]=\lVert D-L\rVert_{F}^{2}$ . With covariance $\Sigma$ , the corresponding matrix is $D\Sigma^{1/2}$ , and a target weight enters by premultiplication.

For $D=\operatorname{diag}(4,2,1)$ , a one-dimensional representation has worst-direction error 2 and isotropic mean-square error $2^{2}+1^{2}=5$ . A two-dimensional representation has both errors equal to one. Reporting “the approximation error” without its norm leaves the claim incomplete.

If $\sigma _{q}=\sigma _{q+1}$ , the optimal error remains unique while the retained singular subspace can be nonunique inside the tied singular-value space. A numerical implementation should report the tied cluster and compare subspaces through projections.

3. A continuous nonlinear encoder still faces a dimension bound

Proposition 2 · Antipodal worst-case lower bound

Let the state set contain the radius- $\rho$ sphere $S_{\rho}^{d-1}$ in $\mathbb{R}^{d}$ , let the target be the state itself, and let $z:S_{\rho}^{d-1}\to \mathbb{R}^{q}$ be continuous with $$q<d$$ . For every decoder $\delta$ ,

\[\operatorname{sup}_{\lVert x\rVert =\rho } \lVert x-\delta (z(x))\rVert \ge \rho .\](2)

Proof. Borsuk–Ulam gives a unit vector $$v$$ with $z(\rho v)=z(-\rho v)$ . Let $$a$$ be the common decoded value. The triangle inequality gives $2\rho=\lVert \rho v-(-\rho v)\rVert \le \lVert \rho v-a\rVert+\lVert -\rho v-a\rVert$ , so at least one error is at least $\rho$ . ∎

Continuity and the spherical state set carry the result. Discontinuous encodings or a finite state set require a different lower-bound argument.

4. From a derivative spectrum to a finite radius

Suppose $\tau$ is twice differentiable and its Hessian operator norm is bounded by $$M$$ on a ball of radius $\rho$ around $x_{0}$ . Taylor's theorem gives

\[\lVert \tau (x_{0}+h)-\tau (x_{0})-D\tau _{x_{0}}h\rVert \le (M/2)\lVert h\rVert ^{2}.\](3)

Combining a rank- $$q$$ approximation to $D\tau_{x_{0}}$ with (3) gives a worst-case upper bound $\sigma _{q+1}\rho+(M/2)\rho ^{2}$ . In the diagonal example with $$q=1$$ , $\rho=0.1$ , and $$M=3$$ , this bound is 0.215.

Failure case · Local singular values reported as global decoder risk

Singular values describe the linearized target at one reference state under declared metrics. A finite-radius risk statement needs Taylor control, a state distribution, or a separate global argument.

5. Implementation, exercises, and sources

Whiten state and target metrics before computing the SVD, report the discarded spectrum, and verify approximation error under the same norm. Estimate or bound Hessian remainders on the actual perturbation region.

Download the volume verification script →

Exercises

Reproduce both error criteria for diag(4,2,1) at ranks zero through three.
Prove (2) for $$d=2$$ and $$q=1$$ directly from the intermediate value theorem.
Derive the covariance-weighted truncation problem for a positive-definite $\Sigma$ .

Partial solutions

1. Spectral errors are 4,2,1,0 and squared Frobenius errors are 21,5,1,0. 3. Write $v=\Sigma^{1/2}u$ with $\operatorname{Cov}(u)=I$ . The mean-square objective becomes the Frobenius approximation of $D\Sigma^{1/2}$ .

C. Eckart and G. Young (1936), “The Approximation of One Matrix by Another of Lower Rank,” Psychometrika 1, 211–218.Best low-rank approximation.
Rajendra Bhatia, Matrix Analysis, Chapter III.Singular-value variational principles.
Jiří Matoušek, Using the Borsuk–Ulam Theorem, Chapter 2.Antipodal coincidence theorem.

6. Audit checkpoint

State the state and target metrics, rank budget, loss norm, covariance weighting, discarded spectrum, nonlinear continuity assumptions, radius, Hessian bound, and whether the result is local or global.

7. Scope boundary

The chapter covers linear spectral approximation, one topological lower bound, and second-order finite-radius control. Nonlinear width theory and statistical selection of rank require additional results.

Prerequisites

I-02 · Linear Maps, Kernels, Quotients, and Information Dimension