Eigenvalues, SVD, and High-Dimensional Vector Spaces in Machine Learning

Eigenvalues, SVD, and High-Dimensional Vector Spaces in Machine Learning

Modern artificial intelligence relies fundamentally on linear transformations over high-dimensional vector spaces. From token embeddings in Transformer models and diffusion latent spaces to low-rank fine-tuning (LoRA), spectral matrix decomposition provides the mathematical framework for understanding dimensionality reduction, attention projections, and representation geometry.

1. Spectral Theorem and Orthonormal Bases

Let $A \in \mathbb{R}^{n \times n}$ be a real symmetric matrix ($A = A^T$). According to the Spectral Theorem, there exists an orthogonal matrix $Q \in \mathbb{R}^{n \times n}$ ($Q^T Q = I$) and a real diagonal matrix $\Lambda = \text{diag}(\lambda_1, \lambda_2, \dots, \lambda_n)$ such that:

$A = Q \Lambda Q^T = \sum_{i=1}^{n} \lambda_i q_i q_i^T$

Where:

• ${q_1, q_2, \dots, q_n}$ form an orthonormal basis of $\mathbb{R}^n$ consisting of eigenvectors of $A$.
• ${\lambda_1 \ge \lambda_2 \ge \dots \ge \lambda_n}$ are the corresponding real eigenvalues.

2. Singular Value Decomposition (SVD)

For arbitrary rectangular matrices $X \in \mathbb{R}^{m \times n}$ (such as a document-term matrix or token co-occurrence matrix):

$X = U \Sigma V^T = \sum_{i=1}^{r} \sigma_i u_i v_i^T$

Where:

• $U \in \mathbb{R}^{m \times m}$ is an orthogonal matrix of left singular vectors (eigenvectors of $X X^T$).
• $\Sigma \in \mathbb{R}^{m \times n}$ is a diagonal matrix of non-negative singular values $\sigma_1 \ge \sigma_2 \ge \dots \ge \sigma_r > 0$.
• $V \in \mathbb{R}^{n \times n}$ is an orthogonal matrix of right singular vectors (eigenvectors of $X^T X$).

3. Eckart-Young-Mirsky Low-Rank Approximation

In machine learning compression and parameter-efficient fine-tuning (such as LoRA), we seek the optimal rank-$k$ approximation matrix $\tilde{X}_k$ ($k < r$) that minimizes reconstruction loss under the Frobenius norm:

$\min_{\text{rank}(B)=k} \|X - B\|_F = \|X - \tilde{X}_k\|_F = \sqrt{\sum_{i=k+1}^{\min(m,n)} \sigma_i^2}$

pythonCopy
import numpy as np

def compute_low_rank_approximation(X: np.ndarray, target_rank: int):
    U, S, Vt = np.linalg.svd(X, full_matrices=False)
    U_k = U[:, :target_rank]
    S_k = S[:target_rank]
    Vt_k = Vt[:target_rank, :]
    X_k = U_k @ np.diag(S_k) @ Vt_k
    error = np.linalg.norm(X - X_k, 'fro')
    return X_k, error

4. References & Academic Citations

1. Eckart, C., & Young, G. (1936). The approximation of one matrix by another of lower rank. Psychometrika, 1(3), 211–218.
2. Strang, G. (2019). Linear Algebra and Learning from Data. Wellesley-Cambridge Press.
3. Hu, E. J., et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685.