Riemann GeoResolver: A Non-Euclidean Attention Framework from Euclidean Resolver to Hyperbolic-Spherical Geometry

Author: Liangchen Ge

arXiv:2608.10416v1 [cs.DS] 11 Aug 2026

This paper presents a theoretical foundation for inverse-distance attention, from its Euclidean prototype (Resolver) to its non-Euclidean realization (Riemann GeoResolver). The work is explicitly a theoretical study: The primary contribution is the set of theorems, lemmas, and architectural principles for inverse-distance attention, first in Euclidean space and then extended to hyperbolic and spherical geometries.

The paper establishes a theoretical arc: from Euclidean attention as a special case, to hyperbolic memory, to spherical retrieval.

The paper motivates inverse-distance attention by contrasting it with softmax attention. The key observation is that The softmax attention mechanism has a well-known property: even when a query exactly matches a key, the output is a weighted average of all values, not a hard retrieval of the matched value. This is intrinsic to the softmax function, which always assigns positive probability to all tokens.

The inverse-distance kernel addresses this by assigning weights inversely proportional to squared distance, which, in the limit ε → 0+, converges to a one-hot selection of the exact match.

The paper identifies three fundamental implications:

1. Expressiveness: The inverse-distance kernel, by contrast, has gradients that remain large near exact matches and a Hessian that is full-rank, suggesting better optimization properties.

2. Optimization: The softmax function, while differentiable and probabilistically interpretable, creates optimization challenges: the gradient is small when the logits are large (saturation), and the Hessian is low-rank when n is large.

3. Generalization: "The softmax's capacity to represent arbitrary functions grows with width, leading to a phenomenon where, once the hidden dimension exceeds the number of training points, the model can memorize arbitrary labels, including noise. This is a form of overparameterization that degrades generalization. The inverse-distance kernel's effective rank is bounded independent of width, preventing this memorization effect."

Statement: There exists an orthogonal-keys instance where IDA achieves exact retrieval with O(1) resources, while any softmax-based architecture requires d = Ω((log n)2) or H = Ω(log n) for ε-approximation.

Key Lemma (Exact Retrieval Property): For pairwise distinct keys and q = k j*, lim ε→0 W j* = 1, lim ε→0 W j = 0 (j ≠ j*).

Proof sketch: The paper constructs an instance with n orthonormal keys e j in R d with k j = Re j and q = k 1 = Re 1. For IDA, the squared distance from query to key j is 0 for j=1 and 2R2 for j≠1, giving W 11 IDA = 1/(1 + ε(n-1)/(2R2+ε)), which approaches 1 as ε→0. For softmax, the weight is W 11 soft = 1/(1 + (n-1)e-R2/√d), requiring d ≥ (R max2/log((n-1)/(2δ)))2 = Ω((log n)2) for δ-approximation.

Additional result (Dense-Key Degradation): If keys have covering radius r from k j, then W jj ≥ 1/(1 + ε(n-1)/(r2+ε)). For Gaussian keys with covariance σ2I, r2 = Θ(d h σ2 n-2/d h).

Proof approach: The Jacobian of the attention weight for IDA involves the derivative of S ij = (‖q i - k j‖2 + ε)-1, which is ∂S ij/∂q i = -2(q i - k j)/(‖q i - k j‖2 + ε)2. The function a/(a2+ε)2 has its maximum at a2 = ε/3, giving O(ε-3/2) per term. For softmax, the derivative is ∂W ij/∂q i = (1/√d)W ij(k j - k̄ i), giving linear n scaling.

Key result: The Hessian spread (ratio of largest to smallest eigenvalue) is Θ(1) for IDA, enabling O(1) escape from saddles, in contrast to softmax's O(n2) escape time.

Corollary: "Since μ IDA > 0, the PL inequality implies that every stationary point of the IDA loss is either a global minimum or a strict saddle (i.e., no spurious local minima). Gradient descent converges linearly as L(t) - L* ≤ e-μt(L(0) - L*)."

Softmax Width Catastrophe: When d h ≥ n, softmax can achieve zero training error on arbitrary labels, with test error → 2η under symmetric noise rate η. The proof constructs orthonormal keys with q i = √d h e i and k j = α√d h e j, showing that as α→∞, softmax attention converges to the identity matrix, allowing memorization of arbitrary labels.

IDA Noise Robustness: For IDA, test error under symmetric noise rate η satisfies E test IDA ≤ Cη2 + O(1/√n), independent of d h.

The Poincaré ball is B d h = x ∈ R d h: ‖x‖ < 1 with conformal factor λ x = 2/(1 - ‖x‖2). The hyperbolic distance is:

d H(x, y) = arcosh(1 + 2‖x - y‖2/((1 - ‖x‖2)(1 - ‖y‖2)))

All embeddings are projected via x = 0.9 tanh(z). The unit sphere is S d-1 = x: ‖x‖ = 1 with great-circle distance d S(x, y) = arccos(⟨x, y⟩).

(A) Information-theoretic capacity: For fixed dimension d, the hyperbolic ball volume V H(R) ∝ e(d-1)R grows exponentially with radius, while Euclidean volume V E(R) ∝ R d grows polynomially.

(B) Effective rank behavior: "Theorem 3 established the bound d min ≥ 2δ/(1 - ρ2); as ρ → 1− (approaching the boundary of the Poincaré ball), this grows without bound. Thus hyperbolic embeddings can achieve arbitrarily large separation between keys without increasing the ambient dimension."

(C) Architectural separation: Proposition 2 establishes an information-theoretic gap between hyperbolic storage and spherical routing.

Lemma 5.1 (Unified Kernel Spectral Properties): For any metric space M with kernel K ij = (d(x i, y j)2 + ε)-1 satisfying (i) d(x i, x i) = 0, (ii) d(x i, y j) ≥ d min > 0 for i ≠ j, (iii) d locally Lipschitz: (a) if x i = y j*, lim ε→0 K ij*/Σ m K im = 1; (b) if ε ≪ d2 min, eff-rank(K) ≤ 1 + nε2/d4 min + O(nε3/d6 min).

M1: Dense-HIDA — The dense hyperbolic inverse-distance attention with complexity Θ(n2d h). Theorem 5.2 establishes circuit separation and rank bound with d min ≥ 2δ/(1 - ρ2). Theorem 5.3 establishes the two-point hyperbolic PL inequality: μ HIDA = Θ(ε2/Δ4 H), μ soft = Θ(e-Δ2 H/√d h ε2/Δ2 H), so μ HIDA/μ soft = Ω(e Δ2 H/√d h/Δ2 H).

M2: FP-HIDA (Fixed-Pattern Sparse) — Uses sparse index sets S i = j: i-j ≤ w ∪ 0, ⌊n/g⌋, ⌊2n/g⌋,... ∪ i ± 2 k ∪ i with w, g = Θ(log n). Theorem 5.4: FP-HIDA requires O(n log n · d h) operations and O(n log n) memory.

M3: L-HIDA (Linear Complexity) — Uses m = Θ(1) learnable anchors with key-anchor and query-anchor weights. Theorem 5.5: L-HIDA requires O(nd h) operations with error bound ‖o L-HIDA - o HIDA‖ F ≤ O(n/(ε2√m)) via Nyström sampling, or O(n·m-α+1/2) + O(n/(ε2√m)) with spectral decay λ k = O(k-α), α > 1/2.

M4: C-HIDA (Constant Complexity) — Maintains c = Θ(1) summary tokens with online hyperbolic k-means updates. Theorem 5.6: The online hyperbolic k-means update has O(c log T) regret: Σ t=1 T min k d H(k t, s k)2 ≤ O(Δ2 H · c · log T) + O(T · ε opt). Per-token attention cost is Θ(1).

For any key k ∈ B d h with polar decomposition k = r·u, given bit-widths b (direction) and b r (radius):

Theorem 6.1 (HCC Reconstruction Error): The reconstruction error satisfies: ‖k - k̂‖22 ≤ 4(2-b + 2-b r).

Theorem 6.2 (HCC Memory Complexity): "The HCC-compressed KV cache requires n((d h - 1)b + b r) bits for keys plus nd h · 32 bits for values. With b = 4, b r = 6, the theoretical compression ratio is ≈ 8× for keys, ≈ 1.8× system-wide (or ≈ 2.4× with half-precision values)."

Three-level gating: head-level κ h, token-level boundary β i, dimension-level G i = diag(σ(W g x i)).

Theorem 7.1 (HyperGate Gradient Lower Bound): ‖∂L/∂x i‖2 ≥ (λ min(G i) - O(‖W g‖‖x i‖)) · ‖∂L/∂h i‖2. In particular, for bounded x i and W g, the lower bound is strictly positive, so gradients do not vanish due to gating alone.

Definition 8.1 (SIDA Kernel): D S ij = d S(q i, k j)2 + ε, W S ij = (D S ij)-1/Σ m(D S im)-1.

Theorem 8.1 (Spherical PL Inequality): "For two keys with spherical angle θ > 0: μ SIDA = Θ(ε2/θ4), μ soft = Θ(e-(1-cos θ) ε2/θ2). Thus μ SIDA/μ soft = Ω(e 1-cos θ/θ2). Unlike the hyperbolic case, spherical compactness ensures θ ≤ π, so the ratio is bounded by a constant depending only on θ."

Cross-Geometric Mapping: Three methods map prototype centroids to the sphere: (1) Norm Normalization: φ A(c) = c/‖c‖; (2) Stereographic Projection: φ B(c) = (2c/(1+‖c‖2), (1-‖c‖2)/(1+‖c‖2)); (3) Learnable Mapping: φ C(c) = MLP θ(c)/‖MLP θ(c)‖.

The prototype pool is P = (E e, c e, t birth e, a e, t last e) e=1 K. Uses sliding-window mean μ t and standard deviation σ t with adaptive threshold τ t = τ base + κσ t + γ·(1/t)Σ i=1 t 1 surprise(i).

Lemma 9.1 (Adaptive Threshold Regret Bound): "Assume the loss l t is sub-Gaussian with mean μ 0 and variance proxy σ2 0 under the null model. For τ t as defined with τ base > μ 0, we have E[T s(T)] ≤ O(log T)."

Theorem 10.1 (GSR Routing Quality): "Let w(1) ≥ w(2) ≥... ≥ w(K pool) be the SIDA weights sorted in descending order. For any query, the approximation error from routing only to the Top-K prototypes is bounded by: ‖o(q) - o*(q)‖2 ≤ 2‖V‖ F · (Σ e>K w(e)/Σ e≤K w(e)) · max e ‖v e‖2. Moreover, as ε → 0+, the SIDA weights concentrate on the nearest prototype."

Theorem 10.2 (GSR Communication Complexity): "GSR has per-query communication cost: O(K pool · d h + K · d h), independent of batch size B. In the α-β model: #messages = O(K pool + K + P), bytes = O((K pool + K)d h · precision). In contrast, All-to-All MoE has: #messages = O(P2), bytes = O(BKd h · precision), which scales linearly with batch size B."

Global Architecture: x → QKV → (HIDA path → o main, SIDA → GSR → DMG → o memory) → HyperGate → o final.

Unified Mathematical Language: All variants share the core structure: o i = Σ j W ij v j, W ij = (d(q i, k j)2 + ε)-1/Σ m∈R i(d(q i, k m)2 + ε)-1, where R i varies by mode and d is Euclidean, hyperbolic, or spherical.

Proposition 1 (Complexity Observations): Dense-HIDA: Θ(n2d h); FP-HIDA: O(n log n · d h); L-HIDA: O(nd h); C-HIDA: Θ(1) per token.

Proposition 2 (Information-Theoretic Justification): I(q S; c S e) ≤ log K pool, I(k; c e) ∝ log(1/(1 - ‖c e‖2)).

Property Softmax Resolver (IDA)

Circuit separation (Thm. 1) Ω((log n)2) O(1)

Lipschitz scaling (Lemma 1) O(n) O(log n) under low-rank/O(n) general

Hessian spread (Lemma 2) Θ(n-2) Θ(1)

PL constant (Thm. 2) Θ(e-Δ2/√d ε2/Δ2) Θ(ε2/Δ4)

Effective rank (Thm. 3) Unbounded ≤ 1 + nε2/d4 min (small ε)

Noise memorization (Thm. 3) At d h ≥ n Structurally limited

The paper explicitly acknowledges: No experimental validation. This paper is purely theoretical. The theorems establish mathematical guarantees, but empirical verification on benchmarks is not provided. Additional limitations include: Compression scope. HCC compresses keys only; values remain full-precision; The two-point PL analysis is a limitation; multi-point effects are future work; and The adaptive threshold requires tuning; the O(log T) regret bound relies on a sub-Gaussian loss assumption.

The paper concludes: "This work establishes that inverse-distance attention is not merely an empirical alternative to softmax, but a mechanism with fundamentally different theoretical properties. The three Euclidean theorems—circuit separation, PL inequality, and effective rank bound—collectively show that IDA is provably stronger than softmax in expressiveness, optimization, and generalization. The non-Euclidean extension shows that these advantages are preserved and amplified in hyperbolic and spherical geometries."

Based on the paper, here are the specific improvements for AI systems:

Improvement: Replace softmax attention with inverse-distance attention (IDA) using kernel weights W ij = (‖q i - k j‖2 + ε)-1.

What the improved system can do: Perform exact one-hot retrieval when a query matches a key (as ε→0), eliminating the softmax dilution problem where outputs are always weighted averages. This enables hard memory lookup in transformers, useful for factual recall, code completion, and retrieval-augmented generation.

Improvement: Embed keys in the Poincaré ball with hyperbolic distance d H(x,y) = arcosh(1 + 2‖x-y‖2/((1-‖x‖2)(1-‖y‖2))) and use the HIDA kernel.

What the improved system can do: Store exponentially more distinct memories in the same dimension (volume grows as e(d-1)R vs. R d for Euclidean), enabling hierarchical knowledge representation (e.g., taxonomies, ontologies) with arbitrarily large key separation without increasing dimension. This is ideal for long-term memory in agents and knowledge graphs.

Improvement: Use FP-HIDA (fixed-pattern sparse) with index sets S i = j: i-j ≤ w ∪ 0, ⌊n/g⌋,... ∪ i ± 2 k ∪ i where w,g = Θ(log n).

What the improved system can do: Achieve O(n log n) attention complexity (vs. O(n2) for dense) while maintaining the exact-retrieval and rank-bound properties. This enables processing of arbitrarily long sequences (e.g., entire books, full codebases) on limited hardware.

Improvement: Use C-HIDA with c = Θ(1) learnable summary tokens updated via online hyperbolic k-means (regret O(c log T)).

What the improved system can do: Process infinite token streams with Θ(1) per-token cost while maintaining bounded memory, enabling real-time applications like live translation, financial tick analysis, and continuous sensor monitoring without context-window limits.

Improvement: Adopt IDA's PL inequality (μ IDA = Θ(ε2/Δ4) vs. softmax's Θ(e-Δ2/√d ε2/Δ2)).

What the improved system can do: Guarantee linear convergence to global minima (no spurious local minima) with O(1) Hessian spread, enabling faster training, better saddle-point escape, and reliable convergence even with deep architectures. This is critical for safety-critical AI where training must be predictable.

Improvement: Use IDA's effective rank bound eff-rank(K) ≤ 1 + nε2/d4 min (independent of hidden width d h).

What the improved system can do: Avoid the softmax width catastrophe where d h ≥ n allows memorization of arbitrary noise. The system generalizes robustly even with large hidden dimensions, with test error bounded by Cη2 + O(1/√n) under label noise—critical for preventing overfitting in large models.

Improvement: Compress keys via polar decomposition k = r·u with bit-widths b (direction) and b r (radius), achieving error ‖k-k̂‖2 ≤ 4(2-b + 2-b r).

What the improved system can do: Reduce KV-cache memory by 8× for keys (≈2.4× system-wide with half-precision values), enabling larger effective context windows or smaller GPU footprints for inference—directly applicable to serving long-context LLMs.

Improvement: Use three-level gating (head, token, dimension) with guaranteed gradient lower bound ‖∂L/∂x i‖ ≥ (λ min(G i) - O(‖W g‖‖x i‖)) · ‖∂L/∂h i‖.

What the improved system can do: Prevent vanishing gradients in gated architectures (e.g., MoE, conditional computation), ensuring stable training even with aggressive sparsity or selective activation—improving reliability of mixture-of-experts models.

Improvement: Route queries to Top-K prototypes using SIDA weights with error bound ‖o(q) - o*(q)‖ ≤ 2‖V‖ F · (Σ e>K w(e)/Σ e≤K w(e)) · max e ‖v e‖.

What the improved system can do: Reduce distributed MoE communication from O(P2) messages and O(BKd h) bytes to O(K pool + K + P) messages and O((K pool + K)d h) bytes—independent of batch size. This enables efficient scaling of MoE models across thousands of GPUs.

Improvement: Use adaptive threshold τ t = τ base + κσ t + γ·(1/t)Σ 1 surprise(i) with regret O(log T) for novelty detection.

What the improved system can do: Automatically detect distribution shifts and novel patterns in streaming data, updating memory prototypes only when statistically significant (sub-Gaussian assumption), enabling continual learning and anomaly detection without catastrophic forgetting.

Summary of system-level capabilities enabled: The improved AI system can (a) perform exact memory retrieval, (b) store exponentially more hierarchical knowledge per dimension, (c) process arbitrarily long sequences with logarithmic or constant complexity, (d) train with provable convergence guarantees, (e) generalize without width-based memorization, (f) compress KV-caches by 8×, (g) train stably with aggressive sparsity, (h) scale MoE to thousands of devices efficiently, and (i) learn continually with principled novelty detection—all with theoretical guarantees absent in current softmax-based architectures.

We present a theoretical foundation for inverse-distance attention, from its Euclidean prototype (Resolver) to its non-Euclidean realization (Riemann GeoResolver). The Euclidean part establishes three core theorems: (1) circuit separation---IDA achieves exact retrieval with O(1) resources while softmax requires ((n) 2) width; (2) a Polyak--Lojasiewicz inequality with (e 2/sqrt d/ 2) stronger constant than softmax, implying linear convergence, O(n) Lipschitz scaling under a low-rank/clustering assumption, (1) Hessian spread, and absence of spurious local minima; (3) a width-independent effective rank bound that limits noise memorization---softmax memorizes arbitrary labels when d h n, while IDA limits test error to O(eta 2). The non-Euclidean extension then builds upon this prototype, replacing Euclidean distance with hyperbolic geodesic distance for storage and spherical geodesic distance for routing. The Riemann GeoResolver framework comprises ten integrated modules: four HIDA operators spanning (n 2) to (1) per token; Hyperbolic Curvature Compression (HCC) with provable error bounds; HyperGate with gradient lower-bound theorem; Spherical Inverse Distance Attention (SIDA) with sphere-analog PL inequalities; Dynamic Memory Genesis (DMG) with O(T) regret bounds; and Geodesic Sparse Routing (GSR) with quality and communication bounds. The Euclidean theorems are proved in full; the non-Euclidean extension theorems are proved with analogous arguments. This work establishes a theoretical arc: from Euclidean attention as a special case, to hyperbolic memory, to spherical retrieval.