Text Has Curvature
cs.LG
Submitted: 2026-02-13
Updated: 2026-09-05
Comments: ICML 2026
Project page: http://skylion007.github.io/OpenWebTextCorpus
License: http://creativecommons.org/licenses/by/4.0/
The gist: Does natural language text have an intrinsic curvature? Language is increasingly modeled in curved geometries - hyperbolic spaces for hierarchy, mixed-curvature manifolds for compositional structure
Terminology
Abstract
Does natural language text have an intrinsic curvature? Language is increasingly modeled in curved geometries - hyperbolic spaces for hierarchy, mixed-curvature manifolds for compositional structure - yet a basic scientific question remains unresolved: what does curvature mean for text itself, in a way that is native to language rather than an artifact of the embedding space we choose? We argue that text does indeed have curvature, and show how to detect it, define it, and use it. To this end, we propose Texture, a text-native, word-level discrete curvature signal, and make three contributions. (a) Existence: We provide empirical and theoretical certificates that semantic inference in natural corpora is non-flat. (b) Definition: We define Texture as a signed two-axis curvature of the word-in-context belief field - the differential of reconciliation between prefix and suffix - measuring, via a debiased Schrödinger transport divergence, whether adding context from one side contracts the semantic effect of context from the other side (focus, positive) or expands it into competing continuations (fan-out, negative). (c) Utility: Texture is actionable: it serves as a general-purpose measurement and control primitive enabling geometry without geometric training; we instantiate it on two representative tasks, improving long-context inference through curvature-guided compression and retrieval-augmented generation through curvature-guided routing. Together, our results establish a text-native curvature paradigm, making Texture practically useful.
Sources
- Spectro-Riemannian Graph Neural Networks
- Position: Beyond Euclidean -- Foundation Models Should Embrace Non-Euclidean Geometries
- LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
- Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach
- RoBERTa: A Robustly Optimized BERT Pretraining Approach
- Pointer Sentinel Mixture Models
- QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks