Learning Concepts, Not Tokens: Self-Supervised Semantic Alignment for Language Models

arXiv:2603.29123 · cs.CL · Submitted 2026-03-31 · Read on arXiv

cs.CL

Submitted: 2026-03-31

Updated: 2026-08-30

Code: https://github.com/christine-zhang1/learning-concepts

Project page: http://skylion007.github.io/OpenWebTextCorpus

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Perplexity — 64× in this paper · explained in 3 episodes
• Clustering — 13× in this paper · explained in 2 episodes
• Alignment — 12× in this paper · explained in 1 episode
• Benchmark — 11× in this paper · explained in 2 episodes
• Embeddings — 9× in this paper · explained in 2 episodes
• Accuracy — 7× in this paper · explained in 1 episode
• Semantic Alignment — 6× in this paper · explained in 3 episodes
• Transformer — 5× in this paper · explained in 2 episodes
• Cosine Similarity — 4× in this paper · explained in 3 episodes
• Semantic Similarity — 4× in this paper · explained in 2 episodes
• Semantic Understanding — 3× in this paper · explained in 2 episodes
• Ablation — 2× in this paper · explained in 1 episode
• Fine-Tuning — 2× in this paper · explained in 5 episodes
• Generalization — 2× in this paper · explained in 13 episodes
• Instruction Tuning — 2× in this paper · explained in 2 episodes
• Abstraction — 1× in this paper · explained in 1 episode
• Adam — 1× in this paper · explained in 2 episodes
• Adaptation — 1× in this paper · explained in 1 episode
• Contrastive Learning — 1× in this paper · explained in 5 episodes
• Diffusion Model — 1× in this paper · explained in 8 episodes
• Domain Adaptation — 1× in this paper · explained in 5 episodes
• Hallucination — 1× in this paper · explained in 3 episodes
• Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes
• Latent Reasoning — 1× in this paper · explained in 2 episodes
• Linear probe — 1× in this paper · explained in 2 episodes
• Loss Function — 1× in this paper · explained in 2 episodes
• Objective Function — 1× in this paper · explained in 2 episodes
• Regularization — 1× in this paper · explained in 3 episodes
• Transfer Learning — 1× in this paper · explained in 10 episodes
• Transformer Architecture — 1× in this paper · explained in 2 episodes

Sources

• Efficient Joint Prediction of Multiple Future Tokens
• Large Concept Models: Language Modeling in a Sentence Representation Space
• Improving Large Language Models with Concept-Aware Fine-Tuning
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• DeepSeek-V3 Technical Report
• MMTEB: Massive Multilingual Text Embedding Benchmark
• The Llama 3 Herd of Models
• ProFit: Leveraging High-Value Signals in SFT via Probability-Guided Token Selection
• SGPT: GPT Sentence Embeddings for Semantic Search
• MTEB: Massive Text Embedding Benchmark
• Qwen3 Technical Report

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering