LLMs are not stochastic parrots: Evidence for meaning-mediated abstraction from conlang-like tasks

arXiv:2609.34187 · cs.CL, cs.AI · Submitted 2026-09-28 · Read on arXiv

cs.CL, cs.AI

Submitted: 2026-09-28

Updated: 2026-10-04

Code: https://github.com/meta-llama/llama3

Terminology

• Abstraction — 42× in this paper · explained in 1 episode
• Cosine Similarity — 38× in this paper · explained in 3 episodes
• Benchmark — 31× in this paper · explained in 2 episodes
• Tokenization — 16× in this paper · explained in 2 episodes
• Transformer — 8× in this paper · explained in 2 episodes
• Consistency — 6× in this paper · explained in 3 episodes
• Failure Modes — 6× in this paper · explained in 3 episodes
• Thinking mode — 6× in this paper · explained in 2 episodes
• Trace — 6× in this paper · explained in 2 episodes
• Alignment — 5× in this paper · explained in 1 episode
• Core — 5× in this paper · explained in 2 episodes
• Distributional Hypothesis — 4× in this paper
• Instruction Tuning — 4× in this paper · explained in 2 episodes
• Generative AI — 3× in this paper · explained in 2 episodes
• Latent Space — 3× in this paper · explained in 7 episodes
• Pragmatics — 3× in this paper · explained in 2 episodes
• Prompt Engineering — 3× in this paper · explained in 4 episodes
• Semantic Similarity — 3× in this paper · explained in 2 episodes
• Adam — 2× in this paper · explained in 2 episodes
• Aggregation — 2× in this paper · explained in 1 episode
• Fine-Tuning — 2× in this paper · explained in 5 episodes
• GRPO — 2× in this paper · explained in 6 episodes
• Group Relative Policy Optimization — 2× in this paper
• In-Context Learning — 2× in this paper · explained in 4 episodes
• Robustness — 2× in this paper · explained in 5 episodes
• AI Literacy — 1× in this paper · explained in 1 episode
• Accountability — 1× in this paper · explained in 2 episodes
• Accuracy — 1× in this paper · explained in 1 episode
• Affordance — 1× in this paper · explained in 1 episode
• Contrastive Learning — 1× in this paper · explained in 5 episodes
• EMMA — 1× in this paper · explained in 2 episodes
• Embeddings — 1× in this paper · explained in 2 episodes
• F1 Score — 1× in this paper · explained in 3 episodes
• Hyperparameters — 1× in this paper · explained in 2 episodes
• In International Conference on Learning Representations — 1× in this paper
• Interpretability — 1× in this paper · explained in 15 episodes
• Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes
• Phase Transition — 1× in this paper · explained in 2 episodes
• Reinforcement Learning — 1× in this paper · explained in 4 episodes
• Resource Constraints — 1× in this paper · explained in 3 episodes
• Scaffolding — 1× in this paper · explained in 3 episodes
• Verifier — 1× in this paper · explained in 2 episodes

Sources

• An enriched category theory of language: from syntax to semantics
• Optimizing Length Compression in Large Reasoning Models
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• "Genlangs" and Zipf's Law: Do languages generated by ChatGPT statistically look human?
• Energy-Based Transformers are Scalable Learners and Thinkers
• Does Prompt Formatting Have Any Impact on LLM Performance?
• Measuring Massive Multitask Language Understanding
• Instruction Following without Instruction Tuning
• Neologism Learning for Controllability and Self-Verbalization
• Lexinvariant Language Models
• Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• Dissociating language and thought in large language models
• Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve
• GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models
• 2 OLMo 2 Furious
• Billions of Sketches Reveal Hidden Cultural Variation in Human Concepts
• Prompt Chaining or Stepwise Prompt? Refinement in Text Summarization
• Hierarchical Reasoning Model
• TruthRL: Incentivizing Truthful LLMs via Reinforcement Learning

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering