Ventriloquist LLMs: Linear Alignment of Late-Stage Representations
cs.AI
Submitted: 2026-03-19
Updated: 2026-09-27
Project page: https://aiforsocialgood.github.io/iclr20
License: http://creativecommons.org/licenses/by/4.0/
The gist: Independently trained language models often learn compatible late-stage representations, despite differences in training objectives, architectures, and data modalities.
Terminology
Abstract
Independently trained language models often learn compatible late-stage representations, despite differences in training objectives, architectures, and data modalities. We ask how far this compatibility extends: can a simple affine map let one model's hidden states be read directly by another model's output head? In this work, we learn affine transformations between the final hidden states of independent models and evaluate them across embedding classification, out-of-distribution detection, and autoregressive text generation. Across model pairs, we find that downstream performance is largely preserved under alignment, with linearly mapped source representations retaining both the decision boundaries and the confidence structure of a target model's classifier. Additionally, we show for the first time that linear alignment sometimes enables text generation across independently trained models, decoding one model's hidden states through another's frozen output head without fine-tuning. This capability is far from universal, and characterizing when it holds is a contribution of our work. We find that success is governed by two factors: tokenizer overlap, which correlates strongly with generation quality, and source-model scale, below which quality degrades sharply. Transfer is also asymmetric, with strong-to-weak mappings substantially outperforming weak-to-strong ones, indicating that source representational capacity, rather than the target head, is the limiting factor. Finally, we characterize where in the network this compatibility arises. A layer-wise analysis localizes this compatibility to the final one or two layers, indicating a late-stage, output-space phenomenon rather than evidence of shared reasoning.
Sources
- Git Re-Basin: Merging Models modulo Permutation Symmetries
- Understanding intermediate layers using linear classifier probes
- Stealing Part of a Production Language Model
- Transferring Linear Features Across Language Models With Model Stitching
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks
- A generic framework for privacy preserving deep learning
- Born Again Neural Networks
- A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
- Training Compute-Optimal Large Language Models
- The Platonic Representation Hypothesis
- Scaling Laws for Neural Language Models
- Convergent Learning: Do different neural networks learn the same representations?
- Nimbus: Secure and Efficient Two-Party Inference for Transformers
- Secure Federated Transfer Learning
- The differential energy distribution and the total integrated binding energy of dynamical models
- SecFormer: Fast and Accurate Privacy-Preserving Inference for Transformer Models via SMPC
- Knowledge Distillation for Federated Learning: a Practical Guide
- On Linear Identifiability of Learned Representations
- Federated Multi-Task Learning
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
- VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection