Amortizing intractable inference in large language models
cs.LG, cs.CL
Submitted: 2023-10-06
Updated: 2026-08-31
Comments: ICLR 2024; 23 pages; code: https://github.com/GFNOrg/gfn-lm-tuning
Code: https://github.com/GFNOrg/gfn-lm-tuning
License: http://creativecommons.org/licenses/by/4.0/
The gist: Autoregressive large language models (LLMs) compress knowledge from their training data through next-token conditional distributions.
Terminology
Abstract
Autoregressive large language models (LLMs) compress knowledge from their training data through next-token conditional distributions. This limits tractable querying of this knowledge to start-to-end autoregressive sampling. However, many tasks of interest -- including sequence continuation, infilling, and other forms of constrained generation -- involve sampling from intractable posterior distributions. We address this limitation by using amortized Bayesian inference to sample from these intractable posteriors. Such amortization is algorithmically achieved by fine-tuning LLMs via diversity-seeking reinforcement learning algorithms: generative flow networks (GFlowNets). We empirically demonstrate that this distribution-matching paradigm of LLM fine-tuning can serve as an effective alternative to maximum-likelihood training and reward-maximizing policy optimization. As an important application, we interpret chain-of-thought reasoning as a latent variable modeling problem and demonstrate that our approach enables data-efficient adaptation of LLMs to tasks that require multi-step rationalization and tool use.
Sources
- Training Verifiers to Solve Math Word Problems
- Language Model Cascades
- Efficient (Soft) Q-Learning for Text Generation with Limited Good Data
- Length Generalization in Arithmetic Transformers
- Teaching Arithmetic to Small Transformers
- Sequential Monte Carlo Steering of Large Language Models using Probabilistic Programs
- G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
- Proximal Policy Optimization Algorithms
- Making Large Language Models Better Reasoners with Alignment
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Reprompting: Automated Chain-of-Thought Prompt Inference Through Gibbs Sampling
- Teaching Algorithmic Reasoning via In-context Learning
- Text Infilling
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks