Aligning Language Models with Observational Data: Opportunities and Risks from a Causal Perspective
cs.LG, econ.EM, stat.ML
Submitted: 2025-05-30
Updated: 2026-09-10
Comments: Accepted at NeurIPS 2025 Workshop on CauScien: Uncovering Causality in Science, 20 pages, 5 figures
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: Large language models are being widely used across industries to generate text that contributes directly to key performance metrics, such as medication adherence in patient messaging and conversion
Terminology
Abstract
Large language models are being widely used across industries to generate text that contributes directly to key performance metrics, such as medication adherence in patient messaging and conversion rates in content generation. Pretrained models, however, often fall short when it comes to aligning with human preferences or optimizing for business objectives. As a result, fine-tuning with good-quality labeled data is essential to guide models to generate content that achieves better results. Controlled experiments, like A/B tests, can provide such data, but they are often expensive and come with significant engineering, logistical, and ethical challenges. Meanwhile, companies have access to a vast amount of historical (observational) data that remains underutilized. In this work, we study the challenges and opportunities of fine-tuning LLMs using observational data. We show that while observational outcomes can provide valuable supervision, directly fine-tuning models on such data can lead them to learn spurious correlations. We present empirical evidence of this issue using various real-world datasets and propose DeconfoundLM, a method that explicitly removes the effect of known confounders from reward signals. In simulation experiments, DeconfoundLM more accurately recovers causal relationships and mitigates failure modes of methods that assume counterfactual invariance, achieving over 16% higher objective score than ODIN and other baselines, when entangled confounding is present. Please refer to the project page for code and related resources.
Sources
- LOLA: LLM-Assisted Online Learning Algorithm for Content Experiments
- Challenges and Future Directions of Data-Centric AI Alignment
- Beyond Mimicry to Contextual Guidance: Knowledge Distillation for Interactive AI
- Fine-Tuning Language Models from Human Preferences
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- A General Language Assistant as a Laboratory for Alignment
- Secrets of RLHF in Large Language Models Part I: PPO
- Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback
- Causal Confusion and Reward Misidentification in Preference-Based Reward Learning
- Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models
- ODIN: Disentangled Reward Mitigates Hacking in RLHF
- Beyond Reward Hacking: Causal Rewards for Large Language Model Alignment
- SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks