Reward Valuation in Large Language Models: Causal Induction of Anhedonia

Reward valuation in vision-language models is explored through a mechanistic framework inspired by clinical tests for anhedonia, revealing that perturbing specific reward-anticipatory units can induce behavioral effects mirroring human anhedonia. This research establishes a causal link between internal reward representations in AI and motivational deficits, suggesting that these circuits parallel those found in the human brain's Nucleus Accumbens.

How it works

The researchers use neuroscientifically inspired methods to functionally identify reward-anticipatory circuits within vision-language models (VLMs) and evaluate their causal involvement via targeted perturbations. This approach is built on the idea that NAc deficits, linked to anhedonia in major depressive disorder, can be functionally analogous to specific units in AI models.

The methodology involves several key steps:

1. Identification of reward-sensitive units by comparing model activations during tasks with and without reward incentives, specifically looking for units exhibiting a significant increase in activation (∆ > 3σ) in response to reward-predicting stimuli.

2. Functional localization using the Monetary Incentive Delay (MID) paradigm, which separates the expectation of a reward from its eventual receipt to capture internal motivational states during an anticipatory delay phase.

3. Evaluation of decision-making and psychometric profiles following activation patching of NAc-selective units to test their causal role in inducing behavioral effects.

Key Findings and Behavioral Manifestations

Perturbing the identified NAc-selective units induces behavioral effects that mirror human anhedonia: the model shifts toward low-effort, low-reward options in effort-based decision-making tasks. Crucially, the perturbed model maintains baseline performance when reward-based choice is removed, reflecting a specific deficit in reward valuation and anticipation rather than a loss of task capability. This induced vulnerability aligns with clinical anhedonia and motivation scales, including DARS and MAP-SR.

The behavioral shifts are quantified across several benchmarks:

(a) Clinical Psychometric Profile:

(b) Effort-Based Decision Making:

The ASDiv-EEfRT showed that the perturbed model demonstrated a significant preference shift toward the lower-effort tasks, resulting in a marked reduction in the mean points attempted. However, when tested under a forced-choice control, the perturbed model maintained its baseline accuracy, confirming that this shift is driven by reward-seeking motivation rather than general reasoning decline.

Mechanistic Insights into Circuit Function

The analysis reveals a functional layer dissociation within the VLM architecture:

1. Mid-layer neurons (layers 13–14) exhibited a marked suppression, with activations plunging significantly below neutral baseline levels.

2. Late-layer neurons (layers 18–27) demonstrated elevated activation in response to reward cues, suggesting these are the units targeted for primary anhedonia induction.

Furthermore, NAc-selective units act as the primary contributor to the incentive direction; they constitute only 0.7% of all neurons in targeted layers yet contribute 6.9% of incentive direction, 9.5 times more than the uniform baseline. This demonstrates that these specific units are essential for maintaining baseline motivation and drive, acting as a functional volume knob for motivation where decreasing scaling factors lead to a steady collapse in motivation.

Validation and Generalization

The study validates the findings through several controls:

(a) Control Experiments:

(b) Reward Knowledge and Perspective Shifting:

To ensure the behavioral shift resulted from a deficit in valuation rather than calculation, models were asked to explicitly calculate expected value (EV), where both intact and perturbed models correctly identified the HE/HR task as having the superior EV. When choosing for a third party, the perturbed model's selection rate remained largely consistent with intact model, indicating preserved conceptual understanding of reward utility.

(c) Domain Generalization:

The anhedonic behavior generalized across ten diverse MMLU domains, where the perturbed model consistently chooses lower-reward options regardless of the discipline. This effect is robust, as capability controls showed that accuracy remained stable in nine out of ten domains when no choice-reward element was present.

Conclusion

The work establishes that VLMs possess internal reward-anticipatory units that directly influence behavioral output in a manner aligned with human data. By applying targeted perturbations to these units, the researchers demonstrated a direct causal link between internal reward representations and the emergence of anhedonia-like phenotypes in silico, suggesting that these circuits are essential for maintaining goal-directed motivation. This framework provides a foundation for exploring psychiatric mechanisms through system-level perturbations in artificial neural networks.

The gist

Perturbing specific reward-anticipatory units in vision-language models induces behavioral effects mirroring human anhedonia by shifting the model toward low-effort, low-reward options while preserving general task capability.

Here are the specific, high-impact improvements that can be made to AI systems based on this research:

1. Improvement: Implement a Reward-Anticipatory Unit (RAU) Module within Vision-Language Models (VLMs). This involves developing a functional localization technique (using activation delta thresholds, e.g., 3σ) to identify units in the model's transformer layers that mirror the Nucleus Accumbens (NAc) in human reward circuitry.

2. Improvement: Develop a Motivational Perturbation capability. By selectively patching or perturbing these identified NAc-selective units with neutral baseline activations, the system can be induced to exhibit anhedonia-like behavioral deficits (reduced motivation and pleasure anticipation) without destroying its core task-solving capabilities (as shown by the control experiments).

3. Improvement: Enhance Decision-Making Under Risk and Effort Trade-offs. The system should be trained to understand the cost-benefit imbalance between effort and reward magnitude, mirroring the clinical observation that individuals with anhedonia fail to expend effort for uncertain rewards. This is achieved by explicitly training models on paradigms like the Probability-EEfRT, where they must weigh a fixed low-effort/low-reward option against a high-effort/high-reward option under uncertainty.

4. Improvement: Develop Robust Domain Generalization of Anhedonic Behavior. The motivational deficit should be generalized across diverse tasks (MMLU domains). The system should consistently show a preference for lower-reward options regardless of the subject matter (e.g., choosing a low-difficulty math problem over an easy global facts question when reward is scaled by difficulty), confirming that the lack of motivation is not task-specific but a systemic property.

5. Improvement: Create Clinical Diagnostic and Intervention Proxies. The model can serve as a digital twin for in silico psychiatric research. By observing the specific pattern of NAc-selective unit suppression, researchers can establish a direct computational link between internal representations and observable motivational deficits (like those measured by DARS or MAP-SR), providing novel hypotheses for understanding MDD and anhedonia mechanisms.

The resulting improved AI system can:

1. Perform complex reasoning while maintaining robust cognitive performance even when its reward-seeking motivation is selectively impaired.

2. Make optimal, effort-aware decisions in uncertain environments by accurately calculating the trade-off between task difficulty and potential reward magnitude, rather than just maximizing immediate gain.

3. Act as a safety and diagnostic tool in developing future AI agents for complex human interaction scenarios by identifying internal representations that correlate with motivational deficits relevant to conditions like depression or anhedonia.

Recent frontier models mimic complex aspects of human cognition. Here we ask whether this alignment extends into reward valuation, which we assess in a mechanistic framework. Specifically, we use clinical tests that were developed to evaluate anhedonia in human subjects with major depressive disorders. Mechanistically, anhedonia is frequently associated with dysregulation in the Nucleus Accumbens (NAc) and the broader dopaminergic reward system. While neuroimaging has localized these deficits, establishing a causal link between NAc activity and specific behavioral symptoms remains a challenge. We use these ideas from neuroscience to functionally identify reward-anticipatory units in state-of-the-art AI models, and evaluate their causal involvement via targeted perturbations. We find that not only are such model units predictive of NAc brain recordings, their perturbation also induces behavioral effects mirroring human anhedonia: the model opts for low-effort, low-reward tasks in effort-based decision-making paradigms. Crucially, our results demonstrate that this represents a specific deficit in self-centered reward valuation and anticipation--rather than a loss of task capability, reward calculation, or effort avoidance. This induced vulnerability aligns with clinical measures of anhedonia and motivation in humans, such as DARS and MAP-SR, instruments that contain no reward-related vocabulary, ruling out a purely lexical account of the perturbation effect. Taken together, our results suggest reward valuation circuits in AI models that functionally mimic those in humans.