Reward Design for Physical Reasoning in Vision-Language Models

summary

Video file (mp4)

The gist

The gist The reward design shapes physical reasoning behavior in Vision-Language Models by comparing four reward signals—format compliance, answer accuracy, a composite rubric reward, and an

In short

The study compared four reward signals—format compliance, answer accuracy, a composite rubric reward, and an internal attention-based reward—to improve physical reasoning in Vision-Language Models. Results showed that accuracy rewards provided the biggest gains overall. The internal attention reward offered a useful grounding signal for spatial reasoning and complemented external supervision.

Key concepts

External Rewards
These are rewards based on outside verification, such as checking if an answer follows a specific format or getting a score from an LLM judge. They rely on human-defined rules or another model to grade the output, providing feedback without requiring labels directly from the model.
Internal Attention-Based Reward
This is a reward calculated using the model's own attention mechanism, looking at which parts of the image tokens it focuses on. It acts as an annotation-free signal that helps improve spatial reasoning by rewarding the model for paying attention to relevant visual regions.
Reward Design
This refers to choosing and combining different signals (like accuracy or attention) to guide a model's training. The paper shows that the specific reward chosen significantly shapes the type of reasoning behavior the VLM develops, not just what it gets right.
GRPO Training
Group Relative Policy Optimization (GRPO) is the specific reinforcement learning algorithm used to train the Vision-Language Model. It uses these various reward signals to iteratively adjust the model's policy, steering its behavior toward better physical reasoning outcomes.

Terminology used across episodes

This episode discusses

The paper

Reward Design for Physical Reasoning in Vision-Language Models · Read on arXiv

Derek Lilienthal, Manisha Mukherjee, Sameera Horawalavithana

Pacific Northwest National Laboratory

Physical reasoning over images requires integrating perception, domain knowledge, and multi-step symbolic inference, yet Vision-Language Models (VLMs) fall far short of human performance on physics benchmarks. Reinforcement learning can improve reasoning via reward signals, but it remains unclear whether these signals must come from outside the model or can be derived from the model itself. We compare external rewards, which rely on an outside verifier (rule-based format and answer-correctness checks, or an LLM-judge rubric), with an internal attention-based reward computed from the model's own attention over image regions, requiring no labels or judge. We find that external rewards drive most of the accuracy gains, while preserving more coherent reasoning when rewarded for reasoning tokens. The internal attention-based reward, provides an annotation-free grounding signal that improves spatial-relation reasoning and is complementary to external rewards: adding it to format and accuracy rewards, at no additional annotation cost, improves average accuracy on Molmo 2 while generalizing to out-of-distribution tasks. Our results suggest that internal attention rewards can supplement external supervision and lead to improved physical reasoning.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Reward Design for Physical Reasoning in Vision-Language Models".

Jane: The gist The reward design shapes physical reasoning behavior in Vision-Language Models by comparing four reward signals—format compliance, answer accuracy, a composite rubric reward,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Okay, so the title is "Reward Design for Physical Reasoning in Vision-Language Models," and honestly, it sets up this comparison of different training signals.

Jane: It’s about taking these models that can see images and read text, like Vision Language Models, and figuring out the best way to give them feedback to help them do physics reasoning better.

Lu: The authors are systematically testing four reward configurations—format compliance, answer accuracy, a composite rubric reward which includes correctness plus principle identification and unit consistency, and then this new internal attention-based reward.

Meng: What's interesting is that they aren't just looking at one setting; they’re checking five different training conditions to see how these signals interact with each other.

Lalam: Essentially, the goal is to find out which combination of these reward signals actually shapes the model into a better physical reasoning machine.

The paper's summary: Tom: The paper summarizes that accuracy-based rewards are generally providing the strongest overall gains across their tests for physics problems.

Jane: For example, when they looked at multiple choice questions, the combination of format, accuracy, and answer correctness emerged as a really strong configuration.

Lu: They found that GRPO training with format plus accuracy plus answer correctness scored zero point four six two overall on multiple choice questions and also ranked second in three other physics domains on the PhyX benchmark.

Meng: But it’s not always the same for open-ended questions, where things get a bit more nuanced.

Lalam: For open-ended problems, they found that format plus accuracy was their strongest configuration overall, giving a score of zero point zero two seven and showing a two times improvement over the standard supervised fine-tuned model.

The paper's improvements: Tom: The authors suggest that reward design isn't just about getting the right answer; it actually induces different types of reasoning behaviors in the model, which is a really important point.

Jane: They show that using an internal attention-based reward, which we call R4, specifically enhances spatial reasoning skills but it can degrade performance when dealing with symbolic domains like thermodynamics and wave or acoustics.

Lu: And they also found that using the composite rubric reward shows an eleven point eight percent accuracy decrease compared to just format plus accuracy for open-ended evaluations.

Meng: That suggests that when you try to optimize a multi-objective reward signal on a smaller model, you can actually destabilize the whole training process with GRPO optimization.

Lalam: So the paper points out that there isn't one single best way to train these models; the choice of which reward signal to use depends entirely on what you want it to do.

Conclusion: Tom: Wrapping up, this whole study suggests that reward design should be seen as a mechanism for controlling the nature of reasoning in these multimodal models.

Jane: It means we need to choose our reward signal based on what we need at deployment: do you want raw accuracy, or do you want interpretable reasoning chains?

Lu: The choice of reward signal should be guided by deployment needs, which means if you want visual grounding, use the attention-based reward, but if you prioritize symbolic reasoning capacity, that’s a different path.

Meng: It's clear that visual grounding supervision and symbolic reasoning capacity end up competing for the representational resources available at the two billion parameter scale.

Lalam: So for anyone working on this, it means matching the reward to the target behavior is just as important as picking the right training algorithm itself.

Tom: Exactly. The paper "Reward Design for Physical Reasoning in Vision-Language Models" shows that there isn't one configuration that dominates across all physics domains or reasoning types, and the choice of reward signal should be guided by what matters most in deployment: raw accuracy, interpretable reasoning chains, or visual grounding.

More episodes

← Home