Contextual Information Policy Optimization for Search Agents

The paper, authored by Xingyu Guo, Wei Chen, Linlin Yang, and Baochang Zhang (Beihang University and Communication University of China), proposes Contextual Information Policy Optimization (CIPO), an evidence-oriented reinforcement learning framework for training search agents that use large language models (LLMs) with external retrieval tools. The core motivation is to address prior-driven reasoning—a failure mode where agents form conclusions from internal parametric knowledge and use retrieval merely to confirm them, resulting in confirmation bias and inefficient use of retrieved evidence.

The paper identifies a fundamental limitation in existing RL training approaches for search agents:

"Existing reinforcement learning approaches for search agents broadly fall into two paradigms. The first relies primarily on sparse outcome rewards... However, the credit assignment problem in long-context reasoning makes supervision based solely on the final outcome excessively sparse. To address this limitation, the second paradigm introduces denser supervision at reasoning steps in addition to outcome rewards."

Despite these advances, the authors argue:

"Although both paradigms can improve agent performance, they share a fundamental limitation: prior driven reasoning caused by reward misalignment. Specifically, existing objectives assess only final answer correctness or the probability that a trajectory converges toward the correct answer. They do not directly evaluate whether reasoning actions after retrieval genuinely depend on newly retrieved evidence."

This misalignment leads to a situation where:

An agent can receive high rewards even when it forms a hypothesis solely from internal parametric knowledge and reduces subsequent retrieval to a mere confirmation procedure for that generated hypothesis.

The paper illustrates this with a Figure 1 example concerning Wernher von Braun and the NASA program he supervised. In prior-driven reasoning, the agent guesses the answer (e.g., Mercury program) and searches only to confirm it, whereas in evidence-driven reasoning, retrieved evidence changes the agent's subsequent reasoning and leads to a different, correct answer (Apollo program). The authors note: When the model has uncertain parametric memory or hallucinated knowledge about facts in the long tail, such seemingly valid but spurious reasoning is harmful and leaves high-quality retrieved evidence entirely unused.

CIPO represents each trajectory τi as a sequence of Ti interaction turns:

Ri,t = Bi,t think ⊕ Bi,t search ⊕ Ei,t, t = 1,..., Ti

where the three spans correspond to the tagged , , and blocks. The prefix hi,t ends at the block of turn t, excluding the current information block Ei,t.

The key innovation is EALR, which quantifies whether retrieved evidence influences the agent's subsequent actions. The target action for evaluation is the next turn's think and search blocks:

ai,t+1 = Bi,t+1 think ⊕ Bi,t+1 search = (y1,..., ym)

The same action is scored under two conditions:

• Evidence-visible: normal causal access to the information block

• Evidence-masked: an additive attention mask prevents target tokens from attending to Ei,t, while all other attention relations remain unchanged

The EALR is defined as:

sealr i,t = (1/m)[lvis(ai,t+1) − lmask(ai,t+1)] = (1/m) log [πθold(ai,t+1 hi,t, Ei,t) / πθold(ai,t+1 hi,t, M(Ei,t))]

Key properties emphasized by the authors:

"Both likelihoods are computed by teacher forcing the same collected next-turn action through the frozen rollout policy πθold. Therefore, EALR requires neither an additional trajectory rollout nor an external reward model."

"A positive EALR means that access to the retrieved information increases the likelihood of the observed next-turn reasoning and search action. A value near zero indicates that the same action is almost equally likely without access to that information, while a negative value indicates that masking the information makes the action more likely."

The paper also shows a theoretical connection to KL divergence: the expectation of the sequence-level log-likelihood ratio under the evidence-visible policy equals DKL(πθold(· h, E) ∥ πθold(· h, M(E))).

CIPO integrates EALR with terminal outcome rewards. Both rewards are standardized separately within each rollout group to avoid scale domination:

Since realr and rout differ in scale, normalizing them independently within the rollout group G ensures stable relative advantage estimates.

The turn-level advantage is computed via discounted accumulation:

Âi,t = Σ j=t Ti γ j−t r̃i,j

where γ ∈ (0, 1] is the discount factor. This integrates local evidence-dependence feedback with global task-success supervision, yielding turn-level advantages that propagate both signals across the full reasoning trajectory.

The final objective uses a clipped importance-sampling objective with a KL divergence penalty:

JCIPO(θ) = Eq,τ [(1/G) Σ i=1 G (1/Mi) Σ s∈Mi LCLIP i,s(θ) − βDKL(πθπref)]

The environment-generated `` tokens are excluded from the optimization loss: Because retrieved evidence is returned by the environment rather than generated by the policy, its tokens serve only as contextual observations and are excluded from the optimization loss.

• In-domain datasets: Natural Questions, TriviaQA, HotpotQA, 2WikiMultiHopQA

• Out-of-domain datasets: MuSiQue, Bamboogle, PopQA

• Metrics: word-level F1 score

• Backbones: Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct

• Baselines: CoT, CoT+RAG, Search-R1, R1-Searcher, DeepResearcher (outcome-only RL); StepSearch, ReasonRAG, GiGPO, IGPO (process-supervised RL)

• Training: verl framework on 8×NVIDIA A100-80G GPUs, discount factor γ = 1.0, 32 prompts per step, 16 rollouts per prompt, at most five interaction turns per rollout

CIPO achieves the highest macro-average F1 for both model sizes:

"utilizing the Qwen2.5-7B-Instruct model allows CIPO to reach a score of 0.504, which exceeds the next best method (IGPO at 0.457) by 4.7 F1 points. Similarly, the Qwen2.5-3B-Instruct backbone enables CIPO to score 0.456, surpassing its strongest competitor (GiGPO at 0.409) by an identical margin of 4.7 points."

The 7B CIPO secures the highest results on Natural Questions, TriviaQA, HotpotQA, MuSiQue, and Bamboogle; the 3B model additionally claims the top score on PopQA. CIPO also demonstrates strong out-of-domain generalization: Even though the training process relies entirely on in-domain datasets, the 7B model outperforms competitors on MuSiQue and Bamboogle, and the 3B model leads across all three out-of-domain benchmarks, including PopQA.

The ablation (Table 2) decomposes the contributions of terminal outcome supervision and EALR. Key findings:

• EALR-only training improves average F1 from 0.336 to 0.393 (3B) and from 0.372 to 0.430 (7B) relative to outcome-only training

• EALR-only also increases supportive-evidence utilization from 18.6% to 43.8% (3B) and 21.9% to 48.5% (7B)

• However, irrelevant-evidence utilization also rises (31.7% and 28.9%), showing EALR alone encourages evidence responsiveness but does not fully distinguish useful evidence from irrelevant content

• Combined EALR + outcome supervision yields the strongest results: average F1 of 0.456 (3B) and 0.504 (7B), supportive-evidence utilization rises to 55.2% and 60.7%, while irrelevant-evidence utilization drops to 10.6% and 9.1%

The paper concludes: EALR encourages the policy to respond to retrieved information, whereas outcome supervision favors evidence use that contributes to correct task completion, resulting in more selective use of answer-supportive evidence.

Using an LLM-assisted evaluation pipeline (GPT-5.5 as judge, with DeepSeek-V4-Flash for evidence annotation), the paper shows:

"In both views, IGPO's measured prior-driven rate increases over training, whereas CIPO maintains a lower rate and decreases steadily. The widening gap shows that CIPO more consistently discourages trajectories whose answer-supporting evidence originates from the model's parametric knowledge rather than from retrieved responses, including among trajectories that reach the correct final answer."

On the test sets, CIPO improves evidence-driven rates:

the overall evidence-driven rate increases from 29.9% for IGPO to 41.9% for CIPO, while the rate among correct trajectories increases from 59.4% to 70.2%.

CIPO is compared against RLOO, PPO, GRPO, DAPO, and GSPO on Qwen2.5-7B-Instruct:

Among the evaluated approaches, CIPO achieves the highest average F1 of 0.504, exceeding the strongest competing average, achieved by GSPO (0.454), by 5.0 F1 points.

The gains span both single-hop and multi-hop benchmarks, e.g., improving from 0.422 to 0.467 on NQ, from 0.358 to 0.460 on 2Wiki, from 0.290 to 0.322 on MuSiQue, and from 0.501 to 0.552 on Bamboogle.

Step-wise validation F1 trajectories (Figure 3 and Appendix B) show CIPO reaches higher F1 than IGPO early in training and maintains this advantage: The trajectory-level comparison shows that the final performance gains of CIPO are reflected throughout training in both evaluation regimes rather than emerging only at the final checkpoint.

The EALR computation requires one additional evidence-masked scoring pass during training. The overhead is modest:

• Qwen2.5-3B-Instruct: per-step time increases from 223.6 to 244.1 seconds (9.17% overhead), with 12.00% relative F1 gain (G/C ratio 1.31)

• Qwen2.5-7B-Instruct: per-step time increases from 417.2 to 436.5 seconds (4.63% overhead), with 17.48% relative F1 gain (G/C ratio 3.78)

Critically: EALR is used only to construct training rewards. At inference time, CIPO requires neither evidence masking nor additional scoring passes beyond the standard search-agent interaction loop.

The paper justifies scoring only the next-turn action rather than the full future trajectory. The next-turn formulation outperforms the full-trajectory variant by 11.4 F1 points (0.456 vs. 0.342 on average). The reason: scoring the full future trajectory would mix the effects of later reasoning steps, newly retrieved evidence, and accumulated context, making it difficult to attribute the likelihood difference specifically to Ei,t.

To verify that CIPO promotes selective rather than indiscriminate evidence use, the paper constructs three counterfactual masking variants. Random-context masking yields an Evidence Selectivity score of only 2.1; non-supportive-evidence-only masking leads to negative selectivity (−20.5), the highest irrelevant-evidence utilization (45.3%), and the lowest average F1 (0.345). CIPO achieves the lowest irrelevant-evidence utilization (10.6%), highest Evidence Selectivity (66.7%), and best average F1 (0.456), leading to the conclusion:

CIPO therefore promotes selective and task-relevant evidence dependence rather than indiscriminate evidence responsiveness.

The paper summarizes its main contributions as: (i) identifying reward misalignment caused by prior-driven reasoning and proposing CIPO; (ii) developing EALR, a dense step-level reward that compares action likelihoods with and without retrieved evidence, requiring neither human annotations nor additional rollouts; and (iii) demonstrating through experiments on seven benchmarks that CIPO substantially reduces prior-driven reasoning and outperforms state-of-the-art baselines on most tasks.

"In this work, we proposed CIPO, a reinforcement learning framework for search agents that rewards evidence-sensitive reasoning while retaining outcome supervision. CIPO assigns dense credit to actions whose generation depends on retrieved evidence. Across seven benchmarks, it improves performance and reduces prior-driven reasoning."

Future directions include expand CIPO to handle longer tasks with multiple tools and test it with different search tools, types of data and constantly changing information, as well as better ways to make the training process for evidence-masked scoring less costly.

Improvements to AI systems:

1. Add a counterfactual evidence-dependence reward (EALR) to RL training of retrieval-augmented agents. Teacher-force the agent's collected next-turn action twice through the frozen policy: once with normal attention to the retrieved `` block, and once with that block masked out. Use the per-token log-likelihood difference as a dense turn-level reward. This directly penalizes confirmation-bias behavior, because if masking the evidence doesn't change the action's likelihood, the model is clearly reasoning from parametric priors rather than evidence.

2. Combine the EALR dense reward with terminal outcome supervision, with each reward stream normalized independently per rollout group. EALR alone makes the agent responsive to all retrieved content (including irrelevant noise); adding outcome supervision makes it selective. This raises supportive-evidence utilization to 60% while dropping irrelevant-evidence utilization to 9%, and yields the best accuracy.

3. Replace expensive process supervision with likelihood-ratio rewards. The EALR requires no human step labels, no external reward model, and no additional rollouts — only one extra masked forward pass during training (4.6–9.2% per-step overhead). This makes dense, evidence-aware credit assignment affordable for long multi-turn reasoning traces.

4. Score only the immediate next-turn action when computing evidence dependence. Restrict the EALR target to the next turn's + blocks and exclude later turns, because scoring the full future trajectory mixes the effects of later evidence and accumulated context. Using the next-turn target outperforms the full-trajectory variant by 11.4 F1 points.

5. Exclude environment-generated evidence tokens from the policy optimization loss. Treat retrieved information purely as contextual observation, not as policy-generated text, so the policy is not optimized to reproduce retrieval output.

6. Use evidence-driven vs. prior-driven trajectory classification as a training monitor and checkpoint-selection metric. An LLM judge can detect spurious correctness — answers that are right but grounded in parametric memory rather than retrieved evidence — which outcome-only rewards silently accept. Selecting checkpoints on this metric prevents rewarding confirmation bias.

7. Generalize the counterfactual-masking reward beyond search to any tool/context source. Apply the same mechanism to calculator outputs, code-execution results, database returns, and live web content, so the agent learns to update its conclusions whenever new context contradicts parametric knowledge — including in constantly changing information environments.

What the improved AI system can do:

• Answer queries correctly even when its parametric knowledge is wrong or hallucinated (e.g., long-tail facts): retrieved evidence can change the final answer instead of merely confirming a guess.

• Achieve markedly higher accuracy on factoid and multi-hop QA: macro F1 of 0.504 at 7B scale (vs. 0.457 for the best baseline) and 0.456 at 3B (vs. 0.409), plus large gains on multi-hop benchmarks like 2Wiki (0.358→0.460) and MuSiQue (0.290→0.322).

• Reduce prior-driven reasoning: evidence-driven rate rises from 29.9% to 41.9% overall, and from 59.4% to 70.2% among correct trajectories.

• Generalize out-of-domain: despite training only on four in-domain sets, it leads on unseen MuSiQue, Bamboogle, and PopQA at 3B scale and on MuSiQue and Bamboogle at 7B scale.

• Use retrieved evidence selectively and usefully, ignoring irrelevant retrieved content; maintain its accuracy advantage throughout training, not just at the final checkpoint.

• Operate at zero additional inference cost — at deployment it runs the standard search-agent loop with no masking or extra scoring passes.