TripScore: Aligning LLMs for Real-World Travel Planning via Expert-Calibrated Reward
cs.AI, cs.CL
Submitted: 2025-10-10
Updated: 2026-09-17
Comments: EMNLP2026 Industry track
License: http://creativecommons.org/licenses/by/4.0/
The gist: In our deployed travel-planning service, most users give minimal inputs or free-form requests rather than the structured constraint checklists assumed by existing benchmarks.
Terminology
Abstract
In our deployed travel-planning service, most users give minimal inputs or free-form requests rather than the structured constraint checklists assumed by existing benchmarks. We therefore present TripScore, a behavior-grounded benchmark and evaluation framework built from real user logs and calibrated against 1,468 pairwise judgments by 203 travel experts. TripScore couples a hierarchical feasibility gate (format and commonsense) with a unified, point-wise reward that aggregates soft quality and preference fulfillment. Using TripScore as both evaluator and reward signal, we benchmark direct prompting, test-time compute, neuro-symbolic solvers, code agents, and fine-tuning. We find that reinforcement learning fine-tuning (e.g., GRPO) provides consistent gains over other approaches under the same base model and practical latency.
Sources
- Gemini: A Family of Highly Capable Multimodal Models
- TravelAgent: An AI Assistant for Personalized Travel Planning
- Training Verifiers to Solve Math Word Problems
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- HyperTree Planning: Enhancing LLM Reasoning via Hierarchical Thinking
- GPT-4 Technical Report
- WebCanvas: Benchmarking Web Agents in Online Environments
- ChinaTravel: An Open-Ended Travel Planning Benchmark with Compositional Constraint Validation for Language Agents
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Qwen3 Technical Report
- Wide-Horizon Thinking and Simulation-Based Evaluation for Real-World LLM Planning with Multifaceted Constraints
- Scaling Relationship on Learning Mathematical Reasoning with Large Language Models
- NATURAL PLAN: Benchmarking LLMs on Natural Language Planning
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
- VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection