PTCG-Bench: Can LLM Agents Master Pok'emon Trading Card Game?
cs.AI
Submitted: 2026-05-28
Updated: 2026-09-19
Code: https://github.com/zjunet/PTCGBench
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: Given a strategically complex board game, human players can quickly learn to devise strategies after playing a few rounds.
Terminology
Abstract
Given a strategically complex board game, human players can quickly learn to devise strategies after playing a few rounds. Autonomous agents require similar capabilities in realistic interactive environments, yet existing agent benchmarks often fail to fully capture such strategic and evolving decision-making scenarios. We present PTCG-Bench, a benchmark built on the Pok' e mon Trading Card Game (PTCG) that evaluates LLM agents at two complementary levels: (1) their decision-making performance within a single complex environment, and (2) their ability to self-evolving through accumulated experience. We further include a modular harness ablation to better interpret agent performance without conflating it with model capability. Our experiments show that, although LLM agents can achieve non-trivial gameplay performance, sustained and stable self-evolution remains challenging, and performance is sensitive to harness design. We hope that PTCG-Bench will facilitate future research on harness-aware and self-evolving agents in realistic interactive environments.
Sources
- GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning
- EvoSkill: Automated Skill Discovery for Multi-Agent Systems
- Measuring Sample Efficiency and Generalization in Reinforcement Learning Benchmarks: NeurIPS 2020 Procgen Benchmark
- SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
- Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Games
- lmgame-Bench: How Good are LLMs at Playing Games?
- GPQA: A Graduate-Level Google-Proof Q&A Benchmark
- The PokeAgent Challenge: Competitive and Long-Context Learning at Scale
- Pok'eChamp: an Expert-level Minimax Language Agent
- ALFWorld: Aligning Text and Embodied Environments for Interactive Learning
- Voyager: An Open-Ended Embodied Agent with Large Language Models
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
- VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection