From Scientific Crop Imagery to Agronomic Reasoning: A Multimodal LLM Benchmark for Plant Phenotyping

High-throughput, effective and comprehensive phenotyping is a critical prerequisite for crop improvement. While phenotypic analysis was traditionally performed manually, recent advances in multimodal foundation models, especially in vision-language models, have offered new opportunities for automated plant phenotyping. However, plant science remains particularly challenging for general-purpose VLMs because it requires domain-specific knowledge, fine-grained visual interpretation, and complex agronomic reasoning. We introduce PlantXpert, an evidence-grounded multimodal reasoning benchmark for plant phenotyping across four major crops: soybean, cotton, rice, and corn. Our benchmark provides a structured and reproducible framework for agronomic adaptation of VLMs, and enables controlled comparison between base models and their domain-adapted counterparts. We constructed a dataset comprising 545 digital images and 4,351 benchmark samples spanning key agronomic domains including disease, pest control, weed management and yield. The benchmark can assess diverse capabilities including visual expertise, quantitative reasoning, and multi-step agronomic reasoning. We trained and evaluated a total of 11 state-of-the-art VLMs. Fine-tuning on our dataset improves overall accuracy for all models, with many achieving around 70% accuracy, and improvements on 9 of 11 remaining statistically significant after Holm correction. We also identify diminishing returns from model scaling, uneven performance across crop subsets, and persistent weaknesses in quantitative reasoning and specialized image analysis. These limitations show that general multimodal competence alone is insufficient for reliable agronomic reasoning and demonstrate the value of domain-specific adaptation. We hope PlantXpert can serve as a foundation for advancing evidence-grounded multimodal reasoning in plant science.