Redemption Score: A Multi-Modal Evaluation Framework for Image Captioning via Distributional, Perceptual, and Linguistic Signal Triangulation
cs.CV, cs.CL
Submitted: 2025-05-22
Updated: 2026-09-16
Comments: Accepted version to IEEE Transactions on Multimedia
License: http://creativecommons.org/licenses/by/4.0/
The gist: Evaluating image captions requires cohesive assessment of both visual semantics and language pragmatics, which is often not entirely captured by most metrics.
Terminology
Abstract
Evaluating image captions requires cohesive assessment of both visual semantics and language pragmatics, which is often not entirely captured by most metrics. As such metrics increasingly guide model development, benchmarking, and system optimization in multimodal AI, inaccuracies in evaluation can misrepresent true progress. We introduce Redemption Score(RS), a novel evaluation framework for multi-modal generation by triangulating three complementary signals: (1) Mutual Information Divergence (MID) for global image-text distributional alignment, (2) DINO-based perceptual similarity of cycle-generated images for visual grounding, and (3) LLM Text Embeddings for contextual text similarity against human references. A calibrated fusion of these signals allows RS to offer a more holistic assessment. On the Flickr8k benchmark, RS achieves a Kendall- τ of 58.42, outperforming most prior methods and demonstrating superior correlation with human judgments without requiring task-specific training. Our framework provides a more robust and nuanced evaluation by thoroughly examining both the visual accuracy and text quality together, with consistent performance across Conceptual Captions and MS COCO.
Sources
- FLEUR: An Explainable Reference-Free Evaluation Metric for Image Captioning Using a Large Multimodal Model
- Towards General Text Embeddings with Multi-stage Contrastive Learning
- Understanding Guided Image Captioning Performance across Domains
- Evaluating Image Caption via Cycle-consistent Text-to-Image Generation
- QLoRA: Efficient Finetuning of Quantized LLMs
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- GIT: A Generative Image-to-text Transformer for Vision and Language
- Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
- BERTScore: Evaluating Text Generation with BERT
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models