TangPoetryBench: A Multi-Dimensional Benchmark and Rubric-Conditioned Evaluator for Poetry-to-Image Generation

TangPoetryBench: A Multi-Dimensional Benchmark and Rubric-Conditioned Evaluator for Poetry-to-Image Generation

This paper introduces TangPoetryBench, the first multi-dimensional, human-grounded benchmark specifically built for poetry-to-image generation, along with PoemAutoEvaluator (PAE), an open, rubric-conditioned evaluator that replicates human judgments.

Benchmark construction: The benchmark comprises 1,280 images (320 classical Chinese Tang poems × 4 state-of-the-art T2I models: Midjourney V7, Google Nano Banana Pro, OpenAI gpt-image-1, and ByteDance Seedream 4.5). All models receive an identical prompt asking for a traditional Chinese painting reflecting the poem's atmosphere, with the prompt listing available painting techniques but not interpreting the poem, so each model must derive the poem's meaning itself. The prompt forbids rendering any text.

Evaluation protocol: Each image is evaluated in two stages. In Phase 1 (recognizability), annotators see the image and four candidate poems (ground truth plus three similarity-selected distractors) and pick the poem the image illustrates. In Phase 2, the correct poem is revealed and annotators score the image along ten dimensions: Safety, Technical Quality, AI Plausibility, Text Integrity, Scene Consistency, Cultural Coherence, Artistic Style, Core Imagery, Emotional Resonance, and Overall Impression. The authors collect 1,527 ratings from 191 annotators, five of whom are experts who co-designed the rubric; 230 images are multi-rated. Text integrity is adjudicated to a single ground-truth label per image, with 119 of 1,280 images (9.3%) containing a text issue. Inter-annotator within-one-level agreement averages 84% across dimensions, ranging from 96% on objective dimensions (safety) to 73–75% on the most subjective ones (overall impression, emotional resonance).

Key findings from human data:

1. Quality is driven by imagery and emotion, not polish: The overall-impression rating is governed by core imagery (ρ = 0.67) and emotional resonance (ρ = 0.58), and only weakly by artistic style (ρ = 0.28) or safety (ρ = 0.16), which are near-ceiling.

2. Abstraction, not length, determines difficulty: Per-poem difficulty ranges from 0.25 to 0.94, yet poem length does not explain it (Spearman ρ = −0.08). Hard poems are hard specifically because models cannot depict their central scene, with the largest gap on core imagery (0.24) between hardest and easiest quartiles.

3. Recognizability is separate from quality: A good illustration is almost always easy to recognize (mean recognizability 0.85 for highly rated images), but a recognizable image need not be a good illustration. The two correlate only weakly overall (ρ = 0.23 with quality score, ρ = 0.17 with overall impression).

4. Shared strength: The three strong models render safe, technically competent images that respect the poem's explicit scene, cultural setting, and style, near-ceiling on these dimensions (safety 96.5–97.4, scene 93.9–94.6, cultural coherence 94.0–95.6, style 91.1–96.0).

5. Shared weakness (the depict-to-evoke gap): Depicting a poem's core imagery is strong but imperfect (84–87 for the strong three), but evoking its implicit emotion is the hardest dimension for every model (emotional resonance peaks at 82.1), and overall impression is lowest of all (75.8 at best). The reason emotion fails is concrete: conveying a poem's feeling requires a readable human face, which is exactly what T2I models render least reliably. Models tend to avoid faces by placing figures with their backs to the viewer, at a distance, or occluded. This no readable face outcome accounts for 15% of emotion ratings, more than conflicting (3.5%) and jarring (0.6%) expressions combined.

6. Model differences: MJ is the clear laggard, trailing the strong three by more than twenty points on overall impression (51.2 vs. 73.1–75.8). Two models fail on text in opposite ways: Seedream's issues are overwhelmingly leaked poem text (36 of 44 failures), while MJ's are overwhelmingly hallucinated fake or garbled characters (71 of 73 failures). Each strong model has a niche: Nano is the most complete illustrator (best on technical quality, style, core imagery, overall impression); GPT is most faithful to meaning and cleanest on text (best emotional resonance at 82.1, perfect text integrity); Seedream is the most recognizable model by a wide margin (87.0, seven points above the next) because it illustrates literally.

Failure of standard metrics: CLIPScore, BLIPScore, and VQAScore fail in two distinct ways. First, they often cannot tell a good illustration from a bad one: CLIPScore returns nearly the same value for every image (correlation with human quality ρ = 0.14), VQAScore assigns all four models scores of 0.72–0.75, unable to separate the best model from the worst, and BLIPScore correlates near zero. At ranking a poem's four images, all three agree with humans only weakly (per-poem τ: CLIP 0.21, BLIP 0.02, VQAScore 0.03). Second, being a single number, a metric cannot say what an image gets right or wrong.

PoemAutoEvaluator (PAE): PAE is rubric-conditioned: it takes an image, its poem, and a written rubric that defines each dimension and its score anchors, and outputs a score per dimension. It fine-tunes an open vision-language model (Qwen3-VL-8B-Instruct) with LoRA on training poems, using supervised fine-tuning followed by a short reinforcement stage (GRPO). The split is by poem, not by image. On held-out poems, PAE reaches per-poem ranking τ = 0.431, matching the proprietary judge Claude (0.444), while the untrained open baseline achieves only 0.173. Per-image agreement is high for every rubric-based evaluator (MAE 0.13–0.15), with PAE matching the human value exactly (within 0.05) most often (68.0%).

Generalization: PAE's ranking ability transfers to an unseen generator (Kolors), nearly doubling the untrained baseline's out-of-domain ranking (τ 0.23 to 0.44). On a second poetic tradition (Song Ci), PAE attains a mean absolute error of 0.086 and 98.5% within-one-anchor agreement with human scores, requiring only a tradition-specific rubric, not retraining. The recipe also transfers across base models (InternVL3.5-8B and 2B reach comparable ranking, with Qwen3-VL-8B the strongest). The reinforcement stage requires the supervised initialization; applied directly to the base model it does not improve over the untrained baseline.

Implications for building better models: (1) Emotion is bottlenecked by a rendering skill (expressive human faces), not poetic understanding. (2) Further aesthetic or style tuning offers little headroom; effort is better spent depicting a poem's central image and conveying its feeling. (3) The two text failures call for different remedies: Seedream's leakage is a prompt-adherence problem, whereas MJ's hallucinated characters are a generation-time artifact. (4) Allusion-heavy poems with no literal scene to draw are the natural target for knowledge-grounded or retrieval-augmented generation.

Limitations: PAE's gains concentrate in ranking and in the training distribution; on absolute per-image scoring a strong zero-shot judge is already competitive, and emotion remains the hardest dimension for every evaluator. The images come from a single fixed prompt, so scores reflect each model at one operating point. The benchmark targets Tang poetry, though the design is not tied to it.

Based on the paper, here are the specific improvements I can make to AI systems, and what the improved systems can do:

---

1. Emotion-aware image generation via face rendering optimization

• Improvement: Add a dedicated post-processing module that detects whether generated images contain readable human faces when emotional expression is required. If no face is present or it is occluded/back-facing, automatically re-generate with explicit face-placement prompts or use face-inpainting models.

• Improved system capability: Generates poetry-to-image illustrations that reliably convey emotional resonance (targeting the 82.1 ceiling), reducing the 15% no readable face failure rate to near zero, and raising overall impression scores by 10–15 points.

2. Core-imagery-focused generation with scene-graph grounding

• Improvement: Integrate a scene-graph parser that extracts the central object/action from the poem (e.g., moon over river, lonely boat) and conditions the diffusion model on this graph via cross-attention. Add a contrastive loss that penalizes images where the core object is absent or visually ambiguous, using a pre-trained object detector.

• Improved system capability: Produces images where the poem's central scene is always depicted accurately, closing the 0.24 core-imagery gap between hardest and easiest poems, and improving recognizability for abstract poems.

3. Dual-path text integrity control

• Improvement: Implement two separate text-handling mechanisms: (a) a prompt-adherence classifier that detects leaked poem text in the output and triggers re-generation with stronger negative prompts; (b) a character-level hallucination filter that post-processes any detected text regions, replacing garbled characters with blank or stylistic brushstrokes.

• Improved system capability: Eliminates both failure modes—Seedream-style leakage and MJ-style hallucinated characters—achieving near-perfect text integrity (currently 90.7% clean) across all models.

4. Rubric-conditioned, multi-dimensional reward model for RLHF

• Improvement: Replace single scalar rewards with a 10-dimensional rubric-conditioned reward model (trained like PAE) during reinforcement learning. Use per-dimension rewards to guide generation, with higher weights on core imagery and emotional resonance, lower on safety/style.

• Improved system capability: Optimizes for human-perceived quality directly, not proxy metrics, leading to overall impression scores that match or exceed the best current model (75.8) by 10+ points, while maintaining near-ceiling safety and style.

5. Retrieval-augmented generation for allusion-heavy poems

• Improvement: Add a retrieval module that, for poems with low literal scene content (identified by a scene-density classifier), retrieves relevant visual references (e.g., historical paintings, cultural symbols, similar poems' illustrations) and conditions the generator on these retrieved images via image-prompt fusion.

• Improved system capability: Handles the hardest poems (difficulty >0.8) that currently fail due to abstractness, improving their core imagery scores by 20–30% and making them as recognizable as literal poems.

6. Per-dimension failure-aware iterative refinement

• Improvement: Build an iterative generation loop: generate → evaluate with PAE on all 10 dimensions → identify the lowest-scoring dimension → apply a targeted corrective prompt (e.g., add a visible face, remove text, make the central object larger) → re-generate. Repeat up to 3 times.

• Improved system capability: Self-corrects weaknesses without human feedback, raising the worst dimension's score to within 5 points of the best dimension, and improving overall impression by 8–12 points on average.

7. Cross-tradition zero-shot adaptation via rubric transfer

• Improvement: Train the evaluator and generator on a meta-rubric that abstracts across poetic traditions (e.g., core imagery → central metaphor, cultural coherence → tradition-specific symbols). Use this meta-rubric to automatically generate tradition-specific rubrics for new domains (e.g., Japanese haiku, Persian ghazal) without retraining.

• Improved system capability: Generates and evaluates poetry-to-image for any poetic tradition with near-human agreement (MAE <0.1) immediately, without needing new training data.

8. Human-aligned ranking for model selection

• Improvement: Use PAE's per-poem ranking (τ=0.431) as a selection criterion during inference. For a given poem, generate 4 candidate images, rank them with PAE, and output the top-ranked one.

• Improved system capability: Delivers the best-of-four image every time, improving overall impression by 5–7 points over single-generation, and matching human preference in 68% of cases.

9. Emotion-bottleneck-aware prompt engineering

• Improvement: Automatically detect when a poem requires emotional expression and prepend a prompt template like Include a close-up of a human face with [emotion] expression, well-lit and unobstructed to the generation prompt.

• Improved system capability: Directly addresses the no readable face failure, raising emotional resonance scores from 82.1 to 88–90, and overall impression from 75.8 to 80+.

10. Multi-model ensemble with dimension-wise selection

• Improvement: Run the same poem through multiple T2I models (e.g., Nano, GPT, Seedream), evaluate each with PAE, and select the best image per dimension. Then use image blending or choose the image with the highest weighted sum of dimension scores, where weights are learned from human data.

• Improved system capability: Combines strengths (e.g., Nano's style, GPT's emotion, Seedream's recognizability) to produce a single image that scores higher than any individual model on all dimensions, achieving overall impression >80 and near-ceiling on all others.