TASTE: A Designer-Annotated Multi-Dimensional Preference Dataset for AI-Generated Graphic Design

arXiv:2605.20731 · cs.CV, cs.AI, stat.AP · Submitted 2026-05-20 · Read on arXiv

cs.CV, cs.AI, stat.AP

Submitted: 2026-05-20

Updated: 2026-09-21

Code: https://github.com/purvanshi-lica/taste

License: http://creativecommons.org/licenses/by/4.0/

The gist: Text-to-image models now generate graphic design at production scale, yet their supervision still comes primarily from photo-style preference datasets with a single overall verdict per comparison.

Terminology

Abstract

Text-to-image models now generate graphic design at production scale, yet their supervision still comes primarily from photo-style preference datasets with a single overall verdict per comparison. Designers evaluate designs along several distinct axes (e.g., typography, layout, color harmony) that a single preference label collapses. We release TASTE(Typography, Aesthetics, Spatial, Tone, Etc.), a multi-dimensional preference dataset in which two disjoint cohorts of five professional designers each ranked outputs from four current text-to-image models across nine criteria along with per-image hallucination flags. We pair the dataset with two contributions. First, a criterion-agnostic signal-validation framework based on Kendall's τ, majority-vote probability, and Condorcet cycles against exact iid-uniform nulls; the analysis reveals significant but moderate designer agreement, with every TASTE criterion rejecting the random-rater null. Second, we benchmark preference models on TASTE and find that off-the-shelf VLM judges and dedicated T2I scorers fail to reach majority agreement with the designer panel, while a small MLP head trained directly on TASTE substantially narrows the gap to the single-rater ceiling, setting a baseline for future TASTE-trained preference models.

Sources

Related papers