Contra Labs finds AI models still can't nail creative taste
By Steven Van ·
The benchmark drew 15,000 judgments from Contra's 1.5M+ vetted creatives across five domains to separate correctness from taste.
Contra Labs has published the Human Creativity Benchmark, a scoring framework for AI-generated creative work built on judgments from its network of creative professionals. Instead of resolving disagreement between evaluators into a single quality score, the benchmark separates it into two signals: convergence, where evaluators agree (shared standards like typography, layout, and visual hierarchy), and divergence, where they don't (aesthetic direction, mood, and creative risk). It finds that no current AI model is reliably both correct where correctness is called for and steerable toward taste where taste is what matters.
The study drew on Contra's network of over 1.5 million verified creatives, who have collectively earned over $250M, to evaluate AI-generated work across five domains (landing pages, desktop apps, ad images, brand images, and product videos) and three phases of the creative process (ideation, mockup, and refinement). Evaluators used three methods: pairwise forced-ranking, scalar ratings on three dimensions, and open-ended qualitative follow-ups, producing roughly 15,000 individual judgments.
The evaluation axes range from objectively verifiable to inherently subjective: prompt adherence sits at the verifiable end, usability falls in the middle, and visual appeal sits at the taste end. Prompt adherence and usability produced higher agreement among evaluators than visual appeal did, and desktop apps and landing pages converged most while ad video and brand assets stayed the most divergent. Read the full announcement for the methodology and complete findings.
