2608.07243-Recipes for Creativity: Iterative Generation and Evaluation in Large Language Models

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Recipes for Creativity: Iterative Generation and Evaluation in Large Language Models".

Jane: The paper was written by Rens Anderson, Tessa Verhoef and Amirhossein (Miros) Zohrehvand from Leiden Institute of Advanced Computer Science and Leiden University.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Paper summary: Tom: Okay, so today's paper is one of those crossovers that sounds like a joke at first and then turns into a real research question. It takes FunSearch — an evolutionary search algorithm that DeepMind built for mathematical discovery — and applies it to writing recipes for the Pillsbury Bake-Off. The question is whether iterating the generate-and-select loop actually makes a language model more creative.

Jane: That is the fun part, but the serious question underneath is one we keep circling in this field. Most evaluations of generative models look at single artifacts, one output in isolation. Human creativity doesn't work like that. People generate, appraise, refine, loop. So the paper is testing whether that loop helps.

Lu: Right, and the loop works like this: the generator proposes a recipe, a separate evaluator model scores it against a rubric, the top scorers survive into a database, and the next round generates new recipes conditioned on those survivors. Semi-isolated islands keep different lineages from collapsing into each other. It's evolution, basically.

Meng: So what did they actually find? I'd have guessed that more iterations would give better recipes.

Tom: That's exactly the guess the results overturn. They ran five, fifteen, and thirty iterations and the creativity scores hovered around the same level. More search barely moved the needle. What actually mattered was which model did the scoring inside the loop.

Jane: And here's the counterintuitive bit. The smaller eight-billion-parameter evaluator produced significantly higher creativity scores than the larger seventeen-billion-parameter one on most dimensions. So a bigger judge steered the search toward something more conservative, and the final outputs got rated as less creative.

Lalam: The broader point is a shift of attention. Generation is cheap now — any model can pour out hundreds of recipe ideas. The interesting design problem is selection: what pressure decides which candidates survive and become parents of the next generation. This paper treats the evaluator as a first-class component, and the results justify that.

Tom: And it isn't just models judging models in a vacuum. They compared the generated recipes against thirty real entries from the 2024 Bake-Off, including the winning recipe. So there's a human benchmark sitting in the middle of the evaluation.

Jane: Let's take the setup apart, then, because there's a lot of careful scaffolding — the recipe constraints, the scoring weights, the evaluation prompts. The first page lays out where FunSearch comes from and why it maps onto creativity research so neatly.

Page 1 of the paper: Jane: So we've got the broad shape of the argument. Now let's back up to the first page, because that's where the paper builds the case for why FunSearch belongs in creativity research at all.

Tom: Yes, and the key is that FunSearch isn't just a fancy prompt loop. It keeps a database of candidate programs, partitioned into semi-isolated islands, and each round the model is prompted with high-scoring examples from its own island to propose new candidates. An automatic scorer decides what gets admitted and what gets discarded. That's how it discovered new mathematical solutions.

Jane: And the paper draws a direct line from that design to a classic creativity theory. Csikszentmihalyi's systems view puts creativity in the interaction among domain, individual, and field — the field being the gatekeepers who judge the work. FunSearch maps onto that cleanly: the generator plays the individual, the programs database plays the domain, and the in-loop scorer plays the field.

Lu: That mapping is doing real work, because it makes the evaluator internal to the creative process. In most LLM creativity work, you generate first and judge at the end. Here the judge sits inside the loop, shaping what gets generated next. That's the conceptual heart of the whole study.

Meng: They also separate themselves from other iterative methods. Self-refine, for example, critiques and revises a single draft. This approach doesn't edit anything. It keeps strong candidates and regenerates new recipes from them. That's a different search dynamic, and the authors come back to it later when they explain why iterations alone didn't improve things.

Tom: Exactly. And the page closes with two research questions. RQ1 asks whether the iterative setup beats a near one-shot baseline and approaches human-level scores. RQ2 asks which factor matters most: iteration count, generator temperature, or in-loop scorer size.

Jane: I also appreciate the early warning about evaluation. Human judgment is still the gold standard for creative artifacts, and LLM judges carry known biases, including self-preference. The authors aren't treating their evaluator as neutral. They're treating it as a design variable.

Lu: Which turns out to be the right instinct, given that the evaluator ended up being the most consequential piece of the whole pipeline.

Tom: So with the framing in place, the next page has to answer a messy practical question: what does it mean to call a recipe creative, and how do you force every candidate to respect the actual Bake-Off rules?

Page 2 of the paper: Tom: Page one left us with the conceptual machinery, the mapping between the search loop and the systems view of creativity. Page two is where it all becomes concrete.

Jane: Very concrete, in fact. Every recipe candidate has to satisfy a skeleton derived from the Bake-Off rules: a title, at most ten ingredients, exactly one official Pillsbury product, prep under thirty minutes, instructions under two thousand characters, and a story under five hundred. Anything that violates those constraints gets discarded before it's even scored.

Tom: And the story is not decoration, because the real competition rubric weighs it too. Their in-loop scorer puts seventy percent of the weight on the recipe itself — taste, appearance, creativity, crowd appeal — and thirty percent on the story, looking at narrative connection, family values, and personal passion.

Lu: But here's the wrinkle that shapes the whole benchmark. The human recipes from the 2024 competition didn't have their stories published, so the final comparison only uses the recipe component. During search, recipes and stories were optimized together. At evaluation time, the story disappears.

Meng: Wait — the stories weren't public? That seems like it changes the comparison quite a bit.

Lu: It does, and the authors are explicit about it. They call the benchmark partial calibration, not a fully matched contest replication. The in-loop scorer was rewarding stories for thirty percent of the weight, and then the story silently vanishes for the final analysis.

Jane: They also had to define the creativity measures at the level of a product rather than a person. So fluency becomes the perceived richness of ideas inside one recipe, flexibility the variety of culinary perspectives it combines, originality the novelty of the concept, and elaboration the amount of concrete detail in the final artifact.

Meng: And I liked the discipline in the evaluation prompts. Fixed persona, a short qualitative rationale before the numeric score, and a strict JSON output format. Keeping those stable across conditions means any score differences trace back to the manipulated factors, not prompt randomness.

Tom: Before the main experiments, they also ran a calibration check — could the model-based scorer rank the human reference set sensibly at all? The official winning recipe landed near the top under both candidate scorers, but not at rank one. So the evaluator has some sensitivity to quality, but it clearly diverges from the human outcome.

Lu: That honest calibration is the paper's signature move, I think. It keeps saying: this is LLM judgment, fallible, a proxy. And that matters because the results are fairly strong, so you want to know how much weight they can carry.

Jane: So the machinery is built. The next page actually sets it running and shows what came back.

Page 3 of the paper: Jane: So the machinery is in place, and the next page finally sets it running. The results come in two experiments, and the first one is genuinely surprising in how flat it is.

Tom: Experiment one varied the iteration count, and the pattern is remarkably stable. Five iterations gave a mean creativity score of 3 point 921, fifteen gave 3 point 835, thirty gave 3 point 927. The human Pillsbury reference set sits at 3 point 638. So every iterative condition clears the human benchmark, but there's no upward trend. The number of rounds just doesn't matter.

Meng: Wait — what exactly is the "near one-shot" baseline they keep comparing against?

Tom: It's the same pipeline with the repeated search removed, basically a single generation pass. The in-loop weighted recipe score dropped from 4 point 71 to 4 point 00, but the final assessed creativity stayed roughly the same, about 4 point 1. The in-loop rubric optimizes for Bake-Off-style criteria, not for the TTCT creativity scores, so the search improves on its own terms without budging the measured creativity.

Lu: There's a variance finding too. The iterative conditions spread more than the human set — standard deviation of 0 point 411 at thirty iterations versus 0 point 328 for the benchmark. The authors read that as the search exploring a broader, less uniform solution space.

Meng: Then experiment two crosses generator temperature at 0 point 5, 1 point 0, and 1 point 5 with the two in-loop scorer sizes, the eight-billion and the seventeen-billion parameter models. They fixed seven islands and a batch size of five so differences trace back to the temperature and the scorer, not search breadth. And the dominant result is the scorer.

Jane: The smaller eight-billion evaluator produces higher scores on average creativity, fluency, flexibility, and elaboration. That's the headline of the whole paper for me. A bigger judge makes the search more conservative, and the final products get rated as less creative.

Tom: And originality is the exception, which we'll see again in the regression numbers. Temperature is much quieter. The only clear effect is that the lowest temperature reduces originality. Higher temperature doesn't significantly improve anything. So adding randomness doesn't enrich creativity — it changes the risk profile.

Lalam: It all points the same direction. The generation side isn't the constraint. The selection side is doing the steering.

Tom: Which is exactly why I want to look at the regression table next, because it separates those effects statistically and shows how much variance remains unexplained.

Page 4 of the paper: Tom: The results point hard at the evaluator, but the figures only tell part of the story. The regression table on page four is where the effects get separated.

Jane: And it uses temperature 1 point 5 with the eight-billion scorer as the reference condition. The large seventeen-billion scorer shows significant negative coefficients on creativity, fluency, flexibility, and elaboration. Flexibility takes the biggest hit, around minus 0 point 307. Creativity drops by 0 point 161.

Tom: Originality is the odd one out. The larger scorer doesn't have a significant effect there. So the penalty isn't about novelty — it's concentrated in the dimensions that reward richness, variety, and detail.

Lu: Temperature shows up only once in the table, as a significant negative coefficient on originality at the lowest temperature. Everything else is statistically quiet. So the temperature story from the figures holds at the regression level: cold sampling makes things more conventional, and that's about it.

Meng: The humble part of the table is the adjusted R-squared. They're all low, between about 0 point 006 and 0 point 037. Temperature and scorer size together explain almost none of the variance in final creativity scores. So a lot is going on that the design variables don't capture — the specific examples retained on each island, prompt framing, and differences among the four evaluators.

Jane: That low explanatory power is actually part of the argument. The evaluator size matters, yet the bigger signal is that the usual knobs we reach for — iterations, temperature — leave the outcome mostly unexplained. The evaluator choice is one of the few systematic effects in a noisy process.

Lalam: And the design implication is fairly direct. If your in-loop scorer rewards well-formed, conservative artifacts, repeated search will settle around competent but unsurprising recipes. A looser evaluator could broaden the exploration, but you might lose coherence. So the interesting design space is the evaluative ecology: the diversity, architecture, and incentives of the scorers that decide what survives.

Lu: The authors are also appropriately cautious about their own method. Both the in-loop scoring and the final evaluation use LLMs, and the seventeen-billion model served as both an in-loop scorer and one of the four final evaluators. Averaging over four models softens the overlap but doesn't erase it. And the calibration check showed only partial agreement with the human outcome — the human winning recipe, for example, got a higher originality assessment than the generated ones.

Meng: So the paper tells us how outputs behave under LLM evaluation, not how a human tasting panel would rank them. That's the scope they claim, and they stick to it.

Jane: Which brings us to the conclusion and what the authors think this means for building creative systems. I think it lands in a useful place.

Conclusion: Jane: We've followed the paper from the conceptual framing through the design and the results, and the message is pretty clear by now.

Tom: Right, let's close this out. The paper took an algorithm proven on mathematics, pointed it at a baking competition, and extracted a clear lesson. Iterative generation and selection can produce recipes that score comparably to human benchmark entries under LLM evaluation, but adding more iterations doesn't push creativity upward.

Jane: And the decisive factor is the in-loop evaluator. The smaller eight-billion scorer produced higher scores across most creativity dimensions than the seventeen-billion one, with significant negative effects on fluency, flexibility, and elaboration. Temperature mattered only for originality, and only when it was low.

Lu: The general point I take from this is that creative systems should be designed around selection. Which models judge the candidates, how diverse they are, what rubrics and incentives they carry — that's where the creative character of the output gets determined. Generating more candidates is not the bottleneck.

Meng: But the scope needs to stay clear. This study says more about how outputs behave under LLM evaluation than about how people would judge them. The benchmark check against the actual competition showed only partial agreement.

Tom: Good point, and the authors would agree. They call recipe generation a bridge case — not fully open-ended like a divergent thinking task, not fully objective like a math problem. A recipe has to be novel, but it also has to be coherent, plausible, and something you'd actually cook. That's exactly why it's a good testbed for iterative creative search.

Jane: What I'll carry from this paper is a reframing: from how many ideas a model can produce, to how we design the pressure that selects among those ideas. And the paper gives that an empirical backbone — measurable effects, a clear negative result on iteration count, and a concrete recommendation to put real care into the evaluator.

Lu: And that negative result is just as valuable as the positive finding. It saves future researchers from a very tempting default.

Tom: Nicely put. We'll say goodbye to this paper and move on to the next one.

Jane: Onward.