2608.07460-CreativeInstruct: Scalably Teaching LLMs to Balance Quality, Creativity, and Diversity

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "CreativeInstruct: Scalably Teaching LLMs to Balance Quality, Creativity, and Diversity".

Jane: The paper was written by Ananya Sahu, Mohit Bansal and Elias Stengel-Eskin from Columbia University and University of North Carolina at Chapel Hill and University of Texas at Austin.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Paper summary: Tom: The paper we're discussing today comes out of Columbia, UNC Chapel Hill, and UT Austin — Ananya Sahu, Mohit Bansal, and Elias Stengel-Eskin — and it tackles a frustration anyone who writes with modern chatbots knows well. Instruction-tuned models follow prompts beautifully, but their stories all start to sound alike. The paper's thesis is that post-training buys quality and obedience at the expense of diversity and creativity, and the authors have built a method to get both back.

Jane: The core trick is a pair of special tokens — one that opens a creative span and one that closes it — which the model learns to insert into its own generation. During training it sees where a diverse base model would take over from a polished aligned model, and at test time a single model decides by itself when to flip into creative mode. No second model required.

Lu: The training data comes from BACo, an inference-time router that runs the base and aligned models together and blends their outputs token by token. What impressed me is that the distilled single model actually beats that router at its own game — on average 29 percent higher semantic diversity and 28 percent higher structural diversity, while using half the test-time compute.

Meng: And the gains are big, not incremental. On LLaMA-3 point 1 8B the method delivers roughly 48 percent relative improvement in semantic diversity and 63 percent in structural diversity over the standard instruct model. Human annotators preferred its generations as more creative in 70 point 3 percent of comparisons.

Jane: There's also a new evaluation tool in the paper — a graph edit distance metric that compares the narrative structure of stories rather than their surface words. It catches formulaic patterns that lexical metrics simply miss.

Lalam: But the result that makes the biggest claim is the reinforcement learning one. They take a checkpoint trained with this creative method, run standard GRPO on math problems, and it beats a normally post-trained checkpoint by about 4 percent on AMC and roughly 5 points on MATH. That's the argument that diversity isn't a luxury for storytelling — it's fuel for exploration in reasoning.

Tom: And the whole recipe scales with ordinary instruction-tuning data, which is what makes it practical beyond research labs. So let's go to page one, where the paper lays out exactly what post-training does to a model's creativity.

Page 1: Tom: We've got the one-paragraph version — the method restores creativity without wrecking quality. Page one is the diagnosis, and it starts with a claim that should worry anyone building on top of these models. The abstract says it plainly: post-trained outputs converge, across model families, and scaling doesn't fix it, because bigger models still cluster into the same repetitive patterns.

Jane: I really appreciate that the paper doesn't frame this as a creative-writing nicety. It argues that reduced diversity actively hurts tasks that need diversity implicitly — most importantly reinforcement learning, where varied rollouts are essential for exploration. If the policy always samples the same reasoning path, learning stalls.

Lu: That's a strong claim.

Tom: It is, and the introduction backs it up. It grounds creativity in the classic definition — unpredictability plus diversity — and calls both essential for generating novel ideas and solving open-ended problems. That's the intellectual frame for the whole paper, and it explains why the RL experiment shows up later.

Meng: So the repetitiveness is baked into alignment itself?

Jane: Exactly, and the concrete symptom is in Figure 1 — the aligned model produces story after story that follows the same template, while the proposed method keeps the aligned model's polish but opens up the creative space. The cure is to teach the model to insert those creativity-triggering token spans during generation, so one model can shift between aligned quality and base-model inventiveness.

Lu: What I appreciate is how early they acknowledge the trade-off. They never claim to improve everything at once — they claim a balance. Quality holds on some axes, diversity goes way up, and the combination is what you actually want.

Lalam: And the abstract already seeds the payoff. The same diverse generation becomes the substrate for RL, where the creatively trained checkpoint learns better on math than the standard aligned one. The whole arc is on page one — problem, mechanism, evidence.

Jane: So the mechanism is clear in outline. What I want to know is why the obvious workarounds — like decoding with two models — don't already solve this. That's exactly where page two takes us.

Page 2: Tom: We've established the disease — post-training homogenizes outputs and that hurts even reasoning. Page two is about the existing remedies and why they're unsatisfying.

Jane: The straightforward fix is test-time routing: run a base model and an aligned model side by side, and route tokens between them, taking diversity from the base and quality from the aligned. The paper names BACo and similar frameworks, then lists the costs. You double the memory and latency at inference, and you need access to the base model — which isn't always released.

Lu: So instead of paying that cost at every generation, the paper pays it once, offline, to build training data. They pull writing-related prompts from general-purpose instruction-tuning data — the Tülu V3 SFT set — and run the router over them, then tag the spans where the base model's tokens came through.

Meng: Those tagged spans become the training signal. The aligned model fine-tunes on the corpus and learns when to switch into creative mode. At test time it self-injects the tokens and decides where the switch is warranted — no second model in the loop.

Lalam: The related-work contrast is sharpest right here. Other training-time approaches, like creative preference optimization, treat diversity as an objective over the whole output. This paper localizes creativity to spans instead, so the model learns where divergence helps — and where it doesn't — rather than forcing variety everywhere.

Jane: And on the RL side, they deliberately avoid adding a diversity reward. Prior work adds diversity bonuses to the policy objective; this paper starts from a more diverse checkpoint and runs vanilla GRPO. That's a cleaner experiment — it isolates what the starting model's diversity contributes.

Tom: There's also a striking claim buried in this section. The single distilled model actually outperforms the multi-model router at test time, even though the router gets two models to work with. The authors read that as evidence the model generalizes beyond mere imitation of the routing signal.

Lu: Then the tags must be carrying real information, not just marking random noise.

Tom: Right, and the way to see that is to look at how the data is actually built. Page three has the full recipe.

Page 3: Tom: We know the design now — one model learns to self-inject creativity tokens. Page three is the recipe, and it starts with BACo, the router that generates the data. BACo operates token by token: punctuation and formatting tokens are routed to the aligned model so sentences stay grammatical, while everything else is routed by entropy. High-entropy tokens go to the base model to encourage diversity, and low-entropy tokens go to the aligned model.

Lu: So the base model makes the surprising choices and the aligned model keeps the prose on the rails. The variant they use is called prob+punc, and it was the best-performing one on diversity metrics in the original work.

Jane: Then comes the data construction. They filter the Tülu V3 SFT set down to English writing prompts — 4,000 unique prompts — and generate three outputs per prompt, giving 12,000 training samples. For each response they track which model produced which token, and they wrap contiguous base-model spans in the creativity markers.

Meng: One detail I really like is that they also mark spans where both models assign nearly identical probabilities — within 0 point 005. When the models agree, the tag still teaches the aligned model that creative switching is permitted even in low-uncertainty territory. That's a thoughtful middle ground.

Lu: And there's a real practical problem solved here. Qwen3 32B has no released base model, so a two-model router can't run on it at all. The authors train on data generated from Qwen2 point 5 32B instead, and the transfer still works — evidence that these tagged examples carry across model families.

Tom: The fine-tuning itself is standard — LoRA on the aligned model, rank 32, applied to attention and MLP projections. One offline routing pass, then a normal instruction-tuning run. That's the whole method.

Lalam: And the RL-related work on this page reinforces the design choice. Optimizing policies for diversity has been shown to help mathematical reasoning, but the paper deliberately doesn't put a diversity reward into GRPO. The cleaner claim is that a more diverse starting point does the work by itself.

Jane: Now, measuring whether that works is subtle. The paper introduces a metric for narrative structure, and that metric does a lot of heavy lifting in the results. Page four explains how it works.

Page 4: Tom: The recipe is done — offline routing, tagged spans, LoRA fine-tuning. Page four switches to measurement, and this is the unglamorous half of the paper that might be its most reusable contribution. The new metric, LLM-GED, asks an LLM judge to convert each story into an abstract event graph — nodes are entities and events, edges are semantic relations plus temporal ordering — and then measures distance between the graphs.

Jane: The key move is canonicalization. Character names become Character1, Character2, locations become Location1, so two stories using different names still match if their narrative shapes are the same. And events are chained with directed next-event edges, which means the order of plot events matters in the comparison.

Lu: They also borrow semantic roles — agent, affected, causes — from Fillmore's case grammar. That gives the graphs real structure. A story where the dog chases the cat is distinguished from one where the cat chases the dog, because the roles swap.

Meng: The distance itself is normalized — raw graph edit distance divided by the larger graph's size, so long stories don't inflate the score. They compute all pairwise distances in a single prompt and get a full matrix back. And they validated the approach against a deterministic pipeline that genuinely computes graph edit distances — equivalent results, but the LLM version is far cheaper.

Jane: The validation details are in the appendix, and they're worth mentioning. They take four controlled settings — identical stories, lexical paraphrases, temporal reorderings, and genuinely different stories — and check that the metric ranks them in the right order. LLM-GED correlates at 0 point 889 with that reference ranking, beating all the semantic metrics.

Lalam: So structural diversity is now measured seriously, not just by word overlap. And the baselines are chosen to answer precise questions — the plain instruct model, BACo at double compute, a distillation baseline trained on the same corpus but without the creativity tags, and creative preference optimization on the LLaMA model. That no-tags baseline is the critical control.

Tom: Right — if the tags weren't doing real work, the model trained without them would perform just as well. Quality is tracked on the side too, with coherence, fluency, relevance, and a writing quality reward model. So with both rulers in hand, we get to the actual numbers — page five brings the main diversity table.

Page 5: Tom: Metric in hand, this is where the paper earns its keep. Table 1 runs the full battery of diversity measures across five models, from 7B to 32B, and the proposed method wins most columns. LLaMA-3 point 1 8B is the showcase: MiniLM cosine dissimilarity jumps from 0 point 309 for the instruct model to 0 point 458 — roughly 0 point 149 over Instruct and 0 point 203 over BACo.

Jane: The structural numbers are even more striking. On that same LLaMA model, the graph edit distance score hits 0 point 545, against 0 point 366 for Instruct and 0 point 374 for BACo — about 17 points higher. That's not just different vocabulary. That's stories with genuinely different narrative skeletons.

Lu: And the pattern holds across models. Qwen2 point 5 7B beats both Instruct and BACo on most metrics, Qwen2 point 5 32B gains 0 point 082 on the Qwen embedding dissimilarity, and Qwen3 8B takes the top spot in the majority of columns. The smaller models seem to benefit enormously from the creative tokens.

Meng: For me, the convincing comparison is the distillation baseline. Same BACo-generated corpus, same fine-tuning, but the creativity tags are stripped out. The full method beats that baseline on structural diversity in every single setting. That isolates the markers as the active ingredient — not just the extra synthetic data.

Lalam: There's one honest wrinkle, and I appreciate that the paper reports it rather than hiding it — Qwen3 32B, where the no-tag baseline edges ahead on most automatic diversity metrics. But the tag-based model still wins on LLM-GED there, 0 point 478 to 0 point 376. And remember, that model was trained on cross-family data because no base Qwen3 32B exists. The transfer case is still quite strong.

Jane: And averaged across all the models, the method beats BACo at test time by 29 percent in semantic diversity and 28 percent in structural diversity — with a single model instead of two. That comparison alone justifies the design.

Tom: Diversity is only half the story, though. If those gains came with broken prose, nobody would adopt this. Page six looks at quality — and at a very concrete symptom, the repetition of character and place names.

Page 6: Tom: We've seen the diversity gains. Page six answers the obvious objection — is this just organized chaos? According to the quality table, no. On the Writing Quality Reward Model, the tag-based method actually posts the highest score within its model family for LLaMA-3 point 1 8B, Qwen2 point 5 32B, and Qwen3 8B. For LLaMA it's 6 point 65 versus 5 point 93 for the instruct baseline.

Jane: And the distillation comparison comes back with a clear verdict. The no-tag baseline — same data, same fine-tuning, no creativity markers — generally scores worse on quality and worse on diversity. So the tags are doing two jobs at once: they boost diversity and they protect quality during fine-tuning. Without them, instruction-tuning on this synthetic data just degrades the model.

Lu: Coherence, fluency, and relevance stay competitive across the board. The paper isn't claiming to win every quality column — it's claiming the diversity gain doesn't come at quality's expense, and the reward model numbers back that up.

Meng: Then they make "repetitive" concrete. They compute proper noun uniqueness — the ratio of unique character and place names to total proper nouns, using spaCy's named entity tagger. At the prompt-group level, the tag-based model scores 37 point 1 percent, versus 26 point 6 percent for the strongest baseline and 18 point 1 percent for Instruct. Corpus-wide it's 24 point 7 percent — more than double Instruct's 12 point 0 percent.

Jane: And that difference is statistically solid — a Mann-Whitney U test with p under 0 point 001. The mechanism is intuitive. Aligned models overuse the same entities across generations, and the creative tokens break that loop.

Lalam: The qualitative examples make it visceral. Given the prompt about being the only person who remembers yesterday, the instruct model opens nearly every sample with the same phrase — "I woke up to an eerie silence" — and reaches for catastrophic motifs. The tag-based model's samples open three completely different ways and build different narrative structures.

Tom: So we have diversity and quality simultaneously, with a mechanistic explanation for why. The natural question is whether this scales — does the method need carefully curated creative data, or can you feed it general instruction data and watch diversity climb? Page seven runs exactly that experiment.

Page 7: Tom: Results so far — big diversity gains, quality intact, entity repetition broken. Page seven tackles scalability, and this is where the method separates itself from bespoke creative-writing systems. Figure 3 plots diversity against training-set size, and the curve is still climbing at 12,000 samples, the largest they used. The scores haven't plateaued, which is a strong hint that more general instruction data would push them further.

Lu: But the smarter experiment is the data-diversity comparison. They train an in-domain variant on narrative generation data only — 2,020 samples, a scarce and fixed pool — and compare it against training on general-purpose Tülu data. The general data wins even at the same dataset size. So you don't need a curated creative corpus. Broad instruction data teaches better creative generalization.

Jane: That result matters because it makes the method cheap to scale. You're not hunting for rare creative-writing data. You recycle the standard instruction mix and run the routing pass once.

Lalam: Then the paper brings humans into the loop. Three annotators — non-author students with NLP backgrounds — judge stories across 50 prompts, where GPT-5 generated diverse topics. Each system produces ten generations per prompt, the team randomly selects five, and annotators see them side by side, anonymized and order-randomized, making pairwise preference judgments on diversity, quality, and creativity.

Meng: The measurement design is careful — 14 of the prompts are judged by all three annotators so they can compute agreement. And the agreement pattern is exactly what you'd predict. Creativity shows high agreement with a Cohen's kappa of 0 point 720, diversity is moderate at 0 point 417, and quality basically doesn't agree at all — negative kappa. That's why they drop quality from the human analysis and lean on automatic metrics for that axis.

Jane: And because the systems are anonymized and order-randomized, the preference signal is about the outputs themselves. The numbers from those annotations are on page eight — along with the RL experiment, which is the other half of why this paper matters.

Page 8: Tom: We're at the payoff page. The human verdict first: the tag-based model beats the instruct baseline on creativity in 70 point 3 percent of comparisons, with a two-sided binomial test confirming significance. Diversity preference sits at 57 point 4 percent — a win, though softer, which matches the moderate agreement annotators showed on that axis.

Jane: And given the high agreement on creativity — kappa 0 point 720 — that 70 point 3 percent is a robust signal, not noise. The annotators are seeing the same structural improvements the metrics detect.

Meng: Then the RL experiment, which is refreshingly standard. Qwen3 8B, GRPO for 1,000 steps, eight rollouts per prompt, trained on a 12,000-problem split of MATH, evaluated on MATH's test set and on AMC as the out-of-domain benchmark, averaged over three seeds.

Lu: The numbers tell the real story. The instruct baseline scores 0 point 374 on MATH and 0 point 432 on AMC; after GRPO it reaches 0 point 409 and 0 point 438. The creatively trained checkpoint starts better on MATH at 0 point 424 but slightly lower on AMC at 0 point 428 — so before RL it's actually worse out-of-domain. Then, after the exact same GRPO training, it jumps to 0 point 459 on MATH and 0 point 478 on AMC. That's about 5 points over the instruct-plus-RL model on MATH, and 4 points on AMC.

Tom: The baseline reversal is the interesting part. Before RL, the creative model is worse on the out-of-domain set. After RL, it's clearly better. That supports the paper's core claim — the diversity isn't directly buying math skill. It's buying exploration, and RL converts that exploration into generalization. The appendix shows the gains are largest at higher difficulty levels.

Lalam: And that reframes creativity as infrastructure rather than personality. A model that generates more varied rollouts gives the learning algorithm more to work with. That broader lesson could outlast this specific method — it applies to any setting where exploration matters.

Jane: And it lines up with the human results. One model, more creative stories for people, and a better substrate for learning. Page nine wraps it all together.

Conclusion: Tom: So here we are at the close. The conclusion ties the threads together — an instruction-tuning approach that teaches a single model to balance aligned quality with base-model creativity, using span-level tokens learned from an offline routing pass. At test time, no second model and no routing heuristics. The model decides internally when to be creative.

Jane: And the evidence stands on three legs. Diversity gains across five models, human annotators preferring the outputs for creativity in 70 point 3 percent of comparisons, and the RL results — about 4 points on AMC and 5 points on MATH over the same training applied to a standard checkpoint. The code is on GitHub, so other groups can build on it directly.

Lalam: The broader point that stays with me is that creativity and diversity aren't decorations on top of language modeling — they're inputs to learning. The RL experiment makes that concrete. A more diverse starting model trains into a better reasoner.

Lu: I keep coming back to the Qwen3 32B case. No base model released, so two-model routing simply can't run there, and the tag-based method still delivers structural diversity gains using data from a different model family. That's practical resilience.

Meng: And the graph edit distance metric is a quiet gift to the field. By measuring narrative structure through canonicalized event graphs, the paper gives everyone a tool to see diversity that word overlap and embedding distance miss entirely.

Tom: That's a good note to end on — the paper leaves us with both a method and a better ruler for measuring what the method improves. I'll be watching for follow-ups on scaling the data further and on creative checkpoints in other RL settings. That wraps up this paper — next up, we're looking at a fresh one on reasoning, so we'll see you then.

Jane: See you then.