POLARIS: Guiding Small Models to Write Long Stories
cs.CL, cs.AI
Submitted: 2026-06-02
Updated: 2026-08-30
Comments: Accepted to EMNLP 2026 (Main Conference) as a long paper
License: http://creativecommons.org/licenses/by/4.0/
The gist: Small open-weight models struggle at long-form creative writing: their generated stories either fall far short of the requested length, or their quality significantly degrades as length increases,
Terminology
Abstract
Small open-weight models struggle at long-form creative writing: their generated stories either fall far short of the requested length, or their quality significantly degrades as length increases, especially when compared to frontier models. We present POLARIS (Policy Optimization with LLM-as-a-judge rewards and Anchored-Reference Injection for Storywriting), a lower-compute GRPO recipe with two key ingredients: a frontier LLM judge with a structured Story Quality rubric as the online reward, and human-reference injection (HRI), where a teacher-forced human-written story serves as a high-reward anchor within each GRPO group. By applying our training recipe to Qwen3.5-9B, using a dataset of approximately 1.4K prompt-story pairs derived from 100 short-story anthologies and 4 A100 GPUs, we obtain POLARIS-9B. Across five benchmarks spanning in-distribution and out-of-distribution prompts and rubrics, POLARIS-9B is competitive with much larger open-weight models while following length instructions more closely. A blinded human evaluation confirms that POLARIS-9B is preferred to the base Qwen3.5-9B and on par with Qwen3.5-27B. Despite training only on stories up to 4k words, POLARIS-9B preserves quality on prompts requesting stories up to 3 times the training length, a regime where most open-weight models degrade substantially in quality, length adherence, or both. More broadly, our results suggest that length generalization is a meaningful stress test for creative-writing models and a useful lens for distinguishing otherwise close models.
Sources
- DPWriter: Reinforcement Learning with Diverse Planning Branching for Creative Writing
- Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewards
- Gemini: A Family of Highly Capable Multimodal Models
- Diverse Preference Optimization
- LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods
- G$^2$RPO-A: Guided Group Relative Policy Optimization with Adaptive Guidance
- RePO: Replay-Enhanced Policy Optimization
- Rewarding Creativity: A Human-Aligned Generative Reward Model for Reinforcement Learning in Storytelling
- R2-Write: Reflection and Revision for Open-Ended Writing with Deep Reasoning
- WritingBench: A Comprehensive Benchmark for Generative Writing
- EQ-Bench: An Emotional Intelligence Benchmark for Large Language Models
- HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models
- Writer-R1: Enhancing Generative Writing in LLMs via Memory-augmented Replay Policy Optimization
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering