From Monolithic to Modular: Segment-level Automatic Prompt Optimization

The paper presents SAPO, a segment-level Automatic Prompt Optimization (APO) method that addresses a persistent limitation in existing APO pipelines: they often rewrite prompts monolithically, which can improve one behavior while degrading others. SAPO decomposes prompts into four explicit segments—role, context, tasks, and output format—then applies targeted improvements based on top-5 and bottom-5 examples. The optimization loop uses one LLM with static meta-prompts and structured outputs for segmentation, weakness analysis, and candidate generation. The authors describe a train/validation protocol and a two-stage generation process: (1) segment-level diagnosis and recommendation extraction, (2) candidate synthesis constrained by weak/strong segment signals. Using evaluation across SQuADv2, TweetEval, XSUM, CommonGen, and GSM8K on GPT-3.5-Turbo and GPT-4o-mini, SAPO achieves the best average score against Zero-shot and strong APO baselines including APE, OPRO, EvoPrompt, GEPA, and StraGO.

The paper motivates the work by noting that Large language models (LLMs) are increasingly used as general-purpose interfaces for NLP tasks, including instruction following, reasoning, and generation. In practical settings, adaptation has shifted from finetuning toward prompt design, where system behavior is controlled through natural-language instructions. Consequently, prompt quality becomes a primary reliability bottleneck: small wording changes can produce large behavioral shifts, while manual prompt iteration remains costly and unstable.

The authors identify a persistent limitation: "many APO pipelines still optimize prompts as monolithic strings. In practice, prompts are compositional artifacts: role framing, context grounding, task directives, and output-format constraints contribute differently to downstream behavior. Prior work reports prompt drifting and instability in iterative optimization, where edits that fix one subset of cases may degrade previously correct behavior. The core challenge is not merely generating more prompt variants, but controlling where and how edits are applied."

SAPO contributes three key elements:

1. Segment-explicit optimization objective: prompts are decomposed into role, context, tasks, and output format, enabling targeted rather than monolithic edits.

2. Constrained candidate synthesis and selection: candidates preserve strong segments, revise weak segments, and use an edit-distance tie-break for conservative updates.

3. Contrastive diagnostic stage: update decisions are grounded in top/bottom evidence, including class-aware handling for discrete per-example metrics.

The evaluation covers five datasets with different task types: SQuADv2, TweetEval, XSUM, CommonGen, and GSM8K, on two model backbones: GPT-3.5-Turbo and GPT-4o-mini. SAPO achieves the best average score against APE, OPRO, EvoPrompt, GEPA, and StraGO, with relative gains of +5.13% on GPT-3.5-Turbo and +7.25% on GPT-4o-mini compared to the strongest competing baseline by average score.

The paper reviews search-based APO methods: APE optimizes instruction candidates via generation and selection; ProTeGi applies textual gradients with beam search and bandit-style selection; OPRO and EvoPrompt extend search with trajectory-conditioned optimization and evolutionary operators; GEPA introduces reflective prompt evolution. The authors note that these methods primarily operate as global prompt-level rewrites rather than as explicit segment-constrained updates.

The paper also reviews structured and modular optimization trends: DSPy and TextGrad cast optimization in programmatic or graph-based forms; Promptomatix emphasizes modular orchestration and cost-aware refinement; section-local optimization explicitly operates over fixed prompt components and reports improved robustness in small-model settings.

The gap summary states: "Existing literature provides either strong search performance or improved robustness diagnostics... but there is still limited evidence on simple black-box pipelines that jointly enforce segment-level controllability, preserve known-strong prompt components, and remain implementation-light under a unified multi-task protocol."

SAPO's positioning: "SAPO sits at the intersection of static meta-prompting and structure-aware optimization. It preserves the deployment simplicity of black-box APO, while introducing explicit segment-level diagnosis and constrained updates that preserve strong components and localize revisions."

The paper formalizes the optimization objective. Given an initial prompt template P(0) with placeholder input, datasets are split into training and validation parts. Given one LLM M and task-dependent metric Qτ, the objective is: P* = arg max Qτ(P; Dval, M). At iteration t, the optimizer applies an update operator U to the current prompt: P(t+1) = U(P(t), Dtrain, Dval). The operator is acceptance-constrained: "if the best candidate generated at iteration t does not improve validation quality, the prompt is kept unchanged. This converts the procedure into a monotone, validation-gated search over prompt space, which directly targets robustness against destructive rewrites."

Prompt structure is represented as four explicit segments: S(P) = srole, scontext, stasks, soutput format.

The optimization loop consists of five steps:

1. Decompose the current prompt into segments role, context, tasks, output format.

2. Run Stage A: evaluate on train, extract top-5/bottom-5 evidence, infer weak/strong segments and recommendations.

3. Run Stage B: generate K constrained prompt candidates.

4. Evaluate candidates on validation; choose the best by score, with edit-distance tie-break.

5. Accept the candidate only if validation score improves; otherwise keep the current prompt.

The four segments map to distinct and operationally separable control dimensions in instruction-based LLM use:

• Role (srole): defines behavioral stance (e.g., classifier, summarizer, analyst) and can materially affect downstream behavior.

• Context (scontext): encodes task grounding, input injection, and domain constraints (including input placement). It controls what information is available and how the model conditions on it.

• Tasks (stasks): specifies actionable requirements and decision rules. It is the main locus for correcting underspecified or ambiguous instructions.

• Output format (soutput format): governs the response schema and formatting constraints. It is critical in tasks where metric outcomes depend on strict label/output conventions.

These four segments provide a compact decomposition that is expressive enough for heterogeneous NLP tasks while remaining small enough for stable, low-cost iterative optimization.

Stage A: Evidence extraction and segment-level diagnosis. At iteration t, the current prompt P(t) is evaluated on Dtrain: ŷi = M(P(t), xi), qi = Qτ(P(t); (xi, yi), M). Examples are ranked by qi, and two contrastive evidence sets are extracted: Bgood = Top-5(qi), Bbad = Bottom-5(qi). For discrete per-example metrics (e.g., ExactMatch with qi ∈ 0, 1), the method uses "class-aware evidence selection: Bgood is drawn first from positive examples (qi = 1), and Bbad is drawn first from negative examples (qi = 0). If one side has fewer than five examples, the remainder is backfilled from the global rank order. Given the evidence sets and segment decomposition, one LLM with static meta-prompts infers structured diagnostic outputs: weak segments, strong segments, and recommendations. Intuitively, strong segments are associated with consistently successful evidence, while weak segments are associated with failure cases and become primary targets for revision."

Stage B: Candidate synthesis. The same LLM receives current prompt, segment decomposition, weak/strong labels, and recommendations, then generates K improved prompt candidates. Candidate synthesis is explicitly constrained to preserve segments listed in strong segments and primarily modify segments listed in weak segments, reducing cross-segment interference. Each candidate is evaluated on Dval. The best candidate is accepted only if it improves the current validation score. Under score ties, the method selects the candidate with the smallest edit distance to the current prompt, which favors minimal edits and helps preserve validated prompt behavior.

Datasets and task coverage: SQuADv2 (extractive QA), TweetEval (social NLP classification), XSUM (abstractive summarization), CommonGen (constrained commonsense generation), and GSM8K (mathematical reasoning). Metrics: BERTScore F1 for XSUM, CG, SQuAD2; F1 for TweetEval; ExactMatch for GSM8K.

Models: GPT-3.5-Turbo and GPT-4o-mini.

Data split and protocol: "Each dataset is split into train/validation/test with sizes 150/100/all. The train split is used for top/bottom evidence extraction, validation is used for candidate selection during optimization, and the held-out test split is used for final reporting. This separation is strictly enforced for all iterative methods to reduce selection leakage."

Generation settings and baselines: K = 5 candidates per iteration, sampling temperature of 0, fixed across all experiments and identical for SAPO and all iterative APO baselines. Baselines include Zero-shot prompting, APE, OPRO, EvoPrompt, GEPA, and StraGO.

Runs and reporting: Each method/model setup is run five times with different random seeds and data samples. Main tables report mean values across runs.

On GPT-3.5-Turbo: SAPO achieves the highest average score. It yields gains on SQuAD2, GSM8K, CommonGen, and TweetEval, while trailing GEPA on XSUM. The largest relative gain appears on GSM8K. Specific scores: SQuAD2 0.922, GSM8K 0.835, CG 0.911, TweetEval 0.575, XSum 0.852, Average 0.819.

On GPT-4o-mini: SAPO again delivers the highest average score and outperforms all compared baselines on all five datasets. Specific scores: SQuAD2 0.931, GSM8K 0.952, CG 0.875, TweetEval 0.593, XSum 0.789, Average 0.828.

The average gain relative to the best competing baseline by average score is +5.13% on GPT-3.5-Turbo and +7.25% on GPT-4o-mini.

Why segment-level updates help: "The observed gains are consistent with the hypothesis that explicit weak/strong segment separation reduces destructive interference between edits. Instead of globally rewriting prompts, SAPO localizes revisions to weak components while preserving high-performing structure."

When improvements are likely: Improvements are most pronounced when tasks depend on strict output conventions or multi-part instructions. In such settings, preserving strong format/context segments appears as important as improving task wording.

Limitations: A key limitation of this work is that SAPO uses a fixed and relatively small segment set. While this decomposition is practical and interpretable, it may not capture finer-grained prompt structure needed for some tasks. Future work should explore richer segment taxonomies and adaptive segment discovery, including LLM-based automatic generation of new segment types during optimization. Additionally, our current setup uses fixed top-5/bottom-5 evidence windows, which may miss finer-grained error modes.

The ablation study quantifies the contribution of individual prompt segments. All ablation families are negative on average, meaning the final SAPO prompt remains the strongest configuration overall. Key findings:

• Reverting tasks to initial wording causes the largest degradation (Avg Δ = −0.1038), with the strongest drop on GSM8K (−0.2800), confirming task-level refinement is the main driver of improvements.

• Removing response-format constraints hurts on average (Avg Δ = −0.0354).

• Role removal is unfavorable overall (Avg Δ = −0.0291).

• Context reversion is the least harmful perturbation (Avg Δ = −0.0031), suggesting context edits are comparatively stable, while tasks and output constraints are the most sensitive levers.

The paper concludes: "We presented SAPO, a modular APO framework that replaces monolithic rewrites with segment-level optimization. Under this protocol, SAPO achieves the strongest average performance against strong APO baselines and offers an interpretable pathway for future extensions in APO."

Based on the paper, here are specific improvements for AI systems:

Improvement: Replace monolithic prompt optimization with a four-segment decomposition (role, context, tasks, output format) and a two-stage generation pipeline.

What the improved system can do:

• Decompose any prompt into explicit segments before optimization

• Diagnose weak vs. strong segments using top-5/bottom-5 contrastive evidence from training data

• Generate candidates that preserve strong segments while revising only weak ones

• Accept updates only when validation performance improves (monotone, validation-gated search)

• Use edit-distance tie-breaking to favor minimal, conservative edits

Improvement: For discrete metrics (e.g., ExactMatch), select top-5 positive examples and bottom-5 negative examples separately, backfilling from global rank order when needed.

Improvement: Explicitly constrain candidate generation to preserve segments labeled strong and primarily modify segments labeled weak.

Improvement: Use one LLM with static, structured meta-prompts for segmentation, weakness analysis, and candidate generation (no dynamic prompt rewriting).

Improvement: Apply dataset-specific metrics within the optimization loop: BERTScore F1 for generation tasks, F1 for classification, ExactMatch for reasoning.

Improvement: Only accept a candidate if validation score strictly improves; otherwise keep the current prompt unchanged.

Improvement: Provide per-segment contribution analysis (role, context, tasks, output format) to identify which components drive performance.

Improvement: Use a call-level complexity proxy: SAPO requires T·Ntr + K·Nval calls per run, compared to T·K·Ntr + K·Nval for APO/StraGO.

Improvement: The method works consistently across GPT-3.5-Turbo and GPT-4o-mini without backbone-specific tuning.

Improvement: Use structured outputs (JSON-like) for diagnostic results: weak segments, strong segments, recommendations.

Summary of measurable improvements:

• Best average score across 5 datasets on both backbones

• +5.13% relative gain over strongest baseline on GPT-3.5-Turbo

• +7.25% relative gain on GPT-4o-mini

• Largest gains on GSM8K (+12.23% relative) and XSUM (+9.58% relative)

• Consistent wins on 9 out of 10 dataset-backbone combinations

Automatic Prompt Optimization (APO) often rewrites prompts monolithically, which can improve one behavior while degrading others. We present SAPO, a segment-level APO method that decomposes prompts into role, context, tasks, and output format, then applies targeted improvements based on top-5 and bottom-5 examples. The optimization loop uses one LLM with static meta-prompts and structured outputs for segmentation, weakness analysis, and candidate generation. We describe a train/validation protocol and a two-stage generation process: (1) segment-level diagnosis and recommendation extraction, (2) candidate synthesis constrained by weak/strong segment signals. Using the evaluation setup across SQuADv2, TweetEval, XSUM, CommonGen, and GSM8K on GPT-3.5-Turbo and GPT-4o-mini, SAPO achieves the best average score against Zero-shot and strong APO baselines including APE, OPRO, EvoPrompt, GEPA, and StraGO.