agentsop-prompt-compilation

name: agentsop-prompt-compilation version: 0.1.0 phase: D tier: core frequency: medium status: opinionated layer: enhance-overlay decision_layer_only: true defers_implementation_to: ["dspy", "dspy-sop", "metric-design"] description: The compile-readiness gate for prompt auto-optimization. Decide whether you have earned the right to run an optimizer (DSPy MIPROv2 / GEPA / BootstrapFewShot) before spending compute. Two preconditions only — a real metric, and enough examples for the optimizer you picked. Garbage metric in, garbage prompt out. Pick the optimizer by data scale; GEPA inverts the scale assumption (~10 examples + textual feedback).

prompt-compilation — The Compile-Readiness Gate

"It's unproductive to launch optimization runs using a poorly designed program or a bad metric." — DSPy core team [dspy.ai/learn/optimization/overview/]

"Compile when you can measure. The optimizer maximizes your metric — garbage metric in, garbage prompt out." — this skill's operating principle (synthesized from the line above + DSPy Case C)

This is an enhancement-overlay decision skill. It answers exactly one question the broad [[dspy]] library skill buries under API surface: have you earned the right to run an optimizer yet, and which one? It produces a go / no-go gate plus an optimizer pick. It defers every implementation detail — Signature syntax, module choice, compile() calls, save/deploy — to [[dspy]] and the full workflow in [[agentsop-dspy]]. It defers metric construction to [[agentsop-metric-design]]; this skill only checks the metric exists and is validated, then uses it as the gate.

The trap it removes: people reach for MIPROv2(auto="heavy") because the API is right there, before they have a metric worth maximizing or enough data to avoid memorization. Compilation is a hyperparameter search costing hundreds-to-thousands of LM calls ($2–$40+, minutes-to-hours) [dspy.ai/faqs/]. Spending that on an un-validated metric or 8 examples is pure waste.

1. 何时激活 (When to Activate)

Activate when all three of these are plausibly true (the gate then confirms them):

• A hand-tuned prompt has plateaued. The team has manually iterated few-shot examples / wording past the point of obvious returns. Symptom from [[agentsop-dspy]] §1: "the team manually tunes few-shot examples; a metric exists but isn't being used to drive prompt design."
• A metric exists (or can be built). There is a metric(example, pred) -> bool|float — or one can be written and human-validated. Without this, do not activate; the optimizer has nothing to maximize.
• Labeled examples exist. There is a dev set. The count determines which optimizer is even legal (§3, §4.2).

Concrete triggers in intent or codebase:

Do NOT activate when:

• The task is one-shot or the Signature/I-O contract is still churning daily — compile only after it stabilizes [dspy.ai/learn/optimization/overview/]; otherwise you pay compile cost for prompts you'll throw away.
• No metric is possible and none will be built — then this is verbose prompting, not compilation. Route to [[agentsop-metric-design]] first; if the user refuses any success criterion, the gate stays closed.
• Compliance requires verbatim human-authored prompts — optimized prompts are machine-generated artifacts.
• You only need parse safety (a typed Signature), not quality optimization — that is [[agentsop-dspy]] Stage 1 and the signature-design overlay. Promoting prose to a typed Signature and compiling it are two different gates.

2. 核心心智模型 (Core Mental Model)

"Compile when you can measure. The optimizer maximizes your metric — garbage metric in, garbage prompt out."

An optimizer (MIPROv2, GEPA, BootstrapFewShot) is a black-box search over prompt instructions + few-shot demos that maximizes metric(pred, example). It has no taste. It will faithfully chase whatever the metric rewards, biases and all. From [[agentsop-dspy]] Case C: "DSPy will optimize toward whatever the metric rewards. A bad metric becomes a bad program at scale." Two corollaries make this a gate, not a step:

1. The metric is the precondition, not a tunable. Before any compute is spent, the metric must (a) exist and (b) agree with human judgment on ≥20 spot-checks. An un-validated metric means the expensive search optimizes the metric's blind spot. This is non-negotiable [dspy.ai/learn/evaluation/metrics/; Case C]. Construction is [[agentsop-metric-design]]'s job; this skill only checks the receipt.

2. Data scale is a hard floor, not a preference. Below the optimizer's example floor you are not training, you are memorizing. The DSPy 20/80 train/val split exists because "prompt-based optimizers often overfit to small training sets" [dspy.ai/learn/optimization/overview/]. The floor differs per optimizer (§4.2).

The two-gate picture

            ┌──────────────────────────────────────────────┐
 Gate 1     │  METRIC: exists?  AND  human-validated ≥20?   │  ── No ─► STOP. Build/validate metric ([[agentsop-metric-design]]).
 (measure)  └──────────────────────────────────────────────┘
                              │ Yes
                              ▼
            ┌──────────────────────────────────────────────┐
 Gate 2     │  EXAMPLES: ≥ floor for the optimizer I want?  │  ── No ─► Pick a lower-floor optimizer, collect data,
 (data)     └──────────────────────────────────────────────┘           or STOP (use LabeledFewShot as a floor).
                              │ Yes
                              ▼
                   COMPILE (cheap probe first: auto="light")

The cost reality (why the gate is worth having)

Compilation runs hundreds-to-thousands of LM calls. Reference run: ~3.2k API calls, $2–$3, 6–20 min for auto="light"; auto="heavy" on 1000+ examples can hit tens of dollars and hours [dspy.ai/faqs/]. Cost scales with num_trials × |trainset| × |program LM calls|. The gate's entire job is to stop you spending that on a metric or dataset that cannot pay it back.

3. SOP (Standard Operating Procedure)

0. Confirm activation (§1): plateaued hand-tuning + metric + examples.
1. GATE 1 — metric exists AND validated ≥20 human spot-checks?   [hard]
2. GATE 2 — example count ≥ floor for the chosen optimizer?      [hard]
3. PICK    — choose optimizer by data scale + feedback signal.   (§4.2)
4. BUDGET  — estimate cost; cap it; choose a cheap optimizer LM.
5. PROBE   — run auto="light" (or smallest config) as a signal.
6. DECIDE  — gain ≥ threshold → escalate; else go back, don't escalate.

Gate 1 — Metric (the "can you measure" gate)

• Is there a metric(example, pred, trace=None) -> bool|float? If not → STOP, route to [[agentsop-metric-design]].
• Has it been validated against a human on ≥20 spot-checks (≥30 if open-ended), with ≥80% agreement? If not → STOP and validate first. "Never compile against a metric you haven't human-validated on ≥20 spot-checks" [[[agentsop-dspy]] Case C; dspy.ai/learn/evaluation/metrics/].
• Is the metric a single holistic LLM-judge? Then it inherits length / self-preference / position bias and the optimizer will chase those biases — decompose it ([[agentsop-metric-design]]) before compiling [arxiv.org/pdf/2506.02592].

Exit: a validated metric callable + a calibration receipt. Otherwise the gate is closed; do not proceed.

Gate 2 — Examples (the "enough data" gate)

Count labeled examples. The DSPy-documented thresholds: ≥30 = minimum useful, ~300 = recommended, 200+ required for MIPROv2 to avoid overfitting [dspy.ai/learn/optimization/overview/]. The floor is per optimizer:

• < 10 → only LabeledFewShot(k=8) (no search, weakest) — usually means STOP, collect data.
• ~10+ with textual feedback → GEPA is legal and sample-efficient (this inverts the usual "more data" rule).
• ~30–50 → BootstrapFewShot / BootstrapFewShotWithRandomSearch.
• 200+ → MIPROv2.

Exit: the example count clears the floor of the optimizer you intend to run. If not, either drop to a lower-floor optimizer or stop.

Step 3 — Pick optimizer by data scale + signal

Use the table in §4.2. The single most important branch: do you have textual error feedback (test diffs, schema violations, judge rationales like "answer was verbose")? If yes, GEPA needs only ~10 examples and converges faster because it reflects on the text of the feedback, not just a scalar [dspy.ai/tutorials/gepa_ai_program/; arxiv.org/abs/2507.19457]. If no, fall back to the data-scale ladder.

Step 4 — Budget

Estimate before launching (§4.4). Cap spend. Use a cheap optimizer LM (e.g. gpt-4o-mini) even when the task LM is expensive — community-reported parity at a fraction of cost [github.com/stanfordnlp/dspy/issues/1596]. Set dspy.configure(track_usage=True) to log actual spend.

Step 5 — Probe cheap first

Run auto="light" (MIPROv2) or the smallest config first. Docs: "start with moderate values, observe behavior, and scale up only if you see clear gains" [github.com/stanfordnlp/dspy/issues/1596]. Never start at auto="heavy".

Step 6 — Decide on the probe result

• light gives <2% lift → do not escalate. The bottleneck is the program/metric, not the optimizer. Loop back to [[agentsop-dspy]] Stage 1 (signature ambiguous? wrong decomposition?) [dspy.ai/learn/optimization/overview/].
• light gives 2–10% lift → escalate to medium; go to heavy only if data ≥300 and you have a held-out test set distinct from val.
• Always confirm the final gain on a held-out test set, not the val set used during optimization.

4. 操作模型 (Operations)

4.1 — Operation registry (Trigger / Action / Output / Evidence)

OP-1 — ReadinessGate (entry, hard gate)

• Trigger: Anyone about to call an optimizer's compile().
• Action: Run Gate 1 (metric exists + validated ≥20) then Gate 2 (examples ≥ optimizer floor). Any failure → STOP.
• Output: Decision token compile-ready | not-ready + the failing gate.
• Evidence: [dspy.ai/learn/optimization/overview/]; [[agentsop-dspy]] Case C, §3 three-stage gate.

OP-2 — MetricExistsCheck

• Trigger: Gate 1.
• Action: Confirm a metric(example, pred, trace=None) -> bool|float exists. If absent → route to [[agentsop-metric-design]], gate stays closed.
• Output: Metric callable or a STOP.
• Evidence: Anti-pattern "compiling without a metric" [dspy.ai/learn/optimization/overview/].

OP-3 — MetricValidatedCheck

• Trigger: Metric exists (OP-2).
• Action: Require ≥20 (open-ended ≥30) human spot-checks at ≥80% agreement, plus a calibration receipt. If a single holistic LLM-judge, require decomposition first.
• Output: validated flag + receipt reference.
• Evidence: [[agentsop-dspy]] Case C step 4; [dspy.ai/learn/evaluation/metrics/]; [arxiv.org/pdf/2506.02592].

OP-4 — ExampleFloorCheck

• Trigger: Gate 2.
• Action: Count labeled examples; compare to the floor of the intended optimizer (LabeledFewShot any; GEPA ~10+feedback; Bootstrap ~30–50; MIPROv2 200+).
• Output: cleared | below-floor + the legal optimizer set.
• Evidence: [dspy.ai/learn/optimization/overview/] (30 min / 300 rec / 200+ MIPROv2).

OP-5 — OptimizerByDataScale

• Trigger: Both gates passed.
• Action: Pick from the §4.2 table by example count and feedback signal. Prefer GEPA when textual feedback exists.
• Output: Optimizer choice + config.
• Evidence: [dspy.ai/learn/optimization/optimizers/]; [dspy.ai/api/optimizers/GEPA/overview/].

OP-6 — CostBudgetGuard

• Trigger: Before launching compile.
• Action: Estimate num_trials × |trainset| × calls; cap spend; set a cheap optimizer LM; enable track_usage=True.
• Output: A budget ceiling + chosen optimizer LM.
• Evidence: [dspy.ai/faqs/]; [github.com/stanfordnlp/dspy/issues/1596].

OP-7 — CheapProbeFirst

• Trigger: Compile-ready, budget set.
• Action: Run auto="light" (or smallest config) first. Never start at heavy.
• Output: A cheap lift signal.
• Evidence: [github.com/stanfordnlp/dspy/issues/1596]; [[agentsop-dspy]] Case A.

OP-8 — EscalateOrReturn

• Trigger: Probe finished.
• Action: <2% lift → return to program/metric design (do NOT escalate). 2–10% → medium. heavy only if data ≥300 + held-out test set. Confirm gain on held-out test.
• Output: Escalate / iterate-back decision.
• Evidence: [[agentsop-dspy]] Case A; [dspy.ai/learn/optimization/overview/].

4.2 — Optimizer-by-data-scale table

4.3 — Readiness checklist (copy/paste)

GATE 1 — MEASURE
[ ] metric(example, pred, trace=None) -> bool|float exists
[ ] validated vs human: >=20 spot-checks (>=30 open-ended), >=80% agreement
[ ] receipt saved {n_spot_checks, agreement, judge_model, task_model, date}
[ ] NOT a lone holistic LLM-judge (else decompose via [[agentsop-metric-design]] first)

GATE 2 — DATA
[ ] labeled examples counted: N = ____
[ ] N clears the floor of the optimizer chosen below
[ ] (prompt-based optimizers) 20/80 train/val split planned; GEPA uses standard split

PICK + BUDGET
[ ] optimizer chosen by §4.2 (GEPA if textual feedback)
[ ] cost ceiling set; cheap optimizer LM chosen; track_usage=True
[ ] plan: probe auto="light" first, escalate only on >=2% lift, confirm on held-out test

4.4 — Cost budget heuristics

Source: [dspy.ai/faqs/]; [github.com/stanfordnlp/dspy/issues/1596]; [arxiv.org/abs/2507.19457].

5. 困境决策案例 (Dilemma Cases)

Dilemma 1 — Optimizer cost vs gain: is compiling even worth it?

困境: A 3-stage RAG pipeline already hits 72% on dev after manual prompt tuning. MIPROv2(auto="heavy") would cost ~$40 and 4 hours. Worth it? (Adapted from [[agentsop-dspy]] Case A.)

约束:

• 250 labeled examples (above the MIPROv2 200-floor) [dspy.ai/learn/optimization/optimizers/].
• Prompts already hand-iterated — diminishing returns suspected.
• Pipeline LM = GPT-4o; per-call cost compounds at trial scale.

决策步骤:

1. Gate 1 first: were the prompts ever scored against the metric, or just eyeballed? If only eyeballed, even auto="light" (~$2) often yields large lifts over hand-tuned baselines (paper reports 25%/65% over standard few-shot) [arxiv.org/abs/2310.03714] — the metric was never actually driving design.
2. Probe light (~$2), never heavy (OP-7). It is a cheap signal for whether more compute helps.
3. <2% lift → do NOT escalate to heavy. Return to program/metric: signature ambiguous? is 3-stage the right decomposition? (OP-8) [github.com/stanfordnlp/dspy/issues/1596].
4. 2–10% lift → medium; heavy only if data ≥300 + a held-out test set distinct from val.
5. Use gpt-4o-mini as the optimizer LM even though the task LM is gpt-4o (OP-6) — community parity.

结果: The $40 heavy run is almost never the right first move. The $2 probe tells you whether to spend more or to go fix the program. Often the answer is "fix the program/metric first."

可提取的操作: OP-1, OP-6 CostBudgetGuard, OP-7 CheapProbeFirst, OP-8 EscalateOrReturn. Lesson: never open at heavy. Probe light with a cheap optimizer LM, and treat <2% as a signal to fix the program, not to add compute.

Dilemma 2 — Only 12 examples: GEPA inverts the data-scale assumption

困境: A code-fix agent has just 12 labeled examples, but each failing run produces rich textual feedback: the failing test diff, a schema-violation message, a linter error. The data-scale ladder says 12 < 30 → "STOP, collect data, you can't optimize." Is that right?

约束:

• 12 examples is below the BootstrapFewShot (~30) and MIPROv2 (200) floors.
• Collecting 200+ labeled examples is weeks of effort.
• But the failures emit machine-readable text feedback, not just a pass/fail scalar.

决策步骤:

1. Gate 1: a metric exists — pass/fail on the test plus a textual feedback string. Validate it (the test is ground truth; spot-check the feedback strings are accurate). Pass.
2. Gate 2 — do NOT apply the scalar-data ladder. The presence of textual feedback changes which optimizer is legal. GEPA needs only ~10 examples because it reflects on the content of the feedback, not a scalar gradient [dspy.ai/tutorials/gepa_ai_program/; arxiv.org/abs/2507.19457]. 12 ≥ ~10 → GEPA is legal.
3. Pick GEPA (OP-5): return dspy.Prediction(score=..., feedback="failing assert: expected X got Y") from the metric; this is GEPA's superpower [dspy.ai/api/optimizers/GEPA/overview/].
4. Probe cheap (OP-7), confirm lift on the held-out examples (OP-8).

结果: A dataset that is far too small for MIPROv2/Bootstrap is sufficient for GEPA. The "you need 200 examples" intuition is specific to scalar-feedback optimizers; textual feedback buys an order-of-magnitude in sample efficiency. GEPA reported beating MIPROv2 by 10–13% on benchmarks while being more sample-efficient [arxiv.org/abs/2507.19457].

可提取的操作: OP-4 ExampleFloorCheck, OP-5 OptimizerByDataScale. Lesson: the example floor is per-optimizer. If you can express why an output failed as text, GEPA inverts the data-scale assumption — ~10 examples suffice. Do not reflexively gate out small datasets that carry rich feedback.

6. 反模式与边界 (Anti-Patterns & Boundaries)

Anti-patterns

Boundaries (when this skill cannot help)

• No willingness to define any success criterion — no metric can exist; the gate stays closed. Problem definition first ([[agentsop-metric-design]], then scientific-critical-thinking).
• Parse-safety, not quality — if you only need a typed Signature so code can consume the output, that is the promote gate (signature-design / [[agentsop-dspy]] Stage 1), a different decision from compile.
• HOW to write/run the optimizer — class syntax, compile() args, save/deploy: that is [[dspy]] and [[agentsop-dspy]], not this overlay.
• Constructing the metric — decomposition, bias probes, calibration receipts: that is [[agentsop-metric-design]]. This skill only checks the receipt exists.
• Non-DSPy auto-optimization — e.g. OpenAI fine-tuning is a weights path, not prompt compilation; the readiness logic (metric + data floor) still applies but the operators differ (§7).

7. 跨框架对照 (Cross-Framework Mapping)

The readiness gate (metric + data floor) is framework-agnostic; the operators differ.

Decision summary: If you have a validated metric and examples clearing a floor, compile (DSPy). If you have a metric but it's never been validated, you are doing manual tuning dressed up — close Gate 1 first. If prompt optimization has plateaued and you have hundreds of stable examples and volume justifies it, consider fine-tuning (or BootstrapFinetune to distill an already-compiled program into a smaller model).

Combination patterns:

• prompt-compilation + [[agentsop-metric-design]]: this skill's Gate 1 is satisfied by a [[agentsop-metric-design]] calibration receipt. No receipt → gate closed.
• prompt-compilation + [[agentsop-dspy]]: this overlay is the sharpened version of [[agentsop-dspy]] §3 Stage 2→3 transition (the "are you allowed to compile yet?" boundary). After the gate opens, hand off to [[agentsop-dspy]] for the full compile/save/deploy workflow.
• prompt-compilation + [[dspy]]: [[dspy]] provides the optimizer APIs; this skill decides whether and which.

References

• references/R1-source-evidence.md — every claim traced to the local dspy-sop skill and the upstream DSPy docs it cites; overlap check vs [[dspy]] and [[agentsop-metric-design]].
• intermediate/operation_candidates.json — the 8 operations + tables in machine-readable form.

Citations: [dspy.ai/learn/optimization/overview/], [dspy.ai/learn/optimization/optimizers/], [dspy.ai/api/optimizers/MIPROv2/], [dspy.ai/api/optimizers/GEPA/overview/], [dspy.ai/api/optimizers/BootstrapFinetune/], [dspy.ai/learn/evaluation/metrics/], [dspy.ai/faqs/], [dspy.ai/cheatsheet/], [dspy.ai/tutorials/gepa_ai_program/], [arxiv.org/abs/2310.03714], [arxiv.org/abs/2507.19457], [arxiv.org/pdf/2506.02592], [github.com/stanfordnlp/dspy/issues/1596]. Primary source: /Users/5imp1ex/Desktop/Skill-Workplace/output/dspy-sop-skill/SKILL.md.