08b-prompt-handauthoring

name: 08b-prompt-handauthoring description: > Default reusable MLflow workshop path for instruction-shaped prompt iteration. Hand-authored prompt revisions guided by failing-scorer rationales from the first scored eval, with full-dataset re-eval and alias-gated promotion. Use when Step 4 evaluation reveals specific prompt-related failure modes (guideline misses, missing structure, wrong tone) AND failure_shape is instruction AND there are no L1 scorer failures. Cheaper, more transparent, and lower-risk than automated optimization. SDLC Step 8b — runs AFTER Step 4. Do NOT use for L1 scorer failures (route to architecture review) or when the quality gap is tool/retrieval-shaped. license: Apache-2.0 compatibility: "Requires Databricks workspace with MLflow >= 3.10.1 and Unity Catalog Prompt Registry. No GEPA / optimize_prompts dependency." clients: [ide_cli, genie_code] bundle_resource: none deploy_verb: none deploy_note: "Hand-authored prompt revisions re-registered + alias-gated in the UC Prompt Registry via the MLflow SDK; full-dataset re-eval on serverless. No bundle resource. Identical on both clients. See skills/genie-code-environment." coverage: full metadata: last_verified: "2026-04-15" volatility: medium upstream_sources: [] author: "prashanth-subrahmanyam" version: "1.0.0" domain: "genai-agents" pipeline_position: "S8b" consumes: "registered_prompts, evaluation_dataset, scorers, predict_fn, agent_spec, llm_role_endpoints" produces: "prompt_iteration_ran, prompt_handauthoring_iterations, prompt_handauthoring_template_diff_summaries" grounded_in: "https://docs.databricks.com/aws/en/mlflow3/genai/prompt-version-mgmt/prompt-registry, https://docs.databricks.com/aws/en/mlflow3/genai/eval-monitor/evaluate-app, docs/retrospective/three-gap-followup.md"

Hand-Authored Prompt Iteration (Default MLflow Workshop Path)

This is the default path for instruction-shaped prompt iteration in the reusable MLflow workshop. It produces hand-authored prompt revisions guided by failing-scorer rationales from the first scored eval, then validates them against the full evaluation dataset before promotion. It is intentionally cheap, transparent, and easy to roll back.

GEPA / automated optimization is NOT in the default flow. If the prompt explicitly declares prompt_iteration_strategy: gepa, route to 08-prompt-optimization instead. That skill owns the AI-Dev-Kit databricks-mlflow-evaluation upstream lineage for optimize_prompts() / GEPA guidance and is maintained as the optional advanced path.

Do I Need This Skill?

Use 08b only when all of the following are true. Otherwise go back to the listed SDLC step.

Refusal Conditions

Refuse to run and surface a typed remediation hint when:

1. target_scorers is empty. There is nothing to iterate against. The first scored eval did not record any failing scorers, so a prompt change cannot be validated. Route the operator back to Step 4 (re-run eval) or to architecture review if the eval itself is misconfigured.
2. target_scorers contains an L1 scorer failure. L1 scorers (safety, PII, hard structural contracts) are not safe to iterate against with a prompt rewrite — a passing prompt that papers over an L1 failure can mask a regression. Route to architecture review (e.g., re-design the agent's tool contract, add input/output guardrails, change retrieval) rather than re-authoring the system prompt.
3. reflection_lm_large_context_probe preflight fails. The role-bound reflection model cannot summarize a representative diff payload. Without a working diff-summary path, hand-authoring becomes opaque and template diff summaries cannot be captured. Route the operator to bind a larger-context model to the reflection_lm role.

Skill Parameters

The Hand-Authored Loop

This skill is not an autonomous optimizer. It runs a structured loop where every prompt change is authored by the operator (with LLM assistance for diff summaries) and validated against the full eval dataset. There is no inner search, no candidate scoring, and no reward signal feeding back into the LLM.

┌──────────────────────────────────────────────────────────────────┐
│ 1. Read failing_scorers_if_regressed from first_scored_eval      │
│ 2. Refuse if target_scorers empty OR has L1 failures             │
│ 3. Preflight: reflection_lm_large_context_probe                  │
│ 4. For each iteration (cap at 3 by default):                     │
│    a. Operator authors candidate prompt revision                 │
│    b. LLM-assisted diff summary (reflection_lm role)             │
│    c. Run full-dataset re-eval (NOT a holdout)                   │
│    d. Compare every target scorer mean to baseline               │
│    e. Capture template_diff_summary into state                   │
│ 5. Promote @staging → @production iff all target scorer means    │
│    meet or beat baseline; else leave @staging in place           │
└──────────────────────────────────────────────────────────────────┘

Step 1 — Read failing scorers

Pull failing_scorers_if_regressed from the live state file's first_scored_eval ## Captured Resource IDs block. This list is the input to target_scorers. Resolve each scorer name to its full definition via agent_spec.governance.scorer_suite[*].

Step 2 — Refuse on empty or L1 failures

Apply the refusal conditions above. Empty target_scorers and L1-classified failures are both routed out of this skill; do not proceed.

Step 3 — Reflection LM preflight

Read llm_role_endpoints[reflection_lm_role].endpoint from state. Run the reflection_lm_large_context_probe preflight (synthetic 80k- and 150k-character probe payloads). Abort with a typed remediation hint if either probe fails — diff summaries cannot be captured against a too-small-context model, and silent truncation would corrupt the iteration record.

Step 4 — Author candidate prompt revisions

For each iteration:

1. The operator hand-authors a candidate revision to the prompt body. The reflection LM is used only to produce a structured diff summary (what guidelines were added, what tone constraints were tightened, what examples were inserted). It is not used to autonomously generate or score candidate prompts.
2. Register the candidate as a new prompt version under the same UC name.
3. Set the write_alias (e.g. @staging) to the new version.
4. Run the eval prompt identified by rerun_prompt_role against the full evaluation dataset — not a holdout, not a sample. Holdout splits are explicitly out of scope: the dataset is the canonical contract and we re-evaluate the entire thing each iteration.
5. Capture an iteration record: candidate version number, target-scorer means before/after, and the LLM-generated template_diff_summary string.

Default iteration cap is 3. After 3 iterations without all target scorer means meeting or beating baseline, exit with iteration_did_not_beat_baseline and leave @production untouched.

Step 5 — Promote (or don't)

Promote @staging → @production only if every target scorer mean meets or beats the baseline mean on the full-dataset re-eval. A single regressed target scorer mean blocks promotion. Partial wins are not promoted; the operator can choose to keep @staging for further iteration or revert.

When promotion is skipped, capture the reason and the per-scorer means in the iteration record so the next operator can pick up where this one left off.

State Capture Contract

Capture the following into ## Captured Resource IDs of the calling prompt's state section:

The state-file capture is the durable record of what was hand-authored and why. Do not collapse iterations into a single summary.

No External Model Fallback

This skill never falls back to Anthropic, OpenAI, or any BYO external model. The reflection_lm role is bound to a workspace-resident endpoint (typically databricks-claude-sonnet-4-6 or equivalent) via llm_role_endpoints in state. If that endpoint is unavailable or the preflight fails, the skill aborts — it does not silently retry against an external provider. Workspace policy and data residency requirements forbid the fallback path.

Validation Checklist

• First scored eval has run and recorded failing_scorers_if_regressed and failure_shape.
• failure_shape == "instruction" and there are no L1 scorer failures in target_scorers.
• prompt_ref resolves via prompts:/{uc_catalog}.{uc_agent_schema}.<name>@<alias> (note: uc_agent_schema, not use_case_slug).
• reflection_lm_large_context_probe preflight passed (80k + 150k payloads).
• Re-eval ran on the full dataset, not a holdout split.
• Each iteration's template_diff_summary was captured into state.
• Promotion happened only when every target scorer mean met or beat baseline.
• prompt_iteration_ran, prompt_handauthoring_iterations, and prompt_handauthoring_template_diff_summaries are present in ## Captured Resource IDs.
• No call was made to mlflow.genai.optimize_prompts() or GepaPromptOptimizer.
• No call was made to an external (non-Databricks-workspace) model serving endpoint.

Do's and Don'ts

Related Skills

• 08-prompt-optimization — optional/advanced GEPA path, only when the prompt declares prompt_iteration_strategy: gepa.
• 01-prompt-registry — prompt registration prerequisite.
• 03-scorers-and-judges — scorer suite (including L1 classification).
• 04-evaluation-runs — first scored eval and re-eval mechanics.

Version History