rag-retrieval

name: rag-retrieval description: RAG & retrieval engineering capability pack. Gives AI agents the judgment rules a senior retrieval engineer applies automatically — chunking strategy selection, embedding model choice, vector database routing, hybrid search with Reciprocal Rank Fusion, two-stage cross-encoder reranking, GraphRAG, and reference-based + LLM-as-judge RAG evaluation. Research-grounded rules with specific numbers from chunking benchmarks, embedding/reranker/vector-DB comparisons, and Ragas-style evaluation. Use for any RAG pipeline design, retrieval quality debugging, chunking/embedding/vector-DB selection, hybrid search fusion, reranker selection, or RAG eval task. keywords: ["RAG", "检索增强", "retrieval", "检索", "chunking", "分块", "embedding", "嵌入", "向量数据库", "vector database", "reranker", "重排序", "hybrid search", "混合检索", "RRF", "BM25", "pgvector", "GraphRAG", "faithfulness", "向量检索"] type: reference-based

CONSUMES: User RAG/retrieval task + corpus description (size, format, language, domain) + optional existing pipeline config (chunker, embedder, vector DB, reranker, eval suite) PRODUCES: Applied retrieval judgment rules + chunking strategy decision + embedding model selection + vector DB routing + hybrid-search/RRF config + reranker selection + RAG eval suite with target thresholds

RAG & Retrieval Engineering Capability Pack

Version: 0.1.0 Compatibility: Claude Code (Phase 1); Codex / Cursor / Gemini in Phase 3 License: Apache 2.0

What This Pack Does

AI agents build RAG pipelines by copying a tutorial: fixed-size chunks, text-embedding-3-small, Chroma, top-k similarity, and a prompt. They reach for semantic chunking because it sounds advanced (it benchmarked < 55% vs recursive-512's 69%). They fuse BM25 and vector scores by adding them directly (mathematically invalid — BM25 is unbounded). They rerank the top-200 (paying latency for ~10% of the accuracy gain). They never separate retrieval evaluation from generation evaluation, and they report a single blended "RAG score" with no domain-calibrated Faithfulness gate — so they can't tell whether the retriever or the generator is the problem, or whether the answer is even grounded.

This pack embeds the judgment rules retrieval engineers apply automatically — rules grounded in 2026 chunking benchmarks, embedding/reranker/vector-DB comparisons, and Ragas-style evaluation, with the specific numbers a no-pack LLM would not produce.

Pack = retrieval judgment. Your workflow system = process constraints. No overlap.

Cross-Cutting Rule: Evaluate Retrieval and Generation Separately, and Gate Faithfulness on a Domain Threshold

A RAG system has two failure surfaces that MUST be measured independently: retrieval (did we fetch the right chunks?) and generation (did the answer stay grounded in them?). Never report a single blended "RAG score." Retrieval is measured with reference-based IR metrics (Precision@k, Recall@k, MRR, nDCG@k) or LLM-judged Context Precision/Recall. Generation is measured with Faithfulness/Groundedness and Answer Relevance. Faithfulness is the fraction of answer claims supported by the retrieved context; the lower it is, the larger the share of unsupported (potentially fabricated) claims. 1.0 is the aspirational target, but Faithfulness is a semi-deterministic LLM-judge score (run variance alone keeps a perfectly-grounded answer below 1.0), so gate on a domain threshold averaged over the eval suite, not a strict ==1.0: block (P0) below ~0.85 general / ~0.90 regulated (finance/health/legal), treat 0.85–0.99 as a P1 risk band, and pair with Answer Relevance ≥ 0.90. Source: findings.md "Rigorous Validation" + "Actionable Recommendations" [35, 36]; Ragas faithfulness docs + 2026 RAG-eval threshold guidance (0.8 general / 0.85 customer-facing / 0.9+ regulated), retrieved 2026-06-13

This rule applies to every RAG eval, every "why is my RAG wrong?" debug (always split: bad retrieval vs bad generation), and every production gate. It is surfaced here because a blended score hides whether the retriever or the generator is the problem — and that determines the entire fix.

Step 0: Context Detection

When the user mentions RAG/retrieval work, detect the context and load the right reference:

Validation Script (deterministic checks — do not hand-recompute)

When the user provides a machine-readable pipeline config, run the linter instead of mentally re-deriving the thresholds:

bash scripts/rag-config-lint.sh <path-to-pipeline-config>

It reads a flat key=value (or JSON) config and emits P0/P1/P2 findings with grounded numbers, exiting 1 on any P0 (raw-score fusion; semantic chunking on academic or unspecified-doc-type corpora; Faithfulness gate below the domain floor — < 0.85 general / < 0.90 regulated), **2 on P1-only** (candidate pool > 50, eval suite < 100 queries, Faithfulness gate in the 0.85–0.99 review band), 0 when clean. P2s cover dedicated vector DB < 100M vectors and missing Contextual Retrieval on legal/medical/academic corpora. The deterministic checks live in code (QUALITY-BAR A10) — the prose rules in references/ remain the source of the why and the judgment calls the script cannot make.

Step 1: Apply Rules

After loading the relevant reference file(s):

1. Read the reference completely — do not skim
2. Apply each rule as a judgment check against the user's pipeline, config, or request
3. For each violated rule: state the violation clearly, then give the specific fix with the grounded number/command
4. Enforce the Cross-Cutting Rule on every eval setup and every retrieval-quality debug — split retrieval vs generation before recommending a fix
5. Always cite the specific threshold/number from the reference — a recommendation without the grounded number is generic advice the user could have gotten from any LLM

Output format per finding:

[P0] Rule CH4 (chunking): Pipeline uses semantic chunking on academic docs — benchmarked < 55% accuracy vs recursive-512's 69% under equal context budget.
→ Switch to Recursive Character Splitting at 512 tokens with 10–20% overlap.

[P1] Rule HR3 (hybrid): BM25 and cosine scores are summed directly. BM25 is unbounded; it dominates the fusion.
→ Fuse by rank with RRF (k=60), not by raw score.

Step 2: Output

Produce a structured retrieval review:

## RAG / Retrieval Review: [pipeline or stage reviewed]

### P0 — Blocking (will produce wrong or hallucinated results)
- [finding + specific fix + grounded number]

### P1 — Required (fix before trusting retrieval quality)
- [finding + specific fix + grounded number]

### P2 — Advisory (improves quality/latency/cost)
- [finding + specific fix + grounded number]

### Retrieval vs Generation Split (Cross-Cutting Rule)
- Retrieval metrics + targets: [Precision@k / Recall@k / MRR / nDCG@k / Context Precision/Recall]
- Generation metrics + targets: [Faithfulness ≥ domain threshold (~0.85 general / ~0.90 regulated; 1.0 aspirational), Answer Relevance≥0.90, Groundedness≥0.95]

### Pipeline Blueprint (if full design requested)
- Chunking → Embedding → Vector DB → Hybrid + RRF → Reranker → Eval gate

Anti-Skip Table

Tool / Model Quick Reference