data-engineering

name: data-engineering description: Data engineering for AI capability pack. Gives AI agents the judgment rules for AI data pipelines — ETL/ELT ingestion (dlt, schema evolution), local-first transformation stacks (DuckDB, Polars, dbt) and train-serve skew defense, orchestration selection (Airflow 3.2 / Dagster 1.13 / Prefect 3.7), data quality gates (Great Expectations v1.0 / Soda Core v4), dimensional modeling and Slowly Changing Dimensions (SCD Type 0-6), and vector + streaming retrieval (metadata filtering, RRF, Kafka/Flink). Research-grounded rules with source citations. Use for any AI data pipeline, feature engineering, warehouse modeling, RAG context-lake, or real-time inference architecture task. keywords: ["数据工程", "data engineering", "数据管道", "data pipeline", "ETL", "ELT", "特征工程", "feature engineering", "dbt", "dlt", "DuckDB", "Polars", "数据质量", "data quality", "Great Expectations", "Soda", "编排", "orchestration", "Airflow", "Dagster", "Prefect", "维度建模", "SCD", "slowly changing dimensions", "向量数据库", "vector database", "RAG", "流式处理", "streaming", "Kafka", "Flink", "feature store", "schema evolution"] type: reference-based

CONSUMES: User data-pipeline task + source/destination description + optional existing pipeline configs, dbt models, warehouse schemas, or RAG/streaming setup PRODUCES: Applied data-engineering judgment rules + ingestion-pattern decisions (ETL vs ELT) + transformation/feature designs + orchestrator selection + data-quality gate configs + SCD/dimensional model + vector-filtering & streaming-inference architecture

Data Engineering for AI Capability Pack

Version: 0.1.0 Compatibility: Claude Code (Phase 1); Codex / Cursor / Gemini in Phase 3 License: Apache 2.0

What This Pack Does

AI agents build data pipelines by reaching for the first tool they remember. They default to Pandas where Polars would parallelize for free. They enforce rigid schemas on raw JSON before storage, discarding the raw history that model retraining needs. They write feature transformations once in a training notebook and again in a serving path — silently introducing train-serve skew. They pick Airflow for a lightweight agentic workflow that Prefect handles better. They run a global vector search and post-filter, then wonder why RAG returns zero results for a tenant. They build SCD Type 2 tables that scan ~160M rows because a query forgot is_current = true.

This pack embeds the judgment rules that data engineers apply automatically — rules grounded in 2026 tooling research (dlt 1.27.2, DuckDB v1.4-LTS, Polars, dbt-core, Apache Airflow 3.x / 3.2.2 docs, Dagster 1.13, Prefect 3.7, Great Expectations / GX Core 1.18.1, Soda Core 4.7.0, Apache Iceberg 1.10.1 / Delta Lake 4.1.0, Kafka/Flink) with source URLs + retrieval dates on every number (see each reference file's "Sources" footer).

Pack = data-engineering judgment. Your workflow system = process constraints. No overlap.

Cross-Cutting Rule: Train-Serve Skew is a Silent Killer

The transformation logic that prepares offline training data and the logic that prepares real-time inference payloads MUST be the same code, not two copies. When they diverge, the model produces invalid predictions in production while reporting high validation accuracy offline — silent degradation that no offline test catches. Enforce a single version-controlled transformation source (dbt model compiled into the warehouse, or a registered Feature View) that feeds BOTH the historical batch training table AND the low-latency serving table.

This rule applies to: feature engineering, dbt model design, feature-store registration, and any pipeline where the same feature is computed for both training and serving. It is surfaced here because burying it in one reference file is exactly how skew enters production.

Source: findings.md "Mitigation of Train-Serve Skew" [8] — dbt as a single version-controlled repository feeding both training tables and real-time serving.

Reference Index

Step 0: Context Detection

When the user mentions data-engineering work, detect the context and load the right reference:

Step 1: Apply Rules

After loading the relevant reference file(s):

1. Read the reference completely — do not skim
2. Apply each rule as a judgment check against the user's pipeline, config, schema, or request
3. For each violated rule: state the violation clearly, then give the specific fix (named tool, CLI command, or threshold from the reference)
4. Enforce the Train-Serve Skew cross-cutting rule on every feature-engineering or dbt design
5. Check determinismLevel annotations — they tell you how reproducible a decision is:
   • deterministic: architectural/classification decision (e.g., ETL-vs-ELT choice, SCD type assignment) — one correct answer given the inputs
   • semi-deterministic: config-driven but data-dependent (e.g., schema-evolution behavior, anomaly thresholds)
   • non-deterministic: runtime/streaming behavior whose outcome varies (e.g., async-inference latency, graph-islanding under strict filters)

Output format per finding:

[P0] Rule ING2 (ingestion): Raw JSON is being schema-enforced and discarded after transform.
→ Switch to ELT: write raw payloads to low-cost storage first, transform in-warehouse. ELT retains full raw history for model retraining; ETL discards it.

[P1] Rule TRN1 (transformation): Feature computed separately in training notebook and serving path.
→ Move the transformation into a single dbt model / registered Feature View feeding both. Train-serve skew causes silent production degradation.

Step 2: Output

Produce a structured data-engineering review:

## Data Engineering Review: [area reviewed]

### P0 — Blocking (must fix before pipeline goes to production)
- [finding + specific fix]

### P1 — Required (fix before trusting the data)
- [finding + specific fix]

### P2 — Advisory (improves robustness / cost)
- [finding + specific fix]

### Architecture Decision Log
[ingestion pattern chosen, orchestrator chosen, SCD types assigned, filtering strategy — each with the rule that drove it]

### Tool Recommendation
[dlt / DuckDB / Polars / dbt / Airflow|Dagster|Prefect / Great Expectations|Soda based on user context]

Anti-Skip Table

Tool Quick Reference

Versions pinned + verified 2026-06-13 (see each reference file's Sources footer for source URL + retrieval date). Pin in your lock file; do not bare---upgrade.

Validation Scripts (deterministic checks — scripts/)

Deterministic checks are code, not "punt to Claude." Run these instead of eyeballing: