Explore AI Agent Skills & Claude Prompts

Schema-level anomaly detection for Databricks Unity Catalog using the Data Quality API (Public Preview). Automatically monitors table freshness and completeness using ML models. Use when setting up schema-wide data reliability monitoring, detecting stale or incomplete tables, configuring anomaly detection alerts, or querying the system results table. **Auto-triggered by Silver and Gold layer setup workflows** to ensure every new schema has baseline freshness/completeness monitoring from day one.

project-planning

Create multi-phase project plans for Databricks data platform solutions with Agent Domain Framework and Agent Layer Architecture. Includes interactive Quick Start with key decisions, industry-specific domain patterns, complete phase document templates (Use Cases, Agents, Frontend), Genie Space integration patterns, deployment order requirements, and worked examples. Use when planning any Databricks solution post-Gold layer — observability, analytics, agent-based frameworks, or multi-artifact projects.

06-table-documentation

Comprehensive documentation standards for Gold layer tables including naming conventions, column descriptions, and metadata requirements. Use when creating Gold layer tables, columns, or writing documentation to ensure dual-purpose descriptions that serve both business users and technical users (including LLMs like Genie). Includes YAML schema consultation patterns, surrogate key patterns, SCD Type 2 documentation, and implementation guidance for Silver table naming conventions.

07-design-validation

Cross-validation of Gold layer design artifacts during the design phase. Use when validating that YAML schemas, ERDs, lineage CSVs, and PK/FK references are internally consistent before handing off to implementation. Catches design-time inconsistencies (e.g., column in ERD but not in YAML, FK referencing non-existent table) that would otherwise surface as runtime bugs.

mlflow-genai-evaluation

MLflow 3 GenAI evaluation patterns with LLM judges, MemAlign judge alignment, GEPA prompt optimization, custom scorers with _extract_response_text() helper, Databricks SDK for scorer LLM calls, metric aliases, threshold checking, 4-6 guidelines best practice, foundation model endpoints. Use when implementing agent evaluation pipelines, creating custom LLM judges, aligning judges with domain feedback, optimizing prompts, setting up UC trace ingestion for production monitoring, or troubleshooting evaluation errors.

mlflow-genai-foundation

Core MLflow 3.0 GenAI patterns for Databricks agents - model signatures, tracing fundamentals, evaluation basics, prompt registry basics. Foundational patterns shared across all agent skills. Triggers on "MLflow GenAI", "MLflow 3.0", "model signature", "autolog", "mlflow.trace".

04-conformed-dimensions

Enterprise integration patterns for Gold layer dimensional models. Covers conformed dimensions, the enterprise data warehouse bus matrix, shrunken/rollup dimensions, conformed facts, and drill-across query patterns. Use when planning dimensions shared across multiple fact tables, creating a bus matrix for enterprise integration, designing rollup dimensions, or enabling cross-process analytics. Triggers on "conformed dimension", "bus matrix", "drill-across", "shrunken dimension", "rollup", "enterprise integration", "cross-process".

04-grain-validation

Pre-merge grain validation patterns for Gold layer fact tables. Use when creating fact table merge scripts to validate that DataFrame grain matches DDL PRIMARY KEY, prevent transaction vs aggregated confusion, and catch grain mismatches before MERGE. Includes grain inference functions, pre-merge validation, validation SQL, and common grain mismatch errors.

03-fact-table-patterns

Advanced fact table design patterns for Gold layer modeling. Covers measure additivity classification (additive, semi-additive, non-additive), factless fact tables, accumulating snapshot facts, consolidated fact tables, header/line fact patterns, late-arriving facts and dimensions, and NULL handling in measures. Use when designing fact tables beyond basic transaction/aggregate patterns, classifying measure types, or handling complex fact scenarios. Triggers on "fact pattern", "factless", "accumulating snapshot", "measure additivity", "semi-additive", "late arriving", "consolidated fact".

05-erd-diagrams

Patterns for creating clean, professional Mermaid ERD diagrams for data modeling documentation. Use when documenting Gold layer data models, creating master ERDs for complete models, domain-specific ERDs for focused views, or summary ERDs for large models (20+ tables). Includes organization strategies, syntax standards, relationship patterns, and cross-domain reference patterns.

05-schema-validation

Runtime schema validation patterns for Gold layer merge scripts. Use when creating merge scripts to ensure DataFrame columns match target DDL schemas, validate column mappings before MERGE operations, and catch schema issues before deployment. Includes the DDL-first workflow, validate_merge_schema() helper, explicit column mapping, DDL schema reader, and pre-deployment validation script. Addresses the "three sources of truth" problem by enforcing DDL as runtime truth.

02-merge-patterns

Provides production-grade patterns for Gold layer MERGE operations from Silver to Gold tables. Covers column mapping, schema evolution, SCD Type 1/2 patterns, fact table aggregation, and preventing variable naming conflicts with PySpark functions. Use when creating Gold layer MERGE operations, handling column name differences between Silver and Gold, implementing SCD Type 1/2 dimensions, aggregating fact tables, or troubleshooting MERGE errors. Triggers on "Gold merge", "MERGE operation", "upsert Gold", "SCD Type 1", "SCD Type 2", "fact table merge", "Silver to Gold", "column mapping", "schema evolution".