databricks-ads-session

name: databricks-ads-session description: Conduct Azure Databricks Architecture Design Sessions (ADS). Orchestrate a structured, multi-turn conversation to gather solution requirements from users across any industry or starting point (on-prem migration, IoT, data warehousing, ML/AI, streaming, etc.), then generate a Databricks-centric architecture diagram as PNG. Use when the user wants to (1) design an Azure Databricks solution, (2) run an architecture design session, (3) scope a Databricks migration or greenfield project, (4) gather requirements for a data platform, or (5) generate an Azure Databricks architecture diagram from requirements. license: MIT compatibility: Works with Claude Code, GitHub Copilot, VS Code, Cursor, and any Agent Skills compatible tool. PNG export requires Node.js (npx @mermaid-js/mermaid-cli). Fallback is Mermaid in Markdown preview. metadata: author: community version: "3.0" domain: Azure Databricks

Azure Databricks ADS Session

This skill provides domain-specific knowledge for Azure Databricks to be used within an Architecture Design Session. The ADS methodology (persona, pacing, session structure, decision narration, trade-off framework, self-critique) is defined in the runtime system prompt — this skill supplies the Databricks-specific questions, patterns, components, and references that the methodology operates on.

Domain: Azure Databricks

This skill covers the Azure Databricks data platform including:

• Data Engineering: LakeFlow Connect, LakeFlow Jobs, LakeFlow Spark Declarative Pipelines (DLT), Auto Loader, Structured Streaming, Apache Flink, Delta Lake
• Data Warehousing: SQL Warehouse (Serverless/Pro/Classic), Lakehouse Federation, dbt integration, materialized views
• AI/ML: Mosaic AI (Model Serving, Feature Store, AI Gateway), MLflow 3.0, serverless GPU compute, distributed training
• GenAI: Mosaic AI Agent Framework, Agent Bricks, Vector Search, MCP Servers, AI Gateway with guardrails
• Governance: Unity Catalog, Delta Sharing, ABAC, column-level masking, data lineage, Compatibility Mode
• Infrastructure: Serverless Workspace, Classic VNet-injected workspace, ADLS Gen2, Azure Key Vault, Microsoft Entra ID

Phase-Specific Databricks Questions

Phase 1: Context Discovery

Ask about:

• Business problem or opportunity driving this initiative
• Industry and regulatory context
• Greenfield project vs. migration from existing system
• Key stakeholders and decision-makers
• Timeline and budget constraints
• Success criteria (what does "done" look like?)
• KPIs, latency targets, and cost envelope

Adapt: If user mentions migration, read references/migration-patterns.md. If user names a specific industry, read references/industry-templates.md for starter context.

Phase 2: Current Landscape

Ask about:

• Data sources (databases, APIs, files, streams, SaaS platforms)
• Current data platform (if migrating: Hadoop, Snowflake, on-prem SQL, etc.)
• Data volumes and growth rate
• Real-time vs. batch requirements
• Data governance and cataloging needs (Unity Catalog considerations)
• Sensitive data classification (PII, PHI, financial)
• Unstructured data (documents, PDFs, images, audio) for AI processing

See references/probing-questions.md for deep-dive question banks when the user's answers are vague or incomplete.

Phase 3: Security & Networking

Ask about:

• Network topology (VNet injection, private endpoints, hub-spoke)
• Identity provider (Entra ID, federation, SCIM provisioning)
• Data access control model (table-level, row-level, column-level, attribute-based access control / ABAC)
• Regulatory compliance (HIPAA, SOC2, GDPR, FedRAMP, industry-specific)
• Encryption requirements (at-rest, in-transit, customer-managed keys)
• Secrets management approach

Phase 4: Operational Requirements

Ask about:

• HA/DR requirements and RPO/RTO targets
• Multi-region or single-region deployment
• Cost optimization priorities (reserved capacity, spot instances, serverless compute, FinOps strategy)
• Workspace deployment model (Serverless Workspace vs Classic with VNet injection)
• Monitoring and alerting requirements
• Environment strategy (dev/staging/prod workspace separation)
• Tagging and cost allocation strategy
• Operating model: who owns data products, who approves access, who triages incidents

Use references/trade-offs-and-failure-modes.md for domain-specific failure scenarios to raise proactively during this phase.

Readiness Gate

After each phase, internally track information completeness using references/readiness-checklist.md.

Decision Logic

IF all must-have items are gathered:
    → Proceed to diagram generation
IF some should-have items are missing:
    → State assumptions explicitly, ask user to confirm or correct
IF must-have items are missing:
    → Ask targeted follow-up questions (max 2 additional turns)
IF user says "just generate something" or expresses impatience:
    → Generate with sensible defaults, document all assumptions

Always tell the user what you know and what you're assuming before generating.

Databricks Diagram Components

When generating diagrams, use these Databricks-specific node shapes (extends the generic style guide from the architecture-diagramming skill):

Pattern Selection

Match gathered requirements to a Databricks architecture pattern. Read references/databricks-patterns.md for the pattern catalog.

Common patterns:

Generating the Diagram

Generate a Mermaid flowchart diagram based on the gathered requirements. Use the pattern templates in references/databricks-patterns.md as a starting point, then customize based on the specific requirements gathered.

Follow the architecture-diagramming skill's style guide for general Mermaid conventions (arrow styles, subgraph naming, layout direction). Apply the Databricks-specific node shapes listed above.

For rendering, Architecture Recap format, and iteration workflow, defer to the architecture-diagramming skill.

Optional: Workload Profiling

These questions are not a mandatory phase — the customer may or may not raise workload-specific topics during the session. Have this content ready to deploy when the conversation naturally moves toward workloads, but do not force it as a separate phase.

If the customer discusses workloads, ask about:

• ETL/ELT pipelines (complexity, frequency, SLAs)
• ML/AI workloads (training, inference, MLOps maturity, Mosaic AI)
• GenAI applications (RAG, chatbots, AI agents, document intelligence, Mosaic AI Agent Bricks)
• BI/reporting and self-service analytics (tools, user count, concurrency, AI/BI Genie)
• Streaming/real-time analytics requirements
• SQL analytics workloads (ad-hoc queries, dashboards)
• Data application hosting needs (Databricks Apps, custom UIs)
• Notebook/interactive development needs
• CI/CD and DevOps practices for data engineering (DABs, Azure DevOps, GitHub Actions)

If a workload area warrants deeper exploration, offer a Technical Deep-Dive using references/technical-deep-dives.md.

Databricks Expertise

You know Databricks inside-out — the tradeoffs between serverless and provisioned, when LakeFlow Connect beats ADF, why Liquid Clustering replaced partitioning. Connect technical decisions to business outcomes: "LakeFlow Spark Declarative Pipelines" isn't just a product name — it's fewer pipeline engineers and faster time-to-insight. Translate tech into value.

Reference Files

Scripts