rocky-new-adapter

name: rocky-new-adapter description: Adding a new warehouse or source adapter crate to the Rocky engine. Use when introducing a new target (e.g. BigQuery, Redshift, ClickHouse) or source (e.g. Airbyte, Singer). Covers crate layout, trait implementation, dialect specifics, auth, conformance tests, and factory registration.

Adding a new Rocky adapter

Rocky adapters are Cargo crates that implement traits from rocky-adapter-sdk. Existing examples:

Kind	Crate	What it talks to
Target (warehouse)	`rocky-databricks`	Databricks SQL Statement Execution API + Unity Catalog
Target (warehouse)	`rocky-snowflake`	Snowflake REST API
Target (local)	`rocky-duckdb`	Embedded DuckDB
Source	`rocky-fivetran`	Fivetran REST API

When to use this skill

Adding a new target adapter (warehouse): e.g. BigQuery, Redshift, ClickHouse, Iceberg
Adding a new source adapter: e.g. Airbyte, Singer, Stitch, database CDC
Forking an existing adapter to support a cousin backend (e.g. Databricks → Databricks Serverless)

Crate scaffold

Create engine/crates/rocky-<name>/ with a Cargo.toml registered in the workspace (engine/Cargo.toml members list). Use rocky-databricks or rocky-snowflake as a template for a warehouse adapter, rocky-fivetran for a source adapter.

Warehouse adapter files (mirror `rocky-databricks/src/`)

File	Responsibility
`lib.rs`	`pub mod` exports + crate-level docs
`adapter.rs`	`impl Adapter for <Name>Adapter` — the main trait from `rocky-adapter-sdk::traits`
`dialect.rs`	`impl SqlDialect` — dialect-specific SQL (quoting, type widening rules, `insert_overwrite_partition`, MV/dynamic-table DDL)
`connector.rs`	REST client or native connection — use `reqwest` for HTTP-based adapters
`auth.rs`	Credential handling (PAT, OAuth M2M, key-pair JWT, etc.) with env var auto-detection
`catalog.rs`	Catalog/schema CRUD (CREATE/ALTER/DESCRIBE)
`permissions.rs`	GRANT/REVOKE + SHOW GRANTS parsing
`batch.rs`	Bulk operations (e.g. `information_schema` UNION ALL batching)
`throttle.rs`	Rate limiting / adaptive concurrency if the backend needs it
`types.rs`	Native → internal type mapping

Source adapter files (mirror `rocky-fivetran/src/`)

File	Responsibility
`lib.rs`	Exports
`adapter.rs`	`impl SourceAdapter`
`client.rs`	Async REST client
`connector.rs`	Source discovery + filtering
`schema.rs`	Schema config parsing (often nested JSON)
`sync.rs`	Sync detection / staleness checks

Trait implementation

Traits live in engine/crates/rocky-adapter-sdk/src/traits.rs. The key ones:

Adapter — warehouse adapter core: execute_sql, describe_table, list_tables, create_catalog, …
SqlDialect — SQL generation specifics. Pay special attention to:
- is_safe_type_widening() — allowlist for ALTER TABLE type changes; conservative by default
- insert_overwrite_partition() — returns Vec<String> so multi-statement transactions work (Snowflake needs BEGIN/DELETE/INSERT/COMMIT)
- Materialized views + dynamic tables DDL
SourceAdapter — source discovery and sync staleness detection
Authenticator — if there's a credential flow worth unifying

Use #[async_trait] for all async trait impls.

Conformance tests

rocky-adapter-sdk/src/conformance.rs hosts a reusable test suite. Wire your adapter into it so rocky test-adapter <name> (the CLI command) validates the baseline behavior: CREATE/DROP, incremental copy, schema drift, permission reconcile.

Test pattern: add an integration test file in your crate that instantiates the adapter and runs the conformance suite. For credential-free local testing, gate behind an env var — see how rocky-databricks skips network tests when DATABRICKS_HOST is unset.

Factory registration

The CLI + config parser needs to know your adapter type string. Search for where existing adapter types are registered (typically in rocky-core/src/config.rs or the CLI's adapter dispatch) and add yours. The config block in rocky.toml will then accept:

[adapter]
type = "<name>"        # ← the string you registered
host = "${MY_HOST}"
# … adapter-specific fields

Auth patterns to copy

PAT-first, OAuth M2M fallback — see rocky-databricks/src/auth.rs. Tries PAT from env, falls back to client_credentials flow if PAT is absent.
Multi-method priority — see rocky-snowflake/src/auth.rs. OAuth (pre-supplied) > RS256 key-pair JWT > password.
Basic auth — see rocky-fivetran/src/client.rs. Simplest case — base64 of key:secret.
SQL identifier validation — always route catalog/schema/table names through rocky-sql/validation.rs before interpolation.

Dialect gotchas

The SqlDialect trait is where adapters actually diverge. Things to get right:

Quoting: backticks vs double-quotes vs brackets. rocky-databricks uses backticks for principals; most warehouses use the standard double-quote identifier.
Safe type widening allowlist: is_safe_type_widening(from, to) controls whether Rocky will ALTER TABLE or fall back to full refresh. Be conservative — data loss from an unsafe cast is worse than an extra full refresh.
Partition overwrite atomicity: Databricks has INSERT INTO ... REPLACE WHERE as a single atomic op. Snowflake and DuckDB need BEGIN; DELETE; INSERT; COMMIT; — hence the Vec<String> return type.
MV / dynamic table semantics: Databricks materialized views and Snowflake dynamic tables have different refresh models. Read engine/CLAUDE.md for the canonical DDL strings.

Adapter SDK process helper

rocky-adapter-sdk/src/process.rs provides helpers for out-of-process adapters (spawning an external binary, parsing JSON, handling exit codes). Useful if you're wrapping a dbt-like adapter or a CLI tool instead of speaking a REST API directly.

Shipping checklist

Crate added to engine/Cargo.toml workspace members
impl Adapter (or SourceAdapter) complete
impl SqlDialect with all required methods
Auth module with env-var auto-detection
Conformance test wired up (rocky test-adapter <name> passes)
Factory registration so rocky.toml type = "<name>" resolves
Capability entry added to engine/crates/rocky-core/src/adapter_capability.rs::capability_for (data / discovery / both). Drives kind-field validation in rocky.toml and feeds the rocky-config skill table.
Env vars documented in engine/README.md and/or docs/src/content/docs/adapters/<name>.md
cargo test -p rocky-<name> green
cargo clippy -p rocky-<name> -- -D warnings clean

Commit style

feat(engine/rocky-bigquery): scaffold BigQuery target adapter
feat(engine/rocky-bigquery): implement SqlDialect + conformance tests
feat(engine/rocky-bigquery): OAuth service-account auth
docs(engine): add BigQuery adapter reference