rocky-test-fixtures

name: rocky-test-fixtures description: Dagster integration test fixture lifecycle. Use when regenerating live-binary fixtures via just regen-fixtures / scripts/regen_fixtures.sh, debugging why a generated fixture differs across runs, or adding scenario data for new CLI commands. Covers determinism (zeroed timings, sentinel timestamps), the relationship between tests/scenarios.py and tests/fixtures_generated/, and the two POCs that back the corpus.

Dagster test fixture regeneration

The dagster integration has two parallel sources of test data with different roles. Conflating them causes confusion, so this skill is worth reading before touching either.

The two test-data surfaces

Tests in test_types.py, test_translator.py, test_component.py, etc. consume the *_json fixtures backed by scenarios.py. The generated corpus is exercised solely by test_generated_fixtures.py to catch shape drift between the Rust binary and the Pydantic models.

A legacy tests/fixtures/*.json directory previously held the parsing-test corpus; it was removed once scenarios.py became the source of truth. See conftest.py for the migration note. The --in-place flag in scripts/regen_fixtures.sh still targets that directory and is effectively a no-op today.

When to use this skill

• Changing any *Output struct that affects a fixture's shape (pairs with the rocky-codegen skill — codegen updates the Pydantic models; the captured fixtures should still parse against them)
• Adding a brand new CLI command that needs a parsing test in test_types.py (you'll add a scenarios.py entry, not a JSON file)
• Investigating why test_generated_fixtures.py fails after a binary or schema change
• Debugging non-determinism in a captured fixture (timestamps, durations, row counts)

Regeneration workflow

just regen-fixtures
# or
./scripts/regen_fixtures.sh

Writes to integrations/dagster/tests/fixtures_generated/. The captured JSON should git diff cleanly except for fields that legitimately changed; non-trivial diffs warrant inspection.

After regenerating, always run from integrations/dagster/:

uv run pytest tests/test_generated_fixtures.py -v

This catches any captured-output shape that no longer parses against the current Pydantic models.

Prerequisites

The script hard-fails if either is missing:

1. Release binary at engine/target/release/rocky — build with:

   cd engine && cargo build --release --bin rocky
   

   or run just codegen (which also builds in release mode and shares the artifact).

2. duckdb CLI on $PATH — used to seed the playground POC's DuckDB file:

   brew install duckdb        # macOS
   

Two POCs back the corpus

regen_fixtures.sh runs the binary against two playground POCs, captured into different subdirectories of fixtures_generated/:

If you need a fixture shape that neither POC produces naturally (e.g. drift.json from a non-zero schema diff), add it as a scenarios.py entry rather than coercing the POC into producing it.

Determinism: why captured JSON is stable across runs

Without post-processing, every regen would produce a slightly different diff because Rocky stamps:

• Duration fields with real wall-clock elapsed time (_ms, _seconds, _secs suffixes)
• Timestamp fields with now() (updated_at, started_at, finished_at, timestamp, captured_at)

The capture helper inside regen_fixtures.sh runs a Python normalizer (scripts/_normalize_fixture.py) over each captured JSON file that:

• Zeroes any numeric field whose key ends in _ms / _seconds / _secs
• Replaces any string field whose key is in the wall-clock set with "2000-01-01T00:00:00Z"

Important invariants the normalizer preserves:

• last_value (watermarks) — NOT touched. It's a logical value derived from seeded data and is genuinely deterministic across runs.
• Anything inside a data payload — NOT touched. Only top-level wall-clock fields get the sentinel.

If you add a new timing or timestamp field to an *Output struct, you may need to extend TIMING_SUFFIXES or WALL_CLOCK_FIELDS in the normalizer. Otherwise your new fixture will drift on every regen.

Exit-code tolerance

The capture helper tolerates non-zero exit codes as long as stdout parses as JSON. This is intentional:

• rocky doctor exits 1 when any check is warning, 2 when critical — and both still emit valid JSON.
• rocky run can exit non-zero on partial success and still emit a valid RunOutput.

The dagster integration's allow_partial=True path mirrors this tolerance in production — see integrations/dagster/src/dagster_rocky/resource.py.

Adding parsing-test coverage for a new CLI command

For a new CLI command whose output the dagster tests need to parse:

1. Rust schema exists (from the rocky-codegen skill).
2. Pydantic binding exists (from just codegen-sdk).
3. Add a scenarios.py entry with a representative output dict, then expose it via a *_json pytest fixture in conftest.py.
4. Add a parsing test in test_types.py that loads the fixture and asserts on key fields.
5. Optionally, add a capture call in scripts/regen_fixtures.sh so the playground POC contributes a live-binary sample under fixtures_generated/. Run just regen-fixtures afterwards.

If the playground POC can't produce the command naturally (e.g. an adapter-specific command that needs Databricks), skip step 5 — the scenarios.py entry is the test source of truth.

Debugging fixture drift

Symptoms and causes:

Reference files

• scripts/regen_fixtures.sh — the capture pipeline + POC list
• scripts/_normalize_fixture.py — wall-clock + timing normalizer
• integrations/dagster/tests/scenarios.py — hand-crafted Python dict scenarios (test source of truth)
• integrations/dagster/tests/conftest.py — *_json fixture exposure
• integrations/dagster/tests/test_generated_fixtures.py — the test that guards the captured corpus
• integrations/dagster/CLAUDE.md — "Adding support for a new Rocky CLI command" checklist (includes the scenarios step)
• justfile — the regen-fixtures recipe (wraps the script)

Related skills

• rocky-codegen — the Rust → Pydantic/TS cascade that fixtures are validated against
• rocky-new-cli-command — the end-to-end checklist that includes scenario/fixture creation
• rocky-poc — authoring the playground POCs that the captured fixture corpus depends on