corpus-test

name: corpus-test description: "Run corpus regression tests against real-world OpenAPI specs. Usage: /corpus-test [short]"

Run integration tests against the corpus of real-world OpenAPI specifications.

Usage:

/corpus-test — run all corpus tests
/corpus-test short — exclude large specs (faster)

Step 1: Check Corpus Availability

Use the Glob tool to check for corpus files:

pattern: testdata/corpus/*.{json,yaml,yml}

If no corpus files found, download them:

make corpus-download

Step 2: Run Corpus Tests

Based on the mode argument:

Full mode (default):

make test-corpus

Short mode (argument = "short"):

make test-corpus-short

Capture and analyze the output.

Step 3: Analyze Results

Note: Corpus specs are real-world documents (Plaid, DigitalOcean, GitHub, etc.) that often contain validation errors intentionally. Validation failures are expected — the key metric is whether the fixer resolves them.

For each corpus spec, report:

✅ Parse/validate/fix succeeded
❌ Failures with error details (fixer failed to resolve validation errors)
⚠️ New warnings compared to expected behavior

Step 4: Report

## Corpus Test Results

| Spec | Parse | Validate | Fix | Notes |
|------|-------|----------|-----|-------|
| petstore.yaml | ✅ | ✅ | ✅ | |
| github-api.json | ✅ | ❌ | ⚠️ | [details] |
| ... | ... | ... | ... | ... |

### Summary
- **Passed**: N/M specs
- **Failed**: [list]
- **New issues**: [list]

Step 5: Offer Actions

If failures are found:

Investigate — deep dive into specific failures
Compare with main — check if failures are regressions
File issues — create GitHub issues for new failures