systematic-debugging

name: systematic-debugging description: Use when encountering any bug, test failure, or unexpected behavior, before proposing fixes

Skill: systematic-debugging

When

Any bug, test failure, or unexpected behavior — before proposing fixes.

Follows ARCS CLI Primer: arcs --commands --json for discovery, --json --lean on all calls.

Flow

flowchart TD
    classDef decision fill:#f59e0b,color:#fff
    classDef stop fill:#ef4444,color:#fff

    Bug[Bug observed] --> ARCS[Check ARCS knowledge]
    ARCS --> Found{Match found?}
    Found -->|Yes| Verify[Verify it applies]
    Found -->|No| Observe

    Verify -->|Applies| Fix
    Verify -->|Doesn't apply| Observe

    Observe[Phase 1: Observe] --> Repro{Reproducible?}
    Repro -->|No| Instrument[Add logging/tracing]
    Instrument --> Observe
    Repro -->|Yes| Hypothesize[Phase 2: Hypothesize]

    Hypothesize --> Compare[Find working example, list differences]
    Compare --> Theory[Form single specific hypothesis]

    Theory --> Isolate[Phase 3: Isolate]
    Isolate --> Test{Smallest change confirms?}
    Test -->|Yes| Fix[Phase 4: Fix]
    Test -->|No| FailCount{3+ failures?}
    FailCount -->|No| Theory
    FailCount -->|Yes| Arch[Question architecture]

    Fix --> WriteFail[Write failing test]
    WriteFail --> Implement[Single targeted fix]
    Implement --> Green{Scoped tests pass?}
    Green -->|Yes| Capture[Capture as ARCS knowledge]
    Green -->|No| FailCount

    class Found,Repro,Test,FailCount,Green decision
    class Arch stop

Phase 1: Observe (Root Cause Investigation)

• Read the actual error message completely
• Reproduce consistently before proceeding
• Check recent changes (git log, git diff)
• Trace data flow backward from failure point
• Instrument component boundaries if cause unclear
• Pre-step: arcs knowledge search <slug> "<error>" --json for gotcha/lesson/pattern entries

Phase 2: Hypothesize (Pattern Analysis)

• Find a working example in the same codebase
• Compare working vs broken — list every difference
• Understand the dependency chain
• Form ONE specific hypothesis (not multiple)

Phase 3: Isolate

• Test with the smallest possible change
• One variable at a time — never stack fixes
• If hypothesis fails, form a new one from evidence
• Escalation: 3+ failed fixes → question the architecture, not the symptom

Phase 4: Fix

• Write a failing test FIRST (proves the bug exists)
• Implement a single targeted fix
• Verify the scoped tests for the files you changed pass (your dispatch VERIFY command — never the full suite; the devil-advocate completion gate owns that)
• If your fix introduces new failures in YOUR scoped tests, revert and return to Phase 2. Failures in files outside your scope are report-only (BLOCKED_BY) — likely a sibling agent's in-flight work; never fix or revert it

Log Triage Protocol

Scan order: failure point → errors → warnings → timing anomalies

rg -n "ERROR|FATAL|panic|exception" <logfile>   # Error grep
jq 'select(.level == "error")' <json-log>       # Structured logs

Output: Timeline of events leading to failure (T-5m, T-3m, T-0).

Git Bisect (Regressions)

git bisect start
git bisect bad HEAD
git bisect good <last-known-good>
git bisect run <test-command>

After finding the commit: read the diff, isolate specific lines, feed into Phase 2.

Dependency Conflict Diagnosis

Diagnose: npm ls <pkg>, npm explain <pkg>, check for multiple copies.

ARCS Knowledge Capture

After root cause identified, persist as knowledge:

• gotcha — environmental/config traps
• lesson — architectural insights from this session
• pattern — reusable solution to recurring problem

Include: root cause summary, evidence, affected files, fix approach.

Capture Resolution as Knowledge

After resolving the issue, persist the learning:

# For a surprising behavior or trap
arcs knowledge create <slug> "Redis connection pool exhaustion under load" \
  --kind=gotcha \
  --summary="Pool size defaults to 10; under concurrent requests >50, connections time out silently" \
  --body="Root cause: default pool size. Fix: set poolSize to max(50, expectedConcurrency). Symptoms: intermittent 503s with no error logs." \
  --json

# For a reusable debugging technique or resolution pattern
arcs knowledge create <slug> "Diagnosing silent connection failures" \
  --kind=lesson \
  --summary="Enable connection-level event logging before load testing" \
  --body="Attach listeners to pool 'error' and 'timeout' events. Default Node.js behavior swallows these." \
  --json

# For a pattern that should be followed going forward
arcs knowledge create <slug> "Connection pool sizing formula" \
  --kind=pattern \
  --summary="Pool size = max(50, 2x expected peak concurrency)" \
  --body="Applies to Redis, Postgres, and HTTP agent pools. Validated under load test 2026-05-26." \
  --json

Kind selection guide:

• gotcha — surprising behavior, trap, or non-obvious failure mode
• lesson — learned technique, debugging approach, resolution method
• pattern — reusable solution that should be applied going forward

Constraints

• NO FIXES WITHOUT ROOT CAUSE INVESTIGATION. If Phase 1 incomplete, you cannot propose fixes.
• One variable at a time. Never apply multiple changes simultaneously.
• 3+ failures = architectural problem. Stop fixing symptoms, question the pattern.
• Test before fix. Failing test proves the bug; green test proves the fix.
• Defense in depth: After fixing root cause, add validation at multiple layers to prevent recurrence.
• Systematic is faster than thrashing. 15-30min systematic vs 2-3h random fixes.

Red Flags (Return to Phase 1)

• "Quick fix for now, investigate later"
• "Just try changing X and see"
• Proposing solutions before tracing data flow
• Each fix reveals a new problem in a different place
• "I don't fully understand but this might work"
• Human says "stop guessing" or "is that not happening?"