trust-dashboard

name: "trust-dashboard" description: "Render per-check Bayesian trust scores from AgentDB reflexion entries. Read-only -- no side effects."

/trust-dashboard -- Validation Trust Scores

What This Skill Does

Queries AgentDB for trust reflexion entries (written by /validate after each run), computes per-check Bayesian trust scores using Beta distributions, and renders a human-readable dashboard. This is a read-only skill with no side effects.

Trust scores start uninformative (0.5) and calibrate over time as more validation runs and /shadow-judge comparisons accumulate.

Usage

/trust-dashboard

No arguments required. Queries all trust:validation:* entries from AgentDB.

How It Works

Step 1: Query AgentDB

reflexion_retrieve(task="trust:validation", limit=500)

This returns all reflexion entries with the trust:validation:* prefix, written by /validate (ADR-004) and /shadow-judge (ADR-006).

Step 2: Group by Check Name

Parse the task prefix to extract tier and check name:

trust:validation:{tier}:{check_name}

Group entries by {tier}:{check_name}.

Step 3: Compute Beta Scores

For each check, compute the Bayesian trust score:

correct = count of entries where reward = 1.0
incorrect = count of entries where reward = 0.0

Trust(check) = (correct + 1) / (correct + incorrect + 2)

This is a Beta(correct+1, incorrect+1) distribution mean. With no data, Trust = 0.5 (uninformative prior). As evidence accumulates, the score converges to the true reliability.

Step 4: Compute Composite Score

Composite = 0.30 * avg(Tier1_checks)
          + 0.30 * avg(Tier2_checks)
          + 0.15 * avg(Tier3_checks)
          + 0.15 * (1 - rework_rate)
          + 0.10 * scope_conformance

Where:

rework_rate = proportion of validation runs that required fix iterations
scope_conformance = proportion of file_scope checks that passed

If a tier has no data, use 0.5 (uninformative) for its average.

Step 5: Render Dashboard

TRUST DASHBOARD ({date})
============================

COMPOSITE SCORE: {score}

PER-CHECK SCORES:
  Tier 1 (Compilation):
    build ............... {score} ({correct}/{total})
    test ................ {score} ({correct}/{total})
    clippy .............. {score} ({correct}/{total})
    anti_stub ........... {score} ({correct}/{total})

  Tier 2 (Process Adherence):
    banned_deps ......... {score} ({correct}/{total})
    stub_scan ........... {score} ({correct}/{total})
    file_scope .......... {score} ({correct}/{total})
    stale_refs .......... {score} ({correct}/{total})
    config_valid ........ {score} ({correct}/{total})

  Tier 3 (Spec Compliance):
    ac_coverage ......... {score} ({correct}/{total})
    test_delta .......... {score} ({correct}/{total})
    new_deps ............ {score} ({correct}/{total})

  Tier 4 (Risk Classification):
    risk_score .......... {score} ({correct}/{total})

SHADOW JUDGE CALIBRATION:
  Total comparisons: {N}
  Human agreed: {M}
  Human disagreed: {K}
  False negative rate: {K/N}

LAST 5 FEATURES:
  | Feature | Tier1 | Tier2 | Tier3 | Tier4 | Overall |
  |---------|-------|-------|-------|-------|---------|
  | {id}    |  {r}  |  {r}  |  {r}  | {risk}|  {r}    |

TREND: {Improving|Stable|Regressing} (based on last 5 composite scores)

Empty State

If no trust entries exist yet:

TRUST DASHBOARD ({date})
============================

No trust data available yet.

Trust scores accumulate after each /validate run.
Run /validate on a feature to start building trust data.
Run /shadow-judge after reviewing to add calibration signal.

Interpreting Scores

Score Range	Meaning
0.90 - 1.00	High trust -- check is reliable
0.70 - 0.89	Good trust -- occasional false negatives
0.50 - 0.69	Uninformative -- not enough data or mixed results
0.30 - 0.49	Low trust -- check frequently misses issues
0.00 - 0.29	Very low trust -- check is unreliable

Important: Scores below 0.5 are unusual and indicate the check is wrong more often than right. This typically means the check itself needs revision, not that the code is bad.

Data Sources

Source	What It Writes	When
`/validate`	`reward=1.0` for each passing check	After every validation run
`/shadow-judge approve`	`reward=1.0` for all checks	When human agrees
`/shadow-judge reject`	`reward=0.0` for missed checks	When human finds issues

Self-reported trust (from /validate alone) is always 1.0. Only /shadow-judge reject adds real calibration signal (reward=0.0).

Statistical Note

The Beta distribution needs approximately 20+ observations per check for statistically meaningful scores. With fewer observations, the score is heavily influenced by the prior (0.5). The dashboard does not attempt significance testing -- it shows raw Beta means and observation counts so the user can judge data sufficiency.

.claude/skills/validate/SKILL.md -- writes trust entries after validation
.claude/skills/shadow-judge/SKILL.md -- human calibration input
product/features/ops-006/architecture/ARCHITECTURE.md -- ADR-004 (trust storage), ADR-005 (dashboard)