qa

name: qa description: "Run comprehensive QA and integration tests against the local Centaur stack. Use when asked to QA the stack, run integration tests, verify a deployment, or check stack health after a refactor."

Centaur QA

Test the Centaur stack in three progressive layers — same core operations at each level, proving successively more surface area works.

Layers

Layer 1: Internal API     kubectl exec → API directly. Proves core works.
Layer 2: External route    curl from host → API route. Proves routing + auth.
Layer 3: User Interfaces   Slackbot webhooks + Thread Viewer UI. Proves E2E UX.

If layer 1 passes but layer 2 fails → problem is nginx/auth. If 1+2 pass but 3 fails → problem is slackbot or web app.

Execution Pipeline

┌─────────────────────────┐
│  Layer 1: Internal API  │  ← Run first, sequential. Must pass before continuing.
│  (services, tools,      │
│   agent/execute,        │
│   personas, logs)       │
└──────────┬──────────────┘
           │ all pass
     ┌─────┴──────┬──────────────┐
     ▼            ▼              ▼
┌─────────┐ ┌──────────┐ ┌────────────┐
│ Layer 2  │ │ Layer 3a │ │ Layer 3b   │   ← Parallel subagents
│ Nginx    │ │ Slackbot │ │ Web App    │
│          │ │          │ │ (dogfood)  │
└─────────┘ └──────────┘ └────────────┘

Use the Task tool to run layers 2, 3a, and 3b as parallel subagents once layer 1 passes.

Setup

Getting the API key: API_SECRET_KEY is NOT in .env — it's in the secrets manager. Fetch it:

API_KEY=$(kubectl exec -n centaur deploy/centaur-centaur-api -- curl -s http://firewall:8081/secrets/API_SECRET_KEY | python3 -c "import sys,json; print(json.load(sys.stdin)['value'])")

Use $API_KEY (not $API_SECRET_KEY) in all curl commands below.

If the user says "QA", "QA the stack", or "health check", start immediately with defaults (sample tools, all layers). Do not ask clarifying questions.

Layer 1: Internal API

All calls via kubectl exec -n centaur deploy/centaur-centaur-api -- curl -s http://localhost:8000/... — bypasses external auth, no key needed.

1a. Services & Health

kubectl get pods -n centaur

All required local deployments must be Ready: postgres, secrets, iron-proxy/firewall-manager, and api. Slackbot is disabled in the default dev overlay.

kubectl exec -n centaur deploy/centaur-centaur-api -- curl -s http://localhost:8000/health/ready
# → {"status":"ok"}

1b. Tool Testing

Test tools via POST /tools/{tool}/{method}.

Sample mode (default): ~10 tools across categories for a fast smoke test:

Auth failures on etherscan/websearch are known — note but don't block.

Full mode (user says "full tools"): Test every registered tool. See references/test-inputs.md for default inputs. Batch by group, append results incrementally.

Classifying results:

Rules: Never call write/mutate methods. Use limit: 2. Chain dependent calls. See references/test-inputs.md.

1c. Agent Execute (connect + execute protocol)

The API uses a two-wire protocol: /agent/connect opens a persistent SSE stream, /agent/execute injects messages into stdin. Output appears on the connect wire.

Step 1: Open connect wire (run in background, write to file):

kubectl exec -n centaur deploy/centaur-centaur-api -- bash -c "curl -s -N -X POST http://localhost:8000/agent/connect \
  -H 'Content-Type: application/json' \
  -d '{\"thread_key\": \"test:qa-execute\", \"harness\": \"amp\"}' \
  > /tmp/sse_qa.txt 2>&1" &
sleep 5

Step 2: Execute a message:

kubectl exec -n centaur deploy/centaur-centaur-api -- curl -s -X POST http://localhost:8000/agent/execute \
  -H "Content-Type: application/json" \
  -d '{"thread_key":"test:qa-execute","message":"Say hello and nothing else","harness":"amp"}'
# → {"ok": true, "injected": true, "turn_id": 1}

Wait ~10s, then check the SSE output:

kubectl exec -n centaur deploy/centaur-centaur-api -- cat /tmp/sse_qa.txt

Verify: SSE output contains wire.ready, system.init, assistant with response text, and exactly one turn.done per turn with non-empty result.

Step 3: Back-to-back follow-up (same thread, same container):

kubectl exec -n centaur deploy/centaur-centaur-api -- curl -s -X POST http://localhost:8000/agent/execute \
  -H "Content-Type: application/json" \
  -d '{"thread_key":"test:qa-execute","message":"Now say goodbye"}'

Verify: Second turn.done appears on the connect wire with turn_id: 2.

1c-ii. Handoff Test

Handoffs are transparent — the amp-wrapper detects handoff tool calls, suppresses handoff noise, kills amp, and chains into the new thread. The API sees one continuous stream.

kubectl exec -n centaur deploy/centaur-centaur-api -- curl -s -X POST http://localhost:8000/agent/execute \
  -H "Content-Type: application/json" \
  -d '{"thread_key":"test:qa-execute","message":"handoff and say hi 3 times"}'

Wait ~30s, then check SSE output:

Verify:

• No tool_use or tool_result events for the handoff tool visible in the stream
• A new system.init with a different session_id appears (the chained thread)
• The chained thread's response appears with a turn.done
• No duplicate content — user should NOT see the same text twice

1c-iii. Auto-chain (context pressure)

The wrapper auto-chains when context usage exceeds 85% (AMP_CONTEXT_THRESHOLD env var). To test, use a low threshold. This requires setting the env var on the sandbox container, which isn't easily done via the API — test locally instead:

cd /tmp && mkdir -p test-autochain && cd test-autochain && git init . 2>/dev/null
(
  echo '{"type":"user","message":{"role":"user","content":[{"type":"text","text":"say hello"}]}}'
  sleep 30
) | AMP_CONTEXT_THRESHOLD=0.01 timeout 30 python3 services/sandbox/amp-wrapper.py 2>/dev/null

Verify: Output shows first thread's response + result, then seamlessly a new system.init with the same session_id (continued thread), followed by the chained thread's response.

Clean up: POST /agent/stop with {"thread_key":"test:qa-execute"}.

1d. Personas

Check loaded personas:

kubectl logs -n centaur deploy/centaur-centaur-api --tail 50 | grep persona_loaded

For each persona (typically eng, legal, invest, events):

kubectl exec -n centaur deploy/centaur-centaur-api -- curl -s -X POST http://localhost:8000/agent/execute \
  -H "Content-Type: application/json" \
  -d '{
    "thread_key": "test:qa-persona-{NAME}",
    "message": "Run: echo $AGENT_PERSONA && head -3 ~/AGENTS.md 2>/dev/null || echo NO_AGENTS_MD",
    "harness": "{NAME}"
  }'

Verify: AGENT_PERSONA is set, prompt content is persona-specific, different cache_creation_input_tokens across personas.

Also test invalid persona — should fall back gracefully.

Clean up all test:qa-persona-* containers.

1e. Log Pipeline

# All services present in VictoriaLogs?
kubectl exec -n centaur deploy/centaur-centaur-api -- curl -s "http://victorialogs:9428/select/logsql/query" \
  --data-urlencode "query=* | uniq_values(service) limit 1000" --data-urlencode "limit=1"

# _msg field populated (not "missing _msg field")?
kubectl exec -n centaur deploy/centaur-centaur-api -- curl -s "http://victorialogs:9428/select/logsql/query" \
  --data-urlencode 'query=service:"api"' --data-urlencode "limit=3"

# Structured fields (level, event) searchable?
kubectl exec -n centaur deploy/centaur-centaur-api -- curl -s "http://victorialogs:9428/select/logsql/query" \
  --data-urlencode 'query=service:"api" AND level:"info" AND event:*' --data-urlencode "limit=3"

# Sandbox container logs collected?
kubectl exec -n centaur deploy/centaur-centaur-api -- curl -s "http://victorialogs:9428/select/logsql/query" \
  --data-urlencode 'query=container:"pipe-"' --data-urlencode "limit=2"

1f. Attachments

Test the attachment pipeline: inline extraction, upload endpoint, and sandbox download.

Inline base64 extraction roundtrip:

# Buffer a message with inline base64 image
B64=$(echo -n "FAKE_PNG_BYTES_FOR_QA_TEST" | base64)
kubectl exec -n centaur deploy/centaur-centaur-api -- curl -s -X POST http://localhost:8000/agent/messages \
  -H "Content-Type: application/json" \
  -d "{
    \"thread_key\": \"test:qa-att-inline\",
    \"messages\": [{
      \"role\": \"user\",
      \"parts\": [
        {\"type\": \"text\", \"text\": \"analyze this\"},
        {\"type\": \"image\", \"source\": {\"type\": \"base64\", \"media_type\": \"image/png\", \"data\": \"$B64\"}}
      ]
    }]
  }"
# → {"ok": true, "inserted": 1}

# List attachments — should have 1 entry
kubectl exec -n centaur deploy/centaur-centaur-api -- curl -s "http://localhost:8000/agent/attachments?thread_key=test:qa-att-inline"
# → [{"id":"att-...","name":"image.png","mime_type":"image/png",...}]

# Download and verify bytes
ATT_ID=$(kubectl exec -n centaur deploy/centaur-centaur-api -- curl -s "http://localhost:8000/agent/attachments?thread_key=test:qa-att-inline" | python3 -c "import sys,json; print(json.load(sys.stdin)[0]['id'])")
kubectl exec -n centaur deploy/centaur-centaur-api -- curl -s "http://localhost:8000/agent/attachments/$ATT_ID/download" | base64 | head -c 40

Verify: List returns 1 attachment with mime_type: image/png. Download returns non-empty bytes. Original message parts in chat_messages contain attachment_ref, not base64 blob.

Upload endpoint:

B64=$(echo -n "UPLOAD_TEST_CONTENT" | base64)
kubectl exec -n centaur deploy/centaur-centaur-api -- curl -s -X POST http://localhost:8000/agent/attachments/upload \
  -H "Content-Type: application/json" \
  -d "{
    \"thread_key\": \"test:qa-att-upload\",
    \"name\": \"test-upload.txt\",
    \"mime_type\": \"text/plain\",
    \"data\": \"$B64\"
  }"
# → {"id":"att-...","name":"test-upload.txt","download_url":"/agent/attachments/att-.../download"}

# Download and verify
ATT_ID=$(kubectl exec -n centaur deploy/centaur-centaur-api -- curl -s -X POST http://localhost:8000/agent/attachments/upload \
  -H "Content-Type: application/json" \
  -d "{\"thread_key\":\"test:qa-att-upload2\",\"name\":\"verify.txt\",\"mime_type\":\"text/plain\",\"data\":\"$(echo -n hello | base64)\"}" | python3 -c "import sys,json; print(json.load(sys.stdin)['id'])")
kubectl exec -n centaur deploy/centaur-centaur-api -- curl -s "http://localhost:8000/agent/attachments/$ATT_ID/download"
# → hello

Verify: Upload returns id, name, download_url. Download returns exact bytes uploaded.

Negative cases:

# Missing fields → 422
kubectl exec -n centaur deploy/centaur-centaur-api -- curl -s -X POST http://localhost:8000/agent/attachments/upload \
  -H "Content-Type: application/json" \
  -d '{"thread_key":"test:qa-att-bad"}'
# → 422

# Nonexistent attachment → 404
kubectl exec -n centaur deploy/centaur-centaur-api -- curl -s -o /dev/null -w "%{http_code}" \
  http://localhost:8000/agent/attachments/att-doesnotexist/download
# → 404

Verify: Missing fields returns 422. Nonexistent attachment returns 404.

Layer 2: Nginx (parallel subagent)

Same operations as layer 1, but via curl http://localhost:8000 from the host — goes through nginx → auth → API. Source .env for $API_SECRET_KEY.

2a. Health & Tools

curl -s http://localhost:8000/health
curl -s http://localhost:8000/tools -H "Authorization: Bearer $API_KEY" | python3 -c "
import sys,json; d=json.load(sys.stdin); print(f'{len(d)} tools via nginx')
"

Watch for: If /tools returns {"detail":"Invalid API key"}, the API key is wrong — re-fetch from secrets (see Setup).

2b. Tool Calls via Nginx

Run the same sample tool tests from 1b, but via nginx with Bearer auth:

curl -s -X POST http://localhost:8000/tools/demo/echo \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" -d '{"message":"hello"}'

2c. Agent Execute via Nginx

curl -s --max-time 120 -X POST http://localhost:8000/agent/execute \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"thread_key":"test:qa-nginx","message":"What persona are you? One word.","harness":"amp"}'

Verify: SSE stream arrives (not 502). turn.done event has non-empty result.

Known issue (fixed): The /agent/ route needs SSE-friendly buffering and timeout settings. Without these, the external route can return 502 while the internal API path works.

2d. Agent Execute — Personas via Nginx

Test each persona (eng, events, invest, legal) through nginx:

for PERSONA in eng events invest legal; do
  echo "=== Persona: $PERSONA ==="
  curl -s --max-time 120 -X POST http://localhost:8000/agent/execute \
    -H "Authorization: Bearer $API_KEY" \
    -H "Content-Type: application/json" \
    -d "{\"thread_key\":\"test:qa-persona-$PERSONA\",\"message\":\"What persona are you? One word.\",\"harness\":\"$PERSONA\"}" \
    | grep 'turn.done'
  echo ""
done

Verify: Each returns a persona-specific answer. Do NOT run these in a bash loop with & — containers need time to spin up; run sequentially with --max-time 120.

2e. Message Persistence

After agent execute, verify both user AND assistant messages are in postgres:

kubectl exec -n centaur statefulset/centaur-centaur-postgres -- psql -U tempo -d ai_v2 -c \
  "SELECT role, substring(parts::text,1,150) FROM chat_messages WHERE thread_key='test:qa-nginx' ORDER BY created_at;"

Verify: Two rows — one user, one assistant with the agent's response text.

Known issue (fixed): stream_exec previously expected turn.done.result to be a dict with a .text field, but it's actually a plain string. This caused assistant messages to never be persisted. If you see user messages but no assistant messages in chat_messages, check the result extraction logic in services/api/api/agent.py → stream_exec().

2f. Fire-and-Forget + Status Poll

Test the async execution flow — fire a job and poll /agent/status for completion:

# Fire (background, don't wait for stream)
curl -s --max-time 5 -X POST http://localhost:8000/agent/execute \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"thread_key":"test:qa-async","message":"Say DONE","harness":"amp"}' > /dev/null 2>&1 &

# Poll status after ~15s
sleep 15
curl -s "http://localhost:8000/agent/status?key=test:qa-async" \
  -H "Authorization: Bearer $API_KEY" | python3 -m json.tool

Verify: Response includes "busy": false and "last_result": "DONE" (or similar).

2h. Cross-Persona Dispatch (from inside sandbox)

Test that one agent can spawn another persona via call agent execute, then poll for results. This is the most important integration test — it proves the full multi-agent orchestration pipeline.

curl -s --max-time 300 -X POST http://localhost:8000/agent/execute \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "thread_key":"test:qa-cross-persona",
    "message":"Run these commands in order:\n1. call agent execute '\''{ \"thread_key\": \"test:qa-legal-sub\", \"message\": \"What is a SAFE? One sentence.\", \"harness\": \"legal\" }'\''\\n2. sleep 30\\n3. call agent status '\''?key=test:qa-legal-sub'\''\\n4. Show me the last_result.",
    "harness":"eng"
  }'

Verify the SSE trace shows:

1. eng agent runs call agent execute → gets SSE stream back (legal container spawned)
2. eng agent runs call agent status → gets busy: false + last_result with legal's answer
3. eng agent presents the result

If call agent execute returns "Tool 'agent' not found": The sandbox container has an old call.sh without the agent case. Fix:

1. Rebuild sandbox image: just build-one agent
2. Delete stale sandbox pods: kubectl delete pod -n centaur -l centaur-agent=true --ignore-not-found
3. Redeploy API: just deploy
4. Wait for rollout, then retry

2g. Auth Gate

# Unauthenticated browser request should redirect to /login
curl -s -o /dev/null -w "%{http_code}" http://localhost:8000/
# → 302

Layer 3a: Slackbot (parallel subagent)

Test the slackbot by crafting HMAC-signed Slack webhook payloads. This proves the full Slack → slackbot → API → sandbox → response path.

Automated integration test script

Run the automated integration test suite (URL verification, signature rejection, app_mention with/without file attachments, thread messages, edge cases):

source .env
{SKILL_DIR}/scripts/integration-slackbot.sh

Set SLACKBOT_URL if the slackbot is only reachable inside the cluster:

kubectl exec -n centaur deploy/centaur-centaur-api -- bash -c 'SLACKBOT_URL=http://slackbot:3001 SLACK_SIGNING_SECRET=<secret> /path/to/integration-slackbot.sh'

3a-i. URL Verification (signed)

source .env
SIGNING_SECRET="$SLACK_SIGNING_SECRET"
TIMESTAMP=$(date +%s)
BODY='{"type":"url_verification","challenge":"test-challenge-qa"}'
SIG_BASESTRING="v0:${TIMESTAMP}:${BODY}"
SIGNATURE="v0=$(echo -n "$SIG_BASESTRING" | openssl dgst -sha256 -hmac "$SIGNING_SECRET" | awk '{print $2}')"

# Direct to slackbot
curl -s -X POST http://localhost:3001/api/slack/events \
  -H "Content-Type: application/json" \
  -H "x-slack-signature: $SIGNATURE" \
  -H "x-slack-request-timestamp: $TIMESTAMP" \
  -d "$BODY"
# → {"challenge":"test-challenge-qa"}

Note: If slackbot is enabled without an external route, use kubectl exec to reach it on the internal network:

kubectl exec -n centaur deploy/centaur-centaur-api -- curl -s -X POST http://slackbot:3001/api/slack/events \
  -H "Content-Type: application/json" \
  -H "x-slack-signature: $SIGNATURE" \
  -H "x-slack-request-timestamp: $TIMESTAMP" \
  -d "$BODY"

3a-ii. Signature Rejection

curl -s -X POST http://localhost:3001/api/slack/events \
  -H "Content-Type: application/json" \
  -H "x-slack-signature: v0=bad" \
  -H "x-slack-request-timestamp: $(date +%s)" \
  -d '{"type":"url_verification","challenge":"test"}'
# → 401 {"error":"Invalid Slack signature"}

3a-iii. Via Nginx (production path)

# Same signed payload through nginx → API webhook proxy
curl -s -X POST http://localhost:8000/api/webhooks/slack \
  -H "Content-Type: application/json" \
  -H "x-slack-signature: $SIGNATURE" \
  -H "x-slack-request-timestamp: $TIMESTAMP" \
  -d "$BODY"

Layer 3b: Web App / Thread Viewer (parallel subagent)

Use the dogfood skill to systematically test the thread viewer UI.

Load skill: skill("dogfood")
Target: http://localhost:8000
Auth: Log in via /login with UI_PASSWORD from .env

Key flows to test:

• Login page works, cookie set after auth
• Thread list view loads, shows recent threads
• Thread detail view renders messages, tool calls, dashboard blocks
• Agent execution from the UI (if supported)
• SSE streaming in thread viewer
• Static assets load (_next/ routes)

The dogfood skill produces its own report with screenshots and repro steps.

Report

Copy the template and fill in results:

cp {SKILL_DIR}/templates/tool-qa-report-template.md {OUTPUT_DIR}/report.md

Known Gotchas

Issue Investigation

When something fails:

1. Service crash — kubectl logs -n centaur deploy/centaur-centaur-{component} --tail 30
2. Schema mismatch — Check DB/API schema vs tool code
3. Missing credentials — kubectl exec -n centaur deploy/centaur-centaur-api -- curl -s http://centaur-centaur-firewall-manager:8081/secrets/{KEY} (NOT direct 1Password access)
4. Connection failure — Check upstream, tunnel, firewall
5. Routing issue — Compare ingress/HTTPRoute config with expected path
6. Postgres persistence — Check chat_messages table: kubectl exec -n centaur statefulset/centaur-centaur-postgres -- psql -U tempo -d ai_v2 -c "SELECT role, parts FROM chat_messages WHERE thread_key='...'"

Note root cause and suggested fix for each failure.

Fixing Issues

1. Fix the bug in the relevant service/tool code
2. Tools: commit + push (hot-reload, no restart)
3. Services: just build-one {service} && just deploy
4. Re-test, update report from FAIL → PASS (fixed)

References

Templates