prod-telemetry

name: prod-telemetry description: Query Executor's production telemetry — Axiom traces (executor-cloud dataset), prod Postgres via PlanetScale, PostHog product analytics — through the Executor MCP. Use when investigating prod errors, latency, usage, churn signals, or verifying a deploy's telemetry; includes the dataset field layout, working APL recipes, and the error-attribution join.

Production telemetry access

All three stores are queryable through the Executor MCP's connected integrations — no dashboards or credentials needed. Verify the connection exists with connections.list if a call fails.

Axiom traces (axiom_mcp)

Tool: axiom_mcp.user.axiomMcpOAuth.querydataset — the argument is apl (NOT query). Dataset: ['executor-cloud'] (worker spans; browser spans join the same traces via traceparent).

Field layout (the part you'd otherwise rediscover by failed queries):

• Custom span attributes live under the JSON map ['attributes.custom'], NOT as top-level attributes.* columns. Read with ['attributes.custom']['mcp.tool.name']. A nonexistent top-level field is a hard query error ("invalid field"), not an empty result.
• Span status: ['status.code'] ("OK"/"ERROR"), ['status.message'].
• Exceptions: the events column carries exception.type / exception.stacktrace JSON.
• OTel basics are top-level: name, trace_id, span_id, parent_span_id, duration, _time.

Span names worth querying (and their custom attrs):

• executor.tool.execute — mcp.tool.name (full address), and since PR #992: executor.tool.outcome (ok/fail), executor.tool.error_code, executor.tool.error_status, executor.tenant, executor.subject.
• mcp.tool.dispatch — mcp.tool.name (sandbox path), mcp.tool.integration, same outcome attrs.
• plugin.openapi.invoke — plugin.openapi.method / path_template / base_url, and since PR #992 http.status_code.
• mcp.request (outer) — mcp.auth.organization_id, mcp.auth.account_id, mcp.tool.name, CF edge fields (cf.country…), MCP client fingerprint (mcp.client.name…).

Recipe — error signatures by class (the daily-digest query):

['executor-cloud']
| where _time > ago(1d)
| where ['status.code'] == "ERROR" and name == "executor.tool.execute"
| extend msg = substring(tostring(['status.message']), 0, 120)
| extend tool = tostring(['attributes.custom']['mcp.tool.name'])
| summarize n = count() by msg, tool
| sort by n desc

Recipe — attribute errors to orgs. Tool spans now carry executor.tenant directly (post-#992). For spans from BEFORE that deploy, join through the outer request span:

['executor-cloud']
| where name == "mcp.request" and isnotnull(['attributes.custom']['mcp.auth.organization_id'])
| project trace_id, org = tostring(['attributes.custom']['mcp.auth.organization_id'])
| join kind=inner (
    ['executor-cloud']
    | where ['status.code'] == "ERROR" and name == "executor.tool.execute"
    | project trace_id, msg = substring(tostring(['status.message']), 0, 60)
  ) on trace_id
| summarize n = count() by org, msg | sort by n desc

Recipe — upstream failure rate per integration (post-#992 attrs):

['executor-cloud']
| where _time > ago(1d) and name == "mcp.tool.dispatch"
| extend outcome = tostring(['attributes.custom']['executor.tool.outcome'])
| extend integration = tostring(['attributes.custom']['mcp.tool.integration'])
| where isnotnull(outcome)
| summarize calls = count(), fails = countif(outcome == "fail") by integration
| extend failRate = todouble(fails) / todouble(calls)
| sort by fails desc

Known signal caveats (audited 2026-06-12):

• Pre-#992 spans: ToolResult.fail outcomes (upstream 4xx/5xx, auth rejections) are INVISIBLE — they rode the Effect success channel with no span marker. Don't conclude "no errors" from old data.
• Many pre-#992 ERROR spans have an EMPTY status.message (tagged errors without a message field) — group those by events exception.type instead.
• [object Object] status messages are the pre-#992 formatting bug.

Prod database (planetscale_mcp)

Read tool needs {organization: "answer-overflow", database: "executor", branch: "main"}. It returns ok: true even when the SQL failed — check the result text for Error:. Use for tenant/integration/connection facts that spans don't carry (row sizes, config shapes, counts).

Product analytics (posthog_api / mcp_posthog_com)

Browser-side events only (the ~60-event typed catalog, PR #987; server-side events not built). The org-key posthog_api connection covers the REST API; the OAuth MCP connection covers the higher-level tools.

Verifying a deploy's telemetry (Layer-0 canary)

After deploying telemetry changes: run a known-failing tool call against prod, then assert the expected attributes arrive in Axiom within ~1 min. Absence of data looks identical to health — query for the NEW attribute explicitly rather than eyeballing dashboards. The e2e equivalent runs on every suite: e2e/cloud/telemetry-contract.test.ts via the Telemetry service (motel /api/spans/search?attr.<key>=<value>).