ops-eng-o2r-phoenix - SKILL.md Agent Skill

name: "ops-eng-o2r-phoenix" description: "Use when working with the Arize Phoenix backend for otel-a2a-relay (o2r). Names the specific Phoenix UI surface for each question (waterfall vs JSON tree vs session list vs agent topology), with direct URLs and click paths. Triggers - phoenix, arize phoenix, o2r phoenix, phoenix sessions, phoenix trace, where is the waterfall, agent graph, openinference, o2r-harness, o2r-view, o2r-phoenix-bootstrap."

Phoenix backend for o2r

Local Phoenix runs at http://localhost:6006. The relay points OTEL_EXPORTER_OTLP_ENDPOINT at the same host (Phoenix accepts OTLP/HTTP on 6006).

Repo: otel-a2a-relay/arize_phoenix/. Upstream: Arize Phoenix.

Which UI surface answers which question

Default to visual views, not the JSON tree. The JSON tree on the trace detail page is a fallback - it is not the waterfall.

"Show me one session, all hops" - Sessions tab: http://localhost:6006/projects/default/sessions. One row per session.id; the entry point for any o2r demo run.
"Show me the visual waterfall for one trace" - click a trace from the Sessions row; the Trace detail page renders the timeline by default. JSON means you clicked the wrong tab.
"Show me the agent topology" - Agent Graph tab. Nodes are agents (agent.role + agent.specialization); edges from graph.node.parent_id (A -> B outbound, B -> A return).
"Show me LLM calls inside a hop" - LLM-kind spans nest under AGENT-kind roots in the Trace waterfall. OpenInference attributes (openinference.span.kind = LLM) drive the rendering.
"Show me datasets / annotations" - Datasets and Annotations sections in the left nav. o2r-phoenix-bootstrap provisions relay-decisions-golden, relay-failures-regression, relay_failure_class, task_outcome_correct.

Operate

From otel-a2a-relay/ workspace root:

uv sync --all-packages
make phoenix-fg              # Phoenix in foreground (operator-owned, separate terminal)
make phoenix-bootstrap       # idempotent: annotation configs + datasets
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:6006 make luca-demo
open http://localhost:6006   # lands on home, click Sessions

Phoenix elsewhere (remote host): OTEL_EXPORTER_OTLP_ENDPOINT=http://phoenix.local:6006 uv run o2r-harness.

CLIs the relay ships

o2r-harness - posts the worked-example trace, prints session.id and the validation checklist.
o2r-view - reduces a session's spans to a readable per-hop log (waterfall in text form for mobile / dictation). Also drives make gif CTX=<context>.
o2r-phoenix-bootstrap - idempotently provisions annotation configs + datasets via Phoenix REST.

Harness validation checklist

After o2r-harness, walk the Sessions / Agent Graph / Trace-tree checklist in docs/harness.md. If any view fails to render, don't start relay implementation - open an issue with screenshots.

If any of those fails to render, do not start relay implementation. Open an issue with screenshots.

Common traps

JSON tree is not the waterfall. The trace detail page has a JSON view and a timeline view. The timeline is the waterfall. If Kai asks "where is the visual view," answer Trace detail -> timeline tab, not the default JSON dump.
DB stuck on alembic error. ~/.phoenix/phoenix.db migration fails across version bumps with ResolutionError. Fix: rm ~/.phoenix/phoenix.db and restart. The harness reseeds in seconds.
No spans showing. Check OTEL_EXPORTER_OTLP_ENDPOINT actually points at :6006 (Phoenix), not :4318 (Tempo's port). Same protocol, different backend.
OpenInference attributes missing. Agent Graph stays empty without openinference.span.kind and graph.node.parent_id. The relay sets these; bespoke instrumentation must too.