debug-local-o2

name: debug-local-o2 description: > Query and inspect logs and traces from the local OpenObserve instance for debugging. Use when investigating errors, tracing request flows, inspecting spans, checking log output, or diagnosing issues in the running application. compatibility: Requires jq to be installed on the system. user-invocable: true disable-model-invocation: false

Debug OpenObserve Skill

Query and inspect logs and traces from the local OpenObserve (O2) instance.

IMPORTANT: The example scripts in scripts/ are reference starting points. You should compose, edit, and synthesize your own curl + jq commands as needed for the specific debugging scenario.

Prerequisites

• pnpm docker:dev must be running (starts O2 at localhost:5080)
• Read about the OpenTelemetry conventions used in this project

Connection

API Reference

Log Search

POST /api/{org}/_search - SQL-based search across any stream.

{
  "query": {
    "sql": "SELECT * FROM {stream} WHERE {conditions}",
    "start_time": 1674789786006000,
    "end_time": 1674793386006000,
    "from": 0,
    "size": 100
  }
}

• Times are microseconds (Unix seconds x 1,000,000)
• SQL is PostgreSQL-compatible
• str_match(field, 'term') - full-text search within a field
• match_all('term') - match any field containing term
• size: -1 - return all results (no pagination)

Response:

{ "took": 155, "hits": [], "total": 27179, "from": 0, "size": 10, "scan_size": 28943 }

Trace Search

CRITICAL: Trace queries require ?type=traces on the search endpoint. Without it, O2 searches the log stream instead and fields like start_time will not be found.

Overview - GET /api/{org}/{stream}/traces/latest

Query params: start_time, end_time, from, size, filter (all required except filter).

Returns high-level trace list with trace IDs, durations, and service names.

Span details - POST /api/{org}/_search?type=traces with SQL against the trace stream:

SELECT * FROM default WHERE trace_id = '{id}' ORDER BY start_time

Use size: -1 to retrieve all spans for a trace.

Log fields: _timestamp, body, severity (numeric: 1=TRACE, 5=DEBUG, 9=INFO, 13=WARN, 17=ERROR, 21=FATAL), service_name, trace_id, span_id, instrumentation_library_name

Trace fields: trace_id, span_id, operation_name, service_name, duration, start_time, end_time, span_kind, span_status (UNSET/ERROR), http_route, http_response_status_code, status_code, status_message

Other Endpoints

Example Scripts

Pre-written queries for common debugging scenarios. Run directly via bash .agents/skills/debug-local-o2/scripts/<script>.sh, or compose your own curl -s ... | jq -c ... commands as needed.

Common Queries

Logs

-- Error+ logs (severity >= 17)
SELECT * FROM default WHERE severity >= 17 ORDER BY _timestamp DESC

-- Logs for a specific user
SELECT * FROM default WHERE str_match(body, 'user@up.edu.ph')

-- Inngest function logs
SELECT * FROM default WHERE str_match(service_name, 'inngest')

-- Logs with a specific attribute
SELECT * FROM default WHERE match_all('draft_id')

Traces (remember ?type=traces)

-- All spans for a trace
SELECT * FROM default WHERE trace_id = '{id}' ORDER BY start_time

-- Slow spans (duration in microseconds)
SELECT * FROM default WHERE duration > 1000000 ORDER BY duration DESC

-- Error spans
SELECT * FROM default WHERE span_status = 'ERROR' ORDER BY start_time DESC

-- Spans for a specific HTTP route
SELECT * FROM default WHERE http_route = '/dashboard' ORDER BY start_time DESC

Noise Filtering

The Inngest dev server polls /api/inngest continuously for auto-discovery, generating significant trace/log noise. Exclude it from queries:

-- Logs: exclude Inngest polling
SELECT * FROM default WHERE NOT str_match(body, '/api/inngest') ...

-- Traces: exclude Inngest polling spans
SELECT * FROM default WHERE http_route != '/api/inngest' ...

Debugging Workflow

Recommended approach - trace-first:

1. Find relevant traces - use recent-traces.sh or filter by time window
2. Inspect spans - use trace-spans.sh {trace_id} to see the full span tree
3. Correlate with logs - use search-by-trace.sh {trace_id} to find log records emitted during that trace
4. Drill into specifics - compose ad-hoc queries targeting specific fields, services, or error conditions

Manual Curl Template

When composing ad-hoc queries, first compute the timestamp with date +%s, then embed it in the curl request:

# Step 1: Get current epoch (run separately)
date +%s

# Step 2: Use the output (e.g., 1708700000) to compose the query
curl -s -u "admin@example.com:password" \
  -X POST "http://localhost:5080/api/default/_search" \
  -H 'Content-Type: application/json' \
  -d '{"query":{"sql":"SELECT * FROM default WHERE ...","start_time":1708399700000000,"end_time":1708700000000000,"from":0,"size":50}}' \
  | jq -c '.hits'

Use jq -c (compact output) to minimize context usage. Use jq -c '.hits[]' to stream individual records.