traces

star 25

Query and investigate trace data from OpenSearch using PPL for agent invocations, tool executions, errors, latency, and token usage analysis.

opensearch-project By opensearch-project schedule Updated 3/22/2026
play_arrow Run Skill in Manus View GitHub

name: traces description: Query and investigate trace data from OpenSearch using PPL for agent invocations, tool executions, errors, latency, and token usage analysis. allowed-tools: - Bash - curl

Trace Querying with PPL

Overview

This skill provides PPL (Piped Processing Language) query templates for investigating trace data stored in OpenSearch. Traces are stored in the otel-v1-apm-span-* index pattern and service dependency maps in otel-v2-apm-service-map. All queries use the OpenSearch PPL API at /_plugins/_ppl with HTTPS and basic authentication.

Credentials are read from the .env file (default: admin / My_password_123!@#). All curl commands use -k to skip TLS certificate verification for local development.

Connection Defaults

All commands below use these variables. Set them in your environment or use the defaults:

Variable Default Description
OPENSEARCH_ENDPOINT https://localhost:9200 OpenSearch base URL
OPENSEARCH_USER admin OpenSearch username
OPENSEARCH_PASSWORD My_password_123!@# OpenSearch password

Base Command

All PPL queries in this skill use this curl pattern:

curl -sk -u "$OPENSEARCH_USER:$OPENSEARCH_PASSWORD" \
  -X POST "$OPENSEARCH_ENDPOINT/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query": "<PPL_QUERY>"}'

The examples below show the full command for clarity, but only the PPL query varies.

Agent Invocation Spans

Query all spans where a GenAI agent was invoked:

curl -sk -u "$OPENSEARCH_USER:$OPENSEARCH_PASSWORD" \
  -X POST "$OPENSEARCH_ENDPOINT/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query": "source=otel-v1-apm-span-* | where `attributes.gen_ai.operation.name` = '\''invoke_agent'\'' | fields traceId, spanId, `attributes.gen_ai.agent.name`, `attributes.gen_ai.request.model`, durationInNanos, startTime | sort - startTime | head 20"}'

Tool Execution Spans

Query all spans where a tool was executed:

curl -sk -u "$OPENSEARCH_USER:$OPENSEARCH_PASSWORD" \
  -X POST "$OPENSEARCH_ENDPOINT/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query": "source=otel-v1-apm-span-* | where `attributes.gen_ai.operation.name` = '\''execute_tool'\'' | fields traceId, spanId, `attributes.gen_ai.tool.name`, durationInNanos, startTime | sort - startTime | head 20"}'

Slow Spans

Identify spans exceeding a latency threshold. The default threshold is 5 seconds (5,000,000,000 nanoseconds). Adjust the value as needed:

curl -sk -u "$OPENSEARCH_USER:$OPENSEARCH_PASSWORD" \
  -X POST "$OPENSEARCH_ENDPOINT/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query": "source=otel-v1-apm-span-* | where durationInNanos > 5000000000 | fields traceId, spanId, serviceName, name, durationInNanos, startTime | sort - durationInNanos | head 20"}'

To find slow agent invocations specifically:

curl -sk -u "$OPENSEARCH_USER:$OPENSEARCH_PASSWORD" \
  -X POST "$OPENSEARCH_ENDPOINT/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query": "source=otel-v1-apm-span-* | where `attributes.gen_ai.operation.name` = '\''invoke_agent'\'' AND durationInNanos > 5000000000 | fields traceId, `attributes.gen_ai.agent.name`, durationInNanos | sort - durationInNanos"}'

Error Spans

Query spans with error status (status.code = 2 means ERROR in OTel):

curl -sk -u "$OPENSEARCH_USER:$OPENSEARCH_PASSWORD" \
  -X POST "$OPENSEARCH_ENDPOINT/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query": "source=otel-v1-apm-span-* | where `status.code` = 2 | fields traceId, spanId, serviceName, name, `status.code`, startTime | sort - startTime | head 20"}'

Token Usage by Model

Aggregate input and output token usage grouped by the requested model:

curl -sk -u "$OPENSEARCH_USER:$OPENSEARCH_PASSWORD" \
  -X POST "$OPENSEARCH_ENDPOINT/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query": "source=otel-v1-apm-span-* | where `attributes.gen_ai.usage.input_tokens` > 0 | stats sum(`attributes.gen_ai.usage.input_tokens`) as total_input, sum(`attributes.gen_ai.usage.output_tokens`) as total_output by `attributes.gen_ai.request.model`"}'

Token Usage by Agent

Aggregate token usage grouped by agent name:

curl -sk -u "$OPENSEARCH_USER:$OPENSEARCH_PASSWORD" \
  -X POST "$OPENSEARCH_ENDPOINT/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query": "source=otel-v1-apm-span-* | where `attributes.gen_ai.usage.input_tokens` > 0 | stats sum(`attributes.gen_ai.usage.input_tokens`) as total_input, sum(`attributes.gen_ai.usage.output_tokens`) as total_output by `attributes.gen_ai.agent.name`"}'

Service Operations Listing

List distinct services and their GenAI operation types with counts:

curl -sk -u "$OPENSEARCH_USER:$OPENSEARCH_PASSWORD" \
  -X POST "$OPENSEARCH_ENDPOINT/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query": "source=otel-v1-apm-span-* | stats count() by serviceName, `attributes.gen_ai.operation.name`"}'

Service Map Queries

Important: The sourceNode, targetNode, sourceOperation, and targetOperation fields in the otel-v2-apm-service-map-* index are nested struct objects, not flat strings. Each node has the structure:

{
  "keyAttributes": { "name": "frontend", "environment": "generic:default" },
  "groupByAttributes": { "telemetry": { "sdk": { "language": "python" } } },
  "type": "service"
}

PPL returns these as JSON objects. When aggregating by node (e.g., stats ... by sourceNode), PPL groups by the full struct which may produce null aggregations. To extract the service name, read the sourceNode field and parse the keyAttributes.name from the returned JSON.

Service Topology (Node Connections)

Query the service dependency map to explore service-to-service connections. Use dedup nodeConnectionHash to get unique connections:

curl -sk -u "$OPENSEARCH_USER:$OPENSEARCH_PASSWORD" \
  -X POST "$OPENSEARCH_ENDPOINT/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query": "source=otel-v2-apm-service-map-* | dedup nodeConnectionHash | fields sourceNode, targetNode, sourceOperation, targetOperation"}'

Service Operations from Service Map

List all operations for a specific service from the service map:

curl -sk -u "$OPENSEARCH_USER:$OPENSEARCH_PASSWORD" \
  -X POST "$OPENSEARCH_ENDPOINT/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query": "source=otel-v2-apm-service-map-* | dedup operationConnectionHash | fields sourceNode, sourceOperation, targetNode, targetOperation"}'

Dependency Count per Service

Count how many downstream dependencies each service calls:

curl -sk -u "$OPENSEARCH_USER:$OPENSEARCH_PASSWORD" \
  -X POST "$OPENSEARCH_ENDPOINT/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query": "source=otel-v2-apm-service-map-* | dedup nodeConnectionHash | stats distinct_count(targetNode) as dependency_count by sourceNode"}'

Remote Service Identification with coalesce()

Different OTel instrumentation libraries use different attributes to identify remote services. Use coalesce() to check multiple fields in priority order:

curl -sk -u "$OPENSEARCH_USER:$OPENSEARCH_PASSWORD" \
  -X POST "$OPENSEARCH_ENDPOINT/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query": "source=otel-v1-apm-span-* | where serviceName = '\''frontend'\'' | where kind = '\''SPAN_KIND_CLIENT'\'' | eval _remoteService = coalesce(`attributes.net.peer.name`, `attributes.server.address`, `attributes.upstream_cluster`, `attributes.rpc.service`, `attributes.peer.service`, `attributes.db.system`, `attributes.gen_ai.system`, `attributes.http.host`, `attributes.messaging.destination.name`, '\'''\'' ) | where _remoteService != '\'''\'' | stats count() as calls by _remoteService | sort - calls"}'

Remote service attribute priority:

Field Used By
attributes.net.peer.name Node.js (frontend)
attributes.server.address Go, Java, .NET (checkout, cart)
attributes.upstream_cluster Envoy/Istio (frontend-proxy)
attributes.rpc.service gRPC services (recommendation)
attributes.peer.service Older OTel conventions
attributes.db.system Database clients (redis, postgresql)
attributes.gen_ai.system LLM clients (openai)
attributes.http.host HTTP clients
attributes.messaging.destination.name Message queues (Kafka, RabbitMQ)

Cross-Signal Correlation

Trace-Log Joins by traceId

Find all logs associated with a specific trace by querying the log index with the same traceId:

curl -sk -u "$OPENSEARCH_USER:$OPENSEARCH_PASSWORD" \
  -X POST "$OPENSEARCH_ENDPOINT/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query": "source=logs-otel-v1-* | where traceId = '\''<TRACE_ID>'\'' | fields traceId, spanId, severityText, body, `resource.attributes.service.name`, `@timestamp` | sort `@timestamp`"}'

Join trace spans with correlated logs using PPL join:

curl -sk -u "$OPENSEARCH_USER:$OPENSEARCH_PASSWORD" \
  -X POST "$OPENSEARCH_ENDPOINT/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query": "source=otel-v1-apm-span-* | where traceId = '\''<TRACE_ID>'\'' | join left=s right=l ON s.traceId = l.traceId logs-otel-v1-* | fields s.spanId, s.name, l.severityText, l.body"}'

Caveat: Cross-index PPL join may return 0 rows on OpenSearch 3.x due to engine limitations. If you get empty results, run two separate queries (one against otel-v1-apm-span-* and one against logs-otel-v1-*) filtered by the same traceId, then correlate the results at the application level.

Trace Tree Reconstruction

Reconstruct the full trace tree by querying all spans for a traceId, sorted by startTime with parentSpanId for hierarchy:

curl -sk -u "$OPENSEARCH_USER:$OPENSEARCH_PASSWORD" \
  -X POST "$OPENSEARCH_ENDPOINT/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query": "source=otel-v1-apm-span-* | where traceId = '\''<TRACE_ID>'\'' | fields traceId, spanId, parentSpanId, serviceName, name, startTime, endTime, durationInNanos, `status.code` | sort startTime"}'

Latency Gap Analysis

Compare parent and child span timing to identify latency gaps within a trace. First retrieve all spans, then compare startTime/endTime values between parent and child:

curl -sk -u "$OPENSEARCH_USER:$OPENSEARCH_PASSWORD" \
  -X POST "$OPENSEARCH_ENDPOINT/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query": "source=otel-v1-apm-span-* | where traceId = '\''<TRACE_ID>'\'' | fields spanId, parentSpanId, name, startTime, endTime, durationInNanos | sort startTime"}'

To find spans where the child started significantly after the parent, look for gaps between a parent's startTime and its children's startTime values. Large gaps indicate queuing, scheduling delays, or uninstrumented work.

Root Span Identification

Find the root span of a trace (where parentSpanId is empty or null):

curl -sk -u "$OPENSEARCH_USER:$OPENSEARCH_PASSWORD" \
  -X POST "$OPENSEARCH_ENDPOINT/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query": "source=otel-v1-apm-span-* | where traceId = '\''<TRACE_ID>'\'' AND parentSpanId = '\'''\'' | fields traceId, spanId, serviceName, name, durationInNanos, startTime, endTime"}'

Find all root spans (entry points) across all traces:

curl -sk -u "$OPENSEARCH_USER:$OPENSEARCH_PASSWORD" \
  -X POST "$OPENSEARCH_ENDPOINT/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query": "source=otel-v1-apm-span-* | where parentSpanId = '\'''\'' | fields traceId, spanId, serviceName, name, durationInNanos, startTime | sort - startTime | head 20"}'

GenAI Operation Types

The OpenTelemetry GenAI semantic conventions define the following operation types in attributes.gen_ai.operation.name:

Operation Type Description
invoke_agent An agent invocation — the top-level span for an agent handling a request
execute_tool A tool execution within an agent's reasoning loop
chat An LLM chat completion call
embeddings A text embedding generation call
retrieval A retrieval operation (e.g., RAG vector search)
create_agent Agent creation/initialization
text_completion A text completion call (non-chat)
generate_content A generic content generation call

Filter by Operation Type

invoke_agent

curl -sk -u "$OPENSEARCH_USER:$OPENSEARCH_PASSWORD" \
  -X POST "$OPENSEARCH_ENDPOINT/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query": "source=otel-v1-apm-span-* | where `attributes.gen_ai.operation.name` = '\''invoke_agent'\'' | fields traceId, spanId, `attributes.gen_ai.agent.name`, durationInNanos, startTime | sort - startTime | head 20"}'

execute_tool

curl -sk -u "$OPENSEARCH_USER:$OPENSEARCH_PASSWORD" \
  -X POST "$OPENSEARCH_ENDPOINT/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query": "source=otel-v1-apm-span-* | where `attributes.gen_ai.operation.name` = '\''execute_tool'\'' | fields traceId, spanId, `attributes.gen_ai.tool.name`, durationInNanos, startTime | sort - startTime | head 20"}'

chat

curl -sk -u "$OPENSEARCH_USER:$OPENSEARCH_PASSWORD" \
  -X POST "$OPENSEARCH_ENDPOINT/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query": "source=otel-v1-apm-span-* | where `attributes.gen_ai.operation.name` = '\''chat'\'' | fields traceId, spanId, `attributes.gen_ai.request.model`, `attributes.gen_ai.usage.input_tokens`, `attributes.gen_ai.usage.output_tokens`, durationInNanos, startTime | sort - startTime | head 20"}'

embeddings

curl -sk -u "$OPENSEARCH_USER:$OPENSEARCH_PASSWORD" \
  -X POST "$OPENSEARCH_ENDPOINT/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query": "source=otel-v1-apm-span-* | where `attributes.gen_ai.operation.name` = '\''embeddings'\'' | fields traceId, spanId, `attributes.gen_ai.request.model`, durationInNanos, startTime | sort - startTime | head 20"}'

retrieval

curl -sk -u "$OPENSEARCH_USER:$OPENSEARCH_PASSWORD" \
  -X POST "$OPENSEARCH_ENDPOINT/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query": "source=otel-v1-apm-span-* | where `attributes.gen_ai.operation.name` = '\''retrieval'\'' | fields traceId, spanId, serviceName, name, durationInNanos, startTime | sort - startTime | head 20"}'

create_agent

curl -sk -u "$OPENSEARCH_USER:$OPENSEARCH_PASSWORD" \
  -X POST "$OPENSEARCH_ENDPOINT/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query": "source=otel-v1-apm-span-* | where `attributes.gen_ai.operation.name` = '\''create_agent'\'' | fields traceId, spanId, `attributes.gen_ai.agent.name`, `attributes.gen_ai.agent.id`, startTime | sort - startTime | head 20"}'

text_completion

curl -sk -u "$OPENSEARCH_USER:$OPENSEARCH_PASSWORD" \
  -X POST "$OPENSEARCH_ENDPOINT/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query": "source=otel-v1-apm-span-* | where `attributes.gen_ai.operation.name` = '\''text_completion'\'' | fields traceId, spanId, `attributes.gen_ai.request.model`, durationInNanos, startTime | sort - startTime | head 20"}'

generate_content

curl -sk -u "$OPENSEARCH_USER:$OPENSEARCH_PASSWORD" \
  -X POST "$OPENSEARCH_ENDPOINT/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query": "source=otel-v1-apm-span-* | where `attributes.gen_ai.operation.name` = '\''generate_content'\'' | fields traceId, spanId, `attributes.gen_ai.request.model`, durationInNanos, startTime | sort - startTime | head 20"}'

Extended GenAI Attributes

The OTel GenAI semantic conventions provide these extended attributes on trace spans:

Attribute Type Description
attributes.gen_ai.agent.id keyword Unique identifier for the agent instance
attributes.gen_ai.agent.name keyword Human-readable agent name
attributes.gen_ai.agent.description keyword Description of the agent's purpose
attributes.gen_ai.agent.version keyword Version of the agent
attributes.gen_ai.conversation.id keyword Identifier for a multi-turn conversation session
attributes.gen_ai.tool.call.id keyword Unique identifier for a specific tool call
attributes.gen_ai.tool.type keyword Type of tool (e.g., function, retrieval)
attributes.gen_ai.tool.call.arguments text JSON-encoded arguments passed to the tool
attributes.gen_ai.tool.call.result text JSON-encoded result returned by the tool
attributes.gen_ai.request.model keyword Model requested for the operation
attributes.gen_ai.usage.input_tokens long Number of input tokens consumed
attributes.gen_ai.usage.output_tokens long Number of output tokens generated
attributes.gen_ai.operation.name keyword Operation type (see GenAI Operation Types above)

Exception and Error Querying

Query Spans with Exceptions

Find spans that contain exception events with type, message, and stacktrace:

curl -sk -u "$OPENSEARCH_USER:$OPENSEARCH_PASSWORD" \
  -X POST "$OPENSEARCH_ENDPOINT/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query": "source=otel-v1-apm-span-* | where `events.attributes.exception.type` != '\'''\'' | fields traceId, spanId, serviceName, name, `events.attributes.exception.type`, `events.attributes.exception.message`, startTime | sort - startTime | head 20"}'

Query Exception Stacktraces

curl -sk -u "$OPENSEARCH_USER:$OPENSEARCH_PASSWORD" \
  -X POST "$OPENSEARCH_ENDPOINT/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query": "source=otel-v1-apm-span-* | where `events.attributes.exception.stacktrace` != '\'''\'' | fields traceId, spanId, `events.attributes.exception.type`, `events.attributes.exception.message`, `events.attributes.exception.stacktrace` | head 10"}'

Note: This query may fail with "insufficient resources" on large indices because the where filter on events.attributes.exception.stacktrace must scan all documents. If this happens, add | head 1000 before the where filter to limit the scan scope: source=otel-v1-apm-span-* | head 1000 | where ...

Query Spans by error_type Attribute

Find spans with a specific error type category:

curl -sk -u "$OPENSEARCH_USER:$OPENSEARCH_PASSWORD" \
  -X POST "$OPENSEARCH_ENDPOINT/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query": "source=otel-v1-apm-span-* | where `attributes.error_type` != '\'''\'' | fields traceId, spanId, serviceName, `attributes.error_type`, `status.code`, startTime | sort - startTime | head 20"}'

Error Spans with Exception Details

Combine error status with exception information for a complete error view:

curl -sk -u "$OPENSEARCH_USER:$OPENSEARCH_PASSWORD" \
  -X POST "$OPENSEARCH_ENDPOINT/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query": "source=otel-v1-apm-span-* | where `status.code` = 2 | fields traceId, spanId, serviceName, name, `events.attributes.exception.type`, `events.attributes.exception.message`, `attributes.error_type`, startTime | sort - startTime | head 20"}'

Conversation Tracking

Track multi-turn conversations by grouping spans with the same conversation ID:

curl -sk -u "$OPENSEARCH_USER:$OPENSEARCH_PASSWORD" \
  -X POST "$OPENSEARCH_ENDPOINT/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query": "source=otel-v1-apm-span-* | where `attributes.gen_ai.conversation.id` != '\'''\'' | stats count() as turns, sum(`attributes.gen_ai.usage.input_tokens`) as total_input_tokens, sum(`attributes.gen_ai.usage.output_tokens`) as total_output_tokens by `attributes.gen_ai.conversation.id`"}'

View all spans in a specific conversation ordered by time:

curl -sk -u "$OPENSEARCH_USER:$OPENSEARCH_PASSWORD" \
  -X POST "$OPENSEARCH_ENDPOINT/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query": "source=otel-v1-apm-span-* | where `attributes.gen_ai.conversation.id` = '\''<CONVERSATION_ID>'\'' | fields traceId, spanId, `attributes.gen_ai.operation.name`, `attributes.gen_ai.agent.name`, startTime, durationInNanos | sort startTime"}'

Tool Call Inspection

Inspect tool call arguments and results for debugging agent behavior:

curl -sk -u "$OPENSEARCH_USER:$OPENSEARCH_PASSWORD" \
  -X POST "$OPENSEARCH_ENDPOINT/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query": "source=otel-v1-apm-span-* | where `attributes.gen_ai.operation.name` = '\''execute_tool'\'' | fields traceId, spanId, `attributes.gen_ai.tool.name`, `attributes.gen_ai.tool.call.id`, `attributes.gen_ai.tool.call.arguments`, `attributes.gen_ai.tool.call.result`, durationInNanos, startTime | sort - startTime | head 20"}'

Inspect tool calls for a specific tool:

curl -sk -u "$OPENSEARCH_USER:$OPENSEARCH_PASSWORD" \
  -X POST "$OPENSEARCH_ENDPOINT/_plugins/_ppl" \
  -H 'Content-Type: application/json' \
  -d '{"query": "source=otel-v1-apm-span-* | where `attributes.gen_ai.operation.name` = '\''execute_tool'\'' AND `attributes.gen_ai.tool.name` = '\''<TOOL_NAME>'\'' | fields traceId, `attributes.gen_ai.tool.call.arguments`, `attributes.gen_ai.tool.call.result`, durationInNanos, startTime | sort - startTime"}'

PPL Commands for Trace Analysis

The following PPL commands are particularly useful when analyzing trace data:

Command Use Case
stats Aggregate token usage, count spans by service, compute latency percentiles
where Filter spans by operation type, status code, duration threshold, attribute values
fields Select specific fields to return (traceId, spanId, attributes, etc.)
sort Order results by startTime, durationInNanos, or other fields
dedup Remove duplicate spans (e.g., deduplicate by traceId to get unique traces)
top Find the most frequent values (e.g., top services, top error types)
rare Find the least frequent values (e.g., rare operation types, rare error messages)
timechart Visualize span counts or latency over time buckets
eval Compute derived fields (e.g., convert nanoseconds to milliseconds)
head Limit result count for quick exploration
rename Rename fields for readability in output
eventstats Add aggregation results as new fields to each row without collapsing rows
trendline Calculate moving averages on latency or token usage over time
streamstats Compute running statistics (e.g., cumulative token count)
ad Anomaly detection on latency — identify spans with unusual duration patterns

Trace Index Field Reference

Key fields available in the otel-v1-apm-span-* index:

Field Type Description
traceId keyword Unique 128-bit trace identifier
spanId keyword Unique 64-bit span identifier
parentSpanId keyword Parent span ID (empty string for root spans)
serviceName keyword Service that produced the span
name text Span operation name
kind keyword Span kind (SERVER, CLIENT, INTERNAL, PRODUCER, CONSUMER)
startTime date Span start timestamp
endTime date Span end timestamp
durationInNanos long Span duration in nanoseconds
status.code integer Status code: 0=Unset, 1=Ok, 2=Error
attributes.gen_ai.operation.name keyword GenAI operation type
attributes.gen_ai.agent.name keyword Agent name
attributes.gen_ai.agent.id keyword Agent identifier
attributes.gen_ai.agent.description keyword Agent description
attributes.gen_ai.agent.version keyword Agent version
attributes.gen_ai.request.model keyword Requested model
attributes.gen_ai.usage.input_tokens long Input token count
attributes.gen_ai.usage.output_tokens long Output token count
attributes.gen_ai.conversation.id keyword Conversation identifier
attributes.gen_ai.tool.name keyword Tool name
attributes.gen_ai.tool.call.id keyword Tool call identifier
attributes.gen_ai.tool.type keyword Tool type
attributes.gen_ai.tool.call.arguments text Tool call arguments (JSON)
attributes.gen_ai.tool.call.result text Tool call result (JSON)
attributes.error_type keyword Error type category
events.attributes.exception.type keyword Exception class/type
events.attributes.exception.message text Exception message
events.attributes.exception.stacktrace text Exception stacktrace

References

AWS Managed OpenSearch

To query traces on Amazon OpenSearch Service, replace the local endpoint and authentication with AWS SigV4:

curl -s --aws-sigv4 "aws:amz:REGION:es" \
  --user "$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY" \
  -X POST https://DOMAIN-ID.REGION.es.amazonaws.com/_plugins/_ppl \
  -H 'Content-Type: application/json' \
  -d '{"query": "source=otel-v1-apm-span-* | where `attributes.gen_ai.operation.name` = '\''invoke_agent'\'' | fields traceId, spanId, `attributes.gen_ai.agent.name`, durationInNanos, startTime | sort - startTime | head 20"}'
  • Endpoint format: https://DOMAIN-ID.REGION.es.amazonaws.com
  • Auth: --aws-sigv4 "aws:amz:REGION:es" with --user "$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY"
  • The PPL API endpoint (/_plugins/_ppl) and query syntax are identical to the local stack
  • No -k flag needed — AWS managed endpoints use valid TLS certificates
Install via CLI
npx skills add https://github.com/opensearch-project/observability-stack --skill traces
Repository Details
star Stars 25
call_split Forks 27
navigation Branch main
article Path SKILL.md
More from Creator
opensearch-project
opensearch-project Explore all skills →