metrics-query-cookbook

name: metrics-query-cookbook description: Cookbook of ready-to-use PromQL queries, preset catalog, metric name dictionaries, and label references for Ceph storage, network traffic, pod statistics, and MTV migrations. Use when you need specific queries, exact metric names, or label filters.

Metrics Query Cookbook

Ready-to-use queries, preset catalog, and metric name/label references for OpenShift clusters with ODF, OVN-Kubernetes, KubeVirt, and Forklift/MTV.

All examples use the kubectl-metrics MCP server tools (metrics_read and metrics_help).

Output format guidance: Use default (markdown) when presenting to user. Use output: "json" only when you need to parse values programmatically. Use selector to filter results by labels post-query.

Preset Catalog

Every preset works as both an instant (default) and range query. Pass start to get a time-series trend.

Cluster & Namespace

Forklift / MTV Migration

Storage Metrics (Ceph / ODF)

Cluster-wide storage health

metrics_read { "command": "query", "flags": { "query": "ceph_health_status", "output": "markdown" } }

Result: 0 = OK, 1 = WARN, 2 = ERR.

Storage capacity

metrics_read { "command": "query", "flags": { "query": "ceph_cluster_total_bytes", "output": "markdown" } }

metrics_read { "command": "query", "flags": { "query": "ceph_cluster_total_used_bytes", "output": "markdown" } }

Pool-level statistics

metrics_read { "command": "query", "flags": { "query": "ceph_pool_percent_used * 100", "output": "markdown" } }

Pool I/O rates

metrics_read { "command": "query", "flags": { "query": "rate(ceph_pool_rd[5m])", "output": "markdown" } }

metrics_read { "command": "query", "flags": { "query": "rate(ceph_pool_wr[5m])", "output": "markdown" } }

OSD operation latency

metrics_read { "command": "query", "flags": { "query": "rate(ceph_osd_op_latency_sum[5m]) / rate(ceph_osd_op_latency_count[5m])", "output": "markdown" } }

Placement group health

metrics_read { "command": "query", "flags": { "query": "ceph_pg_total", "output": "markdown" } }

metrics_read { "command": "query", "flags": { "query": "ceph_pg_degraded", "output": "markdown" } }

Available labels on ceph_* metrics

Storage metrics reference

Network Traffic Metrics

Network traffic by namespace

metrics_read { "command": "preset", "flags": { "name": "namespace_network_rx", "output": "markdown" } }

metrics_read { "command": "preset", "flags": { "name": "namespace_network_tx", "output": "markdown" } }

Network traffic by pod in a namespace

Replace TARGET_NAMESPACE with the actual namespace -- ASK the user if not known.

metrics_read {
  "command": "query",
  "flags": { "query": "topk(10, sort_desc(sum by (pod)(rate(container_network_receive_bytes_total{namespace=\"TARGET_NAMESPACE\"}[5m]))))", "output": "markdown" }
}

metrics_read {
  "command": "query",
  "flags": { "query": "topk(10, sort_desc(sum by (pod)(rate(container_network_transmit_bytes_total{namespace=\"TARGET_NAMESPACE\"}[5m]))))", "output": "markdown" }
}

Network errors and drops by namespace

metrics_read { "command": "preset", "flags": { "name": "namespace_network_errors", "output": "markdown" } }

Node-level network throughput

metrics_read {
  "command": "query",
  "flags": { "query": "instance:node_network_receive_bytes_excluding_lo:rate1m + instance:node_network_transmit_bytes_excluding_lo:rate1m", "output": "markdown" }
}

Available labels on network metrics

Network metrics reference

Pod and Container Statistics

Pod count by namespace

metrics_read { "command": "query", "flags": { "query": "topk(15, count by (namespace)(kube_pod_info))", "output": "markdown" } }

Pod phase summary

metrics_read { "command": "preset", "flags": { "name": "cluster_pod_status", "output": "markdown" } }

Container CPU usage by namespace

metrics_read { "command": "preset", "flags": { "name": "namespace_cpu_usage", "output": "markdown" } }

Container memory usage by namespace

metrics_read { "command": "preset", "flags": { "name": "namespace_memory_usage", "output": "markdown" } }

Container restart counts (instability indicator)

metrics_read { "command": "preset", "flags": { "name": "pod_restarts_top10", "output": "markdown" } }

Pods with high recent restarts (use debug_read for details)

After finding pods with high restarts, use debug_read to get pod details and logs:

debug_read { "command": "list", "flags": { "resource": "pods", "namespace": "<NAMESPACE>", "query": "where status.containerStatuses[0].restartCount > 5", "output": "markdown" } }

debug_read { "command": "logs", "flags": { "name": "<POD_NAME>", "namespace": "<NAMESPACE>", "tail": 100, "query": "where level = 'ERROR'", "output": "markdown" } }

Available labels on pod/container metrics

Pod/container metrics reference

Forklift / MTV Migration Metrics

Available labels on mtv_* metrics

All mtv_* metrics share these labels for filtering and grouping:

MTV migration metrics reference

Migration status overview

metrics_read { "command": "preset", "flags": { "name": "mtv_migration_status", "output": "markdown" } }

Migration plan status

metrics_read { "command": "preset", "flags": { "name": "mtv_plan_status", "output": "markdown" } }

Migration data transfer and throughput

metrics_read { "command": "preset", "flags": { "name": "mtv_data_transferred", "output": "markdown" } }

metrics_read { "command": "preset", "flags": { "name": "mtv_net_throughput", "output": "markdown" } }

metrics_read { "command": "preset", "flags": { "name": "mtv_storage_throughput", "output": "markdown" } }

Migration duration

metrics_read { "command": "preset", "flags": { "name": "mtv_migration_duration", "output": "markdown" } }

metrics_read { "command": "preset", "flags": { "name": "mtv_avg_migration_duration", "output": "markdown" } }

Migration alerts

metrics_read { "command": "query", "flags": { "query": "mtv_plan_alert_status", "output": "markdown" } }

Narrowing migration metrics with label filters

Use {label="value"} in PromQL or use the selector flag:

metrics_read { "command": "query", "flags": { "query": "mtv_migration_data_transferred_bytes", "selector": "provider=vsphere", "output": "markdown" } }

metrics_read { "command": "query", "flags": { "query": "mtv_migration_data_transferred_bytes{mode=\"Cold\"}", "output": "markdown" } }

metrics_read { "command": "query", "flags": { "query": "mtv_migration_data_transferred_bytes{provider=\"ovirt\", mode=\"Warm\"}", "output": "markdown" } }

metrics_read { "command": "query", "flags": { "query": "mtv_migrations_status_total{status=\"Failed\"}", "output": "markdown" } }

metrics_read { "command": "query", "flags": { "query": "mtv_workload_migrations_status_total{plan=\"PLAN_UUID\", status=\"Failed\"}", "output": "markdown" } }

Grouping migration metrics

metrics_read { "command": "query", "flags": { "query": "sum by (provider)(mtv_migration_data_transferred_bytes)", "output": "markdown" } }

metrics_read { "command": "query", "flags": { "query": "sum by (mode)(mtv_migration_data_transferred_bytes)", "output": "markdown" } }

metrics_read { "command": "query", "flags": { "query": "sum by (provider, mode)(mtv_migration_data_transferred_bytes)", "output": "markdown" } }

metrics_read { "command": "query", "flags": { "query": "sum by (status, provider)(mtv_migrations_status_total)", "output": "markdown" } }

metrics_read { "command": "query", "flags": { "query": "avg by (provider)(mtv_migration_duration_seconds)", "output": "markdown" } }

metrics_read { "command": "query", "flags": { "query": "sum by (plan, status)(mtv_workload_migrations_status_total)", "output": "markdown" } }

metrics_read { "command": "query", "flags": { "query": "sum by (provider, status)(mtv_plans_status)", "output": "markdown" } }

Network traffic of migration pods

During active Forklift migrations, data-transfer pods run in the target namespace. Migration pod names follow the pattern {plan-name}-{vm-id}-{random} (e.g. test-vmware-metrics-vm-43-tws62).

Step 1 -- Discover migration pods:

VMware/general migration pods (carry a plan label):

debug_read { "command": "list", "flags": { "resource": "pods", "namespace": "<NAMESPACE>", "selector": "plan", "output": "markdown" } }

oVirt/OpenStack populator pods (named populate-{uuid}-...):

debug_read { "command": "list", "flags": { "resource": "pods", "namespace": "<NAMESPACE>", "query": "where name ~= '^populate-'", "output": "markdown" } }

Step 2 -- Query network traffic for discovered pods:

Use the pod names from Step 1 to build a regex filter (replace POD1|POD2 with the actual names):

metrics_read {
  "command": "query",
  "flags": { "query": "topk(10, sort_desc(sum by (pod)(rate(container_network_receive_bytes_total{namespace=\"TARGET_NAMESPACE\",pod=~\"POD1|POD2\"}[5m]))))", "output": "markdown" }
}

metrics_read {
  "command": "query",
  "flags": { "query": "topk(10, sort_desc(sum by (pod)(rate(container_network_transmit_bytes_total{namespace=\"TARGET_NAMESPACE\",pod=~\"POD1|POD2\"}[5m]))))", "output": "markdown" }
}

Short-lived pod network metrics

Pods that run under 60 seconds (e.g. oVirt/OpenStack populator pods) may not have container-level network metrics (container_network_*). This is because cadvisor needs 1-2 collection cycles (10-20s) to establish network namespace tracking, and the pod may complete before tracking starts. CPU and memory metrics are unaffected.

Node-level network metrics capture the transfer at the node level. Determine which node ran the pod (spec.nodeName or kube_pod_info), then query RX and TX together:

metrics_read {
  "command": "query_range",
  "flags": {
    "query": [
      "instance:node_network_receive_bytes_excluding_lo:rate1m{instance=~\"NODE_NAME.*\"}",
      "instance:node_network_transmit_bytes_excluding_lo:rate1m{instance=~\"NODE_NAME.*\"}"
    ],
    "name": ["node_rx", "node_tx"],
    "start": "<MIGRATION_START>",
    "end": "<MIGRATION_END>",
    "step": "30s",
    "output": "markdown"
  }
}

Compare against baseline before/after the migration window to isolate transfer traffic.

CPU activity confirms the pod was active during the window:

metrics_read { "command": "query_range", "flags": { "query": "rate(container_cpu_usage_seconds_total{pod=\"<POD>\",namespace=\"<NS>\"}[1m])", "start": "<START>", "end": "<END>", "step": "30s", "output": "markdown" } }

Completed migration historical queries

When querying metrics for a migration that already finished, use the plan's start/completion timestamps as absolute time bounds:

1. Get timestamps from the plan:

mtv_read { "command": "describe plan", "flags": { "name": "<PLAN>", "namespace": "<NS>", "output": "markdown" } }

2. Use ISO-8601 start/end in query_range:

metrics_read {
  "command": "query_range",
  "flags": {
    "query": "sum by (pod)(rate(container_network_receive_bytes_total{namespace=\"<NS>\"}[5m]))",
    "start": "2025-06-15T10:00:00Z",
    "end": "2025-06-15T12:30:00Z",
    "step": "60s",
    "output": "markdown"
  }
}

Do not use relative offsets like -1h for completed migrations -- the data may fall outside that window.

Checking migration pod status with debug_read

To investigate migration pod issues alongside metrics:

debug_read { "command": "list", "flags": { "resource": "pods", "namespace": "<NAMESPACE>", "selector": "plan", "output": "markdown" } }

debug_read { "command": "logs", "flags": { "name": "<POD_NAME>", "namespace": "<NAMESPACE>", "tail": 100, "query": "where level = 'ERROR'", "output": "markdown" } }

Network traffic of the Forklift operator itself

metrics_read { "command": "preset", "flags": { "name": "mtv_forklift_traffic", "output": "markdown" } }

KubeVirt VMI migration metrics

These track live VM migrations (vMotion-style), not Forklift cold migrations:

metrics_read { "command": "preset", "flags": { "name": "mtv_vmi_migrations_pending", "output": "markdown" } }

metrics_read { "command": "preset", "flags": { "name": "mtv_vmi_migrations_running", "output": "markdown" } }

Quick Health Dashboard

Run key queries for a cluster overview:

metrics_read { "command": "preset", "flags": { "name": "cluster_cpu_utilization", "output": "markdown" } }

metrics_read { "command": "preset", "flags": { "name": "cluster_memory_utilization", "output": "markdown" } }

metrics_read { "command": "query", "flags": { "query": "ceph_health_status", "output": "markdown" } }

metrics_read { "command": "preset", "flags": { "name": "namespace_network_rx", "output": "markdown" } }

metrics_read { "command": "preset", "flags": { "name": "mtv_migration_status", "output": "markdown" } }