monkit-metrics

name: monkit-metrics description: Use when investigating latency, throughput, error rate, or concurrency of any Storj Go service (satellite, storagenode, gateway, linksharing, etc.), comparing performance across pods/nodes/regions, finding which Go functions are instrumented, or interpreting monkit function / function_times series from Thanos/Prometheus.

Monkit Prometheus Metrics

Overview

Monkit (defer mon.Task()(&ctx)(&err)) emits two metrics for every instrumented Go method:

• function_times — latency (percentiles, min/max, recent samples). Use for "how slow?"
• function — call counters and concurrency. Use for "how often?" / "how many in flight?" / "how many fail?"

Both share the same name, scope, and environment labels, so you usually query them with the same filter set.

function_times — latency

Structure

function_times{name="...", field="...", kind="...", scope="...", ...}

Values are in seconds — multiply by 1000 for ms.

field values

function — call counters & concurrency

Same name/scope/env labels as function_times. No kind label — outcomes are encoded in field.

field values

When to use function vs function_times

Example: compare QPS across regions

sum by (environment_name) (
    rate(function{
        name="__Endpoint__CommitObject",
        scope="storj_io_storj_satellite_metainfo",
        field="successes"
    }[5m])
)

Naming Convention

Same name, different scope

The same name can appear under multiple scope values. Example: __Endpoint__CommitObject lives in storj_io_storj_satellite_metainfo (the handler), and the RPC wrapper for the same call shows up as _metainfo_Metainfo_CommitObject in storj_io_common_rpc_rpctracing. Always include scope (or scope=~"...") when comparing or you will mix unrelated timings.

Note: Storj scopes double — storj_io_storj_satellite_metainfo — because the import path is storj.io/storj/.... Expected.

Finding Instrumented Methods

Methods with defer mon.Task()(&ctx)(&err) emit metrics:

rg "defer mon\.Task\(\)" satellite/metainfo/
rg -n "func.*CommitObject|defer mon\.Task" satellite/metainfo/endpoint_object.go

To trace direct callees: find method calls in the function body, locate their definitions, check for defer mon.Task().

Querying via Grafana MCP

This is the primary path — Thanos sits behind Grafana auth and direct curl won't work.

Datasource UIDs

Verify with mcp__grafana__list_datasources(type="prometheus") if these change.

Discovery workflow

# 1. Confirm the metric exists in this datasource
mcp__grafana__list_prometheus_metric_names(
    datasourceUid="adoggz37zfda8f", regex="function_times")

# 2. Discover real label values (don't guess `environment_name`s)
mcp__grafana__list_prometheus_label_values(
    datasourceUid="adoggz37zfda8f", labelName="environment_name",
    matches=[{"filters":[{"name":"__name__","type":"=","value":"function_times"}]}])

# 3. Query
mcp__grafana__query_prometheus(
    datasourceUid="adoggz37zfda8f",
    expr='function_times{name="__Endpoint__CommitObject", scope=~".*satellite_metainfo", field="r99", kind="success"}',
    queryType="range", startTime="now-1h", endTime="now", stepSeconds=60)

• queryType="instant" → single point right now. Use to sanity-check a label combo exists.
• queryType="range" → time series. Requires startTime, endTime, stepSeconds.
• Times accept RFC3339 (2026-05-19T22:00:00Z) or relative (now, now-1h, now-30m).
• Use mcp__grafana__generate_deeplink to hand the engineer a Grafana Explore URL when reporting findings.

Correlating with logs

When a latency spike lines up with errors, jump to Loki:

mcp__grafana__query_loki_logs(...)            # raw log lines around the spike
mcp__grafana__find_error_pattern_logs(...)    # Sift-based pattern detection

Direct HTTP fallback

Only when Prometheus is reachable without auth (local Prometheus, dev cluster):

curl -sG 'http://HOST:9090/api/v1/query' \
    --data-urlencode 'query=function_times{name="__Endpoint__CommitObject",field="r99",kind="success"}'

Comparing instances / regions

Multiple series per name (one per pod). Aggregate before computing stats.

PromQL — let Prometheus do the work:

quantile by (environment_name, name) (
    0.99,
    function_times{name=~"...", field="r99", kind="success"}
) * 1000

Python — reusable pattern: see compare-instances.py in this directory (range query + aggregate across pods, two endpoints).

Common Mistakes