Prometheus metrics#

GET /metrics serves a Prometheus text exposition. It needs the viewer role unless the agent was started with --metrics-public. The content type is text/plain; version=0.0.4; charset=utf-8, every series carries HELP and TYPE, and the output is sorted, so a diff between two scrapes is readable.

curl -sk "$R/metrics" -H "Authorization: Bearer $T"

Re-derive the list of series names for the version you are running:

grep -o 'wheelhouse_[a-z_]*' agent/metrics.go | sort -u

Every series#

Agent identity#

HTTP#

Both are recorded before the request is authenticated, so an anonymous 401 is counted. status="101" is the telemetry WebSocket, and for that route the duration is the lifetime of the connection rather than a request latency.

The router#

The read cache#

Every router read the cache does not serve forks cli-shell-api on the router, so a falling hit rate shows up as load on the box.

Configuration state#

Accounts#

Single sign-on#

Both series are present but empty on an agent with no identity provider configured.

Interface throughput#

These come from the sampler, not from the telemetry stream: one counter read every five seconds for every interface at once, with the rate derived from the difference between successive reads. A counter that goes backwards — an interface reset — reports 0 for that step rather than a negative rate. An interface the sampler has seen only once has a rate of 0 until its second sample.

Histogram buckets#

Both histograms share one bucket layout, in seconds:

0.005  0.01  0.025  0.05  0.1  0.25  0.5  1  2.5  5  10  +Inf

Each exports _bucket (cumulative), _sum and _count. The buckets are not configurable.

How the route label is built#

Request paths carry object names, and a label built from a client-supplied string would mint a fresh time series per request. Paths are collapsed:

Requests are observed before they are authenticated, so the /api/{unknown} rule is what keeps an anonymous GET /api/<anything> from minting a series.

Label escaping#

Label values escape backslash, double quote and newline, and nothing else. Other control characters become ?, so a path holding a tab character cannot produce an exposition the reference parser rejects.

A scrape config#

scrape_configs:
  - job_name: wheelhouse
    scheme: https
    tls_config: { insecure_skip_verify: true }   # self-signed by default
    authorization:
      type: Bearer
      credentials: wh_…
    static_configs:
      - targets: ['192.0.2.1:8443']

Issue a viewer token for this; Prometheus needs nothing more. --metrics-public removes the need for a credential entirely, and with it the authentication on a route that reports your account count, your session count and your interface names.

What is not here#

Log lines, not metrics#

Some conditions are only ever a log line — a secret file with wrong permissions, a state file that would not parse, a reconcile pass that could not read its file. Those are on Log lines worth alerting on.

See also#

• The HTTP API
• Endpoint index
• The telemetry stream — where the status="101" series comes from.
• Log lines worth alerting on
• Every flag — --metrics-public.
• Telemetry