Prometheus metrics#

GET /metrics serves a hand-rolled Prometheus text exposition — no client library, so the agent stays one dependency-light binary. It needs the viewer role unless the agent was started with --metrics-public. The content type is text/plain; version=0.0.4; charset=utf-8, every series carries HELP and TYPE, and the output is sorted so a diff between two scrapes is readable.

curl -sk "$R/metrics" -H "Authorization: Bearer $T"

Re-derive the list of series names for the version you are running:

grep -o 'wheelhouse_[a-z_]*' agent/metrics.go | sort -u

Every series#

Agent identity#

HTTP#

Both are recorded before the request is authenticated, so an anonymous 401 is counted. status="101" is the telemetry WebSocket, and for that route the duration is the lifetime of the connection rather than a request latency.

The router#

A monotonic failure counter cannot tell an alert that the router is down now rather than having had a blip an hour ago. wheelhouse_router_reachable and wheelhouse_router_last_success_seconds can.

The read cache#

A cache that has quietly stopped hitting is the difference between a fast appliance and a slow one, and there was no way to see it before these existed. Every router read the cache does not serve forks cli-shell-api on the router.

Configuration state#

Accounts#

Single sign-on#

Both series are present but empty on an agent with no identity provider configured.

Interface throughput#

These come from the sampler, not from the telemetry stream: one counter read every five seconds for every interface at once, with the rate derived from the difference between successive reads. A counter that goes backwards — an interface reset — reports 0 for that step rather than a negative rate. An interface the sampler has seen only once has a rate of 0 until its second sample.

Histogram buckets#

Both histograms share one bucket layout, in seconds:

0.005  0.01  0.025  0.05  0.1  0.25  0.5  1  2.5  5  10  +Inf

Each exports _bucket (cumulative), _sum and _count. There is no configuration for the buckets; changing them means changing durationBuckets in agent/metrics.go.

How the route label is built#

Request paths carry object names, and a label built from a client-supplied string is an unbounded exposition — a fresh time series per request. routeLabel collapses them:

The /api/{unknown} rule matters because requests are observed before they are authenticated: without it, an anonymous GET /api/<anything> minted a series.

Label escaping#

Label values escape backslash, double quote and newline, and nothing else; other control characters become ?. Go's %q would have written \t, \x01 and \u00a0 too, which the reference parser rejects — and since every request is observed before authentication, one GET of a path holding a tab character broke the whole scrape for good.

A scrape config#

scrape_configs:
  - job_name: wheelhouse
    scheme: https
    tls_config: { insecure_skip_verify: true }   # self-signed by default
    authorization:
      type: Bearer
      credentials: wh_…
    static_configs:
      - targets: ['192.0.2.1:8443']

Issue a viewer token for this. Prometheus needs nothing else, and a viewer token in a scrape config is the smallest credential that works. --metrics-public removes the need for a credential entirely; it also removes the authentication from a route that reports your account count, your session count and your interface names, so bind accordingly.

What is not here#

Log lines, not metrics#

Some conditions are only ever a log line — a secret file with wrong permissions, a state file that would not parse, a reconcile pass that could not read its file. Those are on Log lines worth alerting on.

See also#

• The HTTP API
• Endpoint index
• The telemetry stream — where the status="101" series comes from.
• Log lines worth alerting on
• Every flag — --metrics-public.
• Telemetry

Checked against#

agent/metrics.go (handleMetrics, durationBuckets, promLabel, History), agent/security.go (observe, routeLabel, apiAreas), agent/cache.go (counts), agent/vyos.go (done, Health, clientFor), agent/primer.go, agent/main.go (routes, newHistory), agent/metrics_test.go, docs/deploy.md "Observability".