Telemetry#

Every live number in the web UI arrives by one of four paths, and they have different freshness, different costs and different failure modes. This page names each one, so that "the graph is flat but the link is busy" has somewhere to start. It also documents /metrics, which is the only path in the product that keeps anything.

The short version: the agent samples interface counters itself on a 5-second ticker and keeps 20 minutes of them in memory; a WebSocket pushes a raw interface read every 2 seconds for liveness; everything else is an HTTP poll served from a read cache; and /metrics exposes all of it in Prometheus text format for something else to store.

1. The interface sampler#

The agent starts a sampler at boot and never stops it — agent/main.go creates it as newHistory(240, 5*time.Second) and runs it in a goroutine.

Each tick it makes one uncached call to show interfaces counters, parses the table, and for every interface stores a Sample:

{ "t": 1756800000000, "rxb": 91827364, "txb": 5544332, "rx": 128410.4, "tx": 9922.0 }

rxb/txb are the router's monotonic byte counters. rx/tx are bytes per second, computed from the difference between this read and the previous one divided by the elapsed milliseconds. That is the only honest way to get a rate out of a monotonic counter, and it has two consequences worth knowing:

• The first sample of any interface has a rate of zero, because there is nothing to difference it against.
• A counter that goes backwards reports zero for that step, not a negative rate. A counter only goes backwards when the interface reset, and a negative throughput would be a lie — nonNegativeRate() in agent/metrics.go.

The ring holds 240 samples per interface, which at a 5-second interval is 20 minutes. Older samples fall off the front. It is in memory only and is deliberately not persisted: this is a live view, not a metrics store.

GET /api/interfaces/history?points=N returns the newest N per interface plus the sampler's interval, so a chart can label its own time axis without assuming a rate:

curl -sk -H "Authorization: Bearer $T" \
  "https://<router>:8443/api/interfaces/history?points=90"

{ "interval_ms": 5000, "series": { "eth0": [ … ], "eth1": [ … ] } }

The default is 120 points when points is absent.

The sampler bypasses the read cache. collectInterfaceCounters() calls vyos.ShowUncached, because rate arithmetic over TTL-stale reads produces rates for intervals that did not happen.

2. The WebSocket stream#

GET /api/stream upgrades to a WebSocket and pushes one frame every 2 seconds. The whole app shares a single socket — StreamProvider in ui/src/lib/stream.tsx — because the Layout and the Dashboard each opening their own doubled the uncached reads the agent made of the router.

A frame is one of two shapes:

{ "ts": 1756800000000, "type": "interface_stats", "data": { … } }
{ "ts": 1756800000000, "type": "error",  "error": "the router's own words" }

The error frame is the point of the design. A stream that goes quiet when the router stops answering looks exactly like an idle router, and the panel wearing the indicator says live. So an unreadable router arrives as an explicit error, and the Dashboard's indicator switches to telemetry unavailable with the router's message on hover.

Two more properties, both deliberate:

• The read is uncached. A telemetry stream replaying a 3-second cache is not live.
• The socket reads as well as writes. A peer that has gone away — a closed laptop lid — is noticed between samples. Without the read pump, the loop would keep asking the router for statistics nobody is watching, every two seconds, indefinitely.

The browser reconnects on close with exponential backoff, doubling from 1 s to a ceiling of 15 s — ui/src/lib/hooks.ts.

3. The read cache and the primer#

Everything else in the UI is an ordinary HTTP poll, and almost every one of those is served from the agent's read cache rather than from the router.

This exists because of one measured property of the base platform: the router serialises its HTTP API. Seven parallel reads take as long as seven sequential ones, and each op-mode read forks cli-shell-api on the router. The only way to keep pages fast is to not ask the router every time — agent/cache.go.

There are two freshness classes:

Configuration correctness is not a matter of timing: the agent is the thing that commits, so it drops every cfg, cmds and raw entry the moment a Configure, ConfigureConfirm or LoadConfig call returns. The one-hour TTL is a backstop against a missed invalidation, not the mechanism.

Both classes are served stale-while-revalidate: a stale hit returns the old value immediately and starts exactly one background refresh. Only the very first read of a key ever blocks. A refresh that began before an invalidation is discarded rather than stored — otherwise a pre-commit answer would be written back with a fresh timestamp and served for the whole TTL.

The primer keeps the cache warm. At startup it fetches 24 configuration subtrees, the whole configuration rendered as commands, 24 op-mode reads, every interface's detail and the installed-app set with its health probes. After that a 10-second ticker re-checks the configuration entries only, and a commit triggers an immediate re-warm of the same set.

4. /metrics#

The agent exposes Prometheus text format at GET /metrics. This is the only place in the product where telemetry is meant to be kept; the charts in the UI are a window, not a store.

By default the route requires the viewer role like every other read. Starting the agent with --metrics-public serves it unauthenticated, which is reasonable on a loopback bind and is a decision to make deliberately on any other — agent/main.go.

curl -sk -H "Authorization: Bearer $T" https://<router>:8443/metrics

What it exposes#

The three desired-state series appear only when --reconcile-file is set.

Two design notes that affect what you can alert on#

A monotonic failure counter cannot tell you the router is down now. wheelhouse_vyos_failures_total rising an hour ago and rising this minute look the same in a counter. That is why wheelhouse_router_reachable and wheelhouse_router_last_success_seconds exist as gauges; alert on those.

Latency is split between the agent and the router. wheelhouse_http_request_duration_seconds is what the browser waited; wheelhouse_vyos_request_duration_seconds is what the router took. When a page is slow, the second series says whether the router or the agent is the slow one. Histogram buckets are, in seconds: 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, +Inf.

A useful first pair of expressions:

wheelhouse_router_reachable == 0
rate(wheelhouse_cache_reads_total{result="miss"}[5m])

The second one is the cache quietly having stopped hitting, which is the difference between a fast appliance and a slow one and was previously invisible.

Reading the failure modes#

What telemetry does not do#

• No retention in the product. 20 minutes of interface rates in memory, and nothing else. There is no embedded time-series database and no long-term graph.
• No per-host history. Top talkers is an aggregation of the live conntrack table. For history, export flows to a collector or run a monitoring app from the catalogue.
• No alerting. The agent sends nothing anywhere. /metrics is the integration point.
• No push. Nothing is posted outward; every metric is scraped.

See also#

• Dashboard — the panels these numbers land in.
• Diagnostics — health — connection-tracking pressure, the other live counter set.
• System — audit — the record that is persisted, and its limits.

Checked against agent/metrics.go, agent/cache.go, agent/primer.go, agent/vyos.go, agent/main.go, agent/opmode.go, ui/src/lib/stream.tsx, ui/src/lib/hooks.ts.