Telemetry#

Every live number in the web UI arrives by one of four paths, with different freshness, different cost and different failure modes.

The agent samples interface counters on a 5-second ticker and keeps 20 minutes of them in memory; the Overview's current rates are the newest of those samples. A WebSocket pushes a raw interface read every 2 seconds, and the router chip in the top bar says whether it is connected. Everything else is an HTTP poll served from a read cache. /metrics exposes all of it in Prometheus text format for something else to store.

1. The interface sampler#

The agent starts a sampler at boot and never stops it. Each tick it makes one uncached call to show interfaces counters, parses the table, and for every interface stores a sample:

{ "t": 1756800000000, "rxb": 91827364, "txb": 5544332, "rx": 128410.4, "tx": 9922.0 }

rxb and txb are the router's monotonic byte counters. rx and tx are bytes per second, computed from the difference between this read and the previous one over the elapsed milliseconds. Two consequences:

• The first sample of any interface has a rate of zero. There is nothing to difference it against.
• A counter that goes backwards reports zero for that step, not a negative rate. A counter goes backwards when the interface has been reset.

The ring holds 240 samples per interface — 20 minutes at a 5-second interval. Older samples fall off the front. It is in memory only and is not persisted.

GET /api/interfaces/history?points=N returns the newest N per interface plus the sampler's interval, so a chart can label its own time axis:

curl -sk -H "Authorization: Bearer $T" \
  "https://<router>:8443/api/interfaces/history?points=90"

{ "interval_ms": 5000, "series": { "eth0": [ … ], "eth1": [ … ] } }

The default is 120 points when points is absent. The Overview asks for points=3 every 5 seconds and shows the newest rate of each uplink.

The sampler bypasses the read cache. Rate arithmetic over stale reads would produce rates for intervals that did not happen.

2. The WebSocket stream#

GET /api/stream upgrades to a WebSocket and pushes one frame every 2 seconds. The whole app shares a single socket, so opening several pages does not multiply the reads the agent makes of the router.

A frame is one of two shapes:

{ "ts": 1756800000000, "type": "interface_stats", "data": { … } }
{ "ts": 1756800000000, "type": "error",  "error": "the router's own words" }

An unreadable router arrives as an explicit error frame. A stream that went quiet instead would be indistinguishable from an idle router.

Every signed-in page holds the stream open. The router chip at the left of the top bar shows whether it is connected: a green dot while it is, amber while it is not. Hover over the chip to read Live data connected or Live data not connected.

The stream's read is uncached. A browser that has gone away — a closed laptop lid — is noticed between samples, and the stream stops reading from the router.

The browser reconnects on close with exponential backoff, doubling from 1 s to a ceiling of 15 s.

3. The read cache#

Everything else in the UI is an ordinary HTTP poll, and almost every one is served from the agent's read cache rather than from the router.

The router serialises its HTTP API: seven parallel reads take as long as seven sequential ones, and each op-mode read forks cli-shell-api on the router.

There are three freshness classes:

The agent is the thing that commits, so it drops every configuration entry the moment a Configure, ConfigureConfirm or LoadConfig call returns. The one-hour TTL is a backstop against a missed invalidation, not the mechanism.

Both classes are served stale-while-revalidate: a stale hit returns the old value immediately and starts one background refresh. Only the first read of a key blocks. A refresh that began before an invalidation is discarded rather than stored.

A primer keeps the cache warm. At startup it fetches 24 configuration subtrees, the whole configuration rendered as commands, 24 op-mode reads, every interface's detail and the installed-app set with its health probes. After that it re-checks the configuration entries on a timer, and a commit triggers an immediate re-warm of the same set. Op-mode entries are not re-warmed on a timer.

4. /metrics#

The agent exposes Prometheus text format at GET /metrics. This is the only place in the product where telemetry is meant to be kept; the charts in the UI are a window, not a store.

The route requires the viewer role like every other read. Starting the agent with --metrics-public serves it unauthenticated — reasonable on a loopback bind, a decision to make carefully on any other.

curl -sk -H "Authorization: Bearer $T" https://<router>:8443/metrics

What it exposes#

The three desired-state series appear only when --reconcile-file is set.

What to alert on#

wheelhouse_vyos_failures_total is monotonic: rising an hour ago and rising this minute look the same. Alert on the gauges instead — wheelhouse_router_reachable and wheelhouse_router_last_success_seconds.

Latency is split in two. wheelhouse_http_request_duration_seconds is what the browser waited; wheelhouse_vyos_request_duration_seconds is what the router took. When a page is slow, the second series says which of the two is slow. Histogram buckets are, in seconds: 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, +Inf.

A first pair of expressions:

wheelhouse_router_reachable == 0
rate(wheelhouse_cache_reads_total{result="miss"}[5m])

The second is the cache having stopped hitting: a rising miss rate means pages are going to the router instead of to memory.

Reading the failure modes#

What telemetry does not do#

• Retention stops at a month, and at the interface. 20 minutes of interface rates in memory, plus a per-interface archive of 24 hours at one-minute and 31 days at one-hour resolution on disk. There is no embedded time-series database, nothing per host, and nothing beyond 31 days.
• No per-host history. Top talkers is an aggregation of the conntrack table as the agent last read it (the read cache's snapshot; the Sessions page shows its age). For history, export flows to a collector or run a monitoring app from the catalogue.
• No alerting. The agent sends nothing anywhere. /metrics is the integration point.
• No push. Nothing is posted outward; every metric is scraped.

See also#

• Overview — the current rates and the traffic chart on the landing page.
• Diagnostics — health — connection-tracking pressure, the other live counter set.
• System — audit — the record that is persisted, and its limits.