Log lines worth alerting on#

The agent logs structured key-value lines with Go's log/slog, to stderr, which under systemd means the journal. Every message string below is a literal from the source, so they are safe to match on.

bash

journalctl -u wheelhouse-agent -f
journalctl -u wheelhouse-agent -p warning --since -1h

--log-level takes debug, info, warn or error; anything else silently becomes info.

The request line#

One per request, and the only high-volume line:

level=INFO msg=request method=GET path=/api/interfaces status=200 bytes=8123 ms=2 ip=192.0.2.44

WARN at 4xx, ERROR at 5xx. The path here is the real path, not the collapsed metric label, so it carries object names. ip is the source address, subject to --trust-proxy.

Alert on the rate of status=5 rather than on any single line.

Page somebody#

Message	Level	Means
`panic serving request`	ERROR	A handler crashed. The stack is on the line. The request got a `500`; the agent kept running. This is a defect — capture it.
`starting with an empty state: every account and API token is gone, and a new admin password follows`	ERROR	Both `state.json` and its backup failed to parse. The agent started empty and made a fresh admin. Every account, token and session is gone. The unreadable files were moved aside with a timestamp.
`the state file will not parse; trying the backup beside it`	ERROR	Half of the above. Recovery may still succeed — look for the next line.
`the backup state file will not parse either`	ERROR	The other half.
`unreadable state file kept for inspection`	ERROR	Names the `.corrupt-<timestamp>` file that was preserved. Keep it.
`listener stopped`	ERROR	The HTTP listener died with something other than a clean shutdown. The process then exits 1; systemd restarts it.
`audit write failed`	ERROR	An audit entry could not be appended or synced. Accountability is broken while this persists.
`audit entry dropped: no audit log is open`	ERROR	Worse: the log is not open at all.
`could not rotate the audit log` / `could not reopen the audit log after rotating`	ERROR	The log will grow past its cap, or stop recording.
`reconcile loop commit failed`	ERROR	The loop tried to commit and the router refused. Reality is drifting and nothing is closing the gap.
`another agent is running against this data dir`	ERROR	Two agents, one data directory. The second one exits 1. Usually a stray process after an upgrade.

Worth a ticket, not a page#

Message	Level	Means
`no licence key: configuration changes are disabled until one is entered (Administration → Licence)`	WARN	At startup. Every write will answer `402`.
`the licence key on file is not usable`	WARN	The stored key does not parse or does not verify.
`licence refresh`	WARN	One refresh attempt failed. Nothing changed — the current token stays good until its expiry plus grace. Alert only if it repeats for days.
`serving management UI over plain HTTP on a non-loopback address`	WARN	The management UI is on the network in clear text. The line names the three fixes.
`replacing the existing self-signed certificate: it was generated as a CA, which it should never have been`	WARN	An older build's certificate is being replaced. Anyone who imported it into a trust store should remove it.
`existing self-signed certificate has expired or expires within 30 days; replacing it`	WARN	Routine. Browsers will warn again after this.
`reconcile loop cannot read its file` / `reconcile loop cannot diff against the router`	WARN	The loop is doing nothing. `GET /api/drift` carries the same error.
`desired-state file: …`	WARN	An advisory from the file, logged once per change, not once per pass.
`secret file is missing; continuing without it`	WARN	Names the flag and the path. Expected for `admin-token` and `license-key`; not expected for anything else.
`could not refresh the state backup`	WARN	The filesystem refused a hard link. Writes are still atomic; there is just no second copy for a recovery.
`could not sync the data dir` / `could not open the data dir to sync it`	WARN	A rename may not be durable across a power cut.
`audit_retention is larger than the agent serves from memory; older entries stay in the log file`	WARN	Names what you asked for and what you got.
`clamping --reconcile-interval to the 10s floor`	WARN	Names what you asked for.
`graceful shutdown timed out`	WARN	Requests were still in flight after 15 seconds at shutdown.
`single sign-on callback failed` / `single sign-on could not complete`	WARN	One sign-on attempt failed. Alert on the rate, not the line — see the metrics below.
`single sign-on group mapping would have demoted the only admin; not doing that`	WARN	A refusal that kept the router manageable. Fix the group mapping.
`could not record token use`	WARN	Cosmetic: "last used" on an API token is stale.
`could not write the initial password file`	WARN	The console banner will not show the first-run password. It is still in the log line above.

Startup lines worth capturing#

Message	Level	Carries
`listening`	INFO	`addr`, `tls`, the router URL, and the agent version. The one line that says what this process actually is.
`licence`	INFO	`state`, `plan`, `expires`, `routers`.
`single sign-on enabled`	INFO	The issuer, client id, redirect URI, provisioning and default role — because a mismatch between that redirect URI and what is registered at the provider is the failure an operator has to diagnose.
`created first admin account — this password is shown once`	WARN	The generated first-run password. Logged exactly once, and never again.
`serving UI`	INFO	The `--ui-dir` in use.
`reconcile loop watching a desired-state file`	INFO	The file, the interval and the mode.
`fleet config loaded`	INFO	The router count, and whether the licence includes `fleet`.
`generated a self-signed certificate — browsers will warn until you add a security exception for this host or install a real certificate`	WARN	The certificate path and its expiry.
`shutting down` / `stopped`	INFO	A clean `SIGTERM`. Their absence around a restart means it was not clean.

Startup failures#

These print and the process exits — see Exit codes.

either --demo or both --api-url and --api-key are required
cannot read secret file
single sign-on is misconfigured
--reconcile-mode must be stage or commit
--tls-cert and --tls-key must be given together
cannot create data dir
cannot open state
app catalog is invalid
bootstrap failed
cannot create self-signed certificate
cannot load fleet config

An agent that will not start and keeps restarting produces one of these every RestartSec=5. That pattern in the journal is the fastest diagnosis there is.

Prefer metrics for rates#

A log line is a good alert for a fault; it is a poor one for a rate. Use the metrics for these:

Question	Series
Is the router reachable now?	`wheelhouse_router_reachable`, `wheelhouse_router_last_success_seconds`
Is the agent up?	`wheelhouse_up`, and the absence of a scrape
Are requests failing?	`wheelhouse_http_requests_total{status=~"5.."}`
Is the router slow?	`wheelhouse_vyos_request_duration_seconds`
Has the cache stopped hitting?	`wheelhouse_cache_reads_total{result="miss"}`
Has the router drifted from its file?	`wheelhouse_desired_drift`
Is somebody being refused at sign-on?	`wheelhouse_oidc_assertions_total{outcome="refused"}`

The audit log is not the journal#

Who did what is in audit.jsonl, not here. A refused request appears in both: as a WARN request line with status=403, and as an audit entry whose operation is denied <METHOD> <path>. The audit entry is the one that names the actor.

The router's own system log is a third thing again, readable at GET /api/log and on the Logs page.

Other programs#

Program	Where it logs
The installer, the seed finder	`/var/log/wheelhouse-install.log`. Under `--answers`, also stderr.
First boot	The journal, plus `logger -t wheelhouse` for its one success line.
The console banner	Nothing. It writes `/etc/issue` and is quiet.

bash

journalctl -t wheelhouse                  # first boot's own tag
sudo tail -f /var/log/wheelhouse-install.log

Checked against#

agent/main.go (newLogger, runDaemon, bootstrapAdmin, loadSecretFiles), agent/security.go (observe, recoverPanics), agent/store.go (load, flush, syncDir, auditLog), agent/license.go (loadLicenseAtStart, runLicenseRefresher), agent/tls.go (ensureSelfSignedCert), agent/desired.go (reconciler.Run, pass, logWarnings), agent/oidc.go, agent/authhttp.go (touchToken), agent/metrics.go, packaging/firstboot.sh, packaging/wheelhouse-install, packaging/wheelhouse-autoinstall, docs/deploy.md "Observability".

Log lines worth alerting on#

The request line#

Page somebody#

Worth a ticket, not a page#

Startup lines worth capturing#

Startup failures#

Prefer metrics for rates#

The audit log is not the journal#

Other programs#

See also#

Checked against#