Troubleshooting#
You will end up knowing which of four things is wrong: the agent, the agent's connection to the router, the router's configuration, or the network. Work in that order. Almost every confusing symptom is the second one dressed up as the third.
The four-question triage#
Answer these in order. Each one tells you which section to jump to.
1. Does the agent answer?
bash
curl -sk https://127.0.0.1:8443/health{"status":"ok","version":"…"} means the agent is alive. No answer means
the agent is not running.
2. Can the agent reach the router?
bash
R=https://<router>:8443
T=wh_...
curl -sk -H "Authorization: Bearer $T" "$R/api/system"A 502 with cannot reach the router means the API key or URL is
wrong. Every page reporting the router unreachable while the
login screen works fine is this, and it is the single most common confusing failure.
3. Is the configuration what you think?
bash
show configuration commands | match <the thing>4. Is the kernel doing what the configuration says?
bash
show interfaces
show ip route
show firewall statisticsSteps 3 and 4 disagreeing is drift, and several pages flag it for you: Interfaces flags anything the configuration does not declare, Static routes flags a route declared but not installed, and QoS shows the attached policy beside the qdisc the kernel installed.
Symptom to cause#
The agent#
| Symptom | Cause and fix |
|---|---|
| Nothing answers on 8443 | The agent is not running, the address is not what you think, or the firewall. Locked out of the UI. |
The agent will not start; the journal says is mode 644; it must not be readable by group or other | A secret file is too permissive. sudo chmod 600 it — the agent refuses to read a loose secret. Applies to the API key, the break-glass token, the licence key, the OIDC secret and the fleet file. |
The agent will not start; missing --api-url or --api-key | The unit lost its flags, or /config/wheelhouse/api-key is gone. Restore from a backup. |
| The address in a document does not answer | The appliance is 8443; the developer install is 8090. The developer install. |
| The UI is blank | The agent was started without --ui-dir, or the assets are missing. After an agent upgrade, hard-reload once: a cached index.html can ask for an asset that no longer exists. |
| A page looks broken right after an upgrade | Cached assets. One hard reload. |
| The browser warns about the certificate | The self-signed certificate is self-signed by definition. Trust it once, or install a real one. |
The agent-to-router link#
| Symptom | Cause and fix |
|---|---|
| Every page reports the router unreachable, but the login screen works | The agent cannot authenticate to VyOS. The API key exists in two places — config.boot under service https api keys, and /config/wheelhouse/api-key — and both must match. This is what happens when a restore took one and not the other. Take a backup. |
502 with cannot reach the router | The same thing, or VyOS' own API is not listening. It is pinned to 127.0.0.1 on first boot. |
| Pages are slow, and the router's load is high | Every call into the VyOS API forks a shell on the router. The agent caches and primes to keep that down; something calling the API in a loop undoes that. Check wheelhouse_vyos_calls_total — Scrape the agent. |
Authentication and authorisation#
| Symptom | Cause and fix |
|---|---|
401 on everything | No principal: a missing or wrong token. |
Every mutation answers 401, reads work | A browser session that is not echoing X-Wheelhouse-CSRF. Use an API token for scripts — Issue a token. |
Every write answers 402 | No usable licence. Admin → Licence, or --license-key-file. Reads keep working. |
403 on one route | Your role. Recorded in the audit log as denied <method> <path> with the actor. The four roles. |
| Forgotten the first-run password | journalctl -u wheelhouse-agent \| grep "shown once", or sudo cat /config/wheelhouse/initial-password. |
| Locked out entirely | Locked out of the UI. |
| Lost the second-factor device | Spend a recovery code (API only today — the browser login box takes six digits), or have an admin clear the factor on Admin → Users, or wheelhouse-rescue totp --user NAME at the console. Locked out of the UI. |
Committing#
| Symptom | Cause and fix |
|---|---|
| The router rebooted after a commit | A commit-confirm window expired unconfirmed. Not a fault: the router is back on its saved configuration, so the unconfirmed changes are gone, and so is anything applied earlier and not saved. Commit-confirm. |
POST /api/stage answers 400 | An op that is not set or delete, a path with no elements, or an empty element in a path. The message names the index. |
A rollback answers 502 | The archive file for that revision is missing. ls /config/archive, and check system config-management commit-revisions. Undo the last commit. |
| A commit fails on a firewall group | A rule references a group that the same commit deletes. Remove the references first. Use groups instead of literals. |
| The Commit Bar has operations nobody staged | The reconcile loop in stage mode shares the working set with people, and its log says what it staged. |
Committing nat static stops the router's API | Known, and Wheelhouse will not do it: committing nat static through the HTTP API has been seen to crash the API process, so the agent refuses those operations at every door that reaches a commit — /api/stage, /api/fleet/{id}/stage, /api/configure, /api/reconcile with apply=true, /api/commit, /api/fleet/{id}/configure and /api/fleet/{id}/commit. Its 1:1 NAT is a destination and source rule pair instead, which the Commit Bar applies. If you hit this, it was not Wheelhouse. 1:1 NAT. |
Network#
| Symptom | Cause and fix |
|---|---|
| A port forward does not work from outside | Two rules are needed, not one. Compare the NAT rule's counter with the firewall accept's: NAT rising and firewall flat means the accept does not match. Publish a service. |
| A port forward works from outside, not from the LAN | No hairpin, or the public address changed. Publish a service, or better, split-horizon DNS. |
| A new VLAN gets leases and reaches nothing | The installer's forward rule names the physical port, and eth0.30 is not eth0. Give the VLAN an access policy on Networks: Add a network (VLAN). |
| Everything works until a failover, then nothing does | The second uplink has no source NAT rule of its own. Masquerade behind each uplink. |
| Failover never fails over | An uplink with no check is never withdrawn; the Uplinks page names any uplink in that state. A second uplink with health-checked failover. |
| A DHCP uplink's failover installs nothing | dhcp-options no-default-route set on an uplink that has a dhcp-interface failover entry. The daemon flushes its routes and installs nothing, while the load balancer's tests on the same targets pass. The Uplinks page flags it. |
| Two uplinks flap | Both WAN interfaces on the same layer-2 network. The failover daemon's per-interface probes interfere. Each uplink needs its own path. |
| A route is in the configuration and not in the table | The next hop is not reachable on a connected interface, or something with a better distance won. The Static routes page flags exactly this. |
| A route is in the table and not in the configuration | Something else installed it: a routing protocol, a DHCP client, or the failover daemon. |
| IPv6 is configured and the neighbour tables do not show it | Expected: they are v4. Network → IPv6 shows v6 addresses, neighbours and routes, and the Static routes page shows the v6 forwarding table. IPv6. |
| Large downloads stall over PPPoE | MTU. PPPoE costs eight bytes. A PPPoE uplink. |
VPN#
| Symptom | Cause and fix |
|---|---|
| WireGuard never handshakes | Keys, endpoint or the firewall. A peer added by hand holds the other side's public key, not a private one; Add device makes a device's keys for it. A WireGuard tunnel for a laptop. |
| WireGuard handshakes, no traffic | Allowed IPs, the forward rule, or — between sites — the route to the far LAN. |
| The WireGuard page says "never connected" for a peer | The kernel has not completed a handshake with it, ever — keys, endpoint or the firewall. Bytes out with nothing in means the router is sending initiations that get no reply. sudo wg show on the router says the same in the kernel's words. |
| IPsec: no IKE at all | The firewall, or the peer address. |
| IPsec: IKE completes, no child SA | Mismatched ESP proposal or traffic selectors. |
| IPsec: up, no traffic passes | Traffic selectors, source NAT applied to the tunnelled traffic, or the forward rule. An IPsec tunnel. |
| Tailscale is online, the LAN is unreachable | The subnet routes are not approved in the Tailscale admin console. Nothing on the router will tell you. Join a tailnet. |
Services and apps#
| Symptom | Cause and fix |
|---|---|
An app install fails on the image pull with toomanyrequests | Docker Hub rate-limits anonymous pulls per source address. Retry, or pull the image by hand on the router. There is no registry mirror or credential setting. |
| A commit fails with Image "…" does not exist locally | The pull step did not happen or did not finish. Install an app. |
| A commit fails with Volume "…" source path does not exist | The directory step did not happen. Off-router, run the mkdir -p the agent handed you. |
| An app is declared and not running | Its own logs: the Logs tab, or sudo podman logs <name>. |
| A restart says the app is down | The second of the two commits failed. The message says down rather than restart failed. Update check and restart. |
Restarting a stopped app answers 409 | There is nothing to bounce. |
| The health probe says not probed — this agent runs off-router | An off-router agent has no route to the container bridge, so a failed probe would say nothing about the app. |
| DNS stopped after installing a filter | The router's forwarder and the container both want port 53. Give the container its own address and forward to it. AdGuard Home. |
| A wildcard DNS record is refused | The router spells the wildcard any; * is refused. Add it on the DNS page, which takes * and writes any. Local names. |
printer.lan does not resolve | Turn on Client names in DNS under DHCP server settings, for a device that sends its name. For one that should always answer to a name, a reservation plus a host name. |
| Conntrack byte counters are missing | nf_conntrack_acct is off. set system conntrack flow-accounting, which the Sessions page stages; a system sysctl parameter for it is undone by the next firewall or NAT commit. Kernel tunables. |
Installing and upgrading#
| Symptom | Where to look |
|---|---|
| The install stopped with a message | /var/log/wheelhouse-install.log, which the installer names in its own error |
| The installer never appeared, just a login prompt | A serial console, or the machine booted an installed system rather than the medium. wheelhouse-install refuses to run on an installed system and says so. |
| The banner says the agent is not active | journalctl -u wheelhouse-agent |
No Web UI: line on the banner | The machine has no address. ip -4 addr from the console. |
| A seed did not install anything | An unattended install runs at most once per boot, and refuses a disk that already holds a system unless the seed says overwrite: true. Build a seed. |
| A seed said the router refused it, and the router had not | The 0.6.0–0.6.2 images. The installer's vbash wrapper ended with exit 0, which inside VyOS' script template leaves the configuration session rather than the script: the commit and the save both succeeded, were then reported as a refusal, and the disk was never written — so the live system came up reconfigured and nothing was installed. Fixed after 0.6.2. On an affected image, install by keyboard. |
| A seed URL was refused | Plain http:// and ftp:// are refused for a seed on the kernel command line. Use https://, or wheelhouse.insecure=1, or pin it with wheelhouse.autoinstall.sha256=<hex>. |
| An air-gapped router cannot be upgraded | POST /api/system/image accepts http(s) URLs only. Use add system image /path/to.iso at the console. |
| The agent went backwards after an image upgrade | A .deb installs into /usr, which an image upgrade replaces. Upgrade the agent only. |
Where the evidence is#
| Question | Read |
|---|---|
| What is the router's software saying? | The journal — Logs, or journalctl |
| What changed, and when? | The commit history — System — history |
| Who changed it? | The audit log — System — audit |
| What is the agent itself doing? | journalctl -u wheelhouse-agent, and /metrics |
| What is on the wire? | Diagnostics → Capture: pick the interface and a BPF filter, Start, then Download the pcap (POST /api/capture, GET /api/capture/download). |
| Does the audit log still add up? | GET /api/audit/verify, admin only. It walks the hash chain and names the first entry that does not fit. |
What to collect before asking for help#
One file. System → Support → Download support bundle, or, at the console:
bash
sudo wheelhouse-support # writes into the current directory
sudo wheelhouse-support /tmp # or into a directory you nameThe console form is the one that works when the web UI does not, which is the point of having it — and it runs in the installer's environment too, so a machine that failed to install can still produce one. Run it as root: the router's API key is readable by root alone, and without it the bundle can describe the agent but not the router, and will say so.
The bundle carries the versions, the agent's status and settings, licence state, the last
500 audit entries, the agent's log and the journal, the running configuration in both
forms, and the router's services, interfaces, routes, resource use and last commits. It is
bounded — 4 MiB an artefact, 16 MiB in total — and its manifest.json lists every item and
the reason for anything missing.
There is nowhere to send it yet. No public support address, issue tracker or chat channel is published. Collect it anyway: it is much easier to gather while the fault is happening.