Troubleshooting#
You will end up knowing which of four things is wrong: the agent, the agent's connection to the router, the router's configuration, or the network. Work in that order. Almost every confusing symptom is the second one dressed up as the third.
The four-question triage#
Answer these in order. Each one tells you which section to jump to.
1. Does the agent answer?
bash
curl -sk https://127.0.0.1:8443/health{"status":"ok","version":"…"} means the agent is alive. No answer means
the agent is not running.
2. Can the agent reach the router?
bash
R=https://<router>:8443
T=wh_...
curl -sk -H "Authorization: Bearer $T" "$R/api/system"A 502 with cannot reach the router means the API key or URL is
wrong. Every page reporting the router unreachable while the
login screen works fine is this, and it is the single most common confusing failure.
3. Is the configuration what you think?
bash
show configuration commands | match <the thing>4. Is the kernel doing what the configuration says?
bash
show interfaces
show ip route
show firewall statisticsSteps 3 and 4 disagreeing is drift, and several pages flag it for you: Interfaces flags anything the configuration does not declare, Static routes flags a route declared but not installed, and QoS shows the attached policy beside the qdisc the kernel installed.
Symptom to cause#
The agent#
| Symptom | Cause and fix |
|---|---|
| Nothing answers on 8443 | The agent is not running, the address is not what you think, or the firewall. Locked out of the UI. |
The agent will not start; the journal says is mode 644; it must not be readable by group or other | A secret file is too permissive. sudo chmod 600 it — the agent refuses to read a loose secret. Applies to the API key, the break-glass token, the licence key, the OIDC secret and the fleet file. |
The agent will not start; missing --api-url or --api-key | The unit lost its flags, or /config/wheelhouse/api-key is gone. Restore from a backup. |
| The address in a document does not answer | The appliance is 8443; the developer install is 8090. They are different products on the same repository. The developer install. |
| The UI is blank | The agent was started without --ui-dir, or the assets are missing. After an agent upgrade, hard-reload once: index.html carries no cache policy yet, so a stale copy can ask for an asset hash that no longer exists. |
| A page looks broken right after an upgrade | Cached assets. One hard reload. |
| The browser warns about the certificate | The self-signed certificate is self-signed by definition. Trust it once, or install a real one. |
The agent-to-router link#
| Symptom | Cause and fix |
|---|---|
| Every page reports the router unreachable, but the login screen works | The agent cannot authenticate to VyOS. The API key exists in two places — config.boot under service https api keys, and /config/wheelhouse/api-key — and both must match. This is what happens when a restore took one and not the other. Take a backup. |
502 with cannot reach the router | The same thing, or VyOS' own API is not listening. It is pinned to 127.0.0.1 on first boot, deliberately. |
| Pages are slow, and the router's load is high | Every call into the VyOS API forks a shell on the router. The agent caches and primes to keep that down; something calling the API in a loop undoes that. Check wheelhouse_vyos_calls_total — Scrape the agent. |
Authentication and authorisation#
| Symptom | Cause and fix |
|---|---|
401 on everything | No principal: a missing or wrong token. |
Every mutation answers 401, reads work | A browser session that is not echoing X-Wheelhouse-CSRF. Use an API token for scripts — Issue a token. |
Every write answers 402 | No usable licence. Administration → Licence, or --license-key-file. Reads keep working, deliberately. |
403 on one route | Your role. Recorded in the audit log as denied <method> <path> with the actor. The three roles. |
| Forgotten the first-run password | journalctl -u wheelhouse-agent \| grep "shown once", or sudo cat /config/wheelhouse/initial-password. |
| Locked out entirely | Locked out of the UI. |
| Lost the second-factor device | There is no admin reset and no recovery codes. Console, agent stopped, state.json edited by hand. Locked out of the UI. |
Committing#
| Symptom | Cause and fix |
|---|---|
| The router rebooted after a commit | A commit-confirm window expired unconfirmed. That is the feature working — the router is back on the previous configuration. Commit-confirm. |
POST /api/stage answers 400 | An op that is not set or delete, a path with no elements, or an empty element in a path. The message names the index. |
A rollback answers 502 | The archive file for that revision is missing. ls /config/archive, and check system config-management commit-revisions. Undo the last commit. |
| A commit fails on a firewall group | A rule references a group that the same commit deletes. Remove the references first. Use groups instead of literals. |
| The Commit Bar has operations nobody staged | The reconcile loop in stage mode shares the working set with people. That is by design; the log says what it staged. |
Committing nat static stops the router's API | Known: on this platform build, committing nat static through the HTTP API crashes the API process. The CLI accepts the same lines, and the 1:1 NAT editor is switched off for this reason. 1:1 NAT, and the console. |
Network#
| Symptom | Cause and fix |
|---|---|
| A port forward does not work from outside | Two rules are needed, not one. Compare the NAT rule's counter with the firewall accept's: NAT rising and firewall flat means the accept does not match. Publish a service. |
| A port forward works from outside, not from the LAN | No hairpin, or the public address changed. Publish a service, or better, split-horizon DNS. |
| A new VLAN gets leases and reaches nothing | The installer's forward rule names the physical port, and eth0.30 is not eth0. Add a VLAN. |
| Everything works until a failover, then nothing does | The second uplink has no source NAT rule of its own. Masquerade behind each uplink. |
| Failover never fails over | An uplink with no check is never withdrawn; the Uplinks page names any uplink in that state. A second uplink with health-checked failover. |
| A DHCP uplink's failover installs nothing | dhcp-options no-default-route set on an uplink that has a dhcp-interface failover entry. The daemon flushes its routes and installs nothing, while the load balancer's tests on the same targets pass. Verified on the bench, 2026-09-02; the page flags it. |
| Two uplinks flap | Both WAN interfaces on the same layer-2 network. The failover daemon's per-interface probes interfere. Each uplink needs its own path. |
| A route is in the configuration and not in the table | The next hop is not reachable on a connected interface, or something with a better distance won. The Static routes page flags exactly this. |
| A route is in the table and not in the configuration | Something else installed it: a routing protocol, a DHCP client, or the failover daemon. |
| IPv6 is configured and nothing shows it | Expected. IPv6 is configurable and not observable — no route view, no uplink model. IPv6. |
| Large downloads stall over PPPoE | MTU. PPPoE costs eight bytes. A PPPoE uplink. |
VPN#
| Symptom | Cause and fix |
|---|---|
| WireGuard never handshakes | Keys, endpoint or the firewall. The router's peer entry holds the laptop's public key, not a private one. A WireGuard tunnel for a laptop. |
| WireGuard handshakes, no traffic | allowed-ips, or the forward rule. |
| The WireGuard page shows no handshake times | It cannot. Handshakes and transfer counters need wg show exposed by the agent, which it does not do. Use sudo wg show. |
| IPsec: no IKE at all | The firewall, or the peer address. |
| IPsec: IKE completes, no child SA | Mismatched ESP proposal or traffic selectors. |
| IPsec: up, no traffic passes | Traffic selectors, source NAT applied to the tunnelled traffic, or the forward rule. An IPsec tunnel. |
| Tailscale is online, the LAN is unreachable | The subnet routes are not approved in the Tailscale admin console. Nothing on the router will tell you. Join a tailnet. |
Services and apps#
| Symptom | Cause and fix |
|---|---|
An app install fails on the image pull with toomanyrequests | Docker Hub rate-limits anonymous pulls per source address. Retry, or pull the image by hand on the router. There is no registry mirror or credential setting. |
| A commit fails with Image "…" does not exist locally | The pull step did not happen or did not finish. Install an app. |
| A commit fails with Volume "…" source path does not exist | The directory step did not happen. Off-router, run the mkdir -p the agent handed you. |
| An app is declared and not running | Its own logs: the Logs tab, or sudo podman logs <name>. |
| A restart says the app is down | The second of the two commits failed. The message says down rather than restart failed because the difference matters. Update check and restart. |
Restarting a stopped app answers 409 | There is nothing to bounce. |
| The health probe says not probed — this agent runs off-router | Correct and honest: an off-router agent has no route to the container bridge, so a failed probe would say nothing about the app. |
| DNS stopped after installing a filter | The router's forwarder and the container both want port 53. Give the container its own address and forward to it. AdGuard Home. |
| A wildcard DNS record is refused | VyOS' forwarder answers names, not wildcards. dnsmasq. |
printer.lan does not resolve | The forwarder does not register DHCP leases. A reservation plus a static host mapping, or dnsmasq. |
| Conntrack byte counters are missing | nf_conntrack_acct is not enabled. Kernel tunables. |
Installing and upgrading#
| Symptom | Where to look |
|---|---|
| The install stopped with a message | /var/log/wheelhouse-install.log, which the installer names in its own error |
| The installer never appeared, just a login prompt | A serial console, or the machine booted an installed system rather than the medium. wheelhouse-install refuses to run on an installed system and says so. |
| The banner says the agent is not active | journalctl -u wheelhouse-agent |
No Web UI: line on the banner | The machine has no address. ip -4 addr from the console. |
| A seed did not install anything | An unattended install runs at most once per boot, and refuses a disk that already holds a system unless the seed says overwrite: true. Build a seed. |
| A seed URL was refused | Plain http:// and ftp:// are refused for a seed on the kernel command line. Use https://, or wheelhouse.insecure=1, or pin it with wheelhouse.autoinstall.sha256=<hex>. |
| An air-gapped router cannot be upgraded | POST /api/system/image accepts http(s) URLs only. Use add system image /path/to.iso at the console. |
| The agent went backwards after an image upgrade | A .deb installs into /usr, which an image upgrade replaces. Upgrade the agent only. |
Where the evidence is#
| Question | Read |
|---|---|
| What is the router's software saying? | The journal — Logs, or journalctl |
| What changed, and when? | The commit history — System — history |
| Who changed it? | The audit log — System — audit |
| What is the agent itself doing? | journalctl -u wheelhouse-agent, and /metrics |
| What is on the wire? | Not from the UI: POST /api/capture answers 501 and hands back the monitor traffic command to run at the console. |
What to collect before asking for help#
There is no support-bundle command. Four calls are the procedure:
bash
R=https://<router>:8443
T=wh_... # a viewer token
curl -sk "$R/health"
curl -sk -H "Authorization: Bearer $T" "$R/api/audit?limit=300"
curl -sk -H "Authorization: Bearer $T" "$R/api/history"
journalctl -u wheelhouse-agent -n 500Where to send it: <TODO: owner>.
See also#
- Locked out of the UI
- Working from the console
- Take a backup that is actually complete
- What is finished, and what is not
- Support
Checked against docs/deploy.md ·
docs/install.md ·
agent/main.go ·
agent/apps.go ·
docs/multi-wan.md ·
SUPPORT.md