Wheelhouse docs

Troubleshooting#

Symptom on the left, what to check on the right. Everything here is a real failure mode with a real cause; where the cause is a design decision rather than a fault, the row says so.

Start here#

bash
systemctl status wheelhouse-agent
journalctl -u wheelhouse-agent -n 50
curl -k https://127.0.0.1:8443/health          # {"status":"ok"}
sudo ls -l /config/wheelhouse

Four commands. If /health answers and the journal is quiet, the agent is fine and the problem is above it or below it.

Installing#

SymptomCheck
The install stopped with a message/var/log/wheelhouse-install.log. The installer names it in its own error, and the failure screen offers to show it.
The installer never appeared, just a login promptYou are on a serial console, or the machine booted an installed system rather than the medium. The banner on a serial console says to run sudo wheelhouse-install; on an installed system that command refuses and says so.
A seeded install did nothing/var/log/wheelhouse-install.log, lines prefixed autoinstall:. Run sudo VERBOSE=1 wheelhouse-autoinstall to see which seed it would pick.
A seeded install ran once and then stopped tryingBy design. An unattended install runs at most once per boot — /run/wheelhouse-autoinstall.attempted.
The seed was ignored on a machine that already has a systemBy design. Set overwrite: true in the seed, or remove the medium.
DISK=auto refusedThere is no disk that is not the boot medium, or there is more than one. Name one.
A seed URL was refusedPlain http:// or ftp://. Serve it over HTTPS, or boot with wheelhouse.insecure=1 if you trust that network.
No Web UI: line on the bannerThe machine has no address yet. ip -4 addr from the console.
The banner says the agent is not activejournalctl -u wheelhouse-agent.
The installer warned about no IPv6 firewallThe image has no firewall ipv6 node, so only the IPv4 ruleset was written. Write the v6 filter before you turn IPv6 on.

Reaching the UI#

SymptomCheck
The address in a document does not answerThe appliance is 8443; the developer install is 8090.
The browser cannot reach 8443The firewall, the wrong interface, or a stopped agent. From the console: curl -k https://127.0.0.1:8443/health.
A box with a WAN and no LAN cannot be reached at allThe default-deny filter admits nothing from the WAN. That is the design. Reinstall with a MGMT_SOURCE, or add a rule from the console.
The browser warns about the certificate--tls-self-signed is self-signed by definition. Add a security exception for this host, or install a real certificate. Do not import it into a root store — it is a leaf certificate for one host.
The UI is blankThe agent was started without --ui-dir, or the assets are missing. After an agent upgrade, hard-reload once: index.html carries no cache policy yet, so a stale copy can ask for an asset hash that no longer exists.

Signing in#

SymptomCheck
Forgot the first-run passwordjournalctl -u wheelhouse-agent \| grep "shown once". It is also in /config/wheelhouse/initial-password and on the console banner until it is changed.
Locked out of the UI entirelyThe break-glass token (--admin-token-file), or stop the agent and edit state.json from the console.
Lost the two-factor deviceThere are no recovery codes, and an admin cannot reset another account's second factor. Console access and a hand-edited state.json is the only route.
Every login answers 429The rate limiters: 8 failures per address per 5 minutes, 20 per account per 30 minutes. Wait, or sign in from elsewhere.
Signed in, but everything answers 403 with totp_enrolment_requiredThe require_totp policy is on and this account has not enrolled. Finish enrolment; the session can reach nothing else until it does.
Single sign-on says the provider is not respondingjournalctl -u wheelhouse-agent \| grep 'single sign-on', and compare the redirect URI in the startup line with what is registered at the provider.

The API#

SymptomCheck
Every mutation answers 401, reads are fineThe CSRF rule. A cookie session must echo X-Wheelhouse-CSRF. Use an API token for scripts.
401 invalid tokenThe token is unknown or expired — the two are deliberately indistinguishable. Issue a new one.
A token in the URL is refusedOnly /api/stream accepts one. Send Authorization: Bearer.
Every write answers 402No usable licence. Administration → Licence, or --license-key-file.
The fleet routes answer 402 with "license":"feature"The licence is fine; its plan does not carry fleet.
403 this action requires the operator roleThe principal's role. The refusal is in the audit log with the role it had.
413The body is over its cap: 1 MiB, or 16 MiB on the six configuration routes.
500 internal error — see the agent logA handler panicked. The stack is in the journal. Report it.

The router#

SymptomCheck
Everything answers 502The agent cannot reach the router. Check --api-url and the API key: sudo cat /config/wheelhouse/api-key, and show configuration commands \| match 'service https'.
502 … answered HTTP 404 … (is service https api rest configured?)The router's REST endpoints are off. With only a key configured, its API answers 404 to everything. set service https api rest.
is mode 644; it must not be readable by group or otherchmod 600 that file. The agent refuses to read a loose secret.
The agent will not start: another agent is running against this data dirA stray process holds the data directory. systemctl stop wheelhouse-agent, check for a second one, start again.
The agent will not start: the state file is newer than this agentYou downgraded. Reinstall the newer agent; opening it would silently drop fields it does not understand.
A commit was refusedThe message is the router's own words. A rejected command arrives as HTTP 200 with an error field, which is why it surfaces as a 502 rather than a false success.
The router rebooted after a commitA commit-confirm window expired unconfirmed. That is the feature working — the reboot is how the rollback is implemented.
A rollback answers 502The archive file for that revision is missing. ls /config/archive.
The 1:1 NAT editor is disabledBy design, and the page says why: committing nat static through this build's HTTP API stops the API process. Make those rules from the command line; the table still reads them.
Packet capture answers 501Not implemented. The response carries the monitor traffic command to run by hand.

Configuration as code#

SymptomCheck
plan exits 2 and you expected 02 means the router differs from the file. That is an answer, not an error.
plan refuses with an empty value is not something VyOS can setA key in the document has an empty value. Give it one, or drop it — a key with nothing after the colon switches a node on.
plan refuses with the router holds "x" here but the file gives it no valueA valueless key over a node that holds a value. No operation can express that.
apply refuses to stageStaging belongs to the running agent. Pass --agent-url, with --admin-token-file.
apply --commit refusesIt needs --confirm-minutes N. A mistake should be revertible.
The reconcile loop is doing nothingGET /api/drift carries the reason: no licence, an unreadable file, or a router that will not answer.
The loop keeps offering an operation you un-stagedBy design. The staging area is the source of truth; an operation removed deliberately is offered again next pass.
apply says its audit entry was not writtenA running agent owns the state file. Stage through --agent-url and the running agent records it.

Apps#

SymptomCheck
An install fails on the image pullThe registry rate-limits anonymous pulls by source address. Retry, or pre-pull the image.
A commit refuses a containerIts image is not on the router yet, or a volume source does not exist. POST /api/apps/pull and POST /api/apps/prepare first.
container name … is stopped — start it instead of restarting itDelete the disable node rather than bouncing it.
A restart left the app downThe second commit failed. The error says so in those words; start it by deleting container name <n> disable.
An app's health shows unknown off-routerThe agent is not on the router it manages, so it has no route to the container network. A failed probe there says nothing about the app.

Upgrades#

SymptomCheck
A stale binary after a developer upgrade./install/install.sh rollback, then upgrade again.
The package upgraded but the old agent is still runningsystemctl status wheelhouse-agent; postinst uses try-restart, which does nothing to a stopped unit.
Accounts gone after an image upgradeThey should not be — /config survives one. Check that the data directory is /config/wheelhouse and not somewhere under /var.

When you need to ask for help#

There is no support-bundle command. The four calls that carry most of what a diagnosis needs:

bash
wheelhouse-agent version
curl -sk "$R/api/system"  -H "Authorization: Bearer $T"
curl -sk "$R/api/license" -H "Authorization: Bearer $T"
curl -sk "$R/api/drift"   -H "Authorization: Bearer $T"
journalctl -u wheelhouse-agent -n 200

Where to send it: <TODO: owner>. No public support address, issue tracker or chat channel exists yet, and the source forge is private.

See also#

Checked against#

docs/deploy.md "Troubleshooting", docs/install.md "If it goes wrong", agent/main.go, agent/authhttp.go, agent/license.go, agent/store.go, agent/vyos.go (decodeVyOS), agent/desired.go, agent/apps.go, agent/security.go, packaging/wheelhouse-install, install/install.sh, SUPPORT.md, docs/security.md.

Updated 2026-09-02 troubleshooting operations