Locked out of the UI#

You will end up signed in again. This page is the order to try things in, from least destructive to most. Most of them need physical or serial access to the machine.

Work out which of these you have before you start, because they need different answers.

Before you start#

The console. Physical keyboard and monitor, a serial connection, or the VM console. Working from the console covers getting a shell there. Almost nothing on this page can be done over the network.

wheelhouse-rescue, which does most of this for you#

Installed at /usr/bin/wheelhouse-rescue on every Wheelhouse router. Run with no subcommand it is a menu:

  1  Why will the agent not start?        (changes nothing)
  2  Reset a web UI password
  3  Remove an account's second factor
  4  Re-run first boot (a new router API key)
  5  Restore a configuration from a file
  6  Forget the saved management listen address
  7  Factory reset: wipe this router's Wheelhouse data
  s  Open a shell
  q  Quit

Each item is also a subcommand: diagnose, password, totp, firstboot, restore FILE, listen --reset and factory-reset.

Four rules it is built around, worth knowing before you rely on it:

• It runs at the console, and only there. Every action refuses unless its controlling terminal is a real console — a virtual terminal, a serial port, the hypervisor's — and refuses inside an SSH session whatever the terminal claims. That check is the whole authorisation model: it resets passwords without knowing one, so what replaces the password is physical presence. There is no option to turn it off.
• It says what it will change before it changes it, including what it will not change, and asks you to type something that is not "y".
• It never weakens anything silently. A password reset leaves two-factor exactly as it found it and says so. The three actions that weaken something — removing a second factor, forgetting the management address, a factory reset — are each their own command, each confirmed on its own.
• Every action goes into the audit log, attributed to rescue and to the console it was run from, and into the log's hash chain.

Where it does not help, said plainly: it cannot recover the console password, which belongs to a router account and not to a web interface account (with the web interface still reachable, set a new one under System → Settings → Router accounts and apply it); it cannot revoke API tokens, so a reset password does not invalidate one; and it removes a two-factor secret rather than recovering it.

The rescue boot entry#

If the machine will not start far enough to run that, the install medium carries Rescue an installed Wheelhouse system in both bootloaders, beside the ordinary live entries. It is appended after them, so nothing you already start by default changes. That entry starts the same live system with wheelhouse.rescue on the kernel command line, which puts wheelhouse-rescue on the first console instead of the installer.

From there the menu has an eighth item, Find an installed system on the disks, to work on an installed Wheelhouse system.

A password you cannot remember#

If the router has never been signed into#

The agent logs a one-time generated password the first time it creates the admin account, and writes it to /config/wheelhouse/initial-password, mode 0600. The console banner shows it until the password is changed, and the file is removed the moment that account's password changes.

sudo cat /config/wheelhouse/initial-password
journalctl -u wheelhouse-agent | grep "shown once"

If you have a break-glass token#

--admin-token-file names a file holding a token that authenticates as admin without an account. The shipped service points it at /config/wheelhouse/admin-token.

R=https://127.0.0.1:8443
T=$(sudo cat /config/wheelhouse/admin-token)
curl -sk -H "Authorization: Bearer $T" "$R/api/auth/me"

With that you can reset your own account's password through the admin API:

curl -sk -X PATCH -H "Authorization: Bearer $T" -H 'Content-Type: application/json' \
  -d '{"password":"a new password"}' "$R/api/admin/users/yourname"

If you have another admin account#

Sign in with it, open Admin → Users, select your account, enter a New password and press Save changes. The account is asked to change it at its next sign-in. That is the reason to give every person their own account — Add an operator.

At the console: wheelhouse-rescue password#

The supported way, and the one to reach for before hand-editing anything.

sudo wheelhouse-rescue password --user admin

It prints exactly what it will do, then does it: replaces that account's Argon2id salt and hash, sets must change password so the new one gets you in once and the web interface asks for another, revokes that account's open sessions, deletes the first-boot initial-password file if it is still there, and stops and restarts the agent around the write so a running agent cannot overwrite the edit from memory. It writes a rescue-password-reset entry, and it prints the new password on the screen and puts it in no file — not the audit log either, which records that a reset happened, not what to.

It hashes by asking the agent's own binary to do it — a throwaway --demo data directory on tmpfs, whose state file it then lifts the salt and hash out of — so the bytes come from exactly the code that will later check them and cannot drift from it.

Two things it leaves alone and says so: two-factor, which stays exactly as it was (if that is what you have lost, run wheelhouse-rescue totp instead), and API tokens, which are independent credentials this does not revoke.

Last resort: edit the store#

Only when wheelhouse-rescue is not available — an old image, or a router whose agent package is gone. With the agent stopped, state.json is a JSON file you can edit. Two versions of this, in increasing order of damage.

Remove one account, so the agent no longer knows it — you then create it again from another admin account, or let the bootstrap run.

sudo systemctl stop wheelhouse-agent
sudo cp -a /config/wheelhouse/state.json /config/wheelhouse/state.json.manual-backup
sudoedit /config/wheelhouse/state.json          # delete the object from "users"
sudo systemctl start wheelhouse-agent

Start over completely, which drops every account, token, session and setting:

sudo systemctl stop wheelhouse-agent
sudo mv /config/wheelhouse/state.json /config/wheelhouse/state.json.old
sudo systemctl start wheelhouse-agent
journalctl -u wheelhouse-agent | grep "shown once"

The agent bootstraps a fresh admin account and logs its password once. The audit log in audit.jsonl is untouched by this, and the router's configuration is untouched entirely — that lives in config.boot, not here.

A second factor you lost#

Three answers, and only the third needs the console.

1. A recovery code#

Enrolment issued ten, shown once. Each signs you in one time, in place of a code from the app. POST /api/auth/login takes one in a recovery field:

R=https://<router>:8443
curl -sk -X POST -H 'Content-Type: application/json' \
  -d '{"username":"jo","password":"…","recovery":"k7m2-pq4r-wxn9"}' "$R/api/auth/login"

The answer says recovery_code_used and how many are left, and the audit log records Signed in with a recovery code.

2. Ask another admin to clear it#

Admin → Users, select the account, and under Two-factor authentication press Clear two-factor authentication, then Clear jo’s two-factor authentication? to go ahead.

T=wh_...                                    # an admin token, or the break-glass token
curl -sk -X POST -H "Authorization: Bearer $T" "$R/api/admin/users/jo/totp/clear"

It deletes the secret and every remaining recovery code, signs the account out everywhere, and is recorded as Cleared a user’s two-factor authentication. The account then signs in with its password alone.

It does not turn off Require two-factor authentication under Admin → Agent settings. With that on, the account is asked to set up two-factor at its next sign-in and can reach nothing else until it has — which is the point.

An admin cannot clear their own factor this way: the page says Turn off your own two-factor authentication on the Account page., and the route refuses with 409. It also refuses when there is nothing to clear. The break-glass token and an admin API token can call it, which is what you want when every admin has lost their phone at once: neither is an account, so neither has a factor of its own to shortcut.

3. At the console#

sudo wheelhouse-rescue totp --user jo

It states what it is about to do — the secret is deleted, not disabled, so the old authenticator's codes will never work again and the account enrols afresh; the password is not changed; no other account is touched; and if two-factor is required, the next sign-in lands in enrolment — and asks you to type the account name to confirm. It stops the agent for the write and starts it again, and records rescue-totp-clear.

Unlike the admin route above, this one leaves recovery.json alone. Nothing there is reachable while the account has no second factor, and enrolling again replaces the set — but if you want the old codes gone rather than superseded, press Issue new codes on the Account page once you are back in.

Then sign in with the password and enrol again, saving the new recovery codes this time.

The port does not answer#

Work down this list. Each step tells you whether to stop.

1. Ask the router why.

sudo wheelhouse-rescue diagnose

It changes nothing, and it checks everything known to stop the agent starting, in the order that would stop it: the binary runs and what version it is; the service's state, and whether first boot succeeded; the data directory's free space; each secret file's presence, size and mode; whether state.json parses and whether it still has an admin who can sign in; whether anything still holds the store lock while the service is stopped; whether the API key the agent holds is the one the router is configured with; whether something else has port 8443; and whether the clock is plausible. It then prints the last twenty journal lines, explains any fatal pattern it recognises in them, and ends with the first problem it found, because that is usually the cause.

A clean run is information too. If nothing is wrong it says so, and tells you the problem is between you and a running agent — the firewall, the address, or TLS.

By hand, if you want the same three answers:

systemctl status wheelhouse-agent
journalctl -u wheelhouse-agent -n 50
curl -sk https://127.0.0.1:8443/health

A "status":"ok" from the router itself means the agent is fine and the problem is the network path. Go to step 3.

2. If it will not start, the journal names the reason. The three that actually happen, all of which diagnose checks for you:

A fourth that diagnose finds and a journal reading will not: the agent's API key is not the router's. The agent starts, cannot authenticate, and every page reports the router unreachable while the sign-in page works perfectly. That is what restoring half a backup looks like, and wheelhouse-rescue firstboot is the fix.

3. Is the address what you think it is?

ip -4 addr

The console banner prints one Web UI: line per addressed interface. If it says "waiting for an address on any interface", the machine has not got one. When a management address is set, the banner shows that one address instead, with the interface it is on — or not on any interface, or this console only for a loopback address.

4. Is the firewall in the way?

This is the common one, and it is usually a rule you just applied.

show configuration commands | match firewall

If a change shut the door, restore the configuration from before it at the console:

configure
rollback 1

Undo a change is the same thing from the web interface, once you are back in. If the change was applied with Apply with auto-revert and nobody pressed Keep changes, the router restarts by itself when the window runs out and starts with its saved configuration, and this problem solves itself — at the cost of any applied change that had not been saved.

5. Is the agent listening where you expect?

The shipped service passes --addr 0.0.0.0:8443, so the agent listens on every address and only the firewall keeps the web interface off the internet. A management address set under Admin → Agent settings → Management address narrows that to one address, from the next agent restart.

• An address that will not bind — a lease that moved, an interface that was deleted — is not a lockout. The agent falls back to the service's address, warns, and records Could not listen on the configured management address in the audit log.
• An address that binds and cannot be reached from where you are — the loopback, or a network you are no longer on — is. Nothing is broken from the agent's point of view, so nothing falls back. Forget the address at the console:

sudo wheelhouse-rescue listen --reset

It clears the saved address and starts the agent again on the service's address, which on the shipped service is every address. It is menu item 6.

Getting in when the network is gone entirely#

The console can make any change the web interface makes: configure, then set and delete lines — Advanced mode in the web interface shows the lines behind each change — then commit and save.

If the router will not start at all, or the new image broke it, the boot menu lists every installed image — choose the previous one. Upgrade the whole system.

The install medium's Rescue an installed Wheelhouse system entry, described at the top of this page, is for the case where the installed system will not give you a usable console at all. From it, wheelhouse-rescue find lists the installed Wheelhouse systems on the machine's disks.

wheelhouse-rescue restore FILE puts a configuration back from the console. It takes either a config.boot or a file of set commands, tells you which it thinks it has and what the difference is against what is running — a commands file merges, a config.boot replaces — copies the present config.boot aside first, applies the whole thing in the router's own transaction so a rejected configuration is rejected whole, and saves it.

Check it worked#

Sign in through the browser, not just through curl. Then do the three things that stop this happening again:

1. Create a break-glass token, as above, and keep it wherever you keep the router's other credentials.
2. Issue and keep new recovery codes — Account → Two-factor authentication → Issue new codes — somewhere that is not this router.
3. Take a backup — Take a backup that is actually complete — so the next time this is twenty minutes rather than an afternoon.

Every rescue action you just took is in the audit log, attributed to rescue and to the console you ran it from. Read it back on the System page's Audit log tab.

See also#

• Working from the console
• Undo a change
• Support — the bundle to produce before you start changing things, if you want somebody else to read what happened.
• Enrol a second factor
• Add an operator
• Accounts and sessions
• Troubleshooting