System — audit log#

The Audit log tab of the System page is the agent's record of what was asked of it: who, from where, with which role, what they did, and whether it worked. It covers the web interface, the API, and the tools that write to the same log. It is one of three separate records the product keeps.

The table#

The Audit log panel lists the most recent 300 entries, newest first, and reads them again every ten seconds. Its header shows how many there are.

Who is one of:

• an account name, for someone signed in to the web interface, or the name that was tried, for a failed sign-in;
• token:<label>, for an API token;
• admin-token, for the break-glass token the agent can be started with;
• reconcile-loop, for the desired-state loop;
• system, for the agent itself, such as creating the first administrator;
• rescue, for wheelhouse-rescue at the router's console;
• upnp or upnp:<device>, for ports the router opened or closed for a device on the network.

An entry#

Selecting an entry opens it. The title is the action, and beneath it the date and time in the browser's time zone. Then:

• Who, Role, From (the address the request came from) and Result.
• Concerning, for an action that names something: the revision restored (Revision 3), the user or token added with its role (alice (operator)), the image, the management address changed (192.0.2.1:8443 to every interface), or the address a restarted agent listens on (Listening on 192.0.2.1:8443).
• "The action failed", with the router's or the agent's reason, when it did.
• A note from the agent where it wrote one, such as "takes effect when the agent restarts" on a changed management address.
• Changes: every change the entry carries, as sentences grouped by area, in the words the pending changes use. Applying changes and discarding them carry the changes; so do the desired-state loop's entries and changes applied to a fleet router.

Adding to the pending changes is not recorded. Applying them is ("Applied changes"), whether with Apply changes or Apply with auto-revert; so are Keep changes ("Kept changes applied with auto-revert") and Discard ("Discarded pending changes"), with what was discarded. Both outcomes are recorded: a failed apply, a refused sign-in and a restore the router rejected are all entries, with the reason.

Every action and the words the tab uses for it are listed in Audit entries.

In Advanced mode#

With Advanced mode turned on (Account → Preferences):

• the table has an Operation column, with the code the agent records (commit, rollback, create-user, …);
• a failure's reason is shown as the router or the agent gave it, in full;
• an entry's panel adds Recorded: the Path the entry names, such as the revision file restored, and the Commands it carried.

Secrets are never recorded. A change that set a password, a private key, a pre-shared key or an SNMP community is recorded with [redacted] in place of the value, for every reader and in the file itself. Entries written by an agent older than 0.9.1 kept their secrets; the page shows them blanked, but the file on the router still holds those older entries as they were written, so treat a log that goes back that far with the same care as the router's own configuration.

Where it is stored#

An append-only file, one entry per line, in the agent's data directory. Each entry is written and flushed to disk at once, so it survives a power cut that follows the action it describes.

Older entries stay in the file; they are not served. A line cut short by a power cut is skipped, and the entries either side of it are still good.

The hash chain#

Every entry carries a sequence number and the hash of the entry before it, so the file adds up or it does not.

• Editing one entry breaks that entry's own hash.
• Deleting one breaks the sequence, and breaks the link of the entry after it.
• Inserting one breaks the same link.

The check that walks the chain re-reads both generations from disk and reports each place the record does not fit: the file, the line, the sequence number, and which of those three things went wrong. It also reports the head: the last sequence number and its hash, which is the value to write down somewhere else. There is no button for it on this tab; it is an API call for an administrator (For automation).

A log written before the agent kept a chain is not rewritten to add one. The first chained entry seals it ("Started the audit log’s integrity chain"): where an entry holds the hash of the one before it, this one holds a digest of everything that came before, and the check reports those older entries as written before the chain, not as tampered with. A file that simply ends because the power went out mid-write is reported as ending, not as broken.

Getting a copy off the router#

Two ways, and they answer different questions.

A witness, continuously. Audit log export on Agent settings (Send audit entries to a syslog server, with Server, Port, Protocol and Facility) copies every entry to a syslog collector as it is written, before it is flushed to disk, so the copy survives the power cut that loses the one on the router. Each message is the exact line the file holds, hash and sequence number included, so the collector's copy verifies the same way. Over UDP, an entry too large for one message is sent without its commands, with a note saying so; TCP carries every entry whole.

It is off by default. It is also not a spooler: while the collector cannot be reached, entries are dropped and counted rather than queued on the router's disk.

A copy, now. The whole log, both generations, byte for byte, is an API call for an administrator (For automation). Taking the log off the router is itself recorded, as "Downloaded the audit log"; the entry is written after the copy, so a copy never contains the record of its own export.

A backup does not carry the audit log. It carries the most recent 500 entries for reading, and a restore never writes them back.

What it cannot do#

• It is not tamper-proof. The chain makes an edit detectable to someone holding an earlier copy of the hashes. It does not stop root from making one, and nothing signs the file.
• There is no verify or export button on this tab. Both are API calls today.
• It records what went through the agent, and only that. A change made at the router's console is in the configuration history and the system log, and not here. wheelhouse-rescue is the exception: it writes its console actions into the chain, as rescue.
• An auto-revert window that runs out is not an entry. The router restarts with its saved configuration on its own; the agent does not record it.
• Entries beyond the retention setting are not served. They are in the file, and the export returns all of them.

For automation#

GET /api/audit needs only the viewer role: everyone who can see the router can see who changed it. It returns the newest entries first, 300 unless limit says otherwise:

curl -sk -H "Authorization: Bearer $T" \
  "https://<router>:8443/api/audit?limit=50" | python3 -m json.tool

GET /api/audit/verify walks the chain and also reports whether the syslog export is reaching its collector and how many entries it has dropped. GET /api/audit/export downloads the whole log. Both need the admin role:

curl -sk -H "Authorization: Bearer $T" \
  https://<router>:8443/api/audit/verify | python3 -m json.tool
curl -sk -OJ -H "Authorization: Bearer $T" https://<router>:8443/api/audit/export

Every field an entry carries is described in Audit entries.

See also#

• System — history — what each revision changed, whoever made it.
• Logs — the router's system log, which leaves the router by a different route.
• Agent settings — the retention setting, and the collector this log is copied to.
• Ship logs off the box — both exports on one page.
• The audit log — what the log is for, and how it differs from the other two records.