The test suites#

There are six suites in this repository and they are not evenly distributed. The agent has over two hundred Go test functions and CI runs them race-clean; the installer and the migration tooling have four Python suites written against replayed prompts and a sample configuration; the catalogue has a shape check that needs no network; this documentation site has its own; and the web UI has none. Knowing which of those is true for the code you just changed is the difference between a green pipeline that means something and one that means the types line up.

What exists#

Every count on this page can drift, so re-derive rather than trusting it:

grep -h '^func Test' agent/*_test.go | wc -l     # Go test functions
grep -c 'def test_' site/tests/test_build.py     # documentation-site cases
grep -hc '    def test_' tools/tests/test_*.py   # each Python suite

The agent's tests#

cd agent
go vet ./...
go test -race -count=1 ./...

-race is not optional. The agent serves concurrent requests, keeps a shared read cache, and mutates a store from several goroutines; several of the tests exist specifically because a race was found — TestSessionWritesDoNotRaceTheSessionList is one of them. -count=1 defeats the test cache, which is what CI wants and what you want after changing a fixture.

They are ordinary Go tests in package main, so they can reach unexported functions directly. There is no mocking framework: where a test needs a router, it stands up an httptest server that speaks the VyOS API's shape.

What the larger files cover:

TestRoutesRegisterWithoutConflict in agent/main_test.go catches a route pattern that collides with an existing one, which is a whole class of mistake that would otherwise appear as a panic at start-up.

The Python suites#

All four run under plain python3 or under pytest, and none needs a dependency.

The OPNsense importer#

tools/tests/test_opnsense_import.py runs the importer against tools/tests/opnsense-sample.xml and asserts the commands a person would check by hand after a migration: the addresses, the forwards with their aliases resolved, the reservations — and, just as important, the things the importer is supposed to refuse rather than guess at. A tunnel is not an Ethernet port; a port the router had switched off does not come back on.

The seed reader#

tools/tests/test_seed.py checks that cloud-config becomes installer answers. Every case runs twice, once with PyYAML available and once with it hidden, and the two must agree — because the image the reader runs on may not have the library.

The installer's commands#

tools/tests/test_install_commands.py uses wheelhouse-install --answers FILE --commands, which prints the set lines an install would run and changes nothing. That makes the part of the installer that decides what a router ends up being testable without a router.

The case worth knowing about is WAN with no LAN. The firewall used to be nested inside the LAN branch, so choosing "set the LAN up later" produced an install with no ruleset at all, on a machine whose only interface faced the internet, with the admin UI listening on every address. That is the highest-consequence installer bug found so far, and this suite is what stops it coming back.

The installer's pty driver#

tools/tests/test_install_driver.py replays the base installer's prompts. The driver is fifteen regular expressions matched against another project's questions, run once, on a disk that is about to be repartitioned — there is no second chance on a real install, so a stand-in installer prints the same strings and the test asserts what the driver did with them.

The last case is the one that matters: a prompt the table does not know has to stop the install quickly and say so. When that guard was written against the stripped string it never fired, and an unexpected question became a fifteen-minute silence after the disk had already been erased.

The catalogue check#

python3 scripts/check-images.py agent/catalog.json --offline

Offline on purpose. It needs no network and no registry credentials, so it cannot flake and cannot be skipped when the network is slow, and it catches the malformed references that once put three broken images in the catalogue. The online run — reachability, and --pin for digests — is deliberate and human, done against a diff someone reads.

The live endpoint script#

agent/test.sh is not a unit test. It drives a running agent against a real router with a bearer token:

bash agent/test.sh https://192.0.2.1:8443 "$TOKEN"

It walks the read plane, checks that an unauthenticated call is refused with a 401, and checks the write path's most common failure honestly: a set the router rejects must come back as a 502 carrying VyOS' own words, not a generic message. If this router accepts the deliberately invalid address, the script takes it back out of the candidate configuration rather than leaving it for the next commit to trip over.

CI does not run it, because CI has no router.

The documentation site#

python3 site/build.py --strict
python3 site/tests/test_build.py

Cases over the Markdown renderer, the link checker and the built output. The build itself fails on a missing title, a broken internal link, a #fragment that names no heading on the target page, a refused link scheme, or a repo: citation naming a file that is not in the checkout — which is how a page that has started lying about the code becomes a build failure. See Contributing.

What is not covered#

Also uncovered, and worth knowing before you rely on a green pipeline:

• The image build. Nothing tests build-iso.sh short of running it. What stands in for a test is the assertions inside it and inside the two branding hooks, plus the inspection step in the ISO workflow.
• An install to disk. The command generation is tested; the install itself is exercised by hand. It has been done in VMs and on the owner's hardware, which is how three installer bugs were found — but there is no automation for it, and no hardware but the owner's has been tried (packaging/README.md).
• An in-place upgrade. An add system image from one Wheelhouse release to the next is not recorded.
• Performance. No CPU, memory or throughput figure has been measured, in CI or on hardware. The <150 MB RSS and <3% idle CPU numbers in PLAN.md §11 are a target.

Before you push#

(cd agent && go vet ./... && go test -race -count=1 ./...)
python3 tools/tests/test_opnsense_import.py
python3 tools/tests/test_seed.py
python3 tools/tests/test_install_commands.py
python3 tools/tests/test_install_driver.py
python3 scripts/check-images.py agent/catalog.json --offline
python3 site/tests/test_build.py
(cd ui && npx tsc --noEmit -p . && npx vite build --logLevel warn)

That is what What CI does will run. The definition of done in AGENTS.md asks for more than the pipeline can check: a feature works against the live bench instance, not just against mocks, and a terminal capture or a screenshot goes in the pull request.

See also#

• What CI does — the pipeline these suites run inside.
• Building the agent · Building the web UI
• Working agreements — the definition of done.
• Contributing — the documentation build's own checks.
• What is finished, and what is not — the product-level version of the gaps listed above.
• Status codes — what the live endpoint script asserts about a rejected set.
• Coming from OPNsense — the migration the importer suite guards.

Checked against#

agent/test.sh · agent/main_test.go · agent/security_test.go · agent/catalog_test.go · agent/endpoints_test.go · tools/tests/test_opnsense_import.py · tools/tests/test_seed.py · tools/tests/test_install_commands.py · tools/tests/test_install_driver.py · scripts/check-images.py · site/tests/test_build.py · .forgejo/workflows/ci.yml · packaging/README.md · docs/ui.md · docs/hardware.md · AGENTS.md