Benchmark: Pavois vs OpenSCAP / Lynis (reproducible)
A reproducible benchmark of Pavois against OpenSCAP and Lynis on a fresh Debian 12 VM. Everything here is rebuilt from a clean snapshot; nothing is hand-asserted. The scripts live in the repo (tools/benchmark/effective-config-bench.sh, tools/coverage_gap.py).
In short: on the SSH drop-in scenario, Pavois FAILs (caught, standard-mapped), OpenSCAP/SSG passes (a file-probed false negative), Lynis warns (it also reads sshd -T, but heuristically). The edge is narrow and honest: it is about drop-in / Include precedence, not runtime reads in general (for sysctl, SSG reads /proc/sys too, so both catch it).
Method
A fresh Debian 12 VM is provisioned, the scanners + CINC are installed once, a clean baseline snapshot is taken, and every run restores that identical snapshot first so the starting state never drifts.
tools/benchmark/effective-config-bench.sh <user@host> <key> <ssg-debian12-ds.xml>sets up the scenario and runs all three scanners.test-vms/bench.shis the lab wrapper (provision → snapshot → restore → run); it is not published because it references the local lab.
| Component | Version |
|---|---|
| Pavois | CINC Auditor 7.1.7 |
| OpenSCAP | oscap 1.3.7 |
| SCAP Security Guide | ssg-debian12-ds.xml 0.1.81 (ComplianceAsCode) |
| SSG datastream SHA-256 | ef4cb97e2566ced5146cbc6861259b79be69999759244e7fa4cbc7eafdbefbfc |
| Lynis | 3.0.8 (462 tests) |
| Target | Debian 12, fresh Incus VM |
| Run date | 2026-06 |
Scenario 1: sshd drop-in overrides PermitRootLogin
The main /etc/ssh/sshd_config says PermitRootLogin no; a drop-in in /etc/ssh/sshd_config.d/ says yes (the value sshd -T resolves).
grep -iE '^\s*PermitRootLogin' /etc/ssh/sshd_config # -> PermitRootLogin no (what a file probe reads)
sudo sshd -T | grep -i '^permitrootlogin' # -> permitrootlogin yes (the effective value)| Scanner | Reads | Result |
|---|---|---|
| Pavois | sshd -T |
FAIL, caught and mapped ssh-disable-root-login (CIS / BP-028 / STIG) |
| OpenSCAP 1.3.7 + SSG | OVAL on the file | PASS, a false negative: it reads the main file's no and misses the drop-in |
| Lynis 3.0.8 | sshd -T |
warns (SSH-7408 suggestion), caught but heuristic, not standard-mapped |
Lynis resolves the drop-in like Pavois; its limit is being heuristic, not this blind spot. OpenSCAP/SSG's file-probed rule misses it outright. Pavois resolves it and maps it to every applicable standard it carries.
Scenario 2: sysctl runtime vs file (no edge, kept for honesty)
We tested a second case to bound the claim: kernel.kptr_restrict = 2 on disk (/etc/sysctl.d/) but sysctl -w kernel.kptr_restrict=0 at runtime.
| Scanner | Result | Why |
|---|---|---|
| Pavois | FAIL | reads the runtime value |
| OpenSCAP + SSG | fail | SSG's sysctl OVAL also reads /proc/sys (the runtime) |
This is not a Pavois differentiator: both catch it. So the effective-config edge is specific to drop-in / Include file precedence (the SSH case), not runtime reads in general. We state this rather than dress up a non-difference as a win.
Coverage gap (auditable)
tools/coverage_gap.py maps Pavois controls to SSG rules (via ssg: tags) and triages the gap with oscap's own verdicts on the real target, so N/A rules never inflate it.
- raw gap: 315 SSG rules Pavois does not map
- triaged: 87 applicable & failing (the real backlog), 74 applicable & already passing, 139 notapplicable on Debian (e.g. SELinux under AppArmor), 15 not auto-checked
- symmetric: Pavois applies to 523 of its 607 Debian 12 controls on a fresh host (84 N/A)
Full per-rule data, a workflow-ready backlog (severity, oscap verdict, derived triage, and empty decision/reason/issue_url/owner/status columns): coverage-gap-debian12.csv (315 rows).
oscap xccdf eval --results gap-res.xml ssg-debian12-ds.xml # on the target, for applicability
python3 tools/coverage_gap.py --os debian12 \
--datastream ssg-debian12-ds.xml --oscap-results gap-res.xml --format csv > coverage-gap-debian12.csvThe high-severity gap entries are mostly arbitration cases on Debian (SELinux is N/A under AppArmor; ntp/timesyncd are alternatives to chrony), not clear missing controls. --create-issues files the applicable-failing gap as backlog, and broken controls become bug issues.