Pavois vs OpenSCAP, Lynis and CIS-CAT: Linux effective-configuration benchmark

At a glance

None of these tools is bad, OpenSCAP/SSG in particular is a mature, standard, broadly-supported stack, and Pavois even cross-validates its mappings against SSG. The point is narrower and only about the service-resolved controls: when a drop-in overrides the main file, an OVAL probe pointed at the file misses it unless it was specifically written for the resolved path, so in practice the file-based tools usually miss that case, where an effective-config read catches it. For inventory, package or filesystem controls the gap mostly closes.

Which tool, when?

Short answer, by goal:

• Use Pavois when you need the effective configuration audited (drop-ins, Includes, runtime overrides resolved), one neutral control mapped to several standards at once, an A:E graded report, and opt-in harden-as-code with a signed evidence bundle.
• Reach for OpenSCAP / SSG when you need a mature, official SCAP pipeline with broadly-supported XCCDF/ARF and vendor-shipped datastreams, especially where file-and-OVAL coverage is enough.
• Reach for Lynis when you want a fast, zero-setup host hardening sweep and a quick heuristic index, with no standard mapping required.
• Reach for CIS-CAT when you need an official CIS attestation (CIS SecureSuite) and the certified benchmark number is what your auditor demands.

They compose: Pavois even cross-validates its mappings against SSG. Pavois's edge is narrow and specific, service-resolved controls; for inventory, package and filesystem controls the tools largely converge.

By evidence type

A tool-vs-tool table flatters whoever wrote it. The honest cut is by what kind of evidence a control needs, read down a tool's column, not across one row:

Pavois leads on the two resolved-state rows, that is the whole reason it exists, and is on par (inventory, filesystem) or genuinely behind (official-standard authority, remediation maturity, ecosystem) on the rest. Pick the tool by the evidence your control actually needs; for a regulated CIS attestation you may still want CIS-CAT, for a broad SCAP pipeline OpenSCAP. They compose.

A reproducible benchmark

We ran the canonical drop-in case on a fresh Debian 12 VM (provision → snapshot → restore, identical every run): the main sshd_config says PermitRootLogin no, a drop-in in sshd_config.d/ says yes, the value that actually applies. Tools as shipped on Debian 12:

The gap is real and narrow, and we state it honestly: Lynis resolves the drop-in like Pavois (it runs sshd -T), but only as a non-normative hint; OpenSCAP/SSG's file-probed rule misses it outright; Pavois resolves it and maps it to every applicable standard it carries. Reproduce it with tools/benchmark/effective-config-bench.sh <user@host> <key> <ssg-debian12-ds.xml>, re-run from an identical snapshot each time.

Breadth matters too, so we also count the rules each tool implements by severity (this is not a winner metric: more rules is not better, and scoping differs). For Debian 12:

SSG carries more raw rules and is more mature; Pavois adds a critical tier SSG's model lacks, every control declares its evidence type and carries every applicable normative mapping, and Lynis has no per-rule severity at all. The drop-in case is about the kind of evidence; this table shows Pavois is comprehensive, not a single trick.

The comparison ships as a tool: tools/coverage_gap.py maps Pavois controls to SSG rules (via the ssg: tags) and triages the gap with oscap's own verdicts on a real target. The raw 315 SSG rules Pavois does not map collapse to 87 applicable and failing once N/A is removed (139 are notapplicable on Debian, e.g. SELinux; 15 are not auto-checked; 74 already pass). Symmetrically, Pavois applies to 523 of its 607 Debian 12 controls on a fresh host (84 N/A). Raw catalogue counts overstate on both sides; the honest, applicable gap is 87, and --create-issues files it as backlog (broken controls become bug issues). Full method, exact commands, tool versions and the per-rule CSV: docs/benchmark.

OpenSCAP (SCAP / OVAL)

OpenSCAP is the reference implementation of SCAP, the NIST-standardised format tying XCCDF (benchmark & rules), OVAL (the low-level checks) and CPE (platform ID). It is genuinely strong: the SCAP Security Guide ships SCAP datastreams (XCCDF/OVAL) mapping rules to CIS, ANSSI BP-028, DISA STIG, PCI-DSS and NIST 800-53, it emits bash and Ansible remediation, and its HTML report is the layout Pavois's renderer aims to equal.

The structural limit is how OVAL checks work: most inspect files on disk (textfilecontent54 greps a config file). An OVAL check pointed at sshd_config never sees a drop-in in sshd_config.d/, it reads the default and prints PASS while the drop-in re-enabled PermitRootLogin yes. Pavois keeps SCAP's chaptered, multi-norm report but swaps the file-reading engine for an effective-config one. It never runs oscap as an engine, only as a visual reference for the report layout.

Lynis

Lynis is a single shell script, no agent, no dependencies, that audits 60+ subsystems (462 tests in v3.0.8, the version Debian 12 ships; 3.1.x is current) in minutes and emits suggestions, warnings and a headline hardening index (0:100). As a fast first-pass triage and an inspiration source it is excellent: it tells you where to look.

But it is heuristic, not normative. Its checks encode good-practice opinions; they are not mapped to a published standard, and the hardening index is a Lynis-specific number, not a compliance verdict an auditor accepts. For SSH it actually runs sshd -T (like Pavois), so it resolves drop-ins there; it only greps the raw file for a few checks (e.g. AllowUsers) and for other subsystems. Its decisive limit is being heuristic, not a drop-in blind spot: use Lynis to spot the obvious and motivate quick wins, use Pavois for the rigorous, standards-mapped, auditable pass. Fabricating a lynis_X reference in place of a real CIS or BP-028 control would be a mistake, Pavois never does that.

CIS-CAT

CIS-CAT is the official CIS Configuration Assessment Tool, a Java scanner that grades a host against a CIS Benchmark. The catch is the membership model: CIS-CAT Lite is free but covers only a handful of benchmarks (Windows 10, Chrome, Ubuntu); the full CIS-CAT Pro Assessor requires paid CIS SecureSuite membership. So the one official tool for self-assessing against a publicly published standard sits behind a paywall: it is not an open rule engine, and the full assessor requires CIS SecureSuite membership.

Pavois is not CIS-CAT and ships no CIS content, it carries its own open rule base where CIS recommendation numbers are tags on neutral, stable IDs (473 controls carry an explicit cis: mapping). Select CIS in the report, pick Level 1 or 2, and Pavois recomposes a CIS-shaped view, openly, against the effective state, with no membership tier.

Remediation tools

Scanning finds the gap; remediation closes it. Here the landscape differs again, and Pavois's angle is that the same corpus that audits also fixes, in one loop.

ansible-lockdown (the MindPoint Group CIS/STIG roles, distinct from the dev-sec.io hardening collections) is mature and widely used, Pavois even cross-validates its mappings against it. But it applies files and runs as a playbook decoupled from the audit, so nothing proves the running daemon ended up compliant. OpenSCAP emits bash/Ansible straight from the XCCDF benchmark, file-oriented, and you trust the benchmark's fix.

Pavois closes the loop instead: pavois harden plan reads the live state, you flip apply per rule, a native Chef run converges, and pavois diff re-scans the effective config to prove every gap is closed and nothing regressed. A fix you can't verify isn't evidence.

Where Pavois is not the right tool

Pavois is strong on resolved-state controls and remediation-as-code, but it is not the answer for everything, and saying so is the point:

• Official CIS self-assessment for a certification: CIS-CAT stays the authority; Pavois's cis: tags are cross-references, not certified claims.
• SCAP-native pipelines (XCCDF/OVAL/ARF, broad enterprise tooling): OpenSCAP/SSG is more mature.
• Lightweight first-pass triage: Lynis is faster and simpler.
• Mature remediation catalogues: SSG and ansible-lockdown are older and broader.
• And harden can be more dangerous: it applies real changes, some risky (a kernel.modules_disabled=1 once bricked an EFI boot), so risky fixes are delivered opt-in, not auto-run, and a reboot campaign proves the result.

Pick the tool by the evidence a control needs and the job at hand; they compose.

The engine: CINC Auditor / InSpec

Pavois is 100% CINC Auditor, the free, trademark-free build of Chef InSpec. A control describes the expected state of a resource in readable language; CINC runs it against the effective configuration. It runs as a native binary so a host scan (local://) and an SSH scan (ssh://) use your own ~/.ssh/config, routing and user, the container is only a zero-install fallback. Effective-config commands need privilege, so Pavois passes --sudo to CINC, over stdin, never the command line.