Hardening logging with journald and rsyslog

The threat: logs are attacked precisely because they hold the proof

Logs are where an intrusion leaves its footprints, and that is exactly why they are attacked. An adversary who lands on a host quickly turns to anti-forensics: wiping /var/log/journal, truncating syslog files, or simply triggering a reboot when the journal lives only in volatile memory. Worse, the danger is often silence: with default Storage=auto, journals can sit in /run and vanish at the next boot; an uncapped journal can fill the log partition, after which journald drops new records and the evidence of the attack in progress is never written. If logs are stored only locally, the same root access that let the attacker in lets them edit or delete the proof. Without trustworthy logs, an incident becomes unprovable.

Why harden it

Logging is the one control whose value is realised after the breach. Every other defense aims to stop the attacker; logging assumes one eventually gets through and asks: can we reconstruct the timeline? Hardening the pipeline guarantees the answer is yes: records survive reboots, the partition does not silently fill, tampering is detectable, and a copy lives somewhere the local attacker cannot reach.

Defense principles applied

• Persistence: Storage=persistent writes journals to /var/log/journal so they outlive crashes, power loss and reboots, instead of evaporating from /run.
• Availability of the log channel: Compress=yes plus retention caps (SystemMaxUse=, SystemKeepFree=, MaxRetentionSec=) stretch history within the disk budget and stop a full partition silently dropping records. RateLimitIntervalSec/RateLimitBurst blunt a log-flood DoS, but set them generously so real security events are never dropped.
• Single, deliberate pipeline: ForwardToSyslog=no avoids blind double-writing; centralized shipping is configured explicitly (an rsyslog @@remote rule or systemd-journal-upload).
• Integrity off-box: forward to a remote collector / SIEM so logs exist beyond the reach of local root.
• Accountability: pull the kernel audit trail into the journal with Audit=yes, and keep a complete time-ordered record.

How do you make logs tamper-evident? Forward Secure Sealing

A local attacker with root can edit journal files. Forward Secure Sealing (FSS) makes that detectable: journald periodically seals the journal with an evolving key, so any retroactive edit breaks the seal. Generate the key pair once, enable sealing, and verify on demand:

journalctl --setup-keys          # generate the sealing key + a verification key (store the latter off-box)
# set Seal=yes in journald.conf, then:
systemctl restart systemd-journald
journalctl --verify              # PASS, or reports the first tampered/garbled entry

FSS detects tampering; it does not prevent it. Pair it with off-box shipping so you also retain an untampered copy.

What Pavois audits: the effective journald configuration

Pavois audits the effective journald configuration the way the daemon resolves it. Because journald merges its drop-in directory /etc/systemd/journald.conf.d/ over the base /etc/systemd/journald.conf, Pavois greps both, so a base file overridden by a drop-in is read as the value that actually applies, where a single-file scan (OVAL/oscap) would report the wrong state. Its Logging (journald) rules check Storage=persistent (journald-storage), Compress=yes (journald-compress), and ForwardToSyslog=no (journald-forwardtosyslog). These map to the CIS journald benchmarks and ANSSI BP-028 R71, and pavois harden apply can write a single drop-in (/etc/systemd/journald.conf.d/99-Pavois.conf) satisfying all three, followed by systemctl restart systemd-journald.

Verify

journalctl --disk-usage                       # how much the journal occupies vs the cap
journalctl --list-boots                       # persistent history across reboots
journalctl --verify                           # FSS integrity check
systemd-analyze cat-config systemd/journald.conf  # the merged effective config
journalctl -k -b -1                           # last boot's kernel log, proving persistence

Pitfalls

• Storage=auto is not persistence: it only persists if /var/log/journal already exists. Use Storage=persistent (or create the directory).
• Aggressive RateLimitBurst drops security events: a brute-force or scan can be silenced by your own rate limit. Tune it high for security-relevant units.
• FSS detects, never prevents: it tells you logs were altered after the fact; only off-box shipping preserves an intact copy.
• A reboot rotates but does not delete: don't confuse --list-boots separation with log loss.

FAQ

How do I prove the logs weren't altered? Enable Forward Secure Sealing (journalctl --setup-keys, Seal=yes) and run journalctl --verify; a broken seal pinpoints the first tampered entry.

Why ForwardToSyslog=no if I want central logging? Because blind forwarding double-writes everything; configure shipping explicitly (rsyslog @@remote or systemd-journal-upload) so there is one authoritative, intentional pipeline.

Why does Pavois read both journald.conf and the drop-in dir? Because the daemon merges them; a drop-in can override the base, so only the merged result is the truth.