A service started by its timer left nothing in journalctl -u, while a manual start logged normally. The timer was innocent: output goes to the journal the same way for both. The logs were written and then vacuumed, because journald was running volatile with a 16 MB ceiling — roughly two hours of history.
Diagnostic order
Check that the timer really ran, and what it ran:
| |
systemctl status shows the last execution only, not the history. For per-run identity:
| |
Look for _SYSTEMD_UNIT, _SYSTEMD_USER_UNIT, INVOCATION_ID, MESSAGE.
Then ask journald whether it kept anything:
| |
--list-boots reports the first and last entry across the whole journal, so it does not prove every unit’s output survived that span.
What the settings mean
| Setting | Effect |
|---|---|
Storage=volatile | journal lives in /run/log/journal (RAM); gone on reboot |
Storage=persistent | journal lives in /var/log/journal, survives reboot |
RuntimeMaxUse= | size cap for the volatile journal |
SystemMaxUse= | size cap for the persistent journal |
SystemMaxFileSize=, RuntimeMaxFileSize= | size of one journal file before rotation |
MaxRetentionSec= | age limit; defaults to unlimited |
The configuration here was:
| |
Journald vacuums archived files when it needs space, not by age. With a 16 MB cap and 2 MB files, rotation lands in the 10–15 minute range, so a full day of history never fits:
| |
First entry of the day was Oct 31 13:19:30 while the machine had been up since August 23 — consistent with a rolling vacuum, not a reboot. journalctl -u systemd-journald returning -- No entries -- is normal; journald does not log its routine work there.
Two journal directories under /run/log/journal mean two machine IDs. Only the one matching cat /etc/machine-id is current; a stale one is left over from another install or image:
| |
The old f37394… directory held a single 8 MB system.journal from August 23. Its history is readable by pointing journalctl --directory= at it, and it can be deleted once that is settled.
Move to persistent storage with 31-day retention
| |
/etc/systemd/journald.conf.d/10-persistent.conf:
| |
| |
No monthly timer is needed; MaxRetentionSec=31day is journald’s own age-based retention. A manual sweep is still available:
| |
--rotate archives the active file so the vacuum can act on it. Delete the stale directory after confirming it holds nothing wanted:
| |
⚠️
SystemMaxUsewins over the age target: logs get dropped early if they hit 500 MB before 31 days. Size the cap for the actual log volume if the full window matters. Nothing already vacuumed comes back — this only preserves entries written after the change.