A service started by its timer left nothing in journalctl -u, while a manual start logged normally. The timer was innocent: output goes to the journal the same way for both. The logs were written and then vacuumed, because journald was running volatile with a 16 MB ceiling — roughly two hours of history.

Diagnostic order Link to this section

Check that the timer really ran, and what it ran:

1
2
3
4
systemctl list-timers --all
systemctl status your-service.service
journalctl -u your-service.service --since "yesterday" --no-pager
journalctl -u your-service.timer --since "yesterday"

systemctl status shows the last execution only, not the history. For per-run identity:

1
journalctl -u your-service.service -o verbose --since today

Look for _SYSTEMD_UNIT, _SYSTEMD_USER_UNIT, INVOCATION_ID, MESSAGE.

Then ask journald whether it kept anything:

1
2
3
4
journalctl --disk-usage
journalctl --list-boots
systemd-analyze cat-config systemd/journald.conf
ls -ld /var/log/journal

--list-boots reports the first and last entry across the whole journal, so it does not prove every unit’s output survived that span.

What the settings mean Link to this section

SettingEffect
Storage=volatilejournal lives in /run/log/journal (RAM); gone on reboot
Storage=persistentjournal lives in /var/log/journal, survives reboot
RuntimeMaxUse=size cap for the volatile journal
SystemMaxUse=size cap for the persistent journal
SystemMaxFileSize=, RuntimeMaxFileSize=size of one journal file before rotation
MaxRetentionSec=age limit; defaults to unlimited

The configuration here was:

1
2
3
[Journal]
Storage=volatile
RuntimeMaxUse=16M

Journald vacuums archived files when it needs space, not by age. With a 16 MB cap and 2 MB files, rotation lands in the 10–15 minute range, so a full day of history never fits:

1
2
3
4
5
/run/log/journal/ebe5d352b5bc43d18d3cfcc13efa0ea9/:
total 15M
-rw-r-----+ 1 root systemd-journal 2.0M Oct 31 13:31 [email protected]
...
-rw-r-----+ 1 root systemd-journal 2.0M Oct 31 16:31 system.journal

First entry of the day was Oct 31 13:19:30 while the machine had been up since August 23 — consistent with a rolling vacuum, not a reboot. journalctl -u systemd-journald returning -- No entries -- is normal; journald does not log its routine work there.

Two journal directories under /run/log/journal mean two machine IDs. Only the one matching cat /etc/machine-id is current; a stale one is left over from another install or image:

1
2
3
cat /etc/machine-id
basename /run/log/journal/*/
journalctl --directory=/run/log/journal/f37394f4a6cf41a0a577d63be479ea22 --list-boots

The old f37394… directory held a single 8 MB system.journal from August 23. Its history is readable by pointing journalctl --directory= at it, and it can be deleted once that is settled.

Move to persistent storage with 31-day retention Link to this section

1
2
3
sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo mkdir -p /etc/systemd/journald.conf.d

/etc/systemd/journald.conf.d/10-persistent.conf:

1
2
3
4
5
6
[Journal]
Storage=persistent
MaxRetentionSec=31day
SystemMaxUse=500M
SystemMaxFileSize=32M
RuntimeMaxUse=100M
1
2
3
4
sudo systemctl restart systemd-journald
sudo journalctl --flush
journalctl --disk-usage
ls -lah /var/log/journal/

No monthly timer is needed; MaxRetentionSec=31day is journald’s own age-based retention. A manual sweep is still available:

1
sudo journalctl --rotate --vacuum-time=31d

--rotate archives the active file so the vacuum can act on it. Delete the stale directory after confirming it holds nothing wanted:

1
sudo rm -rf /run/log/journal/f37394f4a6cf41a0a577d63be479ea22

⚠️ SystemMaxUse wins over the age target: logs get dropped early if they hit 500 MB before 31 days. Size the cap for the actual log volume if the full window matters. Nothing already vacuumed comes back — this only preserves entries written after the change.

References Link to this section