Read, keep and use system logs

Dit deel is nog niet vertaald. Je ziet de Engelse tekst.

$ als gewone gebruiker# als root (of met sudo)

Vaardigheden in dit hoofdstuk

Het journal lezen en logs bewaren na een herstart; de tijd juist houden

Know the two log systems

systemd-journald collects all messages first (kernel, services, boot). rsyslog reads them from the journal and writes classic text files.

	journald	rsyslog
Storage	Binary journal in `/run/log/journal` (lost at reboot) or `/var/log/journal` (kept)	Text files in `/var/log/`
Read with	`journalctl`	`less`, `grep`, `tail`
Strong at	Filters by unit, time, priority, boot; very fast	Simple files, sending logs to a central server

File	What is in it
`/var/log/messages`	Most system messages
`/var/log/secure`	Logins, `sudo`, SSH — security events
`/var/log/cron`	Cron jobs
`/var/log/maillog`	The mail server
`/var/log/boot.log`	Messages of the boot
`/var/log/audit/audit.log`	The audit system, also SELinux denials (written by `auditd`)

5.2

Read a log line

A classic log line has 4 parts: time, host, program[PID], message.

/var/log/secure

Oct 27 12:44:01 server1 sshd[6731]: Failed password for root from 203.0.113.50 port 51122 ssh2
│               │       │           └─ the message
│               │       └─ the program and its PID
│               └─ the host name
└─ the time stamp

#tail -n 20 /var/log/secure
#tail -f /var/log/messages
#grep -i 'error' /var/log/messages | tail

5.3

Search the journal

journalctl shows the whole journal. Filters make it short: by unit, by time, by priority, by boot.

#journalctl
#journalctl -e
#journalctl -n 30
#journalctl -f
#journalctl -u sshd
#journalctl -u httpd --since "1 hour ago"
#journalctl --since "2026-10-27 08:00" --until "2026-10-27 09:00"
#journalctl -p err
#journalctl -p warning -b
#journalctl -b -1
#journalctl -k
#journalctl _PID=1022
#journalctl -o verbose -n 1
$journalctl --user -u report.service

Option	What it does
`-n N` / `-e`	The last N lines / jump to the end
`-f`	Follow new lines live
`-u UNIT`	Only one service
`--since` / `--until`	A time range (`today`, `yesterday`, `"10 min ago"`, a date)
`-p PRIO`	This priority and worse (`emerg alert crit err warning notice info debug`)
`-b` / `-b -1`	This boot / the boot before
`-k`	Only kernel messages (like `dmesg`)
`-o verbose` / `-o json-pretty`	All fields of every entry
`--no-pager`	Print without `less` (for scripts and pipes)

Onthoud

Normal users see their own messages. Members of the group wheel or systemd-journal (and root) see everything.

5.4

Keep the journal after a reboot

When /var/log/journal does not exist, the journal lives in memory (/run/log/journal) and is gone after a reboot. Make it persistent in a drop-in file.

#mkdir -p /etc/systemd/journald.conf.d
#cat > /etc/systemd/journald.conf.d/persistent.conf << 'EOF'
[Journal]
Storage=persistent
SystemMaxUse=500M
EOF
#systemctl restart systemd-journald
#journalctl --list-boots
IDX BOOT ID                          FIRST ENTRY                 LAST ENTRY
 -1 3f9c1e2d4b5a46c7a8e9f0a1b2c3d4e5 Sun 2026-10-26 07:58:01 +03 Mon 2026-10-27 07:59:40 +03
  0 9a8b7c6d5e4f43a2b1c0d9e8f7a6b5c4 Mon 2026-10-27 08:00:55 +03 Mon 2026-10-27 12:50:12 +03
#journalctl --disk-usage
#journalctl --vacuum-time=2weeks

`Storage=`	Effect
`persistent`	Always on disk in `/var/log/journal` (created if needed)
`volatile`	Only in memory
`auto`	On disk only when `/var/log/journal` exists

Examentip

After a reboot, journalctl -b -1 must show the last boot. If it says "no persistent journal", the setting did not work. More: Keep the journal after a reboot.

5.5

Write your own log messages

logger writes a message into the log — good for scripts and for marking the start of a test.

$logger "Backup started"
$logger -t backup -p local0.notice "Backup of /srv finished in 42 s"
$journalctl -t backup -n 5
#grep backup /var/log/messages
$logger -p user.err "Disk check FAILED on /dev/vdb1"
$journalctl -p err -n 3

To send one kind of messages to its own file, add an rsyslog rule:

/etc/rsyslog.d/backup.conf

local0.*    /var/log/backup.log

#systemctl restart rsyslog
$logger -p local0.info "test line"
#tail -n 1 /var/log/backup.log

5.6

Rotate the log files

logrotate renames old logs, compresses them and deletes the oldest, so /var/log does not grow forever. A timer runs it every day.

/etc/logrotate.d/backup

/var/log/backup.log {
    weekly
    rotate 8
    compress
    missingok
    notifempty
}

$cat /etc/logrotate.conf
#logrotate -d /etc/logrotate.d/backup
#logrotate -f /etc/logrotate.d/backup
$ls /var/log/messages*
/var/log/messages  /var/log/messages-20261005  /var/log/messages-20261012

-d = dry run (shows what would happen), -f = force a rotation now.

5.7

Keep the system time right

Logs from many servers can only be compared when all clocks are right. timedatectl shows and sets the time and the time zone; chronyd keeps the clock in sync with NTP servers.

$timedatectl
               Local time: Mon 2026-10-27 12:55:10 +03
           Universal time: Mon 2026-10-27 09:55:10 UTC
                 RTC time: Mon 2026-10-27 09:55:10
                Time zone: Asia/Riyadh (+03, +0300)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no
$timedatectl list-timezones | grep -i riyadh
#timedatectl set-timezone Asia/Riyadh
#timedatectl set-ntp true

5.8

Sync the time with chrony

Tell chrony which NTP servers to use in /etc/chrony.conf, restart it, then check the sources.

/etc/chrony.conf

# Use the company time server, and a public pool as backup
server ntp.example.com iburst
pool 2.rhel.pool.ntp.org iburst
driftfile /var/lib/chrony/drift
makestep 1.0 3
rtcsync

#systemctl restart chronyd
$chronyc sources -v
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* ntp.example.com               2   6   377    35   +120us[ +150us] +/-   12ms
^- 162.159.200.1                 3   6   377    34  -1201us[-1201us] +/-   25ms
$chronyc tracking
#chronyc makestep