Read, keep and manage system logs

$ run as a normal user# run as root (or with sudo)

Exam objectives in this chapter

Find and read system log files and journals
Keep the system journal after a reboot

Know the two log systems

systemd-journald collects all messages (kernel, services, boot) in a binary journal. rsyslogd reads them and writes text files in /var/log/.

	journald	rsyslogd
Storage	Binary journal (`/run/log/journal` or `/var/log/journal`)	Text files in `/var/log/`
Read with	`journalctl`	`less`, `tail`, `grep`
Strong at	Search by unit, time, priority, boot	Sending logs to other servers, long-term files

$systemctl status systemd-journald rsyslog
$ls /var/log/

13.2

Read the classic log files

The most important text logs on RHEL:

File	Contains
`/var/log/messages`	Most system messages
`/var/log/secure`	Logins, sudo, su, SSH (security)
`/var/log/audit/audit.log`	Audit and SELinux events (written by auditd)
`/var/log/boot.log`	Messages of services at boot
`/var/log/cron`	cron jobs
`/var/log/maillog`	Mail server
`/var/log/httpd/`	Apache access and error logs
`/var/log/dnf.log`	Package changes

#less /var/log/messages
#grep -i 'failed password' /var/log/secure | tail

13.3

Understand a log line

A syslog line has: time, host, program[PID], message.

/var/log/secure

Oct 25 09:02:11 server1 sshd[2215]: Accepted publickey for admin from 192.168.122.1 port 51322 ssh2: ED25519 SHA256:k9...

Part	Value
Time	`Oct 25 09:02:11`
Host	`server1`
Program [PID]	`sshd[2215]`
Message	`Accepted publickey for admin …`

13.4

Watch logs live

Follow a log while you test something — you see the error the moment it happens.

#tail -f /var/log/messages
#journalctl -f
#journalctl -fu httpd

At work

Open a second terminal with journalctl -f before you restart a service or change a config. Errors appear right away.

13.5

Search the journal with journalctl

journalctl filters by unit, time, priority, boot and more. Filters can be combined.

#journalctl -u sshd
#journalctl -p err
#journalctl --since today
#journalctl --since "2026-10-25 08:00" --until "2026-10-25 09:00"
#journalctl -b
#journalctl -b -1 -p warning
#journalctl -k
#journalctl _UID=1001
#journalctl -o verbose -n 1

Option	Shows
`-u UNIT`	One service
`-p err`	Priority err and worse (also `warning`, `crit` …)
`--since` / `--until`	A time window (`today`, `yesterday`, `-1h`, a date)
`-b` / `-b -1`	This boot / the boot before
`-k`	Kernel messages
`-n 20`	Last 20 lines
`-x`	Explanations for some messages
`-o verbose`	All fields of an entry
`FIELD=VALUE`	Filter on a field, like `_PID=1010`, `_COMM=sudo`

Remember

Type journalctl -u and press Tab twice: bash completes unit names. In -o verbose you see all fields you can filter on.

13.6

Keep the journal after a reboot

By default the journal may live only in memory (/run/log/journal) and is lost at reboot. With a folder /var/log/journal it is stored on disk.

Check where the journal is now.

$journalctl --list-boots
$ls -d /run/log/journal /var/log/journal

Make it persistent: create a drop-in folder and set Storage=persistent in a file there (or simply create /var/log/journal).
```
#mkdir -p /etc/systemd/journald.conf.d
```
/etc/systemd/journald.conf.d/persistent.conf
```
[Journal]
Storage=persistent
SystemMaxUse=500M
```

Restart journald and flush the memory journal to disk.

#systemctl restart systemd-journald
#journalctl --flush

Check it

More than one boot is listed after a reboot.

$journalctl --list-boots

Storage=	Meaning
`auto`	Disk if `/var/log/journal` exists, else memory (default)
`persistent`	Disk; creates the folder
`volatile`	Memory only
`none`	Keep nothing (only forward to rsyslog)

Exam tip

"The journal must survive a reboot" = Storage=persistent (or mkdir /var/log/journal) and a restart of systemd-journald. Check with journalctl --list-boots after a reboot.

13.7

Configure rsyslog

/etc/rsyslog.conf (and files in /etc/rsyslog.d/) decide which messages go where. A rule is: FACILITY.PRIORITY DESTINATION.

/etc/rsyslog.conf (part)

*.info;mail.none;authpriv.none;cron.none    /var/log/messages
authpriv.*                                  /var/log/secure
mail.*                                      -/var/log/maillog
cron.*                                      /var/log/cron
*.emerg                                     :omusrmsg:*

The config has three parts: modules (inputs like imjournal), global directives, and rules.

13.8

Know facilities and priorities

Every message has a facility (which part of the system) and a priority (how important).

Facilities
`auth`, `authpriv`	Login and security
`cron`, `mail`, `daemon`, `kern`	Cron, mail, services, kernel
`user`	User programs (default of logger)
`local0` … `local7`	Free for your own use

Priority (high → low)	Number
`emerg`	0
`alert`	1
`crit`	2
`err`	3
`warning`	4
`notice`	5
`info`	6
`debug`	7

cron.err means: cron messages with priority err or higher. cron.=err means only err. cron.none means none.

Send all messages of facility local5 to their own file.
/etc/rsyslog.d/app.conf
```
local5.*    /var/log/app.log
```

Restart rsyslog and test with logger.

#systemctl restart rsyslog
#logger -p local5.info "test message for app.log"
#tail -1 /var/log/app.log

At work

Send logs to a central log server so an attacker cannot delete them on the hacked machine: *.* @@logserver.example.com:514 (@@ = TCP, @ = UDP).

13.9

Rotate log files

logrotate renames old logs (like messages-20261025), compresses them and deletes the oldest, so the disk does not fill up. A systemd timer runs it every day.

$cat /etc/logrotate.conf
$ls /etc/logrotate.d/
$systemctl list-timers logrotate.timer

/etc/logrotate.d/app

/var/log/app.log {
    weekly
    rotate 8
    compress
    missingok
    notifempty
    create 0640 root root
}

#logrotate -d /etc/logrotate.d/app
#logrotate -f /etc/logrotate.d/app

Setting	Meaning
`daily` / `weekly` / `monthly` / `size 100M`	When to rotate
`rotate 8`	Keep 8 old files
`compress`	gzip old files
`missingok`, `notifempty`	No error if missing; skip empty files
`create MODE OWNER GROUP`	Make a new empty log with these rights
`postrotate … endscript`	Run commands after the rotation (like a reload)

13.10

Write test messages with logger

logger writes a message to the log from the command line or a script — good to test rules and to log what your scripts do.

$logger "backup started"
$logger -p user.err -t backup "backup failed: disk full"
#journalctl -t backup
#grep backup /var/log/messages

13.11

Remember the main points

The most important facts of this chapter, in short.

RHEL logs with systemd-journald (the journal) and rsyslogd (text files in /var/log).
Important files: /var/log/messages (general), /var/log/secure (logins and sudo), /var/log/audit/audit.log (SELinux and audit).
journalctl reads the journal: -u UNIT, -f follow, -p err priority, -b this boot, --since "1 hour ago", -o verbose.
By default the journal lives in /run and is lost at reboot. Make it persistent with Storage=persistent in a journald drop-in (or create /var/log/journal), then restart journald.
rsyslog sends messages by facility and priority to files: authpriv.* /var/log/secure; own rules go in /etc/rsyslog.d/*.conf.
logrotate rotates, compresses and removes old log files (/etc/logrotate.conf, /etc/logrotate.d/).
logger writes your own message into the logs, good for testing.

13.12

Learn the key terms

The words of this chapter in simple English.

Term	Meaning
journald	The systemd logging service that keeps the journal.
Journal	The binary log of systemd, read with journalctl.
rsyslogd	The service that writes log messages into text files.
Facility	The kind of program that sent the message (auth, cron, mail, local0 …).
Priority (severity)	How important a message is (debug … emerg).
Persistent journal	A journal that is kept on disk in /var/log/journal.
Log rotation	Moving old logs away and starting new files.
auditd	The service that writes the audit log (also SELinux denials).