Find text with regular expressions

Dit deel is nog niet vertaald. Je ziet de Engelse tekst.

$ als gewone gebruiker# als root (of met sudo)

Vaardigheden in dit hoofdstuk

Tekst vinden met reguliere expressies en grep

Know what a regex is

A regex is a search pattern. Normal letters match themselves; special characters (metacharacters) have a meaning.

$grep root /etc/passwd
$grep '^root' /etc/passwd
$grep 'bash$' /etc/passwd

Onthoud

Always put a regex in single quotes. Then the shell does not change *, $ or ? before grep sees them.

Let op

A regex is not a shell wildcard. In a regex * means "the character before, 0 or more times", not "any text". "Any text" is .*.

2.2

Anchor the match at the start or end

^ means the start of the line, $ the end. \< and \> mean the start and end of a word.

Pattern	Matches
`^#`	Lines that start with `#` (comments)
`^$`	Empty lines
`nologin$`	Lines that end with `nologin`
`^PermitRootLogin`	The setting at the start of a line (not a comment)
`\<cat\>`	The word `cat`, not `catalog`

$grep '^PermitRootLogin' /etc/ssh/sshd_config /etc/ssh/sshd_config.d/*.conf
$grep -c '^$' /etc/sudoers
$grep 'nologin$' /etc/passwd | wc -l
$grep -w 'cat' notes.txt

2.3

Match any character and repeats

. is one character of any kind. A quantifier after a character says how many times it may repeat.

Pattern	Meaning
`.`	Any one character
`*`	The part before: 0 or more times
`.*`	Any text, also empty
`\+` / `+` (with `-E`)	1 or more times
`\?` / `?` (with `-E`)	0 or 1 time
`{3}` / `{2,4}` (with `-E`)	Exactly 3 / 2 to 4 times

$grep 'c.t' words.txt
$grep 'Failed.*from' /var/log/secure
$grep -E 'colou?r' text.txt
$grep -E '[0-9]{3}-[0-9]{4}' contacts.txt

2.4

Use character classes

[…] matches one character from a set. [^…] matches one character not in the set. Named classes like [[:digit:]] work in every language setting.

Pattern	Matches one …
`[abc]`	a, b or c
`[0-9]` / `[[:digit:]]`	digit
`[a-z]` / `[[:lower:]]`	small letter
`[A-Za-z]` / `[[:alpha:]]`	letter
`[[:space:]]`	space or tab
`[^#]`	character that is not `#`

$grep '^[^#]' /etc/chrony.conf
$grep -E '^[[:space:]]*$' file.conf
$grep -E '^user[0-9]+:' /etc/passwd

2.5

Use extended regex with grep -E

With grep -E (extended regex) you write + ? { } | ( ) without backslashes. | means "or", ( ) makes a group.

$grep -E 'error|fail|denied' /var/log/messages
$grep -iE '^(PermitRootLogin|PasswordAuthentication)' /etc/ssh/sshd_config
$grep -E '(ab)+' file.txt
$grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}' /var/log/secure | sort -u

Basic (`grep`)	Extended (`grep -E`)
`a\+`	`a+`
`a\?`	`a?`
`a\{2,3\}`	`a{2,3}`
`$ab$`	`(ab)`
`a\|b`	`a\|b`

Onthoud

A dot in an IP address or a file name is a metacharacter. To match a real dot write \. — '192\.168\.', '\.conf$'.

2.6

Use the most useful grep options

grep options decide what is printed: the lines, their number, the file names, or the lines around the match.

$grep -i 'error' /var/log/messages
$grep -v '^#' /etc/fstab
$grep -n 'Listen' /etc/httpd/conf/httpd.conf
$grep -c 'Accepted' /var/log/secure
#grep -r 'PermitRootLogin' /etc/ssh/
#grep -rl 'example.com' /etc/
$grep -A 3 -B 1 'OOM' /var/log/messages
$grep -o 'uid=[0-9]*' /var/log/audit/audit.log | sort | uniq -c
$grep -q 'wheel' /etc/group && echo "group exists"

Option	What it does
`-i`	Ignore big/small letters
`-v`	Print the lines that do not match
`-n`	Show line numbers
`-c`	Only count the matching lines
`-r` / `-R`	Search in all files of a folder
`-l` / `-L`	Only the names of files with / without a match
`-w`	Match whole words only
`-o`	Print only the matching part
`-A N` / `-B N` / `-C N`	N lines after / before / around
`-q`	Print nothing — only the exit code (for scripts)
`-E` / `-F`	Extended regex / plain text, no regex

2.7

Clean up config files and logs

Most daily regex jobs are short: show the real settings of a file, count errors, pull out addresses.

$grep -Ev '^[[:space:]]*(#|$)' /etc/ssh/sshd_config
#grep -E 'Failed password' /var/log/secure | grep -oE 'from [0-9.]+' | sort | uniq -c | sort -rn | head
$journalctl -b --no-pager | grep -Ei 'error|fail' | tail -n 20
$ip -o -4 a | grep -oE 'inet [0-9./]+'
$grep -E '^[^:]+:[^:]*:0:' /etc/passwd

The first line is the most useful one in admin life: it prints a config file without comments and empty lines.

2.8

Use regex in vim, sed and awk

The same patterns work in many tools: search in vim, replace with sed, pick lines in awk.

$sed -n '/^Listen/p' /etc/httpd/conf/httpd.conf
#sed -i.bak -E 's/^#?PermitRootLogin.*/PermitRootLogin no/' /root/sshd_config.test
$sed -E 's/[0-9]{1,3}(\.[0-9]{1,3}){3}/x.x.x.x/g' access.log > access-anon.log
$awk -F: '$3 >= 1000 && $7 ~ /bash$/ {print $1}' /etc/passwd
$awk '/Failed password/ {print $(NF-3)}' /var/log/secure | sort -u

In vim: /^Port searches, :%s/^#$Port$/\1/ removes the # before Port.

Let op

sed -i changes the file at once. Use sed -i.bak to keep a copy, or run without -i first to see the result.

Oefenen

Lab 4.1Get answers from text files and logsGemiddeld · 30 minNog niet begonnen

Oefenen

Alle commando's in dit hoofdstuk