Read, search and change text files

$ run as a normal user# run as root (or with sudo)

Exam objectives in this chapter

Use grep and regular expressions to read text
Create and edit text files

Read long files with less

less shows a file one screen at a time. You can scroll up and down and search, and it never changes the file.

$less /var/log/messages
$less -N /etc/ssh/sshd_config
$journalctl -u sshd | less

Key	What it does
`Space` / `b`	One page down / one page up
`↓` `↑` or `j` `k`	One line down / up
`g` / `G`	First line / last line
`/text`	Search forward; `n` = next match, `N` = previous
`?text`	Search backward
`F`	Follow the file like `tail -f` (Ctrl+C to stop)
`q`	Quit

Remember

less = read, never write. / searches, q quits.

Exam tip

man uses less to show pages, so the same keys work there.

4.2

Show whole files with cat

cat prints the whole file at once. Good for short files; for long ones use less.

$cat /etc/hostname
$cat -n /etc/fstab
$cat -A script.sh
$tac /var/log/dnf.log | head

-n numbers the lines.
-A shows hidden characters: tabs as ^I, line ends as $, Windows ends as ^M$.
tac is cat backwards: the last line first.

At work

cat -A finds the invisible characters that break YAML files and shell scripts.

4.3

Show the start or the end of a file

head shows the first lines, tail the last lines. tail -f keeps showing new lines while a log grows.

$head /etc/passwd
$head -n 3 /etc/passwd
$tail -n 20 /var/log/secure
#tail -f /var/log/messages
$head -n 10 /etc/passwd | tail -n 1

The last line prints only line 10: the first 10 lines, then the last of them.

Remember

head = top, tail = bottom, tail -f = watch live.

4.4

Take columns with cut

cut takes fields (columns) from each line. You give the separator with -d and the field numbers with -f.

$cut -d: -f1 /etc/passwd
$cut -d: -f1,7 /etc/passwd
$cut -c1-8 /etc/passwd

In /etc/passwd the fields are separated by :. Field 1 is the user name, field 7 the shell.

Watch out

cut uses exactly one separator character. For columns separated by several spaces, use awk.

4.5

Sort lines and remove doubles

sort puts lines in order; uniq removes repeated lines that are next to each other. Together they make quick reports.

$sort /etc/passwd
$sort -t: -k3 -n /etc/passwd
$du -sh /var/* 2>/dev/null | sort -h
$cut -d: -f7 /etc/passwd | sort | uniq -c | sort -rn

Option	Meaning
`-n`	Sort as numbers (10 after 9)
`-h`	Sort human sizes (2K, 1M, 3G)
`-r`	Reverse order
`-k3`	Sort on field 3
`-t:`	Fields are separated by `:`
`-u`	Only unique lines

The last example counts how many users use each shell: sort first, then uniq -c counts, then sort the counts.

Remember

sort | uniq -c | sort -rn = "top list" of anything.

4.6

Count lines, words and characters

wc counts. wc -l (lines) is the one you use most.

$wc /etc/passwd
  45   95 2512 /etc/passwd
$wc -l /etc/passwd
$grep -c bash /etc/passwd
$ls /etc | wc -l

The numbers are lines, words, bytes.

4.7

Understand regular expressions

A regular expression (regex) is a pattern that describes text. grep, sed, awk and vim all use them.

Pattern	Matches
`^text`	text at the start of a line
`text$`	text at the end of a line
`.`	any one character
`*`	the character before, 0 or more times
`+`	the character before, 1 or more times (extended)
`?`	the character before, 0 or 1 time (extended)
`{2,4}`	the character before, 2 to 4 times (extended)
`[abc]` / `[0-9]`	one of these characters / one digit
`[^abc]`	one character that is not a, b or c
`\.`	a real dot (the backslash escapes it)
`\b`	a word border
`(a\|b)`	a or b (extended)

$grep '^root' /etc/passwd
$grep 'nologin$' /etc/passwd
$grep -E '^[a-z]+[0-9]{2}$' users.txt
$grep -v '^#' /etc/ssh/sshd_config | grep -v '^$'

Basic regex (BRE) is the default; extended regex (ERE) with grep -E lets you use + ? {} | () without backslashes.

Watch out

Put a regex in single quotes, so the shell does not change *, $ or \ before grep sees them.

Remember

^ start, $ end, . any, * repeat, [] choice, \ escape.

4.8

Search text with grep

grep prints the lines that match a pattern. It is the most used search tool on Linux.

$grep student /etc/passwd
$grep -i error /var/log/messages
$grep -rn "PermitRootLogin" /etc/ssh/
$grep -v '^#' /etc/fstab
$grep -c "Failed password" /var/log/secure
$grep -E 'error|fail' /var/log/messages
$grep -A2 -B2 "Out of memory" /var/log/messages

Option	Meaning
`-i`	Ignore upper/lower case
`-v`	Show lines that do not match
`-r`	Search all files in folders
`-n` / `-c` / `-l`	Show line numbers / count / only file names
`-w`	Whole words only
`-E`	Extended regex
`-A N` `-B N` `-C N`	N lines after / before / around a match
`-o`	Print only the matching part

Exam tip

"Find all lines that contain X in file Y and write them to file Z" is a typical task: grep X Y > Z.

At work

grep -v '^#' FILE | grep -v '^$' shows the active settings of a config file without comments and empty lines.

4.9

Print and calculate columns with awk

awk reads each line, splits it into fields ($1, $2 …) and runs a small program on it. Good for columns separated by spaces.

$awk -F: '{print $1, $7}' /etc/passwd
$awk -F: '$3 >= 1000 {print $1}' /etc/passwd
$df -h | awk 'NR>1 {print $5, $6}'
$ps aux | awk '{sum += $6} END {print sum/1024 " MiB"}'
$awk '/Failed password/ {print $(NF-3)}' /var/log/secure | sort | uniq -c | sort -rn

-F: sets the field separator (default: spaces and tabs).
$0 is the whole line, NF the number of fields, NR the line number.
A pattern before { } selects lines: $3 >= 1000 = normal users.

The last example counts failed SSH logins per IP address.

4.10

Change text with sed

sed (stream editor) changes text as it flows by: replace words, delete lines, print some lines. With -i it changes the file itself.

$sed -n '5,10p' /etc/passwd
$sed 's/enforcing/permissive/' /etc/selinux/config
#sed -i 's/^#PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config
$sed '/^#/d; /^$/d' /etc/chrony.conf
#sed -i.bak 's/8080/9090/g' /etc/myapp.conf

Command	Meaning
`s/old/new/`	Replace the first "old" in each line
`s/old/new/g`	Replace every "old"
`/pattern/d`	Delete lines that match
`-n '5p'`	Print only line 5
`-i`	Change the file itself (`-i.bak` keeps a backup)

Watch out

Test without -i first; look at the output; then add -i (or -i.bak).

4.11

Create text files quickly

For short files you do not need an editor: echo writes one line, and cat with a here-document writes many lines.

$echo "Hello" > hello.txt
$cat > ~/notes.txt <<'EOF'
server1 192.168.122.10
server2 192.168.122.20
EOF
#cat >> /etc/hosts <<'EOF'
192.168.122.30 server3.example.com server3
EOF

<<'EOF' reads lines until a line with only EOF. With quotes around EOF, variables like $HOME stay as they are.

Exam tip

The objective "create and edit text files" can be done with vim, nano, echo or a here-document. Check the file with cat after.

4.12

Remember the main points

The most important facts of this chapter, in short.

less pages through a file (/word to search, q to quit); cat prints a whole file; head and tail show the start and end.
tail -f follows a file live, for example a log file.
cut takes columns, sort orders lines, uniq removes repeated lines, wc counts lines, words and bytes.
A regular expression describes text: ^ start, $ end, . any character, * repeat, [ ] a set.
grep finds lines: -i ignore case, -v invert, -r recursive, -E extended regex, -n line numbers.
awk works with fields (awk -F: '{print $1}'); sed changes text (sed -i 's/old/new/g' file).
Quote regular expressions with single quotes, so the shell does not change them.

4.13

Learn the key terms

The words of this chapter in simple English.

Term	Meaning
Pager	A program that shows text one screen at a time (`less`).
Regular expression	A pattern that describes text, used by grep, sed and awk.
Anchor	`^` (line start) or `$` (line end) in a regular expression.
Field	One column of a line, split by a separator like `:` or a space.
Delimiter	The character between fields (`-d` in cut, `-F` in awk).
Stream editor	`sed`: it changes text line by line as it passes through.
Extended regex	A richer pattern syntax (`grep -E`) with `+`, `?`, `{n}` and `\|`.
Here document	Text given to a command inside a script, between `<<EOF` and `EOF`.