Fix a system that does not boot

Dit deel is nog niet vertaald. Je ziet de Engelse tekst.

$ als gewone gebruiker# als root (of met sudo)

Examendoelen in dit hoofdstuk

Met de hand in een ander target opstarten
Het opstarten onderbreken om toegang tot een systeem te krijgen

Follow the boot procedure

The boot is a chain. When it breaks, the step tells you where to fix it.

Step	What happens	Typical problem → fix
1. Firmware (BIOS/UEFI)	Hardware test, finds a boot device	Wrong boot order → firmware setup
2. GRUB 2	Menu, loads kernel + initramfs, passes arguments	GRUB damaged → reinstall from rescue disk
3. Kernel	Starts, uses the initramfs drivers	Wrong arguments → edit in GRUB (`e`)
4. initramfs	systemd in RAM finds and mounts the root file system	Missing driver/LVM → recreate with dracut
5. switch root	Real root file system becomes `/`	Root FS damaged → fsck / xfs_repair
6. systemd	Starts the default target and its units	fstab error → emergency mode, fix fstab

#journalctl -b -p err
#systemd-analyze critical-chain

18.2

Pass kernel boot arguments

At the GRUB menu press e, go to the line that starts with linux, add arguments at the end, and press Ctrl+X to boot. The change is for this boot only.

Argument	Result
`systemd.unit=rescue.target`	Rescue mode: root login, file systems mounted, no network
`systemd.unit=emergency.target`	Emergency mode: root file system read-only, almost nothing started
`rd.break`	Stop in the initramfs, before the real root is used (root password reset)
`init=/bin/bash`	A root shell instead of systemd (last resort)
`systemd.debug-shell`	A root shell on tty9 during boot (for debugging hangs)
`rd.debug`, `systemd.log_level=debug`	Very detailed boot messages

Onthoud

Remove rhgb quiet from the line to see the boot messages while you test.

18.3

Start a troubleshooting target

Rescue and emergency mode give you a root shell with few services running. You need the root password.

Boot with systemd.unit=emergency.target (or rescue). Type the root password.
In emergency mode the root file system is read-only. Make it writable.
```
#mount -o remount,rw /
```
Fix the problem, then continue the normal boot or reboot.
```
#systemctl default
#systemctl reboot
```

Notitie

Emergency mode starts automatically when a file system in /etc/fstab cannot be mounted.

18.4

Use a rescue disk

When GRUB or the root password is broken and nothing else works, boot from the RHEL installation ISO and choose Troubleshooting → Rescue a Red Hat Enterprise Linux system. It finds your installation and mounts it on /mnt/sysroot.

#chroot /mnt/sysroot
#exit

After chroot the commands work on the installed system as if it booted normally.

18.5

Reinstall GRUB 2

When the boot loader is damaged ("no boot device", a GRUB shell), install it again from a rescue shell.

BIOS systems: write GRUB to the boot disk and rebuild the config.
```
#grub2-install /dev/vda
#grub2-mkconfig -o /boot/grub2/grub.cfg
```

UEFI systems: reinstall the packages that hold the EFI files.

#dnf reinstall -y grub2-efi-x64 shim-x64
#grub2-mkconfig -o /boot/grub2/grub.cfg

Let op

Do not run grub2-install on UEFI systems; it does not set up Secure Boot correctly. Reinstall the packages instead.

18.6

Fix the initramfs

The initramfs has the drivers and tools to find the root file system. When it is damaged or misses a driver, the boot stops in the initramfs ("dracut emergency shell"). Rebuild it with dracut.

#lsinitrd /boot/initramfs-$(uname -r).img | head
#dracut -f
#dracut -f /boot/initramfs-6.12.0-55.el10.x86_64.img 6.12.0-55.el10.x86_64

Notitie

From a rescue disk, uname -r shows the kernel of the ISO. Use the version from ls /lib/modules of the installed system.

18.7

Recover from file system problems

A wrong line in /etc/fstab or a damaged file system drops you into emergency mode. Fix the line, or repair the file system while it is not mounted.

Fix a bad fstab line.

#mount -o remount,rw /
#vim /etc/fstab
#systemctl daemon-reload
#mount -a
#systemctl reboot

Repair a damaged file system (unmounted).

#umount /data
#xfs_repair /dev/vgdata/lvdata
#fsck.ext4 -f /dev/vgdata/lvweb

File system	Check / repair
XFS	`xfs_repair DEVICE` (`-n` = only check)
ext4	`fsck.ext4 -f DEVICE` or `e2fsck -f DEVICE`

Let op

Never repair a mounted file system. For the root file system, repair from a rescue disk or emergency mode with / read-only.

18.8

Reset the root password

Without the root password you can stop the boot in the initramfs with rd.break, then change the password in the real root file system. SELinux must relabel afterwards.

Reboot, press e at the GRUB menu, add rd.break to the end of the linux line, press Ctrl+X.
At the switch_root:/# prompt, make the real root writable and enter it.
```
#mount -o remount,rw /sysroot
#chroot /sysroot
```
Set the new password and ask SELinux to relabel at the next boot.
```
#passwd root
#touch /.autorelabel
```
Leave the chroot and the initramfs shell. The system continues to boot (the relabel takes some minutes and reboots once).
```
#exit
#exit
```

Examentip

The relabel step (touch /.autorelabel) is the one people forget. Without it, /etc/shadow has a wrong SELinux label and nobody can log in.

Op het werk

Anybody at the console can do this. Protect the GRUB menu with grub2-setpassword and use disk encryption (LUKS) for systems in unsafe places.

18.9

Remember the main points

The most important facts of this chapter, in short.

Boot steps: firmware (BIOS/UEFI) → GRUB 2 → kernel + initramfs → systemd → default target.
Useful kernel arguments at the GRUB menu: systemd.unit=rescue.target, systemd.unit=emergency.target, rd.break.
In emergency mode / is read-only: remount with mount -o remount,rw / to fix files like fstab.
Reset a lost root password: rd.break (or init=/bin/bash), mount -o remount,rw /sysroot, chroot /sysroot, passwd root, touch /.autorelabel, exit twice.
A broken GRUB can be fixed from a rescue disk with chroot and grub2-install (BIOS) or by reinstalling the GRUB packages (UEFI).
A broken initramfs is made again with dracut -f.
File system problems: run xfs_repair (XFS) or fsck (ext4) on an unmounted file system.

18.10

Learn the key terms

The words of this chapter in simple English.

Term	Meaning
UEFI / BIOS	The firmware that starts the computer and the boot loader.
initramfs	A small first file system with the drivers needed to find the real root.
rd.break	A kernel argument that stops in the initramfs before the real root is used.
Rescue mode	A minimal system with root login to repair things.
Emergency mode	An even smaller system: only / is mounted, read-only.
chroot	Run a shell with another folder as the root folder.
Relabel	Set all SELinux labels again (`/.autorelabel`).
dracut	The tool that builds the initramfs.

18.11

Check what you learned

Think about the answer first, then open it.

1Name the 5 big steps of the boot process.

Firmware, GRUB 2, kernel with initramfs, systemd, the default target.

2Why must you run touch /.autorelabel after you change the root password from rd.break?

The new /etc/shadow has no correct SELinux label; the relabel at the next boot fixes it, else nobody can log in.

3The system stops in emergency mode after an fstab change. What do you do?

4Which kernel argument starts the system in rescue mode?

systemd.unit=rescue.target.

5How do you build a new initramfs for the running kernel?

dracut -f.

6How do you check an XFS file system?

Unmount it, then xfs_repair /dev/DEVICE.

7Why do you need chroot /sysroot after rd.break?

So commands like passwd work on the real system and not on the initramfs.

8How do you see why the last boot failed?

journalctl -b -1 -p err (needs a persistent journal).

Oefenen

Lab 18.1Break and repair the bootMoeilijk · 45 minNog niet begonnen

Oefenen

Alle commando's in dit hoofdstuk