9 steps of a host incident

Quick reference

Assess — Name the tier in three commands

# Load, memory and bytes in one shot.
$uptime; free -m; df -h
# Inodes and the kernel's last words.
$df -i; dmesg -T | tail -40
# Sockets, states and dead children.
$ss -s; ps -eo stat | grep -c Z

Contain — Stop the damage, not the box

# Stop the loop before it fills the disk.
$systemctl stop api.service
# Put a memory cap in before the restart.
$systemctl edit api.service
# Drop the flood at the door.
$iptables -I INPUT -s 10.0.3.17 -j DROP

Fix — Change one thing, then prove it

# Restart with the new limit in place.
$systemctl restart api.service
# Read the line before the first kill.
$journalctl -u api --since 03:00
# Leave a check behind, not a note.
$systemctl enable --now inode_watch.timer

Explained

Assess

The terminal runs the first three commands before you touch anything. uptime; free -m; df -h gives load, memory and bytes in one line. df -i; dmesg -T | tail -40 adds inodes and the kernel's last words, which is where an OOM kill or a disk error appears first. ss -s; ps -eo stat | grep -c Z counts sockets, states and dead children. Under a minute, and you know which tier you are in.

Contain

systemctl stop api.service stops the loop before it fills the disk. systemctl edit api.service puts a memory cap in place before the restart, so the next spike is contained by the cgroup rather than the kernel. iptables -I INPUT -s 10.0.3.17 -j DROP removes a flood at the door. Containment is temporary by design; it buys the fix, not the week.

Fix

systemctl restart api.service brings the service back with the new limit. journalctl -u api --since 03:00 reads the line before the first kill, which is the actual cause. systemctl enable --now inode_watch.timer leaves a check behind, so the same disk does not surprise the next on-call.

In one line

Assess first

$uptime; df -h; df -i