Triage commands for the first ten minutes

Quick reference

Alerts

$amtool alert query
$amtool silence add alertname=DiskFull
$curl -s localhost:9093/api/v2/alerts
$promtool check rules rules.yml
$jq '.[] | .fingerprint' alerts.json

Timeline

$date -u +%Y-%m-%dT%H:%M:%SZ
$journalctl -u api --since 03:00
$git log --since=03:00 --oneline
$kubectl rollout history deploy/api
$jq -r '.[].started_at' alerts.json

Scope

$kubectl get pods -A -o wide
$kubectl get endpoints payments
$ss -tnp | grep :5432
$dig +short payments.internal
$curl -s localhost:8080/healthz

Handoff

$pd incident:list -s open
$pd incident:ack -i INC-1042
$pd incident:notes -i INC-1042 -n handoff
$jq '.primary,.secondary' rota.json
$date -u +%H:%M

Explained

Alerts

amtool alert query lists what is firing now, not what emailed at 03:00. amtool silence add alertname="DiskFull" stops a known repeat at the source. curl -s localhost:9093/api/v2/alerts returns the raw payload, and promtool check rules rules.yml proves the rule file still parses.

Timeline

date -u gives the clock in UTC, which is the only clock a postmortem can use. journalctl -u api --since 03:00 reads the service log from the incident window. git log --since=03:00 --oneline shows the change before the alert, and kubectl rollout history deploy/api shows the last revision.

Scope

kubectl get pods -A -o wide shows what is running and where. ss -tnp | grep :5432 finds live database connections. kubectl get endpoints payments shows whether the service has any healthy backends left, and dig +short payments.internal answers whether the name still resolves.

Handoff

pd incident:list -s open lists the open incidents. pd incident:ack -i INC-1042 acknowledges the page, and pd incident:notes -i INC-1042 -n handoff writes the handoff line into the incident. jq '.primary,.secondary' rota.json names the next shift.

In one line

Start with alerts

$amtool alert query