s0sta
EN NL عربي
Filters
How to search
  • Plain words find names, texts and examples.
  • "two words" finds the exact phrase.
  • --name or -x finds an option.
  • #word looks only inside commands.
  • @grep finds lessons and labs that use grep.
  • Filters: cat:network · in:labs · in:notes · level:rhcsa · ch:8 · root:yes. Put ! before a word to leave it out. OR between words means one of them.
  • ↑ ↓ to move · Enter to open · Esc to close · / to search

IT operations & incidentsIT series

The first 30 minutes of an outage

When production breaks, the difference between a 10-minute blip and a 2-hour outage is rarely the fix itself. It is how the first 30 minutes are handled.

Quick reference

TimeStepWhat happens
00:00Alert firesError rate crosses the threshold. The pager goes off.
00:02AcknowledgeOne person owns it. Open the incident channel.
00:05AssessWhat changed? Last deploy, config, or a traffic spike.
00:10Stop the bleedingRoll back or scale out before you root-cause.
00:20Confirm recoveryMetrics back to normal. Users are unblocked.
00:30Write it downCapture the timeline while it is still fresh.

Explained

00:00 - Alert fires. The error rate, latency or an uptime check crosses a threshold and the pager goes off. Good alerts are few and meaningful, so this one is trusted.

00:02 - Acknowledge. One person takes ownership and opens the incident channel. Clear ownership stops the two worst failure modes: everyone assuming someone else has it, or five people changing things at once.

00:05 - Assess. The first question is almost always "what changed?" The last deploy, a config change, an expired certificate, or a traffic spike. git log, the deploy history and the dashboards answer this fast.

00:10 - Stop the bleeding. This is the key mindset: restore service before you fully understand the cause. Roll back the deploy, scale out, fail over, or disable the broken feature. Users do not care why it broke, they care that it works.

00:20 - Confirm recovery. Watch the metrics return to normal and verify from the outside, as a user would. Do not declare victory from a single green check.

00:30 - Write it down. Capture the timeline and the actions while memory is fresh. This becomes the blameless postmortem that stops it happening again.

The hardest discipline is resisting the urge to debug deeply while users are down. Stabilise first. Root cause after.

Calm is a technical skill, and it can be practised.

In one line

Calm is a technical skill. Stabilise first. Understand the root cause after.

The first move: acknowledge, then stop the bleeding

More notes: IT operations & incidents

Open in the notes app: search, filters, copy This note on GitHub