Filters
How to search
  • Plain words find names, texts and examples.
  • "two words" finds the exact phrase.
  • --name or -x finds an option.
  • #word looks only inside commands.
  • @grep finds lessons and labs that use grep.
  • Filters: cat:network · in:labs · in:notes · level:rhcsa · ch:8 · root:yes. Put ! before a word to leave it out. OR between words means one of them.
  • ↑ ↓ to move · Enter to open · Esc to close · / to search

Incident response & detectionDevSecOps series

Security operations at a glance

The weekly review opened with a green dashboard and closed with two questions nobody could answer: how long did we take to notice, and how often were we wrong?

Quick reference

Status: This week

MetricValueNote
Open incidents73 high or critical
MTTD18 minmedian, last 30 days
MTTR4h 10malert to service restored
SLA breaches2one critical, overdue
Detections fired146this week, all rules
False positives21%%of 146 alerts

Explained

The week

Open incidents: seven, three of them high or critical. A count is not a workload: two incidents with a named owner and a plan are lighter than seven nobody has touched.

MTTD: eighteen minutes, median over thirty days. Mean time to detect runs from the first event to the alert, so a rule with a one-hour window can never score below an hour.

MTTR: four hours ten minutes. Mean time to resolve runs from the alert to the service restored, and the definition is where meetings go wrong: resolve, respond and remediate are three different stop clocks.

SLA breaches: two, one critical and overdue. A breach is a finding about the process, not about the person who was on call when it landed.

The signal

Detections fired: one hundred and forty-six this week, across every rule. The number says how much the programme produces; it says nothing about how much of it was true.

False positives: twenty-one percent of the 146 alerts. A fifth is tolerable if the queue stays short, and expensive if every one of those thirty-one alerts woke somebody.

The honest caveat: a mean hides the tail. One twenty-six-hour incident disappears inside a four-hour average, so read the median and the ninetieth percentile too.

The lesson: six numbers are a way to ask better questions, not the answer to the review.

In one line

A dashboard is a starting point. Six numbers open the review; they do not close it.

Read the queue

grep -c 'high' alerts.log

More notes: Incident response & detection

Open in the notes app: search, filters, copy This note on GitHub