Filters
How to search
  • Plain words find names, texts and examples.
  • "two words" finds the exact phrase.
  • --name or -x finds an option.
  • #word looks only inside commands.
  • @grep finds lessons and labs that use grep.
  • Filters: cat:network · in:labs · in:notes · level:rhcsa · ch:8 · root:yes. Put ! before a word to leave it out. OR between words means one of them.
  • ↑ ↓ to move · Enter to open · Esc to close · / to search

Observability & SREPlatform series

6 numbers that set your monitoring

Monitoring defaults are not opinions. Six numbers decide how fast you notice, how often you page, and how far back you can look.

Quick reference

NumberWhatWhy it matters
15sScrape intervalA common target: four samples a minute, enough for rate().scrape
5mAlert for: durationA starting point that skips one bad scrape and still pages fast.alert
14.4xBurn rate that pages1h at 14.4x spends 2% of a 30-day budget. Pair it with 5m.SLO
15dRetention defaultLocal Prometheus blocks last 15 days unless you raise it.storage
p99Not the averageThe slowest 1% complains; the mean hides them.latency
30dSLO windowA month of requests makes a 99.9% target measurable.window

Explained

Scrape

15s, the scrape interval. Many teams scrape every 15 seconds: four samples a minute, enough for rate() and small enough to catch a short spike. Prometheus's own default is one minute, which is slower than most symptom alerts want.

Alert

5m, the for: duration. A starting point for symptom alerts: long enough to skip a single bad scrape, short enough that five minutes of pain still gets a response. It is a choice, not a default.

14.4x, the burn rate that pages. At a 30-day budget, burning 14.4 times the normal rate for one hour spends 2% of the budget. That is the multi-window alert from the SRE workbook, paired with a 5m short window.

Retain and read

15d, the retention default. Prometheus keeps local blocks for 15 days unless you raise it. That covers most incident reviews; it does not cover a quarterly trend, which is what remote storage is for.

p99, not the average. The slowest 1% is who complains, and the mean hides them. Put the percentile on the dashboard and the average in the appendix.

30d, the SLO window. A month of requests makes a 99.9% target measurable and stops a quiet week from flattering the number.

The honest caveat: none of these is universal. A 15s scrape is wrong for an hourly batch job, and 15 days of retention is useless if reviews happen quarterly.

The lesson: pick the number that matches the user-visible threshold, then configure the tool.

In one line

Pick the number, then the tool. A default nobody measured is a rumour with a config file.

Set it in code

scrape_interval: 15s

More notes: Observability & SRE

Open in the notes app: search, filters, copy This note on GitHub