See what happens (metrics, logs, traces and alerts)

$ run as a normal user# run as root (or with sudo)

Skills in this chapter

Observability — 7

Prometheus — metrics and alert rules

Prometheus scrapes /metrics endpoints, stores time series and evaluates alert rules; promtool checks configs and rules and runs PromQL.

$promtool check config /etc/prometheus/prometheus.yml
$promtool check rules /etc/prometheus/rules/*.yml
$promtool query instant http://localhost:9090 'up == 0'
$promtool test rules node-test.yml

Security checklist:

No login by default: TLS + basic auth (--web.config.file) or keep it internal.
Avoid high-cardinality labels.

7.2

Alertmanager — route and silence alerts

Alertmanager groups alerts, sends them to the right team (mail, Slack, Teams, PagerDuty) and holds silences; amtool controls it.

$amtool check-config /etc/alertmanager/alertmanager.yml
$amtool alert query severity=critical
$amtool silence add instance="web1:9100" --comment "Kernel update OPS-412" --duration 1h
$amtool config routes test severity=critical

7.3

node_exporter — Linux host metrics

node_exporter shows CPU, memory, disk, network and systemd metrics on port 9100; the textfile collector adds your own metrics (backup age).

$curl -s localhost:9100/metrics | grep node_filesystem_avail_bytes
#firewall-cmd --permanent --add-rich-rule='rule family=ipv4 source address=10.20.0.5/32 port port=9100 protocol=tcp accept'

7.4

Grafana — dashboards

Grafana shows metrics, logs and traces from many sources in dashboards and alerts; provision data sources and dashboards from Git.

#grafana cli admin reset-admin-password 'N3w-Strong-Pass!'
$curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" http://localhost:3000/api/search | jq '.[].title'

Security checklist:

Change admin/admin, no sign-up, no anonymous access, SSO.
Secrets via $__file{} or env vars.

7.5

Loki — logs by labels

Loki stores logs cheaply and indexes only labels; search with LogQL in Grafana or with logcli.

$logcli query '{namespace="shop"} |= "error"' --since=1h
$logcli query '{app="api"} | json | status >= 500' --since=30m
$logcli query '{namespace="shop"}' --tail

7.6

Fluent Bit — ship the logs

fluent-bit collects logs from files, the journal and containers, adds Kubernetes metadata, masks secrets and sends them to Loki, Elasticsearch or Azure Monitor.

#fluent-bit -c /etc/fluent-bit/fluent-bit.conf --dry-run
$helm install fluent-bit fluent/fluent-bit -n logging --create-namespace
$curl -s http://localhost:2020/api/v1/health

Security checklist:

Remove tokens, passwords and personal data before logs leave the host.
TLS for outputs.

7.7

OpenTelemetry Collector — one pipeline for all signals

otelcol receives traces, metrics and logs (OTLP), processes them (batch, redact, sample) and exports them to any backend — vendor-neutral.

$otelcol-contrib validate --config=/etc/otelcol-contrib/config.yaml
$export OTEL_EXPORTER_OTLP_ENDPOINT=http://otelcol:4318
$curl -s http://localhost:13133/

At work

Build the stack once in your home lab with helm install … kube-prometheus-stack, Loki and Grafana; then write 3 alerts (instance down, disk almost full, 5xx error rate) and one dashboard. That is a strong interview story.

All commands in this chapter