See what happens (metrics, logs, traces and alerts)

Prometheus scrapes /metrics endpoints, stores time series and evaluates alert rules; promtool checks configs and rules and runs PromQL.

$promtool check config /etc/prometheus/prometheus.yml
$promtool check rules /etc/prometheus/rules/*.yml
$promtool query instant http://localhost:9090 'up == 0'
$promtool test rules node-test.yml

Alertmanager groups alerts, sends them to the right team (mail, Slack, Teams, PagerDuty) and holds silences; amtool controls it.

$amtool check-config /etc/alertmanager/alertmanager.yml
$amtool alert query severity=critical
$amtool silence add instance="web1:9100" --comment "Kernel update OPS-412" --duration 1h
$amtool config routes test severity=critical

node_exporter shows CPU, memory, disk, network and systemd metrics on port 9100; the textfile collector adds your own metrics (backup age).

$curl -s localhost:9100/metrics | grep node_filesystem_avail_bytes
#firewall-cmd --permanent --add-rich-rule='rule family=ipv4 source address=10.20.0.5/32 port port=9100 protocol=tcp accept'

Grafana shows metrics, logs and traces from many sources in dashboards and alerts; provision data sources and dashboards from Git.

#grafana cli admin reset-admin-password 'N3w-Strong-Pass!'
$curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" http://localhost:3000/api/search | jq '.[].title'

Loki stores logs cheaply and indexes only labels; search with LogQL in Grafana or with logcli.

$logcli query '{namespace="shop"} |= "error"' --since=1h
$logcli query '{app="api"} | json | status >= 500' --since=30m
$logcli query '{namespace="shop"}' --tail

fluent-bit collects logs from files, the journal and containers, adds Kubernetes metadata, masks secrets and sends them to Loki, Elasticsearch or Azure Monitor.

#fluent-bit -c /etc/fluent-bit/fluent-bit.conf --dry-run
$helm install fluent-bit fluent/fluent-bit -n logging --create-namespace
$curl -s http://localhost:2020/api/v1/health

otelcol receives traces, metrics and logs (OTLP), processes them (batch, redact, sample) and exports them to any backend — vendor-neutral.

$otelcol-contrib validate --config=/etc/otelcol-contrib/config.yaml
$export OTEL_EXPORTER_OTLP_ENDPOINT=http://otelcol:4318
$curl -s http://localhost:13133/