promtool

What it does

How to write it

promtool check config|rules|metrics FILE
promtool query instant|range URL 'PROMQL'
promtool test rules TEST.yml

Main parts

promtool commands

check config FILE

Validate prometheus.yml (and the rule files it loads).

check rules FILE...

Validate rule files.

check metrics

Lint metrics text from stdin (naming rules).

test rules FILE

Unit tests for alerts and recording rules.

query instant URL EXPR

Run a PromQL query now.

query range URL EXPR --start --end --step

Query over time.

query labels URL NAME

Values of a label.

tsdb analyze DIR

Show which series use the most space.

tsdb create-blocks-from openmetrics

Backfill old data.

Most used

Nothing matches this filter.

$ run as a normal user# run as root (or with sudo)

1. 1A small prometheus.yml

   $cat /etc/prometheus/prometheus.yml
   global:
     scrape_interval: 15s
   rule_files: [/etc/prometheus/rules/*.yml]
   alerting:
     alertmanagers: [{ static_configs: [{ targets: ["localhost:9093"] }] }]
   scrape_configs:
     - job_name: node
       static_configs:
         - targets: ["web1:9100", "web2:9100", "db1:9100"]

2. 2Check the config and reload

   $promtool check config /etc/prometheus/prometheus.yml
   #systemctl reload prometheus
   $curl -X POST http://localhost:9090/-/reload

3. 3Alert when a server is down or a disk is almost full

   $cat /etc/prometheus/rules/node.yml
   groups:
     - name: node
       rules:
         - alert: InstanceDown
           expr: up == 0
           for: 2m
           labels: { severity: critical }
           annotations: { summary: "{{ $labels.instance }} is down" }
         - alert: DiskAlmostFull
           expr: (node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes) < 0.10
           for: 10m
           labels: { severity: warning }
   $promtool check rules /etc/prometheus/rules/node.yml

4. 4Query from the terminal

   $promtool query instant http://localhost:9090 'up'
   $promtool query instant http://localhost:9090 'sum by (instance) (rate(node_cpu_seconds_total{mode!="idle"}[5m]))'

5. 5Useful PromQL queries

   $cat promql-cheatsheet.txt
   100 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100
   (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100
   sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))
   histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))
   predict_linear(node_filesystem_avail_bytes[6h], 24*3600) < 0

6. 6Ask the HTTP API with curl

   $curl -s 'http://localhost:9090/api/v1/query?query=up' | jq '.data.result[] | [.metric.instance, .value[1]]'
   $curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | [.labels.job, .health]'

7. 7Unit-test an alert

   $cat node-test.yml
   rule_files: [node.yml]
   tests:
     - interval: 1m
       input_series:
         - series: 'up{job="node",instance="web1:9100"}'
           values: '0x5'
       alert_rule_test:
         - eval_time: 5m
           alertname: InstanceDown
           exp_alerts: [{ exp_labels: { severity: critical, job: node, instance: "web1:9100" } }]
   $promtool test rules node-test.yml

8. 8Install the full stack on Kubernetes

   $helm install monitoring prometheus-community/kube-prometheus-stack -n monitoring --create-namespace
   $kubectl -n monitoring port-forward svc/monitoring-kube-prometheus-prometheus 9090

9. 9Find series that use the most memory

   $promtool tsdb analyze /var/lib/prometheus/data | head -30

10. 10Open the firewall for the UI (lab only)

    #firewall-cmd --permanent --add-port=9090/tcp && firewall-cmd --reload

Nothing matches this filter.

Files

/etc/prometheus/prometheus.yml

Main config.

/etc/prometheus/rules/

Rule files.

/var/lib/prometheus/

Time-series data.

Exit codes