node_exporter

What it does

How to write it

node_exporter [--collector.NAME] [--no-collector.NAME] [--web.listen-address=:9100]

Main parts

Important metrics

node_cpu_seconds_total

CPU time per mode (idle, user, system, iowait).

node_memory_MemAvailable_bytes

Memory that apps can still use.

node_filesystem_avail_bytes

Free space per mount.

node_disk_io_time_seconds_total

Disk busy time.

node_network_receive_bytes_total

Network traffic.

node_load1 / node_load5

Load average.

node_systemd_unit_state

State of systemd units (with --collector.systemd).

node_textfile_*

Your own metrics from the textfile collector.

Most used

Nothing matches this filter.

$ run as a normal user# run as root (or with sudo)

1. 1Install from EPEL or the release tarball

   #dnf install -y golang-github-prometheus-node-exporter
   #systemctl enable --now prometheus-node-exporter
   $curl -s localhost:9100/metrics | head

2. 2Run it as a hardened systemd service

   #cat /etc/systemd/system/node_exporter.service
   [Service]
   User=node_exporter
   ExecStart=/usr/local/bin/node_exporter --collector.systemd --collector.textfile.directory=/var/lib/node_exporter
   NoNewPrivileges=yes
   ProtectSystem=strict
   ProtectHome=yes
   ReadOnlyPaths=/

3. 3Allow only the Prometheus server

   #firewall-cmd --permanent --add-rich-rule='rule family=ipv4 source address=10.20.0.5/32 port port=9100 protocol=tcp accept'
   #firewall-cmd --reload

4. 4Check key numbers by hand

   $curl -s localhost:9100/metrics | grep -E '^node_memory_MemAvailable_bytes|^node_load1 '
   $curl -s localhost:9100/metrics | grep 'node_filesystem_avail_bytes{.*mountpoint="/"'

5. 5Add your own metric with the textfile collector

   $cat /usr/local/bin/backup-metric.sh
   #!/bin/bash
   echo "backup_last_success_timestamp_seconds $(date +%s)" > /var/lib/node_exporter/backup.prom.$$
   mv /var/lib/node_exporter/backup.prom.$$ /var/lib/node_exporter/backup.prom

6. 6Alert when the backup is older than 26 hours

   $cat rules/backup.yml
   groups:
     - name: backup
       rules:
         - alert: BackupTooOld
           expr: time() - backup_last_success_timestamp_seconds > 26*3600

7. 7Scrape it from Prometheus

   $cat prometheus.yml
   scrape_configs:
     - job_name: node
       static_configs: [{ targets: ["web1:9100", "web2:9100"] }]

8. 8Protect it with TLS and basic auth

   $cat /etc/node_exporter/web.yml
   tls_server_config: { cert_file: /etc/node_exporter/node.crt, key_file: /etc/node_exporter/node.key }
   basic_auth_users: { prometheus: "$2y$10$examplebcrypthashonly000000000000000000000000000000" }

9. 9See failed systemd units as metrics

   $curl -s localhost:9100/metrics | grep 'node_systemd_unit_state{.*state="failed"} 1'

10. 10Check the exporter itself

    $curl -s localhost:9100/metrics | grep node_exporter_build_info

Nothing matches this filter.

Files

/var/lib/node_exporter/*.prom

Textfile collector files (path is your choice).

Exit codes