Filters
Zo zoek je
  • Gewone woorden vinden namen, teksten en voorbeelden.
  • "twee woorden" vindt precies die zin.
  • --naam of -x vindt een optie.
  • #woord zoekt alleen in commando's.
  • @grep vindt lessen en labs die grep gebruiken.
  • Filters: cat:network · in:labs · in:notes · level:rhcsa · ch:8 · root:yes. Zet ! voor een woord om het weg te laten. OR tussen woorden betekent één ervan.
  • ↑ ↓ bewegen · Enter openen · Esc sluiten · / zoeken

amtool

Alertmanager groups, routes and silences Prometheus alerts and sends them to e-mail, Slack, Teams or PagerDuty; amtool controls it.

Niveau
DevSecOps-service
Groep
Monitoring en logging
Root nodig
Nee
Pakket
alertmanager
Ook
alertmanager

Dit deel is nog niet vertaald. Je ziet de Engelse tekst.

Wat het doet

Prometheus fires alerts; Alertmanager (port 9093) decides who gets what: a route tree matches labels (severity, team), receivers send to channels, grouping joins many alerts into one message, inhibition hides warnings when a critical alert for the same thing fires, and silences mute alerts during maintenance.

amtool checks the config, lists alerts, adds and removes silences, and tests which receiver a label set reaches.

Zo schrijf je het

amtool [--alertmanager.url=URL] COMMAND [ARGS]

Belangrijkste delen

Commands

check-config FILE
Validate alertmanager.yml.
alert query [MATCHERS]
List firing alerts.
alert add NAME LABELS
Send a test alert.
silence add MATCHERS --comment TEXT --duration 2h
Mute alerts.
silence query
List silences.
silence expire ID
End a silence.
config routes show / test LABELS
Show the route tree / which receiver matches.
config show
Show the running config.

Opties 5

Meest gebruikt

Flags

--alertmanager.url=URL
Alertmanager address (or in ~/.config/amtool/config.yml).
-o simple|extended|json
Output format.
--comment TEXT
Why you silence (required by good practice).
--duration 2h / --end TIME
How long the silence lasts.
--author NAME
Who made the silence.

Voorbeelden 10

$ als gewone gebruiker# als root (of met sudo)

  1. 1A route tree with two receivers

    cat /etc/alertmanager/alertmanager.yml
    route:
      receiver: team-mail
      group_by: [alertname, instance]
      group_wait: 30s
      repeat_interval: 4h
      routes:
        - matchers: [severity="critical"]
          receiver: oncall-slack
    receivers:
      - name: team-mail
        email_configs: [{ to: ops@example.com }]
      - name: oncall-slack
        slack_configs: [{ api_url_file: /etc/alertmanager/slack.url, channel: "#oncall" }]
    inhibit_rules:
      - source_matchers: [severity="critical"]
        target_matchers: [severity="warning"]
        equal: [instance]
  2. 2Check the config

    amtool check-config /etc/alertmanager/alertmanager.yml
  3. 3Set the URL once

    cat ~/.config/amtool/config.yml
    alertmanager.url: http://localhost:9093
  4. 4See firing alerts

    amtool alert query
    amtool alert query severity=critical -o extended
  5. 5Silence a server during maintenance

    amtool silence add instance="web1:9100" --comment "Kernel update, ticket OPS-412" --duration 1h
    amtool silence query
    amtool silence expire 7a3c2e1f-5b4d-4c6e-9f8a-1b2c3d4e5f60
  6. 6Test which receiver gets an alert

    amtool config routes test severity=critical instance=db1:9100
    oncall-slack
    amtool config routes show
  7. 7Send a test alert end-to-end

    amtool alert add TestAlert severity=warning instance=lab --annotation=summary="Test from amtool"
  8. 8Query the API with curl

    curl -s http://localhost:9093/api/v2/alerts | jq '.[].labels.alertname'
  9. 9Reload after a change

    curl -X POST http://localhost:9093/-/reload
  10. 10Run in the cluster (kube-prometheus-stack)

    kubectl -n monitoring exec -it alertmanager-monitoring-kube-prometheus-alertmanager-0 -- amtool alert query --alertmanager.url=http://localhost:9093

Let op

  • Too many alerts = people stop reading them. Alert on symptoms (users hurt), not on every cause.
  • Keep webhook URLs and passwords in files (*_file options), not in the YAML in Git.

Bestanden

/etc/alertmanager/alertmanager.yml
Configuration.
~/.config/amtool/config.yml
amtool defaults.

Tips om te onthouden

  • Always write a comment and a ticket number in silences.

Exitcodes

0Success.
1Error.