amtool

What it does

How to write it

amtool [--alertmanager.url=URL] COMMAND [ARGS]

Main parts

Commands

check-config FILE

Validate alertmanager.yml.

alert query [MATCHERS]

List firing alerts.

alert add NAME LABELS

Send a test alert.

silence add MATCHERS --comment TEXT --duration 2h

Mute alerts.

silence query

List silences.

silence expire ID

End a silence.

config routes show / test LABELS

Show the route tree / which receiver matches.

config show

Show the running config.

Most used

Nothing matches this filter.

$ run as a normal user# run as root (or with sudo)

1. 1A route tree with two receivers

   $cat /etc/alertmanager/alertmanager.yml
   route:
     receiver: team-mail
     group_by: [alertname, instance]
     group_wait: 30s
     repeat_interval: 4h
     routes:
       - matchers: [severity="critical"]
         receiver: oncall-slack
   receivers:
     - name: team-mail
       email_configs: [{ to: ops@example.com }]
     - name: oncall-slack
       slack_configs: [{ api_url_file: /etc/alertmanager/slack.url, channel: "#oncall" }]
   inhibit_rules:
     - source_matchers: [severity="critical"]
       target_matchers: [severity="warning"]
       equal: [instance]

2. 2Check the config

   $amtool check-config /etc/alertmanager/alertmanager.yml

3. 3Set the URL once

   $cat ~/.config/amtool/config.yml
   alertmanager.url: http://localhost:9093

4. 4See firing alerts

   $amtool alert query
   $amtool alert query severity=critical -o extended

5. 5Silence a server during maintenance

   $amtool silence add instance="web1:9100" --comment "Kernel update, ticket OPS-412" --duration 1h
   $amtool silence query
   $amtool silence expire 7a3c2e1f-5b4d-4c6e-9f8a-1b2c3d4e5f60

6. 6Test which receiver gets an alert

   $amtool config routes test severity=critical instance=db1:9100
   oncall-slack
   $amtool config routes show

7. 7Send a test alert end-to-end

   $amtool alert add TestAlert severity=warning instance=lab --annotation=summary="Test from amtool"

8. 8Query the API with curl

   $curl -s http://localhost:9093/api/v2/alerts | jq '.[].labels.alertname'

9. 9Reload after a change

   $curl -X POST http://localhost:9093/-/reload

10. 10Run in the cluster (kube-prometheus-stack)

    $kubectl -n monitoring exec -it alertmanager-monitoring-kube-prometheus-alertmanager-0 -- amtool alert query --alertmanager.url=http://localhost:9093

Nothing matches this filter.

Files

/etc/alertmanager/alertmanager.yml

Configuration.

~/.config/amtool/config.yml

amtool defaults.

Exit codes