Prometheus
Alert notifications from Prometheus Alertmanager
This section explains on how to integrate and ingest alerts from Prometheus monitoring tool into CloudFabrix AIOPs platform.
Prometheus Alertmanager is alert management component which supports alert notifications via email, slack, webhook and others. CloudFabrix AIOPs platform uses webhook notification method over HTTP protocol to receive and ingest the alerts or events.
Click here for Alert Sources to create a Webhook URL for Prometheus alert notifications in CloudFabrix OIA application.
Below is the sample configuration to define Alert threshold rules to trigger alerts for monitored assets with alert rules configuration file. (Note: Below alert trigger rules for reference only)
groups:
- name: ALERTENGINE
rules:
- alert: ALERT_MANAGER_FAILURES
expr: rate(alertmanager_notifications_failed_total[5m]) > 0
labels:
severity: CRITICAL
category: ALERTING
annotations:
title: Alertmanager is failing to send notications
description: Alertmanager is seeing errors {{$labels.integration}}
- name: CATASTROPHIC
rules:
- alert: HOST_DOWN
expr: avg_over_time(up{job=~"Hosts|Containers"}[2m]) == 0
labels:
severity: CRITICAL
category: AVAILABILITY
annotations:
summary: "{{$labels.instance}}: Host is unreachable. Host could be down. The Collecors are not accessible. If the host is up, make sure collectors are running."
description: "{{$labels.instance}}: Host is unreachable. Host could be down. The Collecors are not accessible. If the host is up, make sure collectors are running."
- name: HOST
rules:
- alert: HOST_HIGH_MEMORY_USAGE
expr: (((avg_over_time(node_memory_MemTotal_bytes[5m]) - avg_over_time(node_memory_MemFree_bytes[5m]) - avg_over_time(node_memory_Cached_bytes[5m])) / (avg_over_time(node_memory_MemTotal_bytes[5m])) * 100)) > 80
labels:
severity: HIGH
category: HOST_MEMORY
annotations:
summary: "{{$labels.instance}}: Memory Usage detected above 80"
description: "{{$labels.instance}}: Memory usage usage is above 80% (Current Used Memory % is: {{ $value }})"
- alert: HOST_HIGH_DISK_USAGE
expr: ((avg_over_time(node_filesystem_size_bytes{fstype=~"(ext.|xfs)"}[5m]) - avg_over_time(node_filesystem_free_bytes{fstype=~"(ext.|xfs)"}[5m])) * 100 / avg_over_time(node_filesystem_size_bytes{fstype=~"(ext.|xfs)"}[5m])) > 70
labels:
severity: HIGH
category: HOST_DISK
annotations:
summary: "{{$labels.instance}}: Disk {{$labels.device}} Usage detected above 70"
description: "{{$labels.instance}}: Disk {{$labels.device}} usage usage is above 70% (Current Disk Used % is: {{ $value }})"
- alert: HOST_HIGH_CPU_USAGE
expr: (100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)) > 70
labels:
severity: HIGH
category: HOST_CPU
annotations:
summary: "{{$labels.instance}}: CPU Usage detected above 70"
description: "{{$labels.instance}}: CPU usage usage is above 70% (Current CPU % is: {{ $value }})"
- alert: HOST_HIGH_DISK_UTILIZATION
expr: rate(node_disk_io_time_seconds_total[5m]) / 10 > 90
labels:
severity: HIGH
category: HOST_DISK
annotations:
summary: "{{$labels.instance}}: Disk ( {{ $labels.device }} ) utilization is very high."
description: "{{$labels.instance}}: Disk ( {{ $labels.device }} ) utilization is very high. (Current Utilization is: {{ $value }})"
- alert: HOST_HIGH_DISK_INODE
expr: avg_over_time(node_filesystem_files_free{fstype=~"(ext.|xfs)"}[5m]) / avg_over_time(node_filesystem_files{fstype=~"(ext.|xfs)"}[5m]) * 100 <= 20
labels:
severity: HIGH
category: HOST_DISK
annotations:
summary: "{{$labels.instance}}: Disk ( {{ $labels.device }} ) High number of inode usage"
description: "{{$labels.instance}}: Disk ( {{ $labels.device }} ) High number of inode usage. (Current value is: {{ $value }})"
Below is the sample configuration for Prometheus alertmanager to send alert notifications to CloudFabrix AIOps platform over Webhook URL. (config.yml)
route:
repeat_interval: 1m
receiver: cfx-webhook
receivers:
- name: cfx-webhook
webhook_configs:
- url: 'https://<cfx-aiops-webhook-URL>'
send_resolved: true
http_config:
# basic_auth:
# username: <optional>
# password: <optional>
tls_config:
insecure_skip_verify: true
Last modified 10mo ago