Skip to main content

Chapter 11.3 - Alerting

Learning Objectives​

By the end of this chapter, you will be able to:

  • Understand the Prometheus alerting system
  • Configure Alertmanager
  • Create alerting rules
  • Manage notifications (email, Slack, PagerDuty)
  • Configure grouping and routing
  • Implement silence and inhibition

Introduction​

Alerting allows notifying teams when problems are detected in the cluster or applications.

Alerting Architecture​

Prometheus​

  • Collects metrics
  • Evaluates alerting rules
  • Sends alerts to Alertmanager

Alertmanager​

  • Receives alerts from Prometheus
  • Groups and routes alerts
  • Sends notifications

Installing Alertmanager​

Via kube-prometheus-stack​

Alertmanager is included in kube-prometheus-stack.

Verification​

# View Pods
kubectl get pods -n monitoring | grep alertmanager

# Access Alertmanager
kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-alertmanager 9093:9093

Creating Alerting Rules​

PrometheusRule​

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: kubernetes-alerts
  namespace: monitoring
spec:
  groups:
  - name: kubernetes.rules
    rules:
    # Pod CrashLooping
    - alert: PodCrashLooping
      expr: rate(kube_pod_container_status_restarts_total[15m]) > 0
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Pod {{ $labels.pod }} is crash looping"
        description: "Pod {{ $labels.pod }} in namespace {{ $labels.namespace }} has restarted {{ $value }} times in the last 15 minutes"
    
    # High CPU Usage
    - alert: HighCPUUsage
      expr: 100 - (avg(rate(container_cpu_usage_seconds_total[5m])) * 100) < 10
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "High CPU usage on {{ $labels.instance }}"
        description: "CPU usage is above 90% on {{ $labels.instance }}"
    
    # High Memory Usage
    - alert: HighMemoryUsage
      expr: (container_memory_usage_bytes / container_spec_memory_limit_bytes) * 100 > 90
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "High memory usage on {{ $labels.pod }}"
        description: "Memory usage is above 90% on pod {{ $labels.pod }}"

Alertmanager Configuration​

ConfigMap​

apiVersion: v1
kind: ConfigMap
metadata:
  name: alertmanager-config
  namespace: monitoring
data:
  alertmanager.yml: |
    global:
      resolve_timeout: 5m
    
    route:
      group_by: ['alertname', 'cluster', 'service']
      group_wait: 10s
      group_interval: 10s
      repeat_interval: 12h
      receiver: 'default'
      routes:
      - match:
          severity: critical
        receiver: 'critical-alerts'
      - match:
          severity: warning
        receiver: 'warning-alerts'
    
    receivers:
    - name: 'default'
      email_configs:
      - to: 'team@example.com'
        from: 'alertmanager@example.com'
        smarthost: 'smtp.example.com:587'
        auth_username: 'alertmanager@example.com'
        auth_password: 'password'
    
    - name: 'critical-alerts'
      slack_configs:
      - api_url: 'https://hooks.slack.com/services/YOUR/WEBHOOK/URL'
        channel: '#alerts-critical'
        title: '{{ .GroupLabels.alertname }}'
        text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'
    
    - name: 'warning-alerts'
      email_configs:
      - to: 'team@example.com'
        from: 'alertmanager@example.com'
        smarthost: 'smtp.example.com:587'

Notifications​

Email​

receivers:
- name: 'email'
  email_configs:
  - to: 'admin@example.com'
    from: 'alertmanager@example.com'
    smarthost: 'smtp.example.com:587'
    auth_username: 'user'
    auth_password: 'password'
    headers:
      Subject: 'Alert: {{ .GroupLabels.alertname }}'

Slack​

receivers:
- name: 'slack'
  slack_configs:
  - api_url: 'https://hooks.slack.com/services/YOUR/WEBHOOK/URL'
    channel: '#alerts'
    title: '{{ .GroupLabels.alertname }}'
    text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'

PagerDuty​

receivers:
- name: 'pagerduty'
  pagerduty_configs:
  - service_key: 'YOUR_SERVICE_KEY'
    description: '{{ .GroupLabels.alertname }}'

Grouping and Routing​

Grouping​

Similar alerts are grouped together:

route:
  group_by: ['alertname', 'cluster']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 12h

Routing​

Routes alerts based on labels:

route:
  routes:
  - match:
      severity: critical
    receiver: 'critical-team'
  - match:
      severity: warning
    receiver: 'warning-team'

Silence and Inhibition​

Silence​

Temporarily suppress alerts:

# Via the Alertmanager UI
# Or via amtool
amtool silence add alertname=HighCPUUsage --duration=1h

Inhibition​

Suppress less important alerts when a critical alert is active:

inhibit_rules:
- source_match:
    severity: 'critical'
  target_match:
    severity: 'warning'
  equal: ['alertname', 'instance']

Best Practices​

1. Appropriate Rules​

Create rules for real and actionable problems.

2. Realistic Thresholds​

Define thresholds that trigger meaningful alerts.

3. Grouping​

Group alerts to avoid spam.

4. Escalation​

Configure escalation for critical alerts.

5. Documentation​

Document each alerting rule.

Summary​

In this chapter, you learned:

Alerting: Notification system for problems
PrometheusRule: Defines alerting rules
Alertmanager: Manages routing and notifications
Notifications: Email, Slack, PagerDuty
Grouping: Groups similar alerts
Routing: Routes based on labels
Silence/Inhibition: Manages alerts temporarily
Best practices: Appropriate rules, realistic thresholds, grouping

Next Steps​

Module 12: Production Deployment
Lab 11.4: Alerting and Notifications

Chapter created on: December 2024