Alertmanager Standalone

Configurar el routing de alertas con receptores diferenciados por severidad:

# alertmanager.yml
global:
  resolve_timeout: 5m

route:
  receiver: 'slack-default'
  group_by: ['alertname', 'module']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - match:
        severity: critical
      receiver: 'pagerduty-kyc'
      group_wait: 10s
    - match:
        module: liveness
      receiver: 'slack-antifraude'
    - match:
        severity: warning
      receiver: 'slack-default'

Configurar los receptores de PagerDuty para alertas criticas del pipeline:

receivers:
  - name: 'pagerduty-kyc'
    pagerduty_configs:
      - service_key_file: '/etc/alertmanager/secrets/pagerduty_key'
        severity: 'critical'
        description: '{{ .CommonAnnotations.summary }}'
        details:
          module: '{{ .CommonLabels.module }}'
          session_count: '{{ .CommonAnnotations.affected_sessions }}'

Configurar el receptor de Slack para alertas de severidad warning:

  - name: 'slack-default'
    slack_configs:
      - api_url_file: '/etc/alertmanager/secrets/slack_webhook'
        channel: '#kyc-alerts'
        title: '[{{ .Status | toUpper }}] {{ .CommonLabels.alertname }}'
        text: '{{ .CommonAnnotations.description }}'
        send_resolved: true
  - name: 'slack-antifraude'
    slack_configs:
      - api_url_file: '/etc/alertmanager/secrets/slack_webhook'
        channel: '#kyc-antifraude'
        title: 'Alerta Liveness: {{ .CommonLabels.alertname }}'
        text: '{{ .CommonAnnotations.description }}'

Definir reglas de inhibicion para evitar cascadas de alertas:

inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'module']
  - source_match:
      alertname: 'KYCServiceDown'
    target_match:
      alertname: 'KYCHighLatency'
    equal: ['module']

Definir las alerting rules en Prometheus apuntando al Alertmanager independiente:

# prometheus.yml
alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

rule_files:
  - '/etc/prometheus/rules/kyc_alerts.yml'

Crear reglas de alerta especificas para el pipeline KYC:

# kyc_alerts.yml
groups:
  - name: kyc-pipeline
    rules:
      - alert: KYCHighFraudRate
        expr: rate(kyc_fraud_detected_total[5m]) > 0.1
        for: 2m
        labels:
          severity: critical
          module: antifraud
        annotations:
          summary: "Tasa de fraude anormalmente alta"
          description: "Se detectan mas de 0.1 fraudes/s en los ultimos 5 minutos"
      - alert: KYCVerificationLatencyHigh
        expr: histogram_quantile(0.95, rate(kyc_verification_duration_seconds_bucket[5m])) > 8
        for: 3m
        labels:
          severity: warning
        annotations:
          summary: "Latencia p95 supera el objetivo de 8 segundos"

Validar la configuracion con amtool antes de desplegar:

amtool check-config alertmanager.yml
amtool config routes show --config.file=alertmanager.yml