Prometheus Alerting

Label Best Practices

• Keep label cardinality bounded. Never use user IDs, request IDs, session tokens, IP addresses, or timestamps as label values.
• Use labels for dimensions you will filter/aggregate on: method, status_code, endpoint, instance, job.
• Every unique label combination creates a new time series. Adding a label with 1000 values multiplies series count by 1000.
• Drop high-cardinality labels at scrape time with metric_relabel_configs:

metric_relabel_configs:
  - source_labels: [pod_uid]
    action: labeldrop
  - source_labels: [__name__]
    regex: "expensive_metric_.+"
    action: drop

PromQL Essentials

Selectors

http_requests_total{method="GET", status="200"}   # exact match
http_requests_total{status=~"5.."}                 # regex match
http_requests_total{method!="OPTIONS"}             # negative match
http_requests_total{job="api"}[5m]                 # range vector

Aggregation

sum by (method) (rate(http_requests_total[5m]))
avg without (instance) (node_memory_MemAvailable_bytes)
topk(5, sum by (endpoint) (rate(http_requests_total[5m])))
count by (job) (up)

Key Functions

rate(http_requests_total[5m])                       # per-second rate (counters)
increase(http_requests_total[1h])                   # total increase (counters)
histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))
predict_linear(node_filesystem_avail_bytes[1h], 4 * 3600)  # predict 4h ahead
absent(up{job="payment-service"})                   # detect missing series
clamp_min(free_disk_bytes, 0)
rate(http_requests_total[5m])[30m:1m]               # subquery

Common PromQL Patterns

Error Rate (RED)

sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
  / sum by (service) (rate(http_requests_total[5m]))

Saturation (USE)

1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))  # CPU
1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)      # memory
rate(node_disk_io_time_seconds_total[5m])                               # disk I/O

Availability SLI

sum(rate(http_requests_total{status!~"5.."}[5m]))
  / sum(rate(http_requests_total[5m]))

Recording Rules

Pre-compute expensive queries. Use the naming convention level:metric:operations.