Check currently firing Grafana alerts, analyze alert status, and investigate alert issues in the Kagenti platform
This skill helps you check and analyze Grafana alerts in the Kagenti platform.
# Get all currently firing alerts from Grafana
kubectl exec -n observability deployment/grafana -- \
curl -s 'http://localhost:3000/api/alertmanager/grafana/api/v2/alerts' \
-u admin:admin123 | python3 -c "
import sys, json
alerts = json.load(sys.stdin)
firing = [a for a in alerts if a.get('status', {}).get('state') == 'active']
print(f'Firing alerts: {len(firing)}')
for alert in firing:
labels = alert.get('labels', {})
annotations = alert.get('annotations', {})
print(f\"\\n• {labels.get('alertname')} ({labels.get('severity')})\")
print(f\" Component: {labels.get('component')}\")
print(f\" Description: {annotations.get('description', 'N/A')[:100]}...\")
"
# Get all configured alert rules
kubectl exec -n observability deployment/grafana -- \
curl -s 'http://localhost:3000/api/v1/provisioning/alert-rules' \
-u admin:admin123 | python3 -c "
import sys, json
rules = json.load(sys.stdin)
print(f'Total alert rules: {len(rules)}')
for rule in rules:
print(f\" • {rule.get('title')} ({rule.get('labels', {}).get('severity')})\")
"
# Get configuration for a specific alert
kubectl exec -n observability deployment/grafana -- \
curl -s 'http://localhost:3000/api/v1/provisioning/alert-rules' \
-u admin:admin123 | python3 -c "
import sys, json
rules = json.load(sys.stdin)
alert_uid = 'prometheus-down' # Change this to the alert UID
rule = next((r for r in rules if r.get('uid') == alert_uid), None)
if rule:
print(f\"Alert: {rule.get('title')}\")
print(f\"Query: {rule.get('data', [{}])[0].get('model', {}).get('expr')}\")
print(f\"noDataState: {rule.get('noDataState')}\")
print(f\"execErrState: {rule.get('execErrState')}\")
"
# Test an alert's PromQL query
QUERY='up{job="kubernetes-pods",app="prometheus"} == 0'
kubectl exec -n observability deployment/grafana -- \
curl -s -G 'http://prometheus.observability.svc:9090/api/v1/query' \
--data-urlencode "query=${QUERY}" | python3 -m json.tool
# Check why an alert is firing or not firing
kubectl exec -n observability deployment/grafana -- \
curl -s 'http://localhost:3000/api/v1/eval/rules' \
-u admin:admin123 | python3 -m json.tool
Access Grafana: https://grafana.localtest.me:9443 Credentials: admin / admin123
Navigation:
noDataState configuration (should be OK for most alerts)for duration isn't too longnoDataState isn't masking the issuekubectl rollout restart deployment/grafana -n observabilitykubectl get configmap grafana-alerting -n observabilityWhen an alert fires, consult its runbook:
docs/runbooks/alerts/<alert-uid>.mdExample: If "Prometheus Down" alert fires → docs/runbooks/alerts/prometheus-down.md
🤖 Generated with Claude Code