Observability Stack

2. Deploy Custom Dashboards

# Create ConfigMap from project dashboards
kubectl create configmap grafana-dashboards \
  --namespace monitoring \
  --from-file=grafana/dashboards/ \
  --dry-run=client -o yaml | kubectl apply -f -

# Label for Grafana sidecar auto-discovery
kubectl label configmap grafana-dashboards \
  --namespace monitoring \
  grafana_dashboard=1

3. Deploy Custom Alert Rules

# Apply Prometheus alerting rules
kubectl apply -f prometheus/alerting-rules.yaml -n monitoring

# Apply recording rules
kubectl apply -f prometheus/recording-rules.yaml -n monitoring

# Validate rules syntax
promtool check rules prometheus/alerting-rules.yaml
promtool check rules prometheus/recording-rules.yaml

4. Verify Installation

# Check all components
kubectl get pods -n monitoring

# Port-forward Grafana
kubectl port-forward -n monitoring svc/monitoring-grafana 3000:80

# Port-forward Prometheus
kubectl port-forward -n monitoring svc/monitoring-kube-prometheus-prometheus 9090:9090

# Check Prometheus targets
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets | length'

# Verify Grafana datasources
curl -s -u admin:prom-operator http://localhost:3000/api/datasources | jq '.[].name'

Day-2 Operations

Prometheus Operations

# Check Prometheus status
kubectl get pods -n monitoring -l app.kubernetes.io/name=prometheus

# Port forward Prometheus
kubectl port-forward -n monitoring svc/monitoring-kube-prometheus-prometheus 9090:9090

# Query Prometheus API
curl -s http://localhost:9090/api/v1/query?query=up | jq '.data.result'

# Check targets
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets | length'

Grafana Operations

# Check Grafana status
kubectl get pods -n monitoring -l app.kubernetes.io/name=grafana

# Port forward Grafana
kubectl port-forward -n monitoring svc/monitoring-grafana 3000:80

# List data sources
curl -s -u admin:prom-operator http://localhost:3000/api/datasources | jq '.[].name'

Alert Management

# Check alertmanager
kubectl get pods -n monitoring -l app.kubernetes.io/name=alertmanager

# List active alerts
curl -s http://localhost:9093/api/v2/alerts | jq '.[].labels.alertname'

# Validate Prometheus rules
promtool check rules prometheus/alerting-rules.yaml

Troubleshooting

# Check Prometheus logs
kubectl logs -n monitoring -l app.kubernetes.io/name=prometheus --tail=100

# Check Grafana logs
kubectl logs -n monitoring -l app.kubernetes.io/name=grafana --tail=100

# Check for scrape errors
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | select(.health != "up") | {job: .labels.job, health, lastError}'

Project Files Reference

• Helm values: deploy/helm/monitoring/values.yaml
• Alerting rules: prometheus/alerting-rules.yaml
• Recording rules: prometheus/recording-rules.yaml
• Grafana dashboards: grafana/dashboards/
• Terraform module: terraform/modules/observability/

Best Practices

1. Use ServiceMonitors for scrape configuration
2. Set appropriate retention periods (15d default)
3. Configure alert routing correctly (PagerDuty for critical, Teams for warning)
4. Use recording rules for expensive queries
5. Enable persistent storage for Prometheus and Grafana
6. Configure Entra ID SSO for Grafana
7. Monitor ArgoCD and Backstage scrape targets

Output Format

1. Command executed
2. Monitoring status summary
3. Active alerts if any
4. Recommendations

Integration with Agents

Used by: @sre, @devops