Grafana Observability

Environment Variables

*Either service account token or username/password required.

Key Tool Categories

Dashboard Operations

Prometheus (PromQL)

Loki (LogQL)

Alerting

Incident Management

OnCall

Annotations & Rendering

Investigation (Sift)

Workflow: Network Infrastructure Monitoring

When checking network device metrics in Grafana:

1. Find dashboards: search_dashboards with keyword (e.g., "network", "interface", "BGP")
2. Dashboard overview: get_dashboard_summary for panel list without full JSON
3. Query metrics: query_prometheus with PromQL for specific metrics:
   • Interface traffic: rate(ifHCInOctets{instance="router1"}[5m]) * 8
   • BGP peer state: bgp_peer_state{peer="10.1.1.2"}
   • CPU utilization: device_cpu_utilization{device="core-rtr-01"}
   • Interface errors: increase(ifInErrors{device=~".*"}[1h])
4. Check alerts: list_alert_rules to see active alerting thresholds
5. Search logs: query_loki_logs for syslog or SNMP trap data
6. Report: Metrics summary with alert status and log correlation
7. GAIT: Record all queries in audit trail

Example: Interface Utilization Check

search_dashboards(title="Network Interfaces")
get_dashboard_summary(uid="abc123")
query_prometheus(expr="rate(ifHCInOctets{device='core-rtr-01'}[5m]) * 8", time_range="1h")
query_prometheus(expr="rate(ifHCOutOctets{device='core-rtr-01'}[5m]) * 8", time_range="1h")
list_alert_rules(folder="Network")

Workflow: Alert Investigation

When investigating Grafana alerts:

1. List alerts: list_alert_rules — find firing or pending rules
2. Alert details: get_alert_rule_by_uid — thresholds, conditions, datasource
3. Query metrics: query_prometheus — check the metric that triggered the alert
4. Search logs: query_loki_logs — correlate with log events around alert time
5. Check incidents: list_incidents — is this already tracked?
6. Contact points: list_contact_points — verify notification routes
7. Report: Alert analysis with root cause and metric evidence

Workflow: Incident Response

When responding to a Grafana incident:

1. List incidents: list_incidents — find open incidents
2. Incident details: get_incident — timeline, severity, labels
3. OnCall: get_current_oncall_users — who should be notified
4. Correlate metrics: query_prometheus — check affected service metrics
5. Correlate logs: query_loki_logs — find error patterns around incident time
6. Investigate: find_error_pattern_logs — automated error pattern detection
7. Update incident: add_activity_to_incident — add findings to timeline
8. Annotate: create_annotation — mark event on relevant dashboards

Workflow: Log Analysis

When investigating network logs stored in Loki:

1. Discover labels: list_loki_label_names — find available labels (host, severity, facility)
2. Label values: list_loki_label_values — enumerate hosts, severity levels
3. Query logs: query_loki_logs with LogQL:
   • By device: {host="core-rtr-01"}
   • By severity: {host="core-rtr-01"} |= "error"
   • Pattern match: {job="syslog"} |~ "BGP|OSPF"
4. Patterns: query_loki_patterns — detect recurring log structures
5. Stats: query_loki_stats — log volume and rate analysis

Integration with Other Skills

Context Window Management

Grafana dashboards can be large JSON documents. Use these strategies:

1. Always start with get_dashboard_summary — lightweight overview, not full JSON
2. Use get_dashboard_property with JSONPath for specific fields
3. Avoid get_dashboard_by_uid unless you need the complete dashboard definition
4. Use get_dashboard_panel_queries to extract just the query definitions

Important Rules

• Prefer read-only operations — use search_dashboards, get_dashboard_summary, query_prometheus, query_loki_logs, list_alert_rules before any write operations
• Dashboard modifications require ServiceNow CR — unless in lab/dev Grafana instance
• Alert rule changes require approval — creating/updating/deleting alert rules affects production monitoring
• Token-efficient queries — use get_dashboard_summary over get_dashboard_by_uid, use time ranges to limit Prometheus/Loki result size
• GAIT audit mandatory — record all Grafana queries, dashboard modifications, alert changes, and incident updates
• No secrets in queries — never embed credentials or sensitive data in PromQL/LogQL expressions

Error Handling

• Auth fails (401/403): Check GRAFANA_URL and GRAFANA_SERVICE_ACCOUNT_TOKEN in ~/.openclaw/.env. Verify service account has Editor role or required RBAC permissions.
• Datasource not found: Use list_datasources to discover available datasource UIDs and names.
• PromQL/LogQL errors: Use list_prometheus_metric_names or list_loki_label_names to discover valid metric/label names before querying.
• Dashboard not found: Use search_dashboards to find dashboards by title before using UID-based tools.
• Rate limiting: Grafana may rate-limit API requests; space out large query batches.