$37
You have full native Grafana access — query data, create dashboards, set alerts, receive alert notifications, annotate events, explore datasources, push custom data, and deliver visualizations inline. Works with ANY data in Grafana, not just agent metrics.
grafana_explore_datasources first when you need a datasource UID — never guess UIDsgrafana_search before creating a dashboard — avoid duplicatesgrafana_get_dashboard before grafana_share_dashboard — you need exact panel IDsgrafana_get_dashboard before grafana_update_dashboard — you need panel IDs and current structuregrafana_query for direct answers over creating dashboards — "what's my cost?" needs a number, not a URLgrafana_query over grafana_create_dashboard + grafana_share_dashboard for simple data questions — a number is faster than a chartgrafana_query_logs for log searches — LogQL for logs, PromQL for metrics, TraceQL for traces. Never use grafana_query for Loki datasourcesgrafana_query_traces for trace searches — TraceQL for traces, PromQL for metrics, LogQL for logs. Never use or for Tempo datasourcesgrafana_querygrafana_query_logsopenclaw_lens_* metricsgrafana_check_alerts — use the suggestedInvestigation field to go directly to querying (it provides the tool, query, and datasource)grafana_check_alerts with action setup once before alert notifications can reach the agent — this creates the webhook contact pointgrafana_explain_metric for "what is this metric?" questions over manual grafana_query — it returns current value, trend, stats, and metadata in one callqueryNames from push response for PromQL queries — don't guess metric names (counters get _total suffix)openclaw_ext_ prefix for custom metrics — grafana_push_metrics auto-prepends it if missinggrafana_query_logs with metric-over-logs queries (count_over_time, rate, topk) before switching to raw log entriesgrafana_check_alerts with action silence to prevent repeat notifications while investigatinglist_rules for complete alert health — grafana_check_alerts with action list_rules returns all rules with live eval state (normal/firing/pending/nodata/error), health, and lastEvaluation — no need to cross-reference with list actiondashboardUid + panelId to re-run panel queries — don't manually extract PromQL/LogQL from get_dashboard output. Both grafana_query and grafana_query_logs accept these params to auto-resolve the panel's query expression and datasource. The tool handles template variable replacement and datasource routing automaticallygrafana_update_dashboard with operation delete and grafana_check_alerts with action delete_rule are permanent and cannot be undonealloy_pipeline action recipes first when unsure which pipeline recipe fits the user's request — because recipes provide validation, credential handling, and sample queries that raw config does notalloy_pipeline action status after creating a pipeline — because data takes 15-20s to flow through the pipeline, and components may fail silently after reloadconfig only when the user explicitly provides Alloy syntaxconfig — when the user provides a connection string, DSN, password, or API key, ALWAYS use the matching recipe (which routes credentials through sys.env(), keeping secrets off disk). If you must use raw config, wrap sensitive values in sys.env("MY_VAR_NAME") and tell the user to set that env var where Alloy runsenvVarsRequired from every pipeline create response — credential recipes may return pending_credentials status when env vars aren't set yet. Tell the user the exact var names and that they must set them where Alloy runs, then verify with action statusgrafana_list_metrics or grafana_query_logs to discover data, grafana_create_dashboard to visualize, grafana_create_alert to monitoralloy_pipeline action diagnose as first step when user reports pipeline issues — because it checks Alloy connectivity, all pipeline health, config file drift, and limits in one callalloy_pipeline with action delete removes the config and data stops flowingjsonExpressions, labelFields, structuredMetadata, tenantValue, matchRoutes, etc. directly to any log recipe (docker-logs, file-logs, syslog, etc.)samplingPolicies for multi-policy tail sampling — don't create raw config when application-traces can handle it. sampleRate is for simple probabilistic, samplingPolicies is for intelligent multi-policy (keep errors, keep slow, sample rest)tenantValue/tenantSource/matchRoutes work on ALL log recipes. Don't create separate "routing" pipelinesreferences/alloy-components.md before composing raw config — it has copy-pasteable snippets for all common Alloy componentsgrafana_explain_metricgrafana_querygrafana_query_logsgrafana_query_tracesgrafana_query_traces (search error/slow) → grafana_query_traces (get → follow correlationHint) → grafana_query_logs → grafana_query → grafana_annotategrafana_search → grafana_get_dashboard → grafana_share_dashboardgrafana_search (check duplicates) → grafana_create_dashboardgrafana_get_dashboard → grafana_update_dashboardgrafana_update_dashboard with operation delete (confirm with user first)grafana_check_alerts (setup) → grafana_create_alertgrafana_check_alerts with action list_rulesgrafana_check_alerts with action list_rules → delete_rule with ruleUidgrafana_push_metrics (with optional timestamp for historical data, auto-registers, returns queryNames) → grafana_query with queryNamesgrafana_explore_datasourcesgrafana_list_metricsgrafana_search (check existing) → grafana_create_dashboard with llm-command-center → follow suggestedNext chain through remaining templatesgrafana_create_dashboard with genai-observability templategrafana_create_dashboard with session-explorer template → paste session IDgrafana_create_dashboard with llm-command-center template (Loki + Tempo)grafana_create_dashboard with cost-intelligence templategrafana_create_dashboard with tool-performance templategrafana_create_dashboard with sre-operations templategrafana_explore_datasources → grafana_check_alerts (list + list_rules) → grafana_search → grafana_get_dashboard (audit=true for each) → summarizegrafana_get_dashboard (audit=true) → review auditSummary + per-panel healthgrafana_security_checkgrafana_check_alerts (setup) → grafana_create_dashboard (security-overview) → grafana_create_alert (webhook error burst, cost spike, tool loops, injection signals)grafana_security_check → grafana_query_logs (correlate) → grafana_annotate (mark investigation) → grafana_check_alerts (silence)grafana_investigate (multi-signal triage) → follow suggestedHypotheses.testWith for deep-divesgrafana_explain_metric (returns anomaly z-score + seasonality vs 1d/7d ago for 24h period)grafana_check_alerts with action analyze — fatigue reportgrafana_investigate → 5-Phase methodology → postmortem template (see sre-investigation.md §9)grafana_annotate (list, tags: ["deploy"]) → grafana_explain_metric (compareWith: "previous")alloy_pipeline action recipes (filter by category) → select recipe → create → status → query → dashboard → alertalloy_pipeline with recipe scrape-endpoint + params { url }alloy_pipeline with recipe [db]-exporter + params { connectionString }alloy_pipeline (log recipe + processing params: jsonExpressions, labelFields, structuredMetadata)alloy_pipeline with recipe docker-logsalloy_pipeline with recipe file-logs + params { paths }alloy_pipeline with recipe loki-push-apialloy_pipeline with recipe kafka-logs + params { brokers, topics }alloy_pipeline with recipe syslogalloy_pipeline with recipe blackbox-exporter + params { targets }alloy_pipeline with recipe kubernetes-pods + kubernetes-services + kubernetes-logs (3 pipelines)alloy_pipeline with recipe otlp-receiveralloy_pipeline with recipe span-metricsalloy_pipeline with recipe service-graphalloy_pipeline with recipe self-monitoringalloy_pipeline with recipe secret-filter-logs + params { paths }alloy_pipeline with recipe elasticsearch-exporter / kafka-exporteralloy_pipeline with recipe node-exporteralloy_pipeline with recipe docker-metricsalloy_pipeline with recipe application-traces + samplingPolicies array (keep errors, keep slow, filter health checks, sample rest)tenantValue or matchRoutes processing paramalloy_pipeline with recipe continuous-profiling + targetsalloy_pipeline with recipe faro-frontendalloy_pipeline with recipe gelf-logsreferences/alloy-components.md → alloy_pipeline with raw config + optional sampleQueriesalloy_pipeline with action recipesalloy_pipeline with action listalloy_pipeline with action status + namealloy_pipeline with action diagnose → follow remediationalloy_pipeline with action delete + name (confirm with user first)When several Grafana environments are configured (dev, staging, prod), every tool accepts an optional instance parameter. grafana_explore_datasources returns availableInstances — use the name values from that list.
Why this matters: Users often need to query production metrics, create dashboards in dev, or compare environments side by side. Each tool call targets one instance.
Smart defaults: Omitting instance always targets the configured default — safe and invisible for single-environment setups. Only specify instance when the user explicitly names a non-default environment.
Cross-environment workflows: Each call is independent. Query prod, create dashboard in dev — just set instance differently on each call. No context switching needed.
| Tool | What It Does |
|---|---|
grafana_explore_datasources | Discover configured datasources (UIDs, types, query routing) — tells you which tool + query language to use for each datasource |
grafana_list_metrics | Discover available metrics or label values from a datasource. Use compact: true with metadata: true for minimal fields in multi-tool chains |
grafana_query | Run PromQL instant/range queries — get numbers directly |
grafana_query_logs | Run LogQL queries against Loki — search and filter logs |
grafana_query_traces | Run TraceQL queries against Tempo — search traces or get full trace by ID |
grafana_create_dashboard | Create dashboards from templates or custom JSON |
grafana_update_dashboard | Add/remove/update panels, change dashboard metadata, or delete dashboard |
grafana_get_dashboard | Get dashboard summary (panels, queries). Use compact: true for overview scans, audit: true to health-check all panels in one call |
grafana_search | Search existing dashboards by title, tags, or starred status |
grafana_share_dashboard | Render panel as image and deliver inline via messaging |
grafana_create_alert | Create Grafana-native alert rules on any metric |
grafana_annotate | Create or list annotations (events) on dashboards |
grafana_check_alerts | Check, acknowledge, list/delete rules, silence/unsilence, or set up Grafana alert webhook notifications. Use compact: true with list_rules for minimal fields |
grafana_push_metrics | Push custom data (calendar, git, fitness, finance) via OTLP |
grafana_explain_metric | Get metric context: current value, trend, stats, metadata, drill-down queries — agent interprets |
grafana_security_check | Run 6 parallel security checks and return threat-level assessment (green/yellow/red) — "Am I being attacked?" |
grafana_investigate | Multi-signal investigation triage — gathers metrics, logs, traces, and context in parallel, generates hypotheses with specific tool+params for follow-up |
alloy_pipeline | Create and manage Alloy data collection pipelines — 29 recipes for metrics, logs, traces, profiles from any infrastructure (databases, K8s, Docker, apps, profiling, frontend RUM) |
grafana_explore_datasourcesWhen: First step when user mentions data, metrics, or monitoring. Gets datasource UIDs needed by grafana_query, grafana_query_logs, grafana_query_traces, grafana_list_metrics, grafana_create_alert, and grafana_explain_metric.
Params: instance (optional — target Grafana instance, omit for default).
Example: {}
Example (multi-instance): { "instance": "prod" }
Returns: List of datasources with uid, name, type, isDefault, plus routing hints: queryTool (which agent tool to use, e.g. "grafana_query", "grafana_query_logs", or "grafana_query_traces"), queryLanguage (e.g. "PromQL", "LogQL", "TraceQL"), and supported (boolean — whether an agent tool can query this datasource). Use queryTool to pick the right tool for each datasource. When multiple Grafana instances are configured, also returns instance (which instance was queried) and availableInstances (list of { name, url, isDefault } for all configured instances).
grafana_list_metricsWhen: User asks "what metrics are available?" or you need to discover metrics before querying or composing dashboards. Also when grouping metrics by function — metadata mode adds category to each openclaw_* metric. Use purpose when user asks about a specific concern (e.g., "performance metrics", "cost metrics").
Params: datasourceUid (required), prefix (filter by prefix), search (targeted discovery — server-side regex, only matching metrics returned), purpose ("performance" | "cost" | "reliability" | "capacity" — pre-filter by intent, composable with prefix and search), label (list label values instead), metadata (boolean — enriched results with type/help/category), compact (boolean — with metadata, returns only name/type/category, ~60% smaller).
Example names: { "datasourceUid": "prom1", "prefix": "openclaw_lens_" }
Example search: { "datasourceUid": "prom1", "search": "steps" }
Example purpose: { "datasourceUid": "prom1", "purpose": "performance", "metadata": true }
Example combined: { "datasourceUid": "prom1", "prefix": "openclaw_ext_", "search": "fitness" }
Example metadata: { "datasourceUid": "prom1", "metadata": true, "prefix": "openclaw_" }
Example compact: { "datasourceUid": "prom1", "metadata": true, "compact": true }
Returns names: { metrics: ["metric1", "metric2", ...] }. Truncated at 200.
Returns metadata: { metadataSource, categorySummary: { cost: 3, usage: 4, session: 5, ... }, metrics: [{ name, type, help, category?, source? }, ...] }. Use this before composing custom dashboards — type tells you counter vs gauge vs histogram, category groups openclaw_* metrics by function. Search also matches help text. Categories: cost, usage, session, queue, messaging, webhook, tools, agent, custom. categorySummary gives counts per category for quick overview (omitted when no openclaw_* metrics). Purpose maps: performance → session + tools, cost → cost + usage, reliability → webhook + messaging + agent, capacity → queue + session. metadataSource: "prometheus" when Prometheus metadata endpoint has data, "synthetic" when OTLP-only (metadata synthesized from known metric registry — histogram sub-metrics deduplicated, type/help from Grafana Lens definitions). On OTLP stacks, includes hint explaining why metadata is synthetic. source: "synthetic" on individual entries from the registry; source: "custom" on entries from the custom metrics store.
Returns compact: { metadataSource, categorySummary: {...}, metrics: [{ name, type, category? }, ...] }. Same as metadata but drops help, source, labelNames — use in multi-tool chains where you need metric names and types but not full descriptions.
Example label: { "datasourceUid": "prom1", "label": "job" }
Returns label: { label, count, totalCount, values: ["value1", "value2", ...] }. Truncated at 200.
grafana_queryWhen: User asks a data question that needs a direct answer, not a dashboard. Also for re-running an existing dashboard panel's query with different time ranges.
Params: datasourceUid, expr (PromQL), queryType (instant/range), start (range only, required), end (range only, default "now"), step (range only, optional — auto-calculated from time range if omitted, targeting ~300 datapoints), dashboardUid (optional — resolve query from panel), panelId (optional — use with dashboardUid).
Example instant: { "datasourceUid": "prom1", "expr": "sum(increase(openclaw_lens_cost_by_model_total[1d])) or vector(0)" }
Example range (auto-step): { "datasourceUid": "prom1", "expr": "rate(openclaw_tokens_total[5m])", "queryType": "range", "start": "now-30d" }
Example range (explicit step): { "datasourceUid": "prom1", "expr": "rate(openclaw_tokens_total[5m])", "queryType": "range", "start": "now-1h", "end": "now", "step": "60" }
Example panel re-run: { "dashboardUid": "openclaw-command-center", "panelId": 10, "queryType": "range", "start": "now-7d" }
Tip: start/end accept Unix seconds or relative expressions like "now-1h", "now-7d". For range queries, just set start — end defaults to "now" and step is auto-calculated. Override step only when you need specific resolution.
Tip (panel re-run): Set dashboardUid + panelId to re-run a panel's query without manually extracting PromQL. The tool auto-resolves expr and datasourceUid from the panel definition. Template variables are replaced with wildcards. You can still override expr or datasourceUid explicitly if needed. Get panel IDs from grafana_get_dashboard.
Returns instant: { metrics: [{ metric: {...}, value: "1.23", timestamp: "...", healthContext?: { status, thresholds, description, direction } }], datasourceUid, resultCount, warnings?, hint? } — healthContext is included for well-known openclaw_lens_* gauge metrics, providing SRE-grade health assessment: status ("healthy"/"warning"/"critical"), thresholds (warning/critical values), description (what the metric means), direction ("higher_is_worse"/"lower_is_worse"). Omitted for unknown metrics. Capped at 50 results; when exceeded includes truncated: true, totalResults, and truncationHint advising to narrow the query.
Returns range: { series: [{ metric: {...}, values: [{ time, value }...] }], datasourceUid, resultCount, warnings?, hint? } — truncated to 20 points per series and 50 series max. When series are truncated includes truncated: true, totalSeries, and truncationHint. When step is auto-calculated, includes step: { value: "288s", display: "5m", auto: true }.
Returns (panel re-run): Includes resolvedFrom: "panel", panelTitle, panelType, templateVarsReplaced alongside normal query results. If the panel uses a Loki datasource, returns an error directing you to use grafana_query_logs instead.
Returns (warnings): When Prometheus flags a non-fatal issue (e.g., rate() on a gauge), warnings: [{ cause, suggestion, example? }] is included. Example: rate() on a gauge → cause says "rate() applied to 'metric' which appears to be a gauge", suggestion says "use delta() or deriv() instead", example shows the corrected query.
Returns (hint): When the query returns zero results, hint: { cause, suggestion } explains why (metric may not exist, label filters may not match) and suggests using grafana_list_metrics to verify.
Returns (error with guidance): On query failure, includes guidance: { cause, suggestion, example? } alongside the raw error. Pattern-matched for common PromQL mistakes: unclosed parenthesis, missing range selector, timeout, auth failure, rate on gauge, etc. Omitted when the error is unrecognized.
Tip (chaining): Both instant and range responses include datasourceUid — pass it directly to grafana_create_alert or other tools without re-calling grafana_explore_datasources. This enables zero-friction query→alert chains.
grafana_query_logsWhen: User asks about logs, errors, or needs to investigate issues by searching log data. Also for session debugging, OTel log investigation, and re-running existing log panel queries.
Params: datasourceUid, expr (LogQL), queryType (instant/range, default range), start/end (default now-1h/now), step (metric queries only), limit (default 100), direction (backward/forward), lineLimit (max chars per log line, default 500, max 2000), extractFields (boolean, default false — extract structured OTel attributes into a clean fields object), dashboardUid (optional — resolve query from panel), panelId (optional — use with dashboardUid).
Example log search: { "datasourceUid": "loki1", "expr": "{job=\"api\"} |= \"error\"" }
Example with filters: { "datasourceUid": "loki1", "expr": "{job=\"api\"} |~ \"timeout|refused\"", "limit": 50, "direction": "forward" }
Example full stack traces: { "datasourceUid": "loki1", "expr": "{job=\"api\"} |= \"Exception\"", "lineLimit": 2000 }
Example session debugging: { "datasourceUid": "loki1", "expr": "{service_name=\"openclaw\"} | json | component=\"lifecycle\"", "extractFields": true }
Example metric query: { "datasourceUid": "loki1", "expr": "rate({job=\"api\"}[5m])", "queryType": "range", "start": "now-6h", "end": "now", "step": "60" }
Example panel re-run: { "dashboardUid": "openclaw-command-center", "panelId": 18, "start": "now-24h", "extractFields": true }
Returns streams: { entries: [{ labels: {...}, timestamp: "...", line: "..." }], datasourceUid, totalEntries, truncated } — capped at 100 entries, lines at 500 chars (set lineLimit: 2000 for full stack traces).
Returns streams (extractFields): { entries: [{ labels: {...cleaned...}, timestamp: "...", line: "...", fields: { component, event_name, session_id, trace_id, model, duration_s, ... } }], datasourceUid } — infrastructure noise labels removed, openclaw_ prefix stripped from field keys, numeric values auto-converted. Also parses JSON log bodies if present.
Returns streams (traceCorrelation): When extractFields: true and entries contain trace_id, includes traceCorrelation: { traceIds: [...], tool: "grafana_query_traces", tip } — up to 5 unique trace IDs ready for grafana_query_traces with queryType: "get".
Returns metric: Same shape as grafana_query range/instant results (matrix capped at 50 series, vector capped at 50 results — includes datasourceUid, truncated, totalSeries/totalResults, and truncationHint when exceeded).
Returns (panel re-run): Includes resolvedFrom: "panel", panelTitle, panelType, templateVarsReplaced alongside normal results. If the panel uses a Prometheus datasource, returns an error directing you to use grafana_query instead.
Returns (error with guidance): On query failure, includes guidance: { cause, suggestion, example? } alongside the raw error. Pattern-matched for common LogQL mistakes: bare text without stream selector, empty {}, unclosed braces, missing label matchers, auth failure, timeout. Omitted when the error is unrecognized.
Tip: LogQL: {label="value"} selects streams, |= substring filter, |~ regex, != exclude. Metric wrappers: rate(), count_over_time(), bytes_rate(). Use extractFields: true when investigating OTel/lifecycle logs — it surfaces trace_id, session_id, event_name, model, and other attributes as first-class fields instead of buried in raw labels.
Tip (panel re-run): Same as grafana_query — set dashboardUid + panelId to auto-resolve LogQL and datasource. The tool routes Prometheus panels to grafana_query with a helpful error.
grafana_query_tracesWhen: User asks about traces, distributed tracing, slow spans, session trace hierarchies, or needs to debug request flows across services.
Params: datasourceUid, query (TraceQL expression or trace ID), queryType (search/get, default search), start/end (default now-1h/now), limit (default 20, max 50), minDuration/maxDuration (e.g., "1s", "10s"), dashboardUid (optional — resolve query from panel), panelId (optional — use with dashboardUid).
Example search: { "datasourceUid": "tempo1", "query": "{ resource.service.name = \"openclaw\" }" }
Example search slow: { "datasourceUid": "tempo1", "query": "{ resource.service.name = \"openclaw\" }", "minDuration": "5s" }
Example search with time: { "datasourceUid": "tempo1", "query": "{ span.gen_ai.system = \"anthropic\" }", "start": "now-24h", "limit": 50 }
Example get: { "datasourceUid": "tempo1", "query": "abc123def456789...", "queryType": "get" }
Example panel re-run: { "dashboardUid": "openclaw-session-explorer", "panelId": 12, "start": "now-24h" }
Returns search: { traces: [{ traceId, rootServiceName, rootTraceName, startTime, durationMs, spanCount? }], datasourceUid, totalTraces, truncated?, correlationHint? } — capped at 50 traces. When exceeded includes truncated: true and truncationHint. When traces are found, includes correlationHint: { logQuery, tool, tip } with a ready-to-use LogQL expression for grafana_query_logs.
Returns get: { traceId, spans: [{ traceId, spanId, parentSpanId?, operationName, serviceName, startTime, durationMs, status, kind?, attributes: {...} }], datasourceUid, totalSpans, truncated? } — flattened OTLP spans with resolved attributes (string/number/boolean). Capped at 200 spans. Sorted by start time (earliest first).
Returns (panel re-run): Includes resolvedFrom: "panel", panelTitle, panelType, templateVarsReplaced alongside normal results. If the panel uses a Prometheus or Loki datasource, returns an error directing you to use the correct tool.
Returns (error with guidance): On query failure, includes guidance: { cause, suggestion, example? } alongside the raw error. Pattern-matched for common TraceQL mistakes: syntax errors, invalid attributes, auth failure, timeout, not-found, invalid trace ID. Omitted when the error is unrecognized.
Returns (no results): When search returns zero traces, includes hint: { cause, suggestion } suggesting to broaden the query or check the datasource.
Tip: TraceQL: { } matches all traces, resource.service.name for service filter, span.http.status_code for HTTP spans, name for operation name, duration for span duration, status for error/ok filtering. Use minDuration/maxDuration to find performance outliers. Trace-to-Log: search and get results include correlationHint.logQuery — pass it directly to grafana_query_logs to find correlated logs. Log-to-Trace: grafana_query_logs results (with extractFields: true) include traceCorrelation.traceIds — pass any ID to grafana_query_traces with queryType: "get".
Tip (panel re-run): Same as grafana_query — set dashboardUid + panelId to auto-resolve TraceQL and datasource. The tool routes Prometheus/Loki panels to the correct tool with a helpful error.
grafana_create_dashboardWhen: User wants a persistent dashboard for ongoing monitoring.
Params: template or dashboard (custom JSON) — one required. Optional: title (overrides template default), folderUid (target folder), overwrite (default true).
Returns: { uid, url, status, message, suggestedNext?: [{ template, reason }], validation?: DashboardValidation }. For template-based dashboards, suggestedNext lists complementary templates to deploy next. For custom JSON dashboards, validation dry-runs each panel's PromQL and reports per-panel health — check validation.panelsError for broken queries.
Choose the right template (3-tier SRE drill-down hierarchy):
Tier 1 → System: Start here for overall health. Tier 2 → Session: Click a session from Tier 1 to investigate. Tier 3 → Deep Dive: Cost, tool, or SRE details.
| Template | Tier | Domain | Variables | Use When |
|---|---|---|---|---|
llm-command-center | Tier 1 | System overview | $prometheus, $loki, $tempo, $provider, $model, $channel | Golden signals, session table with click-to-drill-down, cost, cache, live feeds |
session-explorer | Tier 2 | Session debug | $prometheus, $loki, $tempo, $session (textbox) | Per-session trace hierarchy, LLM calls, tool calls, conversation flow |
cost-intelligence | Tier 3a | Cost analysis | $prometheus, $loki, $provider, $model | Spending trends, model attribution, cache savings, per-session cost table |
tool-performance | Tier 3b | Tool analytics | $prometheus, $loki, $tempo, $tool | Tool leaderboard, latency ranking, error rates, tool traces |
sre-operations | Tier 3c | SRE operations | $prometheus, $loki | Queue health, webhooks, stuck sessions, tool loops |
genai-observability | — | OTel gen_ai standard | $prometheus, $loki, $tempo, $model, $provider | Industry-standard AI monitoring: token analytics, LLM performance, traces, logs, cache efficiency. Works with any gen_ai data. |
node-exporter | — | System/DevOps | $datasource, $instance | Server CPU, memory, disk, network |
http-service | — | Web/DevOps | $datasource, $job | HTTP request rate, errors, latency (RED signals) |
metric-explorer | — | Any domain | $datasource, $metric | Deep-dive into any single metric from a dropdown |
multi-kpi | — | Any domain | $datasource, $metric1..$metric4 | 4-metric KPI overview (business, fitness, finance, IoT) |
weekly-review | — | Any domain | $datasource, $metric1, $metric2 | Weekly overview of 2 external metrics with trends + all openclaw_ext_* table |
All AI templates have Loki log-to-trace correlation via Tempo + stable UIDs for cross-dashboard navigation.
Example AI health: { "template": "llm-command-center", "title": "My AI Dashboard" }
Example session debug: { "template": "session-explorer", "title": "Session Debug" }
Example cost analysis: { "template": "cost-intelligence", "title": "My AI Costs" }
Example tool analytics: { "template": "tool-performance", "title": "Tool Health" }
Example SRE ops: { "template": "sre-operations", "title": "SRE Health" }
Example GenAI observability: { "template": "genai-observability", "title": "GenAI Observability" }
Example system: { "template": "node-exporter", "title": "Server Health" }
Example generic: { "template": "metric-explorer", "title": "Explore My Data" }
Example multi-KPI: { "template": "multi-kpi", "title": "Business KPIs" }
Example weekly review: { "template": "weekly-review", "title": "My Weekly Review" }
Example custom with validation: { "dashboard": { "title": "Model Comparison", "panels": [{ "id": 1, "title": "Cost by Model", "type": "timeseries", "targets": [{ "refId": "A", "expr": "sum by (model) (rate(openclaw_lens_cost_by_token_type[1h]))", "datasource": { "uid": "prometheus" } }] }] } }
Custom dashboard validation (returned only for dashboard param, not templates):
validation: { panelsTotal: 3, panelsValid: 1, panelsNoData: 1, panelsError: 1, panelsSkipped: 0, details: [{ panelId: 1, title: "Cost by Model", status: "ok", queries: [{ refId: "A", expr: "...", valid: true, sampleValue: 0.42 }] }, { panelId: 2, title: "Latency", status: "nodata" }, { panelId: 3, title: "Bad Query", status: "error", error: "parse error at char 5" }] }
Panel statuses: ok (query returned data), nodata (valid query, no results — metric may not exist yet), error (PromQL syntax error or datasource issue), skipped (no datasource UID found). Dashboard is always created regardless — validation is informational.
grafana_update_dashboardWhen: User wants to add a panel, remove a panel, change a query, update dashboard settings, or delete a dashboard.
Params: uid (required), operation (required: add_panel, remove_panel, update_panel, update_metadata, delete).
add_panel params: panel (object with title, type, targets). Auto-layouts below existing panels.
remove_panel / update_panel params: panelId (preferred) or panelTitle (case-insensitive substring fallback). updates (object) for update_panel.
update_metadata params: title, description, tags, time (e.g., { "from": "now-7d", "to": "now" }), refresh (e.g., "1m").
delete params: None besides uid — permanently removes the dashboard. Always confirm with user first.
Example add: { "uid": "abc123", "operation": "add_panel", "panel": { "title": "Error Rate", "type": "timeseries", "targets": [{ "refId": "A", "expr": "rate(errors_total[5m])", "datasource": { "uid": "prom1" } }] } }
Example add (no datasource): { "uid": "abc123", "operation": "add_panel", "panel": { "title": "Latency", "type": "timeseries", "targets": [{ "refId": "A", "expr": "histogram_quantile(0.99, rate(http_duration_bucket[5m]))" }] } } — validation skipped if no datasource UID found, panel still saved.
Example remove: { "uid": "abc123", "operation": "remove_panel", "panelId": 3 }
Example update panel: { "uid": "abc123", "operation": "update_panel", "panelId": 1, "updates": { "title": "New Title", "targets": [{ "refId": "A", "expr": "new_query" }] } }
Example update metadata: { "uid": "abc123", "operation": "update_metadata", "title": "My Dashboard v2", "time": { "from": "now-7d", "to": "now" }, "refresh": "5m" }
Example delete: { "uid": "abc123", "operation": "delete" }
Returns update: { status: "updated", uid, url, version, operation, panelCount, affectedPanel?: { id, title }, changedFields?: [...], queryValidation?: { validated, results, datasourceUid?, skippedReason? } }.
Returns queryValidation: For add_panel and update_panel (when targets change), PromQL queries are dry-run against Grafana. Each result: { refId, expr, valid: boolean, error?: string, sampleValue?: number }. Panel is always saved — validation is informational. If valid: false, check the error field for PromQL syntax issues. If skippedReason is set, no datasource UID was found — include datasource: { uid: "..." } on targets to enable validation.
Returns delete: { status: "deleted", uid, title, message }.
Tip: targets in update_panel replaces entirely — include all targets, not just changed ones. Include datasource.uid on targets for query validation feedback.
grafana_get_dashboardWhen: Need to inspect a dashboard's panels — find panel IDs for sharing, verify structure, scan multiple dashboards for an overview, or audit which panels are returning data.
Params: uid (required). Optional: compact (boolean, default false) — return panel titles and types only, no queries or metadata (~70% smaller). audit (boolean, default false) — dry-run each panel's query and add health status.
Example (full): { "uid": "abc123" }
Example (compact overview): { "uid": "abc123", "compact": true }
Example (audit): { "uid": "abc123", "audit": true }
Returns (full): { uid, title, description?, url, tags, time?, refresh?, panelCount, panels: [{ id, title, type, queries: [{ refId, expr }] }], folderUid, created?, updated? }.
Returns (compact): { uid, title, url, tags, panelCount, panels: [{ id, title, type }] }.
Returns (audit): Same as full, plus each panel gets health: { status: "ok"|"nodata"|"error"|"skipped", error?, sampleValue? } and the response includes auditSummary: { ok, nodata, error, skipped }. Resolves template variable datasources ($prometheus, $loki) and replaces expression template vars with wildcards.
Tip: Use audit: true when the user asks "which panels are broken?" or "audit my dashboard" — it replaces N separate grafana_query calls with one tool call. Use compact: true for lightweight overview scans. Omit both when you need query details (before update or share).
grafana_searchWhen: User mentions a dashboard by name, before creating one (check duplicates), or for reporting/audit workflows.
Params: query (required). Optional: tags (array — filter by tags), starred (boolean — only starred), sort ("alpha-asc"/"alpha-desc"), limit (number, default 100), enrich (boolean — add updatedAt + panelCount per result, default false).
Example: { "query": "cost" }
Example with tags: { "query": "", "tags": ["production"] }
Example starred: { "query": "", "starred": true, "limit": 10 }
Example enriched: { "query": "", "enrich": true }
Returns: { count, enriched, dashboards: [{ uid, title, url, tags, folderTitle?, folderUid?, updatedAt?, panelCount? }] }. folderTitle/folderUid always included when dashboard is in a folder. updatedAt (ISO 8601) and panelCount only present when enrich: true — enables staleness detection and reporting without per-dashboard get_dashboard calls.
Tip: Use enrich: true for reporting workflows ("which dashboards are stale?", "give me a summary of all dashboards"). Skip enrichment for simple lookups. After finding a dashboard, use grafana_get_dashboard to inspect panels, grafana_share_dashboard to render a chart, or grafana_update_dashboard to modify it.
grafana_share_dashboardWhen: User says "show me" or "send me" a chart/dashboard.
Params: dashboardUid, panelId (required). Optional: from (default "now-6h"), to (default "now"), width (default 1000), height (default 500), theme ("light"/"dark", default "dark").
Example: { "dashboardUid": "abc123", "panelId": 2, "from": "now-6h", "to": "now" }
Returns: Image rendered inline (tier 1), or snapshot URL (tier 2), or deep link (tier 3). Always delivers something. Includes deliveryTier ("image" | "snapshot" | "link"), rendererAvailable (boolean — false when Image Renderer plugin is missing), renderFailureReason (why image rendering failed), and remediation (how to fix it). Tier 3 also includes snapshotFailureReason.
Tip: Use grafana_get_dashboard first to find panel IDs. If rendererAvailable is false, tell the user to install the grafana-image-renderer plugin.
grafana_create_alertWhen: User wants notifications when a metric crosses a threshold.
Params: title, datasourceUid, expr (PromQL), threshold (all required). Optional: evaluation ("instant"/"rate"/"increase", default "instant"), evaluationWindow (default "5m", used with rate/increase), condition (gt/lt/gte/lte, default gt), for (duration, default 5m), folderUid, labels (e.g., { "severity": "warning" }), annotations (e.g., { "summary": "Cost too high" }), noDataState (NoData/Alerting/OK, default NoData).
IMPORTANT: For counter metrics (*_total), always use evaluation: "rate" (per-second rate) or evaluation: "increase" (total change over window). Raw counter values always increase and will immediately breach any threshold. Use "instant" (default) only for gauges.
Example gauge alert: { "title": "High Cost Alert", "datasourceUid": "prom1", "expr": "openclaw_lens_daily_cost_usd", "threshold": 5, "condition": "gt" }
Example rate alert: { "title": "High Error Rate", "datasourceUid": "prom1", "expr": "openclaw_lens_webhook_error_total", "threshold": 0.1, "evaluation": "rate" }
Example increase alert: { "title": "Token Burst", "datasourceUid": "prom1", "expr": "openclaw_lens_tokens_total", "threshold": 10000, "evaluation": "increase", "evaluationWindow": "1h" }
Returns: { uid, title, status: "created", datasourceUid, url, evaluation?: { mode, window, evaluatedExpr }, metricValidation: { valid, error?, sampleValue? }, message }. The datasourceUid echoes back which datasource the rule targets (verify correctness). metricValidation dry-runs the expression before creation — valid: true + sampleValue confirms data exists; valid: false + error warns of typos/missing metrics. Alert is always created regardless (metric may not have data yet). When evaluation is "rate" or "increase", validation runs the wrapped expression.
Note: Auto-creates a "Grafana Lens Alerts" folder if no folderUid is specified.
grafana_annotateWhen: User deploys, changes config, or wants to mark an event for correlation.
Params: action ("create" default, or "list").
Create params: text (required), tags, dashboardUid, panelId, time (epoch ms or relative like "now-2h", default now), timeEnd (epoch ms or relative).
List params: from, to (epoch ms or relative like "now-7d", "now-24h", "now"), tags, limit (default 20).
Time formats: All time params accept epoch ms (e.g., 1700000000000) OR Grafana-style relative strings ("now", "now-1h", "now-7d", "now-30m"). Prefer relative strings — they're simpler and avoid arithmetic errors.
Example create: { "text": "Deployed v2.1.0", "tags": ["deploy", "production"] }
Example create past: { "text": "Incident started", "time": "now-2h", "timeEnd": "now-30m", "tags": ["incident"] }
Example list recent: { "action": "list", "from": "now-7d", "to": "now", "tags": ["deploy"] }
Example list: { "action": "list", "tags": ["deploy"], "limit": 10 }
Returns create: { status: "created", id, message, time, comparisonHint: { beforeWindow: { from, to }, afterWindow: { from, to }, suggestion } }. The comparisonHint provides ready-to-use ISO 8601 time ranges (30-min windows) for before/after comparison via grafana_query — no manual time math needed. For region annotations (with timeEnd), afterWindow starts at timeEnd.
Returns list: { annotations: [{ id, text, tags, time, timeEnd?, dashboardUID?, panelId? }] }.
grafana_check_alertsWhen: Prompt context shows "GRAFANA ALERTS", need to manage alert rules (list/delete), set up the alert webhook, silence alerts during investigation, or acknowledge an investigated alert.
Params: action ("list" default, "acknowledge", "list_rules", "delete_rule", "silence", "unsilence", "setup").
List params: None — returns all pending (unacknowledged) alerts. Instances capped at 5 per alert.
Acknowledge params: alertId (required) — marks an alert as investigated.
List rules params: compact (boolean, default false — returns only uid/title/state/condition). Full mode returns all configured alert rules from Grafana with UID, title, condition (PromQL), folder, labels, annotations, AND live evaluation state (normal/firing/pending/nodata/error), health, and lastEvaluation. One call gives the complete alert health picture.
Delete rule params: ruleUid (required) — permanently deletes an alert rule. Get UIDs from list_rules.
Silence params: matchers (required — array of { name, value, isRegex? } from alert's commonLabels), duration (default "2h"), comment (optional).
Unsilence params: silenceId (required) — removes a silence so alerts resume notifying.
Setup params: webhookUrl (optional, auto-detected) — creates webhook contact point and notification policy route in Grafana.
Example list: {}
Example acknowledge: { "action": "acknowledge", "alertId": "alert-1" }
Example list rules: { "action": "list_rules" }
Example list rules compact: { "action": "list_rules", "compact": true }
Example delete rule: { "action": "delete_rule", "ruleUid": "abc123-def456" }
Example silence: { "action": "silence", "matchers": [{ "name": "alertname", "value": "HighCost" }], "duration": "2h", "comment": "Investigating cost spike" }
Example unsilence: { "action": "unsilence", "silenceId": "silence-uuid-123" }
Example setup: { "action": "setup" }
Returns list: { status: "success", alertCount, alerts: [{ id, status, title, message, receivedAt, commonLabels, totalInstances, truncated?, suggestedInvestigation?: { datasourceUid, condition, tool, queryLanguage, hint }, instances: [{ status, labels, annotations, startsAt, values }] }] }. suggestedInvestigation is auto-enriched by matching the alert to its rule — provides the PromQL/LogQL expression, datasource, and tool to use for immediate investigation (eliminates the need for separate list_rules + explore_datasources calls).
Returns acknowledge: { status: "acknowledged", alertId }.
Returns list_rules: { status: "success", ruleCount, rules: [{ uid, title, folder, ruleGroup, state, health, lastEvaluation, for, labels, annotations, condition, updated }] }. state is the live evaluation state: "normal" (not firing), "firing", "pending" (within for duration), "nodata", or "error". Falls back to "unknown" if the eval state API is unavailable. health is "ok", "nodata", "error", or "unknown". condition is the extracted PromQL expression from the rule's data queries.
Returns list_rules (compact): { status: "success", ruleCount, rules: [{ uid, title, state, condition }] }. Minimal fields for multi-tool chains — use when you need a quick overview of all rules without details.
Returns delete_rule: { status: "deleted", ruleUid, message }.
Returns silence: { status: "silenced", silenceId, duration, matchers, message }.
Returns unsilence: { status: "unsilenced", silenceId, message }.
Returns setup: { status: "created", contactPointUid, webhookUrl } or { status: "already_exists", contactPointUid }.
Note: Setup is idempotent — safe to call multiple times. Only alerts with managed_by=openclaw label route to the webhook (auto-added by grafana_create_alert). Use list_rules → delete_rule for full alert lifecycle management (create via grafana_create_alert, list/delete via grafana_check_alerts).
grafana_push_metricsWhen: User wants to track custom data (calendar events, git commits, fitness stats, financial data) in Grafana.
Params: action ("push" default, "register", "list", "delete").
Push params: metrics (required array) — each: { name, value, labels?, type?, help?, timestamp? }. Names auto-get openclaw_ext_ prefix. timestamp is optional ISO 8601 for historical data (gauge only).
Register params: name (required), type ("gauge"/"counter", default "gauge"), help, labelNames (array), ttlDays.
List params: None — returns all custom metric definitions.
Delete params: name (required) — removes a custom metric.
Example push: { "metrics": [{ "name": "steps_today", "value": 8000 }, { "name": "meetings", "value": 3, "labels": { "type": "standup" } }] }
Example backfill: { "metrics": [{ "name": "steps", "value": 8000, "timestamp": "2025-01-15" }, { "name": "steps", "value": 10500, "timestamp": "2025-01-16" }] }
Example mixed: { "metrics": [{ "name": "steps", "value": 9000, "timestamp": "2025-01-17" }, { "name": "heart_rate", "value": 72 }] }
Example register: { "action": "register", "name": "weight_kg", "type": "gauge", "help": "Body weight", "labelNames": ["person"], "ttlDays": 90 }
Example list: { "action": "list" }
Example delete: { "action": "delete", "name": "old_metric" }
Returns push: { status: "ok", accepted: 2, queryNames: { "openclaw_ext_steps": "openclaw_ext_steps", "openclaw_ext_events": "openclaw_ext_events_total" }, suggestedWorkflow: [{ tool, action, example }], message: "..." }. suggestedWorkflow contains concrete next-step examples using the actual pushed metric names — verify (grafana_query), visualize (grafana_create_dashboard with metric-explorer template), and alert (grafana_create_alert, single-metric only). Partial success supported. Timestamped and real-time points in the same batch are both accepted.
Returns register: { status: "registered", metric: { name, type, help, labelNames, ttlMs }, queryName: "openclaw_ext_events_total", suggestedWorkflow: [{ tool, action, example }] }. suggestedWorkflow shows how to push data and query the registered metric (with rate() wrapping for counters).
Returns list: { count, metrics: [{ name, type, queryName, help, labelNames, createdAt, updatedAt }] }.
Returns delete: { status: "deleted", name }.
Note: Push auto-registers unknown metrics. Response includes queryNames with exact PromQL names and suggestedWorkflow with concrete next steps. Follow suggestedWorkflow to complete the push→visualize pipeline. Timestamped pushes are gauge-only — counters with timestamps are rejected. See external-data.md for naming conventions and backfill patterns.
grafana_explain_metricWhen: User asks "what does this metric mean?", "why did it spike?", "is this normal?", or "show me the trend".
Params: datasourceUid (required), expr (PromQL or plain metric name, required), period (24h/7d/30d, default 24h), compareWith ("previous" — compare current period with the same-length window immediately before it).
Example: { "datasourceUid": "prom1", "expr": "openclaw_lens_daily_cost_usd" }
Example counter: { "datasourceUid": "prom1", "expr": "openclaw_lens_tokens_total" }
Example 7d: { "datasourceUid": "prom1", "expr": "openclaw_lens_daily_cost_usd", "period": "7d" }
Example comparison: { "datasourceUid": "prom1", "expr": "openclaw_lens_daily_cost_usd", "period": "7d", "compareWith": "previous" }
Example PromQL: { "datasourceUid": "prom1", "expr": "rate(http_requests_total[5m])", "period": "24h" }
Returns: { metricType?, trendQuery?, current: { value, timestamp }, healthContext?: { status, thresholds, description, direction }, trend: { changePercent, direction, first, last }, stats: { min, max, avg, samples }, comparison?: { previousPeriod: { from, to, avg, min, max, samples }, change: { absolute, percentage, direction } }, metadata: { type, help, unit }, suggestedQueries?: [{ query, description }], suggestedBreakdowns?: string[] }. Sections omitted when data unavailable. changePercent is null when first value is zero. healthContext is included for well-known openclaw_lens_* gauge metrics — same as grafana_query.
Counter-aware: Auto-detects counter metrics (via metadata type or _total suffix) and wraps the trend query in rate(expr[5m]). The current value stays raw (cumulative total), but trend and stats show rate of change. metricType field tells you the detected type (counter/gauge/histogram). trendQuery shows the actual PromQL used for trend (only present when different from expr).
Drill-down: For multi-dimensional metrics (metrics with labels like model, token_type, provider), the response includes suggestedQueries — ready-to-use PromQL queries for grafana_query that break down the metric by each label. Counter metrics get rate() wrapping automatically. Use these to investigate cost attribution, identify top contributors, or decompose aggregates.
Breakdowns: suggestedBreakdowns provides label names for decomposition — always available for known OpenClaw metrics (cost, session, queue, webhook families) even when the metric has no data yet. For unknown metrics, falls back to labels discovered from the instant query. Use these labels with grafana_query to build sum by (label) (...) queries for root-cause analysis.
Period comparison: Use compareWith: "previous" for period-over-period analysis (e.g., this week vs. last week). Returns a comparison object with the previous period's stats and the change (absolute, percentage, direction). Works with counters too (compares rates). Eliminates the need for manual multi-query workflows.
Tip: For simple trend context, call with just period. For "did things improve?" questions, add compareWith: "previous". Metadata only available for plain metric names (not complex PromQL). No need to manually wrap counters in rate() — the tool does it automatically.
grafana_security_checkWhen: User asks "am I being attacked?", "security status", "security audit", "security check", or wants a comprehensive threat assessment.
Params: lookback (time window, default "1h". Use "24h" for daily review, "7d" for weekly).
Example: {}
Example weekly review: { "lookback": "7d" }
Returns: { overallThreatLevel: "green"|"yellow"|"red", summary, checks: [{ name, status, value, detail }], securityEventLogs, limitations: [...], suggestedActions: [...], dashboardTemplate: "security-overview" }.
Checks (6 parallel, via Promise.allSettled — partial failures return partial results):
webhook_error_ratio — webhook error rate vs received rate. Warning 20%, critical 50%.cost_anomaly — daily cost. Warning $10, critical $50.tool_loops — active tool loops. Warning 1, critical 3.injection_signals — prompt injection patterns detected. Warning 1, critical 5.session_enumeration — unique sessions in 1h. Warning 50, critical 200.stuck_sessions — stuck sessions. Warning 1, critical 3.
Limitations: Auth failures are NOT observable — openclaw's auth middleware emits no diagnostic events. The tool monitors observable signals (webhook errors, cost spikes, prompt injection patterns, session anomalies) but cannot detect silent auth-layer attacks (bad tokens, brute-force, rate-limiter lockouts). Always includes limitations in response.
Auto-discovery: No datasource UID needed — auto-discovers first Prometheus datasource. Loki is optional (skips log check if absent).
Follow-up: Use dashboardTemplate to create a persistent security dashboard. Use suggestedActions for investigation steps.grafana_investigateWhen: First step for "investigate this", "what's wrong?", "triage", "root cause analysis", "debug this issue", or any multi-signal investigation.
Params: focus (required — alert UID, metric name, or symptom text), timeWindow (optional, default "1h", options: "1h", "6h", "24h"), service (optional, default "openclaw").
Example: { "focus": "high error rate" }
Example with alert: { "focus": "alert-abc", "timeWindow": "6h" }
Example with metric: { "focus": "openclaw_lens_daily_cost_usd", "timeWindow": "24h" }
Returns: { timeWindow, focus, metricSignals: { focus: { current, trend, anomalyScore, anomalySeverity }, red: { rate, errorRate, p95Latency } }, logSignals: { totalVolume, errorCount, bySeverity, topPatterns, sampleErrors }, traceSignals: { errorTraces, slowTraces }, contextSignals: { recentAnnotations, alertsActive }, suggestedHypotheses: [{ hypothesis, evidence, confidence, testWith: { tool, params } }], limitations }.
Auto-discovery: No datasource UID needed — auto-discovers Prometheus, Loki, and Tempo datasources. Loki and Tempo are optional (graceful degradation with limitations noted).
Follow-up: Use suggestedHypotheses[].testWith for deep-dives with specific tools. Use grafana_annotate to mark findings. Use grafana_check_alerts to acknowledge investigated alerts.
alloy_pipelineWhen: Trigger whenever the user mentions data collection, metrics pipeline, log forwarding, trace collection, infrastructure monitoring, database monitoring, observability setup, scraping endpoints, collecting Docker/container logs, Kubernetes monitoring, syslog, or any mention of getting data into Grafana. Also trigger for managing existing pipelines (status, health, diagnostics). 26 recipes covering metrics (11), logs (8), traces (4), profiles (1), and infrastructure (3). This is the bridge between "I have a data source" and "I can see it in Grafana" — if the user has data somewhere and wants it in Grafana, this tool is the answer.
Actions: create (deploy pipeline from recipe), list (show managed pipelines), update (change params), delete (remove pipeline), recipes (browse catalog), status (check health), diagnose (Alloy connectivity + all pipelines).
Params (create with recipe): recipe (required), params (recipe-specific), name (optional — auto-generated from recipe name with numeric suffix for uniqueness, e.g., syslog, syslog-2, syslog-3; always read the name field from the response to get the actual assigned name). Log recipes also accept optional processing params: jsonExpressions (object — JSON path extractions), labelFields (object — promote fields to labels), structuredMetadata (object — Loki structured metadata), staticLabels (object), timestampSource (string), outputSource (string), regexExpression (string).
Params (create with raw config): config (raw Alloy River syntax), name (required), signal (optional — "metrics", "logs", "traces", or "profiles", auto-detected from component IDs if omitted), sampleQueries (optional — object with metrics/logs/traces keys containing ready-to-use queries). Response includes exportTargets showing where data is being sent.
Params (update): name (required), params (fields to change — merges with existing). Raw-config pipelines support update via full config replacement — pass config param with the complete new config. Recipe-based pipelines use params for partial updates (merges with existing). Response includes sampleQueries and suggestedWorkflow for chaining into query/dashboard tools.
Params (status): name (required).
Params (recipes): category (optional filter).
Example create (scrape): { "recipe": "scrape-endpoint", "params": { "url": "http://myapp:8080/metrics" } }
Example create (database): { "recipe": "postgres-exporter", "params": { "connectionString": "postgres://user:pass@db:5432/mydb" }, "name": "analytics-db" }
Example create (logs): { "recipe": "docker-logs", "params": { "containerNames": ["myapp", "nginx"] } }
Example create (files): { "recipe": "file-logs", "params": { "paths": ["/var/log/app/*.log"] } }
Example create (K8s): { "recipe": "kubernetes-pods", "params": { "namespaces": ["production"] } }
Example create (OTLP): { "recipe": "otlp-receiver" }
Example create (traces with sampling): { "recipe": "application-traces", "params": { "environment": "staging", "sampleRate": 0.5 } }
Example create (syslog UDP): { "recipe": "syslog", "params": { "protocol": "udp", "listenAddress": "0.0.0.0:5514" } }
Example create (raw config): { "config": "prometheus.scrape ...", "name": "custom-pipeline", "signal": "metrics" }
Example recipes: { "action": "recipes", "category": "logs" }
Example status: { "action": "status", "name": "analytics-db" }
Example diagnose: { "action": "diagnose" }
Returns (create): { status, name, recipe, signal, configFile, reloaded, envVarsRequired, warnings, sampleQueries: { metrics|logs|traces: {...} }, suggestedWorkflow }. status is "created" (active) or "pending_credentials" (env vars not set yet). sampleQueries are ready-to-use PromQL/LogQL/TraceQL — pass directly to query tools. envVarsRequired lists env vars for credential-bearing recipes. warnings contains advisory notes about unrecognized params. Raw-config pipelines may also include credentialWarnings if plaintext credentials were detected.
Returns (recipes): { categories, recipes: [{ name, category, signal, summary, requiredParams, optionalParams, hasCredentials, dashboardTemplate }] }. Use summary to match user intent.
Returns (status): { name, status, components: [{ id, health, detail }], dataVerification: { verifyQuery, tool }, remediation }. Pending pipelines auto-promote to active when components become healthy.
Returns (diagnose): { alloyConnectivity, pipelineHealth, driftDetected, orphanFiles, limits }.
Tip (credential lifecycle): Credential recipes (postgres-exporter, mysql-exporter, mongodb-exporter, redis-exporter, scrape-endpoint with auth, kafka-logs) may return pending_credentials status when env vars aren't set. Config stays on disk — tell user the exact env var names from envVarsRequired, then verify with action status. Pipeline auto-activates once Alloy can connect.
Tip (chaining): sampleQueries → grafana_query/grafana_query_logs. dashboardTemplate → grafana_create_dashboard. Zero-friction pipeline→query→dashboard→alert chains.
Tip (multi-pipeline): For K8s: kubernetes-pods + kubernetes-logs + otlp-receiver (3 separate creates).
Tip (raw config escape hatch): Use config param for patterns not covered by recipes. Signal is auto-detected from component IDs (or set signal explicitly). Add sampleQueries so chaining tools know what to query. Component IDs are auto-extracted for health checking. Supports update via full config replacement. See references/alloy-components.md for copy-pasteable component snippets.
Tip (port conflicts): Trace recipes (otlp-receiver, application-traces, span-metrics, service-graph) default to ports 4317/4318 — creating multiple on the same ports will fail. Set grpcPort/httpPort params to different values for each.
Tip (cross-pipeline isolation): Pipelines are self-contained — each .alloy file has no cross-file references. For combined patterns (e.g., OTLP receiver + span-metrics + service-graph in one config), use raw config param with the full combined pipeline. You cannot wire one pipeline's output into another pipeline.
Honest limitations: OpenClaw's auth middleware does not emit diagnostic events for failed authentication attempts (bad tokens, brute-force, rate-limiter lockouts). Gateway-level auth failures are invisible to Grafana Lens. The security-check tool monitors observable signals (webhook errors, cost spikes, prompt injection patterns, session anomalies) but cannot detect silent auth-layer attacks.
Security PromQL queries:
rate(openclaw_lens_webhook_error_total[5m]) > 0.5rate(openclaw_lens_webhook_error_total[5m]) / (rate(openclaw_lens_webhook_received_total[5m]) + 0.001) > 0.3sum(openclaw_lens_tool_loops_active) > 0sum(increase(openclaw_lens_prompt_injection_signals_total[1h])) > 0openclaw_lens_daily_cost_usd > 10openclaw_lens_unique_sessions_1h > 50sum(increase(openclaw_lens_gateway_restarts_total[24h])) > 3sum by (error_class) (rate(openclaw_lens_tool_error_classes_total[5m]))Security LogQL queries:
{service_name="openclaw"} | json | event_name=~"prompt_injection.detected|gateway.start|gateway.stop"{service_name="openclaw"} | json | component="diagnostic" | event_name="tool.loop"{service_name="openclaw"} | json | event_name="session.reset""How's my agent doing?"
grafana_query with sum(increase(openclaw_lens_cost_by_model_total[1d])) or vector(0) and openclaw_lens_tokens_total for quick numbersgrafana_create_dashboard with llm-command-center template + grafana_share_dashboard for a visual"Set up monitoring for my agent" / "What dashboards should I create?"
grafana_search query: "LLM Command Center" — check for existing agent dashboardsgrafana_create_dashboard template: "llm-command-center" — single-pane health: cost, sessions, errors, latency, cache, live feedssuggestedNext from the response — it chains you through the remaining templates (session-explorer → cost-intelligence → tool-performance → sre-operations → genai-observability)grafana_share_dashboard to show a key panel inlineThe 6 AI observability templates provide comprehensive OpenClaw monitoring with Loki log-to-trace correlation:
"Find error logs in the last hour"
grafana_explore_datasources — find Loki datasource UIDgrafana_query_logs with {job="api"} |= "error""Why did my service crash?"
grafana_explore_datasources — find Loki UIDgrafana_query_logs with {job="myservice"} |~ "fatal|panic|crash"grafana_query — check related metrics around the same time window"Investigate alert: correlate metrics with logs"
grafana_check_alerts — get alert details with suggestedInvestigation (includes ready-to-use query, datasource, and tool)grafana_query / grafana_query_logs — use suggestedInvestigation.tool with suggestedInvestigation.condition and suggestedInvestigation.datasourceUidgrafana_query_logs — search logs around alert time for root cause (if PromQL alert)grafana_annotate — mark findings on dashboardgrafana_check_alerts with action: "acknowledge""What data do I have in Grafana?"
grafana_explore_datasources — find Prometheus datasourcesgrafana_list_metrics with datasourceUid and metadata: true — list available metrics grouped by categorycategorySummary for quick overview (e.g., "5 cost metrics, 8 session metrics, 3 queue metrics")"My agent feels slow — what performance metrics can I look at?"
grafana_explore_datasources — get Prometheus UIDgrafana_list_metrics with purpose: "performance", metadata: true — returns only session + tool metrics (latency, duration, tool calls)grafana_explain_metric on key metrics (e.g., openclaw_lens_session_latency_avg_ms, openclaw_lens_tool_duration_ms) — get current values and trends"Alert me if costs exceed $10/day"
grafana_explore_datasources — get Prometheus UIDgrafana_create_alert with expr: "openclaw_lens_daily_cost_usd", threshold: 10, condition: "gt""Query error rate and alert if it's above 1%" (query→alert chaining)
grafana_query with expr: "rate(http_errors_total[5m])" — response includes datasourceUidgrafana_create_alert with datasourceUid from the query response, same expr, threshold: 0.01, condition: "gt" — no extra datasource discovery needed"Show me my cost trends"
grafana_search with query: "cost" — check for existing dashboardgrafana_create_dashboard with cost-intelligence templategrafana_get_dashboard — find the right panel IDgrafana_share_dashboard — deliver the chart inline"Re-run the latency panel from my Command Center for the last 7 days"
grafana_search with query: "Command Center" — get the dashboard UIDgrafana_get_dashboard with uid — find the latency panel IDgrafana_query with dashboardUid, panelId, queryType: "range", start: "now-7d" — auto-resolves PromQL and datasource from the panel"Mark that I deployed version 2"
grafana_annotate with text: "Deployed v2", tags: ["deploy"]"I deployed v2.3.0 — compare error rates before and after"
grafana_annotate with text: "Deployed v2.3.0", tags: ["deploy"] — response includes comparisonHint with before/after time windowsgrafana_query with comparisonHint.beforeWindow time range — get pre-deploy error rategrafana_query with comparisonHint.afterWindow time range — get post-deploy error rate"What happened around 3pm?"
grafana_annotate with action: "list", from/to relative times (e.g., "now-6h") or epoch ms, or specific tags"Monitor my server's health"
grafana_explore_datasources — get Prometheus UIDgrafana_search with query: "node" — check for existinggrafana_create_dashboard with template: "node-exporter" — user picks instance from dropdown"Show me my e-commerce metrics"
grafana_explore_datasources — get Prometheus UIDgrafana_list_metrics with prefix: "orders_" — discover available metricsgrafana_create_dashboard with template: "multi-kpi" — user picks 4 business metrics from dropdowns"I want to explore my fitness data"
grafana_explore_datasources — get Prometheus UIDgrafana_create_dashboard with template: "metric-explorer" — user browses all fitness_* metrics from dropdown"Set up HTTP API monitoring with alerts"
grafana_explore_datasources — get Prometheus UIDgrafana_create_dashboard with template: "http-service"grafana_create_alert with expr: "http_requests_total{code=~\"5..\"}", threshold: 0.1, evaluation: "rate" — the tool wraps in rate() automatically"Build a custom Redis dashboard"
grafana_explore_datasources — get Prometheus UIDgrafana_list_metrics with metadata: true, prefix: "redis_" — learn metric typesdashboard JSON using panel snippets from the composition guidegrafana_create_dashboard with the custom JSON — check validation.panelsError for broken queries, fix any errors with grafana_update_dashboard"Set up full alert monitoring loop"
grafana_explore_datasources — get Prometheus UIDgrafana_check_alerts with action: "setup" — create webhook contact pointgrafana_create_alert with PromQL condition — rule auto-routes to webhook via managed_by=openclaw label"List all my alert rules and delete one"
grafana_check_alerts with action: "list_rules" — get all configured rules with UIDs, titles, conditions, AND live eval state (normal/firing/nodata/error)grafana_check_alerts with action: "delete_rule", ruleUid — permanently remove the rule"Investigate a firing alert" (triggered by "GRAFANA ALERTS" in prompt context)
grafana_investigate with focus = alert UID or symptom, timeWindow: "1h" — parallel multi-signal triage (metrics, logs, traces, context)grafana_check_alerts with action: "silence", matchers from alert's commonLabels — prevent repeat notificationssuggestedHypotheses[].testWith from investigate response — deep-dive with specific toolsgrafana_query_logs — search logs around alert time for root cause (statistics first: count_over_time, then samples)grafana_query_traces — inspect error/slow traces for span-level detailgrafana_annotate — mark findings on dashboardgrafana_check_alerts with action: "acknowledge" — mark as investigatedgrafana_check_alerts with action: "unsilence", silenceId — restore notifications"Is this metric anomalous?" / "Statistical assessment"
grafana_explore_datasources — get Prometheus UIDgrafana_explain_metric with period: "24h" — response includes anomaly (z-score, severity) and seasonality (vs 1d/7d ago)anomaly.severity >= "significant": grafana_query with z-score PromQL for time series viewgrafana_query with predict_linear(METRIC[6h], 3600) — where is it heading?"Alert fatigue analysis — which alerts are noisy?"
grafana_check_alerts with action: "analyze" — fatigue report with always-firing, flapping, and healthy classificationsfor: durationgrafana_check_alerts with action: "silence" for rules being tuned"Generate a postmortem for the last incident"
grafana_investigate with focus = incident symptom, timeWindow: "6h" — gather all evidencegrafana_check_alerts (list) — find resolved alerts with timelinegrafana_annotate (list, tags: ["investigation"]) — build event timelinegrafana_explain_metric for key metrics — trend/stats for incident periodgrafana_query_logs — error patterns (statistics first)grafana_query_traces — error traces for root cause evidence"Add an error rate panel to my API dashboard"
grafana_search with query: "API" — find the dashboardgrafana_get_dashboard — get current panels and structuregrafana_update_dashboard with operation: "add_panel", panel: { title: "Error Rate", type: "timeseries", targets: [...] }"Remove the old latency panel and add a new one"
grafana_get_dashboard — find panel IDsgrafana_update_dashboard with operation: "remove_panel", panelId: <old_id>grafana_update_dashboard with operation: "add_panel", panel: { ... }"Change my dashboard time range to 7 days"
grafana_update_dashboard with operation: "update_metadata", time: { "from": "now-7d", "to": "now" }"Send my team the weekly dashboard"
grafana_search with the dashboard namegrafana_get_dashboard — get panel IDsgrafana_share_dashboard for each key panel with from: "now-7d""Show me my weekly work review"
grafana_push_metrics — push work data (commits, hours, meetings)grafana_create_dashboard with template: "weekly-review" — weekly trends + all custom metrics tablegrafana_share_dashboard — deliver key panels inline"Track my daily fitness data"
grafana_push_metrics — push fitness metrics (steps, weight, calories). Response has queryNames with exact PromQL names.grafana_create_dashboard with template: "weekly-review" — weekly trendsgrafana_create_alert using queryNames from step 1 for exact PromQL — e.g., threshold: 5000, condition: "lt""Backfill my step count for the past week"
grafana_push_metrics — push 7 data points with timestamp on each:
{ "metrics": [{ "name": "steps", "value": 8000, "timestamp": "2025-01-13" }, { "name": "steps", "value": 10500, "timestamp": "2025-01-14" }, ...] }grafana_create_dashboard with template: "metric-explorer" — visualize the trendgrafana_share_dashboard with from: "now-7d" — deliver the chart"What custom data have I pushed?"
grafana_push_metrics with action: "list" — see all custom metric definitions with queryName per metricgrafana_query using queryName values — see current valuesgrafana_list_metrics with search: "ext" — server-side discovery from Prometheus"System health check — give me a full status report"
grafana_explore_datasources — discover available datasourcesgrafana_check_alerts — check pending alerts + action: "list_rules" for configured rulesgrafana_search with query: "" and enrich: true — find all dashboards with updatedAt and panelCount for staleness triagegrafana_get_dashboard with audit: true for stale or suspicious dashboards — check which panels have data vs brokenauditSummary), stale dashboards, broken panels"Audit my dashboard — which panels are broken?"
grafana_get_dashboard with audit: true — dry-runs every panel's queryauditSummary for counts (ok, nodata, error, skipped)error panels, explain the issue; for nodata panels, suggest fixes (missing metrics, wrong datasource)"What is this metric doing?"
grafana_explore_datasources — get Prometheus UIDgrafana_explain_metric with metric name — get current value, trend, stats, metadata"Compare costs this week vs. last week" / "Did the new model help?"
grafana_explore_datasources — get Prometheus UIDgrafana_explain_metric with period: "7d", compareWith: "previous" — one call gives this week's stats + last week's stats + percentage changecomparison.change object — "costs are down 25% week-over-week, the model switch is working""Compare metric over different time scales"
grafana_explain_metric with period: "24h" — recent trendgrafana_explain_metric with period: "7d" — weekly context"Where are my costs going?" / "Why did my costs spike?"
grafana_explore_datasources — get Prometheus UIDgrafana_explain_metric with openclaw_lens_cost_by_token_type — get trend + suggestedBreakdowns (always: ["model", "token_type", "provider"]) + suggestedQueriessuggestedBreakdowns to know which dimensions matter, or pick from suggestedQueries for ready-to-use PromQLgrafana_query with sum by (model) (rate(openclaw_lens_cost_by_token_type[5m])) — get concrete breakdown numbers"Set up AI observability" / "Show me LLM traces and logs"
grafana_explore_datasources — verify Prometheus, Loki, and Tempo datasources are availablegrafana_search with query: "LLM Command Center" — check for existinggrafana_create_dashboard with template: "llm-command-center" — health overview with live error/session feedsgrafana_create_dashboard with template: "session-explorer" — per-session trace drill-downgrafana_share_dashboard — deliver an LLM latency or token usage panel inline"Why is my LLM slow?" (multi-signal investigation)
grafana_explain_metric with gen_ai_client_operation_duration_seconds — check latency trendgrafana_query_logs with {service_name="openclaw"} | json | component="lifecycle" |= "llm.output", extractFields: true — find slow LLM calls with fields.duration_s and fields.model surfaced directlygrafana_query_traces with queryType: "get" and fields.trace_id from step 2 — inspect spans; response includes correlationHint.logQuery for correlated logsgrafana_query with model-specific duration breakdown — identify which model is slow"Debug a slow or failing session"
grafana_query_traces with { resource.service.name = "openclaw" && status = error } or minDuration: "10s" — find problematic tracesgrafana_query_traces with queryType: "get" and the trace ID — inspect span hierarchy (response includes correlationHint)grafana_query_logs with the correlationHint.logQuery from step 2, extractFields: true — correlated logsgrafana_query with relevant PromQL — check metrics around the same time windowgrafana_annotate with text: "Debug: [findings]", tags: ["debug"] — mark investigation on dashboards"Which dashboards are stale?" / "Give me a dashboard summary report"
grafana_search with query: "" and enrich: true — gets updatedAt, panelCount, folderTitle for every dashboardupdatedAt to identify stale dashboards (e.g., not updated in 30+ days)"Am I being attacked?" / "Security status" / "Security audit"
grafana_security_check — runs 6 parallel security checks, returns threat level (green/yellow/red)suggestedActions from the responsedashboardTemplate is present: grafana_create_dashboard with template: "security-overview" — persistent monitoringgrafana_create_alert using suggested thresholds for ongoing monitoring"Set up security monitoring"
grafana_check_alerts with action: "setup" — create webhook contact point (if not already done)grafana_create_dashboard with template: "security-overview" — 15-panel security dashboardgrafana_create_alert with expr: "rate(openclaw_lens_webhook_error_total[5m]) / (rate(openclaw_lens_webhook_received_total[5m]) + 0.001)", threshold: 0.3, condition: "gt" — webhook error burstgrafana_create_alert with expr: "openclaw_lens_daily_cost_usd", threshold: 10, condition: "gt" — cost spikegrafana_create_alert with expr: "sum(increase(openclaw_lens_prompt_injection_signals_total[1h]))", threshold: 3, condition: "gt" — prompt injection signals"Investigate security alert"
grafana_security_check — get current threat assessmentgrafana_query_logs with {service_name="openclaw"} | json | component="lifecycle" | event_name=~"prompt_injection.detected|gateway.start|gateway.stop" — security event logsgrafana_query with specific PromQL for the affected check (e.g., rate(openclaw_lens_tool_error_classes_total[5m]))grafana_annotate with text: "Security investigation", tags: ["security"] — mark investigation on dashboardsgrafana_check_alerts with action: "acknowledge" — mark alerts as investigated"Clean up old test dashboards"
grafana_search with query: "test" or relevant tags and enrich: true — includes updatedAt for staleness checkgrafana_update_dashboard with operation: "delete" for each confirmed dashboard"Find my production dashboards"
grafana_search with tags: ["production"] or starred: true"Why is the queue backing up?"
grafana_explore_datasources — get Prometheus UIDgrafana_query with openclaw_lens_queue_depth — current queue depthgrafana_query with sum(rate(openclaw_message_queued_total[5m])) by (openclaw_source) — message inflow by sourcegrafana_query with histogram_quantile(0.95, rate(openclaw_queue_wait_ms_milliseconds_bucket[5m])) — p95 queue wait timegrafana_query with sum(rate(openclaw_queue_lane_enqueue_total[5m])) vs sum(rate(openclaw_queue_lane_dequeue_total[5m])) — enqueue vs dequeue rategrafana_query with openclaw_lens_sessions_stuck — check for stuck sessions blocking the queuegrafana_explain_metricFor composing custom dashboards from discovered metrics — template selection, metric-to-panel mapping, JSON snippets, worked examples across domains — see references/dashboard-composition.md.
"Monitor my PostgreSQL database"
alloy_pipeline with recipe postgres-exporter, params { connectionString: "postgres://..." }, name analytics-dbALLOY_POSTGRES_EXPORTER_ANALYTICS_DB_CONNECTIONSTRING env var where Alloy runsalloy_pipeline action status, name analytics-db — verify data flowinggrafana_list_metrics with search pg_ — discover available PostgreSQL metricsgrafana_create_dashboard with template metric-explorer, title Analytics DB Healthgrafana_create_alert — pg_up{job="analytics-db"} == 0 (database down)"Set up full-stack Kubernetes monitoring"
alloy_pipeline recipe kubernetes-pods (auto-discovers pod metrics)alloy_pipeline recipe kubernetes-services (scrapes service endpoints)alloy_pipeline recipe kubernetes-logs (collects pod logs to Loki)alloy_pipeline action diagnose — verify all 3 healthygrafana_create_dashboard with template multi-kpi — cluster healthgrafana_create_alert — pod restart rate, OOMKills"Collect Docker container logs and search errors"
alloy_pipeline recipe docker-logsalloy_pipeline action status, name docker-logsgrafana_query_logs with {source="docker"} |= "error", start now-1h"Parse JSON fields from Kafka logs"
alloy_pipeline recipe kafka-logs, params { brokers: ["kafka:9092"], topics: ["app-logs"], jsonExpressions: { level: "", message: "", request_id: "ctx.rid" }, labelFields: { level: "" }, structuredMetadata: { request_id: "" } }alloy_pipeline action status, name kafka-logsgrafana_query_logs with { expr: '{source="kafka"} | level="error"' }"Generate RED metrics from traces"
alloy_pipeline recipe span-metrics, params { dimensions: ["http.method", "http.status_code"] }alloy_pipeline action statusgrafana_query with { expr: 'sum(rate(traces_spanmetrics_calls_total[5m]))' }grafana_create_dashboard template metric-explorer"Monitor endpoint availability"
alloy_pipeline recipe blackbox-exporter, params { targets: [{ name: "api", address: "https://api.example.com", module: "http_2xx" }] }alloy_pipeline action statusgrafana_create_alert with { expr: 'probe_success == 0', name: 'Endpoint Down' }"My app has /metrics at myapp:9090 — dashboard + alerts"
alloy_pipeline recipe scrape-endpoint, params { url: "http://myapp:9090/metrics" }alloy_pipeline action status — verify scrapinggrafana_list_metrics — discover what's exposedgrafana_create_dashboard using discovered metricsgrafana_create_alert for critical metrics"Set up Linux system monitoring"
alloy_pipeline with recipe node-exporter, name linux-metrics — system CPU, memory, disk metricsalloy_pipeline with recipe journal-logs, name linux-journal — systemd journal logsalloy_pipeline with recipe file-logs, name linux-syslog, params { paths: ["/var/log/syslog", "/var/log/messages"] } — traditional syslog filesalloy_pipeline action diagnose — verify all 3 healthygrafana_create_dashboard with template metric-explorer, title Linux System Health"Are all my data collection pipelines healthy?"
alloy_pipeline action diagnose — connectivity, all pipeline health, drift, limitsalloy_pipeline action status for detailsremediation field for fixesFor metric names, types, labels, and common PromQL — covering both openclaw_* (from diagnostics-otel) and openclaw_lens_* (from grafana-lens) — see references/agent-metrics.md.
For external data naming conventions and integration patterns, see references/external-data.md.
For 5-phase investigation methodology, anomaly detection PromQL, RED/USE method queries, LogQL investigation discipline, TraceQL debugging, SLI/SLO burn rates, cross-signal correlation, and composed investigation workflows — see references/sre-investigation.md.
For the full 26-recipe catalog with detailed parameters, Alloy component mappings, troubleshooting, and advanced usage patterns — see references/alloy-pipelines.md.
For component snippets for raw-config (escape hatch) pipelines — see references/alloy-components.md.