Promql Generator

Interactive Query Planning Workflow

CRITICAL: This skill emphasizes interactive planning before query generation. Always engage the user in a collaborative planning process to ensure the generated query matches their exact intentions.

Follow this workflow when generating PromQL queries:

Stage 1: Understand the Monitoring Goal

Start by understanding what the user wants to monitor or measure. Ask clarifying questions to gather requirements:

1. Primary Goal: What are you trying to monitor or measure?

   • Request rate (requests per second)
   • Error rate (percentage of failed requests)
   • Latency/duration (response times, percentiles)
   • Resource usage (CPU, memory, disk, network)
   • Availability/uptime
   • Queue depth, saturation, throughput
   • Custom business metrics

2. Use Case: What will this query be used for?

   • Dashboard visualization (Grafana, Prometheus UI)
   • Alerting rule (firing when threshold exceeded)
   • Ad-hoc troubleshooting/analysis
   • Recording rule (pre-computed aggregation)
   • Capacity planning or SLO tracking

3. Context: Any additional context?

   • Service/application name
   • Team or project
   • Priority level
   • Existing metrics or naming conventions

Use the AskUserQuestion tool to gather this information if not provided.

When to Ask vs. Infer: If the user's initial request already clearly specifies the goal, use case, and context (e.g., "Create an alert for P95 latency > 500ms for payment-service"), you may acknowledge these details in your response instead of re-asking. Only ask clarifying questions for information that is missing or ambiguous.

Stage 2: Identify Available Metrics

Determine which metrics are available and relevant:

1. Metric Discovery: What metrics are available?

   • Ask the user for metric names
   • If uncertain, suggest common naming patterns
   • Check for metric type indicators in the name:
     • _total suffix → Counter
     • _bucket, _sum, _count suffix → Histogram
     • No suffix → Likely Gauge
     • _created suffix → Counter creation timestamp

2. Metric Type Identification: Confirm the metric type(s)

   • Counter: Cumulative metric that only increases (or resets to zero)
     • Examples: http_requests_total, errors_total, bytes_sent_total
     • Use with: rate(), irate(), increase()
   • Gauge: Point-in-time value that can go up or down
     • Examples: memory_usage_bytes, cpu_temperature_celsius, queue_length
     • Use with: avg_over_time(), min_over_time(), max_over_time(), or directly
   • Histogram: Buckets of observations with cumulative counts
     • Examples: http_request_duration_seconds_bucket, response_size_bytes_bucket
     • Use with: histogram_quantile(), rate()
   • Summary: Pre-calculated quantiles with count and sum
     • Examples: rpc_duration_seconds{quantile="0.95"}
     • Use _sum and _count for averages; don't average quantiles

3. Label Discovery: What labels are available on these metrics?

   • Common labels: job, instance, environment, service, endpoint, status_code, method
   • Ask which labels are important for filtering or grouping

Use the AskUserQuestion tool to confirm metric names, types, and available labels.

Stage 3: Determine Query Parameters

Gather specific requirements for the query.

Pre-confirmation for User-Provided Parameters

IMPORTANT: When the user has already specified parameters in their initial request (e.g., "5-minute window", "500ms threshold", "> 5% error rate"), you MUST:

1. Acknowledge the provided values explicitly in your response
2. Present them as pre-filled defaults in AskUserQuestion with the first option being "Use specified values"
3. Allow quick confirmation rather than re-asking for information already given

Example: If user says "alert when P95 latency exceeds 500ms", use:

AskUserQuestion:
- Question: "Confirm the alert threshold?"
- Options:
  1. "500ms (as specified)" - Use the threshold from your request
  2. "Different threshold" - Let me specify a different value

This respects the user's input and speeds up the workflow while still allowing modifications.

1. Time Range: What time window should the query cover?

   • Instant value (current)
   • Rate over time ([5m], [1h], [1d])
   • For rate calculations: typically [1m] to [5m] for real-time, [1h] to [1d] for trends
   • Rule of thumb: Rate range should be at least 4x the scrape interval

2. Label Filtering: Which labels should filter the data?

   • Exact matches: job="api-server", status_code="200"
   • Negative matches: status_code!="200"
   • Regex matches: instance=~"prod-.*"
   • Multiple conditions: {job="api", environment="production"}

3. Aggregation: Should the data be aggregated?

   • No aggregation: Return all time series as-is
   • Aggregate by labels: sum by (job, endpoint), avg by (instance)
   • Aggregate without labels: sum without (instance, pod), avg without (job)
   • Common aggregations: sum, avg, max, min, count, topk, bottomk

4. Thresholds or Conditions: Are there specific conditions?

   • For alerting: threshold values (e.g., error rate > 5%)
   • For filtering: only show series above/below a value
   • For comparison: compare against historical data (offset)

Use the AskUserQuestion tool to gather or confirm these parameters. When the user has already provided values (e.g., "5-minute window", "> 5%"), present them as the default option for confirmation.

Stage 4: Present the Query Plan

BEFORE GENERATING ANY CODE, present a plain-English query plan and ask for user confirmation:

## PromQL Query Plan

Based on your requirements, here's what the query will do:

**Goal**: [Describe the monitoring goal in plain English]

**Query Structure**:
1. Start with metric: `[metric_name]`
2. Filter by labels: `{label1="value1", label2="value2"}`
3. Apply function: `[function_name]([metric][time_range])`
4. Aggregate: `[aggregation] by ([label_list])`
5. Additional operations: [any calculations, ratios, or transformations]

**Expected Output**:
- Data type: [instant vector/scalar]
- Labels in result: [list of labels]
- Value represents: [what the number means]
- Typical range: [expected value range]

**Example Interpretation**:
If the query returns `0.05`, it means: [plain English explanation]

**Does this match your intentions?**
- If yes, I'll generate the query and validate it
- If no, let me know what needs to change

Use the AskUserQuestion tool to confirm the plan with options:

• "Yes, generate this query"
• "Modify [specific aspect]"
• "Show me alternative approaches"

When the user chooses:

• "Modify [specific aspect]": ask one focused follow-up question about what to change (metric, labels, function, time range, threshold, or output shape), then present an updated plan before generating.
• "Show me alternative approaches": provide at least two valid query plans with trade-offs (accuracy, cost, cardinality, readability), then ask the user to choose one before generating.

Stage 5: Generate the PromQL Query

Once the user confirms the plan, generate the actual PromQL query following best practices.

IMPORTANT: Consult Reference Files Before Generating

Before writing any query code, you MUST:

1. Identify the query category first (histogram, RED, USE, function-specific, optimization, etc.).

2. Read only the relevant reference section(s) using the Read tool:

   • For histogram queries → Read references/metric_types.md (Histogram section)
   • For error/latency patterns → Read references/promql_patterns.md (RED method section)
   • For resource monitoring → Read references/promql_patterns.md (USE method section)
   • For optimization questions → Read references/best_practices.md
   • For specific functions → Read references/promql_functions.md
   • Re-read a section only if requirements changed or you have not consulted it yet in the current thread.

3. If a needed reference cannot be read, state the issue and continue with best-effort generation using the most applicable documented pattern you already have.

4. Cite the applicable pattern or best practice in your response:

   As documented in references/promql_patterns.md (Pattern 3: Latency Percentile):
   # 95th percentile latency
   histogram_quantile(0.95, sum by (le) (rate(...)))
   

5. Reference example files when generating similar queries:

   Based on examples/red_method.promql (lines 64-82):
   # P95 latency with proper histogram_quantile usage
   

This keeps generated queries aligned with documented patterns while avoiding unnecessary full-file rereads on iterative follow-ups.

Best Practices for Query Generation

1. Always Use Label Filters

   # Good: Specific filtering reduces cardinality
   rate(http_requests_total{job="api-server", environment="prod"}[5m])
   
   # Bad: Matches all time series, high cardinality
   rate(http_requests_total[5m])
   

2. Use Appropriate Functions for Metric Types

   # Counter: Use rate() or increase()
   rate(http_requests_total[5m])
   
   # Gauge: Use directly or with *_over_time()
   memory_usage_bytes
   avg_over_time(memory_usage_bytes[5m])
   
   # Histogram: Use histogram_quantile()
   histogram_quantile(0.95,
     sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
   )
   

3. Apply Aggregations with by() or without()

   # Aggregate by specific labels (keeps only these labels)
   sum by (job, endpoint) (rate(http_requests_total[5m]))
   
   # Aggregate without specific labels (removes these labels)
   sum without (instance, pod) (rate(http_requests_total[5m]))
   

4. Use Exact Matches Over Regex When Possible

   # Good: Faster exact match
   http_requests_total{status_code="200"}
   
   # Bad: Slower regex match when not needed
   http_requests_total{status_code=~"200"}
   

5. Calculate Ratios Properly

   # Error rate: errors / total requests
   sum(rate(http_requests_total{status_code=~"5.."}[5m]))
   /
   sum(rate(http_requests_total[5m]))
   

6. Use Recording Rules for Complex Queries

   • If a query is used frequently or is computationally expensive
   • Pre-aggregate data to reduce query load
   • Follow naming convention: level:metric:operations

7. Format for Readability

   # Good: Multi-line for complex queries
   histogram_quantile(0.95,
     sum by (le, job) (
       rate(http_request_duration_seconds_bucket{job="api-server"}[5m])
     )
   )
   

Common Query Patterns

Pattern 1: Request Rate

# Requests per second
rate(http_requests_total{job="api-server"}[5m])

# Total requests per second across all instances
sum(rate(http_requests_total{job="api-server"}[5m]))

Pattern 2: Error Rate

# Error ratio (0 to 1)
sum(rate(http_requests_total{job="api-server", status_code=~"5.."}[5m]))
/
sum(rate(http_requests_total{job="api-server"}[5m]))

# Error percentage (0 to 100)
(
  sum(rate(http_requests_total{job="api-server", status_code=~"5.."}[5m]))
  /
  sum(rate(http_requests_total{job="api-server"}[5m]))
) * 100

Pattern 3: Latency Percentile (Histogram)

# 95th percentile latency
histogram_quantile(0.95,
  sum by (le) (
    rate(http_request_duration_seconds_bucket{job="api-server"}[5m])
  )
)

Pattern 4: Resource Usage

# Current memory usage
process_resident_memory_bytes{job="api-server"}

# Average CPU usage over 5 minutes
avg_over_time(process_cpu_seconds_total{job="api-server"}[5m])

Pattern 5: Availability

# Percentage of up instances
(
  count(up{job="api-server"} == 1)
  /
  count(up{job="api-server"})
) * 100

Pattern 6: Saturation/Queue Depth

# Average queue length
avg_over_time(queue_depth{job="worker"}[5m])

# Maximum queue depth in the last hour
max_over_time(queue_depth{job="worker"}[1h])

Stage 6: Validate the Generated Query

ALWAYS attempt to validate the generated query first using the devops-skills:promql-validator skill:

After generating the query, automatically invoke:
Skill(devops-skills:promql-validator)

The devops-skills:promql-validator skill will:
1. Check syntax correctness
2. Validate semantic logic (correct functions for metric types)
3. Identify anti-patterns and inefficiencies
4. Suggest optimizations
5. Explain what the query does
6. Verify it matches user intent

Validation checklist:

• Syntax is correct (balanced brackets, valid operators)
• Metric type matches function usage
• Label filters are specific enough
• Aggregation is appropriate
• Time ranges are reasonable
• No known anti-patterns
• Query is optimized for performance

If validation fails, fix issues and re-validate until all checks pass.

If the validator skill is unavailable, fails to run, or cannot complete after two fix/re-validate cycles:

• Report the validator failure briefly (tool unavailable, timeout, parsing error, etc.).
• Run a manual fallback check (syntax shape, metric/function compatibility, label filtering, aggregation, time range sanity).
• Mark any unchecked areas as UNVERIFIED and ask the user whether to proceed with best-effort output or provide more context for another validation attempt.

IMPORTANT: Display Validation Results to User

After running validation, you MUST display the structured results to the user in this format:

## PromQL Validation Results

### Syntax Check
- Status: ✅ VALID / ⚠️ WARNING / ❌ ERROR / ⚠️ UNVERIFIED
- Issues: [list any syntax errors]

### Best Practices Check
- Status: ✅ OPTIMIZED / ⚠️ CAN BE IMPROVED / ❌ HAS ISSUES / ⚠️ UNVERIFIED
- Issues: [list any problems found]
- Suggestions: [list optimization opportunities]

### Validation Coverage
- Validator tool run: [successful / failed / unavailable]
- Checks completed: [syntax, semantics, anti-patterns, performance, intent-match]
- Checks skipped: [list any skipped checks, or "None"]

### Query Explanation
- **What it measures**: [plain English description]
- **Output labels**: [list labels in result, or "None (scalar)"]
- **Expected result structure**: [instant vector / scalar / etc.]

This transparency helps users understand the validation process and any recommendations.

Stage 7: Provide Usage Instructions

After generation and validation (or manual fallback validation), provide the user with:

1. The Final Query:

   [Generated and validated PromQL query]
   

2. Query Explanation:

   • What the query measures
   • How to interpret the results
   • Expected value range
   • Labels in the output

3. How to Use It:

   • For Dashboards: Copy into Grafana/Prometheus UI panel query
   • For Alerts: Integrate into Alertmanager rule with threshold
   • For Recording Rules: Add to Prometheus recording rule config
   • For Ad-hoc: Run directly in Prometheus expression browser

4. Customization Notes:

   • Time ranges that might need adjustment
   • Labels to modify for different environments
   • Threshold values to tune
   • Alternative functions if requirements change

5. Related Queries:

   • Suggest complementary queries
   • Mention recording rule opportunities
   • Recommend dashboard panels

Native Histograms (Prometheus 3.x+)

Native histograms are now stable in Prometheus 3.0+ (released November 2024). They offer significant advantages over classic histograms:

• Sparse bucket representation with near-zero cost for empty buckets
• No configuration of bucket boundaries during instrumentation
• Coverage of the full float64 range
• Efficient mergeability across histograms
• Simpler query syntax

Important: Starting with Prometheus v3.8.0, native histograms are fully stable. However, scraping native histograms still requires explicit activation via the scrape_native_histograms configuration setting. Starting with v3.9, no feature flag is needed but scrape_native_histograms must be set explicitly.

Native vs Classic Histogram Syntax

# Classic histogram (requires _bucket suffix and le label)
histogram_quantile(0.95,
  sum by (job, le) (rate(http_request_duration_seconds_bucket[5m]))
)

# Native histogram (simpler - no _bucket suffix, no le label needed)
histogram_quantile(0.95,
  sum by (job) (rate(http_request_duration_seconds[5m]))
)

Native Histogram Functions

# Get observation count rate from native histogram
histogram_count(rate(http_request_duration_seconds[5m]))

# Get sum of observations from native histogram
histogram_sum(rate(http_request_duration_seconds[5m]))

# Calculate fraction of observations between two values
histogram_fraction(0, 0.1, rate(http_request_duration_seconds[5m]))

# Average request duration from native histogram
histogram_sum(rate(http_request_duration_seconds[5m]))
/
histogram_count(rate(http_request_duration_seconds[5m]))

Detecting Native vs Classic Histograms

Native histograms are identified by:

• No _bucket suffix on the metric name
• No le label in the time series
• The metric stores histogram data directly (not separate bucket counters)

When querying, check if your Prometheus instance has native histograms enabled:

# prometheus.yml - Enable native histogram scraping
scrape_configs:
  - job_name: 'my-app'
    scrape_native_histogram: true  # Prometheus 3.x+

Custom Bucket Native Histograms (NHCB)

Prometheus 3.4+ supports custom bucket native histograms (schema -53), allowing classic histogram to native histogram conversion. This is a key migration path for users with existing classic histograms.

Benefits of NHCB:

• Keep existing instrumentation (no code changes needed)
• Store classic histograms as native histograms for lower costs
• Query with native histogram syntax
• Improved reliability and compression

Configuration (Prometheus 3.4+):

# prometheus.yml - Convert classic histograms to NHCB on scrape