Java Observability Metrics

• Counter: monotonic count (requests_total, errors_total).
• Gauge: instantaneous value (queue_depth, memory_used).
• Timer: duration measurements (request_latency).
• Histogram/Distribution: latency/size distributions suitable for percentile-ish analysis.
• Summary (if your backend supports): use carefully; often harder to aggregate across instances.

Golden signals (recommended baseline)

1. Traffic: http_server_requests_total
2. Errors: http_server_errors_total or http_server_requests_total{outcome="error"}
3. Latency: http_server_request_duration_seconds (histogram)
4. Saturation: thread pool utilization, queue size, DB pool usage

Naming conventions (Prometheus-friendly)

• Use snake_case.
• Base unit in name if needed:
  • _seconds, _bytes, _total.
• Prefer a consistent prefix/domain:
  • http_server_*, db_*, cache_*, mq_*.

Label/tag rules (cardinality guardrails)

Allowed labels for HTTP server metrics

• method (bounded)
• route (bounded; avoid raw path)
• status (bounded)
• outcome (bounded: success/error)
• service, env (bounded; usually from resource labels)

Forbidden labels (common cardinality bombs)

• userId, email, ip, sessionId
• requestId, traceId
• raw URL path, query string
• exception messages

Route normalization

• Use templated route names: /v1/orders/{id} not /v1/orders/123.
• If your framework does not provide route templates, implement normalization.

Histograms and timers (latency best practice)

• Prefer histograms for latency:
  • enable bucketed distributions suitable for SLOs.
• Choose buckets that match SLO thresholds:
  • e.g., 50ms, 100ms, 200ms, 500ms, 1s, 2s, 5s.
• Avoid per-endpoint custom histograms unless truly needed.

Instrumentation plan (step-by-step)

1. Define the questions:
   • "What is p95 latency for /orders in prod?"
   • "What is error rate for dependency X?"
2. Define the minimal metric set to answer them.
3. Define label set and enforce boundedness.
4. Implement instrumentation:
   • inbound HTTP
   • outbound HTTP clients
   • DB calls
   • cache
   • messaging consumers/producers
5. Validate on a staging environment:
   • confirm label cardinality is bounded
   • confirm naming conventions
6. Add dashboards and alerts:
   • SLO burn rate alerts (if used)
   • saturation alerts

Testing strategy

• Unit tests for route normalization.
• “Metrics snapshot” tests for presence of required metric names.
• Cardinality budget tests (fail if new labels explode).
• Load test sanity check to observe series count growth.

Outputs / artifacts

• docs/metrics.md (metric catalog + labels + intent)
• metrics/metric-catalog.yml (machine-readable inventory)
• metrics/cardinality-budget.md
• Code changes:
  • instrumentation wrappers
  • timers/histograms config
  • tests for bounded labels

Definition of Done (DoD)

• Baseline golden-signal metrics exist.
• Label sets documented and bounded.
• Cardinality budget validated in staging.
• Dashboards and alerts (at least basic) updated.
• Regression tests prevent accidental cardinality bombs.

Guardrails (What NOT to do)

• Never add high-cardinality labels.
• Never label metrics with IDs or raw error messages.
• Avoid duplicating the same metric under many names.

Common failure modes & fixes

• Symptom: Prometheus/metrics backend memory spikes -> Fix: remove high-cardinality labels, normalize routes.
• Symptom: metrics not aggregatable -> Fix: avoid instance-specific dimensions; use consistent naming.
• Symptom: percentiles inconsistent -> Fix: use histograms with defined buckets; avoid per-instance summaries.

References (see references/)

• references/prometheus-cardinality.md
• references/micrometer-timers-histograms.md
• references/metric-catalog-template.yml