Obs Open Source Observability Stack

Commands to use

# Validate Prometheus config and recording/alert rules
promtool check config observability/prometheus/prometheus.yml
promtool check rules observability/prometheus/*.rules.yml

# Validate Alertmanager config
promtool check-config observability/alertmanager/alertmanager.yml

# Inspect Grafana dashboards (JSON validation)
cd observability/grafana/dashboards && for f in *.json; do jq . "$f" > /dev/null && echo "✓ $f" || echo "✗ $f"; done

# Validate Loki configuration YAML syntax
yamllint observability/loki/loki-config.yaml

# Validate Tempo configuration YAML syntax
yamllint observability/tempo/tempo-config.yaml

# Validate OpenTelemetry Collector configuration
otelcol validate --config observability/otel-collector.yaml
# or if using otel-contrib:
otelcol-contrib validate --config observability/otel-collector.yaml

# Verify metric label cardinality and low-cardinality constraint compliance
rg '\$__(\w+)' observability/prometheus/recording-rules.yml observability/prometheus/alert-rules.yml

# Check for any obvious high-cardinality patterns (user IDs, request IDs in labels)
rg '(user_id|request_id|pod_name|trace_id)' observability/prometheus/ observability/loki/

# Test Prometheus scrape targets (requires running Prometheus)
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, instance: .labels.instance, health: .health}'

# Query test: sample a metric (requires running Prometheus)
curl -s 'http://localhost:9090/api/v1/query?query=up' | jq '.data.result[0]'

Worked example

Example: add API latency dashboard and alert

• Expose/request metrics with low-cardinality labels.
• Add a Prometheus rule for high error rate or high latency.
• Add Grafana panels for request rate, p95 latency, and error ratio.
• Record threshold rationale, metric names, and owner/on-call expectations in the story.

Example: add log+trace correlation

• Send logs to Loki with low-cardinality labels.
• Send traces to Tempo via OTel collector.
• Ensure logs carry trace correlation fields as structured metadata, not as Loki labels.
• Record the correlation strategy and label-cardinality limits in the story.

Failure modes to avoid

• Using pod name, trace ID, request ID, or user ID as stable metric/Loki labels.
• Shipping dashboards without any alert ownership or operational meaning.
• Treating OSS stack config as static infrastructure instead of part of the application change.

Completion checklist

Follow the universal checklist at docs/guidelines/shared-operating-policy.md#completion-checklist and confirm all items pass before marking work complete.