Observability stack: kube-prometheus-stack (Prometheus + Grafana + AlertManager), Grafana Tempo (distributed tracing), Grafana Loki (log aggregation), and Grafana Alloy (OTLP telemetry pipeline). All on K3s ARM64 with local-path storage and Gateway API routing.
| Component | Chart | Version | App Version | Namespace |
|---|---|---|---|---|
| Prometheus + Grafana + AlertManager | prometheus-community/kube-prometheus-stack | 82.17.0 | v0.89.0 | monitoring |
| Tempo (tracing backend) | grafana-community/tempo | 1.26.7 | 2.10.1 | monitoring |
| Loki (log aggregation) | grafana/loki | 6.55.0 | 3.x | monitoring |
| Alloy (telemetry pipeline) | grafana/alloy | 1.7.0 | v1.15.0 | monitoring |
| version-checker (image version tracking) | jetstack/version-checker | 0.10.0 | 0.10.0 | monitoring |
| helm-dashboard (Helm release UI, read-only) | komodorio/helm-dashboard | 2.0.6 | 2.1.1 | monitoring |
| Docker registry:2 (ARM64 image storage) | N/A (kubectl) | 2 | registry |
Grafana is included in kube-prometheus-stack — do NOT install a separate Grafana chart.
# prometheus-community (kube-prometheus-stack)
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
# grafana (alloy, loki, etc.)
helm repo add grafana https://grafana.github.io/helm-charts
# grafana-community (tempo — migrated here after Jan 2026)
helm repo add grafana-community https://grafana-community.github.io/helm-charts
IMPORTANT:
grafana/tempoandgrafana/tempo-distributedare deprecated. Usegrafana-community/tempo(single-binary) orgrafana-community/tempo-distributed.
grafana-community/tempo v2.0.0 has a template bug:
it generates overrides.defaults: {} in the Tempo config which Tempo 2.10.x rejects:
field defaults not found in type overrides.legacyConfig
Use v1.26.7 (latest 1.26.x) — same app version 2.10.1, no bug.
Apps (OpenTelemetry instrumented) All Pod Logs
│ OTLP gRPC :4317 / HTTP :4318 │
▼ ▼
[Alloy DaemonSet] discovery.kubernetes → discovery.relabel
otelcol.receiver.otlp → otelcol.exporter.otlp.tempo (traces)
loki.source.kubernetes → loki.write (logs)
│ │
│ OTLP gRPC │ HTTP push
▼ ▼
[Tempo StatefulSet] [Loki StatefulSet + MinIO]
← single binary, local-path ← single binary, MinIO 1Gi object storage
metricsGenerator → remote_write ← chunksCache/resultsCache disabled (CM4 memory)
│
│ remote_write → :9090/api/v1/write
▼
[Prometheus StatefulSet] ← local-path PVC 20Gi
│
▼
[Grafana Deployment]
datasources: Prometheus + Tempo + Loki
HTTPRoute: grafana.cluster.home → kube-prometheus-stack-grafana:80
These k8s components are embedded in the k3s server process and do NOT expose metrics:
kubeControllerManager:
enabled: false # embedded in k3s
kubeScheduler:
enabled: false # embedded in k3s
kubeEtcd:
enabled: false # embedded etcd, no external port
kubeProxy:
enabled: false # replaced by Cilium BPF
Enabled (work fine on k3s):
kubelet, coreDns, kubeStateMetrics, nodeExporter, prometheusOperatorGrafana uses ClusterIP + HTTPRoute — same pattern as ArgoCD and Pi-hole.
# HTTPRoute (created by install-kube-prometheus-stack role):
apiVersion: gateway.networking.k8s.io/v1