Monitoring Ingestion Pipeline

Environment context

The Grafana MCP is connected to a single Grafana instance scoped to one environment. If the user hasn't specified, ask which environment they want to investigate:

• prod-us — US production (us-east-1)
• prod-eu — EU production (eu-central-1)

Most ingestion app metrics are environment-specific by virtue of which Grafana you're connected to — they don't carry an environment label. CloudWatch metrics are also scoped to the connected Grafana's AWS account.

Cross-environment comparison requires switching Grafana instances (not possible in one session).

All datasource UIDs, dashboard UIDs, ingestion_pipeline/ingestion_lane label values, and ClickHouse type label values are identical across prod-us and prod-eu.

Key differences:

• ingestion-general-turbo deployment exists only in prod-us.
• CloudWatch cluster IDs differ by region suffix (see topology sections below).

Observability landscape

Six telemetry domains, all validated identical across prod-us and prod-eu:

Datasource notes:

• Do NOT use primary Loki (P8E80F9AEF21F6940) — it returns 502 intermittently in both envs. Always use Loki-logs (P44D702D3E93867EC).
• Do NOT use CloudWatch Root (PAAE47F430CFD1449) — it exists only in prod-us.

Stable waypoints

These facts change infrequently and are hard to discover dynamically.

Deployment roles

All PLUGIN_SERVER_MODE=ingestion-v2 deployments (the "analytics ingestion" family), plus specialized modes:

Note: ingestion-general-turbo exists only in prod-us.

Metric prefixes

Every prefix here can be discovered live with list_prometheus_metric_names using datasourceUid: "victoriametrics" and regex: "<prefix>.*".

Redis topology

Ingestion workers depend on up to five Redis instances. Redis health is inferred from ingestion-side metrics and CloudWatch ElastiCache metrics.

Ingestion-side Redis metrics: overflow_redirect_redis_*, cookieless_redis_error. Infrastructure-side: CloudWatch datasource P034F075C744B399F.

prod-eu uses the same logical cluster names but different endpoint suffixes (.mkct36...euc1 instead of .nfjpjm...use1). The prod-eu primary Redis is posthog-prod-redis-encripted (sic — the typo is in the actual cluster name).

Kafka topology

Two backing systems:

• MSK — prod-us: posthog-prod-us-events-2026-03-08 (12 brokers, kafka.m7g.8xlarge, tiered storage); prod-eu: posthog-prod-eu-events-2025-10-16 (15 brokers, kafka.m7g.4xlarge)
• WarpStream — in-cluster agents in warpstream-ingestion namespace; virtual cluster vcn_ingestion_prod_us / vcn_ingestion_prod_eu

KMinion instance for MSK: app_kubernetes_io_instance=~"kminion-msk-analytics".

Postgres topology

Postgres metrics via prometheus-postgres-exporter and prometheus-postgres-persons-exporter.

prod-eu: posthog-cloud-prod-eu-central-1 and posthog-cloud-persons-prod-eu-central-1.

ClickHouse topology

ClickHouse is the ultimate downstream consumer of events the ingestion pipeline produces. Ingestion workers never talk to CH directly — they publish to Kafka topics which CH consumes via its built-in Kafka engine and Kafka Connect (DuckLake). CH health directly impacts perceived ingestion quality: if CH falls behind on consumption, users see stale data.

Cluster roles (discovered via type label on ClickHouseMetrics_*):

Most type label values are identical across prod-us and prod-eu. Minor differences like batch-exports or test may exist only in one env.

Two consumption paths from Kafka to ClickHouse:

1. ClickHouse Kafka Engine — native CH feature. Metrics prefixed ClickHouseProfileEvents_Kafka* (e.g., KafkaMessagesPolled, KafkaRowsRead, KafkaRowsRejected, KafkaCommitFailures). Consumer groups: clickhouse_events_json (prod-us), group1 / group1_recent (prod-eu).
2. Kafka Connect — runs in kafka-connect namespace, uses DuckLake sink connector. Metrics prefixed kafka_connect_* and kafka_connect_ducklake_sink_task_metrics_*. Consumer groups: connect-events-ducklake*.

Key health signals for ingestion operators:

• kminion_kafka_consumer_group_topic_lag{group_id=~"clickhouse_events_json|group1|group1_recent", topic_name="clickhouse_events_json"} — lag between ingestion output and CH consumption (group name differs by env: clickhouse_events_json in prod-us, group1/group1_recent in prod-eu)
• kminion_kafka_consumer_group_topic_lag_seconds with same group filter — same in seconds
• ClickHouseProfileEvents_KafkaRowsRejected — rows CH couldn't parse/insert
• ClickHouseProfileEvents_FailedInsertQuery — insert failures (schema issues, too many parts, etc.)
• ClickHouseAsyncMetrics_MaxPartCountForPartition — rising part count = merge pressure
• ClickHouseMetrics_ReadonlyReplica — replicas that fell behind and went read-only
• ClickHouseAsyncMetrics_ReplicasMaxAbsoluteDelay — max replication delay
• posthog_celery_clickhouse_table_parts_count / _table_row_count — Django-side CH health monitors

Pyroscope services

Profile types: process_cpu:cpu:nanoseconds:cpu:nanoseconds, wall:wall:nanoseconds:wall:nanoseconds, memory:inuse_space:bytes:inuse_space:bytes, memory:inuse_objects:count:inuse_space:bytes.

Grafana dashboards

Discovery workflows

Prometheus / VictoriaMetrics

1. list_prometheus_metric_names — datasourceUid: "victoriametrics", regex: "ingestion_.*" to enumerate app metrics
2. Pick a metric, then list_prometheus_label_names scoped to it — see available dimensions
3. list_prometheus_label_values — discover actual values for a label (e.g. labelName: "cause" on ingestion_event_dropped_total)
4. query_prometheus with PromQL — always scope by app or ingestion_pipeline and set a time range

Repeat with other prefixes: consumed_batch_*, person_*, personhog_*, overflow_redirect_*, ClickHouseMetrics_*, kafka_connect_*, etc.

Loki (logs)

1. list_loki_label_names — datasourceUid: "P44D702D3E93867EC"
2. list_loki_label_values for app — find ingestion containers (values like ingestion-events, ingestion-logs, etc.)
3. query_loki_logs — e.g. {app=~"ingestion-.*"} |= "error" or {namespace="clickhouse"} |= "Exception"

Pyroscope (profiling)

1. list_pyroscope_profile_types — data_source_uid: "pyroscope"
2. fetch_pyroscope_profile — matchers: '{service_name="ingestion/ingestion-events"}', profile_type: "process_cpu:cpu:nanoseconds:cpu:nanoseconds"

Dashboards

1. search_dashboards — query "ingestion" or "clickhouse" or a specific deployment name
2. get_dashboard_by_uid — use a known UID (e.g. "ingestion-general") to get panel details
3. get_dashboard_panel_queries — extract PromQL from existing panels

Redis / ElastiCache

• Ingestion-side: discover overflow_redirect_redis_*, cookieless_redis_* metrics in VictoriaMetrics
• Infrastructure: CloudWatch datasource P034F075C744B399F for ElastiCache (CPU, memory, connections, latency). Cluster IDs: ingestion-prod-redis, posthog-solo (prod-us primary), ingestion-duplicates-prod-redis, cookieless-prod-redis

Postgres / Aurora

• Ingestion-side: discover postgres_error_total, person_*, group_* metrics
• Infrastructure: prometheus-postgres-exporter and prometheus-postgres-persons-exporter metrics
• CloudWatch RDS: datasource P034F075C744B399F with cluster IDs posthog-cloud-prod-us-east-1 / posthog-cloud-persons-prod-us-east-1

ClickHouse

• Cluster health: list_prometheus_metric_names with regex "ClickHouseMetrics_.*" or "ClickHouseProfileEvents_.*" Scope with type label for cluster role (e.g. type="events")
• Kafka engine health: regex "ClickHouseProfileEvents_Kafka.*"
• Kafka Connect: regex "kafka_connect_.*" — scope with namespace="kafka-connect"
• Consumer lag (the bridge): kminion_kafka_consumer_group_topic_lag with group_id=~"clickhouse_events_json|group1|group1_recent|connect-events-ducklake.*" and topic_name="clickhouse_events_json"
• Logs: {namespace="clickhouse"} |= "Exception" or {namespace="kafka-connect"}
• Dashboards: vm-clickhouse-cluster-overview, 8aa35a4a-091a-4645-ac8f-ae46901f0060, cdzv7o1635n9ca

Key metric domains

Categories of what to look for. Discover specific metrics live using the prefixes above.

Kafka consumer health — batch duration, messages consumed per batch, consumer group assignment/rebalance events, consumer lag (via KMinion). Metrics: consumed_batch_*, kafka_consumer_*, kminion_kafka_consumer_group_topic_lag* scoped by group_id.

Pipeline processing — step-level latency and error rates, pipeline result distribution (ingested, filtered, dropped, DLQ'd). Metrics: events_pipeline_step_ms, events_pipeline_step_error_total, ingestion_pipeline_results by result.

Person/group stores — person flush latency, cache hit rates, Postgres write latency, merge failures, properties size. Metrics: person_*, group_*, personhog_*.

Outputs — Kafka production to ClickHouse-bound topics. Message size, latency, errors. Metrics: ingestion_outputs_* by topic.

Overflow routing — stateful overflow decisions, Redis operations for overflow state. Metrics: overflow_redirect_* by type, result, decision.

ClickHouse downstream health — CH cluster QPS, memory, disk, merge pressure, replication lag, Kafka engine consumption (rows read/rejected/failed), Kafka Connect task health and consumer lag. This tells you whether events are actually making it to the query layer. Metrics: ClickHouseMetrics_*, ClickHouseProfileEvents_*, ClickHouseAsyncMetrics_* scoped by type; kafka_connect_* scoped by namespace.

K8s resources — container_* and kube_* for CPU, memory, restarts, HPA state. Scope: namespace="posthog", pod=~"ingestion-.*" (or namespace="clickhouse", pod=~"chi-ingestion-.*" for CH ingestion pods).

Investigation playbooks

See references/investigation-playbooks.md for step-by-step workflows covering: health checks, event drops, latency, consumer lag, person processing, Kafka/MSK issues, Redis, Postgres, session replay, ClickHouse downstream health, and cross-environment comparison.