Replication Guide

Failover Procedures

1. Check replica status: SELECT * FROM system.replicas WHERE is_readonly = 1
2. Verify Keeper connectivity: SELECT * FROM system.zookeeper WHERE path = '/'
3. If replica is readonly due to Keeper disconnect, it auto-recovers when connection restores
4. For permanent failures: SYSTEM DROP REPLICA 'replica_name' FROM TABLE db.table

Quorum Writes

• SET insert_quorum = 2 — wait for N replicas to confirm
• SET insert_quorum_parallel = 1 — parallel quorum inserts (v21.8+)
• SET insert_quorum_timeout = 60000 — timeout in ms
• Use for critical data that must survive node failures

Keeper Management

• ClickHouse Keeper is the recommended replacement for ZooKeeper
• Monitor: system.zookeeper table for browsing ZK tree
• Key paths: /clickhouse/tables/ for table metadata
• Check Keeper health: SELECT * FROM system.asynchronous_metrics WHERE metric LIKE '%Keeper%'

Common Issues

• Split brain: Multiple leaders — usually Keeper issue, restart Keeper
• Readonly replica: Lost Keeper session — check network, Keeper logs
• Queue buildup: Slow fetches — check network bandwidth, disk I/O
• Diverged replicas: SYSTEM SYNC REPLICA db.table to force sync