Redis Cluster

Sentinel vs Cluster Mode

Rule: Never combine Sentinel and Cluster in the same topology.

Setup and Configuration

Node Configuration (redis.conf)

port 7000
cluster-enabled yes
cluster-config-file nodes-7000.conf
cluster-node-timeout 5000
appendonly yes
# Require all slots covered to serve traffic (default yes):
cluster-require-full-coverage yes
# Allow replicas to serve stale reads during failover:
replica-serve-stale-data yes
# Cluster bus port (auto = port + 10000):
# cluster-port 17000

Repeat for each node, changing port. Open both client port AND bus port between all nodes.

Create Cluster

# 6 nodes: 3 masters + 3 replicas (--cluster-replicas 1)
redis-cli --cluster create \
  192.168.1.1:7000 192.168.1.2:7000 192.168.1.3:7000 \
  192.168.1.4:7000 192.168.1.5:7000 192.168.1.6:7000 \
  --cluster-replicas 1

Output:

>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
...
[OK] All 16384 slots covered.

Verify

redis-cli -c -p 7000 CLUSTER INFO
# cluster_state:ok
# cluster_slots_assigned:16384
# cluster_slots_ok:16384
# cluster_known_nodes:6
# cluster_size:3

redis-cli -c -p 7000 CLUSTER NODES
# <id> 192.168.1.1:7000@17000 myself,master - 0 0 1 connected 0-5460
# <id> 192.168.1.4:7000@17000 slave <master-id> 0 0 1 connected
# ...

Replication and Failover

How Failover Works

1. Replica detects master PFAIL (no PONG within cluster-node-timeout).
2. Replica requests votes from other masters via FAILOVER_AUTH_REQUEST.
3. Majority of masters grant vote → replica wins election.
4. Winning replica executes CLUSTER FAILOVER internally, takes over master's slots.
5. New topology propagated via gossip within seconds.

Manual Failover

# Connect to the REPLICA you want to promote:
redis-cli -p 7001 CLUSTER FAILOVER
# OK
# The replica becomes master; old master becomes replica.

Options: CLUSTER FAILOVER FORCE (skip data sync) or CLUSTER FAILOVER TAKEOVER (skip vote, emergency only).

Replica Migration

Set cluster-migration-barrier 1 (default). If a master loses all replicas, an orphan replica from a master with excess replicas migrates automatically.

Hash Tags for Multi-Key Operations

Enclose the shared portion in {} braces. Only the substring inside the first {...} is hashed.

# These keys all hash to the same slot (hashing "user:1000"):
SET {user:1000}:name "Alice"
SET {user:1000}:email "[email protected]"
SET {user:1000}:prefs '{"theme":"dark"}'

# Multi-key operation succeeds because same slot:
MGET {user:1000}:name {user:1000}:email
# 1) "Alice"
# 2) "[email protected]"

# WITHOUT hash tags — FAILS:
MGET user:1000:name user:1000:email
# (error) CROSSSLOT Keys in request don't hash to the same slot

Hash Tag Rules

• First { to first } is extracted. Empty {} → full key is hashed.
• Use consistent tag patterns: {entity:id}:field.
• Beware hotspots: if all keys share one tag, one node handles all traffic. Distribute tags across entities.

Lua Scripts and Transactions

All keys in EVAL/EVALSHA and MULTI/EXEC must hash to the same slot. Pass related keys via hash tags.

# Lua script — all keys must share a slot:
EVAL "return redis.call('GET', KEYS[1]) .. redis.call('GET', KEYS[2])" \
  2 {user:1}:name {user:1}:email
# "[email protected]"

Resharding and Scaling

Add a New Node

# Add as empty master:
redis-cli --cluster add-node 192.168.1.7:7000 192.168.1.1:7000

# Add as replica of a specific master:
redis-cli --cluster add-node 192.168.1.8:7000 192.168.1.1:7000 \
  --cluster-slave --cluster-master-id <master-node-id>

Reshard Slots

# Move 1000 slots from source to new node:
redis-cli --cluster reshard 192.168.1.1:7000 \
  --cluster-from <source-node-id> \
  --cluster-to <target-node-id> \
  --cluster-slots 1000 \
  --cluster-yes

# Or rebalance automatically across all masters:
redis-cli --cluster rebalance 192.168.1.1:7000 --cluster-use-empty-masters

During resharding, clients may receive ASK redirections for migrating slots. Cluster-aware clients handle this transparently.

Remove a Node

1. Reshard all slots away from the node first.
2. Then: redis-cli --cluster del-node 192.168.1.1:7000 <node-id>

Fix Broken State

redis-cli --cluster fix 192.168.1.1:7000
# Repairs open slots, stuck migrations, uncovered slots.

Client-Side Configuration

MOVED Redirection

Permanent redirect — the slot lives on a different node. Client must update its slot map and retry.

> GET mykey
(error) MOVED 3999 192.168.1.2:7000
# Client updates: slot 3999 → 192.168.1.2:7000, retries GET mykey there.

ASK Redirection

Temporary redirect during slot migration. Client sends ASKING then retries the command on the target node. Do NOT update the slot map.

> GET mykey
(error) ASK 3999 192.168.1.3:7000
# Client: connect to 192.168.1.3:7000, send ASKING, then GET mykey.

Client Library Configuration

Python (redis-py):

from redis.cluster import RedisCluster

rc = RedisCluster(
    startup_nodes=[{"host": "192.168.1.1", "port": 7000}],
    decode_responses=True,
    skip_full_coverage_check=False,
    retry_on_timeout=True,
)
rc.set("{user:1}:name", "Alice")
print(rc.get("{user:1}:name"))  # "Alice"

Node.js (ioredis):

const Redis = require("ioredis");
const cluster = new Redis.Cluster([
  { host: "192.168.1.1", port: 7000 },
  { host: "192.168.1.2", port: 7000 },
], {
  redisOptions: { password: "secret" },
  scaleReads: "slave",          // read from replicas
  natMap: {},                    // for NAT/Docker port mapping
  retryDelayOnFailover: 300,
  retryDelayOnClusterDown: 1000,
});

Java (Jedis):

Set<HostAndPort> nodes = new HashSet<>();
nodes.add(new HostAndPort("192.168.1.1", 7000));
nodes.add(new HostAndPort("192.168.1.2", 7000));
JedisCluster jc = new JedisCluster(nodes, 5000, 5000, 3, "password",
    new GenericObjectPoolConfig<>());
jc.set("{user:1}:name", "Alice");

Topology Refresh

Most clients cache the slot map. Force periodic refresh to handle topology changes:

• ioredis: clusterRetryStrategy, auto-refresh on MOVED
• Lettuce: topologyRefreshOptions.enablePeriodicRefresh(Duration.ofSeconds(30))
• redis-py: automatic on MOVED/ASK

Persistence in Cluster Mode

Each node persists independently. Configure per-node:

RDB Snapshots

save 900 1       # snapshot if ≥1 write in 900s
save 300 10      # snapshot if ≥10 writes in 300s
save 60 10000    # snapshot if ≥10000 writes in 60s
dbfilename dump-7000.rdb
dir /var/lib/redis/7000/

AOF (Append Only File)

appendonly yes
appendfilename "appendonly-7000.aof"
appendfsync everysec          # balance durability/performance
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-use-rdb-preamble yes      # hybrid AOF for faster restart

Recommendation: Use AOF with aof-use-rdb-preamble yes for production clusters. RDB alone risks data loss between snapshots.

Monitoring and Diagnostics

Essential Commands

# Cluster health:
redis-cli -c -p 7000 CLUSTER INFO

# Node topology:
redis-cli -c -p 7000 CLUSTER NODES

# Slot distribution:
redis-cli --cluster check 192.168.1.1:7000

# Per-node memory:
redis-cli -p 7000 INFO memory
# used_memory_human:1.2G
# maxmemory_human:4G

# Slow queries:
redis-cli -p 7000 SLOWLOG GET 10

# Client connections per node:
redis-cli -p 7000 INFO clients
# connected_clients:142

# Keyspace stats:
redis-cli -p 7000 INFO keyspace
# db0:keys=1543210,expires=320100,avg_ttl=86400000

Key Metrics to Alert On

Latency Diagnostics

redis-cli -p 7000 --latency-history -i 5
# min: 0, max: 3, avg: 0.45 (100 samples) -- 5.00 seconds range

redis-cli -p 7000 LATENCY LATEST
redis-cli -p 7000 LATENCY HISTORY event-name

Common Pitfalls

1. Cross-Slot Errors

Cause: Multi-key command with keys in different slots. Fix: Use hash tags {tag}:key to colocate related keys.

2. Hotspot Nodes

Cause: Poor key distribution or single popular hash tag. Fix: Spread hash tags across entities. Monitor per-node ops/sec. Rebalance slots.

3. Large Keys (Big Keys)

Cause: Single key > 10 MB blocks cluster operations and resharding. Fix: Break into smaller keys. Use redis-cli --bigkeys to find offenders.

redis-cli -p 7000 --bigkeys
# Biggest string: user:megalist — 45.2 MB
# Biggest hash: session:abc — 12340 fields

4. Cluster Bus Port Blocked

Cause: Firewall blocks port+10000. Nodes can't gossip. Fix: Open both ports. Verify with CLUSTER NODES — nodes stuck in handshake state means bus port blocked.

5. Docker/NAT Issues

Cause: Cluster announces internal IPs that clients can't reach. Fix: Set cluster-announce-ip, cluster-announce-port, cluster-announce-bus-port in redis.conf.

cluster-announce-ip 203.0.113.10
cluster-announce-port 7000
cluster-announce-bus-port 17000

6. Full Coverage Requirement

Cause: cluster-require-full-coverage yes (default). If any slot is uncovered, entire cluster rejects writes. Fix: For availability over consistency, set to no. Cluster serves requests for covered slots even if some slots are down.

7. Stale Client Topology

Cause: Client caches slot map, doesn't refresh after failover/reshard. Fix: Enable topology refresh. Handle MOVED errors as triggers to refresh.

8. Memory Limit Without Eviction Policy

Cause: Node hits maxmemory, rejects writes, breaks replication. Fix: Set maxmemory-policy (e.g., allkeys-lru). Monitor memory per node. Reshard to distribute load.

9. Unbalanced Slot Distribution

Cause: Manual slot assignment or adding nodes without rebalancing. Fix: redis-cli --cluster rebalance <host>:<port> — redistributes slots evenly.

10. Replica Divergence During Network Partition

Cause: Split-brain — clients write to old master while new master is elected. Fix: Set min-replicas-to-write 1 and min-replicas-max-lag 10 to stop accepting writes when replicas are unreachable.

References

In-depth guides for advanced usage, troubleshooting, and operations:

Scripts

Executable helper scripts for cluster lifecycle management:

Usage examples:

# Bootstrap a 6-node dev cluster with Docker:
./scripts/setup-cluster.sh --mode docker --masters 3 --replicas 1

# Check cluster health:
./scripts/health-check.sh 127.0.0.1:7001 --verbose

# Migrate 1000 slots between nodes:
./scripts/resharding.sh --host 127.0.0.1:7001 \
  --from <source-id> --to <target-id> --slots 1000

# Clean up dev cluster:
./scripts/setup-cluster.sh --cleanup --mode docker

Assets

Production-ready templates and configurations:

# Quick start with Docker Compose:
cd assets/
docker compose up -d
# Cluster auto-initializes via the redis-cluster-init service.

# For bare-metal, copy and customize the config template:
cp assets/redis-cluster.conf /etc/redis/redis-7000.conf
# Edit: port, cluster-announce-ip, requirepass, maxmemory