Database Memcached

How to Approach Tasks

When you receive a request:

1. Classify the request:

   • Architecture/internals -- Load references/architecture.md
   • Performance diagnostics -- Load references/diagnostics.md
   • Configuration/operations -- Load references/best-practices.md
   • Comparison with other databases -- Route to parent ../SKILL.md

2. Analyze -- Apply Memcached-specific reasoning. Reference the slab allocator, multi-threaded model, LRU segmentation, and the fact that Memcached is volatile (no persistence). Every recommendation must account for the cache-only nature of the data.

3. Recommend -- Provide actionable guidance with specific startup flags, stats commands, client library configuration, or proxy-layer settings.

4. Verify -- Suggest validation steps (stats, stats slabs, stats items, hit ratio calculations, telnet probes).

Core Expertise

Slab Allocator

Memcached pre-allocates memory into slab classes to eliminate malloc fragmentation. This is the most important architectural concept:

• Memory is divided into pages (default 1 MB each)
• Each page belongs to a slab class with a fixed chunk size
• Chunk sizes grow by a growth factor (-f, default 1.25)
• Items are stored in the smallest chunk that fits (key + value + metadata ~56 bytes overhead)
• Once a page is assigned to a slab class, it cannot be reassigned (unless using slab_reassign)

Default slab class progression (factor 1.25):

Class  1: chunk size     96 bytes (base size)
Class  2: chunk size    120 bytes
Class  3: chunk size    152 bytes
Class  4: chunk size    192 bytes
Class  5: chunk size    240 bytes
...
Class 42: chunk size 1048576 bytes (1 MB, the max item size)

Key parameters:

-m 4096          # Maximum memory in MB (default 64)
-f 1.25          # Growth factor (smaller = more classes, less waste per item)
-n 48            # Minimum space for key+value+flags (chunk overhead added on top)
-I 1m            # Maximum item size (default 1 MB; can increase up to 128 MB)
-C               # Disable CAS (compare-and-swap) to save 8 bytes per item

Slab calcification problem:

• Over time, traffic patterns change but slab pages are permanently assigned
• Class A may have 80% of pages but only 20% of traffic
• Class B may have 5% of pages but 50% of traffic, causing evictions
• Solution: enable slab_reassign and slab_automove (default in 1.6.x)

# Enable automatic slab rebalancing
-o slab_reassign,slab_automove=1
# slab_automove=0: disabled
# slab_automove=1: slow, conservative rebalancing (recommended)
# slab_automove=2: aggressive rebalancing (use with caution)

LRU Eviction (Segmented LRU)

Memcached 1.4.24+ uses a segmented LRU with separate queues per slab class:

LRU crawler and maintainer:

# Enable the background LRU maintainer thread (default in 1.6.x)
-o lru_maintainer
# Enable the LRU crawler to reclaim expired items proactively
-o lru_crawler
# Tune TEMP queue TTL threshold (items with TTL <= this go to TEMP)
-o temporary_ttl=61

Eviction order:

1. Expired items (reclaimed lazily on access or by LRU crawler)
2. COLD queue tail (least recently used items)
3. If all items in a slab class are active, eviction still happens from COLD tail

Consistent Hashing

Memcached itself has no built-in clustering. Distribution is purely client-side:

Ketama consistent hashing (standard algorithm):

• Each server is mapped to ~100-200 points on a hash ring (virtual nodes)
• Keys are hashed to the ring; the nearest server clockwise owns the key
• When a server is added/removed, only ~1/N of keys are remapped (N = number of servers)
• Without consistent hashing, a simple modulo (key % N) remaps nearly all keys on topology change

Client-side configuration (libmemcached example):

memcached_behavior_set(memc, MEMCACHED_BEHAVIOR_DISTRIBUTION,
                       MEMCACHED_DISTRIBUTION_CONSISTENT_KETAMA);
memcached_behavior_set(memc, MEMCACHED_BEHAVIOR_KETAMA_WEIGHTED, 1);

Proxy-based distribution (mcrouter, twemproxy):

• Consistent hashing is handled by the proxy
• Applications connect to the proxy as if it were a single Memcached instance
• Proxy routes commands to the correct backend server
• Benefits: language-agnostic hashing, connection pooling, failover

Multi-Threaded Architecture

Unlike Redis (single-threaded command execution), Memcached is multi-threaded from the ground up:

• Listener thread: Accepts new connections, distributes to worker threads round-robin
• Worker threads (-t N, default 4): Each runs its own event loop (libevent-based), handles commands for assigned connections
• LRU maintainer thread: Background LRU management and slab rebalancing
• LRU crawler thread: Background expired-item reclamation
• Hash table expansion thread: Resizes the hash table without blocking workers
• Slab rebalance thread: Moves pages between slab classes

Thread safety:

• Global hash table protected by per-bucket locks (fine-grained locking)
• Slab class operations use per-class locks
• Item-level operations use per-item reference counting and CAS for consistency
• Connection state is per-thread (no sharing)

Tuning thread count:

# Set to number of CPU cores (hyperthreading cores count)
-t 8
# Monitor per-thread stats
stats conns    # shows connections per thread

Cache Strategies

Memcached supports several caching patterns at the application level:

Cache-Aside (Lazy Loading) -- most common:

1. Application: GET key from Memcached
2. Cache miss: query database, compute result
3. Application: SET key result EX ttl
4. Next request: cache hit (fast path)

Write-Through:

1. Application: write to database
2. Application: SET key new_value in Memcached
3. Reads always find fresh data in cache
4. Downside: every write hits both database and cache

Write-Behind (Write-Back):

1. Application: SET key new_value in Memcached
2. Asynchronous process: flush dirty entries to database
3. Fastest writes but risk data loss (Memcached is volatile)
4. Rarely used with Memcached due to no persistence

Cache Stampede Prevention:

# Problem: popular key expires, 100 threads all query the database
# Solution 1: Locking (use ADD as a lock)
ADD lock:key 1 EX 10 NR        # Only one thread wins
# Winner: fetches from DB, sets cache, deletes lock
# Losers: retry after short sleep

# Solution 2: Early recomputation
# Store TTL metadata in the value; recompute before actual expiry

# Solution 3: Stale-while-revalidate
# Serve stale value while one thread refreshes in the background

Protocol: Text vs Binary

Memcached supports two wire protocols:

Text protocol (ASCII):

# Simple, human-readable, telnet-friendly
set mykey 0 3600 5\r\n
hello\r\n
STORED\r\n

get mykey\r\n
VALUE mykey 0 5\r\n
hello\r\n
END\r\n

Binary protocol:

• Fixed-size header (24 bytes request, 24 bytes response)
• More efficient parsing (no text scanning)
• Supports silent/quiet mutations (no response unless error)
• Required for SASL authentication
• Better pipelining with opcodes and sequence numbers

Meta commands protocol (1.6.x): Memcached 1.6.x introduced meta commands (mg, ms, md, mn, me, ma) that replace and extend both text and binary protocol commands:

# Meta get with flags
mg mykey t v f     # t=TTL remaining, v=value, f=client flags
# Response: VA 5 t=3200 f0\r\nhello\r\n

# Meta set with options
ms mykey 5 T3600 F0\r\nhello\r\n
# T=TTL, F=client flags

# Meta delete
md mykey

# Meta arithmetic
ma mykey D+       # increment; D- for decrement

# Meta noop (pipeline flush)
mn

Meta commands provide richer semantics: win-flag for race prevention, CAS tokens inline, opaque tokens for pipelining, and stale-while-revalidate support.

SASL Authentication

Memcached supports SASL (Simple Authentication and Security Layer) over the binary protocol:

# Enable SASL authentication at startup
-S                        # Enable SASL
-o sasl_env_file=/path/to/envfile  # Specify credentials file (1.6.x+)

SASL mechanisms supported: PLAIN (username/password over binary protocol). Requires clients to use the binary protocol.

Limitations:

• No TLS built-in (use stunnel, sidecar proxy, or VPC networking for encryption)
• No per-key ACLs (authenticated or not, full access)
• AWS ElastiCache Memcached does not support SASL; it relies on VPC security groups

Memcached vs Redis -- Decision Criteria

Choose Memcached when:

• Workload is simple GET/SET with string values
• Need to scale linearly with CPU cores on a single node
• Want the simplest possible caching layer
• Memory efficiency for small objects is critical (slab allocator wastes less than jemalloc for uniform sizes)
• No persistence requirements

Choose Redis when:

• Need data structures (hashes, sorted sets, lists)
• Need persistence or replication
• Need pub/sub or streams
• Need atomic operations beyond CAS (Lua scripting)
• Need built-in clustering

Proxy Layer: mcrouter and twemproxy

mcrouter (Facebook/Meta):

• Full Memcached protocol proxy
• Consistent hashing, replication, failover, warm-up, shadowing
• Connection pooling (reduces connections to backend servers)
• Prefix-based routing (different pools for different key namespaces)
• Cold cache warm-up (reads from old pool, writes to new pool during migration)
• Used at massive scale (trillions of requests/day at Meta)

# mcrouter configuration example
{
  "pools": {
    "A": {
      "servers": [
        "memcached1:11211",
        "memcached2:11211",
        "memcached3:11211"
      ]
    }
  },
  "route": {
    "type": "PoolRoute",
    "pool": "A",
    "hash": {
      "hash_func": "Ch3"
    }
  }
}

twemproxy (Twitter/nutcracker):

• Lightweight proxy for Memcached and Redis
• Consistent hashing (ketama) and modula distribution
• Connection multiplexing (many client connections -> few server connections)
• Automatic server ejection and re-injection on failure
• pipelining support for batching requests
• Simpler configuration than mcrouter; fewer features

# twemproxy (nutcracker) configuration
memcache_pool:
  listen: 0.0.0.0:11211
  hash: fnv1a_64
  distribution: ketama
  auto_eject_hosts: true
  server_retry_timeout: 2000
  server_failure_limit: 3
  timeout: 400
  servers:
    - 10.0.0.1:11211:1
    - 10.0.0.2:11211:1
    - 10.0.0.3:11211:1

AWS ElastiCache for Memcached

Node types and sizing:

• cache.t4g.* (burstable, dev/test), cache.m7g.* (general purpose), cache.r7g.* (memory-optimized)
• Memory available for caching = node memory minus OS/Memcached overhead (~10-15%)
• Up to 300 nodes per cluster (across up to 20 shards for cluster mode)

Auto Discovery:

• ElastiCache provides a configuration endpoint that returns the list of all cache nodes
• Client libraries (e.g., elasticache-cluster-config-net, PHP auto-discovery) query this endpoint
• When nodes are added/removed, clients automatically discover the new topology
• Polling interval configurable; default varies by client library

Key ElastiCache considerations:

• No SASL authentication (rely on VPC security groups and subnet groups)
• Parameter groups control Memcached configuration (chunk_size_growth_factor, max_item_size, etc.)
• Maintenance windows for patching (choose low-traffic periods)
• CloudWatch metrics: CPUUtilization, SwapUsage, Evictions, CurrConnections, NewConnections, BytesUsedForCache
• Scaling: vertical (change node type) or horizontal (add nodes, requires client rehashing)

Client Libraries

libmemcached (C/C++):

• Most widely used C client library
• Supports consistent hashing, binary/text protocol, SASL, async I/O
• Language bindings: PHP (php-memcached), Python (pylibmc), Perl, Ruby

Popular language clients:

Monitoring and Stats Overview

Memcached exposes rich statistics via the stats protocol command:

# Core stats categories
stats                  # General server statistics
stats items            # Per-slab-class item statistics
stats slabs            # Per-slab-class memory statistics
stats sizes            # Item size distribution (locks cache briefly; use stats sizes_enable)
stats cachedump <class> <count>  # Dump keys from a slab class (debugging only)
stats settings         # Server configuration parameters
stats conns            # Per-connection details
stats detail on|off|dump  # Per-prefix (namespace) statistics

Key metrics to monitor:

For deep diagnostic commands, load references/diagnostics.md.

Common Pitfalls

1. Treating Memcached as persistent storage. Memcached is a cache. Any restart, eviction, or node failure means data loss. Always have a backing data store and handle cache misses gracefully.

2. Ignoring slab calcification. Without slab_reassign and slab_automove, slab classes become imbalanced over time. This leads to evictions in one class while another has free space. Always enable slab rebalancing in production.

3. Not using consistent hashing. Simple modulo distribution (key % N) remaps nearly all keys when a server is added or removed. Always use consistent hashing (ketama) for production deployments.

4. Storing items larger than the chunk size. Items that do not fit in any slab class are rejected. If you need items > 1 MB, increase -I (max item size). But very large items reduce cache efficiency -- consider splitting them.

5. Cache stampede on popular keys. When a hot key expires, many threads simultaneously miss and hit the database. Use locking (ADD as mutex), early recomputation, or stale-while-revalidate patterns.

6. Not monitoring eviction rates. Evictions mean you are losing cached data before it expires. This degrades hit ratio and increases database load. Either add memory, add nodes, or reduce the dataset.

7. Running without connection limits. The default maxconns (-c 1024) may be too low for production. Each connection consumes memory for its buffer. Set -c appropriately and monitor listen_disabled_num.

8. Ignoring multi-get hole. When using multi-get across servers, a down server causes a "hole" -- those keys all miss, and the database sees a sudden spike of queries. Implement local fallback caching or handle partial failures gracefully.

9. Using Memcached for data that requires atomicity beyond CAS. Memcached has CAS (check-and-set) but no transactions, no Lua scripting, no WATCH/MULTI. If you need atomic compound operations, use Redis.

10. Not enabling the meta protocol on 1.6.x. Meta commands (mg, ms, md) provide significant advantages over the legacy text protocol: stale-while-revalidate, win flags, inline CAS, and richer responses. Upgrade client libraries to support meta commands.

Reference Files

Load these when you need deep knowledge for a specific area:

• references/architecture.md -- Slab allocator internals, hash table, LRU segmentation, threading model, memory layout, protocol internals, meta commands. Read for "how does Memcached work internally" questions.
• references/diagnostics.md -- stats commands (100+ diagnostic commands), telnet/nc probes, memcached-tool usage, Prometheus exporters, hit ratio calculations, slab analysis, eviction investigation. Read when troubleshooting performance or operational issues.
• references/best-practices.md -- Startup flag tuning, memory sizing, consistent hashing setup, cache strategy patterns, mcrouter/twemproxy configuration, ElastiCache operational guidance, security, monitoring. Read for configuration and operational guidance.