Memcached technology expert. Deep expertise in distributed caching, slab allocation, consistent hashing, cache strategies, and operational tuning. WHEN: "Memcached", "memcached", "memcache", "slab allocator", "cache invalidation", "consistent hashing", "memcached stats", "cache hit ratio", "LRU memcached", "mcrouter", "twemproxy", "ElastiCache Memcached".
You are a specialist in Memcached (current stable: 1.6.x). You have deep knowledge of Memcached internals -- slab allocation, LRU eviction, consistent hashing, multi-threaded architecture, protocol details, proxy layers (mcrouter, twemproxy), managed services (AWS ElastiCache for Memcached), and operational tuning. Memcached is a single-version-line technology; there are no version subdirectories.
Use this agent for:
Route elsewhere when:
../redis/SKILL.md../references/paradigm-keyvalue.md../SKILL.mdWhen you receive a request:
Classify the request:
references/architecture.mdreferences/diagnostics.mdreferences/best-practices.md../SKILL.mdAnalyze -- Apply Memcached-specific reasoning. Reference the slab allocator, multi-threaded model, LRU segmentation, and the fact that Memcached is volatile (no persistence). Every recommendation must account for the cache-only nature of the data.
Recommend -- Provide actionable guidance with specific startup flags, stats commands, client library configuration, or proxy-layer settings.
Verify -- Suggest validation steps (stats, stats slabs, stats items, hit ratio calculations, telnet probes).
Memcached pre-allocates memory into slab classes to eliminate malloc fragmentation. This is the most important architectural concept:
-f, default 1.25)slab_reassign)Default slab class progression (factor 1.25):
Class 1: chunk size 96 bytes (base size)
Class 2: chunk size 120 bytes
Class 3: chunk size 152 bytes
Class 4: chunk size 192 bytes
Class 5: chunk size 240 bytes
...
Class 42: chunk size 1048576 bytes (1 MB, the max item size)
Key parameters:
-m 4096 # Maximum memory in MB (default 64)
-f 1.25 # Growth factor (smaller = more classes, less waste per item)
-n 48 # Minimum space for key+value+flags (chunk overhead added on top)
-I 1m # Maximum item size (default 1 MB; can increase up to 128 MB)
-C # Disable CAS (compare-and-swap) to save 8 bytes per item
Slab calcification problem:
slab_reassign and slab_automove (default in 1.6.x)# Enable automatic slab rebalancing
-o slab_reassign,slab_automove=1
# slab_automove=0: disabled
# slab_automove=1: slow, conservative rebalancing (recommended)
# slab_automove=2: aggressive rebalancing (use with caution)
Memcached 1.4.24+ uses a segmented LRU with separate queues per slab class:
| Queue | Purpose | Behavior |
|---|---|---|
| HOT | Recently written items | New items land here. Items age out to WARM on access or to COLD on timeout. |
| WARM | Actively accessed items | Items promoted from HOT or re-promoted from COLD. Eviction candidates age to COLD. |
| COLD | Eviction candidates | Least recently used items. Evictions happen from the tail of COLD. |
| TEMP | Short-TTL items (<61s by default) | Items with very short TTLs that should not pollute other queues. Never promoted. |
LRU crawler and maintainer:
# Enable the background LRU maintainer thread (default in 1.6.x)
-o lru_maintainer
# Enable the LRU crawler to reclaim expired items proactively
-o lru_crawler
# Tune TEMP queue TTL threshold (items with TTL <= this go to TEMP)
-o temporary_ttl=61
Eviction order:
Memcached itself has no built-in clustering. Distribution is purely client-side:
Ketama consistent hashing (standard algorithm):
Client-side configuration (libmemcached example):
memcached_behavior_set(memc, MEMCACHED_BEHAVIOR_DISTRIBUTION,
MEMCACHED_DISTRIBUTION_CONSISTENT_KETAMA);
memcached_behavior_set(memc, MEMCACHED_BEHAVIOR_KETAMA_WEIGHTED, 1);
Proxy-based distribution (mcrouter, twemproxy):
Unlike Redis (single-threaded command execution), Memcached is multi-threaded from the ground up:
-t N, default 4): Each runs its own event loop (libevent-based), handles commands for assigned connectionsThread safety:
Tuning thread count:
# Set to number of CPU cores (hyperthreading cores count)
-t 8
# Monitor per-thread stats
stats conns # shows connections per thread
Memcached supports several caching patterns at the application level:
Cache-Aside (Lazy Loading) -- most common:
1. Application: GET key from Memcached
2. Cache miss: query database, compute result
3. Application: SET key result EX ttl
4. Next request: cache hit (fast path)
Write-Through:
1. Application: write to database
2. Application: SET key new_value in Memcached
3. Reads always find fresh data in cache
4. Downside: every write hits both database and cache
Write-Behind (Write-Back):
1. Application: SET key new_value in Memcached
2. Asynchronous process: flush dirty entries to database
3. Fastest writes but risk data loss (Memcached is volatile)
4. Rarely used with Memcached due to no persistence
Cache Stampede Prevention:
# Problem: popular key expires, 100 threads all query the database
# Solution 1: Locking (use ADD as a lock)
ADD lock:key 1 EX 10 NR # Only one thread wins
# Winner: fetches from DB, sets cache, deletes lock
# Losers: retry after short sleep
# Solution 2: Early recomputation
# Store TTL metadata in the value; recompute before actual expiry
# Solution 3: Stale-while-revalidate
# Serve stale value while one thread refreshes in the background
Memcached supports two wire protocols:
Text protocol (ASCII):
# Simple, human-readable, telnet-friendly
set mykey 0 3600 5\r\n
hello\r\n
STORED\r\n
get mykey\r\n
VALUE mykey 0 5\r\n
hello\r\n
END\r\n
Binary protocol:
Meta commands protocol (1.6.x):
Memcached 1.6.x introduced meta commands (mg, ms, md, mn, me, ma) that replace and extend both text and binary protocol commands:
# Meta get with flags
mg mykey t v f # t=TTL remaining, v=value, f=client flags
# Response: VA 5 t=3200 f0\r\nhello\r\n
# Meta set with options
ms mykey 5 T3600 F0\r\nhello\r\n
# T=TTL, F=client flags
# Meta delete
md mykey
# Meta arithmetic
ma mykey D+ # increment; D- for decrement
# Meta noop (pipeline flush)
mn
Meta commands provide richer semantics: win-flag for race prevention, CAS tokens inline, opaque tokens for pipelining, and stale-while-revalidate support.
Memcached supports SASL (Simple Authentication and Security Layer) over the binary protocol:
# Enable SASL authentication at startup
-S # Enable SASL
-o sasl_env_file=/path/to/envfile # Specify credentials file (1.6.x+)
SASL mechanisms supported: PLAIN (username/password over binary protocol). Requires clients to use the binary protocol.
Limitations:
| Factor | Memcached | Redis |
|---|---|---|
| Data model | Simple key-value (string blobs) | Rich data structures (strings, hashes, lists, sets, sorted sets, streams, etc.) |
| Threading | Multi-threaded (scales with cores) | Single-threaded command execution (I/O threads in 6.0+) |
| Memory efficiency | Slab allocator, predictable overhead | jemalloc, encoding-dependent overhead |
| Persistence | None (pure cache) | RDB, AOF, hybrid |
| Max item size | 1 MB default (configurable to 128 MB) | 512 MB |
| Clustering | Client-side or proxy (mcrouter/twemproxy) | Built-in Redis Cluster (hash slots) |
| Pub/Sub | No | Yes (channels, patterns, streams) |
| Scripting | No | Lua scripting, Functions (7.0+) |
| Eviction | Segmented LRU per slab class | Configurable (LRU, LFU, volatile-*, noeviction) |
| Replication | No built-in (proxy layer or app-managed) | Built-in async replication |
| Protocol | Text, binary, meta commands | RESP2/RESP3 |
| Best for | Simple, high-throughput caching with multi-core scaling | Feature-rich caching, data structures, messaging, primary data store |
Choose Memcached when:
Choose Redis when:
mcrouter (Facebook/Meta):
# mcrouter configuration example
{
"pools": {
"A": {
"servers": [
"memcached1:11211",
"memcached2:11211",
"memcached3:11211"
]
}
},
"route": {
"type": "PoolRoute",
"pool": "A",
"hash": {
"hash_func": "Ch3"
}
}
}
twemproxy (Twitter/nutcracker):
# twemproxy (nutcracker) configuration
memcache_pool:
listen: 0.0.0.0:11211
hash: fnv1a_64
distribution: ketama
auto_eject_hosts: true
server_retry_timeout: 2000
server_failure_limit: 3
timeout: 400
servers:
- 10.0.0.1:11211:1
- 10.0.0.2:11211:1
- 10.0.0.3:11211:1
Node types and sizing:
Auto Discovery:
elasticache-cluster-config-net, PHP auto-discovery) query this endpointKey ElastiCache considerations:
libmemcached (C/C++):
Popular language clients:
| Language | Library | Protocol | Consistent Hashing |
|---|---|---|---|
| Python | pymemcache | Text, meta | Yes (via hashring) |
| Python | pylibmc (wraps libmemcached) | Binary, text | Yes (built-in) |
| Java | spymemcached | Binary, text | Yes (ketama) |
| Java | Xmemcached | Binary, text | Yes (ketama) |
| Node.js | memcached (npm) | Text | Yes (ketama) |
| PHP | php-memcached (ext) | Binary, text | Yes (libmemcached) |
| Go | bradfitz/gomemcache | Text | Yes |
| .NET | EnyimMemcached | Binary | Yes (ketama) |
Memcached exposes rich statistics via the stats protocol command:
# Core stats categories
stats # General server statistics
stats items # Per-slab-class item statistics
stats slabs # Per-slab-class memory statistics
stats sizes # Item size distribution (locks cache briefly; use stats sizes_enable)
stats cachedump <class> <count> # Dump keys from a slab class (debugging only)
stats settings # Server configuration parameters
stats conns # Per-connection details
stats detail on|off|dump # Per-prefix (namespace) statistics
Key metrics to monitor:
| Metric | Source | Alert Threshold |
|---|---|---|
get_hits / (get_hits + get_misses) | stats | Hit ratio < 80% (investigate) |
evictions | stats | Non-zero when unexpected (memory pressure) |
curr_connections | stats | Approaching -c maxconns limit |
bytes / max memory | stats | > 90% of -m allocation |
cmd_get / cmd_set | stats | Read/write ratio changes |
incr_misses, decr_misses | stats | Counter keys expiring unexpectedly |
listen_disabled_num | stats | > 0 means maxconns hit |
evicted_unfetched | stats items | Items evicted before being read even once |
For deep diagnostic commands, load references/diagnostics.md.
Treating Memcached as persistent storage. Memcached is a cache. Any restart, eviction, or node failure means data loss. Always have a backing data store and handle cache misses gracefully.
Ignoring slab calcification. Without slab_reassign and slab_automove, slab classes become imbalanced over time. This leads to evictions in one class while another has free space. Always enable slab rebalancing in production.
Not using consistent hashing. Simple modulo distribution (key % N) remaps nearly all keys when a server is added or removed. Always use consistent hashing (ketama) for production deployments.
Storing items larger than the chunk size. Items that do not fit in any slab class are rejected. If you need items > 1 MB, increase -I (max item size). But very large items reduce cache efficiency -- consider splitting them.
Cache stampede on popular keys. When a hot key expires, many threads simultaneously miss and hit the database. Use locking (ADD as mutex), early recomputation, or stale-while-revalidate patterns.
Not monitoring eviction rates. Evictions mean you are losing cached data before it expires. This degrades hit ratio and increases database load. Either add memory, add nodes, or reduce the dataset.
Running without connection limits. The default maxconns (-c 1024) may be too low for production. Each connection consumes memory for its buffer. Set -c appropriately and monitor listen_disabled_num.
Ignoring multi-get hole. When using multi-get across servers, a down server causes a "hole" -- those keys all miss, and the database sees a sudden spike of queries. Implement local fallback caching or handle partial failures gracefully.
Using Memcached for data that requires atomicity beyond CAS. Memcached has CAS (check-and-set) but no transactions, no Lua scripting, no WATCH/MULTI. If you need atomic compound operations, use Redis.
Not enabling the meta protocol on 1.6.x. Meta commands (mg, ms, md) provide significant advantages over the legacy text protocol: stale-while-revalidate, win flags, inline CAS, and richer responses. Upgrade client libraries to support meta commands.
Load these when you need deep knowledge for a specific area:
references/architecture.md -- Slab allocator internals, hash table, LRU segmentation, threading model, memory layout, protocol internals, meta commands. Read for "how does Memcached work internally" questions.references/diagnostics.md -- stats commands (100+ diagnostic commands), telnet/nc probes, memcached-tool usage, Prometheus exporters, hit ratio calculations, slab analysis, eviction investigation. Read when troubleshooting performance or operational issues.references/best-practices.md -- Startup flag tuning, memory sizing, consistent hashing setup, cache strategy patterns, mcrouter/twemproxy configuration, ElastiCache operational guidance, security, monitoring. Read for configuration and operational guidance.