Implementing Caching

Routing Logic

Core Principles

1. Cache Strategically, Not Universally

Not everything should be cached. Cache data that is read frequently, expensive to compute, tolerant of brief staleness, and unlikely to change between reads. Caching mutable, low-read data adds complexity without benefit.

Decision test: If the data is read 10x more than it is written and a few seconds of staleness is acceptable, it is a strong caching candidate.

2. Choose the Right Caching Pattern

Default recommendation: Start with cache-aside. It is the simplest, most portable, and gives the application full control over caching behavior.

3. Choose the Right Eviction Policy

Default recommendation: Use allkeys-lru for Redis. It handles the common case (Pareto distribution) well. Add TTL as a safety net on all entries.

4. Layer Your Caches

Apply caching at multiple layers, each serving a different purpose:

Browser Cache -> CDN Edge -> API Gateway -> App In-Memory -> Redis -> Database

• Browser cache -- Eliminates network requests entirely for returning users
• CDN edge -- Serves static and semi-static content from the nearest PoP
• Application in-memory -- Sub-millisecond reads for hot data (bounded by process memory)
• Distributed cache (Redis) -- Shared across instances, survives restarts, millisecond reads
• Database -- Source of truth; materialized views and query caching reduce load

5. Invalidation is Harder Than Caching

"There are only two hard things in Computer Science: cache invalidation and naming things." Start with TTL-based expiration and layer on event-driven invalidation for critical data paths. Never rely solely on manual invalidation.

6. Agentic Workflow Considerations

When an AI agent is implementing or reasoning about caching:

• Identify the consistency requirement first -- Ask whether the consumer tolerates stale data and for how long before choosing a strategy.
• Prefer explicit TTLs over unbounded caching -- Every cache entry should have an expiration. Zombie entries from missing invalidation logic are a common production incident.
• Consider cache warming on deploy -- Cold caches after deployment cause latency spikes. Pre-populate critical entries.
• Log cache hit/miss ratios -- If hit ratio drops below 80%, the cache may be poorly sized or the key space too large.
• Test invalidation paths -- Cache bugs are often invisible until stale data causes user-facing issues. Include invalidation in integration tests.
• When caching LLM responses -- Evaluate whether prompt caching (provider-level, exact prefix match) or semantic caching (application-level, similarity match) is appropriate. Prompt caching is cheaper and simpler; semantic caching handles paraphrased queries.

Workflow

1. Identify -- Determine what data is a caching candidate (high read-to-write ratio, expensive to compute, tolerant of staleness).
2. Select layer -- Choose the appropriate cache layer(s): HTTP, CDN, in-memory, distributed, or database.
3. Select pattern -- Pick a caching pattern (cache-aside, read-through, write-through, write-behind) based on consistency needs.
4. Configure eviction -- Set eviction policy (LRU/LFU), max memory, and TTL values.
5. Implement invalidation -- Define how and when cached data is invalidated (TTL, event-driven, tag-based, versioned keys).
6. Instrument -- Add metrics for cache hit/miss ratio, latency, eviction rate, and memory usage.
7. Test -- Verify cache behavior under load, test invalidation correctness, and simulate cache failures.
8. Monitor -- Track hit ratios in production; tune TTLs and cache sizes based on observed patterns.

Quick Reference

Cache-Aside Pattern (Most Common)

async function getUser(userId: string): Promise<User> {
  // 1. Check cache
  const cached = await redis.get(`user:${userId}`);
  if (cached) return JSON.parse(cached);

  // 2. Cache miss -- read from database
  const user = await db.users.findById(userId);
  if (!user) throw new NotFoundError('User not found');

  // 3. Populate cache with TTL
  await redis.set(`user:${userId}`, JSON.stringify(user), 'EX', 300); // 5 min TTL

  return user;
}

async function updateUser(userId: string, data: Partial<User>): Promise<User> {
  // 1. Update database (source of truth)
  const user = await db.users.update(userId, data);

  // 2. Invalidate cache
  await redis.del(`user:${userId}`);

  return user;
}

HTTP Cache-Control Cheat Sheet

# Static assets (fingerprinted filenames)
Cache-Control: public, max-age=31536000, immutable

# HTML pages
Cache-Control: no-cache
ETag: "abc123"

# API responses (short-lived, revalidatable)
Cache-Control: private, max-age=0, must-revalidate
ETag: "v2-abc123"

# API responses (CDN-friendly)
Cache-Control: public, max-age=60, s-maxage=300, stale-while-revalidate=60

# Sensitive data (never cache)
Cache-Control: no-store

TTL Guidelines

Cache Key Design

# Pattern: {entity}:{identifier}:{optional-variant}