System Design Patterns

Replication & Partitioning

• Split Brain: Multi-leader systems need explicit conflict resolution logic.
• Eventual Consistency: Account for replication lag. After user writes, route their reads to the master node (Read-your-writes consistency).
• Hot Spots: When sharding, beware "hot keys." Add salt to the hash to distribute load.

Scalable Architecture (Alex Xu — System Design)

Scaling from Zero to Millions

• Stateless Web Tier: Servers must not store session state. State belongs in Redis/Memcached.
• Redundancy: Avoid SPOF. Replicate data, duplicate services behind load balancers.
• Caching: Cache frequent reads. CDN for static assets.

Decoupling & Async Processing

• Message Queues: Use brokers (RabbitMQ, SQS, Kafka) as buffers during traffic spikes.
• DAG Processing: Break heavy tasks into directed graphs, process in parallel via queues.

API Protection

• Rate Limiting: Protect public APIs with Token Bucket or Sliding Window algorithms.

Data Distribution

• Consistent Hashing: For cache sharding, use consistent hashing with virtual nodes.
• Delta Sync: When syncing large files, transfer only changed blocks (chunks).

Site Reliability Engineering (Google SRE)

Error Budgets

• 100% Uptime is Impossible: Target 99.9% or 99.99%. The remainder is your error budget for releases and incidents.
• Stop the Line: When error budget is exhausted, freeze features. Fix tech debt only.

Eliminating Toil

• Toil is Toxic: If you do a manual, repetitive task twice — automate it.

Four Golden Signals

Monitor these and alert only when human action is needed:

Overload & Cascading Failures

• Exponential Backoff + Jitter: Clients must increase retry delay + add randomness to avoid thundering herd.
• Graceful Degradation: Under overload, return partial/cached response rather than HTTP 500.

Rollouts & Incidents

• Canary Releases: Roll out to 1% of servers first. Auto-rollback on error spike.
• Blameless Postmortems: Find the system failure, not the person. Fix the process.
• Triage Over Root-Cause: Mitigate first (rollback, restart), investigate later. Prod must recover in <5 minutes.

Pragmatic Survival

Actionable Alerts

• Delete Noisy Alerts: If an alert doesn't require immediate human action — delete it.
• Symptom-Based: Alert on symptoms (user can't log in), not causes (CPU at 80%).

Internal Tooling

• Treat Tools as Production Code: Debug scripts, migration tools, admin panels — build them as reliable CLIs with validation, not throwaway bash scripts.

DORA Metrics (High-Velocity Engineering)

Trunk-Based Development

• No Long-Lived Branches: Branches live max 1-2 days. Merge to main daily.
• Feature Flags: Hide unfinished work behind flags, not branches.
• Loosely Coupled: Teams deploy independently.