System Design

Most system-design writing is aimed at interviews — whiteboard a URL shortener, name-drop “sharding,” move on. This guide is for engineers who then have to run the thing: page when it breaks, scale it when traffic triples, and explain the on-call cost of each architectural choice. Good system design is mostly the disciplined application of a few principles and an honest accounting of trade-offs. There is no free lunch — every capability you add (a cache, a queue, a second region) buys availability or performance and sells consistency, cost, or operational complexity.

Start with requirements

Design begins with constraints, not components.

• Functional requirements — what the system does (post a message, process a payment).
• Non-functional requirements — the ones that shape the architecture: expected traffic (reads vs writes, peak vs average), latency targets, availability target, data volume and growth, consistency needs, and budget.

Quantify them. “Highly available” is meaningless; “99.95% monthly (≈22 minutes downtime), p99 API latency under 200ms, 5k writes/s at peak” drives real decisions. A read-heavy system wants caching and read replicas; a write-heavy one wants partitioning and async processing. You cannot design without these numbers.

The core properties, and their tensions

• Scalability — handle more load by adding resources. Prefer horizontal (more stateless instances behind a balancer) over vertical (a bigger box) — horizontal has no ceiling and no single point of failure, but requires statelessness.
• Availability — the fraction of time the system serves requests. Achieved through redundancy: no single instance, zone, or dependency whose loss takes you down.
• Reliability — it does the right thing, not just responds. Availability without correctness is a fast wrong answer.
• Latency vs throughput — latency is per-request time; throughput is requests per second. Optimizing one can hurt the other (batching raises throughput but adds latency).

Scaling stateless services

Make the application tier stateless — no session in local memory, no local file state — and you can run N identical instances behind a load balancer and add more on demand. State moves to shared stores (a database, a cache, object storage). This is the foundation everything else builds on.

• Load balancing distributes requests (round-robin, least-connections) and, crucially, health-checks instances so traffic skips unhealthy ones. Add autoscaling on a real signal (CPU, request latency, queue depth).
• The database becomes the bottleneck precisely because the app tier scaled so easily — which is why the next sections are mostly about data.

Caching and CDNs

Caching is the highest-leverage performance tool and the richest source of subtle bugs.

• CDN — cache static assets (and cacheable API responses) at the edge, close to users. Cuts latency and offloads origin.
• Application cache (Redis/Memcached) — cache expensive query results and hot objects. The cache-aside pattern (check cache → miss → read DB → populate cache) is the common default.
• The hard part is invalidation — keeping cached data fresh enough. Use TTLs for tolerable staleness; explicit invalidation on write for data that must be current. Beware cache stampedes (many misses hitting the DB at once when a hot key expires) — mitigate with request coalescing or staggered TTLs.

Data: SQL, NoSQL, and scaling storage

Choose the data model for the access pattern, not the hype:

• Relational (SQL) — strong consistency, transactions, flexible queries, joins. The right default for most systems and anything transactional.
• NoSQL — document, key-value, wide-column, graph. Chosen for a specific access pattern, massive scale, or flexible schema — at the cost of joins and (often) strong consistency.

Scaling a database, in the order you typically reach for it:

1. Read replicas — replicate writes to read-only copies; route reads there. Solves read-heavy load. Introduces replication lag (a read right after a write may be stale).
2. Partitioning / sharding — split data across nodes by a shard key. The only way past a single node’s write ceiling — but it complicates queries that span shards and makes a good shard key a critical, hard-to-change decision.
3. Denormalization / caching — trade storage and write complexity for read speed.

Queues and event streaming

Asynchronous messaging decouples producers from consumers — the sender doesn’t wait, and a slow or down consumer doesn’t take the sender with it.

• Message queues (RabbitMQ, SQS) — work distribution, buffering spikes, retrying failures.
• Event streaming (Kafka) — a durable, replayable log of events; multiple consumers, event sourcing, analytics.

Async buys resilience and smoothing but adds real concerns: at-least-once delivery means consumers must be idempotent (processing a duplicate is a no-op); ordering may not be guaranteed across partitions; and you need a dead-letter queue for messages that never succeed.

APIs, gateways, and service boundaries

• API gateway — one entry point handling auth, rate limiting, routing, and TLS termination, so individual services don’t each reimplement them.
• Rate limiting protects you from abuse and from a single client overwhelming a shared resource — token-bucket is the common algorithm; enforce at the gateway.
• Microservices vs monolith — microservices let teams deploy independently and scale services separately, at the cost of network calls, distributed debugging, and operational overhead. A well-structured monolith is the right starting point for most teams; split out services when a specific scaling or team-autonomy need justifies the tax. Don’t buy distributed-systems complexity before you have a distributed-systems problem.

Designing for failure

At scale, something is always degraded. Design so that partial failure stays partial.

• Retries with backoff + jitter — retry transient failures, but exponentially and with randomness, or synchronized retries become a self-inflicted DDoS (“retry storm”).
• Circuit breakers — stop calling a failing dependency after a threshold, fail fast, and probe for recovery. Prevents one slow dependency from exhausting all your threads and cascading.
• Timeouts on everything — a call with no timeout is a resource leak waiting for a bad day. Every network call gets a bounded timeout.
• Graceful degradation — shed non-essential features under load (serve stale cache, disable recommendations) rather than failing entirely.
• Bulkheads — isolate resource pools so one overloaded subsystem can’t starve the rest.

Observability and DR

• Observability — metrics (rates, errors, latency), logs (structured, centralized), and traces (request flow across services). You cannot operate what you can’t see; design it in, don’t bolt it on. See DevOps Practices.
• Multi-region / DR — for the highest availability, run in more than one region. Understand your RPO (how much data you can lose) and RTO (how fast you must recover); active-active is the strongest and most expensive, active-passive a common balance. Whatever you choose, test the failover — an untested DR plan is fiction.

An architecture decision framework

Before adding a component or pattern, walk these five questions. They keep designs honest and reviews productive:

1. What requirement forces this? Tie it to a quantified NFR (a latency target, a write rate, an availability number). No number, no component.
2. What’s the simplest thing that meets it? A monolith + managed DB + CDN handles more scale than most teams admit. Start there.
3. What new failure mode does this introduce, and how will we detect it? If you can’t observe it, you can’t operate it.
4. What’s the operational cost? Who’s on call for it? What’s the runbook? Complexity you can’t run isn’t a solution.
5. Is this reversible? Prefer decisions you can undo (a cache) over ones you can’t (a shard key, a data model). Spend your one-way-door decisions carefully.

Worked example: a highly available web application

Putting it together for a read-heavy web app with a 99.95% target:

• Edge: CDN for static assets + cacheable responses; DNS with health-checked failover.
• Entry: load balancer across ≥2 availability zones, health-checking stateless app instances; autoscale on request latency.
• App tier: stateless services; sessions in Redis, not local memory.
• Data: primary relational DB with a synchronous standby (automatic failover) + read replicas for read load; cache-aside in Redis for hot data.
• Async: a queue for anything that doesn’t need to happen in the request path (emails, thumbnails, webhooks) with idempotent consumers and a DLQ.
• Resilience: timeouts + retries with backoff + circuit breakers on every external call; graceful degradation of non-critical features.
• Ops: metrics/logs/traces to a central stack; runbooks; a tested restore and a tested region-failover.

Every element maps to a requirement — nothing is there because it’s fashionable. That’s the whole discipline.

Frequently asked questions

How do I choose between SQL and NoSQL? Default to relational — it gives you transactions, consistency, and flexible queries. Reach for a specific NoSQL store when you have a well-understood access pattern, extreme scale, or schema flexibility needs that relational can’t meet, and you can live without joins/strong consistency for that data.

When should I move from a monolith to microservices? When a concrete need appears: a component must scale independently, teams need to deploy independently, or a bounded context is genuinely separate. Until then, a well-structured monolith is faster to build and far cheaper to operate. Don’t adopt distributed-systems complexity pre-emptively.

What’s the difference between horizontal and vertical scaling? Vertical = a bigger machine (simple, but has a ceiling and a single point of failure). Horizontal = more machines behind a balancer (no ceiling, built-in redundancy, but requires stateless services and shared state stores). Prefer horizontal for anything that must be highly available.

Why do retries sometimes make outages worse? Naive, immediate retries from many clients synchronize into a “retry storm” that hammers an already-struggling dependency. Use exponential backoff with jitter, cap attempts, and pair with circuit breakers so you stop retrying a dependency that’s clearly down.

What actually gets me to high availability? Redundancy at every layer (multi-AZ, ≥2 instances, DB failover), health-checked load balancing, timeouts + circuit breakers so failures stay contained, and a tested failover/restore. Availability is an operational property, not a diagram.

Related resources

• Guide: DevOps Practices for the operational practices (observability, SRE, DR) that keep these systems running.
• Guide: Cloud Security and Kubernetes Security for securing the architecture, and DevOps Tools for the toolchain.