How do you size a system before you build it?

Vertical scaling makes one machine bigger (more CPU, RAM). Horizontal scaling adds more machines behind a load balancer.

Vertical is simple and needs no code change, but it has a hard ceiling and the machine is still a single point of failure. Horizontal has a much higher ceiling and survives a machine dying, but the app must be stateless and you now need load balancing, health checks and deployment across many nodes.

I usually scale the stateless web tier out from day one (2+ instances for redundancy) and scale the database up first, because databases are harder to split. Follow-up: "What do you do when the database maxes out?" → read replicas, caching, then sharding.

A stateless service keeps no per-user data in its own memory or disk between requests. Session data, carts and uploads live in a shared store (Redis, a database, S3) or in a signed token the client sends.

Why: any instance can serve any request. So the load balancer can use simple round robin, you can add or remove instances freely (autoscaling), and a crash loses nothing. Twelve-Factor calls sticky sessions "a violation".

Trade-off: every request now reads state over the network (a network round trip, instead of a ~100 ns local memory read), and the shared store must itself be made highly available.

Latency is the time for one request; throughput is how many requests complete per second.

Batching is the classic trade: if a service waits to collect 100 writes and flushes them together, throughput rises (one disk sync for 100 writes) but each write waits for the batch, so latency rises. Kafka producers do exactly this with a small linger time.

The other direction: running a server at 95% CPU maximises throughput per machine but queueing makes latency spike. That is why we leave headroom, often targeting around 60–70% utilisation (a rule of thumb, not a law).

Because the average hides the slow requests that users actually complain about. If 99 requests take 10 ms and one takes 2 s, the average is about 30 ms, which looks great, but 1 in 100 users waits two seconds.

p99 says "99% of requests were this fast or faster". The Google SRE book recommends high percentiles because they show "a plausible worst-case value".

Also, heavy users make many requests per page, so they hit the tail often. I would set SLOs on p50 and p99, and alert on p99.

99.9% allows about 43 minutes per 30-day month (0.001 × 30 × 24 × 60 = 43.2). 99.99% allows about 52.6 minutes per year (0.0001 × 525,600).

The quick rule: each extra nine cuts the budget by 10x. Three nines ≈ 8.8 hours a year; four ≈ 53 minutes; five ≈ 5 minutes.

Why it matters: at five nines, a human cannot even get paged and log in before the budget is gone, so recovery must be automatic. These numbers match the Google SRE book's availability table.

SLI is the measurement: for example, the fraction of HTTP requests that succeed in under 300 ms. SLO is the internal target on that SLI: 99.9% over a rolling 28 days. SLA is the external contract with consequences: below 99.5% in a month, customers get a 10% credit.

The SLO is stricter than the SLA so the team reacts before money is owed. 100% minus the SLO is the error budget; when it is spent, teams slow down risky launches.

Follow-up: "Why not target 100%?" → it is impossible to reach and the cost explodes; users cannot tell 99.999% from 100% when their own Wi-Fi is less reliable.

A SPOF is any component with no redundant copy, so its failure stops the whole system.

Here: the single load balancer and the single MySQL instance. The three app servers are fine if they are stateless and spread over zones. Less obvious ones: one DNS provider, one availability zone, one config or deploy pipeline that pushes a bad change everywhere.

Fixes: managed or paired load balancers, MySQL primary + replica with automatic failover (accepting a few seconds of failover time and possible replication lag), multi-AZ placement, and staged rollouts.

L = λ × W: average items in the system equals arrival rate times average time in the system. It holds for long-run averages in a stable system.

Example: the service sends 2,000 queries per second to the database and each holds a connection for 5 ms. L = 2,000 × 0.005 = 10 connections busy on average. I would give the pool maybe 2–3x that for bursts, so 20–30.

The scary part: if a slow query makes W jump to 500 ms, L jumps to 1,000. The pool runs out and requests queue. That is why we set query timeouts and watch pool wait time.

The user waits for the slowest of 100. The chance all 100 are fast is 0.99^100 ≈ 37%, so about 63% of user requests take over a second. This is the "Tail at Scale" example from Dean and Barroso.

Fixes: hedged requests (send a second copy after, say, the p95 time and take whichever finishes first; costs a few percent extra load), tied requests, timeouts with partial results ("good enough" search results), reducing fan-out with better partitioning, and fixing the causes of slow outliers (GC pauses, noisy neighbours, background jobs).

Follow-up: "What does hedging cost?" → extra load, roughly the share of requests that hit the hedge threshold.

In series both must work: 0.999 × 0.999 ≈ 99.8%. So 10 dependencies at 99.9% each give only about 99.0%.

Two independent copies of one service in parallel: 1 − 0.001² = 99.9999%. Then the chain of two such pairs is about 99.9998%.

The catch is "independent". Copies in the same zone, on the same version, sharing the same database, fail together. Real gains come from spreading across zones, staged deploys and removing shared dependencies. This is why microservice chains often have worse availability than a monolith unless calls are made optional or async.

Writes: 100M ÷ (30 × 86,400 ≈ 2.6M s) ≈ 40 writes/s; peak ×3 ≈ 120. Assume reads are 100:1 → about 4,000 reads/s average, 12,000 peak.

Storage: each record (short code 7 B, long URL ~100 B, timestamps, user ID) ≈ 500 B with overhead. 100M × 500 B = 50 GB a month, about 600 GB a year, 3 TB over five years.

So what: writes are tiny, reads are cache-friendly (hot links), and the data fits on one machine for years. I would use a cache in front of a key-value store and not shard on day one. All inputs are assumptions I would confirm with the interviewer.

Requests per day = 20M. Average QPS = 20M ÷ 86,400 ≈ 230; peak at 3x ≈ 700.

Servers depend on cost per request, which I would ask about or measure. If one instance handles about 500 RPS for this workload at safe CPU (an assumption to load-test), then 700 ÷ 500 → 2 servers, and I would run 3 for redundancy across zones (N+1).

The point is to show the chain: users → requests/day → average → peak → per-server capacity → count + redundancy. The per-server capacity must come from a load test, not a guess.

Users get logged out or lose carts at random, because request 2 lands on a server that does not have the session from request 1. In-memory caches also diverge between servers.

Quick fix: sticky sessions (LB pins a user to one server by cookie). It works, but load becomes uneven, a server crash still loses those sessions, and draining servers for deploys is slow.

Proper fix: move session state to Redis/Memcached with a TTL, or use signed tokens (like JWT) so the server needs no session store. Then any server can answer and autoscaling just works.

1,000,000 × 2 Mbps = 2,000,000 Mbps = 2 Tbps (terabits per second). In bytes that is 250 GB/s.

No single data center link or server fleet serves that cheaply. So the design must use a CDN with edge caches close to viewers, and pre-encoded segments at several bitrates so clients can step down on bad networks.

Common slip: mixing bits and bytes. Network speeds are quoted in bits; storage in bytes. Divide bits by 8.

Because requests start waiting in a queue. Near full use, any small burst arrives faster than the server can drain it, and the queue grows. In simple queueing models, waiting time grows roughly like 1 ÷ (1 − utilisation): at 50% it is modest; at 90% it is about 5x worse than at 50%; at 99% it explodes.

By Little's Law, more time waiting means more requests in flight, which uses up threads and connections, which causes timeouts and retries, which adds load: a feedback loop.

So we keep headroom, autoscale on queue length or latency (not only CPU), and shed load (return 429 or 503) instead of letting queues grow without limit.

The monthly budget at 99.9% is about 43 minutes of full downtime (or 0.1% of requests). Using 80% by day 10 means we are burning about 2.4x faster than sustainable.

Actions, per the SRE error-budget idea: freeze or slow risky launches, put engineers on reliability work, find the top causes of bad events, and add burn-rate alerts (alert when the budget would run out in hours or days, not when it is already gone).

Also check that the SLI is right: if users are happy, maybe the SLO is too strict. The budget is a tool to balance speed and reliability, not a punishment.

Messages: 50M × 40 = 2B a day ≈ 23,000 writes/s average, about 70,000 peak. That is write-heavy and too much for one SQL primary, so I would shard by conversation ID and use a store good at appends (for example Cassandra-style wide rows).

Storage: about 100 B of text + 100 B metadata ≈ 200 B per message → 400 GB a day → 146 TB a year, ×3 replicas ≈ 440 TB a year. Media goes to object storage instead.

Connections: 50M daily users, maybe 10M online at once, each holding a WebSocket. At an assumed 50,000–100,000 connections per gateway server, that is 100–200 gateway servers. All inputs are assumptions to confirm.

1) Measure first: find the actual bottleneck (CPU, DB queries, network). 2) Separate the database onto its own machine and scale it up. 3) Make the app stateless (sessions to Redis) and run several copies behind a load balancer across zones. 4) Add a cache for hot reads and a CDN for static files; this often removes most DB reads. 5) Add read replicas. 6) Move slow work (email, image resize) to a queue and workers. 7) Only then shard the database, because sharding is costly to undo.

Each step removes the current bottleneck and a SPOF. I would set p99 and availability SLOs first so I know when a step is "done".