Course · 03

How do you share traffic across servers, and stop a flood?

A load balancer spreads requests over many servers. A rate limiter says "slow down" to anyone who asks too much. This page explains both, from the picture down to the real config, with interview drills at the end.

—Words you need

Each jargon word once, in plain words glossary

01Why load balancers exist

One address outside, many servers inside core

One server has limits. It has a fixed CPU, memory and network. To serve more users, you add servers. This is called horizontal scaling (adding machines, not bigger machines).

But users need one address. The load balancer gives them one. It hides how many servers sit behind it. It also skips servers that are broken.

The LB gives users one stable address and turns many fragile servers into one reliable service.

Source: AWS: ALB health checks · AWS: deregistration delay

02L4 vs L7 load balancing

Does the balancer read the letter, or only the envelope? interview favourite

"Layer" comes from the OSI model, a 7-step picture of networking. Layer 4 is the transport layer (TCP, UDP). Layer 7 is the application layer (HTTP).

An L4 balancer picks a server per connection. It never reads the HTTP request. An L7 balancer reads the request first, then picks a server.

Pick L7 when routing depends on the request. Pick L4 for raw speed, non-HTTP traffic, or a fixed IP.

Source: AWS: NLB introduction · AWS: ALB HTTPS listener

03Load balancing algorithms

How the balancer picks the next server core

The algorithm is the rule that picks a server. Each rule fits a different kind of traffic. Here is a worked example with three servers A, B, C.

upstream api {
    least_conn;                        # default is weighted round robin
    server 10.0.0.1 weight=2;
    server 10.0.0.2 max_fails=3 fail_timeout=10s;
    server 10.0.0.3;
}
upstream cache_nodes {
    hash $request_uri consistent;      # ketama consistent hashing
    server 10.0.1.1;  server 10.0.1.2;
}

Default to round robin, switch to least connections when request lengths vary, and use consistent hashing when a key must keep landing on the same node.

Source: NGINX upstream module · AWS: ALB routing algorithm · HAProxy 2.8: balance · Karger et al. 1997, consistent hashing

04Health checks

How the balancer finds out a server is sick core

A health check is a small test request. The balancer sends it to every server on a timer. Too many failures in a row means "take it out". Enough passes in a row means "put it back".

There are two kinds. Active checks send a test request, like GET /health. Passive checks watch real traffic and count errors.

Worked example: a server dies right after a check. Two more checks fail, 30 s apart. So users may hit it for up to about 60 s. Then it takes 5 passes, about 150 s, to come back.

Make health checks cheap and local. A deep check that tests shared parts can mark every server sick at once.

Source: AWS: ALB health checks · AWS: slow start · NGINX: server max_fails

05Sticky sessions and TLS termination

Two features the LB offers, one you should mostly avoid interview favourite

Sticky sessions

A sticky session pins a user to one server. The LB sets a cookie that names the server. Later requests with that cookie go to the same server.

Teams use it when the server keeps user state in memory, like a shopping cart. It feels easy. It causes problems later.

The fix is a stateless server. Keep sessions in a shared store, like Redis, or in a signed token, like a JWT. Then any server can serve any request.

TLS termination

TLS is the encryption in HTTPS. Terminating TLS means the LB decrypts it. Servers then get plain HTTP inside the private network.

Terminate TLS at the LB, keep servers stateless, and treat sticky sessions as a last resort.

Source: AWS: ALB sticky sessions · AWS: ALB HTTPS listener

06Forward proxy, reverse proxy, load balancer, API gateway

Four middlemen that are easy to mix up interview favourite

Every LB and API gateway is a reverse proxy. The difference is the main job: spreading load versus managing APIs.

Source: RFC 9110 §3.7 · AWS: API Gateway throttling · Azure: Gateway Aggregation pattern

07Global load balancing, and the LB as a single point of failure

Picking a region, and keeping the front door itself alive deep dive

The load balancer is itself a single point of failure

If all traffic goes through one LB box, that box can kill the site. So the LB is always built as a pair or a fleet.

Use DNS or anycast to pick a region, a regional LB to pick a server, and always run the LB as more than one node.

Source: RFC 4786 anycast · AWS Global Accelerator · Route 53 routing policies · RFC 5798 VRRP

08Token bucket and leaky bucket

Two bucket pictures that behave very differently with bursts interview favourite

Token bucket. Tokens drip into a bucket at a fixed rate. The bucket has a maximum size. Each request takes one token. No token, no entry.

Worked example: rate 2 per second, capacity 10. A quiet client saves up 10 tokens. It can then send 10 requests at once. After that it gets only 2 per second.

Leaky bucket. Requests pour into a queue. The queue drains at a fixed rate, like water from a hole. If the queue is full, new requests spill over and are dropped.

Worked example: drain 2 per second, queue size 5. A burst of 8 arrives. 5 wait in line, 3 are dropped. The 5 go out one every half second.

limit_req_zone $binary_remote_addr zone=one:10m rate=1r/s;
location /search/ {
    limit_req zone=one burst=5 nodelay;   # nodelay: serve burst now, reject past it
    limit_req_status 429;                 # default is 503
}

Choose token bucket for APIs, where short bursts are fine. Choose leaky bucket when the thing behind you needs a steady pace.

Source: AWS: API Gateway throttling · NGINX limit_req · Stripe: Scaling your API with rate limiters (2017)

09Fixed window, sliding log, sliding window counter

Counting requests per minute, three ways deep dive

Fixed window. Keep one counter per minute, like user42:10:05. Reset it when the minute changes. Simple and cheap.

The flaw: the edges. Limit 100 per minute. A client sends 100 at 10:05:59 and 100 at 10:06:00. That is 200 in two seconds, and both windows say "fine".

Sliding window log. Store the time of every request. On each new request, drop times older than 60 s and count the rest. Exact, but it stores one entry per request.

Sliding window counter. Keep only two counters: last minute and this minute. Guess the last 60 s by weighting the old counter.

estimate = previous × (60 − 15) / 60 + current
         = 42 × 0.75 + 18
         = 49.5          # limit is 50, so this request is allowed

The guess assumes the old minute's requests were spread evenly. That is rarely exactly true, but it is close enough in practice.

The sliding window counter gives near-exact limits with just two numbers per key. It is the default answer for a high-volume API.

Source: Cloudflare blog, 2017

10Distributed rate limiting with Redis, HTTP 429, and where to put it

Many gateway copies, one shared count deep dive

You run ten gateway copies. Each keeps its own counter in memory. A client spread over all ten gets ten times the limit. So the count must live in one shared place. Redis is the usual choice: it is in memory and runs each command one at a time.

Why atomic matters

The naive code is "read count, check, then add one". Two gateways can read 99 at the same moment. Both allow. Now 101 got in. This is a race condition (two actors interleave and break a rule).

Also, INCR then EXPIRE as two calls can leak. If the app crashes between them, the key never expires. The fix: run both inside one Lua script. Redis runs a script as one step.

local current = redis.call("INCR", KEYS[1])     -- KEYS[1] = "rl:k1:" .. minute
if current == 1 then
  redis.call("EXPIRE", KEYS[1], ARGV[1])           -- ARGV[1] = 60 seconds
end
return current                                    -- caller rejects if current > limit

ZREMRANGEBYSCORE rl:k1 0 (now_ms - 60000)   # drop entries older than 60 s
ZADD rl:k1 now_ms request_id                # record this request
ZCARD rl:k1                                 # how many in the last 60 s
PEXPIRE rl:k1 60000

What to send back: 429 and Retry-After

HTTP/1.1 429 Too Many Requests
Retry-After: 30
Content-Type: application/json

{"error":"rate_limited","limit":100,"window":"60s"}

Retry-After tells the client how long to wait. A good client waits that long, then retries with backoff and jitter (random extra delay).

Where to put the limiter

Shared counter, atomic update, honest 429 with Retry-After. Put coarse limits at the edge and fine limits at the gateway.

Source: Redis INCR: rate limiter pattern · RFC 6585 §4 · RFC 9110 §10.2.3 · Stripe rate limiters

—When to use what

Situation, pick, why decision table

—Common mistakes and traps

Things that look right and break in production traps

—Interview questions

20 questions, easy to design-level. Try first, then open the answer drill

—Cheat sheet

Numbers and facts to remember memorise

—Sources

All checked 2026-10-25 references