Contents

Course · 14

How do you know it works, keep it up, and keep it safe?

Three jobs every real system has. Seeing what is going on inside (observability). Staying up when parts fail (reliability). Letting in only the right people (security). Every number was checked against official docs, RFCs or the Google SRE books on 2026-10-25.

TL;DR

Measure what users feel, alert on how fast you burn your error budget, plan for whole regions failing, and treat every token and secret as something that will leak.

Logs say what happened, metrics say how much, traces say where the time went. One trace ID ties all three together.
Alert on the SLO burn rate, not on CPU. Google's table: page when 2% of a 30-day budget burns in 1 hour (burn rate 14.4).
RPO = how much data you can lose. RTO = how long you can be down. Cheaper DR → bigger numbers.
Use OAuth 2.0 authorization code + PKCE for apps; keep JWT access tokens short-lived because they are hard to revoke.
Hash passwords with Argon2id (min 19 MiB, 2 passes) or bcrypt (cost ≥ 10), per OWASP. The current OWASP Top 10 is the 2025 edition.

golden signals (Google SRE book)

14.4

burn rate that pages: 2% of budget in 1 hour

43–128

PKCE code_verifier length, in characters

600,000

PBKDF2-HMAC-SHA256 iterations (OWASP)

—Words you need

Each hard word once, in plain words glossary

Observability

How well you can tell what is happening inside from outside.

Span / trace

Span = one timed step. Trace = all steps of one request.

SLI / SLO

SLI = a measured number. SLO = the target for it.

Error budget

How much failure the SLO allows: 1 minus the SLO.

Burn rate

How fast you use the error budget. 1 = exactly on pace.

RPO / RTO

RPO = data you may lose. RTO = time you may be down.

Load shedding

Saying "no" to some requests early so the rest succeed.

AuthN / AuthZ

AuthN = who are you? AuthZ = what may you do?

JWT

A signed, self-contained token: header.payload.signature.

OAuth 2.0

A way to give an app limited access without sharing your password.

PKCE

A one-time secret proving the code is redeemed by the same app.

OIDC

mTLS

TLS where both sides show certificates, not only the server.

Salt

Random bytes mixed into each password before hashing.

Observability

01The three pillars: logs, metrics, traces

Each one answers a different question core

One instrumented service, three kinds of signal, three kinds of store.

Pillar	What it is	Question it answers	Cost and limit
Logs	Timestamped event records	"What exactly happened to order 42?"	Huge volume; expensive to keep and search
Metrics	Numbers over time, with labels	"Is the error rate up right now? Since when?"	Cheap and fast, but no per-request detail
Traces	The path and timing of one request across services	"Which service made checkout slow?"	Usually sampled; needs context passed between services

In everyday words

A car: metrics are the dashboard dials, logs are the trip diary, and a trace is the GPS track showing where you got stuck in traffic.

Actually

OpenTelemetry (OTel) is the open standard for making all three. It defines a trace as "the path of a request through your application" and a span as "a unit of work or operation". Context propagation lets signals "be correlated with each other, regardless of where they are generated". Source: OpenTelemetry: traces, context propagation.

Metrics tell you something is wrong; traces tell you where; logs tell you why.

Sources: opentelemetry.io · checked 2026-10-25

02Structured logs and correlation IDs

Logs a machine can filter, joined by one ID core

A plain text log like user 7 paid 20 is easy to read and hard to search. A structured log is one JSON object per line, with named fields. Now you can ask "all errors for order_id=42" in one query.

A correlation ID (or trace ID) is created at the edge and passed to every service in a header. Every log line carries it. One search then shows the whole story of one request.

One structured log line

{"ts":"2026-10-25T10:02:11.482Z","level":"error","service":"payments",
 "trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","span_id":"00f067aa0ba902b7",
 "order_id":42,"msg":"card declined","provider_code":"do_not_honor"}

W3C Trace Context header passed between services

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
#            version-trace-id (32 hex)-parent span id (16 hex)-flags (01 = sampled)

Never log passwords, tokens, or full card numbers.
Use levels (debug, info, warn, error) and sample noisy debug logs.

In everyday words

A parcel tracking number. Every depot scans it, so you can follow one parcel across many depots.

Actually

traceparent is a W3C Recommendation (Trace Context Level 1, 23 Nov 2021): version (2 hex), trace-id (32 hex), parent-id (16 hex), trace-flags (2 hex). The OTel SDK's default propagators are tracecontext,baggage (env var OTEL_PROPAGATORS). Source: W3C Trace Context, OTel SDK environment variables.

JSON logs + a trace ID in every line turn a log pile into something you can query.

Sources: w3.org/TR/trace-context · OTel SDK config spec

03Metrics: types and what to watch

Counter, gauge, histogram; RED, USE and the four golden signals interview favourite

Type (Prometheus)	What it does	Example
Counter	Only goes up (or resets to 0 on restart). You read its `rate()`.	`http_requests_total`
Gauge	A value that goes up and down	Queue length, memory in use
Histogram	Counts values into buckets, plus a sum; percentiles are computed later	Request duration → p99 latency
Summary	Computes quantiles in the client over a sliding window	Latency quantiles for one instance (cannot be added across instances)

Why histograms for latency? Averages hide pain. If 1% of requests take 5 s, the average still looks fine. The p99 (the value 99% of requests beat) shows it.

Method	Signals	Use it for
Four golden signals (Google SRE book)	Latency, traffic, errors, saturation	Any user-facing system
RED (Tom Wilkie)	Rate, errors, duration	Each request-driven service
USE (Brendan Gregg)	Utilization, saturation, errors	Each resource: CPU, disk, network, pool

In everyday words

RED is a restaurant counting orders, mistakes and wait time. USE is checking each oven: how busy, how long the queue, any breakdowns.

Actually

The SRE book says: "If you can only measure four metrics of your user-facing system, focus on these four": latency, traffic, errors, saturation. It also warns to separate the latency of failed requests from successful ones. Brendan Gregg defines saturation as "the degree to which the resource has extra work which it can't service, often queued". Source: SRE book ch. 6, USE method, RED method, Prometheus metric types.

Say "RED for services, USE for resources, golden signals for the user view", and use histograms for latency.

Sources: sre.google · prometheus.io · brendangregg.com · grafana.com

04Distributed tracing

Spans, parents and children, and sampling core

Each bar is a span. Nesting shows who called whom; length shows how long it took. Example data, not real timings.

Every span has a trace ID (shared by the whole request), its own span ID, a parent span ID, a start time, a duration, and attributes like http.status_code. The tracing UI rebuilds the tree from the parent links.

Sampling keeps cost down. Head sampling decides at the start (say, keep 10%). Tail sampling decides after the request ends, so it can keep all errors and slow requests. Tail sampling needs a collector that buffers whole traces.

In everyday words

A stopwatch for each person in a relay race, plus a note of who handed the baton to whom.

Actually

The trace only joins up if every hop forwards the context. Inside a process the OTel SDK does it; across HTTP it is the traceparent header; across a queue like Kafka you must put the context in message headers yourself (or use an instrumented client). A missing hop splits one request into two unrelated traces. Source: OTel context propagation.

Traces answer "where is the time going" across services; propagate context through every hop, including queues.

Sources: OTel traces · OTel context propagation

05SLOs, error budgets and burn-rate alerts

Alert on user pain, at the right speed interview favourite

An SLI is a measurement, such as "share of requests that succeed in under 300 ms". An SLO is the target: "99.9% over 30 days". The error budget is what is left: 1 − 0.999 = 0.1% of requests may fail.

In time terms, 0.1% of 30 days is 43.2 minutes (30 × 24 × 60 × 0.001; our arithmetic, not a quote). Budget left → ship features. Budget gone → slow down and fix reliability.

Burn rate = how fast you spend the budget. Burn rate 1 spends it exactly in 30 days. Burn rate 14.4 spends 2% of it in one hour.

Recommended burn-rate alerts for a 30-day SLO

Burn rate that triggers each alert. Hover a bar for its window and action. From the Google SRE Workbook, ch. 5.

Fast burn: 2% in 1 hour page

14.4

Medium: 5% in 6 hours page

Slow: 10% in 3 days ticket

Show the full table

Budget used	Long window	Short window	Burn rate	Action
2%	1 hour	5 minutes	14.4	Page
5%	6 hours	30 minutes	6	Page
10%	3 days	6 hours	1	Ticket

Prometheus-style rule · fast burn, both windows must agree

(error_ratio[1h] > 14.4 * 0.001) and (error_ratio[5m] > 14.4 * 0.001)
# 0.001 = error budget for a 99.9% SLO; the 5m window makes the alert stop fast after recovery

In everyday words

A monthly phone data plan. Using it a bit fast is a note to yourself. Using a fifth of it in one afternoon is an emergency call.

Actually

The SRE Workbook defines burn rate as "how fast, relative to the SLO, the service consumes the error budget" and recommends multiwindow, multi-burn-rate alerts, with the short window "1/12 the duration of the long window". The long window catches real burns; the short one makes the alert clear soon after recovery. Source: SRE Workbook: Alerting on SLOs, SRE book: Embracing risk.

Page on fast budget burn, ticket on slow burn, and never page on CPU alone.

Sources: sre.google/workbook/alerting-on-slos · sre.google/sre-book/embracing-risk

Reliability

06Staying up under stress

Redundancy, graceful degradation, load shedding, chaos engineering core

Redundancy

No single point of failure

Run at least 2 of everything, in different AZs. Health checks remove bad copies automatically.

Graceful degradation

Lose a feature, not the product

Recommendations down? Show "popular items". Search down? Hide the box. Never block checkout on a nice-to-have.

Load shedding

Reject early, stay fast

When near overload, return 503 or 429 fast for some requests, so accepted ones still meet their latency.

Chaos engineering

Break it on purpose, in a controlled way

Kill instances, add latency, cut a dependency. Check that alerts fire and fallbacks work, before a real outage does it.

Why shed load instead of queueing everything? A server past its limit gets slower for every request. Clients time out and retry, which adds more load. Goodput (useful, on-time responses) can fall to almost zero.

In everyday words

A full restaurant that says "one hour wait" at the door serves the seated guests well. One that seats everyone serves nobody well.

Actually

Amazon's Builders' Library: "When a server approaches overload, it should start rejecting excess requests so that it can focus on the requests it decides to let in." Principles of Chaos defines chaos engineering as "the discipline of experimenting on a system in order to build confidence in the system's capability to withstand turbulent conditions in production". Netflix's Chaos Monkey is "a resiliency tool that helps applications tolerate random instance failures". Source: AWS: Using load shedding to avoid overload, principlesofchaos.org, Netflix Chaos Monkey.

Plan for failure: two of everything, fallbacks for extras, fast rejection when full, and drills to prove it.

Sources: AWS Builders' Library · principlesofchaos.org · github.com/Netflix/chaosmonkey

07Disaster recovery: RPO, RTO, backups, regions

What happens when a whole data center or region is gone interview favourite

RPO looks backward from the failure; RTO looks forward.

RPO (recovery point objective): the most data you accept losing, measured in time. Backups every 24 hours → RPO up to 24 hours. RTO (recovery time objective): the longest you accept being down.

AWS DR strategy	What runs in the second region	RPO / RTO	Cost
Backup & restore	Only backups; build everything after the disaster	Largest	Lowest
Pilot light	Data is replicated; servers are off or minimal	Smaller	Low
Warm standby	A smaller, working copy of everything	Smaller still	Medium
Multi-site active/active	Full copy serving live traffic	Near zero	Highest

Active-passive. In active-active, both regions take traffic and both accept writes.

Setup	Protects against	Main cost or catch
Multi-AZ (one region)	One data center failing	Cheap and common; AZs are close, so sync replication is fine
Multi-region active-passive	A whole region failing	Async copy → some data loss (RPO above 0); failover must be tested
Multi-region active-active	A region failing, with near-zero RTO	Writes in two places → conflicts; needs careful data design

Backups

3-2-1 rule: 3 copies, on 2 different media types, 1 off-site.
Full + incremental backups, plus point-in-time recovery from database logs.
Keep at least one copy that attackers cannot delete (separate account, write-once lock).
A backup you have never restored is a hope, not a backup. Run restore drills.

In everyday words

RPO: how many pages of your diary you can afford to lose. RTO: how long you can live without the diary before you get it back.

Actually

AWS defines RTO as the "maximum acceptable delay between the interruption of service and restoration of service" and RPO as the "maximum acceptable amount of time since the last data recovery point". It says active/active "can reduce your recovery time to near zero". An AZ is "one or more discrete data centers with redundant power, networking, and connectivity"; all AZs in a Region are "within 100 km (60 miles) of each other". The 3-2-1 rule is in the US-CERT/CISA paper "Data Backup Options". Exact RPO/RTO ranges per strategy appear only in a figure we could not read, so they are not proven here. Source: AWS DR whitepaper, AWS Regions and AZs, CISA: Data Backup Options.

Ask the business for RPO and RTO first; they pick the DR strategy and its cost.

Sources: AWS DR whitepaper: BCP · DR options · AWS global infrastructure · CISA

Security

08Authentication, authorization, sessions vs tokens

Who are you, and what may you do? core

Authentication (AuthN) checks identity: password, passkey, one-time code. Authorization (AuthZ) checks permission: may user 7 delete order 42? Getting AuthZ wrong is the number-one web risk (see section 13).

Approach	How it works	Good	Bad
Server session	Server stores session data; browser holds a random session ID in a cookie	Easy to revoke (delete the row); small cookie	Needs a shared session store (Redis) to scale
Self-contained token (JWT)	Token carries signed claims; any server can check it without a lookup	Stateless, fast across services	Hard to revoke before it expires; bigger; claims can go stale

Session cookies should be HttpOnly (JavaScript cannot read them), Secure (HTTPS only) and SameSite (limits cross-site sending, which helps against CSRF).

In everyday words

A session is a coat-check ticket: the cloakroom keeps your coat and can cancel the ticket. A JWT is a signed visitor badge: any guard can read it, but you cannot take it back until it expires.

Common pattern

Browser apps: a server session or a short-lived access token plus a refresh token kept in an HttpOnly cookie. Service-to-service: short-lived tokens or mTLS. Authorization checks happen on the server for every request, never only in the UI. Source: OWASP Session Management Cheat Sheet.

AuthN once, AuthZ on every request, on the server.

Sources: OWASP Session Management Cheat Sheet

09JWT: structure and pitfalls

Three base64url parts, signed, and hard to take back interview favourite

Base64url is encoding, not encryption: anyone can read the payload.

Decoded payload

{ "iss": "https://auth.example.com", "sub": "user_7", "aud": "orders-api",
  "exp": 1761390000, "iat": 1761389100, "scope": "orders:read" }

Pitfall

Cannot revoke easily

A stolen token works until exp. Keep access tokens short (minutes), use refresh tokens you can revoke, or keep a deny-list of jti values.

Pitfall

Trusting the token's own alg

Pin allowed algorithms on the server. Reject none. Watch the RS256 → HS256 confusion attack.

Pitfall

Secrets in the payload

The payload is only encoded. Never put passwords or personal data you would not show the user.

Pitfall

Skipping claim checks

Check signature, exp, iss and aud. A token for another API must be rejected.

Actually

RFC 7519 defines a JWT as base64url parts joined by dots; the registered claims (iss, sub, aud, exp, nbf, iat, jti) are all optional in the spec, so your validator must require the ones you need. RFC 8725 (JWT Best Current Practices) says libraries "MUST enable the caller to specify a supported set of algorithms and MUST NOT use any other algorithms", and describes using an RSA public key as an HMAC secret. Source: RFC 7519, RFC 8725.

JWTs trade revocation for speed. Keep them short-lived and validate every claim you rely on.

Sources: RFC 7519 · RFC 8725 · RFC 7009 (token revocation)

10OAuth 2.0 authorization code + PKCE, and OpenID Connect

The standard way apps get tokens interview favourite

OAuth 2.0 lets a user give an app limited access to their data on another service, without giving the app their password. The app gets an access token with a scope (what it may do).

PKCE (say "pixy") protects the code. The app makes a random secret, the code_verifier. It sends only a hash of it, the code_challenge, at the start. To swap the code for tokens it must show the original verifier. A thief who steals the code does not have the verifier.

Steps 2–4 travel through browser redirects; step 5 is a direct back-channel call.

App makes code_verifier (random, 43–128 characters) and code_challenge = BASE64URL(SHA256(verifier)).
App redirects the browser to /authorize?response_type=code&client_id=…&redirect_uri=…&scope=…&state=…&code_challenge=…&code_challenge_method=S256.
User signs in and agrees.
Auth server redirects to redirect_uri?code=…&state=…. App checks state matches (stops CSRF).
App POSTs to /token with the code and the code_verifier.
Auth server hashes the verifier, compares with the challenge, and returns an access token (plus refresh token; with OIDC, an ID token).
App calls the API with Authorization: Bearer <token> (RFC 6750).

OpenID Connect (OIDC) adds login on top. Ask for scope openid and you also get an ID token: a signed JWT saying who the user is. The access token is for the API; the ID token is for your app.

In everyday words

You tear a ticket in half. You hand the hotel a photo of your half at check-in, and you show the real half to pick up the key. Someone who only grabbed the key slip cannot pick it up.

Actually

RFC 6749 §4.1 defines the five steps (A–E) of the code grant, including that the server checks the redirect_uri matches. RFC 7636 sets the verifier at 43–128 characters and S256 as BASE64URL-ENCODE(SHA256(ASCII(code_verifier))); plain is allowed only if the client technically cannot do S256. RFC 9700 (OAuth 2.0 Security BCP, January 2025) says clients "SHOULD NOT use the implicit grant", public clients MUST use PKCE, and PKCE is RECOMMENDED for confidential clients too. OAuth 2.1 is still an Internet-Draft as of 2026-10-25. OIDC Core requires iss, sub, aud, exp, iat in the ID token. Source: RFC 6749, RFC 7636, RFC 9700, OIDC Core 1.0.

For any app, say "authorization code with PKCE (S256), short access tokens, revocable refresh tokens, and OIDC for login".

Sources: RFC 6749 · RFC 7636 · RFC 9700 · RFC 6750 · OAuth 2.1 draft · OIDC Core

11API keys, mTLS, and encryption in transit vs at rest

Identifying machines, and keeping bytes private core

Tool	What it proves	Use for	Watch out
API key	"This caller has the key"	Identifying a project or partner, rate limits, simple server-to-server	Long-lived shared secret; no user identity; rotate and scope it; never in mobile or browser code
OAuth token	"This user allowed this app this scope"	Acting for a user	Expiry and refresh handling
mTLS	Both sides hold a private key for a trusted certificate	Service-to-service in a zero-trust network, service mesh, partner APIs	Certificate issuing and rotation must be automated

Encryption in transit protects data moving on the network: TLS (HTTPS) everywhere, including inside your VPC. Encryption at rest protects stored data: disks, databases, backups, S3 objects.

At rest, clouds use envelope encryption: a data key encrypts the data, and a master key in a key service (like AWS KMS) encrypts the data key. The master key never leaves the key service, and you can audit and revoke its use.

In everyday words

In transit is a locked armoured van. At rest is a locked safe. You need both: thieves rob vans and break into houses.

Actually

TLS 1.3 was RFC 8446 (2018); it is now obsoleted by RFC 9846 (2026), which is the current TLS 1.3 spec. RFC 8705 defines OAuth mTLS client authentication and certificate-bound access tokens (the x5t#S256 claim), so a stolen token is useless without the client's private key. Source: RFC 9846, RFC 8705, AWS KMS cryptography (envelope encryption).

Keys identify callers, tokens carry user consent, mTLS proves machine identity; encrypt both on the wire and on disk.

Sources: RFC 9846 · RFC 8446 · RFC 8705 · AWS KMS docs

12Passwords, secrets and least privilege

Slow hashes, salts, vaults, and the smallest permission that works core

Never store passwords. Store a slow, salted hash. Fast hashes like SHA-256 are bad here: attackers with GPUs try billions of guesses per second. Password hashes are slow on purpose, and Argon2id and scrypt also need lots of memory, which GPUs lack.

A salt is random bytes stored next to each hash. Two users with the same password get different hashes, so one precomputed table cannot crack everyone. A pepper is an extra secret kept outside the database (optional).

Algorithm (OWASP order)	Minimum settings (OWASP)	Note
Argon2id	19 MiB memory, 2 iterations, 1 thread	First choice
scrypt	N = 2^17 (128 MiB), r = 8, p = 1	If Argon2id is not available
bcrypt	Work factor ≥ 10	Only uses the first 72 bytes of input
PBKDF2	600,000 iterations, HMAC-SHA-256	When FIPS-140 compliance is required

Secrets management

No secrets in code, Git, images or logs. Load them at runtime from a vault (AWS Secrets Manager, HashiCorp Vault).
Rotate automatically; prefer short-lived, dynamic credentials (for example IAM roles instead of access keys).
Least privilege: each service gets only the permissions it needs, on only the resources it needs. Review and trim.

In everyday words

A slow hash is a lock that takes a full second to try each key. You barely notice once; a burglar with a million keys gives up.

Actually

All settings above are from the OWASP Password Storage Cheat Sheet. NIST SP 800-63B-4 says "passwords SHALL be salted and hashed using a suitable password hashing scheme", with a salt of at least 32 bits. OWASP's Secrets Management Cheat Sheet says to "standardize and centralize" secrets and automate rotation. AWS IAM best practices say "apply least-privilege permissions". Source: OWASP Password Storage, NIST SP 800-63B-4, OWASP Secrets Management, AWS IAM best practices.

Argon2id with a per-user salt, secrets from a vault, and the smallest permissions that still work.

Sources: OWASP Password Storage Cheat Sheet · NIST SP 800-63B-4 · checked 2026-10-25

13OWASP Top 10, DDoS and CORS

The headline web risks and two common edge defences core

The current list is OWASP Top 10:2025 (the 8th edition). It replaced the 2021 list, so do not quote the old names in an interview.

#	Risk	One-line example
A01	Broken Access Control	Change `/orders/42` to `/orders/43` and see someone else's order
A02	Security Misconfiguration	Public S3 bucket, debug mode on in production
A03	Software Supply Chain Failures	A hijacked npm package in your build
A04	Cryptographic Failures	Passwords hashed with plain SHA-1, no TLS
A05	Injection	SQL built by string concatenation
A06	Insecure Design	No rate limit on "reset password"
A07	Authentication Failures	No lockout or MFA; sessions never expire
A08	Software or Data Integrity Failures	Auto-update without signature checks
A09	Security Logging and Alerting Failures	A breach nobody notices for months
A10	Mishandling of Exceptional Conditions	An error path that "fails open" and skips the auth check

DDoS protection basics

A DDoS (distributed denial of service) floods you from many machines. Defend in layers: absorb at the edge (CDN, anycast), filter network floods (layers 3–4), rate-limit and use a WAF (web application firewall) for HTTP floods (layer 7), and auto-scale behind it. Keep origin servers hidden behind the edge.

Actually

AWS says: "All AWS customers benefit from the automatic protection of Shield Standard, at no additional charge. Shield Standard defends against the most common, frequently occurring network and transport layer DDoS attacks." HTTP-level floods need AWS WAF rate rules or Shield Advanced. Source: AWS Shield Standard, AWS DDoS resiliency whitepaper.

CORS in one paragraph

Browsers block a page on app.com from reading responses from api.other.com unless that server allows it. CORS (cross-origin resource sharing) is how the server allows it: it answers with Access-Control-Allow-Origin: https://app.com. For "non-simple" requests (for example PUT, or JSON bodies) the browser first sends a preflight OPTIONS request asking what is allowed. The wildcard * is not allowed together with credentials. CORS is enforced only by browsers; it is not an auth system, and curl ignores it.

In everyday words

CORS is a guest list the API hands to the browser's bouncer. Someone who walks in without a browser never meets that bouncer.

Know the 2025 Top 10 headline names, put DDoS defence at the edge, and treat CORS as a browser rule, not security for your API.

Sources: OWASP Top 10:2025 · AWS Shield · MDN: CORS · checked 2026-10-25

Practice

—When to use what

Situation → pick → why decision table

Situation	Pick	Why
"Is the service healthy right now?"	Metrics + SLO dashboards	Cheap, fast, aggregated
"Why did this one request fail?"	Logs filtered by trace ID	Full detail for one event
"Which of 12 services is slow?"	Distributed tracing	Shows time per hop
Deciding when to page someone	Multiwindow burn-rate alerts	Pages on user pain, not noise
Traffic spike past capacity	Load shedding + autoscaling	Keep goodput while scaling up
Must survive one data center loss	Multi-AZ	Cheap, sync replication works
Must survive region loss, some minutes OK	Pilot light or warm standby	Balance cost and RTO
Near-zero downtime, global users	Active-active multi-region	Lowest RTO; accept conflict handling
Web or mobile app login	OIDC + authorization code + PKCE	Current best practice (RFC 9700)
Need instant logout everywhere	Server sessions, or short JWT + revocable refresh token	JWTs alone cannot be recalled
Service-to-service identity	mTLS (service mesh) or short-lived tokens	No shared static secrets
Storing user passwords	Argon2id with salt	OWASP first choice

—Common mistakes

Traps that cost marks and cause outages traps

Trap

Alerting on causes, not symptoms

CPU 90% pages at 3 a.m. while users are fine. Alert on SLO burn; use CPU for dashboards.

Trap

Averages for latency

The mean hides the slow tail. Track p95/p99 with histograms.

Trap

User IDs as metric labels

Every label value is a new time series. Cardinality explodes. Put IDs in logs and traces.

Trap

Untested backups and failover

Restores and failovers fail when first tried in a crisis. Drill them.

Trap

Retries without limits

Every layer retrying 3 times multiplies load during an outage. Use backoff, jitter and retry budgets.

Trap

Long-lived JWTs

A 30-day access token is a 30-day key for a thief. Use minutes plus refresh.

Trap

Implicit grant or no PKCE

RFC 9700 says do not use implicit; use code + PKCE.

Trap

Fast hashes for passwords

SHA-256 or MD5 cracks quickly on GPUs. Use Argon2id, scrypt or bcrypt.

Trap

Thinking CORS protects the API

Only browsers enforce it. Do real auth on the server.

—Interview questions

20 questions, easy to design-level. Try first, then open the answer practice

1. What are logs, metrics and traces, and what does each answer? easy

Show answer

Metrics are numbers over time with labels, like request rate or p99 latency. They are cheap, fast to query and good for dashboards and alerts: "is something wrong, and since when?" Logs are detailed event records: "what exactly happened to this request?" They are expensive at volume, so keep them structured and sampled. Traces follow one request across services as a tree of timed spans: "which hop made it slow or failed?" They are usually sampled. In practice I start from an alert on a metric, jump to example traces for the bad period, and then read the logs for the failing span using its trace ID. OpenTelemetry is the standard way to produce all three with shared context. Follow-up: "Which would you add first to a new service?" RED metrics, then structured logs with trace IDs, then tracing.

2. What is a correlation ID and how does it flow through a system? easy

Show answer

It is a unique ID for one request, created at the edge (load balancer or gateway) or by the first service, and passed along in a header to every downstream call. Every log line includes it, so one search shows the full request story across services. The standard today is the W3C traceparent header: a 32-hex trace ID plus the 16-hex parent span ID and flags, which OpenTelemetry propagates by default. Across message queues you must copy it into message headers, or the chain breaks. Return it to the client in a response header too, so support can ask users for it. Follow-up: "What if a third-party service drops the header?" You lose the link for that hop; start a new span linked to the old trace on return.

3. Counter vs gauge vs histogram: give an example of each. easy

Show answer

A counter only goes up, or resets to zero on restart: http_requests_total, errors_total. You graph its rate(), never the raw value. A gauge goes up and down: queue depth, memory in use, open connections. A histogram counts observations into buckets plus a total sum and count: request duration. From buckets you compute percentiles like p99 on the server side, and you can aggregate across instances, which a Prometheus summary cannot do because it computes quantiles per instance. Choose bucket edges around your SLO threshold (for example 100, 300, 1000 ms) so you can measure "share under 300 ms" exactly. Follow-up: "Why not an average latency gauge?" It hides tail latency and cannot be correctly averaged across instances.

4. Explain RED, USE and the four golden signals. medium

Show answer

The four golden signals from the Google SRE book are latency, traffic, errors and saturation: "if you can only measure four metrics of your user-facing system, focus on these four". RED, from Tom Wilkie, is for request-driven services: rate, errors, duration. It is basically the golden signals minus saturation, applied to every microservice in the same way so dashboards look alike. USE, from Brendan Gregg, is for resources: for every CPU, disk, network link or connection pool, check utilization, saturation (queued work) and errors. I use RED on each service, USE on the hosts and pools under it, and golden signals for the user-facing SLO. Follow-up: "Why separate error latency?" A fast failure can make latency look better while users suffer.

5. What is an error budget and how does it change team behaviour? medium

Show answer

The error budget is 1 minus the SLO. With a 99.9% success SLO over 30 days, 0.1% of requests may fail; in time terms that is about 43.2 minutes of full downtime. It turns reliability into a shared, measurable resource. While budget remains, the team ships features and takes risks. When it is spent, the policy shifts work to reliability: freeze risky launches, fix the top causes. This ends the dev-versus-ops argument with data, and stops chasing 100%, which is too expensive and invisible to users behind their own imperfect networks. The policy must be agreed in advance with product. Follow-up: "What SLO would you pick?" One slightly below what users already get and accept, measured close to the user, like at the load balancer.

6. What is a burn-rate alert, and why use two windows? hard

Show answer

Burn rate is how fast you spend the error budget relative to the SLO; burn rate 1 uses it exactly in the SLO period. Alerting on burn rate pages for real user impact in proportion to severity. The SRE Workbook recommends: page at burn rate 14.4 over 1 hour (2% of a 30-day budget), page at 6 over 6 hours (5%), and ticket at 1 over 3 days (10%). Each alert also needs a short window, 1/12 of the long one (5 min for 1 hour), to agree. The long window avoids paging on brief blips; the short window makes the alert stop soon after the problem is fixed, instead of firing for the rest of the hour. Follow-up: "Low-traffic service?" Too few requests make ratios noisy; use synthetic probes or merge services.

7. How does distributed tracing work, and how do you control its cost? medium

Show answer

Each unit of work creates a span with trace ID, span ID, parent ID, start, duration and attributes. Context (trace ID and parent span) is propagated in headers like traceparent, so downstream spans join the same trace. Spans are exported to a collector, then a backend like Jaeger or Tempo rebuilds the tree as a waterfall. Cost control is sampling: head sampling decides at the start (keep 5%), which is cheap but may drop the interesting failures; tail sampling decides after the trace finishes, keeping all errors and slow traces plus a small share of normal ones, but the collector must buffer whole traces. Also limit attribute size and drop noisy internal spans. Follow-up: "How do you link logs to traces?" Put trace and span IDs into every log line.

8. What is load shedding and how is it different from rate limiting? medium

Show answer

Rate limiting caps each client at an agreed rate, for fairness and abuse control, even when the server is idle. Load shedding protects the server itself: when it nears overload, it rejects excess work quickly so the requests it accepts still finish on time. Amazon's Builders' Library puts it as rejecting excess requests "so that it can focus on the requests it decides to let in". Without it, latency rises for everyone, clients time out and retry, and goodput collapses. Shed cheaply and early (at the load balancer or before parsing), prefer to drop low-priority traffic first, and drop requests that have waited longer than the client's timeout. Return 503 with Retry-After. Follow-up: "What signal triggers shedding?" Concurrency in flight or queue wait time, not CPU alone.

9. What is graceful degradation? Give an example design. easy

Show answer

It means losing non-essential features instead of the whole product when a dependency fails. In an e-commerce page, the core path is product details, cart and checkout. If the recommendation service is slow, the page calls it with a short timeout and a circuit breaker, and falls back to a cached "popular items" list. If reviews are down, the page hides the reviews section. If search is down, category browsing still works. Each dependency is marked critical or optional, optional calls are async or time-boxed, and fallbacks are tested. The result: users can still buy. Follow-up: "How do you know fallbacks work?" Chaos experiments and game days that turn dependencies off on purpose.

10. What is chaos engineering? How would you start safely? medium

Show answer

It is experimenting on a system to build confidence that it can withstand turbulent conditions in production (principlesofchaos.org). The method: define steady state with a metric (for example checkout success rate), form a hypothesis ("losing one instance does not change it"), inject a real-world fault (kill an instance, add 200 ms latency, block a dependency), and compare. Start small: staging first, then production with a tiny blast radius, a clear abort button, during business hours with the team watching. Grow to zone failures and game days. Netflix's Chaos Monkey randomly terminates instances so teams build for it. The value is finding weak fallbacks, missing alerts and bad timeouts before a real outage does. Follow-up: "What if the experiment hurts users?" Abort automatically when the steady-state metric drops.

11. Explain RPO and RTO and pick a DR strategy for a payments system. hard

Show answer

RPO is the maximum acceptable data loss, as time since the last recovery point; RTO is the maximum acceptable downtime. For payments, losing committed transactions is unacceptable, so RPO near zero inside a region: synchronous replication across AZs (for example a Multi-AZ database). For a region loss, async cross-region replication gives an RPO of seconds; to approach zero across regions you need a globally consistent database or synchronous cross-region commit, which costs latency. RTO maybe 15 minutes: warm standby in a second region, with automated, rehearsed failover and idempotent payment APIs so retries after failover do not double-charge. Backup and restore alone would be far too slow. Follow-up: "How do you avoid split brain?" A single authority (or quorum) decides the active region, and the old one is fenced off.

12. Multi-AZ vs multi-region, active-passive vs active-active? medium

Show answer

AZs are separate data centers within one region, within 100 km of each other, with fast links, so synchronous replication is practical. Multi-AZ protects against a data-center failure and is the default for production. Multi-region protects against a whole region failing, but distance makes sync replication slow, so you usually replicate asynchronously and accept some RPO. Active-passive: one region serves, the other is a standby (pilot light or warm) that takes over on failover; simpler, but failover must be tested and RTO is minutes. Active-active: both regions serve live traffic, giving near-zero RTO and lower latency for global users, but writes in two places need conflict handling or partitioning of data by home region. Follow-up: "How is traffic steered?" DNS with health checks or a global anycast load balancer.

13. Authentication vs authorization; sessions vs tokens? easy

Show answer

Authentication proves who you are (password, passkey, OTP); authorization decides what you may do (may user 7 read order 42?). Broken access control, an authorization failure, is A01 in the OWASP Top 10:2025. Sessions keep state on the server: the browser holds a random ID in an HttpOnly, Secure, SameSite cookie, and the server looks it up in a store like Redis. They are easy to revoke but need a shared store. Self-contained tokens like JWTs carry signed claims, so any service can verify them without a lookup; that scales well across microservices but makes revocation hard and claims can go stale. Many systems combine them: short-lived JWT access tokens plus a revocable refresh token or server session. Follow-up: "Where do you check authorization?" On the server, for every request and every object.

14. Describe JWT structure and its main pitfalls. medium

Show answer

A JWT is three base64url parts joined by dots: header (alg, kid), payload (claims like sub, exp, aud, iss), and signature over the first two. It is signed, not encrypted, so anyone can read the payload. Pitfalls: (1) revocation: a stolen token is valid until expiry, so keep access tokens to minutes, use revocable refresh tokens, or a deny-list by jti. (2) Algorithm attacks: never trust the header's alg; pin allowed algorithms, reject none, and avoid RS256-to-HS256 confusion, as RFC 8725 requires. (3) All registered claims are optional in RFC 7519, so you must check exp, iss and aud yourself. (4) Storing them in localStorage exposes them to XSS. Follow-up: "How do you rotate signing keys?" Publish keys in a JWKS endpoint and select by kid.

15. Walk through the OAuth 2.0 authorization code flow with PKCE. hard

Show answer

The client creates a random code_verifier (43–128 characters) and code_challenge = BASE64URL(SHA256(verifier)). It redirects the browser to the authorization endpoint with response_type=code, client_id, redirect_uri, scope, a random state, and the challenge with method S256. The user authenticates and consents. The server redirects back with a short-lived code and the state; the client checks the state. The client then calls the token endpoint directly with the code, redirect URI and verifier. The server hashes the verifier, compares it with the stored challenge, checks the redirect URI, and returns access and refresh tokens. PKCE stops a stolen code from being redeemed. RFC 9700 makes PKCE mandatory for public clients and recommended for all. Follow-up: "Why not the implicit flow?" Tokens in the URL leak; RFC 9700 says clients should not use it.

16. OAuth vs OpenID Connect: what is the difference? medium

Show answer

OAuth 2.0 is about authorization: it gives an app an access token to call an API with limited scope. It says nothing standard about who the user is; the access token is meant for the API, not for the client to read. OpenID Connect is an identity layer on top: the client asks for the openid scope and also receives an ID token, a signed JWT with required claims iss, sub, aud, exp and iat, plus a UserInfo endpoint and discovery metadata. So "Log in with Google" is OIDC; "let this app read my calendar" is OAuth. A common mistake is using an access token as proof of login; it may have been issued to a different client. Follow-up: "What does the client check in the ID token?" Signature, issuer, audience equals its own client ID, expiry and nonce.

17. API keys vs OAuth tokens vs mTLS: when do you use each? medium

Show answer

API keys identify a calling project or partner. They are simple and good for quotas and analytics, but they are long-lived shared secrets with no user identity, so they must be scoped, rotated and kept server-side only. OAuth tokens represent a user's delegated consent with scopes and expiry; use them whenever an app acts on behalf of a user, or use the client-credentials grant for machine identity with short-lived tokens. mTLS proves machine identity at the transport layer: both sides present certificates, so a stolen bearer token is not enough; it suits internal service-to-service traffic (often via a service mesh that rotates certificates automatically) and high-security partner APIs. RFC 8705 can bind OAuth tokens to the client certificate. Follow-up: "Where would you check API keys?" At the API gateway, before traffic hits services.

18. How should passwords be stored? easy

Show answer

Never store them, not even encrypted; store a slow, salted, adaptive hash. OWASP's first choice is Argon2id with at least 19 MiB memory, 2 iterations, parallelism 1. Then scrypt (N=2^17, r=8, p=1), bcrypt with work factor at least 10 (note it uses only the first 72 bytes), or PBKDF2-HMAC-SHA256 with 600,000 iterations where FIPS compliance matters. Each password gets a unique random salt, stored with the hash, so identical passwords hash differently and precomputed tables fail. Optionally add a pepper kept in a secrets manager, outside the database. Fast hashes like SHA-256 are wrong because GPUs try billions per second. Raise the cost over time and rehash at next login. Follow-up: "What else?" Rate-limit logins, check against breached-password lists, and offer MFA.

19. Encryption in transit vs at rest, and how do you manage the keys? medium

Show answer

In transit protects data moving over networks: TLS 1.3 (now RFC 9846) on every hop, including internal traffic, with HSTS for browsers and mTLS between services where identity matters. At rest protects stored data on disks, databases, snapshots and object storage. Key management is the hard part. Use envelope encryption: data is encrypted with a data key; the data key is encrypted with a master key held in a KMS or HSM that never exports it. This allows auditing every use, rotating master keys, and "crypto-shredding" by deleting a key. Access to decrypt is controlled by IAM with least privilege, separately from access to the data. Follow-up: "Does at-rest encryption stop SQL injection?" No; the database decrypts for any query the app is allowed to run.

20. Design security for a public REST API used by web, mobile and partners. hard

Show answer

Edge: CDN and DDoS protection (Shield Standard covers layer 3/4), a WAF with rate rules, TLS only. Identity: web and mobile use OIDC with authorization code + PKCE; partners use client-credentials or mTLS-bound tokens; API keys only for quotas. Gateway validates JWT signature, iss, aud, exp with pinned algorithms and per-client rate limits. Services enforce authorization per object (A01 Broken Access Control), use parameterized queries, and validate input. Short access tokens, rotating refresh tokens, revocation endpoint. Secrets in a vault, least-privilege IAM roles, encryption at rest with KMS. CORS allows only our web origins. Audit logs and security alerts feed the SIEM (A09). Dependencies are pinned and scanned (A03 supply chain). Follow-up: "How would you detect credential stuffing?" Login failure rates per IP and per account, plus breached-password checks.

—Cheat sheet

Numbers and facts to remember memorise

Fact	Value	Source
Four golden signals	Latency, traffic, errors, saturation	SRE book ch. 6
RED / USE	Rate, errors, duration / utilization, saturation, errors	Grafana / Brendan Gregg
99.9% over 30 days	43.2 minutes of budget	Arithmetic from SRE definition
Burn-rate alerts	14.4 (1 h) page · 6 (6 h) page · 1 (3 d) ticket	SRE Workbook ch. 5
Short alert window	1/12 of the long window	SRE Workbook ch. 5
`traceparent`	version-32 hex trace-16 hex span-flags	W3C Trace Context
AZ distance	all within 100 km in a Region	AWS global infrastructure
Backup rule	3 copies, 2 media, 1 off-site	CISA paper
PKCE verifier	43–128 characters, method S256	RFC 7636
OAuth security BCP	RFC 9700 (Jan 2025): no implicit, PKCE everywhere	RFC 9700
ID token required claims	`iss`, `sub`, `aud`, `exp`, `iat`	OIDC Core
Argon2id minimum	19 MiB, 2 iterations, 1 thread	OWASP
bcrypt	cost ≥ 10, 72-byte input limit	OWASP
PBKDF2-HMAC-SHA256	600,000 iterations	OWASP
TLS 1.3 spec	RFC 9846 (obsoletes RFC 8446)	RFC Editor
OWASP Top 10 current	2025 edition; A01 Broken Access Control	owasp.org

Reference

—Sources

All checked on 2026-10-25 official docs

What	Where
Golden signals	Google SRE book ch. 6
Error budget	Google SRE book: Embracing risk
Burn-rate alerts	Google SRE Workbook: Alerting on SLOs
RED method	grafana.com
USE method	brendangregg.com
Metric types	prometheus.io
Traces, spans, context	OTel traces · OTel context propagation · OTel SDK env vars
Trace Context header	W3C Trace Context
Load shedding	AWS Builders' Library
Chaos engineering	principlesofchaos.org · Netflix Chaos Monkey
DR, RPO, RTO	AWS DR whitepaper (BCP) · DR options
Availability Zones	AWS Regions and AZs
3-2-1 backups	CISA: Data Backup Options
Sessions	OWASP Session Management
JWT	RFC 7519 · RFC 8725 · RFC 7009 · RFC 7662
OAuth 2.0	RFC 6749 · RFC 6750 · RFC 7636 · RFC 9700 · OAuth 2.1 draft
OpenID Connect	OIDC Core 1.0
TLS and mTLS	RFC 9846 · RFC 8446 · RFC 8705
Envelope encryption	AWS KMS cryptography
Password storage	OWASP Password Storage · NIST SP 800-63B-4
Secrets and least privilege	OWASP Secrets Management · AWS IAM best practices
OWASP Top 10	OWASP Top 10:2025
DDoS	AWS Shield Standard · AWS DDoS resiliency whitepaper
CORS	MDN: CORS

Course

—Where next

Keep going through the course navigation

Home

Course home

Previous · 13

AWS: databases, caches, messaging, serverless

Next · 15

The interview framework