How do many small services act like one reliable app?

Start with a (modular) monolith for a new product or a small team: one deploy, ACID transactions, simple debugging, fast in-process calls. Move to microservices when the organisation is the bottleneck: many teams blocked on one release, parts with very different scaling needs, or a need to isolate failures. The cost is real: network failures, eventual consistency, distributed tracing, more ops. Fowler's MonolithFirst says not to start new projects with microservices. Prerequisites: automated deploys, monitoring, fast provisioning. Follow-up: "how do you pick boundaries?" → bounded contexts.

One deployable app split into strict modules by business area (orders, users, payments), each with its own internal code and ideally its own tables, talking only through module interfaces. You keep a single deploy and real database transactions, but you get clear ownership and low coupling. If one module later needs independent scaling or a separate team, it can be extracted with little rework because its boundary already exists. The risk is that walls erode without enforcement, so use build rules or architecture tests that block cross-module imports.

So services can change and deploy independently. If two services share tables, a schema change in one can break the other, and they must be released together, which is a distributed monolith. Owning data also lets each service pick the right store (SQL for orders, Redis for sessions, a search index for products). The cost: no joins or ACID transactions across services. You replace joins with API calls, events or read models (CQRS), and transactions with sagas plus the outbox. Owning a database can mean a separate schema on a shared server; the rule is about access, not hardware.

REST over HTTP/JSON for public and browser-facing APIs: simple, cacheable, readable. gRPC (HTTP/2 + Protobuf) for internal calls: strict typed contracts, smaller payloads, streaming, generated clients; harder from browsers. Messaging (Kafka, SQS) when the caller does not need an immediate answer: it decouples availability and absorbs spikes, at the cost of eventual consistency and duplicates. Rule of thumb: sync for queries a user waits on, async for side effects ("send the receipt", "update analytics"). Each sync hop multiplies availability: three 99.9% services in a row ≈ 99.7%.

Client-side: the client queries a registry (Consul, Eureka) for healthy instances and load-balances itself. No extra hop and smart choices (zone-aware), but discovery code is needed in every language. Server-side: the client calls a stable name or load balancer (Kubernetes Service, AWS ALB) that knows the instances. Clients stay simple, at the cost of one more hop and a component that must be highly available. In Kubernetes you get server-side discovery for free: a Service with a stable ClusterIP and a DNS name like orders.prod.svc.cluster.local. Watch DNS TTL caching of dead addresses.

The gateway is the single entry point: routing by path, authentication, TLS termination, rate limiting, request aggregation, caching, and hiding internal service layout from clients. A BFF (backend for frontend) is a gateway per client type, owned by that client's team, so mobile can get small payloads and web can get rich pages without one API serving both badly. Risks: the gateway is a single point of failure (run it redundantly) and can become a monolith if business logic creeps in. Keep it to cross-cutting work.

Base it on the callee's measured latency: a bit above its p99 (or p99.9), so normal slow requests pass but hung ones are cut. Then check it fits inside your own deadline: if your API promises 500 ms and you make two sequential calls, they cannot each have 400 ms. Propagate a deadline down the chain (gRPC deadlines do this) so inner services stop work nobody waits for. Set connect and read timeouts separately. Too short a timeout causes false failures and retries; too long ties up threads. Always pair with a circuit breaker.

Backoff doubles the wait after each failure (100, 200, 400 ms…) up to a cap, giving the callee time to recover. Without jitter, every client that failed at the same moment retries at the same moment, creating synchronised waves that keep the service down. Full jitter picks a random wait between 0 and the backoff value: sleep = random(0, min(cap, base * 2 ** attempt)), which AWS's analysis found spreads load best. Also cap attempts, retry at one layer only (AWS shows 3 tries at 5 layers can mean 243x load), and only retry idempotent operations.

Closed: calls pass and failures are counted in a sliding window. When the failure rate passes a threshold (Resilience4j default 50% over the last 100 calls, after at least 100 calls), it goes Open: calls fail immediately without network, protecting your threads and giving the callee rest. After a wait (default 60 s) it goes Half-open and lets a few test calls through (default 10). If they succeed it closes; if they fail it opens again. Pair with a fallback (cached or default response). Slow calls can count as failures too.

Isolate resources per dependency or per client so one failure cannot exhaust everything. Example: a service has 200 request threads and calls payments and recommendations. If recommendations hangs, all 200 threads can end up waiting on it and checkout dies. With bulkheads, recommendations gets its own pool of 20 threads (or a semaphore of 20 concurrent calls); when full, extra calls fail fast to a fallback, and checkout still has its own pool. The same idea applies at larger scale: separate clusters per tenant tier, or cell-based architecture. Cost: idle capacity in each pool.

2PC needs every participant to support the protocol and hold locks between prepare and commit. That couples availability: one slow or dead participant, or a coordinator failure after prepare, blocks the others with locks held. It also does not scale to high throughput, and many stores (Kafka, most NoSQL, third-party APIs like Stripe) cannot take part. microservices.io simply says 2PC is not an option for microservices. Use a saga: local transactions plus compensating actions, accepting that in-between states are visible for a short time.

Choreography: each service listens for events and emits its own; no central coordinator. Loose coupling, good for 2 to 4 simple steps, but the overall flow is spread across services and hard to see or change, and cyclic event chains can appear. Orchestration: one orchestrator (a service or a workflow engine like Temporal or AWS Step Functions) sends commands and tracks saga state. The flow is explicit, easy to monitor, timeouts and compensations live in one place; the risk is the orchestrator accumulating business logic. For a checkout with payment, stock, shipping and loyalty steps, I would pick orchestration.

A business-level undo of a step that already committed: refund a charge, release reserved stock, cancel a booking. It is not a database rollback, because the original commit is visible and may have side effects. Pitfalls: some steps cannot be undone (an email sent, a parcel shipped), so put them last (a "pivot" step after which the saga only moves forward). Compensations must be idempotent and retried until they succeed. Concurrency: another saga may read the in-between state, so use semantic locks such as an order status of "PENDING".

To avoid the dual-write problem, the service writes its business row and an outbox row in the same local transaction. A relay publishes outbox rows to the broker. A polling publisher queries unsent rows on a schedule: simple, works anywhere, but adds DB load and delay. CDC (Debezium reading MySQL binlog or Postgres WAL) streams committed changes with low delay and in commit order, but adds Kafka Connect to operate. Both may publish twice after a crash, so consumers must be idempotent. Clean up sent outbox rows on a schedule.

Command Query Responsibility Segregation: separate models for writes (enforce rules, normalised) and reads (denormalised views shaped for each screen), often kept in sync by events. Worth it when reads and writes differ a lot: read-heavy dashboards, search, or views that join data owned by several services. Costs: two models, sync code, and eventual consistency, so a user may not see their own write instantly (fix with read-your-writes tricks such as reading the write model for that user briefly). Fowler warns it adds risky complexity for most systems, so do not use it by default.

Instead of storing current state, store every change as an immutable event (Deposited 100, Withdrew 30); state is rebuilt by replaying them. Gains: full audit trail, time travel, new read models by replay, natural fit with events for other services. Problems: event schemas live forever, so you need versioning and upcasting; long streams need snapshots; queries need CQRS projections; deleting personal data (GDPR) is hard in an append-only log (crypto-shredding helps). Good fits: ledgers, bookings, audit-heavy domains. Poor fit: simple CRUD.

The client generates a UUID per user action and sends it in an Idempotency-Key header on every retry. The server inserts the key with status "in progress" under a unique constraint in the same transaction as the payment record. On a repeat: if done, return the stored response; if in progress, return 409 so the client waits; if the body differs, return an error. Keys are kept longer than the client's retry window (Stripe allows pruning after at least 24 hours). Calls to the card network also carry our own idempotency id, so a crash mid-charge can be reconciled.

Strangler fig, never a big-bang rewrite. Put a routing facade (gateway or proxy) in front. Pick a first slice that is valuable, has clear boundaries and few dependencies (e.g. notifications). First modularise it inside the monolith, then extract it; route its traffic to the new service gradually (canary), with a quick way back. Data: let the new service own its tables; sync with CDC from the old database during transition. Add tracing and contract tests before the first split. Repeat slice by slice and delete old code once traffic is zero. Measure deploy frequency and incidents to prove it helps.

Rolling replaces instances gradually (Kubernetes default 25% max surge, 25% max unavailable): cheap, no extra fleet, but old and new run together and rollback is slow. Blue-green runs a full second environment and flips traffic at once: instant rollback, but double capacity during the switch, and the database schema must work for both. Canary sends a small share of traffic to the new version and grows it while metrics stay healthy: smallest blast radius, needs good metrics and weighted routing. In all cases make schema changes backward compatible (expand, then contract). Feature flags separate deploy from release.

With sync calls and no limits, slow payment holds checkout threads, the pool fills, and checkout fails for everyone (cascading failure); retries make it worse. Fixes: timeouts slightly above payment's p99; one retry with jitter only for idempotent calls with an idempotency key; a circuit breaker so checkout fails fast when payment is down; a bulkhead so payment cannot use every thread. Architecturally, make it an orchestrated saga: accept the order as PENDING, charge asynchronously via a queue that levels the peak, reserve stock, compensate on failure, and notify the user. Use the outbox for events and watch queue lag.