How do services talk without waiting for each other?

Three reasons. Load levelling: the queue absorbs spikes, so the worker fleet is sized for average load, not peak. Decoupling: the sender does not need the receiver to be up, or even to know who it is. Fault isolation: if the receiver crashes, messages wait instead of requests failing. Cost: no immediate answer, eventual consistency, duplicates to handle, one more system to run. Example: an upload API that enqueues "resize image" jobs returns in milliseconds even if resizing takes 5 seconds. Follow-up: "what if the queue keeps growing?" → consumers are slower than producers on average; scale consumers or shed load.

In a queue (point-to-point), each message is processed by one consumer; adding consumers splits the work (competing consumers). In pub/sub, each subscriber gets its own copy; adding subscribers adds readers, not workers. Use a queue for tasks ("send this email"), pub/sub for events ("order placed" for billing, email and analytics). Kafka does both: inside a consumer group it is a queue (one consumer per partition), across groups it is pub/sub. On AWS, SNS → several SQS queues gives the same "pub/sub of queues" shape.

A partition is an ordered, append-only log; a topic is split into several. Partitions exist for scale (they spread across brokers, so one topic can exceed one machine's disk and throughput) and parallelism (one consumer per partition per group). The cost is ordering: Kafka orders messages within a partition only. Each message has an offset, its position number. Choosing the count: estimate peak throughput ÷ per-partition throughput, and at least your peak consumer count, because you cannot have more active consumers in a group than partitions.

If the message has a key, the default partitioner hashes it (murmur2) modulo the partition count, so the same key always lands in the same partition, which keeps per-key order. With no key, the producer uses a "sticky" strategy: fill a batch for one partition, then move on, to get big efficient batches. A custom partitioner is possible. Trap: adding partitions changes the modulo, so existing keys can move and per-key order breaks across that moment. Hot keys (one celebrity user) overload a single partition; fix with key salting or a different key.

6 consumers get one partition each and 4 sit idle. Within a group, each partition is read by exactly one consumer, so the partition count caps parallelism. Idle consumers are not useless: they take over if an active one dies (after a rebalance). To use all 10, increase partitions to at least 10 (mind key remapping), or use a share group (KIP-932), where many consumers can read the same partition with per-record acks, at the cost of ordering.

They describe what can happen on a crash. At-most-once: commit the offset before processing; a crash loses the message. At-least-once: process, then commit; a crash causes a repeat. Exactly-once: the effect happens once. Kafka offers it inside Kafka via the idempotent producer + transactions (read-process-write with offsets in the same transaction). End to end with outside systems you get "effectively once" = at-least-once delivery + an idempotent consumer (dedup table keyed by message id, or upserts). Most brokers default to at-least-once.

Options, best first. (1) Store the message id in a "processed" table with a unique key, in the same DB transaction as the work; a repeat fails the insert and is skipped. (2) Make the operation naturally idempotent: "set status = shipped" instead of "increment count". (3) Conditional writes with a version or sequence number: apply only if newer. (4) For external calls, pass an idempotency key (e.g. to a payment API). The id must be stable across retries, so it comes from the producer (event id), not from the broker's delivery. Keep dedup records at least as long as redelivery is possible.

acks=0: the producer does not wait; fastest, any failure loses data. acks=1: the leader writes it to its log and replies; if the leader dies before followers copy it, the write is lost. acks=all: the leader waits until every in-sync replica has it. On its own that is not enough, because the ISR can shrink to just the leader; pair it with min.insync.replicas=2 and replication factor 3, so a write needs 2 copies or is refused with NotEnoughReplicas. Since Kafka 3.0 the producer default is acks=all with idempotence on.

The ISR (in-sync replica set) is the leader plus the followers that are fully caught up; a follower that falls behind for longer than replica.lag.time.max.ms (30 s default) is removed. Only ISR members may become leader, so acked writes survive. If every ISR member dies, Kafka can wait for one to come back (stay unavailable but consistent) or, with unclean.leader.election.enable=true, promote an out-of-sync replica (available again, but acked writes are lost). The default is false: Kafka picks consistency. That is a CAP choice in one config.

The broker assigns the producer a producer id (PID). The producer numbers each batch per partition with a sequence number. The broker remembers the last sequence per PID and partition; a retried batch with a number it already has is acked but not written again, and a gap is an error. It needs acks=all and at most 5 in-flight requests per connection, which also keeps order on retry. Limit: it dedups only within one producer session; a restarted producer gets a new PID unless it uses a transactional.id.

With cleanup.policy=compact, Kafka keeps at least the latest record for each key and removes older ones in the background. Offsets never change. A record with a null value (tombstone) deletes the key; tombstones stay for delete.retention.ms (1 day default) so readers see the delete. Use it for "current state" topics: user profiles, config, CDC changelogs, Kafka's own __consumer_offsets. A new service can rebuild a full table by reading from offset 0. Compare with time retention (default 168 hours), which drops old data regardless of key.

A rebalance happens when a consumer joins, leaves, crashes (missed heartbeats beyond session.timeout.ms) or takes longer than max.poll.interval.ms (5 min default) between polls. Old eager rebalances paused the whole group. Fixes: keep handlers fast or lower max.poll.records; use cooperative rebalancing (only moved partitions pause); use static membership (group.instance.id) so a quick restart does not trigger one; in Kafka 4.x use the new protocol (group.protocol=consumer, KIP-848), where the broker drives incremental assignment. Commit offsets before partitions are revoked to avoid repeats.

Lag = log end offset minus the group's committed offset, per partition. It tells you how far behind readers are, in messages; convert to time by dividing by the arrival rate. Alert on lag that keeps growing, not on a fixed small number. Causes: slow handler, too few consumers, a hot partition, a stuck poison message, or rebalance storms. Actions: scale consumers up to the partition count, batch work, fix the slow dependency, move poison messages to a DLQ, or add partitions. Tools: kafka-consumer-groups.sh --describe, Burrow, or the cloud's metrics.

A message that always fails blocks its partition, because order is per partition (head-of-line blocking). Retry a few times with backoff, then publish it to a dead-letter topic with error details and the original topic/partition/offset in headers, commit the offset and move on. Alert on DLQ size; replay after a fix. For transient failures use retry topics with delays (e.g. 1 min, 10 min) before the DLQ. Trade-off: once a message leaves the main line, later messages with the same key may be processed before it, so per-key order breaks; if order matters, pause that key instead.

When a consumer receives a message, SQS hides it from others for the visibility timeout (default 30 s, max 12 h). The consumer must delete it before the timeout ends, or it becomes visible and another consumer gets it. If jobs take longer than the timeout, you get duplicate processing; fix by setting it above the worst job time or extending it with ChangeMessageVisibility as a heartbeat. After maxReceiveCount receives, a redrive policy moves the message to a DLQ. Use long polling (WaitTimeSeconds up to 20 s) to cut empty receives and cost.

Standard: at-least-once, best-effort order, nearly unlimited throughput. FIFO: strict order per message group id and exactly-once processing via a 5-minute deduplication window, but limited throughput: 300 API calls/s per action, 3,000 messages/s with batching (higher in high-throughput mode). Pick FIFO when order per entity matters (bank transactions per account), and use the account id as message group id so different accounts still run in parallel. Otherwise prefer standard plus idempotent consumers, which scales further and is cheaper.

Use the "task vs event" rule. Tasks done once, with per-message retries and DLQ → SQS (zero operations on AWS) or RabbitMQ (rich routing with exchanges, push with prefetch, priorities). Events many teams consume, need for replay, very high throughput, per-key order → Kafka (kept log, partitions, consumer groups, compaction). Costs: Kafka has more to operate and partition planning; RabbitMQ deletes acked messages so no replay; SQS has no replay and FIFO throughput limits. A common combo: Kafka as the event backbone, SQS or RabbitMQ for job queues fed from it.

Use the transactional outbox. In the same local transaction that writes the business row, insert a row into an outbox table. A relay then publishes outbox rows to the broker, either by polling the table or by CDC (Debezium tailing the MySQL binlog or Postgres WAL). If the transaction rolls back, no event exists; if the relay crashes, it resumes. The relay can publish twice, so consumers must be idempotent. Avoid "write DB, then publish" (crash in between loses the event) and "publish, then write DB" (event about something that never happened).

The order service writes the order + an outbox row; a relay publishes OrderPlaced to a Kafka topic keyed by order id (or customer id for per-customer order). Separate consumer groups: email, SMS, push, analytics, each scaling up to the partition count (size for peak, e.g. 24 partitions). Each sender is idempotent via a dedup table on (event id, channel). Provider failures: retry with backoff, then a dead-letter topic plus alert; rate-limit per provider. Analytics can lag; email should not, so alert on email lag. Replay lets a new channel (WhatsApp) start from history.

All messages for that key go to one partition, so one broker and one consumer carry it: lag grows on that partition while others sit quiet, and adding consumers does not help. Options: (1) salt the key (user42#0..7) to spread it across 8 partitions, then merge downstream, accepting loss of strict per-key order; (2) pre-aggregate in the producer (send counts every second, not every click); (3) give the hot entity its own topic; (4) if order is not needed, use a share group or no key. First, prove it with per-partition lag and bytes-in metrics.