Until now, we have mostly discussed Request-Response models (RPC/REST). This model is tightly coupled: the sender must wait for the receiver.
Event-driven architectures use a Message Broker (e.g., Apache Kafka, RabbitMQ) to decouple the producer of data from the consumer.
Producer-Broker-Consumer Model
In modern brokers like Kafka, the primary data structure is a replicated log. This log is append-only and persistent.
- Partitions: The log is split into partitions for scalability.
- Offsets: Consumers keep track of their position in the log using an offset.
Throughput: RPC vs. Message Stream
What happens if a message is lost or a node crashes during processing?
At-most-once
The sender sends the message and doesn't care if it arrives. No retries.
- Outcome: Messages can be lost, but never duplicated.
At-least-once
The sender retries until an acknowledgement is received.
- Outcome: Messages are never lost, but they can be duplicated.
Exactly-once
The gold standard, ensuring a message is processed exactly once even after retries.
- Mechanism: Usually requires Idempotency (the ability to apply the same operation multiple times without changing the result).
To achieve exactly-once, attach a unique ID to each message. If the receiver sees the same ID again, it ignores the message but returns the original successful result.
| Guarantee | Implementation Cost | Typical Use Case |
|---|---|---|
| At-most-once | Lowest | Sensor data (discardable) |
| At-least-once | Medium | Notification systems |
| Exactly-once | Highest | Payment processing |
Distributed systems are about managing trade-offs between Latency, Consistency, and Availability. There is no "perfect" system; there is only the right system for your specific requirements.