Message Delivery Guarantees
In general, message brokers offer one/all of the following delivery guarantees:
- At least once
- At most once
- Exactly once
Duplicates must be handled when using messaging for asynchronous IPC. Message brokers should ideally deliver a message to a client only once but guaranteeing exactly once delivery is usually too costly.
Most brokers offer at least once delivery guarantee. When a system works normally, an at least once broker will deliver the message only once. But a failure of a client, network, or the broker can result in a message being delivered to the client multiple times; for instance, a client receives a message and updates its database but fails just before acknowledging the message. The broker will deliver the unacknowledged message again - either to the same client when it comes up, or to an available replica of the client (refer Kafka Partition Rebalancing to understand how Kafka does it).
Maintaining Ordering in Message Redelivery
Take example of a messaging client that processes an OrderCreated event followed by an OrderCancelled event for the same Order object, and somehow, the OrderCreated event isn't acknowledged. Therefore, the message broker would attempt redelivering the OrderCreated event. However, the broker must redeliver both OrderCreated as well as OrderCancelled events and in the same order, otherwise it can potentially cause the consumer to undo the OrderCancelled event.
Handling Duplicate Messages and Ensuring Message Ordering
We saw in the example above that the OrderCreated and the OrderCancelled duplicate messages can leave your application in an inconsistent state. There are two ways which can be used to handle duplicates depending on different cases:
- When the message broker can preserve message ordering during redelivery, we can write idempotent message handlers when possible.
- When the message broker cannot preserve ordering during redelivery or the message handler cannot be an idempotent operation, the message handlers must be written to track messages and discard duplicates.
When broker preserves message ordering during redelivery - write idempotent message handlers whenever possible
If the application logic for processing messages is idempotent, then duplicate messages are harmless provided message ordering is preserved by the broker. For e.g. - Cancelling an already cancelled order can be an idempotent operation.
Creating a new Order can also be designed to be idempotent with a client-supplied ID. However unfortunately, application logic is often not idempotent. In such cases, message handlers need to be made idempotent by tracking and discarding duplicates.
Tracking messages and discarding duplicates if broker does not preserve message ordering, or if idempotent message handler is not possible
If the message broker that we might be using does not preserve ordering while redelivering messages, then idempotent message handlers can potentially leave the application in an inconsistent state. Moreover, as mentioned previously, application logic is not or cannot be idempotent most of the times. Consider example of a credit card authorization logic. It must run the authorization logic only once for a card payment for an Order - it is supposed to have a different outcome each time it gets invoked. Such an application logic needs to be made idempotent by detecting and discarding duplicate messages.
A simple solution for a message consumer is to track the messages it has consumed using the message ID and discard duplicate messages. The consumer can store the message ID for each message it consumes in a database table as part of the same transaction that creates and updates business entities. The message IDs can be stored in a dedicated database table, or in an application table. The latter is useful particularly for NoSQL databases that usually do not support updating two tables/collections as part of the same database transaction.
The following diagram roughly explains this solution. An INSERT attempt for a message that was already processed earlier will fail, consequently the consumer can discard the message.
Some libraries and frameworks for messaging, like the Eventuate Tram (Getting started with Eventuate Tram), also offer out-of-the-box duplicate message detection.