Transactional Messaging in Microservices

Introduction

Problem Statement

Transactional messaging is a common problem in Microservices - how do you integrate message publishing with database transactions that update business entities or data. For e.g., we may require to publish domain events when a new entity is created or updated, and also ensure that both the database update as well as the message publishing happen atomically. This can leave the system in an inconsistent state if not done properly - for instance, the service may crash after committing the database update before publishing the message.

However, I must also mention that transactional messaging is a more generic problem that can manifest in any system that has more than one coordinating persistent transactional components, and therefore the patterns discussed in this article are not restricted only to microservices.

Why Not use Distributed Transactions To Solve This?

Traditional solution to maintaining data consistency across services, databases and/or message brokers is to use distributed transactions that span all the databases and message brokers. However, distributed transactions aren't a good choice for modern applications.

The de facto standard for distributed transaction management is X/Open XA (Extended Architecture) - X/Open XA. Open XA uses two-phase commit (2PC) that ensures all the participants in a distributed transaction either commit or rollback. An XA-compliant technology stack consists of XA-compliant databases and message brokers, database drivers and messaging API's, and an interprocess communication mechanism (IPC) that propagates the XA global transaction ID.

Although this may not sound too complicated, there are several problems with distributed transactions:

Although many SQL databases and message brokers are XA compliant, and Java applications can use JTA to perform distributed transactions, many modern databases such as Cassandra and MongoDB do not support it. Same is the case for modern message brokers like RabbitMQ and Apache Kafka. Therefore, we might not be able to use many modern technologies if we decide using distributed transactions. Distributed transactions are actually a form of synchronous IPC which can further reduce the availability of the system, which will shrink even more when new services are added. In most modern applications, availability trumps consistency.

From a developer's perspective distributed transactions follow the same programming model as local transactions. However, because of the reasons above we may not want to choose distributed transactions to solve such kind of problems.

Using Transactional Messaging

Transactional messaging helps sending messages atomically along with database updates. Also, it does not have any of the problems of distributed transactions.

How transactional messaging works

Transactional messaging is a two-fold solution:

1. Using a database table as a temporary message queue
2. Publishing messages from the database to the broker

Using a Database Table as a Temporary Message Queue

Transactional Outbox Pattern

• Transactional Outbox Pattern is a straightforward way of ensuring reliability in publishing messages to the broker - we need to insert messages to be published in a database table in the same transaction.
• When a service uses an RDBMS database:
  • A service that needs to send messages as part of a transaction has an Outbox table that is used as a temporary message queue.
  • Whenever the service creates, updates, or deletes a business entity, it inserts the message/domain event that it needs to publish in the Outbox table.
  • Atomicity is guaranteed piggybacking on RDBMS's ACID properties.
• When a service uses a NoSQL database
  • Due to limited transactional guarantees of NoSQL databases, we need to keep an additional attribute in a business entity that stores a list of messages that need to be published.
  • When a service updates an entity in the database, it also appends a message to that list in the same database operation to ensure atomicity.

Moving messages from database to the Message Broker

The component that reads the Outbox table and publishes the messages to the message broker is called MessageRelay.

There are two ways messages can be moved from the database to the message broker:

1. Polling Publisher Pattern
2. Transaction Log Tailing Pattern

Polling Publisher Pattern

• In case of RDBMS database:
  • MessageRelay does simple periodic polls on the Outbox table for unpublished messages.
    • Next, the MessageRelay publishes those messages to the message broker.
    • Finally, it deletes the published messages from the Outbox table.
• In case of a NoSQL database:
  • MessageRelay needs to query the business entities themselves to get the attribute storing the list of messages that need to be published.
    • How efficiently the list-of-messages-attribute can be fetched, depends on the querying capabilities of the NoSQL database.
• Downsides of Polling Publisher Pattern:
  • Need to decide the frequency of polling the database for to-be-published messages. Polling the database too frequently can be expensive. It can be ineffective if the frequency is too low.
  • Moreover, querying the business entities to fetch the list of messages to be published may or may not be possible to do efficiently in NoSQL depending on its querying capabilities.
• It may work well at low scale, but might not be ideal otherwise.
• The following figure represents how the Polling Publisher Pattern works with the Transactional Outbox Pattern in RDBMS:

Transaction Log Tailing Pattern

• This is a more sophisticated and performant approach, and it also does not have any of the downsides of the Polling Publisher Pattern.
• MessageRelay tails the database transaction log, a.k.a. the commit log, which contains an entry for every committed database update.
• MessageRelay, often called a Transaction Log Miner in this context, simply reads every database commit log entry corresponding to a message inserted in the database, converts it into a broker message, and finally publishes the message to the message broker.
• This approach works for both the RDMBS Outbox table as well as messages appended to the list-of-messages-attribute in a NoSQL database.
• Deleting the already published messages from the database (although can be done) is optional, since the Transaction Log Miner reads only the new inserts and appends.
• Few examples of Transaction Log Miners - Debezium, LinkedIn Databus, DynamoDB Streams, Eventuate Tram.
• Downsides:
  • Although a lot of work can be reduced by using frameworks such as the Eventuate Tram framework (provides both transaction log tailing and polling), it still requires some significant development effort.
• The following figure represents how the Transaction Log Tailing Pattern works with Transactional Outbox Pattern in RDBMS: