Blueprint to Bytes: Ensure Reliable Transactions with the Transactional Outbox Pattern
Ensuring Consistency in Distributed Systems with the Transactional Outbox Pattern
🙌 Catch Up
Previously in this series, we explored how to implement asynchronous messaging with ActiveMQ and JMS.
👋 Introduction to Transactions in Distributed Systems
In our previous articles, we explored the basics of CQRS and Event Sourcing, as well as the role of messaging in distributed systems. We have seen how messaging enables loose coupling between components. This article will focus on transactions within our system to ensure reliable publishing and consumption of messages in distributed systems.
Role of Transactions in Distributed Systems
Depending on the use case of our business domain, we might need transactional semantics to ensure database consistency. Consider our bookstore example where two customers want to order the last book. At some point, we need to notify one of the customers that they cannot buy the book as the stock is empty. This can be done directly by using database transactions with an update on a single item that locks. For event stores and more distributed systems, this is more difficult. In a setup with multiple databases, we can use the SAGA Pattern to implement compensating transactions. This could be, in a simple world, a cancel message to the customer directly after the order. Another approach is the Two-Phase-Commit Protocol, which provides strong consistency but is vulnerable to network errors and may have slower throughput.
For our use case, we need to ensure that publishing events only happens if the triggering domain events are really saved into our database.
Definition of the Transactional Outbox Pattern
The transactional outbox pattern resolves the dual-write operations issue that occurs in distributed systems when a single operation involves both a database write operation and a message or event notification. A dual-write operation occurs when an application writes to two different systems; for example, when a microservice needs to persist data in the database and send a message to notify other systems. A failure in one of these operations might result in inconsistent data. Read more
The Transactional Outbox Pattern ensures that database transactions and the publishing of events are considered a single unit of work. This effectively guarantees that we only publish the message if the transaction is successfully committed and not rolled back. Consider sending an email during the ordering process before saving our state to the database. If we violate some database constraint, our save is rolled back, leaving no order in the database, but the email was sent. This is mitigated by the outbox pattern.
🎯 Problem to solve
For our bookstore, we tried to optimize processing by separating reads and writes. For instance, if we order a book, the stock of that book is decreased asynchronously. The underlying events that trigger such updates should still be consistent; otherwise, we would give the user a false impression and have to deal with inconsistent and non-debuggable states in our database. In the following illustration, we will look at the problem we are trying to solve:
In many implementations, saving to the database happens as the last step. If step 2, which is publishing the event, happens before step 1 and, due to some database constraint, step 1 fails, we will have an inconsistent state as we update our read model but our event store does not have this information. To solve this, we run step 1 and step 2 as a transaction that either fails or succeeds, ensuring that we cannot publish if saving fails.
To show this behavior chronologically, see the sequence diagram below where the event is sent, and our bookstore consumes the update:
The previous problem is effectively mitigated if we handle it via transactions, as shown below, where both actions are treated as a single unit of work:
⚙️ Implementation
The outbox pattern spans several components:
Database of our application: In our case, we consider Couchbase, which has transactional support. A relational database is also suitable, as well as solutions that offer Change-Data-Capture (CDC) like AWS DynamoDB.
Outbox Table for our simple entity with this structure:
Outbox processing: If we have a relational database, we can have a frequent scheduler that checks our table, reads, publishes, and deletes the entry. Another implementation can be done via CDC. We create an event upon saving an outbox entry that is consumed by our processor.
Message Broker: We use ActiveMQ here, but it can be any message broker capable of sending events.
Saving Outbox Events
The first step is to implement the mechanism that saves outbox entries during a transaction. We have a persistence adapter for our book and respective order domain events. The one for books looks like this:
While we save our domain events, we also publish uncommitted events via the outbox pattern to any interested consumer. For simplicity, we assume that topics exist for each type of domain event.
Outbox Processor
Now our processor comes into play. We have a simple scheduler that reads all outbox messages:
Within a transaction, we fetch the entities, send a message, and delete the entry. If you have a very high load, you should page the query. Also, if you want to scale horizontally, you have to use some kind of locking or infrastructure deduplication mechanism to prevent duplicate sending of events, even though consumers should handle them idempotent.
Handling Errors and other Considerations
For managing errors and ensuring data consistency, our outbox processor has to work reliably. If a message cannot be sent, it should be kept in the outbox table and retried.
Depending on our use case, we should also strive for low latency, so we have to check the rate at which we schedule the outbox process. The order of how we send messages can also be considered by ordering based on adding an event time, for example.
How we can consume such events ourselves, for example, for updates of our read model, was covered in-depth in my last article.
Here is an example of a listener:
🔜 Upcoming Stories in This Series
The next story in this series will focus on how to integrate Couchbase with an event store and a store for our read models with Spring Boot and how to enable database migrations.
Afterwards, we will finish the series by showing how to offer with our Bookstore GraphQL API.
🏁 Conclusion
The transactional outbox pattern offers a robust solution for ensuring consistency in distributed systems. We have explored an approach to implement it and additional considerations on how to send and improve the process.