Design Patterns - Transactional Outbox pattern
In the microservice architecture, many times we need that they communicate with each other by sending messages using a message broker, but when these messages are sent or published directly by the microservice to a message broker, many potential points of failure are opened up, for example let’s see these possible scenarios:
Possible problematic scenarios without using the transactional outbox pattern
Scenario 1: Sender microservice have a successful transaction but it fails send the message
In this scenario the receiver microservice never recive the message, so the business process will be truncated and invalid.
Scenario 2: Sender microservice have a unsuccessfully transaction but it successfully send the message
In this scenario the receiver microservice will receive the message and will process it, but the transaction of the sender microservice failed, so the business process will be invalid.
Scenario 3: The message broker is not available
In this scenario the receiver microservice never recive the message, so the business process will be truncated and invalid.
Transactional Outbox pattern
The transactional outbox pattern covers these possible points of failure, which is why it is recommended to use this architectural pattern for sending and publishing events.
To explain this pattern, I’ll use an example app that implements it.
For this sample I will create three projects in C# using RabbitMQ, SQL Server and Quarz, these three projects represent the components of the transactional outbox pattern: An sender, and outbox processor and the receiver.
Repository: https://github.com/JMOrbegoso/OutboxInboxPatternSample
The workflow of this sample project is very simple: We have an API to register users, when a user registers we must send a welcome email to that user, for which we need to call a mail delivery service, which is external to this API, so we must ensure a successful communication between these two components.
To achieve this we implement the transactional outbox pattern, but before explaining this implementation we must define what an outbox is:
Outbox: Envelope that wraps a message, it have properties like Id, creation date, message type, message data and send date.
This example consists of three projects:
Sender
Users are registered in this API, when a new user registers, we must send a message to the mail delivery service, but instead of sending it directly to RabbitMQ, it will wrap this message inside an outbox object and save it to the outbox database table.
To ensure the complete consistency of application state, the outbox object and the user are saved in the database within the same transaction, so if the transaction fails, the outbox object will never be saved to the database, this can be easily achieved by making use of the Unit of Work pattern.
Those are all the changes to be made in the sender, as you can see this microservice does not have any reference to the message broker or the mail delivery service.
Outbox Processor
This project could be an application console, a worker, or it could even be a background process within the Sender microservice, but in this approach it is a worker.
This project is responsible for checking from time to time the outbox table in the database looking for unpublished outbox objects, fetch them and post them to the RabbitMQ queue.
First we need to checking from time to time the outbox table in the database looking for unpublished outbox objects, to perform we will use Quarz, it is a task scheduler for C#, this Quartz job will check each 5 seconds in outbox table in the database looking for unpublished outbox objects, fetch them and post them to the RabbitMQ queue, after that it update the outbox object updating its send date property.
Receiver
This is the mail delivery service, is subscribed to a RabbitMQ queue, when receiving a message, it handles it.
Previous scenarios but now using the transactional outbox pattern
Let’s go back to the initial scenarios:
Scenario 1: Sender microservice have a successful transaction but it fails send the message
This can no longer happen because the first microservice do not directly send the message.
Scenario 2: Sender microservice have a unsuccessfully transaction but it successfully send the message
In this scenario the outbox would never have been stored in the database because the business and the outbox are written in the same database transaction.
Scenario 3: The message broker is not available
The outbox processor could not publish the messages immediately, but these will be sent as soon as the message broker is available again.