Blueprint to Bytes: Modern App Architectures with Event-Driven Design, Event Sourcing & CQRS
Transforming Theoretical Designs into Practical Applications
"Blueprint to Bytes" is a series of articles showcasing the journey from architectural blueprints to implementation. I will explore some existing patterns and approaches and implement them based on my own understanding. Stay tuned!
👋 Introduction to Modern Architecture Patterns
Modern software development requires advanced architecture approaches to meet increasing requirements for scalability, flexibility, and maintainability. Traditional monolithic architectures face difficulties in terms of speed of development or ease of deployment.
Even though it might be fine to start with a monolith, you might refactor later and split. If you decide to cut an application into more services, you could do this by teams, bounded contexts, or other measures.
As microservices also introduce challenges, there is also a concept called the logical monolith. It aims to combine the benefits of monoliths with a runtime that can scale parts independently. Jakarta Enterprise Beans tried to do something similar and failed by abstracting the network part.
Architecture patterns provide proven solutions for certain problems, but they also come with costs, and using a certain pattern is a tradeoff. You might trade increased scalability for more development effort.
Architecture patterns play an important role in building modern applications. The decision process for architecture involves evaluating tradeoffs, understanding system requirements, and aligning with business goals. So you should gather enough information and be able to justify your decision. For documenting you should consider Architecture Decision Records.
💡 Understanding CQRS (Command Query Responsibility Segregation)
CQRS is an architecture pattern that separates read and write operations within an application, allowing fine-grained optimizations and scalability of operations. By separating the read and write model, you can face stale read data, so you have to deal with eventual consistency here.
Benefits:
Scalability:
Independent scaling of read and write operations enables optimized resource allocation based on your requirements. For example, you can scale up reads with more instances by scaling horizontally. Scaling writes is more difficult and depends on your database architecture. (Reference: Leaderless Replication and Multi Write Nodes)
Flexibility and Performance:
You can have a read and a write model and maintain them separately. An update to your write model triggers an update to your read model, for example, by events. Upon reading, you do not have to transform or query multiple tables as the data is already there, leading to better performance.
Separation of Concerns:
Separating reads and writes can lead to better maintainability and easy-to-understand code. Having commands and queries enables easier understanding of the system.
Drawbacks:
Complexity:
Management and synchronization of multiple data models add complexity. Eventual consistency requires planning and analysis of the impact of stale data. The Saga pattern could come into play here. Different communication mechanisms between queries and commands also add complexity.
Effort and Costs:
Your system might become more resource-intensive as you have more databases and synchronization processes. If your team does not know CQRS, they need to learn and will have a learning curve. Debugging and error troubleshooting might be more difficult and cost-intensive.
Use Cases to Avoid CQRS:
Simple Applications:
For applications with mainly CRUD operations, CQRS is overengineering.
Limited Resources:
If you have limited resources in manpower or infrastructure, the complexity of CQRS could delay development further, and other simpler architectures might fit better.
Strong Consistency Requirements:
If you need strong consistency over read and write operations, CQRS might be the wrong choice. If you need real-time updates reflected consistently, you should not implement CQRS.
📌 Event-Driven Architecture (EDA)
An EDA is centered around the production, consumption, and reaction to events. Events reflect changes in state, and the EDA is designed to respond to those changes.
Benefits:
Fast Reaction:
EDA enables fast reactions to changes and requirements, which is desirable in fast-growing markets. The flexible structure allows for easy addition of events and processes.
Scalability:
Scaling independent parts of the system enables easy scaling without impacting other parts of the system. The loose coupling of components helps here. Microservices can be designed with event-driven principles, achieving loose coupling, robustness, and efficiency.
Ease of Maintenance:
As we are independent and do not have to touch every part of the system, maintaining and adding new features becomes easy. For example, in an e-commerce application, adding a new payment method can be done by creating a new event and handler without modifying existing parts of the system.
Drawbacks:
Complexity:
Systems can be complex in understanding the operations within as well as the whole system. You have to ensure the correctness of events you send as well as those that you receive. Consumer-driven contract tests can help here if you consume from external sources like other teams to ensure the correct format. This increases overhead on your CI/CD but ensures robustness.
Overengineering for Simple Applications:
For simple applications with not much expected change, EDA could be overengineering. You can have more ways to fail than with monolithic systems, and debugging async event systems is more difficult.
Cost:
It costs more money than simple applications, so you should consider if the tradeoff is worth it.
Greenfield Deployment:
EDA is viable for greenfield application deployment. Many modern digital platforms are based on EDA. You can integrate with your existing application and migrate step by step like with a Strangler Application. Start with a simple project and build up from there.
📍Event Sourcing
Event sourcing describes a pattern where every change in our state is captured as a sequence of events.
Capturing State Changes:
We capture all changes to our internal state as a sequence of events. Each event is stored in our event store and allows easy recoverability at any given point.
Auditing and Debugging:
Easy auditing, compliance, and debugging. We can easily trace the state back in time. Encourages immutability of the event store.
The drawbacks overlap with those of EDA and CQRS. You will likely encounter higher complexity, higher infrastructure costs, and increased engineering costs.
✅ Benefits of Combining CQRS, Event Sourcing and EDA
Event sourcing is often used with CQRS in event-driven architectures to improve scalability and flexibility of applications. Describe with self created picture below how it works.
Performance and Maintainability:
The combination of CQRS and EDA can improve performance and maintainability of applications.
Clear Separation:
The combination allows for a clear separation and enables updates to the read model after writes happen due to domain events by listening to those events.
Reliable Event Propagation:
The event bus propagates events so that they are reliably delivered to interested subscribers. Consumers can be in the same application or in other services.
Careful Planning Required:
Requires careful planning to manage complexity and ensure consistency and error handling.
Robust Synchronization:
EDA provides a robust mechanism to handle the synchronization between the command and query sides of CQRS. State changes in the domain are propagated and should reduce the time of stale read data.
Auditability:
Saves to our event store drive the update of our read models. We have easy auditability as all events are stored, ideally in an immutable fashion.
Efficient State Management:
For read models, we do not have to build from all the events as this would incur performance costs. For write updates, we can build up from the events, keeping a consistent state. If the store grows too big, we can use state snapshots to avoid loading all the history. Use cases that could benefit include inventory management with inventory audits, where we would have a complete log of all stock changes, allowing auditors to replay events to verify the current state.
✏️ Case Study: Bookstore
To show these concepts in a hands-on style, I will show an example application implemented with CQRS and event sourcing in an EDA style. The example will demonstrate a bookstore application. Kurz drauf eingehen was wir rauslassen, mails etc. bzw. message communism noch mit rein
Preview of Case Study:
The case study will show an example application with CQRS, Event Sourcing, and EDA.
The application demonstrates how to implement theoretical concepts and shows benefits and challenges. The goal is to give the reader a good understanding and practical insights into these architecture patterns.
Use Cases:
getBook: Display information of a specific book.
addBookToCatalog: Adds a new book to the catalog of books.
viewOrder: Shows the current state of your order.
placeOrder: Places a new order for books.
cancelOrder: Cancels an existing order for books.
Essential Domain Events:
BookAddedToCatalogEvent: Book has been added to the catalog.
StockDecreasedEvent: Stock of a certain book decreased.
StockIncreasedEvent: Stock of a certain book increased.
OrderCancelledEvent: An order was canceled.
OrderPlacedEvent: An order was placed.
Architecture of Sample Application:
Monolithic Service: We will start by implementing a monolithic application with a single database that serves for the event store and read models.
Separation of Concerns: The architecture is created in a way that enables a later separation into multiple services and datastores to increase scalability.
CQRS: We will separate reads and writes.
Event Sourcing: We will capture all changes for traceability and recovery of our state by such events.
EDA: We will have some kind of message broker for asynchronous communication between our components.
Ports & Adapters Architecture: This architecture is a proven pattern designed for loose coupling. Ports define an abstract API that an adapter will implement. Drivers, such as the User through a UI, and adapters funnel into the domain and may trigger a database update through a driven adapter. For example, you could replace the current database with a new one. You would need to implement a new adapter, but no changes would be required in the domain.
Technologies: Using Spring Boot, Couchbase, GraphQL, and JMS with ActiveMQ.
🔜 Upcoming Stories in this Series:
The next story in this series will focus on messaging with JMS and ActiveMQ for asynchronous communication and to increase the robustness of our application.
Afterwards, we will focus on transactions and the outbox pattern, which should ensure we publish correctly and do not produce inconsistent data.
The fourth article of this series will describe Couchbase and database migrations and how they are applied to our use case.
In the fifth article, we will show how we are using a GraphQL API that is exposed to clients and from which we can fetch books and place orders.
The last and final article will offer a recap of the whole series and show how each component works with each other.
🏁 Conclusion
The combination of CQRS, Event Sourcing, and EDA is a proven pattern for complex use cases with requirements like traceability, recovery at certain points in time, or auditing. For other use cases, it might be overengineering and should not be used.
The pattern is really good for systems that need to be scalable and flexible.
A thorough understanding of tradeoffs and careful planning is essential to effectively implement these patterns.