Event bus for dummies
As a software developer at Heureka Group, I'm offering a brief introduction to software architecture, recipes, and insights from my experience. I highly recommend this article to all developers and product managers. It's a valuable resource for anyone looking to enhance their understanding of software architecture.
What is the primary purpose of software architecture?
I believe that creating a highly organized application structure is the primary objective of all software engineers. We want to develop sustainable, maintainable, and expandable solutions. As developers, we should always keep the future in mind.
Many applications during development become something like a plate of spaghetti. There are a lot of connections between vast amounts of services. This is not precisely an architecture – it is only a plate of spaghetti. No one knows what is connected where. No one has a complete picture of the application.
The only solution to untangle the spaghetti structure is to find an Italian meal with a better structure. We are looking for something structured and layered – what about lasagna?
The Story of SW Architecture
A monolithic solution is a prime example of a spaghetti structure. Such a monolith structure has long grown, regardless of how developers, technologies, and architectural philosophy have changed during its development.
Microservices – a way out of spaghetti
Microservices are the only way out from the monolithic spaghetti knot. So, we have separated the code and its data into simple, independent applications.
The threat: Rest Nest
Once you break a monolith into services, it works nicely. It will separate the functionality into logical parts, standardize communication between services (REST), and everything is fine.
But the new idea brings new problems. The usual one is when one service calls another service, and it calls other services, etc. This case creates undefined response times and complicates monitoring of what is happening in the system at any given moment. Also, past states of the system are lost forever.
This is something like the higher level of spaghetti architecture (I personally call it Rest-Nest).
When an architect is in a tight corner, he always uses one solution (except suicide, of course). He adds another layer. So, let's use our microservices structure and glue them using another layer, and we finally get a nice portion of lasagna.
Request-driven vs. event-driven
The microservices with REST interfaces are typical Request-driven architecture. In this type of architecture, users or services send commands to the system. The command tells the system what to do.
On the other hand, Event architecture describes what happens in the system. The message describes the change as an event. The state of the system is described by a chain of events. Let's call it the event bus.
The event bus is a system built around the bus. There are applications that send data to the bus, called producers, and applications that read data from the bus, called consumers.
Event bus design patterns
Design patterns are bricks we use for building complex systems.
Producer= The producer sends event messages to the bus.
Consumer = The consumer reads event messages from the bus.
Dead letter queue
When an incoming message can't be processed for any reason, the system cannot stop processing queues or simply forget about it. So, we always create a special event to store these "invalid" messages.
Typical case: Message is not JSON valid
When an application consumes a message and then produces another message, we call it a processor.
Typical case: We have a user ID and want to add more data about the user to the message.
The application collects incoming message data to the database. We call this app materialiser.
Typical case: Visit counter: If our event bus contains one message for one user visit on the web, we can count it using materialiser.
When there is some third-party service that is not reliable (in extreme, this case is all services), we need to store the message for the following process round.
Typical case: The invoice system generates invoices, which is time-consuming. So, the processor needs to implement a retry queue to wait for the generated invoice.
Pros of event bus architecture
- Small apps
The system is divided into applications as small as possible.
Adding (or removing) this new app to this structure is easy – it is extensible. Great for the future, remember?
We can have (nearly) an unlimited amount of consumer app instances.
Apps do not depend on each other. Apps depend only on incoming events – easy-to-test
Cons of event bus architecture
Even a tiny solution needs significant infrastructure.
The event bus requires precise specification that describes Event flow.
The structure and meaning of events are not a part of the event bus.
When the bus is down, everything is down.
There could be a long way from the start to the end of event processing.
It is always asynchronous. It could be mostly an advantage and sometimes a pain – it is hard to mix with synchronous systems.
Our project contains 15+ services and 30+ event messages. It is not such a simple task to, e. g., change the event name throughout the whole project. So these are conventions we decided to follow.
One topic per event
Every topic contains only one type of event.
Why: Events have different importance, count, and configuration (e. g., retention in Apache Kafka)
One publisher per topic
There could be only one event producer.
Why: solves questions: Who sent this?
Consumer as simple as possible
The consumer has to be as simple as possible. Reads data, processes data, sends or resends data, nothing else.
Processor as fast as possible
Processors could be the bottlenecks. It slows the flow of events.
Why: The processor is in the middle of two events. So, its processing time is a delay in event flow. Don't forget to measure processing time.
Event names are made from two parts – the subject and the action, written in dot notation.
The subject is typically a singular noun – the name of processed content (such as credit, payment, invoice, notification) or service name (for dead-letter-queue).
The action is a verb in past tense and describes what happened with the subject (such as created, requested, expired, confirmed, canceled).
Every service specification or subsystem contains an Event flow chart made in Mermaid, allowing us to join these small charts into the big one.
The complete bus event flow chart also works as the interface between developers and project managers. It easily allows you to check different use cases.
Now that you understand the benefits of using an event bus and you had a look at how to implement it, do not hesitate. And please hop on the bus.
- The Concept of Event-Driven Microservice Architecture by Sumith Fernando
- Event-Driven Architecture: Getting Started with Kafka by Jean-François SIMON