Your Event-Driven Apps Are Only as Good as Your Event Broker

Blog: Solutions Review - Business Process Management

As part of Solutions Review’s Contributed Content Series—a collection of contributed articles written by our enterprise tech thought leader community—Shawn McAllister, the Chief Technology Officer and Chief Product Officer at Solace, explains why a company’s event-driven apps are only as good as their event brokers.

Event-driven architecture (EDA)—a software design pattern enabling real-time, event-driven communication among diverse applications via an event broker—is becoming pervasive among enterprises that want to power time-sensitive applications, business processes, and insights (advanced analytics, ML, AI) at scale. However, achieving your desired outcome with EDA often depends on your choice of event broker. Not all event brokers are created equal; some are better suited for particular tasks and use cases.

Implementations of Event-Driven Architecture (EDA) platforms are now on the fast track, with 82 percent of IT leaders saying their company plans to apply EDA to 2-3 new use cases within the next 24 months. A new IDC Infobrief charts this rise. EDA adoption goes hand in hand with digital maturity, with 47 percent of respondents describing their EDA journey as either maturing (“centralized”) or “advanced.”

The rise of EDA is industry-agnostic. Research shows that more and more businesses across all industries—retail, financial services, aviation, manufacturing, transportation and logistics, and more—are seeing the need to adopt this pattern of application connectivity to effectively enable modern use cases such as curbside pickup, preventative maintenance of machinery, and digital twinning.

At the heart of EDA transformation is the event broker—middleware software that routes events and other data between various applications, systems, and devices. As EDA has risen in appreciation and use, so has the market for event brokers grown to the point that today, there is a wide variety of brokers to choose from. Event broker capabilities can vary drastically, so it is essential for architects and application teams to consider their evaluation criteria carefully.

Event Brokers Can Make or Break Your Real-Time Application Integration Strategy

An event broker is a message-oriented middleware that enables the reliable transmission of events between different system components, acting as a mediator between publishers and subscribers. It is the cornerstone of event-driven architecture, and all event-driven applications use some form of an event broker to send and receive information.

As a first and critical step, IT leaders now have to choose not whether to embrace EDA but rather which event broker to choose to underpin it or which ones to use for which use cases. Often, an enterprise might need more than one type of broker because some are better suited to particular use cases.

Analysts are joining the debate, notably David Mooter of Forrester, who recently outlined the choice between a “log stream broker” and a “smart broker.” When combined with real-time analytics and event sourcing, a log stream broker can support high data throughput, tolerate some degree of complexity, and message replay. On the other hand, a smart broker can support complex message routing, granular controls on message filtering, global order guarantees, transactional commits, and many other capabilities.

Use Cases Signpost the Choice: Analytical vs. Operational

Mooter explains that the fundamentals of event brokers allow organizations to build their applications as a collection of composable services that are decoupled in nature. This provides the benefits of agility, scalability, and resilience, but depending on the use cases, choosing between a log stream broker or a smart broker can make all the difference.

Apache Kafka best epitomizes log stream brokers. Over the last few years, it has taken the world of data streaming by storm because it’s very good at aggregating massive amounts of “log” data and streaming it to analytics engines and big data repositories.

Not All Event Brokers Are Equal to the Job at Hand

Unfortunately, the popularity and prevalence of Kafka have led many developers to use it for use cases for which it is not ideally suited—namely operational, “run-the-business” scenarios, which often involve a mix of applications, systems, and devices that need to tap into specific event streams to run effectively.

Log-based brokers use rigid, flat topic structures to describe the data they transmit, which puts the onus on the applications to filter through all the streamed data. This can be like drinking from a firehose, as applications need to consume, filter, and discard events they don’t need, which leads to increased cost and complexity, not to mention security concerns.

As Businesses Get More Connected, The Broker Must Get Smarter

On the flip side, “smart” brokers do a lot of the thinking, filtering, and routing—especially as enterprises work to become more integrated, connected, and real-time across diverse Information Technology (IT) and Operational Technology (OT).

These brokers feature-rich and flexible topic hierarchies, enabling applications to easily publish and subscribe to the specific subsets of data they are interested in. From an event streaming perspective, smart brokers support a wide range of message exchange patterns beyond publish/subscribe, including request/reply, streaming, and replay, as well as different service qualities, such as best effort and guaranteed delivery.

This makes smart brokers ideal for operational use cases, acting as the “digital nervous system” of a distributed enterprise. Smart brokers go beyond simply analytics—imagine a global bank streaming over 150 billion events a day between low-latency trading platforms and market data centers in multiple locations across the globe. Prices could be published in New York and consumed in London quickly, allowing orders to be placed quickly based on the latest information.

Choosing the Right Smart Broker to Run Your Business

The choice becomes clear: if an organization is looking to address operational “run-the-business” applications and use cases across a distributed enterprise, they need a smart broker, providing them three critical attributes:

1. Fine-grained routing & filtering

When dealing with complex business operations, events must be intelligently routed and filtered to be made available to the right people at the right time, not just served up en masse. A smart event broker allows consumers to subscribe to only receive the subset of events they need in the original sequence. Take the operational example of a retailer using microservices to stream order-related events – but different functions are being dealt with in other locations. Those application users who only want to process new orders do not require information such as shipped orders or returns; they need information based on new orders.

It’s also the job of a smart broker to publish information in a way that allows fine-grained filtering of events based on topic taxonomy. Consider an example of an aviation authority required to manage incoming and outgoing flights from a particular airport. It sounds straightforward, but there will be many events related to every flight. A smart broker can handle the volume of events about all flights but allow subscribers to drill down by airline, arrival/departure, on time/delayed, and inbound and outbound gate, meaning they can receive detailed information based on the factors relevant to their job function or area of responsibility.

2. Event mesh-ability

Organizations requiring real-time event streaming for operational use cases must look at how their broker can support an “event mesh.” An event mesh is an architecture layer consisting of a network of interconnected event brokers that allows events from one application to be dynamically routed and received by any other application, no matter where these applications are deployed—no-cloud, private cloud, or public cloud. An event mesh makes all data that touches the mesh available on demand in a secure, reliable manner—exactly where and when needed.

It provides the ability to integrate legacy applications, data stores, modern microservices, SaaS, IoT, and mobile devices dynamically and all in real-time. If Information Technology and analytics are legacy integration and Operational Technology is the sensing of devices, then an event mesh is the glue that ties old technology with the new.

Consider a manufacturing organization using IoT edge devices for asset tracking, product quality monitoring, and predictive maintenance. With an event mesh, sensor events in manufacturing processes can be leveraged with real-time streaming analytics to improve quality and detect machine maintenance issues sooner. Operations are always on, and so are customer and employee expectations. An event mesh supporting real-time data delivery ultimately means no waiting. Spikes in demand are deftly handled so systems don’t crash.

3. Rich application integration

Catering to the complex business ecosystems they support, a smart broker easily connects various applications and devices without worrying about protocol translation from one to another. A distributed organization, for example, has many custom applications in many different languages that they need to share information.

Operational use cases will usually involve this complex mix of legacy and modern applications and taking on further data streams from IoT devices—all communicating using different languages and protocols. These streams may also need to be accessed by partners or third parties using standards-based protocols such as AMQP, MQTT, or webhooks—hence the need for a smart broker to support integration across this application landscape.

Integration of SaaS, legacy, packaged applications, and middleware is also required across an organization. This involves the use of specialized event connectors to event-enable these applications, so the event broker you choose needs to have many of these “off-the-shelf” connectors as well.

Heineken: The Case for Rich Application Integration

Heineken, the multinational brewing company, has business units spread across 190 countries and is the perfect use case of how EDA deployments have increased in sophistication. Heineken uses an event-driven integration approach to support event streaming across thousands of business-critical applications, including payments, logistics, inventory management, and more. Only a smart broker enables this consistently fast, reliable, and robust integration process across all critical business functions.

Making Business Operations Smarter with the Right Broker for the Job

It’s an event-driven world. Business operations will only become more data-driven, dynamic, and real-time, demanding event streaming support beyond analytics-based log brokering.

This is where the smart broker needs to step up to the plate and provide the architecture to keep its stakeholders connected, always on the same page, and able to make up-to-date data-driven decisions. We’re talking developers, users, consumers, employees, partners, and end-customers. It’s a community of stakeholders that is only going to keep growing.