Enterprise Integration Patterns Revisited for the Era of Big Data, Internet of Things, Cloud, and Microservices
Blog: The Tibco Blog
In 2015, I gave talks about Enterprise Integration Patterns at OOP 2015 in Munich, Germany and at JavaDay 2015 in Kiev, Ukraine. I reused a talk from 2013 and updated it with current trends to show how Enterprise Integration Patterns (EIP) are used everywhere today and what’s coming for the future.
It was great to see that Gregor Hohpe, founder and author of the EIP book and currently Head of IT at Allianz, was attending one of the sessions:
Introduction to Enterprise Integration Patterns
Data exchanges between companies increase frequently and the number of applications that must be integrated increases as well. The realization of these integration scenarios is a complex and time-consuming task because different applications and services do not use the same concepts, interfaces, data formats, or technologies.
Originated and published over 10 years ago by Gregor Hohpe and Bobby Woolf, Enterprise Integration Patterns became the worldwide de facto standard for describing integration problems. These patterns offer a standardized way to split huge, complex integration scenarios into smaller recurring problems. The following slide deck revisits EIPs and gives an overview about the status quo and its relevance regarding modern concepts such as big data, Internet of Things, cloud, and microservices. The following shows a simple, but powerful diagram using several different EIPs, such as a splitter or router (source: http://www.enterpriseintegrationpatterns.com/):
Fortunately, EIPs offer more possibilities than modeling integration problems in a standardized way. Frameworks and tools implement these patterns, so the developer does not have to implement EIPs on his own.
Trends: Hybrid Integration and the Citizen Integrator
End users’ behavior has changed, yet mission critical systems still need to be integrated on the enterprise level. However, technical people (no developer, but some technical knowledge) want to realize simple integrations with partners by themselves. “Hybrid Integration,” as Forrester calls it, can help by offering integration in the cloud via web UIs and self-service. Business users also want (and need) to be more practical without having to ask IT departments for help. This is what Gartner calls the “Citizen Developers” or “Citizen Integrators.” Business users do simple integrations by themselves. No matter how you name these new trends, they are on the way to the IT world of most enterprises.
TIBCO is following this trend! Take a look at the TIBCO NOW Tour in Paris and London in 2015: TIBCO announced different tools for different end users.
Relation between Integration Offerings and Enterprise Integration Patterns
How do EIPs map to this trend? They are used everywhere, in varying degrees of complexity. The future integration portfolio will contain offerings for business users, citizen developers, and classical enterprise integration scenarios. EIPs will be used without your awareness. Let’s take a look at different offerings and how to use EIPs within:
- TIBCO Simplr offers an intuitive web-based user interface for the Business User, or the “Citizen Integrator”, to realize personal integration without technical knowledge, such as synchronizing data via self-service from SaaS offerings like Salesforce and Marketo. As this person does not have much technical knowledge, just some basic, but powerful, EIPs are used. Pipes and filters are used to transfer data from one system to another. This, in combination with a content-based router, is enough to satisfy the business user.
- TIBCO Cloud Integration offers intuitive web-based integration for people with some technical understandings, called “Incidental Integrators” by Gartner, to create and deploy REST services or to configure connections and policies of open APIs. It can be used to build microflows. Contrary to TIBCO Simplr, it uses more complex patterns such as invalid message channel or dead letter queue to realize error handling, policies, and other unexpected behavior.
- TIBCO ActiveMatrix BusinessWorks Enterprise Edition is TIBCO’s flagship integration product for enterprise integration scenarios. It can be deployed on-premise on physical hardware, on virtual machines, or in IaaS cloud infrastructures such as Amazon EC2. The full stack of EIPs is used. Powerful composites of EIPs are used to map, route, and integrate different on-premise and cloud applications. Unexpected behavior can be implemented, as well as distributed systems, scalability, correlation of different events, and guaranteed delivery. Patterns such as the correlation identifier, message expiration, dynamic router and recipient list are used.
- TIBCO ActiveMatrix BusinessWorks Container Edition is used to develop integration applications that are deployed natively onto the Cloud Foundry PaaS. “Cloud native applications” can be developed to be more agile, quickly change, and provide web scale. Similar to TIBCO ActiveMatrix BusinessWorks Enterprise Edition, the full stack of EIPs is utilized to implement complex integration scenarios.
EIPs for Streaming Analytics, IoT and Microservices
EIPs are important for “classical integration“ using an Enterprise Service Bus like TIBCO ActiveMatrix BusinessWorks or new cloud integration services such as TIBCO Cloud Integration or TIBCO Simplr.
EIPs are also used with frameworks and tools for stream processing (e.g. TIBCO StreamBase), API management and application services governance (e.g. TIBCO Mashery), and any other tool which uses some kind of integration logic, like Internet of Things, or building business logic via microservices. The tasks of EIPs are very similar to integration offerings: you route, filter or map data from one endpoint to another one. Sometimes, you combine the EIPs with different concepts.
While “classical integration” is usually based on concepts such as request-response or publish-subscribe communication, stream processing processes data while it is in motion using concepts such as continuous queries and sliding windows. However, in both scenarios, EIPs are needed to route, filter or map data between endpoints.
EIPs for Log Management and IT Operations Analytics (ITOA)
Log management (TIBCO LogLogic) and IT operations analytics (ITOA, TIBCO LogLogic Unity) is another hot topic in the market today, where EIPs are used for processing machine data and other events. Take a look at the typical workflow, which translates to several different EIPs:
Slide Deck “Enterprise Integration Patterns Revisited”
Here is the slide deck, which discusses the above content in more detail:
Integration is Key to a Successful Future
The number of different data sources and technologies has increased more today than ever before:
- Legacy systems such as CRM, ERP, Mainframe or B2B via EDIFACT will not disappear
- Big data clusters such as Data Warehouse, Apache Hadoop or NoSQL databases will store exponentially more data year by year
- Streaming server and in-memory data grids will process more data in real time in the future
- Cloud, mobile, APIs, big data, Fast Data, Internet of Things are not an option, but everybody’s future!
Therefore: EVERYTHING HAS TO BE INTEGRATED! Enterprise Integration Patterns are a key piece in this puzzle, regardless of which problem you have to solve or which concepts you need to use.
Leave a Comment
You must be logged in to post a comment.
