Contributed on July 21, 2015

5.5K

1.4K

Placement of BPM runtime components� in an SOA environment

Description

The service oriented architecture (SOA) reference architecture is intentionally simplistic at a high level but it holds some surprises when you look closely at how components really interact. This is especially true in relation to the placement of business process management (BPM) componentry. We discuss the most common design questions including: Is BPM a consumer or provider of services? To what extent should a user interface, be decoupled from the BPM runtime? How do we retain agility in BPM while adhering to the architectural separation of SOA? These subtleties are critical when designing solutions to reap benefits of both SOA and BPM simultaneously.

Transcript

© 2012 IBM Corporation
IBM Software Group
Placement of BPM runtime components
in an SOA environment
Presenter: Kim Clark
Authors:
Kim Clark (kim.clark@uk.ibm.com)
Brian Petrini (petrini@us.ibm.com)
Version 1.6
IBM Software Group
© 2012 IBM Corporation
Related article
Please note, the following article has been published that
covers the concepts in the first half of this presentation:
“The placement of BPM runtime components in an SOA”
http://www.ibm.com/developerworks/bpm/bpmjournal/1404_clark/1404_clark.html
25/01/14
IBM Software Group
© 2012 IBM Corporation
How complicated can a simple sequence of boxes be?
Process
IBM Software Group
© 2012 IBM Corporation
Common questions arising when considering BPM and SOA
https://collaboration.opengroup.org/projects/soa-ref-arch
•  Should all requests go through business process management (BPM)?
•  Is BPM above or below the enterprise service bus (ESB)?
•  Are processes also services?
•  Should BPM handle page navigation?
•  Should BPM perform integration?
IBM Software Group
© 2012 IBM Corporation
Consumers
Are business processes just another type of consumer?
Services
(Exposure)
Operational Systems
(Applications & Data)
Process Runtime
(BPM as a consumer)
Other
consumer
ServiceProvidersServiceConsumers
Other
consumer
Could/should we remove the business process layer altogether?
Removes temptation to assert that all requests should go through
business process layer.
IBM Software Group
© 2012 IBM Corporation
Sequences of steps
Where could they occur / Where should they occur?
Service
Exposure
Operational
Systems
(Applications
& Data)
Integration
Consumers
Business
Process
IBM Software Group
© 2012 IBM Corporation
People based processes
Service
Exposure
Operational
Systems
(Applications
& Data)
Integration
Consumers
Business
Process
IBM Software Group
© 2012 IBM Corporation
What makes a business process suitable for BPM?
Performing the process provides value to the business
The process contains individually business relevant steps
Business relevant data flows through the process
The process follows a relatively structured path
The steps within the process are performed by multiple roles/teams.
The process changes over time as a result of changes in the business
Create
Account
Capture
Details
Send
Documents
Sign
Documents
Activate
Account
Customer
Call centre
Back office
Notify
Customer
New
revenue!
Existing customer
New customer
— ———– ————- ——- –
— ———– ————- ——- –
— ———– ————- ——- –
— ———– ————- ——- — ———– ————- ——- –
— ———– ————- ——- –
IBM Software Group
© 2012 IBM Corporation
Consumers
BPM as a consumer of services
Services
(Exposure)
Operational Systems
(Applications & Data)
Business Process
(Orchestration)
Process Runtime (as a consumer)
ServiceProvidersServiceConsumers
A business process orchestrates (consumes) services in order to automate steps in the process
Human
task
System
task
Exposed
service
Back end
system
Misleading
representation
of an interaction
Interaction
Key
IBM Software Group
© 2012 IBM Corporation
Consumers
Positioning of the integrated BPM UI
Services
(Exposure)
Operational Systems
(Applications & Data)
Business Process
(Orchestration)
Process Runtime (consumer)
ServiceProvidersServiceConsumers
Typically an internal BPM UI is provided to enable users to view and work their tasks.
Integrated BPM UI
IBM Software Group
© 2012 IBM Corporation
Consumers
Introducing an external BPM user interface
Services
(Exposure)
Operational Systems
(Applications & Data)
Business Process
(Orchestration)
Process Runtime (consumer)
Integrated BPM UI
ServiceProvidersServiceConsumers
Often some, or all of the human tasks need to be served up by an external
custom portal due to the specific requirements of the graphical user interface.
retrieveTask()
updateTask()
completeTask()
Independent BPM UI
IBM Software Group
© 2012 IBM Corporation
Systems acting as both providers and consumers simultaneously
§  The layered architecture is a logical model, not a physical deployment
§  Components may appear more than once if they perform different roles
§  Many operational systems are both providers and consumers
Services
(Exposure)
Operational Systems
(Applications & Data)
System A
(as a provider)
ServiceProvidersServiceConsumers
Consumers System B
(as a consumer)
System A
(as a consumer)
System B
(as a provider)
IBM Software Group
© 2012 IBM Corporation
Consumers
The process runtime acting as a provider of services
Services
(Exposure)
Operational Systems
(Applications & Data)
Business Process
(Orchestration)
Process Runtime (as a consumer)
Integrated BPM UI
Process Runtime
(as a provider)
Independent BPM UI
ServiceProvidersServiceConsumers
In reality, the business process runtime also becomes a service provider in this situation
providing services for external systems to read and manipulate tasks. So the BPM
component now lives in two places on the diagram, each with a specific role.
BPM is both above and below the service layer
retrieveTask()
updateTask()
completeTask()
Internalinteractionwithin
businessprocessruntime
API
IBM Software Group
© 2012 IBM Corporation
Pros and Cons of fully decoupling the process API
Pros:
•  Enables hiding of routing, versioning, security models etc.
•  Becomes a standardize process capability, independent of technology and
implementation
•  Standardized monitoring and controlling service level agreements, throttling
traffic
•  Standardized governance: Enforcing policies for things such as encryption,
deprecation.
•  Optionally enables standardization of data model, but see Cons below.
Cons:
•  Significant up front work before the first project can use the API
•  Results in another layer to be maintained, more skillsets involved operationally.
•  May render the standard API product documentation less effective, and new
documentation may need to be created and maintained. Particularly true if a new
data model is introduced.
Recommendations
•  Only consider decoupling where multiple highly independent consumers are
present, each with strong reliance on well defined service level agreements.
•  Focus on requirements around traffic management, and security.
•  Avoid replacing/translating the APIs data model. This is a lot of work both up
front and ongoing for very little gain.
IBM Software Group
© 2012 IBM Corporation
Page navigation
Service
Exposure
Operational
Systems
(Applications
& Data)
Integration
Consumers
Business
Process
IBM Software Group
© 2012 IBM Corporation
Separating Process, Page Navigation and Page Design
Loan Requester
Loan Approver
System
Approve
Loan
Process
Loan
Capture
Loan Details
Capture
Personal Details
Capture
Financial Details
First name
Second name
Date of birth
Personal details
etc.
What’s wrong with
this process model?
Multiple steps in the same human swimlane of a BPMN diagram suggest there is an issue with
the granularity of the process modeling.
It implies they could have been performed by the same person in one go, so they are not really
separate steps.
We should not model this low level page navigation in the BPMN diagram.
IBM Software Group
© 2012 IBM Corporation
Separating Process, Page Navigation and Page Design
Loan Requester
Loan Approver
Capture
Loan Application
System
Approve Loan
Process Loan
Process
Capture Loan Application
Loan
Details
Personal
Details
Financial
Details
First name
Second name
Date of birth
Personal details
etc.
Page navigation
Page
Page navigation should be
modeled separately from the
BPMN swimlane diagram.
IBM Software Group
© 2012 IBM Corporation
Separation of concerns when creating a custom UI for BPM
Independent user interfaces for BPM
should also take on the page navigation.
Example issues
•  Page navigation depends on the
page data
•  There are often data dependencies
across pages
•  Pages/data may need to be pre-
loaded based on navigation
•  The ideal UI data model will be
different to the process model
Process Layer
Presentation Layer
Page navigation
Page
Process
IBM Software Group
© 2012 IBM Corporation
Consumers
Is a process also a service?
Services
(Exposure)
Operational Systems
(Applications & Data)
Business Process
(Orchestration)
Process Runtime (consumer)
Integrated BPM UI
ServiceProvidersServiceConsumers
Process Runtime
(provider)
submitLoanApplication()
Service
Components
startProcess()
User Interface
Internalinteractionwithin
businessprocessengine
API
“Is a process a service?” – No!
But a service can result in the initiation of a process.
IBM Software Group
© 2012 IBM Corporation
Processes with integration
Service
Exposure
Operational
Systems
(Applications
& Data)
Integration
Consumers
Business
Process
IBM Software Group
© 2012 IBM Corporation
Single Transaction
for all invocations
Multiple Transactions
across invocations
Human Tasks
Terminology: Orchestration vs. Process vs. Composition?
Orchestration is a sequence of “invocations” of systems, or humans
Appear to the business as a single step.
Do not include human interaction from the
business.
Take long enough that the business need
to be aware of their intermediate in-
progress states.
Include human interaction, or interaction
with slow responding systems.
Orchestration
Composition Process
Note: These are not industry standard terms,
but derive from common usage observed in the field
IBM Software Group
© 2012 IBM Corporation
Core types of Orchestration: Process vs. Composition
Process
Makes calls to mature high level services
Often triggered (i.e. one way call) rather than invoked as a two way call
Where it is invoked as a two way interaction, the caller is typically asynchronous (i.e. not
a user) and therefore the service level agreement is throughput based rather than
response time based
Stateful persistence of the steps in the process
Events can correlate with the running process
Often involves human interaction to perform some tasks within the process
Composition
Grouping of relatively fine grained interactions
Typically called in real-time in a request/response (two way) interaction style.
Response time is the primary driver for the service level agreement
Common for aggregation functions
Some or all the granular interactions may not themselves be exposed as re-usable
services
Generally state free
Never involves human interaction during the composition
IBM Software Group
© 2012 IBM Corporation
Separating the business process from low level composition
Loan Requester
Loan Approver
System
Approve
Loan
Save
Loan Details
Capture
Loan Application
What’s wrong with
this process model?
Multiple steps in the same system swimlane of a BPMN diagram suggest there is an
issue with the granularity of the process modeling.
It is likely we are modeling the separate systems users have to update today.
In the future process this multiple update will be automated, so it is no longer be
relevant to model the individual steps on the business process diagram.
We should not model this low level integration logic in BPMN diagram.
Save
Personal Details
Save
Financial Details
IBM Software Group
© 2012 IBM Corporation
Pros/cons of moving composition out of the process layer
Orchestration modelled and implemented within the
process layer.
Good for visibility, but process much more complex.
Granular services exposed for re-use.
Interactions traverse many layers (poor performance)
Orchestration modelled and implemented in the
integration layer.
Hides integration complexity from process.
Course grained services exposed for re-use
Interactions are close to systems (good
performance)
Service
Exposure
Operational
Systems
(Applications & Data)
Integration
Layer
Business
Process
Layer
Composition
Business Process
a b c
Service
Exposure
Operational
Systems
(Applications & Data)
Integration
Layer
Business
Process
Layer
Business Process
ba c
IBM Software Group
© 2012 IBM Corporation
Just a selection of the different patterns of orchestration
Integrated
workflow
Stateful
integration
Aggregation
Isolated
transaction
Composition
Exceptions
only
Process
Stateless* engine Stateful* engine
* This refers to persistence of orchestration state
Real time retrieval
of data from
multiple systems
Real time updates
to multiple systems
that can be
combined into a
single update
Straight through
processing across
systems that can’t be
combined
transactionally
People based
exception handling
Processes
integrating people
and systems
IBM Software Group
© 2012 IBM Corporation
“Process” patterns
Integrated
workflow
Stateful
integration
Exceptions
only
Process
Stateful engine
IBM Software Group
© 2012 IBM Corporation
Pure workflow process
People based process with no system integration
All data is stored in the process. State is
stored as each user works a task
Model changes can be in more in the hands
of the users.
Examples
• Paper based processes
• Procedures involving multiple teams of
people
Primary benefits:
• Reduction/removal of paper
• Improved work distribution/prioritisation
• Management visibility on process status
• Common understanding of process model
Implementation: Stateful orchestration
engine with good task user interface
capabilities.
System based activity
People-based activity
Queued events
State persistence
IBM Software Group
© 2012 IBM Corporation
Integrated workflow process
People and systems are involved in the process
Enables progressive automated of tasks
Users directly involved in modelling of process.
IT required for integration.
Examples
• Processes where elements of data are
duplicated across multiple systems
• Processes where system updates are
unnecessarily complex for users
Benefits:
• Benefits of Pure Workflow
• Reduce/re-assign workforce
• Improve end-to-end time
• Improve data consistency
Implementation: Stateful orchestration engine
with good integration capabilities. Significantly
simplified if implemented in conjunction with a
service oriented architecture.
System based activity
People-based activity
Queued events
State persistence
IBM Software Group
© 2012 IBM Corporation
Process with people for exceptions only
Main path is straight through, people handle known exception paths
Users can progressively refine
exception path rules to increase
automation. A step in the direction of
Straight Through Processing (STP)
Examples
• Approval processes rules enable
some auto-approval
Benefits:
• Benefits of Integrated Workflow
Process
• Manage proportionally higher
volumes
Implementation: Orchestration
engine with integration and rules
capabilities.
System based activity
People-based activity
Queued events
80%
20%
State persistence
IBM Software Group
© 2012 IBM Corporation
“Composition” patterns
Stateful
integration
Aggregation
Isolated
transaction
Composition
Stateless engine
IBM Software Group
© 2012 IBM Corporation
Aggregation
No transactions required, just gathering data
Data gathered from multiple sources and merged into a single
result.
By far the most common requirement. Most solutions need
convenient access to rich data.
Examples
• Gathering quotations from multiple insurance brokers (parallel)
• Combining customer data from multiple systems (sequenced)
Benefits
• Relatively simple implementation since no data integrity/
transactionality requirements.
• Failures part way through can be simply re-tried
• Hides complexities of data translation/merge and connectivity to
disparate systems
Implementation: Ideally performed as close to the back end
systems as possible. Integration engine preferable due to strong
data merging capabilities, and more direct connectivity to back
end systems.
Challenges
•  Should you wait for all responses
•  How do you merge parallel requests
•  Which layer in the architecture should this be performed.
System based activity
IBM Software Group
© 2012 IBM Corporation
Isolated transaction
Multiple transactional calls to the same system
Provides a real-time response regarding the status of the
update to the caller. Commonly used to commit changes
requested by user interfaces providing immediate feedback of
success.
Examples
• Updates to multiple tables in the same database. e.g to
create an order and its order items.
Benefits
• Ensures data integrity across multiple calls to the same
system
• Failures will be fully rolled back, so requests can be simply
re-tried.
Implementation: Ideally done as close to the data as possible
for performance, and simplicity of the transactional protocols.
Highest performance will be within the back end system (e.g. a
stored procedure). Next best is a integration engine that can
handle the transactional protocol (such as JDBC).
Challenges
• Transactions typically hold resources (e.g. threads/memory)
whilst in progress
• The aggregate of all the steps may take longer than the
transaction timeout.
• Transactions containing multiple steps can cause deadlocks
if coding guidelines are not adhered to.
Transaction Boundary
System based activity
Single
phase
commit
IBM Software Group
© 2012 IBM Corporation
Distributed transaction
Data integrity critical across multiple systems
Requires a independent transaction controller to perform a
global transaction across more than one system. This requires
that the systems are enabled for two phase commit transactions.
Examples
• Committing changes to multiple databases
• Committing changes to a queue and a database.
In reality two phase commit is more commonly used “within” a
system, rather than across multiple systems. For the reasons
noted in “Challenges”
Benefits
• Ensures data integrity across calls multiple systems
• Failures will be fully rolled back, so requests can be simply re-
tried.
Challenges
• Few system owners will allow a two phase commit to be
performed against their systems by external parties due
concerns over resource locking.
• Two phase commit require the maintaining of a transaction log
so systems can re-align should a failure happen mid transaction.
This adds interdependencies that complicate design of high
availability and disaster recovery.
• Deadlocks are much more difficult to diagnose than with single
phase commit.
• Two phase commit is much more heavyweight in CPU both for
the end systems, and for the caller than single phase commit.
Transaction Boundary
System based activity
Two
phase
commit
IBM Software Group
© 2012 IBM Corporation
Enriched update
Only the final update matters, everything else is preparation
Transaction Boundary
System based activity
read non-critical
update
critical
update
*Definition: “Critical Update” Any request that performs a change to data (create, update
or delete) that MUST be resolved in the case of a failure, either with a transactional
rollback, or with compensatory actions if the update has already been committed.
A complex scenario is simplified to focus on the
one key update, thereby removing the need for
intermediate state.
Examples
• Checking for existence of a parent objects
before creating a child object – e.g. checking
customer exists before creating order.
• Validating data before performing update –
e.g. checking for stock inventory before
submitting an order.
Benefits
• Enables a complex composition yet still
removes the need for stateful orchestration.
Challenges
• Re-tries result in repeating all of the pre-
critical steps
• Invocations deemed to be “non-critcial” may
result in confusion during diagnosis of issues.
IBM Software Group
© 2012 IBM Corporation
Event distribution/broadcast
We only need to know it will be done, not that it has been done
Messages are placed in a number of queues within a
single transaction.
Examples
• Useful for updates to systems that are not expected to
be instantaneously up to date. E.g. data warehouses,
audit logs
• Useful for notification type requests (SMS, email etc.)
Benefits
• Provides a swift answer to the caller
• With persistent queues, the messages should never
be lost, and will be applied to their destinations
eventually.
Challenges
• Destination systems have not received the data when
the response is given
• The time taken for destinations to receive the update
is not in the control of the composition
• Should problems occur processing the messages, the
composition, and therefore also the caller are unaware.
Therefore reliant on external exception handling.
Transaction Boundary
System based activity
Queued events
IBM Software Group
© 2012 IBM Corporation
Basic Interaction Types
ProviderRequester
getCustomer(CustomerID)
returns Customer
ProviderRequester
submitOrder(Order)
a)  Fire and forget
b) Request-response
A messaging transport is used. The request is “one-way” sending data to the provider, but with no response.
The requester need only wait for the message to be received by the transport – i.e. it is non-blocking.
This is the style typically used to initiate a process. i.e. we do not wait for the process to complete.
A synchronous transport is used. The requester requires response data, and must wait for the provider to
respond. The provider must be available at the time of the request and must process the request
immediately.
This is the style typically used with compositions. i.e. the caller requires
a response from the composition.
IBM Software Group
© 2012 IBM Corporation
Stateful Integration – The pattern with two homes
Integrated
workflowAggregation Isolated
transaction
Composition
Exceptions
only
Process
Stateless engine Stateful engine
Stateful
integration
IBM Software Group
© 2012 IBM Corporation
Stateful Integration Process
Zero or near zero human interaction
All steps are system invocations. Multiple separate
transactional interactions are required. State must
be stored in between each step, ideally as part of
the same transaction.
Examples
• Data synchronisation.
e.g. “Change of Address”
• “Straight Through Processing” (STP).
e.g. payments processing
Benefits
• Enables orders of magnitude higher volumes.
e.g. moving sales to the internet
• Enables precise data integrity across disparate
systems
Implementation: Depends significantly on non-
functional requirements – throughput/volumes.
See next slide for options.
System based activity
Queued events
State persistence
Transaction Boundary
IBM Software Group
© 2012 IBM Corporation
Implementation options for Stateful Integration Process
Orchestrated interactions
Pros
Mature products available (e.g. BPM)
First class notion of “process”
Modelled process visible in implementation
Monitoring/reporting at process level
Process version migration support
Clearer modelling of exception paths
Easier to introduce human interaction
Cons
Higher CPU/network due to persistence
Solution will be split between business process
integration layers
Chained interactions
Pros
Higher performance due to minimal persistence
Can be implemented entirely in the integration layer
Cons
Largely a custom solution
No notion of “process”, only the individual steps
Process “model” not visible in implementation
Custom work for effective monitoring/reporting
Exception paths harder to understand
Custom work to migrate to new processes
Custom work to introduce human interaction
Queued eventsState persistence
IBM Software Group
© 2012 IBM Corporation
What orchestration patterns are required?
Service Exposure
Operational Systems
(Applications & Data)
Integration
Consumers
Business Process Integrated workflow
Page Navigation
Enriched Update
Aggregation Stateful integration
IBM Software Group
© 2012 IBM Corporation
Core characteristics for assessing orchestration requirements
Time-span
Granularity
Human interaction
Principal objects
Application integration
Complexity
Monitoring
Flow ownership
Dynamicity
Transactionality
Performance
Volumes
Business value
State management
Security
Infrastructure
IBM Software Group
© 2012 IBM Corporation
Many patterns, many possible implementation options
Integration
flow pub/
sub
Business
process
short
running
Business
process
briefly
persisted
Integration
flow 1:n
Integration
flow 1:1
with
translation
Business
Process
long
running
system
centric
Business
Process
long
running
human
centric
Integration
flow 1:1
routing only
Many attempts at decision trees for implementation choices fail because they try to cover the whole many-
to-many mapping from all possible requirements to all possible implementations. This is typically an
impossibly hard decision tree to draw, and inconceivably complex to navigate.
Implementation Options
Requirements
more…more…
IBM Software Group
© 2012 IBM Corporation
Narrowing the complexity using a two stage decision tree
Patterns
Implementation
Types
Use architectural principles
to determine the relevant
implementation choices for
that pattern
Use requirements to
choose the pattern
IBM Software Group
© 2012 IBM Corporation
Types of orchestration in relation to transactionality
Do any of the requests perform critical updates*?
No
Aggregation Enriched update
Yes
Can all critical updates* be
performed in one transaction?
Distributed
Transaction
Stateful
Integration
Yes No
How many critical updates*?
=1 >1
*Definition: “Critical Update”
Any request that performs a change to
data (create, update or delete) that MUST
be resolved in the case of a failure, either
with a transactional rollback, or with
compensatory actions if the update has
already been committed.
Non-transactional Isolated transaction Global transaction Multi-transaction
IBM Software Group
© 2012 IBM Corporation
No single answer
Process solutions are always a combination of orchestration patterns
Service Exposure
Operational Systems
(Applications & Data)
Integration
Consumers
Business Process
Data
synchronization
Microflow
In-app.
workflow
Screenflow/
page navigation
Stored procedure
Straight-through
processing
Business process
IBM Software Group
© 2012 IBM Corporation
References
The placement of BPM runtime components in an SOA
http://www.ibm.com/developerworks/bpm/bpmjournal/1404_clark/1404_clark.html
Process implementation types: Patterns based design for process-based solutions
http://www.ibm.com/developerworks/websphere/library/techarticles/1004_clark/1004_clark.html
Process-oriented modeling for SOA
http://www.ibm.com/developerworks/library/ar-procmod1