Contributed on October 3, 2013

3.1K

1.4K

BPM in the Cloud: Early Research Works and Challenges

Description

With an estimated market volume of US$ 130 billion and double-figure growth rates, cloud computing is on the path to change many aspects of our digital lives. In this talk I consider the question of potential benefits and open challenges when bringing BPM research into the cloud, such as (i) how can BPM benefit from the cloud, (ii) what should BPM in the cloud look like, and (iii) what can BPM bring to cloud computing practices? In this talk I will present three works around those questions, namely (a.) end-user-friendly process modelling in the cloud, (b.) process-aware auto-scaling of cloud resources, and (c.) process mining for error diagnostics in cloud management processes.

Transcript

NICTA Copyright 2012 From imagination to impact
BPM in the Cloud:
Early Research Works
and Challenges
Dr. Ingo Weber
Senior Researcher
NICTA, Sydney (ATP)
September 2013
NICTA Copyright 2012 From imagination to impact
Cloud Service Market Growth vs. BPM Market
Gartner (2013)
• The cloud is big, and here to stay
NICTA Copyright 2012 From imagination to impact
Cloud Service Market Growth vs. BPM Market
Gartner (2013)BPM market estimated at $2.6B in 2013, $7B in 2018
(PRWeb 16/6/2013)
• The cloud is big, and here to stay
• By comparison, BPM is a dwarf
NICTA Copyright 2012 From imagination to impact
Why BPM in the Cloud?
• BPM as a standalone system may not take off…
– But there are opportunities to use BPM thinking,
techniques, and tools in other domains / applications
• BPM in the cloud will happen
– If something provides value, the industry will build it
– This is an opportunity for the BPM research
community to have practical impact on a promising
area
• Many avenues to explore…
– This talk aims to provide a short overview of where
we are at, what is needed, and some of my works.
4
NICTA Copyright 2012 From imagination to impact
Cloud Computing Overview
5
Service Model
SaaS
Software as a Service
Examples: CRM, email, online games, vitual
desktops, etc.
PaaS
Platform as a Service
Examples: web server, database, execution
runtime, development tools, etc.
IaaS
Infrastructure as a Service
Examples: virtual machines, storage, load
balancers, network, etc.
Examples in BPM
In-browser modelling, e.g., Signavio, Oryx
Specialized environments, e.g., IFTTT, Effektif
Complete environments, e.g., Intalio|Create, Cordys
(now OpenText)
BPM execution platforms, e.g., Intalio
Workflow tools, e.g., AWS Simple Workflow Service
–
Left-hand side based on Wikipedia article Cloud Computing, 09/2013
NICTA Copyright 2012 From imagination to impact
SaaS-level BPM for end-users?
• Characteristics needed
– Limit functionality to something small but useful
– User experience
– Simplicity
– Social aspects for easy connection to
colleagues/friends
– Short learning curve: people need to understand the
tool very quickly
• Are we there yet?
– …or should we try and re-think what BPM in the cloud
can be?
6
Top part inspired by T. Baeyen„s talk at BPM, 08/2013
NICTA Copyright 2012 From imagination to impact
Example: IFTTT does this quite well
7
www.ifttt.com, 09/2013
NICTA Copyright 2012 From imagination to impact
Example recipes in IFTTT
8
www.ifttt.com, 09/2013
Whenever a game of
the Boston Red Sox
ends, send me an
email
NICTA Copyright 2012 From imagination to impact
Example recipes in IFTTT
9
www.ifttt.com, 09/2013
Whenever a game of
the Boston Red Sox
ends, send me an
email
Whenever someone
tags me on a facebook
photo, download that
photo to my dropbox
account
NICTA Copyright 2012 From imagination to impact
Example: IFTTT
10
www.ifttt.com, 09/2013
If there is a birthday of
a friend in my Google
calendar for today,
post „happy birthday“
on their wall
NICTA Copyright 2012 From imagination to impact
Are we there yet?
• Questions to drive research directions
– How can BPM benefit from the cloud?
– What should BPM in the cloud look like?
– What can BPM bring to cloud computing practices?
• Next: some of my early works
11
NICTA Copyright 2012 From imagination to impact
How can BPM benefit from the cloud?
• SaaS
– No installation, very little burden for people to get
started – if the tool fulfils the characteristics while
being useful
– My work: FormSys Process Designer [1,2]
• PaaS / IaaS
– Use cloud resources to enable process execution
– Benefit from elasticity to stay within SLAs, but not
over-pay for over-provisioning
– My work: process-aware auto-scaling of cloud
resources (on the IaaS level) [3]
12
NICTA Copyright 2012 From imagination to impact
Process-aware auto-scaling of cloud resources
• Assumption
– Executing process model that calls Web services (WSs)
– WSs are fully automated and perform compute-intensive tasks
– Horizontal scaling of the VMs running the WSs is possible
– Complexity of a task can depend on the input
• Example
– Integration logic around proprietary systems (e.g., in COBOL)
– Translation, transformation, aggregation can create high
utilization of resources (CPU, disk I/O, etc.) and slow down
process execution
• Problem
– Say we have external SLAs for the whole process, e.g., to
complete 95% of instances within 300s
– How many resources to schedule for each task, when the
requests change (complexity, frequency)?
13
NICTA Copyright 2012 From imagination to impact
• Setup of our experiment
14
Process-aware auto-scaling of cloud resources
Elastic Load Balancer 1
Auto Scaling Group 1
Instances for Service 1
Elastic Load Balancer 2
Auto Scaling Group 2
Instances for Service 2
Elastic Load Balancer 3
Auto Scaling Group 3
Instances for Service 3
ServiceProvider
Service 1 Service 2 Service 3
NICTA Copyright 2012 From imagination to impact
• Early work: use a fuzzy controller for process-aware auto-scaling
• Process backlog on cloud IaaS: process-aware vs. standard auto-scaling
Process-aware auto-scaling of cloud resources
Load profile (different
scale)
Complexity of the
requests increases,
plateaus, then
decreases, and finally
no new instances are
created
NICTA Copyright 2012 From imagination to impact
• Early work: use a fuzzy controller for process-aware auto-scaling
• Process backlog on cloud IaaS: process-aware vs. standard auto-scaling
Process-aware auto-scaling of cloud resources
Backlog: in a given
minute, how many
more process
instances are started
than completed?
Load profile (different
scale)
NICTA Copyright 2012 From imagination to impact
• Early work: use a fuzzy controller for process-aware auto-scaling
• Process backlog on cloud IaaS: process-aware vs. standard auto-scaling
Process-aware auto-scaling of cloud resources
Load profile (different
scale)
“Cut-offs” show where
time-outs happened,
caused by the ELBs.
NICTA Copyright 2012 From imagination to impact
What should BPM in the cloud look like?
• SaaS
– My work: FormSys Process Designer (2010-2011) [1,2]
– Metaphor: forms – ideally the ones known by users
– Approach: scripting (not Programming by Demonstration), but
made easy
– For execution & to enable technical editing of end-user-created
process models: translation to BPEL
– Features:
• user-editable forms are linked to Web services
• restricted, yet powerful generic composition language
• immediate automated process verification
• process simulation in the design phase
• automatic code generation: parallelized orchestrations
– Evaluated in a user study
18
NICTA Copyright 2012 From imagination to impact
FormSys Process Designer – core ideas
• Service: name + icon
– Visualize Data Service
• Messages to / from services
– Example: Sirca, computing GARCH Daily Volatility
19
Form through which
the user usually
controls a service
NICTA Copyright 2012 From imagination to impact
FormSys Process Designer – core ideas
• Service: name + icon
– Visualize Data Service
• Messages to / from services
– Example: Sirca, computing GARCH Daily Volatility
20
Form through which
the user usually
controls a service
Fields in the form
can be linked to
parts of WSDL
messages (by a
developer)
NICTA Copyright 2012 From imagination to impact
FormSys Process Designer – example
• Financial data analysis process:
1. Find news data: e.g., news data on the company „BHP‟
2. Find performance data: e.g., on company code „BHP.AX‟
3. Aggregate the performance data, e.g., avg. hourly stock prices
4. Merge datasets: e.g., merge the result data sets from the first
and third steps
5. Visualize dataset: e.g., to see influence of news on prices
21
Control flow
modelling
Data flow
modelling, see
[1,2] for details
NICTA Copyright 2012 From imagination to impact
FormSys Process Designer – User study [1]
• Ability to solve tasks in FormSys Process Designer
– Tech.: users with technical expertise
– Fin.: users with financial expertise
– Gen.: users with expertise in both
22
0
1
2
3
4
5
6
7
8
Tech. Gen. Fin.
Solved
Not solved
Not attempted
NICTA Copyright 2012 From imagination to impact
• Times by expertise
23
0:00
0:14
0:28
0:43
0:57
1:12
1:26
Gen. Tech. Fin.
Time(h:mm,successonly)
Expertise
FormSys Process Designer – User study [1]
NICTA Copyright 2012 From imagination to impact
• Times by expertise
24
0:00
0:14
0:28
0:43
0:57
1:12
1:26
Gen. Tech. Fin.
Time(h:mm,successonly)
ExpertiseMean time using
Intalio|bpms
FormSys Process Designer – User study [1]
NICTA Copyright 2012 From imagination to impact
What can BPM bring to cloud computing
practices?
• My work: using process mining in systems operations
(a.k.a. cloud administration / management)
– Disclaimer: started only 2 months ago
• Core question: can a process view help in operations?
1. Understanding how a tool is running
2. Understanding what the best
practice is (manual processes)
3. Detecting when process steps are
supposedly complete, but failed
4. Detecting deviations from
best-practice / usual path
5. Detecting unusual log messages
25
NICTA Copyright 2012 From imagination to impact
What can BPM bring to cloud computing
practices?
• My work: using process mining in systems operations
(a.k.a. cloud administration / management)
– Disclaimer: started only 2 months ago
• Core question: can a process view help in operations?
1. Understanding how a tool is running
2. Understanding what the best
practice is (manual processes)
3. Detecting when process steps are
supposedly complete, but failed
4. Detecting deviations from
best-practice / usual path
5. Detecting unusual log messages
26
Process discovery
Assertion checks
Conformance checks
Classification
Approaches:
NICTA Copyright 2012 From imagination to impact
Process Discovery for System Operations
• Challenges:
– How to get the right logs
– How to correlate events from several logs
• Especially in a large-scale distributed system
• What is a process instance
– How to handle concurrent sub-processes
– How to deal with low-level log messages
– How to analyze incomplete log messages
– How to deal with 1000s of log lines per instance
27
NICTA Copyright 2012 From imagination to impact
Process Discovery – Example: Asgard
• Netflix Asgard is a cloud management tool
– Open source
– For Amazon Web Services (AWS)
– Sunny day example: 1 log source, high quality logs, …
• Example process: rolling upgrade
– “Rolling”: upgrade one virtual machine (VM) at a time
– Auto-scaling group (ASG): set of VMs from the same
Amazon machine image (AMI), usually behind an
elastic load balancer (ELB)
– Upgrade by replacing AMI for the ASG, then de-
register a VM from the ELB, terminate the VM, wait
for ASG to create a new VM from the new AMI
28
NICTA Copyright 2012 From imagination to impact
Process Discovery – Asgard logs: 3 lines
[2013-07-18 15:37:31,369] [Task:Pushing ami-ad059597 into group hadoopcluster–
firstASG for app hadoopcluster] com.netflix.asgard.Task 2013-07-18_15:37:31 135:
{Ticket: null} {User: null} {Client: localhost 0:0:0:0:0:0:0:1%0} {Region: apsoutheast-
2} [Pushing ami-ad059597 into group hadoopcluster– firstASG for app
hadoopcluster] Started on thread Task:Pushing ami-ad059597 into group
hadoopcluster–firstASG for app hadoopcluster.
[2013-07-18 15:37:31,996] [Task:Pushing ami-ad059597 into group hadoopcluster–
firstASG for app hadoopcluster] com.netflix.asgard.Task 2013-07-18_15:37:31 135:
{Ticket: null} {User: null} {Client: localhost 0:0:0:0:0:0:0:1%0} {Region: apsoutheast-
2} [Pushing ami-ad059597 into group hadoopcluster– firstASG for app
hadoopcluster] Updating launch from hadoopcluster–firstASG-20130718120001
with ami-ad059597 into hadoopcluster–firstASG-20130718153731
[2013-07-18 15:37:31,998] [Task:Pushing ami-ad059597 into group hadoopcluster–
firstASG for app hadoopcluster] com.netflix.asgard.Task 2013-07-18_15:37:31 135:
{Ticket: null} {User: null} {Client: localhost 0:0:0:0:0:0:0:1%0} {Region: apsoutheast-
2} [Pushing ami-ad059597 into group hadoopcluster– firstASG for app
hadoopcluster] Create Launch Configuration ‘hadoopcluster–firstASG-
20130718153731’ with image ‘amiad059597’
29
NICTA Copyright 2012 From imagination to impact
Process Discovery – Asgard logs: 3 lines
[2013-07-18 15:37:31,369] [Task:Pushing ami-ad059597 into group hadoopcluster–
firstASG for app hadoopcluster] com.netflix.asgard.Task 2013-07-18_15:37:31 135:
{Ticket: null} {User: null} {Client: localhost 0:0:0:0:0:0:0:1%0} {Region: apsoutheast-
2} [Pushing ami-ad059597 into group hadoopcluster– firstASG for app
hadoopcluster] Started on thread Task:Pushing ami-ad059597 into group
hadoopcluster–firstASG for app hadoopcluster.
[2013-07-18 15:37:31,996] [Task:Pushing ami-ad059597 into group hadoopcluster–
firstASG for app hadoopcluster] com.netflix.asgard.Task 2013-07-18_15:37:31 135:
{Ticket: null} {User: null} {Client: localhost 0:0:0:0:0:0:0:1%0} {Region: apsoutheast-
2} [Pushing ami-ad059597 into group hadoopcluster– firstASG for app
hadoopcluster] Updating launch from hadoopcluster–firstASG-20130718120001
with ami-ad059597 into hadoopcluster–firstASG-20130718153731
[2013-07-18 15:37:31,998] [Task:Pushing ami-ad059597 into group hadoopcluster–
firstASG for app hadoopcluster] com.netflix.asgard.Task 2013-07-18_15:37:31 135:
{Ticket: null} {User: null} {Client: localhost 0:0:0:0:0:0:0:1%0} {Region: apsoutheast-
2} [Pushing ami-ad059597 into group hadoopcluster– firstASG for app
hadoopcluster] Create Launch Configuration ‘hadoopcluster–firstASG-
20130718153731’ with image ‘amiad059597’
30
„Instance ID“
NICTA Copyright 2012 From imagination to impact
Process Discovery – Asgard logs: 3 lines
[2013-07-18 15:37:31,369] [Task:Pushing ami-ad059597 into group hadoopcluster–
firstASG for app hadoopcluster] com.netflix.asgard.Task 2013-07-18_15:37:31 135:
{Ticket: null} {User: null} {Client: localhost 0:0:0:0:0:0:0:1%0} {Region: apsoutheast-
2} [Pushing ami-ad059597 into group hadoopcluster– firstASG for app
hadoopcluster] Started on thread Task:Pushing ami-ad059597 into group
hadoopcluster–firstASG for app hadoopcluster.
[2013-07-18 15:37:31,996] [Task:Pushing ami-ad059597 into group hadoopcluster–
firstASG for app hadoopcluster] com.netflix.asgard.Task 2013-07-18_15:37:31 135:
{Ticket: null} {User: null} {Client: localhost 0:0:0:0:0:0:0:1%0} {Region: apsoutheast-
2} [Pushing ami-ad059597 into group hadoopcluster– firstASG for app
hadoopcluster] Updating launch from hadoopcluster–firstASG-20130718120001
with ami-ad059597 into hadoopcluster–firstASG-20130718153731
[2013-07-18 15:37:31,998] [Task:Pushing ami-ad059597 into group hadoopcluster–
firstASG for app hadoopcluster] com.netflix.asgard.Task 2013-07-18_15:37:31 135:
{Ticket: null} {User: null} {Client: localhost 0:0:0:0:0:0:0:1%0} {Region: apsoutheast-
2} [Pushing ami-ad059597 into group hadoopcluster– firstASG for app
hadoopcluster] Create Launch Configuration ‘hadoopcluster–firstASG-
20130718153731’ with image ‘amiad059597’
31
Payload
AMI ID
ASG ID
NICTA Copyright 2012 From imagination to impact
Process Discovery – Example: Asgard
32
NICTA Copyright 2012 From imagination to impact
Process Discovery – Approach
33
NICTA Copyright 2012 From imagination to impact
Open Challenges
34
Analysis Design Implementation Enactment Monitoring Evaluation
On-demand self
service
Broad network
access
Resource pooling
Rapid elasticity
Measured
service
SaaS
PaaS
IaaS
Private cloud
Community
cloud
Public cloud
Hybrid cloud
BPMlife-cycle (Mendling 2007)
NISTDefinitionofCloudComputing
EssentialCharacteristics
Service
Model
Deployment
Models
• Some dimensions when thinking about solutions
Disclaimer: likely incomplete and needs per-tool analysis
NICTA Copyright 2012 From imagination to impact
Summary & Outlook
• BPM in the cloud will happen
– If something provides value, the industry will build it
– This is an opportunity for the BPM research
community to have practical impact on a promising
area
• Presented some early work
– Process-aware auto-scaling of cloud resources [3]
– Domain-user BPM [1,2]
– Process view on system operations using mining
• Contact us for more information
– http://ssrg.nicta.com.au/people/?cn=Ingo+Weber
35
NICTA Copyright 2012 From imagination to impact
References
1. Ingo Weber, Hye young Paik, and Boualem Benatallah. Forms-
based web service composition for domain experts. ACM
Transactions on the Web (TWEB), conditionally accepted, 2013.
2. Ingo Weber, Hye young Paik, Boualem Benatallah. Forms-based
Service Composition. Accepted for publication in ICSOC’11: 9th
International Conference on Service Oriented Computing, short
paper, Paphos, Cyprus, December 2011.
3. Christian Janiesch, Ingo Weber, Michael Menzel, and Jörn
Kuhlenkamp. Optimizing the performance of automated business
processes executed on virtualized resources. In HICSS’14:
Hawaiian International Conference on System Science, January
2014, to appear.
Acknowledgements:
Thanks to Christian Janiesch and Len Bass for their input to the slides.
And thanks to many colleagues for the discussions and feedback.
36