How to create intelligent Business Processes thanks to Big Data (BPM, Apache Hadoop, Talend, Open Source)
Description
BPM is established, tools are stable, many companies use it successfully. However, today’s business processes are based on data from relational databases or web services. Humans make decisions due to this information. Companies also use business intelligence and other tools to analyze their data. Though, business processes are executed without access to this important information because technical challenges occur when trying to integrate big masses of data from many different sources into the BPM engine. Additionally, bad data quality due to duplication, incompleteness and inconsistency prevents humans from making good decisions. That is status quo. Companies miss a huge opportunity here!
This session explains how to achieve intelligent business processes, which use big data to improve performance and outcomes. A live demo shows how big data can be integrated into business processes easily – just with open source tooling. In the end, the audience will understand why BPM needs big data to achieve intelligent business processes.
Transcript
Next-Generation BPM –How to create intelligent Business Processesthanks to Big DataTalend, Global Leader in Open Source Integra7on Solu7ons Kai Wähnerkwaehner@talend.com@KaiWaehnerXing / LinkedInwww.kai-waehner.de © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner ConsultingDevelopingCoachingSpeakingWritingMain TasksRequirements EngineeringEnterprise Architecture ManagementBusiness Process ManagementArchitecture and Development of ApplicationsService-oriented ArchitectureIntegration of Legacy ApplicationsCloud ComputingBig DataContactEmail: kontakt@kai-waehner.deBlog: www.kai-waehner.de/blogTwitter: @KaiWaehnerSocial Networks: Xing, LinkedInKai Wähner © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Key messagesBPM should be used (just) for optimizing business processes!Intelligent business processes need big data and integration!Big data will reduce human interactions in BPM further! © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner • Big data paradigm shiL • Use cases for big data • Intelligent business processes • Technology and product perspec7ve • Live demo Agenda © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner • Big data paradigm shiL • Use cases for big data • Intelligent business processes • Technology and product perspec7ve • Live demo Agenda © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner William Edwards Deming (1900 –1993) American sta7s7cian, professor, author, lecturer and consultant “If you cant measure it, you cant manage it.” Why should you care about big data? © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner è „Silence the HiPPOs“ (highest-‐paid person‘s opinion) è Being able to interpret unimaginable large data stream, the gut feeling is no longer jus7fied! Why should you care about big data? © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Why should you care about big data?“Accessing data is now [again] the cri7cal path in making good decisions!” h`p://www.inforbix.com/friday-‐data-‐stories-‐big-‐data-‐driven-‐decision-‐making/ © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Why does big data exist?Changing InteractionsChanging ScaleChanging ExpectationsCloudSensors © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner The Vs of big dataVolume (terabytes, petabytes) Variety (social networks, blog posts, logs, sensors, etc.) Velocity (real7me or near-‐real7me) Value © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Big Data Integra3on – Land data in a Big Data cluster – Implement or generate parallel processes Big Data Manipula3on – Simplify manipula7on, such as sort and filter – Computa7onal expensive func7ons Big Data Quality & Governance – Iden7fy linkages and duplicates, validate big data – Match component, execute basic quality features Big Data Project Management – Place frameworks around big data projects – Common Repository, scheduling, monitoring Big data tasks to solve – before analysis © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner • Big data paradigm shiL • Use cases for big data • Intelligent business processes • Technology and product perspec7ve • Live demo Agenda © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner h`p://www.slideshare.net/brocknoland/common-‐and-‐unique-‐use-‐cases-‐for-‐apache-‐hadoop Replacing ETL jobs: Binary files © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner h`p://hkotadia.com/archives/5021 Deduce Customer Defec7ons Forecast: Risk management © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner ➜ With revenue of almost USD 30 billion and a network of 800 loca7ons, Macys is considered the largest store operator in the USA ➜ Daily price check analysis of its 10,000 ar7cles in less than two hours ➜ Whenever a neighboring compe7tor anywhere between New York and Los Angeles goes for aggressive price reduc7ons, Macys follows its example ➜ If there is no market compe7tor, the prices remain unchanged h`p://www.t-‐systems.com/about-‐t-‐systems/examples-‐of-‐successes-‐companies-‐analyze-‐big-‐data-‐in-‐record-‐7me-‐l-‐t-‐systems/1029702 Logistics: Flexible pricing © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Content manipula7on at a traveling portal – Which customer reviews are trustworthy? Jean-‐Paul Schmetz, Harvard Business Manager, November 2012 Fraud detection: Fraud mining © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner ➜ A lot of data must be stored „forever“ ➜ Numbers increase exponen7ally ➜ Goal: As cheap as possible ➜ Problem: (Fast) queries must s7ll be possible ➜ Solu7on: Commodity servers and „Hadoop querying“ Global Parcel Service h`p://archive.org/stream/BigDataImPraxiseinsatz-‐SzenarienBeispieleEffekte/Big_Data_BITKOM-‐Leisaden_Sept.2012#page/n0/mode/2up Storage: Compliance © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner ➜ How do you put this big data easily in the hands of the people that need it? ➜ Making the data “ac7onable” is the real challenge. ➜ Seeing the informa7on that helps make a decision on a composite dashboard is just the first step and where too many companies stop. ➜ A business must be able to fire off the business process to execute the decision made regarding the data. Great big data use cases, but …Kai Wähnerh`p://smartdatacollec7ve.com/ma`-‐davies/104576/data-‐driven-‐bpm-‐making-‐big-‐data-‐ac7onable © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner • Big data paradigm shiL • Use cases for big data • Intelligent business processes • Technology and product perspec7ve • Live demo Agenda © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner ➜ Humans have to interpret large data to make decision. Using gut feeling is nothing but gambling. ➜ Just doing big data analy7cs is not enough. Systema7c and monitored human interac7ons are as important to get best outcomes. ➜ An intelligent business process combines big data and BPM. This enables humans to make data-‐driven decisions based on big data analy7cs. Intelligent business processes © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner ➜ Process starts ac7on (Big Data PULL) • Manual or automated • Faster responses (e.g. „catching Osama“) • Be`er outcomes (e.g. „recommenda7on engine“) ➜ Data starts ac7on (Big Data PUSH) • (Usually) automated • Predic7ve processes (e.g. „preven7ng flu epidemic“) • Handle before it happens (e.g. „customer deduc7on“) Intelligent business processes © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner ➜ How are they related? ➜ How to combine? ➜ How to realize this technically? Combination of big data and BPM © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner increase efficiencybetter qualityreduce costsincrease flexibilityenable new business modelsWhy BPM?Kai WähnerBusiness-IT-Alignment © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner • long-running stateful workflows• frequently changing processes• human interactionWhen BPM?Kai Wähner © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner How BPM?Kai WähnerScript Task Service TaskGroovyJavaScriptetc.SOAP Web ServiceEverythingfrom Cobolto Ruby…… or a„big dataservice“ © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner ChallengeKai WähnerSeparation of Concerns © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Separation of ConcernsKai WähnerIntegration• ETL• Connectivity / adaptors to connect to externalsystems using a variety of different protocols • Predefined EIP for message routingBig Data• Processing• AnalyticsBPM• Do queries to make decisions• Human or machineC R E A T E U S E © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner • Big data paradigm shiL • Use cases for big data • Intelligent business processes • Technology and product perspec7ve • Live demo Agenda © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Kai WähnerIntegration• ETL• Connectivity / adaptors to connect to externalsystems using a variety of different protocols • Predefined EIP for message routingBig Data• Processing• AnalyticsBPM• Do queries to make decisions• Human or machineTechnical / product perspective © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Enterprise Integration Patterns © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Enterprise Integration Patterns © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner AmazonS3 s3 = new AmazonS3Client(new Proper7esCreden7als( S3Sample.class.getResourceAsStream(“AwsCreden7als.proper7es”))); String bucketName = ”my-‐first-‐s3-‐bucket-‐” + UUID.randomUUID(); String key = ”MyObjectKey”; try { s3.createBucket(bucketName); s3.putObject(new PutObjectRequest(bucketName, key, createSampleFile())); S3Object object = s3.getObject(new GetObjectRequest(bucketName, key)); ObjectLis7ng objectLis7ng = s3.listObjects(new ListObjectsRequest() .withBucketName(bucketName) .withPrefix(“My”)); s3.deleteObject(bucketName, key); s3.deleteBucket(bucketName); } catch (AmazonServiceExcep7on ase) { // error handling… } catch (AmazonClientExcep7on ace) { // error handling… } Integration framework © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner // Producerfrom(“ftp:toS3″).setHeader(S3Constants.KEY, simple(“order.txt”)).to(“aws-s3://myBucket?accessKey=” + a+ “&secretKey= ” + s)// Consumerfrom(„salesforce://orders__c?user=dummy1“).filter(„customer == ${dummyCustomer}).to(“ibm-database:orderData”)Integration framework (e.g. Apache Camel) © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Enterprise Service Bus (e.g. Talend ESB) © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner ConnectivityRoutingTransformationComplexityof IntegrationEnterprise Service Bus Integra7on Suite Low HighIntegrationFrameworkINTEGRATIONToolingMonitoringSupport+BUSINESS PROCESS MGT.BIG DATA / MDMREGISTRY / REPOSITORYRULES ENGINE„YOU NAME IT“Alternatives for integration+ © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Kai WähnerIntegration• ETL• Connectivity / adaptors to connect to externalsystems using a variety of different protocols • Predefined EIP for message routingBig Data• Processing• AnalyticsBPM• Do queries to make decisions• Human or machineTechnical / product perspective © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Technology perspectiveHow to process big data? © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Slides: h`p://www.slideshare.net/pavlobaron/100-‐big-‐data-‐0-‐hadoop-‐0-‐java Video: h`p://www.infoq.com/presenta7ons/Big-‐Data-‐Hadoop-‐Java How to process big data? © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner The defacto standard for big data processing How to process big data? © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Even MicrosoL (the .NET house) relies on Hadoop since 2011 How to process big data?“A big part of [the company’s strategy] includes wiring SQL Server 2012 (formerly known by the codename “Denali”) to the Hadoop distributed compu7ng plasorm, and bringing Hadoop to Windows Server and Azure” © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Apache Hadoop, an open-‐source soLware library, is a framework that allows for the distributed processing of large data sets across clusters of commodity hardware using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computa7on and storage. What is Hadoop? © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner How to process big data? © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner MapReduceHDFSEcosystemFeaturesincludedHadoop Distribu7on Integra7on Suite few manyApacheHadoopPackagingDeployment-ToolingSupport+Tooling / ModelingCode GenerationSchedulingOther Tools (ESB, BPM, …)+Hadoop alternatives © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Kai WähnerIntegration• ETL• Connectivity / adaptors to connect to externalsystems using a variety of different protocols • Predefined EIP for message routingBig Data• Processing• AnalyticsBPM• Do queries to make decisions• Human or machineTechnical / product perspective © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner StandardsKai WähnerBPMBPEL BPMNXPDLWF-XMLBPEL4PeoplejPDLARIS EPC © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner BPMNKai Wähner„Business Process Model and Notation (BPMN) is a graphicalrepresentation for specifying business processes in a business processmodel.“Wikipedia © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner • BPMN is a standard notation for designing business processes(versus: UML is a standard modeling language best suited for designing andimplementing software)• not just flow charts! sufficient restrictions / constraints => executable!• standardized XML format• orchestration and choreography• extension points => add specific needs without breaking interoperability• optional mapping of a BPMN subset to BPEL(restricted to block-structured flows without cycles)BPMN 2.0Kai Wähner © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Alternatives for BPM (technology perspective)Kai WähnerComplexity ofOrchestrationLow HighBPM Framework BPM Suite Integra7on Suite CodingService TasksHuman InteractionGUIBPMToolingMonitoringSupport+ESBBIG DATA / MDMREGISTRY / REPOSITORYRULES ENGINE„YOU NAME IT“+ © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Kai WähnerIntegration• ETL• Connectivity / adaptors to connect to externalsystems using a variety of different protocols • Predefined EIP for message routingBig Data• Processing• AnalyticsBPM• Do queries to make decisions• Human or machineLet‘s realize it !!! Technical / product perspective © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Alternatives for creating intelligent BPsKai WähnerComplexity ofOrchestrationLow HighFrameworks Specific Tools Suite of Tools e.g.Talend Unified Platformi.e ALL-IN-ONE(Integration, Big Data, BPM)e.g.Camel (Integration)Hadoop (Big Data)Activiti (BPM)e.g.Mule ESB (Integration)MapR (Big Data)Camunda (BPM) © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Custom combination of integration, big data and BPM?Kai Wähner• A lot of glue code• Testing• Bugfixing• No supportSome other people already hadthe problems you would have! © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner • Big data paradigm shiL • Use cases for big data • Intelligent business processes • Technology and product perspec7ve • Live demo Agenda © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Live demoComplexity ofOrchestrationLow HighFrameworks Specific Tools Suite of Tools e.g.Talend Unified Platformi.e ALL-IN-ONE(Integration, Big Data, BPM)e.g.Camel (Integration)Hadoop (Big Data)Activiti (BPM)e.g.Mule ESB (Integration)MapR (Big Data)Camunda (BPM) © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner DataQualityDataIntegration MDM ESBBigDataBig Data¾ Open source license ¾ Free of charge ¾ OpDonal support ¾ Based on open source projects such as Eclipse or Apache Camel, CXF, Hadoop ¾ Commercial license ¾ SubscripDon model ¾ Support included DataQualityDataIntegration MDM ESB BPMTalend Unified Platform © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner „Talend Integra7on Suite“ in ac7on… Live demo © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Example (Talend): Integration © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Example (Talend): Big Data Processing © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Example (Talend): Business Process © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Example (Talend): Big Data Web Service © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Did you get the key message? © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Key messagesBPM should be used (just) for optimizing business processes!Intelligent business processes need big data and integration!Big data will reduce human interactions in BPM further! © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Did you get the key message? Thank you for your attention.Questions?KAI WÄHNERkwaehner@talend.comwww.kai-waehner.deLinkedIn / Xing@KaiWaehner
Leave a Comment
You must be logged in to post a comment.
