O dear, what will we do with the data?
Blog: Process transformation - interventions for meaningful change
In my previous post, I wrote about the possible scenario’s to deal with integration with existing data-centric applications. In this blog-post, I want to cover situations, in which we want to digitize the process (with a BPMS/ACMS) and there is data, but no existing data-centric applications to store it.
Let’s first investigate the typical data one encounters when analyzing a process that needs to be digitized…
|Customer data||Names, addresses, contact persons, etc.|
|Product/transaction data||Type of service or product, certain service / product specifications, pricing, other conditions and agreements|
|Decisions||Specific decisions (what what the decision, who took it, when, and why was this decision taken/on what grounds)|
|Process and task performance data||Durations of process and tasks (or even effort and costs associated with it)|
|Audit trails||What was done, when, why and by who|
|Other process linked data||Comments/communication between process participants, certain checklists/QC outcomes|
– The customer wants their service or product first time right, on time delivered
– The process participants need it to know what to do, and do it correctly
– Process management wants to manage the flow, work-distribution, work-load and compliance
– Responsible managers want to be able to report outcomes, performance and compliance
– Accountants / Audit / QA-people want to understand what happened, and see if it was compliant to regulations
– Future generations (Public sector mainly) want to historically research what happened
When we digitize a process, this data is, in the as-is situation, usually scattered in various forms:
– In unstructured forms, on paper, in folders, or in e-mails
– In semi-structured form, in (often) Excel files
And the user requirements are typically:
– We don’t want Excel anymore, the new BPMS/ACMS solution should help us to digitize the process
– We want to store and report on all of this data somehow
Many times, I encounter organizations that are mostly paper-based, sometimes with low-volumes. So when implementing a BPMS/ACMS solution, we do not have the luxury of a CRM-system, and ERP-system (or other Product administration), a Datawarehouse. Sure, organizations should have these systems in place, but often, introducing these technologies are not in our scope. Then the inevitable question comes: O dear, what will we do with the data? Ouch.
From a solution architecture point of view, we have a number of paths.
1. Let parts of the data behind in existing office automation
– Separate files that need to be updated manually (more effort)
– If the data is also stored in other places: risk of becoming out of sync
– Not addressing client needs
– Hard to query
2. Store parts of the data in office automation and integrate with it
– Data is updated, no need for manually updates
– Less risk of becoming out of sync
– Low cost
– Usually quite risky, as business users can access the files and change structures
– Not suitable for large data volumes
– Technically unstable sometimes, and transaction integrity/locking issues
– Hard to query
3. Store parts of the data in structured form in the process instance
This scenario is tricky. My design rule is: keep the process instance very light, as:
– Performance might get bad, when too much data per instance
– Often hard to query on specific items (usually stored as XML/blobs)
With very light I mean:
– Customer/transactional data is only temporarily stored in the process instance, but as quick as possible stored in a other structure
– Data that is really process instance specific (a contact person name for this order)
– Only keys to the data (customerid, productid) are stored
– Only data is stored that is needed to execute business rules (preferably retrieved from other structure just in time).
4. Store parts of the data in digital documents (unstructured)
– Simple (you will need a Document Management solution, and often a ‘Case’ concept)
– No reporting on structured data possible (Text mining too difficult)
5. Store parts of the data in a custom database-structure (linked to a case)
– Clear structured place to store data
– Depending on complexity of data, can grow large and complex in terms of effort and architecture
– If the BPMS needs to support multiple processes, each with their own set of data, complexity can become even more
What I sometimes encounter is that scope explodes, because customers would like access to this data through a non-BPMS user interface: a new data-centric app is born and included in the project. Usually not a nice thing.
6. Store parts of the data in structured form in a generic Case data – solution
In this scenario, we create (of buy) a component in which one can define meta-data schemes per process-class or case type. In each scheme, one can define possible data-elements, . When creating a process instance or case, the BPMS/ACMS allows users to maintain the defined data-elements.
– Flexible, quick to configure and deploy
– Can be hard to query
7. Store parts of the data in a (small/custom) Datawarehouse
For some types of data, I would recommend to create a small custom set of Star-scheme tables with the relevant dimensions. This allows to query on performance of processes and tasks, by dimensions as process / case type, period, organization unit, etc. In addition, it would be possible the extent the star-scheme with more transactional data (order volume, price, certain decisions) to allow more complex queries (‘In period X what was the total of orders in Euro, in which we refused to deliver’)
Data governance can be a tricky question in BPMS/ACMS projects. It not properly handled it can require lots of effort, effort that we rather would focus on the process and BPMS-advantages. If a project is oversold in terms of BPMS/ACMS promises (and we all encounter this from time to time :-)), it can be hard to manage client’s expectations and influence the client to select and implement additional components, such as a CRM-system.
As BPMS-specialist I think we need to be honest to the clients on the risks and limitations of the possible scenario’s, and help them to make a sound decision.