Blog Posts Business Management Process Management

O dear, what will we do with the data?

Blog: Process transformation - interventions for meaningful change

In my previous post, I wrote about the possible scenario’s to deal with integration with existing data-centric applications. In this blog-post, I want to cover situations, in which we want to digitize the process (with a BPMS/ACMS) and there is data, but no existing data-centric applications to store it.

Let’s first investigate the typical data one encounters when analyzing a process that needs to be digitized…

Type Description
Customer data Names, addresses, contact persons, etc.
Product/transaction data Type of service or product, certain service / product specifications, pricing, other conditions and agreements
Decisions Specific decisions (what what the decision, who took it, when, and why was this decision taken/on what grounds)
Process and task performance data Durations of process and tasks (or even effort and costs associated with it)
Audit trails What was done, when, why and by who
Other process linked data Comments/communication between process participants, certain checklists/QC outcomes

All this data is crucial for the stakeholders around a process:
– The customer wants their service or product first time right, on time delivered
– The process participants need it to know what to do, and do it correctly
– Process management wants to manage the flow, work-distribution, work-load and compliance
– Responsible managers want to be able to report outcomes, performance and compliance
– Accountants / Audit / QA-people want to understand what happened, and see if it was compliant to regulations
– Future generations (Public sector mainly) want to historically research what happened

When we digitize a process, this data is, in the as-is situation, usually scattered in various forms:
– In unstructured forms, on paper, in folders, or in e-mails
– In semi-structured form, in (often) Excel files

And the user requirements are typically:
– We don’t want Excel anymore, the new BPMS/ACMS solution should help us to digitize the process
– We want to store and report on all of this data somehow

Many times, I encounter organizations that are mostly paper-based, sometimes with low-volumes. So when implementing a BPMS/ACMS solution, we do not have the luxury of a CRM-system, and ERP-system (or other Product administration), a Datawarehouse. Sure, organizations should have these systems in place, but often, introducing these technologies are not in our scope. Then the inevitable question comes: O dear, what will we do with the data? Ouch.

From a solution architecture point of view, we have a number of paths.

1. Let parts of the data behind in existing office automation
– Simple

– Separate files that need to be updated manually (more effort)
– If the data is also stored in other places: risk of becoming out of sync
– Not addressing client needs
– Hard to query

2. Store parts of the data in office automation and integrate with it 
– Data is updated, no need for manually updates
– Less risk of becoming out of sync
– Low cost

– Usually quite risky, as business users can access the files and change structures
– Not suitable for large data volumes
– Technically unstable sometimes, and transaction integrity/locking issues
– Hard to query

3. Store parts of the data in structured form in the process instance
This scenario is tricky. My design rule is: keep the process instance very light, as:
– Performance might get bad, when too much data per instance
– Often hard to query on specific items (usually stored as XML/blobs)

– You never know when the data gets cleaned out (you need strict rules on instance archiving)

With very light I mean:
– Customer/transactional data is only temporarily stored in the process instance, but as quick as possible stored in a other structure
– Data that is really process instance specific (a contact person name for this order)
– Only keys to the data (customerid, productid) are stored
– Only data is stored that is needed to execute business rules (preferably retrieved from other structure just in time).

4. Store parts of the data in digital documents (unstructured)
– Simple (you will need a Document Management solution, and often a ‘Case’ concept)

– No reporting on structured data possible (Text mining too difficult)

5. Store parts of the data in a custom database-structure (linked to a case)
– Clear structured place to store data

– Depending on complexity of data, can grow large and complex in terms of effort and architecture
– If the BPMS needs to support multiple processes, each with their own set of data, complexity can become even more

What I sometimes encounter is that scope explodes, because customers would like access to this data through a non-BPMS user interface: a new data-centric app is born and included in the project. Usually not a nice thing.

6. Store parts of the data in structured form in a generic Case data – solution
In this scenario, we create (of buy) a component in which one can define meta-data schemes per process-class or case type. In each scheme, one can define possible data-elements, . When creating a process instance or case, the BPMS/ACMS allows users to maintain the defined data-elements. 
– Flexible, quick to configure and deploy

– Can be hard to query

7. Store parts of the data in a (small/custom) Datawarehouse
For some types of data, I would recommend to create a small custom set of Star-scheme tables with the relevant dimensions. This allows to query on performance of processes and tasks, by dimensions as process / case type, period, organization unit, etc.  In addition, it would be possible the extent the star-scheme with more transactional data (order volume, price, certain decisions) to allow more complex queries (‘In period X  what was the total of orders in Euro, in which we refused to deliver’)

Data governance can be a tricky question in BPMS/ACMS projects. It not properly handled it can require lots of effort, effort that we rather would focus on the process and BPMS-advantages. If a project is oversold in terms of BPMS/ACMS promises (and we all encounter this from time to time :-)), it can be hard to manage client’s expectations and influence the client to select and implement additional components, such as a CRM-system.
As BPMS-specialist I think we need to be honest to the clients on the risks and limitations of the possible scenario’s, and help them to make a sound decision.

Leave a Comment

Get the BPI Web Feed

Using the HTML code below, you can display this Business Process Incubator page content with the current filter and sorting inside your web site for FREE.

Copy/Paste this code in your website html code:

<iframe src="" frameborder="0" scrolling="auto" width="100%" height="700">

Customizing your BPI Web Feed

You can click on the Get the BPI Web Feed link on any of our page to create the best possible feed for your site. Here are a few tips to customize your BPI Web Feed.

Customizing the Content Filter
On any page, you can add filter criteria using the MORE FILTERS interface:

Customizing the Content Filter

Customizing the Content Sorting
Clicking on the sorting options will also change the way your BPI Web Feed will be ordered on your site:

Get the BPI Web Feed

Some integration examples