Blog Posts

Process Mining for Usability Tests

You might have noticed: Products—and especially consumer electronics—are becoming more and more complex. As a result, people are not always able to deal with these complexities and usability becomes a distinguishing factor in brand reputation and customer satisfaction.

Process mining is a new technology that makes invisible process flows visible by analyzing existing log data in a bottom-up manner. Earlier, we have seen how process mining can be applied to the test process of ASML and an HR process. But can process mining also help to improve the usability of products?

Usability Testing

In the context of the Master’s project of Pieter Hofstra, together with Jeroen Keijzers, Yuan Lu, and Ton Weijters, we investigated the applicability of process mining for usability tests1.

Usability tests, for example first-use consumer tests, can help to get early feedback from the field while there is still time to adapt the product before releasing it to the market. Traditional usability measures include mostly static information, for example:

The results typically do not reflect the temporal aspects of the test data. So, in this project, we looked at how process mining can be used to get insights into the actual user behavior.

Experimental Setup

For the project, a group of 29 Dutch volunteers (from age of 22 to 66) participated in a usability test for a new television. 19 participants were male and 10 were female. The usability test took place in a simulated living room to make them feel at home as much as possible.

The participants were asked to complete the following three tasks:

  1. Channel selection. After installation of the television, channel RTL 7 has been automatically programmed on channel 25. The participants were asked to put RTL 7 on channel 7.

  2. Dual screen. The Dual screen function is innovative in comparison with previous versions of the product. It is one of the features promoted by marketing to sell the product. The participants were asked to watch the channels NEDERLAND 2 and NET 5 simultaneously.

  3. Digital picture. Another function that is new in comparison with previous versions of the television is the Digital picture function, which allows to view digital pictures from a USB stick on the television screen.

The “correct procedures” to solve each of these three usability tasks is shown in the picture below. Further down, you can see the process models of the actual user behavior for the middle task (Dual Screen).

(Process models of the optimal user behavior for solving each of the three television usability tasks.)

The entire experiment took about 15-40 minutes per person, depending on the participants’ performance. During the experiment, the participants and the television screen were captured on a video camera. From these video recordings, an event log of the actual usage behavior was created semi-automatically. You can find more details about the event log creation in Pieter’s Master thesis.


One of the goals of the study was to assess the effect of a consumer’s (product) knowledge on usability. People with high product knowledge are assumed to be more familiar with the product, and to have more experience in using it.

The test participants were divided into ‘High Knowledge’ (13 people) and ‘Low Knowledge’ (16 people) groups based on their knowledge ratings in the questionnaire2. Process mining was done on the usage logs of these two groups separately.

(Process model discovered for the ‘High Knowledge’ group performing the Dual Screen task. The numbers and coloring indicate frequencies.)

Look at the behavior of the ‘High Knowledge’ group performing the Dual Screen task above. One can nicely see the paths that were taken from the start to the end of the task.

(Process model discovered for the ‘Low Knowledge’ group performing the Dual Screen task. Compared to the ‘High Knowledge’ group this model is more complex showing more variability among the group of users.)

In the model above it is very visible that the people in the ‘Low Knowledge’ were even further away from the optimal solution. One person even had to give up.

I find this is a nice example that through the visualization of actual user behavior it is possible to reveal usage patterns, which provide both qualitative and quantitative feedback.

Not only for Consumers

Also in a business context usage behavior can be crucial. For example, in call centers there is an increasing use of analytics for operational performance management. Agents often have to switch between 4-6 different applications (Siebel, SAP, etc.) while handling a call. Desktop analysis tools can analyze the key strokes of an agent and the resulting insight can be used to build an abstracting layer on top of the actual applications that matches the typical call flow.

Do you see other examples where understanding user behavior is important? Let us know in the comments.

  1. See details in P.P.H.J. Hofstra. Analysing the Effect of Consumer Knowledge on Product Usability Using Process Mining Techniques. Masters thesis, Eindhoven University of Technology, Department of Industrial Design, Eindhoven, The Netherlands, 2009. ↩︎

  2. The knowledge level of each participant was measured by asking them questions assessing their “familiarity” and “expertise” with televisions and computers. For example, participants had to react to statements such as “Compared to most other people, I know less about televisions/computers” (familiarity) and “I usually talk with friends and colleagues about new developments regarding televisions/computers” (expertise) on a 5-point Likert scale (ranging from total disagreement to total agreement). For more details see Pieter’s thesis. ↩︎

Leave a Comment

Get the BPI Web Feed

Using the HTML code below, you can display this Business Process Incubator page content with the current filter and sorting inside your web site for FREE.

Copy/Paste this code in your website html code:

<iframe src="" frameborder="0" scrolling="auto" width="100%" height="700">

Customizing your BPI Web Feed

You can click on the Get the BPI Web Feed link on any of our page to create the best possible feed for your site. Here are a few tips to customize your BPI Web Feed.

Customizing the Content Filter
On any page, you can add filter criteria using the MORE FILTERS interface:

Customizing the Content Filter

Customizing the Content Sorting
Clicking on the sorting options will also change the way your BPI Web Feed will be ordered on your site:

Get the BPI Web Feed

Some integration examples