Case Study: Auditing With Process Mining — Part XI: Conclusion
This is the 11th and last article in our case study series on auditing with process mining. The series is written by Jasmine Handler and Andreas Preslmayr from the City of Vienna. You can find an overview of all the articles in the series here.
Within this case study, we followed the 9-step model from Figure 3 to apply process mining in our audit. Throughout our journey, we experienced that the nine steps were not in a strict sequence. We frequently could use the things we learned in later phases of the project to improve deliverables from earlier steps. For example, we reworked our analysis questions multiple times as we gained new information regarding data availability and quality in later phases of the project.
Thus, the model has to be seen as an iterative approach where a rolling review contains validation steps on multiple levels. The image at the top shows the most important validation steps (click here to see a larger version).
First, we evaluated whether the data sets used as the input for the process mining analysis matched the raw data and the data from the productive system. Within this validation step, we ensured that we based our data analysis on a reliable data source.
Then, we validated the process itself by checking if the discovered model represents the process model defined in an earlier phase of the project. Within this step, we examined if we used the correct data or if there was any need to adapt the data model.
Next, we checked whether the analysis answers covered all the analysis questions. This way, we could check for any analysis question we had not answered yet, whether that was due to a lack of data or simply forgetting it.
Finally, we validated the results of the data analysis by checking if they met the requirements of the data analysis concept and if we had considered the primary audit objective sufficiently. With this evaluation, we could determine if the final analysis covered what we planned to audit.
All these validation steps helped us to get reliable results on a certain level of assurance and quality and improve the deliverables made throughout the process mining project.
Challenges and Limitations
Figure 17: Challenges and limitations
We encountered a number of challenges during our process mining project (see Figure 17).
Data preparation was one of the main challenges. As previously discussed, only for some process steps we wanted to consider was data available. Thus, we could not answer all the analysis questions we had in mind in the first place.
The available data was spread over different tables, which had to be linked to each other. 1:n and n:m relationships made data preparation more complicated, and we had to implement multiple versions of the data transformation workflow before it provided reliable and validated data we were confident in.
The lack of direct access to the productive systems made data validation even more time-consuming because there was a dependency on the audited party to provide the data for cross-checking. Furthermore, as we performed the audit during the covid-19-pandemic, only limited on-site visits were possible due to contact restrictions.
The high complexity of the analyzed process and the vast number of events in the data sets made the explorative analysis quite challenging. As a result, analysis questions No. 1 (“Do the real processes fit the should-process?“) and No. 12 („Are there any bottlenecks within the process?”) were hard to analyze. We had to simplify the data to reduce complexity to a level that made the process analyzable. Ultimately, it was impossible to state a definite percentage of how many cases did or did not fit the should-be process.
The 1:n and n:m relationships made it necessary to work with different data sets for different analysis questions. As a result, we made our process mining analyses from the order perspective as well as from the invoice perspective.
Despite the challenges and limitations listed above, the process mining analysis gave us an excellent insight into how the purchase-to-pay process was performed in reality. Thus, the benefits of using process mining in the audit (see Figure 18) did exceed the challenges we needed to overcome.
Benefits of using process mining in an audit
Figure 18: Benefits of using process mining in our audit
Within our audit work, we are often confronted with massive amounts of data and a high number of cases. As our resources are limited and we need to finish our audits within a specific timeframe, traditionally, we choose a sample of cases from the relevant process data and look at those cases in more detail.
Using process mining, we do not need to pick a sample anymore. Instead, we can analyze all cases regardless of the total number. For example, in the audit of this study, we could perform a complete examination of all 2,550 cases and give statements about order and invoice releases for all purchase orders from the year 2019.
We still used sampling techniques for a targeted investigation of those cases where we detected irregularities. This way, we allocate our resources more efficiently. From our experience, this led to a higher quality of the audit results and a higher assurance of detecting potential weaknesses in the internal control systems.
Furthermore, working with enormous amounts of data, it is often also hard to present the analysis results in a way everyone can follow. The graphical interface of the process mining software is very beneficial regarding this aspect. We could easily perform the analysis steps in attendance of the audited party to make transparent what we had done to come to the specific result. Of course, this benefit is even bigger when the audited party also has experience with process mining. In this case, they can retrace the analysis and measure if the changes they made due to the auditors’ recommendations have the expected impact.
Finally, using process mining to evaluate the effect of changes made to the process can also be very beneficial from the auditor’s perspective. If a follow-up audit is performed, process mining can be used once again to fully examine all cases and verify whether the implemented changes have improved the quality of the process.
As shown in this article, using process mining in an audit can be very beneficial and allows a deeper insight into the process of interest. Getting started with process mining in audit work is undoubtedly challenging, but it gets easier with more experience. We started using process mining in our audits in 2016 and have worked on improving our practice ever since. Every process mining project has been a new chance to improve our approach and make the audit trail more transparent.
With this article, we want to encourage other auditors to learn more about process mining and incorporate it into their audit method. We value exchanges with our peers and invite you to contact us to discuss your experiences with us via the contact details below.
Jasmine Handler, MA MSc – email@example.com (corresponding author)
Ing. Dipl.Ing.(FH) Andreas Preslmayr, MSc – firstname.lastname@example.org
You can also download the whole article as a PDF here for easier printing or sharing with others.
Leave a Comment
You must be logged in to post a comment.