Blog Posts Case Management Process Management Process Modeling

DMN TCK Turns Two

Blog: Collaborative Planning & Social Business

Attending bpmNEXT conference this week. It was here, two years ago, that we decided to start the DMN TCK effort. How far have we come?

The original idea was to simply to back up the written spec with some real running code. The specification can’t possibly express the full detail necessary for an actual implementation. Anyone faced with implementing DMN would have to make thousands of small design decisions between, say, behavior X or behavior Y. Most of those are arbitrary in the sense that the a user could use behavior of either of X or Y, but what matters is that the product is consistent. Moving from product to product requires that all implementations make the same choices at those levels. The specification is, by its nature, ambiguous on these many small points.

After the first couple of months we abandoned the idea of creating a reference implementation, and pivoted successfully to the idea of providing a framework for publishing test cases which any successful implementation could use to verify and validate their execution. The tests would be files constructed in XML: the DMN model in the standard format, and the test data in another XML which we designed containing input values to submit to the decision model, and output values to compare the output to. To use the tests one needs to write a small “runner” which is code that reads the files, and runs the test.

All of the DMN examples and all the test files are available on GitHub and usable for free with a Creative Commons license.

A Example Test

Here is the XML representation of a DMN decision

This decision simply calculates the yearly amount by multiplying the monthly amount by 12. This tests the ability for an engine to parse that FEEL expression, and correctly evaluate it. And here is the XML test file:

As you can see, it puts 10,000 as the monthly amount, and 120,000 as the yearly result to compare to. Of course the DMN models can be considerably more complex, but the test files generally remains simple because it always is simple a list of inputs and outputs. It really is that easy to create tests, and because it is easy we hope for widespread adoption.

The results are then collected and listed in this Results Site.

Collecting Tests

Once the basic pieces were in place, we started collecting tests. The first few tests were models that Bruce Silver had developed while writing his book DMN Method and Style. The next chunk of tests came from Red Hat during the testing and finalization of their FEEL evaluator. Camunda added a number of tests. Finally, in July of 2017 Actico entered the group, and converted hundreds of their internal tests into the standard form that could be used by everyone bringing the total number of tests to 588.

We welcome more tests. Now that the framework is complete and functional, it is easy to add a DMN model, and a set of input / output values, and that constitutes a new test that anyone can run. Anyone can contribute who can make a GitHub style pull request. The team members will review the test, and assure that it runs on at least one existing engine, and then include it in the official set.

Improving the DMN Specification

While it is not the goal for TCK members to discuss modifications to the DMN spec itself, it is necessary to discuss how to implement parts of the DMN spec, and on many occasions we have run into parts that had been interpreted differently by different vendors, or even parts which were impossible to implement. In many cases the disagreement could be resolved by a careful re-reading of the spec. In some cases the spec was ambiguous, and so a proposed clarification was submitted to the RTF group maintaining the spec.

Spec did not allow a time duration to be divided by another time duration
Spec did not allow a date and time type to be parsed from a string
Spec considered that a list with a single element was in all cases exactly the same as that element alone. When we got into testing, we found that causes any number of paradoxes in the parsing of formulas.

For each of these we raised an issue with the RTF to get the spec changed or clarified.

Support

The current effort has received a lot of support from these organizations:

There are seven vendors that have run the tests and contributed results to the results site. However, the DM Community lists 18 vendors who claim DMN support. Why are these lists different? Why have so many vendors not run the tests? What assurance do you, as a consumer, have that those vendors actually run DMN? These are questions to ask those vendors directly.

What is Left to Do?

We are waiting for DMN 1.2 to be released so that we can then add some tests that are not compatible with 1.1, and we expect they will be supported by the new release. When that happens we will create a number of tests around 1.2.

We have not really explored error cases beyond one or two. Clearly there are many more error cases to check for. We might also test DMN models with invalid FEEL expressions to make sure that engines react consistently to those.

The main focus, however, will be adding real-world DMN models which have been developed for actual operating business applications. Those models test not only the execution of the engine in cases that are realistic, but also the suitability of DMN in general to real world problems.

Resources

A video of my talk at bpmNEXT 2018 (when available)
A Summary by Sandy Kemsley