Blog Posts Case Management Process Management Process Modeling

What if the new Standard breaks the old tests?

Blog: Collaborative Planning & Social Business

Would you release a product to the public before you run the tests? Whether you are manufacturing, or software, or agriculture, or anything, if you have a set of tests, you would run the tests before providing the final product to the pubic. Makes sense, right? The world of technology standards is different.

DMN 1.2 Spec Released

The OMG approved the release of DMN 1.2 a couple weeks ago, and it continues to go through various rounds of formalization that will take a couple more months, but the content of the standard is promised to be entirely fixed at this point.

The DMN TCK has more than 750 tests used to verify the correct running of the DMN implementations. Currently these run on the 1.1 version of the spec. The next step will be for vendors to implement the changes for version 1.2 into their engines, and then we will run the existing tests on those modified modules.

What if a Test Fails?

We would call this an upward compatibility problem. It can happen anytime a new feature in the specification changes the way something works. The committee always tries to avoid this when possible, but a new behavior is a new behavior and if it happens to infringe on a prior behavior that was being used some someone, it causes an upward compatibility problem. Obvious cases can be avoided, but the ones that are not avoided can still be there by accident.

Such problems are hard to recognize in advance. This is why most serious software projects have a large list of test cases that are brought from version to version. (In one project here in Fujitsu, we have tests that were written more than 15 years that still run today unmodified in order to prove compatibility to the older versions.) Before a new version is release, all the old tests are run. If an incompatibility somewhere deep in the interactions of the various scenarios is found — then it can be addressed before shipping the product.

What if a DMN test fails now? What if an accidental incompatibility is found now, after the spec has been released? That will have to be decided once the problem turns up. Later. After the spec is in circulation.

OMG Standards are not Tested

The OMG process for creating a standard does not involve any demonstration of an implementation actually running the standard. No running code is required before the standard is “released.” No matter how sharp your designers are, you simply can not verify that everything works the way you think until you get code actually running.

That is one big reason we started the TCK is to assure that running implementation are actually tested and compared. Demonstrating that the standard works is a critical part in the development of any standard.

What happens if a compatibility problem is found? A decision will be made at that time. The incompatibility might be left int he standard, because sometimes to progress a break with the past must be done. Implementers then have a choice of which version to support, and possibly including a flag to specify which version of behavior you want.

The more insidious problems are the statements in the spec that are found to be unimplementable. Several of these exist in the 1.1 spec: for example the statement that claims that a reference to a list with one element in it is exactly the same as a reference to the only element of that list. Implementing that was simply impossible, but it is still there in the 1.1 version of the spec. It was changed in 1.2, but because the spec is released before any implementation, there are going to be things in the spec which are wrong. Individual vendors decide not to implement when it is impossible — and sometimes don’t implement things that are possible — and this makes the field of implementations lack the uniformity we all would like. The de jure standard does not match the de facto standard.

Summary

Because the scope of change in DMN 1.2 is small, and due to the large amount of scrutiny, I don’t expect to find any large surprises after implementation. There could be small ones, and they will be dealt with when discovered.

Still, it would be a benefit to the entire community if the OMG required at least one running implementation before releasing the spec. The OMG won’t change in this regard, so it is up to the rest of the community to test and various implementations, and to discover what the real technology turns out to be.