Blog Posts Decision / Rules Management DMN

Our DMN 2.0 Wish List I: Decision Logic

Blog: Lux Magi - Decision Management for Finance

While James Taylor and I were collaborating on our Decision Modelling book, and discussing our experiences of using DMN with clients, the question “what additional features should be adopted in the next major release?” was a common subject of conversation. We found that our respective wish lists had a lot in common, reflecting our views on decision modeling best practice, so we elected to describe these features in our book and summarize some of them here. We’ve started with our wishes for the DMN logic level.

DMN (The Decision Model and Notation) is a way of representing Decision Models using diagrams and text, it does not address issues such as method or approach, therefore wishes must be constrained to notational items. Given this: do you agree with our items? What features would you like to see included in the decision logic level of the standard?

Decision Logic Level Wish List

The decision logic level of DMN addresses the way in which decision logic is defined using the standard using Decision Tables, Boxed Expressions and the expression language on which they are constructed: FEEL. Here are the extra features we would like to see.

Support Expressions in Decision Table Input Entries

Currently a Decision Table condition cell (formally a Input Entry in DMN) is a FEEL unary tests permitting conditions which test an input expression for:

equality (or inequality) with a…
- literal numeric, string, Boolean or date time values (e.g., ‘>100’);
- qualified name (variable, e.g., ‘<=Credit Limit’)
inclusion within a interval (e.g., ‘[0..100]’)
null a value indicating the value of the input variable is not known
‘–‘ which is a test that passes regardless of the input expressions value.
a (comma separated) list combining any of these (which is satisfied if any one of the members is) .

In practice this restriction is rather irksome and means that many decision tables have to resort to contexts or other FEEL-heavy means to achieve their ends. This is something we see a lot ‘in the field’, business users of DMN are perplexed that condition cells cannot perform tests using fully featured arithmetic expressions. It would be valuable to extend input entries to support simple expression or even a new type combining unary tests with arithmetic expression, simple value and function invocation to allow conditions like:

‘>=(Expiry Date + Grace Period)’ (simple expression)
‘>max(Applicant Credit Limit, Guarantor Credit Limit)’ (function invocation)

Even better would be to allow Decision Table Input Entries to support all FEEL functions directly, either by allowing expressions that use a reserved word to represent the condition value or by using simplified FEEL functions based on existing functions with implicit arguments. Take the case of a Decision Table with a condition (input expression) that is a string. Remember FEEL defines a function for testing if a string contains a substring (e.g., contains(“shire”,”ir”)). In this case, FEEL functions could be reused in an Input Entry by:

Using a reserved word to represent the Input Expression being tested (e.g., ‘contains(value,”ir”)’)
Using a simplified version of the function (e.g., ‘contains(”ir”)’) where the first argument is always the input value (condition)

This would allow Decision Tables like the one shown.

Decision Table with function input entry

This may strike some as over complicating Decision Tables and it would require discipline to use effectively. Undisciplined use of complex conditions runs the risk of making the Decision Table incomprehensible and hard to validate—but the alternative (FEEL contexts) is far more ‘heavyweight’ in our experience.

Decision Table Output Entries

By a similar argument to the above, the content of conclusion cells (output entries) which are already simple expressions, should be extended to support FEEL function calls. This would allow Decision Tables to represent a selection of outcome calculations which currently cannot be performed without resorting to a context.

Decision Tables Supporting Collection Input Expressions

Currently Input Entries (the contents of a condition cell in DMN Decision Tables) cannot directly handle collections in DMN. Therefore, although an Input Expression (or input variable) may be a collection, the only way to process a collection is to use FEEL’s ‘for .. in .. return’ construct to allow iteration through the collection, applying the Decision Table to each member, or use an input expression or context that aggregates the collection to a single value.

Simple conditions on the entire collection, for example checking an input collection for the existence therein of a specific value, requires a FEEL context. This should not be necessary. A range of condition tests should be developed for collection inputs, perhaps based on the simplified functions idea, above. This would support condition tests like:

Checking collection membership (‘list contains(USD)’)
Checking collection size (‘count()>5’)
Or checking aggregate properties (‘min()<4.5’).

If TDM can manage this, so can DMN.

Rule Level Annotations in Decision Tables

DMN Decision Requirement Diagrams benefit from text comments called annotations to clarify their meaning (e.g., to express assumptions, denote use of patterns, etc.). So, as some vendors have rightly suggested, why not have a rule level annotation as well? A rule level annotation is an additional conclusion in a Decision Table that acts like a comment and post-execute explanation (much like the Message construct of TDM) which:

Documents the intent, idiosyncrasies and rationale of each rule (as required)
Act as a means of supporting after-the-fact behavioural explanations of why specific rules were selected, including the ability to evaluate expressions to assist with diagnostics (see examples below)
(optionally) Links a rule with a specific section of a Knowledge Source that it satisfies

Here is our informal proposal for what this would look like:

Decision Table with rule level annotation referencing sections in Knowledge Source (click for closer look)

Note:

Annotations are confined to the right-most conclusion columns (or bottom rows in a ‘rules-as-columns’ table).
Annotations contain references to specific sections of named Knowledge Sources (traceability to requirements). In this example, IASIC is a nickname for a Knowledge Source and 5-19-4b is a section reference to the document referenced by this source.
Annotations are string expressions which can embed diagnostic information (see the annotation for rules 3 and 4)
Annotations are an effective means of making your Decision Tables easier to understand.

Completeness

Earlier versions of DMN supported the notional of completeness: a flag, denoted alongside the hit policy, to indicate that a Decision Table has a rule to cover every possible case—every combination of logical scenarios. Many of our clients found this very useful and were disappointed when the concept of completeness was discontinued in DMN 1.1 because of implementation concerns.

In our view, the concept of completeness should be reintroduced with a subtly different meaning. DMN should include a completeness marker to indicate the decision modeler’s intent that a Decision Table has complete logical coverage in practice. In other words, a Decision Table is complete if:

For every possible combination of input values that can occur in practice at least one rule is satisfied

The counter-argument has been made that this is not required because Decision Tables should always be explicitly complete. That is: each impossible combination of inputs should be supported by a rule with a null conclusion to deliberately document ‘impossible cases’. We have some sympathy with this argument but, in practice, there are many ‘real-world’ Decision Tables with enumerated inputs that have many nonsensical combinations. Listing them all would ‘bloat’ the table and obscure its business intent. Business analysts need a means of saying: “this set of rules covers all business valid combinations, all others either cannot happen or are not addressed here.”

Other Logical Representations

We would like to see the introduction of a DMN standard Decision Tree and a more business like interface to Analytic Models (business users find PMML is hard work).

Hit Policies

The hit policies ‘R’ (Rule Order) and ‘O’ (Output Order) are virtually useless and lead to some spectacularly obscure logic—please, get rid of them!

Missing FEEL Functions

FEEL incorporates some useful built-in functions for manipulating and aggregating collections of values. This is excellent as far as it goes, but many of the included operations lack associated functions: union() is supported but not intersection(); sum() is supported but not product(); there is flatten() but no zip() and mean() is supported but not median(), mode() or stddev().

Naturally there is an interface to add new functions as needed and the standard should not be encumbered with too many functions. However we would like to see a more inclusive set of built-in functions.

Glossary

Some have said DMN already defines all the metadata needed to interact with a glossary – anything else is tool specific. They assert that it’s not the job of a decision modeling standard to dictate the structure of a business glossary.

This is true, but we feel the DMN standard could go further in defining its interface to a glossary and what it requires of a glossary. As a result the importance a business glossary has been lost on many new users of DMN and DMN handles business terms in a naïve way, standing aloof from the data it manipulates.

DMN should not define what constitutes a glossary, but it should define what the glossary provides to a decision model, specifically:

Robust handling of enumerations

Enumerations should be a first class data type and not “strings with benefits”. Decision models should be able to manage externally defined enumerations and import them fully. In our experience of finance, more than two thirds of data types manipulated by decision models are enumerated.

Value lists should refer to named enumerated types in glossary

In practice a value list (a way of constraining the enumerated values of conditions and

Decision Table with enumeration type in value list

conclusions in Decision Tables) is not very useful when the Decision Table is dealing with enumerated types that: have tens of permitted values or more, have constantly changing permitted values and are used in tens of Decision Tables. The maintenance of the enumerations is very user-intensive and error prone—leading to inconsistencies. A much better scheme is to use a glossary to define named enumerated types and allow Decision Tables to refer to these named types in the value list. Consider the example shown.

This proposal would ease the maintenance of value list constraints considerably.

Robust handling of context names

Contexts in DMN can be used to rename information requirements to make a decision table more readable. As with enumerations, the same renaming may be used in multiple decision tables and this should be managed once not repeated each time. We propose a means of introducing contexts that can be easily shared between many tables.

Conclusion

We hope you found these interesting and we will post again soon with our wishes for additional features in the Decision Requirements level of the DMN. Meanwhile, we would be interested to hear your views: what additional features would you like to see in DMN.