Efesto refactoring – technical details

Blog: Drools & jBPM Blog

This post is meant as a description of the APIs and other technical details of the Efesto framework. It continues the introduction made in the previous one.

Base concepts.

There are some concepts around which the APIs are implemented:

Generated resource
Unique identifier
Context of execution

The framework provides and manage default implementations of the classes representing those concepts. Those classes could be extended by different engines for their specific needs (e.g. the Kie-drl compilation plugin define a context that contains a KnowledgeBuilderConfiguration) but this specific addition should never leak out of the engine itself, and the functionality of the framework itself should never rely on such "custom" details.

Generated resource

A GeneratedResource represent the result of a compilation. By itself is just a marker interface because there are different kind of generated resources:

executable resources (GeneratedExecutableResource)
redirect resources (GeneratedRedirectResource)
“container” resources (like GeneratedClassResource).

Executable resources represents the "entry point" for execution at runtime, and it contains information required to "instantiate" the executable unit. For some code-generation models (e.g. rules, predictions) this means store the class to instantiate at runtime, that will be used to start the evaluation. For models that does not rely on code-generation for execution (e.g. decisions), this resource contains the name of the class to be instantiated and/or the methods/parameters to be invoked.
Redirect resources contains information needed to forward the execution request to a different engine, and it contains the informatio about the ewngine to be invoked.
Container resources are meant to store other informations needed at runtime (e.g. the classes generated during compilation).

Unique identifier

The unique identifier (ModelLocalUriId) contains the information required to uniquely identify an executable or redirect generated resource. ModelLocalUriId contains information about:

the model/engine to invoke
the full path to the given resource

The unique identifier is represented a "path" whose root is the model/engine to invoke, and the path describe all the elements required to get to the specific resource. Stateless engines (e.g. DMN, PMML) describe that as "/namespace/model_name" or "/filename/model_name". Statefull engines would require further path compoenents to identify the specific "state" to be invoked (e.g. "/drl/rule_base/session_name/session_identifier").
ModelLocalUriId is a property of both GeneratedExecutableResource and GeneratedRedirectResource, since both of them have to be retrieved during runtime execution. ModelLocalUriId implements LocalId and is a feature that was initially implemented in the Kogito Incubation API, for which an explanation is available here.
For each module, client code should be able to invoke a method like that to retrieve the unique identifier:

        ModelLocalUriId modelLocalUriId = appRoot("")
                .get(PmmlIdFactory.class)
                .get(fileNameNoSuffix, modelName);

This is a fluent API, and each get invocation corresponds to an element in the generated path. The appRoot parameter is only used to differentiate multiple applications (e.g. in distributed context).
The first get is needed to start the path building. Each module should implement its own factory extending ComponentRoot, that, in turn, will be used to generate the full path.

Each of the following get should return an object that extends ModelLocalUriId, since each it represent the path until that specific segment. Each module may provide its own strategy to define such paths, so each module may implement its own subclasses, depending on the needs. Since the The ModelLocalUriId constructor requires a LocalUri instance, any of its subclasses should implement a way to call that constructor with such instance.

In the following example:

public class PmmlIdFactory implements ComponentRoot {

    public LocalComponentIdPmml get(String fileName, String name) {
        return new LocalComponentIdPmml(fileName, name);
    }

}

the PmmlIdFactory expose a get method ( the fluent API) that requires fileName and name parameters. This, in turns, are used to invoke the LocalComponentIdPmml constructor.

public class LocalComponentIdPmml extends ModelLocalUriId {
    public static final String PREFIX = "pmml";

    public LocalComponentIdPmml(String fileName, String name) {
        super(LocalUri.Root.append(PREFIX).append(fileName).append(name));
    }

}

This snippet:

LocalUri.Root.append(PREFIX).append(fileName).append(name)

will lead to the creation of the following path:
/{PREFIX}/{fileName}/{name}

Context of execution.

The EfestoContext contains basic information about the current execution. It contains informations about the generated classes and the unique identifiers generated during compilation.

EfestoRuntimeContext is the specialization used at runtime to retrieve the generated classes. EfestoRuntimeContextImpl is the default implementation.

Engines may extends the above as per their needs.
For example, DrlCompilationContext (the EfestoCompilationContext used inside the rule engine) defines KnowledgeBuilderConfiguration for its needs.

Compilation context

EfestoCompilationContext is the specialization used at compile time, and it is used to store the classes generated during compilation. EfestoCompilationContextImpl is the default implementation.

EfestoCompilationContext provide a static method to retrieve the default implementation (EfestoCompilationContextImpl) with all the classes eventually compiled from a previous compilation.

That static method, behind the scenes, invokes the constructor that scan the classloader to look for efesto-related classes.

Runtime context

EfestoRuntimeContext is the specialization used at runtime to retrieve the generated classes. EfestoRuntimeContextImpl is the default implementation.

EfestoRuntimeContext provide a static method to retrieve the default implementation (EfestoRuntimeContextImpl) with all the efesto-related compiled classes. That static method, behind the scenes, invokes the constructor that scan the classloader to look for efesto-related classes.

Public APIs

The framework consists basically of two set of APIs, the "compilation" and the "runtime" ones. Those APIs are defined inside CompilationManager and RuntimeManager. Those are the APIs that "client code" is expected to invoke. Said differently, "client code" is expected to interact with engines only through those APIs.

Compilation API

void processResource(EfestoCompilationContext context, EfestoResource... toProcess);

This is the method that "External applications" (e.g. kie-maven-plugin) should invoke to create executables units out of given models.

EfestoResource is the DTO wrapping a single model to be processed. Its only method
T getContent();

is invoked by the compilation manager to get the object to be processed.
The more common usage is to provide an actual File to the compilation manager, in which case there already is an implementation, EfestoFileResource.

EfestoSetResource is a specific abstract implementations that wraps a Set of models. As for the previous, there already exist an implementation to manage FIles, EfestoFileSetResource.

Runtime API

 Collection<EfestoOutput> evaluateInput(EfestoRuntimeContext context,
                                           EfestoInput... toEvaluate);

This is the method that "External applications" (e.g. kogito execution) should invoke to retrieve a result out of executable units generated at compile-time.

EfestoInput is the DTO wrapping a the data to be evaluated and the unique identifier of the executable units. It has two methods:

  ModelLocalUriId getModelLocalUriId();

  T getInputData();

the former returns the unique identifier of the executable units; the latter returns the data to use for evaluation.

Currently there are no "default" implementations of it, since the input structure is generally model-specific; so, every plugin should provide its own implementation.

Internal APIs

Behind the scenes, when CompilationManager and RuntimeManager receives a request, they scan the classloader for engine plugins. Such plugins should implement, respectively, the KieCompilerService and the KieRuntimeService.

CompilerService API

KieCompilerService declares three methods:

boolean canManageResource(EfestoResource toProcess);
List<E> processResource(EfestoResource toProcess, U context);
String getModelType();

The first one is invoked by the CompilationManager to verify if the specific implementation is able to manage the given resource. The evaluation could be based on the actual type of the resource, on some details of the content, or on a mix of them. It is responsibility of the implementation to find the appropriate logic. The only requirement to keep in mind is that, during execution, there should be at most one implementation that return true for a given EfestoResource, otherwise an exception is thrown.

The following snippet is an example where a given EfestoResource is considered valid if it is an DecisionTableFileSetResource:

@Override
public boolean canManageResource(EfestoResource toProcess) {
    return toProcess instanceof DecisionTableFileSetResource;
}

The above implementation works because DecisionTableFileSetResource is a class specifically defined by the plugin itself, so there are no possible "overlaps" with other implementations.

On the other side, the following snippet is an example where a given EfestoResource is considered valid if it is an EfestoFileResource and if the contained model is a PMML:

@Override
public boolean canManageResource(EfestoResource toProcess) {
    return toProcess instanceof EfestoFileResource && ((EfestoFileResource) toProcess).getModelType().equalsIgnoreCase(PMML_STRING);
}

In this case, the actual class of EfestoResource is not enough, since EfestoFileResource is one of the default implementations provided by the framework. So, a further check is needed, that is about the model that is wrapped in the resource.

A single plugin may manage multiple representations of the same model. For example, a plugin may manage both an EfestoFileResource and an EfestoInputStreamResource. There are different possible strategies to do that. For example, the plugin may provide one single "compilation-module" with two classes implementing the KieCompilerService; or it may define two "compilation-modules", each of which with one implementation, or one single class may manage both kind of inputs. Again, this is responsibility of the plugin itself.

This also push toward code reusage. For a given model, there could be a common path that provide the final compilation output, and different entry point depending on the model representation. It is so possible that multiple compilation models creates a compilation output that, in turns, it is also an EfestoResource. Then, there could be another implementation that accept as input the above intermediate resuorce, and transform it to the final compilation outpout. This chaining is managed by the efesto framework out of the box.

An example of that is featured by drools-related pmml models. During compilation, the PMML compiler generates an EfestoRedirectOutputPMMLDrl that is both an EfestoResource and an EfestoCompilationOutput. When the CompilationManager retrieves that compilation output, being it an EfestoResource, scans the plugins to find someone that is able to compile it.

The KieCompilerServiceDrl fullfill this requirement, and proceed with drl-specific compilation.

One thing to notice here is that different modules should limit as much as possible direct dependency between them.

The second method is invoked by the compilation manager if the previous one returned true. That method receives also an EfestoCompilationContext as parameter. Code-generating implementations should rely on that context for compilation and classloading.

The third method is used by the framework to discover, at execution time, which models can actually be managed. Thanks to that method, there is a complete de-coupling between the framework and the implementation themselves, since the framework can discover dynamically the available models, and every plugin may freely define its own model.

Last critical bit is that every compilation module should contain an org.kie.efesto.compilationmanager.api.service.KieCompilerService

file inside

src/main/resources/META-INF

directory, and that file should contain all the KieCompilationService implementations provided by that module.

RuntimeService API

KieRuntimeService declares three methods:

boolean canManageInput(EfestoInput toEvaluate, K context);

Optional<E> evaluateInput(T toEvaluate, K context);

String getModelType();

The first one is invoked by the RuntimeManager to verify if the specific implementation is able to manage the given input. The evaluation could be based on the actual type of the resource, on some details of the content, or on a mix of them. It is responsibility of the implementation to find the appropriate logic. The only requirement to keep in mind is that, during execution, there should be at most one implementation that return true for a given EfestoInput, otherwise an exception is thrown.

The following snippet is an example where a given EfestoInput is considered valid if it is an EfestoInputPMML and the given identifier has already been compiled:

 public static boolean canManage(EfestoInput toEvaluate, EfestoRuntimeContext runtimeContext) {
        return (toEvaluate instanceof EfestoInputPMML) && isPresentExecutableOrRedirect(toEvaluate.getModelLocalUriId(), runtimeContext);
    }

The above implementation works because EfestoInputPMML is a class specifically defined by the plugin itself, so there are no possible "overlaps" with other implementations. The difference with the compilation side is that the KieRuntimeService implementation should also check that the model related to the given unique identifier has already been compiled.

A single plugin may manage different types of input for the same model. For example, the rule plugin may manage both an EfestoInputDrlKieSessionLocal and an AbstractEfestoInput that contains an EfestoMapInputDTO. There are different possible strategies to do that. For example, the plugin may provide one single "runtime-module" with two classes implementing the KieRuntimeService; or it may define two "runtime-modules", each of which with one implementation, or one single class may manage both kind of inputs. Again, this is responsibility of the plugin itself.

This also push toward code reusage. For a given model, there could be a common code-path that provides the final runtime result, and different entry point depending on the input format. It is so possible that a runtime implementation would need a result from another implementation. In that case, the calling runtime will create a specifically-crafted EfestoInput and will ask the RuntimeManage the result for it. This chaining is managed by the efesto framework out of the box.

An example of that is featured by drools-related pmml models. During execution, the PMML runtime generates an EfestoInput<EfestoMapInputDTO> and send it to the RuntimeManager. The RuntimeManager scans the plugins to find someone that is able to execute it.

The KieRuntimeServiceDrlMapInput fullfill this requirement, and proceed with drl-specific execution.

One thing to note here is thet modules should limit as much as possible direct dependency between them!

The second method is invoked by the runtime manager if the previous one returned true. That method receives also an EfestoRuntimeContext as parameter. Code-generating implementations should rely on that context to retrieve/load classes generated during compilation.

Last critical bit is that every compilation module should contain an org.kie.efesto.runtimemanager.api.service.KieRuntimeService