Application¶

ForML applications represent a high-level concept built on top of ForML projects as deliverables for the serving layer.

The term application in this context does not hold the common meaning of a general computer program covering a wide range of potential functions. ForML applications obviously focus just on the ML inference (providing predictions in response to the presented data points) representing the apply stage of the production life cycle.

While the purpose of projects is to implement a portable solution to the given ML problem, applications aim to expose it (by means of gateway providers) in a domain-specific form suitable for integration with the actual decision-making process.

ForML platform persists published applications within a special application inventory where they are picked from at runtime by the serving engine.

Project-Application Relationship¶

As shown in the diagram below, relationships between projects and applications can have any possible cardinality. Projects might not be associated with any application (not exposed for serving - e.g. Project B), on the other hand, an application can possibly span multiple (compatible) projects (its model selection strategy can involve multiple projects - e.g. Application Y) or a single project might be utilized by several different applications (e.g. Project A).

flowchart LR
    subgraph registry ["Registry"]
        subgraph prja ["Project A"]
            gena1[("Generation 1")]
            gena2[("Generation 2")]
        end
        subgraph prjb ["Project B"]
            genb1[("Generation 1")]
        end
        subgraph prjc ["Project C"]
            genc1[("Generation 1")]
        end
    end
    subgraph inventory ["Inventory"]
        app1(["Application X"]) --- gena1 & gena2
        app2(["Application Y"]) --- gena2 & genc1
    end
    subgraph gw ["Gateway"]
        eng["Engine"] --- app1 & app2
    end

It makes sense to manage an application (descriptor) in the scope of some particular project if they form a 1:1 relationship (perhaps the most typical scenario). More complex applications might need to be maintained separately though.

Request Dispatching¶

Applications play a key role in the serving process taking control over the following steps:

sequenceDiagram
    Engine ->> Application: receive(Request)
    Application --) Engine: Entry, Scope
    Engine ->> Application: select(Scope, Stats)
    Application --) Engine: Model
    Engine ->> Model: predict(Entry)
    Model --) Engine: Outcome
    Engine ->> Application: respond(Outcome, Scope)
    Application --) Engine: Response

Data Interpretation¶

Applications define how to understand the received query and how to turn it into a model-prediction request, as well as how to present the predicted outcomes as the domain-specific response.

This is implemented within the following steps:

1. Formally receiving the query by:

   1. Decoding its content according to the implemented payload semantic. Applications might choose to support a number of different encodings.

   2. Optionally compile the query into prediction-relevant data points. This might possibly involve certain domain mapping (e.g. a recommender application receiving click-stream events would turn it at this point into a set of product features to be passed down for scoring, etc.).

   3. Optionally assemble custom metadata to constitute an application context to be carried through the serving layers for reference.

2. Producing a response based on:

   1. Composing the domain-specific result message out of the prediction outcomes (generated by the engine using the selected model). This might again involve particular domain mapping (e.g. turning probabilities into a selection of products, etc.).

   2. Encoding the response payload into a client-accepted representation.

Model Selection¶

Another powerful way an application exerts control over the serving process is a dynamic selection of the specific model generation to be used for serving each particular request.

Applications can base the selection logic on the following available facts:

• the actual content of the model registry (any existing model generation to choose from)

• custom metadata stored in the application context (e.g. as part of the query receiving)

• various serving metrics provided by the system (e.g. number of requests already served by this application - using which model - including actual performance tracking results of each of the models, etc.)

The model-selection mechanism allows implementations of complex serving strategies including A/B testing, multi-armed bandits, cold-start/fallback models, etc. It is due to this dynamic ability to select a particular model/generation on the fly that the project-application relationship can potentially have higher than just the ordinary 1:1 cardinality.

Implementation¶

Similarly to the principal project components, applications are delivered in form of a python module (single file with the .py suffix) providing an implementation of the application.Descriptor:

class forml.application.Descriptor[source]¶

Application descriptor abstract base class.

The serving layer is using Application descriptors to control the query processing.

Active descriptors are deployed through asset.Inventory used by the serving engine.

abstract property name : str¶

Name of the application represented by this descriptor.

Caution

Application name is expected to be globally unique. This name will be used to register the application when publishing and to target it when serving.

Returns:

Application name.

abstract receive(request: layout.Request) → layout.Request.Decoded[source]¶

Receive the raw payload and turn it into a structure suitable for predicting.

This involves at least payload decoding plus potentially also any further data compilation necessary for prediction. Additionally, it might also produce custom metadata representing an application context to be passed down the chain all the way to select() and respond().

Parameters:

request: layout.Request¶

Native request format.

Returns:

Decoded entry (adjusted for prediction) with optional custom (serializable!) context.

Raises:

layout.Encoding.Unsupported – If the received encoding is not supported.

abstract select(registry: asset.Directory, context: Any, stats: runtime.Stats) → asset.Instance[source]¶

Select the model instance to be used for serving the request.

This can implement an arbitrary model-selection strategy with the use of the provided information.

Parameters:

registry: asset.Directory¶

Model registry to select the model from.

context: Any¶

Optional metadata carried over from receive().

stats: runtime.Stats¶

Application specific serving metrics.

Returns:

Model instance.

abstract respond(outcome: layout.Outcome, encoding: Sequence[layout.Encoding], context: Any) → layout.Payload[source]¶

Turn the application result into a native response to be passed back to the requester.

This involves assembling the resulting structure and encoding it into a native format.

Parameters:

outcome: layout.Outcome¶

Result to be returned.

encoding: Sequence[layout.Encoding]¶

Accepted encoding media types.

context: Any¶

Optional metadata carried over from receive().

Returns:

Encoded native response payload.

Raises:

layout.Encoding.Unsupported – If none of the accepted encodings is supported.

The descriptor needs to be registered within the delivering module via a call to the application.setup() function:

forml.application.setup(descriptor: application.Descriptor) → None[source]¶

Interface for registering application descriptor instances.

This function is expected to be called exactly once from within the application descriptor file.

The true implementation of this function is only provided when imported within the application loader context (outside the context this is effectively no-op).

Parameters:

descriptor: application.Descriptor¶

Application descriptor instance.

Examples

titanic.py¶

 from forml import application

 APP = application.Generic('forml-example-titanic')
 application.setup(APP)

Generic Application¶

In addition to the abstract application.Descriptor, ForML for convenience also provides a generic out-of-the-box implementation suitable for most typical scenarios.

This implements the data interpretation simply using the available layout.get_decoder and layout.get_encoder codecs and for the model selection it introduces a concept of pluggable application.Selector strategies.

class forml.application.Generic(name: str, selector: application.Selector | None = None)[source]¶

Bases: Descriptor

Generic application descriptor for basic serving scenarios.

It simply runs the directly decoded (using the available decoders) request payload through the model/generation selected using the provided application.Selector and returns the directly encoded (using the available encoders) outcomes as the response.

Parameters:

name: str¶

The (unique) name for this application registration/lookup.

selector: application.Selector | None = None¶

Implementation of a particular model-selection strategy (defaults to application.Latest selector expecting the project name to be matching the application name).

Examples

>>> APP = application.Generic('forml-example-titanic')

Generic applications are configured with particular model selection strategies provided as implementations of the following application.Selector base class:

class forml.application.Selector[source]¶

Abstract base class for the model selection strategy to be used by the application.Generic descriptors.

abstract select(registry: asset.Directory, context: Any, stats: runtime.Stats) → asset.Instance[source]¶

Select the model instance to be used for serving the request.

See also

This serves the same purpose as the application.Descriptor.select method only extracted as a separate object.

Parameters:

registry: asset.Directory¶

Model registry to select the model from.

context: Any¶

Optional metadata carried over from the application.Descriptor.receive.

stats: runtime.Stats¶

Application specific serving metrics.

Returns:

Model instance.

Strategies¶

Following are available implementations of model selection strategies to be used when configuring any generic application.

class forml.application.Explicit(project: str | asset.Project.Key, release: str | asset.Release.Key, generation: str | int | asset.Generation.Key)[source]¶

Bases: Selector

Model selection strategy always choosing an explicit model generation.

Parameters:

project: str | asset.Project.Key¶

Project reference of the selected model.

release: str | asset.Release.Key¶

Project release reference of the selected model.

generation: str | int | asset.Generation.Key¶

Project generation reference of the selected model.

class forml.application.Latest(project: str | asset.Project.Key, release: str | asset.Release.Key | None = None, refresh: float = 30)[source]¶

Bases: Selector

Model selection strategy choosing an instance of the most recent model release/generation.

Parameters:

project: str | asset.Project.Key¶

Project reference to choose the most recent generation from.

release: str | asset.Release.Key | None = None¶

Optional release to choose the most recent generation from.

refresh: float = 30¶

Interval in seconds for refreshing the latest instance from its registry.

class forml.application.ABTest(avar: application.ABTest.Variant, bvar: application.ABTest.Variant, *others: application.ABTest.Variant)[source]¶

Bases: Selector

Model selection strategy choosing among a number of defined variants according to their target weights.

The target weights can be defined either as fractions in the float interval (0, 1) or as positive integers that will get normalized internally. If omitted for any variant, the particular target weight gets calculated either as a complement to 1 (if all other provided weights are floats below 1) or as a mean of the provided integer weights.

Attention

Instances are expected to be created via the Builder.

Examples

Setting up an ABTest starts with the compare() method:

>>> selector = application.ABTest.compare(
>>>     "forml-tutorial-titanic", "0.1.dev1", 1, 0.9
>>> ).against(generation=2, target=0.1)

classmethod compare(project: str | asset.Project.Key, release: str | asset.Release.Key, generation: str | int | asset.Generation.Key, target: float | None = None) → application.ABTest.Builder[source]¶

Bootstrap method for creating the ABTest builder.

Parameters:

project: str | asset.Project.Key¶

Project reference of the selected model.

release: str | asset.Release.Key¶

Project release reference of the selected model.

generation: str | int | asset.Generation.Key¶

Project generation reference of the selected model.

target: float | None = None¶

Relative engagement share demand.

Returns:

ABTest builder.

class forml.application.ABTest.Builder(project: str | asset.Project.Key, release: str | asset.Release.Key, generation: str | int | asset.Generation.Key, target: float | None)¶

Internal builder for setting up the application.ABTest.

Attention

Instances are expected to be created via the application.ABTest.compare method.

over(generation: str | int | asset.Generation.Key, *, release: str | asset.Release.Key | None = None, project: str | asset.Project.Key | None = None, target: float | None = None) → application.ABTest.Builder¶

Intermediate method for adding a non-last variant (ABTest of more than 2 variants).

Parameters:

project: str | asset.Project.Key | None = None¶

Project reference of the selected model.

release: str | asset.Release.Key | None = None¶

Project release reference of the selected model.

generation: str | int | asset.Generation.Key¶

Project generation reference of the selected model.

target: float | None = None¶

Relative engagement share demand.

Returns:

ABTest builder.

against(generation: str | int | asset.Generation.Key, *, release: str | asset.Release.Key | None = None, project: str | asset.Project.Key | None = None, target: float | None = None) → application.ABTest¶

ABTest builder completer for adding the last variant and constructing the ABTest instance.

Parameters:

project: str | asset.Project.Key | None = None¶

Project reference of the selected model.

release: str | asset.Release.Key | None = None¶

Project release reference of the selected model.

generation: str | int | asset.Generation.Key¶

Project generation reference of the selected model.

target: float | None = None¶

Relative engagement share demand.

Returns:

ABTest instance.

Publishing¶

Applications get deployed by publishing into an application inventory used by the particular serving engine. Unlike the project artifacts, applications are not versioned and are only held in a flat namespace depending on the uniqueness of each application name. (Re)publishing an application with an existing name overwrites the original instance.

$ forml application put titanic.py
$ forml application list
forml-tutorial-titanic