Source Feed

Feed is a runtime platform component responsible for resolving the project defined ETL query providing the requested data.

forml.lib.feed.static

Static feed implementation.

Reader

forml.lib.reader.sqlite

SQLite3 reader implementation.

API

class forml.io.feed.Provider(*args, **kwargs)[source]

Feed is the implementation of a specific datasource provider.

Return type

Interface

class Reader(sources, columns, **kwargs)[source]

Abstract reader of the feed.

property columns

The explicit columns mapping implemented by this feed to be used by the query parser.

Returns

Columns mapping.

classmethod format(data)[source]

Optional post-formatting to be applied upon obtaining the columnar data from the raw reader.

Parameters

data (Sequence[Any]) – Input Columnar data to be formatted.

Returns

Formatted data.

Return type

Any

load(source, lower=None, upper=None)[source]

Provide a pipeline composable segment implementing the etl actions.

Parameters
  • source (component.Source) – Independent datasource descriptor.

  • lower (Optional[kindmod.Native]) – Optional ordinal lower bound.

  • upper (Optional[kindmod.Native]) – Optional ordinal upper bound.

Returns

Pipeline segment.

Return type

forml.flow.pipeline.Segment

classmethod reader(sources, columns, **kwargs)[source]

Return the reader instance of this feed (any callable, presumably extract.Reader).

Parameters
  • sources (Mapping[frame.Source, Source]) – Source mappings to be used by the reader.

  • columns (Mapping[series.Column, Column]) – Column mappings to be used by the reader.

  • kwargs (Any) – Optional reader keyword arguments.

Returns

Reader instance.

Return type

Callable[[frame.Query], Sequence[Any]]

classmethod slicer(schema, columns)[source]

Return the slicer instance of this feed, that is able to split the loaded dataset column-wise.

This default slicer is plain positional sequence slicer.

Parameters
  • schema (Sequence[series.Column]) – List of expected columns to be sliced from.

  • columns (Mapping[series.Column, Column]) – Column mappings to be used by the selector.

Returns

Slicer instance.

Return type

Callable[[Sequence[Any], Union[slice, int]], Sequence[Any]]

abstract property sources

The explicit sources mapping implemented by this feed to be used by the query parser.

Returns

Sources mapping.