ForML provides a custom testing framework for user-defined operators. It
is built on top of the standard unittest library with an API
specialized to cover all the standard operator outcomes while minimizing any boiler-plating.
Internally it uses the
Virtual launcher to carry out the
particular test scenarios as a genuine ForML workflow wrapping the tested
The tests need to be placed under the
tests/ folder of your project (note
unittest requires all test files and the
itself to be python modules hence it needs to contain the appropriate
The testing framework is available after importing the
from forml import testing
See the tutorials for a real case unit test implementations.
Operator Test Case Outcome Assertions¶
The testing framework allows to assert following possible outcomes of the particular operator under the test:
is a scenario, where the operator raises an exception right upon initialization. This can be used to assert an expected (hyper)parameter validation.
mytest1 = testing.Case(arg1='foo').raises(ValueError, 'invalid value of arg1')
asserts an exception to be raised when executing the apply-mode of an operator without any previous train execution.
mytest2 = testing.Case(arg1='bar').apply('foo').raises(RuntimeError, 'Not trained')
is an assertion of an output value of the successful outcome of the apply-mode executed again without the previous train-mode.
mytest3 = testing.Case(arg1='bar').apply('baz').returns('foo')
checks the train-mode of given operator fails with the expected exception.
mytest4 = testing.Case(arg1='bar').train('baz').raises(ValueError, 'wrong baz')
compares the output value of the successfully completed train-mode with the expected value.
mytest5 = testing.Case(arg1='bar').train('foo').returns('baz')
asserts an exception to be raised from the apply-mode when executed after the previous successful train-mode.
mytest6 = testing.Case(arg1='bar').train('foo').apply('baz').raises(ValueError, 'wrong baz')
is a scenario, where the apply-mode executed after the previous successful train-mode returns the expected value.
mytest7 = testing.Case(arg1='bar').train('foo').apply('bar').returns('baz')
Operator Test Suite¶
All test case assertions of the same operator are defined within the operator test suite that’s created simply as follows:
class TestMyTransformer(testing.operator(mymodule.MyTransformer)): """MyTransformer unit tests.""" # Test scenarios invalid_params = testing.Case('foo').raises(TypeError, 'takes 1 positional argument but 2 were given') not_trained = testing.Case().apply('bar').raises(ValueError, "Must be trained ahead") valid_transformation = testing.Case().train('foo').apply('bar').returns('baz')
You simply create the suite by inheriting your
Test... class from the
utility wrapping your operator under the test. You then put your operator scenarios (test case
outcome assertions) right into the body of your test suite class.
Running Your Tests¶
All the suites are transparently expanded into a full-blown
definition so from here you would treat them as normal unit tests, which means you can simply run
them using the usual:
$ forml project test running test TestNaNImputer Test of Invalid Params ... ok TestNaNImputer Test of Not Trained ... ok TestNaNImputer Test of Valid Imputation ... ok TestTitleParser Test of Invalid Params ... ok TestTitleParser Test of Invalid Source ... ok TestTitleParser Test of Valid Parsing ... ok ---------------------------------------------------------------------- Ran 6 tests in 0.591s OK
Custom Value Matchers¶
.returns() assertions are implemented using the
unittest.TestCase.assertEqual() which compares the expected and actual values
object.__eq__() equality. If this is not a valid comparison for the
particular data types used by the operator, you have to supply a custom matcher as a second
parameter to the assertion. The matcher needs to be a callable with the following signature of
typing.Callable[[typing.Any, typing.Any], bool], where the first argument is expected and
the second is the actual value.
This can be useful for example for
pandas.DataFrame, which does not support simple
boolean equality check. The following example uses a custom matcher for comparing the memory
consumption of the tested datasets:
def size_equals(expected: object, actual: object) -> bool: """Custom object comparison logic based on their size.""" return sys.getsizeof(actual) == sys.getsizeof(expected) class TestFooBar(testing.operator(FooBar)): """Unit testing the FooBar operator.""" # Dataset fixtures INPUT = ... EXPECTED = ... # Test scenarios valid_parsing = testing.Case().apply(INPUT).returns(EXPECTED, size_equals)
For convenience, there is a number of explicit matchers provided as part of the