Using the Integration Test Suite

Along with the pFIRE library and executable, the codebase also includes unit tests and integration tests for validation, benchmarking and validation of pFIRE’s functionality. The use of these is documented in detail here.

Integration Testing

Integration testing involves running the complete program against various input data and comparing either to an accepted result, either in the form of predetermined result file or the output of another program (this could be an earlier version of pFIRE or a different image registration code).

Integration testing for pFIRE can be performed using the pfire_benchmarking python module, located in the benchmarking/ subdirectory of the source code repository.

Installation

Since it is provided as a standard python module the test suite may be installed on any system with python version 3.5 or later, either using pip or by manually running the provided setup.py:

$ cd benchmarking
# then
$ pip install --user .
#or
$ python3 setup.py

The test suite is dependent on several common python packages. If using pip these should be installed for you automatically. If installing manually it is up to you to install them.

Once installed, the testsuite may be run using the pfire-integration-test command.

Running Integration Tests

The integration test suite can be run either on a single test file or on all tests in a directory tree. There are two types of test that the suite supports, comparison with a result file, or by running both pFIRE and its predecessor (ShIRT) against the same test data and comparing the results.

To run against a single configuration file, pass that test file as a parameter to the testsuite

$ pfire-integration-test /path/to/testfile.testconf

or to run against all test files in a directory tree, pass the root directory of that tree

$ pfire-integration-test /path/to/root/of/testtree/

the testsuite will then be run against the supplied test(s), and an html report generated detailing the results. Note that for a test configuration file to be detected in the test tree it’s name should end with .testconf.

Included Tests

A set of comparison and regression tests are included with the pFIRE source code, in the testdata/integration subdirectory.

Integration Test Configuration

Integration tests are described and configured using a short configuration file, with ini syntax. This configuration file should have the extension .testconf in order to be detected by the testsuite when part of a tree of tests.

The complete syntax of the config file is:

name = # string: Name of the Test
type = # string: one of "comparison" or "regression"
pfire_config = # string: the pfire config file to use
accepted_image = # string: the image to compare with ("regression" mode only)
accepted_map = # string: the map to compare with ("regression" mode only)

The test suite will run pFIRE using the supplied configuration file, and compare the output to either the provided file(s) (type=regression) or to the results of running ShIRT with the same configuration (type=comparison). If the type is specified as comparison then the ShIRT binary must be located on the system $PATH.