Collaboration diagram for Testing Utilities (testutils):

Description

Common helper classes and functions for writing tests using Google Test.

General documentation for the testing in GROMACS can be found in the developer guide. This page provides an overview of the actual API provided by the testutils module.

There are several distinct functionalities provided:

gmx::test::TestFileManager (in testfilemanager.h) provides functionality for locating test input files from the source directory and managing temporary files that need to be created during the test.
gmx::test::TestFileInputRedirector (in testfileredirector.h) provides functionality for capturing file existence checks in code that uses gmx::FileInputRedirectorInterface.
GMX_TEST_OPTIONS macro provides facilities for adding custom command line options for the test binary.
testasserts.h provides several custom test assertions for better exception and floating-point handling than built-in Google Test assertions.
gmx::test::TestReferenceData and related classes (in refdata.h) provide utilities to write regression-style tests that check that the test produces the same results as an earlier run of the same test. The reference data is stored as XML files. For certain types of tests, the XML can also be easier to inspect manually for correctness than writing the checks in C++, providing an alternative method to write assertions.
gmx::test::CommandLine and related classes (in cmdlinetest.h) provide utilities for constructing command line argument arrays for use in tests that invoke actual commands.
gmx::test::StringTestBase provides a test fixture for tests that need to test long strings for correctness.
gmx::test::IntegrationTestFixture provides a test fixture for tests that execute legacy code where stdin reading etc. cannot be easily mocked.

Additionally, testinit.h and mpi-printer.h, and their corresponding source files, provide functionality that is not visible on the API level: they provide initialization routines for the above functionality, which are automatically called by the main() function provided in unittest_main.cpp.

mpi-printer.h provides a Google Test listener that is installed when the tests are compiled with MPI. This listener allows the test binary to be run on multiple MPI ranks, and synchronizes the execution and output from the test cases, as well as makes the test fail on even if an assertion fails only on one rank.

Assertions for exceptions
These macros replace `(ASSERT\|EXPECT)(_NO)?_THROW` from Google Test. They are used exactly like the Google Test ones, but also print details of any unexpected exceptions using GROMACS-specific routines. This makes it much easier to see at one glance what went wrong. See Google Test documentation for details on how to use the macros.
#define	EXPECT_THROW_GMX(statement, expected_exception)
	Asserts that a statement throws a given exception.

#define	EXPECT_NO_THROW_GMX(statement)
	Asserts that a statement does not throw.

#define	ASSERT_THROW_GMX(statement, expected_exception)
	Asserts that a statement throws a given exception.

#define	ASSERT_NO_THROW_GMX(statement)
	Asserts that a statement does not throw.

Assertions for floating-point comparison
These routines extend `(EXPECT\|ASSERT)_(FLOAT\|DOUBLE)_EQ` and `(EXPECT\|ASSERT)_NEAR` from Google Test to provide more flexible assertions for floating-point values. See gmx::test::FloatingPointTolerance for the possible ways to specify the tolerance, and gmx::test::FloatingPointDifference for some additional details of the difference calculation.
#define	EXPECT_FLOAT_EQ_TOL(value1, value2, tolerance)
	Asserts that two single-precision values are within the given tolerance.

#define	EXPECT_DOUBLE_EQ_TOL(value1, value2, tolerance)
	Asserts that two double-precision values are within the given tolerance.

#define	EXPECT_REAL_EQ_TOL(value1, value2, tolerance)
	Asserts that two `real` values are within the given tolerance.

#define	ASSERT_FLOAT_EQ_TOL(value1, value2, tolerance)
	Asserts that two single-precision values are within the given tolerance.

#define	ASSERT_DOUBLE_EQ_TOL(value1, value2, tolerance)
	Asserts that two double-precision values are within the given tolerance.

#define	ASSERT_REAL_EQ_TOL(value1, value2, tolerance)
	Asserts that two `real` values are within the given tolerance.

Assertions for NULL comparison
These macros should be used instead of `(EXPECT\|ASSERT)_EQ(NULL, ...)`, because Google Test doesn't support the NULL comparison with xlC++ 12.1 on BG/Q.
#define	EXPECT_NULL(val) EXPECT_EQ((void *) NULL, val)
	Asserts that a pointer is null. More...

#define	ASSERT_NULL(val) ASSERT_EQ((void *) NULL, val)
	Asserts that a pointer is null. More...

Classes
class	gmx::test::CommandLine
	Helper class for tests that need to construct command lines. More...

class	gmx::test::CommandLineTestHelper
	Helper class for tests that construct command lines that need to reference existing files. More...

class	gmx::test::CommandLineTestBase
	Test fixture for tests that call a single command-line program with input/output files. More...

class	gmx::test::IntegrationTestFixture
	Test fixture for integration tests. More...

class	gmx::test::TestReferenceData
	Handles creation of and comparison to test reference data. More...

class	gmx::test::TestReferenceChecker
	Handles comparison to test reference data. More...

class	gmx::test::StringTestBase
	Test fixture for tests that check string formatting. More...

class	gmx::test::FloatingPointDifference
	Computes and represents a floating-point difference value. More...

class	gmx::test::FloatingPointTolerance
	Specifies a floating-point comparison tolerance and checks whether a difference is within the tolerance. More...

class	gmx::test::TestException
	Exception class for reporting errors in tests. More...

class	gmx::test::TestFileManager
	Helper for tests that need input and output files. More...

class	gmx::test::TestFileInputRedirector
	In-memory implementation for FileInputRedirectorInterface for tests. More...

class	gmx::test::TestOptionsProvider
	Provides additional options for the test executable. More...

Macros
#define	GMX_TEST_OPTIONS(name, options)
	Macro to add additional command-line options for the test binary. More...

Enumerations
enum	gmx::test::ReferenceDataMode { gmx::test::erefdataCompare, gmx::test::erefdataCreateMissing, gmx::test::erefdataUpdateAll }
	Mode of operation for reference data handling. More...

Functions
void	gmx::test::initReferenceData (Options *options)
	Initializes reference data handling. More...

	gmx::test::FloatingPointDifference::FloatingPointDifference (float value1, float value2)
	Initializes a single-precision difference.

	gmx::test::FloatingPointDifference::FloatingPointDifference (double value1, double value2)
	Initializes a double-precision difference.

bool	gmx::test::FloatingPointDifference::isNaN () const
	Whether one or both of the compared values were NaN. More...

double	gmx::test::FloatingPointDifference::asAbsolute () const
	Returns the difference as an absolute number (always non-negative).

gmx_uint64_t	gmx::test::FloatingPointDifference::asUlps () const
	Returns the difference as ULPs (always non-negative). More...

bool	gmx::test::FloatingPointDifference::signsDiffer () const
	Whether the compared values were of different sign. More...

bool	gmx::test::FloatingPointDifference::isDouble () const
	Whether the difference is between single- or double-precision numbers.

std::string	gmx::test::FloatingPointDifference::toString () const
	Formats the difference as a string for assertion failure messages.

	gmx::test::FloatingPointTolerance::FloatingPointTolerance (float singleAbsoluteTolerance, double doubleAbsoluteTolerance, gmx_uint64_t singleUlpTolerance, gmx_uint64_t doubleUlpTolerance, bool bSignMustMatch)
	Creates a tolerance with the specified values. More...

bool	gmx::test::FloatingPointTolerance::isWithin (const FloatingPointDifference &difference) const
	Checks whether a difference is within the specified tolerance. More...

std::string	gmx::test::FloatingPointTolerance::toString (const FloatingPointDifference &difference) const
	Formats the tolerance as a string for assertion failure messages.

void	gmx::test::registerTestOptions (const char name, TestOptionsProvider provider)
	Registers a test option provider with the test framework. More...

void	gmx::test::initTestOptions (Options *options)
	Initializes the options from all registered test providers. More...

static FloatingPointTolerance	ulpTolerance (gmx_uint64_t ulpDiff)
	Creates a tolerance that only allows a specified ULP difference. More...

FloatingPointTolerance	relativeToleranceAsFloatingPoint (double magnitude, double tolerance)
	Creates a tolerance that allows a difference in two compared values that is relative to the given magnitude. More...

static FloatingPointTolerance	relativeToleranceAsPrecisionDependentUlp (double magnitude, gmx_uint64_t singleUlpDiff, gmx_uint64_t doubleUlpDiff)
	Creates a tolerance that allows a precision-dependent relative difference in a complex computation. More...

static FloatingPointTolerance	absoluteTolerance (double tolerance)
	Creates a tolerance that allows a specified absolute difference.

static FloatingPointTolerance	relativeToleranceAsUlp (double magnitude, gmx_uint64_t ulpDiff)
	Creates a tolerance that allows a relative difference in a complex computation. More...

static FloatingPointTolerance	defaultRealTolerance ()
	Returns the default tolerance for comparing `real` numbers.

Directories
directory	testutils
	Testing Utilities (testutils)

directory	tests
	Unit tests for Testing Utilities (testutils).

Files
file	cmdlinetest.h
	Declares utilities testing command-line programs.

file	integrationtests.h
	Declares test fixture for integration tests.

file	refdata.h
	Functionality for writing tests that can produce their own reference data.

file	stringtest.h
	Declares gmx::test::StringTestBase.

file	testasserts.h
	Extra assertions for unit tests.

file	testexceptions.h
	Exception classes for errors in tests.

file	testfilemanager.h
	Declares gmx::test::TestFileManager.

file	testfileredirector.h
	Declares generic mock implementations for interfaces in fileredirector.h.

file	testoptions.h
	Functions for accessing test command-line options.

Macro Definition Documentation

#define ASSERT_NULL ( val ) ASSERT_EQ((void *) NULL, val)

Asserts that a pointer is null.

Works exactly like ASSERT_EQ comparing with a null pointer.

#define EXPECT_NULL ( val ) EXPECT_EQ((void *) NULL, val)

Asserts that a pointer is null.

Works exactly like EXPECT_EQ comparing with a null pointer.

#define GMX_TEST_OPTIONS	(	name,
		options
	)

Macro to add additional command-line options for the test binary.

Parameters

name	Unique name for the set of options.
options	Placeholder name for an gmx::Options object for adding options.

The macro should be followed by a block that adds the desired command-line options to options using gmx::Options::addOption(). Extensible Handling of Options (options) provides an overview of the options machinery.

name must be unique within the executable to which the options are added. If the macro is within an unnamed namespace, then it is sufficient that it is unique within the file.

Typical usage:

#include "gromacs/options/basicoptions.h"
#include "gromacs/options/options.h"
#include "testutils/testoptions.h"
namespace gmx
{
namespace
{
bool g_optionValue = false;
//! \cond
GMX_TEST_OPTIONS(MyTestOptions, options)
{
    options->addOption(BooleanOption("flag").store(&g_optionValue)
                           .description("My description"));
}
//! \endcond
} // namespace
} // namespace gmx

\cond and \endcond statements are necessary around the macro to avoid Doxygen warnings.

One macro invocation per an added option, with more of the implementation details hidden inside the macro, could be nicer. But that requires more elaborate macro machinery, so it is probably not worth the effort and complexity.

Enumeration Type Documentation

enum gmx::test::ReferenceDataMode

Mode of operation for reference data handling.

There should be no need to use this type outside the test utility module.

Enumerator

erefdataCompare

Compare to existing reference data.

If reference data does not exist, or if the test results differ from those in the reference data, the test fails.

erefdataCreateMissing

Create missing reference data.

If reference data does not exist for a test, that test behaves as if erefdataUpdateAll had been specified. Tests for which reference data exists, behave like with erefdataCompare.

erefdataUpdateAll

Update reference data, overwriting old data.

Tests utilizing reference data should always pass in this mode unless there is an I/O error.

Function Documentation

gmx_uint64_t gmx::test::FloatingPointDifference::asUlps ( ) const

inline

Returns the difference as ULPs (always non-negative).

The ULPs are calculated for the type that corresponds to the constructor used to initialize the difference. The ULP difference between 0.0 and -0.0 is zero.

gmx::test::FloatingPointTolerance::FloatingPointTolerance	(	float	singleAbsoluteTolerance,
		double	doubleAbsoluteTolerance,
		gmx_uint64_t	singleUlpTolerance,
		gmx_uint64_t	doubleUlpTolerance,
		bool	bSignMustMatch
	)

inline

Creates a tolerance with the specified values.

Parameters

[in]	singleAbsoluteTolerance	Allowed absolute difference in a single-precision number.
[in]	doubleAbsoluteTolerance	Allowed absolute difference in a double-precision number.
[in]	singleUlpTolerance	Allowed ULP difference in a single-precision number.
[in]	doubleUlpTolerance	Allowed ULP difference in a double-precision number.
[in]	bSignMustMatch	Whether sign mismatch fails the comparison.

void gmx::test::initReferenceData ( Options * options )

Initializes reference data handling.

Adds command-line options to options to set the reference data mode. By default, erefdataCompare is used, but --ref-data create or --ref-data update can be used to change it.

This function is automatically called by initTestUtils().

void gmx::test::initTestOptions ( Options * options )

Initializes the options from all registered test providers.

Parameters

options The options are added here.

This is called automatically by initTestUtils().

bool gmx::test::FloatingPointDifference::isNaN ( ) const

Whether one or both of the compared values were NaN.

If this returns true, other accessors return meaningless values.

bool gmx::test::FloatingPointTolerance::isWithin ( const FloatingPointDifference & difference ) const

Checks whether a difference is within the specified tolerance.

NaNs are always treated outside the tolerance.

void gmx::test::registerTestOptions	(	const char *	name,
		TestOptionsProvider *	provider
	)

Registers a test option provider with the test framework.

Parameters

[in]	name	Name of the options provider (for ordering).
[in]	provider	The provider to register.

Exceptions

std::bad_alloc	if out of memory.
tMPI::system_error	on mutex failures.

Typically not used directly in test code, but through the GMX_TEST_OPTIONS macro.

This gets called from constructors for global variables, so ideally it would not throw to avoid unhandled exceptions. But since this is only test code, it is not worth the effort to try to remove those rare exceptions (mutex failures and out-of-memory from STL).

FloatingPointTolerance relativeToleranceAsFloatingPoint	(	double	magnitude,
		double	tolerance
	)

Parameters

[in]	magnitude	Magnitude of the numbers the computation operates in.
[in]	tolerance	Relative tolerance permitted (e.g. 1e-4).

In addition to setting an ULP tolerance equivalent to tolerance for both precisions, this sets the absolute tolerance such that values close to zero (in general, smaller than magnitude) do not fail the check if they differ by less than tolerance evaluated at magnitude. This accounts for potential loss of precision for small values, and should be used when accuracy of values much less than magnitude do not matter for correctness.

The ULP tolerance for different precisions will be different to make them both match tolerance.

static FloatingPointTolerance relativeToleranceAsPrecisionDependentUlp	(	double	magnitude,
		gmx_uint64_t	singleUlpDiff,
		gmx_uint64_t	doubleUlpDiff
	)

Parameters

[in]	magnitude	Magnitude of the numbers the computation operates in.
[in]	singleUlpDiff	Expected accuracy of single-precision computation (in ULPs).
[in]	doubleUlpDiff	Expected accuracy of double-precision computation (in ULPs).

This works as relativeToleranceAsUlp(), but allows setting the ULP difference separately for the different precisions. This supports cases where the double-precision calculation can acceptably has a higher ULP difference, but relaxing the single-precision tolerance would lead to an unnecessarily loose test.

static FloatingPointTolerance relativeToleranceAsUlp	(	double	magnitude,
		gmx_uint64_t	ulpDiff
	)

Parameters

[in]	magnitude	Magnitude of the numbers the computation operates in.
[in]	ulpDiff	Expected accuracy of the computation (in ULPs).

In addition to setting the ULP tolerance as ulpTolerance(), this sets the absolute tolerance such that values close to zero (in general, smaller than magnitude) do not fail the check if they differ by less than ulpDiff evaluated at magnitude. This accounts for potential loss of precision for small values, and should be used when accuracy of values much less than magnitude do not matter for correctness.

bool gmx::test::FloatingPointDifference::signsDiffer ( ) const

inline

Whether the compared values were of different sign.

0.0 and -0.0 are treated as positive and negative, respectively.

static FloatingPointTolerance ulpTolerance ( gmx_uint64_t ulpDiff )

The tolerance uses the given ULP value for both precisions, i.e., double precision will have much stricter tolerance.

Description

Assertions for exceptions

Assertions for floating-point comparison

Assertions for NULL comparison

Classes

Macros

Enumerations

Functions

Directories

Files

Macro Definition Documentation

Enumeration Type Documentation

Function Documentation