This page tries to list and discuss several considerations for C++ testing frameworks, and occasionally mentions which testing frameworks have what features.
It is intended to help me, AndyGlew, decide which of the widely available C++ Unit testing frameworks I may want to use or, failing that, steal features from for inclusion in my own test framework.
Most of these considerations are not specific to C++ unit testing frameworks, but in fact apply to any C++ library intended to be widely used.
See also WhySoManyCeePlusPlusTestFrameworks.
----
'''Summary'''
What started out as a fairly small attempt to compare test suites is turning into BigDesignUpFront. Or, maybe not BDUF, since there really are only a few design alternatives being considered, but perhaps AnalysisParalysis.
Basically, I want the following:
* I want to be able to write a test, and have it automatically added to the test suite
** If I have TestCollector scripts, I want to use them
** If I have no TestCollector scripts, I want to use a registry
* I want libraries with different test frameworks to peacefully coexist
There are two basic ways to structure tests
* void functions
* classes and objects
** classes with a single test() methods
The former work marginally better with TestCollector scripts and multiple test frameworks, but cannot self-register. The latter imposes conventions on the TestCollector scripts, but can self register and are thus easier to use in the absence of scripts.
My listing of pros and cons goes on and on. It may actually contain useful design principles for C++ libraries. But there doesn't seem to be one single best solution for C++, so I think I must get on with life.
(Updated May 2005: Noel Llopis has written a nice survey of C++ unit test frameworks, http://www.gamesfromwithin.com/articles/0412/000061.html, using criteria very similar to my own, concluding that cxxtest is the easiest to use.)
----
After a 72 hour break, I realized that it ''does'' matter, that I was responding to a CodeSmell in CppUnit (CppUnitCodeSmell), and that a small "nugget" satisfied my goals: AndyGlewMinimalCeePlusPlusTestFramework.
----
'''HeaderOnlyCeePlusPlusPackages'''
It is especially convenient if a library or package can be expressed completely in C/C++ header files.
* CxxTest is distributed entirely as a set of header files.
* CppUnit is not header only.
* My personal test library has always started out as header files. I often eventually add object files, but I usually regret doing so.
'''Dependence on "Advanced" C++ Language Features'''
Unfortunately, C++ is very inconsistently implemented.
* CxxTest makes a big point over not requiring RunTimeTypeInformation, member template functions, and exception handling.
* RwCppUnit is for the essentially obsolete RogueWave environment, with no Standard Template Library.
* My personal rule is to use features that are in GnuCpp.
I.e. G++ is my standard of portability.
Programming in the least common denominator of all C++ implementations would be too constraining.
Certain language features are frequently problematic:
* RTTI, RunTimeTypeInformation, is rather incompletely defined by the standard. Even when implemented, it cannot be relied on.
* Templates were a fairly recent addition, and advanced features such as member templates or interior classes are often broken.
* Exception handling is missing in some embedded C++ compilers.
'''OnceAndOnlyOnce'ing'''
C++ is a language that requires lots of repetitive code.
For example, you may need to define a class in one place, and declare it in another, with a forward declaration still somewhere else.
Or you may need to define a testcase in one place, and link it onto a test suite somewhere else.
Or, you may need to type a test case name in both the class and as a printable string.
There are several ways of reducing this repetition, but they have varying degrees of ugliness.
'''''OnceAndOnlyOnceingPrintableNamesOfCodeObjects'''''
Test frameworks often want to be able to report a name for a passing (or failing) test, or suite, or... Often the "print name" and the internal C++ name for the test class or method or function are almost identical.
There are several ways of doing this:
* The name can be entered twice. This is what CppUnit originally required.
* RTTI, RunTimeTypeInformation, but it is extremely ill-defined in the C++ standard and even when present, as in GnuCpp, it may be ugly.
* Macros. E.g. recent CppUnit
* Standard ANSI C __FILE__. Quite restrictive.
* You can use the GNU GCC/GnuCpp predefined strings. Restricting yourself to __FUNCTION__ or _func_ is one reason to use free functions, as opposed to classes, as the basic testcase.
* Outside language scripts. Requires naming conventions, and conventions as to the basic structure for testcases, etc.
'''TestCollector'''
A TestCollector that makes it unnecessary to first code a test, and then enter the test into a list of tests, is a good idea.
Unfortunately, such a tool has to be written outside the C/C++ language as a script.
TestCollector scripts require naming and data structure conventions. For example, a TestCollector may
* process only files that look like {*-test,test-*}.{h,hh,cc}
* grep for class test_XXXX or XXXX_test
* grep for classes that inherit from testcase
* grep for void XXX_test(void) functions
'''Minimize External Library Dependencies'''
The more non-standard libraries a package depends on, the less the chance of being able to use it on a different platform.
You should endeavour to only depend on libraries in the language standard, such as the STL (and even that is problematic), or on libraries whose source code can be distributed with your package.
* CxxTest doesn't depend on external libraries.
* Q: does CppUnit depend on external libraries?
'''Separate GUI Dependencies'''
In particular, as a special case of minimizing external library dependencies, C/C++ have no global standards for Graphical User Interfaces. Therefore, try to separate GUI code out.
'''Text Interface'''
The only reasonably portable I/O system for C and C++ is to try to provide a text-only or command line interface using C++ iostreams or C stdio FILE*s.
C++ iostreams are preferable, because they can be extended.
Thus, for example, standard iostreams can be used in a text only interface. But, a GUI oriented interface might extend the ostream to scroll results in a window, or parse the stream into a still more graphical interface like a folding browser, or one that pattern matches to reports tests passed and failed.
'''''HTML/XML Text Interface'''''
If you must try for "advanced" features, vanilla HTML, and slightly less vanilla XML, are a good place to start.
E.g. XML can indicate arbitrarily nested test suites
.
.
.
These are fairly easily understandable as ordinary text, can be reformatted by standard tools, and can be converted into useful GUI visualizations.
'''Script Dependencies'''
In limited languages like C and C++, it is often convenient to write extra-linguistic scripts, in Perl, etc., to avoid repetitive typing. (The motivation is usually to have a tool that can run before anything can be compiled. In interpretive languages like LISP (or Java with local filesystem access, there is much less reason to write such scripts --- except that scripting languages are often easier to code in, a secondary motivation.)
However, such script dependencies can make it harder to port your code. Now you have to port the C/C++ compiler *and* the scripting language :-(
I prefer to use or abuse language facilities such as the pre-processor (not m4) in order to avoid script dependencies. Occasionally, I will write C/C++ code that performs the sort of file munging actions usually performed by scripting languages.
For testing frameworks, however, pretty much the only way to implement a TestCollector is to use a scripting language.
'''Peaceful Coexistence'''
Big monolithic projects may be able to enforce a single, standard, big monolithic test framework.
However, libraries that are intended to be used in many different projects must have minimal dependencies, and must be prepared to coexist peacefully with other code.
C++ NameSpace''''''s help. Unfortunately, they often have compiler bugs.
From the point of view of test frameworks, peaceful coexistence means, again, minimal dependencies. The library may be embedded in a project that uses a completely different test framework. It should therefore be possible to adapt or wrapperize the library's tests, so that they can be called from the enclosing project's test framework.
(I am assuming that the library is also under development
along with the project. The library is not read-only golden code. The library's unit tests should be run.)
Conversely, a project's test framework should not make
assumptions that make it difficult to embed a library with
a different test framework within the project.
For example:
* TestCollector scripts that scan a directory tree looking for tests should have a mechanism that stops them from going into subdirectories that use a different test framework. In such "interface" directories, adapter scripts can be placed
* Interfacing different test frameworks is easiest when both use textual output. Interfacing different test frameworks that use idiosyncratic internal C++ test result collection is awkward.
'''Testing Extensibility via Inheritance and Templates'''
Often, the whole purpose of the library is as a basis for extension and derivation. Therefore, one of the most important things the library can do is provide tests so that the derived and extended code can be verified.
I.e. the library's tests don't just test the library itself. The library's tests also test the user code that extends the library. Particular extensions or implementations get tested.
In my opinion, this is the key missing factor for code reuse. It has often been hard to extend libraries because the libraries have lacked code to test the extensions. Fragile base class problems arise in the lack of tests for the contract between library and users that extend the library.
If the extension mechanism is inheritance, such testing requires use of virtual functions.
Inheritance, however, is often deprecated. Similarly, virtual functions are often deprecated. They are deprecated both for efficiency reasons, and for code maintainability reasons.
Templates are a WONDERFUL C++ mechanism for tests, allowing tests to be applied to classes unrelated by inheritance.
'''VoidFuncVsTestCaseObjects'''
There are two main ways of defining a test case:
(1) Void functions
void testComplexEquality(void) {
TEST_ASSERT( Complex(10,1) == Complex(10,1) );
TEST_ASSERT( !(Complex(10,1) == Complex(10,2)) );
}
void testComplexAddition(void) {
TEST_ASSERT( Complex(10,1) + Complex(1,1) == Complex(11,2) );
}
(2) Test case objects
class ComplexNumberTest : public CppUnit::TestFixture {
private:
Complex *m_10_1, *m_1_1, *m_11_2;
protected:
void setUp()
{
m_10_1 = new Complex( 10, 1 );
m_1_1 = new Complex( 1, 1 );
m_11_2 = new Complex( 11, 2 );
}
void tearDown()
{
delete m_10_1;
delete m_1_1;
delete m_11_2;
}
void testEquality()
{
CPPUNIT_ASSERT( *m_10_1 == *m_10_1 );
CPPUNIT_ASSERT( !(*m_10_1 == *m_11_2) );
}
};
CppUnit::T''''''estCaller
test( "testEquality",
&ComplexNumberTest::testEquality );
CppUnit::TestResult result;
test.run( &result );
Now, the difference in complexity between these two examples is pretty damned extreme!!!! It's not quite fair, though: CppUnit's test jigs, etc., are quite useful.
In general, however, anything that can be done with a test object can be done with a VoidFuncTest.
VoidFuncTest-s may lead to NameSpace collisions. Objects conserve NameSpace. However, collisions can be avoided be avoided by using file static. Moreover, naming conventions such as test_CLASSNAME are unlikely to lead to collisions if the classes don't collide.
'''''Test Object Structure'''''
When using test objects, there are several different ways to relate the test cases (test functions) and the test case class.
One might do as CppUnit did, making the test functions methods of the class.
class ComplexNumberTest : public CppUnit::TestFixture {
private:
Complex *m_10_1, *m_1_1, *m_11_2;
protected:
void setUp()
{
m_10_1 = new Complex( 10, 1 );
m_1_1 = new Complex( 1, 1 );
m_11_2 = new Complex( 11, 2 );
}
void tearDown()
{
delete m_10_1;
delete m_1_1;
delete m_11_2;
}
void testEquality()
{
CPPUNIT_ASSERT( *m_10_1 == *m_10_1 );
CPPUNIT_ASSERT( !(*m_10_1 == *m_11_2) );
}
};
Or one might make each test case inherit from the test jig that it uses:
class ComplexNumberTestFixture : public CppUnit::TestFixture {
private:
Complex *m_10_1, *m_1_1, *m_11_2;
protected:
void setUp()
{
m_10_1 = new Complex( 10, 1 );
m_1_1 = new Complex( 1, 1 );
m_11_2 = new Complex( 11, 2 );
}
void tearDown()
{
delete m_10_1;
delete m_1_1;
delete m_11_2;
}
};
class ComplexTestEquality : public ComplexNumberTestFixture {
void run()
{
CPPUNIT_ASSERT( *m_10_1 == *m_10_1 );
CPPUNIT_ASSERT( !(*m_10_1 == *m_11_2) );
}
};
In fact, when CppUnit does
CppUnit::T''''''estCaller test( "testEquality",
&ComplexNumberTest::testEquality );
it is doing basically this.
However, T''''''estCaller<>::run takes a result argument, and so succeeds in hiding the details of the result from the user.
'''''Aside''''': a related convention is that a class have a "test()" method, or a test subclass. Taxonomically
* free function VoidFuncTest()
* class Foo, with class FooTest
** class test methods FooTest::method()
* class Foo, with void Foo::test()
'''''Parameterized Test Objects'''''
Test objects (or classes) can be parameterized, so that something like a loop can be used to do range testing. With void functions, the loop would have to be embedded.
However, such parameterized testing is unlikely to be accessible to a TestCollector script.
It seems that VoidFuncTest may be useful as the interface that a TestCollector handles. Class and object test interfaces may be useful inside a test suite, but are less convenient for TestCollectors.
'''''Test Collections'''''
In C++, objects in a collection must have compatible types. E.g. all objects in a test suite must be related by inheritance, and must share the same virtual function "run".
This allows suites and testcases to be intermixed, as long as they are related by inheritance.
But it gets in the way when different test frameworks are used in a package P, and its sublibraries L1 and L2. Since the frameworks are different, you cannot place P's, L1's, and L2's tests in the same collection. You must write adapters.
''One Possible Moral'': if you use free void functions as your top level test object, a TestCollector script can likely collect them. I.e. VoidFuncTest are a lowest common denominator that can allow different test frameworks to coexist with minimal adapter writing.
''Test Registries''
Registries are a C++ idiom that allows instances of a class to be registered centrally. Each file contains a static variable that performs the registration. The main program does not need to enumerate all the tests; the addition of a test or suite to the registry is decentralized.
Unfortunately, only objects can register themselves in this way. Void functions cannot.
This sort of registration is convenient when we do not have a TestCollector script that greps test cases out of the source file. It works in combination with a simpler TestCollector, that simply compiles and links all source files obeying a naming convention like foo-test.{cc,o}
However, it does require that the main program of the test have an explicit call to the test registry's run command.
If we have two different test frameworks in the same program, it might be possible to write a main() by hand that calls the registry for each framework, separately. (As opposed to VoidFuncTests that mush all the different framework's tests together.)
'''TANSTAAFL'''
I kept hoping that there was a way to get the best of both worlds -- the generic nature of a VoidFuncTest, and the flexibility of a TestCaseObject. Unfortunately, there does not seem to be such a way in C++.
If C++ had reflection, it would probably be pretty easy to get both.
'''Test Object Structure and TestCollector Scripts'''
After a night's sleep, and a day doing something else, I realized that using free void functions, and/or using test objects with a test() method that inherit their test fixture, are easier to use with TestCollector scripts than the CppUnit style where test case are methods in the fixture, and must be bound to create a test case object.
Consider CppUnit style testcases implemented as methods of a fixture:
class ComplexNumberTest : public CppUnit::TestFixture {
protected:
void testEquality()
{
CPPUNIT_ASSERT( *m_10_1 == *m_10_1 );
CPPUNIT_ASSERT( !(*m_10_1 == *m_11_2) );
}
void testAddition() { ... }
};
to create a test case object, one must "bind" the various
test methods:
CppUnit::T''''''estCaller
test( "testEquality",
&ComplexNumberTest::testEquality );
CppUnit::TestResult result;
test.run( &result );
A TestCollector script that recognizes such methods and tries to automatically extract them into test case objects must understand enough of C++ language syntax to recognize what class they are methods of. That's a real pain.
For a simpler approach, consider testcases as free void functions
void test_foo(void) {
TEST_ASSERT( foo(1) = "foo" );
}
and consider testcases as objects with a test() method:
class ComplexNumberTestFixture { ... };
class TextComplexNumberEquality : public ComplexNumberTestFixture
{
void test() { TEST_ASSERT( m_1_2 == m_1_2 ); }
}
A TestCollector script does not need to understand much C++ syntax. It does not need to figure out what class they are a member of. In fact, it might simply use naming conventions, grepping for a string in a function name, like VoidFuncTest
void
VoidFuncTest_foo(void) {
or a string in a class name, like
class ClassWithTestMethod_ComplexEqualityTest
or similarly with comments
void
test_foo(void) // extract test VoidFuncTest
{
or
class
ComplexEqualityTest // extract test ClassWithTestMethod
: public arbitrary_test_case
The TestCollector script can then simply generate
testsuite.add( voidfunc_to_testcase( test_foo, "test_foo" ) );
testsuite.add( cwtm_to_testcase( new ComplexEqualityTest, "ComplexEqualityTest" ) );
with the suite arranging to call the function, or instantiate the class as an object and call the test method.
I.e. the tests cases need make no assumptions about the test suite that is running them. The test suite can wrapperize arbitrary code. This promotes peaceful coexistence and makes it easier to automate writing adapters.
'''Location of Test Code'''
Where should the test code be located? There are several common ways:
* IfdeffedTestCode
* ClassesWithTestMethod
* SeparateTestClasses
* SeparateTestFiles
'''Test Output Verbosity'''
As noted above, positive confirmation of tests is good.
But different test suites have different standards for test output. E.g. some may expect to see messages such as "TEST PASSED", and others "test completed successfully". Such small differences in test output make peaceful coexistence of test suites harder.
One advantage of TestCollectors is that they may be able to adapt one test case to any test suite. The test suite driver then would be responsible for generating output in the format it desires.
If this is done, then we want the test cases to be relatively quiet:
void test_complex_equality(void) {
TEST_ASSERT( Complex(1,2) == Complex(1,2) );
}
and not
void test_complex_equality(void) {
cout << "TEST STARTING: test_complex_equality\n";
if( Complex(1,2) == Complex(1,2) ) {
cout << "TEST ENDING SUCCESSFULLY: test_complex_equality\n";
}
else {
cout << "TEST ENDING UNSUCCESSFULLY: test_complex_equality\n";
}
}
The test driver will also be responsible for handling exceptions.
What should you do if you want to wrapperize or adapt a set of test cases that is verbose?
* Trivial: print "Ignore this output" and then allow their verbosity
* Redirect their verbosity - if you can redirect whatever I/O package they are using
* Wrap in a separate process, and redirect their verbosity. The last is good at silencing distractingly verbose tests, but the extra levels of indirection involved make debugging such failing tests more difficult.
MORAL: write silent tests. Let the rest suite handle printing and exceptions.
'''''TestAssertions'''''
Most suites use TestAssertions. The assertions may themselves be verbose,
and/or record how many have been run, a form of TestInventory.
----
'''''Handling Memory Access Errors'''''
Something you might want to consider is whether the test framework can trap memory access errors in the tested code and report them as test outcomes in the GUI or text trace. CppUnit cannot do this - a segfault will abort the entire test run. The 'check' library for C runs tests in a separate process and can therefore detect crashes caused by bad pointer usage, etc.
''AndyGlew responds:''
I've done exactly this, running tests in a separate process, for all sorts of errors that may cause the program to abort.
Note: this is particularly useful to do in a regression test. You can indicate "Bug #2333 that caused a segmentation violation is probably still present".
I have also found separate process debugging to be useful for non-object oriented C code. It may not be possible to create multiple instances of the C data structures in a program. But, fork a separate process, exec if necessary, and you can test them and verify that they haven't been broken, as you refactor them into objects.
However, if you don't bother with separate process debugging, take care to look at the TestSummary and/or TestInventory. I have been burned by tests that silently crashed after one sub-test was finished but before the next had started, or in a subtest that had not announced "Test starting". There was no apparent error, just a lack of failure messages. TestPositiveConfirmation (e.g. the bar turning green) is always a good idea.
----
'''TestingPrivateMethodsInCeePlusPlus'''
Some people may debate
(1) whether private methods should be used at all, and
(2) whether it should be necessary to test private methods,
or whether the whole class should be exercisable from the public interface.
Nevertheless, if you want to, here's a discussion of some ways to do so: TestingPrivateMethodsInCeePlusPlus.
----
'''TestSpeed'''
The topic TestSpeed describes some techniques to manage the amount of time tests take to run.
----
'''TestIntegrationWithBuildSystems'''
Another good criteria is to see if each test framework integrates well with your build system.
----
CategoryCpp CategoryTesting CategoryFramework