The striking difference in attitude between commercial users and open source users (e.g. more professionalism, gratitude and patience in the former than the latter) is most likely due to one of Cialdini's Influences: Commitment (in the book, "commitment and consistency").
It seems that the initial decision to spend money on software causes an emotional investment in the user. One would therefore expect that when a user has had the choice of software forced upon them (e.g. the use of Windows in a corporate environment, the use of an unfamiliar version control utility in a dev department, etc), they would be as impatient, rude, and generally unprofessional as a typical open source user.
Informal observation bears this out, but it would be nice to see some studies demonstrating the effects -- a simple review of ticketing systems for similar projects (Apache/IIS, Eclipse/VisualStudio, etc) should bear some fruit.
Showing posts with label software engineering. Show all posts
Showing posts with label software engineering. Show all posts
Saturday, May 14, 2011
Sunday, August 15, 2010
Unit Testing in C
There are a number of unit test frameworks for C, but they all seem to lack something.
Sure, each can be used to perform automated testing, and some will even generate test stubs for you. But all of them are limited to API testing; that is, the only functions that can be unit-tested are the public API exported in the header files.
C differs from object-oriented languages in that most of the work -- the units to be tested, as it were -- is done in static functions that are not exported. This makes unit-testing, as it stands, all but useless from a C programmer's perspective; there is no compelling reason to use unit tests over the usual "test programs running on test data and returning 0 or 1 to the makefile" approach.
With some small work, however, it is possible to apply a unit test framework to a C project, and to perform actual unit tests (i.e. on static functions). It is a bit ugly, it is a bit invasive (naturally each .c file must contain its own unit tests), but it is scalable and maintainable.
Unit Testing with CHECK
check is a unit testing framework for C (manual). It is primarily designed to be used with the GNU autotools (autoconf, automake, etc), and the documentation is not clear on integrating check with a standard Makefile-based project. The unit tests developed here will serve as an example.
To begin, assume a project with the following directory structure:
The main program, the shared library libstuff.so, has its API in stuff.h, its code in stuff.c, and is built by the Makefile. The unit tests run by check will be in the tests director.y
stuff.h
stuff.c
Makefile
So far, this is all pretty straightforward stuff. The Makefile in the tests directory will contain all of the check-specific settings.
tests/Makefile
# Test Definitions (to be added later)
There are a few things to take note of here.
First, the compiler options profile-arcs and test-coverage cause check to perform test coverage profiling. See the the manual for details.
Next, libcheck.a (there is no .so) is added to the linker options, enclosed in --whole-archive directives so that the linker will not discard what it thinks are unused object files. This last point is important; when linking a static library into a dynamic library, --whole-archive must be used.
Finally, the preprocessor definition UNIT_TEST is added to the compiler options. This will be used to ensure that unit tests do not get built into a distribution version of libstuff.so.
Simple Unit Test
The first unit test will be a simple one that ensures the library links with no errors. This is a common problem when developing a shared library; unresolved symbols will not be reported until an executable is linked to the library.
tests/test_link.c
Add a make target for this first test.
tests/Makefile
TEST1_FLAGS = $(CC_FLAGS) $(LD_FLAGS)
TEST1_LIBS = $(LIBS) -l$(TGT_NAME)
Note that the lines before # ... go in the definitions (top) part of the Makefile, while the lines after it go in the targets (bottom) part of it.
This first test can now be run:
Success! Note that the link test does not involve check; make will error out if the linking fails. Pedantic unit testers may want to take the route of making the test fail first (by invoking, say, do_nothing() in test_link.c) in order to convince themselves that it works.
Testing Static Functions
Testing a static function requires embedding the test code in the file that contains the function. The UNIT_TEST preprocessor directive will prevent the unit test from being compiled in non-unit-test binaries.
First, append the test code to the end of the library code.
stuff.c
The START_TEST and END_TEST macros are provided by check, as are the tcase_ routines. The strategy here is to define a single exported function, create_static_testcase, which the unit-test binaries will link to.
The next step is to add a unit test suite to the test directory.
tests/test_stuff.c
This file creates a test suite that contains two tests of the library API (defined in test_stuff.c) and three tests of the static functions (defined in stuff.c). The entire suite will be run when the unit-test binary is executed.
Finally, the new test must be added to tests/Makefile, as discussed earlier.
tests/Makefile
TEST2_LIBS = $(LIBS)
Now all of the tests will be run when the make target is invoked:
Sure, each can be used to perform automated testing, and some will even generate test stubs for you. But all of them are limited to API testing; that is, the only functions that can be unit-tested are the public API exported in the header files.
C differs from object-oriented languages in that most of the work -- the units to be tested, as it were -- is done in static functions that are not exported. This makes unit-testing, as it stands, all but useless from a C programmer's perspective; there is no compelling reason to use unit tests over the usual "test programs running on test data and returning 0 or 1 to the makefile" approach.
With some small work, however, it is possible to apply a unit test framework to a C project, and to perform actual unit tests (i.e. on static functions). It is a bit ugly, it is a bit invasive (naturally each .c file must contain its own unit tests), but it is scalable and maintainable.
Unit Testing with CHECK
check is a unit testing framework for C (manual). It is primarily designed to be used with the GNU autotools (autoconf, automake, etc), and the documentation is not clear on integrating check with a standard Makefile-based project. The unit tests developed here will serve as an example.
To begin, assume a project with the following directory structure:
Makefile
stuff.c
stuff.h
tests/
tests/Makefile
tests/Makefile
The main program, the shared library libstuff.so, has its API in stuff.h, its code in stuff.c, and is built by the Makefile. The unit tests run by check will be in the tests director.y
stuff.h
#ifndef STUFF_H
#define STUFF_H
double do_stuff( double i );
#endif
stuff.c
#include <math.h>
#include "stuff.h"
static double step1( double i ) { return i * i; }
static double step2( double i ) { return i + i; }
static double step3( double i ) { return pow( i, i ); }
double do_stuff( double i ) { return step3( step2( step1( i ) ) ); }
#include "stuff.h"
static double step1( double i ) { return i * i; }
static double step2( double i ) { return i + i; }
static double step3( double i ) { return pow( i, i ); }
double do_stuff( double i ) { return step3( step2( step1( i ) ) ); }
Makefile
# Library Makefile
# -------------------------------------------------------------------
NAME = stuff
LIBNAME = lib$(NAME).so
ARCHIVE = lib$(NAME).a
DEBUG = -DDEBUG -ggdb
OPT = -O2
ERR = -Wall
INC_PATH = -I.
LIB_PATH =
#----------------------------------------------------------------
CC = gcc
LD = ld
AR = ar rc
RANLIB = ranlib
RM = rm -f
LIBS = -lm
CC_FLAGS = $(INC_PATH) $(DEBUG) $(OPT) $(ERR) -fPIC
LD_FLAGS = $(LIB_PATH) $(LIBS) -shared -soname=$(LIBNAME)
SRC = stuff.c
OBJ = stuff.o
#---------------------------------------------------------- Targets
all: $(LIBNAME)
.PHONY: all clean check
$(ARCHIVE): $(OBJ)
$(AR) $(ARCHIVE) $^
$(RANLIB) $(ARCHIVE)
$(LIBNAME): $(ARCHIVE)
$(LD) $(LD_FLAGS) --whole-archive $< --no-whole-archive -o $@
.c.o: $(SRC)
$(CC) $(CC_FLAGS) -o $@ -c $<
clean:
[ -f $(LIBNAME) ] && $(RM) $(LIBNAME)|| [ 1 ]
[ -f $(ARCHIVE) ] && $(RM) $(ARCHIVE)|| [ 1 ]
[ -f $(OBJ) ] && $(RM) $(OBJ) || [ 1 ]
cd tests && make clean
check: $(LIBNAME)
cd tests && make && make check
So far, this is all pretty straightforward stuff. The Makefile in the tests directory will contain all of the check-specific settings.
tests/Makefile
# Unit-test Makefile
#--------------------------------------------------------- Definitions
TGT_NAME = stuff
TGT_SRC = ../stuff.c
OPT = -O2 -fprofile-arcs -ftest-coverage
ERR = -Wall
INC_PATH = -I. -I..
LIB_PATH = -L..
LD_PATH = ..
#---------------------------------------------------------
CC = gcc
RM = rm -f
# NOTE: check libs must be enclosed by --whole-archive directives
CHECK_LIBS = -Wl,--whole-archive -lcheck -Wl,--no-whole-archive
LIBS = -lm $(CHECK_LIBS)
LIBS = -lm $(CHECK_LIBS)
# NOTE: UNIT_TEST enables the static-function test case in stuff.c
CC_FLAGS = $(INC_PATH) $(OPT) $(ERR) -DUNIT_TEST
# NOTE: check libs must be enclosed by --whole-archive directives
LD_FLAGS = $(LIB_PATH)
# Test Definitions (to be added later)
TESTS =
#---------------------------------------------------------- Targets
all: $(TESTS)
.PHONY: all clean check
clean:
$(RM) $(TESTS) *.gcno *.gcda
check:
@for t in $(TESTS); do \
LD_LIBRARY_PATH='$(LD_PATH)' ./$$t; \
done
There are a few things to take note of here.
First, the compiler options profile-arcs and test-coverage cause check to perform test coverage profiling. See the the manual for details.
Next, libcheck.a (there is no .so) is added to the linker options, enclosed in --whole-archive directives so that the linker will not discard what it thinks are unused object files. This last point is important; when linking a static library into a dynamic library, --whole-archive must be used.
Finally, the preprocessor definition UNIT_TEST is added to the compiler options. This will be used to ensure that unit tests do not get built into a distribution version of libstuff.so.
Simple Unit Test
The first unit test will be a simple one that ensures the library links with no errors. This is a common problem when developing a shared library; unresolved symbols will not be reported until an executable is linked to the library.
tests/test_link.c
#include <stuff.h>
int main(void) { return (int) do_stuff( 32.0 ); }
int main(void) { return (int) do_stuff( 32.0 ); }
Add a make target for this first test.
tests/Makefile
# Test 1 : Simple test to ensure that linking against the library succeeds
TEST1 = test_link
TEST1_SRC = test_link.c
TEST1_FLAGS = $(CC_FLAGS) $(LD_FLAGS)
TEST1_LIBS = $(LIBS) -l$(TGT_NAME)
TESTS = $(TEST1)
# ...
$(TEST1): $(TEST1_SRC)
$(CC) $(TEST1_FLAGS) -o $@ $^ $(TEST1_LIBS)
Note that the lines before # ... go in the definitions (top) part of the Makefile, while the lines after it go in the targets (bottom) part of it.
This first test can now be run:
bash# make check
gcc -I. -DDEBUG -ggdb -O2 -Wall -fPIC -o stuff.o -c stuff.c
ar rc libstuff.a stuff.o
ranlib libstuff.a
ld -lm -shared -soname=libstuff.so --whole-archive libstuff.a --no-whole-archive -o libstuff.so
cd tests && make && make check
make[1]: Entering directory `stuff/tests'
gcc -I. -I.. -O2 -fprofile-arcs -ftest-coverage -Wall -DUNIT_TEST -L.. -Wl,--whole-archive -lstuff -lcheck -Wl,--no-whole-archive -o test_link test_link.c
make[1]: Leaving directory `stuff/tests'
make[1]: Entering directory `stuff/tests'
make[1]: Leaving directory `stuff/tests'
gcc -I. -DDEBUG -ggdb -O2 -Wall -fPIC -o stuff.o -c stuff.c
ar rc libstuff.a stuff.o
ranlib libstuff.a
ld -lm -shared -soname=libstuff.so --whole-archive libstuff.a --no-whole-archive -o libstuff.so
cd tests && make && make check
make[1]: Entering directory `stuff/tests'
gcc -I. -I.. -O2 -fprofile-arcs -ftest-coverage -Wall -DUNIT_TEST -L.. -Wl,--whole-archive -lstuff -lcheck -Wl,--no-whole-archive -o test_link test_link.c
make[1]: Leaving directory `stuff/tests'
make[1]: Entering directory `stuff/tests'
make[1]: Leaving directory `stuff/tests'
Success! Note that the link test does not involve check; make will error out if the linking fails. Pedantic unit testers may want to take the route of making the test fail first (by invoking, say, do_nothing() in test_link.c) in order to convince themselves that it works.
Testing Static Functions
Testing a static function requires embedding the test code in the file that contains the function. The UNIT_TEST preprocessor directive will prevent the unit test from being compiled in non-unit-test binaries.
First, append the test code to the end of the library code.
stuff.c
/* ================================================================= */
/* UNIT TESTS */
#ifdef UNIT_TEST
#include <check.h>
#include <stdlib.h> /* for rand() */
START_TEST (test_step1)
{
double d = (double) rand();
fail_unless( step1(d) == (d * d), "Step 1 does not square" );
}
END_TEST
START_TEST (test_step2)
{
double d = (double) rand();
fail_unless( step2(d) == (d + d), "Step 2 does not double" );
}
END_TEST
START_TEST (test_step3)
{
double d = (double) rand();
fail_unless( step3(d) == pow(d, d), "Step 3 does not exponentiate" );
}
END_TEST
TCase * create_static_testcase(void) {
TCase * tc = tcase_create("Static Functions");
tcase_add_test(tc, test_step1);
tcase_add_test(tc, test_step2);
tcase_add_test(tc, test_step3);
return tc;
}
#endif
#include <stdlib.h> /* for rand() */
START_TEST (test_step1)
{
double d = (double) rand();
fail_unless( step1(d) == (d * d), "Step 1 does not square" );
}
END_TEST
START_TEST (test_step2)
{
double d = (double) rand();
fail_unless( step2(d) == (d + d), "Step 2 does not double" );
}
END_TEST
START_TEST (test_step3)
{
double d = (double) rand();
fail_unless( step3(d) == pow(d, d), "Step 3 does not exponentiate" );
}
END_TEST
TCase * create_static_testcase(void) {
TCase * tc = tcase_create("Static Functions");
tcase_add_test(tc, test_step1);
tcase_add_test(tc, test_step2);
tcase_add_test(tc, test_step3);
return tc;
}
#endif
The START_TEST and END_TEST macros are provided by check, as are the tcase_ routines. The strategy here is to define a single exported function, create_static_testcase, which the unit-test binaries will link to.
The next step is to add a unit test suite to the test directory.
tests/test_stuff.c
#include <check.h>
#include <math.h>
#include <stdlib.h>
#include <stuff.h>
#define SUITE_NAME "Stuff"
/* ----------------------------------------------------------------- */
/* TESTS */
START_TEST (test_stuff_diff)
{
double d = (double) rand();
fail_unless ( do_stuff(d) != d, "do_stuff doesn't!" );
}
END_TEST
START_TEST (test_stuff_rand)
{
double d = (double) rand();
double dd = d * d;
double sumdd = dd + dd;
fail_unless ( do_stuff(d) == pow(sumdd, sumdd), "Incorrect result" );
}
END_TEST
/* ----------------------------------------------------------------- */
/* SUITE */
extern TCase * create_static_testcase(void);
Suite * create_suite(void) {
Suite *s = suite_create( SUITE_NAME );
/* Create test cases against library API */
TCase *tc_core = tcase_create ("Core");
tcase_add_test(tc_core, test_stuff_diff);
tcase_add_test(tc_core, test_stuff_rand);
suite_add_tcase(s, tc_core);
/* Create test cases against static functions */
suite_add_tcase( s, create_static_testcase() );
return s;
}
int main( void ) {
int num_fail;
Suite *s = create_suite();
SRunner *sr = srunner_create(s);
srunner_run_all (sr, CK_NORMAL);
num_fail = srunner_ntests_failed (sr);
srunner_free (sr);
return (num_fail == 0) ? EXIT_SUCCESS : EXIT_FAILURE;
}
#include <math.h>
#include <stdlib.h>
#include <stuff.h>
#define SUITE_NAME "Stuff"
/* ----------------------------------------------------------------- */
/* TESTS */
START_TEST (test_stuff_diff)
{
double d = (double) rand();
fail_unless ( do_stuff(d) != d, "do_stuff doesn't!" );
}
END_TEST
START_TEST (test_stuff_rand)
{
double d = (double) rand();
double dd = d * d;
double sumdd = dd + dd;
fail_unless ( do_stuff(d) == pow(sumdd, sumdd), "Incorrect result" );
}
END_TEST
/* ----------------------------------------------------------------- */
/* SUITE */
extern TCase * create_static_testcase(void);
Suite * create_suite(void) {
Suite *s = suite_create( SUITE_NAME );
/* Create test cases against library API */
TCase *tc_core = tcase_create ("Core");
tcase_add_test(tc_core, test_stuff_diff);
tcase_add_test(tc_core, test_stuff_rand);
suite_add_tcase(s, tc_core);
/* Create test cases against static functions */
suite_add_tcase( s, create_static_testcase() );
return s;
}
int main( void ) {
int num_fail;
Suite *s = create_suite();
SRunner *sr = srunner_create(s);
srunner_run_all (sr, CK_NORMAL);
num_fail = srunner_ntests_failed (sr);
srunner_free (sr);
return (num_fail == 0) ? EXIT_SUCCESS : EXIT_FAILURE;
}
This file creates a test suite that contains two tests of the library API (defined in test_stuff.c) and three tests of the static functions (defined in stuff.c). The entire suite will be run when the unit-test binary is executed.
Finally, the new test must be added to tests/Makefile, as discussed earlier.
tests/Makefile
# Test 2 : Actual unit tests against source code
# NOTE: this cannot link against the library as it incorporates the source code
TEST2 = test_$(TGT_NAME)
TEST2_SRC = $(TEST2).c \
$(TGT_SRC)
TEST2_FLAGS = $(CC_FLAGS) $(LD_FLAGS)
TEST2_LIBS = $(LIBS)
TESTS = $(TEST1) $(TEST2)
# ...
$(TEST2): $(TEST2_SRC)
$(CC) $(TEST2_FLAGS) -o $@ $^ $(TEST2_LIBS)
Now all of the tests will be run when the make target is invoked:
gcc -I. -DDEBUG -ggdb -O2 -Wall -fPIC -o stuff.o -c stuff.c
ar rc libstuff.a stuff.o
ranlib libstuff.a
ld -lm -shared -soname=libstuff.so --whole-archive libstuff.a --no-whole-archive -o libstuff.so
cd tests && make && make check
make[1]: Entering directory `stuff/tests'
gcc -I. -I.. -O2 -fprofile-arcs -ftest-coverage -Wall -DUNIT_TEST -L.. -Wl,--whole-archive -lstuff -lcheck -Wl,--no-whole-archive -o test_link test_link.c
gcc -I. -I.. -O2 -fprofile-arcs -ftest-coverage -Wall -DUNIT_TEST -L.. -Wl,--whole-archive -lstuff -lcheck -Wl,--no-whole-archive -o test_stuff test_stuff.c ../stuff.c
make[1]: Leaving directory `stuff/tests'
make[1]: Entering directory `stuff/tests'
Running suite(s): Stuff
100%: Checks: 5, Failures: 0, Errors: 0
make[1]: Leaving directory `stuff/tests'
ar rc libstuff.a stuff.o
ranlib libstuff.a
ld -lm -shared -soname=libstuff.so --whole-archive libstuff.a --no-whole-archive -o libstuff.so
cd tests && make && make check
make[1]: Entering directory `stuff/tests'
gcc -I. -I.. -O2 -fprofile-arcs -ftest-coverage -Wall -DUNIT_TEST -L.. -Wl,--whole-archive -lstuff -lcheck -Wl,--no-whole-archive -o test_link test_link.c
gcc -I. -I.. -O2 -fprofile-arcs -ftest-coverage -Wall -DUNIT_TEST -L.. -Wl,--whole-archive -lstuff -lcheck -Wl,--no-whole-archive -o test_stuff test_stuff.c ../stuff.c
make[1]: Leaving directory `stuff/tests'
make[1]: Entering directory `stuff/tests'
Running suite(s): Stuff
100%: Checks: 5, Failures: 0, Errors: 0
make[1]: Leaving directory `stuff/tests'
Labels:
c/c++,
software engineering
Subscribe to:
Posts (Atom)