The past few months, the BIND 10 developers have been using a test-driven development model. As classes and functions are coded, corresponding unit tests are also coded to help verify the routines do what is expected — with good or bad input providing correct results. Sometimes the unit tests are written before the new code or the tests are written soon after. (We don’t always follow the definition as defined at the Wikipedia where the developer first writes the test case which fails.) Providing full unit testing is a requirement of our human code review processes which we follow before importing code for official public release.
BIND 10 is developed in C++ and Python. Back in October, one of our developers, Jinmei experimented with some simple C++ test cases usingCppUnit, CxxTest, googletest, and Boost.Test. (These historical experiments are in our subversion repo.) We decided to use googletest (aka gtest) for C++ and the standard PyUnit for Python. (We didn’t do any experiments of Python unit testing frameworks.)
We provide code coverage reports to show if all code is adequately tested by our unit tests. The reports also indicate what lines of code and functions aren’t used. To build BIND 10 with this support, configure with the –with-lcov option and after building use “make coverage” to generate an HTML report.
We use LCOV which is a front-end to GCC’s gcov report what lines of code are actually executed when running the unittests. LCOV generates HTML webpages to highlight the code (for lines covered and not covered) and showing coverage rates with bar graphs.
The GCC gcov(1) manual page says:
“Software developers also use coverage testing in concert with testsuites, to make sure software is actually good enough for a release. Testsuites can verify that a program works as expected; a coverage program tests to see how much of the program is exercised by the testsuite. Developers can then determine what kinds of test cases need to be added to the testsuites to create both better testing and a better final product.”
We automatically generate new reports when changes are committed to the subversion source repository. The latest C++ unit test code coverage report is at http://bind10.isc.org/~tester/LATEST_UNITTEST_COVERAGE/ and the latest Python unit test code coverage report is at http://bind10.isc.org/~tester/LATEST_PYTHON_UNITTEST_COVERAGE/. (Note that we are still working on automating and improving the python tests.)
The unit tests and the unit test coverage reports have been useful to recognize bugs and to improve the code. (For example, see revisions 361, 387, 404, 426, 451 and many others.) In addition, when we have noticed bugs (outside of the unit tests framework), new test cases have been added to catch them.
We also do other automated source code checking which I will introduce in a different blog article. We will be expanding our automated testing to include higher-level feature tests on different hardware platforms and operating systems and microbenchmarks for charting performance of key components and functions. If you’d like to participate by coding or implementing unit tests or higher level test suites or by running a build slave, please let me know.