Gnome buildbot and integration of unit tests

Using buildbot you can include a “make check” step in your automatic build process.This way you can realize if your tests succeed or fail on each automatic build. This is very useful, along with the option to include references to the commits made to the projet repository, to find out what commit broke something.

When something goes wrong with the tests, the first step is to find out what is actually failing. To do this, buildbot provides a complete text log with all the stdout generated by the “make check” command. Although this may be enough for developers, it is not very confortable, you have to explore by hand a complete text log looking for the test(s) that have failed, skipping passed tests and any other kind of (maybe) irrelevant output generated by the tests.
In order to make this easier, I’ve playing a little with the gnome-buildbot setup by Dape some time ago, trying to add the capability to parse the tests output and generate a more visual and efficient html report with a summary of the tests and the error messages.

This parser looks for strings like: “PASS: ” or “FAIL: “, which are the regular automake output for each test program executed. Besides, I’ve also included support for Check based unit tests, so you can get a detail of each unit tests done for each test program.

Let’s see this with an example, in the following image one can see the buildbot automated building of the gtk+ project. In orange you can see the tests phase, which is actually the result of executing “make check”, previously, I’ve added a set of Check based tests to the project. The orange color of that phase means there are failed tests. Besides the number of passed and failed tests one can see a link to the buildbot plain text stdout log and a link to access the html summary report.

In the next image one can see a piece of the html report with the tests summary, which shows one table per test done. The table title is the name of the test as prompted by automake and each table row is a single test (done with Check) in that test program. If you do not use Check for your unit tests, then you’d only get the table titles. Of course, green means test passed, red means test failed ;). In case a test fails, the error message is shown too.

Finally, for an easy access to the tests of interest, I’ve added a header table with links to each test detail in the report: