{"id":36,"date":"2007-02-13T11:26:36","date_gmt":"2007-02-13T10:26:36","guid":{"rendered":"http:\/\/blogs.igalia.com\/itoral\/?p=36"},"modified":"2007-02-13T11:26:36","modified_gmt":"2007-02-13T10:26:36","slug":"howto-integrate-unit-tests-with-buildbot","status":"publish","type":"post","link":"https:\/\/blogs.igalia.com\/itoral\/2007\/02\/13\/howto-integrate-unit-tests-with-buildbot\/","title":{"rendered":"Howto: Integrate unit tests with buildbot"},"content":{"rendered":"

After my latest post about unit tests integration with buildbot<\/a> some people asked me how I’ve done it, and got interested in setting up something like that for their own projects. I hope this post helps with this. <\/p>\n

Here we go:<\/p>\n

1. Setup Automake unit tests for your project<\/b><\/p>\n

The first step is to provide a set of Automake compliant unit tests<\/a> for the project. In my case, I used simple tests, so I just needed to define in my Makefile.am<\/em> something like this:<\/p>\n

\nif HAVE_CHECK\n    TESTS = gtkbutton \n            gtkentry \n            ...\n    TESTS_ENVIRONMENT = CK_FORK=yes\n    CK_VERBOSITY=verbose\nelse\n    TESTS =\nendif\n<\/code><\/pre>\n
By defining the TESTS<\/em> variable I’m instructing Automake<\/em> to create a “check”<\/em> target for make, that will build and execute each of the programs listed. For each test program, Automake<\/em> will assume that an exit code 0 means successful, while any other exit code means failure. Once a tests is executed, Automake prints to stdout a message like:<\/p>\n
PASS: [test]<\/code>
\nFAIL: [test]<\/code><\/p>\n
as appropriate. This is important, because I’ll use this later on, when setting up buildbot<\/em>, to detect the passed and failed tests.<\/p>\n
The HAVE_TESTS<\/em> variable should be set at configure time, after checking we have all the tools we need to build and run the tests, so we can disable the tests in case the systemt does not provide the necessary stuff. We’ll see how to do this in the next section.<\/p>\n
2. Adding unit tests to your project<\/b><\/p>\n
Ok, now we have to provide an implementation for all those tests. You can use any unit testing framework to implement them (of course, you can decide not to use any framework too, it is up to you).  I have used Check<\/a> for my tests, so I’ll explain how to set up autotools to use it properly. In case you use use a different tool, a similar setup should be needed.<\/p>\n
First, you need to define in your Makefile.am all the stuff needed to build these test programs, for example, in my case, I added something like:<\/p>\n
\nDEPS =                                                                                  \n    $(top_builddir)\/gdk-pixbuf\/libgdk_pixbuf-$(GTK_API_VERSION).la   \n    $(top_builddir)\/gdk\/$(gdktargetlib)                                           \n    $(top_builddir)\/gtk\/$(gtktargetlib)\n\nINCLUDES =                                        \n    -I$(top_srcdir)                                \n    -I$(top_builddir)\/gdk                         \n    -I$(top_srcdir)\/gdk                           \n    -I$(top_srcdir)\/gtk                            \n    -DGDK_PIXBUF_DISABLE_DEPRECATED   \n    -DGDK_DISABLE_DEPRECATED              \n    -DGTK_DISABLE_DEPRECATED              \n    $(GTK_DEBUG_FLAGS)                         \n    $(GTK_DEP_CFLAGS)                           \n    $(CHECK_CFLAGS)\n\nLDADD =                                                                                \n    $(CHECK_LIBS)                                                                   \n    $(top_builddir)\/gdk-pixbuf\/libgdk_pixbuf-$(GTK_API_VERSION).la  \n    $(top_builddir)\/gdk\/$(gdktargetlib)                                          \n    $(top_builddir)\/gtk\/$(gtktargetlib)\n\ngtkbutton_SOURCES =    \n    check-main.c           \n    check-utils.c            \n    check-gtkbutton.c\n\n...\n<\/code><\/pre>\n
$(CHECK_LIBS) and $(CHECK_CFLAGS) are provided by Check<\/em> at configure time, providing the neccessary libs and flags to compile Check<\/em> based tests. <\/p>\n
There is only one thing missing, that is to detect whether the system has Check<\/em> installed. As I said before, we have to check this at configure time, so add this lines to your configure.in<\/em> script:<\/p>\n
\nAM_PATH_CHECK([0.9.2-4],[have_check=\"yes\"],\nAC_MSG_WARN([Check not found; cannot run unit tests!])\n[have_check=\"no\"])\nAM_CONDITIONAL(HAVE_CHECK, test x\"$have_check\" = \"xyes\")\n<\/code><\/pre>\n
The AM_PATH_CHECK<\/em> macro is provided by Check<\/em> (you might want to add it to your acinclude.m4 file), and is used here to ensure that an appropriate version of Check is installed, setting HAVE_CHECK<\/em> to True in case it is (and thus, enabling the build of the Check<\/em> based tests defined in Makefile.am<\/em>).<\/p>\n
Now, re-run your autogen.sh<\/em> and configure<\/em> scripts. If all goes well you should be able to run “make check”<\/em> to execute your tests:<\/p>\n
\niago@venus:[\/home\/iago\/tests\/gtk+\/ut]# make check\nmake  check-TESTS\nmake[1]: Entering directory `\/home\/iago\/tests\/gtk+\/ut'\nRunning suite(s): GtkButton\n[...]\n100%: Checks: 7, Failures: 0, Errors: 0\ncheck-gtkbutton.c:144:P:new_with_label:test_new_with_label_regular: Passed\ncheck-gtkbutton.c:202:P:new_with_mnemonic:test_new_with_mnemonic_regular: Passed\ncheck-gtkbutton.c:251:P:new_from_stock:test_new_from_stock_regular: Passed\ncheck-gtkbutton.c:290:P:set_get_label:test_set_get_label_regular: Passed\ncheck-gtkbutton.c:313:P:set_get_label:test_set_get_label_invalid: Passed\ncheck-gtkbutton.c:349:P:pressed_released:test_pressed_released_clicked_regular: Passed\ncheck-gtkbutton.c:359:P:pressed_released:test_pressed_released_clicked_invalid: Passed\nPASS: gtkbutton<\/font>\nRunning suite(s): GtkEntry\n[...]\n<\/code><\/pre>\n
Do you see the blue line? That’s an Automake<\/em> output. The lines above that one are Check<\/em> output stating the result for each unit test executed.<\/p>\n
3. Setting up buildbot to build and test your project<\/b><\/p>\n
Next, you need to install buildbot and configure it to build your project(s). I’ll assume you’ve already done this, but if you haven’t yet, you can follow chapter 2 of this manual:<\/p>\n
http:\/\/buildbot.sourceforge.net\/manual-0.7.5.html<\/a><\/p>\n
It is very easy, really.<\/p>\n
Once the above is done, we need to add the build step that will take care of testing. To do this, in the master setup of your project, edit the master.cfg<\/em> file. Go to the Builders<\/em> section, where you configured the build phases of your project, it might look more or less like this:<\/p>\n
\nf = factory.BuildFactory()\nf.addStep(SVN, svnurl=projecturl)\nf.addStep(step.ShellCommand, command=[\"make\", \"all\"])\nf.addStep(step.ShellCommand, command=[\"make\", \"install\"])\n<\/code><\/pre>\n
Now, let’s add a new step which will take care of the testing:<\/p>\n
\nf.addStep(step.ShellCommand, command=[\"make\", \"check\"])<\/code>\n<\/code><\/pre>\n
You can reboot buildbot<\/em> now to see how it works. Once buildbot<\/em> finishes to build the project you can see that all you get for the phase that takes care of the tests is a plain text log with the “make check”<\/em> command stdout. Let’s now see how we can get a better report.<\/p>\n
4. Adding the tests HTML report<\/b><\/p>\n
To get the HTML report I showed in my latest post I created a new customized build step class, inheriting from steps.shell.ShellCommand, which is a base class for shell based commands. This new class will be specialized for “make check” commands: <\/p>\n
\nclass TestCommand(steps.Shell.ShellCommand):\n    failedTestsCount = 0\n    passedTestsCount = 0\n    testsResults = []\n\n    def __init__(self, stage=None,module=None, moduleset=None, **kwargs):\n        steps.shell.ShellCommand.__init__(self, description=\"Testing\",\n                                                      descriptionDone=\"Tests\", \n                                                      command=[\"make\", \"check\"], **kwargs)\n        self.failedTestsCount = 0\n        self.passedTestsCount = 0\n        self.testsResults = []\n        testFailuresObserver = UnitTestsObserver ()\n        self.addLogObserver('stdio', testFailuresObserver)\n\n    def createSummary(self, log):\n        if self.failedTestsCount > 0 or self.passedTestsCount > 0:\n            self.addHTMLLog ('tests summary', self.createTestsSummary())\n\n    def getText(self, cmd, results):\n        text = steps.shell.ShellCommand.getText(self, cmd, results)\n        if self.failedTestsCount > 0 or self.passedTestsCount > 0:\n            text.append(\"tests failed: \" + str(self.failedTestsCount))\n            text.append(\"tests passed: \" + str(self.passedTestsCount))\n        return text\n\n    def evaluateCommand(self, cmd):\n        if self.failedTestsCount > 0:\n            return WARNINGS\n        else:\n            return SUCCESS\n\n    def createTestsSummary (self):\n            # Create a string with your html report and return it\n            ...\n<\/code><\/pre>\n
The most interesting stuff is in the __init__<\/em> method, where we create an observer (UnitTestsObserver)<\/em> for the stdout log. This means that each time that a new line is output to stdout, that observer will be warned, so it can process it. <\/p>\n
The getText<\/em> method provides the text that is shown in the phase box of the Waterfall<\/em> view of the project. In this case it will show the number of passed and failed tests.<\/p>\n
The createSummary<\/em> method is used to add additional information (for example,  extra logs). In this case I use this method to link a new log with the html summary of the tests done.<\/p>\n
The evaluateCommand<\/em> method is called when the “make check”<\/em> command finishes, to decide the final status of the phase. In this case I set the status to “WARNING” (orange color in Waterfallview) when there are failed tests, or SUCCESS otherwise. I could set it to FAILURE if there are failed tests, but I decided not to flag a build as FAILED when there are failed tests.<\/p>\n
Finally, the createTestsSummary<\/em> method is used to generate the HTML with the tests summary that is being linked in  createSummary<\/em>. In this method you must create and return a string with the HTML page contents.<\/p>\n
Ok, so as we’ve seen the main stuff here is the log observer, which will be responsible for parsing and extracting all the interesting information from stdout in order to provide the data we need to generate the results (passed and failed tests). Let’s see how I implemented it:<\/p>\n
\nclass UnitTestsObserver(buildstep.LogLineObserver):\n    regroupfailed = []\n    regrouppassed = []\n    reunittest = []\n    unittests = []\n\n    def __init__(self):\n        buildstep.LogLineObserver.__init__(self)\n        if len(self.regroupfailed) == 0:\n            self.regroupfailed.append((re.compile('^(FAIL:) (.*)$'), 1))\n        if len(self.regrouppassed) == 0:\n            self.regrouppassed.append((re.compile('^(PASS:) (.*)$'), 1))\n        if len(self.reunittest) == 0:\n            self.reunittest.append((re.compile('^([^:]*):([^:]*):([^:]*):([^:]*):([^:]*):([^:]*).*$'), 4, 5))\n\n    def outLineReceived(self, line):\n        matched = False\n        for r in self.regroupfailed:\n            result = r[0].search(line)\n            if result:\n                self.step.failedTestsCount += 1\n                self.step.testsResults.append((result.groups()[r[1]].strip(), False, self.unittests))\n                self.unittests = []\n                matched = True\n        if not matched:\n            for r in self.regrouppassed:\n                result = r[0].search(line)\n                if result:\n                    self.step.passedTestsCount += 1\n                    self.step.testsResults.append((result.groups()[r[1]].strip(), True, self.unittests))\n                    self.unittests = []\n                    matched = True\n        if not matched:\n            for r in self.reunittest:\n                result = r[0].search(line)\n                if result:\n                    err_msg = result.groups()[r[2]].strip()\n                    if err_msg == \"Passed\":\n                        self.unittests.append((result.groups()[r[1]].strip(), True, err_msg))\n                    else:\n                        self.unittests.append((result.groups()[r[1]].strip(), False, err_msg))\n                    matched = True\n<\/code><\/pre>\n
regroupfailed<\/em> and regrouppassed<\/em> are lists of regular expressions that match failed and passed tests. In my case, because I’m using Automake,<\/em> I know that failed tests output a FAIL: [testname]<\/tt> to stdout while passed tests output PASSED: [testname]<\/tt>, so I added regular expressions to match these cases. This provides integration with Automake<\/em>. reunittest<\/em> is a list of regular expressions that match Check<\/em>‘s output for each unit test executed. When Check<\/em> is used in verbose mode it prompts, for each unit test done, a line like this one to stdout:<\/p>\n
check-gtkfilechooser.c:80:P:set_get_action:test_set_get_action_regular: Passed<\/code><\/p>\n
In this example, test_set_get_action_regular<\/em> is the name of the unit test, and the last component is “Passed” if the test was successful or an error message otherwise.Thus, I added to the list a regular expression to matches such lines and extracts the interesting information from them.<\/p>\n
Because Automake<\/em> does not print its output until all the unit tests of the test program are done, I do not know which test program the unit tests belong to until I get the Automake<\/em> output. That’s why I keep the matched unit tests in the unittests<\/em> variable until I match an Automake<\/em> passed\/failed line (at that moment, I add all the unit tests matched to that test program and reset the unittests variable).<\/p>\n
After processing the entire stdout log, the testsResults<\/em> attribute of the TestCommand<\/em> instance will provide a list with one element per test done. If we name one of those elements as ‘t’, then:<\/p>\n