GTK+ unit tests, the whole story

This post tries to give some links and context to my last one (GTK+ unit tests brainstorming (conclusions)), so people interested in this topic can follow all the story behind the work on Gtk+ unit tests.

The last GUADEC in Barcelona the build-brigade was born. Some of the tasks of the build-brigade have to do with the deployment of a continuous integration system for Gnome, as well as to take care of testing and the integration of these tests, among other quality tools, into the continuous integration system. You can have a look here (Gtk+ integration loop), here (Gtk+ module integration loop details), here (Gtk+ module tests results) and here (Gtk+ module tests code coverage statistics) to see a sample demo of what I mean for GTK+, but the idea would be to extend it to all gnome modules.

Igalia, the company I work for, is interested in helping with this to become real, and thus, encouraged me and my partner -and also friend- Dape, to work on the build-brigade to this purpose. Dape has been working more on the continuous integration tool side (you can see his latest posts on his work here, here or here), while I’ve been working more on the testing side.

Regarding my work on the testing side, thomasvs, another member of the build-brigade, told us that there was people inside GTK+ that would be interested in having some Check based unit tests. We were already using Check here in Igalia, so that was nice.

Fernando Herrera warned me that there was a previous project of Sun that aimed the same target, gtkvts, and a forked (and maintained) version of LSB, so I first took a look at it, then I tried to (unseccessfuly) get feedback from GTK+ people and finally drop my conclusions here and here. Basically, I think that starting from scratch and trying to get feedback from GTK+ guys (using a buildbot demo as Fer suggested) would be better.

Thus, I started writing some unit tests for GTK+ using Check and following a similar approach as the one pointed out by Stefan Kost in his GUADEC presentation on check based unit testing, while, at the same time, I tried to get them integrated into the buildbot demo that Dape was preparing. Once I had something functional to show, and integrated into the buildbot demo so people can see the whole picture underneath, I asked for feedback to Gtk+ guys.

After that, a successful brainstorming was started by Tim Janik on this topic, and now, I’m trying to get a set of conclusions to start the work in a proper way, which was the topic of my last post. I think this has been all till now ๐Ÿ™‚

GTK+ unit tests brainstorming and conclusions

These last weeks there has been an interesting brainstorming in gtk-devel about developing unit tests for Gtk+. A lot of people joined and give insight on this topic, it is great seeing so many people interested in this stuff. Now it is time to reach conclusions and there are some hot topics that still need to be decided:

  • Will we use a testing framework like Check or finally develop something specific for Gtk+?
  • Should we stop the test suite after the first failed test or should we get a complete list of failed/passed tests?
  • Should we fail a test whenever we reach a critical/warning?
  • Should we test segmentation faults by forking every single test?
  • and more … ๐Ÿ™‚

If you are interested too, check it out and help us make the best choices ๐Ÿ˜‰