José Dapena blog

Yes! Today, 21-September-2006, Igalia celebrates its 5th anniversary!

An example of party at Igalia office (year 2003 ;))

Buildbot

Yesterday I’ve ended the documentation and upload of my JHbuild buildbot integration scripts. I’ve published a manual (jhbuild buildbot manual). With this bot, I’ve got:

• All Gnome 2.16 modules compiling in Buildbot. The configuration files use the JHBuild Gnome 2.16 moduleset. My scripts are easily configurable to use other modulesets.
• There’s a preliminar support for tests and coverage. Currently, it runs “make check” for all Gnome modules supporting this Makefile rule, and then it runs lcov to explore the code coverage of those tests.
• I’ve done some tricks to avoid a huge load produced by buildbot managing and compiling too much modules.

This week I’ll try to get this server in production (currently I’m running it in my PC). There’s a CVS with the scripts, and you can browse them with our Bonsai.

Fisterra

These days I’ve also been preparing some patches for Fisterra framework and apps to get them easily compiled in JHBuild. I’ll try to merge the patches this week. They fix compilation with gcc 4, and make it easier to get the modules compiled in a non-standard prefix (as JHBuild requires).

5th aniversary of Igalia

Next thursday Igalia will be 5 years old. Fine to be such a long time with these guys :).
Weekend

Last saturday, I’ve gone with my girlfriend to an interesting restaurant in Vigo (Rosalia de Castro street). Tasty food (in special the hot chocolate fritters, or the veal tenderloin with strawberries and cheese sauce).

Thanks to Thomas for giving me this clue to get coverage reports with code compiled with gcc 4.x.

In order to get coverage logs, CFLAGS should include -fprofile-arcs -ftest-coverage (and preferrable to get also -O0). It’s what the gcc manual says. It was enough for gcc 3.x. But in gcc 4.x it may not. When you link, you can get this error:

hidden symbol `__gcov_init' in /usr/lib/gcc/i486-linux-gnu/4.0.3/libgcov.a(_gcov.o)

is referenced by DSO

The problem seems to be caused by the linker. In theory, adding -ftest-coverage should imply linking gcov automatically. But it’s not true, at least in my case. The solution: adding -lgcov to the LDFLAGS. Finally I have coverage reports for my gcc 4.1.2 compiler.

CFLAGS=-fprofile-arcs -ftest-coverage

LDFLAGS=-lgcov

First, hello to planet.gnome.org! Thanks to jdub for making it possible. Briefly, I’m working at igalia as a member of the Gnome technologies team, and currently I’m devoted to continuous integration and testing tasks under the umbrella of the Build brigade group. Hackergotchie is comming up, stay tuned.

Back from holidays. Some notes:

• I’ve been for nearly a week in Portugal (Lisbon, Fatima, Castelo Branco, Alto Douro, Viana do Castelo, …). Very nice country… and food.
• Nearly one month of relax far away from a computer in Vigo. I did some local tourism there (visited Gondomar, Mondariz, Villasobroso, A Cañiza, and some nice places in Vigo like Monte Alba, or the boat restaurant in the harbour).
• The worst thing was the terrible forest fire wave in Galicia. In Vigo I couldn’t even see the sun for a week, as all the sky was covered by a dense smoke cloud (photos in flicker Vigo Harbour, Rande Bridge).

But now I’m back. These days I’ll try to go on with the efforts around Buildbot and jhbuild integration:

• I’ve got Buildbot running with jhbuild, as I told in my last post.
• Buildbot is not designed for this use, and it’s a problem because I want to provide better aggregated views (build status of a slave, last builds of a module, information about an specific module, etc). I’ll try to improve a bit these views.
• I need better unit test support in buildbot (specifically I would like to integrate coverage reports as Thomas has done in gstreamer buildbot, and better unit tests log reports).

This week I’ve been playing a bit with buildbot, in order to complete my hack to get gnome jhbuild integrated with it. And it’s running now. I’ve uploaded a patch to buildbot project webpage (buildbot jhbuild support patch in sf).

The ideas of the patch are the following:

• Each module in a jhbuild module set gets a factory
• The factory can be used to get builders in many slaves. Then we’ll get a builder for each factor and slave.
• An scheduler for each builder. In my case the first module uses a Periodic scheduler, and the others are Serial depending on the previous module.

I added an example of a configuration file for buildbot master to get this working. Now I’m running gnome jhbuild in my machine. Saddenly the UI of buildbot does not scale very well for large sets of modules :(.

A capture here:

Today I’ve found a problem trying to create a patch with cvs. I haven’t got write access to gnome repository. This way I have only read-only access to it. The problem is when I try to create a patch with new files involved.

If I run:

$ cvs diff -N

it ignores the new files. If I run:

$ cvs diff -N changedfile1 changedfile2 newfile1 newfile2

it says something like this for every new file:

cvs server: I know nothing about newfile1

An easy solution would be creating two patches:

• One patch using standard cvs diff
• A second patch for new files, using diff command against /dev/null.

But there’s another trick to get a standard patch from cvs. You have to edit the CVS/Entries file of the directory you want to add a file. For example, for a newfile1 in directory directory1, you would change the directory1/CVS/Entries file and add an entry like this one:

/newfile1/0/Initial newfile1//

Then I can do a standard cvs diff command like this one:

$ cvs diff -N

and this will include the new files in the patch. Of course you can use the -u/-U parameters to get unified format patches.

Since the last post, I’ve began to implement all the required methods to provide a complete DBus functionality in jhbuild. I added the following DBus interfaces:

• dict:string,dict:string,string info (array:string modulelist): returns the information about the modules requested.
• array:struct:string,string list (array:string modulelist): returns the list of required dependencies required to compile the module list.
• async update (array:string modulelist, array:string skiplist, string start, string datespec): updates the modules in modulelist and their dependencies, starting in start parameter, ignoring the modules in skiplist. The checkouts will be obtained using the datespec.
• async updateone (array:string modulelist, string datespec): updates the modules in modulelist. The checkouts will be obtained using the datespec.
• async build (array:string modulelist, array:string skiplist, string start, string datespec, dict:string,string options): builds the modules in modulelist and their dependencies, starting in start parameter, ignoring the modules in skiplist. The checkouts will be obtained using the datespec. The options parameter is a dictionary of additional options. They can be autogen (forces autogen.sh of the module), clean (calls make clean before compiling the modules) and nonetwork (forces avoiding access to network, and then, avoids accessing version control repositories).
• async buildone (array:string modulelist, string datespec, dict:string,string options): builds the modules in modulelist. The checkouts will be obtained using the datespec. The options work in the same way as in build command.
• void set_autogen(boolean enabled): set the global autogen parameter value. If set, it will run always autogen.sh before compiling modules.
• boolean get_autogen(): obtains the current global autogen parameter value.
• void set_clean(boolean enabled): set the global makeclean parameter value. If set, it will run always run make clean before compiling modules.
• boolean get_ckean(): obtains the current global autogen parameter value.
• void set_nonetwork(boolean enabled): set the global nonetwork parameter value. If set, it will run always avoid networks access (and checkouts) for build and buildone commands.
• boolean get_nonetwork(): obtains the current global nonetwork parameter value.
• struct:string,string,string get_status(): obtains the current compilation status. The struct has three fields: the current command (build, buildone, updateone, update), the current module, and the status/phase (idle, build, checkout, …).

You can hook to the following signals:

• start_build_signal (): called when a build starts.
• end_build_signal (array:string failures):called when the build ends. failures contains the list of modules that did fail.
• start_module_signal (string module): called when a module build starts. module contains the name of the module.
• end_module_signal (string module, boolean failed): called when a module build ends. module contains the name of the module. failed tells if the module build has failed.
• start_phase_signal (string module, string state): called when a phase in the build of a module begins. module contains the name of the module. state contains the name of the phase.
• end_phase_signal (string module, string state, boolean failed): called when a phase in the build of a module ends.module contains the name of the module. state contains the name of the phase. failed tells if the build phase has failed.
• message (string message): all the messages from compilation logs are sent through this signal. If you want to retrieve the compilation log, you should hook and get all this messages. This signal is called in each phase with a frequency of 5 seconds.

The biggest problem I’ve faced these days was related to the way I could launch the subprocess for build/update commands. As it’s called in a dbus handler, I couldn’t put the jhbuild script in another thread. And the method should return immediately without waiting for the end of the command. I had to use the gobject.Idle interface to hook the build to the gobject mainloop, this way:

1. I implemented a gobject.Idle child, and add a callback implementation calling the builder script and returning False (in order to be called only one time).
2. I attach this idle object to the mainloop, and then end the dbus procedure implementation.

The code for the idle child is something like this:

class JHBuildBuilderIdle(gobject.Idle):
def __init__(self, builder):
gobject.Idle.__init__(self)
self.set_callback(self.callback, builder)

def callback(self, builder):
builder.build()
return False

And the call in the dbus handler is this:

@dbus.service.method('org.gnome.JHBuildIFace')
def build(self, modulelist = [], skiplist = [], start = None, datespec = None, options = {}):
[...]

build = jhbuild.frontends.get_buildscript(ownconfig, module_list)

idle = JHBuildBuilderIdle(build)
idle.attach()
return

Tomorrow I’ll add this work to the jhbuild bugzilla in order to begin the discussion upstream.

Today I’ve began the experiment to add DBus support to JHBuild. To do this I’ve established these goals:

• Be able to launch JHBuild as a DBus service. Also test how can I set it up with DBus activation.
• Implement a list method, equivalent to jhbuild list command.
• Implement one of the build methods.

And guys, I did it! I’ve added a new dbus command to jhbuild to launch it as a DBus service, and begin to wait for user calls.And added a dbus frontend to buildscript. This frontend offers signals to know the status of a compilation (offering methods to get the logs, and to know the current status of compilation.

If I run:

dape@bonus:~$ dbus-send --print-reply --dest='org.gnome.JHBuild'

/org/gnome/JHBuildObject org.gnome.JHBuildIFace.build

it launches a complete build. For example, I can use dbus-monitor to view events, and I get a log like this one:

signal sender=:1.95 -> dest=(null destination) interface=org.gnome.JHBuildIFace; member=end_phase_signal

string "libxml2"    string "checkout"    string "false"

signal sender=:1.95 -> dest=(null destination) interface=org.gnome.JHBuildIFace; member=start_phase_signal

string "libxml2"    string "build"

signal sender=:1.95 -> dest=(null destination) interface=org.gnome.JHBuildIFace; member=message_signal

string "Building libxml2"

signal sender=:1.95 -> dest=(null destination) interface=org.gnome.JHBuildIFace; member=message_signal

string "make  all-recursive make[1]:

se ingresa al directorio `/usr/local/devel/dape/cvs/libxml2'

Making all in include

This way we can communicate with the JHBuild compilation loop. It should be interesting to avoid running jhbuild (and loading python) for each compilation in an integration loop. But it also adds an asynchronous channel to communicate with continuous integration tools. I’ll go on completing the dbus interface for most used commands of jhbuild tomorrow, and also add customizability (currently it does not wait for many kinds of parameters, it should be improved). When it’s more polished I’ll propose the patch for jhbuild.

These days I’ve been doing some work in JHBuild to make life easier for those who want to run it from a Continuous Integration loop. In particular, it should improve the experience for those using Buildbot (as we intend in the Gnome Build Brigade).

Checkout modes for JHBuild

In Buildbot, most used build scripts  enable you to decide how checkouts are done. The options are currently four:

• Update mode: checkout in a directory, then updates over the same folder you’re compiling the tree. It’s the way jhbuild works currently. Problem: sometimes there are generated files in CVS/SVN/whatever. When you update them, they can raise merge conflicts. It’s very typical in gtk-doc templates.
• Clobber mode: it wipes the build directory before checking out. Then, every time you compile, you need to download the full tree. It avoids conflicts as there are no merges. If you use clobber always, it’s good to have a compilation cache to avoid compiling all the trees every time.
• Export mode: similar to clobber, but it uses export instead of checkout. Basically, the difference is you don’t get version control information in the build tree (and then you can use it directly to generate source package, for example).
• Copy mode: works as in update mode, but version control is done in a different directory. Then, after checking out/updating, it copies all the files to the directory it will use to compile. It avoids the merge conflicts problem, enables you to do development over the version control directory being able to create patches easily. Unfortunately it stores the files in version control 2 times.

For me, the most interesting options are the copy mode and the clobber mode. In continuous integration loops, we usually don’t add modifications over the compilation trees, but merge conflicts raise frequently. With those options, you get an easy solution.

I’ve added a bug  and a patch to JHBuild bugzilla to support checkout options.  With this patch, you can establish a global checkout mode, and specific checkout modes for each module in jhbuildrc file.

Running make check from jhbuild without breaking all the moduleset build every time

Currently JHBuild lets you run make check for modules using the autotools module type. If you set the makecheck variable in jhbuildrc, it will run make check before installing the packages. The problem with this is: if the checks fail, then JHBuild considers the module is broken, and then it will not compile the modules depending on it.

I’ve written a patch to change this behavior. With this, you can set up a unittestfails variable (currently defaulting to True to maintain current behavior) in jhbuildrc. If this variable is false, make check failures don’t break compilations of the module. It goes on compiling, but shows a message in the log warning about this.

There was an old bug talking about this in JHBuild bugzilla. I added the patch to this bug.

Future work

Two ideas for JHBuild integration with other tools:

• Convert JHBuild to a Python module. Doing this we could have a version control independent library for python scripts. Buildbot is implemented in Python, so it can be a solution to integrate JHBuild with more flexibility.
• Add DBus support. It would be interesting that applications could fire compilations, and register to wait for different events of the compilations (end of each module, stage or so). Then we could have a better control of the compilation process from external tools.

I’ll be working on this next week.

As I wrote last week, I’ve been attending to Guadec 2006 in Vilanova i la Geltru. A very positive experience, with lots of meetings, talks and interesting events in general.

For last thursday, there was a BOF about Continuous Integration hosted by Juan José Sánchez. There, I presented the work with Tinderbox here in Igalia, and we discussed a bit about the requirements for a Continuos Integration service in Gnome project. The interest was more than I expected, and we shared our experiences and decided to create a work group to get the infrastructure up. It’s called Gnome Build Brigade, and we’ve set up a wiki (yes, all belongs to wiki!)
Currently we have the fantastic work from Frederic Peters (JHAutobuild). It’s running last months, and offers RSS information about compilation status for developers.

But Thomas Vander Stichele proposed Buildbot, as they’re using it in Fluendo for gstreamer. Main reasons: well maintained, big community and more mature. So it seems the way to go, and I began to test it yesterday, as I’m planning to move Fisterra continuous integration to Buildbot.

Yes, buildbot is very easy. It’s more strict about clients (slaves in its notation), but it lets server administrator get a better control over them. And the model is more secure. I’ve also been learning a bit about Twisted, the framework it uses to implement an asynchronous events system. I also implemented a prototype of integration with jhbuild that is working for small sets of modules, but does not handle dependencies yet.
My compromise with Build brigade group is about jhbuild work anyway. I’ll be these days working on JHBuild integration, to extend it to fit better with CI systems (I’ve prepared a patch for stages splitting, and I would want to add more options to checkout). I’ll write about these next days.