Some time ago I needed to jump into the fix-compile-test loop for WebKitGTK+, but in the armhf architecture, speaking in terms of Debian/Ubuntu.
To whom don’t know, WebKitGTK+ is huge, it is humongous, and it takes a lot of resources to compile. For me, at first glance, was impossible to even try to compile it natively in my hardware, which, by the way, is an Odroid-X2. So I setup a cross-compilation environment.
And I failed. I could not cross-compile the master branch of WebKitGTK+ using as root file system, a bootstrapped Debian. It is supposed to be the opposite, but all the multiarch thing made my old and good cross-compilation setup (based on scratchbox2) a bloody hell. Long story short, I gave up and I took more seriously the idea of native builds. Besides, Ubuntu and Debian does full native builds of their distributions for armhf, not to say that the Odroid-X2 has enough power for give it a try.
It is worth to mention that I could not use Yocto/OE or buildroot, though I would love to use them, because the target was a distribution based on Debian Sid/Ubuntu, and I would not afford a chroot environment only for WebKitGTK+.
With a lot of patience I was able to compile, in the Odroid, a minimalist configuration of WebKitGTK+ without symbols. As expected, it took ages (less than 3 hours, if I remember correctly)
Quickly an idea popped out in the office: to use distcc. I grabbed as many board based on ARMv7 I could find: another Odroid-X2, a couple Pandaboards, an Arndaleboard, and an IFC6410, installed in them a distcc compilation setup.
And yes, the compilation time went down, but not that much, though I don’t remember how much.
Many of the colleagues at the office migrated from distcc to icecream. Particularly, Juan A. Suárez told me about his experiments with icecc and his Raspberry pi. I decided to give it a shoot.
Icecream permits to do cross-compilation because the scheduler can deliver, into the compilation host, the required tool-chain by the requester.
First, you should have one or several cross tool-chains, one for each compilation tuple. In this case we will have only one: to compile in X86_64, generating code for armfh. Luckily, embdebian provides it, out of the box. Nevertheless you could use any other mean to obtain it, such as crosstool.
Second, you need the icecc-create-env script to create the tarball that the scheduler will distribute to the compilation host.
$ /usr/lib/icecc/icecc-create-env \
--gcc /usr/bin/arm-linux-gnueabihf-gcc-4.7 \
/usr/bin/arm-linux-gnueabihf-g++-4.7
The output of this script is an archive file containing all the files necessary to setup the compiler environment. The file will have a random unique name like “ddaea39ca1a7c88522b185eca04da2d8.tar.bz2” per default. You will need to rename it to something more expressive.
Third, copy the generated archive file to board where your code will be compiled and linked, in this case WebKitGTK+.
For the purpose of this text, I assume that the board has already installed and configured the icecc daemon. Beside, I use ccache too. Hence my environment variables are more or less like these:
CCACHE_PREFIX=icecc CCACHE_DIR=/mnt/hd/.ccache # /mnt/hd is a mounted hard disk through USB. PATH=/usr/lib/ccache:.. # where Debian sets the compiler's symbolic links
Finally, the last pour of magic is the environment variable ICECC_VERSION. This variable needs to have this pattern
<native_archive_file>(,<platform>:<cross_archive_file>=<target>)*.
Where <native_archive_file> is the archive file with the native tool-chain. <platform> is the host hardware architecture. <cross_archive_file> is the archive file with the cross tool-chain. <target> is the target architecture of the cross tool-chain.
In my case, the target is not needed because I’m doing native compilation in armhf. Hence, my ICECC_VERSION environment variable looks like this:
ICECC_VERSION=/mnt/hd/tc/native-compiler.tar.gz,x86_64:/mnt/hd/tc/arm-x86-compiler.tar.gz
And that’s it! Now I’m using the big iron available in the office, reducing the time of a clean compilation in less than an hour.
As a final word, I expect that this compilation time will be reduced a bit more using the new cmake build infrastructure in WebKitGTK+.