Boosting WebKitGTK+ compilation for armhf with icecream

Some time ago I needed to jump into the fix-compile-test loop for WebKitGTK+, but in the armhf architecture, speaking in terms of Debian/Ubuntu.

To whom don’t know, WebKitGTK+ is huge, it is humongous, and it takes a lot of resources to compile. For me, at first glance, was impossible to even try to compile it natively in my hardware, which, by the way, is an Odroid-X2. So I setup a cross-compilation environment.

And I failed. I could not cross-compile the master branch of WebKitGTK+ using as root file system, a bootstrapped Debian. It is supposed to be the opposite, but all the multiarch thing made my old and good cross-compilation setup (based on scratchbox2) a bloody hell. Long story short, I gave up and I took more seriously the idea of native builds. Besides, Ubuntu and Debian does full native builds of their distributions for armhf, not to say that the Odroid-X2 has enough power for give it a try.

It is worth to mention that I could not use Yocto/OE or buildroot, though I would love to use them, because the target was a distribution based on Debian Sid/Ubuntu, and I would not afford a chroot environment only for WebKitGTK+.

With a lot of patience I was able to compile, in the Odroid, a minimalist configuration of WebKitGTK+ without symbols. As expected, it took ages (less than 3 hours, if I remember correctly)

Quickly an idea popped out in the office: to use distcc. I grabbed as many board based on ARMv7 I could find: another Odroid-X2, a couple Pandaboards, an Arndaleboard, and an IFC6410, installed in them a distcc compilation setup.

And yes, the compilation time went down, but not that much, though I don’t remember how much.

Many of the colleagues at the office migrated from distcc to icecream. Particularly, Juan A. Suárez told me about his experiments with icecc and his Raspberry pi. I decided to give it a shoot.

Icecream permits to do cross-compilation because the scheduler can deliver, into the compilation host, the required tool-chain by the requester.

First, you should have one or several cross tool-chains, one for each compilation tuple. In this case we will have only one: to compile in X86_64, generating code for armfh. Luckily, embdebian provides it, out of the box. Nevertheless you could use any other mean to obtain it, such as crosstool.

Second, you need the icecc-create-env script to create the tarball that the scheduler will distribute to the compilation host.

    $ /usr/lib/icecc/icecc-create-env \
          --gcc /usr/bin/arm-linux-gnueabihf-gcc-4.7 \
                /usr/bin/arm-linux-gnueabihf-g++-4.7

The output of this script is an archive file containing all the files necessary to setup the compiler environment. The file will have a random unique name like “ddaea39ca1a7c88522b185eca04da2d8.tar.bz2” per default. You will need to rename it to something more expressive.

Third, copy the generated archive file to board where your code will be compiled and linked, in this case WebKitGTK+.

For the purpose of this text, I assume that the board has already installed and configured the icecc daemon. Beside, I use ccache too. Hence my environment variables are more or less like these:

CCACHE_PREFIX=icecc
CCACHE_DIR=/mnt/hd/.ccache # /mnt/hd is a mounted hard disk through USB.
PATH=/usr/lib/ccache:..    # where Debian sets the compiler's symbolic links

Finally, the last pour of magic is the environment variable ICECC_VERSION. This variable needs to have this pattern

<native_archive_file>(,<platform>:<cross_archive_file>=<target>)*.

Where <native_archive_file> is the archive file with the native tool-chain. <platform> is the host hardware architecture. <cross_archive_file> is the archive file with the cross tool-chain. <target> is the target architecture of the cross tool-chain.

In my case, the target is not needed because I’m doing native compilation in armhf. Hence, my ICECC_VERSION environment variable looks like this:

ICECC_VERSION=/mnt/hd/tc/native-compiler.tar.gz,x86_64:/mnt/hd/tc/arm-x86-compiler.tar.gz

And that’s it! Now I’m using the big iron available in the office, reducing the time of a clean compilation in less than an hour.

As a final word, I expect that this compilation time will be reduced a bit more using the new cmake build infrastructure in WebKitGTK+.

devhelp and jhbuild v2

A couple years ago I wrote a small tip about how to integrate the devhelp installed by your distribution with the documentation generated through jhbuild.

Well, that way is plain wrong. I noticed it when I upgraded to GNOME 3: modifying the environment variable XDG_DATA_DIRS, the gnome-session goes crazy.

So, I found a less intrusive way to make your distro-based devhelp reads the jhbuild-generated documentation:

ln -s /opt/jhbuild/share/gtk-doc/ ~/.local/share/

And that is it!

using bugzilla, the right way

Yes, I admit it, I hate bugzilla. I mean, the bug tracking is something indispensable for every software project, but digging in its web interface for quick searches, I just find it unbearably awful.

But I guess I came across with the right tool for overcome this web madness: pybugz

On the other hand, I’ve to say that I agree about using bugzilla to keep track of patches, it’s a bad idea too. A properly organised mailing list and maybe a patchwork, if the traffic is massive, as helper.

apache configuration for video tag in firefox

In contrast with Epiphany-WebKit, Firefox seems to be quite picky at rendering videos through the HTML5 tag <video>: it demands the correct HTTP headers.

It doesn’t assume anything. If the stream doesn’t have the correct headers, Firefox just will put a black box in you face without any further explanation.

In order to overcome this issue, struggling a little with the sparse information available through the Internet, I came with this .htaccess file:

AddType video/ogg          .ogv
Header set Access-Control-Allow-Origin "*"
Header unset Etag
FileETag None

First of all, my Apache server doesn’t recognize the video/ogg mime type.

Second, the header MUST NOT include the ETag key.

And finally, there’s new header, which isn’t fully implemented by the all browsers, but it will be, and will mess up all our pages with embedded videos, is the cross domain accessibility.

More information:

http://diveintohtml5.org/video.html
http://www.bluishcoder.co.nz/2008/11/video-audio-and-cross-domain-usage.html
http://en.flossmanuals.net/TheoraCookbook/MyHosting
http://www.askapache.com/htaccess/apache-speed-etags.html
http://httpd.apache.org/docs/2.0/mod/mod_headers.html

misc

Do you remember that I promised not to use a minimalistic window manager? Well, sorry, another broken promise. Since I started to play around with imapfilter, I discovered lua. Moreover, a comment in a previous post made a mention of awesome, a minimalistic window manager configured through lua. So, I installed it, play with it, and suddenly I got delighted with all of its features: Awesome comes along pretty well with Gnome and its panel, which I didn’t want to lose at all. Besides, Awesome provides its own panel (called widget box, a.k.a. wibox), which includes a systray (sadly, awesome’s systray steals the icons from the gnome-panel’s systray). I’ve found that a tidy desktop, which avoids to the user unnecessary mouse interactions, is much more relaxed and helps the user to focus on her task. We’ll see how this experiment ends.

Meanwhile, Joaquin, a colleague from Dextra, told about they were having troubles with the gst-openmax JPEG decoder element, because it needed a JPEG parser, while the gst-goo one mimic the official JPEG decoder provided by GStreamer in gst-plugins-good. In other words, the last two elements actually parse the buffer and validates the presences of a complete image in a single buffer, while the first doesn’t, it just assumes it, relying thou on a parser after the data source, which will deliver the image meta-data through fixed capabilities in the parser’s source pad.

Loathed by HAM and all the release processes, I though it could be nice to wet my feet again in the GStreamer waters. Besides, I need to help Iago with his codec base classes, so this JPEG parser, would help me to ramp up.

As an historical note, the first element I took in charge when I get in the GStreamer development group in Dextra was, precisely, the JPEG decoder.

As soon as I chatted with Joaquin, I found a bug report about an element for that purpose, but it still missed a couple features to turn it useful for our needs. And start to hack it. First, I moved from gst-plugins-good, to gst-plugins-bad, and then parse the image header in order to find its properties such as width, height, if the image is progressive, color format, etc. And the set these data in a fixed capability. Also, the frame-rate is negotiated in the sink pad of the parser, such as in the official JPEG decoder.

Finally I got something which seems to work OK and posted it in the same bugzilla report. I hope to receive feedback soon.

On the other hand, I’m still waiting for the approval of my last patches to the GStreamer’s Vala bindings (592346, 592345 and 591979). 591979 can be particularly controversial, given that it changes a typical class method, into a static function. I guess I need to ping somebody.

On the Bacon Video Widget port to Vala, some advances had came, but still there’s nothing to show yet.

the true imap usage: imapfilter

As I said before, I have been using mutt as my e-mail client, and I’m really getting into it. It does what it is supposed to do, cleanly and fast.

Nevertheless its support to IMAP lacks of several nice features. One I miss the most is to notice the unread messages in all of my IMAP folders. Yes, what it does is to notices the new e-mails, but if you ignore the announcement, Mutt will not remind you those not-yet-read messages later on.

And that was my biggest fear: Have I missed an important email?

Then I found imapfilter, the IMAP Swiss Army Knife. Using the LUA programming language, you would manipulate your IMAP resources as you want: you can do complex searches, apply operations on messages (move, copy, delete), put flags on them, etc.

So, my task was obtain the list of unread messages in my IMAP account. After a few of LUA exploration, I managed to get this script:

server = IMAP {
    server = 'mail.server.com',
    username = 'vjaquez',
    password = 'myubberpassword',
    ssl = 'tls1',
}

mailboxes, folders = server:list_all ('*')
for i,m in pairs (mailboxes) do
   -- server[m]:check_status ()
   messages = server[m]:is_unseen () -- + server[m]:is_new ()
   subjects = server[m]:fetch_fields ({ 'subject' }, messages)
   if subjects ~= nil then
      print (m)
      for j,s in pairs (subjects) do
         print (string.format ("t%s", string.sub (s, 0, s:len () - 1)))
      end
   end
end

It prints out nicely all the messages I haven’t read, so I can go to those IMAP folders, and read them.

Sadly, there’s no way to integrate (as far as I can see) the imapfilter results with Mutt. But that would be amazing! Imagine an IMAP email client manipulated by commands like these.

And those feelings made me recall the Philip’s ideas about integrating Tinymail with Tracker.

And I wonder, does imapfilter support pipelining?

password management

Be on-line means have user accounts in a lot of services, such as email, social web sites, blogs, etc.. And have multiple user accounts implies have to remember user names and passwords.

I know people that only have one user name and password and repeat the same for all their accounts. This approach may simplify the need to memorise a lot of different pair of words. But this method is not reliable at all: 1) if your password is compromised, you’ll have to change it in all your accounts, and 2) you may forget an account if you don’t keep track of all the accounts you sign up.

Another big issue is that most of the people have passwords easy to remember, and those passwords usually easy to crack.

In other words, we have two problems: 1) keep track of all our user accounts (resource, user name and password), and 2) the password must be not guessable.

For the second problem you may follow hints and craft each one. But I am lazy guy and computers execute algorithms better than me. So I installed an automated password generator (apg) and let the program offer me a set of possible passwords, choosing the most appealing one.

$ apg -M NCL -c cl_seed -t -x 8
Awnowov6 (Awn-ow-ov-SIX)
Biuj7qua (Bi-uj-SEVEN-qua)
RyecGod9 (Ryec-God-NINE)
Ojonrag1 (Oj-on-rag-ONE)
9KnecOng (NINE-Knec-Ong)
ClagHog0 (Clag-Hog-ZERO)

Neat, don’t you think?

Now, the straightforward solution for the first problem is write down, in a plain text file, the list of resources, user names and password, of every user account you have. This file can be consulted if you don’t remember the account data.

As a personal choice, I use, in Emacs, the org mode to organise the user data, because its table editor is just beautiful. Furthermore, I have several types of outlined user accounts (web sites, email servers, WEP keys, etc.), what it is also handled by org mode.

* web
| site                   | user         | password   |
|------------------------+--------------+------------|
| https://sitio.guay.com | mi_nick_guay | Ojonrag1   |
| ...

* email servers
| server         | user          | password         |
|----------------+---------------+------------------|
| mi_empresa.com | mi_nick_serio | ClagHog0         |
| ...

* wep
| essid          | password  |
|----------------+-----------|
| essid_del_piso | Awnowov6  |
| ...

But now we have a problem: have a plain text file with all your passwords is more insecure than just have one shared among all your user accounts. If somebody gain access to this file, (s)he will own you.

The solution can’t be more simpler: encrypt the file! Well, yes, you’ll have to remember one password, but only one! In order to encrypt GPG is the way to go. GPG not only support asymmetric encryption, but also symmetric, which may be handy if you don’t like or you are not used to use private/public keys. Nevertheless, is worthy learn how to interact with the asymmetric encryption.

Well, if you use Emacs, you have the EasyPG mode, which will easy the GPG interaction, avoiding you to run the gpg command each time you want to read or save your file, the mode will detect if the file is encrypted and it will ask you for the pass phrase to decrypt it transparently for the user.

Once you have encrypted your password file, you can put it in your home page for backup and roaming purposes.

Neat, don’t you think?

This post is heavily inspired in Keeping your secrets secret.

Programming tips: Google’s codesearch

Let’s face it: every programmer loves the “copy&paste” when he is prototyping a solution, learning how to use a new library or just needs a quick fix, and why not?

I mean, yes, it is important understand what we are doing, the implications of every single line of code, keep the elegance, coherency and the simplicity, but those characteristics don’t come just from inspiration neither reading and understanding the theory, nor from a big flame war at email or IRC. We need examples. We learn, as good ape descendants, from imitation.

We are learning/fixing/prototyping and we need fast feedback in order to keep us happy and motivated. We need perceive the progress of our iterative work. The “copy&paste” is great for those purposes. Then, the discipline must do its work sustaining the new acquired knowledge, but that is another history.

But even in the “copy&paste” we must be smart and responsible with ourselves: we must seek the best examples to imitate, we must look at the alpha male, to borrow the brightest resources available. “Copy&Paste” the code of a lousy programmer and you will be another one; “copy&paste” the the code of a great programmer and maybe, someday, you and I will be one.

A great resource to search, easily and fast, great sources of code, since a couple years to now, is Google’s CodeSearch. It does searches along a great number of successful and recognized open projects sources (released tarballs mostly).

When I am working on a programming task which implies new code, my usual work-flow is to visualize the solution as a sequential execution of macro operations which will be defined to to a more fine grained at each iteration. At each iteration usually I found situations when I want to have either a quick solution, or I want to find how others have solved it, or just how to use a specific function/method call, so I go the CodeSearch site and type the used programming language, the related API, et voilà, several possible solutions are shown.

Further more, if I need know how to achieve something with autoconf-fu, script-fu, or even system-configuration-fu, I can do searches with specific file names as Makefile.am or configure.ac.

My two cents in programming tips.

devhelp and jhbuild

When I started to use jhbuild I resolved to not compile a whole new
Gnome desktop because my work laptop is too slow and freeze easily: I
had to reduce the dependencies at minimum and try reuse the libraries
already provided by my distro (Ubuntu by the way).

The first trouble I had was to have access to the gtk-doc API
documentation through the installed devhelp. The solution is simply,
but not obvious (at least for me):

Append in ~/.profile

XDG_DATA_DIRS="${XDG_DATA_DIRS}":/opt/jhbuild/share

export XDG_DATA_DIRS

Well, you know you’ve to change the path to your jhbuild deployment
directory