Good news! Finally, we are using containers in Travis CI for Grilo!. Something I was trying for a while, but we achived it now. I must say that a post Bassi wrote was the trigger for getting into this. So all my kudos to him!

In this post I’ll explain the history behind using Travis CI for Grilo continuous integration.

The origin

It all started when one day exploring how GitHub integrates with other services, I discovered Travis CI. As you may know, Travis is a continuous integration service that checks every commit from a project in GitHub, and for each one it starts a testing process. Roughly, it starts a “virtual machine”¹ running Ubuntu², clones the repository at that commit under test, and runs a set of commands defined in the .travis.yml file, located in the same project GitHub repository. In that file, beside the steps to execute the tests, it contains the instructions about how to build the project, as well as which dependencies are required.

Note that before Travis, instead of a continuous integration system in Grilo we had a ‘discontinuous’ one: run the checks manually, from time to time. So we could have a commit entering a bug, and we won’t realize until we run the next check, which can happen way later. Thus, when I found Travis, I thought it would be a good idea to use it.

Setting up .travis.yml for Grilo was quite easy: in the before_install section we just use apt-get to install all requirements: libglib2.0-dev, libxml2-dev, and so on. And then, in the script section we run autogen.sh and make. If nothing fails, we consider the test is successful. We do not run any specific test because we don’t have any in Grilo.

For the plugins, the same steps: install dependencies, configure and build the plugins. In this case, we also run make check, so tests are run always. Again, if nothing fails Travis gives us a green light. Otherwise, a red one. The status is shown in the main web page. Also, if the test fail, an email is sent to the commit author.

Now, this has a small problem when testing plugins: they require Grilo, and we were relying in the package provided by Ubuntu (it is listed in the dependencies). But what happens if the current commit is using a feature that was added in Grilo upstream, but not released yet? One option could be cloning Grilo core, building and installing it, before the plugins, and then compiling the plugins, depending on this version. This means that for each commit in plugins, we need to build two projects, adding lot of complexity in the Travis file. So we decided to go with a different approach: just create a Grilo package with the required unreleased Grilo core version (only for testing), and put it in a PPA. Then we can add that PPA in our .travis.yml file and use that version instead.

A similar problem happens with Grilo itself: sometimes we require a specific version of a package that is not available in the Ubuntu version used by Travis (Ubuntu 12.04). So we need to backport it from a more recent Ubuntu version, and add it in the same PPA.

Summing up, our .travis.yml files just add the PPA, install the required dependencies, build and test it. You can take a look at the core and plugins file.

Travis and the Peter Pan syndrome

Time passes, we were adding more features, new plugins, fixing problem, adding new requirements or bumping up the required versions… but Travis continues using Ubuntu 12.04. My first thoughts were “OK, maybe Travis wants to rely only in LTS releases”. So we need to wait until the next LTS is released, and meanwhile backporting everything we need. No need to say that doing this becomes more and more complicated as time is passing. Sometimes backporting a single dependency requires to backport a lot of other dependencies, which can end up in a bloody nightmare. “Only for a while, until the new LTS is released”, repeated to myself.

And good news! Ubuntu 14.04, the new LTS, is released. But you know what? Travis is not updated, and still uses the old LTS!. What the hell!

Moreover, two years later after this release, Ubuntu 16.04 LTS is also released, and Travis still uses 12.04!

At that moment, backporting were so complex that basically I gave up. And Continuous Integration was basically broken.

Travis and the containers.

And we were under this broken status until I read Travis was adding support for containers. “This is what we need”. But the truth is that even I knew that it would fix all the problems, I wasn’t very sure how to use the new feature. I tried several approaches, but I wasn’t happy with none of them.

Until Emmanuele Bassi published a post about using Meson in Epoxy. That post included an explanation about using Docker containers in Travis, which solved all the doubts I had, and allowed me to finally move to use containers. So again, thank you, Emmanuele!

What’s the idea? First, we have created a Docker container that has preinstalled all the requirements to build Grilo and the plugins. We tagged this image as base.

When Travis is going to test Grilo, we instruct Travis to build a new container, based on base, that builds and installs Grilo. If everything goes fine, then our continous integration is successful, and Travis gives green light. Otherwise it gives red light. Exactly like it happened in the old approach.

But we don’t stop here. If everything goes fine, we push the new container into Docker register, tagging it as core. Why? Because this is the image we will use for building the plugins.

And in the case of plugins we do exactly the same as in the core. But this time, instead of relying in the base image, we rely in the core one. This way, we always use a version that has an up-to-date version of Grilo, so we don’t need to package it when introducing new features. Only if either Grilo or the plugins require a new dependency we need to build a new base image and push it. That’s all.

Also, as a plus, instead of discarding the container that contains the plugins, we push it in Docker, tagged as latest. So anyone can just pull it with Docker to have a container to run and test Grilo and all the plugins.

If interested, you can take a look at the core and plugins files to check how it looks like.

Oh! Last but not least. This also helped us to test the building both using Autotools and Meson, both supported in Grilo. Which is really awesome.

Summing up, moving to containers provides a lot of flexibility, and make things quite easier.

Please, leave any comment or question either in Facebook or Google+.

UPDATE (26/11/2017): comments are available again.