Words from the Inside

Grilo, Travis CI and Containers

Thu, 09 Mar 2017 00:00:00 +0100

Good news! Finally, we are using containers in Travis CI for Grilo!. Something I was trying for a while, but we achived it now. I must say that a post Bassi wrote was the trigger for getting into this. So all my kudos to him!

In this post I’ll explain the history behind using Travis CI for Grilo continuous integration.

The origin

It all started when one day exploring how GitHub integrates with other services, I discovered Travis CI. As you may know, Travis is a continuous integration service that checks every commit from a project in GitHub, and for each one it starts a testing process. Roughly, it starts a “virtual machine”¹ running Ubuntu², clones the repository at that commit under test, and runs a set of commands defined in the .travis.yml file, located in the same project GitHub repository. In that file, beside the steps to execute the tests, it contains the instructions about how to build the project, as well as which dependencies are required.

Note that before Travis, instead of a continuous integration system in Grilo we had a ‘discontinuous’ one: run the checks manually, from time to time. So we could have a commit entering a bug, and we won’t realize until we run the next check, which can happen way later. Thus, when I found Travis, I thought it would be a good idea to use it.

Setting up .travis.yml for Grilo was quite easy: in the before_install section we just use apt-get to install all requirements: libglib2.0-dev, libxml2-dev, and so on. And then, in the script section we run autogen.sh and make. If nothing fails, we consider the test is successful. We do not run any specific test because we don’t have any in Grilo.

For the plugins, the same steps: install dependencies, configure and build the plugins. In this case, we also run make check, so tests are run always. Again, if nothing fails Travis gives us a green light. Otherwise, a red one. The status is shown in the main web page. Also, if the test fail, an email is sent to the commit author.

Now, this has a small problem when testing plugins: they require Grilo, and we were relying in the package provided by Ubuntu (it is listed in the dependencies). But what happens if the current commit is using a feature that was added in Grilo upstream, but not released yet? One option could be cloning Grilo core, building and installing it, before the plugins, and then compiling the plugins, depending on this version. This means that for each commit in plugins, we need to build two projects, adding lot of complexity in the Travis file. So we decided to go with a different approach: just create a Grilo package with the required unreleased Grilo core version (only for testing), and put it in a PPA. Then we can add that PPA in our .travis.yml file and use that version instead.

A similar problem happens with Grilo itself: sometimes we require a specific version of a package that is not available in the Ubuntu version used by Travis (Ubuntu 12.04). So we need to backport it from a more recent Ubuntu version, and add it in the same PPA.

Summing up, our .travis.yml files just add the PPA, install the required dependencies, build and test it. You can take a look at the core and plugins file.

Travis and the Peter Pan syndrome

Time passes, we were adding more features, new plugins, fixing problem, adding new requirements or bumping up the required versions… but Travis continues using Ubuntu 12.04. My first thoughts were “OK, maybe Travis wants to rely only in LTS releases”. So we need to wait until the next LTS is released, and meanwhile backporting everything we need. No need to say that doing this becomes more and more complicated as time is passing. Sometimes backporting a single dependency requires to backport a lot of other dependencies, which can end up in a bloody nightmare. “Only for a while, until the new LTS is released”, repeated to myself.

And good news! Ubuntu 14.04, the new LTS, is released. But you know what? Travis is not updated, and still uses the old LTS!. What the hell!

Moreover, two years later after this release, Ubuntu 16.04 LTS is also released, and Travis still uses 12.04!

At that moment, backporting were so complex that basically I gave up. And Continuous Integration was basically broken.

Travis and the containers.

And we were under this broken status until I read Travis was adding support for containers. “This is what we need”. But the truth is that even I knew that it would fix all the problems, I wasn’t very sure how to use the new feature. I tried several approaches, but I wasn’t happy with none of them.

Until Emmanuele Bassi published a post about using Meson in Epoxy. That post included an explanation about using Docker containers in Travis, which solved all the doubts I had, and allowed me to finally move to use containers. So again, thank you, Emmanuele!

What’s the idea? First, we have created a Docker container that has preinstalled all the requirements to build Grilo and the plugins. We tagged this image as base.

When Travis is going to test Grilo, we instruct Travis to build a new container, based on base, that builds and installs Grilo. If everything goes fine, then our continous integration is successful, and Travis gives green light. Otherwise it gives red light. Exactly like it happened in the old approach.

But we don’t stop here. If everything goes fine, we push the new container into Docker register, tagging it as core. Why? Because this is the image we will use for building the plugins.

And in the case of plugins we do exactly the same as in the core. But this time, instead of relying in the base image, we rely in the core one. This way, we always use a version that has an up-to-date version of Grilo, so we don’t need to package it when introducing new features. Only if either Grilo or the plugins require a new dependency we need to build a new base image and push it. That’s all.

Also, as a plus, instead of discarding the container that contains the plugins, we push it in Docker, tagged as latest. So anyone can just pull it with Docker to have a container to run and test Grilo and all the plugins.

If interested, you can take a look at the core and plugins files to check how it looks like.

Oh! Last but not least. This also helped us to test the building both using Autotools and Meson, both supported in Grilo. Which is really awesome.

Summing up, moving to containers provides a lot of flexibility, and make things quite easier.

Please, leave any comment or question either in Facebook or Google+.

UPDATE (26/11/2017): comments are available again.

Let’s call Virtual Machine, container, whatever. In this context it doesn’t matter. ↩
Ubuntu 12.04 LTS, to be exact. ↩

Highlights in Grilo 0.2.11 (and Plugins 0.2.13)

Mon, 29 Sep 2014 00:00:00 +0200

Hello, readers!

Some weeks ago we released a new version of Grilo and the Plugins set (yes, it sounds like a 70’s music group :smile:). You can read the announcement here and here. If you are more curious about all the detailed changes done, you can take a look at the Changelog here and here.

But even when you can read that information in the above links, it is always a pleasure if someone highlights what are the main changes. So let’s go!

Launch Tool

Regarding the core system, among the typical bug fixes, I would highlight a new tool: grl-launch. This tool, as others, got inspiration from GStreamer gst-launch. So far, when you wanted to do some operation in Grilo, like performing a search in YouTube or getting the title of a video on disk, the recommended way was using Grilo Test UI. This is a basic application that allows you to perform the typical operations in Grilo, like browsing or searching, and everthing from a graphical interface. The problem is that this tool is not flexible enough, so you can’t control all the details you could require. And it is also useful to visually check the results, but not to export the to manage with another tool.

So while the Test UI is still very useful, to cover the other cases we have grl-launch. It is a command-line based tool that allows you to perform most of the operations allowed in Grilo, with a great degree of control. You can browse, search, solve details from a Grilo media element, …, with a great control: how many elements to skip or return, the metadata keys (title, author, album, …) to retrieve, flags to use, etc.

And on top of that, the results can be exported directly to a CSV file so it can be loaded later in a spreadsheet.

As example, getting the 10 first trailers from Apple’s iTunes Movie Trailers site:

$ grl-launch-0.2 browse -c 10 -k title,url grl-apple-trailers
23 Blast,http://trailers.apple.com/movies/independent/23blast/23blast-tlr_h480p.mov
A Most Wanted Man,http://trailers.apple.com/movies/independent/amostwantedman/amostwantedman-tlr1_h480p.mov
ABC's of Death 2,http://trailers.apple.com/movies/magnolia_pictures/abcsofdeath2/abcsofdeath2-tlr3_h480p.mov
About Alex,http://trailers.apple.com/movies/independent/aboutalex/aboutalex-tlr1b_h480p.mov
Addicted,http://trailers.apple.com/movies/lionsgate/addicted/addicted-tlr1_h480p.mov
"Alexander and the Terrible, Horrible, No Good, Very Bad Day",http://trailers.apple.com/movies/disney/alexanderterribleday/alexanderterribleday-tlr1_h480p.mov
Annabelle,http://trailers.apple.com/movies/wb/annabelle/annabelle-tlr1_h480p.mov
Annie,http://trailers.apple.com/movies/sony_pictures/annie/annie-tlr2_h480p.mov
Are You Here,http://trailers.apple.com/movies/independent/areyouhere/areyouhere-tlr1_h480p.mov
As Above / So Below,http://trailers.apple.com/movies/universal/asabovesobelow/asabovesobelow-tlr1_h480p.mov
10 results

As said, if you re-direct the output to a file and you import it from a spreadsheet program as CSV you will read it better.

dLeyna/UPnP plugin

Regarding the plugins, here is where the fun takes place. Almost all plugins were touched, in some way or other. In most cases, for fixing bugs. But there are other changes I’d like to highlight. And among them, UPnP is one that suffered biggest changes.

Well, strictly speaking, there is no more UPnP plugin. Rather, it was replaced by new dLeyna plugin, written mainly by Emanuele Aina. From an user point of view, there shouldn’t be big differences, as this new plugin also provides access to UPnP/DLNA sources. So where are the differences?

First off, let’s specify what is dLeyna. So far, if you want to interact with a UPnP source, either you need to deal with the protocol, or use some low-level library, like gupnp. This is what the UPnP plugin was doing. Still it is a rather low-level API, but higher and better than dealing with the raw protocol.

On the other hand, dLeyna, written by the Intel Open Source Technology Center, wraps the UPnP sources with a D-Bus layer. Actually,not only sources, but also UPnP media renderers and controllers, though in our case we are only interested in the UPnP sources. Thanks to dLeyna, you don’t need any more to interact with low-level UPnP, but with a higher D-Bus service layer. Similar to the way we interact with other services in GNOME or in other platforms. This makes easier to browser or search UPnP sources, and allows us to add new features. dLeyna also hides some details specific to each UPnP server that are of no interest for us, but we would need to deal with in case of using a lower level API. The truth is that though UPnP is quite well specified, each implementation doesn’t follow it at 100%: there are always slight differences that create nasty bugs. In this case, dLeyna acts (or should act) as a protection, dealing itself with those differences.

And what is needed to use this new plugin? Basically, having dleyna-service D-Bus installed. When the plugin is started, it wakes up the service, which will expose all the available UPnP servers in the network, and the plugin would expose them as Grilo sources. Everything as it was happening with the previous UPnP source.

In any case, I still keep a copy of the old UPnP plugin for reference, in case someone want to use it or take a look. It is in “unmaintained” mode, so try to use the new dLeyna plugin instead.

Lua Factory plugin

There isn’t big changes here, except fixes. But I want to remark it here because it is where most activity is happening. I must thank Bastien and Victor for the work they are doing here. Just to refresh, this plugin allows to execute sources written in Lua. That is, instead of writing your sources in GObject/C, you can use Lua. The Lua Factory plugin will load and run them. Writing plugins in Lua is a pleasure, as it allows to focus on fixing the real problems and leave the boiler plate details to the factory. Honestly, if you are considering writing a new source, I would really think about writing it in Lua.

And that’s all! It is a longer post than usual, but it is nice to explain what’s going on in Grilo. And remember, if you are considering using Grilo in your product, don’t hesitate to contact with us.

Another year, another GUADEC

Thu, 24 Jul 2014 00:00:00 +0200

It’s 2014, and like previous years:

This time I won’t give any talk, just relax and enjoy talks from others, and hope Strasbourg.

And what is more important, meet those hackers you interact with frequently, and maybe share some beers.

So if you go there, and you want to have a nice chat with me, or talk about Grilo project, don’t hesitate to do it. Igalia, which is kindly sponsoring my attendance, will have a place there during the core days, so likely you could find me around or ask anyone there for me.

Enjoy!

Yum Search Extended

Thu, 06 Mar 2014 00:00:00 +0100

Hi again! Let me tell you something. I’m a Fedora user since several releases ago, probably since Fedora 13 or 14.

Before that, I was using Ubuntu, but decided to switch to Fedora for several reasons that are not worth explaining here. In any case, after switching to Fedora there was something that I was missing quite a lot: the aptitude package manager. aptitude is a deb package manager, similar to apt. What I really like about aptitude is its flexibility when searching packages.

While apt or yum allows to specify the search term, they just get all the packages matching the search text, but they don’t allow you where to search. Do you want to get only packages that are not installed? Or do you just remember the package had python in the name, and part of the description? With aptitude this is not a problem, as it allows you to specify such search expressions.

Though search in yum is not so flexible, as far as I know, it has a nice feature: it allows plugins to implement new features. So several months ago I wrote a plugin to mimic the aptitude search flexibility: yum searchex (search extended).

It is worth saying that I didn’t want to imitate the full aptitude functionality; only those features that I really missed from Ubuntu.

The basic idea is specifying for each term where to search. This is done by prefixing the text with ~ and a letter that expresses where to search. In some cases, the text to search is not needed. For instance, to search only in the list of installed packages, we would use ~i.

The full list of the available options can be found in the project forge.

As an example is worth a thousand words, let’s show how to search a package that we know it contains python in the name, it is not installed, and also we remember it has something to do with KDE:

yum searchex ~apython~dKDE

Hope this plugin is as useful for you as it is for me!

See you at GUADEC 2013!

Mon, 22 Jul 2013 00:00:00 +0200

GUADEC 2013 is around the corner.

Igalia is kindly sponsoring my attendance, as well as other mates, to this wonderful conference, where all years I meet good friends, and do new ones.

I’ll be there from August, 1st to 5th, both included. On saturday 3rd I’ll give a talk about Grilo. If you are using Grilo or willing to, join us to the talk. Of course, I always welcome any question, so if you see me and want to ask anything, don’t hesitate to address me.

Also, I expect to attend on 5th the gnome-music BoF. gnome-music is one of the programs I collaborate with that heavily use Grilo. I really suggest to give a try. I’ts so nice!

Besides all above, Igalia will have a booth during all the event, where we will be showing some of the cool things we do. I still don’t know where it be exactly located, but if you see us, come there!

Grilo 0.2.8 released

Sat, 25 May 2013 00:00:00 +0200

I did a new release of grilo plugins only one week after the previous release because it includes a patch that people from gnome-photos would like to see in next GNOME 3.9 pre-release.

Besides that patch, this new release also includes a new plugin to get content from the Magnatune service. Credits go to Victor Toso, who did a great job on it.

Happy weekend from Igalia headquarter!

What's going on in Grilo?

Sat, 18 May 2013 00:00:00 +0200

There’s a lot of time I don’t blog about Grilo. But it doesn’t mean we are not working on it! Here at Igalia we do, and we also get lot of contributions from community. All the announcements are sent to the Grilo mailing list.

During this week we have released a new version of Grilo, both for core (v0.2.6) and for the plugins (v0.2.7). You can see the announcements here and here. If you are interested in more detailed changelogs, you can see them here and here.

So what happens in this release?

As usual, lot of bugfixes.
Bastien added support for non-file URIs in Filesystem plugin. What this means? In Filesystem plugin you can configure the base path from which the source can show content. Thus, if you setup the base path as ~/Music, it means Filesystem will show only content placed in that directory. With the new approach, you could use as base-path something like recent://, getting the list of recently used items. Cool, uh?
My mate Sergio fixed the cache system in GrlNet. Seems we broke it at some point in the history, and we didn’t realize. We rely on libsoup and its caching feature, and Sergio is very skilled in libsoup. So who’s better than him to fix it? :grinning:
We improved grl-inspect tool. Inspired in gst-inspect from GStreamer, this tool helps to list all the available sources in the system, and list all the supported features. Now, it can also list all the available metadata keys developer can use. Moreover, it also list which sources support each key.
We added support for i18n! Now we speak Brazilian, Czech, Galician, Greek, Polish, Serbian, Slovenian, Spanish and Tajik. And we hope to support more languages in the coming releases. Many thanks to Rafael, Marek, Fran, Dimitris, Piotr, Мирослав, Martin, Miguel and Victor for their work on translating Grilo.
Marek Chalupa added support for GOA in the Flickr plugin. This means that if you have a Flickr account configured in GOA, you will get for free a Flickr source dealing with that account. Nice!

I think that’s all. If you use Grilo and needs help or support, don’t hesitate to contact us!

Grilo at GUADEC 2012

Thu, 19 Jul 2012 00:00:00 +0200

A new year, a new GUADEC. And as in previous years:

This year is a bit special for different reasons:

GUADEC 2012 is allocated in the same city as Igalia headquarter. And in the same region I was born.
I’ll give a talk about Grilo, reviewing its current state and talking about what’s coming next.
There will be also a BoF where we can talk for 2 hours about next features and what would be good to have in Grilo.

Of course, do not hesitate to ask me (or other Grilo contributors) whatever you want to know about Grilo if you see us around. We will be more han happy to answer any doubt.

Grilo at Desktop Summit 2011

Fri, 29 Jul 2011 00:00:00 +0200

Are you interested on Grilo? Are you going to Desktop Summit 2011? Then let’s meet there. Among other igalians,

Also, I and my fellow Guillaume have organized a hacking session on Wednesday 10th, 9:00 in room 1.401/2. If you want to learn more about Grilo, and how to use it to develop your ideas, then write down in the attendant list. It will be a pleasure to count on you, and have a funny time.

And don’t miss neither the BoF about using SeedKit to develop your applications, organized by Alexandre.

If you don’t have time, but still you have any question related to Grilo, don’t hesitate to contact with any of us. See you all there!

UPDATE: Do not forget neither the lightning talk about Grilo extension for GNOME Shell, by Philippe!

GUADEC'2010 talks about Grilo

Mon, 06 Sep 2010 00:00:00 +0200

Thanks to Flumotion, you can access and view the awesome talks that happened at GUADEC 2010.

I have got those related with Grilo, and put them here. Besides the original WebM format videos, I provide also Theora version (in lower quality, intended to those who can not play WebM yet), and the slides too.

The first is a complete talk about Grilo: what is Grilo, what provides, and some of its features.

The second one is a 5 minutes lightning talk, about using Grilo to create a daemon that is able to provide content to other clients through DBus.

The third one, is also a 5 minutes lightning talk, that explains the port of Grilo to Maemo 5, and how it was used to add more multimedia sources to N900‘s Mediaplayer.

Enjoy them!

Iago Toral Quiroga, Grilo: Feeding applications with multimedia content (video, slides)
Juan A. Suárez Romero, Rygel-Grilo (video, slides)
Xabier Rodríguez Calvar, MAFWGriloSource (video, slides)