Words from the Inside Uninteresting things from an uninteresting developer https://blogs.igalia.com/jasuarez/ Fri, 05 Jan 2024 10:28:27 +0100 Fri, 05 Jan 2024 10:28:27 +0100 Jekyll v4.3.3 Implementing Performance Counters in V3D driver <p>Let me talk here about how we implemented the support for performance counters in the <a href="https://www.mesa3d.org/">Mesa</a> <a href="https://docs.mesa3d.org/drivers/v3d.html">V3D driver</a>, the OpenGL driver used by the <a href="https://www.raspberrypi.org/products/raspberry-pi-4-model-b/">Raspberry Pi 4</a>. For reference, the implementation is very similar to the one already available (not done by me, by the way) for the <a href="https://docs.mesa3d.org/drivers/vc4.html">VC4</a>, OpenGL driver for the <a href="https://www.raspberrypi.org/products/raspberry-pi-3-model-b-plus/">Raspberry Pi 3</a> and prior devices, also part of Mesa. If you are already familiar with how this is implemented in VC4, then this will mostly be a refresher.</p> <p>First of all, what are these performance counters? Most of the processors nowadays contain some hardware facilities to get measurements about what is happening inside the processor. And of course graphics processors aren’t different. In this case, the graphics chips used by Raspberry Pi devices (manufactured by Broadcom) can record a bunch of different graphics-related parameters: how many quads are passing or failing depth/stencil tests, how many clock cycles are spent on doing vertex/fragment shading, hits/misses in the GPU cache, and many others values. In fact, with the V3D driver it is possible to measure around 87 different parameters, and up to 32 of them simultaneously. Quite a few less in VC4, though. But still a lot.</p> <p>On a hardware level, using these counters is just a matter of writing and reading some GPU registers. First, write the registers to select what we want to measure, then a few more to start to measure, and finally read other registers containing the results. But of course, much like we don’t expect users to write GPU assembly code, we don’t expect users to write registers in the GPU directly. Moreover, even the Mesa drivers such as V3D can’t interact directly with the hardware; rather, this is done through the kernel, the one that can use the hardware directly, through the DRM subsystem in the kernel. For the case of V3D (and same applies to VC4, and in general to any other driver), we have a driver in user-space (whether the OpenGL driver, V3D, or the Vulkan driver, V3DV), and a kernel driver in the kernel-space, unsurprisingly also called V3D. The user-space driver is in charge of translating all the commands and options created with the OpenGL API or other API to batches of commands to be executed by the GPU, which are submitted to the kernel driver as DRM jobs. The kernel does the proper actions to send these to the GPU to execute them, including touching the proper registers. Thus, if we want to implement support for the performance counters, we need to modify the code in two places: the kernel and the (user-space) driver.</p> <h2 id="implementation-in-the-kernel">Implementation in the kernel</h2> <p>Here we need to think about how to deal with the GPU and the registers to make the performance counters work, as well as the API we provide to user-space to use them. As mentioned before, the approach we are following here is the same as the one used in the VC4 driver: <a href="https://cgit.freedesktop.org/drm/drm/tree/drivers/gpu/drm/v3d/v3d_perfmon.c?id=26a4dc29b74a137f45665089f6d3d633fcc9b662">performance counters monitors</a>. That is, the user-space driver <a href="https://cgit.freedesktop.org/drm/drm/tree/include/uapi/drm/v3d_drm.h?id=26a4dc29b74a137f45665089f6d3d633fcc9b662#n53">creates</a> one or more monitors, specifying for each monitor <a href="https://cgit.freedesktop.org/drm/drm/tree/include/uapi/drm/v3d_drm.h?id=26a4dc29b74a137f45665089f6d3d633fcc9b662#n281">what counters it is interested in</a> (up to 32 simultaneously, the hardware limit). The kernel returns a <a href="https://cgit.freedesktop.org/drm/drm/tree/include/uapi/drm/v3d_drm.h?id=26a4dc29b74a137f45665089f6d3d633fcc9b662#n375">unique identifier</a> for each monitor, which can be used later to do the measurement, <a href="https://cgit.freedesktop.org/drm/drm/tree/include/uapi/drm/v3d_drm.h?id=26a4dc29b74a137f45665089f6d3d633fcc9b662#n57">query the results</a>, and finally <a href="https://cgit.freedesktop.org/drm/drm/tree/include/uapi/drm/v3d_drm.h?id=26a4dc29b74a137f45665089f6d3d633fcc9b662#n55">destroy it</a> when done.</p> <p>In this case, there isn’t an explicit start/stop the measurement. Rather, every time the driver wants to measure a job, it includes <a href="https://cgit.freedesktop.org/drm/drm/tree/include/uapi/drm/v3d_drm.h?id=26a4dc29b74a137f45665089f6d3d633fcc9b662#n141">the</a> <a href="https://cgit.freedesktop.org/drm/drm/tree/include/uapi/drm/v3d_drm.h?id=26a4dc29b74a137f45665089f6d3d633fcc9b662#n278">identifier</a> of the monitor it wants to use for that job, if any. Before submitting a job to the GPU, the kernel <a href="https://cgit.freedesktop.org/drm/drm/tree/drivers/gpu/drm/v3d/v3d_sched.c?id=26a4dc29b74a137f45665089f6d3d633fcc9b662#n133">checks</a> if the job has a monitor identifier attached. If so, then it needs to check if the previous job executed by the GPU was also using the same monitor identifier, in which case it doesn’t need to do anything other than send the job to the GPU, as the performance counters required are already enabled. If the <a href="https://cgit.freedesktop.org/drm/drm/tree/drivers/gpu/drm/v3d/v3d_sched.c?id=26a4dc29b74a137f45665089f6d3d633fcc9b662#n69">monitor is different</a>, then it needs first to <a href="https://cgit.freedesktop.org/drm/drm/tree/drivers/gpu/drm/v3d/v3d_perfmon.c?id=26a4dc29b74a137f45665089f6d3d633fcc9b662#n74">read the current counter values</a> (through proper GPU registers), adding them to the current monitor, <a href="https://cgit.freedesktop.org/drm/drm/tree/drivers/gpu/drm/v3d/v3d_perfmon.c?id=26a4dc29b74a137f45665089f6d3d633fcc9b662#n76">stop the measurement</a>, <a href="https://cgit.freedesktop.org/drm/drm/tree/drivers/gpu/drm/v3d/v3d_perfmon.c?id=26a4dc29b74a137f45665089f6d3d633fcc9b662#n35">configure the counters</a> for the new monitor, <a href="https://cgit.freedesktop.org/drm/drm/tree/drivers/gpu/drm/v3d/v3d_perfmon.c?id=26a4dc29b74a137f45665089f6d3d633fcc9b662#n53">start the measurement again</a>, and finally <a href="https://cgit.freedesktop.org/drm/drm/tree/drivers/gpu/drm/v3d/v3d_sched.c?id=26a4dc29b74a137f45665089f6d3d633fcc9b662#n147">submit</a> the new job to the GPU. In this process, if it turns out there wasn’t a monitor under execution before, then it only needs to execute the last steps.</p> <p>The reason to do all this is that multiple applications can be executing at the same time, some using (different) performance counters, and most of them probably not using performance counters at all. But the performance counter values of one application shouldn’t affect any other application so we need to make sure we don’t mix up the counters between applications. Keeping the values in their respective monitors helps to accomplish this. There is still a small requirement in the user-space driver to help with accomplishing this, but in general, this is how we avoid the mixing.</p> <p>If you want to take a look at the full implementation, it is available in a <a href="https://cgit.freedesktop.org/drm/drm/commit/?id=26a4dc29b74a137f456">single commit</a>.</p> <h2 id="implementation-in-the-driver">Implementation in the driver</h2> <p>Once we have a way to create and manage the monitors, using them in the driver is quite easy: as mentioned before, we only need to <a href="https://gitlab.freedesktop.org/mesa/mesa/-/blob/685281278ebd39114c3007e76443eaaa66cf833/src/gallium/drivers/v3d/v3d_query_perfcnt.c#L222">create a monitor</a> with the counters we are interested in and <a href="https://gitlab.freedesktop.org/mesa/mesa/-/blob/685281278ebd39114c3007e76443eaaa66cf833/src/gallium/drivers/v3d/v3d_job.c#L507">attach it</a> to the job to be submitted to the kernel. In order to make things easier, we keep a <a href="https://gitlab.freedesktop.org/mesa/mesa/-/blob/685281278ebd39114c3007e76443eaaa66cf833/src/gallium/drivers/v3d/v3d_context.h#L307">mirror-like version</a> of the monitor inside the driver.</p> <p>This approach is adequate when you are developing the driver, and you can add code directly on it to check performance. But what about the final user, who is writing an OpenGL application and wants to check how to improve its performance, or check any bottleneck on it? We want the user to have a way to use OpenGL for this.</p> <p>Fortunately, there is in fact a way to do this through OpenGL: the <a href="https://www.khronos.org/registry/OpenGL/extensions/AMD/AMD_performance_monitor.txt">GL_AMD_performance_monitor</a> extension. This OpenGL extension provides an API to query what counters the hardware supports, to create monitors, to start and stop them, and to retrieve the values. It looks very similar to what we have described so far, except for an important difference: the user needs to start and stop the monitors explicitly. We will explain later why this is necessary. But the key point here is that when we start a monitor, this means that from that moment on, until stopping it, any job created and submitted to the kernel will have the identifier of that monitor attached. This implies that only <a href="https://gitlab.freedesktop.org/mesa/mesa/-/blob/685281278ebd39114c3007e76443eaaa66cf833/src/gallium/drivers/v3d/v3d_query_perfcnt.c#L209">one monitor</a> can be enabled in the application at the same time. But this isn’t a problem, as this restriction is part of the extension.</p> <p>Our driver does not implement this API directly, but through <a href="https://gitlab.freedesktop.org/mesa/mesa/-/blob/685281278ebd39114c3007e76443eaaa66cf833/src/gallium/drivers/v3d/v3d_query.c">“queries”</a>, which are used then by the Gallium subsystem in Mesa to implement the extension. For reference, the V3D driver (as well as the VC4) is implemented as part of the Gallium subsystem. The <a href="https://docs.mesa3d.org/gallium/index.html">Gallium</a> part basically handles all the hardware-independent OpenGL functionality, and just requires the driver hook functions to be implemented by the driver. If the driver implements the proper functions, then Gallium exposes the right extension (in this case, the GL_AMD_performance_monitor extension).</p> <p>For our case, it requires the driver to implement functions to return which <a href="https://gitlab.freedesktop.org/mesa/mesa/-/blob/685281278ebd39114c3007e76443eaaa66cf833/src/gallium/drivers/v3d/v3d_query_perfcnt.c#L160">counters are available</a>, to <a href="https://gitlab.freedesktop.org/mesa/mesa/-/blob/685281278ebd39114c3007e76443eaaa66cf833/src/gallium/drivers/v3d/v3d_query_perfcnt.c#L310">create</a> or <a href="https://gitlab.freedesktop.org/mesa/mesa/-/blob/685281278ebd39114c3007e76443eaaa66cf833/src/gallium/drivers/v3d/v3d_query_perfcnt.c#L183">destroy</a> a query (in this case, the query is the same as the monitor), <a href="https://gitlab.freedesktop.org/mesa/mesa/-/blob/685281278ebd39114c3007e76443eaaa66cf833/src/gallium/drivers/v3d/v3d_query_perfcnt.c#L202">start</a> and <a href="https://gitlab.freedesktop.org/mesa/mesa/-/blob/685281278ebd39114c3007e76443eaaa66cf833/src/gallium/drivers/v3d/v3d_query_perfcnt.c#L244">stop</a> the query, and once it is finished, <a href="https://gitlab.freedesktop.org/mesa/mesa/-/blob/685281278ebd39114c3007e76443eaaa66cf833/src/gallium/drivers/v3d/v3d_query_perfcnt.c#L272">to get the results back</a>.</p> <p>At this point, I would like to explain a bit better what it implies to stop the monitor and get the results back. As explained earlier, stopping the monitor or query means that from that moment on, any new job submitted to the kernel (and thus to the GPU) won’t contain a performance monitor identifier attached, and hence won’t be measured. But it is important to know that the driver submits jobs to the kernel to be executed at its own pace, but these aren’t executed immediatly; the GPU needs time to execute the jobs, and so the kernel puts the arriving jobs in a queue, to be submitted to the GPU. This means when the user stops the monitor, there could be still jobs in the queue that haven’t been executed yet and are thus pending to be measured.</p> <p>And how do we know that the jobs have been executed by the GPU? The hook function to implement getting the query results has a <a href="https://gitlab.freedesktop.org/mesa/mesa/-/blob/685281278ebd39114c3007e76443eaaa66cf833/src/gallium/drivers/v3d/v3d_query_perfcnt.c#L272">“wait” parameter</a>, which tells if the function <a href="https://gitlab.freedesktop.org/mesa/mesa/-/blob/685281278ebd39114c3007e76443eaaa66cf833/src/gallium/drivers/v3d/v3d_query_perfcnt.c#L282">needs to wait</a> for all the pending jobs to be measured to be executed or not. If it doesn’t but there are pending jobs, then it just returns telling the caller this fact. This allows to do other work meanwhile and query again later, instead of becoming blocked waiting for all the jobs to be executed. This is implemented through sync objects. Every time a job is sent to the kernel, there’s a <a href="https://gitlab.freedesktop.org/mesa/mesa/-/blob/685281278ebd39114c3007e76443eaaa66cf833/src/gallium/drivers/v3d/v3d_context.h#L516">sync object</a> that is used to signal when the job has finished executing. This is mainly used to have a way to synchronize the jobs. In our case, when the user finalizes the query we <a href="https://gitlab.freedesktop.org/mesa/mesa/-/blob/685281278ebd39114c3007e76443eaaa66cf833/src/gallium/drivers/v3d/v3d_query_perfcnt.c#L263">save this fence</a> for the last submitted job, and we use it to know when this last job has been executed.</p> <p>There are quite a few details I’m not covering here. If you are interested though, you can take a look at the <a href="https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10666">merge request</a>.</p> <h2 id="gallium-hud">Gallium HUD</h2> <p>So far we have seen how the performance counters are implemented, and how to use them. In all the cases it requires writing code to create the monitor/query, start/stop it, and querying back the results, either in the driver itself or in the application through the GL_AMD_performance_monitor extension<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup>.</p> <p>But what if we want to get some general measurements without adding code to the application or the driver? Fortunately, there is an <a href="https://docs.mesa3d.org/envvars.html">environmental variable</a> <code class="language-plaintext highlighter-rouge">GALLIUM_HUD</code> that, when correctly, will show on top of the application some graphs with the measured counters.</p> <p>Using it is very easy; set it to <code class="language-plaintext highlighter-rouge">help</code> to know how to use it, as well as to get a list of the available counters for the current hardware.</p> <p>As example:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ env GALLIUM_HUD=L2T-CLE-reads,TLB-quads-passing-z-and-stencil-test,QPU-total-active-clk-cycles-vertex-coord-shading scorched3d </code></pre></div></div> <p>You will see:</p> <p><img src="/jasuarez/assets/post_images/2021-09-01-v3d-gallium-hud.png" alt="Performance Counters in Scorched 3D" class="center-block" /></p> <p>Bear in mind that to be able to use this you will need a kernel that supports performance counters for V3D. At the moment of writing this, no kernel has been released yet with this support. If you don’t want to wait for it, you can download the <a href="https://cgit.freedesktop.org/drm/drm/commit/?id=26a4dc29b74a137f456">patch</a>, apply it to your <a href="https://github.com/raspberrypi/linux">raspberry pi kernel</a> (which has been tested in the 5.12 branch), <a href="https://www.raspberrypi.org/documentation/computers/linux_kernel.html#building">build and install it</a>.</p> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:1" role="doc-endnote"> <p>All this is for the case of using OpenGL; if your application uses Vulkan, there are other similar extensions, which are not yet implemented in our V3DV driver at the moment of writing this post. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> Wed, 01 Sep 2021 00:00:00 +0200 https://blogs.igalia.com/jasuarez/2021/09/01/v3d-perfcounters/ https://blogs.igalia.com/jasuarez/2021/09/01/v3d-perfcounters/ graphics raspberrypi Grilo, Travis CI and Containers <p>Good news! Finally, we are using containers in <a href="https://travis-ci.org">Travis CI</a> for <a href="https://wiki.gnome.org/Projects/Grilo">Grilo</a>!. Something I was trying for a while, but we achived it now. I must say that a <a href="https://www.bassi.io/articles/2017/02/11/epoxy">post Bassi wrote</a> was the trigger for getting into this. So all my kudos to him!</p> <p>In this post I’ll explain the history behind using Travis CI for Grilo continuous integration.</p> <h2 id="the-origin">The origin</h2> <p>It all started when one day exploring how GitHub integrates with other services, I discovered <a href="https://travis-ci.org">Travis CI</a>. As you may know, Travis is a continuous integration service that checks every commit from a project in GitHub, and for each one it starts a testing process. Roughly, it starts a “virtual machine”<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup> running Ubuntu<sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote" rel="footnote">2</a></sup>, clones the repository at that commit under test, and runs a set of commands defined in the <code class="language-plaintext highlighter-rouge">.travis.yml</code> file, located in the same project GitHub repository. In that file, beside the steps to execute the tests, it contains the instructions about how to build the project, as well as which dependencies are required.</p> <p>Note that before Travis, instead of a continuous integration system in Grilo we had a ‘discontinuous’ one: run the checks manually, from time to time. So we could have a commit entering a bug, and we won’t realize until we run the next check, which can happen way later. Thus, when I found Travis, I thought it would be a good idea to use it.</p> <p>Setting up <code class="language-plaintext highlighter-rouge">.travis.yml</code> for Grilo was quite easy: in the <code class="language-plaintext highlighter-rouge">before_install</code> section we just use <code class="language-plaintext highlighter-rouge">apt-get</code> to install all requirements: <code class="language-plaintext highlighter-rouge">libglib2.0-dev</code>, <code class="language-plaintext highlighter-rouge">libxml2-dev</code>, and so on. And then, in the <code class="language-plaintext highlighter-rouge">script</code> section we run <code class="language-plaintext highlighter-rouge">autogen.sh</code> and <code class="language-plaintext highlighter-rouge">make</code>. If nothing fails, we consider the test is successful. We do not run any specific test because we don’t have any in Grilo.</p> <p>For the plugins, the same steps: install dependencies, configure and build the plugins. In this case, we also run <code class="language-plaintext highlighter-rouge">make check</code>, so tests are run always. Again, if nothing fails Travis gives us a green light. Otherwise, a red one. The status is shown in the <a href="https://wiki.gnome.org/Projects/Grilo">main web page</a>. Also, if the test fail, an email is sent to the commit author.</p> <p>Now, this has a small problem when testing plugins: they require Grilo, and we were relying in the package provided by Ubuntu (it is listed in the dependencies). But what happens if the current commit is using a feature that was added in Grilo upstream, but not released yet? One option could be cloning Grilo core, building and installing it, before the plugins, and then compiling the plugins, depending on this version. This means that for each commit in plugins, we need to build two projects, adding lot of complexity in the Travis file. So we decided to go with a different approach: just create a Grilo package with the required unreleased Grilo core version (only for testing), and put it in a <a href="https://launchpad.net/~grilo-team/+archive/ubuntu/travis">PPA</a>. Then we can add that PPA in our <code class="language-plaintext highlighter-rouge">.travis.yml</code> file and use that version instead.</p> <p>A similar problem happens with Grilo itself: sometimes we require a specific version of a package that is not available in the Ubuntu version used by Travis (Ubuntu 12.04). So we need to backport it from a more recent Ubuntu version, and add it in the same PPA.</p> <p>Summing up, our <code class="language-plaintext highlighter-rouge">.travis.yml</code> files just add the PPA, install the required dependencies, build and test it. You can take a look at the <a href="https://git.gnome.org/browse/grilo/tree/.travis.yml?id=ce1fa94cc8759616a6aacfe94c09ffbf3432f7c0">core</a> and <a href="https://git.gnome.org/browse/grilo-plugins/tree/.travis.yml?id=f93e959f0243f5207dd23bbed21b8be20dfa76b4">plugins</a> file.</p> <h2 id="travis-and-the-peter-pan-syndrome">Travis and the Peter Pan syndrome</h2> <p>Time passes, we were adding more features, new plugins, fixing problem, adding new requirements or bumping up the required versions… but Travis continues using Ubuntu 12.04. My first thoughts were <em>“OK, maybe Travis wants to rely only in LTS releases”</em>. So we need to wait until the next LTS is released, and meanwhile backporting everything we need. No need to say that doing this becomes more and more complicated as time is passing. Sometimes backporting a single dependency requires to backport a lot of other dependencies, which can end up in a bloody nightmare. <em>“Only for a while, until the new LTS is released”</em>, repeated to myself.</p> <p>And good news! Ubuntu 14.04, the new LTS, is released. But you know what? Travis is not updated, and still uses the old LTS!. What the hell!</p> <p>Moreover, two years later after this release, Ubuntu 16.04 LTS is also released, and Travis still uses 12.04!</p> <p>At that moment, backporting were so complex that basically I gave up. And Continuous Integration was basically broken.</p> <h2 id="travis-and-the-containers">Travis and the containers.</h2> <p>And we were under this broken status until I read Travis was adding support for containers. <em>“This is what we need”</em>. But the truth is that even I knew that it would fix all the problems, I wasn’t very sure how to use the new feature. I tried several approaches, but I wasn’t happy with none of them.</p> <p>Until <a href="https://twitter.com/ebassi">Emmanuele Bassi</a> published a <a href="https://www.bassi.io/articles/2017/02/11/epoxy">post about using Meson in Epoxy</a>. That post included an explanation about using Docker containers in Travis, which solved all the doubts I had, and allowed me to finally move to use containers. So again, thank you, Emmanuele!</p> <p>What’s the idea? First, we have created a <a href="https://hub.docker.com/r/grilofw/grilo">Docker container</a> that has preinstalled all the requirements to build Grilo and the plugins. We tagged this image as <em>base</em>.</p> <p>When Travis is going to test Grilo, we instruct Travis to build a new container, based on <em>base</em>, that builds and installs Grilo. If everything goes fine, then our continous integration is successful, and Travis gives green light. Otherwise it gives red light. Exactly like it happened in the old approach.</p> <p>But we don’t stop here. If everything goes fine, we push the new container into Docker register, tagging it as <em>core</em>. Why? Because this is the image we will use for building the plugins.</p> <p>And in the case of plugins we do exactly the same as in the core. But this time, instead of relying in the <em>base</em> image, we rely in the <em>core</em> one. This way, we always use a version that has an up-to-date version of Grilo, so we don’t need to package it when introducing new features. Only if either Grilo or the plugins require a <strong>new dependency</strong> we need to build a new <em>base</em> image and push it. That’s all.</p> <p>Also, as a plus, instead of discarding the container that contains the plugins, we push it in Docker, tagged as <em>latest</em>. So anyone can just pull it with Docker to have a container to run and test Grilo and all the plugins.</p> <p>If interested, you can take a look at the <a href="https://git.gnome.org/browse/grilo/tree/.travis.yml?id=fcdcd29b1bc6aec03f57dac39b7b5a7df60c8cae">core</a> and <a href="https://git.gnome.org/browse/grilo-plugins/tree/.travis.yml?id=8a8f1a829cc222230ca16aa0a5f522dfb394225d">plugins</a> files to check how it looks like.</p> <p>Oh! Last but not least. This also helped us to test the building both using <a href="https://en.wikipedia.org/wiki/GNU_Build_System">Autotools</a> and <a href="http://mesonbuild.com">Meson</a>, both supported in Grilo. Which is really awesome.</p> <p>Summing up, moving to containers provides a lot of flexibility, and make things quite easier.</p> <p>Please, leave any comment or question either in <a href="https://www.facebook.com/jsuarezr/posts/1633824283311177">Facebook</a> or <a href="https://plus.google.com/+jasuarez/posts/df1d8wufjZj">Google+</a>.</p> <p><span style="color:green"><em>UPDATE (26/11/2017):</em> comments are available again.</span></p> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:1" role="doc-endnote"> <p>Let’s call Virtual Machine, container, whatever. In this context it doesn’t matter. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:2" role="doc-endnote"> <p>Ubuntu 12.04 LTS, to be exact. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> Thu, 09 Mar 2017 00:00:00 +0100 https://blogs.igalia.com/jasuarez/2017/03/09/grilo-travis-containers/ https://blogs.igalia.com/jasuarez/2017/03/09/grilo-travis-containers/ grilo multimedia New Year, New Blog! <p>A new year has come! And with the new year, the usual new proposals: be better person, do more exercise, blog more, … :smile:</p> <p>More than two years without blogging. Lot of time. So let’s start with the last proposal.</p> <p>But I also wanted to do a clean restart, and entirely reboot my blog. This is something I was thinking of during the last months, specially after <a href="https://blogs.igalia.com/mrego/">Rego</a> moved his blog to <a href="http://jekyllrb.com">Jekyll</a>. I really like how clean and simple it looks like.</p> <p>I have (or better had) my blog hosted in a WordPress server. <a href="https://wordpress.org">WordPress</a> is a very popular and also, why not, very good system to create and manage website content and blogs. It’s entirely an online service, where you create your content, and it’s served to the world. And there are tons of plugins that practically allow to do whatever you need.</p> <p>But to be honest, it seems too much cathedral for me. I just wanted something simpler, that serves the content I have, and nothing else. Specially, after reading lot of times about bugs and different security problems it has (fortunately, most of them fixed quickly). Nevertheless, I’m very lucky in this topic, because the WordPress server I was using is hosted by <a href="http://www.igalia.com">Igalia</a>, and their sysadmins are awesome professional people that keep everything updated. But still, can’t avoid the feeling that there’s still a risk, and that implies spending time from our sysadmins to maintain it.</p> <p>Besides that, there were other reasons that made me to consider a static blog system (again, these reasons are entirely for my personal case).</p> <ul> <li> <p>As said, too much big just to keep a simple blog. Using a static blog seems more suitable.</p> </li> <li> <p>I’m a FLOSS person, and as such, I really like to know what’s going on under the hood. And in this case it is something I don’t really know. Yes, WordPress is a free software, and source code is available out there. What I mean is that I can’t go to the server and freely change things, because I’m not the sysadmin. If I want to use a modified version of a plugin, I can’t do it. Basically, I see it as a service like others in the cloud, that you just use and full stop. On the other hand, static blogging is different: you have the source code, that you can inspect or modify (like I did for this blog), and that runs in your own host. And once the final content is generated, can be served by any webserver. No need to install anything at all in the server.</p> </li> <li> <p>I’m a developer person, and as such, I like the process of <em>writing</em> something in clear text, <em>compile</em> it, and get a <em>final result</em> I can use. Which is something that perfectly matches with the way of doing static blogging: you write your posts in clear text (usually, in <a href="http://daringfireball.net/projects/markdown">MarkDown</a> or <a href="http://docutils.sourceforge.net/rst.html">reStructuredText</a>), you run a <a href="https://www.staticgen.com">generator</a> which transforms the posts in HTML + CSS, and you get everything inside a directory. You only need to copy the generated content in the proper webserver. Nothing else.</p> </li> <li> <p>A very important one for me, that I think triggered the change from WordPress: I really like <a href="https://git-scm.com">Git</a>. And really missed to have my posts under with Git. But now, as posts are just clear text files, I can easily handle them with Git: I can push, amend, branch, and even accept fixes through pull requests!</p> </li> </ul> <p>Probably there are more reasons that made me to switch from WordPress. But those above are enough.</p> <p>So the next question was: which one? There are lot of different <a href="https://www.staticgen.com">static site generators</a>. Lot time ago I had done some shy attempts with <a href="https://blog.getpelican.com">Pelican</a> first, and with <a href="http://jekyllrb.com">Jekyll</a> later. As Jekyll is the most popular one, I decided to go with it. It has lot of plugins that covers all my needs, and very big community. For sure it is not the fastest one, but I don’t mind spending some minutes if required to generate content. The good thing is that in the future I can move to a different generator if needed, and just use the same posts.</p> <p>Once I decided to use Jekyll, a crucial question came up: which theme? Themes define how your content looks alike. For sure, I wanted something simple, like <a href="https://blogs.igalia.com/mrego/">Rego’s blog</a>, but not the same theme. I’m not a designer, so doing it myself from scratch was discarded. I could buy a theme from a professional designer, but I think it is too early to do that at this moment. Maybe in the future. Thus, I spent several days trying different free themes, checking how they look, until finding one that suited what I wanted: <a href="https://github.com/dirkfabisch/mediator">Mediator theme</a>. I used it as a starting point, fixing some problems (most of those fixes were merged in the original theme), doing some modifications to adapt to my own wishes, and voilà! What you see here is the final result.</p> <p>What’s next? Very likely, search for a way of allowing comments. This is not handled natively in Jekyll, but usually through third-party services, like <a href="https://disqus.com">Disqus</a> or <a href="http://www.discourse.org">Discourse</a>. I could use any of them, but again, I would be in the same situation as with WordPress.</p> <p>So for now, I’ll leave a couple of links in <a href="https://www.facebook.com/jsuarezr/posts/1573891925971080">Facebook</a> and <a href="https://plus.google.com/+jasuarez/posts/e2v3YD3mGbL">Google+</a> where people can leave comments.</p> <p>Happy new year!</p> <p><span style="color:green"><em>UPDATE (26/11/2017):</em> comments are available again.</span></p> Thu, 12 Jan 2017 00:00:00 +0100 https://blogs.igalia.com/jasuarez/2017/01/12/new-year-new-blog/ https://blogs.igalia.com/jasuarez/2017/01/12/new-year-new-blog/ Highlights in Grilo 0.2.11 (and Plugins 0.2.13) <p>Hello, readers!</p> <p>Some weeks ago we released a new version of <a href="https://wiki.gnome.org/Projects/Grilo">Grilo</a> and the Plugins set (yes, it sounds like a 70’s music group :smile:). You can read the announcement <a href="https://mail.gnome.org/archives/grilo-list/2014-August/msg00000.html">here</a> and <a href="https://mail.gnome.org/archives/grilo-list/2014-August/msg00001.html">here</a>. If you are more curious about all the detailed changes done, you can take a look at the Changelog <a href="https://download.gnome.org/sources/grilo/0.2/grilo-0.2.11.changes">here</a> and <a href="https://download.gnome.org/sources/grilo-plugins/0.2/grilo-plugins-0.2.13.changes">here</a>.</p> <p>But even when you can read that information in the above links, it is always a pleasure if someone highlights what are the main changes. So let’s go!</p> <h2 id="launch-tool">Launch Tool</h2> <p>Regarding the core system, among the typical bug fixes, I would highlight a new tool: <strong>grl-launch</strong>. This tool, as others, got inspiration from <a href="http://gstreamer.freedesktop.org">GStreamer</a> <a href="http://docs.gstreamer.com/display/GstSDK/gst-launch">gst-launch</a>. So far, when you wanted to do some operation in Grilo, like performing a search in YouTube or getting the title of a video on disk, the recommended way was using Grilo Test UI. This is a basic application that allows you to perform the typical operations in Grilo, like browsing or searching, and everthing from a graphical interface. The problem is that this tool is not flexible enough, so you can’t control all the details you could require. And it is also useful to visually check the results, but not to export the to manage with another tool.</p> <p>So while the Test UI is still very useful, to cover the other cases we have grl-launch. It is a command-line based tool that allows you to perform most of the operations allowed in Grilo, with a great degree of control. You can browse, search, solve details from a Grilo media element, …, with a great control: how many elements to skip or return, the metadata keys (title, author, album, …) to retrieve, flags to use, etc.</p> <p>And on top of that, the results can be exported directly to a <a href="http://en.wikipedia.org/wiki/Comma-separated_values">CSV</a> file so it can be loaded later in a spreadsheet.</p> <p>As example, getting the 10 first trailers from <a href="http://trailers.apple.com">Apple’s iTunes Movie</a> Trailers site:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ grl-launch-0.2 browse -c 10 -k title,url grl-apple-trailers 23 Blast,http://trailers.apple.com/movies/independent/23blast/23blast-tlr_h480p.mov A Most Wanted Man,http://trailers.apple.com/movies/independent/amostwantedman/amostwantedman-tlr1_h480p.mov ABC's of Death 2,http://trailers.apple.com/movies/magnolia_pictures/abcsofdeath2/abcsofdeath2-tlr3_h480p.mov About Alex,http://trailers.apple.com/movies/independent/aboutalex/aboutalex-tlr1b_h480p.mov Addicted,http://trailers.apple.com/movies/lionsgate/addicted/addicted-tlr1_h480p.mov "Alexander and the Terrible, Horrible, No Good, Very Bad Day",http://trailers.apple.com/movies/disney/alexanderterribleday/alexanderterribleday-tlr1_h480p.mov Annabelle,http://trailers.apple.com/movies/wb/annabelle/annabelle-tlr1_h480p.mov Annie,http://trailers.apple.com/movies/sony_pictures/annie/annie-tlr2_h480p.mov Are You Here,http://trailers.apple.com/movies/independent/areyouhere/areyouhere-tlr1_h480p.mov As Above / So Below,http://trailers.apple.com/movies/universal/asabovesobelow/asabovesobelow-tlr1_h480p.mov 10 results </code></pre></div></div> <p>As said, if you re-direct the output to a file and you import it from a spreadsheet program as CSV you will read it better.</p> <h2 id="dleynaupnp-plugin">dLeyna/UPnP plugin</h2> <p>Regarding the plugins, here is where the fun takes place. Almost all plugins were touched, in some way or other. In most cases, for fixing bugs. But there are other changes I’d like to highlight. And among them, UPnP is one that suffered biggest changes.</p> <p>Well, strictly speaking, there is no more UPnP plugin. Rather, it was replaced by new dLeyna plugin, written mainly by <a href="http://nerd.ocracy.org/em">Emanuele Aina</a>. From an user point of view, there shouldn’t be big differences, as this new plugin also provides access to UPnP/DLNA sources. So where are the differences?</p> <p>First off, let’s specify what is <a href="https://01.org/dleyna">dLeyna</a>. So far, if you want to interact with a UPnP source, either you need to deal with the protocol, or use some low-level library, like <a href="https://wiki.gnome.org/Projects/GUPnP"><em>gupnp</em></a>. This is what the UPnP plugin was doing. Still it is a rather low-level API, but higher and better than dealing with the raw protocol.</p> <p>On the other hand, dLeyna, written by the <a href="https://01.org">Intel Open Source Technology Center</a>, wraps the UPnP sources with a D-Bus layer. Actually,not only sources, but also UPnP media renderers and controllers, though in our case we are only interested in the UPnP sources. Thanks to dLeyna, you don’t need any more to interact with low-level UPnP, but with a higher D-Bus service layer. Similar to the way we interact with other services in GNOME or in other platforms. This makes easier to browser or search UPnP sources, and allows us to add new features. dLeyna also hides some details specific to each UPnP server that are of no interest for us, but we would need to deal with in case of using a lower level API. The truth is that though UPnP is quite well specified, each implementation doesn’t follow it at 100%: there are always slight differences that create nasty bugs. In this case, dLeyna acts (or should act) as a protection, dealing itself with those differences.</p> <p>And what is needed to use this new plugin? Basically, having dleyna-service D-Bus installed. When the plugin is started, it wakes up the service, which will expose all the available UPnP servers in the network, and the plugin would expose them as Grilo sources. Everything as it was happening with the previous UPnP source.</p> <p>In any case, I still keep a <a href="https://github.com/jasuarez/grilo-upnp-plugin">copy of the old UPnP plugin</a> for reference, in case someone want to use it or take a look. It is in “unmaintained” mode, so try to use the new dLeyna plugin instead.</p> <h2 id="lua-factory-plugin">Lua Factory plugin</h2> <p>There isn’t big changes here, except fixes. But I want to remark it here because it is where most activity is happening. I must thank <a href="http://www.hadess.net">Bastien</a> and <a href="http://www.victortoso.com">Victor</a> for the work they are doing here. Just to refresh, this plugin allows to execute sources written in <a href="http://www.lua.org">Lua</a>. That is, instead of writing your sources in GObject/C, you can use Lua. The Lua Factory plugin will load and run them. Writing plugins in Lua is a pleasure, as it allows to focus on fixing the real problems and leave the boiler plate details to the factory. Honestly, if you are considering writing a new source, I would really think about writing it in Lua.</p> <p>And that’s all! It is a longer post than usual, but it is nice to explain what’s going on in Grilo. And remember, if you are considering using Grilo in your product, don’t hesitate to <a href="http://www.igalia.com/contact">contact with us</a>.</p> Mon, 29 Sep 2014 00:00:00 +0200 https://blogs.igalia.com/jasuarez/2014/09/29/highlights-in-grilo-0-2-11/ https://blogs.igalia.com/jasuarez/2014/09/29/highlights-in-grilo-0-2-11/ grilo multimedia Another year, another GUADEC <p>It’s 2014, and like previous years:</p> <p><a href="https://2014.guadec.org"><img src="/jasuarez/assets/post_images/2014-07-24-guadec2014.png" alt="GUADEC 2014" class="center-block" /></a></p> <p>This time I won’t give any talk, just relax and enjoy <a href="http://www.igalia.com/nc/igalia-247/news/item/meet-us-at-guadec-2014-strasbourg-july-26-august-1">talks from others</a>, and hope <a href="http://www.strasbourg.eu">Strasbourg</a>.</p> <p>And what is more important, meet those hackers you interact with frequently, and maybe share some beers.</p> <p>So if you go there, and you want to have a nice chat with me, or talk about <a href="https://wiki.gnome.org/Projects/Grilo">Grilo</a> project, don’t hesitate to do it. <a href="http://www.igalia.com">Igalia</a>, which is kindly sponsoring my attendance, will have a place there during the core days, so likely you could find me around or ask anyone there for me.</p> <p>Enjoy!</p> Thu, 24 Jul 2014 00:00:00 +0200 https://blogs.igalia.com/jasuarez/2014/07/24/another-year-another-guadec/ https://blogs.igalia.com/jasuarez/2014/07/24/another-year-another-guadec/ guadec Yum Search Extended <p>Hi again! Let me tell you something. I’m a <a href="https://fedoraproject.org">Fedora</a> user since several releases ago, probably since Fedora 13 or 14.</p> <p>Before that, I was using <a href="https://fedoraproject.org">Ubuntu</a>, but decided to switch to Fedora for several reasons that are not worth explaining here. In any case, after switching to Fedora there was something that I was missing quite a lot: the <a href="https://wiki.debian.org/Aptitude">aptitude package manager</a>. aptitude is a deb package manager, similar to <a href="https://wiki.debian.org/Apt">apt</a>. What I really like about aptitude is its flexibility when searching packages.</p> <p>While apt or <a href="http://yum.baseurl.org">yum</a> allows to specify the search term, they just get all the packages matching the search text, but they don’t allow you where to search. Do you want to get only packages that are not installed? Or do you just remember the package had <em>python</em> in the name, and part of the description? With aptitude this is not a problem, as it allows you to specify such search expressions.</p> <p>Though search in yum is not so flexible, as far as I know, it has a nice feature: it allows <a href="http://yum.baseurl.org/wiki/WritingYumPlugins">plugins</a> to implement new features. So several months ago I wrote a plugin to mimic the aptitude search flexibility: <a href="https://github.com/jasuarez/yum-plugin-searchex"><strong>yum searchex (search extended)</strong></a>.</p> <p>It is worth saying that I didn’t want to imitate the full aptitude functionality; only those features that I really missed from Ubuntu.</p> <p>The basic idea is specifying for each term where to search. This is done by prefixing the text with <code class="language-plaintext highlighter-rouge">~</code> and a letter that expresses where to search. In some cases, the text to search is not needed. For instance, to search only in the list of installed packages, we would use <code class="language-plaintext highlighter-rouge">~i</code>.</p> <p>The full list of the available options can be found in the <a href="https://github.com/jasuarez/yum-plugin-searchex/blob/master/README.md">project forge</a>.</p> <p>As an example is worth a thousand words, let’s show how to search a package that we know it contains <em>python</em> in the name, it is not installed, and also we remember it has something to do with KDE:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>yum searchex ~apython~dKDE </code></pre></div></div> <p>Hope this plugin is as useful for you as it is for me!</p> Thu, 06 Mar 2014 00:00:00 +0100 https://blogs.igalia.com/jasuarez/2014/03/06/yum-search-extended/ https://blogs.igalia.com/jasuarez/2014/03/06/yum-search-extended/ fedora yum See you at GUADEC 2013! <p><a href="https://www.guadec.org">GUADEC</a> 2013 is around the corner.</p> <p><img src="/jasuarez/assets/post_images/2013-07-22-guadec2013.png" alt="GUADEC 2013" class="center-block" /></p> <p><a href="http://www.igalia.com">Igalia</a> is kindly sponsoring my attendance, as well as other mates, to this wonderful conference, where all years I meet good friends, and do new ones.</p> <p>I’ll be there from August, 1st to 5th, both included. On saturday 3rd I’ll give a talk about <a href="https://wiki.gnome.org/Grilo">Grilo</a>. If you are using Grilo or willing to, join us to the talk. Of course, I always welcome any question, so if you see me and want to ask anything, don’t hesitate to address me.</p> <p>Also, I expect to attend on 5th the <a href="https://wiki.gnome.org/Hackfests/Music2013">gnome-music BoF</a>. gnome-music is one of the programs I collaborate with that heavily use Grilo. I really suggest to give a try. I’ts so nice!</p> <p>Besides all above, Igalia will have a booth during all the event, where we will be showing some of the cool things we do. I still don’t know where it be exactly located, but if you see us, come there!</p> Mon, 22 Jul 2013 00:00:00 +0200 https://blogs.igalia.com/jasuarez/2013/07/22/see-you-at-guadec-2013/ https://blogs.igalia.com/jasuarez/2013/07/22/see-you-at-guadec-2013/ grilo guadec Grilo 0.2.8 released <p>I did a new <a href="https://mail.gnome.org/archives/grilo-list/2013-May/msg00017.html">release of grilo plugins</a> only one week after the <a href="https://mail.gnome.org/archives/grilo-list/2013-May/msg00012.html">previous release</a> because it includes a <a href="https://bugzilla.gnome.org/show_bug.cgi?id=700517">patch</a> that people from <a href="https://live.gnome.org/GnomePhotos">gnome-photos</a> would like to see in next <a href="http://www.gnome.org">GNOME</a> 3.9 pre-release.</p> <p>Besides that patch, this new release also includes a new plugin to get content from the <a href="http://magnatune.com">Magnatune</a> service. Credits go to <a href="http://www.victortoso.com">Victor Toso</a>, who did a great job on it.</p> <p>Happy weekend from <a href="http://www.igalia.com">Igalia</a> headquarter!</p> Sat, 25 May 2013 00:00:00 +0200 https://blogs.igalia.com/jasuarez/2013/05/25/grilo-plugins-0-2-8-released/ https://blogs.igalia.com/jasuarez/2013/05/25/grilo-plugins-0-2-8-released/ grilo multimedia What's going on in Grilo? <p>There’s a lot of time I don’t blog about <a href="https://live.gnome.org/Grilo">Grilo</a>. But it doesn’t mean we are not working on it! Here at <a href="http://www.igalia.com">Igalia</a> we do, and we also get lot of contributions from community. All the announcements are sent to the Grilo <a href="https://mail.gnome.org/mailman/listinfo/grilo-list">mailing list</a>.</p> <p>During this week we have released a new version of Grilo, both for core (v0.2.6) and for the plugins (v0.2.7). You can see the announcements <a href="https://mail.gnome.org/archives/grilo-list/2013-May/msg00011.html">here</a> and <a href="https://mail.gnome.org/archives/grilo-list/2013-May/msg00012.html">here</a>. If you are interested in more detailed changelogs, you can see them <a href="http://ftp.gnome.org/pub/GNOME/sources/grilo/0.2/grilo-0.2.6.changes">here</a> and <a href="http://ftp.gnome.org/pub/GNOME/sources/grilo-plugins/0.2/grilo-plugins-0.2.7.changes">here</a>.</p> <p>So what happens in this release?</p> <ul> <li> <p>As usual, lot of bugfixes.</p> </li> <li> <p><a href="http://www.hadess.net">Bastien</a> added support for non-file URIs in Filesystem plugin. What this means? In Filesystem plugin you can configure the base path from which the source can show content. Thus, if you setup the base path as <code class="language-plaintext highlighter-rouge">~/Music</code>, it means Filesystem will show only content placed in that directory. With the new approach, you could use as base-path something like <code class="language-plaintext highlighter-rouge">recent://</code>, getting the list of recently used items. Cool, uh?</p> </li> <li> <p>My mate <a href="http://blogs.igalia.com/svillar">Sergio</a> fixed the cache system in GrlNet. Seems we broke it at some point in the history, and we didn’t realize. We rely on libsoup and its caching feature, and Sergio is very skilled in libsoup. So who’s better than him to fix it? :grinning:</p> </li> <li> <p>We improved <code class="language-plaintext highlighter-rouge">grl-inspect</code> tool. Inspired in <code class="language-plaintext highlighter-rouge">gst-inspect</code> from <a href="http://gstreamer.freedesktop.org">GStreamer</a>, this tool helps to list all the available sources in the system, and list all the supported features. Now, it can also list all the available metadata keys developer can use. Moreover, it also list which sources support each key.</p> </li> <li> <p>We added support for i18n! Now we speak Brazilian, Czech, Galician, Greek, Polish, Serbian, Slovenian, Spanish and Tajik. And we hope to support more languages in the coming releases. Many thanks to <a href="https://l10n.gnome.org/users/rafaelff1">Rafael</a>, <a href="https://l10n.gnome.org/users/Marv">Marek</a>, <a href="https://l10n.gnome.org/users/frandieguez">Fran</a>, <a href="https://l10n.gnome.org/users/dmtrs32">Dimitris</a>, <a href="https://l10n.gnome.org/users/raven">Piotr</a>, <a href="https://l10n.gnome.org/users/MirosNik">Мирослав</a>, Martin, <a href="https://l10n.gnome.org/users/migueldemalaga">Miguel</a> and <a href="https://l10n.gnome.org/users/ibragimov">Victor</a> for their work on translating Grilo.</p> </li> <li> <p>Marek Chalupa added support for <a href="https://live.gnome.org/GnomeOnlineAccounts">GOA</a> in the Flickr plugin. This means that if you have a Flickr account configured in GOA, you will get for free a Flickr source dealing with that account. Nice!</p> </li> </ul> <p>I think that’s all. If you use Grilo and needs help or support, don’t hesitate to <a href="http://www.igalia.com/contact">contact us</a>!</p> Sat, 18 May 2013 00:00:00 +0200 https://blogs.igalia.com/jasuarez/2013/05/18/whats-going-on-in-grilo/ https://blogs.igalia.com/jasuarez/2013/05/18/whats-going-on-in-grilo/ grilo multimedia Grilo at GUADEC 2012 <p>A new year, a new <a href="http://guadec.org">GUADEC</a>. And as in previous years:</p> <p><img src="/jasuarez/assets/post_images/2012-07-19-guadec2012.png" alt="GUADEC 2012" class="center-block" /></p> <p>This year is a bit special for different reasons:</p> <ul> <li> <p>GUADEC 2012 is allocated in the same city as <a href="http://www.igalia.com">Igalia</a> headquarter. And in the same <a href="http://en.wikipedia.org/wiki/Galicia_(Spain)">region</a> I was born.</p> </li> <li> <p>I’ll give a <a href="https://www.gpul.org/indico/contributionDisplay.py?contribId=34&amp;confId=0">talk</a> about <a href="https://live.gnome.org/Grilo">Grilo</a>, reviewing its current state and talking about what’s coming next.</p> </li> <li> <p>There will be also a <a href="https://live.gnome.org/GUADEC/2012/BOFs">BoF</a> where we can talk for 2 hours about next features and what would be good to have in Grilo.</p> </li> </ul> <p>Of course, do not hesitate to ask me (or other Grilo contributors) whatever you want to know about Grilo if you see us around. We will be more han happy to answer any doubt.</p> Thu, 19 Jul 2012 00:00:00 +0200 https://blogs.igalia.com/jasuarez/2012/07/19/grilo-at-guadec-2012/ https://blogs.igalia.com/jasuarez/2012/07/19/grilo-at-guadec-2012/ grilo guadec