Herostratus’ legacy

Notes on hacking gfxreconstruct to enable H.265 video encoding

2026-07-03T00:00:00Z

While working on Vulkan Video encode support in Mesa, I needed to capture H.265 encoding traces. gfxreconstruct already handled H.264 video and several other extensions, but VK_KHR_video_encode_h265 was explicitly blocked. Here’s how I unblocked it and what I learned about gfxreconstruct’s code generation machinery along the way.

gfxreconstruct is LunarG’s suite of tools for capturing and replaying graphics API calls. It intercepts Vulkan (and D3D12) calls at the layer level, serializes them into a compressed binary trace file, and can later replay that trace verbatim. This is useful for driver regression testing, GPU bring up, architecture simulation, and bug reporting.

The blocking mechanism #

gfxreconstruct has two independent mechanisms that prevent an extension from being captured.

The first is a runtime blocklist in framework/encode/vulkan_entry_base.cpp. A static array called kVulkanUnsupportedDeviceExtensions lists extension name strings that the layer strips from vkEnumerateDeviceExtensionProperties results. If your extension is on that list, applications cannot even see it when the capture layer is loaded, so they never attempt to use it and nothing gets recorded.

The second is a generation-time exclusion list in the Python code generator. gfxreconstruct does not hand-write capture and replay handlers for each Vulkan function. Instead, it parses the Khronos XML registry (vk.xml, video.xml) and auto-generates thousands of lines of C++ for encoding, decoding, and consuming API calls. The generator has exclusion lists that tell it which extensions and struct families to skip entirely.

VK_KHR_video_encode_h265 was on both lists.

Removing the runtime block #

This part is trivial: open framework/encode/vulkan_entry_base.cpp and delete the line

VK_KHR_VIDEO_ENCODE_H265_EXTENSION_NAME,

from kVulkanUnsupportedDeviceExtensions. One line. After rebuilding, the layer reports the extension to applications. But that is not enough: without generated capture/replay code, intercepted calls would have no handlers.

Removing the generator exclusions #

The code generator lives under framework/generated/. The file khronos_generators/vulkan_generators/vulkan_base_generator.py maintains two exclusion lists:

_remove_extensions: extensions whose entire API surface the generator should skip.
_remove_video_extensions: lower-level codec standard headers (vulkan_video_codec_h265std*) that define the StdVideoH265* struct families.

From _remove_extensions I removed:

VK_KHR_video_decode_h265
VK_KHR_video_encode_h265
VK_KHR_video_maintenance2

From _remove_video_extensions I removed:

vulkan_video_codec_h265std
vulkan_video_codec_h265std_decode
vulkan_video_codec_h265std_encode

I included VK_KHR_video_maintenance2 because its structs interact with video session parameters relevant to H.265 sessions.

The XML patch: when the generator breaks #

After removing the exclusions, I ran the generator and hit an error. The generator could not resolve a len attribute in video.xml for StdVideoH265HrdParameters.

The problematic members are pSubLayerHrdParametersNal and pSubLayerHrdParametersVcl. Both are pointers, and video.xml specifies len="*_max_sub_layers_minus1 + 1" for them. The expression references *_max_sub_layers_minus1, a field that lives in an outer struct (VPS or SPS), not in StdVideoH265HrdParameters itself.

gfxreconstruct’s code generator resolves len expressions by walking the current struct’s members, but it cannot follow cross-struct references. This is a reasonable limitation: the generator would need to understand the full semantics of the Vulkan Video specification to know which outer struct provides the length field.

The fix is a small XML tree patch in gencode.py. Before the generator runs, I strip the len attribute from both members:

hrd_type = video_tree.find('types/type[@name="StdVideoH265HrdParameters"]')
if hrd_type is not None:
    for member_name in ('pSubLayerHrdParametersNal', 'pSubLayerHrdParametersVcl'):
        for member in hrd_type.findall('member'):
            name_elem = member.find('name')
            if name_elem is not None and name_elem.text == member_name:
                member.attrib.pop('len', None)

Without a len attribute, the generator falls back to treating each pointer as pointing to a single element. This is safe for practical capture scenarios where only one sub-layer is in use.

Regenerating the code #

With the generator changes in place, regenerating is a single command:

uv run --with pyparsing python3 framework/generated/generate_vulkan.py

uv run --with pyparsing ensures the pyparsing dependency is available without a manual pip install.

The generator overwrites all files under framework/generated/generated_vulkan_*.cpp and framework/generated/generated_vulkan_*.h. The diff was substantial: hundreds of new functions for encoding and decoding StdVideoH265* structs, plus all the video session parameter handling infrastructure.

The bigger picture #

The interesting part of this exercise was understanding gfxreconstruct’s architecture. The capture layer is not a monolithic block of hand-written interceptors. It is a generator pipeline: Python scripts consume the Khronos XML registry and emit C++ that handles serialization, deserialization, and replay for every struct and function in the Vulkan API surface.

This means enabling a new extension is mostly a matter of telling the generator to stop ignoring it, then fixing any edge cases where the XML description does not match the generator’s assumptions. The actual capture and replay logic comes for free once the generator produces code for the extension’s types and entry points.

If you are considering enabling other video extensions (H.264 encode is still blocked), the same recipe applies: remove from the runtime blocklist, remove from _remove_extensions and _remove_video_extensions, regenerate, and fix any XML len expression issues that surface.

Getting ready for the GStreamer Spring Hackfest 2026 in Nice

2026-05-22T00:00:00Z

Next week, several Igalians will be heading to Nice for a bunch of open source gatherings and hackfests.

I’ll attend the Media Summit remotely (I’m interested in the discussion about the usage of Vulkan Video API in embedded devices), then I’ll travel, as other multimedia folks, to the GStreamer Hackfest. While other Igalians will be at the Display Next Hackfest and Embedded Recipes.

My goal for the hackfest is to chat with other GStreamer developers about hardware-accelerated encoders and how to test them, especially those using VA-API and Vulkan.

Here’s what my multimedia colleagues are planning for the hackfest:

Stéphane Cerveau will talk about new features in GstPipelineStudio and its future integration with GstPrinceOfParser, along with several core GStreamer improvements.

Alicia Boya will tackle subtle timing issues, out-of-order and race conditions in frame processing, and other GStreamer bugs.

Xabier Rodríguez Calvar plans to close the issues around static compilation of gstreamer-rs, including selected plugins with system-deps.

Thibault Saunier will aim to finish upstreaming some new features and focus on WebAssembly support.

GStreamer Conference 2025

2025-11-20T00:00:00Z

The GStreamer Conference is an annual gathering that brings together developers, contributors, and users of the GStreamer multimedia framework. It serves as a platform for sharing knowledge, discussing the latest advancements, and fostering collaboration within the open-source multimedia community.

This year’s conference was held in London at the impressive Barbican Center, located within the Barbican Estate, a residential complex rebuilt after World War II in the brutalist architectural style.

Barbican Estate in the London City

It was a pleasure to meet in person with the colleagues, from different companies and backgrounds, I usually collaborate with remotely, sharing and discussing their projects and results.

Recently, UbiCast, which generously records and streams the conference, has uploaded all talks of this year conference to their site, exclusively for the GStreamer Conference.

In this blog post, I’ll share the talks delivered by my fellow igalians:

Animate Your Subtimelines in GES #

Direct link

GstVA and GStreamer-VAAPI updates #

Direct link

Time Remapping and GES: Implementation Details and Latest Updates #

Direct link

soothe: a proposal for encoder testing #

Direct link

GstWebRTC in WebKit, current status & plans #

Direct link

VVC/H.266 in GStreamer #

Direct link

Video Reshaping with Skia #

Direct link

Vulkan Video: pipeline update #

Direct link

Following the GStreamer Conference, we hosted our Autumn hackfest at Amazon’s offices in the City of London. This time I worked on GStreamer Vulkan.

This year, two other conferences typically held in the US, FOMS and Demuxed, also took place in London. I attended FOMS, where I discovered the vibrant MOQ project.

Finally, I’d like to thank Centricular for organizing the event, especially Tim-Philipp Müller, and even more particularly Igalia for sponsoring it and allowing me to participate in this project that’s close to my heart.

And that’s all, mates. Cheers!

Pint of Guinness

Summer updates

2025-07-25T00:00:00Z

Somehow I internalized that my duty as software programmer was to silently work in a piece of code as if it were a magnum opus, until it’s finished, and then release it to the world with no need of explanations, because it should speak for itself. In other words, I tend to consider my work as a form of art, and myself as an artist. But I’m not. There’s no magnus opus and there will never be one. I’m rather a craftsman, in the sense of Richard Sennett: somebody who cares about their craft, making small, quick but thoughtful and clean changes, here and there, hoping that they will be useful to someone, now and in the future. And those little efforts need to be exposed openly, in spaces as this one and social media, as if I were a bazaar merchant.

This reflection invites me to add another task to my duties as software programmer: a periodical exposition of the work done. And this is the first intent to forge a (monthly) discipline in that direction, not in the sense of bragging, or looking to overprice a product (in the sense of commodity fetishism), but to build bridges with those that might find useful those pieces of software.

Let’s start.

GStreamer YUV4MPEG2 encoder and decoder #

We have been working lately on video encoding, and we wanted an easy way to test our work, using common samples such as those shared by the Derf’s collection. They are in a file format known as YUV4MPEG2, or more commonly known as y4m, because of their file name extension.

YUV4MPEG2 is a simple file format designed to hold uncompressed frames of YUV video, formatted as YCbCr 4:2:0, YCbCr 4:2:2 or YCbCr 4:4:4 data for the purpose of encoding. Instead of using raw YUV streams, where the frame size and color format have to be provided out-of-band, these metadata are embedded in the file.

There were already GStreamer elements for encoding and decoding y4m streams, but y4mdec was in gst-plugins-bad while y4menc in gst-plugins-good.

Our first task was to fix and improve y4menc [!8654], then move y4mdec to gst-plugins-good [!8719], but that implied to rewrite the element and add unit tests, while add more features such as handling more color formats.

Soothe — video encoders testing framework #

Heavily inspired by Fluster, a testing framework written in Python for decoder conformance, we are sketching Soothe, a script that aims to be a testing framework for video encoders, using VMAF, a perceptual video quality assessment algorithm.

GStreamer Vulkan H.264 encoder #

This is the reason of the efforts expressed above: vulkanh264enc, a H.264 encoder using the Vulkan Video extension [!7197].

One interesting side of this task was to propose a base class for hardware accelerated H.264 encoders, based on the vah264enc, the GStreamer VA-API H.264 encoder. We talked about this base class in the GStreamer Conference 2024.

Now the H.264 encoder merged and it will be part of the future release of GStreamer 1.28.

Removal of GStreamer-VAAPI subproject #

We’re very grateful with GStreamer-VAAPI. When its maintenance were handed over to us, after a few months we got the privilege to merge it as an official GStreamer subproject.

Now GStreamer-VAAPI functionality has been replaced with the VA plugin in gst-plugins-bad. Still, it isn’t a full featured replacement [#3947], but it’s complete and stable enough to be widely deployed. As Tim said in the GStreamer Conference 2024: it just works.

So, GStreamer-VAAPI subproject has been removed from main branch in git repository [!9200], and its Gitlab project, archived.

Vulkan Video Status page #

We believe that Vulkan Video extension will be one of the main APIs for video encoding, decoding and processing. Igalia participate in the Vulkan Video Technical Sub Group (TSG) and helps with the Conformance Test Suite (CTS).

Vulkan Video extension is big and constantly updated. In order to keep track of it we maintain a web page with the latest news about the specification, proprietary drivers, open source drivers and open source applications, along with articles and talks about it.

https://vulkan-video-status.igalia.com

GStreamer Planet #

Last but not least, GStreamer Planet has been updated and overhauled.

Given that the old Planet script, written in Python 2, is unmaintained, we worked on a new one in Rust: planet-rs. It internally uses tera for templates, feed-rs for feed parsing, and reqwest for HTTP handling. The planet is generated using Gitlab scheduled CI pipelines.

https://gstreamer.freedesktop.org/planet

GStreamer 1.26 and Igalia

2025-05-29T00:00:00Z

The release of GStreamer 1.26, last March, delivered new features, optimization and improvements. Igalia played its role as long standing contributor, with 382 commits (194 merge requests) from a total of 2666 of commits merged in this release.This blog post takes a closer look on those contributions.

gst-devtools #

This module contains development and validation tools.

gst-dot-viewer #

gst-dot-viewer is a new web tool for real-time pipeline visualization. Our colleague, Thibault, wrote a blog post about its usage.

validate #

GstValidate is a tool to check if elements are behaving as expected.

Added support for HTTP Testing.
Scenario fixes such as reset pipelines on expected errors to avoid inconsistent states, improved error logging, and async action handling to prevent busy loops.

gst-editing-services #

GStreamer Editing Services is a library to simplify the creation of multimedia editing applications.

Enabled reverse playback, by adding a reverse property to nlesource for seamless backward clip playback.
Added internal tests for Non-Linear Engine elements.

gst-libav #

GStreamer Libav plug-in contains a set of many popular decoders and encoders using FFmpeg.

As part of the effort to support VVC/H.266 in GStreamer FFmpeg VVC/H.266 decoder was exposed.
Optimized framerate renegotiation in avviddec without decoder resets.
Mapped GST_VIDEO_FORMAT_GRAY10_LE16 format to FFmpeg’s equivalent.

gstreamer #

Core library.

Added a tracer for gst-dots-viewer.
Log tracers improvements such as, replaced integer codes with readable strings, to track pad’s sticky events, and simplify parameters handling, etc.
On pads, don’t push sticky events in response to a FLUSH_STOP event.
On queue element, fixed missing notify signals for level changes.
Pipeline parser now logs bus error messages during pipeline construction.
Fixed gst_util_ceil_log2 utility function.

gst-plugins-base #

GStreamer Base Plugins is a well-groomed and well-maintained collection of plugins. It also contains helper libraries and base classes useful for writing elements.

audiorate: respect tolerance property to avoid unnecessary sample adjustments for minor gaps.
audioconvert: support reordering of unpositioned input channels.
videoconvertscale: improve aspect ratio handling.
glcolorconvert: added I422_10XX, I422_12XX, Y444_10XX, and Y444_16XX color formats, and fixed caps negotiation for DMABuf.
glvideomixer: handle mouse events.
pbutils: added VVC/H.266 codec support
encodebasebin: parser fixes.
oggdemux: fixed seek to the end of files.
rtp: fixed precision for UNIX timestamp.
sdp: enhanced debugging messages.
parsebin: improved caps negotiation.
decodebin3: added missing locks to prevent race conditions.
streamsynchronizer: improved documentation.

gst-plugins-good #

GStreamer Good Plugins is a set of plugins considered to have good quality code, correct functionality, and uses LGPL/LGPL+compatible licenses.

hlsdemux2: handle empty segments at the beginning of a stream.
qtmux and matroska: add support for VVC/H.266.
matroskademux:support seek with stop in push mode.
rtp: several fixes.
osxaudio: fixes.
videoflip: support Y444_16LE and Y444_16BE color formats.
vpx: enhance error and warning messages.

gst-plugins-bad #

GStreamer Bad Plug+ins is a set of plugins that aren’t up to par compared to the rest. They might be close to being good quality, but they’re missing something, be it a good code review, some documentation, a set of tests, etc.

dashsink: a lot of improvements and cleanups, such as unit tests, state and event management.
h266parse: enabled vvc1 and vvi1 stream formats, improved code data parsing and negotiatios, along with cleanups and fixes.
mpegtsmux and tsdemux: added support for VVC/H.266 codec.
vulkan:
- Added compatibility for timeline semaphores and barriers.
- Initial support of multiple GPU and dynamic element registering.
- Vulkan image buffer pool improvements.
- vulkanh264dec: support interlaced streams.
- vulkanencoding: rate control and quality level adjustments, update SPS/PPS, support layered DPBs.
webrtcbin:
- Resolved duplicate payload types in SDP offers with RTX and multiple codecs.
- Transceivers are now created earlier during negotiation to avoid linkage issues.
- Allow session level in setup attribute in SDP answer.
wpevideosrc:
- code cleanups
- cached SHM buffers are cleared after caps renegotiation.
- handle latency queries and post progress messages on bus.
srtdec: fixes
jpegparse: handle avi1 tag for progressive images
va: improve encoders configuration when properties change in run+time, specially rate control.

Using pre-commit in GStreamer

2025-03-12T00:00:00Z

Recently, GStreamer development story integrated the usage of pre-commit. pre-commit is a Git hook script that chain different linters, checkers, validators, formatters, etc., that are executed at git commit. This script is in Python. And there’s other GStreamer utility in Python: hotdoc

The challenge is that Debian doesn’t allow to install Python packages through pip, they have to be installed as Debian packages or inside virtual environments, such as venv.

So, instead of activating a virtual environment when I work in GStreamer, let’s just use direnv to activate it automatically.

Here’s a screencast of what I did to setup a Python virtual environment, within direnv, and installing pre-commit, hotdoc and gst-indent-1.0.

UPDATE: Tim told me that wit pipx we can do the same without the venv hassle.

https://mastodon.social/@tp_muller@fosstodon.org/114150178786863565

Igalia Multimedia achievements (2024)

2024-12-18T00:00:00Z

As 2024 draws to a close, it’s a perfect time to reflect on the year’s accomplishments done by the Multimedia team in Igalia. In our consideration, there were three major achievements:

WebRTC’s support in WPE/WebKitGTK with GStreamer.
GES maturity improved for real-life use-cases.
Vulkan Video development and support in GStreamer.

WebRTC support in WPE/WebKitGTK with GStreamer #

WPE and WebKitGTK are WebKit ports maintained by Igalia, the former for embedded devices and the latter for applications with a full-featured Web integration.

WebRTC is a web API that allows real-time communication (RTC) directly between web browser and applications. Examples of these real-time communications are video conferencing, cloud gaming, live-streaming, etc.

Some WebKit ports support libwebrtc, an open-source library that implements the WebRTC specification, developed and maintained by Google. WPE and WebKitGTK originally also supports libwebrtc, but we started to use also GstWebRTC, a set of GStreamer plugins and libraries that implement WebRTC, which adapts perfectly to the multimedia implementation in both ports, also in GStreamer.

This year the fruits of this work have been unlocked by enabling Amazon Luna gaming:

https://www.youtube.com/watch?v=lyO7Hqj1jMs

And also enabling a CAD modeling, server-side rendered service, known as Zoo:

https://www.youtube.com/watch?v=CiuYjSCDsUM

WebKit Multimedia #

WebKit made significant improvements in multimedia handling, addressing various issues to enhance stability and playback quality. Key updates include preventing premature play() calls during seeking, fixing memory leaks. The management of track identifiers was also streamlined by transitioning from string-based to integer-based IDs. Additionally, GStreamer-related race conditions were resolved to prevent hangs during playback state transitions. Memory leaks in WebAudio and event listener management were addressed, along with a focus on memory usage optimizations.

The handling of media buffering and seeking was enhanced with buffering hysteresis for smoother playback. Media Source Extensions (MSE) behavior was refined to improve playback accuracy, such as supporting markEndOfStream() before appendBuffer() and simplifying playback checks. Platform-specific issues were also tackled, including AV1 and Opus support for encrypted media and better detection of audio sinks. And other improvements on multimedia performance and efficiency.

GES maturity improved for real-live use-cases #

GStreamer Editing Services (GES) is a set of GStreamer plugins and a library that allow non-linear video editing. For example, GES is what’s behind of Pitivi, the open source video editor application.

Last year, GES was deployed in web-based video editors, where the actual video processing is done server-side. These projects allowed, in great deal, the enhancement and maturity of the library and plugins.

Tella is a browser-based tool that allow to screen record and webcam, without any extra software. Finished the recording, the user can edit, in the browser, the video and publish it.

https://www.youtube.com/watch?v=uSWqWHBRDWE

Sequence is a complete, browser-based, video editor with collaborative features. GES is used in the backend to render the editing operations.

https://www.youtube.com/watch?v=bXNdDIiG9lE

Vulkan Video development and GStreamer support #

The last but not the least, this year we continue our work with the Vulkan Video ecosystem by working the task subgroup (TSG) enabling H.264/H.265 encoding, and AV1 decoding and encoding.

Early this year we delivered a talk in the Vulkanised about our work, which ranges from the Conformance Test Suite (CTS), Mesa, and GStreamer.

https://www.youtube.com/watch?v=z1HcWrmdwzI

Conclusion #

As we wrap up 2024, it’s clear that the year has been one of significant progress, driven by innovation and collaboration. Here’s to continuing the momentum and making 2025 even better!

GStreamer Conference 2024

2024-10-23T00:00:00Z

Early this month I spent a couple weeks in Montreal, visiting the city, but mostly to attend the GStreamer Conference and the following hackfest, which happened to be co-located with the XDC Conference. It was my first time in Canada and I utterly enjoyed it. Thanks to all those from the GStreamer community that organized and attended the event.

For now you can replay the whole streams of both days of conference, but soon the split videos for each talk will be published:

GStreamer Conference 2024 - Day 1, Room 1 - October 7, 2024

https://www.youtube.com/watch?v=KLUL1D53VQI

GStreamer Conference 2024 - Day 1, Room 2 - October 7, 2024

https://www.youtube.com/watch?v=DH64D_6gc80

GStreamer Conference 2024 - Day 2, Room 2 - October 8, 2024

https://www.youtube.com/watch?v=jt6KyV757Dk

GStreamer Conference 2024 - Day 2, Room 1 - October 8, 2024

https://www.youtube.com/watch?v=W4Pjtg0DfIo

And a couple pictures of igalians :)

GStreamer Skia plugin

GStreamer VA

GstWebRTC

GstVulkan

gstreamer editing services

GStreamer Vulkan Operation API

2024-06-06T00:00:00Z

Two weeks ago the GStreamer Spring Hackfest took place in Thessaloniki, Greece. I had a great time. I hacked a bit on VA, Vulkan and my toy, planet-rs, but mostly I ate delicious Greek food ☻. A big thanks to our hosts: Vivia, Jordan and Sebastian!

First day of the GStreamer Spring Hackfest 2024 - https://floss.social/@gstreamer/112511912596084571

And now, writing this supposed small note, I recalled that I have in my to-do list an item to write a comment about GstVulkanOperation, an addition to GstVulkan API which helps with the synchronization of operations on frames, in order to enable Vulkan Video.

Originally, GstVulkan API didn’t provide almost any synchronization operation, beside fences, and that appeared to be enough for elements, since they do simple Vulkan operations. Nonetheless, as soon as we enabled VK_VALIDATION_FEATURE_ENABLE_SYNCHRONIZATION_VALIDATION_EXT feature, which reports resource access conflicts due to missing or incorrect synchronization operations between action [*], a sea of hazard operation warnings drowned us [*].

Hazard operations are a sequence of read/write commands in a memory area, such as an image, that might be re-ordered, or racy even.

Why are those hazard operations reported by the Vulkan Validation Layer, if the programmer pushes the commands to execute in queue in order? Why is explicit synchronization required? Because, as the great blog post from Hans-Kristian Arntzen, Yet another blog explaining Vulkan synchronization, (make sure you read it!) states:

[…] all commands in a queue execute out of order. Reordering may happen across command buffers and even vkQueueSubmits

In order to explain how synchronization is done in Vulkan, allow me to yank a couple definitions stated by the specification:

Commands are instructions that are recorded in a device’s queue. There are four types of commands: action, state, synchronization and indirection. Synchronization commands impose ordering constraints on action commands, by introducing explicit execution and memory dependencies.

Operation is an arbitrary amount of commands recorded in a device’s queue.

Since the driver can reorder commands (perhaps for better performance, dunno), we need to send explicit synchronization commands to the device’s queue to enforce a specific sequence of action commands.

Nevertheless, Vulkan doesn’t offer fine-grained dependencies between individual operations. Instead, dependencies are expressed as a relation of two elements, where each element is composed by the intersection of scope and operation. A scope is a concept in the specification that, in practical terms, can be either pipeline stage (for execution dependencies), or both pipeline stage and memory access type (for memory dependencies).

First let’s review execution dependencies through pipeline stages:

Every command submitted to a device’s queue goes through a sequence of steps known as pipeline stages. This sequence of steps is one of the very few implicit ordering guarantees that Vulkan has. Draw calls, copy commands, compute dispatches, all go through certain sequential stages, which amount of stages to cover depends on the specific command and the current command buffer state.

In order to visualize an abstract execution dependency let’s imagine two compute operations and the first must happen before the second.

Operation 1
Sync command
Operation 2

The programmer has to specify the Sync command in terms of two scopes (Scope 1 and Scope 2), in this execution dependency case, two pipeline stages.
The driver generates an intersection between commands in Operation 1 and Scope 1 defined as Scoped operation 1. The intersection contains all the commands in Operation 1 that go through up to the pipeline stage defined in Scope 1. The same is done with Operation 2 and Scope 2 generating Scoped operation 2.
Finally, we got an execution dependency that guarantees that Scoped operation 1 happens before Scoped operation 2.

Now let’s talk about memory dependencies:

First we need to understand the concepts of memory availability and visibility. Their formal definition in Vulkan are a bit hard to grasp since they come from the Vulkan memory model, which is intended to abstract all the ways of how hardware access memory. Perhaps we could say that availability is the operation that assures the existence of the required memory; while visibility is the operation that assures it’s possible to read/write the data in that memory area.

Memory dependencies are limited the Operation 1 that be done before memory availability and Operation 2 that have to be done after its visibility.

But again, there’s no fine-grained way to declare that memory dependency. Instead, there are memory access types, which are functions used by descriptor types, or functions for pipeline stage to access memory, and they are used as access scopes.

All in all, if a synchronization command defining a memory dependency between two operations, it’s composed by the intersection of between each command and a pipeline stage, intersected with the memory access type associated with the memory processed by those commands.

Now that the concepts are more or less explained we could see those concepts expressed in code. The synchronization command for execution and memory dependencies is defined by VkDependencyInfoKHR. And it contains a set of barrier arrays, for memory, buffers and images. Barriers express the relation of dependency between two operations. For example, Image barriers use VkImageMemoryBarrier2 which contain the mask for source pipeline stage (to define Scoped operation 1), and the mask for the destination pipeline stage (to define Scoped operation 2); the mask for source memory access type and the mask for the destination memory access to define access scopes; and also layout transformation declaration.

A Vulkan synchronization example from Vulkan Documentation wiki:

vkCmdDraw(...);

... // First render pass teardown etc.

VkImageMemoryBarrier2KHR imageMemoryBarrier = {
  ...
  .srcStageMask = VK_PIPELINE_STAGE_2_FRAGMENT_SHADER_BIT_KHR,
  .dstStageMask = VK_PIPELINE_STAGE_2_COLOR_ATTACHMENT_OUTPUT_BIT_KHR,
  .dstAccessMask = VK_ACCESS_2_COLOR_ATTACHMENT_WRITE_BIT_KHR,
  .oldLayout = VK_IMAGE_LAYOUT_READ_ONLY_OPTIMAL,
  .newLayout = VK_IMAGE_LAYOUT_ATTACHMENT_OPTIMAL
  /* .image and .subresourceRange should identify image subresource accessed */};

VkDependencyInfoKHR dependencyInfo = {
    ...
    1,                      // imageMemoryBarrierCount
    &imageMemoryBarrier,    // pImageMemoryBarriers
    ...
}

vkCmdPipelineBarrier2KHR(commandBuffer, &dependencyInfo);

... // Second render pass setup etc.

vkCmdDraw(...);

First draw samples a texture in the fragment shader. Second draw writes to that texture as a color attachment.

This is a Write-After-Read (WAR) hazard, which you would usually only need an execution dependency for - meaning you wouldn’t need to supply any memory barriers. In this case you still need a memory barrier to do a layout transition though, but you don’t need any access types in the src access mask. The layout transition itself is considered a write operation though, so you do need the destination access mask to be correct - or there would be a Write-After-Write (WAW) hazard between the layout transition and the color attachment write.

Other explicit synchronization mechanisms, along with barriers, are semaphores and fences. Semaphores are a synchronization primitive that can be used to insert a dependency between operations without notifying the host; while fences are a synchronization primitive that can be used to insert a dependency from a queue to the host. Semaphores and fences are expressed in the VkSubmitInfo2KHR structure.

As a preliminary conclusion, synchronization in Vulkan is hard and a helper API would be very helpful. Inspired by FFmpeg work done by Lynne, I added GstVulkanOperation object helper to GStreamer Vulkan API.

GstVulkanOperation object helper aims to represent an operation in the sense of the Vulkan specification mentioned before. It owns a command buffer as public member where external commands can be pushed to the associated device’s queue.

It has a set of methods:

gst_vulkan_operation_begin wraps vkBeginCommandBuffer.
gst_vulkan_operation_end wraps both vkEndCommandBuffer and either vkQueueSubmit or vkQueueSubmit2 (the object internally handle both depending on the driver support).
gst_vulkan_operation_wait wraps vkWaitForFences.

Internally, GstVulkanOperation contains two arrays:

The array of dependency frames, which are the set of frames, each representing an operation, which will hold dependency relationships with other dependency frames.

gst_vulkan_operation_add_dependency_frame appends frames to this array.

When calling gst_vulkan_operation_end the frame’s barrier state for each frame in the array is updated.

Also, each dependency frame creates a timeline semaphore, which will be signaled when a command, associated with the frame, is executed in the device’s queue.
The array of barriers, which contains a list of synchronization commands. gst_vulkan_operation_add_frame_barrier fills and appends a VkImageMemoryBarrier2KHR associated with a frame, which can be in the array of dependency frames.

Here’s a generic view of video decoding example:

gst_vulkan_operation_begin (priv->exec, ...);

cmd_buf = priv->exec->cmd_buf->cmd;

gst_vulkan_operation_add_dependency_frame (exec, out,
    VK_PIPELINE_STAGE_2_VIDEO_DECODE_BIT_KHR,
    VK_PIPELINE_STAGE_2_VIDEO_DECODE_BIT_KHR);

/* assume a engine where out frames can be used for DPB frames, */
/* so a barrier for layout transition is required */
gst_vulkan_operation_add_frame_barrier (exec, out,
    VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT,
    VK_ACCESS_2_VIDEO_DECODE_WRITE_BIT_KHR,
    VK_IMAGE_LAYOUT_VIDEO_DECODE_DPB_KHR, NULL);

for (i = 0; i < dpb_size; i++) {
  gst_vulkan_operation_add_dependency_frame (exec, dpb_frame,
      VK_PIPELINE_STAGE_2_VIDEO_DECODE_BIT_KHR,
      VK_PIPELINE_STAGE_2_VIDEO_DECODE_BIT_KHR);
}

barriers = gst_vulkan_operation_retrieve_image_barriers (exec);
vkCmdPipelineBarrier2 (cmd_buf, &(VkDependencyInfo) {
    ...
    .pImageMemoryBarriers = barriers->data,
    .imageMemoryBarrierCount = barriers->len,
});
g_array_unref (barriers);

vkCmdBeginVideoCodingKHR (cmd_buf, &decode_start);
vkCmdDecodeVideoKHR (cmd_buf, &decode_info);
vkCmdEndVideoCodingKHR (cmd_buf, &decode_end);

gst_vulkan_operation_end (exec, ...);

Here, just one memory barrier is required for memory layout transition, but semaphores are required to signal when an output frame and its DPB frames are processed, and later, the output frame can be used as a DPB frame. Otherwise, the output frame might not be fully reconstructed with it’s used as DPB for the next output frame, generating only noise.

And that’s all. Thank you.

GStreamer Hackfest 2024

2024-05-24T00:00:00Z

Last weeks were a bit hectic. First, with a couple friends we biked the southwest of the Netherlands for almost a week. The next week, the last one, I attended the 2024 Display Next Hackfest

https://mastodon.social/@igalia@floss.social/112438486139416587

This week was Igalia’s Assembly meetings, and next week, along with other colleagues, I’ll be in Thessaloniki for the GStreamer Spring Hackfest

https://mastodon.social/@gstreamer@floss.social/112473279155437759

I’m happy to meet again friends from the GStreamer community and talk and move things forward related with Vulkan, VA-API, KMS, video codecs, etc.

GstVA library in GStreamer 1.22 and some new features in 1.24

2024-02-13T00:00:00Z

I know, it’s old news, but still, I was pending to write about the GstVA library to clarify its purpose and scope.I didn’t want to have a library for GstVA, because to maintain a library is a though duty. I learnt it in the bad way with GStreamer-VAAPI. Therefore, I wanted GstVA as simple as possible, direct in its libva usage, and self-contained.

As far as I know, the main usage of GStreamer-VAAPI library was to have access to the VASurface from the GstBuffer, for example, with appsink. Since the beginning of GstVA, a mechanism to access the surface ID was provided, as `GStreamer OpenGL** offers access to the texture ID: through a flag when mapping a GstGL memory backed buffer. You can see how this mechanism is used in the one of the GstVA sample apps.

Nevertheless, later another use case appeared which demanded a library for GstVA: Other GStreamer elements needed to produce or consume VA surface backed buffers. Or to say it more concretely: they needed to use the GstVA allocator and buffer pool, and the mechanism to share the GstVA context along the pipeline too. The plugins with those VA related elements are msdk and qsv, when they operate on Linux. Both elements use Intel OneVPL, so they are basically competitors. The main difference is that the first is maintained by Intel, and has more features, specially for Linux, whilst the former in maintained by Seungha Yang, and it appears to be better tested in Windows.

These are the objects exposed by the GstVA library API:

GstVaDisplay #

GstVaDisplay represents a VADisplay, the interface between the application and the hardware accelerator. This class is abstract, and it’s supposed not to be instantiated. Instantiation has to go through it derived classes, such as

GstVaDisplayDRM, for DRM devices.
GstVaDisplayWrapped, for user-injected display connections, for example, VA X11 or Wayland connection, through the GstContext bus message GST_MESSAGE_NEED_CONTEXT. See the mentioned example.
The experimental VA backend for Windows: GstVaDisplayWin32, merged for the next release 1.24.

This class is shared among all the elements in the pipeline via GstContext, so all the plugged elements share the same connection the hardware accelerator. Unless the plugged element in the pipeline has a specific name, as in the case of multi GPU systems.

Let’s talk a bit about multi GPU systems. This is the gst-inspect-1.0 output of a system with an Intel and an AMD GPU, both with VA support:

$ gst-inspect-1.0 va
Plugin Details:
  Name                     va
  Description              VA-API codecs plugin
  Filename                 /home/igalia/vjaquez/gstreamer/build/subprojects/gst-plugins-bad/sys/va/libgstva.so
  Version                  1.23.1.1
  License                  LGPL
  Source module            gst-plugins-bad
  Documentation            https://gstreamer.freedesktop.org/documentation/va/
  Binary package           GStreamer Bad Plug-ins git
  Origin URL               Unknown package origin

  vaav1dec: VA-API AV1 Decoder in Intel(R) Gen Graphics
  vacompositor: VA-API Video Compositor in Intel(R) Gen Graphics
  vadeinterlace: VA-API Deinterlacer in Intel(R) Gen Graphics
  vah264dec: VA-API H.264 Decoder in Intel(R) Gen Graphics
  vah264lpenc: VA-API H.264 Low Power Encoder in Intel(R) Gen Graphics
  vah265dec: VA-API H.265 Decoder in Intel(R) Gen Graphics
  vah265lpenc: VA-API H.265 Low Power Encoder in Intel(R) Gen Graphics
  vajpegdec: VA-API JPEG Decoder in Intel(R) Gen Graphics
  vampeg2dec: VA-API Mpeg2 Decoder in Intel(R) Gen Graphics
  vapostproc: VA-API Video Postprocessor in Intel(R) Gen Graphics
  varenderD129av1dec: VA-API AV1 Decoder in AMD Radeon Graphics in renderD129
  varenderD129av1enc: VA-API AV1 Encoder in AMD Radeon Graphics in renderD129
  varenderD129compositor: VA-API Video Compositor in AMD Radeon Graphics in renderD129
  varenderD129deinterlace: VA-API Deinterlacer in AMD Radeon Graphics in renderD129
  varenderD129h264dec: VA-API H.264 Decoder in AMD Radeon Graphics in renderD129
  varenderD129h264enc: VA-API H.264 Encoder in AMD Radeon Graphics in renderD129
  varenderD129h265dec: VA-API H.265 Decoder in AMD Radeon Graphics in renderD129
  varenderD129h265enc: VA-API H.265 Encoder in AMD Radeon Graphics in renderD129
  varenderD129jpegdec: VA-API JPEG Decoder in AMD Radeon Graphics in renderD129
  varenderD129postproc: VA-API Video Postprocessor in AMD Radeon Graphics in renderD129
  varenderD129vp9dec: VA-API VP9 Decoder in AMD Radeon Graphics in renderD129
  vavp9dec: VA-API VP9 Decoder in Intel(R) Gen Graphics

  22 features:
  +-- 22 elements

As you can observe, each card registers the supported element. The first GPU, the Intel one, doesn’t insert renderD129 after the va prefix, while the AMD Radeon, does. As you could imagine, renderD129 is the device name in /dev/dri:

$ ls /dev/dri
by-path  card0  card1  renderD128  renderD129

The appended device name expresses the DRM device the elements use. And only after the second GPU the device name is appended. This allows to use the untagged elements either for the first GPU or to allow the usage of a wrapped display injected by the user application, such as VA X11 or Wayland connections.

Notice that sharing DMABuf-based buffers between different GPUs is theoretically possible, but not assured.

Keep in mind that, currently, nvidia-vaapi-driver is not supported by GstVA, and the driver will be ignored by the plugin register since 1.24.

VA allocators #

There are two types of memory allocators in GstVA:

VA surface allocator. It allocates a GstMemory that wraps a VASurfaceID, which represents a complete frame directly consumable by VA-based elements. Thus, a GstBuffer will hold one and only one GstMemory of this type. These buffers are the very same used by the system memory buffers, since the surfaces generally can map their content to CPU memory (unless they are encrypted, but GstVA haven’t been tested for that use case).
VA-DMABuf allocator. It descends from GstDmaBufAllocator, though it’s not an allocator in strict sense, since it only imports DMABufs to VA surfaces, and exports VA surfaces to DMABufs. Notice that a single VA surface can be exported as multiple DMABuf-backed GstMemories, and the contrary, a VA surface can be imported from multiple DMABufs.

Also, VA allocators offer a couple methods that can be useful for applications that use appsink, for example gst_va_buffer_get_surface and gst_va_buffer_peek_display.

One particularity of both allocators is that they keep an internal pool of the allocated VA surfaces, since the mapping of buffers/memories is not always 1:1, as in the case of DMABuf; to reuse surfaces even if the buffer pool ditches them, and to avoid surface leaks, keeping track of them all the time.

GstVaPool #

This class is only useful for other GStreamer elements capable of consume VA surfaces, so they could instantiate, configure and propose a VA buffer pool, but not for applications.

And that’s all for now. Thank you for bear with me up to here. But remember, the API of the library is unstable, and it can change any time. So, no promises are made :)

GStreamer Conference 2023

2023-10-31T00:00:00Z

This year the GStreamer Conference happened in A Coruña, basically at home, along with the hackfest.The conference was the first after a long hiatus of four years of pandemics. The community craved it and long expected it. Some igalians helped to the GStreamer Foundation and our warm community with the organization and logistics. I’m very thankful with my peers and the sponsors of the event. Personally, I’m happy with the outcome. Though, I ought to say, organizing a conference like this is quite a challenge and very demanding.

Marina from Palexco

The conference were recorded and streamed by Ubicast. And you can watch any presentation of the conference in their GStreamer Archive.

Tim sharing the State of the Union

Lunch time in Palexco

This is the list of talks where fellow Igalians participated:

There were two days of conference. The following two were for the hackfest, at Igalia’s Head Quarters.

Day one of the Hackfest

DMABuf modifier negotiation in GStreamer

2023-08-08T00:00:00Z

It took almost a year of design and implementation but finally the DMABuf modifier negotiation in GStreamer is merged. Big kudos to all the people involved but mostly to He Junyan, who did the vast majority of the code.## What’s a DMAbuf modifier?

DMABuf are the Linux kernel mechanism to share buffers among different drivers or subsystems. A particular case of DMABuf are the DRM PRIME buffers which are buffers shared by the Display Rendering Manager (DRM) subsystem. They allowed sharing video frames between devices with zero copy.

When we initially added support for DMABuf in GStreamer, we assumed that only color format and size mattered, just as old video frames stored in system memory. But we were wrong. Beside color format and size, also the memory layout has to be considered when sharing DMABufs. By not considering it, the produced output had horrible tiled artifacts in screen. This memory layout is known as modifier, and it’s uniquely described by an uint64 number.

How was designed and implemented? #

First, we wrote a design document for caps negotiation with dmabuf modifiers, where we added a new color format (DMA_DRM) and a new caps field (drm-format). This new caps field holds a string, or a list of strings, composed by the tuple DRM_color_format : DRM_modifier.

Second, we extended the video info object to support DMABuf with helper functions that parse and construct the drm-format field.

Third, we added the dmabuf caps negotiation in glupload. This part was the most difficult one, since the capability of importing DMABufs to OpenGL (which is only available in EGL/GLES) is run-time defined, by querying the hardware. Also, there are two code paths to import frames: direct or RGB-emulated. Direct would be the most efficient, but it depends on the presence of GLES2 API in the driver; while RGB-emulated is imported as a set of RGB images where each component is an image. At the end more than a thousand lines of code were added to the glupload element, beside the code added to EGL context object.

Fourth, and unexpectedly, waylandsink also got DMABuf caps negotiation support.

And lastly, decoders in va plugin merged theirs DMABuf caps negotiation support.

How I can test it? #

You need, of course, to user the current main branch of GStreamer, since it’s just fresh and there’s no release yet. Then you need a box with VA support. And if you inspect, for example, vah264dec, you might see this output if your box is Intel (but also AMD through Mesa is supported though the negotiated memory is linear so far):

Pad Templates:
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-h264
                profile: { (string)main, (string)baseline, (string)high, (string)progressive-high, (string)constrained-high, (string)constrained-baseline }
                  width: [ 1, 4096 ]
                 height: [ 1, 4096 ]
              alignment: au
          stream-format: { (string)avc, (string)avc3, (string)byte-stream }

  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-raw(memory:VAMemory)
                  width: [ 1, 4096 ]
                 height: [ 1, 4096 ]
                 format: NV12
      video/x-raw(memory:DMABuf)
                  width: [ 1, 4096 ]
                 height: [ 1, 4096 ]
                 format: DMA_DRM
             drm-format: NV12:0x0100000000000002
      video/x-raw
                  width: [ 1, 4096 ]
                 height: [ 1, 4096 ]
                 format: NV12

What it’s saying, for memory:DMABuf caps feature, the drm-format to negotiate is NV12:0x0100000000000002.

Now some tests:

NOTE: These commands assume that va decoders are primary ranked (see merge request 2312), and that you’re in a Wayland session.

gst-play-1.0 --flags=0x47 video.file --videosink=waylandsink

GST_GL_API=gles2 gst-play-1.0 --flags=0x47 video.file --videosink=glimagesink

gst-play-1.0 --flags=0x47 video.file --videosink='glupload ! gtkglsink'

Right now it’s required to add --flags=0x47 to playbin because it adds video filters that still don’t negotiate the new DMABuf caps.

GST_GL_API=gles2 instructs GStreamer OpenGL to use GLES2 API, which allows direct importation of YUV images.

Thanks to all the people involved in this effort!

As usual, if you would like to learn more about DMABuf, VA-API, GStreamer or any other open multimedia framework, contact us!

Review of Igalia Multimedia activities (2022)

2023-03-14T00:00:00Z

We, Igalia’s multimedia team, would like to share with you our list of achievements along the past 2022.

WebKit Multimedia #

WebRTC #

Phil already wrote a first blog post, of a series, on this regard: WebRTC in WebKitGTK and WPE, status updates, part I. Please, be sure to give it a glance, it has nice videos.

Long story short, last year we started to support Media Capture and Streams in WebKitGTK and WPE using GStreamer, either for input devices (camera and microphone), desktop sharing, webaudio, and web canvas. But this is just the first step. We are currently working on RTCPeerConnection, also using GStreamer, to share all these captured streams with other web peers. Meanwhile, we’ll wait for the second episode of Phil’s series :)

MediaRecorder #

We worked in an initial implementation of MediaRecorder with GStreamer (1.20 or superior). The specification goes about allowing a web browser to record a selected stream. For example, a voice-memo or video application which could encode and upload a capture of your microphone / camera.

Gamepad #

While WebKitGTK already has Gamepad support, WPE lacked it. We did the implementation last year, and there’s a blog post about it: Gamepad in WPEWebKit, with video showing a demo of it.

Capture encoded video streams from webcams #

Some webcams only provide high resolution frames encoded in H.264 or so. In order to support these resolutions with those webcams we added the support for negotiate of those formats and decode them internally to handle the streams. Though we are just at the beginning of more efficient support.

Flatpak SDK maintenance #

A lot of effort went to maintain the Flatpak SDK for WebKit. It is a set of runtimes that allows to have a reproducible build of WebKit, independently of the used Linux distribution. Nowadays the Flatpak SDK is used in Webkit’s EWS, and by many developers.

Among all the features added during the year we can highlight added Rust support, a full integrity check before upgrading, and offer a way to override dependencies as local projects.

MSE/EME enhancements #

As every year, massive work was done in WebKit ports using GStreamer for Media Source Extensions and Encrypted Media Extensions, improving user experience with different streaming services in the Web, such as Odysee, Amazon, DAZN, etc.

In the case of encrypted media, GStreamer-based WebKit ports provide the stubs to communicate with an external Content Decryption Module (CDM). If you’re willing to support this in your platform, you can reach us.

Also we worked in a video demo showing how MSE/EME works in a Raspberry Pi 3 using WPE:

Your browser does not support playing HTML5 video. You can download the file instead. Here is a description of the content: EME demo

WebAudio demo #

We also spent time recording video demos, such as this one, showing WebAudio using WPE on a desktop computer.

Your browser does not support playing HTML5 video. You can download the file instead. Here is a description of the content: WPE WebAudio

GStreamer #

We managed to merge a lot of bug fixes in GStreamer, which in many cases can be harder to solve rather than implementing new features, though former are more interesting to tell, such as those related with making Rust the main developing language for GStreamer besides C.

Rust bindings and GStreamer elements for Vonage Video API / OpenTok #

OpenTok is the legacy name of Vonage Video API, and is a PaaS (Platform As a Service) to ease the development and deployment of WebRTC services and applications.

We published our work in Github of Rust bindings both for the Client SDK for Linux and the Server SDK using REST API, along with a GStreamer plugin to publish and subscribe to video and audio streams.

GstWebRTCSrc #

In the beginning there was webrtcbin, an element that implements the majority of W3C RTCPeerConnection API. It’s so flexible and powerful that it’s rather hard to use for the most common cases. Then appeared webrtcsink, a wrapper of webrtcbin, written in Rust, which receives GStreamer streams which will be offered and streamed to web peers. Later on, we developed webrtcsrc, the webrtcsink counterpart: an element which source pads push streams from web peers, such as another browser, and forward those Web streams as GStreamer ones in a pipeline. Both webrtcsink and webrtcsrc are written in Rust.

Behavior-Driven Development test framework for GStreamer #

Behavior-Driven Development is gaining relevance with tools like Cucumber for Java and its domain specific language, Gherkin to define software behaviors. Rustaceans have picked up these ideas and developed cucumber-rs. The logical consequence was obvious: Why not GStreamer?

Last year we tinkered with GStreamer-Cucumber, a BDD to define behavior tests for GStreamer pipelines.

GstValidate Rust bindings #

There have been some discussion if BDD is the best way to test GStreamer pipelines, and there’s GstValidate, and also, last year, we added its Rust bindings.

GStreamer Editing Services #

Though not everything was Rust. We work hard on GStreamer’s nuts and bolts.

Last year, we gathered the team to hack GStreamer Editing Services, particularly to explore adding OpenGL and DMABuf support, such as downloading or uploading a texture before processing, and selecting a proper filter to avoid those transfers.

GstVA and GStreamer-VAAPI #

We helped in the maintenance of GStreamer-VAAPI and the development of its near replacement: GstVA, adding new elements such as the H.264 encoder, the compositor and the JPEG decoder. Along with participation on the debate and code reviewing of negotiating DMABuf streams in the pipeline.

Vulkan decoder and parser library for CTS #

You might have heard about Vulkan has now integrated in its API video decoding, while encoding is currently work-in-progress. We devoted time on helping Khronos with the Vulkan Video Conformance Tests (CTS), particularly with a parser based on GStreamer and developing a H.264 decoder in GStreamer using Vulkan Video API.

You can check the presentation we did last Vulkanised.

https://www.youtube.com/watch?v=5PV_yTculaM

WPE Android Experiment #

In a joint adventure with Igalia’s Webkit team we did some experiments to port WPE to Android. This is just an internal proof of concept so far, but we are looking forward to see how this will evolve in the future, and what new possibilities this might open up.

Your browser does not support playing HTML5 video. You can download the file instead. Here is a description of the content: WPE View

If you have any questions about WebKit, GStreamer, Linux video stack, compilers, etc., please contact us.

GStreamer compilation with third party libraries

2023-01-03T00:00:00Z

Suppose that you have to hack a GStreamer element which requires a library that is not (yet) packaged by your distribution, nor wrapped as a Meson’s subproject. How do you do?In our case, we needed the latest version of

Which are interrelated CMake projects.

For these cases, GStreamer’s uninstalled development scripts can use a special directory: gstreamer/prefix. As the README.md says:

NOTE: In the development environment, a fully usable prefix is also configured in gstreamer/prefix where you can install any extra dependency/project.

This means that gstenv.py script (the responsible of setting up the uninstalled development environment) will add

gstreamer/prefix/bin in PATH for executable files.
gstreamer/prefix/lib and gstreamer/prefix/share/gstreamer-1.0 in GST_PLUGIN_PATH, for out-of-tree elements.
gstreamer/prefix/lib in GI_TYPELIB_PATH for GObject Introspection metadata.
gstreamer/prefix/lib/pkgconfig in PKG_CONFIG_PATH for third party dependencies (our case!)
gstreamer/prefix/etc/xdg for XDG_CONFIG_DIRS for XDG compliant configuration files.
gstreamer/prefix/lib and gstreamer/prefix/lib64 in LD_LIBRARY_PATH for third party libraries.

Therefore, the general idea, is to compile those third party libraries with their installation prefix as gstreamer/prefix.

In our case, Vulkan repositories are interrelated so they need to be compiled in certain order. Also, we decided, for self-containment, to clone them in gstreamer/subprojects.

Vulkan-Headers #

cd ~/gst/gstreamer/subprojects
git clone git@github.com:KhronosGroup/Vulkan-Headers.git
cd Vulkan-Headers
mkdir build
cd build
cmake -GNinja -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=/home/vjaquez/gst/gstreamer/prefix ..
cmake --build . --install

Vulkan-Loader #

cd ~/gst/gstreamer/subprojects
git clone git@github.com:KhronosGroup/Vulkan-Loader.git
cd Vulkan-Loader
mkdir build
cd build
cmake  -DCMAKE_BUILD_TYPE=Debug -DVULKAN_HEADERS_INSTALL_DIR=/home/vjaquez/gst/gstreamer/prefix DCMAKE_INSTALL_PREFIX=/home/vjaquez/gst/gstreamer/prefix ..
cmake --build . --install

Vulkan-Tools #

cd ~/gst/gstreamer/subprojects
git clone git@github.com:KhronosGroup/Vulkan-Tools.git
cd Vulkan-Tools
mkdir build
cd build
cmake  -DCMAKE_BUILD_TYPE=Debug -DVULKAN_HEADERS_INSTALL_DIR=/home/vjaquez/gst/gstreamer/prefix DCMAKE_INSTALL_PREFIX=/home/vjaquez/gst/gstreamer/prefix ..
cmake --build . --install

Right now we have the Vulkan headers and the Vulkan loader pkg-config file in place. And we should be able to compile GStreamer. Right?

Not exactly, because gstenv.py only sets the environment variables for the development environment, not for GStreamer compilation. But the solution is simple, because we have all set in the proper order: just to set PKG_CONFIG_PATH when executing meson setup:

PKG_CONFIG_PATH=/home/vjaquez/gst/gstreamer/prefix/lib/pkgconfig meson setup --buildtype=debug build

Video decoding in GStreamer with Vulkan Video extension (part 2)

2022-12-18T00:00:00Z

Its has been a while since I reported my tinkering with the Vulkan Video provisional extension. Now the specification will have its final release soon_ish_, and also there has been more engagement within the open source communities, such as the work-in-progress FFmpeg implementation by Lynne (please, please, read that post), and the also work-in-progress Mesa 3D drivers both for AMD and Intel by Dave Airlie! Along with the well known NVIDIA beta drivers for Vulkan.From our side, we have been trying to provide an open source alternative to the video parser used by the Conformance Test Suite and the NVIDIA vk_video_samples, using GStreamer: GstVkVideoParser, which intends to be a drop-in replacement of the current proprietary parser library.

Along the way, we have sketched the Vulkan Video support in gfxreconstruct, for getting traces of the API usage. Sadly, its kind of bit-rotten right now, even more because the specification has changed since then.

Regarding the H.264 decoder for GStreamer, we just restarted its hacking. The merge request was moved to monorepo, but for the sake of the well needed complete re-write, we changed the branch to this one (vkh264dec). We needed to re-write it because, besides the specification updates, we have learned many things along the journey, such as the out-of-band parameters update, Vulkan’s recommendation for memory pre-allocation as much as possible, the DPB/references handling, the debate about buffer vs. slice uploading, and other friction points that Lynne has spotted for future early adopters.

The way to compile it is grab the branch and compile as usually GStreamer is compiled with meson:

meson setup builddir -Dgst-plugins-bad:vulkan-video=enabled --buildtype=debug
ninja C builddir

And run simple pipelines such as

gst-launch-1.0 filesrc location=INPUT ! parsebin ! vulkanh264dec ! fakesink -v

Our objective is to have a functional demo for the next Vulkanised in February. We are very ambitious, we want it to work in Linux, Windows and in many GPU as possible. Wish us luck. And happy December festivities!

Gamepad in WPEWebKit

2022-07-20T00:00:00Z

This is the brief story of the Gamepad implementation in WPEWebKit.It started with an early development done by Eugene Mutavchi (kudos!). Later, by the end of 2021, I retook those patches and discussed them with my fellow igalian Adrián, and we decided to come with a slightly different approach.

Before going into the details, let’s quickly review the WPE architecture:

WPE architecture diagram

cog library — it’s a shell library that simplifies the task of writing a WPE browser from the scratch, by providing common functionality and helper APIs.
WebKit library — that’s the web engine that, given an URI and other following inputs, returns, among other outputs, graphic buffers with the page rendered.
WPE library — it’s the API that bridges cog (1) (or whatever other browser application) and WebKit (2).
WPE backend — it’s main duty is to provide graphic buffers to WebKit, buffers supported by the hardware, the operating system, windowing system, etc.

Eugene’s implementation has code in WebKit (implementing the gamepad support for WPE port); code in WPE library with an API to communicate WebKit’s gamepad and WPE backend, which provided a custom implementation of gamepad, reading directly the event in the Linux device. Almost everything was there, but there were some issues:

WPE backend is mainly designed as a set of protocols, similar to Wayland, to deal with graphic buffers or audio buffers, but not for input events. Cog library is the place where input events are handled and injected to WebKit, such as keyboard.
The gamepad handling in a WPE backend was ad-hoc and low level, reading directly the events from Linux devices. This approach is problematic since there are plenty gamepads in the market and each has its own axis and buttons, so remapping them to the standard map is required. To overcome this issue and many others, there’s a GNOME library: libmanette, which is already used by WebKitGTK port.

Today’s status of the gamepad support is that it works but it’s not yet fully upstreamed.

merged libwpe pull request.
cog pull request – there are two implementations: none and libmanette. None is just a dummy implementation which will ignore any request for a gamepad provider; it’s provided if libmanette is not available or if available libwpe hasn’t gamepad support.
WebKit pull request.

To prove you all that it works my exhibit A is this video, where I play asteroids in a RasberryPi 4 64 bits:

https://www.youtube.com/watch?v=LwNjAKRpaj4

The image was done with Buildroot, using its master branch (from a week ago) with a bunch of modifications, such as adding libmanette, a kernel patch for my gamepad device, kernel 5.15.55 and its corresponding firmware, etc.

GstVA H.264 encoder, compositor and JPEG decoder

2022-07-18T00:00:00Z

There are, right now, three new GstVA elements merged in main: vah264enc, vacompositor and vajpegdec.Just to recap, GstVA is a GStreamer plugin in gst-plugins-bad (yes, we agree it’s not a great name anymore), to differentiate it from gstreamer-vaapi. Both plugins use libva to access stateless video processing operations; the main difference is, precisely, how stream’s state is handled: while GstVA uses GStreamer libraries shared by other hardware accelerated plugins (such as d3d11 and v4l2codecs), gstreamer-vaapi uses an internal, tightly coupled and convoluted library.

Also, note that right now (release 1.20) GstVA elements are ranked NONE, while gstreamer-vaapi ones are mostly PRIMARY+1.

Back to the three new elements in GstVA, the most complex one is vah264enc wrote almost completely by He Junyan, from Intel. For it, He had to write a H.264 bitwriter which is, until certain extend, the opposite for H.264 parser: construct the bitstream buffer from H.264 structures such as PPS, SPS, slice header, etc. This API is part of libgstcodecparsers, ready to be reused by other plugins or applications. Currently vah264enc is fairly complete and functional, dealing with profiles and rate controls, among other parameters. It still have rough spots, but we’re working on them. But He Junyan is restless and he already has in the pipeline an encoder common class along with a HEVC and AV1 encoders.

The second element is vacompositor, wrote by Artie Eoff. It’s the replacement of vaapioverlay in gstreamer-vaapi. The suffix compositor is preferred to follow the name of primary video mixing (software-based) element: compositor, successor of videomixer. See this discussion for further details. The purpose of this element is to compose a single video stream from multiple video streams. It works with Intel’s media-driver supporting alpha channel, and also works with AMD Mesa Gallium, but without alpha channel (in other words, a custom degree of transparency).

The last, but not the least, element is vajpegdec, which I worked on. The main issue was not the decoder itself, but jpegparse, which didn’t signal the image caps required for the hardware accelerated decoders. For instance, VA only decodes images with SOF marker 0 (Baseline DCT). It wasn’t needed before because the main and only consumer of the parser is jpegdec which deals with any type of JPEG image. Long story short, we revamped jpegparse and now it signals sof marker, color space (YUV, RGB, etc.) and chroma subsampling (if it has YUV color space), along with comments and EXIF-like metadata as pipeline’s tags. Thus vajpegdec will expose in caps template the supported color space and chroma subsampling supported by the driver. For example, Intel supports (more or less) RGB color space, while AMD Mesa Gallium don’t.

And that’s all for now. Thanks.

From gst-build to local-projects

2022-05-02T00:00:00Z

Two years ago I wrote a blog post about using gst-build inside of WebKit SDK flatpak. Well, all that has changed. That’s the true upstream spirit.There were two main reason for the change:

Since the switch to GStreamer mono repository, gst-build has been deprecated. The mechanism in WebKit were added, basically, to allow GStreamer upstream, so keeping gst-build directory just polluted the conceptual framework.
By using gst-build one could override almost any other package in WebKit SDK. For example, for developing gamepad handling in WPE I added libmanette as a GStreamer subproject, to link a modified version of the library rather than the one in Flatpak. But that approach added an unneeded conceptual depth in tree.

In order to simplify these operations, by taking advantage of Meson’s subproject support directly, gst-build handling were removed and new mechanism was set in place: Local Dependencies. With local dependencies, you can add or override almost any dependency, while flatting the tree layout, by placing at the same level GStreamer and any other library. Of course, in order add dependencies, they must be built with meson.

For example, to override libsoup and GStreamer, just clone both repositories below of Tools/flatpak/local-projects/subprojects, and declare them in WEBKIT_LOCAL_DEPS environment variable:

export WEBKIT_SDK_LOCAL_DEPS=libsoup,gstreamer-full
export WEBKIT_SDK_LOCAL_DEPS_OPTIONS="-Dgstreamer-full:introspection=disabled -Dgst-plugins-good:soup=disabled"
build-webkit --wpe

Digging further into Flatpak with NVIDIA

2022-01-17T00:00:00Z

As you may know the development environment used by WebKitGTK and WPE is based on Flatpak. I feel hacking software within Flatpak like teleoperating a rover in Mars, since I have to go through Flatpak commands to execute the commands I want to execute. The learning curve is steeper in exchange of a common development environment.I started to work on another project where is required to use an NVIDIA GPU, without stopping to work on WebKitGTK/WPE. So I needed to use the card within Flatpak, and it’s well known that, currently, that setup is not available out-of-the-box. Furthermore, I have to use a very specific version of the graphics card drive for Vulkan.

This is the story of how I make it work.

My main reference is, of course, the blog post of my colleague TingPing: Using host Nvidia driver with Flatpak, besides Flatpak for NVIDIA GL runtime platform.

As TingPing explained, Flatpak does not use host libraries, that’s why it might need runtimes and extensions for specific hardware setups, with the libraries for user-space, such as NVIDIA GL platform runtime. And it must have the same version as the one running in kernel.

NVIDIA GL platform extension is a small project which generates Flatpak’s runtimes for every public NVIDIA driver. The interesting part is that those runtimes are not created in building time, but at install-time. When the user installs the runtime, a driver blob is downloaded from NVIDIA servers (see --extra-data in flatpak build-finish for reference), and a small program is executed, which extracts the embedded tarball in the blob, and from it, it extracts the required libraries. In a few words, initially, the runtime is composed only by a definition of the file to download, and the small program that populates the Flatpak’s file system at install-time.

The trick here, that took me a lot to be aware, is that this small program has to be statically compiled, since it has be executed regardless the available runtime.

This little program uses libarchive to extract the libraries from NVIDIA’s tarball, but libarchive is not available statically in any Flatpak’s SDK. Furthermore, our use of libarchive will depend on libz and liblzma, both statically compile as well. Gladly, there’s only one, very old version, obsolete, of freedesktop SDK, which offers static versions of libz and liblzma: 1.6. And that’s why org.freedesktop.Platform.GL.nvidia demands that specific old version of the SDK. Then, the manifest of the extension contains basically the static compilation of libarchive and the static compilation of the next-to-be apply_extra.

Update: There’s a merge request to use current freedesktop SDK 21.08, which, basically, builds statically libz and liblzma, besides libarchive.

I needed to modify org.freedesktop.Platform.GL.nvidia sources a bit, since it, by default, consist in a big loop of downloading, hashing, templating a json manifest, and building, for every supported driver. But, as my case is just one custom driver, I don’t want to waste time in that loop. The hack to achieve it is fairly simple:

diff --git a/versions.sh b/versions.sh
index 8b72664..86686c0 100755
--- a/versions.sh
+++ b/versions.sh
@@ -15,4 +15,5 @@ TESLA_VERSIONS="450.142.00 450.119.04 450.51.06 450.51.05 440.118.02 440.95.01 4
# Probably never: https://ahayzen.com/direct/flathub_downloads_only_nvidia_runtimes.txt
UNSUPPORTED_VERSIONS="390.147 390.144 390.143 390.141 390.138 390.132 390.129 390.116 390.87 390.77 390.67 390.59 390.48 390.42 390.25 390.12 387.34 387.22 387.12 384.130 384.111 384.98 384.90 384.69 384.59 384.47 381.22 381.09 378.13 375.82 375.66 375.39 375.26 370.28 367.57"

-DRIVER_VERSIONS="$BETA_VERSIONS $VULKAN_VERSIONS $NEW_FEATURE_VERSIONS $PRODUCTION_VERSIONS $LEGACY_VERSIONS $TESLA_VERSIONS $UNSUPPORTED_VERSIONS"
+#DRIVER_VERSIONS="$BETA_VERSIONS $VULKAN_VERSIONS $NEW_FEATURE_VERSIONS $PRODUCTION_VERSIONS $LEGACY_VERSIONS $TESLA_VERSIONS $UNSUPPORTED_VERSIONS"
+DRIVER_VERSIONS="470.XX.XX"

But in order to make it work, it needs a file in data/ directory with the specification of the file to download, with the format: NAME:SHA256:DOWNLOAD-SIZE:INSTALL-SIZE:URL.

--- /dev/null
+++ b/data/nvidia-470.XX.XX-x86_64.data
@@ -0,0 +1 @@
+:34...checksum-sha264...:123456789::http://compu.home.arpa/NVIDIA/NVIDIA-Linux-x86_64-470.XX.XX.run

The last parameter is the URL where the driver shall be downloaded. In my case is a local server to ease the testing.

Long story short, the command to execute are:

To setup the building environment:

flatpak install org.freedesktop.Sdk//1.6 org.freedesktop.Platform//1.6

To build the Flatpak repository and package:

make

The command will output a repo directory in the current one. There’s where the generated Flatpak package is stored.

To install the local repository and the extension:

flatpak --user remote-add --no-gpg-verify nvidia-local repo
flatpak -v install nvidia-local org.freedesktop.Platform.GL.nvidia-470-XX-XX

To remove the obsolete SDK and platform once built:

flatpak uninstall org.freedesktop.Sdk//1.6 org.freedesktop.Platform//1.6

To remove the local repository and the extension if something went wrong:

flatpak -v uninstall org.freedesktop.Platform.GL.nvidia-470-62-15
flatpak --user remote-delete nvidia-local

One way to verify if the libraries are installed correctly and if they match with the driver running in the kernel’s host, is to install and run GreenWithEnvy:

flatpak install com.leinardi.gwe
flatpak run com.leinardi.gwe

If you want to install the driver in your WebKit development environment, you just need to set the environment variable FLATPAK_USER_DIR:

FLATPAK_USER_DIR=~/WebKit/WebKitBuild/UserFlatpak flatpak --user remote-add --no-gpg-verify nvidia-local repo
FLATPAK_USER_DIR=~/WebKit/WebKitBuild/UserFlatpak flatpak -v install nvidia-local org.freedesktop.Platform.GL.nvidia-470-XX-XX

GstVA in GStreamer 1.20

2021-12-08T00:00:00Z

It was a year and half ago when I announced a new VA-API H.264 decoder element in gst-plugins-bad. And it was bundled in GStreamer release 1.18 a couple months later. Since then, we have been working adding more decoders and filters, fixing bugs, and enhancing its design. I wanted to publish this blog post as soon as release 1.20 was announced, but, since the developing window is closed, which means no more new features will be included, I’ll publish it now, to create buzz around the next GStreamer release.Here’s the list of new GstVA decoders (of course, they are only available if your driver supports them):

vah265dec
vavp8dec
vavp9dec
vaav1dec
vampeg2dec

Also, there are a couple new features in vah264dec (common to all gstcodecs-based H.264 decoders):

Supports interlaced streams (vah265dec and vampeg2dec too).
Added a compliance property to trick the specification conformance for lower the latency, for example, or to enable non-standard features.

But not only decoders, there are two new elements for post-processing:

vapostproc
vadeinterlace

vapostproc is similar to vaapipostproc but without the interlace operation, since it was moved to another element. The reason for this is because there are deinterlacing methods which require to hold a list of referenced frames, thus these methods are broken in vaapipostproc, and adding them would increase the complexity of the element with no need. To keep things simple it’s better to handle deinterlacing in a different element.

This is the list of filters and features supported by vapostproc:

Color conversion
Resizing
Cropping
Color balance (Intel only -so far-)
Video direction (Intel only)
Skin tone enhancement (Intel only)
Denoise and Sharpen (Intel only)

And, I ought to say, HDR is in the pipeline, but it will be released after 1.20.

While vadeinterlace only does that, deinterlacing. But it supports all the available methods currently in the VA-API specification, using the new way to select the field to extract, since the old one (used by GStreamer-VAAPI and FFmpeg) is a bit more expensive.

Finally, both video filters, if they cannot handle the income format, they are configured in passthrough mode.

But there are not only new elements, there’s also a new library!

Since many others elements need to share a common VADisplay in the GStreamer pipeline, the new library expose only the GstVaDisplay object by now. The new library must be thin and lean, exposing only what it’s requested by other elements, such as gst-msdk. We have pending to merge after 1.20, for example, the add of GstContext helpers, and the plan is to expose the allocators and bufferpools later.

Another huge task are encoders. After the freeze, we’ll merge the first implementation of the H.264 encoder, and add, in different iterations, more encoders.

As I said in the previous blog post, all these elements are ranked as none, so the won’t be autoplugged, for example by playbin. To do so, users need to export the environment variable GST_PLUGIN_FEATURE_RANK as documented.

GST_PLUGIN_FEATURE_RANK=vah264dec:MAX,vah265dec:MAX,vampeg2dec:MAX,vavp8dec:MAX,vavp9dec:MAX gst-play-1.0 stream.mp4

Thanks a bunch to He Junyan, Seungha Yang and Nicolas Dufresne, for all the effort and care.

Still, the to-do list is large enough. Just to share what I have in my notes:

Add a new upload method in glupload to interop with VA surfaces – though this hardly will be merged since it creates a circular dependency between -base and -bad.
vavc1dec – it might need a rewrite of vc1parse.
vajpegdec – it needs a rewrite of jpegparse.
vaalphacombine – decoding alpha channel with VA within vp9alphacodebin and vp8alphacodebin
vamixer – similar to compositor, glmixer or vaapioverlay, to compose a single frame from different video streams.
And encoders (mainly H.264 and H.265).

As a final mode, GStreamer-VAAPI has enter into maintenance mode. The general plan, without any promise or dates, is to deprecate it when most of its use cases were covered by GstVA.

Video decoding in GStreamer with Vulkan

2021-07-09T00:00:00Z

Warning: Vulkan video is still work in progress, from specification to available drivers and applications. Do not use it for production software just yet.## Introduction

Vulkan is a cross-platform Application Programming Interface (API), backed by the Khronos Group, aimed at graphics developers for a wide range of different tasks. The interface is described by a common specification, and it is implemented by different drivers, usually provided by GPU vendors and Mesa.

One way to visualize Vulkan, at first glance, is like a low-level OpenGL API, but better described and easier to extend. Even more, it is possible to implement OpenGL on top of Vulkan. And, as far as I am told by my peers in Igalia, Vulkan drivers are easier and cleaner to implement than OpenGL ones.

A couple years ago, a technical specification group (TSG), inside the Vulkan Working Group, proposed the integration of hardware accelerated video compression and decompression into the Vulkan API. In April 2021 the formed Vulkan Video TSG published an introduction to the specification. Please, do not hesitate to read it. It’s quite good.

Matthew Waters worked on a GStreamer plugin using Vulkan, mainly for uploading, composing and rendering frames. Later, he developed a library mapping Vulkan objects to GStreamer. This work was key for what I am presenting here. In 2019, during the last GStreamer Conference, Matthew delivered a talk about his work. Make sure to watch it, it’s worth it.

Other key components for this effort were the base classes for decoders and the bitstream parsing libraries in GStreamer, jointly developed by Intel, Centricular, Collabora and Igalia. Both libraries allow using APIs for stateless video decoding and encoding within the GStreamer framework, such as Vulkan Video, VAAPI, D3D11, and so on.

When the graphics team in Igalia told us about the Vulkan Video TSG, we decided to explore the specification. Therefore, Igalia decided to sponsor part of my time to craft a GStreamer element to decode H.264 streams using these new Vulkan extensions.

Assumptions #

As stated at the beginning of this text, this development has to be considered unstable and the APIs may change without further notice.

Right now, the only Vulkan driver that offers these extensions is the beta NVIDIA driver. You would need, at least, version 455.50.12 for Linux, but it would be better to grab the latest one. And, of course, I only tested this on Linux. I would like to thank NVIDIA for their Vk Video samples. Their test application drove my work.

Finally, this work assumes the use of the main development branch of GStreamer, because the base classes for decoders are quite recent. Naturally, you can use gst-build for an efficient upstream workflow.

Work done #

This work basically consists of two new objects inside the GstVulkan code:

GstVulkanDeviceDecoder: a GStreamer object in GstVulkan library, inherited from GstVulkanDevice, which enables VK_KHR_video_queue and VK_KHR_video_decode_queue extensions. Its purpose is to handle codec-agnostic operations.
vulkanh264dec: a GStreamer element, inherited from GstH264Decoder, which tries to instantiate a GstVulkanDeviceDecoder to composite it and is in charge of handling codec-specific operations later, such as matching the parsed structures. It outputs, in the source pad, memory:VulkanImage featured frames, with NV12 color format.

So far this pipeline works without errors:

gst-launch-1.0 filesrc location=big_buck_bunny_1080p_h264.mov ! parsebin ! vulkanh264dec ! fakesink

As you might see, the pipeline does not use vulkansink to render frames. This is because the Vulkan format output by the driver’s decoder device is VK_FORMAT_G8_B8R8_2PLANE_420_UNORM, which is NV12 crammed in a single image, while for GstVulkan a NV12 frame is a buffer with two images, one per component. So the current color conversion in GstVulkan does not support this Vulkan format. That is future work, among other things.

You can find the merge request for this work in GStreamer’s Gitlab.

Future work #

As was mentioned before, it is required to fully support VK_FORMAT_G8_B8R8_2PLANE_420_UNORM format in GstVulkan. That requires thinking about how to keep backwards compatibility. Later, an implementation of the sampler to convert this format to RGB will be needed, so that decoded frames can be rendered by vulkansink.

Also, before implementing any new feature, the code and its abstractions will need to be cleaned up, since currently the division between codec-specific and codec-agnostic code is not strict, and it must be fixed.

Another important cleanup task is to enhance the way the Vulkan headers are handled. Since the required headers files for video extensions are beta, they are not expected to be available in the system, so temporally I had to add the those headers as part of the GstVulkan library.

Then it will be possible to implement the H.265 decoder, since the NVIDIA driver also supports it.

Later on, it will be nice to start thinking about encoders. But this requires extending support for stateless encoders in GStreamer, something I want do to for the new VAAPI plugin too.

Thanks for bearing with me, and thanks to Igalia for sponsoring this work.

Review of Igalia Multimedia activities (2020/H2)

2021-04-21T00:00:00Z

As the first quarter of 2021 has already come to a close, we reckon it’s time to recap our achievements from the second half of 2020, and update you on the improvements we have been making to the multimedia experience on the Web and Linux in general.Our previous reports:

WPE / WebKitGTK #

We have closed ~100 issues related with multimedia in WebKitGTK/WPE, such as fixed seek issues while playback, plugged memory leaks, gardening tests, improved Flatpak-based developing work-flow, enabled new codecs, etc… Overall, we improved a bit the multimedia’s user experience on these Webkit engine ports.

To highlight a couple tasks, we did some maintenance work on WebAudio backends, and we upstreamed an internal audio mixer, keeping only one connection to the audio server, such as PulseAudio, instead of multiple connections, one for every audio resource. The mixer combines all streams into a single audio server connection.

Adaptive media streaming for the Web (MSE) #

We have been working on a new MSE backend for a while, but along the way many related bugs have appeared and they were squashed. Also many code cleanups has been carried out. Though it has been like yak shaving, we are confident that we will reach the end of this long and winding road soonish.

DRM media playback for the Web (EME) #

Regarding digital protected media playback, we worked to upstream OpenCDM, support with Widevine, through RDK’s Thunder framework, while continued with the usual maintenance of the others key systems, such as Clear Key, Widevine and PlayReady.

For more details we published a blog post: Serious Encrypted Media Extensions on GStreamer based WebKit ports.

Realtime communications for the Web (WebRTC) #

Just as EME, WebRTC is not currently enabled by default in browsers such as Epiphany because license problems, but they are available for custom adopters, and we are maintaining it. For example, we collaborated to upgrade LibWebRTC to M87 and fixed the expected regressions and gardening.

Along the way we experimented a bit with the new GPUProcess for capture devices, but we decided to stop the experimentation while waiting for a broader adoption of the process, for example in graphics rendering, in WPE/WebKitGTK.

GPUProcess work will be retaken at some point, because it’s not, currently, a hard requirement, since we already have moved capture devices handling from the UIProcess to the WebProcess, isolating all GStreamer operations in the latter.

GStreamer #

GStreamer is one of our core multimedia technologies, and we contribute on it on a daily basis. We pushed ~400 commits, with similar number of code reviews, along the second half of 2020. Among of those contributions let us highlight the following list:

A lot of bug fixing aiming for release 1.18.
Reworked and enhanced decodebin3, the GstTranscoder API and encodebin.
Merged av1parse in video parsers plugin.
Merged qroverlay plugin.
Iterated on the mono-repo proposal, which requires consensus and coordination among the whole community.
gstwpe element has been greatly improved from new user requests.
Contributed on the new libgstcodecs library, which enables stateless video decoders through different platforms (for example, v4l2, d3d11, va, etc.).
Developed a new plugin for VA-API using this library, exposing H.264, H.265, VP9, VP8, MPEG2 decoders and a full featured postprocessor, with better performance, according our measurements, than GStreamer-VAAPI.

Conferences #

Despite 2020 was not a year for conferences, many of them went virtual. We attended one, the Mile high video conference, and participated in the Slack workspace.

Thank you for reading this report and stay tuned with our work.

Notes on using Emacs (LSP/ccls) for WebKit

2020-11-26T00:00:00Z

I used to regard myself as an austere programmer in terms of tooling: Emacs —with a plain configuration— and grep. This approach forces you to understand all the elements involved in a project.Some time ago I have to code in Rust, so I needed to learn the language as fast as possible. I looked for packages in MELPA that could help me to be productive quickly. Obviously, I installed rust-mode, but I also found racer for auto-completion. I tried it out. It was messy to setup and unstable, but it helped me to code while learning. When I felt comfortable with the base code, I uninstalled it.

This year I returned to work on WebKit. The last time I contributed to it was around five years ago, but now in a different area (still in the multimedia stack). WebKit is huge, and because of C++, I found gtags rather limited. Out of curiosity I looked for something similar to racer but for C++. And I spent a while digging on it.

The solution consists in the integration of three MELPA packages:

lsp-mode: a client for Language Server Protocol for Emacs.
company-mode: a text completion framework.
ccls: A C/C++ language server. Besides emacs-ccls adds more functionality to lsp-mode.

(I known, there’s a simpler alternative to lsp-mode, but I haven’t tried it yet).

First we might explain what’s LSP. It stands for Language Server Protocol, defined with JSON-RPC messages, between the editor and the language server. It was originally developed by Microsoft for Visual Studio, which purpose is to support auto-completion, finding symbol’s definition, to show early error markers, etc., inside the editor. Therefore, lsp-mode is an Emacs mode that communicates with different language servers in LSP and operates in Emacs accordingly.

In order to support the auto-completion use-case lsp-mode uses the company-mode. This Emacs mode is capable to create a floating context menu where the editing cursor is placed.

The third part of the puzzle is, of course, the language server. There’s a language servers for different programming languages. For C & C++ there are two servers: clangd and ccls. The former uses Clang compiler, the last can use either Clang, GCC or MSVC. Along this text ccls will be used for reasons exposed later. In between, emacs-ccls leverages and extends the support of ccls in lsp-mode, though it’s not mandatory.

In short, the basic .emacs configuration, using use-package, would have these lines:

(use-package company
  :diminish
  :config (global-company-mode 1))

(use-package lsp-mode
  :diminish "L"
  :init (setq lsp-keymap-prefix "C-l"
              lsp-enable-file-watchers nil
              lsp-enable-on-type-formatting nil
              lsp-enable-snippet nil)
  :hook (c-mode-common . lsp-deferred)
  :commands (lsp lsp-deferred))

(use-package ccls
  :init (setq ccls-sem-highlight-method 'font-lock)
  :hook ((c-mode c++-mode objc-mode) . (lambda () (require 'ccls) (lsp-deferred))))

The snippet first configures company-mode. It is enabled globally because, normally, it is a nice feature to have, even in non-coding buffers, such as this very one, for writing a blog post in markdown format. Diminish mode hides or abbreviates the mode description in the Emacs’ mode line.

Later comes lsp-mode. It’s big and aims to do a lot of things, basically we have to tell it to disable certain features, such as file watcher, something not viable in massive projects as WebKit; as I don’t use snippet (generic text templates), I also disable it; and finally, lsp-mode tries to format the code at typing, I don’t know how the code style is figured out, but in my experience, it’s always detected wrong, so I disabled it too. Finally, lsp-mode is launched when a text uses the c-mode-common, shared by c++-mode too. lsp-mode is launched deferred, meaning it’ll startup until the buffer is visible; this is important since we might want to delay ccls session creation until the buffer’s .dir-locals.el file is processed, where it is configured for the specific project.

And lastly, ccls-mode configuration, hooked until c-mode or c++-mode are loaded up in a deferred fashion (already explained).

It’s important to understand how ccls works in order to integrate it in our workflow of a specific project, since it might need to be configured using Emacs’ per-directory local variables.

We are living in a post-Makefile world (almost), proof of that is ccls, which instead of a makefile, it uses a compilation database, a record of the compile options used to build the files in a project. It’s commonly described in JSON and it’s generated automatically by build systems such as meson or cmake, and later consumed by ninja or ccls to execute the compilation. Bear in mind that ccls uses a cache, which can eat a couple gigabytes of disk.

Now, let’s review the concrete details of using these features with WebKit. Let me assume that WebKit local repository is cloned in ~/WebKit.

As you may know, the cool way to compile WebKit is with flatpak. Flatpak adds an indirection in the compilation process, since it’s done in an isolated environment, above the native system. As a consequence, ccls has to be the one inside the Flatpak environment. In ~/.local/bin/webkit-ccls:

#!/bin/sh
set -eu
cd $HOME/WebKit/
exec Tools/Scripts/webkit-flatpak -c ccls "$@"

Basically the scripts calls ccls inside Flatpak, which is available in the SDK. And this is why ccls instead of clang, since clang is not provided.

By default ccls assumes the compilation database is in the project’s root directory, but in our case, it’s not, thus it is required to configure the database directory for our WebKit setup. For it, as we already said, a .dir-locals.el file is used.

((c-mode
  (indent-tabs-mode . nil)
  (c-basic-offset . 4))
 (c++-mode
  (indent-tabs-mode . nil)
  (c-basic-offset . 4))
 (java-mode
  (indent-tabs-mode . nil)
  (c-basic-offset . 4))
 (change-log-mode
  (indent-tabs-mode . nil))
 (nil
  (fill-column . 100)
  (ccls-executable . "/home/vjaquez/.local/bin/webkit-ccls")
  (ccls-initialization-options . (:compilationDatabaseDirectory "/app/webkit/WebKitBuild/Release"
                                  :cache (:directory ".ccls-cache")))
  (compile-command . "build-webkit --gtk --debug")))

As you can notice, ccls-execute is defined here, though it’s not a safe local variable. Also the ccls-initialization-options, which is a safe local variable. It is important to notice that the compilation database directory is a path inside Flatpak, and always use the Release path. I don’t understand why, but Debug path didn’t work for me. This mean that WebKit should be compiled as Release frequently, even if we only use Debug type for coding (as you may see in my compile-command).

Update: Now we can explain why it’s important to configure lsp-mode as deferred: to avoid connections to ccls before processing the .dir-locals.el file.

And that’s all. Now I have early programming errors detection, auto-completion, and so on. I hope you find these notes helpful.

Update: Sadly, because of Flatpak indirection, symbols’ definition finding won’t work because the file paths stored in ccls cache are relative to Flatpak’s file system. For that I still rely on global and its Emacs mode.

Review of Igalia Multimedia activities (2020/H1)

2020-09-07T00:00:00Z

This blog post is a review of the various activities the Igalia Multimedia team was involved in during the first half of 2020.Our previous reports are:

Just before a new virus turned into pandemics we could enjoy our traditional FOSDEM. There, our colleague Phil gave a talk about many of the topics covered in this report.

GstWPE #

GstWPE’s wpesrc element, produces a video texture representing a web page rendered off-screen by WPE.

We have worked on a new iteration of the GstWPE demo, focusing on one-to-many, web-augmented overlays, broadcasting with WebRTC and Janus.

https://www.youtube.com/watch?v=QNZJYOuVGiE

Also, since the merge of gstwpe plugin in gst-plugins-bad (staging area for new elements) new users have come along spotting rough areas and improving the element along the way.

Video Editing #

GStreamer Editing Services (GES) is a library that simplifies the creation of multimedia editing applications. It is based on the GStreamer multimedia framework and is heavily used by Pitivi video editor.

Implemented frame accuracy in the GStreamer Editing Services (GES) #

As required by the industry, it is now possible to reference all time in frame number, providing a precise mapping between frame number and play time. Many issues were fixed in GStreamer to reach the precision enough for make this work. Also intensive regression tests were added.

Implemented time effects support in GES #

Important refactoring inside GStreamer Editing Services have happened to allow cleanly and safely change playback speed of individual clips.

Implemented reverse playback in GES #

Several issues have been fixed inside GStreamer core elements and base classes in order to support reverse playback. This allows us to implement reliable and frame accurate reverse playback for individual clips.

Implemented ImageSequence support in GStreamer and GES #

Since OpenTimelineIO implemented ImageSequence support, many users in the community had said it was really required. We reviewed and finished up imagesequencesrc element, which had been awaiting review for years.

This feature is now also supported in the OpenTimelineIO GES adapter.

Optimized nested timelines preroll time by an order of magnitude*

Caps negotiation, done while the pipeline transitions from pause state to playing state, testing the whole pipeline functionality, was the bottleneck for nested timelines, so pipelines were reworked to avoid useless negotiations. At the same time, other members of GStreamer community have improved caps negotiation performance in general.

Last but not least, our colleague Thibault gave a talk in The Pipeline Conference about The Motion Picture Industry and Open Source Software: GStreamer as an Alternative, explaining how and why GStreamer could be leveraged in the motion picture industry to allow faster innovation, and solve issues by reusing all the multi-platform infrastructure the community has to offer.

WebKit multimedia #

There has been a lot of work on WebKit multimedia, particularly for WebKitGTK and WPE ports which use GStreamer framework as backend.

WebKit Flatpak SDK #

But first of all we would like to draw readers attention to the new WebKit Flatpak SDK. It was not a contribution only from the multimedia team, but rather a joint effort among different teams in Igalia.

Before WebKit Flatpak SDK, JHBuild was used for setting up a WebKitGTK/WPE environment for testing and development. Its purpose to is to provide a common set of well defined dependencies instead of relying on the ones available in the different Linux distributions, which might bring different outputs. Nonetheless, Flatpak offers a much more coherent environment for testing and develop, isolated from the rest of the building host, approaching to reproducible outputs.

Another great advantage of WebKit Flatpak SDK, at least for the multimedia team, is the possibility of use gst-build to setup a custom GStreamer environment, with latest master, for example.

Now, for sake of brevity, let us sketch an non-complete list of activities and achievements related with WebKit multimedia.

General multimedia #

Upstreamed external-oes support for hardware-accelerated video rendering. On some GPUs the presence of native YUV samplers allows the video sink to optimize rendering.
Allow video assets in <img>. This can be used as replacement for GIFs. For example, here is a demo and a blog post to explain it lengthy.
WhatsApp fixes related with blob URIs playback and buffering.
Added GstGLContext which a singleton wrapper of WebKit’s GL shared context wrapper.
Progress with GStreamer support for WebInspector. It adds a new tab in the inspector allowing users to view pipeline graphs.
Progress with Internal audio mixer. It is related with reducing the number of PulseAudio clients WebKit creates, by doing mixing internally and maintain only one connection with PulseAudio.
Ongoing work on sandboxing GStreamer pipelines in the new WebKit’s GPUProcess.
General WebKit multimedia and layout tests bug fixing.

Media Source Extensions (MSE) #

Fixed delayed audio and seek issues.
Preparation for the new MSE backend involved a large playbin3 track switching.

Encrypted Media Extension (EME) #

One of the major results of this first half, is the upstream of ThunderCDM, which is an implementation of a Content Decryption Module, providing Widevine decryption support. Recently, our colleague Xabier, published a blog post on this regard.

And it has enabled client-side video rendering support, which ensures video frames remain protected in GPU memory so they can’t be reached by third-party. This is a requirement for DRM/EME.

WebRTC #

Collaborated with the upgrade of LibWebRTC to M82.

GStreamer #

Though we normally contribute in GStreamer with the activities listed above, there are other tasks not related with WebKit. Among these we can enumerate the following:

Contributed to the new stateless GStreamer codecs library.
A new H264 VAAPI video decoder implementation
Helped with the AV1 parser review, a task that has been followed since we were Google Summer of Code mentors last year.
Contributed a new Rust GStreamer plugin for audio noise removal support

GStreamer VAAPI #

Reviewed a lot of patches.
Support for media-driver (iHD), the new VAAPI driver for Intel, mostly for Gen9 onwards. There are a lot of features with this driver.
A new vaapioverlay element.
Deep code cleanups. Among these we would like to mention:
- Added quirk mechanism for different backends.
- Change base classes to GstObject and GstMiniObject of most of classes and buffers types.
Enhanced caps negotiation given current driver’s constraints

Conclusions #

The multimedia team in Igalia has keep working, along the first half of this strange year, in our three main areas: browsers (mainly on WebKitGTK and WPE), video editing and GStreamer framework.

We worked adding and enhancing WebKitGTK and WPE multimedia features in order to offer a solid platform for media providers.

We have enhanced the Video Editing support in GStreamer.

And, along these tasks, we have contributed as much in GStreamer framework, particularly in hardware accelerated decoding and encoding and VA-API.

New VA-API H.264 decoder in gst-plugins-bad

2020-07-10T00:00:00Z

Recently, a new H.264 decoder, using VA-API, was merged in gst-plugins-bad.Why another VA-based H.264 decoder if there is already gstreamer-vaapi?

As usual, an historical perspective may give some clues.

It started when Seungha Yang implemented the GStreamer decoders for Windows using DXVA2 and D3D11 APIs.

Perhaps we need one step back and explain what are stateless decoders.

Video decoders are magic and opaque boxes where we push encoded frames, and later we’ll pop full decoded frames in raw format. This is how OpenMAX and V4L2 decoders work, for example.

Internally we can imagine those magic and opaque boxes has two main operations:

Codec state handling
Signal processing like Fourier-related transformations (such as DCT), entropy coding, etc. (DSP, in general)

The codec state handling basically extracts, from the stream, the frame’s parameters and its compressed data, so the DSP algorithms can decode the frames. Codec state handling can be done with generic CPUs, while DSP algorithms are massively improved through specific purpose processors.

These video decoders are known as stateful decoders, and usually they are distributed through binary and closed blobs.

Soon, silicon vendors realized they can offload the burden of state handling to third-party user-space libraries, releasing what it is known as stateless decoders. With them, your code not only has to push frames into the opaque box, but now it shall handle the codec specifics to provide all the parameters and references for each frame. VAAPI and DXVA2 are examples of those stateless decoders.

Returning to Seungha’s implementation, in order to get theirs DXVA2/D3D11 decoders, they also needed a state handler library for each codec. And Seungha wrote that library!

Initially they wanted to reuse the state handling in gstreamer-vaapi, which works pretty good, but its internal library, from the GStreamer perspective, is over-engineered: it is impossible to rip out only the state handling without importing all its data types. Which is kind of sad.

Later, Nicolas Dufresne, realized that this library can be re-used by other GStreamer plugins, because more stateless decoders are now available, particularly V4L2 stateless, in which he is interested. Nicolas moved Seungha’s code into a library in gst-plugins-bad.

Currently, libgstcodecs provides state handling of H.264, H.265, VP8 and VP9.

Let’s return to our original question: Why another VA-based H.264 decoder if there is already one in gstreamer-vaapi?

The quick answer is «to pay my technical debt».

As we already mentioned, gstreamer-vaapi is big and over-engineered, though we have being simplifying the internal libraries, in particular He Junyan, has done a lot of work replacing the internal base class, GstVaapiObject, withGstObject or GstMiniObject. Also, this kind of projects, where there’s a lot of untouched code, it carries a lot of cargo cult decisions.

So I took the libgstcodecs opportunity to write a simple, thin and lean, H.264 decoder, using VA new API calls (vaExportSurfaceHandle(), for example) and learning from other implementations, such as FFmpeg and ChromeOS. This exercise allowed me to identify where are the dusty spots in gstreamer-vaapi and how they should be fixed (and we have been doing it since then!).

Also, this opportunity lead me to learn a bit more about the H.264 specification since I implemented the reference picture list handling, and fixed a small bug in Chromium.

Now, let me be crystal clear: GStreamer VA-API is not going anywhere. It is, right now, one of the most feature-complete implementations using VA-API, even with its integration issues, and we are working on them, particularly, Intel folks are working hard on a new AV1 decoder, enhancing encoders and adding new video post-processing features.

But, this new vah264dec is an experimental VA-API decoder, which aims towards a tight integration with GStreamer, oriented to provide a good experience in most of the common use cases and to enhance the common libgstcodecs library shared with other stateless decoders, looking to avoid Intel specific nuances.

These are the main characteristics and plans of this new decoder:

It use, by default, a DRM connection to VA display, avoiding the troubles of choosing X11 or Wayland.
- It uses the first found DRM device as VA display
- In the future, users will be able to provide their custom VA display through the pipeline’s context.
It requires libva >= 1.6
No multiview/stereo profiles, neither interlaced streams, because libgstcodecs doesn’t handle them yet
It is incompatible with gstreamer-vaapi: mixing elements might lead to problems.
Even if memory:VAMemory is exposed, it is not handled yet by any other element yet.
- Users will get VASurfaces via mapping as GstGL does with textures.
Caps templates are generated dynamically generated by querying VAAPI
YV12 and I420 are added for system memory caps because they seem to be supported for all the drivers when downloading frames onto main memory, as they are used by xvimagesink and others, avoiding color conversion.
Decoding surfaces aren’t bounded to context, so they can grow beyond the DBP size, allowing smooth reverse playback.
There isn’t yet error handling and recovery.
The element is supposed to spawn if different renderD nodes with VA-API driver support are found (like gstv4l2), but it hasn’t been tested yet.

Now you may be asking how do I use vah264dec?

Currently vah264dec has NONE rank, which means that it will never be autoplugged, but you can use the trick of the environment variable GST_PLUGIN_FEATURE_RANK:

GST_PLUGIN_FEATURE_RANK=vah264dec:259 gst-play-1.0 ~/video.mp4

And that’s it!

Thanks.

WebKit Flatpak SDK and gst-build

2020-06-16T00:00:00Z

This post is an annex of Phil’s Introducing the WebKit Flatpak SDK. Please make sure to read it, if you haven’t already.Recapitulating, nowadays WebKitGTK/WPE developers —and their CI infrastructure— are moving towards to Flatpak-based environment for their workflow. This Flatpak-based environment, or Flatpak SDK for short, can be visualized as a software sandboxed-container, which bundles all the dependencies required to compile, run and debug WebKitGTK/WPE.

In a day-by-day work, this approach removes the potential compilation of the world in order to obtain reproducible builds, improving the development and testing work flow.

But what if you are also involved in the development of one dependency?

This is the case of Igalia’s multimedia team where, besides developing the multimedia features for WebKitGTK and WPE, we also participate in the GStreamer development, the framework used for multimedia.

Because of this, in our workflow we usually need to build WebKit with a fix, hack or new feature in GStreamer. Is it possible to add in Flatpak our custom GStreamer build without messing its own GStreamer setup? Yes, it’s possible.

gst-build is a set of scripts in Python which clone GStreamer repositories, compile them and setup an uninstalled environment. This uninstalled environment allows a transient usage of the compiled framework from their build tree, avoiding installation and further mess up with our system.

The WebKit scripts that wraps Flatpak operations are also capable to handle the scripts of gst-build to build GStreamer inside the container, and, when running WebKit’s artifacts, the scripts enable the mentioned uninstalled environment, overloading Flatpak’s GStreamer.

How do we unveil all this magic?

First of all, setup a gst-build installation as it is documented. In this installation is were the GStreamer plumbing is done.

Later, gst-build operations through WebKit compilation scripts are enabled when the environment variable GST_BUILD_PATH is exported. This variable should point to the directory where the gst-build tree is placed.

And that’s all!

But let’s put these words in actual commands. The following workflow assumes that WebKit repository is cloned in ~/WebKit and the gst-build tree is in ~/gst-build (please, excuse my bashisms).

Compiling WebKitGTK with symbols, using LLVM as toolchain (this command will also compile GStreamer):

cd ~/WebKit
CC=clang CXX=clang++ GST_BUILD_PATH=/home/vjaquez/gst-build Tools/Scripts/build-webkit --gtk --debug

Running the generated minibrowser (remind GST_BUILD_PATH is required again for a correct linking):

GST_BUILD_PATH=/home/vjaquez/gst-build Tools/Scripts/run-minibrowser --gtk --debug

Running media layout tests:

GST_BUILD_PATH=/home/vjaquez/gst-build ./Tools/Scripts/run-webkit-tests --gtk --debug media

But wait! There’s more….

What if you I want to parameterize the GStreamer compilation. To say, I would like to enable a GStreamer module or disable the built of a specific element.

gst-build, as the rest of GStreamer modules, uses meson build system, so it’s possible to pass arguments to meson through the environment variable GST_BUILD_ARGS.

For example, I would like to enable gstreamer-vaapi 😇

cd ~/WebKit
CC=clang CXX=clang++ GST_BUILD_PATH=/home/vjaquez/gst-build GST_BUILD_ARGS="-Dvaapi=enabled" Tools/Scripts/build-webkit --gtk --debug

Review of the Igalia Multimedia team Activities (2019/H2)

2020-03-16T00:00:00Z

This blog post is a review of the various activities the Igalia Multimedia team was involved along the second half of 2019.Here are the previous 2018/H2 and 2019/H1 reports.

GstWPE #

Succinctly, GstWPE is a GStreamer plugin which allows to render web-pages as a video stream where it frames are GL textures.

Phil, its main author, wrote a blog post explaining at detail what is GstWPE and its possible use-cases. He wrote a demo too, which grabs and previews a live stream from a webcam session and blends it with an overlay from wpesrc, which displays HTML content. This composited live stream can be broadcasted through YouTube or Twitch.

These concepts are better explained by Phil himself in the following lighting talk, presented at the last GStreamer Conference in Lyon:

https://www.youtube.com/watch?v=-lxm4EfpfAc

Video Editing #

After implementing a deep integration of the GStreamer Editing Services (a.k.a GES) into Pixar’s OpenTimelineIO during the first half of 2019, we decided to implement an important missing feature for the professional video editing industry: nested timelines.

Toward that goal, Thibault worked with the GSoC student Swayamjeet Swain to implement a flexible API to support nested timelines in GES. This means that users of GES can now decouple each scene into different projects when editing long videos. This work is going to be released in the upcoming GStreamer 1.18 version.

Henry Wilkes also implemented the support for nested timeline in OpenTimelineIO making GES integration one of the most advanced one as you can see on that table:

Feature	OTIO	EDL	FCP7 XML	FCP X	AAF	RV	ALE	GES
Single Track of Clips	✔	✔	✔	✔	✔	W-O	✔	✔
Multiple Video Tracks	✔	✖	✔	✔	✔	W-O	✔	✔
Audio Tracks & Clips	✔	✔	✔	✔	✔	W-O	✔	✔
Gap/Filler	✔	✔	✔	✔	✔	✔	✖	✔
Markers	✔	✔	✔	✔	✖	N/A	✖	✔
Nesting	✔	✖	✔	✔	✔	W-O	✔	✔
Transitions	✔	✔	✖	✖	✔	W-O	✖	✔
Audio/Video Effects	✖	✖	✖	✖	✖	N/A	✖	✔
Linear Speed Effects	✔	✔	✖	✖	R-O	✖	✖	✖
Fancy Speed Effects	✖	✖	✖	✖	✖	✖	✖	✖
Color Decision List	✔	✔	✖	✖	✖	✖	N/A	✖

Along these lines, Thibault delivered a 15 minutes talk, also in the GStreamer Conference 2019:

https://www.youtube.com/watch?v=P6iMcUWPqpA

After detecting a few regressions and issues in GStreamer, related to frame accuracy, we decided to make sure that we can seek in a perfectly frame accurate way using GStreamer and the GStreamer Editing Services. In order to ensure that, an extensive integration testsuite has been developed, mostly targeting most important container formats and codecs (namely mxf, quicktime, h264, h265, prores, jpeg) and issues have been fixed in different places. On top of that, new APIs are being added to GES to allow expressing times in frame number instead of nanoseconds. This work is still ongoing but should be merged in time for GStreamer 1.18.

GStreamer Validate Flow #

GstValidate has been turning into one of the most important GStreamer testing tools to check that elements behave as they are supposed to do in the framework.

Along with our MSE work, we found that other way to specify tests, related with produced buffers and events through specific pads, was needed. Thus, Alicia developed a new plugin for GstValidate: Validate Flow.

Alicia gave an informative 30 minutes talk about GstValidate and the new plugin in the last GStreamer Conference too:

https://www.youtube.com/watch?v=_NiHTKax6yg

GStreamer VAAPI #

Most of the work along the second half of 2019 were maintenance tasks and code reviews.

We worked mainly on memory restrictions per backend driver, and we reviewed a big refactor: internal encoders now use GstObject, instead of the custom GstVaapiObject. Also we reviewed patches for new features such as video rotation and cropping in vaapipostproc.

Servo multimedia #

Last year we worked integrating media playing in Servo. We finally delivered hardware accelerated video playback in Linux and Android. We worked also for Windows and Mac ports but they were not finished. As natural, most of the work were in servo/media crate, pushing code and reviewing contributions. The major tasks were to rewrite the media player example and the internal source element looking to handle the download playbin’s flag properly.

We also added WebGL integration support with <video> elements, thus webpages can use video frames as WebGL textures.

Finally we explored how to isolate the multimedia processing in a dedicated thread or process, but that task remains pending.

WebKit Media Source Extension #

We did a lot of downstream and upstream bug fixing and patch review, both in WebKit and GStreamer, for our MSE GStreamer-based backend.

Along this line we improved WebKitMediaSource to use playbin3 but also compatibility with older GStreamer versions was added.

WebKit WebRTC #

Most of the work in this area were maintenance and fix regressions uncovered by the layout tests. Besides, the support for the Rasberry Pi was improved by handling encoded streams from v4l2 video sources, with some explorations with Minnowboard on top of that.

Conferences #

GStreamer Conference #

Igalia was Gold sponsor this last GStreamer Conference held in Lyon, France.

All team attended and five talks were delivered. Only Thibault presented, besides the video editing one which we already referred, another two more: One about GstTranscoder API and the other about the new documentation infrastructure based in Hotdoc:

https://www.youtube.com/watch?v=AEkqLuDV5PI

https://www.youtube.com/watch?v=1q7i_w7ggWw

We also had a productive hackfest, after the conference, where we worked on AV1 Rust decoder, HLS Rust demuxer, hardware decoder flag in playbin, and other stuff.

Linaro Connect #

Phil attended the Linaro Connect conference in San Diego, USA. He delivered a talk about WPE/Multimedia which you can enjoy here:

https://www.youtube.com/watch?v=9vomDBGk8N0

Demuxed #

Charlie attended Demuxed, in San Francisco. The conference is heavily focused on streaming and codec engineering and validation. Sadly there are not much interest in GStreamer, as the main focus is on FFmpeg.

RustFest #

Phil and I attended the last RustFest in Barcelona. Basically we went to meet with the Rust community and we attended the “WebRTC with GStreamer-rs” workshop presented by Sebastian Dröge.

GStreamer-VAAPI 1.16 and libva 2.6 in Debian

2020-01-08T00:00:00Z

Debian has migrated libva 2.6 into testing. This release includes a pull request that changes how the drivers are selected to be loaded and used. As the pull request mentions:

   libva will try to load iHD firstly, if it failed. then it will load i965.

Also, Debian testing has imported that iHD driver with two flavors: intel-media-driver and intel-media-driver-non-free. So basically iHD driver is now the main VAAPI driver for Intel platforms, though it only supports the new chips, the old ones still require i965-va-driver.

Sadly, for current GStreamer-VAAPI stable, the iHD driver is not included in its driver white list. And this will pose a problem for users that have installed either of the intel-media-driver packages, because, by default, such driver is ignored and the VAAPI GStreamer elements won’t be registered.

There are three temporal workarounds (mutually excluded) for those users (updated):

Uninstall intel-media-driver* and install (or keep) the old i965-va-driver-shaders/i965-va-driver.
Export, by default in your session, export LIBVA_DRIVER_NAME=i965. Normally this is done adding the variable exportation in $HOME/.profile file. This environment variable will force libva to load the i965 driver.
And finally, export, by default in your sessions, GST_VAAPI_ALL_DRIVERS=1. This is not advised since many applications, such as Epiphany, might fail.

We prefer to not include iHD in the stable white list because most of the work done for that driver has occurred after release 1.16.

In the case of GStreamer-VAAPI master branch (actively in develop) we have merged the iHD in the white list, since the Intel team has been working a lot to make it work. Though, it will be released for GStreamer version 1.18.

Generating a GStreamer-1.14 bundle for TravisCI with Ubuntu/Trusty

2019-02-13T00:00:00Z

For having continuous integration in your multimedia project hosted in GitHub with TravisCI, you may want to compile and run tests with a recent version of GStreamer. Nonetheless, TravisCI mainly offers Ubuntu Trusty as one of the possible distributions to deploy in their CI, and that distribution packages GStreamer 1.2, which might be a bit old for your project’s requirements.A solution for this problem is to provide to TravisCI your own GStreamer bundle with the version you want to compile and test on you project, in this case 1.14. The present blog is recipe I followed to generate that GStreamer bundle with GstGL support.

There are three main issues:

The packaged libglib version is too old, hoping that we will not find an ABI breakage while running the CI.
The packaged FFmpeg version is too old
As we want to compile GStreamer using gst-build, we need a recent version of meson, which requires python3.5, not available in Trusty.

schroot #

Old habits die hard, and I have used schroot for handle chroot environments without complains, it handles the bind mounting of /proc, /sys and all that repetitive stuff that seals the isolation of the chrooted environment.

The debootstrap’s variant I use is buildd because it installs the build-essential package.

sudo mkdir /srv/chroot/gst-trusty64
sudo debootstrap --arch=amd64 --variant=buildd trusty ./gst-trusty64/ http://archive.ubuntu.com/ubuntu
sudo vim /etc/schroot/chroot.d

This is the schroot configuration I will use. Please, adapt it to your need.

[gst]
description=Ubuntu Trusty 64-bit for GStreamer
directory=/srv/chroot/gst-trusty64
type=directory
users=vjaquez
root-users=vjaquez
profile=default
setup.fstab=default/vjaquez-home.fstab

I am overriding the fstab default file for a custom one where the home directory of vjaquez user aims to a clean directory.

mkdir -p ~/home-chroot/gst
sudo vim /etc/schroot/default/vjaquez-home.fstab

# fstab: static file system information for chroots.
# Note that the mount point will be prefixed by the chroot path
# (CHROOT_PATH)
#
#
/proc           /proc           none    rw,bind         0       0
/sys            /sys            none    rw,bind         0       0
/dev            /dev            none    rw,bind         0       0
/dev/pts        /dev/pts        none    rw,bind         0       0
/home           /home           none    rw,bind         0       0
/home/vjaquez/home-chroot/gst   /home/vjaquez   none    rw,bind 0       0
/tmp            /tmp            none    rw,bind         0       0

configure chroot environment #

We will get into the chroot environment as super user in order to add the required packages. For that purpose we add universe repository in apt.

libglib requires: autotools-dev gnome-pkg-tools libtool libffi-dev libelf-dev libpcre3-dev desktop-file-utils libselinux1-dev libgamin-dev dbus dbus-x11 shared-mime-info libxml2-utils
Python requires: libssl-dev libreadline-dev libsqlite3-dev
GStreamer requires: bison flex yasm python3-pip libasound2-dev libbz2-dev libcap-dev libdrm-dev libegl1-mesa-dev libfaad-dev libgl1-mesa-dev libgles2-mesa-dev libgmp-dev libgsl0-dev libjpeg-dev libmms-dev libmpg123-dev libogg-dev libopus-dev liborc-0.4-dev libpango1.0-dev libpng-dev libpulse-dev librtmp-dev libtheora-dev libtwolame-dev libvorbis-dev libvpx-dev libwebp-dev pkg-config unzip zlib1g-dev

And for general setup: language-pack-en ccache git curl

$ schroot --user root --chroot gst
(gst)# sed -i "s/main$/main universe/g" /etc/apt/sources.list
(gst)# apt update
(gst)# apt upgrade
(gst)# apt --no-install-recommends --no-install-suggests install \
autotools-dev gnome-pkg-tools libtool libffi-dev libelf-dev \
libpcre3-dev desktop-file-utils libselinux1-dev libgamin-dev dbus \
dbus-x11 shared-mime-info libxml2-utils \
libssl-dev libreadline-dev libsqlite3-dev \
language-pack-en ccache git curl bison flex yasm python3-pip \
libasound2-dev libbz2-dev libcap-dev libdrm-dev libegl1-mesa-dev \
libfaad-dev libgl1-mesa-dev libgles2-mesa-dev libgmp-dev libgsl0-dev \
libjpeg-dev libmms-dev libmpg123-dev libogg-dev libopus-dev \
liborc-0.4-dev libpango1.0-dev libpng-dev libpulse-dev librtmp-dev \
libtheora-dev libtwolame-dev libvorbis-dev libvpx-dev libwebp-dev \
pkg-config unzip zlib1g-dev

Finally we create our installation prefix. In this case /opt/gst to avoid the contamination of /usr/local and logout as root.

(gst)# mkdir -p /opt/gst
(gst)# chown vjaquez /opt/gst
(gst)# exit

compile FFmpeg 3.2 #

Now, let’s login again, but as the unprivileged user, to build the bundle, starting with FFmpeg. Notice that we are using ccache and building out-of-source.

$ schroot --chroot gst
(gst)$ git clone https://git.ffmpeg.org/ffmpeg.git ffmpeg
(gst)$ cd ffmpeg
(gst)$ git checkout -b work n3.2.12
(gst)$ mkdir build
(gst)$ cd build
(gst)$ ../configure --disable-static --enable-shared \
--disable-programs --enable-pic --disable-doc --prefix=/opt/gst
(gst)$ PATH=/usr/lib/ccache/:${PATH} make -j8 install

compile glib 2.48 #

(gst)$ cd ~
(gst)$ git clone https://gitlab.gnome.org/GNOME/glib.git
(gst)$ cd glib
(gst)$ git checkout -b work origin/glib-2-48
(gst)$ mkdir mybuild
(gst)$ cd mybuild
(gst)$ ../autogen.sh --prefix=/opt/gst
(gst)$ PATH=/usr/lib/ccache/:${PATH} make -j8 install

install Python 3.5 #

Pyenv is a project that allows the automation of installing and executing, in the user home directory, multiple versions of Python.

(gst)$ curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash
(gst)$ ~/.pyenv/bin/pyenv install 3.5.0

Install meson 0.50 #

We will install the last available version of meson in the user home directory, that is why PATH is extended and exported.

(gst)$ cd ~
(gst)$ ~/.pyenv/verion/3.5.0/pip3 install --user meson
(gst)$ export PATH=${HOME}/.local/bin:${PATH}

build GStreamer 1.14 #

PKG_CONFIG_PATH is exported to expose the compiled versions of FFmpeg and glib. Notice that the libraries are installed in /opt/lib in order to avoid the dispersion of pkg-config files.

(gst)$ cd ~/
(gst)$ export PKG_CONFIG_PATH=/opt/gst/lib/pkgconfig/
(gst)$ git clone https://gitlab.freedesktop.org/gstreamer/gst-build.git
(gst)$ cd gst-build
(gst)$ git checkout -b work origin/1.14
(gst)$ meson -Denable_python=false \
-Ddisable_gst_libav=false -Ddisable_gst_plugins_ugly=true \
-Ddisable_gst_plugins_bad=false -Ddisable_gst_devtools=true \
-Ddisable_gst_editing_services=true -Ddisable_rtsp_server=true \
-Ddisable_gst_omx=true -Ddisable_gstreamer_vaapi=true \
-Ddisable_gstreamer_sharp=true -Ddisable_introspection=true \
--prefix=/opt/gst build --libdir=lib
(gst)$ ninja -C build install

testing #

(gst)$ cd ~/
(gst)$ LD_LIBRARY_PATH=/opt/gst/lib \
GST_PLUGIN_SYSTEM_PATH=/opt/gst/lib/gstreamer-1.0/ \
/opt/gst/bin/gst-inspect-1.0

And the list of available elements shall be shown.

archive the bundle #

(gst)$ cd ~/
(gst)$ tar zpcvf gstreamer-1.14-x86_64-linux-gnu.tar.gz -C /opt ./gst

update your .travis.yml #

These are the packages you shall add to run this generated GStreamer bundle:

libasound2-plugins
libfaad2
libfftw3-single3
libjack-jackd2-0
libmms0
libmpg123-0
libopus0
liborc-0.4-0
libpulsedsp
libsamplerate0
libspeexdsp1
libtdb1
libtheora0
libtwolame0
libwayland-egl1-mesa
libwebp5
libwebrtc-audio-processing-0
liborc-0.4-dev
pulseaudio
pulseaudio-utils

And this is the before_install and before_script targets:

     before_install:
       - curl -L http://server.example/gstreamer-1.14-x86_64-linux-gnu.tar.gz | tar xz
       - sed -i "s;prefix=/opt/gst;prefix=$PWD/gst;g" $PWD/gst/lib/pkgconfig/*.pc
       - export PKG_CONFIG_PATH=$PWD/gst/lib/pkgconfig
       - export GST_PLUGIN_SYSTEM_PATH=$PWD/gst/lib/gstreamer-1.0
       - export GST_PLUGIN_SCANNER=$PWD/gst/libexec/gstreamer-1.0/gst-plugin-scanner
       - export PATH=$PATH:$PWD/gst/bin
       - export LD_LIBRARY_PATH=$PWD/gst/lib:$LD_LIBRARY_PATH

     before_script:
       - pulseaudio --start
       - gst-inspect-1.0 | grep Total

Review of Igalia's Multimedia Activities (2018/H2)

2019-02-11T00:00:00Z

This is the first semiyearly report about Igalia’s activities around multimedia, covering the second half of 2018.Great length of this report was exposed in Phil’s talk surveying multimedia development in WebKitGTK and WPE:

https://www.youtube.com/watch?v=jGaW_UofoYA

WebKit Media Source Extensions (MSE) #

MSE is a specification that allows JS to generate media streams for playback for Web browsers that support HTML 5 video and audio.

Last semester we upstreamed the support to WebM format in WebKitGTK with the related patches in GStreamer, particularly in qtdemux, matroskademux elements.

WebKit Encrypted Media Extensions (EME) #

EME is a specification for enabling playback of encrypted content in Web browsers that support HTML 5 video.

In a downstream project for WPE WebKit we managed to have almost full test coverage in the YoutubeTV 2018 test suite.

We merged our contributions in upstream, WebKit and GStreamer, most of what is legal to publish, for example, making demuxers aware of encrypted content and make them to send protection events with the initialization data and the encrypted caps, in order to select later the decryption key.

We started to coordinate the upstreaming process of a new implementation of CDM (Content Decryption Module) abstraction and there will be even changes in that abstraction.

Lighting talk about EME implementation in WPE/WebKitGTK in GStreamer Conference 2018.

https://www.youtube.com/watch?v=ZenVmiiVfGw

WebKit WebRTC #

WebRTC consists of several interrelated APIs and real time protocols to enable Web applications and sites to captures audio, or A/V streams, and exchange them between browsers without requiring an intermediary.

We added GStreamer interfaces to LibWebRTC, to use it for the network part, while using GStreamer for the media capture and processing. All that was upstreamed in 2018 H2.

Thibault described thoroughly the tasks done for this achievement.

Talk about WebRTC implementation in WPE/WebKitGTK in WebEngines hackfest 2018.

https://www.youtube.com/watch?v=LHSVMWOYViw

Servo/media #

Servo is a browser engine written in Rust designed for high parallelization and high GPU usage.

We added basic support for <video> and <audio> media elements in Servo. Later on, we added the GStreamerGL bindings for Rust in gstreamer-rs to render GL textures from the GStreamer pipeline in Servo.

Lighting talk in the GStreamer Conference 2018.

https://www.youtube.com/watch?v=yLyZuIBSU4g

GstWPE #

Taking an idea from the GStreamer Conference, we developed a GStreamer source element that wraps WPE. With this source element, it is possible to blend a web page and video in a single video stream; that is, the output of a Web browser (to say, a rendered web page) is used as a video source of a GStreamer pipeline: GstWPE. The element is already merged in the gst-plugins-bad repository.

Talk about GstWPE in FOSDEM 2019

https://www.youtube.com/watch?v=no7rvUk8GqM

Demo #1

https://www.youtube.com/watch?v=1JUa3Ldb3vE

Demo #2

https://www.youtube.com/watch?v=J2eb-KRk9gw

GStreamer VA-API and gst-MSDK #

At last, but not the least, we continued helping with the maintenance of GStreamer-VAAPI and gst-msdk, with code reviewing and on-going migration of the internal library to GObject.

Other activities #

The second half of 2018 was also intense in terms of conferences and hackfest for the team:

GStreamer conference in Edinburgh.
Web Engines Hackfest in Coruña.
WebKit meeting in Silicon Valley
Finally, we participated as mentors in the last Google Summer of Code. Georg Ottinger, among other completed tasks, sketched an AV1 parser for GStreamer, though it needs a couple more iterations to get it merged.

Thanks to bear with us along all this blog post and to keeping under your radar our work.

Rust bindings for GStreamerGL: Memoirs

2019-01-16T00:00:00Z

Rust is a great programming language but the community around it’s just amazing. Those are the ingredients for the craft of useful software tools, just like Servo, an experimental browser engine designed for tasks isolation and high parallelization.Both projects, Rust and Servo, are funded by Mozilla.

Thanks to Mozilla and Igalia I have the opportunity to work on Servo, adding it HTML5 multimedia features.

First, with the help of Fernando Jiménez, we finished what my colleague Philippe Normand and Sebastian Dröge (one of my programming heroes) started: a media player in Rust designed to be integrated in Servo. This media player lives in its own crate: servo/media along with the WebAudio engine. A crate, in Rust jargon, is like a library. This crate is (very ad-hocly) designed to be multimedia framework agnostic, but the only backend right now is for GStreamer. Later we integrated it into Servo adding an initial support for audio and video tags.

Currently, servo/media passes, through a IPC channel, the array with the whole frame to render in Servo. This implies, at least, one copy of the frame in memory, and we would like to avoid it.

For painting and compositing the web content, Servo uses WebRender, a crate designed to use the GPU intensively. Thus, if instead of raw frame data we pass OpenGL textures to WebRender the performance could be enhanced notoriously.

Luckily, GStreamer already supports the uploading, downloading, painting and composition of video frames as OpenGL textures with the OpenGL plugin and its OpenGL Integration library. Even more, with plugins such as GStreamer-VAAPI, Gst-OMX (OpenMAX), and others, it’s possible to process video without using the main CPU or its mapped memory in different platforms.

But from what’s available in GStreamer to what it’s available in Rust there’s a distance. Nonetheless, Sebastian has putting a lot of effort in the Rust bindings for GStreamer, either for applications and plugins, sadly, GStreamer’s OpenGL Integration library (GstGL for short) wasn’t available at that time. So I rolled up my sleeves and got to work on the bindings.

These are the stories of that work.

As GStreamer shares with GTK+ the GObject framework and its introspection mechanism, both projects have collaborated on the required infrastructure to support Rust bindings. Thanks to all the GNOME folks who are working on the intercommunication between Rust and GObject. The quest has been long and complex, since Rust doesn’t map all the object oriented concepts, and GObject, being a set of practices and software helpers to do object oriented programming with C, its usage is not homogeneous.

The Rubicon that ease the generation of Rust bindings for GObject-based projects is GIR, a tool, written in Rust, that reads gir files, along with metadata in toml, and outputs two types of bindings: sys and api.

Rust can call external functions through FFI (foreign function interface), which is just a declaration of a C function with Rust types. But these functions are considered unsafe. The sys bindings, are just the exporting of the C function for the library organized by the library’s namespace.

The next step is to create a safe and rustified API. This is the api bindings.

As we said, GObject libraries are quite homogeneous, and even following the introspection annotations, there will be cases where GIR won’t be able to generate the correct bindings. For that reason GIR is constantly evolving, looking for a common way to solve the corner cases that exist in every GObject project. For example, these are my patches in order to generate the GstGL bindings.

The done tasks were:

For this document we assume that the reader has a functional Rust setup and they know the basic concepts.

Clone and build gir

cd ~/ws
git clone https://github.com/gtk-rs/gir.git
cd gir
cargo build --release

the reason to build gir in release mode is because, otherwise would be very slow.

For sys bindings.

These kind of bindings are normally straight forward (and unsafe) since they only map the C API to Rust via FFI mechanism.

cd ~/ws
git clone https://gitlab.freedesktop.org/gstreamer/gstreamer-rs-sys.git
cd gstreamer-rs-sys
cp /usr/share/gir-1.0/GstGL-1.0.gir gir-files/

Verify if the gir file is more o less correct
1. If there something strange, we should fix the code that generated it.
2. If that is not possible, the last resource is to fix the gir file directly, which is just XML, not manually but through a script using xmlstartlet. See fix.sh in gtk-rs as example.
Create the toml file with the metadata required to create the bindings. In other words, this file contains the exceptions, rules and options used by the tool to generated the bindings. See Gir_GstGL.toml in gstreamer-rs-sys as example. The documentation of the toml file is in the gir’s README.md file.
```
~/ws/gir/target/release/gir -c Gir_GstGL.toml
```

This command will generate, as specified in the toml file (target_path), a crate in the directory named gstreamer-gl-sys.

Api bindings.

These type of bindings may require more manual work since their purpose is to offer a rustified API of the library, with all its syntactic sugar, semantics, and so on. But in general terms, the process is similar:

cd ~/ws
git clone https://gitlab.freedesktop.org/gstreamer/gstreamer-sys.git
cd gstreamer-sys
cp /usr/share/gir-1.0/GstGL-1.0.gir gir-files/

Again, it would be possible to end up applying fixes to the gir file through a fix.sh script using xmlstartlet.

And again, the confection of the toml file might take a lot of time, by trial and error, by cleaning and tidying up the API. See Gir_GstGL.toml in gstreamer-rs as example.

~/ws/gir/target/release/gir -c Gir_GstGL.toml

A good way to test your bindings is by crafting a test application, which shows how to use the API. Personally I devoted a ton of time in the test application for GstGL, but worth it. It made me aware of a missing part in the crate used for GL applications in Rust, named Glutin, which was a way to get the used EGLDisplay. So also worked on that and sent a pull request that was recently merged. The sweets of the free software development.

Nowadays I’m integrating GstGL API in servo/media and later, Servo!

Building gst-msdk with MediaSDK opensource

2018-11-23T00:00:00Z

I tried, several months ago, the open source version of Intel MediaSDK and it was a complete mess. In order to review some patches for gst-msdk I tried it again. I am surprised how the situation has improved since then.## Install dependencies

sudo apt get install libva-dev vainfo cmake ccache
sudo apt build-dep gstreamer1.0 gst-plugins-{base,good,bad}1.0
sudo apt remove libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev

Setting up the workplace #

sudo mkdir /opt/intel
sudo chown usuario:usuario /opt/intel
mkdir ~/msdk
cd ~/msdk

Build MediaSDK #

It will be built in its source directory: ~/msdk/MediaSDK/build

It will be installed in /opt/intel

git clone https://github.com/Intel-Media-SDK/MediaSDK.git
cd MediaSDK
mkdir build
cd build
cmake ..
make
make install

Build media-driver #

cd ~/msdk
git clone https://github.com/intel/media-driver.git
git clone https://github.com/intel/gmmlib.git
mkdir build
cd build
cmake ../media-driver
make

Let’s install media-driver in /opt/intel too

cd ~/msdk/build
cp ./media_driver/iHD_drv_video.so /opt/intel

But don’t remove, rename or move the directory ~/msdk/build because iHD_drv_video.so links against libigdgmm.so.5 which is there. Thus either you keep the directory or you install that library in a path searchable by the linker, or set the environment variable LD_LIBRARY_PATH

Test environment #

$ LIBVA_DRIVERS_PATH=/opt/intel LIBVA_DRIVER_NAME=iHD vainfo
  libva info: VA-API version 1.3.0
  libva info: va_getDriverName() returns -1
  libva info: User requested driver 'iHD'
  libva info: Trying to open /opt/intel/iHD_drv_video.so
  libva info: Found init function __vaDriverInit_1_3
  libva info: va_openDriver() returns 0
 vainfo: VA-API version: 1.3 (libva 2.2.0)
 vainfo: Driver version: Intel iHD driver - 1.0.0
 vainfo: Supported profile and entrypoints
   VAProfileNone                   : VAEntrypointVideoProc
   VAProfileNone                   : VAEntrypointStats
   VAProfileMPEG2Simple            : VAEntrypointVLD
   VAProfileMPEG2Simple            : VAEntrypointEncSlice
   VAProfileMPEG2Main              : VAEntrypointVLD
   VAProfileMPEG2Main              : VAEntrypointEncSlice
   VAProfileH264Main               : VAEntrypointVLD
   VAProfileH264Main               : VAEntrypointEncSlice
   VAProfileH264Main               : VAEntrypointFEI
   VAProfileH264Main               : VAEntrypointEncSliceLP
   VAProfileH264High               : VAEntrypointVLD
   VAProfileH264High               : VAEntrypointEncSlice
   VAProfileH264High               : VAEntrypointFEI
   VAProfileH264High               : VAEntrypointEncSliceLP
   VAProfileVC1Simple              : VAEntrypointVLD
   VAProfileVC1Main                : VAEntrypointVLD
   VAProfileVC1Advanced            : VAEntrypointVLD
   VAProfileJPEGBaseline           : VAEntrypointVLD
   VAProfileJPEGBaseline           : VAEntrypointEncPicture
   VAProfileH264ConstrainedBaseline: VAEntrypointVLD
   VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
   VAProfileH264ConstrainedBaseline: VAEntrypointFEI
   VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP
   VAProfileVP8Version0_3          : VAEntrypointVLD
   VAProfileHEVCMain               : VAEntrypointVLD
   VAProfileHEVCMain               : VAEntrypointEncSlice
   VAProfileHEVCMain               : VAEntrypointFEI

Setup gst-build #

It will be build in its source directory: ~/msdk/gst-build/build

cd ~/msdk
git clone https://gitlab.freedesktop.org/gstreamer/gst-build.git
cd gst-build
export INTELMEDIASDKROOT=/opt/intel/mediasdk
meson build -Dpython=disabled -Dgst-plugins-bad:msdk=enabled
ninja -C build

Check for build elements #

$ ninja -C ~/msdk/gst-build/build  uninstalled
[gst-master] $ GST_VAAPI_ALL_DRIVERS=1 \
               LIBVA_DRIVERS_PATH=/opt/intel \
               LIBVA_DRIVER_NAME=iHD \
               gst-inspect-1.0 | egrep "vaapi|msdk"
vaapi:  vaapijpegdec: VA-API JPEG decoder
vaapi:  vaapimpeg2dec: VA-API MPEG2 decoder
vaapi:  vaapih264dec: VA-API H264 decoder
vaapi:  vaapivc1dec: VA-API VC1 decoder
vaapi:  vaapivp8dec: VA-API VP8 decoder
vaapi:  vaapih265dec: VA-API H265 decoder
vaapi:  vaapipostproc: VA-API video postprocessing
vaapi:  vaapidecodebin: VA-API Decode Bin
vaapi:  vaapisink: VA-API sink
vaapi:  vaapimpeg2enc: VA-API MPEG-2 encoder
vaapi:  vaapih265enc: VA-API H265 encoder
vaapi:  vaapijpegenc: VA-API JPEG encoder
vaapi:  vaapih264enc: VA-API H264 encoder
msdk:  msdkvpp: MSDK Video Postprocessor
msdk:  msdkvc1dec: Intel MSDK VC1 decoder
msdk:  msdkvp8enc: Intel MSDK VP8 encoder
msdk:  msdkvp8dec: Intel MSDK VP8 decoder
msdk:  msdkmpeg2enc: Intel MSDK MPEG2 encoder
msdk:  msdkmpeg2dec: Intel MSDK MPEG2 decoder
msdk:  msdkmjpegenc: Intel MSDK MJPEG encoder
msdk:  msdkmjpegdec: Intel MSDK MJPEG decoder
msdk:  msdkh265enc: Intel MSDK H265 encoder
msdk:  msdkh265dec: Intel MSDK H265 decoder
msdk:  msdkh264enc: Intel MSDK H264 encoder
msdk:  msdkh264dec: Intel MSDK H264 decoder

Remember #

Remember to export these environment variables (perhaps you what to create a script file to set them):

export GST_VAAPI_ALL_DRIVERS=1
export LIBVA_DRIVERS_PATH=/opt/intel
export LIBVA_DRIVER_NAME=iHD

How to setup a gst-build environment with Intel's VA-API stack

2018-04-17T00:00:00Z

gst-build is far superior than gst-uninstalled scripts for developing GStreamer, mainly because its meson and ninja usage. Nonetheless, to integrate external dependencies it is not as easy as in gst-uninstalled.This guide aims to show how to integrate GStreamer-VAAPI dependencies, in this case with the Intel VA-API driver.

Dependencies #

For now we will need meson master, since it this patch is required. The pull request is already merged but it is unreleased yet.

Installation #

clone gst-build repository #

git clone git://anongit.freedesktop.org/gstreamer/gst-build
cd gst-build

apply custom patch for gst-build #

The patch will add the repositories for libva and intel-vaapi-driver.

wget https://people.igalia.com/vjaquez/gst-build-vaapi/0001-add-libva-and-intel-vaapi-driver-as-subprojects.patch
git am 0001-add-libva-and-intel-vaapi-driver-as-subprojects.patch

0001-add-libva-and-intel-vaapi-driver-as-subprojects.patch

configure #

Running this command, all dependency repositories will be cloned, symbolic links created, and the build directory configured.

meson bulid

apply custom patches for libva, intel-vaapi-driver and gstreamer-vaapi #

libva #

This patch is required since the headers files uninstalled paths doesn’t match with the ones in the “include” directives.

cd libva
wget https://people.igalia.com/vjaquez/gst-build-vaapi/0001-build-add-headers-for-uninstalled-setup.patch
git am 0001-build-add-headers-for-uninstalled-setup.patch
cd -

0001-build-add-headers-for-uninstalled-setup.patch

intel-vaapi-driver #

The patch handles libva dependency as a subproject.

cd intel-vaapi-driver
wget https://people.igalia.com/vjaquez/gst-build-vaapi/0001-meson-support-libva-as-subproject.patch
git am 0001-meson-support-libva-as-subproject.patch
cd -

0001-meson-support-libva-as-subproject.patch

gstreamer-vaapi #

Note to myself: this patch must be split and merged in upstream.

cd gstreamer-vaapi
wget https://people.igalia.com/vjaquez/gst-build-vaapi/0001-build-meson-libva-gst-uninstall-friendly.patch
git am 0001-build-meson-libva-gst-uninstall-friendly.patch
cd -

0001-build-meson-libva-gst-uninstall-friendly.patch updated: 2018/04/24

build #

ninja -C build

And wait a couple minutes.

run uninstalled environment for testing #

$ ninja -C build uninstalled
[gst-master] $ gst-inspect-1.0 vaapi
Plugin Details:
 Name                     vaapi
 Description              VA-API based elements
 Filename                 /opt/gst/gst-build/build/subprojects/gstreamer-vaapi/gst/vaapi/libgstvaapi.so
 Version                  1.15.0.1
 License                  LGPL
 Source module            gstreamer-vaapi
 Binary package           gstreamer-vaapi
 Origin URL               http://bugzilla.gnome.org/enter_bug.cgi?product=GStreamer

 vaapih264enc: VA-API H264 encoder
 vaapimpeg2enc: VA-API MPEG-2 encoder
 vaapisink: VA-API sink
 vaapidecodebin: VA-API Decode Bin
 vaapipostproc: VA-API video postprocessing
 vaapivc1dec: VA-API VC1 decoder
 vaapih264dec: VA-API H264 decoder
 vaapimpeg2dec: VA-API MPEG2 decoder
 vaapijpegdec: VA-API JPEG decoder

 9 features:
 +-- 9 elements

GStreamer VA-API Troubleshooting

2018-03-28T00:00:00Z

GStreamer VA-API is not a trivial piece of software. Even though, in my opinion it is a bit over-engineered, the complexity relies on its layered architecture: the user must troubleshoot in which layer is the failure.So, bear in mind this architecture:

So, bear in mind this architecture:

libva architecture

And the point of failure could be anywhere.

Drivers #

libva is a library designed to load another library called driver or back-end. This driver is responsible to talk with the kernel, windowing platform, memory handling library, or any other piece of software or hardware that actually will do the video processing.

There are many drivers in the wild. As it is an API aiming to stateless video processing, and the industry is moving towards that way to process video, it is expected more drivers would appear in the future.

Nonetheless, not all the drivers have the same level of maturity, and some of them are abandon-ware. For this reason we decided in GStreamer VA-API, some time ago, to add a white list of functional drivers, basically, those developed by Mesa3D and this one from Intel™. If you wish to disable that white-list, you can do it by setting an environment variable:

export GST_VAAPI_ALL_DRIVERS=1

Remember, if you set it, you are on your own, since we do not trust on the maturity of that driver yet.

Internal libva↔driver version #

Thus, there is an internal API between libva and the driver and it is versioned, meaning that the internal API version of the installed libva library must match with the internal API exposed by the driver. One of the causes that libva could not initialize a driver could be because the internal API version does not match.

Drivers path and driver name #

By default there is a path where libva looks for drivers to load. That path is defined at compilation time. Following Debian’s file-system hierarchy standard (FHS) it should be set by distributions in /usr/lib/x86_64-linux-gnu/dri/. But the user can control this path with an environment variable:

export LIBVA_DRIVERS_PATH=${HOME}/src/intel-vaapi-driver/src/.libs

The driver path, as a directory, might contain several drivers. libva will try to guess the correct one by querying the instantiated VA display (which could be either KMS/DRM, Wayland, Android or X11). If the user instantiates a VA display different of his running environment, the guess will be erroneous, the library loading will fail.

Although, there is a way for the user to set the driver’s name too. Again, by setting an environment variable:

export LIBVA_DRIVER_NAME=iHD

With this setting, libva will try to load iHD_drv_video.so (a new and experimental open source driver from Intel™, targeted for MediaSDK —do not use it yet with GStreamer VAAPI—).

vainfo #

vainfo is the diagnostic tool for VA-API. In a couple words, it will iterate on a list of VA displays, in try-and-error strategy, and try to initialize VA. In case of success, vainfo will report the driver signature, and it will query the driver for the available profiles and entry-points.

For example, my skylake board for development will report

$ vainfo
error: can't connect to X server!
libva info: VA-API version 1.1.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /home/vjaquez/gst/master/intel-vaapi-driver/src/.libs/i965_drv_video.so
libva info: Found init function __vaDriverInit_1_1
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.1 (libva 2.1.1.pre1)
vainfo: Driver version: Intel i965 driver for Intel(R) Skylake - 2.1.1.pre1 (2.1.0-41-g99c3748)
vainfo: Supported profile and entrypoints
     VAProfileMPEG2Simple            : VAEntrypointVLD
     VAProfileMPEG2Simple            : VAEntrypointEncSlice
     VAProfileMPEG2Main              : VAEntrypointVLD
     VAProfileMPEG2Main              : VAEntrypointEncSlice
     VAProfileH264ConstrainedBaseline: VAEntrypointVLD
     VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
     VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP
     VAProfileH264ConstrainedBaseline: VAEntrypointFEI
     VAProfileH264ConstrainedBaseline: VAEntrypointStats
     VAProfileH264Main               : VAEntrypointVLD
     VAProfileH264Main               : VAEntrypointEncSlice
     VAProfileH264Main               : VAEntrypointEncSliceLP
     VAProfileH264Main               : VAEntrypointFEI
     VAProfileH264Main               : VAEntrypointStats
     VAProfileH264High               : VAEntrypointVLD
     VAProfileH264High               : VAEntrypointEncSlice
     VAProfileH264High               : VAEntrypointEncSliceLP
     VAProfileH264High               : VAEntrypointFEI
     VAProfileH264High               : VAEntrypointStats
     VAProfileH264MultiviewHigh      : VAEntrypointVLD
     VAProfileH264MultiviewHigh      : VAEntrypointEncSlice
     VAProfileH264StereoHigh         : VAEntrypointVLD
     VAProfileH264StereoHigh         : VAEntrypointEncSlice
     VAProfileVC1Simple              : VAEntrypointVLD
     VAProfileVC1Main                : VAEntrypointVLD
     VAProfileVC1Advanced            : VAEntrypointVLD
     VAProfileNone                   : VAEntrypointVideoProc
     VAProfileJPEGBaseline           : VAEntrypointVLD
     VAProfileJPEGBaseline           : VAEntrypointEncPicture
     VAProfileVP8Version0_3          : VAEntrypointVLD
     VAProfileVP8Version0_3          : VAEntrypointEncSlice
     VAProfileHEVCMain               : VAEntrypointVLD
     VAProfileHEVCMain               : VAEntrypointEncSlice

And my AMD board with stable packages replies:

$ vainfo
libva info: VA-API version 0.40.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib64/dri/radeonsi_drv_video.so
libva info: Found init function __vaDriverInit_0_40
libva info: va_openDriver() returns 0
vainfo: VA-API version: 0.40 (libva )
vainfo: Driver version: mesa gallium vaapi
vainfo: Supported profile and entrypoints
     VAProfileMPEG2Simple            : VAEntrypointVLD
     VAProfileMPEG2Main              : VAEntrypointVLD
     VAProfileVC1Simple              : VAEntrypointVLD
     VAProfileVC1Main                : VAEntrypointVLD
     VAProfileVC1Advanced            : VAEntrypointVLD
     VAProfileH264ConstrainedBaseline: VAEntrypointVLD
     VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
     VAProfileH264Main               : VAEntrypointVLD
     VAProfileH264Main               : VAEntrypointEncSlice
     VAProfileH264High               : VAEntrypointVLD
     VAProfileH264High               : VAEntrypointEncSlice
     VAProfileNone                   : VAEntrypointVideoProc

Does this mean that VA-API processes video? No. It means that there is an usable VA display which could open a driver correctly and libva can extract symbols from it.

I would like to mention another tool, not official, but I like it a lot, since it extracts almost of the VA information available in the driver: vadumpcaps.c, written by Mark Thompson.

GStreamer VA-API registration #

When GStreamer is launched, normally it will register all the available plugins and plugin features (elements, device providers, etc.). All that data is cache and keep until the cache file is deleted or the cache invalidated by some event.

At registration time, GStreamer VA-API will instantiate a DRM-based VA display, which works with no need of a real display (in other words, headless), and will query the driver for the profiles and entry-points tuples, in order to register only the available elements (encoders, decoders. sink, post-processor). If the DRM VA display fails, a list of VA displays will be tried.

In the case that libva could not load any driver, or the driver is not in the white-list, GStreamer VA-API will not register any element. Otherwise gst-inspect-1.0 will show the registered elements:

$ gst-inspect-1.0 vaapi
Plugin Details:
 Name                     vaapi
 Description              VA-API based elements
 Filename                 /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstvaapi.so
 Version                  1.12.4
 License                  LGPL
 Source module            gstreamer-vaapi
 Source release date      2017-12-07
 Binary package           gstreamer-vaapi
 Origin URL               http://bugzilla.gnome.org/enter_bug.cgi?product=GStreamer

 vaapijpegdec: VA-API JPEG decoder
 vaapimpeg2dec: VA-API MPEG2 decoder
 vaapih264dec: VA-API H264 decoder
 vaapivc1dec: VA-API VC1 decoder
 vaapivp8dec: VA-API VP8 decoder
 vaapih265dec: VA-API H265 decoder
 vaapipostproc: VA-API video postprocessing
 vaapidecodebin: VA-API Decode Bin
 vaapisink: VA-API sink
 vaapimpeg2enc: VA-API MPEG-2 encoder
 vaapih265enc: VA-API H265 encoder
 vaapijpegenc: VA-API JPEG encoder
 vaapih264enc: VA-API H264 encoder

 13 features:
 +-- 13 elements

Beside the normal behavior, GStreamer VA-API will also invalidate GStreamer’s cache at every boot, or when any of the mentioned environment variables change.

Conclusion #

A simple task list to review when GStreamer VA-API is not working at all is this:

Check your LIBVA_* environment variables
Verify that vainfo returns sensible information
Invalidate GStreamer’s cache (or just delete the file)
Check the output of gst-inspect-1.0 vaapi

And, if you decide to file a bug in bugzilla, please do not forget to attach the output of vainfo and the logs if the developer asks for them.

GStreamer VA-API 1.14: what's new?

2018-03-27T00:00:00Z

As you may already know, there is a new release of GStreamer, 1.14. In this blog post we will talk about the new features and improvements of GStreamer VA-API module, though you have a more comprehensive list of changes in the release notes.Most of the topics explained along this blog post are already mentioned in the release notes, but a bit more detailed.

DMABuf usage #

We have improved DMABuf usage, mostly at downstream.

In the case of upstream, we just got rid a nasty hack which detected when to instantiate and use a buffer pool in sink pad with a dma-buf based allocator. This functionality has been already broken for a while, and that code was the wrong way to enabled it. The sharing of a dma-buf based buffer pool to upstream is going to be re-enabled after bug 792034 is merged.

For downstream, we have added the handling of memory:DMABuf caps feature. The purpose of this caps feature is to negotiate a media when the buffers are not map-able onto user space, because of digital rights or platform restrictions.

For example, currently intel-vaapi-driver doesn’t allow the mapping of its produced dma-buf descriptors. But, as we cannot know if a back-end produces or not map-able dma-buf descriptors, gstreamer-vaapi, when the allocator is instantiated, creates a dummy buffer and tries to map it, if it fails, memory:DMABuf caps feature is negotiated, otherwise, normal video caps are used.

VA-API usage #

First of all, GStreamer VA-API has support now for libva-2.0, this means VA-API 1.10. We had to guard some deprecated symbols and the new ones. Nowadays most of distributions have upgraded to libva-2.0.

We have improved the initialization of the VA display internal structure (GstVaapiDisplay). Previously, if a X based display was instantiated, immediately it tried to grab the screen resolution. Obviously, this broke the usage of headless systems. We just delay the screen resolution check to when the VA display truly requires that information.

New API were added into VA, particularly for log handling. Now it is possible to redirect the log messages into a callback. Thus, we use it to redirect VA-API message into the GStreamer log mechanisms, uncluttering the console’s output.

Also, we have blacklisted, in autoconf and meson, libva version 0.99.0, because that version is used internally by the closed-source version of Intel MediaSDK, which is incompatible with official libva. By the way, there is a new open-source version of MediaSDK, but we will talk about it in a future blog post.

Normally, the object GstVaapiDisplay is shared among the pipeline through the GstContext mechanism. But this class is defined internally and it is not exposed to users since release 1.6. This posed a problem when an application wanted to create its own VA Display and share it with an embedded pipeline. The solution is a new context application message: gst.vaapi.app.Display, defined as a GstStructure with two fields: va-display with the application’s vaDisplay, and x11-display with the application’s X11 native display. In the future, a Wayland’s native handler will be processed too. Please note that this context message is only processed by vaapisink.

vaapisink

One precondition for this solution was the removal of the VA display cache mechanism, a lingered request from users, which, of course, we did.

Interoperability with appsink and similar #

A hardware accelerated driver, as the Intel one, may have custom offsets and strides for specific memory regions. We use the GstVideoMeta to set this custom values. The problem comes when downstream does not handle this meta, for example, appsink. Then, the user expect the “normal” values for those variable, but in the case of GStreamer VA-API with a hardware based driver, when the user displays the frame, it is shown corrupted.

In order to fix this, we have to make a memory copy, from our custom VA-API images to an allocated system memory. Of course there is a big CPU penalty, but it is better than delivering wrong video frames. If the user wants a better performance, then they should seek for a different approach.

Resurrection of GstGLTextureUploadMeta for EGL renders #

I know, GstGLTextureUploadMeta must die, right? I am convinced of it. But, Clutter video sink uses it, an it has a vast number of users, so we still have to support it.

Last release we had remove the support for EGL/Wayland in the last minute because we found a terrible bug just before the release. GLX support has always been there.

With Daniel van Vugt efforts, we resurrected the support for that meta in EGL. Though I expect the replacement of Clutter sink with glimagesink someday, soon.

vaapisink demoted in Wayland #

vaapisink was demoted to marginal rank on Wayland because COGL cannot display YUV surfaces.

This means, by default, vaapisink won’t be auto-plugged when playing in Wayland.

The reason is because Mutter (aka GNOME) cannot display the frames processed by vaapisink in Wayland. Nonetheless, please note that in Weston, it works just fine.

Decoders #

We have improved a little bit upstream renegotiation: if the new stream is compatible with the previous one, there is no need to reset the internal parser, with the exception of changes in codec-data.

low-latency property in H.264 #

A new property has added only to H.264 decoder: low-latency. Its purpose is for live streams that do not conform the H.264 specification (sadly there are many in the wild) and they need to twitch the spec implementation. This property force to push the frames in the decoded picture buffer as soon as possible.

base-only property in H.264 #

This is the result of the Google Summer of Code 2017, by Orestis Floros. When this property is enabled, all the MVC (Multiview Video Coding) or SVC (Scalable Video Coding) frames are dropped. This is useful if you want to reduce the processing time or if your VA-API driver does not support those kind of streams.

Encoders #

In this release we have put a lot of effort in encoders.

Processing Regions of Interest #

It is possible, for certain back-ends and profiles (for example, H.264 and H.265 encoders with Intel driver), to specify a set of regions of interest per frame, with a delta-qp per region. This mean that we would ask more quality in those regions.

In order to process regions of interest, upstream must add to the video frame, a list of GstVideoRegionOfInterestMeta. This list then is traversed by the encoder and it requests them if the VA-API profile, in the driver, supports it.

The common use-case for this feature is if you want to higher definition in regions with faces or text messages in the picture, for example.

New encoding properties #

quality-level: For all the available encoders. This is number between 1 to 8, where a lower number means higher quality (and slower processing).
aud: This is for H.264 encoder only and it is available for certain drivers and platforms. When it is enabled, an AU delimiter is inserted for each encoded frame. This is useful for network streaming, and more particularly for Apple clients.
mbbrc: For H.264 only. Controls (auto/on/off) the macro-block bit-rate.
temporal-levels: For H.264 only. It specifies the number of temporal levels to include a the hierarchical frame prediction.
prediction-type: For H.264 only. It selects the reference picture selection mode.

The frames are encoded as different layers. A frame in a particular layer will use pictures in lower or same layer as references. This means decoder can drop frames in upper layer but still decode lower layer frames.
- hierarchical-p: P frames, except in top layer, are reference frames. Base layer frames are I or B.
- hierarchical-b: B frames , except in top most layer, are reference frames. All the base layer frames are I or P.
refs: Added for H.265 (it was already supported for H.264). It specifies the number of reference pictures.
qp-ip and qp-ib: For H.264 and H.265 encoders. They handle the QP (quality parameters) difference between the I and P frames, the the I and B frames respectively.

Set media profile via downstream caps #

H.264 and H.265 encoders now can configure the desired media profile through the downstream caps.

Contributors #

Many thanks to all the contributors and bug reporters.

         1  Daniel van Vugt
        46  Hyunjun Ko
         1  Jan Schmidt
         3  Julien Isorce
         1  Matt Staples
         2  Matteo Valdina
         2  Matthew Waters
         1  Michael Tretter
         4  Nicolas Dufresne
         9  Orestis Floros
         1  Philippe Normand
         4  Sebastian Dröge
        24  Sreerenj Balachandran
         1  Thibault Saunier
        13  Tim-Philipp Müller
         1  Tomas Rataj
         2  U. Artie Eoff
         1  VaL Doroshchuk
       172  Víctor Manuel Jáquez Leal
         3  XuGuangxin
         2  Yi A Wang

Enabling HuC for SKL/KBL in Debian/testing

2017-12-07T00:00:00Z

Recently, our friend Florent complained that it was impossible to set a constant bitrate when encoding H.264 using low-power profile with gstreamer-vaapi .Low-power (LP) profiles are VA-API entry points, available in Intel SkyLake-based processor and successors, which provide video encoding with low power consumption.

Later on, Ullysses and Sree, pointed out that CBR in LP is ony possible if HuC is enabled in the kernel.

HuC is a firmware, loaded by i915 kernel module, designed to offload some of the media functions from the CPU to GPU. One of these functions is bitrate control when encoding. HuC saves unnecessary CPU-GPU synchronization.

In order to load HuC, it is required first to load GuC, another Intel’s firmware designed to perform graphics workload scheduling on the various graphics parallel engines.

How we can install and configure these firmwares to enable CBR in low-power profile, among other things, in Debian/testing?

Check i915 parameters #

First we shall confirm that our kernel and our i915 kernel module is capable to handle this functionality:

$ sudo modinfo i915 | egrep -i "guc|huc|dmc"
firmware:       i915/bxt_dmc_ver1_07.bin
firmware:       i915/skl_dmc_ver1_26.bin
firmware:       i915/kbl_dmc_ver1_01.bin
firmware:       i915/kbl_guc_ver9_14.bin
firmware:       i915/bxt_guc_ver8_7.bin
firmware:       i915/skl_guc_ver6_1.bin
firmware:       i915/kbl_huc_ver02_00_1810.bin
firmware:       i915/bxt_huc_ver01_07_1398.bin
firmware:       i915/skl_huc_ver01_07_1398.bin
parm:           enable_guc_loading:Enable GuC firmware loading (-1=auto, 0=never [default], 1=if available, 2=required) (int)
parm:           enable_guc_submission:Enable GuC submission (-1=auto, 0=never [default], 1=if available, 2=required) (int)
parm:           guc_log_level:GuC firmware logging level (-1:disabled (default), 0-3:enabled) (int)
parm:           guc_firmware_path:GuC firmware path to use instead of the default one (charp)
parm:           huc_firmware_path:HuC firmware path to use instead of the default one (charp)

Install firmware #

sudo apt install firmware-misc-nonfree

UPDATE: In order to install this Debian package, you should have enabled the non-free apt repository in your sources list.

Verify the firmware are installed:

$ ls -1 /lib/firmware/i915/
bxt_dmc_ver1_07.bin
bxt_dmc_ver1.bin
bxt_guc_ver8_7.bin
bxt_huc_ver01_07_1398.bin
kbl_dmc_ver1_01.bin
kbl_dmc_ver1.bin
kbl_guc_ver9_14.bin
kbl_huc_ver02_00_1810.bin
skl_dmc_ver1_23.bin
skl_dmc_ver1_26.bin
skl_dmc_ver1.bin
skl_guc_ver1.bin
skl_guc_ver4.bin
skl_guc_ver6_1.bin
skl_guc_ver6.bin
skl_huc_ver01_07_1398.bin

Update modprobe configuration #

Edit or create the configuration file /etc/modprobe.d/i915.con

$ sudo vim /etc/modprobe.d/i915.conf
....
$ cat /etc/modprobe.d/i915.conf
options i915 enable_guc_loading=1 enable_guc_submission=1

Reboot #

sudo systemctl reboot

Verification #

Now it is possible to verify that the i915 module kernel loaded the firmware correctly by looking at the kernel logs:

$ journalctl -b -o short-monotonic -k | egrep -i "i915|dmr|dmc|guc|huc"
[   10.303849] miau kernel: Setting dangerous option enable_guc_loading - tainting kernel
[   10.303852] miau kernel: Setting dangerous option enable_guc_submission - tainting kernel
[   10.336318] miau kernel: i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem
[   10.338664] miau kernel: i915 0000:00:02.0: firmware: direct-loading firmware i915/kbl_dmc_ver1_01.bin
[   10.339635] miau kernel: [drm] Finished loading DMC firmware i915/kbl_dmc_ver1_01.bin (v1.1)
[   10.361811] miau kernel: i915 0000:00:02.0: firmware: direct-loading firmware i915/kbl_huc_ver02_00_1810.bin
[   10.362422] miau kernel: i915 0000:00:02.0: firmware: direct-loading firmware i915/kbl_guc_ver9_14.bin
[   10.393117] miau kernel: [drm] GuC submission enabled (firmware i915/kbl_guc_ver9_14.bin [version 9.14])
[   10.410008] miau kernel: [drm] Initialized i915 1.6.0 20170619 for 0000:00:02.0 on minor 0
[   10.559614] miau kernel: snd_hda_intel 0000:00:1f.3: bound 0000:00:02.0 (ops i915_audio_component_bind_ops [i915])
[   11.937413] miau kernel: i915 0000:00:02.0: fb0: inteldrmfb frame buffer device

That means that HuC and GuC firmwares were loaded successfully.

Now we can check the status of the modules using sysfs

$ sudo cat /sys/kernel/debug/dri/0/i915_guc_load_status
GuC firmware status:
       path: i915/kbl_guc_ver9_14.bin
       fetch: SUCCESS
       load: SUCCESS
       version wanted: 9.14
       version found: 9.14
       header: offset is 0; size = 128
       uCode: offset is 128; size = 142272
       RSA: offset is 142400; size = 256

GuC status 0x800330ed:
       Bootrom status = 0x76
       uKernel status = 0x30
       MIA Core status = 0x3

Scratch registers:
        0:     0xf0000000
        1:     0x0
        2:     0x0
        3:     0x5f5e100
        4:     0x600
        5:     0xd5fd3
        6:     0x0
        7:     0x8
        8:     0x3
        9:     0x74240
       10:     0x0
       11:     0x0
       12:     0x0
       13:     0x0
       14:     0x0
       15:     0x0
$ sudo cat /sys/kernel/debug/dri/0/i915_huc_load_status
HuC firmware status:
       path: i915/kbl_huc_ver02_00_1810.bin
       fetch: SUCCESS
       load: SUCCESS
       version wanted: 2.0
       version found: 2.0
       header: offset is 0; size = 128
       uCode: offset is 128; size = 218304
       RSA: offset is 218432; size = 256

HuC status 0x00006080:

Test GStreamer #

$ gst-launch-1.0 videotestsrc num-buffers=1000 ! video/x-raw, format=NV12, width=1920, height=1080, framerate=\(fraction\)30/1 ! vaapih264enc bitrate=8000 keyframe-period=30 tune=low-power rate-control=cbr ! mp4mux ! filesink location=test.mp4
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Got context from element 'vaapiencodeh264-0': gst.vaapi.Display=context, gst.vaapi.Display=(GstVaapiDisplay)"\(GstVaapiDisplayGLX\)\ vaapidisplayglx0";
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Got EOS from element "pipeline0".
Execution ended after 0:00:11.620036001
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...
$ gst-discoverer-1.0 test.mp4
Analyzing file:///home/vjaquez/gst/master/intel-vaapi-driver/test.mp4
Done discovering file:///home/vjaquez/test.mp4

Topology:
 container: Quicktime
   video: H.264 (High Profile)

Properties:
 Duration: 0:00:33.333333333
 Seekable: yes
 Live: no
 Tags:
     video codec: H.264 / AVC
     bitrate: 8084005
     encoder: VA-API H264 encoder
     datetime: 2017-12-07T14:29:23Z
     container format: ISO MP4/M4A

Mission accomplished!

References #

Intel MediaSDK on Debian (testing)

2017-11-24T00:00:00Z

Everybody knows it: install Intel MediaSDK in GNU/Linux is a PITA. With CentOS or Yocto is less cumbersome, if you trust blindly on scripts ran as root.I don’t like CentOS, I feel it like if I were living in the past. I like Debian (testing, of course) and I also wanted to understand a little more about MediaSDK. And this is what I did to have Intel MediaSDK working in Debian/testing.

First, I did a pristine installation of Debian testing with a netinst image in my NUC 6i5SYK, with a normal desktop user setup (Gnome3).

The madness comes later.

Intel’s identifies two types of MediaSDK installation: Gold and Generic. Gold is for CentOS, and Generic for the rest of distributions. Obviously, Generic means you’re on your own. For the purpose of this exercise I used as reference Generic Linux* Intel® Media Server Studio Installation.

Let’s begin by grabbing the Intel® Media Server Studio – Community Edition. You will need to register yourself and accept the user agreement, because this is proprietary software.

At the end, you should have a tarball named MediaServerStudioEssentials2017R3.tar.gz

Extract the files for Generic installation #

cd ~
tar xvf MediaServerStudioEssentials2017R3.tar.gz
cd MediaServerStudioEssentials2017R3
tar xvf SDK2017Production16.5.2.tar.gz
cd SDK2017Production16.5.2/Generic
mkdir tmp
tar -xvC tmp -f intel-linux-media_generic_16.5.2-64009_64bit.tar.gz

Kernel #

Bad news: in order to get MediaSDK working you need to patch the mainlined kernel.

Worse news: the available patches are only for the version 4.4 the kernel.

Still, systemd works on 4.4, as far as I know, so it would not be a big problem.

Grab building dependencies #

sudo apt install build-essential devscripts libncurses5-dev
sudo apt build-dep linux

Grab kernel source #

I like to use the sources from the git repository, since it would be possible to do some rebasing and blaming in the future.

cd ~
git clone https://github.com/torvalds/linux.git
git pull -v --tags
git checkout -b 4.4 v4.4

Extract MediaSDK patches #

cd ~/MediaServerStudioEssentials2017R3/SDK2017Production16.5.2/Generic/tmp/opt/intel/mediasdk/opensource/patches/kmd/4.4
tar xvf intel-kernel-patches.tar.bz2
cd intel-kernel-patches
PATCHDIR=$(pwd)

Patch the kernel #

cd ~/linux
git am $PATCHDIR/*.patch

The patches should apply with some warnings but no fatal errors (don’t worry, be happy).

Still, there’s a problem with this old kernel: our recent compiler doesn’t build it as it is. Another patch is required:

wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.8-rc2/0002-UBUNTU-SAUCE-no-up-disable-pie-when-gcc-has-it-enabl.patch
git am 0002-UBUNTU-SAUCE-no-up-disable-pie-when-gcc-has-it-enabl.patch

TODO: Shall I need to modify the EXTRAVERSION string in kernel’s Makefile?

Build and install the kernel #

Notice that we are using our current kernel configuration. That is error prone. I guess that is why I had to select NVM manually.

cp /boot/config-4.12.0-1-amd64 ./.config
make olddefconfig
make nconfig # -- select NVM
scripts/config --disable DEBUG_INFO
make deb-pkg
sudo dpkg -i linux-image-4.4.0+_4.4.0+-2_amd64.deb linux-headers-4.4.0+_4.4.0+-2_amd64.deb linux-firmware-image-4.4.0+_4.4.0+-2_amd64.deb

Configure GRUB2 to boot Linux 4.4. by default #

This part was absolutely tricky for me. It took me a long time to figure out how to specify the kernel ID in the grubenv.

sudo vi /etc/default/grub

And change the line GRUB_DEFAULT=saved. By default it is set to 0. And update GRUB.

sudo update-grub

Now look for the ID of the installed kernel image in /etc/grub/grub.cfg and use it:

sudo grub-set-default "gnulinux-4.4.0+-advanced-2c246bc6-65bb-48ea-9517-4081b016facc>gnulinux-4.4.0+-advanced-2c246bc6-65bb-48ea-9517-4081b016facc"

Please note it is twice and separated by a >. Don’t ask me why.

Copy MediaSDK firmware (and libraries too) #

I like to use rsync rather normal cp because there are the options like --dry-run and --itemize-changes to verify what I am doing.

cd ~/MediaServerStudioEssentials2017R3/SDK2017Production16.5.2/Generic/tmp
sudo rsync -av --itemize-changes ./lib /
sudo rsync -av --itemize-changes ./opt/intel/common /opt/intel
sudo rsync -av --itemize-changes ./opt/intel/mediasdk/{include,lib64,plugins} /opt/intel/mediasdk

All these directories contain blobs that do the MediaSDK magic. They are dlopened by hard coded paths by mfx_dispatch, which will be explain later.

In /lib lives the firmware (kernel blob).

In /opt/intel/common… I have no idea what are those shared objects.

In /opt/intel/mediasdk/include live header files for programming an compilation.

In /opt/intel/mediasdk/lib64 live the driver for the modified libva (iHD) and other libraries.

In /opt/intel/mediasdk/plugins live, well, plugins…

In conclusion, all these bytes are darkness and mystery.

Reboot #

sudo systemctl reboot

The system should boot, automatically, in GNU/Linux 4.4.

Please, log with Xorg, not in Wayland, since it is not supported, as far as I know.

GStreamer #

For compiling GStreamer I will use gst-uninstalled. Someone may say that I should use gst-build because is newer and faster, but I feel more comfortable doing the following kind of hacks with the old&good autotools.

Basically this is a reproduction of Quick-start guide to gst-uninstalled for GStreamer 1.x.

sudo apt build-dep gst-plugins-{base,good,bad}1.0
wget https://cgit.freedesktop.org/gstreamer/gstreamer/plain/scripts/create-uninstalled-setup.sh -q -O - | sh

I will modify the gst-uninstalled script, and keep it outside of the repository. For that I will use the systemd file-hierarchy spec for user’s executables.

cd ~/gst
mkdir -p ~/.local/bin
mv master/gstreamer/scripts/gst-uninstalled ~/.local/bin
ln -sf ~/.local/bin/gst-uninstalled ./gst-master

Do not forget to edit your ~/.profile to add ~/.local/bin in the environment variable PATH.

Patch ~/.local/bin/gst-uninstalled #

The modifications are to handle the three dependencies libraries that are required by MediaSDK: libdrm, libva and mfx_dispatch.

diff --git a/scripts/gst-uninstalled b/scripts/gst-uninstalled
index 81f83b6c4..d79f19abd 100755
--- a/scripts/gst-uninstalled
+++ b/scripts/gst-uninstalled
@@ -122,7 +122,7 @@ GI_TYPELIB_PATH=$GST/gstreamer/gst:$GI_TYPELIB_PATH
 export LD_LIBRARY_PATH
 export DYLD_LIBRARY_PATH
 export GI_TYPELIB_PATH
-
+
 export PKG_CONFIG_PATH="\
 $GST_PREFIX/lib/pkgconfig\
 :$GST/gstreamer/pkgconfig\
@@ -140,6 +140,9 @@ $GST_PREFIX/lib/pkgconfig\
 :$GST/orc\
 :$GST/farsight2\
 :$GST/libnice/nice\
+:$GST/drm\
+:$GST/libva/pkgconfig\
+:$GST/mfx_dispatch\
 ${PKG_CONFIG_PATH:+:$PKG_CONFIG_PATH}"

 export GST_PLUGIN_PATH="\
@@ -227,6 +230,16 @@ export GST_VALIDATE_APPS_DIR=$GST_VALIDATE_APPS_DIR:$GST/gst-editing-services/te
 export GST_VALIDATE_PLUGIN_PATH=$GST_VALIDATE_PLUGIN_PATH:$GST/gst-devtools/validate/plugins/
 export GIO_EXTRA_MODULES=$GST/prefix/lib/gio/modules:$GIO_EXTRA_MODULES

+# MediaSDK
+export LIBVA_DRIVERS_PATH=/opt/intel/mediasdk/lib64
+export LIBVA_DRIVER_NAME=iHD
+export LD_LIBRARY_PATH="\
+/opt/intel/common/mdf/lib64\
+:$GST/drm/.libs\
+:$GST/drm/intel/.libs\
+:$GST/libva/va/.libs\
+:$LD_LIBRARY_PATH"
+

Now, initialize the gst-uninstalled environment:

cd ~/gst
./gst-master

libdrm #

Grab libdrm from its repository and switch to the branch with the supported version by MediaSDK.

cd ~/gst/master
git clone git://anongit.freedesktop.org/mesa/drm
cd drm
git checkout -b intel libdrm-2.4.67

Extract the distributed tarball in the cloned repository.

tar -xv --strip-components=1 -C . -f ~/MediaServerStudioEssentials2017R3/SDK2017Production16.5.2/Generic/tmp/opt/intel/mediasdk/opensource/libdrm/2.4.67-64009/libdrm-2.4.67.tar.bz2

Then we could check the big delta between upstream and the changes done by Intel for MediaSDK.

Let’s put it in a commit for later rebases.

git add -u
git add .
git commit -m "mediasdk changes"

Get build dependencies and compile.

sudo apt build-dep libdrm
./configure
make -j8

Since the pkgconfig files (*.pc) of libdrm are generated to work installed, it is needed to modify them in order to work uninstalled.

prefix=${HOME}/gst/master/drm
sed -i -e "s#^libdir=.*#libdir=${prefix}/.libs#" ${prefix}/*.pc
sed -i -e "s#^includedir=.*#includedir=${prefix}#" ${prefix}/*.pc

In order to C preprocessor could find the uninstalled libdrm header files we need to make them available in the expected path according to the pkgconfig file and right now they are not there. To fix that it is possible to create proper symbolic links.

cd ~/gst/master/drm
ln -s include/drm/ libdrm

libva #

This modified a version of libva. These modifications messed a bit with the opensource version of libva, because Intel decided not to prefix the library, or some other strategy. In gstreamer-vaapi we had to blacklist VA-API version 0.99, because it is the version number, arbitrary set, of this modified version of libva for MediaSDK.

Again, grab the original libva from repo and change the branch aiming to the divert point. It was difficult to find the divert commit id since even the libva version number was changed. Doing some archaeology I guessed the branch point was in version 1.0.15, but I’m not sure.

cd ~/gst/master
git clone https://github.com/01org/libva.git
cd libva
git checkout -b intel libva-1.0.15
tar -xv --strip-components=1 -C . -f ~/MediaServerStudioEssentials2017R3/SDK2017Production16.5.2/Generic/tmp/opt/intel/mediasdk/opensource/libva/1.67.0.pre1-64009/libva-1.67.0.pre1.tar.bz2
git add -u
git add .
git commit -m "mediasdk"

Before compile, verify that Makefile is going to link against the uninstalled libdrm. You can do that by grepping for LIBDRM in Makefile.

Get compilation dependencies and build.

sudo apt build-dep libva
./configure
make -j8

Modify the pkgconfig files for uninstalled

prefix=${HOME}/gst/master/libva
sed -i -e "s#^libdir=.*#libdir=${prefix}/va/.libs#" ${prefix}/pkgconfig/*.pc
sed -i -e "s#^includedir=.*#includedir=${prefix}#" ${prefix}/pkgconfig/*.pc

Fix header path with symbolic links

cd ~/gst/master/libva/va
ln -sf drm/va_drm.h

mfx_dispatch #

This static library which must be linked with MediaSDK applications. In our case, to the GStreamer plugin.

According to its documentation (included in the tarball):

the dispatcher is a layer that lies between application and the SDK implementations. Upon initialization, the dispatcher locates the appropriate platform-specific SDK implementation. If there is none, it will select the software SDK implementation. The dispatcher will redirect subsequent function calls to the same functions in the selected SDK implementation.

In the tarball there is the source of the mfx_dispatcher, but it only compiles with cmake. I have not worked with cmake on uninstalled setups, but we are lucky, there is a repository with autotools support:

cd ~/gst/master
git clone https://github.com/lu-zero/mfx_dispatch.git

And compile. After running ./configure it is better to confirm, grepping the generated Makefie, that the uninstalled versions of libdrm and libva are going to be used.

autoreconf  --install
./configure
make -j8

Finally, just as the other libraries, it is required to fix the pkgconfig files:d

prefix=${HOME}/gst/master/mfx_dispatch
sed -i -e "s#^libdir=.*#libdir=${prefix}/.libs#" ${prefix}/*.pc
sed -i -e "s#^includedir=.*#includedir=${prefix}#" ${prefix}/*.pc

Test it #

At last we are in a position where it is possible to test if everything works as expected. For it we are going to run the pre-compiled version of vainfo bundled in the tarball.

We will copy it to our uninstalled setup, thus we would running without specifying the path.

$ sync -av /home/vjaquez/MediaServerStudioEssentials2017R3/SDK2017Production16.5.2/Generic/tmp/usr/bin/vainfo ./prefix/bin/
$ vainfo
libva info: VA-API version 0.99.0
libva info: va_getDriverName() returns 0
libva info: User requested driver 'iHD'
libva info: Trying to open /opt/intel/mediasdk/lib64/iHD_drv_video.so
libva info: Found init function __vaDriverInit_0_32
libva info: va_openDriver() returns 0
vainfo: VA-API version: 0.99 (libva 1.67.0.pre1)
vainfo: Driver version: 16.5.2.64009-ubit
vainfo: Supported profile and entrypoints
     VAProfileH264ConstrainedBaseline: VAEntrypointVLD
     VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
     VAProfileH264ConstrainedBaseline: <unknown entrypoint>
     VAProfileH264ConstrainedBaseline: <unknown entrypoint>
     VAProfileH264Main               : VAEntrypointVLD
     VAProfileH264Main               : VAEntrypointEncSlice
     VAProfileH264Main               : <unknown entrypoint>
     VAProfileH264Main               : <unknown entrypoint>
     VAProfileH264High               : VAEntrypointVLD
     VAProfileH264High               : VAEntrypointEncSlice
     VAProfileH264High               : <unknown entrypoint>
     VAProfileH264High               : <unknown entrypoint>
     VAProfileMPEG2Simple            : VAEntrypointEncSlice
     VAProfileMPEG2Simple            : VAEntrypointVLD
     VAProfileMPEG2Main              : VAEntrypointEncSlice
     VAProfileMPEG2Main              : VAEntrypointVLD
     VAProfileVC1Advanced            : VAEntrypointVLD
     VAProfileVC1Main                : VAEntrypointVLD
     VAProfileVC1Simple              : VAEntrypointVLD
     VAProfileJPEGBaseline           : VAEntrypointVLD
     VAProfileJPEGBaseline           : VAEntrypointEncPicture
     VAProfileVP8Version0_3          : VAEntrypointEncSlice
     VAProfileVP8Version0_3          : VAEntrypointVLD
     VAProfileVP8Version0_3          : <unknown entrypoint>
     VAProfileHEVCMain               : VAEntrypointVLD
     VAProfileHEVCMain               : VAEntrypointEncSlice
     VAProfileVP9Profile0            : <unknown entrypoint>
     <unknown profile>               : VAEntrypointVideoProc
     VAProfileNone                   : VAEntrypointVideoProc
     VAProfileNone                   : <unknown entrypoint>

It works!

Compile GStreamer #

I normally make a copy of ~/gst/master/gstreamer/script/git-update.sh in ~/.local/bin in order to modify it, like adding support for ccache, disabling gtkdoc and gobject-introspection, increase the parallel tasks, etc. But that is out of the scope of this document.

cd ~/gst/master/
./gstreamer/scripts/git-update.sh

Everything should be built without issues and, at the end, we could test if the gst-msdk elements are available:

$ gst-inspect-1.0 msdk
Plugin Details:
 Name                     msdk
 Description              Intel Media SDK encoders
 Filename                 /home/vjaquez/gst/master/gst-plugins-bad/sys/msdk/.libs/libgstmsdk.so
 Version                  1.13.0.1
 License                  BSD
 Source module            gst-plugins-bad
 Source release date      2017-11-23 16:39 (UTC)
 Binary package           GStreamer Bad Plug-ins git
 Origin URL               Unknown package origin

 msdkh264dec: Intel MSDK H264 decoder
 msdkh264enc: Intel MSDK H264 encoder
 msdkh265dec: Intel MSDK H265 decoder
 msdkh265enc: Intel MSDK H265 encoder
 msdkmjpegdec: Intel MSDK MJPEG decoder
 msdkmjpegenc: Intel MSDK MJPEG encoder
 msdkmpeg2enc: Intel MSDK MPEG2 encoder
 msdkvp8dec: Intel MSDK VP8 decoder
 msdkvp8enc: Intel MSDK VP8 encoder

 9 features:
 +-- 9 elements

Great!

Now, let’s run a simple pipeline. Please note that gst-msdk elements have rank zero, then they will not be autoplugged, it is necessary to craft the pipeline manually:

$ gst-launch-1.0 filesrc location= ~/test.264 ! h264parse ! msdkh264dec ! videoconvert ! xvimagesink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
libva info: VA-API version 0.99.0
libva info: va_getDriverName() returns 0
libva info: User requested driver 'iHD'
libva info: Trying to open /opt/intel/mediasdk/lib64/iHD_drv_video.so
libva info: Found init function __vaDriverInit_0_32
libva info: va_openDriver() returns 0
Redistribute latency...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Got EOS from element "pipeline0".
Execution ended after 0:00:02.502411331
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

\o/

GStreamer Conference 2017

2017-10-30T00:00:00Z

This year, the GStreamer Conference happened in Prague, along with the traditional autumn Hackfest.Prague is a beautiful city, though this year I couldn’t visit it as much as I wanted, since the Embedded Linux Conference Europe and the Open Source Summit also took place there, and Igalia, being a Linux Foundation sponsor, had a booth in the venue, where I talked about our work with WebKit, Snabb, and obviously, GStreamer.

But, let’s back to the GStreamer Hackfest and Conference.

One of the features that I like the most of the GStreamer project is its community, the people involved in it, by writing code, sharing their work with many others. They might appear a bit tough at beginning (or at least that looked to me) but in real they are all kind and talented persons. And I’m proud of consider myself part of this community. Nonetheless it has a diversity problem, as many other Open Source communities.

During the Hackfest, Hyunjun and I, met with Sree and talked about the plans for GStreamer-VAAPI, the new features in VA-API and libva and how we could map them to the GStreamer’s design. Also we talked about the future developments in the msdk elements, merged one year ago in gst-plugins-bad. Also, I talked a bit with Nicolas Dufresne regarding kmssink and DMABuf.

In the Conference, which happened in the same venue as the hackfest, I talked wit the authors of gstreamer-media-SDK. They are really energetic.

I delivered my usual talk about GStreamer-VAAPI. You can find the slides, as a web presentation, here. Also, as every year, our friends of Ubicast, recorded the talks, and made them available for streaming almost instantaneously:

https://www.youtube.com/watch?v=tg_sErLpaG0

My colleague Enrique talked in the Conference about the Media Source Extensions (MSE) on WebKit, and Hyunjun shared his experience with VA-API on Rust.

Also, in the conference venue, we showed a couple demos. One of them was a MinnowBoard running WPE, rendering videos from YouTube using gstreamer-vaapi to decode video.

GstSpringHackfest2017: a quick report

2017-05-30T00:00:00Z

Two weeks ago was the GStreamer Spring Hackfest 2017 and I am very happy about how it went. I have the feeling that most of the attendees had a good time, and made some progress in their projects. I want to thank all the people that participated, in some way or another.Along that weekend when the hackfest happened, besides my duties as organizer (with a lot of help from my colleagues at Igalia), I managed to hack a bit on GstPlayer, proposing the missing API for setting the subtitles font description (782858). Also I helped Nicolas a bit with the upstreaming of the v4l2 video encoder (728438). Julien Isorce and I talked about the missing parts of DMABuf support in gstreamer-vaapi, in particular the action path when the new libva API, for importing and exporting DMABuf, got merged (779146). With Thibault we played with the idea of a Jenkins server doing CI for gstreamer-vaapi. Also I did some kernel debugging, and found out why kmssink failed in db410c when the caps changed from RGB to YUV, thus Rob Clark cooked a patch.

Finally, I worked on a time-lapse video of the hackfest’s main room, only using GStreamer with gstreamer-vaapi in an Atom-based NUC. You can glance the code of the video grabber. Thanks to Luis de Bethencourt for the original idea and code.

GStreamer Spring Hackfest 2017 & GStreamer 1.12

2017-05-09T00:00:00Z

Greetings earthlings!Two things:

One #

GStreamer 1.12 is out! And with it, gstreamer-vaapi. Among other new features and improvements we have:

GstVaapiDisplay now inherits from GstObject, thus the VA display logging messages are better and tracing the context sharing is more readable.
When uploading raw images into a VA surfaces now VADeriveImages are tried first, improving the upload performance, if it is possible.
The decoders and the post-processor now can push dmabuf-based buffers to downstream under certain conditions. For example: GST_GL_PLATFORM=egl gst-play-1.0 video-sample.mkv --videosink=glimagesink
Refactored the wrapping of VA surface into gstreamer memory, adding lock when mapping and unmapping, and many other fixes.
Now vaapidecodebin loads vaapipostproc dynamically. It is possible to avoid it usage with the environment variable GST_VAAPI_DISABLE_VPP=1.
Regarding encoders: they have primary rank again, since they can discover, in run-time, the color formats they can use for upstream raw buffers and caps renegotiation is now possible. Also the encoders push encoding info downstream via tags.
About specific encoders: added constant bit-rate encoding mode for VP8 and H265 encoder handles P010_10LE color format.
Regarding decoders, flush operation has been improved, now the internal VA encoder is not recreated at each flush. Also there are several improvements in the handling of H264 and H265 streams.
VAAPI plugins try to create their own GstGL context (when available) if they cannot find it in the pipeline, to figure out what type of VA Display they should create.
Regarding vaapisink for X11, if the backend reports that it is unable to render correctly the current color format, an internal VA post-processor, is instantiated (if available) and converts the color format.

And

Two #

GStreamer Spring Hackfest 2017 is in less than two weeks!

It is going to be held at Igalia premises in A Coruña. Keep an eye on it ;)

GStreamer VAAPI 1.11.x (development branch)

2017-03-17T00:00:00Z

Greetings GstFolks!Last month the unstable release 1.11.2 of GStreamer hit the streets, and I would like to share with you all a quick heads-up of what we are working on in gstreamer-vaapi, since there are a lot of new stuff:

GstVaapiDisplay inherits from GstObject

GstVaapiDisplay is a wrapper for VADisplay. Before it was a custom C structure shared among the pipeline through the GstContext mechanism. Now it is a GObject based object, which can be queried, introspected and, perhaps later on, exposed in a separated library.
Direct rendering and upload

Direct rendering and upload are mechanisms based on using vaDeriveImage to upload an raw image into a VASurface, or to download a VASurface into a raw image, which is faster rather than exporting the VASurface to a VAImage.

Nonetheless we have found some issues with the direct rendering in new Intel hardware (Skylake and above), and we are still assessing if we keep it as default.
Improve the GstValidate pass rate

GstValidate provides a battery of tests for the whole GStreamer object, sadly, using gstreamer-vaapi, the battery didn’t output the same pass rate as without it. Though we still have some issues with the vaapsink that might need to be tackled in VA API, the pass rate has increased a lot.
Refactor the GstVaapiVideoMemory

We had refactor completely the internals of the VAAPI video memory (related with the work done for the direct download and upload). Also we have added locks when mapping and unmapping, to avoid race conditions.
Support dmabuf sharing with downstream

gstreamer-vaapi already had support to share dmabuf-based buffers with upstream (e.g. cameras) but now it is also capable to share dmabuf-based buffers with downstream with sinks capable of importing them (e.g. glimagesink under supported EGL).
Support compilation with meson

Meson is a new compilation machinery in GStreamer, along with autotools, and now it is supported also by gstreamer-vaapi.
Headless rendering improvements

There has been a couple of improvements in the DRM backend for vaapisink, for headless environments.
Wayland backend improvements

Also there has been improvements for the Wayland backend for vaapisink and GstVaapiDisplay.
Dynamically reports the supported raw caps

Now the elements query in run-time the VA backend to know which color formats does it support, so either the source or sink caps are negotiated correctly, avoiding possible error conditions (like negotiating a unsupported color space). This has been done for encoders, decoders and the post-processor.
Encoders enhancements

We have improve encoders a lot, adding new features such as constant bit rate support for VP8, the handling of stream metadata through tags, etc.

And many, many more changes, improvements and fixes. But there is still a long road to the stable release (1.12) with many pending tasks and bugs to tackle.

Thanks a bunch to Hyunjun Ko, Julien Isorce, Scott D Phillips, Stirling Westrup, etc. for all their work.

Also, Intel Media and Audio For Linux was accepted in the Google Summer Of Code this year! If your are willing to face this challenge, you can browse the list of ideas to work on, not only in gstreamer-vaapi, but in the driver, or other projects surrounding VAAPI.

Finally, do not forget these dates: 20th and 21h of May @ A Coruña (Spain), where the GStreamer Spring Hackfest is going to take place. Sign up!

GStreamer-VAAPI 1.10 (now 1.10.1)

2016-11-19T00:00:00Z

A lot of things have happened last month and I have neglected this report. Let me try to fix it.First, the autumn GStreamer Hackfest and the GStreamer Conference 2016 in Berlin.

The hackfest’s venue was the famous C-Base, where we talked with friends and colleagues. Most of the discussions were leaned towards the release 1.10. Still, Julien Isorce and I talked about our approaches for DMABuf sharing with downstream; we agreed on to explore how to implement an allocator based on GstDmaBufAllocator. We talked with Josep Torra about the performance he was getting with gstreamer-vaapi encoders (and we already did good progress on this). Also we talked with Nicolas Dufresne about v4l2 and kmssink. And many other informal chats along with beer and pizza.

GStreamer Hackfest a the C-Base (from @calvaris tweet)

Afterwards, in the GStreamer Conference, I talked a bit about the current status of GStreamer-VAAPI and what we can expect from release 1.10. Here’s the video that the folks from Ubicast recorded. You can see the slides here too.

https://www.youtube.com/watch?v=KP2NnLp0ESo

I spent most of the conference with Scott, Sree and Josep tracing the bottlenecks in the frame uploading to VA Intel driver, and trying to answer the questions of the folks who gently approached me. I hope my answers were any helpful.

And finally, the first of November, the release 1.10 hit the streets!

As it is already mentioned in the release notes, these are the main new features you can find in GStreamer-VAAPI 1.10

All decoders have been split, one plugin feature per codec. So far, the available ones, depending on the driver, are: vaapimpeg2dec, vaapih264dec, vaapih265dec, vaapivc1dec, vaapivp8dec, vaapivp9dec and vaapijpegdec (which already was split).
Improvements when mapping VA surfaces into memory, differentiating from negotiation caps and allocations caps, since the allocation memory for surfaces may be bigger than one that is going to be mapped.
vaapih265enc got support to constant bitrate mode (CBR).
Since several VA drivers are unmaintained, we decide to keep a white list with the va drivers we actually test, which is mostly the i915 and, in some degree, gallium from mesa project. Exporting the environment variable GST_VAAPI_ALL_DRIVERS disable the white list.
The plugin features are registered, in run-time, according to its support by the loaded VA driver. So only the decoders and encoder supported by the system are registered. Since the driver can change, some dependencies are tracked to invalidate the GStreamer registry and reload the plugin.
DMA-Buf importation from upstream has been improved, gaining performance.
vaapipostproc now can negotiate through caps the buffer transformations.
Decoders now can do reverse playback! But they only shows I frames, because the surface pool is smaller than the required by the GOP to show all the frames.
The upload of frames onto native GL textures has been optimized too, keeping a cache of the internal structures for the offered textures by the sink, improving a lot the performance with glimagesink.

This is the short log summary since branch 1.8:

1 Allen Zhang
19 Hyunjun Ko
1 Jagyum Koo
2 Jan Schmidt
7 Julien Isorce
1 Matthew Waters
1 Michael Olbrich
1 Nicolas Dufresne
13 Scott D Phillips
9 Sebastian Dröge
30 Sreerenj Balachandran
1 Stefan Sauer
1 Stephen
1 Thiago Santos
1 Tim-Philipp Müller
1 Vineeth TM
154 Víctor Manuel Jáquez Leal

Thanks to Igalia, Intel Open Source Technology Center and many others for making GStreamer-VAAPI possible.

VA-API and DRM/KMS in MinnowBoard

2016-07-01T00:00:00Z

In 2012 I started to work on a video renderer for GStreamer which uses directly the DRM/KMS kernel subsystem to display images. I even blogged about it, but I didn’t finished it.Nonetheless, in December last year, a customer asked us to finish the element, but this time focusing on i.MX6, rather than OMAP4. Consequently, early this year, kmssink got merged into upstream.

The video sink has some nice features, such as DMA-buf importing through the PRIME kernel interface. This feature makes possible a zero-copy path when the video decoder delivers dmabuf based frames. For this particular project, the kernel we used supports the CODA media subsystem, and through the GStreamer element v4l2videodec we could link pipelines negotiating dmabuf sharing, and thus we played videos very efficiently.

During the last GStreamer Hackfest, Nicolas Dufresne got working the MFC media subsystem, for Exynos, with v4l2videodec and kmssink, but I don’t remember if he also got the dmabuf sharing working.

All in all, kmssink had only been tested in a couple of ARM devices, so I wonder, as a gstreamer-vaapi co-maintainer, if it also works in x86 devices.

Some colleagues at Igalia, use a Minnowboard to test WebKit4Wayland, so I borrowed it, installed a minimal Fedora 23 in it, and, surprisingly, I found out that it has a nice VA-API support:

$ vainfo
error: can't connect to X server!
libva info: VA-API version 0.38.1
libva info: va\_getDriverName() returns 0
libva info: Trying to open /usr/lib64/dri/i965\_drv\_video.so
libva info: Found init function \_\_vaDriverInit\_0\_38
libva info: va\_openDriver() returns 0
vainfo: VA-API version: 0.38 (libva 1.6.2)
vainfo: Driver version: Intel i965 driver for Intel(R) Bay Trail - 1.6.2
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Simple            : VAEntrypointEncSlice
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointEncSlice
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointEncSlice
      VAProfileH264High               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointEncSlice
      VAProfileH264StereoHigh         : VAEntrypointVLD
      VAProfileVC1Simple              : VAEntrypointVLD
      VAProfileVC1Main                : VAEntrypointVLD
      VAProfileVC1Advanced            : VAEntrypointVLD
      VAProfileNone                   : VAEntrypointVideoProc
      VAProfileJPEGBaseline           : VAEntrypointVLD

Afterwards, I compiled GStreamer using gst-uninstalled in order to have kmssink and the master branch of gstreamer-vaapi. And got it! I had hardware accelerated decoding with VA-API, and video rendering using KMS/DRM. But, sadly, no zero copy, since, in order to have dmabuf sharing, we have to finish and to merge bug 755072 in gstreamer-vaapi.

Running the typical Big Bunny video (1080p, H.264) the playback consumes less than the 50% of the CPU usage, compared with the 100% of CPU usage with software decoders (and a lot of frames dropped).

I recorded a small video with the experiment as a proof.

Update: Javer Martinez told me in twitter, that MFC can do dma-buf export since uses vb2 and Exynos DRM can import, but he couldn’t get zero copy to work due missing format support.

GStreamer Hackfest 2016

2016-05-20T00:00:00Z

Yes, it happened again: the GStreamer Spring Hackfest 2016. This time in the beautiful city of Thessaloniki. Thanks a lot, Vivia and Sebastian, for making it happen.

My objective this time was to work with dma-buf support in gstreamer-vaapi. Though it is supported already, it needs a major clean up, and to extend its usage for downstream buffers (bugs 755072 and 765435).

In the way I learned that we need to update our internal API (called libgstvaapi), when handling dma-buf, to support mult-plane formats.

On the other hand, Nicolas Dufresne and I talked a bit about kmssink, libdrm and dma-buf. He managed to hack his Odroid U (Exynos3) to enable its V4L2 mem2mem video decoder and share buffers with kmssink. It was amazing. By the way, he promised me to write a blog post with the instructions to replicate his deed.

Photo by Luis de Bethencourt

Finally, we had a preview of Edward Hervey’s decodebin3. It is fun his test of switching the different audio streams in a media container (the different available languages) in every second or less. It was truly a multi-language audio!

In the meantime, we shared beers and meals, learning and laughing.

gstreamer-vaapi 1.8: the codec split

2016-04-06T00:00:00Z

On march 23th GStreamer 1.8 was released, along with all its bundled modules, and, of course, one of those modules is gstreamer-vaapi.Let us talk about this gstreamer-vaapi release, since there are several sweets!

First thing to notice is that the encoders have been renamed. Before they followed the pattern vaapiencode_{codec}, now they follow the pattern vaapi{codec}enc. The purpose of this change is twofold: to fix the plugins gtk-docs and to keep the usual element names in GStreamer. The conversion table is this:

Old	New
vaapiencode_h264	vaapih264enc
vaapiencode_h265	vaapih265enc
vaapiencode_mpeg2	vaapimpeg2enc
vaapiencode_jpeg	vaapijpegenc
vaapiencode_vp8	vaapivp8enc

But those were not the only name changes, we also have split the vaapidecode. Now we have a vaapijpegdec, which only decodes JPEG images, while keeping the old vaapidecode for video decoding. Also, vaapijpegdec was demoted to a marginal rank, because there are some problems in the Intel VA driver (which is the only one which supports JPEG decoding right now).

Note that in future releases, all the decoders will be split by codec, just as we did the JPEG decoder now; but first, we need to modify vaapidecodebin to choose a decoder in run-time based on the negotiated caps.

Please, update your scripts and code accordingly.

There are a ton of enhancements and optimizations too. Let me enumerate some of them: Vineeth TM fixed several memory leaks, and some compilations issues; Tim enabled vaapisink to send unhandled keyboard or mouse events to the application, making the usage of apps like gst-play-1.0 or apps based on GstPlayer be more natural; Thiago fixed the h264/h265 parsers, meanwhile Sree fixed the vp9 and the h265 ones too; Scott also fixed the h265 parser; et cetera. As may you see, H265/HEVC parser has been very active lately, it is the new thing!

I have to thank Sebastian Dröge, he did all the release work and also fixed a couple compilation issues.

This is the short log summary since 1.6:

2 Lim Siew Hoon
1 Scott D Phillips
8 Sebastian Dröge
3 Sreerenj Balachandran
5 Thiago Santos
1 Tim-Philipp Müller
8 Vineeth TM
16 Víctor Manuel Jáquez Leal

GStreamer VA-API under the umbrella of GStreamer

2016-02-09T00:00:00Z

We have a new GStreamer VA-API release: 1.6.0!Wait a minute, you might say, weren’t the last release 0.7?, and you will be correct; but something big has happened: GStreamer VA-API is now part of the official GStreamer project!

And this means a couple changes.

First of all, the official repository of the project has been moved, now it’s co-hosted along with the rest of the GStreamer components, in freedesktop.org (fdo):

Anonymous Git repository: git://anongit.freedesktop.org/gstreamer/gstreamer-vaapi

Developer Git repository: ssh://username@git.freedesktop.org/git/gstreamer/gstreamer-vaapi

Web Git gateway: https://cgit.freedesktop.org/gstreamer/gstreamer-vaapi

Second, the bug tracking has also moved, since now GStreamer VA-API is now a component of GStreamer, the new bugs must be filled here:

https://bugzilla.gnome.org/enter_bug.cgi?product=GStreamer&component=gstreamer-vaapi

And the bug list is here:

https://bugzilla.gnome.org/buglist.cgi?component=gstreamer-vaapi&product=GStreamer

What will happen with the old bugs? you will ask. Well, we will move them as soon as we have reviewed all them.

The third change, as you already noticed, is the version scheme. Now we are following the GStreamer version numbering to avoid confusion and to simplify our development. Hence, this release, 1.6.0, supports the current GStreamer stable version (1.6.3), and our current development branch is the 1.7.x. The future releases will follow the GStreamer version schemes and dependencies.

Sweet! but, what’s new in this release?. The answer is, not much. Really. Most of the changes are related with the coupling with the upstream processes (autoconf setup, documentation, etc.). Perhaps the most remarkable thing is the removal of the support libraries (libgstvaapi-*.so) used by the vaapi plugin: now they are compiled as one static library and linked to the GStreamer’s plugin. Also, the custom parsers were removed, and the plugin and elements documentation got better shape.

At code level, we had to push a huge indentation commit, in order to align with the GStreamer code style. This commit artificially kills the blame history, but it was our better option.

I ought to say that those were not the only changes at code level, Michael Olbrich fixed a missing frame release in the Wayland backend. And Sree, as usual, fixed a bunch of hardcore stuff. But specially I want to thank Tim-Philipp Müller, for helping us along the upstreaming process. And obviously to the Intel Open Source Technology Center, for let this happen.

Here’s the git’s short log summary since the last 0.7.0 release:

1 Joel Holdsworth
1 Michael Olbrich
9 Sreerenj Balachandran
4 Tim-Philipp Müller
42 Víctor Manuel Jáquez Leal

By the way, Igalia is hiring!

GStreamer VA-API 0.7.0

2015-12-08T00:00:00Z

GStreamer VA-API 0.7.0 is here! As is usually said, “grab it while it is fresh”, specially the distributions.In a previous blog post we explained in detail what is GStreamer VA-API. Also, last October, we talked about it in the GStreamer Conference 2015; you can glance the slides here.

This release includes three major changes:

Added support for VP9 decoding.
Improved a lot the HEVC decoder.
Some fixes in the integration with OpenGL platforms.

Please note that VP9 decoding is only supported from Braswell and Skylake onwards, using the Intel’s hybrid driver for libva, which is a subset of a VA-API backend, and it can be plugged to Intel’s VA-API backend (adding the –hybrid-codec parameter to the intel-driver’s configure script).

HEVC (a.k.a. H265) decoding is more stable and has a better performance. And, remember, it is only available in Skylake and Cherry View chipsets).

Now, OpenGL3 is handled correctly through GstGLTextureUploadMeta.

But there are a lot of fixes more, improvements and enhancements, such as better handling of H264 corrupted streams, better GstContext handling, the enabling of vaapidecodebin (finally!), etc.

Here’s the git’s short log summary since the last 0.6.0 release.

5 Gwenole Beauchesne
1 Jan Schmidt
2 Lim Siew Hoon
1 Mark Nauwelaerts
1 Olivier Crete
61 Sreerenj Balachandran
72 Víctor Manuel Jáquez Leal

GStreamer Conference 2015

2015-10-23T00:00:00Z

Last October 8th and 9th, the GStreamer Conference 2015 had happened in Dublin. It was plenty of interesting talks and colleagues. I really enjoyed it.I participated with our latest work in gstreamer-vaapi. You can glance the slides of my talk.

By the way, if you are in Dublin, let me recommend you to visit the Chester Beatty Library. There are a lot of antique, beautiful and rare books

GStreamer VA-API: A new release!

2015-07-16T00:00:00Z

A new release of gstreamer-vaapi is now available!! Since we have been working on it for the last months, I would like talk you about it.gstreamer-vaapi is a set of GStreamer plugins and libraries for hardware accelerated video processing using VA-API.

VA-API stands for “Video Acceleration - Application Programming Interface”, and is, simultaneously, a specification of an application programming interface, and its library implementation under the Open Source MIT license, whose purpose is to offer applications access to the GPU accelerated video processing capabilities, offloading those tasks from the CPU. Accelerated processing includes video decoding, sub-picture blending and rendering.

The library implementation (libva) is designed to be a front-end for many different back-ends. I am aware only of these back-ends:

As you can notice, only the Intel driver is well maintained, meanwhile the others haven’t been updated since several years ago. And of course, this issue has impacted on the bugs handled in gstreamer-vaapi (for example, bug #749554).

VA-API is designed to use various entry-points of the hardware accelerated processing in the GPU driver, such as VLD (also known as slice level acceleration), iDCT, among others.

The fact that VA-API uses slice level decode is important, since it is different from the software-based video decoders, were the decoding is at frame level. As consequence, more information is required from the current video parsers (bug #691712).

But let us back on track. VA-API, as we already said, is a set of GStreamer elements (vaapidecode, vaapipostroc, vaapisink, and several encoders) and a GObject-friendly library named libgstvaapi. This library wraps libva under a GObject/GStreamer semantics. I might talk about this library in another opportunity.

vaapisink is video sink with support for multiple display server protocols:

UPDATE (2015/07/20): As Sree commented, vaapisink does not use GLX/EGL for rendering, it uses VA-API, but rather vaapidecode/vaapipostproc can deal with GLX/EGL contexts so they can share buffers with other OpenGL-based elements.

This element is the most efficient way to render the video processed by other VA-API elements such as decoders and post-processors, since no media transformations are required. If we want to render the stream in other video sinks, some extra operations might be necessary. The worst case scenario is where we need to download the whole image from the GPU memory onto the CPU one, and the overall performance of the pipeline might be impacted.

vaapidecode is single decoding element that handles several codecs, depending on the back-end and the hardware available. The comprehensive list of possible enabled codecs is:

MPEG-2 (simple and main profiles)
MPEG-4 (simple, main and advance-simple profiles) / DivX / Xvid
H263 (baseline profiles)
H264 (baseline, constrained-baseline, main, high, multiview-high and stereo-high profiles)
H265 (main and main10)
WMV3 (simple and main profiles) / VC1
VP8 (version 0.3)
JPEG (baseline profile)

In the particular case of Intel chip-sets, the next table shows the codec support per each generation:

CHIPSET	MPEG2	H264	H265	VC1	VP8	JPEG
Skylake	Y	Y	Y	Y	Y	Y
Cherry View	Y	Y	Y	Y	Y	Y
Broadwell	Y	Y		Y		Y
Haswell	Y	Y		Y		Y
Ivybridge	Y	Y		Y		Y
Sandybridge	Y	Y		Y
Ironlake	Y	Y
G4x	Y

In the case of H264, there are multiple profiles, and every chip-set offers different profile sets. But for now, I will not dig in more on it.

In other back-ends the support shall be different. For example, I have a box with an GPU NVidia GeForce GT which supports MPEG2, MPEG4, H264 and VC1.

There is a branch for G45 GPU, where H264 decoding support is added. Nevertheless, is not official, it doesn’t merge with current master anymore, and the supported output color space is NV12 only (bug #745660).

An interesting thing to know is because of the slice level decoding, gstreamer-vaapi does its own parsing inside the decoder, even if a video parser was plugged before.

vaapipostproc is a video transform element which can do color space conversions, de-interlacing, sharpening and many other types of video filtering, such as:

Color conversion
Resize / Scale
Noise reduction
De-interlacing
Sharpening
Color Balance
Skin tone enhancement

Also, the availability of this elements and its filters, depends on the back-end and the hardware. For example, the VDPAU back-end doesn’t provide any post-processing capability.

In the case of Intel’s chip-sets these are the provided post-processing capabilities:

CHIPSET	Format	Noise	Deinterlace	Sharp	C.B.	STE
Skylake	Y	Y	Y	Y	Y	Y
Cherry View	Y	Y	Y	Y	Y	Y
Broadwell	Y	Y	Y	Y	Y	Y
Haswell	Y	Y	Y	Y	Y	Y
Ivybridge	Y	Y	Y
Sandybridge	Y	Y	Y
Ironlake	Y
G4x

In the case of the video encoders, they are split in different elements. Those implemented in gstreamer-vaapi are:

vaapiencode_mpeg2
vaapiencode_h264
vaapiencode_h265
vaapiencode_vp8
vaapiencode_jpeg

As far as I know, the only VA-API back-end that provides encoder is the Intel one, and this is the table of the encoders per chip-set:

CHIPSET	MPEG2	H264	H265	VP8	JPEG
Skylake	Y	Y	Y	Y	Y
Cherry View	Y	Y			Y
Broadwell	Y	Y
Haswell	Y	Y
Ivybridge	Y	Y
Sandybridge		Y
Ironlake
G4x

vaapidecodebin is a new bin composed by vaapidecode, a queue and vaapipostproc. Its purpose is bundle a complete decoding solution, with de-interlacing support. The purpose of the queue is to set the decoder to its full speed. The problem is, that since each back-end and chip-set might or might not support vaapipostproc we have to check for it in run-time (bug #749554).

    (-------------------------------------------------------)
    |                     vaapidecodebin                    |
    |   (-------------)    (-------)    (---------------)   |
    |-->| vaapidecode |--->| queue |--->| vaapipostproc |-->|
    |   (-------------)    (-------)    (---------------)   |
    |                                                       |
    (-------------------------------------------------------)

In my opinion, this should be just a temporal workaround meanwhile we have auto-plugging support of de-interlacers (bug #687182).

So far we have explained what is GStreamer VA-API. Let us now talk about what is new and hot in this release.

This is the short log summary since the last release, 0.5.10:

1 Adrian Cox
2 Alban Browaeys
30 Gwenole Beauchesne
1 Jacobo Aragunde Pérez
2 Jan Schmidt
1 Julien Isorce
1 Lim Siew Hoon
1 Martin Sherburn
4 Michael Olbrich
12 Olivier Crete
3 Simon Farnsworth
74 Sreerenj Balachandran
75 Víctor Manuel Jáquez Leal
1 Wind Yuan

The major changes in this release includes:

HEVC (H265) decoding and encoding support (available only in Skylake and Cherry View chip-sets)
VP8 encoder (Skylake)
JPEG encoder (Skylake and Cherry View)
vaapidecodebin element, which we have talked about above.
Support for EGL either in vaapidecode, vaapipostproc and vaapisink.
Skin tone enhancement support in vaapipostproc
Support for H.264 Multiview High profile encoding with more than 2 views, which encompasses Jan Schmidt’s efforts for stereoscopic / multi-view support in GStreamer (bug #611157)
A lot of improvements and round rough corners all over the place. Only to mention that we closed more than 70 bug reports along this cycle

Now, please test this new release, enjoy it, and if you find something ugly, let us known!!!

GStreamer Hackfest 2015

2015-03-18T00:00:00Z

Last weekend was the GStreamer Hackfest in Staines, UK, in the Samsung’s premises, who also sponsored the dinners and the lunches. Special thanks to Luis de Bethencourt, the almighty organizer!My main purpose was to sip one or two pints with the GStreamer folks and, secondarily, to talk about gstreamer-vaapi, WebKitGTK+ and the new OpenGL/ES support in gst-plugins-bad.

GStreamer Hackfest in Staines

About gstreamer-vaapi, there were a couple questions about some problems shown in downstream (stable releases in distributions) which I was happy to announce that they are mostly fixed in upstream. On the other hand, Sebastian Dröge was worried about the existing support of GStreamer 0.10 and I answered him that its removal is already in the pipeline. He looked pleased.

Related with gstreamer-vaapi and the new GstGL, we tested and merged a patch for GLES2/EGL, so now it is possible to render VA-API decoded video through glimagesink with (nearly) zero-copy. Sadly, this is not currently possible using GLX. Along the way I found a silly bug that came from a previous patch of mine and fixed it; also, we fixed other small bug in the gluploader .

In the WebKitGTK+ realm, I worked on a new functionality: to share the OpenGL context and the display of the browser with the GStreamer pipeline. With it, we could add gl filters into the pipeline. But honour to whom honour is due: this patch is a split of a previous patch done by Philippe Normand. The ultimate goal is to ditch the custom video sink in WebKit and reuse the glimagesink, with it’s new off-screen rendering feature.

Finally, on Sunday’s afternoon, I walked around Richmond and it is beautiful.

Richmond

Thanks to Igalia, Intel and all the sponsors that make possible the hackfest and my attendance.

https://www.youtube.com/watch?v=74vZ3ODUo00

GUADEC 2014

2014-08-05T00:00:00Z

The last Friday 25 of July, National Day of Galicia, started very early because I had to travel to Strasbourg, official seat of the European Parliament, not for any political duty, but for the GNOME Users and Developers European Conference, the GUADEC!My last GUADEC was in The Hague, in 2010, though in 2012, when it was hosted in Coruña, I attended a couple talks. Nonetheless, it had been a long time since I met the community, and it was a pleasure to me meet them again.

My biggest impression was the number of attendees. I remember the times in Turkey or in Gran Canaria where hundreds packed the auditoriums and halls. Nowadays the audience was smaller, but that is a good thing, since now you get in touch with the core of developers who drive and move the project easily.

We, Igalia, as sponsors, had a banner in the main room and a table in a corridor. Here is a picture of Juan to prove it:

Juan at the Igalia’s booth.

Also I ran across with Emmanuele Bassi, setting up a booth to show up the Endless Mobile OS, based on GNOME 3. The people at GUADEC welcomed with enthusiasm the user experience provided by it and the purpose of the project. Personally, I love it. If you don’t know the project, you should visit their web site.

The first talk I attended what the classic GStreamer update by Sebastian Dröge and Tim Müller. They talked about the new features in GStreamer 1.4. Neat stuff in there. I like the new pace of GStreamer, rather of the old stagnated evolution of 0.10 version.

Afterwards, Jim Hall gave us a keynote about Usability in GNOME. I really enjoyed that talk. He studied the usability of several GNOME applications such as Nautilus (aka Files), GEdit, Epiphany (aka Web), etc., as part of his Masters’ research. It was a pleasure to hear that Epiphany is regarded as having a good usability.

After lunch I was in the main room hearing Sylvain Le Bon about sustainable business models for free software. He talked about crowd funding, community management and related stuff.

The next talk was Christian Hergert about his project GOM, an object mapper from GObjects to SQLite, which is used in Grilo to prevent SQL injection by some plugins that use SQLite.

Later on, Marina Zhurakhinskaya gave us one of the best talks of the GUADEC: How to be an ally to women in tech. I encourage you to download the slides and read them. There I learned about the unicorn law and the impostor syndrome.

The day closed with the GNOME Foundation’s teams reports.

Sunday came and I arrived to the venue for the second keynote: Should We Teach The Robot To Kill by Nathan Willis. In his particular style, Nathan, presented a general survey of GNU/Linux in the Automotive Industry.

Next, one of main talks from Igalia: Web 3.12: a browser to make us proud, presented by Edu. It was fairly good. Edu showed us the latest development in WebKitGTK+ and Epiphany (aka Web). There were quite a few questions at the end of the talk. Epiphany nowadays is actively used by a lot of people in the community.

After, Zeeshan presented his GNOME boxes, an user interface for running virtual machines. Later on Alberto Ruiz showed us Fleet Commander, a web application to handle large desktop deployments.

And we took our classic group photo:

Group photo

That Sunday closed with the intern’s lighting talks. Cool stuff is being cooked by them.

On Monday I was in the venue when Emmanuele Bassi talked us about GSK, the GTK+ Scene Graph Kit, his new project, using as a starting point the lessons learned in Clutter. Its objective is to have a scene graph library fully integrated in GTK+.

After the lunch and the second part of the Foundation’s Annual General Meeting, Benjamin Otte gave an amusing talk about the CSS implementation in GTK+. Later, Jasper St. Pierre talked about the Wayland support in GNOME.

When the coffee break ended, the almighty Žan Doberšek gave the other talk from Igalia: Wayland support in WebKit2GTK+.

In the last day of the GUADEC, I attended Bastien Nocera’s talk: Hardware integration, the GNOME way, where he reviewed the history of his contributions to GNOME related with hardware integration and the goal of nicely support most of the hardware in GNOME, like compasses, gyroscopes, et cetera.

Afterwards, Owen Taylor talked us about the GNOME’s continuous integration performance testing, in order to know exactly why one release of GNOME is faster or slower than the last.

And the third keynote came: Matthew Garrett talked us about his experiences with the GNOME community and his vision about where it should go: to enhance the privacy and security of the users, something that many GNOMErs are excited about, such as Federico Mena.

Later on, David King talked about his plans for Cheese, the webcam application, turning it into a DBus service, using the current development of kdbus to sandbox the interaction with the hardware.

Afterwards Christian Hergert talked us about his plans for Builder, a new IDE for GNOME. Promising stuff, but we will see how it goes. Christian said that he is going to take a full year working on this project.

The GUADEC ended with the lighting talks, where I enjoyed one about the problems around the current encryption and security tools.

Finally, the next GUADEC host was unveiled: the Sweden Conspiracy: Gothenburg!

See you at GUADEC!

2014-07-24T00:00:00Z

Woah! Has been long time since I attended a GUADEC, besides the one in Coruña, of course. And I am very excited about it!My plan is to chill with other GNOME developers and talk about the future of GNOME OS.

I’m going to GUADEC 2014

Munich

2014-03-24T00:00:00Z

I spent a week in Munich. I went there for two reasons: to attend the Web & TV workshop organized by the W3C and to hack along with the gst-gang in the GStreamer Hackfest 2014. All these sponsored by Igalia, my company.I arrived to Munich on Tuesday evening, and when I reached the Marienplatz metro station, I ran across with a crowd of Bayern Munich fans, chanting songs about the glory of their team, huddling and dancing. And a lot of police officers surrounding the tracks.

The workshop was organized by the W3C Web and TV Interest Group, and intended to spark discussions around how to integrate and standardize TV technologies and the Web.

The first slide of the workshop

On Wednesday morning, the workshop began. People from Espial and Samsung talked about HbbTV, and japanese broadcasters talked about their Hybridcast. Both technologies seek to enhance the television experience, using the Internet Protocols, the first for Europe, and the former for Japan. Also, broadcasters from Chine showed their approach using ad-hoc technologies. I have to say that Hybridcast awed me.

http://www.youtube.com/watch?v=ThrOv2NqI00

Afterwards, most of the workshop was around the problem of the companion device. People showed their solutions and proposals, in particular about device discovering, and data sharing and control. All the solutions relied on WebSockets and WebRTC for the data sharing between devices.

http://www.youtube.com/watch?v=0xraXLZY2gc

During the panels, I enjoyed a lot the participation of Jon Piesing, in particular his slide summarizing the specifications used by the HbbTV V2. It’s like juggling specs!

Specifications used by HbbTV V2

Broadcaster are very interested in Encrypted Media Extension and Media APIs, for example the specification for a Tuner API. Also there’s a lot of expectation about meta-data handling and defining common TV ontologies.

Finally, there were a couple talks about miscellaneous technologies surrounding the IPTV broadcasting.

The second stage of my visit to Bavaria’s Capital, was the GStreamer Hackfest. It was in the Google Offices, near to the Marienplatz.

Christan Schaller has made a very good summary of what happened along the hackfest. From my side, I worked with Nicolas Dufresne with the v4l2 video converter for the Exynos4, which is a piece required for the hardware acceleration decoding for that platform using v4l2 video decoder.

Boosting WebKitGTK+ compilation for armhf with icecream

2013-12-31T00:00:00Z

Some time ago I needed to jump into the fix-compile-test loop for WebKitGTK+, but in the armhf architecture, speaking in terms of Debian/Ubuntu.To whom don’t know, WebKitGTK+ is huge, it is humongous, and it takes a lot of resources to compile. For me, at first glance, was impossible to even try to compile it natively in my hardware, which, by the way, is an Odroid-X2. So I setup a cross-compilation environment.

And I failed. I could not cross-compile the master branch of WebKitGTK+ using as root file system, a bootstrapped Debian. It is supposed to be the opposite, but all the multiarch thing made my old and good cross-compilation setup (based on scratchbox2) a bloody hell. Long story short, I gave up and I took more seriously the idea of native builds. Besides, Ubuntu and Debian does full native builds of their distributions for armhf, not to say that the Odroid-X2 has enough power for give it a try.

It is worth to mention that I could not use Yocto/OE or Buildroot, though I would love to use them, because the target was a distribution based on Debian Sid/Ubuntu, and I would not afford a chroot environment only for WebKitGTK+.

With a lot of patience I was able to compile, in the Odroid, a minimalist configuration of WebKitGTK+ without symbols. As expected, it took ages (less than 3 hours, if I remember correctly)

Quickly an idea popped out in the office: to use distcc. I grabbed as many board based on ARMv7 I could find: another Odroid-X2, a couple Pandaboards, an Arndaleboard, and an IFC6410, installed in them a distcc compilation setup.

And yes, the compilation time went down, but not that much, though I don’t remember how much.

Many of the colleagues at the office migrated from distcc to icecream. Particularly, Juan A. Suárez told me about his experiments with icecc and his Raspberry pi. I decided to give it a shoot.

Icecream permits to do cross-compilation because the scheduler can deliver, into the compilation host, the required tool-chain by the requester.

First, you should have one or several cross tool-chains, one for each compilation tuple. In this case we will have only one: to compile in X86_64, generating code for armfh. Luckily, embdebian provides it, out of the box. Nevertheless you could use any other mean to obtain it, such as crosstool.

Second, you need the icecc-create-env script to create the tarball that the scheduler will distribute to the compilation host.

  $ /usr/lib/icecc/icecc-create-env \\
          --gcc /usr/bin/arm-linux-gnueabihf-gcc-4.7 \\
                /usr/bin/arm-linux-gnueabihf-g++-4.7

The output of this script is an archive file containing all the files necessary to setup the compiler environment. The file will have a random unique name like “ddaea39ca1a7c88522b185eca04da2d8.tar.bz2” per default. You will need to rename it to something more expressive.

Third, copy the generated archive file to board where your code will be compiled and linked, in this case WebKitGTK+.

For the purpose of this text, I assume that the board has already installed and configured the icecc daemon. Beside, I use ccache too. Hence my environment variables are more or less like these:

CCACHE_PREFIX=icecc CCACHE_DIR=/mnt/hd/.ccache # /mnt/hd is a mounted hard disk through USB. PATH=/usr/lib/ccache:… # where Debian sets the compiler’s symbolic links

Finally, the last pour of magic is the environment variable ICECC_VERSION. This variable needs to have this pattern

<native_archive_file>(,<platform>:<cross_archive_file>=<target>)*.

Where <native_archive_file> is the archive file with the native tool-chain. <platform> is the host hardware architecture. <cross_archive_file> is the archive file with the cross tool-chain. <target> is the target architecture of the cross tool-chain.

In my case, the target is not needed because I’m doing native compilation in armhf. Hence, my ICECC_VERSION environment variable looks like this:

ICECC_VERSION=/mnt/hd/tc/native-compiler.tar.gz,x86_64:/mnt/hd/tc/arm-x86-compiler.tar.gz

And that’s it! Now I’m using the big iron available in the office, reducing the time of a clean compilation in less than an hour.

As a final word, I expect that this compilation time will be reduced a bit more using the new cmake build infrastructure in WebKitGTK+.

Composited video support in WebKitGTK+

2013-07-26T00:00:00Z

A couple months ago we started to work on adding support for composited video in WebKitGTK+. The objective is to play video in WebKitGTK+ using the hardware accelerated path, so we could play videos at high definition resolutions (1080p).

How does WebKit paint? #

Basically we can perceive a browser as an application for retrieving, presenting and traversing information on the Web.

For the composited video support, we are interested in the presentation task of the browser. More particularly, in the graphical presentation.

In WebKit, each HTML element on a web page is stored as a tree of Node objects called the DOM tree.

Then, each Node that produces visual output has a corresponding RenderObject, and they are stored in another tree, called the Render Tree.

Finally, each RenderObject is associated with a RenderLayer. These RenderLayers exist so that the elements of the page are composited in the correct order to properly display overlapping content, semi-transparent elements, etc.

It is worth to mention that there is not a one-to-one correspondence between RenderObjects and RenderLayers, and that there is a RenderLayer tree as well.

Render Trees in WebKit (from GPU Accelerated Compositing in Chrome).

WebKit fundamentally renders a web page by traversing the RenderLayer tree.

What is the accelerated compositing? #

WebKit has two paths for rendering the contents of a web page: the software path and hardware accelerated path.

The software path is the traditional model, where all the work is done in the main CPU. In this mode, RenderObjects paint themselves into the final bitmap, compositing a final layer which is presented to the user.

In the hardware accelerated path, some of the RenderLayers get their own backing surface into which they paint. Then, all the backing surfaces are composited onto the destination bitmap, and this task is responsibility of the compositor.

With the introduction of compositing an additional conceptual tree is added: the GraphicsLayer tree, where each RenderLayer may have its own GraphicsLayer.

In the hardware accelerated path, it is used the GPU for compositing some of the RenderLayer contents.

Accelerated Compositing in WebKit (from Hardware Acceleration in WebKit).

As Iago said, the accelerated compositing, involves offloading the compositing of the GraphicsLayer onto the GPU, since it does the compositing very fast, releasing that burden to the CPU for delivering a better and more responsive user experience.

Although there are other options, typically, OpenGL is used to render computing graphics, interacting with the GPU to achieve hardware acceleration. And WebKit provides cross-platform implementation to render with OpenGL.

How does WebKit paint using OpenGL? #

Ideally, we could go from the GraphicsLayer tree directly to OpenGL, traversing it and drawing the texture-backed layers with a common WebKit implementation.

But an abstraction layer was needed because different GPUs may behave differently, they may offer different extensions, and we still want to use the software path if hardware acceleration is not available.

This abstraction layer is known as the Texture Mapper, which is a light-weight scene-graph implementation, which is specially attuned for an efficient usage of the GPU.

It is a combination of a specialized accelerated drawing context (TextureMapper) and a scene-graph (TextureMapperLayer):

The TextureMapper is an abstract class that provides the necessary drawing primitives for the scene-graph. Its purpose is to abstract different implementations of the drawing primitives from the scene-graph.

One of the implementations is the TextureMapperGL, which provides a GPU-accelerated implementation of the drawing primitives, using shaders compatible with GL/ES 2.0.

There is a TextureMapperLayer which may represent a GraphicsLayer node in the GPU-renderable layer tree. The TextureMapperLayer tree is equivalent to the GraphicsLayer tree.

How does WebKitGTK+ play a video? #

As we stated earlier, in WebKit each HTML element, on a web page, is stored as a Node in the DOM tree. And WebKit provides a Node class hierarchy for all the HTML elements. In the case of the video tag there is a parent class called HTMLMediaElement, which aggregates a common, cross platform, media player. The MediaPlayer is a decorator for a platform-specific media player known as MediaPlayerPrivate.

All previously said is shown in the next diagram.

Video in WebKit. Three layers from top to bottom

In the GTK+ port the audio and video decoding is done with GStreamer. In the case of video, a special GStreamer video sink injects the decoded buffers into the WebKit process. You can think about it as a special kind of GstAppSink, and it is part of the WebKitGTK+ code-base.

And we come back to the two paths for content rendering in WebKit:

In the software path the decoded video buffers are copied into a Cairo surface.

But in the hardware accelerated path, the decoded video buffers shall be uploaded into a OpenGL texture. When a new video buffer is available to be shown, a message is sent to the GraphicsLayer asking for redraw.

Uploading video buffers into GL textures #

When we are dealing with big enough buffers, such as the high definition video buffers, copying buffers is a performance killer. That is why zero-copy techniques are mandatory.

Even more, when we are working on a multi-processor environment, such as those where we have a CPU and a GPU, switching buffers among processor’s contexts, is also very expensive.

It is because of these reasons, that the video decoding and the OpenGL texture handling, should happen only in the GPU, without context switching and without copying memory chunks.

The simplest approach could be that decoder deliver an EGLImage, so we could blend the handle into the texture. As far as I know, the gst-omx video decoder in the Raspberry Pi, works in this way.

GStreamer added a new API, that will be available in the version 1.2, to upload video buffers into a texture efficiently: GstVideoGLTextureUploadMeta. This API is exposed through buffer’s metadata, and ought be implemented by any downstream element that deals with the decoded video frames, most commonly the video decoder.

For example, in gstreamer-vaapi there are a couple patches (which still are a work-in-progress) in bugzilla, enabling this API. In the low level, calling gst_video_gl_texture_upload_meta_upload() will call vaCopySurfaceGLX(), which will do an efficient copy of the VA-API surface into a texture using a GLX extension.

Demo #

This is an old demo, when all the pieces started to fit, but no the current performance. Still, it shows what has been achieved:

https://www.youtube.com/watch?v=ekt3oj5oApo

Future work #

So far, all these bits are already integrated in WebKitGTK+ and GStreamer. Nevertheless there are some open issues.

gstreamer-vaapi et all: GStreamer 1.2 is not released yet, and its new API might change. Also, the port of gstreamer-vaapi to GStreamer 1.2 is still a work in progress, where the available patches may have rough areas.Also, there are many other projects that need to be updated with this new API, such as clutter-gst and provide more feedback to the community. Another important thing is to have more GStreamer elements implementing these new API, such as the texture upload and the caps features
Tearing: The composited video task unveiled a major problem in WebKitGTK+: it does not handle the vertical blank interval at all, causing tearing artifacts, clearly observable in high resolutions videos with high motion.WebKitGTK+ composites the scene off-screen, using X Composite redirected window, and then display it at a X Damage callback, but currently, GTK+ does not take care of the vertical blank interval, causing this tearing artifact in heavy compositions. At Igalia, we are currently researching for a way to fix this issue.
Performance: There is always room for performance improvement. And we are always aiming in that direction, improving the frame rate, the CPU, GPU and memory usage, et cetera. So, keep tuned, or even better, come and help us.

GPhone v0.2

2013-05-05T00:00:00Z

It is almost 5 months since I published the first release of GPhone. Though I haven’t abandoned it, I swork on it slowly but steadily. And to prove it, I’m releasing a new version, the v0.2.What does it include? Mainly bug fixes. The new features are still a work in progress. You can browse the next branch in git to glance it.

Change Log:

Able to use GSimpleAsyncResult for glib <= 2.35
Added Vala (0.16) back compatibility, including a gstreamer-1.0.vapi
Added a script for build and update a jhbuild environment: update-jhbuild-env. It handles the gphone’s dependencies.
Notify to the user if his NAT type is not compatible with a VoIP session.
Renamed accounts to registrars.
Handle networking signaling
Create a global header file
Many bug fixes

https://github.com/ceyusa/gphone/tags

GStreamer Hackfest 2013 - Milan

2013-04-08T00:00:00Z

Last week, from 28th to 31th of March, some of us gathered at Milan to hack some bits of the GStreamer internals. For me was a great experience interact with great hackers such as Sebastian Dröge, Wim Taymans, Edward Hervey, Alessandro Decina and many more. We talked about GStreamer and, more particularly, we agreed on new features which I would like to discuss here.

For sake of completeness, let me say that I have been interested in hardware accelerated multimedia for a while, and just lately I started to wet my feet in VAAPI and VDPAU, and their support in our beloved GStreamer.

GstContext #

The first feature that reached upstream is the GstContext. Historically, in 2011, Nicolas Dufresne added GstVideoContext as an interface to a share video context (such as display name, X11 display, VA-API display, etc.) among the pipeline elements and the applications. But now, Sebastian, generalized the interface to a container to stores and shares any kind of contexts between multiple elements and the application.

The first approach, that is still living in gst-plugins-bad, was merely a wrapper to a custom query to set or request a video context. But now, the context sharing is part of the pipeline setup.

An element that needs a shared context must follow these actions:

Check if the element already has a context
Query downstream for the context
Post a message in the bus to see if the application has one to share.
Create the context if there is none, post a message and send an event letting know that the element has the context.

You can see the example of the eglglessink to know how to use this feature.

GstVideoGLTextureUploadMeta #

Also in 2011, Nicolas Dufresne, added a helper class to upload a buffer into a surface (OpenGL texture, VA API surface, Wayland surface, etc.). This is quite important since the new video players are scene based, using framework such as Clutter or OpenGL directly, where the video display is composed by various actors, such as the multimedia controls widgets.

But still, this interface didn’t fit well for GStreamer 1.0, until now, where it was introduced in the figure of a buffer’s meta, though this meta is only specific for OpenGL textures. If the buffer provides this new GstVideoGLTextureUploadMeta meta, a new function gst_video_gl_texture_upload_meta_upload() is available to upload that buffer into an OpenGL texture specified by its numeric identifier.

Obviously, in order to use this meta, it should be proposed for allocation by the sink. Again, you can see the case of eglglessink as example.

Caps Features #

The caps features are a new data type for specify a specific extension or requirement for the handled media.

From the practical point of view, we can say that caps structures with the same name but with a non-equal set of caps features are not compatible, and, if a pad supports multiple sets of features it has to add multiple equal structures with different feature sets to the caps.

Empty GstCapsFeatures are equivalent with the GstCapsFeatures handled by the common system memory. Other examples would be a specific memory types or the requirement of having a specific meta on the buffer.

Again, we can see the example of the capsfeatures in eglglessink, because now the gst-inspect also shows the caps feature of the pads:

    Pad Templates:
      SINK template: 'sink'
        Availability: Always
        Capabilities:
          video/x-raw(memory:EGLImage)
                     format: { RGBA, BGRA, ARGB, ABGR, RGBx,
                               BGRx, xRGB, xBGR, AYUV, Y444,
                               I420, YV12, NV12, NV21, Y42B,
                               Y41B, RGB, BGR, RGB16 }
                      width: [ 1, 2147483647 ]
                     height: [ 1, 2147483647 ]
                  framerate: [ 0/1, 2147483647/1 ]
          video/x-raw(meta:GstVideoGLTextureUploadMeta)
                     format: { RGBA, BGRA, ARGB, ABGR, RGBx,
                               BGRx, xRGB, xBGR, AYUV, Y444,
                               I420, YV12, NV12, NV21, Y42B,
                               Y41B, RGB, BGR, RGB16 }
                      width: [ 1, 2147483647 ]
                     height: [ 1, 2147483647 ]
                  framerate: [ 0/1, 2147483647/1 ]
          video/x-raw
                     format: { RGBA, BGRA, ARGB, ABGR, RGBx,
                               BGRx, xRGB, xBGR, AYUV, Y444,
                               I420, YV12, NV12, NV21, Y42B,
                               Y41B, RGB, BGR, RGB16 }
                      width: [ 1, 2147483647 ]
                     height: [ 1, 2147483647 ]
                  framerate: [ 0/1, 2147483647/1 ]

Parsers meta #

This is a feature which has been pulled by Edward Hervey. The idea is that the video codec parsers (H264, MPEG, VC1) attach a meta into the buffer with a defined structure that carries that new information provided by the codified stream.

This is particularly useful by the decoders, which will not have to parse again the buffer in order to extract the information they need to decode the current buffer and the following.

For example, here is the H264 parser meta definition.

VDPAU #

Another task pulled by Edward Hervey, for which I feel excited, is the port of VDPAU decoding elements to GStreamer 1.0.

Right now only the MPEG decoder is upstreamed, but MPEG4 and H264 are coming.

As a final note, I want to thank Collabora and Fluendo for sponsoring dinners. A special thank you, as well, for Igalia which covered my travel expenses and attendance to the hackfest.

GStreamer Hackfest 2013

2013-03-26T00:00:00Z

Next Thursday I’ll be flying to Milan to attend the 2013 edition of the GStreamer Hackfest. My main interests are hardware codecs and GL integration, particularly VA API integrated with GL-based sinks.Thanks Igalia for sponsoring my trip!

Igalia (three) week in WebKit: Media controls, Notifications, and Clang build

2013-03-04T00:00:00Z

Hi all,Three weeks have passed since I wrote the last WebKit report, and they did so quick that it scares me. Many great things have happened since then.

Let’s start with my favorite area: multimedia. Phil landed a patch that avoids muting the sound if the audio pitch is preserved. And Calvaris finally landed his great new media controls. Now watching videos in WebKitGTK+ is a pleasure.

Video controls

Rego keeps his work on adding more tests to WebKitGTK+, also he also wrote a fine document on how to build Epiphany with WK2 from git/svn.

Claudio, besides his work in the snapshots API that we already commented, retook the implementation of the notifications API for WebKitGTK+. And, while implementing it, he fixed some crashers in WK2’s implementation. He has also given us an early screencast with the status of the notifications implementation: Check it out! (video).

Carlos García Campos, besides working hard on the stable and development releases of WebKitGTK+ library, has also landed a couple of fixes. Meanwhile, Dape removed some dependencies, making the code base more clean.

Žan, untiredly, has been poking all around the WebKit project, keeping the GTK port kicking and healthy: He fixed code; cleaned up scripts, autogenerated code and enhanced utility scripts; he also enabled more tests in the development builds. But his most impressive work in progress is enabling the Clang build of the GTK port.

But there’s more! Žan setup a web page were you can visualize the status of WebKit2 on several ports: http://www.iswk2buildbrokenyet.com/

Igalia week in WebKit: Fixes, gardening, resources API and MathML

2013-02-01T00:00:00Z

It is time for another weekly report on what is going on WebKit and Igalia.Sergio reported and helped to debug different crashes in WebKit accessibility. He improved the robustness of build-webkit. Also, he fixed a bug in the defined network buffer size, simplified a bit the code that close the network connections, and fixed a bug in libsoup within a crash in webkit

And this week we have a new face too, at least in these reports: Adrián. He enabled the opus codec in the MIME type list. After that, he decided to give a try to a native compilation in armv5tel board. Crazy? A bit, but fun too.

Rego continues his hard work on enabling tests for WebKitGTK+, like testing the text direction setting, or unflag blocked tests that already pass, and many other that are still work in progress. Also, he got landed his patch for the bug found in the GtkLauncher.

Claudio, whilst he waits for the review of his API to retrieve a snapshot, he retook the Notifications API work for WebKitGTK+, particularly for WebKit2. Also, and just for sake of landing something, he fixed a couple of minor compilation glitches.

Philippe still kicking the fullscreen support in WebKit using GStreamer. And for relaxing purposes, he updated this patch for AudioSourceProvider.

Žan works tirelessly keeping the bots working as good as possible: disabling the build of wk2 on EWSs, meanwhile the wk2 storm appeases; cleaned building configuration options, removed deprecated methods for unit tests, enhanced the webkit-patch script and much more. Besides this needed work, he also started to hack on completing the WTFURL implementation for the WebCore’s KURL interface. WTFURL is pretty much what WebKit (as a project) would like to use in the future. WTFURL is based on the GoogleURL code (which is what the Chromium port is at the moment using as the backend of KURL).

The great WebKit hacker, Martin Robinson is exploring uncharted territory in the project: He’s trying to get away from port-specific things, by scratching in the core stuff, but the serendipity showed up, so he found some pretty serious port-specific bugs that have relatively straight-forward fixes.

Martin started to work on MathML and noticed that WebKit MathML support for the mathvariant attribute is unfinished. And this issue led him done to set of patches to overcome this situation, like fixing the freetype usage, or adding the logic to append UChar32 onto StringBuilder

In addition to all this work, Martin is working on a patch for mathvariant itself. The mathvariant attribute allows easily using the Mathematical Alphanumeric Symbols in MathML without having to type out XML entities. For instance this:

<mi mathvariant="fraktur">HI GUYZ!</mi>

will be rendered like this:

output

Carlos García Campos cooked a patch for fixing the WebKit2 GTK+ API by implementing the resources API, removed recently, using injected bundle. This is another effort to bring back WebKit2 into WebKitGTK+.

Dape is still pushing the Qt/WebKit2 spellcheck support with a new iteration of the patch. He also worked on the removal of the GDK dependency from ImageDiff.

Finally, I finished a first iteration of my patch for pitch preservation, and also it got landed!

Announcing GPhone v0.10

2013-01-28T00:00:00Z

Hi folks!As many of you may know, lately I have been working on Ekiga and Opal. And, as usually happens to me, I started to wonder how I would re-write that piece of software. My main ideas growth clearly: craft a GObject library, ala WebKitGTK+, wrapping Opal’s classes, paying attention to the gobject-introspection, also, redirecting the Opal’s multimedia rendering to a GStreamer player, and, finally, write the application with Vala.

The curiosity itched me hard, so I started to hack, from time to time, these ideas. In mid-November, last year, I had a functional prototype, which only could make phone calls in a disgraceful user interface. But I got my proof of concept. Nonetheless, as I usually do, I didn’t dropped the pet project, continuing the development of more features.

And today, I am pleased to present you, the release v0.1 of GPhone.

Well, I guess a screencast is mandatory nowadays:

http://www.youtube.com/watch?v=uLXuF1a3V5I

Igalia WebKit week: welcome Žan and Rego!

2013-01-24T00:00:00Z

This a new weekly WebKit Igalia’s report. And the last week has been a shaky one.Let’s start with a warm welcome to Žan Doberšek as collaborator student, who is working hard in a lot of cool matters: gardening the bots, cleaning up the Coverity run output, modifying the WebCoreTestSupport to decrease the dependency WebKit2 API, digging in a stack size problem in JSC, and much more fun stuff.

Meanwhile Joanie, after a lot of restless hacking hours, could fix a crash in tables accessibility, and saving us from many other accessibility crashes. She is working to make the world safer for table users everywhere!

But we have more new faces around: Our dear colleague, Rego, is getting his feet wet, and he started finding and fixing new bugs and he is enabling more tests in the Gtk+ port.

Calvaris is following his efforts for enhancing the user experience with the media controls for HTML5. Here is a recent screenshot of these controls:

Video controls

Claudio is pushing the accelerator pedal for the snapshot API in WebKit2. With this API the applications would retrieve a snapshot from a webview as in the Overview mode in Epiphany.

Philippe is working on the fullscreen video porting the GStreamerGWorld module to GStreamer 1.0, while he is still hacking on the AudioSourceProvider for WebAudio.

And the last but not the least, Dape is working real hard on the Qt spell-check support, and he also proposed solutions for Qt WebNotification support.

And that is all for now. See you!

Two video streams simultaneously in Ekiga

2013-01-21T00:00:00Z

As we talked earlier, we added support for H.239 in Ekiga, and we were able to show the main role video and the slides role video, one stream at the time.But now we did a step forward: we wanted to view both videos simultaneously. So, we hacked again Ekiga and we added that feature, when a second video stream arrives, another window is showed with the secondary role stream.

Here is the mandatory screencast:

http://www.youtube.com/watch?v=Bj_k0jZ2_aQ

Weekly Igalia's report on WebKit

2013-01-17T00:00:00Z

Hi webkitters,This weekly report project was supposed to start after the last WebKit hackfest, but my holidays got in between and now I’m recovering of them ;)

Here we go:

In summary, in these last three weeks we have had 15 commits and done 23 reviews.

Martin and Carlos have been working on the authentication mechanisms. Now they can be hooked ,through the web view API, by the applications, which could take control of the dialogues and credentials handling.

Martin has also been dealing with text rendering with complex layouts (such as Arabic). This effort leaded, finally, to the removal of Pango in favor of Harfbuzz.

Now let’s talk about Carlos’ baby monster: the injected bundle patch. As you know, in WebKit2, the engine has been split in two isolated processes the UI and Web processes. The first is in charge of the user interface, and the former deals with HTML, CSS and JavaScript handling. Meanwhile this approach adds more robustness and responsiveness, also imposes more complexity because it is required an IPC mechanism to interact with the Web process. This is particularly hard for accessing to the DOM bindings.

Carlos, since the last year, has been working on his injected bundle patch, which offers a mean to support loading plugins in the web process using injected bundle. Hence, through DBus, an application could load a plugin to communicate, indirectly, with the Web process. This approach is supposed to be the milestone for the DOM bindings in WK2GTK, and also provides a mean to pre-fetch DNS registries. This patch has been happily pushed just recently, in the second week of January.

If this was not enough, Carlos also released the development version of WebKitGTK+ v1.11.4.

Now let us go to the multimedia realm, my favorite land.

Philippe finished the port of his patch for WebAudio support to GStreamer 1.0 as backend. And now he is porting the full-screen support in Gst 0.10 to Gst 1.0 in order to reuse ans share the same base code. Aligned with WebAudio, Philippe is developing a new audio source provider that gathers raw audio data from the MediaPlayer and pipe them into the AudioBus if it is required.

Xabier has been working to deliver a nice and neat HTML5 media controls, using stylable GTK+ controls. And myself, I’m still playing with the audio pitch preservation.

Another great landmark for us is a11y, and here Joanie has been working hard to bring back the accessibility tests on GTK to a sane state. And also keeps her efforts to enable an access to WebKit for Orca.

In other sort of things, Berto has been fighting against a bug on GtkLaunch, which was shown in Epiphany too when displaying only images. Meanwhile, Dape, lurked on spell checking support for Qt WebKit2. And Sergio enabled, by default, the WebP image handling.

The thrilling story about the H.239 support in Ekiga

2012-09-25T00:00:00Z

The development of Ekiga 4.0 has begun, and three months ago I started a project aimed to support H.239 in Ekiga.H.239 is an ITU recommendation and it is titled “Role management and additional media channels for H.300-series terminals”. Its purpose is the definition of procedures for use more than one video channel, and for labelling those channels with a role.

A traditional video-conference has an audio channel, a video channel and an optional data channel. The video channel typically carries the camera image of the participants. The H.239 recommendation defines rules and messages for establishing an additional video channel, often to transmit presentation slides, while still transmitting the video of the presenter.

For presentations in multi-point conferencing, H.239 defines token procedures to guarantee that only one endpoint in the conference sends the additional video channel which is then distributed to all conference participants.

Ekiga depends on Opal, a library that implements, among other things, the protocols used to send voice over IP networks (VoIP). And, according to the documentation, Opal had already support for H.239 since a while, for what we assumed that enable it in Ekiga would be straight forward.

After submitting a few patches to Opal and its base library, PTlib, then I setup a jhbuild environment to automate the building of Ekiga and its dependencies. Soon, I realized that the task was not going to be as simple as we initially assumed: the support for H.323, in Ekiga, is not as mature as the SIP support.

I have to mention that along the development process, in order to test my scratched code, I used this H.239 application sample, a small MS Windows program which streams two video channels (the main role and the slides role). The good news is that it works fine in Wine.

Well, the true is that the activation of the client support for H.239 in Ekiga was easy. The real problem began with the video output mechanism. Ekiga has a highly complex design for drawing video frames. Too complex in my opinion. One of the first thing I did to understand the video frames display, was to sketch a sequence diagram from when a decoded frame is delivered by Opal to when the frame is displayed in the screen. And here is the result:

Sequence frame

As I said, it is too complex for me, but the second stream had to be displayed. Fearfully, I started to refactor some code inside Ekiga and adding more parameters to handle another video stream. But finally, I just stepped forward, and submitted a bunch of patches that finally displays a second stream.

The current outcome is shown in the next screen cast: Screencast Ekiga H.239 (OGV / 7.8M)

A GStreamer Video Sink using KMS

2012-06-20T00:00:00Z

The purpose of this blog post is to show the concepts related to the GstKMSSink, a new video sink for GStreamer 1.0, co-developed by Alessandro Decina and myself, done during my hack-fest time in the Igalia’s multimedia team.One interesting thing to notice is that this element shows it is possible to write DRI clients without the burden of X Window.

Brief introduction to graphics in Linux #

If you want to dump images onto your screen, you can simply use the frame buffer device. It provides an abstraction for the graphics hardware and represents the frame buffer of the video hardware. This kernel device allows user application to access the graphics hardware without knowing the low-level details [1].

In GStreamer, we have two options for displaying images using the frame buffer device; or three, if we use OMAP3: fbvideosink, fbdevsink and gst-omapfb.

Nevertheless, since the appearance of the GPUs, the frame buffer device interface has not been sufficient to fulfill all their capabilities. A new kernel interface ought to emerge. And that was the Direct Rendering Manager (DRM).

What in the hell is DRM? #

The DRM layer is intended to support the needs of complex graphics devices, usually containing programmable pipelines well suited to 3D graphics acceleration [2]. It deals with [3]:

A DMA queue for graphic buffers transfers [4].
It provides locks for graphics hardware, treating it as shared resource for simultaneous 3D applications [5].
And it provides secure hardware access, preventing clients from escalating privileges [6].

The DRM layer consists of two in-kernel drivers: a generic DRM driver, and another which has specific support for the video hardware [7]. This is possible because the DRM engine is extensible, enabling the device-specific driver to hook out those functionalities that are required by the hardware. For example, in the case of the Intel cards, the Linux kernel driver i915 supports this card and couples its capabilities to the DRM driver.

The device-specific driver, in particular, should cover two main kernel interfaces: the Kernel Mode Settings (KMS) and the Graphics Execution Manager (GEM). Both elements are also exposed to the user-space through the DRM.

With KMS, the user can ask the kernel to enable native resolution in the frame buffer, setting certain display resolution and colour depth mode. One of the benefits of doing it in kernel is that, since the kernel is in complete control of the hardware, it can switch back in the case of failure [8].

In order to allocate command buffers, cursor memory, scanout buffers, etc., the device-specific driver should support a memory manager, and GEM is the manager with more acceptance these days, because of its simplicity [9].

Beside to the graphics memory management, GEM ensures conflict-free sharing of data between applications by managing the memory synchronization. This is important because modern graphics hardware are essentially NUMA environments.

The following diagram shows the components view of the DRM layer:

What is the deal with KMS? #

KMS is important because on it relies GEM and DRM to allocate frame buffers and to configure the display. And it is important to us because almost all of the ioctls called by the GStreamer element are part of the KMS subset.

Even more, there are some voices saying that KMS is the future replacement for the frame buffer device [10].

To carry out its duties, the KMS identifies five main concepts [11,12]:

Frame buffer:

The frame buffer is just a buffer, in the video memory, that has an image encoded in it as an array of pixels. As KMS configures the ring buffer in this video memory, it holds a the information of this configuration, such as width, height, color depth, bits per pixel, pixel format, and so on.

CRTC:

Stands for Cathode Ray Tube Controller. It reads the data out of the frame buffer and generates the video mode timing. The CRTC also determines what part of the frame buffer is read; e.g., when multi-head is enabled, each CRTC scans out of a different part of the video memory; in clone mode, each CRTC scans out of the same part of the memory.Hence, from the KMS perspective, the CRTC abstraction contains the display mode information, including, resolution, depth, polarity, porch, refresh rate, etc. Also, it has the information of the buffer region to display and when to change to the next frame buffer.

Overlay planes:

Overlays are treated a little like CRTCs, but without associated modes our encoder trees hanging off of them: they can be enabled with a specific frame buffer attached at a specific location, but they don’t have to worry about mode setting, though they do need to have an associated CRTC to actually pump their pixels out [13].

Encoder:

The encoder takes the digital bitstream from the CRTC and converts it to the appropriate format across the connector to the monitor.

Connector:

The connector provides the appropriate physical plug for the monitor to connect to, such as HDMI, DVI-D, VGA, S-Video, etc…

And what about this KMSSink?

KMSSink is a first approach towards a video sink as a DRI client. For now it only works in the panda-board with a recent kernel (I guess, 3.3 would make it).

For now it only uses the custom non-tiled buffers and use an overlay plane to display them. So, it is in the to-do, add support for more hardware.

Bibliography #

Aura

2012-04-10T00:00:00Z

In the last weeks Miguel, Calvaris and myself, developed an application for the N9/N950 mobile phone and we called it Aura.Basically it uses the device’s camera (either the main one or the frontal one) for video recording, as a normal camera application, but also it exposes a set of effects that can be applied, in real time, to the video stream.

For example, here is a video using the historical effect:

Aura is inspired in the Gnome application, Cheese, and it uses many of the effects available in Gnome Video Effects.

The list of effects that were possible to port to the N9/N950 are:

dice, edge, flip, historical, hulk, mauve, noir/blanc, optical illusion, quark, radioactive, waveform, ripple, saturation, shagadelic, kung-fu, vertigo and warp.

Besides of these software effects, it is possible to add, simultaneously, another set of effects that the hardware is capable, such as sepia colors. These hardware capabilities do not impose extra processing as the software effects do.

Because of this processing cost, imposed by the non-hardware video effects, Aura has a fixed video resolution. Otherwise the performance would make the application unusable. Also, we had a missing feature: the still image capture. But, hey! there is good news: Aura is fully open source, you can checkout the code at github and we happily accept patches.

Honoring Cheese, the name of Aura is taken from a kind of Finnish blue cheese.

We hope you enjoy this application as we enjoyed developing it.

Debian's mutt with notmuch support

2012-02-26T00:00:00Z

One of my weekend tasks was to reorganize my email environment. For reading email I use mutt, configured to grab the email from an IMAP server. For sending email, I have a minimal exim setup securely relaying to a smart host.Mutt is a great email browser, but it is very bad at handling IMAP. Besides, I started to need searching through all my email. The solution for the first problem is offlineimap, a program wrote in Python, that “synchronizes emails between two repositories”. It downloads my email from the IMAP server into my laptop, so I work in my email locally, and if I delete an email locally, offlineimap will delete it in the next sync operation.

The solution for the second problem, the search, is notmuch, which is a email indexer, enabling fast searches among a vast mail collection. So, once new mail arrive (or is deleted) with offlineimap, notmuch (de)indexes it.

But another problem appear: how to query to notmuch in an integrated way with my mail reader? One solution is provided by mutt-kz, a fork of mutt with notmuch support tightly integrated.

But I use Debian, and I like its package management. So I needed to craft a Debian package for mutt-kz.

I grabbed the Debian’s repository for mutt and re-based, one by one, the patches from mutt-kz.

The result is stored in this repository.

And now, I can query notmuch in mutt and immediately browse the result set.

GStreamer video decoder for SysLink

2012-01-16T00:00:00Z

This blog post is another one of our series about SysLink (1, 2): finally I came with a usable GStreamer element for video decoding, which talks directly with the SysLink framework in the OMAP4’s kernel.As we stated before, SysLink is a set of kernel modules that enables the initialization of remote processors, in a multi-core system (which might be heterogeneous), that run their own operating systems in their own memory space, and also, SysLink, enables the communication between the host processor with the remotes ones. This software and hardware setup could be viewed as an Asymmetric Multi-Processing model.

TI provides a user-space library to access the SysLink services, but I find its implementation a bit clumsy, so I took the challenge of rewrite a part of it, in a simple and straightforward fashion, as gst-dsp does for DSP/Bridge. The result is the interface syslink.h.

Simultaneously, I wrote the utility to load and monitor the operating system into the Cortex-M3 processors for the PandaBoard. This board, such as all the OMAP4-based SoCs, has two ARM Core-M3 as remote processors. Hence, this so called daemon.c, is in charge of loading the firmware images, setting the processor in its running state, allocating the interchange memory areas, and monitoring for any error message.

In order to load the images files into the processors memory areas, it is required to parse the ELF header of the files, and that is the reason of why I decided to depend on libelf, rather than write another ELF parser. Yes, one sad dependency for the daemon. The use of libelf is isolated in elf.h.

When I was developing the daemon, for debugging purposes, I needed to trace the messages generated by the images in the remote processors. For that reason I wrote tracer.c, whose only responsibility is to read and to parse the ring buffer used by the images, in the remote processors, for logging.

Now, in OMAP4, the subsystem comprised by the two Cortex-M3 processors is called Ducati. The first processor is used only for the exchange of notification messages among the host processor and the second M3 processor, where all the multimedia processing is done.

There are at least two images for the second Cortex-M3 processor: DOMX, which is closed source and focused, as far as I know, on the OMX IL interface; and, in the other hand, DCE, which is open source, it’s developed by Rob Clark, and it provides a simple interface of buffers interchange.

My work use DCE, obviously.

But, please, let me go back one step in this component description: in order to send and receive commands between the host processor and one remote processor, SysLink uses a packet based protocol, called Remote Command Messaging, or just RCM for the friends. There are two types of interfaces of RCM, the client and the server. The client interface is used by the applications running in the host processor, and they request services to the server interface, exposed by the systems running in the remote processors, it is accepting the requests and it returns results.

The RCM client interface is in rcm.h.

Above the RCM client, sits my dce.h interface, which is in charge of control the state of the video decoders and it is also in charge of handling the buffers.

But these buffers are tricky. They are not memory areas allocated by a simple malloc, instead they are buffers allocated by a mechanism in the kernel called tiler. The purpose of this mechanism is to provide buffers with capacity of 2D operations by hardware (in other words, cheap and quick in computations). These buffers are shared along all the processing pipeline, so the copies of memory areas are not needed. Of course, in order to achieve this paradise, the video renderer must handle this type of buffers too.

In my code, the interface to the tiler mechanism is in tiler.h.

And finally, the all mighty GStreamer element for video decoding: gstsyslinkvdec.c! Following the spirit of gst-dsp, this element is intended to deal with all the available video decoders in the DCE image, although for now, the H264 decoding is the only one handled.

For now, I have only tested the decoder with fakesink, because the element pushes tiled buffers onto the source pad, and, in order to have an efficient video player, it is required a video renderer that handles this type of tiled buffers. TI is developing one, pvrvideosink, but it depends on EGL, and I would like to avoid X whenever is possible.

I have not measured either the performance of this work compared with the TI’s combo (syslink user-space library / memmgr / libdce / gst-ducati), but I suspect that my approach would be little more efficient, faster, and, at least, simpler ;)

The sad news, as in every hard paced development, all these kernel mechanisms are already deprecated: SysLink and DMM-Tiler will never be mainlined into the kernel, but their successors, rproc/rpmsg and omapdrm, have a good chance. And both have a very different approach since their predecessors. Nevertheless, SysLink is already here and it is being used widely, so this effort has an opportunity for being worthy.

My current task is to decide if I should drop the 2D buffers in the video decoders or if I should develop a video renderer for them.

devhelp and jhbuild v2

2011-12-16T00:00:00Z

A couple years ago I wrote a small tip about how to integrate the devhelp installed by your distribution with the documentation generated through jhbuild.Well, that way is plain wrong. I noticed it when I upgraded to GNOME 3: modifying the environment variable XDG_DATA_DIRS, the gnome-session goes crazy.

So, I found a less intrusive way to make your distro-based devhelp reads the jhbuild-generated documentation:

ln -s /opt/jhbuild/share/gtk-doc/ \~/.local/share/

And that is it!

OpenMAX: a rant

2011-11-29T00:00:00Z

I used to work with OpenMAX a while ago. I was exploring an approach to wrap OpenMAX components with GStreamer elements. The result was gst-goo. Its purpose was to test the OpenMAX components in complex scenarios and it was only focused for the Texas Instruments’ OpenMAX implementation for the OMAP3 processor.Some time after we started gst-goo, Felipe Contreras released gst-openmax, which had a more open development, but with a hard set of performance objectives like zero-copy. And also only two implementations were supported at that moment: Bellagio and the TI’s one mentioned before.

Recently, Sebastian Dröge has been working on a redesign of gst-openmax, called gst-omx. He explained the rational behind this new design in his talk in the GStreamer Conference 2011. If you are looking for a good summary of the problems faced when wrapping OpenMAX with GStreamer, because of their semantic impedance mismatch, you should watch his talk.

In my opinion, the key purpose of OpenMAX is to provide a common application interface to a set of different and heterogeneous multimedia components: You could take different implementations, that could offer hardware-accelerated codecs or either any other specialized ones, and build up portable multimedia applications. But this objective has failed utterly: every vendor delivers a incompatible implementation with the others available. One of the causes, as Dröge explained, is because of the specification, it is too ambiguous and open to interpretations.

From my perspective, the problem arises from the need of a library like OpenMAX. It is needed because the implementer wants to hide (or to abstract if you prefer) the control and buffer management of his codification entities. By hiding this, the implementer has the freedom to develop his own stack closely, without any kind of external review.

In order to explain the problem brought by the debauchery in the hind of OpenMAX, let me narrow the scope of the problem: I will not fall on the trap of portability among different operative systems, specially in those of non-Unix. Even more, I will only focus on the ARM architecture of the Linux kernel. Thus, I will not consider the software-based codecs, only the hardware-accelerated ones. The reason upholding these constrains is that, beside the PacketVideo’s OpenCORE, I am not aware of any other successful set of non-Unix / software-based multimedia codecs, interfaced with OpenMAX.

As new and more complex hardware appears, with its own processing units, capable of off-loading the main processor, silicon vendors must deliver also the kernel drivers to operate them. This problem is very recurrent among the ARM vendors, where the seek of added value gives the competitive advantage, and the Linux kernel has the virtues required for a fast time to market.

But these virtues have turned into a ballast: It has been observed excessive churn in the ARM architecture, duplicated code, board-specific data encoded in source files, and conflicts at kernel’s code integration. In other words, every vendor has built up their own software stack without taking care of developing common interfaces among all of them. And this has been particularly true for the hardware-accelerated multimedia components, where OpenMAX promised to the user-space developers what the kernel developers could not achieve.

First we need a clear and unified kernel interface for hardware accelerated multimedia codecs, so the user-space implementations could be straight, lean and clean. Those implementations could be OpenMAX, GStreamer, libav, or whatever we possibly want and need.

But there is hope. Recently there has happened a lot of effort bringing new abstractions and common interfaces for the ARM architecture, so in the future we could expect integrated interfaces for all these new hardware, independently of the vendor.

Though, from my perspective, if we reach this point (and we will), we will have less motivation for a library like OpenMAX, because a high level library, such as GStreamer, it would cover a lot of hardware within a single element. Hence, it is a bit pointless to invest too much in OpenMAX or its wrappers nowadays.

Of course, if you think that I made a mistake along these reasons, I would love to read your comments.

And last but not least, Igalia celebrates its 10th anniversary! Happy igalian is happy :)

How I installed the Linaro's toolchain

2011-08-03T00:00:00Z

Last week, because of a distraction, I dd’ed my external hard disk, where I used to store my cross compilation setup. And yes, I lost everything: my OpenEmbedded working directory, my installation of the CodeSourcery toolchain, and a lot more.But, as the people say, when the going gets rough, the rough get going. So I considered this mistake as a sane tabula rasa to start all over again, learning a few new tricks along the process.

My objective was to substitute the CodeSourcery toolchains for the Linaro’s one, because my main development boards are a beagle-board and a panda-board.

But I use Debian, not Ubuntu, where the Linaro’s toolchain can be deployed out-of-the-box. I had to compile it. For that I came aware of crosstool-ng, a nice utility, based on kconfig, to build toolchains, and also it supports Linaro in its last release.

To have only a toolchain is not enough. For example, my syslink wrapper depends upon libelf. A cross-compiled sysroot is also required. And for that I chose Buildroot, which is another kconfig based utility to build complete embedded systems. And it integrates with crosstool-ng too!

Sadly, the version of crosstool-ng handled currently by Buildroot is a bit older and do not include the Linaro’s toolchain. Nevertheless it was pretty much easy to modify in order to use the latest release of crosstool-ng (patch).

Now I will try to work with this environment, which, at first sight, looks comfortable and neat.

I have to add that OE is quite complete and it also includes the Linaro’s toolchain. Just I wanted to explore other options this time.

Diving into SysLink v2

2011-07-13T00:00:00Z

Following with our SysLink saga, now we will dive into its internals.As we stated before, SysLink is a software stack which implements an Inter-Processors Communication (IPC), whose the purpose is enabling the Asymmetric Multi-Processing (AMP).

Most of the readers are more familiar with the concept of the Symmetric Multi-Processing (SMP), which is the most common approach to handle multiple processors in a computer, where a single operating system controls and spreads the processing load among the available processors. Typically these processors are identical.

On the contrary, AMP is designed to deal with different kind of processors, each one running different instances of operative systems, with different architectures and interfaces. The typical approach is to have a master processor with one or more slave units, all of them sharing the same memory area, hence the operating system running in the master processor expose the other processors as other devices available in the system (figure 1).

Figure 1

The main advantage of AMP, as we mention in the previous post, is that we can integrate specialized processors and delegate them tasks that are rather expensive to execute in our general purpose processor, such as multimedia, cryptography, real-time processing, etc.

All the operating systems, executed in an AMP system, must share, at least, one key component: a mechanism of Inter-Processors Communication, because with it, they will be able to interchange data and synchronize tasks among each other.

Basically there are two types of IPC: shared memory and message passing. With the shared memory we avoid copying the data to process in other unit, but also we will demand complicated semaphoring protocols, in order to avoid overlapping, what lead to the data corruption. On the other hand, message passing is oriented to the transfer of small chunks of data, called messages, sent by one processor and received by another, and are used to notify, to control and synchronize.

Therefore, SysLink version 2.0, provides a set of components for IPC, for both shared memory and message passing (figure 2):

MultiProc: identifies and names each available processor in the configuration.
SharedRegion: handles memory areas which will be shared across the different processors.
Gate: provides local and remote context protection, preventing preemption by another thread locally and protects memory regions from remote processors.
Message: supports the variable length message passing across processors.
Notify: registers callback what will be executed when a remote event is triggered.
HeapBuf: manages fixed size buffers that can be used by multiple processors within the shared memory.
HeapMem: manages variable size buffers that can be used by multiple processors within the shared memory.
List: provides a way to create, access and manipulate double linked list in a the shared memory.
NameServer: a dictionary table within the shared memory.

Figure 2

All these operations are done through ioctl operations to the /dev/syslink_ipc device.

Nonetheless, SysLink v2.0, being a master/slave configuration, it must provide a way to control the slave processors, operations like starting and stopping, to load an operative system into them, and so on. In that regard, there is a module called the Processor Manager, which is operated also through ioctl calls to the /dev/syslink-procmgr device, and also to the specific processors, what are also mapped to the device file-system as /dev/omap-rproc[0-9], in the case of the OMAP processors.

Now, on top of all these IPC facilities, there must be a façade for all the remote operation that we could execute, and that façade is called Remote Command Messaging (RCM). Our application, which is running in the master processor, will act as client RCM and will request remote commands on the RCM server, which is running in a slave processor. DCE (Distributed Codec Engine) is an open source RCM, developed by Rob Clark, to expose distributed multimedia processing in the OMAP4.

And I continue developing a thin API for SysLink 2.0 :)

References

SysLink chronology

2011-06-08T00:00:00Z

IntroductionSince a while the processor market has observed a Moore’s law decline: the #

processing velocity cannot be duplicated each year anymore. That is why, the old and almost forgotten discipline of parallel processing have had a kind of resurrection. GPUs, DSPs, multi-cores, are names that are kicking the market recently, offering to the consumers more horse power, more multi-tasking, but not more giga-hertz per core, as it used to be.

Also an ecologist spirit has hit the chips manufacturers, embracing the green-computing concept. It fits perfectly with the Moore’s law decay: more velocity, more power consumption and more injected heat into the environment. Not good. A better solution, they say, is to have a set of specialized processors which are activated when their specific task is requested by the user: do you need extensive graphics processing? No problem, the GPU will deal with it. Do you need decode or encode high resolution multimedia? We have a DSP for you. Are you only typing in an editor? We will turn off all the other processors thus saving energy.

Even though this scenario seems quite idyllic, the hardware manufacturers have placed a lot of responsibility upon the software. The hardware is there, it is already wired into the main memory, but now all the heuristics and logic to control the data flow among them is a huge and still open problem, yet with multiple partial solutions currently available.

And that is the case of Texas Instrument, who, since his first generation of OMAP, has delivered to the embedded and mobile market a multicore chip, with a DSP and a high end ARM processor. But deliver a chip is not enough, so TI had had to provide the system software capable to squeeze those heterogeneous processors.

At the beginning, TI only delivered to its clients a mere proof of concept, with which the client could take it as a reference for his own implementation. That was the case of dsp-gateway[1], developed by Nokia as an open source project.

But as the OMAP processor capacities were increasing, TI was under more pressure from his customers to deliver a general mechanism to communicate with the embedded processors.

For that reason TI started to develop a mechanism of Inter-Processors Communication (IPC), whose approach is based on the concept that the general purpose processor (GPP**, the host processor in charge of the user interactions, can control and interact with the other processors as if they were just another devices in the system. Those devices are called slave processors.

Thus, this designed IPC mechanism runs in the host processor’s kernel space and it fulfills the following responsibilities:

It allows the exchange of messages between the host processor with the slaves.
It can map files from the host’s file system into the memory space of the slave processor.
It permits the dynamic loading of basic operating systems and programs into the slave processors.

Also, TI has developed a programming library that provides an API with which the developers could build applications that use the processing power of the slave processors.

Sadly, whilst the development in the host processor, typically Linux environments, is open and mostly free, the development for the slave cores (DSP in general) is still close and controlled by TI’s licences. Nevertheless, TI has provided, gratis but closed, binary objects which can be loaded into the slaves cores and they do multimedia processing.

Well, actually there have been some efforts to develop a GCC backend[2] for the C64x DSP, and also a LLVM backend[3], with which, least theoretically, we could write programs to be loaded and executed through these IPC mechanisms. But they are not mature enough to use them seriously.

DSPLink and DSPBridge #

In order to develop an general purpose IPC for his OMAP processors family, TI has designed the DSPBridge[4]. Oriented to multi-slave systems, agnostic to operating system, handle power management demands and other industrial weight buzzwords.

But the DSPBridge was not ready for production until the OMAP3 came out to the market. That is why, another group inside TI, narrowed the scope of the DSPBridge’s design and slimmed its implementation, bringing out DSPLink[5], capable to run in OMAP2, OMAP3 and also in the DaVinci family.

DSPLink is distributed as an isolated kernel module and a programming library, along with the closed binaries that run in the DSP side. Nevertheless, the kernel module does not meet the requirements to be mainlined into the kernel tree. Also, it lacks of power management features and a dynamic MMU support.

On the other hand, DSPBridge has been brewed to be mainlined into the kernel, though it has stuck in the Greg’s staging tree for a long time. It seems that all the resources within TI are devoted to SysLink, the next generation of IPC. Nonetheless, many recent OMAP3 based devices uses this IPC mechanism for their multimedia applications.

Initially, TI offered an OpenMAX layer on top of the DSPBridge user-space API to process multimedia in the C64x DSP, but that solution was too bloated for some developers, and the project gst-dsp[6] appeared, which reuse the codecs for the DSP available in the TI’s OpenMAX implementation, along with the DSPBridge kernel module, to provide a thin and lean interface through GStreamer framework.

SysLink and OMAP4 #

Then OMAP4 came to existence. It is not only anymore a DSP and a high end ARM processor. It has a DSP, a dual ARM Cortex-M3, and, as host processor, a dual ARM Cortex-A9. Five processing units in a single silicon! How in hell we will share information among all of them? DSPBridge was not designed for this scenario in mind.

The ARM Cortex-M3 has the purpose to process video and images, and for that reason a tiler-based memory allocation is proposed, where the memory buffers are already perceived as 2D, where fast operations of mirroring and rotation are available.

Regretfully, in the case of the Pandaboard (OMAP4330), the available DSP has lower capacities than the one in the Beagleboards OMAP3, so the published codecs, for the OMAP3 DSP, can not be reused in the pandaboard. But video codecs for the M3 cores are currently available, and they are capable to process high definition resolutions.

The answer is SysLink. Where, besides the three operation developed for DSPBridge, two more core responsibilities were added:

Zero-copy shared memory: ability to "pass** data buffers to other processors by simply providing its location in shared memory
TILER-based memory allocation: allocate 2D-buffers with mirroring and rotation options.
Remote function calls: one processor can invoke functions on a remote processor

The stack offered is similar than the OMAP3: in the user space we start with the SysLink API, then an OpenMAX layer, called now as DOMX, and finally the gst-openmax elements for the GStreamer framework. And again, a bloated, buzzworded, stack for multimedia.

In his spare time, Rob Clark developed a proof of concept to remove the DOMX/gst-openmax layers and provide a set of GStreamer elements that talk directly with the SysLink API: libdce[7]/gst-ducati[8].

Either way, I feel more comfortable with the approach proposed by Felipe Contreras in gst-dsp: a slim and simple API to SysLink and plain GStreamer elements using that API. And because of that reason, I started to code a minimalist API, copying the spirit of the dsp_bridge[9], for the SysLink interface: https://gitorious.org/vj-pandaboard/syslink/

Software architects and API designers

2011-05-26T00:00:00Z

Don’t you hate API designers when they obfuscate all the data type system?

Custom Ubuntu's root file-system for the panda-board

2011-02-07T00:00:00Z

Since I started to work with the beagle-board, one of my main concerns has been generate a custom image with the minimal setup required for the task: In my opinion it is important to have installed only the necessary and do not bloat the SD with a full featured distribution. In this way, we will control the dependencies, and we will remove possible points of failure which only will drowse our cycle of development/testing.In the case of a OMAP3 base board or any other board with inferior capabilities, the solutions were in builder systems with cross-compilation support, such as OpenEmbedded, Buildroot, etc… Those builders use the cross-compilation technique to generate custom images for the target hardware.

But in OMAP4 we are in the border line between an embedded system and a PC, and because of this, I needed a kind of a mixed approach: a native compiling environment within a custom image, in order to set it up as a builbot slave.

As I attained certain experience with OpenEmbedded I started to craft a custom native-sdk image, with rather thwarted results: after several days of patches and bug fixes, I was not able to compile, in the panda-board, the software I was willing to test.

Dodging the OE’s approach, I decided to uproot it and I started to explore new paths. Soon I stumbled with rootstock: a shell script which generates custom images of the Ubuntu distribution for the ARM architecture.

Nonetheless I tried it under Debian, since I don’t have any machine with Ubuntu. The script is difficult to trace and its logs are not useful on many occasions, mainly when the processing occurs in the qemu virtual machine. Anyway, I found that the qemu distributed by Squeeze was not functional in this particular case, so I grabbed the latest code from its repository, I compiled it and it worked like a charm.

NOTE: use the latest code of the RootStock hosted in the Launchpad bazaar repository.

At the end I came with a script where I set the parameters for my particular use case:

#!/bin/sh

export TMPDIR=$(pwd)/tmp export PATH=$PATH:/usr/sbin/:/sbin:/opt/qemu/bin

[ -d $TMPDIR ] || mkdir -p $TMPDIR

BASE="ubuntu-minimal,build-essential,openssh-server,python-twisted-core,nfs-common,portmap"
V8_DEPS="scons,libreadline-dev"
JSC_DEPS="autoconf,libtool,bison,flex,gperf,libicu-dev,libxml2-dev,libglib2.0-dev,libsoup2.4-dev"

./rootstock --fqdn panda1.local.igalia.com --login user --password 000
  --imagesize 512M --dist maverick --serial ttyO2 --seed $BASE,$V8_DEPS,$JSC_DEPS

In this way I can generate a minimal image, in a tarball, for my panda build-slave.

AAC decoder for gst-dsp

2010-12-13T00:00:00Z

One of my purposes for this year was collaborate with the gst-dsp project. But the JPEG decoder was not enough, as there are other released socketnodes by TI which are not yet wrapped in gst-dsp, such as, in this case, the AAC decoder.gst-dsp is a project with only video decoding use case in mind, and audio streams might not be optimally handled by it. The biggest concern was about the memory mapping of small buffers, which could consume more CPU rather than the direct decoding. Nevertheless I decided give it a try.

These last two weeks I devoted them to pull out the dspadec element and it seems to perform quite good without any significant modification in the gst-dsp core. You can find the submitted patches in the gst-dsp mailing list.

These patches are still in review process and perhaps they will not land on the repository, nevertheless, I also updated the marmita’s recipes in order to provide an installable image for the Beagleboard, so anybody could test this new element.

Among other candies, this marmita snapshot brings libsoup and the souphttpsrc GStreamer element, besides all the alsa stuff for the audio rendering. Also gdb hit into the tarball.

Enjoy it!

These bytes were brought to you thanks to Igalia, who sponsored this development.

btw II

2010-11-13T00:00:00Z

By the way, a group of igalians, including myself, will be tomorrow in Dublin for the Meego Conference 2010. I’ll be at the Igalia’s exhibition stand with a demo of the Pandaboard running Meego and -hopefully- rendering on-line multimedia through Grilo.

Meego in the Pandaboard

2010-11-07T00:00:00Z

The objective was rather simple: run Meego on the Pandaboard.I found that there is a wiki page for this topic, which provides a kick-start file for generating the file system, which is more than enough as a starting point.

Note: With the purpose to keep track of the Meegos’ kick-start files available for OMAP, I setup a repository in gitorious.

Nevertheless, the kernel did not work for me. Neither the boot.src provides the correct kernel parameters for the video subsystem. Hence, I borrowed what currently works: the x-loader, u-boot and kernel of the Ubuntu Maverick Meerkat for the Panda.

For the x-loader and the u-boot I just copied the binaries from the Ubuntu SD. In the boot.src I got rid of the initrd fanciness. And for the kernel, I grabbed the latest Ubuntu’s kernel from its git repository and using the config file also from the SD.

My first approach was to have GLX/EGL through the SGX kernel module, which also comes in the Ubuntu image. You can find the kernel module and the proprietary libraries in the OMAP4 PPA.

This approach failed though: The Meego’s compositor (mcompositor) could not render correctly in both monitors where I tested the result.

So, my alternative approach was get rid of the compositor and launch directly the duihome, the window manager. In order to do that, you must change the file /etc/sysconfig/uxlaunch, with

-session=/usr/bin/mcompositor +session= /usr/bin/duihome -software -show-cursor

The -software parameter request to use software rendering, not accelerated. The -show-cursor requests for a mouse pointer. Those options are consumed by meegotouch library. Nevertheless, each application must be launched explicitly with them.

In order to circumvent this issue, there is already an environment variable for the -software: M_USE_SOFTWARE_RENDERING=1. But for the -show-cursor there is none. Although it is not difficult to implement (patch).

Et voilà, we have a singing & dancing Meego environment. But still, the UI transitions are really missed.

Next step: Syslink, DOMX and GStreamer.

btw

2010-10-19T00:00:00Z

By the way, I will be at Cambridge, UK, the next Monday, I will attend the GStreamer Conference and afterwards the ELC-E 2010. I hope will see you there!

Pandaboard - Chapter One

2010-10-18T00:00:00Z

Finally, the last Friday the Pandaboard arrived to the office. Thanks to all the guy in TI who generously decided to give me one, specially to Jayabharath Goluguri and Rob Clark.

Pandaboard 1

Pandaboard 2

Pandaboard 3

Pandaboard 4

The next Saturday, Ryan Lortie and I came to the office to fool around with the new toy. Just one word: Impressive.

Ryan set up the Ubuntu Maverick Meerkat image for the Pandaboard. At the beginning we ran with a couple problems, but with the help of Ogra we solved them. First lesson learnt:

The SD for Maverick must be, at least, a 4Gb card, not less.

Second lesson learnt:

The USB is unable the power the Pandaboard. Thou shall power the board with a normal 5V power supply.

Also we found that my monitor, a LG Flatron W2261VP, is not well handled by the PVR X driver, but still it’s usable under 640x480. Ryan filed a bug about this issue.

It was great start up experience. The next stage is play with syslink and DOMX. Anyway, I won’t use Ubuntu for it, my plan is go with the minimal-fs stuff.

grilo network library

2010-10-14T00:00:00Z

Well, after almost a month of sketching out and playing with the grilo network library, finally I pushed it into the official repository.The story is this: almost all the grilo plugins fetch their media metadata from the Web. Their initial implementation used Gvfs, but this approach implied some issues: Gvfs is not well supported in non-Gnome environments, no cache handling and it doesn’t provide a throttling mechanism.

As we desire to be nice guys with the Web services providing the metadata, and also we want to play well with other players besides Gnome, so We though it could be a good idea provide an utility class for grilo plugins which can be used as a libsoup wrapper, which also would provide that functionality still doesn’t provided by libsoup yet (cache and throttling).

Nevertheless this web client functionality, semantically speaking, is not part of grilo, the plugins could depend on other HTTP client, so we set it up as a different library distributed by grilo’s core: grl-net. So whether your plugin want to used, you just have to link the library against it.

The web client uses the GAsync interface of GIO, so it is intuitive and easy to port to GI and Vala, which are already supported in upstream too (more tests are needed).

As a final note, the web client is not finished yet, the memory cache and the sync methods are work in progress, but it is already used by all the plugins provided by grilo.

GUADEC 2010

2010-08-03T00:00:00Z

Thanks to my employer, Igalia, this GUADEC was my third attendance to the conference in a row. You can see my previous impressions about the first one in Istanbul and the next in Canarias. This year the conference took place in Den Haag, Netherlands.About the GUADEC, I must state this: in my perception, Gnome is going through a crisis. A middle age crisis if you want. In 2008 Gnome was on a peak, attracting the attention of everyone, especially from the mobile industry. Everybody seemed to be a happy Gnome hacker. But then came 2009 and Nokia said farewell to GTK placing all its chips on Qt. The community seemed disenchanted and disappointed. In the air was floating a sense of disbandment.

And now that the project had lost much of the media’s attention, the community is questioning to herself where to go, which is… great! It means that the community has retaken the direction of the project, and the responsibility of its success, away from the huge corporations and the PR propaganda.

And that was the spirit of this GUADEC: hard work, serious proposals, and freedom. Yes, that sense of new freedom which can also be perceived and exposed as fear, uncertainty and doubts ;) And those feelings were more remarked because of the absence of several of the old and renown hackers of the Gnome scene, such as Miguel, Federico or Behdad.

The main trend shown almost in each talk was the importance of the World Wide Web integrated to the Gnome user experience. That was what Luis Villa talked on his keynote, and was repeated in many other presentations. Xan López and Gustavo Noronha echoed this idea making emphasis on the usage of WebKitGTK+ to achieve it. WebKitGTK+ is a reusable library, with a stable API, in difference with Mozilla, for example, which only focus on Firefox.

On the other hand, and complementary to the idea of a Web Desktop, the use of Javascript was stressed by John Palmieri: Coding for the Desktop must be just as coding for the Web. And the Gnome Shell project is the flag ship of this concept, and it continues steadily as the window manager for Gnome 3.0

Here we can observe a shift of perspective since Istanbul: in those days people were trying to bring the Web information into the Desktop (do you remember the Online Desktop?); nowadays the objective is to turn the Desktop development into the same mind frame as Web development, thus the information integration would be more seamless.

An essential ingredient for this coupling is the introspection. And good achievements were seen along this past year. Must of the Gnome core libraries are making an effort to annotate their APIs, and Python, Vala and the GJS implementation are merging their introspection work.

But what everybody is asking “What’s going on with GTK+?”. Its development seems to be at a very slow pace, with no real innovation to show. In 2008 Mark Shuttleworth suggested to integrate Qt into Gnome, replacing GTK+, and this year another voice repeated the same suggestion at the General Meeting.

In a more technical issue, Benjamin Otte exploded against the so called “Walled Gardens”. Benjamin have been trying to bring video rendering in GStreamer with Cairo, but that effort means modify many different projects, bottom-up almost along all the stack. As a result, he has found many features overlapped among the different projects and also hit against the resistance from the coders of the different projects to collaborate each other, phenomenon that he called the “Walled Gardens”.

And maybe, just maybe, this is what is happening with GTK+, is a fortified garden, where selfless programmers are patiently sealing the castle for a future exposition in the wild. I don’t know, maybe GTK+ should break its X Windows ties. I don’t know. But GTK+ is not gaining any momentum for now among the user interface programmers.

Another voices raised the question about “What is Gnome from the developer perspective?”. Andrew Savory regretted the lack of a SDK for Gnome; meanwhile Alberto Ruiz praised its absence and pushed in favor of better developer documentation and tutorials. I think that Gnome, as a developing framework, is a loose set of (almost) homogeneous libraries, utilities and guide lines, where the idea of an IDE/SDK implies a centralization of all the projects which is practically impossible. And, because of this, more and more projects are integrated into the FreeDesktop community.

Also Dave Neary and German Poo showed their results from their own analysis in the Gnome repositories. Dave tried to answer the question “Who makes GNOME?”, so he extracted gross numbers from all the current data to get the actual great iron committer; meanwhile German tried to be more lean and took in count historical aspects as the different milestones of the project, sadly German only got a lighting talk to show all his results.

Finally, other projects are still floating around, the most important is Clutter, now integrating accessibility thanks to Alejandro Piñeiro. Zeitgeist seems to me a more and more marginal project in Gnome. Telepathy, which is the de facto IM communication library for Gnome applications, and many others.

In conclusion, my over all impression of the GUADEC 2010, is that the Gnome project is in a deep insight of what is it and where is it going. I reckon that during this period of introspection (wink) we will see few innovation, but rather the new foundations of the project are putting down the required roots for the future innovation.

dsp-exec landed on dsp-tools

2010-07-06T00:00:00Z

In the DSP bridge realm, usually when the kernel module is loaded, it in turn loads the so called DSP base image, which is a file what encompass the DSP/BIOS kernel and the DSP/BIOS Bridge.Usually, in a development cycle, you may want to test different base images, and removing and reloading the Linux bridgedriver module is not very practical. For this case, TI provides the cexec.out utility, which uses the bloated libdspbridge API, to load in runtime different DSP base images.

But we all know that cool boys use dsp_bridge instead of libdspbridge, which is much more clean, small and nice. And also we have a neat set of utilities called dsp-tools. Nevertheless a utility like cexec.out was missing, and because of that dsp-exec has born.

Last week, my patches were committed by FelipeC into the stage repository in github, and I’m enjoying them while I’m poking around in the audio decoding :)

by the way

2010-06-21T00:00:00Z

I am attending GUADEC

jpeg decoder in gst-dsp

2010-06-13T00:00:00Z

Do you remember this comment? Well, I took the challenge, and it has been hard to accomplish.I started with an early implementation of FelipeC, but then I realised that I didn’t understand a bit about what it was going on.

So, I outlined a strategy with two milestones: try to clean up the TI OpenMAX IL implementation and then to add the jpeg decoder to gst-dsp.

In order to clean up the TI’s OMXIL, FelipeC recommended me to rewrite libdspbridge using dsp_bridge beneath, but I didn’t see a real gain on that task. It makes sense for a progressive TI-OMXIL update without breaking the ABI, but that was not my purpose, so I stepped further and decided to rewrite the LCML in terms of dsp_bridge instead.

The LCML is the acronym for Linux Common Multimedia Layer, and it is a shared library, loaded at run-time by the OMX components, and it provides the communication between the ARM-side application and the multimedia DSP socket node (SN). It is build upon the libdspbridge for the interaction with the DSPBridge kernel module.

My task was to rewrite the LCML, removing the libdspbridge linking dependency. You can see the result in my lcml-ng branch.

Along the rewrite process I understood the communication protocol used by the socket nodes. The clean up was really painful because the LCML code is very messy and poorly designed. And I have to say this, the Hungarian notation must be buried deep down into the oblivion.

It was my intention to keep the ABI compatibility but I rather preferred be readable, so I ended breaking it.

With a clear idea of how the LCML library works, I retook the challenge, with some degree of success: the SN was loaded up and allocated correctly, and also I found that the input port only admits one buffer, and not two as rest of the video decoders in gstdspvdec. But, when everything looked promising and the input buffer was pushed, the SN threw an critical error event and the output buffer was never received.

I had to do more than merely understand the LCML, I had must rewrite the JPEG decoder OMX component too. But this time the code was even more obfuscated than the one in the LCML.

And I had an epiphany: developing software in community implies have clean and readable code, for sake of the peer reviewing, each one with heterogeneous backgrounds. Meanwhile, under the closed and internal development approach, the QA is based on black box testing, where the cleanness of the code is not a praised virtue, but somewhat the opposite.

I rewrote all the jpeg decoder bits, from the component to its test application, but I have not pushed that branch yet into gitorious.

Finally I came across with the missing parts: each buffer pushed into the SN must have metadata, which is a structure with information about the buffer; in the case of the jpeg decoder, there were also a couple of magic numbers. The output buffer also comes with metadata, which, among other information, expresses if the buffer was decoded correctly.

Yesterday, before meeting my mates for cinema, I emailed a couple patches with the initial support for the jpeg decoding in gst-dsp.

The next task is find a strategy for the buffer number assignation on each port, so it could be defined as late as possible.

my DSP related activities

2010-05-17T00:00:00Z

When I started to play with the Beagleboard, my objective was to poke with the DSP accelerated codecs through OpenMAX and GStreamer. But soon I realized that it would be a hard task to achieve since the framework, developed by Texas Instrument, is in part proprietary (though free), and the another is open source, but it is not developed with an open source community in mind.When I started to pulled it out, the first decision I had to face was to choose a cross compiling environment. As you know, there are plenty: scratchbox, OpenEmbedded, Buildroot, PTXdist, etc. But just because people I know from TI began to write recipes for Poky, I devoted some time learning about it. Although, after a while, I jumped into OpenEmbedded. The reason were the slow updating rate against upstream which Poky had (and as I don’t follow the project anymore, I’m not aware about its current state).

But bitbake and OpenEmbedded are not the magic wand to build up a complete image to boot up in a device. There are a lot of things to define previously. Just to mention one, the distribution to build. By default OpenEmbedded offers Angstrom. But I did not want a full featured distribution, I wondered for something thin and lean, only a serial shell to start with, something I could set as a workbench for my multimedia experiments.

And for that reason marmita born.

As you may see, I mimicked the “Poky’s way”, making an overlay of OpenEmbedded, but as soon as I was getting involved in the OE’s community, I realized that maybe that it was not the correct decision, maybe I should push my changes into Angstrom instead.

Anyway, right now I have a steady set of recipes which allows me to build images for the Beagleboard, with the latest dspbridge kernel branch, and many of the TI bytes (either proprietary and open source) required for running the DSP accelerated codecs.

On the other hand, I have revamped the DSP how-to at elinux.org wiki, with the instructions to build up a kernel with DSP/BIOS Bridge support, and the means to test the communication with the DSP through it.

Along this process I became aware of the TI problems to release his DSP user space stack to the open source community. Even though the kernel side is moving quite well towards the kernel mainline, the user space bits are not doing that well, even more, they will be completely deprecated soon, because the kernel interface is still evolving.

At the lowest layer, on the ARM side of the DSP users pace stack, there is a library known as libbridge, which is basically an abstraction of the ioctl to the dspbridge kernel module, and it offers an interface with a nice semantics, but it is too much aligned to the old win16/32 API style (a bad idea in my opinion).

But the problem does not start with the API style, it begins at locating the library inside of a chaotic bunch of files, insanely bundled in a git repository, along with binaries (either for Windows and Linux), tarballs, and all sort of unrelated documentation.

Furthermore, the image building machinery within TI is a custom set of invasive makefiles, which all the projects must include and being conform with them. As result of this highly coupled build engine, extracting and isolating a project for it release is a painful and error prone process.

Given those problems, I got lost as soon as I began. So I decided to emulate the Felipe Contreras’ approach: get rid of libbridge and use his minimalist dspbridge ioctl wrapper: dsp_bridge, what he uses for gst-dsp. In order to train myself in these topics I wrote a clone of the TI’s ping application, using the dsp_bridge instead. And later on I wrote the DSP socket node counterpart. Both included in the dsp-samples repository.

Currently, the ping application is also included in the Meego’s dsp-tools repository.

Nevertheless, most of the DSP multimedia codecs are exposed, out of the box, through the TI’s OpenMAX IL implementation. And it depends on libbridge. For that reason I ripped out the libbridge from the userspace-dspbridge repository and pushed it into a different repository, cleaned it up its building machinery and removed other unneeded bytes too.

Finally, I had to do the same for the OpenMAX IL, which is not only messed up with the inside building machinery, but it is not released through a git repository yet, using instead old fashioned tarballs.

The future work will be integrate gst-openmax into marmita and try to participate with gst-dsp development. Also, FelipeC came up with the idea of rewrite libbridge in terms of dsp_bridge, task that I have been exploring lately.

using bugzilla, the right way

2010-04-30T00:00:00Z

Yes, I admit it, I hate bugzilla. I mean, the bug tracking is something indispensable for every software project, but digging in its web interface for quick searches, I just find it unbearably awful.But I guess I came across with the right tool for overcome this web madness: pybugz

On the other hand, I’ve to say that I agree about using bugzilla to keep track of patches, it’s a bad idea too. A properly organised mailing list and maybe a patchwork, if the traffic is massive, as helper.

Jhbuild beneath OpenEmbedded

2010-03-09T00:00:00Z

Since a while I’ve been working on a OpenEmbedded overlay called marmita. But this post is not about it. Where I want to aim now is about a nice trick: how to use Jhbuild using an OpenEmbedded setup.First of all, install jhbuild. And, of course, install Marmita.

In Marmita, just as in Poky, in order to get into the OE environment, the user source the script marmita-init-build-env.

Then, I setup another script, which is though to be the rcfile of a new bash session: marmita-simple-cross-compiling-env.

So, at this moment we have set all those environment variables needed to run a jhbuild session. There is also an alias for the jhbuild build, which specifies the jhbuildrc file, crafted for a cross-compiled environment: marmita.jhbuildrc.

For the moment I’ve only built GStreamer. And as a matter of fact, in the process, I came up with a simple patch for gst-plugins-bad.

By the way, the destination directory is in the /opt directory, under the stage directory; so, if you want to play with the generated output in a device, just copy that directory tree into the device’s file system.

Yes, I cannot say that I achieved a full integration between jhbuild and OE, but what I can state is that cooperation is quite possible.

apache configuration for video tag in firefox

2010-03-09T00:00:00Z

In contrast with Epiphany-WebKit, Firefox seems to be quite picky at rendering videos through the HTML5 tag <video>: it demands the correct HTTP headers.It doesn’t assume anything. If the stream doesn’t have the correct headers, Firefox just will put a black box in you face without any further explanation.

In order to overcome this issue, struggling a little with the sparse information available through the Internet, I came with this .htaccess file:

    AddType video/ogg          .ogv
    Header set Access-Control-Allow-Origin "\*"
    Header unset Etag
    FileETag None

First of all, my Apache server doesn’t recognize the video/ogg mime type.

Second, the header MUST NOT include the ETag key.

And finally, there’s new header, which isn’t fully implemented by the all browsers, but it will be, and will mess up all our pages with embedded videos, is the cross domain accessibility.

More information:

Grilo meets Vala

2010-02-12T00:00:00Z

Last week, after the internal presentation of Grilo, I got pretty excited about the project: basically, using Grilo I could make epris consume feeds from Jamendo.I started to generate the bindings for Vala. I never thought it could be that hard: the heavy use of atypical callbacks in Grilo made me find a bug in the code writer of Vala. And eventually I came with a small patch, which I’d just pushed.

Those problems brought into the discussion to use GAsyncResult within Grilo instead of the custom callbacks mechanism. We’ll see where we can go.

Finally I got my small test snippet. Cute, isn’t it?

Today also pushed another patches I’d in my Vala queue. The interesting part is, after talking with Zeeshan, I understood that the gstreamer vapi must be generated with the latest release of GStreamer. Something logical but I never stopped to think about it.

Slides of my talk at FOSDEM 2010

2010-02-08T00:00:00Z

I still don’t know how to submit my slides into the FOSDEM website, so I’m linking them here by now:

gstjpegparser

2010-01-29T00:00:00Z

Back in August 2009 I was chatting with my old peers in Mexico, and they told me that they needed a JPEG parser element in GStreamer for their DSP accelerated JPEG decoder. So, I went to bugzilla and found a bug report about the issue and a proposed patch. But the published patch still missed some features so I took it and worked on it.After attaching my first try, Arnout, the first author of the patch, came with some comments to improve the element. Several weeks after I retook the element and almost rewrote it again. So I was waiting for the OK from a GStreamer developer.

Finally, this week, Stefan review it and pushed it. Sadly for me, I didn’t notice, when I rebased my local commits, squashing my change set into one single commit, that this commit had as author Arnout, not me :(

Now the commit is under the Arnout’s credit.

Yeah, sometimes I’m so absentminded.

Ping DSP task node

2010-01-24T00:00:00Z

DSP task nodes, under the TI Omap3 domain, are a separate execution threads running on the DSP that implement control or signal processing algorithms.I’ve just pushed a rewrite for the ping dsp task node to my dsp-samples repository. It works with the dsp-ping program included in dps-tools.

An interesting thing is that it’s nearly 5 times smaller than the dll provided by TI:

-rw-r--r-- 1 1001 1001 3920 Jan 19 15:01 pingdyn_3430.dll64P -rw-r--r-- 1 root root 19816 Jan 19 14:44 pingdyn_3430.dll64P.bak

And too!

And maybe I’ll talk about this in the embedded devroom… maybe…

Minimal CD Ripper

2010-01-14T00:00:00Z

Back in London I bought a couple CDs. Obviously I don’t use CD players anymore, I mostly stream all the music I hear (jamendo, spotify, last.fm). Though, if I want to hear music using my n900 without any network connection, I ought drop in there the music files. So, the solution is rip out the music from the CDs, encode them et voilà.The obvious solution to rip music is SoundJuicer, and I started to compile it within my jhbuild environment, but I found a huge list of dependencies which I didn’t want to install, such as brasero. As everybody knows the next logical think then is “let’s code a simple cd ripper”.

Vala was my chosen language (and don’t ask why). What I wanted was have metadata in the files (life without metadata is not feasible anymore), also I want to encode the files in AAC/MPEG4, and finally I didn’t want any user interaction: just run the program and have my directory with my music.

The first problem I found was that Vala hasn’t bindings for libmusicbrainz, so I started to cook one for libmusicbrainz v2.x, which I found terrible bad to port to Vala and also it is already deprecated. Then I cooked another for libmusicbrainz3.

After that, I also found that GstTagSetter wasn’t bind to Vala either, so I made the patches for it and another for the GstIterator.

Solved all those issues, finally I came with my mcdripper!

http://gitorious.org/vjaquez-misc/mcdripper

Ah, by the way, it uses async methods, so you’ll need a recent Vala (I use the git’s version).

And finally I’ve been ripping my new CDs and storing the files in my N900.

Moving out apt metadata

2010-01-12T00:00:00Z

As some of you may know, in the N900, the root file system is stored in a OneNAND chip with 256M of space. Meanwhile /home and /home/user/MyDocs are in a eMMC in two different partitions: ~2GB (ext2) for /home and ~29GB (vfat) for /home/user/MyDocs.The OneNAND is faster than the eMMC, and it’s intended to host only the Maemo main system, moving out the third party applications to the eMMC. Though, this new layout has brought new limitations, the more visible one is the /opt problem [2].

One of the debates about what left and what not in the OneNAND is the apt database and metadata. Moving out the apt database out from the OneNAND to the eMMC, in my personal opinion, is very risky: It will slow down the database processing (which is already slow given the size of the Fremantle repositories), and if the eMMC gets corrupted, the base system wouldn’t be upgreadable either, because apt couldn’t read its database. And that’s why I’m against the proposal.

Nevertheless I’m aware that the apt metadata and database could be huge, consuming much of the precious OpenNAND storing space. Just to mention it, I’ve found myself, in my development cycles, moving out those files.

That’s why I cooked this script: move-apt-dirs.sh

WARNING: this script is not official. You’re at your own if you run it: no promises, no guaranties.

shinning new HAM

2009-12-14T00:00:00Z

A new version HAM will hit the streets soon, and we, the HAM team, are very proud of all the effort done.There have been 178 commits since the first public release in the HAM repository, all of them affording user experience and trying to cover several corner cases on the SSU realm, specially dealing with reduced disk space in the OneNAND.

New section view in HAM

There are several new features and some eye candy:

The section view has been improved greatly GtkIconView instead of the old buttons grid.
Several user interaction (work flows and dialogues appearance) optimizations.
Keep the cursor position in the package lists among operations.
Add live search support, dropping the old search dialog.
Avoid the update icon blink when the screen is blank, saving power
maemo-confirm-text can show the package name who launched it.
Minor fixes in logic strings and text display.
Speed up the HAM launching loading the back-end using a lazy strategy.
Speed up the package list processing in the back-end, so the package list are shown more quickly in the UI.

For the packagers there are also some bits:

Adapt the .install files in order to interact with the packaged catalogs.
Initial support for OVI store packages.
Add a dbus function to search packages so other applications can interact with HAM.

And for the SSU, specially handling the reduced space disk in the root file system:

Use always the eMMC for downloaded packages, avoiding the rootfs even as fallback.
Stop as much process as possible when going into the SSU (stop prestarted apps, camera-ui, browser, rtcom-messaging-ui, alarmd, etc.) in order to reduce the double mappings of large files.
Go into rescue mode if the SSU fails and change its looks to a less scary one.
Sync the disk before fetching it status, moving the operation to the back-end.
Because the documentation use a lot of disk space, we hack a way to get rid of it during the SSU.
Use the higher disk compression during the SSU

Special thanks to Lokesh, David Kedves, Mario, Marius, Gabriel, and all whom patient had helped us to make HAM a better piece of software to Fremantle users.

The SSU nightmare (2)

2009-12-01T00:00:00Z

Do you remember the third party policy nightmare [1]? Well, the SSU saga continues.As some of you may know, in the N900, the root file system is stored in a OneNAND chip with 256M of space. Meanwhile /home and /home/user/MyDocs are in a eMMC in two different partitions: ~2GB (ext2) for /home and ~29GB (vfat) for /home/user/MyDocs. This new layout has brought new limitations. The more visible one is the /opt problem [2]. But others, more subtle, have been arose, specially in HAM.

The SSU is the operation to upgrade the system in the device without reflashing, and this is the preferred way to keep it updated. The release 1.0 of Maemo 5 is already in the wild, and has been thoroughly tested and promptly fixed, thus the release 1.1 has a huge change set. And the SSU failed miserably in the early testing: the root file system ran out of space before the end of the process.

A first analysis exposed a regression in HAM (it was not closing the running applications before installing the SSU), and a problem with Gtk+ icon cache [3]: as it is a mapped file, when a package updates the icon cache, the rest of running processes kept a whole copy of the previous file. And closing the running user applications did not fix the issue, because there are many other system processes linked with the toolkit library. The Gtk guys cunningly proposed a fix using the triggers in dpkg [4].

Nevertheless, only unstable apt uses the trigger delaying strategy [5]. So we extracted the related commits and cherry picked them into Maemo’s apt [6]. I must say that I’m still crossing my fingers begging for the stability of those commits.

But the problem remained: the space in the root file system got exhausted. Other mapped files came noticed. “Kill’em all!”, they shouted, referring to the all running processes when performing the SSU. At this moment all the dirty ad-hoc hacks started get integrated into the source code of HAM (shame on me): first a list of services to stop (camera-ui, browserd, hildon-desktop, among others), and then list of processes to kill (yes, signal 9) [8].

Yet the free space got exhausted. “HAM does not calculate the free space correctly!”, they complained. There was a remarkable difference of what df shown and the amount of bytes that HAM recognized as free. We reviewed the source code of df in busybox [9], finding that it uses the f_bavail value in the statfs structure, meanwhile HAM uses f_bfree since Diablo. So in the spirit of prevention we changed it to f_bavail.

Still the required free space was more than the estimated. “Use the rescue mode as the last resource”, they demanded. So we changed added a new ad-hoc code path: if the SSU fails because of space exhaustion, reboot and get into the rescue mode. In this way is assured that none process is running. But this is not nice use a rescue mode for a normal operation. The upgrade must be as smooth as possible.

A further analysis shown that the packages install their documentation, and only a posteriori docpurge [10] deletes it. We should avoid write the documentation in the root file system. “Do a bind-mounting to another partition dumping all the documentation there”, they said, and another ad-hoc hacks were integrated. At least Marius brought some sanity and proposed a patch to dpkg to filter out some directories [11] before the got wrote in disk. Sadly, his change did not make it in the release 1.1.

At the end all these madness and hackish solutions seems to fix the SSU for now. But we shall clean out all this mess from HAM.

Embedded Linux Conference Europe 2009

2009-10-17T00:00:00Z

A couple days ago Juanjo asked me if I wanted to attend the Embedded Linux Conference at Grenoble, France, sponsored by the CE Linux forum and I usually don’t say “no” to travel. So one day after I unpacked from Barcelona, I packed again.One of the bad sides of living in A Coruña is the fly connections needed to go to anywhere. This time it was A Coruña - Madrid - Lyon (St Exupéry Airport) - Grenoble. Even though I found entertained and challenging all these heterogeneous connections.

In the registration area surprisingly I came across with Rob Taylor and Mark Doffman, from Codethink. Later on I saw also David Neary and Christian Schaller around.

The opening key talk was in charge of Jon Masters, and was about “Porting Linux” to other hardware platforms, where he just highlight some hints and landmarks in the porting process, such as run the vendor tests in the board, port the u-boot, read the hardware erratas (which are more insightful than the rest of the documentation); read the linux kernel mailing lists; use the flattened device tree to specify the device’s addresses (there’s no BIOS neither ACPI in embedded system); gave some recommendations using cscope; and finally insisted in not to take the x86 architecture implementation as references, because is too dirty, he rather recommended use the score arch (“x86 wouldn’t be designed today”). At the end he strongly advised about reading the development-process under the kernel documentation.

The next talk I attended was “Use of the Fast IRQ (FIQ) in ARM-Linux” by Allesandro Rubini. The FIQ is an interrupt request with a higher priority and can be handled while another IRQ. Allesandro use them as near-real-time signal processing under a non-real-time linux kernel. In this talk he showed his project, which hasn’t been accepted in mainstream, because is too task specific.

Then I stayed in the talk “Constrained Power Management” by Patrick Bellasi where he described multiple-policy power management optimization. Patrick distinguish two PM policies: device specific and system-wide. So he propose a coordinator entity among the local policies, and this coordinator uses some kind of linear programming optimization. Cool research stuff.

My objective for the conference was twofold: to learn about embedded linux distributions build systems, and to come aware of the community opinions about Android. So I’d the intention to attend the Matt Porter’s talk, nevertheless Masters recommended the PTXdist one, and I bought it: failed! The Porter’s talk won the “best talk” conference’s award.

PTXdist is just another embedable-linux-distribution builder, based on Kconfig to “cook” your distro and in makefiles (GAR style) to build the packages and express their dependencies. An interesting feature is the ltmain.sh fixup to avoid path hard-coding in libraries lookups.

Afterward I assisted to Nina Wilner’s talk about “Porting Android to Power Architecture”. She’s technology chief at IBM for the revamped PowerPC platform. Sadly she just make advertising for Mentor Graphics, the guys who employ Matt Porter and actually done the PowerPC port of Linux to Android. A couple interesting insights is that even though Google insist that Android is multi-platform it’s full of ARMisms, and porting its Bionic C Library to other architectures painfully. Nevertheless she stated that Android could make, in the embedded stage, what Linux made for Unix in the servers: do all over again and do it right.

Then I backed to the hardcore topics with “Analyzing Kernel Function Execution with ftrace” by Tom Bird. He’s also one the organizers of the conference. ftrace is the first generic tracing system to get mainlined. In this talk Tom explained his adventures to measure the duration of each kernel’s function at boot. With ftrace you can execute logging functions at every filtered function call. Meanwhile the community is expecting the port of dtrace to Linux.

In the afternoon Marcin Juszkiewicz talked about “Hacking with OpenEmbedded”. It was about different use cases for OpenEmbedded beside the embedded distro building.

Finally, the BoFs of the conference. I stayed at first in the Buildroot, another embedded-distro-building which is too similar to PTXdist. At the middle of it I ran away to the Android BoF which I catch at the conclusions. Again, Android doesn’t bring nothing new, nothing breakthrough, but a lot of questions and doubts, as its ties with Google, its GPL licensing avoidance efforts, and so on.

And the Small Business BoF, organized by Michael Opdenacker, the owner of the company Free Electrons. Most of the companies there were quite small, several just one-person companies. The hottest topics there were about the lawyers need, the time tracking and billing, the hiring process and local vs abroad clients.

At the night we’d the social event: a cocktail buffet at Restaurant du Téléphérique, with a great panoramic view of the city.

At the next and last day, the opening keynote was in charge of Philippe Gerum: “State of real-time Linux: Don’t stop until history follows”. I arrive late to the talk but as far as I understood, he talked about the advances in the Linux kernel in the Real Time arena. He put special attention to the project Xenomai, which bring hard real-time support in the GNU/Linux user-space.

I flipped a coin and decided to go to “The LLVM MIPS and ARM backends” but I was gladly surprised with the talks swapping to “Coccinelle: A program matching and transformation tool” by Nicolas Palix. I came aware of the powerful concept of semantic patch. The principal use case for it, is to detect the collateral effects of an API breakage, also to fix common programming mistakes. It has been used in the kernel finding tons of potential unseen bugs. Neat!

After a short coffee break I ran into the “Survey of Linux measurement and diagnostics tools” by Frank Rowand. He shown a quick survey of several software tools for performance measurement such as smem (quite interesting); oprofile (quote: “sampling tools are not that good” also is turning obsolete); systemtap (measure the processor use in a non-intrusive way); time (the old & honest); perf (here Frank talked about the interesting problem of cache misses); bootchart; the kernel function trace; the LLTng, Timechart, an finally the famous ftrace.

Meanwhile the lunch time was getting arranged, a showcase took place in a room, where a couple projects were shown. The most impressive IMHO was the Nsec Linux system boot by Montavista. There was also Codethink with a all-microbloggers-on-one application; an LZO compression algorithm for the kernel; a demo of Android in PowerPC, etc.

In the second part of the second day, Gilad Ben-Yossef, from Codefidence, talked about “The good, the bad and ugly: on threads, processes and co-processes”, where he probed that in the Linux realm, threads and processes have no performance differences. In theory the big difference between a process and a thread is that the first owns a memory space and the second share it, so the context switching among process should be more expensive that in threads. So he decided to do some measurements with a modified version of lat_ctx, and surprise! no appreciable differences were shown. Thus, his conclusion, is that the reason why people prefer use threads instead of process is because the API. The mental model offered by the threads is more appeal to the common programmer than the process model, so Gilad decided to create a wrapper library to implement process API in terms of the threads API called coproc.

“A computer is a state machine. Threads are for people who can’t program state machines” – Alan Cox

The last talk I assisted was “Technical features and components of open source build systems” by Alex de Vries from Wind River Systems. It tried to bring a survey of all the requirements of a “perfect” embedded-distro-builder and then a pseudo BoF was raised with the question why we don’t have already one? There are tons of these system on the wild, but every has a very specific and reduced set of features. There was no conclusions but at least the seed was planted.

Finally, the closing keynote was carried on by Tom Bird and Klaas van Gend, in a mixed talk about the purpose of the CE Linux forums its goals and achievements with a contest of the butterfly effect. It was fun.

The conference closed with a round of drinks. I got a nice Chinese beer.

GIT quick notes

2009-10-06T00:00:00Z

I want to work on a new plug-in inside of gst-plugins-bad. But also I want to expose my development to a personal GIT repository, so my friends could fetch my changes.As Linus Torvalds explained once keeping a nice linear regression set of patches above an upstream development implies the use of git-rebase, nevertheless that also implies that I will have to force my pushes and my friends will also have trouble keeping their repositories in sync with mine.

So this is a kind of compromise between be nice with the people who pull your changes, or ease your daily work.

libgoo & gst-goo

2009-09-19T00:00:00Z

Back in 2007 I started to work integrating OpenMAX IL components into the GStreamer platform.OpenMAX is a set of programming interfaces, in C language, for portable multimedia processing. Specifically the Integration Layer (IL) defines the interface to communicate with multimedia codecs implemented by hardware or software.

Texas Instrument started to work on an implementation of the OpenMAX IL for their DSP accelerated codecs for OMAP platform.

A quick and rough view of the software architecture implemented to achieve this processing is more or less exposed in the next diagram:

    +---------------------+
    | OpenMAX IL          |
    +---------------------+
    | libdspbridge        |
    +---------------------+
    | Kernel (DSP Bridge) |
    +---------------------+

The DSP Bridge driver is a Linux Kernel device driver designed to supply a direct link between the GPP program and the assigned DSP node. Basically the features offered by the driver are:

Messaging: Ability to exchange fixed size control messages with DSP
Dynamic memory management: Ability to dynamically map files to DSP address space
Dynamic loading: Ability to dynamically load new nodes on DSP at run time
Power Management: Static and dynamic power management for DSP

The libdspbridge is part of the user-space utilities of the DSP bridge, which purpose is to provide a simple programming interface to the GPP programs for the driver services.

In the DSP side, using the C/C++ compiler for the C64x+ and the libraries contained in the user-space utilities, it is possible to compile a DSP program and package it as a DSP node, ready to be controlled by the DSP bridge driver. But right now TI provides a set of out-of-the-box DSP multimedia codecs for non-commercial purposes. These nodes are contained in the tiopenmax package.

So, as I said before, my job was to wrap up the OpenMAX IL components delivered by TI as a GStreamer plug-in. In that way a lot of available multimedia consumers could use the hardware accelerated codecs. But also, our team did the test of the delivered OpenMAX components.

After trying several approaches we came to the conclusion that we need a new layer of software which will provide us

Facilitate a great testing coverage of the components without the burden of the upper framework (GStreamer in this case).
Improve the code reuse.
Use an object oriented programming through GObject.
Facilitate the bug’s workaround for each component and maintenance of those workarounds.
A playground for experimenting with features such as (OpenMAX specific) tunneling and the (TI specific) DSP Audio Software Framework (DASF).

For those reasons we started to develop an intermediate layer called GOO (GObject OpenMAX).

    +---------------------+
    | GStreamer / gst-goo |
    +---------------------+
    | libgoo              |
    +---------------------+
    | OpenMAX             |
    +---------------------+

libgoo is a C language library that wraps OpenMAX using GObject. The follow diagram shows part of its class hierarchy.

                               +--------------+
                               | GooComponent |
                               +--------------+
                                       |
    +---------------+ +---------------+ +---------------+ +---------------+
    | GooTiAudioEnc | | GooTiAudioDec | | GooTiVideoDec | | GooTiVideoEnc |
    +---------------+ +---------------+ +---------------+ +---------------+
            |                 |                 |                 |
     +-------------+   +-------------+  +---------------+ +---------------+
     | GooTiAACEnc |   | GooTiAACDec |  | GooTiMpeg4Dec | | GooTiMpeg4Enc |
     +-------------+   +-------------+  +---------------+ +---------------+

At the top there is GooComponent which represents any OpenMAX component. If the OMX IL implementation is neat and clean, there shouldn’t need to add subclasses underneath it, just parameterize it, and should be ready to use as any other OMX IL component. But reality, as usual, is quite different: Every implementation is different from each other; and to make it worst, each component in a same implementation might behave differently, and that was the case of the TI implementation.

Finally, over libgoo there is gst-goo, the set of GStreamer elements which use the libgoo components. GstGoo also sketched some proof of concepts such as ghost buffers (to be used with the OpenMAX interop profile), and dasfsink and dasfsrc (TI specific).

In those days, before I move to the GStreamer team, an old fellow, Felipe Contreras, worked on gomx, which is the precedent of libgoo, before he got an opportunity in Nokia and started to code on GstOpenMAX. An interesting issue at this point is that FelipeC is pushing boldly for a new set of GStreamer elements which ditched OpenMAX and talks directly to the kernel’s DSP bridge: gst-dsp.

What’s the future of libgoo and GstGoo? I couldn’t say. Since I moved to Igalia, I left its development. I’ve heard about a couple companies showed some kind of interest on it, sadly, the current developers are very constrained by the TI workload.

misc

2009-08-23T00:00:00Z

Do you remember that I promised not to use a minimalist window manager? Well, sorry, another broken promise. Since I started to play around with imapfilter, I discovered lua. Moreover, a comment in a previous post made a mention of awesome, a minimalist window manager configured through lua. So, I installed it, play with it, and suddenly I got delighted with all of its features: Awesome comes along pretty well with Gnome and its panel, which I didn’t want to lose at all. Besides, Awesome provides its own panel (called widget box, a.k.a. wibox), which includes a systray (sadly, awesome systray steals the icons from the gnome-panel systray). I’ve found that a tidy desktop, which avoids to the user unnecessary mouse interactions, is much more relaxed and helps the user to focus on her task. We’ll see how this experiment ends.Meanwhile, Joaquin, a colleague from Dextra, told about they were having troubles with the gst-openmax JPEG decoder element, because it needed a JPEG parser, while the gst-goo one mimic the official JPEG decoder provided by GStreamer in gst-plugins-good. In other words, the last two elements actually parse the buffer and validates the presences of a complete image in a single buffer, while the first doesn’t, it just assumes it, relying thou on a parser after the data source, which will deliver the image meta-data through fixed capabilities in the parser’s source pad.

Loathed by HAM and all the release processes, I though it could be nice to wet my feet again in the GStreamer waters. Besides, I need to help Iago with his codec base classes, so this JPEG parser, would help me to ramp up.

As an historical note, the first element I took in charge when I get in the GStreamer development group in Dextra was, precisely, the JPEG decoder.

As soon as I chatted with Joaquin, I found a bug report about an element for that purpose, but it still missed a couple features to turn it useful for our needs. And start to hack it. First, I moved from gst-plugins-good, to gst-plugins-bad, and then parse the image header in order to find its properties such as width, height, if the image is progressive, color format, etc. And the set these data in a fixed capability. Also, the frame-rate is negotiated in the sink pad of the parser, such as in the official JPEG decoder.

Finally I got something which seems to work OK and posted it in the same bugzilla report. I hope to receive feedback soon.

On the other hand, I’m still waiting for the approval of my last patches to the GStreamer’s Vala bindings (592346, 592345 and 591979). 591979 can be particularly controversial, given that it changes a typical class method, into a static function. I guess I need to ping somebody.

On the Bacon Video Widget port to Vala, some advances had came, but still there’s nothing to show yet.

the true imap usage: imapfilter

2009-08-09T00:00:00Z

As I said before, I have been using mutt as my e-mail client, and I’m really getting into it. It does what it is supposed to do, cleanly and fast.Nevertheless its support to IMAP lacks of several nice features. One I miss the most is to notice the unread messages in all of my IMAP folders. Yes, what it does is to notices the new e-mails, but if you ignore the announcement, Mutt will not remind you those not-yet-read messages later on.

And that was my biggest fear: Have I missed an important email?

Then I found imapfilter, the IMAP Swiss Army Knife. Using the LUA programming language, you would manipulate your IMAP resources as you want: you can do complex searches, apply operations on messages (move, copy, delete), put flags on them, etc.

So, my task was obtain the list of unread messages in my IMAP account. After a few of LUA exploration, I managed to get this script:

server = IMAP {
    server = 'mail.server.com',
    username = 'vjaquez',
    password = 'myubberpassword',
    ssl = 'tls1',
}

mailboxes, folders = server:list\_all ('\*')
for i,m in pairs (mailboxes) do
   -- server\[m\]:check\_status ()
   messages = server\[m\]:is\_unseen () -- + server\[m\]:is\_new ()
   subjects = server\[m\]:fetch\_fields ({ 'subject' }, messages)
   if subjects ~= nil then
      print (m)
      for j,s in pairs (subjects) do
         print (string.format ("t%s", string.sub (s, 0, s:len () - 1)))
      end
   end
end

It prints out nicely all the messages I haven’t read, so I can go to those IMAP folders, and read them.

Sadly, there’s no way to integrate (as far as I can see) the imapfilter results with Mutt. But that would be amazing! Imagine an IMAP email client manipulated by commands like these.

And those feelings made me recall the Philip’s ideas about integrating Tinymail with Tracker.

And I wonder, does imapfilter support pipelining?

The SSU nightmare

2009-07-27T00:00:00Z

In the Maemo domain, there’s a concept called SSU, Seamless Software Updates, which is the mean to update the whole operative system without reflashing the device.This concept is relatively new in the mobile device domain, where reflashing has been the common upgrade path. But this idea is as old as GNU/Linux distributions.

Maemo uses the Debian format packages and dpkg /APT for their handling.

Meanwhile in Debian we are used to the dist-upgrade command in APT to upgrade to the new OS release, in Maemo is not the case because each OS release has a specific hardware target, so there is no need to handle complex dependency handling.

And that is the historical reason that the Hildon Application Manager does not use the APT algorithms for dependency handling. Instead of that, it use custom chicken-like algorithms: if the dependencies are not fulfilled without any foreseen problem, the package installation, upgrade or removal is rejected.

The Maemo OS upgrades are done through a meta-packages, which is merely a list of package dependencies to conform the new release. And this is all the magic in the famous SSU.

But this approach posses a couple drawbacks, which, in Fremantle, had grown disproportionately given the dimensions of the project.

First, the number of packages which conform a new release is so big, that the package section in the list file is bigger than the buffer size allocated by APT to parse it. We already filed a bug in Debian about this matter. It seems the problem is an integer overflow.

Second. It is quite easy to break the SSU process: if you install a third party application with a hard dependency to an OS package, the meta-package of the next release will fail, given the chicken-like nature of H-A-M when solving dependencies problems.

The adopted solution is impose a dependency policy for third party packages, which current implementation had triggered a community discussion.

The other proposed solution, use the tiger-like algorithms available in APT, was discarded given the risk of the needed changes in the apt-worker versus the available time frame.

No, there is not conclusion yet about the third party policy issue.

lit

2009-07-19T00:00:00Z

Given the Igalia’s OLPI program (One Laptop Per Igalian) I got a new Thinkpad, the x301. I chose this one just because Alex chose it too, and I trust in the Alex judgement, and I don’t regret it: it’s a great machine.All my computers had receive a name. The first computer I bought with my own resources was named Rowena. I still like that name. My first laptop was named Angelina. Meanwhile my third desktop computer was named Toaster.

So, in order to not break the tradition, I named the new laptop, and its name is lit :)

lit as noun is the contraction of literature. As adjective is the contraction of illuminated or lighted. Also, in French means bed.

I took a different approach to work with this laptop. First of all, I installed Debian Squeeze on it. Not Ubuntu, nor Gentoo. The reason of this is follow the standard Igalia’s software stack.

The second difference is an reinforce in my console orientation :) Instead of using Evolution, I’m trying Mutt; instead of XChat, IRSSI; right now I’m testing Newbeuter as feed reader, all of them in a single terminal using GNU/Screen. Besides, I’m continuing with my Emacs dependency curve.

And, if you’re wondering, I’m not going to use RatPoison or any other minimalist window manager, and maybe never will.

Here’s a screenshot of my current desktop:

GCDS ramblings

2009-07-13T00:00:00Z

The last two years the GUADEC has been pushing the geographical limits of Europe: in 2008 in Istanbul, more near to Asia, and now, in 2009, in Canarias, more near to Africa. And that’s OK, I like it actually, is better than repeating the old hosts of the conference.Comparing my impressions with that two last GUADECs, I found something interesting: In Istanbul the new ideas were boiling the environment, bold proposals: the seminal concepts for Gnome Shell, Zeitgeist, Gnome 3.0, and so on. And now, in Gran Canaria, the spirit was an evaluation the development of those ideas, a revision of where the project is and what’s missing to achieve the goals.

In Istanbul, the hackers takeoff. In Gran Canaria, the hackers landed.

Nokia stepped back from GTK+ in their products and announced their official support to its recently acquired Qt framework. And that announcement staggered the gnome mobile gang, forcing to rethink their objectives.

The online desktop died and from its rotten corpse a new flower appeared: Gnome Shell: The desktop is a canvas driven by javascript applets, just like the Web 2.0. Moblin also adopted that idea, and many other are going in that path.

The Federico’s crazy idea about a chronological desktop evolved into Zeitgeist, which is taking over as the Tracker user interface for Gnome.

The furor unleashed in Istanbul with the announce of the efforts for Gtk±3.0 had been diminished in this time. Nevertheless, Garnacho showed in the GUADEC-ES some of his unfinished branches, some of them quite amazing, but unmerged into mainstream yet.

Almost of the attention was hogged by Clutter, and the Moblin guys are using this to push the rest of their framework. I guess that everybody is waiting a miraculous merge among Gtk+ and Clutter, but that’s really hard to happen.

Vincent is still pushing for Gnome 3.0. And I really wish for that to happen. And Gnome 3 means GnomeShell, Zeitgeist, and possible Gtk+ 3.0 and GStreamer 1.0. Those new versions doesn’t mean necessary new features, but more clean ups in the code.

What excited me the most was the discussion towards GStreamer 1.0. There’s plenty of work to do. Hopefully I’d find a spot to collaborate on it.

Also I went the ConMan talk and a couple about WebKit, DBus, and a couple more.

10 blast moments at GCDS

2009-07-13T00:00:00Z

10 blast moments at GCDS1. When Edu played Gimme the power in the Gnome party

When we rented a car to pass an afternoon in Maspalomas
When Owen Taylor showed a picture with me and others as if we were the authors of GnomeShell :)
The Robert Lefkowitz keynote
That actually my talk in GUADEC-ES had public!
The girls taking the sun in the beaches
The hacking sessions at hotel’s lobby
Met Jürg Billeter at Collabora’s party
Gossip with Marius
The weather… (it didn’t rain!)

Gran Canaria Desktop Summit

2009-07-12T00:00:00Z

My biggest surprise in the GCDS was in the Owen Taylor’s talk about Gnome shell: he attributed the authoring of Gnome shell idea to Kimmo, Rodrigo, Miguel and myself! and I had no idea! Sadly, I looked in the gnome-shell git repository and I didn’t find any reference to myself. So, no, I’m not an hypnobate coder.

My GUADEC-ES talk

2009-07-10T00:00:00Z

Last Wednesday I talked in the GUADEC-ES about Vala. A slight introduction to the language. Here are the slides.

Hildon Application Manager goes public

2009-05-25T00:00:00Z

Almost since my arrival to Igalia I started to work in the Hildon Application Manager or H-A-M for friends. The project was developed using SVN in garage, but after the GUADEC in Turkey, we move it to GIT, within an internal server.But after an unexpected and bold movement of Marius Vollmer, the latest development version of H-A-M, for Fremantle more specifically, was pushed into the wild. Thanks mvo!

H-A-M was finally sync with a repository in Gitorious, making available its source code for everyone. So, everyone is invited to submit your patches! :D

The big patch theory

2009-04-09T00:00:00Z

It is well known that the free/open source software major asset is the sense of community surrounding each project. That idea is the constant in almost all the promotional videos which participate in the Linux Foundation contest We’re Linux.A community driven project, exposed to the masses by Eric Raymond in his paper The Cathedral and the Bazaar, has been gaining moment, turning the software development, rather than a isolated, almost monastic, craft work, into a collaborative, openly discussed, engineering task.

All those ideas sound exciting and appealing, but from the trenches, how is it done? How does a bunch of hackers collaborate all together to construct a coherent and useful piece of software?

We could be tempted to think about complex, full solution, software such as forges, distributed revision control software, etc. But I won’t go there. That’s buzzword. From my point of view, the essence of any collaborative software project is the patch****.

According the software engineering, an important activity, in the software development process, is the peer reviewing. Even ESR gave a name to it: the Linus’s Law. But the software peer review is not an task done once or periodically, it would be an almost impossible duty, or at least impractical, because it’d imply read and analyse thousands of lines of code. Instead of this, the peer review it’s a continuum, it’s a process that must be carried on during all the code writing.

So far we stated two purposes for a patch file: collaborative software construction and peer review. For the first purpose the activities implied are, from the developer point of view, modifying the code, extract the diff file and hand it out; from the integrator point of view, the application of the patch and keep a register of it. It’s relatively simple process, and it can be automatised in some degree with the help of a SCM software. But there’s hidden trick: the patch must be small and specific enough to survive several changes in the base code, and could be applied cleanly even if the surrounding code had changed since it was created.

Nevertheless, for the second purpose, the process from the developer point of view, is more complex; the crafting of a patch for peer review is not a trivial task, quite the opposite, good patches are fruit of experience.

A good patch has set of subjective and objectives values that the reviewers and the integrator will take in count at the moment to commit it in the official repository. A patch must be small enough to be easily understandable, attack a single issue, self-contained (in must have all it needs to solve an issue), well documented, complete, honour the project code-style, and so on.

A perfect patch must transmit the needed confidence to the maintainer, with a simple glance, to apply it immediately. And only the experience can give this skills. There’s not a Patch 101 lecture, but maybe is a interesting idea.

fosdem'09

2009-02-12T00:00:00Z

Last weekend some igalians and I went to Brussels to attend the FOSDEM’09. We arrived the Friday just in time for the FOSDEM Beer Event (amazing coincidence!). In the Delirium Café I got my first epiphany about the True Spirit of FOSDEM: it is not the talks and the meetings, it is the beer. So, after that night, I could say “mission accomplished”.Nevertheless I went further and also assisted to a couple talks:

The People Framework: it is about having a unified method to access to “contacts” backends (google, ldap, etc.) Go Vala!
The Hynerian Empire: It was about Rygel, a UPnP media server. Go Vala! * Bringing geolocation into GNOME: I slept this one.
Tracker: Philip tried to expose tracker as the ultimate object locator.
Xfce 4.6 and then?: It was a “What’s new” in Xfce.
Maemo on BeagleBoard: Nokia employees says that their software also runs in other hardware.
WebKit on ebook readers: A WebKit implementation for a specific embedded device.
Ext4: What is and what is new in Ext4: a featured Ext3.

But now, when all the craziness of Brussels had gone, reviewing the whole schedule, I realized that I should went to other talks. These are my actual chooses… too late…

Wt, a C++ web toolkit, for rich web interfaces to embedded systems
Reverse Engineering of Proprietary Protocols, Tools and Techniques
Building Embedded Linux Systems with PTXdist
A talk on FLOSSMetrics
CMake - what can it do for your project
Syslinux and the dynamic x86 boot process
Emdebian 1.0 release - small & super small Debian
Mozilla Headless back-end

By the way, I just loved Brussels.

password management

2009-02-01T00:00:00Z

Be on-line means have user accounts in a lot of services, such as email, social web sites, blogs, etc… And have multiple user accounts implies have to remember user names and passwords.I know people that only have one user name and password and repeat the same for all their accounts. This approach may simplify the need to memorise a lot of different pair of words. But this method is not reliable at all: 1) if your password is compromised, you’ll have to change it in all your accounts, and 2) you may forget an account if you don’t keep track of all the accounts you sign up.

Another big issue is that most of the people have passwords easy to remember, and those passwords usually easy to crack.

In other words, we have two problems: 1) keep track of all our user accounts (resource, user name and password), and 2) the password must be not guessable.

For the second problem you may follow hints and craft each one. But I am lazy guy and computers execute algorithms better than me. So I installed an automated password generator (apg) and let the program offer me a set of possible passwords, choosing the most appealing one.

$ apg -M NCL -c cl\_seed -t -x 8
Awnowov6 (Awn-ow-ov-SIX)
Biuj7qua (Bi-uj-SEVEN-qua)
RyecGod9 (Ryec-God-NINE)
Ojonrag1 (Oj-on-rag-ONE)
9KnecOng (NINE-Knec-Ong)
ClagHog0 (Clag-Hog-ZERO)

Neat, don’t you think?

Now, the straightforward solution for the first problem is write down, in a plain text file, the list of resources, user names and password, of every user account you have. This file can be consulted if you don’t remember the account data.

As a personal choice, I use, in Emacs, the org mode to organise the user data, because its table editor is just beautiful. Furthermore, I have several types of outlined user accounts (web sites, email servers, WEP keys, etc.), what it is also handled by org mode.

* web
| site                   | user         | password   |
|------------------------+--------------+------------|
| https://sitio.guay.com | mi_nick_guay | Ojonrag1   |
| ...

* email servers
| server         | user          | password         |
|----------------+---------------+------------------|
| mi_empresa.com | mi_nick_serio | ClagHog0         |
| ...

* wep
| essid          | password  |
|----------------+-----------|
| essid_del_piso | Awnowov6  |
| ...

But now we have a problem: have a plain text file with all your passwords is more insecure than just have one shared among all your user accounts. If somebody gain access to this file, (s)he will own you.

The solution can’t be more simpler: encrypt the file! Well, yes, you’ll have to remember one password, but only one! In order to encrypt GPG is the way to go. GPG not only support asymmetric encryption, but also symmetric, which may be handy if you don’t like or you are not used to use private/public keys. Nevertheless, is worthy learn how to interact with the asymmetric encryption.

Well, if you use Emacs, you have the EasyPG mode, which will easy the GPG interaction, avoiding you to run the gpg command each time you want to read or save your file, the mode will detect if the file is encrypted and it will ask you for the pass phrase to decrypt it transparently for the user.

Once you have encrypted your password file, you can put it in your home page for backup and roaming purposes.

Neat, don’t you think?

This post is heavily inspired in Keeping your secrets secret.

no country for boy's compilers

2008-11-03T00:00:00Z

During the first revision by the mythical committee of ALGOL, called ALGOL60, Donald Knuth proposed a mean for evaluating implementations of the language: the man or boy test.This test involves a heavy use of closures and evaluating functions as first-class citizen in the programming language. If your programming language, either its design and implementation, can solve the test, you’re working on a mature compiler.

Coroutines, closures and continuations

2008-10-08T00:00:00Z

I ought start with a disclaimer: I don’t have experience with functional programming. I never studied it. Nevertheless recently I have had some contact with some of its concepts, specially with the idea of continuations and closures. From these contacts came up a question, an hypothesis: by definition, continuations ⊆ closures ⊆ coroutines?.The first step should be find their formal definitions and establish their relationship as subsets. But find and understand formal definitions is not a piece of cake, at least for me, so I will try to expose the most clear definitions, in simple words, which I found in the net:

coroutine #

A coroutine is a generalization of a subroutine which can be paused at some point and returns a value. When the coroutine is called again, it is resumed right from the point it was paused and its environment remains intact.

In other words, a coroutine is a subroutine which has multiple entry points and multiple return points. The next entry point is where the last return point occurred or at the beginning of the coroutine if it reached its end previously. Meanwhile a subroutine only has one entry point and one or more return points.

Example:

In this example, the pause and return point is controlled by the token yield.

coroutine foo {
  yield 1;
  yield 2;
  yield 3;
}

print foo (), "n";
print foo (), "n";
print foo (), "n";
print foo (), "n";
print foo (), "n";
print foo (), "n";

This code will print:

closure #

A closure is a block of code (can be anonymous or a named subroutine) which can contains variables defined out of its lexical scope (free or bound variables). When that block of code is evaluated, all the external variables and arguments are used to resolve its free variables.

Example:

function foo (mylist) {
  threshold = 150
  return mylist.select ({ myitem, myitem.value > threshold })
}

In this example, the closure is the block code of { e, e.salary > threshold } which is evaluated for every item in the list (the select method traverse the list executing the passed closure for each element), and the free variable of threshold is resolved in the lexical scope of the foo subroutine.

continuation #

A continuation is a subroutine which represents the rest of the program.

In order to visualize this, you must break with the concept of the return of a subroutine, the subroutines don’t return values, they continue the execution of the program with another subroutine. The continuation, obviously, must receive the whole state of the program.

In other words, a continuation is a glorified “goto” sentence, where the execution pointer is transferred from a subroutine to another subroutine. The concept of the function stack and their dependency relationships are removed.

function foo (value, continuation) {
  continuation (value + 1)
}

In the previous example, the continuation is passed to the foo subroutine as an argument, an contains, besides the next subroutine to execute, the whole state of the program, and receives the an argument which is processed by foo.

conclusion #

Each concept, although extends or modifies the wide-used concept of a lexical scoped subroutine, is independent among them, but not mutually exclusive: in terms of implementation you may have a coroutine which is a closure, a continuation which is a closure, a coroutine which is a continuation or vice versa.

So, my hypothesis is incorrect, we must rephrase it as, by definition coroutine ∩ closure ∩ continuation = ∅

But, by implementation, ∃ coroutine ∪ closure ∪ continuation

references #

what the heck is: a coroutine http://www.sidhe.org/~dan/blog/archives/000178.html
Wikipedia Coroutine http://en.wikipedia.org/wiki/Coroutines
Closure http://martinfowler.com/bliki/Closure.html
A Definition of Closures http://gafter.blogspot.com/2007/01/definition-of-closures.html
Wikipedia Closure http://en.wikipedia.org/wiki/Closure_(computer_science)
Fake threads http://mail.python.org/pipermail/python-dev/1999-July/000467.html
Continuations made simple and illustrated http://www.ps.uni-sb.de/~duchier/python/continuations.html
Wikipedia Continuations http://en.wikipedia.org/wiki/Continuations

async gio test

2008-09-11T00:00:00Z

In order to understand how to use the asynchronous API of gio, I have cooked a small test. It was a little hard to figure out the use of the API, mostly because I could not find implementations using Google’s codesearch, neither doing grand-greps ™ on my jhbuild’s directory of checkouts.I hope this little test be useful for those who are trying to use gio in their async operations.

Programming tips: Google's codesearch

2008-08-26T00:00:00Z

Let’s face it: every programmer loves the “copy&paste” when he is prototyping a solution, learning how to use a new library or just needs a quick fix, and why not?I mean, yes, it is important understand what we are doing, the implications of every single line of code, keep the elegance, coherency and the simplicity, but those characteristics don’t come just from inspiration neither reading and understanding the theory, nor from a big flame war at email or IRC. We need examples. We learn, as good ape descendants, from imitation.

We are learning/fixing/prototyping and we need fast feedback in order to keep us happy and motivated. We need perceive the progress of our iterative work. The “copy&paste” is great for those purposes. Then, the discipline must do its work sustaining the new acquired knowledge, but that is another history.

But even in the “copy&paste” we must be smart and responsible with ourselves: we must seek the best examples to imitate, we must look at the alpha male, to borrow the brightest resources available. “Copy&Paste” the code of a lousy programmer and you will be another one; “copy&paste” the the code of a great programmer and maybe, someday, you and I will be one.

A great resource to search, easily and fast, great sources of code, since a couple years to now, is Google’s CodeSearch. It does searches along a great number of successful and recognized open projects sources (released tarballs mostly).

When I am working on a programming task which implies new code, my usual work-flow is to visualize the solution as a sequential execution of macro operations which will be defined to to a more fine grained at each iteration. At each iteration usually I found situations when I want to have either a quick solution, or I want to find how others have solved it, or just how to use a specific function/method call, so I go the CodeSearch site and type the used programming language, the related API, et voilà, several possible solutions are shown.

Further more, if I need know how to achieve something with autoconf-fu, script-fu, or even system-configuration-fu, I can do searches with specific file names as Makefile.am or configure.ac.

My two cents in programming tips.

summer hack

2008-08-18T00:00:00Z

I have wrote a Rhythmbox plugin for playing mp3 streams from goear.com. It searches, requests several pages on demand, fill the metadata when the stream begins and sorts the search results.What is missing is the capability to make playlists with selected streams.

You can find the patch in Rhythmbox’s bugzilla. Just patch the code, add the goear.png in the plugins/goear directory, build et voila. The patch was done with the current subversion HEAD, but also is applied cleanly in the Hardy Ubuntu’s source package.

By the way, you can find a Hardy Ubuntu package for 32bits with the patch here (the metadata filling doesn’t work here :()

devhelp and jhbuild

2008-07-24T00:00:00Z

When I started to use jhbuild I resolved to not compile a whole new Gnome desktop because my work laptop is too slow and freeze easily: I had to reduce the dependencies at minimum and try reuse the libraries already provided by my distro (Ubuntu by the way).The first trouble I had was to have access to the gtk-doc API documentation through the installed devhelp. The solution is simply, but not obvious (at least for me):

Append in ~/.profile

XDG_DATA_DIRS="${XDG_DATA_DIRS}":/opt/jhbuild/share

export XDG_DATA_DIRS

Well, you know you’ve to change the path to your jhbuild deployment directory

Guadec 2008

2008-07-21T00:00:00Z

Last week I attended the Guadec 2008 in Turkey. It was my second Guadec. The first one was in Sevilla in 2002, when I was backpacking Europe. The big difference between my previous and this last guadec is that now I have a better understanding of the state of the art, so I could appreciate more the talks and proposals done, which I will try to summarize in this post.

The Gnome community faces new and not so new challenges. The framework is trying to push its limits to new frontiers besides renovate its own core.

First there is the discussion about the strategy for Gtk+. Imendio is pushing for a ABI breakage in order to polish the API, sealing the private data in the components, among other things. Nevertheless, Miguel, being the voice of several ISV, is disagree with proposed path.

In the Gtk+ road map, a great discussion has been raised about the new canvas. Seems that the favorite contender is Clutter, pushed by OpenedHand.

In other front, Mark Shuttleworth came up with the idea to integrate QT in Gnome, replacing Gtk+ in the long term.

Alp Toker presented the advances in WebKit development, and in my perception, a lot of gnomies are webkit enthusiast, because have a tighter integration with the Gtk+ widgets and better performance than Firefox currently (which is always temporal). So I expect more applications using webkit rather than embedded Mozilla.

Following the web area, there’s a big effort, basically from the RedHat guys for the online desktop, integrating the desktop with different and heterogeneous web feeds. I think this is a must for the next desktop environment. More and more applications are web integrated, and give a framework to do this and mix all that information smartly is a huge need.

All the multimedia stack is healthy managed by the guys at Collabora and Fluendo. There’s an important effort in data communication (Telepathy and Farsight) using the GStreamer framework. Meanwhile another project, Ekiga, which doesn’t use GStreamer is steady and currently the better application for VoIP in Gnome.

Finally, Miguel is pushing for a HTTP desktop applications model, using Moonlight as front end machinery, and a custom HTTP server as gluing proxy for the back end systems applications. A bold idea with not too many supporters.

As colophon we must say in the mobile area are great efforts (Nokia as main bidder) to keep sync with the mainstream development and improve the performance in both fronts.

The ideas are in the table, the discussion is on going, and the code being committed. Where do you want to work in?