Herostratus’ legacy

words from a lazy coder

GStreamer Hackfest 2013 - Milan

Last week, from 28th to 31th of March, some of us gathered at Milan to hack some bits of the GStreamer internals. For me was a great experience interact with great hackers such as Sebastian Dröge, Wim Taymans, Edward Hervey, Alessandro Decina and many more. We talked about GStreamer and, more particularly, we agreed on new features which I would like to discuss here.GStreamer Hackers at Milan

For sake of completeness, let me say that I have been interested in hardware accelerated multimedia for a while, and just lately I started to wet my feet in VAAPI and VDPAU, and their support in our beloved GStreamer.

GstContext #

The first feature that reached upstream is the GstContext. Historically, in 2011, Nicolas Dufresne added GstVideoContext as an interface to a share video context (such as display name, X11 display, VA-API display, etc.) among the pipeline elements and the applications. But now, Sebastian, generalized the interface to a container to stores and shares any kind of contexts between multiple elements and the application.

The first approach, that is still living in gst-plugins-bad, was merely a wrapper to a custom query to set or request a video context. But now, the context sharing is part of the pipeline setup.

An element that needs a shared context must follow these actions:

  1. Check if the element already has a context
  2. Query downstream for the context
  3. Post a message in the bus to see if the application has one to share.
  4. Create the context if there is none, post a message and send an event letting know that the element has the context.

You can see the example of the eglglessink to know how to use this feature.

GstVideoGLTextureUploadMeta #

Also in 2011, Nicolas Dufresne, added a helper class to upload a buffer into a surface (OpenGL texture, VA API surface, Wayland surface, etc.). This is quite important since the new video players are scene based, using framework such as Clutter or OpenGL directly, where the video display is composed by various actors, such as the multimedia controls widgets.

But still, this interface didn’t fit well for GStreamer 1.0, until now, where it was introduced in the figure of a buffer’s meta, though this meta is only specific for OpenGL textures. If the buffer provides this new GstVideoGLTextureUploadMeta meta, a new function gst_video_gl_texture_upload_meta_upload() is available to upload that buffer into an OpenGL texture specified by its numeric identifier.

Obviously, in order to use this meta, it should be proposed for allocation by the sink. Again, you can see the case of eglglessink as example.

Caps Features #

The caps features are a new data type for specify a specific extension or requirement for the handled media.

From the practical point of view, we can say that caps structures with the same name but with a non-equal set of caps features are not compatible, and, if a pad supports multiple sets of features it has to add multiple equal structures with different feature sets to the caps.

Empty GstCapsFeatures are equivalent with the GstCapsFeatures handled by the common system memory. Other examples would be a specific memory types or the requirement of having a specific meta on the buffer.

Again, we can see the example of the capsfeatures in eglglessink, because now the gst-inspect also shows the caps feature of the pads:

    Pad Templates:
      SINK template: 'sink'
        Availability: Always
        Capabilities:
          video/x-raw(memory:EGLImage)
                     format: { RGBA, BGRA, ARGB, ABGR, RGBx,
                               BGRx, xRGB, xBGR, AYUV, Y444,
                               I420, YV12, NV12, NV21, Y42B,
                               Y41B, RGB, BGR, RGB16 }
                      width: [ 1, 2147483647 ]
                     height: [ 1, 2147483647 ]
                  framerate: [ 0/1, 2147483647/1 ]
          video/x-raw(meta:GstVideoGLTextureUploadMeta)
                     format: { RGBA, BGRA, ARGB, ABGR, RGBx,
                               BGRx, xRGB, xBGR, AYUV, Y444,
                               I420, YV12, NV12, NV21, Y42B,
                               Y41B, RGB, BGR, RGB16 }
                      width: [ 1, 2147483647 ]
                     height: [ 1, 2147483647 ]
                  framerate: [ 0/1, 2147483647/1 ]
          video/x-raw
                     format: { RGBA, BGRA, ARGB, ABGR, RGBx,
                               BGRx, xRGB, xBGR, AYUV, Y444,
                               I420, YV12, NV12, NV21, Y42B,
                               Y41B, RGB, BGR, RGB16 }
                      width: [ 1, 2147483647 ]
                     height: [ 1, 2147483647 ]
                  framerate: [ 0/1, 2147483647/1 ]

Parsers meta #

This is a feature which has been pulled by Edward Hervey. The idea is that the video codec parsers (H264, MPEG, VC1) attach a meta into the buffer with a defined structure that carries that new information provided by the codified stream.

This is particularly useful by the decoders, which will not have to parse again the buffer in order to extract the information they need to decode the current buffer and the following.

For example, here is the H264 parser meta definition.

VDPAU #

Another task pulled by Edward Hervey, for which I feel excited, is the port of VDPAU decoding elements to GStreamer 1.0.

Right now only the MPEG decoder is upstreamed, but MPEG4 and H264 are coming.

As a final note, I want to thank Collabora and Fluendo for sponsoring dinners. A special thank you, as well, for Igalia which covered my travel expenses and attendance to the hackfest.