New VA-API H.264 decoder in gst-plugins-bad

Recently, a new H.264 decoder, using VA-API, was merged in gst-plugins-bad.

Why another VA-based H.264 decoder if there is already gstreamer-vaapi?

As usual, an historical perspective may give some clues.

It started when Seungha Yang implemented the GStreamer decoders for Windows using DXVA2 and D3D11 APIs.

Perhaps we need one step back and explain what are stateless decoders.

Video decoders are magic and opaque boxes where we push encoded frames, and later we’ll pop full decoded frames in raw format. This is how OpenMAX and V4L2 decoders work, for example.

Internally we can imagine those magic and opaque boxes has two main operations:

  • Codec state handling
  • Signal processing like Fourier-related transformations (such as DCT), entropy coding, etc. (DSP, in general)

The codec state handling basically extracts, from the stream, the frame’s parameters and its compressed data, so the DSP algorithms can decode the frames. Codec state handling can be done with generic CPUs, while DSP algorithms are massively improved through specific purpose processors.

These video decoders are known as stateful decoders, and usually they are distributed through binary and closed blobs.

Soon, silicon vendors realized they can offload the burden of state handling to third-party user-space libraries, releasing what it is known as stateless decoders. With them, your code not only has to push frames into the opaque box, but now it shall handle the codec specifics to provide all the parameters and references for each frame. VAAPI and DXVA2 are examples of those stateless decoders.

Returning to Seungha’s implementation, in order to get theirs DXVA2/D3D11 decoders, they also needed a state handler library for each codec. And Seungha wrote that library!

Initially they wanted to reuse the state handling in gstreamer-vaapi, which works pretty good, but its internal library, from the GStreamer perspective, is over-engineered: it is impossible to rip out only the state handling without importing all its data types. Which is kind of sad.

Later, Nicolas Dufresne, realized that this library can be re-used by other GStreamer plugins, because more stateless decoders are now available, particularly V4L2 stateless, in which he is interested. Nicolas moved Seungha’s code into a library in gst-plugins-bad.

Currently, libgstcodecs provides state handling of H.264, H.265, VP8 and VP9.

Let’s return to our original question: Why another VA-based H.264 decoder if there is already one in gstreamer-vaapi?

The quick answer is «to pay my technical debt».

As we already mentioned, gstreamer-vaapi is big and over-engineered, though we have being simplifying the internal libraries, in particular He Junyan, has done a lot of work replacing the internal base class, GstVaapiObject, withGstObject or GstMiniObject. Also, this kind of projects, where there’s a lot of untouched code, it carries a lot of cargo cult decisions.

So I took the libgstcodecs opportunity to write a simple, thin and lean, H.264 decoder, using VA new API calls (vaExportSurfaceHandle(), for example) and learning from other implementations, such as FFMpeg and ChromeOS. This exercise allowed me to identify where are the dusty spots in gstreamer-vaapi and how they should be fixed (and we have been doing it since then!).

Also, this opportunity lead me to learn a bit more about the H.264 specification since I implemented the reference picture list handling, and fixed a small bug in Chromium.

Now, let me be crystal clear: GStreamer VA-API is not going anywhere. It is, right now, one of the most feature-complete implementations using VA-API, even with its integration issues, and we are working on them, particularly, Intel folks are working hard on a new AV1 decoder, enhancing encoders and adding new video post-processing features.

But, this new vah264dec is an experimental VA-API decoder, which aims towards a tight integration with GStreamer, oriented to provide a good experience in most of the common use cases and to enhance the common libgstcodecs library shared with other stateless decoders, looking to avoid Intel specific nuances.

These are the main characteristics and plans of this new decoder:

  • It use, by default, a DRM connection to VA display, avoiding the troubles of choosing X11 or Wayland.
    • It uses the first found DRM device as VA display
    • In the future, users will be able to provide their custom VA display through the pipeline’s context.
  • It requires libva >= 1.6
  • No multiview/stereo profiles, neither interlaced streams, because libgstcodecs doesn’t handle them yet
  • It is incompatible with gstreamer-vaapi: mixing elements might lead to problems.
  • Even if memory:VAMemory is exposed, it is not handled yet by any other element yet.
    • Users will get VASurfaces via mapping as GstGL does with textures.
  • Caps templates are generated dynamically generated by querying VAAPI
  • YV12 and I420 are added for system memory caps because they seem to be supported for all the drivers when downloading frames onto main memory, as they are used by xvimagesink and others, avoiding color conversion.
  • Decoding surfaces aren’t bounded to context, so they can grow beyond the DBP size, allowing smooth reverse playback.
  • There isn’t yet error handling and recovery.
  • The element is supposed to spawn if different renderD nodes with VA-API driver support are found (like gstv4l2), but it hasn’t been tested yet.

Now you may be asking how do I use vah264dec?

Currently vah264dec has NONE rank, which means that it will never be autoplugged, but you can use the trick of the environment variable GST_PLUGIN_FEATURE_RANK:

$ GST_PLUGIN_FEATURE_RANK=vah264dec:259 gst-play-1.0 ~/video.mp4

And that’s it!

Thanks.

WebKit Flatpak SDK and gst-build

This post is an annex of Phil’s Introducing the WebKit Flatpak SDK. Please make sure to read it, if you haven’t already.

Recapitulating, nowadays WebKitGtk/WPE developers —and their CI infrastructure— are moving towards to Flatpak-based environment for their workflow. This Flatpak-based environment, or Flatpak SDK for short, can be visualized as a software sandboxed-container, which bundles all the dependencies required to compile, run and debug WebKitGtk/WPE.

In a day-by-day work, this approach removes the potential compilation of the world in order to obtain reproducible builds, improving the development and testing work flow.

But what if you are also involved in the development of one dependency?

This is the case of Igalia’s multimedia team where, besides developing the multimedia features for WebKitGtk and WPE, we also participate in the GStreamer development, the framework used for multimedia.

Because of this, in our workflow we usually need to build WebKit with a fix, hack or new feature in GStreamer. Is it possible to add in Flatpak our custom GStreamer build without messing its own GStreamer setup? Yes, it’s possible.

gst-build is a set of scripts in Python which clone GStreamer repositories, compile them and setup an uninstalled environment. This uninstalled environment allows a transient usage of the compiled framework from their build tree, avoiding installation and further mess up with our system.

The WebKit scripts that wraps Flatpak operations are also capable to handle the scripts of gst-build to build GStreamer inside the container, and, when running WebKit’s artifacts, the scripts enable the mentioned uninstalled environment, overloading Flatpak’s GStreamer.

How do we unveil all this magic?

First of all, setup a gst-build installation as it is documented. In this installation is were the GStreamer plumbing is done.

Later, gst-build operations through WebKit compilation scripts are enabled when the environment variable GST_BUILD_PATH is exported. This variable should point to the directory where the gst-build tree is placed.

And that’s all!

But let’s put these words in actual commands. The following workflow assumes that WebKit repository is cloned in ~/WebKit and the gst-build tree is in ~/gst-build (please, excuse my bashisms).

Compiling WebKitGtk with symbols, using LLVM as toolchain (this command will also compile GStreamer):

$ cd ~/WebKit
% CC=clang CXX=clang++ GST_BUILD_PATH=/home/vjaquez/gst-build Tools/Scripts/build-webkit --gtk --debug
...

Running the generated minibrowser (remind GST_BUILD_PATH is required again for a correct linking):

$ GST_BUILD_PATH=/home/vjaquez/gst-build Tools/Scripts/run-minibrowser --gtk --debug
...

Running media layout tests:

$ GST_BUILD_PATH=/home/vjaquez/gst-build ./Tools/Scripts/run-webkit-tests --gtk --debug media

But wait! There’s more...

What if you I want to parametrize the GStreamer compilation. To say, I would like to enable a GStreamer module or disable the built of a specific element.

gst-build, as the rest of GStreamer modules, uses meson build system, so it’s possible to pass arguments to meson through the environment variable GST_BUILD_ARGS.

For example, I would like to enable gstreamer-vaapi 😇

$ cd ~/WebKit
% CC=clang CXX=clang++ GST_BUILD_PATH=/home/vjaquez/gst-build GST_BUILD_ARGS="-Dvaapi=enabled" Tools/Scripts/build-webkit --gtk --debug
...

Review of the Igalia Multimedia team Activities (2019/H2)

This blog post is a review of the various activities the Igalia Multimedia team was involved along the second half of 2019.

Here are the previous 2018/H2 and 2019/H1 reports.

GstWPE

Succinctly, GstWPE is a GStreamer plugin which allows to render web-pages as a video stream where it frames are GL textures.

Phil, its main author, wrote a blog post explaning at detail what is GstWPE and its possible use-cases. He wrote a demo too, which grabs and previews a live stream from a webcam session and blends it with an overlay from wpesrc, which displays HTML content. This composited live stream can be broadcasted through YouTube or Twitch.

These concepts are better explained by Phil himself in the following lighting talk, presented at the last GStreamer Conference in Lyon:

Video Editing

After implementing a deep integration of the GStreamer Editing Services (a.k.a GES) into Pixar’s OpenTimelineIO during the first half of 2019, we decided to implement an important missing feature for the professional video editing industry: nested timelines.

Toward that goal, Thibault worked with the GSoC student Swayamjeet Swain to implement a flexible API to support nested timelines in GES. This means that users of GES can now decouple each scene into different projects when editing long videos. This work is going to be released in the upcoming GStreamer 1.18 version.

Henry Wilkes also implemented the support for nested timeline in OpenTimelineIO making GES integration one of the most advanced one as you can see on that table:

Feature OTIO EDL FCP7 XML FCP X AAF RV ALE GES
Single Track of Clips W-O
Multiple Video Tracks W-O
Audio Tracks & Clips W-O
Gap/Filler
Markers N/A
Nesting W-O
Transitions W-O
Audio/Video Effects N/A
Linear Speed Effects R-O
Fancy Speed Effects
Color Decision List N/A

Along these lines, Thibault delivered a 15 minutes talk, also in the GStreamer Conference 2019:

After detecting a few regressions and issues in GStreamer, related to frame accuracy, we decided to make sure that we can seek in a perfectly frame accurate way using GStreamer and the GStreamer Editing Services. In order to ensure that, an extensive integration testsuite has been developed, mostly targeting most important container formats and codecs (namely mxf, quicktime, h264, h265, prores, jpeg) and issues have been fixed in different places. On top of that, new APIs are being added to GES to allow expressing times in frame number instead of nanoseconds. This work is still ongoing but should be merged in time for GStreamer 1.18.

GStreamer Validate Flow

GstValidate has been turning into one of the most important GStreamer testing tools to check that elements behave as they are supposed to do in the framework.

Along with our MSE work, we found that other way to specify tests, related with produced buffers and events through specific pads, was needed. Thus, Alicia developed a new plugin for GstValidate: Validate Flow.

Alicia gave an informative 30 minutes talk about GstValidate and the new plugin in the last GStreamer Conference too:

GStreamer VAAPI

Most of the work along the second half of 2019 were maintenance tasks and code reviews.

We worked mainly on memory restrictions per backend driver, and we reviewed a big refactor: internal encoders now use GstObject, instead of the custom GstVaapiObject. Also we reviewed patches for new features such as video rotation and cropping in vaapipostproc.

Servo multimedia

Last year we worked integrating media playing in Servo. We finally delivered hardware accelerated video playback in Linux and Android. We worked also for Windows and Mac ports but they were not finished. As natural, most of the work were in servo/media crate, pushing code and reviewing contributions. The major tasks were to rewrite the media player example and the internal source element looking to handle the download playbin‘s flag properly.

We also added WebGL integration support with <video> elements, thus webpages can use video frames as WebGL textures.

Finally we explored how to isolate the multimedia processing in a dedicated thread or process, but that task remains pending.

WebKit Media Source Extension

We did a lot of downstream and upstream bug fixing and patch review, both in WebKit and GStreamer, for our MSE GStreamer-based backend.

Along this line we improved WebKitMediaSource to use playbin3 but also compatibility with older GStreamer versions was added.

WebKit WebRTC

Most of the work in this area were maintenance and fix regressions uncovered by the layout tests. Besides, the support for the Rasberry Pi was improved by handling encoded streams from v4l2 video sources, with some explorations with Minnowboard on top of that.

Conferences

GStreamer Conference

Igalia was Gold sponsor this last GStreamer Conference held in Lyon, France.

All team attended and five talks were delivered. Only Thibault presented, besides the video editing one which we already referred, another two more: One about GstTranscoder API and the other about the new documentation infrastructure based in Hotdoc:

We also had a productive hackfest, after the conference, where we worked on AV1 Rust decoder, HLS Rust demuxer, hardware decoder flag in playbin, and other stuff.

Linaro Connect

Phil attended the Linaro Connect conference in San Diego, USA. He delivered a talk about WPE/Multimedia which you can enjoy here:

Demuxed

Charlie attended Demuxed, in San Francisco. The conference is heavily focused on streaming and codec engineering and validation. Sadly there are not much interest in GStreamer, as the main focus is on FFmpeg.

RustFest

Phil and I attended the last RustFest in Barcelona. Basically we went to meet with the Rust community and we attended the “WebRTC with GStreamer-rs” workshop presented by Sebastian Dröge.

GStreamer-VAAPI 1.16 and libva 2.6 in Debian

Debian has migrated libva 2.6 into testing. This release includes a pull request that changes how the drivers are selected to be loaded and used. As the pull request mentions:

libva will try to load iHD firstly, if it failed. then it will load i965.

Also, Debian testing has imported that iHD driver with two flavors: intel-media-driver and intel-media-driver-non-free. So basically iHD driver is now the main VAAPI driver for Intel platforms, though it only supports the new chips, the old ones still require i965-va-driver.

Sadly, for current GStreamer-VAAPI stable, the iHD driver is not included in its driver white list. And this will pose a problem for users that have installed either of the intel-media-driver packages, because, by default, such driver is ignored and the VAAPI GStreamer elements won’t be registered.

There are three temporal workarounds (mutually excluded) for those users (updated):

  1. Uninstall intel-media-driver* and install (or keep) the old i965-va-driver-shaders/i965-va-driver.
  2. Export, by default in your session, export LIBVA_DRIVER_NAME=i965. Normally this is done adding the variable exportation in $HOME/.profile file. This environment variable will force libva to load the i965 driver.
  3. And finally, export, by default in your sessions, GST_VAAPI_ALL_DRIVERS=1. This is not advised since many applications, such as Epiphany, might fail.

We prefer to not include iHD in the stable white list because most of the work done for that driver has occurred after release 1.16.

In the case of GStreamer-VAAPI master branch (actively in develop) we have merged the iHD in the white list, since the Intel team has been working a lot to make it work. Though, it will be released for GStreamer version 1.18.

Generating a GStreamer-1.14 bundle for TravisCI with Ubuntu/Trusty

For having continous integration in your multimedia project hosted in GitHub with TravisCI, you may want to compile and run tests with a recent version of GStreamer. Nonetheless, TravisCI mainly offers Ubuntu Trusty as one of the possible distributions to deploy in their CI, and that distribution packages GStreamer 1.2, which might be a bit old for your project’s requirements.

A solution for this problem is to provide to TravisCI your own GStreamer bundle with the version you want to compile and test on you project, in this case 1.14. The present blog is recipe I followed to generate that GStreamer bundle with GstGL support.

There are three main issues:

  1. The packaged libglib version is too old, hoping that we will not find an ABI breakage while running the CI.
  2. The packaged ffmpeg version is too old
  3. As we want to compile GStreamer using gst-build, we need a recent version of meson, which requires python3.5, not available in Trusty.

schroot

Old habits die hard, and I have used schroot for handle chroot environments without complains, it handles the bind mounting of /proc, /sys and all that repetitive stuff that seals the isolation of the chrooted environment.

The debootstrap’s variant I use is buildd because it installs the build-essential package.

$ sudo mkdir /srv/chroot/gst-trusty64
$ sudo debootstrap --arch=amd64 --variant=buildd trusty ./gst-trusty64/ http://archive.ubuntu.com/ubuntu
$ sudo vim /etc/schroot/chroot.d

This is the schroot configuration I will use. Please, adapt it to your need.

[gst]
description=Ubuntu Trusty 64-bit for GStreamer
directory=/srv/chroot/gst-trusty64
type=directory
users=vjaquez
root-users=vjaquez
profile=default
setup.fstab=default/vjaquez-home.fstab

I am overrinding the fstab default file for a custom one where the home directory of vjaquez user aims to a clean directory.

$ mkdir -p ~/home-chroot/gst
$ sudo vim /etc/schroot/default/vjaquez-home.fstab
# fstab: static file system information for chroots.
# Note that the mount point will be prefixed by the chroot path
# (CHROOT_PATH)
#
#                
/proc           /proc           none    rw,bind         0       0
/sys            /sys            none    rw,bind         0       0
/dev            /dev            none    rw,bind         0       0
/dev/pts        /dev/pts        none    rw,bind         0       0
/home           /home           none    rw,bind         0       0
/home/vjaquez/home-chroot/gst   /home/vjaquez   none    rw,bind 0       0
/tmp            /tmp            none    rw,bind         0       0

configure chroot environment

We will get into the chroot environment as super user in order to add the required packages. For that pupose we add universe repository in apt.

  • libglib requires: autotools-dev gnome-pkg-tools libtool libffi-dev libelf-dev libpcre3-dev desktop-file-utils libselinux1-dev libgamin-dev dbus dbus-x11 shared-mime-info libxml2-utils
  • Python requires: libssl-dev libreadline-dev libsqlite3-dev
  • GStreamer requires: bison flex yasm python3-pip libasound2-dev libbz2-dev libcap-dev libdrm-dev libegl1-mesa-dev libfaad-dev libgl1-mesa-dev libgles2-mesa-dev libgmp-dev libgsl0-dev libjpeg-dev libmms-dev libmpg123-dev libogg-dev libopus-dev liborc-0.4-dev libpango1.0-dev libpng-dev libpulse-dev librtmp-dev libtheora-dev libtwolame-dev libvorbis-dev libvpx-dev libwebp-dev pkg-config unzip zlib1g-dev
  • And for general setup: language-pack-en ccache git curl
$ schroot --user root --chroot gst
(gst)# sed -i "s/main$/main universe/g" /etc/apt/sources.list
(gst)# apt update
(gst)# apt upgrade
(gst)# apt --no-install-recommends --no-install-suggests install \
autotools-dev gnome-pkg-tools libtool libffi-dev libelf-dev \
libpcre3-dev desktop-file-utils libselinux1-dev libgamin-dev dbus \
dbus-x11 shared-mime-info libxml2-utils \
libssl-dev libreadline-dev libsqlite3-dev \ 
language-pack-en ccache git curl bison flex yasm python3-pip \
libasound2-dev libbz2-dev libcap-dev libdrm-dev libegl1-mesa-dev \
libfaad-dev libgl1-mesa-dev libgles2-mesa-dev libgmp-dev libgsl0-dev \
libjpeg-dev libmms-dev libmpg123-dev libogg-dev libopus-dev \
liborc-0.4-dev libpango1.0-dev libpng-dev libpulse-dev librtmp-dev \
libtheora-dev libtwolame-dev libvorbis-dev libvpx-dev libwebp-dev \
pkg-config unzip zlib1g-dev

Finally we create our installation prefix. In this case /opt/gst to avoid the contamination of /usr/local and logout as root.

(gst)# mkdir -p /opt/gst
(gst)# chown vjaquez /opt/gst
(gst)# exit

compile ffmpeg 3.2

Now, let’s login again, but as the unprivileged user, to build the bundle, starting with ffmpeg. Notice that we are using ccache and building out-of-source.

$ schroot --chroot gst
(gst)$ git clone https://git.ffmpeg.org/ffmpeg.git ffmpeg
(gst)$ cd ffmpeg
(gst)$ git checkout -b work n3.2.12
(gst)$ mkdir build
(gst)$ cd build
(gst)$ ../configure --disable-static --enable-shared \
--disable-programs --enable-pic --disable-doc --prefix=/opt/gst 
(gst)$ PATH=/usr/lib/ccache/:${PATH} make -j8 install

compile glib 2.48

(gst)$ cd ~
(gst)$ git clone https://gitlab.gnome.org/GNOME/glib.git
(gst)$ cd glib
(gst)$ git checkout -b work origin/glib-2-48
(gst)$ mkdir mybuild
(gst)$ cd mybuild
(gst)$ ../autogen.sh --prefix=/opt/gst
(gst)$ PATH=/usr/lib/ccache/:${PATH} make -j8 install

install Python 3.5

Pyenv is a project that allows the automation of installing and executing, in the user home directory, multiple versions of Python.

(gst)$ curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash
(gst)$ ~/.pyenv/bin/pyenv install 3.5.0

Install meson 0.50

We will install the last available version of meson in the user home directory, that is why PATH is extended and exported.

(gst)$ cd ~
(gst)$ ~/.pyenv/verion/3.5.0/pip3 install --user meson
(gst)$ export PATH=${HOME}/.local/bin:${PATH}

build GStreamer 1.14

PKG_CONFIG_PATH is exported to expose the compiled versions of ffmpeg and glib. Notice that the libraries are installed in /opt/lib in order to avoid the dispersion of pkg-config files.

(gst)$ cd ~/
(gst)$ export PKG_CONFIG_PATH=/opt/gst/lib/pkgconfig/
(gst)$ git clone https://gitlab.freedesktop.org/gstreamer/gst-build.git
(gst)$ cd gst-build
(gst)$ git checkout -b work origin/1.14
(gst)$ meson -Denable_python=false \
-Ddisable_gst_libav=false -Ddisable_gst_plugins_ugly=true \
-Ddisable_gst_plugins_bad=false -Ddisable_gst_devtools=true \
-Ddisable_gst_editing_services=true -Ddisable_rtsp_server=true \
-Ddisable_gst_omx=true -Ddisable_gstreamer_vaapi=true \
-Ddisable_gstreamer_sharp=true -Ddisable_introspection=true \
--prefix=/opt/gst build --libdir=lib
(gst)$ ninja -C build install

test!

(gst)$ cd ~/
(gst)$ LD_LIBRARY_PATH=/opt/gst/lib \
GST_PLUGIN_SYSTEM_PATH=/opt/gst/lib/gstreamer-1.0/ \
/opt/gst/bin/gst-inspect-1.0

And the list of available elemente shall be shown.

archive the bundle

(gst)$ cd ~/
(gst)$ tar zpcvf gstreamer-1.14-x86_64-linux-gnu.tar.gz -C /opt ./gst

update your .travis.yml

These are the packages you shall add to run this generated GStreamer bundle:

  • libasound2-plugins
  • libfaad2
  • libfftw3-single3
  • libjack-jackd2-0
  • libmms0
  • libmpg123-0
  • libopus0
  • liborc-0.4-0
  • libpulsedsp
  • libsamplerate0
  • libspeexdsp1
  • libtdb1
  • libtheora0
  • libtwolame0
  • libwayland-egl1-mesa
  • libwebp5
  • libwebrtc-audio-processing-0
  • liborc-0.4-dev
  • pulseaudio
  • pulseaudio-utils

And this is the before_install and before_script targets:

      before_install:
        - curl -L http://server.example/gstreamer-1.14-x86_64-linux-gnu.tar.gz | tar xz
        - sed -i "s;prefix=/opt/gst;prefix=$PWD/gst;g" $PWD/gst/lib/pkgconfig/*.pc
        - export PKG_CONFIG_PATH=$PWD/gst/lib/pkgconfig
        - export GST_PLUGIN_SYSTEM_PATH=$PWD/gst/lib/gstreamer-1.0
        - export GST_PLUGIN_SCANNER=$PWD/gst/libexec/gstreamer-1.0/gst-plugin-scanner
        - export PATH=$PATH:$PWD/gst/bin
        - export LD_LIBRARY_PATH=$PWD/gst/lib:$LD_LIBRARY_PATH

      before_script:
        - pulseaudio --start
        - gst-inspect-1.0 | grep Total

Review of Igalia’s Multimedia Activities (2018/H2)

This is the first semiyearly report about Igalia’s activities around multimedia, covering the second half of 2018.

Great length of this report was exposed in Phil’s talk surveying mutimedia development in WebKitGTK and WPE:

WebKit Media Source Extensions (MSE)

MSE is a specification that allows JS to generate media streams for playback for Web browsers that support HTML 5 video and audio.

Last semester we upstreamed the support to WebM format in WebKitGTK with the related patches in GStreamer, particularly in qtdemux, matroskademux elements.

WebKit Encrypted Media Extensions (EME)

EME is a specification for enabling playback of encrypted content in Web bowsers that support HTML 5 video.

In a downstream project for WPE WebKit we managed to have almost full test coverage in the YoutubeTV 2018 test suite.

We merged our contributions in upstream, WebKit and GStreamer, most of what is legal to publish, for example, making demuxers aware of encrypted content and make them to send protection events with the initialization data and the encrypted caps, in order to select later the decryption key.

We started to coordinate the upstreaming process of a new implementation of CDM (Content Decryption Module) abstraction and there will be even changes in that abstraction.

Lighting talk about EME implementation in WPE/WebKitGTK in GStreamer Conference 2018.

WebKit WebRTC

WebRTC consists of several interrelated APIs and real time protocols to enable Web applications and sites to captures audio, or A/V streams, and exchange them between browsers without requiring an intermediary.

We added GStreamer interfaces to LibWebRTC, to use it for the network part, while using GStreamer for the media capture and processing. All that was upstreamed in 2018 H2.

Thibault described thoroughly the tasks done for this achievement.

Talk about WebRTC implementation in WPE/WebKitGTK in WebEngines hackfest 2018.

Servo/media

Servo is a browser engine written in Rust designed for high parallelization and high GPU usage.

We added basic support for <video> and <audio> media elements in Servo. Later on, we added the GstreamerGL bindings for Rust in gstreamer-rs to render GL textures from the GStreamer pipeline in Servo.

Lighting talk in the GStreamer Conference 2018.

GstWPE

Taking an idea from the GStreamer Conference, we developed a GStreamer source element that wraps WPE. With this source element, it is possible to blend a web page and video in a single video stream; that is, the output of a Web browser (to say, a rendered web page) is used as a video source of a GStreamer pipeline: GstWPE. The element is already merged in the gst-plugins-bad repository.

Talk about GstWPE in FOSDEM 2019

Demo #1

Demo #2

GStreamer VA-API and gst-MSDK

At last, but not the least, we continued helping with the maintenance of GStreamer-VAAPI and gst-msdk, with code reviewing and on-going migration of the internal library to GObject.

Other activities

The second half of 2018 was also intense in terms of conferences and hackfest for the team:


Thanks to bear with us along all this blog post and to keeping under your radar our work.

Rust bindings for GStreamerGL: Memoirs

Rust is a great programming language but the community around it’s just amazing. Those are the ingredients for the craft of useful software tools, just like Servo, an experimental browser engine designed for tasks isolation and high parallelization.

Both projects, Rust and Servo, are funded by Mozilla.

Thanks to Mozilla and Igalia I have the opportunity to work on Servo, adding it HTML5 multimedia features.

First, with the help of Fernando Jiménez, we finished what my colleague Philippe Normand and Sebastian Dröge (one of my programming heroes) started: a media player in Rust designed to be integrated in Servo. This media player lives in its own crate: servo/media along with the WebAudio engine. A crate, in Rust jargon, is like a library. This crate is (very ad-hocly) designed to be multimedia framework agnostic, but the only backend right now is for GStreamer. Later we integrated it into Servo adding an initial support for audio and video tags.

Currently, servo/media passes, through a IPC channel, the array with the whole frame to render in Servo. This implies, at least, one copy of the frame in memory, and we would like to avoid it.

For painting and compositing the web content, Servo uses WebRender, a crate designed to use the GPU intensively. Thus, if instead of raw frame data we pass OpenGL textures to WebRender the performance could be enhanced notoriously.

Luckily, GStreamer already supports the uploading, downloading, painting and composition of video frames as OpenGL textures with the OpenGL plugin and its OpenGL Integration library. Even more, with plugins such as GStreamer-VAAPI, Gst-OMX (OpenMAX), and others, it’s possible to process video without using the main CPU or its mapped memory in different platforms.

But from what’s available in GStreamer to what it’s available in Rust there’s a distance. Nonetheless, Sebastian has putting a lot of effort in the Rust bindings for GStreamer, either for applications and plugins, sadly, GStreamer’s OpenGL Integration library (GstGL for short) wasn’t available at that time. So I rolled up my sleeves and got to work on the bindings.

These are the stories of that work.

As GStreamer shares with GTK+ the GObject framework and its introspection mechanism, both projects have collaborated on the required infrastructure to support Rust bindings. Thanks to all the GNOME folks who are working on the intercommunication between Rust and GObject. The quest has been long and complex, since Rust doesn’t map all the object oriented concepts, and GObject, being a set of practices and software helpers to do object oriented programming with C, its usage is not homogeneous.

The Rubicon that ease the generation of Rust bindings for GObject-based projects is GIR, a tool, written in Rust, that reads gir files, along with metadata in toml, and outputs two types of bindings: sys and api.

Rust can call external functions through FFI (foreign function interface), which is just a declaration of a C function with Rust types. But these functions are considered unsafe. The sys bindings, are just the exporting of the C function for the library organized by the library’s namespace.

The next step is to create a safe and rustified API. This is the api bindings.

As we said, GObject libraries are quite homogeneous, and even following the introspection annotations, there will be cases where GIR won’t be able to generate the correct bindings. For that reason GIR is constantly evolving, looking for a common way to solve the corner cases that exist in every GObject project. For example, these are my patches in order to generate the GstGL bindings.

The done tasks were:

For this document we assume that the reader has a functional Rust setup and they know the basic concepts.

Clone and build gir

$ cd ~/ws
$ git clone https://github.com/gtk-rs/gir.git
$ cd gir
$ cargo build --release

The reason to build gir in release mode is because, otherwise would be very slow.

For sys bindings.

These kind of bindings are normally straight forward (and unsafe) since they only map the C API to Rust via FFI mechanism.

$ cd ~/ws
$ git clone https://gitlab.freedesktop.org/gstreamer/gstreamer-rs-sys.git
$ cd gstreamer-rs-sys
$ cp /usr/share/gir-1.0/GstGL-1.0.gir gir-files/
  1. Verify if the gir file is more o less correct
    1. If there something strange, we should fix the code that generated it.
    2. If that is not possible, the last resource is to fix the gir file directly, which is just XML, not manually but through a script using xmlstartlet. See fix.sh in gtk-rs as example.
  2. Create the toml file with the metadata required to create the bindings. In other words, this file contains the exceptions, rules and options used by the tool to generated the bindings. See Gir_GstGL.toml in gstreamer-rs-sys as example. The documentation of the toml file is in the gir’s README.md file.
$ ~/ws/gir/target/release/gir -c Gir_GstGL.toml

This command will generate, as specified in the toml file (target_path), a crate in the directory named gstreamer-gl-sys.

Api bindings.

These type of bindings may require more manual work since their purpose is to offer a rustified API of the library, with all its syntactic sugar, semantics, and so on. But in general terms, the process is similar:

$ cd ~/ws
$ git clone https://gitlab.freedesktop.org/gstreamer/gstreamer-sys.git
$ cd gstreamer-sys
$ cp /usr/share/gir-1.0/GstGL-1.0.gir gir-files/

Again, it would be possible to end up applying fixes to the gir file through a fix.sh script using xmlstartlet.

And again, the confection of the toml file might take a lot of time, by trial and error, by cleaning and tidying up the API. See Gir_GstGL.toml in gstreamer-rs as example.

$ ~/ws/gir/target/release/gir -c Gir_GstGL.toml

A good way to test your bindings is by crafting a test application, which shows how to use the API. Personally I devoted a ton of time in the test application for GstGL, but worth it. It made me aware of a missing part in the crate used for GL applications in Rust, named Glutin, which was a way to get the used EGLDisplay. So also worked on that and sent a pull request that was recently merged. The sweets of the free software development.

Nowadays I’m integrating GstGL API in servo/media and later, Servo!

Building gst-msdk with MediaSDK opensource

I tried, several months ago, the open source version of Intel MediaSDK and it was a complete mess. In order to review some patches for gst-msdk I tried it again. I am surprised how the situation has improved since then.

Install dependencies

$ sudo apt get install libva-dev vainfo cmake ccache
$ sudo apt build-dep gstreamer1.0 gst-plugins-{base,good,bad}1.0
$ sudo apt remove libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev

Seting up the workplace

$ sudo mkdir /opt/intel
$ sudo chown usuario:usuario /opt/intel
$ mkdir ~/msdk
$ cd ~/msdk

Build MediaSDK

It will be built in its source directory: ~/msdk/MediaSDK/build

It will be installed in /opt/intel

$ git clone https://github.com/Intel-Media-SDK/MediaSDK.git
$ cd MediaSDK
$ mkdir build
$ cd build
$ cmake ..
$ make
$ make install

Build media-driver

$ cd ~/msdk
$ git clone https://github.com/intel/media-driver.git
$ git clone https://github.com/intel/gmmlib.git
$ mkdir build
$ cd build
$ cmake ../media-driver
$ make

Let’s install media-driver in /opt/intel too

$ cd ~/msdk/build
$ cp ./media_driver/iHD_drv_video.so /opt/intel

But don’t remove, rename or move the directori ~/msdk/build because iHD_drv_video.so links against libigdgmm.so.5 which is there. Thus either you keep the directory or you install that library in a path searchable by the linker, or set the environment variable LD_LIBRARY_PATH

Test environment

$ LIBVA_DRIVERS_PATH=/opt/intel LIBVA_DRIVER_NAME=iHD vainfo
  libva info: VA-API version 1.3.0
  libva info: va_getDriverName() returns -1
  libva info: User requested driver 'iHD'
  libva info: Trying to open /opt/intel/iHD_drv_video.so
  libva info: Found init function __vaDriverInit_1_3
  libva info: va_openDriver() returns 0
 vainfo: VA-API version: 1.3 (libva 2.2.0)
 vainfo: Driver version: Intel iHD driver - 1.0.0
 vainfo: Supported profile and entrypoints
   VAProfileNone                   : VAEntrypointVideoProc
   VAProfileNone                   : VAEntrypointStats
   VAProfileMPEG2Simple            : VAEntrypointVLD
   VAProfileMPEG2Simple            : VAEntrypointEncSlice
   VAProfileMPEG2Main              : VAEntrypointVLD
   VAProfileMPEG2Main              : VAEntrypointEncSlice
   VAProfileH264Main               : VAEntrypointVLD
   VAProfileH264Main               : VAEntrypointEncSlice
   VAProfileH264Main               : VAEntrypointFEI
   VAProfileH264Main               : VAEntrypointEncSliceLP
   VAProfileH264High               : VAEntrypointVLD
   VAProfileH264High               : VAEntrypointEncSlice
   VAProfileH264High               : VAEntrypointFEI
   VAProfileH264High               : VAEntrypointEncSliceLP
   VAProfileVC1Simple              : VAEntrypointVLD
   VAProfileVC1Main                : VAEntrypointVLD
   VAProfileVC1Advanced            : VAEntrypointVLD
   VAProfileJPEGBaseline           : VAEntrypointVLD
   VAProfileJPEGBaseline           : VAEntrypointEncPicture
   VAProfileH264ConstrainedBaseline: VAEntrypointVLD
   VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
   VAProfileH264ConstrainedBaseline: VAEntrypointFEI
   VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP
   VAProfileVP8Version0_3          : VAEntrypointVLD
   VAProfileHEVCMain               : VAEntrypointVLD
   VAProfileHEVCMain               : VAEntrypointEncSlice
   VAProfileHEVCMain               : VAEntrypointFEI

Setup gst-build

It will be build in its source directory: ~/msdk/gst-build/build

$ cd ~/msdk
$ git clone https://gitlab.freedesktop.org/gstreamer/gst-build.git
$ cd gst-build
$ export INTELMEDIASDKROOT=/opt/intel/mediasdk
$ meson build -Dpython=disabled -Dgst-plugins-bad:msdk=enabled
$ ninja -C build

Check for build elements

$ ninja -C ~/msdk/gst-build/build  uninstalled
[gst-master] $ GST_VAAPI_ALL_DRIVERS=1 \
               LIBVA_DRIVERS_PATH=/opt/intel \
               LIBVA_DRIVER_NAME=iHD \
               gst-inspect-1.0 | egrep "vaapi|msdk"
 vaapi:  vaapijpegdec: VA-API JPEG decoder
vaapi:  vaapimpeg2dec: VA-API MPEG2 decoder
vaapi:  vaapih264dec: VA-API H264 decoder
vaapi:  vaapivc1dec: VA-API VC1 decoder
vaapi:  vaapivp8dec: VA-API VP8 decoder
vaapi:  vaapih265dec: VA-API H265 decoder
vaapi:  vaapipostproc: VA-API video postprocessing
vaapi:  vaapidecodebin: VA-API Decode Bin
vaapi:  vaapisink: VA-API sink
vaapi:  vaapimpeg2enc: VA-API MPEG-2 encoder
vaapi:  vaapih265enc: VA-API H265 encoder
vaapi:  vaapijpegenc: VA-API JPEG encoder
vaapi:  vaapih264enc: VA-API H264 encoder
msdk:  msdkvpp: MSDK Video Postprocessor
msdk:  msdkvc1dec: Intel MSDK VC1 decoder
msdk:  msdkvp8enc: Intel MSDK VP8 encoder
msdk:  msdkvp8dec: Intel MSDK VP8 decoder
msdk:  msdkmpeg2enc: Intel MSDK MPEG2 encoder
msdk:  msdkmpeg2dec: Intel MSDK MPEG2 decoder
msdk:  msdkmjpegenc: Intel MSDK MJPEG encoder
msdk:  msdkmjpegdec: Intel MSDK MJPEG decoder
msdk:  msdkh265enc: Intel MSDK H265 encoder
msdk:  msdkh265dec: Intel MSDK H265 decoder
msdk:  msdkh264enc: Intel MSDK H264 encoder
msdk:  msdkh264dec: Intel MSDK H264 decoder

Rembember

Remember to export these environment variables (perhaps you what to create a script file to set them):

export GST_VAAPI_ALL_DRIVERS=1
export LIBVA_DRIVERS_PATH=/opt/intel
 export LIBVA_DRIVER_NAME=iHD

How to setup a gst-build environment with Intel’s VA-API stack

gst-build is far superior than gst-uninstalled scripts for developing GStreamer, mainly because its meson and ninja usage. Nonetheless, to integrate external dependencies it is not as easy as in gst-uninstalled.

This guide aims to show how to integrate GStreamer-VAAPI dependencies, in this case with the Intel VA-API driver.

Dependencies

For now we will need meson master, since it this patch is required. The pull request is already merged but it is unreleased yet.

Installation

clone gst-build repository

$ git clone git://anongit.freedesktop.org/gstreamer/gst-build
$ cd gst-build

apply custom patch for gst-build

The patch will add the repositories for libva and intel-vaapi-driver.

$ wget https://people.igalia.com/vjaquez/gst-build-vaapi/0001-add-libva-and-intel-vaapi-driver-as-subprojects.patch
$ git am 0001-add-libva-and-intel-vaapi-driver-as-subprojects.patch

0001-add-libva-and-intel-vaapi-driver-as-subprojects.patch

configure

Running this command, all dependency repositories will be cloned, symbolic links created, and the build directory configured.

$ meson bulid

apply custom patches for libva, intel-vaapi-driver and gstreamer-vaapi

libva

This patch is required since the headers files uninstalled paths doesn’t match with the ones in the “include” directives.

$ cd libva
$ wget https://people.igalia.com/vjaquez/gst-build-vaapi/0001-build-add-headers-for-uninstalled-setup.patch
$ git am 0001-build-add-headers-for-uninstalled-setup.patch
$ cd -

0001-build-add-headers-for-uninstalled-setup.patch

intel-vaapi-driver

The patch handles libva dependency as a subproject.

$ cd intel-vaapi-driver
$ wget https://people.igalia.com/vjaquez/gst-build-vaapi/0001-meson-support-libva-as-subproject.patch
$ git am 0001-meson-support-libva-as-subproject.patch
$ cd -

0001-meson-support-libva-as-subproject.patch

gstreamer-vaapi

Note to myself: this patch must be split and merged in upstream.

$ cd gstreamer-vaapi
$ wget https://people.igalia.com/vjaquez/gst-build-vaapi/0001-build-meson-libva-gst-uninstall-friendly.patch
$ git am 0001-build-meson-libva-gst-uninstall-friendly.patch
$ cd -

0001-build-meson-libva-gst-uninstall-friendly.patch updated: 2018/04/24

build

$ ninja -C build

And wait a couple minutes.

run uninstalled environment for testing

$ ninja -C build uninstalled
[gst-master] $ gst-inspect-1.0 vaapi
Plugin Details:
  Name                     vaapi
  Description              VA-API based elements
  Filename                 /opt/gst/gst-build/build/subprojects/gstreamer-vaapi/gst/vaapi/libgstvaapi.so
  Version                  1.15.0.1
  License                  LGPL
  Source module            gstreamer-vaapi
  Binary package           gstreamer-vaapi
  Origin URL               http://bugzilla.gnome.org/enter_bug.cgi?product=GStreamer

  vaapih264enc: VA-API H264 encoder
  vaapimpeg2enc: VA-API MPEG-2 encoder
  vaapisink: VA-API sink
  vaapidecodebin: VA-API Decode Bin
  vaapipostproc: VA-API video postprocessing
  vaapivc1dec: VA-API VC1 decoder
  vaapih264dec: VA-API H264 decoder
  vaapimpeg2dec: VA-API MPEG2 decoder
  vaapijpegdec: VA-API JPEG decoder

  9 features:
  +-- 9 elements

GStreamer VA-API Troubleshooting

GStreamer VA-API is not a trivial piece of software. Even though, in my opinion it is a bit over-engineered, the complexity relies on its layered architecture: the user must troubleshoot in which layer is the failure.

So, bear in mind this architecture:

GStreamer VA-API is not a trivial piece of software. Even though, in my opinion it is a bit over-engineered, the complexity relies on its layered architecture: the user must troubleshoot in which layer is the failure.

So, bear in mind this architecture:

libva architecture
libva architecture

And the point of failure could be anywhere.

Drivers

libva is a library designed to load another library called driver or back-end. This driver is responsible to talk with the kernel, windowing platform, memory handling library, or any other piece of software or hardware that actually will do the video processing.

There are many drivers in the wild. As it is an API aiming to stateless video processing, and the industry is moving towards that way to process video, it is expected more drivers would appear in the future.

Nonetheless, not all the drivers have the same level of maturity, and some of them are abandon-ware. For this reason we decided in GStreamer VA-API, some time ago, to add a white list of functional drivers, basically, those developed by Mesa3D and this one from Intel™. If you wish to disable that white-list, you can do it by setting an environment variable:

$ export GST_VAAPI_ALL_DRIVERS=1

Remember, if you set it, you are on your own, since we do not trust on the maturity of that driver yet.

Internal libva↔driver version

Thus, there is an internal API between libva and the driver and it is versioned, meaning that the internal API version of the installed libva library must match with the internal API exposed by the driver. One of the causes that libva could not initialize a driver could be because the internal API version does not match.

Drivers path and driver name

By default there is a path where libva looks for drivers to load. That path is defined at compilation time. Following Debian’s file-system hierarchy standard (FHS) it should be set by distributions in /usr/lib/x86_64-linux-gnu/dri/. But the user can control this path with an environment variable:

$ export LIBVA_DRIVERS_PATH=${HOME}/src/intel-vaapi-driver/src/.libs

The driver path, as a directory, might contain several drivers. libva will try to guess the correct one by querying the instantiated VA display (which could be either KMS/DRM, Wayland, Android or X11). If the user instantiates a VA display different of his running environment, the guess will be erroneous, the library loading will fail.

Although, there is a way for the user to set the driver’s name too. Again, by setting an environment variable:

$ export LIBVA_DRIVER_NAME=iHD

With this setting, libva will try to load iHD_drv_video.so (a new and experimental open source driver from Intel™, targeted for MediaSDK —do not use it yet with GStreamer VAAPI—).

vainfo

vainfo is the diagnostic tool for VA-API. In a couple words, it will iterate on a list of VA displays, in try-and-error strategy, and try to initialize VA. In case of success, vainfo will report the driver signature, and it will query the driver for the available profiles and entry-points.

For example, my skylake board for development will report

$ vainfo
error: can't connect to X server!
libva info: VA-API version 1.1.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /home/vjaquez/gst/master/intel-vaapi-driver/src/.libs/i965_drv_video.so
libva info: Found init function __vaDriverInit_1_1
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.1 (libva 2.1.1.pre1)
vainfo: Driver version: Intel i965 driver for Intel(R) Skylake - 2.1.1.pre1 (2.1.0-41-g99c3748)
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Simple            : VAEntrypointEncSlice
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointEncSlice
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP
      VAProfileH264ConstrainedBaseline: VAEntrypointFEI
      VAProfileH264ConstrainedBaseline: VAEntrypointStats
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointEncSlice
      VAProfileH264Main               : VAEntrypointEncSliceLP
      VAProfileH264Main               : VAEntrypointFEI
      VAProfileH264Main               : VAEntrypointStats
      VAProfileH264High               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointEncSlice
      VAProfileH264High               : VAEntrypointEncSliceLP
      VAProfileH264High               : VAEntrypointFEI
      VAProfileH264High               : VAEntrypointStats
      VAProfileH264MultiviewHigh      : VAEntrypointVLD
      VAProfileH264MultiviewHigh      : VAEntrypointEncSlice
      VAProfileH264StereoHigh         : VAEntrypointVLD
      VAProfileH264StereoHigh         : VAEntrypointEncSlice
      VAProfileVC1Simple              : VAEntrypointVLD
      VAProfileVC1Main                : VAEntrypointVLD
      VAProfileVC1Advanced            : VAEntrypointVLD
      VAProfileNone                   : VAEntrypointVideoProc
      VAProfileJPEGBaseline           : VAEntrypointVLD
      VAProfileJPEGBaseline           : VAEntrypointEncPicture
      VAProfileVP8Version0_3          : VAEntrypointVLD
      VAProfileVP8Version0_3          : VAEntrypointEncSlice
      VAProfileHEVCMain               : VAEntrypointVLD
      VAProfileHEVCMain               : VAEntrypointEncSlice

And my AMD board with stable packages replies:

$ vainfo
libva info: VA-API version 0.40.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib64/dri/radeonsi_drv_video.so
libva info: Found init function __vaDriverInit_0_40
libva info: va_openDriver() returns 0
vainfo: VA-API version: 0.40 (libva )
vainfo: Driver version: mesa gallium vaapi
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileVC1Simple              : VAEntrypointVLD
      VAProfileVC1Main                : VAEntrypointVLD
      VAProfileVC1Advanced            : VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointEncSlice
      VAProfileH264High               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointEncSlice
      VAProfileNone                   : VAEntrypointVideoProc

Does this mean that VA-API processes video? No. It means that there is an usable VA display which could open a driver correctly and libva can extract symbols from it.

I would like to mention another tool, not official, but I like it a lot, since it extracts almost of the VA information available in the driver: vadumpcaps.c, written by Mark Thompson.

GStreamer VA-API registration

When GStreamer is launched, normally it will register all the available plugins and plugin features (elements, device providers, etc.). All that data is cache and keep until the cache file is deleted or the cache invalidated by some event.

At registration time, GStreamer VA-API will instantiate a DRM-based VA display, which works with no need of a real display (in other words, headless), and will query the driver for the profiles and entry-points tuples, in order to register only the available elements (encoders, decoders. sink, post-processor). If the DRM VA display fails, a list of VA displays will be tried.

In the case that libva could not load any driver, or the driver is not in the white-list, GStreamer VA-API will not register any element. Otherwise gst-inspect-1.0 will show the registered elements:

$ gst-inspect-1.0 vaapi
Plugin Details:
  Name                     vaapi
  Description              VA-API based elements
  Filename                 /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstvaapi.so
  Version                  1.12.4
  License                  LGPL
  Source module            gstreamer-vaapi
  Source release date      2017-12-07
  Binary package           gstreamer-vaapi
  Origin URL               http://bugzilla.gnome.org/enter_bug.cgi?product=GStreamer

  vaapijpegdec: VA-API JPEG decoder
  vaapimpeg2dec: VA-API MPEG2 decoder
  vaapih264dec: VA-API H264 decoder
  vaapivc1dec: VA-API VC1 decoder
  vaapivp8dec: VA-API VP8 decoder
  vaapih265dec: VA-API H265 decoder
  vaapipostproc: VA-API video postprocessing
  vaapidecodebin: VA-API Decode Bin
  vaapisink: VA-API sink
  vaapimpeg2enc: VA-API MPEG-2 encoder
  vaapih265enc: VA-API H265 encoder
  vaapijpegenc: VA-API JPEG encoder
  vaapih264enc: VA-API H264 encoder

  13 features:
  +-- 13 elements

Beside the normal behavior, GStreamer VA-API will also invalidate GStreamer’s cache at every boot, or when any of the mentioned environment variables change.

Conclusion

A simple task list to review when GStreamer VA-API is not working at all is this:

#. Check your LIBVA_* environment variables
#. Verify that vainfo returns sensible information
#. Invalidate GStreamer’s cache (or just delete the file)
#. Check the output of gst-inspect-1.0 vaapi

And, if you decide to file a bug in bugzilla, please do not forget to attach the output of vainfo and the logs if the developer asks for them.