GStreamer VA-API: A new release!
A new release of gstreamer-vaapi is now available!! Since we have been working on it for the last months, I would like talk you about it.gstreamer-vaapi is a set of GStreamer plugins and libraries for hardware accelerated video processing using VA-API.
VA-API stands for “Video Acceleration - Application Programming Interface”, and is, simultaneously, a specification of an application programming interface, and its library implementation under the Open Source MIT license, whose purpose is to offer applications access to the GPU accelerated video processing capabilities, offloading those tasks from the CPU. Accelerated processing includes video decoding, sub-picture blending and rendering.
The library implementation (libva) is designed to be a front-end for many different back-ends. I am aware only of these back-ends:
As you can notice, only the Intel driver is well maintained, meanwhile the others haven’t been updated since several years ago. And of course, this issue has impacted on the bugs handled in gstreamer-vaapi (for example, bug #749554).
VA-API is designed to use various entry-points of the hardware accelerated processing in the GPU driver, such as VLD (also known as slice level acceleration), iDCT, among others.
The fact that VA-API uses slice level decode is important, since it is different from the software-based video decoders, were the decoding is at frame level. As consequence, more information is required from the current video parsers (bug #691712).
But let us back on track. VA-API, as we already said, is a set of GStreamer elements (vaapidecode, vaapipostroc, vaapisink, and several encoders) and a GObject-friendly library named libgstvaapi. This library wraps libva under a GObject/GStreamer semantics. I might talk about this library in another opportunity.
vaapisink is video sink with support for multiple display server protocols:
UPDATE (2015/07/20): As Sree commented, vaapisink does not use GLX/EGL for rendering, it uses VA-API, but rather vaapidecode/vaapipostproc can deal with GLX/EGL contexts so they can share buffers with other OpenGL-based elements.
This element is the most efficient way to render the video processed by other VA-API elements such as decoders and post-processors, since no media transformations are required. If we want to render the stream in other video sinks, some extra operations might be necessary. The worst case scenario is where we need to download the whole image from the GPU memory onto the CPU one, and the overall performance of the pipeline might be impacted.
vaapidecode is single decoding element that handles several codecs, depending on the back-end and the hardware available. The comprehensive list of possible enabled codecs is:
- MPEG-2 (simple and main profiles)
- MPEG-4 (simple, main and advance-simple profiles) / DivX / Xvid
- H263 (baseline profiles)
- H264 (baseline, constrained-baseline, main, high, multiview-high and stereo-high profiles)
- H265 (main and main10)
- WMV3 (simple and main profiles) / VC1
- VP8 (version 0.3)
- JPEG (baseline profile)
In the particular case of Intel chip-sets, the next table shows the codec support per each generation:
| CHIPSET | MPEG2 | H264 | H265 | VC1 | VP8 | JPEG |
|---|---|---|---|---|---|---|
| Skylake | Y | Y | Y | Y | Y | Y |
| Cherry View | Y | Y | Y | Y | Y | Y |
| Broadwell | Y | Y | Y | Y | ||
| Haswell | Y | Y | Y | Y | ||
| Ivybridge | Y | Y | Y | Y | ||
| Sandybridge | Y | Y | Y | |||
| Ironlake | Y | Y | ||||
| G4x | Y |
In the case of H264, there are multiple profiles, and every chip-set offers different profile sets. But for now, I will not dig in more on it.
In other back-ends the support shall be different. For example, I have a box with an GPU NVidia GeForce GT which supports MPEG2, MPEG4, H264 and VC1.
There is a branch for G45 GPU, where H264 decoding support is added. Nevertheless, is not official, it doesn’t merge with current master anymore, and the supported output color space is NV12 only (bug #745660).
An interesting thing to know is because of the slice level decoding, gstreamer-vaapi does its own parsing inside the decoder, even if a video parser was plugged before.
vaapipostproc is a video transform element which can do color space
conversions, de-interlacing,
sharpening and many other types of video filtering, such as:
- Color conversion
- Resize / Scale
- Noise reduction
- De-interlacing
- Sharpening
- Color Balance
- Skin tone enhancement
Also, the availability of this elements and its filters, depends on the back-end and the hardware. For example, the VDPAU back-end doesn’t provide any post-processing capability.
In the case of Intel’s chip-sets these are the provided post-processing capabilities:
| CHIPSET | Format | Noise | Deinterlace | Sharp | C.B. | STE |
|---|---|---|---|---|---|---|
| Skylake | Y | Y | Y | Y | Y | Y |
| Cherry View | Y | Y | Y | Y | Y | Y |
| Broadwell | Y | Y | Y | Y | Y | Y |
| Haswell | Y | Y | Y | Y | Y | Y |
| Ivybridge | Y | Y | Y | |||
| Sandybridge | Y | Y | Y | |||
| Ironlake | Y | |||||
| G4x |
In the case of the video encoders, they are split in different elements. Those implemented in gstreamer-vaapi are:
- vaapiencode_mpeg2
- vaapiencode_h264
- vaapiencode_h265
- vaapiencode_vp8
- vaapiencode_jpeg
As far as I know, the only VA-API back-end that provides encoder is the Intel one, and this is the table of the encoders per chip-set:
| CHIPSET | MPEG2 | H264 | H265 | VP8 | JPEG |
|---|---|---|---|---|---|
| Skylake | Y | Y | Y | Y | Y |
| Cherry View | Y | Y | Y | ||
| Broadwell | Y | Y | |||
| Haswell | Y | Y | |||
| Ivybridge | Y | Y | |||
| Sandybridge | Y | ||||
| Ironlake | |||||
| G4x |
vaapidecodebin is a new bin composed by vaapidecode, a queue and
vaapipostproc. Its purpose is bundle a complete decoding solution, with
de-interlacing support. The purpose of the queue is to set the decoder to its
full speed. The problem is, that since each back-end and chip-set might or might
not support vaapipostproc we have to check for it in run-time (bug
#749554).
(-------------------------------------------------------)
| vaapidecodebin |
| (-------------) (-------) (---------------) |
|-->| vaapidecode |--->| queue |--->| vaapipostproc |-->|
| (-------------) (-------) (---------------) |
| |
(-------------------------------------------------------)In my opinion, this should be just a temporal workaround meanwhile we have auto-plugging support of de-interlacers (bug #687182).
So far we have explained what is GStreamer VA-API. Let us now talk about what is new and hot in this release.
This is the short log summary since the last release, 0.5.10:
1 Adrian Cox
2 Alban Browaeys
30 Gwenole Beauchesne
1 Jacobo Aragunde Pérez
2 Jan Schmidt
1 Julien Isorce
1 Lim Siew Hoon
1 Martin Sherburn
4 Michael Olbrich
12 Olivier Crete
3 Simon Farnsworth
74 Sreerenj Balachandran
75 Víctor Manuel Jáquez Leal
1 Wind YuanThe major changes in this release includes:
- HEVC (H265) decoding and encoding support (available only in Skylake and Cherry View chip-sets)
- VP8 encoder (Skylake)
- JPEG encoder (Skylake and Cherry View)
- vaapidecodebin element, which we have talked about above.
- Support for EGL either in vaapidecode, vaapipostproc and vaapisink.
- Skin tone enhancement support in vaapipostproc
- Support for H.264 Multiview High profile encoding with more than 2 views, which encompasses Jan Schmidt’s efforts for stereoscopic / multi-view support in GStreamer (bug #611157)
- A lot of improvements and round rough corners all over the place. Only to mention that we closed more than 70 bug reports along this cycle
Now, please test this new release, enjoy it, and if you find something ugly, let us known!!!
- Previous: GStreamer Hackfest 2015
- Next: GStreamer Conference 2015