It was a year and half ago when I announced a new VA-API H.264 decoder element in gst-plugins-bad. And it was bundled in GStreamer release 1.18 a couple months later. Since then, we have been working adding more decoders and filters, fixing bugs, and enhancing its design. I wanted to publish this blog post as soon as release 1.20 was announced, but, since the developing window is closed, which means no more new features will be included, I’ll publish it now, to create buzz around the next GStreamer release.
Here’s the list of new GstVA decoders (of course, they are only available if your driver supports them):
Also, there are a couple new features in
vah264dec (common to all
gstcodecs-based H.264 decoders):
- Supports interlaced streams (
- Added a
complianceproperty to trick the specification conformance for lower the latency, for example, or to enable non-standard features.
But not only decoders, there are two new elements for post-processing:
vapostproc is similar to
vaapipostproc but without the interlace operation, since it was moved to another element. The reason for this is because there are deinterlacing methods which require to hold a list of referenced frames, thus these methods are broken in
vaapipostproc, and adding them would increase the complexity of the element with no need. To keep things simple it’s better to handle deinterlacing in a different element.
This is the list of filters and features supported by
- Color conversion
- Color balance (Intel only -so far-)
- Video direction (Intel only)
- Skin tone enhancement (Intel only)
- Denoise and Sharpen (Intel only)
And, I ought to say, HDR is in the pipeline, but it will be released after 1.20.
vadeinterlace only does that, deinterlacing. But it supports all the available methods currently in the VA-API specification, using the new way to select the field to extract, since the old one (used by GStreamer-VAAPI and FFMPEG) is a bit more expensive.
Finally, both video filters, if they cannot handle the income format, they are configured in passthrough mode.
But there are not only new elements, there’s also a new library!
Since many others elements need to share a common
VADisplay in the GStreamer pipeline, the new library expose only the
GstVaDisplay object by now. The new library must be thin and lean, exposing only what it’s requested by other elements, such as gst-msdk. We have pending to merge after 1.20, for example, the add of
GstContext helpers, and the plan is to expose the allocators and bufferpools later.
Another huge task are encoders. After the freeze, we’ll merge the first
implementation of the H.264 encoder, and add, in different iterations, more encoders.
As I said in the previous blog post, all these elements are ranked as none, so the won’t be autoplugged, for example by
playbin. To do so, users need to export the environment variable
GST_PLUGIN_FEATURE_RANK as documented.
$ GST_PLUGIN_FEATURE_RANK=vah264dec:MAX,vah265dec:MAX,vampeg2dec:MAX,vavp8dec:MAX,vavp9dec:MAX gst-play-1.0 stream.mp4
Thanks a bunch to He Junyan, Seungha Yang and Nicolas Dufresne, for all the effort and care.
Still, the to-do list is large enough. Just to share what I have in my notes:
- Add a new upload method in
gluploadto interop with VA surfaces — though this hardly will be merged since it creates a circular dependency between -base and -bad.
vavc1dec— it might need a rewrite of
vajpegdec— it needs a rewrite of
vaalphacombine— decoding alpha channel with VA within
vamixer— similar to
vaapioverlay, to compose a single frame from different video streams.
- And encoders (mainly H.264 and H.265).
As a final mode, GStreamer-VAAPI has enter into maintenance mode. The general plan, without any promise or dates, is to deprecate it when most of its use cases were covered by GstVA.