Video decoding in GStreamer with Vulkan Video extension (part 2)
Its has been a while since I reported my tinkering with the Vulkan Video provisional extension. Now the specification will have its final release soon_ish_, and also there has been more engagement within the open source communities, such as the work-in-progress FFmpeg implementation by Lynne (please, please, read that post), and the also work-in-progress Mesa 3D drivers both for AMD and Intel by Dave Airlie! Along with the well known NVIDIA beta drivers for Vulkan.From our side, we have been trying to provide an open source alternative to the video parser used by the Conformance Test Suite and the NVIDIA vk_video_samples, using GStreamer: GstVkVideoParser, which intends to be a drop-in replacement of the current proprietary parser library.
Along the way, we have sketched the Vulkan Video support in gfxreconstruct, for getting traces of the API usage. Sadly, its kind of bit-rotten right now, even more because the specification has changed since then.
Regarding the H.264 decoder for GStreamer, we just restarted its hacking. The merge request was moved to monorepo, but for the sake of the well needed complete re-write, we changed the branch to this one (vkh264dec). We needed to re-write it because, besides the specification updates, we have learned many things along the journey, such as the out-of-band parameters update, Vulkan’s recommendation for memory pre-allocation as much as possible, the DPB/references handling, the debate about buffer vs. slice uploading, and other friction points that Lynne has spotted for future early adopters.
The way to compile it is grab the branch and compile as usually GStreamer is compiled with meson:
meson setup builddir -Dgst-plugins-bad:vulkan-video=enabled --buildtype=debug
ninja C builddirAnd run simple pipelines such as
gst-launch-1.0 filesrc location=INPUT ! parsebin ! vulkanh264dec ! fakesink -vOur objective is to have a functional demo for the next Vulkanised in February. We are very ambitious, we want it to work in Linux, Windows and in many GPU as possible. Wish us luck. And happy December festivities!