Dabrain34's igalian blog

Mulimedia engineer and free-software enthousiast

Vulkan video with NVK driver

This blog post might interest you if you want to try the bleeding edge NVK driver which allows to decode H264/5 video with the power of the Vulkan extensions VK_KHR_video_decode_h26x.

This is a summary of the instructions provided in the MR. This work needs a recent kernel with new features, so it will describe the steps to add this feature and build this new kernel on an Ubuntu based system.

To accomplish that what you would need, is a recent NVIDIA hardware such as GeForce RTX 4060, a Linux kernel greater than 6.12.8 and the mesa branch supporting Vulkan Video using NVK: https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31867

Lets start by the root, the kernel.

Kernel build #

To run the NVK driver, you need a custom patch to be applied on top of the Nouveau driver. This patch applies on minimum 6.12 kernel so you need to build a new kernel except if your distribution is running bleeding edge kernel which i doubt so here is my method I used to build this kernel.

I mainly got my inspiration from https://askubuntu.com/questions/718381/how-to-compile-and-install-custom-mainline-kernel but I had to adapt it a bit to make it work.

All of this blog post has been performed on a regular ubuntu 24.04.

First you need to install the following ubuntu package:

$ sudo apt install -y git build-essential kernel-package fakeroot libncurses5-dev libssl-dev ccache

You’ll have to download a recent kernel. I ended up using a git clone as it was easier than a tarball to build all necessary bits.

$ git clone --branch v6.12.8 --depth 1 git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git linux-stable
$ cd linux-stable

Then you’ll have to apply the given patch to your work tree:

diff --git a/drivers/gpu/drm/nouveau/nouveau_abi16.c b/drivers/gpu/drm/nouveau/nouveau_abi16.c
index 2a0617e5fe2a..eaef5effbb3d 100644
--- a/drivers/gpu/drm/nouveau/nouveau_abi16.c
+++ b/drivers/gpu/drm/nouveau/nouveau_abi16.c
@@ -379,6 +379,9 @@ nouveau_abi16_ioctl_channel_alloc(ABI16_IOCTL_ARGS)
 			case NOUVEAU_FIFO_ENGINE_CE:
 				engine = NV_DEVICE_HOST_RUNLIST_ENGINES_CE;
 				break;
+			case 0x300:
+				engine = NV_DEVICE_HOST_RUNLIST_ENGINES_NVDEC;
+				break;
 			default:
 				return nouveau_abi16_put(abi16, -ENOSYS);
 			}
@@ -456,10 +459,12 @@ nouveau_abi16_ioctl_channel_alloc(ABI16_IOCTL_ARGS)
 			goto done;
 		break;
 	case NV_DEVICE_INFO_V0_TURING:
-		ret = nvif_object_ctor(&chan->chan->user, "abi16CeWar", 0, TURING_DMA_COPY_A,
-				       NULL, 0, &chan->ce);
-		if (ret)
-			goto done;
+		if (engine != NV_DEVICE_HOST_RUNLIST_ENGINES_NVDEC) {
+			ret = nvif_object_ctor(&chan->chan->user, "abi16CeWar", 0, TURING_DMA_COPY_A,
+					       NULL, 0, &chan->ce);
+			if (ret)
+				goto done;
+		}
 		break;
 	default:
 		break;

Next step will be to configure the kernel. The best option I’ll recommend you is to copy the kernel config, your distribution is shipping with. On Ubuntu you can find it in /boot with the name config-6.8.0-52-generic for example.

$ cp /boot/config-6.8.0-52-generic .config

Then to get the default config, your kernel will use, including the specific options coming with Ubuntu, you’ll have to run:

$ make defconfig

This will setup the build and make it ready to compile with this version of the kernel, auto configuring the new features.

Two options CONFIG_SYSTEM_TRUSTED_KEYS and CONFIG_SYSTEM_REVOCATION_KEYS must be disabled to avoid compilation errors with missing certificates. For that you can set it up within menuconfig or you can edit .config and set these values to ""

Then you should be ready to go for a break ☕, short or long depending on your machine to cook the brand new kernel debian packaged, ready to use:

$ make clean
$ make -j `getconf _NPROCESSORS_ONLN` deb-pkg LOCALVERSION=-custom

The process should end up with a new package named linux-image-6.12.8-custom_6.12.8-3_amd64.deb in the upper folder which can then be installed along your previous kernel.

$ sudo dpkg -i ../linux-image-6.12.8-custom_6.12.8-3_amd64.deb

The first one will replace your current default menulist item in grub upon installation. This means that if you install it, next time you reboot, you’ll boot into that kernel.

Mesa build for NVK #

Now that we have a working kernel which allows to discuss with nvidia codec engine using nouveau driver. We can start the build of mesa.

Prerequisites #

Mesa depends on various system packages in addition to python modules and the the rust toolchain. So first we’ll have to install the given package which are all present in Ubuntu 24.04:

$ sudo apt install build-essential ninja-build glslang-tools python3-pip llvm byacc flex pkg-config bindgen libclc-19-dev libllvmspirvlib-18-dev libdrm-dev libclang-cpp-dev libclang-18-dev

$ sudo apt install libelf-dev libwayland-bin libwayland-dev libwayland-egl-backend-dev libx11-xcb-dev libxcb-randr0-dev libxfixes-dev libxcb-glx0-dev libxcb-shm0-dev libxcb-dri3-dev

$ sudo apt install libxcb-present-dev libxshmfence-dev libxxf86vm-dev

Then you’ll have to install the rust toolchain and some cargo package:

$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
$ cargo install bindgen-cli
$ cargo install cbindgen

When all the system package are installed, you can install the python modules needed by Mesa build system.

$ sudo pip3 install --break-system-packages meson mako pyyaml

Configure and build #

As Mesa is highly depending on the system, you might need some additional packages but worth a try to launch the build with the given instructions:

$ cd ~/DEV/
$ git clone https://gitlab.freedesktop.org/dwlsalmeida/mesa.git --branch=nvk-vulkan-video
$ meson setup builddir --libdir lib64 --prefix `pwd`/builddir/install -Dbuildtype=debug -Dvulkan-drivers=nouveau -Dgallium-drivers=nouveau -Dtools=nouveau
$ ninja -C builddir install

Use of NVK driver #

Now that the kernel and the Mesa driver have been built and are available for your machine, you should be able to decode your first h264 stream with the NVK driver.

Important: Nouveau driver might be disabled !! #

As you might have use the Nvidia driver first and installed with your regular kernel, you might hit a weired error when invoking vulkaninfo such as:

ERROR: [Loader Message] Code 0 : setup_loader_term_phys_devs:  Failed to detect any valid GPUs in the current config
ERROR at ./vulkaninfo/./vulkaninfo.h:247:vkEnumeratePhysicalDevices failed with ERROR_INITIALIZATION_FAILED

Indeed nouveau driver can not live along the Nvidia driver, so you’ll have to uninstall the Nvidia driver first to be able to use nouveau properly and the vulkan extensions. One other solution is to boot on your new custom kernel and modify the file /etc/modprobe.d/nvidia-installer-disable-nouveau.conf to get something like:

# generated by nvidia-installer
#blacklist nouveau
options nouveau modeset=1

In that case the modeset=1 option will enable the driver and allow to use it. Then you’ll have to reboot with this new configuration

Run vulkaninfo #

As you may have noticed, during the configure stage that we chose to install the artifacts of the build in a folder named mesa/builddir/install.

Here is a script run-nvk.sh which you can use before calling any binary which will use this folder as a base to set the environment variable dedicated to the NVK Vulkan driver

#!/bin/sh
MESA=$HOME/DEV/mesa/builddir/install \
LD_LIBRARY_PATH=$MESA/lib64:$MESA/lib:$LD_LIBRARY_PATH \
VK_ICD_FILENAMES=$MESA/share/vulkan/icd.d/nouveau_icd.x86_64.json \
VK_DRIVER_FILES=$MESA/share/vulkan/icd.d/nouveau_icd.x86_64.json \
NVK_I_WANT_A_BROKEN_VULKAN_DRIVER=true \
exec "$@"

After all these stage you should be able to run vulkaninfo such as:

$ ~/DEV/SCRIPTS/run-nvk.sh vulkaninfo | grep video_decode

	VK_KHR_video_decode_h264                    : extension revision 9
	VK_KHR_video_decode_h265                    : extension revision 8
	VK_KHR_video_decode_queue                   : extension revision 8

Run GStreamer playback #

Now its time to run a real application exploiting the power of Vulkan to decode multimedia content. For that I’ll recommend you to use GStreamer which ship with Vulkan elements for decoding in 1.24.2 version bundled in Ubuntu 24.04.

First of all, you’ll have to install the ubuntu packages for GStreamer

sudo apt install gstreamer1.0-plugins-bad gstreamer1.0-tools

Check that the elements are available with:

$ gst-inspect vulkan

  vulkancolorconvert: Vulkan Color Convert
  vulkandeviceprovider: Vulkan Device Provider
  vulkandownload: Vulkan Downloader
  vulkanh264dec: Vulkan H.264 decoder
  vulkanh265dec: Vulkan H.265 decoder
  vulkanimageidentity: Vulkan Image Identity
  vulkanoverlaycompositor: Vulkan Overlay Compositor
  vulkanshaderspv: Vulkan Shader SPV
  vulkansink: Vulkan video sink
  vulkanupload: Vulkan Uploader
  vulkanviewconvert: Vulkan View Convert

If you succeed to see this list of elements, you should be able to run a GStreamer pipeline with Vulkan Video extensioms. Here is a pipeline to decode a content:

$ ~/DEV/SCRIPTS/run-nvk.sh gst-launch-1.0 urisourcebin uri=https://test-videos.co.uk/vids/bigbuckbunny/mp4/h264/1080/Big_Buck_Bunny_1080_10s_1MB.mp4 ! parsebin ! vulkanh264dec ! vulkandownload ! autovideosink

Hope you enjoyed this blog post and you are now able to decode h26x content with the open Vulkan Video extensions.

Thanks a lot to the work of David Airlie and Daniel Almeida who allow to decode h26x content with NVK driver.