Vulkan Video Ecosystem Status #

This page intends to provide up-to-date links and information for the latest driver and application support for Khronos Vulkan Video extension. It gathers efforts from different persons, organizations and companies. Special thanks to Lynne (FFmpeg) and Dave Airlie (Mesa / RedHat) for all their work, along with Igalia (GStreamer / Mesa)

Last update: 2024-02-29

Drivers #

SDK and specification #

• Latest Vulkan SDK
• Vulkan Specification SDK 1.3.239 is the first to support the decode extensions for H.264 and H.265.
• Vulkan Specification SDK 1.3.275 is the first to support the encode extensions for H.264 and H.265.
• Vulkan Specification 1.3.277 is the first to support the decode extensions for AV1. (It’s not a SDK release).

Applications #

• GStreamer support for H.264 decode is merged and will be available in GStreamer 1.23.1

• GStreamer support for H.265 decode is merged and will be available in GStreamer 1.23.2

• 🚧 GStreamer support for H.264 and H.265 encoding MR

• FFmpeg 6.0.1 “Von Neumann” supports Vulkan accelerated decode for H.264 and H.265.

• 🚧 FFmpeg development branch for H.264 and H.265 encode.

• Vulkan Video support in mpv player

• Fluster is a testing framework written in Python for decoder conformance, and it includes the support to test GStreamer and FFmpeg decoding with Vulkan.

References #

• Dave Airlie Linux Graphics blog for the latest on his Mesa driver work.
• Lynne’s compiled musings on FFmpeg Vulkan Video development.
• Khronos Releases AV1 Decode in Vulkan Video with SDK Support for H.264/H.265 Encode February 1, 2024
• Experience report of the decode extension December 20, 2023.
• Khronos Finalizes Vulkan Video Extensions for Accelerated H.264 and H.265 Encode December 19, 2023.
• Vulkan Video encoder in GStreamer December 10, 2023.
• A helpful introduction from a game engine perspective May 7, 2023.
• Concise overview of video compression basics May 9, 2021.
• Khronos: An introduction to Vulkan Video April 13, 2021.

Presentations #

Vulkanised 2024: Implementing a Video Encoder From Mesa to GStreamer - Stéphane Cerveau & Hyunjun Ko

Vulkanised 2024: Vulkan Video Encode API: A Deep Dive - Tony Zlatinski

XDC 2023: Status of the Vulkan Video ecosystem. Charles Turner, Igalia

GStreamer Conference 2023: Vulkan Video in GStreamer. Stéphane Cerveau & Victor Jáquez, Igalia

Vulkanised 2023: A deep dive into Vulkan Video. Tony Zlatinski, NVIDIA

Vulkanised 2023: Implementing a Vulkan decoder with GStreamer. Victor Jáquez & Stéphane Cerveau, Igalia

XDC 2021: Video decoding in Vulkan: VK_KHR_video_queue/decode APIs. Víctor Jáquez, Igalia

Examples #

GStreamer #

GStreamer decode #

gst-launch-1.0 filesrc location=./video.mp4 ! parsebin ! vulkanh264dec ! vulkandownload ! videoconvert ! autovideosink

GStreamer encode #

gst-launch-1.0 videotestsrc ! video/x-raw, format=NV12 ! vulkanupload ! vulkanh264enc ! vulkanh264dec ! vulkandownload ! videoconvert ! autovideosink sync=false

FFmpeg #

• Setting vk:N allows you to select a specific device.
• Setting debug=1 turns on API validation.
• When dealing with unframed YUV data, there are addition switches to pass before the -i flag: e.g. -f rawvideo -pixel_format nv12 -video_size 352x240 -i ….

FFmpeg decode #

ffmpeg -hwaccel vulkan -init_hw_device vulkan=vk:0,debug=0 -hwaccel_output_format vulkan -i input -vf 'hwdownload,format=nv12' -pix_fmt yuv420p output.y4m

🚧 FFmpeg encode #

ffmpeg -y -hide_banner -hwaccel vulkan -hwaccel_output_format vulkan -i input.y4m -vf 'format=nv12,hwupload' -c:v h264_vulkan -frames 100 output_h264_vulkan.mp4