VVC/H.266 in GStreamer 1.26

23 April 2025
vvc
h266
multimedia
gstreamer

With the release of GStreamer 1.26, we now have playback support for Versatile Video Coding (VVC/H.266). In this post, I’ll describe the pieces of the puzzle that enable this, the contributions that led to it, and hopefully provide a useful guideline to adding a new video codec in GStreamer.

Playing back VVC with GStreamer #

With GStreamer 1.26 and the relevant plugins enabled, one can play multimedia files containing VVC content, for example, by using gst-play-1.0:

gst-play-1.0 vvc.mp4

By using gst-play-1.0, a pipeline using playbin3 will be created and the appropriate elements will be auto-plugged to decode and present the VVC content. Here’s what such a pipeline looks like:

Although the pipeline is quite large, the specific bits we’ll focus on in this blog are inside parsebin and decodebin3:

qtdemux → ... → h266parse → ... → avdec_h266

I’ll explain what each of those elements is doing in the next sections.

(De)muxing files with VVC video #

To store multiple kinds of media (e.g. video, audio and captions) in a way that keeps them synchronized, we typically make use of container formats. This process is usually called muxing, and in order to play back the file we perform de-muxing, which separates the streams again. That is what the qtdemux element is doing in the pipeline above, by extracting the audio and video streams from the input MP4 file and exposing them as the audio_0 and video_0 pads.

Support for muxing and demuxing VVC streams in container formats was added to:

qtmux and qtdemux: for ISOBMFF/QuickTime/MP4 files (often saved with the .mp4 extension)
matroskamux and matroskademux: for Matroska (MKV) files (often saved with .mkv extension)
mpegtsmux and tsdemux: for MPEG transport stream (MPEG-TS) files (often saved with the .ts extension)

Besides the fact that the demuxers are used for playback, by also adding support to VVC in the muxer elements we are then also able to perform remuxing: changing the container format without transcoding the underlying streams.

Some examples of simplified re-muxing pipelines (only taking into account the VVC video stream):

MP4 to MKV:

gst-launch-1.0 filesrc location=vvc.mp4 ! qtdemux ! matroskamux ! filesink location=vvc.mkv

MKV to MPEG-TS:

gst-launch-1.0 filesrc location=vvc.mkv ! matroskademux ! h266parse ! mpegtsmux ! filesink location=vvc.ts

But why do we need h266parse when re-muxing from Matroska to MPEG-TS? That’s what I’ll explain in the next section.

Parsing and converting between VVC bitstream formats #

Video codecs like H.264, H.265, H.266 and AV1 may have different stream formats, depending on which container format is used to transport them. For VVC specifically, there are two main variants, as shown in the caps for h266parse:

Pad Templates:
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-h266
  
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-h266
                 parsed: true
          stream-format: { (string)vvc1, (string)vvi1, (string)byte-stream }
              alignment: { (string)au, (string)nal }

byte-stream or so-called Annex-B format (as in Annex B from the VVC specification): it separates the NAL units by start code prefixes (0x000001 or 0x00000001), and is the format used in MPEG-TS, or also when storing VVC bitstreams in files without containers (so-called “raw bitstream files”).

ℹ️ Note: It’s also possible to play raw VVC bitstream files with gst-play-1.0. That is achieved by the typefind element detecting the input file as VVC and playbin taking care of auto-plugging the elements.

vvc1 and vvi1: those formats use length field prefixes before each NAL unit. The difference between the two formats is the way that parameter sets (e.g. SPS, PPS, VPS NALs) are stored, and reflected in the codec_data field in GStreamer caps. For vvc1, the parameter sets are stored as container-level metadata, while vvi1 allows for the parameter sets to be stored also in the video bitstream.

The alignment field in the caps signals whether h266parse will collect multiple NALs into an Access Unit (AU) for a single GstBuffer, where an AU is the smallest unit for a decodable video frame, or whether each buffer will carry only one NAL.

That explains why we needed the h266parse when converting from MKV to MPEG-TS: it’s converting from vvc1/vvi1 to byte-stream! So the gst-launch-1.0 command with more explicit caps would be:

gst-launch-1.0 filesrc location=vvc.mkv \
  ! matroskademux \
  ! video/x-h266,stream-format=vvc1 ! h266parse ! video/x-h266,stream-format=byte-stream \
  ! mpegtsmux ! filesink location=vvc.ts

The base library for parsing VVC streams and the h266parse element were contributed by Intel, with initial support only for the byte-stream format, and I added support for the vvc1 and vvi1 stream formats.

VVC decoder elements #

As of today, there are three VVC decoder implementations available in GStreamer: FFmpeg, VVdeC and VAAPI.

FFmpeg #

FFmpeg 7.1 has a native VVC decoder which is considered stable. In GStreamer 1.26, we have allowlisted that decoder in gst-libav, and it is now exposed as the avdec_h266 element.

VVdeC #

Another software decoder implementation is VVdeC. The vvdec element has been merged as part of gst-plugins-rs. To achieve that, I wrote safe Rust bindings for VVdeC, which is also published on crates.io.

VAAPI #

Intel has added the vah266dec element in GStreamer 1.26, which enables hardware-accelerated VVC decoding on Intel Lunar Luke CPUs. However, it still has rank of 0 in GStreamer 1.26, so in order to test it out, one would need to, for example, manually set GST_PLUGIN_FEATURE_RANK.

Similar to h266parse, initially vah266dec was added with support for only the byte-stream format. I implemented support for the vvc1 and vvi1 modes in the base h266decoder class, which fixes the support for them in vah266dec as well. However, it hasn’t yet been merged and I don’t expect it to be backported to 1.26, so likely it will only be available in GStreamer 1.28.

Here’s a quick demo of vah266dec in action on an ASUS ExpertBook P5. In this screencast, I perform the following actions:

Run vainfo and display the presence of VVC decoding profile
gst-inspect vah266dec
export GST_PLUGIN_FEATURE_RANK='vah266dec:max'
Start playback of six simultaneous 4K@60 DASH VVC streams. The stream in question is the classic Tears of Steel, sourced from the DVB VVC test streams.
Run nvtop, showing GPU video decoding & CPU usage per process.
Show pipeline dump via gst-dots-viewer.

Testing decoder conformance with Fluster #

A tool that is handy for testing the new decoder elements is Fluster. It simplifies the process of testing decoder conformance and comparing decoders by using test suites that are adopted by the industry. It’s worth checking it out, and it’s already common practice to test new decoders with this test framework. I added the GStreamer VVC decoders to it: vvdec, avdec_h266 and vah266dec.

TODO: Encoding VVC with GStreamer and VVenC #

We’re still missing the ability to encode VVC video in GStreamer. I have a work-in-progress branch that adds the vvenc element, by using VVenC and safe Rust bindings (similarly to the vvdec element), but it still needs some work. I intend to work on it during the GStreamer Spring Hackfest 2025 to make it ready to submit upstream 🤞

Carlos Bentzen's Blog

Multimedia, Web Engines and other random bits