Dabrain34's igalian blog

Introducing GstPrinceOfParser 0.4.3

2026-04-08T00:00:00Z

GstPrinceOfParser: An All-in-One Tool to Play With GStreamer on Any Platform #

Introducing gst-pop, the GStreamer Prince of Parser — a tool to make interaction with GStreamer easier, global, and remotely accessible.

What is GStreamer? #

GStreamer is an open-source multimedia framework started in 1999. It lets you build pipelines of interconnected elements to stream, encode, decode, and manipulate media. The core idea is simple: a source element produces data, passes it through one or more transform elements, and delivers it to a sink. For example, here is a pipeline that decodes an MP3 audio file:

    filesrc --> mp3dec --> audiosink

For more than 20 years, GStreamer has relied on its in-house toolbox to demonstrate the power of its pipelines. As this toolbox is used in thousands of projects and serves as a reference implementation, modifications and enhancements are deliberately kept minimal to maintain stability. gst-pop was created to go beyond these limitations.

A Unified Interface for GStreamer #

Accessible over the network, via CLI arguments, or through D-Bus, gst-pop aims to provide a multi-pipeline-capable command-line tool.

With a simple invocation of gst-pop (or its alias gst-popd), you can run a daemon that accepts multiple pipelines simultaneously, accessible through D-Bus or WebSocket via the pipeline ID. You’ll be able to control, query, and get information about each pipeline — all of that over a remote network, secured with API key authentication and origin validation to prevent unauthorized access.

As demonstrated in the blog post related to GstPipelineStudio, it will be possible to connect to a remote pipeline or launch new pipelines through the GStreamer GUI. If a GUI is not available on the platform, it will soon be possible to use a web interface to control GStreamer, offering everything GStreamer can provide and more, limited only by your imagination.

Remote Element Inspection #

gst-pop (or its alias gst-pop-inspect) is also capable of listing the elements on a local or remote host, inspecting their capabilities, and providing a remote way to interact with your GStreamer installation.

Media Discovery #

It can also provide information on a media file using GStreamer’s discovery interface using gst-pop-discovery, offering an easy and remote-capable media discovery system for your setup.

Playback #

And of course, it can serve as an alternative to the gst-play tool, with gst-pop-play, allowing you to instantiate as many playback sessions as you need, with the ability to use any sink you want.

The possibilities are vast: provide multimedia services such as transcoding, media analysis, or remote playback to your setup using the power of a remote machine, all controllable from your terminal or a GUI such as GstPipelineStudio.

Cross-Platform and Language Support #

The tool is written in Rust for memory safety and reliability and provides client libraries in both Rust and C, offering all the flexibility needed for your existing applications. It is available on Linux (deb, rpm or docker), MacOS, and Windows, see the release page.

Examples #

# Start the daemon
gst-pop

# Launch a pipeline
gst-pop launch videotestsrc ! autovideosink

# Inspect an element
gst-pop inspect videotestsrc

# Discover media info
gst-pop discover file:///path/to/video.mp4

# Play a media file
gst-pop play file:///path/to/video.mp4

# Create a pipeline with the client
gst-popctl create "videotestsrc ! autovideosink"

# List pipelines on a remote daemon
gst-popctl list

# Play the pipeline with ID 0
gst-popctl play 0

# Stop the pipeline with ID 0
gst-popctl stop 0

# Run via Docker
docker run -d -p 9000:9000 ghcr.io/dabrain34/gstpop:latest

Give it a try and let us know what ideas you might have — we have plenty coming, so stay tuned.

As usual, if you would like to learn more about gst-pop, GStreamer, or any other open multimedia framework, please contact us!

Introducing GstPipelineStudio 0.5.1

2026-04-07T00:00:00Z

GstPipelineStudio 0.5.1 #

Your GStreamer Pipelines, at a Glance #

New version of GstPipelineStudio is out!

After months of improvements and intermediate releases since October 2024, it’s time for an official announcement for 0.5.1.

GstPipelineStudio provides a visual interface to GStreamer, the marvelous Swiss Army knife of multimedia pipelines. But what is GStreamer exactly?

What is GStreamer? #

    filesrc --> mp3dec --> audiosink

GStreamer is written in C, with a growing ecosystem of plugins in Rust and bindings for languages such as Python and C++. It ships with many command-line tools to build and test pipelines, but validating ideas still requires writing C/Rust/Python code or using the command line. That’s where GstPipelineStudio comes in — providing a visual interface to help newcomers discover and adopt GStreamer, and skilled developers debug their pipelines.

The Story Behind GstPipelineStudio #

The GstPipelineStudio project started in 2021 with the idea to provide the same environment that brought me to multimedia: GraphEdit on Windows with DirectShow. Indeed, DirectShow and GStreamer share the same idea of plugins sharing data. As I started to implement a DVB decoder with DirectShow, the graphical interface made it easier to validate which filters to use. But DirectShow only works natively on Windows, unlike GStreamer which can run everywhere — Linux, macOS, Windows, iOS, Android, and even low-power devices such as a Raspberry Pi.

GstPipelineStudio aims to work on all these platforms, easing GStreamer adoption where its use was not always obvious, such as on Windows. GStreamer is based on GLib, a cross-platform toolkit that abstracts system calls and provides a common base layer. For the GUI, since Rust was offering very good bindings, GTK was the natural choice to achieve cross-platform support. There was an attempt to create a GUI using Qt, named pipeviz, which has been a great inspiration for GPS, but the Qt Rust bindings were not mature enough, unlike those for GTK.

The first official release of GPS was 0.3.4, and you can read its official blog post published in 2023. Since then, we have been devoted to providing new features to bring GPS to another level.

A first revision, GPS 0.4.0, came out before Christmas 2024 with a refreshed interface — including zoom on the graph and contextual menus on any element or pad of the pipeline. The versions of GStreamer and GTK have also been updated to get the latest plugins and features from both frameworks. A new icon has also been introduced to let GPS dive into another dimension.

What’s New in 0.5.1 #

0.5.1 is here, and it brings a game changer: the dot file reader. Previously, it was possible to open a command-line pipeline or save/open pipelines with an XML-based format, but now you can also open the generated dot files, the native format in GStreamer, to display a pipeline graphically. This is still a beta version as it can only display high-level pipelines such as those described with the command line. Nevertheless this is a great improvement and allows users to see their pipeline and manipulate it.

Here is the list of other improvements you’ll find in this release:

Open Dot Folder menu entry for loading dot files from the common GStreamer folder
Remote pipeline introspection using the GStreamer tracers
App ID renamed to dev.mooday.GstPipelineStudio
Improved look and feel of the interface
Auto-connect on node click (node-link-request)
File selector button for location property
Logger copy to clipboard with multi-selection support
Auto-arrange elements on screen
GStreamer 1.28.0
GTK 4.20
RPM and AppImage artifacts

Remote Pipeline Introspection #

The remote pipeline introspection is a new way to connect to the WebSocket tracer available in GStreamer, pipeline-snapshot. In addition to dot file loader, it allows you to visualize a pipeline directly in GPS from an external process running with the tracer.

As you may know, GStreamer pipelines can be very complex, so one dream was to be able to visualize them live. There is already a mini tool in GStreamer named gst-dots-viewer which creates a web server to display pipelines in a browser from the $XDG_CACHE_DIR folder, see the blog post from Thibault about it.

Now with GPS, you can directly create a WebSocket server and let the tracer connect to it and provide available dot files to be displayed.

For example, to visualize a running pipeline in GPS:

In GPS: Menu → Remote Pipeline → Listen…
Enter the WebSocket address (e.g., ws://localhost:8080)
Run your GStreamer pipeline with the pipeline-snapshot tracer:

GST_TRACERS="pipeline-snapshot(dots-viewer-ws-url=ws://localhost:8080)" \
  gst-launch-1.0 videotestsrc ! autovideosink

The pipeline graph will appear in GPS once the tracer connects.

These dot files are converted to GPS pipelines, making it possible to modify them. That’s a first step for real interaction with GStreamer pipelines — and there are more features coming in the pipeline.

Coming in 0.6.0 #

In parallel, a new tool named GstPrinceOfParser (gst-pop) has also been implemented. This tool allows remote control of all pipelines instantiated locally or over the network. It is a multi-pipeline daemon accessible through WebSocket or D-Bus, aiming to centralize all GStreamer options in one tool for launch, inspection, and discovery. GstPipelineStudio will be able to control this daemon, making gst-pop the backbone of the GStreamer GUI. A blog post will come soon, stay tuned…

A new tracer is under development: a WebSocket server that will allow you to inspect and interact with the current pipeline — modify the play state (pause, seek), fetch the logs, and of course see the current dot representation, all from the GstPipelineStudio interface.

In addition, more features are on the way: a new look and feel based on libadwaita on Linux/macOS/Windows, better localization, an auto-plug feature, seek and step-by-step playback, and bug fixes on demand.

We hope you’ll enjoy this new version of the tool and please feel free to propose new features with an RFC here or merge requests here.

Stay tuned for the next GStreamer Spring hackfest 2026 coming soon (end of May) where new features and deeper interaction with GStreamer pipelines will be discussed.

As usual, if you would like to learn more about GstPipelineStudio, GStreamer, or any other open multimedia framework, please contact us!

Vulkan video with NVK driver

2025-04-28T00:00:00Z

This blog post might interest you if you want to try the bleeding edge NVK driver which allows to decode H264/5 video with the power of the Vulkan extensions VK_KHR_video_decode_h26x.

This is a summary of the instructions provided in the MR. This work needs a recent kernel with new features, so it will describe the steps to add this feature and build this new kernel on an Ubuntu based system.

To accomplish that what you would need, is a recent NVIDIA hardware such as GeForce RTX 4060, a Linux kernel greater than 6.12.8 and the mesa branch supporting Vulkan Video using NVK: https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31867

Lets start by the root, the kernel.

Kernel build #

To run the NVK driver, you need a custom patch to be applied on top of the Nouveau driver. This patch applies on minimum 6.12 kernel so you need to build a new kernel except if your distribution is running bleeding edge kernel which i doubt so here is my method I used to build this kernel.

I mainly got my inspiration from https://askubuntu.com/questions/718381/how-to-compile-and-install-custom-mainline-kernel but I had to adapt it a bit to make it work.

All of this blog post has been performed on a regular ubuntu 24.04.

First you need to install the following ubuntu package:

$ sudo apt install -y git build-essential kernel-package fakeroot libncurses5-dev libssl-dev ccache

You’ll have to download a recent kernel. I ended up using a git clone as it was easier than a tarball to build all necessary bits.

$ git clone --branch v6.12.8 --depth 1 git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git linux-stable
$ cd linux-stable

Then you’ll have to apply the given patch to your work tree:

diff --git a/drivers/gpu/drm/nouveau/nouveau_abi16.c b/drivers/gpu/drm/nouveau/nouveau_abi16.c
index 2a0617e5fe2a..eaef5effbb3d 100644
--- a/drivers/gpu/drm/nouveau/nouveau_abi16.c
+++ b/drivers/gpu/drm/nouveau/nouveau_abi16.c
@@ -379,6 +379,9 @@ nouveau_abi16_ioctl_channel_alloc(ABI16_IOCTL_ARGS)
 			case NOUVEAU_FIFO_ENGINE_CE:
 				engine = NV_DEVICE_HOST_RUNLIST_ENGINES_CE;
 				break;
+			case 0x300:
+				engine = NV_DEVICE_HOST_RUNLIST_ENGINES_NVDEC;
+				break;
 			default:
 				return nouveau_abi16_put(abi16, -ENOSYS);
 			}
@@ -456,10 +459,12 @@ nouveau_abi16_ioctl_channel_alloc(ABI16_IOCTL_ARGS)
 			goto done;
 		break;
 	case NV_DEVICE_INFO_V0_TURING:
-		ret = nvif_object_ctor(&chan->chan->user, "abi16CeWar", 0, TURING_DMA_COPY_A,
-				       NULL, 0, &chan->ce);
-		if (ret)
-			goto done;
+		if (engine != NV_DEVICE_HOST_RUNLIST_ENGINES_NVDEC) {
+			ret = nvif_object_ctor(&chan->chan->user, "abi16CeWar", 0, TURING_DMA_COPY_A,
+					       NULL, 0, &chan->ce);
+			if (ret)
+				goto done;
+		}
 		break;
 	default:
 		break;

Next step will be to configure the kernel. The best option I’ll recommend you is to copy the kernel config, your distribution is shipping with. On Ubuntu you can find it in /boot with the name config-6.8.0-52-generic for example.

$ cp /boot/config-6.8.0-52-generic .config

Then to get the default config, your kernel will use, including the specific options coming with Ubuntu, you’ll have to run:

$ make defconfig

This will setup the build and make it ready to compile with this version of the kernel, auto configuring the new features.

Two options CONFIG_SYSTEM_TRUSTED_KEYS and CONFIG_SYSTEM_REVOCATION_KEYS must be disabled to avoid compilation errors with missing certificates. For that you can set it up within menuconfig or you can edit .config and set these values to ""

Then you should be ready to go for a break ☕, short or long depending on your machine to cook the brand new kernel debian packaged, ready to use:

$ make clean
$ make -j `getconf _NPROCESSORS_ONLN` deb-pkg LOCALVERSION=-custom

The process should end up with a new package named linux-image-6.12.8-custom_6.12.8-3_amd64.deb in the upper folder which can then be installed along your previous kernel.

$ sudo dpkg -i ../linux-image-6.12.8-custom_6.12.8-3_amd64.deb

The first one will replace your current default menulist item in grub upon installation. This means that if you install it, next time you reboot, you’ll boot into that kernel.

Mesa build for NVK #

Now that we have a working kernel which allows to discuss with nvidia codec engine using nouveau driver. We can start the build of mesa.

Prerequisites #

Mesa depends on various system packages in addition to python modules and the the rust toolchain. So first we’ll have to install the given package which are all present in Ubuntu 24.04:

$ sudo apt install build-essential ninja-build glslang-tools python3-pip llvm byacc flex pkg-config bindgen libclc-19-dev libllvmspirvlib-18-dev libdrm-dev libclang-cpp-dev libclang-18-dev

$ sudo apt install libelf-dev libwayland-bin libwayland-dev libwayland-egl-backend-dev libx11-xcb-dev libxcb-randr0-dev libxfixes-dev libxcb-glx0-dev libxcb-shm0-dev libxcb-dri3-dev

$ sudo apt install libxcb-present-dev libxshmfence-dev libxxf86vm-dev

Then you’ll have to install the rust toolchain and some cargo package:

$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
$ cargo install bindgen-cli
$ cargo install cbindgen

When all the system package are installed, you can install the python modules needed by Mesa build system.

$ sudo pip3 install --break-system-packages meson mako pyyaml

Configure and build #

As Mesa is highly depending on the system, you might need some additional packages but worth a try to launch the build with the given instructions:

$ cd ~/DEV/
$ git clone https://gitlab.freedesktop.org/dwlsalmeida/mesa.git --branch=nvk-vulkan-video
$ meson setup builddir --libdir lib64 --prefix `pwd`/builddir/install -Dbuildtype=debug -Dvulkan-drivers=nouveau -Dgallium-drivers=nouveau -Dtools=nouveau
$ ninja -C builddir install

Use of NVK driver #

Now that the kernel and the Mesa driver have been built and are available for your machine, you should be able to decode your first h264 stream with the NVK driver.

Important: Nouveau driver might be disabled !! #

As you might have use the Nvidia driver first and installed with your regular kernel, you might hit a weired error when invoking vulkaninfo such as:

ERROR: [Loader Message] Code 0 : setup_loader_term_phys_devs:  Failed to detect any valid GPUs in the current config
ERROR at ./vulkaninfo/./vulkaninfo.h:247:vkEnumeratePhysicalDevices failed with ERROR_INITIALIZATION_FAILED

Indeed nouveau driver can not live along the Nvidia driver, so you’ll have to uninstall the Nvidia driver first to be able to use nouveau properly and the vulkan extensions. One other solution is to boot on your new custom kernel and modify the file /etc/modprobe.d/nvidia-installer-disable-nouveau.conf to get something like:

# generated by nvidia-installer
#blacklist nouveau
options nouveau modeset=1

In that case the modeset=1 option will enable the driver and allow to use it. Then you’ll have to reboot with this new configuration

Run vulkaninfo #

As you may have noticed, during the configure stage that we chose to install the artifacts of the build in a folder named mesa/builddir/install.

Here is a script run-nvk.sh which you can use before calling any binary which will use this folder as a base to set the environment variable dedicated to the NVK Vulkan driver

#!/bin/sh
MESA=$HOME/DEV/mesa/builddir/install \
LD_LIBRARY_PATH=$MESA/lib64:$MESA/lib:$LD_LIBRARY_PATH \
VK_ICD_FILENAMES=$MESA/share/vulkan/icd.d/nouveau_icd.x86_64.json \
VK_DRIVER_FILES=$MESA/share/vulkan/icd.d/nouveau_icd.x86_64.json \
NVK_I_WANT_A_BROKEN_VULKAN_DRIVER=true \
exec "$@"

After all these stage you should be able to run vulkaninfo such as:

$ ~/DEV/SCRIPTS/run-nvk.sh vulkaninfo | grep video_decode

	VK_KHR_video_decode_h264                    : extension revision 9
	VK_KHR_video_decode_h265                    : extension revision 8
	VK_KHR_video_decode_queue                   : extension revision 8

Run GStreamer playback #

Now its time to run a real application exploiting the power of Vulkan to decode multimedia content. For that I’ll recommend you to use GStreamer which ship with Vulkan elements for decoding in 1.24.2 version bundled in Ubuntu 24.04.

First of all, you’ll have to install the ubuntu packages for GStreamer

sudo apt install gstreamer1.0-plugins-bad gstreamer1.0-tools

Check that the elements are available with:

$ gst-inspect vulkan

  vulkancolorconvert: Vulkan Color Convert
  vulkandeviceprovider: Vulkan Device Provider
  vulkandownload: Vulkan Downloader
  vulkanh264dec: Vulkan H.264 decoder
  vulkanh265dec: Vulkan H.265 decoder
  vulkanimageidentity: Vulkan Image Identity
  vulkanoverlaycompositor: Vulkan Overlay Compositor
  vulkanshaderspv: Vulkan Shader SPV
  vulkansink: Vulkan video sink
  vulkanupload: Vulkan Uploader
  vulkanviewconvert: Vulkan View Convert

If you succeed to see this list of elements, you should be able to run a GStreamer pipeline with Vulkan Video extensioms. Here is a pipeline to decode a content:

$ ~/DEV/SCRIPTS/run-nvk.sh gst-launch-1.0 urisourcebin uri=https://test-videos.co.uk/vids/bigbuckbunny/mp4/h264/1080/Big_Buck_Bunny_1080_10s_1MB.mp4 ! parsebin ! vulkanh264dec ! vulkandownload ! autovideosink

Hope you enjoyed this blog post and you are now able to decode h26x content with the open Vulkan Video extensions.

Thanks a lot to the work of David Airlie and Daniel Almeida who allow to decode h26x content with NVK driver.

Vulkanised 2025

2025-03-11T00:00:00Z

UK calling #

Long time no see this beautiful grey sky, roast beef on sunday and large but full packed pub when there is a football or a rugby game (The rose team has been lucky this year, grrr).

It was a delightful journey in the UK starting with my family visiting London including a lot (yes a lot…) of sightviews in a very short amount of time. But we managed to fit everything in. We saw the changing of the guards, the Thames river tide on a boat, Harry Potter gift shops and the beautiful Arsenal stadium with its legendary pitch, one of the best of England.

It was our last attraction in London and now it was time for my family to go to Standsted back home and me to Cambridge and its legendary university.

To start the journey in Cambridge, first I got some rest on Monday in the hotel to face the hail of information I will get during the conference. This year, Vulkanised took place on Arm’s campus, who kindly hosted the event, providing everything we needed to feel at home and comfortable.

The first day, we started with an introduction from Ralph Potter, the Vulkan Working Group Chair at Khronos, who introduced the new 1.4 release and all the extensions coming along including “Vulkan Video”. Then we could start this conference with my favorite topic, decoding video content with Vulkan Video. And the game was on! There was a presentation every 30 minutes including a neat one from my colleague at Igalia Ricardo Garcia about Device-Generated Commands in Vulkan and a break every 3 presentations. It took a lot of mental energy to keep up with all the topics as each presentation was more interesting than the last. During the break, we had time to relax with good coffee, delicious cookies, and nice conversations.

The first day ended up with a tooling demonstrations from LunarG, helping us all to understand and tame the Vulkan beast. The beast is ours now!

As I was not in the best shape due to a bug I caught on Sunday, I decided to play it safe and went to the hotel just after a nice indian meal. I had to prepare myself for the next day, where I would present “Vulkan Video is Open: Application Showcase”.

Vulkan Video is Open: Application showcase ! #

First Srinath Kumarapuram from Nvidia gave a presentation about the new extensions made available during 2024 by the Vulkan Video TSG. It started with a brief timeline of the video extensions from the initial h26x decoding to the latest VP9 decode coming this year including the 2024 extensions such as the AV1 codec. Then he presented more specific extensions such as VK_KHR_video_encode_quantization_map, VK_KHR_video_maintenance2 released during 2024 and coming in 2025, VK_KHR_video_encode_intra_refresh. He mentioned that the Vulkan toolbox now completely supports Vulkan Video, including the Validation Layers, Vulkan Profiles, vulkaninfo or GFXReconstruct.

After some deserved applause for a neat presentation, it was my time to be on stage.

During this presentation I focused on the Open source ecosystem around Vulkan Video. Indeed Vulkan Video ships with a sample app which is totally open along with the regular Conformance Test Suite. But that’s not all! Two major frameworks now ship with Vulkan Video support: GStreamer and FFmpeg.

Before this, I started by talking about Mesa, the open graphics library. This library which is totally open provides drivers which support Vulkan Video extensions and allow applications to run Vulkan Video decode or encode. The 3 major chip vendors are now supported. It started in 2022 with RADV, a userspace driver that implements the Vulkan API on most modern AMD GPUs. This driver supports all the vulkan video extensions except the lastest ones such as VK_KHR_video_encode_quantization_map or VK_KHR_video_maintenance2 but this they should be implemented sometime in 2025. Intel GPUs are now supported with the ANV driver, this driver also supports the common video extensions such as h264/5 and AV1 codec. The last driver to gain support was at the end of 2024 where several of the Vulkan Video extensions were introduced to NVK, a Vulkan driver for NVIDIA GPUs. This driver is still experimental but it’s possible to decode H264 and H265 content as well as its proprietary version. This completes the offering of the main GPUs on the market.

Then I moved to the applications including GStreamer, FFmpeg and Vulkan-Video-Samples. In addition to the extensions supported in 2025, we talked mainly about the decode conformance using Fluster. To compare all the implementations, including the driver, the version and the framework, a spreadsheet can be found here. In this spreadsheet we summarize the 3 supported codecs (H264, H265 and AV1) with their associated test suites and compare their implemententations using Vulkan Video (or not, see results for VAAPI with GStreamer). GStreamer, my favorite playground, can now decode H264 and H265 since 1.24 and recently got the support for AV1 but the merge request is still under review. It supports more than 80% of the H264 test vectors for the JVT-AVC_V1 and 85% of the H265 test vectors in JCT-VC-HEVC_V1. FFMpeg is offering better figures passing 90% of the tests. It supports all the avaliable codecs including all of the encoders as well. And finally Vulkan-Video-Samples is the app that you want to use to support all codecs for both encode and decode, but its currently missing support for mesa drivers when it comes to use Fluster decode tests…

Vulkanised on the 3rd day #

During the 3rd day, we had interesting talks as well demonstrating the power of Vulkan, from Blender, a free and open-source 3D computer graphics software tool switching progressively to Vulkan, to the implementation of 3D a game engine using Rust, or compute shaders in Astronomy. My other colleague at Igalia, Lucas Fryzek, also had a presentation on Mesa with Lavapipe: a Mesa’s Software Renderer for Vulkan which allows you to have a hardware free implementation of Vulkan and to validate extensions in a simpler way. Finally, we finished this prolific and dense conference with Android and its close collaboration with Vulkan.

If you are interested in 3D graphics, I encourage you to attend future Vulkanised editions, which are full of passionate people. And if you can not attend you can still watch the presentation online.

If you are interested in the Vulkan Video presentation I gave, you can catch up the video here:

Or follow our Igalia live blog post on Vulkan Video:

https://blogs.igalia.com/vjaquez/vulkan-video-status/

As usual, if you would like to learn more about Vulkan, GStreamer or any other open multimedia framework, please feel free to contact us!

Vulkan Video encoder in GStreamer

2023-12-10T00:00:00Z

Vulkan Video encoder in GStreamer #

During the last months of 2023, we, at Igalia, decided to focus on the latest provisional specs proposed by the Vulkan Video Khronos TSG group to support encode operations in an open reference.

As the Khronos TSG Finalizes Vulkan Video Extensions for Accelerated H.264 and H.265 Encode the 19th of December 2023, here is an update on our work to support both h264 and h265 codec in GStreamer.

This work has been possible thanks to the relentless work of the Khronos TSG during the past years, including the IHV vendors, such as AMD, NVIDIA, Intel, but also Rastergrid and its massive work on the specifications wording and the vulkan tooling, and of course Cognizant for its valuable contributions to the Vulkan CTS.

This work started with moving structures in drivers and specifications along the CTS tests definition. It has been a joint venture to validate the specifications and check that the drivers suppports properly the features necessary to encode a video with h264 and h265 codecs. Breaking news, more formats should come very soon…

The GStreamer code #

Building a community #

Thanks first to Lynne and David Airlie, on their RADV/FFmpeg effort, a first implementation was available last year to validate the provisional specifications of the encoders. This work helped us a lot to design the GStreamer version and offer a performant solution for it.

The GStreamer Vulkan bits have been also possible thanks to the tight collaboration with the CTS folks which helped us a lot to understand the crashes we could experience without any clues in the drivers.

To write this GStreamer code, we got inspiration from the code written by the GStreamer community along the GstVA encoders and v4l2codecs plugins. So great kudos to the GStreamer community and especially to He Junayan from Intel, Collabora folks paving the way to stateless codecs and of course Igalia’s past contributions.

And finally the active and accurate reviews from Centricular folks to achieve a valid synchronized operation in Vulkan, see for example https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/5079 in addition to their initial work to support Vulkan.

Give me the (de)coded bits #

The code is available on the GStreamer MR and we plan to have it ready in the 1.24 version.

This code needs at least the Vulkan SDK version 1.3.274 with KHR Vulkan Video encoding extension support. It should be available publicly when the specifications and the tooling will be available along the next SDK release early 2024.

To build it, nothing is more simple than (after installing the LunarG SDK and the GST requirements):

$ meson setup builddir -Dgst-plugins-bad:vulkan-video=enabled
$ ninja -C builddir

And to run it, you’ll need an IHV driver supporting the KHR extension, which should happen very soon, or the community driven driver such as Igalia’s Intel ANV driver. You can also try the AMD RADV driver available here when KHR will be available.

To encode and validate the bitstream you can run this transcode GStreamer pipeline:

$ gst-launch-1.0 videotestsrc ! vulkanupload ! vulkanh265enc ! h265parse ! avdec_h265 ! autovideosink

As you can see a vulkanupload is necessary to upload the GStreamer buffers to the GPU memory, then the buffer will be encoded through the vulkanh265enc which will generate a byte-stream AU aligned stream which can be decoded with any decoder, here the FFMpeg one.

We are planning to simplify this pipeline and allow to select an encoder from a multiple device configuration.

It supports most of the Vulkan video encoder features and can encode I, P and B Frames for better streaming performance. See this blog post

Challenges #

One of the most important challenge here was to put all the bits together and understand when the drivers tell you that you are doing something wrong with only an obscure segfault.

The vulkan process to encode stateless a raw bitstream is quite simple as it will be:

Initialize the video session with the correct video profile (h264,h265 etc…)
Initialize the sessions paramaters such as SPS/PPS which will be used by the driver during the encode session
Reset the codec state
Encode the buffers:
- Change the quality/rate control if necessary
- Retrieve the video session parameters from the driver if necessary
- Set the bitstream standard parameters such as slice header data.
- Set the begin video info to give the context to the encoder for the reference
- Encode the buffer
- Query the encoded result
- End the video coding operation
- repeat …

The challenge here was not the state machine design but more the understanding of the underlying bits necessary to tell the driver what and how to encode it. Indeed to encode a video stream, a lot of parameters (more than 200…) are necessary and a bad use of these parameters can lead to obscure crashes in the driver stack.

So first it was important to validate the data provided to the Vulkan drivers through the Validation layer. This part in implementing an application using Vulkan is the first and essential step to avoid headaches, see here

You can checkout the code from here to get the latest one and configure it to validate your project. Otherwise it should come from the Vulkan SDK.

This step helped us a lot to understand the mechanics expected by the driver to provide the frames and their references to encode those properly. It also helped to use some API such as the rate control or the quality level.

But Validation Layers cant do everything and that’s why designing, reviewing a solid codebase in CTS as a reference is crucial to implement the GStreamer application or any other 3rd party application. So a lot of time in Igalia has been dedicated to ease and simplify the CTS tests to facilitate its usage.

Indeed the Validation Layers do not validate the Video Standard parameters for example such as the one provided for the format underlying specific data.

So we spent a few hours checking some missing parameters in CTS as in GStreamer which was leading to obscure crash, especially when we were playing with references. A blog post should follow explaining this part.

To debug this missing bits, we used some facilitating tools such as GfxReconstruct Vulkan layer or the VK_LAYER_LUNARG_api_dump which allows you to get an essential dump of your calls and compare the results with other implemetations.

Conclusion #

It was a great journey providing a cross platform solution to the community to encode video using largely adopted codec such as h264 and h265. As said before other codecs will come very soon, so stay tuned!

I mentioned the team work in this journey as without the dedication of all, it wouldn’t have been possible to achieve this work in a short time span.

And last but not least, I’d like to special thank also Valve to sponsor this work without whom it wouldn’t have been possible to go as far and as fast!

We’ll attend the Vulkanised 2024 event where we’ll demonstrate this work. Feel free to come and talk to us there. We’ll be delighted to hear and discuss your doubts or brilliant ideas !

As usual, if you would like to learn more about Vulkan, GStreamer or any other open multimedia framework, please contact us!

Introducing GstPipelineStudio 0.3.4

2023-10-20T00:00:00Z

GstPipelineStudio #

As it’s not always convenient to use the powerful command line based, gst-launch tool and also manage all the debug possibilities on all the platforms supported by GStreamer, I started this personal project in 2021 to facilitate the adoption to the GStreamer framework and help newbies as confirmed engineers enjoy the power of it.

Indeed a few other projects, such as Pipeviz (greatly inspired from…) or gst-debugger, already tried to offer this GUI capability, my idea with GPS was to provide a cross-platform tool written in Rust with the powerful framework, gtk-rs.

The aim of this project is to provide the GUI for GStreamer but also being able to remote debug existing pipeline while offering a very simple and accessible interface as back in the days I discovered DirectShow with the help of graphedit or GraphStudioNext

Project details #

The interface includes 5 important zones:

(1) The registry area gives you an access to the GStreamer registry including all the plugins/elements available on your system. It provides you details on each elements. You can also access to a favorite list.
(2) the main drawing area is where you can add elements from the registry area and connect them together. This area allows you to have multiple independent pipelines with its own player for each drawing area.
(3) The control playback area, each pipeline can be controlled from this area including basic play/stop/pause but also a seekbar.
(4) the debug zone where you’ll receive the messages from the application.
(5) The render zone where you can have a video preview if a video sink has been added to the pipeline. Future work includes to have tracers or audio analysis in this zone.

The project has been written in Rust to offer more stability and thanks to the wonderful work to use the GTK framework, it was perfectly fitting to this project as it gives an easy way to use it over the 3 platforms targeted such as GNU/Linux, MacOS and Windows. On this last platform which is quite well “implanted” in the desktop eco-system, the use of GStreamer can lead to difficulties, that’s why GstPipelineStudio Windows MSI will be a perfect match to test the power of the GStreamer framework.

This project has been written under the GPL v3 License.

How it works under the hood #

The trick is quite simple as it uses the power of gst-parse-launch API to build a pipeline as a transformation of the visual pipeline to a command line.

So its a clearly a sibling of gst-launch.

Right now its directly linked to the GStreamer installed on your system but future work could be to connect it over daemons such as GstPrinceOfParser or gstd

What’s new in 0.3.4 #

The main feature of this release is the cross platform ready state. These are beta versions but the CI is now ready to build and deploy Flathub (Linux), Mac OS and Windows version of GstPipelineStudio.

You can download the installers from the project page or with:

Linux: flatpak install org.freedesktop.dabrain34.GstPipelineStudio
MacOS: DMG file
Windows: MSI file

Here is a list of main features added to the app:

Open a pipeline from the command line and it will be drawn automatically on the screen. This feature allows you to take any command line pipeline and draw it on the screen to allow any new play tricks with the pipeline, such as change of elements, properties etc.
Multiple graphview allows you to draw multiple independent pipeline in the same instance of GstPipelineStudio. The playback state is totally independent for each of the views.
Capsfilter has been added to the links allowing to add this crucial feature of GStreamer pipelines.
gstreamer-1.0 wrap support to the build system. So you can build your own version of GPS using a dedicated GStreamer version.

What’s next in the pipeline #

Among multiple use case, key and debug features, the most upcoming features are:

Support the zoom on the graphview. As a pipeline can be quite big, the zoom is a key/must features for GPS. See MR
Debug sections such as receiving events/tags/messages or tracers and terminal support, see MR
Elements compatibility to check if an element can connect to the previous/next one.
Remote debugging: A tracer wsserver is currently under development allowing to send over websocket the pipeline events such as connections, properties or element addition. A MR is under development to connect this tracer and render the corresponding pipeline.
Auto plugging according to the rank of each compatible elements for a given pad caps.
Display the audio signal in a dedicated render tab.
Translations
Documentation
Unit tests

Here is a lighning talk, I gave about this release (0.3.3), during the 2023 GStreamer conference.

Hope you’ll enjoy this tool and please feel free to provide new features with an RFC here or merge requests here.

As usual, if you would like to learn more about GstPipelineStudio, GStreamer or any other open multimedia framework, please contact us!

This is a post with no description

2023-10-06T00:00:00Z

When the post has not description: field, this first paragraph is the excerpt but we want it to be rendered using markdown. The rest of the post content is not relevant for this particular case.

Discover GStreamer full static mode

2023-07-25T00:00:00Z

How to embed statically your own tailored version of GStreamer in your application #

Since the gstreamer-full effort, it was possible to create a shared library which will embed the GStreamer framework in addition to its set of plugins.

Within this effort, it was also possible to register the selected plugins/features automatically by calling the gst_init method in your application linking with gstreamer-full.

This method was offering a gstreamer-full package with library, headers and pc files but it was not possible to embed GStreamer statically in your application and use it transparently.

GstVkVideoParser: a standalone solution #

In the journey to bring an open source solution for a video parser to the Vulkan Conformance Test Suite, we chose first to use GStreamer as it was bringing all the parsing facilities necessary to support the needed codecs such as H26x or VPx. This solution was supposed to be also cross platform and dragging as less as possible system dependencies. Seen that GStreamer is usually dragging its own dependencies such as glib or orc and as we wanted to have a standalone GstVkVideoParser library supported on Windows, a little bit of work and love was necessary to add this to GStreamer.

Unfortunately this solution has not be retained by the Vulkan Video TSG, not because it was not working but another parser has been made available and easy to integrate to the CTS at source level avoiding binary linkage, see Vulkan Video change.

GStreamer as a full static library #

With the gstreamer-full work, everything was almost ready to be used except to have gstreamer-full as a real static library and be able to link with it in any application.

Here is the MR merged and the challenges taken up:

Adding gst-full-target-type=static #

To generate the gstreamer-full dependency which will be statically linked into the application, we decided to introduce a new gst meson options, gst-full-target-type.

By default the gstreamer-full will be built as a shared library as before.

By passing gst-full-target-type=static, only static object will be generated and a package config file will be generated for gstreamer-full allowing the application to avoid to know what static library it needs to add the link line. The GStreamer build system will take care of enabling/disabling the features/libraries you (do/don’t) need.

Initialize the plugins/features automatically #

To avoid multiple call necessary to initialize GStreamer, it was also necessary to call the gst_init_static_plugins along with gst_init only in full-static mode but it was leading to a build issue.

Indeed most of tools/examples/tests are linking with libgstreamer-1.0 which owns gst_init () but to facilitate the plugins registration, it was necessary to move all the tools build after the gstreamer-full stage. A first MR has been performed to let gstreamer tools be built against gstreamer-full but additional work was necessary for some core tools or helpers such as gst-transcoder or gst-plugin-scanner to avoid a linking issue.

Disable tests and examples #

In a future work all the tools/examples/tests should support the full-static mode but as GStreamer aims to be a shared object framework, we decided to leave this work for later and disable all the examples/tests in full static mode as most of the application using a tailor build won’t need the examples and tests.

Windows support #

One of the goal of this work was to provide a Windows library to the Vulkan CTS free of dependency, which has been achieved but some additional work might be necessary to support all of the use case, the GStreamer framework offer, especially on supporting library-dependent plugins.

Give me an example … #

In the GstVkVideoParser project, various jobs are building Linux and Windows versions generating a library without any GStreamer/glib dependencie, everything is embedded inside the library, as you can see in this GitHub’ Actions.

In this project, GStreamer is used as a meson subproject/wrap which allows to build GStreamer along of GstVkVideoParser. This can be possible easily by adding the following file to your meson project

subprojects/gstreamer-1.0.wrap

[wrap-git]
directory=gstreamer-1.0
url=https://gitlab.freedesktop.org/gstreamer/gstreamer.git
revision=main

[provide]
dependency_names = gstreamer-1.0, gstreamer-base-1.0, gstreamer-video-1.0, gstreamer-audio-1.0

and then add the following lines to your meson.build to depend on gstreamer-full

meson.build

gstreamer_full_dep = dependency('gstreamer-full-1.0', fallback: ['gstreamer-1.0'], required :true)

In order to build a project, library or application which is using a tailored version of GStreamer you can follow this configure example:

$ meson buildfull-static --default-library=static --force-fallback-for=gstreamer-1.0,glib,libffi,pcre2 -Dauto_features=disabled -Dglib:tests=false -Djson-glib:tests=false -Dpcre2:test=false -Dvkparser_standalone=enabled -Dgstreamer-1.0:libav=disabled -Dgstreamer-1.0:ugly=disabled -Dgstreamer-1.0:ges=disabled -Dgstreamer-1.0:devtools=disabled -Dgstreamer-1.0:default_library=static -Dgstreamer-1.0:rtsp_server=disabled -Dgstreamer-1.0:gst-full-target-type=static_library -Dgstreamer-1.0:gst-full-libraries=gstreamer-video-1.0, gstreamer-audio-1.0, gstreamer-app-1.0, gstreamer-codecparsers-1.0 -Dgst-plugins-base:playback=enabled -Dgst-plugins-base:app=enabled -Dgst-plugins-bad:videoparsers=enabled -Dgst-plugins-base:typefind=enabled

In this case we are disabling everything in GStreamer by using -Dauto_features=disabled and some enabled features such as ges, libav, etc. and enable only what we need as plugins, playback, app, videoparsers and typefind.

And finally we are enabling the static build with --default-library=static and -Dgstreamer-1.0:gst-full-target-type=static_library.

Next … #

As you can see, it’s quite easy now to build an application and depends on gstreamer-full static build, but there is still some issues to address such as the plugins dependencies which might be not static and some other platform specific issue such as the gstreamer-full symbols export on Windows.

You can follow some open issues such as:

As usual, if you would like to learn more about Vulkan Video, GStreamer or any other open multimedia framework, please contact us!

ESExtractor: how to integrate a dependency-free library to the Khronos CTS

2023-04-19T00:00:00Z

ESExtractor, how to integrate a dependency-free library to the Khronos CTS #

Since the Vulkan CTS is now able to test and check Vulkan Video support including video decoding, it was necessary to define the kind of media container to be used inside the test cases and the library to extract the necessary encoded data.

In a first attempt, the FFmpeg media toolkit had been chosen to extract the video packets from the A/V ISO base media format chosen as a container reference. This library was provided as a binary package and loaded dynamically at each test run.

As Vulkan video aims to test only video contents, it was not necessary to choose a complex media container, so first all the videos were converted to the elementary stream format for H264 and H265 contents. This is a very elementary format based on MPEG start codes and NAL unit identification.

To avoid an extra multimedia solution integrable only with binaries, a first attempt to replace FFmpeg was, to use GStreamer and an in-house helper library called demuxeres. It was smaller but needed to be a binary still to avoid the glib/gstreamer system dependencies (self contained library). it was a no-go still because the binary package would be awkward to support on the various platforms targetted by the the Khronos CTS.

So at Igalia, we decided to implement a minimal, dependency-free, custom library, written in C++ to be compliant with the Khronos CTS and simple to integrate into any build system.

This library is called ESExtractor

A proposed change :) please close

What is ESExtractor ? #

ESExtractor aims to be a simple elementary stream extractor. For the first revision it was able to extract video data from a file in the NAL standard. The first official release was 0.2.4. In this release, only the NAL was supported with both the H264 and H265 streams supported.

As Vulkan Video aims to support more than H264 and H265 including format such as AV1 or VP9, the ESExtractor had to support multiple format. A redesign has been started to support multiple format and is now available in the version 0.3.2.

How ESExtractor works #

A simple C interface is provided to maximise portability and use by other languages.

es_extractor_new #

ESExtractor extractor = es_extractor_new(filePath, "options"));

This interface returns the main object which will give you access to the packets according to the file path and the options given in the arguments. This interface returns a ready to use object where the stream has been initially parsed to determine the kind of video during the object creation.

Then you can check the video format with:

ESEVideoFormat eVideoFormat = es_extractor_video_format(extractor);

or the video codec with:

ESEVideoCodec eVideoCodec = es_extractor_video_codec(extractor);

It supports H264, H265, AV1 and VP9 for now.

es_extractor_read_packet #

This API is the main function to retrieve the available packets from the file. Each time this API is called, the library will return the next available packet according to the format and the specific alignment (ie NAL) and a status to let the application decide what to do next. The packet should be freed using es_extractor_clear_packet.

Has CI powered by github #

To test the library usage, we have implementing a testing framework in addition to a CI infrastructure As github offers a very powerful worklow, we decided to use this platform to test the library on various architectures and platforms. The CI is now configured to release packages for 64 and 32 bits on Linux and Windows.

As usual, if you would like to learn more about Vulkan Video, ESExtractor or any other open multimedia framework, please contact us!

This is a post without header.

2023-02-09T00:00:00Z

Leverage agile frameworks to provide a robust synopsis for high level overviews. Iterative approaches to corporate strategy foster collaborative thinking to further the overall value proposition. Organically grow the holistic world view of disruptive innovation via workplace diversity and empowerment. Bring to the table win-win survival strategies to ensure proactive domination. At the end of the day, going forward, a new normal that has evolved from generation X is on the runway heading towards a streamlined cloud solution. User generated content in real-time will have multiple touchpoints for offshoring.

Section Header #

Capitalize on low hanging fruit to identify a ballpark value added activity to beta test. Override the digital divide with additional clickthroughs from DevOps. Nanotechnology immersion along the information highway will close the loop on focusing solely on the bottom line.

This is a post with a header picture.

2023-02-08T00:00:00Z

list item
list item 2

numbered list
second item

Code #

Styled (with Syntax) #

Bring to the table win-win survival strategies to ensure proactive domination. At the end of the day, going forward, a new normal that has evolved from generation X is on the runway heading towards a streamlined cloud solution. User generated content in real-time will have multiple touchpoints for offshoring.

// this is a command
function myCommand() {
	let counter = 0;
	counter++;
}

// Test with a line break above this line.
console.log('Test');

Unstyled #

// this is a command
function myCommand() {
	let counter = 0;
	counter++;
}

// Test with a line break above this line.
console.log('Test');