Dabrain34's igalian blogDabrain34's blog.2023-12-10T00:00:00Zhttps://blogs.igalia.com/Stéphane Cerveauscerveau@igalia.comVulkan Video encoder in GStreamer2023-12-10T00:00:00Zhttps://blogs.igalia.com/scerveau/vulkan-video-encoder-in-gstreamer/<h1 id="vulkan-video-encoder-in-gstreamer" tabindex="-1">Vulkan Video encoder in GStreamer <a class="header-anchor" href="https://blogs.igalia.com/scerveau/vulkan-video-encoder-in-gstreamer/">#</a></h1>
<p>During the last months of 2023, we, at <a href="https://www.igalia.com/about/">Igalia</a>, decided to focus on the latest provisional specs proposed
by the <a href="https://www.khronos.org/news/press/vulkan-sdk-is-vulkan-video-ready">Vulkan Video Khronos TSG group</a> to support encode operations in an open reference.</p>
<p>As the <a href="https://www.khronos.org/blog/khronos-finalizes-vulkan-video-extensions-for-accelerated-h.264-and-h.265-encode?mc_cid=e4afdbcd22&mc_eid=47d3c1b7bb">Khronos TSG Finalizes Vulkan Video Extensions for Accelerated H.264 and H.265 Encode</a>
the 19th of December 2023, here is an update on our work to support both h264 and h265 codec in GStreamer.</p>
<p>This work has been possible thanks to the relentless work of the Khronos TSG during the past years, including the IHV vendors, such as
AMD, NVIDIA, Intel, but also Rastergrid and its massive work on the specifications wording and the vulkan tooling, and of course Cognizant for its valuable
contributions to the <a href="https://github.com/KhronosGroup/VK-GL-CTS">Vulkan CTS</a>.</p>
<p>This work started with moving structures in drivers and specifications along the CTS tests definition. It has been a joint venture
to validate the specifications and check that the drivers suppports properly the features necessary to encode a video
with h264 <strong>and</strong> h265 codecs. <em>Breaking news, more formats should come very soon...</em></p>
<h2 id="the-gstreamer-code" tabindex="-1">The GStreamer code <a class="header-anchor" href="https://blogs.igalia.com/scerveau/vulkan-video-encoder-in-gstreamer/">#</a></h2>
<h3 id="building-a-community" tabindex="-1">Building a community <a class="header-anchor" href="https://blogs.igalia.com/scerveau/vulkan-video-encoder-in-gstreamer/">#</a></h3>
<p>Thanks first to <a href="https://lynne.ee/pages/about.html">Lynne</a> and <a href="https://airlied.blogspot.com/">David Airlie</a>, on their RADV/FFmpeg effort,
a first implementation was available last year to validate the provisional specifications of the encoders.
This work helped us a lot to design the GStreamer version and offer a performant solution for it.</p>
<p>The GStreamer Vulkan bits have been also possible thanks to the tight collaboration with the CTS folks which helped us a lot to understand the
crashes we could experience without any clues in the drivers.</p>
<p>To write this GStreamer code, we got inspiration from the code written by the GStreamer community along the <a href="https://gstreamer.freedesktop.org/documentation/va/index.html?gi-language=c">GstVA encoders</a> and
v4l2codecs plugins. So great kudos to the GStreamer community and especially to He Junayan from Intel, Collabora folks paving the way to stateless codecs and of course
Igalia's past contributions.</p>
<p>And finally the active and accurate reviews from Centricular folks to achieve a valid synchronized operation in Vulkan,
see for example <a href="https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/5079">https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/5079</a> in addition to their initial work to
support Vulkan.</p>
<h3 id="give-me-the-de-coded-bits" tabindex="-1">Give me the (de)coded bits <a class="header-anchor" href="https://blogs.igalia.com/scerveau/vulkan-video-encoder-in-gstreamer/">#</a></h3>
<p>The code is available on the <a href="https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/5739">GStreamer MR</a> and we plan to have it ready in the 1.24 version.</p>
<p>This code needs at least the Vulkan SDK version 1.3.274 with KHR Vulkan Video encoding extension support. It should be available publicly when the specifications and the tooling will be available
along the next SDK release early 2024.</p>
<p>To build it, nothing is more simple than (after installing the <a href="https://vulkan.lunarg.com/#new_tab">LunarG SDK</a> and the <a href="https://gstreamer.freedesktop.org/documentation/installing/building-from-source-using-meson.html?gi-language=c">GST requirements</a>):</p>
<pre><code>$ meson setup builddir -Dgst-plugins-bad:vulkan-video=enabled
$ ninja -C builddir
</code></pre>
<p>And to run it, you'll need an IHV driver supporting the KHR extension, which should happen very soon, or the community driven driver such as <a href="https://gitlab.freedesktop.org/zzoon/mesa/-/tree/h264enc_anv_4?ref_type=heads">Igalia's Intel ANV driver</a>.
You can also try the AMD RADV driver available <a href="https://gitlab.freedesktop.org/airlied/mesa/-/commits/radv-vulkan-video-encode-h2645-spec-latest/?ref_type=undefined">here</a>
when KHR will be available.</p>
<p>To encode and validate the bitstream you can run this transcode GStreamer pipeline:</p>
<pre><code>$ gst-launch-1.0 videotestsrc ! vulkanupload ! vulkanh265enc ! h265parse ! avdec_h265 ! autovideosink
</code></pre>
<p>As you can see a <code>vulkanupload</code> is necessary to upload the GStreamer buffers to the GPU memory, then the buffer will be encoded
through the <code>vulkanh265enc</code> which will generate a byte-stream AU aligned stream which can be decoded with any decoder, here the FFMpeg one.</p>
<p>We are planning to simplify this pipeline and allow to select an encoder from a multiple device configuration.</p>
<p>It supports most of the Vulkan video encoder features and can encode I, P and B Frames for better streaming performance. See this
<a href="https://www.rastergrid.com/blog/multimedia/2021/05/video-compression-basics/">blog post</a></p>
<h2 id="challenges" tabindex="-1">Challenges <a class="header-anchor" href="https://blogs.igalia.com/scerveau/vulkan-video-encoder-in-gstreamer/">#</a></h2>
<p>One of the most important challenge here was to put <strong>all</strong> the bits together and understand when the drivers tell you that you are doing
something wrong with only an obscure segfault.</p>
<p><picture><source type="image/avif" srcset="https://blogs.igalia.com/scerveau/img/Q5VBs5lDkE-300.avif 300w"><source type="image/webp" srcset="https://blogs.igalia.com/scerveau/img/Q5VBs5lDkE-300.webp 300w"><img alt="Vulkan Video" loading="lazy" decoding="async" src="https://blogs.igalia.com/scerveau/img/Q5VBs5lDkE-300.png" width="300" height="320"></picture></p>
<p>The vulkan process to encode <em>stateless</em> a raw bitstream is quite simple as it will be:</p>
<ul>
<li>Initialize the video session with the correct video profile (h264,h265 etc...)</li>
<li>Initialize the sessions paramaters such as SPS/PPS which will be used by the driver during the encode session</li>
<li>Reset the codec state</li>
<li>Encode the buffers:
<ul>
<li>Change the quality/rate control if necessary</li>
<li>Retrieve the video session parameters from the driver if necessary</li>
<li>Set the bitstream standard parameters such as slice header data.</li>
<li>Set the begin video info to give the context to the encoder for the reference</li>
<li>Encode the buffer</li>
<li>Query the encoded result</li>
<li>End the video coding operation</li>
<li><em>repeat ..</em></li>
</ul>
</li>
</ul>
<p>The challenge here was not the state machine design but more the understanding of the underlying bits necessary to tell
the driver what and how to encode it. Indeed to encode a video stream, a lot of parameters (more than 200..) are necessary and a bad
use of these parameters can lead to obscure crashes in the driver stack.</p>
<p>So first it was important to validate the data provided to the Vulkan drivers through the Validation layer. This part
in implementing an application using Vulkan is the first and essential step to avoid headaches, see <a href="https://vulkan-tutorial.com/Drawing_a_triangle/Setup/Validation_layers">here</a></p>
<p>You can checkout the code from <a href="https://github.com/KhronosGroup/Vulkan-ValidationLayers">here</a> to get the latest one and configure it to validate
your project. Otherwise it should come from the Vulkan SDK.</p>
<p>This step helped us a lot to understand the mechanics expected by the driver to provide the frames and their references
to encode those properly. It also helped to use some API such as the rate control or the quality level.</p>
<p>But Validation Layers cant do everything and that's why designing, reviewing a solid codebase in CTS as a reference
is crucial to implement the GStreamer application or any other 3rd party application. So a lot of time in Igalia has been
dedicated to ease and simplify the CTS tests to facilitate its usage.</p>
<p>Indeed the Validation Layers do not validate the Video Standard parameters for example such as the one provided for the format
underlying specific data.</p>
<p>So we spent a few hours checking some missing parameters in CTS as in GStreamer which was leading to obscure crash, especially
when we were playing with references. A blog post should follow explaining this part.</p>
<p>To debug this missing bits, we used some facilitating tools such as <a href="https://vulkan.lunarg.com/doc/view/latest/linux/capture_tools.html">GfxReconstruct Vulkan layer</a>
or the <code>VK_LAYER_LUNARG_api_dump</code> which allows you to get an essential dump of your calls and compare the results with other implemetations.</p>
<h2 id="conclusion" tabindex="-1">Conclusion <a class="header-anchor" href="https://blogs.igalia.com/scerveau/vulkan-video-encoder-in-gstreamer/">#</a></h2>
<p>It was a great journey providing a cross platform solution to the community to encode video using largely adopted codec
such as h264 and h265. As said before other codecs will come very soon, so stay tuned!</p>
<p>I mentioned the team work in this journey as without the dedication of all, it wouldn't have been possible to achieve this work
in a short time span.</p>
<p>And last but not least, I'd like to special thank also Valve to sponsor this work without whom it wouldn't have been possible to go as far and as fast!</p>
<p>We'll attend the <a href="https://www.khronos.org/events/vulkanised-2024">Vulkanised 2024</a> event where we'll demonstrate this work.
Feel free to come and talk to us there. We'll be delighted to hear and discuss your doubts or brilliant ideas !</p>
<p>As usual, if you would like to learn more about Vulkan, GStreamer or any other open multimedia framework, please contact <a href="https://www.igalia.com/">us</a>!</p>
Introducing GstPipelineStudio 0.3.42023-10-20T00:00:00Zhttps://blogs.igalia.com/scerveau/introducing-gstpipelinestudio-0-3-4/<h2 id="gstpipelinestudio" tabindex="-1">GstPipelineStudio <a class="header-anchor" href="https://blogs.igalia.com/scerveau/introducing-gstpipelinestudio-0-3-4/">#</a></h2>
<p>As it's not always convenient to use the powerful command line based, <code>gst-launch</code> tool and also manage all the debug possibilities on all the platforms supported by GStreamer,
I started this personal project in 2021 to facilitate the adoption to the GStreamer framework and help newbies as confirmed engineers enjoy the power of it.</p>
<p>Indeed a few other projects, such as <a href="https://github.com/virinext/pipeviz">Pipeviz</a> (greatly inspired from...) or <a href="https://gitlab.gnome.org/GNOME/gst-debugger">gst-debugger</a>, already tried to offer this GUI capability,
my idea with GPS was to provide a cross-platform tool written in Rust with the powerful framework, gtk-rs.</p>
<p>The aim of this project is to provide the GUI for GStreamer but also being able to remote debug existing pipeline while offering
a very simple and accessible interface as back in the days I discovered DirectShow with the help of <a href="https://learn.microsoft.com/en-us/windows/win32/directshow/using-graphedit">graphedit</a> or <a href="https://www.videohelp.com/software/GraphStudio">GraphStudioNext</a></p>
<h2 id="project-details" tabindex="-1">Project details <a class="header-anchor" href="https://blogs.igalia.com/scerveau/introducing-gstpipelinestudio-0-3-4/">#</a></h2>
<p><picture><source type="image/avif" srcset="https://blogs.igalia.com/scerveau/img/YyhaJBWbN6-791.avif 791w"><source type="image/webp" srcset="https://blogs.igalia.com/scerveau/img/YyhaJBWbN6-791.webp 791w"><img alt="GstPipelineStudio interface" loading="lazy" decoding="async" src="https://blogs.igalia.com/scerveau/img/YyhaJBWbN6-791.png" width="791" height="411"></picture></p>
<p>The interface includes 5 important zones:</p>
<ul>
<li>(1) The registry area gives you an access to the GStreamer registry including all the plugins/elements available on your system. It provides
you details on each elements. You can also access to a favorite list.</li>
<li>(2) the main drawing area is where you can add elements from the registry area and connect them together. This area
allows you to have multiple independent pipelines with its own player for each drawing area.</li>
<li>(3) The control playback area, each pipeline can be controlled from this area including basic play/stop/pause but also a seekbar.</li>
<li>(4) the debug zone where you'll receive the messages from the application.</li>
<li>(5) The render zone where you can have a video preview if a video sink has been added to the pipeline. Future work includes to have tracers or audio analysis in this zone.</li>
</ul>
<p>The project has been written in Rust to offer more stability and thanks to the wonderful work to use the GTK framework, it
was perfectly fitting to this project as it gives an easy way to use it over the 3 platforms targeted such as GNU/Linux, MacOS and
Windows.
On this last platform which is quite well "implanted" in the desktop eco-system, the use of GStreamer
can lead to difficulties, that's why <a href="https://gitlab.freedesktop.org/dabrain34/GstPipelineStudio/uploads/ce4b8443e2d3161eeaed089071cdc402/GstPipelineStudio-0.3.4.msi">GstPipelineStudio Windows MSI</a> will be a perfect match to test the power of the GStreamer framework.</p>
<p>This project has been written under the GPL v3 License.</p>
<h2 id="how-it-works-under-the-hood" tabindex="-1">How it works under the hood <a class="header-anchor" href="https://blogs.igalia.com/scerveau/introducing-gstpipelinestudio-0-3-4/">#</a></h2>
<p>The trick is quite simple as it uses the power of <a href="https://gstreamer.freedesktop.org/documentation/gstreamer/gstparse.html?gi-language=c#gst_parse_launch">gst-parse-launch API</a> to build a pipeline as a transformation of the visual pipeline to a command line.</p>
<p>So its a clearly a sibling of <code>gst-launch</code>.</p>
<p>Right now its directly linked to the GStreamer installed on your system but future work could be to connect it over daemons
such as <a href="https://github.com/dabrain34/gpop">GstPrinceOfParser</a>
or <a href="https://developer.ridgerun.com/wiki/index.php/GStreamer_Daemon">gstd</a></p>
<h2 id="what-s-new-in-0-3-4" tabindex="-1">What's new in <a href="https://gitlab.freedesktop.org/dabrain34/GstPipelineStudio/-/releases">0.3.4</a> <a class="header-anchor" href="https://blogs.igalia.com/scerveau/introducing-gstpipelinestudio-0-3-4/">#</a></h2>
<p>The main feature of this release is the cross platform ready state. These are beta versions but the CI is now ready
to build and deploy Flathub (Linux), Mac OS and Windows version of GstPipelineStudio.</p>
<p>You can download the installers from the project <a href="https://dabrain34.pages.freedesktop.org/GstPipelineStudio/">page</a> or with:</p>
<ul>
<li>Linux: <code>flatpak install org.freedesktop.dabrain34.GstPipelineStudio</code></li>
<li>MacOS: <a href="https://gitlab.freedesktop.org/dabrain34/GstPipelineStudio/uploads/5ac641779cfb7e8fffdf9be6a61fba17/GstPipelineStudio-0.3.4.dmg">DMG file</a></li>
<li>Windows: <a href="https://gitlab.freedesktop.org/dabrain34/GstPipelineStudio/uploads/ce4b8443e2d3161eeaed089071cdc402/GstPipelineStudio-0.3.4.msi">MSI file</a></li>
</ul>
<p>Here is a list of main features added to the app:</p>
<ul>
<li>
<p>Open a pipeline from the command line and it will be drawn automatically on the screen. This feature
allows you to take any command line pipeline and draw it on the screen to allow any new play tricks with the pipeline, such as change of elements, properties etc.</p>
</li>
<li>
<p>Multiple graphview allows you to draw multiple independent pipeline in the same instance of GstPipelineStudio.
The playback state is totally independent for each of the views.</p>
</li>
<li>
<p>Capsfilter has been added to the links allowing to add this crucial feature of GStreamer pipelines.</p>
</li>
<li>
<p>gstreamer-1.0 wrap support to the build system. So you can
build your own version of GPS using a dedicated GStreamer version.</p>
</li>
</ul>
<h2 id="what-s-next-in-the-pipeline" tabindex="-1">What's next in the pipeline <a class="header-anchor" href="https://blogs.igalia.com/scerveau/introducing-gstpipelinestudio-0-3-4/">#</a></h2>
<p>Among multiple use case, key and debug features, the most upcoming features are:</p>
<ul>
<li>Support the zoom on the graphview. As a pipeline can be quite big, the zoom is a key/must features for GPS. See <a href="https://gitlab.freedesktop.org/dabrain34/GstPipelineStudio/-/merge_requests/53">MR</a></li>
<li>Debug sections such as receiving events/tags/messages or tracers and terminal support, see <a href="https://gitlab.freedesktop.org/dabrain34/GstPipelineStudio/-/merge_requests/55">MR</a></li>
<li>Elements compatibility to check if an element can connect to the previous/next one.</li>
<li>Remote debugging: A tracer wsserver is currently under development allowing to send over websocket the pipeline events such as connections, properties or element addition.
A <a href="https://gitlab.freedesktop.org/dabrain34/GstPipelineStudio/-/merge_requests/52">MR</a> is under development to connect this tracer and render the corresponding pipeline.</li>
<li>Auto plugging according to the rank of each compatible elements for a given pad caps.</li>
<li>Display the audio signal in a dedicated render tab.</li>
<li>Translations</li>
<li>Documentation</li>
<li>Unit tests</li>
</ul>
<p>Here is a <a href="https://gstconf.ubicast.tv/videos/gstpipelinestudio-version-030-is-out/">lighning talk</a>, I gave about this release (0.3.3), during the 2023 GStreamer conference.</p>
<p>Hope you'll enjoy this tool and please feel free to provide
new features with an RFC <a href="https://gitlab.freedesktop.org/dabrain34/GstPipelineStudio/-/issues/new">here</a> or merge requests <a href="https://gitlab.freedesktop.org/dabrain34/GstPipelineStudio/-/merge_requests">here</a>.</p>
<p>As usual, if you would like to learn more about GstPipelineStudio, GStreamer or any other open multimedia framework, please contact <a href="https://www.igalia.com/">us</a>!</p>
Discover GStreamer full static mode2023-07-25T00:00:00Zhttps://blogs.igalia.com/scerveau/discover-gstreamer-full-static-mode/<h1 id="how-to-embed-statically-your-own-tailored-version-of-gstreamer-in-your-application" tabindex="-1">How to embed statically your own tailored version of GStreamer in your application <a class="header-anchor" href="https://blogs.igalia.com/scerveau/discover-gstreamer-full-static-mode/">#</a></h1>
<p>Since the <a href="https://dabrain34.github.io/2021/10/04/shrinking_gstreamer.html">gstreamer-full effort</a>, it was possible to create a shared library which will embed the GStreamer framework in addition to its set of plugins.</p>
<p>Within this effort, it was also possible to register the selected plugins/features automatically by calling the <code>gst_init</code> method in your application linking with <code>gstreamer-full</code>.</p>
<p>This method was offering a <code>gstreamer-full</code> package with library, headers and pc files but it was not possible to embed GStreamer statically in your application and use it transparently.</p>
<h1 id="gstvkvideoparser-a-standalone-solution" tabindex="-1">GstVkVideoParser: a standalone solution <a class="header-anchor" href="https://blogs.igalia.com/scerveau/discover-gstreamer-full-static-mode/">#</a></h1>
<p>In the journey to bring an open source solution for a video parser to the <a href="https://github.com/KhronosGroup/VK-GL-CTS">Vulkan Conformance Test Suite</a>, we chose first to use GStreamer as it was bringing all the parsing facilities
necessary to support the needed codecs such as H26x or VPx. This solution was supposed to be also cross platform and dragging as less as possible system dependencies.
Seen that GStreamer is usually dragging its own dependencies such as glib or orc and as we wanted to have a standalone <a href="https://github.com/Igalia/GstVkVideoParser">GstVkVideoParser</a> library supported on Windows, a little bit
of work and love was necessary to add this to GStreamer.</p>
<p>Unfortunately this solution has not be retained by the Vulkan Video TSG, not because it was not working but another parser has been made available and easy to integrate to the CTS at source level avoiding binary linkage, see Vulkan Video <a href="https://github.com/KhronosGroup/VK-GL-CTS/commit/e5db10e7ae436dbd9b46dad9518f3254dd6eeea2">change</a>.</p>
<h1 id="gstreamer-as-a-full-static-library" tabindex="-1">GStreamer as a full static library <a class="header-anchor" href="https://blogs.igalia.com/scerveau/discover-gstreamer-full-static-mode/">#</a></h1>
<p>With the <code>gstreamer-full</code> work, everything was almost ready to be used except to have <code>gstreamer-full</code> as a real static library and be able to link with it in any application.</p>
<p>Here is the <a href="https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/4128">MR</a> merged and the challenges taken up:</p>
<h2 id="adding-gst-full-target-type-static" tabindex="-1">Adding gst-full-target-type=static <a class="header-anchor" href="https://blogs.igalia.com/scerveau/discover-gstreamer-full-static-mode/">#</a></h2>
<p>To generate the <code>gstreamer-full</code> dependency which will be statically linked into the application, we decided to introduce a new gst meson options, <code>gst-full-target-type</code>.</p>
<p>By default the <code>gstreamer-full</code> will be built as a <em>shared library</em> as before.</p>
<p>By passing <code>gst-full-target-type=static</code>, only static object will be generated and a package config file will be generated for <code>gstreamer-full</code> allowing the application to avoid to know what static library it needs to add the link line.
The GStreamer build system will take care of enabling/disabling the features/libraries you (do/don't) need.</p>
<h2 id="initialize-the-plugins-features-automatically" tabindex="-1">Initialize the plugins/features automatically <a class="header-anchor" href="https://blogs.igalia.com/scerveau/discover-gstreamer-full-static-mode/">#</a></h2>
<p>To avoid multiple call necessary to initialize GStreamer, it was also necessary to call the <code>gst_init_static_plugins</code> along with <code>gst_init</code> only in full-static mode but it was leading to a build issue.</p>
<p>Indeed most of tools/examples/tests are linking with <code>libgstreamer-1.0</code> which owns <code>gst_init ()</code> but to facilitate the plugins registration, it was necessary to move all the tools build after the <code>gstreamer-full</code> stage. A first <a href="https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1581">MR</a> has been performed to let gstreamer tools be built against gstreamer-full
but additional work was necessary for some core tools or helpers such as <code>gst-transcoder</code> or <code>gst-plugin-scanner</code> to avoid a linking issue.</p>
<h2 id="disable-tests-and-examples" tabindex="-1">Disable tests and examples <a class="header-anchor" href="https://blogs.igalia.com/scerveau/discover-gstreamer-full-static-mode/">#</a></h2>
<p>In a future work all the tools/examples/tests should support the <code>full-static</code> mode but as GStreamer aims to be a shared object framework, we decided to leave this work for later and disable all the examples/tests in full static mode as most of the application using a tailor build won't need the examples and tests.</p>
<h2 id="windows-support" tabindex="-1">Windows support <a class="header-anchor" href="https://blogs.igalia.com/scerveau/discover-gstreamer-full-static-mode/">#</a></h2>
<p>One of the goal of this work was to provide a Windows library to the Vulkan CTS free of dependency, which has been achieved but some additional work might be necessary to support
all of the use case, the GStreamer framework offer, especially on supporting library-dependent plugins.</p>
<h2 id="give-me-an-example" tabindex="-1">Give me an example ... <a class="header-anchor" href="https://blogs.igalia.com/scerveau/discover-gstreamer-full-static-mode/">#</a></h2>
<p>In the <a href="https://github.com/Igalia/GstVkVideoParser">GstVkVideoParser project</a>, various jobs are building Linux and Windows versions generating a library without any GStreamer/glib dependencie, everything is embedded inside the library, as you can see in this <a href="https://github.com/Igalia/GstVkVideoParser/actions">GitHub' Actions</a>.</p>
<p>In this project, GStreamer is used as a meson subproject/wrap which allows to build GStreamer along of GstVkVideoParser. This can be possible easily by adding the following file to your meson project</p>
<p><em>subprojects/gstreamer-1.0.wrap</em></p>
<pre><code>[wrap-git]
directory=gstreamer-1.0
url=https://gitlab.freedesktop.org/gstreamer/gstreamer.git
revision=main
[provide]
dependency_names = gstreamer-1.0, gstreamer-base-1.0, gstreamer-video-1.0, gstreamer-audio-1.0
</code></pre>
<p>and then add the following lines to your <code>meson.build</code> to depend on <code>gstreamer-full</code></p>
<p><em>meson.build</em></p>
<pre><code>gstreamer_full_dep = dependency('gstreamer-full-1.0', fallback: ['gstreamer-1.0'], required :true)
</code></pre>
<p>In order to build a project, library or application which is using a tailored version of GStreamer you can follow this <a href="https://github.com/Igalia/GstVkVideoParser/blob/main/configure_gst_full.py">configure example</a>:</p>
<pre><code>$ meson buildfull-static --default-library=static --force-fallback-for=gstreamer-1.0,glib,libffi,pcre2 -Dauto_features=disabled -Dglib:tests=false -Djson-glib:tests=false -Dpcre2:test=false -Dvkparser_standalone=enabled -Dgstreamer-1.0:libav=disabled -Dgstreamer-1.0:ugly=disabled -Dgstreamer-1.0:ges=disabled -Dgstreamer-1.0:devtools=disabled -Dgstreamer-1.0:default_library=static -Dgstreamer-1.0:rtsp_server=disabled -Dgstreamer-1.0:gst-full-target-type=static_library -Dgstreamer-1.0:gst-full-libraries=gstreamer-video-1.0, gstreamer-audio-1.0, gstreamer-app-1.0, gstreamer-codecparsers-1.0 -Dgst-plugins-base:playback=enabled -Dgst-plugins-base:app=enabled -Dgst-plugins-bad:videoparsers=enabled -Dgst-plugins-base:typefind=enabled
</code></pre>
<p>In this case we are disabling everything in GStreamer by using <code>-Dauto_features=disabled</code> and some enabled features such as <code>ges</code>, <code>libav</code>, etc. and enable only what we need as plugins, <code>playback</code>, <code>app</code>, <code>videoparsers</code> and <code>typefind</code>.</p>
<p>And finally we are enabling the static build with <code>--default-library=static</code> and <code>-Dgstreamer-1.0:gst-full-target-type=static_library</code>.</p>
<h2 id="next" tabindex="-1">Next ... <a class="header-anchor" href="https://blogs.igalia.com/scerveau/discover-gstreamer-full-static-mode/">#</a></h2>
<p>As you can see, it's quite easy now to build an application and depends on <code>gstreamer-full</code> static build, but there is still some issues to address such as the plugins dependencies which might be not static and some other platform specific issue such as the <code>gstreamer-full</code> symbols export on Windows.</p>
<p>You can follow some open issues such as:</p>
<ul>
<li>
<p><a href="https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/1629">static build is failing on Windows platform</a></p>
</li>
<li>
<p><a href="https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/2104">gstfull: Build only static library on Windows</a></p>
</li>
</ul>
<p>As usual, if you would like to learn more about Vulkan Video, GStreamer or any other open multimedia framework, please contact <a href="https://www.igalia.com/">us</a>!</p>
ESExtractor: how to integrate a dependency-free library to the Khronos CTS2023-04-19T00:00:00Zhttps://blogs.igalia.com/scerveau/esextractor-how-to-integrate-a-dependency-free-library-to-the-khronos-cts/<h1 id="esextractor-how-to-integrate-a-dependency-free-library-to-the-khronos-cts" tabindex="-1">ESExtractor, how to integrate a dependency-free library to the Khronos CTS <a class="header-anchor" href="https://blogs.igalia.com/scerveau/esextractor-how-to-integrate-a-dependency-free-library-to-the-khronos-cts/">#</a></h1>
<p>Since the <a href="https://github.com/KhronosGroup/VK-GL-CTS">Vulkan CTS</a> is now able to test and check <a href="https://www.khronos.org/news/press/vulkan-sdk-is-vulkan-video-ready">Vulkan Video support</a>
including video decoding, it was necessary to define the kind of media container to be used inside the test cases and the library
to extract the necessary encoded data.</p>
<p>In a first attempt, the <a href="https://ffmpeg.org/">FFmpeg media toolkit</a> had been chosen to extract the video packets from the A/V ISO base media
format chosen as a container reference. This library was provided as a binary package and loaded dynamically at each
test run.</p>
<p>As Vulkan video aims to test only <em>video</em> contents, it was not necessary to choose a complex media container,
so first all the videos were converted to the elementary stream format for
H264 and H265 contents.
This is a very elementary format based on MPEG start codes and NAL unit identification.</p>
<p>To avoid an extra multimedia solution integrable only with binaries, a first attempt to replace FFmpeg was,
to use GStreamer and an in-house helper library called <a href="https://github.com/Igalia/GstVkVideoParser/tree/main/lib/demuxeres">demuxeres</a>.
It was smaller but needed to be a binary still to avoid the glib/gstreamer system dependencies (self contained library).
it was a no-go still because the binary package would be awkward to support on the various platforms targetted by the the Khronos CTS.</p>
<p>So at <a href="https://www.igalia.com/">Igalia</a>, we decided to implement a minimal, dependency-free, custom library, written in C++
to be compliant with the Khronos CTS and simple to integrate into any build system.</p>
<p>This library is called <a href="https://github.com/Igalia/ESExtractor">ESExtractor</a></p>
<h2 id="what-is-esextractor" tabindex="-1">What is ESExtractor ? <a class="header-anchor" href="https://blogs.igalia.com/scerveau/esextractor-how-to-integrate-a-dependency-free-library-to-the-khronos-cts/">#</a></h2>
<p>ESExtractor aims to be a simple elementary stream extractor. For the first revision it was able to extract video data from
a file in the <a href="https://en.wikipedia.org/wiki/Network_Abstraction_Layer">NAL standard</a>.
The first official release was <a href="https://github.com/Igalia/ESExtractor/releases/tag/release-v0.2.4">0.2.4</a>. In this release,
only the NAL was supported with both the H264 and H265 streams supported.</p>
<p>As Vulkan Video aims to support more than H264 and H265 including format such as AV1 or VP9, the ESExtractor had to support multiple format.
A redesign has been started to support multiple format and is now available in the version 0.3.2.</p>
<h2 id="how-esextractor-works" tabindex="-1">How ESExtractor works <a class="header-anchor" href="https://blogs.igalia.com/scerveau/esextractor-how-to-integrate-a-dependency-free-library-to-the-khronos-cts/">#</a></h2>
<p>A simple C interface is provided to maximise portability and use by other languages.</p>
<h3 id="es-extractor-new" tabindex="-1">es_extractor_new <a class="header-anchor" href="https://blogs.igalia.com/scerveau/esextractor-how-to-integrate-a-dependency-free-library-to-the-khronos-cts/">#</a></h3>
<pre><code>ESExtractor extractor = es_extractor_new(filePath, "options"));
</code></pre>
<p>This interface returns the main object which will give you access to the packets according to the file path and the options given in the arguments.
This interface returns a ready to use object where the stream has been initially parsed to determine the kind of video during
the object creation.</p>
<p>Then you can check the video format with:</p>
<pre><code>ESEVideoFormat eVideoFormat = es_extractor_video_format(extractor);
</code></pre>
<p>or the video codec with:</p>
<pre><code>ESEVideoCodec eVideoCodec = es_extractor_video_codec(extractor);
</code></pre>
<p>It supports H264, H265, AV1 and VP9 for now.</p>
<h3 id="es-extractor-read-packet" tabindex="-1">es_extractor_read_packet <a class="header-anchor" href="https://blogs.igalia.com/scerveau/esextractor-how-to-integrate-a-dependency-free-library-to-the-khronos-cts/">#</a></h3>
<p>This API is the main function to retrieve the available packets from the file. Each time this API is called,
the library will return the next available packet according to the format and the specific alignment (ie NAL) and
a status to let the application decide what to do next. The packet should be freed using <code>es_extractor_clear_packet</code>.</p>
<h2 id="has-ci-powered-by-github" tabindex="-1">Has CI powered by github <a class="header-anchor" href="https://blogs.igalia.com/scerveau/esextractor-how-to-integrate-a-dependency-free-library-to-the-khronos-cts/">#</a></h2>
<p>To test the library usage, we have implementing a testing framework in addition to a CI infrastructure
As github offers a very powerful worklow, we decided to use this platform to test the library on various architectures and platforms.
The <a href="https://github.com/Igalia/ESExtractor/actions">CI</a> is now configured to release packages for 64 and 32 bits on Linux and Windows.</p>
<p>As usual, if you would like to learn more about Vulkan Video, ESExtractor or any other open multimedia framework, please contact <a href="https://www.igalia.com/">us</a>!</p>