opengl – Herostratus’ legacy

Digging further into Flatpak with NVIDIA

As you may know the development environment used by WebKitGTK and WPE is based on Flatpak. I feel hacking software within Flatpak like teleoperating a rover in Mars, since I have to go through Flatpak commands to execute the commands I want to execute. The learning curve is steeper in exchange of a common development environment.

I started to work on another project where is required to use an NVIDIA GPU, without stopping to work on WebKitGTK/WPE. So I needed to use the card within Flatpak, and it’s well known that, currently, that setup is not available out-of-the-box. Furthermore, I have to use a very specific version of the graphics card drive for Vulkan.

This is the story of how I make it work.

My main reference is, of course, the blog post of my colleague TingPing: Using host Nvidia driver with Flatpak, besides flatpak’s NVIDIA GL runtime platform.

As TingPing explained, flatpak does not use host libraries, that’s why it might need runtimes and extensions for specific hardware setups, with the libraries for user-space, such as NVIDIA GL platform runtime. And it must have the same version as the one running in kernel.

NVIDIA GL platform extension is a small project which generates flatpak’s runtimes for every public NVIDIA driver. The interesting part is that those runtimes are not created in building time, but at install-time. When the user installs the runtime, a driver blob is downloaded from NVIDIA servers (see --extra-data in flatpak build-finish for reference), and a small program is executed, which extracts the embedded tarball in the blob, and from it, it extracts the required libraries. In a few words, initially, the runtime is composed only by a definition of the file to
download, and the small program that populates the flatpak’s filesystem at install-time.

The trick here, that took me a lot to be aware, is that this small program has to be statically compiled, since it has be executed regardless the available runtime.

This little program uses libarchive to extract the libraries from NVIDIA’s tarball, but libarchive is not available statically in any flatpak’s SDK. Furthermore, our use of libarchive will depend on libz and liblzma, both statically compile as well. Gladly, there’s only one, very old version, obsolete, of freedesktop SDK, which offers static versions of libz and liblzma: 1.6. And that’s why org.freedesktop.Platform.GL.nvidia demands that specific old version of the SDK. Then, the manifest of the extension contains basically the static compilation of libarchive and the static compilation of the next-to-be apply_extra.

Update: There’s a merge request to use current freedesktop SDK 21.08, which, basically, builds statically libz and liblzma, besides libarchive.

I needed to modify org.freedesktop.Platform.GL.nvidia sources a bit, since it, by default, consist in a big loop of downloading, hashing, templating a json manifest, and building, for every supported driver. But, as my case is just one custom driver, I don’t want to waste time in that loop. The hack to achieve it is fairly simple:

diff --git a/versions.sh b/versions.sh
index 8b72664..86686c0 100755
--- a/versions.sh
+++ b/versions.sh
@@ -15,4 +15,5 @@ TESLA_VERSIONS="450.142.00 450.119.04 450.51.06 450.51.05 440.118.02 440.95.01 4
# Probably never: https://ahayzen.com/direct/flathub_downloads_only_nvidia_runtimes.txt
UNSUPPORTED_VERSIONS="390.147 390.144 390.143 390.141 390.138 390.132 390.129 390.116 390.87 390.77 390.67 390.59 390.48 390.42 390.25 390.12 387.34 387.22 387.12 384.130 384.111 384.98 384.90 384.69 384.59 384.47 381.22 381.09 378.13 375.82 375.66 375.39 375.26 370.28 367.57"

-DRIVER_VERSIONS="$BETA_VERSIONS $VULKAN_VERSIONS $NEW_FEATURE_VERSIONS $PRODUCTION_VERSIONS $LEGACY_VERSIONS $TESLA_VERSIONS $UNSUPPORTED_VERSIONS"
+#DRIVER_VERSIONS="$BETA_VERSIONS $VULKAN_VERSIONS $NEW_FEATURE_VERSIONS $PRODUCTION_VERSIONS $LEGACY_VERSIONS $TESLA_VERSIONS $UNSUPPORTED_VERSIONS"
+DRIVER_VERSIONS="470.XX.XX"

But in order to make it work, it needs a file in data/ directory with the specification of the file to download, with the format: NAME:SHA256:DOWNLOAD-SIZE:INSTALL-SIZE:URL.

--- /dev/null
+++ b/data/nvidia-470.XX.XX-x86_64.data
@@ -0,0 +1 @@
+:34...checksum-sha264...:123456789::http://compu.home.arpa/NVIDIA/NVIDIA-Linux-x86_64-470.XX.XX.run

The last parameter is the URL where the driver shall be downloaded. In my case is a local server to ease the testing.

Long story short, the command to execute are:

To setup the building environment:

$ flatpak install org.freedesktop.Sdk//1.6 org.freedesktop.Platform//1.6

To build the flatpak repository and package:

$ make

The command will output a repo directory in the current one. There’s where the generated flatpak package is stored.

To install the local repository and the extension:

$ flatpak --user remote-add --no-gpg-verify nvidia-local repo
$ flatpak -v install nvidia-local org.freedesktop.Platform.GL.nvidia-470-XX-XX

To remove the obsolete SDK and platform once built:

$ flatpak uninstall org.freedesktop.Sdk//1.6 org.freedesktop.Platform//1.6

To remove the local repository and the extension if something went wrong:

$ flatpak -v uninstall org.freedesktop.Platform.GL.nvidia-470-62-15
$ flatpak --user remote-delete nvidia-local

One way to verify if the libraries are installed correctly and if they match with the driver running in the kernel’s host, is to install and run GreenWithEnvy:

$ flatpak install com.leinardi.gwe
$ flatpak run com.leinardi.gwe

If you want to install the driver in your WebKit development environment, you just need to set the environment variable FLATPAK_USER_DIR:

$ FLATPAK_USER_DIR=~/WebKit/WebKitBuild/UserFlatpak flatpak --user remote-add --no-gpg-verify nvidia-local repo
$ FLATPAK_USER_DIR=~/WebKit/WebKitBuild/UserFlatpak flatpak -v install nvidia-local org.freedesktop.Platform.GL.nvidia-470-XX-XX

Rust bindings for GStreamerGL: Memoirs

Rust is a great programming language but the community around it’s just amazing. Those are the ingredients for the craft of useful software tools, just like Servo, an experimental browser engine designed for tasks isolation and high parallelization.

Both projects, Rust and Servo, are funded by Mozilla.

Thanks to Mozilla and Igalia I have the opportunity to work on Servo, adding it HTML5 multimedia features.

First, with the help of Fernando Jiménez, we finished what my colleague Philippe Normand and Sebastian Dröge (one of my programming heroes) started: a media player in Rust designed to be integrated in Servo. This media player lives in its own crate: servo/media along with the WebAudio engine. A crate, in Rust jargon, is like a library. This crate is (very ad-hocly) designed to be multimedia framework agnostic, but the only backend right now is for GStreamer. Later we integrated it into Servo adding an initial support for audio and video tags.

Currently, servo/media passes, through a IPC channel, the array with the whole frame to render in Servo. This implies, at least, one copy of the frame in memory, and we would like to avoid it.

For painting and compositing the web content, Servo uses WebRender, a crate designed to use the GPU intensively. Thus, if instead of raw frame data we pass OpenGL textures to WebRender the performance could be enhanced notoriously.

Luckily, GStreamer already supports the uploading, downloading, painting and composition of video frames as OpenGL textures with the OpenGL plugin and its OpenGL Integration library. Even more, with plugins such as GStreamer-VAAPI, Gst-OMX (OpenMAX), and others, it’s possible to process video without using the main CPU or its mapped memory in different platforms.

But from what’s available in GStreamer to what it’s available in Rust there’s a distance. Nonetheless, Sebastian has putting a lot of effort in the Rust bindings for GStreamer, either for applications and plugins, sadly, GStreamer’s OpenGL Integration library (GstGL for short) wasn’t available at that time. So I rolled up my sleeves and got to work on the bindings.

These are the stories of that work.

As GStreamer shares with GTK+ the GObject framework and its introspection mechanism, both projects have collaborated on the required infrastructure to support Rust bindings. Thanks to all the GNOME folks who are working on the intercommunication between Rust and GObject. The quest has been long and complex, since Rust doesn’t map all the object oriented concepts, and GObject, being a set of practices and software helpers to do object oriented programming with C, its usage is not homogeneous.

The Rubicon that ease the generation of Rust bindings for GObject-based projects is GIR, a tool, written in Rust, that reads gir files, along with metadata in toml, and outputs two types of bindings: sys and api.

Rust can call external functions through FFI (foreign function interface), which is just a declaration of a C function with Rust types. But these functions are considered unsafe. The sys bindings, are just the exporting of the C function for the library organized by the library’s namespace.

The next step is to create a safe and rustified API. This is the api bindings.

As we said, GObject libraries are quite homogeneous, and even following the introspection annotations, there will be cases where GIR won’t be able to generate the correct bindings. For that reason GIR is constantly evolving, looking for a common way to solve the corner cases that exist in every GObject project. For example, these are my patches in order to generate the GstGL bindings.

The done tasks were:

For this document we assume that the reader has a functional Rust setup and they know the basic concepts.

Clone and build gir

$ cd ~/ws
$ git clone https://github.com/gtk-rs/gir.git
$ cd gir
$ cargo build --release

The reason to build gir in release mode is because, otherwise would be very slow.

For sys bindings.

These kind of bindings are normally straight forward (and unsafe) since they only map the C API to Rust via FFI mechanism.

$ cd ~/ws
$ git clone https://gitlab.freedesktop.org/gstreamer/gstreamer-rs-sys.git
$ cd gstreamer-rs-sys
$ cp /usr/share/gir-1.0/GstGL-1.0.gir gir-files/

Verify if the gir file is more o less correct
1. If there something strange, we should fix the code that generated it.
2. If that is not possible, the last resource is to fix the gir file directly, which is just XML, not manually but through a script using xmlstartlet. See fix.sh in gtk-rs as example.
Create the toml file with the metadata required to create the bindings. In other words, this file contains the exceptions, rules and options used by the tool to generated the bindings. See Gir_GstGL.toml in gstreamer-rs-sys as example. The documentation of the toml file is in the gir’s README.md file.

$ ~/ws/gir/target/release/gir -c Gir_GstGL.toml

This command will generate, as specified in the toml file (target_path), a crate in the directory named gstreamer-gl-sys.

Api bindings.

These type of bindings may require more manual work since their purpose is to offer a rustified API of the library, with all its syntactic sugar, semantics, and so on. But in general terms, the process is similar:

$ cd ~/ws
$ git clone https://gitlab.freedesktop.org/gstreamer/gstreamer-sys.git
$ cd gstreamer-sys
$ cp /usr/share/gir-1.0/GstGL-1.0.gir gir-files/

Again, it would be possible to end up applying fixes to the gir file through a fix.sh script using xmlstartlet.

And again, the confection of the toml file might take a lot of time, by trial and error, by cleaning and tidying up the API. See Gir_GstGL.toml in gstreamer-rs as example.

$ ~/ws/gir/target/release/gir -c Gir_GstGL.toml

A good way to test your bindings is by crafting a test application, which shows how to use the API. Personally I devoted a ton of time in the test application for GstGL, but worth it. It made me aware of a missing part in the crate used for GL applications in Rust, named Glutin, which was a way to get the used EGLDisplay. So also worked on that and sent a pull request that was recently merged. The sweets of the free software development.

Nowadays I’m integrating GstGL API in servo/media and later, Servo!