Review of Igalia Multimedia activities (2022)
We, Igalia’s multimedia team, would like to share with you our list of achievements along the past 2022.## WebKit Multimedia
WebRTC #
Phil already wrote a first blog post, of a series, on this regard: WebRTC in WebKitGTK and WPE, status updates, part I. Please, be sure to give it a glance, it has nice videos.
Long story short, last year we started to support Media Capture and Streams in WebKitGTK and WPE using GStreamer, either for input devices (camera and microphone), desktop sharing, webaudio, and web canvas. But this is just the first step. We are currently working on RTCPeerConnection, also using GStreamer, to share all these captured streams with other web peers. Meanwhile, we’ll wait for the second episode of Phil’s series :)
MediaRecorder #
We worked in an initial implementation of MediaRecorder with GStreamer (1.20 or superior). The specification goes about allowing a web browser to record a selected stream. For example, a voice-memo or video application which could encode and upload a capture of your microphone / camera.
Gamepad #
While WebKitGTK already has Gamepad support, WPE lacked it. We did the implementation last year, and there’s a blog post about it: Gamepad in WPEWebKit, with video showing a demo of it.
Capture encoded video streams from webcams #
Some webcams only provide high resolution frames encoded in H.264 or so. In order to support these resolutions with those webcams we added the support for negotiate of those formats and decode them internally to handle the streams. Though we are just at the beginning of more efficient support.
Flatpak SDK maintenance #
A lot of effort went to maintain the Flatpak SDK for WebKit. It is a set of runtimes that allows to have a reproducible build of WebKit, independently of the used Linux distribution. Nowadays the Flatpak SDK is used in Webkit’s EWS, and by many developers.
Among all the features added during the year we can highlight added Rust support, a full integrity check before upgrading, and offer a way to override dependencies as local projects.
MSE/EME enhancements #
As every year, massive work was done in WebKit ports using GStreamer for Media Source Extensions and Encrypted Media Extensions, improving user experience with different streaming services in the Web, such as Odysee, Amazon, DAZN, etc.
In the case of encrypted media, GStreamer-based WebKit ports provide the stubs to communicate with an external Content Decryption Module (CDM). If you’re willing to support this in your platform, you can reach us.
Also we worked in a video demo showing how MSE/EME works in a Raspberry Pi 3 using WPE:
WebAudio demo #
We also spent time recording video demos, such as this one, showing WebAudio using WPE on a desktop computer.
GStreamer #
We managed to merge a lot of bug fixes in GStreamer, which in many cases can be harder to solve rather than implementing new features, though former are more interesting to tell, such as those related with making Rust the main developing language for GStreamer besides C.
Rust bindings and GStreamer elements for Vonage Video API / OpenTok #
OpenTok is the legacy name of Vonage Video API, and is a PaaS (Platform As a Service) to ease the development and deployment of WebRTC services and applications.
We published our work in Github of Rust bindings both for the Client SDK for Linux and the Server SDK using REST API, along with a GStreamer plugin to publish and subscribe to video and audio streams.
GstWebRTCSrc #
In the beginning there was
webrtcbin, an
element that implements the majority of W3C
RTCPeerConnection API.
It’s so flexible and powerful that it’s rather hard to use for the most common
cases. Then appeared
webrtcsink,
a wrapper of webrtcbin, written in Rust, which receives GStreamer streams
which will be offered and streamed to web peers. Later on, we developed
webrtcsrc,
the webrtcsink counterpart: an element which source pads push streams from web
peers, such as another browser, and forward those Web streams as GStreamer ones
in a pipeline. Both webrtcsink and webrtcsrc are written in Rust.
Behavior-Driven Development test framework for GStreamer #
Behavior-Driven Development is gaining relevance with tools like Cucumber for Java and its domain specific language, Gherkin to define software behaviors. Rustaceans have picked up these ideas and developed cucumber-rs. The logical consequence was obvious: Why not GStreamer?
Last year we tinkered with GStreamer-Cucumber, a BDD to define behavior tests for GStreamer pipelines.
GstValidate Rust bindings #
There have been some discussion if BDD is the best way to test GStreamer pipelines, and there’s GstValidate, and also, last year, we added its Rust bindings.
GStreamer Editing Services #
Though not everything was Rust. We work hard on GStreamer’s nuts and bolts.
Last year, we gathered the team to hack GStreamer Editing Services, particularly to explore adding OpenGL and DMABuf support, such as downloading or uploading a texture before processing, and selecting a proper filter to avoid those transfers.
GstVA and GStreamer-VAAPI #
We helped in the maintenance of GStreamer-VAAPI and the development of its near replacement: GstVA, adding new elements such as the H.264 encoder, the compositor and the JPEG decoder. Along with participation on the debate and code reviewing of negotiating DMABuf streams in the pipeline.
Vulkan decoder and parser library for CTS #
You might have heard about Vulkan has now integrated in its API video decoding, while encoding is currently work-in-progress. We devoted time on helping Khronos with the Vulkan Video Conformance Tests (CTS), particularly with a parser based on GStreamer and developing a H.264 decoder in GStreamer using Vulkan Video API.
You can check the presentation we did last Vulkanised.
WPE Android Experiment #
In a joint adventure with Igalia’s Webkit team we did some experiments to port WPE to Android. This is just an internal proof of concept so far, but we are looking forward to see how this will evolve in the future, and what new possibilities this might open up.
If you have any questions about WebKit, GStreamer, Linux video stack, compilers, etc., please contact us.