Igalia Multimedia achievements (2024)
As 2024 draws to a close, it’s a perfect time to reflect on the year’s accomplishments done by the Multimedia team in Igalia. In our consideration, there were three major achievements:
- WebRTC’s support in WPE/WebKitGTK with GStreamer.
- GES maturity improved for real-life use-cases.
- Vulkan Video development and support in GStreamer.
WebRTC support in WPE/WebKitGTK with GStreamer #
WPE and WebKitGTK are WebKit ports maintained by Igalia, the former for embedded devices and the latter for applications with a full-featured Web integration.
WebRTC is a web API that allows real-time communication (RTC) directly between web browser and applications. Examples of these real-time communications are video conferencing, cloud gaming, live-streaming, etc.
Some WebKit ports support libwebrtc, an open-source library that implements the WebRTC specification, developed and maintained by Google. WPE and WebKitGTK originally also supports libwebrtc, but we started to use also GstWebRTC, a set of GStreamer plugins and libraries that implement WebRTC, which adapts perfectly to the multimedia implementation in both ports, also in GStreamer.
This year the fruits of this work have been unlocked by enabling Amazon Luna gaming:
And also enabling a CAD modeling, server-side rendered service, known as Zoo:
WebKit Multimedia #
WebKit made significant improvements in multimedia handling, addressing various
issues to enhance stability and playback quality. Key updates include preventing
premature play() calls during seeking, fixing memory leaks. The management of
track identifiers was also streamlined by transitioning from string-based to
integer-based IDs. Additionally, GStreamer-related race conditions were resolved
to prevent hangs during playback state transitions. Memory leaks in WebAudio and
event listener management were addressed, along with a focus on memory usage
optimizations.
The handling of media buffering and seeking was enhanced with buffering
hysteresis for smoother playback. Media Source
Extensions (MSE) behavior was refined to
improve playback accuracy, such as supporting markEndOfStream() before
appendBuffer() and simplifying playback checks. Platform-specific issues were
also tackled, including AV1 and Opus support for encrypted media and better
detection of audio sinks. And other improvements on multimedia performance and
efficiency.
GES maturity improved for real-live use-cases #
GStreamer Editing Services (GES) is a set of GStreamer plugins and a library that allow non-linear video editing. For example, GES is what’s behind of Pitivi, the open source video editor application.
Last year, GES was deployed in web-based video editors, where the actual video processing is done server-side. These projects allowed, in great deal, the enhancement and maturity of the library and plugins.
Tella is a browser-based tool that allow to screen record and webcam, without any extra software. Finished the recording, the user can edit, in the browser, the video and publish it.
Sequence is a complete, browser-based, video editor with collaborative features. GES is used in the backend to render the editing operations.
Vulkan Video development and GStreamer support #
The last but not the least, this year we continue our work with the Vulkan Video ecosystem by working the task subgroup (TSG) enabling H.264/H.265 encoding, and AV1 decoding and encoding.
Early this year we delivered a talk in the Vulkanised about our work, which ranges from the Conformance Test Suite (CTS), Mesa, and GStreamer.
Conclusion #
As we wrap up 2024, it’s clear that the year has been one of significant progress, driven by innovation and collaboration. Here’s to continuing the momentum and making 2025 even better!
- Previous: GStreamer Conference 2024