Review of Igalia's Multimedia Activities (2018/H2)
This is the first semiyearly report about Igalia’s activities around multimedia, covering the second half of 2018.Great length of this report was exposed in Phil’s talk surveying multimedia development in WebKitGTK and WPE:
WebKit Media Source Extensions (MSE) #
MSE is a specification that allows JS to generate media streams for playback for Web browsers that support HTML 5 video and audio.
Last semester we upstreamed the support to WebM
format in WebKitGTK with the
related patches in GStreamer, particularly in qtdemux, matroskademux
elements.
WebKit Encrypted Media Extensions (EME) #
EME is a specification for enabling playback of encrypted content in Web browsers that support HTML 5 video.
In a downstream project for WPE WebKit we managed to have almost full test coverage in the YoutubeTV 2018 test suite.
We merged our contributions in upstream, WebKit and GStreamer, most of what is legal to publish, for example, making demuxers aware of encrypted content and make them to send protection events with the initialization data and the encrypted caps, in order to select later the decryption key.
We started to coordinate the upstreaming process of a new implementation of CDM (Content Decryption Module) abstraction and there will be even changes in that abstraction.
Lighting talk about EME implementation in WPE/WebKitGTK in GStreamer Conference 2018.
WebKit WebRTC #
WebRTC consists of several interrelated APIs and real time protocols to enable Web applications and sites to captures audio, or A/V streams, and exchange them between browsers without requiring an intermediary.
We added GStreamer interfaces to LibWebRTC, to use it for the network part, while using GStreamer for the media capture and processing. All that was upstreamed in 2018 H2.
Thibault described thoroughly the tasks done for this achievement.
Talk about WebRTC implementation in WPE/WebKitGTK in WebEngines hackfest 2018.
Servo/media #
Servo is a browser engine written in Rust designed for high parallelization and high GPU usage.
We added basic support for <video> and <audio> media elements in Servo.
Later on, we added the GStreamerGL bindings for Rust in
gstreamer-rs to render
GL textures from the GStreamer pipeline in Servo.
Lighting talk in the GStreamer Conference 2018.
GstWPE #
Taking an idea from the GStreamer Conference, we developed a GStreamer source element that wraps WPE. With this source element, it is possible to blend a web page and video in a single video stream; that is, the output of a Web browser (to say, a rendered web page) is used as a video source of a GStreamer pipeline: GstWPE. The element is already merged in the gst-plugins-bad repository.
Talk about GstWPE in FOSDEM 2019
GStreamer VA-API and gst-MSDK #
At last, but not the least, we continued helping with the maintenance of GStreamer-VAAPI and gst-msdk, with code reviewing and on-going migration of the internal library to GObject.
Other activities #
The second half of 2018 was also intense in terms of conferences and hackfest for the team:
- GStreamer conference in Edinburgh.
- Web Engines Hackfest in Coruña.
- WebKit meeting in Silicon Valley
- Finally, we participated as mentors in the last Google Summer of Code. Georg Ottinger, among other completed tasks, sketched an AV1 parser for GStreamer, though it needs a couple more iterations to get it merged.
Thanks to bear with us along all this blog post and to keeping under your radar our work.