This blog post is a review of the various activities the Igalia Multimedia team was involved in during the first half of 2020.
Our previous reports are:
Just before a new virus turned into pandemics we could enjoy our traditional FOSDEM. There, our colleague Phil gave a talk about many of the topics covered in this report.
wpesrc element, produces a video texture representing a web page rendered off-screen by WPE.
We have worked on a new iteration of the GstWPE demo, focusing on one-to-many, web-augmented overlays, broadcasting with WebRTC and Janus.
Also, since the merge of gstwpe plugin in gst-plugins-bad (staging area for new elements) new users have come along spotting rough areas and improving the element along the way.
GStreamer Editing Services (GES) is a library that simplifies the creation of multimedia editing applications. It is based on the GStreamer multimedia framework and is heavily used by Pitivi video editor.
Implemented frame accuracy in the GStreamer Editing Services (GES)
As required by the industry, it is now possible to reference all time in frame number, providing a precise mapping between frame number and play time. Many issues were fixed in GStreamer to reach the precision enough for make this work. Also intensive regression tests were added.
Implemented time effects support in GES
Important refactoring inside GStreamer Editing Services have happened to allow cleanly and safely change playback speed of individual clips.
Implemented reverse playback in GES
Several issues have been fixed inside GStreamer core elements and base classes in order to support reverse playback. This allows us to implement reliable and frame accurate reverse playback for individual clips.
Implemented ImageSequence support in GStreamer and GES
Since OpenTimelineIO implemented ImageSequence support, many users in the community had said it was really required. We reviewed and finished up
imagesequencesrc element, which had been awaiting review for years.
This feature is now also supported in the OpentimelineIO GES adapater.
Optimized nested timelines preroll time by an order of magnitude
Caps negotiation, done while the pipeline transitions from pause state to playing state, testing the whole pipeline functionality, was the bottleneck for nested timelines, so pipelines were reworked to avoid useless negotiations. At the same time, other members of GStreamer community have improved caps negotiation performance in general.
Last but not least, our colleague Thibault gave a talk in The Pipeline Conference about The Motion Picture Industry and Open Source Software: GStreamer as an Alternative, explaining how and why GStreamer could be leveraged in the motion picture industry to allow faster innovation, and solve issues by reusing all the multi-platform infrastructure the community has to offer.
There has been a lot of work on WebKit multimedia, particularly for WebKitGTK and WPE ports which use GStreamer framework as backend.
WebKit Flatpak SDK
But first of all we would like to draw readers attention to the new WebKit Flatpak SDK. It was not a contribution only from the multimedia team, but rather a joint effort among different teams in Igalia.
Before WebKit Flatpak SDK, JHBuild was used for setting up a WebKitGTK/WPE environment for testing and development. Its purpose to is to provide a common set of well defined dependencies instead of relying on the ones available in the different Linux distributions, which might bring different outputs. Nonetheless, Flatpak offers a much more coherent environment for testing and develop, isolated from the rest of the building host, approaching to reproducible outputs.
Another great advantage of WebKit Flatpak SDK, at least for the multimedia team, is the possibility of use
gst-build to setup a custom GStreamer environment, with latest master, for example.
Now, for sake of brevity, let us sketch an non-complete list of activities and achievements related with WebKit multimedia.
- Upstreamed external-oes support for hardware-accelerated video rendering. On some GPUs the presence of native YUV samplers allows the video sink to optimize rendering.
- Allow video assets in <img>. This can be used as replacement for GIFs. For example, here is a demo and a blog post to explain it lenghtly.
- Whatsapp fixes related with blob URIs playback and buffering.
- Added GstGLContext which a singleton wrapper of WebKit’s GL shared context wrapper.
- Progress with GStreamer support for WebInspector. It adds a new tab in the inspector allowing users to view pipeline graphs.
- Progress with Internal audio mixer. It is related with reducing the number of PulseAudio clients WebKit creates, by doing mixing internally and maintain only one connection with PulseAudio.
- Ongoing work on sandboxing GStreamer pipelines in the new WebKit’s GPUProcess.
- General WebKit multimedia and layout tests bugfixing.
Media Source Extensions (MSE)
- Fixed delayed audio and seek issues.
- Preparation for the new MSE backend involved a large playbin3 track switching.
Encrypted Media Extension (EME)
One of the major results of this first half, is the upstream of ThunderCDM, which is an implementation of a Content Decryption Module, providing Widevine decryption support. Recently, our colleague Xabier, published a blog post on this regard.
And it has enabled client-side video rendering support, which ensures video frames remain protected in GPU memory so they can’t be reached by third-party. This is a requirement for DRM/EME.
- Collaborated with the upgrade of LibWebRTC to M82.
Though we normally contribute in GStreamer with the activities listed above, there are other tasks not related with WebKit. Among these we can enumerate the following:
- Contributed to the new stateless GStreamer codecs library.
- A new H264 VAAPI video decoder implementation
- Helped with the AV1 parser review, a task that has been followed since we were Google Summer of Code mentors last year.
- Contributed a new Rust GStreamer plugin for audio noise removal support
- Reviewed a lot of patches.
- Support for media-driver (iHD), the new VAAPI driver for Intel, mostly for Gen9 onwards. There are a lot of features with this driver.
- A new
- Deep code cleanups. Among these we would like to mention:
- Added quirk mechanism for different backends.
- Change base classes to
GstMiniObjectof most of classes and buffers types.
- Enhanced caps negotiation given current driver’s constraints
The multimedia team in Igalia has keep working, along the first half of this strange year, in our three main areas: browsers (mainly on WebKitGTK and WPE), video editing and GStreamer framework.
We worked adding and enhancing WebKitGTK and WPE multimedia features in order to offer a solid platform for media providers.
We have enhanced the Video Editing support in GStreamer.
And, along these tasks, we have contribuited as much in GStreamer framework, particulary in hardware accelerated decoding and encoding and VA-API.