Review of the Igalia Multimedia team Activities (2019/H2)
This blog post is a review of the various activities the Igalia Multimedia team was involved along the second half of 2019.Here are the previous 2018/H2 and 2019/H1 reports.
GstWPE #
Succinctly, GstWPE is a GStreamer plugin which allows to render web-pages as a video stream where it frames are GL textures.
Phil, its main author, wrote a blog post explaining at detail what is GstWPE
and its possible
use-cases.
He wrote a
demo too,
which grabs and previews a live stream from a webcam session and blends it with
an overlay from wpesrc, which displays HTML content. This composited live
stream can be broadcasted through YouTube or Twitch.
These concepts are better explained by Phil himself in the following lighting talk, presented at the last GStreamer Conference in Lyon:
Video Editing #
After implementing a deep integration of the GStreamer Editing Services (a.k.a GES) into Pixar’s OpenTimelineIO during the first half of 2019, we decided to implement an important missing feature for the professional video editing industry: nested timelines.
Toward that goal, Thibault worked with the GSoC student Swayamjeet Swain to implement a flexible API to support nested timelines in GES. This means that users of GES can now decouple each scene into different projects when editing long videos. This work is going to be released in the upcoming GStreamer 1.18 version.
Henry Wilkes also implemented the support for nested timeline in OpenTimelineIO making GES integration one of the most advanced one as you can see on that table:
| Feature | OTIO | EDL | FCP7 XML | FCP X | AAF | RV | ALE | GES |
|---|---|---|---|---|---|---|---|---|
| Single Track of Clips | ✔ | ✔ | ✔ | ✔ | ✔ | W-O | ✔ | ✔ |
| Multiple Video Tracks | ✔ | ✖ | ✔ | ✔ | ✔ | W-O | ✔ | ✔ |
| Audio Tracks & Clips | ✔ | ✔ | ✔ | ✔ | ✔ | W-O | ✔ | ✔ |
| Gap/Filler | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✖ | ✔ |
| Markers | ✔ | ✔ | ✔ | ✔ | ✖ | N/A | ✖ | ✔ |
| Nesting | ✔ | ✖ | ✔ | ✔ | ✔ | W-O | ✔ | ✔ |
| Transitions | ✔ | ✔ | ✖ | ✖ | ✔ | W-O | ✖ | ✔ |
| Audio/Video Effects | ✖ | ✖ | ✖ | ✖ | ✖ | N/A | ✖ | ✔ |
| Linear Speed Effects | ✔ | ✔ | ✖ | ✖ | R-O | ✖ | ✖ | ✖ |
| Fancy Speed Effects | ✖ | ✖ | ✖ | ✖ | ✖ | ✖ | ✖ | ✖ |
| Color Decision List | ✔ | ✔ | ✖ | ✖ | ✖ | ✖ | N/A | ✖ |
Along these lines, Thibault delivered a 15 minutes talk, also in the GStreamer Conference 2019:
After detecting a few regressions and issues in GStreamer, related to frame accuracy, we decided to make sure that we can seek in a perfectly frame accurate way using GStreamer and the GStreamer Editing Services. In order to ensure that, an extensive integration testsuite has been developed, mostly targeting most important container formats and codecs (namely mxf, quicktime, h264, h265, prores, jpeg) and issues have been fixed in different places. On top of that, new APIs are being added to GES to allow expressing times in frame number instead of nanoseconds. This work is still ongoing but should be merged in time for GStreamer 1.18.
GStreamer Validate Flow #
GstValidate has been turning into one of the most important GStreamer testing tools to check that elements behave as they are supposed to do in the framework.
Along with our MSE work, we found that other way to specify tests, related with produced buffers and events through specific pads, was needed. Thus, Alicia developed a new plugin for GstValidate: Validate Flow.
Alicia gave an informative 30 minutes talk about GstValidate and the new plugin in the last GStreamer Conference too:
GStreamer VAAPI #
Most of the work along the second half of 2019 were maintenance tasks and code reviews.
We worked mainly on memory restrictions per backend driver, and we reviewed a
big refactor: internal encoders now use GstObject, instead of the custom
GstVaapiObject. Also we reviewed patches for new features such as video
rotation and cropping in vaapipostproc.
Servo multimedia #
Last year we worked integrating media playing in Servo. We
finally delivered hardware accelerated video playback in Linux and Android. We
worked also for Windows and Mac ports but they were not finished. As natural,
most of the work were in servo/media crate,
pushing code and reviewing contributions. The major tasks were to rewrite the
media player example and the internal source element looking to handle the
download playbin’s flag properly.
We also added WebGL integration support with <video> elements, thus webpages can use video frames as WebGL textures.
Finally we explored how to isolate the multimedia processing in a dedicated thread or process, but that task remains pending.
WebKit Media Source Extension #
We did a lot of downstream and upstream bug fixing and patch review, both in WebKit and GStreamer, for our MSE GStreamer-based backend.
Along this line we improved
WebKitMediaSource
to use playbin3 but also compatibility with older GStreamer versions was
added.
WebKit WebRTC #
Most of the work in this area were maintenance and fix regressions uncovered by the layout tests. Besides, the support for the Rasberry Pi was improved by handling encoded streams from v4l2 video sources, with some explorations with Minnowboard on top of that.
Conferences #
GStreamer Conference #
Igalia was Gold sponsor this last GStreamer Conference held in Lyon, France.
All team attended and five talks were delivered. Only Thibault presented, besides the video editing one which we already referred, another two more: One about GstTranscoder API and the other about the new documentation infrastructure based in Hotdoc:
We also had a productive hackfest, after the conference, where we worked on AV1 Rust decoder, HLS Rust demuxer, hardware decoder flag in playbin, and other stuff.
Linaro Connect #
Phil attended the Linaro Connect conference in San Diego, USA. He delivered a talk about WPE/Multimedia which you can enjoy here:
Demuxed #
Charlie attended Demuxed, in San Francisco. The conference is heavily focused on streaming and codec engineering and validation. Sadly there are not much interest in GStreamer, as the main focus is on FFmpeg.
RustFest #
Phil and I attended the last RustFest in Barcelona. Basically we went to meet with the Rust community and we attended the “WebRTC with GStreamer-rs” workshop presented by Sebastian Dröge.