I am at the airport getting ready to board beginning my trip to Karlsruhe for GUADEC 2016. I’ll be there with my colleague Michael Catanzaro.Please talk to us if you are interested in browsers or Igalia.
In this post I am going to talk about the implementation of the Media Source Extensions (known as MSE) in the WebKit ports that use GStreamer. These ports are WebKitGTK+, WebKitEFL and WebKitForWayland, though only the latter has the latest work-in-progress implementation. Of course we hope to upstream WebKitForWayland soon and with it, this backend for MSE and the one for EME.
My colleague Enrique at Igalia wrote a post about this about a week ago. I recommend you read it before continuing with mine to understand the general picture and the some of the issues that I managed to fix on that implementation. Come on, go and read it, I’ll wait.
The idea of the our AppendPipeline is to put a data stream into it and get it processed at the other side. It has an appsrc, a demuxer (qtdemux currently) and an appsink to pick up the processed data. Something tricky of the spec is that when you append data into the SourceBuffer, that operation has to block it and prevent with errors any other append operation while the current is ongoing, and when it finishes, signal it. Our main issue with this is that the the appends can contain any amount of data from headers and buffers to only headers or just partial headers. Basically, the information can be partial.
First I’ll present again Enrique’s AppendPipeline internal state diagram:
First let me explain the easiest case, which is headers and buffers being appended. As soon as the process is triggered, we move from Not started to Ongoing, then as the headers are processed we get the pads at the demuxer and begin to receive buffers, which makes us move to Sampling. Then we have to detect that the operation has ended and move to Last sample and then again to Not started. If we have received only headers we will not move to Sampling cause we will not receive any buffers but we still have to detect this situation and be able to move to Data starve and then again to Not started.
Our first approach was using two different timeouts, one to detect that we should move from Ongoing to Data starve if we did not receive any buffer and another to move from Sampling to Last sample if we stopped receiving buffers. This solution worked but it was a bit racy and we tried to find a less error prone solution.
We tried then to use custom downstream events injected from the source and at the moment they were received at the sink we could move from Sampling to Last sample or if only headers were injected, the pads were created and we could move from Ongoing to Data starve. It took some time and several iterations to fine tune this but we managed to solve almost all cases but one, which was receiving only partial headers and no buffers.
If the demuxer received partial headers and no buffers it stalled and we were not receiving any pads or any event at the output so we could not tell when the append operation had ended. Tim-Philipp gave me the idea of using the need-data signal on the source that would be fired when the demuxer ran out of useful data. I realized then that the events were not needed anymore and that we could handle all with that signal.
Both need-data signal and probe information come in GStreamer internal threads so we could use mutexes to overcome any race conditions. We thought though that deferring the operations to the main thread through the pipeline bus was a better idea that would create less issues with race conditions or deadlocks.
To finish I prefer to give some good news about performance. We use mainly the YouTube conformance tests to ensure our implementation works and I can proudly say that these changes reduced the time of execution in half!
That’s all folks!
First thing to say is that it happened in March at Apple campus in Cupertino and I atteded as part of the Igalia gang.
My goal when I went there was to discuss with Youenn Fablet about Streams API and we are implementing and see how we could bootstrap the reviews and being to get the code reviewed and landed efficiently. Youenn and I also made a presentation (mainly him) about it. At that moment we got some comments and help from Benjamin Poulain and nowadays we are also working with Darin Adler and Geoffrey Garen so the work is ongoing.
WebRTC was also a hot topic and we talked a bit about how to deal with the promises as they seem to be involved in the WebRTC standard was well. My Igalian partner Philippe was missed in this regard as he is involved in the development of WebRTC in WebKit, but he unfortunately couldn’t make it because of personal reasons.
I also had an interesting talk with Jer Noble and Eric Carlson about Media Source and Encrypted Media Extensions. I told them about the several downstream implementations that we are or were working on, specially the WebKit4Wayland one and that we expect to begin to upstream soon. They commented that they still have doubts about the abstractions they made for them and of course I promised to get back to them when we begin with the job. Actually I already discussed some issues with Quique, another fellow Igalian.
Among the other interesting discussions, I found very necessary the migration of Mac port to CMake. Actually, I am experiencing now the
painbenefits of using XCode to add files, specially the generated ones to the compilation. I hope that Alex succeeds with the task and soon we have a common build system for all main ports.
In December we organized in A Coruña the WebKitGTK+ hackfest at the Igalia premises as usual and also as usual it was an awesome oportunity to meet the rest of the team. For more information about the progress done in the hackfest, you can have a look at KaL’s post.
Some of the non-aesthetic changes are the following:
The captions icon problem was interesting because I found out that the one we were using was “user-invisible-symbolic” and it was hardcoded directly in the CSS code. I changed it to be loaded from the theme but it raised the issue of using the incorrect metaphor though the current icon looks nice for captions. I filed a GNOME bug (and another WebKit bug to follow this up) so that a new icon can be created for captions/subtitles with the correct metaphor.
And which are the controls aesthetic changes?
As I already said the aesthetic differences with the former C++ are not a big deal unless you compare them with the original controls:
To appreciate the new controls I cannot just show a screenshot, because the nicest thing are the animations. Therefore a video is needed (and if you have WebKit compiled you can experience them yourself)):
Of course, I thank our hackfest sponsors as the it was possible because of them:
I departed from A Coruña, flying to Barcelona (where I had time to have a coffee with a friend), then to Vienna and then to Brno by bus. It was a bit tiring so after having a quick drink at our hotel with the rest of the Igalians, I just went to bed.
Web, the future is now by my Igalian friend Claudio Saavedra (different from Garnacho) was my first talk (apart from the keynote). Even after knowing the content already, I found quite interesting the way Claudio spoke about the latest changes in the development of Web (a.k.a. Epiphany), specially the way WebKit 2 helps to provide a much better user experience.
I liked a lot Alex Larsson’s talk of High resolution display support in GNOME. His approach of the abstract pixels and all the way down in the stack from Gtk+ was very interesting.
Matt Dalio and his Endless Mobile project just rock, not only because of the tech and GNOME involved but also because of its social implications. Keep going!
Given the rest of my career and my work in WebKit, specially in the WebKitGTK+ port, I am always interested in GStreamer, as it is the framework we use use for multimedia playback (and other ports like EFL and Qt in Linux). What’s cooking in GStreamer talk by Sebastian Dröge and Tim-Philipp Müller was of course mandatory and worth going.
Jan-Christoph Borchardt’s talk about GNOME and ownCloud: desktop plus web for a holisic experience showed everything that is and is coming with ownCloud. I introduced him to Claudio as he mentioned that he would like to see at least Ephy’s bookmarks syncronized with ownCloud. It looks interesting.
I missed Philip Withnall’s talk about Testing online services because it was at the same time as the ownCloud one, but I was interested because of my wife‘s research project about the same subject. Philip, if you read this, leave me a comment or get in touch directly with her as she was very interested.
It is always refreshing to attend Marina Zhurakhinskaya’s talk about Outreach Program for Women: a lesson in collaboration, because I think it is very important to keep reminding ourselves about the relevance of programs to integrate more women into our community and I think we, and specially Marina and the OPW program, get always awesome results, though we cannot fall asleep and keep pushing till, at least, the 50%.
My Igalian friend Juan Suárez had a talk called Writing multimedia applications with Grilo where he showed the easy but powerful possibilities of the Grilo API and its plugins. There was also a lightning talk by an intern about adding support to build Lua plugins for Grilo.
Juan Pablo Ugarte was showing off his Glade skills in his talk about Rich custom user interfaces with Glade and CSS. His slides were made with Glade. Cool!
More secure with less “security” by Stef Walter had a lot of interesting ideas about how to improve security and making at the same time the user’s life easier.
Just after lunch it was the turn of my Igalian friend Martin Robinson who gave the talk about Webkit2 and you and explained many things about the WebKit 2 model, like for example all its layers and how the processes work.
There was another talk given by other fellow Igalians, Alejandro Piñeiro and Joanmarie Diggs, called Tag, your PDF is it that I couldn’t attend. In that case I preferred to prioritize other talks as I can always easily talk to them.
The rest of the talks I attended were:
I really enjoyed the tour at the city, which is very beatiful and peaceful, though we had to finish it prematurely because of the impresive storm. On our way back from our unofficial GNOME Hispano dinner, which of course was great, to our hotel, we could see some broken tree branches and even what we think that were some fallen young trees.
The venue was really a nice place (with funny chairs) and the volunteers did an awesome job (THANKS!). Of course, there are always issues, like the AC problem (these things can always happen) and the internet connection which is an inherent problem to almost every GUADEC.
Of course, I cannot forget to thank Igalia for sponsoring my trip.
Strasbourg, there we go!
First of all, thanks to Igalia for sponsoring me and see you all in Brno!
Btw, I will be in the Avanti Hotel, one of the recommended by the organization. I am alone in the room by now, but if there is anybody open to share room, please get in touch with me.
Starting point was:
As you can see the controls look like an old Gtk+ application without any theming. Jon suggested that we could began with mimicing Chromium controls as they look closer to any modern themed GNOME application and adapt them to use the GNOME symbolic icons and keep some other stuff like the volume bar, but of course making it look nicer.
What was done:
I had a small issue with a Chromium guy landing a patch that forced me to change the
display of some components from
-webkit-flex and of course, rebasing all related tests. This created a small delay in landing the patch, but it finally did as 143463.
I don’t know about you guys, but I like it!
Tired as hell but with my hacking batteries completelly full.
There was always something to do here and there, which I needed to help with, but I think the most important things that I did were:
The only bad thing I can think of is that I missed some talks because of having a lot of things to do.
I attended all the talks I could and just tried to avoid the ones by Igalians because for obvious reasons I already knew what they were about, so I’ll mention the ones that I liked most:
And of course, I need to thank Igalia for sponsoring me by attending and helping at GUADEC:
Wow, something I think I would not see, but GUADEC is happening less than 3km from home in less than a week!
I have collaborated in organizing the Lightning Talks and BoFs/Hackfests and of course, I’ll help wherever I can. Poor Zeenix will be at my place so he will need to wake up early to come with me and my wife Laura to the faculty.
I also want to thank Igalia for sponsoring me with time to help organizing and attending the event and I might be also helping the Advisory Board meeting that will happen at Igalia headquarters on the 25th.