This post is a brief and quick introduction to Accelerated Compositing and the role this plays in WebKit.
The point of accelerated compositing in WebKit is to take advantage of the GPU to accelerate the rendering of web content. To understand how this works one needs to know how a web page is actually rendered by WebKit.
Rendering in WebKit
WebKit groups DOM elements in layers that are rendered separately and then composited together to create the final page. This enables proper handling of transparent objects and overlapping contents for example. These layers, just like the DOM, conform a tree.
WebKit defines some rules to decide when a new layer is needed and which DOM elements should be included in it. For example, if you have a WebGL canvas or a video element, these will go to separate layers, if you use explicit CSS position properties on an object you get another layer for it, etc.
When it is time to paint the web page on the browser, the work consists of traversing the layer tree and composite the layers together to create the final view of the web page. For example, if you have a transparent layer sitting on top of some other layer in the page, the composition will take the two individual layers that have been rendered independently and blend them together in the final page, in the right order, at the right position and with the right
Before we had accelerated compositing, the compositing process happend in software, that is, it was the CPU that would do all the layer compositing work, which is expensive and can hog the CPU, making for a worse user experience. Accelerated compositing however, involves offloading the compositing of the layers to the GPU hardware. It turns out that GPUs can do the compositing very fast and doing so would also free the CPU, delivering a better, more responsive, user experience too.
Accelerated compositing is all about making a better use of the available graphics hardware, offloading the work required to composite the final view of the webpage from the various layers it contains, which results in a better user experience and better overall rendering performance.
So first things first, check out the video below to see the demo, it showcases Web (Epiphany), the default browser of the GNOME platform, running on WebKit1 under Wayland (Weston) and illustrates:
Browsing of regular text/image based sites
Embedded HTML5 video playback (Youtube and native)
2D and 3D CSS transforms
If you are intrigued to know more about WebKit, Wayland and Web, keep reading. If you already know about this and are more interested in knowing what is still missing, go here.
So WebKit you say, what is that and why should I care?
There is a significant chance that when you are reading this you are using WebKit. WebKit is an open source web browser engine that powers a variety of software that we run on our desktops, phones and a myriad of other gadgets every day and with the popularity that HTML5 has achieved this will only get bigger and bigger in the future.
Igalia has always considered the web the platform of the future and with the passing of the years we have seen this become more and more real. In this process we are certain that WebKit has played a very significant role, providing an open framework where many companies and individuals have worked together for years, building something that is at the core of most HTML5 based solutions out there. These companies and individuals have done a remarkable work in pushing the technology forward and make all this possible, and at Igalia we are very proud to be a part of this too.
Ok, so WebKit is important… but what does that have to do with Wayland?
WebKit and HTML5 are not the only big things happening out there. Wayland will progressively replace X in the coming years, something outstanding when you consider the fact that X has been there for 3 decades! This replacement is not trivial though, X has established the basis for graphical user interfaces since… well since before I even knew computers existed! Much of the software we use every day is based on X directly or indirectly, so with the change, a lot of software that we love will need to undergo some changes and adaptations to work with Wayland.
Platforms like GNOME and KDE have plans to support Wayland. GNOME in particular plans to have complete Wayland support some time in 2014, but for that to happen efforts have already started at many levels. GTK+, the graphical toolkit of the GNOME platform, is already compatible with Wayland, which is a big part of what needs to be done to make GNOME itself run on Wayland. Still, there are plenty of X bits in many other parts of the platform that will need to be addressed. WebKitGTK+ is no exception to this.
The work to get WebKitGTK+ ported to Wayland was initiated by my colleague José Dapena and I have recently joined this effort. I started from Jose’s patches, checking the current state of things and trying to identify the missing bits. I hope the video at the top gives a good idea of where we are now and later in this post I will explain what is still missing. For now I have focused mostly on WebKit1, but I believe most of my findings are just as valid for WebKit2. Hopefully this will help us define specific tasks and make steady progress in the months to come.
And what did you do to get that demo exactly?
Since WebKitGTK+ is based on GTK+ and GTK3 already provides a Wayland backend, we need to make sure that we use GTK3 in WebKitGTK+ and Web, which we already do (except for the plugin process which still needs GTK2, more on this later), so thankfully a lot of the heavy lifting had already been taken care of.
The second thing we need to check is the backing store backend we use. We need to replace the X11 implementation with something more agnostic. Good that we already have a Cairo implementation of the backing store available!
A few other minor fixes aside, these two items represent most of what I needed to get the demo running.
How far are you from full Wayland support then?
1. Accelerated compositing
Current implementation of accelerated compositing depends on XComposite and XDamage. We need to identify how to re-implement this functionality in Wayland (at least the bits that depend on X directly) while we make sure that we keep the same performance advantage.
2. Fullscreen video playback
Fullscreen video playback with GStreamer is implemented using the video overlay interface which is platform agnostic. However, even if the interface is platform agnostic, it requires GSteamer to provide a platform-specific implementation. There is a Wayland sink in GStreamer, however this sink does not implement the video overlay interface and I still don’t know if this is to be addressed in GStreamer or when. If the waylandsink in GStreamer is extended to implement the video overlay interface then the good news is that the work on the WebKit end should be fairly straight forward.
Various plugins won’t work in Wayland. The Flash plugin in particular is known to only work with GTK2 (this is the reason the Plugin process in WebKit2 requires GTK2 actually). Since GTK2 does not support Wayland the consequence is that we won’t be seeing the Flash plugin in Wayland for now. We contacted Adobe some time ago on this subject to see if they would be interested in porting their Flash plugin to GTK3, but apparently this is not a priority for them. Even more, this restricts all plugins to use GTK2 and hence, it will pretty much disable all plugins under Wayland (well, at least the ones that need to render). Another example of a plugin needing work is Java, which seems to have direct X dependencies in its code leading to a nice crash when you load a Java applet for example. I did not do a comprehensive analysis of all the plugins, but this is definitely a problematic area in the short term. Of course, this will probably change with time, when Wayland becomes mainstream plugin developers will have to port the plugins or lose user base and be replaced by other implementations. Some plugins may even become completely obsolete before Wayland becomes mainstream too, that could be the case of the Flash plugin maybe. Bottom line: plugins suck and if you are a web developer you probably want to avoid them if you have an alternative.
WebKit2 brings a bigger challenge to the Wayland port due to the split process architecture that it introduces. In this context we need to make sure that all the rendering and painting that takes place in coordinated form between the UI process and the Web process use Wayland mechanisms for GPU acceleration efficiently. We still need to design a good solution for this before getting hands-on with fixing it and we will need to approach the Wayland community and discuss how WebKit2′s architecture fits in Wayland as it is today. Wayland has just integrated support for sub-surfaces and this is something that may come in handy in the context of this problem too.
5. Web (Epiphany)
Web is the default browser of the GNOME platform and a good test ground for WebKitGTK+ too. Making WebKitGTK+ Wayland compatible would not be good enough for GNOME if Web isn’t. Fortunately, WebKitGTK+ and GTK3 do most of the work for Web in this regard, although there seem to be some X interactions in Web itself they do not look difficult to replace or find a workaround for. I will use the video above to prove my point
So that’s it, even if there are still some issues that need further work, WebKitGTK+ and Web can run in Wayland with few changes as they are today, including things like WebGL, CSS Transforms or embedded video playback. Next in the roadmap are accelerated compositing and WebKit2. We are planning to make progress in these areas in the coming months, so if you are as excited as we are for full Wayland support in WebKitGTK+ including WebKit2 and Accelerated Compositing, stay tuned!
That was real fast! The video of my presentation on Grilo at the MeeGo Conference is now available here. The video starts a bit late so it missed the first slides but hopefully people can still follow the presentation properly.
I got my talk about Grilo accepted for the next MeeGo Conference in San Francisco. I intend to use this opportunity to introduce the project to the MeeGo community, where I think Grilo can be very useful for all the developers out there interested in crafting multimedia solutions on this platform.
Also, I have realised that my blog has been silent for way too long, that is mostly because I am not the main developer of Grilo nowadays, Juan Suárez and Guillaume Emont have been doing most of the work for quite some time now, and even though I am still quite involved in the project my contributions happen mostly in the form of patch reviews, mailing list participation and help with feature planning and API design. I hope I can get back to write some code again soon though, I miss that.
And talking about code, in the last 6 months or so we have been doing quite a lot of things in Grilo, we should have blogged more about that for sure, but let’s just say that all that work has helped with making Grilo more mature and flexible. In this regard I want to thank the media explorer developers for using Grilo and giving us feedback, bug reports and code contributions. It is really great to see other developers using Grilo in their projects… and now we also have a strong and good looking demo of Grilo, thank you guys!
Now, getting back to the MeeGo Conference topic, other than my talk about Grilo I will also be at Igalia’s booth most of the time so, in case you are planning to attend the event, I should be easy to find, feel free to come and talk to us, I will be happy to show you some of the cool stuff we are doing at Igalia, Grilo included
If you are in Berlin for LinuxTag and you are interested in Grilo or you are simply curious about it you probably want to attend my presentation: ‘Seamless Integration of Multimedia Services’ tomorrow at 4:00pm in room Berlin II.
This weekend I got a confirmation from the GUADEC organization that my talk “Grilo: Feeding applications with multimedia content” has been accepted. That’s great news, I am really looking forward to delivering this presentation and get feedback from other GNOME hackers about what we are doing. See you all in The Hague in July!
After knowing about Grilo, Bastien proposed to use it as a helper library to implement Rygel’s MediaServer D-Bus API spec. This D-Bus API specifies how content providers can expose content over D-Bus that Rygel can consume and then export to the UPnP world. Although this spec was developed for Rygel in particular, it aims to be generic enough so other applications could use it to consume content over D-Bus directly.
The idea is to decouple providers and consumers, this way one does not have to worry about the language they are written in and providers don’t have to be loaded in the consumer’s address space, instead they are communicated through D-Bus.
So, what could be Grilo‘s role in this context? As you know, Grilo is a pluggable framework that provides a single, high-level API to consume contents from various sources (Youtube, Jamendo, SHOUTcast, etc) which are implemented as plugins for the framework. As such, the role of Grilo would be that of a content provider. Juan has been working on a daemon that would use Grilo to get access to all the content exposed by the framework and expose it over D-Bus according to Rygel’s MediaServer spec. This will enable Rygel (or any other application), to get access to all this content over D-Bus.
As of today, I think only Rygel has implemented the consumer side of this API, but the idea is that in the future other applications like Totem or Rhythmbox would have plugins to consume it too. Also, since Rygel is a UPnP server, and it is feeding on Grilo through this D-Bus interface, it is also exposing Grilo’s contents to the UPnP world, so any UPnP client should be able to access these contents over UPnP thanks to Rygel. Totem for example, has a UPnP plugin already, and because of that it can consume the contents from Grilo through Rygel even when it does not implement the consumer API of the MediaServer spec.
If you are interested in more details check Juan’s post here. There you will also find a screencast showcasing a Grilo-powered daemon that’s exposing content from various of its plugins over D-Bus, then Rygel feeding on that and exposing them over UPnP, and finally Totem consuming all the content though its UPnP plugin. Of course, you may ask why wouldn’t Totem feed on D-Bus directly, well just because it has the UPnP plugin already and not the D-Bus one, but it will come .
Talking about that, another thing that Bastien proposed and on which we will be working too, is a set of helper libraries to ease development of content producers and consumers. On the producer side (backends) the idea would be to hide all the D-Bus stuff and provide a clean, easy to use interface for exposing content to the bus (for those not wanting to deal with D-Bus directly), and on the consumer side (frontends), we could do the same and maybe add some extra bonus stuff, like widgets to manipulate available sources and the like.