Jul 14

A tour around the world of Mesa and Linux graphics drivers

For some time now I have decided to focus my work at Igalia on the graphics stack. As a result of this I had the chance to participate in a couple of very interesting projects like implementing Wayland support in WebKitGtk+ (a topic I have visited in this blog a number of times) and, lately, work on graphics drivers for Linux in the Mesa framework.

The graphics stack in Linux is complex and it is not always easy to find information and technical documentation that can aid beginners in their firsts steps. This is usually a very demanding domain, the brave individuals who decide to put their energy into it usually have their hands full hacking on the code and they don’t have that much room for documenting what they do in a way that is particularly accessible to newcomers.

As I mentioned above, I have been hacking on Mesa lately (particularly on the Intel i965 driver) and so far it as been a lot of fun, probably the most exciting work I have done at Igalia in all these years, but it is also certainly challenging, requiring me to learn a lot of new things and some times fairly complex stuff.

Getting involved in this is no easy endeavor, the learning curve is steep because the kind of work you do here is probably unlike anything you have done before: for starters it requires a decent understanding of OpenGL and capacity to understand OpenGL specifications and what they mean in the context of the driver, you also need to have a general understanding of how modern 3D-capable GPUs work and finally, you have to dig deeper and understand how the specific GPU that your driver targets works and what is the role that the driver needs to play to make that hardware work as intended. And that’s not all of it, a driver may need to support multiple generations of GPUs which sometimes can be significantly different from each other, requiring driver developers to write and merge multiple code paths that handle these differences. You can imagine the maintenance burden and extra complexity that comes from this.

Finally, we should also consider the fact that graphics drivers are among the most critical pieces of code you can probably have in a system, they need to be performant and stable for all supported hardware generations, which adds to the overall complexity.

All this stuff can be a bit overwhelming in the beginning for those who attempt to give their first steps in this world but I believe that this initial steep learning curve can be smoothed out by introducing some of the most important concepts in a way that is oriented specifically to new developers. The rest will still not be an easy task, it requires hard work, some passion, be willing to learn and a lot of attention to detail, but I think anyone passionate enough should be able to get into it with enough dedication.

I had to go through all this process myself lately, so I figured I am in a very good situation to try and address this problem myself, so that’s why I decided to write a series of posts to introduce people to the world of Mesa and 3D graphics drivers, with a focus on OpenGL and Intel GPUs, which is the area were I am currently developing my work. Although I’ll focus on Intel hardware I believe that many of the concepts that I will be introducing here are general enough so that they are useful also to people interested in other GPUs. I’ll try to be clear about when I am introducing general concepts and when I am discussing Intel specific stuff.

My next post, which will be the first in this series, will serve as an introduction to the Linux graphics stack and Linux graphics drivers. We will discuss what Mesa brings to the table exactly and what we mean when we talk about graphics drivers in Linux exactly. I think that should put us on the right track to start looking into the internals of Mesa.

So that’s it, if you are interested in learning more about Linux graphics and specifically Mesa and 3D graphics drivers, stay tuned! I’ll try my best to post regularly and often.

Feb 14

Epiphany + WebKitGTK/WebKit2 + Wayland + Accelerated Compositing

In my previous post I shared that I had managed to get a basic implementation of WebKitGTK+WebKit2 to work under Wayland. I also discussed some of the pieces that were still missing, most important of which was supporting for multiple views, that is, having the possibility to run multiple browser windows/tabs that render accelerated content simultaneously.


In the last weeks I have continued making progress and I am happy to say that I have finally implemented support for this too, proof in the video below:


Support for multiple views required to implement an extension to the Wayland protocol so that we can effectively map widgets and their corresponding Wayland surfaces in our nested compositor. This is needed to know which surface provides the graphics for which widget. Thanks to Pekka Paalanen for introducing me into the world of Wayland extensions!


My work also uncovered a number of hidden bugs in WebKitGTK that were hidden by the fact that we always use a sharing context for all our GL contexts. In Wayland, however, my colleague Zan Dobersek is working on implementing support for the sharing context separately and our patches still need to be merged together, so I have been working all this time without a sharing context and that uncovered these bugs that show up when we deal with multiple views (and hence multiple GL contexts). I am still working on fixing these but in any case merging my work with Zan’s should be enough to prevent them from actually producing any harm, just like in X11. Actually, one of these bugs is the one behind the rendering issues I mentioned in my last post when clicking on the browser’s back button.


One more thing worth mentioning: I needed a full browser to test multiple browser windows and tabs, so that also led me to test all my work with Epiphany/Web, which I had not done yet (so far I had restricted myself to work only with WebKit’s MiniBrowser), that is of course the browser I use in the video above.


If you are interested in following progress more closely or want to look at the patches that enable Accelerated Compositing for WebKitGTK/WebKit2 under Wayland, here is the bug.


Finally, I would like to thank my sponsor, Igalia, for supporting this work since its inception.

Jan 14

WebKitGTK Wayland: Initial support for WebKit2 and Accelerated Compositing

Quick Recap

In my last post on the subject I explained how during the last WebKitGTK hackfest my colleague Eduardo Lima and I got a working GTK application that made use of the nested compositor design we need in WebKitGTK to get WebKit2 to work under Wayland and how the next steps would involve developing a similar solution inside WebKitGTK.


Current Status

During the last 2 weeks I have been working on this, and today I got Accelerated Compositing to work under Wayland and WebKit2. There are still a lot of rough edges of course since this milestone is mostly a prototype that only covers the basics. Its purpose was solely  to investigate how the nested compositor approach would work in WebKitGTK to support the Accelerated Compositing code path. Still, this is an important milestone: Accelerated Compositing and WebKit2 were the biggest missing pieces to bring Wayland support to WebKitGTK and even if this is  only a prototype it provides a solution for these two aspects, which is great news.


To Do List

There are probably a lot of things that need more work to convert this prototype into a proper solution. I have not tested it thoroughly but here is a quick list of things that I already know need more work:

  • Support for multiple windows and tabs (the prototype only supports one tab in one window)
  • For some pages the first composition can be very slow (as in taking >5 seconds). This problem only happens the first time the page is loaded, but it does not happen when  reloading the same page (the demo video below shows this)
  • Rendering of text selections does not seem to work
  • There are rendering artifacts when going back using the browser’s back button to a previously visited page that activates the Accelerated Compositing code path. If the page is reloaded things go back to normal though
  • There are some style rendering issues I have not looked into yet, might be on the side of GTK though
  • All this was tested in a Wayland environment inside an X session, so it can be that some things still need to be fixed for this to work in a pure Wayland environment (with no X server running).
  • Ideally we would like a solution that can make run-time decisions about the underlying platform (X or Wayland) so that we don’t have to build WebKitGTK specifically for one or the other. This is particularly important now that adoption of Wayland is still low. My prototype, however, only supports Wayland at the moment and would require more work to select between X and Wayland at run-time.

And there is probably a lot more to do that we will find out once we start testing this more seriously.



So here is a small video demoing the prototype. The demo uses WebKit’s MiniBrowser (WebKit2) to load 3 pages that activate the Accelerated Compositing code path in WebKitGTK. The browser is restarted for every page load only because it made it easier for me to record the video. You will see that for some pages, the first time the page  is composited it takes a long time which is one of the issues I mentioned above. The demo also shows how this is not the case when the page is reloaded:



Next Steps

Once I reached this milestone I think we should start moving things to get this  upstream as soon as possible: the current implementation provides the basics for Wayland support in WebKit2 and it would allow other interested developers to step in and help with testing, completing and fixing this initial implementation. I am sure there is still a lot of work to do for a fully operational Wayland port of WebKitGTK, so the more people who can contribute to this, the better.


I presume that upstreaming my code will still require a significant effort: my current implementation is a bit too hackish right now so there will be a lot of cleanups to do and a lot of back and forth with upstream maintainers to get the code in position to be merged, so the sooner we start the better. I also need to rebase my code against up-to-date versions of WebKitGTK and Wayland since I froze my development environment during the last WebKitGTK hackfest.


So that’s it. It is always good to reach milestones and I am happy to have reached this one in particular. If you are excited about WebKitGTK and Wayland I hope you enjoyed the news as much as I enjoyed working on it!


I would like to thank Igalia for sponsoring my work on this as well as all the other Igalians who helped me in the process, it would have not been possible without this support!

Dec 13

WebKitGTK+ 2013 hackfest: On the road to WebKit2 Wayland support in WebKitGTK+

So this was my first participation in the WebKitGTK+ hackfest. It was great to have some time to focus on WebKitGTK+ hacking for a few days as well as meeting other colleagues face to face to discuss various related topics, specifically the one I am most interested in: Wayland support in WebKit2.

A few months back I was reviewing the status of WebKitGTK+ in Wayland and mentioned that one of the main challenges was the multi-process architecture introduced with WebKit2, the one that Web/Epiphany is currently using.

The problem is simple enough to explain: In WebKit2, scene composition is done entirely in the Web Process and then painted on the screen in the UI Process, so we need to share a graphics surface between these two processes. In X11 we do this by having the UI Process create an offscreen XWindow and sharing the window ID with the Web Process, but in Wayland there is no direct means to share a graphics surface between two Wayland clients.

The solution for this is to use the same means that a Wayland compositor uses to share graphics with its clients. The meaning of this in the context of WebKitGTK+ is that we need to implement a small Wayland compositor that we can use to share the rendering surface. The way this would work in is like this: the UI Process would play the role of Wayland compositor and the WebProcess would ask it for  a surface using regular Wayland APIs. Since the UI Process implements a Wayland compositor it will have access to the graphics buffers rendered by the client (the Web Process) and we have our problem solved. This is quite a bit more of work than simply sharing a Xwindow ID though.

Some months ago I was prototyping a proof of concept for how this would work taking WebKit out of the equation to keep things easy as a first step. During the hackfest I had the opportunity to complete the work and bring it up to date with the current status of Wayland together with the help of my colleague Eduardo Lima This small prototype has two parts: a GTK+ application with a custom GtkContainer widget (which would play the role of the UIProcess in WebKitGTK+) and a separate Wayland client that renders a simple GL scene. The GTK program spawns another process to run the Wayland client when started and also implements the required bits of the Wayland compositor interface to serve Wayland requests as required by the client.

The point of the experiment was to get the GTK program to use the rendering results of the client to paint its own widget contents. This is basically what we need in WebKitGTK+, where the GtkWidget would be the WebView running in the UI Process and the client process would be the Web Process rendering the results of the scene composition to a GL texture.

The next step is to implement this solution in WebKitGTK+, which is a work in progress at the moment. It is still quite a bit of work since the WebKit code base is quite large and complex, but at this point I think it is only a matter of time to get a basic solution to work. Then of course we will have to deal with a lot of other details that this initial proof of concept did not care about, like  resizing, managing surfaces for multiple windows and probably a lot more stuff that will pop up along the way.

Finally, there is another interesting consideration to make. Even if the UI Process can share a graphics surface with Web Process, it still has to render it on the GTK widget’s surface. The problem here is that GTK on Wayland uses a cairo image surface as backing or the window surface, so this process involves a copy that results in bad performance. I guess this should be fixed at some point in GTK+ so we can have the same performance we currently have on X11. In the past I tried to go around this by creating accelerated Wayland subsurfaces for the widget and render to that instead. This worked well for performance but it had to be done completely outside GTK+ , and hence it breaks a number of things (for example you have to position the surface manually within the window surface, events are not managed properly, etc), so it was a no go. I suppose that if GTK+ can provide means to manage Wayland subsurfaces for a widget natively, this would also be another option to fix the performance  problem.

Jun 13

On WebKit, Layers and Accelerated Compositing

This post is a brief and quick introduction to Accelerated Compositing and the role this plays in WebKit.

The point of accelerated compositing in WebKit is to take advantage of the GPU to accelerate the rendering of web content. To understand how this works one needs to know how a web page is actually rendered by WebKit.

Rendering in WebKit

WebKit groups DOM elements in layers that are rendered separately and then composited together to create the final page. This enables proper handling of transparent objects and overlapping contents for example. These layers, just like the DOM, conform a tree.

WebKit defines some rules to decide when a new layer is needed and which DOM elements should be included in it. For example, if you have a WebGL canvas or a video element, these will go to separate layers, if you use explicit CSS position properties on an object you get another layer for it, etc.

When it is time to paint the web page on the browser, the work consists of traversing the layer tree and composite the layers together to create the final view of the web page. For example, if you have a transparent layer sitting on top of some other layer in the page, the composition will take the two individual layers that have been rendered independently and blend them together in the final page, in the right order, at the right position and with the right

Accelerated Compositing

Before we had accelerated compositing, the compositing process happend in software, that is, it was the CPU that would do all the layer compositing work, which is expensive and can hog the CPU, making for a worse user experience. Accelerated compositing however, involves offloading the compositing of the layers to the GPU hardware. It turns out that GPUs can do the compositing very fast and doing so would also free the CPU, delivering a better, more responsive, user experience too.


Accelerated compositing is all about making a better use of the available graphics hardware, offloading the work required to composite the final view of the webpage from the various layers it contains, which results in a better user experience and better overall rendering performance.

May 13

WebKitGTK+ / Wayland Demo and Future Work

So first things first, check out the video below to see the demo, it showcases Web (Epiphany), the default browser of the GNOME platform, running on WebKit1 under Wayland (Weston) and illustrates:

  • Browsing of regular text/image based sites
  • Embedded HTML5 video playback (Youtube and native)
  • 2D and 3D CSS transforms
  • WebGL

If you are intrigued to know more about WebKit, Wayland and Web, keep reading. If you already know about this and are more interested in knowing what is still missing, go here.

So WebKit you say, what is that and why should I care?

There is a significant chance that when you are reading this you are using WebKit. WebKit is an open source web browser engine that powers a variety of software that we run on our desktops, phones and a myriad of other gadgets every day and with the popularity that HTML5 has achieved this will only get bigger and bigger in the future.

Igalia has always considered the web the platform of the future and with the passing of the years we have seen this become more and more real. In this process we are certain that WebKit has played a very significant role, providing an open framework where many companies and individuals have worked together for years, building something that is at the core of most HTML5 based solutions out there. These companies and individuals have done a remarkable work in pushing the technology forward and make all this possible, and at Igalia we are very proud to be a part of this too.

Ok, so WebKit is important… but what does that have to do with Wayland?

WebKit and HTML5 are not the only big things happening out there. Wayland will progressively replace X in the coming years, something outstanding when you consider the fact that X has been there for 3 decades! This replacement is not trivial though, X has established the basis for graphical user interfaces since… well since before I even knew computers existed! Much of the software we use every day is based on X directly or indirectly, so with the change, a lot of software that we love will need to undergo some changes and adaptations to work with Wayland.

Platforms like GNOME and KDE have plans to support Wayland. GNOME in particular plans to have complete Wayland support some time in 2014, but for that to happen efforts have already started at many levels. GTK+, the graphical toolkit of the GNOME platform, is already compatible with Wayland, which is a big part of what needs to be done to make GNOME itself run on Wayland. Still, there are plenty of X bits in many other parts of the platform that will need to be addressed. WebKitGTK+ is no exception to this.

The work to get WebKitGTK+ ported to Wayland was initiated by my colleague José Dapena and I have recently joined this effort. I started from Jose’s patches, checking the current state of things and trying to identify the missing bits. I hope the video at the top gives a good idea of where we are now and later in this post I will explain what is still missing. For now I have focused mostly on WebKit1, but I believe most of my findings are just as valid for WebKit2. Hopefully this will help us define specific tasks and make steady progress in the months to come.

And what did you do to get that demo exactly?

Since WebKitGTK+ is based on GTK+ and GTK3 already provides a Wayland backend, we need to make sure that we use GTK3 in WebKitGTK+ and Web, which we already do (except for the plugin process which still needs GTK2, more on this later), so thankfully a lot of the heavy lifting had already been taken care of.

The second thing we need to check  is the backing store backend we use. We need to replace the X11 implementation with something more agnostic. Good that we already have a Cairo implementation of the backing store available!

A few other minor fixes aside, these two items represent most of what I needed to get the demo running.

How far are you from full Wayland support then?

1. Accelerated compositing

Current implementation of accelerated compositing depends on XComposite and XDamage. We need to identify how to re-implement this functionality in Wayland (at least the bits that depend on X directly)  while we make sure that we keep the same performance advantage.

2. Fullscreen video playback

Fullscreen video playback with GStreamer is implemented using the video overlay interface which is platform agnostic. However, even if the interface is platform agnostic, it requires GSteamer to provide a platform-specific implementation. There is a Wayland sink in GStreamer, however this sink does not implement the video overlay interface and I still don’t know if this is to be addressed in GStreamer or when. If the waylandsink in GStreamer is extended to implement the video overlay interface then the good news is that the work on the WebKit end should be fairly straight forward.

3. Plugins

Various plugins won’t work in Wayland. The Flash plugin in particular is known to only work with GTK2 (this is the reason the Plugin process in WebKit2 requires GTK2 actually). Since GTK2 does not support Wayland the consequence is that we won’t be seeing the Flash plugin in Wayland for now. We contacted Adobe some time ago on this subject to see if they would be interested in porting their Flash plugin to GTK3, but apparently this is not a priority for them. Even more, this restricts all plugins to use GTK2 and hence, it will pretty much disable all plugins under Wayland (well, at least the ones that need to render). Another example of a plugin needing work is Java, which seems to have direct X dependencies in its code leading to a nice crash when you load a Java applet for example. I did not do a comprehensive analysis of all the plugins, but this is definitely a problematic area in the short term. Of course, this will probably change with time, when Wayland becomes mainstream plugin developers will have to port the plugins or lose user base and be replaced by other implementations. Some plugins may even become completely obsolete before Wayland becomes mainstream too, that could be the case of the Flash plugin maybe. Bottom line: plugins suck and if you are a web developer you probably want to avoid them if you have an alternative.

4. WebKit2

WebKit2 brings a bigger challenge to the Wayland port due to the split process architecture that it introduces. In this context we need to make sure that all the rendering and painting that takes place in coordinated form between the UI process and the Web process use Wayland mechanisms for GPU acceleration efficiently. We still need to design a good solution for this before getting hands-on with fixing it and we will need to approach the Wayland community and discuss how WebKit2′s architecture fits in Wayland as it is today. Wayland has just integrated support for sub-surfaces and this is something that may come in handy in the context of this problem too.

5. Web (Epiphany)

Web is the default browser of the GNOME platform and a good test ground for WebKitGTK+ too. Making WebKitGTK+ Wayland compatible would not be good enough for GNOME if Web isn’t. Fortunately, WebKitGTK+ and GTK3 do most of the work for Web in this regard, although there seem to be some X interactions in Web itself they do not look difficult to replace or find a workaround for. I will use the video above to prove my point ;)


So that’s it, even if there are still some issues that need further work, WebKitGTK+ and Web can run in Wayland with few changes as they are today, including things like WebGL, CSS Transforms or embedded video playback. Next in the roadmap are accelerated compositing and WebKit2. We are planning to make progress in these areas in the coming months, so if you are as excited as we are for full Wayland support in WebKitGTK+ including WebKit2 and Accelerated Compositing, stay tuned!

Jun 11

Grilo @ MeeGo Conference: Video available now

That was real fast! The video of my presentation on Grilo at the MeeGo Conference is now available here. The video starts a bit late so it missed the first slides but hopefully people can still follow the presentation properly.

May 11

Grilo @ MeeGo Conference

I got my talk about Grilo accepted for the next MeeGo Conference in San Francisco. I intend to use this opportunity to introduce the project to the MeeGo community, where I think Grilo can be very useful for all the developers out there interested in crafting multimedia solutions on this platform.

Also, I have realised that my blog has been silent for way too long, that is mostly because I am not the main developer of Grilo nowadays, Juan Suárez and Guillaume Emont have been doing most of the work for quite some time now, and even though I am still quite involved in the project my contributions happen mostly in the form of patch reviews, mailing list participation and help with feature planning and API design. I hope I can get back to write some code again soon though, I miss that.

And talking about code, in the last 6 months or so we have been doing quite a lot of things in Grilo, we should have blogged more about that for sure, but let’s just say that all that work has helped with making Grilo more mature and flexible. In this regard I want to thank the media explorer developers for using Grilo and giving us feedback, bug reports and code contributions. It is really great to see other developers using Grilo in their projects… and now we also have a strong and good looking demo of Grilo, thank you guys! :)

Now, getting back to the MeeGo Conference topic, other than my talk about Grilo I will also be at Igalia’s booth most of the time so, in case you are planning to attend the event, I should be easy to find, feel free to come and talk to us, I will be happy to show you some of the cool stuff we are doing at Igalia, Grilo included ;)

Aug 10

Grilo session at GUADEC – slides

For those interested, you can find the slides of my presentation on Grilo here.

Jun 10

Grilo at LinuxTag

If you are in Berlin for LinuxTag and you are interested in Grilo or you are simply curious about it you probably want to attend my presentation: ‘Seamless Integration of Multimedia Services’ tomorrow at 4:00pm in room Berlin II.