All posts by agomes

Chromium, ozone, wayland and beyond

Over the past 2 months my fellow Igalian Frédéric Wang and I have been working on improving the Wayland support in Chromium. I will be detailing on this post some of the progresses we’ve made and what our next steps are.

Background

The more Wayland evolved as a mature alternative to the traditional X11-based environments, the more the Chromium community realized efforts were needed to run Chrome natively and flawlessly on such environments. On that context, the Ozone project began back in 2013 with the main goal of making porting Chrome to different graphics systems easier.

Ozone consists in a set of C++ classes for abstracting different window systems on Linux.
It provides abstraction for the construction of accelerated surfaces underlying Aura UI framework, input devices assignment and event handling.

Yet in 2013, Intel presented an Ozone/Wayland backend to the Chromium community. Although project progressed really well, after some time of active development, it got stalled: its most recent update is compatible with Chromium m49, from December/2015.

Leaping forward to 2016, a Wayland backend was added to Chromium upstream by Michael Forney. The implementation is not as feature complete as Intel’s (for example, drag n’ drop support is not implemented, and it only works with specific build configurations of Chrome), but it is definitively a good starting point.

In July, I experimented a bit this existing Wayland backend in Chromium. In summary, I took the existing wayland support upstream, and “ported” the minimal patches from Intel’s 01.org repository, being able to get ContentShell running with Wayland. However, given how the Wayland backend in Chromium upstream is designed and implemented, the GPU process failed to start, and HW rendering path was not functional.

weston_wayland_chromium

Communication with Google

Our main goal was to make Chrome capable of running natively on a Wayland-based environment, with hardware accelerated rendering enabled through the GPU process. After the investigation phase, the natural follow up step for us at Igalia was to contact Chrome hackers at Google, share our plans to Chromium/Wayland, and gather feedback. We reached out to @forney and @rjkroege, and once we heard back, we could understand why “porting” the missing bits from Intel’s original Ozone/Wayland implementation could not be the best choice on the long run.

Basically, the Ozone/Wayland backend present in Chromium upstream is designed in accordance to the original Mus+Ash Project’s architecture: the UI and GPU components would live in the same process. This way, the Wayland connection and other objects, that belong to the UI component, would be accessible to the GPU component and accelerated HW rendering could be implemented without any IPC.

This differs from Intel’s original implementation, where the GPU process owns the Wayland connections and objects, and the legacy Chromium IPC system is used to pass Wayland data from the GPU process to the Browser process.

The latest big Chromium refactor, since the Content API modularization, is the so called Mus+Ash (read "mustash") effort. In an over-summarized definition, it consists of factoring out specific parts of Chromium out of Chrome into "services", each featuring well defined interfaces for communication via Mojo IPC subsystem.

It was also pointed to us that eventually the UI and GPU components will be split into different processes. At that point, the same approach as the one done for the DRM/GBM Ozone backend was advised in order to implement communication between UI and GPU components.

Igalia improvements

Given that “ash” is a ChromeOS-oriented UI component, it seems logical to say that “mus+ash” project’s primarily focus is on the ChromeOS platform. Well, with that in mind, we thought we just needed to: (1) make a ChromeOS/Ozone/Wayland build of Chrome; (2) run ‘chrome –mash –ozone-platform=wayland’; (c) voilà, all should work.

WRONG!

When Fred and I started on this project two months ago, the situation was:

  • Chrome/Ozone/Wayland configuration was only build-able on ChromeOS and failed to run by default.
  • The Ozone project documentation was out of date (e.g. it referred to GYP and the obsolete Intel Ozone/Wayland code)
  • The Wayland code path was not tested by any buildbots, and could break without being noticed at this time.

After various code clean-ups and build fixes, we could finally launch Chrome off of a ChromeOS/Mash/Ozone/X11 build.

With further investigation, we verified that ChromeOS/Mash/Ozone/Wayland worked in m52 (when hardware accelerated rendering support was added to the Wayland backend), m53 and m54 checkouts, but not in m55. We bisected commits between m54 and m55, identified the regressions, and fixed them in issues 2387063002 and 2389053003. Finally, chrome –mash –ozone-platform=wayland was running with hardware accelerated rendering enabled on “off-device” ChromeOS builds.

screenshot-from-2016-11-14-00-51-02

 

Linux desktop builds

Our next goal at this point of the collaboration was to make it possible to build and run chrome –mash off of a regular Linux desktop build, through Ozone/Wayland backend.

After discussing about it on the ozone-dev mailing, we submitted and upstreamed various patches, converging to a point where a preliminary version of Chrome/mash/Ozone/Wayland was functional on a non-chromeos build, as shown below.

chrome_mash_ozone_wayland_non-chromeos-oct10

Note that this version of Chrome is stated above as “preliminary” because although it launches off of a regular desktop Linux build, it still runs with UI components from the ChromeOS UI environment, including a yellow bottom tray, and other items part of the so called “ash” Desktop environment.

Improving developers’ ecosystem

As mentioned, when we started on this collaboration, the Ozone project featured an out of date documentation, compilation and execution failures, and was mostly not integrated to Chromium’s continuous testing system (buildbots, etc). However, given that Ozone is the abstraction layer we would work on in order to get a Wayland backend up and running for Chrome, we found it sensible to improve this scenario.

By making use of Igalia’s community work excellence, many accomplishments were made
on this front, and now Ozone documentation is up-to-date.

Also, after having fixed Ozone/Wayland recurring build and execution failures, Fred and I realized the Ozone/Wayland codepath in Chromium needed to be exercised by a buildbot, so that its maintenance burden is reduced. Fred emailed ozone-dev about it and we started contributing to it, together with the Googler thomasanderson@. As a result,  Ozone/Wayland backend was made part of Chromium’s continuous integration testing system.

Also, Fred made a great post about the current Ozone/Wayland code in Chromium, its main classes and some experimentation done about adding Mojo capability to it. I consider it is a must-read follow on on this post.

Next steps

Today chrome —mash launches within an “ash” window, where the “ash” environment works as a shell for child internal windows, being Chrome one of them.
Being “mash” a shortener for “mus+ash”, our next step is to “eliminate”
the “ash” aspects from the execution path, resulting on a “Chrome Mus” launch.

This project is sponsored by Renesas Electronics …

renesas_logomark_l

… and has been performed by the great Igalian hacker Frédéric Wang and I on behalf of Igalia.

igalia-logo-364x130

Making Chromium’s PDFium greater

One of the coolest things about working at Igalia, is that we have the chance to work on real world problems from day 1, either by improving an existing solution or creating new ones, based on open source technologies.

During the past month, I worked on a specific aspect of Chromium’s PDF engine, PDFium: extending PDFium’s capability so that Annotations can be manipulated by Acrobat JS scripting.

In practice, the goal was to make the specific scenario described on bug 492 to behave similarly in Chromium/PDFium, when compared to IE/Acrobat.

“Mission given, is mission accomplished”

The following Acrobat JS APIs were implemented:

Document::URL
Document::gotoNamedDest
Document::syncAnnotScan (only stubbed out)
Document::getAnnot
Document::getAnnots

… and Acrobat JS scripts like the snippet below can now be successfully executed by PDFium, making 200k+ PDF files to behave in Chromium/PDFium similarly to IE/Acrobat –
with regards to the way Annotations are manipulated.

1 this.disclosed = true;
2 this.syncAnnotScan();
3 var u = this.URL;
4 var args = u.split('calcrtid=');
5 var tags;
6 if (args.length > 1) {
7   tags = args[1].split('&');
8   this.gotoNamedDest(tags[0]);
9   var a = this.getAnnot(this.pageNum, tags[0]);
10   if (a != null) {
11     a.hidden = false;
12     var rect = a.rect;
13     if (rect != null) {
14       this.scroll(rect[2] + 100, rect[3] + 100);
15     }
16   }
17 }

Also as part of this project, it was possible to fix two discrepancies in PDFium, again if compared to Acrobat: 1) Hidden annotations were shown; 2) PDFium’s positioning Text Markup annotations.

About (2) specifically, on Acrobat if an “/AP” clause is present as part of an “/Annot” definition, the coordinates used to draw annotations come from “/Rect” values. On the other hand, when “/AP” is not defined, “/QuadPoints” values are used to grab the annotation coordinates from. The divergence was coming from the fact that PDFium uses “/Rect” regardless the presence or absence of “/AP”. This CL fixed it.

To wrap up the work, I was also able to extend PDFium’s main testing tool (pdfium_test) to actually run Acrobat JS tests (follow up), as well as make lots of driven-by clean ups, and change PDFium’s main client (Chromium) to deal better when its async nature.

The PDFium community

Throughout the project, interactions with the PDFium community were smooth:

  • PDFium contribution flow follows pretty much the Chromium’s:
    https://bugs.chromium.org to report issues and Codereview to get the work reviewed.
  • Once approved, a CL is committed through a “Commit Queue” bot – which ensures a CLs correctness by running PDFium regression suite.
  • Owners are really responsive.

Acknowledgments

My work in PDFium is sponsored by Bloomberg. Thank you all!

igalia-bloomberg

Understanding Chromium’s runtime ozone platform selection

Requisites

For the context of this post, it is assumed a ‘content_shell’ ChromeOS GN build produced with the following commands:

$ gn gen --args='target_os="chromeos"
                 use_ozone=true
                 ozone_platform_wayland=true
                 use_wayland_egl=false
                 ozone_platform_x11=true
                 ozone_platform_headless=true
                 ozone_auto_platforms=false' out-gn-ozone

$ ninja -C out-gn-ozone blink_tests

Ozone at runtime

Looking at the build arguments above, one can foresee that the content_shell app will be able to run on the following Graphics backends: Wayland (no EGL), X11, and “headless” – and it indeed is. This happens thanks to Chromium’s Graphics layer abstraction, Ozone.

So, in order to run content_shell app on Ozone platform “bleh”, one simply does:

$ out-gn-ozone/content_shell --ozone-platform="bleh"

Simple no? Well, yes .. and not as much.

The way the desired Ozone platform classes/objects are instantiated is interesting, involving c++ templates, GYP/GN hooks, python generated code, and more. This post aims to detail the process some more.

Ozone platform selection logic

Two methods kick off OzonePlatform instantiation: ::InitializeForUI and ::InitializeForGPU . They both call ::CreateInstance(), which is our starting point.

This is how simple it looks:

63 void OzonePlatform::CreateInstance() {
64     if (!instance_) {
(..)
69         std::unique_ptr<OzonePlatform> platform =
70             PlatformObject<OzonePlatform>::Create();
71
72         // TODO(spang): Currently need to leak this object.
73         OzonePlatform* pl = platform.release();
74         DCHECK_EQ(instance_, pl);
75     }
76 }

Essentially, when PlatformObject<T>::Create is ran (lines 69 and 70), it ends up calling a method named Create{T}Bleh, where

  • “T” is the template argument name, e.g. “OzonePlatform”.
  • “bleh” is the value passed to –ozone-platform command line parameter.

For instance, in the case of ./content_shell –ozone-platform=x11, the method called would CreateOzonePlatformX11, following the pattern Create{T}Bleh (i.e. “Create”+”OzonePlatform”+”X11”).

The actual logic

In order to understand how PlatformObject class works, lets start by looking at its definition (ui/ozone/platform_object.h & platform_internal_object.h):

template <class T> class PlatformObject {
 public:
  static std::unique_ptr<T> Create();
};
16 template <class T>
17 std::unique_ptr<T> PlatformObject<T>::Create() {
18     typedef typename PlatformConstructorList<T>::Constructor Constructor;
19
20     // Determine selected platform (from --ozone-platform flag, or default).
21     int platform = GetOzonePlatformId();
22
23     // Look up the constructor in the constructor list.
24     Constructor constructor = PlatformConstructorList<T>::kConstructors[platform];
26     // Call the constructor.
27     return base::WrapUnique(constructor());
28 }

In line 24 (highlighted above), the ozone platform runtime selection machinery actually happens. It retrieves a Constructor, which is a typedef for a PlatformConstructorList<T>::Constructor.

By looking at the definition of PlatformConstructorList class (below), Constructor is actually a pointer to a function that returns a T*.

14 template <class T>
15 struct PlatformConstructorList {
16     typedef T* (*Constructor)();
17     static const Constructor kConstructors[kPlatformCount];
18 };

Ok, so basically here is what we know this far:

  1. OzonePlatform::CreateInstance method calls OzonePlatform<bleh>::Create
  2. OzonePlatform<bleh>::Create picks up an index and retrieves a PlatformConstructorList<bleh>::Constructor (via kConstructor[index])
  3. PlatformConstructorList<bleh>::Constructor is a typedef to a function pointer that returns a bleh*.
  4. (..)
  5. This chain ends up calling Create{bleh}{ozone_platform}()

But wait! kConstructors, the array of pointers to functions – that solves the puzzle – is not defined anywhere in src/! 🙂

This is because its actual definition is part of some generated code triggered by specific GN/GYP hooks. They are:

  • generate_ozone_platform_list which generates out/../platform_list.cc,h,txt
  • generate_constructor_list which generates out/../constructor_list.cc though generate_constructor_list.py

In the end out/../generate_constructor_list.cc has the definition of kConstructors.

Again, in the specific case of the given GN build arguments, kConstructors would look like:

template <> const OzonePlatformConstructor
PlatformConstructorList<ui::OzonePlatform>::kConstructors[] = {
    &ui::CreateOzonePlatformHeadless,
    &ui::CreateOzonePlatformWayland,
    &ui::CreateOzonePlatformX11,
};

Logic wrap up

  • GYP/GN hooks are ran at build time, and generate plaform_list.{txt,c,h} as well as constructor_list.cc files respecting ozone_platform_bleh parameters.
    • constructor_list.cc has PlatformConstructorList<bleh>kConstructors actually populated.
  • ./content_shell –ozone-platform=bleh is called
  • OzonePlatform::InitializeFor{UI,Gpu}()
  • OzonePlatform::CreateInstance()
  • PlaformObject<OzonePlatformBleh>::Create()
  • PlatformConstructorList<bleh>::Constructor is retrieved – it is a pointer to a function stored in PlatformConstructorList<bleh>::kConstructor
  • function is ran and an OzonePlatformBleh instance is returned.

[Chromium] content_shell running on Wayland desktop (Weston Compositor)

During my first weeks at Igalia, I got an interesting and promising task of Understand the status of Wayland support in Chromium upstream.

At first I could see clear signs that Wayland is being actively considered by the Chromium community:

  1. Ozone/Wayland project by Intel – which was my starting point as described later on.
  2. The meta bug “upstream wayland backend for ozone (20% project)“, which has some recent activity by the Chromium community.
  3. This comment in the chromium-dev mailing list (by a Googler):
  4. “(..) I ‘d recommend using ToT. Wayland support is a work-in-progress and newer trees will probably be better.”.

    Chromium’s DEPS file has “wayland” as one its core dependency (search for “wayland”).

Next step, naturally, was get my hands dirty, compiling and experimenting with it. I decided to start with content_shell.


Environment:

Ubuntu 16.04 LTS, regular Gnome session (with X) and Weston Wayland compositor running (no ‘xwayland’ installed) to run Wayland apps.

GYP_DEFINES

component=static_library use_ash=1 use_aura=1 chromeos=0 use_ozone=1 ozone_platform_wayland=1 use_wayland_egl=1 ozone_auto_platforms=0 use_xkbcommon=1 clang=0 use_sysroot=0 linux_use_bundled_gold=0

(note: GYP was used for personal familiarity with it, but GN applies fine here).

Chromium version

Base SHA 5da650481 (as of 2016/May/17)

Initial results

As is, content_shell built fine, but hung at runtime upon launch, hitting a CHECK at desktop_factory_ozone.cc(21).

Analysis and Action

After understanding current Ozone/Wayland upstream support, compare designs/code against 01.org, I could start connecting some of the missing dots.

The following files were “ported” from 01.org:

  • ui/views/widget/desktop_aura/desktop_drag_drop_client_wayland.cc / h
  • ui/views/widget/desktop_aura/desktop_screen_ozone_wayland.cc / h
  • ui/views/widget/desktop_aura/desktop_window_tree_host_ozone_wayland.cc / h

And then, I implemented DesktopFactoryOzoneWayland class (ui/views/widget/desktop_factory_ozone_wayland.cc/h) – it inherits from DesktopFactoryOzone, and implements the following pure virtual methods ::CreateWindowTreeHost and ::CreateDesktopScreen.

Initial result

After that, I could build and run content_shell with Weston Wayland Compositor (with no ‘xwayland’ installed). See a quick preview below.

Remarks

As is, the UI process owns the the Wayland connection, and GPU process runs without GL support.

UI processes initializes Ozone by calling:

#0 ui::WaylandSurfaceFactory::WaylandSurfaceFactory
#1 ui::OzonePlatformWayland::InitializeUI
#2 ui::OzonePlatform::InitializeForUI();
#3 aura::Env::Init()
#4 aura::Env::CreateInstance()
#5 content::BrowserMainLoop::InitializeToolkit()
(…)
#X content::ContentMain()
<UI PROCESS LAUNCH>
On the other side, GPU process gets initialized by calling:
#0 ui::WaylandSurfaceFactory::WaylandSurfaceFactory
#1 ui::OzonePlatformWayland::InitializeGPU
#2 ui::OzonePlatform::InitializeForGPU();
#3 gfx::InitializeStaticGLBindings()
#4 gfx::GLSurface::InitializeOneOffImplementation()
#5 gfx::GLSurface::InitializeOneOff()
#6 content::GpuMain()
<GPU PROCESS LAUNCH>

Differently from UI process, the GPU process call does not initialize OzonePlatformWayland::display_ and instead passes ‘nullptr’ to WaylandSurfaceFactory ctor.

Down the road on the GPU processes initialization WaylandSurfaceFactory::LoadEGLGLES2Bindings is called but bails out earlier explicitly because display_ is NIL.
Then, UI process falls back to software rendering (see call to WaylandSurfaceFactory::CreateCanvasForWidget).

Next step

  • So far I have experimented Ozone/Wayland support using “linux” as the target OS. As far as I can tell, most of the Ozone work upstream though has been focusing on “chromeos” builds instead (e.g. ozone/x11).
  • Hence the idea is to clean up the code and agree with Googlers / 01.org (Intel) people about how to best make use of this code.
  • It is being also discussed with some Googlers what the best way to tackle this lack of GL support is. Some real nice stuff are on the pipe here.