Category Archives: chromium

Chromium Mus/Ozone update (H1/2017): wayland, x11

browsers, chrome, chromium, igalia, mus, ozone, wayland, weston

Since January, Igalia has been working on a project whose goal is to make the latest Chromium Browser able to run natively on Wayland-based environments. The project has various phases, requires us to carve out existing implementations and align our work with the direction Chromium’s mainline is taking.

In this post I will provide an update on the progresses we have made over 2017/H1, as well as our plans coming next.

In order to jump straight to the latest results section (including videos) without the details, click here.

Background

In 2016/Q4, my fellow Igalian Frédéric Wang and I ran a warm-up project to check the status of the existing Wayland support in Chromium’s mainline repository, and estimate how much work was needed to get the full (and latest) Chromium browser running on Wayland.

As part of this warm-up we were able to build and launch ChromeOS’s Chrome for both desktop and embedded Linux distributions, featuring either X11 or Wayland. Automotive Grade Linux running on the Renesas’ R-car M3 board is an example of the embedded environments we tested.

Mus+ash on LinuxOS (Nov/2016).
Mus+ash on LinuxOS (Nov/2016)

Although this was obviously not our end goal (some undesirable ChromeOS widgets were visible at the bottom), it allowed us to verify the overall performance of the build, and experiment with things a bit. Here is a brief summary of the most relevant findings:

  • There is an ongoing effort to reimplement the Ash (aka “cash” or classic Ash) Window Manager on top of the new UI Service (aka Mus). It is the so called mus+ash project, a ChromeOS oriented effort that is the continuation of Mandoline project.

  • It is possible to build mus+ash for various platforms including Linux, ChromeOS and Windows. On Linux specifically, it is possible to make off-device ChromeOS builds of mus+ash, and run it on desktop Linux for testing purposes. A more minimalistic Window Manager version is also available in //mash/simple_wm, and should run on regular Linux builds too.

  • mus+ash can be built with Ozone enabled. This means that it can run with the various backends Ozone has. It is worth saying that the upstream focus seems to be the Ozone/DRM-GBM backend, for ChromeOS.

  • Ozone itself has morphed over time from an abstraction layer underneath the Aura toolkit, to be a layer underneath Mus.

Last, we could publish some worth reading content:

2017 developments

At the beginning of this new phase of the project, we knew we needed to work on two different levels, in order to have the Chromium browser running on desktop Linux, ideally without functionality losses if compared against the stock Chromium browser on X11: both Mus and Ozone needed to support ‘external window’ mode.

For the sake of completeness, the term external window mode above is the terminology we chose to represent a regular desktop application on Linux, where the host Window Manager takes care of windowing actions like maximize, minimize, restore and fullscreen the application window. Also, the application itself reacts to content size changes accordingly. Analogously, when we say an application runs in internal window mode, it runs within the (M)ash shell environment, the builtin Window Manager that powers ChromeOS builds. Applications in this mode do not interact with the host WM.

A huge pro about how mus+ash is being implemented is that the Chrome browser itself already works as it ought to in non-ChromeOS Mus-based environments: either we are running Mus in internal or external window modes, Chrome will work just like Chrome for a Linux desktop ought to.

That being said, we identified the following set of tasks, on both Ozone and Mus sides.

Ozone tasks:

Extend Ozone so that both Window Manager provided window decorations (like a regular X11 window on Ubuntu) and Chromium’s builtin window decoration work flawlessly.
On Wayland, window decorations can be provided either by the client side (application), or by the Wayland server (compositor). The fact that Weston does not provide window decorations by default, forces us to support Chromium’s builtin one for the good.
In case of the Chromium’s builtin window decorations …
… add support for basic windowing functionality like maximize, minimize, restore and fullscreen, as well as window dragging and resizing.
Add support to “window close”.
In internal window mode, there is no concept of window closing, because the outer/native Ozone window represents the Window Manager display, which is not supposed to get closed. In external window mode, windows can be closed freely, as per the needs of the user.
Add support for multi window browsing.
Each browser window should be backed by its own acceleratedWidget. This also includes being able to draw widgets that on stock Linux/X11 builds use native windows: tooltips, (nested and context) menus.
Handle keyboard focus activation when switching windows.
Again in ‘internal window’ mode the outer/native Ozone window is unique and represents the Window Manager display, not losing or gaining focus at any time. Focus switching of inner windows is handled by mus+ash. In ‘external window’ mode, user can open as many Browser windows as he wants, and focus switches at the Window Manager level, should reflect on the application focus.

Mus tasks:

Fix assumptions that make sense for mus+ash on ChromeOS only.
The fact that a display::Display instance mapped always to a single ui::ws::Display instance.
Ownership model
Some Mus objects have slightly different ownership in external window mode: ws::Display, ws::WindowManagerState, ws::WindowManagerDisplayRoot and ws::WindowTree
The plan

After meeting with rjkroege@ at BlinkOn 7, we defined a highlevel plan to tackle the project. These were the main action points:

1) Extend the mus_demo to work in ‘external window’ mode.
2) Start fixing 1:1 assumptions in the code, e.g. display::Display ui::ws::Display.
3) Extend Mus to work on ‘external window’ mode.
4) Extend Ozone to work on ‘external window’ mode.
5) Make the code that handles the existing –mus command line parameter non-ChromeOS specific.

With this 5 highlevel steps done, we would be able to get Chrome/Mus running on desktop Linux, on the various Ozone backends.

The action
Mus Demo

We were able to get mus_demo working in ‘external window’ mode, by making use of the existing WindowTreeHostFactory API.

1:1 assumptions

Although WindowTreeHostFactory was in place for the creation WindowTreeHost instances, both Mus and Ozone still had assumptions that only applied in a ChromeOS context. The Googler kylechar@ jumped in and fixed some of them, helping out on our effort.

Mus and Ozone carve out

In order to get the 3rd and 4th steps going, we decided to switch our main development repository to a GitHub fork, so that we could expedite reviews and progresses. Given Igalia’s excellence in carrying downstream forks of large projects forward, we established a contribution process and a rebase strategy that would allow us to move at a good pace, and still stay as close as possible to Chromium’s tip of trunk.

These are some of the main changes in our downstream repository:

  • Mus and Aura/Mus were changed to support an ‘external window’ mode flow: there is a single WindowTreeClient (Aura/Mus) instance that maps to another single WindowTree instance (Mus). The relationship is invariable and is ensured by ui::ws::WindowTreeHostFactoryRegistrar class.

  • In this new set up, ui::ws::WindowTreeHostFactory::CreatePlatformWindow can create as many WindowTreeHost / ui::ws::Display instances as needed. ui::ws::Display triggers creation of PlatformDisplay objects, which hold Ozone window handles. Hence, every Chromium window (and some browser widgets) gets backed by its own acceleratedWidget.

  • In mus+ash, there are some operations accomplished through a cooperation between both Mus and Ash, or Mus and Aura/Mus sides. For example, setting “frame decorations” values in mus+ash goes through the following path:

    1) ash::mus::WindowManager get frame decoration values as per the “material design” in use and passes it to aura::WindowTreeClient::SetFrameDecorationValues.
    2) WindowTree::WmSetFrameDecorationValues
    3) WindowManagerState::SetFrameDecorationValues
    4) UserDisplayManager::OnFrameDecorationValuesChanged
    5) ScreenMus::OnDisplays()
    6) These values are used later on to draw “non client frame” area of the Browser window, which “frame” that contains the Web contents area.

    On Chrome/Mus LinuxOS, we skip this round trip by using the same “non client frame view” as stock Linux/X11 Chrome: OpaqueBrowserFrameView.

  • In mus+ash all Browser widgets creation take the DesktopNativeWidgetAura path. This implies a new WindowPort and new WindowTreeHost instances per widget. Adding support for this in Mus and Ozone sides would require lots of work and refactory. Hence, we again decided to use the stock Linux/X11 flow: for widgets currently backed by a native window (tooltips, menus) we use the NativeWidgetAura path, whereas for others widgets (bookmark banner and zoom in/out banners, URL completion window, status bubble, etc) we use NativeWidgetAura. Also, this choice made extending Ozone accordingly simpler.

Status and next steps

We have reached a point where we can show Chrome Ozone/Mus on desktop Linux, on using both X11 and Wayland backends, and here is how it is looking like today:

Wayland:

X11:

The –mus and –ozone-platform={name} command line parameters control the Chrome configuration. Please note that the same Chrome binary is used.

Some of our next steps for Chromium Mus/Ozone are:

  • Continue to fix the windowing features (namely window resize and dragging, as well as drag and drop) when Chromium’s builtin window decorations are used.
  • Provide updated yocto builds on Igalia’s meta-browser fork.
  • Support newer shell protocols like XDG v6, supported by Fedora 25.
  • Ensure no feature losses when compared to stock Chromium X11/Linux.
  • Ensure there is no performance penalties when compared to stock Chromium X11/Linux.
  • Start to upstream some of the changes.

We are also considering providing prebuilt binaries, so that earlier adopters can test the status.

This project is sponsored by Renesas Electronics …

renesas_logomark_l

… and is being performed by Igalian hacker Maksim Sisov and Antonio Gomes (me) on behalf of Igalia, being Frederic Wang an emeritus contributor.

igalia-logo-364x130

Chromium, ozone, wayland and beyond

Over the past 2 months my fellow Igalian Frédéric Wang and I have been working on improving the Wayland support in Chromium. I will be detailing on this post some of the progresses we’ve made and what our next steps are.

Background

The more Wayland evolved as a mature alternative to the traditional X11-based environments, the more the Chromium community realized efforts were needed to run Chrome natively and flawlessly on such environments. On that context, the Ozone project began back in 2013 with the main goal of making porting Chrome to different graphics systems easier.

Ozone consists in a set of C++ classes for abstracting different window systems on Linux.
It provides abstraction for the construction of accelerated surfaces underlying Aura UI framework, input devices assignment and event handling.

Yet in 2013, Intel presented an Ozone/Wayland backend to the Chromium community. Although project progressed really well, after some time of active development, it got stalled: its most recent update is compatible with Chromium m49, from December/2015.

Leaping forward to 2016, a Wayland backend was added to Chromium upstream by Michael Forney. The implementation is not as feature complete as Intel’s (for example, drag n’ drop support is not implemented, and it only works with specific build configurations of Chrome), but it is definitively a good starting point.

In July, I experimented a bit this existing Wayland backend in Chromium. In summary, I took the existing wayland support upstream, and “ported” the minimal patches from Intel’s 01.org repository, being able to get ContentShell running with Wayland. However, given how the Wayland backend in Chromium upstream is designed and implemented, the GPU process failed to start, and HW rendering path was not functional.

weston_wayland_chromium

Communication with Google

Our main goal was to make Chrome capable of running natively on a Wayland-based environment, with hardware accelerated rendering enabled through the GPU process. After the investigation phase, the natural follow up step for us at Igalia was to contact Chrome hackers at Google, share our plans to Chromium/Wayland, and gather feedback. We reached out to @forney and @rjkroege, and once we heard back, we could understand why “porting” the missing bits from Intel’s original Ozone/Wayland implementation could not be the best choice on the long run.

Basically, the Ozone/Wayland backend present in Chromium upstream is designed in accordance to the original Mus+Ash Project’s architecture: the UI and GPU components would live in the same process. This way, the Wayland connection and other objects, that belong to the UI component, would be accessible to the GPU component and accelerated HW rendering could be implemented without any IPC.

This differs from Intel’s original implementation, where the GPU process owns the Wayland connections and objects, and the legacy Chromium IPC system is used to pass Wayland data from the GPU process to the Browser process.

The latest big Chromium refactor, since the Content API modularization, is the so called Mus+Ash (read "mustash") effort. In an over-summarized definition, it consists of factoring out specific parts of Chromium out of Chrome into "services", each featuring well defined interfaces for communication via Mojo IPC subsystem.

It was also pointed to us that eventually the UI and GPU components will be split into different processes. At that point, the same approach as the one done for the DRM/GBM Ozone backend was advised in order to implement communication between UI and GPU components.

Igalia improvements

Given that “ash” is a ChromeOS-oriented UI component, it seems logical to say that “mus+ash” project’s primarily focus is on the ChromeOS platform. Well, with that in mind, we thought we just needed to: (1) make a ChromeOS/Ozone/Wayland build of Chrome; (2) run ‘chrome –mash –ozone-platform=wayland’; (c) voilà, all should work.

WRONG!

When Fred and I started on this project two months ago, the situation was:

  • Chrome/Ozone/Wayland configuration was only build-able on ChromeOS and failed to run by default.
  • The Ozone project documentation was out of date (e.g. it referred to GYP and the obsolete Intel Ozone/Wayland code)
  • The Wayland code path was not tested by any buildbots, and could break without being noticed at this time.

After various code clean-ups and build fixes, we could finally launch Chrome off of a ChromeOS/Mash/Ozone/X11 build.

With further investigation, we verified that ChromeOS/Mash/Ozone/Wayland worked in m52 (when hardware accelerated rendering support was added to the Wayland backend), m53 and m54 checkouts, but not in m55. We bisected commits between m54 and m55, identified the regressions, and fixed them in issues 2387063002 and 2389053003. Finally, chrome –mash –ozone-platform=wayland was running with hardware accelerated rendering enabled on “off-device” ChromeOS builds.

screenshot-from-2016-11-14-00-51-02

 

Linux desktop builds

Our next goal at this point of the collaboration was to make it possible to build and run chrome –mash off of a regular Linux desktop build, through Ozone/Wayland backend.

After discussing about it on the ozone-dev mailing, we submitted and upstreamed various patches, converging to a point where a preliminary version of Chrome/mash/Ozone/Wayland was functional on a non-chromeos build, as shown below.

chrome_mash_ozone_wayland_non-chromeos-oct10

Note that this version of Chrome is stated above as “preliminary” because although it launches off of a regular desktop Linux build, it still runs with UI components from the ChromeOS UI environment, including a yellow bottom tray, and other items part of the so called “ash” Desktop environment.

Improving developers’ ecosystem

As mentioned, when we started on this collaboration, the Ozone project featured an out of date documentation, compilation and execution failures, and was mostly not integrated to Chromium’s continuous testing system (buildbots, etc). However, given that Ozone is the abstraction layer we would work on in order to get a Wayland backend up and running for Chrome, we found it sensible to improve this scenario.

By making use of Igalia’s community work excellence, many accomplishments were made
on this front, and now Ozone documentation is up-to-date.

Also, after having fixed Ozone/Wayland recurring build and execution failures, Fred and I realized the Ozone/Wayland codepath in Chromium needed to be exercised by a buildbot, so that its maintenance burden is reduced. Fred emailed ozone-dev about it and we started contributing to it, together with the Googler thomasanderson@. As a result,  Ozone/Wayland backend was made part of Chromium’s continuous integration testing system.

Also, Fred made a great post about the current Ozone/Wayland code in Chromium, its main classes and some experimentation done about adding Mojo capability to it. I consider it is a must-read follow on on this post.

Next steps

Today chrome —mash launches within an “ash” window, where the “ash” environment works as a shell for child internal windows, being Chrome one of them.
Being “mash” a shortener for “mus+ash”, our next step is to “eliminate”
the “ash” aspects from the execution path, resulting on a “Chrome Mus” launch.

This project is sponsored by Renesas Electronics …

renesas_logomark_l

… and has been performed by the great Igalian hacker Frédéric Wang and I on behalf of Igalia.

igalia-logo-364x130

Making Chromium’s PDFium greater

browsers, chromium, igalia, pdfium

One of the coolest things about working at Igalia, is that we have the chance to work on real world problems from day 1, either by improving an existing solution or creating new ones, based on open source technologies.

During the past month, I worked on a specific aspect of Chromium’s PDF engine, PDFium: extending PDFium’s capability so that Annotations can be manipulated by Acrobat JS scripting.

In practice, the goal was to make the specific scenario described on bug 492 to behave similarly in Chromium/PDFium, when compared to IE/Acrobat.

“Mission given, is mission accomplished”

The following Acrobat JS APIs were implemented:

– Document::URL
– Document::gotoNamedDest
– Document::syncAnnotScan (only stubbed out)
– Document::getAnnot
– Document::getAnnots

… and Acrobat JS scripts like the snippet below can now be successfully executed by PDFium, making 200k+ PDF files to behave in Chromium/PDFium similarly to IE/Acrobat –
with regards to the way Annotations are manipulated.

1 this.disclosed = true;
2 this.syncAnnotScan();
3 var u = this.URL;
4 var args = u.split('calcrtid=');
5 var tags;
6 if (args.length > 1) {
7   tags = args[1].split('&');
8   this.gotoNamedDest(tags[0]);
9   var a = this.getAnnot(this.pageNum, tags[0]);
10   if (a != null) {
11     a.hidden = false;
12     var rect = a.rect;
13     if (rect != null) {
14       this.scroll(rect[2] + 100, rect[3] + 100);
15     }
16   }
17 }

Also as part of this project, it was possible to fix two discrepancies in PDFium, again if compared to Acrobat: 1) Hidden annotations were shown; 2) PDFium’s positioning Text Markup annotations.

About (2) specifically, on Acrobat if an “/AP” clause is present as part of an “/Annot” definition, the coordinates used to draw annotations come from “/Rect” values. On the other hand, when “/AP” is not defined, “/QuadPoints” values are used to grab the annotation coordinates from. The divergence was coming from the fact that PDFium uses “/Rect” regardless the presence or absence of “/AP”. This CL fixed it.

To wrap up the work, I was also able to extend PDFium’s main testing tool (pdfium_test) to actually run Acrobat JS tests (follow up), as well as make lots of driven-by clean ups, and change PDFium’s main client (Chromium) to deal better when its async nature.

The PDFium community

Throughout the project, interactions with the PDFium community were smooth:

  • PDFium contribution flow follows pretty much the Chromium’s:
    https://bugs.chromium.org to report issues and Codereview to get the work reviewed.
  • Once approved, a CL is committed through a “Commit Queue” bot – which ensures a CLs correctness by running PDFium regression suite.
  • Owners are really responsive.

Acknowledgments

My work in PDFium is sponsored by Bloomberg. Thank you all!

igalia-bloomberg

Understanding Chromium’s runtime ozone platform selection

browsers, chromium, igalia, ozone, wayland

Requisites

For the context of this post, it is assumed a ‘content_shell’ ChromeOS GN build produced with the following commands:

$ gn gen --args='target_os="chromeos"
                 use_ozone=true
                 ozone_platform_wayland=true
                 use_wayland_egl=false
                 ozone_platform_x11=true
                 ozone_platform_headless=true
                 ozone_auto_platforms=false' out-gn-ozone

$ ninja -C out-gn-ozone blink_tests

Ozone at runtime

Looking at the build arguments above, one can foresee that the content_shell app will be able to run on the following Graphics backends: Wayland (no EGL), X11, and “headless” – and it indeed is. This happens thanks to Chromium’s Graphics layer abstraction, Ozone.

So, in order to run content_shell app on Ozone platform “bleh”, one simply does:

$ out-gn-ozone/content_shell --ozone-platform="bleh"

Simple no? Well, yes .. and not as much.

The way the desired Ozone platform classes/objects are instantiated is interesting, involving c++ templates, GYP/GN hooks, python generated code, and more. This post aims to detail the process some more.

Ozone platform selection logic

Two methods kick off OzonePlatform instantiation: ::InitializeForUI and ::InitializeForGPU . They both call ::CreateInstance(), which is our starting point.

This is how simple it looks:

63 void OzonePlatform::CreateInstance() {
64     if (!instance_) {
(..)
69         std::unique_ptr<OzonePlatform> platform =
70             PlatformObject<OzonePlatform>::Create();
71
72         // TODO(spang): Currently need to leak this object.
73         OzonePlatform* pl = platform.release();
74         DCHECK_EQ(instance_, pl);
75     }
76 }

Essentially, when PlatformObject<T>::Create is ran (lines 69 and 70), it ends up calling a method named Create{T}Bleh, where

  • “T” is the template argument name, e.g. “OzonePlatform”.
  • “bleh” is the value passed to –ozone-platform command line parameter.

For instance, in the case of ./content_shell –ozone-platform=x11, the method called would CreateOzonePlatformX11, following the pattern Create{T}Bleh (i.e. “Create”+”OzonePlatform”+”X11”).

The actual logic

In order to understand how PlatformObject class works, lets start by looking at its definition (ui/ozone/platform_object.h & platform_internal_object.h):

template <class T> class PlatformObject {
 public:
  static std::unique_ptr<T> Create();
};
16 template <class T>
17 std::unique_ptr<T> PlatformObject<T>::Create() {
18     typedef typename PlatformConstructorList<T>::Constructor Constructor;
19
20     // Determine selected platform (from --ozone-platform flag, or default).
21     int platform = GetOzonePlatformId();
22
23     // Look up the constructor in the constructor list.
24     Constructor constructor = PlatformConstructorList<T>::kConstructors[platform];
26     // Call the constructor.
27     return base::WrapUnique(constructor());
28 }

In line 24 (highlighted above), the ozone platform runtime selection machinery actually happens. It retrieves a Constructor, which is a typedef for a PlatformConstructorList<T>::Constructor.

By looking at the definition of PlatformConstructorList class (below), Constructor is actually a pointer to a function that returns a T*.

14 template <class T>
15 struct PlatformConstructorList {
16     typedef T* (*Constructor)();
17     static const Constructor kConstructors[kPlatformCount];
18 };

Ok, so basically here is what we know this far:

  1. OzonePlatform::CreateInstance method calls OzonePlatform<bleh>::Create
  2. OzonePlatform<bleh>::Create picks up an index and retrieves a PlatformConstructorList<bleh>::Constructor (via kConstructor[index])
  3. PlatformConstructorList<bleh>::Constructor is a typedef to a function pointer that returns a bleh*.
  4. (..)
  5. This chain ends up calling Create{bleh}{ozone_platform}()

But wait! kConstructors, the array of pointers to functions – that solves the puzzle – is not defined anywhere in src/! 🙂

This is because its actual definition is part of some generated code triggered by specific GN/GYP hooks. They are:

  • generate_ozone_platform_list which generates out/../platform_list.cc,h,txt
  • generate_constructor_list which generates out/../constructor_list.cc though generate_constructor_list.py

In the end out/../generate_constructor_list.cc has the definition of kConstructors.

Again, in the specific case of the given GN build arguments, kConstructors would look like:

template <> const OzonePlatformConstructor
PlatformConstructorList<ui::OzonePlatform>::kConstructors[] = {
    &ui::CreateOzonePlatformHeadless,
    &ui::CreateOzonePlatformWayland,
    &ui::CreateOzonePlatformX11,
};

Logic wrap up

  • GYP/GN hooks are ran at build time, and generate plaform_list.{txt,c,h} as well as constructor_list.cc files respecting ozone_platform_bleh parameters.
    • constructor_list.cc has PlatformConstructorList<bleh>kConstructors actually populated.
  • ./content_shell –ozone-platform=bleh is called
  • OzonePlatform::InitializeFor{UI,Gpu}()
  • OzonePlatform::CreateInstance()
  • PlaformObject<OzonePlatformBleh>::Create()
  • PlatformConstructorList<bleh>::Constructor is retrieved – it is a pointer to a function stored in PlatformConstructorList<bleh>::kConstructor
  • function is ran and an OzonePlatformBleh instance is returned.