To stream or not to stream

Multimedia blog and other fancy stuff

The WPE Platform API

How to use the new WPE Platform API to replace the old WPE WebKit backends.

In the past, when you needed to adapt WPE WebKit to a new platform, or integrate it with your own system/application not based on Wayland, you had to develop a specific backend. I wrote two blog posts in the past about this topic (One about the process of creating a new WPE backend and another about using EGLStreams in a WPE backend) and, to be honest, it was not really straightforward.

Since WPE WebKit 2.50, a new system has been designed to replace all this by a more modern and intuitive approach, making a lot easier to integrate WPE WebKit into your application. The new API is called WPE Platform and allows you to define the equivalent of the old backend system into your own executable. At the moment of this post (version 2.52), the WPE Platform is still in development and the official release is foreseen for the next stable version 2.54. Nevertheless, it is already stable enough to start playing with it.

This post is going to explain step by step how you can use WPE WebKit as a web view inside a simple X11 window using EGL with full hardware acceleration and zero-copy of the graphical hardware buffers while avoiding the complexity of writing an external backend. The X11/EGL window itself will be managed by the GLFW library.

N.B. GLFW is only used here as a convenient, cross-platform way of creating a window and an EGL context with a minimal amount of code. It is not the topic of this post. What matters here is the generic contract to follow to implement a custom WPE Platform. It consists basically in three GObject classes and a small set of virtual methods that would look exactly the same if we had chosen SDL, Qt, or directly a raw X11 Window instead of GLFW.

The reference project for this post is blog-the-wpe-platform-api. It implements a complete and minimal WPE Platform taking into account the basic user’s interactions (keyboard, scrolling, mouse, etc…).

The version 0.0 of this code implements the bases to initialize a web view using WPE (it will need a Wayland compositor to run). While the version 1.0 also integrates all the components needed to implement the WPE Platform. From the implementation point of view, the only difference is the usage of a custom WPEDisplay instead of the default one:

// Create and connect our GLFW-backed WPE display
WPEDisplay* display = wpe_glfw_display_new();
GError* error = NULL;
if (!wpe_display_connect(display, &error))
{
    ...
    return EXIT_FAILURE;
}

// Create the WebKitWebView
WebKitWebView* web_view = WEBKIT_WEB_VIEW(g_object_new(WEBKIT_TYPE_WEB_VIEW, "display", display, NULL));
...

Building and running the example project #

If your operating system has a development package for libWPEWebKit-2.0 version 2.52 or above, the easiest way is to install this package from your distribution. Else, you can build WPE WebKit by yourself:

  • or by using the webkit-container-sdk reading the instructions provided on the project,
  • or by building the library locally.

If you already have WPE WebKit installed, you can go directly to the build of the example project.

Build WPE WebKit locally #

The following instructions are for building the library locally out of any container. You first need to clone the WPE WebKit project, version 2.52 or above:

mkdir wpe-webkit
cd wpe-webkit
git clone --depth 1 -b wpewebkit-2.52.5 https://github.com/WebKit/WebKit.git

Then install clang and all the needed development dependencies by calling: ./WebKit/Tools/wpe/install-dependencies.

Download the build-wpe.sh and set_dev_env.sh scripts and copy them to your working folder containing the WebKit source code. Then execute:

chmod 755 ./build-wpe.sh ./set_dev_env.sh
./build-wpe.sh --configure
./set_dev_env.sh

It will build and install WPE WebKit into ./dist-wpe. The set_dev_env.sh script will set the environment variables to use the files in ./dist-wpe for pkg-config and for the runtime.

Build the example project #

Clone the example project:

git clone https://github.com/neodesys/blog-the-wpe-platform-api.git wpe-glfw-platform

Install the GLFW development dependency (package libglfw3-dev on Ubuntu/Debian).

Configure and build it:

cd wpe-glfw-platform
meson setup build
ninja -C build

You can now run it by calling ./build/wpe-glfw [url].

N.B. In the version 2.52.5 of WPE WebKit, the Skia multithreaded hardware-accelerated compositor is not fully stable with some specific GPUs. In particular, with NVidia graphic cards, it may crash when the allocated surfaces are resized to resolutions bigger than HD. If this is your case, you can disable the Skia hardware-accelerated compositor by setting the environment variable WEBKIT_SKIA_ENABLE_CPU_RENDERING=1. In some cases, you don’t need to disable the whole hardware-acceleration for the compositor. Sometimes just configuring the hardware compositor to use only one thread is enough. You can do that by setting the environment variable WEBKIT_SKIA_PAINTING_THREADS=1.

From a WPE Backend to a WPE Platform #

A WPE Backend sits at the crossroads between the WPEWebProcess, in charge of running the ThreadedCompositor, and the application process, which presents the resulting frames. Both processes must load the same backend shared library, and that library has to implement an IPC layer to move each handle from one process to the other.

graph LR;
    subgraph SA[<b>WPEWebProcess</b>]
        A(ThreadedCompositor)
    end
    subgraph SB[<b>Application Process</b>]
        C(User Application)
    end
    A -->|draw| B([WPE Backend shared library<br/>Loaded by both processes]) --> C

The WPE Platform API removes this shared library and all the IPC burden. WPE WebKit still spawns a WPEWebProcess to run the ThreadedCompositor but the transfer of the rendered frames from the WPEWebProcess to the application process is now handled internally by WPE WebKit itself. As an application developer, you no longer need to implement any IPC: you only receive a ready-to-use WPEBuffer object, backed by a DMA buffer or by shared memory, directly into your application process.

graph LR;
    subgraph SA[<b>WPEWebProcess</b>]
        A(ThreadedCompositor)
    end
    subgraph SB[<b>Application Process</b>]
        B[WPEDisplay<br/>WPEToplevel<br/>WPEView]
        C(User Application)
    end
    A -->|"WPEBuffer (handled internally)"| B --> C

What used to be a shared library exposing five libwpe interfaces is now just three plain GObject classes (WPEDisplay, WPEToplevel and WPEView) that you need to subclass and link directly into your application binary.

The three pillars of a WPE Platform #

  • WPEDisplay is the entry point. It owns the connection to the native graphical system (here, an EGL display bootstrapped through GLFW) and acts as a factory for the toplevel window and the web views.
  • WPEToplevel is roughly the equivalent of a native window. It owns the actual GLFW window, translates native window events (keyboard, mouse, scroll, focus and resize) into WPEEvent instances, and exposes the usual window operations to the WPE Platform API (resizing, setting the window title and switching to fullscreen).
  • WPEView is where the web page is actually drawn. It receives a new WPEBuffer each time the ThreadedCompositor has produced a frame and is responsible for presenting it on screen.
graph TB;
    subgraph SA[<b>WPEGLFWDisplay</b>]
        A["connect(): create the EGL display through GLFW"]
    end
    A -->|create_toplevel| B[<b>WPEGLFWToplevel</b><br/>owns the GLFW window]
    A -->|create_view| C[<b>WPEGLFWView</b><br/>renders one WPEBuffer per frame]
    B -.attach view and broadcast window events.-> C

A single WPEDisplay can create several toplevel windows, and a toplevel window can have several views attached to it (think of tabs sharing one native window). In our example application we only create a single toplevel window with a single view.

Implementing WPEGLFWDisplay: bootstrapping EGL through GLFW #

The WPEGLFWDisplay::connect(…) override is called only once, when wpe_display_connect(...) is invoked from main(). It initializes GLFW, requests an EGL/GLES2 context for every window that will be created afterwards, and creates a tiny hidden bootstrap window used only to force GLFW to set up its EGL connection:

// Request EGL + GLES 2 context for all subsequent window creations
glfwWindowHint(GLFW_CONTEXT_CREATION_API, GLFW_EGL_CONTEXT_API);
glfwWindowHint(GLFW_CLIENT_API, GLFW_OPENGL_ES_API);
glfwWindowHint(GLFW_CONTEXT_VERSION_MAJOR, 2);
glfwWindowHint(GLFW_CONTEXT_VERSION_MINOR, 0);
glfwWindowHint(GLFW_VISIBLE, GLFW_FALSE);

// Create a tiny hidden window so GLFW can initialize its EGL connection
self->init_window = glfwCreateWindow(1, 1, "", NULL, NULL);
...
self->egl_display = glfwGetEGLDisplay();

If we want to be able to transfer the frames with zero-copy, keeping them in the GPU memory, we will need a valid WPEDRMDevice. It is also initialized during the display connection, using the EGL_EXT_device_query extension to fetch the device associated with the current EGL display:

EGLDeviceEXT egl_device = EGL_NO_DEVICE_EXT;
if (eglQueryDisplayAttribEXT(self->egl_display, EGL_DEVICE_EXT, (EGLAttrib*)&egl_device) &&
    egl_device != EGL_NO_DEVICE_EXT)
{
    const char* drm_device = eglQueryDeviceStringEXT(egl_device, EGL_DRM_DEVICE_FILE_EXT);
    const char* drm_render_node = eglQueryDeviceStringEXT(egl_device, EGL_DRM_RENDER_NODE_FILE_EXT);
    self->drm_device = wpe_drm_device_new(drm_device, drm_render_node);
}

If WPEGLFWDisplay::get_drm_device(…) returns NULL, the produced frames will be transferred to the application using shared memory, which implies copying the frames content back and forth between the GPU and the CPU.

Implementing WPEGLFWToplevel: the window and its events #

WPEGLFWToplevel is where the actual GLFW window is created. It is important to manage this creation in the WPEGLFWToplevel::constructed(…) override rather than in the init() function because WPEToplevel::constructed(...) is resetting the toplevel registered dimensions.

static void wpe_glfw_toplevel_constructed(GObject* object)
{
    // It is important to initialize the window in the `constructed` virtual
    // method and not in the `init` method because the parent class
    // (WPETopLevel) resets the toplevel window size in this call.
    G_OBJECT_CLASS(wpe_glfw_toplevel_parent_class)->constructed(object);

    WPEGLFWToplevel* self = WPE_GLFW_TOPLEVEL(object);

    // All window hints have already been configured by the display
    self->window = glfwCreateWindow(DEFAULT_WIDTH, DEFAULT_HEIGHT, "", NULL, NULL);
    ...

    // Communicate the initial window size to the WPETopLevel, so when
    // the WPEView is attached, it can be resized immediately to the correct
    // dimensions.
    wpe_toplevel_resized(WPE_TOPLEVEL(self), DEFAULT_WIDTH, DEFAULT_HEIGHT);
}

Once the window exists, each GLFW events callback (for the keyboard, mouse, window resizing, etc…) is translating the window events into the corresponding WPEEvent and broadcasts those events to every view currently attached to this toplevel window.

The collection and dispatch of the GLFW events themselves are ensured by calling glfwPollEvents(). As this function is managing the events for all the GLFW windows, it is configured in the WPEGLFWDisplay as a Glib source. This way all GLFW window events are collected, dispatched, translated into WPEEvent and broadcasted to each view from the main application thread running the Glib main loop.

The rest of the class is a set of straightforward virtual method overrides mapping WPE window operations onto GLFW calls to allow the WPE Platform API to set the window title, change the toplevel window size or switch to fullscreen.

Implementing WPEGLFWView: turning a WPEBuffer into pixels #

WPEGLFWView is the class doing the actual OpenGL ES drawing. The main interesting override is render_buffer(…), called by WPE WebKit every time a new frame is ready to be presented. The rest of the code is basically some boilerplate used to render a texture on a plane.

The WPEBuffer provided by WPE WebKit can be a wrapper for a DMA buffer allowing to draw the frame without copying it to the main memory, or it can be the wrapper of a classical block of shared memory if the DRM device was not available in WPEGLFWDisplay::get_drm_device(…).

So, when drawing, we first try to get the EGLImage wrapped by the WPEBuffer and, if not available, we fall back to the shared memory:

// Try to import the WPEView content through an EGLImage to allow a
// zero-copy transfer. This is only going to work if the EGLDisplay is
// associated with a valid DRM device returned by
// WPEDisplay::get_drm_device().
gpointer egl_image = wpe_buffer_import_to_egl_image(buffer, NULL);
if (egl_image)
{
    self->glEGLImageTargetTexture2DOES(GL_TEXTURE_2D, (GLeglImageOES)egl_image);
    glUniform1f(self->uniform_swap_rb, 0.f);
}
else
{
    // Else, fall back to the SHM buffer. In this case the WPEView content
    // is copied through the CPU into shared memory.
    GBytes* pixels = wpe_buffer_import_to_pixels(buffer, &shm_err);
    ...
    gconstpointer data = g_bytes_get_data(pixels, NULL);
    ...
    glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, (GLsizei)buf_w, (GLsizei)buf_h, 0, GL_RGBA, GL_UNSIGNED_BYTE, data);
    glUniform1f(self->uniform_swap_rb, 1.f);
}

All the resources (the EGLImage or the GBytes pixels array) are held by the WPE WebKit ThreadedCompositor and so, once the drawing is finished, we must signal that the buffer has been rendered and can be recycled:

wpe_view_buffer_rendered(view, buffer);
wpe_view_buffer_released(view, buffer);

The first call (wpe_view_buffer_rendered(...)) triggers the rendering of the next frame, while the second call (wpe_view_buffer_released(...)) informs that the internal EGLImage or GBytes pixels array can be re-used for a future frame content, avoiding the allocation of new buffers for each new frame.

N.B. The drawing loop in the example is very simplified for the purpose of this post because we are not repainting the window content when it is damaged for example. We are only doing the drawing sequentially at one place when receiving a new frame from the web view. In a real application, we may want to keep the current WPEBuffer for intermediate repainting, until receiving the next frame. In this case, we would call wpe_view_buffer_rendered(...) for buffer A once it has been drawn but we would call wpe_view_buffer_released(...) only after receiving buffer B with the next frame content. So, buffer A may be used more than once to repaint the window content like shown in the following sequence diagram.

sequenceDiagram
    participant A as WPEWebProcess
    participant B as Application Process
    activate A
    A ->> A: Render frame 1 in buffer A
    A ->> B: WPEBuffer A
    deactivate A
    activate B
    B ->> B: Draw frame 1 from buffer A
    B ->> A: wpe_view_buffer_rendered(A)
    deactivate B
    activate A
    A ->> A: Render frame 2 in buffer B
    activate B
    B ->> B: Repaint buffer A
    deactivate B
    A ->> B: WPEBuffer B
    deactivate A
    activate B
    B ->> A: wpe_view_buffer_released(A)
    B ->> B: Draw frame 2 from buffer B
    B ->> A: wpe_view_buffer_rendered(B)
    deactivate B
    activate A
    A ->> A: Render frame 3 in buffer A
    A ->> B: WPEBuffer A
    deactivate A
    activate B
    B ->> A: wpe_view_buffer_released(B)
    B ->> B: Draw frame 3 from buffer A
    deactivate B

The rendering in the current example is not optimized either because the ThreadedCompositor must wait for the complete presentation of the current frame with glfwSwapBuffers(...) blocking until the drawing is finished. We can perfectly imagine a multithreaded view where the render_buffer(...) override just posts the current WPEBuffer to a separate drawing thread. This way the WPE WebKit drawing of the next frame and the presentation of the current frame on screen would run in parallel instead of waiting for each other.