Rego’s Everyday Life

A blog about my work at Igalia.

aamtest: Testing accessibility in WPT

Testing in the web platform is done in many different ways, but a very important one is through the Web Platform Tests (WPT) repository, which contains around 130,000 tests (note that almost 54,000 are from test262, testing JavaScript), which make a total of about 2.25 million subtests. This shared test suite covers most of the web specifications, or at least that’s the aim. All this is relevant because the different web engines run these tests constantly to ensure compatibility and avoid regressions.

WPT is great, but until very recently it couldn’t properly test platform accessibility (a11y) APIs, which means it was leaving out a very important part of the web, and not finding interop issues, catching regressions and identifying implementation gaps in that area. Browser engines have usually been testing a11y through internal tests, which is nice, but not enough in many cases. WPT can now test platform a11y APIs, and, as we’ll see later in this blog post, these new tests have already found bugs in web engines, and even in the specs.

Testing a11y on the web platform #

Igalia has been exploring ways to solve this problem for quite a while. A first attempt was a project called Acacia, which was sponsored by the Sovereign Tech Fund. Given all platform a11y APIs have Python bindings, we were working on a Python library that would allow us to inspect them directly. After many conversations with different parties, that project was put on hold, and we moved to find a solution directly inside WPT based on all we’ve learnt in that project.

That solution is now a reality, a new type of WPT tests called aamtest, which aim to test the AAM (Accessibility API Mappings) for the different platform a11y APIs. My colleague Valerie Young introduced this new type in March (RFC 204: WPT testing for AAMs) and started to write new tests based on that directly in WPT. As of today, there are 190 aamtest tests in WPT, covering 4 specs: ARIA, Core-AAM, HTML-AAM and MathML-AAM. We’re glad to mention that the recent work on this project has been funded by the NLnet Foundation.

Let’s use a diagram by my other coworker Alice Boxhall to try to clarify a little bit more about what these new tests do. If you want a full description, the diagram is fully explained in the previously linked RFC.

Diagram showing the interaction between a browser and accessibility APIs by Alice Boxhall

Diagram showing the interaction between a browser and accessibility APIs by Alice Boxhall

In this diagram you can see the different components related to the a11y support in browsers, and how they interact with each other. The green dashed line is the new proposed way to directly test the platform a11y APIs from WPT.

It’s worth mentioning that back in June, Valerie gave a great talk at the Web Engines Hackfest covering all the details related to this work; don’t miss it if you want to dig deeper.

What does an aamtest look like? #

These tests are written in Python and have different sections covering each platform a11y API. They’re all inside an aamtests folder, which is how WPT identifies this kind of test and knows it has to enable a11y in the browser for them.

Let’s take a look at one example (core-aam/aamtests/role/blockquote.py) and explain the different parts.

# Testing: https://w3c.github.io/core-aam/#role-map-blockquote

TEST_HTML = "<div role='blockquote' id='test'>content</div>"

def test_atspi(atspi, session, inline):
    session.url = inline(TEST_HTML)

    # Spec:
    # Role: ROLE_BLOCK_QUOTE

    node = atspi.find_node("test", session.url)
    assert atspi.Accessible.get_role(node) == atspi.Role.BLOCK_QUOTE

def test_axapi(axapi, session, inline):
    session.url = inline(TEST_HTML)

    # Spec:
    # AXRole: AXGroup
    # AXSubrole: <nil>

    node = axapi.find_node("test", session.url)
    role = axapi.AXUIElementCopyAttributeValue(node, "AXRole", None)[1]
    assert role == "AXGroup"
    subrole = axapi.AXUIElementCopyAttributeValue(node, "AXSubrole", None)[1]
    assert subrole == None

def test_ia2(ia2, session, inline):
    session.url = inline(TEST_HTML)

    # Spec:
    # Role: ROLE_SYSTEM_GROUPING
    # Role: IA2_ROLE_BLOCK_QUOTE

    node = ia2.find_node("test", session.url)
    assert ia2.get_role(node) == "IA2_ROLE_BLOCK_QUOTE"
    assert ia2.get_msaa_role(node) == "ROLE_SYSTEM_GROUPING"


def test_uia(uia, session, inline):
    session.url = inline(TEST_HTML)

    # Spec:
    # Control Type: Group
    # Localized Control Type: blockquote

    node = uia.find_node("test", session.url)
    assert node.CurrentControlType == uia.ControlType.Group
    assert node.CurrentLocalizedControlType == "blockquote"

As you can see, these are quite simple; in this case it’s a test to check that role="blockquote" is properly exposed.

The first thing in the test is some HTML code that we’ll use to test that the role is properly exposed; in this case it’s quite simple, consisting of a DIV element with role="blockquote":

TEST_HTML = "<div role='blockquote' id='test'>content</div>"

Next you can see 4 different functions prefixed with test_, one per platform a11y API:

All of them start with the same line, which inlines the HTML to be able to test it (this uses WebDriver). Then they have a comment with the mapping information from the spec. And after that, they use the Python library to check that the role is properly exposed to the platform a11y APIs.

Let’s take a closer look at the AT-SPI one:

    node = atspi.find_node("test", session.url)
    assert atspi.Accessible.get_role(node) == atspi.Role.BLOCK_QUOTE

First it finds the a11y node corresponding to the DOM element with ID test, and then uses the a11y library to get the role of that element. Finally it verifies that it’s the expected one.

You can run these tests locally with the WPT tooling, but you can also check the results for the different platforms on the wpt.fyi website. For example, you can see the results for this particular test.

Screenshot of the results at wpt.fyi for core-aam/aamtests/role/blockquote.py, showing it passing for AT-SPI on Chrome and Firefox, and IA2 and UIA on Edge, while not running on Safari

Screenshot of results for core-aam/aamtests/role/blockquote.py at wpt.fyi

We have also tweaked wpt.fyi so the tests show as missing for the platforms they don’t belong to. You can see that AT-SPI is only tested on Linux (Chrome and Firefox), while IA2 and UIA are tested on Windows (Edge). These tests cannot run on WPT’s macOS (Safari) CI infrastructure yet, though the WebKit team is working on adding support for them; for now, you can run them locally if you have a Mac.

All this is great, allowing us to directly test the platform a11y APIs, and verify that things are properly exposed to assistive technology users. And being in WPT, it also helps us to check that implementations are compatible with each other, and that there are no regressions that go unnoticed.

Adding events support #

In the last few months, I’ve had the chance to help with this project. And I’ve worked on adding one missing piece, the capability of testing a11y events. When something changes on a page (like a form entry that is marked as disabled), platform a11y APIs emit events to notify assistive technologies, and thus the users, about it.

In order to be able to test those events, we have added a new method called expect_event() to the testing infrastructure, which adds an event listener for a given event in a node, executes an action to trigger it, waits for the event and returns it. That way we can make the test suite more comprehensive by also covering events.

Let’s see an example of how it works in the wild (core-aam/aamtests/event/aria_checked.py).

# Testing: https://w3c.github.io/aria/core-aam/#event-aria-checked

TEST_HTML = "<div id='target' role='checkbox' aria-checked='false'>"

def test_atspi(atspi, session, inline):
    session.url = inline(TEST_HTML)

    # Spec:
    # object:state-changed:checked

    node = atspi.find_node("target", session.url)
    assert "STATE_CHECKED" not in atspi.get_state_list_helper(node)

    def toggle_aria_checked():
        session.execute_script(
            "target.ariaChecked = target.ariaChecked === 'true' ? 'false' : 'true'",
        )

    event = atspi.expect_event(
        "object:state-changed:checked", dom_id="target",
        action=toggle_aria_checked,
    )

    assert event.detail1 == 1
    assert "STATE_CHECKED" in atspi.get_state_list_helper(node)

    event = atspi.expect_event(
        "object:state-changed:checked", dom_id="target",
        action=toggle_aria_checked,
    )

    assert event.detail1 == 0
    assert "STATE_CHECKED" not in atspi.get_state_list_helper(node)

In this case the test checks the event triggered when the aria-checked attribute is modified.

Again it has quite simple HTML, and a similar structure to the previous one, though there are a couple of new things.

We define a function toggle_aria_checked() that modifies the value of the aria-checked attribute. That function is passed as an action to expect_event() to which we also pass the event name object:state-changed:checked and the ID of the node in which we’re going to wait for the event. When the event is triggered, we get it and we can check different things on it to verify that it’s working correctly. Then we run it twice to check that the event is also triggered when you do the opposite change to the aria-checked attribute.

How useful are these tests? #

One question you might have is why we are doing all this, knowing web engines already have internal tests covering this (as mentioned before). That’s true, but just with the first tests we’ve added, we have discovered interop issues and missing features in different places. Let’s review a few examples.

While working on events support, we started with a simple example changing the value of aria-disabled and catching the expected events, which for AT-SPI is expected to trigger 2 events: object:state-changed:enabled and object:state-changed:sensitive. We realized that Firefox wasn’t doing that, as it was only triggering one of them (enabled). This was a small issue and easy to fix, so we went ahead and fixed the issue in Firefox; the patch landed even before the support for testing events in aamtest was merged in WPT.

Another example, while testing an ARIA “must” statement about aria-errormessage when aria-invalid="false", we found again that Firefox wasn’t fulfilling the spec. In this case, this was a known old bug that we also fixed upstream.

And it’s not only Firefox; for example, when we added tests for Core-AAM attributes, we discovered that Chromium wasn’t properly exposing aria-sort="none", which again we fixed upstream.

Last but not least, while working on these tests we have also found spec issues, not only implementation ones. For example, adding a test for the <abbr> element, we found a wrong statement in the HTML-AAM spec, which has already been fixed.

There are more examples like these; these are just a few. In the early days of these new tests, they have already proven to be very valuable for identifying issues and improve the a11y support in different engines. It’s really nice to see how useful they are while things are coming along.

Future work #

This doesn’t finish here; this work can be considered the foundation, which is now in place and people can build things on top of it.

Now that WPT has a new type of tests to cover a11y platform APIs, we expect that aamtest tests start to be used more and more to cover a11y support for new and existing web platform features. Our recent work has been focused on the Linux platform, but there is more work to do covering other platforms, completing tests for platforms that aren’t being tested yet. If you want to do that, you can check the documentation.

Not only that, with all these tests, we’re finding interoperability issues here and there, so we hope people start to fix them where convenient. Apart from that, there is some flakiness when running these tests in the WPT CI for both Chromium and Firefox, as they’re the first ones testing against the platform a11y APIs. This needs more investigation to come up with a solution.

All in all, we’re very happy to have reached this point and to see these tests already having a positive impact on the web platform. We’d like to thank everyone who has been helping make this happen, whether through reviews, feedback, participation in discussions, and so on.