Android has all kinds of nice development tools, but sometimes you just want to run an apk and don’t need all the surrounding tooling. In my case, I have already have my Chromium setup, which can produce binaries for several platforms including Android.
I usually test on a physical device, a smartphone, but I would like to try a device with a tablet form factor and I don’t have one at hand.
Chromium provides different user experiences for smartphones and tablets.
I’ve set up the most stripped-down environment possible to run an Android emulator using the same tools provided by the platform. Notice these are generic instructions, not tied to Chromium tooling, despite I’m doing this mainly to run Chromium.
The first step is to download the latest version of the command line tools, instead of Android Studio, from the Android developer website. We will extract the tools to the location ~/Android/Sdk, in the path where they like to find themselves:
Now we have the tools installed in ~/Android/Sdk/cmdline-tools/latest, we need to add their binaries to the path. We will also add other paths for tools we are about to install, with the command:
Run the command above for a one-time use, or add it to ~/.bashrc or your preferred shell configuration for future use.
You will also need to setup another envvar:
export ANDROID_HOME=~/Android/Sdk
Again, run it once for a one-time use or add it to your shell configuration in ~/.bashrc or equivalent.
Now we use the tool sdkmanager, which came with Android’s command-line tools, to install the rest of the software we need. All of this will be installed inside your $ANDROID_HOME, so in ~/Android/Sdk.
In the command above I have installed the emulator, platform tools (adb and friends), and system libraries and a system image for Android 10 (API level 29). You may check what other things are available with sdkmanager --list, you will find multiple variants of the Android platform and system images.
Finally, we have to setup a virtual machine, called AVD (“Android Virtual Device”) in Android jargon. We do that with:
Here I have created a virtual device called “Android_API_29_Google” with the system image I had downloaded, and the form factor and screen size of a Pixel C, which is a tablet. You may get a list of all the devices that can be simulated with avdmanager list device.
Now we can already start an emulator running the AVD with the name we have just chosen, with:
emulator -avd Android_API_29_Google
The files and configuration for this AVD are stored in ~/.android/avd/Android_API_29_Google.avd, or a sibling directory if you used a different name. Configuration is in the config.ini file, you can set here many options you would normally configure from Android Studio, and even more. I recommend to change at least this value, for your convenience:
hw.keyboard = yes
Otherwise you will find yourself typing URLs with a mouse on a virtual keyboard… Not fun.
Finally, your software will be the up-to-date after a fresh install but, when new versions are released, you will be able to install them with:
sdkmanager --update
Make sure to run this every now and then!
At this point, your setup should be ready, and all the tools you need are in the PATH. Feel free to reuse the commands above to create any number of virtual devices with different Android system images, different form factors… Happy hacking!
Back in March of 2021, as part of my maintenance work on Chrome/Chromium accessibility at Igalia, I inadvertently started a long, deep dive into the accessible name calculation code, that had me intermittently busy since then.
This is the third and last post in the series. We started with an introduction to the problem, and followed by describing how we fixed things in Chromium. In this post, we will describe some problems we detected in the accessible name computation spec, and the process to address them.
This post extends what was recently presented in a lightning talk at BlinkOn 17, in November 2022:
Hidden content and aria-labelledby
Authors may know they can use aria-labelledby to name a node after another, hidden node. The attribute aria-describedby works analogously for descriptions, but in this post we will refer to names for simplicity.
This is a simple example, where the produced name for the input is “foo”:
if computing a name, and the current node has an aria-labelledby attribute that contains at least one valid IDREF, and the current node is not already part of an aria-labelledby traversal, process its IDREFs in the order they occur:
or, if computing a description, and the current node has an aria-describedby attribute that contains at least one valid IDREF, and the current node is not already part of an aria-describedby traversal, process its IDREFs in the order they occur:
i. Set the accumulated text to the empty string.
ii. For each IDREF:
a. Set the current node to the node referenced by the IDREF.
b. Compute the text alternative of the current node beginning with step 2. Set the result to that text alternative.
c. Append the result, with a space, to the accumulated text.
iii. Return the accumulated text.
When processing each IDREF per 2B.ii.b, it will stumble upon this condition in 2A:
If the current node is hidden and is not directly referenced by aria-labelledby or aria-describedby, nor directly referenced by a native host language text alternative element (e.g. label in HTML) or attribute, return the empty string.
So, a hidden node referenced by aria-labelledby or aria-describedby will not return the empty string. Instead, it will go on with the subsequent steps in the procedure.
Unfortunately, the spec did not provide clear answers to some corner cases: what happens if there are elements that are explicitly hidden inside the first hidden node? What about nodes that aren’t directly referenced? How deep are we expected to go inside a hidden tree to calculate its name? And what about aria-hidden nodes?
Given the markup above, WebKit generated the name “d”, Firefox said “abcd” and Chrome said “bcd”. In absence of clear answers, user agent behaviors differ.
There was an ongoing discussion in Github regarding this topic, where I shared my findings and concerns. Some conclusions that came out from it:
It appears that the original intention of the spec was allowing the naming of nodes from hidden subtrees, if done explicitly, and to include only those children that were not explicitly hidden.
It’s really hard to implement a check for explicitly hidden nodes inside a hidden subtree. For optimization purposes, that information is simplified away in early stages of stylesheet processing, and it wouldn’t be practical to recover or keep it.
Given the current state of affairs and how long the current behavior has been in place, changing it radically would cause confusion among authors, backwards compatibility problems, etc.
The agreement was to change the spec to match the actual behavior in browsers, and clarify the points where implementations differed. But, before that…
A dead end: the innerText proposal
An approach we discussed which looked promising was to use the innerText property for the specific case of producing a name from a hidden subtree. The spec would say something like: “if node is hidden and it’s the target of an aria-labelledby relation, return the innerText property”.
It’s easy to explain, and saves user agents the trouble of implementing the traversal. It went as far as to have a proposed wording and I coded a prototype for Chromium.
Unfortunately, innerText has a peculiar behavior with regard to hidden content: it will return empty if the node was hidden with the visibility:hidden rule.
1. If this is not being rendered or if the user agent is a non-CSS user agent, then return this’s descendant text content.
Where:
An element is being rendered if it has any associated CSS layout boxes, SVG layout boxes, or some equivalent in other styling languages.
An element with visibility:hidden has an associated layout box, hence it’s considered “rendered” and follows the steps 2 and beyond:
Let results be a new empty list.
For each child node node of this:
i. Let current be the list resulting in running the rendered text collection steps with node. Each item in results will either be a string or a positive integer (a required line break count).
Where “rendered text collection steps” says:
Let items be the result of running the rendered text collection steps with each child node of node in tree order, and then concatenating the results to a single list.
If node’s computed value of ‘visibility’ is not ‘visible’, then return items.
The value of visibility is hidden, hence it returns items, which is empty because we had just started.
This behavior makes using innerText too big of a change to be acceptable for backwards compatibility reasons.
Clarifying the spec: behavior on naming after hidden subtrees
In the face of the challenges and preconditions described above, we agreed that the spec would document what most browsers are already doing in this case, which means exposing the entire, hidden subtree, down to the bottom. There was one detail that diverged: what about nodes with aria-hidden="true"? Firefox included them in the name, but Chrome and WebKit didn’t.
Letting authors use aria-hidden to really omit a node in a hidden subtree appeared to be a nice feature to have, so we first tried to codify that behavior into words. Unfortunately, the result became too convoluted: it added new steps, it created differences in what “hidden” means… We decided not to submit this proposal, but you can still take a look at it if you’re curious.
Instead, the proposal we submitted does not add or change many words in the spec text. The new text is:
“If the current node is hidden and is not part of an aria-labelledby or aria-describedby traversal, where the node directly referenced by that relation was hidden […], return the empty string.”
Instead, it adds a lot of clarifications and examples. Although it includes guidelines on what’s considered hidden, it does not make differences in the specific rules used to hide; as a result, aria-hidden nodes are expected to be included in the name when naming after a hidden subtree.
Updating Chromium to match the spec: aria-hidden in hidden subtrees
We finally settled on including aria-hidden nodes when calculating a name or a description after a hidden subtree, via the aria-labelledby and aria-describedby relations. Firefox already behaved like this but Chromium did not. It appeared to be an easy change, like removing only one condition somewhere in the code… The reality is never like that, though.
The change landed successfully, but it’s much more than removing a condition: we found out the removal had side effects in other parts of the code. Additionally, we had some corrections in Chromium to prevent authors to use aria-hidden on focusable elements, but they weren’t explicit in the code; our change codifies them and adds tests. All this was addressed in the patch and, in general, I think we left things in a better shape than they were.
A recap
In the process to fix a couple of bugs about accessible name computation in Chrome, we detected undocumented cases in the spec and TODOs in the implementation. To address all of it, we ended up rewriting the traversal code, discussing and updating the spec and adding even more tests to document corner cases.
Many people were involved in the process, not only Googlers but also people from other browser vendors, independent agents, and members of the ARIA working group. I’d like to thank them all for their hard work building and maintaining accessibility on the Web.
But there is still a lot to do! There are some known bugs, and most likely some unknown ones. Some discussions around the spec still remain open and could require changes in browser behavior. Contributions are welcome!
Back in March last year, as part of my maintenance work on Chrome/Chromium accessibility at Igalia, I inadvertently started a long, deep dive into the accessible name calculation code, that had me intermittently busy since then. Part of this work was already presented in a lightning talk at BlinkOn 15, in November 2021:
This is the second post in a series bringing a conclusion to the work presented in that talk. You may read the previous one, with an introduction to accessible name computation and a description of the problems we wanted to address, here.
Why traverse the DOM?
In our previous post, we found out we used a DOM tree traversal in AXNodeObject::TextFromDescendants to calculate the name for a node based on its descendants. There are some reasons this is not a good idea:
There was duplicate code related to selection of relevant nodes (cause of the whitespace bug). For example, the accessibility tree creation already selects relevant nodes, and simplifies whitespace nodes to one single unit whenever relevant.
The DOM tree does not contain certain relevant information we need for accessible names (cause of the ::before/after bug). Relevant pseudo-elements are already included in the accessibility tree.
In general, it would be more efficient to reuse the cached accessibility tree.
There must have been a good reason to traverse the DOM again instead of the cached accessibility tree, and it was easy enough to switch our code to do the latter, and find out what was breaking. Our tests were kind enough to point out multiple failures related with accessible names from hidden subtrees: it is possible, under certain conditions, to name elements after other hidden elements. Because hidden elements are generally excluded from the accessibility tree, to be able to compute a name after them we had resorted to the DOM tree.
Hidden content and aria-labelledby
Authors may know they can use aria-labelledby to name a node after another, hidden node. This is a simple example, where the produced name for the input is “foo”:
Unfortunately, the spec did not provide clear answers to some corner cases: what happens if there are elements that are explicitly hidden inside the first hidden node? What with nodes that aren’t directly referenced? How deep are we expected to go inside a hidden tree to calculate its name? And what about aria-hidden nodes?
The discussion and changes to the spec that resulted from it are worth their own blog post! For this one, let me jump directly to the conclusion: we agreed to keep browsers current and past behavior for the most part, and change the spec to reflect it.
Addressing the traversal problem in Chromium
The goal now is to change name computation code to work exclusively with the accessibility tree. It would address the root cause of the problems we had identified, and simplify our code, potentially fixing other issues. To do so, we need to include in the accessibility tree all the information required to get a name from a hidden subtree like explained previously.
We have the concept of “ignored” nodes in the Blink accessibility tree: these nodes exist in the tree, are kept up-to-date, etc. But they are not exposed to the platform accessibility. We will use ignored nodes to keep name-related items in the tree.
At first, I attempted to include in the accessibility tree those nodes whose parent was a target of an aria-labelledby or aria-describedby relation, even if they were hidden. For that purpose, I had to track the DOM for changes in these relations, and maintain the tree up-to-date. This proved to be complicated, as we had to add more code, more conditions… Exactly the opposite to our goal of simplifying things. We ended up abandoning this line of work.
We considered the best solution would be the simplest one: keep all hidden nodes in the tree, just in case we need them. The patch had an impact in memory consumption that we decided to assume, taking into account that the advantages of the change (less and cleaner code, bug fixes) were more than the disadvantages, and its neutral-to-slightly-positive impact in other performance metrics.
There were some non-hidden nodes that were also excluded from the accessibility tree: that was the case of <label> elements, to prevent ATs from double-speaking the text label and the labeled element next to it. We also had to include these labels in the Blink accessibility tree, although marked “ignored”.
Along the way, we made a number of small fixes and improvements in surrounding and related code, which hopefully made things more stable and performant.
Next in line: spec work
In this post, we have explained the big rework in accessible naming code to be able to use the accessibility tree as the only source of names. It was a long road, with a lot unexpected problems and dead-ends, but worth for the better code, improved test coverage and bugs fixed.
In the next and final post, I’ll explain the work we did in the spec, and the adjustments required in Chromium, to address imprecision when naming from hidden subtrees.
Back in March last year, as part of my maintenance work on Chrome/Chromium accessibility at Igalia, I inadvertently started a long, deep dive into the accessible name calculation code, that had me intermittently busy since then.
As a matter of fact, I started drafting this post over a year ago, and I already presented part of my work in a lightning talk at BlinkOn 15, in November 2021:
This series of posts is the promised closure from that talk, that extends and updates what was presented there.
Introduction
Accessible name and description computation is an important matter in web accessibility. It conveys how user agents (web browsers) are expected to expose the information provided by web contents in text from, in a way it’s meaningful for users of accessibility technologies (ATs). This text is meant to be spoken, or routed to a braille printer.
We know a lot of the Web is text, but it’s not always easy. Style rules can affect what is visible and hence, what is part of a name or description and what is not. There are specific properties to make content hidden for accessibility purposes (aria-hidden), and also to establish relations between elements for naming and description purposes (aria-labelledby, aria-describedby) beyond the markup hierarchy, potentially introducing loops. A variety of properties are used to provide naming, specially for non-text content, but authors may apply them anywhere (aria-label, alt, title) and we need to decide their priorities. There are a lot of factors in play.
There is a document maintained by the ARIA Working Group at W3C, specifying how to do this: https://w3c.github.io/accname/. The spec text is complex, as it has to lay out the different ways to name content, settle their priorities when they interact and define how other properties affect them.
Accessible name/description in Chromium
You may have heard the browser keeps multiple tree structures in memory to be able to display your web sites. There is the DOM tree, which corresponds with the nodes as specified in the HTML document after it’s been parsed, any wrong markup corrected, etc. It’s a live tree, meaning that it’s not static; JavaScript code and user interaction can alter it.
There is a layout tree, which I won’t dare explaining because it’s out of my experience field, but we can say it’s related with how the DOM is displayed: what is hidden and what is visible, and where it’s placed.
And then we have the accessibility tree: it’s a structure that keeps the hierarchy and information needed for ATs. It’s based on the DOM tree but they don’t have the same information: nodes in those trees have different attributes but, more importantly, there is not a 1:1 relation between DOM and accessibility nodes. Not every DOM node needs to be exposed in the accessibility tree and, because of aria-owns relations, tree hierarchies can be different too. It’s also a live tree, updated by JS code and user interaction.
In Chromium implementation, we must also talk about several accessibility trees. There is one tree per site, managed by Blink code in the renderer process, and represented in an internal and platform-agnostic way for the most part. But the browser process needs to compose the tree for the active tab contents with the browser UI and expose it through the means provided by the operating system. More importantly, these accessibility objects are different from the ones used in Blink, because they are defined by platform-specific abstractions: NSAccessibility, ATK/AT-SPI, MSAA/UIA/IAccessible2…
When working on accessible tree computation, most of the code I deal with is related with the Blink accessibility tree, save for bugs in the translation to platform abstractions. It is, still, a complex piece of code, more than I initially imagined. The logic is spread across twelve functions belonging to different classes, and they call each other creating cycles. It traversed the accessibility tree, but also the DOM tree in certain occasions.
The problems
Last year, my colleague Joanmarie Diggs landed a set of tests in Chromium, adapted from the test cases used by the Working Group as exit criteria for Candidate Recommendations. There were almost 300 tests, most of them passing just fine, but a few exposed bugs that I thought could be a good introduction to this part of the accessibility code.
The first issue I tried to fix was related to whitespace in names, as it looked simple enough to be a good first contact with the code. The issue consisted on missing whitespace in the accessible names: while empty or whitespace nodes translated to spaces in rendered nodes, the same didn’t happen in the accessible names and descriptions. Consider this example:
The label is rendered as “123 45” but Chromium produced the accessible name “12345”. It must be noticed that the spec is somewhat indefinite with regard of the whitespace, which is already reported, but we considered Chromium’s behavior was wrong for two reasons: it didn’t match the rendered text, and it was different from the other implementations. I reported the issue at the Chromium bug tracker.
Another, bigger problem I set to fix was that the CSS pseudo-elements ::before and ::after weren’t added to accessible names. Consider the markup:
The rendered text is “fancy fruit”, but Chromium was producing the accessible name “fancy”. In addition to not matching the rendered text, the spec explicitly states those pseudo-elements must contribute to the accessible name. I reported this issue, too.
I located the source of both problems at AXNodeObject::TextFromDescendants: this function triggers the calculation of the name for a node based on its descendants, and it was traversing the DOM tree to obtain them. The DOM tree does not contain the aforementioned pseudo-elements and, for that reason, they were not being added to the name. Additionally, we were explicitly excluding whitespace nodes in this traversal, with the intention of preventing big chunks of whitespace being added to names, like new lines and tabs to indent the markup.
A question hit me at this point: why were we using the DOM for this in the first place? That meant doing an extra traversal for every name calculated, and duplicating code that selected which nodes are important. It seems the reasons for this traversal were lost in time, according to code comments:
// TODO: Why isn't this just using cached children?
While I could land (and I did) temporary fixes for these individual bugs, it looked like we could fix “the traversal problem” instead, addressing the root cause of the problems while streamlining our code and, potentially, fixing other issues. I will address this problem, the reasons to traverse the DOM in first place and the interesting detours this work lead me, in a future post. Stay tuned, and happy hacking!
It’s been a bit more than a year since I started working on Chromium accessibility. Although I’ve worked on several kinds of issues, a lot of them had to do with UI dialogs of some kind. Browser interfaces present many kinds of dialogs and these problems affected all the desktop platforms. This post is a recap of all the kinds of things that have been fixed related with different uses of dialogs in Chromium!
