GNOME Shell accessibility status

These days some people have started to ask about the current GNOME Shell accessibility status, probably a collateral effect of the Boston Summit, as Joanmarie Diggs and Alejandro Leiva (Orca maintainers) were talking there with GNOME Shell developers, mainly about the current Universal Access UI and how fit Orca on the new ui experience. They are planning to make a summary post so stay tunned.

Anyway, I would like to share my view of the current accessibility status on GNOME Shell. Using the GNOME Shell roadmap as a guide:

  • Magnification: Joseph work has been integrated on GNOME Shell. Not as a extension or something like this, it is a real feature of GNOME Shell. So good news.
  • Design and implement keyboard navigation: Some months before GUADEC Dan Winship started to work on on keyboard navigation, and it has been really active these past months. This includes support on the Shell Toolkit itself (already on master), and then use this on the different regions of the shell, like the overview or the chrome, which provides a promising keyboard navigation. Note that in the case of the overview this is somewhat on a idle, as it is planned a overview relayout (check the overview-relayout branch).
  • Hook up accessibility to ATK: basic ATK support is right now directly provided without loading any Cally module, as Cally is now part of Clutter, and part of the stable Clutter 1.4 release. It is true that Cally can be more complete (this statement applies to any library), but the current status should be enough for GNOME Shell purposes. Of course, new bugs can be detected, and I should need to take a look to this bug, but I don’t see Cally as the main issue here. In the same way, I provided a patch proposal to load the atk-bride on GNOME Shell. The main problem with this patch is that although working, it is not clear if it is the final solution.
  • Comprehensively make controls accessibility: new elements are being defined on the Shell Toolkit, and most of them would require to export appropriate roles and actions. I have started to review this, so I have already created a patch for StLabel, and I have some other items in my TODO, like StWidget tooltips, and the new objects declared on the Javascript code. This is related with the previous keyboard navigation task, as I’m detecting most of then while testing the keyboard navigation patches. So I reported some additional bugs related to provide extra ATK information on the overview navegation (with patches) and the chrome. As there isn’t anything equivalent to gtk widget mnemonics those patches are using direct calls to ATK methods.
  • Theming: it is already reported as a bug, but AFAIK, there is not immediate plans to implement it.

So, if you apply all these patches, you get Orca speeching out most of the events from GNOME Shell. Part of the sections of GNOME Shell can be used using just the keyboard. In the cases that GNOME Shell is using the new core keyboard navigation support Orca reacts to the focus change, although the information exposed is only meaningful on part of the cases.

Apart from this, GNOME Shell includes an accessibility tray icon, with the intention of switch on/off the accessibility features, or start the Universal Access UI to configure those tools. But it is not still complete for the moment, and it seems more a mock-up. The screen reader switch doesn’t starts any screen reader, and although you can start the GNOME Shell magnifier you can’t configure it on the settings dialog (right now you need Orca to do that).

Conclusion

Most (all?) of the missing accessibility holes are detected, and a lot of them have provisional solutions. The process to manage the GNOME Shell with just Orca and the keyboard has started. In fact in my short-medium term plans was updating and finishing the current additional ATK support required on the keyboard navigation support, and create something like a branch or PPA and ask some user feedback (so new bugs). This will be postponed, due last changes on keyboard navigation and the overview-relayout.

But, most of then are provisional solutions, and it is still required to analyze them. Although it works, it is not clear if the current proposal to load the atk bridge, or start to add atk calls to the ui javascript code are the best options. Those should be also good solutions. Using Owen Taylor words “In my mind accessibility has to be held to the same high standards as everything else in GNOME”.

So as it was already stated, it would be really unlikely that GNOME Shell will ship with full accessibility support for GNOME 3.0.

BUT, as the roadmap states, accessibility is being taking into account. People are working on it, with several bugs, patches, and continuous patch review. So what now is a “rough draft”, at that moment will be a “real draft” of the required accessibility support on GNOME Shell, enough at least to start to ask users for feedback, and with a perspective of what it is missing, with the purpose of polish them for the following releases.

Cally integrated on Clutter

I will stop making Cally releases. This is because this Wednesday something important (from Cally POV) happened. Cally source code was moved to Clutter repository.

Some people could think that this is the same that happened to GAIL, that started as a isolated library, and then was moved to GTK+ repository. But there is a major difference. GAIL is still a runtime loadable module, and compiles against GTK+ as a independent library. So this was just basically a GAIL source code move.

But now Cally is not a loadable module anymore. It is part of Clutter. Clutter calls Cally directly in order to start accessibility support, so accessibility becomes one of the pieces of Clutter, instead of a external plugin providing this feature. I would like to thank Emmanuele Bassi for all the work done in this integration. In fact this integration was his idea, during the discussion of two bugs related to how to use/load Cally in gnome-shell, during the GNOME accessibility hackfest on CSUN.

I hope this change makes things easier. Now we still have to implement a proper way to load the bridge between Cally (the ATK implementation) and AT-SPI, the atk-bridge.

Bridge to Cally

BTW, I’m going to guadec, so this means that I need to put here this:

Going to GUADEC

And I will have there a talk about Cally, at first scheduled on Wednesday 28th July. See you there.

Cally going to GUADEC


I'm attending GUADEC

Somewhat late but just to comment that my talk “Cally: One Year Later” was accepted, so if nothing strange happens, I will attend this year to GUADEC. Four in a row.

A lot of work these days related to Cally. During the accessibility hackfest and talking about how apps should load cally or extend it, Emmanuele Bassi suggested that we should stop to consider Cally as a isolated module. These weeks I have been working in order to make this real, a lot of changes that involves the current Cally and Clutter itself to support it (summarized in one metabug), and AT applications like Orca required some changes when I removed the gdk/gtk dependency.

All the related patches are uploaded, and waiting to be applied, but blocked right now for reasons out of my hands. I hope this will be ready for GUADEC. So with this solved we could forgot to load a Cally module using gmodule. It still remains the atk-bridge, but I think that in this case it still has sense to be an optional module.

And to finalize this rant, this week I uploaded on maemo.gitorious.org a version of at-spi (here) . On previous weeks some people on maemo-developers list were interested in use at-spi and other a11y technologies (in this case for automatic testing purposes). The only place to find a proper stack to maemo was an old hildon automatic testing project.

So I just uploaded version of at-spi more recent that the one we can find there. It is still somewhat old, and still CORBA based, but it was tested with maemo, and have some specific maemo fixes applied, so it can be still useful.

The ideal would be try to migrate the current a11y stack, as much as possible, including the DBUS at-spi to maemo, but right now I don’t have enough time to do that.

Gnome-shell starts to talk

After spent some time improving cally, reviewing mx new focusable/focus-manager objects and several days configuring my environment (karmic upgrade, broken linkage in my jhbuild environment, etc) I started to check again how to use cally on gnome shell (first look here).

One of the entries in my TODO is start to make the module loading more that a hacking patch. A first solution proposal and a gratuitous rant here.

Other point on my TODO list is check why accerciser and orca froze gnome-shell. Well, accerciser still frozes the shell, but, fortunately it seems that orca now works (more or less) fine without doing anything special (black magic probably):

orca running on gnome-shell

I know that it would be more useful with sound, but as the gnome shell screencast recording feature doesn’t record audio, and I wasn’t able to use recordmydesktop or istanbul, finally I just recorded the sound with my N900, and I was too lazy to create a video with both. If you are curious enough, you can hear the audio of the previous video screencast here.

How I run gnome-shell

During this environment configuration time, I was also looking for a convenient way to run gnome shell. On live gnome, there are two proposed options to run gnome shell:

  • Replace your own WM (with –replace option)
  • To avoid to replace your own WM, run gnome-shell nested (with –xephyr option).

As I said, running some of the accessibility tools leads to froze the gnome-shell. Additionally in my case, running it on xephyr had a horrible performance, so both options were not really useful to me. Finally I chose a mixed option. I just launch a second X server, and launch the gnome-shell here.

So, in brief:

  • Move to other tty with Ctrl+F1
  • Launch other x server: xinit -- :1
  • Execute gnome-session
  • jhbuild shell
  • ./gnome-shell --replace

Then you can just use Ctrl+F9 and Ctrl+F7 to move between your “normal environment” and your “gnome-shell” environment. Probably someone can wonder why it is required to run gnome-session, and not execute directly gnome-shell (without the replace). Well, for any reason if I do that, the performance is also as horrible as with the –xephyr option. In the same way gnome-session load all my configuration, etc. Not a big issue anyway.

And finally Cally talks

Cally was somewhat stuck during December due Christmas and parallel projects, but January was a productive month.

A ClutterText bug was committed, and this allowed to push several cally bugs solutions and other blocked local commits, and going forward to do a real test of Cally with ORCA. The idea was being able to run the toy-cally examples with ORCA before move to bigger things. So bug 1946 was created with several comparative cally-gail test cases. I want to give thanks to Joanmarie for all her help as ORCA expert, testing and bug detection.

As I said, this bug leads to detect some extra bugs (on clutter and on cally). Finally these bugs were solved, so now you can use ORCA with the cally text examples.

During this task, I personally discovered that the manipulation of keyboard codes is something near to hell, and that currently Gdk and Clutter has a slightly different behaviour. This would lead to problems using the ATK interfaces, as AtkKeyEventStruct was defined really tied to GDK (at that moment the use case to contemplate). I tried to minimize that on Cally as far as possible.

Next steps? While I was working in this bug MX team announce the 0.5 release. This announce include the addition of MxFocusable and MxFocusManager, used to improve the keyboard focus management. I would like to check if it will be useful for keyboard navigation bugs in gnome-shell, as Owen Taylor suggested in the mailing list.

BTW, I have added a Cally page on live gnome, http://live.gnome.org/Cally

Finally home after Boston Summit

Although Boston Summit has finished the Monday, I have just arrived at home. In theory I should start to do housecleanings and other home-boring-related things, but basically, after a trip that started the 13th on the afternoon, and finished the 14th on the noon, I simply don’t feel like it.

But about Boston Summit. This was my first Boston Summit, and although the two days wasted on flights and the airports were a pain in the neck (specially american ones if you are non-american), I think that this worth. I was present to several interesting talks like the ones about Clutter 1.2 and the design and internals of Gnome-shell. In the same way I met really interesting people, and I would like to focus on Willie Walker and Joanmarie Diggs, and we have several conversations about Cally, and about the a11y support in future clutter-based applications like Gnome-shell. Joanmarie help me a lot to prioritize the current Cally bugs in order to have a good ORCA support in clutter based applications.

Although Owen Taylor said that Gnome-shell has still a lot of work to do before start seriously to think on the a11y support (as the user interaction could still change), he also says that once the Gnome-shell is there, it will require a good a11y support from the core (Clutter-Cally?), so the support could be added straighfordwardly.

As start to develop a library without a focus would be a bad idea (just start with things not really important), my current idea is try to use Gnome-shell/mutter as a target-aplication to implement the missing features on Cally, as a example of application using it. Although it would change in the future, at least a lot of work would have been done. In the same way, was good to know that Emmanuel Bassi is really open to get a better a11y support and that he includes it in the list of “things” that could require add new API on Clutter 1.2.

At thats all. I hope have written a coherent post, I barely slept on the flight.

Taking a quick look to the gnome-shell/mutter a11y

Randomly, when I have a litte time to, I’m trying to take a look about how the a11y support could be added to gnome-shell, and decide if Cally fits here.

So the first think to do is check the current status. After some stupid mistakes from my side (as trying to compile GnomeShell using the normal moduleset, instead of doing as the guide says), I was able to compile GnomeShell and get it running. Using the nested X server I got the next errors:

(mutter:24120): atk-bridge-WARNING **: AT_SPI_REGISTRY was not started at session startup.

(mutter:24120): atk-bridge-WARNING **: IOR not set.

(mutter:24120): atk-bridge-WARNING **: Could not locate registry

This means that I was not able to load the atk-bridge using the nested server, so I used the replace option. Not a big issue right now, as gnome-shell is going really slow there, but I would like to use that on real development. Any quick ideas?

Ok, let’s go with the replace option, running Accerciser. What we get? Here a screenshoot:

Current status

I suppose that this gnomeshell-taskpanel came from gnome- shell ;), but we can see that just under mutter there isn’t any child. So, how to interact with this “Activities” button just on the top-left corner of the window?

Well, probably now I should have started to look at the mutter+gnome-shell code trying to analyze all their parts, but before that, I wanted to test a quick hack. Basically, see what happens if we just load Cally instead of GAIL. This can’t be done using the GTK_MODULES env (so don’t forget to clean it), as it is required to be done after init Clutter, and as there is no general way to do that on Clutter, this is required to made by hand. And by hand I mean modify directly Mutter, with a hackish patch.

Wow, wow. This is working, it doesn’t crash, both modules are loaded. Let’s test Accerciser. What we get? All the applications frozen. Hmm, well, this isn’t exactly what we wanted, but probably has sense. Probably Accerciser uses all the fancy AtkUtil methods related to event listeners, not implemented yet.

Well, no problem at all, I can return to the old times, and use dogtail. The good news is that now I’m able to see things under mutter application, and get their objects. For example the “Activities” button just in the top left corner. And click it! As easy as the screenshot shows:


Using Cally

Ok, so I’m writing on the python terminal, and getting the objects, and I click on “Activities”. What happens? That all are moved to another view (probally called activities view or something like that). See a example:


Activities View

Of course, I lost the focus on the terminal, so I can’t just type again “activities.click()” ir order to go back to the previous view. Since the POV of the automatic testing this is not a issue. For example, this little dogtail test works without problem. It clicks the activities button, and after two seconds, it click it again, so the view goes back to the normal one, no problem. But from the “classic a11y” POV this could be a issue. Although normally I don’t use AT apps like Orca, I always think that if I’m able to control a app just with the python terminal and dogtail stuff, any AT app could do that (this is not 100% true, I know). Here I found that this wouldn’t be as easy.

Anyway, this is not only about the AT apps. For example, in order to take the screenshot, I just clicked the “Impr Pant” so the save screenshot dialog appear. But of course, on one of the working areas. These working areas minimized at that moment, so to interact to that I needed to click on the working area and so on. IMHO, this is confusing, but after all, gnome-shell is just on development, but this leads if we need a app on the top of that view…

Well, and now the next steps should be do a real work instead of just change the module to load. About that, probably one of the other big issues is that we will need to manage a mixed environment GAIL-Cally, so I need to check how to do that. Other thing to review is just print all the things currently exposed (just mutter.dump()) and see if we miss something.

Clutter 1.0 released -> Cally 1.0 has born

Today Emmanuel Bassi finally announced the release of Clutter 1.0.0. Great news.

So, for some hours the Cally master didn’t compile against the clutter master. A new branch (cally-0.9) was created (although probably not required), and the master version was updated. As I explain on the README file at Cally, in order to make life easier, the version of Cally is tied to the version of Clutter, in order to avoid questions like “which version of Cally I need to install in order to support Clutter A.B.C?”. So, Cally 0.9.X will grant accessibility support for Clutter 0.9.Y. X and Y will kept independent, for internal usage, in a similar way to GAIL with Gtk+.

In the same way, I started to reduce the awful TODO file, and create some bugs on the clutter bugzilla, in order to trace the missing stuff and bugs in a sensible way. So, since today, Cally has several bugs.

Cally moved to Clutter repository and other news

As some of you already know, at first the Cally git repository was placed on Igalia server, waiting to be moved to the Clutter repositories. As one of the GCDS post-effects, this move was being made. So, the new way to get the Cally source code is:

  • git clone git://git.clutter-project.org/cally

And the development will continue there, so the Igalia one would be closed soon.

In the same way, a new component was added to the O-hand Clutter bugzilla, so now you can open a bug against Cally. Now I will start to transform the über TODO file at Cally in small bugs, in order to track the missing features and other bugs in a sensible way.

I would like to thank Emmanuel Bassi, Ross Burton and Richard Purdie for this move.

Other news:

  • For the people who went to my GCDS talk, I added a branch with the Clutter example I used. This is a heavily modified version of the clutter toy foofone called CrapPhone. This also includes a dogtail script. For the moment it will be maintained as a different branch, as I’m not sure if it is worth to include it with the rest of the examples on the master branch.
  • A first commit related to the AtkText support on ClutterText (CallyText) was pushed on the master, containing the implementation of most of the methods of this interface. It also adds an example (on the examples directory), that tries to use all this methods. This implementation is based on GailEntry, and the methods that still remains are the ones that on GailEntry was based on GailMisc. This library is really tied to Gtk, and some methods are not 100% useful on Cally. Probably a equivalent object (ie: CallyMisc) would be created in order to fulfill this functionality. In the same way, I think that worths to not create über commits, and split this a little. And probably I will start the support for AtkEditableText before finish that.

My last day in GCDS

Like a lot of other fellow igalians, I has been this week on the GCDS (Guadec+Akademy) at the beatiful city of Gran Canaria (beatiful and not as hot as I feared).

Probably I should have made more posts about that, but there were several talks to see, but more important, I used most of my free time to prepare my talk on the GUADEC. This task was scheduled yesterday, at 10:00. Yes, the first talk in the morning. And yes, the day after the sponsored Igalia party 😉 . But in the end there were more people than I expected (although in the same way, I started 5 minutes late by purpose 😛 ).

My talk was about Cally, the Clutter Accessibility Library implementation. In fact we can say that the Tuesday morning was the “Clutter morning”, as there were three talks about Clutter.

Except for the fact that in the middle of the talk the microphone fall to the floor, I think that it was fine. If you are interested, you can download the slides here.

As I said on the talk, currently Cally works, but for a specific scope. So, what about the future?

  • Complete the missing functionality (AtkText: work in progress)
  • Review new clutter-based toolkits (glitter, nbtk, …)
  • Review applications using clutter directly (gnome-shel, …)

Although the GCDS will finish on Friday, this is my last day here (I preferred to go back earlier) . As the song says, “all come to an end“, so I will come back to Coruña tomorrow.

I saw several interesting talks, I met some interesting people and I enjoy all the guadec, but in the same way I’m really tired right now 😉