Accessibility testing

Accessibility in mobile apps means that assistance systems such as VoiceOver or Talkback have technical access to all control elements. The challenge in testing lies in the fact that settings for different disability groups, i.e. the blind, visually impaired and motor-impaired, can interfere with each other. A standardized test matrix does not yet exist.

Key Takeaways

• Assistance systems for different types of disabilities are mutually exclusive on mobile devices: Talkback helps the blind, but at the same time prevents meaningful keyboard use for the motor-impaired.
• A test matrix for accessibility on mobile devices must take hundreds of switch combinations into account, which is why teams must document exactly which settings were active and which were intentionally left deactivated.
• Accessible IT is not just a technical problem: errors occur at the designer and document creator stage, not just during development.
• The curb-cut effect makes accessibility features useful for everyone: high-contrast mode and assistive touch also help people without disabilities, for example in sunlight or when distracted.
• Providing accessible PDF documents is technically feasible, but fails in practice due to the sheer volume of existing documents in organizations.

Accessibility on mobile devices is a different game than on the PC

Anyone who tests software for accessibility knows the setup on the desktop: Keyboard, mouse, maybe a camera. However, as soon as the application to be tested runs on an iPad or Android device, the rules of the game change fundamentally. The assistive systems are already part of the operating system.

Thorsten Schröder and Dirk Haas are testing applications in the context of German pension insurance, including a mobile patient file on the iPad. Just turning the device from portrait to landscape format changed the app’s behavior. Individual elements were no longer focused when swiping via VoiceOver and were therefore inaccessible for blind users.

On the mobile device, you switch on functions such as VoiceOver on iOS or Talkback on Android to have content read aloud to you. The standard behavior of the app, swiping and tapping, should then suddenly work via a keyboard. This is exactly where the application starts to behave in an unusual way.

Why assistance functions block each other

The biggest problem when testing mobile accessibility is that settings for different types of disabilities are mutually exclusive. What helps one group excludes another.

A concrete example: For people with motor impairments, there is the switch control. On an iPhone, the camera can react to head movements and thus navigate from element to element, triggered by a sound, for example. However, this control cannot be operated simultaneously with VoiceOver. A blind person would have no chance with this combination.

This is a dilemma for the test report. The test criteria are all on an equal footing. There is no criterion that determines whether a requirement applies only to blind users or only to users with motor impairments. The sentence is always the same. The restriction must be documented by the tester himself.

Who is responsible for accessibility, the device or the software?

Mobile devices are offering more and more alternative input options, and this raises a tough evaluation question. One example illustrates the problem precisely.

The WCAG contains a test criterion for gesture and motion control: an action such as shaking the device to discard data must also be able to be performed with a finger as an alternative. The software should actually offer this alternative itself. However, you can activate AssistiveTouch on an iOS device. This allows the operating system to simulate shaking with a tap of the finger.

Does this make the software accessible or not? It does not provide the alternative itself, but the task still works in a specific environment. The person operating the device doesn’t care where the help comes from. The main thing is that it works.

Every release brings new user aids, and every time we have to look again because there really is a lot that is new. Dirk Haas

How to set up a test matrix for hundreds of combinations

The practical way is through systematic trial and error and complete documentation of the respective constellation. There is no ready-made solution for all cases.

Schröder and Haas sit down several times, testing button and switch combinations and checking the consequences. If a certain switch is not set and a blind person can operate everything with it, the criterion for this type of disability is considered passed. But precisely this condition must be explicitly noted, because the next user may not activate the switch control and not make any progress.

This makes the scope of a test report very narrow. There are now hundreds of combinations of switch options, all of which are impossible to map. Both the activated settings and explicitly those that lead to errors are therefore documented.

A fixed rule limits the effort. The operating mode is not changed in between within a use case.

Either it works with one setting or it works with the other. But having to change in the middle is not reasonable. Thorsten Schröder

The client provides the relevant use cases. A tester cannot judge whether a blood pressure value that is recorded five times a day has a different relevance to a number that is only recorded once at the start of the recording.

Why accessibility can hardly be tested automatically

Accessibility tests for assistance systems are purely manual, and there is a logical reason for this. The test checks whether the assistive technology can access the content at all.

A screen reader needs access to an element, its properties and its status in order to be able to read it out. Test software would need the same access. If a field cannot be accessed, an automatic test cannot be written for it. However, this lack of accessibility is often the flaw that the test uncovers.

Many common test tools check things like valid HTML or the parsing of HTML. For a person with a disability working with the application, this often has no relevance. Modern browsers also interpret non-valid HTML reliably.

This shift is reflected in a regulatory premiere. With WCAG 2.2, a test criterion was removed from the register for the first time: the criterion for parsing HTML. Tools that specialize in precisely this would thus lose a considerable part of their findings.

However, this change is not yet effective in practice. Current laws and standards refer to the WCAG, not to individual adaptations. The legal situation will not change until the European standard EN 301549 is updated and the relevant point is deleted. Such adaptations take a long time at European level.

iOS and Android are converging, but habit binds

Apple’s earlier lead in assistance systems has now largely been overtaken. The functionalities can now be found almost one-to-one in both systems.

Apple has integrated screen readers and assistive systems right from the start. With each new iOS version, additional features are added, such as door or object recognition via the magnifying glass, which uses a camera to identify people or objects. For testers, this means more work, not less, because each new option increases the size of the test matrix.

One practical factor remains: Once you have become accustomed to a device and its operation, you rarely change. This applies to all users and especially to those who rely on assistance systems.

What is built for the disabled benefits everyone

Functions for people with disabilities are also useful for everyone else. This curb-cut effect describes precisely the phenomenon.

A high contrast or black and white mode doesn’t just help visually impaired people. As soon as you are standing outside in the sun and the surface of your cell phone is reflective, high contrast is often the only chance you have of recognizing anything at all. With pink on green, you can no longer see anything.

Such aids are used in everyday life without realizing the original intention. This is one of the strongest arguments for communicating accessibility: It’s not just for the disabled, it benefits a broad user group.

Why testers with disabilities are specifically included, but not for every test

People with disabilities are included selectively, but a complete test with all disability groups would not be affordable. The effort involved would explode.

A blind person cannot see what they cannot see. If an element is not controlled at all, they cannot judge whether it has been implemented correctly or incorrectly. For a comprehensive test, you would need hearing-impaired, motor-impaired, severely visually impaired and blind people, each with different levels of impairment.

Schröder and Haas cover a broad portfolio with their own test and seek specific expertise in the event of uncertainties. A blind employee, a former software developer for screen readers, is involved in the development of a new design system right from the start. His technical contribution is one effect.

The other effect is communication. Anyone working with a blind colleague cannot simply send them a screenshot without a description. This forces the team to communicate in an accessible way themselves. The team also has its own presentation documents checked, because anyone who talks about the topic should make their content accessible to at least the same extent.

This shows how differently users work. There are blind people who do not master Braille and only work with voice output, and others who rely exclusively on the Braille display. Both hardly understand how the other works. It is beyond any realistic scope to fully compare this diversity in a test.

The testers themselves take a deliberately naive approach and use the software as it is available. Experienced screen reader users have developed their own procedures, know the problem areas and work around them automatically. However, this cannot be the benchmark, because then everyone else would have to be able to do the same.

Accessibility is not just a technical issue

The biggest misconception is that accessible IT is solely a matter of technology. If the structure of an application isn’t right, even the best technical implementation won’t help.

If a design is poorly constructed and the technical structure does not correspond to the visual presentation, a blind person will never find their way around. This happens, for example, when someone works with layout tables. The responsibility therefore does not lie with the developer at the end of the chain, but with the designer, the copywriters and the creators of the documents.

A document can technically be made seemingly accessible immediately by marking all content as an artifact. The checking tool then reports green because there is nothing left for the screen reader. For the blind user, however, the document is empty. Accessibility must be considered from the top down, in the program design and in the provision of time and resources.

The next major construction site is accessible documents

The topic that is currently on everyone’s mind is the provision of accessible documents and forms. The move away from paper to electronically accessible content is huge.

The effort per document is not endless, but the number of documents is almost. If you look at your own company, you will find a daunting number of instructions, forms and texts. Each of these would have to be processed once to make them accessible. Partial automation using appropriate tools is in progress, but this does not change the fact that bad content remains bad even if it is barrier-free.

The pressure is distributed differently. Public institutions such as pension insurance are already under legal obligation. It is now becoming relevant for the free market: Anyone who wants to establish a web store in Europe and does not want to risk penalties from market surveillance must start implementing accessibility, otherwise it will be expensive.