Accessibility Testing Software Compared

Steve Faulkner, Web Accessibility Consultant, Vision Australia Foundation [HREF1], 454 Glenferrie Rd, Kooyong 3144. steven.faulkner@visionaustralia.org.au

Andrew Arch, Manager Online Accessibility Consulting, Vision Australia Foundation [HREF1], 454 Glenferrie Rd, Kooyong 3144. andrew.arch@visionaustralia.org.au

Contents: introduction | conformance | investigations | conclusion | references | detailed results

Abstract

Web accessibility for people with disabilities and other disadvantaged groups is becoming increasingly important for government and educational institutions as they try to meet their obligations under the Disability Discrimination Act and various policies and guidelines for online publishing. Business are also obligated under the DDA not to discriminate against people with disabilities as a result of their online activities.

A plethora of web accessibility testing tools have been released on to the market over the past eighteen months at prices ranging from a few hundred dollars to many thousands of dollars. This study looks at the ability of four of these testing tools to accurately assess the accessibility issues on a web site against the W3C Web Content Accessibility Guidelines 1.0 Priority 1 checkpoints. We conclude that they all have strengths and weaknesses and that none of them are able to identify all the accessibility issues. At this stage these tools can only aid accessibility testing, not provide a definitive assessment.

Introduction

With the increasing requirement for Government, education and business to provide accessible online services in order to provide access to all citizens including people with disabilities, many web managers are turning to the "robotic" tools to spider through their sites and tell them the problems and accessibility 'hot spots'. This has led to a veritable industry of software developers trying to solve this problem for those requiring "accessible" web sites, spured along with the recently enacted Section 508 [Section 508] law in the United State requring Federal Government agencies to build accessible web sites and generally to "buy accessible".

However, as we shall demonstrate in this paper, not all accessibility assessment software tools are created equal. Some overstate the problems while others understate the problems that exist on a web site. And they don't even do this consistently across the sixty five checkpoints of the Web Content Accessibility Guidelines 1.0 (WCAG) [Chisholm, et al ].

Web accessibility testing tools is a relatively new software area - most tools have been available for less than two years. Comparisons of the efficacy of the tools have not been undertaken as the market has matured. Graves [2001] in Government Computer News conducted one of the earliest comparisons between InFocus 508 and PageScreamer 2.3 noting the problems both products had with tables. Harrison [2002] and Harrison and O' Grady [2002] presented a more rigorous comparison of six analysis and repair tools noting the difficulty in isolating errors identified by some of the tools and the emphasis on US accessibility standards rather than the international WCAG ones.

Our aim has been to investigate the efficacy and accuracy of some of the available software applications used for web accessibility testing , in order to help assess the value of such tools in the broader process of ensuring the accessibility of web sites [Brewer & Letourneau].

How is conformance ascertained

What is accessibility?

An accessible web is available to people with disabilities, including those with:

• Vision impairment (e.g. low vision or colour blindness) or vision loss affecting their ability to discern or see the screen
• Physical impairment affecting their ability to use a mouse or keyboard
• Hearing impairment or loss affecting their ability to discern or hear online audio
• Cognitive impairments (e.g. dyslexia, ADD, learning difficulties, memory impairment) affecting their ability to comprehend or understand your site
• Literacy impairments (e.g. low reading skills or English is not their first language) possibly affecting their ability to fully understand your site and its messages

Beneficiaries from an accessible web, however, are a much wider group than just people with disabilities and also include:

• People with poor communications infrastructure, especially rural Australians
• Older people and new users, often computer illiterate
• People with old equipment (not capable of running the latest software)
• People with "non-standard" equipment (e.g. WAP phones and PDA's)
• People with restricted access environments (e.g. locked-down corporate desktops)
• People with temporary impairments or who are coping with environmental distractions

How do we check for an accessible web site?

The Web Accessibility Initiative (WAI) outlines approaches for preliminary and conformance reviews of web sites [Brewer & Letourneau]. Both approaches recommend the use of 'accessibility evaluation tools' to identify some of the issues that occur on a web site. The WAI web site includes a large list of software tools to assist with conformance evaluations [Chisholm & Kasday]. These tools range from automated spidering tools such as the infamous Bobby [ Watchfire, 2003], to tools to assist manual evaluation such as The WAVE [WebAIM], to tools to assist assessment of specific issues such as colour blindness. Some of the automated accessibility assessment software tools also have options for HTML repair.

What role can automated tools play in assessing the accessibility of a web site?

WCAG 1.0 comprises 65 Checkpoints. Some of these are qualified with "Until user agents ..." and with the advances in browsers and assistive technology since 1999, some of these are no longer applicable - leaving us with 61 Checkpoints. Of these only 13 are clearly capable of being tested definitively, with another 27 that can be tested for the presence of the solution or potential problem, but not whether it has definitively been resolved satisfactorily. With intelligent algorithms many of the tools can narrow down the instances of potential issues that need manual checking, e.g. the use of "spacer" as the alt text for spacer.gif used to position elements on the page.

These automated tools are very good at identifying pages and lines of code that need to be manually checked for accessibility. Unfortunately, many people misuse these tools and place a "passed" (e.g. XYZ Approved) graphic on their site when the tool can not identify any specific accessibility issues, but the site has not been competently manually assessed for issues that are not software checkable.

So, automated software tools can:

• check the syntax of the site's code
• identify some actual accessibility problems
• identify some potential problems
• identify pages containing elements that may cause problems
• search for known patterns that humans have listed

However, automated software tools cannot:

• check for appropriate meaning
• check for appropriate rendering (auditory, variety of visual)

The interpretation of the results from the automated tools requires assessors trained in accessibility techniques with an understanding of the technical and usability issues facing people with disabilities. A thorough understanding of accessibility is also required in order to competently assess the checkpoints that the automated tools cannot check such as consistent navigation, and appropriate writing and presentation style.

Investigation of efficacy of accessibility software

Choice of software

The choice of tools to review was based on a number of factors:

• The software needed to have the ability check WCAG 1.0 checkpoints (some tools check only for US Section 508 problems). This decision was based on the applicability of the WCAG 1.0 to Australian disability regulations.
• The least expensive desktop software products from each software producer. This decision was based upon the hypothesis that the more sophisticated versions of the testing software are built upon the same 'testing engines' and algorithms as the entry level products.
• Potential users were considered more likely to purchase the less expensive products as Web Accessibility testing would not be a major priority for many organisations.
• The availability of trial versions of the software products to be reviewed and the resources available to conduct the review.

We intend to expand the list of software products reviewed as software and resources become available

1. A new version of Bobby has been released since the testing was conducted.

It is evident that there is quite a difference in cost across vendors for their entry level products. This difference in cost is partially reflected in the functionality of the software; some tools automatically fix certain problems, but the core functionality, the testing and reporting on WCAG 1.0 issues, is present in all the software reviewed.

Investigation methodology

Site used for testing

The site "The University of Antarctica" used for the review is a demonstration site developed by WebAIM, a non profit organisation whose stated goal "is to improve accessibility to online learning opportunities for all people".

The site contains examples of the many potential barriers to accessibility. The site is hosted by WebAIM at http://www.webaim.org/tutorials/uofa/.

The site consists of:

• 28 HTML documents
• 1 CSS file
• 16 GIF files
• 24 JPG files
• 1 SWF file (Flash)
• 1 Java file
• 13 AU files (audio)
• 1 MOV file (multimedia)
• 3 MPEG files (multimedia)

The site was chosen because it was built to demonstrate accessibility problems. The scope of the site is quite small and therefore instances of problems are easily quantified. Furthermore, the site was constructed using plain HTML files, there are no pages generated "on the fly' from a database, making the process of manual checking and quantification of the site a manageable task.

It was also reasoned that the site content and structure is relatively stable and therefore further testing of the site at a later time will still produce accurate comparison results. Further to this, a copy of the site will be stored at http://it-test.com.au/UOFA by Vision Australia Foundation to ensure the sites continuing integrity for testing purposes.

Process followed

Each of the products in the review were set to produce reports detailing issues in reference to the WCAG 1.0 Priority 1 Checkpoints. The reporting options if present were set to produce the standard reports. All reports were produced in HTML format.

AccVerify report comprised 60 HTML files:(Accverify report [zip file 1,128kb])

• 3 summary pages, comprising a listing of Priority 1 checkpoint errors and instances and Priority 1 visual checkpoints (needing human judgment) and instances. Graphical representation of this information is also presented along with a page listing the files that failed against any of the checkpoints along with a link to the associated detailed report and checklist
• 5 statistical pages listing some of the structural elements of the files tested, e.g. tables, forms, images, with links to the pages containing the elements
• 26 detailed report pages (1 for each file tested):
• individual page summary and graphs of checkpoint errors and visual checkpoints
  • detailing of instances of specific issues with links to the associated checkpoints on the W3C web site.
  • divided into Priority 1, 2 and 3 issues
• 26 checkpoint pages (1 for each file tested):
• listing of all WCAG 1.0 checkpoints with indication of whether the page passed/failed, not applicable, or needed visual checking for each checkpoint
• short explanation of each checkpoint, paraphrased from WCAG 1.0 guidelines
• divided into Priority 1, 2 and 3 issues

Bobby report comprised 72 HTML files: (Bobby report [zip file 183kb])

• 1 summary page consisting of a short description of all (possible) issues found and links to the site files where an occurrence of the issue was found
• 1 index page consisting of a list of all the files tested with links to their corresponding detailed report
• 35 detailed report pages (1 for each file tested):
• detailing instances of specific issues with links to (locally stored) explanations of the issues and links to the associated checkpoints on the W3C web site
• divided into Priority 1, 2 and 3 issues
• further divided into issues either needing, or not needing, user checking to confirm their existence
• 35 (1 for each file tested) 'text only' versions of the files tested

InFocus report comprising 28 HTML files: (Infocus report [zip file 74kb])

• 3 summary/index pages:
• a summary compliance report page, listing instances of and explaining each violation with links to the reports of "top 5 pages containing this violation".
• a page with links to a detailed report for each page tested, also listing the number of checkpoint 'violations' found on each page
• a page with links to a detailed report for each page tested ordered by (descending) total number of 'violations' found on each page
• 25 detailed 'compliance report' pages (1 for each file tested)
• listing each violation found, the associated WCAG 1.0 checkpoint and the offending elements within the HTML code of the page.

PageScreamer report comprising 66 HTML files: (Pagescreamer report [zip file 256kb])

• 1 page containing a copy the WCAG 1.0 guidelines from the W3C web site
• 1 'detail' page listing every HTML element that triggered a checkpoint violation, the elements' URL location and line number
• 1 verification summary page for the site
  • containing a graph showing the number of 'tags' in violation of each checkpoint
  • a table listing all the checkpoints with links to the related sections of the locally stored copy of the WCAG guidelines.
  • numbers of Compliance violation instances and whether the site passed/failed the checkpoint or the checkpoint needed further verification
  • 1 file containing a text description of the graph
• 27 verification summary pages (one for each URL found) containing:
• containing a table listing all the checkpoints with links to the related sections of the locally stored copy of the WCAG guidelines.
• numbers of compliance violation instances and whether the site passed/failed the checkpoint or the checkpoint needed further verification
• 35 pages containing lists of links to URL's (with links to associated report files) containing instances of a violation of a particular checkpoint (1 page per checkpoint)

Overview of results

A significant measure of the software tools and their ability to report on the accessibility problems of a site is the ability to find all the URL's on the target site. Some of the apparent discrepancies between software of the URL count could be apportioned to the software only listing the URL's of files that contain errors. But in this situation all 28 HTML files contained at least one instance of non conformance with a WCAG 1.0 checkpoint.

AccVerify
• Failed to find any instances of the MPEG (3) (multimedia) or AU (13) (audio) files
• Failed to find files (2) that were the targets of forms

Bobby
• Identified instances of audio and multimedia files, although it failed to report some instances.
• Failed to find a file linked via meta based redirect
• Tested and reported twice on the 'Home' page [index.html]
• Failed to find a file linked via an image map based link
• Failed to find files (2) that were the targets of forms

InFocus
• Failed to find files (3) that were the targets of forms

PageScreamer
• Tested and reported twice on the 'Home' page [index.html]
• Failed to find files (2) that were the targets of forms
• Found instances of the MPEG (3) (multimedia) or AU (13) (audio) files but passed site on checkpoints relating to multimedia without verification.

Accuracy and Definitive nature of results

Another significant measure of the products efficacy is its ability to produce both accurate and definitive results without the need for further human interpretation.

1. Bobby and InFocus did not report upon issues that could be checked by the software and found not applicable e.g. they found no server-side image maps therefore did not report about issues concerning client side image maps
2. PageScreamer failed to provide any information in the report about 2 Priority 1 checkpoints (4.1, 14.1).
3. Bobby only reported on those checkpoints that it found the site to be in breach of.
4. PageScreamer incorrectly reported the site having passed 2 checkpoints (1.3, 1.4) that the manual check revealed to be fails.

Graphical representation of data from Table 3

1. AccVerify overstated total potential failures in relation to actual failures
2. Bobby overstated total potential failures in relation to actual failures
3. InFocus overstated total potential failures in relation to actual failures
4. PageScreamer defined the least checkpoints as 'potential failures', but understated total potential failures in relation to actual failures

Graphical of representation of data from Table 4.

Over reporting

'Over reporting of instances of potential checkpoint failures was a common feature of all the products reviewed.

Discussion

While Bobby was the most successful tool in identifying multimedia and audio files it had the most problems identifying HTML files linked via the META and IMAGE MAP elements as well as those files that were the targets of FORM elements. The inability to identify some of the targets of Forms was a common defect among all the software.

The product's ability to give a quantitative answer as to whether the site passed or failed against 15 WCAG 1.0 Priority 1 checkpoints varied from 60% for Pagescreamer to 28% for AccVerify, though it should be noted that the Pagescreamer quantitative results produced 2 false passes. The results highlight that the software tools either did not produce a quantitative report or could not produce an accurate report on the status of the site in relation to the majority of the Priority 1 checkpoints tested. For approximately 10 of the 15 Priority 1 checkpoints tested the reports inform us that the checkpoints need a 'visual', 'manual' or 'user' check. Furthermore, although along with the instructions to do a 'manual' check the reports detail potential instances of checkpoint violations (Table 5), which should be helpful in tracking down issues, when comparing the number of potential instances reported against the actual occurrence, it is evident that none of the tools does a very good job at identifying potential errors. All of the products over-reported potential checkpoint errors/violations. A number of checkpoints were detailed as potential errors on every page by some software tool reports.

All of the products produce a report that upon initial consideration may appear as a detailed analysis of the accessibility problems found by the product on the site tested. Upon closer examination it is revealed that the software tools fail at the initial hurdle of correctly identifying the files to be tested. Furthermore, many of the accessibility issues that may occur on the site are not within the reach of the 'mechanical' rules based analysis that these products undertake. In an attempt to ensure that issues are not missed by the software, the reports tend to overstate the occurrence of potential problems, up to the point where a potential instance of a checkpoint violation is flagged for every file checked, thus undermining even the heuristic values of the report.

Conclusion

The research into the automated accessibility tools reported here and conducted at Vision Australia Foundation indicates that the rule of "caveat emptor" applies as equally to the field of accessibility testing tools as it does to buying a used car.

All of the tools tested have advantages over free online testing tools [Steven Faulkner], the main adavantage being their ability check a whole site rather than a limited number of pages. Also files do not have to be published to the web before they can be tested, the user has greater control of what rules are applied when testing, the style and formatting of reports, and in some cases the software will automatically correct problems found.

None of the tools evaluated, as expected, were able to identify all the HTML files and associated multimedia files that needed to be tested for accessibility on the site.

All the software tools evaluated will assist the accessibility quality assurance process, however none of them will replace evaluation by informed humans. The user needs to be aware of their limitations and needs a strong understanding of accessibility issues and the implications for people with disabilities in order to interpret the reports and the accessibility issues or potential issues flagged by the software tools.

We hope the analysis reported here will aid web site quality assurance managers in their choice of web accessibility testing software, and in their understanding of the limitations that automated tools have in the broader process of ensuring accessible web sites.

References

Chisholm, W. et al (Eds) 1999, Web Content Accessibility Guidelines 1.0, World Wide Web Consortium. [HREF2]

Brewer, J & Letourneau, C. (Eds) 2002, Evaluating Web Sites for Accessibility, World Wide Web Consortium. [HREF3]

Chisholm, W & Kasday, L (Eds) 2002, Evaluation, Repair, and Transformation Tools for Web Content Accessibility, World Wide Web Consortium. [HREF4]

Watchfire, 2003, Welcome to Bobby.[HREF5]

WebAIM, undated, WAVE 3.0 Accessibility Tool.[HREF6]

Section 508: http://www.usdoj.gov/crt/508/508law.html [HREF7]

Graves, Steve 2001, Check sites for 508 with audit-edit tools, Government Computer News [HREF8]

Harrison, Laurie 2002, Web Accessibility Validation and Repair - Which Tool and Why? (introduction), Center On Disabilities Technology And Persons With Disabilities Conference 2002 (CSUN-2002) [HREF9]

Harrison, L & O'Grady, L 2002, Web Accessibility Validation and Repair: Which Tool and Why? (analysis), ATRC, University of Toronto.[HREF10]

Steven Faulkner, Vision Australia Foundation, 2003, Free Web Development and Accessibility Tools. [HREF11]

Hypertext References

HREF1

http://www.visionaustralia.org.au/webaccessibility/

HREF2

http://www.w3.org/TR/WCAG10/

HREF3

http://www.w3.org/WAI/eval/

HREF4

http://www.w3.org/WAI/ER/existingtools.html

HREF5

http://bobby.watchfire.com/bobby/html/en/index.jsp

HREF6

http://www.wave.webaim.org:8081/wave/index.jsp

HREF7

http://www.usdoj.gov/crt/508/508law.html

HREF8

http://www.gcn.com/20_23/reviews/16783-1.html

HREF9

http://www.csun.edu/cod/conf/2002/proceedings/279.htm

HREF10

http://snow.utoronto.ca/access/evaltoolreview/validation.html

HREF11

http://www.visionaustralia.org.au/webaccessibility/workshops/references.html#acheck

 

Appendix - Table of results

Copyright