Accessibility Testing Software Compared

Steve Faulkner, Web Accessibility Consultant, Vision Australia Foundation [HREF1], 454 Glenferrie Rd, Kooyong 3144.

Andrew Arch, Manager Online Accessibility Consulting, Vision Australia Foundation [HREF1], 454 Glenferrie Rd, Kooyong 3144.

Contents: introduction | conformance | investigations | conclusion | references | detailed results


Web accessibility for people with disabilities and other disadvantaged groups is becoming increasingly important for government and educational institutions as they try to meet their obligations under the Disability Discrimination Act and various policies and guidelines for online publishing. Business are also obligated under the DDA not to discriminate against people with disabilities as a result of their online activities.

A plethora of web accessibility testing tools have been released on to the market over the past eighteen months at prices ranging from a few hundred dollars to many thousands of dollars. This study looks at the ability of four of these testing tools to accurately assess the accessibility issues on a web site against the W3C Web Content Accessibility Guidelines 1.0 Priority 1 checkpoints. We conclude that they all have strengths and weaknesses and that none of them are able to identify all the accessibility issues. At this stage these tools can only aid accessibility testing, not provide a definitive assessment.


With the increasing requirement for Government, education and business to provide accessible online services in order to provide access to all citizens including people with disabilities, many web managers are turning to the "robotic" tools to spider through their sites and tell them the problems and accessibility 'hot spots'. This has led to a veritable industry of software developers trying to solve this problem for those requiring "accessible" web sites, spured along with the recently enacted Section 508 [Section 508] law in the United State requring Federal Government agencies to build accessible web sites and generally to "buy accessible".

However, as we shall demonstrate in this paper, not all accessibility assessment software tools are created equal. Some overstate the problems while others understate the problems that exist on a web site. And they don't even do this consistently across the sixty five checkpoints of the Web Content Accessibility Guidelines 1.0 (WCAG) [Chisholm, et al ].

Web accessibility testing tools is a relatively new software area - most tools have been available for less than two years. Comparisons of the efficacy of the tools have not been undertaken as the market has matured. Graves [2001] in Government Computer News conducted one of the earliest comparisons between InFocus 508 and PageScreamer 2.3 noting the problems both products had with tables. Harrison [2002] and Harrison and O' Grady [2002] presented a more rigorous comparison of six analysis and repair tools noting the difficulty in isolating errors identified by some of the tools and the emphasis on US accessibility standards rather than the international WCAG ones.

Our aim has been to investigate the efficacy and accuracy of some of the available software applications used for web accessibility testing , in order to help assess the value of such tools in the broader process of ensuring the accessibility of web sites [Brewer & Letourneau].

How is conformance ascertained

What is accessibility?

An accessible web is available to people with disabilities, including those with:

Beneficiaries from an accessible web, however, are a much wider group than just people with disabilities and also include:

How do we check for an accessible web site?

The Web Accessibility Initiative (WAI) outlines approaches for preliminary and conformance reviews of web sites [Brewer & Letourneau]. Both approaches recommend the use of 'accessibility evaluation tools' to identify some of the issues that occur on a web site. The WAI web site includes a large list of software tools to assist with conformance evaluations [Chisholm & Kasday]. These tools range from automated spidering tools such as the infamous Bobby [ Watchfire, 2003], to tools to assist manual evaluation such as The WAVE [WebAIM], to tools to assist assessment of specific issues such as colour blindness. Some of the automated accessibility assessment software tools also have options for HTML repair.

What role can automated tools play in assessing the accessibility of a web site?

WCAG 1.0 comprises 65 Checkpoints. Some of these are qualified with "Until user agents ..." and with the advances in browsers and assistive technology since 1999, some of these are no longer applicable - leaving us with 61 Checkpoints. Of these only 13 are clearly capable of being tested definitively, with another 27 that can be tested for the presence of the solution or potential problem, but not whether it has definitively been resolved satisfactorily. With intelligent algorithms many of the tools can narrow down the instances of potential issues that need manual checking, e.g. the use of "spacer" as the alt text for spacer.gif used to position elements on the page.

These automated tools are very good at identifying pages and lines of code that need to be manually checked for accessibility. Unfortunately, many people misuse these tools and place a "passed" (e.g. XYZ Approved) graphic on their site when the tool can not identify any specific accessibility issues, but the site has not been competently manually assessed for issues that are not software checkable.

So, automated software tools can:

However, automated software tools cannot:

The interpretation of the results from the automated tools requires assessors trained in accessibility techniques with an understanding of the technical and usability issues facing people with disabilities. A thorough understanding of accessibility is also required in order to competently assess the checkpoints that the automated tools cannot check such as consistent navigation, and appropriate writing and presentation style.

Investigation of efficacy of accessibility software

Choice of software

The choice of tools to review was based on a number of factors:

We intend to expand the list of software products reviewed as software and resources become available

Table 1: Software Reviewed
Software Vendor URL Cost
AccVerify 4.9 HiSoftware US $495
Bobby 4.0.11 Watchfire US $99
InFocus 4.2 SSB Technologies US $1,795
PageScreamer 4.1 Crunchy Technologies US $1,495

1. A new version of Bobby has been released since the testing was conducted.

It is evident that there is quite a difference in cost across vendors for their entry level products. This difference in cost is partially reflected in the functionality of the software; some tools automatically fix certain problems, but the core functionality, the testing and reporting on WCAG 1.0 issues, is present in all the software reviewed.

Investigation methodology

Site used for testing

The site "The University of Antarctica" used for the review is a demonstration site developed by WebAIM, a non profit organisation whose stated goal "is to improve accessibility to online learning opportunities for all people".

The site contains examples of the many potential barriers to accessibility. The site is hosted by WebAIM at

The site consists of:

The site was chosen because it was built to demonstrate accessibility problems. The scope of the site is quite small and therefore instances of problems are easily quantified. Furthermore, the site was constructed using plain HTML files, there are no pages generated "on the fly' from a database, making the process of manual checking and quantification of the site a manageable task.

It was also reasoned that the site content and structure is relatively stable and therefore further testing of the site at a later time will still produce accurate comparison results. Further to this, a copy of the site will be stored at by Vision Australia Foundation to ensure the sites continuing integrity for testing purposes.

Process followed

Each of the products in the review were set to produce reports detailing issues in reference to the WCAG 1.0 Priority 1 Checkpoints. The reporting options if present were set to produce the standard reports. All reports were produced in HTML format.

AccVerify report comprised 60 HTML files:(Accverify report [zip file 1,128kb])

Bobby report comprised 72 HTML files: (Bobby report [zip file 183kb])

InFocus report comprising 28 HTML files: (Infocus report [zip file 74kb])

PageScreamer report comprising 66 HTML files: (Pagescreamer report [zip file 256kb])

Overview of results

A significant measure of the software tools and their ability to report on the accessibility problems of a site is the ability to find all the URL's on the target site. Some of the apparent discrepancies between software of the URL count could be apportioned to the software only listing the URL's of files that contain errors. But in this situation all 28 HTML files contained at least one instance of non conformance with a WCAG 1.0 checkpoint.

Table 2: HTML files found
AccVerify Bobby InFocus PageScreamer Manual Check

Accuracy and Definitive nature of results

Another significant measure of the products efficacy is its ability to produce both accurate and definitive results without the need for further human interpretation.

Table 3: Status of (15 priority 1) checkpoints tested
AccVerify Bobby InFocus PageScreamer Manual Check
n/a 3
4 4
intervention (%)
11 (73%)
11 (73%)
8 (53%)
6 (40%)
Not reported
2 1
2 1
2 2
  1. Bobby and InFocus did not report upon issues that could be checked by the software and found not applicable e.g. they found no server-side image maps therefore did not report about issues concerning client side image maps
  2. PageScreamer failed to provide any information in the report about 2 Priority 1 checkpoints (4.1, 14.1).
  3. Bobby only reported on those checkpoints that it found the site to be in breach of.
  4. PageScreamer incorrectly reported the site having passed 2 checkpoints (1.3, 1.4) that the manual check revealed to be fails.

Graphical representation of data from Table 3
Graphical represenation of data from Table 3 [Status of (15 priority 1) checkpoints tested]

Table 4: Totals of reported potential/actual failures of (15 priority 1) checkpoints tested
AccVerify 1 Bobby 2 InFocus3 PageScreamer4 Manual Check
Potentials Failures
(human Intervention)
  1. AccVerify overstated total potential failures in relation to actual failures
  2. Bobby overstated total potential failures in relation to actual failures
  3. InFocus overstated total potential failures in relation to actual failures
  4. PageScreamer defined the least checkpoints as 'potential failures', but understated total potential failures in relation to actual failures

Graphical of representation of data from Table 4.
Graphical represenation of data from Table 4 [Totals of reported potential/actual failures of (15 priority 1) checkpoints tested]

Over reporting

'Over reporting of instances of potential checkpoint failures was a common feature of all the products reviewed.

Table 5: Examples of reported Instances of potential/actual checkpoints failures
AccVerify Bobby InFocus PageScreamer Manual Check
2.1 - Ensure that all information conveyed with color is also available without color
5.1 - For data tables, identify row and column headers.
5.2 - For data tables that have two or more logical levels of row or column headers, use markup to associate data cells and header cells.


While Bobby was the most successful tool in identifying multimedia and audio files it had the most problems identifying HTML files linked via the META and IMAGE MAP elements as well as those files that were the targets of FORM elements. The inability to identify some of the targets of Forms was a common defect among all the software.

The product's ability to give a quantitative answer as to whether the site passed or failed against 15 WCAG 1.0 Priority 1 checkpoints varied from 60% for Pagescreamer to 28% for AccVerify, though it should be noted that the Pagescreamer quantitative results produced 2 false passes. The results highlight that the software tools either did not produce a quantitative report or could not produce an accurate report on the status of the site in relation to the majority of the Priority 1 checkpoints tested. For approximately 10 of the 15 Priority 1 checkpoints tested the reports inform us that the checkpoints need a 'visual', 'manual' or 'user' check. Furthermore, although along with the instructions to do a 'manual' check the reports detail potential instances of checkpoint violations (Table 5), which should be helpful in tracking down issues, when comparing the number of potential instances reported against the actual occurrence, it is evident that none of the tools does a very good job at identifying potential errors. All of the products over-reported potential checkpoint errors/violations. A number of checkpoints were detailed as potential errors on every page by some software tool reports.

All of the products produce a report that upon initial consideration may appear as a detailed analysis of the accessibility problems found by the product on the site tested. Upon closer examination it is revealed that the software tools fail at the initial hurdle of correctly identifying the files to be tested. Furthermore, many of the accessibility issues that may occur on the site are not within the reach of the 'mechanical' rules based analysis that these products undertake. In an attempt to ensure that issues are not missed by the software, the reports tend to overstate the occurrence of potential problems, up to the point where a potential instance of a checkpoint violation is flagged for every file checked, thus undermining even the heuristic values of the report.


The research into the automated accessibility tools reported here and conducted at Vision Australia Foundation indicates that the rule of "caveat emptor" applies as equally to the field of accessibility testing tools as it does to buying a used car.

All of the tools tested have advantages over free online testing tools [Steven Faulkner], the main adavantage being their ability check a whole site rather than a limited number of pages. Also files do not have to be published to the web before they can be tested, the user has greater control of what rules are applied when testing, the style and formatting of reports, and in some cases the software will automatically correct problems found.

None of the tools evaluated, as expected, were able to identify all the HTML files and associated multimedia files that needed to be tested for accessibility on the site.

All the software tools evaluated will assist the accessibility quality assurance process, however none of them will replace evaluation by informed humans. The user needs to be aware of their limitations and needs a strong understanding of accessibility issues and the implications for people with disabilities in order to interpret the reports and the accessibility issues or potential issues flagged by the software tools.

We hope the analysis reported here will aid web site quality assurance managers in their choice of web accessibility testing software, and in their understanding of the limitations that automated tools have in the broader process of ensuring accessible web sites.


Chisholm, W. et al (Eds) 1999, Web Content Accessibility Guidelines 1.0, World Wide Web Consortium. [HREF2]

Brewer, J & Letourneau, C. (Eds) 2002, Evaluating Web Sites for Accessibility, World Wide Web Consortium. [HREF3]

Chisholm, W & Kasday, L (Eds) 2002, Evaluation, Repair, and Transformation Tools for Web Content Accessibility, World Wide Web Consortium. [HREF4]

Watchfire, 2003, Welcome to Bobby.[HREF5]

WebAIM, undated, WAVE 3.0 Accessibility Tool.[HREF6]

Section 508: [HREF7]

Graves, Steve 2001, Check sites for 508 with audit-edit tools, Government Computer News [HREF8]

Harrison, Laurie 2002, Web Accessibility Validation and Repair - Which Tool and Why? (introduction), Center On Disabilities Technology And Persons With Disabilities Conference 2002 (CSUN-2002) [HREF9]

Harrison, L & O'Grady, L 2002, Web Accessibility Validation and Repair: Which Tool and Why? (analysis), ATRC, University of Toronto.[HREF10]

Steven Faulkner, Vision Australia Foundation, 2003, Free Web Development and Accessibility Tools. [HREF11]

Hypertext References


Appendix - Table of results

Priority 1 Checkpoints and Indication of software report on each checkpoint
In General (Priority 1) Manual Check AccVerify Bobby InFocus Page Screamer
1.1 Provide a text equivalent for every non-text element (e.g., via "alt", "longdesc", or in element content). This includes: images, graphical representations of text (including symbols), image map regions, animations (e.g., animated GIF's), applets and programmatic objects, ascii art, frames, scripts, images used as list bullets, spacers, graphical buttons, sounds (played with or without user interaction), stand-alone audio files, audio tracks of video, and video. fail fail fail fail fail
2.1 Ensure that all information conveyed with color is also available without color, for example from context or markup. fail visual user check manual check verify
4.1 Clearly identify changes in the natural language of a document's text and any text equivalents (e.g., captions). fail visual user check manual check not reported
6.1 Organize documents so they may be read without style sheets. For example, when an HTML document is rendered without associated style sheets, it must still be possible to read the document. fail visual user check manual check verify
6.2 Ensure that equivalents for dynamic content are updated when the dynamic content changes. fail visual user check manual check verify
7.1 Until user agents allow users to control flickering, avoid causing the screen to flicker. pass visual user check manual check verify
14.1 Use the clearest and simplest language appropriate for a site's content. pass visual user check manual check not reported
And if you use images and image maps (Priority 1) Manual Check AccVerify Bobby InFocus Page Screamer
1.2 Provide redundant text links for each active region of a server-side image map. pass pass not reported not reported pass
9.1 Provide client-side image maps instead of server-side image maps except where the regions cannot be defined with an available geometric shape. pass pass not reported not reported pass
And if you use tables (Priority 1) Manual Check AccVerify Bobby InFocus Page Screamer
5.1 For data tables, identify row and column headers. fail visual user check fail verify
5.2 For data tables that have two or more logical levels of row or column headers, use markup to associate data cells and header cells. fail visual user check fail verify
And if you use frames (Priority 1) Manual Check AccVerify Bobby InFocus Page Screamer
12.1 Title each frame to facilitate frame identification and navigation. fail fail fail fail fail
And if you use applets and scripts (Priority 1) Manual Check AccVerify Bobby InFocus Page Screamer
6.3 Ensure that pages are usable when scripts, applets, or other programmatic objects are turned off or not supported. If this is not possible, provide equivalent information on an alternative accessible page. fail visual user check manual check fail
And if you use multimedia (Priority 1) Manual Check AccVerify Bobby InFocus Page Screamer
1.3 Until user agents can automatically read aloud the text equivalent of a visual track, provide an auditory description of the important information of the visual track of a multimedia presentation. fail visual user check fail pass
1.4 For any time-based multimedia presentation (e.g., a movie or animation), synchronize equivalent alternatives (e.g., captions or auditory descriptions of the visual track) with the presentation. fail visual user check manual check pass
And if all else fails (Priority 1) Manual Check AccVerify Bobby InFocus Page Screamer
11.4 If, after best efforts, you cannot create an accessible page, provide a link to an alternative page that uses W3C technologies, is accessible, has equivalent information (or functionality), and is updated as often as the inaccessible (original) page. not tested


Vision Australia Foundation, © 2000. The authors assign to Southern Cross University and other educational and non-profit institutions a non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The authors also grant a non-exclusive licence to Southern Cross University to publish this document in full on the World Wide Web and on CD-ROM and in printed form with the conference papers and for the document to be published on mirrors on the World Wide Web.