The Pros and Cons of Automated Accessibility Testing Tools Chetan Bakhru Matt Baillargeon

1. The Pros and Cons of Automated Accessibility Testing Tools Chetan Bakhru Matt Baillargeon

2. About SSB BART Group • Unmatched Experience • Accessibility Focus • Implementation-Oriented Solutions • Solutions That Reduce Legal Risk • Organizational Stability and Continuity • Knowledge That Is Up-to-Date, All the Time • Published and Peer Review Auditing Methodology • Fourteen hundred organizations (1445) • Fifteen hundred individual accessibility best practices (1595) • Twenty-two core technology platforms (22) • Fifty-five thousand audits (55,930) • One hundred fifty million accessibility violations (152,351,725) • Three hundred sixty-six thousand human validated accessibility violations (366,096)

3. Review of Accessibility Standards WCAG (International) The World Wide Web Consortium (W3C) publishes the Web Content Accessibility Guidelines (WCAG) The WCAG has two versions 1.0 (1999) and 2.0 (2009) These form the basis of most Web accessibility standards including Section 508 Section 508 (U.S.) The current Section 508 standards are based on the WCAG 1.0 standards but are structured around technical, functional and support requirements The Section 508 refresh standards are due in 2010 and will maintain a relationship with WCAG Application US Public Sector organizations generally require Section 508 compliance US private sector organizations generally require ADA compliance – likely standards are WCAG 2.0 Level A and AA conformance

4. Accessibility Auditing Requirements Technical Requirements Requires a system to have a conformant technical implementation Testing requirements are split between those that can be tested Automatically (24.8%), Manually (48.3%) and Globally (26.9%) Automatic testing is the cheapest and most common testing but covers only a small fraction of legal requirements Functional Requirements Requires a system to be usable to people with disabilities using current assistive technologies Functional testing coverage for sensory and mobility impairments is generally required What is Accessibility Testing?

5. Automated Testing Tools - Overview What is an Automated Testing Tool? • A piece of software which can evaluate a web page, or even an entire website, for accessibility • Such tools have existed for a long time and have historically been a useful way of checking websites for accessibility • Some come in the form of a browser add-on or toolbar • Some are free while others have either one-time or recurring costs associated with them • Some are more feature-rich than others

6. Automated Tool Limitations - Overview • Cons of Automated Tools • Limited testing coverage for all accessibility auditing requirements • False sense of security for risk mitigation • Potential for false positives • Staying up to date on changing laws and accessibility guidelines (especially in free solutions) • Many solutions do not provide for manual testing guidance and reporting • Many solutions do not provide support for functional testing and reporting • Literal interpretation of guidelines

7. Limitations – False Sense of Security • Many automated tools provide reports which do not call out the level of accessibility testing that has been completed • Reports based on automated testing alone may contain favorably high compliance scores creating a false sense of security • Many automated tools do not provide coverage for other technologies outside of the web • Many only have the ability to test public facing content and cannot test sites which are behind a firewall, in a staging environment, require user interaction, authentication, etc

8. Limitations – False Positives • Automated testing solutions often do not test the true Document Object Model (DOM) • Many solutions do not allow for organizations to edit or extend the automated testing logic • Inaccurate results can skew the validity of the resulting accessibility reports • Providing inaccurate results to remediation teams can prove to decrease efficiency and increase cost • Many automated testing solutions struggle with accurately testing content created using the latest technologies

9. Limitations – Changing Guidelines • Some automated testing tools do not provide support for changing or newly developed accessibility guidelines • Working with out of date or irrelevant accessibility guidelines is not only a waste of recourses, but also exposes the organization to a greater level of risk • Remediation efforts based on out of date accessibility guidelines may actually hinder the compliance level of a site

10. Limitations – Manual Testing • Many solutions do not provide for manual testing guidance and reporting • A solution which lacks accurate guidance on how to accomplish manual testing leaves 75% of the auditing requirements unfinished • Many automated tools do not allow for the cohesive reporting of both automated and manual testing results • Testers are commonly left to their own varying interpretations of accessibility guidelines when performing manual testing

11. Automated Testing Tools Benefits - Overview • Pros of Automated Testing Tools • A great way to jumpstart an accessibility initiative • Quick results are obtained (entire site or single page testing) • Websites can be scanned on a specified schedule • Low cost entry point into accessibility testing • Empowers all roles within an organization to perform a level of compliance testing • A level of statistical data can be produced via automated results providing management with baseline information of the overall web compliance state • Addresses many of the low hanging fruit accessibility violations

12. Benefits – Jumpstart Accessibility Initiative • Automated accessibility testing tools are a great way for an organization to roll out an accessibility initiative • Organizations can rapidly enable all users to perform accessibility testing on their own • Quickly obtain baseline assessments of an organization’s web properties • Great way to educate users on all accessibility auditing requirements such as manual testing • Allows an organization to deploy a standardized testing control to all users

13. Benefits – Usable by All Roles • An automated testing tool can be integrated into all roles of an organization’s development lifecycle • Designers can use to detect preliminary accessibility violations on templates • Developers can use on content that they are in the process of developing • Compliance officers can use to scan production sites to track compliance changes over time

14. Benefits – Establish a Baseline • For many organizations that are starting an accessibility initiative, the current level of accessibility compliance is a complete unknown • Automated tools allow an organization to quickly scan an entire site and obtain some understanding of the current level of accessibility • These types of reports and results can help an organization to scope budget and put together a more informed plan to address accessibility

15. Benefits – Address Low Hanging Fruit • Although automated testing alone does not cover all accessibility requirements, it does identify many violations • Results typically identify issues which are highly impactive on overall site compliance, but require minimal effort to fix • Resolving accessibility violations detected from automated testing alone can greatly increase the overall compliance level of a site • For sites developed without accessibility in mind, automated testing results are beneficial in achieving a baseline level of accessibility

16. What is the Best Solution for Your Organization? Supports the specific accessibility standards relevant to your organization Allows for the modification and/or extension of automated testing logic Provides clear and easy to understand manual testing guidance Delivers accurate data rich reports which can be utilized by all members of your organization

17. What is the Best Solution for Your Organization? (cont.) User friendly with supporting documentation and training Offers a variety of testing workflows Offers the ability to perform continuous monitoring Stays up to date with changes in accessibility laws and guidelines Keeps up with rapid changes in technology Offers seamless integration into current organization development processes and existing applications

18. AMP Demonstration

19. Conclusion Automated accessibility testing solutions play an important role within an organization’s accessibility initiative Leveraging effective automated solutions can result in increased productivity, efficiency and accuracy of results An organization generally should not rely solely on the use of an automated tool End user training is critical for successful deployment Educate users on all accessibility testing requirements

20. Questions?

21. Thank You Contact Us Chetan Bakhru Assistive Technology Tester chetan.bakhru@ssbbartgroup.com Matt Baillargeon Application Solutions Engineer matt.baillargeon@ssbbartgroup.com SSB Contact Information info@ssbbartgroup.com (800) 889-9659 Follow Us Twitter @SSBBARTGroup LinkedIn www.linkedin.com/company/ssb-bart-group Facebook www.facebook.com/ssbbartgroup Blog www.ssbbartgroup.com/blog