Archived
Tips For Conducting Usability Studies With Participants With Disabilities
Over the last few years, I ran several usability studies with participants with various disabilities. I thought it would help others if I shared some of my experiences.
In this article, I provide lessons learned or tips to consider in planning and executing usability testing with participants with disabilities. The lessons learned are divided into general that can apply to all types of disabilities; and lessons learned for three specific disability categories: visual, motor, and cognitive. These tips will help you regardless where you work: If you work with an established user research team where usability testing is part of your design process or if you work on your own with limited resources but want to improve the quality of the user research by expanding the diversity of participants.
Windows 10 high contrast mode of Google.com. (Large preview)
Background
Several of our clients from a state government agency to several fortune 500 companies came to us at the User Experience Center (UXC) for help with their websites. They wanted to make sure users with disabilities could access their site and accomplish their goals.
There are many different kinds of disabilities, however, there is a general agreement to categorize people with disability into four general categories: visual, auditory, motor (also referred to as “physical”), and cognitive. There are different conditions and much variability within each category, e.g., within visual disabilities, color blindness, low vision, and blindness. There is also a distinction as to when a disability is contracted, e.g., a person who was born blind as opposed to one who lost vision later on in life.
Getting the process just right ain’t an easy task. That’s why we’ve set up ‘this-is-how-I-work’-sessions — with smart cookies sharing what works really well for them. A part of the Smashing Membership, of course.
Explore features →
Furthermore, as we age or encounter unique situations (such as multi-tasking), we may have a similar experience to people we think of as disabled. Therefore, disabilities should be thought of as a spectrum of abilities that should be accounted for during the design of all user interfaces and experiences.
Typically, in order to ensure that disabled people can use their digital products and services, companies aim for compliance with accessibility guidelines such as the Web Content Accessibility Guidelines (WCAG 2.0). While this is critical, it is also important to have users with disabilities try to accomplish real tasks on the site in usability testing. There may be gaps in the overall user experience…
Think about the typical doors found in buildings. How many times have you tried to open a door one way and realized they actually open the other, for example, push instead of pull. Technically the door is accessible, but it is usable?
Example of doors that are technically accessible, but not usable (handles give impression you should push but must pull to open). (Large preview)
Even if a site follows the accessibility guidelines and is technically accessible, users may not be able to accomplish their goals on the site.
Lesson Learned
In most ways, usability testing with this segment of the population is no different than testing with anyone else. However, there are several areas you need to pay just a bit more attention to so your sessions run smoothly. The lessons or tips are broken down into general ones that can apply to all participants and specific tips for various disability types such as visual, motor, and cognitive.
General Lessons Learned
1. Ensure a baseline level of accessibility before usability testing
Ensure a baseline level of accessibility before usability testing: Planning usability testing, especially recruiting participants can take time both for the project team and the recruited participants.
Two good examples of basic accessibility issues that should be addressed prior to usability testing are:
- Missing alternative (alt) text.
Usability testing can be used to see if the alt text used is appropriate and makes sense to participants, but if all the participants are doing is confirming that the alt text is missing then this is not a good use of their time. - Appropriate color contrast.
All page designs should be reviewed beforehand to make sure all foreground and background colors meet WCAG 2.0 AA color contrast ratios.
2. Focus the recruiting strategy
If you work with an external recruiter ask them if they have experience recruiting people with disabilities; some do. If you are recruiting internally (without an external recruiter), you may need to reach out to organizations that have access to people with disabilities. For example, if you need to recruit participants with visually disabilities in the United States, you should contact a local chapter of the National Federation of the Blind (https://nfb.org/state-and-local-organizations) or a local training center such as the Carroll Center for the Blind in Massachusetts (http://carroll.org/). If you use social media to advertise your study, a good approach is to use the hashtag #a11y (stands for accessibility — there are 11 letters between the “a” and “y”) in your post.
3. Bring their own equipment/assistive technology
Allow and encourage participants to bring their own equipment such as their own laptop, especially if they use assistive technology. This way, you can truly see how people customize and use assistive technology.
4. Have a backup plan for assistive technology
As stated above in #3. It is best if participants can bring their own equipment. However, it is always wise to plan for the worst, for example, if a participant does not bring their equipment or if there is a technical problem such as you can’t connect their equipment to your Wi-Fi network. In the case of visually impaired participants, install assistive technology (AT) such as screen reader software they will be bringing in on a backup PC. For many of the AT software packages, you can get a free trail that should cover you for the usability testing period. This has saved us several times. Even though the configuration was different than what the participants had, we were able to run the session. Participants were quickly able to go into the settings and make some adjustments (e.g., increase the speech rate) and get started with the session.
5. Allow additional time
Provide additional time in-between sessions. Typically we like to reserve 30 minutes between participants. However, when participants plan to bring in their own equipment additional time may be required for setting up and resolving any issues that may arise. When testing with individuals with disabilities, we schedule an hour between sessions, so we have extra time for setting up assistive technology and testing it.
6. Confirm participant needs
Either with the recruiting screener or via email or telephone, confirm what equipment participants will bring in and need to be supplied beforehand. In our lab, we can connect external laptops (that in this case, were outfitted with special accessibility software and settings) to our 1Beyond system via an HDMI cable. In a recent study, all of our participants’ laptops had HDMI ports. However, we forgot to check this beforehand. This is an example of a small but important thing to check to prevent show-stopping issues at the time of the test.
7. Consider additional cost
Depending on the disability type transportation to the usability testing location may add additional burden or cost. Consider the cost of transportation in the incentive amount. If feasible, consider providing an extra $25-$40 in your incentive amount so participants can take a taxi/Uber/Lyft, etc. to and from your location. Depending on access to public transportation and taxi/ride-sharing rates in your area the amount may vary. Our participants came to the UXC in different ways — some more reliable and timely than others.
8. Revise directions
Check the directions you provide for accessibility. Make sure they include an accessible path into your building. Test them out beforehand. Do you need to provide additional signage? If so, ensure all signs are clear, concise, and use plain-language directions.
9. Review the emergency evacuation plan
Review the plan in the event of a fire or other emergency. Map out the emergency evacuation plan in advance.
10. Consider logistics
Consider remote usability testing as an option. One of the benefits of bringing individuals with disabilities into the lab for usability testing is observing first-hand participants’ use of the product or website in question. However, the logistics of getting to your location may be just too much for participants. If it’s possible to test remotely (we typically do this through Zoom or GoToMeeting), it should be considered. This poses the additional challenge of making sure your process for capturing the remote session is compatible with all of the participant’s assistive technology, as well as accessible itself. Troubleshooting remotely is never fun and could be more difficult with this segment of the population.
11. Hearing impaired participants
Some participants may have a hearing impairment where the position of the moderator and participant is critical for adequate communication. In the case of hearing-impaired participants, it is important to get their attention before talking to them and also to take turns when engaging in conversation.
To get the most of this research, it is best if the participants are not discovering basic accessibility issues that should have been discovered during an accessibility review and/or testing.
Lessons Learned For Participants With Visual Disabilities
Participants with visual disabilities range from people who are blind and use screen readers such as JAWS, to people that need to the text or the screen to be enlarged using software such as ZoomText or relying on the native screen enlargement in the browser. People that are color-blind also fall into this category.
- For any documents needed prior to the study such as the consent form, send via email beforehand and ask them review and send back in lieu of a physical signature. If you don’t, be prepared to read aloud the consent form and assist in signing the documents for some participants.
- Make sure directions provide step-wise directions; do not rely only on graphical maps as these may not be accessible.
- For all documents, make sure color is not used as the sole cue to convey information. Print out all documents on a black and white printer to make sure color is not required to make sense of the information.
- Get participants mobile phone numbers in advance and meet them at their drop-off point. Be prepared to guide them to the testing location. Review best practice for guiding blind individuals:
- While Braille documents can be helpful for participants that read Braille, the time and cost involved may not be feasible. Furthermore, all blind people do not read Braille, especially people that have lost sight later in life. It is best to make sure all documents can be read via a screen reader. Unless you are sure if there are no accessibility issues avoid PDF documents and send out simple Word documents or text-based emails.
- If participants bring guide dogs do not treat them as pets, they are working. Provide space for the dog and do not pet it unless the participant gives you permission.
- Make sure to explain beforehand any sounds or noise that are or may be present in the room such as unique audio from recording software. This may avoid the participant from becoming startled or confused during the session.
- Initially when I started to work with blind participants I was worried my choice in language might offend. However, over the years I have learned that most blind participants are fairly relaxed when it comes to speech. Therefore, during moderation do not be afraid to use phrases such as “see” or “look” and similar words when talking to blind participants; for example, “please take a look at the bottom of the page” or “what do you see in the navigation menu?” In my experience, blind participants will not be offended and will understand the figurative meaning rather than the literal meaning.
- Test out all recording equipment/processes beforehand. Ensure all audio including both human speech in the room and audio/speech from AT such as screen readers will be recorded correctly. During testing of the equipment adjust the locations of the microphones for optimal recording.
Lessons Learned For Participants With Motor Disabilities
Motor disabilities refer to disabilities that affect the use of arms or legs and mobility. These individuals may need to use a wheelchair. Some people may not have full use of their hands or arms and cannot use a standard mouse and keyboard. These people may need to voice recognition software which allows to use voice input or use a special pointing device, for example, one that is controlled by their mouth.
- In the directions, make sure the route is accessible and routes them via elevators rather than stairs. Also, if participants are driving note the location of accessible parking.
- Note if doors have accessible door controls. If not you may need to meet the participant and guide them to the testing location.
- Make a note of the nearest accessible restrooms to the testing location.
- As with all participants with disabilities, it is best if they can bring in their own laptop with their assistive technology software installed and any other required assistive technology. However, in the case of participants (such as Adriana in Figure 3) that use voice recognition software such as Dragon Naturally Speaking this is critical because they have trained the software to recognize their voice.
- Make sure the desk or table where the participant will be working can accommodate a wheelchair and the height is adjustable. According to the American with Disabilities Act (ADA), conference tables must be 27 inches high in order to accommodate knee clearance for individuals in wheelchairs..
Adriana Mallozzi conducting a usability test at the User Experience Center. Adriana has a motor disability (cerebral palsy) which affects her entire body. She uses a wheelchair, a sip-and-puff joystick that acts as a mouse, along with Dragon Naturally Speaking. (Large preview)
Lessons Learned For Participants With Cognitive Disabilities
Individuals with these disabilities cover a wide range of relatively mild learning disabilities such as Dyslexia to individuals with a more profound cognitive disability such as Down syndrome. In general, people with cognitive disabilities have challenges with one or more mental tasks. Rather than looking at specific clinical definitions it best to consider functional limitations in key areas such as memory, problem-solving, attention, reading or verbal compensation. Consider how best to accommodate participants during usability testing. Many of the tips below should also apply to all participants, however for this group you need to be extra aware.
- Sometimes participants will be accompanied by a caretaker or an aide. This person may assist with transportation or may need to be present with the participant during the usability test. If the caretaker is present during the usability test, make sure they understand the structure of the usability test and what will be required of the participant. If you know the participant will be accompanied before the study, you review the goals and protocol prior to arrival via email or phone. That is as much as possible the participant should be one conducting the usability testing, and the caretaker should not be involved unless it is completely necessary.
- In some cases, the caretaker or aide may act like an interpreter. You may need to communicate with this interpreter in order to communicate with the participant. If this is the case, make sure you record the audio coming from both the participant and the interpreter.
- Provide instructions in multiple modalities, for example, both written and verbal. Be patient and be prepared to repeat the task or ask the same question multiple times.
- Be prepared to break tasks into smaller sub-tasks to support memory/attention challenges or fatigue that may set in.
- Ideally, it is best to be consistent with tasks for all participants however for some participants with cognitive disabilities you should be prepared to go off-script or modify tasks on the fly if the current approach is not working.
- Have the participant’s comfort and well-being the number one priority at all times. Don’t be afraid to take multiple breaks or end the session early if things are just not working out or the participant is not comfortable.
Amazon.com home page zoomed in at 250%. (Large preview)
Additional Insights
The tips above should serve as guidelines. Each participant is unique and may require various accommodations depending on their situation. Furthermore, while some of the tips are categorized for specific disability types, specific individuals may have multiple disabilities and/or benefit from a tip from a different category than their primary disability.
If you or your company have conducted user or customer research, you know the value of gathering feedback about the issues and benefits of products and systems. Testing with individuals with disabilities is no different, as you learn many insights that you would not gain otherwise. However, an additional takeaway for us was the realization that people use assistive technologies in different ways. The following example is specific to people with visual disabilities, but there are similar examples across all groups.
An assumption might be someone that is blind only uses a screen reader such as JAWS and is an expert at it. We found that people with visual impairments actually differ greatly in the level of support needed from assistive technology.
- Some users need a screen reader for accessing all content.
- Some users (with more sight/with low vision) only need to enlarge content or invert page colors to increase contrast.
- Others may need a combination of approaches. One visually impaired participant used both a screen reader along with the zoom function embedded in the web browser. She only used a screen reader for large paragraphs of text, but otherwise simply zoomed in with the web browser and got very close to the screen when navigating around the website.
Furthermore, just like anyone, all users are not experts on the software they use. While some users would consider themselves experts, some only learn enough about the software to accomplish what they need and no more.
Moving Forward
Hopefully you have learned some useful information that will help you include more diversity into your usability testing. However, since there is variability with different disabilities, this may seem overwhelming. I recommend starting small; for example by including one or two participants with disabilities as part of a larger group of 5 to 10 participants. In addition, initially bring in someone that has both experience with usability testing and a lot of experience with their assistive technology so you can focus on getting their feedback rather than how the usability testing process works or their use of their assistive technology.
Acknowledgements
I would like to thank Jocelyn Bellas, UX Researcher at Bank of America and Rachel Graham, UX Researcher at Amazon. When Rachel and Jocelyn worked at the User Experience Center as Research Associates in 2016, they worked with me on some of the projects referenced in this article and also contributed to a related blog post on this topic.
References
- “Conducting Usability Testing,” Henry, S.L. (2007), Just Ask: Integrating Accessibility Throughout Design
- “A Web for Everyone: Designing Accessible User Experiences,” Horton S., Quesenbery W. (2013), Rosenfeld Media
- “ICT and People with Cognitive Disabilities: Variations in Assistive Technology,” Lewis, User Experience Magazine
- “Why Usability Testing Should be Part of your Accessibility Testing Strategy,” McNally P. (2017, October), The 2017 ICT Accessibility Testing Symposium: Automated & Manual Testing
- “Usability Studies and Participants with Disabilities: What you Need to Know,” McNally P., Graham R., Bellas J. (2016, August)
- “The Design of Everyday Things,” Norman D. (1988), Basic Books
- “Handbook of Usability Testing,” Rubin J. (1994), John Wiley & Sons
(cc, ra, yk, il)
From our sponsors: Tips For Conducting Usability Studies With Participants With Disabilities
