Thursday, November 29, 2012

How many testers do you need?

When I was planning my usability tests, I optimistically hoped for a dozen usability testers. But "from the discussions I've had with thought leaders in usability testing, I may not need as many as a dozen folks to get usability good results - so I'll be satisfied if I only attract half that (five or six)."

For my usability tests, ten people signed up, but three were "no-shows," leaving seven people who participated in the usability tests. And it was amazing how quickly those few testers were able to "zero in" to the usability problems and highlight positive trends for "what worked well."

From Jakob Nielsen, it turns out 5 testers is usually enough to get you there, if you're just interested in finding the usability problems for a product: "If you want a single number, the answer is simple: test 5 users in a usability study. This lets you find almost as many usability problems as you'd find using many more test participants."

If you want to dig deeper or search for other information, you may need more testers, however:

As with any human factors issue, however, there are exceptions:
  • Quantitative studies (aiming at statistics, not insights): Test at least 20 users to get statistically significant numbers; tight confidence intervals require even more users.
  • Card sorting: Test at least 15 users.
  • Eyetracking: Test 39 users if you want stable heatmaps.

I've commented before that it doesn't take much to pull together a few testers and ask them to run through test scenarios that represent how the product is used.  Interesting to note a comment I see repeated when I ask open source software folks about usability: "I'm not a usability expert." But you don't need to be a usability expert to apply usability to open source software. As Redish shared with me in our interview, you can learn a lot just by sitting down with a few users, and watching them use the software. By extension, you don't need to be a usability expert to grab a friend or family member, and ask them to try out a scenario for 15 minutes.

No comments:

Post a Comment