I wanted to share some thoughts about this usability project, and how I would apply it outside of open source software. The project was very interesting and insightful about how to improve the usability of open source software, but usability methods don't just apply to software.
At work, I am the Director of Information Technology for a small liberal arts university. As part of my role, I often interact with students and faculty. For example, last year we held several "listening sessions" to solicit direct feedback from our customers about the level of service we currently provide, and where we can improve. These listening sessions aren't very different in concept to usability testing. In usability testing, you gather input from your target audience, and use that input to inform a process.
Where usability methods are used in software, that process is the software development lifecycle to bring new improvements to the software. Where the same methods are used in listening sessions, that process drives improvements in the organization. These are strongly parallel.
Going forward, I now recognize you don't need very many people to provide effective and useful feedback. In our listening sessions, we were concerned about attracting a "sufficient" number of folks to the sessions to garner useful feedback. But now I know from Jakob Nielsen, it turns out 5 testers is usually enough to get you there, if you're just interested in finding the core problems for a product. Per Nielsen: "If you want a single number, the answer is simple: test 5 users in a usability study. This lets you find almost as many usability problems as you'd find using many more test participants."
And in retrospect, that rule would have applied equally well to our listening sessions. We started the sessions with 20-40 people, depending on attendance, and asked participants to identify three to four ideas that would raise the level of technology on campus. But from each roomful of contributors, we ended up with about a dozen unique ideas. There was quite a lot of overlap. Five participants at three or four ideas each would have generated 15-20 ideas. So yes, Nielsen seems to be correct, even for this different process: "Test 5 users in a usability study. This lets you find almost as many usability problems as you'd find using many more test participants."
That's a key takeaway from this usability study that can be applied to non-usability areas: it can be statistically valid to use a smaller sample size. Of course, if you have the larger group available and the resources to test them, certainly do so. But if you can only scrape together a few participants, you can get by with as few as five interested testers.