Tuesday, March 5, 2019

Outreachy GNOME usability testing wrap-up

The December-March cycle of Outreachy has finished, and I wanted to do a quick recap of the work from our intern, Clarissa.

As I mentioned when we started week 1 GNOME usability testing, most of our work in the internship was testing designs that haven't gone "live" yet (this is called "prototype usability testing"). Allan and Jakub created mock-ups of new designs, and Clarissa did usability testing on them.

That means a lot of the applications were still being worked on, and were often delivered in Flatpak format since these "in development" versions were not part of a systemwide release (which would have likely been included natively in a Linux distribution somewhere).

As a result, we ran this cycle of usability testing in a more "loose" fashion that we have in previous cycles. We didn't have a long run-up to usability testing, where we could take our time learning about usability testing and carefully constructing new scenario tasks. Rather, Allan provided an overview of what he was hoping to get out of each usability test (how do users respond to this new feature, do they interact with the user interface differently, etc?) and Clarissa had to quickly assemble scenario tasks based on that. I helped with focus and wording on the scenario tasks.

We didn't expect anyone in the internship to come with previous experience, but we needed someone who could learn quickly. We expected that the intern would "learn as you go" and Clarissa did a great job with that!

Clarissa ran three usability tests, which she describes in her Final internship report on her blog. The three tests were:
  1. A new GNOME Sound Settings design (see results)
  2. New designs for GNOME Files and GNOME Notes (see results)
  3. Updated design for GNOME gedit (see results)
In Clarissa's Final internship report, she also shares some thoughts and lessons learned from the internship. Please read Clarissa's blog for her takeaways, but I wanted to highlight this one:
A bigger number of testers does not always give you more precise results. When I applied to the internship, I tried to look for the bigger amount of volunteers as I could because I thought it would bring me better results and I would have a better contribution, and, consequently, I would be chosen for the internship (hehe :P). On the first week of the internship I studied with the help of some articles that Jim sent me and I discovered that the time you spend running tests with more testers than you actually need can be spent with writing results faster and giving the design team more time to work on new designs. I discovered that it was true on the first round, when I ran tests with 7 volunteers, when I needed only 5. I wrote about how many testers do we need here.
I've highlighted the important bit.

You don't need very many testers to get useful results. This is especially important if you are doing iterative testing: create a design, test it, tweak the design based on results, test it again, tweak the design again, test it again, final tweaks based on results. You can learn enough about the design by using only five testers. If each tester exercises about 31% of usability problems, after five testers you've uncovered 85% of the issue. That's enough to make changes to the design.

Doing more tests with more testers doesn't really get you much further down the road. Do each round of usability testing with about five testers and you'll be fine. And that's exactly what Clarissa found.

I'm very proud of Clarissa for her work in this cycle of Outreachy. She great work and provided many useful results that I'm sure the Design team will be able to use to tweak future designs. That's why we do usability testing.

On a personal note, it was great to stay involved in GNOME and be part of usability testing. I believe being a mentor is a valuable experience. If you haven't mentored an intern as part of your work in open source software, I encourage you to do so. Outreachy is a wonderful opportunity because it provides paid internships for women and other underrepresented groups to work in open source software.

But there are other ways to mentor someone. Find an outlet that works for you, and bring someone under your wing. By helping others get involved in open source software, we make the entire open source community stronger.

No comments:

Post a Comment