Monday, April 7, 2014

Comparing my predictions

Before starting the usability test, I made some predictions for how things would go, what themes I would uncover, and how users would respond to the tasks. Here is a quick comparison of my predictions to what actually happened:

The 4 (now 5) themes will continue
I definitely saw a continuation of the 5 themes: Familiarity, Consistency, Menus, Obviousness, and Unexpected Behavior. The "hot corner" remained an issue for several testers, although this wasn't as prevalent as my previous usability test. Fortunately, when participants experienced the hot corner problem, most were able to quickly get back to the program before I stepped in to do it for them. But in general, this unexpected behavior was confusing and startling. 
Consistency was an important theme. Testers frequently said that because all programs looked more or less the same, they were able to apply what they learned in one program to the other programs. But a challenge was Familiarity; GNOME just isn't very much like Mac or Windows, so everyone had to re-learn how to use GNOME. 
Obviousness was also important, and in fact some testers thought that they hadn't completed an action because GNOME didn't give a very obvious indication that their action had an effect. A specific example is find and replace in Gedit. Menus were also an issue, but I'll talk more about that next.
Users will be confused about menus
While testers were confused about having menus split across two different "areas" (the Gear menu and the Application menu) they weren't as confused as I thought they would be. Several participants shared comments that the Gear menu (which they called "Options" or some similar name) seemed to have "program actions" and the Application menu had more "top level" actions. But overall, I'd say that users were confused about split menus.
Users will think the test didn't go well, even if they were able to complete all the tasks
I thought this would be the case, and I made a big deal out of it in my predictions, citing rhetoric principles. But it turns out that users thought the overall test went well if the last program went well. (This is an example of primacy-recency, a different principle of rhetoric.)
Specific problems with applications and tasks
I said this would be exhibited by specific problems with applications and tasks, such as: changing the default font in Gedit and Notes, creating a shortcut to a folder in Nautilus, searching for a file that's not under the current folder tree, bookmarking a website in Web, increasing the font size in Web, and deleting all notes in Notes. And for the most part, I was right. Look at the heat map to see how testers fared in these tasks. 
"Hot" areas include: changing the default font in Gedit and Notes (as predicted) and creating a shortcut to a folder in Nautilus (as predicted). Other less critical (but still important) usability issues include: find and replace in Gedit, bookmarking a website in Web (as predicted), increasing the font size in Web, and installing programs in Software. 
While I thought users would have a problem with Selection Mode in Notes, users didn't experience problems here. But importantly, neither did they use Selection Mode. When asked to delete all notes, testers invariably right-clicked a note, then clicked the Delete button - then repeated the process for the second note. Testers didn't seem to see a need to use Selection Mode to delete multiple notes.
Problems caused by possible bugs
Fortunately, I didn't experience any unexpected bugs. One problem I knew about beforehand is that if you install a program (Robots) using Software, then immediately try to remove the same program, you get an error. You seem to have to exit the Software program after installing a program, then re-launch Software before you can remove it. That seems like a program bug. As a workaround, I asked testers to exit the Software program before they tried to delete the Robots game, so we avoided that bug in Software.

1 comment:

  1. great work thank you so much for sharing this blog great work.