Wednesday, August 28, 2013

My own usability test

To demonstrate the formal usability test, I reviewed three common open source software projects. I solicited advice here and on other forums, asking which open source software programs to test. Sorting through the suggestions, three open source software projects met the criteria for the usability test:

  1. Gedit (a text editor)
  2. Firefox (a web browser)
  3. Nautilus (a file manager)


I didn't ask for a specific level of technological expertise, as I was looking for typical users with average knowledge. In most formal usability tests, it’s common to give each tester a small gratuity; I gave out free pop and candy.

Seven people volunteered for my usability test. They ranged in age from about 20 to about 40, three men and four women. Most claimed “low” experience with computers, and almost all used Windows as their primary desktop. Each tester started with a fresh instance to test against: their own account on a laptop running Fedora Linux. Since everyone had a separate account, they started from the same “default” settings.

At the start of the usability test, I gave each tester a brief background. I explained that this was a usability test, so it was about the software, not about them. If the tester experienced problems during the test, that was okay and we could move on to the next section. I was there to observe their interaction with the software, not to judge. I would take notes and watch what was happening on their screen.

I also asked each tester to speak aloud what was going through their mind during the usability test. For example, if they were looking for a Print button, they should simply say, “I’m looking for a Print button.” And I encouraged them to track the mouse on the screen with their eyes so I could observe where they were trying to find menus and buttons.

During the usability test, I presented the testers with a number of scenarios, each providing a brief context and an action they were asked to complete. For example, after asking testers to navigate to the BBC News website in the Firefox browser, one scenario asked them to increase the font size for the site. Importantly, the scenario does not use the same wording as the menu action that applies the font change:
You don't have your glasses with you, so it's hard to read the text on the BBC News website. Please make the text bigger on the BBC News website.

Overall, the usability test included nineteen scenarios, which testers completed in 30-40 minutes.

So, what were the usability issues? Interestingly, almost everyone experienced the same four issues:

  1. Change the default font in Gedit.
  2. Change the default colors in Gedit.
  3. Create a bookmark or shortcut to a folder in Nautilus.
  4. Search for a file in Nautilus.

In Gedit, testers were very confused about how to set the default font and colors. Part of this confusion stemmed from thinking of the editor as if it were a word processor such as Microsoft Word, which uses icons on the toolbar to accomplish either action.

In Nautilus, testers became frustrated while trying to create a bookmark or shortcut to a folder (used for a project that collected photos, and was saved under the Pictures folder). The only user who successfully created the bookmark only did so by accident.

Similarly, most testers found searching for a file in Nautilus a difficult task. They did not realize that the Search function starts from the current folder, not from their Home directory. Only two testers were able to use Search successfully. Of these, one happened to click on the Search button from their home directory. The other tried changing options in the drop-down Search action until they eventually picked a combination that worked. One tester gave up with Search and simply navigated into each folder in turn to find the file they were looking for.

And while GNOME was not part of the usability test, almost all testers experienced difficulty with the GNOME “Activities” menu hot corner. In the GNOME desktop environment, the “Activities” menu shows the list of available programs plus a view of the running applications. Users can bring up the “Activities” by clicking the menu icon in the upper left corner of the desktop, or simply by moving the mouse into that corner (the “hot corner”). Usually right away in the first scenario, testers would “overshoot” the program menu they were looking for, and hit the GNOME hot corner. Although testers were able to recover from the hot corner, it caused frequent disruption throughout the test.

What worked well for usability? Throughout the test, I observed four themes of good usability that allowed all testers to quickly pass through those parts of the usability test:

1. Familiarity
Testers commented that the programs seemed to operate more or less like their counterparts in Windows or MacOS X. For example, Gedit isn't very different from Windows Notepad, or even Microsoft Word. Firefox looks like other web browsers. Nautilus is quite similar to Windows Explorer or MacOS X Finder. To some extent, these testers had been “trained” under Windows or MacOS X, so having functionality (and paths to that functionality) that was approximately equivalent to the Windows or MacOS X experience was an important part of their success.

2. Consistency
User interface consistency between the programs worked strongly in favor of the testers, and was a recurring theme for good usability. Right-click worked in all programs to bring up a context-sensitive menu. Programs looked and acted the same, so testers didn't have to “re-learn” how to use the next program. While the toolbars differed, all programs shared a familiar menu system that featured File, Edit, View, and Help.

3. Menus
Testers preferred to access the programs’ functionality from the menus rather than via “hotkeys” or icons on the toolbar. For example, the only toolbar icon that testers used in the Gedit scenarios was the Save button. For other scenarios, testers used the drop-down menus such as File, Edit, View, and Tools.

4. Obviousness
When an action produced an obvious result, or clearly indicated success—such as saving a file in the editor, creating a folder in the file manager, opening a new tab in the web browser—testers were able to quickly move through the scenarios. Where an action did not produce obvious feedback, the testers tended to become confused. The contrast was evident when trying to create a bookmark or shortcut in the Nautilus file manager. In this case, Nautilus did not provide feedback, failing to indicate whether or not the bookmark had been created, so testers were unsure if they had successfully completed the activity.

These are good lessons in open source software and usability. Your program's user interface doesn't have to be a beautiful impediment to understanding. Instead, leverage existing user interface paradigms: Be consistent with other programs on the same platform, whether other open source software or proprietary software. Use menus that are clearly labelled. Ensure every action has a result that is obvious to the end user, especially if that result indicates failure.

No comments:

Post a Comment

Note: Only a member of this blog may post a comment.