Monday, April 27, 2015

A brief glimpse into usability testing

I am mentoring in GNOME Outreachy this summer. As part of the Outreachy application process, we ask applicants to make an initial contribution to their project of interest. For usability testing, I ask applicants to read my article about usability testing, and conduct a 1-person usability test of their own. One of the applicants (Gina Dobrescu) provided a very complete analysis in her 1-person test. While this is only a 1-person test, so does not generate actionable results, I wanted to share her results on my blog as a "guest post," as an example for future applicants. This is from Gina, posted with her permission:
This provides the description and results of my 1-person usability test for GNOME version 3.14.2. I set up the test environment on a virtual machine running Debian Jessie, and conducted the usability test with only one participant on February 21, 2015.

When doing usability testing, I strongly believe that understanding the participants is a very important point that we shouldn't ignore. Here, the tester was a 23 year old female student in computer science, who self-reported a medium level of computer expertise.

Test objectives & context

The principal objective of my 1-person usability test was to identify areas in GNOME interface where users may encounter difficulties. Other objectives of this test were to determine if:
  • The proposed tasks can be successfully accomplished
  • The proposed tasks were clear for the user
  • The given explanations clear enough for the user

Test Method

As the participant was not previously familiar with GNOME, I started with an explanation of what GNOME was, and informed the participant about this usability test's purpose. In a usability test, we want to evaluate the user interface and not the tester's individual skills.

Before starting the test, I set some rules, asking the participant to "speak out loud" when doing a task. The participant also agreed to be recorded during the test, and was assured that the data was collected anonymously.

For the usability test, I gave the participant a single sheet of paper containing a list of 23 usability test scenario tasks. The duration of the test was about 30 minutes, including the introduction.

The test results

In order to evaluate the usability test results, I used Jim Hall's heat map technique, presented below. In this heat map, you can see that the participant was able to complete most of the tasks without much difficulty (green boxes), but she encountered some problems in accomplishing others (red boxes). Tasks that the tester was unable to complete are shown in black boxes.

Heat map

The tester was easily able to search for an image in the system and change the screen and the lock screen background. The same happened with the tasks related to the browser, as the tester was already familiar with Mozilla Firefox. The tester was also easily able to play a song and repeat a song in the Music application, the tasks were also quickly and correctly accomplished.

However, during the test, some usability issues were revealed. The tester was not able to complete some tasks like bookmarking a page in a PDF document, or change the name of a bookmark. When testing gedit, the tester was unable to display line numbers in the file or to change the letters of a word to upper-case.

Furthermore, one of most difficult encountered issues was related to Nautilus, where the participant took a long time to discover the option to "bookmark this location". After this option was selected, the tester was unable to locate where the bookmark was saved. Indeed, when her attention was in the top-right corner in order to select the "bookmark this location" option, she did not see that the folder was added on the left-side pane in Nautilus(see screenshot).

Bookmarking a location

Package installation

Another problem that the participant struggled with was the installation of the gparted program. Her first guess was to search for the keyword "install" in the GNOME search bar, but the displayed programs names were not explicit enough to convince her that she was following the right procedure. At first, she started the Synaptic package manager application, but then reacted with "no, this is not the right program". Then, she launched the Packages application, searched for "gparted" program and clicked on "Install Package", but after the button name changed into "Remove package" she thought that the program was installed. When she was asked to verify if the program was installed, she could not find it in the system. It was only after a closer look in the Packages application that she realized that she missed the "Apply settings" button, in the right corner of the window (see screenshot).


For her first interaction with GNOME environment, the tester successfully accomplished most of the basic tasks. However, I believe some of the difficulties in fulfilling the other tasks were due to the user's non-familiarity with GNOME.

Reflecting on the usability test itself, I realized that using a virtual machine with limited performance could have an impact on the test quality. The next time I do usability testing, I plan to use a dedicated machine.

Another uncovered issue was due to my scenario task design: some tasks were not explained well enough, so the tester sometimes had to ask about what a task was prompting her to do. The next time I do usability testing, I will put more focus on the scenario task descriptions.

To improve the quality of future usability tests, I would plan to develop more scenario tasks. For each category of GNOME features, tasks could be classified by their level of difficulty and designed for various types of subjects (for example people familiar with GNOME, or people familiar with Linux but not with GNOME, etc).

This is an interesting first test, although usability testing with only one person makes it difficult to determine if a product like GNOME has good usability.

In conclusion, usability testing can help GNOME to resolve its issues, by using testers' feedback to improve its features. This kind of testing can also help us to understand users and their expectations in order to offer a better user experience.
A few minor edits by me, for style and grammar. -jh


  1. Quite a useful result, even for a one-person test. It's quite clear that the package management tool has some usability issues, and that Nautilus could use a "bookmark this location" action located near the existing bookmarks.

  2. I agree, this provides some interesting first-glance results. Gina and I will be working together during Outreachy, and it would be interesting to expand on this usability test as part of that.

  3. Nice work as usual. That's more about Debian usability then GNOME itself though, since we have software in place of synaptic and the core apps tested changed quite a bit in the latest iteration (3.16) trying to address issues spotted by your user testing even ;)


Note: Only a member of this blog may post a comment.