Monday, April 27, 2015

A brief glimpse into usability testing

I am mentoring in GNOME Outreachy this summer. As part of the Outreachy application process, we ask applicants to make an initial contribution to their project of interest. For usability testing, I ask applicants to read my article about usability testing, and conduct a 1-person usability test of their own. One of the applicants (Gina Dobrescu) provided a very complete analysis in her 1-person test. While this is only a 1-person test, so does not generate actionable results, I wanted to share her results on my blog as a "guest post," as an example for future applicants. This is from Gina, posted with her permission:
This provides the description and results of my 1-person usability test for GNOME version 3.14.2. I set up the test environment on a virtual machine running Debian Jessie, and conducted the usability test with only one participant on February 21, 2015.

When doing usability testing, I strongly believe that understanding the participants is a very important point that we shouldn't ignore. Here, the tester was a 23 year old female student in computer science, who self-reported a medium level of computer expertise.

Test objectives & context

The principal objective of my 1-person usability test was to identify areas in GNOME interface where users may encounter difficulties. Other objectives of this test were to determine if:
  • The proposed tasks can be successfully accomplished
  • The proposed tasks were clear for the user
  • The given explanations clear enough for the user

Test Method

As the participant was not previously familiar with GNOME, I started with an explanation of what GNOME was, and informed the participant about this usability test's purpose. In a usability test, we want to evaluate the user interface and not the tester's individual skills.

Before starting the test, I set some rules, asking the participant to "speak out loud" when doing a task. The participant also agreed to be recorded during the test, and was assured that the data was collected anonymously.

For the usability test, I gave the participant a single sheet of paper containing a list of 23 usability test scenario tasks. The duration of the test was about 30 minutes, including the introduction.

The test results

In order to evaluate the usability test results, I used Jim Hall's heat map technique, presented below. In this heat map, you can see that the participant was able to complete most of the tasks without much difficulty (green boxes), but she encountered some problems in accomplishing others (red boxes). Tasks that the tester was unable to complete are shown in black boxes.

Heat map

The tester was easily able to search for an image in the system and change the screen and the lock screen background. The same happened with the tasks related to the browser, as the tester was already familiar with Mozilla Firefox. The tester was also easily able to play a song and repeat a song in the Music application, the tasks were also quickly and correctly accomplished.

However, during the test, some usability issues were revealed. The tester was not able to complete some tasks like bookmarking a page in a PDF document, or change the name of a bookmark. When testing gedit, the tester was unable to display line numbers in the file or to change the letters of a word to upper-case.

Furthermore, one of most difficult encountered issues was related to Nautilus, where the participant took a long time to discover the option to "bookmark this location". After this option was selected, the tester was unable to locate where the bookmark was saved. Indeed, when her attention was in the top-right corner in order to select the "bookmark this location" option, she did not see that the folder was added on the left-side pane in Nautilus(see screenshot).

Bookmarking a location

Package installation

Another problem that the participant struggled with was the installation of the gparted program. Her first guess was to search for the keyword "install" in the GNOME search bar, but the displayed programs names were not explicit enough to convince her that she was following the right procedure. At first, she started the Synaptic package manager application, but then reacted with "no, this is not the right program". Then, she launched the Packages application, searched for "gparted" program and clicked on "Install Package", but after the button name changed into "Remove package" she thought that the program was installed. When she was asked to verify if the program was installed, she could not find it in the system. It was only after a closer look in the Packages application that she realized that she missed the "Apply settings" button, in the right corner of the window (see screenshot).


For her first interaction with GNOME environment, the tester successfully accomplished most of the basic tasks. However, I believe some of the difficulties in fulfilling the other tasks were due to the user's non-familiarity with GNOME.

Reflecting on the usability test itself, I realized that using a virtual machine with limited performance could have an impact on the test quality. The next time I do usability testing, I plan to use a dedicated machine.

Another uncovered issue was due to my scenario task design: some tasks were not explained well enough, so the tester sometimes had to ask about what a task was prompting her to do. The next time I do usability testing, I will put more focus on the scenario task descriptions.

To improve the quality of future usability tests, I would plan to develop more scenario tasks. For each category of GNOME features, tasks could be classified by their level of difficulty and designed for various types of subjects (for example people familiar with GNOME, or people familiar with Linux but not with GNOME, etc).

This is an interesting first test, although usability testing with only one person makes it difficult to determine if a product like GNOME has good usability.

In conclusion, usability testing can help GNOME to resolve its issues, by using testers' feedback to improve its features. This kind of testing can also help us to understand users and their expectations in order to offer a better user experience.
A few minor edits by me, for style and grammar. -jh

Wednesday, April 1, 2015

ChromeOS getting an update

I was among the first to order when Google released the Chromebook, an ultra-portable low-cost laptop that instantly connects you to the Internet. The idea behind the Chromebook is that we don't really need to store things locally anymore. Instead, we use the Cloud for email, documents, collaboration, video, and pretty much everything we do. So the Chromebook's goal is to get you online as quickly and easily as possible, and connect you to those Cloud services. As suggested by the name, the Chromebook comes pre-loaded with Google's Chrome web browser.

Google's Chromebook uses a desktop environment called "Aura." It presents a somewhat simplified desktop experience, where all applications actually run inside the Chrome web browser. For example, clicking a "YouTube" icon actually launches in a new Chrome browser tab. Frequently-used programs can be added to the desktop, or you can browse other applications by clicking the Applications menu icon. Other desktop functions (clock, wireless network connections, battery indicator, etc) are displayed in the lower-right corner. While there is no "bar" or "panel" like in Windows or MacOSX, the Aura desktop appears to do the same by the way it displays icons and desktop functions.

It's no secret that I love the Aura desktop. I find it has good usability! My wife has a Chromebook and loves it. I use a Chromebook at work about half the time, and it's great. The Aura desktop has a simplified look that is both easy for new users and flexible for power users. Aura provides very good usability: The desktop is familiar to both Windows users and Mac users, and the desktop functions more or less like the desktops on these other platforms. Since (almost) everything in the Chromebook runs inside the browser, programs share consistent behavior. The Chrome designers have done a great job of making web apps easy to find and launch, and settings easy to search and update.

Overall, if the Aura desktop were available on a "stock" Linux distribution and had the ability to launch local programs like LibreOffice or a terminal, it would be a great desktop for most folks.

So I was particularly interested to read an article recently from Hot Hardware that ChromeOS is getting a makeover with material design and Google Now support. I really should update my Chromebook at work to use the Beta channel, so I can try it out.

As reported by Hot Hardware: "New in this version is Chrome Launcher 2.0, which gives you quick access to a number of common features, including the apps you use most often (examples are Hangouts, Calculator, and Files). Some apps have also received a fresh coat of paint, such as the file manager, seen below. Google notes that this is just the start, so there will be more updates rolling out to the beta OS as time goes on."

And here are the screenshots:

It's hard to make a heuristic usability evaluation based on two screenshots, but I think I can make a few fair comments here:

As always, I like the clarity in the presentation. Even though Aura doesn't use a traditional "menu bar" or "application bar" like GNOME's top bar or XFCE's launcher, it's easy to see how to launch different applications. The first screenshot shows a number of GNOME Apps icons, and I only have issues with a few of them (Google Calendar's icon looks like a shopping bag to me, for example). In the second screenshot, you can see icons for Chrome and Files. In the lower-right, you can find the clock, wireless status, and battery. Overall, the icons are clear and accurately represent the action.

I love the clearly defined window "tab" in the Files app. When the app is in "windowed" mode, it is a convenient place to "grab" the window to relocate it, and provides icons to minimize, maximize, and close the window using standard icons.

The Google Now screenshot provides only content, so I'll focus on the second screenshot instead. The Files app provides a basic menu bar that I find quite usable. There's a clearly define "breadcrumb" navigation trail in the upper-left, and the upper-right has a search menu and an application menu. I dislike the mixed use of the "three lines" menu with the "three dots" menu. I don't know what each of them does. But overall, this is very clear.

I also appreciate the bright, happy colors used in Aura. No dark, moody colors here. Aura's use of bright blues at the top of the windows would likely reflect feelings of "sky." Especially so because the blue is limited to the top of the screen.

But oh my—the wallpaper! The wallpaper uses muted colors, but is still garish. I don't know if I should blame Google for this (was it the default desktop wallpaper?) or the reviewer (did they pick this particular wallpaper?) but either way, I would have preferred a happier, calmer image.

I'm pleased to see this example of good software usability. Although Aura isn't open source software,My bad. Aura is open source. See comments, below. -jh it's important to note positive examples of usability so we can learn from them: what worked well, and what could be improved.
Chrome icon: Wikimedia
screenshots: Hot Hardware