Monday, March 16, 2015

Preview of GNOME usability results

I have been mentoring Sanskriti Dawle as part of the GNOME Outreach Program for Women. Sanskriti has been working on a usability test of GNOME, an update from my own usability testing which I also shared at GUADEC 2014.

I encourage you to watch Sanskriti's blog for the final results, but I wanted to share a view into her excellent work. You might treat this as a preview of Sanskriti's results.

Sanskriti's usability test included about equal men and women, and about equally divided between "high" and "low" mobile OS exposure (but none at "moderate," which is interesting). Most were Windows OS users, and the testers were equally distributed in expertise. Her test participants had a good distribution in age groups. Click to view a larger version of each chart:

This is important, because GNOME wants to be useful to what I call "average users with average experience." And Sanskriti's usability test participants represent that.

These testers required an average of 38 minutes (±11 minutes) to do all scenario tasks in the full usability test. The median test time was 37.5 minutes. Click to see a larger version of the chart:

I asked Sanskriti to use the heat map method to display her usability test results. This is a method I developed during my master's program research, and refined in later usability testing. In a usability heat map, each scenario task in the usability test is displayed in rows, and each tester is shown as columns. Each tester's experience for every scenario task is represented using a colored block: green if the tester completed this task successfully with little or no problems, orange if the tester experienced some problems but was still able to complete the task successfully, red if the tester encountered great difficulty but still completed the task. And black if the tester was completely unable to finish the task.

Sanskriti's heap map shows some very useful and interesting results. Click to view a larger version of Sanskriti's data:

I can make a few initial observations from this data. Looks like testers had the most difficulty with tasks Gedit.6 and Photos.3 and Photos.4, with noticeable difficulty in tasks Notes.1 and Photos.2. There's some interesting data around tasks Gedit.1 and Music.1 that might reflect testers 9, 11, and 12.

This is only a preview of Sanskriti's results. I encourage you to watch Sanskriti's blog for the final results, which I hope to see in the next week as she wraps up her work in the internship.
image: Outreach Program for Women


  1. May I suggest getting a new and more appealing default icon set? The default brown icons, in my opinion, are quite ugly.

  2. What do you mean by brown icons? The GNOME Nautilus icons are probably what you mean, but they're more gold than brown. What distro are you running? Ubumtu has a brown theme, I think.

  3. I don't think that "The default brown icons, in my opinion, are quite ugly" will really affect the usability much.

    1. They can affect the ability to understand and find the needed program actions. It can also cause people to feel intimidated and that might prevent them from exploring a program.

    2. The "brown" icons can influence the perception of usability, but I think the icons clearly represent what they do. So the usability is good, but maybe the "brown" color is not so great.

      This is an example that Usability and "User Experience" (XP) are two different concepts that are often confused with one another. In short: Usability is how easy a user interface is for a user to accomplish his or her goals quickly, easily, and efficiently. User Experience (XP) is about the overall emotional engagement between a user and the program.

      For most cases, usability and user experience are aligned: programs with good usability tend to have good XP, programs with poor usability tend to have poor XP. However, it is possible to have a program with good usability but poor XP, and poor usability but good XP.

      For example, a program that was really easy to use (clearly defined menus, etc) would have good usability .. but if it used a "red text on black background" for the user interface, users would probably report that the program just didn't "feel" right. They could do all their tasks, but they would likely say they didn't have a good experience while using the program.

      On the other hand, a program might be difficult to use and/or difficult to learn, but have a positive XP. These are probably less likely, but one example category is games: the game is fun to play once you figure out how to move around. For me, the HedgeWars game is fun to play, but hell to learn. I never remember how to change weapons or move around. I had a blast while playing it (good XP) but the game didn't do anything to help me play it (poor usability). I was pretty much at the mercy of the AI, but it was cute and I enjoyed it.

  4. Missing Context menu icons...

  5. HeavensRevenge's comment on colors is an important observation. Colors are an important part of the user experience, and I've commented on this in another post from last year: What you think about desktop colors. The colors used in a desktop environment can influence your perception of usability. It's no surprise that Apple's MacOS desktop uses bright, cheerful colors. My other post discusses the dark, moody colors used in GNOME 3. My recommendation is to update the colors to use a brighter theme.

    From my other post:

    This may be why users at large perceive the GNOME desktop to have poor usability, despite usability testing that demonstrates GNOME actually has pretty good usability. The dark, moody colors used in GNOME provoke feelings of tension and insecurity. Bang(1991) included color in her analysis of how people perceive images, mentioning that white or light colors feel "safer" than darker colors, because we can see well in the day but not at night. This is especially true of background colors. Of all the "dark" colors, the top associations were forest, brick, deep, and night. Also, the top associations with "black" were dark, night, darkness, and deep.

    Open source software projects can learn from these color associations. GNOME currently uses a dark color scheme. These colors correlate to open, calm, and moody attributes. Meanwhile, the lighter colors used throughout MacOS X provide affiliations with open, soft, and calm adjectives. It is not surprising that users typically view MacOS X as having stronger usability. Certainly, Apple has put a lot of effort into usability testing in MacOS X, but I believe their choice of light colors was a conscious one. Even people who do not use MacOS X as their primary desktop describe MacOS X as easy to use.

    I recommend that GNOME adopt lighter colors in future releases. Avoid dark colors, especially in backgrounds or the desktop wallpaper. Embrace light, airy colors in the interface instead of somber, melancholy color tones.