Sunday, March 30, 2014

Predictions for the usability test

An important aspect of experimentation is to test a theory. And while the purpose of a usability test is to examine how real users interact with a system, this is my second usability test and I would like to make my predictions for how users will respond to GNOME 3.12. To be clear, this is not meant as a negative criticism of GNOME, but simply my own heuristic review of how users will respond to the GNOME design patterns.

Let's lay some groundwork. From my previous usability test, I found four themes of successful usability in GNOME:

1 .Familiarity
Testers commented that the programs seemed to operate more or less like their counterparts in Windows or Mac. For example, Gedit isn't very different from Windows Notepad, or even Microsoft Word. Firefox looks like other web browsers. Nautilus is quite similar to Windows Explorer or Mac Finder. To some extent, these testers had been “trained” under Windows or Ma, so having functionality (and paths to that functionality) that was approximately equivalent to the Windows or Mac experience was an important part of their success.
2. Consistency
User interface consistency between the programs worked strongly in favor of the testers, and was a recurring theme for good usability. Right-click worked in all programs to bring up a context-sensitive menu. Programs looked and acted the same, so testers didn't have to “re-learn” how to use the next program. While the toolbars differed, all programs shared a familiar menu system that featured File, Edit, View, and Help.
3. Menus
Testers preferred to access the programs’ functionality from the menus rather than via “hotkeys” or icons on the toolbar. For example, the only toolbar icon that testers used in the Gedit scenarios was the Save button. For other scenarios, testers used the drop-down menus such as File, Edit, View, and Tools.
4. Obviousness
When an action produced an obvious result, or clearly indicated success—such as saving a file in the editor, creating a folder in the file manager, opening a new tab in the web browser—testers were able to quickly move through the scenarios. Where an action did not produce obvious feedback, the testers tended to become confused. The contrast was evident when trying to create a bookmark or shortcut in the Nautilus file manager. In this case, Nautilus did not provide feedback, failing to indicate whether or not the bookmark had been created, so testers were unsure if they had successfully completed the activity.
And while it wasn't part of the test, users experienced problems with the "Activities" hot corner. I can generalize this into a fifth theme:

5. Unexpected behavior
Those who experienced the "hot corner problem" typically did so right away in the first exercise when attempting to use the program menus. While testers were able to recover from the hot corner, it definitely caused disruption several times throughout the test. None of the users in the usability test had used GNOME before, so they did not have previous experience in using GNOME. The programs they were evaluating in the usability test were running "maximized" so when they needed to use the program menus (say, the "File" menu) they would naturally bring the mouse towards the upper left corner of the screen, to the "File" menu - and then "overshoot" the menu, activating the hot corner. Suddenly, the screen changed completely - all the program windows from the test were represented as smaller "tiles" on the screen. Testers exhibited clear confusion, irritation, and surprise at this unexpected behavior, even when they experienced the "hot corner problem" multiple times throughout the test.
My question in this usability study is How well do users navigate or understand the new design patterns in GNOME? The differences in GNOME between version 3.4 (my previous usability test, using Fedora 17) and GNOME 3.10 (Fedora 20) & GNOME 3.12 (latest release) appear to be largely "under the hood." With the exception of Gedit, the user interface differences appear minimal. Perhaps the largest change is the loss of "title bars."

My predictions:

The 4 (now 5) themes will continue
Despite UI differences from GNOME to Mac or Windows, I believe testers will comment that the programs are somewhat familiar. Familiarity will likely be the strongest theme: Gedit is not too different from Notepad, Web is similar to Firefox or Internet Explorer, Nautilus is not unlike Windows Explorer or Mac Finder, etc. Notes may be a new experience, however. But the fact that all programs act similarly, and share the same design patterns, will make the programs easier to learn. Once you figure out one program, you can apply that new knowledge to the other programs. The other themes will similarly continue. Obviousness will get positive comments due to improved messages and feedback in GNOME 3.10 & 3.12. But I expect usability issues with the loss of obvious menus, and more "hot corner" problems.
Users will be confused about menus
In my previous usability test, menus were an important part of usability. Menus were also a theme of good usability, because GNOME 3.4 still used menus for most actions. But in GNOME 3.10 & 3.12, actions have been moved into a "gear" menu. Some "top level" or "global" actions may only be accessed from the application menu. The loss of these obvious menus will likely cause problems during the test. I expect users will discover some functionality by clicking on the "gear" menu. Having found that menu, I don't know that users will experiment with the application menu. So any functionality that must be accessed through the application menu may be rendered unusable.
Users will think the test didn't go well, even if they were able to complete all the tasks
This is a visual rhetoric statement, and probably the most important. In rhetorical discourse, if you are performing a rhetorical ritual and the audience feels uncomfortable, it's because you left out an important part of the ritual. 
An easy example is a wedding ceremony. There's nothing that requires the use of "I take thee" vows. A wedding is a form of a legal contract, and you need to have some language in there that defines the wedding, but "I take thee" statements are not part of that contractual ceremony. But they are part of the traditional ceremony, the ritual. If you leave out the "I take thee" vows, it's pretty much guaranteed that family and friends in the audience will feel uncomfortable, as though something was missing. They may not be able to figure out what was missing, but they will know something got left out. And that's when some family and friends may begin to wonder "did they really get married?" 
I think it's the same with this usability test. The loss of certain visual cues, the missing direction provided by visually rhetorical devices (such as "title bars" and other common desktop metaphors) will make the testers uncomfortable. At the end of the test, when I ask how things went, I predict a common theme where users initially say they aren't sure, then shift their answer to "it didn't go well." And I think that will happen even if the testers were able to complete all of the scenario tasks. 
This isn't to say that user interfaces can't ever change, that they need to remain static. But it does suggest that interface designers need to pay close attention to these visual cues. Changes in these areas must be done with care. Sometimes you can get away with total change by shifting the context (such as the new interface on tablets, versus the traditional "menus and windows" interface on desktops and laptops - most people weren't surprised when this totally different computer had a totally different interface). But in general, if it's a visual cue, a rhetorical part of the desktop interaction, you can only change it in stages.
Specific problems with applications and tasks
I predict that "training" in other desktop environments (Windows or Mac) will translate to behavior expectation. In addition, while I attempted to reflect common user actions in the scenario tasks, I feel some of these actions happen so rarely in other desktop environments that the testers will be unfamiliar with how they might do them under a different environment. (How often do you change the default font in a text editor, or a sticky-notes program? Maybe once, when you first start using it, but never after that.) This will be exhibited by specific problems with applications and tasks, such as: changing the default font in Gedit and Notes, creating a shortcut to a folder in Nautilus, searching for a file that's not under the current folder tree, bookmarking a website in Web, increasing the font size in Web, and deleting all notes in Notes.
Problems caused by possible bugs
It's possible that we'll uncover a few bugs along the way, and users may experience usability issues because of programmatic errors. One problem I've found is that if you install a program (Robots) using Software, then try to remove the same program, you get an error. You seem to have to exit the Software program after installing a program, then re-launch Software to remove it. That seems like a program bug. As a workaround, I may ask testers to exit the Software program before they try out the Robots game, so we don't trigger the bug in Software.

Saturday, March 29, 2014

Institutional review

When you do your own usability tests, it's a good idea to gather some information about who is doing the test. Are your testers mostly men, mostly women, or a mix? What are the ages of your testers? (Sometimes, programs that are easy for one age group to use may be difficult for another age group.) How much previous knowledge do your testers have about the programs they will be using during the test?

And when you ask these questions in a higher education research setting, as mine is, you need to go through what's called an Institutional Review Board. As the name implies, the IRB reviews your research project proposal to make sure it doesn't harm the participants, and that any information that you gather doesn't put the participants at risk. My IRB review was pretty straightforward: a usability test doesn't harm the test participants or put them at risk, and I'm only asking a few informational questions that don't identify the person participating in the test.

Here are the questions I'm asking each tester to answer, before we do the test:
1. Your age: (please mark one)
  • 18 - 24
  • 25 - 34
  • 35 - 44
  • 45 - 54
  • 55 - 64
  • 65 - 74
2. Your gender: (fill in the blank)
3. Please indicate (circle) your level of computer expertise:
  1. I know very little about computers
  2. I know some things, but not a lot
  3. I am pretty average
  4. I am better than most
  5. I am a computer expert

My wrap-up script

Just as important as the "welcome" script, I need to have a "wrap-up" script. This makes sure I don't forget anything, and provides a good opportunity for gathering final reflections and thoughts.

Here is my wrap-up script:
Thank you for your time.

Let’s talk about the tasks you did today. Generally, did the tasks seem to get easier, more difficult, or about the same as you went through them?

  • When did things start to seem easier/harder?
  • What got easier/harder, and why did it suddenly seem so?

What are some themes or common areas where you thought things went really well, or were more difficult to do?
Give parting gift. (As a "thank you," I am giving out $5 gift cards to our local coffee shop, Higbies.)

My welcome script

When I sit down with each of the usability testers, it's good to have a "script" to go from, to make sure that I provide the same experience to each tester. I don't want to leave anything out. Everyone needs to come into the usability test with the same expectations.

While I may "tweak" the welcome script as I go, this is the base script that I'll use:
Brief context, what my project is about (inform tester about the purpose of the study, and it’s output).

Have you participated in a usability test before? This may be a new experience for you, and that’s okay.

Short explanation for what a “usability test” is. Not testing you, but you are helping us examine software; there are no “wrong” answers

Quick rundown of what the test will cover: Gedit, Nautilus, etc. (This is a good opportunity to share comparisons. For example, Nautilus is a file manager, like Windows Explorer, or Finder on Mac. Having that shared reference can help set the stage, but be careful about drawing too direct a comparison.)

Please “speak aloud” when you are looking for something, etc. I’ll be taking notes, but don’t let that distract you.

For each program, we’ll start by giving you a set of written scenarios: one at a time, one scenario per page. Do your best to complete that scenario. Also, have you used this program before? How would you expect to use the program?

When you complete each task, let me know. Then we’ll move on to the next task. (Each scenario task should be printed on a separate piece of paper. At the end of all scenarios for a program, take a short break before moving to the next program. This is where I do some basic Q & A by asking follow-up questions.)
Typical usability follow-up questions include:

  • “What were you thinking when you tried to do X?”
  • “You seemed lost when you were working on Y - what would have made it easier?”
  • “What were you expecting to see when you were doing Z?”

Also, a quick plus/delta: what worked well, and what needs to be improved?

My usability scenario tasks

My usability test starts on Monday, and I've gone through GNOME 3.12 & 3.10 to check and re-check that my scenario tasks make sense, and work under GNOME. It's important that I don't ask a tester to do something that you can't do in this version of the software, because the feature isn't there. In my previous usability test, I had testers do various tasks in Firefox; for this usability test, we'll use GNOME Web - but not every feature from Firefox is available in Web.

I also tried to pare down my list of usability tasks, to make sure I wasn't asking testers to do something that was too obvious. We won't learn much from the usability test if the tasks are too easy, or if they are repetitive (testing the same thing over and over). So I dropped a few tasks that I had used in the previous usability test.

And as I said in my other post, due to a few bugs, I won't be able to do the whole test in GNOME 3.12. I'll need to do some of them in GNOME 3.10.

Here are my scenario tasks that I will use in the GNOME 3.12 & 3.10 usability test:

1. You want to finish writing a draft of a blog post that you are using in a project. You start the Gedit text editor (this has been done for you).

Please open the file blog post draft.txt from the Documents folder, into the Gedit text editor..

2. You realize that you got a few details wrong. Please make these edits:
  1. In the first paragraph, change the dash (“—”) to a semicolon (“;”) - Relationships are currency—you to Relationships are currency; you
  2. In the second paragraph, change “me” to “others” - relationship to me. to relationship to others.
  3. About 2/3 into the document, there’s a list of the “4 I’s” of relationships, but the first two items are out of order. Put these into the correct order, so the list reads like this:
  1. Initiate
  2. Inquire
  3. Invest
  4. Inspire
When you are done, please save the file. You can use the same filename.

3. Some of the names are incorrect in the file. Please replace every occurrence of Applejack with Fluttershy, and all instances of Rainbow Dash with Twilight Sparkle.

When you are done, please save the file. You can use the same filename.

4. You'd like to make a copy of the note, using a different name that you can find more easily later. Please save a copy of this note as Leadership lessons.txt in the Documents folder.

For the purposes of this exercise, you do not need to delete the original file.

5. You decide the text in the editor is difficult to read, and you would prefer to use a different style. Please change the text style to be Nimbus Mono L, 12 point.
1. You are about to start working on a new project, and you would like to keep all your files in the same folder.

Please create a new folder called My Project in the Documents folder.

2. Your research partner has given you a bunch of files that you can use in your project, but you’ll need to copy them from his USB flash drive to your computer. Copy the Project001 folder (all the folders and files) from the USB flash drive (provided) into the My Project folder in the Documents folder.

3. Oops! You realize that the files in your new project will be photos, so you prefer to keep them under the Pictures folder instead of Documents.

Please move the My Project folder from Documents to Pictures.

4. Your project now has a new name, so you decide to rename the project folder to use the new name.

Please rename the My Project folder to Space Photo Project.

5. As you work on your project, you expect to go back to the folder frequently. A shortcut to the folder would be handy, rather than having to navigate to it each time.

Create a shortcut to Space Photo Project.

6. Your research partner included a few files that aren’t needed for the project. Specifically, the email to mom.txt file was included by accident. Find the email to mom.txt file in the files you copied.

7. 6. You don’t need the email to mom.txt file that was accidentally included in the project files, the same file that you searched for in the previous task. Delete the email to mom.txt file in the Mars photos folder. This is located under Space01, then under Photos, then under Mars photos.
Update (March 30)
I realized that the file-search action being tested in #6 will generate artificially positive results. In Nautilus, when you search for something, the default is to search starting from your current location. Since the previous tests were all in the Space Photo Project folder, and the file to search for is buried within that folder tree, users would immediately find it. But I want to see how well people can use the search function in Nautilus, even if the file they are looking for is elsewhere. I'm modifying the test to find a file not under the current folder tree. The new task #7 replaces the old task #6.

7. You would like to update a draft of your Space article, which you started several weeks ago. You don't remember where you last saved the file, so you need to find it. Find the file Draft of space article.txt. When you find it, open it.
1. Your friend has told you about an interesting arcade-style game that you’d like to try out on your computer. Install the game Robots.

2. After installing Robots, try it out to see what it’s like.

3. This is the last task for Software. Please uninstall the game Robots.
1. You would like to see what’s happening in the news. Your favorite news website is BBC News, but you don't remember the website’s address.

Find the BBC News website, and bring up the site.

2. You don't have your glasses with you, so it’s hard to read the text on the BBC News website. Please make the text bigger on the website.

3. You now would like to see what's new with your favorite “open source” project, FreeDOS. The website's address is

However, you want to keep an eye on the news while you are looking at the FreeDOS website. In a new site, navigate to the FreeDOS website.

4. You decide to download a copy of the FreeDOS program screenshot, the sample program image on the right side, under “Welcome to FreeDOS.” This image might come in handy for a document you are working on (writing the document is not part of this exercise).

Save a copy of this image as freedos.png in the Pictures folder.

5. You would like to visit the FreeDOS website later, but don't want to keep typing in the address every time. Create a shortcut for the FreeDOS website.

When you create the shortcut, name it simply FreeDOS.
1. You need to type up a few quick reminders for yourself, so you don’t forget to do them later. Enter these two reminders into the Notes program. Match the formatting as best as you can:

First reminder:
Don't forget: Jeff’s surprise party this Thursday. Check with Mark.

Second reminder:
Things to buy at the grocery store:

  • Milk
  • Eggs
  • Cheese

2. You decide to be more clear in your reminder about Jeff’s surprise party, so you don’t forget the time. Update that note to say this:
Don't forget:

Jeff’s surprise party this Thursday.
Get together at 5:30.
Party at 7:00.

Check with Mark.

3. You decide the text in Notes is difficult to read, and you would prefer to use a different style. Please change the default text style to be PT Sans, 14 point.

4. This is the last task for Notes. Please delete all notes you may have created during this part of the usability test.

A few bugs

I was very excited that GNOME 3.12 was released on March 26. But in getting ready for usability testing, I found a few bugs. Unfortunately, they affect stability. Programs crash or restart, or don't display anything altogether, when I try to run through my usability test scenarios. Gedit seems okay, but Web, Software, Nautilus, and Notes are just too unstable to use. I won't learn anything in a usability test if the participants aren't able to use the applications.

Unfortunately, I've already invited usability testers for next week. The usability tests run from Monday through Thursday. I don't think it's likely I'll get a fix for this by Monday.

So what I think I'll do is this:
  1. I'll boot the test laptop into GNOME 3.12, using the bootable USB flash drive (the one from the Getting GNOME page)
  2. Tester does the scenario tasks for gedit, on GNOME 3.12
  3. I'll reboot the laptop into Fedora 20 & GNOME 3.10
  4. Tester does the scenario tasks for Web, Software, Nautilus, and Notes
It's not ideal, but the big UI changes in GNOME between 3.10 and 3.12 (for my scenarios) seem to be in gedit, anyway. The other programs seem to have the same interface, the same behavior. Or if there are differences, I didn't notice them right away. This seems to be suggested by GNOME designer Allan Day recommendation to test "gedit (development version only)," from when I was planning my research.

Friday, March 28, 2014

It's about the User: Applying Usability in Open-Source Software

I wrote an article for Linux Journal about usability in open source software, based on my usability study a year or so ago. They ran the article in their December 2013 issue. You can also read the article online at the Linux Journal website:

It's about the User: Applying Usability in Open-Source Software

Thursday, March 27, 2014

Let's go

I've officially started the ball rolling on my usability test! This afternoon, I downloaded the GNOME 3.12 bootable image, and did a little testing in that. I'm currently verifying my test scenarios against this version, but things look good so far.

So I have taken the next step, and invited people to participate in a usability test. As a "thank you," I am giving out $5 gift cards to our local coffee shop, Higbies. Here's the email:
Hi there! I am working on a project to look at how people use certain computer programs, and I would like your help. It won't take much of your time, and in return you'll get a $5 gift card for Higbies!

It's easy: I'll ask you to complete a few tasks in 4 or 5 different computer programs. These tasks represent typical usage, such as using a web browser, editing a document, or copying files. The tasks aren't designed to be tricky or hard. After you do the tasks for each program, I'll ask a few questions to see how things went, and to get your opinion of the program. You should be done in about 45 minutes overall.

I am looking to do these sessions next week (Monday, 3/31 - Thursday, 4/3) in the afternoon and evening. You can pick a time that works for you by reserving an appointment for yourself on the "Usability testing" sign-up page.

Please email me if you have problems reserving a time on the sign-up page. (Note that the test should only take 45 minutes, but the appointment slots are 60 minutes so there's extra time before the next person shows up.)


Wednesday, March 26, 2014

GNOME 3.12 is released

I'm very excited that GNOME version 3.12 has been released! I look forward to using this new version in my usability test of GNOME.

For more information, visit GNOME 3.12 released on the GNOME website. They also shared a video that demonstrates some of the new features in GNOME:

Sunday, March 16, 2014

Getting ready for usability testing

A recent email to the GNOME development-announcement mailing list has me excited: GNOME 3.12 is coming soon!

Specifically, the email announces the "3.11.92 release candidate" which is the "last stop before 3.12." Checking the GNOME schedule on the wiki suggests that GNOME 3.12.0 will be available by the end of March (around March 26). This is perfect timing for my usability test! I have planned to do my usability test around the end of March or beginning of April. Looks like I'll be able to do those usability tests with the new GNOME 3.12.

Update (March 26)
GNOME 3.12 has been released, right on time! You can find it on the Getting GNOME page. There's even a live ISO image that you can use to try out this version: "GNOME 3.12 is the latest version and will be available in distributions in the future. In the mean time, you can try our live demo image." I'm downloading now.

Update (March 27)
This was announced on Slashdot, as well. Slashdot is a technology news portal. It's interesting to read the comments, as it supports my previous conclusions about usability. From my usability test last year, the four themes of successful usability were: Familiarity, Consistency, Menus, and Obviousness. In reading through the comments on Slashdot, they echo the importance of these four themes. In particular, I see a lot of comments on Slashdot that perceive GNOME as lacking in Familiarity (it's very different from Windows or Mac) and Menus (menu actions have moved under "gear icons" and the application menu). So it will be interesting to compare this "group think" perception with the results of my own usability test.

Saturday, March 8, 2014

Testing design patterns

Following up on my audience analysis, my usability test needs to examine design patterns in GNOME, rather than specific applications or programs. Allan Day of the GNOME Project recommends several such design patterns that GNOME would like to see in a usability test:

I'd be particularly interested in looking at selection mode, header bars, notifications, content overviews (browsing within the content apps), full screen, app menus and gear menus. The patterns look a bit different in some places, so it might be nice to compare, say, selection mode in Software compared with selection mode in Clocks. For full screen I would probably look at Videos (the new version that's been done for 3.12) or Documents.

The GNOME Applications wiki entry currently lists some 130 GNOME programs. That's a lot; it's impossible to cover so many programs in a single usability test. Fortunately, Day suggested a list of ten GNOME applications that use the new design patterns concept:

  1. Clocks
  2. Contacts
  3. Documents
  4. gedit (development version only)
  5. Maps
  6. Music
  7. Notes
  8. Photos
  9. Software
  10. Web

An ideal usability test would examine several design patterns across a few applications. But timing is an issue. If the usability test goes on for more than an hour, I'll "lose" the testers to exhaustion. It's hard to get volunteers to participate in a test longer than about 45-60 minutes. And that time needs to include the "welcome" where you explain the usability test, understand the tester, etc. - and the "wrap up" where you go back to ask about themes, or areas that seemed particularly easy or difficult to complete. As I plan the usability test, and decide what programs to include in it, I need to watch time carefully.

In order to draw comparisons between the results of this usability test and a previous usability test conducted with GNOME 3.4 and Firefox, the new test design should include Nautilus, Web, and gedit. Nautilus is the GNOME file manager; while the GNOME project is hesitant to make further changes to Nautilus, testing this program in a usability study would provide an interesting comparison to earlier results: How have GNOME design patterns improved the usability of the file manager? Determining which other programs to include in a usability test requires further examination of the GNOME design patterns.

I went back and forth as I analyzed how each of the ten programs used the GNOME design patterns. I commented on part of this process as I discussed my areas of investigation. I found it helpful to compare the ten programs and what functionality they exercised within the design patterns. From that analysis, I eventually focused on these design patterns, which is pretty much the same list that Day suggested:
  1. Application Menus (1)
  2. Header Bars (2)
  3. Content Overview
  4. Selection Mode (5)
  5. Gear Menu
  6. Notifications
(numbers correlate to diagram, from HIG wiki)

These are the design patterns that get exercised the most, and/or have the largest impact to user task scenarios. And as I further examined the ten programs, I determined that a usability study of only a few programs should provide sufficient exercise to the six design patterns to draw effective comparisons and derive common themes. These programs are:
  1. gedit
  2. Nautilus
  3. Notes
  4. Software
  5. Web
This list also incorporates three programs that will let me expand upon my previous usability study. Looking back, those programs were GNOME's gedit, GNOME's file manager (Nautilus) and the Firefox web browser. In my new usability study, I would like to examine programs that let me expand the analysis - but at the same time, provide a deeper analysis of my previous results. I think the new list of five programs will allow me to do that.

Audience analysis: GNOME

The first step in preparing a usability test is to understand the users. Often, project owners will identify users in broad terms, then create "user profiles" that define a set of theoretical users of the system. By creating these user profiles, all members of the project can use the same starting point in discussing usability, and how design decisions will affect certain users. With an agreement on user profiles, a designer can say "This change will help users like Frank" or provide critiques like "This updated design actually makes it harder for users like Susan." This takes design discussion away from personal experience, and helps designers remain focused on the users.

For my usability study, I plan to examine several programs from the GNOME desktop environment. So, who are the target users for GNOME?

Allan Day of the GNOME Project shared that GNOME does not identify a subset of users, such as developers or computer experts. Rather, GNOME targets all users. GNOME looks at the user experience across three categories. From Allan's email to me:

1. Core experience - this is the operating system, it includes the shell and the rest of the "bare" system. It is a very "thin" platform for running apps. ie. We provide as little functionality outside of applications as possible.

2. Core apps - these cover the basic things that almost everyone needs, as well as some essential utilities that are needed by the system. You get a browser, apps for viewing files and other content, handy utilities like a calculator, and so on.

3. 3rd party apps - these are how diverse users fulfil their more specialist requirements. Applications are how you extend the functionality of your system.

I wouldn't necessarily say that the core apps are targeting a particular kind of user. Rather, they are targeting what the majority of users have in common.

As a result, the design vision in GNOME addresses a broad audience. Users may be anyone, from computer novices to computer experts, from programmers to graphics designers to writers and grandmothers. GNOME seeks the lowest common denominator. Accordingly, the usability test design must not target a particular kind of test participant. Testers should represent a mixture of backgrounds, gender, and age groups.

So I guess I get to skip the step of defining the target users. The usability test should instead examine design patterns in GNOME, rather than specific applications or programs.

Friday, March 7, 2014

Preparing for usability testing

You can do a usability study in different ways. Alice Preston described the types of usability methods for the STC usability community (Vol 10, No. 3). In the list, she describes 11 different ways to evaluate usability in a design. Her list:
  1. Interviews/Observations: One-on-one sessions with users.
  2. Focus Groups: Often used in marketing well before there is any kind of prototype or product to test, a facilitated meeting with multiple attendees from the target user group.
  3. Group Review or Walk-Through: A facilitator presents planned workflow to multiple attendees, who present comments on it.
  4. Heuristic Review: Using a predefined set of standards, a professional usability expert reviews someone else's product or product design and presents a marked checklist back to the designer.
  5. Walk-Around Review: Copies of the design/prototype/wireframe are tacked to the walls, and colleagues are invited to comment.
  6. Do-it-Yourself Walk-Through: Make mock-ups of artifacts, but make the scenarios realistic. Walk through the work yourself.
  7. Paper Prototype Test: Use realistic scenarios but a fake product.
  8. Prototype Test: A step up from a paper prototype, using some type of animated prototype with realistic scenarios.
  9. Formal Usability Test: Using a stable product, an animated prototype, or even a paper prototype, test a reasonably large number of subjects against a controlled variety of scenarios.
  10. Controlled Experiment: A comparison of two products, with careful statistical balancing, etc
  11. Questionnaires: Ask testers to complete a formal questionnaire, or matching questionnaire.
In 2012,  I suggested that only 4 of Preston's usability methods would apply to the study of open source usability:
  1. Heuristic Review
  2. Prototype Test
  3. Formal Usability Test
  4. Questionnaires
In sharing that list of 4 methods, I used an unspoken assumption of open source programs that are suitably large, yet ones that had usability issues that still needed to be uncovered and analyzed.

The heuristic review is interesting. A “Usability Analysis” or “Usability Critical Analysis” is basically a heuristic review, and is essentially a “plus/delta” group exercise, focused on what is working in the design (plus) and what needs to be improved (delta). In this review, a panel of usability experts provide comments on a program's user interface design, and predict how users might respond to such a design. The interface might be a working prototype, or a mock-up on paper. A parallel in other fields of academia is the “Literary Critical Analysis,” a discussion of a work in literature. The use of criticism doesn't imply disapproval or a negative review, but instead a full review of the work.

However, the heuristic review relies on trust between the reviewer and the program maintainer. If that trust is lacking, or if the maintainer doesn't "buy into" the usability review, the maintainer will likely dismiss the heuristic review. This is one reason why many open source software programs lack good usability. From a previous conversation with Eric Raymond, Raymond commented that most programmers view "Menus and icons are like the frosting on the cake after you've baked it" and that any results that try to provide correction to usability is "swimming against a strong cultural headwind." While a mixed metaphor (intentional) it's very descriptive of the problem.

A more compelling way to achieve usability "buy in" with developers is to perform a usability test. There's nothing quite like seeing a user experience problems with your program.

In my next post, I'll share some notes on how to do a usability test, and planning for your usability test audience.

Saturday, March 1, 2014

A look back, and looking ahead

In another post about my research, I shared that there has been very little academic research about usability in open source software. While the field of “Usability Testing” has received significant attention, not much research has been applied to open source software.

Usability studies of open source software seem to be somewhat rare in academic literature. Most notably, David M. Nichols and Michael B. Twidale have authored several articles on the subject (for example, “Usability Processes in Open Source Projects”). Calum Benson, Matthias Müller-Prove, and Jiri Mzourek wrote similar reviews of usability tests in open source software (ACM: “Professional Usability in Open Source Projects: GNOME,, NetBeans”).

A 2009 PhD thesis by Paula M. Bach (College of Information Sciences and Technology, Pennsylvania State University) explored “Supporting the user experience in free/libre/open source software development.” Similarly, a 2011 bachelor’s thesis by Jan-Christoph Borchardt (information design, Stuttgart Media University) examined “Coordinating low-cost usability testing in independent free & open source software projects,” and is now an open document (CC BY-SA 3.0) summarizing “Usability in Free Software.”

Other research has typically been conducted informally, or sponsored by corporate interests. In a 2013 presentation at the GUADEC 2013 conference (GNOME Users’ And Developers’ European Conference, held annually in cities around Europe), Fabiana Simoes discussed issues surrounding the capture and reporting of UX issues in open source software.

I researched the literature surrounding usability studies of open source software. For a list of academic resources on this topic, see my previous comment listing some 8 academic articles on the usability of open source software. These are the most-often-referenced resources. While this is not a complete list of everything out there, we can assume a literature review is fairly complete with you start to get significant overlap: when multiple sources keep references the same other sources.

The names that keep coming up most often in this field (usability in open source software) are Benson et al, Frishberg et al, and Nichols & Twidale.

But doing a study of the usability of open source software is a pretty big field; open source software covers a lot of programs. You can't test everything. I see two ways forward: focus the study, so you investigate the usability of a specific open source software program, or you study a system of programs to discover common themes in the usability of open source software.

In a previous study, I examined the usability of several common applications, including GNOME's gedit, GNOME's file manager (Nautilus) and the Firefox web browser. In my new usability study, I would like to examine programs that let me expand the analysis - but at the same time, provide a deeper analysis of my previous results. That means including the same or similar programs in this study that I investigated last time.

So for this test, I'm keeping my focus on GNOME. And as suggested by Andersen (2013), academia and practice need to develop a reciprocal relationship. It's a cycle; academia needs to provide accessible, actionable research that is published in places visited by practice. That allows the practice to advance, which provides future research opportunity for academia. The cycle continues, and both academia and practice benefit. That's why I'm sticking with GNOME applications for this usability study: doing a usability test on several GNOME applications provides a reciprocal benefit to academia (my work) and practice (GNOME).