Thursday, November 29, 2012

A quick link

I haven't read through the actual study yet, but I wanted to share this paper: "Current Issues of Usability Characteristics and Usability Testing" by Ayobami, Hector, Hammed. The abstract:

A well given design’s attention to Learnability, Contextuality and Operationability as Usability characteristics issues of software, and information systems generally is an evolving and challenging area in the field of Information systems, and Human Computer Interaction as an area of expertise. The procedure of testing and evaluating usability as quality characteristics in information systems has equally attracted researchers and professionals more in the recent times due to the developers’ attempt to meet the diverse users’ needs in the development of usable information systems. Hence, the paper aims to bring to the limelight; these current issues of usability and usability testing, the trend of the challenges inherent in the usability testing and evaluation process, then suggest the necessary attention that must be provided by subsequent researches in the bid to solve these lingering design research problems.

Why not usability?

I mentioned this earlier, but I'll share it again:

Interesting to note a comment I see repeated when I ask open source software folks about usability: "I'm not a usability expert." But you don't need to be a usability expert to apply usability to open source software. As Redish said to me in our interview, you can learn a lot just by sitting down with a user, and watching them use the software. You don't need to be a usability expert to grab a friend or family member, and ask them to try out a scenario for 15 minutes.

But not everyone is so easily convinced of the value of usability testing. A List Apart discusses this topic (July 31, 2012).  From the article:

Like any research method, though, usability testing has its drawbacks. Most importantly, it isn’t cheap. In a classic study, you’ve got the expenses of developing the testing plan, recruiting and compensating the participants, sometimes even travel. Even remote testing—which has its limitations—can be expensive, because analysis of the testing data is still no easy feat. In fact, analysis is often the biggest component of the project. Going through all those testing videos and cataloging the qualitative data just can’t be sped up without undermining the integrity of the study.
[...]
Fortunately, we have other usability research methods at our disposal. The standouts, expert review and heuristic evaluation, are both easy to add to a design and development process almost regardless of budget or resource concerns. Both involve minimal costs, as compared to full-scale usability testing, with expert review in particular being scalable to almost any project. For projects with fuller budgets, both techniques can maximize the cost-benefit ratios of subsequent usability testing.


The article highlights "discount usability engineering" which is basically a "heuristic review" where a usability expert evaluates an information product (such as a website) across several areas, including:


  • Appearance: The appeal and effectiveness of the site’s look and feel, from major layout features to small typographical details.
  • Content: The quality and strategic significance of the site’s content—including not only page copy but page metadata, PDFs and other files, and element-specific content (e.g., image descriptions).
  • Interface Usability: Ease of use of the site’s interactive components, from the simplest navigation structures to the most complex forms.
  • Accessibility and Technology: On a basic level, the site’s ability to adapt to diverse user needs, encompassing not only accessibility guidelines compliance but also browser compatibility and mobile- and tablet-friendliness.


While I (personally) view in-person usability tests as being the most useful, if you have experienced usability experts who can help you, doing a heuristic review can identify the "low hanging fruit" on your design - which will result in a better end product.


Usability testing by flash mobs

I liked this article about "usability testing by flash mob" from Usability Testing Howto Blog. From the article:

Ever heard of Improv Everywhere? This was the UX equivalent. Researchers just appeared out of the crowd to ask people to try out a couple of designs and then talk about their experiences. Most of the interactions with participants were about 20 minutes long. That’s it. But by the time the sun was over the yardarm (time for cocktails, that is), we had data on two designs from 40 participants. The day was amazingly energizing. 

What an amazing idea! Using "flash mobs" and other on-the-spot venues for usability testing is a great innovation. If your test subject is willing to spare a few minutes, and with a "mob" of individual testers and evaluators, you can glean valuable information in a short amount of time.

The article lists ten suggestions to making "flash mob testing" work well:

  1. Organize up front
  2. Streamline your data collection
  3. Test the data collection forms
  4. Minimize scripting
  5. Brief everyone on the dos and don'ts of usability evaluation
  6. Practice intercepting people
  7. Do an inventory check before starting
  8. Be flexible!
  9. Check in
  10. Reconvene on the same day for a wrap-up

This idea of "flash mob" usability testing can be extended to other domains, too. You can do this type of impromptu usability testing on website visitors. Chad Fennell shared a few links with me, including a software package "Ethnio" that allows you to intercept visitors so you can recruit them for live UX research.

Ethnio and other similar packages will likely be used more frequently in future by web designers as they seek active usability feedback from users. It doesn't take much for a usability test, only 5 or so testers can identify usability issues or positive trends. Imagine a design team setting aside a day to intercept web visitors, asking them to respond to a quick (5 minute) usability scenario. How many visitors does your site get in a day? How many visitors are on your website right now? Even if only 1/4 of your visitors give you their 5 minutes, it won't take long to get useful feedback.

How many testers do you need?

When I was planning my usability tests, I optimistically hoped for a dozen usability testers. But "from the discussions I've had with thought leaders in usability testing, I may not need as many as a dozen folks to get usability good results - so I'll be satisfied if I only attract half that (five or six)."

For my usability tests, ten people signed up, but three were "no-shows," leaving seven people who participated in the usability tests. And it was amazing how quickly those few testers were able to "zero in" to the usability problems and highlight positive trends for "what worked well."

From Jakob Nielsen, it turns out 5 testers is usually enough to get you there, if you're just interested in finding the usability problems for a product: "If you want a single number, the answer is simple: test 5 users in a usability study. This lets you find almost as many usability problems as you'd find using many more test participants."

If you want to dig deeper or search for other information, you may need more testers, however:

As with any human factors issue, however, there are exceptions:
  • Quantitative studies (aiming at statistics, not insights): Test at least 20 users to get statistically significant numbers; tight confidence intervals require even more users.
  • Card sorting: Test at least 15 users.
  • Eyetracking: Test 39 users if you want stable heatmaps.

I've commented before that it doesn't take much to pull together a few testers and ask them to run through test scenarios that represent how the product is used.  Interesting to note a comment I see repeated when I ask open source software folks about usability: "I'm not a usability expert." But you don't need to be a usability expert to apply usability to open source software. As Redish shared with me in our interview, you can learn a lot just by sitting down with a few users, and watching them use the software. By extension, you don't need to be a usability expert to grab a friend or family member, and ask them to try out a scenario for 15 minutes.

What about joining a community?

This study of open source software and usability neatly intersects two of my interests: information design (and usability) with open source software.

I've been involved in the open source community for a very long time. I first became aware of "Free software" around 1992 (or 1993?) while I was an undergraduate physics student. I had several classes that required use of the campus computer lab, and the lab manager installed many Free software (such as the GNU C Compiler, the GNU Emacs editor, and other programs) for us to use on the central Unix systems. I found this software to be of equally high quality to the commercial "payware" programs that I had used up to that time.

Later in 1993, I decided that I could use Free software to run my computer, so I installed the Linux operating system on my small '386 PC. I've used Linux at home (and at work) ever since, although I continued to run DOS on my computer for the rest of my undergrad years.

During my undergrad years, I extended my interest in open source software. As a physics student, I had learned some computer programming - and had learned more about it outside of class. In 1994, I started work on my own Free software version of the DOS operating system. I first called this "PD-DOS" for "Public Domain DOS," I quickly renamed it to "Free-DOS" and later "FreeDOS" as much of the operating system was built on Free software.

FreeDOS was my first "real" entry into the open source community. Since then, I've contributed to other Free software and open source software projects, including operating systems, editors, tools, music programs, and games.

More recently, I became interested in information design, through my graduate studies. Usability is strongly related to information design - and in fact, is a method to improve information products.

So in a recent conversation with my advisor, I started looking to communities surrounding usability and "UX". One popular destination is the UXPA - Usability / User Experience Professionals Association (formerly "UPA"). I'm sure there are other communities that have formed around LinkedIn and other social media venues. I plan to join UXPA and LinkedIn communities, even if only as a "lurker" (reading and listening, while rarely posting or contributing topics). What other communities can you suggest?

Asking the right questions

An article "Modifying Your Usability Testing Methods to Get Early-Stage Design Feedback" on UX Matters discusses the early stage design questions for your usability test. It's a great quick read about giving a boost to your usability tests.

What about sharing with OS cons?

In a discussion with my advisor today, I considered the idea of sharing my findings with a broader in-person open source forum. There are two open source conventions that I can think of, which might be interested in "open source usability" as a feature topic:

O'Reilly Open Source Convention ("OSCON")
As its name implies, an open source convention sponsored by O'Reilly (the publisher of many useful technical reference manuals and how-to books). I attended an OSCON about ten years ago, chairing a birds of a feather session about FreeDOS. Topics at OSCON span a broad range, and I think a presentation on open source usability would be well attended.

Penguicon
An open source software (and science fiction) convention held annually. I was a Nifty Guest some years back, speaking about FreeDOS, Linux, open source in the enterprise, and building open source communities. The date and location for Penguicon 2013 haven't been set yet, but based on last year's schedule I would expect it in late April or early May.

Monday, November 19, 2012

Draft outline

While I am busy analyzing the usability test findings, I don't want to lose focus on the eventual outcome of this study: how can I help the open source community improve the usability of open source software? At the start of this project, I suggested an article in a journal or magazine related to open source software. I'd like to continue with that goal, using Linux Journal as a target.

LJ is a front-runner among Linux and open source magazines. First published in 1994, LJ used to be a traditional subscription print journal, but recently (2011) switched to electronic only. LJ focuses mostly on Linux, but discusses open source projects in general. Articles published in LJ will see the widest distribution in the open source software community. Senior Editor: Doc Searls. Associate Editor: Shawn Powers.

So, what would be the outline of such an article? Here's one draft outline: (Note: I've updated this a bit since yesterday. Adjusted 1, 7, 8 with new content.)

  1. A bit of background: we've all experienced open source programs that were difficult to use or confusing. Or you may have heard from someone, "this is a great program, once you figure out the menus, or once you learn how to use it." That's a problem. i.e. bad usability in many open source programs makes people unlikely to use them, OR the main uptake for open source programs in the non open source community is programs that are easy to use/have good usability, so how do we improve usability to make open source more widely used. - Make this a strong beginning with the thesis of the study, uses an anecdote to draw in the readers.
  2. What is "usability," and what is good usability?
  3. Why do so many open source programs have bad usability? In short, because most open source programs are written by developers for other developers. The functionality is most important; the menus and usability are generally second thoughts - if they are considered at all.
  4. How can developers improve usability? Ideally, you should start with user profiles and usage scenarios - but a usability test will get you there. Talk about the methods of usability: heuristic review, prototype tests, formal usability tests, questionnaires, etc. But you don't need to do anything big and complicated to get useful/helpful information - a simple in-person usability test is all you need.
  5. How to do a usability test.
  6. My usability test.
  7. What I learned from my usability test: what works in usability, what needs work. - What makes these programs usable, and how I determined that.
  8. Summary and next steps: how open source programmers can do their own usability tests, you don't need many users to get useful feedback. The challenge is how to incorporate usability into your project so it becomes part of the "culture." Summary should include wrap up of paper (this is what makes for good usability, etc.) as well as tease the next steps.

Sunday, November 18, 2012

What worked well?

I almost titled this post "What works in open source usability," but I think I'm not quite there yet. So for now, let me discuss what worked well in the usability tests this week, and the comments from the testers.

Testers were easily able to accomplish the following tasks:

Gedit (text editor)

  • Write a simple note. The scenario asked them to type in a short note (about two paragraphs) so I could observe them recovering from typos and errors.
  • Basic editing. Replacing a single instance of a few words, and move a sentence from one paragraph to the other.
  • Replacing text. This asked testers to replace all occurrences of two names with new spellings of the same names.
  • Saving a file as a new copy of the file, with a new name.

Firefox (web browser)

  • Search the web for a particular news site, and navigate to that site.
  • Set that website as the browser's default startup page.
  • Increase the font size.
  • Create a new tab or window, and navigate to a particular URL.
  • Save the page for off-line viewing.
  • Download an image from the website.
  • Make a bookmark to a website for later use, and give the bookmark a new name.

Nautilus (file manager)

  • Create a new folder.
  • Move a folder to a new location.
  • Give a folder a new name.
  • Delete a file.

So, what's common that made these scenarios go well for the testers? I can provide commentary, but let me first share a few of the testers' responses to the questions "Across all three programs, what went well? Can you suggest why this went well for you?":

  • "Replace All" worked well to replace Jeff with Geoff, and Jenny with Ginny.
  • Having right-click menus everywhere helped a lot.
  • All programs sort of look the same.
  • Firefox was "good" and straightforward, seemed easy to use.
  • Right-click options were good.
  • Having the "+" button in Firefox to make a new tab made it easy, and because it was next to the other tab made it easy to find.
  • Highlighting was good, especially in the editor.
  • Firefox is like other browsers.

In general, I see Familiarity as a theme. Testers indicated that the programs seemed to operate more or less like their counterparts in Windows, and remember that most (6 out of 7) testers used Windows as their primary desktop. Gedit wasn't too different from Windows Notepad, or even Word. Firefox is like other browsers. Nautilus is very similar to Windows Explorer. To some extent, these testers had been trained under Windows, so having functionality - and paths to that functionality - that was approximately equivalent to the Windows experience was an important part of their success.

Also, Consistency was a recurring theme in the feedback. Right-click menus worked in all programs, and the programs looked and acted the same.

Menus were also helpful. While some said that icons (such as in the toolbar) were helpful to them during the test, most testers did not actually use the toolbar in Gedit - except to use the Save button. Instead, they used the program's drop-down menus: File, Edit, View, Tools, etc. In fact, testers experienced problems when the menus did not present possible actions clearly.

And to extend the analysis slightly further, allow me to contrast with what didn't work well - Obviousness. When an action produced an obvious result, or clearly indicated success - such as saving a file, creating a folder, opening a new tab, etc. - the testers were able to quickly move through the scenarios. When an action did not produce obvious feedback, the testers became confused. (These problems were especially evident when trying to create a bookmark or shortcut in the Nautilus file manager, but the program did not provide feedback, did not indicate whether or not the bookmark had been created.)

    GNOME: this needs fixing

    One other thing to comment on before talking about what works in usability - in doing the usability test, 6 of 7 testers experienced difficulty with the GNOME "hot corner." In GNOME, you can open the "Activities" menu (basically, the "programs" menu) by clicking on the "Activities" icon in the upper left corner of the screen, or simply by moving your mouse all the way to the upper left corner of the screen (the "hot corner"). You can also do this by pressing the "Windows" key on your keyboard.

    Those who experienced the "GNOME hot corner problem" typically did so right away in the first exercise when attempting to use the program menus. While testers were able to recover from the hot corner, it definitely caused disruption several times throughout the test.

    None of the users in the usability test had used GNOME, so they did not have previous experience in using GNOME. The programs they were evaluating in the usability test were running "maximized" so when they needed to use the program menus (say, the "File" menu) they would naturally bring the mouse towards the upper left corner of the screen ... and then "overshoot" the menu, activating the hot corner. Suddenly, the screen changed completely - all the program windows from the test were represented as smaller "tiles" on the screen.

    Most of these users (5 of the 6 testers) recognized that they needed to click on the "tile" for the program they were in, to bring up the program again. But in doing so, I heard these quotes from the testers:
    • "Dammit!"
    • "Oops!"
    • "What?"
    • "Hey, what happened."
    • "Whoa."

    I think the quotes represent confusion and surprise at the hot corner. Testers frequently experienced the "hot corner problem" multiple times throughout the entire usability test. I am not sure why 1 tester happened never to hit the hot corner during the entire test.

    I'll add that a GNOME extension, "No Topleft Hot Corner," disables this hot corner behavior. However, users would have to know to seek out and install this add-on, as it is not available by default.

    Nautilus: what needs work

    In the Nautilus file manager usability test, users were asked to complete a series of short scenarios. Here are the ones that caused problems:

    "4. As you work on your project, you expect to go back to the folder frequently. A shortcut or bookmark to the folder would be handy, rather than having to navigate to it each time. Create a bookmark for Space Photo Project."

    In the preceding scenarios, the testers were asked to create a folder, move it, and rename it. As explained in the exercises, the folder would be used in a project that collected photos from space, stored under the Pictures folder.

    However, when asked to create a bookmark or shortcut to the folder, almost everyone (6) had problems. Quotes from the testers in this exercise:

    • "I'm not sure if it created the bookmark."
    • "It's not speaking to me." (Not telling me anything.)
    • "That didn't work."
    • "I'm looking for something that says 'Bookmark'." (Later commented didn't notice the Bookmarks menu.)
    • "Nothing shows up."
    • "I want to grab the address of the folder."
    • "Nothing shows up."
    • "Where did that go? I'm confused."
    • (To the computer) "Why do you not want to bookmark it?"
    • "Oops, did I do that wrong?"
    • "What about Edit - Make Link?"
    • "I don't think that did anything."
    • "Something should pop up to tell me."
    • "Nothing popped up, it didn't work."

    Even though I used the same term in the scenario ("shortcut or bookmark") as in the menus ("Bookmarks"), testers were unsure what to do. The most common action was to go into the Pictures folder, highlight the project folder with a single mouse click, then select Bookmarks - Add Bookmark. When the menu action didn't do anything, testers also looked in File and Edit menus.

    Two testers were eventually able to create a bookmark to the project folder. Of these, one first used the Help menu and searched for the answer - but admitted he would not have done that if he didn't know about the feature already. The other, having exhausted a number of menu actions, tried creating the bookmark from within the folder she needed to mark. And that, of course, is how you create a bookmark in Nautilus.

    It seems clear that Nautilus should display a message when the user tries to create a bookmark. If the action is successful, it should note the success. If the action fails, Nautilus should explain why in simple language.

    After the usability test, several testers asked how to create a bookmark, since they were unable to do this. When I explained how to create a bookmark, and why their earlier attempts to create the mark seemed to fail, they thought they could do it successfully the next time. (For those curious: selecting Bookmarks - Add Bookmark from within the Pictures folder, no matter what is highlighted, will attempt to create a bookmark for Pictures ... but Pictures is effectively bookmarked already as a default, so Nautilus doesn't take any action.)

    "6. Do you remember the image that you downloaded earlier, in the web browser exercise? The filename was freedos.png, but you don't remember where you saved it. Find the freedos.png file and move it to the Pictures folder, if it is not there already."

    In this scenario, testers were asked to search for a file. If I noted that they had (earlier) successfully downloaded the web image into the Pictures folder, I simply asked them to find the file.

    Two of the testers opted not to use the file manager to search for the missing file, but clicked on each folder in sequence until they found the necessary file.

    The other five testers used the Search button in the upper-right of Nautilus, but searching for the file was confusing for them. Quotes heard during this section:

    • "I'm not finding it."
    • "What else to search on?"
    • "Reload my search? Nothing happened!"
    • "Grrrr!"
    • "There is no general search?"
    • "Not found, huh."
    • "No option for my home."

    Only two users were able to use Search successfully. Of these, one happened to click on the Search button from their home directory. The other tried changing things in the drop-down search action and eventually picked a combination that worked.

    Saturday, November 17, 2012

    Gedit: what needs work

    It's easier to talk about what needs to be improved (the "deltas") than to analyze what worked well (the "plusses") so let me take a moment to discuss this for Gedit.

    "5. You decide the font in the editor is difficult to read, and you would prefer to use a different font. Please change the font to DejaVu Sans Mono, 12 point."

    In Gedit, testers were really confused about how to set the font and the color. Part of this confusion stemmed from thinking of the editor like it was Word, which makes it easy to change the font. But even Windows Notepad lets you change the default font using a Format menu item.

    Testers reported they were looking for a menu item called "Font". Quotes from users during this part of the test:

    • "Looking for Font."
    • "Looking for the font change menu."
    • "Not showing a Font menu."
    • "Where do I change the font?"
    • "Looking for a Font menu, but it's not in the toolbar."
    • "Is it an 'input method'?"

    While looking to set the font, testers looked in all the menus, but mostly in the File, Edit, View, and Tools menus.

    A few testers (3) eventually discovered how to set the font, but all of these took several minutes to do so. Upon discovering the correct menu item (Edit - Preferences - Font & Colors) to change the font, two testers actually exited the menu because they didn't know how to change the font. They did not realize that they needed to uncheck the "Use the system fixed width font" item.

    "6. You decide the black-on-white text is too bright for your eyes, and you would prefer to use different colors. Please change the colors to the Oblivion color scheme."

    Changing the default colors presented similar problems. Quotes during this part of the test:

    • "Looking in Tools, not seeing it."
    • "Looking for a menu with a font."
    • "Looking for a toolbar to do this."
    • "Look for colors."

    The 3 testers who were able to set the font in the previous scenario also were able to change the colors, and one reported that it was easier because they saw the colors in the other exercise.

    Testers looked in all the menus to find the colors setting, but mostly in the File, Edit, View, and Tools menus - the same as in the font exercise.

    A preview of usability test results

    In my discussions with thought leaders about usability testing, I learned that you can often get a surprisingly accurate picture of usability in as few as 3 testers. It's one thing to hear that said; it's another to see it happen.

    This week, I worked with 7 usability testers in an informal lab setting: I set up a laptop in my office, with a USB keyboard and mouse, booted into a USB fob drive installation of Fedora 17 running the Gnome desktop. Each tester was logged into their own account, thus each tester started from the same "default" state.

    Testers ranged from about age 20 to mid-40s, both male (3) and female (4). Most testers (5) claimed "low" experience with computers, and almost all (6) used Windows as their primary desktop.

    I'll do a more detailed write-up of results soon, but here's a preview of the findings:

    Gedit

    Testers were easily able to accomplish the following tasks:

    • Write a simple note. The scenario asked them to type in a short note (about two paragraphs) so I could observe them recovering from typos and errors.
    • Basic editing. Replacing a single instance of a few words, and move a sentence from one paragraph to the other.
    • Replacing text. This asked testers to replace all occurrences of two names with new spellings of the same names.
    • Saving a file as a new copy of the file, with a new name.

    However, they experienced great difficulty in these tasks:

    • Changing the default font. Everyone agreed that an editor, even a simple editor like Windows Notepad (and Gedit is similar in function to Notepad), should let them change the text.
    • Changing the default colors. Most said that they would expect to change the colors, but either had trouble finding this function or were were unable to find it altogether.

    Firefox

    With a few one-off exceptions, testers had no problems using Firefox to achieve these tasks:

    • Search the web for a particular news site, and navigate to that site.
    • Set that website as the browser's default startup page.
    • Increase the font size.
    • Create a new tab or window, and navigate to a particular URL.
    • Save the page for off-line viewing.
    • Download an image from the website.
    • Make a bookmark to a website for later use, and give the bookmark a new name.

    Nautilus

    Testers were easily able to complete these tasks:

    • Create a new folder.
    • Move a folder to a new location.
    • Give a folder a new name.
    • Delete a file.

    However, testers experienced great difficulty with these tasks:

    • Create a new bookmark or shortcut for a folder.
    • Search for a file.


    I'll dig into these results further in another post.

    Usability tests this week

    This week, I conducted a usability test of three open source programs:
    1. Gedit, a text editor
    2. Firefox, a web browser
    3. Nautilus, a file manager
    These programs were selected based on your input of open source programs that feature good usability.

    I asked for volunteers to participate in a usability test. Since I work on a university campus, it was fairly straightforward to get volunteers for this usability test; I offered free candy and pop to anyone who sat in the test. In all, 10 volunteers stepped up. Unfortunately, a few were no-shows, leaving 7 who helped out (plus my wife who participated in a "dry run" of he usability test the weekend before - so technically there were 8 people). My ideal was about a dozen testers, but I said I would be happy with 5 or 6. So I'm good.

    At the start of each usability test, I gave the tester a brief context of the study, and why I am looking at open source usability. I explained that this was a usability test, so it was about the software. If the tester experienced problems, that's okay. Don't feed judged. I'm here to observe. And along the way, I'll be taking notes and looking at your screen.

    I also asked the testers to speak aloud what was going through their mind. For example, if they were looking for a Print button, just say "I'm looking for a Print button." And if they were looking for something on the screen, they move the mouse to where they were looking, so I could get an idea what they were searching for.

    During the usability test, the subject was presented with a number of scenarios. Each scenario was printed at 16 pt Garamond on a separate sheet of paper, and provided a brief context and an action they were asked to complete. For example: You are about to start working on a new project, and you would like to keep all your files in the same folder. Please create a new folder called My Project in the Documents folder.

    I listed the usability scenarios in a previous blog post.

    The usability tests went for about 30-40 minutes, depending on the tester. So I was pretty might right-on with my time estimates. I took a few minutes at the end of each program to ask some followup questions, and at the end of the test we did a quick "plus/delta" to wrap up.

    Wednesday, November 14, 2012

    Make FLOSS usable!

    "FLOSS" is the general term for "Free/Libre and Open Source Software", a more inclusive term than simply "open source". But the concept behind the OpenUsability project applies to all open source software: make FLOSS usable!

    From the About page:
    The idea of OpenUsability originated at the KDE developers conference in 2003. Jan Muehlig and his Berlin-based company relevantive had performed a comparative usability study of KDE 3.1 and Windows XP. The developers appreciated the work of the usability professionals, and a discussion came up how to involve usability feedback continuously with development. So far, it had been difficult for both sides to get in touch with each other: On the one side, self-acclaimed usability experts had reduced the developers' trust in the effects of usability, and on the other side trustful usability specialists often had problems to get in touch with the right people in a project.

    OpenUsability came to fill this gap. Since 2004, Open Source Software Projects can register to get usability support. A growing number of usability practitioners started to work under the label "OpenUsability", and provided the projects with usability feedback, attended developer conferences and fairs.
    (Note: the comparative study between KDE and Windows XP no longer exists on the relevantive.de site, so I have dropped the links.)

    They provide a list of projects that are specifically targeted to help certain open source programs: ArgoUML, Edyk, GCstar, ProjectPier, and so on. However, I don't see any general guidelines for how open source developers can apply usability practices into their own work, or successful ideas for good usability.

    Sunday, November 11, 2012

    Usability scenarios

    This weekend, my wife helped me with the study by agreeing to be a usability tester, by doing a "dry run" of the usability test scenarios. This helped me to gauge time and determine if the scenarios could be completed by a "typical" user with "average" knowledge. (Although she runs Linux on her laptop, my wife admits she is not very technical - she has "average" knowledge.)

    I previously shared a draft of the usability scenarios/exercises for Gedit, Firefox, and Nautilus. Here are the new questions after the dry run: (I also updated the previous blog posts, but this is for reference.)

    Gedit

    1. You need to type up a quick note for yourself, briefly describing an event that you want to remember later. You start the Gedit text editor (this has been done for you).

    Please type the following short paragraphs into the text editor:

    Note for myself:

    Jenny and I decided to throw a surprise party for Jeff, for Jeff's birthday. We'll get together at 5:30 on Friday after work, at Jeff's place. Jenny arranged for Jeff's roommate to keep him away until 7:00.
    We need to get the decorations up and music ready to go by 6:30, when people will start to arrive. We asked everyone to be there no later than 6:45.

    Save this note as party reminder.txt in the Documents folder.

    2. After typing the note, you realize that you got a few details wrong. Please make these edits:

    • In the first paragraph, change Friday to Thursday.
    • Change 5:30 to 5:00.
    • Move the entire sentence Jenny arranged for Jeff's roommate to keep him away until 7:00. to the end of the second paragraph, after no later than 6:45.

    When you are done, please save the file. You can use the same filename.

    3. Actually, Jeff prefers to go by Geoff, and Jenny prefers Ginny. Please replace every occurence of Jeff with Goeff, and all instances of Jenny with Ginny.

    When you are done, please save the file. You can use the same filename.

    4. You'd like to make a copy of the note, using a different name that you can find more easily later. Please save a copy of this note as Geoff surprise party.txt in the Documents folder.

    For the purposes of this exercise, you do not need to delete the original file.

    5. You decide the font in the editor is difficult to read, and you would prefer to use a different font. Please change the font to DejaVu Sans Mono, 12 point.

    6. You decide the black-on-white text is too bright for your eyes, and you would prefer to use different colors. Please change the colors to the Oblivion color scheme.

    Firefox

    1. You would like to see what's happening in the news. Your favorite news website is BBC News, but you don't remember the website's address.

    Search for the BBC News website, and bring up the site.

    2. You would like to bring up the BBC News website every time you start up the web browser. Please set the BBC News website as the default page.

    3. You don't have your glasses with you, so it's hard to read the text on the BBC News website. Please make the text bigger on the BBC News website.

    4. You now would like to see what's new with your favorite "open source" project, FreeDOS. The website's address is http://www.freedos.org/

    However, you want to keep an eye on the news while you are looking at the FreeDOS website. In a new site, navigate to the FreeDOS website.

    5. You would like to save a copy of the FreeDOS front page, so you can view it later on your laptop during your bus ride home tonight. Save a copy of the page to the Documents folder.

    6. You decide to download a copy of the FreeDOS program screenshot, the sample program image on the right side, under "Welcome to FreeDOS". This image might come in handy for a document you are working on (writing the document is not part of this exercise).

    Save a copy of this image as freedos.png in the Pictures folder.

    7. You would like to visit the FreeDOS website later, but don't want to keep typing in the address every time. Create a bookmark for the FreeDOS website.

    When you create the bookmark, name it simply FreeDOS.

    Nautilus

    1. You are about to start working on a new project, and you would like to keep all your files in the same folder.

    Please create a new folder called My Project in the Documents folder.

    2. Oops! You realize that the files in your new project will be photos, so you prefer to keep them under the Pictures folder instead of Documents.

    Please move the My Project folder from Documents to Pictures.

    3. Your project now has a new name, so you decide to rename the project folder to use the new name.

    Please rename the My Project folder to Space Photo Project.

    4. As you work on your project, you expect to go back to the folder frequently. A shortcut or bookmark to the folder would be handy, rather than having to navigate to it each time.

    Create a bookmark for Space Photo Project.

    5. Do you remember the party reminder that you created earlier, in the text editor exercise? In that exercise, you created a copy of party reminder.txt as Geoff surprise party.txt, but did not delete the first file.

    Please delete the party reminder.txt file in the Documents folder.

    6. Do you remember the image that you downloaded earlier, in the web browser exercise? The filename was freedos.png, but you don't remember where you saved it.

    Find the freedos.png file and move it to the Pictures folder, if it is not there already.

    Friday, November 9, 2012

    More interesting links

    Here are a few other interesting links shared with me by Chad Fennell, from the U of M:

    Ethnio, a web-based tool to intercept users on your website, so you can recruit them for UX live research.

    Regarding the Drupal tests, some of the resulting documentation - gives some sense of impact and adoption of the recommendations that resulted from the tests: Chad is also very involved in the Drupal project, working in the area of usability. Drupal is an interesting case, and appears to be fairly unique among open source projects for their community focus on usability.

    The last round of testing:
    Drupal usability
    The resources section contains links to participant videos, the issues cue matrix that resulted, etc.

    Here is where most ongoing usability discussions happen (outside of IRC):
    Drupal usability discussion

    A good place to ask more question: #drupal-usability on freenode.

    A few interesting links

    I wanted to share a few links, originally shared with me by Nick Rosencrans from the U of M, and Drupal:

    The case for open source design : a description of the way open source development works, with a bunch of helpful diagrams and such

    OS Alt : search engine for open source alternatives to commercial software

    Drupal usability : results from our usability test conducted for a Drupal team

    Wednesday, November 7, 2012

    Nautilus usability scenarios

    In the usability test for the Nautilus file manager, I will ask testers to do these scenarios:

    (Before beginning this phase of the usability test, I will start Nautilus for the tester. Note that Nautilus starts up in the Home folder.)

    1. You are about to start working on a new project, and you would like to keep all your files in the same folder.

    Please create a new folder called My Project in the Documents folder.

    2. Oops! You realize that the files in your new project will be photos, so you prefer to keep them under the Pictures folder instead of Documents.

    Please move the My Project folder from Documents to Pictures.

    3. Your project now has a new name, so you decide to rename the project folder to use the new name.

    Please rename the My Project folder to Space Photo Project.

    4. As you work on your project, you expect to go back to the folder frequently. A shortcut or bookmark to the folder would be handy, rather than having to navigate to it each time.

    Create a bookmark for Space Photo Project.

    5. Do you remember the party reminder that you created earlier, in the text editor exercise? In that exercise, you created a copy of party reminder.txt as Geoff surprise party.txt, but did not delete the first file.

    Please delete the party reminder.txt file in the Documents folder.

    6. Do you remember the image that you downloaded earlier, in the web browser exercise? The filename was freedos.png, but you don't remember where you saved it.

    Find the freedos.png file and move it to the Pictures folder, if it is not there already.

    7. You decide you are done working with the test program you wrote earlier, in the text editor exercise. Delete the test.sh program file in the Documents folder.

    (UPDATE 11/11/12) Since I've removed the program scenario from Gedit, I will drop scenario 7.

    Firefox usability scenarios

    For this part of the usability test, I will ask the testers to complete these scenarios:

    (Before beginning this phase of the test, I will start Firefox for the tester. Note that on Fedora Linux, Firefox starts up with a default home page from the Fedora project.)

    1. You would like to see what's happening in the news. Your favorite news website is BBC News, but you don't remember the website's address.

    Search for the BBC News website, and bring up the site.

    2. You would like to bring up the BBC News website every time you start up the web browser. Please set the BBC News website as the default page.

    3. You don't have your glasses with you, so it's hard to read the text on the BBC News website. Please make the text bigger on the BBC News website.

    4. You now would like to see what's new with your favorite "open source" project, FreeDOS. The website's address is http://www.freedos.org/

    However, you want to keep an eye on the news while you are looking at the FreeDOS website. In a new site, navigate to the FreeDOS website.

    5. You would like to save a copy of the FreeDOS front page, so you can view it later on your laptop during your bus ride home tonight. Save a copy of the page to the Documents folder.

    6. You decide to download a copy of the FreeDOS program screenshot, the sample program image on the right side, under "Welcome to FreeDOS". This image might come in handy for a document you are working on (writing the document is not part of this exercise).

    Save a copy of this image as freedos.png in the Pictures folder.

    7. You would like to visit the FreeDOS website later, but don't want to keep typing in the address every time. Create a bookmark for the FreeDOS website.

    When you create the bookmark, name it simply FreeDOS.

    8. You don't like looking at advertisements on websites. A friend said you can block ads in your web browser by installing something called Adblock Plus.

    Install Adblock Plus in your web browser.

    I'm not really sure the Adblock Plus scenario is a common usage example for "average" users with "typical" experience. What do you think?

    (UPDATE 11/8/12) After some discussion, I've decided to drop scenario 8 from the usability test. Installing Adblock Plus is not a task that "typical" users with "average" knowledge would be expected to do.

    * See, this is one of those times that user personas and usage scenarios would come in handy! We would be able to refer back to those to decide if Firefox's "typical" users would need to install add-ons like Adblock Plus.

    Gedit usability scenarios

    In asking testers to exercise the Gedit text editor, I will ask them to run through these scenarios:

    (Before beginning the Gedit usability test, I will start Gedit for the tester. Note that Gedit starts up with an empty document.)

    1. You need to type up a quick note for yourself, briefly describing an event that you want to remember later. You start the Gedit text editor (this has been done for you).

    Please type the following short paragraphs into the text editor:

    Note for myself:

    Jenny and I decided to throw a surprise party for Jeff, for Jeff's birthday. We'll get together at 5:30 on Friday after work, at Jeff's place. Jenny arranged for Jeff's roommate to keep him away until 7:00.

    We need to get the decorations up and music ready to go by 6:30, when people will start to arrive. We asked everyone to be there no later than 6:45.

    Save this note as party reminder.txt in the Documents folder.

    2. After typing the note, you realize that you got a few details wrong. Please make these edits:

    • In the first paragraph, change Friday to Thursday.
    • Change 5:30 to 5:00.
    • Move the entire sentence Jenny arranged for Jeff's roommate to keep him away until 7:00. to the end of the second paragraph, after no later than 6:45.

    When you are done, please save the file. You can use the same filename.

    3. Actually, Jeff prefers to go by Geoff, and Jenny prefers Ginny. Please replace every occurence of Jeff with Goeff, and all instances of Jenny with Ginny.

    When you are done, please save the file. You can use the same filename.

    4. You'd like to make a copy of the note, using a different name that you can find more easily later. Please save a copy of this note as Geoff surprise party.txt in the Documents folder.

    For the purposes of this exercise, you do not need to delete the original file.

    5. Gedit can also be used to write programs. For the purposes of this exercise, let's assume you are a programmer. Please open the test.sh program file in the Documents folder. It should look like this:

    #/bin/bash

    echo "Hello there!"
    echo -en "The date is: "
    date

    if [ "$COLS" -lt 80 ] ; then
        echo "this is a narrow terminal"
    fi

    if [ "$UID" -eq 0 ] ; then
        echo "you appear to be the administrator"
    fi

    You realize that the middle section that refers to COLS is not working, so as a debugging step you decide to comment out the three lines, starting with if and ending with fi.

    Comment out these lines by placing a # at the start of each of the three lines. When you are done, the file should look like this:

    #/bin/bash

    echo "Hello there!"
    echo -en "The date is: "
    date

    #if [ "$COLS" -lt 80 ] ; then
    #    echo "this is a narrow terminal"
    #fi

    if [ "$UID" -eq 0 ] ; then
        echo "you appear to be the administrator"
    fi

    When you are done, please save the file.

    (UPDATE 11/11/12) I've decided to drop scenario 5 from the usability test. This really isn't something that an "average" user with "typical" knowledge would do, and it would skew the results. I did a "dry run" usability test with family over the weekend, and it caused problems and added too much time to the test.

    6. You decide the font in the editor is difficult to read, and you would prefer to use a different font. Please change the font to DejaVu Sans Mono, 12 point.

    7. You decide the black-on-white text is too bright for your eyes, and you would prefer to use different colors. Please change the colors to the Oblivion color scheme.

    Scenarios in usability testing

    Usually when creating an information project, you would start by describing the people who are likely to use your product, and what they need to get out of your product. These are called user personas and are very helpful in generating a common understanding with everyone on the project about the users and their goals. To make the personas more concrete, you should include photos of people who look like the users you describe. (Stock photos are fine as long as they are casual, and always make sure you have permission to use the photo.) Photos put a human touch on the personas.

    The next step after personas is to describe actions that your users would take to accomplish their goals. These are called user scenarios. Scenarios can be very detailed, or they can be left a high level - but they always describe how your users do their work. There's nothing like being with your users when they try to accomplish a task, so you can watch them first-hand, but a good "second best" is to capture that behavior in a user scenario.

    In usability testing, you start not with the user personas, but with the scenarios. These usability scenarios are usually described at a high level, as you do not want to burden your usability tester with a preconceived notion for how they should accomplish a task, simply provide a description of the task or tasks they need to perform using the information product.

    In my example of usability testing, I will ask usability testers to complete several tasks, described as scenarios. The scenarios will start with a brief context, then a textual description of the task the tester is asked to perform. In the usability testing field, these are elaborated scenarios.

    I'll describe the scenarios in other posts here.

    Monday, November 5, 2012

    Slashdot question on open source usability

    A few weeks ago, the popular technology blog Slashdot polled the community for interview questoins to ask open source advocate Bruce Perens. I was glad to see Bruce answered my question about usability in open source software:

    Usability in open source software?
    by Jim Hall

    Bruce, I'm doing a study of usability in open source software - how user interfaces can be designed in Free / open source programs so the program is easy to use by real people. So my question is twofold: What Free / open source program really got it right with usability? What qualities make for good usability in Free / open source software?

    Bruce: Although I evangelize the issue of building our software for people outside of our community, I am no expert in usability at all. So, what that means is that there is room for others to evangelize best practices. I'd like to learn. I'll gladly pass your message along, if you build a compelling case.

    If I had to guess, I'd say that Mozilla has more expertise in this than many Free Software projects. I have been running the GNOME 3 desktop in Debian Sid for a month or so, and I am not sure GNOME 3 is quite there yet. Keep it up, folks!

    Unfortunately, that's a pretty short answer. I didn't expect a longer answer, though - usability in open source software seems to be an overlooked topic. Not a lot of research in this area.

    Interesting to note a comment I see repeated when I ask open source software folks about usability: "I'm not a usability expert." But you don't need to be a usability expert to apply usability to open source software. As Redish said to me in our interview, you can learn a lot just by sitting down with a user, and watching them use the software. You don't need to be a usability expert to grab a friend or family member, and ask them to try out a scenario for 15 minutes. (I'm doing the same with my family when I return from a trip next weekend - the start of my usability testing!)

    As of this writing, none of the comments on the Slashdot article discuss usability in open source software. But I wanted to share anyway.

    Sunday, November 4, 2012

    Usability testing

    (This post has been updated with a few improvements, based on suggestions. Edits are in blue.)

    Next week, I plan to start doing usability testing on the three programs: Gedit, Nautilus, Firefox.

    Since I work on a university campus, I'll start by inviting students, facutly, and staff to participate in the usability study. In most usability tests, you give your testers a small gratuity for their time. Since I don't have a budget ($$) to give out cash rewards, I'll instead offer free cans of pop to take with them, and as much candy as they can carry away with them. That's pretty inexpensive for me - and on a university campus, that should still garner a fair turnout.

    I'll also invite folks from the surrounding town (we're small, it's easy to get word around) to participate in the test too. And I'l try to get friends and family to submit to a usabilty test, as well.

    My goal is about a dozen testers from different age groups. From the discussions I've had with "thought leaders" in usability testing, I may not need as many as a dozen folks to get usability good results - so I'll be satisfied if I only attract half that (five or six).

    Each tester will start with a fresh instance to test against. This needn't be a separate install; it can be the same workstation, but different test accounts so that subsequent testers always get the same "vanilla" starting point as the first tester. My thought is to have the tester work from a bootable USB fob drive running Fedora 17, with separate test accounts for each tester: "test1", "test2", "test3", ... Since each test account can only modify it's own settings, everyone starts from the same "default" settings. This is important, especially since I'll likely ask testers to create or modify bookmarks (Firefox) or change a default font (Gedit).

    The goal will be to have the tester engaged in a usability test for no longer than 45 minutes - that's 15 minutes per program. I'm thinking about the following steps:

    Welcome (less than 5 minutes)
    • Brief context of the study, what is my study about (inform tester about the purpose of the study, and it's output)
    • Short explanation for what a "usability test" is - we are not testing you, but you are helping us examine software; there are no "wrong" answers
    • Quick rundown of what the test will cover: Gedit, Nautilus, Firefox. (This is a good opportunity to share comparisons. For example, Nautilus is a file manager, like Windows Explorer, or Finder on Mac. Having that shared reference can help set the stage for the tester, but I need to be careful about drawing too direct a comparison.)
    • Ask if they have participated in a usability test before. This is probably a new experience for many of the potential testers.
    • Introduction to how to do a usability test - please "speak aloud" when you are looking for something, etc

    Testing (30-45 minutes, depending on the tester)
    • Gedit (10-15 minutes)
    • Firefox (10-15 minutes)
    • Nautilus (10-15 minutes)
    For each program, start by giving the tester a set of written scenarios - one at a time, one scenario per page - and ask him/her to complete that scenario. Also, ask the tester if they have used this program before, or how they would expect to use the program. 
    When the tester has completed a scenario, move on to the next scenario: each scenario should be printed on a separate piece of paper. At the end of all scenarios for a program, take a short break before moving to the next program. This is where I do some basic Q & A by asking followup questions. Typical usability followup questions include: "What were you thinking when you tried to do X?" "You seemed lost when you were working on Y - what would have made it easier?" "What were you expecting to see when you were doing Z?" Also, a quick "plus/delta": what worked well, and what needs to be improved?

    Wrap-up (less than 5 minutes)
    • "Thank you for your time."
    • Provide them with a 1-page information sheet about the three programs, as a giveaway. (Testers may like to refer back to this afterwards, as a way to better connect with the test.)
    • Give parting gift, and send them on their way.

    Friday, November 2, 2012

    And then there were three

    Earlier this week, I wrote about the two candidates for good usability that I will use in my usability study. After some discussion, I've decided to add Firefox back on the list of programs to examine. This now means I'll do a usability study of three successful open source programs, instead of two.

    When I asked for suggestions of programs to include in the study, many folks recommended a web browser - specifically Chrome and Firefox. But I was concerned that the ubiquity of these programs on other platforms (Mac, Windows) could mean we wouldn't be testing the usability of these programs during the usability test, but rather how well the testers navigate these browsers on another platform.

    But there's a lot to be learned by exploring the usability of Firefox. Just because the usability testers might already be familiar with Firefox shouldn't be a reason to exclude a program that so many people recommended as a candidate for good usability. So it's in.

    As a reminder, the three programs that I will study:

    1. Gedit (a text editor)
    2. Nautilus (a file manager)
    3. Firefox (a web browser)

    We'll start usability testing next week. My next step is to craft scenarios for each, which we will use in the usability test.