Friday, May 16, 2014

Free software outpaces proprietary code quality

I was interested to read this recent article in CIOL, describing the analysis of 750 million lines of free software and open source software code through the Coverity Scan service, and comparing commercial usage of the Coverity Development Testing Platform. According to the article, this is the largest sample size that the report has studied to date. Their results: free software code quality outpaces proprietary code code quality.

Key findings from the report include:

  • Open source code quality surpasses proprietary code quality in C/C++ projects. Coverity's analysis found an average defect density of .59 for open source C/C++ projects that leverage the Scan service, compared to an average defect density of .72 for proprietary C/C++ code developed for enterprise projects.
  • Linux continues to be a benchmark for open source quality.
  • C/C++ developers fixed more high-impact defects. Not so much for projects written in Java, for example.

While I would be cautious about overconfidence, these results support Eric S. Raymond's maxim (which he terms "Linus's Law") that "given enough eyeballs, all bugs are shallow," from The Cathedral and the Bazaar.

Saturday, May 10, 2014

Usability testing through prototypes

Usability in all software is a problem, not just free software and open source software. But with free software and open source software, we have a unique opportunity. Our users are our collaborators. We often think of this from a "developer" mindset; it really helps to have people out there writing code, submitting bug fixes, adding new features. Sometimes we step outside this mode of thinking and ask collaborators to do other helpful things such as writing documentation, testing functionality, and reporting bugs. With everyone working together, we all win.

But in free software and open source software, our user/collaborators can and should include usability testing. Developers can learn a lot about how their programs get used just by sitting down with a few people and watching them explore the program. You can use a variety of methods to do usability testing; my paper lists 11 methods, and demonstrates a full usability test. But there are other ways to do usability testing.

One way is a variation of the "Paper prototype" method. In a traditional paper prototype test, you print out on paper all the screens that the user might encounter when using the program. Then you ask people to simulate using the program by telling you what they might click on, and you present new menus or screens for them to interact with.

But it's hard to do a paper prototype test when you're a small free software or open source software project. Getting people together to test can take some doing.

Instead, leverage a website to do the prototype testing for you. Create a few mockups of your program screens, showing every menu that users can interact with. Then "animate" these using a series of simple web pages. The tricky part is getting all the screens in order. That means you'll need to grab screenshots of your program, or create new screenshot mockups.

If your program includes the menus and functionality already, you can just grab screenshots for each menu and every screen. If your program isn't quite "there" yet, then you'll need to creat screenshot mockups to use. Here's an example:

Let's say I wanted to write a simple Tic Tac Toe game program. I'm not sure what menus I'll use, or what items I would need to use in each menu. So I decide to create screenshot mockups. To start, I'll grab a screenshot of another program, just to get all the major screen elements in order. Here's the GNOME Terminal program:

(click to enlarge)

Open this screenshot image into GIMP, and do some quick edits to remove the "Terminal" parts of the image. This is the first step to creating simple screenshot mockups.

(click to enlarge)

Add elements that represent what your program screen might look like. In this example, I've added the tic tac toe grid, edited the title, and simplified the menus to suit my game:

(click to enlarge)

Note that the screenshot mockup doesn't need to look exactly like the final version of the program. This is just a mockup that testers can recognize. Don't spend a lot of time making this perfect. You can see in my example that I wasn't too careful in removing all the elements from the original Terminal screenshot, and I created a rough tic tac toe grid by drawing two rectangles on the screen and chopping off the ends. It's not perfect, but testers will be able to understand the mockup if it is at least close. You can always add a disclaimer later that the mockup is just that: a simulation of what the program screens might look like.

The final step is to create screens showing all the menus and other interactions that your program might allow. In this example, I only have a few menus. Maybe your program also allows right-click context sensitive menus, or uses pop-up windows to display other information. You'll need to create screenshot mockups for these screens as well. Here's one screenshot for the File menu in my Tic Tac Toe game. It's not perfect, but it's enough for people to understand and interact with the screenshot:

(click to enlarge)

Once you create the menus for your program, create a very simple website that lets testers explore the mockup. No need to use fancy javascript here; you can do it very simply. For example, you might put each screenshot mockup on its own web page, and use the html <area> tag to to define regions where users can click to view the next screenshot mockup, on a different web page. In my Tic Tac Toe game, I might use <area shape="rect"> for each of the red rectangles, to let testers access each of the menus in my game program. So testers can "exit" out of the menus, I might define <area shape="poly"> for the purple region:

(click to enlarge)

I'll add that it took me all of 30 minutes to create these five simple screenshot mockups in GIMP. This doesn't need to take a lot of time. To create mockups for the other menus might take me another 20 minutes.

Having created the screenshot mockups and the simple website that lets users explore the mockup, you'll need to invite testers to try out the mockup. Define a few simple scenarios that "exercise" different parts of the program. I find it's best to limit testers to two or three scenarios. Create a separate web page with these scenarios, and ask testers to open the "mockup website" in a separate tab in their web browser. They can flip back and forth between the scenarios and the mockup as they test.

Your scenarios should be worded very simply and plainly, but avoid using terms from the program. For example, for the Tic Tac Toe game, you might ask the tester to change the "look" of the tic tac toe board, not change its "view."

Testers can then run through the mockup and click on each screen as they test the scenario. At the end of each scenario, ask the testers to fill out a brief survey, such as via a simple web form:
  1. Do you think you completed this task?
  2. Please describe what you did (what you clicked on) to complete the task.
  3. What worked well?
  4. What would you change?
  5. Rate the difficulty of this task. (You might use a form with a numerical scale, -5 to +5, where -5 is hard, and +5 is easy.)
You can learn a lot about your program if you ask just a few users to run through these screenshot mockups. What you think is easy might be challenging for someone else. Menu items or descriptions that make sense to you might be difficult for someone else to understand. But by running through a simple "animated prototype" described here, you can quickly find out what areas of your program need more work, and improve the usability. And with good usability, everyone wins.

Friday, May 9, 2014

What's next?

As you know from my other post, my Master's capstone title was Usability Themes in Open Source Software. I feel very passionate about this topic, and I plan to continue working with usability in free software and open source software. As a start, I will keep writing here.

I'm going to submit a version of my capstone project to the Journal of Usability Studies. I think I can get three good JUS articles out of it: the usability study & results, using a heat map to display usability results, and a lit review of usability in open source software.

Both Linux Journal and Linux Voice said they would like to run a version of my article, so I'll write separate versions for them. And the blog has asked me to write an ongoing series of articles about usability in open source software. And I will be submitting proposals at open source software conferences like GUADEC, O'Reilly OSCON, Penguicon, and FOSDEM. I hope to see you there!

And the GNOME folks have asked if I can help them with future usability tests of new versions of GNOME. That would be a great continuation of this work!
Subscribe to the RSS feed to stay updated on Open Source Software & Usability! (RSS)

Thursday, May 1, 2014

Usability Themes in Open Source Software

For about the last two years, I've been working hard on my Master's degree. This was an interesting program, M.S. in Scientific & Technical Communication. I had a lot of great classes, including a directed study in Usability, in which I explored Open Source Software & Usability. I expanded this into my Master's capstone project, "Usability Themes in Open Source Software."

It is my pleasure to share my capstone project with you:
Note on the different versions: When I generated the EPUB ebook, I fixed a few minor formatting errors from the PDF. The MOBI ebook is a conversion from EPUB.