Sunday, September 25, 2016

Great first year at LAS GNOME!

This was the first year of the Libre Application Summit, hosted by GNOME (aka "LAS GNOME"). Congratulations to the LAS GNOME team for a successful launch of this new conference! I hope to see more of them.

In case you missed LAS GNOME, the conference was in Portland, Oregon. I thoroughly enjoyed this very walkable city. Portland is a great place for a conference venue. When I booked my hotel, I found lots of hotel options within easy walking distance to the LAS GNOME location. I walked every day, but you could also take any of the many light rail or bus or trolley options running throughout the city.

I encourage you to review this year's conference schedule to learn about the different presentations. I'll share only a few highlights of my own. I also live-tweeted through many of the presentations, and I'll share some of those tweets here:

Alexander Larsson gave a great presentation ("Taking back the apps from the distributions") about Flatpak. I'd followed Flatpak before, but until Alexander's presentation, I never really grokked what Flatpak can do for us. With Flatpak, anyone can provide an application or app anywhere, on any distribution.

Today, you may be comfortable installing whatever application you need through your distribution. But what if your distribution doesn't provide the application you are looking for? And what if you want to run an application that a distribution isn't likely to want to provide (such as a commercial third-party office application or game)? In these cases, you either hope the person providing the application can provide a package for your distribution, or you go without.

With Flatpak, you can bundle an application to run on any Linux distribution, and it should just run. I see this as a huge opportunity for indie games on Linux! Imagine the next version of PuzzleQuest also being available for Linux, or a version of Worms for Linux, or a version of Inside for Linux. With Flatpak, these indie developers could bundle up their apps for Linux—and sell them.

Also, a great thing about going to conferences is the opportunity to finally meet people you've only chatted with online. Ciarrai was one of my interns during the May-August cycle of Outreachy, and we finally got to meet at LAS GNOME! We had been emailed back and forth in the weeks ahead of time, since we knew we'd both be there. But it was great to finally meet them in person!

LAS GNOME had scheduled talks in the morning, and "unconference" topics in the afternoon. With an "unconference," people suggest topics for an impromptu presentation or workshop, and attendees vote on them.

In the afternoon of day 1, I hosted an informal workshop on usability testing, at the same time my friend Asheesh gave an unconference presentation on web app packaging in Sandstorm. I was surprised to see so many developers at my how-to presentation about usability testing. Thanks to all who attended! Scott snapped this selfie of the two of us:

On day 2, Asheesh Laroia gave a presentation ("How to make open source web apps viable") about Sandstorm. Asheesh and I have known each other for a few years now, and I've followed his work with Sandstorm. But it was great to see Asheesh walk us through a demo of Sandstorm to really show off what it can do. In short, Sandstorm provides an open source software platform that helps people collaborate over the web.

And Sandstorm is designed with several layers of security, so it's very locked down. Even if you can "break out" of one web application, you can't get elsewhere on the system. As Asheesh put it during his talk, it's like Google Docs, but more secure:

On day 3, Stephano Cetola shared a wonderful story about how he got involved in open source software and made it his career ("Endless Summer of Code: Getting Involved in OSS"). I loved how Stephano talked about his experimental nature, and how he learned about technology by letting the "magic smoke" out of things.

Because of his eagerness and willingness to learn, Stephano found new opportunities to grow—eventually landing a position where he works for Intel, working on the Yocto project. A great journey that was a joy to experience!

Finally, I gave my presentation on usability testing for GNOME ("GNOME Usability Testing"). I opened by talking about different ways (direct and indirect) that you can test usability of software. From there, I gave an overview of the Outreachy internship, and what Renata, Ciarrai and Diana worked on as part of their internships.

I was thrilled for Ciarrai to share their part of usability testing from this cycle of Outreachy. Ciarrai's project was a paper prototype test of a new version of GNOME Settings. Renata conducted a traditional usability test of other ongoing work in GNOME, and Diana worked on a User eXperience (UX) test of GNOME.

I think my presentation went well, and we had a lot of great questions. Thanks to everyone!

Again, congratulations to the LAS GNOME team for a successful first year!
image: LAS GNOME
photo: Scott

Sunday, September 18, 2016

Headed to LAS GNOME!

By the time this gets posted on the blog, I will be headed to LAS GNOME. I'm really looking forward to being there!

I'm on the schedule to talk about usability testing. Specifically, I'll discuss how you can do usability testing for your own open source software projects. Maybe you think usability testing is hard—it's not! Anyone can do usability testing! It only takes a little prep work and about five testers to get enough useful feedback that you can improve your interface.

As part of my presentation, I'll use our usability tests from this summer's GNOME usability tests as examples. Diana, Renata and Ciarrai did a great job on their usability testing.

These usability testing projects also demonstrate three different methods you can use to examine usability in software:

Ciarrai did a paper prototype test of the new GNOME Settings. A paper prototype is a useful test if you don't have the user interface nailed down yet. The paper prototype test allows you to examine how real people would respond to the interface while it's still mocked up on paper. If you have a demo version of the software, you can do a variation of this called an "animated prototype" that looks a lot like a traditional usability test.

Renata performed a traditional usability test of other areas of ongoing development in GNOME. This is the kind of test most people think of when they hear about usability testing. In a traditional usability test, you ask testers to perform various tasks ("scenario tasks") while you observe what they do and take notes of what they say or how they attempt to complete the tasks. This is a very powerful test that can provide great insights to how users approach your software.

Diana led a user experience (UX) test of users who used GNOME for the first time. User experience is technically different from usability. Where usability is about how easily real people can accomplish real tasks using the software, user experience focuses more on the emotional reaction these people have when using the software. Usability and user experience are often confused with each other, but they are separate concepts.

I hope to see you at the conference!
Interested in my slides from LAS GNOME? You can find them on my personal website at I'll keep them there for at least the next few weeks.
image: LAS GNOME

Saturday, September 10, 2016

Wrap-up from this cycle of Outreachy

Now that all interns have completed their work, I wanted to share a few final thoughts about this cycle of Outreachy. Hopefully, this post will also help us in future usability testing.

This was my third time mentoring for Outreachy, but my first time with more than one intern at a time. As in previous cycles, I worked with GNOME to do usability testing. Allan Day and Jakub Steiner from the GNOME Design Team also pitched in with comments and advice to the interns when they were working on their tests and analysis.

As before, I structured the usability testing internship along similar lines to an online class. In fact, I borrowed from the usability testing internship schedule when I taught CSCI 4609 Processes, Programming, and Languages: Usability of Open Source Software, an online elective computer science program at the University of Minnesota Morris. We first learned about usability testing, then we practiced usability testing. In this way, the internship was broken up into two phases: a research phase and a project phase.


With three interns, we decided from the beginning that we should focus on three different areas for our usability tests. In previous cycles, with only one intern at a time, we conducted traditional usability tests that spanned several common design patterns used in GNOME. In this cycle, we wanted to continue with a traditional usability test, but we wanted to explore other areas of GNOME usability.

Based on the previous experience and interests of Renata, Ciarrai and Diana, we assigned them to do these three usability tests:

  1. Renata: A traditional usability test of ongoing development in GNOME.
  2. Ciarrai: A paper prototype test of the new GNOME Settings.
  3. Diana: A first-time GNOME User eXperience (UX) test.

A paper prototype test uses a slightly different test design from a traditional usability test, but the analysis is the same. In a paper prototype test, as in a traditional usability test, you ask a tester to respond to various scenario tasks. But in a paper prototype test, the tester is acting against a paper mock-up of the design. In this case, Ciarrai used screenshot mock-ups of the updated GNOME Settings.

Renata led a traditional usability test. In her test, we asked Renata to further examine areas that had improved based on the last cycle of GNOME usability testing. This allowed us to measure any usability gains in the design.

Diana performed a different kind of test. Technically, a user experience test is not a usability test; usability and user experience are different concepts. Usability is how easily users can use the software to accomplish real tasks in a reasonable amount of time. But user experience is more about the emotional engagement of the user when using the software. It's possible for a program to have good usability and poor UX, or for a program to have poor usability but good UX, but in most cases you will find a positive correlation between usability and user experience. Most of the time, if a program has good usability, it will likely have positive UX. And if a program has poor usability, it will likely have negative UX.


When planning any usability test, it's important to start with an understanding of who uses the software. Typically, a project should have these documented as personas, fake personalities that reflect real-world users. With personas, a project can discuss design changes in a more concrete way. Rather than a designer or developer planning a change "because it's cool" or "because I like it," they can plan changes that will help the "Sally" persona do her work, or will make the "Stephen" persona more productive, or will allow the "Chris" persona to more easily work with the software.

Once you understand the users, you need to define how the users will use the software. These are the use scenarios. Users might have different needs that carry different expectations. For a word processor program, one user might use the program to write a report at work, while another user might use the program to write a research paper (with footnotes!) for class, while another user might use the program to jot a quick letter to a friend.

From this understanding of use scenarios, you can then create a usability test. In most usability tests, the test is usually comprised of scenario tasks, which you give one at a time to each tester. The scenario task should set up a brief context for the task, then ask the tester to do something specific. The task needs to be general enough that the tester has latitude to complete the task the way they would normally do it, yet specific enough that both the test moderator and test volunteer would know when the task is completed.

Take notes while your testers use the software. This is usually made easier if you ask each tester to speak aloud what they are doing or what they are looking for (if they are looking for a Print button, they should say "I need to print, and I'm looking for a Print button"). Also note any comments they make while working on each task ("This seems difficult" or "I can't seem to find the Print button" or "That was easy").

And that's the process Diana, Renata and Ciarrai followed when they did their tests. Diana's UX test was a little different, because UX is not the same as usability so required a slightly different test design, but still she asked testers to explore GNOME via three broad scenario tasks.

After the test, you collate your notes and analyze your data. For a traditional usability test, I find it is easiest to start with a heat map. I discuss heat maps elsewhere on my blog. From the heat map, it's easy to look for "cool" and "hot" rows, which reflect scenario tasks that were easy for testers or more difficult. This is often my first step to examine themes.

Both Renata and Ciarrai used heat maps in their analysis. Ciarrai's heat map analysis method was a variant on the typical method, due to their test being a paper prototype test. The testers didn't interact with a live version of the software, but a paper mock-up using screenshots of the new design. So Ciarrai's analysis was made more difficult, but they did a great job.

For Diana's test, I asked her to try a different analysis method. After each tester explored GNOME, she asked them to describe their initial reaction (their first few minutes) to GNOME by pointing to an emoji. Diana had prepared a list of ten emojis that ranged from "ewww" to "happy." She also asked testers to describe their experience at the end of the test using the same emoji scale. Diana then followed up with a brief interview designed to explore how testers perceived GNOME.

Notes on the UX test for next time:

I have mentored several traditional usability test before, as part of Outreachy and as an instructor for CSCI 4609, so we had a lot of prior experience going into Renata's traditional usability test and Ciarrai's paper prototype test. That's probably why both went very well. But this was our first time doing a user experience test, and while I think we got some useful results from it, I think we can improve the user experience test the next time we do a UX test.

First, I think we need more testers in the user experience test. In most usability tests, you don't need very many testers to uncover most problems and get actionable results. This assumes that each tester will uncover many of the same problems found by the next tester. Some usability researchers claim you only need to test with five testers. I agree with that, but you need to remember the assumption: each tester uncovers many of the same problems the next tester will find. Nielsen finds a single tester will uncover about 31% of the problems in a single usability test.

But in UX testing, you aren't uncovering usability problems. Instead, you are examining how real people respond to the software. You are looking for something akin to emotional attachment, not ease of use. So five testers is not enough. Diana initially included five testers for her UX test, and later expanded to seven. Even that isn't enough. I think the next time we do a GNOME user experience test, we will want around twenty testers.

We should also closely examine the number of emoji that testers respond to. I worked with Diana to select ten emoji that span a range of emotions. I based this test design on a common contemporary UX test design, including that used in an article in the Washington Post where the newspaper asked readers to respond to political events (specifically, the Republican National Convention) by selecting from a list of emoji. Over a thousand readers responded in that case. With such great numbers, the results quickly narrowed down to show the top one or two emoji responses for each political moment. For example, when responding to "I declare Trump and Pence to be nominees for President and Vice President," the responses tailed off quickly: 553, 334, 90, 79, 70, … That makes it easy to identify the one or two dominant reactions.

The next time we do a user experience test, increasing the testers will help us to consistently identify emotional response via emoji. We might also reduce the number of emoji, so fewer testers have fewer options to choose from. At the same time, we need to be careful not to restrict the responses too much or we risk artificial results.

To offset this, and in addition to emoji, we might also ask the testers to pick three or five adjectives from a list that describe their reaction to the test. We can use synonyms of various emotions, so testers have a wide range of words with which to describe their reactions, yet we can combine results more easily. For example, synonyms of "confused" include baffled, bewildered, befuddled and dazed. If a tester felt confused at the beginning of the UX test, they might pick three adjectives "baffled, bewildered, confused" that would give a score of +3 to "confused." Similarly, synonyms of "happy" include cheerful, contented, delighted and ecstatic.

A few thoughts for future mentors:

Mentoring in Outreachy is a very rewarding experience, but it's also a lot of work.

When mentoring three interns in this cycle in Outreachy, I assumed the effort would be about the same as my online class with ten students. The research phase was certainly straightforward; we were researching the same topics. But once we entered the project phase, and Diana, Renata and Ciarrai each started work on their own projects, I was constantly switching gears as I reviewed each intern's work and provided direction and guidance. I think I did a good job here, but it came with a ton of effort on my end.

As weeks progressed in the project phase, I began to rely more heavily on Jakub and Allan to help me out. I seemed to constantly send them reminders to review that week's work and respond to the interns. I hope I haven't made a nuisance of myself or pushed too hard.

Looking back, mentoring three interns on three different usability tests was more work than my online class with ten students because the three interns were working on completely different projects, while students in my CSCI 4609 class worked on similar projects. In my online class, students had different starting points in their Personas and Use Scenarios, but they worked together on the same program for a "dry run" usability test. For the final project, students could pick any open source program that appealed to them, but in the "dry run" I picked the program for them to examine (based on common interests we explored in the first few weeks of class). This meant most of my energies as professor were pointed in the same direction. But in mentoring three projects for Outreachy, I was always getting pulled in a different direction.

I'm not sure how my experience compares to a project that focuses on coding. At a guess, I think it's not very comparable. Your mileage may vary.

My guess is the effort to mentor several interns in a coding project is probably the same as being the project maintainer for any active open source software project. I've been involved in open source software since 1993, and an active developer since 1994, so I know the effort isn't too great if you are reviewing code patches for a codebase that you are very familiar with. As maintainer of the program, you know where the puzzle pieces go, and you should have a good idea what the side effects may be for a new patch.

If you volunteer as a mentor to a future cycle in Outreachy, make sure to plan your time very carefully. Be aware of time commitments and how your work in Outreachy will intersect with other things you want to do. I over-committed myself, for example. I can't do that again.

A few thoughts for future interns:

One of the steps in your application to Outreachy is the first contribution. For the usability testing internship, I asked applicants to pick a few scenario tasks from previous usability tests, and do a usability test with a few test volunteers, then write a short article about your first contribution. I intentionally asked applicants to test with only a few users, usually less than five, because I wanted them to see what usability testing was like before they committed to the internship. Three isn't enough testers to get actionable results, so I think I will increase this to five testers next time.

I find that this first contribution really does provide an indicator to later performance. Applicants who did well in their initial contribution did well in their internship. We were able to predict performance (to some degree) based on their initial contribution. Diana, Ciarrai and Renata were accepted into the program because their initial contributions represented good work. Other applicants to the program who weren't selected may have done a good job but were not selected due to space, while others didn't seem to take the first contribution very seriously (they thought it was "busywork") or they waited until the last moment to do their initial contribution, leaving no time to evaluate their work before we had to make a decision.

If you plan to apply for a future cycle in Outreachy, don't forget the initial contribution. Take it seriously. Reach out to the project's mentors and discuss the initial contribution and how to approach it. I know I took into consideration each applicant's engagement and relative success in the initial contribution, and that mattered when selecting interns for the program.
image: Outreachy

Friday, September 9, 2016

Keeping DOS alive with FreeDOS

I wanted to share a recent interview with about the FreeDOS Project, an open source software project that's been close to me since 1994. Jason Baker from Red Hat interviewed me about FreeDOS, why we started it, and what to expect in FreeDOS 1.2 (out later this year).

Here's a brief excerpt from "Keeping DOS alive and kicking with open source":

DOS: the Disk Operating System. For many of us whose computer coming-of-age story spanned across the 80s and 90s, we remember it, fondly or not, as the gateway into our computers. But somewhere along the way, DOS gave way to graphical environments, and some of us opted to move to open source alternatives.

But in the free software world, DOS is still alive and well in the form of FreeDOS, an open source operating system maintained by Jim Hall and a team of dedicated developers who are keeping the DOS legacy alive well into the twenty-first century. And more than simply an existing code base, FreeDOS is still being actively developed and is approaching a new release in the near future ("When it's ready," according to Jim). FreeDOS has proved to be an important tool to numerous legacy applications powering critical systems which were never migrated to a more modern operating system.

I caught up with Jim to learn more about the FreeDOS project and where it's headed in this interview.

Read the article for the full interview»
image: Official FreeDOS fish (cc-by)
by Bas Snabilie, adapted by Mateusz Viste
from FreeDOS Images

Presenting at the Government IT Symposium

I'm pleased to announce that I will be presenting "Usability testing in open source software" at the 35th Annual Government IT Symposium, in December 2016!

As you might guess from the title, this presentation is about how to do usability testing. I'll use my previous experience in usability testing in open source software as an example, and walk through three different usability tests from this year's Outreachy internship with GNOME.

Are you planning to attend the Symposium? If so, here are the presentation details:
In this session, you’ll learn about different ways to perform usability testing. Jim has mentored usability testing for the GNOME Foundation, including three usability tests during Summer 2016. This presentation will discuss different methods you can use for usability testing, and general lessons that you can use now. Jim will also present results from the usability tests of GNOME from Summer 2016.
My presentation is scheduled for Thursday, December 8th, 2016 1:15 p.m. to 2:15 p.m.

Also, I'll be doing a separate presentation about "Leadership lessons from unusual places." Check out my other blog Coaching Buttons to learn more about that!
image: MN Government IT Symposium

Tuesday, September 6, 2016

Wrapping up user experience testing

This is a follow-up to my other item about wrapping up Outreachy. Diana posted part two of her analysis from first-time GNOME user experience testing.

In this test, Diana asked testers to simulate an "unboxing" of a new system. The tester logged in to GNOME using a fresh "test" account so they get the first-time user experience. After allowing each tester to explore GNOME via three broad scenario tasks, Diana asked them to rate their reaction to GNOME using emoji, and followed up with several interview questions.

What went well:

  • File management (Nautilus)
  • Browsing the web (GNOME Web)

What was difficult:

  • GNOME initial setup
  • Email setup (Evolution)

Diana's user experience test used only a few testers (initially five, expanded to seven) so it's hard to evaluate trends from this research. However, Diana makes these general recommendations based on her observations:

  • Add more colors
  • Reduce number of steps in initial setup
  • Simplify password experience
  • Relocate the input language settings

Diana goes into more depth in her discussion. Read Diana's analysis for more»
image: Outreachy

Friday, September 2, 2016

Making your first contribution to open source software

A reader recently emailed me to ask how to get involved in open source software if you haven't done that before. Great question!

I believe that open source software needs to be personal to you if you're going to do it at all. Open source software is a hobby—but it's more than that. I find myself thinking about the open source software projects I maintain, contribute to, and use almost daily. So if you're going to contribute to open source software, make it something you care about.

But how to get started? It can be somewhat intimidating to look at the software that's already been created and think "How can I contribute to that?" One easy way to contribute is by fixing bugs. Every program maintainer is always interested to receive bug fixes! So play with a program that you like. If you find a bug, dig into the source code and fix it. Or if you think the program would be better if it did things a little differently, add that new feature! Email your patches to the program maintainer and see if they accept it. Pretty much all the open source software developers I know are happy to receive patches that fix bugs or add new functionality.

Start by looking at the project website for the program you like. They may have a bug list or a bug tracker or some other bug database of "known bugs" or "known feature requests" that you could work on.

But here are a few projects I would recommend to you. These are of special interest to me:

​The FreeDOS Project
I'm also the founder and project coordinator for FreeDOS, a free implementation of the DOS operating system. We just discussed/reviewed a bunch of bugs on the mailing list, and folks have been updating the ​FreeDOS bug tracker. So the bug list should be updated. Most of the FreeDOS Project (and certainly the bugs listed in the bug tracker) are written in C and x86 Assembly. We use OpenWatcom for our C compiler, and NASM for our Assembler. Both are open source. You will need to install FreeDOS somewhere to work on it - I recommend installing FreeDOS in a virtual machine or PC emulator such as QEMU, Bochs, Plex86, VMWare, Virtualbox, or any other PC emulator.

Simple Senet
I wrote Simple Senet a few years ago. My original intention had been to write a playable version of the board game Senet. But as an experiment, I distracted myself by creating a Senet simulator, where two AI players compete: an "aggressive" player vs a "defensive" player. It's interesting to see which play style wins. It's written in C using ncurses for display to a text terminal. A few suggestions:
  1. Update Simple Senet so a real player can move pieces on the board. The rule system for Senet is very simple, so I imagine a human would use a keyboard up/down/left/right to select the piece they want to move, then space or Enter or some other key to move the piece.
  2. Implement a rules subsystem. Senet is a very very old game. It may be the oldest game known! Because it's so old, there are no written rules from ancient times. Experts have had to figure out how to play the game based on evidence; there is some agreement on the rules (PDF), but not everyone plays the same rules. I wanted to separate the rules into a "rules system" that you could select from. One way to do this is via command line options: senet --kendall to play with Kendall's rules, or senet --jequier to play with Jequier's rules, and so on for --tait and --bell. I was going to make these "top level" options as "macros" that are the same as combining a bunch of other command line options like --stop-squares=15,26,28,29,30 or --start=on or --start=off and so on for other options.
  3. Port the game to GNOME. But this is likely better accomplished as a complete re-write.

Atomic Tanks
This is a fun little game that I used to play a lot. I've even submitted some patches, long ago. I recall Atomic Tanks is written in C. In Atomic Tanks, you control a little tank and you combat other (AI) tanks using outlandish weapons. It's basically a trajectory game: angle and power. One change I would recommend is the fonts: the text fonts are pretty basic. You might see if you can find a better free font (for example, from Google Fonts) that you could load instead.

GNU Robots
I wrote this game a long time ago, but others have updated it after me. It was last updated in 2008. It's a game where you write a program for a little robot, then set it loose and watch it explore a virtual world. GNU Robots is written in C, but the robot programs are written in Scheme (implemented by GNU Guile). I recall the build process needs updating, so you might do that. Also, the game currently displays the whole world at once using tiles for each square. They are pretty small and can be difficult to see. I think it would be more fun to have the game "zoom into" the immediate play area (say, the surrounding 7×7 squares, assuming the robot is always in the center square) and display a smaller version of the entire map off to the side with a "highlight" square around the area that is currently zoomed. That would allow players to watch their robot more closely, but still see where the robot is positioned relative to the map.
image: Official FreeDOS fish (cc-by)
by Bas Snabilie, adapted by Mateusz Viste
from FreeDOS Images

Thursday, September 1, 2016

Wrapping up Outreachy

This summer, I mentored three interns as part of Outreachy. Outreachy (formerly the Outreach Program for Women) helps people from groups underrepresented in free and open source software get involved. This was my third time working with Outreachy, but my first time working with more than one intern at a time.

Diana, Ciarrai and Renata have wrapped up their usability tests and posted their analysis. I've highlighted several of their analysis already, but I wanted to share a brief update here, all in one article:

First-time user experience test
In this test, Diana asked testers to simulate an "unboxing" of a new system. The tester turned on the computer, watched the computer start up, and logged in to a fresh "test" account so they get first-user experience. After allowing them to explore GNOME via three broad scenario tasks, Diana asked the testers to rate their reaction to GNOME using emoji.

Diana has posted part one of her analysis, discussing the participants and providing an overview of their reactions. She will soon post part two of her analysis, to share results of the UX test in more detail and suggest a few recommendations for GNOME based on what was observed in the test.

Paper prototype test of new Settings application
The usability test with a paper prototype can be very similar to a traditional usability test with a final product. Ciarrai's usability test examined the new GNOME Settings application. This application is still being designed, so Ciarrai used a paper prototype in her usability test.

Traditional usability test of other areas of GNOME development
Renata's test was a traditional usability test of GNOME. This is basically the same usability test that applicants performed during their initial contribution to Outreachy. In fact, Renata's test examined how certain design areas in GNOME Photos and GNOME Calendar have improved since the usability test performed by Gina in a previous cycle of Outreachy.
image: Outreachy