1. Figure out who are the target users for your program.
You probably already know this. Are you writing a program for general users with average computer knowledge? Or are you writing a specialized tool that will be used by experts? Take an honest look. For example, one open source software project I work on is the FreeDOS Project, and we figured out long ago that it wasn't just DOS experts who wanted to use FreeDOS. We determined there were three different types of users: people who want to use FreeDOS to play DOS games, people who need to use FreeDOS to run work applications, and developers who need FreeDOS to create embedded systems.
2. Identify the typical tasks for your users.
What will these people use your program for? How will your users try to use your program? Come up with a list of typical activities that people would actually do. Don’t try to think of the steps they will use in the program, just the types of activities. For example, if you were working on a web browser, you’d want to list things like: visiting a website, bookmarking a website, or printing a web page.
3. Use these tasks to build a usability test scenario.
Write up each task in plain language that everyone can understand, with each task on its own page. Put a typical scenario behind the tasks, so that each step seems natural. You don’t have to build on each step – each task can be separate with its own scenario. But here’s the tricky part: be careful not to use terminology from your program in the scenario. Also avoid abbreviations, even if they seem common to you – they might not be common to someone else. For example, if you were writing a scenario for a simple text editor, and the editor has a “Font” menu, try to write your scenario without using the word “Font.” For example, instead of this: “To see the text more clearly, you decide to change the font size. Increase the font size to 14pt.” write your scenario like this: “To see the text more clearly, you decide to make the text bigger. Increase the size of the text to 14 point.”
In my research question, I propose investigating the new design patterns in GNOME, asking "How well do users navigate/understand/… the new design patterns in GNOME?" A practical approach to this research question would be to measure the success/impact of these design patterns on typical behavior/activities. And usability can be measured; it is a quantitative practice. One way to explore the research question would be to conduct a usability test against prototypes (functional or on-paper) of the updated applications, using the new design patterns. By testing several applications that use the same or similar design patterns, the results might identify areas where the design patterns work well vs where they do not work well.
So my first step is to determine who are the target users for GNOME. From that, I can then decide the applications that should be tested. GNOME designer Allan Day suggested a list of GNOME applications that use the new design patterns concept. I might use any of these in my usability test:
While these applications are all good candidates for a usability test, the target users may define which applications (and thus, design patterns) would make best candidates for a usability test.