Diana will examine a "user experience" of a user's first exposure to GNOME
In this test, Diana will ask testers to simulate an "unboxing" of a new system. The tester will turn on the laptop or computer, watch the computer start up, and login to a fresh "test" account so they get first-user experience.Ciarrai will perform a paper prototype test of the new Settings app
The tester will experiment with the system, using a few scenarios (not scenario tasks) to suggest real tasks that real people might do, so the tester gets an opportunity to poke around and launch a few applications (such as web, music, files, etc). Since the tester will have a limited time to experiment with GNOME, we are suggesting three scenarios:
1. Documents, photos and music management
This makes a lot of sense for a first experience because new users will need to copy over their old data before they can use the computer. This is also a great opportunity to see how new users approach the folder structure: Documents, Downloads, Music, etc. Diana will also create a "forest" of sample files for the tester to work with: a bunch of music files, a bunch of photos, a bunch of documents in different formats (MS Word .doc or .docx, MS Excel .xls or .xlsx, LibreOffice/OpenOffice .odt and .ods, and plain text). We won't put them in the same folder names that GNOME uses, but will organize them both randomly (files of different types mixed in a folder like "My backup" or "Copies of old files") and structurally with different folder names (like "Songs" and "My files" and "Class papers").
2. Sending an email
This is also a good scenario because almost every person uses a computer to send email at some point. We will leave this scenario open-ended. For example, rather than specify the Evolution mail program, the scenario will provide a general direction like "After you start up your new computer, you want to check your email." I'm curious how people will interact with this. Gmail users and other webmail users may use the browser to access their email. Others may explore Evolution.
3. Setting up a browser
This is another great first experience thing to do; almost everyone would likely use a web browser in the first few minutes of using a new computer. This scenario will encourage general use of the browser, such as asking the tester to visit a website like Google or their favorite news site.
After this brief experience, Diana will do a brief interview with each tester, asking questions like "Are you excited about it, or just interested, sort of neutral, or not that interested, or did you not like it?" Ask questions that uncovered how they approached the system. "Did it seem friendly in the brief time you got to use the desktop?" And the tester will sum up their first impression with an emoji to suggest their emotional response, and use a word association ("Describe this first experience in one word.")
The usability test with a paper prototype can be very similar to a traditional usability test with a final product. With the paper prototype test, we'll need to be careful about the test design.Renata will perform a traditional usability test of other ongoing work in GNOME
The major change in the new GNOME Settings is this: "A major feature of the latest settings designs is a rethink of the GNOME Settings “shell” (that is, the overall framework of the settings application). We want to move from the current model, that uses an icon grid and fixed window size, to one that uses a list sidebar for navigation, and has a resizeable window."
Ciarrai has already written several draft scenarios for the prototype test, and now she is refining them into scenario tasks.
In the paper prototype test, Ciarrai will present these scenario tasks to the testers and ask them to indicate what list item they would pick. Remember that a scenario task sets up a brief context, then asks the tester to do something specific. For example: "You brought your laptop to a local cafe and you've already connected to the wireless network using the shop's shared wireless password, which was printed on your receipt but you've since thrown it in the trash. (Connecting to wireless networks happens outside Settings.) The person next to you asks how to get on the wireless network. Which Setting would you try first to find the wireless password so you can share it with the person sitting next to you?" (That one is kind of long.) Or "You're at the office and you want to show a work presentation with a projector. Which Setting would you try first to connect your laptop to the projector?"
This is basically the same usability test that applicants performed during their initial contribution to Outreachy.
In other cycles of Outreachy, we've started the traditional usability test analysis by looking at the design patterns that we'd like to evaluate, and building up the test based on patterns and features, matching the patterns/features of interest to the scenario tasks. But we've probably carried that as far as we can in previous usability tests. Sanskriti and Gina (and my own work) have uncovered design patterns that work well and others that need improvements. There's not much to be learned by repeating tests on design patterns that work well, so we will shift this test to look at those design patterns and design pattern implementations that have been improved in recent GNOME design iterations.
Renata has also performed her own analysis of GNOME design patterns.
Looking at Gina's usability test results, testers struggled to find the zoom button in Photos (header bar button), changing the month/year in Calendar (header bar buttons), searching (header bar button) and copying in Characters (primary window button), annotating and bookmarking in Evince (header bar menus), and other tasks in Nautilus (several were header bar menus). So the scenario tasks for this usability test will focus on Photos, Calendar, and Evince—with attention to header bar buttons and header bar menus (other design patterns too, but those two seem important).
This week, Renata is writing scenario tasks that exercise header bar buttons, primary window buttons, header bar menus and application menus. She will reference Gina's test and re-use any scenario tasks from her usability test that uncovered problems for testers. Re-using scenario tasks will allow us to better compare how design patterns have improved.
In each case, Diana, Ciarrai and Renata will also write a pre-test "script" to "set up" the test. You need to have something written down so you say the same thing to every tester. This script will inform the tester that our usability test will examine a free software desktop system. Something like: Just like Windows is a desktop system and MacOS is a desktop system, GNOME is a free software desktop system. And you'll be doing a few tasks using GNOME and a few GNOME applications. I'll give you the tasks one at a time. We aren't "testing" you. The test is all about the software. If you have problems with part of the test, that's okay; that's exactly what we are trying to find. So don't feel embarrassed if you have problems doing something. Also, I'm going to take notes while you're doing these tasks. It will help me if you talk out loud when you are doing something, so I can take notes. So if you're looking for a Print button, just say "I'm looking for a Print button." And move the mouse to wherever you are looking on the screen, so I can see where you're looking.
Diana, Ciarrai and Renata will also write their own wrap-up interview questions. These will likely be specific to each test, and they may ask some questions based on how each tester performed in the test (for example, "It looked like you had problems doing __ in the test; what were you looking for, or what would have made that more obvious.") But they will also need to have some questions about the test itself: What seemed hardest to do? What seemed easiest to do?
Watch in the next few days for updates!