This usability test was a paper prototype usability test of the updated GNOME Settings application. Ciarrai included ten testers between twenty and forty years old, with good gender distribution (male, female, nonbinary) and pretty equal preferred OS platform (Mac, Windows, Linux) with slightly more non-technology professionals than technology professionals. Every participant claimed to use the internet daily. Only three of the participants had ever seen GNOME before.
Ciarrai presented testers with a typical scenario most users would likely encounter, and asked them to select the menu item in Settings they would try first. Ciarrai used twenty-three of these scenario tasks:
1. You set an alarm on your laptop. Now you want to make sure that it will be loud enough to hear from across the room. Where would you look in the settings app to change the volume of the alarm?The last three questions focused on the WiFi panel.
2. You want to simultaneously be to able to view your web browser and a text editor where you are working on a document. You have two monitors and want to spread your work space across them. Which setting would you use to enable this function?
3. You want to have your Thunderbird account to be the email that opens whenever GNOME is asked to open an e-mail client. Which setting would you try to make Thunderbird your official email?
4. You would like to know which version of GNOME you are running so that you can see if you are up to date. Which setting would you try to find out this information?
5. You are about to watch a movie on your computer. The internal sound is too low so you get out a Bluetooth speaker. Where in the settings app would you go to connect the speaker to your computer?
6. You are working on a few design projects and want to adjust the way your monitor handles colors so that none of the image quality is lost. Which setting would you try first to change the default colors?
7. You would like to be able to access photos from GNOME shell search. Which setting would you try first to see if photos can be indexed in a search?
8. You are a very skilled typist. You notice that a lag on your keyboard and want to speed up the computers’ response to your key strokes. Where would you look in the settings app for a way to change the keyboard speed?
9. You are going out of town for a week and having a friend housesit for you. She wants to use your desktop computer while you are gone and you are happy to oblige. Your friend’s native language is Spanish while yours is English. You think it would be nice to set your computer to Spanish for her so that she can use it more efficiently. Where would you look to switch the language to Spanish?
10. You prefer watching movies with the video player VLC. You would like to set VLC as the video player that GNOME connects to automatically. Which setting would you try first to make the change?
11. In a few minutes you are expected to be in a two hour long lecture. You want to take notes on your laptop during the talk. Where in settings would you look to see how much battery charge you have left in order to know whether you have to run to your car and get the charger before the lecture starts?
12. You’re at the office and you want to show a work presentation with a projector. Which Setting would you try first to connect your laptop to the projector?
13. You’re preparing for a remote work meeting. You wish to set a password that you will give to your coworkers so they can see and control your screen during the meeting. With which setting would you first attempt to do this task?
14. You find that you don’t like only having 5 minutes of idleness before your screen locks on your system. Where in the settings would you go first to adjust the wait time from 5 to 10 minutes.
15. You just bought a new printer. Which setting would you choose first to connect the new printer to your computer over WiFi?
16. You are living abroad for several months in a country with a time zone 6 hours ahead of yours. Where in setting would you go to adjust the time zone to reflect your new location?
17. You are listening to music on your computer. At the same time, you are torrenting a number of files. You don’t want the notification sound to disrupt your music each time a file successfully downloads. Which setting would you try first to mute the volume of the notification?
18. You don’t want to have alert messages come up when your screen is locked. Which setting would you try first to turn off these messages?
19. You are tired of looking at the default lock screen image on your computer. Where would you look first in the settings application in order to replace the lock screen image with a picture of your family that you have saved on your computer?
20. You want to be able to scroll with two fingers on your touchpad. Which setting would you try first to enable this scrolling on your touchpad?
21. You have the WiFi password for your favorite local coffee shop saved on your computer. While you are there one day working on your laptop the person next to you asks if you know the WiFi password. Where in settings would you look to locate the saved password in order to share it?
22. You’re at the office trying to get work done. The WiFi has gone down three times already today and you are frustrated with waiting. You have an Ethernet cable plugged in and set up already, which panel would you try first in the Network settings to switch over from WiFi to Ethernet?
23. You’re not getting a great signal to the internet. Where in the Network settings app would you go to change which WiFi network you are connected to?
Ciarrai started their analysis using a heat map, which I have reproduced here (original was html; I copied/pasted that into LibreOffice to update the formatting). Each tester is a separate column (labelled 1–10) and each scenario task is a separate row. A brief recap of each scenario task is on the left side, and the corresponding correct selection is on the right side. In Ciarrai's heat map, green represents a correct response, red an incorrect response, and orange a correct selection of Devices or Details but an incorrect sub-menu selection:
|click to view larger version|
With this heat map, Ciarrai performs a breakdown of observed themes: Which tasks were easy for everyone? Which tasks were hard for everyone? Which tasks were somewhat difficult? Read Ciarrai's post for the full analysis»