This written summary works well, and it's important to report your findings accurately, but the summary requires a lot of reading on the part of anyone who reviews the results. And it can be difficult to spot problem areas. While a well-written summary should highlight these pain points, the reality is that the reader will need to dig through the report to understand where testers ran into problems, and which areas of the software seemed to be okay.
When presenting my usability test results, I still provide a summary of the findings. But I also include a "heat map." The heat map is a simple information design tool that presents a summary of the test results in a novel way.
I first developed the heat map method in 2014 when sharing my research results in the usability of the GNOME desktop. The heat map I used then was a primitive visualization, using simple colored blocks.
Over time, I modified the heat map presentation to use a grid style. For example, I used this style of heat map when I discussed an online class I taught about the usability of open source software.
Throughout the development of my heat map tool, I have used the same basic three rules to create the heat map:
- Scenario tasks (from the usability test) are arranged in rows.
- Test participants (for each tester) are arranged in columns.
- Represent each tester's difficulty with each scenario task with a colored block.
The color indicates the relative difficulty of each task for each tester:
—Green if the tester easily completed the task. For example, if the tester seemed to know exactly what to do, what menu item to activate or which icon to click, you would code the task in green for that tester.
—Yellow if the tester experienced some (but not too much) difficulty in the task.
—Orange if the tester had some trouble in the task. For example, if the tester had to poke around the menus for a while to find the right option, or had to hunt through toolbars and selection lists to locate the appropriate icon, you would code the task in orange for that tester.
—Red if the tester experienced severe difficulty in completing the task.
—Black if the tester was unable to figure out how to complete the task, and gave up.
The colors borrow from the standard green-yellow-red indicators to suggest go-caution-stop from stoplights. The extra orange and black colors provide gradation to difficulty. In the heat map, red indicates a task with extreme difficulty, yet the tester was still able to complete the task. And black represents a task that was so difficult, the tester was unable to figure it out.
The colors also lend the heat map its name. The gradient from "cool" colors (green and yellow) to "hot" (orange and red, to black) imply increasingly difficulty.
You can create a heat map using different methods. In my first usability test results, I created the heat map by inserting different colored graphical "block" elements into my document. The constant width of the blocks produced the intended "grid" effect, where each row represented a scenario task, and each column represented an individual tester. In my subsequent usability test results, I created the heat map using a spreadsheet.
The spreadsheet method is much easier. You can create a heat map of your usability results in just a few minutes. Here is a demonstration of how to create a heat map using LibreOffice Calc, using data from a first contribution to usability testing (used here with permission):
First, enter your usability scenario tasks in the spreadsheet. I like to add some whitespace before my results; this will become obvious at the end when we set the borders to white.
Write a brief summary of the usability scenario task on each row. If your usability test involves different groupings of scenario tasks (in this example, different programs) then type those in a separate column. These will be used as "headings" in a later step:
When entering your usability test results, use "G" for green, "Y" for yellow, and so on:
Every modern spreadsheet supports conditional formatting. In LibreOffice Calc, just highlight the data cells you want to format, then select Format → Conditional Formatting → Condition… and set your formatting. For a heat map, define any cells with a value equal to "G" to have a green background, and so on for the other colors:
(Note to the LibreOffice folks: Conditional formatting in LibreOffice Calc is kind of rough. Can you make this easier to do? I knew the functionality was in there somewhere, but it took me about a minute to find conditional formatting, then a few minutes more to actually figure out how LibreOffice wanted me to do it. Hint for the rest of us: For each rule, use Apply Style → New Style… and set the cell style background to whatever color you need.)
The spreadsheet will automatically apply consistent formatting to every data cell you highlighted. Note that you can also set the text color to be the same as the background color, which will effectively "hide" the data text. But in my example I have left the text color as the default so you can easily see how the spreadsheet applies the conditional formatting:
Now you just need to do a bit of manual formatting to make everything look nice. I like to adjust the columns and rows so the data cells are square with centered text. Also set the vertical alignment for everything to "Middle." Merge the heading cells with empty cells to the right, then set a grey background:
Finally, highlight the heat map and a few rows and columns around it, then set the borders to solid white. This effectively erases the grid lines, but leaves nice-looking white lines between each data cell so you can easily follow columns and rows. In this example, I've not merged the heading rows, so you can see the results of the white borders:
The final heat map after merging the headings. Just take a screenshot of the heat map, and insert it into your results:
images: mine (data from first contribution to usability testing)