Test Fest: 5-in-1 Usability Testing

Overview

Background

The idea for this study was borrowed from a study conducted at the Harvard User Research Center. The test consisted of five 20-minute usability tests, with five participants who rotated through each of the tests over a 2-hour period.

Goals

Test Fest was about more than trying something new. The undertaking required our team to work together and provided an opportunity to work with others curious about UX. The format also allowed our team to conduct usability tests 5 distinct pieces of the (massive) UNC Libraries website simultaneously.

Purpose

Why did we select this experimental design?

Logistics

Recruitment

Test Fest Flyer

Due to the long format of the test, we used a variety of techniques to ensure our recruitment goal would be met:

Overall, our efforts were very successful, with 80 total respondents!

People

Staff: 2 fulltime staff and 6 grad students divided across 5 tests (4 moderated and one automated) including one time-keeper

Participants: 10 UNC Chapel Hill undergraduate students (5 for each day of testing)

Setup

Location was the largest hurdle. We needed a big space with lots of little spaces inside. In the end, we opted to rent out a suite of empty study carrels in the University library.

Pros:

  • Small, private rooms
  • Quiet environment
  • Less preparation time needed

Cons:

  • Difficult for participants to find
  • Rooms a bit too small
  • Sunlight made some rooms uncomfortably warm

Other preparations included borrowing laptops, moving furniture, stocking refreshments, and creating signage such as room labels and directional signs.

Take a look at the final setup:

Test Fest Map

Schedule of Events

Pre-Fest: welcome, name tags, consent, paperwork

Hour One: participants are run through their first 3 tests, rotating every 20 minutes

Hour Two: break for snacks, followed by the remaining 2 tests (again, 20 minutes each)

Post-Fest: general debrief of participants

Pilot Test

With so many moving parts, we opted to run a full 2-hour pilot test the week before the study. Five grad students with an interest in UX were recruited as stand-in “participants,” and offered free donuts and experience. We tested the schedule of events as well as each of the tests, making changes as needed before the true testing would begin.

Test #4: The Sketch Test

My Role

Each of us in the Usability Department had the opportunity to design and moderate one of the usability tests. For my test (#4), I evaluated the homepage of the UNC Libraries website using a method deemed the "Sketch Test."

The Test

The Sketch Test required participants to create sketches of their ideal webpage design. Participants were alloted five minutes for each drawing, and asked to think aloud as they worked on their sketches. The test consisted of three individual tasks:

  1. "Design your ideal library website homepage from scratch."
  2. "Please annotate this image of UNC's current library homepage, making any changes you see fit."
  3. "Thinking about the previous tasks, design a new library website hompage for UNC."

Results

You can learn a lot about design from people who don't know what they're doing. Even more important than the actual sketches produced were the stories my participants told as they created them.

sketch 1
sketch 2
sketch 3

Afterwards

Data Analysis

In addition to compiling individual usability reports from the results of each test, we wanted to look at the big picture. Because the tests shared a common theme (i.e., the UNC Libraries website), we thought it would be interesting to dump all of our findings into one large pool and look for patterns. After typing up our notes, we met a week later to create an affinity diagram from the results. This was later coded into a spreadsheet for easy reference.

lots of post-its
organized post-its

Lessons Learned

Don't:

Do:

Special thanks to my usability team: Sarah Arnold, Chad Haefele, Stephanie Hsieh, and Grace Sharrar, and to our student volunteers: Gordon Chadwick, Julie Riegel, and David Tenenholtz.