Test Fest: 5-in-1 Usability Testing
The idea for this study was borrowed from a study conducted at the Harvard User Research Center. The test consisted of five 20-minute usability tests, with five participants who rotated through each of the tests over a 2-hour period.
Test Fest was about more than trying something new. The undertaking required our team to work together and provided an opportunity to work with others curious about UX. The format also allowed our team to conduct usability tests 5 distinct pieces of the (massive) UNC Libraries website simultaneously.
Why did we select this experimental design?
- Saves time: Conduct 5 tests at once!
- Saves money: One larger incentive over many small ones.
- Backlog of testing: Lots of things to test? Try it.
- Experiment: Try a new methodology.
- Involve more people: Spread the word about usability.
Due to the long format of the test, we used a variety of techniques to ensure our recruitment goal would be met:
- Email to contacts collected from a previous usability survey
- Flyers posted around campus with links to sign up
- Heavy incentive ($40 gift card in exchange for a 2-hour session)
- Free snacks and drinks on testing day (helps with fatigue!)
Overall, our efforts were very successful, with 80 total respondents!
Staff: 2 fulltime staff and 6 grad students divided across 5 tests (4 moderated and one automated) including one time-keeper
Participants: 10 UNC Chapel Hill undergraduate students (5 for each day of testing)
Location was the largest hurdle. We needed a big space with lots of little spaces inside. In the end, we opted to rent out a suite of empty study carrels in the University library.
- Small, private rooms
- Quiet environment
- Less preparation time needed
- Difficult for participants to find
- Rooms a bit too small
- Sunlight made some rooms uncomfortably warm
Other preparations included borrowing laptops, moving furniture, stocking refreshments, and creating signage such as room labels and directional signs.
Take a look at the final setup:
Schedule of Events
Pre-Fest: welcome, name tags, consent, paperwork
Hour One: participants are run through their first 3 tests, rotating every 20 minutes
Hour Two: break for snacks, followed by the remaining 2 tests (again, 20 minutes each)
Post-Fest: general debrief of participants
With so many moving parts, we opted to run a full 2-hour pilot test the week before the study. Five grad students with an interest in UX were recruited as stand-in “participants,” and offered free donuts and experience. We tested the schedule of events as well as each of the tests, making changes as needed before the true testing would begin.
Test #4: The Sketch Test
Each of us in the Usability Department had the opportunity to design and moderate one of the usability tests. For my test (#4), I evaluated the homepage of the UNC Libraries website using a method deemed the "Sketch Test."
The Sketch Test required participants to create sketches of their ideal webpage design. Participants were alloted five minutes for each drawing, and asked to think aloud as they worked on their sketches. The test consisted of three individual tasks:
- "Design your ideal library website homepage from scratch."
- "Please annotate this image of UNC's current library homepage, making any changes you see fit."
- "Thinking about the previous tasks, design a new library website hompage for UNC."
You can learn a lot about design from people who don't know what they're doing. Even more important than the actual sketches produced were the stories my participants told as they created them.
In addition to compiling individual usability reports from the results of each test, we wanted to look at the big picture. Because the tests shared a common theme (i.e., the UNC Libraries website), we thought it would be interesting to dump all of our findings into one large pool and look for patterns. After typing up our notes, we met a week later to create an affinity diagram from the results. This was later coded into a spreadsheet for easy reference.
Schedule your test the morning after a big game.
February 9, 2017: UNC plays Duke in basketball (and loses).
February 10, 2017: We lose 2 of our 5 participants.
- Collect more data than you can reasonably analyze.
After two full days of test-festing and five completed usability tests, we were left with A LOT of data to analyze. It took us two 3-hour lunch meetings just to organize one giant affinity diagram!
- Be flexible.
- Try new things!
Despite the no-shows, our efforts would not go to waste! We ran the test according to plan. Later, we set up a third mini test session, running just those tests which required more participants.
The team learned a lot from experimenting with this new method. It was both fun and an efficient use of time. It also allowed us to include outsiders and educate them about the usability testing process while gathering insight from a variety of perspectives.
Special thanks to my usability team: Sarah Arnold, Chad Haefele, Stephanie Hsieh, and Grace Sharrar, and to our student volunteers: Gordon Chadwick, Julie Riegel, and David Tenenholtz.