Moderating Usability Studies

Presentation Date: 

Tuesday, May 24, 2016

Location: 

User Research Center, Lamont Library

Presentation Slides: 

This presentation was given on May 24, 2016 by Dorian Freeman, as part of a workshop for the User Research Center.

Full content of the presentation:

A timeline

Moderating a test should be easy, right? Not any different than a casual conversation with a friend? Nope, it's not quite the same. In order for the study to be successful, there are a lot of things to keep in mind. I’ve broken down the timeline of a usability study into 5 sections and included tips for each one:

  • Preparation
  • Welcoming the participant
  • Getting ready
  • During the study
  • After the study

Preparation: before the study begins

  • Be prepared
    • Test your test! You'd be surprised what will turn up. Use a coworker, a family member, a neighbor, the dog-- well maybe not the dog.
    • Expect the unexpected - be ready to pivot. Things never go quite the way you had planned, so be flexible, and have contingency plans.
    • Practice! (see test your test! above)
  • Be aware of your assumptions
    • Figure these out before you do the study. Are you assuming you know some things already? Try putting each thing you think you know about your users or about what they do on a sticky note and then sort each note into one of three categories: True (based on data), False (also based on data), and Need to Verify. It’s the ones you need to verify that you will need to test, if not in this study, in another, and at the very least, to be aware of.
  • Be aware of your biases
    • Soooo many things can influence our judgment. Take a look. Whut- there's even one for not understanding when you have a bias?

The beginning: welcoming participants to the session

  • Build rapport
    • Be just the ‘right amount of friendly’ to build a rapport. Too much and the session becomes too chatty. Too little and things get formal and cold. Find your inner Goldilocks.
  • Describe the study
    • Describe what the study is about and what your goals are briefly, but don’t go into so much detail as to influence the participant's perception of the study.
  • Make them comfortable
    • Remind the participant:
      • “We're not testing you, we’re testing the software.”
      • “What you tell us will help us make things better.”
      • “Please tell us everything, the good, the bad, the ugly. You won't hurt our feelings.”
      • “You may come across some things you aren’t sure of, that’s okay, that’s why we’re testing- I can answer your questions after we’re done.”

The beginning, part two: getting the tools ready

  • Forms
    • Have them sign the informed consent form (you have one, right?)
  • Discuss recording and data privacy
    • You should have a policy which states how long you retain recordings. Be clear about what you are using the recordings for. This may be in the consent form as well.
  • Explain "think aloud"
    • If this is a think aloud test, talk about what that is. Some people find it unnatural at first, but most fall into it after a little bit. 
  • Overview of the technology
    • Let them see the technology they will be using.

The middle: during the session

  • Your attention
    • Note-taking: it's usually better to have another person with you taking notes, so you are not distracted by writing during the session. If you can't have someone else join, then only write down the big revelations. Remember if you are recording, you can always go back and watch and listen later.
    • Active listening: stay quiet, except when you need clarity (see the ask questions bullet below). The impulse to react and to converse is very strong for some of us, but it will steal the focus from the study and the participant. Fight it!
    • Active watching: pay attention to what they are doing more than what they are saying; also remember to look at the participant's body language & facial expressions. What are they telling you?
  • Your demeanor
    • Stay neutral: control your own body language (show that you are engaged but objective), and keep facial expressions neutral (time to work on your poker face!) 
    • Let the participant lead, and follow their pace. Some people are really fast, that's okay. When they are really slow, be patient.
    • Don’t help until absolutely necessary: be patient and watch as long as you feel they are handling it ok. If frustration levels start getting high, have them move on with something cheerful like, "ok, let's move to the next part".
  • Ask questions (but be careful)
    • Probing- not leading questions: again, we are trying not to influence the answers. "Could you explain more about when you said, 'the pointy thing'?"
    • Use the 5 whys technique: after the 4th or 5th "why?" typically you get some clarity.

The end: after the session

  • Ask if there are any questions
    • Ask if anything wasn’t discussed that should have been. Make sure you have time for a followup conversation.
  • Give the gift
    • Water bottles for the win!
  • Say thank you and see them out
    • ...Like your parents taught you.

Conclusion

The success of your study is critically hinged on the value and quality of the data you gather. The way you interact with participants can make or break that quality. These tips should take you well in the right direction. 

Learn more:

How To Moderate Effectively In Usability Research, By Colleen Roller  
12 Expert Tips for Moderating Like a Pro, by Peter Hughes  
Effectively Moderating Usability Tests, by Beth Loring (UIE webinar)  
Identifying and Validating Assumptions and Mitigating Biases in User Research
Interviewing Users- How to Uncover Compelling Insights, by Steve Portigal 

Happy moderating!
Dorian