Recruiting and Fielding UX Research Study Participants

Moderated by Helen Lee Lin as part of the UX Researchers’ Guild
This is an abridgment; view the full video presentation here.
Session 3 – January 7, 2023

Once you have completed a research plan, which is aligned with the goals of the stakeholders, and have chosen your methodology, it’s time to recruit and field participants for your UX research study. In this session, Helen discussed screeners, incentives, interviews, cardsorts, and diary studies. 

To begin, she offered the following tips to aid in this effort:

• Due to the high rate of no-shows in some segments of users, consider over-recruiting. Plan on over-recruiting by 15-20% for diary studies, especially if the study lasts longer than one week or if tasks present a considerable burden.
• Consider recruiting distinct and extreme segments (users versus non-users, power users versus newbies, users who feel somewhat strongly to very strongly about X topic).
• Place knockout questions near the beginning when logical.
• For qualitative studies: Use 1-2 open-ended questions to vet the participant for insightfulness, level of detail, fluency of thought, organization of information as well as  what POV they can offer.
• UserTesting: break your segments into narrow recruitment “audiences” to better distribute recruitment between gender and age groups.

She also shared two common mistakes that can mar your recruiting: not considering who should be in your sample, using leading questions when screening, or giving away the study topic in the screener intro.

Using Screeners in UX Research

If you know the participants you will be interviewing, a screener may not be needed. But if you don’t and need to recruit outside your organization or a known pool of users, use this tool before signing on with people for an upcoming study. 

In looking at questions to use in a screener, move beyond yes and no queries. If you were screening participants to use a mountain biking app, for example, a question such as “Have you ever ridden a bike?” clues in the participant as to what you’ll be studying. If they’re motivated to get into your study because they wish to be paid, or for some other reason, they know to respond, “Yes.” These kinds of pointed questions, therefore, would not be ideal. It’s better to hide the target behavior of interest or the target attitude of interest and not make it so obvious. 

A better question for the above example would be, “Which of these have you done?” and then list options such as: “Rode a mountain bike; Cooked food; Called a friend; or Bought a ticket.”

The best approach to screen for this example, however, would be to ask questions along the lines of: “In the recent past, have you participated in any of the following activities at least once a month?” with options such as: 

• Completed a day hike
• Went mountain biking 
• Trained for a 5K
• Used a single-person kayak
• Swam for 30 minutes
• Attended a sports clinic

This route is specific and targets the type of person that you want. Check with your stakeholders to know how engaged of a user  that person needs to be for the upcoming study. Using this last approach, you can hide your desired response among other specific responses. This is certainly not the only question you could ask to achieve this goal. But this example shows a much more effective way to identify participants that would be target users of a mountain biking app.

Compensation for UX Research Study Participants

If you find people are not as willing to participate as you had hoped, look at what you are offering as an incentive or compensation. Consider a non-monetary benefit that the organization can offer to these individuals. Compensations or incentives will increase trust and communicate that you do value their feedback. Offering compensation of some kind is one of the best practices,  

Surveys and Unmoderated Interviews for UX Research

An unmoderated interview is a study that can be run on User Testing or on UserZoom, where the participant is usually either voice- or video-recorded. Similar to a survey, participants respond to a series of questions. In this scenario, you are not there to correct them if they go off course, you can’t probe them further if they don’t go into enough detail nor can you redirect them if they misunderstand the question.

Here are some tips to make un-moderated interviews work for you:

• Do a soft launch with a small N, then review the data or recordings to ensure participants understand all the instructions and questions and that all questions/images/skip logic are programmed correctly.
• Make sure participants understand what they are giving feedback on.
• Use carefully labeled & highlighted screenshots or photos of the experience they are responding to.

Some common mistakes that can and should be addressed before launching an un-moderated interview include:

• Using the wrong/broken/duplicate links
• Not allowing ‘share’ or preview access to Figma/Google files
• Writing unclear instructions or questions
• Using ambiguous or idiomatic phrases with international markets
• Not triple-checking branching and skipping logic

A last consideration is to instruct participants to record in a quiet, well-lit space and to check that their microphones are not covered up. The recording needs to be the best it can since it will be your main source of data.

In-depth Moderated Interviews for UX Research (IDIs)

The ideal scenario when running moderated interviews is to record the session or have another researcher who is also a trained note-taker. This will free you up to focus on participants’ expressions and nonverbal queues helping you to know when to probe more deeply. 

But sometimes you won’t have this ideal situation and will do all this by yourself. Whether you’re on your own or have the luxury of a second person or a recording of the interview, it’s always a good idea to take time to debrief time after the first one or two participants. Set aside an hour or so right after those first couple of participants to see how many questions you were able to get through. You might find that there are questions you can throw out while prioritizing or rewording others that participants seem to find confusing. You can then complete the rest of your interviews with a more streamlined set of questions and go back to the more low-priority questions if there’s time. 

Some common mistakes to be aware of with moderated interviews include the following:

• Not reminding participants to download needed software,forgetting to prep needed devices ahead of the session
• Not including an initial icebreaker with “easy” questions about them
• Noting the participant’s talking speed, whether they will require reining in or encouragement to speak
• Talking over/interrupting participants during quotable moments
• Spending too much time trying to build rapport or establish commonalities: this is an interview, not a conversation with a friend. Plan on listening at least 75% of the length of the interview.
• Sticking too rigidly to scripted questions (it’s okay to go off-script and dive into a tangent, especially if you have saturation on research questions from previous participants)
• Asking a scripted question about something they’ve already answered, making participants feel like you weren’t listening
• Leading the participant with biased phrasing, or using too many examples

Using Cardsorts for UX Research

If using cardsorts, conduct these in-person whenever possible or with screen share recording allowing participants to think out loud as they sort the cards. This is really where your insights will come from. After the session, run the analysis to see how they grouped the cards. It’s also important to know why they’ve chosen to group them a certain way, because people may group cards similarly, but not have the same reason for doing so. And don’t be afraid to jump in and ask any questions that may come to mind during this process

Some other considerations are to have participants review the cards they placed under each category at the end and make final adjustments to their categorizations before submitting the cardsort. If you intend to ask follow-up questions about specific cards (e.g., “Were there any cards that belong to more than one category?” or “Were any cards difficult to understand?”), ask them before the cardsort is submitted so they can look back at their sorted cards.

If you are running your cardsorts as an un-moderated study, ensure that the text is as clear as possible and that if examples are necessary, participants know where to find those examples. 

Some common mistakes with this method might be: including too few or too many cards (the task typically takes 10-20 minutes); not vetting the text on the cards which could cause ambiguity; and not providing context or examples for cards that may require some explanation.

Diary Studies for UX Research

Think of a diary study as a variable that you track on a daily or regular basis. If people have kept a calorie counter or food diary or tracked the number of steps they walk every day, that’s a form of a diary study. For participants to follow through with recording their experiences, make it simple and require no more than 10 to 15 minutes each day, with the initial onboarding questions taking a bit longer.

When using diary studies, keep the tests clear and make the study no more than 8 to 10 days long. The length is crucial because there’s going to be a lot of attrition unless you have highly invested users who are engaged and willing to do something for longer than that. In academia, this might include participants from your department at your university, people who are basically a more adaptive audience and would likely be available for a longer study. In industry, plan on a shorter study –  no more than 10 days. After that, there would be a lot of drop-off, and you would still need to compensate them accordingly. If questions were not thorough enough, invite these participants to come back for post-study interviews. These might include participants who offered interesting insights or those with very different points of view. During these later interviews, dive deeper into things only touched on in the diary process.

 

About Helen: Helen Lee Lin received her Ph.D. in Social Psychology and has worked in applied research in adolescent literacy and children’s nutrition, and with combat veterans with traumatic brain injuries. She took a career pause while living in Ankara, Turkey for 6 years and then transitioned to UXR in 2018. She volunteered for Hack for LA for six months and broke into industry in January 2021 with a contract at TikTok. She is currently a contract UXR at Meta, working on Facebook Groups.