When to Use Which Quantitative UX Research Methods

Moderated by Michele L. Oliver, Ph.D., UXr Guild Board President
This is an abridgment; view the full video presentation here.
Session 1 – September 8, 2022

When to use Quantitative Methods

In simple terms, quantitative methods are used anytime we want to observe users’ interaction with products and to better understand the impact of those interactions.

But quantitative methods, however useful, may seem daunting at first. By taking time to familiarize oneself with these methods, and know when to use them, they will become some of the most important tools in the UX researcher’s toolbox.

When to Use Usability Testing

This method is used when we want to evaluate users’ behavior as they interact with a product. Here’s an example where this method would be useful.

“I would like to examine the validity of some assumptions I have about  how medical professionals navigate a site designed to present medical information to healthcare providers.” 

Usability testing is all about getting real people to interact with a product: whether it be a website, an app, or anything that you built, and to observe their behavior and their reactions to it. From usability testing, we can measure effectiveness by way of success, rates and the number of errors. We can measure efficiency by way of average time on task, the number of path deviations and then satisfaction through some type of survey. We don’t need to look at advanced statistical data; we can just look at number/frequency counts. The raw data alone can tell us a lot about what’s going on in our study. Usability testing allows us to validate our product, the prototype which we want to confirm that our product meets users expectations. Usability testing helps us develop empathy, and to avoid that tunnel vision that can occur when we think we understand how a user feels.

Pros of usability testing: Involves users in the design process; Identifies users’ expectations; Improves users’ experience; Discovers hidden issues; Ensures the system’s functionality meets business requirements.

Cons of usability testing: Recruiting challenges; Random selection is not 100% representative of the population; May be costly (in time and money).

When to Use Surveys

A survey approach helps when we want to assess users’ opinions and experiences and to see how to incorporate quant methods in research. Surveys are also used to monitor ongoing satisfaction, identifying any opportunities for improvement. This method would be effective for the following scenario:

“I want to understand how satisfied users are with their current registration process as well as how important it is that they have specific (identified) needs addressed in the solution.”

Through interviews, we are able to derive needs and create need statements. It’s important to keep the following points in mind when creating surveys:

• In general, it’s best to keep them short. At times, however, it may be important to understand as many needs as possible which may require longer surveys. Best practice would say to shoot for at least 80 needs.
• Ask each participant to rate how important each need is for them.
• Ask how satisfied they are with the current solution.
• From the responses, which needs are being underserved, as well as those that are being overserved can be determined.

Pros of surveys: Economical in time (and perhaps money); Anonymity; Provides a measure of users’ opinions or experiences; Can be conducted frequently to foster customer engagement.

Cons of surveys: No personal interaction; Questions may be biased (priming effect, social desirability, yea-saying, etc.); Possibility of carryover or order effects; Survey fatigue.

When to Use A/B Testing

A/B Testing is used when we want to assess how well different options perform. The following scenario would be best analyzed through A/B Testing:

“I have a hypothesis about this, but I’d like to test which page option will bring in the most donations.”

It’s really straightforward. We have two groups where one is given one condition, and the other, the second condition. We then do a simple test, called a high-test, to see if there are any significant differences. 

Pros of A/B Testing: Clear evidence based on behavior; Improved content engagement; reduced abandon/bounce rates; Increased conversion rates.

Cons of A/B Testing: Time consuming; Only works for specific goals; Doesn’t improve a site that is already fundamentally flawed; No additional behavior insights.

When to Use Card Sorts

Card sort is a quant method in which participants group individual labels that are written on cards. This method uncovers how users think about things and when we want to review existing IA or plan new IA. This would be the best method with the following research request:

“I want to gain insight into how our products’ menu structure matches users’ mental models.”

This method has a qualitative feel because you’re using words and mental models, but it also has quant data underpinning it.

Pros of Card Sorts: Simple and inexpensive; Quick; User-focused; Provides great information.

Cons of Card Sorts: It Ignores tasks; Analysis takes more time than other methods; It doesn’t uncover the “why”.

When to Use Tree Testing

Tree testing addresses the idea of how easily users can find information and exactly where people get lost trying to locate what they are looking for; when we want to quickly see where the problems areas of a site structure exist. A general request that would benefit from Tree Testing might be:

“I want to see how easily users can find information on our app/website.”

Other questions that could be addressed through Tree Testing might include:

• Do my labels make sense to users?
• Is my content grouped  logically to users? 
• Can they find what they need easily?
• And if not, what are some of those blockers that are keeping them from doing so?

Pros of Tree Testing: Short test sessions; Can easily be conducted remotely; Quick data analysis.

Cons of Tree Testing: Basic form of the test means that there will not be visual elements that may help users navigate; Remote testing does not allow for follow-up questions (and the “why”).

Beyond these five methods, these additional quant methods can also be used in UX research:

• Analytics look at what people are doing with a live product. Where do they go? Where do they click? What features are they using? What pages did they decide to abandon? We can get really good information from analytics.
• Desirability Studies identify attributes associated with a product or brand by showing a prototype to users and asking them to describe the design and features from a list of descriptive words. 
• Eye Tracking Testing requires special equipment and shows where a users’ eyes go as they’re interacting with an interface.
• Heatmaps focus on a user’s mouse allowing visualization of which records they hover over, where they are scrolling and clicking. 

With all these methods, it’s important to remember that nothing we do is done strictly in isolation. We can couple quantitative methods with qualitative methods and follow up with asking questions. And some of the methods listed here can be both qualitative and quantitative. 

“There is a big difference between making a simple product and making a product simple.”
Des Traynor, Co-Founder of Intercom

How can qualitative researchers use quantitative methods?

• As a qual researcher, you already have transferable skills: you are a great moderator, listener and thinker.
• You can synthesize data, even if it’s numerical data.
• You already know how to put people at ease and have developed empathy for working with users. 
• You do need to be familiar with available tools, but you don’t need to have a strong statistical background

How do you demonstrate quant skills in a UX research portfolio?

• If you are not a designer, consider showing the jobs you’ve done and include a sampling of questions you’ve asked. 
• Include how you analyzed the data without making it overly complicated.
• Because sharing a product might include proprietary information, focus instead on telling  the story of the research. In this way, you are able to show your strengths and how you applied them in certain scenarios.

How do you interview participants who are reluctant to talk?

• Building up trust engenders confidence that what is done will be kept secure and confidential. 
• Users and researchers may both be nervous. Building rapport will help minimize many issues that might arise on either side.
• Making users feel comfortable will help balance the scale, and hopefully help them act as normally as they would in other situations.
• Spending some extra time just talking with users before diving into the research part of the interview may help to calm them down and loosen them up a bit. 
• Creating and using standard license forms can also help build trust in your team and will make people who might feel uncomfortable before the research starts.

How do you handle research participants who tell you that they liked everything?

• Remind users that you did not design the prototype, so they know that they are not going to hurt your feelings.
• Tell them that you want to learn from them, and know how they really feel. This helps them to let down their guard and be willing to share their honest thoughts. 

What job titles should a person target if they are more interested in research operations for UX?

• Look into joining Research Ops channels on Slack.
• Networking with others is a great way to find out how others have navigated this same path.

The Q&As of Quantitative Methods, Session 2 will be held October 13 with Session 3 on  November 10, 2022. Register here.

Michele has a Ph.D. in Experimental Psychology with an emphasis on Psychophysiology, Statistics, and Research Methods. She has been a Senior Lecturer and Adjunct faculty member. She is currently Principal UX Researcher at Ellucian, a provider of SAAS solutions for higher education. Contact Michele at michele@uxrguild.com or through the UXr Guild Slack Channel.