How Important are Quant Skills to UX Research?

Hosted by the UX Researchers’ Guild
Moderated by Danielle Green and Jess Vice
This abridged version was written by Booker Harrap. View the full video presentation here.
Session 3 – September 21, 2023

It is clear that the definition of “quantitative skills” varies among researchers (and hiring managers). Even when definitions are aligned, folks have different views on the level of quant required to be a successful UX researcher. 

Danielle Green interviewed dozens of experienced UX researchers about their views on quantitative skills. The results fell into two common themes: 

1) Many researchers lack confidence in their quantitative capabilities. Sometimes these researchers are ashamed because they have no training in statistics, while some are ashamed because they “only know basic things like multiple regression and factor analysis.” Some researchers actively avoid any job descriptions that contain the word “quantitative.” They identify as “qualitative researchers” but are often happy to design, analyze, and report on surveys containing only Likert scale items. 

2) Some researchers self-report “mastery” of quantitative research practices but cannot define terms like “standard deviation” or “confidence interval.” This then naturally leads to the question: What should mastery consist of?

So, are quant skills required or just a bonus for UX researchers? Let’s dive into the discussion of the pros and cons of quantitative skills in UX research.

Pros of Requiring Quant Skills

Minimum Quant Skills Needed in UX Research

• UXRs should be able to calculate estimates of true values given a sample and statistically compare two or more groups. 

The minimum level of quantitative skills would include being able to compute estimates of the true value based on a sample of data, as in reporting confidence intervals, and having the ability to statistically compare two or more groups. This points to basic inferential statistics knowledge, such as familiarity with Anova, Chi-square, and T-tests. Comparing groups usually means proportions and reporting confidence intervals around proportions. 

Quant Skills Contribute to UX Research Success

• The two-part definition of success is 1) stakeholder buy-in and 2) making decisions based on research that generate value for the organization.

Success in this arena boils down to two main ideas. The first is stakeholder buy-in. When you report findings and present them to stakeholders, you want them to take these findings seriously. And second to this is research data-backed decision-making. When the organization takes an action based on your research findings, that action produces value, whether revenue, clicks, signups, etc. Your research insights will turn out to be right most of the time. 

Quant Skills Increase Chances to Be Hired and Retained

• When two resumes are roughly equal, the one with quant will likely win. Business leaders struggle to value qual-only research.

So, why is it good to be familiar with quant skills? The simple answer is getting hired, especially in this job market. All things being equal, the candidate with quant skills will probably get the job over the candidate who lacks those skills. Getting the job is just half the battle; you also want to keep that job. You do this by being seen as a contributor with unique value among your team members. Business leaders can struggle to note the difference between your qualitative insights and the junior PM who talked to five customers this morning and now has a two-year roadmap. Even if you have a Ph.D. in anthropology, they may not recognize the difference between qualitative research and the one who talked to a handful of customers this morning. Quant can help you retain that unique value that you are contributing to the team.

Qual Combined with Quant Skills Improves Operational Efficiency and Effectiveness

• When quant and qual exist together in the same person, the two types of data can be combined more effectively.

When researchers have both quant and qual skillsets, they combine both types of data in a single person’s mind and then deliver more powerful and effective insights. They can do this because they’re able to process both types of data together. If you have both these skills, you don’t have to involve other team members when synthesizing insights. 

Quant Skills Contribute to Accuracy

• If you make decisions based on counts and averages alone, research will be biased to “build it,” which is not advantageous for most orgs.

A final thought centers around accuracy and making the right calls based on research. Whereas those with quant skills may be biased to say that two things, groups, or designs are not different, those with less quantitative skills may report differences that may not be true in the user population.

Philosophically speaking, quantitative researchers tend to be more conservative and less likely to say yes to building an app, making changes, or adopting a more costly option. Less quantitative folks will be more prone to want to try new things. While there is a time and place for both, organizations generally hire researchers to de-risk things and will look for conservative and quantitatively inclined candidates with a sensitivity to those risks.

Cons of Requiring Quant Skills

Accurate (Qualitative Researchers) vs Precise (Quantitative Researchers)

• Because user experience centers around humans, it is impossible to be precise in most scenarios.

Jared Spool points to the difference between accurate and precise; qualitative researchers tend to be accurate, while quantitative researchers tend to be precise. In real-life scenarios, the only time we actually talk about precision is when the client asks about running a survey to statistical significance to which the response is to inform the client of a lack of funds and time. 

But generally speaking, most of the work done as UX researchers deals with humans who are unpredictable by nature. Because of this, it’s usually good, or at least good enough, to be accurate, to represent the trends rather than to be precise. The focus that quant brings a more conservative, granular, and specific approach sometimes causes us to miss the bigger picture. 

Qualitative Researchers Build Collaboration with Data and Teams

• Rather than stepping on toes by taking on someone else’s role, build collaboration with data, analytics, and business analyst teams.

There’s a symbiosis between purely qualitative researchers and purely quantitative data teams. Rather than trying to acquire the skills the quant team already has, can you show up instead as someone who complements them and brings a different perspective? In this way, you can challenge how they look at their jobs and build more collaboration internally, something every company wants.

Since a qual skillset is looking at people patterns, perhaps your time and effort would be better spent focusing on partnering with the business analyst teams already in place rather than trying to be a business analyst. Examining roles and responsibilities within any organization is something all researchers need to better understand. 

Cost-Benefit Analysis of Acquiring Quantitative Skills – Is it Worth It?

• Are quant skills going to be additive? How long will they take to acquire and how much will it cost? What will the real outcome be? Will this effort help me be a better researcher or get that dream job or promotion? 

These are legitimate questions that need to be addressed, especially when quant, or advanced math skills, are not your forte. If these questions make you feel uneasy or uncomfortable, then maybe the effort to acquire such skills is not worth your time and energy. In this case, perhaps it does make more sense to partner a quant person with a qual person, thus allowing each to be the expert in their field.

Data Misrepresentation Without Adequate Quant Training

• Without enough training and accountability, quantitative data can be skewed to tell the story we want. 

Another thing to consider is that without adequate training in quant skills or internal accountability on your part, can you guarantee that the quant numbers you’re sharing are accurate? The numbers might tell the story you want to tell, the one that makes your boss happy, or even the one that guarantees that raise you’ve been aiming for. But is it the correct story? There’s always the chance that you’ve skewed the data without realizing it, especially if you’ve recently been hired or are trying on a new set of skills you haven’t mastered yet. These conclusions need to be left to the experts. Companies would be wise to hire a business analyst or someone with a Ph.D. in numbers who has worked in CPA environments: an expert who can see the gaps and knows what to do about them. Without those quant skills, you, as a newbie, might not even know what could go wrong and how to fix those errors. Do you want that responsibility? Being familiar and careful with the data is crucial.

Quantitative Research and Masked Biases

• The diversity, representation, and accuracy of sources is easy to hide or ignore in quantitative data.

As good as quant tools are, they contain inherent biases that can’t be overlooked. Pollfish is one of those tools. It can be a UX researcher’s best friend, but keep in mind that you don’t know where the user panels are coming from. And with tools such as this, there’s no guarantee you will get the diversity or accurate representation of the groups you are surveying. 

There are unknowns behind the scenes that we don’t understand unless it’s made transparent. In quant specifically, people are more inclined to keep those details in the background because they don’t represent an interesting story. So, instead of getting the full picture, we end up with masked biases that, from the beginning, skew the data. This, of course, can be disconcerting when we’re talking about human data, about real people.

As a UX researcher, you will need to determine whether having quant skills in your toolkit is right for you. These pros and cons can serve as guidelines to make that all-important decision.

. . .

About Danielle Green (she/her): Danielle is a product and UX professional specializing in research and strategy (high growth, product-market fit). She is an Instructor and mentor with eight years in product, and five years leading teams. As a professor of practices at Claremont Graduate University, Danielle teaches the core courses for the User Experience MA in Applied Cognitive Psychology. She is also the founder and director of the Claremont UXR Laboratory (claremontuxrlab.com), a graduate student lab for UX Research, and has extensive experience in many domains, such as: e-commerce, SaaS, Edtech, Virtual Reality, and hardware.

About Jess Vice (they/them): Jess loves working with people and is curious and excited to understand what drives them to make decisions. Jess offers a deep background in qualitative research, user experience best practices, and high-level strategic planning and is particularly good at making meaning from research and using it to create data-informed strategies for creative and development teams. Jess is also intensely aware that the first point of contact is always the internal teams they work with — if a relationship is not built on trust, they know they won’t be a successful researcher and strategist. Jess has been working in marketing and advertising, CRO, SaaS, and product for over 14 years, and is consistently thrilled with how much more there is to learn.