How to Use Statistical Tests in UX Research

Moderated by Michele L. Oliver, Ph.D., UXr Guild Board President
This is an abridgment; view the full video presentation here.
Session 2 – October 13, 2022

This session in The Q&As of Quantitative UX Research Methods series is an extension of the first session. In this session, Michele discussed different statistical tests and how and when to use them. 

What are the Different Types of Data?

Identify data to identify which test to use:

• Nominal – For this kind of data, think name. You can use any type of labeling (words or numbers) but the categories assigned don’t mean anything. For example, 1 = Mac, 2 = PC, 3 = Mobile. It’s more helpful to think of a name. You can use any kind of label. Categories are different, but there is no difference between the name and the label. 
• Ordinal – For this data, think rank or order. These have a specific order and are typically labeled with an order or rank. However, there is no difference between the categories. Rank or order is going to be important. Example – the months of the year: there’s an order. But the differences between the months don’t matter. In research, we speak of the five-point Likert scale, where one is very difficult and five is very easy. Those ranks mean something. 
• Interval – This categorizes precise and continuous intervals, but it has no true zero. As an example, think of temperature. If we went to zero, it doesn’t mean there is an absence of temperature – zero degrees mean something. 
• Ratio – This is the same as Interval data, but it has a true zero. Because the two are so similar, except that true zero, that data is analyzed the same way, so it is usually called interval/ratio data.

What Kind of Test Should I Run Based on the Type of Data?

When you can identify what kind of data you have, it will help narrow down the types of tests which would be used in any given scenario. First, let’s break down Categorical Data, then we’ll look at Numerical Data.

A. Categorical:

• Nominal – How many variables do you have?
  a. Only one variable – Chi-Square test – tests one variable with a distribution. Test against the norm.
  b. Two or more variables – Chi-Square test of independence. Testing them against each other.
• Ordinal Data:  Are you investigating relationships between variables or differences between groups?
  a. Relationships – Use Spearman’s Rank Correlation
  b. Differences – Are you testing differences between independent groups or within your groups (repeated measures)?
  i. Independent groups – Use Mann Whitney U
  ii. Within groups –  Use Wilcoxon Signed Rank Test

B. Numerical – Are you investigating relationships between variables or differences between groups?

1. Relationship – Do you have a true Independent variable (IV)?
   a. Yes – Regression.
   b. No – Is the distribution normal – can we assume a linear relationship?
   i. No – Spearman’s Rank Correlation
   ii. Yes – Pearson’s r
2. Differences – How many groups are we comparing?
   a. One group – Do I  know the population Standard Deviation?
   i. Yes – Z-test
   ii. No – 1 sample t-test
   b. Only two groups – Can we assume equal variances?
   i. Yes – Independent samples t-test (or paired t-test for repeated measures)
   ii. No – Mann-Whitney U or Wilcoxon Rank sum test
   c. More than two groups – Can we assume equal variances?
   i. Yes – Analysis of Variance (ANOVA)
   ii. No – Kruskal-Wallis Test

Statistics in Action

You must have a clear understanding of what you and your teams want to know and understand what your variables are so that you can identify the most appropriate test for your study. 

Usability Testing: Create a New Communication Plan

Steps for Running an Independent T-test Using Excel

1. Select “Data” from Menu
2. Select “Data Analysis”
3. Select t-Test: Two Square  …  Equal Variances
4. Select the range for the first variable
5. Repeat the second variable
6. Ensure that “Labels” is checked
7. Set “Alpha” to 0.05
8. Set a range for the output to appear
9. Write up your findings!

Some tips on using Usability Tests:

• As far as the sample size for a quant usability test, try to get about 30. The idea is that as you approach 30, your distribution starts to look more normal. That’s the rationale since your goal is to get as close to our normal distribution.
• Don’t be afraid to collect more than what’s asked. You have participants right there, so get as much information as possible. 
• The 0.05 is standard – you will want to allow yourself a 5% error
• If you wonder when time on task is most appropriate in a usability test, it depends on what your team wants to know. 
• Use a one-tail test if you are predicting the difference to be in one direction. But if you have two prototypes and you’re considering going in two different directions, then you should go with a two-tail test. 

Surveys: (JTBD) Jobs to be done

An example of when surveys would be used is when you have no existing data anywhere. You may have assumptions but no data. Conduct several interviews to identify needs and then create needs statements from that data, to be put into a survey to validate those needs. This is where the value of JTBD or Jobs To Be Done, as described by Tony Ulwick, comes into play.

Steps for Conducting a JTBD Framework

1. Conduct interviews to identify needs.
2. Validate needs via a survey (importance satisfaction.)
3. Create frequency counts for each of the responses. How many people reported Not at all, Somewhat for both important and satisfaction metrics.
4. Create importance and Satisfaction proportions. 
5. Divide the proportion by 10.
6. Calculate Satisfaction Gap Score = Importance Score – Satisfaction Score
7. Calculate the Opportunity Score = Importance Score + Satisfaction Gap
8. Rank order the opportunity scores.

What About Correlations?

The good thing about correlations is that you can throw them all into the analysis. Be mindful that the output will only look at pairs at a time. Because of this, you don’t have to do separate correlations for each pair that you’re interested in. 

Steps for Conducting a Correlation 

1. Select “Data” and then “Data Analysis.”
2. Select “Correlation.”
3. Select the Input Range from your spreadsheet.
4. Select the output range.
5. Select Ok.
6. Write up your results!

In the third and final session in The Q&As of Quantitative Methods series, Michele will explore more detailed statistics including A/B Testing, Chi-Square, and effect sizes. These more advanced calculations may seem daunting, but Michele will explain how all these, and many more calculations, can be done using an Excel spreadsheet. 

Michele Oliver has a Ph.D. in Experimental Psychology with an emphasis on Psychophysiology, Statistics, and Research Methods. She has been a Senior Lecturer and Adjunct faculty member. She is currently a Principal UX Researcher at Ellucian, a provider of SAAS solutions for higher education. Contact Michele at michele@uxrguild.com or through the UXr Guild Slack Channel.