Moderated by Helen Lee Lin as part of the UX Researchers’ Guild
This is an abridgment; view the full video presentation here.
Session 2 – December 3, 2022

As you approach your UX proposals, there are three crucial documents: a Research Plan, a Moderation Guide, and a Screener. Becoming familiar with these will help you be successful as a UXr Freelancer.

Compiling a Research Plan

Research plans, or research briefs or proposals, are similar from team to team. They use similar formats, although the headings may differ or be referenced by different names. Some teams may even have their unique format but they will likely revolve around the same information. 

To begin with, list the stakeholders with whom you have worked. Whenever you get to the interview stage with a company, they will want to know if you have experience working with project managers, engineers, or developers, as well as people in the C-Suite, including heads of Strategy, Content Designers (also known as Ux Writers). A potential employer or client wants to know with whom you’ve communicated and from whom you have gathered ideas. When you work on a team, research is collaborative. You may be the only researcher in the group, but you will receive feedback from different functions and give recommendations that will impact these areas.

Be sure and talk about your background. This is analogous to when you were in academia and conducting your Lit Review. You will create similar documents in industry although they might look different and be shorter. For these reviews, choose the best resources you have used internally at your company, helpful academic articles, or a mix of both. This helps explain why you are doing this study, why you are doing it now, what is the question, and why it is important to address.

Research plans don’t have to be long; shoot half a page to two pages at the most. Include relevant products you’ve worked on as well as past research you have done on the topic. Then you will discuss your research questions at a very high level to focus on general themes.

The research questions are not the actual questions you would ask in a survey or interview. These questions give your stakeholders an idea of what will be covered so they can give feedback and make suggestions. This information will help shape your survey or Moderation Guide when you get to that point.

Within the research plan, talk about what methods you’ve chosen. Typically you would not include why you chose the methods that you did. However, be prepared to explain this in an interview. Preparing and documenting this information while it is still fresh in your mind will also be helpful down the road when the details of a previous research plan might be forgotten. 

The need for incentives is something that we take for granted in the industry. Usually, there is money to pay participants, but in the case of research, you might do as a freelancer, think of some nonmonetary incentives. 

Materials you would use might include a survey or Moderation Guide, designs, or a live website or app – anything to illustrate your plan to your users.

Preparing a Moderation Guide

For your Moderation Guide, don’t overlook the basic logistics of a one-on-one interview. Make sure you start the recording and that your participants have signed their research participation agreement. 

You would then prepare an introductory script which should:

• Set the context for the interview.
• Verify if participants have done this type of research before.
• Make them feel comfortable about participating, especially if it’s their first time.
• Get their consent to be recorded.
• Notify participants that they’re agreeing to screen share if you need them to do an on-screen walkthrough.

Communicate that there are no right or wrong answers; in research, we’re looking to improve. That means that they should not shy away from being critical and negative in sharing their thoughts and experiences. They will give more honest feedback if they know they are not going to hurt your feelings. 

Below the introductory script will be your question guide with questions organized by theme or task, followed by probes and links to stimuli.

Creating a Screener

A screener helps you select the right users for your study. If you have a niche product and already know and selected people to recruit, you may be able to just ask some demographic questions.

But if you are planning to recruit in some more anonymous, or large space, with people you are not familiar with, you need to know their motivation. Do they want to participate to troll you or because you have a great incentive? You want to be very careful with the questions you ask to weed out those who don’t qualify without tipping them off to what you’re looking for.

Here are a few tips for creating an effective Screener.

• Knockout questions placed at the beginning will help eliminate unqualified participants so they don’t waste their time going through the Screener.  
• Avoid leading questions which can keep potential participants from trying to guess the “correct” response.
• Consider including open-ended questions for one-on-one interviews. These will help vet participants for insightfulness, level of detail, and fluency of thought, and allow you to understand better what you might be getting with that particular participant if you choose to interview them.

As a freelance UX researcher, you will frequently use Research Plans, Moderation Guides, and Screeners. Familiarize yourself with them to be prepared for every client and their UX needs that may come along.

About Helen: Helen Lee Lin received her Ph.D. in Social Psychology and has worked in applied research in adolescent literacy and children’s nutrition, and with combat veterans with traumatic brain injuries. She took a career pause while living in Ankara, Turkey for 6 years and then transitioned to UXR in 2018. She volunteered for Hack for LA for six months and broke into industry in January 2021 with a contract at TikTok. She is currently a contract UXR at Meta, working on Facebook Groups.

Moderated by Helen Lee Lin as part of the UX Researchers’ Guild
This is an abridgment; view the full video presentation here.
Session 1 – November 5, 2022

When transitioning from academia to UX research, a portfolio can be a great way to introduce yourself, and showcase your talents, experience, and case studies. In this session, Helen Lee Lin explores aspects of portfolios and how to make connections to expand your offering as you move into the workplace.

Do I Need a UXR Portfolio?

While it is possible to get hired without a portfolio, you need case studies to present in an interview. So, at the very least, you need some visual format or presentation. Keep this presentation to yourself until it’s time for an interview or provide it to recruiting or hiring managers beforehand. Either way, you need something that demonstrates you have the skills they’re looking for. 

Either way, it doesn’t hurt to have a portfolio. There were times when Helen was going through a job application process when a portfolio was required and she didn’t have one. She knew she didn’t have time to pull one together, so she had to pass on those particular positions. If you already have a portfolio, you can plug it right in and that’s another job application that you wouldn’t have been able to fill out otherwise. 

What Should I Include in a UXR Portfolio?

You have probably seen portfolios for UX designers with splashy graphics and mock-ups of websites or smartphone screens. But that is not expected of a UX researcher. You would likely have a deck or a document that contains the following:

• Any literature search or background research that you have done before conducting your study.
• The findings and your recommendations. 
• The potential impact, if you are aware of it.

In this way, a portfolio is like a condensed research presentation or a poster, in that you might assume that the audience is not necessarily a researcher. It might be a designer, engineer, or product manager. Beyond these suggestions, don’t forget to address the thought process you went through in your various case studies.

• Why did you make the decisions that you did? 
• Why did you choose the methods, the particular users to survey or interview, or the length of the study that you chose? 
• How did you decide to deal with problems you encountered, and why? 

This is what recruiters and hiring managers want to see: the behind-the-scenes actions that led you to the study in the first place.

Can I Use Academic Research in a UXR Portfolio?

The short answer is yes, you certainly can, and people do. Depending on what types of companies you’re targeting, some of them would find this par for the course. For example, companies like Microsoft, Google, and Meta where Ph.D’s are often hired, are used to seeing academic research in a UXR Portfolio. They understand that you haven’t done industry research and tend to be fine with that.

However, there are other companies that are new to hiring people with advanced degrees or they want somebody who has demonstrable industry research experience. If unsure, talk to people who already work in such a company to learn their perspective. Or communicate with the recruiter or hiring manager to understand what they want to see in your portfolio. 

Because of this, your portfolio might be a collection of three or four case studies that showcase your various strengths, such as different methods, topics, or industries where you’ve done research. If you have more studies than this, you can cycle them out, depending on where you’re applying. There’s no need to include every possible case study that you have. You want to present those which will carry the greatest weight with the companies you are pursuing. 

Can I Include Ongoing Projects or NDAs (Non-Disclosure Agreements)?

Absolutely. You can include expected findings, recommendations, and anticipated impact. Though not as preferred as a completed case study, you can still present work that is under NDA, even if you need to make it vague. Be sure and communicate the status of any such projects to prospective employers. They absolutely will understand. They wouldn’t want you to violate any agreement you have with a previous employer because if they hire you, they want to know that they can trust you in the future.

If you need to leave out particular variables of a study, you can change some of the details and include more generalized findings. Then give the amount of information that impact-wise is either already publicly known or that you can share.

How Should I Present My UXR Portfolio?

The form of a UXr portfolio can be simple and manageable. While it might include a website, it can be as simple as a Google slides or Powerpoint presentation you’ve saved as a pdf. Upload your document to something like Slideshare and include that link on your application. 

Should I Stay Within a Certain Industry or Go into Different Fields?

Take time to think strategically about where you want to go in your career. For example, if you’re interested in fintech, start moving in that direction and conduct research in that area. You’ll gain experience with the variables and competitors and build connections in that industry.

That said, If you are happy where you are, and want to target a future career in that area, then continue to lean into that industry because you are already a expert in that subject are.

How To Identify Warm Connections for Potential Stakeholders

When we speak of warm connections, we’re referring to people or companies with whom you are affiliated in some way. You’re familiar with the space and more likely aware of what needs to be done. This is a huge benefit, considering we frequently have a relatively short time to complete a project. 

When Helen began her freelance career, she wondered how to gain experience. She didn’t know anyone who had a startup, ran a website, or had a small business. She quickly realized, however, that UX is everywhere and doesn’t have to be a digital product. The key is locating warm connections and then discovering ways you can assist them with your UX experience.

So, think about places where you currently volunteer professionally or in your personal life. Are you completing any type of service of running a program for a community or business? Any program evaluation is a type of user experience research. Do you have a side hustle like an Etsy shop? Or a Youtube channel where you interact with a community? These places, and many others, are where you might find those warm connections you’re looking for.

No matter the organization or team to whom you propose change, recognize the impact your suggestions could have on those in decision-making positions. Bring those individuals in early in that process, so they can offer their input and see the benefit of your recommendations. This can help minimize any possible resistance to changes you are proposing.

Where Can I Find Cold Connections?

Helen has met interesting entrepreneurs at Lunch Club. She has also gone to Fishbowl, something like LinkedIn, but anonymous. When posting on Fishbowl, you can see what role somebody works in, or what company they’d work for, even though you don’t know their name. However, that is a potential place to look around for ideas. Angel.co is part of Angel List, where co-investors and founders find each other. But, they are also often hiring as well. You could look on there for at least company ideas, if not an actual job. 

Also, look at volunteer organizations where you can find opportunities to gain experience. Consider taking an idea of your own to say Code for America, Hack for LA, or explore freelance talent sites like Upwork, Toptal, or Hirect. 

And then keep an eye out for any new startups or apps related to your passions. This might be a relatively new app or website you are familiar with, and they may not have a UX researcher or know about UX research. You could teach them about it and then propose a project. There are places you could approach to find out if they’re looking for UX help. Keep your eyes and ears open; those opportunities are out there.

How Important is Networking?

In closing, Helen highly recommends networking with other professionals. “I have found the majority of my positions through networking,” she shares. “Your network is everything career-wise. The people in our networks are important resources to not only look for potential stakeholders but also to expand our experience and increase our strengths as researchers.”

About Helen: Helen Lee Lin received her Ph.D. in Social Psychology and has worked in applied research in adolescent literacy and children’s nutrition, and with combat veterans with traumatic brain injuries. She took a career pause while living in Ankara, Turkey for 6 years and then transitioned to UXR in 2018. She volunteered for Hack for LA for six months and broke into industry in January 2021 with a contract at TikTok. She is currently a contract UXR at Meta, working on Facebook Groups.

Moderated by Susan Shaw, as part of
The UX Researchers’ Guild
This is an abridged version. View the full presentation here.

​In this session, Susan Shaw shares her expertise in UX Research by exploring the most effective methods she has used in the automotive industry.

Types of UX Research

Numerous methods are available and can be used in UX research. As explained on the Nielsen Norman website, the following chart compares “20 methods mapped across 3 dimensions and over time within a typical product-development process.” For more information, visit https://www.nngroup.com/articles/which-ux-research-methods/.

Of these, Susan chose to discuss in this session the following methods which are most common in the automotive industry:

• Usability Benchmarking
• Journal/Diary Studies
• Focus Groups
• Card Sorting
• Interviews
• Usability Testing

Using Usability Benchmarks in UX Research

Usability benchmarking is tightly scripted usability studies using precise and predetermined measures of performance, usually to track usability improvements of a product over time or compare with competitors.

All OEMs (Original Equipment Manufacturers) look at what other OEMs are doing well to improve their own features. An effective way to do this is by creating a form or questionnaire of what you want to know. Then using that as a guide, look at that feature in every vehicle you come across.

The following image shows the icons each OEM uses for the same dashboard feature. They are similar but may not be clear from one vehicle make to another, what the image means.

Other examples of where usability benchmarks are used in the automotive industry include:

• Exploring how many steps users take to turn off Forward Collision Warning systems.
• Understanding how users access the preset button for their favorite radio station, what shape/size the climate control knob is, and whether a driver can reach/adjust these controls without looking directly at them.

Journal/Diary Studies

Diary Studies are used to understand long-term user behavior and experiences, give contextual insights about real-time user behaviors and needs, and help define UX feature requirements. Participants in a journal/diary study use a mechanism (e.g., paper or digital diary, camera, smartphone app) to record and describe aspects of their lives that are relevant to a product or service or simply core to the target audience.

Diary Studies are typically longitudinal over an extended period of time (a few days to a few months) and can be done only for data that participants can easily record. During the defined reporting period, study participants are asked to keep a diary and log specific information about activities you as the researcher, are studying. To help participants remember to fill in their diaries, they may be periodically prompted through a notification received daily or at select times during the day.

(Image from: https://www.nngroup.com/articles/diary-studies/)

In a previous diary study, Susan combined the written log with video recordings. Qualified participants were given an instrumented vehicle (with cameras and other equipment) for 100+ days and asked to log their use daily. In addition to the data from the daily logs, researchers watched the video recordings from the installed cameras to record actual use versus reported use. In this study, she wanted to discover if people used a newly installed alternative technology or defaulted to using what currently existed.

These studies usually take place over an extended period for multiple reasons: (1) to get long-term data and (2) to ensure people were comfortable and past the initial awkwardness and acting naturally. Over time participants tend to “forget” cameras are there and act more naturally.

Focus Groups

Focus groups are an informal technique to assess user needs and feelings both before interface design and long after implementation. In a focus group, 4-15 users come together to discuss issues and concerns about the features of a user interface. The group typically lasts about 1-2 hours and is run by a moderator who maintains the group’s focus.

In a focus group setting, spontaneous reactions and ideas can be more readily observed.  In these groups, you can ask participants to discuss how they perform activities that span many days or weeks: something that is expensive to observe directly. However, researchers can only assess what customers say they do and not the way customers actually operate the product. Since there are often differences between what people say and what they do, direct observation of one user at a time should supplement focus groups.

Problems that can arise with focus groups is when one person monopolizes the conversation, when a participant doesn’t talk, or when someone just goes along with the crowd. The key to getting good data from a focus group is:

• To have a good moderator, one that not only keeps the group on task but also,
• Ensures that opinions and feedback from all group members are being represented.

(Image from: https://www.questionpro.com/blog/focus-group)

Card Sorting

Card sorting is a quantitative or qualitative method that asks users to organize items into groups and assign categories to each group. This method helps create or refine the information architecture of a site by exposing users’ mental models.

Part of making a site easy to use is organizing information so that people find what they’re looking for. Too often, content is structured based on what makes sense to the company, not to the users.  One of the primary ways to figure out an organization scheme that best matches users’ mental model is through card sorting.

Let’s imagine that you’re designing a car-rental site. Your company offers around 60 vehicle models from which customers can choose. How would you organize those vehicles into categories where customers can quickly find their ideal car rental? Your company might use technical terms such as family car, executive car, and full-size luxury car. But your users might not understand the difference these terms. This is where card sorting can help: ask your users to organize vehicles into groups that make sense to them, and then, see what patterns emerge.

As it relates to the automotive industry, Susan has done studies to determine the best locations for controls on a vehicular dashboard. She created three categories: Steering Wheel, Touch Screen, and Center Stack.  Her intent was to discover which controls needed to be duplicated and which did not. Instead of always relying on post-its and paper, she and her team have also put users in the vehicle and given them actual-sized buttons with velcro and asked them to place them where they best saw fit/what made sense to them when driving down the road and needing to make an adjustment.

Interviews Versus Usability Testing

A user interview is a UX research method during which a researcher asks questions about a topic of interest (e.g., use of a system, behaviors, and habits) with the goal of learning about that topic. Unlike focus groups, which involve multiple users simultaneaously, user interviews are one-on-one sessions, although several facilitators may take turns asking questions.

(Images from:https://www.nngroup.com/articles/user-interviews/)

Although you may feel that doing a UX user interview is simple and straightforward, there is more to a good interview than many people realize. You need to know how to dig a little more, make participants feel comfortable and willing to talk and keep participants on track.

Interviews give insights into what users think about a site, application, product, or process. They can point out what site content is memorable, what people feel is important on the site, and what ideas for improvement they may have.

Usability Testing/Clinics

Usability testing participants come into a lab, one-on-one with a researcher, and given a set of scenarios that lead to tasks and usage of specific interest within a product or service. In a usability-testing session, a researcher (called a “facilitator” or a “moderator”) asks a participant to perform tasks, usually using one or more specific user interfaces. While the participant completes each task, the researcher observes the participant’s behavior and listens for feedback. These sessions are best done in person but due to the pandemic, many have had to be done remotely as well.

(Images from: https://www.nngroup.com/articles/usability-testing-101/)

Which UX Study is best

So, now that you know a bit more about the most common research methods in Automotive, which one should you use? No one method is best in any given circumstance. It all depends on where you are in your research, what you are trying to find out, and how much information you want and need.

The following chart is a helpful guide to determine which method will work best at which stage of a research project.

You also need to understand what your budget is before looking at how many participants are involved, or where the testing will take place. Susan shared that when she was doing a study on Lincoln Sedans, she focused on an older population. In contrast, while testing the KIA Soul, she’d go to campuses or large cities such as Los Angeles or New York where she would find younger populations.

While this session focused on identifying methods for the automotive industry, these principles can be applied to any project. Be willing to do the work to understand your project, stakeholders’ expectations, and user involvement to create the most effective and successful study under any situation.

About Susan: Susan Shaw has a BSE, in Systems Engineering from the University of Illinois and an MSE, in Industrial Engineering from the University of Michigan, and is a Certified Professional Ergonomist. She has worked as an ergonomics engineer and consultant in automotive usability.

For seven years Susan was a Senior Human-Machine Interface/UX Engineer at Hyundai Kia America Technical Center, Inc. She is currently the ADAS Customer Experience Research Lead at Ford. Susan is also a Board Member of the UX Researchers’ Guild.

Moderated by Jillian Hudson, UXC; UXr Guild Board of Directors
This is an abridgment; view the full video presentation here.
Session 3 – December 8, 2022

Jillian Hudson shared in her first session that UX Strategy is the user plus the business. As the researcher, you combine users’ needs with business goals and create projects that help both sides. 

In her second session, Jillian shared some tips on identifying and creating UX champions, those stakeholders who can help you bring your UXr expertise to the table.

So, now that you have those stakeholders, what do you do with them? That was the focus of Jillian Hudson’s third session in her Leveling Up with UX Strategy series.

Collaborating with UX Champions

UX champions can be at multiple levels in a company. What you can accomplish may depend on where they sit within the organization.

With a product owner, for example, work with them on a project that you know has value to both the user and the business. Then, when you make the project a success, you both win. In this way, you can talk to and influence these stakeholders. Unfortunately, this isn’t a magic formula that is 100% successful. But you can work with the ones that will work with you. This will result in viable projects and a win/win situation.

It is then important to make others aware of the work that you’re doing together and how amazing it is that they are working with you. Take advantage of opportunities to provide kudos and praise within the company to be seen as a generous contributor.

If it is a VP of Sales that you need to collaborate with, showing them the ROI of fixing some of the user’s pain points can help them meet some of their goals. You can then help them see that they will sell more by removing inefficiencies. Find ways to make it easier for them which will help address the pain points and solve the problem.

Persuasion Tips with Stakeholders

Robert Cialdini and Steve Martin wrote a book in 1984 called “The Science of Persuasion.” In this work, they outline six suggestions on how to work with, and persuade people to your side. But these ideas must be used authentically and ethically to bring about the desired results.

Here are their six shortcuts and how they might apply in your interactions with stakeholders.

1. Reciprocity: Obligation to give when you receive. Key: Be the first to give and make it personalized and unexpected
2. Scarcity: People want more of the things they can have less of. Key: Focus on the unique benefits and what they stand to lose
3. Authority: People follow the lead of credible, knowledgeable experts. Key: Signal to others what makes you an expert before you attempt to influence 
4. Consistency: People like to be consistent with previous behaviors & actions. Key: Ask for small commitments to new behaviors before requesting larger ones
5. Liking: We respond best to people who complement and cooperate with us. Key: Seek ”common ground.” Share genuine compliments.
6. Consensus: People will look to the actions of others to determine their own. Key: Highlight what many “similar” others are already doing

Understanding Stakeholder Expectations

When working with a new group or product team, try to meet with them one-on-one. Ask: “What are you working on? What are your pain points? How can I help you?” Go in assuming that whoever you’re going to meet with wants to work with you to be successful until they prove otherwise. 

If they’re not going to be cooperative, help them as best as they will allow, and then spend your energy with the people who want to work with you.  

To understand expectations, you must understand your communication style and that of the stakeholders you’re working with. This is more than just speaking the language of business to business stakeholders. Knowing this will make your client more comfortable and receptive to what you share. It’s all about what Jillian refers to as “turning the UX lens inside.” It’s not about the users so much as it is about the user experience of working together on a product team.

Building Authority

Since we are all fallible, how can you effectively build authority amid mistakes and encourage others to share their thoughts with you? This is indeed a process requiring confidence on your part and trust on the part of the stakeholders. 

If ever the adage “Honesty is the best policy” applies, it is here. Owning your mistakes may seem like a step backward, but in reality, it helps your stakeholders know that you are taking responsibility and want to make things right. This in no way lessens your authority; it makes you human. And if you admit your mistake before anybody else notices, you’re coming from a place of power.

There’s no need to beat yourself up over errors or apologize profusely. Admit your mistake, clearly communicate how you will resolve the error, and then do it. You will be seen as honest, ethical, and trustworthy which will strengthen any relationship. And you want to be a trusted resource, a sounding board, and a brainstorming partner when your stakeholders are thinking through a business problem. 

Create a Long-term map

Jared Spool speaks on “envisionment” or looking down the road to where you want to be. You can do this as a researcher by bringing together key stakeholders and asking them where they want to be in five to eight years. They will likely state what they can accomplish with current resources. But to get them to envision a different future, you need to help them to focus on where they want to be, regardless of current circumstances. It be a challenge for many stakeholders. But it is the only way to create effective and lasting change. Once they have this in mind, do some reverse engineering to identify what needs to be done to reach that future objective. 

It will take time and effort and will be a new approach for many clients. But the benefits, both in the short-term and the long-term, will outweigh any discomfort. 

Using Strategic Discovery in UX Research

What is Strategic Discovery? It is continuous research: focusing not on a single project in a research study but on a user persona. For example, if you’re working with a sales team, you have a sales manager, a sales rep, and a sales system; that’s three personas. Strategic discovery emphasizes the need to look at all three personas to learn how they’re doing on the tasks that you’re making changes on and then identify their pain points. Ask yourself: “How can I help improve things for these users?” That will get the stakeholders that they want.

Presenting the results of a Strategic Discovery study is more complex and requires much more analysis as you’ll cover many disparate initiatives in the same session. But once again, the benefits will be worth the effort.

By incorporating these principles into your stakeholder relationships, your UX champions, you can be the person who sees the whole picture and can then guide them toward success.  

Jillian Hudson is NN/g UX certified in Research and Management. She worked as a UX Product Designer/Usability Expert for American Airlines and as a UX Strategist and Researcher for Bank of America, and Wells Fargo Bank. She is currently the Principal UX Research Strategist at XPO Logistics. Contact Jillian through the UXr Guild Slack channel.

Moderated by Michele L. Oliver, Ph.D., UXr Guild Board President, with Guest Speaker Reem Ayouby
This is an abridgment; view the full video presentation here.
Session 3 – November 10, 2022

In session two of “The Q&As of Quantitative UX Research Methods,” Michele Oliver identified various tests and when to use them. The first section here is a continuation of this previous discussion.

How to Do a Chi Square Test in Excel

For A/B Testing, we’re going to do a Chi Square test. An example of using this test was when users were asked to update financial aid holds, and they were in one of two conditions. In one case, the null hypothesis is that there was no difference. Before doing the test, there were a few calculations we needed to do. This would be easier in Spss, but if you don’t have access to that, here are the steps using Excel:

1. Set up your table with frequency counts. Here you will see the raw data. Then convert the raw data into frequency counts to see how many were successful in each group and how many failed per group.
2. Calculate the totals for the rows and columns.
3. Calculate what is called Expected Values in a new table. In a Chi Square test, the observed values are what you get in the test itself, and the expected values are the calculations that were made. We then need to compare these two values. Do this in a new table to clearly see the data. The expected value is equal to the row total multiplied by the column and then divided by the overall total. Exp = (Row total x Column total) / Overall totals.
4. Calculate the actual Chi Square value. To find this, take the observed value and subtract it from the expected value. Once you do that, you square it. Then divide that by the expected value. X 2 = ∑(O – E)2 /E. This is easier to do in Excel because you just build the formula, and then you can do this with all your values separately. Then add them all up to find that Chi Square value.
5. Calculate what is called Degrees of Freedom (df). Use a 2×2 matrix with two columns and two rows. To calculate this, it is Row minus one, multiplied by the Column minus one. df = (r-1) x (c-1).
6. Then put this calculation into the formula bar in Excel to calculate the p-value.
7. Summarize your results by comparing the two groups.

Using Advanced Statistics for UX Research

(The following sections were presented by Reem Ayouby.)

While basic statistics are sufficient for many studies, advanced tools result in a more in-depth analysis. Some researchers might shy away from complex mathematical equations, but it is possible to do this without knowing all the formulas. But first, let’s set the stage.

The Difference Between Function and Feature

A function is what needs to be done. This can be related to the Jobs to be Done framework. A feature is how the app enables the user to get that job done. These definitions are important because they can be used interchangeably. An example is the A/B Test where the same function can be performed in two different ways and one may work better. The function is the same; there are just two ways to implement it through features.

The Importance of Data Visualization

Data visualization allows users and stakeholders to see, understand, and interact with the information you are presenting.

Visualization tools show how different factors will change; how two variables behave together. If you add a third variable, you can observe any interactions among them. This is helpful when presenting information and helping others see the ramifications of adding other variables into the equation. And if you add an interaction term, it can get even more difficult. Exploring these tools can help identify which research questions these can answer for you.

Can we use correlation? (IV and DV) – the correlation visualization. 

http://statkat.com/visualization-regression-one-predictor.php

Can we use regression? (2 IV and DV) – the linear correlation visualization

http://statkat.com/visualization-regression-two-predictors.php

What if there is interaction effect visualization? – the interaction effect visualization

http://statkat.com/visualization-regression-moderation.php

What is Correlation?

Correlation tells us when two variables are moving together, and one is higher, the other one will also be higher, and if one drops, the other will drop also. That’s the correlation, but sometimes there is more than one variable. For example, perhaps you want to know which products an individual will use as they grow older. Instead of just looking at someone’s age, you may want to add a variable such as a personality characteristic. If both go up, then a certain dependent variable will change similarly. 

If two independent variables may influence an outcome, and then a third is added, it starts to get complex in terms of how we can visualize this and how well we can make sense of what’s going on with the data.

What is Quantitative UX Research?

To answer this, a group of research scientists at Facebook came up with the following definition:

“Quantitative UX research delivers insights about people. UX researchers often approach research projects with questions such as: What are the human motivations for using these products? How do people perceive and use the product? How do they react emotionally and physically to it? What do they like and dislike about specific features? What role does the  product play in their daily life?” Mary Nguyen and colleagues, 2017 

By removing human motivations, perceived use, reactions, likes, and dislikes, you can often discern some of these through your qualitative research. So, why use quantitative research to revisit these ideas?

Qualitative research usually involves small sample sizes, not interviews with hundreds or thousands of people. With this small sample size,  research results may or may not generalize to the wider population you hope will purchase your product. Quantitative research can give a sense of the magnitude and strength of the relationship and how to generalize these findings to this wider population.

An example of a framework that is used in UX research is the Perceived Ease of Use and Perceived Usefulness. As researchers, you may frequently focus on Perceived Ease of Use to see how easy a product is for people to use. If they are not using it, it might be too difficult. You will then need to determine how to make the process easier and more intuitive. 

But then there’s also the Perceived Usefulness. Getting the Perceived Ease of Use right will remove hurdles. But it still has to be useful, which means it needs to be perceived as having useful functionality. With that, you can go into a lot of depth.

This is a typical model often used in user research in academia, but it also introduces the idea of causal inference. The idea is that when the perceived functionalities are in place, that leads to perceived usefulness, which leads to use, which then would lead to an increase in subscription level. It may even lead to customer satisfaction and loyalty. 

The Dichotomy of Qualitative and Quantitative Research

The following breakdown of research approaches moves from qualitative methods through mixed methods to quantitative methods is from a presentation by Chris Chapman at the Quantitative UX Research Conference held in June 2022.¹

Example: Social Ethnography:

• Field Work
• Ethnography
• Interviews
• Discussion groups

Example: Digital pen

• Usability assessment
• Beta/trusted testers
• International studies
• Prototype testing

Example: Profile CBC

• Qual/Quant groups
• Satisfaction & CSat
• User profiles
• Survey research

Example: Sentiment overview

• Conjoint analysis
• Sentiment analysis
• Competitive analysis
• Psychometrics

Example: Feature demand

• Regression models
• Multivariate stats
• Casual modeling
• Market estimation

When looking at regression models, you are trying to understand how various factors affect your variable of interest, often referred to as a dependent variable in multivariate statistics. But, you cannot consider that many variables by using linear regression. This is because the relationship between these variables may become complex. The complexity must be captured by the statistical model for useful conclusions to be drawn from the analysis. 

A solution to this is to use structural equation modeling. A friendly and more forgiving variant of structural equation modeling is partial least squares structural equation modeling (PLS-SEM). It is friendly because it is visually appealing. And it is forgiving because the statistical assumptions which must be respected are less demanding than for other statistical methods with similar capabilities.

An example of where using structural equation models was helpful in a recent study was exploring the impact of social media use on well-being. This model helped address questions of the influence of motives and perceived social media functionality on social media use; how social media use affects an individual’s well-being; and how emotion regulation is involved in the outcome.

This scientific research revealed that although there is a risk for negative impacts, it is possible for social media use to have positive impacts on individuals’ well-being. With this knowledge, researchers could then help companies research and design electronic social networks that would have the desired impact on users and society.

The full presentation offers an in-depth analysis of the use of structural equation models in this study.

About the speakers:

Michele Oliver has a Ph.D. in Experimental Psychology with an emphasis on Psychophysiology, Statistics, and Research Methods. She has been a Senior Lecturer and Adjunct faculty member. She is currently a Principal UX Researcher at Ellucian, a provider of SAAS solutions for higher education. Contact Michele at michele@uxrguild.com or through the UXr Guild Slack Channel.

Reem Ayouby, Ph.D., is a UX researcher and data scientist who leverages a combination of qualitative and quantitative approaches in her research. Her doctoral research focused on the motives, perceptions, use, and impacts of social media platforms such as Facebook. Reem holds a doctorate in Business Technology Management, a master’s degree in Management Information Systems, and a bachelor’s degree in Commerce with a major in Management Information Systems from the John Molson School of Business, Concordia University.

¹ CN Chapman (2022). Two UX Research Cases. In CN Chapman, KZ Xu, M Callegaro, F Gao, and M Cipollone, eds. (2022). Proceedings of the 2022 Quantitative User Experience Conference (Quant UX Con). June 2022, Sunnyvale, CA.

Moderated by Michele L. Oliver, Ph.D., UXr Guild Board President
This is an abridgment; view the full video presentation here.
Session 2 – October 13, 2022

This session in The Q&As of Quantitative UX Research Methods series is an extension of the first session. In this session, Michele discussed different statistical tests and how and when to use them. 

What are the Different Types of Data?

Identify data to identify which test to use:

• Nominal – For this kind of data, think name. You can use any type of labeling (words or numbers) but the categories assigned don’t mean anything. For example, 1 = Mac, 2 = PC, 3 = Mobile. It’s more helpful to think of a name. You can use any kind of label. Categories are different, but there is no difference between the name and the label. 
• Ordinal – For this data, think rank or order. These have a specific order and are typically labeled with an order or rank. However, there is no difference between the categories. Rank or order is going to be important. Example – the months of the year: there’s an order. But the differences between the months don’t matter. In research, we speak of the five-point Likert scale, where one is very difficult and five is very easy. Those ranks mean something. 
• Interval – This categorizes precise and continuous intervals, but it has no true zero. As an example, think of temperature. If we went to zero, it doesn’t mean there is an absence of temperature – zero degrees mean something. 
• Ratio – This is the same as Interval data, but it has a true zero. Because the two are so similar, except that true zero, that data is analyzed the same way, so it is usually called interval/ratio data.

What Kind of Test Should I Run Based on the Type of Data?

When you can identify what kind of data you have, it will help narrow down the types of tests which would be used in any given scenario. First, let’s break down Categorical Data, then we’ll look at Numerical Data.

A. Categorical:

• Nominal – How many variables do you have?
  a. Only one variable – Chi-Square test – tests one variable with a distribution. Test against the norm.
  b. Two or more variables – Chi-Square test of independence. Testing them against each other.
• Ordinal Data:  Are you investigating relationships between variables or differences between groups?
  a. Relationships – Use Spearman’s Rank Correlation
  b. Differences – Are you testing differences between independent groups or within your groups (repeated measures)?
  i. Independent groups – Use Mann Whitney U
  ii. Within groups –  Use Wilcoxon Signed Rank Test

B. Numerical – Are you investigating relationships between variables or differences between groups?

1. Relationship – Do you have a true Independent variable (IV)?
   a. Yes – Regression.
   b. No – Is the distribution normal – can we assume a linear relationship?
   i. No – Spearman’s Rank Correlation
   ii. Yes – Pearson’s r
2. Differences – How many groups are we comparing?
   a. One group – Do I  know the population Standard Deviation?
   i. Yes – Z-test
   ii. No – 1 sample t-test
   b. Only two groups – Can we assume equal variances?
   i. Yes – Independent samples t-test (or paired t-test for repeated measures)
   ii. No – Mann-Whitney U or Wilcoxon Rank sum test
   c. More than two groups – Can we assume equal variances?
   i. Yes – Analysis of Variance (ANOVA)
   ii. No – Kruskal-Wallis Test

Statistics in Action

You must have a clear understanding of what you and your teams want to know and understand what your variables are so that you can identify the most appropriate test for your study. 

Usability Testing: Create a New Communication Plan

Steps for Running an Independent T-test Using Excel

1. Select “Data” from Menu
2. Select “Data Analysis”
3. Select t-Test: Two Square  …  Equal Variances
4. Select the range for the first variable
5. Repeat the second variable
6. Ensure that “Labels” is checked
7. Set “Alpha” to 0.05
8. Set a range for the output to appear
9. Write up your findings!

Some tips on using Usability Tests:

• As far as the sample size for a quant usability test, try to get about 30. The idea is that as you approach 30, your distribution starts to look more normal. That’s the rationale since your goal is to get as close to our normal distribution.
• Don’t be afraid to collect more than what’s asked. You have participants right there, so get as much information as possible. 
• The 0.05 is standard – you will want to allow yourself a 5% error
• If you wonder when time on task is most appropriate in a usability test, it depends on what your team wants to know. 
• Use a one-tail test if you are predicting the difference to be in one direction. But if you have two prototypes and you’re considering going in two different directions, then you should go with a two-tail test. 

Surveys: (JTBD) Jobs to be done

An example of when surveys would be used is when you have no existing data anywhere. You may have assumptions but no data. Conduct several interviews to identify needs and then create needs statements from that data, to be put into a survey to validate those needs. This is where the value of JTBD or Jobs To Be Done, as described by Tony Ulwick, comes into play.

Steps for Conducting a JTBD Framework

1. Conduct interviews to identify needs.
2. Validate needs via a survey (importance satisfaction.)
3. Create frequency counts for each of the responses. How many people reported Not at all, Somewhat for both important and satisfaction metrics.
4. Create importance and Satisfaction proportions. 
5. Divide the proportion by 10.
6. Calculate Satisfaction Gap Score = Importance Score – Satisfaction Score
7. Calculate the Opportunity Score = Importance Score + Satisfaction Gap
8. Rank order the opportunity scores.

What About Correlations?

The good thing about correlations is that you can throw them all into the analysis. Be mindful that the output will only look at pairs at a time. Because of this, you don’t have to do separate correlations for each pair that you’re interested in. 

Steps for Conducting a Correlation 

1. Select “Data” and then “Data Analysis.”
2. Select “Correlation.”
3. Select the Input Range from your spreadsheet.
4. Select the output range.
5. Select Ok.
6. Write up your results!

In the third and final session in The Q&As of Quantitative Methods series, Michele will explore more detailed statistics including A/B Testing, Chi-Square, and effect sizes. These more advanced calculations may seem daunting, but Michele will explain how all these, and many more calculations, can be done using an Excel spreadsheet. 

Michele Oliver has a Ph.D. in Experimental Psychology with an emphasis on Psychophysiology, Statistics, and Research Methods. She has been a Senior Lecturer and Adjunct faculty member. She is currently a Principal UX Researcher at Ellucian, a provider of SAAS solutions for higher education. Contact Michele at michele@uxrguild.com or through the UXr Guild Slack Channel.