Moderated by Danielle Cooley
Hosted by the UX Researchers’ Guild
View the full video presentation here.

The decision to become a UXR Consultant can be a difficult one to make. Beyond just making the decision, if you choose to leap, when is the best time to do so? Should you start with consulting on a part-time basis or jump right into full-time? You will also need to look at establishing and building your business and brand, determining which clients and projects to pursue, pricing, and more!

As a 15-year veteran UXR Consultant, Danielle Cooley brought a wealth of knowledge and experience to this discussion. Though not an easy decision, leaving the corporate world to become an independent consultant can bring great rewards.

Finding Your Why

In his book, “Know Your Why: Finding and Fulfilling Your Calling in Life,” author Ken Costa emphasizes the need to understand one’s sense of purpose or direction. Without knowing why you want to be an independent consultant, it’s easy to burn out and get frustrated. You may find yourself doing great work and making great recommendations, only to see them end up in a metaphorical drawer, unimplemented. This can leave you wondering why you should even bother. Knowing your personal why will help you find the right projects and keep you motivated to see your efforts and dreams to completion.

Identifying Where You Are and What You Bring to the Table

Once you understand your motivation, it’s important to determine where you are on your path to being an independent UXR Consultant and what characteristics you currently have to help you on your journey. Consider what traits you have that are conducive to success as an independent consultant. A UX researcher does not have to have all these skills, nor is this a comprehensive list. But it is a helpful set of characteristics to think about. 

* I am comfortable taking risks to achieve my goals. As an independent consultant, you will be placed in situations you may not have experienced before. Being able to take those steps into the unknown comes with the territory.

* I am a good communicator and networker. Learning how to negotiate and communicate goes a long way. People will learn to trust you and what you say you will do.

* I can sell my services confidently. While selling is not something that UX researchers tend to do very often, it is a critical skill for entrepreneurship.

* I have experienced UXR work in a variety of different organizational environments. This background may or may not be necessary depending on how you target your practice, but as Danielle shared, “It has been helpful to me to have worked in large enterprises, scrappy startups, consulting agencies, and digital marketing agencies. So, if you’ve always worked in enterprise UX research and your clients will be in enterprise UX research, that’s great. But having that variety gives you a few more opportunities.”

* I have built up a strong network of UX professionals (or am committed to using every opportunity to do so going forward). Danielle shared the importance of this skill because most of her work comes from individuals she has previously worked with – something that’s true for many other independent UXR consultants she collaborates with.

* I’m comfortable working extra hours at times to keep up with multiple projects and nonbillable activities. Your schedule is something you can control to a certain extent but beware: it can get out of hand pretty quickly.

* I have the discipline to create a financial safety net for times when I do not have enough billable work. It is the nature of this work: feast or famine. Danielle said that in 2022, she made about 70% of her annual income in November and December. While unusual in some ways, it shows you can’t assume that because there is money in your bank account today, it will be there in two months. You have to be okay with the lulls that will come in this business and be able to save for them.

* I know the buck stops with me, and I can handle setbacks emotionally. You have to take responsibility for your work and work through challenges that may come. It feels personal when something falls through, even though you know it’s not. But it can be tough.

Once you have identified what characteristics you already have,  you will be better prepared to address other pressing questions of the business. For a more in-depth look into these skills, check out “The UX Careers Handbook” by Cory Lebson.

Should I Charge an Hourly or Fixed Fee as a UXR Consultant?

No matter where you are on your path, the first question that generally arises is how to price your services.

One way to bill is to charge an hourly rate. To estimate your hourly salary or consulting rate, it is approximately what your annual salary should be* divided by a thousand. Why 1,000 when you work 2,000 hours annually in a corporate job? As a consultant, not only will you have hundreds of unbillable hours for which you need to be compensated, but you will also have a higher tax burden. For easy math, if your desired annual salary is $100,000 a year, divide that by 1,000 and your hourly rate is about $100 an hour. 

* Are you underpaid now? If so, be sure to account for that when determining your rate. Your contribution is valuable. Make sure you’re paid what you’re worth.

The alternative to billing an hourly rate is to have fixed-fee, project-based billing. In this scenario, you would offer your services,  A, B, and C  for $X,000. Even though this is a fixed rate, your hourly rate is still something to consider. You will need to estimate how long the project will take, but it is ultimately based on the value to the customer. 

When you as a consultant give stakeholders research-informed recommendations for what they need to change in their interface or how they need to move forward with their strategic planning, consider the following questions: What is that worth to them? What are they going to gain in revenue or efficiency? What costs will be cut in terms of returned product or expensive call center volume? 

If the value to the company is less than what your hourly rate would be to do that project, you don’t want to take it on. If your fixed fee is about your hourly rate, then you can start looking at other factors. And if that value is higher, then that value is higher and that’s okay. 

It’s important to not underestimate your contribution. Your work has value. Danielle has worked for the second largest healthcare system in the United States, and if they can increase their patient load by one-quarter of 1%, that’s worth millions of dollars. If she were to charge only $2,000 because it only took her 15-20 hours to complete, she is not being fair to herself. Her work is worth much more.

This graph from the UXR Guild offers some guidelines for what you ought to charge based on your title and work experience. This and other resources can be found at https://uxrguild.com/faq/#researchers-faq.

Pay rates vary based on the experience required, type of research, length of engagement, and the budget of the client. We cannot guarantee what rate you may be offered for a specific job, but these are representative pay ranges:

Title	Experience	Rate range ($/hour):
Mid-level Researcher	3-5 years	$80.00 to $110.00
Senor-level Researcher	5-10 years	$110.00 to $140.00
Lead Researcher	7-12 years	$120.00 to $150.00
Principal Researcher	8-15+ years	$130.00 to $160.00
Manager or Director	10-20+ years	$140.00 to $200.00+

What Does an Independent Consultant Do?

As Danielle’s friend Brian O’Neill shared with her, “If you want to do UX research all day, do not become an independent consultant.” There is so much more you will need to do. Simply put, “Be ready to be the chief everything officer.” The reality is that your duties will more than likely include, but not be limited to, the following roles:

• Salesperson
• Accountant
• Marketer
• Chief Strategist
• Contracts lawyer
• HR Specialist
• Operations Officer
• Office Manager
• IT Support

All this is in addition to doing the research work, managing recruitment, identifying the best software tools for the job, and delivering reports and test plans. 

Finding Your Niche in UX Research

The next thing to consider when diving into being an independent consultant is what kind of work you are going to do in your consulting practice. Another one of Danielle’s friends, Jim Barnthouse, shared this important maxim: “The riches are in the niches.” Identifying a niche will help you target your marketing and your messaging, whether in industry, health care, finance, or a sub-niche of one of these. 

While this may be true in theory, the practice can be rather intimidating. What would you do if someone were to ask you to do something else? You may still be able to do it. Niches may have different practices, but your familiarity with the process could be applied to other businesses. 

Possible niches could include:

• Industries (automotive, FinTech, healthcare, medical devices)
• Company size (startups, small, mid-sized, large, enterprise)
• Product categories (mobile, enterprise, B2B, B2C)
• Product types (Intranets, social media apps, Sharepoint)
• UXR stages (early discovery through alpha, beta, and release)
• UXR methods (journey mapping, field research, usability testing

If you don’t want to niche but still want to narrow your work, you can certainly find clients. For example, you could market yourself as a thought leader and find a variety of work. But be aware that if you are designing (or marketing) for everyone, you are, in essence, designing (and marketing) for no one. While you may feel uncomfortable identifying a niche as a UX researcher, specializing will generally be better for most consultants. 

When to Transition to UXR Consulting

While many characteristics listed above are factors to know when to make the transition, the most important one is having a financial safety net. Just because you leave your employment and take the leap, does not mean the money will start flowing right away.

Your consulting practice is business to business: a business is hiring you to do the work, and that takes time. You and the product owner or VP must be in complete agreement on the terms, scope, and timeline. Beyond this is the legal aspect of being hired to work for a company, with procurement taking months to get everything signed off and done, and before any revenue starts flowing in.

The ideal scenario would be to have a client lined up before you jump. In any case, you will want to start building up that safety net as soon as you can.

Owning your own business does offer a measure of flexibility of working whenever you want. But what that flexibility will look like for you will depend on other obligations you may have, such as family.

Technology Needed to Be a UXR Consultant

Another question that arises frequently is what technology or digital tools an independent consultant needs to get started. Besides the obvious need for a computer, Danielle strongly suggests getting a web domain and an associated email address. A domain costs about $20 and a linked Gmail will run about $6 a month.

Beyond these basics, you will also want to consider the following:

• Some sort of presentation, spreadsheet, or documentation tool. If you’re not ready to invest in Office, a less expensive option is Google Workspace.
• A cloud-based storage or collaboration system. Google Drive is included in a Google Workspace account.
• A meeting tool. Zoom accounts are available for free but come with limitations. Google Meet can help you accomplish the same objectives.
• An accounting tool. Wave is free. Also, FreshBooks and QuickBooks are some other options. 

Another aspect of the business to look at is specialized tools. But do you need tools specific to the UX industry and work? “Not necessarily,” says Danielle. UserTesting is expensive to purchase as an independent consultant, since it is targeted to the enterprise, and a license can cost tens of thousands of dollars. Danielle offered that she only uses UserTesting if her client has a license already. Optimal Workshop is a less expensive option at about $2,500 a year. Yet even with this, Danielle will do a monthly subscription for however long the project is going to last and then cancel it, making sure she accounts for that amount in the project proposal. The same with Loop 11.

If you plan on developing prototypes for testing, Figma is not the only option out there. Danielle has used PowerPoint, Mural, or Miro, to accomplish these same tasks for a whole lot less. 

While many of these tools are helpful, you don’t have to have a ton of capital to invest in starting your practice. It is possible to start on the side and build towards full-time consultant work as long as your current employment contract is amenable to that. 

How to Find Clients as a UXR Consultant

There are many different methods to find clients as an independent consultant Danielle has had the most success with word of mouth or referrals, either from past clients or co-workers. But what do you do if you are just beginning your practice and have yet to build up a large network?

Here are a few suggestions:

• Hone your internet search skills. Know how algorithms work and what you can do to get more noticed online.
• Look into opportunities to speak at conferences. This can be an especially effective way to get yourself noticed, especially if the conference is local. Even if it is the smallest attended session during that block, it is a way to make connections. The people who hear you may not need your services at the time, but they will remember you when that need comes in the future.
• LinkedIn is another resource. Posting about your skillset and availability can draw attention to what you have to offer. You can also use the site to demonstrate your thought leadership by posting frequently to expand your reach.

All of these have mixed levels of success depending on what you’ve decided to specialize in, what is easiest for you to do, and what you’re most likely to do. It’s like exercise; the best exercise is the one you will do. The best course of action to find clients is the one you can and will do consistently.

Is There a Difference Between Consulting and Freelancing?

While some people use the terms “independent consultant” and “freelancer” interchangeably, Danielle Cooley begs to differ. She intentionally does not use the term “freelancer” because of the connotation that can suggest it’s a side gig. But when you think about yourself as an independent consultant and small business owner who runs a consulting practice or consulting firm, this changes your mindset about who you are and what you do. 

Another word Danielle avoids using is contractor. She describes the difference this way: contractors are extra hands; consultants are extra brains. A contractor will often have a full-time 40-hour-week contract for X number of months or indefinitely. A contractor is hired to do specific research or build wireframes or take existing wireframes and add a layer of visual design using a specific style guide or design system. Whereas a consultant is hired with this directive: “Here’s the problem we’re trying to solve. How do you think we should approach it?” In this scenario, a company is buying a consultant’s expertise instead of (or in addition to) a developed skill. 

Differences Between Independent Consulting and In-house Consulting Services

As an independent consultant, you have experience or education in other fields that make you more valuable. Danielle has an Engineering degree, and her Human Factors degree is from a business school which gives her a deeper understanding of these three areas. In contrast, some consultants without similar experience will propose great UX but are not technically skilled to implement it, move the business forward enough, or support the business goals.

In addition, this is one place where niching can help you stand out. You may feel that your automotive experience is only valuable in that industry. But many skills carry over to, say, a financial services company where you can take what worked in this other context and with a new perspective, apply it in this new environment.

Independent consultants have every bit of value as those large companies or competitors that offer research services.

“I decided to start being an independent consultant after I got laid off from a consulting company,” shared Danielle. “My first prospective client told me they were considering hiring my former employer. I explained that if they had done so three weeks earlier, they would have gotten me and paid twice as much for the same work because of the overhead. They chose to hire me as an independent consultant. I did great work for them and they hired me again later.”

Should I Hire a Lawyer as a UXR Consultant?

Do you need to hire a lawyer, or can you rely on online templates found for legal documents such as contractual agreements and mutual NDAs?  

While Danielle is not a lawyer nor professes to offer legal advice, she suggests that those online templates are a good place to begin if you’re just getting started. But, she added, be sure and read them with a critical eye. You can also ask other consultants if they would be willing to share the subcontractor agreement they use. If you are subcontracted to a company, they also likely had their legal team look over the agreement they want you to sign.  

At a minimum, any contract should have a timeline, acceptance criteria, and payment terms. But if there’s a question or points of concern, you want to include those in the contract. Don’t leave anything open to interpretation if you can. Be as clear as possible. 

So, the short answer is no, you don’t need a lawyer. Just be aware of the risks you may encounter. (And yes, if you can afford one, you should get one.)

Best Lead Generating Activities

There are myriad activities to generate UX leads. But here are three that have worked well for Danielle and her colleagues:

• LinkedIn: Cory Lebson uses LinkedIn with much success. He will post his availability in the coming months and has had a great response. He enjoys doing in-person testing, so if he has a gap, he might offer to waive his travel expenses, for example. This works very well for him and expands his network.
• SEO: Carol Barnum of UX Firm uses search engine optimization. By working with an SEO specialist, she keeps herself in the rankings where she wants to be. She pays a monthly retainer to do so, but it has paid off for her.
• Word of mouth and past employees: This has been Danielle’s greatest lead-generating activity. The success of this will depend on where your niche and target audience are, and your unique skillset. If you don’t have an expansive network of UX professionals, then word of mouth will not get you as far as if you did have that deeper network. If you are still working toward building out your UX sphere, this can be a good strategy until that network expands.
• Presenting at conferences: If you have chosen a narrow specialty, look into local conferences for that niche and see if you can present or exhibit there. While the ROI on exhibiting isn’t great, presenting might be hugely successful. The folks who write the checks may not be there, but when a need arises for a consultant, those who attended your session will remember you and tell their boss. 

How do you choose the best lead-generating activity for you? Find the tool that suits you the best and be open to other activities. Some may require greater courage, but don’t forget others have been where you are now. You will find success as you continue to persevere.

. . .

Danielle Cooley has been the owner of DGCooley & Co., a bespoke independent UX research and strategy consulting practice since 2009. Her past work has included Hyundai, Graco, Pfizer, Sargento Foods, and Mayo Clinic. She has a BE in Biomedical and Electrical Engineering from Vanderbilt University and an MS in Human Factors in Information Design from Bentley University.

Moderated by Raymond Lee, Founder, UXr Guild
This is an abridgment; view the full video presentation here.

In this final How to Freelance session, Raymond discussed completing a Request for Proposal (RFP) and the characteristics of a good UX project,  then jumped into the business side of freelancing. 

How to Respond to a Request for Proposal (RFP)

Some companies solicit bids for research projects by sending out requests for proposals. When responding to an RFP, it’s important to know your objective and target audience. If they are budget-conscious, you might want to be the low bidder. If they are seeking special expertise, you might demonstrate yours with a case study. Any questions provided by the client can give you clues to what they are looking for. 

While it is ideal to know the client’s budget ahead of time, frequently, you won’t. What should you do then? One strategy is to provide low, medium, and high-cost options and let them choose the one that best fits their budget.  If you want to see a basic template for services, you can view the Guild’s outline for usability testing services here: https://uxrguild.com/services/usability-testing-services/

Typically consultants generate a bid by estimating their hours and multiplying by their desired hourly rate. If you’re anxious for work, you’ll probably want to bid low. If you’ve worked for a client before, have ideal experience, or can handle a tight deadline, you might bid higher and still get the work.  Estimate a few extra hours in case some tasks were not spelled out upfront or work takes longer than expected.

Be prepared for some back-and-forth in reaching a final agreement. If you’re working with someone you don’t know, there may not be much discussion. If you know them, you might have more opportunities to ask additional questions and adjust your proposal accordingly. 

Here are some final tips for RFPs: 

• RFPs take time but can pay off if you’re strategic. 
• Bid if you know the client, have ideal experience, or a special advantage (such as being local and able to work onsite). 
• For a repeat client, you can ask for an exclusive first-bid opportunity. If they like your bid, they may not need to reach out to other researchers.
• Pass on an RFP if you know it has been sent out to dozens of researchers and you have no connection with the company.

What Makes a Good Freelance Project?

Is there a way to identify a good UX project before submitting a response to an RFP? Absolutely! Do some research by asking the following questions:

• Does the company or UX team have a positive reputation in the industry? Are their products highly regarded?
• How long is the project? Most consultants prefer longer engagements, so they are worth more effort to win.
• Does the company have an established process with tools, support, and recruiting techniques?
• Are you interested in the study? 
• Is there sufficient time to complete the project?

Can Freelancers Charge for Travel?

When is it appropriate to charge for airfare, hotel, rental car, per diems, and travel time? All these items are negotiable, and most companies are willing to pay these expenses, but you will more than likely need to ask. Depending on where you live, where the client is, and if they want in-person meetings, clients are usually willing to pay for travel that is not considered commuting.

A per diem is a fixed amount, such as $75 per day for meals and transportation. Travel time is the most challenging expense to get reimbursed. You might ask for 50% of your normal hourly rate when travel is required to conduct interviews, attend meetings, or present findings. 

How to Hire Additional Freelance Help

After examining the scope of the project, you might find that it is bigger than anticipated, requiring more work than you can do in the allotted time. You may need a second moderator for a big study, a participant recruiter, a note taker, or an administrative assistant. It’s best to hire these individuals as independent contractors if possible. If you hire them as employees, you will face the complexities of payroll, tax withholding, government filings, etc., which can take a significant amount of time and expertise. 

What Records Do Freelancers Need to Keep?

Keeping good records is an absolute must for a freelancer. This may not be the most enjoyable part of being a freelancer, but without good records, it will make the business side of your work  much more difficult.

While this list is far from inclusive, at the minimum, you should keep:

• Quotes and proposals
• Contracts
• Copies of your studies and reports
• Client communications
• Time and task tracking
• Invoices and payment receipts
• Expense receipts

Use an organizational system that works for you to have these documents easily accessible as needed.

Bookkeeping Basics for Freelancers

Bookkeeping may feel like a daunting task. If that’s the case, and you’re concerned about doing it right, consider hiring a bookkeeper and a CPA for your taxes. The important thing is to maintain a separation between your personal and business income and expenses. 

Create a separate business bank account (even if you’re a sole proprietor). This will help you track income and expenses for taxes. Having a business credit card for business expenses or using checks, Zelle, or Venmo will also aid in making these payments.

Do Freelancers Need Bookkeeping Software?

If you decide to take on the bookkeeping yourself, do you need software to track hours, issue invoices, and track income and expenses? While not mandatory, options are available, some at low or no cost. The most challenging part of bookkeeping is staying on top of your billable hours, invoices, and expenses. 

If you do decide to invest in software, Quick Books has a self-employed edition, which is a scaled-down version of their full product and an easier option if you’re an independent contractor.  

The best bookkeeping options may be the freelancer specific apps, such as Moxie, and Fiverr Workspace. Wave Apps is free accounting/bookkeeping software that is especially good for invoicing. Getharvest.com, is good for time tracking. Moxie will help you send out proposals, manage a project, and do the bookkeeping at the same time.

Freelance Business Expenses

Shifting now to the expense side of freelancing, one of the advantages of being an independent contractor is that you can write off some expenses that you couldn’t as an employee. What are some of those things? 

• Office supplies, laptop, phone, furniture
• Website development, hosting, internet, email
• Work-related software, programs, apps, fees
• Memberships, training, prof. development
• Design, marketing, accounting, legal, services
• Work-related travel and mileage
• Business insurance, advertising

A couple of other items are more complex, and you will want to check with a CPA to see if you qualify for these.

• Home office rent deduction
• Health insurance (individual premiums)

You will want to take advantage of these deductions if you are freelancing to reduce your overall income thus lowering your tax liability.

Preparing Freelancer Taxes

When preparing your taxes, make sure you have accurate records of your income, whether from freelancing (1099) or an employer (W-2), if you are freelancing on the side.

Acquire a business credit card as mentioned above, and use it for business expenses. This will simplify your record-keeping. If you need to pay via other means, such as cash, a personal card, or check, or when transferring funds from your business to your personal account, make sure you have detailed receipts and records to account for those transactions.  

This is where bookkeeping software, or at the very least, a spreadsheet, will come in handy and make the task of doing your taxes much easier for you and your CPA.

Quarterly Tax Deposits for Freelancers

If you work for an employer, your taxes are withheld from each paycheck and deposited for you. But what do you do as a freelancer when no taxes are automatically withheld? This is where filing and paying taxes quarterly can come into play, particularly if your freelance income is over $20,000 per year.

The IRS requires you to pay annual taxes in four different deposits, called Estimated Taxes, spread throughout the year. Note that the deadlines for these payments differ from regular filing dates. For information on payment schedules and other requirements, go to https://www.irs.gov/businesses/small-businesses-self-employed/estimated-taxes. 

You can also find online calculators where you can input your details to estimate these deposits. One such resource is https://workspace.fiverr.com/quarterly-tax-calculator/.  Again, check with your CPA to find out if you are required to pay these quarterly taxes.

Health Insurance for Freelancers

If you’re leaving your employment and need to get your own insurance, how do you do that? This is less of an issue than it’s been in the past.  Your first option is to keep your current coverage, as long as you can. This might be a viable route to go if you like the coverage and the rate is good. 

If that does not work for your particular situation, the changes that have recently been made to the Affordable Care Act in the United States allow for more options. These plans are now subsidized, making them accessible and affordable in most instances, and are worth looking into.

Here is a breakdown of these available options:

Medical Insurance:

• COBRA-keep your employer plan
• ACA marketplace plans: Healthcare.gov
• Off-marketplace: UnitedHealth, Kaiser, Anthem BC/BS, Centene, Humana, Aetna/CVS, HCSC, Cigna, Molina
• Short-term medical; DPC; Healthshare; MDSave; Cash.

Dental options include Delta Dental, MetLife, Aetna, Cigna, …

In Canada: Provincial Health Care.

Saving for Retirement as a Freelancer

A final consideration that needs to be addressed is retirement. As a freelancer, there are no employer 401k or pension plans. If you intend to freelance long-term, it is a good idea to set aside money for retirement.

There is an instrument called a Solo or One Participant 401(k) which can be either a Roth or a 401k or both, depending on whether you are putting money in pre-tax or post-tax. You do, however, have to use W-2 wages to go into the plan initially. Visit the IRS website https://www.irs.gov/retirement-plans/one-participant-401k-plans to learn more. 

For freelancers making over $50,000 in 1099 income per year, it might make sense to pay yourself 35 to 40% of that total in wages each year. That would give you the W-2 wages to contribute to a plan (and can also be a good tax saving strategy). See Freelancers Can Save Taxes with an LLC, here: https://uxrguild.com/business-entity-freelancers/

The business side of freelancing may feel complex and confusing at times. But with an abundance of online and in-person resources available to help you navigate the process, it should not stop you. If you’re still feeling a bit nervous, take time to explore the Guild’s resources before taking the plunge. You might find that you really can do this after all and if not, there are services to handle it for you.

 

Raymond Lee is the Founder of the UX Researchers Guild. You can find him on LinkedIn. This presentation included guest speaker John Thompson, a New York-based freelance lawyer, who deals with freelance contracts and legal issues on a regular basis. You can view John’s full video appearance here.

Hosted by the UX Researchers’ Guild
Moderated by Danielle Green and Jess Vice
View the full video presentation here.
Session 4 – December 7, 2023

Artificial Intelligence is all the rage, but do generative chatbots belong in UX Research? To what extent should UX Researchers leverage these tools to answer research questions? We hear both praise and protest from UX Researchers across the US and Canada around AI use cases. Research planning, data analysis, and simulated user feedback are just a few ways researchers are attempting to leverage AI tools. With these new processes come various concerns over privacy, ethics, the accuracy of insights, the validity of findings, and more.

AI: Definitions, Goals, and Outcomes for UX Research

Earlier this year, a state of the UX research industry report asked UX researchers about their use of AI tools. Twenty percent of UX researchers reported that they use these tools at work. That number has more than likely increased since then. In preparation for this session, Danielle Green and Jess Vice had a researcher named Riley Mertog do qualitative interviews with seven UX researchers across different domains about their generative chatbot experiences. Riley found that those UX researchers mostly use tools like ChatGPT for research planning, qualitative analysis, writing copy or reports, and brainstorming.

To adequately discuss AI in UX Research, we need to also address its definitions, goals, and outcomes. What exactly do we mean when we say AI and what tools are we referring to? 

AI is an umbrella term referencing the field that encompasses different algorithms, including machine learning and natural language processing. If you have used Qualtrics, SurveyMonkey, Dovetail, Grain, or any transcription tool, or if you’ve categorized qualitative responses by sentiment or summarized qualitative responses by themes, you’ve probably used an AI tool. If you are overwhelmed by all the AI terminology and are trying to figure out what is different among these definitions, you’re not alone. 

So, let’s look at the relationship among these terms. Machine learning is a subset of AI, and natural language processing is also included here. Not all AI is machine learning and not all machine learning leverages natural language processing. Algorithms, a subset of machine learning, leverage something called simulated neural networks. 

However, generative AI uses deep learning to create new content based on its training. So, large language models, or LLMs, are a subset of generative AI. And they are trained on a large amount of language data. In the case of GPTs, that large amount of data is just a chunk of the internet taken from a snapshot in time. So in GPT, as in ChatGPT, the P stands for “pre-trained” on that chunk of the Internet, which is why ChatGPT doesn’t know what happened yesterday because it was pre-trained on a snapshot in time of a chunk of the Internet. ChatGPT is the interface used to interact with the GPT model.

For this Research Rumble, when AI is discussed, this will reference general-purpose chatbots like ChatGPT, with AI, bot, and ChatGPT used interchangeably. 

Pros of AI in UX Research

There are several positive aspects of AI in user research, especially in reference to chatbots or other GPT devices. Here are a few of them.

No more blinking cursor dread

Have you ever opened up an empty document to craft a report without an idea of what to write? AI can alleviate this problem when used as a peer or second brain to consult or even write a whole research report. You will still need to review and finetune the results since no research report should be completely AI-generated. But AI can certainly help you get started. 

Reduce time to task

Most AI chatbots can generate a survey template, interview script, or persona. The survey tool Pollfish is an excellent platform for this. A simple paragraph describing who you’re looking for, why you want to ask them questions, and what answers you’d like to see can result in a decent survey template. As in the report writing example above, you will need to tweak the generated result. But these AI chatbots are a great way to fast forward through some of the more menial tasks. 

Affordable semantic analysis

Once you’ve done your research, what do you do with your results? Consider AI as an alternative to writing down these qualitative, open-ended, feelings-based answers and rearranging them on sticky notes on the wall. Dump all that information into AI and ask questions such as, “Tell me how many times this keyword showed up. How do people feel about this thing? Can you summarize it? What are the positives and the negatives? Can you identify people who are outliers?” While AI is very good with responses such as “I hated this” or “I loved that,” remember that there are still gray areas when clear feelings are not involved. But it’s a faster way to sift through and organize information than using sticky notes.

Web crawler extraordinaire

Hunting for reviews, customer sentiment, esoteric knowledge, or even just combing through forums goes 10 times faster with AI. Using AI to crawl the web for you is an incredible boost. If you’ve ever tried to find an answer that you knew was buried in a thread somewhere, you know the frustration. You could spend time going through post after post or thread after thread in reddit only to end up in a digital wormhole. But why bother if AI can pull up those answers for you? 

One of Jess’ colleagues had two 3D rendering softwares they were trying to get to talk to each other. A couple of developers and a UX designer weren’t having much success until they thought to go to ChatGPT. By asking, “Can you tell me how to have this program talk to the other program and do this outcome?” AI laid out the steps, suggested things to look out for, and how to troubleshoot – all in a matter of minutes. For those more esoteric issues, AI can be a useful web crawler. 

Free video or interview transcription

Studies have compared free AI chatbot transcriptions to services like Otter.ai and found almost no difference in results. As with everything AI-related, the transcription will not be perfect. But if you’ve ever had to transcribe a 20-minute interview from a Zoom video, it’s a giant headache. Even a quick transcription that you edit and compare to the audio file can be a huge time-saving tool with some ChatGPT help. Just be sure you have explicit permission from the participants to upload the recording to an AI tool – once that information is shared, it cannot be unshared.

Cons of AI in UX Research

While there are many positives when it comes to AI, there are also plenty of negatives. With that thought in mind, let’s look at some of the cons.

AI Is Coming for Your Job! (sort of) 

The first thought in most domains is, “Is AI coming for my job?” This is a legitimate concern given that AI-powered tools are convincing to untrained stakeholders, even when the research quality is poor.

ChatGPT and the like will never replace UX researchers; AI is not coming for your job anytime soon. Yet, there is a veil of mystery surrounding UX research that leads people to think they don’t need training to conduct good research to de-risk big ventures. 

Unfortunately, those people will only be encouraged by ChatGPT’s confidence and the level of detail it provides. We will likely see many leaders and business folks who already thought that research specialists weren’t necessary. And these tools will help them confirm this view they already have. These people are going to try their hand at research and these tools will actually elevate some of the work they’re doing and get them to a place where they can do better and better quality research. So, the new issue with these tools is that bad research is harder to detect by people who are not trained researchers. And that’s the biggest hurdle that you have. It’ll be more difficult for untrained people to detect when research quality is poor.

Privacy and Ethics

There are ethical concerns about how bots are trained and retrained, and how data is used. A major issue with AI chatbots is ethics/privacy. Some of these models were trained on copyrighted information without permission. Therefore, it’s unclear if and how some companies are storing the data provided and whether they’re using that data to finesse the training of their models. If you don’t have permission to share information, or if you’re sharing personal information in these interfaces like ChatGPT’s interface, you’re taking a legal, and potentially, ethical risk. If you copy and paste responses from a qualitative survey, for example, you may be violating the rights of your participants if you haven’t asked for their permission to use their data that way. The same goes for anything that includes identifying information of participants such as names, where they work, email addresses, phone numbers, etc. Some of these privacy and ethical issues are why many countries have banned certain generative chatbots until the privacy laws and regulations catch up.

Proprietary Information and Security

Supplying sensitive information to a tool that is not controlled by your organization could have unwanted consequences. In fact, there have been stories in the news recently concerning security and proprietary information, where workers entered proprietary information into tools like ChatGPT and were later fired for doing so. How would this play out? It could begin with pasting an R script from your organization’s repository into ChatGPT and requesting changes to be made. It may seem innocent, but doing so may violate your company’s NDA or security policies. A good rule of thumb is if you wouldn’t do it on a public computer at your local library, only do it on ChatGPT if approved by your organization. So think about security.

Validity of Output

Next is the issue of the validity of the output. ChatGPT’s training comes from a chunk of the internet. In theory, all the information it has been trained on is publicly available. So if we could effectively do our jobs based entirely on public information, we would only do desk research. It also means that if people wouldn’t publicly share that data, like portfolios or work examples from senior-level researchers, ChatGPT would not have access to that information: it’s not in ChatGPT’s worldview because it wasn’t trained on that information. Bots will not pull from cutting-edge, mind-blowing expert case studies because that information isn’t available for it to train on.

An example from Danielle’s experience is evidence of this. The internet loves wordy, double-barreled survey questions. Therefore, ChatGPT loves writing wordy, double-barreled survey questions. But no matter how much Danielle coaches it, she cannot get it to stop writing survey questions in this way. ChatGPT can’t unlearn its fundamental training. And then there was the time when ChatGPT told one of her graduate students to assemble a board of directors for their four-week-long homework assignment. Understandably, ChatGPT can make logical errors. But this also places the validity of the output in question.

Big bias 

Another thing to address is bias, whether that of the internet or ChatGPT. You can’t test accessibility. You can’t with confidence take into account certain cultural contexts with ChatGPT or other bots. That might sound like something that doesn’t apply to you depending on your line of work. Going back to ChatGPT’s insistence to write surveys in a certain way, a bot cannot write a survey that makes sense to 15-year-olds. It can’t write simple survey items to use with that population. Think about your user population. What language do they use and comprehend? What kind of cognitive load can they endure? ChatGPT cannot take those things into account even if you coach it. Bots cannot create, for example, good protocols or research materials for neurodivergent populations. Simply put, bots are a product of their training, and the inputs are not representative of many diverse groups. 

As with any topic, there are pros and cons of the role and future of AI in UX Research. Do your research and constantly ask yourself, “Is the time I’m saving worth the risk I’m taking?” Only then can you truly understand how to safely and ethically use AI in your role as a UX researcher.

For more information on how to use AI in UX Research, check out this quick guide.

. . .

About Danielle Green (she/her): Danielle is a product and UX professional specializing in research and strategy (high growth, product-market fit). She is an Instructor and mentor with eight years in product, and five years leading teams. As a professor of practices at Claremont Graduate University, Danielle teaches the core courses for the User Experience MA in Applied Cognitive Psychology. She is also the founder and director of the Claremont UXR Laboratory (claremontuxrlab.com), a graduate student lab for UX Research, and has extensive experience in many domains, such as: e-commerce, SaaS, Edtech, Virtual Reality, and hardware.

About Jess Vice (they/them): Jess loves working with people and is curious and excited to understand what drives them to make decisions. Jess offers a deep background in qualitative research, user experience best practices, and high-level strategic planning and is particularly good at making meaning from research and using it to create data-informed strategies for creative and development teams. Jess is also intensely aware that the first point of contact is always the internal teams they work with — if a relationship is not built on trust, they know they won’t be a successful researcher and strategist. Jess has been working in marketing and advertising, CRO, SaaS, and product for over 14 years, and is consistently thrilled with how much more there is to learn.

Hosted by the UX Researchers’ Guild
Moderated by Danielle Green and Jess Vice
This abridged version was written by Booker Harrap. View the full video presentation here.
Session 3 – September 21, 2023

It is clear that the definition of “quantitative skills” varies among researchers (and hiring managers). Even when definitions are aligned, folks have different views on the level of quant required to be a successful UX researcher. 

Danielle Green interviewed dozens of experienced UX researchers about their views on quantitative skills. The results fell into two common themes: 

1) Many researchers lack confidence in their quantitative capabilities. Sometimes these researchers are ashamed because they have no training in statistics, while some are ashamed because they “only know basic things like multiple regression and factor analysis.” Some researchers actively avoid any job descriptions that contain the word “quantitative.” They identify as “qualitative researchers” but are often happy to design, analyze, and report on surveys containing only Likert scale items. 

2) Some researchers self-report “mastery” of quantitative research practices but cannot define terms like “standard deviation” or “confidence interval.” This then naturally leads to the question: What should mastery consist of?

So, are quant skills required or just a bonus for UX researchers? Let’s dive into the discussion of the pros and cons of quantitative skills in UX research.

Pros of Requiring Quant Skills

Minimum Quant Skills Needed in UX Research

• UXRs should be able to calculate estimates of true values given a sample and statistically compare two or more groups. 

The minimum level of quantitative skills would include being able to compute estimates of the true value based on a sample of data, as in reporting confidence intervals, and having the ability to statistically compare two or more groups. This points to basic inferential statistics knowledge, such as familiarity with Anova, Chi-square, and T-tests. Comparing groups usually means proportions and reporting confidence intervals around proportions. 

Quant Skills Contribute to UX Research Success

• The two-part definition of success is 1) stakeholder buy-in and 2) making decisions based on research that generate value for the organization.

Success in this arena boils down to two main ideas. The first is stakeholder buy-in. When you report findings and present them to stakeholders, you want them to take these findings seriously. And second to this is research data-backed decision-making. When the organization takes an action based on your research findings, that action produces value, whether revenue, clicks, signups, etc. Your research insights will turn out to be right most of the time. 

Quant Skills Increase Chances to Be Hired and Retained

• When two resumes are roughly equal, the one with quant will likely win. Business leaders struggle to value qual-only research.

So, why is it good to be familiar with quant skills? The simple answer is getting hired, especially in this job market. All things being equal, the candidate with quant skills will probably get the job over the candidate who lacks those skills. Getting the job is just half the battle; you also want to keep that job. You do this by being seen as a contributor with unique value among your team members. Business leaders can struggle to note the difference between your qualitative insights and the junior PM who talked to five customers this morning and now has a two-year roadmap. Even if you have a Ph.D. in anthropology, they may not recognize the difference between qualitative research and the one who talked to a handful of customers this morning. Quant can help you retain that unique value that you are contributing to the team.

Qual Combined with Quant Skills Improves Operational Efficiency and Effectiveness

• When quant and qual exist together in the same person, the two types of data can be combined more effectively.

When researchers have both quant and qual skillsets, they combine both types of data in a single person’s mind and then deliver more powerful and effective insights. They can do this because they’re able to process both types of data together. If you have both these skills, you don’t have to involve other team members when synthesizing insights. 

Quant Skills Contribute to Accuracy

• If you make decisions based on counts and averages alone, research will be biased to “build it,” which is not advantageous for most orgs.

A final thought centers around accuracy and making the right calls based on research. Whereas those with quant skills may be biased to say that two things, groups, or designs are not different, those with less quantitative skills may report differences that may not be true in the user population.

Philosophically speaking, quantitative researchers tend to be more conservative and less likely to say yes to building an app, making changes, or adopting a more costly option. Less quantitative folks will be more prone to want to try new things. While there is a time and place for both, organizations generally hire researchers to de-risk things and will look for conservative and quantitatively inclined candidates with a sensitivity to those risks.

Cons of Requiring Quant Skills

Accurate (Qualitative Researchers) vs Precise (Quantitative Researchers)

• Because user experience centers around humans, it is impossible to be precise in most scenarios.

Jared Spool points to the difference between accurate and precise; qualitative researchers tend to be accurate, while quantitative researchers tend to be precise. In real-life scenarios, the only time we actually talk about precision is when the client asks about running a survey to statistical significance to which the response is to inform the client of a lack of funds and time. 

But generally speaking, most of the work done as UX researchers deals with humans who are unpredictable by nature. Because of this, it’s usually good, or at least good enough, to be accurate, to represent the trends rather than to be precise. The focus that quant brings a more conservative, granular, and specific approach sometimes causes us to miss the bigger picture. 

Qualitative Researchers Build Collaboration with Data and Teams

• Rather than stepping on toes by taking on someone else’s role, build collaboration with data, analytics, and business analyst teams.

There’s a symbiosis between purely qualitative researchers and purely quantitative data teams. Rather than trying to acquire the skills the quant team already has, can you show up instead as someone who complements them and brings a different perspective? In this way, you can challenge how they look at their jobs and build more collaboration internally, something every company wants.

Since a qual skillset is looking at people patterns, perhaps your time and effort would be better spent focusing on partnering with the business analyst teams already in place rather than trying to be a business analyst. Examining roles and responsibilities within any organization is something all researchers need to better understand. 

Cost-Benefit Analysis of Acquiring Quantitative Skills – Is it Worth It?

• Are quant skills going to be additive? How long will they take to acquire and how much will it cost? What will the real outcome be? Will this effort help me be a better researcher or get that dream job or promotion? 

These are legitimate questions that need to be addressed, especially when quant, or advanced math skills, are not your forte. If these questions make you feel uneasy or uncomfortable, then maybe the effort to acquire such skills is not worth your time and energy. In this case, perhaps it does make more sense to partner a quant person with a qual person, thus allowing each to be the expert in their field.

Data Misrepresentation Without Adequate Quant Training

• Without enough training and accountability, quantitative data can be skewed to tell the story we want. 

Another thing to consider is that without adequate training in quant skills or internal accountability on your part, can you guarantee that the quant numbers you’re sharing are accurate? The numbers might tell the story you want to tell, the one that makes your boss happy, or even the one that guarantees that raise you’ve been aiming for. But is it the correct story? There’s always the chance that you’ve skewed the data without realizing it, especially if you’ve recently been hired or are trying on a new set of skills you haven’t mastered yet. These conclusions need to be left to the experts. Companies would be wise to hire a business analyst or someone with a Ph.D. in numbers who has worked in CPA environments: an expert who can see the gaps and knows what to do about them. Without those quant skills, you, as a newbie, might not even know what could go wrong and how to fix those errors. Do you want that responsibility? Being familiar and careful with the data is crucial.

Quantitative Research and Masked Biases

• The diversity, representation, and accuracy of sources is easy to hide or ignore in quantitative data.

As good as quant tools are, they contain inherent biases that can’t be overlooked. Pollfish is one of those tools. It can be a UX researcher’s best friend, but keep in mind that you don’t know where the user panels are coming from. And with tools such as this, there’s no guarantee you will get the diversity or accurate representation of the groups you are surveying. 

There are unknowns behind the scenes that we don’t understand unless it’s made transparent. In quant specifically, people are more inclined to keep those details in the background because they don’t represent an interesting story. So, instead of getting the full picture, we end up with masked biases that, from the beginning, skew the data. This, of course, can be disconcerting when we’re talking about human data, about real people.

As a UX researcher, you will need to determine whether having quant skills in your toolkit is right for you. These pros and cons can serve as guidelines to make that all-important decision.

. . .

About Danielle Green (she/her): Danielle is a product and UX professional specializing in research and strategy (high growth, product-market fit). She is an Instructor and mentor with eight years in product, and five years leading teams. As a professor of practices at Claremont Graduate University, Danielle teaches the core courses for the User Experience MA in Applied Cognitive Psychology. She is also the founder and director of the Claremont UXR Laboratory (claremontuxrlab.com), a graduate student lab for UX Research, and has extensive experience in many domains, such as: e-commerce, SaaS, Edtech, Virtual Reality, and hardware.

About Jess Vice (they/them): Jess loves working with people and is curious and excited to understand what drives them to make decisions. Jess offers a deep background in qualitative research, user experience best practices, and high-level strategic planning and is particularly good at making meaning from research and using it to create data-informed strategies for creative and development teams. Jess is also intensely aware that the first point of contact is always the internal teams they work with — if a relationship is not built on trust, they know they won’t be a successful researcher and strategist. Jess has been working in marketing and advertising, CRO, SaaS, and product for over 14 years, and is consistently thrilled with how much more there is to learn.

Moderated by Kuldeep Kelkar and Rima Campbell
View the full presentation here.
December 1, 2023

If you’re a UX leader who wants to have more impact, the following strategies will help affect change, increase your influence in your organization, and help get a seat at the table with decision-makers to determine product strategy, corporate strategy, and budgets.

Strategic Influence to Increase Executive Buy-in with Kuldeep Kelkar

Approaching this topic can be overwhelming. But let’s look at two keywords that will help break it down a bit: Executive and Influence. 

What Executives Need

To create strategic influence, first identify key executives within the organization, then contemplate their needs, and lastly, consider their concerns. Following these steps will help you focus your efforts where they will have the greatest impact.

Step 1. Identify Key Executives:

Reflect on two to three key executives at your company whom you want to influence. Consider their roles, responsibilities, and the scope of their influence within the organization. These could be anyone within your organization: a product leader, someone in marketing, your reporting line VP, and so on. 

Step 2. Contemplate Executive Needs:

Beyond the realm of UX, contemplate what these executives might need. What are their broader goals, challenges, and priorities? Think about how your work intersects with their objectives. 

Step 3. Consider Their Concerns: 

What potential issues or challenges keep these executives up at night?  Try to empathize with their position and the pressures they face. 

Ways to Grow Your Influence as a UX Leader

The second keyword in today’s topic is Influence. How can research leaders influence stakeholders? The following suggestions are by no means the only tools you can use to grow your influence. But these are the ones that I have found valuable throughout my career.

Build Credibility 

Building credibility is both an art and a science. Since you are already working with an organization, you can demonstrate tremendous value and build credibility as you execute research; whether in a large study, design validation, or part of a research roadmap.

Forge Strong Relationships 

Forging strong relationships across the organization cannot be overlooked. While this applies to designers or product managers, look for allies in marketing, engineering, HR, or finance. Eventually, an opportunity may come to have a wider influence, but it starts with building those strong relationships. 

Become a Master Storyteller and Share Success Stories

None of us can be influential unless we are master storytellers. I have never interacted with a successful executive who was not also a good storyteller. Learning to share those sound bites and narrate success stories is key to influence. I work with many executives, and those I have tried to emulate are all good storytellers. 

Please note that this goes beyond a PowerPoint deck or research report. Consider adding sound bites and enticing stories into general conversations. Success stories of what you are doing will increase your influence over time. 

Facilitate Workshops 

The next one is near and dear to my heart. As researchers, AI will make us more productive and efficient in defining screeners, analysis, and reporting. But I don’t know that AI can unite us as humans. Most UX researchers and designers are proficient at reading people. We empathize with end users, product managers, and people we work with. As researchers, we are best suited to bring people together. Anytime you have an opportunity, jump in and facilitate a workshop, whether at the divergent or convergent phase of an engagement. Doing so will bring you into strategic conversations you were not previously aware of.

Domain Expertise and Industry Knowledge

The next suggestion centers around domain expertise that is more than the user experience or user research domain. By this, I mean industry knowledge. In the last year, I have worked with product leaders and investors and have worked with several startups. I also speak with the investment community. I have learned that successful leaders within any organization are generally those who have been in that industry and are familiar with competitor pricing, personas, or the ICP, the ideal customer profile. They know what the competitors are doing. 

The more you know about your industry, including pricing trends and growth, the more you know about your competitors. Information for public companies is likely found online. Keep up with the knowledge of whichever industry you work within. Try to speak those sound bites so that people understand that you know how things work within your industry. And again, this is not just about UX but the implications of UX within your industry.

Hire Consultants and Invite Guest Speakers

The next suggestion is to hire consultants and invite wise speakers. You might wonder why this is important when you are a consultant yourself. When I worked at PayPal, before my consulting days, I brought in guest speakers within my organization. Ironically, this made me more influential because I could introduce industry thought leaders to my company. Even if these guest speakers said the same thing I had shared, they offered additional value and validated what I was saying. Doing this will increase your influence.

Mentoring and Being a Continuous Learner

The following two suggestions focus on mentorship and seeking mentors within and across the organization. You don’t have to have a particular agenda in doing this. Being a continuous learner is an important attribute that ties into the need to forge strong relationships. 

Learning from others is crucial, but it is also vital to mentor others. You have experience the larger community can benefit from, even within your company. Every day, product managers, designers, and researchers are being hired. Network with them and be available to educate others to the degree you can.  

Measuring Your Impact and Influence

It is sometimes difficult to measure impact and influence. But you cannot manage something that you cannot measure. I encourage you to figure out ways to measure, improve, and re-measure your current influence. It’s all about understanding where your influence exists. 

Perfect Your Elevator Pitch

The theory of an elevator pitch is envisioning yourself on an elevator with an executive. As you enter the ground floor together, you will have five to 10 floors before exiting. That’s 20 to 30 seconds. That may not seem like a lot of time, but it is enough to do a few simple things.

The elevator pitch typically has three components: sharing who you are, what you do at the company, and why what you do is important. Take advantage of every opportunity to connect with executives, even if for this short period of time.

Think about sound bites to have at the ready. These can be cliffhangers or dramatic headlines, or anything that will cause the other person to pause and want to know more. These hooks are conversation starters that encourage a response and leave an impression.

Speak the Business Language

To connect with executives, you will need to understand the needs of the company and the business metrics that drive performance. On the simplest level, if the person you are speaking to within your industry refers to the end user as a customer, patient, or guest, use those key terms and understand why they are used. 

On a more complex level, take time to research basic concepts of business and finance. You don’t need an MBA or PhD to do this. Someone somewhere in your organization is working on how to beat the bottom line or the beta or gross margin goals that determine how much money the company has to spend on different functions. Every company that I’ve worked for and known has someone who is looking at what the top line number should be for the next four quarters. This forecasting determines how much money there is to spend. The more you understand and speak the business language and understand the company, the more influential you will be simply because you will participate in conversations that you were not part of before. 

Getting a Seat at the Table with Rima Campbell

UX Challenges in Business

A major challenge UX researchers face is getting executive buy-in for UX research. One reason this may not happen as often as we would like concerns the relationship between user experience and key performance indicators. Executives in the C-suite are looking for KPIs and the bottom line and how that impacts their business. The UX professionals and leaders are looking for insights. So, how do they bridge the gap between the C-suite and the user experience? 

McKinsey & Company ran a study about the potential for design-driven growth.¹ They analyzed the impact of the design effort of 300 publicly listed companies over five years across a variety of industries and countries. The report found that design-forward organizations performed at twice the rate of their industry competitors when it came to generating revenue.

How were they able to do this? As explained in the study, “Setting a North Star and combining qualitative data and quantitative measures can demonstrate the value and effectiveness of design.” They started with continuous iteration of their design validation, then consistently embedded research throughout the product development lifecycle, which led to making data-driven decisions by using a higher sample size. They also measured the performance of their design and its impact on the financial success of their business. 
_______________________________

¹ https://www.mckinsey.com/capabilities/mckinsey-design/our-insights/the-business-value-of-design

Competition in the World of UX Research

We live in a world of constant competition in which every executive wants to know their company’s rank among competitors because seeing their ranking can be a motivational booster.

I experienced this firsthand in a study while at Citigroup. The business wanted to increase online credit card applications submitted by 22%. As a researcher, I wanted to make an impact and get executive buy-in for more UX research to grow the budget and the team and be able to embed research across the product development lifecycle. To do this, I had to show metrics and the impact UX research and design make on the business. We looked at the analytics, trying to figure out why people did not submit their credit card applications online, as well as why there was a large call volume related to that issue. Looking at just the analytical data wasn’t giving us the answers. 

So, we looked at the survey where we had interviewed 15 individuals. We chose to simplify our approach by asking participants to do one task: choose a credit card that matched their needs without submitting their application. We also determined that we needed a significant sample size to prove the accuracy of our results.

We ran a baseline benchmarking study where we examined behavioral data and also ran a “talk out loud” study to hear the participant’s frustration along the way. We discovered that the success rate for completing that journey and selecting the current card was 63%; satisfaction was below 60, and the ease of use was equally low. We now understood why participants did not complete that task in the journey. Our comparison experience was unsatisfactory and left participants with a sense of mistrust. When we shared this information, the executive wanted to know what the success rate norm was. We knew why and had statistical data to prove our findings. But how did we know that a 63% success rate was not good?

To verify this, we ran the same study on American Express, Discover, and Wells Fargo and found out that we were, in fact, not doing a good job. To improve the success rate, we needed to improve the customer satisfaction.

As we improved the comparison experience, success, satisfaction, and ease of use increased. We then ran more studies on the prototype and made improvements, iterated design, and tested it to ensure the success rate and satisfaction were on par with our competitors. Within 12 months, we realized that enhancing the experience improved the online application by 22% and reduced the call volume from 63% to 30%. Imagine the impact we created in that particular case study. We got the executive buy-in to do research across the product development lifecycle by starting with this case in that project. 

How to Measure UX Influence

We all agree that measuring influence can be tricky. Consider sharing a survey or questionnaire with stakeholders to collect structured feedback every six months. 

The User Research Team …

• … is regarded as a credible and reliable source within the organization.
• … demonstrates exceptional expertise and knowledge in their field.
• … consistently provides value.
• … regularly introduces innovative ideas that significantly benefit our projects.
• … effectively engages with stakeholders, fostering positive collaborations and outcomes.
• … is frequently recognized within the organization for its contributions to success.
• … plays a crucial role in positively influencing and shaping our organizational culture.
• … exerts influence beyond its immediate domain to other departments and areas.

Being open to their feedback will go a long way in measuring the influence you have on executives and the company as a whole.

Additional Thoughts on the Elevator Pitch

Adding to what Kuldeep shared on this topic, being on an elevator with an executive is a perfect opportunity to talk about UX research and what we’re doing in that world. But if you launch into a long story, the executive will lose interest. 

How do you do an elevator pitch that captures their interest and encourages them to want to know more? Look at sharing specifics, such as, “We ran a benchmark study and Blue Cross Blue Shield won the competition. They scored 77 out of 100, whereas Humana scored the lowest. Their score was 57 (this example is fictitious).”

Sharing this story of scoring and comparison will spark the executive’s curiosity. I could then share that Humana paid a significant amount of money to redesign its website. So why did they score the lowest? Now, with their attention, I am invited to the table. I can go into greater detail to show how this impacts business metrics and KPIs.This will lead to even more discussions because every executive cares about their KPIs and is accountable for business metrics. 

As a UX research leader, you need to know your business metrics and how to tie them to what matters most to your organization. The QXscore will help you do that. It also includes the loyalty or NPS score which most executives resonate with. 

Connecting UX Metrics to Revenue

The bottom line is you don’t have to remember a million scores; you only have to remember one score that is out of a hundred. Having information like this at your fingertips will allow you to capture an executive’s attention. This then trickles up into what we call the 21st-century metrics framework from level four to level one as an impact on the financial goal. Having a deep understanding of a key executive’s ambition and desire, and tying those UX metrics to the business metrics, will make a huge impact and increase your influence.

You Got a Seat at the Table: Now What?

Several organizations ran the measurement program and made a huge impact. One of them is Kimberly Clark. Because they were able to improve before and after from a task success perspective and time spent, they were able to improve the business metrics and show how the UX metrics improve their business metrics and what they end up with. One big success for them is driving prioritization and demonstrating progress as UX leaders and practitioners. In this way, you can impact the prioritization of your project from a product and IT organizational level.  

Scalable Solutions with Customer Value

Another study was with Kroger, where they incorporated scalable solutions with customer value as they ran these measurement programs. Doing so allowed them to get the business to invest more at the beginning of the product development lifecycle and do more discovery, ensuring they were creating the right solution for the customer problem. This occurred at the beginning, not just validating the design in the middle of the product development lifecycle. 

Tying OKRs to QXscore Results

Adobe tied the OKRs to QXscore results and embedded the UX metrics in the design and performance reviews. This desire came from the UX design team to improve their score and the impact they’re having on their organization. So, it was something they included in their performance reviews. 

UX Measurement Program Framework: Activate, Ignite, and Drive

A UX measurement program framework includes three categories: Activate, Ignite, and Drive. 

In the Activate category, there are three components: 

• Identify what matters most to executives;
• Identify a pilot project that focuses on a user journey; and
• Connect UX metrics to the business metrics. 

The Ignite category includes the following steps:

• Know the QXscore and the competitor’s QXscore;
• Educate and build trust with the QXscore; and 
• Prioritize improvements with stakeholders, test and iterate with benchmark.

The last category, Design, incorporates these three elements:

• Visualize, track, and share progress regularly;
• Embed QXscore into team performance review; and 
• Tell inspiring stories regularly.

UX researchers face challenges in today’s enterprise, and focusing on them may seem overwhelming. But don’t give up. Your dedicated efforts will eventually bear fruit. Remember that you can demonstrate value, get executive buy-in, and gain a seat at the table.

About Rima Campbell and Kuldeep Kelkar:

Presenters Rima Campbell and Kuldeep Kelkar worked together as research leaders and consultants at UserZoom (now UserTesting) for the last seven years. They consulted with leaders at Global 500 companies, helping them to conduct effective research, refine products, and increase their influence. Previously, they had long stints leading research in large enterprises; Rima as SVP and Global Research Manager at Citi for 16 years, and Kuldeep as Head of Global UX Research at PayPal for 10 years.

Kuldeep Kelkar
kuldeep@uxreactor.com
https://www.linkedin.com/in/kuldeepkelkar/
https://uxreactor.com/blog/

Rima Campbell
https://www.linkedin.com/in/rima-campbell/
https://www.usertesting.com/resource-library

A UX Researcher’s Guild Online Event
Moderated by Danielle Green and Jess Vice
This abridged version was written by Booker Harrap. View the full video presentation here.
Session 2 – July 14, 2023

As a UX researcher, it’s essential to stay up-to-date with the ever-evolving perspectives on research best practices. Scrutinizing our standard tools and methodologies is important to catch any gaps in our ability to generate sound, actionable insights. In a recent Research Rumble, we took a deep dive into one such tool: personas.

Are Personas Good?

(By Jess Vice)

Distilling Research for Easy Consumption
For design and development teams, who tend to be very visual and think big picture, personas provide a concise summary. Instead of being overwhelmed with data, they get a snapshot, making it easier to assimilate and act upon.

Preventing Self-referential Design
Alan Cooper’s introduction of personas in 1998 aimed to curb the “elastic user” problem, where we imagine what the user might want or who the user might be. Designing based on assumptions is problematic; personas ground these decisions in research.

Maintaining a Research-Backed Focus
Personas help in combating scope creep. They serve as a constant reminder of the real user, discouraging the creation of imaginary users based on our biases or assumptions.

Building Team Alignment
When multiple stakeholders are involved, differing opinions can delay projects. Personas, being research-backed, offer a common ground and guide aligned decision-making.

Cultivating Empathy
With diverse teams contributing to product development, it’s easy to lose sight of the user. Personas serve as a tool to humanize users, fostering empathy, and ensuring that the end product caters to genuine needs.

What’s Wrong with Personas?

(By Danielle Green)

Inclusion of Irrelevant Data
When personas become more of a formality than a functional tool, they risk containing unnecessary information. This dilutes their effectiveness and confuses more than clarifies. 

Too Many Personas
Creating a multitude of personas can paradoxically make them less useful. Instead of clarity, teams face choice paralysis, questioning which persona to prioritize.

Unsubstantiated Foundations
A persona built on minimal or irrelevant data risks being a fictional character rather than a representative of real users.

Risk of Extrapolation
Ironically, while personas aim to curb assumptions, they can sometimes encourage them. Stereotyping of characteristics present in the persona can lead to misplaced assumptions. “Did we just assume that the mother of five who is preparing a family’s meal is a homemaker? Or did we actually ask that question?”

Mental Shortcuts to Different Internal Definitions
Similar to extrapolating beyond the data, the mental shortcuts that personas provide can turn sour when these shortcuts lead to different internal definitions. Maybe memories of the data have warped over time, leaving stakeholders overconfident in what are essentially assumptions.

Designing for the Average, Not the Real
Assuming these challenges of execution and socialization have been overcome – there is a decent amount of data from an appropriately representative sample – what could go wrong? Often, this data is represented by the averages across the various characteristics, creating a persona of a user that doesn’t exist. This can lead to a persona with conflicting items that don’t align.

Creating for the Current Market, Not the Target Market
Maybe the problem of the non-existent average user has been accounted for with a cluster analysis to see which factors hang together, leaving the team with an accurate snapshot of the current user. Perfect! Or is it? There is a difference between the target market and the current market which is not being accounted for by this approach, and this can be a strategic pitfall. Company mission, company values, and a changing marketplace mean it may be very important to design for the ideal user, but this excellent persona of the current user is blinding, and the new market out there that could be served is being ignored. 

An interesting data point here: Despite these concerns, a recent UXPA salary survey revealed that 68% of UX researchers still employ personas.

Why Are Personas Good?

Group Discussion Insights In Favor of Personas

Echoing the Opening Arguments

Cultivating Empathy
It comes as no surprise that this point was echoed by other groups of Rumble attendees.

Maintaining a Research-Backed Focus, and Building Team Alignment
This quote reflects these points well. It also makes a fresh point that personas create company-wide prioritization:

“We need to have consumer surveys with significant data that show that these characteristics, these priorities, these pain points match up with this group that is statistically significant [and] fits within a mainstream user base, right? And we have a general sense of the jobs that they want to get done with our product, right? And so my approach with personas is I will incorporate anything into them that works, that will make it more credible… make it easier for designers to act on it and to keep the company, the product designers, the product managers, the executives–strategically keep everyone aligned on [the] top priorities and the biggest pain-points or use cases that we’re solving for that appeal to a large mainstream user base.”

Fresh Insights

High Utility for Those Involved in the Design Process
This was brought up by two different groups, who called out that personas are particularly useful for stakeholders involved in the design process. Here is a quote:

“[Personas are] very valuable to creatives, designers, and people making decisions on behalf of users and a product and ensuring that they’re building or designing the right thing.”

Replacing Personas… With What?
Two groups independently brought up the following question: “If we throw out personas, what would we replace them with?” It’s a good question that no one had an answer to.

Differentiating Target Markets
A more specific use case for personas mentioned is their utility in differentiating target markets.

“The buyer of a [truck] is very different than the buyer of a [sports car]. [Personas] help us. But then … at a second level, we try to expand our personas because it is possible that that single mom not living in Texas does want to drive a truck. So, we might have who we think is our target, but we try to expand them beyond that.”

Aspirational, Building Towards the Ideal Customer
This callout echoed the opening remarks about considering the ideal user:

“We see a lot of benefit for them being useful for long-term or aspirational use. Who is it that we want to be targeting and can we build towards that sort of ideal customer?

Why Are Personas Bad?

Group Discussion Insights Against Personas

Echoing the Opening Arguments

Risk of Extrapolation
One group specifically referenced the opening remarks:

“As we’ve already discussed, it’s also very easy to get a big idea about something with very little data, which can lead you down a false path.”

Fresh Insights

This is where things got interesting. Although most of the favorable aspects of personas had already been mentioned in the opening arguments, our Rumble attendees had a lot of new insights when speaking about the pitfalls of personas.

Relying on Old Data / Not Updating Persona
One group of Rumble attendees brought up the need to constantly check where the data is sourced from and how recently it was updated. This is a good point; personas should not become stagnant artifacts that provide a false sense of diligence in research-backed decision-making. They must reflect the user we are currently trying to target.

Risk of Not Designing for Limited Abilities
This was a great point that can so easily be overlooked when creating a persona, and focusing on your largest addressable market. All products should keep in mind the need to design for people with ability limitations: not just because it is the ethical (which it very much is!) and the effective thing to do, but companies who fail to do this can find themselves in very hot water, legally or as a matter of public image.

Less Useful for Stakeholders Not Involved in Design
On the flip side of the benefit that personas provide to people involved in designing and building products, personas can have a limited utility for stakeholders not involved in the design lifecycle. Like all presentation of research, it is important to remain intentional in the building of an artifact and who it is designed to help.

Quickly Forgotten After Lots of Work
This can easily happen since personas are such a standard practice. If they are being built as a matter of checking a box in the product development process, and a low effort is made to integrate their use during design and decision-making processes, that can make them effectively a waste of time and resources. Have a plan to revisit research artifacts throughout design!

Limited Resolution of Target Market
Looking back to the benefit personas provide in differentiating target markets, this only happens if the opportunity to do so is noticed and acted on. If you rely on only one persona, there’s a risk of excluding a segment of the market that represents a substantial opportunity. Don’t fall into this trap! As one of our Rumble attendees noted: “It is possible that that single mom living in Texas does want to drive a truck.”

Sometimes They Don’t Account for the “Why”
While personas provide a snapshot of the user’s characteristics and needs, they often lack deeper insights into the underlying reasons or motivations behind certain behaviors or preferences. This “why” aspect is critical for truly understanding the user’s pain-points, desires, and context:–”They may watch TV, but why?”

Potential solution offered by rumble attendee:

Integrate the user journey and product mapping into the persona and vice versa. This underscores the importance of combining personas with other research methods and artifacts to gain a more holistic view of the user.

Conclusion

Personas and UX research are seemingly inseparable; when you think of UX research methods, personas are likely top of mind.  Personas have become a UX tradition, and like all traditions, it’s worth taking a hard look at them and seeing if they still serve us. In this post, we’ve taken a look at the highlights from our recent Research Rumble where we discussed how well, or poorly, they fit into the modern UXer’s toolkit.

. . .

About Danielle Green (she/her): Danielle is a product and UX professional specializing in research and strategy (high growth, product-market fit). She is an Instructor and mentor with eight years in product, and five years leading teams. As a professor of practices at Claremont Graduate University, Danielle teaches the core courses for the User Experience MA in Applied Cognitive Psychology. She is also the founder and director of the Claremont UXR Laboratory (claremontuxrlab.com), a graduate student lab for UX Research, and has extensive experience in many domains, such as: e-commerce, SaaS, Edtech, Virtual Reality, and hardware.

About Jess Vice (they/them): Jess loves working with people and is curious and excited to understand what drives them to make decisions. Jess offers a deep background in qualitative and quantitative research, user experience best practices, and high-level strategic planning and is particularly good at making meaning from research and using it to create data-informed strategies for creative and development teams. Jess is also intensely aware that the first point of contact is always the internal teams they work with — if a relationship is not built on trust, they know they won’t be a successful researcher and strategist. Jess has been working in marketing and advertising, CRO, SaaS, and product for over 14 years, and is consistently thrilled with how much more there is to learn.

About Booker Harrap (he/him): Booker recently graduated with a master’s degree in UX Research from Claremont Graduate University. He is currently a UX Researcher with Atticus Capital and is working with the Games & Interactive Technology Lab on an emotional wellness program for middle school students using gamified mobile applications. Booker is actively seeking employment opportunities in UX research. For more information,  please send an email to hire@uxrguild.com.

Presented by Jeanette Fucella
Director, User Research & Insights, Pendo
This is an abridgment; view the full video presentation here.
February 2, 2023

Modern product teams are simultaneously balancing speed to delivery of new features and the need to test hypotheses, uncover unmet needs, and gain new insights about their users. So many user research questions; so little time! ​Given these competing pressures, how do you determine which studies to prioritize? How should a product team determine how to invest their limited time as well as their customers’ time? When is it worth investing in time-consuming primary research, and when is it ok to “ship it and measure it”?

Jeanette Fucella, Director, User Research and Insights at Pendo, answers these questions and many others in her recent session with the UX Researchers’ Guild.

Scaling the Impact of a Research Team with a Prioritization Framework

When Jeanette first put together this prioritization framework, she thought it might generate a few likes when she posted it online. But then, as she describes it,  “the thing went berserk” and has since been translated into multiple languages. This simple 2×2 matrix has helped Jeanette keep her sanity, and she encourages others to modify it to use in their unique work environments. 

Ratio of Researchers to Designers to Engineers/Developers

According to a study published by the Nielsen/Norman Group some years ago, the ratio of researchers to designers to developers or product can be skewed. This is where the prioritization framework stems from. According to these findings, the most common extended ratio is one researcher to five designers to 50 developers. And these ratios are probably changing with recent layoffs occurring in industry. Whereas in the past, these ratios may have been livable, now they are totally out of proportion. 

With this ratio, it is imperative to prioritize your time because people requesting research, whether designers, engineers, or product managers, outnumber those trying to provide the research. 

In the UX industry, there is a lot of talk about democratization. Whether you’re against or for it, is an irrelevant conversation. Researchers can’t survive without delegating a portion of their research activities and tasks.

This is where the framework comes into play. To use this effectively, however, there must be a change in the way researchers think about and approach their work.

The Role of a UX Researcher – Myth versus Reality

To those who know nothing about a researcher’s role, it might look like a slew of experiments, making amazing things happen while navigating successes and failures. And that would be true.

But what if we reframe how a researcher looks at their role? Instead of coming in to save the day, researchers are frequently putting out the fires to ensure that things don’t metaphorically go up in flames.

As an example, researchers want product managers to do the most effective and rigorous research possible. But what if the product manager is the problem? Then researchers need to explore a solution in the most effective, rigorous way possible. This results in a reframing of who you are as a researcher, and your role in relation to your colleagues, and your work.

The User Research Prioritization Framework

The framework came about as Jeanette discussed with her boss how to decide what to work on personally versus what to delegate to other people. That’s when he suggested a 2×2 matrix. Once she put her ideas into this format, she realized that it resonated with her colleagues and it became a meaningful and effective tool.

The framework begins with a vertical axis representing “Problem Clarity” (with a range from high at the top to low at the bottom). This first axis centers on problem clarity, which researchers deal with regularly.

This problem space is specific to customer problems, as opposed to business and product problems. This then begs the question: how well do you understand the specific concerns of your customers that you as a researcher are trying to solve or address? It might be high; it might be low. These customer problems, often referred to as Jobs to Be Done, are the issues customers are talking about and trying to achieve with or without our product. That’s different from product problems which show up as feedback requests. Product problems exist because a product exists and business problems exist because of the business. 

While there are multiple problems, stakeholders must focus on the customers’ problems within this framework. These other problems do exist and should not be minimized, but where innovation occurs is at the intersection of the three.

All design processes, including the double-diamond shown below, begin with identifying the customer problem which shows WHAT researchers are trying to solve for the customer. But it’s also to identify and mitigate risks as early as possible. Without a clear understanding of the problem, it’s difficult to know what risks might lie ahead. 

(For a more in-depth look at dealing with problems as a researcher, see Jeanette’s article, “The Problem with Problems,” https://dovetailapp.com/outlier/the-problem-with-problems.)

The horizontal axis represents the “Risk of Getting it Wrong” (also from high to low) or in other words, how bad is the risk if you get things wrong? Will customers bail right and left, or will they hardly notice?

The four quadrants created by the two axes, and recommended research methods, are as follows:

1. Research Heavy – Low problem clarity, high risk of getting it wrong. For this quadrant, Jeanette recommends doing Generative Research: JTBD interviews, Contextual inquiry, and Diary studies
2. Design Heavy – High problem clarity, high risk of getting it wrong. Recommendations: Iterative design and Evaluative research: Prototype evaluations, Click tests, and Comprehension tests
3. Research Light – Low problem clarity; low risk of getting it wrong. Recommendations: Generative research: Semi-structured interviews
4. Ship It and Measure – HIgh problem clarity, low risk of getting it wrong. Recommendations: Experimental/evaluative research: A/B testing, Behavioral analysis.

Variations on the Prioritization Framework

The great thing about this quadrant is that researchers can make it their own to fit any given circumstance. The following articles, and modified framework below, demonstrate variations on this framework and how individual researchers have leveraged it to meet individual needs. 

https://uxdesign.cc/building-a-framework-for-prioritizing-user-research-ed46622ead99
https://www.notably.ai/blog/how-to-choose-the-right-qualitative-research-methods
https://dovetailapp.com/blog/research-method-discovery-product-management/

Collaborating with Cross-functional Partners

On the issue of how to collaborate with cross-functional partners, there is no one right answer. It’s very organizationally dependent. Jeanette shared that she has data analysts and scientists that work right around the corner from her who organizationally report differently. Oftentimes, data scientists, especially in an early-stage startup, are focused more on revenue modeling. So that depends on what their remit is. But if you have people for whom that is part of their remit, those are great people to partner with. Part of the challenge is once you’ve set the KPIs or the measurements, remembering to monitor and check it, and then report back to the team.

A researcher’s role is to define what those measurements are and be accurate about what behavioral change accurately measures the success of the new functionality or solution you’re trying to implement.

Thoughts on the Prioritization Framework

Some people may be surprised that they hadn’t seen this framework before. But most would agree that its simplified nature works very well. One use of this matrix might be to plot things on the framework as a group to facilitate a good conversation with stakeholders. Another aspect to consider might revolve around how to layer in the dimension of time or window of opportunity in which research can make a difference, even complicate the prioritization of the work.

Research Impact

Some questions to consider alongside this prioritization framework might include: How impactful will this research project be? How does it align with the business priorities and the business objectives? In our current economic times, the more tightly you can connect the work you do to the priorities of the business, the better. Knowing what the priorities of the business are, and using those to help also prioritize your work, is vital. 

About Jeanette: Jeanette Fuccella started her career at IBM, as a human factors engineer, designer, researcher, strategist, and evangelist. After 16 years in this area, she moved to LexisNexis where she led design ops and research ops initiatives as well as a large global research team. She is now the Director of User Research and Insights at Pendo.