How to do Mixed Methods Research without being a Quant Expert
The way that companies think about user experience has changed. The end user now makes the initial purchase decision to solve their own pain, not an executive thinking of bottom line impact.
This is called Product-Led Growth and it is creating a tidal wave of change across tech, particularly in UX and product teams. If the buyer is now the end user, then user experience is your new salesperson. There is less tolerance for bad UX than ever before.
Understanding end user pains is a discovery-driven research process that relies heavily on qualitative methods. But in nearly all cases, tech companies subscribe to quantitatively-justifed decision making. This conflict between qual research and quant appetite has driven a renewed interest in democratising a blend of the two — aka mixed methods research.
Companies like Facebook, Amazon and Microsoft are all recruiting mixed methods researchers RIGHT NOW. If they ask a new research hire to jump from user interviews into quantative analysis they don't want to hear a meek "But that's not part of my research specialism." That's not how these companies work. They want solutions.
If you're a user researcher who leans heavily on a qualitative skillset, you're probably thinking "Shit, I did all this work to become a qual specialist. How am I going to learn this complicated quant stuff?". Quant can often feel pretty alien and intimidating to a people-driven qual researcher. But you don't need a PhD in statistics to become a mixed methods researcher. There are only a handful of core skills you really need.
🥉 Master the Basics
Pareto's 80/20 rule says that 20% of your effort will account for 80% of your results. This applies to learning new skills too. Master the basics and you'll be covered for the majority of day-to-day mixed methods requirements. This won't apply at every company, but it's a great place to start either way.
1: Surveys (🥉)
Learning how to conduct and analyse quantitative surveys is the bread and butter of mixed methods. Online surveys can help you to validate and measure the insights gathered from qualitative research like user interviews.
Surveys should come with a big fat disclaimer though; it's easy to create a survey but hard to write a quality questionnaire or gather accurate and actionable insights. And we've all participated in a terrible survey before, so I don't even need to go into this further. Be careful out there.
Analysing surveys can seem straightforward with automated dashboards on most survey platforms, but be warned — bias and poor representation are the carbon monoxide of surveys. They'll kill you in your sleep if you're not aware of them.
Quantitative UX research, just like qualitative, delivers insights about people. All user research is human-centric, don't lose sight of this as you start diving deeper into the numbers.
The first step to better surveys is writing good questions. Good questions steer clear of your own biases and avoid leading the respondents towards predisposed answers. Suggested reading: How to Create Effective User Surveys.
In order to be confident that your results represent the user segment you say they do, you'll need to learn how to do sample sizing. This is often the area of most friction between user research and product teams. Understand how sample size robustness, margin of error and participant screening work. Here's a good starting point: Determining Sample Size: How Many Survey Participants Do You Need?
Once you're ready to analyse results, Hubspot's How to Analyze Survey Results Like a Data Pro can guide you through the process.
The most important survey format for startups is called Customer Problem Stack Ranking — a research format from Stripe for stack ranking people’s priorities to inform better deicions with real data.
2: Analytics (🥉)
“Without big data analytics, companies are blind and deaf, wandering out onto the web like deer on a freeway.” – Geoffrey Moore, author of Crossing the Chasm
Analytics platforms like Mixpanel and Google Analytics make it easy to track user behaviour without needing to write any code. Installing tracking codes on your website is pretty straightforward and you can start collecting data and generating insights right from the get go.
Analytics is rabbit hole that you can easily get lost in for hours on end. Set clear goals to avoid getting distracted by shiney but unactionable insights. As an analytics newbie, focus on audience and issue identification.
Analytics tools help you to segment users with criteria like demographics, device and behaviour. In this way, you can build more detailed personas and better understand segment commonalities. Google Analytics Academy offers free mini-courses with everything you need to get started.
Digging into metrics like bounce rate, high drop off points and rage clicking can help you to identify undiagnosed issues in your user experience. Mixpanel is widely used for this type of analytics. It's great for tracking millions of events to get a really comprehensive understanding of how individual users interact with your product. Start with their free course, Introduction to Mixpanel.
🥈 Level Up
Surveys and analytics are the most common quant methods used in mixed methods research. If you'd rather not move beyond these two research methods today, then click here to jump to examples of these quant basics being used in mixed methods research projects. Continue reading to learn about intermediate-level methods that will position you as an equally capable qual and quant researcher.
3: Quantitative Usability Testing (🥈)
Quantitative and qualitative usability testing both cover similar territories by asking users to perform everyday tasks on your product. Quant usability testing focuses on measurable aspects like user performance (eg. time spent on a task, percentage that successfully completed a task) or user perception (eg. satisfaction ratings). These metrics can be gathered alongside observational insights, blending qual and quant together.
Unmoderated usability testing is another quantitative method. Rather than personally facilitating a usability session, unmoderated tests observe larger sample sizes. Unmoderated tests can be set up through platforms like UserTesting.com or by analysing existing user behaviour data through a platform like Mixpanel.
4: A/B Testing (🥈)
"The goal of a test if to get a learning, not a lift. With enough learnings, you can get the real lift." - Dr. Flint McGlaughlin, Founder of MECLABS
Start with a goal in mind, eg. 'I want to increase conversion rate', create a control and variable version of what you're testing and observe the data to see which version performs best. A/B testing can be made increasingly advanced, adding more variable versions and testing with different user segments.
A/B testing can be broken down into a 5 step process: (1) Identify a goal, (2) Form a hypothesis, (3) Design and run the test, (4) Analyse the results, (5) Implement the results.
A/B testing enables incremental improvements to your product that make a big difference to user experience over time. The best teams methodically use A/B tests to optimise landing pages, buttons, copy and more. A/B testing removes the designer bias by outsourcing product decisions to users via their behaviour.
Researchers rarely carry out A/B tests on their own. Understanding how A/B tests work will enable you to better collaborate with product or development teams who typically own these experiments.
UX Booth has written a great 5-step guide to getting started. Other helpful resources include Smashing Magazine's Ultimate Guide to A/B Testing and Unbounce's How to Formulate A Smart A/B Test Hypothesis (and Why They’re Crucial).
🥇 Go for Gold
At this stage we're moving onto the opposite side of Pareto's 80/20 rule — the methods that will require 80% of your time to master but only account for 20% of your quant requirements as a mixed methods researcher. These methods aren't for everyone. If you want to learn how to combine the topics already covered in mixed methods projects, jump to the examples section. Alternatively, if you feel up for a challenge, I salute you! ⏩
5: Statistics (🥇)
Most qual-first users researchers have fled in fear by now. But if you decide to become an expert Full Stack Researcher, a foundation in statistics will be important.
Statistics enable a deeper dive beyond the insights capabilities of generic tools. It can uncover hidden correlations and relationships between user behaviour data points and inform segmentation decisions. In the best scenarios, statistics can enable you to move beyond segmentation altogether towards true user experience personalisation.
If you want to dip your toe into areas of statistics useful for UX, consider reading the book Quantifying The User Experience: Practical Statistics For User Research by Jeff Sauro (who was featured on our list of 21 User Researchers to Follow on Twitter). Or for a quicker overview, check out Andy Park's blogposts Using statistics in UX design.
If you want to jump into more specific skill-based learning, give SPSS a go with the book Discovering Statistics Using IBM SPSS Statistics.
🏆 Mixed Methods Research Designs
The best way to test your new quant knowledge is by giving a mixed methods research project a try. There are three main types of mixed methods: exploratory, explanatory and dynamic.
Exploratory Mixed Methods (Qual → Quant) 🔭
Exploratory Mixed Methods Research puts the qualitative step first to establish foundational knowledge about your research topic. Exploratory research is particularly helpful when you have a lot of unknown unknowns.
A common approach to exploratory mixed methods is to conduct user interviews to investigate a topic in-depth by gathering opinions, perspectives, motivations and unmet needs of research participants.
The hypothesis formed from this qualitative step will inform your quantitative survey questions. The survey results enable you to robustly measure and validate your qual insights with a statistically significant sample size.
Check out a real life exploratory mixed methods example here.
Explanatory Mixed Methods (Quant → Qual) 🔍
Explanatory Mixed Methods Research puts the quantitative step first. It's especially useful when you've already conducted a quant research project and you need to understand unexplained or surprising insights.
A common approach for explanatory mixed methods is to analyse user analytics or survey data. During this step it is likely that you will see what people are doing but it may be difficult to understand why. Bring these why questions into a qualitative deep-dive session to learn more about the context behind the figures. Participants can share the reason for their actions through an interview which will enable you to build a more comprehensive understanding of the topic.
Check out a real life explanatory mixed methods example here.
Dynamic Mixed Methods (Qual + Quant) 🔮
Dynamic research overcomes the time-consuming aspect of mixed methods research by blending qualitative and quantitative data together at the same time. An entire mixed methods research project can be run within one process with a dynamic research tool, unlike the multiple steps required for exploratory and explanatory research.
For example, OpinionX enables dynamic mixed methods for discovery research by letting participants vote on each other's opinions — ie. adding a quant dimension of data to otherwise unstructured qual input. This removes the need for the researcher to manually take insights from the first step and translate them into a different research method, which can be a veeeeery time consuming task. And who doesn't like faster, richer and more actionable insights :)
The result is a harmonious blend of qualitative user opinions that are easy to prioritise with quantitative sorting methods based on consensus and importance.
Check out a real life dynamic mixed methods example here.
👉 Learning as you go
The UX research industry has been heading towards mixed methods for years. The introduction of Product-Led Growth has only served to accelerate this trend.
Adapting your wheelhouse to introduce mixed methods research can feel intimidating. The reality is that you don't need a formal background in quantitative research to get started with mixed methods research.
Through the resources linked above (bookmark this page to revisit them) along with a bias towards hands-on experimentation and trial & error, you'll be on your way to becoming a Full-Stack Researcher in no time.
👊 — Written by Éamon Cullen and Daniel Kyne from OpinionX
🙌 Bonus Tip: Immerse yourself in online communities
UX research is one of the fastest changing functions in tech. Staying on top of best practice and the latest content will help you to explore the topic deeper and get the most from your findings. One of the best ways to stay up to date is by following the experts.
Here are 21 User Researchers you NEED to Follow on Twitter in 2021
These are the Top UX conferences to attend in 2021
Get an email every two weeks on the future of user research by subscribing to our newsletter: