7 Types of Ranking Exercises For Focus Groups (Online & Offline)
Most online guides to creating ranking exercises for focus groups are over a decade old and focus on theoretical frameworks like “Participatory Ranking Methodology”. That’s not what this post is.
Instead of boring you with more academic theory and jargon, this post shares 7 simple methods you can use to create engaging ranking exercises for your focus groups, along with some useful tips and real examples too.
Topics covered:
3 In-Person Ranking Exercises
Ok, let’s dive in…
What should we consider when designing ranking exercises for focus groups?
These principles apply regardless of whether your focus group is in-person, online, or a hybrid of both.
Principle 1: Comparative
In real life, decisions are comparative; you always have a bunch of different options, trade-offs, problems, ideas, or opportunities to consider. Ranking exercises should reflect this and ensure that participants are making decisions between available options rather than simply rating each one independently. An example of what not to do is the Likert Scale (rating each option on a 1-5 range from “very satisfied” to “very dissatisfied”). If someone can rate every option 4/5 and ruin your output data, then your ranking exercise was poorly designed.
Principle 2: Repeatable
Robust research is repeatable research. This is particularly relevant in focus groups, where you’ll likely have to combine the data from each focus group into one overall result to present back to your team. The best way to ensure repeatability is to have a structured ranking process — if it can’t be repeated identically each time (eg. using an online tool), at least hire an experienced facilitator to keep participants from unduly influencing each other’s voting.
Principle 3: Countable
Ranking exercises are great for focus groups because they translate qualitative insights into quantitative data better suited for informing decisions. If you allow participants to decide on the ranking list themselves, you risk ending up with a list of qualitative statements that can’t be counted. However, at the end of this post, I explain one way to do crowdsourced ranking with focus groups.
Online Ranking Methods for Focus Group Research
There are four online ranking methods we can use that meet all three of our design principles:
1. Pair Ranking
Pair Ranking, also known as pairwise comparison, is a ranking method that uses a series of head-to-head votes to score and rank a list of options. Instead of asking participants a painstakingly drag and drop a long list of options into their order of preference, Pair Ranking simplifies the process by giving participants easier mini-tasks that culminate in the same valuable ranked data at the end.
You can create Pair Ranking exercises for free on OpinionX. Each option gets a score from 0-100 showing what percentage of pair votes it won. You can even customize how many pairs you want each participant to vote on — whether you want a quick 5-vote sample from each person or a full ranking of all possible pair combinations.
2. Order Ranking
This is the traditional “drag-and-drop” list that you’ll find on many survey tools. For shorter lists of 10 options or less, Order Ranking is a quick and easy way to identify people’s priorities. Beyond 10 ranking options, using Pair Ranking will ensure higher-quality results.
The Order Ranking question type on OpinionX uses a scoring formula called the Dowdall Method. When a participant ranks a list of options, each option’s score is 100/rank (ie. 1st gets 100 points, 2nd gets 100/2 = 50, 3rd gets 100/3 = 33, and so on). The Dowdall Method is ideal because it’s very easy to understand and each participant isn’t required to rank the same amount of options to produce a normalized final score.
3. Image Ranking
The best way to do online Image Ranking is to use the same format as Pair Ranking — breaking the list of ranking options into a series of head-to-head pair votes. This is particularly relevant for survey respondents using mobile devices (over 56% of surveys are completed by phone as of Q4 2022), where there’s just not enough screen space to drag-and-drop rank multiple large images.
4. Points-Based Ranking
Comparative ranking methods like Pair and Order Ranking are great at understanding the relative importance between a set of options, but sometimes they can hide the true magnitude of difference in preference. A great way to study the magnitude of importance is to allocate a certain number of points to each participant and allow them to ‘spend them’ amongst the ranking options. You may soon realize there’s a larger gap between your top two options than expected!
Benefits of Online Ranking Methods
i. Scale
Online ranking methods work just as fast with 2 people as they do with 2,000. This scale gives you peace of mind that your tooling won’t let you down, regardless of your research requirements.
ii. Segmentation
You can create segments for each of your focus groups, allowing you to use one survey to calculate the overall results for all participants and then filter the ranking to compare how each individual focus group differed from each other.
iii. Speed
Because results are instantly calculated by OpinionX, you can spend your extra time sharing the results with the group. This can open an iterative discussion where participants self-reflect and share their reactions to the overall results, helping you understand the “why” behind their priorities.
iv. Sophistication
Online ranking tools come with more advanced features that allow you to do things at scale that you can’t do with paper-based exercises. For example, you can run a ranking survey in two steps, first collecting people’s written opinions and then adding those opinions to a Pair Ranking block to see whose statement is the best articulation of the room’s collective opinion.
3 Offline Ranking Formats for Focus Group Discussions
Time to break out the sticky notes…
1. Dot Voting
From 2014 to 2020, I was involved in organizing hackathon events called Startup Weekends. Anyone who pitched a product idea would have to create a quick A1 poster on flipchart paper. Then, we’d give every participant 3 sticky notes to vote for their favorite ideas.
Dot-voting is a very quick and easy way to surface the overall favorite options within a large group (although it would take all-hands-on-deck to count and sort 300+ votes in just a couple of minutes. It’s also a popular online ranking method, with free tools available like Lucidchart and Miro.
2. Group Sticky-Note Stack Ranks
Group stack ranks are where each person ranks their personal priorities and then the overall results are analyzed for similarities. It’s a great way to move a situation from open conversion to deeper discussion about group priorities. At Startup Weekend hackathons, we used to get teams to use both of these techniques on Saturday mornings to align what they should be working on.
→ 2a. Freelisted
If teams are still early in the brainstorming process, a freelisted stack rank will help surface consensus opinions. Each participant gets 10 sticky notes to write the 10 different things (anything they want) which they think are most important in the situation and sticks them in a vertical line on a blank wall or board. Freelisted sticky-note stack ranks break all three of our design principles and should really just be used as a shortlisting exercise early in the conversation, rather than a research method to understand team priorities.
→ 2a. Shortlisted
The same approach is followed for shortlisted sticky-note stack ranks, except the top ~10 priorities are already largely agreed upon. This time, each person is ranking the same list of 10 options from 1st to 10th. This approach can be scored by counting the positions for each opinion (eg. ranked 4th = 4 points), then sorting the results list from lowest points at the top to most points at the bottom. This is an inclusive decision-making approach for workshops, hackathons and focus groups.
3. Yes/No Secret Ballots
The easiest of the focus group ranking exercises is the yes/no secret ballot. Each participant writes Y or N on a piece of paper, folds it up, and throws it into the middle. All the slips of paper are counted up (often by the group to speed things up) and then the next ballot is cast.
While this method meets many of our principles by being repeatable, countable, and minimizing group voting influence through anonymous voting, it is generally not comparative. You can give multiple options and allow participants to cast a secret ballot for their top option (eg. numbering the list 1-10 and participants write the number of the option they like most), but you’re losing a bunch of rich insights into their overall priorities — which is especially important in non-consensus topics with a wide range of opinions.
Challenges with paper-based group ranking methods
i. Repeatability
Interpersonal influence is much more likely to impact participants in offline ranking exercises where others can see how you vote. An experienced facilitator can help level the playing field of personalities in the room to a certain extent, but this is an unavoidable aspect of in-person, discussion-based group settings.
ii. Scale
Paper-based ranking formats require more time to count votes. The bigger your group or the longer your list of ranking options, the more time that’s required. As a result, offline ranking methods are difficult to scale to larger group settings.
iii. Time
Due to the time-consuming nature of manually analyzing offline ranking results, facilitators tend to have less opportunity to bring the results of the exercise back to the group for further discussion. This recursive, self-analyzing process often produces the richest insights into the “why” behind a focus group’s collective priorities.
How to pick the right activity format for your focus group research
Ranking exercises are a great way to create engagement and participation in your focus group while also generating valuable quantitative data for future decisions. When you’re paying for participant and facilitator time, you want to be sure that you’re getting the most value possible per minute in your research — online ranking is the most efficient way to achieve this.
You can create unlimited ranking surveys for free on OpinionX using every online ranking method mentioned in this guide. Companies like Google, Shopify and Amazon trust OpinionX to help them identify and measure what matters most to their customers, colleagues and communities.
Create a focus group ranking survey in under 5 minutes for free.
About the author:
Daniel Kyne is the Co-Founder & CEO of OpinionX, a research tool for stack ranking people’s priorities. He was previously Digital & Innovation Lead at Unilever UK, where he received a consumer research award and organized focus groups with some of the top market research agencies in the world.