A Simple Method For Ranking A Long List of Options
There’s an endless number of survey tools you can find online for ranking 5 options. But what should you use when you’re trying to rank a list of 10 or more options?
Or what if you’ve got 100+ options to rank — what then?
All the major survey companies make it pretty clear that you shouldn’t use their products:
SurveyMonkey says “limit the number of ranking choices to about 5. Asking respondents to rank too many items in relation to each other can be overwhelming.”
Qualtrics says “limit your ranking question options to between 6 and 10 items, and don’t read too much into the middle rankings.”
Slido says “we recommend keeping the ranking list short — sometimes less is more :)”
What they’re basically saying is that you should shortlist the options yourself, which defeats the whole purpose of using a research tool in the first place. But the good news is that there are two survey formats specially designed for ranking long option lists.
In this post, I explain (1) why the big survey companies don’t want you to use their drag-and-drop rank order format, (2) two better methods for ranking long lists, and (3) some real example of them in action.
Why are survey companies saying not to use their products for long rank order questions?
The amount of mental work required to drag-and-drop rank a list increases exponentially with each new option you add.
To rank an option effectively, you have to compare it against every other option you’ve currently got ranked. To rank 3 options we would have to make 6 comparisons, for 9 options that grows to 45 comparisons, and for 15 options you’d need 120 mental comparisons ( (n²+n)/2 ). What seems like a simple task is actually a cognitive challenge that scales exponentially…
It’s not surprising that a 2016 study found rank order questions become increasingly error-prone and time-consuming as more options are added. So, in a way, the survey companies are right — you shouldn’t use their tools to rank long lists of 10+ options. Well, this begs the question…
Why aren’t there question formats suited for longer ranking lists?
There are! Despite these companies raising billions in funding, they simply don’t care about ranking research. Personally, I find that to be pretty outrageous — comparative ranking models are one of the best research methods for understanding people’s priorities. But really tools like SurveyMonkey aren’t meant for real research; they’re just online versions of traditional pen-and-paper questionnaires.
In 1927, a scientist called L. L. Thurstone wrote a paper about a ranking idea where a list of options is broken down into a series of head-to-head pair votes to measure people’s values or attitudes. Today, we call this approach “Pairwise Comparison” (or ‘Pair Rank’ for short).
^ Pairwise comparison example for ranking a set of images
The Advantages of Pair Ranking
The best part about Pair Ranking is its flexibility.
If you’re trying to rank a personal list of options, you can just complete all possible pair combinations and you’ll get a full ranked list at the end. If other people will help you rank the list, you can give each of them a smaller set of pairs to vote on and then aggregate everyone’s results together to get the final ranking.
This flexibility can be pushed to the extremes. In 2021, I helped a researcher run a pair ranking survey with 800+ options and over 30,000 votes cast by participants.
How does scoring work in a Pair Rank survey?
Whether your pair rank list will have 1 voter or 10,000, you’ll still end up with the same output — each option is ranked from 1st to last using a 0-100 score based on the percentage of pair votes that option won. For example, if an option appears in 5 pairs and wins 4, its score would be 80 (ie. 4/5 = 80%).
^ Results table for a Pair Ranking survey on OpinionX
[Real Example] Using Pair Rank To Rank A Long List of Options
Hazel Byrne was completing her Master’s in Global Challenges for Sustainability and had to pick a topic for her capstone project.
She was struggling to prioritize from her list of 33 options, so she decided to run a personal pair ranking survey to help her decide. The survey gave her every combination of the 33 options as head-to-head pairs and asked her to pick between Option A or Option B each time.
In the end, Hazel had a clear winner with a 100% win rate that she decided to pursue.
What survey tools have a pair ranking question?
None of the major survey tools offer pairwise comparison questions. This includes Google Forms, Microsoft Forms, Qualtrics, SurveyMonkey, Typeform, Jotform, Slido, Mentimeter, etc.
However, we created a free tool called OpinionX specifically for ranking surveys because we weren’t happy with the research formats offered by the traditional survey tools. Today, thousands of teams from companies like Google, Amazon and Microsoft use OpinionX to let their customers, colleagues and communities rank what matters most to them.
You can set exactly how many pairs you want each participant to vote on and you can also include other survey questions like multiple choice and email collection. For users that require advanced analysis like segmentation and branching, we offer those features on our premium tiers.
Below is an example of a free pair ranking survey run on OpinionX:
left: voting / right: results
There’s no reason not to give OpinionX a quick try — it’s completely free and you can have your first pair ranking survey setup and ready to go in under 5 minutes. Get started here.
Need an even faster way to collect data?
MaxDiff Analysis is very similar to Pair Ranking, except instead of showing just two options at a time, it shows 3-6 and asks participants to pick the best and worst choices from the list. By showing more options per vote, MaxDiff is a more information-dense collection method than Pairwise Comparison.
Alongside Pair Ranking, OpinionX also offers free MaxDiff Analysis surveys with unlimited participants → more info here.