What Is Stack Ranking: Meaning, Examples, Templates, Advice
This guide covers everything you need to know about stack ranking — from its meaning and examples of stack ranking used in real life, to some simple templates and actionable advice to consider.
Contents:
What is stack ranking? How is it different to normal ranking?
Definition: stack ranking is a method for ranking a list of options based on their relative importance.
Unlike regular ranking methods, stack ranking is used when there’s no objective data points that can be used to rank the options. Instead, stack ranking compares the options against each other subjectively in order to put them in order from first to last place.
Here are three examples that can’t be considered stack ranking:
▸ Spotify ranks musicians based on their total number of streams in the previous 30 days.
▸ The Premier League ranks soccer teams in England based on their points on the leaderboard.
▸ IMDb ranks the “Top 250 Movies Of All Time” based on average viewer ratings from 1-10 stars.
None of these examples can be considered stack ranking because they each use an objective data point to sort the list from first place down (monthly streams, league points, average user rating).
Here are three scenarios that can be considered stack ranking:
▸ Asking customers to rank a set of problems based on which they’re most keen to get solved.
▸ Each team member ranks the interview candidates based on suitability for the role.
▸ Ranking your favorite movies from the Marvel cinematic universe.
In these three examples, there’s no obvious data you can use to rank the options, but by comparing the options against each other, we can stack rank them in order of relative preference.
The Origins of Stack Ranking
[Then] Stack Ranking Employees
Stack ranking originally referred to the employee evaluation method (popularized by G.E. CEO Jack Welsh in the 1980s) where managers rank their employees and rewards the top 10% while firing the bottom 10% of performers. Employee stack ranking went out of fashion during the 2000s — Microsoft officially stopped using it in 2013 and companies like Facebook and Amazon have been claiming for years (to mixed degrees of believability) that they also no longer use it.
[Now] Stack Ranking for Agile Prioritization
Stack ranking took on a new meaning after the launch of The Agile Manifesto in February 2001. Agile’s focus on maximizing value for customers and continuously improving the product after you deliver it. This meant teams had to become excellent at prioritization — figuring out which customer needs were most important to address first, which parts of the existing product to improve next, which ideas had the most risk or reward potential. Stack ranking is a perfect tool for these exact scenarios. While agile stack ranking started off with sticky notes on a wall, it has since developed into more advanced methodologies like Pairwise Comparison, the KANO Model, the MOSCOW Method, RICE scoring, and more.
Example Of Stack Ranking Used In A Real Life Scenario
Back in early 2021, 6 months after launching my first startup, we had just lost our only customer and were trying to figure out why nobody seemed to understand the value of our product. We decided to run one last experiment before giving up, where we would ask a bunch of our target customers to stack rank a list of problems so we could see where the key problem we were focused on solving ranked in their list of priorities.
In just one evening, we drafted a list of problem statements based on our customer discovery interviews, shared them with 600 target customers, and could already see in the first two hours of voting that our key problem statement was ranked dead last out of 45 options. But the biggest surprise for us was that 5 of the top 7 highest-ranking problems were actually very similar to the one we were trying to solve!
So the next morning we rewrote our landing page and product onboarding flow to match these 5 high-ranking problems instead. Within a week of starting this stack ranking experiment, we had multiple paying customers, our landing page was converting to trial 3x better than before, and we even had our first customer testimonials!
To learn more about this “customer problem stack ranking” experiment, check out the short video case study below or read this viral blog post explaining the process we followed.
Stack Ranking Methods & Scoring Techniques
1. Sticky Notes
Method: Find a wall, a stack of sticky notes, and a sharpie. The process is the same whether one or many people are ranking — stack your options from highest personal preference down to lowest.
Scoring: Two techniques depending on situation. If participants are all ranking the same list of options, then a Simple Borda Count will do — 1 point to first place, 2 points to second, etc., calculate the average per option, lowest points ranks highest. If people have different length rankings, use the Dowdall Count method instead — 1st place gets 100 points, 2nd place gets 100/2 = 50 points, 3rd place gets 100/3 = 33 points, etc. Using Dowdall Count means everyone’s 1st place gets the same amount of points while also allowing people to only rank what’s relevant to them without penalization.
2. Pairwise Comparison
Method: Pairwise comparison is an online survey format that takes two of your ranking options at a time and puts them in a head-to-head pair vote. These pair votes are repeated multiple times for each stack rank participant.
Scoring: For each option, calculate the number of wins divided by the total number of pairs it appeared in. When using a pairwise survey tool like OpinionX, this calculation is done automatically.
Example of Pairwise Ranking on OpinionX
3. Points Allocation
Method: Each participant is given a budget of points to allocate amongst options at their own discretion. Ideal for understanding the magnitude of people’s preferences, not just the relative order. Available in many online surveys (check out this list of free points allocation tools) or can be done offline using dot voting.
Scoring: Add up the total number of points per option, if needed you can average the result based on number of participants.
Example of Points Allocation ranking on OpinionX
4. Drag-And-Drop Rank Ordering
Method: Online survey format where participants are shown all ranking options and asked to drag-and-drop them into their order of preference. Only suitable for ranking a small list of options (maximum of 6-10 options).
Scoring: Dowdall Count gives each option 100 points divided by its rank position, for example 100/4 for 4th place = 25 points. Calculate the average number of points, rank from highest to lowest. Dowdall Count doesn’t require every participant to rank the same number of options, meaning people can leave out ones that they don’t understand or aren’t relevant to them.
Example of Rank Ordering on OpinionX
5. Consensus Ranking
Method: Ask participants to vote “agree” or “disagree” on each option. While this isn’t technically stack ranking, as options aren’t being directly compared against each other, it is great for finding the highest consensus option rather than the option with the highest perceived importance.
Scoring: Number of agrees over total votes gives you the consensus percentage for each option.
Example of Consensus Ranking on OpinionX
6. Image Ranking
Method: Using the same underling voting format as Pairwise Comparison, image ranking lets you use visual mockups, sketches or photographs to make voting more engaging.
Scoring: Wins divided by number of pair votes (same as Pairwise Comparison).
Example of Image Ranking on OpinionX
7. MaxDiff Analysis
Method: MaxDiff is a more complex version of Pairwise Comparison. Rather than just showing 2 options at a time, it shows 3-6 and asks respondents to pick their best and worst candidate from the list. Note: MaxDiff is regarded as an advanced research method and there are no free/cheap survey tools that offer MaxDiff questions (here’s a list of the 10 most popular tools for MaxDiff).
Scoring: MaxDiff tools tend to use Bayesian statistical models to analyze voting patterns in order to stack rank your list of options, so the scoring methods they use are pretty advanced and complicated. Alternatively, you can manually calculate MaxDiff results by subtracting the number of times an option was chosen as least important from the total count of times it was chosen as most important by respondents.
Example of a MaxDiff Analysis survey
When To Use Stack Ranking (Use Cases)
I built a number of stack ranking templates, which can be found on the Sample Survey gallery inside OpinionX — a research tool we built specifically for stack ranking (used by teams at Google, Amazon, and Microsoft). My stack rank templates include ones for roadmap prioritization, idea validation and team workshops.
1. Roadmap Prioritization
At OpinionX, we look at all the feedback we’ve received from users each quarter and create a list of problem statements. We take that list of problem statements and share it with our entire user base using a Pairwise Comparison ranking survey. Then we filter the ranked results to look at the top ranking problems according to customers on each of our pricing tiers. This data then helps us figure out what problems to focus on each month — eg. what’s preventing activation and conversion (top problems for free users), as well as retention (the biggest pain for each pricing plan cohort). More info on stack ranking for roadmap prioritization.
2. Idea Validation
The best way to validate your product idea is to prove that it is a high-priority problem that your target customer is urgently trying to solve. The easiest way to do this is to interview a sample of those target customers, take all the problems they tell you about, stack rank them to measure their relative importance, and then see how high up your target problem ranks. Here’s a full guide to data-driven idea validation using stack ranking techniques.
3. Team Consensus Check
Ever found yourself in a meeting where everyone seems to disagree on the topic being discussed? Bridging disagreement in these scenarios is hard without establishing common ground. Ask each person to write down their opinion, put it into a stack ranking survey, and share the link for people to vote. Active participation helps to get buy in from participants and quantitative results helps to solidify a clear direction to the conversation.
4. Customer Segmentation
One of the most common mistakes that teams make is trying to target multiple customer segments with the same generic value proposition. It’s often feels like the only way to truly map each segment’s main need would be to spend a week interviewing them each separately — which is crazy expensive and time consuming. Alternatively, stack ranking their collective priorities and then filtering the results to compare how they change depending on job title, seniority, geography, industry category, etc. can help you see which needs are driving behavior for each segment (ie. Needs-Based Segmentation).
5. Message Testing
When it comes to picking your value proposition, landing page copy, or sales deck messaging, you’re generally left with one options — build the whole thing and see how target customers react *after* you ship it. There’s a better way to do this! Pairwise comparison is a great way to model how people make decisions in the real world because it doesn’t rely on stated preferences — you’re not asking people to tell you what they like, instead their voting reveals what they like. That means you can give people a list of messaging options and measure which ones will get the most clicks, all without having to build one mockup or prototype.
6. Measuring Feature Value
It can be hard to know which features on your premium plans you should be shouting about loudest — especially when your pricing tiers start to fill up with more and more paywalled features. There’s a simple way to use stack ranking to measure which features your customers perceive to be the highest value. Invite users to a pairwise comparison survey that includes a list of “capability statements” (the underlying ability each feature enables) along with the question “If our product could only do one of the following things for you, which would you pick?”. Whichever capabilities rank highest in each tier are the features driving the most value — increase their visibility to users in lower pricing tiers. For more on this, check out this guide to the Value-Adoption Matrix.
Tips To Consider When Stack Ranking
1. Pick Your Stack Ranking Method Carefully
The most commonly available online ranking format — drag-and-drop rank order questions — should not be used if you have 6+ ranking options. If in doubt, check out the range of stack ranking methods here.
2. Crowdsource Your Stack Ranking Options
In a 2021 study, we found that 81% of stack ranking surveys that allowed participants to submit new ranking options had participant-submitted options in their top three ranked results. Especially when it comes to Customer Problem Stack Ranking projects, your participants likely know the right terminology and context to describe their own problems better than you can. My advice: after your stack ranking question, include an optional Text Response box for participants to submit new statements if there were options they considered but didn’t see while voting.
3. Always Segment Your Results
The most impactful way to understand stack rank results is to compare how those results change when you look at isolated segments of your participants. Whether your participants can be split into different job types, pricing plans, geographies — whatever way they can be divided — compare the ranked results for each group separately (ideally using an automated segmentation filter) to understand how priorities change between segments. Here’s a 60-second video that shows exactly why you need to segment your stack ranked results:
Choose A Purpose-Built Stack Ranking Tool
We built a research tool specifically for stack ranking called OpinionX. Thousands of companies (including teams at Google, Amazon, Shopify, Salesforce and more) use our tool to measure what matters most to their customers, colleagues and communities.
You can create as many free stack ranking surveys as you need using any of the ranking methods described in this post — with no time limit or credit card required. Create your first stack ranking survey right now in under 5 minutes:
— — —
Thanks for taking the time to read this post! If you enjoyed it, consider subscribing to my newsletter The Full-Stack Researcher, where I share actionable research advice with thousands of product teams and startup founders: