Customer Problem Stack Ranking: How Stripe Validates New Product Ideas
80% of products and features are rarely or never used, according to Pendo's 2019 Feature Adoption Report, which analysed feature usage across 615 subscriptions. Most of that waste comes from teams building for problems customers don't care enough about.
I spent over a year on 100+ customer discovery interviews only to follow the wrong user need. I've also spent 5+ years as a Techstars Community Leader and Global Facilitator, helping hundreds of early-stage entrepreneurs validate their ideas and build first-concept products. Validation is hard.
At the end of March 2021, I came across a Twitter thread by Stripe product manager Shreyas Doshi describing a validation method called Customer Problem Stack Ranking, which he had first named in May 2020.
At the time, we were struggling to get website visitors to convert and people just didn't understand why they would use our product. We decided right away to give CPSR a try on our startup. The value proposition that we spent 7 months of discovery research crafting came dead last with both of our top two target customer segments.
Dead. Fucking. Last.
We learned more in 2 hours using this stack ranking approach than we did in 100+ customer discovery interviews.
So I took that experience and built a CPSR tool to help you validate your ideas and find product-market fit.
To explain how the method works, I'm going to take a startup idea I've heard at countless hackathons: an app that makes it easier for a group of people to split the payments for their upcoming holiday. Call it Splitzies. This guide covers what CPSR is, how to write the question, how to turn your idea into a problem statement, how to find peripheral problems and participants, and how to read the results.
What is Customer Problem Stack Ranking?
Customer Problem Stack Ranking (CPSR) tells you how important your idea is compared to the other problems your target customers experience. It's a data-driven way to find out whether your idea solves a burning pain point or a mild inconvenience.
What you get back is a position in the customer's priority list, instead of a score for your idea sitting on its own. That's what separates CPSR from a rating-scale survey, where almost everything comes back clustered in the middle and nothing separates.
It also sits earlier in the process than the frameworks it gets confused with. RICE and the Kano Model rank ideas a team already believes in, using estimates the team supplies. CPSR tests whether the problem deserves a slot at all, which is why it now gets written up as a standalone product framework outside the company.
Step 1: Write your question
CPSR is a type of survey, so it needs a question. CPSR questions usually go along the lines of "What is frustrating about ____ ?"
Your question should be broad enough that participants can explore every problem attached to the activity, not just the specific one your idea will solve.
For our imaginary app Splitzies, our 'activity of focus' is booking a group holiday, so our question is: What is frustrating about booking a group holiday?
Step 2: Turn your idea into a problem statement
Asking target customers to tell you if they like your idea is a bad approach. As Rob Fitzpatrick's book The Mom Test explains, if you ask people about your idea they will just tell you it's great so that they don't hurt your feelings.
Instead, turn your idea into a problem statement so that you can compare it to the other problems customers face during the activity of focus.
For Splitzies, our problem statement could be: "Dividing the cost of a hotel booking is frustrating and complicated when planning a group holiday."
You can create multiple problem statements to explore the different pain points your idea might solve and the different words your target customers might use to talk about the activity of focus.
Step 3: Create peripheral problem statements
Brainstorm other problem statements that fall under the same activity of focus but aren't related to your idea. These can be informed by asking open-ended questions during a handful of interviews, or by reading pain-point blog posts and forums. Problem libraries like 99 startup problems and B2B customer problems are another place to draw from.
Don't worry if you feel like you've missed some peripheral problems. Running your stack ranking on OpinionX means participants can submit new problem statements to cover areas you missed at the start.
Some peripheral problem statements for Splitzies:
It's difficult to plan activities when I haven't organised a transport method like car rental or public transport.
It's hard to find out how expensive a destination is for general things like food and transport.
Agreeing on dates that suit everyone is a pain.
Some destinations are very different depending on the time of year, but good information on seasonality is hard to come across.
Keeping a list of potential Airbnbs and hotels turns into a giant messy spreadsheet.
Step 4: Send your stack rank to target customers
Send your survey link to target customers. Pick one specific segment instead of a generic demographic, to avoid noisy data. If I send my Splitzies stack rank to both young parents planning a family holiday and student backpackers, they're going to have very different priority problems and the data gets messed up. Segmenting the results afterwards works better than mixing the audiences at collection.
If you haven't got a pre-release waitlist, hit people's DMs on online communities, forums, and social media. When we used CPSR on ourselves, we joined a few Slack communities for product managers and got a +25% response rate on a couple hundred messages. That outreach took a couple of hours one evening. Here are more free tactics for finding participants.
Step 5: Iterate
The most important part of running a CPSR is growing your problems list as you discover pain points you missed at the start. Based on OpinionX product data, 81% of stack rankings run on the platform finish with new opinions added by participants sitting in the top 3.
That's why we recommend against using a normal survey for this. Tools like SurveyMonkey, Typeform and Google Forms lock you in with whatever blind spots you had when you wrote the multiple-choice list, because participants can only pick from what you thought of first. Traditional surveys fail at idea validation for exactly this reason.
Step 6: Read the results
Once you've got enough participants, sort all the problems by highest importance to stack rank your statements. In one click, you'll see how your value proposition and your idea rank against the other problems your target customers face.
As I said, the results of our stack ranking showed us that the value proposition we had spent 7 months building came dead last with our target customers. What surprised us was that the big picture problem we were solving really was important to those customers. The words we had wrapped around it were the problem.
Having the right problem and the wrong words for it costs you the same as chasing the wrong problem.
After our CPSR, we took 5 of the top 6 most important problems from our stack rank results and rewrote our entire landing page and onboarding experience. By the end of that week, conversion, onboarding success and referrals had all multiplied.
Step 7: Go
Run it now. CPSR works the same whether the job is validating a startup idea, fixing your messaging, or picking the next big problem to tackle for your product.
The full write-up of the experiment we ran on ourselves is on HackerNoon. The Discovery Sandwich covers where the method fits alongside qualitative work. For long lists, see pairwise comparison and how to rank long lists. If the underlying method is new to you, start with stack ranking.
OpinionX was built for Customer Problem Stack Ranking, and other tools have since added their own versions of it. Every question type and every analysis feature is unlocked on the free tier, capped at 25 participants per survey, so you can run a full CPSR and read the results before deciding whether to pay. Setup takes about 4 minutes. Create a free stack ranking survey
Daniel Kyne is the Co-Founder and CEO of OpinionX, an advanced market research survey platform used by product teams to validate ideas and find product-market fit. Daniel is a Techstars Global Facilitator, a Global Shaper at the World Economic Forum, and a former Digital & Innovation Lead at Unilever UK.
Over 42,000 researchers and product people get one step-by-step guide like this each week in The Full-Stack Researcher.