New Ranking Formulas (April 2023 Product Update)
On April 12th 2023, we shipped new ranking algorithms for Pair Rank and Order Rank questions on OpinionX. This post explains why we made that change, the math behind the new methods, and what this means for your stack ranking surveys going forward.
Contents:
Why we’re updating the OpinionX ranking algorithms
Until today, April 12th 2023, OpinionX has calculated the score of opinions in stack ranking surveys using a data science approach called Glicko-2 — a Bayesian rating system for scoring players’ skill levels in popular online games like Counter-Strike, Chess.com and Pokémon Go.
Every opinion on OpinionX starts with a score of 1500 points. Depending on how participants vote, those scores can end up anywhere on a range from ~800 up to +2100. The most common question we get from users every single week from OpinionX users is to explain how Glicko-2 calculates these scores.
Unfortunately for researchers, Glicko-2 is a bit of a black box. We could run simulations to ensure that it ranked data accurately for our Pair Rank and Order Rank questions, but we couldn’t find a way to explain individual survey results beyond a frustrating “you just have to trust the system” response. This often left researchers in a very difficult position. If even one teammate questioned the validity of their results, it often completely destroyed the entire research project.
Beyond this, it also carried a bunch of annoying conditions. Glicko-2 is made for scoring large data sets like multiplayer games. This meant that stack ranking surveys needed a bunch of votes before the algorithm would have a high level of confidence in its scores. For researchers with small participant pools (like internal surveys or many B2B projects), this pretty much ruled OpinionX out immediately. The ‘robustness’ and ‘volatility’ variables also added additional confusion and friction.
Our aim at OpinionX is to build the best research tool for understanding people’s priorities. We should be able to say that OpinionX has the best ranking methods of any similar research tool, but with so researchers unable to accomplish their objective on OpinionX, this clearly wasn’t the case. That’s why we decided to update the ranking methods that power OpinionX surveys.
Dowdall: The new scoring method for Order Rank questions
The Dowdall Method is a Borda Count variation used for elections on the island nation of Nauru. Like OpinionX, the Dowdall Method’s story starts in Ireland — it was created by Irishman Desmond Dowdall, Nauru’s Secretary for Justice in the 1970s (🇮🇪!).
When a participant submits their ranked list, each opinion is scored 100/rank. For example, 4th would get 100/4 = 25 points. If a participant excludes an opinion from their ranked list, the opinion gets 0 points. The overall score for each opinion is average(sum of all scores).
Dowdall’s Advantages:
Exclusions — It deals with excluded opinions in a logical manner.
Flexible — It allows survey creators to add new opinions mid-survey and for its final score to be comparable with the original opinions.
Variable — It handles submissions of different sizes extremely well. If Participant1 ranks 10 opinions while Participant2 submits only 6 ranked opinions, both participants have a pretty similar impact on the overall results.
Win Rate: The new scoring method for Pair Rank questions
Pair Rank is what we call pairwise comparison surveys on OpinionX. It involves breaking a list of opinions down into a series of head-to-head votes which are used to compare and score the overall set.
Instead of using a probabilistic model like Glicko-2, the new Win Rate Method simply takes for each opinion its number of wins and divides by its total number of pair comparisons. The resulting score on a range from 0-100 is the opinion’s percentage win rate.
Example: Opinion1 appears in 100 pairwise comparisons, where it wins 60 pairs, loses 35, and 5 pairs were skipped by participants. It’s score would be 60/95 = 63.16. How to interpret this → if Opinion1 is paired against any randomly selected opinion from the same survey, it has a 63% chance of winning.
Win Rate’s Advantages:
Scale — This scoring system works for Pair Rank surveys regardless of their size, whether you’ve got anywhere from 5 participants up to 5,000 or anything from 5 opinions or 500 ranking options.
Simple — No need for “robustness” or a confusing ‘opinions : participants’ ratio. Your Pair Rank results are correct from Participant1 onwards and every participant has an equal impact on the final results.
Intuitive — You don’t need a background in data science to understand what the score means, making it much easier to explain the methodology to your teammates.
Comparable — As the most complex ranking method on OpinionX, the Win Rate Method puts the scoring output for Pair Rank blocks on the same 0-100 range as all our other ranking blocks, making it much easier to compare results across different ranking formats.
Top Tip! Don’t forget that all users (free tier included) can now set a custom number of pair votes for each participant. If you don’t want the default 10 votes, go to survey setup tab > click on the pair rank block > go to the ( ⁝ ) icon bottom-right > hit the ‘Custom number of pair votes’ checkbox.
What this means for your current OpinionX surveys
Surveys created before this update went live on April 12th will continue to use the old Glicko-2 scoring system. Any survey created after the update went live, or any new Pair Rank or Order Rank block added to an old survey, will use the new Dowdall and Win Rate formulas instead. There is no way to get the old Glicko-based system on new surveys created from now onwards.
Participants will not notice anything different as a result of this update — their experience of your surveys will be the exact same on old and new scoring systems. This update applies to all users across the free and paid tiers of OpinionX.
One final note — if you duplicate a survey created before this update went live on April 12th, your duplicated survey will retain the old ranking algorithms from the original survey. To access the new algorithms, you must create a new survey.
Try out the new ranking types by creating a free stack ranking survey on OpinionX today. We are extremely keen to hear people’s opinions on the new format, so if you have something to share, send us a message over our support chat or email hello@opinionx.co.