Did you knowNeuralis taking the stage this fall? Together with an amazing line-up of experts, we will explore the future of AI during TNW Conference 2021.Secure your online ticket now!
A 2002National Geographic surveydemonstrated that only about 17% of adults in the US could point to Afghanistan on a map. And this was less than a year removed from the 9/11 terrorist attacks, so you can be certain the Middle East was splayed across 24-hours news networksad infinitum.
You’d be forgiven, then, if you didn’t think highly of the so-called “wisdom of crowds.”
But there’s a new AI-human hybrid forecasting paradigm that’s showing incredible improvements over the old “majority rules” method of crowd-sourcing predictions.
Calling all Scaleup founders! Join the Soonicorn Summit on November 28 in Amsterdam.
Meet with the leaders of Picnic, Miro, Carbon Equity and more during this exclusive event dedicated to Scaleup Founders!
Up front:Remember when the New York Times and everyone else predictedabout an 85% chancethat Hillary Clinton would win the 2016 US presidential election?
Well, that’s because everybody and their mother said they were going to vote for Hillary Clinton. Nobody took the scattered millions here and there who said they weren’t very seriously.
When people were asked who they thoughtwas going to win the election, almost all of them answered: Hillary Clinton.
We should have been asking them to give their opinion on how other voters were going to answer that question.
Background:Researchers at Princeton University came up with a paradigm called the “surprisingly popular answer” that apparently outperforms other crowd wisdom models.
It works by asking an objective question such as, is Philadelphia the capital of Pennsylvania? And then you also ask, will most other people answer yes or no?
People familiar with US geography will know that Harrisburg is the capital, so the correct answer is “no.”
And people from Pennsylvania (as well as students of human nature) will know that a lot of people who aren’t from the state are going to get that one wrong.
The outcome of the survey will, according to the paradigm, clearly show that the correct answer is asurprisingly popularanswer.
The number of people who know the correct answer but also assume others won’t, coupled with the number of people who answer incorrectly while assuming others will answer as they did, makes the actual number of correct “no” answers surprising – they don’t follow what the crowdexpected.
This might not sound ground-breaking, but when researchers applied math models to algorithms using thesurprisingly popularanswer paradigm, they found a marked increase in accuracy.
Pera 2017 press release:
Across all topics, the researchers found that the “surprisingly popular” algorithm reduced errors by 21.3 percent compared to simple majority votes, and by 24.2 percent compared to basic confidence-weighted votes (where people express how confident they are in their answers).
It also reduced errors by 22.2 percent compared to answers with the highest average confidence levels. On the 50 test questions related to state capitals—such as the Harrisburg-Philadelphia question—the SP method reduced incorrect decisions by 48 percent compared to the majority vote.
What’s new:A lot’s changed since 2017 in the AI world and the surprisingly popular answer paradigm has gotten a massive upgrade.
According toa new press release, this one from Pennsylvania State University, researchers have figured out how to useSPon ranked lists.
This means it’s as efficacious at finding ground-truth rankings, in order, as it is at predicting the answer to a yes/no question.
Per team lead Hadi Hassani:
One does not require to fully elicit complete rankings and predictions of others. We are able to recover the ground truth by combining both the vote and predictions without eliciting full distributions over all possible n! rankings. And this is true for recovering either only the top choice or the full ranking.
Quick take:Crowds can be stupid, but trying to figure out which individuals are and aren’t subject-matter experts or trustworthy representatives of their population can be as political an exercise as running for office.
We need a method by which people in a crowd who know what they’re talking about can be heard louder than those who don’t – but one that doesn’t rely onidentityto discern who deserves consideration.
Behold the power of math! By sorting out who’s right based on predictions aboutwho’s really right about who’s right, we end up in a weird place where the numbers make more sense than just trying to pick a winner.
You can read the Pennsylvania team’s whole research paperhere.
Story byTristan Greene
Tristan is a futurist covering human-centric artificial intelligence advances, quantum computing, STEM, physics, and space stuff. Pronouns:(show all)Tristan is a futurist covering human-centric artificial intelligence advances, quantum computing, STEM, physics, and space stuff. Pronouns: He/him
Get the TNW newsletter
Get the most important tech news in your inbox each week.
