An initial thanks for the many excellent comments and emails, which I’m trying to absorb. We’ve been discussing several methods for aggregating views: markets a la Hayek, group deliberation, and wikis (with a brief mention of open source software). One emphasis has been on problems with group deliberation, because like-minded people usually end up thinking a more extreme version of what they thought before.
In his fun and illuminating book, The Wisdom of Crowds, James Surowiecki emphasizes another method of aggregating opinions: ask a lot of people and take the average answer. In many cases, this method seems to work magically well. If you put a bunch of jelly beans in a jar, and ask 200 people how many beans are in the jar, the average answer is likely to be eerily good. Often the average answer of a large group is right on the mark.
Surowiecki doesn’t explain why this happens, but the answer lies in the Condorcet Jury Theorem. If you have a group of people, and if each person is more than 50% likely to be right, the likelihood that the average answer will be right approaches 100% as the size of the group increases. (The math here is so simple that even we lawyers can almost understand it. For nonbinary choices with plurality voting, the math isn’t so simple, and this lawyer can’t even almost understand it, but there’s a result that explains why Condorcet’s basic insight applies there too.) Condorcet’s result has implications for many practices; it hasn’t been adequately exploited by people in business, law, and politics.
Here’s a problem, though. If group members are less than 50% likely to be right, the likelihood that the average will be right approaches ZERO as the size of the group increases. (I asked members of the faculty at the University of Chicago Law School to estimate the weight of the horse who won the Kentucky Derby, the number of lines in Antigone, and the number of Supreme Court invalidations of state and federal law. The group average did really well with the first question, pretty badly with the second, and horrendously with the third!) Condorcet was well aware of this point, and hence he emphasized that we can’t rely on the wisdom of group averages when most group members are likely to be biased or wrong.
Are group averages likely to be worse than what emerges from group deliberation? The answer is mixed. Sometimes deliberation does help to correct errors (especially when people are considering a “eureka” problem, where the answer is clearly right once identified). But sometimes deliberating groups do little better, and sometimes even worse, than predeliberation averages.
Are markets likely to do better than group averages? The simplest answer is yes, because participants have strong incentives to be right, and won’t participate unless they think they have something to gain.
Cass states: Here’s a problem, though. If group members are less than 50% likely to be right, the likelihood that the average will be right approaches ZERO as the size of the group increases.
Actually, this isn’t a good way to describe the jelly bean scenario, though it may be applicable to other types of decision making.
The correct way to statistically describe the jelly bean scenario is to say that each person has an uncertainty “S” in their estimate of the number of jelly beans. If they don’t talk to each other (and thereby influence other people’s decisions), then assuming no optical illusions or other effects that could generate a systematic bias, the uncertainty of the average of the guesses of everybody in the crowd is S/sqrt(N), where N is the number of persons making independent guesses and sqrt(N) means the square root of N.
If their ability to estimate gets worse, S gets larger, and you just need more people (large N) to get an exact count of the number of jelly beans. Or put another way, if your population size is kept constant, as the individuals ability to estimate gets worse, then so does their corresponding answer.
There are other types of decision making, e.g., categorical choices like “is this color chartreuse or a lime green”? If enough subjects don’t know the difference between the two colors, you can make N as large as you want and still never get the correct answer.
There is another example of collective intelligence…. social insects such as ants. These insects as a group show some pretty cool abilities, such as “remembering” where different types os food are, regulating how much brood goes to the the workers versus the alates, etc.
This all ties in, by the way, to the importance of an informed populace. As the data coming from the media gets worse, the ability of the democracy to make appropriate choices becomes more and more impaired. The emergence of a blogosphere which is divided into a left and right hemisphere, with very different social characteristics, also suggests that there will be an asymmetric advantage to the hemisphere which is able to arrive at truthful and accurate conclusions. The hemisphere that spends most of its time spinning, on the other hand, may actually further screw up the functioning of its associated party.
I haven’t looked carefully to see if I’m not repeating something already mentioned, but google is an aggregator of aggregators, complete with feedback loops.
In some ways it’s like a stock market measuring prices, except that the measure is based on link weighting rather than money.
Creation of new links is also biased by this measurement: for example, I’ll often search for new links for my Critiques Of Libertarianism web site with google, and those that come up first are more likely to be evaluated and eventually linked. So there are positive feedback loops involved.
Carrick may have pointed out the fly in the ointment (better mathematical minds than mine will have to weigh in on that), but the theorem as stated by Cass suggests a built-in way to measure how good the average guess is: simply take two groups, one significantly larger than the other, and compare their answers. If the larger group has the better answer, the average agent is more than 50% likely to be right.
This kind of information is useful in playing Who Wants To Be A Millionaire. You can ask for a poll of the audience in their multiple-choice questions. If you are completely clueless, but know it to be a common category and relatively easy question, the winner in the poll is almost invariably right. If you have some knowledge in a trickier category, but are from from sure of the right answer, reading the poll requires interpretation: A clear winner is less likely, and you might see two choices equally popular–in this case, you must see if you can figure the “trick” answer and discount those who are fooled by it.
Similarily like in Who Wants to be a Millionaire you can bias the audience into picking the completely wrong answer. If one were to announce to the audience that they have some knoweldge on the question, and that they go as far as to label the correct answer as being incorrect, then the audience is more than likely to vote for the wrong answers. Case in point, there was a question that asked which toy translates to english as “play well” from its native language. The contestant said “It couldn’t be Lego.” So when he polled the audience everyone chose the the wrong answers, and a very small percentage of individuals still voted for Lego.
So the minority that still voted for Lego can either fit into two categories:
A.) They knew the answer.
B.) They didn’t know it was the answer, but they voted for Lego anyway because they couldn’t trust the contestant said.
It is obvious though, that any question can be loaded to provide a majority consent, even if it were a false one. It’s just part of the rhetorical schemantics in our language.