Mass Intelligence

May 25, 2004 2:45 PM

The wisdom of crowds and the miracle of aggregation, arguably, are the reasons why markets and democracy work as well as they do. As New Yorker James Surowiecki explains in his new book, "consider the show Who Wants to Be a Millionaire. When a contestant on the show is stumped by a question, he has a couple of choices in asking for help: the audience or someone he's designated as an expert. The experts do a reasonable job: They get the answer right 65% of the time. But the audience is close to perfect: It gets the answer right 91% of the time, even though it's made up of people who have nothing better to do than sit in a TV studio and watch Regis Philbin." The new, new tipping point?
posted by kliuless (24 comments total)

From your first link:

For the crowd to be intelligent, the people within it have to be making decisions on their own, while drawing on diverse sources of information. During a bubble or panic, people's decisions become dependent on others'--"If they're selling, I better sell, too"--and diversity vanishes as people get caught up in the prevailing frenzy.

From your second link:

Plenty of questions remain about this theory, such as: Why assume that ignorance leads to random decision-making? Maybe elites, demagogues or the press can sway uninformed voters in one direction, thereby drowning out the votes of the informed minority.

This is good stuff (thanks kliuless!) but I'm of the mind that the caveats open holes so large that they overwhelm the value of the original speculations about mass wisdom.
posted by vacapinta at 2:54 PM on May 25, 2004

haven't read it, but fwiw, here're some more neat articles (i thought :) on information markets!

a couple by michael mauboussin, a CSFB strategist - the invisible lead steer & the janitor's dream

an HP paper - predicting the future

a few articles on PAM from way back when - decisions, decisions, what weird futures can you buy? & betting on terror

some suggestions on how to harness group intelligence - team players, information economies, the dynamics of learning & assassination politics

posted by kliuless at 2:55 PM on May 25, 2004

Based on my viewing of Who Wants to Be a Millionaire, that's a slightly unfair comparison. Contestants are more likely to use their "Ask the Audience" lifeline earlier in the game--on easier questions--than they use their "Phone-a-Friend" lifeline. This may be controlled for in the book (I haven't read it), but that's something to think about in terms of this data point.
posted by skynxnex at 2:57 PM on May 25, 2004

Really interesting stuff, kliuless. Surowiecki may discuss this in his book, but I don't think it's quite right to say that a crowd is smarter than an expert.

An expert has a single mind. There may be many competing ideas within his mind, but there's some kind of higher-up filtering method that chooses one of them to articulate or act upon.

A crowd is a huge number of minds, all with different ideas. So to make a crowd act like a mind, you need the crowd PLUS a filter.

On "Millionaire," the crowd ISN'T always right. The crowd PLUS a filter that shows what the majority of people in the crowd think PLUS a contestant choosing to go with the majority is always right.

The tricky thing is designing the filter. Is majority always best? What if you're trying to come up with a new-novel idea rather than a fact? What is the filtering mechanism for this.

I know that many cognitive scientists feel that we basically have a crowd in our heads -- a bunch of competing groups of neurons, all coming up with different ideas. The key here is to also figure out what mechanism decides who wins.
posted by grumblebee at 2:59 PM on May 25, 2004

The other examples in the article are good, but the Millionaire Ask the Audience one is flawed, because contestants usually ask the Audience general knowledge questions (i.e. the Easy ones).
posted by Stan Chin at 3:02 PM on May 25, 2004

Who was it that said If you want to see why democracy doens't work very well, just go out and talk to people who vote.
posted by Postroad at 3:32 PM on May 25, 2004

I think the Forbes article makes an interesting point -- that Google is essentially doing this with their pagerank algorithm. But, they are adding an additional step, they are weighting each vote. Definitely an interesting idea, almost seems like something that ought to be self-evident. I agree with the remarks on the millionaire point, my experience also indicates the Audience gets the easier, earlier questions.
posted by nads at 3:35 PM on May 25, 2004

There's a rather wide gulf between answering a single question on WWTBAM and selecting a political candidate to vote for.

That said, this is a good post, kliuless.

And I should know - I'm right approximately 91% of the time.
posted by troutfishing at 3:36 PM on May 25, 2004

This is - by the way - why liberals are more often right.

They usually are of many minds on any given subject.

Voters, however, cleave to the candidate who displays the greatest conviction of being right.

Hence, Iraq.
posted by troutfishing at 3:38 PM on May 25, 2004

Yes, that old conviction/confidence thing - as they lead you the your bright and certain future ... over a cliff (standing aside at the last moment to let you pass).
posted by Blue Stone at 4:24 PM on May 25, 2004

This is - by the way - why liberals are more often right. They usually are of many minds on any given subject.


Huh? How can they be both "of many minds" and "right"?

Or are you joking?
posted by grumblebee at 5:11 PM on May 25, 2004

well, from the book excerpt, surowiecki admits, "the results of 'who wants to be a millionaire?' would never stand up to scientific scrutiny," so i guess as an exemplum it's busted (does not hold up, bad example :) but! nevertheless! he thinks...

There are two lessons to draw from these experiments. First, in most of them the members of the group were not talking to each other or working on a problem together. They were making individual guesses, which were aggregated and then averaged. This is exactly what Galton did, and it is likely to produce excellent results. (In a later chapter, we'll see how having members interact changes things, sometimes for the better, sometimes for the worse.)

Second, the group's guess will not be better than that of every single person in the group each time. In many (perhaps most) cases, there will be a few people who do better than the group. This is, in some sense, a good thing, since especially in situations where there is an incentive for doing well (like, say, the stock market) it gives people reason to keep participating. But there is no evidence in these studies that certain people consistently outperform the group.

[/em added :] to be cont'd...

also, btw, in the back of my mind, something i just remembered that this kinda reminded me of is a neat piece by SDB (i know some people regard him as persona non grata 'round these parts, himself included :) but it's really neat!

like his speculation on a noospheric hive mind:

With speech, the collaborative process of creation of knowledge expanded from the person to the tribe. With writing, it spread to the level of citystates. With printing it encompassed nations and even continents. With computer networking, everyone in the world is involved whether they like it or not. There's nowhere left to hide.

i.e. advances in telecommunication (ability to aggregate information) allows greater social cohesion and more complex and larger group affinities, such that people not only behave like particle-agents, but become like a BEC, and a schizophrenic decohering one at that, i find interesting (against my better judgement; i'd be wary of such analogies :) also i think presenting it as some how inevitable is a deterministic fallacy, altho i agree with it as a condition!
posted by kliuless at 6:02 PM on May 25, 2004

Huh? How can they be both "of many minds" and "right"?

That is obviously something your feeble conservative mind cannot comprehend!

/elitist liberal joke
posted by inpHilltr8r at 7:02 PM on May 25, 2004

Metafilter: lots of dumb people outsmarting the experts.
posted by alms at 7:55 PM on May 25, 2004

Count me still as a skeptic. The oft-cited jellybean example seems to be just the law of averages at work: Most of us have *some* sense of how many jellybeans are in a jar. We can tell its not 1 or 10 and we can tell its not a billion. With a little intuition we can hone in on a guess within a "reasonable" range. If a hundred people all do this and we average it out, its not surprising to me that the average guess will be pretty damn close (for everybody that overshot someone undershot)

The democracy link in the FPP also seems to contradict the main argument. What it says is that given an already established preference order among a population, the choice of *voting system* can then affect the outcome. And, among other paradoxes, the winner may be someone that is not favored by any majority. I don't see how this supports the intelligent crowds argument.
posted by vacapinta at 1:01 AM on May 26, 2004

Hmm... are the 'phone-a-friend' people actually experts on the questions they're asked to answer? It seems like there's some smoke-and-mirrors obfuscation going on here between the 'individual' and the 'expert'. I'm pretty sure that if contestants could ring up an historian when they got a history question, a mathematician when they got a maths question, or a doctor when they got a medical question, the experts would roundly thrash the group's guesses.
So there.
posted by RokkitNite at 5:30 AM on May 26, 2004

MetaFilter: the crowd PLUS a filter.
posted by AwkwardPause at 5:52 AM on May 26, 2004

It gets the answer right 91% of the time

As others have said, only because of the contestant's pre-selection. Judging by the UK show, it's not that the crowd is asked "easy" questions as such. It's asked questions in popular culture territory, such as names of forgettable sporting and pop stars.
posted by raygirvan at 6:28 AM on May 26, 2004

"Men, it has been well said, think in herds; it will be seen that they go mad in herds, while they only recover their senses slowly, and one by one."
posted by dodgygeezer at 11:14 AM on May 26, 2004

The oft-cited jellybean example ... With a little intuition we can hone in on a guess within a "reasonable" range

However, I'm skeptical about this too. Like the Ask the Audience question, this seems to be a problem pre-selected to guide people toward a reasonable answer; the number of beans is limited to a comfortably intuitive range. To the contrary, the many examples of innumeracy cited by John Allen Paulos suggest that people in general are extremely bad at numerical assessment, especially when it gets into major orders-of-magnitude such as distinguishing between millions and billions.
posted by raygirvan at 12:13 PM on May 26, 2004

i don't think surowiecki would disagree that people use heuristics or that the "wisdom of crowds" is just a fancy way to repackage the law of averages; i actually think that's his point! (not having read it :)

that what we know about the world is only an approximation of reality and depending on what filters are used (to grumblebee's point) and how you aggregate it -- as SDB sez, "the collaborative process of creation of knowledge" -- the 'group' oftentimes may exhibit (dare-i-say emergent) properties that are 'smarter' than what any individual can muster over a period of time.

and, it occurs to me (extemporaneously :) -- not speaking for surowiecki! -- that that group could well be considered one's own mind, hence making one an 'expert', and then like a group of experts one might call a university or something, and so on and so on, on up.

anyway, in finance, who i think's the crowd surowiecki is writing for, the notion of price discovery follows the efficient market hypothesis (which has strong, semi-strong, and weak formations) that all hinge upon information; that it's 'all' reflected in the price! (nevermind differential pricing mechanisms :)

this 'all' is disputed, hence the different versions, but that in itself is sorta fascinating because it gets kinda philosophical, like when is something actually 'known' or what really constitutes an 'exchange of information', stuff like that. (oh and like, can counterfactuals count? to what extent is proleptic knowledge (or subjunctive thinking) valid?)

practically, "elites, demagogues or the press can sway" market participants, indeed do, and, moreoever, they influence each other, all the time; everyone reads what others are doing (and reading) and is trying to figure out/model/magick what others are thinking (and gonna do), and it can get 'meta' (or arcane) fairly quickly -- if ever a panoptical infinite regress there was, financial markets should qualify.

prices, reflecting 'all' available information including the relative pricing/'shadow prices'/convertibility against all other things, can then change chaotically, unpredictably and discontinuously. with all the variables and feedback and stuff it's a very tough (partial-)differential calculus, if it can even be described mathematically at all; it may take quantum computation and/or the multiverse :D

literally, neal stephenson describes the process thusly in the confusion!

The coin had been passing from hand to hand and purse to purse for more than a hundred years, and probably had more tales to tell than a ship full of Irish sailors—yet it was just a single mote in the dust-pile that was the English money supply. In a certain way to take that dust and shovel it into the maw of the crucibles was monstrous, like burning a library.

the notional amount traded everyday, recycled, in the crucible of electronic currency markets approaches $2 trillion (that's the world's annual economic output traded every three and half weeks). if you count the number of mutual funds, closed-end funds and hedge funds (and funds of funds), there are more than there are securites that make them up. to what purpose does all this trading, swapping and repackaging suit? it all seems rather bizarre and, by all accounts, none of it should work, yet, for the most part, they usually function pretty well; markets and the institutions that support them, supposedly, are able to keep track of and smoothly allocate it all.

cosma shalizi, in that 'dynamics of learning' link above, has a really neat (proto-)explanation why:

The Dynamics of Learning project takes computational mechanics in a new direction. Anything people are willing to call an agent has inputs and outputs. In organisms, the inputs are all the senses, and the outputs all the motions of the animal. In machines (say, a mobile robot), the inputs would come from sensors (e.g., cameras, heat detectors, wireless links) and the outputs would go to "effectors\" (e.g., motors in wheels and wireless links). Some mechanism connects them, making an agent into what computer science calls a transducer or a channel with memory. Computational mechanics now has the tools to discover the patterns of intrinsic computation going on in a transducer, including the way it changes its own organization in response to inputs. These tools work even when the transducer is a "learning channel\" and works by building a model of its input—we can do pattern discovery on pattern discoverers!

Using these methods, Crutchfield and his former student Dave Feldman have already calculated how much internal complexity an agent must have in order to adequately model its environment. Excessively simple agents can't grasp all the structure in the environment, and therefore see it as more random than it really is—and the amount of excess randomness depends on the mismatch between the agent's cognitive complexity and the environment's structural complexity.

The next stage of the project will go beyond single-agent learning, to learning and adaptation in multi-agent systems. A collective of agents is like a network of interconnected transducers. Computational mechanics can show how the local behavior of the agents builds up into the global behavior of the network, and can identify the intrinsic computation the collective performs. Given that, the group will begin seeing when the collective can do things that individuals cannot—how an adaptation can be distributed, just as a computation can be. The ultimate goal of the project is to understand collective cognition, the way groups can sometimes solve problems and learn things better than any of their members could. It's only fitting that what amounts to a quantitative sociology of science be initiated at SFI.

[/em added :] kinda like a sociological takeoff from chaitin's work on algorithmic information theory/program-size complexity.

The democracy link in the FPP also seems to contradict the main argument.

yeah, i put that in there for balance :D cuz, of course, like any decent hypothesis should be arguable! and that brings up your point that any voting procedure can be gamed and, like a three-body problem, impossible to predict.
posted by kliuless at 4:47 PM on May 26, 2004

The wisdom of crowds can be seen at the racetrack, where the odds on horses coincide very nicely with their probability of winning. (That is, if you look at a large collection of horses that went off at 4-1 odds, you find that 20% won.)

Am I the only one that considers an absolute difference of 5% to be a pretty big differential? I mean, 4-1 odds is a full 25% higher than a 20% win-potential. Perhaps its due to the inherent variability in the data, but the article should really mention it. I guess you can't expect that kind of attention to detail from a rich kid rag like forbes, though.
posted by toothless joe at 12:53 AM on May 27, 2004

(Fascinating links and discussion; thanks for initiating, kliuless)
posted by carter at 2:19 PM on May 27, 2004

hey (just as a followup addendum :) i was just reading that world 'GDP' is $36 trillion -- the ~$50 trillion amount is based on PPP -- so that would allow the 'velocity' of money to churn through/'turn over' global output every two and half weeks or so!

which reminds me of another measurement problem... oh and a fascinating look at "what are numbers, really? a cerebral basis for number sense" [via interconnected's links miniblog - who, btw, has a hunch that:

[S]ynaesthesia is the pre-dawn of a higher-level linguistic/symbolic sense becoming hardwired in much the same way. Transhumans will be mentally superior in so many ways. Meaning will exhibit pop-out. Objects will be declared by semantics, not by early visual processing edge detection.

(mine is that our brains are quantum computers linked thru the multiverse, and like SDB's supposition, we're just the substrate for higher order intelligences we cannot fathom -- kinda like understanding higher dimensions, the 'neuron' cannot know the mind! or can it? :) and some interesting thoughts on comparative species' cognition besides! :]
posted by kliuless at 1:09 PM on May 28, 2004