The wisdom of crowds is a book by James Surowiecki which describes how the masses can come up with better solutions to problems than experts. It also became part of Tim O’Reilly’s seminal paper describing the nature of web 2.0.

If it were merely an amplifier, blogging would be uninteresting. But like Wikipedia, blogging harnesses collective intelligence as a kind of filter. What James Suriowecki calls “the wisdom of crowds” comes into play, and much as PageRank produces better results than analysis of any individual document, the collective attention of the blogosphere selects for value.

To summarise what O’Reilly is saying, he feels that the most popular blog postings, or the ones that achieve the best search engine ranking, are also the best ones. Google works by analysing the number of inbound links to a site, using this as a determination of quality. Since (in the best possible world) all of those links were created by individual human beings acting intelligently, the Google algorithm is using the sum of all of our individual judgements to decide how useful a page is.

There are several ways to spam Google, but they do seem to get found out eventually. It’s also questionable how discriminating the majority is. Linkage is not necessarily a foolproof determiner of quality. An article by David Beckham on the existence of God would probably gain a higher rank than one by A J Ayers. In addition, it’s hard for new sites to ‘break in’ to Google. Nobody has links to your site because it didn’t exist before, and you’ll have to compete against the thousands of links to your competitors’ sites before you’ll be able to get a look in.

Is this better than having a human editor? The question doesn’t really arise since there are billions of pages. There are 47 million blogs indexed on Technorati alone, with 75,000 new blogs started every day. The only practical way to organise the information is by machine. Since machines can’t (yet) decide how clever or funny or moving a page is, then involving some human judgement, the number of inbound links, seems to be the best kludge we have available. Google is the number one search engine because people think it produces the best results. If it didn’t, people would go elsewhere. Nobody, not even Google, is saying that their algorithm is perfect, but it is quite clearly ‘good enough for now’, harnessing the wisdom of crowds to produce better search results than human beings alone could possibly produce, no matter how qualified or dedicated.

I’d also like to briefly point out the example of ebay. Getting a high reputation within the e-bay community is such a motivating factor that it drives really high levels of service. Yes, there have been a number of bad experiences there and people who abuse the system, but on the whole, the success of e-bay could not have happened without a community that cared enough to make it successful. It’s very hard to quantify the overall level of service satisfaction on ebay, but it could not continue to exist if it wasn’t ‘high enough’. Again, this is the product of a crowd of individuals making their own decisions and thinking: “If I deliver a high level of service (a) I will make more sales and (b) more people will trust shopping at e-bay. I win from both of those things.” Their collective intelligence has made e-bay a good place to buy things. Like Google, it is not perfect, but it certainly seems ‘good enough’ to fund an enormously successful internet company.

The wisdom of crowds does exist on the internet. These are two examples. The problem comes when the conditions are not correct for that wisdom to happen. For mobs to be smart, they have to all operate individually using only their own agenda and they have to care. Surowiecki gives the example of a lost submarine. It had been several hours since the vessel had lost contact and the possible search area was impossibly massive. The man in charge of the rescue operation, John Craven, solicited opinions from all sorts of people as to where the sub might be: naval officers, salvage experts and mathematicians. Then he drew a map and pinpointed all the suggestions. Then he calculated the median of all their answers — not where the majority thought it would be, but the average. Lo and behold, that’s where the sub lay. The important things to note are that this was a group of people making independent judgements based on their own experience, agenda and training. That’s how a crowd is able to make a wise communal decision.

When internet sites operate on majority rule and the community that runs it is relatively small and it’s very easy to make your impression felt — such as news voting sites, like digg and reddit, or some of the social bookmarking sites — then the results become less accurate. Wisdom loses ground to lobbyists, unintelligent clicking at buzzwords and sheep-like approval of anything other people have said was important. Visitors to such sites see the stories or sites that have already been found popular by previous visitors and this creates a vicious circle whereby only stories reflecting the hard-core of users — the sort of people who spend hours of their day on the sites — ever get promoted near to the top of the list. A new model needs to be found for these sites, in my opinion, that is able to operate by aggregating a host of genuinely individual decisions — the true wisdom of crowds.