Then Shall I Know Even as Also I am Known

By Ian, on December 31st, 2006

I wrote yesterday that the emerging news about Search Wikia seemed a tad confusing. There’s good reason for that. In an interview between Danny Sullivan and Jimmy Wales, the Wikimedia foundation boss says that he didn’t entirely intend the project to be made public yet, since it hasn’t really been started. Wales told a reporter on the Times that he had an idea for search, and suddenly it became big news.

As far as I can see, the idea is a web search engine in which the answers and even the algorithm are tweaked by the ‘crowd’. Though similar in some respects to the Open Directory, everyone would have some sort of a say, in a similar way to Wikipedia.

Q. How do you envision the community participating? Will they be selecting sites? Will this leverage material in Wikipedia? Will they rate sites?
If I had to speculate about it, I would say it’s several of those things, not just community involved with rating URLs but also community rating for whole web sites, what to include or not to include and also the whole algorithm … That’s a human type process that we can empower people to guide the spider.

The problem with search as it is will be familiar to anyone who has searched for a commercial service. If you search for London Hotels or want to buy a wii, then there’s no way to know which sites to trust. The top results seem to be the best optimised for Google, not the best ones for you to buy from. Wales believes transparency in the reasons for a site’s position in the listings is the only way to create trust in search results.

Exactly how this will happen is still very much open to debate. It’s unclear, for example, how this will differ from the human-generated searches using collective intelligence on services like del.icio.us and wink. I’m also interested in how the site might go about counteracting spam — businesses and their agents, perhaps automated, recommending their own sites. However, Wales suggests that there aren’t nearly so many spammers as you might think, and identifying and banning them is not inconceivable. The success of the human-powered comment spam service Akismet gives me hope that this is possible.

One other reason they might succeed: Search Wikia has already raised $4mn in angel funding. Money isn’t a cure for bad ideas, but it gives you time to make a few mistakes finding out how to make the thing work.