Life just got better. At the end of last week, Google announced that its personalised search had now become available to ‘signed-out’ users.

What does that mean?

Well, personalised search means that Google uses its history of what you have searched for before to provide more relevant results for subsequent search queries. It records everything you’ve searched for and every result you’ve clicked. This allows it to profile you and produce results that are more likely to be about what you’re interested in. If you live in Birmingham, UK, for example, and often click on results for places in that city, then you’ll be less likely to get results relating to Birmingham, Alabama.

Signed-out users are people who don’t log into a Google account prior to conducting a search. That would include people who haven’t opted in to have their search results saved. This is done through a cookie file saved on your computer. Unless your Internet privacy settings are set very high, this will happen without you noticing.

So, whoever you are, your search history is saved and analysed. Without your permission.

In a similar vein, the rollout of real-time search means that Twitter comments are instantly catalogued. And don’t worry – you don’t need to change any account settings or opt-in to anything. They’re doing it anyway. There really is no ‘undo’ button on the web.

Any lily-livered liberals clinging to outmoded ideas like a right to privacy need to move on. As Google CEO Eric Schmidt told CNBC last week:

If you have something that you don’t want anyone to know, maybe you shouldn’t be doing it in the first place.

Well, maybe I shouldn’t. Or maybe I just don’t think it’s any of your business. Or that you should ask me first.

Oh wait – you are allowed to delete your history and opt out. But you’ll need to explicitly opt-out of survei personalisation on every computer you use.

I really wish Bing produced better results.

Postscript: I notice Alan beat me to the punch on this and is typically incisive.

I understand that Microsoft has to do something to build on its web strategy/presence. No-one uses Live Search, Live Spaces, or any of the rest. (OK. About one percent of people do). To build up any future trade for advertising, web services or development platforms, they have to increase market share.

I understand that Yahoo! has to do something. Their share of the search market is pitiful compared to the almighty Google. Their share of the search marketing budget is about 20% compared to Google’s 70%. And they’d just been forced to lay off a load of staff.

So if they combine forces, they end up with a market competitor?

I don’t think so.

Microsoft’s problem and Yahoo!‘s has been that they have not been able to identify what they do well. Microsoft used to do operating systems and business productivity software. They were quite good at that. YMMV.

Yahoo! used to have this great directory of editor-approved, quality websites. Then they diversified. They tried to make yahoo.com all things to all wo/men. That failed disastrously because there’s no such thing. They brought on some cool people and acquired a load of cool sites like del.icio.us, flickr and upcoming. But still it didn’t work for them because advertisers don’t buy cool; they buy results. Yahoo! announced 1400 job losses just last week.

Why didn’t it work and why isn’t MS able to make any inroads on the web?

Because neither of them have a core value proposition when it comes to the web. You couldn’t sum up what either of them do on the web in one sentence. If a business can’t do that, then they are in trouble, normally.

Don’t get me wrong. There are bits within both companies’ web presence that have considerable value. Flickr is a cool photo site. Microsoft’s technet is actually very good, IMHO. Live Spaces is arguably a much better platform than Blogger or Vox.

However, for end-users, if you want good search, go to Google. For businesses, if you want SEM, go to Google. What exactly would you willingly go to a Yahoo or MS website for?

Microo! doesn’t appear to me to provide a compelling alternative to any of that.

Well, you’d probably try to work out how Google works. You want to come top of the search results for things like err.. ‘glue’. You’d probably also want to come at the top or near the top for things like ‘DIY products’ and ‘craft supplies’. You can’t just buy your way to the top – well, you can, but it would be better to be returning top positions in the organic results as well as sponsored positions.

How would you do that? Well, you’d go about making sure that the Bostik site was the best site on the web when it comes to glue. You’d have sections on the history of glue, glue tips and tricks, glue industry news, glue formulae, learned articles on the future of adhesives. It’ll take a little resource, but hey, you’re a brand – part of the Total group in this case – you aren’t short of a few bob. Plus, if you get it right, you’ll be saving a packet on advertising. Depending on your marketplace, you might not have to go overboard here – you just need more, better stuff than those bastards at Copydex.

You’d also get into this whole Web 2.0 thing. It’ll help you generate more content and get linked to. Pay someone to write the best glue blog on the planet for you. Get some message boards on the go about DIY, handicrafts and other glue issues. You’ll get a widget put together – maybe it gives you a DIY tip every day or something. You’ll provide RSS feeds for all your content so it can travel as far as possible.

Google loves all this stuff – pretty soon, you’ll be ranking for as many glue, DIY and handicrafting terms as you’d care to mention. And it’s all relatively easy.

Most brands aren’t currently doing this stuff. That’s because corporations are slow-moving and stupid, not to mention psychotic. Most brands behave like toddlers screaming for attention at the moment; the brands of the future will behave like best mates and learned counsellors – people you actually want to have a relationship with.

They will come round eventually – it’s common sense. If you are a brand you will sooner or later be working as hard as possible to either create or acquire the definitive site on the Internet when it comes to your subject matter.

Anything wrong with this? In some senses, it’s great. Google is rehabilitating corporations in some senses – forcing them to offer stuff that’s useful and interesting rather than the old raping and pillaging shenanigans they used to do.

Well, it’s great if you work in advertising or marketing (hey, kill yourself). Not so much if you are a citizen of the world. Brands are still psychotic underneath, you see. They are being rehabilitated in the same sense that a mass murderer growing flowers in the prison garden is rehabilitated. You know exactly what they’d really like to do with those shears. The problem is brands have been given the means to take control of the message once again, if only they had the sense to realise it.

They don’t want you to consider their competitors; they don’t want you thinking about buying nails instead of glue; they don’t want you to know about the Marvin Medium massacre of ‘37. And it will only take them a very little time and resource to achieve it. And remember, they’re brands – their resources and energy are pretty much limitless. Particularly compared to you, Mister competitor Glue Blogger. They’ll buy you up and shut you down in an instant. Here’s a real example – a search for Mattel Toy Recall takes you straight their consumer relations page as the top result. Potentially more vital information about the results of lead poisoning in young children appear half way down the page, where nobody clicks.

So the next Google. Not only will it be better at searching – we’re only impressed by the current Google because the competition is so absolutely dismal. It will also be about expressing diversity rather than hierarchy. About delivering the truth in all its facets rather than the definitive answer. It won’t produce a list; it will produce a crystal.

There’s little to add to Heather’s excellent post, which I just wanted to draw attention to. It’s a great example of social networks propelling someone from zero to hero. As she says, the search engine traffic didn’t really start until he’d become a phenomenon on MySpace and Bebo. Early on, these sites apparently counted for 64% of the upstream traffic to the Lil’ Chris site.

It is interesting how social network traffic (brown line) has tailed off as search traffic has increased (green line). The reasons for this will be familiar to anyone who went to school. Once a cult phenomenon goes mainstream then it seems it just isn’t cool for cats any more. Let’s hope that the recent dip in searches will have been curtailed by the release of his shot at Christmas No. 1, Gettin’ Enough last week. (YouTube Link. Chris seems to have forsaken Gene Simmons for Pete Shelley and/or Feargal Sharkey nowadays. Good move.)

I’m not sure whether the stats prove an uncomplicated victory for social search and crowdsourced problem-solving, for a start. I’ve really no idea which service produces better answers, being one issue. It probably depends on the question. ‘What’s a good Italian restaurant in Cardiff?’ will work well with the Yahoo! model because it has a wider reach. On the other hand, you might not want to trust folk wisdom for a solution to matters that require a specialised knowledge.

It does show that a free-for-all, give-and-take knowledge source is very addictive and, presumably, helpful enough. Involving people like Stephen Hawking and Oprah Winfrey bought Yahoo! a vital share of attention Google never bothered with. Also, as Brady Forrest points out, Yahoo!‘s model could scale organically, while Google’s required the recruitment and vetting of answerers, a time-consuming and distracting business.

Is this victory analagous to what will happen in the battle between the Wikipedia and the Britannica? It seems very similar on face value. Not entirely, though, since their business models are different: Wikipedia survives on charitable donations and drubbing the opposition when it comes to traffic is not nearly as helpful as it has been to Yahoo!

[I interviewed Steven Taylor, RVP of Yahoo! UK here, back in August and he talked a little about the Answers service]

They haven’t been Techcrunched or boingboing’ed as yet so the number of sites in the system is low, which limits the utility of the service. However, a search for ‘Search Engines’ has been pre-populated. It would be useful if there were a way to seed the system with results from another service. I also think real money would be an interesting addition. As it is, only the ongoing addition and betting on sites by sufficient users will reveal whether the model actually works. But I have to say.. it looks good on paper. Would I invest real money in Tall Street? No. But then I’m not an investor on the lookout for innovative new ideas.

Yes, it’s just another student project, and it looks a bit rough. But haven’t we seen something fitting that description before…?

Gary was also good enough to answer a few questions for me:

Could you explain briefly how tall street works.

Traders make investments on sites belonging under keywords. They get rewarded (their investments increase) if when people who search those keywords find their sites useful (based on if the link gets clicked and any ratings the link gets) and punished if not.

Can prediction markets work for search when there are so many ‘correct answers’?

This is a problem any search engine struggles with.

The ideal situation is:
The user has some idea about what they want, they express this in “search terms”, the search engine returns the best answer for what the user wanted.

A search engine works by:
Returning all the pages that contain that term (or had that term in the anchor text to that page). Then it ranks those pages based on some algorithm which takes into account the number of pages linking to it, (and linking to those pages) among other things.

Tall Street works by:
Returning all the pages that someone thought belonged under that search term, and the rankings are influenced by

1) how many people thought that site belonged under the term (if lots of people invest in a site it gets lots of investment then it gets a high ranking),

2) how much they thought the site should be ranked in comparison to other sites under that term (some unpopular terms might only have one investor in, and they could completely control the rankings for that search term, this is fine since if someone disagrees then they are able to invest more heavily and change the rankings),

3) how much influence (money) the investors investing in the sites under the term have over the directory i.e. their net worth (if that investor is good investing in popular sites then over time they will get more influence since they will accumulate more money)

Why is this better?

Rankings are up-to-date — the link structure of the web that search engines analyse to determine rankings isn’t dynamic — links remain long after a site moves. What could happen with Tall Street is if some site added a new cool feature that no other site under the keyword had, then investment in that site could increase and it’d end up at the top of the results straight away. You wouldn’t see something like this in the link structure until long after. ( i.e. The feature would have to be very cool that people decided to link to the site and then the search engine would have to recrawl the web, and then update the results)

Users can find sites by type and easily find similar sites — Sites are under keywords because people thought that the site belonged under the key words. With search engines you get lots of results many aren’t relevant. They just appear because they contain the term you are searching for on the page. We have a feature where you can drill down to find pages under keyword1 and keyword2, which lets you easily find similar sites e.g. http://www.tallstreet.com/view/PHP/tutorial/
Fair Ranking system — Everyone gets an input into the rankings. If you think a site doesn’t belong, you can mark it as spam. If you find a site that is better then what is showing as the best then you can sign up and invest in it. If you’re right and other people agree, then you’ll make more money and have more influence over the directory.

At a later stage, we could allow traders to group together and publish their portfolio. This means you could search for ‘what do New Zealanders think is the best news site’. Or ‘what do web developers think is the best reference site’. This isn’t something a search engine can do as easily.

Is this a web 2.0 product, and how so?

My aim in the project is to create a cool fun website that I think it’ll help make finding good sites easier. I guess if you follow the definitions on

http://twopointouch.com/2006/08/17/10-definitions-of-web-20-and-their-shortcomings/

It has to some degree:
The wisdom of crowds
Shared Web Applications
User Participation

But mainly because I think these elements will help produce the best result.

I try to allow the site to work even if you don’t have javascript (we do not rely on AJAX) and provide an AJAX way if I think it can help usability.

I would argue that you should narrow down a little on what you search for, to gain a little traction in a few limited ‘best of’s. What’s your answer?

My view is to provide some value to people as early as possible. You are right. Limiting would show exactly how the product works. But one way we give users value is to allow them to add their own site under keywords. This is fine since if they if they are spamming they will be punished, and if they want to earn lots of money to invest in their site they will need to play the game well and make good investments in other sites. We could publish some of the popular search terms on the front page, or if people want to recommend some categories to promote we could do that.

Who are you and where are you from (professionally and geographically)?

I’m a software engineering student from the bottom of the world (New Zealand). I have been working for an ecommerce store which is where I picked up the skills to do this.

How are you funded currently?

Currently the project is experimental rather then an actual company, it’s self funded.

What are your plans for this going forward?

The real problem is gaining users, the site isn’t useful unless people use it, and people won’t use it unless its useful. This is made harder by the fact we have $0 marketing budget.

The key things we are concentrating on at the moment are:
– We are trying to add in content to make the site more useful.
– Getting users.

For some reason, my request for a face-to-face interview with Larry Page and Sergey Brin was unsuccessful. Apparently, I needed to ask in 1996 to get an appointment any time soon. Nonetheless, the Google people were keen to answer my questions about the business. On the less positive side, I had to do the whole thing by email and the answers need to be attributed to a ‘Google spokesperson’. As I’ve said before, I think email interviews are less than satisfactory. Being a big company, they have to be pretty circumspect and so some of the answers are a bit bland, to say the least. Nonetheless, thought I’d share a portion of what they provided. Thank you, Oliver at Google UK, for co-ordinating this.

How do you define Web 2.0, if indeed you consider it worthy of a definition?

Here at Google we have no single definition of web 2.0. For us, the development of our services rests on keeping creative and innovative, maintaining our focus on improving user experience, and our goal to organise the world’s information and make it universally accessible and useful.

Two commonly used indicators of Web 2.0 are “the web as platform” and “the wisdom of crowds”. How is Google currently implementing these? A third element that tends to be cited is “the web as a social experience”. Are you active in this space?

Our overall view is “Don’t bet against the Internet”. What’s exciting today is that technologies like AJAX are making it possible for browser-based applications to have the rich functionality that used to only be possible with desktop-based applications.

A big objection to web applications is trust. How might this be overcome?

Google is committed to protecting our users’ privacy. We recognise that our continued success is based on earning — and keeping — our users’ trust. Unlike a credit card company or mobile phone operator, our users can switch to a competing search provider with the click of a mouse. All our work at Google is guided by clear privacy principles.

At Google:

- We build privacy protections into our products from the ground up;

- None of our products use any personal data unless fully disclosed in a privacy policy;

- We always ask people actively to opt-in to services that use sensitive data;

- We write our privacy policies in simple clear language so that users can easily understand them — they are not the usual legal jargon; and

- We allow people to use most of our services anonymously, and we even tell them how they can disable our cookies that they have been sent.

In addition, Google will take legal action to protect its users’ privacy. In 2005 the US Government asked us to provide two months worth of users’ search queries and billions of web addresses (URLs) as part of its effort to defend the Child Online Protection Act, a 1998 federal law that seeks to ban Internet sites from displaying content deemed ”harmful to minors”. The US Supreme Court ruled that the law couldn’t be enforced unless the Government could prove that less intrusive measures (such as Internet filtering) had proven inadequate.

Google challenged these demands. As Nicole Wong, Associate General Counsel, has explained:

While privacy was not the most significant legal issue in this case (because the Government wasn’t asking for personally identifiable information), privacy was perhaps the most significant to our users. As we noted in our briefing to the court, we believe that if the Government was permitted to require Google to hand over search queries that could have undermined confidence that our users have in our ability to keep their information private. (Google Blog, 17th March 2006).

A US judge largely ruled in our favor — forcing the Government to exclude all search queries from its demand and to limit the number of URLs to just 50,000. We believe that this was a real victory for privacy, and for our users.

How does AJAX help Google deliver a better service/experience?

With AJAX, web sites can be more dynamic and interactive, and the user interface can be more responsive. For example, AJAX technologies are behind the draggable maps on Google Maps. Likewise, AJAX technologies are behind Google Suggest, letting us display query suggestions from Google’s legions of servers in real time as the user types. These types of interactive features are enabled by AJAX, and they have let Google develop novel and innovative interfaces that improve our ability to deliver information to our users.

So-called “mashups” using the Google Maps API (in particular) are causing a lot of excitement. Why do you think this is the case? Any favourites there? Is this the future? (i.e. will the trend towards open APIs become dominant? why?)

Mapping has traditionally been a very expensive and complex technology. With the Google Maps API, web developers can incorporate maps into their web pages for free and with very little programming experience. Because the API is so accessible, web developers whose web sites had geographic data quickly adopted Google Maps to make their web sites more interactive, attractive, and useful for their users. We are really excited to see such wide adoption from such a wide range of web sites. There is such a variety of sites that use the Google Maps API, everyone on the team has their own favourite. We even do our own mash-ups sometimes, such as our integration of the stages of the Tour de France 2006 into Google Earth, which you can read about here: http://googleblog.blogspot.com/2006/07/tour-de-france-goes-3d-with-google.html

Google’s business model is generally to provide free services and monetize those services with targeted advertising. Open APIs like the Google Maps API are very compatible with this business model, and we hope to open up more of our computing infrastructure to developers in the future.

One driver (IMHO) of Web 2.0 is a demand for authentic, word-of-mouth relationships and referrals. Is web searching going out of fashion any time soon?

While it is difficult to speculate about future trends, we feel that providing relevant answers to search queries is something that users want and value, and we continue to devote the majority of our time to developing and improving upon our search services.

It’s important to remember that today only around 10 to 15% of the world’s information is available online. The more information that comes online, the more important search will become.

Sex and pornography have been trounced by business and e-commerce as the most popular internet search topics, new research shows.

In their mid-90s heyday, sex-related topics accounted for 17 per cent of web searches, but that figure has shrunk to an unsexy 3.8 per cent, Queensland University of Technology’s Professor Amanda Spinks said today.

But Prof Spinks said business and commerce-related topics, including buying and selling on the net, had outstripped sex to make 30 per cent of web searches.

I can confirm that in my mid-90s heyday, sex-related topics accounted for 17% of my thoughts.

Sounds like good news for booksellers. “Google Book Search may be facilitating sales of Books.” Well, maybe.

Heather then goes on to give a picture of the people who follow this search path: “The 55+ age group was 69% more likely to be on Google Book Search than average for the UK internet population … The Types that are most highly over-represented on Google Book Search UK are really interesting as they represent young uni students and the elderly.” One thing that links many members of those two groups together is, of course, low incomes.

Google’s book search is, among other things, a low-cost way to gain access to books and to learn more about books you might consider buying. One great feature is links from the book lists to worldcat, which allows you to see which libraries in your area stock the book. Perhaps the 16% downstream from book search to book shops represents the people who are frustrated that they can’t get the works they want as a free PDF or loan it from a local library?

Elsewhere: Students Seek Alternatives as Textbook Prices Mount

and Google’s PDFs not very usable.