…in the Semantic Web individual machine-based agents (or a collaborating group of agents) will be able to understand and use information by translating concepts and deducing new information rather than just matching keywords.

Once machines can understand and use information, using a standard ontology language, the world will never be the same. It will be possible to have an info agent (or many info agents) among your virtual AI-enhanced workforce each having access to different domain specific comprehension space and all communicating with each other to build a collective consciousness.

You’ll be able to ask your info agent or agents to find you the nearest restaurant that serves Italian cuisine, even if the restaurant nearest you advertises itself as a Pizza joint as opposed to an Italian restaurant. But that is just a very simple example of the deductive reasoning machines will be able to perform on information they have.

Far more awesome implications can be seen when you consider that every area of human knowledge will be automatically within the comprehension space of your info agents. That is because each info agent can communicate with other info agents who are specialized in different domains of knowledge to produce a collective consciousness (using the Borg metaphor) that encompasses all human knowledge. The collective “mind” of those agents-as-the-Borg will be the Ultimate Answer Machine, easily displacing Google from this position, which it does not truly fulfil.

There’s little to dislike about a better web usage model which will find answers rather than match search terms. The name Web 3.0 is as misleading as Web 2.0, though. Arguably, more so. We’re not talking about a new internet, or even, really, a better internet. For one thing, for the most part, it will be the same old web — you can’t retrofit a gazillion billion pages with a semantic markup that there’s still considerable debate about. What we’re actually talking about is improved ways into that information base.

And yes… I have still got a year and a half left on the domain name.

The problem here is that, for many types of news and information, newspaper stories don’t cut it anymore.

So much of what local journalists collect day-to-day is structured information: the type of information that can be sliced-and-diced, in an automated fashion, by computers. Yet the information gets distilled into a big blob of text — a newspaper story — that has no chance of being repurposed.

When he says ‘repurposed’, Holovaty isn’t talking about RSS or putting the content on mobile phones. He means that the atoms of the event/story could be identified and potentially recombined to suit a multitude of purposes. Any given story has attributes — places, people, dates, types of event — that could be tagged and recorded systematically.

A story about a burglary, for example, could automatically be added to ongoing, currently unwritten stories about where and when burglaries take place, presence and type of alarm, modes of entry, types of victim and perpetrator. At the moment, because newspapers’ basic unit of currency is the story, it’s almost impossible to extract this sort of information. The focus on stories represents a ‘lost opportunity’ for shared research, re-use and the discovery of those hidden stories that lie between the experiences of many journalists.

Adrian doesn’t mention the words semantic or microformats, but it’s fairly clear that he’s talking about a semantic news. If news stories were machine-readable, then fairly labour-intensive tasks such as “what’s the weather likely to be on election day, and what effect will this have on voter turnout” could be a piece of cake.

I don’t think the news story is quite dead yet. People prefer a story to a heap of facts. It helps us to contextualise events, to decide what’s important and worth caring about. We need people (journalists and editors) working full time on connecting and narrativising (if that’s a word) the things that happen because we haven’t got time to do it ourselves. It takes time, brain power and imagination to work out that there might be a connection between different phenomenon, to establish the links and to present the results as a digestible whole.

However, this more semantic approach to news gathering would make it possible to produce more in-depth, factually rich stories more easily. And make pursuing stories based on a hunch quickly proven or disproved without wasting anyone’s time. That’s good news for journalists, news publishers and the public.

It turns out that that is one of the more difficult questions. (Damn!) It depends on what you mean. You might mean turning the billions of existing web sites semantic or only about possible future sites or services. The second of these options is the most likely outcome at the present. Semantic web is partly about annotating web pages to make them amenable to machines. John prefers the expression ‘semantic technologies’ to avoid this confusion.

At the moment, information on the web is pretty much designed for human consumption. You and I know when we go to a shopping site that the figure in bold is the price, that a certain number is the product code and that this piece of information is about the shipping information. To a machine, it may make no sense whatsoever. If machines are to be able to bring together all these different pages to make the web more useful, then they need to be able to read them.

We’ll see the first applications of semantic technologies in the enterprise space. Its need is more acute. They have lots of databases, all built by different people according to different rules. Integrating the information from those is already a very costly and time-consuming activity. One database may talk about CustomerName, another may refer to CustomerID, for example. Joining these things together, so perhaps, a support department knows about what equipment the logistics department has installed for a customer, improves business efficiency. Semantic technologies put what Davies called a “wrapper” around these different data sources to create overarching access, connecting different datasources in a way that doesn’t require nearly so much human effort.

People developing semantic technologies work by developing an ontology for understanding the sort of data it’s looking at and the technology will be able to do some reasoning based on this. An ontology means a form of classification system for whatever it is that’s being examined. For foods, that might include their ingredients, nutritional properties, suppliers and type. It’s not just a list, though, but will also understand the relationships between different items. An ontology developed for food might come across E101 and additives and CrispyPop bar. It will know that E101 falls under additives which are part of the ingredients of the bar. If that description then gets combined with a database of shops at a wholesaler where you might send the bar, then the semantic agent will calculate that health food shops aren’t going to stock CrispyPop bars. It’s not intelligence in any way, but the application of rules that the creators have decided upon.

Because the semantic technologies are lightweight and open source, they are potentially available to any company. For this reason, enterprises that get some of their data from external sources will still be able to use semantic approaches to integrate and drill the ensuing combination. These are my words, not John’s, but one way to think of the technologies is as providing a toolkit for more easily creating web mashups. Companies already exist, such as Cerebra and Ontoprise, to sell ways to integrate enterprise information using OWL, the web ontology language.

I kind of understood, so far, but I needed a good example. I suspect you may be the same.

BT works closely with the National Health Service in the UK. The Service has already gone a long way into digitising and collating its information through the Electronic Patient Record system and also information on medical knowledge through the SNOMED classification system. Unfortunately, though, the data can still be very dispersed. The X-Ray department might have a patient’s data on a different system to the Pharmacy, for example, and those might be completely unconnected to the systems used in a different hospital or by a clinic.

What semantic web technologies bring to this is unity and also what John called Description Logics. It can prise open the different databases, allow an overview, but also calculate with it. Imagine a patient’s medical record says that they are allergic to almonds. Then a doctor misses this and somehow prescribes a nut-based food. When the nurse enters this into the patient’s record, the semantic application will use its ontology to work out that almonds are nuts and that therefore this is a very bad idea. Semantic technologies that can perform calculations like this, and potentially save lives, are already in use in the UK Health Service.

I’ll leave this there for today. My head hurts already. If there’s interest, I’ll be happy to do a follow-up on another day.

In the list that follows, I’ve taken a lot of these characteristics or definitions from Tim O’Reilly’s What is Web 2.0?, and also Paul Graham’s Web 2.0 and Jason Fried’s user survey about the term.

1. The wisdom of crowds: We’re thinking here of things like digg that harness collective judgements to decide the importance of news stories. People talk about the power of ‘network effects’ when they’re keen on this definition. Google Search works like this by using the number and quality of inbound links to decide a page’s importance. But the whole idea does not apply to Google Maps, or any of the other Software-as-a-Service (SaaS) crowd e.g. Basecamp, Writely, 30boxes, etc., which are nonetheless thought of as being Web 2.0. Nor does it apply to social networks that are just about developing and maintaining friendships, like MySpace, though they do benefit from network effects, of course.

2. Shared Web Applications. One of the definitions from Jason Fried’s list and quite promising. Almost the opposite of our first definition, since it quite clearly applies to things like Basecamp, Writely and 30Boxes. However, there are some Web 2.0 applications that have no social element whatsoever, e.g. Pandora, Google Maps, Orchestrate, goowy. I’m also struggling with the idea of web applications. I can see why digg and Google Search are applications, but to have this as a defining feature of Web 2.0 would mean classifying MySpace as an application. And if I allow that, then almost any web site becomes an application.

3. Web as platform: It’s hard to know where this one starts and ends. In some sense, every web page is using the web as a platform. For Tim O’Reilly, who came up with this explanation, it means services that could not exist without the web, and he’s thinking of things like eBay, craigslist, Wikipedia, del.icio.us, Skype and Dodgeball. For me, that means that every online community could fall into this category. Are message boards and usenet Web 2.0? Most people would say not. Too broad.

4. User Participation: This is about the pointing out the differences between old-fashioned newspaper and magazine sites and new services like YouTube, flickr, and OhMyNews where the consumers are also the creators. The expression ‘Read/Write web’ crops up among proponents of this definition. Again, it’s rather too broad, so it could equally apply to message boards, but also too narrow in a different way, since it misses the SaaS sites.

5. Rich User Experience: Web 2.0 sites use CSS, AJAX and other technologies to enhance usability and create dynamic pages able to display more information in the same space. But hang on, the default MySpace page is probably one of the least “rich” imaginable. Oh, apart from craigslist. And until they introduced search term prediction earlier this year, Google Search didn’t use any fancy presentation technologies at all. Also, the presence of an AJAX-enhanced shopping cart on an etailer site doesn’t really capture what people mean by Web 2.0. Dell.com, for example, has had a ‘live’ shopping cart for years. It’s a good cart, but Web 2.0?

6. Marketing Buzzword: This is what all the sceptics say. So Google Search and Amazon and eBay and craigslist, all of which are believed to be Web 2.0 applications, because they match some of the other characteristics I’ve described here, are just some sort of modern fad that’s going to fade away, are they? The same thing goes for anyone who wants to describe Web 2.0 as “the new stuff on the web”. I do agree, incidentally, that Web 2.0 has become a marketing buzzword, it’s just that I think that it’s also more than that.

7. Data is the next Intel Inside: Though it’s a bit of a mouthful, I actually quite like this one. Again, it’s from the O’Reilly paper. Data management is a core competency of Web 2.0 companies. “SQL is the new HTML,” is another quotation from the paper along these lines. All the Web 2.0 crowd, and we can go from giants like Amazon and Google to startups like 30boxes and Orchestrate, operate mainly from databases to contain and present personalised views on that data. There’s two problems here: (a) data management isn’t quite such a sexy idea as people would want and (b) a lot of the Web 1.0 companies were also about finding clever ways to use databases e.g. Altavista, Lastminute.com.

8. Permanent Beta: Web 2.0 applications are re-released, re-written and revised on an ongoing basis, putting paid to the yearly release cycle that characterised earlier software development. Most Google applications, for example, are still in Beta. flickr is rumoured to sometimes be revised every 30 minutes. MySpace and the other social networks add extra features every couple of weeks. I think that this is a clear characteristic of Web 2.0 apps. But it’s also become a feature of mainstream applications. Windows and MacOS, for example, get new fixes and patches every month. Antivirus programs are updated every day, but they aren’t Web 2.0, are they? The same thing goes for ‘lightweight programming models’. Also, I think people mean more by the term than the way in which it’s programmed. Most users couldn’t care less, they just want it to work well.

9. Using the web as it was meant to be used: This one is from Paul Graham’s essay on the subject. He’s referring to the increases in usability that are achieved through very good design as well as things like AJAX, but also by allowing users to develop their own ways of organising the information they have, the way del.icio.us and flickr do. Again, I have a couple of problems here. Firstly, it’s a bit loose: I’m sure that there were always some very well-designed sites that worked exactly as you wanted them to. The old (and now defunct) UK train timetable site was a perfect web app in many senses: it got you train times quickly and easily. But no-one would call it Web 2.0. Second, it’s a little self-satisfied as a definition and implies we’re reaching an end-point. A lot of the sites described as Web 2.0 have quite clearly got it wrong.

10. Nothing: One of the more popular answers in Jason Fried’s user poll. It’s a hard one for me to evade given that I have just come up with counter-examples or objections to all the definitions I’ve been able to find. Still, I resist the idea that this is nothing. Here are two answers to the question I think are true. (a) A Web 2.0 application, site or service will have a combination of the features given above. Just as black and white aren’t satisfactory for describing the colour of everything, neither is Web 1.0 vs. Web 2.0. It isn’t a binary division of the web, or a revolution. Instead, we have a spectrum. Those sites and services which satisfy a number of these criteria or characteristics are more Web 2.0 than those which don’t. That is not a value judgement, of course. Sites with no Web 2.0 features can still be wonderful. Sites with a lot of them can be awful. Also (b) Web 2.0 is still too young as an expression to have reached the point where we have consensus about what it means. It means different things to different people at the moment. It may only be with hindsight that we come to be able to narrow things down enough to be able to say what it was in one sentence.

Did you invent Web 2.0 or discover it?

Neither! It’s a name attempting to point people at something that existed. It wasn’t even me who came up with the expression. However, it’s an idea that I’ve been pursuing since 1997. I started talking about ‘infoware’, which is much the same thing, at the same conference [Linux Kongress, May 1997] that Eric Raymond started talking about The Cathedral and the Bazaar.

Many applications and services that use the web as a platform (e.g. Writely) seem very different from those that use the alleged wisdom of crowds (e.g. digg). Hasn’t it been misleading to call them the same thing?

Web 2.0 is a catch-all term, for sure. But when I talk about the web as platform, we’re talking about using the network as a platform. And that does include the examples you point to, albeit with different emphases. This means a completely different approach to software development and to distributing that software. We’re still getting used to that and adjusting.

In the same way, the original PC applications were very much like mainframe applications. It took a long time before we arrived at the idea of shrink-wrapped software you can buy in a regular store. In the same way, we’re still getting used to the idea of what Web 2.0 means. The people who realise where the leverage points are will win. There’s a shift in power from software APIs to big databases. The people who own the databases will win. Those databases might be records of people, or it might be devices, behaviours or geographical information.

A lot of critics of Web 2.0 ventures point to flakey business models built on CPC advertising. Is this a fair characterisation?

Focusing on the failure of companies and ventures is always a big mistake. It stops people making real progress and draws attention away from what is successful. However, Web 2.0 is not about these bubble companies, it’s about the new approaches we are trying.

Most of the experimentation happening now is wrong. But by having those experiments it means we are learning what distinguishes the survivors. These new paradigms mean that there is a lower barrier to innovation. I think maybe the top ten of the Web 2.0 experiments that are big now will survive.

In any case, I think bubbles are a good thing. That’s how you get capital redeployed.

There’s a lot of controversy at the moment about paying the users of, or contributors to, Web 2.0 applications. What’s your take on that?

It’s what we ultimately have to figure out. The applications have to give the users a payback of some kind, whether that be in the experience or the outputs they get from them, If the applications aren’t working well in that way for users, then they’ll want to get paid in cash. There’s more than one answer.

To what extent do you think that Web 2.0 principles like communities, social networking, openness and software-as-a-service will become a permanent feature of the internet?

Communities and social networking have always been with us on the internet and they always will be. However, I think that other things will change. It will become harder and more closed. It’s like when the internet first started, everyone was equal, then barriers started to appear. Access to data will become more guarded in Web 2.0, I think, and so there’ll be fewer, more powerful players as time goes by. That’s not so true of the software, where I think openness is a lot more important to success.

How long do you think the term Web 2.0 will last before we start talking about something different?

I originally thought is was good for a couple of years. Now, I think it’s probably got another four to five years in it. There’s still a lot to talk about and learn.

Is that something different the semantic web?

Hmm. Before we had the web, there was going to be something called Open Systems Interconnect (OSI). It had been researched by all the top academics and was mandated by the government. It was a lot more comprehensive and clever. There wouldn’t have been things like 404 errors or out-of-date pages.

Then came along this crappy thing called the internet. And, as it turned out, though it was inferior to the OSI in many respects, it was good enough for most people, and as we know it’s never looked back. I think the academics think way too hard about these things. In a lot of ways, worse is better.

That’s not to say that I think the semantic web people have got it wrong. They have a lot of ideas that are right. However, I believe that Web 2.0 is already the semantic web. We are building meaning into the pages. Ultimately, people will solve the problems that need solving and ignore the little things that don’t bother anyone. Only the solutions that offer value to lots of people will be propagated.

As a publisher, doesn’t this boom in self-publishing make you uneasy?

Inasmuch as it does threaten what we do, so that has to change. People buy much fewer reference books than they used to, so we don’t publish as many. But we’re also interacting with this movement. We’re doing more to build interactivity into our books, even building Web 2.0 apps ourselves to extend the book experience onto the internet. Make magazine is closest to our new model. That was put together by people we found on the internet who had something new and different to share. We can help them do that. We’re also experimenting with using internet wikis as a way of putting together books.

The role of the publisher is in selecting and adding value to information, and the need for that won’t go away. Our business is changing the world by spreading the knowledge of innovators. However, as with all this stuff, there’s going to be a period of upset and disruption before we discover the new rules.