So, you use the internet? Congratulations, you have a couple of thousand terabytes of charted web @ your disposal: company websites, twitter streams, the magic Kingdom of Facebook, and the wondrous tentacles of Google land. But all of this mindboggling information is only a tiny percentage of what the internet really is: a gargantuan monster…
Picture this: The World Wide Web is rather huge, really… Google found more than a 1 trillion (that’s 1,000,000,000,000) unique URLs on the web, and is still trying to index all of those ( in 2006, 25 billion sites were fully indexed). However, most experts refer to this “visible” part of the web as the “surface web”.
Surface web is an adequate term, if you currently draw your nets in the ocean of online info; you’re barely scratching the surface. The Dark Web, or Hidden Web is approximately 540 times bigger than the web you experience daily. Apart from secret military streams, long lost and forgotten early-day-experiments, over machine-to-machine botnets and criminal set-ups, there are whole sections of the web (like freenet for instance) that are concealed from the normal user.
While big players as Google, Bing and Facebook desperately try to chart, map, reach and index this Deep Web or Dark Web, none of them are making remarkable progress: the Dark Web is still uncomfortably dark, and “hidden”. However, in this Dark Web, people are storing data, having conversations, expressions, opinions… that are now mostly lost for the indexing, tracking and measuring giants.
Michael Bergman is an American academic, specializing in this Deep Web. He found the deep web to be approximately 550 times larger than surface world wide web. His study says that: “The deep web is the fastest growing category of new information on the internet … The value of deep web content is immeasurable … internet searches are searching only 0.03% … of the [total web] pages available.”
Tim Berners-Lee, CERN scientist who stood at the very cradle of the world wide web has a compelling vision: “I have a dream for the web in which computers become capable of analyzing all the data on the web – the content, links, and transactions between people …” His dream of a Semantic, indexed and holistic web is still a distant dreamy thought however… But the key to a better understanding of knowledge, sentiment and vision might be found in the dark web.
Content is Gold. Measuring is knowing. 99% of the web remains unexplored. Leave the charted waters, Go West. Again ;-).
