The challenges that the major search engines face in penetrating vaster Web of hidden data that remain largely invisible to search engines is so-called Deep Web. Now new technologies taking shape that’ll extend the reach of search engines into the Web’s hidden corners. “Most search engines try to help you find a needle in a haystack, but what we’re trying to do is help you explore the haystack,” said Anand Rajaraman, co-founder of Kosmix, a Deep Web search start-up.

Prof. Juliana Freire at the University of Utah is working on an ambitious project called DeepPeep that eventually aims to crawl and index every database on the public Web. Extracting the contents of so many far-flung data sets requires a sophisticated kind of computational guessing game.

Full Article