Under the Hood: How Google's Search by Image Uses Computer Vision Tech to Analyze Your Image?

Ever thought, how Google's Search by Image feature, which allows you to search using a picture instead of typing in words works?. Google Inside Search blog today, shared under the hood at how Search by Image uses computer vision technology to analyze your image, determine what it is, and return relevant results to you."Computer vision […]

Ever thought, how Google's Search by Image feature, which allows you to search using a picture instead of typing in words works?. Google Inside Search blog today, shared under the hood at how Search by Image uses computer vision technology to analyze your image, determine what it is, and return relevant results to you.

"Computer vision technology is an active area of computer science research because it's difficult for a computer to match a person's ability to see and understand. Search by Image uses computer vision techniques to "see" what's in the image. Computer vision technology doesn't look at the image filename or where the image came from -- rather, it looks at the content of the image itself to determine what that image is," informs Google.

Google explains that "When you upload an image to Search by Image, the algorithms analyze the content of the image and break it down into smaller pieces called "features". These features try to capture specific, distinct characteristics of the image - like textures, colors, and shapes. Features and their geometric configuration represent the computer's understanding of what the image looks like." "These features are then sent to our backend servers and compared against the billions of images in our index to see if a good match exists. When the algorithm is very confident that it's found a matching image, you'll see a "best guess" of what your image is on the results page. Whether or not we've a best guess, you'll also see results for images that're visually similar -- though they mayn't be related to your original image."

"Because our algorithm sees the world through the "features" that're extracted from images, those define what it can "see" well and what it can't. We're more likely to find a good match if your image query has a unique appearance. Other things that lack distinctive features or a consistent shape, like a crumpled blanket or a puppy, don't result in confident matches, but will return images which're visually similar in appearance. You can refine your results in those cases by giving the algorithm a hint. Add a word or two into the search box that describes the image, and the results may display better "Similar Images" results," Google said.

"The results page summarizes a variety of information that we can match for your query image, including our best guess for the image, related web results, and images that are visually similar to the one you've uploaded," added Google.

Check this video below for an animated look at how Search by Image works:

[Source:Inside Search blog]