Bing Visual Search Offer Detailed View within Image

Bing upgrades its image search capabilities lets people search for specific items shown within a image, follows Pinterest and Google.

Share online:

Bing is enabling people to automatically detect intent and provide great search results for images with a new feature called Bing Visual Search, available on Thursday, as part of existing Bing image search tools.

Bing already provide 'Search By Image' capability, that lets a user search by an image. Now, what if a user want to search for a certain object in an web image, or the one they photographed, this is where the new Visual Search tool will help along side the existing image search tools.

The tool available now on Bing.com on both desktop and mobile, as well as in Bing mobile app. In addition, Bing is also making its visual search available to developers via Image Search APIs.

Explaining how this visual search tools work, bing in an example says, "for example when you look bing for kitchen decoration inspiration, and is attracted with an image — now a click on the thumbnail result will bring up the 'Detail View' with a magnifying glass in the top left of the image called "visual search button."

bing visual image search: example kitchen decor ideas
Bing Visual Image Search Example

Clicking this button displays a visual search box on the image, just click and drag it to cover just the object of interest — or simply draw a box around the oject. Bing will instantly trigger a visual search using the selected portion as the query.

For related products, Bing automatically detect the shopping intent and, in addition to regular image search, also run a product search for matching products. Now, just click on the right product, pick a merchant on the detail page and finalize the purchase.

bing visual image search: example related products for kiteche decor
Bing Visual Image Search Related product example

Not happy with results, you can continue with exploration of similar images by clicking on "Related Images."

For better results, Bing advises to "tweak the visual search box to fully capture the object of interest to get the best results."

Bing shared some example screenshots of its new visual image search:

bing visual image search example with related products

So how does it all work under the hood?

Bing explains, "as a first step to understand the query-image, they run Image Processing Service to perform object detection, extraction of various image features including DNN features, recognition features, and additional features used for duplicate detection.

Next, it generate best text query to represent the input image based on a pipeline used in its image search to identify BRQs ('best representative query'). Subsequently, leveraging expertise used in Bing answers, a model triggers to identify different scenarios for search by image.

In the next step i.e. matching, it employs a technique known as Visual Words, which narrow down a set of candidates from billions to several millions.

After the matching step, we enter the stage of multilevel ranking. We need to rank millions of image candidates, and we'll do that based on feature vector distances. To speed up the calculations, we use an innovative algorithm developed by Microsoft Research in collaboration with University of Science and Technology of China called bing visual image search under the hood architecture

After this, an innovative algorithm called 'Optimized Product Quantization' decomposes original high-dimensional vector into many low-dimensional sub-vectors that are then quantized separately.

Then, the distances between query-image and result-image vectors are calculate through a table lookup against a set of pre-calculated values to speed things up even further.

As a result, they now perform a multiple levells of ranking images, which is followed by the de-duplication to end up with the final result set that will be returned to the user."

bing visual image search under the hood

About The Author

Deepak Gupta is a IT & Web Consultant. He is the founder and CEO of diTii.com & DIT Technologies, where he’s engaged in providing Technology Consultancy, Design and Development of Desktop, Web and Mobile applications using various tools and softwares. Sign-up for the Email for daily updates. Google+ Profile.