Bing's Research & Development team explores the benefits of having a dynamic number of search results and provides valuable insights that led Bing to challenge the status quo.
Bing said that most search engines still provide a fixed number of web links on the search results page (SERPs), typically ""10 blue links" in addition to instant answers."
Adding, it said that intuitively blue links are no more the right number of links for all queries and scenarios. That's why, Bing adapt the search results page size dynamically.
For example, "showing fewer search results in navigational queries may enable users get to their desired URL faster. Also, users may benefit from seeing more blue links when they return to a search results page after clicking on the browser's back button," explains Ronny Kohavi.
After the 50 percent of users that click the top result, Bing says that only about 4-6 percent click the third result (depending on if it's an Instant Answer or a regular web page link). Only about 2-3 percent of users click on the fourth result, and so on … down to fewer than one percent click-thru rate on the eighth search result. (Oddly, the graph and blog post completely skip the CTR for the No. 2 search result.)
On average, over 50% of users click on the first result on the page. Bing says from there it see a significant drop--"only about 4-6% click the third result (depending on if it's an Instant Answer or a regular web page link). Only about 2-3% click on the fourth result, and less than 1 percent clicks on the 8th link on the page."
Figure above shows click-through rate dropping from position 3 on, where position 3 could contain an Instant Answer, or the 2ndweb result that was pushed down because there was an Instant Answer above it, etc.
"When users click on a result, then hit the browser back button, they typically look lower on the page. Statistics showed that the click-through rate on lower positions are a factor of five to eight times higher after a back button," Kohavi expalined.
This observation, led Bing to a change that SERP initially showed eight algorithmic results, in the US in late May 2012-then the page was extended to show 12 after a back button is hit.
With this controlled experimentsm Bing says it saw improvements in key metrics such as "users were executing fewer queries per session, pages rendered faster on average, and pagination to page 2, 3, etc. reduced by almost 2%."
Kohavi also discusses users interact with "deep links" -- multiple results from one source grouped together--explaining that the click-through rate on the first result (a large block of results) is over 75%.
"We experimented with truncating such result pages more aggressively, showing only the first four elements (either algorithmic results or instant answers) and news when relevant. Conversely, we extended the SERP after a back button to 14 algorithmic results," Kohavi said.
The results further improved key metrics: "users were more successful in sessions, they reached the web link they wanted sooner, queries per session declined, pages rendered faster on average, and pagination was reduced by a further 5%."
This led, Bing to ship the feature to all users on April 22nd, 2013.