Microsoft fires back at Yale professor who calls 'Bing It On' claims bogus

This week, a Yale law professor took aim at Microsoft's "Bing It On" campaign, which purports to show that users prefer the company's search engine to Google's in a majority of blind tests. Writing in Freakonomics, Ian Ayres writes that Bing's claims are misleading — and he set up a trial of his own to prove it. He worked with four Yale Law School students to run the blind test at BingItOn.com with 1,000 people recruited through Amazon's Mechanical Turk. And Ayres' results differed sharply from Microsoft's: "We found that, to the contrary of Microsoft's claim, 53 percent of subjects preferred Google and 41 percent Bing (6 percent of results were "ties"). This is not even close to the advertised claim that people prefer Bing 'nearly two-to-one.'" Ayres called Microsoft's Bing ads misleading and even suggested that Google could file a deceptive advertising lawsuit against the company. "We simply don't track the results." Today, Microsoft offered a rejoinder from Matt Wallaert, a behavioral psychologist at Bing. Wallaert argues that Ayres ignored key differences in the studies Microsoft used to generate its claims, and said the company would not release a complete data set from its trials because it doesn't track the data generated there for reasons of science and privacy. "It isn't conducted in a controlled environment, people are free to try and game it one way or another, and it has Bing branding all over it," Wallaert writes. "So we simply don't track their results, because the tracking itself would be incredibly unethical. And we aren't basing the claim on the results of a wildly uncontrolled website, because that would also be incredibly unethical (and entirely unscientific)." Wallaert sidesteps the question of why Google's results bested Bing's own in most cases on BingItOn.com. Notably, Bing performed best when it suggested queries to users taking the test. Ayres suggests that Bing is choosing queries that it knows generate better results on Bing; Wallaert says it took those queries from popular search terms in Google's 2012 Zeitgeist report. "it may be because we provide better results for current news topics than Google does," Wallaert writes. There's nothing definitive in Ayres' study to suggest that Microsoft is wildly overstating the claims from its "Bing It On" studies. At the same time, Wallaert can't quite shake the suggestion that the blind trial is a shoddy way of determining which search engine is superior in everyday use. A better way is to occasionally run queries on different sites and see which one performs better over time — something that most people concerned about the quality of their search results are likely already doing.