Google’s top search result? Surprise! It’s Google
The search engine dedicated almost half of the first page of results in our test to its own products, which dominated the coveted top of the page
Pushed down the page
To determine the amount of space Google dedicates on the search page to direct answers and its own products, we built a custom scraper to gather all trending Google searches for two months, starting in November 2019. Then we built another scraper to run the searches as they would appear on mobile devices, where the majority of searches now occur. We wrote more than 1,000 lines of code to parse and analyze the resulting dataset, which contained more than 1 million rows.
We found that the majority of links to and results for non-Google sites were pushed down to the bottom-middle of the page, wheredatashows users are less likely to click.
We categorized search results intofour types: Google, non-Google, ads, and AMP, a web format created by Google four years ago. AMP pages are hosted by Google but created and monetized by publishers.
AMP has been controversial, with somepublishersanddeveloperssaying it gives Google too much control over the web. The companytells website ownersthat using it makes them eligible “for more prominent presentation of thumbnail images in search results and in the Google Discover feed.” Because AMP has some features of a Google result and some features of a non-Google result, we gave it its own category.
Levin objected to that decision, saying AMP results should be non-Google. “Those are outbound links to publishers and other web creators. To suggest otherwise is not factual,” she said. She also said our results may be heavy on AMP content because our sample, Google Trends, leans toward breaking news, which triggers “top stories” carousels. The news stories in those carousels were often delivered in AMP in our data.
Direct answers include “featured snippets,” which excerpt content from websites, and “knowledge panels,” which show summaries and facts drawn from the “knowledge graph,” Google’s fact database curated from other sources. They also include weather, sports statistics, and dictionary definitions. All of these appeared on the first search results page, typically at the top, without the need to click through.
Google acknowledgedin written comments to Congresslast fall that one major reason people end a search is because Google’s modules provided the answer on the search page.
In our sample, Google featured its dictionary definitions before Urban Dictionary, Cambridge Dictionary, Dictionary.com, Wikipedia, Merriam Webster, and Investopedia, among others. And searches for song titles typically returned a YouTube video in the top spot, followed by the lyrics, displayed in full on the search results page.
Levin said Google does not give preference to YouTube, its subsidiary. Recent tests by The Wall Street Journalfoundthat it did.
The quantity of Google results in some searches in our sample was quite large. A search for the Shania Twainsong“Man! I Feel Like a Woman,” which was trending during our study, returned the following on the first page: links to four YouTube videos in various positions on the page; a box labeled “about” with some hyperlinked facts that led to new Google searches; a box labeled “people also search for,” which led to a new Google search; and a “people also ask” box. Together, direct answers and results leading to Google products took up 67 percent of the first search results page for that query. Non-Google results took up only 22 percent of the page.
Even some of the “traditional” results that appeared after the Google results in that query were touched by its hand. Genius.com and Biography.com’s results were delivered in AMP.
Competing with advertisers
Travel websites are among those who say they have suffered from Google Search’s preferential treatment of Google products. TripAdvisor, whichlaid off 200 workersin January, pointed to Google “siphoning off quality traffic that would otherwise find TripAdvisor” as its “most significant challenge.”
In queries for specific airlines that appeared in our sample, Google presented Google Flights at the top of the results page, before links to the airlines’ own websites. A search for “nonstop flight” also returned Google Flights in the top spot, above competitors.
And travel sites aren’t just Google’s competitors; they’re also its customers. Together, Expedia and Booking spent $5.8 billion on Google advertising in the 12 months ending in September 2019, according Skift, the travel research firm.
“When they compete against their advertisers … I think it’s bad practice,” Barry Diller, chairman of Expedia Group, said during an earnings call in February in which he called Google an “existential” threat.
In the industry, Google Flights is not seen as the best product. Itdid not crack the top 10of Frommer’s 2020 ranking of airfare search engines, for instance.
The Markup found that Google Flights did not always display the cheapest fares or all available flights, even when those fares and flights showed up in ITA Matrix, which is powered by the same software Google acquired in 2010 and used to launch Google Flights.
For example, a search on Google Flights for a one-way flight from Billings, Mont., to London’s Heathrow on Feb. 19 showed the cheapest flight was $1,068. The same search on Orbitz turned up a flight for $772, while ITA Matrix offered $766.69.
And when The Markup searched for a one-way flight from Morgantown, W.Va., to New York City on Feb. 21, Orbitz produced a long list of results, including a three-hour-and-40-minute journey that combined flights from Southern Airways and American Airlines with a stop in Pittsburgh for $253.47. The same search on ITA Matrix showed the same flight at a different time for $242.40. But a Google Flights search showed nothing: “Aw snap, no results.”
The FTC found the same thing in 2012, according to the memo accidentally disclosed to The Wall Street Journal: “Although it displays its flight search above any natural search results for flight-booking sites, Google does not provide the most flight options for travelers,” the regulatorswrote.
Levin would not directly address that issue with The Markup. Google’stravel support webpagesaid that “a partnership with Google is required” to appear in Google Flights but gave no further details.
Once up-and-coming travel website Hipmunk blames its death in part on Google Search’s high ranking of Google’s travel products. A Silicon Valley favorite, Hipmunk was among the first to compare prices for Airbnb alongside hotels. It invented an “agony” ranking to help customers decide if that cheap flight was really worth it.
To drum up traffic, it created webpages with highly specific travel tips like “How long does it take to get to the airport from downtown San Francisco?” This strategy led to booming “organic” traffic from Google, said its CEO at the time, Adam Goldstein. Then Google started to feature Google Flights, Google Hotels, and its“destinations on Google” travel guide prominently in search results.
“One quarter, that traffic started to really fall from what we were planning. And then the next quarter it fell short by even more,” Goldstein said. “This was around the time that I realized that the doors were starting to close in on us and all the small players in the travel industry.”
As Google Flights grew, airlines realized they didn’t need referrals from smaller players like Hipmunk, Goldstein said. The airlines lowered fees to nearly zero for small players, and airlines even began demanding that online travel agencies hide certain routes in exchange for access to fare data, according to Goldstein andcomplaintsto the U.S. Department of Transportation. The department launched an inquiry but has not taken any action.
Goldstein gave up and sold the company to the corporate travel giant Concur, which he thought had enough heft to compete. Concur shut Hipmunk down in January.
Levin, the Google spokesperson, declined to address Goldstein’s complaints.
Page, Google’s co-founder, had bashed competitors like AOL and MSN in an interview with Playboy in 2004 for prominently placing their editorial content above competitors.
“Most portals show their own content above content elsewhere on the web,” he said. “We feel that’s a conflict of interest, analogous to taking money for search results. Their search engine doesn’t necessarily provide the best results; it provides the portal’s results.”
Long after leaving AOL and MSN in the dust, Google now says users want answers on the page—or, in their smart speakers, as officialssaid in a 2019 SEC filing, because it’s “quicker, easier and more natural to find what you’re looking for.”
This philosophical switch started around 2007, when Googleintroducedwhat it called “universal search,” which incorporated Google Maps, Google Books, and Google Video into general search results, frequently at the top.
At a conference in Seattle in 2007, Marissa Mayer, then Google’s vice president of search products and user experience, was asked about the sudden appearance of links to Google Finance ahead of Yahoo Finance in the design of the stock ticker info boxes on the search results page at that time. Sheresponded, “It seems only fair, right? We do all the work for the search page and all these other things, so we do put it first.”
There are now many varieties of Google-created results featuring its products or information taken from other sites delivered as direct answers. Google even serves up results that lead nowhere and seem purely for entertainment, like a module that producesanimal sounds.
Other search engines, including DuckDuckGo and Bing, also provide answers on the search results page but attract less criticism because of their small U.S. market share. DuckDuckGo has 1.5 percent, and Bing has 7 percent, according toStatcounter.
“Fundamentally, consumers do want instant answers,” said Gabriel Weinberg, CEO of DuckDuckGo. “That is a tension with the open web and the sites that produce that information.”
DuckDuckGo does not have any products to promote in its search results, but Bing, which mirrors Google’s features closely, directs users to Microsoft-owned products such as Bing Maps and its flight booking tool. Microsoft declined to comment.
“It’s not a snippet on Google”
Some of the information Google presents in direct answers is gathered from websites that agree to it, including the crowdsourced encyclopedia Wikipedia, which allows anyone to publish its content as long as it’s credited. Wikipedia said it has not seen a drop in donations as a result.
Dario Taraborelli, former head of research at Wikimedia Foundation, which operates Wikipedia, said he is concerned, however, about the effect on media literacy.
“It’s become really difficult to understand where information comes from. What is the provenance of what we’re learning?” he said. “It’s going to be become much harder for a new contributor to understand … that Wikipedia exists as a separate project, that it’s not a snippet on Google.”
A 2017studyfound people were more than five times more likely to mistakenly attribute information to Google itself after reading it in a Google direct answer than when no direct answer appeared in results.
Google also scrapes websites without explicit permission, a practice it has defended byarguing the “fair use”exemption to copyright laws and noting that websites canopt outof inclusion in featured snippets while remaining in the search index for other results.
Brian Warner, founder of the website CelebrityNetWorth, said he declined Google’s request to include his site’s information in the “knowledge graph,” but two years later Google started showing information taken from CelebrityNetWorth in “featured snippets” results.
His traffic crumbled, and Warner said he had to lay off half of his 12-person staff.
The song lyrics site Genius said its click-through rate from search results fell from 60 to 80 percent to 5 to 20 percent after Google started displaying lyrics on the search results page. The company also alleges that one of Google’s lyrics providers, LyricFind,had stolenGenius’s watermarked lyrics and they were showing up in direct answers.
“The fact that Google often populates its lyrics boxes with lyrics misappropriated from our website makes Google’s behavior even more unfair,” Ben Gross, chief strategy officer at Genius, said in an email.
GeniussuedGoogle in December. Google filed a demand for a jury trial. Levin declined to comment on the company’s claims or its lawsuit.
Some websites try to get into featured snippets because it’s high up on the page. Levin said “it can drive meaningful traffic to their sites” but declined to provide specifics.
Some sites take whatever traffic Google search provides and don’t complain. “My goal in life is not to cross the Google gods,” said one sports media executive who did not want to be quoted by name or the name of the website for fear of being shut out by Google.
Cummings, of SpanishDict.com, said something similar. “Google delivers the traffic for the whole internet. Unless your name is Facebook, you rely on Google,” he said. “It’s very risky to speak out at Google because you don’t know what type of retaliation you’ll face.”
Frederic Lambert said his site was demoted in search listings when Google rebooted its shopping product in 2012.
A penny-pinching roadside ad salesman, Lambert launched Acheter-moins-cher—“buy cheaper”—in 1998 to aggregate products from around the web and rank them by price. It allowed consumers to set alerts if the price dropped. When users clicked or bought something, sellers paid him a commission.
At its peak, about 40,000 shoppers visited the French site, and it took in more than a million euros a year in revenue. Starting in 2012, revenues decreased by half, until he shut it down in 2018.
“They did kill us,” he said. “But not by making the best product.”
In 2017, the European Commission levied a then-record-breaking €2.42 billion fine against Google, finding it gave preferential treatment to Google Shopping while demoting comparison shopping sites in the rankings. Google said itdisagreeswith the commission’s decision, which it hasappealed.
Still, it agreed to allow comparison-shopping sites to buy ad slots in Google Shopping, a remedy the commission approved. Levin, the Google spokesperson, said 600 sites are participating, and their “click and impression” share grew from 20 to 40 percent between 2018 and 2019.
Last year, 41 comparison shopping companiessent a letter to the commissionsaying that they “have not experienced any substantial overall increase in traffic to their websites.”
Google continues to abuse its market power, they said, and the industry has not rebounded, despite the remedy.
“Many competitors died,” said Thomas Höppner, a partner at the law firm Hausfeld, who represented three Google Shopping competitors in the European Commission case. “And some argue that the industry will never fully recover.”
This article wasoriginally published on The Markupby Adrianne Jeffries and Leon Yin and was republished under theCreative Commons Attribution-NonCommercial-NoDerivativeslicense.
