Google’s search monopoly is becoming an AI monopoly

What happened

Google became the king of internet search and created a free cornucopia of knowledge by striking a clever bargain with anyone who made a website: Let the company’s crawler index your pages and Google would send human readers back. That trade looks very different today thanks to Google’s strategy of using its search infrastructure to strengthen its artificial-intelligence products.

There’s nothing wrong with bots doing things that are useful, but the internet will become vapid, less original and ultimately less useful if human contribution is no longer rewarded and peters out. That’s what Google’s new search system threatens, as people create content only for machines to copy and condense into AI answers and generating economic rewards that never reach the original authors.

Google could have addressed that by letting websites opt out of having their data used for AI while still appearing in search results. The company has since faced a reckoning on that decision. Prince says Cloudflare gave Google an ultimatum: From Sept. 15, the infrastructure firm will block mixed-purpose crawlers for its customers who are ad-supported — and do so by default.

Google could lose access to millions of websites using Cloudflare. Further pressure came from the UK’s antitrust watchdog. In June, the Competition and Markets Authority ordered Google to give website owners a clear choice to block their content from being used for AI products while remaining in search results. Prince told me there were "rumblings" that Google wouldn’t just follow that standard for the UK but apply it globally.

A Google spokesperson confirmed to me that the company was indeed testing a new setting that lets websites opt out of its AI answers without hurting their search rankings, and that it will roll that out worldwide once UK testing is done. But the company's crawlers remain technically bundled.

Websites could then block AI scraping themselves and at their own door rather than relying on Google's word — which, as the internal slides from 2024 showed, tends to bend when revenue is at stake.

Why it matters

The technique, which has largely flown under the radar, gives the company an unfair advantage over other AI firms while also accelerating the rise of automated traffic on the internet at the expense of humans. Google is now meeting some resistance from a plucky British regulator and a crusading industry player, which may finally bring some much-needed change.

Googlebot, the firm’s automated crawler, has for many years scoured the Internet and downloaded web pages to feed its massive index so that the freshest and most relevant pages could appear on Google’s search results.

More recently, though, the line between content collected for search and content used to generate AI answers has become blurred, as the bot is used to help feed both search and Google’s flagship AI model, Gemini. ’s, according to Matthew Prince, chief executive officer of web security company Cloudflare.

Prince’s firm is a traffic gatekeeper for about a fifth of the web, handling traffic and blocking attacks from bad actors, and he’s become a vocal critic of Google for bundling its search and scraping systems. He says the strategy makes it difficult for publishers to block AI training without losing the search traffic they rely on. That advantage also has broader consequences.

AI agents generated more than 57 per cent of web traffic in 2026 versus about 42 per cent for humans, according to Cloudflare, marking the first time in history that machine activity surpassed humans on the Internet. Prince sees that divergence getting more extreme in the next few years, a trend that echoes the so-called dead internet theory, the idea that machines increasingly create and consume online content.

“Humans will be a rounding error on the internet versus what their agents and their bots are doing,” he tells me. Agents are still largely the preserve of software developers and companies; but Mark Zuckerberg said last week that Meta Platforms Inc.

What to watch

The same bot still collects a site’s content for both search and AI, and websites have to trust Google to respect their wishes. Fully separating Google's crawler into two — indexing for search, and AI scraping — would be a better remedy.