Publishers Push Back Against Google's AI Crawling

Publishers accuse Google of unfair AI crawling

At the Fortune Brainstorm Tech conference this week, Neil Vogel, CEO of People, Inc., publicly accused Google of acting as a "bad actor" by using a single web crawler both to index sites for Search and to gather content for its AI features.

Vogel said the practice has materially changed referral economics: Google Search previously drove as much as 65% (and at times nearly 90%) of People Inc.'s open-web traffic; today that share sits in the high 20s. His point: the same crawler that still brings search clicks is also being used to feed AI products that repurpose publisher content.

That dynamic creates a dilemma. Publishers want to block AI-only crawlers to force licensing conversations, but Vogel says Google cannot practically be blocked without cutting off the remaining search traffic because its crawler isn't split. "They know this, and they're not splitting their crawler," he said.

People Inc. has adopted Cloudflare's crawler-blocking tools to deny non-paying AI crawlers. Vogel said this approach prompted outreach from "large LLM providers" and advanced content negotiations — though no new deals were finalized at the time of his remarks. He noted People Inc. already has a relationship with OpenAI, which he called a "good actor."

Other media leaders on the panel echoed the frustration. Janice Min of Ankler Media described Big Tech as longtime "content kleptomaniacs," and said her company also blocks AI crawlers. The debate is no longer just technical; it's commercial and political.

Cloudflare CEO Matthew Prince, whose company builds the blocking tools, offered two counterpoints. First, he questioned whether copyright litigation — built for the pre-AI era — is the right lever. Second, he predicted market shifts and regulations could push platforms to pay creators for crawl-and-model usage within a year.

The panel highlighted a practical toolkit publishers are already testing and the trade-offs involved: block crawlers to gain negotiating leverage, accept the loss of some search traffic, or pursue legal and regulatory remedies. Each path carries cost, timing, and strategic implications.

What this means for the media-AI ecosystem:

• Publishers gain new leverage when they can technically deny non-consensual data collection.
• AI companies face growing pressure to negotiate licenses or build compliant crawling practices to avoid being blocked or regulated.
• Search economics will continue to shift as platforms balance indexed traffic with the commercial value of content for models.

For publishers and content owners, the immediate choices are tactical and strategic: implement selective crawler blocks, diversify referral channels, build direct user relationships, and enter negotiations armed with data about usage and value. Those who act deliberately can convert the current pressure into revenue and control.

QuarkyByte's approach focuses on measurable decisions: model how crawl-blocking affects search referrals and revenue, quantify negotiating leverage against AI providers, and design evidence-based policies that protect content value while preserving audience growth. For media companies, government stakeholders, and tech leaders, that means turning technical controls into clear commercial outcomes.

As the dust settles between publishers and AI platforms, expect a mix of market deals, product changes at major tech firms, and regulatory attention. The short-term battleground will be crawlers and contracts; the long-term fight will be over how value from trained models is shared.