Google Gets Banned
Cloudflare announced that, starting September 15, it will default-block “mixed use” search crawlers that also feed AI training, potentially disrupting Google, Bing, and Apple because many websites rely on Cloudflare and can now prevent indexing bots from doubling as data scrapers.
MAIN POINTS FROM TRANSCRIPT
- Cloudflare will default-block mixed-use crawlers that both index sites and collect AI training data.
- Roughly a quarter of websites use Cloudflare, giving the policy broad reach across the web.
- Google is especially exposed because its search bots are central to indexing and search visibility.
- robots.txt can separate indexing from training for some bots, but enforcement has historically been voluntary.
TAKEAWAYS
- Search engines may lose access to fresh content if many sites choose to block their crawlers.
- The old bargain of free indexing in exchange for traffic is weakening as AI scraping becomes a concern.
- Infrastructure providers like Cloudflare can shift web policy at internet scale.
- Companies that reuse search bots for AI training may face growing resistance from publishers and site owners.