From September 15, new domains will block training and agent bots by default on ad-bearing pages, with dual-purpose crawlers like Googlebot judged by the strictest rule.
Cloudflare has added three purpose-based groups for blocking AI bots, letting website operators allow or deny each kind of automated visitor on its own. The controls are available now to every current customer, including accounts on the Free tier.
The settings sit under the Security section of each zone. Cloudflare sorts bots by what they do: Search covers behavior that collects or indexes a site's content to answer questions about it later, with site owners expected to get referral traffic or other equitable compensation in return; Agent means automation working live to carry out a task for a person, such as chat-fetch bots like ChatGPT-User or Gemini and Claude driving a Chrome browser; and Training describes a crawler that takes content to train or fine-tune a model, permanently absorbing the data into it.
The defaults will shift on September 15, 2026. From that date, any new domain joining the platform will have the Training and Agent categories blocked by default on pages that carry ads, while Search stays permitted unless the owner changes it.
The same change catches crawlers that serve several purposes at once by mixing Search with Training. The company says these will be judged on the whole of their activity, governed by whichever rule is strictest — so bots such as Googlebot, Applebot and BingBot will be shut out for any customer who has chosen to block Training.
Cloudflare casts the move as a second Content Independence Day, roughly a year after it rolled out one-click AI-bot blocking and a Pay-Per-Crawl marketplace that lets operators earn payment when their material is crawled. The shift arrives as Google Search has stepped back from an AI-centered design and now works as a full answer engine, resolving a query directly on the results page.
Site owners who do not want the new defaults can decline them by adjusting an option in their Security dashboard at any point before September 15.
Cover image: “Lava lamp wall at Cloudflare office -1” by HaeB, Wikimedia Commons, CC BY-SA 4.0, resized, re-encoded.
