Cloudflare introduces tools to block AI scrapers while maintaining search visibility

Cloudflare has launched a new feature designed to help website owners manage how their content is accessed by AI bots. As generative AI models increasingly rely on web scraping to train their systems, many site operators face the dilemma of wanting to remain discoverable in search engines while preventing their data from being used for AI training. Cloudflare’s new solution allows users to easily block AI scrapers with a single click, while simultaneously ensuring that legitimate search engine crawlers, such as Googlebot or Bingbot, can still index their pages. This approach provides site owners with greater control over their intellectual property and data usage. By distinguishing between beneficial search traffic and unauthorized AI harvesting, Cloudflare aims to balance the needs of the open web with the growing demand for data privacy and content protection in the era of large language models.