More: Alongside that change, Cloudflare has launched PayPerCrawl, a new marketplace that allows website owners to charge AI companies per page crawled. If you're running a blog, a digital magazine, a startup product page, or even a knowledge base, you now have the option to set a price for access. AI bots must identify themselves, send payment, and only then can they index your content.
The core issue behind this shift is how AI models are trained. Large language models like OpenAI's GPT or Anthropic's Claude rely on huge amounts of data from the open web. They scrape everything, including articles, FAQs, social posts, documentation, even Reddit threads, to get smarter. But while they benefit, the content creators see none of that upside.
Unlike traditional search engines that drive traffic back to the sites they crawl, generative AI tends to provide full answers directly to users, cutting creators out of the loop.
According to Cloudflare, the data is telling: OpenAI's crawl-to-referral ratio is around 1,700 to 1. Anthropic's is 73,000 to 1. Compare that to Google, which averages about 14 crawls per referral, and the imbalance becomes clear.
