Cloudflare CEO Matthew Prince gave an example: If you ask an AI agent to recommend a restaurant, it may visit the websites of 1,000 restaurants and read 1,000 menus, but in the end you only go to one of them.

What about the remaining 999? Their server still processed the request, the bandwidth was still used, but not a single customer was waited for.

What he wants to say is that after the intelligent agent replaces people to browse the web, the cost and benefit of the website are decoupled. In the past, traffic was roughly equal to potential customers, and server fees were considered customer acquisition costs. Now the traffic is up, but the people are not coming.

News and content stations are more direct. After the agent reads the article and generates a summary, the user gets the answer in the AI ​​interface. The website has neither advertisement display nor subscription conversion, leaving only a bandwidth bill.

Prince gave two paths.

The first is to make crawling less wasteful. Cloudflare is cooperating with OpenAI to provide prompts only when the content of the web page changes. There is no need to repeatedly crawl the pages that have not changed. This saves both parties money.

The second item is charges. To read website content, the AI ​​service pays the website. Websites can set strategies based on scenarios: the agent that helps users search for information can be set to provide content only after paying; the agent that helps users place orders and book reservations is a door-to-door business, so just let them go. Charging based on the number of crawls and usage is already Cloudflare’s clear product direction.

You have to be careful here, Cloudflare itself sells this set of tools, and it has every incentive to make the problem more serious. But the question itself is true. Anyone who has worked on a website will know if they look at the logs of the past six months that the number of requests from various AI crawlers and agents has increased much faster than human visits.

There is not much that small webmasters can do: count crawler traffic separately, and first find out how much they pay for them in a month.