No AI Crawler Pays Publishers Through 402 Yet. Build This Instead.
Direct Answer
No major AI crawler pays for content today. OpenAI, Anthropic, Perplexity and Google document robots.txt as the entire control surface, with no mention of payment or HTTP 402, and Google treats a 402 as content that does not exist. Build the agent-readable layer first, because that one already has consumers.
Key Takeaways
- Cloudflare's crawler-monetization stack is announced, not shipped. Pay per crawl has been in beta since July 2025, the Monetization Gateway is waitlist only, and a reserved Cloudflare Wallet handle cannot send, receive or hold funds. The two products available on all plans charge nobody anything.
- Nobody documents paying, on either side of the market. No major crawler operator's documentation mentions payment or HTTP 402, and across five consecutive weekly runs of our own readiness index, zero of 56 benchmarked organizations passed any of the five agentic commerce protocols.
- A 402 does not bill Googlebot, it deindexes you. Google documents that all 4xx errors except 429 mean the content does not exist, and that already-indexed URLs returning a 4xx are removed from the index. HTTP 402 is not listed as an exception anywhere.
- The core of x402 proves payment, not identity. It establishes that a key holder authorized a transfer and that it settled on chain. Binding that key to a named crawler requires an optional extension built on Web Bot Auth, which is the real prerequisite and is barely adopted.
- The agent-readable layer already has named consumers. Anthropic and OpenAI both document reading remote MCP servers from their own APIs, and Google documents fetching an A2A agent card at runtime. No operator documents consuming llms.txt, and Google states that Search ignores it.
Can You Actually Charge AI Crawlers for Your Content Today?
Not if your intended buyer is a major AI crawler. Every managed product that would let a website charge one is gated behind a closed beta, a waitlist, or a name reservation, and no major crawler operator documents paying a 402 challenge. You can stand up an x402 paywall yourself today, because the protocol is open and needs nobody's permission. What you cannot do is get a major crawler to pay it.
Cloudflare is the clearest case because it has announced the most and documents its own availability precisely. The table below is assembled entirely from Cloudflare's own product pages and developer documentation.
| Product | Announced | Status in September 2026 |
|---|---|---|
| Pay per crawl | July 1, 2025 | Closed beta, admission by form or account executive |
| AI Index | September 26, 2025 | Private beta, no product documentation published |
| Content Signals Policy | September 24, 2025 | Shipped, available on all plans |
| Monetization Gateway | July 1, 2026 | Waitlist only, no product documentation published |
| Pay Per Use | July 1, 2026 | Stated direction, no product and no documentation |
| Cloudflare Wallets | August 4, 2026 | Handle reservation only, cannot hold funds |
| AI Crawl Control | Formerly AI Audit | Generally available on all plans |
The pattern is the point. AI Crawl Control, which shows you which AI services access your content and lets you allow or block them, is generally available on all plans and works with zero configuration. The Content Signals Policy, which lets you state your terms inside robots.txt, is also shipped on all plans. Both are available on all plans, and neither collects a cent from anyone. Cloudflare, Cloudflare
Everything that would actually bill a crawler sits behind a gate. Pay per crawl entered private beta on announcement in July 2025 and its documentation still described a closed beta when last updated in July 2026. The Monetization Gateway, which would let you charge for any web page, dataset, API or MCP tool behind Cloudflare, opened a waitlist and has no product documentation. Cloudflare, Cloudflare
The wallet is the sharpest illustration. Cloudflare opened handle reservations during Agents Week in August 2026. Cloudflare's own documentation states plainly that a reserved handle does not yet let you send, receive, or hold funds. Reserving a name is not the same as having a wallet, and the distinction matters if you are planning revenue around it. Cloudflare
Do Any AI Crawlers Pay for Content Right Now?
No major AI crawler operator documents paying for content, and I found no public example of a publisher documenting revenue collected from one. The documented crawler-to-publisher market is effectively empty, which is checkable rather than inferred, because it rests on the operators' own published documentation.
Each of the four operators that matter most publishes crawler documentation, and in every case the entire control surface offered to a site owner is robots.txt and user-agent identification:
- OpenAI: documents GPTBot, OAI-SearchBot, OAI-AdsBot and ChatGPT-User, describes what each is used for, and offers robots.txt directives and IP ranges. Payment, compensation and HTTP 402 are not mentioned.
- Anthropic: documents ClaudeBot, Claude-User and Claude-SearchBot, and describes robots.txt opt-out plus crawler identification as the available controls. Payment is not mentioned.
- Perplexity: documents PerplexityBot and Perplexity-User, and notes that the user-initiated fetcher generally ignores robots.txt because a person requested it. Its crawler documentation does not mention payment, though the company runs a separate publisher programme that is not surfaced there.
- Google: documents its crawler fleet and its HTTP status code handling in detail, and mentions payment nowhere. It is the only one that documents what a 402 would actually do, covered in the next section.
Cloudflare, which built the marketplace, has not named a crawler that has paid through pay per crawl. Its bot reference has no payment or price column, its pay per crawl documentation addresses a prospective operator without naming any participant, and its one-year retrospective on the agentic internet publishes no revenue figure and no roster of paying crawlers. OpenAI, Anthropic, Perplexity
The obvious objection is Amazon Bedrock AgentCore payments, which reached general availability on August 18, 2026 and lets an agent discover, access and pay for APIs, MCP servers and content over x402 and MPP. It is real and it is shipped, but it is buyer-side tooling a developer wires into their own agent, not a crawler fleet. An agent paying for an API call it chose to make is a different economic event from a search or training crawler compensating a publisher whose pages it harvested at scale, and it is the second one that publishers are being told to price for. AWS
The supply side is just as thin, and here we can report our own measurements. The Agentic Web Readiness Index scans roughly 60 organizations working in agentic search and agentic commerce every week. Across five consecutive runs between August 22 and September 13, 2026, zero of the 56 benchmarked organizations passed any of the five agentic commerce protocols we test for, which are ACP, UCP, AP2, x402 and MPP. Not one company, not one protocol, not one week.
What Happens If You Return HTTP 402 to Googlebot?
Google stops using the URL, and a page already indexed is dropped from the index. Google documents that all 4xx errors except 429 are treated identically, with the crawler informing the next processing system that the content does not exist, that its systems stop using such a URL over time, and that URLs already indexed which return a 4xx are removed from the index. HTTP 402 is not listed as an exception anywhere.
This is the single most important fact for anyone considering a paywall for agents, because it converts an abstract tradeoff into a documented consequence. Serving a 402 to Googlebot does not present an invoice. It tells Google the page does not exist, and Google acts on that over time.
The reason no crawler knows what to do with a 402 is that the specification has never told anyone. Three details are worth holding together:
- The status code says nothing: RFC 9110 section 15.5.3 defines 402 in a single sentence, that the status code is reserved for future use. There is no required header, no payment format, and no client behaviour.
- It has said nothing since January 1997: 402 first appeared in RFC 2068 with the words that this code is reserved for future use, language carried forward verbatim into RFC 2616 and then reworded, but unchanged in substance, in RFC 7231 and the current specification.
- Everything else is a convention on top: x402 gives 402 a practical meaning, but that meaning lives in a separate specification that a crawler has to have implemented. A crawler that has not implemented it sees an unexplained client error.
So the risk of gating early is asymmetric in a way that is easy to underestimate. If you are right about the timing you collect fractions of a cent, and if you are early you lose the indexing and citation surface that currently sends you whatever value you do get. Google's own status code documentation is the place to check this before shipping, not after. Google Search Central, RFC 9110
Why Is Cloudflare Moving From Pay Per Crawl to Pay Per Use?
Because a fetch is not usage, and Cloudflare says so in its own words. Its July 2026 press release states that it is evolving Pay Per Crawl into Pay Per Use so publishers are paid when their content actually creates value, not just when it is fetched. The company that built per-crawl billing moved off it within a year.
The reasoning Cloudflare publishes is the clearest statement of the problem available: a single page might be crawled once and then cited in thousands of answers, or crawled over and over and never used at all. Cloudflare data suggests that over 50 percent of crawl traffic from AI crawlers is spent re-fetching unchanged pages, which means per-crawl pricing bills most heavily for the least valuable requests. Cloudflare, Cloudflare
That correction creates a different problem, and both models have an unresolved side:
- Pay per crawl is verifiable but measures the wrong thing: you can see the request and the settlement yourself, and neither tells you whether the content influenced an answer. It prices your work by the byte.
- Pay per use measures the right thing but you cannot audit it: once your page leaves your server you cannot observe whether it shaped a response, so you are relying on the platform to report what happened and what it was worth.
- The opaque end of that spectrum already exists: Google has been reported to pay some publishers when their content contributes to AI answers, and has published no public documentation of the programme. No publicly accessible Google page names it, defines eligibility, or explains the calculation.
What Google does document is instructive by contrast. It publishes a control for whether a site appears in and helps ground its AI features, and performance reports showing impressions from generative AI surfaces, though not clicks. Measurement it will document, if only partially. Payment it will not, at least not publicly, which is the structural weakness of every platform-reported model. The distinction worth holding is that paying for downstream contribution to an answer is licensing by another name, settled after the fact on the platform's own measurement. It is not Googlebot satisfying a payment challenge at request time, and the two models leave a publisher in very different positions. Google Search Console Help, Google Search Console Help
Why Does x402 Alone Not Tell You Who Is Paying?
Because identity lives in an optional extension rather than in the core protocol. The core x402 specification establishes that a payment was authorized by the holder of a private key and settled on a blockchain, and nothing more. Binding that key to a named crawler operator is possible, but only through an extension both sides must implement.
The mechanics are worth understanding because they explain both the appeal and the gap. In version 2 of the specification, a server answers with a PAYMENT-REQUIRED header describing price and terms, the client retries with a PAYMENT-SIGNATURE header carrying a signed authorization, and a facilitator verifies and settles before the server returns the content with a PAYMENT-RESPONSE header. Under the exact scheme on EVM chains, the facilitator pays the gas while the client controls the flow of funds through its signature. The specification reserves the phrase truly gasless for EIP-3009 tokens such as USDC, since its Permit2 fallback has the payer cover a one-time approval unless the facilitator sponsors it. x402 specification, x402 specification
What none of that provides is an answer to who is on the other end:
- The signature proves key control, not identity: the authorization carries a payer address, and nothing in the core payload binds that address to a company, a person, or an agent rather than a script.
- Authentication is explicitly optional: the core specification offers integration with sign-in systems as a way to give verified users different pricing, which is a feature for discounts rather than a requirement for identity.
- Identity exists, as an extension nobody is obliged to use: the http-message-signatures extension establishes the identity of the paying agent through RFC 9421 signatures, carries a web-bot-auth tag, and names Cloudflare as an implementing network, pointing at Cloudflare's own pay per crawl crawler-verification documentation.
- The data reflects the gap: TRM Labs examined roughly 198.9 million settlement transactions worth 52.7 million dollars through known x402 facilitators on Base, Solana and Polygon since May 2025, screened that down to 25.62 million dollars of plausible commerce, and put the share that appears agentic at between 0.6 and 7.5 percent. Its stated reason for the wide band is that agency cannot be read off a transaction, because a scheduled script and an agent produce the same record. TRM adds that its strict test may understate the space, because a single-purpose agent paying one service repeatedly reads as a script.
The missing prerequisite is cryptographic crawler identity, and its status explains the empty market. Web Bot Auth, which signs requests so a server can verify which crawler is asking, is deployed at Cloudflare but still rests on individual IETF drafts rather than a working group document. Cloudflare's pay per crawl requires a paying crawler to implement it, to pass a manual verified-bot approval, and to complete Stripe onboarding, three prerequisites that together amount to an enterprise procurement flow standing between an agent and a penny. TRM Labs, Cloudflare
What Does the Agent-Readable Layer Have That the Payment Layer Does Not?
Named consumers in first-party vendor documentation. The difference between the two layers is not maturity or elegance, it is whether anything on the other end reads what you publish. For the agent-readable surfaces, the vendors document reading them in their own API references. For the payment surface, no crawler operator documents anything.
The table below pairs each surface with the vendor documentation that describes consuming it.
| Agent surface | Documented consumer | Where it is documented |
|---|---|---|
| Remote MCP server | Claude connects to remote MCP servers directly from the Messages API, no separate client needed (beta) | Anthropic platform documentation |
| Remote MCP server | The Responses API accepts an mcp tool pointed at any public MCP server | OpenAI API documentation |
| A2A agent card | The Agent Development Kit resolves an agent card over HTTP at runtime | Google ADK documentation |
| A2A agent card | The GKE ingestion controller queries the card to populate agent skills | Google Cloud documentation |
| llms.txt | None. Google states that Google Search ignores these files | Google Search Central |
| HTTP 402 payment | None. No major crawler operator documents presenting payment | OpenAI, Anthropic, Perplexity, Google bot docs |
The contrast with llms.txt is the useful one, because it is the deliverable most commonly sold as agent readiness. No major operator documents consuming it, and Google addresses it directly, stating that you do not need to create machine readable files, AI text files, markup or Markdown to appear in Google Search, and that Google Search ignores them. All four operators publish an llms.txt for their own developer documentation, which is the opposite direction and is routinely misread as endorsement. Google Search Central
Our own index data shows how few organizations have acted on any of this, and the split is revealing. The cheap signals are nearly universal while the surfaces that make a site genuinely operable are almost absent, measured across the 56 benchmarked organizations in the September 13, 2026 run:
- Widely adopted and inherited: robots.txt at 98 percent, a sitemap at 96 percent, and named AI bot rules at 95 percent. These mostly arrive with a platform rather than a decision.
- Adopted by a minority: Link headers at 36 percent, OAuth discovery at 23 percent, Markdown negotiation at 20 percent, and Content-Signal at 11 percent.
- Effectively absent: an API catalogue, an MCP server card and a WebMCP surface each at 3.6 percent, an A2A agent card and an ARD catalogue each at 1.8 percent.
- The median tells the story: the typical organization in this field passes 4 of the 16 readiness checks we score, and these are companies whose own positioning is agentic search or agentic commerce.
That is the opportunity, stated plainly. The layer with documented consumers is the layer almost nobody has built, which means the work is available now and pays off against systems that exist today rather than against a market that has not opened. Anthropic, OpenAI
The sequence below is deliberate, and it works because it front-loads everything with a consumer today and defers everything without one. If the rails arrive on schedule you are a configuration change away from charging, and if they slip another year you have spent the time making your site usable by the agents already reading it. Either way the work is not wasted, which is not true of a 402 shipped this quarter.
How to Sequence Agent Readiness Before Agent Payments
A five-step order of work that makes a site usable by the agents reading it today, while keeping the option to charge for access as a configuration change rather than a rebuild. Each step is useful on its own, none of it depends on a payment rail existing, and the last one keeps the option open without paying for it early.
Declare your terms in robots.txt
Add Content-Signal directives stating your position on search, ai-input and ai-train, and set the use directive. This is a stated reservation of rights that costs nothing and blocks no traffic, and it is shipped on all Cloudflare plans today.
Make your content retrievable
Serve clean Markdown twins for your key pages under content negotiation, keep a current sitemap, and publish Link headers. An agent that cannot parse your page will not pay for it either.
Expose a real interface
Stand up a remote MCP server over Streamable HTTP for the data an agent actually needs, and publish an api-catalog under RFC 9727 plus an OpenAPI description. Both Anthropic and OpenAI document reading remote MCP servers from their APIs, so this is the surface with shipping first-party clients already capable of consuming it.
Publish identity and authorization discovery
Add an A2A agent card at the well-known path, an MCP server card, and OAuth discovery with protected resource metadata under RFC 9728. Adopt Web Bot Auth verification when you need to know which crawler is asking, since that is the prerequisite any charging model will require.
Keep payment as a switch, not a wall
Put the pricing decision in configuration so a route can start charging in one change, and decide now which resources would ever be paid. Original datasets, research and tools are the candidates. Explainers that earn citations should stay open.
Frequently Asked Questions
- Can I charge AI crawlers for my content right now?
- In practice no. Cloudflare's pay per crawl remains in closed beta more than a year after launch, the Monetization Gateway is waitlist only, and no major crawler operator documents presenting payment. You can build the capability, but there is currently no documented buyer on the other end.
- What happens if my site returns HTTP 402 to Googlebot?
- Your pages get deindexed. Google documents that all 4xx status codes except 429 are treated the same way, with crawlers informing the next system that the content does not exist, and that already-indexed URLs returning a 4xx are removed from the index. 402 is not an exception.
- Is Cloudflare Pay Per Crawl available yet?
- Not generally. It was announced on July 1, 2025 in private beta, and its developer documentation still described a closed beta when last updated in July 2026. Admission requires a signup form or an enterprise account executive, and a paying crawler needs Web Bot Auth, verified-bot approval and Stripe onboarding.
- Do AI crawlers support x402 payments?
- No major crawler operator documents doing so. TRM Labs analysed 198.9 million x402 settlement transactions and estimated that between 0.6 and 7.5 percent of screened commerce volume appears agentic, noting that a scheduled script and an autonomous agent produce an identical on-chain record.
- Is llms.txt worth creating in 2026?
- There is no documented consumer. Google states that you do not need to create AI text files or Markdown to appear in Google Search and that Google Search ignores them, and OpenAI, Anthropic and Perplexity document robots.txt rather than llms.txt. All four publish one for their own docs, which is the reverse direction.
- Should I block AI crawlers instead of charging them?
- Blocking earns nothing and reduces your presence in AI answers and citations. A more useful middle path is to declare your terms with Content-Signal directives in robots.txt, which states a reservation of rights without refusing traffic, then make the content you want cited genuinely easy for agents to retrieve.
The Toll Booth Can Wait. The Front Door Cannot.
Charging agents for access is a switch you can flip later, but only if an agent can read, authenticate against, and transact with your site in the first place. That readiness is the part with consumers shipping today. The A.G.E.N.T.I.C. Framework scores it across every agentic surface, from discovery files and MCP integration through agent-completed transactions.
Ready to find out where you stand? Book your free Agentic Web audit
About This Article and Author
Authored by Shahzad Safri, Founder and Agentic Web expert at agenticplug.ai, combining primary sources from Cloudflare, Google Search Central, the x402 specification, the Linux Foundation, TRM Labs, Anthropic, OpenAI, Perplexity, the IETF, and agenticplug.ai's own Agentic Web Readiness Index.