A working demonstration of pay-per-crawl content monetization went live in August, showing that AI agents can request a page, check its price, pay one cent in USDC and receive the content — all without an account, API key or human intervention.

The demo, built by Suganthan and running at paid.suganthan.com since August 8, mirrors infrastructure Cloudflare is building and taps the open x402 protocol, which puts a practical purpose behind HTTP's long-dormant 402 Payment Required status code, reserved in the specification since 1997.

The timing matters for publishers. Google confirmed to Digiday that it is running an "AI contribution pilot," paying publishers when their content significantly contributes to answers generated in AI Overviews, AI Mode or the Gemini app. Barry Schwartz covered the program at Search Engine Roundtable on September 14, after finding the relevant help page back in April. Publishers who join see a monthly earnings figure in Search Console — but no breakdown of how Google calculated it. One publishing executive familiar with the program described it as a black box. Digiday reported that Google has approached at least dozens of publishers, with smaller publishers showing more interest than larger ones.

How the payment flow works

The x402 exchange runs in four steps. The agent requests a page; the server responds with HTTP 402 and a PAYMENT-REQUIRED header describing the price and payment terms. The agent signs an authorization to transfer the exact amount in USDC and retries the request with a PAYMENT-SIGNATURE header. A payment facilitator — Coinbase's facilitator at x402.org in this setup — verifies the signature and settles the transaction on the blockchain. The server returns the content with a settlement receipt in the PAYMENT-RESPONSE header.

The demo site runs on a Cloudflare Worker, roughly 300 lines of Hono code plus the x402 middleware. It has three routes: a free landing page, a research article at /research/agentic-seo/ that charges $0.01 per crawl, and a public /earnings/ page recording every paid crawl since launch, with timestamps, payer addresses and transaction hashes.

The offer asks for one cent in testnet USDC on Base Sepolia. USDC uses six decimal places, so the amount field reads 10000, representing 0.01 USDC.

On the morning of September 15, five payments went through, each settled on-chain with a public transaction hash. One payment came from Claude Code running on the author's own machine, using a hook that watches WebFetch calls: when Claude hits a 402, the hook fetches the payment offer, checks it against spending caps, pays and passes the content back as context.

Caveats publishers should note

All payments so far came from the author's own agents. No search crawler has paid for access, and the tokens are testnet USDC with no real monetary value. The buyer script also runs budget guardrails — a $0.05 per-call cap and a $0.25 daily allowance tracked in an on-disk ledger — checked before any payment signature is created. The buyer wallet held zero ETH, yet every settlement went through because the facilitator covered gas fees.

Transaction volume claims in this market deserve scrutiny. Decrypt reported on September 13 that TRM Labs examined 198.9 million x402 settlements and found most payment volume did not come from AI agents. Scheduled scripts, load tests and self-directed transfers produce similar blockchain records, and TRM's methods cannot conclusively distinguish an agent from a script.

The broader monetization picture

Publishers are responding to AI extraction in three broad ways. Smaller sites often block AI crawlers outright, accepting reduced visibility in training data and AI answers. Large publishers negotiate licensing deals — News Corp, the Financial Times and Reddit all have agreements with AI companies. Press Gazette reported on September 4 that People Inc.'s chief executive told investors that blocking crawlers through Cloudflare had helped bring AI companies to the negotiating table.

Cloudflare told Press Gazette it is shifting its default from pay per crawl toward pay per use, running pilots with Ceramic.ai and You.com, on the reasoning that fetching a page doesn't prove the content was used. That distinction cuts both ways. Pay per use measures value more accurately, but the site owner cannot independently observe whether a page influenced an answer — the platform decides what happened and what it was worth. With pay per crawl, the site owner can verify the request and the payment.

Cloudflare announced its Monetization Gateway on July 1, still waitlist-only. During Agents Week on August 4 it announced Wallets — Account Wallets for people who fund and set limits, Virtual Wallets for agents that spend within them. Reserved handles cannot yet send, receive or hold funds, according to Cloudflare's documentation. Its existing Pay Per Crawl product remains in closed beta, requiring Web Bot Auth verification, Stripe onboarding and program approval.

The x402 protocol has heavyweight backing. The x402 Foundation became operational under the Linux Foundation on July 14 with 40 members, including Visa, Mastercard, Google, AWS, Stripe and Cloudflare. AWS made AgentCore payments generally available on August 18.

What site owners should do now

The author's advice: don't start charging this month expecting GPTBot, ClaudeBot, PerplexityBot or Googlebot to arrive with a wallet. Until crawlers support payment, requiring it simply blocks them, potentially costing visibility in AI answers with nothing earned in return.

Not every page is a candidate. General explainers may retain value as free discovery assets; original datasets, research and tools are stronger candidates for per-request pricing. And if Google offers a monthly pilot payment, ask how it was calculated — one publishing executive argued that accepting early terms could weaken publishers' position in future negotiations.

Source: Search Engine Journal (https://www.searchenginejournal.com/i-made-my-website-charge-ai-agents-a-penny-per-page-then-i-watched-claude-pay-it/589447/) / Source: suganthan.com (https://suganthan.com/blog/how-chatgpt-picks-sources-part-2/)