Blog Profitable Founder
Guide

Cloudflare's Pay Per Crawl: 3 Agent Internet Businesses to Start Now

Greg Isenberg says Cloudflare's pay-per-crawl will mint 1,000+ AI millionaires. The 3 agent internet business ideas from the episode, with real numbers.

Greg Isenberg thinks Cloudflare just quietly launched the next gold rush.

Not crypto. Not another model release. A payment layer that lets AI agents pay fractions of a cent to read your content, query your data, or call your tools. He put it bluntly in the title: "Cloudflare will make 1000+ AI millionaires."

Big claim. But the episode is one of the clearest breakdowns I've seen of where agent internet business ideas are heading, and he backs it up with three specific businesses you could start this month, with the exact wedge, first customer, and pricing for each.

I watched the whole thing so you don't have to. Here's the full breakdown, with my take as someone who publishes three articles a day and watches AI crawlers eat them for free.

The old bargain of the internet is dead

Greg starts with what he calls the old bargain.

You let Google crawl your site. Google sends you traffic. A human lands on your page, and you monetize the attention: ads, email capture, a subscription, an affiliate link.

That trade funded a massive part of the internet for 20 years.

AI agents broke it. An agent reads your page, pulls the answer, hands it to the user. The content still created the value. The website lost the visit, the ad impression, the email, everything.

Publishers are furious about this, and I get it. I feel it on this blog. But Greg's point is that the publisher outrage is only the first inning.

The bigger shift: agents are starting to use the internet the way software uses infrastructure. They request things, call tools, compare products, retrieve data, take action. And that means the internet needs pricing for machine usage.

A human will never pay a third of a cent to read a recipe. (Greg's line: he'd close the laptop and order tacos out of spite.) A machine doesn't care. If the data helps the agent finish the job, the agent pays, automatically, in microtransactions.

→ The human web monetized attention. The agent web monetizes useful resources.

What Cloudflare actually shipped, in plain English

Three pieces, stacked on top of each other:

1. AI Crawl Control. See which AI crawlers hit your site, allow some, block others. Visibility first.

2. Pay Per Crawl. The monetization piece. You can now charge AI crawlers to access your content. The crawler either sends payment intent with the request, or gets an HTTP 402 "payment required" response with your price, pays, and retries.

3. The monetization gateway. The bigger version. Not just pages: any resource behind Cloudflare can carry payment rules. A dataset charging per lookup. An API charging per successful call. An MCP tool call. A search index. A file.

The payment rail is x402, built on that 402 status code that's been sitting unused in the HTTP spec for decades. Cloudflare verifies payment at the edge before the request even hits your server.

There's no checkout flow and no account creation. The request itself is the transaction.

Greg's summary: Cloudflare is trying to make paid access feel like part of the internet itself. Agents get wallets, and every useful resource on the web becomes a tiny toll booth you can own.

He has no affiliation with Cloudflare, for the record. He's just early on the implication: thousands of small, profitable businesses built on tiny paid doors that agents walk through all day.

Idea 1: The niche data refinery

This is the one Greg would start with, and honestly, me too.

Pick one niche where valuable information is messy, fragmented, changing, and annoying to collect. Then refine it into what he calls "clean fuel for agents."

His example: med spas. (He lives in Miami. There are a lot of med spas.)

A med spa owner wants to know what competitors charge, which treatments are trending, what local reviews complain about, who's hiring injectors (which means they're expanding capacity). That data exists today, scattered across Google reviews, Instagram, job posts, ad libraries, and pricing pages.

An agent with that data cleaned up could tell an owner: "Your Botox pricing is above the local median, but your reviews don't support premium positioning yet." That's a real insight someone pays for.

The wedge Greg lays out:

→ One niche, one city. Track 100 businesses, not more.
→ Do it manually first. A spreadsheet: services, prices, review counts, top complaints, hiring signals, booking flow.
→ Build 10 outputs from it: a local pricing map, a competitor gap report, a review complaint summary, a monthly market movement report.

And the part most people miss: your first customer probably isn't the med spa owner.

It's the person already selling into the niche. The marketing agency charging spas $5K a month for a growth package will happily pay you $300 to $800 a month if your data helps them close one more client. Same story for consultants and the folks building business automations for local clients: they all need clean market data to make their own work better, and none of them want to collect it.

Then you climb the ladder: report, dashboard, API, MCP tool. And when the x402 rails mature, agents pay you per lookup while you sleep.

The filter for picking your niche, straight from the episode: the data must be valuable (drives money decisions), repeated (needed again and again), changing (freshness matters), fragmented (one person can't easily collect it), and annoying (that's your margin).

Works for roofing (storm events, permits, insurance signals), real estate investing (zoning, rent comps, tax delinquencies), e-commerce (competitor SKUs, review complaints, influencer rates), law firms. Pick the one where you have an unfair advantage.

Idea 2: Agent readiness audits (SEO for the agent internet)

Think about how a B2B SaaS buyer used to work: land on the homepage, click around, read pricing, book a demo, ask a friend.

Agents compress all of that. Someone asks their assistant "find me the best payroll provider for a 15-person company in California," and the agent needs to answer: who is this for, what does it cost, what does it replace, what are the risks, how does it compare.

Most websites make this nearly impossible. Hidden pricing. Docs buried in PDFs from 2002. "Unlocking operational excellence" copywriting that says nothing.

The business: make companies easy for agents to understand, trust, compare, and recommend.

Greg's wedge is a paid audit, and the sales motion is brutal in the best way:

→ Pick one vertical. Run 20 to 50 buyer-intent prompts across the major AI tools.
→ Screenshot the answers.
→ Show the founder: "When buyers ask AI about your category, you don't show up. And when you do, AI says you cost $8 a month. Your actual price is $20."

You're not selling the future. You're selling the screenshot.

The fix you deliver: a clean llms.txt file, a pricing page agents can parse, honest comparison pages, structured FAQs built from real buyer questions, schema markup, a changelog, maybe a lightweight MCP server if they have enough useful data.

Pricing from the episode: $3,000 to $10,000 for the audit and cleanup, $10,000 to $20,000 for bigger B2B companies. Then a monthly retainer to rerun the prompts and track whether the AI answers improve.

After 10 clients in the same niche you'll see the same missing docs and the same unclear pricing pages every time. That's when you productize into software. Cash flow from day one, software valuation later.

This one pairs well with what I covered in my breakdown on selling to AI agents: the companies that win the next five years are the ones agents can actually read.

Idea 3: Turn expert archives into agent tools

This one's for creators, analysts, and consultants sitting on years of content.

A creator with 300 sales videos is monetizing them with pre-roll ads on 7-year-old interviews. Greg's pitch: turn that archive into a tool agents can use.

Not "chat with an expert." That's too broad, and every version of it has flopped. One archive, one painful job:

→ 300 sales videos become a cold email critique tool that scores your draft, rewrites it using the expert's frameworks, and cites the source lessons.
→ 500 startup podcast episodes become an idea feedback tool that gives you the wedge, the customer, and what to validate this week.
→ A decade of design teardowns becomes a landing page critique tool.

The part everyone skips is the tagging. Don't dump everything into a vector database and call it a day (Greg's words: that gives you "a search box with confidence"). Tag a sales archive by prospecting, subject lines, objections, follow-up, close. The structure is what people actually pay for.

The creator already has the distribution and the trust, so customer acquisition is solved. Charge $19 to $50 a month to their audience, bundle it into a paid community, or license it to agencies.

And when agent payments go live, the creator gets paid every time their knowledge gets used. Way better than hoping someone sits through an ad.

Greg is doing this himself with Ideabrowser, where the MCP integration is one of the most used features. He's practicing what he's preaching.

My take: the window is 18 months

There's a moment in the episode where Greg addresses the doom on X. Levels.io talking about traffic and revenue dropping across indie projects, AI overviews eating clicks, people vibe-coding their own tools instead of buying yours.

Greg's answer: the world is moving, move with it. Stop building little tools that can be vibe-coded in an afternoon, start building things with actual moats: proprietary data, structure, trust.

His analogy is the one that stuck with me. This is like building an app when the App Store opened in 2009. The opportunities look small right now because the agent internet is small. It won't stay small.

All three ideas share one question, and it's the one Greg says to ask yourself before building anything here: what resource does an agent need badly enough, often enough, and reliably enough to pay for?

Notice something else: none of these require the Cloudflare rails to be mature. You sell the human version now (reports, audits, tools), build the data asset, and you're already standing there when agents start paying per request. That's the same crawl-walk-run logic behind the tiny AI arbitrage businesses I broke down from Greg's earlier episode.

The founders I interview keep proving the same thing: the boring, specific plays win when you start them before everyone else sees them. This is one of those.

FAQ

What is Cloudflare's Pay Per Crawl?

Pay Per Crawl lets website owners charge AI crawlers for accessing their content. The crawler either includes payment intent in its request or receives an HTTP 402 "payment required" response with the price, pays (usually a fraction of a cent), and retries. Cloudflare verifies the payment at the edge before the request reaches your server.

What is x402?

x402 is the payment protocol behind this shift. It uses the long-dormant HTTP 402 status code to turn any web request into a transaction: an agent requests a resource, the server responds with a price, the agent pays and retries with proof of payment. No accounts, no checkout flows, no invoices.

Do AI agents actually pay for content yet?

Barely. The rails are live but adoption is early, which is exactly Greg's point. His advice is to build the manual version of these businesses now (reports, audits, tools humans pay for), so your data asset and distribution already exist when agent payments become normal over the next 12 to 18 months.

Which agent internet business should I start first?

The niche data refinery is the most practical starting point. It needs no code on day one (a spreadsheet tracking 100 businesses in one city), it has a clear first customer (agencies and consultants already selling into the niche at $300 to $800 a month), and every later stage, from dashboard to API to MCP tool, builds on the same asset.

Steal playbooks like this every week

I break down stories like this on the Profitable Founder Podcast: bootstrapped founders sharing the real numbers behind how they grew, including the parts that went wrong.

If you're building toward $100K MRR, it's the closest thing to sitting in on the conversations I wish I'd had at $15K a month.

Listen to the podcast →

Florian Darroman, founder of Distribb and host of Profitable Founder
About the author

Florian Darroman

Florian Darroman is a French distribution guy based in Bali, founder of Distribb and host of Profitable Founder. He interviews bootstrapped founders making $100K-$10M/year and documents the journey of growing Distribb to $100K MRR.

Experience: affiliate SEO to 6 figures, infoproducts to 7 figures, and built and sold Les Makers for $130K.

Read more in Guide

Keep reading

Building a SaaS toward $100K MRR?

Profitable Founder Club is a mastermind for founders doing $5K–$50K MRR. Bi-weekly calls, monthly Q&As with founders past $100K MRR.

Join the Club