Guestlist Only

What Cloudflare’s AI move means for B2B Marketers.

When I imagined the big players in the AI content war a couple of years ago, I envisaged battles between Google and OpenAI. Maybe a few attorney-generals.

Then Cloudflare stood up and said “now I run the internet”.

The developing situation is this…

• Google’s AI Overviews have been increasingly stealing your organic search clicks, a ~30-50% CTR loss when appearing.
  
• Publishers are now reporting 30-60% organic traffic loss this year too.
  
• AI chatbots and answer engines are increasingly bypassing websites altogether. A client saw a >50% drop after answer engines/chatbots virtually obsoleted their category of informational queries entirely. Years of content value gone in a few weeks.
  
• Now Cloudflare, which sits in front of ~20% of the internet, has started blocking most AI crawlers by default.

I’ve had several conversations with marketers recently (mostly B2B) who are trying to get a handle on this and what it means for them. 

First question is often, “WTF *is* Cloudflare?” 

Well, among other things, a CDN (content delivery network) + WAF (Web Application Firewall) which sits in front of your site to help keep nasties out. 

If your website was a nightclub, Cloudflare would be the bouncer. Juiced on venture capital and with fewer tattoos (probably). 

We’ve used Cloudflare on the marketing sites we manage, and have done for years. For all the traditional reasons; security, speed, uptime, etc.

Alas, if it’s not on your site, you’ll have been routed through it around the web many, many times.

Now, this bouncer has decided that Perplexity, Claude, and others are banned from your club. Unless they’re on the guestlist.

That’s due to the likes of OpenAI sending 750× more crawlers than real actual human users. And some AI crawlers have been… let’s say creative… in getting around existing blocks. Climbing in through the toilet windows, to keep the analogy going. Hey Perplexity! 

Cloudflare also knows visibility in AI assistants is becoming as critical as ranking in Google is was for marketing goals. 

Yet their PR play is all around ‘protecting publishers’ and saving the internet. Sure thing… 

Let’s also remember that every crawler request consumes Cloudflare’s bandwidth, processing power, and security resources. 

Fair dinkum, they don’t want to lose out in the AI space race either. 

As such, a lot of the kerfuffling is also about infrastructure cost and new revenue streams and slices of cake(s). And yes, they already have a pay per crawl service in beta.

On the other hand, the AI firms see it as a power move that risks blocking legit, user-triggered assistants. We don’t know what their practical response will entail, but you can bet your bottom dollar that there will most certainly be one.

Cloudflare’s effectively asking you, website owner, to opt in if you’d like AI to scrape your content. And then seeks to charge AI firms crawls  – on your behalf, they take a clip and process the payments – for providing the transport. 

Pretty much Deliveroo for AI bots.

So, your door policy requires a decision.

Depending on your situation, it might be a tricky one too.

Because first-party research, publishing guides, and lifting the gate on… well, gated content… has been the advice for gaining ground with search and social in recent times.

And generally speaking, B2B content is pretty valuable. It often deals with niche and important things, and does so for intelligent people.

Blocking everything is by design a self-limiting move. Handing over all your hard-earned content without a plan might be a missed opportunity though.

So surely the play for the content rich is to figure out how to show up in AI-powered answers where buyers are already, without giving away the farm. 

Thinking of AI crawler access as you would any other distribution channel:

• Let in only the bots that earn their keep.
• Structure open content so your brand gets credited.
• Keep the high-value stuff like first-party research gated to build your own audience.
• Track whether AI visibility is actually driving results, the same way you would with any other channel.

So will your bouncer work the door, run a VIP list, or just let everyone in?