Has Cloudflare changed the internet?

Whatever your stance on AI and the fair use of original content, I think it’s fair to say that Cloudflare’s July update is making waves in the digital world.

The Shifting Landscape of Online Content

Since the dawn of the search engine, websites have, in one way or another, allowed their pages to be crawled and indexed. In return, search engines promote these sites, generating valuable traffic. Websites offering better content and user experiences were typically rewarded as users discovered new sites and relevant information.

However, this model has been significantly disrupted by the influx of AI agents. These crawlers have been collecting vast amounts of information from a wide range of media, often without rewarding the websites from which this data originated, either in the form of traffic or monetary value. There’s no “pay-per-click” equivalent, no revenue per citation, and as reported, organic traffic is dropping while “zero-click” searches are increasing. So, what’s the incentive for content creators?

Cloudflare’s Bold Move: Empowering Content Owners

Cloudflare is now looking to put the power back into the hands of the content creator. They are actively blocking AI crawlers from accessing content by default, unless there’s express permission or compensation. This means that AI crawlers will now have to declare their intention, and website owners will have the control to decide what to allow.

The Potential Impact on AI and SEO

Given that Cloudflare manages and protects traffic for around 20% of the web, this change drastically reduces the opportunity for AI agents to crawl the latest news and insights from some of the world’s leading publishers, media, and technology companies. This could, in turn, impact the quality of responses from AI models. If AI visibility is something you’re interested in for your content, it’s definitely best to check your Cloudflare settings.

But the ripple effects don’t stop there. On a day-to-day level, widely used tools like Screaming Frog and SEMrush may struggle to crawl sites without specific permission. It’s crucial that your SEO team is aware of this development and starts putting measures in place to mitigate any potential issues if your website uses Cloudflare.

The Future of Data Use and Compensation

Data scraping has long been the elephant in the room when it comes to AI tools, and Cloudflare is taking a clear stance to protect content creators and ensure they are rewarded for the use of their data. While some immediate knock-on effects, like the ability to legitimately crawl a site, are already evident, other consequences will likely take more time to fully materialise.