Worried about AI scrapers and don’t know where to start? Here are five lessons for news publishers.
Share this story:
Note from the editor
The news industry is split on how to deal with generative AI scraping published journalism.
Some organizations are leaning into optimizing content for bot and agent traffic, similar to how news publishers embraced search engine optimization (SEO) as a way to help audiences more easily discover their content through search engines like Google.
Other outlets are focused on blocking AI traffic as a way to create demand and — hopefully — be compensated by AI companies for access to published journalism.
Right now, the only “right” option is to do something about AI. Doing nothing puts news publishers behind other content creators that are working to figure out how to reach audiences who are increasingly adopting AI. Basically, an AI-internet is the next big change in technology that news publishers need to adapt to in order to be sustainable.
To help you decide on your next steps, here are five valuable lessons news publishers need to be aware of as they plan for the digital future of publishing news.
1. Individual defensive tactics don’t solve collective action problems
Litigation over AI crawling content has not yet been fruitful for news publishers.
A great example of this is The New York Times suing OpenAI and Microsoft in December 2023. News Machines reported that The Times spent $4.4 million on legal costs in Q1 2025 alone. But, as of today, no trial date has been set.
Legal action along these lines may take years to reach a result. And while news organizations wait for the decisions, they continue to pay in time and money.
Rather than going the route of litigation alone, news publishers are shifting to collaboration after seeing cases like Bartz v. Anthropic end in a massive $1.5 billion settlement. Authors had filed a class action lawsuit against Anthropic for using their works to train its AI system.
Other individual defensive tactics that aren’t working include outright blocking AI bots, which has led to lost human traffic, and licensing deals between a single publisher and a single AI company, which has been linked to lower click-through rates.
2. Structured data feeds generate revenue where article licensing doesn’t
The Associated Press, Reuters, and AFP — all wire services — for years have invested in creating structured feeds of their news content. Now, that investment in infrastructure has turned into a way for monetizing AI bot and agent traffic.
AI bots like Google’s Gemini and Microsoft’s Copilot use the wire services’ feeds as a source of real-time news, paying the news organizations in bulk rather than by individual story. Combined, these three wire services have generated an estimated $65 million annually from structured feeds, according to News Machines.
The reason this commodity-scale model works is because the infrastructure is already in place, and the sheer volume of content gives AI companies a lot to work with.
3. Direct audience relationships generate more value
When search traffic from Google started to plummet for publishers thanks to AI summaries, the news organizations that already had direct relationships with their audiences still had a reliable way to deliver news.
Subscription paywalls, membership signups and newsletters are all forms of creating a direct audience relationship. Rather than relying on a third party to send audiences to content, directly communicating with audience members reminds them of the news product’s value. And, compared to getting pennies per pageview (think ads or a potential pay-per-crawl AI model), one annual subscription can generate a lot more.
To create these connections, publishers need to consider their existing technology stack — the tools that make digital content publishing possible. For example, some content management systems have paywall infrastructure built in, but are missing newsletter capabilities for reaching subscribers. Depending on what your tech stack looks like, some tools are a better fit than others.
Newer publishers, smaller publishers and independent journalists have turned to all-in-one platforms like Beehiiv and Ghost, which lend themselves well to creating a direct connection to readers without having a technology team.
4. Embracing AI tools can mean cost savings
Using AI tools internally at news organizations is helping cut costs. Publishers report 20% to 30% cost reductions from adopting AI tools, according to News Machines.
Of course, selecting these tools has to be done intentionally and without causing harm to the editorial product.
Publishers beginning their journey to finding the right AI tools should start with conversations and looking for pain points within a news organization that need solving (think arduous workflows, repetitive tasks or even tactics that require a big time commitment for very little revenue gains). These problems are opportunities where implementing AI could make a big difference — and even save money.
Then, see which AI tools are successfully helping solve problems for other publishers. Feel free to reach out to us at the Help Desk if you need help navigating options.
5. Learn more about your AI traffic before deciding how to monetize it
According to News Machines, no AI marketplace participant has disclosed revenue yet. That fact, combined with the lack of transparency around deals and pricing, means news publishers could easily buy into a program that doesn’t end up being worthwhile.
News Machine outlines the fundamental tension between individual and collective solutions for monetizing AI traffic, so we won’t get into that. Instead, we want to propose a low-risk move that benefits publishers as they plan for the future of AI: Start using an AI-focused analytics tool.
In our case study highlighting TollBit, Digital Trends gave us the inside scoop on their strategy of getting data now to set themselves up for getting revenue from AI later. The AI monitoring made possible with TollBit has already proven valuable for informing future content and business strategies at the technology reporting outlet.
TollBit isn’t the only tool for monitoring. Local news publishers have also turned to Known Agents, Cloudflare and others to get better insights about how the traffic to their sites are changing.
AI transparency note: The lead art for this story was created using AI tools on Canva.com.
Share this story:
