Technical.ly welcomed AI bot traffic instead of blocking it, and gained subscribers
Share this story:
With the significant decline of search traffic to websites and rise in traffic from AI and bots, news leaders face the challenge of how to adjust their distribution strategies to reach human audiences.
Some are considering monetizing AI bot traffic; some are blocking bots as much as possible; and larger organizations can license their work. But there’s no one-size-fits-all approach, which means each publisher needs to decide the best path forward for itself.
Technical.ly, a national network of regional newsrooms covering the local effects of technology startups and the economies they create, decided to optimize for AI in ways that could help them reach new audiences.
That decision was informed by AI monitoring tools Known Agents and Spyglasses. Here’s a look at how the news organization is using the tools and why the optimization strategy made sense.
Newsroom overview
Funded by a mix of philanthropy, research production, marketing packages and advertising, Technical.ly aims to engage, inform and connect local technology entrepreneurs and their communities of investors, supporters, researchers and policymakers.
With a lean staff of 13, their coverage currently spans eight cities — including Philadelphia, Baton Rouge and Washington, D.C. — along with national trend stories in local tech-driven “innovation economies.” The organization’s goals to reach and deeply engage their niche audiences drove them to approach AI discoverability and distribution in an innovative way.
Problem: Overwhelming AI bot traffic
Last year, Technical.ly audience product manager McKenzie Morgan and her team were balancing infrastructure projects along with research on where their content was being seen by readers through Google’s AI Overviews and bots.
During that time, the news site began to slow significantly, taking minutes for articles to save in their WordPress CMS and longer to navigate to published pages. It was so bad that their 404 error page became the top pageview one month.
The team quickly discovered the culprit: AI bot scrapers hitting their site several thousand times a day. At first, she and her developer colleague explored different server options to handle the influx of new traffic that wasn’t resulting in increased audience metrics, but the cost for a small newsroom was astronomical.
During their research, Morgan’s teammate discovered two tools that allowed them to track where the influx of non-reader traffic was coming from, including from China and Singapore. As a newsroom covering and reaching U.S. domestic audiences about local technology economies, they started to test tools that allowed them to make decisions on which bots to block and which to allow. They also wanted to learn how AI bots were using their content to train and surface information beyond Google search.
Solution: Learning about the bot traffic
After Technical.ly blocked all traffic from China and Singapore, Morgan and her colleague used the Known Agents WordPress plugin to audit where their AI referrals were coming from and get more information about what the bots were doing. The user-friendly tool also allows them to see the implications of blocking access to certain crawlers, scrapers and AI agents.
“In Known Agents, you can see AI [chatbots] and referrals and filter session source for main referrers. I don’t think you need a developer to use this; if you’re a small news organization, it’s easy for anyone to do,” says Morgan.
The second tool they used, Spyglasses, allowed them to see more detailed metrics on the visibility of their news stories, including the number of AI assistants referencing their articles in chatbot results and human visits referred by AI. At first, the team blocked AI that were training on their content and allowed inference bots (like Google Gemini and other popular LLM chatbots that are already trained) access for revenue purposes.
With information from these two tools, editorial director Danya Henninger was empowered to make strategic decisions about which bots helped or hurt the news organization’s business. The data also helped her understand that optimizing for AI discoverability could help them reach human readers.
As audiences increasingly search for answers using chatbots like Claude or ChatGPT, Henninger decided that Technical.ly’s place in establishing their expertise and subject matter authority was to inform the LLM models as much as possible. By allowing the AI models access to their content, the goal is to reach potential audience members searching for the specific tech startups and topics that their newsroom covers.
“We are optimizing for ‘hits’ from AI assistants and AI agent bots, used to retrieve fresh information for queries,” Henninger says. “We purposely do not block AI trainer bots, [which are] used to train models.”
Specifically, in AI assistant responses where people were asking questions related to Technical.ly’s local innovation coverage, they optimized for linked citations to Technical.ly stories and mentions for the businesses they covered within those stories. This establishes Technical.ly as a good resource on a niche topic, and surfaces the tech companies in relevant ways to the person using ChatGPT or another chatbot.
Henninger mentioned that in their markets, an emerging technology company’s first press mention is often through Technical.ly, due to their specific topical emphasis. Traditionally, publishing newsworthy topics before other outlets establishes authority in search and how often the article is referenced for further reporting by other outlets.
This was the case in 2014, when Technical.ly covered the sale of Scribewise, a Philadelphia-based marketing agency, to a company called Trellist. In 2016, the original Scribewise owner, John Miller, bought the company back.
However, when Miller’s teammate asked Google Gemini about Scribewise, the response was that Scribewise was still a part of Trellist and that the acquisition date was in 2024. Gemini could be prompted to fix that information in an individual conversation, but that isn’t a fix for other chatbot users. Miller’s team decided to contact Technical.ly to update their coverage.
That resulted in a new article about Scribewise with current information that Gemini was able to learn from.
“This is why journalism is important in AI; someone needs to be feeding the training models correct, updated information,” says Henninger, who edited the updated Scribewise piece published in March.
Impact: Optimizing for AI leads to audience
As part of the AI optimization strategy, Henninger’s team prioritizes publishing explainers in their coverage areas beyond the general descriptions that chatbots return, including defining key terms like innovation districts and adding TL;DR bullets, which search and LLM chatbots scrape more easily to use as responses to related to questions.
The strategy has led to positive results. Since September 2025, referrals from AI chatbots to Technical.ly have newsletter subscription conversion rates that are twice the rate of search referrals and 1.5 times that of direct traffic.
Though mentions and links in search still drive a larger number of newsletter subscribers, the Technical.ly team is working toward an important loyalty KPI (Key Performance Indicator) involving authoritative mentions and citations within LLM responses.
Verdict: Better visibility of your AI traffic leads to smart strategies
Though their team is still assessing both tools for long-term usefulness, Henninger and Morgan have been able to make significant changes to their AI strategy in a short amount of time.
A year ago, AI bots had limited readers from accessing their site. Now, Technical.ly is leveraging AI bots to help them reach human audiences.
Both Known Agents and Spyglasses are free to use via WordPress plugin or CDN integration that protects your organization’s data. For enterprise-level usage, including custom integrations and unlimited monitoring for larger newsrooms, custom pricing options are available.
Though rules of discoverability are changing every day, Morgan cites assessing tools and continuing to learn from a community of other news leaders as a key part of helping inform Technical.ly’s AI optimization strategy.
“The biggest thing that has helped me is knowledge sharing … in News Nerdery, NPA and ONA,” Morgan says. “Everyone is helpful and more than happy to get on a call to discuss what our news orgs are doing and offer insights. We’re all in this together.”
This story has been updated to correct the name of the AI monitoring tool Spyglasses and the tool’s URL. A sentence about fixing incorrect information in a Gemini chatbot conversation has also been clarified.
Share this story:
