Getting started with AI monitoring tool Known Agents: A guide for news publishers

Deciding what AI bot to block or allow starts with understanding the consequences of those decisions and aligning those bot behaviors with your audience strategies. One way to do this is to use an AI monitoring tool like Known Agents.

At its launch the tool was called Dark Visitors and it was a simple list of AI bots, compiled by the engineer Gavin King. 

“Half of the internet is bot [traffic] across e-commerce and publisher sites,” King says, noting that the Known Agents tool aims to help publishers understand where AI agents, web crawlers and scrapers are coming from, how they affect website traffic and performance, and the consequences of allowing or blocking them. 

Here’s how to get started with this easy-to-use AI monitoring tool.

Step 1: Create an account 

Making an account on Known Agents costs nothing. The account can also be used to connect to multiple websites and reveal insights about bot traffic to the domains. On the free plan, you will only be able to connect to one domain.

Be sure to use an email address for this account creation step that has appropriate visibility and access for the people who may need it. The email address will be used for logging in to the site and it will receive email notifications when new AI bots visit your site as well as weekly agent and LLM activity reports. 

Step 2: Connect your website(s)

Connecting your Known Agents account to a website is simple and, in most cases, can be done without any coding background. 

Websites hosted on WordPress can use the Known Agents plugin to get started. You can also connect via CDN, like Cloudflare, Fastly, Amazon Web Services (AWS) and others. Additionally, you can connect with an API. (Find the full set of options for connecting to Known Agents here.)

Depending on how many websites you want to connect, you may need to move up from the free plan to one of the three other options. If you want to connect to more than one website or your website gets a lot of traffic, you should plan for a paid tier.

The free plan only includes 100,000 “events per month,” which is the number of visits your website gets. According to Known Agents, “you can estimate how many events you’ll need by doubling your human monthly visit count.” If you’re not sure or want to start with the free plan, the tool will email you and ask you to upgrade if you begin to approach the events limit.

Step 3: Start collecting data

After you’ve connected your website, you may need to wait for an hour or two before beginning to see traffic populate on the Known Agents project dashboard. If you are connected to multiple websites, each site will have its own dashboard.

While Known Agents does allow users to export the data it gathers, you will not be able to export data on free plans.

Step 4: Learn about your agent vs. human baseline

The longer your account collects data about a website, the more clear your baseline for bot traffic becomes. On the main project dashboard, users can see a real time overview of agent visits in the last 30 minutes, the total number of agent visits in the selected timeframe (28 days is the default) and the total agent volume, which indicates the percentage of your website traffic that is not human.

The dashboard also pre-filters the agent traffic to show you visits broken down by type or by which pages are being visited. All the data is visualized and has opportunities for users to see additional insights alongside explainers that can help you understand what an agent is actually doing on your website.

It’s also easy to zero in on a specific bot to see how often it’s visiting your site and where the bot is located. This may be useful if you’ve asked a bot to stop scraping your site and want to see if it’s listening to the robots.txt file instructions.

Step 5: Monitor, learn and (maybe) block

Generally, it’s a good idea to gather at least a month’s worth of data about your site’s bot activity before drawing conclusions. During that time, you can use Known Agents to read about different bots and start to become familiar with what the different types of events mean. 

If you conclude that blocking is necessary, the free Known agents tier can help you update your robots.txt file to ask one of the agents to stop crawling your site. This is especially useful for organizations that don’t have a developer on staff to update this file manually.

Unfortunately, the robots.txt is not a true traffic blocker for agents. So, if you’ve asked an agent to skip your site and still see traffic from it, you will need to take additional steps. (Read the Electronic Frontier Foundation’s tips on additional agent mitigation tactics here.)

Is Known Agents a good fit for your organization?

This is a low-risk monitoring tool that can help news organizations gain a base-level understanding of their AI agent audience. For smaller publishers with traffic that won’t hit the 100,000 event cap, the tool is a great free starting point for gathering data to create an AI agent strategy.  

Larger publishers looking to gain similar insights will need to consider which paid plan is the right option. But they should also consider how the cost compares to similar AI monitoring tools. Check out our alternates section at the end of this guide.

In either case, make sure there’s a person or team at your organization who can take the lead on helping your organization use the data. Without this dedicated resource — someone with the capacity to (1) learn about AI strategies for news publishers and (2) keep up with changes in this space — the tool will collect insights without benefiting the newsroom’s sustainability.

Pro tips and known issues

• Known Agents’ FAQs and written documentation are robust. It even includes a publisher-specific FAQ that shows screenshots of the dashboard and other data pages. Referring to the guidance will be helpful, but you can also reach out to the tool creator for a demo.
• Setup is simple through CDN integration or a WordPress plugin, and is free to use until you exceed the free tier plans or want to use custom integrations.
• The tool does a good job explaining what certain agents are doing on your site and its tips provide insight on what may happen if you attempt to block certain crawlers, including revenue, search discoverability and relevance markers.
• Known Agents also wants to hear from people who operate agents and other bots. It says listing an agent with the tool could result in better transparency for people who encounter the agent, which may mean more credibility or trust is established. (Scroll to the end of Known Agents main about page to contact them about listing an agent.)
• Currently, there are no monetization features within the Known Agents platform. Instead, the focus is on gathering data about your agent traffic and learning what meaning the traffic (or lack of traffic) has for your organization.

Alternatives to Known Agents for AI monitoring

Spyglasses

Though not a direct alternative, Spyglasses is another monitoring dashboard that allows a detailed technical level of insight on AI agent traffic with freemium tiers. If you have access to a developer team, you may want to explore using both tools, like the Technical.ly news organization is doing, to see how they meet your organizational needs.

TollBit

Another AI monitoring tool that offers a free plan for news publishers is TollBit. On top of monitoring and analytics, TollBit offers tools for managing bot behavior and monetization features for a fee. Read our case study on Digital Trends’ use of TollBit for more background.

CDN/WAF bot mitigation (Cloudflare, Akamai, Fastly)

Tools like Cloudflare, Akamai and Fastly can detect and block (or challenge) automated traffic like agents and crawlers. Each of these tools also is part of a much larger ecosystem of tools for website security and verification of human visitors. Generally, these tools are much more robust than Known Agents and publishers can expect to pay for these services.

This story has been updated to correct the name of the AI monitoring tool Spyglasses and to update the tool’s URL.

• Category: Getting Started Guides
• Tag: AI, AI monitoring, Known agents