Verify AI Bot Hits With Published IP Ranges Before You Trust Your Logs

A user-agent string is a label, and anyone can type any label. If your log report counts every request that says it is an AI bot, some of those hits are probably not from that platform. The fix is simple: check the requesting IP address against the ranges the platform publishes, then count only the matches.

This matters because many owners now compare AI crawler activity with AI referral visits. If the crawler side is padded with fake hits, the comparison misleads you. A short verification step keeps both sides of that ratio honest.

Why a User-Agent Alone Proves Nothing

The user-agent is a header the visitor's software sends with each request. Real crawlers use it to identify themselves politely. Scrapers and scanners can copy the same text in seconds.

Some people copy AI bot names on purpose. They hope your site will treat them as welcome guests and skip rate limits or blocks. Others do it by accident through a poorly configured tool. Either way, your report shows a bot name that the platform never sent.

An IP address is much harder to fake for a visit that needs a response. That is why the platforms publish their address lists, and why Google documents a way to check a crawler against its network.

What the Platforms Publish

Each platform offers a slightly different tool. Here is what the primary documentation says.

• OpenAI: It publishes IP address files for its bots. There are separate JSON files for OAI-SearchBot, GPTBot and ChatGPT-User, so you can check each agent against its own list.
• Perplexity: Its documentation describes PerplexityBot and Perplexity-User and recommends allowing requests from its published IP ranges.
• Google: Its documentation describes two methods. You can run a reverse DNS lookup followed by a forward lookup by hand, or you can match the address against Google's published range files for large volumes.

Use the exact files named in each platform's own pages. Ranges change over time, so do not copy a list once and forget it.

Two Ways to Verify

Method one: match against the published list

This method suits a regular report. Download the platform's JSON file, extract the address ranges, and test each logged IP against them. A request that claims to be ChatGPT-User but falls outside the ChatGPT-User list is unverified.

Your web team or hosting provider can often automate this inside a log processing job. Refresh the lists on a schedule, such as weekly, so new ranges do not show up as false alarms.

Method two: reverse and forward DNS

Google describes this method for its crawlers. You look up the host name for the IP address and confirm it belongs to a Google domain. Then you look up that host name and confirm it returns the original IP address.

Both steps must pass. The two-step design matters because anyone can set a reverse record that sounds right, but only the real owner controls the forward record. Use this method for one-off checks when a suspicious spike appears.

Build a Three-Way Split in Your Report

Once you verify, change how you label the data. A simple split works well for most small business reports.

1. Verified: The user-agent and the IP range both match the platform's published information.
2. Claimed but unverified: The user-agent names an AI bot, but the IP does not match.
3. Unknown: The request has no clear label and no match.

Report the first group as AI bot activity. Keep the second group visible, because a large share tells you something about your exposure to scrapers. Do not add the second group to your AI totals.

If you already sort bots into crawlers, user-triggered fetchers and browser agents, run the verification inside each of those buckets. A verified ChatGPT-User fetch means a person asked a question that touched your page. An unverified one tells you nothing about a person.

Do Not Block on the Label Alone

Verification also protects you from mistakes in the other direction. If your firewall blocks every request that claims to be an AI bot, you may block real visitors who use odd software. If it allows every request that claims to be one, you may let a scraper through.

Better rules combine the label and the address. Allow verified bots that you want, such as a search bot that can surface your pages in answers. Rate limit or challenge claimed bots that fail the check. Keep your robots.txt file accurate as well, since it is the public statement of what you welcome.

Remember that some user-triggered fetchers may not follow robots.txt in the same way as automatic crawlers. OpenAI and Perplexity both explain this in their bot pages. For those agents, the IP check and your server rules do the work that robots.txt cannot.

How This Changes Your Decisions

Clean bot counts help you in three practical ways.

• Hosting load: You see which traffic is real crawling and which is noise, so you can size your server and your rate limits correctly.
• Access choices: You can decide which verified bots to allow based on real activity instead of inflated numbers.
• Referral comparisons: A verified crawl count paired with referral visits gives you a fairer picture of how often answer engines read your pages compared with how often people click through.

Keep one caution in mind. Verification tells you who made a request, not whether the request led to a visit or a citation. Treat the verified count as one input, next to your referral data and your direct traffic, not as a result on its own.

A Monthly Routine That Takes Minutes

1. Export one month of server log lines that contain an AI bot name.
2. Match each IP against the current published files.
3. Record the verified and unverified counts for each bot name.
4. Note any bot with more than half its hits unverified, and ask your host to review it.
5. Update your report labels and save the date of the list refresh.

Ask your hosting provider or web team to do the matching if you do not read logs yourself. Give them the three platform pages listed below so they work from the primary documentation. Once the routine is in place, your bot counts will be defensible when someone asks where the numbers came from.