User-Triggered AI Fetchers: The Third Bucket Your Traffic Report Needs

AI assistants now send a third kind of hit to your site, and most reports lump it in with something else. These are user-triggered fetchers. They request your page because a person asked an assistant a question, and they usually ignore robots.txt. Count them in their own bucket, apart from search crawlers and apart from human visitors.

This matters because the two familiar buckets lead you astray. Treat a fetch as a visit and your traffic looks bigger than it is. Treat it as a crawler and you miss a sign that someone is asking about your business right now.

What a User-Triggered Fetcher Is

Google describes user-triggered fetchers as tools that are initiated by users to perform a fetching function inside a Google product. Its documentation says they generally ignore robots.txt rules, because a user requested the fetch and Google did not start it on its own.

OpenAI draws the same line. Its documentation lists ChatGPT-User as the agent that handles user-initiated actions in ChatGPT. It says robots.txt rules may not apply to those actions, and that the agent is not used to crawl the web in an automatic fashion.

Google added a fetcher called Google-Agent to its official list in March 2026. It lets agents hosted on Google infrastructure browse the web and act when a user asks. It has its own IP range file, so you can separate it from Googlebot.

The Three Buckets

A clean traffic report sorts every request into one of three groups. Each group answers a different business question.

• Search and training crawlers. These run on their own schedule and follow robots.txt. Examples are OAI-SearchBot and GPTBot. They tell you whether an assistant can find and learn from your pages.
• User-triggered fetchers. These run when a person asks. Examples are ChatGPT-User and Google-Agent. They tell you which pages people are asking assistants about.
• Human visits. These are people who clicked a link and loaded your page in a browser. They are the only group that should feed your conversion numbers.

Why the Middle Bucket Is Different

A crawler hit says little about demand. A fetcher hit says a real person wanted something from your page at that moment. A burst of fetches on your pricing page, for example, is worth a look.

It still is not a visit. The assistant reads the page and summarizes it, and the person may never click through. Your analytics script often never runs, because these requests may not load a full browser. That is why server logs show fetches that your dashboard never records.

The opposite error also happens. Some fetchers do run in a browser-like environment, and a tool that counts them as sessions can inflate your numbers. Check how your reporting tool treats them before you trust a spike.

How to Sort Your Logs

Start with your server or hosting logs, since they record every request. Follow these steps once, then repeat monthly.

1. Export 30 days of requests. Keep the date, path, user-agent string, IP address and status code.
2. Filter on known names. Search the user-agent field for OAI-SearchBot, GPTBot, ChatGPT-User and Google-Agent. Put each into the bucket described above.
3. Verify with IP ranges. OpenAI publishes JSON files for each agent, including chatgpt-user.json. Google publishes user-triggered-agents.json and two related files. Match your IP addresses against them.
4. Flag the leftovers. Requests that claim a known name but come from an unlisted IP address are likely imitations. Keep them out of every bucket.
5. Compare against human sessions. Line up fetch counts by page with human visits to the same pages.

The IP check matters because any program can type a trusted name into its user-agent field. A name alone proves nothing. A name plus a published IP range is far stronger evidence.

Do Not Rely on robots.txt to Control This Bucket

Many owners add a disallow rule and assume the matter is closed. For crawlers such as OAI-SearchBot and GPTBot, that rule works as intended. OpenAI notes that opting out of OAI-SearchBot keeps a site out of ChatGPT search answers, though it can still appear as a navigational link.

For user-triggered fetchers, the rule is not the control you think it is. Both OpenAI and Google say these requests may ignore it. If you must restrict access, use your server or firewall settings, and base the rule on the published IP ranges.

Think about the business cost before you block anything. A blocked fetch means the assistant cannot read your page for that person, and it may answer from a competitor's page instead. Most small businesses gain more from being readable than from being hidden.

Turning the Numbers Into Decisions

Once the buckets are separate, each one points to a clear action.

• High fetch counts, low human visits. Assistants are reading the page, but people are not clicking. Put the key facts, such as hours, service area and next step, in the first lines of the page.
• Fetches on pages you rarely update. People are asking about those pages. Check that prices, policies and contact details are current.
• Crawler hits with no fetches. Assistants can find you, but few people ask about you yet. Look at your page titles and headings for the questions customers actually use.
• Human visits from assistant links. These are your real referral visits. Track them by referrer and measure what they do, such as form submissions and calls.

Keep Privacy in Mind

This method needs no personal data about visitors. User-agent strings and IP ranges identify software, not people. Keep log retention short, remove human IP addresses when you finish the analysis, and tell visitors in your privacy policy what you collect.

A privacy-first approach also keeps your reporting honest. You are measuring machine requests and human sessions separately, so you never need to profile individuals to understand demand.

A Simple Monthly Habit

Build a one-page summary with three numbers per month: crawler hits, fetcher hits and human visits from assistants. Watch the ratios, not the totals. If fetches rise while human visits stay flat, your content is being read but not clicked, and that is the gap to fix next.

Review the published IP files each quarter, because both companies can add or change agents. Their documentation pages are the source of truth, so recheck them when your numbers shift without a clear cause.