Separate AI User Requests From Crawler Hits in Your Traffic Report

AI traffic reports become more useful when they separate human clicks, assistant-triggered requests, and ordinary crawler activity. Start by labeling each request instead of combining every AI-related user-agent into one traffic number.

A person may click a link in an assistant's answer and create a normal browser session. An assistant may also request a page after a person asks a question, while a crawler independently fetches pages for search or discovery.

Why one AI traffic bucket creates bad decisions

Server logs record requests, not intent. If a report groups browser referrals, assistant fetches, and crawler activity together, the total can look like visitor demand when it actually measures several different events.

The fix is to define separate data fields before reviewing totals. At minimum, record whether the request appears to be a browser visit, an assistant-triggered fetch, a crawler request, or an unresolved event.

• Human referral: A browser arrives after following a link from an assistant.
• Assistant-triggered request: An assistant fetches a page in response to a person's question.
• Crawler request: An automated system requests pages for indexing, search, training, or related processing.
• Unresolved request: Available evidence does not support a reliable classification.

These labels answer different business questions. Human referrals help measure visits and conversions, while crawler requests help explain access patterns and content exposure.

Use the request details you already collect

A practical review starts with request-level logs containing the timestamp, time zone, requested URL, referrer, user-agent, status, method, request ID, cache status, and edge or origin source.

Keep the original values, then add a classification field. Do not overwrite the raw user-agent or referrer, because later verification may require the unedited request.

Referrer and campaign fields

A browser referral may include an assistant hostname in the referrer, while a campaign parameter may preserve the source when the referrer is absent. Treat both as evidence of a possible referral, not proof that a person completed the visit.

Review the landing page, session cookie, JavaScript activity, and conversion events alongside the source. A request that contains a source value but lacks normal browser activity should remain separate from a confirmed visitor session.

User-agent fields

Separate user-agent patterns for user-triggered requests from crawler patterns when the platform publishes distinct identifiers. For example, the documented guidance distinguishes ChatGPT-User from GPTBot and OAI-SearchBot, and Perplexity-User from PerplexityBot.

Do not treat a matching user-agent as conclusive identity. The same guidance recommends checking additional request evidence and assigning labels such as validated, unverified, or rejected.

A three-step classification routine

1. Preserve the raw request. Store the original referrer, user-agent, URL, method, status, timestamp, and request ID before applying filters.
2. Apply a preliminary label. Mark likely browser referral, likely assistant request, likely crawler, or unresolved based on the available signals.
3. Verify before reporting. Compare the request with published platform information, source IP ranges where available, browser behavior, and nearby requests from the same session or system.

Published IP ranges can support verification, but they should not replace other checks. A matching address can strengthen a classification, while a missing match should keep the event unverified instead of forcing a conclusion.

Record the verification date and rule used. Platform identifiers and published ranges can change, so a classification without its evidence becomes difficult to audit later.

Keep people and machines out of the same conversion report

Only confirmed browser sessions should enter a human referral conversion report. Assistant fetches and crawler requests belong in a separate operational report, even when they involve the same content page.

This separation prevents a page request from appearing as a lead, purchase, form completion, or engaged visit. It also keeps content decisions grounded in the behavior of people rather than automated access volume.

Build two simple views from the same source data:

• Visitor view: Sessions, landing pages, engagement, conversions, and revenue from browser activity.
• Request view: Automated requests, requested URLs, response status, crawl frequency, and classification confidence.

When the two views point in different directions, keep them different. High automated access with few human referrals may support a technical review, but it does not prove that the page attracts customers.

Handle missing referrers without guessing

Some assistant links may lose referrer information, and some visitors may appear as direct traffic. Treat missing attribution as an uncertainty problem, not as evidence that AI played no role.

Use several signals together when permitted by your privacy practices. Review campaign parameters, landing pages, request logs, assistant-related user-agents, and conversion paths, then report the result as an observed minimum when the source cannot be confirmed.

Do not identify individual people from incomplete traffic data. Aggregate counts, remove unnecessary identifiers, limit retention, and follow the consent and privacy requirements that apply to your website visitors.

Turn the labels into decisions

Review the visitor and request views on the same schedule, but ask different questions of each. This keeps measurement useful without treating every automated request as a marketing outcome.

• If validated human referrals rise, compare their landing pages and conversion rates with other acquisition sources.
• If assistant-triggered requests rise without browser referrals, check whether the content answers common questions clearly and whether access rules permit the intended fetching.
• If crawler requests increase sharply, inspect requested URLs, response codes, cache behavior, and server load before changing content.
• If many records remain unresolved, improve log fields and classification rules before drawing trend conclusions.

Use a confidence field such as validated, unverified, or rejected. Report totals by confidence level so readers can see what the data supports and what remains uncertain.

A compact report structure

A monthly AI traffic report can stay short while preserving the distinctions that matter. Include the date range, human referrals, assistant-triggered requests, crawler requests, unresolved requests, conversions from validated sessions, and the rules used for classification.

Add one sentence describing missing data, such as stripped referrers, blocked scripts, incomplete logs, or unavailable platform verification. That note prevents readers from treating the report as a complete measure of all AI influence.

The goal is not to produce one impressive AI number. The goal is to show which requests represent people, which represent machines, and which require better evidence before anyone changes a budget or page.