Why Google-Extended Never Appears in Your Server Logs

Google-Extended will never show up as a line in your server logs, because it is not a crawler. Google documents it as a robots.txt token that controls how already-crawled content is used. If your bot filter or traffic report searches for that name, it will find nothing, and that is normal.

This matters because many owners build AI bot reports from a list of names. If one name on the list can never appear, the report can mislead you. This post explains what the token does and how to read your data correctly.

What Google says Google-Extended is

Google's crawler documentation states that Google-Extended has no separate HTTP request user agent string. In Google's words, crawling is done with existing Google user agent strings, and the robots.txt token is used in a control capacity.

The token lets publishers manage whether content Google crawled can be used for training future Gemini models and for grounding. Google also says it does not affect a site's inclusion in Google Search and is not a ranking signal.

So the token is a permission switch. It is not a visitor, and it leaves no footprint of its own in your logs.

How this differs from other AI bots

Several AI companies publish bots that do show up by name. OpenAI documents three that matter most for a business website.

• OAI-SearchBot powers ChatGPT search features. OpenAI says sites that disallow it will not be shown in ChatGPT search answers, though they can still appear as navigational links.
• GPTBot crawls content that may be used to train generative AI models. Disallowing it signals that your content should not be used for that training.
• ChatGPT-User handles actions a person starts inside ChatGPT. OpenAI notes that because a user starts these actions, robots.txt rules may not apply.

Each of these sends its own name in the user agent header. You can count them in a log file. Google-Extended gives you no such line to count.

Where the confusion causes bad decisions

Three mistakes come up often when owners read bot reports.

1. Reading zero as safe. A report that shows no Google-Extended hits does not mean Google is staying away. Googlebot may still crawl your pages, and the token only governs later use.
2. Reading zero as a broken block. If you added a disallow rule and see no change in logs, the rule is not failing. A token never produced log lines to begin with.
3. Counting Googlebot as AI traffic. Googlebot serves Google Search. Adding its hits to an AI crawler total inflates the number and hides real AI activity.

Each mistake leads to the same problem. You end up making a policy choice from a number that does not measure what you think it measures.

How to read your data correctly

Step 1: Sort names into three groups

Label each entry in your bot list as a training crawler, a search crawler or a user-triggered fetcher. This matches how the platforms describe their own tools. Keep the token on a separate line marked as a policy setting, not a traffic source.

Step 2: Remove names that cannot appear

Delete or relabel any list item that is only a robots.txt token. Leaving it in suggests a measurement that does not exist. A clean list is easier for your team to trust.

Step 3: Keep your policy and your measurement apart

Write down your robots.txt choices in one place. Write down your measured crawler hits and AI referral visits in another. When someone asks what an AI tool did on your site, you can answer from the right record.

Step 4: Note the date of every change

OpenAI says it can take about 24 hours after a robots.txt update for its search systems to adjust. Log the day you edit the file. Then wait before judging results, and compare the same weekdays on each side of the change.

What you can and cannot measure

You can measure named bots in your server logs, and you can measure visits that arrive with an AI assistant as the referrer. You can compare the two over time to see whether crawler activity moves with visits.

You cannot measure the use of your content inside a model. No log line records that. Treat any report that claims to count it as an estimate, and ask how it was built.

You also should not assume a robots.txt change will move your visit counts quickly. Search features, training and user fetches follow different rules, and each reacts on its own schedule.

A quick audit you can run this week

• Open your robots.txt file and list every AI-related token and bot name in it.
• Mark which ones are tokens only and which are real user agents.
• Check your log or bot report for each real user agent over the last 30 days.
• Confirm that Googlebot hits are filed under search, not under AI crawlers.
• Record the result with a date so you can compare next month.

This audit takes less than an hour for most small sites. It also gives you a clear starting point before you change any rule.

The bottom line

A name in your robots.txt file is not always a name in your logs. Google-Extended is the clearest case, since Google says it has no user agent string of its own. Build your reports around bots that can actually be counted, keep policy notes separate, and your AI traffic numbers will stay honest.