Server logs: where AI crawler activity is real

Your citation tracker can tell you that no AI answer named your pages last month. It cannot tell you why. There are two reasons, and they could not be more different. Either the machines read your pages and preferred somebody else's, or they never fetched them at all.

One file on your own servers settles that, with facts rather than estimates. Every fetch by GPTBot, ClaudeBot, PerplexityBot or Google-Extended leaves a line in your server log. A timestamp, a URL, a status code. No panel, no sampling, no modeling.

The catch is the file itself. An SEO manager who asks for it usually finds out it belongs to another department. So this article is about reading those lines, and about getting hold of them in the first place. It is also the cheapest check in this series to repeat, which matters, because crawler behavior drifts month to month whether anyone is watching or not.

Fetches and citations are different facts

A bot fetching your pages is activity. Citations are the outcome: an answer naming one of your pages as a source underneath its text. One check reads the first straight from your logs. Crawl volume per AI bot, the status codes those fetches came back with, and how many unique pages each covered.

Put the two facts side by side and you get a matrix worth memorizing. Heavy fetching with no citations means the machines read you and chose other sources, which is a content and authority problem no robots rule will fix.

No fetching at all is the opposite diagnosis. The pipeline is broken before your content ever gets a vote. That is an access problem, and it has a specific fix.

The AI crawler readiness article covers that layer, robots directives and CDN rules included, and this log check is how you confirm a fix took.

The matrix also settles arguments about where to spend next quarter. A team certain it has a content problem can discover in one log check that the crawlers never arrived. The reverse is just as common, and it stings more: access is clean, fetches are healthy, and the missing citations are a competitive problem that no amount of technical work is going to fix.

See also: the robots and CDN rules that decide whether bots get in →

Which bot actually visits you most?

Often the one nobody is watching. On a site we looked at, the user-triggered fetcher did most of the visiting. The training crawler gets the headlines. The fetches that arrive because a real person asked a question in that moment outnumbered it many times over.

That changes what a failed fetch costs you. A training crawl that hits an error costs you a little future representation, and you never see the bill. A user-triggered fetch that fails costs you a live answer. Somebody asked about your topic right then, and the machine could not read your page.

So the status codes per bot are triage rather than trivia. Errors on the on-demand fetchers are your urgent rows, and the check sorts them out of the noise. The split shows up cleanly in the per-bot table. It deserves a standing row in any AI readiness report.

What the status codes actually say

More than the totals above them do. A wall of redirects tells you bots are spending fetches on moved addresses your internal links never caught up with. Errors concentrated on one template point at a fault in that template rather than a site-wide illness. A sudden block pattern usually traces back to an edge rule shipped for an unrelated reason, with crawlers as the accidental casualty.

Read the codes per bot before you generalize. Crawlers differ in how they honor directives, how they retry, and which paths they favor. A clean report for one bot certifies nothing about the rest. The per-bot cut is the same table one filter deeper. It is where the surprises live.

Spoofers come out before the count

The player below shows the log check in flight. Verification runs before counting, because a log full of impostors flatters every number in it.

Getting the logs is the actual project

The analysis is the easy half. Access is the hard one. It is where most log projects die. Logs live with the CDN or the platform team, and the search person who wants them needs an approval that crosses an organizational boundary nobody drew with this use in mind.

We watched a manager solve it the practical way. He asked for an executive summary he could paste verbatim into the approval ticket, in the approver's language rather than his own. The ticket got approved because the case arrived pre-written.

If log access is your bottleneck, write that summary before anything else. It is one paragraph, and it beats a quarter of waiting. The same summary doubles as the opening slide of your recurring report, so the access request and the reporting routine become one artifact instead of two. Three lines carry it.

• What the logs would reveal
• Which decisions they would speed up
• What it costs to keep flying blind

What the log check feeds next

Three things. Every one of them is already waiting on it. Your non-200 rows become a repair worklist with owners: redirects that should be direct links, errors on pages that matter, blocks that were meant to be temporary.

Coverage numbers feed your crawl budget conversation, because a page no bot fetches is a page no answer can lean on. And the verified-versus-spoofed split feeds security and analytics hygiene at once.

Monthly beats deep, every time. A log check each month catches rot while it is still one ticket's worth of work. The Server Logs source page describes what the feed carries, and the fake Googlebots article covers the verification layer this check depends on. Between them they turn the least glamorous dataset you own into the one your AI visibility numbers stand on.

See also: how to tell a real crawler from an impostor →