AEO monitoring
AI crawler readiness: you can't be cited if you were never crawled
The R1 gate: verified bot fetches joined to the pages you want cited.
Six weeks into a content sprint aimed at an AI-visibility problem, an engineer at acme.example found the robots rule. No AI crawler had ever fetched the pages the writers were rewriting.
That sprint was unwinnable on day one. Nothing in the brief could have said so. You cannot be cited if you were never crawled, and being cited means an AI answer names one of your pages as a source. A lot of AI-visibility panic dissolves at exactly this check.
It is also the cheapest diagnosis in AI search, because it reads data you already have.
One zero, two very different causes
The answer engines are the tools that answer a question directly and cite their sources. When they leave you out of the prompts you track, there are two explanations, and they need opposite fixes.
Either the crawlers fetched your pages and the answers chose other sources, which is a content problem. Or the crawlers never got in, which is an access problem that no amount of writing will fix.
On a dashboard those two zeros look identical. Server logs are the only place they come apart, because the logs record which bots arrived, what they asked for, and what your server said back.
The panic pattern has one shape. A citation number drops or flatlines, the content team gets handed the problem, and weeks later somebody finds the rule that made the whole exercise unwinnable. The log check costs minutes. It should have run first.
Rendering belongs in the access category too. A page the bot fetched but could not read past the JavaScript is crawled in the log and absent in practice. So the check reads status codes and indexability together rather than fetch counts on their own.
What the check actually joins
It joins verified crawler activity to the pages you want cited, in one conversation.
One read shows per-bot fetch volume and the status codes each bot got back. Another separates real crawlers from traffic pretending to be them. Two more find the pages nothing can index and the pages nothing links to. The ai-crawler-readiness skill, a packaged analysis you run by name in your assistant, chains those in order. What comes back is a short list with a cause attached to each entry.
That crossing, crawl logs against citation targets, is a read most stacks cannot produce, because logs and AI-visibility data almost never live in the same place.
Verification matters more than it sounds. A meaningful slice of traffic claiming to be a crawler is not one, and conclusions drawn from unverified logs inherit the noise.
Ask it yourself
Which of the pages we want cited weren't fetched by any AI crawler recently, and what did the ones that came get back?
See also: separating real crawlers from the traffic impersonating them →
Each cell names its own fix
Cross crawled against cited and you get four boxes. Each one points at different people.
That matrix keeps teams from writing their way out of a technical problem and from engineering their way out of a content problem, which are the two most expensive wrong turns in AI search.
It also sets expectations plainly. Fixing an access problem restores eligibility, not citations. The content still has to win the answer, and it may well lose. At least the contest is real again, which is more than you could say the week before.
So here are the four, and the fix each one names.
- Crawled and cited: it is working, so protect it.
- Crawled but never cited: a content and passage-quality problem, so brief it.
- Not crawled and not cited: an access problem, so fix the robots rule or the rendering before anyone writes a word.
- Cited with no crawl on record: a flag to audit the measurement itself.
Bot access regresses without a routine
It regresses the way all technical health regresses. A deploy. A CDN rule. A robots edit made for good reasons two quarters ago.
Frequency matters because the answer engines re-fetch on their own schedule, not yours. A page that went unreachable in March is a citation that quietly expires by May, and nothing on your site looks different in the meantime.
The bots also differ from each other. Training crawlers and live fetchers gate different surfaces, so readiness gets checked per bot rather than as one AI average. A site can be wide open to one engine and dark to another without anybody having decided that.
See also: where AI crawler activity is real, in the server logs →
This check protects every number above it
On the Quattr Method it belongs to R1 · Reachable, the first question: can search engines and AI bots fetch, render, and index your pages at all? It gates the prompt-basket work explicitly. Conclusions about the prompts you track wait until the bot check passes.
That ordering is what makes the citation numbers trustworthy when they finally arrive. Every zero left standing is a real zero, with a workstream attached to it.
It is the least glamorous check in AI search. It is also the reason anybody believes the exciting numbers that come after it.
See also: the first question, and everything that depends on it →
The workflow that does this: ChatGPT citations by cited page →
Frequently asked
- We do not have server logs. Can we still run this?
- Partly. Indexability and orphan checks work without logs, and they catch a real share of access problems. What you lose without logs is the ability to tell a bot that never came from a bot that came and got refused, and that distinction is usually the one that settles the argument.
- How often should this run?
- Often enough to catch a regression before the engines re-fetch, which in practice means a standing check rather than an annual audit. Access breaks on deploy days, not on a schedule you control.
- If we fix the robots rule, do the citations come back?
- Eligibility comes back. Citations are a separate contest that your content still has to win. The honest framing is that the fix removes a disqualification, and what happens next depends on whether the page deserves to be a source.