Unique pages crawled
also called Unique URLs fetched by botsPages reached by bots
Definition
The number of distinct URLs a named crawler fetched in the period, counted from server logs.
How it's calculated
A distinct count of the language-variant-normalised full URL over the matching log records.
Scope, grain and dimensions
- Grain
- One count per bot per date range, or per bot per time bucket.
- Dimensions
- bot_type · date
- Required filters
- date_range
- Aggregation
- A distinct count, so it is never summed across bots or buckets, the same page fetched by two bots is one page for each of them and cannot be added.
- Metric type
- raw measure · distinct URLs
Data sources
Where you'll see this
Named reports that normally include this metric.
AI bot activity reportCrawl log reportInternal linking reportTechnical SEO audit report
Skills and analyses that use it
Skills carry the judgment; the analysis verbs do the reading.
Analysis verbs
ai_crawler_analysis
Method rungs and levers
This sits on the bottom rung: if it reads badly, every measurement above it is unreliable.
levers L1 Technical & crawl health
Ask Quattr
- "How much of our site are AI crawlers reaching?" Simulate this →
- "Is crawl coverage growing?" Simulate this →
How to read it
The coverage half of crawl activity. Read against the hit count it separates two different situations that look alike in a total: broad shallow crawling, and heavy repeat fetching of a small set.
Caveats, freshness and failure modes
A distinct count, so bot-level or bucket-level values cannot be added into a site total.
URLs are normalised across language variants before counting, so localised versions of one page collapse together.
It counts what was fetched, not what exists, it is not a share of your site unless you separately know the site's page count.
Same bot-identification and CDN caveats as the hit count: user-agent matching only, and origin-served requests only.
- Freshness
- hours to nightly, depends on how logs are shipped; the card carries the ingestion time.
Common failure modes
- Summing per-bot unique-page counts.
- Presenting it as a percentage of the site without a denominator from the crawl.
Not the same as
The confusions that cause the most wrong decisions.
Unique pages crawled Bot hits Compare definitions →
One counts requests, the other counts distinct URLs. The ratio between them is the crawl-repetition signal and neither number carries it alone.
Watch this metric read in a real run
All runs →
Related metrics and workflows
Verification
A definition is the smallest part of this.
The measurement matters because something acts on it. Here is the rest of the showcase, in the order most people find useful.