Crawl log report
also called Log file reportServer log analysisCrawl budget reportBot crawl reportLogfile insights
Server-log truth about crawling: requests by bot, status-code distribution, response times, which URL groups absorb the crawl budget and which valuable pages are fetched rarely or not at all. It is the R1 gate, nothing above it is possible for a page no crawler fetched, and it is the single most-run family in the estate, which reflects how often it is checked rather than how rarely it changes.
Who it is for, and what it settles
- Audience
- Technical SEO
- Job to be done
- Show what search crawlers actually fetched, with what status, and where crawl effort is being spent or wasted.
- Cadence
- Weekly
The metrics, in three tiers
Headline numbers carry the story. Diagnostics explain a headline that moved. Guardrails stop a wrong conclusion. Every row links to the definition, the formula, the caveats and what the metric is not all live on the metric page.
Headline What you lead with 3
Diagnostic Reached for when a headline moves 4
Required data sources
Skills that produce or interpret it
Method rungs covered
- R1 Reachable , covered
- R2 Retrieved , not covered
- R3 Referenced , not covered
- R4 Represented , not covered
- R5 Rewarded , not covered
It covers the bottom rung, so this report can tell you whether the numbers above it are sound in the first place.
Ask for it in one sentence
The prompt pack for this report, copy one, or watch it play in the simulator.
- "What is Googlebot crawling on our site?" Simulate this → Prompt gallery →
- "Where is crawl budget being wasted?" Simulate this → Prompt gallery →
- "Are crawlers hitting errors?" Simulate this → Prompt gallery →
- "Which important pages are not being crawled?" Simulate this → Prompt gallery →
What to say out loud when you present it
Log coverage depends on which hosts, CDNs and edge layers forward logs, a bot missing from the file may be blocked at the edge rather than absent, and the two look identical here. Bot identity is claimed in the user agent unless reverse-DNS verification is in place, so 'Googlebot' counts include impostors. Log delivery lags from hours to overnight and varies by site; check the max available date before reading the last day.
Related workflows and use cases
Watch a run that builds this report
All runs →How this entry was verified
Knowing the contents is step one.
Someone still has to assemble it, defend the numbers and present it. Here is the rest of the showcase.