Crawl budget goes where you're not looking

Two counts describe your site. Your CMS knows how many pages you published. Your logs know how many addresses the site actually answers to, because overnight the crawlers walked every one of them. The second number is the larger one.

The difference between those counts is where your crawl budget goes. Parameterized variants breeding in a faceted nav. Paths buried many clicks deep. Addresses that answer with an error or a redirect. Crawlers spend real fetches on all of it, and every fetch spent there is one your money pages did not get.

None of this is visible from inside a CMS, which shows you the site you meant to publish. The crawler sees the site your links actually generate. And none of it is about pleasing a crawler for its own sake. Fetch allocation is the upstream constraint on how fast new pages get discovered and updated ones get re-read, which makes it an editorial speed problem wearing a technical costume.

Waste comes in three shapes

Deep, parameterized, and broken. One check separates them, and it separates them because each shape belongs to a different owner. Deep URLs sit many clicks from your homepage, so crawlers reach them rarely and rank them accordingly. Parameterized URLs are where sort orders, session tokens and locale switches mint endless addresses for identical content. Non-200s are the redirects and errors that consume a fetch and hand back nothing.

Keeping them apart is the point. Depth is an architecture and internal-linking problem. Parameters are a link-generation problem. Non-200s are a repair queue. Roll all three into one crawl-health score and you have hidden the fact that three different owners need three different tickets.

Depth deserves one more note, because it is the shape teams dispute most. A page many clicks from your homepage may still get discovered through a sitemap, but the link graph sets how often it gets revisited, and revisit frequency is what keeps its ranking current. Deep pages go stale in the index long before anyone notices them going stale on the site.

One parameter can hijack your international presence

The sharpest version we have watched came from a locale and currency parameter, minting URL variants of every product page. Canonical tags pointed at the clean versions. The search engine ignored them at scale and chose the parameterized variants anyway, and wrong-language pages ended up ranking across most of that site's international markets. Buyers in one country landed on pages priced and phrased for another.

Two lessons traveled out of it. Canonicals are hints. When enough internal links point at parameterized addresses the hint loses to the link graph, so the durable fix was a ticket that stripped the parameters out of internally generated links, rather than another round of canonical tuning.

The second lesson is who built the detector afterwards. Their own engineer did, which is the right instinct. A waste pattern that expensive deserves standing detection, and the crawl check is that detector without the engineering project attached.

Do small duplicates really matter?

At scale they matter more than the dramatic ones do. A trailing-slash variant answering alongside its clean twin looks harmless in isolation. Multiplied across a large site it becomes a steady tax: two fetches for every one page of value, forever. We have watched that exact pattern eat a meaningful slice of a site's crawl activity while every individual URL looked fine.

These are the leaks a quarterly skim never catches, because no single row looks alarming enough to write down. Ranking waste by fetch volume catches them, and a small duplication pattern at scale outranks a dramatic one in a corner of the site nobody visits.

There is a second cost underneath the fetches. Duplicate addresses split link equity and confuse canonical selection, so the wasted fetches are usually the smallest of the damages. They are just the most measurable one.

The fix ships as a ranked worklist

Crawl budget has no setting to flip. The only output that changes anything is a ranked worklist with names on it: the parameter family for your platform team, the depth problem for whoever owns navigation and internal linking, the non-200 queue for the maintenance rotation. The internal linking article covers depth in detail, because linking is how depth gets fixed without an architecture rebuild.

Rank the list by fetches wasted. Cap it at what those teams can actually ship this month, and rerun the check after each batch lands. Budget reallocates on its own once the waste stops, and the reward arrives as faster discovery of the pages you care about.

One caution on the measurement itself. Run this on verified crawler data, because impostor traffic inflates fetch counts precisely on the junk URLs that scrapers love to walk. The article on fake Googlebots covers the verification step that keeps a worklist trustworthy.

See also: how to tell a real Googlebot from a costume →

AI crawlers pay the same tax

Every argument above doubles for them. GPTBot and its peers fetch with budgets of their own, and a site that spends a third of its fetches on parameters and duplicates is teaching every crawler, search and AI alike, that its address space is expensive to explore.

This is the ground floor of readiness. Can crawlers fetch, render, and index the pages that matter, without drowning in the ones that do not. The technical and crawl health article frames where this sits in your larger program. The indexation and vitals article picks up the next question, which is whether the pages that do get fetched make it into the index and load fast enough to matter.

Clean the waste first. Everything downstream inherits the savings.

See also: whether the fetched pages make it into the index and load fast enough →