Crawl budget: what it is & when it actually matters

Crawl budget is the number of URLs a search engine will fetch from your site in a given period. Google frames it as the product of two things: the crawl rate (capacity) limit, which is how fast Googlebot can fetch without degrading your server, and crawl demand, which is how much it wants to fetch based on popularity, staleness and its assessment of quality.

Both halves matter and they fail differently. A site can have plenty of rate headroom and no demand, or abundant demand and a server that forces Googlebot to back off. Treating them as one number is how teams end up buying faster hosting to solve a content problem.

Crawl budget = rate limit x demand

The rate limit is a health signal. Googlebot increases concurrent fetches while response times stay low and status codes stay clean, and reduces them when the server slows or starts returning 5xx. You can lower it deliberately by returning a 503/429 status code (with a Retry-After header); you cannot raise it above what Google thinks is safe. A faster origin makes more crawl available, which is not the same as Google choosing to use it.

Demand is the interesting half. It rises with external links, with traffic, with how often the content genuinely changes, and with Google’s view of whether the URLs are worth having. It falls when a site is full of near-duplicates, soft 404s and pages that have never earned an impression.

Most crawl-budget problems presented to me are demand problems dressed as rate problems. The pages are not being crawled because Google has assessed the site and decided the marginal URL is not worth the fetch. Buying more server does nothing for that.

When crawl budget matters, and when it is a waste of attention

• Catalogues above roughly half a million URLs, especially where faceted navigation multiplies them.
• Sites where a large share of URLs are machine-generated: filters, parameters, sorts, internal search results.
• Inventories that change constantly: marketplaces, job boards, classifieds, travel, ticketing. Here the cost of slow crawl is stale prices and availability in the SERP.
• Any site that has just multiplied its URL count, whether through a replatform, a new filter component or a programmatic build.
• The weeks after a migration, when you need Google through the redirect map quickly.

When crawl budget does not matter: a 3,000-page B2B SaaS site. If your pages are not indexed, it is almost never because Google ran out of budget on three thousand URLs. It is because Google crawled them, assessed them, and declined. “Crawled — currently not indexed” in Search Console is a quality verdict, not a capacity message, and the difference is worth understanding before anyone opens a log file.

I have sat in rooms where crawl budget was the comfortable explanation for a content problem, and I have watched a quarter go into parameter handling on a site whose real issue was four hundred pages nobody wanted to read. The diagnosis is cheap. Do it first.

Log files are the only direct evidence

Everything else is inference. Search Console’s Crawl Stats report is genuinely useful for trend and for host status, and it is aggregated. Server logs tell you which agent fetched which URL, when, with what status and how long it took. There is no substitute.

• Pull at least 30 days, with full path, query string, status code, user agent and response time.
• Take them from the CDN, not just the origin (Cloudflare Logpush, Fastly, Akamai). Origin logs miss everything served from cache, which on a well-configured site is most of it.
• Verify Googlebot by reverse DNS. A substantial share of traffic claiming to be Googlebot in raw logs is not.
• Segment by directory and by template, not by URL. The finding is usually “39% of Googlebot requests went to /search“, never a single page.

The number I care about most is the share of verified Googlebot requests that hit URLs in your canonical sitemap set. On healthy large sites I expect that to be the majority. When it comes back at 20%, you have found the project.

When crawl budget is an issue, there are usual culprits

• Faceted navigation, and particularly combinations of two and three facets, which is where the URL count goes exponential.
• Tracking, session and affiliate parameters appended to otherwise fine URLs.
• Sort orders, view toggles, items-per-page switches, every one of which duplicates the entire category.
• Internal search result pages, which are infinite by construction and occasionally linked from templates by accident.
• Calendar and booking widgets with next-month links, which are literally infinite and will be crawled into 2043 if you let them.
• Print, AMP and alternate-format variants that nobody has used in years.
• Staging and dev subdomains that got indexed, and redirect chains left over from the last two migrations.

Crawl budget controls and why sequence matters

There are four controls and they are not interchangeable. Applied in the wrong order they cancel each other out, which is the most common technical mistake I find on large sites.

1. Do not link to it. This is the only control that operates before a fetch, and it is free. Crawlers are link-driven; a URL that is never linked is mostly never discovered. Solving faceted navigation in the front-end component (e.g. deciding which combinations render as <a href> at all) beats every downstream directive.
2. robots.txt disallow. Stops the fetch, which saves crawl. It does not prevent indexing: a disallowed URL with external links can still appear as a URL-only result. And because the page is never fetched, any noindex on it is never read. Never combine the two on the same URL. That single conflict is responsible for more stuck deindexing projects than anything else I see.
3. noindex. Requires the fetch, so it costs crawl, but it reliably removes the URL from the index, and over time Google crawls persistently noindexed URLs less. This is the right control for pages that must exist for users and must not exist in search.
4. rel=canonical. The weakest of the four. It is a hint, not a directive, it requires the fetch, and substantially lags any downstream crawl savings. It consolidates signals between genuine duplicates, which is a different job from budget management.

Teams reach for canonical first because it is the easiest thing to ship: one line in a template, no negotiation with the front-end team about which links render. It is also the least effective of the four at the problem they are trying to solve. Do the linking work.

Two supporting details worth getting right: return 410 rather than 404 for content deliberately removed in bulk, because Google retires those faster; and make sitemap lastmod honest, because a sitemap where every URL claims to have changed today is a sitemap Google stops trusting for prioritization.

Where to start

1. Check Search Console first: if the problem is “crawled — currently not indexed”, stop and go look at the content.
2. Get 30 days of CDN logs and calculate the share of verified Googlebot requests hitting sitemap URLs.
3. Segment the waste by directory and rank it. The top three segments are usually 80% of the loss.
4. For each one, work down the control order: stop linking first, disallow second, and only then consider directives that require a fetch.
5. Re-measure the same ratio after two crawl cycles. Rankings are the wrong success metric here; crawl distribution is the right one.

On a large e-commerce site this work usually pays for itself in a quarter. On a mid-size site it usually does not need doing at all, and I would rather tell you that early.