Skip to content
Search and discovery

What is Crawl budget?

Crawl budget is the number of URLs a search engine will fetch from a site in a given period, set by how much load the server can absorb and how much demand the engine has for the site’s content. It becomes a practical constraint mainly on large sites.

Definition

Definition

Crawl budget is the number of URLs a search engine will fetch from a site in a given period, set by how much load the server can absorb and how much demand the engine has for the site’s content. It becomes a practical constraint mainly on large sites.

Most sites never hit the limit. Google’s own guidance is that sites under a few thousand URLs generally do not need to think about it.

Where it does bite, the cause is usually self-inflicted: faceted navigation generating near-infinite parameter combinations, session IDs in URLs, or paginated archives with no crawl ceiling.

The usual error

Where this goes wrong in practice

Blocking crawl of pages in robots.txt to "save budget" when those pages carry canonical or noindex signals. A blocked URL cannot be read, so the signal never gets seen.