Crawl budget
Crawl budget is the set of URLs a search engine can and wants to crawl on a site, set by how much crawling the server can handle (crawl capacity) and how much the engine wants the content (crawl demand).
Google defines crawl budget as the set of URLs Googlebot can and wants to crawl on a site. Two things set it. The crawl capacity limit is how much Google can crawl without overloading the server; it rises when the site responds quickly and falls when it slows or returns errors. Crawl demand is how much Google wants to crawl, based on the site’s size, how often it changes and how popular its pages are.
Most sites never need to think about it. Google’s guide is written for sites with more than a million unique pages that change about weekly, sites with more than 10,000 pages that change daily, and sites with many URLs reported as “Discovered, currently not indexed”. Google calls these rough estimates.
Why it matters now: sites that publish pages at scale, including AI-generated ones, can create more URLs than Google wants to crawl, and the excess sits undiscovered or unindexed.
Practical note: cut duplicate and parameter URLs, fix soft 404s and server errors, keep sitemaps current and make sure every URL you publish is one you want crawled.