Crawl budget is the set of URLs Google can and wants to crawl on your site. It is set by how much your server can handle and how much Google wants your content. Per Google, only very large or fast-changing sites need to manage it. A small site whose new pages get crawled quickly can ignore it.
What does crawl budget mean?
Google's documentation defines it as the set of URLs Googlebot can and wants to crawl, counted per hostname. Two things shape it, according to Google's guide to managing crawl budget for large sites.
- Crawl capacity limit. How much Google can fetch without overloading your server. It goes up when the site responds quickly and steadily, and down when Google sees slowdowns, server errors (5xx) or rate-limit responses (429).
- Crawl demand. How much Google wants to crawl. It depends on how many URLs Google knows about, how popular they are, and how stale they are.
The budget is the lower of what your server allows and what Google wants. Fixing demand does nothing if your server is slow, and a fast server does not create demand for pages nobody links to.
Does your site need to worry about crawl budget?
Probably not. Google says its guide is for these cases:
- Sites with more than 1 million unique pages whose content changes about once a week.
- Sites with more than 10,000 unique pages whose content changes daily.
- Sites where a large share of URLs show as "Discovered - currently not indexed" in Search Console.
Google adds that smaller sites whose pages are crawled the same day they are published generally do not need the guide. These are rough estimates, not hard cutoffs, but they show the scale. A blog with 200 posts is nowhere near it.
If your pages are not getting indexed and you run a small site, crawl budget is rarely the cause. Quality, duplication and weak internal links are far likelier. Start with why Google crawls pages but does not index them.
What wastes crawl budget?
Google lists these as the main drains on a large site:
- Duplicate URLs, such as the same list sorted in different ways.
- Soft 404 pages, which look like errors to a visitor but return a normal status code, so Google keeps fetching them.
- Long redirect chains.
- Pages marked
noindex, which Google still has to request before it can see the tag. - Permanently removed pages that do not return a 404 or 410 status.
How do you improve crawl efficiency?
- Consolidate duplicates. Pick one URL per page and point the others at it. Our canonical URL guide covers how.
- Block pages that should never be crawled with robots.txt. Google notes that blocking is not a way to move budget to other pages on demand. Freed budget only matters if you are already hitting your limit.
- Return 404 or 410 for pages that are gone for good. Google calls a 404 a strong signal not to crawl that URL again.
- Fix soft 404s. Search Console's Page Indexing report lists them.
- Shorten redirect chains so each old URL goes straight to its final destination. See 301 vs 302 redirects.
- Keep your sitemap current and include a
lastmoddate for pages that changed. More on that in do you need an XML sitemap. - Speed up the server. Fast, steady responses raise the capacity limit.
Should you use robots.txt or noindex to control it?
For crawl control, robots.txt is the right tool and noindex is the wrong one. Google says to avoid noindex for this purpose, because the page still has to be requested before the tag is read. If your goal is to keep a page out of search results rather than save crawling, the answer changes. See robots.txt vs noindex for how to choose.
Where can you see how Google crawls your site?
Search Console has a Crawl Stats report. Per Google's Crawl Stats help page, it shows total crawl requests, download size, average response time, host status, response codes, file types, crawl purpose and Googlebot type. You find it under Settings, then Crawl stats, and it is only available for root-level properties.
Look for three things: a sudden drop in requests, a rising average response time, and a high share of non-200 responses. Each points to a different fix.
Frequently Asked Questions
Is crawl budget a ranking factor?
No. It controls how much of your site Google fetches, not how your pages rank. A page must be crawled to be indexed, but being crawled does not make it rank.
How do I know if I have a crawl budget problem?
Signs include many URLs stuck in "Discovered - currently not indexed", new pages taking weeks to appear, and Crawl Stats showing most requests going to low-value URLs. On a small site, check content quality first.
Does a bigger sitemap increase crawl budget?
No. A sitemap helps Google find URLs and see which changed. It does not raise the number of requests Google makes to your server.
Do 404 errors hurt crawl budget?
A few do not. Google treats a 404 as a signal to stop crawling that URL. The waste comes from soft 404s and from removed pages that still return a normal status.
Can I request a higher crawl rate?
Google's guide points to server speed and consistency as what raises your capacity, and to useful, linked content as what raises demand. There is no setting that asks for more.