Crawl budget is the number of URLs Googlebot can and wants to crawl on your site within a given period. For most small sites it is not worth worrying about. For large shops, classified sites, publishers and anything with tens of thousands of URLs, it becomes an important part of technical SEO.
The two parts of crawl budget
- Crawl capacity limit: how many requests Google can make without overloading your server. Fast, stable servers get a higher limit; slow responses and 5xx errors lower it.
- Crawl demand: how much Google wants to crawl, based on how popular and how frequently updated your pages are.
Who needs to care?
Google’s own guidance says crawl budget mainly matters for sites with over a million pages, or over 10,000 pages that change daily, or sites with many URLs stuck as “Discovered – currently not indexed”. If your blog has 200 posts, focus on content and links instead.
What wastes crawl budget?
- Faceted navigation creating endless filter combinations.
- Session IDs and tracking parameters in URLs.
- Infinite calendar or pagination spaces.
- Redirect chains and soft 404 pages.
- Large numbers of low-value or duplicate pages.
- Internal search result pages being crawled.
- Slow server response times.
How to optimise crawl budget
- Block worthless URL patterns with robots.txt (crawling) rather than noindex (which still requires crawling).
- Fix redirect chains and update internal links to final URLs.
- Return 404 or 410 for removed content rather than soft 404s.
- Keep the XML sitemap clean, with accurate lastmod dates.
- Improve server speed and caching.
- Strengthen internal links to important pages so Google sees them as a priority.
- Consolidate thin pages into stronger ones.
How to monitor crawling
The Crawl Stats report in Google Search Console (Settings > Crawl stats) shows total requests, response codes, file types and average response time. Server log analysis gives the full picture: exactly which URLs Googlebot requested and how often.
Myths about crawl budget
- Crawl rate is not a ranking factor. More crawling does not mean better rankings.
- Nofollow on internal links does not reliably save crawl budget.
- Requesting indexing in bulk does not increase your allowance.
Frequently asked questions
Does page speed affect crawl budget?
Yes. Faster server responses let Google crawl more pages in the same time.
Can I ask Google to crawl more?
Not directly. Improve speed, demand and site quality, and crawling follows.
Related terms
See how search engines crawl and index, HTTP status codes and the SEO encyclopedia.
Mr. SEO
Mr. SEO is the webmaster at SEO TT - SEO tips and tricks, a publication about all things SEO related.