What is the crawl budget?
The crawl budget is the amount of URLs that a search engine bot, such as Google, allows a certain website to crawl per crawl, how often it crawls the upper levels and how often it crawls deep. The information architecture of the website plays an elementary role here: if all the content of a website is easily accessible, i.e. accessible with just a few clicks from the homepage, the crawler can crawl the website more easily. Broken links and pages without inbound links, on the other hand, hinder the crawler. Providing a complete XML sitemap also helps to use the crawl budget more efficiently.
Google itself determines how high the crawl budget is for a website. The more trust a website has, the higher the PageRank etc., the higher the budget. By excluding certain subpages with little content, e.g. the legal notice or the login page, the website operator can "control" the crawler and make better use of the crawl budget.
Further information:
https://moz.com/blog/an-illustrated-guide-to-matt-cutts-comments-on-crawling-indexation
https://www.sistrix.de/news/crawling-und-indexierung-umfangreicher-webseiten/
Video:
