Monitoring, Optimizing, and Staying Ahead

Factors That Affect Your Crawl Budget

According to Google, the biggest issue that affects crawl budget is low-value URLs. Having too many URLs that present little or no value, yet are still on the crawler’s path, will use up the available budget and prevent Googlebot from accessing more important assets. 

The problem is that you might not even realize that you have many low-value URLs - many are created without your direct impact. Let’s see how that happens, typically.
 

How Websites Generate Low-Value URLs

#1. Faceted Navigation

Faceted navigation refers to different ways users can filter or sort out results on a web page based on different criteria. For example, any time you use Ross-Simons’ filters to fine-tune the product listing, you use faceted navigation.

But faceted navigation, although helpful for users, can create issues for the search engines. 

Filters often create dynamic URLs which, to Googlebot, might look like individual URLs to crawl and index. This can use up your crawl budget and create a lot of duplicate content issues on the site. 

Faceted navigation can dilute the link equity on the page too, passing it to dynamic URLs that you don’t even want to be indexed. 

There are ways to overcome this:

Option 1

Add a “nofollow” tag to any faceted navigation link.

This will minimize the crawler’s discovery of unnecessary URLs and therefore reduces the potentially explosive crawl space that can occur with faceted navigation. 

Option 2

Add a “noindex” tag to inform bots of which pages are not to be included in the index.

This will remove pages from the index, but there will still be crawl budget wasted and link equity that is diluted.

Option 3

Use a robots.txt disallow. For URLs with unnecessary parameters, include a directory that will be robots.txt disallowed. This lets all search engines freely crawl URLs that you want a bot to crawl.

For example: we could disallow prices under $100 in the robots file.

Disallow: *?prefn1=priceRank&prefv1=%240%20-%20%24100

Option 4

Canonical tags allow you to instruct Google that a group of pages have a preferred version. Link equity can be consolidated into the chosen preferred page utilizing this method. However, the crawl budget will still be wasted.

#2. Session Identifiers/On-site duplicate Content

Similarly, URL parameters - like session IDs or tracking IDs - or forms sending information with the GET method will create many unique instances of the same URL. 

Those dynamic URLs, in turn, can cause the duplicate content issue on the site and use up much of the crawl budget, in spite of the fact that none of those assets are, in fact, unique.

#3. Soft 404

A "soft 404" occurs when a web server responds with a 200 OK HTTP status code, rather than the 404 Not Found, even though the page doesn’t exist.  

In this case, the Googlebot will attempt to crawl the page, using up the allocated budget, instead of moving on to actual, existing URLs.
 

#4. Hacked Pages

Unfortunately, hacked pages, included in the hacker attack, can increase the list of URLs a crawler might attempt to access. If your site got hacked, remove those pages from the site and serve Googlebot with a 404 Not Found response code. 

Seeing hacked pages isn’t anything new to Google, and the search engine will drop them from the index promptly. But it will do so only if you serve it a 404, of course. 

#5. Infinite Spaces and Proxies

Infinite spaces are near-endless lists of URLs that Googlebot will attempt to crawl. Infinite spaces can occur in many ways - but the most common includes auto-generated URLs by the site search. Some sites list on-site searches on pages, which leads to the creation of almost an infinite number of low-value URLs that Google will consider to crawl. 

Another common scenario is displaying a calendar on a page with a “next month” link. Each URL will have such a link, meaning that the calendar can generate thousands of unnecessary infinite spaces.

Google suggests some ways of dealing with infinite spaces. One is to eliminate the entire categories of those links in the robots.txt file. Doing so will restrict Googlebot from accessing those URLs from the start and save your crawl budget for other pages.

How to Optimize Crawl Budget

Based on the information above, you can tell how severe the issues with the crawl budget can be. The good news is that you can also optimize your site for the budget to maximize the time crawlers allocate for your website. 

There are some generic things that help. Improving overall site speed, for one. Avoiding duplicate content is another. Similarly, eliminating broken pages or simplifying the site’s architecture will help users and crawlers access your most critical content easily. 

However, below are other factors you should optimize to save the crawl budget from going to waste.
 

Reduce the number of crawlable URLs

The key to optimizing for the crawl budget is ensuring that the number of crawlable URLs does not exceed the budget. Otherwise, Google will cut off crawling the site once the budget is up. 

But with fewer URLs to crawl than the allocated number of requests, you stand a much greater chance for crawlers to access all your content. 

You can achieve this in many different ways. Here are some of the most common approaches:

#1. Fix 30x Redirects

Any broken link or redirect is a dead-end for Googlebot. When it comes to broken links, the crawler might consider that there is nowhere else to go and move to another website. With the redirects, it can travel through some hops. However, even Google recommends not to exceed five hops, otherwise, the crawler will move on. 

To avoid those issues, ensure that all redirected URLs point to the final destination directly, and fix any broken links.
 

#2. Remove 4xx - URL No Longer Active - Links

Removing any links to 404 pages also optimizes the crawl budget usage. As the website gets older, it's a higher risk to have internal links on your site that point to URLs that are no longer active.

#3. Optimize the Faceted Navigation

We’ve talked about the issue of faceted navigation already. You know that filters on a page can create multiple low-value URLs that use up the crawl budget. 

But that doesn’t mean that you can’t use the faceted navigation. Quite the contrary. However, you must take measures to ensure that crawlers do not try to access dynamic URLs that the navigation will create. 

When trying to determine how to solve this faceted navigation conundrum, there are a few solutions that you can implement. Which one to use, however, will rely heavily on what parts of the site should be indexed.

No Index

"Noindex" tags can be implemented to inform bots of which pages not to include in the index. This method will remove pages from the index, however, there will still be a crawl budget spent on them and link equity that is diluted.

Canonicalization

Canonical tags allow you to instruct Google that a group of similar pages has a preferred version of the page.

Nofollow

The simplest solution is adding the “nofollow” tag to those internal links. It will prevent crawlers from following those links and attempting to crawl the content.
 

#4. Remove Outdated Content

You do not have to physically delete those pages. However, blocking crawlers from accessing it will immediately reduce the number of crawlable URLs to free up crawl budget. 

#5. Block Crawlers from Accessing URLs that Should Not Be Indexed

You can save much of the crawl budget from going to waste by simply blocking crawlers from accessing URLs that you simply don’t need indexed.

These could be pages with legal information that your organization must provide to users, tags, content categories, or other assets that do not provide much value for searchers. 

The simplest way to do so is by either adding the “noindex” tag to those assets or a canonical tag pointing to a page you want to indexed instead.
 

#6. Cleaning the Sitemap

As we’ve discussed already, Google will prioritize URLs in the sitemap over those it discovers while crawling the site.

Unfortunately, though, without regular updates, the sitemap might end up clogged with inactive URLs or pages you don’t necessarily need to have indexed. Regular updates to the sitemap and cleaning up those unwanted URLs will free up crawl budget as well. 

#7. Using the Robots.txt File

A robots.txt file tells search engine crawlers which pages or files the crawler can or can't request from your site. Typically, the file is used to prevent crawlers from overloading sites with requests, however, it can help steer Googlebot away from certain sections of the site and also free up the crawl budget. 

One thing to remember, though, is that the robots.txt is only a suggestion to Googlebot - it’s not a directive it has to absolutely and unconditionally follow every single time. 

#8. Improve the Site Speed

Google has stated openly that improving site speed provides not only a much better user experience but also increases the crawl rate. Making pages load faster, therefore, is bound to improve the crawl budget usage. 

Optimizing page speed is a vast topic, of course, and involves working on many technical SEO factors.

At minimum, at seoClarity, we recommend that you enable compression, remove render-blocking JavaScript, leverage browser caching, and optimize images to ensure that Googlebot has enough time to visit and index all of your pages.
 

#9. Improve the Internal Linking Structure

Search engine bots find content on a site in two ways. First, they consult the sitemap. But they also navigate the site by following internal links.

This means that if a certain page is well interlinked from other content, its chances of having the bot discover it are significantly higher. An asset with little or no internal links can, most likely, remain unnoticed by the bot.

But this also means that you can use internal links to direct crawlers to pages or content clusters that you absolutely need to have indexed. 

For example, you can link those pages from content with many backlinks and high crawl frequency. By doing so, you’ll increase the chances that Googlebot will reach and index those pages quicker. 

Optimizing the entire site’s architecture can also help with freeing up the crawl budget. Having a flat but wide architecture, meaning that the most critical pages are not too far from the homepage, will make it easy for Googlebot to reach those assets within the available crawl budget.

How to Monitor Your Crawl Budget

An important aspect of optimizing your crawl budget is monitoring how Googlebot visits your site and how it accesses the content. 

There are three ways to do it, two of which lie within GSC:

Google Search Console

Google Search Console includes a breadth of information about your site’s stance in the index and the search performance. It also provides certain insight into your crawl budget. 

First of all, in the Legacy tools section, you’ll find the Crawl Stats report, showing Googlebot activity on your site within the last 90 days. 

From the report you can see that, on average, Google crawls 48 pages a day on this site. Assuming that this rate remains consistent, you can calculate the average crawl budget for the site with this formula:

Average pages per day * 30 days = Crawl Budget

In this case, it looks like this:

48 pages per day * 30 days = 1440 pages per month. 

Naturally, this is a crude estimate but it can give some insight into your available crawl budget. 

Note: Optimizing the crawl budget using the tips above should increase the number.

The Coverage Report in GSC will also show how many pages Google has indexed on the site and excluded from indexation. You can compare that number with the actual volume of content assets to identify if there are any pages the Googlebot has missed.

Server Log File Analysis

Without a doubt, the server log file is one of the biggest sources of truth about a site’s crawl budget. 

Why, because the server log file will tell you exactly when search engine bots are visiting your site. The file will also reveal what pages they visit the most often, and the size of those crawled files. 

11.222.333.44 - - [11/Mar/2020:11:01:28 –0600] “GET https://www.seoclarity.net/blog/keyword-research HTTP/1.1” 200 182 “-” “Mozilla/5.0 Chrome/60.0.3112.113”

On Mar 11, 2020 someone using Google Chrome tried to load https://www.seoclarity.net/blog/keyword-research. The “200” means the server found the file, weighing in at 182 bytes.

Now, it's true that server log-file analysis is not a simple task. You have to go through thousands of rows of server requests to identify the right bot and analyze its activity. 

At seoClarity, you have access to Bot Clarity, our site audit tool that, among other functionalities, provides a detailed log-file analysis.

With Bot Clarity, you can:

• Understand what the most important pages on your site are to the search engine crawl.
• Optimize crawl budgets to ensure bots crawl and index as many important pages on your site as possible.
• Find broken links and errors that search engine bots have encountered while crawling your site.
• Audit your redirects.
• Tie bot activity to performance, indicating which areas of the site you should focus your efforts.
  
  

Frequently Asked Questions

Do URLs I disallowed through robots.txt affect my crawl budget in any way?

No, disallowed URLs do not affect crawl budget. 

Does the meta noindex tag help to "save" crawling budget?

Unfortunately, no. It can, however, reduce the number of crawlable URLs and affect crawl budget positively this way. 

Does the nofollow directive affect crawl budget?

It actually depends. Any URL the Googlebot crawls will affect crawl budget.

The nofollow directive on a URL does not restrict the crawler from accessing the page as if another page on your site, or any page on the web, doesn't label the link as nofollow.

Can I control Googlebot with the "crawl-delay" directive?

No, the "crawl-delay" directive in the robots.txt is not processed by Googlebot.

Do alternate URLs and embedded content count in your crawl budget?

Yes, they do. Any URL that the crawler accesses counts against the crawl budget. As a result, any alternate versions of the content, AMP, hreflang tag defined content, or content triggered by JS or CSS will affect the crawl budget. 

Is crawling a ranking factor?

Google's ability to access and crawl the site’s content impacts its ability to rank it for relevant phrases.

So, although crawling may not be a ranking factor per se, it is the first step to getting the search engine to discover the content, understand it, index, and rank. 

Does site speed affect my crawl budget? 

Yes, and this has been confirmed by Google.

Do canonical tags save crawl budget?

Probably not (or not much). You have to pick a canonical and have to crawl the dups to see that they're dups anyway.

How do I increase the crawl rate?

You can adjust your crawl rate in the Google Search Console by using the Crawl Rate Settings option.