SEO Fundamentals 4 min read

Crawl Budget

Definition

Crawl budget is the amount of attention Google is likely to spend crawling your site over a period of time. It is how many URLs Google can and wants to visit before moving on. Google explains crawl budget as a mix of crawl rate and crawl demand. It also says this is mainly a concern for very large or frequently updated sites. For many smaller sites, keeping your sitemap updated and monitoring index coverage is usually enough.

Related terms to know

A few technical terms make crawl budget easier to understand.

  • Crawl rate is how fast Google crawls your site without overwhelming your server.
  • Crawl demand is how much Google wants to crawl your pages based on things like importance, freshness, and popularity.
  • A canonical URL is the main version of a page when several very similar or duplicate versions exist.
  • A sitemap is a file that tells Google which pages on your site exist and which ones you consider important.
  • Noindex tells Google a page should not appear in search results. For that to work, Google must still be able to access the page and see the instructions. (Read about crawlability and indexability to learn more).
  • Robots.txt tells crawlers which URLs they are allowed to access, but it is not a reliable way to keep a page out of Google's index.

Why is crawl budget important for fitness and wellness brands?

Fitness and wellness websites tend to create dozens of dedicated pages (class pages, trainer pages, location pages, blog tags, product filters, supplement variants, campaign URLs, and so much more.) These can all add up quickly.

When that happens, Google can end up crawling pages that aren't relevant to your business goal, or even pages that won't contribute significantly to your organic traffic. All the while, your key service, product, and content pages get less attention.

How to help Google crawl the right pages on your site

Start by deciding which pages need the most search visibility and which ones should get priority, in line with your business goals. That's usually core service pages, main product pages, key location pages, and your best educational content.

Then make sure your sitemap prioritizes these. That means only including the most important pages. Google will likely still find the rest if you have good internal linking, but since your sitemap is a major directory, that's where the crawler starts.

Additionally, Google specifically recommends listing canonical URLs in your sitemap, because that helps search engines understand which version you want treated as the main one.

How to talk to your agency about this

Questions to ask your agency

"How many duplicate, low-value, and non-converting URLs do we have on our sitemap? How are we allocating our crawl budget and focusing on the most important revenue-driving pages?"