The Crawl Coverage report shows you how often Googlebot crawls your pages.
In this article we will explain:
The Crawl Coverage Report
How the Crawl Coverage Report Works
How to use the Crawl Coverage Report
The Crawl Coverage Report
The Crawl Coverage report is an alternative log file report that helps SEO, product and digital teams understand how often Googlebot crawls your important pages.
It is available for Scale and Company accounts. See our pricing page for more details.
The Crawl Coverage report groups crawl times into five Days Since Last Crawl time buckets:
1-30 Days - Pages crawled in the last 30 days.
31 - 60 Days - Pages crawled in the last 31-60 days.
61 - 100 Days - Pages crawled in the last 61-100 days.
101 - 130 Days - Pages crawled in the last 101-130 days.
130+ Days - Pages crawled over 130 days.
The report shows the last 90 days of crawl stats for your website:
As well as the 90-day chart you can also view all the page URLs in a data table that is sorted by earliest Days Since Last Crawl:
As well as the Crawl Coverage report we also provide a breakdown the Days Since Last Crawl time bucket reports (1-30 days, 31-60 days, 61 - 100 days, etc.). And within the time bucket reports we show indexed vs not indexed pages within these time buckets.
How the Crawl Coverage Report Works
The Crawl Coverage report works by using the Last Crawl Time metric from the URL Inspection API.
Our tool uses the Last Crawl Time metric to record a Days Since Last Crawl metric for each page URL.
Then we group the pages on the Days Since Last Crawl metric.
Why do we group the pages into the 5 time buckets?
The reason we have chosen these time buckets is based on our research of 1.4 million URLs into the 130-day indexing rule.
Our research found that if you break down the indexing data from the URL Inspection API into time buckets you can spot whether your pages are at risk of being deindexed.
As we have the data and the technical expertise, we now allow our customers to spot which page URLs are likely to be deindexed.
How to use the Crawl Coverage Report
The Crawl Coverage report can be used to identify the following:
Indexed pages at risk of being deindexed
Frequently crawled not indexed pages
Alternative log file analysis
1) Indexed pages at risk of being deindexed
The Crawl Coverage reports can be used to identify indexed pages at risk of being deindexed.
Our tool can help you quickly identify the indexed pages at risk of being deindexed by looking at the Days Since Last Crawl: 101 - 130 days report.
Our first-party research study identified that if a page has not been crawled in 130+ days then it has 99% chance of being deindexed. And if its not been crawled in 150 days then it has 100% chance of being deindexed.
By looking the Days Since Last Crawl: 101 - 130 days report…
…and filtering on Submitted and Indexed…
…you will be able to spot the important indexed pages which are at risk of being deindexed in seconds.
We plan on allowing customers to get custom alerts in the future so you don’t have to rely on doing the work manually. And adding this data will be added to our daily email alert.
2) Frequently crawled not indexed pages
The Crawl Coverage report can be used to identify the not indexed pages which have been recently crawled by Googlebot.
Why is this useful?
When Google actively removes pages from its search results (indexed → not indexed) there are still signals which tell the Googlebot crawler to recrawl page URLs.
The most common signal is pages are still linking to this not indexed page. This usually happens because:
A website is internally linking to these not indexed pages, or
An external website is externally linking to these not indexed pages.
Our report can help identify important not indexed pages that need action from your team.
By looking the Days Since Last Crawl: 1 - 30 days report…
…and filtered on not indexed pages using the chart (by unselecting the indexed pages)…
…you will be able to spot the important not indexed pages which have been crawled in 30 days or less.
3) Alternative log file analysis
The Crawl Coverage report can be used as an alternative log file analyser.
Log file analysis is the technique of using log files to understand how Googlebot interacts with the website. Indexing Insight turns the indexing data into an alternative log file analyser.
The Crawl Coverage report can help digital teams understand:
Which important pages are being crawled frequently
Which important pages are being crawled infrequently
Which important pages are less of a priority to crawl
Which important pages are at risk of being deindexed
Which pages have been actively forgotten by Google
Any digital team can use the Crawl Coverage report to understand how Googlebot is interacting with important pages on your website. And combine crawling and indexing data to get even more insights.
The best bit is that you don’t need to spend endless hours working with multiple teams to get this data.
Once you set up Indexing Insight the Crawl Coverage report data is automatically calculated and reported for any website being monitored by our tool.