Regular scanning of the website by search robots ensures the updating of data in the index. The regularly crawled pages will also have an up-to-date snippet in the SERP. However, sometimes crawling is excessive and can cause server overload.
If your website is popular and constantly updated, extreme crawling by search engines can even lead to a lot of 5xx status codes. In this case, users can also receive 5xx if they use the site during peak crawling.
You may be interested in: How Googlebot almost killed an e-commerce website with extreme crawling.
Different search engines have different scanning frequencies. So, it is needed to monitor each search engine in order to identify in time the bot with the highest load on your webserver.
In this article, we will tell you how to monitor the frequency of scanning by search engines.
The most important information about the scan frequency can be found in the “Overview” report – “Logs” menu. On the “Dynamics of bots visits” chart, you can see the dynamics of visits by various bots and search engines for the selected period.
In this diagram, you can easily see the extreme crawling. In most cases, the scanning dynamics is displayed in real time. Therefore, you will quickly see if a particular bot has dramatically increased its scanning frequency. You can also set up a scan boost alert.
Pay attention to the fact that you can analyze crawled pages or bots visits.
If the number of pages on your website did not change, but the number of visits increased, this is one issue. The problem may be that search robots started scanning URLs with get-parameters, there were changes in the robots.txt file, indexing rules changed, etc.
Another issue is when the number of pages has increased, but the number of visits has not changed. This could mean that crawlers are not finding new pages or the crawling budget is not increasing because crawlers are scanning useless pages.
In addition, you can analyze the frequency of search robots crawling by subdomains. In the “Overview” report, there is a separate “Dynamics by subdomain” chart, where we show the ratio of visits by subdomains (in total for all bots).
It may be interesting: How to analyze what types of search robots visit your website.
For an in-depth analysis of the frequency of crawling by different robots, we have created a separate report “Bot dynamics”. Here you can find information about how each type of search engine crawls your website and the crawling budget for the selected search engine.
Select the desired period, bots type and domain.
Also, you can choose a segment of pages.
More information: How to use segments.
In the “Bot visits” chart you can find a comparison of the crawl rate (number of visits) and the number of visited pages.
Clicking on the chart will take you to the data table with a list of all visits.
When analyzing scan frequency, keep the following cases in focus: