How to analyze what types of GoogleBots are visiting your website and why it matters

Google is one of the most popular search engines. Many SEO specialists closely follow the updates of algorithms of this search engine. In particular, a lot of attention is paid to changing the version of the browser with which the Googlebot imitates users. Also, the ratio of crawling performed by desktop and mobile robots is constantly changing. This is important for prioritizing the tasks of working on the website too.

With the help of JetOctopus, you can analyze what types of Google search robots are visiting your website. Additionally, we’ll explain why it’s important to monitor the types of Google robots that visit your website.

Audit of the types of Google robots crawling your website

Google has different types of robots. SEO specialists are most interested in analyzing GoogleBot Search. This robot includes two types of robots: those that simulate mobile devices (GoogleBot Mobile or Smartphone) and those that simulate desktop users (GoogleBot Desktop).

If your website is crawled mainly by GoogleBot Smartphone, this is one of the signs of Mobile First Indexing. Please note that if your site has not yet been switched to Mobile First Indexing, and the frequency of crawling by GoogleBot Mobile is increasing, this may indicate a switching to MFI.

To check which types of GoogleBot Search are crawling your website, go to the “Logs” menu – the “Bot Dynamics” report. Next, select bots type – “GoogleBot Search” (includes Smartphone and Desktop).

On the “Mobile vs desktop” chart, you can see the ratio of visits by mobile and desktop bots. The screenshot shows that the website is scanned mainly by GoogleBot Desktop. Probably, this website has not yet switched to Mobile First Indexing.

However, if you change the visualization to “Pages”, you can see that most pages are scanned by GoogleBot Mobile.

It is also necessary to analyze the dynamics of scanning your website by different types of robots. We show this data in the “Dynamics of mobile/desktop bots” report. Here you can also choose how to analyze the data: by the number of visits or by the number of scanned pages.

Analysis of GoogleBot versions is important too. This will help you and the developers to optimize the performance of your website. After all, for different Chrome/browser versions, performance metrics can have different results.

To analyze GoogleBot versions, select a bot type in the “Bot Dynamics” report and go to the “GoogleBot versions” chart. Here you can also select pages or visits.

On the “Dynamics of GoogleBot versions” chart, you can find the dynamics of different versions during the selected period.

What other types of Google robots should be analyzed?

In addition to Google Search, there are other Google robots. For example, GoogleBot News, GoogleBot Images, Web Light, AdsBot, etc. The effectiveness of your website, including paid traffic, also depends on the crawling of your site by these robots.

Google has published a complete list of the robots it uses. You can analyze any user agent from this list using JetOctopus.

We have selected the most popular bots in the “Bots” filter. You can use this filter on most reports in the “Logs” menu.

When analyzing the activity of these robots, we recommend checking:

• scan frequency;
• status codes (200, not 200, 5xx, 4xx, etc.);
• load time;
• bytes loaded;
• referers;
• blinking status codes.

If you did not find the desired robot in the filter, go to the data tables – “Raw logs”. Next, filter robots using the “User Agent String” filter. You can also add a “Bot Status” filter to check if the bot is valid (not a fake bot). Add a filter for IPs used by the required robots to see the activity of 100% valid bots.

Why do you need an analysis of the traffic of different types of GoogleBots

Google is one of the most popular search engines. If you expect to receive traffic from this search engine, it is necessary to optimize your website in accordance with Google’s requirements. But Google includes many different types of bots. Knowing which specific robots visit your website, you can more effectively build your optimization strategy and create technical tasks.

In addition, with the analysis of the visits of different types of robots, you can audit the following points:

• does Mobile First Indexing work for your website;
• which pages to optimize first: if most pages are scanned by GoogleBot Smartphone, and only some by GoogleBot Desktop, this may mean that the latter are not well optimized for mobile devices;
• and vice versa, you can strengthen those pages that are most often scanned by GoogleBot Mobile;
• how to prioritize developer tasks to improve performance metrics;
• strengthen the image optimization strategy to get organic image traffic;
• check advertising campaigns;
• check the effectiveness of the news site – if GoogleBot News often crawls your news website, but you do not receive traffic, you need to review the strategy.