Why is not all the traffic of search robots displayed in the logs?

In this article, we will explain why a situation may occur when you cannot see all the visits of search engines in the logs data tables in JetOctopus. We will show you what to do if you are sure that a search engine visited your website, but there are no loglines displaying these visits. You will also find out why not all search engines are displayed in the JetOctopus reports and how to configure reports for the specific search engines.

Why not all search engines are in the reports

You will find most of the popular search and advertising robots in the logs data tables and reports in JetOctopus. Among them are GoogleBot (mobile and desktop), Bing, AdsBot, Baidu, Google Images, etc. You can configure filtering in any report from the “Logs” menu to see reports only for selected robots.

Separately, we collected marketing bots, including SemRush, Ahrefs, Majestic, etc. There are a lot of marketing tools and bots scanning your website on a constant basis. They usually don’t overload websites, so we didn’t single them out.

We collect the data with all other bots, which are not so popular, in the “Unknown bots” filter. Among them are PetalBot, Naver, etc.

If you need to customize reports for a special bot, you can manually extract the desired bot by user agent from the raw logs. 

If you want to explore the dynamics of visits for DuckDuckGo, for example, go to the logs data table – “Raw logs”. Next, select the filter “User Agent String” – Contains – “duckduckgo”.

We also recommend finding the official IP addresses of the desired search engine and adding an IP filter.

Also, you can use validation status in filters. Use “Bot validation status” – “=Equal” – “Valid” to see bots with corresponding reverse DNS lookup.

What to do if there are no visits of GoogleBot and other well-known search robots in the JetOctopus logs

If you meet a situation where you are sure that search engines have visited your website, but the records of those visits are not shown in the JetOctopus logs, check the following points.

1. If you imported the log files yourself, check that the files contain all periods you want to analyze and that you have downloaded all the needed files. Check it in “Manage logs” – “Upload files”. Here you will find a list of all downloaded files.

2. Check whether the logs for the HTTP and HTTPS versions of your website are recorded in the same location/files. Also, check whether the logs from different subdomains of your site are recorded in the same location. It is possible that there are different logs for each domain and/or HTTP/HTTPS versions of a website. If this is your case, then additional integration is required for each subdomain or for HTTP/HTTPS versions.

If your question is not resolved after reading this article, we will be happy to help. Contact the technical support department. Thank you!