July 25, 2022
by Sofia

Why is not all the traffic of search robots displayed in the logs?

In this article, we will explain why a situation may occur when you cannot see all the visits of search engines in the logs data tables in JetOctopus. We will show you what to do if you are sure that a search engine visited your website, but there are no loglines displaying these visits. You will also find out why not all search engines are displayed in the JetOctopus reports and how to configure reports for the specific search engines.

Why not all search engines are in the reports

You will find most of the popular search and advertising robots in the logs data tables and reports in JetOctopus. Among them are GoogleBot (mobile and desktop), Bing, AdsBot, Baidu, Google Images, etc. You can configure filtering in any report from the “Logs” menu to see reports only for selected robots.

Why is not all the traffic of search robots displayed in the logs - JetOctopus - 1

Separately, we collected marketing bots, including SemRush, Ahrefs, Majestic, etc. There are a lot of marketing tools and bots scanning your website on a constant basis. They usually don’t overload websites, so we didn’t single them out.

Why is not all the traffic of search robots displayed in the logs - JetOctopus - 2

We collect the data with all other bots, which are not so popular, in the “Unknown bots” filter. Among them are PetalBot, Naver, etc.

If you need to customize reports for a special bot, you can manually extract the desired bot by user agent from the raw logs. 

If you want to explore the dynamics of visits for DuckDuckGo, for example, go to the logs data table – “Raw logs”. Next, select the filter “User Agent String” – Contains – “duckduckgo”.

Why is not all the traffic of search robots displayed in the logs - JetOctopus - 3

We also recommend finding the official IP addresses of the desired search engine and adding an IP filter.

Also, you can use validation status in filters. Use “Bot validation status” – “=Equal” – “Valid” to see bots with corresponding reverse DNS lookup.

Why is not all the traffic of search robots displayed in the logs - JetOctopus - 4

What to do if there are no visits of GoogleBot and other well-known search robots in the JetOctopus logs

If you meet a situation where you are sure that search engines have visited your website, but the records of those visits are not shown in the JetOctopus logs, check the following points.

1. If you imported the log files yourself, check that the files contain all periods you want to analyze and that you have downloaded all the needed files. Check it in “Manage logs” – “Upload files”. Here you will find a list of all downloaded files.

Why is not all the traffic of search robots displayed in the logs - JetOctopus - 5

2. Check whether the logs for the HTTP and HTTPS versions of your website are recorded in the same location/files. Also, check whether the logs from different subdomains of your site are recorded in the same location. It is possible that there are different logs for each domain and/or HTTP/HTTPS versions of a website. If this is your case, then additional integration is required for each subdomain or for HTTP/HTTPS versions.

If your question is not resolved after reading this article, we will be happy to help. Contact the technical support department. Thank you!

About Sofia
Technical SEO specialist. Sofia has almost 10 years of experience, of which the last 5 years in JavaScript SEO. She is convinced that SEO is a very technical part of digital marketing. And without logs and in-depth data analysis, you can't do effective SEO.

Search

Categories

Find technical SEO errors at your website at free trial
Get exclusive tech SEO insights
We are tech SEO geeks who believe that SEO is predictable and numeric. Don’t miss our insigths!
Start your free trial
free 7-days trial