Log file integration: everything you need to know

Analyzing search engine logs is a regular task for most SEO specialists. However, it is inconvenient to analyze logs using standard applications such as Kibana, Datadog, etc. And if you export loglines to CSV or other files, then working with this data using tables can be difficult. Therefore, it is a logical step to use JetOctopus as a log analyzer. In this article, we will talk about how to integrate logs into JetOctopus and what errors can occur during integration. Of course, we will tell you how to fix these errors.

Note that you can use the free 7-days trial to see how easy it is to integrate logs and how quickly you can get the first insights from search engines logs!

Log analysis is a cool thing to do to stay ahead of Google warnings and errors for your website. Therefore, we recommend that you try and get the maximum benefit from this information. Read about what to focus on when analyzing logs in the article: The Ultimate Guide to Log Analysis – a 21 Point Checklist.

Where to get logs of search engines

If you haven’t used a log analyzer before, you may be wondering where to get the logs from. 

1. If you have a large website and development team, then it is 100% that the logs are already being collected. Usually, various programs and services are used for visualization, for example, Kibana, Datadog, Greylog, Grafana, Databox, Looker, and so on. Unfortunately, logs in such services may not be stored for a long time, because their number is very large. The reason for this is that there are collected not only logs of search engines but also logs of all users, parsers and other bots.

To get the logs, you can contact the development teams (best to contact DevOps) and get access to one of the services that collect the logs. Next, you can select the desired period and export the data to a text file.

This is what a record of a Googlebot visit looks like in one of the services.

2. Another way is to get Access Log records from hosting. Access Log contains raw logs of all users of your website. Mostly, the data is stored for the last day. Then the logs are archived. Logs are deleted at the end of each month. 

What do raw logs mean? This means that you will receive a large file with dozens and hundreds of lines without separated sections with user agents, messages, IPs, etc. Handling such files manually is very inconvenient. 

To access the raw logs, you need to connect to your account with FTP or SSH.

3. You can also contact DevOps to integrate logs directly from your web server.

Logs contain standard data: 

• user agent, 
• status code, 
• IP address, 
• access date, 
• URL/path/query, 
• referrer, 
• HTTP version, 
• and HTTP request type (GET or POST).

This data is usually transmitted by the user agent in the request header to your web server, which is why the logs are also called “server logs”. And they should not be confused with logging changes on your website. For example, logging can contain information about when and which users edited the content of the page.

How to integrate logs into JetOctopus

There are three ways, which are described in detail in the article “3 ways of log file integration with JetOctopus”. Therefore, we will briefly talk about it here.

1. Integration directly from your NGINX or Cloudflare web server. Using this method, you will receive data in real-time. This is a very cool feature that allows you to monitor the behaviour of bots immediately during site migrations or updates.

2. Automatic export of log files daily. You need to create a location and choose a way to store the log files with our technical team (FTP, for example), and JetOctopus will fetch these files daily. This method allows you to receive data with a delay of 1 hour to 1 day. You won’t have to do anything manually.

3. Manual export of files directly in the JetOctopus interface. Processing such files may take a little longer than integrating directly from your web server. But you can independently choose the period and data for analysis according to your limits.

What types of log files do we support?

Actually, JetOctopus supports all types of log files. If there is an error with the file type during the integration, our technical team will manually integrate your file within 24 hours and the logs will be displayed in JetOctopus reports.

Do I need to separate logs of search engines from users yourself?

No, you don’t need to do this. We use only search engine logs, user logs in complete security: we do not use any personal data of users. However, you can separate user logs to reduce the file size when downloading manually. This will speed up the processing of log files. But please note that if you are unsure about which logs are 100% from users and which are not, we recommend not separating the loglines. After all, JetOctopus also checks whether the Googlebot is real or it was a fake bot emulating Google. We do reverse DNS Lookup. Fake bot data is very important for analyzing your website.

More information: How to analyze what types of search robots visit your website.

Possible problems when integrating logs and how to solve it

1. An error occurred with the file type during manual download.

Our technical team will solve your question within 24 hours, no need to worry!

2. Logs stopped being updated in logs reports.

Usually, the reason lies in changes in your web server settings or access settings. You need to contact DevOps or the site administrator to verify the connection with JetOctopus. If you use Cloudflare or AWS, you can check the connection with JetOctopus in the interface.

If everything is fine on your side, contact our online chat: we will help you!

3. Outdated logs are displayed in the data tables.

If you downloaded the log files manually, we recommend re-uploading them. Pay attention to the type of download: Append logs data, Skip updates for uploaded days, Update log data for existing days. If you need to update data for days that already have logs uploaded, select the last type of managing logs. If you want to add all log lines from the file, select the first one. If it is necessary to add data only for dates without logs, choose “Skip updates for uploaded days”.

4. The number of logs in JetOctopus differs from the number of downloaded loglines. It is likely that there are many records of fake bot visits among the uploaded data. Check the type of user agent for which you filtered the logs in JetOctopus.

Our technical team will be happy to help you solve all log issues.