How to integrate logs manually

Analysis of the activity of search bots is an important part of SEO, which should be paid a lot of attention. If the search engines do not visit the site regularly, do not update the pages and do not discover new ones, the website will not get the full potential of organic traffic. Therefore, we recommend analyzing the logs of search robots to get valuable information about their activity.

Logs are text files (rarely other files) that contain useful information about users, in particular about bots. In the logs, you will find data on all bot visits, user agent (so you will know what types of bots are scanning your website), date (timestamp), IP (allows you to detect fake bots), referrer, etc. But the number of logs is quite large. If your website has thousands of visits every day, manually segmenting the bot logs and highlighting problematic visits will be extremely difficult. Therefore, you can use various tools – log analyzers. As a comprehensive tool for technical SEO, JetOctopus also allows you to import and analyze search engine logs.

If you are new to log analysis and have never collected logs before, we recommend that you start by manually uploading logs to JetOctopus.

To get access to the logs, you need to contact DevOps or the hosting administrator. Usually, most hosts allow website owners to view and download logs.

By default, logs are stored for 30 days, but depending on the hosting settings, there may be other settings.

Of course, it is much more convenient to set up automatic import of logs, but if the resources of developers or DevOps are limited, you can integrate logs manually. Don’t forget to do this regularly.

To manually import logs into JetOctopus and be able to view logs in a convenient format, with a lot of dashboards save the files in the default Apache and Nginx logs format. Compressed (zip) folders are also supported.

They look like this:

66.249.64.90 – – [28/Oct/2021:06:52:10 +0200] “GET /robots.txt HTTP/1.1” 200 14 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”.

If you have logs in another format, we will help you import them into JetOctopus. Write to our technical support.

To import logs manually, go to the logs section, then select “Upload files”.

Next, drop the files into the window and select “Import mode”.

Append logs data – all data for all dates will be loaded.

Skip updates for uploaded days – days for which there are already uploaded logs will be skipped.

Update log data for existing days – if there is a match, the data for existing days will be updated.

Next, click “Import files”.

If everything is successful and the files are of the appropriate format, after a few seconds your logs will be available in the appropriate section.

There are also other ways to upload logs and automate the integration to get real-time bot activity data. 

More information: Log file integration: everything you need to know.

You can find all imported log files at the bottom of the “Import files” menu in the “Manage Logs” section.

In case of any problems or if the logs are not in the default format, please contact our technical support. We will be happy to help you and import logs of another format manually within 24 hours.

Using log analysis, you can find a lot of technical issues related to the activity of search robots:

How to find blinking status codes in search robots logs

Step-by-step instructions: analysis of 404 URLs in search engine logs

How to audit non 200 response codes in search engine logs