Step-by-step instructions: analysis of 404 URLs in search engine logs

If search engines receive a lot of 404 response codes, they can reduce the scanning frequency of your website. Also, 404 in the logs can indicate the following problems:

• broken URLs in sitemaps;
• 404 pages in the code of your site (broken internal linking);
• old URLs that no longer exist on your website but they are still in the index of search engines and do not have the correct redirect.

Also, 404 response codes in the logs can highlight problems with the site or web pages that broke due to releases, updates, etc.

How to find 404 status codes in logs with Jet Octopus

Using JetOctopus, you can easily find pages that have returned 404 status codes to search engines. To do this, you need to have automatic integration of logs or import log files manually through the log manager.

After log integration, we will start immediately collecting information about all visits of search, advertising and marketing robots, including GoogleBot, Google Image, Baidu, Bing, AdsBot etc.

To see general information about 404 status codes in bot logs, go to “Logs” – “Bot Dynamics” report.

Next, select the desired bot or select “All”, select period and domain.

In the “Status codes” chart, you will see the ratio of all response status codes for the selected robot during the selected period. Click on “404” to go to the data table with a list of all 404 pages.

We recommend paying attention to the “Dynamics of status codes” chart in the “Bot Dynamics” report. Here you can easily find out in which period search engines received the most 404 status codes. This may indicate a massive breakdown of web pages.

You can also filter 404 status codes in search engine logs using data tables. Go to the “Logs” menu and select “Raw Logs” in data tables. Next, click the “+Add filter” button and select “Status code” – “=Equal” – “404” – “Apply” button.

Here you can set up all the other filters you need, including choosing URLs, period, bots, and more.

How to check the list of URLs for 404 in search engine logs

With JetOctopus, you can check whether specific URLs return a 404 status code for crawler visits.

To do this, go to the “Logs” menu – data tables – “Raw logs”. Next, add filter “Status code” – “=Equal” – “404” and filter “URL” – “=Equal” and in the text field add the exact list of URLs with HTTPS protocol and domain. Each URL in a new line without extra characters.

To bulk export 404 status codes in the logs of GoogleBot and other robots, click the “Export button” in the right corner above the data table and select the desired format.