July 6, 2022
by Sofia

Step-by-step instructions: analysis of 404 URLs in search engine logs

If search engines receive a lot of 404 response codes, they can reduce the scanning frequency of your website. Also, 404 in the logs can indicate the following problems:

  • broken URLs in sitemaps;
  • 404 pages in the code of your site (broken internal linking);
  • old URLs that no longer exist on your website but they are still in the index of search engines and do not have the correct redirect.

Also, 404 response codes in the logs can highlight problems with the site or web pages that broke due to releases, updates, etc.

How to find 404 status codes in logs with Jet Octopus

Using JetOctopus, you can easily find pages that have returned 404 status codes to search engines. To do this, you need to have automatic integration of logs or import log files manually through the log manager.

Step-by-step instructions analysis of 404 URLs in search engine logs - JetOcotopus - 1

After log integration, we will start immediately collecting information about all visits of search, advertising and marketing robots, including GoogleBot, Google Image, Baidu, Bing, AdsBot etc.

To see general information about 404 status codes in bot logs, go to “Logs” – “Bot Dynamics” report.

Step-by-step instructions analysis of 404 URLs in search engine logs - JetOcotopus - 2

Next, select the desired bot or select “All”, select period and domain.

Step-by-step instructions analysis of 404 URLs in search engine logs - JetOcotopus - 3

In the “Status codes” chart, you will see the ratio of all response status codes for the selected robot during the selected period. Click on “404” to go to the data table with a list of all 404 pages.

Step-by-step instructions analysis of 404 URLs in search engine logs - JetOcotopus - 4

We recommend paying attention to the “Dynamics of status codes” chart in the “Bot Dynamics” report. Here you can easily find out in which period search engines received the most 404 status codes. This may indicate a massive breakdown of web pages.

Step-by-step instructions analysis of 404 URLs in search engine logs - JetOcotopus - 5

You can also filter 404 status codes in search engine logs using data tables. Go to the “Logs” menu and select “Raw Logs” in data tables. Next, click the “+Add filter” button and select “Status code” – “=Equal” – “404” – “Apply” button.

Step-by-step instructions analysis of 404 URLs in search engine logs - JetOcotopus - 6

Here you can set up all the other filters you need, including choosing URLs, period, bots, and more.

How to check the list of URLs for 404 in search engine logs

With JetOctopus, you can check whether specific URLs return a 404 status code for crawler visits.

To do this, go to the “Logs” menu – data tables – “Raw logs”. Next, add filter “Status code” – “=Equal” – “404” and filter “URL” – “=Equal” and in the text field add the exact list of URLs with HTTPS protocol and domain. Each URL in a new line without extra characters.

Step-by-step instructions analysis of 404 URLs in search engine logs - JetOcotopus - 7

To bulk export 404 status codes in the logs of GoogleBot and other robots, click the “Export button” in the right corner above the data table and select the desired format.

Step-by-step instructions analysis of 404 URLs in search engine logs - JetOcotopus - 7 - bulk export

About Sofia
Technical SEO specialist. Sofia has almost 10 years of experience, of which the last 5 years in JavaScript SEO. She is convinced that SEO is a very technical part of digital marketing. And without logs and in-depth data analysis, you can't do effective SEO.

Search

Categories

Get exclusive tech SEO insights
We are tech SEO geeks who believe that SEO is predictable and numeric. Don’t miss our insigths!