How to check 5xx URLs in logs

5xx response codes indicate a broken web server. These status codes are usually temporary. However, a large number of 5xx and regular repeats can affect the crawling of your website by search engines.

Why do you need to monitor 5xx response codes?

First, Google search engines will slow down the crawling of your website if the web server returns a large number of 5xx. This also means that search engines will be slower to find and index new pages and update current ones.

Secondly, if the indexed page will always return 5xx status codes, over time the search engines will drop it out from the SERP.

Thirdly, you should especially monitor the sitemaps and the robots.txt file. If the robots.txt file constantly returns a 5xx response code, Google may begin to consider that your website has no scanning restrictions. Accordingly, search engines may index duplicate pages or pages that should not be displayed in search results. In the case of a 5xx response code returned by sitemaps, search engines will not be able to detect new pages in time.

How to find 5xx pages in search engine logs in two clicks?

Using JetOctopus, you can quickly find URLs with the 5xx response codes that the web server returned to search engines.

Step 1. Go to the “Logs” section, select the “Bot Impact” dashboard. Next, choose the type of search engine, period and domain.

On the “Status codes” diagram, you will see the ratio of 5xx response codes and all others. The more 5xx codes, the more likely there is a problem with your web server. This can reduce how often your website is crawled by search engines.

Pay attention to the “Dynamics of status codes” diagram. You can see how the number of 5xx response codes changed during the period you selected.

Step 2. For a detailed analysis of pages that returned a 5xx response code, go to the “Raw Logs” data table. Then select the “5xx statuses” section.

Click the “+Add filter” button to select the desired URLs, bots, etc. You can also select the desired type of 5xx response code for analysis here.

Step 3. Select the desired segments or join datasets. This will allow you to identify the 5xx URLs that have the highest risk of being deindexed.

More information: How to join datasets.

Step 4. Bulk export  URLs returned 5xx status codes to search engines. Click the “Export” button and select the desired format. In addition to CSV, Google Sheets, Excel, you can export URLs to DataStudio or sitemaps.

More information: How to generate a sitemap with JetOctopus and how to submit it to Google.

Step 5. Recheck for 5xx errors. Select “Crawl the list” in “Tools” to run a crawl of URLs with 5xx status codes and check that they are not still repeated.

You may be interested in: What if you run a crawl and your website returns the 503 response code?