July 19, 2022
by Sofia

How to check 5xx URLs in logs

5xx response codes indicate a broken web server. These status codes are usually temporary. However, a large number of 5xx and regular repeats can affect the crawling of your website by search engines.

Why do you need to monitor 5xx response codes?

First, Google search engines will slow down the crawling of your website if the web server returns a large number of 5xx. This also means that search engines will be slower to find and index new pages and update current ones.

Secondly, if the indexed page will always return 5xx status codes, over time the search engines will drop it out from the SERP.

Thirdly, you should especially monitor the sitemaps and the robots.txt file. If the robots.txt file constantly returns a 5xx response code, Google may begin to consider that your website has no scanning restrictions. Accordingly, search engines may index duplicate pages or pages that should not be displayed in search results. In the case of a 5xx response code returned by sitemaps, search engines will not be able to detect new pages in time.

How to find 5xx pages in search engine logs in two clicks?

Using JetOctopus, you can quickly find URLs with the 5xx response codes that the web server returned to search engines.


Step 1. Go to the “Logs” section, select the “Bot Impact” dashboard. Next, choose the type of search engine, period and domain.

How to check 5xx URLs in logs with JetOctopus - 1

On the “Status codes” diagram, you will see the ratio of 5xx response codes and all others. The more 5xx codes, the more likely there is a problem with your web server. This can reduce how often your website is crawled by search engines.

How to check 5xx URLs in logs with JetOctopus - 2

Pay attention to the “Dynamics of status codes” diagram. You can see how the number of 5xx response codes changed during the period you selected.

How to check 5xx URLs in logs with JetOctopus - 3

Step 2. For a detailed analysis of pages that returned a 5xx response code, go to the “Raw Logs” data table. Then select the “5xx statuses” section.

How to check 5xx URLs in logs with JetOctopus - 4

Click the “+Add filter” button to select the desired URLs, bots, etc. You can also select the desired type of 5xx response code for analysis here.

How to check 5xx URLs in logs with JetOctopus - 5

Step 3. Select the desired segments or join datasets. This will allow you to identify the 5xx URLs that have the highest risk of being deindexed.

More information: How to join datasets.

How to check 5xx URLs in logs with JetOctopus - 6

Step 4. Bulk export  URLs returned 5xx status codes to search engines. Click the “Export” button and select the desired format. In addition to CSV, Google Sheets, Excel, you can export URLs to DataStudio or sitemaps.

How to check 5xx URLs in logs with JetOctopus - 7

More information: How to generate a sitemap with JetOctopus and how to submit it to Google.

Step 5. Recheck for 5xx errors. Select “Crawl the list” in “Tools” to run a crawl of URLs with 5xx status codes and check that they are not still repeated.

How to check 5xx URLs in logs with JetOctopus - 8

You may be interested in: What if you run a crawl and your website returns the 503 response code?

About Sofia
Technical SEO specialist. Sofia has almost 10 years of experience, of which the last 5 years in JavaScript SEO. She is convinced that SEO is a very technical part of digital marketing. And without logs and in-depth data analysis, you can't do effective SEO.

Categories

Find technical SEO errors at your website at free trial
Get exclusive tech SEO insights
We are tech SEO geeks who believe that SEO is predictable and numeric. Don’t miss our insigths!
Start your free trial
free 7-days trial