How to audit non-200 response codes in search engine logs

If your website is perfectly on-page optimized, but the search engines get a lot of non-200 response codes, this can decrease the crawling budget. Search engines will crawl your website less often, pages will be updated less often in the SERP, and new ones will be shown with a lag. To avoid this, we recommend analyzing search engine logs. Analysis of non-200 status codes is one of the most important points.

Using JetOctopus, you can quickly analyze non-200 pages in logs of any search engine.

Where to find a list of all non-200 status codes

To see the general statistics of non-200 status codes, go to the “Logs” menu – the “Bot Dynamics” report. Next, select the desired period, type of search engine and domain.

Pay attention to the chart “Status codes”: here you can see the ratio of 200 and other status codes for the selected search robot and period. Clicking on a chart segment or status code will take you to the data table.

On the “Dynamics of status codes” chart, you can find information on how the number of non-200 status codes changed during the period you selected. A sharp jump of non-200 pages with an unchanged number of visited pages may indicate large errors on your website.

For an in-depth analysis of non-200 status codes in search engine logs, go to the data tables – “Raw logs” – “Non 200 Status”. Here you can configure the necessary filters: click “+Add filter” to set the required URLs, status code, referers, bot IPs, etc.

To get additional data, configure the columns by clicking the “Setup Columns” button. Next, select the desired ones.

For bulk export of non-200 status codes for search robots, for example, for GoogleBot, click the “Export” button and select the desired format.

What types of non-200 status codes should be paid special attention to

1. When analyzing non-200 response codes in search engine logs, pay attention to the dynamics of these status codes and the ratio of non-200 visits to pages.

2. Also, we recommend an in-depth analysis of 404 status codes in the logs. More information: Step-by-step instructions: analysis of 404 URLs in search engine logs. 

3. Pay attention to 304 status codes: How to audit 304 pages visited by GoogleBot.

4. And, of course, keep the blinking status codes in focus. Pages with blinking status codes are those pages that returned to bot at least two different status codes during the selected period. This is usually a 200 and 404 or 5xx response code. More information: How to find blinking status codes in search robots logs.

5. Analyze the 5xx status code separately. They are usually related to temporary web server problems. However, a large number of 5xx crawlers reduces the frequency of crawling of your website.