How to audit 304 pages visited by GoogleBot/other bots

When the search robot user-agent requests the web page using the header “If-Modified-Since” or “If-None-Match” and the page has not changed since the requested time (usually this is the last visit of the GoogleBot), the server returns a 304 response code. This signals to the search engine that the web page has not changed since the requested date. Therefore, the crawler can use the previous cached version.

This is a great option that allows search engines to save resources. However, there may be a problem if the page returns a 304, but it was updated. Then the search robot will have an outdated version and this version will be shown in the search results. In such a situation, your SEO will be ineffective, because the robot will not see any changes.

To avoid such problems, we recommend diving deep into the analysis of 304 response codes in search engines logs. 

Step 1. Analyze the total number of 304 status codes.

Go to the “Logs” menu, select the “Bot Dynamics” report. Next, set the desired period, select the type of search engine and domain. You can also select a segment of pages to analyze.

In the “Status Codes” chart, you will see the ratio of all status codes on your website, including 304. Pay attention to the ratio of 200 and 304 response codes.

If 304s predominate, it’s likely that you’re having problems with search results due to non-updating pages (or the server returning an incorrect status code).

Step 2: Dive deep into analyzing 304 response codes.

For advanced analysis, go to the data tableі – “Pages”. Next, select the “Pages with 304” section. Here we show all the pages that received at least one 304.

You can set filters, choose a period, type of crawler, etc.

Pay attention to what types of pages returned a 304 response code. Among these pages may be static web pages, for example, “About us” page or “Terms of use”. However, you should be alarmed if you see category pages or product pages with the 304s. This is most important for large and e-commerce websites that have a dynamic assortment, many reviews, promotional prices, a lot of articles etc.

It may be that 304s are returned for only one type of search bot. If so, you need to check the server settings for this user agent.

Also, check if the 304s are returned by different pages or by the same ones all the time. Perhaps the problem is in particular pages or in a certain type of pages, and the server does not update the last modified field.

You can export data in a convenient format. JetOctopus supports export to CSV, Excel and Google Sheets.