June 23, 2022
by Sofia

How to find pages ignored by GoogleBot

Before indexing a page, search engines must make at least one visit to that page. For JavaScript sites, the number of visits before indexing will be at least 2 times. Accordingly, you need to monitor the pages ignored by GoogleBot to understand which of your pages will not be indexed.

Why is it important to monitor pages that are not visited by search engines?

We recommend auditing pages ignored by GoogleBot regularly. It helps to find some problems, for example, to check if crawlers can find new pages.

If search engines do not visit a certain page for several months, the page will likely be dropped from the index.

If the search robot visits the pages very rarely, then the bot does not see content updates. Accordingly, the SERP may contain outdated snippets or outdated information.

Step-by-step instruction for analyzing pages ignored by GoogleBot

This analysis is available to users who have integrated search engine logs.

Go to the “Logs” menu – the “Impact” report.

How to find pages ignored by GoogleBot with JetOctopus - 1

Next, go to the “Pages visited by bot” chart. Here you will see the ratio of pages visited and ignored by the search engine. Perfect when you have all the pages “Visited and in the structure”.

Click on the “NOT Visited, In structure” column to go to the data table with a list of pages ignored by GoogleBot.

How to find pages ignored by GoogleBot with JetOctopus - 2

On this page, you can configure the dataset. Choose the period for which you want to see the pages ignored by the bot.

How to find pages ignored by GoogleBot with JetOctopus - 3

You can also choose a search robot. For example, you can see pages ignored by all search engines, AdWords bot, GoogleBot Images or GoogleBot Mobile, etc.

How to find pages ignored by GoogleBot with JetOctopus - 4

Select the desired domain or analyze all domains. You can also add any other filters or join datasets.

How to find pages ignored by GoogleBot with JetOctopus - 5

In the finished table, you will see a list of all URLs that the search robot did not visit during the period you selected. Note that this list contains only those URLs that were found in the crawl results.

You can export the dataset in a convenient format, save the selected URLs as a separate segment, create a KPI or save them as a problem.

How to find pages ignored by GoogleBot with JetOctopus - 6

Why search engines aren’t visiting pages on your website

Possible reasons why crawlers are not visiting certain pages:

  • there are no entry links for the search robot (the pages are not in the sitemaps and they are not found in the code of the pages inside <a href=>);
  • poor internal linking (search engines consider the page unimportant because of this);
  • overly complicated or unclear site structure;
  • pages are blocked for search robots crawling, for example, by the robots.txt file.

Once you find the reason why the search robot is ignoring your pages, it is recommended to fix the problem so that these pages are indexed in the future.

About Sofia
Technical SEO specialist. Sofia has almost 10 years of experience, of which the last 5 years in JavaScript SEO. She is convinced that SEO is a very technical part of digital marketing. And without logs and in-depth data analysis, you can't do effective SEO.



Get exclusive tech SEO insights
We are tech SEO geeks who believe that SEO is predictable and numeric. Don’t miss our insigths!