We recommend auditing pages ignored by GoogleBot regularly. It helps to find some problems, for example, to check if crawlers can find new pages.
If search engines do not visit a certain page for several months, the page will likely be dropped from the index.
If the search robot visits the pages very rarely, then the bot does not see content updates. Accordingly, the SERP may contain outdated snippets or outdated information.
This analysis is available to users who have integrated search engine logs.
Go to the “Logs” menu – the “Impact” report.
Next, go to the “Pages visited by bot” chart. Here you will see the ratio of pages visited and ignored by the search engine. Perfect when you have all the pages “Visited and in the structure”.
Click on the “NOT Visited, In structure” column to go to the data table with a list of pages ignored by GoogleBot.
On this page, you can configure the dataset. Choose the period for which you want to see the pages ignored by the bot.
You can also choose a search robot. For example, you can see pages ignored by all search engines, AdWords bot, GoogleBot Images or GoogleBot Mobile, etc.
Select the desired domain or analyze all domains. You can also add any other filters or join datasets.
In the finished table, you will see a list of all URLs that the search robot did not visit during the period you selected. Note that this list contains only those URLs that were found in the crawl results.
You can export the dataset in a convenient format, save the selected URLs as a separate segment, create a KPI or save them as a problem.
Possible reasons why crawlers are not visiting certain pages:
Once you find the reason why the search robot is ignoring your pages, it is recommended to fix the problem so that these pages are indexed in the future.