Why are URLs being displayed as non-indexable in crawl results?

Indexability is one of the most important characteristics to pay attention to when analyzing crawling results. We’ve created a separate segment to make work with indexable pages easier. You can use this segment in all JetOctopus reports.

Why the page is not displayed in the segment “Indexable”

An indexable page has 200 response code and can be crawled and indexed by a search engine. All pages that meet these requirements will be displayed in the segment “Indexable”.

The page will not be displayed in the segment “Indexable” if:

• it does not respond with 200 status code (3xx, 5xx, 4xx URLs – all these are non-indexable pages);
• page is redirected;
• the page is blocked by the robots.txt file;
• the page contains the “canonical” to another page;
• the page contains a “noindex” meta tag on HTML pages;
• the page contains “none” or “noindex” X-Robots-Tag in an HTTP header;
• if indexing is not prohibited and the page does not contain any other indexation rules.

In all the cases listed above, search engines will not be able to either scan the page, or read information about the indexability status, or there are noindex directives in HTML or HTTP header. To find out why the page is non-indexable, go to the “Indexation” report. There is a chart with the number of non-indexable pages for various reasons.

We also recommend working with the data tables to see the reason the page is non-indexable.

What does mean “Is indexable” in Data Tables

The “Is indexable” filter in Data Tables is the same as the “Indexable” segment. This filter does not include pages with non-200 status codes, with canonical tag, with noindex meta tag or noindex/none specification in the X-Robots-Tag, and pages blocked by the robots.txt file.

To find reasons for non-indexable status use additional filters in the “Indexation” block:

• Is Robots.txt indexable – pages that are not blocked from scanning by search engines in the file robots.txt;
• Is meta tag indexable – pages that contain the meta tag “index” in HTML;
• X-Robots Header Index – pages without the X-Robots-Tag prohibiting indexing.

To find out if the reason for non-indexability is in the canonical, you can use the filters in the “Canonicals” block.

• Is Canonical Page – select “No” to see if pages contain canonicals to another pages.

To analyze pages that are non-indexable due to a non-200 status, select the needed value in the status code filter.

You can also set the “Non-Indexable Reason” column in the data tables. JetOctopus shows the reason for non-indexability in this column.

Of course, you can export your data to CSV, Excel or Google Spreadsheets.