While checking the crawl results, you may notice that some pages are missing. Therefore, it is very important to understand why JetOctopus did not find some pages because search engines are just as difficult to find these pages when crawling your site.
To understand why JetOctopus did not find all the pages, you need to understand how our crawler works. Scanning of your website starts from the home page (or the one you entered in the URL field when setting up a crawl).
Our crawler looks for URLs with <a href= > tag in the code of the main page. GoogleBot similarly scans your website.
Then our crawler goes to the links he found on the first page and looks for <a href= > links in the code of these pages. And so on, until the crawler reaches the page limit specified in the “Page Limit” field.
Or until our crawler will find all the <a href= > links on the website.
Now, you know how JetOctopus scans your website. So, you can understand why there are missing pages in the crawl results:
It can be both reaching the page limit and settings of the crawl. Check if you have activated the “Respect robots rules” checkbox and if you disallow crawling pages that are closed for indexing. If so, our crawler will follow all the scanning directives for search engines, including the “nofollow” and a robots.txt file.
Analysis of missing pages is a great option to improve internal linking and identify weaknesses of the website structure. If the pages are not included in the structure of the website, users will not be able to access them. Useful content will simply be lost. As a result, you may lose traffic.
For search engines, internal linking is also important, as it is the way to find and index a page. Search engines use the number of internal links to the page as a ranking factor.
Where to find the missing URLs in the crawl?
You can analyze the results of the crawl to see which pages are missing. There are several ways to find them.
We want to finalize the main reasons why you can’t find all URLs in the crawl results: