How to identify missing URLs in your sitemap

Sitemaps help Google and other search engines find new pages on your website and discover pages with updated content. Therefore, it is important to assure that all indexable pages are included in sitemaps. On the other hand, non-indexable and non-200 pages should not be added to sitemaps. In this article, we will tell you how to find indexable pages on your website that are not included in the sitemap.

Step 1

Scan your entire website to find all the pages. Activate the “Process sitemaps” checkbox during crawl settings.

If your sitemaps have a non-standard address, go to “Advanced Settings” and enter a list of sitemap addresses in the “Sitemaps” field.

Step 2

After the crawl is complete, analyze missing URLs in your sitemap. Go to the “Pages” data table.

Click the “+Add filter” button, and select “Found in Sitemap” – “No” and “Is Indexable” – “Yes”. Then click “Apply”. Make sure to filter indexable pages, as only indexable pages should be included in sitemaps to prevent search engines from wasting their crawl budget on non-indexable links.

In the results, you will see a table with a list of all indexable pages that were not found in the sitemap. We recommend adding these links to your sitemap.

Step 3 

Export the data to a convenient format or generate a sitemap with missing URLs using JetOctopus. To generate a sitemap with JetOctopus, export the URLs list to the sitemap by selecting the “Export” option and then choosing “To Sitemap”. 

In the pop-up window that appears, specify the file name, the number of URLs per file, and the date lastmod and change freq attributes (“Change freq” is an optional attribute). Download the generated sitemap file and upload it to the domain root folder of your website.

Submit the sitemap to Google by adding it to the sitemap report in Search Console or by using the ping tool to send a GET request with the URL of the sitemap. You can ping the sitemap here: https://www.google.com/ping?sitemap=FULL_URL_OF_SITEMAP. You can also add a link to your sitemap in the robots.txt file.

By following these steps, you can generate a custom sitemap with missing URLs and submit it to Google to ensure that your corrected and updated pages are crawled and indexed correctly.

More information: How to generate a sitemap with JetOctopus and how to submit it to Google

Why is it important to create full sitemaps? 

• Creating full sitemaps is important because they help search engines like Google find and index all the pages on your website, including those with updated content. 
• Sitemaps provide a roadmap for search engine crawlers to navigate and understand the structure of your website. 
• Including all indexable pages in your sitemap ensures that search engines can efficiently crawl and index your website, which can improve your website’s visibility in search results. 

Additionally, excluding non-indexable and non-200 pages from your sitemap can help prevent search engines from wasting their crawl budget on pages that are not worth indexing. By following the steps outlined in the article, you can generate a custom sitemap with missing pages and submit it to Google to ensure that all your website’s pages are crawled and indexed correctly.

You may be interested in:

How To Audit XML Sitemaps
How to use JetOctopus to create effective sitemaps
Sitemap as a True Damager