How to use JetOctopus to create effective sitemaps

Sitemaps are an important part of website optimization. Effective sitemaps include all indexable pages and help search bots discover new and updated URLs on your website. However, sitemaps are not always really efficient. In this article, we will tell you how to create really useful sitemaps taking into account the structure of the website because it really positively affects the crawl budget and the ranking of your website in search results.

What to focus on when creating sitemaps

The structure of sitemaps depends on the structure of the website, the number of pages on the website and the frequency of creation of new URLs. These are the main three things that you should pay attention to in the first place. 

By the way, the recommendations in the article can be used both for creating xml sitemaps and for creating HTML sitemaps. Because both should be structured, relevant and understandable for users and search engines.

If your website has a branched structure and many URLs, then the sitemaps should also correspond to the structure of the website. And even more: sitemaps should systematize URLs to facilitate site crawling by search engines.

For example, you can create sitemaps by type of pages (one sitemap contains all products, the second contains categories, the third contains blog articles and informational pages, etc.). Or you can create sitemaps by directories in the URL or by main sections of the site – there are quite a few options. However, pay attention to the fact that depending on the URLs that are in the sitemap, search engines may scan the sitemaps more or less often. So, you should make sure that there are no important URLs that are frequently updated in the sitemap with static pages.

Also, consider how quickly your website is growing. If there were only three categories and 10,000 products on your website at the time the sitemaps were created, but you have plans to expand this category to 18 more categories and 500,000 products by the end of the year, take this into account when creating sitemap structure.

What else is important to consider:

• availability of subdomains: each subdomain must have its own sitemaps;
• using of hreflang in sitemaps;
• do you need to create sitemaps with images and videos.

All these technical solutions must meet the goals of the site. 

Where to start planning the structure of the sitemap

Once you’ve decided whether your sitemaps will include images, URLs with hreflangs, etc., you need to think about the structure of your sitemaps. In order not to miss any necessary pages, we recommend starting with a full crawl of your website. By the way, it is also worth scanning non-indexable pages, if you are not sure that there are no useful ones among them.

A full crawl of your website will help you understand:

• how many pages do you have;
• how many pages of each type;
• whether all pages return a 200 status code;
• what is the distance from the index of your website;
• what the structure is like in general and how effective it is.

Pay attention to the “Structure Efficiency” dataset. Here we show which directories are on your website, how many pages are in each directory, and how many of those pages are indexable.

If you have two large directories on your website, you can focus on that and create three sitemap index files that will contain all the URLs for those directories.

For example, your website has a default version in English https://example.com; the French version of the website https://example.com/fr and the German version https://example.com/de. You can then create three sitemap index files that contain smaller sitemaps for each directory.

After analyzing the structure of the website, decide which of these pages are needed in the SERP. By the way, if you don’t have many pages on your website, you can use JetOctopus to create a sitemap. Filter the desired URLs in the “Pages” database, then click the “Export” button and select “To sitemap”.

More information: How to generate a sitemap with JetOctopus and how to submit it to Google.

What types of pages should be in sitemaps?

In general, sitemaps should include all pages that can bring organic traffic. These pages must return a 200 status code for users and search robots (so, check the Geo IP redirects), be indexable and not blocked by the robots.txt file.

Don’t forget to add less obvious pages to your sitemaps. For example, for an e-commerce website, it can be information pages with contacts, a list of stores, pages with vacancies, etc.

If pages are created frequently on your website, you can create a separate sitemap with new pages. This tip allows search engines to discover new pages on your website even faster.

By the way, plugins do not always work ok. And we often come across situations when plugins generate sitemaps with broken or non-indexable URLs, so you can generate sitemaps yourself and submit it to Google to avoid such situations. If you decide to use a plugin for creating sitemaps, pay special attention to checking URLs in sitemaps.

After all, checking the relevance and correctness of the sitemap is important regardless of the way you create this sitemap: independently, using a plugin, or sitemaps will be updated automatically based on updates in your database. Sitemaps are not a part of a website that can be created and submitted once, and then forgotten about. Sitemaps require regular analysis and checks.

You can do most sitemap checks with JetOctopus.

More information: How To Audit XML Sitemaps.

Life Hacks for creating sitemaps

We’ve been experimenting with sitemaps to understand how sitemap structure affects page crawling and indexing. We realized that sitemaps work differently for each website, depending on the overall crawlability of the site, internal linking, distance from the index, etc. You can read about an interesting case studies: Sitemap as a True Damager (about the negative experience of a large website with incorrect generation of site maps) and Case Study: How DOM.RIA Doubled Their Googlebot Visits Using JetOctopus (about how small sitemaps improved Googlebot’s crawling of pages).

But the following lifehacks have worked very well for some large websites:

• small sitemaps for different types of URLs depending on the frequency of page updates;
• different types of pages are separated in multiple sitemaps;
• separate sitemaps for each type of content: video, news, etc.;
• it is better to highlight new URLs separately – then the scanning of this sitemap will be more frequent and more effective than other sitemaps;
• if the products (or other pages) often change the status code, and you do not have time to update the sitemaps, it is better to abandon such pages in the sitemaps. For example, if out of stock products return a 404 status code and remain in the sitemap for a long period, which is greater than the average sitemap scan frequency, try not to use such URLs in the sitemap.

And one more tip: if you’ve updated your sitemap, don’t forget to submit it to Google. Thus, Google will be notified in time about changes in the sitemap and will quickly scan pages from this part of the site.