How to generate a sitemap with JetOctopus and how to submit it to Google

If you find errors in crawl results or logs, you can submit the corrected/updated pages for scanning to search engines. This can be done with a custom sitemap. JetOctopus allows you to generate a custom sitemap in two clicks.

In which cases do you need to generate a sitemap? For example, due to a technical error, some pages have become closed from indexing, and search engines have already crawled those pages. And until the pages dropped out of the index, you submit the list with corrected URLs to Google. We recommend using sitemaps for this. Of course, you need to submit a sitemap after you’ve fixed all the bugs.

How to generate a sitemap with JetOctopus

With JetOctopus, you can easily generate a custom sitemap with any URLs.

1. To get started, select the required project and crawl from your list.

2. Go to the appropriate report and configure the filters. Or use Data Tables.

3. Export the URLs list to the sitemap.

4. Upload a sitemap into the domain root folder of your website and submit it to Google.

How to configure sitemaps

When you select the option “Export”- “To Sitemap”, a pop-up window appears.

1. In this window, specify the file name (for example, “sitemap-goods”). After all the settings you can download a file with this name.

2. You can also change the number of URLs in the sitemap. We remind you that the number of URLs in the sitemap cannot be more than 50,000. If the number of URLs is more than 50,000, JetOctopus will generate several sitemaps.

3. “Date lastmod” is the date the page was refreshed. JetOctopus automatically uses the date of generation of the sitemap, but you can use your date. We recommend using either the date when the page was fixed or today’s date.

4. “Change freq” is an optional attribute in sitemaps. This attribute indicates how often the page changes. If the page changes frequently, for example, the main page – specify hourly. If it’s a stable page, choose monthly or yearly. Now Google ignores <changefreq>, but other search engines use it during processing the sitemap. Then export sitemap.

How to submit a sitemap to Google

You can submit a sitemap to Google with Search Console. Just add a new sitemap in a sitemap report. But first, upload a sitemap into the domain root folder of your website.

Also, you can use the ping tool. Send a GET request in your browser or the command line with the URL of the sitemap. Use absolute URL with HTTPS protocol and domain name. 

Ping the sitemap here: https://www.google.com/ping?sitemap=FULL_URL_OF_SITEMAP

You can also add a link to your sitemap in the robots.txt file. The link must be absolute.

Which pages should not be added to a sitemap

1. If the bug has not been fixed.
2. Pages with non 200 response code.
3. Pages blocked by robots.txt. 
4. Pages with noindex in meta-robots or X-Robots tag. 
5. Non-canonical pages.

How to exclude pages from a sitemap

To exclude pages from a sitemap, use the “Page URL” filter – “Not contain”. Then enter the URL path or URL list. Enter each URL in a new line. You can enter a maximum of 10,000 lines.

Or use Page URL” filter – “Not contain by REGEXP” and enter the regex rule to exclude pages from a sitemap.