The JetOctopus team is thrilled to announce a new product update for our customers. In our work, we strive to ensure that our crawler works just like a real search engine when visiting websites. However, we understand that our customers have different needs and have released an update that includes new crawler location logic.
The updated version of JetOctopus allows users to choose the location from which their website will be crawled from the first request. Also, with this new update all your sitemaps also will be scanned from the selected location. This feature is especially important for websites that use geo IP-based redirects. With the new update, all requests will go from the exact location selected by the user, regardless of the crawl mode or user agent chosen.
The update allows users to check how geo IP-based redirects work for users or even how they work for Googlebot if it scans sitemaps and so on. This update is crucial for some websites because some indicators in the crawl results of a website may depend on the location of the web server. With the new update, users can be sure that all requests are coming from the exact location they chose.
To help users configure the crawl to receive the most interesting and useful data, we have provided some useful links.
How to configure a crawl of your website
How to use the list mode: 5 tips
By the way, we will be happy to receive feedback from our users on how we can improve JetOctopus even further. We are always open to suggestions and ideas that can make our SEO crawler more powerful and useful to our customers.