If you have crawled the same website twice or more, you may notice that the crawl results are different. In this article, we will explain why this happens and what to keep in focus when analyzing differences in crawls. Next, we will describe in points the possible reasons why the results of crawls differ. Different […]
Understanding how the JetOctopus crawler works will help you better understand the results of your crawling. The main principle of our crawler is very similar to search engines. However, you can make many additional settings with JetOctopus. We want to emphasize that if there are errors in the crawl results or there are not all […]
We recommend using advanced settings to crawl JavaScript websites. Thanks to the specific settings you will be able to analyze your JS version in more detail. We’ve written an article about crawl configuration before, go to this article to read about standard crawl configuration for all types of websites. And in the article “How to […]
If you crawl your website with a regular JetOctopus user agent, it will not be blocked by Cloudflare. However, if you want to check how your site is seen by Google bot using crawling, Cloudflare will block unconfirmed Googlebot. We remind you that you can crawl your website with the Googlebot Mobile or Googlebot Desktop […]
For large multinational/multilingual websites, it can be difficult to track traffic dynamics in different countries. Monitoring the performance of all countries in total is an ineffective method for SEO. Because the fall in 1% of traffic in the most traffic-generating country will be very noticeable, and in the general chart this fall will be almost […]
If you find errors in crawl results or logs, you can submit the corrected/updated pages for scanning to search engines. This can be done with a custom sitemap. JetOctopus allows you to generate a custom sitemap in two clicks. In which cases do you need to generate a sitemap? For example, due to a technical […]
Happens when you run a crawl and the website returns a 503 response code. “503 Service Unavailable” response code means that your web server cannot process the request at this time. What are the reasons for the 503 status code? There is good news: the 503 response code is temporary, and as soon as the […]
Before you start a new crawl we recommend preparing basic information about your website. It allows you to set up the optimal configuration of the crawl. Of course, the basic settings of JetOctopus crawl are suitable for most websites, but it is worth delving into the settings to get the most useful information about your […]
We want to have clean and accessible code on our website. Also, we want search engines to scan only pages needed in search results because we worry about our crawling budget. But sometimes there are situations when URLs in the code are blocked by the robots.txt file. It means that search engines cannot access the […]
Get exclusive tech SEO insights
