Blog
Get exclusive tech SEO insights
Recent Articles
Why does the crawl of your website fail?
In this article, we’ll talk about why a crawl of your website can fail. There are several reasons why a crawl of your website may not finish. Some of the reasons may be due to the settings of your web server and the fact that your server perceives our crawler as a potential security threat. […]
How to use segments
Using JetOctopus, you can customize as many segments as you need. We do not limit the number of segments. You can use URL segments when analyzing the results of crawling, logs and Google Search Console. Why you should segment your website Slice your site into as many segments as you need and watch your errors […]
Why the results of crawls differ
If you have crawled the same website twice or more, you may notice that the crawl results are different. In this article, we will explain why this happens and what to keep in focus when analyzing differences in crawls. Next, we will describe in points the possible reasons why the results of crawls differ. Different […]
How does the JetOctopus crawler work?
Understanding how the JetOctopus crawler works will help you better understand the results of your crawling. The main principle of our crawler is very similar to search engines. However, you can make many additional settings with JetOctopus. We want to emphasize that if there are errors in the crawl results or there are not all […]
Why do JavaScript and HTML versions differ in crawl results?
When you crawl your JavaScript site, be prepared to see that JavaScript and HTML versions differ. The difference between these versions may affect the indexing results. However, if the most important SEO elements are the same, the difference between JS and HTML is normal. Below we will explain why the results of crawling JS and […]
How to check JS website with JetOctopus
JavaScript is fast, dynamic technology. It saves your server resources which is a great benefit for you. However, in order for the JavaScript site to generate a lot of organic traffic, you need to be careful about the settings and auditing of SEO. In this article, we will explain how to audit JS websites with […]
How to configure crawl for JavaScript websites
We recommend using advanced settings to crawl JavaScript websites. Thanks to the specific settings you will be able to analyze your JS version in more detail. We’ve written an article about crawl configuration before, go to this article to read about standard crawl configuration for all types of websites. And in the article “How to […]
How to check canonicals with JetOctopus
Rel=”canonical” is a powerful method that allows you to manage duplicate content. When you use the canonical, you tell search engines which page to show in search results. However, there are situations when due to the incorrect use of the canonicals, unwanted pages are displayed in the search results or, conversely, the most important ones […]
How to compare crawls
JetOctopus represents a great opportunity to compare crawls. You can compare any crawls, not just the last two. This option is free and available to all users who use the JetOctopus crawling tool. LINK TO THE HOME Read about the benefits of our crawl comparison tool in the article “Product Update: Compare Crawls”. How to […]
Why are there missing pages in the crawl results?
While checking the crawl results, you may notice that some pages are missing. Therefore, it is very important to understand why JetOctopus did not find some pages because search engines are just as difficult to find these pages when crawling your site. Why JetOctopus didn’t find all the pages To understand why JetOctopus did not […]
How to schedule a crawl
You can run the crawl yourself or set up a regular crawl. Also, you can set the start of a one-time crawl at the time you need, for example, at night. With a regular crawl, you can track changes to your website after releases, the number of new pages created over a while, and more. […]
How to find links to broken pages
Using JetOctopus, you can find links to broken pages easily. Start a new crawl or check for broken links in the results of the previous crawl. However, keep in mind that the number of broken links may have changed since the last crawl. Links to broken pages are the links that lead to non-200 pages. […]
How to find orphan pages with JetOctopus
Orphan pages are URLs that cannot be found on your website during a crawl. They are not included in the structure of internal linking. As a consequence, users will not be able to find these pages. But search engines can follow the orphan pages from external links, from search index (for example, if the orphan […]
The easy way to check for duplicate content: duplicated titles and meta descriptions
Unique content is an essential factor in the ranking of your website in search results. If there is a lot of duplicate content, search engines may incorrectly identify the most relevant page or exclude duplicate content URLs from search results. However, the most important thing is that duplications are wasting your crawling budget. A lot […]
How to check XML Sitemaps with JetOctopus
Sitemaps are one of the most important sources of new URLs for search engines. They scan URLs from sitemaps regularly. And if the URLs meet the requirements of search engines, they will be indexed. However, if sitemaps contain broken links, 4xx, 3xx pages, etc., this will negatively affect your crawling budget. Therefore, it is wise […]
How to crawl websites using Cloudflare with the Googlebot user-agent
If you crawl your website with a regular JetOctopus user agent, it will not be blocked by Cloudflare. However, if you want to check how your site is seen by Google bot using crawling, Cloudflare will block unconfirmed Googlebot. We remind you that you can crawl your website with the Googlebot Mobile or Googlebot Desktop […]
How to crawl a staging website with JetOctopus
JetOctopus can crawl staging websites and websites closed from scanning by search engines. You can also run a crawl of a website that needs authentication (password/login). To crawl staging websites, just set up special crawl settings. When you actually need to crawl a staging website How to crawl a staging website blocked by robots.txt If […]
Why are URLs being displayed as non-indexable in crawl results?
Indexability is one of the most important characteristics to pay attention to when analyzing crawling results. We’ve created a separate segment to make work with indexable pages easier. You can use this segment in all JetOctopus reports. Why the page is not displayed in the segment “Indexable” An indexable page has 200 response code and […]
How to analyze Core Web Vitals with JetOctopus
Core Web Vitals is a set of specific features that Google considers important for users. The better your Core Web Vitals, the better your site is at the top in the SERP. Core Web Vitals consist of three main metrics: the largest contentful paint (LCP), first input delay (FID), and cumulative layout shift (CLS). Also, […]
Product Update. GSC by Countries
For large multinational/multilingual websites, it can be difficult to track traffic dynamics in different countries. Monitoring the performance of all countries in total is an ineffective method for SEO. Because the fall in 1% of traffic in the most traffic-generating country will be very noticeable, and in the general chart this fall will be almost […]
How to check hreflang with JetOctopus
Hreflang attributes show search engines that a website has localized versions of pages. For example, you may have English and Ukrainian versions, or separate English language versions for the United Kingdom and the United States. You can use the hreflang attributes in regular HTML, HTTP headers or sitemaps. Most hreflang problems are with the incorrect […]