The last six months have been interesting and exciting for the JetOctopus team. It was difficult at times, but we managed it. And as a result, many useful updates were released. As we know, our beloved customers are actively using many new features. We are happy to improve JetOctopus and make it more convenient for […]
Use the Include and Exclude feature to prevent JetOctopus from scanning unwanted pages. We recommend that you use this option if you need to scan one or more types of pages. You can use this feature if you need to scan a part of the large website. Read about other ways to scan a large […]
Security is important to all websites. For JetOctopus, the security of your data is also in the first place. Therefore, we want to tell you which user agents JetOctopus uses when scanning your website. DevOps or website administrators will also need this information so they don’t block the JetOctopus crawler. We emphasize that JetOctopus is […]
Custom extraction is needed to find certain elements in the HTML code of the page. Our web scraper will search selected elements while crawling your website. With custom extraction, you can use JetOctopus more efficiently and get non-standard information about your website. With JetOctopus, you can configure up to 50 custom extraction rules during a […]
If you use separate mobile domains, you need a special crawl analysis. In the article “How to crawl separate mobile domains with JetOctopus” we wrote about how to configure crawling for a separate mobile site. In this article, we will tell you what to focus on during the crawl analysis of a separate mobile domain. […]
If your website is global/international, it is important to track performance by countries. Consider an example why a general analysis of effectiveness in the SERP is not enough. Your website has visitors from two countries. Each country has a certain number of clicks and impressions and different conversion rates. Country A has a high conversion […]
In this article, we’ll talk about why a crawl of your website can fail. There are several reasons why a crawl of your website may not finish. Some of the reasons may be due to the settings of your web server and the fact that your server perceives our crawler as a potential security threat. […]
Using JetOctopus, you can customize as many segments as you need. We do not limit the number of segments. You can use URL segments when analyzing the results of crawling, logs and Google Search Console. Why you should segment your website Slice your site into as many segments as you need and watch your errors […]
If you have crawled the same website twice or more, you may notice that the crawl results are different. In this article, we will explain why this happens and what to keep in focus when analyzing differences in crawls. Next, we will describe in points the possible reasons why the results of crawls differ. Different […]
Understanding how the JetOctopus crawler works will help you better understand the results of your crawling. The main principle of our crawler is very similar to search engines. However, you can make many additional settings with JetOctopus. We want to emphasize that if there are errors in the crawl results or there are not all […]
We recommend using advanced settings to crawl JavaScript websites. Thanks to the specific settings you will be able to analyze your JS version in more detail. We’ve written an article about crawl configuration before, go to this article to read about standard crawl configuration for all types of websites. And in the article “How to […]
If you crawl your website with a regular JetOctopus user agent, it will not be blocked by Cloudflare. However, if you want to check how your site is seen by Google bot using crawling, Cloudflare will block unconfirmed Googlebot. We remind you that you can crawl your website with the Googlebot Mobile or Googlebot Desktop […]
For large multinational/multilingual websites, it can be difficult to track traffic dynamics in different countries. Monitoring the performance of all countries in total is an ineffective method for SEO. Because the fall in 1% of traffic in the most traffic-generating country will be very noticeable, and in the general chart this fall will be almost […]
If you find errors in crawl results or logs, you can submit the corrected/updated pages for scanning to search engines. This can be done with a custom sitemap. JetOctopus allows you to generate a custom sitemap in two clicks. In which cases do you need to generate a sitemap? For example, due to a technical […]
Happens when you run a crawl and the website returns a 503 response code. “503 Service Unavailable” response code means that your web server cannot process the request at this time. What are the reasons for the 503 status code? There is good news: the 503 response code is temporary, and as soon as the […]
Before you start a new crawl we recommend preparing basic information about your website. It allows you to set up the optimal configuration of the crawl. Of course, the basic settings of JetOctopus crawl are suitable for most websites, but it is worth delving into the settings to get the most useful information about your […]
We want to have clean and accessible code on our website. Also, we want search engines to scan only pages needed in search results because we worry about our crawling budget. But sometimes there are situations when URLs in the code are blocked by the robots.txt file. It means that search engines cannot access the […]
In JetOctopus crawl you can find the pages with a 404 response code. This response code means that the page was not found and the content is not available to your users. To find all 404 pages on your website, use JetOctopus crawler. Among other things, you can use the free 7-days trial after demo […]
Building a new website for your business? Don’t make the mistake of treating search engine optimization (SEO) as an afterthought. Why? Because SEO holds tremendous power as a marketing, brand-building (awareness and credibility), and revenue-driving channel for your business. Here are just some of the numerous statistics that corroborate the importance of SEO for your […]
Google updates its algorithm every other day. Hence, SEO is increasingly evolving into a complex and dynamic discipline. Most of these updates are minor and do not significantly change the SEO landscape. However, a few updates like the Core update and Page Experience update have changed the way search experts approach SEO. Consequently, the search […]
You can choose several scanning modes: scanning the entire website, scanning URLs from sitemaps or crawling pages from a list. Crawl list mode is very useful when you need to check metadata, status codes, load time and get other data about specific pages. It’s a very handy tool, but there are a few life hacks […]
Get exclusive tech SEO insights
