Custom extraction is needed to find certain elements in the HTML code of the page. Our web scraper will search selected elements while crawling your website. With custom extraction, you can use JetOctopus more efficiently and get non-standard information about your website. With JetOctopus, you can configure up to 50 custom extraction rules during a […]
If your website is global/international, it is important to track performance by countries. Consider an example why a general analysis of effectiveness in the SERP is not enough. Your website has visitors from two countries. Each country has a certain number of clicks and impressions and different conversion rates. Country A has a high conversion […]
In this article, we’ll talk about why a crawl of your website can fail. There are several reasons why a crawl of your website may not finish. Some of the reasons may be due to the settings of your web server and the fact that your server perceives our crawler as a potential security threat. […]
Using JetOctopus, you can customize as many segments as you need. We do not limit the number of segments. You can use URL segments when analyzing the results of crawling, logs and Google Search Console. Why you should segment your website Slice your site into as many segments as you need and watch your errors […]
If you have crawled the same website twice or more, you may notice that the crawl results are different. In this article, we will explain why this happens and what to keep in focus when analyzing differences in crawls. Next, we will describe in points the possible reasons why the results of crawls differ. Different […]
Understanding how the JetOctopus crawler works will help you better understand the results of your crawling. The main principle of our crawler is very similar to search engines. However, you can make many additional settings with JetOctopus. We want to emphasize that if there are errors in the crawl results or there are not all […]
We recommend using advanced settings to crawl JavaScript websites. Thanks to the specific settings you will be able to analyze your JS version in more detail. We’ve written an article about crawl configuration before, go to this article to read about standard crawl configuration for all types of websites. And in the article “How to […]
If you crawl your website with a regular JetOctopus user agent, it will not be blocked by Cloudflare. However, if you want to check how your site is seen by Google bot using crawling, Cloudflare will block unconfirmed Googlebot. We remind you that you can crawl your website with the Googlebot Mobile or Googlebot Desktop […]
For large multinational/multilingual websites, it can be difficult to track traffic dynamics in different countries. Monitoring the performance of all countries in total is an ineffective method for SEO. Because the fall in 1% of traffic in the most traffic-generating country will be very noticeable, and in the general chart this fall will be almost […]
If you find errors in crawl results or logs, you can submit the corrected/updated pages for scanning to search engines. This can be done with a custom sitemap. JetOctopus allows you to generate a custom sitemap in two clicks. In which cases do you need to generate a sitemap? For example, due to a technical […]
Happens when you run a crawl and the website returns a 503 response code. “503 Service Unavailable” response code means that your web server cannot process the request at this time. What are the reasons for the 503 status code? There is good news: the 503 response code is temporary, and as soon as the […]
Before you start a new crawl we recommend preparing basic information about your website. It allows you to set up the optimal configuration of the crawl. Of course, the basic settings of JetOctopus crawl are suitable for most websites, but it is worth delving into the settings to get the most useful information about your […]
We want to have clean and accessible code on our website. Also, we want search engines to scan only pages needed in search results because we worry about our crawling budget. But sometimes there are situations when URLs in the code are blocked by the robots.txt file. It means that search engines cannot access the […]
In JetOctopus crawl you can find the pages with a 404 response code. This response code means that the page was not found and the content is not available to your users. To find all 404 pages on your website, use JetOctopus crawler. Among other things, you can use the free 7-days trial after demo […]
Building a new website for your business? Don’t make the mistake of treating search engine optimization (SEO) as an afterthought. Why? Because SEO holds tremendous power as a marketing, brand-building (awareness and credibility), and revenue-driving channel for your business. Here are just some of the numerous statistics that corroborate the importance of SEO for your […]
Google updates its algorithm every other day. Hence, SEO is increasingly evolving into a complex and dynamic discipline. Most of these updates are minor and do not significantly change the SEO landscape. However, a few updates like the Core update and Page Experience update have changed the way search experts approach SEO. Consequently, the search […]
You can choose several scanning modes: scanning the entire website, scanning URLs from sitemaps or crawling pages from a list. Crawl list mode is very useful when you need to check metadata, status codes, load time and get other data about specific pages. It’s a very handy tool, but there are a few life hacks […]
In this article, we will tell you how to crawl large websites with millions of pages. With the help of different settings, you will be able to crawl your website most efficiently and get reliable results. Running a full crawl of your website is a great idea to apply the results to your entire website. […]
Being a bit competitive is always a good thing, especially in business. Keeping an eye on your top rivals — what they’re doing, how they’re doing it, and the output of their efforts — spurs innovation, discourages complacency, and provides inspiration to improve. The same holds true in SEO. Whether your website is ranking on […]
It has already become a tradition — to reflect on the results of the past year. Dear customers and colleagues, we are happy to share with you how well we fared in 2021. The volume of data processed since our inception (2018) log lines: 80bln crawled pages: 20bln websites: 40k Our customers Our customer geography […]
Have you ever had a chance to look into the live tail of your website’s access log? You’d expect to see a lot of different user-agents of users’ browsers on your website. But more likely, you’ll find a variety of bots. What is bot traffic? In simple words, when a user uses an internet browser […]
Get exclusive tech SEO insights
