Category: Crawlers
October 12, 2022
by Sofia
Why can the number of indexed pages differ in Google’s SERP data and Google Search Console? Why is the number of indexable pages found by JetOctopus also different from these...
October 5, 2022
by Sofia
As you know, you can use a custom robots.txt file when configuring a crawl. This is a useful option if, for example, you want to scan a staging website. But there are other...
September 13, 2022
by Sofia
JetOctopus is a comprehensive tool for SEO that has many advantages and cool features. We would like to remind you about several cool features that will help you use JetOctopus...
September 9, 2022
by Sofia
The last six months have been interesting and exciting for the JetOctopus team. It was difficult at times, but we managed it. And as a result, many useful updates were released....
September 8, 2022
by Sofia
Use the Include and Exclude feature to prevent JetOctopus from scanning unwanted pages. We recommend that you use this option if you need to scan one or more types of pages. You...
August 12, 2022
by Sofia
Security is important to all websites. For JetOctopus, the security of your data is also in the first place. Therefore, we want to tell you which user agents JetOctopus uses when...
July 14, 2022
by Sofia
Custom extraction is needed to find certain elements in the HTML code of the page. Our web scraper will search selected elements while crawling your website. With custom...
July 11, 2022
by Sofia
Merging data from different tables is much more convenient if you use the “Join Dataset” tool. You can use this option in all data tables: logs, Google Search Console and crawl....
July 7, 2022
by Sofia
We offer several options if you want to share a JetOctopus report with your colleagues. Choose the most convenient way depending on what your needs are.
1. Share the PDF file...
June 27, 2022
by Sofia
If you use separate mobile domains, you need a special crawl analysis. In the article “How to crawl separate mobile domains with JetOctopus” we wrote about how to configure...
