Category: Crawlers
May 17, 2022
by Sofia
If you crawl your website with a regular JetOctopus user agent, it will not be blocked by Cloudflare. However, if you want to check how your site is seen by Google bot using...
May 11, 2022
by Sofia
For large multinational/multilingual websites, it can be difficult to track traffic dynamics in different countries. Monitoring the performance of all countries in total is an...
May 9, 2022
by Sofia
If you find errors in crawl results or logs, you can submit the corrected/updated pages for scanning to search engines. This can be done with a custom sitemap. JetOctopus allows...
May 6, 2022
by Sofia
Happens when you run a crawl and the website returns a 503 response code. “503 Service Unavailable” response code means that your web server cannot process the request at this...
May 2, 2022
by Sofia
Before you start a new crawl we recommend preparing basic information about your website. It allows you to set up the optimal configuration of the crawl. Of course, the basic...
April 29, 2022
by Sofia
We want to have clean and accessible code on our website. Also, we want search engines to scan only pages needed in search results because we worry about our crawling budget...
April 14, 2022
by Sofia
In JetOctopus crawl you can find the pages with a 404 response code. This response code means that the page was not found and the content is not available to your users. ...
April 10, 2022
by Julia Nesterets
Building a new website for your business? Don’t make the mistake of treating search engine optimization (SEO) as an afterthought.
Why? Because SEO holds tremendous power as a...
April 7, 2022
by Julia Nesterets
Google updates its algorithm every other day. Hence, SEO is increasingly evolving into a complex and dynamic discipline. Most of these updates are minor and do not significantly...
April 6, 2022
by Sofia
You can choose several scanning modes: scanning the entire website, scanning URLs from sitemaps or crawling pages from a list. Crawl list mode is very useful when you need to...
