Why your crawl can’t be finished.
Sometimes crawl fails because the JetOctopus bot is blocked by the webserver. If this problem occurs, you will the following information in your crawl project: a lot of URLs returning a 401 or 403 status code; the crawl slows down progressively, and appears to stop running completely; crawler shows ‘crawl error’ status; crawl shows that […]