What if you run a crawl and your website returns the 503 response code?

Happens when you run a crawl and the website returns a 503 response code. “503 Service Unavailable” response code means that your web server cannot process the request at this time.

What are the reasons for the 503 status code?

1. The server is under maintenance.
2. The server is overloaded.

There is good news: the 503 response code is temporary, and as soon as the server has the resources, it will process the request.

What to do when the crawl doesn’t continue because of 503

Check on your browser how you see the page or check how Googlebot sees your website. To do this, use the URL Testing Tool in Google Search Console or run a Mobile Friendly test. If the page has visible content, likely, your server is not processing requests from the JetOctopus user agent which crawls your website.

Therefore, change the user agent when starting a new JetOctopus crawl. Select “Googlebot Mobile” if the page loaded while testing in a mobile-friendly test or try “Googlebot Desktop”. You can change user agent in the “Basic settings”.

Also, you can change the crawler’s location in the “Advanced settings”.

If the 503 response codes appear regularly during a crawl, JetOctopus may be overloading your web server. We recommend reducing the number of threads to a minimum in the “Basic settings”.

You can also set a longer timeout in the “Advanced settings”.

We recommend checking for 503 in the search engine logs. If your server returns 503 to a search engine, your crawling budget will be reduced, as a consequence. Pages can be deindexed.

The 503 response code is temporary, that’s why you can save all 503 pages and recrawl them when your server will be more stable. Filter all 503 in the “Data Table” and click the buttom “Crawl the list” to start recrawl of 503 pages.

If you still see a lot of 503 response codes in the crawl results after all the crawl settings we described above, and if 503 pages are in the logs of search engines, contact the developers or DevOps. That can be a problem with your web server.