May 6, 2022
by Sofia

What if you run a crawl and your website returns the 503 response code?

Happens when you run a crawl and the website returns a 503 response code. “503 Service Unavailable” response code means that your web server cannot process the request at this time.

What are the reasons for the 503 status code?

  1. The server is under maintenance.
  2. The server is overloaded.

There is good news: the 503 response code is temporary, and as soon as the server has the resources, it will process the request.

What to do when the crawl doesn’t continue because of 503

Check on your browser how you see the page or check how Googlebot sees your website. To do this, use the URL Testing Tool in Google Search Console or run a Mobile Friendly test. If the page has visible content, likely, your server is not processing requests from the JetOctopus user agent which crawls your website.

Therefore, change the user agent when starting a new JetOctopus crawl. Select “Googlebot Mobile” if the page loaded while testing in a mobile-friendly test or try “Googlebot Desktop”. You can change user agent in the “Basic settings”.

What to do when the crawl doesn’t continue because of 503 - JetOctopus

Also, you can change the crawler’s location in the “Advanced settings”.

What to do when the crawl doesn’t continue because of 503 - crawler location- JetOctopus

If the 503 response codes appear regularly during a crawl, JetOctopus may be overloading your web server. We recommend reducing the number of threads to a minimum in the “Basic settings”.

What to do when the crawl doesn’t continue because of 503 - threds - JetOctopus

You can also set a longer timeout in the “Advanced settings”.

What to do when the crawl doesn’t continue because of 503 - timeout - JetOctopus

We recommend checking for 503 in the search engine logs. If your server returns 503 to a search engine, your crawling budget will be reduced, as a consequence. Pages can be deindexed.

The 503 response code is temporary, that’s why you can save all 503 pages and recrawl them when your server will be more stable. Filter all 503 in the “Data Table” and click the buttom “Crawl the list” to start recrawl of 503 pages.

What to do when the crawl doesn’t continue because of 503 - crawl the list - JetOctopus

If you still see a lot of 503 response codes in the crawl results after all the crawl settings we described above, and if 503 pages are in the logs of search engines, contact the developers or DevOps. That can be a problem with your web server.

About Sofia
Technical SEO specialist. Sofia has almost 10 years of experience, of which the last 5 years in JavaScript SEO. She is convinced that SEO is a very technical part of digital marketing. And without logs and in-depth data analysis, you can't do effective SEO.

Search

Categories

Find technical SEO errors at your website at free trial
Get exclusive tech SEO insights
We are tech SEO geeks who believe that SEO is predictable and numeric. Don’t miss our insigths!
Start your free trial
free 7-days trial