Why there are 5xx pages in the crawl results and why they are not reproduced with the manual checking

5xx response codes indicate that the web server cannot process a request from the client. During the crawl, the JetOctopus user agent is a client. 5xx status code is not a sign that the URL is incorrect or broken. This only indicates that the web server could not respond when the server received this request from the client. The server can process many requests from different clients at the same time, so the status codes may differ for each client.

Types of 5xx status codes

500 status code indicates an internal server error. It can be reproduced for visitors. 500 status code may also show an error in the code that JetOctopus detected during the scanning.

The server returns a 501 status code if it does not recognize the client method. Usually, there are no such mistakes during the crawl.

The 502 response code returns if your intermediate web server cannot receive a response from another server. May indicate a broken SSR.

503 status code indicates that the server is currently unavailable (overloaded or running technical work). This status code is often found in crawl results. Reasons 503 in the results of the crawl:

• too many requests from the JetOctopus user agent, so you can find 503 or another type of 5xx in the crawl results, and everything works fine for users;
• JetOctopus overloaded the server.

To avoid the 503 response codes in crawl results, reduce the number of threads in the “Basic settings” when setting the crawl. Your crawl will take more time, but you will get more accurate results.

Read more about the reasons for 503 and what to do with it in the article: “What if you run a crawl and your website returns the 503 response code?”.

504 response code indicates a timeout. 

There are also other types of 5xx status codes. There are specific response codes from CloudFlare, such as 520, 521, 522, etc. They show that there is a problem with the interaction between the client browser and CloudFlare.

How to find the reasons for 5xx response codes

To understand that the web server returns 5xx status codes only to the JetOctopus crawler, you need to reduce the number of threads and run a test crawl. You can also check if there were similar response codes for users and GoogleBot at the same time the crawl was running. If the server returned 5xx status codes for both users/search engines and JetOctopus, then at that time there were problems with your web server or it was overloaded.

When you manually check the URLs with a 5xx response code, the conditions may not be reproduced and you will receive a 200 response code. For example, your server is no longer overloaded or web server maintenance is over.

It is recommended to recrawl 5xx pages to see if the problem still exists. Most likely, 5xx will not be reproduced during the second crawl. If your web server returns the 5xx response codes, contact DevOps or the web server administrator to fix the problem.