How to find JS blocked requests

The popularity of JavaScript technologies is growing rapidly. Every year, more and more websites are migrating to JavaScript. As a result, there is a growing need for high-quality SEO tools that are specifically designed for JavaScript, since SEO for JavaScript has its own unique characteristics. One important aspect of technical JS SEO is the analysis of requests. If certain parts of the JS requests or events are not functioning properly or are blocked, search engines will not be able to process the page correctly and some of the page’s content may be inaccessible. Therefore, in this article, we will explain how to analyze blocked JS requests on a massive scale.

What are JS blocked requests?

JS blocked requests refer to JavaScript files that are blocked from being loaded and executed by search engine robots due to restrictions specified in the robots.txt file.

When analyzing the page with the Mobile Friendly test tool, you could see Page resources that couldn’t be loaded in the list of problems. Sometimes, this list includes JavaScript files that are blocked by the robots.txt file. 

Such JS files are typically located on technical domains that are completely blocked by the robots.txt file. As a result, the Google search engine is unable to access these files and execute them. This can lead to issues with the proper processing of the page and result in certain content being inaccessible.

But manually checking all pages is not possible using Mobile Friendly Test. That’s why we at JetOctopus decided to make such a feature available to all our users.

How to find JS blocked requests on your website

To identify JS blocked requests on your website, you can use JetOctopus, a powerful SEO crawler that offers a JavaScript site analysis feature. Follow these steps to find the list of blocked requests on each page.

1. Go to the crawl settings and start the JavaScript crawl.

More information: How to configure crawl for JavaScript websites

2. After the crawl is complete, go to the data table with the results and select the “JS” data table. Then, select “Blocked Requests” to open a data table with a list of all requests (i.e., JavaScript files) that search engines will not be able to access.

3. Analyze if it is the same JavaScript file for all pages or if there are several different files blocked. It’s important to talk to the developers to find out what function these JavaScript files perform, as not all JavaScript requests should be accessible to bots. For example, access to customers’ favorite products may not need to be given to Google.

4. You can export all data to a format convenient for you, namely Google Sheets, CSV, Excel, and others.

Clicking on “use in API” will provide a request to the JetOctopus API, which you can use for your software environment. For instance, to automatically detect blocked JS requests after each release.

Why you should pay attention to JS blocked requests

It’s important to check for JS blocked requests on your website because if search engine robots are unable to access and execute certain JavaScript files, it can result in issues with processing the page and some content may become inaccessible for indexing and ranking. Identifying and addressing these blocked requests can help ensure that your website is fully optimized for search engines and users alike. 

JetOctopus offers many technical JavaScript SEO options. Here are some of them: 

How to check JS website with JetOctopus
Why do JavaScript and HTML versions differ in crawl results?
How to see how Googlebot renders JavaScript website
Webinar: How to Migrate a website to JavaScript. Best practices