How to configure crawl for JavaScript websites

We recommend using advanced settings to crawl JavaScript websites. Thanks to the specific settings you will be able to analyze your JS version in more detail. 

We’ve written an article about crawl configuration before, go to this article to read about standard crawl configuration for all types of websites. And in the article “How to configure crawl for JavaScript websites” we will focus on special crawl settings that apply only when scanning JS websites.

Step 1. Start the JS crawling

If you want to start crawling on the JS website, go to the desired project and click “+ New Crawl” in the top right corner of the screen.

Next, activate the “Execute JavaScript” checkbox in the “Basic settings”. Note that 1 page of JS crawl costs 2 regular pages.

Step 2. Make basic crawl settings

Configure the crawl as recommended in the article “How to configure a crawl of your website”.

Step 3. Make specific settings to scan the JS website.

Go to the “Advanced settings” field to make specific settings for JS crawling.

JS Execution max time

Set a longer timeout if the previous crawl of the JS version had timeout pages. This can be seen in the “Technical” – “Status codes report”. The default timeout for all pages is 10 seconds (most browsers use the same timeout by default). Some JavaScript websites need more time to run JavaScript. The reason may be both slow execution and a large amount of Java Script code. Or if you are scanning a test/staging website.

We recommend closely monitoring the timeout in the crawl results, as search engines will not crawl the rendered page if the timeout is longer than standard.

JS Execution render viewport size

There are cases when the execution of JavaScript depends on the viewport size. Therefore, you need to set the viewport size (width x height) correctly during crawling so that the JavaScript will be executed correctly. If you use only CSS to adapt the content of the page, you probably don’t need these settings.

You can customize the viewport size for desktop and mobile devices.

JS End Render Event

You can choose after which event the rendering of the JS version will end. Each page has its own specific lifecycle. Typically, the page lifecycle contains the following events:

• init;
• DOMContentLoaded;
• firstPaint;
• firstContentfulPaint;
• firstMeaningfulPaint;
• firstMeaningfulPaintCandidate;
• firstimagepaint;
• load;
• networkAlmostIdle;
• networkIdle.

The execution time of these events is an important metric when analyzing the JavaScript version. However, there may be events and processes in this lifecycle that are still active but not used by clients. And if the rendering ends before this event, the results of the crawl will not change in essence. However, choosing the right JS End Render Event will save the client browser resources.

 You can choose one of the options as JS End Render Event:

• networkAlmostIdle – default setting; after performing all active processes on the JS page, the crawler will move to the next page;

• networkIdle – crawler will wait for all processes on the page, even if they involve reloading the page. We recommend that you use this item only when you are sure that it is appropriate and that these elements will be useful in the results of the crawl.