Using JetOctopus, you can configure a JS crawl and see how the page is rendered by search bots. When checking a JS crawl, it is important to compare static HTML and JS. In particular, indexing rules and meta data should match. However, when comparing the results of a JS crawl and manually inspecting a particular page, you may notice some discrepancies. In this article, we will explain why this happens and what problems this may indicate.
You can choose to emulate Googlebot Desktop or Googlebot Smartphone when setting up the user agent crawl. However, even if you choose a desktop or mobile browser without Googlebot specification, JetOctopus will still crawl your website just like Googlebot does. After receiving data from your web server, the JetOctopus crawler renders the page using the headless version of the Chrome browser, just like Googlebot does. In most cases, the results of the crawl will reflect the real problems and issues that Googlebot is encountering on your website. However, the results of rendering by a verified Googlebot will differ since they depend on the load on your web server at a certain time of day and many other factors.
This can happen for various reasons.
If the results of manual inspection are different, check the JS crawl settings in the “Crawler” – “Settings” section.
Pay attention to the following points.
“Disable cache resources” – “Yes” – Googlebot does not cache resources during page processing, but a regular browser caches. Maybe the cached version in your browser is out of date and you need to clear the cache.
“JS Execution max time” – Googlebot usually waits for all JS to load, just like your browser (except in cases of extra-slow loading). On the other hand, if you set a small number in the crawl settings, results may be different because the crawler did not wait for slow scripts to load.
“JS Execution render viewport size” – if you use a non-standard browser viewport size when manually checking, results may vary.
JetOctopus saves data at the time of page check. Make sure there have been no developer releases or changes to the website.
If you use dynamic rendering for different user agents or bots, or if you have blocks that are only displayed for returning/registered users, or if there are other behaviors for users who have never visited your website before, the rendering results may vary. Check the page in an anonymous tab by clearing cookies and cache.