How to scan a password protected website with JetOctopus?

Using JetOctopus, you can easily scan a staging or test website, even if it is password-protected or requires login credentials for access. Here’s how to scan a password-protected website using JetOctopus.

Configure the crawl settings by entering the link to the home page of the test/staging website in the “URL” field.

Pay attention to the configuration of the robots.txt file for the staging/test domain. If the file contains the lines “User-agent: * Disallow: /” and the staging domain is blocked from scanning, you need to add the custom robots.txt file in the “Custom robots.txt” field under “Advanced Settings”. Otherwise, JetOctopus will not be able to crawl your website because our crawler follows the same rules as search engines.

It is recommended to use the same robots.txt file that will be used after the release of the staging website. This will help you identify any crawlability issues in advance. However, you can also block specific parts of the staging website in the robots.txt file if you don’t want JetOctopus to crawl them, such as a test admin panel.

Take note of the meta robots and HTTP headers used on the test website. If the default setting is “noindex, nofollow”, you need to disable the “Respect robots rules” and “Follow links on pages with” checkboxes. This allows JetOctopus to ignore the indexing restrictions and crawl all pages on your test website.

Pay attention to the crawling load. Test websites often have limited resources and can be overloaded by the crawler. To prevent this, it is advisable to use a smaller number of threads unless you are certain that the staging website can handle the load. If the website gets overloaded, you may receive 5xx status codes, and it will not be possible to analyze metadata, titles, and other important SEO elements.

If you need to add a password, enter login and password in the “HTTP Auth” field under “Advanced Settings” to allow JetOctopus to crawl your staging website. Rest assured that passwords are not stored and are protected, ensuring the safety of your data.

Finally, click on the “Start Crawl” button. The provided password and login will only be used for this particular crawl and will not be stored thereafter.

By following these steps, you can effectively scan password-protected staging or test websites using JetOctopus.