If you crawl your website with a regular JetOctopus user agent, it will not be blocked by Cloudflare. However, if you want to check how your site is seen by Google bot using crawling, Cloudflare will block unconfirmed Googlebot.
We remind you that you can crawl your website with the Googlebot Mobile or Googlebot Desktop user-agent. To do this, select the user agent type in “Basic settings”.
Cloudflare has powerful protection for mitigating bot traffic, so when crawling using a Googlebot user agent with an unconfirmed IP, the crawler will be blocked. The crawler will receive a 403 response code.
Cloudflare will perceive the crawler as a fake bot because the crawler’s IP will not match the original IP address of the Googlebot.
Go to your Cloudflare account, and select the desired website from the list. Then select “Bots” in the “Security” menu. On the “Bot Report” chart, go to the “Configure Super Bot Fight Mode” menu.
The setting depends on your Cloudflare subscription:
After these settings, you can start crawling with JetOctopus using the Googlebot user agent.
After the crawl is finished be sure to reset all Cloudflare settings to ensure strong protection from fake bots.
You can also add a list of exceptions by IP address. To do this, select the desired site from the list in Cloudflare. Next, go to the “Security” menu and select “WAF (Web Application Firewall)”. Click “Create firewall rule” to add a list of IPs that Cloudflare should not block.
Select “IP Source Address” from the drop-down filter, select “equals “ and enter JetOctopus IP-s:
Needs to select the “OR” directive between IP addresses.
Click “Allow” and save the rule.