May 25, 2022
by Sofia

How to schedule a crawl

You can run the crawl yourself or set up a regular crawl. Also, you can set the start of a one-time crawl at the time you need, for example, at night. With a regular crawl, you can track changes to your website after releases, the number of new pages created over a while, and more.

To set up a regular crawl, go to the desired project and click “New scheduled crawl”. You can schedule a crawl only in the project menu. 

How to schedule a crawl - JetOctopus - Step 1

When creating a crawl, you can specify the frequency, day and time when the crawl will start. These settings can be made in the “Schedule” window. 

“Frequency”  – select daily, weekly, monthly or every two weeks. You can also select “Once” to start crawling later.

“Week Day”  – choose the day of the week when the crawl will run.

“Start From” is the date of the first crawl.

“Time Zone”  – select the time zone for running your crawl.

How to schedule a crawl - JetOctopus - Step 2

Next, make the default crawl settings.

More information: “How to configure a crawl of your website”.

Note that you can set up a regular crawl for the URL list. To do this, select the “URL list” crawl mode in “Basic settings”.

How to schedule a crawl - JetOctopus - Step 3

After completing all the settings, click the “Schedule crawl” button.

How to schedule a crawl - JetOctopus - Step 4

Why do you need a scheduled crawl?

With JetOctopus, you can set up a regular recurring crawl or run a crawl at the time you need. This is a great option for:

  • starting a crawl at a time when the website has the least number of visitors not to overload the web server;
  • check the releases of SEO tasks;
  • check the number of new URLs on the site during a period of a time;
  • check if the bug is fixed.

About Sofia
Technical SEO specialist. Sofia has almost 10 years of experience, of which the last 5 years in JavaScript SEO. She is convinced that SEO is a very technical part of digital marketing. And without logs and in-depth data analysis, you can't do effective SEO.



Find technical SEO errors at your website at free trial
Get exclusive tech SEO insights
We are tech SEO geeks who believe that SEO is predictable and numeric. Don’t miss our insigths!
Start your free trial