May 25, 2022
by Sofia

How to schedule a crawl

You can run the crawl yourself or set up a regular crawl. Also, you can set the start of a one-time crawl at the time you need, for example, at night. With a regular crawl, you can track changes to your website after releases, the number of new pages created over a while, and more.

To set up a regular crawl, go to the desired project and click “New scheduled crawl”. You can schedule a crawl only in the project menu. 

How to schedule a crawl - JetOctopus - Step 1

When creating a crawl, you can specify the frequency, day and time when the crawl will start. These settings can be made in the “Schedule” window. 

“Frequency”  – select daily, weekly, monthly or every two weeks. You can also select “Once” to start crawling later.

“Week Day”  – choose the day of the week when the crawl will run.

“Start From” is the date of the first crawl.

“Time Zone”  – select the time zone for running your crawl.

How to schedule a crawl - JetOctopus - Step 2

Next, make the default crawl settings.

More information: “How to configure a crawl of your website”.

Note that you can set up a regular crawl for the URL list. To do this, select the “URL list” crawl mode in “Basic settings”.

How to schedule a crawl - JetOctopus - Step 3

After completing all the settings, click the “Schedule crawl” button.

How to schedule a crawl - JetOctopus - Step 4

Why do you need a scheduled crawl?

With JetOctopus, you can set up a regular recurring crawl or run a crawl at the time you need. This is a great option for:

  • starting a crawl at a time when the website has the least number of visitors not to overload the web server;
  • check the releases of SEO tasks;
  • check the number of new URLs on the site during a period of a time;
  • check if the bug is fixed.

About Sofia
Technical SEO specialist. Sofia has almost 10 years of experience, of which the last 5 years in JavaScript SEO. She is convinced that SEO is a very technical part of digital marketing. And without logs and in-depth data analysis, you can't do effective SEO.



Get exclusive tech SEO insights
We are tech SEO geeks who believe that SEO is predictable and numeric. Don’t miss our insigths!