Product Update. Searchbot’s behavior in dynamics

JetOctopus offers a new solution for in-depth analysis of robots’ behavior on your website. The “Bot Dynamics” report is our powerful tool for tracking the behavior of bots: Google search robots, Google Image Bot, Bing, Baidu, marketing and other bots. This report allows you to effectively analyze changes in crawling on your website in two clicks, in particular, to track the impact of crawl budget optimization measures.

This update allows you to compare the crawl budget for the entire website and different types of pages against other periods, detect new and lost pages in the logs, and monitor the technical aspects of crawling your website by bots.

Missed and New Pages in Logs

Now you can highlight the pages that the crawler has stopped crawling, as well as determine which new pages the search bot has discovered during the selected period. If measures have been taken to optimize the crawl budget, then you will be able to see whether search engines have stopped scanning unnecessary pages, or vice versa: whether bots can more efficiently detect and scan new pages.

Compare crawl budget in pages and in visits

Analyze the total number of pages visited by search engines and compare them with the number of visits. All chart columns are clickable. Clicking on the desired column of the chart will take you to the data table with a list of pages.

Status codes in dynamics

Manually checking data and comparing status codes is time-consuming. That’s why we created this chart so you can spot anomalies in your website’s health and bots’ behavior in time. If the number of non-200 status codes has increased compared to the previous period, this can reduce the frequency of crawling of the site by search engines and negatively affect the crawl budget.

Average Page Size and Load Time

Two great technical reports that make it possible to analyze two more factors that have an impact on the crawl budget. If the load time increases, the number of scanned pages may decrease. As a result, search bots may not detect new pages in time or find updates on already existing pages. Page Size also affects the crawling budget, because not only the total number of pages is important for scanning, but also the number of megabytes downloaded.

To learn more about the capabilities of JetOctopus log analyzer, check out several articles on our blog. If you have not yet experienced the benefits of our log analyzer, we invite you to a demo.