Using search engine log analysis, you can determine how often search engines crawl your website, how many pages are visited during a certain period, and how long it takes to crawl the entire website to discover new pages and update content on existing pages. In general, this is called the crawl budget of your website. […]
JetOctopus offers a new solution for in-depth analysis of robots’ behavior on your website. The “Bot Dynamics” report is our powerful tool for tracking the behavior of bots: Google search robots, Google Image Bot, Bing, Baidu, marketing and other bots. This report allows you to effectively analyze changes in crawling on your website in two […]
According to statistics, medium-sized websites are most often attacked by fake and malicious bots. Therefore, it is important to monitor the scanning statistics of not only verified Google, Bing and other search engine bots, but also to detect and block malicious bots. In this article, we will tell you how to detect such bots using […]
Analysis of the activity of search bots is an important part of SEO, which should be paid a lot of attention. If the search engines do not visit the site regularly, do not update the pages and do not discover new ones, the website will not get the full potential of organic traffic. Therefore, we […]
Google is one of the most popular search engines. Many SEO specialists closely follow the updates of algorithms of this search engine. In particular, a lot of attention is paid to changing the version of the browser with which the Googlebot imitates users. Also, the ratio of crawling performed by desktop and mobile robots is […]
The frequency of crawling of different subdomains by search engines may differ. This may depend on the indexing settings, the robots.txt file (for each subdomain there must be a separate robot.txt), correctly formed sitemaps, etc. But in addition to the domains open for indexing and scanning, the logs may contain technical, staging, and other subdomains […]
Ajax URLs may not be displayed correctly in search results because they are an asynchronously processed part of the page that depends on user action. Although the Ajax URLs have a different address, the content may be identical with the main page. Therefore, it is necessary to ensure that search bots do not scan Ajax […]
5xx response codes indicate a broken web server. These status codes are usually temporary. However, a large number of 5xx and regular repeats can affect the crawling of your website by search engines. Why do you need to monitor 5xx response codes? First, Google search engines will slow down the crawling of your website if […]
You may have noticed that the number of crawled URLs in the Google Search Console “Crawl stats” report may differ from the number of URLs visited by search engines that you see from the JetOctopus logs. Why this is so and how to correctly count the number of visited pages, read in this article. 1. […]
Get exclusive tech SEO insights
