Using search engine log analysis, you can determine how often search engines crawl your website, how many pages are visited during a certain period, and how long it takes to crawl the entire website to discover new pages and update content on existing pages. In general, this is called the crawl budget of your website. […]
JetOctopus offers a new solution for in-depth analysis of robots’ behavior on your website. The “Bot Dynamics” report is our powerful tool for tracking the behavior of bots: Google search robots, Google Image Bot, Bing, Baidu, marketing and other bots. This report allows you to effectively analyze changes in crawling on your website in two […]
According to statistics, medium-sized websites are most often attacked by fake and malicious bots. Therefore, it is important to monitor the scanning statistics of not only verified Google, Bing and other search engine bots, but also to detect and block malicious bots. In this article, we will tell you how to detect such bots using […]
Analysis of the activity of search bots is an important part of SEO, which should be paid a lot of attention. If the search engines do not visit the site regularly, do not update the pages and do not discover new ones, the website will not get the full potential of organic traffic. Therefore, we […]
We want to share key insights from Serge Bezborodov’s brilliant talk at BrightonSEO. Serge talked about how to control Googlebot and why it is needed for all SEOs. The presentation was based on a long-term analysis of the activity of search robots. 1. On average, ~40-60% of pages of most sites are uncrawled by Googlebot. […]
Google algorithms have recently been updated: now only the first 15 MB of page loading are taken into account for indexing and ranking. At the same time, images, videos, CSS and JavaScript files are fetched separately. Sounds threatening, doesn’t it? However, it is worth noting that a 15 MB page is quite a large page. […]
Google is one of the most popular search engines. Many SEO specialists closely follow the updates of algorithms of this search engine. In particular, a lot of attention is paid to changing the version of the browser with which the Googlebot imitates users. Also, the ratio of crawling performed by desktop and mobile robots is […]
In this article, we will explain why a situation may occur when you cannot see all the visits of search engines in the logs data tables in JetOctopus. We will show you what to do if you are sure that a search engine visited your website, but there are no loglines displaying these visits. You […]
The frequency of crawling of different subdomains by search engines may differ. This may depend on the indexing settings, the robots.txt file (for each subdomain there must be a separate robot.txt), correctly formed sitemaps, etc. But in addition to the domains open for indexing and scanning, the logs may contain technical, staging, and other subdomains […]
Get exclusive tech SEO insights
