Serge Bezborodov
Author

Serge Bezborodov

Co-founder and CTO, JetOctopus

Serge Bezborodov is the co-founder and CTO of JetOctopus, where he leads product engineering and has been building software for more than 10 years. His main focus is log file analysis and the way Googlebot and AI crawlers actually move through large websites, a topic he regularly takes to conference stages including BrightonSEO. On the blog, Serge shares findings from billions of crawled pages and log lines and shows how SEO teams can turn that data into fixes.

Recent Articles

How to calculate the crawl budget of your website with JetOctopus

Using search engine log analysis, you can determine how often search engines crawl your website, how many pages are visited during a certain period, and how long it takes to crawl the entire website to discover new pages and update content on existing pages. In general, this is called the crawl budget of your website. […]

Product Update. Searchbot’s behavior in dynamics

JetOctopus offers a new solution for in-depth analysis of robots’ behavior on your website. The “Bot Dynamics” report is our powerful tool for tracking the behavior of bots: Google search robots, Google Image Bot, Bing, Baidu, marketing and other bots. This report allows you to effectively analyze changes in crawling on your website in two […]

Fake bots: what it is and why you need to analyze it

According to statistics, medium-sized websites are most often attacked by fake and malicious bots. Therefore, it is important to monitor the scanning statistics of not only verified Google, Bing and other search engine bots, but also to detect and block malicious bots. In this article, we will tell you how to detect such bots using […]

Product Update. Verifying Googlebot with JetOctopus

We are happy to announce a new release from the JetOctopus team. From now on, you can verify Googlebot using our service. This tool is very useful for all websites, because in this way you will be able to detect spam crawlers and troublemaking web scrappers that emulate Googlebot. How JetOctopus verifies Googlebot For Googlebot […]

Product Update. Compare JS vs non-JS content with JetOctopus

The JetOctopus team continues to make new features and tools that meet the actual needs of SEO specialists. One of the main needs today is setting up technical SEO for JavaScript websites. Using JetOctopus, you can configure a JS crawl and analyze the results. And now we are happy to present another release that will […]

How to integrate logs manually

Analysis of the activity of search bots is an important part of SEO, which should be paid a lot of attention. If the search engines do not visit the site regularly, do not update the pages and do not discover new ones, the website will not get the full potential of organic traffic. Therefore, we […]

What is DFI (distance from index) and how to analyze it

As you know, internal linking is extremely important for users of your website and of course for search engines. All visitors, even artificial intelligence, want to access any page of your site in just a few clicks. And the simpler and clearer the path from the home page to the desired one, the better the […]

How to find lazy load images and how to analyze it

Website loading speed and performance are very important for search engines and users. Every year, the requirements for these metrics grow. One way to make a website faster is to optimize images. Lazy loading can help with this. However, how do you know which images are already lazy loaded and which are not? How to […]

How to analyze logs by subdomains

The frequency of crawling of different subdomains by search engines may differ. This may depend on the indexing settings, the robots.txt file (for each subdomain there must be a separate robot.txt), correctly formed sitemaps, etc. But in addition to the domains open for indexing and scanning, the logs may contain technical, staging, and other subdomains […]

How to check Ajax URLs in logs

Ajax URLs may not be displayed correctly in search results because they are an asynchronously processed part of the page that depends on user action. Although the Ajax URLs have a different address, the content may be identical with the main page. Therefore, it is necessary to ensure that search bots do not scan Ajax […]

How to check 5xx URLs in logs

5xx response codes indicate a broken web server. These status codes are usually temporary. However, a large number of 5xx and regular repeats can affect the crawling of your website by search engines. Why do you need to monitor 5xx response codes? First, Google search engines will slow down the crawling of your website if […]

How to find out the frequency of scanning by search robots

Regular scanning of the website by search robots ensures the updating of data in the index. The regularly crawled pages will also have an up-to-date snippet in the SERP. However, sometimes crawling is excessive and can cause server overload. If your website is popular and constantly updated, extreme crawling by search engines can even lead […]

How to audit 304 pages visited by GoogleBot/other bots

When the search robot user-agent requests the web page using the header “If-Modified-Since” or “If-None-Match” and the page has not changed since the requested time (usually this is the last visit of the GoogleBot), the server returns a 304 response code. This signals to the search engine that the web page has not changed since […]

How to find the slowest pages visited by search engines

Page load time affects how often your website is crawled by search engines. The slower the pages, the less often the search engine will visit them. That is, pages that respond slowly during the visit of search robots negatively affect the crawling budget of your website. So, we recommend analyzing the slowest pages and optimizing […]

How to analyze what types of search robots visit your website

Analysis of the activity of search robots is an important part of technical SEO. Some robots may visit your website too often and overload your web server. At the same time, the number of sessions and conversions with this search engine may be close to zero. Therefore, the activity of such search robots should be […]

Meet all our authors

Julia Nesterets

Julia Nesterets

Co-founder & CEO, JetOctopus

View articles
Serge Bezborodov

Serge Bezborodov

Co-founder and CTO, JetOctopus

View articles
Maksym Bohdan

Maksym Bohdan

Senior Content Writer, JetOctopus

View articles
Sofia Vatuliak

Sofia Vatuliak

Tech SEO Manager, JetOctopus

View articles

Let's connect

Join our community

For powerful insights, product updates, and proven SEO strategies — straight from the JetOctopus team.