According to statistics, medium-sized websites are most often attacked by fake and malicious bots. Therefore, it is important to monitor the scanning statistics of not only verified Google, Bing and other search engine bots, but also to detect and block malicious bots. In this article, we will tell you how to detect such bots using […]
Analysis of the activity of search bots is an important part of SEO, which should be paid a lot of attention. If the search engines do not visit the site regularly, do not update the pages and do not discover new ones, the website will not get the full potential of organic traffic. Therefore, we […]
We want to share key insights from Serge Bezborodov’s brilliant talk at BrightonSEO. Serge talked about how to control Googlebot and why it is needed for all SEOs. The presentation was based on a long-term analysis of the activity of search robots. 1. On average, ~40-60% of pages of most sites are uncrawled by Googlebot. […]
Google is one of the most popular search engines. Many SEO specialists closely follow the updates of algorithms of this search engine. In particular, a lot of attention is paid to changing the version of the browser with which the Googlebot imitates users. Also, the ratio of crawling performed by desktop and mobile robots is […]
The frequency of crawling of different subdomains by search engines may differ. This may depend on the indexing settings, the robots.txt file (for each subdomain there must be a separate robot.txt), correctly formed sitemaps, etc. But in addition to the domains open for indexing and scanning, the logs may contain technical, staging, and other subdomains […]
Ajax URLs may not be displayed correctly in search results because they are an asynchronously processed part of the page that depends on user action. Although the Ajax URLs have a different address, the content may be identical with the main page. Therefore, it is necessary to ensure that search bots do not scan Ajax […]
5xx response codes indicate a broken web server. These status codes are usually temporary. However, a large number of 5xx and regular repeats can affect the crawling of your website by search engines. Why do you need to monitor 5xx response codes? First, Google search engines will slow down the crawling of your website if […]
You may have noticed that the number of crawled URLs in the Google Search Console “Crawl stats” report may differ from the number of URLs visited by search engines that you see from the JetOctopus logs. Why this is so and how to correctly count the number of visited pages, read in this article. 1. […]
If search engines receive a lot of 404 response codes, they can reduce the scanning frequency of your website. Also, 404 in the logs can indicate the following problems: Also, 404 response codes in the logs can highlight problems with the site or web pages that broke due to releases, updates, etc. How to find […]
You may meet a situation when you check a page that returned a 4xx or 5xx response code to the search engine (you found this data in the logs). And when checking manually, the page returns a 200 response code. What could be the reason for such a situation? 1. In the logs, you can […]
Regular scanning of the website by search robots ensures the updating of data in the index. The regularly crawled pages will also have an up-to-date snippet in the SERP. However, sometimes crawling is excessive and can cause server overload. If your website is popular and constantly updated, extreme crawling by search engines can even lead […]
When the search robot user-agent requests the web page using the header “If-Modified-Since” or “If-None-Match” and the page has not changed since the requested time (usually this is the last visit of the GoogleBot), the server returns a 304 response code. This signals to the search engine that the web page has not changed since […]
Page load time affects how often your website is crawled by search engines. The slower the pages, the less often the search engine will visit them. That is, pages that respond slowly during the visit of search robots negatively affect the crawling budget of your website. So, we recommend analyzing the slowest pages and optimizing […]
Analysis of the activity of search robots is an important part of technical SEO. Some robots may visit your website too often and overload your web server. At the same time, the number of sessions and conversions with this search engine may be close to zero. Therefore, the activity of such search robots should be […]
Analysis of search engine logs is an important part of the work of SEO specialists. With the help of log analysis, you can get insights on why your website is displayed in the SERP in a certain way. In addition, you may find issues in the logs that prevent your pages from being crawled and […]
Building a new website for your business? Don’t make the mistake of treating search engine optimization (SEO) as an afterthought. Why? Because SEO holds tremendous power as a marketing, brand-building (awareness and credibility), and revenue-driving channel for your business. Here are just some of the numerous statistics that corroborate the importance of SEO for your […]
Google updates its algorithm every other day. Hence, SEO is increasingly evolving into a complex and dynamic discipline. Most of these updates are minor and do not significantly change the SEO landscape. However, a few updates like the Core update and Page Experience update have changed the way search experts approach SEO. Consequently, the search […]
Being a bit competitive is always a good thing, especially in business. Keeping an eye on your top rivals — what they’re doing, how they’re doing it, and the output of their efforts — spurs innovation, discourages complacency, and provides inspiration to improve. The same holds true in SEO. Whether your website is ranking on […]
Have you ever had a chance to look into the live tail of your website’s access log? You’d expect to see a lot of different user-agents of users’ browsers on your website. But more likely, you’ll find a variety of bots. What is bot traffic? In simple words, when a user uses an internet browser […]
Google is going all-in on user experience. With the introduction of Core Web Vitals and a host of other user-centric updates, the increasing obsession on optimizing the user’s search journey — from entering a search query to landing on the right page with the best possible content quality, relevancy, and load speed — is Google’s […]
With each update, Google’s ranking algorithm is becoming increasingly smarter in determining the quality of content and user experience on websites. Sure, time-tested SEO tactics such as link building and keywords optimization aren’t going anywhere anytime soon — and you must continue working on these as usual — but there are certain SEO trends worth […]
Get exclusive tech SEO insights
