In this article, we will explain why a situation may occur when you cannot see all the visits of search engines in the logs data tables in JetOctopus. We will show you what to do if you are sure that a search engine visited your website, but there are no loglines displaying these visits. You […]
The frequency of crawling of different subdomains by search engines may differ. This may depend on the indexing settings, the robots.txt file (for each subdomain there must be a separate robot.txt), correctly formed sitemaps, etc. But in addition to the domains open for indexing and scanning, the logs may contain technical, staging, and other subdomains […]
Ajax URLs may not be displayed correctly in search results because they are an asynchronously processed part of the page that depends on user action. Although the Ajax URLs have a different address, the content may be identical with the main page. Therefore, it is necessary to ensure that search bots do not scan Ajax […]
5xx response codes indicate a broken web server. These status codes are usually temporary. However, a large number of 5xx and regular repeats can affect the crawling of your website by search engines. Why do you need to monitor 5xx response codes? First, Google search engines will slow down the crawling of your website if […]
You may have noticed that the number of crawled URLs in the Google Search Console “Crawl stats” report may differ from the number of URLs visited by search engines that you see from the JetOctopus logs. Why this is so and how to correctly count the number of visited pages, read in this article. 1. […]
If your website is perfectly on-page optimized, but the search engines get a lot of non-200 response codes, this can decrease the crawling budget. Search engines will crawl your website less often, pages will be updated less often in the SERP, and new ones will be shown with a lag. To avoid this, we recommend […]
If search engines receive a lot of 404 response codes, they can reduce the scanning frequency of your website. Also, 404 in the logs can indicate the following problems: Also, 404 response codes in the logs can highlight problems with the site or web pages that broke due to releases, updates, etc. How to find […]
You may meet a situation when you check a page that returned a 4xx or 5xx response code to the search engine (you found this data in the logs). And when checking manually, the page returns a 200 response code. What could be the reason for such a situation? 1. In the logs, you can […]
Regular scanning of the website by search robots ensures the updating of data in the index. The regularly crawled pages will also have an up-to-date snippet in the SERP. However, sometimes crawling is excessive and can cause server overload. If your website is popular and constantly updated, extreme crawling by search engines can even lead […]
Get exclusive tech SEO insights
