Web page returned non 200 to search bot. Why do I get a different response code while manual checking?
You may meet a situation when you check a page that returned a 4xx or 5xx response code to the search engine (you found this data in the logs). And when checking manually, the page returns a 200 response code. What could be the reason for such a situation? 1. In the logs, you can […]
How to find out the frequency of scanning by search robots
Regular scanning of the website by search robots ensures the updating of data in the index. The regularly crawled pages will also have an up-to-date snippet in the SERP. However, sometimes crawling is excessive and can cause server overload. If your website is popular and constantly updated, extreme crawling by search engines can even lead […]
Product Update. GSC URL Inspection Tool
We are happy to announce a big update. Now with the help of JetOctopus, you will be able to work with Google Search Console data faster and more efficiently. Features of the new GSC URL Inspection tool If you have a large website with millions of pages, it can be difficult to monitor the process […]
How to audit 304 pages visited by GoogleBot/other bots
When the search robot user-agent requests the web page using the header “If-Modified-Since” or “If-None-Match” and the page has not changed since the requested time (usually this is the last visit of the GoogleBot), the server returns a 304 response code. This signals to the search engine that the web page has not changed since […]
How to find the slowest pages visited by search engines
Page load time affects how often your website is crawled by search engines. The slower the pages, the less often the search engine will visit them. That is, pages that respond slowly during the visit of search robots negatively affect the crawling budget of your website. So, we recommend analyzing the slowest pages and optimizing […]
How to analyze what types of search robots visit your website
Analysis of the activity of search robots is an important part of technical SEO. Some robots may visit your website too often and overload your web server. At the same time, the number of sessions and conversions with this search engine may be close to zero. Therefore, the activity of such search robots should be […]
How to analyze crawl results for a separate mobile domain
If you use separate mobile domains, you need a special crawl analysis. In the article “How to crawl separate mobile domains with JetOctopus” we wrote about how to configure crawling for a separate mobile site. In this article, we will tell you what to focus on during the crawl analysis of a separate mobile domain. […]
How to analyze the URLs most and least visited by GoogleBot
Google and other search engines regularly crawl the pages of most websites. However, search engine resources are limited, so bots can crawl certain pages very often and certain pages extremely rarely. Most often, regularly scanned pages are more visible in SERP. And the least scanned pages may even have outdated snippets. Therefore, it is important […]
How to find pages ignored by GoogleBot
Before indexing a page, search engines must make at least one visit to that page. For JavaScript sites, the number of visits before indexing will be at least 2 times. Accordingly, you need to monitor the pages ignored by GoogleBot to understand which of your pages will not be indexed. Why is it important to […]