Blog
Get exclusive tech SEO insights
Recent Articles
How to check Ajax URLs in logs
Ajax URLs may not be displayed correctly in search results because they are an asynchronously processed part of the page that depends on user action. Although the Ajax URLs have a different address, the content may be identical with the main page. Therefore, it is necessary to ensure that search bots do not scan Ajax […]
How to check 5xx URLs in logs
5xx response codes indicate a broken web server. These status codes are usually temporary. However, a large number of 5xx and regular repeats can affect the crawling of your website by search engines. Why do you need to monitor 5xx response codes? First, Google search engines will slow down the crawling of your website if […]
Why is a different number of logs displayed in GSC and in your logs?
You may have noticed that the number of crawled URLs in the Google Search Console “Crawl stats” report may differ from the number of URLs visited by search engines that you see from the JetOctopus logs. Why this is so and how to correctly count the number of visited pages, read in this article. 1. […]
Custom Extraction Guide with multiple use-cases
Custom extraction is needed to find certain elements in the HTML code of the page. Our web scraper will search selected elements while crawling your website. With custom extraction, you can use JetOctopus more efficiently and get non-standard information about your website. With JetOctopus, you can configure up to 50 custom extraction rules during a […]
Step-by-step instructions: analysis of 404 URLs in search engine logs
If search engines receive a lot of 404 response codes, they can reduce the scanning frequency of your website. Also, 404 in the logs can indicate the following problems: Also, 404 response codes in the logs can highlight problems with the site or web pages that broke due to releases, updates, etc. How to find […]
Web page returned non 200 to search bot. Why do I get a different response code while manual checking?
You may meet a situation when you check a page that returned a 4xx or 5xx response code to the search engine (you found this data in the logs). And when checking manually, the page returns a 200 response code. What could be the reason for such a situation? 1. In the logs, you can […]
How to find out the frequency of scanning by search robots
Regular scanning of the website by search robots ensures the updating of data in the index. The regularly crawled pages will also have an up-to-date snippet in the SERP. However, sometimes crawling is excessive and can cause server overload. If your website is popular and constantly updated, extreme crawling by search engines can even lead […]
Product Update. GSC URL Inspection Tool
We are happy to announce a big update. Now with the help of JetOctopus, you will be able to work with Google Search Console data faster and more efficiently. Features of the new GSC URL Inspection tool If you have a large website with millions of pages, it can be difficult to monitor the process […]
How to audit 304 pages visited by GoogleBot/other bots
When the search robot user-agent requests the web page using the header “If-Modified-Since” or “If-None-Match” and the page has not changed since the requested time (usually this is the last visit of the GoogleBot), the server returns a 304 response code. This signals to the search engine that the web page has not changed since […]