We want to have clean and accessible code on our website. Also, we want search engines to scan only pages needed in search results because we worry about our crawling budget. But sometimes there are situations when URLs in the code are blocked by the robots.txt file. It means that search engines cannot access the […]
In JetOctopus crawl you can find the pages with a 404 response code. This response code means that the page was not found and the content is not available to your users. To find all 404 pages on your website, use JetOctopus crawler. Among other things, you can use the free 7-days trial after demo […]
Building a new website for your business? Don’t make the mistake of treating search engine optimization (SEO) as an afterthought. Why? Because SEO holds tremendous power as a marketing, brand-building (awareness and credibility), and revenue-driving channel for your business. Here are just some of the numerous statistics that corroborate the importance of SEO for your […]
Google updates its algorithm every other day. Hence, SEO is increasingly evolving into a complex and dynamic discipline. Most of these updates are minor and do not significantly change the SEO landscape. However, a few updates like the Core update and Page Experience update have changed the way search experts approach SEO. Consequently, the search […]
You can choose several scanning modes: scanning the entire website, scanning URLs from sitemaps or crawling pages from a list. Crawl list mode is very useful when you need to check metadata, status codes, load time and get other data about specific pages. It’s a very handy tool, but there are a few life hacks […]
In this article, we will tell you how to crawl large websites with millions of pages. With the help of different settings, you will be able to crawl your website most efficiently and get reliable results. Running a full crawl of your website is a great idea to apply the results to your entire website. […]
Being a bit competitive is always a good thing, especially in business. Keeping an eye on your top rivals — what they’re doing, how they’re doing it, and the output of their efforts — spurs innovation, discourages complacency, and provides inspiration to improve. The same holds true in SEO. Whether your website is ranking on […]
It has already become a tradition — to reflect on the results of the past year. Dear customers and colleagues, we are happy to share with you how well we fared in 2021. The volume of data processed since our inception (2018) log lines: 80bln crawled pages: 20bln websites: 40k Our customers Our customer geography […]
Have you ever had a chance to look into the live tail of your website’s access log? You’d expect to see a lot of different user-agents of users’ browsers on your website. But more likely, you’ll find a variety of bots. What is bot traffic? In simple words, when a user uses an internet browser […]
Get exclusive tech SEO insights
