What is the “URLs Query in Crawl” report and how to use it

The “URLs Query in Crawl” report in JetOctopus provides information about all query parameters found during the crawl of your website.

What are query parameters?

Query parameters, also known as URL query parameters or GET parameters, are components of a URL that are used to request specific resources or information from a web server through the HTTP protocol. They typically consist of a name-value pair. These parts are separated by the sign “=”. 

A query parameter always begins with a question mark (?).

An example of a URL query with a query parameter: https://jetoctopus.com/blog?find=internal_links.

Why is it important to analyze URLs with query parameters?

Analyzing URLs with query parameters is important for several reasons. 

Firstly, having numerous query parameters can result in the inefficient usage of your crawl budget. Yes, Google and other bots can freely scan URLs with query parameters, if they are not blocked by the robots.txt file. So, excessive URLs with query parameters can consume crawl budget unnecessarily because URLs with query parameters can include search queries, sorting options, filtering criteria, and more. And most of these URLs with query parameters will be duplicates of the page without the GET parameter: whether the offers are sorted in the most popular order or in the most recent order, the information on the page will not change. Many query parameters can be created unnoticeable to users, but can be scanned by bots. 

How to find URLs with query parameters?

To identify URLs with query parameters, follow these steps using JetOctopus.

Start a new crawl of your website using the JetOctopus platform. During the crawl configuration, ensure that the “Do not crawl URLs with GET parameters” checkbox is deactivated, allowing the crawling of URLs with query parameters.

After the crawl is complete, access the crawl results and navigate to the data table. Select the “Others” section and choose the “URLs Query in Crawl” dataset.

The results will be displayed in descending order, with the query parameters having more associated URLs ranked higher.

All numbers are clickable. Click on the number next to GET-parameter to go to the data table with a detailed list of URLs with this query parameter. 

What to do with URLs with query parameters?

When dealing with URLs containing GET parameters, perform the following analysis.

• Determine the importance of these query parameters. Some websites use query parameters for pagination or creating filters, which can be beneficial for search engine results pages (SERPs).
• Assess whether search bots should crawl these pages. If certain pages with query parameters are not significant, you can block them using the robots.txt file.
• Consider the indexability of these pages and whether they contain canonical tags. If the pages are not crucial for your website, you can make them non-indexable.
• Remove internal links pointing to unnecessary URLs with query parameters.

Based on the analysis, make decisions on how to proceed. Retain only the necessary query parameters, ensuring they serve a specific purpose and contribute to the overall functionality and visibility of your website.