July 20, 2022
by Sofia

How to check Ajax URLs in logs

Ajax URLs may not be displayed correctly in search results because they are an asynchronously processed part of the page that depends on user action. Although the Ajax URLs have a different address, the content may be identical with the main page. Therefore, it is necessary to ensure that search bots do not scan Ajax URLs and do not waste the crawling budget with this type of URLs. Using JetOctopus, you can quickly detect Ajax URLs in search engine logs.

Step 1. Go to the “Logs” section, select the “Health” dashboard. Select the desired type of search engine, period and domain.

How to check Ajax URLs in logs with JetOctopus

Find in the “Problems list (Bot’s behavior)” whether search crawlers have scanned Ajax URLs. Click on the number next to the problem to go to the data table for a detailed analysis.

How to check Ajax URLs in logs with JetOctopus - 1

In the “Problems list (Visitors’s behaviour) ” chart you will find similar information about Ajax URLs visited by users.

Pay attention to the “Dynamics of problems (bots)” chart to see when search engines were most actively crawling potential Ajax URLs.

How to check Ajax URLs in logs with JetOctopus - 2

Also focus on the “Dynamics of problems (organic visits)” diagram. Here you will see the dynamics of Ajax URLs visited by users during the period you selected. What does it mean? This means that Ajax URLs are already indexed and users go to such URLs from SERP. However, Ajax URLs may not contain all information that users need.

How to check Ajax URLs in logs with JetOctopus - 3

Step 2. Go to the “Raw Logs” data table to perform an in-depth analysis of the Ajax URLs visited by search robots.

How to check Ajax URLs in logs with JetOctopus - 4

You can add any other URL path to the filter, except /api|/react|/ajax, if you are sure how Ajax URLs are generated on your website. Also, you can add any other filters you need here.

How to check Ajax URLs in logs with JetOctopus - 5

Step 3. Bulk export Ajax URLs. Click the “Export” button and select the desired format. You can also export Ajax URLs from logs to DataStudio.

How to check Ajax URLs in logs with JetOctopus - 6

What to do if you find Ajax URLs in the logs:

  • inspect the dynamics of Ajax URLs; you can set up an alert that will notify you about the increase of Ajax URLs in the logs;
  • block Ajax URLs with the robots.txt file;
  • find entry points to Ajax URLs for search robots, for example, check if inlinks are contained it in texts, descriptions, etc., and replace those URLs;
  • check which Ajax URLs are indexed and, if necessary, adjust the canonical to the main page.

Remember that you should not immediately block Ajax URLs in robots.txt if they are in SERP. Analyze the potential loss of traffic and choose the best option to save it: canonical, redirect or block pages.

About Sofia
Technical SEO specialist. Sofia has almost 10 years of experience, of which the last 5 years in JavaScript SEO. She is convinced that SEO is a very technical part of digital marketing. And without logs and in-depth data analysis, you can't do effective SEO.


Find technical SEO errors at your website at free trial
Get exclusive tech SEO insights
We are tech SEO geeks who believe that SEO is predictable and numeric. Don’t miss our insigths!
Start your free trial
free 7-days trial