Why are there not all titles, descriptions and headings in the crawl results?

Previously, we already wrote about why there could be missed pages in the crawl results. The reason can be both in the crawl settings and in the internal linking structure of your website. But there are situations when all pages in the crawl results are displayed, but without titles, meta descriptions or with other titles and meta descriptions. And when checking manually, everything is displayed correctly. What could be the reasons? Does it affect search engines? What metadata does Google view? You will find all this information in this article.

Pages are displayed with empty titles, meta data and headings in crawl results

Crawled pages may be displayed with empty or missing title and meta descriptions if your website uses JavaScript. JetOctopus does not process JS by default. So run a JavaScript crawl to get all the processed metadata.

For the same reason, the crawl results may show the same titles and descriptions for all scanned pages.

Many websites on JS use a default template for the title and meta data. And only after the execution of all JavaScripts files by the client browser, unique meta data for all pages is displayed.

The results of the crawl do not show the same meta-data as during the manual check

One of the reasons can also be JavaScript. If you are sure that your website uses static code, you need to check how many titles and meta descriptions are in your website code. JetOctopus displays in the crawl results only the first title and meta-description it finds in the code. To check this, go to the crawl results – the “Pages” data table.

Next, set the “<title> count” – “> Greater than” – “1” filter.

In the results, you can find a list of all pages that have two or more titles.

Such a check can be done for the meta description. Set the filter “<meta name=”description”> count” – “> Greater than” – “1” to see pages with two or more meta descriptions in the HTML code.

Of course, you can check the number of headings on the page. We remind you that there may be several H1s on the page, this is not a problem for search robots.

We have listed common problems that we faced most often. But there may be other reasons:

• your server returns various content to users and search engines;
• the server replaces content based on GeoIP (different content is shown to users from different countries);
• if the status codes are configured incorrectly and 200 is displayed instead of 404 or 5xx (it is likely that in this case the title will be the same for all pages with the wrong status code).

How to check that JavaScript is the cause

To make sure that the crawl results show the wrong metadata because of JS, do a manual check or run a small test crawl using JavaScript.

To configure JS crawl, activate the checkbox “Execute JavaScript”. 

See the article “How to configure crawl for JavaScript websites” for more information about JavaScript crawling.

To check manually, open Chrome Dev Tools: press Fn+F12 or F12. Next, open the command line using Control+Shift+P. Select “Disable JavaScript” and launch the page.

After that, check the title, headings and meta description in the “Elements” tab.

Does it affect search results?

Yes, some of the points listed above can affect search results. For example, search engines can use the title without the right keywords for ranking. Similarly, the wrong meta-description may be displayed in the snippet.

If you have Geo IP redirects or display different content for users and robots, this can also have a negative impact on search results.