How to crawl separate mobile domains with JetOctopus
Although Google recommends using the responsive version of the website for mobile devices, many websites still use a separate domain for mobile users. This means that the website for desktop and mobile has different URL addresses.
For example, your main desktop domain is located at example.com and your mobile domain is located at m.example.com. When visitors using a mobile phone go to the example.com domain, they are redirected by the 302 redirect to m.example.com.
In this article we will tell you how to crawl separate mobile and desktop versions in a simple way.
Option 1. Select a mobile user agent.
If you have 302 redirects for visitors who use mobile devices, you can select one of the mobile user agents when setting up crawling for the main domain. You can choose a standard mobile browser or scan your website with a mobile GoogleBot user agent. Only the pages of the mobile domain will be in the crawl results.
Please note that if your mobile version contains less internal links than the desktop version, the JetOctopus crawler will scan only those URLswhich will be found in the mobile version’s code.
Option 2. Use a mobile domain as a starting page for crawling.
In the “URL” field, enter the URL-address of the mobile domain and deactivate the “Crawl Subdomains” checkbox so that our crawler does not switch to the desktop version of the website. Additionally, we recommend choosing a mobile user agent.
If you want to crawl both the desktop and mobile versions, activate the “Crawl Subdomains” checkbox. And be sure to activate the “Special crawling mode for mobile subdomain” checkbox.
This checkbox allows our crawler to process canonicals correctly for the separate mobile domain. If there are problems with the canonicals on the mobile domain, we will show it in special reports.
Thanks to “Special crawling mode for mobile subdomain”, our crawler will be able to check whether rel = “canonical” corresponds to rel = “alternate”. After all, in order for search engines to correctly process the connection between the mobile and desktop versions, you need to add these elements in the code of the pages.
The desktop version must contain rel = “alternate” with a link to the relevant mobile URL-address. And the mobile address must contain rel = “canonical” in the <head> with a link to the corresponding desktop address inside the canonical.
How to scan desktop and mobile domains in one crawl
To check how the desktop and mobile versions combine/differ, you can scan them as part of one crawl.
With JetOctopus, you can scan desktop and mobile separate domains within a single crawl. To do this, make the following settings:
- enter the address of the main desktop domain in the start URL field;
- activate the “Crawl Subdomains” checkbox;
- activate checkbox “Special crawling mode for mobile subdomain”.
How to check the correctness of redirects for mobile devices
If you use 302 redirects for mobile devices from the desktop site to the mobile domain, we recommend checking the correctness of redirects for the mobile GoogleBot.
You can use the “URL List” mode and enter a list of desktop addresses in the test field.
Next, select the GoogleBot Mobile user agent and start crawling.
You can also take a list of URLs from the desktop crawl and send them to the recrawl using a mobile user agent.
- Category:
- Crawlers
- How to guides
- On-page Tech SEO
About Sofia
Technical SEO specialist. Sofia has almost 10 years of experience, of which the last 5 years in JavaScript SEO. She is convinced that SEO is a very technical part of digital marketing. And without logs and in-depth data analysis, you can't do effective SEO.