Igor Simic
4 years ago

AdsBot Google ignoring wildcard in robots.txt




Google Ad Bot is bot used by Google to check the landing pages where Google Ad is leading to, and it will ignore wildcard in robots.txt file. For example 
User-agent: *
Disallow: /
will be ignored and AdsBot-Google will still crawl your page.

To prevent AdsBot Google crawling your page you have to be specific and target only Google As bot. To prevent AdWords from crawling your entire site, add the following to your robots.txt file:

User-agent: AdsBot-Google
Disallow: /

To prevent AdWords from crawling your mobile site, add the following to your robots.txt file: 

User-agent: AdsBot-Google-Mobile
Disallow: /

To prevent Ads Bot crawling some part of your page, for example /shoping_card you can put this in your robots.txt file
User-agent: AdsBot-Google
Disallow: /shopping_cart/

But you have to be careful, disallowing landing pages used by your Google Adwords will lead to suspension of complete site or Google could disapprove your ad.

More info you can find here:
https://support.google.com/adwords/answer/2404197