Clever Geek Handbook

How can I prevent googlebot from bypassing Ajaxified Links?

I have a bunch of ajaxified links that do things like poll, vote, post-standard moderation community flag.

The problem is googlebot crawls these links and votes, votes down and flag items.

Will this be added to my robots.txt file to prevent googlebot from crawling those links? Or is there something else I need to do?

User-agent: Googlebot 
Disallow: /item/*/flag/
Disallow: /item/*/vote/

Thank!

EDIT: Modified method for post and googlebot is still tagged with posts.

Syntax

:

<a href="onclick="$.ajax({data:'authenticity_token=' + encodeURIComponent('blahblahblah'), dataType:'script', type:'post', url:'/item/flag/236?method=post'}); return false;">

any thoughts?

+2


source to share


2 answers


You should change these links as POST and not GET, then you don't have to worry about crawlers. There are other reasons for this as well: web caches may decide to cache the GET and therefore not actually push to your server when the user clicks on the link. They will not cache POST.



+4


source


Apart from making POST requests, since they are transformative, guest users (people who are not logged in) should not be allowed to vote on community moderation. So just exclude them from the vote and that fixes your bot problem.



0


source






More articles:


All Articles