Allow Search robots
This does the opposite of above, allowing ONLY these web robots access. Other than rogue robots, configuring your robots.txt file correctly will keep most robots where you want them.
SetEnvIfNoCase User-Agent .*google.* search_robot SetEnvIfNoCase User-Agent .*yahoo.* search_robot SetEnvIfNoCase User-Agent .*bot.* search_robot SetEnvIfNoCase User-Agent .*ask.* search_robot Order Deny,Allow Deny from All Allow from env=search_robot
If you enjoyed this post, please consider to leave a comment or subscribe to the feed and get future articles delivered to your feed reader.


Comments
No comments yet.
Leave a comment