Allow Search robots

This does the opposite of above, allowing ONLY these web robots access. Other than rogue robots, configuring your robots.txt file correctly will keep most robots where you want them.

SetEnvIfNoCase User-Agent .*google.* search_robot
SetEnvIfNoCase User-Agent .*yahoo.* search_robot
SetEnvIfNoCase User-Agent .*bot.* search_robot
SetEnvIfNoCase User-Agent .*ask.* search_robot

Order Deny,Allow
Deny from All
Allow from env=search_robot

If you enjoyed this post, please consider to leave a comment or subscribe to the feed and get future articles delivered to your feed reader.

Powered by WP | WP Remix Copyright 2007. Baldwinit.com Website Design. All rights reserved

• Home
• About
• Archives
• Tags