- Add robots: http://forums.oscomm...ost__p__1602572
- Stop worthless traffic: http://forums.oscomm...thless-traffic/
- High traffic: http://forums.oscomm...y-high-traffic/
- .htaccess (required Apache with modrewrite enabled)
- robots.txt -> as default robots.txt for good (whitelist) search engine
- robots.php -> provide dynamic robots.txt
.htaccess file
Options +FollowSymLinks Options -Indexes ServerSignature Off #BADENGINE #empty user-agent SetEnvIfNoCase User-Agent (^(\s+)?$) BADENGINE #others user-agent SetEnvIfNoCase User-Agent (some_user_agent) BADENGINE SetEnvIfNoCase User-Agent (another1_user_agent) BADENGINE SetEnvIfNoCase User-Agent (another2_user_agent) BADENGINE #UNCOMMENT for TESTING #SetEnv BADENGINE 1 #let robot access robots.txt SetEnvIfNoCase Request_URI "robots\.txt" ROBOTS_LET_IN <LimitExcept CONNECT> Order Allow,Deny Allow from all Deny from env=BADENGINE Allow from env=ROBOTS_LET_IN </LimitExcept> RewriteEngine On RewriteBase / RewriteRule ^robots\.txt$ /robots.php [NC,L]
Default robots.txt
User-agent: * Disallow: /includes/ Disallow: /cgi-bin/ Disallow: /admin/ Disallow: /some_others_folder/
Dynamic robots.php
<?php
error_reporting(0);
$ROBOTS_LET_IN = false;
$ROBOTS_BADENGINE = false;
$ROBOTS_NAME = '-';
if (isset($_SERVER["ROBOTS_LET_IN"]) || isset($_SERVER["REDIRECT_ROBOTS_LET_IN"])) {
$ROBOTS_LET_IN = true;
}
if (isset($_SERVER["BADENGINE"]) || isset($_SERVER["REDIRECT_BADENGINE"])) {
$ROBOTS_BADENGINE = true;
}
if (!$ROBOTS_LET_IN) {
//accessing robots.php directly
$header = array( "HTTP/1.1 404 Not Found", "HTTP/1.1 404 Not Found", "Content-Length: 0" );
foreach ( $header as $sent ) {
header( $sent );
}
die();
}
header("Content-Type:text/plain");
if ($ROBOTS_BADENGINE) {
//disallow all files and folders on all badengine
echo "User-agent: *\n";
echo "Disallow: /\n";
} else {
//print default robots.txt
echo @file_get_contents('robots.txt');
}
?>
You can test it by uncomment one line in .htaccess, than try to access yourdomain.com/robots.txt
#UNCOMMENT for TESTING SetEnv BADENGINE 1
@zaenal
Edited by Parikesit, 05 October 2011 - 10:10 AM.
