Django robots.txt rewrite depending on domain

Question

I'm trying to set up my robots.txt file such that when coming from certain third-level domains they will be shown a no-crawl robots.txt. I've been using the advice in this post, http://community.webfaction.com/questions/3649/separate-robotstxt-for-different-subdomains-that-point-to-the-same-application, but I can't get it to work with my site.

I'm using wsgi to serve django and I put the following in my httpd.conf file.

<Location "/robots.txt">
 SetHandler None
</Location>
alias /robots.txt /home/username/webapps/django_dev/robots.txt

<Location "/robots_no_crawl.txt">
 SetHandler None
</Location>
alias /robots_no_crawl.txt /home/username/webapps/django_dev/robots_no_crawl.txt

RewriteEngine on
RewriteCond ${HTTP_HOST} ^dev.mywebsite.com$ [NC,OR]
RewriteRule ^/robots.txt$ /robots_no_crawl.txt [P,L]

This results in the server eventually timing out with a 502 error.

Answer 1

Those directives should work. I would write them like this,

Alias /robots.txt /home/you/webapps/appname/robots.txt
RewriteEngine on
RewriteCond %{HTTP_HOST} ^dev.mywebsite.com$ [NC,OR]
RewriteCond %{REQUEST_URI} ^/robots.txt$
RewriteRule .* https://%{HTTP_HOST}/robots_no_crawl.txt [R,L]

There might be a minor syntax error and debugging in real-time would be easier. You may submit a support ticket if you would like us to do this.

Be sure to use curl to test the domain, as it does not cache or forward the result, but show the raw request which is useful for debugging as most browsers cache,

$ curl webfaction.com
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>
</head><body>
<h1>Found</h1>
<p>The document has moved <a href="http://www.webfaction.com/">here</a>.</p>
</body></html>

Django robots.txt rewrite depending on domain

Follow this question

Related questions