WebFaction
Community site: login faq

I'm trying to set up my robots.txt file such that when coming from certain third-level domains they will be shown a no-crawl robots.txt. I've been using the advice in this post, http://community.webfaction.com/questions/3649/separate-robotstxt-for-different-subdomains-that-point-to-the-same-application, but I can't get it to work with my site.

I'm using wsgi to serve django and I put the following in my httpd.conf file.

<Location "/robots.txt">
 SetHandler None
</Location>
alias /robots.txt /home/username/webapps/django_dev/robots.txt

<Location "/robots_no_crawl.txt">
 SetHandler None
</Location>
alias /robots_no_crawl.txt /home/username/webapps/django_dev/robots_no_crawl.txt

RewriteEngine on
RewriteCond ${HTTP_HOST} ^dev.mywebsite.com$ [NC,OR]
RewriteRule ^/robots.txt$ /robots_no_crawl.txt [P,L]

This results in the server eventually timing out with a 502 error.

asked 23 Sep '13, 14:28

theUNCHARTED
14
accept rate: 0%

edited 23 Sep '13, 14:52


Those directives should work. I would write them like this,

Alias /robots.txt /home/you/webapps/appname/robots.txt
RewriteEngine on
RewriteCond %{HTTP_HOST} ^dev.mywebsite.com$ [NC,OR]
RewriteCond %{REQUEST_URI} ^/robots.txt$
RewriteRule .* https://%{HTTP_HOST}/robots_no_crawl.txt [R,L]

There might be a minor syntax error and debugging in real-time would be easier. You may submit a support ticket if you would like us to do this.

Be sure to use curl to test the domain, as it does not cache or forward the result, but show the raw request which is useful for debugging as most browsers cache,

$ curl webfaction.com
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>
</head><body>
<h1>Found</h1>
<p>The document has moved <a href="http://www.webfaction.com/">here</a>.</p>
</body></html>
permanent link

answered 23 Sep '13, 18:26

johns ♦♦
5.3k212
accept rate: 23%

Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×904
×28
×5

question asked: 23 Sep '13, 14:28

question was seen: 1,700 times

last updated: 23 Sep '13, 18:26

WEBFACTION
REACH US
SUPPORT
AFFILIATE PROGRAM
LEGAL
© COPYRIGHT 2003-2019 SWARMA LIMITED - WEBFACTION IS A SERVICE OF SWARMA LIMITED
REGISTERED IN ENGLAND AND WALES 5729350 - VAT REGISTRATION NUMBER 877397162
5TH FLOOR, THE OLD VINYL FACTORY, HAYES, UB3 1HA, UNITED KINGDOM