How do I force Apache to have /robots.txt cast to a file regardless of domain?
I am running a local server with the following urls:
foo.self
bar.self
blah-blah.self
The above url is processed by the following expression VirtualHost
:
<VirtualHost *:80>
UseCanonicalName Off
ServerName self
ServerAlias *.self
VirtualDocumentRoot C:\Users\Foo\PhpstormProjects\%-2
<Directory C:\Users\Foo\PhpstormProjects\*>
Options Indexes FollowSymLinks Includes ExecCGI MultiViews
Order deny,allow
Allow from all
Require all granted
</Directory>
</VirtualHost>
Each has its own /robots.txt
, but I need any of the URLs to return the same, no matter what contains them /robots.txt
, or even if they don't exist. For example, the following URLs:
- foo.self / robots.txt
- bar.self / robots.txt
- blah-blah.self / robots.txt
... will return the same text:
User-agent: *
Disallow: /
It doesn't resort to 301 Redirect
or RewriteRule
.
source to share
Just create an alias for /robots.txt inside your hosts config pointing to the same file. In addition, the Location directive is required to grant access.
<VirtualHost *:80>
UseCanonicalName Off
ServerName self
ServerAlias *.self
Alias /robots.txt C:\Somfolder\robots.txt
<Location "C:\Somfolder\robots.txt">
Order deny,allow
Allow from all
</Location>
VirtualDocumentRoot C:\Users\Foo\PhpstormProjects\%-2
<Directory C:\Users\Foo\PhpstormProjects\*>
Options Indexes FollowSymLinks Includes ExecCGI MultiViews
Order deny,allow
Allow from all
Require all granted
</Directory>
</VirtualHost>
You add this config to all hosts, all of these directives Alias
point to the same file. The file C:\Somfolder\robots.txt
is a "regular" robots.txt file as you described it.
You can simplify this by including this directive as a template. Thus, you place the directive inside a separate config file and add the include directive to your host config:
<VirtualHost *:80>
UseCanonicalName Off
ServerName self
ServerAlias *.self
Include C:\path\to\file\robots.inc
VirtualDocumentRoot C:\Users\Foo\PhpstormProjects\%-2
<Directory C:\Users\Foo\PhpstormProjects\*>
Options Indexes FollowSymLinks Includes ExecCGI MultiViews
Order deny,allow
Allow from all
Require all granted
</Directory>
</VirtualHost>
File C:\path\to\file\robots.inc
:
Alias /robots.txt C:\Somfolder\robots.txt
<Location "C:\Somfolder\robots.txt">
Order deny,allow
Allow from all
</Location>
Please note that I am not aware of MS-Windows systems. So the examples of paths I've noticed might not make sense. But you should be able to get the idea :-)
source to share