Prev: IIS7 on Win7 Home Premium
Next: IIS 7 Catastrophic Failure :: COMException :: Microsoft.JScript
From: S N on 31 Jan 2010 12:07
How do I restrict specific folder access to web spiders on IIS (on Classic
ASP VBScript).
I have used Robots.txt. But I dont want people to even able to read the
contents of robots.txt (which they can read by just typing
http://mysite.com/robots.txt

Please help.


From: Marvin Landman on 31 Jan 2010 12:22
Hi,

What you want is basically impossible since you just cannot determine if
a user is a human or is a spider.

Also note that browsers usually ignore robots.txt so you are only able
to restrict spiders using robots.txt rather than browsers.

The best thing you can do is to examine
Request.ServerVariables("HTTP_USER_AGENT") and determine whether the
User-Agent header is representing a browser or a spider.

If you want to serve robots.txt using ASP then you either have to use a
404 error handler ASP page or map .txt to asp.dll in IIS manager.

Marvin

S N wrote:
> How do I restrict specific folder access to web spiders on IIS (on Classic
> ASP VBScript).
> I have used Robots.txt. But I dont want people to even able to read the
> contents of robots.txt (which they can read by just typing
> http://mysite.com/robots.txt
>
> Please help.
>
>
From: Dooza on 1 Feb 2010 04:01
On 31/01/2010 17:22, Marvin Landman wrote:
> Hi,
>
> What you want is basically impossible since you just cannot determine if
> a user is a human or is a spider.
>
> Also note that browsers usually ignore robots.txt so you are only able
> to restrict spiders using robots.txt rather than browsers.
>
> The best thing you can do is to examine
> Request.ServerVariables("HTTP_USER_AGENT") and determine whether the
> User-Agent header is representing a browser or a spider.

Also be aware that not all robots say they are robots, for instance the
msn-bot often pretends to be IE6 and can even execute JavaScript. I have
seen one page hits from the msn-bot range of IP addresses in stats that
are JavaScript triggered.

Dooza
From: Ken Schaefer on 27 Feb 2010 09:50
You need to implement some kind of authentication system.

You can either enable Basic/Digest/Integrated Windows Authentication, in
which case users will need to supply valid Windows credentials, or you can
implement something in ASP itself.

Cheers
Ken

--
http://adOpenStatic.com/blog

"S N" <sn(a)invalid.invalid> wrote in message
news:O7UzbfpoKHA.4044(a)TK2MSFTNGP06.phx.gbl...
> How do I restrict specific folder access to web spiders on IIS (on Classic
> ASP VBScript).
> I have used Robots.txt. But I dont want people to even able to read the
> contents of robots.txt (which they can read by just typing
> http://mysite.com/robots.txt
>
> Please help.
>
From: Dan on 1 Mar 2010 08:54

"S N" <sn(a)invalid.invalid> wrote in message
news:O7UzbfpoKHA.4044(a)TK2MSFTNGP06.phx.gbl...
> How do I restrict specific folder access to web spiders on IIS (on Classic
> ASP VBScript).
> I have used Robots.txt. But I dont want people to even able to read the
> contents of robots.txt (which they can read by just typing
> http://mysite.com/robots.txt
>
> Please help.

If you don't want anyone reading the contents of robots.txt then remove it -
it only works because the contents are read by spiders/bots that adhere to
the robots.txt specification, it's not like .htaccess on Apache where the
server uses the contents of the file to block access, so if you stop anyone
reading the file then you've just rendered it useless.

You can get add-ons for IIS that add .htaccess or similar server side file
handling, or you could code your own handler.

--
Dan

 | 
Pages: 1
Prev: IIS7 on Win7 Home Premium
Next: IIS 7 Catastrophic Failure :: COMException :: Microsoft.JScript