What goes in a robots.txt file for more protection?
in [FrontPage]
|
From: Tom Willett on 3 Mar 2010 08:44 And, any visitor can easily open the robots.txt file and look at it. -- Tom [Pepper] Willett Microsoft MVP - FrontPage Since 1997 --------------------------- "Hot-text" <hot-text(a)hotmail.com> wrote in message news:CA8F70AF-7955-41C0-B089-DD8AC5860188(a)microsoft.com... : # : # From HTML-Kit : # This is an empty robots.txt file that can be used to indicate to : # WWW robots which parts of a web site should not be indexed. : # : # Following page has details on how to setup a robots.txt file and : # related security issues: : # : # http://www.chami.com/tips/internet/010198I.html : # : : : # 1. Uncomment following two lines by removing "#" at the beginning to : # indicate that all robots should stay away from the /cgi-bin/ folder : : #User-agent: * : #Disallow: /cgi-bin/ : : : # 2. Following two lines indicate that robots named "WWW_Robot_Name" : # should stay away from this site. : : #User-agent: WWW_Robot_Name : #Disallow: / : : : # 3. Following two lines indicate that robots named "WWW_Wanderer_Name" : # should not index files in the /private/ folder. : : #User-agent: WWW_Wanderer_Name : #Disallow: /private/ : : : "robots" <robots(a)discussions.microsoft.com> wrote in message : news:288033DF-3FFE-45FB-85F3-BEAA1AC9FBE6(a)microsoft.com... : > If there is a robots.txt file in the root directory, how can spiders still : > list all my files even in the MEMBERS-ONLY section :
From: Hot-text on 3 Mar 2010 18:55 And he need to Generator a Google Sitemap/XML and a Yahoo Map/Text for the Robots will help too Tom [Pepper] Willett If he Read to here he see this link http://wonderwebware.com/sitemap-generator/ But he get the robots.txt he ask for right! "Tom Willett" <tom(a)youreadaisyifyoudo.com> wrote in message news:eZV2tetuKHA.800(a)TK2MSFTNGP04.phx.gbl... > And, any visitor can easily open the robots.txt file and look at it. > -- > Tom [Pepper] Willett > Microsoft MVP - FrontPage Since 1997 > --------------------------- > "Hot-text" <hot-text(a)hotmail.com> wrote in message > news:CA8F70AF-7955-41C0-B089-DD8AC5860188(a)microsoft.com... > : # > : # From HTML-Kit > : # This is an empty robots.txt file that can be used to indicate to > : # WWW robots which parts of a web site should not be indexed. > : # > : # Following page has details on how to setup a robots.txt file and > : # related security issues: > : # > : # http://www.chami.com/tips/internet/010198I.html > : # > : > : > : # 1. Uncomment following two lines by removing "#" at the beginning to > : # indicate that all robots should stay away from the /cgi-bin/ folder > : > : #User-agent: * > : #Disallow: /cgi-bin/ > : > : > : # 2. Following two lines indicate that robots named "WWW_Robot_Name" > : # should stay away from this site. > : > : #User-agent: WWW_Robot_Name > : #Disallow: / > : > : > : # 3. Following two lines indicate that robots named "WWW_Wanderer_Name" > : # should not index files in the /private/ folder. > : > : #User-agent: WWW_Wanderer_Name > : #Disallow: /private/ > : > : > : "robots" <robots(a)discussions.microsoft.com> wrote in message > : news:288033DF-3FFE-45FB-85F3-BEAA1AC9FBE6(a)microsoft.com... > : > If there is a robots.txt file in the root directory, how can spiders > still > : > list all my files even in the MEMBERS-ONLY section > : > >
First
|
Prev
|
Pages: 1 2 Prev: Files do not show outside Frontpage Next: CSS ( ul li ul = Hidden) not working |