Topic: robots/slurpers/spiders/site-downloaders

[Craig Jensen]

Hi.

I have been studying the options available through my SME 5.6 server to 'control' those who insist on downloading my entire site with their slurping applications.  I have noted a few that offer limited help.  

I have lately been stopping httpd for a given time and then re-starting, just so I can use a bit of MY OWN bandwidth for awhile  (BTW, does anyone else find that this seems to be more and more of a problem?)

Please if you will, enlighten me on methods you have used that efficiently 'choke' these bandwidth hogs...

Thank you for your replies.

Craig Jensen

[Reinhold]

Craig:

For those automatic grabbing tools you may want to make a robots.txt file...
for more info have a look at http://www.searchengineworld.com/robots/robots_tutorial.htm

[Michael Smith]

Not a problem here, because I'm not making anything available that anybody would want much of ... no binaries!

[Kirk Ferguson]

Hello.  We're using a .htaccess in the primary, something like is discussed at this site:

http://www.webmasterworld.com/forum13/687-3-15.htm

We copied 90e-smithAccess10primary to /etc/e-smith/templates-custom/etc/httpd/conf/httpd.conf/90e-smithAccess10primary

and added :

in

Options +FollowSymLinks

and changed:

AllowOverride None to :

AllowOverride All

Expanded and restarted httpd

This allows you to choose which spiders you want to allow, and to redirect those you don't want to a 404 page or another site.

[Craig Jensen]

Thank you for the responses.  I can see use for each option mentioned.

Thanks again

Craig Jensen

[Charlie Brady]

Kirk Ferguson wrote:

> Hello.  We're using a .htaccess in the primary, something
> like is discussed at this site:

There's never any need to use .htaccess, since you can use template fragments.

> This allows you to choose which spiders you want to allow,
> and to redirect those you don't want to a 404 page or another
> site.

You can do that with a custom template fragment. Using .htaccess means that you have your spider information in two places rather than one, and if you are not careful, might open up a security hole.

Charlie

[Kirk Ferguson]

Charlie,

Thanks for the help.   So I should try to include the commands (or something like them) in a custom template fragment rather than using that .htaccess file?  

Should this be part of the httpd.conf section for the primary?  

I sure don't need any more security worries, but the server this runs on is a a slow line and seems to attract more spiders than other servers I work with for some reason.

Kirk