Topic: How to block ill-behaved spiders?

[holck]

I want to block all requests from the "Bytespider" robot from Bytedance, but I can't make it work.

I've made a custom template fragment in /etc/e-smith/templates-custom/etc/httpd/conf/httpd.conf/VirtualHosts/26RewriteTraceAndTrack like this:

Code: [Select]

{
    $OUT =<<'HERE';
    RewriteEngine on
    RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK)
    RewriteRule .* - [F]
# Block Bytespider from Bytedance
    RewriteCond %{HTTP_USER_AGENT} "Bytespider"
    RewriteRule .* - [F,L]
#

HERE
}
And then made a signal-event e-smith-apache-update. The file /etc/httpd/conf/httpd.conf looks all right.
But the block doesn't seem to have any effect.

[ReetP]

Not sure but the line might need to earlier like here.

https://stackoverflow.com/questions/51972679/how-to-block-a-specific-user-agent-in-apache

Similar to this straight after RewriteEngine on (JP will undoubtedly point out how wring I am!)

Code: [Select]

RewriteEngine on
RewriteCond %{HTTP_USER_AGENT}  ^.*BingPreview.*$
RewriteRule . - [R=403,L]

And probably needs the regex too. Eg

Code: [Select]

^.*Bytespider.*$

If you have webapps installed run

Code: [Select]

signal-event webapps-update

Test the conf with

Code: [Select]

apachectl -t
Let us know.

[holck]

Sorry, my bad, it works smoothly.

I didn't notice that all requests from a Bytespider now get a 403 reply, just as they should.

Sorry!

[Jean-Philippe Pialasse]

another approach would be a local rule for fail2ban. 
i use to deploy one with a long list of spiders.  you might need to remove some according to your needs (could block yum in some cases if you have a repo on your server)