Page 1 of 1

Bots are running amok

Posted: 29 December 14 3:07 pm
by caughtatwork
Folks.

We've been suffering from some major issues recently with performance of the database. I've tracked it down to a couple of rogue spiders / bots hitting us, causing very long running queries and gobs of data heading outbound.

We have a robots.txt file, but it looks like at least one of them doesn't respect the robots.txt file so I may need to brute force it out.

Please bear with us for a while as we try to keep the bots out which will hopefully put the system back in better shape.

Thanks.

Re: Bots are running amok

Posted: 30 December 14 10:59 am
by SamCarter
Spiders are an occupational hazard when caching ...

Love your work; good luck with the pest eradication.

Re: Bots are running amok

Posted: 30 December 14 4:16 pm
by CraigRat
Image

Re: Bots are running amok

Posted: 30 December 14 6:06 pm
by caughtatwork
I've 403'd the most obnoxious and robots.txt'd the other 2, so they should die off in a day or so.
All things look back to normal.

Re: Bots are running amok

Posted: 30 December 14 10:14 pm
by WazzaAndWenches
Love your work C@W.
=D> =D> =D> =D>

Re: Bots are running amok

Posted: 02 January 15 6:38 pm
by blossom*
Yep I vote you 2 guys are still as wonderful as when I first started caching =D>