Bots are running amok
Posted: 29 December 14 3:07 pm
Folks.
We've been suffering from some major issues recently with performance of the database. I've tracked it down to a couple of rogue spiders / bots hitting us, causing very long running queries and gobs of data heading outbound.
We have a robots.txt file, but it looks like at least one of them doesn't respect the robots.txt file so I may need to brute force it out.
Please bear with us for a while as we try to keep the bots out which will hopefully put the system back in better shape.
Thanks.
We've been suffering from some major issues recently with performance of the database. I've tracked it down to a couple of rogue spiders / bots hitting us, causing very long running queries and gobs of data heading outbound.
We have a robots.txt file, but it looks like at least one of them doesn't respect the robots.txt file so I may need to brute force it out.
Please bear with us for a while as we try to keep the bots out which will hopefully put the system back in better shape.
Thanks.