Bots are running amok

Discussion about the Geocaching Australia web site
Post Reply
User avatar
caughtatwork
Posts: 17015
Joined: 17 May 04 12:11 pm
Location: Melbourne
Contact:

Bots are running amok

Post by caughtatwork » 29 December 14 3:07 pm

Folks.

We've been suffering from some major issues recently with performance of the database. I've tracked it down to a couple of rogue spiders / bots hitting us, causing very long running queries and gobs of data heading outbound.

We have a robots.txt file, but it looks like at least one of them doesn't respect the robots.txt file so I may need to brute force it out.

Please bear with us for a while as we try to keep the bots out which will hopefully put the system back in better shape.

Thanks.

User avatar
SamCarter
1400 or more caches found
1400 or more caches found
Posts: 650
Joined: 13 March 07 10:32 am
Location: Hobart

Re: Bots are running amok

Post by SamCarter » 30 December 14 10:59 am

Spiders are an occupational hazard when caching ...

Love your work; good luck with the pest eradication.

User avatar
CraigRat
850 or more found!!!
850 or more found!!!
Posts: 7015
Joined: 23 August 04 3:17 pm
Twitter: CraigRat
Facebook: http://facebook.com/CraigRat
Location: Launceston, TAS
Contact:

Re: Bots are running amok

Post by CraigRat » 30 December 14 4:16 pm

Image

User avatar
caughtatwork
Posts: 17015
Joined: 17 May 04 12:11 pm
Location: Melbourne
Contact:

Re: Bots are running amok

Post by caughtatwork » 30 December 14 6:06 pm

I've 403'd the most obnoxious and robots.txt'd the other 2, so they should die off in a day or so.
All things look back to normal.

User avatar
WazzaAndWenches
5000 or more caches found
5000 or more caches found
Posts: 395
Joined: 08 April 07 10:28 pm
Location: Echuca, Vic

Re: Bots are running amok

Post by WazzaAndWenches » 30 December 14 10:14 pm

Love your work C@W.
=D> =D> =D> =D>

User avatar
blossom*
3000 or more caches found
3000 or more caches found
Posts: 1589
Joined: 25 February 09 1:59 pm
Location: West Ryde

Re: Bots are running amok

Post by blossom* » 02 January 15 6:38 pm

Yep I vote you 2 guys are still as wonderful as when I first started caching =D>

Post Reply