Site Outage 9/1/2019

Geocaching Australia governance issues
Post Reply
User avatar
CraigRat
850 or more found!!!
850 or more found!!!
Posts: 7015
Joined: 23 August 04 3:17 pm
Twitter: CraigRat
Facebook: http://facebook.com/CraigRat
Location: Launceston, TAS
Contact:

Site Outage 9/1/2019

Post by CraigRat » 09 January 19 8:36 pm

Here's the lowdown on today's outage

* reports early AM that site wasn't working.
* Initial thoughts were a full Hard Drive
* Remote diagnostics revealed our two SSDs used for our database were failed. Weird.
* Reboot caused server to go fully offline
* C@W attended site at 5:30pm and him and I worked through recovering the SSDs.
* After a lot of joggery-pokery we determined one SSD was totally NFG.
* Restored RAID config from the second drive (after having to purge a PERC RAID cache)
* Rebooted and tested, however for reasons unknown, databases were an old version from May 2018!!!!
* Restored from Last backup, however this is 24 hours old as the backup at 00:30 on 9/1/2019 hadn't run due to this failure, meaning we lost all data for 8/1/2019
* Tested, back online at ~ 20:35

We now need to replace the faulty SSD(s).

User avatar
whitewebbs
6500 or more caches found
6500 or more caches found
Posts: 358
Joined: 05 February 11 6:39 pm
Location: Sandford

Re: Site Outage 9/1/2019

Post by whitewebbs » 09 January 19 9:09 pm

Thanks for all your time, effort and expertise in getting the site up and running again.

crew 153
9000 or more caches found
9000 or more caches found
Posts: 1099
Joined: 09 October 04 7:51 pm
Location: Calamvale, Brisbane
Contact:

Re: Site Outage 9/1/2019

Post by crew 153 » 09 January 19 10:40 pm

Thanks for the update and all your hard work to get it up and running again.

I have copies of the imports which I sent through last night so I will resend them.

User avatar
stainless-steel-rat
Posts: 131
Joined: 13 November 11 12:16 am
Location: Hobart

Re: Site Outage 9/1/2019

Post by stainless-steel-rat » 09 January 19 10:53 pm

Top work guys well done on working thru the issues and getting the site up and running again.

=D>

User avatar
Zalgariath
5500 or more caches found
5500 or more caches found
Posts: 1749
Joined: 17 August 09 10:44 am
Location: Sydney, NSW

Re: Site Outage 9/1/2019

Post by Zalgariath » 10 January 19 7:39 am

Double your pay this month. :mrgreen:

User avatar
mattyrx
850 or more found!!!
850 or more found!!!
Posts: 264
Joined: 21 May 10 9:57 pm
Location: The Channon, NSW
Contact:

Re: Site Outage 9/1/2019

Post by mattyrx » 10 January 19 8:07 pm

We really are lucky have developers with so much talent. What could have been an absolute disaster has been managed and fixed and now we're all back playing the game. Geocaching for me revolves around GCA - the stats, the caches, the games and the people here all put it well ahead of any other geocaching sites, in my opinion. Thankyou to all that make GCA the wonderful place that it is.

2y'stassies
10000 or more caches found
10000 or more caches found
Posts: 370
Joined: 19 January 10 7:54 pm
Location: Ulverstone Tasmania

Re: Site Outage 9/1/2019

Post by 2y'stassies » 11 January 19 7:33 am

Well done in getting everything up and running so soon. We all appreciate the work you do to give us access to so many useful tools. =D> =D>

Post Reply