Page 1 of 1

Site Outage 9/1/2019

Posted: 09 January 19 8:36 pm
by CraigRat
Here's the lowdown on today's outage

* reports early AM that site wasn't working.
* Initial thoughts were a full Hard Drive
* Remote diagnostics revealed our two SSDs used for our database were failed. Weird.
* Reboot caused server to go fully offline
* C@W attended site at 5:30pm and him and I worked through recovering the SSDs.
* After a lot of joggery-pokery we determined one SSD was totally NFG.
* Restored RAID config from the second drive (after having to purge a PERC RAID cache)
* Rebooted and tested, however for reasons unknown, databases were an old version from May 2018!!!!
* Restored from Last backup, however this is 24 hours old as the backup at 00:30 on 9/1/2019 hadn't run due to this failure, meaning we lost all data for 8/1/2019
* Tested, back online at ~ 20:35

We now need to replace the faulty SSD(s).

Re: Site Outage 9/1/2019

Posted: 09 January 19 9:09 pm
by whitewebbs
Thanks for all your time, effort and expertise in getting the site up and running again.

Re: Site Outage 9/1/2019

Posted: 09 January 19 10:40 pm
by crew 153
Thanks for the update and all your hard work to get it up and running again.

I have copies of the imports which I sent through last night so I will resend them.

Re: Site Outage 9/1/2019

Posted: 09 January 19 10:53 pm
by stainless-steel-rat
Top work guys well done on working thru the issues and getting the site up and running again.

=D>

Re: Site Outage 9/1/2019

Posted: 10 January 19 7:39 am
by Zalgariath
Double your pay this month. :mrgreen:

Re: Site Outage 9/1/2019

Posted: 10 January 19 8:07 pm
by mattyrx
We really are lucky have developers with so much talent. What could have been an absolute disaster has been managed and fixed and now we're all back playing the game. Geocaching for me revolves around GCA - the stats, the caches, the games and the people here all put it well ahead of any other geocaching sites, in my opinion. Thankyou to all that make GCA the wonderful place that it is.

Re: Site Outage 9/1/2019

Posted: 11 January 19 7:33 am
by 2y'stassies
Well done in getting everything up and running so soon. We all appreciate the work you do to give us access to so many useful tools. =D> =D>