Geocaching Australia
https://forum.geocaching.com.au/

Site Outage 9/1/2019
https://forum.geocaching.com.au/viewtopic.php?f=24&t=20012
Page 1 of 1

Author:  CraigRat [ 09 January 19 8:36 pm ]
Post subject:  Site Outage 9/1/2019

Here's the lowdown on today's outage

* reports early AM that site wasn't working.
* Initial thoughts were a full Hard Drive
* Remote diagnostics revealed our two SSDs used for our database were failed. Weird.
* Reboot caused server to go fully offline
* C@W attended site at 5:30pm and him and I worked through recovering the SSDs.
* After a lot of joggery-pokery we determined one SSD was totally NFG.
* Restored RAID config from the second drive (after having to purge a PERC RAID cache)
* Rebooted and tested, however for reasons unknown, databases were an old version from May 2018!!!!
* Restored from Last backup, however this is 24 hours old as the backup at 00:30 on 9/1/2019 hadn't run due to this failure, meaning we lost all data for 8/1/2019
* Tested, back online at ~ 20:35

We now need to replace the faulty SSD(s).

Author:  whitewebbs [ 09 January 19 9:09 pm ]
Post subject:  Re: Site Outage 9/1/2019

Thanks for all your time, effort and expertise in getting the site up and running again.

Author:  crew 153 [ 09 January 19 10:40 pm ]
Post subject:  Re: Site Outage 9/1/2019

Thanks for the update and all your hard work to get it up and running again.

I have copies of the imports which I sent through last night so I will resend them.

Author:  stainless-steel-rat [ 09 January 19 10:53 pm ]
Post subject:  Re: Site Outage 9/1/2019

Top work guys well done on working thru the issues and getting the site up and running again.

=D>

Author:  Zalgariath [ 10 January 19 7:39 am ]
Post subject:  Re: Site Outage 9/1/2019

Double your pay this month. :mrgreen:

Author:  mattyrx [ 10 January 19 8:07 pm ]
Post subject:  Re: Site Outage 9/1/2019

We really are lucky have developers with so much talent. What could have been an absolute disaster has been managed and fixed and now we're all back playing the game. Geocaching for me revolves around GCA - the stats, the caches, the games and the people here all put it well ahead of any other geocaching sites, in my opinion. Thankyou to all that make GCA the wonderful place that it is.

Author:  2y'stassies [ 11 January 19 7:33 am ]
Post subject:  Re: Site Outage 9/1/2019

Well done in getting everything up and running so soon. We all appreciate the work you do to give us access to so many useful tools. =D> =D>

Page 1 of 1 All times are UTC + 10 hours [ DST ]
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
http://www.phpbb.com/