Site Outage 9/1/2019
- CraigRat
- 850 or more found!!!
- Posts: 7019
- Joined: 23 August 04 3:17 pm
- Twitter: CraigRat
- Facebook: http://facebook.com/CraigRat
- Location: Launceston, TAS
- Contact:
Site Outage 9/1/2019
Here's the lowdown on today's outage
* reports early AM that site wasn't working.
* Initial thoughts were a full Hard Drive
* Remote diagnostics revealed our two SSDs used for our database were failed. Weird.
* Reboot caused server to go fully offline
* C@W attended site at 5:30pm and him and I worked through recovering the SSDs.
* After a lot of joggery-pokery we determined one SSD was totally NFG.
* Restored RAID config from the second drive (after having to purge a PERC RAID cache)
* Rebooted and tested, however for reasons unknown, databases were an old version from May 2018!!!!
* Restored from Last backup, however this is 24 hours old as the backup at 00:30 on 9/1/2019 hadn't run due to this failure, meaning we lost all data for 8/1/2019
* Tested, back online at ~ 20:35
We now need to replace the faulty SSD(s).
* reports early AM that site wasn't working.
* Initial thoughts were a full Hard Drive
* Remote diagnostics revealed our two SSDs used for our database were failed. Weird.
* Reboot caused server to go fully offline
* C@W attended site at 5:30pm and him and I worked through recovering the SSDs.
* After a lot of joggery-pokery we determined one SSD was totally NFG.
* Restored RAID config from the second drive (after having to purge a PERC RAID cache)
* Rebooted and tested, however for reasons unknown, databases were an old version from May 2018!!!!
* Restored from Last backup, however this is 24 hours old as the backup at 00:30 on 9/1/2019 hadn't run due to this failure, meaning we lost all data for 8/1/2019
* Tested, back online at ~ 20:35
We now need to replace the faulty SSD(s).
- whitewebbs
- 6500 or more caches found
- Posts: 368
- Joined: 05 February 11 6:39 pm
- Location: Sandford
Re: Site Outage 9/1/2019
Thanks for all your time, effort and expertise in getting the site up and running again.
-
- 9000 or more caches found
- Posts: 1099
- Joined: 09 October 04 7:51 pm
- Location: Calamvale, Brisbane
- Contact:
Re: Site Outage 9/1/2019
Thanks for the update and all your hard work to get it up and running again.
I have copies of the imports which I sent through last night so I will resend them.
I have copies of the imports which I sent through last night so I will resend them.
- stainless-steel-rat
- Posts: 131
- Joined: 13 November 11 12:16 am
- Location: Hobart
Re: Site Outage 9/1/2019
Top work guys well done on working thru the issues and getting the site up and running again.
- Zalgariath
- 5500 or more caches found
- Posts: 1749
- Joined: 17 August 09 10:44 am
- Location: Sydney, NSW
Re: Site Outage 9/1/2019
Double your pay this month.
- mattyrx
- 850 or more found!!!
- Posts: 267
- Joined: 21 May 10 9:57 pm
- Location: The Channon, NSW
- Contact:
Re: Site Outage 9/1/2019
We really are lucky have developers with so much talent. What could have been an absolute disaster has been managed and fixed and now we're all back playing the game. Geocaching for me revolves around GCA - the stats, the caches, the games and the people here all put it well ahead of any other geocaching sites, in my opinion. Thankyou to all that make GCA the wonderful place that it is.
-
- 10000 or more caches found
- Posts: 372
- Joined: 19 January 10 7:54 pm
- Location: Ulverstone Tasmania
Re: Site Outage 9/1/2019
Well done in getting everything up and running so soon. We all appreciate the work you do to give us access to so many useful tools.