Currently the backup has been restored. Accounts that were not included could have been over the standard backup size or newly created with in the last 6 days. For those accounts they will be restored beginning now. Once completed, accounts who need more recent data - such as changed in the last 48 hours - will be able to request it once the next backup phase is restored.
Here are the details so far which we know:
Around 7AM EST the server monitoring displayed the webserver as down. The system was responding to ping and ssh accepted an initial connection but it was not completed. No details were on console.
The system was rebooted but failed to boot, due to a boot loader/grub error. The system was brought into a rescue mode to check the grub config and repair it. This was completed, however the server failed to boot giving little information on an error.
The system was brought into a live cd. Some partitions could be seen, but it was returning an error on a file system check. At this point a new server build was began in case backups needed to be restored from by another team
After several hours, it appeared nothing further could be done. With the new server ready the backup drives were moved to the new server to begin a restore.
Currently we do not know why the system failed, as it didn't show any harddrive errors before hand and the system had no noticeable errors otherwise. The server was also a recent build, less than 2 months, and had been a build to replace a server reaching end of life. A weekly backup was running during the time. Once the second and third phase backups are completed further testing will be done to try to determine where the error may have come from. The data on the previous server is now readable by our techs.