We successfully copied all data from the second failed drive to a fresh drive, then rebuilt the array. The SAN is back online and all VPSs have been restarted. We apologise for this outage and hope it hasn't inconvenienced our clients. This was a unique situation with two drives that relied on each other failing within 30 minutes (while the rebuild after the first failure was occurring), thankfully we had previously been assured that if this happened data should still be recoverable as SSDs tend to fail on write rather than read.
Posted Apr 23, 2015 - 03:36 AEST
Update
We are currently running DDRescue on the second failed drive, we're not sure how long this will take but are hopeful it will be done within a couple of hours. This mirrors the failed drive sector by sector onto a new drive, which we should then be able to replace into the RAID array recovering it to degraded status, at which time it should be active and able to rebuild to it's mirror.
Posted Apr 23, 2015 - 00:58 AEST
Identified
We have had a SAN RAID failure and are currently investigating. We do not have an ETA at this time. We ask all clients to be patient while we work on the problem.