Outage
Incident Report for VPSBlocks Pty Ltd

Incident Report

Date: 29/08/2016

Service Affected: All Services

All times provided are AEST

Incident Time Started: 5:27AM

Incident Resolution: 10:10AM

Incident Description At 5:16AM utility power to the Omniconnect Data Center which houses VPSBlocks equipment was cut (due to a nearby traffic accident). The UPS systems kept all infrastructure online at the data center until the batteries depleted at 5:27AM.

The backup generator failed to start (usually this occurs within 30 seconds of a power failure). Attempts made to restart the generator manually failed initially. Power was restored at 6:25AM.

The data center has reported that the generator failed to start due to a faulty trickle charger. The generator undergoes weekly tests, but failed to start during the outage due to a faulty trickle charger which is designed to keep the generator battery at an optimum charge. This meant that the generator battery had depleted and therefore the generator could not start.

The trickle charger and battery have been replaced as well as upgraded, and the generator has been successfully tested.

To avoid a recurrence OMNIConnect will be adding a battery check and trickle charge test to their generator testing schedule.

The restoration of power brought most VPSBlocks services back online by 7am. However, we had to investigate and resolve issues relating to some of the systems caused by the power outage. Out of the servers which did not boot quickly, we managed to get all but one server back online by 8:30am ready for business hours. The final server took a little longer and we had that online by 10:10am.

During the outage we endeavoured to answer as many phone calls as possible to keep clients informed of the issues.

We will be working closely with OMNIConnect over the next few days to ensure that all precautions are taken at the data center to prevent any occurrences of this nature in the future.

We apologise for any inconvenience caused this morning.

Thank you for your continued support

The Team at VPSBlocks

Posted over 1 year ago. Aug 29, 2016 - 16:36 AEST

Resolved
This incident has been resolved.
Posted over 1 year ago. Aug 29, 2016 - 15:16 AEST
Monitoring
All services should be back online. If your service is not back online please call 1300 839 448.

Please note that we do have a backlog of tickets at this time.
Posted over 1 year ago. Aug 29, 2016 - 10:20 AEST
Update
All services are online except for one server which needs to be rebuilt as a result of the power outage.
Posted over 1 year ago. Aug 29, 2016 - 09:29 AEST
Update
We are working on restoring services after a major power failure at the data center.
Posted over 1 year ago. Aug 29, 2016 - 07:19 AEST
Identified
The data center has had a major power outage it is being worked on.
Posted over 1 year ago. Aug 29, 2016 - 05:58 AEST
Investigating
We are currently investigating an outage
Posted over 1 year ago. Aug 29, 2016 - 05:34 AEST