Incident Report

Date: 29/08/2016

Service Affected: All Services

All times provided are AEST

Incident Time Started: 5:27AM

Incident Resolution: 10:10AM

Incident Description At 5:16AM utility power to the Omniconnect Data Center which houses VPSBlocks equipment was cut (due to a nearby traffic accident). The UPS systems kept all infrastructure online at the data center until the batteries depleted at 5:27AM.

The backup generator failed to start (usually this occurs within 30 seconds of a power failure). Attempts made to restart the generator manually failed initially. Power was restored at 6:25AM.

The data center has reported that the generator failed to start due to a faulty trickle charger. The generator undergoes weekly tests, but failed to start during the outage due to a faulty trickle charger which is designed to keep the generator battery at an optimum charge. This meant that the generator battery had depleted and therefore the generator could not start.

The trickle charger and battery have been replaced as well as upgraded, and the generator has been successfully tested.

To avoid a recurrence OMNIConnect will be adding a battery check and trickle charge test to their generator testing schedule.

The restoration of power brought most VPSBlocks services back online by 7am. However, we had to investigate and resolve issues relating to some of the systems caused by the power outage. Out of the servers which did not boot quickly, we managed to get all but one server back online by 8:30am ready for business hours. The final server took a little longer and we had that online by 10:10am.

During the outage we endeavoured to answer as many phone calls as possible to keep clients informed of the issues.

We will be working closely with OMNIConnect over the next few days to ensure that all precautions are taken at the data center to prevent any occurrences of this nature in the future.

We apologise for any inconvenience caused this morning.

Thank you for your continued support

The Team at VPSBlocks