Network Issue
Incident Report for VPSBlocks Pty Ltd
Postmortem

A rogue VM had been hacked and a scan rootkit had been uploaded onto the server. Although we have limits for network bandwidth, we believe the rootkit scanner had saturated connection limits within our network.

The VM in question has been decomissioned and the owner alerted. We are investigating the best method to prevent a re-occurrence of this should a similar thing occur again in the future.

Please be aware this is the first time an incident of this nature has occurred in over three years of operation. We do apologise for the downtime and are working hard to find a resolution to ensure it does not reoccur. Since our implementation of bandwidth outbound limitations and inbound DDOS protection our network has been solid, and we expect this to continue into the future.

We have implemented a monitor which picks up large spikes in packets sent on VMs and takes action accordingly. This is being trialed over the next week and will then be put into production. This should ensure that any re-occurrence of the above issue should not impact clients.

All our clients are valued clients, and we would like to take the opportunity to thank you for your continued support.

Posted Jun 16, 2015 - 22:27 AEST

Resolved
This incident has been resolved.
Posted Jun 16, 2015 - 22:15 AEST
Identified
The incident was caused by a rogue VM flooding the network beyond the virtualization limits. We are investigating exactly how this is possible.
Posted Jun 16, 2015 - 21:55 AEST
Investigating
We are investigating a network outage.
Posted Jun 16, 2015 - 21:04 AEST