Dear All,

Yesterday (19th November 2018) we experienced a major outage approximately 2 hours in length, as well as some network latency / dropping of packets leading up to the outage.

A number of factors led to the outage not only occurring but failing to be dealt with in a timely manner. The underlying cause of the outage was a substantial high packet, low bandwidth attack that although some of the attack had been partially scrubbed by the DDOS scrubbing center - which we employ to scrub our international traffic before it reaches us - enough packets were reaching our network to cause a substantial outage.

Unfortunately, the initial problem was misdiagnosed by our staff as a possible hardware issue, we had also brought in a second network engineer to investigate shortly after the major outage began and he wasn’t able to immediately pinpoint the issue either.

We have not had an attack of this nature cause any kind of major outage in a number of years, and have spent this morning reviewing the incident, the procedures and how the misdiagnoses came about, as well as the correct ways to identify and mitigate in the future before any major outage begins.

We are also working with the DDOS scrubbing center to improve communication with them if any kind of substantial attack occurs, so that we can black-hole any target IP before it affects our network, and thus our clients and their services.

We are truly sorry for this incident, we value all our clients and realise you have a large range of providers to choose from. We always do our best to provide a great service, and as a result of this incident changes in our procedures will occur immediately to prevent and/or mitigate any interruption to client services as a result of this kind of attack in the future.

We look forward to your continued support as part of the VPSBlocks community.

Thanks,

Will Kruss
Technical Director