This morning around 3:30AM (EST) we started to experience packet loss that was causing slowness and timeout across our entire network. We contacted GoRACK at 4:05AM after attempting to troubleshoot and determining the problem was outside of our network. GoRACK began investigating and found that the cause of the disruption was a large inbound DDoS attack (the size of the attack was enough to impact their 10Gbps lines). At around 5:00AM we noticed that the network was getting better and we were able to access our servers and websites again, but this was just the calm before the storm it appears. At 6:07AM the whole network went down including GoRACK's website which was previously unaffected by the DDoS attack. At 7:01AM the whole network returned online and all tests are showing "all clear" (no packet loss or slowness). One thing we have noticed is that our tests are showing the traffic is now routing through TWTC when it was previously going over Level3.
Unfortunatelty there was nothing we could have done to prevent this from happening as we were not the focus of the attack (at least based on the monitoring tools GoRACK provides us). GoRACK was quick to respond though and we are thankful for their hard work.
One thing that we have learned from this is our need to provide communication to our clients during such distasters. We were providing updates over Twitter (@SecureDragonLLC) but we understand that this does not reach everyone in a timely manner. We are working to provide a failover website that we can use in the event of another disaster so SecureDragon.net will always be accessible. Lastly, we will be focusing on completing our information site (Drgn.biz) and move it to an outside host to allow another means of communication for our clients.
Feel free to contact us if you have any questions or concerns and we will be happy to help you as best we can. Thank you.
-The Secure Dragon Staff
Friday, December 23, 2011
