On August 5th we received multiple alerts that our network in Tampa, FL. was offline. We contacted our data center and they informed us their network engineers were working on the problem. The total downtime was roughly 44 minutes according to our monitoring. Here is the RFO provided by our data center:

Root Cause:
At 7:02 PM Eastern on August 5th 2016 a DDOS attack was launched against a customer. The size of the attack was over 18Gbps. This massive amount of data saturated the uplinks into the catalyst fabric of the data center and caused them to drop packets. This in turn overloaded the CPU on the core routers of the data center causing them to respond slowly and to drop OSPF and BGP sessions. This in turn caused further stress on the core routers and caused an outage to the internet due to saturation from the DDOS attack.

Corrective Action:

Upon determination that routers were over saturated with IP processing, bandwidth logs were reviewed. Once that happened we discovered the DDOS attack and blocked the attack. After the block was in place, the saturation of the core was alleviated and services returned to normal. The new Cisco Nexus platform provides the bandwidth and CPU power to avoid such overloads. We are continuing to migrate customers to the new fabric. Again this platform can better mitigate issues such as this.

Outage began: Aug 05 2016 17:59:06 GMT-5.0
Outage resolved: Aug 05 2016 18:43:35 GMT-5.0

-The Secure Dragon Staff

Tuesday, August 9, 2016





« Back