While on site at E Solutions, we had been performing some tests on our primary and backup routers and discovered our BGP was not working properly (primarily our blackhole community). We worked with the data center staff and got this resolved and we resumed our DOS testing. After 5 successful tests, we began returning the router to it's normal configurations and began receiving alerts that our network was down. We attempted to restore connectivity by rebooting our primary router so it would failover to our backup but this had no effect. We got the data center staff involved again and after an hour and 12 minutes of troubleshooting, digging, and attempted fixes the network came back online on it's own while we were discussing our options. The primary router was back online and working normally but our backup router is still unable to establish any BGP connections at this time. In addition to these issues, we had 2 minor DDOS attacks right after connectivity was restored but these stopped shortly after they began.
We are still trying to determine our course of action but we do have 1 router functioning normally for now. We will continue to monitor the network and hope to have it restored to 100% by this weekend.
UPDATE: We have rebuilt the backup router and it is working properly at this time. We have successfully failed the primary router over to the backup without any network loss and are running on our backup router at this time. We will switch back after we've been on the backup router for 15 minutes.
-The Secure Dragon Staff
Friday, September 7, 2012
