At 11:15pm last night we experienced a loss of signal on almost all of our
Tukwila to Portland routes. We believe this to have been a maintenance period
that we were not notified of by our underlaying dark fiber carrier. Our guess
is that the southbound TX fiber strand on our 360/Zayo fiber path had a micro
bend inserted into it while a case was open as the timing is very suspect. The
error manifested itself as a very low signal to noise ratio on all the channels
in that direction. Unfortunately, this happened the same week that our high
capacity (8X 10G) protect path is being turned up so that diversity was
unavailable to us and only smaller links were have in place were available and
promptly saturated.
At 11:30PM we engaged our provider to go out and do testing on the circuit
involved, while we also dispatched to two of the sites in order to make sure
our equipment was working correctly. They all arrived on-site by 2AM and began
testing andworking to configure a work around in order to service our customers
traffic. From initial testing last night we believe that the problem occurred
somewhere between Spanaway, WA and Portland, OR. All services were restored at
around 3:30AM. We also placed some additional capacity within Portland in order
to protect against any further disruption and should have full diversity
available by early next week which will allow us to take down the entire system
for a full once-over of all components.
RESOLVED:
Hello, Our techs have completed cleaning and testing the primary path between
Portland and Tukwila. We have moved all traffic back over to the primary path
and it appears to be stable, running with no errors. We have left the backup
path in place for the time being until the primary path has continued to run
clean for serveral days. We appreciate your patience with us while we resolved
problems with this path.
-The Secure Dragon Staff
Viernes, Augosto 23, 2013
