On September 12th we experienced a brief power outage in our Tampa, FL location that impacted all of our services. This outage occurred despite having A+B power setup in our cabinet but was quickly resolved by our data center resulting in less than 6 minutes of downtime for our cabinet although some VPSs were offline for up to 30 minutes while our nodes ran a disk check to prevent corruption or data loss. We also learned that an incorrect setting in the BIOS for fl1kvm01 prevented it from being powered back on after power was restored resulting in manual intervention to power it on, this will be addressed once the current IPv6 issue is also resolved.

Here is the information our data center provided us at the time:
During a routine power transfer test performed this morning, the UPS system failed to hold the customer power load, causing a brief power outage for our customers powered from the 8th floor infrastructure.

As a result of this, the UPS and generator vendors were contacted to perform emergency testing the following day to ensure the outage would not happen again. The testing involved the UPS system functionality, generator load banking, and transfer switch operation.

Here are the details provided by our data center as of today:
After an examination by the UPS manufacturer of the UPS logs generated on September 12, 2015, the manufacturer has determined the root cause of the UPS failure was an internal error in the static switch module.  A replacement module was immediately ordered upon notification of the manufacturer's determination.  On Sunday October 3, 2015 beginning at 9AM EST the manufacturer will be on site to perform the replacement. We expect the replacement to be complete before 12PM EST.

Another announcement will be generated regarding this planned maintenance.

Outage began: Sep 12 2015 07:36:20 GMT-7.0
Outage resolved: Sep 12 2015 07:39:16 GMT-7.0

-The Secure Dragon Staff

Wednesday, September 30, 2015





« Back