While performing our routine maintenance and updating our software, we ran into an issue with installing the latest stable OpenVZ kernel. The problem was due to the ploop-lib library which is not used on CentOS 5 nodes but is still installed for some reason. According to OpenVZ.org, on October 16th they decided to remove ploop from the CentOS 5 package for new installs and in doing so, required us to manually remove it to continue our upgrades. Upon following the commands provided on their website, the load for the nodes shot up to over 50.00 and a reboot was required. Removal of ploop-lib also removed our OpenVZ service due to the reboot and replaced our vz.conf file with a default one. After re-installing OpenVZ, replacing the vz.conf with out previous one, and booting into the OpenVZ kernel we were back online with only a few lingering issues.
While trying to boot some VPSs manually, we were receiving some memory errors in regards to the virtual NIC (venet0) and the problem was the amount of IPv6 on the hardware nodes. We were forced to remove some unused IPv6 addresses and increase the kernel limit to get the remaining few VPSs to boot.
At this time all VPSs are online and functioning.
Only our OpenVZ nodes running CentOS 5 were impacted by this (fire01, fire02, and fire03).
Our Backup VPS node (gold01) has not been touched yet and we are waiting for another time to continue the maintenance.
Our new CentOS 6 node (fire04) is not impacted by this problem and was patched without issue.
Our Xen and KVM nodes were not impacted and were also patched without issue.
Our cPanel server (cpanel01) was also not impacted by this.
Our internal servers (den01 and den02) have not been touched yet and will be patched later this evening with the possibility of our Client Area and SolusVM Master going offline during a reboot.
Please open a ticket with out Billing department and we will issue a credit in accordance with our 99.9% uptime SLA. Thank you.
-The Secure Dragon Staff
Wednesday, November 7, 2012
