We are issuing an emergency reboot of both of our Xen PV nodes (shadow01 and shadow02) after being notified of a recently discovered critical Xen PV exploit that could result in the comprimise of our hardware nodes.
More information about the exploit can be found here: http://lists.xen.org/archives/html/xen-announce/2012-06/msg00001.html
Update 1: I am on-site at the DC now trying to resolve the issues with shadow01 and shadow02. At this time, failing back to an older kernel is not an option as it would put all of our clients at risk along with all of the data on our nodes.
Update 2: The problem appears to be with the Xen bridge. We are still working to get this resolved. No ETA at this time.
Update 3: See latest announcement "Xen PV Outage" for more details.
-The Secure Dragon Staff
Tuesday, June 12, 2012