After much painful testing and even more painful fixing and re-testing, we have finally made some headway and can see the light at the end of the tunnel. Here are some updates to our current Infrastructure Upgrade project:
- GOLD0001 was rebuilt with hardware RAID10 which worked very nicely for 24 hours... until the hardware RAID10 stopped working. Luckily the server was still in the testing phase and no clients were affected. We have been working for the past 24 hours in attempts to get the hard drives to either work with the onboard fakeRAID or software RAID but with no luck. Current write speeds for a 6 drive RAID10 configuration are 30MB/s which is not acceptable so we are exploring other options. We apologize to those clients who were on GOLD0001 and are being impacted by the performance decrease on the temporary holding server.
- We rebuilt the hardware firewall with newer hardware only to find the software is not compatible with the server's video card (OUCH!). We have been exploring other avenues for this but for the time being we are putting the hardware firewall on the back burner while we focus on other things. As it stands, we have been able to successfully combat the DoS attacks using a software firewall that has been working wonderfully which is why we reduced the priority of the hardware firewall.
- FIRE0003(NEW) was built today after the last of the clients were migrated off of FIRE0003 and FIRE0004 (we used the existing hardware to build a much more powerful server). FIRE0003(NEW) is currently running a hardware RAID10 configuration with multiple SAS drives and more CPU power than FIRE0003 and FIRE0004 combined. The results speak for themselves... 176MB/s write speeds! In comparison, FIRE0003 was only able to obtain 75MB/s speeds so this is a huge improvement that our clients will greatly benefit from. FIRE0003(NEW) is also running CentOS 6.0 and takes advantage of the new memory management so all clients on the new server will be assigned vSwap instead of Burstable RAM.
- FIRE0001 and FIRE0002 are currently experiencing higher than normal loads due to all clients on FIRE0003 and FIRE0004 being migrated to them. Luckily because we greatly undersell our VPS nodes, there is no fear that the nodes will become unstable with the added VPSs. Once FIRE0003(NEW) has been thoroughly tested and has survived it's stress test, we will begin migrating clients onto that node. We have not determined how we will decide which clients will be moved and which ones will stay on FIRE0001 and FIRE0002 at this time so please do not open a ticket requesting to be moved.
In addition to these changes listed, we have been making MANY MANY MANY behind-the-scenes changes and improvements which, while not visible to you directly, have impacted you in someway or another... in a good way of course. We've been working on improving our current VPS plans and once we are done with GOLD0001 and FIRE0003(NEW), we have 3 servers available to us just for testing which will provide us plenty of resources for development (a little birdy keeps saying something about KVM and a new control panel in the future...). Once FIRE0003(NEW) is settled and proves to us and our clients that it was the right choice, expect more of such servers to be arriving as we slowly phase out our current Blade servers.
Lastly, I want to use this opportunity to communicate to you about communication. I (Joe) am personally very sorry for the lack of updates. I made attempts to provide them as frequently as possible but in all honestly I didn't feel it was appropriate to update Twitter with every roadblock we hit and up until a few hours ago, there was nothing of value to report. In my downtime (i.e. while waiting for a 16TB array to build) I have been working on a website specifically to increase communication. While it's not nearly complete, I feel it is at a point where I can share it with everybody: http://drgn.biz/index.php (without the index.php you're just going to get an empty page since it's under construction still). We are also looking into a newsletter/mailing service that will help us improve communication since our current mailservers are constantly limiting our mass e-mails which prevents our clients from receiving the critical updates (which is why I try to focus on Announcements to make sure everybody has access to the news and to keep everything transparent). I will try to utilize Twitter more as a communication tool also.
Thank you for all of you understanding and support. It's been a wild couple of months, but I'm sure we all like some excitement in our lives... stable excitement of course. :)
-The Secure Dragon Staff
Lastly, I want to use this opportunity to communicate to you about communication. I (Joe) am personally very sorry for the lack of updates. I made attempts to provide them as frequently as possible but in all honestly I didn't feel it was appropriate to update Twitter with every roadblock we hit and up until a few hours ago, there was nothing of value to report. In my downtime (i.e. while waiting for a 16TB array to build) I have been working on a website specifically to increase communication. While it's not nearly complete, I feel it is at a point where I can share it with everybody: http://drgn.biz/index.php (without the index.php you're just going to get an empty page since it's under construction still). We are also looking into a newsletter/mailing service that will help us improve communication since our current mailservers are constantly limiting our mass e-mails which prevents our clients from receiving the critical updates (which is why I try to focus on Announcements to make sure everybody has access to the news and to keep everything transparent). I will try to utilize Twitter more as a communication tool also.
Thank you for all of you understanding and support. It's been a wild couple of months, but I'm sure we all like some excitement in our lives... stable excitement of course. :)
-The Secure Dragon Staff
Saturday, November 12, 2011
Powered by WHMCompleteSolution
