New Blog Software and OS 2
Since this is a technology blog, I figure some of you may be interested in a major change that happened out of necessity here today.
This is the very first blog article on our new physical server, running in a completely different virtual machine. For the next week, everything here is a test.
Due to some sort of outage issue earlier today, I was forced to upgrade everything involved with this blog. I had attempted to perform this upgrade previously and failed. As you can see, this time, there was success. Nobody was shocked more than I.
The Changes
|
Xen paravirtual VM |
KVM HMV VM |
|---|---|
| Ubuntu Server 8.04 LTS | Ubuntu Server 10.04 LTS |
| Ruby 1.8.6 | Ruby 1.8.7 |
| Rails 2.x.x | Rails 3.1.0 |
| Mongrel 1.1.5 | uh, still need a cluster manager. Mongrel didn't work and thin is failing too. |
| nginx 1.0.4 | nginx 0.7.x (yes, I downgraded) |
That's a bunch of changes for something thrown together due to an outage. Something bad happened earlier today that prevented the Typo blog app to startup properly. Normally that has been due to either database corruption or having too many session records left in the database. Restoring from a backup DB didn't help this time. What is really confusing to me was that I'd used the blog ealier today without issue, so the backed up DB was fine. I was unable to get even bring the system up without crashing the VM it ran inside. So far the new VM feels faster even if it has fewer back end servers.
Other Updates
Whenever the blog software is updated, that forces other software updates for monitoring, backups, web analytics, etc. I have successfully migrated the web analytics over to the new machine. We're running the same system performance statistics tools and I'll probably end up with a different approach to VM backups under KVM than we had under Xen. Having this VM running under KVM gives me a reason to solve this issue. I'm just happy we didn't loose our 365 days of analytic data too. We did switch virtual servers AND physical servers after all.
Xen is on the way out.
I still have a number of VMs running under Xen VMs on the other machine. We've had a plan to migrate away from Xen for about a year, but never were forced. This change today has forced that issue. We won't be going back.
As long as you have a backup of your daabstae which get’s updated more frequently you are good in case of a crash. Images can be backed up manually once in a while, you should have a copy of the images on your local disk where you uploaded from. Chances are slim of having a blog crash and your computer simultaneously. Make sure you also backup your local disk / development area.
@Nini: Our backup methods are a little more stringent here. They are daily, automatic, and pushed to different physical hardware. We retain at least 30 days, but usually 60 days of backups for this system. That includes the OS, apps, blog, logs, performance data, blog DB and static web files. All are very well backed up. I take the blog software down during the backup window every day to ensure ZERO database corruption.
Last night’s blog server backup took …
The service interruption was probably only 45 seconds.
Typo has been more stable recently, but it still does run into a too many open files issue every few weeks which locks 1 instance of the typo app up. A reboot is usually needed to fix it. My searches point that bug in the directions of Rails.
Many of the blog software elements have been updated to current levels since this article was written. For example, nginx is running the latest stable version always now – something in the 1.2.x range, I believe. Thin is working well too.