JDPFu.com 2025

Update May 2021

dpkg -l 'linux*' | awk '/^rc/{print $2}' | xargs sudo apt purge -y

Original Article

If you run Ubuntu Server LTS releases like I do, you are probably wasting disk space and, because of that, backup storage because old files tied to old kernels aren’t cleaned up automatically.

After locating and cleaning up the old kernel files, I regained over 1GB of space. I was getting low on storage on / otherwise I wouldn’t have bothered.

First Method – Package Manager

When you use a package manager for installations, you want to use it for removals too.
First, I want a list of installed kernel specific packages

sudo dpkg-query -l | egrep -i ‘2.6.17|2.6.2[0-3]’

Ouch. That’s a big list with many old, unused packages still installed. If you have Synaptic installed, using the search in that tool will let you easily multi-select packages for removal. Without X/Windows, you’ll be at the command line with me. Time to start removing them with cmds like this.

sudo apt-get remove linux-source-2.6.17 vmware-player-kernel-modules-2.6.17-10 xen-image-xen0-2.6.17-6-generic-xen0 linux-image-2.6.20-17-generic linux-restricted-modules-2.6.20-17-generic

Package managers remove libraries and programs, but avoid removing configuration files, since the next version of a tool probably needs them. If you are truly removing an application, you’ll probably want to purge the install to remove the conf files too.

Second Method – Find and Locate

I use locate and updatedb. I can’t imagine running servers without these tools. To start, I wanted a list of locations to look through and determine how much waste I had. On the server, we are using kernel 2.6.24-26-generic today. To find almost all the files, use

locate  2.6.24 | egrep -vi /backup > /tmp/old-kernel-files

These commands return a list of files, remove backups from that list and put the list into a /tmp file for reference.

On my Ubuntu system, files were located in:

/boot

/lib/modules

/lib/firmware

/linux-restricted-modules/

/usr/src

I had about 10 old kernels lying around beginning with 2.6.17. Because I wanted to be very careful with removing these files, I manually typed the cleanup commands in each directory. A few examples:

sudo rm -rf 2.6.17-*

sudo rm -rf 2.6.24-2[3-4]*

sudo rm -rf linux-headers-2.6.17-1*

sudo rm linux

sudo rm -rf linux-source-2.6.17 orig-linux-headers-2.6.17-10*

Definitely be careful. It is easy to remove the wrong files by accident with a bad pattern.

Good luck getting that space back!

So, I found a few free minutes today and decided to upgrade this blog to the latest release. This was a security related update, seems Typo had a few security vulnerabilities. As usual, things mostly went fine following the instructions provided by the upgrade web page. Mostly.

Overview of the Steps

Sadly, the answer to which virtualization is best for Linux isn’t an easy one to answer. There are many different factors that go into the answer. While I cannot answer the question, since your needs and mine are different, I can provide a little background on what I chose and why. We won’t discuss why you should be running virtualization or which specific OSes to run. You already know why.

Key things that go into my answer

1. I’m not new to UNIX. I’ve been using UNIX since 1992.
2. I don’t need a GUI. Actually, I don’t want a GUI and the overhead that it demands.
3. I would prefer to pay for support, when I need it, but not be forced to pay to do things we all need to accomplish – backups for example.
4. My client OSes won’t be Windows. They will probably be the same OS as the hypervisor hosting them. There are some efficiencies in doing this like reduced virtualization overhead.
5. I try to avoid Microsoft solutions. They often come with additional requirements that, in turn, come with more requirements. Soon, you’re running MS-ActiveDirectory, MS-Sharepoint, MS-SQL, and lots of MS-Windows Servers. With that come the MS-CALs. No thanks.
6. We’re running servers, not desktops. Virtualization for desktops implies some other needs (sound, graphics acceleration, USB).
7. Finally, we’ll be using Intel Core 2 Duo or better CPUs. They will have VT-x support enabled and 8GB+ of RAM. AMD makes fine CPUs too, but during our recent upgrade cycle, Intel had the better price/performance ratio.

Major Virtualization Choices

1. VMware ESXi 4 (don’t bother with 3.x at this point)
2. Sun VirtualBox
3. KVM as provided by RedHat or Ubuntu
4. Xen as provided by Ubuntu

I currently run all of these except KVM, so I think I can say which I prefer and which is proven.

ESXi 4.x

I run this on a test server just to gain knowledge. I’ve considered becoming VMware Certified and may still get certified, which is really odd. I don’t believe many mainstream certifications mean much, except CISSP, VMware, Oracle DBA and Cisco. I dislike that VMware has disabled things that used to work in prior versions to encourage full ESX deployments over the free ESXi. Backups at the hypervisor level, for example. I’ve been using some version of VMware for about 5 years.

A negative, VMware can be picky about which hardware it will support. Always check the approved hardware list. Almost every desktop motherboard will not have a supported network card and may not like the disk controller, so spending another $30-$200 on networking will be necessary.

ESXi is rock solid. No crashes, ever. There are many very large customers running thousands of VMware ESX server hosts.

Sun VirtualBox

I run this on my laptop because it is the easiest hypervisor to use. Also, since this works on desktops, it includes USB pass thru capabilities. That’s a good thing, except, it is also the least stable hypervisor that I use. That system locks up about once a month for no apparent reason. That is unacceptable for a server under any conditions. The host OS is Windows7 x64, so that could be the stability issue. I do not play on this Windows7 machine. The host OS is almost exclusively used as a platform for running VirtualBox and very little else.

Until VirtualBox gains stability, it isn’t suitable for use on servers, IMHO.

Xen (Ubuntu patches)

I run this on 2 servers each running about 6 client Linux systems. During system updates, another 6 systems can be spawned as part of the backout plan or for testing new versions of stuff. I built the systems over the last few years using carefully selected name brand parts. I don’t use HVM mode, so each VM runs with 97% of native hardware performance by running the same kernel.

There are downsides to Xen.

1. Whenever the Xen kernel gets updated, this is a big deal, requiring the hypervisor be rebooted. In fact, I’ve had to reboot the hypervisor 3 times after a single kernel update before it takes in all the clients. Now I plan for that.
2. Kernel modules have to be manually copied into each VM, which isn’t a big deal, but does have to be done.
3. I don’t use a GUI, that’s my preference. If you aren’t experienced with UNIX, you’ll want to find a GUI to help create, configure and manage Xen infrastructure. I have a few scripts – vm_create, kernel_update, and lots of chained backup scripts to get the work done.
4. You’ll need to roll your own backup method. There are many, many, many, many options. If you’re having trouble determining which hypervisor to use, you don’t have a chance to determine the best backup method. I’ve discussed backup options extensively on this blog.
5. No USB pass thru, that I’m aware. Do you know something different?

I’ve only had 1 crash after a kernel update with Xen and that was over 8 months ago. I can’t rule out cockpit error.
Xen is what Amazon EC2 uses. They have millions of VMs. Now, that’s what I call scalability. This knowledge weighed heavily on my decision.

KVM

I don’t know much about KVM. I do know that both RedHat and Ubuntu are migrating to KVM as the default virtualization hypervisor in their servers since the KVM code was added to the Linux kernel. Conanacal’s 10.04 LTS release will also include an API 100% compatible with Amazon’s EC2 API, binary compatible VM images, and VM cluster management. If I were deploying new servers today, I’d at least try the beta 9.10 Server and these capabilities. Since we run production servers on Xen, until KVM and the specific version of Ubuntu required are supported by those apps, I don’t see us migrating.

Did I miss any important concerns?

It is unlikely that your key things match mine. Let me know in the comments.

The script below was created as part of an Alfresco upgrade process and meant to be run manually. This is fairly trivial cold backup script for Alfresco 2.9b, which is a dead release tree from our friends at Alfresco. It hasn’t been tested with any other version and only backs up locally, but could easily backup remote with sshfs or nfs mounts or even rdiff-backup commands swapped in.

For nightly backup of our production servers, we actually perform rdiff-backups of shutdown virtual machines, which take about 3 minutes each. That little amount of downtime to have a differential backup of the entire VM is worth it to us.

#!/bin/sh
# ###############################################################
# This script should not be run from cron. It will wait for the mysql
# DB password to be entered.
# 
#  Created by JDPFU 10/2009
# 
# ###############################################################
# Alfresco Backup Script - tested with Alfresco v2.9b
#   Gets the following files
#    - alf_data/
#    - Alfresco MySQL DB
#    - Alf - Extensions
#    - Alf - Global Settings
# ###############################################################
export TOP_DIR=/opt/Alfresco2.9b
DB_NAME=alfresco_2010_8392
export EXT_DIR=$TOP_DIR/tomcat/shared/classes/alfresco/extension
export BACK_DIR=/backup/ALFRESCO
export BACKX_DIR=$BACK_DIR/extension

# Shutdown Alfresco
/etc/init.d/alfresco.sh stop

# Backup the DB and important files.
# dir.root setting will change in the next version
/usr/bin/mkdir  -p $BACK_DIR
cd  $BACK_DIR/; 
/usr/bin/rsync  -vv -u -a --delete --recursive --stats --progress $TOP_DIR/alf_data $BACK_DIR/

echo "
  Reading root MySQL password from file
"
/usr/bin/mysqldump -u root \
    -p`cat ~root/bin/$DB_NAME.passwd.root` $DB_NAME | \
    /bin/gzip > $BACK_DIR/${DB_NAME}_`date +%Y%m%d`.gz
/usr/bin/find  $BACK_DIR -type f -name "$DB_NAME"/* -atime 60 -delete

/usr/bin/cp  $TOP_DIR/*sh $BACK_DIR
/usr/bin/mkdir  -p $BACKX_DIR
/usr/bin/rsync  -vv -u -a --delete --recursive --stats --progress  $EXT_DIR/* $BACKX_DIR/

# Start Alfresco
/etc/init.d/alfresco.sh start

Why a cold backup? Unless you have a really large DB, being down a few minutes isn’t really a big deal. If you can’t afford to be down, you would already be mirroring databases and automatically fail over anyway. Right?

We use a few extensions for Alfresco, that’s why we bother with the extensions/ directory.
There are many ways to make this script better. It was meant as a trivial example or starting point to show simple scripting methods while still being useful.

Last week, I setup and configured a special desktop for the accounting system for the company. Basically, it is a Windows7 Pro desktop running under ESXi 4 that the folks responsible for accounting remote (RDP) into after connecting via VPN to the special network for it. We’re small and only a few people even need access – never more than 1 at a time.

It was fairly painless to setup, install Accounting, load Payroll CD, then validate remote VPN access (which is never trivial), then setup daily backup jobs. Of course, AV, automatic patchs and nasty IE settings were configured too. Each daily backup set is about 250MB, which isn’t too bad, but more than I would have thought given the machine is idle most of the time and won’t be used more than 3 days a month. These backups are Microsoft VHD files using the built-in backup, which could be useful, but I’d rather have a complete VDMK, VDI, or Xen img file to restore.

Of course, it isn’t possible to connect to this VM without going through our VPN.

Next I need to perform a test restore to another machine under some virtualization tool that we use. Yeah, I know with the VHD, I can perform a restore someplace else, but with the VM-image file, I just point a hypervisor at it and go. Now that VirtualBox supports VMware, vdmk, files, this test really should be trivial. If it goes well, I’ll take my WinXP (MS-Office, Visio and other WinXP-only tools VM) and put it under a server-based VM too. It will be better to not travel with that stuff on my laptop anyway.

I came across an old article that I wrote on backups that had some clock times for the different VMs. Since that article was written, I’ve changed the backup methodology from rsync to rdiff-backup.

dms44 → 1m:52s   Alfresco

crm46 → 3m:36s    vTiger

xen41 → 3m:10s    Typo

pki42 →  1m:17s

mon45 → 1m:8s

zcs43 →  3m:53s   Zimbra

Those are real “downtime” numbers to ensure completely safe backups were made with all files closed. Actually, the virutal machine is shutdown during the backup periods. Email is unavailable for 4 minutes at around 2am daily. We can live with that. Recovery works perfectly too. I’ve recovered the largest VM twice in under 20 minutes after some cockpit errors.

This works because we use Xen virtual machines and rdiff-backup. Most of the VMs are 20GB in disk size, but use less actual storage.

As I ponder how to build a redundant file server that serves Linux, Solaris, VMware, Xen, VirtualBox, FreeBSD, FreeNAS, TiVo and Windows systems, a few interesting articles have come to light.

• A home file server using ZFS
• Snapshots into the cloud
• Backups with versioning
• 10 Reasons You need to Consider ZFS

Requirements

Basically, I’d like

1. reasonable amounts of redundancy
2. hardware agnostic
3. FOSS (non-commercial)
4. Enterprise ready – support for iSCSI, CIFS, Samba, NFSv4, RAID levels, snapshots, and versioning
5. remote backup capabilities – rdiff-backup would be ideal
6. Offsite backup capabilities – any type of external storage “in the cloud”
7. Encryption of offsite backups
8. high performance capabilities
9. Suitable for file system, database and raw disk device access

More on this as I work through the solution over the next few days and weeks.

BTRFS

Of course, I came across this article on btrfs a few days later explaining the it will likely be the default Linux file system in a few years. It also explains that any file systems created prior to kernel 2.6.30 are incompatible and with later kernels. Today, I’m running 2.6.24-24-generic SMP. No go.