JDPFu.com 2024

Lots of Big WTV Recordings

With the Olympics on TV, I find myself dealing with lots of really large files – often between 25G and 35GB in size due to the 6 hour recording periods and 1080i resolutions. Moving those files around to remove commercials and sports that I don’t have any interest in takes time and CPU. Improving performance for both network and disk I/O on a KVM virtual server has made a huge difference in utilization.

The server is running Ubuntu x64 10.04 with KVM in the Dom0. The client VM, DomU, getting the improved performance is running Windows7 Ultimate x32.

Disk I/O

The solution summary is to use pre-allocated files (either raw or QCOW2) and avoid KVM disk caching. Don’t use Write-Thru caching EVER and avoid the system default cache setting. It is best to disable all caching and let the clientOS (DomU) handle it, not the hostOS (Dom0). I was already using pre-allocated files, but the default disk caching was selected and is less than ideal according to the source article.

Be aware that using the GUI to allocate QCOW2 doesn’t support pre-allocated virtual HDDs, or it didn’t when the source article was written. To be safe, drop to the CLI to do that and import the virtual HDD to use QCOW2. Be certain to use metadata pre-allocation with QCOW2 too. Make a 10G pre-allocated qcow2 virtual disk file:

$ qemu-img create -f qcow2 -o size=10000000000,preallocation=metadata VM-file.img

I stole that command directly from the source article.

If we use raw disks, pre-allocation works perfectly in the GUI.
If you use raw partitions and not image-based files, you’ll want to research the performance improving options further.

Disk Performance Differences

If you look at the source article, he has produced some simple but clear graphs showing how cache and pre-allocation parameters impact performance. I found the graphs compelling enough to make the changes immediately.

Before the change to no-caching, disk transfers from that VM were around 15MB/s.
After the change, disk transfers were either 22MB/s using Teracopy or about 52MB/s using a perl script. The files tested were 15G and 17G in size. The reported through-puts were sustained over the entire file copy processes.

I was already using VirtIO drivers for this Windows VM. Those show up inside Windows device manager as Red Hat VirtIO SCSI Disk Devices. Virtio is the default disk and network under KVM now for Linux clients, so no special installation instructions are needed.

Notice the 54% network utilization.

Network Devices

While you are in the VM settings, check that the Network Card for the VM is set to either a e1000 or a virtio card, not Hypervisor Default too. The default is only a 100base-tx card. This is slow, especially on a GigE network. Even if the real physical NIC is 100base-tx, setting the virtual network cards to faster devices removes that as a bottleneck, unless you want to specifically cause a network bottleneck.

Windows Vista and Windows7 have the Intel PRO/E1000 driver built-in. WindowsXP does not, so be certain you have a driver disk available (or have discovered it online) before changing the model in a WinXP DomU.

CPU Features

KVM is designed to support portable virtual machines to some extent. To achieve that goal, it avoids using CPU features that are vendor specific. Basically, it uses the least common denominator model. However, if the CPUs you will use for the VM all have a minimum standard feature set, you can significantly enhance for the performance for a single VM and the cooperation across multiple VMs running on the same host by allowing the VM to use the capabilities of the CPU. If you have a mixed CPU vendor environment, this is probably a bad idea, but if you have all Intel CPUs, even if they aren’t all identical, on the Processor Settings page inside virt-manager, select the oldest CPU running on your servers and press the Copy host CPU configuration button. A full VM shutdown and startup is needed for the change to occur. A restart is not sufficient.

As an example, we have Core2Duos and Core i5 systems on the network here. Selecting the Core2Duo as the Model for the CPU drastically improved performance and sharing of CPU resources across the VMs.

Be cautious when selecting the CPU features if you use live migrations between your KVM hosts. You could end up with either a non-booting VM or corrupted data.

Summary

Steps to the best possible disk and network I/O for a KVM/QEMU Virtual Machine:

1. Use VirtIO devices for KVM virtual machines whenever possible.
2. Use SATA disk devices if virtio isn’t available for the client OS.
3. Pre-allocate the storage used inside VMs.
4. Use e1000 network devices if virtio isn’t available for the client OS.
5. Disable disk caching for all virtual disk devices.
6. Set the CPU Processor settings to match the installed CPU in the VM server.

I didn’t test QCOW2 disks. I’ve preferred raw VM disks for many years. When using QCOW2 disks, pre-allocation needs to be performed manually, from a shell. This may not still be the case under recent releases.

I think these suggestions apply equally to other hypervisors.

A few other articles about improving virtualization performance.

• Make Windows Almost 50% Faster Under VirtualBox – provides more implementation details and actual times
• Faster Linux Software RAID Rebuilds
• KVM Tuning This is a MUST READ article.