Pondering File Transfer Speeds 1

Posted by JD 01/26/2010 at 08:17

I move files around my network alot. Most of the time, these transfers are between wired GigE connected systems and are limited by disk performance, not the network. It is good and fast. Multi-gigabyte files transfer in seconds.

However, there are some tools that only work on Windows and my only Windows machine is a WiFi (G) connected laptop. Yes, I can wire it into the GigE network and see huge transfer speeds only limited by the laptop disk drive, but that is usually not how I do it. Yes, I use WiFi for convenience.

Why is 1 tool 2x faster than others? Why?

I’ve recently started using a VMware backup tool for Windows – Trilian VM-Explorer. It use ssh to connect to my ESXi 4 server and does some fancy stuff before safely transferring an entire virtual machine. As I watched the transfers, I noticed they were fast. 2+x faster than any other method I have. Blisteringly fast at almost 50% of the theoretical bandwidth for my WiFi network. I am impressed.

So, I fire up WinSCP and normal Linux scp in a VM and run them. Yep, they both run at 24% of the theoretical WiFi bandwidth. This begs the question

What is different about VM-Explorer scp transfers that make it sooooo fast?

Just to be clear, these test were performed on the same Win7 x64 machine with no changes to the IP stack settings. The machine placement didn’t change. Basically, everything is identical, except the program used to transfer files.

If I could, I’d use VM-Explorer to transfer files from my laptop to the servers, but it is tailored for transfers between VMware servers and Windows. Too bad.

I don’t know what is special about VM-Explorer transfers. Do you?

I can only guess these could be part of the answer.

  • Compression enabled
  • Larger buffers
  • VM-Explorer is optimized for VMware VMDK files and does other tricks based on this knowledge like a pre-built, pre-optimized LZH array.

I wish I knew.

Trackbacks

Use the following link to trackback from your own site:
https://blog.jdpfu.com/trackbacks?article_id=450

  1. JD 03/14/2010 at 10:04

    Since this article was written, I’ve built a new server and moved some OS installs around. There is now a Win7 x32 machine wired to the GigE network. Moving files from it to my Linux server over samba connections often gets 65% of the GigE bandwidth, about 650Mbps, thanks to large disk buffers on both sides. It is impressive to see that performance for the first 500MB of a file transfer.

    The VM with VM-Explorer has been unavailable for transfers since the laptop failed. I never did find an answer to what that program does differently to make transfers so much faster.