JDPFu.com 2025

For network copies, some form of SSL encryption was used. That could be ssh tunnels, rsync of ssh, sshfs or simply scp. The OpenSSL libraries were not patched for higher performance.

Performance Testing is Hard

The results appear to be all over the place. This means I have at least 1 major flaw in my testing methods. I didn’t reboot the systems between test scenarios. I didn’t flush the cache, nor did I stop all other programs running. Basically, the systems involved were doing what they do all the time … running. I did avoid high CPU, IO and bandwidth tasks and I performed the tests during off-peak periods.

Anyway, here are the results.

Results

14 different test runs were performed. To understand the details, both the order of the tests and the program used need to be considered.

The table below provides more information about each scenario. They are in order. The left group of copies were performed within a single machine across file systems on different disks. The second group of copies were over a GigE network which routinely sees 650Mbps throughput. The disk IO subsystems and CPU required due to the program used were more important. Also, the use of the FUSE sshfs could have impacted performance.

To see the image full-size, right-click and choose View Image.

A table showing the details for each scenario is available.

Network Throughput Chart
Below is a graph that shows the bandwidth for each scenario. As expected, disk throughput was the limiting factor, not network throughput. Disks are faster than networks for small files, not big ones where the disk performance is king. About 70Mbps is the throughput seen with cp, scp, and rsync for the first copy (before any differencing can be performed). As expected, higher perceived throughput is achieved with differential copies, but not as high as I would have thought. bigsync provided the highest effective throughput with the 3rd run at about 580 Mbps.

bigsync runs showed some interesting trends. The first run, before the target file exists, is slightly slower than cp or scp. The 2nd run was at least 100% faster and sometimes almost 500% faster. The 3rd run was another 50% faster than the 2nd run, but that was almost certainly due to disk caches.

The bigsync process leaves a file in the target directory which contains SHA-1 signatures for each block. Also, if the target arguement does not specify a filename, then the source file name is not utilized. This seems like a poor design choice. The C code is simple enough to make that modification an easy change.

The use of FUSE sshfs probably slowed things down slightly, but as network performance didn’t seem the be the bottle neck anywhere in these tests as shown by internal system disk-to-disk copy performance, I don’t plan to perform any additional network tuning.

What Does This Mean?

The data doesn’t show any trends and definitely didn’t show what I’d expected. This means the tests were flawed … or that it doesn’t really matter which of the methods are used.

However, the data did confirm a few things that I already knew.

• rsync isn’t efficient with large files, we are better off copying the files without rsync
• I expected the first bigsync to be similar to a copy or scp in time. It was.
• I expected non-first bigsyncs to be much quicker, especially for an unchanged file. The bigsync code is not very complex.
• I learned that bigsync had issues accessing VirtualBox shared file systems from a hostOS. It never worked. When I reviewed the code, I couldn’t see any complex file system calls and have concluded it is a bug in VirtualBox based on other strange behaviors and poor performance that I’ve seen VirtualBox host-shared files display.

I’m still looking for a solution and I need to work on my test procedures.