File Copy Performance for Large Files 3

Posted by JD 10/27/2010 at 18:36

The last few days, I’ve been trying to improve the manner that I copy large (2+GB) files around both locally and between systems.

I looked at 4 different programs and captured very simple stats using the time command. The programs tested were:

  1. cp
  2. scp
  3. rsync
  4. bigsync

I’d considered trying other programs like BigSync, but really wanted something at supported incremental backups to the same file and handled it without too much complexity. I would have liked to use zsync, but my understanding is that is an HTTP protocol and can’t be used for local copies. I wasn’t interested in setting up apache with SSL between internal servers.

Diagram of Linux Distro History

Posted by JD 10/26/2010 at 07:04

A picture is worth …. 1,000,000 words in this case.
This link shows the way that different distributions are related, started, and some died. It only shows the most popular distros, perhaps 300 of them. I didn’t count.

DC404 Presentation-SysUsage in 5 Minutes of Effort

Posted by JD 10/15/2010 at 07:43

This Saturday (tomorrow) I’ll be presenting to the DC404 crowd how to get system monitoring working on Linux in 5 minutes. Sadly, the presentation is running about 30 minutes because I really only need 5 minutes to show a complete install. Come and check it out, say hello.

There isn’t really much to it because SysUsage is really easy to setup and run. It works on desktops, laptops and servers, but there is no GUI for setup so you will need to use a terminal or console. With newer releases (OS or SysUsage) the dependencies may change, but it is still really easy to setup. This is so simple there really isn’t any excuse NOT to have performance graphs for all your Linux machines.
SysUsage Thumbnail

VMware Hiding the vSphere Client

Posted by JD 09/02/2010 at 19:59

Where to download the latest vSphere Client?

I just spent the last 90 minutes hunting the vmware.com web site for the latest x64 Windows7 vSphere client. VMware has lots of things you can download prominently displayed … just not the latest vsphere client.

It still bothers me that I’m required to run MS-Windows to use the mandatory client.

In the end, I had to select a new download of ESXi, agree to give my first born to them, sign a contract, then they showed a list of about 5 more downloads – including the damn vsphere client that I wanted. While I was there, I grabbed the 4.0 to 4.1 ESXi upgrade package.

For the last 2 years, ESXi has been rock solid … until about 10 days ago. That’s when a 32-bit Linux VPN server stopped responding. The other machines are available and working, just that one is not. Got an error trying to use the v4.0 vsphere client to connect.

The error that the old vsphere client, v4.x, displayed was error parsing clients.xml This error happened on both x64 and x32 machines running either Win7 or WinXP. Basically, only the unsupported ssh login was available. According to the google results, a new vsphere client will fix everything. I hope it does.

I can’t wait for KVM to be as good.

Missing URL Attacks and Counter Measures 1

Posted by JD 08/25/2010 at 05:49

Yesterday, I took a look at the missing URL statistics for my blog and saw lots of attacks to phpMyAdmin. That’s sorta funny since I don’t use that tool and if I did, it wouldn’t be available publicly. When I say lots of phpMyAdmin attacks, I mean hundreds with every possible way of spelling it and every possible version listed. Certainly, these are scripts, but still they are many hundreds of attempts. My little blog isn’t THAT popular and it is non-commercial. You don’t need an ad-blocker here and most of the website works just fine without javascript enabled.

Anyway, I’ve added a reverse proxy filter to my web cluster that will prevent undesired requests. I’ve always had a reverse proxy, but never bothered to filter anything except PUSH size to this point. If something seems to be broken, please let me know, since I could have missed a few needed allow rules.

Between the blog spammers and random attacks, it sorta makes me wonder how many counter measures the average blogger installs and uses to pro-actively protect their work. Do you review your logs?

I did see that someone had a sense of humor in their attacks

/oh.hey.there.how.you.doin
was seen as a missing URL request in the log files.

Here’s what I do to protect my websites
  • DB – Limit any DB access to local, no network access and especially no internet access to DBs
  • Firewall – Limit the ports which can get inside to the reverse web proxy, not to the actual ports where the applications are running.
  • Remote – remote access from outside the network is only via key-based ssh or over VPN
  • Use non-standard software that doesn’t have millions of users – low value target
  • Limit plugins to simple ones so I can review the code
  • Stay patched – weekly patching happens on all systems involved
  • Stay up to date with released software that is not patched
  • Review the server logs daily using summary tools like log watch
  • Generally keep up with security issues for the platform and software installed
  • Backups – incremental, daily, automatic, full. I can recover any file or the entire server as needed for any day from today to 30 days ago.
  • Virtualization used for server platforms. If I do have a hardware failure, the VM can be brought back up on almost any other hardware, provided the VM host/Dom0 runs.
  • Run 1 main service per VM. The blog servers don’t do email or vpn or monitoring or …. – they are each single purpose machines. The VM hosts are not available to the outside world.
  • Avoid Microsoft solutions for servers – yes, this is a security feature to me
  • Block full search engine access
  • Perform local analytics – no outside service is used – this is for your privacy too

Anyway, I hope I wasn’t too restrictive on the verbs allowed to pass into this site. Again, if you see issues, please let me know. The script kiddies will need to find another site to bother and someone else’s bandwidth to waste.

What do you do to protect your websites from hackers? Tell me in the comments.

Simple Disk Encryption for Laptops 2

Posted by JD 08/21/2010 at 11:28

When you have a laptop, you expect to take it with you. When you have a portable computer, the data on it is at higher than normal risk for theft. This means you need to take steps to protect that data. In 2010, the Commonwealth of Massachusetts passed laws that require non-public consumer data be encrypted on disk and networks. Basically, disk encryption is here to stay, but there are some drawbacks. Below I’ll describe why you want to encrypt much of your laptop disk storage, why you probably don’t want whole disk encryption, and the setup that I’m using.

Windows7 Disk Shrinking Drama 1

Posted by JD 08/19/2010 at 16:38

Just got a new laptop yesterday, but I didn’t like the way that the 500GB HDD is partitioned.

  • Unknown (50MB)
  • RECOVERY (18GB)
  • WIN7 (whatever remained of the 500GB – over 400+GB)

Only a noob would want that partitioning because they simply don’t know any better. Why do manufacturers still do this? Ah – because it is easy. Sorry for the dumb question.

I want:

  • Unknown (50MB)
  • RECOVERY (10GB – it actually only uses less than 8GB – resize)
  • WIN7 (60GB for apps and the OS – resize + move)
  • Data (size determined by my backup disk … I’m guessing about 250GB) Much of it will be encrypted, since this is where my VMs are placed.
  • Scratch area for local disk backups or snapshots of VMs or …. unimportant temp data.

Ah, if it were only that simple. Below I’ll discuss the methods used to shrink / resize disk partitions and re-align the data so reduce wasted areas on the disk.

VirtualBox on FLOSS Weekly

Posted by JD 08/16/2010 at 15:00

The FLOSS Weekly podcast did an entire episode on VirtualBox recently. There were some interesting comments in the podcast from the VBox team.

The team believes they are the closest to native for both network and storage virtualization. They recommended a number of specific NICs and how to connect to storage that does not disagree with my Improve VirtualBox Performance by almost 50% article. That was good to hear.

They talked about:

  1. Oracle’s commitment to VirtualBox
  2. Virtualization Overview (why, features, security, virtualized hardware, etc)
    1. Most interesting to me was that VBox supports OpenGL 2.x – that means XBMC should run in a vbox VM.
  3. Differences between the OSE and PEUL licenses – what is contained in just the PEUL?
  4. vboxmanage and how 3rd party folks have written some nice tools
  5. Teleportation (V-Motion like) that does not require identical
  6. Page-Fusion – shared memory across guest OSes (this was new to me). The intent is to support hundreds of client VMs per server.
  7. Mac virtualization challenges – it works, but only on Mac hardware due to Apple DRM in hardware checks.

Anyway, find the VirtualBox specific podcast here.
A link to the show notes

By the way, if you are a dentist, FLOSS Weekly has nothing to do with teeth. FLOSS is *F*ree *L*ibre *O*pen *S*ource *S*oftware.

11 Windows Software Programs to Avoid 4

Posted by JD 08/10/2010 at 10:15

Over the years, we all come across software that we decide to uninstall and never use again (unless forced to do so by a company). I expect that some software on my hate it list is loved by someone. Most of those tools that make it to this list became bloated, lost their core purpose or did something nasty like installed a search bar or some other spyware.

Anyway, here’s my list for Windows.

Why I Use a Linux Desktop 2

Posted by JD 07/29/2010 at 07:51

Each of us use a computer for various reasons. Some just want a system that works, without any hassle. Most of us want to run specific software, work with specific file types, connect with everyone else, and possibly just do what the people around us are doing to be the same.

I’ll be as honest as I can and put the reasons in order.