JDPFu.com 2025

As I ponder how to build a redundant file server that serves Linux, Solaris, VMware, Xen, VirtualBox, FreeBSD, FreeNAS, TiVo and Windows systems, a few interesting articles have come to light.

• A home file server using ZFS
• Snapshots into the cloud
• Backups with versioning
• 10 Reasons You need to Consider ZFS

Requirements

Basically, I’d like

1. reasonable amounts of redundancy
2. hardware agnostic
3. FOSS (non-commercial)
4. Enterprise ready – support for iSCSI, CIFS, Samba, NFSv4, RAID levels, snapshots, and versioning
5. remote backup capabilities – rdiff-backup would be ideal
6. Offsite backup capabilities – any type of external storage “in the cloud”
7. Encryption of offsite backups
8. high performance capabilities
9. Suitable for file system, database and raw disk device access

More on this as I work through the solution over the next few days and weeks.

BTRFS

Of course, I came across this article on btrfs a few days later explaining the it will likely be the default Linux file system in a few years. It also explains that any file systems created prior to kernel 2.6.30 are incompatible and with later kernels. Today, I’m running 2.6.24-24-generic SMP. No go.

Software Usability is an art form. It isn’t easy, but here’s a suggestion for all the product managers out there.

Give the software to your Mother. Can she use it? Can she figure it out with 10 minutes of 1-on-1 training?

If she can’t then, you’ve failed. Try again.

If there is a trick, then you’ve failed too. Apple likes their tricks. The first time I used an ipod, I couldn’t turn it off or change tracks because they were too lazy to properly mark the buttons.
The first time I used a Mac after college, I needed to format a floppy disk and couldn’t figure out how. The menus didn’t have “format” on them until the disk was inserted.

rdiff-backup rocks, mostly. But there are times when it doesn’t work as expected or doesn’t work at all. Usually, the not working at all part is a cockpit error, but sometimes not.

Key rdiff-backup features

1. Simple 1 line backup command; rdiff-backup source target
2. Reverse Incremental backup sets
3. Extremely FAST backups. Entire server installations are just a few minutes, after the initial backup set is created.
4. Last backup set is available as a complete copy of the files. Need to recover? Just copy the file(s) back.
5. Control over how old backup sets can be. Deletion of “older than x days” sets is trivial.
6. Compressed older differential backup sets
7. Current backup is 1-for-1 sized. Older backups are tiny. As an example, a 5GB backup with hundreds of files with 30 days of incremental backups only takes 6GB total. Each daily backup is relatively tiny and based on changes made that day. Usually those changes are just a 10-40MB. Impressive.
8. Recovery by date/time
9. FOSS – we like Free and Open Source Software
10. Cross platform – Unix, Linux, MacOS and MS-Windows.

Things that just work

1. Linux local rdiff-backup runs, just work. Backing up a directory structure or an entire VM (not as a single huge file) to another mounted disk works very nicely with all the key features listed above.
2. Win32 local rdiff-backup runs, provided there isn’t any networking involved nor huge files.
3. Recovery of an entire VM fileset. I’ve needed a few recoveries the last 6 months due to user error. They worked flawlessly and only took 20 minutes from problem discovery to full recovery. That was manual recovery. If this were scripted, it could be less than 5 minutes.

Things that don’t work or work poorly

This is mostly on MS-Windows platforms, but some Linux stuff doesn’t work nicely either. Windows howto that wasn’t really much use.

1. remote transmission over ssh on a non-standard port is broken regardless of platform. That doesn’t mean it can’t work, but I’ve never been successful in getting it to work. Neither push, nor pull command versions have worked. It shouldn’t be this hard.
2. Large file differencing doesn’t seem to work on Linux or Windows. In theory, that means 4GB files, but smaller files get confused and end up as a complete copy too, not a block level differential copy.
3. MS-Windows network backups don’t really work, even over samba connections. Ok, there are many strange things about rdiff-backup on Windows. For example:
   1. You have to `cd` to the drive and directory of the source if you want it to work.
   2. You have to use ‘/’ instead of ‘\’ characters, most of the time. This is a python thing, I guess.
   3. Backups to samba shares may or may not work. I haven’t figured the reason why or why not yet.
   4. Backups over ssh require less than trivial setup. Only push will work from windows unless an ssh server is setup. Then the complexity is exponentially more difficult.
   5. Many people use cygwin with all those faults (slow, heavy, bad directory access) to get around the win32/64 API issues.

So, rdiff-backup is great for local Linux system backups, but for remote backups, you’ll want to use different technology, like rsync. If you’re on Windows and want remote backups, check out some other solutions.

Good writeup on rdiff-backup features, method, and algorithm.

S1/Disk1 -rdff-backup→ S1/Disk2 -rsync/ZFS send→ R2/Disk1

If you have a lead or solution for my woes, please let me know! I often miss trivial solutions.

Here’s an actual rdiff-backup set to clarify:

        Time                       Size        Cumulative size

-—————————————————————————————————————

Fri Jul 10 01:32:13 2009         4.37 GB           4.37 GB   (current mirror)

Thu Jul  9 01:32:13 2009         36.3 MB           4.41 GB

Wed Jul  8 01:32:13 2009         37.5 MB           4.45 GB

Tue Jul  7 01:32:14 2009         31.4 MB           4.48 GB

Mon Jul  6 01:32:13 2009         31.1 MB           4.51 GB

Sun Jul  5 01:32:13 2009         27.0 MB           4.53 GB

Sat Jul  4 01:32:14 2009         41.3 MB           4.57 GB

Fri Jul  3 01:32:12 2009         33.9 MB           4.61 GB

Thu Jul  2 01:32:13 2009         37.9 MB           4.64 GB

Wed Jul  1 01:32:14 2009         35.4 MB           4.68 GB

Tue Jun 30 01:32:13 2009         37.3 MB           4.71 GB

Mon Jun 29 01:32:14 2009         38.9 MB           4.75 GB

Sun Jun 28 01:32:13 2009         38.7 MB           4.79 GB

Sat Jun 27 01:32:15 2009         42.0 MB           4.83 GB

Fri Jun 26 01:32:13 2009         49.3 MB           4.88 GB

Thu Jun 25 01:32:13 2009         37.3 MB           4.92 GB

Wed Jun 24 01:32:14 2009         36.4 MB           4.95 GB

Tue Jun 23 01:32:13 2009         43.0 MB           4.99 GB

Mon Jun 22 01:32:15 2009         33.4 MB           5.03 GB

Sun Jun 21 01:32:15 2009         31.0 MB           5.06 GB

Sat Jun 20 01:32:13 2009         41.6 MB           5.10 GB

Fri Jun 19 01:32:14 2009         31.7 MB           5.13 GB

Thu Jun 18 01:32:14 2009         32.0 MB           5.16 GB

Wed Jun 17 01:32:15 2009         31.0 MB           5.19 GB

Tue Jun 16 01:32:17 2009         31.6 MB           5.22 GB

Mon Jun 15 01:32:16 2009         31.7 MB           5.25 GB

Sun Jun 14 01:32:14 2009         31.3 MB           5.28 GB

Sat Jun 13 01:32:14 2009         30.7 MB           5.31 GB

Fri Jun 12 01:32:14 2009         31.3 MB           5.34 GB

Thu Jun 11 01:32:15 2009         32.3 MB           5.37 GB

I’ve been running Vista on a laptop since Sept. Generally, I don’t really use Vista for anything other than a platform to run virtual machines. What I need from Vista is

1. a stable host platform
2. Disk access and storage for VMs
3. Network access and bandwidth for VMs
4. Video access for VMs
5. Accurate Time Service since VMs get time from the host OS. There’s no way to disconnect time from the host provider.

Not all computers will go through airports or traveling to client locations in your car. So why do you need to encrypt some of the data on desktop systems?

Tech Support

At some point, tech support will be needed for every computer. Unless you are or have access to a computer nerd, a tech support call will be involved. If you are a retail user, you’ll take your non-working system into a big box electronics retainer or computer store and leave it there a few days or more to be worked on. During that time, all your media and any interesting files (spreadsheets, financial data, programs) will be copied off by the minimum wage techs.

Which files? Quicken files, emails, EI browsing history, games, high cost programs, and all audio and video files will all be available to and copied by the tech. Also passwords, PKI keys, and any certificates will be copied. Expect that this copying happens and plan for it.

Wouldn’t you rather have those files encrypted inside a volume that is password protected so nobody gains access to them? The bad news is you need to set this up before there’s an issue, since after the issue happens, you PC probably can’t be used.

House Guests

We all have guests in our homes and the first thing my guest expect is the ability to check their email and blogs. Some bring their own PCs, but family members usually don’t bring any since there are many computers all around here.

Internal IT Support

If you are an executive in your company or temporarily store sensitive data on your work PC, you also need truecrypt. I’ve seen all stock option grants for an entire company stored on a network shared disk, unencrypted. At least this data was put someplace that got backed up, unlike desktops, but sensitive data needs to have additional protection.

Summary

Your mother was right. Better safe than sorry. Think about the sensitive data you have on your PC. Think carefully, since most people will say I don’t any anything important on my PC. How would you or your company feel if that data was on the front page of the New York Times? That is what is at risk.

There’s a new competitor to Blackberry, Windows Mobile and Smartphones available, the Peek Pronto.

It looks like the RIM 957 with color. No phone or web browser, just email and texting. This is great for companies that want their people connected, but not with a cell phone ripe for abuse.

I’m concerned when an email-only device doesn’t clearly state the security features. A lack of network and data encryption and remote wiping is discouraging. At a minimum, HTTPS and IMAPS and POP3S need to be clearly supported. A device password lock with encrypted file system would be easy to add, IME. In that way, even if the device were lost, the data on it would be protected provided the password wasn’t hacked. Of course, real security goes beyond a “password” and complex passwords, autolocking, mandatory change periods, no password reuse, etc. are needed too.

But keeping it simple is a good thing. The Pronto seems to do this.

• email (5 acnts),
• texting,
• view images,
• view DOCs and PDFs.
• No web.
• No cell phone.

4/2009

• $80 for the device.
• $20/month for nationwide GSM service

There is an older device that is cheaper, has the same monthly plan costs, but doesn’t support text or anything other than email.

Blackberry Still Wins

Blackberry security still beats all the hand held devices, that hasn’t changed. Windows Mobile devices win on flexibility. Both cost significantly more than the Peek-Pronto.

Netbooks are becoming more and more viable to replace all these devices for those who need to get work done while on the road, not just check email.

Nokia Internet Tablets

Anyone who knows me, knows that I love the Nokia N800/N810 Internet Tablets. These devices should be on any list that a Peek Pronto is on and any list that an iTouch, WM6, Blackberry or Netbook is on too. Both the N800 and iTouch use WiFi and Bluetooth for connectivity – no data plan is required, therefore, no monthly data plan is required. This is a major plus.

Summary

The Peek Pronto is a low end email device that requires a monthly data plan to be useful. Security may or may not meet your requirements. We can’t tell based on the advertising.

This page was written without actually touching or seeing the device ourselves. It is based on what the getpeek website says (and doesn’t say). Without touching the device, it is impossible to determine whether the keyboard feel is good or not. That can be a critical decision factor for hand held devices.

Step 1 – backup your data.
Step 2 – recover your data as a test from a friends home or business.

The stuff you learn in step 2 is critical. We don’t backup data just to see it complete. We intend to get that data back at some point.

• Do you have access to the encryption keys used during backup? No encryption? – WHAT!? ARE YOU CRAZY?
• Do you have enough of the backup software (or can you down load it) to recover your data from bare metal, if needed?

Testing is critical to know what does and what doesn’t work. Don’t forget to fix the uncovered restore issues.

Recently, I’ve been running IT for a small business. Backups and Disaster Recovery are critical for us AND our customers. With our background in enterprise solutions, we were limited in knowledge for low-end solutions that didn’t cost an arm AND a leg to implement. High end solutions from EMC, Sun/StorageTek, IBM, and HP were our expertise. OTOH, we know that we need to do lower cost solutions better than anyone else does since technical architecture is our business. Having an outage due to a system failure is unacceptable. If a disaster occurs, we need to be up and running with acceptable data loss the next day. Period. Unplanned downtime for a trivial reason simply isn’t allowed. It can’t happen.

Requirements

1. Trivial Backup of data – Backups need to be easy to automate. If they aren’t completely automatic, then they won’t happen.
2. Even easier restoration of data – Backup is 10% of the problem. Recovery is 90%. Recovery at 3am after a bad day and little sleep is the goal.
3. Encrypted transfer – No peaking at our data, please. Strong, industry standard encryption. Claiming FIPS compliance, but not saying the real encryption used is just scary.
4. Encrypted disk storage – No peaking at our data, Mr. Service provider. Strong, industry standard encryption.
5. Differential backups / Incremental – We only want to transmit data that has changed since yesterday of the internet. This keeps bandwidth costs low after the initial backup seed.
6. Selection of recovery from last night or 3 weeks ago – 30 days worth of backups where we can recover data from last night or 3 weeks ago just as easily.
7. Compression – 60% compression of data is fairly standard.
8. Pay by use – Pay by the GB or TB, not by the amount of packets transferred (Sorry Amazon). 1TB needs to be in the $150/month range at most.
9. Windows / Linux support – Desktop, laptop, and Server OS supported.
10. Open file backup – Windows is known for keeping files open and preventing a good backup from completing. Linux will keep a few files open all the time too, but that doesn’t mean we don’t want them backed up.
11. Recovery by the file, not the entire backup set.
12. Full Virtual Machine backup from outside a VM. VMware and Xen and VirtualBox supported.
13. Near and Far Backup support – Off-site backups are great, until your network connection is down. If a user just noticed a lost a file due to corruption that happened last week, it needs to be easy to recover the single file from 8 days ago.

Possible Solutions

There are too many to list, but in our search, we found:

• Home made script; cron job, rdiff-backup, gzip, mcrypt, and rsync to remote location.
• Numerous backup solutions, but no low cost solutions appear to run on both Linux and Windows-whatever. If a backup server platform can be dictated, the an optimal solution may be possible. AMANDA, Bacula, BackupPC, rdiff-backup and many others may be suitable.
• Mozy.com – part of EMC now and appears to have everything we need, except Linux support.
• Rotate USB drives connected nightly/weekly then a mirror or incremental backup. Then take them home or off-site daily/weekly with the rotation.

My Answer and Why

I’ve selected the home made script with rdiff-backup at the core. Most of our production infrastructure runs on Linux inside Xen-based virtual machines. We automatically shutdown each VM nightly for a few minutes, run the rdiff-backup and bring the machine back up. All of our efforts require less than 3 minutes of downtime each. Very acceptable for a known clean file system, IMHO. Then the output is packaged into a single file per server, compressed, encrypted and transmitted to another local machine with protected disks. The amount of daily change data is relatively small – 10MB per server for a complete VM (OS, applications and all data). Over a 30 day period, retaining 30 days of incremental backups, about 1GB of extra data is incurred above the compressed initial full backup size. Most of our Xen VM backups are under 2GB is size. 5 servers, 10GB, not bad? There are exceptions to these sizes. One of our server VM backups is 14GB compressed.

Mozy would be a viable solution for Windows provided you don’t have too much data to backup. The cost really explodes since 100GB is over $100/month. Also, it doesn’t support local backups. The cost and security are well within reason for your most critical data and Mozy is part of EMC – who else would you trust with important data. If critical data on your CxO laptops aren’t being backed up nightly, What are you thinking? Be certain the data is encrypted on the laptop too by using Truecrypt too.

Background for SaaS

SaaS, Software as a Service, is the rage in the computing world today. Every big company is getting into it and using Web 2.0 as the initial buzzword. I read a few IT news aggregaters every day and each of them constantly link to SaaS websites. You know them too. A short list:

• Google, Google Mail, Google Docs, Google Voice, Google Maps, Google-Checkout, Google-whatever
• Yahoo Mail, Yahoo Small Biz
• eBay, Paypal, RIM/Blackberry
• online stock trading
• Twitter, 30boxes, iTunes, Don’t forget the milk, Wordpress, Blogger, salesforce
• Microsoft-Live, Live Meeting, Microsoft SaaS, Small business packages
• GotoMyPC.com, Pandora, Hulu,
• 37signals – project management, contacts, collaboration, chat, project organization for small businesses

Basically, any website that provides a useful interface and requires your internet connection is SaaS. Email is the simplest SaaS even if you use Outlook or Thunderbird as clients. Without the server, there isn’t any communication happening.

If you are a SaaS provider, you probably love cloud computing, but that’s a different article.

The Good about SaaS

1. Quick deployment. Usually, you’re up and running in a few hours.
2. Low initial costs – $20/month per user per service is common. It really is impressive what you can get for this small price and you didn’t have to buy a server, tapes, backup, IT guru, anything. Of course, $99+/month for higher end solutions exists too.
3. You don’t run any network or server infrastructure for the application
4. In theory, THE experts run the software
5. Upgrades are handled by the experts. Usually the people who created to software. If something bad happens, it isn’t just your 10 users, so they work really hard to get it fixed, now.
6. Backups aren’t your problem
7. Disaster Recovery isn’t your problem
8. System Security isn’t your problem
9. You don’t have to have people to manage this system, send them to training, keep them up to date or worry about their career goals.

The Bad about SaaS

1. Internet down? So are your applications
2. Privacy? What’s that? Google applications are mostly free. How can they do that? By indexing everything and building a profile of you based on it. Any data that goes through the SaaS provider can be misappropriated for alternate use that you didn’t approve, didn’t know about and can’t stop. It may not even be the service providers’ fault since the server may be hacked for months before they realize it. | Monster Hacked. It is unlikely you’ll ever know that a service is hacked. Current laws in most locations do not require service providers to tell anyone.
3. Run by experts? Perhaps not. SaaS providers can be Fortune 50 companies or some guy in their basement. You can’t tell. The guy in the basement isn’t necessarily bad, but he probably doesn’t have the resources or skill to build fault tolerance, redundancy, and disaster recovery into his systems.
4. System redundancy? How many physical locations is your data stored? If the WAN link goes down, what happens? Google provides 3 addresses, but 37signals only provides 1. That doesn’t mean there are 20 systems behind that single address, but it usually means a single location for all the servers.
5. Secure Customer Data? Maybe not. Most of these SaaS solutions store your data in the same database with all the other customers’ data. If you main competitor uses the same SaaS provider, it is just a software glitch away from being provided to any of the users.
6. Security and Access Controls? Can you configure the SaaS access to be connected to your internal enterprise authentication? What happens to these accounts when someone leaves your company? Are they automatically disabled?
7. Backups performed – maybe not
8. Disaster Recovery
9. All that data you enter – how do you get it out? Suppose you go with Salesforce.com, a leader in SaaS CRM applications. Your sale team enters 1,000s of leads and contacts into this system. The more you use it, the more useful it is, so the more they can charge you. Makes sense. For 100 users, it is $150K/yr subscription for the enterprise version. Don’t get trapped.
10. You want to switch providers – how do you migrate from SaaS-A to SaaS-B or bring the solution internal?
11. Larger WAN connectivity needs and you’ll probably want a redundant link too. You hope the provider has redundant WAN connectivity too.
12. Even the best providers fail. gmail down 1 | gmail down 2 | gmail down 3 If google can’t keep email up, what hope do the other SaaS providers have?
13. Who owns the data? You or the service provider? Be certain that you know.

SaaS isn’t all good and it isn’t all bad. It definitely has a place, but be certain that if you go with this model for your business, be certain you understand the limitations and get a contract executed that addresses your downtime concerns sufficiently.

Backup Schedule

Backups are boring, when done properly. They’re boring when done wrong too, but you can’t tell the difference until the day comes – and it will – that you need to recover data. There’s nothing new below, but it may be new to you.

Over the last 40 years, a standard, minimal, backup schedule has been developed to address many of the shortcomings that non-standard backup schedules experience. There are too many reasons to describe all the issues that the schedule below solves. Know that straying too far (or any) away from this schedule places your systems and data at risk.

Below is THE STANDARD monthly backup schedule:

S M T W T F S

- – - – - – - 

D D D D D D M

D D D D D D F

D D D D D D F

D D D D D D F

• D = Daily differential backup – changed data only, unless you can perform full backups within your backup window without impacting users
• F = Full backups – this limits the number of daily backups to be restored if there is an issue that week to 6 or 7
• M = Monthly – mark the first full backup of each month as the monthly and store it
• If you have limited backup infrastructure, you may need to split the weekly/Full backups between multiple days like Friday, Saturday and Sunday. Since this complicates your overall solution, this should be avoided.

Backup Retention

Doing the backup is just a small part of this solution. You also need to retain the backup as long as it is useful, plus 1 extra copy.

• Dailies – keep at least 2 weeks
• Weeklies – keep at least 4 weeks
• Monthlies – keep at least 2 months, but consider retaining 6 months

1. Legal requirements may demand longer or shorter retention periods. If you can, keep all backups on the same retention schedule. Be extremely clear on the backup media what the purpose and retention is.
2. Test the backups. If you don’t test it, you’ll never know whether that hard work is actually working.
3. So we’re keeping a bunch of copies following this method. You’ll retain 5 full backups between weekly and monthly versions. If you have issues or get hacked, you’ll have ample recovery options. With all these copies, you understand that the lowest cost media is usually deployed.