How-To Migrate Debian/Ubuntu Systems and Data Overview 2

Posted by JD 12/11/2013 at 21:00

From time to time, we all need to migrate our systems from 1 machine to another. With Linux, often it is possible to swap the old HDD into the newer machine to get that accomplished. Sometimes that is not an option and we need to migrate the data, settings and installed programs some other way.

A few options to accomplish this are below. This is an APT package manager solution. Other Linux package managers should work in a similar way, just replace the dpkg commands with whatever the package manager needs for your specific system.

Just Swap the HDD

This is the easiest method. Take the HDD from the old system and physically move it into the newer system. If UUIDs were used for point points in the /etc/fstab and the CPU is at the same level or newer (x64) then this should just work. Only the wired ethernet devices might be differently named. To fix that, just delete the old rules from under /etc/udev/… At the next reboot, the current ethernet device should be discovered and added as eth0.

Mirror/Replicate Everything

Sometimes, swapping the HDD is not possible – like when you do not own the equipment – so migrating everything (all the files) is desirable – using something like rsync or ddrescue if the target HDD does not have any OS yet.

If the HDDs have different sector sizes (512B vs 4K), then dd/ddrescue are not a good idea. Terrible performance might result. If the HDD sector sizes are different, a mirror method that works at the file system level is needed. Something like fsarchieve. How to use that is left as an exercise to the reader, but it is not very difficult. I’ve done it a few times.

Migration

Other times, the other OS is pre-installed and moving everything just is not possible, so a limited migration is needed. That’s the point of this article. From a very high level, here are the steps to migrate between 2 systems. I use this method all the time. It is also a way to migrate from one OS to another with the least impacts. I prefer this to letting the OS upgrade happen.

  1. update and dist-upgrade your Ubuntu. You want all software to be current before starting.
  2. Make a backup and verify it. I’m serious. Things can go really wrong. This backup needs to be to a disk that honors Linux file permissions and ownership. EXT4 is probably the best answer today, but jfs, xfs, ext3, zfs, and many others will work just as well. This disk needs to be accessible from both the VM AND the running Ubuntu OS later on the physical machine. Network connectivity is fine, so the storage area does not need to be physically connected.
  3. Create a list of all packages running on the VM – dpkg —get-selections > ~/pkg-list does that. Shove the list into a file and save that file.
  4. Remove any VM “guest additions”, unless the new VM uses the same technology. However, if this is what you are doing, then using the built-in VM-Export function is probably better AND easier.
  5. Install the exact same version of Ubuntu to a new HDD installed in the machine. If 64-bit, use 64-0bit. If 32-bit, use a 32-bit version. This is not strictly required and migrating between 32-64-bit is possible. The risk here is incompatible programs and data files, so only you can decide which settings and programs will migrated easily. I have used this method to migrate from x32 to x64 bit, but there were a few extra manual steps needed that I can’t recall now. It will mostly work.
  6. update and dist-upgrade your new physical Ubuntu.
  7. Take that backup disk and connect it to the newly installed OS – a network connection is fine.
  8. Restore the settings that you changed in the old system. If you have a great backup tool, you can just restore from the backup into the new machine and all will be fine … er … mostly. Things that will likely ruin your day include any funky Apple hardware, UEFI, the different network card, …. that should be it. Be prepared to correct those issues. The network card is usually just easy to remove the old lines inside /etc/udev/rules.d/70-persistent-net.rules and reboot. With all the data – often just /etc and /home, we are ready to add the packages.
  9. Take the list of packages that you made previously and use dpkg —set-selections < ~/pkg-list to tag all that you want installed on the new system. This just sets the list inside the APT system and does not install anything.
  10. Now tell APT to install everything in the current list. If you didn’t install any programs outside the package manager, you are now 100% current with software between both machines. Pretty great, huh?
  11. Generally, it is 100% safe to restore your HOME without any concern. Occasionally, different desktop environments and/or settings will cause issues. An easy short-term work around for this is to create a new account, use the problem program and see which settings are conflicting between the new account settings and the old, migrated, account settings.
  12. /etc – will have a few hardware specific items like the fstab and udev stuff previously mentioned.
  13. /var probably has your website stuff, but if this is a dynamic site, then you need to worry about the DBMS files and things shoved under /usr/share or /usr/local or elsewhere. It is impossible for me to say all the places that files might be stored on the old system.
  14. Any proprietary video drivers that you want to install. The F/LOSS drivers are getting to be very good. If you are migrating a VM, then forget about the physical video card. It almost always doesn’t matter unless you have gone way-way-way out of the norm and are using direct GPU passthru. That is non-trivial and requires extremely specific hardware and driver support.
  15. If you installed anything outside the package manager, you need to migrate that over too … manually. After doing this a few times, you’ll decide like I have to avoid non-packaged software in the future.

Anyway, I hope this is helpful. I’ve migrated about 15 VMs to other VM technologies and this has worked for me. Have you considered creating a VM server? Then you can put multiple client OSes under it, do development for different clients in a separate VM and not risk any corruption between the different systems. You can also freeze an entire VM, push it to any external disk, perhaps on a shelf, and have it there with all the development environment ready should that client come back later. A 1TB HDD can hold the backups for thousands of client VMs. You’ll also be able to offer a “web-site restore” feature to those clients after they try to change something in the config and break it.

Sharing a VM with others is also a good use for this backup method. There won’t be any question as to whether the VM has the dependencies for a specific development environment. If the frozen VM is recent, things should just work under a new VM host.

There are other ways to accomplish this same outcome

Another AskUbuntu.com answer that looks really, really good to me.

  1. JD 02/27/2014 at 15:13

    New ways to save/restore package lists
    Backup packages:

    apt-mark showauto > pkgs_auto.lst
    apt-mark showmanual > pkgs_manual.lst

    Restore packages:

    sudo apt-mark auto $(cat pkgs_auto.lst)
    sudo apt-mark manual $(cat pkgs_manual.lst)

  2. JD 05/04/2014 at 17:07

    I tried to use the apt-mark method in the previous comment. It didn’t work as cleanly as the
    dpkg --get-selections/--set-selections method listed above.