Out of Disk Space-inodes! 4

Posted by JD 02/01/2013 at 00:00

Some days we just can’t win.

This little blog VM is out of disk space … er … sorta.
It is out of inodes. Fortunately, it is at 93% use for sector storage, but unfortunately, there are less than 3K inodes available.

apt-get update cannot complete. Out of disk error.

The VM has 1.5G of swap that never gets used. The RAM amount is set to prevent that need on purpose, so I decided to

  • change the swap partition into a data partition,
  • rsync over a few thousand files from /var/www,
  • mount /var/www from the other partition (vda5) and
    … life would be good. Yes, everything under /var/old-www was removed after mounting.

It was all scripted to minimize downtime. AND there is a good backup from last night should anything terrible happen. Downtime was a little over 30 seconds. Good enough. Heck, www.amazon.com was down for at least 10 minutes today, so my 30 seconds should be fine.

After all that, now there are fewer inodes available on /!!!

$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/vda1 296000 293371 2629 100% /
/dev/vda5 97344 3751 93593 4% /var/www

Looks like it is time to build a larger VM file. The blog storage was already expanded last year – which is probably why the inodes are so tight. Copying the entire VM will take longer than a few minutes.

What a pain. I hate downtime, but I hate leaving a system sorta-patched must more.

Some days we just can’t win.

  1. PerseP 02/01/2013 at 14:26

    Hello,
    what kind of vm do you use and what settings does it have?. Is it hosted on a vps or home computer?

    Thanks

  2. JD 02/01/2013 at 17:51

    There is a list of “tags” on the right side of this page that might help your discover the answers to your questions.

    Update
    Wow, those links aren’t working. Well, there is a REST interface to the tags. domain/tag/{tag} that will find those articles.
    So, to find all the articles here tagged with virtualization, then the URL would be blog.jdpfu.com/tag/virtualization. Simple, easy and I can’t believe the sidebar is broke over that. The blog software has a bug already reported about this.

    I looked at the code to see if it could be easily fixed. I’m sad to say, that I couldn’t understand it and the method called that builds the URLs does not seem to exist anywhere else in the code directory systems. It is the magic of Rails it seems.

  3. Defron 02/02/2013 at 18:40

    Have you thought about thin provisioning your blog VM? It’ll make adding storage easier in the future. I’m not sure how it’s done in KVM, which IIRC is the hypervisor you use for your blog’s VM. It’s not without downsides, though. It’s VMWare-centric, but it gives you a good idea of the general pros/cons of thin provisioning as well: http://lonesysadmin.net/2009/11/06/vmware-vsphere-4-thin-provisioning-pros-cons/

    I’m sure there’s KVM-centric pros/cons lists on the web too. I’m just taking a coffee break at work right now :P

  4. JD 02/02/2013 at 22:55

    @Defron: The performance hit for sparse file system allocations is huge. I’ve performed testing and would never consider it a viable option for any VMs on spinning HDDs. I learned this the hard way about 6 yrs ago. I’ve used most of the virtualization hosts/hypervisors out there and they all suffer from this performance drop when using sparse file allocations.

    However, if the VMs were running off SSDs, then I would always use sparse virtual disk allocations. SSDs are so fast.

    I would be shocked if the number of inodes for the file system were variable. It is not a big deal to create a new, larger FS for a VM and copy all the data over.

    But since I’ve been burned by running out of inodes a few times with perl and ruby application servers, I think I’ll force a much higher ratio of inodes to total storage for these VMs. Why perl and ruby – these languages have lots and lots of tiny files and when you use the best practices for each, then the entire environment for every application becomes self contained along with all the script addons (gems and cpan modules).

    I share these issues so when other people experience them, they don’t have to spend too much time trying to figure out what is really happening. There are many more new-to-Linux users out there now than ever before. Seeing 65% disk usage, but out of storage errors can be confusing.
    By sharing, everyone will see the root cause a little quicker.