root'ing the netRamblings from a a coding sysadmin

Yesterday I found The Admin Zen. Recommended reading for all the systems administrators out there. I liked the fact that some of the principles are loosely based on The Zen of Python, which you can read by just typing the following in a console:

python -c 'import this'

Enjoy… and keep your stuff up and running

So I am right now running Ubuntu 8.10 “Intrepid” in my workstation. In general, the update was smooth, but I have a word of warning: if you use OpenBox as window manager in the GNOME desktop environment, it will stop working. If you try to log in from GDM using the “GNOME/OpenBox” session kind, you will be kicked back to the login screen. As a workaround you can execute openbox --replace from the “Execute” dialog from a regular session as a workaround.

Just in case you did not know about that, I spent some minutes today figuring out how to resize a LVM logical volume there are snapshots of it: the volume cannot be active. This is a pity if your root filesystem is inside a logical volume. Fortunately, you can stop the bootup process of recent Debian installations in the initial RAM disk by passing break=mount; unfortunately this minimal system has the vgchange tool (needed to start/stop volumes) but it does not have lvresize (guess what it is needed for?).

Finally I ended up firing up the bootup process from a Debian “netinstall” CD, entered rescue in the bootloader prompt and switched to the second console just after the installer detects hardware and before it mounts disks. Then it was only a matter of typing:

 # vgchange -an vg
 # lvresize -L +300M vg/base

Then reboot the installed system and make the XFS filesystem grow with:

 # xfs_growfs /

…and you are done

As I have written before snapshots is a extremely fast and convenient way of perfoming live backups which can be mounted and handled like any other filesystem. But I also told you that LVM snaphots could render themselves as unusable when they run out of free blocks. Today I hacked up a quick script which automagically increments sizes of snapshots logical volumes. Just drop the following snippet into /etc/cron.hourly and relax:

#! /bin/bash

THRESHOLD="80"
INCREMENT="15"

set -e
IFS=':'

/sbin/lvs --noheadings --units M --separator : | \\
while read lv vg attr lsize origin snapp move log copyp
do
  # Check whether this is a snapshot or not
  [ "${snapp}" ] || continue

  snapp=${snapp%.*}
  lv=${lv// /}

  # Check whether the thing needs resizing
  [ "${snapp}" -ge "${THRESHOLD}" ] || continue

  lsize=${lsize%.*}
  isize=$(( INCREMENT * lsize / 100 ))
  echo "lvresize -L +${isize}M ${vg}/${lv}"
  /sbin/lvresize -L "+${isize}M" "${vg}/${lv}"
done

You can change the following settings:

• THRESHOLD is the percent of usage which triggers resizing. When actual usage is greater than this value snapshots will grow.
• INCREMENT is used to calculate how much size is added to the volume, it is a percent of the current volume size.

Of course the script could be improved (i.e. it could check whether there are space for growing in the volume group), but this naïve implementation is enough to make me happy and not to worry about checking status of my snapshots periodically

Let us suppose we have a volume group named vg. Let us suppose we have a logical volume named base which holds our precious data in a 800 MB file system which has support for freezing (e.g. XFS). So we can take snapshots at any time with a single command:

# lvcreate --size 200M --snapshot --name snappy /dev/vg/base

Now we can mount it whenever we want to recover from the saved status:

# mount -o nouuid,ro /dev/vg/snappy /mnt

The nouuid option is needed for XFS filesystems, otherwise the driver will think it is mounting the same filesystem and will refuse to mount the snapshot. Take into account that:

• Blocks in the source volume are stored on-demand in the snapshot volume when they are changed.
• If mounting the snapshot in write mode, written blocks will be stored in the snapshot volume.
• You can make the original filesystem grow, but it cannot be shrunk.
• Snapshots can be grown and shrunk.
• If a snapshot runs out of free blocks, it will render itself unusable: make sure you check its status periodically with lvdisplay and grow it as needed.

Currently I am using snapshots as a quick-come-on-let’s-go backup method for the root filesystem of a Debian installation I am using for a project. If somethings goes wrong one can mount the snapshot and restore files from the snapshot… or one can even boot up from a snapshot by setting root=/dev/mapper/vg-snappy in the kernel command line…

As I am currently working with KVM KVM and I will have some big disk image files hanging around, I thought it could be a good opportunity to give a try to the btrfs snapshot and subvolume support. You can take a look at my previous post

First of all: snapshots and subvolumes are the same to btrfs. To be fair: snapshots are subvolumes with initial content. The nice thing about btrfs is the Copy-On-Write (COW) semantics: snapshots do not take physical space until blocks are overwritten, and only new versions of overwritten blocks are allocated from free space. This is a lot similar to KVM/Qemu snapshots when dealing with images in the qcow2 format. The difference is that you can mount a btrfs volume: you get features designed for virtualization right on the filesystem level! Once again old technology is applied in a novel way

I have a btrfs volume mounted at /mnt/misc, and a 1GB Debian “Etch” image and a file with some notes about the image:

# ls -l /mnt/misc
-rw-r--r-- 1 root root 1000000000 2008-06-24 20:44 etch-base.img
-rw-r--r-- 1 root root         60 2008-06-24 18:35 etch-base.txt

Now let us create a snapshot named snappy:

# btrfsctl -s snappy /mnt/misc
# ls -l /mnt/misc
-rw-r--r-- 1 root root 1000000000 2008-06-24 20:44 etch-base.img
-rw-r--r-- 1 root root         60 2008-06-24 18:35 etch-base.txt

Hey! Wait a second! Nothing seems to happen!… This is due to all subvolumes and snapshots hanging from the filesystem root, and the default subvolume is mounted (you guessed it) by default. Let’s remount another subvolume:

# umount /mnt/misc
# mount -o subvol=snappy /dev/hda1 /mnt/misc
# ls -l /mnt/misc
-rw-r--r-- 1 root root 1000000000 2008-06-24 20:44 etch-base.img
-rw-r--r-- 1 root root         60 2008-06-24 18:35 etch-base.txt

Nothing impressive… now for the fun part we will add some garbage, and mount all the subvolumes at once:

# touch /mnt/misc/hello-btrfs.txt
# echo 'btrfs rocks' >> /mnt/misc/etch-base.txt
# umount /mnt/misc
# mount -o subvol=. /dev/hda1 /mnt/misc
# ls /mnt/misc
default  snappy

Now the mount point shows a directory for each subvolume, and the files are really there:

# ls /mnt/misc/*
/mnt/misc/default:
etch-base.img  etch-base.txt

/mnt/misc/snappy:
etch-base.img  etch-base.txt  hello-btrfs.txt

Now please run df -h /mnt/misc by yourself and get surprised: only about 1GB of physical space is really in use. In fact that can even be less than 1GB, as btrfs supports sparse files as well (if there are “gaps” they do not take physical space). Final remark: you can also make snapshots of snapshots.

Last weekend I was able of emptying a partition and had the opportunity of testing btrfs for first time. I made some simple timings using the Bash time builtin. The test was building a Linux kernel with the default configuration (make defconfig). I timed the time needed for unpacking kernel sources, building and removing the tree afterwards. The results against XFS are as follows:

All measurements are shown in seconds, so smaller numbers means better performance. I was surprised with how fast was unpacking kernel sources on btrfs, so I was expecting a slightly better build time. The reason for the similar build times can be the fact that the tools (compiler, linker, etc) were running from a XFS partition, and that it was a CPU-bound. Both test over XFS and btrfs were done after a clean system boot and freshly created filesystems.

What I loved about btrfs was the ability of growing and shrinking the filesystem while mounted and being accessed, which makes it a perfect candidade for its use over LVM. I still have to try out subvolumes, live writable snapshots, and making some serious testing. Even so, btrfs looks like a Good Thing™ to me.