Full disclosure: I’m one of those people who assembled such a NAS. And I was one of those people who spent an inordinate amount of time worrying about the 4K alignment problem. I eventually learned that I’d been wasting my time. When you find people on various mailing lists, forums, and blogs talking about this, please point them here.

The fundamental problem is the need to align data structures to the physical storage device’s boundaries. If your device stores data in block of 4K, then you want to begin writing your data at a multiple of 4K. You do not want to start writing 4K of data beginning 2K from the start of the device, because that will mean reading and writing two blocks of data. It would be more efficient to align to the boundary and write only one block.

All low-level storage systems, from filesystems (ext4) to logical volume managers (lvm2) to software raid (mdadm) use various block sizes. The terms vary from blocks to chunks to stripes, but the concept is the same. We want these blocks to align to what is best for the underlying layers, from any RAID configuration down to the physical devices.

To solve this problem, libblkid provides topology information for block devices. This includes physical and logical block sizes. As of spring 2010, support for libblkid is available in all the standard tools:

1. fdisk, since util-linux >= 2.15. You should start with ‘-c -u’ to disable DOS compatibility and use sectors instead of cylinders.
2. parted, since parted >= 2.1. Parted defaults to 1 MiB alignment for unknown disks, which should align to just about any sector size. This includes gparted and other graphical tools such as the partitioner used in the Ubuntu desktop installer.
3. mdadm, since util-linux >= 2.15. It is no longer necessary to manually specify stride and stripe-width. All MD superblocks are positioned either at the end of the array (0.9 and 1.0) or in multiples of 4K from the beginning (1.1, 1.2).
4. lvm2, since util-linux >= 2.15.
5. mkfs.{ext,xfs,gfs2,ocfs2} all support libblkid directly.

If you’re uncertain, verify that your tool depends on util-linux or libblkid. You can use the package manager, although you may need to examine a dependant library:

user@server:~$ dpkg -s libparted0debian1 | grep libblkid
Depends: libblkid1 (>= 2.17), libc6 (>= 2.8), libdevmapper1.02.1 (>= 2:1.02.36), libuuid1 (>= 2.16)

Or use ldd, which is more direct:

user@server:~$ ldd /sbin/parted | grep libblkid
        libblkid.so.1 => /lib/libblkid.so.1 (0x00007f5833228000)

Conclusion: if you are running Ubuntu 10.04, Fedora 13, or RHEL 6 or later, you do not have to do anything to use your 4K disk efficiently. Create the RAID array, logical volume, and filesystem exactly as you normally would.

The problem with this design is that conventional duplication/archival tools like rsync and tar can’t be used to duplicate the cpool in a reasonable period of time. Any file with more than one hardlink must be tracked and restored correctly on the destination host or filesystem. Copying my 700 GB pool with rsync between two identical disks with rsync would take a minimum of 3 days. Copying the same amount of raw data would take only a few hours.

So we can’t use filesystem-aware tools to backup a BackupPC server’s pool. Instead, we must make a bit-for-bit copy of the disk image.

One suggestion I’ve seen is to make the cpool part of a RAID1 array of two disks. When you want to make a backup, pull one disk and replace it. Or make it a 3-disk mirror with a removable drive, and periodically swap it out and carry it elsewhere. This is a sound solution, but it is still sneakernet.

What I want is a solution that:

1. Duplicates the entire BackupPC storage to a second server,
2. in a matter of hours not days,
3. over the network,
4. such that the secondary server can quickly replace the primary server if it fails,
5. without interrupting normal operations on the disk, including BackupPC itself.

To accomplish this, we build the BackupPC storage on LVM, and use LVM snapshots and netcat to copy the disk image.

My BackupPC server uses a 3ware hardware RAID card with two arrays:

1. /dev/sda is a RAID1 of two disks, and provides the / and /var filesystems.
2. /dev/sdb is a RAID5 array of 5 disks, and provides the physical volume for the backuppc logical volume and filesystem

The server’s filesystem looks like this:

root@backuppc:~$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             9.2G  1.2G  8.0G  13% /
/dev/sda3              63G   21G   41G  34% /var
/dev/mapper/volgroup-backuppc
                      1.4T  711G  644G  53% /var/lib/backuppc

If you’re planning to install BackupPC on a new server, this is easy to prepare. First, install the latest Ubuntu LTS server (I’m still using 8.04 “Hardy Heron”) on /dev/sda, using / and /var partitions as above. It is possible to configure LVM through the installer UI, but I’m not going to cover that here.

Once the base OS is installed, login via SSH and prepare the LVM setup. LVM is discussed in excellent detail elsewhere. I strongly urge you to read this How-To and play with LVM a little before building the final system.

All commands are run as root. Use sudo if you prefer.

1. Create one partition of type 8e (Linux LVM) on the RAID disk.
   fdisk /dev/sdb

   Command (m for help): o
   Building a new DOS disklabel with disk identifier 0xf2704d76.
   Changes will remain in memory only, until you decide to write them.
   After that, of course, the previous content won't be recoverable.
   
   Command (m for help): n
   Command action
      e   extended
      p   primary partition (1-4)
   p
   Partition number (1-4): 1
   First cylinder (1-243147, default 1):
   Using default value 1
   Last cylinder, +cylinders or +size{K,M,G} (1-243147, default 243147):
   Using default value 243147
   
   Command (m for help): t
   Selected partition 1
   Hex code (type L to list codes): 8e
   Changed system type of partition 1 to 8e (Linux LVM)
   
   Command (m for help): p
   
   Disk /dev/sdb: 1999.9 GB, 1999957393408 bytes
   255 heads, 63 sectors/track, 243147 cylinders
   Units = cylinders of 16065 * 512 = 8225280 bytes
   Disk identifier: 0x92cf8945
   
   Device Boot      Start         End      Blocks   Id  System
   /dev/sdb1          1      243147  1953078277   8e  Linux LVM
   
   Command (m for help): w
   The partition table has been altered!
   
   Syncing disks.

2. Prepare the physical volume and volume group.
   pvcreate /dev/sdb1
vgcreate volgroup /dev/sdb1
3. Create a logical volume which is 75% of the available space. We’ll reserve the remaining space for the snapshot.
   lvcreate -l '75%VG' -n backuppc volgroup
4. Create a filesystem on the logical volume. Some prefer XFS, but I’m happy with ext3 (with an eye to upgrading to ext4 later).
   mkfs.ext3 /dev/volgroup/backuppc

5. Add the new logical volume to /etc/fstab.
   vi /etc/fstab
   Add a new line. We mount with the most efficient options since BackupPC doesn’t need them.

   /dev/volgroup/backuppc /var/lib/backuppc ext3 noatime 0 2

6. Now mount it for the first time.
   mkdir /var/lib/backuppc
mount /var/lib/backuppc
7. Finally, install BackupPC. This will also install Apache2, Perl, and a mail server.
   apt-get install backuppc

You now have a working BackupPC system using LVM for storage. Now we create an LVM snapshot.

Why use LVM snapshots? Because we must take a copy of the filesystem as it exists at one moment in time. If we install directly on /dev/sdb1 and then try to copy the filesystem while it is mounted, the copy will be inconsistent. Some data blocks will represent the time we started copying, others the time we ended, or anything in between. There is a disk performance penalty to using snapshots, as this creates a copy-on-write fork of the filesystem from the moment the snapshot is taken. The best practice is to create the snapshot when the server is idle, copy it to the secondary server, and then remove it immediately after.

Create the snapshot using all remaining free space.
lvcreate -l '100%free' -s -n backuppcsnapshot /dev/volgroup/backuppc

If you are curious, you can now mount the logical volume /dev/volgroup/backuppcsnapshot and compare the difference between it and the original filesystem. But we don’t need to do that. We just need to copy it to our secondary server.

The secondary server should be configured exactly like the first. At minimum, it must have a partition or logical volume that is the same size or greater than /dev/volgroup/backuppc on the primary server. To copy it, we’ll use dd and SSH.

I previously recommended using netcat to perform this step. However, I’ve had a lot of difficulty getting cron, SSH, dd, and netcat to behave together when starting the listener on the destination server. I’ve also found that SSH is just as fast as using netcat, which is surprising given that encryption should be slower than transmitting the data in the clear.

Now we copy the data from primary to secondary via SSH. On the primary:
dd if=/dev/volgroup/backuppcsnapshot | ssh root@192.168.0.2 "dd of=/dev/volgroup/backuppc"

This reads the logical volume block for block and sends it to the secondary server on 192.168.0.2, via SSH. The secondary server receives and writes it to the logical volume.

One advantage of using dd is that you can easily check progress:

root@backuppc:~# ps -ef | grep dd
root      4958  4495  0 18:39 pts/0    00:00:03 dd if=/dev/volgroup/backuppcsnapshot
root@backuppc:~# kill -USR1 4958

On the terminal where dd is running, you’ll see this.

126689+0 records in
126689+0 records out
64864768 bytes (65 MB) copied, 9.97271 s, 6.5 MB/s

You may wish to use bzip to compress the data after the first dd operation and decompress it before the second, or use ssh -C. Using compression will save bandwidth at the cost of CPU usage. In my tests, a raw copy gave sustained speeds of 200 mbit over gigabit Ethernet. Using bzip2, throughput was reduced to 20 mbit. Therefore I do not recommend using compression if copying over your own gigabit LAN.

As a last step, archive the BackupPC configuration whenever you clone the storage pool. Since you don’t want the secondary to start running backups, put it in a safe place. On the secondary:

mkdir /etc/backuppc/backup/

On the primary, after copying the snapshot:

rsync -av --delete /etc/backuppc/ root@192.168.0.2:/etc/backuppc/backup/

We now have a complete BackupPC server with LVM storage and a means of completely backing up to a secondary server, over the network, without interrupting the operation of the primary. But what if you have an existing BackupPC pool directly on disk partition? You’ll need to migrate it to LVM without losing the data. I’ll discuss how to do that in my next post.

Updated 2010-09-17: removed directions for netcat and recommended SSH method.

One of the reasons I started this blog is to write about problems I’ve solved that I can’t find answers to already on the web. Today, let’s talk about moving your linux install to linux software raid (md raid / mdadm). This post assumes you are running Ubuntu Linux 8.04, but any Debian-based distro from the past two years, or most other distros, will have similar commands.

We start with an install on a single 80 GB SATA drive, partitioned as follows:

/dev/sda1 as /, 10GB, ext3, bootable
/dev/sda2 as swap, 1GB
/dev/sda3 as /home, 69GB, ext3

We want to add a second 80GB SATA drive and move the entire install to use RAID1 between the two drives. So the final configuration will appear:

/dev/md0 as /, 10GB, ext3
/dev/md1 as swap, 1GB
/dev/md2 as /home, 69GB, ext3

Where the raid arrays are:

md0 : active raid1 sda1[0] sdb1[1]
md1 : active raid1 sda2[0] sdb2[1]
md2 : active raid1 sda3[0] sdb3[1]

Here there be dragons. As always, back up your data first. If you don’t know how to use rsync, now is an excellent time to learn.

The general plan is:

1. Partition the new drive
2. Create RAID arrays and filesystems on the new drive
3. Copy the data from drive 1 to the new RAID arrays on drive 2
4. Install grub on drive 2
5. Configure fstab and mdadm.conf, and rebuild initramfs images
6. Reboot on the RAID arrays on drive 2
7. Repartition drive 1 and add it to RAID

All commands are run as root. Use sudo if you prefer.

Step 1: Partition the new drive

Assuming you want to partition the second drive the same way as the first, this is easy. Just clone the partitions from /dev/sda to /dev/sdb:

sfdisk -d /dev/sda | sfdisk /dev/sdb

You may want to format the new disk first to clear it of old data, especially if it previously had software RAID partitions on it. You may get unusual results if it has a similar partition structure or RAID setup as the original disk.

Now use parted to mark the partitions for software RAID, with the first partition to boot:

parted /dev/sdb
(parted) toggle 1 raid
(parted) toggle 2 raid
(parted) toggle 3 raid
(parted) toggle 1 boot
(parted) print

Disk /dev/sdb: 80.0GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type     File system  Flags
 1      0.51kB  10.0GB  10.0GB  primary  ext3         boot, raid
 2      10.0GB  11.0GB  2000MB  primary  linux-swap   raid
 3      11.0GB  80.0GB  69.0GB  primary  ext3         raid
(parted) quit

Parted will show you a file system in each partition, but the reality is that they will be plain linux software raid partitions (type 0xfd):

fdisk -l /dev/sdb

Disk /dev/sdb: 80.0 GB, 80026361856 bytes
255 heads, 63 sectors/track, 9729 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0xa3181e57

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1        1216     9765625   fd  Linux RAID autodetect
/dev/sdb2            1216        1338      976562+  fd  Linux RAID autodetect
/dev/sdb3            1338        9730    67408556   fd  Linux RAID autodetect

Step 2: Create RAID arrays and filesystems on the new drive

Now we create RAID 1 arrays on each partition. These arrays will have just one member each when we create them, which isn’t normal for RAID 1. So we’ll have to force mdadm to let us:

apt-get install mdadm
mdadm --create /dev/md0 --level=1 --force --raid-devices=1 /dev/sdb1
mdadm --create /dev/md1 --level=1 --force --raid-devices=1 /dev/sdb2
mdadm --create /dev/md2 --level=1 --force --raid-devices=1 /dev/sdb3

Next, you’ll need to create filesystems on the new RAID devices. Assuming the same formats as your first partition:

mkfs.ext3 /dev/md0
mkswap /dev/md1
mkfs.ext3 /dev/md2

Step 3: Copy the data from drive 1 to the new raid arrays on drive 2

This is a job for rsync. First, there are some directories on a running Linux system that we do not want to copy, like /dev and /proc. We also want to ignore tempfs directories, like /var/run. The best way to avoid these is to make an excludes file. Create a file, /root/rsync-filter, with the following content:

- /dev/
- /lib/modules/*/volatile/
- /mnt/
- /proc/
- /sys/
- /var/lock/
- /var/run/

These lines define directories we will not copy over. You may wish to add /tmp, apt’s cache, etc, but if you do you must manually create the directories themselves on the new filesystem.

Mount the new RAID array:

mount /dev/md0 /mnt
mkdir /mnt/home
mount /dev/md2 /mnt/home

If you are using a different mount structure, just be sure to recreate it and mount it in the right places in the new filesystem under /mnt/.

And start the rsync copy:

rsync -avx --delete -n --exclude-from /root/rsync-filter / /mnt/
rsync -avx --delete -n /home/ /mnt/home/

You will see a list of files that will be changed, but nothing actually happens. This is the job of the -n argument, which performs a dry-run. Always do this before actually starting a copy. You WILL make a painful mistake with rsync some day, so learn to be cautious. Repeat the above commands without the -n when you are sure all is well.

The -x argument ensures that you will not cross filesystem boundaries, which is why you must copy /home separately, and any other mounted filesystems. If you omit this you only need one command. But make sure you have a good rsync filter file, and that you have nothing mounted like /media/cdrom that you don’t want an archive of.

Finally, create the directories that you skipped with your filter:

cd /mnt/
mkdir -p dev/ mnt/ proc/ sys/ var/lock/ var/run/
for i in /lib/modules/*/volatile ; do mkdir -p /mnt/$i ; done

Step 4: Install grub on drive 2

OK, we almost have a working RAID install on the second drive. But it won’t boot yet. Let’s use chroot to switch into it.

mount --rbind /dev /mnt/dev
mount --rbind /proc /mnt/proc
chroot /mnt

Now we have a working /dev and /proc inside the new RAID array, and by using chroot we are effectively in the root of the new array. Be absolutely sure you are in the chroot, and not the real root of drive 1. Here’s an easy trick: make sure nothing is in /sys:

ls /sys

If not, you’re in the new chroot.

Now we need to update grub to point to the UUID of the new filesystem on the RAID array. Get the UUID of /dev/md0′s filesystem:

tune2fs -l /dev/md0 | grep UUID

Filesystem UUID:          8f4fe480-c7ab-404e-ade8-2012333855a6

We need to tell grub to use this new UUID. How to do this depends on which version of grub you are using.

Step 4.1: If you are using GRUB 2, which is the default on Ubuntu since 9.10 Karmic Koala:

Edit /boot/grub/grub.cfg and replace all references to the old UUID with the new. Find all the lines like this:

search --no-floppy --fs-uuid --set 9e299378-de65-459e-b8b5-036637b7ba93

And replace them with the new UUID for /dev/md0:

search --no-floppy --fs-uuid --set 8f4fe480-c7ab-404e-ade8-2012333855a6

Do not run grub-mkconfig, as this will undo your changes. Install grub on the second drive.

grub-install /dev/sdb

Go on to step 5 below.

Step 4.2: If you are using GRUB 1 (Legacy), which is the default on Ubuntu before 9.10 Karmic Koala:

Edit grub’s device.map and menu.list files. Edit /boot/grub/device.map and make sure both drives are listed:

(hd0)   /dev/sda
(hd1)   /dev/sdb

Edit /boot/grub/menu.list and find the line like this:

# kopt=root=UUID=9e299378-de65-459e-b8b5-036637b7ba93 ro

Replace the UUID with the one you just found. Leave the line commented, and save the file. Now rebuild menu.lst to use the new UUID:

update-grub

Double-check that each boot option at the bottom of menu.lst is using the right UUID. If not, edit them too. Finally, install grub on the second drive.

grub-install /dev/sdb

Step 5: Configure fstab and mdadm.conf, and rebuild initramfs images

We’re almost ready to reboot. But first, we need to build an initramfs that is capable of booting from a RAID array. Otherwise your boot process will hang at mounting the root partition. Still in the chroot, edit /etc/fstab and change the partition entries to the new filesystems or devices.

If your /etc/fstab has “UUID=” entries like the following, change them to the new entries:

proc            /proc           proc    defaults        0       0
UUID=8f4fe480-c7ab-404e-ade8-2012333855a6 /               ext3    relatime,errors=remount-ro 0       1
UUID=c1255394-4c42-430a-b9cc-aaddfb024445 none            swap    sw              0       0
UUID=94f0d5db-c525-4f52-bdce-1f93652bc0b1 /var/           ext3    relatime 0       1
/dev/scd0       /media/cdrom0   udf,iso9660 user,noauto,exec,utf8 0       0

In the above example, the first UUID cooresponds to /dev/md0, the second to /dev/md1, and so on. Find the UUIDs with:

/lib/udev/vol_id /dev/md0

If your distribution uses a newer udev, you may not have the vol_id command. Use:

/sbin/blkid /dev/md0

If your /etc/fstab has “/dev/sda1″ entries like it’s a bit easier. Just change them to /dev/md0 and so on:

proc            /proc           proc    defaults        0       0
/dev/md0        /               ext3    relatime,errors=remount-ro 0     1
/dev/md1        none            swap    sw              0       0
/dev/md2        /var/           ext3    relatime        0       1
/dev/scd0       /media/cdrom0   udf,iso9660 user,noauto,exec,utf8 0       0

Now, while still in the chroot, edit /etc/mdadm/mdadm.conf to list each RAID array:

DEVICE partitions
ARRAY /dev/md0 UUID=126a552f:57b18c5b:65241b86:4f9faf62
ARRAY /dev/md1 UUID=ca493308:7b075f97:08347266:5b696c99
ARRAY /dev/md2 UUID=a983a59d:181b32c1:bcbb2b25:39e64cfd
MAILADDR root

Find the UUID of each RAID array, which is not the same as the UUID of the filesystem on it (!), using mdadm:

mdadm --detail /dev/md0 | grep UUID

Now, rebuild your current kernel’s initramfs image:

update-initramfs -u

Or all of them:

update-initramfs -u -k all

Step 6: Reboot on the RAID arrays on drive 2

Now we’re ready to reboot. First, exit the chroot and power off the machine cleanly. You have three options:

1. If your BIOS allows you to select which drive to boot from, elect to boot from drive 2.
2. Swap drives 1 and 2 so drive 2 becomes /dev/sda, and restart
3. Use a USB recovery stick to boot from drive 2

When the system restarts, you should reboot on the new RAID drive. Make sure:

df -h

Filesystem            Size  Used Avail Use% Mounted on
/dev/md0              9.2G  1.9G  6.9G  21% /
varrun                498M   56K  498M   1% /var/run
varlock               498M     0  498M   0% /var/lock
udev                  498M   68K  498M   1% /dev
devshm                498M     0  498M   0% /dev/shm
lrm                   498M   39M  459M   8% /lib/modules/2.6.24-19-generic/volatile
/dev/md2               63G  130M   60G   1% /home

Step 7: Repartition drive 1 and add it to RAID

Finally, we add the old drive into the array. Assuming you chose option 1 and didn’t swap the drive’s cables:

sfdisk -d /dev/sdb | sfdisk --force /dev/sda

If you receive a warning message here, reboot now.

Re-reading the partition table ...
BLKRRPART: Device or resource busy
The command to re-read the partition table failed
Reboot your system now, before using mkfs

If not, continue on:

mdadm /dev/md0 --grow -n 2
mdadm /dev/md1 --grow -n 2
mdadm /dev/md2 --grow -n 2
mdadm /dev/md0 --add /dev/sda1
mdadm /dev/md1 --add /dev/sda2
mdadm /dev/md2 --add /dev/sda3

If you chose option 2 and your drives now have a different ordering, swap “/dev/sda” and “/dev/sdb” everywhere above.

The RAID array will now rebuild. To check its progress:

cat /proc/mdstat

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sdb1[1] sda1[0]
      9767424 blocks [2/2] [UU]

md1 : active raid1 sdb1[1] sda1[0]
      1951808 blocks [2/2] [UU]

md2 : active raid1 sdb2[2] sda2[0]
      68693824 blocks [2/1] [U_]
      [>....................]  recovery =  0.4% (16320/68693824) finish=22.4min speed=8160K/sec

unused devices:

Once the array is finished rebuilding, reinstall grub on the new drive. We’ll do both, for good measure.

grub-install /dev/sda
grub-install /dev/sdb

You should now be able to reboot without either drive, and your system will come up cleanly. If you ever need to replace a failed drive, remove it, use step 7 above to clone the partition scheme to the new drive and add it to the array.

Gotchas:

If you are having issues with your RAID setup, especially if you have an older RAID setup or older release already, you might need these packages:

apt-get install lvm2 evms dmsetup