I run CyanogenMod 7 on my tablet, which means I have root and the rsync binary. I assume you do too. I also assume you have a working BackupPC installation, and are familiar using SSH key authentication and rsync as a BackupPC transfer method. If not, this is not the time to figure it out. Get it working on a normal Linux backup target first, then come back here.

First, install an SSH server on your Android device. I recommend you use QuickSSHd, but SSHDroidPro works as well. QuickSSHd has a nicer GUI and can run quietly (without persistent notification) in the background. Both are paid apps.

Open QuickSSHd, check “Password”, and enter a password for your root user. Then find the device’s IP address (displayed in the app), and verify that you can SSH in. Assuming its IP address is 192.168.0.100:

ssh root@192.168.0.100

QuickSSHD for Android
# 

Both QuickSSHd and SSHDroidPro are really just GUIs wrapped around dropbear, so you can (hopefully) trust the cryptography to be implemented correctly. Once you login, verify you have root access, and install your SSH keys.

# id
uid=0(root) gid=0(root) groups=0(root)
# pwd
/data/data/com.teslacoilsw.quicksshd/home
# umask 077
# mkdir .ssh
# cat > .ssh/authorized_keys

Now paste in your SSH keys. I have one for myself, and one for my BackupPC server:

ssh-rsa AAAABjkkgGSukkA....Yt/Ew== tyler@laptop
ssh-rsa AAAABAADsUXeWwh....wkq9F== backuppc@backup-server

And press Ctrl-D to close the file. Finally, set them to be owned by the QuickSSHd app user. This varies between devices, so first find it:

# ls -la /data/data/com.teslacoilsw.quicksshd/home
drwxrwx--x    3 app_84   app_84        4096 Jun  2 12:10 .
drwxr-xr-x    7 app_84   app_84        4096 Jun  9 17:16 ..
drwxrwx--x    2 app_84   app_84        4096 May 29 20:36 .ssh

Here the user is “app_84″, so:

chown app_84:app_84 .ssh/authorized_keys
chmod 600 .ssh/authorized_keys

Now open QuickSSHd, check “Shared keys”, and uncheck “Password” (optional, for enhanced security). You should now see your SSH keys listed by comment in the main app window. If you didn’t change the ownership on the authorized_keys file, they’ll still work for login, but won’t be visible in the app.

In QuickSSHd, Menu / Settings, set the options “Keep Awake” to on, to prevent the phone from sleeping while clients are connected; “Start at Boot” to on; and “Show Notification” to off, so that SSH can run quietly in the background all the time. Contrary to popular belief, a background process on Android doesn’t necessarily consume battery time. A sleeping SSH server does nothing until something tries to connect to it. So your battery is safe, as long as you don’t have a script kiddie roommate, constantly brute-forcing SSH logins against your phone.

Now test that you can login from the “backuppc” user of your BackupPC server. If so, you’re ready to set up this device in BackupPC.

Create a new host, named for your Android device. In the device’s config, go the “Xfer” tab. Set:

• RsyncShareName = /
• BackupFilesOnly = /data/, /mnt/asec/, /mnt/emmc/, and /mnt/sdcard/
• RsyncClientPath = /system/xbin/rsync (may be specific to CyanogenMod, check from the shell first)
• Optional: Any excludes for files you don’t need to back up. Here, I’ve set BackupFilesExclude to ignore Music and Videos, since I also have those files on my PC.

Here is my configuration:

Why backup only those directories?

• /data/, for all installed applications’ apk files and their configurations.
• /mnt/asec/, for APP2SD-compatible apps’ apk files.
• /mnt/emmc/, because my tablet mounts additional internal storage there.
• /mnt/sdcard/, for obvious reasons.

Why not backup the root directory? Because it’s a rootfs, which is just tempfs. Even if you wanted to back it up, you can’t restore to it. You only need a few subdirectories.

Why not backup /system/? Because I haven’t remounted /system/ read-write, nor changed the firmware from the regular CyanogenMod install. For disaster recovery, I plan to reinstall CyanogenMod, then restore the lost apps and their configurations.

Once done with the Xfer settings, test it by starting a full backup of your Android device. If you get the dread “unable to read 4 bytes” message, check that your SSH authentication is actually working. My server ran the first full backup in 20 minutes, and now runs nightly in about one minute.

Aside: SSH logins on Android default to a minimal bourne-type shell. It has no colourisation and no tab-completion. If you plan on doing anything else from the command line, you may find it convenient to start bash immediately after login.

ssh -t root@192.168.0.100 'cd / ; echo ; bash'

Minor updates to the repo for Ubuntu 10.10 Maverick Meerkat:

• BackupPC 3.2.0 has been added. This is the same package from Debian Sid, and works on Lucid as well. I don’t recommend a non-LTS release for your backup solution, but if your hardware requires a newer kernel, you might need this. I’ve tested it myself.
• Clementine 0.5.3 has been removed. You can now find a newer release in maverick-backports.

BackupPC 3.2.0 is now in the repository for both Ubuntu Lucid and Hardy. The Lucid version is taken directly from Debian sid. The Hardy version is backported from the same package, with edits to make the graphs’ fonts work (older rrdtool on hardy) and depend on samba-common instead of samba-common-bin (changed after hardy).

This version fixes the “Download Zip archive” issue in Ubuntu lucid. You no longer have to manually install an older Archive::Zip Perl module.

It’s also got a shiny new favicon, courtesy yours truly. That was a hard 10 minutes’ work with GIMP, right there. Astute readers of changelogs may believe that the developers didn’t use the favicon I submitted. But according to md5sum they did, and that is awesome.

This problem is easy to verify. Login to your server and inspect one of its RRD files.

root@backuppc:~# cd /var/lib/backuppc/log
root@backuppc:/var/lib/backuppc/log# rrdtool info pool.rrd
ERROR: This RRD was created on other architecture

To correct it, export the RRD files to XML format before * upgrading the server:

rrdtool dump pool.rrd > pool.xml

* What if you’ve already upgraded? You can use any other server with the same architecture. Just copy your RRD files to it and run the same commands there.

After upgrading, import your XML files into the new RRD format:

rrdtool restore -f pool.xml pool.rrd

The above example is for BackupPC, but these commands work for any RRD file. If you have a lot of files (such as on a Cacti server), here is a handy loop:

for i in *.rrd; do rrdtool dump $i > $i.xml ; done

And another for restoring:

for i in *.xml; do rrdtool restore -f $i `basename $i .xml`.rrd ; done

Don’t forget to set expected file ownership and permissions (chown --reference and chmod --reference are your friends), and remove the XML files afterward.

Note to developers: when releasing new versions of your library, try to avoid breaking its core functionality.

If you try decompressing a corrupted file, you’ll see a message like this:

tyler@laptop:~$ unzip restore.zip
Archive:  restore.zip
  End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.  In the
  latter case the central directory and zipfile comment will be found on
  the last disk(s) of this archive.
unzip:  cannot find zipfile directory in one of restore.zip or
        restore.zip.zip, and cannot find restore.zip.ZIP, period.

I posted a solution on the mailing list. I’ve since uploaded an older version (1.18-1) of this package to my repository, so the process is even more streamlined.

If you are using my repo:
apt-get install libarchive-zip-perl=1.18-1
echo "libarchive-zip-perl hold" | dpkg --set-selections

If you aren’t using my repo but want a quick fix:
wget http://www.tolaris.com/apt/pool/main/liba/libarchive-zip-perl/libarchive-zip-perl_1.18-1_all.deb
dpkg -i libarchive-zip-perl_1.18-1_all.deb
echo "libarchive-zip-perl hold" | dpkg --set-selections

I wrote a post 18 months ago on moving a Linux root partition to software RAID. Moving to LVM shares a lot of the same problems. The differences are:

• We’re not touching the root (or /boot) filesystem. Therefore we don’t have to deal with BIOS or GRUB. In fact, this entire process can be done without rebooting.
• We must copy the filesystem bit-for-bit instead of using file-aware tools like rsync, cp, or tar. This is due to the design of the BackupPC storage pool.

If you have another disk to use as a swap space, this problem is easy to solve. Simply unmount your backuppc partition and use dd to clone the data to the new drive. Then make create a logical volume on the old partition and copy it back.

If you don’t have another disk, you must convert the disk layout in place. The instructions in this post can be used to move any filesystem to LVM; just disregard the parts specific to BackupPC. I assume that your server’s drive and filesystem layout is as described in my last post:

• /dev/sda provides the root filesystem (and possibly others). We’re not concerned about it here.
• /dev/sdb is a RAID array with one partition. It does not yet use LVM, and contains an ext3 filesystem mounted on /var/lib/backuppc.

The general plan is:

1. Reduce filesystem usage on /var/lib/backuppc to 49% or less
2. Resize the filesystem and partition to 49%
3. Create a second partition and logical volume with the free space
4. Copy the filesystem to the logical volume
5. Add the first partition to the logical volume
6. Expand the logical volume and filesystem
7. Mount the logical volume

Warning: this is a multi-day process with many long delays for large data transfers. I’ll identify the delays as we go along. Never interrupt a filesystem operation unless you know exactly what you are doing. I recommend executing all long-running processes in a GNU screen session in case your network connection is interrupted.

Step 1: Reduce filesystem usage on /var/lib/backuppc to 49% or less

If your BackupPC pool filesystem is larger than 49%, you must reduce disk usage.

1. Remove old backups. Reduce the BackupPC variables $Conf{FullKeepCnt} and $Conf{IncrKeepCnt} to keep fewer backups. Alternately, use a script like BackupPC_deleteBackup.sh to delete old ones manually.
2. Set $Conf{BackupPCNightlyPeriod} = 1
3. Wait for the next BackupPC_Nightly run. BackupPC will purge old backups and clean the pool of unnecessary files. This may take several days, but during this time you can perform backups as normal.

Do not proceed until filesystem usage is below 49%. Verify this with df.

Step 2: Resize the filesystem and partition to 49%

You may wish to read this guide first. It is possible to use the parted “resize” function for this, but it doesn’t work for all filesystems. We’ll do it manually.

1. Stop BackupPC. From this point no new backups will be kept until the entire process is finished, which may be several days.
   /etc/init.d/backuppc stop
2. Unmount the partition
   umount /var/lib/backuppc
3. Get the block size and block count for the partition, and calculate the new size in K.
   tune2fs -l /dev/sdb1 | grep Block

   Block count:              2620595
   Block size:               4096

   I’ve used a 10 GB partition in this example, and we now know that this filesystem consists of 2620595 blocks of 4K size. Calculate the new size as follows:

   new size = (current size in blocks) * (block size in K) * 0.49

   Round this number down. Our new size will be 2620595 * 4 * 0.49 = 5136366 K.

4. Check the filesystem. This may take several hours or days.
   fsck -f /dev/sdb1
5. Resize the filesystem.
   resize2fs /dev/sdb1 5136366K

   resize2fs 1.40.8 (13-Mar-2008)
   Resizing the filesystem on /dev/sdb1 to 1284091 (4k) blocks.
   The filesystem on /dev/sdb1 is now 1284091 blocks long.

6. Resize the partition by deleting it and creating a new one of 5136366K.
   fdisk /dev/sdb

   Command (m for help): p
   
   Disk /dev/sdb: 10.7 GB, 10737418240 bytes
   255 heads, 63 sectors/track, 1305 cylinders
   Units = cylinders of 16065 * 512 = 8225280 bytes
   Disk identifier: 0x0005eeec
   
      Device Boot      Start         End      Blocks   Id  System
   /dev/sdb1               1        1305    10482381   83  Linux
   
   Command (m for help): d 1
   Selected partition 1
   
   Command (m for help): n
   Command action
      e   extended
      p   primary partition (1-4)
   p
   Partition number (1-4): 1
   First cylinder (1-1305, default 1):
   Using default value 1
   Last cylinder or +size or +sizeM or +sizeK (1-1305, default 1305): +5136366K
   
   Command (m for help): p
   
   Disk /dev/sdb: 10.7 GB, 10737418240 bytes
   255 heads, 63 sectors/track, 1305 cylinders
   Units = cylinders of 16065 * 512 = 8225280 bytes
   Disk identifier: 0x0005eeec
   
      Device Boot      Start         End      Blocks   Id  System
   /dev/sdb1               1         640     5140768+  83  Linux
   
   Command (m for help): w
   The partition table has been altered!
   
   Calling ioctl() to re-read partition table.
   Syncing disks.

Step 3: Create a second partition and logical volume with the free space

1. If you haven’t already, install LVM.
   apt-get install lvm2
2. Use fdisk again to create a second partition. You can combine this step with the last one. Hit enter at the prompts to use all available space, and change the new partition to type 8e, Linux LVM.
   fdisk /dev/sdb

   Command (m for help): n
   Command action
      e   extended
      p   primary partition (1-4)
   p
   Partition number (1-4): 2
   First cylinder (641-1305, default 641):
   Using default value 641
   Last cylinder or +size or +sizeM or +sizeK (641-1305, default 1305):
   Using default value 1305
   
   Command (m for help): p
   
   Disk /dev/sdb: 10.7 GB, 10737418240 bytes
   255 heads, 63 sectors/track, 1305 cylinders
   Units = cylinders of 16065 * 512 = 8225280 bytes
   Disk identifier: 0x0005eeec
   
      Device Boot      Start         End      Blocks   Id  System
   /dev/sdb1               1         640     5140768+  83  Linux
   /dev/sdb2             641        1305     5341612+  83  Linux
   
   Command (m for help): t
   Partition number (1-4): 2
   Hex code (type L to list codes): 8e
   Changed system type of partition 2 to 8e (Linux LVM)
   
   Command (m for help): p
   
   Disk /dev/sdb: 10.7 GB, 10737418240 bytes
   255 heads, 63 sectors/track, 1305 cylinders
   Units = cylinders of 16065 * 512 = 8225280 bytes
   Disk identifier: 0x0005eeec
   
      Device Boot      Start         End      Blocks   Id  System
   /dev/sdb1               1         640     5140768+  83  Linux
   /dev/sdb2             641        1305     5341612+  8e  Linux LVM
   
   Command (m for help): w
   The partition table has been altered!
   
   Calling ioctl() to re-read partition table.
   Syncing disks.

3. The second partition must be equal to or larger than the first partition. If it is not, check your math or try resizing to a smaller percentage and try again.
4. Create a volume group and logical volume.
   pvcreate /dev/sdb2
vgcreate volgroup /dev/sdb2
lvcreate -l 100%VG -n backuppc volgroup

Step 4: Copy the filesystem to the logical volume

1. Use dd, the sysadmin’s oldest friend, to copy the filesystem bit-for-bit from the old partition to the new. This may take several hours or days.
   dd if=/dev/sdb1 of=/dev/volgroup/backuppc
2. If you get impatient during the copy, you can force dd to output the current progress. Find the process ID of the running dd, and send a USR1 signal.
   ps -ef | grep dd

   root      4786  4642 26 15:44 pts/0    00:00:01 dd if /dev/sdb1 of /dev/volgroup/backuppc

   kill -USR1 4786

   18335302+0 records in
   18335302+0 records out
   9387674624 bytes (9.4 GB) copied, 34.6279 seconds, 271 MB/s

3. A neater way to do this is:
   dd if=/dev/sdb1 of=/dev/volgroup/backuppc &
pid=$!
kill -USR1 $pid
4. When dd finishes you will see one last progress report which should total the size of /dev/sdb1.

Step 5: Add the first partition to the logical volume

We’ve copied our data to the logical volume, so we no longer need the original partition. Add it to the volume group.

1. Use fdisk to change partition 1 to type 8e.
   fdisk /dev/sdb

   Command (m for help): p
   
   Disk /dev/sdb: 10.7 GB, 10737418240 bytes
   255 heads, 63 sectors/track, 1305 cylinders
   Units = cylinders of 16065 * 512 = 8225280 bytes
   Disk identifier: 0x0005eeec
   
      Device Boot      Start         End      Blocks   Id  System
   /dev/sdb1               1         640     5140768+  83  Linux
   /dev/sdb2             641        1305     5341612+  8e  Linux LVM
   
   Command (m for help): t
   Partition number (1-4): 1
   Hex code (type L to list codes): 8e
   Changed system type of partition 1 to 8e (Linux LVM)
   
   Command (m for help): p
   
   Disk /dev/sdb: 10.7 GB, 10737418240 bytes
   255 heads, 63 sectors/track, 1305 cylinders
   Units = cylinders of 16065 * 512 = 8225280 bytes
   Disk identifier: 0x0005eeec
   
      Device Boot      Start         End      Blocks   Id  System
   /dev/sdb1               1         640     5140768+  8e  Linux LVM
   /dev/sdb2             641        1305     5341612+  8e  Linux LVM
   
   Command (m for help): w
   The partition table has been altered!
   
   Calling ioctl() to re-read partition table.
   Syncing disks.

2. Add it to the volume group.
   pvcreate /dev/sdb1
vgextend volgroup /dev/sdb1

Step 6: Expand the logical volume and filesystem

We’ve moved the data to a logical volume and added all the disk space to the volume group. Now we expand the logical volume and filesystem to use the available space.

1. Extend the logical volume. I advise using only 75% so you can use LVM snapshots.
   lvextend -l 75%VG /dev/volgroup/backuppc
2. Resize the filesystem to use the entire logical volume. This may take several hours or days.
   resize2fs /dev/volgroup/backuppc
3. If at any point you’ve mounted the filesystem, you’ll have to run a new fsck before resizing. This will cause another long delay, so try to avoid it.

Step 7: Mount the logical volume

We’re now ready to mount the new logical volume on /var/lib/backuppc and restart BackupPC.

1. Edit /etc/fstab and replace any references to /dev/sdb1 with /dev/volgroup/backuppc. If your fstab uses UUIDs instead of explicit device paths, you shouldn’t have to change anything.
2. Mount the filesystem.
   mount /var/lib/backuppc
3. Restart BackupPC
   /etc/init.d/backuppc start

You have successfully migrated to LVM!

The problem with this design is that conventional duplication/archival tools like rsync and tar can’t be used to duplicate the cpool in a reasonable period of time. Any file with more than one hardlink must be tracked and restored correctly on the destination host or filesystem. Copying my 700 GB pool with rsync between two identical disks with rsync would take a minimum of 3 days. Copying the same amount of raw data would take only a few hours.

So we can’t use filesystem-aware tools to backup a BackupPC server’s pool. Instead, we must make a bit-for-bit copy of the disk image.

One suggestion I’ve seen is to make the cpool part of a RAID1 array of two disks. When you want to make a backup, pull one disk and replace it. Or make it a 3-disk mirror with a removable drive, and periodically swap it out and carry it elsewhere. This is a sound solution, but it is still sneakernet.

What I want is a solution that:

1. Duplicates the entire BackupPC storage to a second server,
2. in a matter of hours not days,
3. over the network,
4. such that the secondary server can quickly replace the primary server if it fails,
5. without interrupting normal operations on the disk, including BackupPC itself.

To accomplish this, we build the BackupPC storage on LVM, and use LVM snapshots and netcat to copy the disk image.

My BackupPC server uses a 3ware hardware RAID card with two arrays:

1. /dev/sda is a RAID1 of two disks, and provides the / and /var filesystems.
2. /dev/sdb is a RAID5 array of 5 disks, and provides the physical volume for the backuppc logical volume and filesystem

The server’s filesystem looks like this:

root@backuppc:~$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             9.2G  1.2G  8.0G  13% /
/dev/sda3              63G   21G   41G  34% /var
/dev/mapper/volgroup-backuppc
                      1.4T  711G  644G  53% /var/lib/backuppc

If you’re planning to install BackupPC on a new server, this is easy to prepare. First, install the latest Ubuntu LTS server (I’m still using 8.04 “Hardy Heron”) on /dev/sda, using / and /var partitions as above. It is possible to configure LVM through the installer UI, but I’m not going to cover that here.

Once the base OS is installed, login via SSH and prepare the LVM setup. LVM is discussed in excellent detail elsewhere. I strongly urge you to read this How-To and play with LVM a little before building the final system.

All commands are run as root. Use sudo if you prefer.

1. Create one partition of type 8e (Linux LVM) on the RAID disk.
   fdisk /dev/sdb

   Command (m for help): o
   Building a new DOS disklabel with disk identifier 0xf2704d76.
   Changes will remain in memory only, until you decide to write them.
   After that, of course, the previous content won't be recoverable.
   
   Command (m for help): n
   Command action
      e   extended
      p   primary partition (1-4)
   p
   Partition number (1-4): 1
   First cylinder (1-243147, default 1):
   Using default value 1
   Last cylinder, +cylinders or +size{K,M,G} (1-243147, default 243147):
   Using default value 243147
   
   Command (m for help): t
   Selected partition 1
   Hex code (type L to list codes): 8e
   Changed system type of partition 1 to 8e (Linux LVM)
   
   Command (m for help): p
   
   Disk /dev/sdb: 1999.9 GB, 1999957393408 bytes
   255 heads, 63 sectors/track, 243147 cylinders
   Units = cylinders of 16065 * 512 = 8225280 bytes
   Disk identifier: 0x92cf8945
   
   Device Boot      Start         End      Blocks   Id  System
   /dev/sdb1          1      243147  1953078277   8e  Linux LVM
   
   Command (m for help): w
   The partition table has been altered!
   
   Syncing disks.

2. Prepare the physical volume and volume group.
   pvcreate /dev/sdb1
vgcreate volgroup /dev/sdb1
3. Create a logical volume which is 75% of the available space. We’ll reserve the remaining space for the snapshot.
   lvcreate -l '75%VG' -n backuppc volgroup
4. Create a filesystem on the logical volume. Some prefer XFS, but I’m happy with ext3 (with an eye to upgrading to ext4 later).
   mkfs.ext3 /dev/volgroup/backuppc

5. Add the new logical volume to /etc/fstab.
   vi /etc/fstab
   Add a new line. We mount with the most efficient options since BackupPC doesn’t need them.

   /dev/volgroup/backuppc /var/lib/backuppc ext3 noatime 0 2

6. Now mount it for the first time.
   mkdir /var/lib/backuppc
mount /var/lib/backuppc
7. Finally, install BackupPC. This will also install Apache2, Perl, and a mail server.
   apt-get install backuppc

You now have a working BackupPC system using LVM for storage. Now we create an LVM snapshot.

Why use LVM snapshots? Because we must take a copy of the filesystem as it exists at one moment in time. If we install directly on /dev/sdb1 and then try to copy the filesystem while it is mounted, the copy will be inconsistent. Some data blocks will represent the time we started copying, others the time we ended, or anything in between. There is a disk performance penalty to using snapshots, as this creates a copy-on-write fork of the filesystem from the moment the snapshot is taken. The best practice is to create the snapshot when the server is idle, copy it to the secondary server, and then remove it immediately after.

Create the snapshot using all remaining free space.
lvcreate -l '100%free' -s -n backuppcsnapshot /dev/volgroup/backuppc

If you are curious, you can now mount the logical volume /dev/volgroup/backuppcsnapshot and compare the difference between it and the original filesystem. But we don’t need to do that. We just need to copy it to our secondary server.

The secondary server should be configured exactly like the first. At minimum, it must have a partition or logical volume that is the same size or greater than /dev/volgroup/backuppc on the primary server. To copy it, we’ll use dd and SSH.

I previously recommended using netcat to perform this step. However, I’ve had a lot of difficulty getting cron, SSH, dd, and netcat to behave together when starting the listener on the destination server. I’ve also found that SSH is just as fast as using netcat, which is surprising given that encryption should be slower than transmitting the data in the clear.

Now we copy the data from primary to secondary via SSH. On the primary:
dd if=/dev/volgroup/backuppcsnapshot | ssh root@192.168.0.2 "dd of=/dev/volgroup/backuppc"

This reads the logical volume block for block and sends it to the secondary server on 192.168.0.2, via SSH. The secondary server receives and writes it to the logical volume.

One advantage of using dd is that you can easily check progress:

root@backuppc:~# ps -ef | grep dd
root      4958  4495  0 18:39 pts/0    00:00:03 dd if=/dev/volgroup/backuppcsnapshot
root@backuppc:~# kill -USR1 4958

On the terminal where dd is running, you’ll see this.

126689+0 records in
126689+0 records out
64864768 bytes (65 MB) copied, 9.97271 s, 6.5 MB/s

You may wish to use bzip to compress the data after the first dd operation and decompress it before the second, or use ssh -C. Using compression will save bandwidth at the cost of CPU usage. In my tests, a raw copy gave sustained speeds of 200 mbit over gigabit Ethernet. Using bzip2, throughput was reduced to 20 mbit. Therefore I do not recommend using compression if copying over your own gigabit LAN.

As a last step, archive the BackupPC configuration whenever you clone the storage pool. Since you don’t want the secondary to start running backups, put it in a safe place. On the secondary:

mkdir /etc/backuppc/backup/

On the primary, after copying the snapshot:

rsync -av --delete /etc/backuppc/ root@192.168.0.2:/etc/backuppc/backup/

We now have a complete BackupPC server with LVM storage and a means of completely backing up to a secondary server, over the network, without interrupting the operation of the primary. But what if you have an existing BackupPC pool directly on disk partition? You’ll need to migrate it to LVM without losing the data. I’ll discuss how to do that in my next post.

Updated 2010-09-17: removed directions for netcat and recommended SSH method.

Bacula is very complicated. That complication provides flexibility – if you want your level-0 backups to run on Tuesdays at 1900 and Thursdays at 22:30, and to exclude some files on Wednesdays only, it can – but it also makes it brittle. One daemon hangs for any reason, and half the backups never complete. It scales well to very large enterprise sizes, but I don’t really need that yet.

What Bacula does really well is dump data to tape. I don’t use tape. I use a RAID array. When you dump data to tape, you do it in series. But when you dump data to disk, you can just as easily do it in parallel. The process of calculating checksums and compressing files limits the rate at which you actually transfer those files to the backup store. With most of my servers, that turns out to be about 20 mbit/sec at max. But my servers are typically connected via gigabit LAN, so why not backup several at the same time? Further, my backup window is limited to 12 hours at night so as to avoid affecting normal traffic during the day. If you are running backups in series and one takes 4 hours, the other backups run the risk of pushing into the morning work hours.

How to install and configure BackupPC is already well documented elsewhere. Where my install differs is that we don’t run Windows, so we don’t have to deal with backing it up. All of my defaults are configured to use rsync and expect Linux filesystems.

What BackupPC does really well is just back stuff up, to disk, whenever it can. It is possible to force exactly when backups run, but in general the system isn’t designed that way. I tried to force everything to kick off at 20:00 UTC nightly, and run until finished. But by default, BackupPC is configured to try to backup every hour on the hour, and to run the next backup 24 hours after the last one finished. I soon realised that it was best to do it that way. When a server is down or the backup fails, BackupPC tries again every hour until it succeeds. That is far more robust than “the backup failed last night, so we have to wait until next night”.

So what about not running backups during the day? It can do that too. You can specify exactly which hours to wake up and perform pending jobs:

$Conf{WakeupSchedule} = [ '19', '20', '21', '22', '23', '0', '1', '2', '3', '4', '5', '6' ];

But a better way is to run every hour anyway, and instead use blackout periods:

$Conf{WakeupSchedule} = [ '12', '13', '14', '15', '16',
'17', '18', '19', '20', '21', '22', '23', '0', '1',
'2', '3', '4', '5', '6', '7', '8', '9', '10', '11' ];

$Conf{BlackoutBadPingLimit} = '3';
$Conf{BlackoutGoodCnt} = '7';
$Conf{BlackoutPeriods} = [ {
    'weekDays' => [ '0', '1', '2', '3', '4', '5', '6' ],
    'hourBegin' => '6.5'
    'hourEnd' => '18.5',
  } ];

Now jobs run every hour. But for any host that has stayed online for 7 days (tested by ping, where 3 are sent before declaring failure), backups will not start between 6:30 and 18:30. Unfortunately all times are hours, so you must use decimal HH.HH instead of HH:MM. You’ll only make a few mistakes before you get used to it.

The advantage to the blackout system is that it backs up a recently failed server within an hour of it coming online again, even during the day, while restricting backups of functioning servers to the later hours. Servers just added to the system may also back up during the day because they don’t have 7 good pings yet. The easy solution is to add new servers at night. Backups tend to stay near to when they last ran, but can move within the normal job period.

While backup times can move around, every server is still backed up every 24 hours, at some point during the night. Now that I’ve run this for 5 weeks, on 47 servers, I’m very pleased with it. It has proven more reliable than Bacula, and the disk usage is far, far better. BackupPC compresses and pools stored files, and uses hashes to determine if a file needs to be archived again. If you have multiple copies of the same file, across all servers, you only have one on disk on the BackupPC server. Backing up every copy of /usr/ suddenly makes no difference at all. From the web interface:

There are 47 hosts that have been backed up, for a total of:
    * 177 full backups of total size 1331.97GB (prior to pooling and compression),
    * 395 incr backups of total size 783.92GB (prior to pooling and compression).

Pool is 337.28GB comprising 4081543 files and 4369 directories (as of 2009-10-08 13:46),
Pool hashing gives 543 repeated files with longest chain 21,
Nightly cleanup removed 63257 files of size 37.75GB (around 2009-10-08 13:46),
Pool file system was recently at 29% (2009-10-08 21:25), today's max is 29% (2009-10-08 12:00)
and yesterday's max was 29%. 

BackupPC pool statistics

That’s 5 weeks of weekly level-zeroes plus the last 14 daily incrementals. The same data in Bacula for only 27 servers came to 900 GB. BackupPC is configured to keep backups going back 5 months; at best I could only manage a month of Bacula backups for the same period.

What about data throughput? Bacula topped out at about 20 mbit for even gigabit-connected hosts, and it could only run in serial (unless I configured multiple “storage devices” (directories), but there is no way to pool them for dynamic use. Allowing only 3 simultaneous backups, the server routinely exceeds 60 mbit of traffic during the first hour of the run.

Network statistics

Unfortunately I don’t have useful Cacti graphs from the time we ran Bacula, but it was at best 80% as fast for only one backup. And of course it could run only one.

What else does BackupPC have over Bacula?

• A good web interface. Yes, webacula exists, but you are still limited to editing config files and using an interactive terminal interface to do anything useful. With BackupPC you can use any standards-compliant browser to review statistics and logs about recent backups, edit the config, browse backups in tree format and restore from them, and start additional backups outside of the normal schedule. You can also grant users the ability to browse and schedule backups for specific servers or desktops, but not see any others.
• Inline help. When editing BackupPC’s config in the web GUI, all configuration variables are hyperlinks to the HTML documentation.
• Several ways to restore. You can directly restore file to the server by the same transport that you use to backup (rsync, ssh+tar, SMB). You can also download a ZIP or tar.gz through your browser.
• No database dependency. The filesystem is the database. Files are stored in disk trees, and logs are text on disk. I’m not against using MySQL, but it adds one more point of failure.
• No client daemon. That’s right, nothing to install on the clients. Just add the server’s SSH key to your root user, and rsync does the rest. If you think that’s a security risk, it is. Any other backup system has the same vulnerability. I suggest using key access limits.

I’ve tested basic restores and bare-metal ones. So far there are no issues with BackupPC, and I’m much happier using it. It is simpler, less prone to failure, and less likely to require user intervention. It may not scale to 100 distributed backup stores with tape drives, but it is excellent for the small-to-medium enterprise.