Bacula is very complicated. That complication provides flexibility – if you want your level-0 backups to run on Tuesdays at 1900 and Thursdays at 22:30, and to exclude some files on Wednesdays only, it can – but it also makes it brittle. One daemon hangs for any reason, and half the backups never complete. It scales well to very large enterprise sizes, but I don’t really need that yet.

What Bacula does really well is dump data to tape. I don’t use tape. I use a RAID array. When you dump data to tape, you do it in series. But when you dump data to disk, you can just as easily do it in parallel. The process of calculating checksums and compressing files limits the rate at which you actually transfer those files to the backup store. With most of my servers, that turns out to be about 20 mbit/sec at max. But my servers are typically connected via gigabit LAN, so why not backup several at the same time? Further, my backup window is limited to 12 hours at night so as to avoid affecting normal traffic during the day. If you are running backups in series and one takes 4 hours, the other backups run the risk of pushing into the morning work hours.

How to install and configure BackupPC is already well documented elsewhere. Where my install differs is that we don’t run Windows, so we don’t have to deal with backing it up. All of my defaults are configured to use rsync and expect Linux filesystems.

What BackupPC does really well is just back stuff up, to disk, whenever it can. It is possible to force exactly when backups run, but in general the system isn’t designed that way. I tried to force everything to kick off at 20:00 UTC nightly, and run until finished. But by default, BackupPC is configured to try to backup every hour on the hour, and to run the next backup 24 hours after the last one finished. I soon realised that it was best to do it that way. When a server is down or the backup fails, BackupPC tries again every hour until it succeeds. That is far more robust than “the backup failed last night, so we have to wait until next night”.

So what about not running backups during the day? It can do that too. You can specify exactly which hours to wake up and perform pending jobs:

$Conf{WakeupSchedule} = [ '19', '20', '21', '22', '23', '0', '1', '2', '3', '4', '5', '6' ];

But a better way is to run every hour anyway, and instead use blackout periods:

$Conf{WakeupSchedule} = [ '12', '13', '14', '15', '16',
'17', '18', '19', '20', '21', '22', '23', '0', '1',
'2', '3', '4', '5', '6', '7', '8', '9', '10', '11' ];

$Conf{BlackoutBadPingLimit} = '3';
$Conf{BlackoutGoodCnt} = '7';
$Conf{BlackoutPeriods} = [ {
    'weekDays' => [ '0', '1', '2', '3', '4', '5', '6' ],
    'hourBegin' => '6.5'
    'hourEnd' => '18.5',
  } ];

Now jobs run every hour. But for any host that has stayed online for 7 days (tested by ping, where 3 are sent before declaring failure), backups will not start between 6:30 and 18:30. Unfortunately all times are hours, so you must use decimal HH.HH instead of HH:MM. You’ll only make a few mistakes before you get used to it.

The advantage to the blackout system is that it backs up a recently failed server within an hour of it coming online again, even during the day, while restricting backups of functioning servers to the later hours. Servers just added to the system may also back up during the day because they don’t have 7 good pings yet. The easy solution is to add new servers at night. Backups tend to stay near to when they last ran, but can move within the normal job period.

While backup times can move around, every server is still backed up every 24 hours, at some point during the night. Now that I’ve run this for 5 weeks, on 47 servers, I’m very pleased with it. It has proven more reliable than Bacula, and the disk usage is far, far better. BackupPC compresses and pools stored files, and uses hashes to determine if a file needs to be archived again. If you have multiple copies of the same file, across all servers, you only have one on disk on the BackupPC server. Backing up every copy of /usr/ suddenly makes no difference at all. From the web interface:

There are 47 hosts that have been backed up, for a total of:
    * 177 full backups of total size 1331.97GB (prior to pooling and compression),
    * 395 incr backups of total size 783.92GB (prior to pooling and compression).

Pool is 337.28GB comprising 4081543 files and 4369 directories (as of 2009-10-08 13:46),
Pool hashing gives 543 repeated files with longest chain 21,
Nightly cleanup removed 63257 files of size 37.75GB (around 2009-10-08 13:46),
Pool file system was recently at 29% (2009-10-08 21:25), today's max is 29% (2009-10-08 12:00)
and yesterday's max was 29%. 

BackupPC pool statistics

That’s 5 weeks of weekly level-zeroes plus the last 14 daily incrementals. The same data in Bacula for only 27 servers came to 900 GB. BackupPC is configured to keep backups going back 5 months; at best I could only manage a month of Bacula backups for the same period.

What about data throughput? Bacula topped out at about 20 mbit for even gigabit-connected hosts, and it could only run in serial (unless I configured multiple “storage devices” (directories), but there is no way to pool them for dynamic use. Allowing only 3 simultaneous backups, the server routinely exceeds 60 mbit of traffic during the first hour of the run.

Network statistics

Unfortunately I don’t have useful Cacti graphs from the time we ran Bacula, but it was at best 80% as fast for only one backup. And of course it could run only one.

What else does BackupPC have over Bacula?

• A good web interface. Yes, webacula exists, but you are still limited to editing config files and using an interactive terminal interface to do anything useful. With BackupPC you can use any standards-compliant browser to review statistics and logs about recent backups, edit the config, browse backups in tree format and restore from them, and start additional backups outside of the normal schedule. You can also grant users the ability to browse and schedule backups for specific servers or desktops, but not see any others.
• Inline help. When editing BackupPC’s config in the web GUI, all configuration variables are hyperlinks to the HTML documentation.
• Several ways to restore. You can directly restore file to the server by the same transport that you use to backup (rsync, ssh+tar, SMB). You can also download a ZIP or tar.gz through your browser.
• No database dependency. The filesystem is the database. Files are stored in disk trees, and logs are text on disk. I’m not against using MySQL, but it adds one more point of failure.
• No client daemon. That’s right, nothing to install on the clients. Just add the server’s SSH key to your root user, and rsync does the rest. If you think that’s a security risk, it is. Any other backup system has the same vulnerability. I suggest using key access limits.

I’ve tested basic restores and bare-metal ones. So far there are no issues with BackupPC, and I’m much happier using it. It is simpler, less prone to failure, and less likely to require user intervention. It may not scale to 100 distributed backup stores with tape drives, but it is excellent for the small-to-medium enterprise.

It all began when I accidentally copied /home/tyler/bin on my laptop to another workstation’s /bin. I have a lot of little scripts and doodads for my personal use in /home/tyler/bin, but none of them are named for real executables in /bin. So I think, “No big deal, just manually remove those scripts from /bin and no harm done.”

So I quickly listed the programs in bin, copied and pasted them into the other terminal with ‘rm’, and let fly.

Wait, was that /bin or /home/tyler/bin I just listed? Oh shit.

root@boned-workstation:/bin#
ls
bash: ls: command not found

When you’ve just deleted /bin, you’re screwed. You can’t do anything. Quickly rsync replacements over? Nope. How about a tarball? Nope. In fact you probably cannot even login, so don’t logout of the SSH session you’ve still got open. Booting from a Live CD/USB/DVD is about your only hope.

And then I remembered Bacula. The Bacula file daemon needs only itself to do its job, and it doesn’t even need that once the daemon is running. You could delete /usr/sbin/bacula-fd and it will still work unless you stop the daemon.

We don’t back up system files on workstations, just user files and the dpkg database. But Ubuntu workstations and servers run the same software. So I restored the previous night’s backup of /bin/ from the backup server itself to the affected workstation. Bacula’s restore process is quite flexible, and is happy to let you restore from any archive to any host it knows.

A quick rsync from a similar workstation to make sure all the binaries are there, and we’re saved!