It has been *0* days since the last ‘rm -rf *’

Sometimes running Bacula is a real pain. It’s too much infrastructure for someone who doesn’t use tape backups (we just use a big RAID store). But today it saved me.

It all began when I accidentally copied /home/tyler/bin on my laptop to another workstation’s /bin. I have a lot of little scripts and doodads for my personal use in /home/tyler/bin, but none of them are named for real executables in /bin. So I think, “No big deal, just manually remove those scripts from /bin and no harm done.”

So I quickly listed the programs in bin, copied and pasted them into the other terminal with ‘rm’, and let fly.

Wait, was that /bin or /home/tyler/bin I just listed? Oh shit.

root@boned-workstation:/bin#
ls
bash: ls: command not found

When you’ve just deleted /bin, you’re screwed. You can’t do anything. Quickly rsync replacements over? Nope. How about a tarball? Nope. In fact you probably cannot even login, so don’t logout of the SSH session you’ve still got open. Booting from a Live CD/USB/DVD is about your only hope.

And then I remembered Bacula. The Bacula file daemon needs only itself to do its job, and it doesn’t even need that once the daemon is running. You could delete /usr/sbin/bacula-fd and it will still work unless you stop the daemon.

We don’t back up system files on workstations, just user files and the dpkg database. But Ubuntu workstations and servers run the same software. So I restored the previous night’s backup of /bin/ from the backup server itself to the affected workstation. Bacula’s restore process is quite flexible, and is happy to let you restore from any archive to any host it knows.

A quick rsync from a similar workstation to make sure all the binaries are there, and we’re saved!

Tags: ,

  1. Craig’s avatar

    Very interesting that. I o really need play with Bacula at some stage.

    Thanks for the useful info. :)

    Reply

  2. Daryl’s avatar

    Have to admit, first I’ve heard of Bacula (gotta love the name), but it’s always nice when apps like this can pull victory out of the vicious jaws of inglorious defeat. I do love those… “omg whatever you do, *do not* shut down the ssh session” moments… =]

    Reply

  3. kjw’s avatar

    Bad sysadmin! No Donut!

    Reply

Reply to kjw Cancel reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.