It all began when I accidentally copied /home/tyler/bin on my laptop to another workstation’s /bin. I have a lot of little scripts and doodads for my personal use in /home/tyler/bin, but none of them are named for real executables in /bin. So I think, “No big deal, just manually remove those scripts from /bin and no harm done.”

So I quickly listed the programs in bin, copied and pasted them into the other terminal with ‘rm’, and let fly.

Wait, was that /bin or /home/tyler/bin I just listed? Oh shit.

root@boned-workstation:/bin#
ls
bash: ls: command not found

When you’ve just deleted /bin, you’re screwed. You can’t do anything. Quickly rsync replacements over? Nope. How about a tarball? Nope. In fact you probably cannot even login, so don’t logout of the SSH session you’ve still got open. Booting from a Live CD/USB/DVD is about your only hope.

And then I remembered Bacula. The Bacula file daemon needs only itself to do its job, and it doesn’t even need that once the daemon is running. You could delete /usr/sbin/bacula-fd and it will still work unless you stop the daemon.

We don’t back up system files on workstations, just user files and the dpkg database. But Ubuntu workstations and servers run the same software. So I restored the previous night’s backup of /bin/ from the backup server itself to the affected workstation. Bacula’s restore process is quite flexible, and is happy to let you restore from any archive to any host it knows.

A quick rsync from a similar workstation to make sure all the binaries are there, and we’re saved!