I try to post a few times each month, but somehow January (and most of February) fell through the cracks. Lately I’ve been busy with operational tasks, which hasn’t left me much room for engineering. I haven’t solved any particularly hard or unusual problems, which is usually what I write about. Instead, I’ll write about a routine problem that is nonetheless tricky enough to warrant discussion.
Most of the time I’m not in the same country as the servers I administer. Which means I can’t just drive down and fix something when it goes wrong. It also means that making changes to the network is particularly dangerous. So is updating the kernel, initrd, or GRUB configuration. It is possible to leave a server in a state that requires you to be physically present to fix it. I call this kind of work “flying without a net”. Here are my techniques for safely working without console access.