Most of the time I’m not in the same country as the servers I administer. Which means I can’t just drive down and fix something when it goes wrong. It also means that making changes to the network is particularly dangerous. So is updating the kernel, initrd, or GRUB configuration. It is possible to leave a server in a state that requires you to be physically present to fix it. I call this kind of work “flying without a net”. Here are my techniques for safely working without console access.

The most useful tool in my bag is GNU screen. Screen acts as a window/terminal manager inside a terminal. This means you can open one SSH session to a server but start multiple bash logins and multi-task from one terminal window. What’s more, screen is persistent. This means you can disconnect from it and return later, and your programs are still running. Even if your network access is interrupted, anything running in screen will complete rather than terminating when your login session does.

Screen is the safest way, short of console login, to make network changes. Want to run “ifdown eth0 ; ifup eth0“? Try it and you’ll likely find that the first command will interrupt the network and the second command will never run. Your server is now off the network, and you can’t fix it without console access. If you had used screen, you wouldn’t be ringing the on-call staff and suffering a service outage.

For me, the most common way to lose access to a server is to make a mistake while changing the network configuration. This could be the firewall, the IP configuration, tunable kernel networking parameters (sysctl), or something as simple as stopping the OpenSSH server. The first rule is: the server always boots in a known good configuration. Don’t write changes to any startup configuration files until they are tested. When testing, tell the server to restart if the new configuration is bad.

How do you do this? With shutdown -r +5m. This will instruct the server to reboot in 5 minutes. Then, make your changes. If they work, cancel the shutdown with shutdown -c. If they don’t, the server will reboot in 5 minutes on the known, working configuration. Five minutes of down time may be unpleasant, but it is a lot better than sending an engineer.

This means you must test your configuration changes with temporary files. Most commands that use configuration files take an argument to read an alternate file.

cp /etc/network/interfaces /etc/network/interfaces.new
vi /etc/network/interfaces.new
ifdown eth0
ifup -i /etc/network/interfaces.new eth0

Another simple trick is to have a private IP address configured on a virtual interface (eth0:1). Do this for all your servers. If you lose access to one server, you can SSH into another on the same LAN, then SSH into the affected server using the private IP. In rare cases I’ve set a bad route or default gateway, but found I could reach the server from another on the same IP subnet.

Always keep an active SSH session when making changes to the OpenSSH server. Don’t test by logging out of your current shell and then logging in again. Open a new terminal and test in that one. The OpenSSH server only controls new SSH sessions, not existing ones. So you can upgrade or restart the SSH service without losing your open shell.

This applies to firewall changes as well. A typical firewall config accepts all existing, recognised connections before filtering new ones. If you firewall port 22/tcp, you may not be able to open a new session, but your existing ones will probably continue to work.

Install webmin. If you break the OpenSSH server or run afoul of your own firewall, you may be able to login via webmin and fix it. Webmin has native brute-force protection and uses SSL, so you aren’t decreasing your security by installing it. It has saved me more than once.

Be wary of anything that affects the boot process itself. That means GRUB, the kernel, and the initrd image. If you use software RAID on the root partition, make sure that initrd image has all the tools you may need to boot:

apt-get install mdadm evms dmsetup lvm2

In general, it is safe to upgrade or reboot if you use your distribution’s standard software repositories and don’t change the default configurations. If you compile your own kernel or do anything fancy with GRUB, ensure you have console access.

If you have the resources, model changes to a server in a virtual machine first. Today, “resources” only means a mid-range laptop and Virtualbox. Create a virtual network environment that is identical to the real one, using the same IP addresses but only accessible to you. Test your changes, then apply them to the real server only if they work.

These techniques are no substitute for professional server administration. Standard policy at my company is that all servers are installed on network-aware PDUs and connected to IP-capable KVM units. This means I can remotely reboot a server by the power switch, or login as if I were on a local keyboard and monitor. This means I have access to BIOS, bootable network/RAID cards, the GRUB boot menu, a minimal recovery shell (such as when fsck fails), or the regular console. You can’t get any of that with GNU screen or any tool that runs on the server itself.

If you have more than four physical servers, installing a network PDU and KVM will cost less than 15% of the servers themselves and will save you a lot of trouble in on-site support fees and lost time. And they will allow your engineers to do scheduled maintenance at late hours from home.

We use KVM for virtualisation, but I’m not going to discuss the details here. Like any virtualisation solution, KVM starts a virtual machine and attaches its virtual network hardware to a network interface on the host OS. What I want to discuss is how to implement the networking layer.

I previously wrote about creating a network bridge for Virtualbox virtual machines, and we’re going to do something similar here. However, we want to implement VLAN support and native 802.1q VLAN tagging at the same time.

The design:

• The physical server is connected to the Ethernet switch via a 802.1q VLAN tagged trunk port
• The host OS is aware of the trunk port, and implements several virtual network interfaces. Each virtual interface is associated with one VLAN. Any traffic on that virtual interface exits the physical network interface as tagged VLAN packets.
• The host OS provides a network bridge for each VLAN, and adds the virtual VLAN interface to the bridge.
• The guest OSes (virtual machines) are not VLAN-aware. They have a normal ethernet interface which requires no special configuration.
• The virtualisation software (KVM, in this case) attaches the network interface of the guest OS to the VLAN-specific network bridge.

Warning: changing the Ethernet setup to your server can cause you to lose access to it. At all times during this process, ensure you have console access to the server, and network or console access to the switch.

That said, we’ll try to time the network interruption such that we don’t lose access. First, configure your server for VLAN networking. This guide assumes the server runs Ubuntu 8.04 “Hardy Heron”, but the steps are similar for any recent Ubuntu release. I have no idea how Red Hat handles VLANs and bridging, but I invite you to provide the steps in comments.

This guide also assumes the server is connected to a VLAN unaware (access) port, on VLAN 100, with an existing /etc/network/interfaces like so:

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
	address 192.168.0.2
	netmask 255.255.255.0
	gateway 192.168.0.1

Finally, we assume that virtual machines will be attached to either VLAN 100 (the same as the host OS’s own traffic) or VLAN 200. So we’ll prepare support for both.

Install vlan and bridge support.
apt-get install vlan bridge-utils

Then edit /etc/network/interfaces. Modify the existing network setup to be aware of VLAN 100, and to create a bridge on it.

# The loopback network interface
auto lo
iface lo inet loopback          

# LAN uses VLAN trunking, so set any IP addresses on appropriate bridge
auto eth0
iface eth0 inet manual
	up ifconfig eth0 up

# VLAN 100
auto eth0.100
iface eth0.100 inet manual
	up ifconfig eth0.100 up

# KVM bridge, VLAN 100, via eth0
auto br100
iface br100 inet static
	address 192.168.0.2
	netmask 255.255.255.0
	gateway 192.168.0.1
	bridge_ports eth0.100
	bridge_maxwait 5
	bridge_fd 1
	bridge_stp on

We must bring eth0 up before we can create eth0.100, and we must bring eth0.100 up before we can attach it to the bridge interface. Finally, we must configure the host OS’s IP address on the bridge. If you don’t want to attach virtual machines to VLAN 100, you could configure the IP directly on eth0.100 and leave out the stanza for br100.

Now, apply changes. Because we’re going to move the existing IP from eth0 to br100, the easiest way is to reboot. Alternatively, you may try /etc/init.d/networking restart, but make sure you run this from console so you can fix any problems.

Network interruption begins as soon as you run the above command, or reboot. To restore access to the host OS, we must now configure the Ethernet port on the switch. For Cisco, this is easy. Assuming the server is connected to gigabit Ethernet port 1, run:

configure terminal
interface GigabitEthernet0/1
 switchport mode trunk

You should now be able to ping your server. Once you verify that works, add the configuration for VLAN 200 to /etc/network/interfaces.

# VLAN 200
auto eth0.200
iface eth0.200 inet manual
	up ifconfig eth0.200 up

# KVM bridge, VLAN 200, via eth0
auto br200
iface br200 inet manual
	bridge_ports eth0.200
	bridge_maxwait 5
	bridge_fd 1
	bridge_stp on

VLAN 200 doesn’t need an IP on the host OS, so it lacks the static IP configuration. Configure any additional VLANs the same way.

Now we are ready to attach virtual machines to the new bridges. For KVM, use virt-install to create a machine with 20 GB hard disk, 1 GB of RAM, booting the hardy iso, and attached to VLAN 100.

virt-install --connect qemu:///system -n guestname -r 1024 -f /path/to/virtual/disks/guestname.qcow2 -s 20 -c /path/to/isos/ubuntu-8.04.3-server-i386.iso --vnc --os-type linux --os-variant ubuntuHardy --accelerate --network=bridge:br100

Now use virt-manager to connect to the guest and configure it.

The last step can be replaced with any other virtualisation solution. For instance, with Virtualbox, you can create a VM and then attach it to the bridged interface br100.

Unlike the Virtualbox NAT/routing setup, we don’t need to enable IP forwarding (sysctl -w net.ipv4.ip_forward=1). This method uses layer 2 switching only. However, if you use an iptables firewall, make sure the FORWARD chain of the filter table defaults to “ACCEPT”, or use an equivalent rule matching traffic to/from the virtual bridges.

At the time of this writing, my host OS is Ubuntu 9.10 Karmic 64-bit (actually an Intel Core 2 Duo 2.0 MHZ), with VirtualBox 3.0.10 PUEL edition. I run several 32-bit clients: Ubuntu 6.06 LTS (just in case, we still encounter them at some customer sites), Ubuntu 8.04 LTS, Xubuntu 9.10, and Windows XP SP3 (my actual legal copy that came with the laptop).

I left the Windows XP VM running but idle, and noticed that my keyboard and mouse occasionally skipped, and my CPU worked harder than expected. So I ran a few tests with top in batch mode:

top -b -d 10 -n 10

I then redirected that to a file, waited the 10*10 seconds to finish, and grepped the results. This is with VT-x enabled, and shows only the relevant VM (not the GUI or other VMs):

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
30763 tyler     20   0 1020m 609m  73m S   10 15.4   0:28.05 VirtualBox
30763 tyler     20   0 1020m 609m  73m S    9 15.4   0:28.98 VirtualBox
30763 tyler     20   0 1020m 609m  73m S    5 15.4   0:29.52 VirtualBox
30763 tyler     20   0 1020m 609m  73m S    6 15.4   0:30.10 VirtualBox
30763 tyler     20   0 1020m 609m  73m S   20 15.4   0:32.06 VirtualBox
30763 tyler     20   0 1020m 609m  73m S    6 15.4   0:32.63 VirtualBox
30763 tyler     20   0 1020m 609m  73m S   10 15.4   0:33.60 VirtualBox
30763 tyler     20   0 1020m 609m  73m S   16 15.4   0:35.18 VirtualBox
30763 tyler     20   0 1020m 609m  73m S    6 15.4   0:35.78 VirtualBox
30763 tyler     20   0 1020m 609m  73m S    4 15.4   0:36.20 VirtualBox

Here is the same data, after rebooting the guest and disabling VT-x:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
30217 tyler     20   0 1027m 625m  73m S    4 15.8   2:21.32 VirtualBox
30217 tyler     20   0 1027m 625m  73m R    5 15.8   2:21.79 VirtualBox
30217 tyler     20   0 1027m 625m  73m S    1 15.8   2:21.89 VirtualBox
30217 tyler     20   0 1027m 625m  73m S   11 15.8   2:23.04 VirtualBox
30217 tyler     20   0 1027m 625m  73m S   11 15.8   2:24.17 VirtualBox
30217 tyler     20   0 1027m 625m  73m S    5 15.8   2:24.69 VirtualBox
30217 tyler     20   0 1027m 625m  73m S    2 15.8   2:24.94 VirtualBox
30217 tyler     20   0 1027m 625m  73m S   10 15.8   2:25.94 VirtualBox
30217 tyler     20   0 1027m 625m  73m S    5 15.8   2:26.44 VirtualBox
30217 tyler     20   0 1027m 625m  73m S   13 15.8   2:27.72 VirtualBox

Two conclusions:

1. Software virtualisation uses more RAM. The guest is configured with 1 GB of RAM. With VT-x, this uses 1020 M. Without, it uses 1027 M. Whether that represents a memory overhead of 7M per VM or 0.7% of total RAM per VM, I don’t care. It’s small enough.
2. Software virtualisation, under these conditions, is more efficient than using VT-x. The average CPU usage with VT-x was 9.2%. Without, it was 6.7%.

Engineering and science students will recall that all experimenters must note potential flaws and sources of error.

1. This tests only a guest which is completely idle. However, my general impression when doing actual work in the guest VM supports the idle observation.
2. top isn’t the best measure of performance, but it is indicative.
3. I only tested a 64-bit Intel host with a 32-bit Windows guest. As always, more data is better.

I also note that my keyboard and mouse no longer skip when the guest OS is busy. Clearly something is up with VT-x and Ubuntu 9.10 hosts. The right thing to do is to perform more tests confirm the initial observations, but I’m not going to spend any more time on this. Software virtualisation, so far, has been plenty fast for me. I’ll just disable VT-x on all guests, and test again after the next upgrade for VirtualBox or the host kernel.

Note to hardware developers: if your network-based device does not have a standards-compliant HTTP interface, you lose. If it has a Windows-based management tool instead, you lose twice. I will buy your product only if I have no other choice.

I imagine running Windows apps is what 90% of VirtualBox users use it for, but it can do so much more than that. I also run several Linux-based VMs, and use them to test server configs, or even whole networks before rolling out the real thing. If you do this, you probably want to use more than the basic NAT networking that VirtualBox uses by default. For instance, wouldn’t it be nice to install an SSH server in the VM, minimise the VirtualBox GUI, and SSH in from a terminal just like you would a real server?

I assume you are using VirtualBox 2.1.4 from a Linux host running Ubuntu 8.04 “Hardy Heron”. Similar commands can be used on any recent Debian or Ubuntu release. You’ll have to adapt some things to use it on RPM- or source-based Linux distributions. Assume all commands are run as root (directly or with sudo).

Update 2010-08-18: These instructions are still valid as of VirtualBox 3.2.8 and Ubuntu 10.04 “Lucid Lynx”. VirtualBox now creates a “vboxnet0″ interface by default, but this is not a bridge. Follow the instructions below.

Step 1: Create a bridge interface

First, we have to create a bridge interface for the VMs. Install the bridge utilities:
apt-get install bridge-utils

Now make the bridge start on boot. Add the following to /etc/network/interfaces:

# VirtualBox NAT bridge
auto vnet0
iface vnet0 inet static
        address 172.16.0.1
        netmask 255.255.255.0
        bridge_ports none
        bridge_maxwait 0
        bridge_fd 1
        up iptables -t nat -I POSTROUTING -s 172.16.0.0/24 -j MASQUERADE
        down iptables -t nat -D POSTROUTING -s 172.16.0.0/24 -j MASQUERADE

Either reboot or start it manually:
ifup vnet0

We now have a bridge interface to which VirtualBox can attach virtual machines. That traffic will be NATed to your host’s IP address when the guest OS accesses the Internet. However, the traffic won’t yet route.

Note: if you are already using a firewall such as iptables, shorewall, or ufw, you should remove the two iptables lines above and add equivalent commands to your firewall configuration. Otherwise NAT will probably not function.

Step 2: Enable IP forwarding

Now you must tell the kernel to route traffic. Find the ‘net.ipv4.ip_forward’ line in /etc/sysctl.conf, and uncomment it:

# Uncomment the next line to enable packet forwarding for IPv4
net.ipv4.ip_forward=1

And load it:
sysctl -p

Step 3: Setup DHCP and DNS for clients

OK, now you can forward and NAT traffic from client VMs. But you still have to configure static IPs in each guest’s OS. Here is where DNSMasq shines. It provides an all-in-one DHCP/DNS server in a small footprint. Install it:
apt-get install dnsmasq

And edit /etc/dnsmasq.conf to include:

interface=vnet0
dhcp-range=172.16.0.2,172.16.0.254,1h

That’s all you really need, but you may want to explicitly define DNS servers and domains for the guests, or static assignments. Add:

dhcp-option=option:dns-server,172.16.0.1,208.67.222.222,208.67.220.220
dhcp-option=option:domain-name,example.com
dhcp-host=08:00:27:00:00:02,vmxp,172.16.0.2       # Windows XP
dhcp-host=08:00:27:00:00:03,vmubuntu,172.16.0.3   # Ubuntu

This defines the host OS and the OpenDNS servers as the DNS servers (instead of passing on whatever your host OS uses), tells all guests they are part of the domain example.com, and defines two static assignments by MAC address.

Step 4: Set up the virtual machine

1. Start the VirtualBox interface, and edit your virtual machine’s settings.
2. Choose “Network”.
3. Enable a network adaptor.
4. Under “Attached to:”, select “Host Interface”.
5. If you assigned a static DHCP assignment above, be sure to set the same MAC address.
6. Under “Host Interfaces”, select the bridge you created in step 1, vnet0.

Example:

Your virtual machines will now automatically receive an IP address in the 172.16.0.0/24 network, will resolve DNS, will NAT to your host’s external IP address, and can directly address each other.