iperf is a tool for measuring bandwidth and reporting on throughput, jitter, and data loss. Others have written handy tutorials, but I’ll summarise the basics here.

iperf will run on any Linux or Unix (including Mac OSX), and must be installed on both hosts. Additionally, the “server” (receiving) host must allow incoming traffic to some port (which defaults to 5001/UDP and 5001/TCP). If you want to run bidirectional tests with UDP, this means you must open 5001/UDP on both hosts’ firewalls.

iptables -I INPUT -p udp -m udp --dport 5001 -j ACCEPT

A network path is really two paths – the downstream path and the upstream (or return) path. With iperf, the “client” is the transmitter and the “server” is the receiver. So we’ll use the term “downstream” to refer to traffic transmitted from the client to the server, and “upstream” to refer to the opposite. Since these two paths can have different bandwidths and entirely different routes, we should measure them separately.

Start by opening terminal windows to both the client and server hosts, as well as the iperf man page. On the server, you only have to start listening. This runs iperf as a server on the default 5001/UDP.

root@server:~# iperf -s -u
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size:   124 KByte (default)
------------------------------------------------------------

The server will output test results, as well as report them back to the client for display.

On the client, you have many options. You can push X data (-b) for Y seconds (-t). For example, to push 1 mbit for 10 seconds:

root@client:~# iperf -u -c server.example.com -b 1M -t 10
------------------------------------------------------------
Client connecting to 172.16.0.2, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size:   110 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.1 port 37731 connected with 172.16.0.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.19 MBytes  1000 Kbits/sec
[  3] Sent 852 datagrams
[  3] Server Report:
[ ID] Interval       Transfer     Bandwidth       Jitter   Lost/Total Datagrams
[  3]  0.0-10.0 sec  1.19 MBytes  1.00 Mbits/sec  0.842 ms    0/  852 (0%)

You can request that the server perform a reverse connection to test the return path, either at the same time (-d, dual test) or in series (-r, tradeoff). This causes both ends to temporarily start both a client and a server.

root@client:~# iperf -u -c server.example.com -b 1M -t 10 -r
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size:   110 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 172.16.0.2, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size:   110 KByte (default)
------------------------------------------------------------
[  4] local 192.168.1.1 port 46297 connected with 172.16.0.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  1.19 MBytes  1000 Kbits/sec
[  4] Sent 852 datagrams
[  4] Server Report:
[ ID] Interval       Transfer     Bandwidth       Jitter   Lost/Total Datagrams
[  4]  0.0-10.0 sec  1.19 MBytes    998 Kbits/sec  0.250 ms    2/  852 (0.23%)
[  3] local 192.168.1.1 port 5001 connected with 172.16.0.2 port 34916
[ ID] Interval       Transfer     Bandwidth       Jitter   Lost/Total Datagrams
[  3]  0.0-10.0 sec  1.19 MBytes  1.00 Mbits/sec  0.111 ms    0/  851 (0%)
[  3]  0.0-10.0 sec  1 datagrams received out-of-order

The above shows first the client->server transmission, then the server->client transmission. If it seems hard to read, each simultaneous link has an ID such as “[ 3]“, and look for port 5001 to identify the host that is receiving data.

You can also specify the datagram size. Many devices have limits on packets per second, which means you can push more data with 1470-byte datagrams than with 64-byte datagrams. The same link tested with 64-byte datagrams (requiring nearly 20,000 packets where previously we needed only 852) showed 6% packet loss:

root@client:~# iperf -u -c server.example.com -b 1M -t 10 -l 64
------------------------------------------------------------
Client connecting to 172.16.0.2, UDP port 5001
Sending 64 byte datagrams
UDP buffer size:   110 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.1 port 47784 connected with 172.16.0.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.19 MBytes  1000 Kbits/sec
[  3] Sent 19533 datagrams
[  3] Server Report:
[ ID] Interval       Transfer     Bandwidth       Jitter   Lost/Total Datagrams
[  3]  0.0-10.0 sec  1.11 MBytes    933 Kbits/sec  0.134 ms 1294/19533 (6.6%)

To find the total packet size, add 28 bytes to the datagram size for UDP+IP headers. For instance, setting 64-byte datagrams causes iperf to send 92-byte packets. Exceeding the MTU can produce even more interesting results, as packets are fragmented.

iperf provides final throughput results at the end of each test. However, I sometimes find it handy to get results as the test is running, or to report on packets/second. That’s when I use bwm-ng.

Try opening two more terminals, one each to the client and server. In each, start bwm-ng.

root@client:~# bwm-ng -u bits -t 1000

  bwm-ng v0.6 (probing every 1.000s), press 'h' for help
  input: /proc/net/dev type: rate
  |         iface                   Rx                   Tx                Total
  ==============================================================================
               lo:           0.00 Kb/s            0.00 Kb/s            0.00 Kb/s
             eth0:           0.00 Kb/s         1017.34 Kb/s         1017.34 Kb/s
             eth1:           0.00 Kb/s            0.00 Kb/s            0.00 Kb/s
  ------------------------------------------------------------------------------
            total:           0.00 Kb/s         1017.34 Kb/s         1017.34 Kb/s

By default, bwm-ng shows bytes/second. Press ‘u’ to cycle through bytes, bits, packets, and errors per second. Press ‘+’ or ‘-’ to change the refresh time. I find that 1 or 2 seconds produces more accurate results on some hardware. Press ‘h’ for handy in-line help.

Now, start the same iperf tests. Any packet losses will be immediately apparent, as the throughput measurements won’t match. The client will show 1 mbit in the Tx column, while the server will show a lower number in the Rx column.

However, bwm-ng will not differentiate between traffic from iperf and other traffic at the same time. When that happens, it is still useful to use the packets/sec display to find the maximum packet throughput limits of your hardware.

One warning to those who want to test TCP throughput with iperf: you cannot specify the data rate. Instead, iperf in TCP mode will scale up the data rate until it finds the maximum safe window size. For low-latency links, this is generally 85% of the true channel bandwidth as measured by UDP tests. However, as latency increases, TCP bandwidth decreases.

Sniffing is easy when you have access to the intermediate router, but that isn’t always the case. What if you’re just another PC on the same switch? That’s impossible, right?

I’m not the first person to write about this. Switches are not immune to sniffing. There are three techniques for sniffing traffic on an Ethernet switch:

1. Use a monitor port. With a managed switch, configure one port to receive copies of data sent to other port(s). Useful if you have administrator access to a managed switch.
2. MAC flooding. Fill up the switch’s MAC table with false entries, causing it to failover into broadcast mode. All packets will be sent to all ports. Easy to detect and hard to do on modern switches.
3. ARP poisoning. Send unsolicited ARP responses to the target devices (for instance, a PC and its default gateway), telling them to direct traffic to you. Silently route their traffic so they don’t notice an interruption.

It is the last technique that I want to discuss. Many tools exist to poison ARP, from simple command line utilities like arpspoof (part of dsniff), seringe, and arp-sk, to full-featured man-in-the-middle attack suites like Ettercap. Ettercap can poison ARP caches, sniff packets and write them to files, route traffic, and filter/change traffic content. You can find a very helpful step-by-step Ettercap tutorial here, so I won’t cover the same ground. I’d rather explain what these tools do.

ARP poisoning attacks work by sending gratuitous ARP replies to two targets, informing each of them that the attacker’s PC is the opposite target. For instance, suppose I have three devices:

1. gateway, a router on 192.168.0.1, with MAC 00:1d:73:36:ad:0c
2. alice, a PC on 192.168.0.101, with MAC 00:13:02:02:56:da
3. My PC on 192.168.0.102, with MAC 00:17:09:d2:30:25

Alice knows Gateway’s MAC address:

user@alice:~$ ip neigh show
192.168.0.1 dev wlan0 lladdr 00:1d:73:36:ad:0c REACHABLE

Gateway knows Alice’s MAC address:

user@gateway:~$ ip neigh show
192.168.0.101 dev eth1 lladdr 00:13:02:02:56:da REACHABLE

When I start the ARP poisoning attack, both Alice and Gateway learn different MAC addresses for each other:

user@alice:~$ ip neigh show
192.168.0.1 dev wlan0 lladdr 00:17:09:d2:30:25 REACHABLE

user@gateway:~$ ip neigh show
192.168.0.101 dev eth1 lladdr 00:17:09:d2:30:25 REACHABLE

Now Alice will send traffic for Gateway (probably all Internet traffic) to my PC, and Gateway will do the same for traffic intended for Alice. If we stop here, we’ll be able to see what Alice is trying to do, but Alice will quickly realise something is wrong because traffic will stop without explanation.

This is the second part of ARP poisoning (and of any Man-in-the-Middle attack): silently routing traffic onward. You can use the kernel’s IP forwarding to do this:

echo 1 > /proc/sys/net/ipv4/ip_forward

But if you do, you’ll appear in Alice’s traceroute path:

 1  192.168.0.102   2.580 ms  2.754 ms  3.312 ms
 2  192.168.0.1  1.530 ms  2.321 ms  2.964 ms
 3  192.168.1.254  60.537 ms  60.028 ms  59.573 ms
 4  79.86.25.1  19.854 ms  16.007 ms  17.413 ms
 5  ....

If you use “unified sniffing” with Ettercap, it will silently route traffic in software. Other tools, such as fragrouter (also part of dsniff), can do the same thing from the command line. Both of these tools forward traffic without decrementing the TTL, which makes them invisible to traceroute.

ARP poisoning is not silent. It can be detected by both hosts, because you are manipulating their ARP tables. These gratuitous ARP replies are easily visible with a sniffer:

 0.636848 00:17:09:d2:30:25 -> 00:1d:73:36:ad:0c ARP 192.168.0.1 is at 00:17:09:d2:30:25
 0.637622 00:17:09:d2:30:25 -> 00:13:02:02:56:da ARP 192.168.0.101 is at 00:17:09:d2:30:25
 10.680712 00:17:09:d2:30:25 -> 00:1d:73:36:ad:0c ARP 192.168.0.1 is at 00:17:09:d2:30:25
 10.684618 00:17:09:d2:30:25 -> 00:13:02:02:56:da ARP 192.168.0.101 is at 00:17:09:d2:30:25

Most tools send these ARP replies regularly, to keep ARP table entries from timing out. For instance, Ettercap sends them every 10 seconds. This is suspicious, as ARP requests are normally only seen when a host is first discovering another one. It is necessary to prevent true ARP requests from briefly changing the ARP table back to its correct state. Several tools exist to detect or prevent ARP poisoning, like ArpON, arpwatch, and DHCP snooping.

I’ve used traditional security terms like “victim” and “attacker” in this post, and of course ARP poisoning is a useful technique for unauthorised spying. However, it has legitimate uses. I’ve used it a number of times in situations where I had to sniff but couldn’t interrupt the physical cabling or didn’t have a managed switch. Whatever the problem, there is no faster way to understand a network problem than sniffing the raw data.

Around the same time, we also experienced intermittent problems logging in to MSN Messenger, and some users complained of issues accessing certain web pages, including a lot of HTTPS links. I began to suspect these were related.

When the going gets tough, the tough sniff packets. We can sniff at any point in our core routing network, which is by far the most effective way to find a networking problem. And so I found this:

root@router1:~# tshark
  1   0.000000 10.0.2.1 -> 192.168.3.155 ICMP Destination unreachable (Fragmentation needed)

10.0.2.1 is the upstream router from one of our network providers. 192.168.3.155 is one of our servers. So something is definitely wrong upstream of us.

It’s possible to do most troubleshooting with tshark (wireshark’s command-line form). But I prefer to use SCP to copy the capture file back to my PC and investigate it in the GUI. I captured a few more packets and saved them:

tshark -i eth0 -f 'host 10.0.2.1' -w /tmp/upstream.cap -S

This sniffs on eth0 for packets to or from 10.0.2.1 and saves them to /tmp/upstream.cap while showing me a running summary. I let this run until a few more ICMP errors are captured, then hit Ctrl-C to cancel. I can then either copy the file back to my PC and open it with Wireshark, or directly open it using KDE’s “fish://” KIO handler. Try opening a URL like this in Dolphin:

fish://root@192.168.0.1/tmp/

Just right-click and open the file with Wireshark, and KDE will transparently download the file and start Wireshark for you.

I selected one of the offending packets and saw a packet from 10.0.2.1 with this:

This shows that the MTU of the link between 10.0.2.1 and the next router is 1496 bytes. To explain further, we need to understand how IP and Ethernet MTU interact.

The MTU, or maximum transmission unit, for a normal Ethernet network is typically 1500 bytes. This means that an Ethernet packet may contain up to 1500 bytes of data exclusive of the Ethernet headers. As data moves from one Ethernet to the next, as in the case of all traffic routed over the Internet, it must not exceed that Ethernet’s MTU. If it does, the packet is fragmented, split into smaller packets and transmitted as two or more smaller packets.

IP normally allows for fragmentation. This is an inefficient but effective solution, given that a host on the Internet doesn’t usually know the lowest MTU of the entire path between it and the target it would like to reach. However, IP packets contain a flag in the header, “Do not fragment”. If this bit is set, and a packet needs to be fragmented due to a smaller MTU, it cannot be transmitted onward. The router where this happens will drop the packet and return an ICMP type 3, code 4 message, “Destination unreachable, fragmentation needed but ‘do not fragment’ is set.”

Now that I know what to look for, I can find all kinds of MTU issues.

tshark -i eth0 -f 'icmp[icmptype] == 3 and icmp[icmpcode] == 4' -w /tmp/frag.cap -S

Watching this for a while, I found a lot of remote networks with lower than normal MTUs. ICMP error messages contain copies of the IP and TCP/UDP headers of the offending packet, which makes it easy to find what triggered it. I found several examples of SMTP traffic with “do not fragment” set, trying to enter networks with MTUs as low as 300 bytes. But these are remote networks, and are someone else’s problem to fix. What I shouldn’t have is a low MTU immediately upstream, affecting all my traffic.

Now I know I have an MTU problem. But is this the cause of all my problems? I sniffed again, this time trying to login to MSN. The router at 10.0.2.1 again returned an error:

This one told me the cause of the MSN problem. As part of its authentication, MSN Messenger logs in (to 65.54.165.177) via HTTPS (TCP port 443), with a 1500-byte packet which has the “do not fragment” bit set.

Some solutions to the problem of MTU path discovery exist. Sniffing identified the problem to my satisfaction. But you can test more directly by using a traceroute tool that supports MTU discovery. The process is simple: perform traceroute using IP packets of size 1500 bytes with “do not fragment” set. If they are rejected, back off to the “MTU of next hop” value and repeat until you reach your target. On Linux, use tracepath or traceroute --mtu.

root@router1:~# tracepath -n 4.2.2.2
 1:  192.168.101.3    0.236ms pmtu 1500
 1:  192.168.101.1    14.080ms
 1:  192.168.101.1    1.985ms
 2:  192.168.1.254    70.445ms
 3:  10.86.24.1       32.215ms
 4:  no reply
 5:  no reply
 6:  212.113.15.19    37.143ms asymm  7
 7:  4.69.139.97      34.641ms
 8:  4.69.132.134     44.419ms
 9:  4.69.141.169     38.904ms
10:  4.69.133.90      41.321ms
11:  4.69.141.149     41.615ms
12:  4.69.132.138     45.783ms
13:  4.69.140.22      46.130ms
14:  4.68.23.195      46.663ms
15:  no reply
16:  4.2.2.2          45.328ms reached
     Resume: pmtu 1500 hops 16 back 241

A path MTU problem will appear as a drop in MTU:

 1:  192.168.101.3    0.281ms pmtu 1500
 1:  192.168.101.1    12.194ms
 2:  192.168.1.254    69.332ms pmtu 1496
 3:  10.86.24.1       35.182ms

On Windows, try mturoute. On Mac OS X, try traceroute -F google.com 1500.

I reported the problem to our upstream provider and included the tracepath output and a few packets captured during testing. They scheduled a maintenance window for the following afternoon and corrected the problem with two minutes of down time. We haven’t had an issue with email, MSN, or web pages since.

If you want to do something more complicated, or prefer editing iptables rules yourself, you’ll have to do it by hand. When I first started doing this I found a template online and edited it to suit my need. Over time I’ve learned a lot more about iptables, and my templates have evolved.

Here is a simple basic firewall. The outline is taken from the output of iptables-save, and then hand-edited. I’ve left comments in to explain what each section does.

*filter
:FORWARD ACCEPT [0:0]
:INPUT DROP [0:0]
:OUTPUT ACCEPT [0:0]
# Accept all loopback traffic
-A INPUT -i lo -j ACCEPT
# Drop invalid/unknown/spoofed TCP sessions
-A INPUT -p tcp ! --syn -m state --state NEW -j DROP
# Accept new sessions
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
# Accept important ICMP types
-A INPUT -p icmp -m icmp --icmp-type 0 -j ACCEPT
-A INPUT -p icmp -m icmp --icmp-type 3 -j ACCEPT
-A INPUT -p icmp -m icmp --icmp-type 4 -j ACCEPT
-A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A INPUT -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A INPUT -p icmp -m icmp --icmp-type 12 -j ACCEPT
# Accept known services
-A INPUT -p tcp -m tcp -m multiport --dports 22,25,80,443,465,587,993,995 -j ACCEPT
-A INPUT -p udp -m udp -m multiport --dports 53,5353 -j ACCEPT
COMMIT

Edit the last two lines for your specific services, and then save the file as /etc/iptables.up.rules and restore it:

iptables-restore < /etc/iptables.up.rules

The above is a default-deny firewall which allows all outgoing and loopback traffic, but filters incoming connections. This means that anything not specifically allowed will be dropped without notice.

Suppose you want to do this at your router to protect all your servers. Assuming:

1. The router is the default gateway for both servers
2. server1 at 192.168.0.100 is a mail server offering SMTP, POP, POP-SSL, IMAP, and IMAP-SSL
3. server2 at 192.168.0.101 is a web server offering HTTP, HTTPS, and DNS
4. all servers offer SSH, including the router

For routed (forwarded) traffic, you must use the FORWARD chain. In addition to ACCEPT and DROP, chains can call other named chains. I recommend creating a chain for each server you want to protect.

*filter
:FORWARD DROP [0:0]
:INPUT DROP [0:0]
:OUTPUT ACCEPT [0:0]
:server1 - [0:0]
:server2 - [0:0]
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp ! --syn -m state --state NEW -j DROP
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -p icmp -m icmp --icmp-type 0 -j ACCEPT
-A INPUT -p icmp -m icmp --icmp-type 3 -j ACCEPT
-A INPUT -p icmp -m icmp --icmp-type 4 -j ACCEPT
-A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A INPUT -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A INPUT -p icmp -m icmp --icmp-type 12 -j ACCEPT
# if just a single port, you can avoid the multiport match extension and use dport
-A INPUT -p tcp -m tcp --dport 22  -j ACCEPT
# put common rules first
-A FORWARD -p tcp ! --syn -m state --state NEW -j DROP
-A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT
-A FORWARD -p icmp -m icmp --icmp-type 0 -j ACCEPT
-A FORWARD -p icmp -m icmp --icmp-type 3 -j ACCEPT
-A FORWARD -p icmp -m icmp --icmp-type 4 -j ACCEPT
-A FORWARD -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A FORWARD -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A FORWARD -p icmp -m icmp --icmp-type 12 -j ACCEPT
# all servers offer SSH
-A FORWARD -p tcp -m tcp --dport 22  -j ACCEPT
# send specific traffic to custom rules
-A FORWARD -d 192.168.0.100 -j server1
-A FORWARD -d 192.168.0.101 -j server2
-A server1 -p tcp -m tcp -m multiport --dports 25,110,143,993,995 -j ACCEPT
-A server2 -p tcp -m tcp -m multiport --dports 80,443 -j ACCEPT
-A server2 -p udp -m udp --dport 53 -j ACCEPT
COMMIT

It's possible to reduce complexity by putting the common rules in another chain and calling it from both INPUT and FORWARD:

*filter
:FORWARD DROP [0:0]
:INPUT DROP [0:0]
:OUTPUT ACCEPT [0:0]
:common_fw - [0:0]
:server1 - [0:0]
:server2 - [0:0]
-A INPUT -i lo -j ACCEPT
-A INPUT -j common_fw
-A FORWARD -j common_fw
-A FORWARD -d 192.168.0.100 -j server1
-A FORWARD -d 192.168.0.101 -j server2
-A common_fw -p tcp ! --syn -m state --state NEW -j DROP
-A common_fw -m state --state ESTABLISHED,RELATED -j ACCEPT
-A common_fw -p icmp -m icmp --icmp-type 0 -j ACCEPT
-A common_fw -p icmp -m icmp --icmp-type 3 -j ACCEPT
-A common_fw -p icmp -m icmp --icmp-type 4 -j ACCEPT
-A common_fw -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A common_fw -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A common_fw -p icmp -m icmp --icmp-type 12 -j ACCEPT
-A common_fw -p tcp -m tcp --dport 22  -j ACCEPT
-A server1 -p tcp -m tcp -m multiport --dports 25,110,143,993,995 -j ACCEPT
-A server2 -p tcp -m tcp -m multiport --dports 80,443 -j ACCEPT
-A server2 -p udp -m udp --dport 53 -j ACCEPT
COMMIT

Where to save and how to load your rules depends on the distribution. Ubuntu and Debian don't have a standard way to load an iptables ruleset on boot. So I borrowed Webmin's practice of putting a line in /etc/network/interfaces:

# The primary network interface
auto eth0
iface eth0 inet static
        address 192.168.0.2
        netmask 255.255.255.0
        gateway 192.168.0.1
        post-up iptables-restore < /etc/iptables.up.rules

Users of RPM-based systems like Fedora, RHEL, and CentOS have it easier. Save your rules in /etc/sysconfig/iptables, and they'll be loaded at boot or whenever you restart networking. However, Red Hat systems come with a simple firewall manager named lokkit, which creates its own unusual chains like "RH-Firewall-1-INPUT". Here is an example from a CentOS 4 server I administer:

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:RH-Firewall-1-INPUT - [0:0]
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited

You should never expect your hand-edited files to work with a firewall management tool like ufw or lokkit. However, if you are careful to obey the expected format these tools use, you can make small edits by hand while still using the tool. For instance, all of my templates are compatible with Webmin's firewall editor.

Most of the time I’m not in the same country as the servers I administer. Which means I can’t just drive down and fix something when it goes wrong. It also means that making changes to the network is particularly dangerous. So is updating the kernel, initrd, or GRUB configuration. It is possible to leave a server in a state that requires you to be physically present to fix it. I call this kind of work “flying without a net”. Here are my techniques for safely working without console access.

The most useful tool in my bag is GNU screen. Screen acts as a window/terminal manager inside a terminal. This means you can open one SSH session to a server but start multiple bash logins and multi-task from one terminal window. What’s more, screen is persistent. This means you can disconnect from it and return later, and your programs are still running. Even if your network access is interrupted, anything running in screen will complete rather than terminating when your login session does.

Screen is the safest way, short of console login, to make network changes. Want to run “ifdown eth0 ; ifup eth0“? Try it and you’ll likely find that the first command will interrupt the network and the second command will never run. Your server is now off the network, and you can’t fix it without console access. If you had used screen, you wouldn’t be ringing the on-call staff and suffering a service outage.

For me, the most common way to lose access to a server is to make a mistake while changing the network configuration. This could be the firewall, the IP configuration, tunable kernel networking parameters (sysctl), or something as simple as stopping the OpenSSH server. The first rule is: the server always boots in a known good configuration. Don’t write changes to any startup configuration files until they are tested. When testing, tell the server to restart if the new configuration is bad.

How do you do this? With shutdown -r +5m. This will instruct the server to reboot in 5 minutes. Then, make your changes. If they work, cancel the shutdown with shutdown -c. If they don’t, the server will reboot in 5 minutes on the known, working configuration. Five minutes of down time may be unpleasant, but it is a lot better than sending an engineer.

This means you must test your configuration changes with temporary files. Most commands that use configuration files take an argument to read an alternate file.

cp /etc/network/interfaces /etc/network/interfaces.new
vi /etc/network/interfaces.new
ifdown eth0
ifup -i /etc/network/interfaces.new eth0

Another simple trick is to have a private IP address configured on a virtual interface (eth0:1). Do this for all your servers. If you lose access to one server, you can SSH into another on the same LAN, then SSH into the affected server using the private IP. In rare cases I’ve set a bad route or default gateway, but found I could reach the server from another on the same IP subnet.

Always keep an active SSH session when making changes to the OpenSSH server. Don’t test by logging out of your current shell and then logging in again. Open a new terminal and test in that one. The OpenSSH server only controls new SSH sessions, not existing ones. So you can upgrade or restart the SSH service without losing your open shell.

This applies to firewall changes as well. A typical firewall config accepts all existing, recognised connections before filtering new ones. If you firewall port 22/tcp, you may not be able to open a new session, but your existing ones will probably continue to work.

Install webmin. If you break the OpenSSH server or run afoul of your own firewall, you may be able to login via webmin and fix it. Webmin has native brute-force protection and uses SSL, so you aren’t decreasing your security by installing it. It has saved me more than once.

Be wary of anything that affects the boot process itself. That means GRUB, the kernel, and the initrd image. If you use software RAID on the root partition, make sure that initrd image has all the tools you may need to boot:

apt-get install mdadm evms dmsetup lvm2

In general, it is safe to upgrade or reboot if you use your distribution’s standard software repositories and don’t change the default configurations. If you compile your own kernel or do anything fancy with GRUB, ensure you have console access.

If you have the resources, model changes to a server in a virtual machine first. Today, “resources” only means a mid-range laptop and Virtualbox. Create a virtual network environment that is identical to the real one, using the same IP addresses but only accessible to you. Test your changes, then apply them to the real server only if they work.

These techniques are no substitute for professional server administration. Standard policy at my company is that all servers are installed on network-aware PDUs and connected to IP-capable KVM units. This means I can remotely reboot a server by the power switch, or login as if I were on a local keyboard and monitor. This means I have access to BIOS, bootable network/RAID cards, the GRUB boot menu, a minimal recovery shell (such as when fsck fails), or the regular console. You can’t get any of that with GNU screen or any tool that runs on the server itself.

If you have more than four physical servers, installing a network PDU and KVM will cost less than 15% of the servers themselves and will save you a lot of trouble in on-site support fees and lost time. And they will allow your engineers to do scheduled maintenance at late hours from home.

We use KVM for virtualisation, but I’m not going to discuss the details here. Like any virtualisation solution, KVM starts a virtual machine and attaches its virtual network hardware to a network interface on the host OS. What I want to discuss is how to implement the networking layer.

I previously wrote about creating a network bridge for Virtualbox virtual machines, and we’re going to do something similar here. However, we want to implement VLAN support and native 802.1q VLAN tagging at the same time.

The design:

• The physical server is connected to the Ethernet switch via a 802.1q VLAN tagged trunk port
• The host OS is aware of the trunk port, and implements several virtual network interfaces. Each virtual interface is associated with one VLAN. Any traffic on that virtual interface exits the physical network interface as tagged VLAN packets.
• The host OS provides a network bridge for each VLAN, and adds the virtual VLAN interface to the bridge.
• The guest OSes (virtual machines) are not VLAN-aware. They have a normal ethernet interface which requires no special configuration.
• The virtualisation software (KVM, in this case) attaches the network interface of the guest OS to the VLAN-specific network bridge.

Warning: changing the Ethernet setup to your server can cause you to lose access to it. At all times during this process, ensure you have console access to the server, and network or console access to the switch.

That said, we’ll try to time the network interruption such that we don’t lose access. First, configure your server for VLAN networking. This guide assumes the server runs Ubuntu 8.04 “Hardy Heron”, but the steps are similar for any recent Ubuntu release. I have no idea how Red Hat handles VLANs and bridging, but I invite you to provide the steps in comments.

This guide also assumes the server is connected to a VLAN unaware (access) port, on VLAN 100, with an existing /etc/network/interfaces like so:

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
	address 192.168.0.2
	netmask 255.255.255.0
	gateway 192.168.0.1

Finally, we assume that virtual machines will be attached to either VLAN 100 (the same as the host OS’s own traffic) or VLAN 200. So we’ll prepare support for both.

Install vlan and bridge support.
apt-get install vlan bridge-utils

Then edit /etc/network/interfaces. Modify the existing network setup to be aware of VLAN 100, and to create a bridge on it.

# The loopback network interface
auto lo
iface lo inet loopback          

# LAN uses VLAN trunking, so set any IP addresses on appropriate bridge
auto eth0
iface eth0 inet manual
	up ifconfig eth0 up

# VLAN 100
auto eth0.100
iface eth0.100 inet manual
	up ifconfig eth0.100 up

# KVM bridge, VLAN 100, via eth0
auto br100
iface br100 inet static
	address 192.168.0.2
	netmask 255.255.255.0
	gateway 192.168.0.1
	bridge_ports eth0.100
	bridge_maxwait 5
	bridge_fd 1
	bridge_stp on

We must bring eth0 up before we can create eth0.100, and we must bring eth0.100 up before we can attach it to the bridge interface. Finally, we must configure the host OS’s IP address on the bridge. If you don’t want to attach virtual machines to VLAN 100, you could configure the IP directly on eth0.100 and leave out the stanza for br100.

Now, apply changes. Because we’re going to move the existing IP from eth0 to br100, the easiest way is to reboot. Alternatively, you may try /etc/init.d/networking restart, but make sure you run this from console so you can fix any problems.

Network interruption begins as soon as you run the above command, or reboot. To restore access to the host OS, we must now configure the Ethernet port on the switch. For Cisco, this is easy. Assuming the server is connected to gigabit Ethernet port 1, run:

configure terminal
interface GigabitEthernet0/1
 switchport mode trunk

You should now be able to ping your server. Once you verify that works, add the configuration for VLAN 200 to /etc/network/interfaces.

# VLAN 200
auto eth0.200
iface eth0.200 inet manual
	up ifconfig eth0.200 up

# KVM bridge, VLAN 200, via eth0
auto br200
iface br200 inet manual
	bridge_ports eth0.200
	bridge_maxwait 5
	bridge_fd 1
	bridge_stp on

VLAN 200 doesn’t need an IP on the host OS, so it lacks the static IP configuration. Configure any additional VLANs the same way.

Now we are ready to attach virtual machines to the new bridges. For KVM, use virt-install to create a machine with 20 GB hard disk, 1 GB of RAM, booting the hardy iso, and attached to VLAN 100.

virt-install --connect qemu:///system -n guestname -r 1024 -f /path/to/virtual/disks/guestname.qcow2 -s 20 -c /path/to/isos/ubuntu-8.04.3-server-i386.iso --vnc --os-type linux --os-variant ubuntuHardy --accelerate --network=bridge:br100

Now use virt-manager to connect to the guest and configure it.

The last step can be replaced with any other virtualisation solution. For instance, with Virtualbox, you can create a VM and then attach it to the bridged interface br100.

Unlike the Virtualbox NAT/routing setup, we don’t need to enable IP forwarding (sysctl -w net.ipv4.ip_forward=1). This method uses layer 2 switching only. However, if you use an iptables firewall, make sure the FORWARD chain of the filter table defaults to “ACCEPT”, or use an equivalent rule matching traffic to/from the virtual bridges.

Jul 12 14:15:26 mailserver.example.com postfix/smtpd[19885]: NOQUEUE: reject: RCPT from 206.12.0.10.in-addr.arpa[10.0.12.206]: 554 5.7.1 <symons@yahoo.co.uk>: Relay access denied; from=<yyjaqveh@lpsb.com> to=<symons@yahoo.co.uk> proto=ESMTP helo=<206.12.0.10.in-addr.arpa>

IP addresses have been obscured to protect the guilty (or ignorant, as this is certainly a botnet). Unfortunately, a large number of the IP addresses in question belonged to my own satellite customers. Mail servers for our other domains were almost entirely unaffected. Which tells me that some bastard has written a botnet spam client that looks up its own public IP, finds the reverse DNS entry, looks up the MX record of the corresponding domain, and then attempts to relay mail through that server. This is particularly mean, as it will encourage your own ISP to shut you down.

I ignored these attempts at first. The server was rejecting them and the botnet didn’t appear to be trying to relay through anyone else (according to traffic logs). With our Postfix settings it was unlikely they’d ever succeed. Then the log entries moved from “annoying” to “problematical”:

Jul 13 20:34:50 mailserver.example.com postfix/master[18904]: warning: service “smtp” (25) has reached its process limit “200″: new clients may experience noticeable delays
Jul 13 20:34:50 mailserver.example.com postfix/master[18904]: warning: to avoid this condition, increase the process count in master.cf or reduce the service time per client
Jul 13 20:34:50 mailserver.example.com postfix/master[18904]: warning: see http://www.postfix.org/STRESS_README.html for examples of stress-adapting configuration settings

Time to do something about it. We already use fail2ban to protect other services, so I enabled the postfix rules. In /etc/fail2ban/jail.local:

[postfix]
enabled  = true
ignoreip = 127.0.0.1/8   # suggest listing your valid server IP ranges here
port     = smtp,ssmtp
filter   = postfix
logpath  = /var/log/mail.log
bantime  = 3600

I then reloaded fail2ban, and waited for results. Within a few minutes (fail2ban takes time to parse existing logs), 65 IP addresses were banned. Here are my results 24 hours later, edited for brevity:

root@mailserver:/etc/fail2ban# fail2ban-client status postfix

Status for the jail: postfix
|- filter
|  |- File list:        /var/log/mail.log
|  |- Currently failed: 49
|  `- Total failed:     58171
`- action
   |- Currently banned: 68
   |  `- IP list: ...
   `- Total banned:     3883

Now to hunt down the customers with botnet infections and start sending “violation of terms of service” warnings.

I set up OpenVPN so our offices and laptops could securely access internal resources. This lets me print documents directly to another office, for instance. Or access web-based applications that we don’t make available to the public. Or remotely SSH into a PC and fix a problem. A month after setting it up, I realised that:

1. I could traceroute from a host on the LAN to a host on another office LAN, via the VPN
2. I could traceroute from a host on the LAN to a host in the Internet, where the traffic never enters the VPN
3. I could not traceroute from a host on the LAN, through the VPN to the central VPN server, and from there to a server on the public Internet

Why would I want to do #3? Because I sometimes deal with ISPs and autocratic governments that filter Internet traffic. In those circumstances, I back-haul all my traffic to the VPN server and break out there, bypassing any annoying proxies.

How I do it is a story for another day. Suffice it to say that much complicated NAT trickery and careful routing is involved.

So, #1 (through the VPN to the VPN server, and again to another office) is good:

tyler@baal:~$ sudo traceroute -n -I 192.168.6.50

traceroute to 192.168.6.50 (192.168.6.50), 30 hops max, 40 byte packets
 1  192.168.3.1  1.596 ms  2.905 ms  3.394 ms
 2  10.1.0.1  21.868 ms  26.078 ms  27.751 ms
 3  10.1.0.10  162.386 ms  166.004 ms  168.540 ms
 4  192.168.6.50  171.394 ms  175.397 ms  181.904 ms

And #2 (avoiding the VPN entirely) is good:

tyler@baal:~$ sudo traceroute -n -I google.com

traceroute to google.com (74.125.45.100), 30 hops max, 40 byte packets
 1  192.168.3.1  1.718 ms  3.435 ms  4.546 ms
 2  10.1.110.1  12.975 ms  16.903 ms  17.462 ms
 ...
16  74.125.45.100  114.961 ms  110.625 ms  112.908 ms

But #3 (through the VPN to the VPN server) gives no joy. If I change my router’s VPN setup to route all traffic through the VPN (or create a specific route for Google), I see:

tyler@baal:~$ sudo traceroute -n -I google.com

traceroute to google.com (74.125.45.100), 30 hops max, 40 byte packets
 1  192.168.3.1  1.618 ms  2.005 ms  3.704 ms
 2  10.1.0.1  21.889 ms  29.736 ms  30.924 ms
 3  172.16.64.17  33.234 ms  35.010 ms  37.247 ms
 4  * * *
 5  * * *
 ...
15  * * *
16  74.125.45.100  126.346 ms  114.346 ms  114.358 ms

So I can ping it, and the final step in the trace shows the answer, but no routers between the VPN server and Google appear.

Or does it? A little sniffing at my router showed something interesting:

root@router:~# tshark -i tun0 -f 'icmp[icmptype] = icmp-timxceed'

0.000079 192.168.3.1 -> 192.168.3.10 ICMP Time-to-live exceeded (Time to live exceeded in transit)
0.020700 10.1.0.1 -> 192.168.3.10 ICMP Time-to-live exceeded (Time to live exceeded in transit)
...
0.076001 146.188.15.237 -> 192.168.3.10 ICMP Time-to-live exceeded (Time to live exceeded in transit)
0.107338 146.188.69.102 -> 192.168.3.10 ICMP Time-to-live exceeded (Time to live exceeded in transit)

So the traffic is coming in the VPN tunnel. But what is going out?

root@router:~# tshark -i eth1 -f 'icmp[icmptype] = icmp-timxceed'

0.000083 192.168.3.1 -> 192.168.3.10 ICMP Time-to-live exceeded (Time to live exceeded in transit)
0.021200 10.1.0.1 -> 192.168.3.10 ICMP Time-to-live exceeded (Time to live exceeded in transit)
0.024700 172.16.64.17 -> 192.168.3.10 ICMP Time-to-live exceeded (Time to live exceeded in transit)
...

I could see the initial “TTL expired” messages from the first few hops, but none after that. Something is stopping them between tun0 (the OpenVPN tun interface) and eth1 (the inside LAN interface).

Can you see where I’m going with this? It’s our old friend, the reverse-path filter. Disable it with sysctl or echo.

echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter

For some reason, the kernel doesn’t mind most traffic moving between eth1 and tun0, but something about the ICMP “TTL expired” messages triggered rp_filter. Disable it and traceroute works again.

Aside: I always use mtr in place of regular traceroute. What traceroute does in O(n), mtr does in O(1). Actually O(n/5), but explaining that needs it own post. mtr is fast. You should use mtr. Your distro has it.

• failover shared IP addresses using heartbeat
• routing announcements via Quagga BGPd
• 802.1q VLAN tagging
• multiple physical interfaces

During debugging of this network, I encountered an odd scenario whereby traffic coming in from the external interface (eth0) could not reach the IP address of the secondary (inactive) router’s internal interface (eth1, VLAN tagged).

When router B is inactive, it is available on IP addresses on both eth0 and one VLAN of eth1. Because it is inactive, traffic bound for an IP on router B eth1 moves in a big clock-wise circle around the above diagram. It comes from outside to router A eth0, exits router A via eth1, moves to the internal switch, then back to router B eth1. When active, traffic comes directly to it from outside, and instead router A behaves this way. The problem is that when in the inactive state, the inactive router’s return traffic goes directly out eth0. That is, it comes in eth1 and goes back out eth0.

When such a situation occurs (source and destination traffic to the same IP using different interfaces), the Linux kernel (by default in Debian/Ubuntu) drops the traffic as potentially spoofed. This is called reverse-path filtering, and it is useful for preventing obviously bad traffic from entering your network (sourced from a local IP address, multicast, etc). However, you can safely disable it if you perform the same sanity checks in your iptables firewall (and you do, right?).

To disable this check, use the net.ipv4.conf.all.arp_filter sysctl variable:

root@routera:~# sysctl -a | grep \\.rp_filter

net.ipv4.conf.lo.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.eth0.rp_filter = 1
net.ipv4.conf.eth1.rp_filter = 1
net.ipv4.conf.eth1/10.rp_filter = 1
net.ipv4.conf.eth1/20.rp_filter = 1

In this example, my router has two physical interfaces, eth0 and eth1, and has two tagged VLANs on eth1 (VLANs 10 and 20). All are set to perform basic reverse-path filtering. Change them by editing /etc/sysctl.conf. Find this section:

# Comment the next two lines to disable Spoof protection (reverse-path filter)
# Turn on Source Address Verification in all interfaces to
# prevent some spoofing attacks
net.ipv4.conf.default.rp_filter=1
net.ipv4.conf.all.rp_filter=1

Comment-out the two lines here, and apply changes with sysctl -p. If any interfaces still show rp_filter=1, reboot or set them to zero yourself:

sysctl -w net.ipv4.conf.eth0.rp_filter=0

This problem with rp_filter seems to specifically cause issues with VLAN interfaces. I have another pair of routers doing the same thing, except they don’t use VLAN tagging on eth1. They have no issues with rp_filter at all.

If you suspect your traffic is being filtered by rp_filter, you can find out with log_martians:

sysctl -w net.ipv4.conf.all.log_martians=1

Your syslog will now show dropped packets.

Note that the value of rp_filter is not what you may expect.

Note to hardware developers: if your network-based device does not have a standards-compliant HTTP interface, you lose. If it has a Windows-based management tool instead, you lose twice. I will buy your product only if I have no other choice.

I imagine running Windows apps is what 90% of VirtualBox users use it for, but it can do so much more than that. I also run several Linux-based VMs, and use them to test server configs, or even whole networks before rolling out the real thing. If you do this, you probably want to use more than the basic NAT networking that VirtualBox uses by default. For instance, wouldn’t it be nice to install an SSH server in the VM, minimise the VirtualBox GUI, and SSH in from a terminal just like you would a real server?

I assume you are using VirtualBox 2.1.4 from a Linux host running Ubuntu 8.04 “Hardy Heron”. Similar commands can be used on any recent Debian or Ubuntu release. You’ll have to adapt some things to use it on RPM- or source-based Linux distributions. Assume all commands are run as root (directly or with sudo).

Update 2010-08-18: These instructions are still valid as of VirtualBox 3.2.8 and Ubuntu 10.04 “Lucid Lynx”. VirtualBox now creates a “vboxnet0″ interface by default, but this is not a bridge. Follow the instructions below.

Step 1: Create a bridge interface

First, we have to create a bridge interface for the VMs. Install the bridge utilities:
apt-get install bridge-utils

Now make the bridge start on boot. Add the following to /etc/network/interfaces:

# VirtualBox NAT bridge
auto vnet0
iface vnet0 inet static
        address 172.16.0.1
        netmask 255.255.255.0
        bridge_ports none
        bridge_maxwait 0
        bridge_fd 1
        up iptables -t nat -I POSTROUTING -s 172.16.0.0/24 -j MASQUERADE
        down iptables -t nat -D POSTROUTING -s 172.16.0.0/24 -j MASQUERADE

Either reboot or start it manually:
ifup vnet0

We now have a bridge interface to which VirtualBox can attach virtual machines. That traffic will be NATed to your host’s IP address when the guest OS accesses the Internet. However, the traffic won’t yet route.

Note: if you are already using a firewall such as iptables, shorewall, or ufw, you should remove the two iptables lines above and add equivalent commands to your firewall configuration. Otherwise NAT will probably not function.

Step 2: Enable IP forwarding

Now you must tell the kernel to route traffic. Find the ‘net.ipv4.ip_forward’ line in /etc/sysctl.conf, and uncomment it:

# Uncomment the next line to enable packet forwarding for IPv4
net.ipv4.ip_forward=1

And load it:
sysctl -p

Step 3: Setup DHCP and DNS for clients

OK, now you can forward and NAT traffic from client VMs. But you still have to configure static IPs in each guest’s OS. Here is where DNSMasq shines. It provides an all-in-one DHCP/DNS server in a small footprint. Install it:
apt-get install dnsmasq

And edit /etc/dnsmasq.conf to include:

interface=vnet0
dhcp-range=172.16.0.2,172.16.0.254,1h

That’s all you really need, but you may want to explicitly define DNS servers and domains for the guests, or static assignments. Add:

dhcp-option=option:dns-server,172.16.0.1,208.67.222.222,208.67.220.220
dhcp-option=option:domain-name,example.com
dhcp-host=08:00:27:00:00:02,vmxp,172.16.0.2       # Windows XP
dhcp-host=08:00:27:00:00:03,vmubuntu,172.16.0.3   # Ubuntu

This defines the host OS and the OpenDNS servers as the DNS servers (instead of passing on whatever your host OS uses), tells all guests they are part of the domain example.com, and defines two static assignments by MAC address.

Step 4: Set up the virtual machine

1. Start the VirtualBox interface, and edit your virtual machine’s settings.
2. Choose “Network”.
3. Enable a network adaptor.
4. Under “Attached to:”, select “Host Interface”.
5. If you assigned a static DHCP assignment above, be sure to set the same MAC address.
6. Under “Host Interfaces”, select the bridge you created in step 1, vnet0.

Example:

Your virtual machines will now automatically receive an IP address in the 172.16.0.0/24 network, will resolve DNS, will NAT to your host’s external IP address, and can directly address each other.