I recently created a very complex network using routers running Ubuntu Hardy. These routers were configured with the following features:
- failover shared IP addresses using heartbeat
- routing announcements via Quagga BGPd
- 802.1q VLAN tagging
- multiple physical interfaces
During debugging of this network, I encountered an odd scenario whereby traffic coming in from the external interface (eth0) could not reach the IP address of the secondary (inactive) router’s internal interface (eth1, VLAN tagged).
When router B is inactive, it is available on IP addresses on both eth0 and one VLAN of eth1. Because it is inactive, traffic bound for an IP on router B eth1 moves in a big clock-wise circle around the above diagram. It comes from outside to router A eth0, exits router A via eth1, moves to the internal switch, then back to router B eth1. When active, traffic comes directly to it from outside, and instead router A behaves this way. The problem is that when in the inactive state, the inactive router’s return traffic goes directly out eth0. That is, it comes in eth1 and goes back out eth0.
When such a situation occurs (source and destination traffic to the same IP using different interfaces), the Linux kernel (by default in Debian/Ubuntu) drops the traffic as potentially spoofed. This is called reverse-path filtering, and it is useful for preventing obviously bad traffic from entering your network (sourced from a local IP address, multicast, etc). However, you can safely disable it if you perform the same sanity checks in your iptables firewall (and you do, right?).
To disable this check, use the net.ipv4.conf.all.arp_filter sysctl variable:
root@routera:~# sysctl -a | grep \\.rp_filter
net.ipv4.conf.lo.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.eth0.rp_filter = 1
net.ipv4.conf.eth1.rp_filter = 1
net.ipv4.conf.eth1/10.rp_filter = 1
net.ipv4.conf.eth1/20.rp_filter = 1
In this example, my router has two physical interfaces, eth0 and eth1, and has two tagged VLANs on eth1 (VLANs 10 and 20). All are set to perform basic reverse-path filtering. Change them by editing /etc/sysctl.conf. Find this section:
# Comment the next two lines to disable Spoof protection (reverse-path filter)
# Turn on Source Address Verification in all interfaces to
# prevent some spoofing attacks
Comment-out the two lines here, and apply changes with
sysctl -p. If any interfaces still show rp_filter=1, reboot or set them to zero yourself:
sysctl -w net.ipv4.conf.eth0.rp_filter=0
This problem with rp_filter seems to specifically cause issues with VLAN interfaces. I have another pair of routers doing the same thing, except they don’t use VLAN tagging on eth1. They have no issues with rp_filter at all.
If you suspect your traffic is being filtered by rp_filter, you can find out with log_martians:
sysctl -w net.ipv4.conf.all.log_martians=1
Your syslog will now show dropped packets.
Note that the value of rp_filter is not what you may expect.