I set up OpenVPN so our offices and laptops could securely access internal resources. This lets me print documents directly to another office, for instance. Or access web-based applications that we don’t make available to the public. Or remotely SSH into a PC and fix a problem. A month after setting it up, I realised that:

1. I could traceroute from a host on the LAN to a host on another office LAN, via the VPN
2. I could traceroute from a host on the LAN to a host in the Internet, where the traffic never enters the VPN
3. I could not traceroute from a host on the LAN, through the VPN to the central VPN server, and from there to a server on the public Internet

Why would I want to do #3? Because I sometimes deal with ISPs and autocratic governments that filter Internet traffic. In those circumstances, I back-haul all my traffic to the VPN server and break out there, bypassing any annoying proxies.

How I do it is a story for another day. Suffice it to say that much complicated NAT trickery and careful routing is involved.

So, #1 (through the VPN to the VPN server, and again to another office) is good:

tyler@baal:~$ sudo traceroute -n -I 192.168.6.50

traceroute to 192.168.6.50 (192.168.6.50), 30 hops max, 40 byte packets
 1  192.168.3.1  1.596 ms  2.905 ms  3.394 ms
 2  10.1.0.1  21.868 ms  26.078 ms  27.751 ms
 3  10.1.0.10  162.386 ms  166.004 ms  168.540 ms
 4  192.168.6.50  171.394 ms  175.397 ms  181.904 ms

And #2 (avoiding the VPN entirely) is good:

tyler@baal:~$ sudo traceroute -n -I google.com

traceroute to google.com (74.125.45.100), 30 hops max, 40 byte packets
 1  192.168.3.1  1.718 ms  3.435 ms  4.546 ms
 2  10.1.110.1  12.975 ms  16.903 ms  17.462 ms
 ...
16  74.125.45.100  114.961 ms  110.625 ms  112.908 ms

But #3 (through the VPN to the VPN server) gives no joy. If I change my router’s VPN setup to route all traffic through the VPN (or create a specific route for Google), I see:

tyler@baal:~$ sudo traceroute -n -I google.com

traceroute to google.com (74.125.45.100), 30 hops max, 40 byte packets
 1  192.168.3.1  1.618 ms  2.005 ms  3.704 ms
 2  10.1.0.1  21.889 ms  29.736 ms  30.924 ms
 3  172.16.64.17  33.234 ms  35.010 ms  37.247 ms
 4  * * *
 5  * * *
 ...
15  * * *
16  74.125.45.100  126.346 ms  114.346 ms  114.358 ms

So I can ping it, and the final step in the trace shows the answer, but no routers between the VPN server and Google appear.

Or does it? A little sniffing at my router showed something interesting:

root@router:~# tshark -i tun0 -f 'icmp[icmptype] = icmp-timxceed'

0.000079 192.168.3.1 -> 192.168.3.10 ICMP Time-to-live exceeded (Time to live exceeded in transit)
0.020700 10.1.0.1 -> 192.168.3.10 ICMP Time-to-live exceeded (Time to live exceeded in transit)
...
0.076001 146.188.15.237 -> 192.168.3.10 ICMP Time-to-live exceeded (Time to live exceeded in transit)
0.107338 146.188.69.102 -> 192.168.3.10 ICMP Time-to-live exceeded (Time to live exceeded in transit)

So the traffic is coming in the VPN tunnel. But what is going out?

root@router:~# tshark -i eth1 -f 'icmp[icmptype] = icmp-timxceed'

0.000083 192.168.3.1 -> 192.168.3.10 ICMP Time-to-live exceeded (Time to live exceeded in transit)
0.021200 10.1.0.1 -> 192.168.3.10 ICMP Time-to-live exceeded (Time to live exceeded in transit)
0.024700 172.16.64.17 -> 192.168.3.10 ICMP Time-to-live exceeded (Time to live exceeded in transit)
...

I could see the initial “TTL expired” messages from the first few hops, but none after that. Something is stopping them between tun0 (the OpenVPN tun interface) and eth1 (the inside LAN interface).

Can you see where I’m going with this? It’s our old friend, the reverse-path filter. Disable it with sysctl or echo.

echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter

For some reason, the kernel doesn’t mind most traffic moving between eth1 and tun0, but something about the ICMP “TTL expired” messages triggered rp_filter. Disable it and traceroute works again.

Aside: I always use mtr in place of regular traceroute. What traceroute does in O(n), mtr does in O(1). Actually O(n/5), but explaining that needs it own post. mtr is fast. You should use mtr. Your distro has it.

• failover shared IP addresses using heartbeat
• routing announcements via Quagga BGPd
• 802.1q VLAN tagging
• multiple physical interfaces

During debugging of this network, I encountered an odd scenario whereby traffic coming in from the external interface (eth0) could not reach the IP address of the secondary (inactive) router’s internal interface (eth1, VLAN tagged).

When router B is inactive, it is available on IP addresses on both eth0 and one VLAN of eth1. Because it is inactive, traffic bound for an IP on router B eth1 moves in a big clock-wise circle around the above diagram. It comes from outside to router A eth0, exits router A via eth1, moves to the internal switch, then back to router B eth1. When active, traffic comes directly to it from outside, and instead router A behaves this way. The problem is that when in the inactive state, the inactive router’s return traffic goes directly out eth0. That is, it comes in eth1 and goes back out eth0.

When such a situation occurs (source and destination traffic to the same IP using different interfaces), the Linux kernel (by default in Debian/Ubuntu) drops the traffic as potentially spoofed. This is called reverse-path filtering, and it is useful for preventing obviously bad traffic from entering your network (sourced from a local IP address, multicast, etc). However, you can safely disable it if you perform the same sanity checks in your iptables firewall (and you do, right?).

To disable this check, use the net.ipv4.conf.all.arp_filter sysctl variable:

root@routera:~# sysctl -a | grep \\.rp_filter

net.ipv4.conf.lo.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.eth0.rp_filter = 1
net.ipv4.conf.eth1.rp_filter = 1
net.ipv4.conf.eth1/10.rp_filter = 1
net.ipv4.conf.eth1/20.rp_filter = 1

In this example, my router has two physical interfaces, eth0 and eth1, and has two tagged VLANs on eth1 (VLANs 10 and 20). All are set to perform basic reverse-path filtering. Change them by editing /etc/sysctl.conf. Find this section:

# Comment the next two lines to disable Spoof protection (reverse-path filter)
# Turn on Source Address Verification in all interfaces to
# prevent some spoofing attacks
net.ipv4.conf.default.rp_filter=1
net.ipv4.conf.all.rp_filter=1

Comment-out the two lines here, and apply changes with sysctl -p. If any interfaces still show rp_filter=1, reboot or set them to zero yourself:

sysctl -w net.ipv4.conf.eth0.rp_filter=0

This problem with rp_filter seems to specifically cause issues with VLAN interfaces. I have another pair of routers doing the same thing, except they don’t use VLAN tagging on eth1. They have no issues with rp_filter at all.

If you suspect your traffic is being filtered by rp_filter, you can find out with log_martians:

sysctl -w net.ipv4.conf.all.log_martians=1

Your syslog will now show dropped packets.

Note that the value of rp_filter is not what you may expect.