Reverse-path filter part 2: OpenVPN and traceroute

After my recent adventure with reverse-path filtering, I didn’t expect to see it again so soon. And then I took another look at a long-standing annoyance in our OpenVPN network.

I set up OpenVPN so our offices and laptops could securely access internal resources. This lets me print documents directly to another office, for instance. Or access web-based applications that we don’t make available to the public. Or remotely SSH into a PC and fix a problem. A month after setting it up, I realised that:

1. I could traceroute from a host on the LAN to a host on another office LAN, via the VPN
2. I could traceroute from a host on the LAN to a host in the Internet, where the traffic never enters the VPN
3. I could not traceroute from a host on the LAN, through the VPN to the central VPN server, and from there to a server on the public Internet

Why would I want to do #3? Because I sometimes deal with ISPs and autocratic governments that filter Internet traffic. In those circumstances, I back-haul all my traffic to the VPN server and break out there, bypassing any annoying proxies.

How I do it is a story for another day. Suffice it to say that much complicated NAT trickery and careful routing is involved.

So, #1 (through the VPN to the VPN server, and again to another office) is good:

tyler@baal:~$ sudo traceroute -n -I 192.168.6.50

traceroute to 192.168.6.50 (192.168.6.50), 30 hops max, 40 byte packets
 1  192.168.3.1  1.596 ms  2.905 ms  3.394 ms
 2  10.1.0.1  21.868 ms  26.078 ms  27.751 ms
 3  10.1.0.10  162.386 ms  166.004 ms  168.540 ms
 4  192.168.6.50  171.394 ms  175.397 ms  181.904 ms

And #2 (avoiding the VPN entirely) is good:

tyler@baal:~$ sudo traceroute -n -I google.com

traceroute to google.com (74.125.45.100), 30 hops max, 40 byte packets
 1  192.168.3.1  1.718 ms  3.435 ms  4.546 ms
 2  10.1.110.1  12.975 ms  16.903 ms  17.462 ms
 ...
16  74.125.45.100  114.961 ms  110.625 ms  112.908 ms

But #3 (through the VPN to the VPN server) gives no joy. If I change my router’s VPN setup to route all traffic through the VPN (or create a specific route for Google), I see:

tyler@baal:~$ sudo traceroute -n -I google.com

traceroute to google.com (74.125.45.100), 30 hops max, 40 byte packets
 1  192.168.3.1  1.618 ms  2.005 ms  3.704 ms
 2  10.1.0.1  21.889 ms  29.736 ms  30.924 ms
 3  172.16.64.17  33.234 ms  35.010 ms  37.247 ms
 4  * * *
 5  * * *
 ...
15  * * *
16  74.125.45.100  126.346 ms  114.346 ms  114.358 ms

So I can ping it, and the final step in the trace shows the answer, but no routers between the VPN server and Google appear.

Or does it? A little sniffing at my router showed something interesting:

root@router:~# tshark -i tun0 -f 'icmp[icmptype] = icmp-timxceed'

0.000079 192.168.3.1 -> 192.168.3.10 ICMP Time-to-live exceeded (Time to live exceeded in transit)
0.020700 10.1.0.1 -> 192.168.3.10 ICMP Time-to-live exceeded (Time to live exceeded in transit)
...
0.076001 146.188.15.237 -> 192.168.3.10 ICMP Time-to-live exceeded (Time to live exceeded in transit)
0.107338 146.188.69.102 -> 192.168.3.10 ICMP Time-to-live exceeded (Time to live exceeded in transit)

So the traffic is coming in the VPN tunnel. But what is going out?

root@router:~# tshark -i eth1 -f 'icmp[icmptype] = icmp-timxceed'

0.000083 192.168.3.1 -> 192.168.3.10 ICMP Time-to-live exceeded (Time to live exceeded in transit)
0.021200 10.1.0.1 -> 192.168.3.10 ICMP Time-to-live exceeded (Time to live exceeded in transit)
0.024700 172.16.64.17 -> 192.168.3.10 ICMP Time-to-live exceeded (Time to live exceeded in transit)
...

I could see the initial “TTL expired” messages from the first few hops, but none after that. Something is stopping them between tun0 (the OpenVPN tun interface) and eth1 (the inside LAN interface).

Can you see where I’m going with this? It’s our old friend, the reverse-path filter. Disable it with sysctl or echo.

echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter

For some reason, the kernel doesn’t mind most traffic moving between eth1 and tun0, but something about the ICMP “TTL expired” messages triggered rp_filter. Disable it and traceroute works again.

Aside: I always use mtr in place of regular traceroute. What traceroute does in O(n), mtr does in O(1). Actually O(n/5), but explaining that needs it own post. mtr is fast. You should use mtr. Your distro has it.