Troubleshooting path MTU problems

Not long ago we started having very unusual issues our email servers. Mail would be inexplicably held for delivery, bounce back, or fail to send for hours and then send without issue later. Some users couldn’t fetch email by POP until they restarted their mail client. We investigated the mail software, but weeks of investigation turned up nothing.

Around the same time, we also experienced intermittent problems logging in to MSN Messenger, and some users complained of issues accessing certain web pages, including a lot of HTTPS links. I began to suspect these were related.

When the going gets tough, the tough sniff packets. We can sniff at any point in our core routing network, which is by far the most effective way to find a networking problem. And so I found this:

10.0.2.1 is the upstream router from one of our network providers. 192.168.3.155 is one of our servers. So something is definitely wrong upstream of us.

It’s possible to do most troubleshooting with tshark (wireshark’s command-line form). But I prefer to use SCP to copy the capture file back to my PC and investigate it in the GUI. I captured a few more packets and saved them:

tshark -i eth0 -f 'host 10.0.2.1' -w /tmp/upstream.cap -S

This sniffs on eth0 for packets to or from 10.0.2.1 and saves them to /tmp/upstream.cap while showing me a running summary. I let this run until a few more ICMP errors are captured, then hit Ctrl-C to cancel. I can then either copy the file back to my PC and open it with Wireshark, or directly open it using KDE’s “fish://” KIO handler. Try opening a URL like this in Dolphin:

fish://root@192.168.0.1/tmp/

Just right-click and open the file with Wireshark, and KDE will transparently download the file and start Wireshark for you.

I selected one of the offending packets and saw a packet from 10.0.2.1 with this:

This shows that the MTU of the link between 10.0.2.1 and the next router is 1496 bytes. To explain further, we need to understand how IP and Ethernet MTU interact.

The MTU, or maximum transmission unit, for a normal Ethernet network is typically 1500 bytes. This means that an Ethernet packet may contain up to 1500 bytes of data exclusive of the Ethernet headers. As data moves from one Ethernet to the next, as in the case of all traffic routed over the Internet, it must not exceed that Ethernet’s MTU. If it does, the packet is fragmented, split into smaller packets and transmitted as two or more smaller packets.

IP normally allows for fragmentation. This is an inefficient but effective solution, given that a host on the Internet doesn’t usually know the lowest MTU of the entire path between it and the target it would like to reach. However, IP packets contain a flag in the header, “Do not fragment”. If this bit is set, and a packet needs to be fragmented due to a smaller MTU, it cannot be transmitted onward. The router where this happens will drop the packet and return an ICMP type 3, code 4 message, “Destination unreachable, fragmentation needed but ‘do not fragment’ is set.”

Now that I know what to look for, I can find all kinds of MTU issues.

tshark -i eth0 -f 'icmp[icmptype] == 3 and icmp[icmpcode] == 4' -w /tmp/frag.cap -S

Watching this for a while, I found a lot of remote networks with lower than normal MTUs. ICMP error messages contain copies of the IP and TCP/UDP headers of the offending packet, which makes it easy to find what triggered it. I found several examples of SMTP traffic with “do not fragment” set, trying to enter networks with MTUs as low as 300 bytes. But these are remote networks, and are someone else’s problem to fix. What I shouldn’t have is a low MTU immediately upstream, affecting all my traffic.

Now I know I have an MTU problem. But is this the cause of all my problems? I sniffed again, this time trying to login to MSN. The router at 10.0.2.1 again returned an error:

This one told me the cause of the MSN problem. As part of its authentication, MSN Messenger logs in (to 65.54.165.177) via HTTPS (TCP port 443), with a 1500-byte packet which has the “do not fragment” bit set.

Some solutions to the problem of MTU path discovery exist. Sniffing identified the problem to my satisfaction. But you can test more directly by using a traceroute tool that supports MTU discovery. The process is simple: perform traceroute using IP packets of size 1500 bytes with “do not fragment” set. If they are rejected, back off to the “MTU of next hop” value and repeat until you reach your target. On Linux, use tracepath or traceroute --mtu.

A path MTU problem will appear as a drop in MTU:

On Windows, try mturoute. On Mac OS X, try traceroute -F google.com 1500.

I reported the problem to our upstream provider and included the tracepath output and a few packets captured during testing. They scheduled a maintenance window for the following afternoon and corrected the problem with two minutes of down time. We haven’t had an issue with email, MSN, or web pages since.

Tags: ,

  1. 网通传奇SF’s avatar

    我们老家的赤脚医生最拿手的就是打吊针(输液)

    Reply

  2. 壶关新闻网’s avatar

    当今的ZF的悲哀 在毛时代什么都没有也造出来了2弹1星 没有常规潜艇 却造出来了核潜艇 现在的ZHONG GUO 不知道比60 70年代强多少倍 却没有一点作为 真是为我们的领导人感到悲哀!!!!!!!!

    Reply

  3. 鹿泉新闻网’s avatar

    我发现现在fanqiang的人越来越多了

    Reply

  4. 永济门户网’s avatar

    讨厌韩国,自大的国家,还瞧不起中国人。韩国人比日本人差两条街……

    Reply

  5. 古交新闻网’s avatar

    跟屁!我们要做的不该模仿别国,我们该做的是,研制美国佬见到就跑的战备!醒醒吧!

    Reply

  6. 贵溪门户网’s avatar

    比较支持!拿下小日本。让小日本女人来中国当慰安妇

    Reply

  7. www.qjhgnm.com’s avatar

    当年美国媒体臆测中国不可能出兵朝鲜,孩子们将回家过感恩节。。。。。再开战:琉球群岛回归或者独立,中美划太平洋而治!南沙群岛一战定乾坤,全部回归祖国怀抱!

    Reply

  8. 深圳新闻网’s avatar

    太不可思议了!这种女人、这种案例发人深思!不能让这个女人轻易死去……让她好好思考、哈哈反省、再好好的讲讲:她的出生、成长、结婚生子的人生历程…..让更多的男人女人都接受一次人生的家庭教育、做女人的品德、待人、为人做母、为人作父的良知教育……太可恶了!可给四川人丢人了!给天下的女人丢人了!给天下的父母丢人了!气愤!气愤!气愤!恶有恶报!善有善报!罪不可赦!

    Reply

  9. 宜宾市秀江园林有限公司’s avatar

    GJ涨价委宣布气财油涨价就不让评论,还没得到我军证实的军演就让评论了。真是服了!!

    Reply

  10. COCO奶茶加盟’s avatar

    就你,不配有老婆孩子,去搞男人屁眼都委屈了别人。

    Reply

  11. www.wuqiao.net.cn’s avatar

    支持医生反击!打击医闹!打击试图攻击医务人员肆无忌惮的暴徒!支持修改法律医生打死医闹无罪!!!

    Reply

  12. 广宗新闻网’s avatar

    当哥看了这条新闻,做事不腰疼了,干活也不累了,精神也好多了。睡的也香了。

    Reply

  13. www.cnLianguo.com’s avatar

    连菲律宾的拆迁办都出马了 俺们的马跑那溜达去啦

    Reply

  14. www.sf123.com’s avatar

    学习不是张开嘴说说,说完就继续闯红灯,插队,吐痰,看到那些媚日的狗B教训国人的嘴脸,真他妈想吐,有民族自卑心的麻烦去派出所改姓〃犬养〃

    Reply

  15. www.qingxu.net.cn’s avatar

    我也刚当爸爸不久,是个女儿!看到这新闻,心痛哦!老婆刚怀孕的时候,老婆说要女儿,我说要儿子,到后来生出来的时候,老婆见是女儿有点失望,我就不能理解了,我都不失望,她失望什么?说实话,我现在比她更疼我女儿!自己的孩子,不管是女是男,都一样会疼爱!

    Reply

  16. 我本沉默版本’s avatar

    定期查体体检就好了!工厂里还不是一个大家庭吗?东西都是公用也没出什么问题,但是必须定期体检,说的公筷纯属扯淡,你是怎么长大的还不是父母喂大的,也没见你不喘气了!脱裤子放屁的事儿,只要家人都健康用什么公筷。你谈恋爱还不能和你男(女)朋友亲嘴了,不能秀恩爱吃一个棒棒糖了!一个盘里夹菜也不行,你要隔离了,另类!

    Reply

  17. www.chongqingnews.net.cn’s avatar

    医生医术低劣还打人,真是做人低下贱到家了。

    Reply

  18. 重庆网站建设’s avatar

    等你老公在你月子期间跟你吵架,你再说吧,你嫁的好,怎么能理解别人的辛酸
    重庆网站建设 http://www.cqwzjs.com.cn

    Reply

  19. 新洲网站建设’s avatar

    这个老公不是因为老婆生了个女儿才和她吵架吧!哪能月子里面就吵架,吵架原因都不愿意说,说明是他的错。女人需要的关心爱和理解去哪了?这样的窝囊男人不光是没钱,是没爱。女人摔孩子确实不对,既然有第一次丈夫为什么不阻止,还有第二次的悲剧发生。是否这个男人就根本不喜欢这个孩子呢?
    新洲网站建设 http://www.xzwzjs.com.cn

    Reply

  20. 重庆新闻网’s avatar

    联合国递交国书,任何侵占我国一寸一土的国家将给予还击。美国在菲律宾的战舰敢在我领土海域演习,立刻击沉。给以颜色,在干嚣张,就核战。
    重庆新闻网 http://www.chongqingnews.net.cn/

    Reply

  21. 佛山网站建设’s avatar

    看到这信息!心里好舒服!这才体現出东方雄狮的威猛!
    佛山网站建设 http://www.fswzjs.net.cn

    Reply

  22. 拓展’s avatar

    用公筷子不管是家里还是外面这都是要养成良好习惯。这和你是谁的父亲没有关系。注重细节才能体现民族素质!省的去国外旅行让人家说中国人素质差。
    拓展 https://www.tstysjy.com/

    Reply

  23. 单职业传奇’s avatar

    中国这么多年没有实战了,需要通过实战培养一批人才 了
    单职业传奇 https://www.7000sf.com/

    Reply

  24. 法库新闻网’s avatar

    妈离的远,婆婆照顾的老报怨,其实现在还提倡二胎,我坚决不要,生孩和死一次差不多
    法库新闻网 http://www.fakunews.cn/

    Reply

  25. 丰台网站建设’s avatar

    如果养不起的话,我每个月都可以捐点,也号召大家捐点,但是要在保证没有贪污的情况下!
    丰台网站建设 http://www.ftwzjs.cn/

    Reply

  26. 陕西誉丰源物资有限公司’s avatar

    我觉得很正常,习惯就好,像我老公,他一般不会把它吃过的筷子夹菜给孩子!
    陕西誉丰源物资有限公司 http://www.sxyfywz.com

    Reply

  27. 广州双城热恋创意文化传播有限公司’s avatar

    别老是说得老百姓苦好不好~~~~~你回去查一下资料~~~~现在的中国是历代以来~!~最和平最稳定最富强最自由的政府来的了~!~!中国还处于一个发展阶段~~~你不要老是跟发达国家来比较~~~当然~!~我们提出意见~!~让社会更和谐是好事~~~
    广州双城热恋创意文化传播有限公司 http://www.999Loveyou.com

    Reply

  28. 青山网站建设’s avatar

    我等不急了,我特想杀小日本,快点干,需要杀小日本猛将,我第一个参加,请像我们农民通知
    青山网站建设 http://www.Qswzjs.cn

    Reply

  29. 宁波网站建设’s avatar

    买下的人有功劳!应该奖励,人家欺负到门口才想到航母,再造晚了点,就应该便宜时买,金融危机时铁矿石那么便宜,国家资金多的是不去买矿山和储备铁矿石,担心还要跌,结果等过了人家高价勒索,
    宁波网站建设 http://www.nbwzjs.cn

    Reply

  30. 德兴在线’s avatar

    讲究卫生从自己做起,你讲卫生的人你真的讲卫生吗?不使用公筷就不卫生啦?你夫妻俩接吻就卫生吗?烧烤店的食品好吃吧,它卫生吗?
    德兴在线 http://www.dexingnews.cn/

    Reply

  31. 街道口网站建设’s avatar

    还是日本人最了解中国人,中国人除了侵略历史在对日本人根本不了解。
    街道口网站建设 http://www.Jdkwzjs.cn

    Reply

  32. 玉田新闻网’s avatar

    什么屁话,这不是理由,俺就说她脾气大没什么的。
    玉田新闻网 http://www.yutiannews.cn/

    Reply

  33. 怀柔网站建设’s avatar

    只要中国死刑法加上假冒伪劣者判死刑看谁还敢做伪劣产品出来
    怀柔网站建设 http://www.hrwzjs.com.cn/

    Reply

  34. 北京环世旧机动车经纪有限公司’s avatar

    在日本生活过,去过日本的人就知道日本真的是好!
    北京环世旧机动车经纪有限公司 http://www.bjjjdc.com

    Reply

  35. 东西湖网站建设’s avatar

    女人虽然可恶,但是男的也是人渣,才生孩子11天你吵什么?吵得家破人亡了好了
    东西湖网站建设 http://www.Dxhwzjs.cn

    Reply

  36. 霍州新闻网’s avatar

    虽说看了有点高兴,但还是希望我们自己搞个大型的海陆空军演,在日本海附近。
    霍州新闻网 http://www.huozhounews.com.cn/

    Reply

  37. sf666’s avatar

    人民只不过是国家的理财者,你口袋里的钱还不是国家的钱。只要不流到国外,钞票就会一直在国内流通。怕的就是我们一直在消费国外的产品,当然有今天的局势就是因为我们慢慢觉醒,已减少消费国外的各种产品。
    sf666 http://www.sf666.es

    Reply

  38. 上海新闻网’s avatar

    没有永远的朋友或敌人,只有永恒不变的利益。
    上海新闻网 http://www.shanghainews.net.cn/

    Reply

  39. 东城网站建设’s avatar

    既然 不敢跟人家一争高下 就不要整天嚷嚷着那些东西是自己的 耍嘴皮子 谁不会啊
    东城网站建设 http://www.dcwzjs.com.cn/

    Reply

Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.