IPv6 path discovery oddities - flushing the routing cache resolves
Subsystems (3)
Hello list,
I'm experiencing difficulties with IPv6 path discovery. The setup is quite simple, a machine with native IPv6, no special routing - let's call it the "server" for now. Unfortunately, I seem to loose connectivity multiple times a day - and after digging in, I found this to be "too big" messages that weren't honored at the server. The network consists of something like:
server --- hosting --- others ---- SIXXS tunnel with 1280 MTU --- me.
A "ip -6 route list cache" would show a cached route to my "client", but one without MTU. Then after "ip -6 route flush cache", and after trying to send a large packet (for example issuing "ps uaxww" on an ssh prompt), "ip -6 route list cache" will show a correct MTU.
But after a while, things start to go wrong again, and another "ip -6 route flush cache" is needed.
The server is running 3.8.0-something (ubuntu 12.04 with a newer kernel).
tcpdump shows that on reception of icmpv6 "too big", nothing happens (i.e. the "too big" packet will be sent time and again), and after the "ip -6 route flush cache", suddenly the "too big" message is honored.
I saw a couple of path discovery issues on this list, more specifically one with the subject "IPv6 path MTU discovery broken" earlier this month - but I'm not sure it's the same issue (because the original submitter specifically mentions kernels 3.10 and 3.11 and has a much more complicated routing table).
The "server" just has:
auto br0
iface br0 inet6 static
address 2a01:xxxx:xxx:xxxx::2
netmask 64
gateway fe80::1
bridge_ports eth0
bridge_maxwait 0
Nothing special, no special routes, no routing daemons.
If I can do anything to shine more light on this issue, please tell me so.
Best regards,
Valentijn
Quoted 9 lines · Valentijn Sessink
Hello list,
I'm experiencing difficulties with IPv6 path discovery. The setup is quite simple, a machine with native IPv6, no special routing - let's call it the "server" for now. Unfortunately, I seem to loose connectivity multiple times a day - and after digging in, I found this to be "too big" messages that weren't honored at the server. The network consists of something like:
Quoted 6 lines
server --- hosting --- others ---- SIXXS tunnel with 1280 MTU --- me.
A "ip -6 route list cache" would show a cached route to my "client", but one without MTU. Then after "ip -6 route flush cache", and after trying to send a large packet (for example issuing "ps uaxww" on an ssh prompt), "ip -6 route list cache" will show a correct MTU.
Oh, thats another very good hint.
Quoted 9 lines
But after a while, things start to go wrong again, and another "ip -6 route flush cache" is needed.
The server is running 3.8.0-something (ubuntu 12.04 with a newer kernel).
tcpdump shows that on reception of icmpv6 "too big", nothing happens (i.e. the "too big" packet will be sent time and again), and after the "ip -6 route flush cache", suddenly the "too big" message is honored.
Quoted 5 lines
I saw a couple of path discovery issues on this list, more specifically one with the subject "IPv6 path MTU discovery broken" earlier this month
- but I'm not sure it's the same issue (because the original submitter specifically mentions kernels 3.10 and 3.11 and has a much more complicated routing table).
I do think these two issues are connected. Could you send me a the corresponding ip route output and /proc/net/ipv6_route output for when it works and mtus are correctly handled and when it does not work?
Thanks,
Hannes
Hi,
Thanks for your fast response, please see below. Sorry for the rather long mail, due to the routing tables. Even though they're really short, the output of ipv6_route is extensive...
Quoted 3 lines · Hannes Frederic Sowa
I do think these two issues are connected. Could you send me a the corresponding ip route output and
/proc/net/ipv6_routeoutput for when it works and mtus are correctly handled and when it does not work?
When it works (please note that the routing cache at first doesn't show an MTU, that only happens after a "too big" has been received, hence the difference before and after a "ps uaxww"), I have the following - this is from last night:
# ip -6 ro li 2a01:4f8:200:546b::/64 dev br0 proto kernel metric 256 fe80::/64 dev eth0 proto kernel metric 256 fe80::/64 dev br0 proto kernel metric 256 fe80::/64 dev vnet0 proto kernel metric 256 fe80::/64 dev vnet1 proto kernel metric 256 default via fe80::1 dev br0 metric 1024
# ip -6 ro li cache
2001:7b8:1529:0:cd7a:7681:b261:af09 via fe80::1 dev br0 metric 0
cache
2001:1af8:ff03:3:219:66ff:fe26:6dd via fe80::1 dev br0 metric 0
cache# ps uaxww [... output cut for brevity ...]
# ip -6 ro li cache
2001:7b8:1529:0:cd7a:7681:b261:af09 via fe80::1 dev br0 metric 0
cache
2001:1af8:ff03:3:219:66ff:fe26:6dd via fe80::1 dev br0 metric 0
cache expires 577sec mtu 1280# cat /proc/net/ipv6_route 200107b815290000cd7a7681b261af09 80 00000000000000000000000000000000 00 fe800000000000000000000000000001 00000000 00000001 000062b2 01000023 br0 20011af8ff030003021966fffe2606dd 80 00000000000000000000000000000000 00 fe800000000000000000000000000001 00000000 00000001 00000002 01400023 br0 2a0104f80000a0a10000000000020001 80 00000000000000000000000000000000 00 fe800000000000000000000000000001 00000000 00000000 00000001 01000003 br0 2a0104f80200546b0000000000000000 40 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001 br0 fe800000000000000000000000000000 40 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001 eth0 fe800000000000000000000000000000 40 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001 br0 fe800000000000000000000000000000 40 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001 vnet0 fe800000000000000000000000000000 40 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001 vnet1 00000000000000000000000000000000 00 00000000000000000000000000000000 00 fe800000000000000000000000000001 00000400 00000000 00000000 00000003 br0 00000000000000000000000000000000 00 00000000000000000000000000000000 00 00000000000000000000000000000000 ffffffff 00000001 0001b230 00200200 lo 00000000000000000000000000000001 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00000037 80200001 lo 2a0104f80200546b0000000000000002 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000003 000e6bc4 80200001 lo fe80000000000000d63d7efffee306e9 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 0000003b 80200001 lo fe80000000000000d63d7efffee306e9 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00000000 80200001 lo fe80000000000000fc5400fffe17ee5b 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00000000 80200001 lo fe80000000000000fc5400fffe497447 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00000000 80200001 lo ff000000000000000000000000000000 08 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001 br0 ff000000000000000000000000000000 08 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001 eth0 ff000000000000000000000000000000 08 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001 vnet0 ff000000000000000000000000000000 08 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001 vnet1 00000000000000000000000000000000 00 00000000000000000000000000000000 00 00000000000000000000000000000000 ffffffff 00000001 0001b230 00200200 lo
(Please note, while trying, I found a rather interesting output, showing negative time; if it's normal, please ignore:
# ip -statistics -6 ro li cache
2001:1af8:ff03:3:219:66ff:fe26:6dd via fe80::1 dev br0 metric 0
cache expires -23sec users 2 used 2 age 637sec mtu 1280)Anyway, today, suddenly it doesn't work anymore:
# ip -6 ro li 2a01:4f8:200:546b::/64 dev br0 proto kernel metric 256 fe80::/64 dev eth0 proto kernel metric 256 fe80::/64 dev br0 proto kernel metric 256 fe80::/64 dev vnet0 proto kernel metric 256 fe80::/64 dev vnet1 proto kernel metric 256 fe80::/64 dev vnet2 proto kernel metric 256 default via fe80::1 dev br0 metric 1024
# ip -6 ro li cache
2001:7b8:2ff:37a::2 via fe80::1 dev br0 metric 0
cache
2001:7b8:1529::f via fe80::1 dev br0 metric 0
cache
2001:7b8:1529:0:213:8fff:feda:140c via fe80::1 dev br0 metric 0
cache
2001:7b8:1529:0:b845:bb25:5d99:2a07 via fe80::1 dev br0 metric 0
cache
2a01:4f8:200:546b::5 via 2a01:4f8:200:546b::5 dev br0 metric 0
cache
2a01:4f8:200:546b::2eef via 2a01:4f8:200:546b::2eef dev br0 metric 0
cache# ip -statistics -6 ro li cache
2001:7b8:1529:0:213:8fff:feda:140c via fe80::1 dev br0 metric 0
cache users 4 used 9448
2001:7b8:1529:0:b845:bb25:5d99:2a07 via fe80::1 dev br0 metric 0
cache users 4 used 260467 age 0sec
2a01:4f8:200:546b::2eef via 2a01:4f8:200:546b::2eef dev br0 metric 0
cache users 1 used 987 age 205sec
# ip -6 ro li cache
2001:7b8:1529:0:213:8fff:feda:140c via fe80::1 dev br0 metric 0
cache
2001:7b8:1529:0:b845:bb25:5d99:2a07 via fe80::1 dev br0 metric 0
cache
2a01:4f8:200:546b::2eef via 2a01:4f8:200:546b::2eef dev br0 metric 0
cache
(The second time suddenly a couple of routes were gone - I don't know why). After "cat /proc/net/ipv6_route > /tmp/route-bad" I typed a "route -6 flush cache" and TCP started flowing again. Here's ipv6_route:200107b802ff037a0000000000000002 80 00000000000000000000000000000000 00 fe800000000000000000000000000001 00000000 00000000 00000006 01000003
br0200107b815290000000000000000000f 80 00000000000000000000000000000000 00 fe800000000000000000000000000001 00000000 00000000 00000006 01000003
br0200107b81529000002138ffffeda140c 80 00000000000000000000000000000000 00 fe800000000000000000000000000001 00000000 00000004 000024aa 01000023
br0200107b815290000b845bb255d992a07 80 00000000000000000000000000000000 00 fe800000000000000000000000000001 00000000 00000004 0003f79f 01000023
br02a0104f80200546b0000000000000005 80 00000000000000000000000000000000 00 2a0104f80200546b0000000000000005 00000000 00000000 00000018 01000001
br02a0104f80200546b0000000000002eef 80 00000000000000000000000000000000 00 2a0104f80200546b0000000000002eef 00000000 00000001 000003db 01000001
br02a0104f80200546b0000000000000000 40 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001
br0fe800000000000000000000000000000 40 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001
eth0fe800000000000000000000000000000 40 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001
br0fe800000000000000000000000000000 40 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001 vnet0 fe800000000000000000000000000000 40 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001 vnet1 fe800000000000000000000000000000 40 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001 vnet2 00000000000000000000000000000000 00 00000000000000000000000000000000 00 fe800000000000000000000000000001 00000400 00000000 00000000 00000003
br0
00000000000000000000000000000000 00 00000000000000000000000000000000 00 00000000000000000000000000000000 ffffffff 00000001 00064566 00200200
lo
00000000000000000000000000000001 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00000167 80200001
lo2a0104f80200546b0000000000000000 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00000000 00300001
lo2a0104f80200546b0000000000000002 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 0000000b 002a4bfe 80200001
lofe800000000000000000000000000000 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00000000 00300001
lofe800000000000000000000000000000 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00000000 00300001
lofe800000000000000000000000000000 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00000000 00300001
lofe800000000000000000000000000000 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00000000 00300001
lofe800000000000000000000000000000 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00000000 00300001
lofe80000000000000d63d7efffee306e9 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 000001ed 80200001
lofe80000000000000d63d7efffee306e9 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00000000 80200001
lofe80000000000000fc5400fffe17ee5b 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00000000 80200001
lofe80000000000000fc5400fffe497447 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00000000 80200001
lofe80000000000000fc5400fffedd5a81 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00000000 80200001
loff000000000000000000000000000000 08 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001
br0ff000000000000000000000000000000 08 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001
eth0ff000000000000000000000000000000 08 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001 vnet0 ff000000000000000000000000000000 08 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001 vnet1 ff000000000000000000000000000000 08 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001 vnet2 00000000000000000000000000000000 00 00000000000000000000000000000000 00 00000000000000000000000000000000 ffffffff 00000001 00064566 00200200
lo
I hope this helps.
Best regards,
Valentijn
Quoted 7 lines · Valentijn Sessink
Hi,
Thanks for your fast response, please see below. Sorry for the rather long mail, due to the routing tables. Even though they're really short, the output of
ipv6_routeis extensive...On 16-10-13 17:48, Hannes Frederic Sowa wrote:
Quoted 8 lines
Quoted 3 lines
I do think these two issues are connected. Could you send me a the corresponding ip route output and
/proc/net/ipv6_routeoutput for when it works and mtus are correctly handled and when it does not work?When it works (please note that the routing cache at first doesn't show an MTU, that only happens after a "too big" has been received, hence the difference before and after a "ps uaxww"), I have the following - this is from last night:
Thanks, I needed this to verify I am on the right track replicating this. 2001:1af8:ff03:3:219:66ff:fe26:6dd is the other end of the connection, I guess?
Greetings,
Hannes
Quoted 3 lines · Hannes Frederic Sowa
Thanks, I needed this to verify I am on the right track replicating this. 2001:1af8:ff03:3:219:66ff:fe26:6dd is the other end of the connection, I guess?
Yes, the working connection (first example) is from 2001:1af8:ff03:3:219:66ff:fe26:6dd. The non-working connection should have an MTU of 1280 on the 2001:7b8:1529:: subnet connections (those are tunneled, with the tunnel restricting the MTU).
Best regards,
Valentijn
Quoted 9 lines · Valentijn Sessink
Quoted 3 lines · Hannes Frederic Sowa
Thanks, I needed this to verify I am on the right track replicating this. 2001:1af8:ff03:3:219:66ff:fe26:6dd is the other end of the connection, I guess?
Yes, the working connection (first example) is from 2001:1af8:ff03:3:219:66ff:fe26:6dd. The non-working connection should have an MTU of 1280 on the 2001:7b8:1529:: subnet connections (those are tunneled, with the tunnel restricting the MTU).
I got access to a nice test box yesterday where I could brute force the problem in parallel (it was a PITA). This is what I found:
This first patch solves the problem of a complete lockdown of all sockets towards one ipv6 destination. This can happen if we recheck the ipv6 fib (expiration is ok) and we get back a rt6_info where we apply the new metrics information on. After the check the dst entry expires and we do a relookup. We try to insert the same routing information into the fib which results only in a call to rt6_clean_expires. Because we don't reset the dst.expires value a later update of mtu information won't update the expiration time because of the strange semantics in rt6_update_expires. This patch should fix this.
include/net/ip6_fib.h +1 -0
@@ -164,6 +164,7 @@ static inline struct inet6_dev *ip6_dst_idev(struct dst_entry *dst)static inline void rt6_clean_expires (struct rt6_info *rt) { + rt->dst.expires =0 ; rt->rt6i_flags &= ~RTF_EXPIRES ; }
The second patch resolves the problem that the socket keeps hanging on outdated mtu information which gets invalidated just after processing. We need to relookup the destination entry in case the socket expires. This helps a socket to free the cached dst before applying the mtu information to an already expired dst which will be reinserted (see above, it will only call rt6_clean_expires on the dst_entry). This is normally not a problem, but in the process of the creation of the cloned dst_entry we end up copying the metric information from the non-DST_CACHEd route to the dst_entry (ip6_rt_copy/dst_copy_metrics). Because the information are held in inetpeer storage and the key for the expired dst and the new dst have the same key we overwrite the metrics store which currently is in use by two rt6_infos. So we just invalidate the newly installed metrics information and will use the interface mtu just after the PACKET_TOO_BIG notification, which leads to hangs of the connection. A flush of the cached routing entries causes relookups, so this a workaround.
This patch should fix this:
net/ipv6/route.c +6 -3
@@ -1064,10 +1064,13 @@ static struct dst_entry *ip6_dst_check(struct dst_entry *dst, u32 cookie)if (rt->rt6i_genid !=rt_genid_ipv6 (dev_net (rt->dst.dev)))return NULL ; -if (rt->rt6i_node && (rt->rt6i_node->fn_sernum == cookie)) -return dst; +if (!rt->rt6i_node && (rt->rt6i_node->fn_sernum != cookie)) +return NULL ; -return NULL ; +if (rt6_check_expired (rt)) +return NULL ; + +return dst; }static struct dst_entry *ip6_negative_advice (struct dst_entry *dst)
I had the patches in test for a few hours on some VMs where I could normally reproduce this issue within 5 minutes. They are for testing only and I don't know if they resolve all issues. I also have to check why rt6_update_expires has such strange expiration update logic.
Steinar and Valentijn could you give them a test drive?
Greetings,
Hannes
Quoted 7 lines · Hannes Frederic Sowa
This patch should fix this:
net/ipv6/route.c +0
-0
net/ipv6/route.c static struct dst_entry *ip6_dst_check(struct dst_entry *dst, u32 cookie) · line 1068
Quoted 7 lines
net/ipv6/route.c +2
-2@@ -1064,10 +1064,13 @@ static struct dst_entry *ip6_dst_check(struct dst_entry *dst, u32 cookie)if (rt->rt6i_genid !=rt_genid_ipv6 (dev_net (rt->dst.dev)))return NULL ; -if (rt->rt6i_node && (rt->rt6i_node->fn_sernum == cookie)) -return dst; +if (!rt->rt6i_node && (rt->rt6i_node->fn_sernum != cookie)) +return NULL ;
I just rewrote this patch to have a clean diff. I missed to rewrite the '&&' to '||'.
Correct patch is:
net/ipv6/route.c +6 -3
@@ -1064,10 +1064,13 @@ static struct dst_entry *ip6_dst_check(struct dst_entry *dst, u32 cookie)if (rt->rt6i_genid !=rt_genid_ipv6 (dev_net (rt->dst.dev)))return NULL ; -if (rt->rt6i_node && (rt->rt6i_node->fn_sernum == cookie)) -return dst; +if (!rt->rt6i_node || (rt->rt6i_node->fn_sernum != cookie)) +return NULL ; -return NULL ; +if (rt6_check_expired (rt)) +return NULL ; + +return dst; }static struct dst_entry *ip6_negative_advice (struct dst_entry *dst)
Quoted 2 lines · Hannes Frederic Sowa
I got access to a nice test box yesterday where I could brute force the problem in parallel (it was a PITA). This is what I found:
Thanks so much for looking into this! It's great that to hear that it's not just me being weird =) I'll try rebooting the machine in question into a kernel with these patches later today; given how intermittent it is, it can of course only show if the patches are not fully working (or, over time, make it likely that they are).
/* Steinar */
Signature · 1 line
Homepage: http://www.sesse.net/
op 19-10-13 12:12, Steinar H. Gunderson schreef:
Quoted 2 lines
Thanks so much for looking into this! It's great that to hear that it's not just me being weird =)
I'm not sure the patch addresses weirdness ;-P
On the more serious side: I have this problem with a 3.8.0 kernel (Ubuntu 12.04 with updated kernel). I saw that you (Steinar) experience this with 3.10 and 3.11, iirc. I'm a little bit worried by the fact that this has not been seen for at least six months - and now suddenly pops up.
Does anyone have an idea when this race condition was introduced?
(It could be, that certain patches have been included in Ubuntu 3.8.0-something but not in an official 3.8).
Best regards,
Valentijn
p.s. I won't be able to test any patches until tuesday - sorry.
Quoted 4 lines · Valentijn Sessink
op 19-10-13 12:12, Steinar H. Gunderson schreef:
Quoted 2 lines
Thanks so much for looking into this! It's great that to hear that it's not just me being weird =)
Quoted 8 lines
I'm not sure the patch addresses weirdness ;-P
On the more serious side: I have this problem with a 3.8.0 kernel (Ubuntu 12.04 with updated kernel). I saw that you (Steinar) experience this with 3.10 and 3.11, iirc. I'm a little bit worried by the fact that this has not been seen for at least six months - and now suddenly pops up.
Does anyone have an idea when this race condition was introduced?
Do a bisect. ;)
I had a look at the history but could not find a single commit to blame. Even reference counting changes (e.g. removal of neighbour caching in routes) or something in the way the socket release callbacks are processed by tcp could introduce a way where these problems are more likely to happen. I wondered, too.
A lot of people of the IPv6 community often state that mtu discovery is broken because of filtering packet_too_bigs. Maybe most people assumed broken firewalls and did not actual check if it is the kernels problem. So, thanks again for reporting this issue.
Greetings,
Hannes