Skip to contentSkip to conversation

IPv6 path discovery oddities - flushing the routing cache resolves

DISCUSSION Valentijn Sessink 3 participants 10 messages Oct 16, 2013 – Oct 19, 2013

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

Hello list,

I'm experiencing difficulties with IPv6 path discovery. The setup is quite simple, a machine with native IPv6, no special routing - let's call it the "server" for now. Unfortunately, I seem to loose connectivity multiple times a day - and after digging in, I found this to be "too big" messages that weren't honored at the server. The network consists of something like:

server --- hosting --- others ---- SIXXS tunnel with 1280 MTU --- me.

A "ip -6 route list cache" would show a cached route to my "client", but one without MTU. Then after "ip -6 route flush cache", and after trying to send a large packet (for example issuing "ps uaxww" on an ssh prompt), "ip -6 route list cache" will show a correct MTU.

But after a while, things start to go wrong again, and another "ip -6 route flush cache" is needed.

The server is running 3.8.0-something (ubuntu 12.04 with a newer kernel).

tcpdump shows that on reception of icmpv6 "too big", nothing happens (i.e. the "too big" packet will be sent time and again), and after the "ip -6 route flush cache", suddenly the "too big" message is honored.

I saw a couple of path discovery issues on this list, more specifically one with the subject "IPv6 path MTU discovery broken" earlier this month - but I'm not sure it's the same issue (because the original submitter specifically mentions kernels 3.10 and 3.11 and has a much more complicated routing table).

The "server" just has:

auto br0
iface br0 inet6 static
address 2a01:xxxx:xxx:xxxx::2
netmask 64
gateway fe80::1
bridge_ports eth0
bridge_maxwait 0

Nothing special, no special routes, no routing daemons.

If I can do anything to shine more light on this issue, please tell me so.

Best regards,

Valentijn

#1 lore raw
Quoted 9 lines · Valentijn Sessink

Hello list,

I'm experiencing difficulties with IPv6 path discovery. The setup is quite simple, a machine with native IPv6, no special routing - let's call it the "server" for now. Unfortunately, I seem to loose connectivity multiple times a day - and after digging in, I found this to be "too big" messages that weren't honored at the server. The network consists of something like:

Quoted 6 lines

server --- hosting --- others ---- SIXXS tunnel with 1280 MTU --- me.

A "ip -6 route list cache" would show a cached route to my "client", but one without MTU. Then after "ip -6 route flush cache", and after trying to send a large packet (for example issuing "ps uaxww" on an ssh prompt), "ip -6 route list cache" will show a correct MTU.

Oh, thats another very good hint.

Quoted 9 lines

But after a while, things start to go wrong again, and another "ip -6 route flush cache" is needed.

The server is running 3.8.0-something (ubuntu 12.04 with a newer kernel).

tcpdump shows that on reception of icmpv6 "too big", nothing happens (i.e. the "too big" packet will be sent time and again), and after the "ip -6 route flush cache", suddenly the "too big" message is honored.

Quoted 5 lines

I saw a couple of path discovery issues on this list, more specifically one with the subject "IPv6 path MTU discovery broken" earlier this month
- but I'm not sure it's the same issue (because the original submitter specifically mentions kernels 3.10 and 3.11 and has a much more complicated routing table).

I do think these two issues are connected. Could you send me a the corresponding ip route output and /proc/net/ipv6_route output for when it works and mtus are correctly handled and when it does not work?

Thanks,

Hannes

#2 lore raw

Hi,

Thanks for your fast response, please see below. Sorry for the rather long mail, due to the routing tables. Even though they're really short, the output of ipv6_route is extensive...

Quoted 3 lines · Hannes Frederic Sowa

I do think these two issues are connected. Could you send me a the corresponding ip route output and /proc/net/ipv6_route output for when it works and mtus are correctly handled and when it does not work?

When it works (please note that the routing cache at first doesn't show an MTU, that only happens after a "too big" has been received, hence the difference before and after a "ps uaxww"), I have the following - this is from last night:

# ip -6 ro li
2a01:4f8:200:546b::/64 dev br0  proto kernel  metric 256
fe80::/64 dev eth0  proto kernel  metric 256
fe80::/64 dev br0  proto kernel  metric 256
fe80::/64 dev vnet0  proto kernel  metric 256
fe80::/64 dev vnet1  proto kernel  metric 256
default via fe80::1 dev br0  metric 1024
# ip -6 ro li cache
2001:7b8:1529:0:cd7a:7681:b261:af09 via fe80::1 dev br0  metric 0
    cache
2001:1af8:ff03:3:219:66ff:fe26:6dd via fe80::1 dev br0  metric 0
    cache
# ps uaxww
[... output cut for brevity ...]
# ip -6 ro li cache
2001:7b8:1529:0:cd7a:7681:b261:af09 via fe80::1 dev br0  metric 0
    cache
2001:1af8:ff03:3:219:66ff:fe26:6dd via fe80::1 dev br0  metric 0
    cache  expires 577sec mtu 1280
# cat /proc/net/ipv6_route
200107b815290000cd7a7681b261af09 80 00000000000000000000000000000000 00
fe800000000000000000000000000001 00000000 00000001 000062b2 01000023
  br0
20011af8ff030003021966fffe2606dd 80 00000000000000000000000000000000 00
fe800000000000000000000000000001 00000000 00000001 00000002 01400023
  br0
2a0104f80000a0a10000000000020001 80 00000000000000000000000000000000 00
fe800000000000000000000000000001 00000000 00000000 00000001 01000003
  br0
2a0104f80200546b0000000000000000 40 00000000000000000000000000000000 00
00000000000000000000000000000000 00000100 00000000 00000000 00000001
  br0
fe800000000000000000000000000000 40 00000000000000000000000000000000 00
00000000000000000000000000000000 00000100 00000000 00000000 00000001
 eth0
fe800000000000000000000000000000 40 00000000000000000000000000000000 00
00000000000000000000000000000000 00000100 00000000 00000000 00000001
  br0
fe800000000000000000000000000000 40 00000000000000000000000000000000 00
00000000000000000000000000000000 00000100 00000000 00000000 00000001
vnet0
fe800000000000000000000000000000 40 00000000000000000000000000000000 00
00000000000000000000000000000000 00000100 00000000 00000000 00000001
vnet1
00000000000000000000000000000000 00 00000000000000000000000000000000 00
fe800000000000000000000000000001 00000400 00000000 00000000 00000003
  br0
00000000000000000000000000000000 00 00000000000000000000000000000000 00
00000000000000000000000000000000 ffffffff 00000001 0001b230 00200200
   lo
00000000000000000000000000000001 80 00000000000000000000000000000000 00
00000000000000000000000000000000 00000000 00000001 00000037 80200001
   lo
2a0104f80200546b0000000000000002 80 00000000000000000000000000000000 00
00000000000000000000000000000000 00000000 00000003 000e6bc4 80200001
   lo
fe80000000000000d63d7efffee306e9 80 00000000000000000000000000000000 00
00000000000000000000000000000000 00000000 00000001 0000003b 80200001
   lo
fe80000000000000d63d7efffee306e9 80 00000000000000000000000000000000 00
00000000000000000000000000000000 00000000 00000001 00000000 80200001
   lo
fe80000000000000fc5400fffe17ee5b 80 00000000000000000000000000000000 00
00000000000000000000000000000000 00000000 00000001 00000000 80200001
   lo
fe80000000000000fc5400fffe497447 80 00000000000000000000000000000000 00
00000000000000000000000000000000 00000000 00000001 00000000 80200001
   lo
ff000000000000000000000000000000 08 00000000000000000000000000000000 00
00000000000000000000000000000000 00000100 00000000 00000000 00000001
  br0
ff000000000000000000000000000000 08 00000000000000000000000000000000 00
00000000000000000000000000000000 00000100 00000000 00000000 00000001
 eth0
ff000000000000000000000000000000 08 00000000000000000000000000000000 00
00000000000000000000000000000000 00000100 00000000 00000000 00000001
vnet0
ff000000000000000000000000000000 08 00000000000000000000000000000000 00
00000000000000000000000000000000 00000100 00000000 00000000 00000001
vnet1
00000000000000000000000000000000 00 00000000000000000000000000000000 00
00000000000000000000000000000000 ffffffff 00000001 0001b230 00200200
   lo

(Please note, while trying, I found a rather interesting output, showing negative time; if it's normal, please ignore:

# ip -statistics -6 ro li cache
2001:1af8:ff03:3:219:66ff:fe26:6dd via fe80::1 dev br0  metric 0
    cache  expires -23sec users 2 used 2 age 637sec mtu 1280)

Anyway, today, suddenly it doesn't work anymore:

# ip -6 ro li
2a01:4f8:200:546b::/64 dev br0  proto kernel  metric 256
fe80::/64 dev eth0  proto kernel  metric 256
fe80::/64 dev br0  proto kernel  metric 256
fe80::/64 dev vnet0  proto kernel  metric 256
fe80::/64 dev vnet1  proto kernel  metric 256
fe80::/64 dev vnet2  proto kernel  metric 256
default via fe80::1 dev br0  metric 1024
# ip -6 ro li cache
2001:7b8:2ff:37a::2 via fe80::1 dev br0  metric 0
    cache
2001:7b8:1529::f via fe80::1 dev br0  metric 0
    cache
2001:7b8:1529:0:213:8fff:feda:140c via fe80::1 dev br0  metric 0
    cache
2001:7b8:1529:0:b845:bb25:5d99:2a07 via fe80::1 dev br0  metric 0
    cache
2a01:4f8:200:546b::5 via 2a01:4f8:200:546b::5 dev br0  metric 0
    cache
2a01:4f8:200:546b::2eef via 2a01:4f8:200:546b::2eef dev br0  metric 0
    cache
# ip -statistics -6 ro li cache
2001:7b8:1529:0:213:8fff:feda:140c via fe80::1 dev br0  metric 0
    cache  users 4 used 9448
2001:7b8:1529:0:b845:bb25:5d99:2a07 via fe80::1 dev br0  metric 0
    cache  users 4 used 260467 age 0sec
2a01:4f8:200:546b::2eef via 2a01:4f8:200:546b::2eef dev br0  metric 0
    cache  users 1 used 987 age 205sec
# ip -6 ro li cache
2001:7b8:1529:0:213:8fff:feda:140c via fe80::1 dev br0  metric 0
    cache
2001:7b8:1529:0:b845:bb25:5d99:2a07 via fe80::1 dev br0  metric 0
    cache
2a01:4f8:200:546b::2eef via 2a01:4f8:200:546b::2eef dev br0  metric 0
    cache
(The second time suddenly a couple of routes were gone - I don't know why). After "cat /proc/net/ipv6_route > /tmp/route-bad" I typed a "route -6 flush cache" and TCP started flowing again. Here's ipv6_route:

200107b802ff037a0000000000000002 80 00000000000000000000000000000000 00 fe800000000000000000000000000001 00000000 00000000 00000006 01000003
br0
200107b815290000000000000000000f 80 00000000000000000000000000000000 00 fe800000000000000000000000000001 00000000 00000000 00000006 01000003
br0
200107b81529000002138ffffeda140c 80 00000000000000000000000000000000 00 fe800000000000000000000000000001 00000000 00000004 000024aa 01000023
br0
200107b815290000b845bb255d992a07 80 00000000000000000000000000000000 00 fe800000000000000000000000000001 00000000 00000004 0003f79f 01000023
br0
2a0104f80200546b0000000000000005 80 00000000000000000000000000000000 00 2a0104f80200546b0000000000000005 00000000 00000000 00000018 01000001
br0
2a0104f80200546b0000000000002eef 80 00000000000000000000000000000000 00 2a0104f80200546b0000000000002eef 00000000 00000001 000003db 01000001
br0
2a0104f80200546b0000000000000000 40 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001
br0
fe800000000000000000000000000000 40 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001
eth0
fe800000000000000000000000000000 40 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001
br0
fe800000000000000000000000000000 40 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001 vnet0 fe800000000000000000000000000000 40 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001 vnet1 fe800000000000000000000000000000 40 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001 vnet2 00000000000000000000000000000000 00 00000000000000000000000000000000 00 fe800000000000000000000000000001 00000400 00000000 00000000 00000003
br0
00000000000000000000000000000000 00 00000000000000000000000000000000 00 00000000000000000000000000000000 ffffffff 00000001 00064566 00200200
lo
00000000000000000000000000000001 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00000167 80200001
lo
2a0104f80200546b0000000000000000 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00000000 00300001
lo
2a0104f80200546b0000000000000002 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 0000000b 002a4bfe 80200001
lo
fe800000000000000000000000000000 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00000000 00300001
lo
fe800000000000000000000000000000 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00000000 00300001
lo
fe800000000000000000000000000000 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00000000 00300001
lo
fe800000000000000000000000000000 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00000000 00300001
lo
fe800000000000000000000000000000 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00000000 00300001
lo
fe80000000000000d63d7efffee306e9 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 000001ed 80200001
lo
fe80000000000000d63d7efffee306e9 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00000000 80200001
lo
fe80000000000000fc5400fffe17ee5b 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00000000 80200001
lo
fe80000000000000fc5400fffe497447 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00000000 80200001
lo
fe80000000000000fc5400fffedd5a81 80 00000000000000000000000000000000 00 00000000000000000000000000000000 00000000 00000001 00000000 80200001
lo
ff000000000000000000000000000000 08 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001
br0
ff000000000000000000000000000000 08 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001
eth0
ff000000000000000000000000000000 08 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001 vnet0 ff000000000000000000000000000000 08 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001 vnet1 ff000000000000000000000000000000 08 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001 vnet2 00000000000000000000000000000000 00 00000000000000000000000000000000 00 00000000000000000000000000000000 ffffffff 00000001 00064566 00200200
lo

I hope this helps.

Best regards,

Valentijn

#3 lore raw
Quoted 7 lines · Valentijn Sessink

Hi,

Thanks for your fast response, please see below. Sorry for the rather long mail, due to the routing tables. Even though they're really short, the output of ipv6_route is extensive...

On 16-10-13 17:48, Hannes Frederic Sowa wrote:

Quoted 8 lines
Quoted 3 lines

I do think these two issues are connected. Could you send me a the corresponding ip route output and /proc/net/ipv6_route output for when it works and mtus are correctly handled and when it does not work?

When it works (please note that the routing cache at first doesn't show an MTU, that only happens after a "too big" has been received, hence the difference before and after a "ps uaxww"), I have the following - this is from last night:

Thanks, I needed this to verify I am on the right track replicating this. 2001:1af8:ff03:3:219:66ff:fe26:6dd is the other end of the connection, I guess?

Greetings,

Hannes

#4 lore raw
Quoted 3 lines · Hannes Frederic Sowa

Thanks, I needed this to verify I am on the right track replicating this. 2001:1af8:ff03:3:219:66ff:fe26:6dd is the other end of the connection, I guess?

Yes, the working connection (first example) is from 2001:1af8:ff03:3:219:66ff:fe26:6dd. The non-working connection should have an MTU of 1280 on the 2001:7b8:1529:: subnet connections (those are tunneled, with the tunnel restricting the MTU).

Best regards,

Valentijn

#5 lore raw
Quoted 9 lines · Valentijn Sessink
Quoted 3 lines · Hannes Frederic Sowa

Thanks, I needed this to verify I am on the right track replicating this. 2001:1af8:ff03:3:219:66ff:fe26:6dd is the other end of the connection, I guess?

Yes, the working connection (first example) is from 2001:1af8:ff03:3:219:66ff:fe26:6dd. The non-working connection should have an MTU of 1280 on the 2001:7b8:1529:: subnet connections (those are tunneled, with the tunnel restricting the MTU).

I got access to a nice test box yesterday where I could brute force the problem in parallel (it was a PITA). This is what I found:

This first patch solves the problem of a complete lockdown of all sockets towards one ipv6 destination. This can happen if we recheck the ipv6 fib (expiration is ok) and we get back a rt6_info where we apply the new metrics information on. After the check the dst entry expires and we do a relookup. We try to insert the same routing information into the fib which results only in a call to rt6_clean_expires. Because we don't reset the dst.expires value a later update of mtu information won't update the expiration time because of the strange semantics in rt6_update_expires. This patch should fix this.

include/net/ip6_fib.h +1 -0
index 6738f34..3932633 100644
@@ -164,6 +164,7 @@ static inline struct inet6_dev *ip6_dst_idev(struct dst_entry *dst)
 
 static inline void rt6_clean_expires(struct rt6_info *rt)
 {
+	rt->dst.expires = 0;
 	rt->rt6i_flags &= ~RTF_EXPIRES;
 }
 

The second patch resolves the problem that the socket keeps hanging on outdated mtu information which gets invalidated just after processing. We need to relookup the destination entry in case the socket expires. This helps a socket to free the cached dst before applying the mtu information to an already expired dst which will be reinserted (see above, it will only call rt6_clean_expires on the dst_entry). This is normally not a problem, but in the process of the creation of the cloned dst_entry we end up copying the metric information from the non-DST_CACHEd route to the dst_entry (ip6_rt_copy/dst_copy_metrics). Because the information are held in inetpeer storage and the key for the expired dst and the new dst have the same key we overwrite the metrics store which currently is in use by two rt6_infos. So we just invalidate the newly installed metrics information and will use the interface mtu just after the PACKET_TOO_BIG notification, which leads to hangs of the connection. A flush of the cached routing entries causes relookups, so this a workaround.

This patch should fix this:

net/ipv6/route.c +6 -3
index c3130ff..7629022 100644
@@ -1064,10 +1064,13 @@ static struct dst_entry *ip6_dst_check(struct dst_entry *dst, u32 cookie)
 	if (rt->rt6i_genid != rt_genid_ipv6(dev_net(rt->dst.dev)))
 		return NULL;
 
-	if (rt->rt6i_node && (rt->rt6i_node->fn_sernum == cookie))
-		return dst;
+	if (!rt->rt6i_node && (rt->rt6i_node->fn_sernum != cookie))
+		return NULL;
 
-	return NULL;
+	if (rt6_check_expired(rt))
+		return NULL;
+
+	return dst;
 }
 
 static struct dst_entry *ip6_negative_advice(struct dst_entry *dst)

I had the patches in test for a few hours on some VMs where I could normally reproduce this issue within 5 minutes. They are for testing only and I don't know if they resolve all issues. I also have to check why rt6_update_expires has such strange expiration update logic.

Steinar and Valentijn could you give them a test drive?

Greetings,

Hannes

#6 lore raw
Quoted 7 lines · Hannes Frederic Sowa

This patch should fix this:

net/ipv6/route.c +0 -0
index c3130ff..7629022 100644

net/ipv6/route.c static struct dst_entry *ip6_dst_check(struct dst_entry *dst, u32 cookie) · line 1068

Quoted 7 lines
net/ipv6/route.c +2 -2
@@ -1064,10 +1064,13 @@ static struct dst_entry *ip6_dst_check(struct dst_entry *dst, u32 cookie)
 	if (rt->rt6i_genid != rt_genid_ipv6(dev_net(rt->dst.dev)))
 		return NULL;
 
-	if (rt->rt6i_node && (rt->rt6i_node->fn_sernum == cookie))
-		return dst;
+	if (!rt->rt6i_node && (rt->rt6i_node->fn_sernum != cookie))
+		return NULL;

I just rewrote this patch to have a clean diff. I missed to rewrite the '&&' to '||'.

Correct patch is:

net/ipv6/route.c +6 -3
index c3130ff..17b3cc1 100644
@@ -1064,10 +1064,13 @@ static struct dst_entry *ip6_dst_check(struct dst_entry *dst, u32 cookie)
 	if (rt->rt6i_genid != rt_genid_ipv6(dev_net(rt->dst.dev)))
 		return NULL;
 
-	if (rt->rt6i_node && (rt->rt6i_node->fn_sernum == cookie))
-		return dst;
+	if (!rt->rt6i_node || (rt->rt6i_node->fn_sernum != cookie))
+		return NULL;
 
-	return NULL;
+	if (rt6_check_expired(rt))
+		return NULL;
+
+	return dst;
 }
 
 static struct dst_entry *ip6_negative_advice(struct dst_entry *dst)
#7 lore raw
In reply to Hannes Frederic Sowa
Quoted 2 lines · Hannes Frederic Sowa

I got access to a nice test box yesterday where I could brute force the problem in parallel (it was a PITA). This is what I found:

Thanks so much for looking into this! It's great that to hear that it's not just me being weird =) I'll try rebooting the machine in question into a kernel with these patches later today; given how intermittent it is, it can of course only show if the patches are not fully working (or, over time, make it likely that they are).

/* Steinar */

Signature · 1 line
Homepage: http://www.sesse.net/
#8 lore raw

op 19-10-13 12:12, Steinar H. Gunderson schreef:

Quoted 2 lines

Thanks so much for looking into this! It's great that to hear that it's not just me being weird =)

I'm not sure the patch addresses weirdness ;-P

On the more serious side: I have this problem with a 3.8.0 kernel (Ubuntu 12.04 with updated kernel). I saw that you (Steinar) experience this with 3.10 and 3.11, iirc. I'm a little bit worried by the fact that this has not been seen for at least six months - and now suddenly pops up.

Does anyone have an idea when this race condition was introduced?

(It could be, that certain patches have been included in Ubuntu 3.8.0-something but not in an official 3.8).

Best regards,

Valentijn
p.s. I won't be able to test any patches until tuesday - sorry.

#9 lore raw
Quoted 4 lines · Valentijn Sessink

op 19-10-13 12:12, Steinar H. Gunderson schreef:

Quoted 2 lines

Thanks so much for looking into this! It's great that to hear that it's not just me being weird =)

Quoted 8 lines

I'm not sure the patch addresses weirdness ;-P

On the more serious side: I have this problem with a 3.8.0 kernel (Ubuntu 12.04 with updated kernel). I saw that you (Steinar) experience this with 3.10 and 3.11, iirc. I'm a little bit worried by the fact that this has not been seen for at least six months - and now suddenly pops up.

Does anyone have an idea when this race condition was introduced?

Do a bisect. ;)

I had a look at the history but could not find a single commit to blame. Even reference counting changes (e.g. removal of neighbour caching in routes) or something in the way the socket release callbacks are processed by tcp could introduce a way where these problems are more likely to happen. I wondered, too.

A lot of people of the IPv6 community often state that mtu discovery is broken because of filtering packet_too_bigs. Maybe most people assumed broken firewalls and did not actual check if it is the kernels problem. So, thanks again for reporting this issue.

Greetings,

Hannes