Does IPv6 support Jumbograms ?

Hi there,

I've been recently running performance tests with the loopback interface increasing the MTU over the 65535 byte limit.
I was really surprised to see that a simple scp onto the ::1 address systematically blocked after transferring about 2,4 MB.
My interpretation of this behavior is that the current IPv6 kernel layer does not support Jumbograms at all. Am I wrong ?
If that's not the case, what could then the right interpretation of this issue ?
And whenever I'm right, is there any plan to support this feature in a near future ?

Thanks in advance for any help.
Regards,

François

Hi!

IPv6 should handle the reception of jumbo frames quite fine, even the UDP size override IIRC. But there is no support for large outgoing UDP frames, and I currently don't know about TCP-MSS override works.

I guess you should be able to send jumbos with rawsocket if you attach the dsthop header manually, but I have never tried it (maybe it will get blocked by some check in ip6_output, HDRINCL and H-b-H option setsockopt could work also).

I currently don't see the need for that option if no one also takes care about UDP and TCP paths, so I guess support is coming as soon as someone cares. ;)

Greetings,

Hannes

What do you mean by blocked ?

Please give more details (kernel version, exact mtu...), because it should not happen !

# ifconfig lo mtu 100000
# scp -6 vmlinux edumazet@ip6-localhost:/tmp
Password: 
vmlinux
100%   24MB  23.5MB/s   00:00    
# ls -l /tmp/vmlinux
-rwxr-xr-x 1 edumazet eng 24668915 Apr  9 14:37 /tmp/vmlinux

I couldn't test it on my development system with a recent net-next kernel yet, but on my laptop with a distribution 3.13.9 kernel this happened too.

I tested with two netcats and threw a dd from /dev/zero into it with a 1MB block size. ssh seems not to be aggressive enough to trigger this:

# tcpdump -i lo -vvv -nKS tcp and port 9988
# tcpdump -i lo -nKS -vvv tcp and port 9988
tcpdump: listening on lo, link-type EN10MB (Ethernet), capture size 65535 bytes
00:26:02.914511 IP6 (hlim 64, next-header TCP (6) payload length: 40) ::1.52141 > ::1.nsesrvr: Flags [S], seq 2750352628, win 43690, options [mss 65535,sackOK,TS val 43288646 ecr 0,nop,wscale 7], length 0
00:26:02.914593 IP6 (hlim 64, next-header TCP (6) payload length: 40) ::1.nsesrvr > ::1.52141: Flags [S.], seq 1418051855, ack 2750352629, win 43690, options [mss 65535,sackOK,TS val 43288647 ecr 43288646,nop,wscale 7], length 0
00:26:02.914659 IP6 (hlim 64, next-header TCP (6) payload length: 32) ::1.52141 > ::1.nsesrvr: Flags [.], seq 2750352629, ack 1418051856, win 342, options [nop,nop,TS val 43288647 ecr 43288647], length 0
00:26:02.915013 IP6 (hlim 64, next-header TCP (6) payload length: 8224) ::1.52141 > ::1.nsesrvr: Flags [P.], seq 2750352629:2750360821, ack 1418051856, win 342, options [nop,nop,TS val 43288647 ecr 43288647], length 8192
00:26:02.915089 IP6 (hlim 64, next-header TCP (6) payload length: 32) ::1.nsesrvr > ::1.52141: Flags [.], seq 1418051856, ack 2750360821, win 1366, options [nop,nop,TS val 43288647 ecr 43288647], length 0
00:26:02.915289 IP6 (hlim 64, next-header TCP (6) payload length: 8224) ::1.52141 > ::1.nsesrvr: Flags [P.], seq 2750360821:2750369013, ack 1418051856, win 342, options [nop,nop,TS val 43288647 ecr 43288647], length 8192
00:26:02.915342 IP6 (hlim 64, next-header TCP (6) payload length: 32) ::1.nsesrvr > ::1.52141: Flags [.], seq 1418051856, ack 2750369013, win 2389, options [nop,nop,TS val 43288647 ecr 43288647], length 0
00:26:02.915467 IP6 (hlim 64, next-header TCP (6) payload length: 8224) ::1.52141 > ::1.nsesrvr: Flags [P.], seq 2750369013:2750377205, ack 1418051856, win 342, options [nop,nop,TS val 43288647 ecr 43288647], length 8192
00:26:02.915558 IP6 (hlim 64, next-header TCP (6) payload length: 32) ::1.nsesrvr > ::1.52141: Flags [.], seq 1418051856, ack 2750377205, win 3413, options [nop,nop,TS val 43288648 ecr 43288647], length 0
00:26:02.915723 IP6 (hlim 64, next-header TCP (6) payload length: 8224) ::1.52141 > ::1.nsesrvr: Flags [P.], seq 2750377205:2750385397, ack 1418051856, win 342, options [nop,nop,TS val 43288648 ecr 43288648], length 8192
00:26:02.915778 IP6 (hlim 64, next-header TCP (6) payload length: 32) ::1.nsesrvr > ::1.52141: Flags [.], seq 1418051856, ack 2750385397, win 3639, options [nop,nop,TS val 43288648 ecr 43288648], length 0
00:26:02.915898 IP6 (hlim 64, next-header TCP (6) payload length: 8224) ::1.52141 > ::1.nsesrvr: Flags [P.], seq 2750385397:2750393589, ack 1418051856, win 342, options [nop,nop,TS val 43288648 ecr 43288648], length 8192
00:26:02.915952 IP6 (hlim 64, next-header TCP (6) payload length: 32) ::1.nsesrvr > ::1.52141: Flags [.], seq 1418051856, ack 2750393589, win 3639, options [nop,nop,TS val 43288648 ecr 43288648], length 0
00:26:02.916072 IP6 (hlim 64, next-header TCP (6) payload length: 8224) ::1.52141 > ::1.nsesrvr: Flags [P.], seq 2750393589:2750401781, ack 1418051856, win 342, options [nop,nop,TS val 43288648 ecr 43288648], length 8192
00:26:02.916126 IP6 (hlim 64, next-header TCP (6) payload length: 32) ::1.nsesrvr > ::1.52141: Flags [.], seq 1418051856, ack 2750401781, win 3639, options [nop,nop,TS val 43288648 ecr 43288648], length 0
00:26:02.916245 IP6 (hlim 64, next-header TCP (6) payload length: 8224) ::1.52141 > ::1.nsesrvr: Flags [P.], seq 2750401781:2750409973, ack 1418051856, win 342, options [nop,nop,TS val 43288648 ecr 43288648], length 8192

00:26:02.916850 IP6 (hlim 64, next-header TCP (6) payload length: 19) ::1.52141 > ::1.nsesrvr: [|tcp]
00:26:02.918682 IP6 (hlim 64, next-header TCP (6) payload length: 19) ::1.52141 > ::1.nsesrvr: [|tcp]
00:26:02.919852 IP6 (hlim 64, next-header TCP (6) payload length: 19) ::1.52141 > ::1.nsesrvr: [|tcp]
00:26:02.920966 IP6 (hlim 64, next-header TCP (6) payload length: 19) ::1.52141 > ::1.nsesrvr: [|tcp]
00:26:02.922092 IP6 (hlim 64, next-header TCP (6) payload length: 19) ::1.52141 > ::1.nsesrvr: [|tcp]
00:26:02.923194 IP6 (hlim 64, next-header TCP (6) payload length: 19) ::1.52141 > ::1.nsesrvr: [|tcp]
00:26:02.955634 IP6 (hlim 64, next-header TCP (6) payload length: 32) ::1.nsesrvr > ::1.52141: Flags [.], seq 1418051856, ack 2750409973, win 3639, options [nop,nop,TS val 43288688 ecr 43288648], length 0
00:26:02.955832 IP6 (hlim 64, next-header TCP (6) payload length: 19) ::1.52141 > ::1.nsesrvr: [|tcp]
00:26:03.160698 IP6 (hlim 64, next-header TCP (6) payload length: 19) ::1.52141 > ::1.nsesrvr: [|tcp]
00:26:03.572148 IP6 (hlim 64, next-header TCP (6) payload length: 19) ::1.52141 > ::1.nsesrvr: [|tcp]
00:26:04.394117 IP6 (hlim 64, next-header TCP (6) payload length: 19) ::1.52141 > ::1.nsesrvr: [|tcp]

Hmm... that is strange.

Bye,

Hannes

Oh well, it seems ip6_mtu() needs to cap mtu to max mtu for non jumbograms...

- return mtu;
+ return min_t(unsigned int, mtu, IP6_MAX_MTU);

#define IP6_MAX_MTU (65535 + 40)

I thought ip6_default_advmss will deal with this which correctly limits the mtu. Capping dst_mtu would make it harder to implement jumbograms some day.

I'll look into it, thank you.

I doubt we'll do this (jumbo grams)

This is a terrible academic idea.

^ minus

I think we should do that. Otherwise we would have to start supporting jumbograms in the output path generally.

Greetings,

Hannes

Why 'minus' ?

hdr->payload_len = htons(seg_len);

seg_len doesn't include the IPv6 header.

So it seems we can use up to 65575 for the device mtu ?

Anyway I am testing the patch ;)

From: Eric Dumazet <[email protected]>

Francois reported that setting big mtu on loopback device could prevent tcp sessions making progress.

We do not support (yet ?) IPv6 Jumbograms and cook corrupted packets.

We must limit the IPv6 MTU to (65535 + 40) bytes in theory.

In practice, its better to align to a multiple of 4 for optimal TCP performance.

Tested:

ifconfig lo mtu 70000
netperf -H ::1

Before patch :

-> Throughput : 0.05 Mbits

After patch :

-> Throughput : 36624.24 Mbits

---
 include/net/ip6_route.h |    7 +++++++
 net/ipv6/route.c        |    5 +++--
 2 files changed, 10 insertions(+), 2 deletions(-)

index 3c3bb184eb8f..8510a0c14b1d 100644
@@ -32,6 +32,13 @@ struct route_info {
 #define RT6_LOOKUP_F_SRCPREF_PUBLIC	0x00000010
 #define RT6_LOOKUP_F_SRCPREF_COA	0x00000020
 
+/* We do not (yet ?) support IPv6 jumbograms (RFC 2675)
+ * Unlike IPv4, hdr->seg_len doesn't include the IPv6 header
+ * In theory, limit is 0xFFFF + 40, but we round-down to a multiple
+ * of 4 for better TCP performance.
+ */
+#define IP6_MAX_MTU (0xFFFC + sizeof(struct ipv6hdr))
+
 /*
  * rt6_srcprefs2flags() and rt6_flags2srcprefs() translate
  * between IPV6_ADDR_PREFERENCES socket option values

index 5015c50a5ba7..5ea462eacd9f 100644
@@ -1338,7 +1338,7 @@ static unsigned int ip6_mtu(const struct dst_entry *dst)
 	unsigned int mtu = dst_metric_raw(dst, RTAX_MTU);
 
 	if (mtu)
-		return mtu;
+		goto out;
 
 	mtu = IPV6_MIN_MTU;
 

@@ -1348,7 +1348,8 @@ static unsigned int ip6_mtu(const struct dst_entry *dst)
 		mtu = idev->cnf.mtu6;
 	rcu_read_unlock();
 
-	return mtu;
+out:
+	return min_t(unsigned int, mtu, IP6_MAX_MTU);
 }
 
 static struct dst_entry *icmp6_dst_gc_list;

Same story for IPv4 btw : current lo mtu=65536 is suboptimal

lpq83:~# ifconfig lo mtu 65536
lpq83:~# ./netperf 
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
localhost () port 0 AF_INET
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

87380 16384 16384 10.00 33020.76

lpq83:~# ifconfig lo mtu 65533
lpq83:~# ./netperf 
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
localhost () port 0 AF_INET
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

87380 16384 16384 10.00 39010.09

It is a TCP issue. We should not limit the mtu to 65532+40. I am for 65535+40. Otherwise, other protocol such as UDP cannot use full mtu as before.

--yoshfuji

I have not seen problems with max ipv6 mtu limit of 65535+40 and tcp.

I agree this would be a better approach, and maybe & ~3 the mtu/mss in tcp code? I assume people expect maximum udp packet sizes working over loopback?

Greetings,

Hannes

Yes, recent Intel cpus have really fast mem copy, even with different alignments for source/dest.

But old cpus or other arches might have an issue here. Apparently nobody really cares, so I wont argue and send a V2.

From: Eric Dumazet

Yes, they've finally thrown a small amount of logic at it. Probably a cache line sized barrel shifter.
'rep movsb' has also been optimised for short transfers (but is slower for longer ones).

At least the zero length 'rep movsb' at the end of the inlined memcpy will not take 40+ clocks on those cpus.

David

From: Hannes Frederic Sowa

The 'trick' would be to generate slightly short messages in the tcp transmit code when doing so wouldn't generate an extra packet. So (using very small numbers) if the max userdata was 63 bytes a 120 byte transmit would generate two 60 byte packets, but 121 to 126 byte transmits would still generate two packets.

David

From: Eric Dumazet <[email protected]>

Francois reported that setting big mtu on loopback device could prevent tcp sessions making progress.

We do not support (yet ?) IPv6 Jumbograms and cook corrupted packets.

We must limit the IPv6 MTU to (65535 + 40) bytes in theory.

Tested:

ifconfig lo mtu 70000
netperf -H ::1

Before patch : Throughput : 0.05 Mbits

After patch : Throughput : 35484 Mbits

---
v2: set the max limit, not round-down it.

 include/net/ip6_route.h |    5 +++++
 net/ipv6/route.c        |    5 +++--
 2 files changed, 8 insertions(+), 2 deletions(-)

index 3c3bb184eb8f..6c4f5eac98e7 100644
@@ -32,6 +32,11 @@ struct route_info {
 #define RT6_LOOKUP_F_SRCPREF_PUBLIC	0x00000010
 #define RT6_LOOKUP_F_SRCPREF_COA	0x00000020
 
+/* We do not (yet ?) support IPv6 jumbograms (RFC 2675)
+ * Unlike IPv4, hdr->seg_len doesn't include the IPv6 header
+ */
+#define IP6_MAX_MTU (0xFFFF + sizeof(struct ipv6hdr))
+
 /*
  * rt6_srcprefs2flags() and rt6_flags2srcprefs() translate
  * between IPV6_ADDR_PREFERENCES socket option values

index 5015c50a5ba7..5ea462eacd9f 100644
@@ -1338,7 +1338,7 @@ static unsigned int ip6_mtu(const struct dst_entry *dst)
 	unsigned int mtu = dst_metric_raw(dst, RTAX_MTU);
 
 	if (mtu)
-		return mtu;
+		goto out;
 
 	mtu = IPV6_MIN_MTU;
 

@@ -1348,7 +1348,8 @@ static unsigned int ip6_mtu(const struct dst_entry *dst)
 		mtu = idev->cnf.mtu6;
 	rcu_read_unlock();
 
-	return mtu;
+out:
+	return min_t(unsigned int, mtu, IP6_MAX_MTU);
 }
 
 static struct dst_entry *icmp6_dst_gc_list;

--yoshfuji

Eric,

Oops, you need to change the subject as well.
Otherwise, I am okay.

Thanks a lot.

--yoshfuji

From: Eric Dumazet <[email protected]>
Date: Thu, 10 Apr 2014 20:20:27 -0700

Ugh, sorry, I had to run. I'll resend.

From: Eric Dumazet <[email protected]>

Francois reported that setting big mtu on loopback device could prevent tcp sessions making progress.

We do not support (yet ?) IPv6 Jumbograms and cook corrupted packets.

We must limit the IPv6 MTU to (65535 + 40) bytes in theory.

Tested:

ifconfig lo mtu 70000
netperf -H ::1

Before patch : Throughput : 0.05 Mbits

After patch : Throughput : 35484 Mbits

---
v3: fix title... humpff...

 include/net/ip6_route.h |    5 +++++
 net/ipv6/route.c        |    5 +++--
 2 files changed, 8 insertions(+), 2 deletions(-)

index 3c3bb184eb8f..6c4f5eac98e7 100644
@@ -32,6 +32,11 @@ struct route_info {
 #define RT6_LOOKUP_F_SRCPREF_PUBLIC	0x00000010
 #define RT6_LOOKUP_F_SRCPREF_COA	0x00000020
 
+/* We do not (yet ?) support IPv6 jumbograms (RFC 2675)
+ * Unlike IPv4, hdr->seg_len doesn't include the IPv6 header
+ */
+#define IP6_MAX_MTU (0xFFFF + sizeof(struct ipv6hdr))
+
 /*
  * rt6_srcprefs2flags() and rt6_flags2srcprefs() translate
  * between IPV6_ADDR_PREFERENCES socket option values

index 5015c50a5ba7..5ea462eacd9f 100644
@@ -1338,7 +1338,7 @@ static unsigned int ip6_mtu(const struct dst_entry *dst)
 	unsigned int mtu = dst_metric_raw(dst, RTAX_MTU);
 
 	if (mtu)
-		return mtu;
+		goto out;
 
 	mtu = IPV6_MIN_MTU;
 

@@ -1348,7 +1348,8 @@ static unsigned int ip6_mtu(const struct dst_entry *dst)
 		mtu = idev->cnf.mtu6;
 	rcu_read_unlock();
 
-	return mtu;
+out:
+	return min_t(unsigned int, mtu, IP6_MAX_MTU);
 }
 
 static struct dst_entry *icmp6_dst_gc_list;

Eric, the optimization for limiting the mtu in tcp case is still possible. It is not that people don't care. ;)

Thanks,

Hannes

I am not sure I want to add an additional test in TCP, as current systems do not really have performance issues. I would 99.99% of the time, MSS is already a multiple of 4.

I think we have more pressing problems ;)

From: Eric Dumazet <[email protected]>
Date: Thu, 10 Apr 2014 21:23:36 -0700

Applied and queued up for -stable, thanks Eric.

Sorry about my wrong statement earlier in this thread, you're right.

Btw. this patch fixes a flaw in the UDP output path, too, where we didn't generate EMSGSIZE error correctly before in all cases. udp_sendmsg only checks for INT_MAX - sizeof(udphdr) and defers further checking to ip6_append_data which seems to be prepared for jumbograms. ;)

Very cool catch with the possible ipv4 performance improvment, too. ;)

Thanks,

Hannes