TSO Reloaded

Ok, here it is, first draft of the new TSO handling I promised so long ago :-) I was lazy and waited until today to implement the entire thing.

It works with basic testing over tg3.

I'll discuss the changes in more detail tomorrow, but the only potentially sore spot right now is the tcp_push_one() avoidance done in tcp_sendmsg() and tcp_sendpages(). It may need to be changed to do so mething like "if not TSO then tcp_push_one(), else wait for N packets to accumulate" where N is configurable or dynamically measured in some way.

It would be nice to know that this thing works with e1000 and other TSO capable cards, also I did not do much sendfile() testing at all.

Thanks.

@@ -280,8 +280,7 @@ struct tcp_sock {
 	__u32	snd_wnd;	/* The window we expect to receive	*/
 	__u32	max_window;	/* Maximal window ever seen from peer	*/
 	__u32	pmtu_cookie;	/* Last pmtu seen by socket		*/
-	__u32	mss_cache;	/* Cached effective mss, not including SACKS */
-	__u16	mss_cache_std;	/* Like mss_cache, but without TSO */
+	__u16	mss_cache;	/* Cached effective mss, not including SACKS */
 	__u16	ext_header_len;	/* Network protocol overhead (IP/IPv6 options) */
 	__u8	ca_state;	/* State of fast-retransmit machine 	*/
 	__u8	retransmits;	/* Number of unrecovered RTO timeouts.	*/

@@ -1128,13 +1128,16 @@ static inline void sk_stream_moderate_sn
 static inline struct sk_buff *sk_stream_alloc_pskb(struct sock *sk,
 						   int size, int mem, int gfp)
 {
-	struct sk_buff *skb = alloc_skb(size + sk->sk_prot->max_header, gfp);
+	struct sk_buff *skb;
+	int hdr_len;
 
+	hdr_len = SKB_DATA_ALIGN(sk->sk_prot->max_header);
+	skb = alloc_skb(size + hdr_len, gfp);
 	if (skb) {
 		skb->truesize += mem;
 		if (sk->sk_forward_alloc >= (int)skb->truesize ||
 		    sk_stream_mem_schedule(sk, skb->truesize, 0)) {
-			skb_reserve(skb, sk->sk_prot->max_header);
+			skb_reserve(skb, hdr_len);
 			return skb;
 		}
 		__kfree_skb(skb);

@@ -2060,7 +2060,7 @@ static int tcp_v4_init_sock(struct sock 
 	 */
 	tp->snd_ssthresh = 0x7fffffff;	/* Infinity */
 	tp->snd_cwnd_clamp = ~0;
-	tp->mss_cache_std = tp->mss_cache = 536;
+	tp->mss_cache = 536;
 
 	tp->reordering = sysctl_tcp_reordering;
 

@@ -2021,7 +2021,7 @@ static int tcp_v6_init_sock(struct sock 
 	 */
 	tp->snd_ssthresh = 0x7fffffff;
 	tp->snd_cwnd_clamp = ~0;
-	tp->mss_cache_std = tp->mss_cache = 536;
+	tp->mss_cache = 536;
 
 	tp->reordering = sysctl_tcp_reordering;
 

Is anyone testing this or is everyone too scared? :-)

A lot of people wanted this TSO stuff fixed, so I spent time implementing the fix. Yet nobody seems interested enough to even say "it works" or "it's busted" :-/

I can give it a quick sniff on e1000 and r8169 (on amd64), though only to verify that its works (ie, no real perf). Is that acceptable?

Thanks,
Jon

It worked fine for both r8169 and e1000 on my amd64 box (running 2.6.12-rc3-mm2).

When I can get my machines booting to userspace Im definitely going to try it out :)

Anton

Do you need anything besides that early-packet fix from Herbert?

Nope. I'll spin the patch up now and report back.

Anton

It works on tg3 that is too stupid to do TSO at least. More tests coming

It's off by default, "ethtool -K eth0 tso on" is necessary. I'll change that soon.

Or do you have a chip revision that really can't do TSO?

It's in the queue, Dave :).
Let you know if there are problems...

thanks,
Nivedita

We will be testing on 10GbE NICs in the next couple weeks; will let you know.
BTW - any plans for IPv6 support?

What exactly does your NIC expect? Do you use the NETIF_F_HW_CSUM flag to indicate generic checksumming support? Otherwise, there is no other way to support ipv6 checksum offload at the moment, and that is a requirement for ipv6 TSO.

For TSO, the ipv6 header handling seems very non-trivial. What is supposed to happen in cases where certain optional extension headers should be present in some of the frames but not the others?

A specification of how your NIC support ipv6 TSO is necessary in order for support to be written, see? Seems like you are the most qualified person to write the support, therefore :-) Really, it isn't that hard and you have something on which to test whatever you write, whereas I don't.

Our ASIC supports ipv6 CSUM and TSO (and header splitting) even if extension headers are present, but I suspect the majority ipv6-capable NICs will not implement this; the stack needs to query NIC header-processing capabilities (for both CSUM and TSO) and act accordingly.

We are planning to release the ASIC programming manual to the community fairly soon, this will provide a better view on IPv6 LSO and some other features.

We will probably get to it sometime down the road; at the moment support for UDP LSO and receive side offloads are much higher on our list.

Getting a vanilla (no support for extension headers) implementation from someone who knows the stack better than we do would be a good thing :-), I suspect this should be useful for more than one NIC vendor.

Leonid

Hi Dave,

I fired off socklib (just a TCP stream test) and things look much better. I dumped the size of the data+frags for each packet that the e1000 created. It looks like frags are working in multiples of MTU, but we also have to split when we hit page boundaries. Is this expected?

BTW the 4 bytes at the end is due to an e1000 workaround for a hw bug.

Thanks for working on this :)

Anton

data: 42 frags:
data: 74 frags:
data: 66 frags: 1448 1444 4
data: 66 frags: 1024 424 1444 4
data: 66 frags: 1448 512 932 4
data: 66 frags: 1448 1444 4
data: 66 frags: 1448
data: 66 frags: 1448 1024 424 1448 1448 512 936 1444 4
data: 66 frags: 1448 1444 4
data: 66 frags: 1448 1024 420 4
data: 66 frags: 1448 1444 4
data: 66 frags: 512 936 1448 1448 1448 1448 1024 424 1448 1448 512 936 1444 4
data: 66 frags: 1448 1448 1444 4
data: 66 frags: 1024 424 1448 1448 512 936 1448 1448 1444 4
data: 66 frags: 1448 1024 424 1448 1448 372 4
data: 66 frags: 128 1320 1448 1152 296 1448 1448 640 808 1444 4
data: 66 frags: 1448 128 1320 1444 4
data: 66 frags: 1152 296 1448 1448 640 804 4
data: 66 frags: 1448 1448 128 1316 4
data: 66 frags: 1448 1152 296 1444 4
data: 66 frags: 1448 640 808 1448 1448 128 1320 1448 1152 296 1448 1448 640 808 1448 1444 4
data: 66 frags: 128 1320 1448 1152 296 1448 1444 4
data: 66 frags: 640 808 1448 1444 4
data: 66 frags: 128 1320 1448 1152 296 1448 1448 372 4
data: 66 frags: 256 1192 1448 1280 168 1448 1448 768 680 1448 1448 256 1192 1448 1280 168 1444 4
data: 66 frags: 1448 768 680 1448 1444 4
data: 66 frags: 256 1192 1448 1280 168 1448 1444 4

Hi,

With a 9000 byte MTU I see page sized frags as expected. There are still some pages broken in two which is interesting.

Anton

data: 66 frags: 3584 4096 1268 2816 4096 2032 4
data: 66 frags: 2048 4096 2804 1280 4096 3568 4
data: 66 frags: 512 4096 4096 244 3840 4096 1008 4
data: 66 frags: 3072 4096 1780 2304 4096 2544 4
data: 66 frags: 1536 4096 3316 768 4096 4080 4
data: 66 frags: 4096 4096 756 3328 4096 1520 4
data: 66 frags: 2560 4096 2292 1792 4096 3056 4
data: 66 frags: 1024 4096 3828 256 4096 4096 496 4
data: 66 frags: 3584 4096 1268 2816 4096 2032 4
data: 66 frags: 2048 4096 2804 1280 4096 3568 4
data: 66 frags: 512 4096 4096 244 3840 4096 1008 4
data: 66 frags: 3072 4096 1780 2304 4096 2544 4
data: 66 frags: 1536 4096 3316 768 4096 4080 4
data: 66 frags: 4096 4096 756 3328 4096 1520 4
data: 66 frags: 2560 4096 2292 1792 4096 3056 4
data: 66 frags: 1024 4096 3828 256 4096 4096 496 4
data: 66 frags: 3584 4096 1268 2816 4096 2032 4

I know what is causing the breaks, it's the "cache line align" crap in tcp_sendmsg().

Comment out this alignment code in net/ipv4/tcp.c and see how it goes:

					/* If page is cached, align
					 * offset to L1 cache boundary
					 */
					off = (off + L1_CACHE_BYTES - 1) &
					      ~(L1_CACHE_BYTES - 1);

This alignment causes TSO coalescing to fail at these spots.

You should not see this effect when sendfile() is used for the transfer.

Nice work! Its looking much better now.

Anton

1500 MTU:
eth0 data: 74 frags:
eth0 data: 66 frags: 2892 4
eth0 data: 66 frags: 1200 1692 4
eth0 data: 66 frags: 2400 492 4
eth0 data: 66 frags: 2892 4
eth0 data: 66 frags: 704 744
eth0 data: 66 frags: 3352 4096 1236 4
eth0 data: 66 frags: 2856 36 4

1500 MTU+sendfile:
eth0 data: 74 frags:
eth0 data: 66 frags: 2892 4
eth0 data: 66 frags: 1200 1692 4
eth0 data: 66 frags: 2400 492 4
eth0 data: 66 frags: 2892 4
eth0 data: 66 frags: 704 744
eth0 data: 66 frags: 3348 4
eth0 data: 66 frags: 4092 4

9000 MTU:
eth0 data: 74 frags:
eth0 data: 66 frags: 4096 4096 4096 4096 1508 4
eth0 data: 66 frags: 2584 4096 2268
eth0 data: 66 frags: 1828 4096 3024
eth0 data: 66 frags: 1072 4096 4096 4096 4096 4096 4096 1192 4 eth0 data: 66 frags: 2900 4096 4096 4096 2704 4
eth0 data: 66 frags: 1388 4096 3464
eth0 data: 66 frags: 632 4096 4096 4096 4096 876 4

9000 MTU+sendfile:
eth0 data: 74 frags:
eth0 data: 66 frags: 4096 4096 4092 4
eth0 data: 66 frags: 4096 4096 4096 4096 1508 4 eth0 data: 66 frags: 2584 4096 4092 4
eth0 data: 66 frags: 4096 4096 4096 4096 1508 4 eth0 data: 66 frags: 2584 4096 4096 4096 3020 4 eth0 data: 66 frags: 1072 4096 4096 4096 4096 436 4 eth0 data: 66 frags: 3656 4096 4096 4096 1948 4

Great, thanks for testing.

BTW:

What does this "66 frags" mean? Is it "tso_segs" from the SKB? skb_shinfo(skb)->nr_frags can never be that large (MAX_SKB_FRAGS is (65536/PAGE_SIZE + 2)) so it can't be that.

Nah I just dump the skb mapping then the subsequent frags. So 66 is the size of the header. In this case there is only 2 frags.

Anton

Though it may be moot by now, I just ran some tests of my own with this patch. This was on a dual 2.4 GHz Xeon booted with a UP kernel, running iperf over an e1000 at a 1500 byte MTU. I measured idle CPU by running a process which sits in a gettimeofday() loop. Numbers are CPU utilization, all turned out +/- 1%.

No TSO:  86.6%
Old TSO: 61.0%
New TSO: 88.5%

-John

From: John Heffner <[email protected]>
Date: Tue, 17 May 2005 22:26:09 -0400

Yeah, TSO Reloaded really stinks. :-)

Try the "Super TSO" patch I just posted instead.

Interesting timing. :)
With the new patch I'm getting 78.0%.

-John

From: John Heffner <[email protected]>
Date: Tue, 17 May 2005 22:51:53 -0400

Thanks for testing.

I actually expected the new code to do better. Splitting could be a little bit expensive, but that only occurs at the beginning of the connection as we ramp up the congestion and send window. Afterwards, we should be able to release full unsplit TSO frames onto the wire.

With different (larger than default) buffer sizes, I'm getting 63.4%. Not surprising I guess.

-John

From: John Heffner <[email protected]>
Date: Tue, 17 May 2005 23:47:48 -0400

Thanks for the data.

How long are your transfers? Just curious.

I think what I need to investigate is some kind of light cwnd prediction. This, plus some TSO packet coalescing logic when we undershoot, should do the trick.

But first I'll study the segmenting behavior of the current code. It could be simply a matter of tweaking when we wake up the user when he sleeps on the send buffer filling up.

I was actually measuring 10 second intervals in the middle of a long running flow.

-John