Super TSO v2

PATCH In review David S. Miller 2 participants 3 messages May 20, 2005 – May 23, 2005

Subsystems (4)

Networking Network drivers TCP Sockets

lore

Share & tools

lore

David S. Miller

author maintainer

2005-05-20 18:51 UTC #0 lore raw

PATCH 1/1 Super TSO v2

Ok, new version of the patch. Besides some cleanups, the main change is packet send deferral at ACK arrival time.

The issue is that an ACK can open up the window enough for one MSS sized segment, but not necessarily enough to send a significant portion of the TSO frame at the head of the queue.

John Heffner described an algorithm to me which attempts to defer so that we accumulate some more send window space. It's quite effective.

At first, this change had a side effect I couldn't account for, the congestion window would grow slowly and sometimes not at all with this change. It turns out that we do not grow the congestion window unless we had filled the current congestion window. These tests are in tcp_input.c:tcp_ack() where it goes:

	prior_in_flight = tcp_packets_in_flight(tp);

...

		if ((flag & FLAG_DATA_ACKED) &&
		    (tcp_vegas_enabled(tp) || prior_in_flight >= tp->snd_cwnd))
			tcp_cong_avoid(tp, ack, seq_rtt);

tcp_cong_avoid() is what advances the congestion window. This precondition is part of congestion window validation (RFC2861), which basically states that the congestion window should not be advanced if the connection is application limited (and thus we are not really testing the link fully).

If we defer, to leave more space for sending larger chunks of TSO frames, the "prior_in_flight >= tp->snd_cwnd" test will not pass when the next ACK arrives. In order to fix this, yet still preserve the intentions of the test, I added a tp->deferred which keeps track of how much send window space we left intentionally free to accumulate space.

I'm slowly gaining a lot of confidence in this patch, so more testing would greatly be appreciated. It's pretty clear that the "TSO Reloaded" scheme is not worth pursuing much further, and that this stuff below is the way to go.

Thanks.

include/linux/tcp.h renamed from include/linux/tcp.h.~1~ +4 -2

@@ -280,13 +280,15 @@ struct tcp_sock {
 	__u32	snd_wnd;	/* The window we expect to receive	*/
 	__u32	max_window;	/* Maximal window ever seen from peer	*/
 	__u32	pmtu_cookie;	/* Last pmtu seen by socket		*/
-	__u32	mss_cache;	/* Cached effective mss, not including SACKS */
-	__u16	mss_cache_std;	/* Like mss_cache, but without TSO */
+	__u16	mss_cache;	/* Cached effective mss, not including SACKS */
+	__u16	xmit_size_goal;	/* Goal for segmenting output packets	*/
+	__u32	xmit_size_cache;/* Cache for keeping xmit_size_goal uptodate */
 	__u16	ext_header_len;	/* Network protocol overhead (IP/IPv6 options) */
 	__u8	ca_state;	/* State of fast-retransmit machine 	*/
 	__u8	retransmits;	/* Number of unrecovered RTO timeouts.	*/
 
 	__u16	advmss;		/* Advertised MSS			*/
+	__u16	deferred;	/* Packets deferred for segmentation	*/
 	__u32	window_clamp;	/* Maximal window to advertise		*/
 	__u32	rcv_ssthresh;	/* Current window clamp			*/

include/net/sock.h renamed from include/net/sock.h.~1~ +5 -2

@@ -1130,13 +1130,16 @@ static inline void sk_stream_moderate_sn
 static inline struct sk_buff *sk_stream_alloc_pskb(struct sock *sk,
 						   int size, int mem, int gfp)
 {
-	struct sk_buff *skb = alloc_skb(size + sk->sk_prot->max_header, gfp);
+	struct sk_buff *skb;
+	int hdr_len;
 
+	hdr_len = SKB_DATA_ALIGN(sk->sk_prot->max_header);
+	skb = alloc_skb(size + hdr_len, gfp);
 	if (skb) {
 		skb->truesize += mem;
 		if (sk->sk_forward_alloc >= (int)skb->truesize ||
 		    sk_stream_mem_schedule(sk, skb->truesize, 0)) {
-			skb_reserve(skb, sk->sk_prot->max_header);
+			skb_reserve(skb, hdr_len);
 			return skb;
 		}
 		__kfree_skb(skb);

include/net/tcp.h renamed from include/net/tcp.h.~1~ +26 ~~-110~~

204 lines not on this page · raw message

net/ipv4/tcp.c renamed from net/ipv4/tcp.c.~1~ +17 ~~-22~~

@@ -634,7 +634,7 @@ static ssize_t do_tcp_sendpages(struct s
 			 size_t psize, int flags)
 {
 	struct tcp_sock *tp = tcp_sk(sk);
-	int mss_now;
+	int mss_now, size_goal;
 	int err;
 	ssize_t copied;
 	long timeo = sock_sndtimeo(sk, flags & MSG_DONTWAIT);

@@ -647,6 +647,7 @@ static ssize_t do_tcp_sendpages(struct s
 	clear_bit(SOCK_ASYNC_NOSPACE, &sk->sk_socket->flags);
 
 	mss_now = tcp_current_mss(sk, !(flags&MSG_OOB));
+	size_goal = tp->xmit_size_goal;
 	copied = 0;
 
 	err = -EPIPE;

@@ -660,7 +661,7 @@ static ssize_t do_tcp_sendpages(struct s
 		int offset = poffset % PAGE_SIZE;
 		int size = min_t(size_t, psize, PAGE_SIZE - offset);
 
-		if (!sk->sk_send_head || (copy = mss_now - skb->len) <= 0) {
+		if (!sk->sk_send_head || (copy = size_goal - skb->len) <= 0) {
 new_segment:
 			if (!sk_stream_memory_free(sk))
 				goto wait_for_sndbuf;

@@ -671,7 +672,7 @@ new_segment:
 				goto wait_for_memory;
 
 			skb_entail(sk, tp, skb);
-			copy = mss_now;
+			copy = size_goal;
 		}
 
 		if (copy > size)

@@ -712,7 +713,7 @@ new_segment:
 		if (!(psize -= copy))
 			goto out;
 
-		if (skb->len != mss_now || (flags & MSG_OOB))
+		if (skb->len != size_goal || (flags & MSG_OOB))
 			continue;
 
 		if (forced_push(tp)) {

@@ -732,6 +733,7 @@ wait_for_memory:
 			goto do_error;
 
 		mss_now = tcp_current_mss(sk, !(flags&MSG_OOB));
+		size_goal = tp->xmit_size_goal;
 	}
 
 out:

@@ -773,15 +775,11 @@ ssize_t tcp_sendpage(struct socket *sock
 
 static inline int select_size(struct sock *sk, struct tcp_sock *tp)
 {
-	int tmp = tp->mss_cache_std;
+	int tmp = tp->mss_cache;
 
-	if (sk->sk_route_caps & NETIF_F_SG) {
-		int pgbreak = SKB_MAX_HEAD(MAX_TCP_HEADER);
+	if (sk->sk_route_caps & NETIF_F_SG)
+		tmp = 0;
 
-		if (tmp >= pgbreak &&
-		    tmp <= pgbreak + (MAX_SKB_FRAGS - 1) * PAGE_SIZE)
-			tmp = pgbreak;
-	}
 	return tmp;
 }

@@ -792,7 +790,7 @@ int tcp_sendmsg(struct kiocb *iocb, stru
 	struct tcp_sock *tp = tcp_sk(sk);
 	struct sk_buff *skb;
 	int iovlen, flags;
-	int mss_now;
+	int mss_now, size_goal;
 	int err, copied;
 	long timeo;

@@ -811,6 +809,7 @@ int tcp_sendmsg(struct kiocb *iocb, stru
 	clear_bit(SOCK_ASYNC_NOSPACE, &sk->sk_socket->flags);
 
 	mss_now = tcp_current_mss(sk, !(flags&MSG_OOB));
+	size_goal = tp->xmit_size_goal;
 
 	/* Ok commence sending. */
 	iovlen = msg->msg_iovlen;

@@ -833,7 +832,7 @@ int tcp_sendmsg(struct kiocb *iocb, stru
 			skb = sk->sk_write_queue.prev;
 
 			if (!sk->sk_send_head ||
-			    (copy = mss_now - skb->len) <= 0) {
+			    (copy = size_goal - skb->len) <= 0) {
 
 new_segment:
 				/* Allocate new segment. If the interface is SG,

@@ -856,7 +855,7 @@ new_segment:
 					skb->ip_summed = CHECKSUM_HW;
 
 				skb_entail(sk, tp, skb);
-				copy = mss_now;
+				copy = size_goal;
 			}
 
 			/* Try to append data to the end of skb. */

@@ -891,11 +890,6 @@ new_segment:
 					tcp_mark_push(tp, skb);
 					goto new_segment;
 				} else if (page) {
-					/* If page is cached, align
-					 * offset to L1 cache boundary
-					 */
-					off = (off + L1_CACHE_BYTES - 1) &
-					      ~(L1_CACHE_BYTES - 1);
 					if (off == PAGE_SIZE) {
 						put_page(page);
 						TCP_PAGE(sk) = page = NULL;

@@ -956,7 +950,7 @@ new_segment:
 			if ((seglen -= copy) == 0 && iovlen == 0)
 				goto out;
 
-			if (skb->len != mss_now || (flags & MSG_OOB))
+			if (skb->len != size_goal || (flags & MSG_OOB))
 				continue;
 
 			if (forced_push(tp)) {

@@ -976,6 +970,7 @@ wait_for_memory:
 				goto do_error;
 
 			mss_now = tcp_current_mss(sk, !(flags&MSG_OOB));
+			size_goal = tp->xmit_size_goal;
 		}
 	}

@@ -2135,7 +2130,7 @@ void tcp_get_info(struct sock *sk, struc
 
 	info->tcpi_rto = jiffies_to_usecs(tp->rto);
 	info->tcpi_ato = jiffies_to_usecs(tp->ack.ato);
-	info->tcpi_snd_mss = tp->mss_cache_std;
+	info->tcpi_snd_mss = tp->mss_cache;
 	info->tcpi_rcv_mss = tp->ack.rcv_mss;
 
 	info->tcpi_unacked = tp->packets_out;

@@ -2185,7 +2180,7 @@ int tcp_getsockopt(struct sock *sk, int 
 
 	switch (optname) {
 	case TCP_MAXSEG:
-		val = tp->mss_cache_std;
+		val = tp->mss_cache;
 		if (!val && ((1 << sk->sk_state) & (TCPF_CLOSE | TCPF_LISTEN)))
 			val = tp->rx_opt.user_mss;
 		break;

net/ipv4/tcp_input.c renamed from net/ipv4/tcp_input.c.~1~ +22 ~~-31~~

133 lines not on this page · raw message

net/ipv4/tcp_ipv4.c renamed from net/ipv4/tcp_ipv4.c.~1~ +2 -1

@@ -2060,7 +2060,8 @@ static int tcp_v4_init_sock(struct sock 
 	 */
 	tp->snd_ssthresh = 0x7fffffff;	/* Infinity */
 	tp->snd_cwnd_clamp = ~0;
-	tp->mss_cache_std = tp->mss_cache = 536;
+	tp->mss_cache = 536;
+	tp->xmit_size_cache = ~0;
 
 	tp->reordering = sysctl_tcp_reordering;

net/ipv4/tcp_output.c renamed from net/ipv4/tcp_output.c.~1~ +455 ~~-124~~

700 lines not on this page · raw message

net/ipv6/tcp_ipv6.c renamed from net/ipv6/tcp_ipv6.c.~1~ +2 -1

@@ -2021,7 +2021,8 @@ static int tcp_v6_init_sock(struct sock 
 	 */
 	tp->snd_ssthresh = 0x7fffffff;
 	tp->snd_cwnd_clamp = ~0;
-	tp->mss_cache_std = tp->mss_cache = 536;
+	tp->mss_cache = 536;
+	tp->xmit_size_cache = ~0;
 
 	tp->reordering = sysctl_tcp_reordering;

Herbert Xu

2005-05-23 09:54 UTC #1 lore raw

Quoted 4 lines · David S. Miller

Ok, new version of the patch. Besides some cleanups, the main change is packet send deferral at ACK arrival time.

Thanks for the great work Dave.

Quoted 22 lines

net/ipv4/tcp_output.c renamed from net/ipv4/tcp_output.c.~1~ +14 -1

@@ -712,7 +713,7 @@ new_segment:
 		if (!(psize -= copy))
 			goto out;
 
-		if (skb->len != mss_now || (flags & MSG_OOB))
+		if (skb->len != size_goal || (flags & MSG_OOB))
 			continue;
 
+unsigned int tcp_current_mss(struct sock *sk, int large_allowed)
 
+
+	if (tp->xmit_size_cache != xmit_cache) {
+		u16 xmit_size_goal = mss_now;
+
+		if (doing_tso) {
+			xmit_size_goal = 65535 -
+				tp->af_specific->net_header_len -
+				tp->ext_header_len - tp->tcp_header_len;
+
+			if (tp->rx_opt.eff_sacks)
+				xmit_size_goal -= (TCPOLEN_SACK_BASE_ALIGNED +
+						   (tp->rx_opt.eff_sacks *

net/ipv4/tcp_output.c new_segment: · line 708

Quoted 10 lines

net/ipv4/tcp_output.c renamed from net/ipv4/tcp_output.c.~1~ +8 -0

+						    TCPOLEN_SACK_PERBLOCK));
+
+			xmit_size_goal -= (xmit_size_goal % mss_now);
+		}
+		tp->xmit_size_goal = xmit_size_goal;
+		tp->xmit_size_cache = xmit_cache;
+	}
+
 	return mss_now;
 }

Perhaps we should set the goal based on the cwnd as we do now?

As it is I believe we may be doing more work compared to the status quo because it's always building 64K blocks and then cutting them up as dictated by the cwnd. On the other hand, the current code simply builds the packets up to the size that cwnd allows.

Cheers,

Signature · 4 lines

Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

David S. Miller

author maintainer

2005-05-23 18:43 UTC #2 lore raw

From: Herbert Xu <[email protected]>
Date: Mon, 23 May 2005 19:54:17 +1000

Quoted 7 lines

Perhaps we should set the goal based on the cwnd as we do now?
As it is I believe we may be doing more work compared to the status quo because it's always building 64K blocks and then cutting them up as dictated by the cwnd. On the other hand, the current code simply builds the packets up to the size that cwnd allows.

I played with this, and it makes no difference.

It would only help when:

1) we are building sk->sk_send_head, and
2) sk->sk_send_head is the tail of the write queue

But the SKB splitting is so cheap that it doesn't
show up.

And if we sleep or defer for some reason, say while copying the data from userspace or whatever, all of this information changes and we don't bunch up as much as we could have.

Also, in my current code we defer sending not just at ACK arrive processing, but also when doing normal sends from sendmsg() and sendpages(). Which totally invalidates your suggested scheme :-)