Skip to contentSkip to conversation

mptcp: adjust tcp rcvspace on rx

PATCH In review Florian Westphal 3 participants 8 messages May 25, 2020 – May 27, 2020

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v2 cover mptcp: adjust tcp rcvspace on rx

These two patches improve mptcp throughput by making sure tcp grows the receive buffer when we move skbs from subflow socket to the mptcp socket.

The second patch moves mptcp receive buffer increase to the recvmsg path, i.e. we only change its size when userspace processes/consumes the data. This is done by using the largest rcvbuf size of the active subflows.

#1 lore raw

PATCH v2 1/2 mptcp: adjust tcp rcvspace after moving skbs from ssk to sk queue

TCP does tcp rcvbuf tuning when copying packets to userspace, e.g. in tcp_read_sock(). In case of mptcp, that function is only rarely used (when discarding retransmitted duplicate data).

Instead, skbs are moved from the tcp rx queue to the mptcp socket rx queue.
Adjust subflow rcvbuf when we do so, its the last spot where we can adjust the ssk rcvbuf -- later we only have the mptcp-level socket.

This greatly improves performance on mptcp bulk transfers.

---
v2: no change, added new patch, 2/2

 net/mptcp/protocol.c | 4 ++++
 1 file changed, 4 insertions(+)
net/mptcp/protocol.c +4 -0
index ba9d3d5c625f..dbb86cbb9e77 100644
@@ -248,6 +248,9 @@ static bool __mptcp_move_skbs_from_subflow(struct mptcp_sock *msk,
 
 	*bytes = moved;
 
+	if (moved)
+		tcp_rcv_space_adjust(ssk);
+
 	return done;
 }
 
@@ -1263,6 +1266,7 @@ static int mptcp_init_sock(struct sock *sk)
 		return ret;
 
 	sk_sockets_allocated_inc(sk);
+	sk->sk_rcvbuf = sock_net(sk)->ipv4.sysctl_tcp_rmem[1];
 	sk->sk_sndbuf = sock_net(sk)->ipv4.sysctl_tcp_wmem[2];
 
 	return 0;
Signature · 1 line
2.26.2
#2 lore raw

PATCH v2 2/2 mptcp: move recbuf adjustment to recvmsg path

In reply to Florian Westphal

From: Paolo Abeni <[email protected]>

Place receive window tuning in the recvmsg path.
This makes sure the size is only increased when userspace consumes data.

Previously we would grow the sk receive buffer towards tcp_rmem[2], now we so only if userspace reads data.

Simply adjust the msk rcvbuf size to the largest receive buffer of any of the existing subflows.

---
This patch is new in v2.

 net/mptcp/protocol.c | 32 ++++++++++++++++++++++++--------
 1 file changed, 24 insertions(+), 8 deletions(-)
net/mptcp/protocol.c +24 -8
index dbb86cbb9e77..89a35c3fc499 100644
@@ -190,13 +190,6 @@ static bool __mptcp_move_skbs_from_subflow(struct mptcp_sock *msk,
 		return false;
 	}
 
-	if (!(sk->sk_userlocks & SOCK_RCVBUF_LOCK)) {
-		int rcvbuf = max(ssk->sk_rcvbuf, sk->sk_rcvbuf);
-
-		if (rcvbuf > sk->sk_rcvbuf)
-			sk->sk_rcvbuf = rcvbuf;
-	}
-
 	tp = tcp_sk(ssk);
 	do {
 		u32 map_remaining, offset;
@@ -933,6 +926,25 @@ static bool __mptcp_move_skbs(struct mptcp_sock *msk)
 	return moved > 0;
 }
 
+static void mptcp_rcv_space_adjust(struct mptcp_sock *msk)
+{
+	const struct mptcp_subflow_context *subflow;
+	struct sock *sk = (struct sock *)msk;
+	const struct sock *ssk;
+	int rcvbuf = 0;
+
+	if (sk->sk_userlocks & SOCK_RCVBUF_LOCK)
+		return;
+
+	mptcp_for_each_subflow(msk, subflow) {
+		ssk = mptcp_subflow_tcp_sock(subflow);
+		rcvbuf = max(ssk->sk_rcvbuf, rcvbuf);
+	}
+
+	if (rcvbuf)
+		sk->sk_rcvbuf = rcvbuf;
+}
+
 static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 			 int nonblock, int flags, int *addr_len)
 {
@@ -962,6 +974,8 @@ static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 	target = sock_rcvlowat(sk, flags & MSG_WAITALL, len);
 	__mptcp_flush_join_list(msk);
 
+	mptcp_rcv_space_adjust(msk);
+
 	while (len > (size_t)copied) {
 		int bytes_read;
 
@@ -975,8 +989,10 @@ static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 		copied += bytes_read;
 
 		if (skb_queue_empty(&sk->sk_receive_queue) &&
-		    __mptcp_move_skbs(msk))
+		    __mptcp_move_skbs(msk)) {
+			mptcp_rcv_space_adjust(msk);
 			continue;
+		}
 
 		/* only the master socket status is relevant here. The exit
 		 * conditions mirror closely tcp_recvmsg()
Signature · 1 line
2.26.2
#3 lore raw

Re: [PATCH v2 net-next 2/2] mptcp: move recbuf adjustment to recvmsg path

Hello,

Quoted 45 lines · Florian Westphal

From: Paolo Abeni <[email protected]>

Place receive window tuning in the recvmsg path.
This makes sure the size is only increased when userspace consumes data.

Previously we would grow the sk receive buffer towards tcp_rmem[2], now we so only if userspace reads data.

Simply adjust the msk rcvbuf size to the largest receive buffer of any of the existing subflows.

---
This patch is new in v2.

 net/mptcp/protocol.c | 32 ++++++++++++++++++++++++--------
 1 file changed, 24 insertions(+), 8 deletions(-)
net/mptcp/protocol.c +3 -7
index dbb86cbb9e77..89a35c3fc499 100644
@@ -190,13 +190,6 @@ static bool __mptcp_move_skbs_from_subflow(struct mptcp_sock *msk,
                return false;
        }
 
-       if (!(sk->sk_userlocks & SOCK_RCVBUF_LOCK)) {
-               int rcvbuf = max(ssk->sk_rcvbuf, sk->sk_rcvbuf);
-
-               if (rcvbuf > sk->sk_rcvbuf)
-                       sk->sk_rcvbuf = rcvbuf;
-       }
-
        tp = tcp_sk(ssk);
        do {
                u32 map_remaining, offset;
@@ -933,6 +926,25 @@ static bool __mptcp_move_skbs(struct mptcp_sock *msk)
        return moved > 0;
 }
 
+static void mptcp_rcv_space_adjust(struct mptcp_sock *msk)
+{
+       const struct mptcp_subflow_context *subflow;

net/mptcp/protocol.c static bool __mptcp_move_skbs(struct mptcp_sock *msk) · line 941

Quoted 10 lines
net/mptcp/protocol.c +10 -0
+       struct sock *sk = (struct sock *)msk;
+       const struct sock *ssk;
+       int rcvbuf = 0;
+
+       if (sk->sk_userlocks & SOCK_RCVBUF_LOCK)
+               return;
+
+       mptcp_for_each_subflow(msk, subflow) {
+               ssk = mptcp_subflow_tcp_sock(subflow);
+               rcvbuf = max(ssk->sk_rcvbuf, rcvbuf);

tcp_rcv_space_adjust is called even when the app is not yet reading, thus wouldn't this mean that we still end up with an ever-growing window?
E.g., imagine an app that does not read at all at the beginning. The call to tcp_rcv_space_adjust in patch 1/2 will make the subflow's window grow. Now, the app comes and reads one byte. Then, the window at MPTCP-level will jump regardless of how much the app actually read.

I think what is needed is to size the MPTCP-level window to 2 x the amount of data read by the application within an RTT (maximum RTT among all the active subflows). That way if an app reads 1 byte a second, the window will remain low. While for a bulk-transfer it will allow all subflows to receive at full speed [1].

Or do you think that kind of tuning can be done in a follow-up patch?

Christoph

[1]: https://www.usenix.org/system/files/conference/nsdi12/nsdi12-final125.pdf
-> Section 4.2

#4 lore raw

Re: [PATCH v2 net-next 2/2] mptcp: move recbuf adjustment to recvmsg path

Quoted 3 lines · Christoph Paasch

tcp_rcv_space_adjust is called even when the app is not yet reading, thus wouldn't this mean that we still end up with an ever-growing window?

Window is based on available mptcp sk recvbuf. When data is moved from ssk to the mptcp sk, the skb truesize is charged to the mptcp rmem.

Quoted 4 lines

E.g., imagine an app that does not read at all at the beginning. The call to tcp_rcv_space_adjust in patch 1/2 will make the subflow's window grow. Now, the app comes and reads one byte. Then, the window at MPTCP-level will jump regardless of how much the app actually read.

Yes, the rcvbufsz value will jump, regardless homw much the app actually read.

Quoted 5 lines

I think what is needed is to size the MPTCP-level window to 2 x the amount of data read by the application within an RTT (maximum RTT among all the active subflows). That way if an app reads 1 byte a second, the window will remain low. While for a bulk-transfer it will allow all subflows to receive at full speed [1].

Sounds like the idea to move skbs to msk was bad one? Sorry, I don't see how I can make this work.

Even deferring tcp_rcv_space_adjust() until recv() time won't work, given data has been pulled to the mptcp socket already.

NOT calling tcp_rcv_space_adjust() at all might work, but that would require something else entirely. We would still have to adjust the subflow rcvbufsz in this case, else we may announce a window that is larger than the memory limit of the ssk (and we will end up dropping data at tcp level if the worker can't move the skbs fast enough).

Quoted 1 line

Or do you think that kind of tuning can be done in a follow-up patch?

This sounds completely different so I don't think that makes sense.

maintainer
#5 lore raw
In reply to Florian Westphal

From: Florian Westphal <[email protected]>
Date: Mon, 25 May 2020 20:15:06 +0200

Quoted 8 lines

These two patches improve mptcp throughput by making sure tcp grows the receive buffer when we move skbs from subflow socket to the mptcp socket.

The second patch moves mptcp receive buffer increase to the recvmsg path, i.e. we only change its size when userspace processes/consumes the data. This is done by using the largest rcvbuf size of the active subflows.

What's the follow-up wrt. Christoph's feedback on patch #2?

#6 lore raw
Quoted 7 lines · David Miller

From: Florian Westphal <[email protected]>
Date: Mon, 25 May 2020 20:15:06 +0200

Quoted 4 lines

These two patches improve mptcp throughput by making sure tcp grows the receive buffer when we move skbs from subflow socket to the mptcp socket.

Quoted 6 lines
Quoted 4 lines

The second patch moves mptcp receive buffer increase to the recvmsg path, i.e. we only change its size when userspace processes/consumes the data. This is done by using the largest rcvbuf size of the active subflows.

What's the follow-up wrt. Christoph's feedback on patch #2?

Please drop these patches, I have no idea (yet?) how to address it.

maintainer
#7 lore raw

From: Florian Westphal <[email protected]>
Date: Wed, 27 May 2020 13:55:45 +0200

Quoted 8 lines
Quoted 7 lines · David Miller

From: Florian Westphal <[email protected]>
Date: Mon, 25 May 2020 20:15:06 +0200

Quoted 4 lines

These two patches improve mptcp throughput by making sure tcp grows the receive buffer when we move skbs from subflow socket to the mptcp socket.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

The second patch moves mptcp receive buffer increase to the recvmsg path, i.e. we only change its size when userspace processes/consumes the data. This is done by using the largest rcvbuf size of the active subflows.

What's the follow-up wrt. Christoph's feedback on patch #2?

Please drop these patches, I have no idea (yet?) how to address it.

Ok, thank you.