Skip to contentSkip to conversation

mptcp: adjust tcp rcvbuf size

PATCH Superseded Florian Westphal 2 participants 4 messages May 25, 2020

Subsystems (2)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Florian Westphal

author
#0 lore raw

v2 cover mptcp: adjust tcp rcvbuf size

This is a v2 of the patch I sent to netdev previously. No change. The second patch is new: It makes sure we track/use the largest rcvbuf size of a subflow.

Futhermore, msk rcvbuf tracking is only done from the recvmsg path. i.e., if userspace never calls recvmsg, the msk socket rcvbuffer won't be adjusted.

This still provides the same benefits as 1/2 while preventing unbounded growth to tcp_rmem[2].

author
#1 lore raw

PATCH v2 1/2 mptcp: adjust tcp rcvspace after moving skbs from ssk to sk queue

TCP does tcp rcvbuf tuning when copying packets to userspace, e.g. in tcp_read_sock(). In case of mptcp, that function is only rarely used (when discarding retransmitted duplicate data).

Instead, skbs are moved from the tcp rx queue to the mptcp socket rx queue.
Adjust subflow rcvbuf when we do so, its the last spot where we can adjust the ssk rcvbuf -- later we only have the mptcp-level socket.

This greatly improves performance on mptcp bulk transfers.

Signed-off-by Florian Westphal <fw(a)strlen.de>

---
v2: no changes.

 net/mptcp/protocol.c | 4 ++++
 1 file changed, 4 insertions(+)
net/mptcp/protocol.c +4 -0
index ba9d3d5c625f..dbb86cbb9e77 100644
@@ -248,6 +248,9 @@ static bool __mptcp_move_skbs_from_subflow(struct mptcp_sock *msk,
 
 	*bytes = moved;
 
+	if (moved)
+		tcp_rcv_space_adjust(ssk);
+
 	return done;
 }
 
@@ -1263,6 +1266,7 @@ static int mptcp_init_sock(struct sock *sk)
 		return ret;
 
 	sk_sockets_allocated_inc(sk);
+	sk->sk_rcvbuf = sock_net(sk)->ipv4.sysctl_tcp_rmem[1];
 	sk->sk_sndbuf = sock_net(sk)->ipv4.sysctl_tcp_wmem[2];
 
 	return 0;
Signature · 1 line
2.26.2
author
#2 lore raw

PATCH v2 2/2 mptcp: move recbuf adjustment to recvmsg path

In reply to Florian Westphal

From: Paolo Abeni <pabeni(a)redhat.com>

Place receive window tuning in the recvmsg path.
This makes sure the buffer size is only changed when userspace consumes data.

Previously we would grow sk_rcvbuf towards tcp_rmem[2], now we so only if userspace reads data.

Furthermore, we simply track the largest receive buffer size of the subflow, i.e. we will also reduce the size instead of only increasing it.

Signed-off-by Paolo Abeni <pabeni(a)redhat.com>
Signed-off-by Florian Westphal <fw(a)strlen.de>

---
v2: new patch.

 net/mptcp/protocol.c | 30 ++++++++++++++++++++++--------
 1 file changed, 22 insertions(+), 8 deletions(-)
net/mptcp/protocol.c +22 -8
index dbb86cbb9e77..2a4792cfe85e 100644
@@ -190,13 +190,6 @@ static bool __mptcp_move_skbs_from_subflow(struct mptcp_sock *msk,
 		return false;
 	}
 
-	if (!(sk->sk_userlocks & SOCK_RCVBUF_LOCK)) {
-		int rcvbuf = max(ssk->sk_rcvbuf, sk->sk_rcvbuf);
-
-		if (rcvbuf > sk->sk_rcvbuf)
-			sk->sk_rcvbuf = rcvbuf;
-	}
-
 	tp = tcp_sk(ssk);
 	do {
 		u32 map_remaining, offset;
@@ -933,6 +926,22 @@ static bool __mptcp_move_skbs(struct mptcp_sock *msk)
 	return moved > 0;
 }
 
+static void mptcp_rcv_space_adjust(struct mptcp_sock *msk)
+{
+	struct sock *sk = (struct sock *)msk;
+	const struct mptcp_subflow_context *subflow;
+	const struct sock *ssk;
+	int rcvbuf = 0;
+
+	mptcp_for_each_subflow(msk, subflow) {
+		ssk = mptcp_subflow_tcp_sock(subflow);
+		rcvbuf = max(ssk->sk_rcvbuf, rcvbuf);
+	}
+
+	if (rcvbuf)
+		sk->sk_rcvbuf = rcvbuf;
+}
+
 static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 			 int nonblock, int flags, int *addr_len)
 {
@@ -962,6 +971,9 @@ static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 	target = sock_rcvlowat(sk, flags & MSG_WAITALL, len);
 	__mptcp_flush_join_list(msk);
 
+	if (!(sk->sk_userlocks & SOCK_RCVBUF_LOCK))
+		mptcp_rcv_space_adjust(msk);
+
 	while (len > (size_t)copied) {
 		int bytes_read;
 
@@ -975,8 +987,10 @@ static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 		copied += bytes_read;
 
 		if (skb_queue_empty(&sk->sk_receive_queue) &&
-		    __mptcp_move_skbs(msk))
+		    __mptcp_move_skbs(msk)) {
+			mptcp_rcv_space_adjust(msk);
 			continue;
+		}
 
 		/* only the master socket status is relevant here. The exit
 		 * conditions mirror closely tcp_recvmsg()
Signature · 1 line
2.26.2
#3 lore raw

[MPTCP] Re: [PATCH v2 mptcp-next 2/2] mptcp: move recbuf adjustment to recvmsg path

net/mptcp/protocol.c static bool __mptcp_move_skbs(struct mptcp_sock *msk) · line 934

Quoted 10 lines · Florian Westphal
net/mptcp/protocol.c +6 -0
@@ -933,6 +926,22 @@ static bool __mptcp_move_skbs(struct mptcp_sock *msk)
 	return moved > 0;
 }
 
+static void mptcp_rcv_space_adjust(struct mptcp_sock *msk)
+{
+	struct sock *sk = (struct sock *)msk;
+	const struct mptcp_subflow_context *subflow;
+	const struct sock *ssk;
+	int rcvbuf = 0;

Reverse x-mas tree ;)

net/mptcp/protocol.c static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, · line 996

Quoted 12 lines
net/mptcp/protocol.c +3 -1
@@ -975,8 +987,10 @@ static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 		copied += bytes_read;
 
 		if (skb_queue_empty(&sk->sk_receive_queue) &&
-		    __mptcp_move_skbs(msk))
+		    __mptcp_move_skbs(msk)) {
+			mptcp_rcv_space_adjust(msk);
 			continue;
+		}
 
 		/* only the master socket status is relevant here. The exit
 		 * conditions mirror closely tcp_recvmsg()

We currently don't check the SOCK_RCVBUF_LOCK here, while we do that in the other mptcp_rcv_space_adjust() call-site.

It likely should not matter as ssk userlocks mimics msk's ones, but for consistency we could always do the test before trying to update the buffer (e.g. move the userlocks check inside mptcp_rcv_space_adjust()), WDYT?

Thanks!

Paolo