Skip to contentSkip to conversation

mptcp: rework mptcp_sendmsg

PATCH In review Florian Westphal 2 participants 10 messages May 11, 2020 – May 12, 2020

Subsystems (2)
lore
Share & tools
lore

Florian Westphal

author
#0 lore raw

v2 cover mptcp: rework mptcp_sendmsg

This series reworks mptcp_sendmsg logic to avoid blocking on the subflow socket.
This is another iteration, because there were problems with the previous one.

1. Add a new patch 1/7 to stop mptcp from accepting megabytes worth of data when tcp data gets acked at subflow level quickly, but not at mptcp level.

2. drop the patch that cached an skb in sk->sk_tx_skb_cache. One problem with that patch was that it sabotages an extra check that tcp has: when it allocates and skb it will behave slightly differently when the tcp write and retrans queues are empty wrt. wmem schedule.
By preallocating in mptcp layer that code gets bypassed. Moreover, the preallocation doesn't buy anything; the allocation can schedule but it won't sleep forever.

3. for the 3rd patch I moved the initialisation to a single location, also the timer is only started before returning to userspace. As long was we keep looping, the timer doesn't have to be started here as it will be kicked by mptcp_data_acked() for the incoming acks.

4. xmit via lo and MPTCP is SLOOOOOOOOW in 'mmap' mode. However it turned out this is already the case without these patches, so I decided to send these patches out anyway.
I will send another email to followup up on this.

I've preserved Paolos ACKs on patches that had no changes vs. v1.

Florian Westphal (7):
mptcp: break and restart in case mptcp sndbuf is full
mptcp: avoid blocking in tcp_sendpages
mptcp: make sure timer is inited once data is pending for transmission
mptcp: fill skb extension cache outside of mptcp_sendmsg_frag
mptcp: fill skb page frag cache outside of mptcp_sendmsg_frag
mptcp: remove inner wait loop from mptcp_sendmsg_frag
net: allow __skb_ext_alloc to sleep

 include/linux/skbuff.h |   2 +-
 net/core/skbuff.c      |   8 ++--
 net/mptcp/protocol.c   | 116 ++++++++++++++++++++++++++++++++++++-------------
 3 files changed, 91 insertions(+), 35 deletions(-)
author
#1 lore raw

PATCH v2 1/7 mptcp: break and restart in case mptcp sndbuf is full

Its not enough to wait for TCP-level acks (free tcp send space). We also hold on to any transmitted data for mptcp-level retransmits, so we can't continue sending more data when there is no more mptcp send buffer space available.

Therefore add needed check in mptcp_sendmsg, flush tcp data and wait until mptcp snd space becomes available.

Signed-off-by Florian Westphal <fw(a)strlen.de>
---
 net/mptcp/protocol.c | 37 ++++++++++++++++++++++++++++++-------
 1 file changed, 30 insertions(+), 7 deletions(-)
net/mptcp/protocol.c +30 -7
index e1f23016ed3f..d14e2aa30a14 100644
@@ -653,6 +653,15 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,
 	return ret;
 }
 
+static void mptcp_nospace(struct mptcp_sock *msk, struct socket *sock)
+{
+	clear_bit(MPTCP_SEND_SPACE, &msk->flags);
+	smp_mb__after_atomic(); /* msk->flags is changed by write_space cb */
+
+	/* enables sk->write_space() callbacks */
+	set_bit(SOCK_NOSPACE, &sock->flags);
+}
+
 static struct sock *mptcp_subflow_get_send(struct mptcp_sock *msk)
 {
 	struct mptcp_subflow_context *subflow;
@@ -666,13 +675,8 @@ static struct sock *mptcp_subflow_get_send(struct mptcp_sock *msk)
 		if (!sk_stream_memory_free(ssk)) {
 			struct socket *sock = ssk->sk_socket;
 
-			if (sock) {
-				clear_bit(MPTCP_SEND_SPACE, &msk->flags);
-				smp_mb__after_atomic();
-
-				/* enables sk->write_space() callbacks */
-				set_bit(SOCK_NOSPACE, &sock->flags);
-			}
+			if (sock)
+				mptcp_nospace(msk, sock);
 
 			return NULL;
 		}
@@ -740,6 +744,7 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 
 	mptcp_clean_una(sk);
 
+wait_for_sndbuf:
 	__mptcp_flush_join_list(msk);
 	ssk = mptcp_subflow_get_send(msk);
 	while (!sk_stream_memory_free(sk) || !ssk) {
@@ -777,6 +782,24 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 		}
 
 		copied += ret;
+
+		/* memory is charged to mptcp level socket as well, i.e.
+		 * if msg is very large, mptcp socket may run out of buffer
+		 * space.  mptcp_clean_una() will release data that has
+		 * been acked at mptcp level in the mean time, so there is
+		 * a good chance we can continue sending data right away.
+		 */
+		if (unlikely(!sk_stream_memory_free(sk))) {
+			tcp_push(ssk, msg->msg_flags, mss_now,
+				 tcp_sk(ssk)->nonagle, size_goal);
+			mptcp_clean_una(sk);
+			if (!sk_stream_memory_free(sk)) {
+				/* no luck, can't send more for now */
+				mptcp_nospace(msk, sk->sk_socket);
+				release_sock(ssk);
+				goto wait_for_sndbuf;
+			}
+		}
 	}
 
 	mptcp_set_timeout(sk, ssk);
Signature · 1 line
2.26.2
author
#4 lore raw

PATCH v2 2/7 mptcp: avoid blocking in tcp_sendpages

In reply to Florian Westphal

The transmit loop continues to xmit new data until an error is returned or all data was transmitted.

For the blocking i/o case, this means that tcp_sendpages() may block on the subflow until more space becomes available, i.e. we end up sleeping with the mptcp socket lock held.

Instead we should check if a different subflow is ready to be used.

This restarts the subflow sk lookup when the tx operation succeeded and the tcp subflow can't accept more data or if tcp_sendpages indicates -EAGAIN on a blocking mptcp socket.

In that case we also need to set the NOSPACE bit to make sure we get notified once memory becomes available.

In case all subflows are busy, the existing logic will wait until a subflow is ready, releasing the mptcp socket lock while doing so.

The mptcp worker already sets DONTWAIT, so no need to make changes there.

v2:
* set NOSPACE bit
* add a comment to clarify that mptcp-sk sndbuf limits need to be checked as well.

Signed-off-by Florian Westphal <fw(a)strlen.de>
---
 net/mptcp/protocol.c | 33 ++++++++++++++++++++++++++++++---
 1 file changed, 30 insertions(+), 3 deletions(-)
net/mptcp/protocol.c +30 -3
index d14e2aa30a14..13fb04d45083 100644
@@ -590,7 +590,7 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,
 	 * access the skb after the sendpages call
 	 */
 	ret = do_tcp_sendpages(ssk, page, offset, psize,
-			       msg->msg_flags | MSG_SENDPAGE_NOTLAST);
+			       msg->msg_flags | MSG_SENDPAGE_NOTLAST | MSG_DONTWAIT);
 	if (ret <= 0)
 		return ret;
 
@@ -718,6 +718,7 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 	struct socket *ssock;
 	size_t copied = 0;
 	struct sock *ssk;
+	bool tx_ok;
 	long timeo;
 
 	if (msg->msg_flags & ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL))
@@ -742,6 +743,7 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 		return ret >= 0 ? ret + copied : (copied ? copied : ret);
 	}
 
+restart:
 	mptcp_clean_una(sk);
 
 wait_for_sndbuf:
@@ -764,11 +766,17 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 	pr_debug("conn_list->subflow=%p", ssk);
 
 	lock_sock(ssk);
-	while (msg_data_left(msg)) {
+	tx_ok = msg_data_left(msg);
+	while (tx_ok) {
 		ret = mptcp_sendmsg_frag(sk, ssk, msg, NULL, &timeo, &mss_now,
 					 &size_goal);
-		if (ret < 0)
+		if (ret < 0) {
+			if (ret == -EAGAIN && timeo > 0) {
+				release_sock(ssk);
+				goto restart;
+			}
 			break;
+		}
 		if (ret == 0 && unlikely(__mptcp_needs_tcp_fallback(msk))) {
 			/* Can happen for passive sockets:
 			 * 3WHS negotiated MPTCP, but first packet after is
@@ -783,11 +791,30 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 
 		copied += ret;
 
+		tx_ok = msg_data_left(msg);
+		if (!tx_ok)
+			break;
+
+		if (!sk_stream_memory_free(ssk)) {
+			set_bit(SOCK_NOSPACE, &sk->sk_socket->flags);
+			tcp_push(ssk, msg->msg_flags, mss_now,
+				 tcp_sk(ssk)->nonagle, size_goal);
+			release_sock(ssk);
+			goto restart;
+		}
+
 		/* memory is charged to mptcp level socket as well, i.e.
 		 * if msg is very large, mptcp socket may run out of buffer
 		 * space.  mptcp_clean_una() will release data that has
 		 * been acked at mptcp level in the mean time, so there is
 		 * a good chance we can continue sending data right away.
+		 *
+		 * Normally, when the tcp subflow can accept more data, then
+		 * so can the MPTCP socket.  However, we need to cope with
+		 * peers that might lag behind in their MPTCP-level
+		 * acknowledgements, i.e.  data might have been acked at
+		 * tcp level only.  So, we must also check the MPTCP socket
+		 * limits before we send more data.
 		 */
 		if (unlikely(!sk_stream_memory_free(sk))) {
 			tcp_push(ssk, msg->msg_flags, mss_now,
Signature · 1 line
2.26.2
author
#5 lore raw

PATCH v2 3/7 mptcp: make sure timer is inited once data is pending for transmission

In reply to Florian Westphal

The mptcp_connect self-tests with additional '-m mmap' argument triggers a (harmless) WARN_ON_ONCE splat in mptcp_reset_timer() as the msk sockets timer_ival is still 0.

This is because the timer is only set up after the transmit loop, but the splat can be triggered once the first mptcp-level ACK is processed.

Signed-off-by Florian Westphal <fw(a)strlen.de>
---
 net/mptcp/protocol.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
net/mptcp/protocol.c +3 -2
index 13fb04d45083..3e8117837b41 100644
@@ -647,6 +647,9 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,
 out:
 	if (!retransmission)
 		pfrag->offset += frag_truesize;
+
+	mptcp_set_timeout(sk, ssk);
+
 	*write_seq += ret;
 	mptcp_subflow_ctx(ssk)->rel_write_seq += ret;
 
@@ -829,7 +832,6 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 		}
 	}
 
-	mptcp_set_timeout(sk, ssk);
 	if (copied) {
 		ret = copied;
 		tcp_push(ssk, msg->msg_flags, mss_now, tcp_sk(ssk)->nonagle,
@@ -1195,7 +1197,6 @@ static void mptcp_worker(struct work_struct *work)
 	dfrag->offset = orig_offset;
 	dfrag->data_len = orig_len;
 
-	mptcp_set_timeout(sk, ssk);
 	release_sock(ssk);
 
 reset_unlock:
Signature · 1 line
2.26.2
author
#6 lore raw

PATCH v2 4/7 mptcp: fill skb extension cache outside of mptcp_sendmsg_frag

In reply to Florian Westphal

The mptcp_sendmsg_frag helper contains a loop that will wait on the subflow sk.

It seems preferrable to only wait in mptcp_sendmsg() when blocking io is requested. mptcp_sendmsg already has such a wait loop that is used when no subflow socket is available for transmission.

This is a preparation patch that makes sure we call mptcp_sendmsg_frag only if a skb extension has been allocated.

Moreover, such allocation currently uses GFP_ATOMIC while it could use sleeping allocation instead.

Followup patches will remove the wait loop from mptcp_sendmsg_frag() and will allow to do a sleeping allocation for the extension.

Acked-by Paolo Abeni <pabeni(a)redhat.com>
Signed-off-by Florian Westphal <fw(a)strlen.de>
---
 net/mptcp/protocol.c | 18 ++++++++++++++----
 1 file changed, 14 insertions(+), 4 deletions(-)
net/mptcp/protocol.c +14 -4
index 3e8117837b41..778b2627c76a 100644
@@ -672,6 +672,9 @@ static struct sock *mptcp_subflow_get_send(struct mptcp_sock *msk)
 
 	sock_owned_by_me((const struct sock *)msk);
 
+	if (!mptcp_ext_cache_refill(msk))
+		return NULL;
+
 	mptcp_for_each_subflow(msk, subflow) {
 		struct sock *ssk = mptcp_subflow_tcp_sock(subflow);
 
@@ -798,7 +801,8 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 		if (!tx_ok)
 			break;
 
-		if (!sk_stream_memory_free(ssk)) {
+		if (!sk_stream_memory_free(ssk) ||
+		    !mptcp_ext_cache_refill(msk)) {
 			set_bit(SOCK_NOSPACE, &sk->sk_socket->flags);
 			tcp_push(ssk, msg->msg_flags, mss_now,
 				 tcp_sk(ssk)->nonagle, size_goal);
@@ -1146,7 +1150,7 @@ static void mptcp_worker(struct work_struct *work)
 {
 	struct mptcp_sock *msk = container_of(work, struct mptcp_sock, work);
 	struct sock *ssk, *sk = &msk->sk.icsk_inet.sk;
-	int orig_len, orig_offset, ret, mss_now = 0, size_goal = 0;
+	int orig_len, orig_offset, mss_now = 0, size_goal = 0;
 	struct mptcp_data_frag *dfrag;
 	u64 orig_write_seq;
 	size_t copied = 0;
@@ -1168,6 +1172,9 @@ static void mptcp_worker(struct work_struct *work)
 	if (!dfrag)
 		goto unlock;
 
+	if (!mptcp_ext_cache_refill(msk))
+		goto reset_unlock;
+
 	ssk = mptcp_subflow_get_retrans(msk);
 	if (!ssk)
 		goto reset_unlock;
@@ -1179,8 +1186,8 @@ static void mptcp_worker(struct work_struct *work)
 	orig_offset = dfrag->offset;
 	orig_write_seq = dfrag->data_seq;
 	while (dfrag->data_len > 0) {
-		ret = mptcp_sendmsg_frag(sk, ssk, &msg, dfrag, &timeo, &mss_now,
-					 &size_goal);
+		int ret = mptcp_sendmsg_frag(sk, ssk, &msg, dfrag, &timeo,
+					     &mss_now, &size_goal);
 		if (ret < 0)
 			break;
 
@@ -1188,6 +1195,9 @@ static void mptcp_worker(struct work_struct *work)
 		copied += ret;
 		dfrag->data_len -= ret;
 		dfrag->offset += ret;
+
+		if (!mptcp_ext_cache_refill(msk))
+			break;
 	}
 	if (copied)
 		tcp_push(ssk, msg.msg_flags, mss_now, tcp_sk(ssk)->nonagle,
Signature · 1 line
2.26.2
author
#7 lore raw

PATCH v2 5/7 mptcp: fill skb page frag cache outside of mptcp_sendmsg_frag

In reply to Florian Westphal

The mptcp_sendmsg_frag helper contains a loop that will wait on the subflow sk.

It seems preferrable to only wait in mptcp_sendmsg() when blocking io is requested. mptcp_sendmsg already has such a wait loop that is used when no subflow socket is available for transmission.

This is another preparation patch that makes sure we call mptcp_sendmsg_frag only if the page frag cache has been refilled.

Followup patch will remove the wait loop from mptcp_sendmsg_frag().

The retransmit worker doesn't need to do this refill as it won't transmit new mptcp-level data.

Acked-by Paolo Abeni <pabeni(a)redhat.com>
Signed-off-by Florian Westphal <fw(a)strlen.de>
---
 net/mptcp/protocol.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
net/mptcp/protocol.c +6 -1
index 778b2627c76a..7e91abcfb9db 100644
@@ -721,6 +721,7 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 {
 	int mss_now = 0, size_goal = 0, ret = 0;
 	struct mptcp_sock *msk = mptcp_sk(sk);
+	struct page_frag *pfrag;
 	struct socket *ssock;
 	size_t copied = 0;
 	struct sock *ssk;
@@ -749,13 +750,16 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 		return ret >= 0 ? ret + copied : (copied ? copied : ret);
 	}
 
+	pfrag = sk_page_frag(sk);
 restart:
 	mptcp_clean_una(sk);
 
 wait_for_sndbuf:
 	__mptcp_flush_join_list(msk);
 	ssk = mptcp_subflow_get_send(msk);
-	while (!sk_stream_memory_free(sk) || !ssk) {
+	while (!sk_stream_memory_free(sk) ||
+	       !ssk ||
+	       !mptcp_page_frag_refill(ssk, pfrag)) {
 		ret = sk_stream_wait_memory(sk, &timeo);
 		if (ret)
 			goto out;
@@ -802,6 +806,7 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 			break;
 
 		if (!sk_stream_memory_free(ssk) ||
+		    !mptcp_page_frag_refill(ssk, pfrag) ||
 		    !mptcp_ext_cache_refill(msk)) {
 			set_bit(SOCK_NOSPACE, &sk->sk_socket->flags);
 			tcp_push(ssk, msg->msg_flags, mss_now,
Signature · 1 line
2.26.2
author
#8 lore raw

PATCH v2 6/7 mptcp: remove inner wait loop from mptcp_sendmsg_frag

In reply to Florian Westphal

previous patches made sure we only call into this function when these prerequisites are met, so no need to wait on the subflow socket anymore.

Acked-by Paolo Abeni <pabeni(a)redhat.com>
Signed-off-by Florian Westphal <fw(a)strlen.de>
---
 net/mptcp/protocol.c | 14 --------------
 1 file changed, 14 deletions(-)
net/mptcp/protocol.c +0 -14
index 7e91abcfb9db..da5f2a1c465d 100644
@@ -510,20 +510,6 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,
 	 * fooled into a warning if we don't init here
 	 */
 	pfrag = sk_page_frag(sk);
-	while ((!retransmission && !mptcp_page_frag_refill(ssk, pfrag)) ||
-	       !mptcp_ext_cache_refill(msk)) {
-		ret = sk_stream_wait_memory(ssk, timeo);
-		if (ret)
-			return ret;
-
-		/* if sk_stream_wait_memory() sleeps snd_una can change
-		 * significantly, refresh the rtx queue
-		 */
-		mptcp_clean_una(sk);
-
-		if (unlikely(__mptcp_needs_tcp_fallback(msk)))
-			return 0;
-	}
 	if (!retransmission) {
 		write_seq = &msk->write_seq;
 		page = pfrag->page;
Signature · 1 line
2.26.2
author
#9 lore raw

PATCH v2 7/7 net: allow __skb_ext_alloc to sleep

In reply to Florian Westphal

mptcp calls this from the transmit side, from process context. Allow a sleeping allocation instead of unconditional GFP_ATOMIC.

Acked-by Paolo Abeni <pabeni(a)redhat.com>
Signed-off-by Florian Westphal <fw(a)strlen.de>
---
 include/linux/skbuff.h | 2 +-
 net/core/skbuff.c      | 8 +++++---
 net/mptcp/protocol.c   | 4 +++-
 3 files changed, 9 insertions(+), 5 deletions(-)
include/linux/skbuff.h +1 -1
index 3000c526f552..531843952809 100644
@@ -4165,7 +4165,7 @@ struct skb_ext {
 	char data[] __aligned(8);
 };
 
-struct skb_ext *__skb_ext_alloc(void);
+struct skb_ext *__skb_ext_alloc(gfp_t flags);
 void *__skb_ext_set(struct sk_buff *skb, enum skb_ext_id id,
 		    struct skb_ext *ext);
 void *skb_ext_add(struct sk_buff *skb, enum skb_ext_id id);
net/core/skbuff.c +5 -3
index 1bf0c3d278e7..35a133c6d13b 100644
@@ -6087,13 +6087,15 @@ static void *skb_ext_get_ptr(struct skb_ext *ext, enum skb_ext_id id)
 /**
  * __skb_ext_alloc - allocate a new skb extensions storage
  *
+ * @flags: See kmalloc().
+ *
  * Returns the newly allocated pointer. The pointer can later attached to a
  * skb via __skb_ext_set().
  * Note: caller must handle the skb_ext as an opaque data.
  */
-struct skb_ext *__skb_ext_alloc(void)
+struct skb_ext *__skb_ext_alloc(gfp_t flags)
 {
-	struct skb_ext *new = kmem_cache_alloc(skbuff_ext_cache, GFP_ATOMIC);
+	struct skb_ext *new = kmem_cache_alloc(skbuff_ext_cache, flags);
 
 	if (new) {
 		memset(new->offset, 0, sizeof(new->offset));
@@ -6188,7 +6190,7 @@ void *skb_ext_add(struct sk_buff *skb, enum skb_ext_id id)
 	} else {
 		newoff = SKB_EXT_CHUNKSIZEOF(*new);
 
-		new = __skb_ext_alloc();
+		new = __skb_ext_alloc(GFP_ATOMIC);
 		if (!new)
 			return NULL;
 	}
net/mptcp/protocol.c +3 -1
index da5f2a1c465d..6c8ff8a504ea 100644
@@ -367,8 +367,10 @@ static void mptcp_stop_timer(struct sock *sk)
 
 static bool mptcp_ext_cache_refill(struct mptcp_sock *msk)
 {
+	const struct sock *sk = (const struct sock *)msk;
+
 	if (!msk->cached_ext)
-		msk->cached_ext = __skb_ext_alloc();
+		msk->cached_ext = __skb_ext_alloc(sk->sk_allocation);
 
 	return !!msk->cached_ext;
 }
Signature · 1 line
2.26.2
#2 lore raw

[MPTCP] Re: [PATCH net-next 1/7] mptcp: break and restart in case mptcp sndbuf is full

In reply to Florian Westphal
Quoted 19 lines · Florian Westphal

Its not enough to wait for TCP-level acks (free tcp send space). We also hold on to any transmitted data for mptcp-level retransmits, so we can't continue sending more data when there is no more mptcp send buffer space available.

Therefore add needed check in mptcp_sendmsg, flush tcp data and wait until mptcp snd space becomes available.

Signed-off-by Florian Westphal <fw(a)strlen.de>
---
 net/mptcp/protocol.c | 37 ++++++++++++++++++++++++++++++-------
 1 file changed, 30 insertions(+), 7 deletions(-)
net/mptcp/protocol.c +0 -0
index e1f23016ed3f..d14e2aa30a14 100644
@@ -653,6 +653,15 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,
 	return ret;

net/mptcp/protocol.c static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk, · line 663

Quoted 10 lines
net/mptcp/protocol.c +8 -0
 }
 
+static void mptcp_nospace(struct mptcp_sock *msk, struct socket *sock)
+{
+	clear_bit(MPTCP_SEND_SPACE, &msk->flags);
+	smp_mb__after_atomic(); /* msk->flags is changed by write_space cb */
+
+	/* enables sk->write_space() callbacks */
+	set_bit(SOCK_NOSPACE, &sock->flags);
+}

(sharing on the ML a topic already discussed privatelly) I think we can use this helper also in ssk_check_wmem().

Other than this, series LGTM!

Thanks Floerian,

/P

author
#3 lore raw

[MPTCP] Re: [PATCH net-next 1/7] mptcp: break and restart in case mptcp sndbuf is full

Quoted 22 lines · Paolo Abeni
Quoted 21 lines · Florian Westphal

Its not enough to wait for TCP-level acks (free tcp send space). We also hold on to any transmitted data for mptcp-level retransmits, so we can't continue sending more data when there is no more mptcp send buffer space available.

Therefore add needed check in mptcp_sendmsg, flush tcp data and wait until mptcp snd space becomes available.

Signed-off-by Florian Westphal <fw(a)strlen.de>
---
 net/mptcp/protocol.c | 37 ++++++++++++++++++++++++++++++-------
 1 file changed, 30 insertions(+), 7 deletions(-)
net/mptcp/protocol.c +0 -0
index e1f23016ed3f..d14e2aa30a14 100644
@@ -653,6 +653,15 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,
 	return ret;
 }
 
Quoted 11 lines

net/mptcp/protocol.c static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk, · line 663

Quoted 8 lines
net/mptcp/protocol.c +8 -0
+static void mptcp_nospace(struct mptcp_sock *msk, struct socket *sock)
+{
+	clear_bit(MPTCP_SEND_SPACE, &msk->flags);
+	smp_mb__after_atomic(); /* msk->flags is changed by write_space cb */
+
+	/* enables sk->write_space() callbacks */
+	set_bit(SOCK_NOSPACE, &sock->flags);
+}

(sharing on the ML a topic already discussed privatelly) I think we can use this helper also in ssk_check_wmem().

Thanks for pointing this out, I've split this into a new patch to reduce size of this one.