Skip to contentSkip to conversation

mptcp: add and use wmem accounting for rtx queue

RFC PATCH Superseded Florian Westphal 3 participants 15 messages Nov 8, 2019 – Nov 12, 2019

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Florian Westphal

author
#0 lore raw

cover mptcp: add and use wmem accounting for rtx queue

This series adds wmem accounting and makes mptcp sockets block once the sndbuf size is reached. rtx cleanup is changed to wakeup any blocked mptcp socket once enough wmem is available again.

sndbuf is set to the maximum allowed size of the tcp wmem. There is no auto-tuning, I think we can add that later on as an enhancement.

This is an RFC series in order to prevent the patches from getting applied, I will send a v2 as soon as the open questions are sorted out. Everthing continues to work for me.

I've tested these patches with an 4k-sized sndbuf, which results in frequent wmem blocks and mptcp connection recovers once the acks come in.

1. (see patch 5): add a new work queue or re-use existing one?
2. Should all patches get squashed or should there be a "add wmem accouting" patch? A mix of both?

2, 3 and 4 would be squash candidates for example even if we add a extra "wmem patch".

I will send the next iteration of the scheduler patches after the above is sorted out.
The scheduler patches have no major changes aside from addressing a comment from Paolo.

Meanwhile I am working on removal of __tcp_poll and on make the mptcp socket conn list store tcp_sk structs rather than require a 'struct socket'.

Florian Westphal (6):
mptcp: add wmem_queued accounting
mptcp: allow partial cleaning of rtx head dfrag
subflow: store known available bytes in subflow
options: ack pending sequence
mptcp: add ack work queue
sendmsg: block until mptcp sk is writeable

 net/mptcp/options.c  | 13 ++++++--
 net/mptcp/protocol.c | 86 +++++++++++++++++++++++++++++++++++++++++++++++-----
 net/mptcp/protocol.h |  5 +--
 net/mptcp/subflow.c  | 18 ++++++++---
 4 files changed, 105 insertions(+), 17 deletions(-)
author
#1 lore raw

RFC PATCH 1/6 mptcp: add wmem_queued accounting

Peer could ack data at TCP level but refrain from sending mptcp-level ACKs. This could result in a growing the mptcp socket backlog indefinitely.

We should thus block mptcp_sendmsg until the peer has acked some of the sent data.

In order to be able to do so, increment the mptcp socket wmem_queued counter on memory allocation and decrement it when releasing the memory on mptcp-level ack reception.

Because TCP performns sndbuf auto-tuning up to tcp_wmem_max[2], make this the mptcp sk_sndbuf limit.

In the future we could add experiment with autotuning as TCP does in tcp_sndbuf_expand().

Signed-off-by Florian Westphal <fw(a)strlen.de>
---
 net/mptcp/protocol.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
net/mptcp/protocol.c +8 -1
index 7bea6b62f66e..65f10bb372aa 100644
@@ -139,8 +139,11 @@ static inline bool mptcp_frag_can_collapse_to(const struct mptcp_sock *msk,
 
 static void dfrag_clear(struct sock *sk, struct mptcp_data_frag *dfrag)
 {
+	int len = dfrag->data_len + dfrag->overhead;
+
 	list_del(&dfrag->list);
-	sk_mem_uncharge(sk, dfrag->data_len + dfrag->overhead);
+	sk_mem_uncharge(sk, len);
+	sk_wmem_queued_add(sk, -len);
 	put_page(dfrag->page);
 }
 
@@ -304,6 +307,9 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,
 		if (!dfrag_collapsed) {
 			get_page(dfrag->page);
 			list_add_tail(&dfrag->list, &msk->rtx_queue);
+			sk_wmem_queued_add(sk, frag_truesize);
+		} else {
+			sk_wmem_queued_add(sk, ret);
 		}
 
 		/* charge data on mptcp rtx queue to the master socket
@@ -711,6 +717,7 @@ static int mptcp_init_sock(struct sock *sk)
 		return ret;
 
 	sk_sockets_allocated_inc(sk);
+	sk->sk_sndbuf = sock_net(sk)->ipv4.sysctl_tcp_wmem[2];
 
 	return 0;
 }
Signature · 1 line
2.23.0
author
#2 lore raw

RFC PATCH 2/6 mptcp: allow partial cleaning of rtx head dfrag

In reply to Florian Westphal

After adding wmem accouting for the mptcp socket we could get into a situation where the mptcp socket can't transmit more data, and mptcp_clean_una doesn't reduce wmem even if snd_una has advanced because it currently will only remove entire dfrags.

Allow advancing the dfrag head sequence and reduce wmem, even though this isn't correct (as we can't release the page).

Because we will soon block on mptcp sk in case wmem is too large, call sk_stream_write_space() in case we reduced the backlog so userspace task blocked in sendmsg or poll will be woken up.

Signed-off-by Florian Westphal <fw(a)strlen.de>
---
 net/mptcp/protocol.c | 28 +++++++++++++++++++++++++---
 1 file changed, 25 insertions(+), 3 deletions(-)
net/mptcp/protocol.c +25 -3
index 65f10bb372aa..b93050056195 100644
@@ -137,13 +137,18 @@ static inline bool mptcp_frag_can_collapse_to(const struct mptcp_sock *msk,
 		df->data_seq + df->data_len == msk->write_seq;
 }
 
+static void dfrag_uncharge(struct sock *sk, int len)
+{
+	sk_mem_uncharge(sk, len);
+	sk_wmem_queued_add(sk, -len);
+}
+
 static void dfrag_clear(struct sock *sk, struct mptcp_data_frag *dfrag)
 {
 	int len = dfrag->data_len + dfrag->overhead;
 
 	list_del(&dfrag->list);
-	sk_mem_uncharge(sk, len);
-	sk_wmem_queued_add(sk, -len);
+	dfrag_uncharge(sk, len);
 	put_page(dfrag->page);
 }
 
@@ -152,14 +157,31 @@ static void mptcp_clean_una(struct sock *sk)
 	struct mptcp_sock *msk = mptcp_sk(sk);
 	struct mptcp_data_frag *dtmp, *dfrag;
 	u64 snd_una = atomic64_read(&msk->snd_una);
+	bool cleaned = false;
 
 	list_for_each_entry_safe(dfrag, dtmp, &msk->rtx_queue, list) {
 		if (after64(dfrag->data_seq + dfrag->data_len, snd_una))
 			break;
 
 		dfrag_clear(sk, dfrag);
+		cleaned = true;
+	}
+
+	dfrag = mptcp_rtx_head(sk);
+	if (dfrag && after64(snd_una, dfrag->data_seq)) {
+		u64 delta = dfrag->data_seq + dfrag->data_len - snd_una;
+
+		dfrag->data_seq += delta;
+		dfrag->data_len -= delta;
+
+		dfrag_uncharge(sk, delta);
+		cleaned = true;
+	}
+
+	if (cleaned) {
+		sk_mem_reclaim_partial(sk);
+		sk_stream_write_space(sk);
 	}
-	sk_mem_reclaim_partial(sk);
 }
 
 /* ensure we get enough memory for the frag hdr, beyond some minimal amount of
Signature · 1 line
2.23.0
#3 lore raw

[MPTCP] Re: [RFC PATCH 2/6] mptcp: allow partial cleaning of rtx head dfrag

Quoted 7 lines · Florian Westphal

After adding wmem accouting for the mptcp socket we could get into a situation where the mptcp socket can't transmit more data, and mptcp_clean_una doesn't reduce wmem even if snd_una has advanced because it currently will only remove entire dfrags.

Allow advancing the dfrag head sequence and reduce wmem, even though this isn't correct (as we can't release the page).

The approach seems ok to me, since it only ever has excess bytes from one page per MPTCP socket.

No changes to suggest in the code either.

Mat

Quoted 10 lines

Because we will soon block on mptcp sk in case wmem is too large, call sk_stream_write_space() in case we reduced the backlog so userspace task blocked in sendmsg or poll will be woken up.

Signed-off-by Florian Westphal <fw(a)strlen.de>

---
net/mptcp/protocol.c | 28 +++++++++++++++++++++++++---

1 file changed, 25 insertions(+), 3 deletions(-)

--
Mat Martineau
Intel

author
#4 lore raw

RFC PATCH 3/6 subflow: store known available bytes in subflow

In reply to Florian Westphal

Instead of just a flag, store the number of bytes that are known to be available for reading.

This will allow a followup patch to include this number in the dss ack. At this time, we can only ack up to msk->ack_seq, but that number is only updated when userspace reads from the socket.

This could lead to a needless retransmission if userspace doesn't read fast enough.

Ideally we could update this number to include all the pending data, i.e. tp->rcv_nxt - tp->copied_seq, but there is no guarantee that the next skb in sk_receive_queue contains the next packet in logical mptcp sequence space.

Signed-off-by Florian Westphal <fw(a)strlen.de>
---
 net/mptcp/protocol.h |  2 +-
 net/mptcp/subflow.c  | 18 ++++++++++++++----
 2 files changed, 15 insertions(+), 5 deletions(-)
net/mptcp/protocol.h +1 -1
index 83b06382e56a..bc7e8fa54afa 100644
@@ -215,8 +215,8 @@ struct mptcp_subflow_context {
 		conn_finished : 1,
 		map_valid : 1,
 		backup : 1,
-		data_avail : 1,
 		rx_eof : 1;
+	u32	data_avail;
 	u32	remote_nonce;
 	u64	thmac;
 	u32	local_nonce;
net/mptcp/subflow.c +14 -4
index 76156a7aaea8..0d686f91e0b4 100644
@@ -612,7 +612,8 @@ static bool subflow_check_data_avail(struct sock *ssk)
 bool mptcp_subflow_data_available(struct sock *sk)
 {
 	struct mptcp_subflow_context *subflow = mptcp_subflow_ctx(sk);
-	struct sk_buff *skb;
+	const struct sk_buff *skb;
+	u32 data_avail = 0;
 
 	/* check if current mapping is still valid */
 	if (subflow->map_valid &&
@@ -636,9 +637,18 @@ bool mptcp_subflow_data_available(struct sock *sk)
 	}
 
 	skb = skb_peek(&sk->sk_receive_queue);
-	subflow->data_avail = skb &&
-		       before(tcp_sk(sk)->copied_seq, TCP_SKB_CB(skb)->end_seq);
-	return subflow->data_avail;
+	if (skb) {
+		const struct tcp_sock *tp = tcp_sk(sk);
+		u32 map_remaining = subflow->map_data_len -
+				    mptcp_subflow_get_map_offset(subflow);
+
+		data_avail = TCP_SKB_CB(skb)->end_seq - tp->copied_seq;
+		data_avail = min_t(u32, data_avail, map_remaining);
+	}
+
+	subflow->data_avail = data_avail;
+
+	return subflow->data_avail > 0;
 }
 
 static void subflow_data_ready(struct sock *sk)
Signature · 1 line
2.23.0
#5 lore raw

[MPTCP] Re: [RFC PATCH 3/6] subflow: store known available bytes in subflow

Quoted 51 lines · Florian Westphal

Instead of just a flag, store the number of bytes that are known to be available for reading.

This will allow a followup patch to include this number in the dss ack. At this time, we can only ack up to msk->ack_seq, but that number is only updated when userspace reads from the socket.

This could lead to a needless retransmission if userspace doesn't read fast enough.

Ideally we could update this number to include all the pending data, i.e. tp->rcv_nxt - tp->copied_seq, but there is no guarantee that the next skb in sk_receive_queue contains the next packet in logical mptcp sequence space.

Signed-off-by Florian Westphal <fw(a)strlen.de>

---
net/mptcp/protocol.h | 2 +-
net/mptcp/subflow.c | 18 ++++++++++++++----

2 files changed, 15 insertions(+), 5 deletions(-)
net/mptcp/protocol.h +1 -1
index 83b06382e56a..bc7e8fa54afa 100644
@@ -215,8 +215,8 @@ struct mptcp_subflow_context {
 		conn_finished : 1,
 		map_valid : 1,
 		backup : 1,
-		data_avail : 1,
 		rx_eof : 1;
+	u32	data_avail;
 	u32	remote_nonce;
 	u64	thmac;
 	u32	local_nonce;
net/mptcp/subflow.c +2 -1
index 76156a7aaea8..0d686f91e0b4 100644
@@ -612,7 +612,8 @@ static bool subflow_check_data_avail(struct sock *ssk)
 bool mptcp_subflow_data_available(struct sock *sk)
 {
 	struct mptcp_subflow_context *subflow = mptcp_subflow_ctx(sk);
-	struct sk_buff *skb;
+	const struct sk_buff *skb;
+	u32 data_avail = 0;
 
 	/* check if current mapping is still valid */
 	if (subflow->map_valid &&
@@ -636,9 +637,18 @@ bool mptcp_subflow_data_available(struct sock *sk)
 	}

net/mptcp/subflow.c bool mptcp_subflow_data_available(struct sock *sk) · line 644

Quoted 10 lines
net/mptcp/subflow.c +5 -3
 
 	skb = skb_peek(&sk->sk_receive_queue);
-	subflow->data_avail = skb &&
-		       before(tcp_sk(sk)->copied_seq, TCP_SKB_CB(skb)->end_seq);
-	return subflow->data_avail;
+	if (skb) {
+		const struct tcp_sock *tp = tcp_sk(sk);
+		u32 map_remaining = subflow->map_data_len -
+				    mptcp_subflow_get_map_offset(subflow);
+

It looks like the mapping can be assumed valid here, since subflow_check_data_avail() returned true. It's not obvious looking at this function that map_valid should be true at this point - is it worth a WARN_ONCE or a comment?

Quoted 16 lines
net/mptcp/subflow.c +7 -0
+		data_avail = TCP_SKB_CB(skb)->end_seq - tp->copied_seq;
+		data_avail = min_t(u32, data_avail, map_remaining);
+	}
+
+	subflow->data_avail = data_avail;
+
+	return subflow->data_avail > 0;
 }
 
 static void subflow_data_ready(struct sock *sk)
Signature · 4 lines
2.23.0
_______________________________________________
mptcp mailing list -- mptcp(a)lists.01.org
To unsubscribe send an email to mptcp-leave(a)lists.01.org

--
Mat Martineau
Intel

author
#6 lore raw

[MPTCP] Re: [RFC PATCH 3/6] subflow: store known available bytes in subflow

Quoted 6 lines · Mat Martineau
Quoted 6 lines
	skb = skb_peek(&sk->sk_receive_queue);
net/mptcp/subflow.c +2 -3
-	subflow->data_avail = skb &&
-		       before(tcp_sk(sk)->copied_seq, TCP_SKB_CB(skb)->end_seq);
-	return subflow->data_avail;
+	if (skb) {
+		const struct tcp_sock *tp = tcp_sk(sk);
Quoted 8 lines

net/mptcp/subflow.c bool mptcp_subflow_data_available(struct sock *sk) · line 644

Quoted 3 lines
net/mptcp/subflow.c +3 -0
+		u32 map_remaining = subflow->map_data_len -
+				    mptcp_subflow_get_map_offset(subflow);
+

It looks like the mapping can be assumed valid here, since subflow_check_data_avail() returned true. It's not obvious looking at this function that map_valid should be true at this point - is it worth a WARN_ONCE or a comment?

I think I'll go with a WARN_ON_ONCE().

author
#7 lore raw

RFC PATCH 4/6 options: ack pending sequence

In reply to Florian Westphal

This allows us to include some of the bytes already pending on the subflow in the DSS ack instead of having to wait for the next recv (which updates msk->ack_seq value).

Signed-off-by Florian Westphal <fw(a)strlen.de>
---
 net/mptcp/options.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
net/mptcp/options.c +9 -2
index 80dbe7662cea..b488bf07ada5 100644
@@ -418,13 +418,20 @@ static bool mptcp_established_options_dss(struct sock *sk, struct sk_buff *skb,
 		ack_size += TCPOLEN_MPTCP_DSS_BASE;
 
 	if (ack_size <= remaining) {
+		const struct mptcp_subflow_context *subflow;
 		struct mptcp_sock *msk;
 
 		dss_size += ack_size;
 
-		msk = mptcp_sk(mptcp_subflow_ctx(sk)->conn);
+		subflow = mptcp_subflow_ctx(sk);
+		msk = mptcp_sk(subflow->conn);
 		if (msk) {
-			opts->ext_copy.data_ack = msk->ack_seq;
+			u64 ack_seq = READ_ONCE(msk->ack_seq);
+
+			if (subflow->map_valid)
+				ack_seq += subflow->data_avail;
+
+			opts->ext_copy.data_ack = ack_seq;
 		} else {
 			mptcp_crypto_key_sha1(mptcp_subflow_ctx(sk)->remote_key,
 					      NULL, &opts->ext_copy.data_ack);
Signature · 1 line
2.23.0
#8 lore raw

[MPTCP] Re: [RFC PATCH 4/6] options: ack pending sequence

Quoted 21 lines · Florian Westphal

This allows us to include some of the bytes already pending on the subflow in the DSS ack instead of having to wait for the next recv (which updates msk->ack_seq value).

Signed-off-by Florian Westphal <fw(a)strlen.de>

---
net/mptcp/options.c | 11 +++++++++--

1 file changed, 9 insertions(+), 2 deletions(-)
net/mptcp/options.c +1 -0
index 80dbe7662cea..b488bf07ada5 100644
@@ -418,13 +418,20 @@ static bool mptcp_established_options_dss(struct sock *sk, struct sk_buff *skb,
 		ack_size += TCPOLEN_MPTCP_DSS_BASE;
 
 	if (ack_size <= remaining) {
+		const struct mptcp_subflow_context *subflow;
 		struct mptcp_sock *msk;
 
 		dss_size += ack_size;

net/mptcp/options.c static bool mptcp_established_options_dss(struct sock *sk, struct sk_buff *skb, · line 432

Quoted 10 lines
net/mptcp/options.c +6 -2
 
-		msk = mptcp_sk(mptcp_subflow_ctx(sk)->conn);
+		subflow = mptcp_subflow_ctx(sk);
+		msk = mptcp_sk(subflow->conn);
 		if (msk) {
-			opts->ext_copy.data_ack = msk->ack_seq;
+			u64 ack_seq = READ_ONCE(msk->ack_seq);
+
+			if (subflow->map_valid)
+				ack_seq += subflow->data_avail;

I'm not sure this works well with multiple subflows. It could ack a later sequence number on this subflow, then send an earlier sequence number in a bare ack on another subflow. They should be close enough that the earlier seq would be ignored, but it would be better to send consistent acks.

Mat

Quoted 11 lines
net/mptcp/options.c +2 -0
+
+			opts->ext_copy.data_ack = ack_seq;
 		} else {
 			mptcp_crypto_key_sha1(mptcp_subflow_ctx(sk)->remote_key,
 					      NULL, &opts->ext_copy.data_ack);
Signature · 4 lines
2.23.0
_______________________________________________
mptcp mailing list -- mptcp(a)lists.01.org
To unsubscribe send an email to mptcp-leave(a)lists.01.org

--
Mat Martineau
Intel

author
#9 lore raw

[MPTCP] Re: [RFC PATCH 4/6] options: ack pending sequence

Quoted 13 lines · Mat Martineau
Quoted 13 lines
net/mptcp/options.c +1 -0
index 80dbe7662cea..b488bf07ada5 100644
@@ -418,13 +418,20 @@ static bool mptcp_established_options_dss(struct sock *sk, struct sk_buff *skb,
 		ack_size += TCPOLEN_MPTCP_DSS_BASE;
 
 	if (ack_size <= remaining) {
+		const struct mptcp_subflow_context *subflow;
 		struct mptcp_sock *msk;
 
 		dss_size += ack_size;
 
Quoted 14 lines

net/mptcp/options.c static bool mptcp_established_options_dss(struct sock *sk, struct sk_buff *skb, · line 432

Quoted 9 lines
net/mptcp/options.c +6 -2
-		msk = mptcp_sk(mptcp_subflow_ctx(sk)->conn);
+		subflow = mptcp_subflow_ctx(sk);
+		msk = mptcp_sk(subflow->conn);
 		if (msk) {
-			opts->ext_copy.data_ack = msk->ack_seq;
+			u64 ack_seq = READ_ONCE(msk->ack_seq);
+
+			if (subflow->map_valid)
+				ack_seq += subflow->data_avail;

I'm not sure this works well with multiple subflows. It could ack a later sequence number on this subflow, then send an earlier sequence number in a bare ack on another subflow. They should be close enough that the earlier seq would be ignored, but it would be better to send consistent acks.

That would require far more changes since this means we need to move msk->ack_seq update away from recvmsg.

I think its best to drop this patch for now, its annoying to have this behaviour of acking data that was consumed by userspace rather than what was received but I don't see a simple solution nor is it related to wmem accouting.

author
#10 lore raw

RFC PATCH 5/6 mptcp: add ack work queue

In reply to Florian Westphal

This is needed in the (unlikely) case that userspace is blocked in mptcp_sendmsg because wmem is exhausted.

In that case, only the rtx work queue will clean the rtx backlog, but it could take several milliseconds until it runs next, so add a dedicated work queue that will run as soon as possible.

We can avoid scheduling the work if the mptcp socket is writeable, next send/recv call will also clean acked dfrags.

An alternative approach is to add a new mptcp flag and re-use the existing retrans worker, let me know if this is the preferred solution.

Signed-off-by Florian Westphal <fw(a)strlen.de>
---
 net/mptcp/options.c  |  2 +-
 net/mptcp/protocol.c | 35 +++++++++++++++++++++++++++++++----
 net/mptcp/protocol.h |  3 ++-
 3 files changed, 34 insertions(+), 6 deletions(-)
net/mptcp/options.c +1 -1
index b488bf07ada5..81d7c3c026c8 100644
@@ -590,7 +590,7 @@ static void update_una(struct mptcp_sock *msk,
 		old_snd_una = atomic64_cmpxchg(&msk->snd_una, snd_una,
 					       new_snd_una);
 		if (old_snd_una == snd_una) {
-			mptcp_reset_timer((struct sock *)msk);
+			mptcp_data_acked((struct sock *)msk);
 			break;
 		}
 	}
net/mptcp/protocol.c +31 -4
index b93050056195..5c6948085ef4 100644
@@ -40,7 +40,7 @@ static bool mptcp_timer_pending(struct sock *sk)
 	return timer_pending(&inet_csk(sk)->icsk_retransmit_timer);
 }
 
-void mptcp_reset_timer(struct sock *sk)
+static void mptcp_reset_timer(struct sock *sk)
 {
 	struct inet_connection_sock *icsk = inet_csk(sk);
 	unsigned long tout;
@@ -52,6 +52,15 @@ void mptcp_reset_timer(struct sock *sk)
 	sk_reset_timer(sk, &icsk->icsk_retransmit_timer, jiffies + tout);
 }
 
+void mptcp_data_acked(struct sock *sk)
+{
+	mptcp_reset_timer(sk);
+
+	if (!sk_stream_is_writeable(sk) &&
+	    schedule_work(&mptcp_sk(sk)->ack_work))
+		sock_hold(sk);
+}
+
 static void mptcp_stop_timer(struct sock *sk)
 {
 	struct inet_connection_sock *icsk = inet_csk(sk);
@@ -647,6 +656,21 @@ static void mptcp_retransmit_timer(struct timer_list *t)
 	sock_put(sk);
 }
 
+static void mptcp_ack_work(struct work_struct *work)
+{
+	struct mptcp_sock *msk;
+	struct sock *sk;
+
+	msk = container_of(work, struct mptcp_sock, ack_work);
+	sk = &msk->sk.icsk_inet.sk;
+
+	lock_sock(sk);
+	mptcp_clean_una(sk);
+	release_sock(sk);
+
+	sock_put(sk);
+}
+
 static void mptcp_retransmit(struct work_struct *work)
 {
 	int orig_len, orig_offset, ret, mss_now = 0, size_goal = 0;
@@ -716,6 +740,7 @@ static int __mptcp_init_sock(struct sock *sk)
 	INIT_LIST_HEAD(&msk->rtx_queue);
 
 	INIT_WORK(&msk->rtx_work, mptcp_retransmit);
+	INIT_WORK(&msk->ack_work, mptcp_ack_work);
 
 	/* re-use the csk retrans timer for MPTCP-level retrans */
 	timer_setup(&msk->sk.icsk_retransmit_timer, mptcp_retransmit_timer, 0);
@@ -755,12 +780,14 @@ static void __mptcp_clear_xmit(struct sock *sk)
 		dfrag_clear(sk, dfrag);
 }
 
-static void mptcp_cancel_rtx_work(struct sock *sk)
+static void mptcp_cancel_work(struct sock *sk)
 {
 	struct mptcp_sock *msk = mptcp_sk(sk);
 
 	if (cancel_work_sync(&msk->rtx_work))
 		sock_put(sk);
+	if (cancel_work_sync(&msk->ack_work))
+		sock_put(sk);
 }
 
 static void mptcp_close(struct sock *sk, long timeout)
@@ -792,7 +819,7 @@ static void mptcp_close(struct sock *sk, long timeout)
 	__mptcp_clear_xmit(sk);
 	release_sock(sk);
 
-	mptcp_cancel_rtx_work(sk);
+	mptcp_cancel_work(sk);
 
 	sk_common_release(sk);
 }
@@ -802,7 +829,7 @@ static int mptcp_disconnect(struct sock *sk, int flags)
 	lock_sock(sk);
 	__mptcp_clear_xmit(sk);
 	release_sock(sk);
-	mptcp_cancel_rtx_work(sk);
+	mptcp_cancel_work(sk);
 	return tcp_disconnect(sk, flags);
 }
 
net/mptcp/protocol.h +2 -1
index bc7e8fa54afa..aeda56466219 100644
@@ -134,6 +134,7 @@ struct mptcp_sock {
 	unsigned long	flags;
 	u16		dport;
 	struct work_struct rtx_work;
+	struct work_struct ack_work;
 	struct list_head conn_list;
 	struct list_head rtx_queue;
 	struct socket	*subflow; /* outgoing connect/listener/!mp_capable */
@@ -290,7 +291,7 @@ void mptcp_get_options(const struct sk_buff *skb,
 
 void mptcp_finish_connect(struct sock *sk, int mp_capable);
 void mptcp_finish_join(struct sock *sk);
-void mptcp_reset_timer(struct sock *sk);
+void mptcp_data_acked(struct sock *sk);
 
 int mptcp_token_new_request(struct request_sock *req);
 void mptcp_token_destroy_request(u32 token);
Signature · 1 line
2.23.0
#11 lore raw

[MPTCP] Re: [RFC PATCH 5/6] mptcp: add ack work queue

Quoted 7 lines · Florian Westphal

This is needed in the (unlikely) case that userspace is blocked in mptcp_sendmsg because wmem is exhausted.

In that case, only the rtx work queue will clean the rtx backlog, but it could take several milliseconds until it runs next, so add a dedicated work queue that will run as soon as possible.

Quoted 5 lines

We can avoid scheduling the work if the mptcp socket is writeable, next send/recv call will also clean acked dfrags.

An alternative approach is to add a new mptcp flag and re-use the existing retrans worker, let me know if this is the preferred solution.

It looks like core TCP uses both approach for timers (overloading a timer with multiple duties, and using multiple timers for different activities).

Since mptcp_ack_work() matches closely the initial bits of mptcp_retransmit(), In this specific scenario I think reusing the existing worker (plus flag) would be better.

Cheers,

Paolo

author
#12 lore raw

[MPTCP] Re: [RFC PATCH 5/6] mptcp: add ack work queue

Quoted 3 lines · Paolo Abeni

Since mptcp_ack_work() matches closely the initial bits of mptcp_retransmit(), In this specific scenario I think reusing the existing worker (plus flag) would be better.

Done: https://git.breakpoint.cc/cgit/fw/mptcp-next.git/commit/?h=wmem_acct_05&id=fea55d959e29da51c605dad9c5e84544622d1b49

I'll resend once others had a chance to look at this series as well.

#13 lore raw

[MPTCP] Re: [RFC PATCH 5/6] mptcp: add ack work queue

Quoted 8 lines · Florian Westphal
Quoted 3 lines · Paolo Abeni

Since mptcp_ack_work() matches closely the initial bits of mptcp_retransmit(), In this specific scenario I think reusing the existing worker (plus flag) would be better.

Done: https://git.breakpoint.cc/cgit/fw/mptcp-next.git/commit/?h=wmem_acct_05&id=fea55d959e29da51c605dad9c5e84544622d1b49

I'll resend once others had a chance to look at this series as well.

I think the combined worker should be ok for DATA_FIN too (which Florian had mentioned sharing the ack workqueue with).

--
Mat Martineau
Intel

author
#14 lore raw

RFC PATCH 6/6 sendmsg: block until mptcp sk is writeable

In reply to Florian Westphal

This disables transmit of new data until the peer has acked enough mptcp data to get below the wspace write threshold (more than half of wspace upperlimit is available again).

Also have poll not report EPOLLOUT in this case, its not relevant if a subflow is writeable.

The latter is a temporary workaround that is needed because mptcp_poll walks the subflows and calls __tcp_poll on each of them. Because subflow ssk is usually writable, we will have to undo-that if the mptcp sndbuf is exhausted. This won't be needed anymore once __tcp_poll is removed, I am working on this.

Signed-off-by Florian Westphal <fw(a)strlen.de>
---
 net/mptcp/protocol.c | 18 ++++++++++++++++--
 1 file changed, 16 insertions(+), 2 deletions(-)
net/mptcp/protocol.c +16 -2
index 5c6948085ef4..db0261abe4df 100644
@@ -406,6 +406,18 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 		return ret;
 	}
 
+	timeo = sock_sndtimeo(sk, msg->msg_flags & MSG_DONTWAIT);
+
+	mptcp_clean_una(sk);
+
+	while (!sk_stream_is_writeable(sk)) {
+		ret = sk_stream_wait_memory(sk, &timeo);
+		if (ret)
+			goto out;
+
+		mptcp_clean_una(sk);
+	}
+
 	ssk = mptcp_subflow_get(msk);
 	if (!ssk) {
 		release_sock(sk);
@@ -421,8 +433,6 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 	pr_debug("conn_list->subflow=%p", ssk);
 
 	lock_sock(ssk);
-	mptcp_clean_una(sk);
-	timeo = sock_sndtimeo(sk, msg->msg_flags & MSG_DONTWAIT);
 	while (msg_data_left(msg)) {
 		ret = mptcp_sendmsg_frag(sk, ssk, msg, NULL, &timeo, &mss_now,
 					 &size_goal);
@@ -1328,6 +1338,10 @@ static __poll_t mptcp_poll(struct file *file, struct socket *sock,
 		tcp_sock = mptcp_subflow_tcp_socket(subflow);
 		ret |= __tcp_poll(tcp_sock->sk);
 	}
+
+	if (!sk_stream_is_writeable(sk))
+		ret &= ~(EPOLLOUT|EPOLLWRNORM);
+
 	release_sock(sk);
 
 	return ret;
Signature · 1 line
2.23.0