Skip to contentSkip to conversation

mptcp: just another complete datapath refactor

PATCH Superseded Paolo Abeni 3 participants 16 messages Nov 13, 2020 – Nov 18, 2020

Subsystems (2)

v2 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Paolo Abeni

author
#0 lore raw

v2 cover mptcp: just another complete datapath refactor

This itration addresses Mat's feedback and bring many bugfixes, all noted in the individual patches.

Now it survives a few self-test iterations, and I would consider this for merging.

Note: depend on "mptcp: refine MPTCP-level ack scheduling"

Paolo Abeni (6):
mptcp: separate accouting for wmem forward alloc mem
mptcp: protect the rx path with the msk socket spinlock
mptcp: allocate TX skbs in msk context.
mptcp: avoid a few atomic ops in the rx path
mptcp: simplify mptcp_nospace
mptcp: use mptcp backlog.

 net/mptcp/mptcp_diag.c |   2 +-
 net/mptcp/options.c    |  47 +--
 net/mptcp/protocol.c   | 659 +++++++++++++++++++++++++++++------------
 net/mptcp/protocol.h   |  23 +-
 net/mptcp/subflow.c    |  31 +-
 5 files changed, 534 insertions(+), 228 deletions(-)
Signature · 1 line
2.26.2
author
#1 lore raw

PATCH v2 1/6 mptcp: separate accouting for wmem forward alloc mem

Account separatelly the memory forward allocated for write operation. When sendmsg() need more memory it looks at wforward_alloc first and if that is exaused, move more space from sk_forward_alloc.

This will simplify the next patch

Signed-off-by Paolo Abeni <pabeni(a)redhat.com>
---
 net/mptcp/protocol.c | 52 +++++++++++++++++++++++++++++++++++++++-----
 net/mptcp/protocol.h |  1 +
 2 files changed, 48 insertions(+), 5 deletions(-)
net/mptcp/protocol.c +47 -5
index 6397083476e3..87d8659b9bcc 100644
@@ -852,9 +852,48 @@ static bool mptcp_frag_can_collapse_to(const struct mptcp_sock *msk,
 		df->data_seq + df->data_len == msk->write_seq;
 }
 
+static bool mptcp_wmem_alloc(struct sock *sk, int size)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+	int amount;
+
+	if (msk->wforward_alloc >= size)
+		goto account;
+
+	if (!sk_wmem_schedule(sk, size))
+		return false;
+
+	/* try to keep half fwd alloc memory for each direction */
+	amount = max(size, sk->sk_forward_alloc >> 1);
+	sk->sk_forward_alloc -= amount;
+	msk->wforward_alloc += amount;
+
+account:
+	msk->wforward_alloc -= size;
+	return true;
+}
+
+static void mptcp_wmem_uncharge(struct sock *sk, int size)
+{
+	mptcp_sk(sk)->wforward_alloc += size;
+}
+
+static void mptcp_mem_reclaim_partial(struct sock *sk)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+
+	sk->sk_forward_alloc += msk->wforward_alloc;
+	msk->wforward_alloc = 0;
+	sk_mem_reclaim_partial(sk);
+
+	/* split the remaining fwd allocated memory between rx and tx */
+	msk->wforward_alloc = sk->sk_forward_alloc >> 1;
+	sk->sk_forward_alloc -= msk->wforward_alloc;
+}
+
 static void dfrag_uncharge(struct sock *sk, int len)
 {
-	sk_mem_uncharge(sk, len);
+	mptcp_wmem_uncharge(sk, len);
 	sk_wmem_queued_add(sk, -len);
 }
 
@@ -909,8 +948,8 @@ static void mptcp_clean_una(struct sock *sk)
 	}
 
 out:
-	if (cleaned)
-		sk_mem_reclaim_partial(sk);
+	if (cleaned && tcp_under_memory_pressure(sk))
+		mptcp_mem_reclaim_partial(sk);
 }
 
 static void mptcp_clean_una_wakeup(struct sock *sk)
@@ -1334,11 +1373,12 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 		offset = dfrag->offset + dfrag->data_len;
 		psize = pfrag->size - offset;
 		psize = min_t(size_t, psize, msg_data_left(msg));
-		if (!sk_wmem_schedule(sk, psize + frag_truesize))
+		if (!mptcp_wmem_alloc(sk, psize + frag_truesize))
 			goto wait_for_memory;
 
 		if (copy_page_from_iter(dfrag->page, offset, psize,
 					&msg->msg_iter) != psize) {
+			mptcp_wmem_uncharge(sk, psize + frag_truesize);
 			ret = -EFAULT;
 			goto out;
 		}
@@ -1354,7 +1394,6 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 		 * Note: we charge such data both to sk and ssk
 		 */
 		sk_wmem_queued_add(sk, frag_truesize);
-		sk->sk_forward_alloc -= frag_truesize;
 		if (!dfrag_collapsed) {
 			get_page(dfrag->page);
 			list_add_tail(&dfrag->list, &msk->rtx_queue);
@@ -1977,6 +2016,7 @@ static int __mptcp_init_sock(struct sock *sk)
 	INIT_WORK(&msk->work, mptcp_worker);
 	msk->out_of_order_queue = RB_ROOT;
 	msk->first_pending = NULL;
+	msk->wforward_alloc = 0;
 
 	msk->ack_hint = NULL;
 	msk->first = NULL;
@@ -2167,6 +2207,8 @@ static void __mptcp_destroy_sock(struct sock *sk)
 
 	sk->sk_prot->destroy(sk);
 
+	sk->sk_forward_alloc += msk->wforward_alloc;
+	msk->wforward_alloc = 0;
 	sk_stream_kill_queues(sk);
 	xfrm_sk_free_policy(sk);
 	sk_refcnt_debug_release(sk);
net/mptcp/protocol.h +1 -0
index 07542f7b349a..2a842d228950 100644
@@ -219,6 +219,7 @@ struct mptcp_sock {
 	u64		ack_seq;
 	u64		rcv_wnd_sent;
 	u64		rcv_data_fin_seq;
+	int		wforward_alloc;
 	struct sock	*last_snd;
 	int		snd_burst;
 	int		old_wspace;
Signature · 1 line
2.26.2
author
#2 lore raw

[MPTCP] Re: [PATCH v2 1/6] mptcp: separate accouting for wmem forward alloc mem

Quoted 79 lines · Paolo Abeni

Account separatelly the memory forward allocated for write operation. When sendmsg() need more memory it looks at wforward_alloc first and if that is exaused, move more space from sk_forward_alloc.

This will simplify the next patch

Signed-off-by Paolo Abeni <pabeni(a)redhat.com>
---
 net/mptcp/protocol.c | 52 +++++++++++++++++++++++++++++++++++++++-----
 net/mptcp/protocol.h |  1 +
 2 files changed, 48 insertions(+), 5 deletions(-)
net/mptcp/protocol.c +42 -3
index 6397083476e3..87d8659b9bcc 100644
@@ -852,9 +852,48 @@ static bool mptcp_frag_can_collapse_to(const struct mptcp_sock *msk,
 		df->data_seq + df->data_len == msk->write_seq;
 }
 
+static bool mptcp_wmem_alloc(struct sock *sk, int size)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+	int amount;
+
+	if (msk->wforward_alloc >= size)
+		goto account;
+
+	if (!sk_wmem_schedule(sk, size))
+		return false;
+
+	/* try to keep half fwd alloc memory for each direction */
+	amount = max(size, sk->sk_forward_alloc >> 1);
+	sk->sk_forward_alloc -= amount;
+	msk->wforward_alloc += amount;
+
+account:
+	msk->wforward_alloc -= size;
+	return true;
+}
+
+static void mptcp_wmem_uncharge(struct sock *sk, int size)
+{
+	mptcp_sk(sk)->wforward_alloc += size;
+}
+
+static void mptcp_mem_reclaim_partial(struct sock *sk)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+
+	sk->sk_forward_alloc += msk->wforward_alloc;
+	msk->wforward_alloc = 0;
+	sk_mem_reclaim_partial(sk);
+
+	/* split the remaining fwd allocated memory between rx and tx */
+	msk->wforward_alloc = sk->sk_forward_alloc >> 1;
+	sk->sk_forward_alloc -= msk->wforward_alloc;
+}
+
 static void dfrag_uncharge(struct sock *sk, int len)
 {
-	sk_mem_uncharge(sk, len);
+	mptcp_wmem_uncharge(sk, len);
 	sk_wmem_queued_add(sk, -len);
 }
 
@@ -909,8 +948,8 @@ static void mptcp_clean_una(struct sock *sk)
 	}
 
 out:
-	if (cleaned)
-		sk_mem_reclaim_partial(sk);
+	if (cleaned && tcp_under_memory_pressure(sk))
+		mptcp_mem_reclaim_partial(sk);
 }
 
 static void mptcp_clean_una_wakeup(struct sock *sk)

net/mptcp/protocol.c static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) · line 1377

Quoted 6 lines
net/mptcp/protocol.c +1 -1
@@ -1334,11 +1373,12 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 		offset = dfrag->offset + dfrag->data_len;
 		psize = pfrag->size - offset;
 		psize = min_t(size_t, psize, msg_data_left(msg));
-		if (!sk_wmem_schedule(sk, psize + frag_truesize))
+		if (!mptcp_wmem_alloc(sk, psize + frag_truesize))
 			goto wait_for_memory;

As mentioned before, one possible follow-up to this series is open-code a MPTCP-specific variant of lock_sock()/release_sock() which will allow performing some additional/MPTCP specific actions under the msk level spinlock.

Specifically:

- in recevmsg(), in lock_sock(), splice sk->sk_receive_queue in msk-

Quoted 1 line

receive_queue

- still in recvmsg(), in release_sock(), update the bulk freed rmem

Overall this 2 will allow acquiring no additional spinlock at all in the average case for recvmsg()

- in sendmsg(), in lock_sock(), move as much memory as needed from sk_forward_memory to wforward_memory, eventually allocating it.
- in sendmsg(), in release_sock(), move again the unused wforward_memory to sk_forward_memory.

Overall this 2 will allow acquiring no additional spinlock at all in the average case for sendmsg() and will avoid artifacts due to the sk_forward_memory/wforward_memory separation.

But they additionally will allow replacing the wforward_memory field with a local variable.

So I'm wondering if I should already code the above, to avoid introducing this additional new msk field just to remove it in a few patches !?!

Any opinion more than welcome!

Paolo

#3 lore raw

[MPTCP] Re: [PATCH v2 1/6] mptcp: separate accouting for wmem forward alloc mem

Quoted 23 lines · Paolo Abeni

As mentioned before, one possible follow-up to this series is open-code a MPTCP-specific variant of lock_sock()/release_sock() which will allow performing some additional/MPTCP specific actions under the msk level spinlock.

Specifically:

- in recevmsg(), in lock_sock(), splice sk->sk_receive_queue in msk-

Quoted 1 line

receive_queue

- still in recvmsg(), in release_sock(), update the bulk freed rmem

Overall this 2 will allow acquiring no additional spinlock at all in the average case for recvmsg()

- in sendmsg(), in lock_sock(), move as much memory as needed from sk_forward_memory to wforward_memory, eventually allocating it.
- in sendmsg(), in release_sock(), move again the unused wforward_memory to sk_forward_memory.

Overall this 2 will allow acquiring no additional spinlock at all in the average case for sendmsg() and will avoid artifacts due to the sk_forward_memory/wforward_memory separation.

Quoted 6 lines

But they additionally will allow replacing the wforward_memory field with a local variable.

So I'm wondering if I should already code the above, to avoid introducing this additional new msk field just to remove it in a few patches !?!

Tricky. Do you have hunch of how much additional code churn that means?

If its not too much 'rewrite', I would prefer to do it later on even if that means axing the newly added field.

If it means to rewrite most of this again then its probably better to do it right away.

author
#4 lore raw

[MPTCP] Re: [PATCH v2 1/6] mptcp: separate accouting for wmem forward alloc mem

Quoted 24 lines · Florian Westphal
Quoted 23 lines · Paolo Abeni

As mentioned before, one possible follow-up to this series is open-code a MPTCP-specific variant of lock_sock()/release_sock() which will allow performing some additional/MPTCP specific actions under the msk level spinlock.

Specifically:

- in recevmsg(), in lock_sock(), splice sk->sk_receive_queue in msk-

Quoted 1 line

receive_queue

- still in recvmsg(), in release_sock(), update the bulk freed rmem

Overall this 2 will allow acquiring no additional spinlock at all in the average case for recvmsg()

- in sendmsg(), in lock_sock(), move as much memory as needed from sk_forward_memory to wforward_memory, eventually allocating it.
- in sendmsg(), in release_sock(), move again the unused wforward_memory to sk_forward_memory.

Overall this 2 will allow acquiring no additional spinlock at all in the average case for sendmsg() and will avoid artifacts due to the sk_forward_memory/wforward_memory separation.

Quoted 8 lines
Quoted 6 lines

But they additionally will allow replacing the wforward_memory field with a local variable.

So I'm wondering if I should already code the above, to avoid introducing this additional new msk field just to remove it in a few patches !?!

Tricky.

The trickest part is probably getting a corret estimate for the wmem we are going to use. But we don't really need to be very accurate.

Quoted 8 lines

Do you have hunch of how much additional code churn that means?

If its not too much 'rewrite', I would prefer to do it later on even if that means axing the newly added field.

If it means to rewrite most of this again then its probably better to do it right away.

eh... the only way I had for a reasonable answer to the above was... trying to code it, do I did it ;)

In the end I had to retain the newly added field - with a slightly different semantic and name:

https://github.com/pabeni/mptcp/tree/worker_must_die_6

I think overall it cleans things more than a bit...

/P

#5 lore raw

[MPTCP] Re: [PATCH v2 1/6] mptcp: separate accouting for wmem forward alloc mem

Quoted 49 lines · Paolo Abeni
Quoted 24 lines · Florian Westphal
Quoted 23 lines · Paolo Abeni

As mentioned before, one possible follow-up to this series is open-code a MPTCP-specific variant of lock_sock()/release_sock() which will allow performing some additional/MPTCP specific actions under the msk level spinlock.

Specifically:

- in recevmsg(), in lock_sock(), splice sk->sk_receive_queue in msk-

Quoted 1 line

receive_queue

- still in recvmsg(), in release_sock(), update the bulk freed rmem

Overall this 2 will allow acquiring no additional spinlock at all in the average case for recvmsg()

- in sendmsg(), in lock_sock(), move as much memory as needed from sk_forward_memory to wforward_memory, eventually allocating it.
- in sendmsg(), in release_sock(), move again the unused wforward_memory to sk_forward_memory.

Overall this 2 will allow acquiring no additional spinlock at all in the average case for sendmsg() and will avoid artifacts due to the sk_forward_memory/wforward_memory separation.

Quoted 8 lines
Quoted 6 lines

But they additionally will allow replacing the wforward_memory field with a local variable.

So I'm wondering if I should already code the above, to avoid introducing this additional new msk field just to remove it in a few patches !?!

Tricky.

The trickest part is probably getting a corret estimate for the wmem we are going to use. But we don't really need to be very accurate.

Quoted 8 lines

Do you have hunch of how much additional code churn that means?

If its not too much 'rewrite', I would prefer to do it later on even if that means axing the newly added field.

If it means to rewrite most of this again then its probably better to do it right away.

eh... the only way I had for a reasonable answer to the above was... trying to code it, do I did it ;)

Quoted 7 lines

In the end I had to retain the newly added field - with a slightly different semantic and name:

https://github.com/pabeni/mptcp/tree/worker_must_die_6

I think overall it cleans things more than a bit...

Yes, I find the version of this patch in
https://github.com/pabeni/mptcp/tree/worker_must_die_6.1 to be cleaner than the version posted here without mptcp_lock_sock().

Did you intend to use __sk in the body of the mptcp_lock_sock() macro?

Thanks,

--
Mat Martineau
Intel

author
#6 lore raw

[MPTCP] Re: [PATCH v2 1/6] mptcp: separate accouting for wmem forward alloc mem

Quoted 54 lines · Mat Martineau
Quoted 52 lines · Paolo Abeni
Quoted 24 lines · Florian Westphal
Quoted 23 lines · Paolo Abeni

As mentioned before, one possible follow-up to this series is open-code a MPTCP-specific variant of lock_sock()/release_sock() which will allow performing some additional/MPTCP specific actions under the msk level spinlock.

Specifically:

- in recevmsg(), in lock_sock(), splice sk->sk_receive_queue in msk-

Quoted 1 line

receive_queue

- still in recvmsg(), in release_sock(), update the bulk freed rmem

Overall this 2 will allow acquiring no additional spinlock at all in the average case for recvmsg()

- in sendmsg(), in lock_sock(), move as much memory as needed from sk_forward_memory to wforward_memory, eventually allocating it.
- in sendmsg(), in release_sock(), move again the unused wforward_memory to sk_forward_memory.

Overall this 2 will allow acquiring no additional spinlock at all in the average case for sendmsg() and will avoid artifacts due to the sk_forward_memory/wforward_memory separation.

Quoted 8 lines
Quoted 6 lines

But they additionally will allow replacing the wforward_memory field with a local variable.

So I'm wondering if I should already code the above, to avoid introducing this additional new msk field just to remove it in a few patches !?!

Tricky.

The trickest part is probably getting a corret estimate for the wmem we are going to use. But we don't really need to be very accurate.

Quoted 8 lines

Do you have hunch of how much additional code churn that means?

If its not too much 'rewrite', I would prefer to do it later on even if that means axing the newly added field.

If it means to rewrite most of this again then its probably better to do it right away.

eh... the only way I had for a reasonable answer to the above was... trying to code it, do I did it ;)

In the end I had to retain the newly added field - with a slightly different semantic and name:

Quoted 8 lines
Quoted 4 lines

https://github.com/pabeni/mptcp/tree/worker_must_die_6

I think overall it cleans things more than a bit...

Yes, I find the version of this patch in
https://github.com/pabeni/mptcp/tree/worker_must_die_6.1 to be cleaner than the version posted here without mptcp_lock_sock().

That was my impression, too ;)

Quoted 1 line

Did you intend to use __sk in the body of the mptcp_lock_sock() macro?

That is typo on my side! Thanks for finding it. The macro actually works only because a 'sk' variable is always available in the scopes where the macro is used/invoked. I'll fix that.

Since looks like _6 is a viable option I'll post a v3 with that, ok?

Thanks!

Paolo

author
#7 lore raw

PATCH v2 2/6 mptcp: protect the rx path with the msk socket spinlock

In reply to Paolo Abeni

Such spinlock is currently used only to protect the 'owned' flag inside the socket lock itself. With this patch we extend it's scope to protect the whole msk receive path and sk_forward_memory.

Given the above we can always move data into the msk receive queue (and OoO queue) from the subflow.

We leverage the previous commit, so that we need to acquire the spinlock in the tx path only when moving fwd memory to wfwd and vice versa.

recvmsg() must now explicitly acquire the socket spinlock when moving skbs out of sk_receive_queue. We use a second rx queue and splice the first into the latter to reduce the number locks required.

Signed-off-by Paolo Abeni <pabeni(a)redhat.com>

---
v1 -> v2:
- keep skb->destructor set for skbs in the msk receive queue and clear it at dequeue time. This simplify rmem accounting and reduces the deltas
- added {READ,WRITE}_ONCE annotation for ack_hint
- drop no-more-needed check in __mptcp_move_skbs(): we are always called under socket lock and can't race anymore with close

---
 net/mptcp/protocol.c | 148 +++++++++++++++++++++++++++++--------------
 net/mptcp/protocol.h |   4 ++
 2 files changed, 103 insertions(+), 49 deletions(-)
net/mptcp/protocol.c +99 -49
index 87d8659b9bcc..915d69957227 100644
@@ -446,7 +446,7 @@ static void mptcp_send_ack(struct mptcp_sock *msk, bool force)
 
 		/* if the hintes ssk is still active, use it */
 		pick = ssk;
-		if (ssk == msk->ack_hint)
+		if (ssk == READ_ONCE(msk->ack_hint))
 			break;
 	}
 	if (!force && pick) {
@@ -606,13 +606,13 @@ static bool __mptcp_move_skbs_from_subflow(struct mptcp_sock *msk,
 			break;
 		}
 	} while (more_data_avail);
-	msk->ack_hint = ssk;
+	WRITE_ONCE(msk->ack_hint, ssk);
 
 	*bytes += moved;
 	return done;
 }
 
-static bool mptcp_ofo_queue(struct mptcp_sock *msk)
+static bool __mptcp_ofo_queue(struct mptcp_sock *msk)
 {
 	struct sock *sk = (struct sock *)msk;
 	struct sk_buff *skb, *tail;
@@ -658,34 +658,27 @@ static bool mptcp_ofo_queue(struct mptcp_sock *msk)
 /* In most cases we will be able to lock the mptcp socket.  If its already
  * owned, we need to defer to the work queue to avoid ABBA deadlock.
  */
-static bool move_skbs_to_msk(struct mptcp_sock *msk, struct sock *ssk)
+static void move_skbs_to_msk(struct mptcp_sock *msk, struct sock *ssk)
 {
 	struct sock *sk = (struct sock *)msk;
 	unsigned int moved = 0;
 
-	if (READ_ONCE(sk->sk_lock.owned))
-		return false;
-
-	if (unlikely(!spin_trylock_bh(&sk->sk_lock.slock)))
-		return false;
-
-	/* must re-check after taking the lock */
-	if (!READ_ONCE(sk->sk_lock.owned)) {
-		__mptcp_move_skbs_from_subflow(msk, ssk, &moved);
-		mptcp_ofo_queue(msk);
+	if (inet_sk_state_load(sk) == TCP_CLOSE)
+		return;
 
-		/* If the moves have caught up with the DATA_FIN sequence number
-		 * it's time to ack the DATA_FIN and change socket state, but
-		 * this is not a good place to change state. Let the workqueue
-		 * do it.
-		 */
-		if (mptcp_pending_data_fin(sk, NULL))
-			mptcp_schedule_work(sk);
-	}
+	mptcp_data_lock(sk);
 
-	spin_unlock_bh(&sk->sk_lock.slock);
+	__mptcp_move_skbs_from_subflow(msk, ssk, &moved);
+	__mptcp_ofo_queue(msk);
 
-	return moved > 0;
+	/* If the moves have caught up with the DATA_FIN sequence number
+	 * it's time to ack the DATA_FIN and change socket state, but
+	 * this is not a good place to change state. Let the workqueue
+	 * do it.
+	 */
+	if (mptcp_pending_data_fin(sk, NULL))
+		mptcp_schedule_work(sk);
+	mptcp_data_unlock(sk);
 }
 
 void mptcp_data_ready(struct sock *sk, struct sock *ssk)
@@ -860,13 +853,17 @@ static bool mptcp_wmem_alloc(struct sock *sk, int size)
 	if (msk->wforward_alloc >= size)
 		goto account;
 
-	if (!sk_wmem_schedule(sk, size))
+	mptcp_data_lock(sk);
+	if (!sk_wmem_schedule(sk, size)) {
+		mptcp_data_unlock(sk);
 		return false;
+	}
 
 	/* try to keep half fwd alloc memory for each direction */
 	amount = max(size, sk->sk_forward_alloc >> 1);
 	sk->sk_forward_alloc -= amount;
 	msk->wforward_alloc += amount;
+	mptcp_data_unlock(sk);
 
 account:
 	msk->wforward_alloc -= size;
@@ -878,12 +875,11 @@ static void mptcp_wmem_uncharge(struct sock *sk, int size)
 	mptcp_sk(sk)->wforward_alloc += size;
 }
 
-static void mptcp_mem_reclaim_partial(struct sock *sk)
+static void __mptcp_mem_reclaim_partial(struct sock *sk)
 {
 	struct mptcp_sock *msk = mptcp_sk(sk);
 
 	sk->sk_forward_alloc += msk->wforward_alloc;
-	msk->wforward_alloc = 0;
 	sk_mem_reclaim_partial(sk);
 
 	/* split the remaining fwd allocated memory between rx and tx */
@@ -891,6 +887,13 @@ static void mptcp_mem_reclaim_partial(struct sock *sk)
 	sk->sk_forward_alloc -= msk->wforward_alloc;
 }
 
+static void mptcp_mem_reclaim_partial(struct sock *sk)
+{
+	mptcp_data_lock(sk);
+	__mptcp_mem_reclaim_partial(sk);
+	mptcp_data_unlock(sk);
+}
+
 static void dfrag_uncharge(struct sock *sk, int len)
 {
 	mptcp_wmem_uncharge(sk, len);
@@ -1442,13 +1445,12 @@ static void mptcp_wait_data(struct sock *sk, long *timeo)
 
 static int __mptcp_recvmsg_mskq(struct mptcp_sock *msk,
 				struct msghdr *msg,
-				size_t len)
+				size_t len, int *released)
 {
-	struct sock *sk = (struct sock *)msk;
 	struct sk_buff *skb;
 	int copied = 0;
 
-	while ((skb = skb_peek(&sk->sk_receive_queue)) != NULL) {
+	while ((skb = skb_peek(&msk->receive_queue)) != NULL) {
 		u32 offset = MPTCP_SKB_CB(skb)->offset;
 		u32 data_len = skb->len - offset;
 		u32 count = min_t(size_t, len - copied, data_len);
@@ -1468,7 +1470,10 @@ static int __mptcp_recvmsg_mskq(struct mptcp_sock *msk,
 			break;
 		}
 
-		__skb_unlink(skb, &sk->sk_receive_queue);
+		/* we will bulk release the skb memory later */
+		skb->destructor = NULL;
+		*released += skb->truesize;
+		__skb_unlink(skb, &msk->receive_queue);
 		__kfree_skb(skb);
 
 		if (copied >= len)
@@ -1576,25 +1581,45 @@ static void mptcp_rcv_space_adjust(struct mptcp_sock *msk, int copied)
 	msk->rcvq_space.time = mstamp;
 }
 
-static bool __mptcp_move_skbs(struct mptcp_sock *msk, unsigned int rcv)
+static void __mptcp_update_rmem(struct sock *sk, int *released)
 {
-	unsigned int moved = 0;
-	bool done;
+	atomic_sub(*released, &sk->sk_rmem_alloc);
+	sk_mem_uncharge(sk, *released);
+	*released = 0;
+}
 
-	/* avoid looping forever below on racing close */
-	if (((struct sock *)msk)->sk_state == TCP_CLOSE)
-		return false;
+static void mptcp_update_rmem(struct sock *sk, int *released)
+{
+	if (!*released)
+		return;
+
+	mptcp_data_lock(sk);
+	__mptcp_update_rmem(sk, released);
+	mptcp_data_unlock(sk);
+}
+
+static bool __mptcp_move_skbs(struct mptcp_sock *msk, unsigned int rcv, int *released)
+{
+	struct sock *sk = (struct sock *)msk;
+	unsigned int moved = 0;
+	bool ret, done;
 
 	__mptcp_flush_join_list(msk);
 	do {
 		struct sock *ssk = mptcp_subflow_recv_lookup(msk);
 		bool slowpath;
 
-		if (!ssk)
+		/* we can have data pending in the subflows only if the msk
+		 * receive buffer was full at subflow_data_ready() time,
+		 * it is an likely slow path
+		 */
+		if (likely(!ssk))
 			break;
 
 		slowpath = lock_sock_fast(ssk);
+		mptcp_data_lock(sk);
 		done = __mptcp_move_skbs_from_subflow(msk, ssk, &moved);
+		mptcp_data_unlock(sk);
 		if (moved && rcv) {
 			WRITE_ONCE(msk->rmem_pending, min(rcv, moved));
 			tcp_cleanup_rbuf(ssk, 1);
@@ -1603,17 +1628,27 @@ static bool __mptcp_move_skbs(struct mptcp_sock *msk, unsigned int rcv)
 		unlock_sock_fast(ssk, slowpath);
 	} while (!done);
 
-	if (mptcp_ofo_queue(msk) || moved > 0) {
-		mptcp_check_data_fin((struct sock *)msk);
-		return true;
+	/* acquire the data lock only if some input data is pending */
+	ret = moved > 0;
+	if (!RB_EMPTY_ROOT(&msk->out_of_order_queue) ||
+	    !skb_queue_empty_lockless(&sk->sk_receive_queue)) {
+		mptcp_data_lock(sk);
+		__mptcp_update_rmem(sk, released);
+		ret |= __mptcp_ofo_queue(msk);
+		skb_queue_splice_tail_init(&sk->sk_receive_queue,
+					   &msk->receive_queue);
+		mptcp_data_unlock(sk);
 	}
-	return false;
+	if (ret)
+		mptcp_check_data_fin((struct sock *)msk);
+	return !skb_queue_empty(&msk->receive_queue);
 }
 
 static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 			 int nonblock, int flags, int *addr_len)
 {
 	struct mptcp_sock *msk = mptcp_sk(sk);
+	int released = 0;
 	int copied = 0;
 	int target;
 	long timeo;
@@ -1631,7 +1666,8 @@ static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 	for (;;) {
 		int bytes_read, old_space;
 
-		bytes_read = __mptcp_recvmsg_mskq(msk, msg, len - copied);
+		bytes_read = __mptcp_recvmsg_mskq(msk, msg, len - copied,
+						  &released);
 		if (unlikely(bytes_read < 0)) {
 			if (!copied)
 				copied = bytes_read;
@@ -1640,8 +1676,8 @@ static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 
 		copied += bytes_read;
 
-		if (skb_queue_empty(&sk->sk_receive_queue) &&
-		    __mptcp_move_skbs(msk, len - copied))
+		if (skb_queue_empty(&msk->receive_queue) &&
+		    __mptcp_move_skbs(msk, len - copied, &released))
 			continue;
 
 		/* be sure to advertize window change */
@@ -1671,8 +1707,15 @@ static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 			if (test_and_clear_bit(MPTCP_WORK_EOF, &msk->flags))
 				mptcp_check_for_eof(msk);
 
-			if (sk->sk_shutdown & RCV_SHUTDOWN)
+			if (sk->sk_shutdown & RCV_SHUTDOWN) {
+				/* race breaker: the shutdown could be after the
+				 * previous receive queue check
+				 */
+				if (__mptcp_move_skbs(msk, len - copied,
+						      &released))
+					continue;
 				break;
+			}
 
 			if (sk->sk_state == TCP_CLOSE) {
 				copied = -ENOTCONN;
@@ -1691,26 +1734,29 @@ static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 		}
 
 		pr_debug("block timeout %ld", timeo);
+		mptcp_update_rmem(sk, &released);
 		mptcp_wait_data(sk, &timeo);
 	}
 
-	if (skb_queue_empty(&sk->sk_receive_queue)) {
+	if (skb_queue_empty_lockless(&sk->sk_receive_queue) &&
+	    skb_queue_empty(&msk->receive_queue)) {
 		/* entire backlog drained, clear DATA_READY. */
 		clear_bit(MPTCP_DATA_READY, &msk->flags);
 
 		/* .. race-breaker: ssk might have gotten new data
 		 * after last __mptcp_move_skbs() returned false.
 		 */
-		if (unlikely(__mptcp_move_skbs(msk, 0)))
+		if (unlikely(__mptcp_move_skbs(msk, 0, &released)))
 			set_bit(MPTCP_DATA_READY, &msk->flags);
 	} else if (unlikely(!test_bit(MPTCP_DATA_READY, &msk->flags))) {
 		/* data to read but mptcp_wait_data() cleared DATA_READY */
 		set_bit(MPTCP_DATA_READY, &msk->flags);
 	}
 out_err:
+	mptcp_update_rmem(sk, &released);
 	pr_debug("msk=%p data_ready=%d rx queue empty=%d copied=%d",
 		 msk, test_bit(MPTCP_DATA_READY, &msk->flags),
-		 skb_queue_empty(&sk->sk_receive_queue), copied);
+		 skb_queue_empty_lockless(&sk->sk_receive_queue), copied);
 	mptcp_rcv_space_adjust(msk, copied);
 
 	release_sock(sk);
@@ -2014,6 +2060,7 @@ static int __mptcp_init_sock(struct sock *sk)
 	INIT_LIST_HEAD(&msk->join_list);
 	INIT_LIST_HEAD(&msk->rtx_queue);
 	INIT_WORK(&msk->work, mptcp_worker);
+	__skb_queue_head_init(&msk->receive_queue);
 	msk->out_of_order_queue = RB_ROOT;
 	msk->first_pending = NULL;
 	msk->wforward_alloc = 0;
@@ -2428,6 +2475,9 @@ static struct sock *mptcp_accept(struct sock *sk, int flags, int *err,
 
 void mptcp_destroy_common(struct mptcp_sock *msk)
 {
+	struct sock *sk = (struct sock *)msk;
+
+	skb_queue_splice_tail_init(&sk->sk_receive_queue, &msk->receive_queue);
 	skb_rbtree_purge(&msk->out_of_order_queue);
 	mptcp_token_destroy(msk);
 	mptcp_pm_free_anno_list(msk);
net/mptcp/protocol.h +4 -0
index 2a842d228950..54d5d117ae28 100644
@@ -239,6 +239,7 @@ struct mptcp_sock {
 	struct work_struct work;
 	struct sk_buff  *ooo_last_skb;
 	struct rb_root  out_of_order_queue;
+	struct sk_buff_head receive_queue;
 	struct list_head conn_list;
 	struct list_head rtx_queue;
 	struct mptcp_data_frag *first_pending;
@@ -255,6 +256,9 @@ struct mptcp_sock {
 	} rcvq_space;
 };
 
+#define mptcp_data_lock(sk) spin_lock_bh(&(sk)->sk_lock.slock)
+#define mptcp_data_unlock(sk) spin_unlock_bh(&(sk)->sk_lock.slock)
+
 #define mptcp_for_each_subflow(__msk, __subflow)			\
 	list_for_each_entry(__subflow, &((__msk)->conn_list), node)
 
Signature · 1 line
2.26.2
author
#8 lore raw

PATCH v2 3/6 mptcp: allocate TX skbs in msk context.

In reply to Paolo Abeni

Move the TX skbs allocation in mptcp_sendmsg() scope, and tentativelly allocate a skbs based on the goal size on the last used subflow.

Use the ssk tx skb cache to prevent the subflow allocation.

This allows removing the msk skb extension cache and will make possible the later patch.

Signed-off-by Paolo Abeni <pabeni(a)redhat.com>
---
 net/mptcp/protocol.c | 215 +++++++++++++++++++++++++++++++++++--------
 net/mptcp/protocol.h |   4 +-
 2 files changed, 180 insertions(+), 39 deletions(-)
net/mptcp/protocol.c +177 -38
index 915d69957227..1f5c2b7cf678 100644
@@ -797,16 +797,6 @@ static void mptcp_check_for_eof(struct mptcp_sock *msk)
 		inet_sk_state_store(sk, TCP_CLOSE);
 }
 
-static bool mptcp_ext_cache_refill(struct mptcp_sock *msk)
-{
-	const struct sock *sk = (const struct sock *)msk;
-
-	if (!msk->cached_ext)
-		msk->cached_ext = __skb_ext_alloc(sk->sk_allocation);
-
-	return !!msk->cached_ext;
-}
-
 static struct sock *mptcp_subflow_recv_lookup(const struct mptcp_sock *msk)
 {
 	struct mptcp_subflow_context *subflow;
@@ -968,19 +958,12 @@ static void mptcp_clean_una_wakeup(struct sock *sk)
 	}
 }
 
-/* ensure we get enough memory for the frag hdr, beyond some minimal amount of
- * data
- */
-static bool mptcp_page_frag_refill(struct sock *sk, struct page_frag *pfrag)
+static void mptcp_enter_memory_pressure(struct sock *sk)
 {
 	struct mptcp_subflow_context *subflow;
 	struct mptcp_sock *msk = mptcp_sk(sk);
 	bool first = true;
 
-	if (likely(skb_page_frag_refill(32U + sizeof(struct mptcp_data_frag),
-					pfrag, sk->sk_allocation)))
-		return true;
-
 	sk_stream_moderate_sndbuf(sk);
 	mptcp_for_each_subflow(msk, subflow) {
 		struct sock *ssk = mptcp_subflow_tcp_sock(subflow);
@@ -990,6 +973,18 @@ static bool mptcp_page_frag_refill(struct sock *sk, struct page_frag *pfrag)
 		sk_stream_moderate_sndbuf(ssk);
 		first = false;
 	}
+}
+
+/* ensure we get enough memory for the frag hdr, beyond some minimal amount of
+ * data
+ */
+static bool mptcp_page_frag_refill(struct sock *sk, struct page_frag *pfrag)
+{
+	if (likely(skb_page_frag_refill(32U + sizeof(struct mptcp_data_frag),
+					pfrag, sk->sk_allocation)))
+		return true;
+
+	mptcp_enter_memory_pressure(sk);
 	return false;
 }
 
@@ -1036,6 +1031,128 @@ static int mptcp_check_allowed_size(struct mptcp_sock *msk, u64 data_seq,
 	return avail_size;
 }
 
+static bool __mptcp_add_ext(struct sk_buff *skb, gfp_t gfp)
+{
+	struct skb_ext *mpext = __skb_ext_alloc(gfp);
+
+	if (!mpext)
+		return false;
+	__skb_ext_set(skb, SKB_EXT_MPTCP, mpext);
+	return true;
+}
+
+static struct sk_buff *__mptcp_do_alloc_tx_skb(struct sock *sk)
+{
+	struct sk_buff *skb;
+
+	skb = alloc_skb_fclone(MAX_TCP_HEADER, sk->sk_allocation);
+	if (likely(skb)) {
+		if (likely(__mptcp_add_ext(skb, sk->sk_allocation))) {
+			skb_reserve(skb, MAX_TCP_HEADER);
+			skb->reserved_tailroom = skb->end - skb->tail;
+			return skb;
+		}
+		__kfree_skb(skb);
+	} else {
+		mptcp_enter_memory_pressure(sk);
+	}
+	return NULL;
+}
+
+static bool mptcp_tx_cache_refill(struct sock *sk, int size,
+				  struct sk_buff_head *skbs, int *total_ts)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+	struct sk_buff *skb;
+	int space_needed;
+
+	if (unlikely(tcp_under_memory_pressure(sk))) {
+		mptcp_mem_reclaim_partial(sk);
+
+		/* under pressure pre-allocate at most a single skb */
+		if (msk->skb_tx_cache.qlen)
+			return true;
+		space_needed = msk->size_goal_cache;
+	} else {
+		space_needed = msk->tx_pending_data + size -
+			       msk->skb_tx_cache.qlen * msk->size_goal_cache;
+	}
+
+	while (space_needed > 0) {
+		skb = __mptcp_do_alloc_tx_skb(sk);
+		if (unlikely(!skb)) {
+			/* under memory pressure, try to pass the caller a
+			 * single skb to allow forward progress
+			 */
+			while (skbs->qlen > 1) {
+				skb = __skb_dequeue_tail(skbs);
+				__kfree_skb(skb);
+			}
+			return skbs->qlen > 0;
+		}
+
+		*total_ts += skb->truesize;
+		__skb_queue_tail(skbs, skb);
+		space_needed -= msk->size_goal_cache;
+	}
+	return true;
+}
+
+static bool __mptcp_alloc_tx_skb(struct sock *sk, struct sock *ssk)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+	struct sk_buff *skb;
+
+	if (ssk->sk_tx_skb_cache) {
+		skb = ssk->sk_tx_skb_cache;
+		if (unlikely(!skb_ext_find(skb, SKB_EXT_MPTCP) &&
+			     !__mptcp_add_ext(skb, sk->sk_allocation)))
+			return false;
+		return true;
+	}
+
+	skb = skb_peek(&msk->skb_tx_cache);
+	if (skb) {
+		if (likely(sk_wmem_schedule(ssk, skb->truesize))) {
+			skb = __skb_dequeue(&msk->skb_tx_cache);
+			if (WARN_ON_ONCE(!skb))
+				return false;
+
+			mptcp_wmem_uncharge(sk, skb->truesize);
+			ssk->sk_tx_skb_cache = skb;
+			return true;
+		}
+
+		/* over memory limit, no point to try to allocate a new skb */
+		return false;
+	}
+
+	skb = __mptcp_do_alloc_tx_skb(sk);
+	if (!skb)
+		return false;
+
+	if (likely(sk_wmem_schedule(ssk, skb->truesize))) {
+		ssk->sk_tx_skb_cache = skb;
+		return true;
+	}
+	kfree_skb(skb);
+	return false;
+}
+
+static bool mptcp_must_reclaim_memory(struct sock *sk, struct sock *ssk)
+{
+	return !ssk->sk_tx_skb_cache &&
+	       !skb_peek(&mptcp_sk(sk)->skb_tx_cache) &&
+	       tcp_under_memory_pressure(sk);
+}
+
+static bool mptcp_alloc_tx_skb(struct sock *sk, struct sock *ssk)
+{
+	if (unlikely(mptcp_must_reclaim_memory(sk, ssk)))
+		mptcp_mem_reclaim_partial(sk);
+	return __mptcp_alloc_tx_skb(sk, ssk);
+}
+
 static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,
 			      struct mptcp_data_frag *dfrag,
 			      struct mptcp_sendmsg_info *info)
@@ -1047,7 +1164,7 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,
 	struct sk_buff *skb, *tail;
 	bool can_collapse = false;
 	int avail_size;
-	size_t ret;
+	size_t ret = 0;
 
 	pr_debug("msk=%p ssk=%p sending dfrag at seq=%lld len=%d already sent=%d",
 		 msk, ssk, dfrag->data_seq, dfrag->data_len, info->sent);
@@ -1055,6 +1172,7 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,
 	/* compute send limit */
 	info->mss_now = tcp_send_mss(ssk, &info->size_goal, info->flags);
 	avail_size = info->size_goal;
+	msk->size_goal_cache = info->size_goal;
 	skb = tcp_write_queue_tail(ssk);
 	if (skb) {
 		/* Limit the write to the size available in the
@@ -1103,8 +1221,11 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,
 		goto out;
 	}
 
-	mpext = __skb_ext_set(tail, SKB_EXT_MPTCP, msk->cached_ext);
-	msk->cached_ext = NULL;
+	mpext = skb_ext_find(tail, SKB_EXT_MPTCP);
+	if (WARN_ON_ONCE(!mpext)) {
+		/* should never reach here, stream corrupted */
+		return -EINVAL;
+	}
 
 	memset(mpext, 0, sizeof(*mpext));
 	mpext->data_seq = data_seq;
@@ -1177,9 +1298,6 @@ static struct sock *mptcp_subflow_get_send(struct mptcp_sock *msk,
 	sock_owned_by_me((struct sock *)msk);
 
 	*sndbuf = 0;
-	if (!mptcp_ext_cache_refill(msk))
-		return NULL;
-
 	if (__mptcp_check_fallback(msk)) {
 		if (!msk->first)
 			return NULL;
@@ -1288,6 +1406,15 @@ static void mptcp_push_pending(struct sock *sk, unsigned int flags)
 			if (ssk != prev_ssk || !prev_ssk)
 				lock_sock(ssk);
 
+			/* keep it simple and always provide a new skb for the
+			 * subflow, even if we will not use it when collapsing
+			 * on the pending one
+			 */
+			if (!mptcp_alloc_tx_skb(sk, ssk)) {
+				mptcp_push_release(sk, ssk, &info);
+				goto out;
+			}
+
 			ret = mptcp_sendmsg_frag(sk, ssk, dfrag, &info);
 			if (ret <= 0) {
 				mptcp_push_release(sk, ssk, &info);
@@ -1298,6 +1425,7 @@ static void mptcp_push_pending(struct sock *sk, unsigned int flags)
 			dfrag->already_sent += ret;
 			msk->snd_nxt += ret;
 			msk->snd_burst -= ret;
+			msk->tx_pending_data -= ret;
 			copied += ret;
 			len -= ret;
 		}
@@ -1341,8 +1469,9 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 	mptcp_clean_una(sk);
 
 	while (msg_data_left(msg)) {
+		int total_ts, frag_truesize = 0;
 		struct mptcp_data_frag *dfrag;
-		int frag_truesize = 0;
+		struct sk_buff_head skbs;
 		bool dfrag_collapsed;
 		size_t psize, offset;
 
@@ -1376,9 +1505,17 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 		offset = dfrag->offset + dfrag->data_len;
 		psize = pfrag->size - offset;
 		psize = min_t(size_t, psize, msg_data_left(msg));
-		if (!mptcp_wmem_alloc(sk, psize + frag_truesize))
+		total_ts = psize + frag_truesize;
+		__skb_queue_head_init(&skbs);
+		if (!mptcp_tx_cache_refill(sk, psize, &skbs, &total_ts))
+			goto wait_for_memory;
+
+		if (!mptcp_wmem_alloc(sk, total_ts)) {
+			__skb_queue_purge(&skbs);
 			goto wait_for_memory;
+		}
 
+		skb_queue_splice_tail(&skbs, &msk->skb_tx_cache);
 		if (copy_page_from_iter(dfrag->page, offset, psize,
 					&msg->msg_iter) != psize) {
 			mptcp_wmem_uncharge(sk, psize + frag_truesize);
@@ -1407,8 +1544,6 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 			 dfrag->data_seq, dfrag->data_len, dfrag->already_sent,
 			 !dfrag_collapsed);
 
-		if (!mptcp_ext_cache_refill(msk))
-			goto wait_for_memory;
 		continue;
 
 wait_for_memory:
@@ -1420,8 +1555,10 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 			goto out;
 	}
 
-	if (copied)
+	if (copied) {
+		msk->tx_pending_data += copied;
 		mptcp_push_pending(sk, msg->msg_flags);
+	}
 
 out:
 	release_sock(sk);
@@ -2009,9 +2146,6 @@ static void mptcp_worker(struct work_struct *work)
 	if (!dfrag)
 		goto unlock;
 
-	if (!mptcp_ext_cache_refill(msk))
-		goto reset_unlock;
-
 	ssk = mptcp_subflow_get_retrans(msk);
 	if (!ssk)
 		goto reset_unlock;
@@ -2022,6 +2156,9 @@ static void mptcp_worker(struct work_struct *work)
 	info.sent = 0;
 	info.limit = dfrag->already_sent;
 	while (info.sent < dfrag->already_sent) {
+		if (!mptcp_alloc_tx_skb(sk, ssk))
+			break;
+
 		ret = mptcp_sendmsg_frag(sk, ssk, dfrag, &info);
 		if (ret <= 0)
 			break;
@@ -2029,9 +2166,6 @@ static void mptcp_worker(struct work_struct *work)
 		MPTCP_INC_STATS(sock_net(sk), MPTCP_MIB_RETRANSSEGS);
 		copied += ret;
 		info.sent += ret;
-
-		if (!mptcp_ext_cache_refill(msk))
-			break;
 	}
 	if (copied)
 		tcp_push(ssk, 0, info.mss_now, tcp_sk(ssk)->nonagle,
@@ -2061,9 +2195,12 @@ static int __mptcp_init_sock(struct sock *sk)
 	INIT_LIST_HEAD(&msk->rtx_queue);
 	INIT_WORK(&msk->work, mptcp_worker);
 	__skb_queue_head_init(&msk->receive_queue);
+	__skb_queue_head_init(&msk->skb_tx_cache);
 	msk->out_of_order_queue = RB_ROOT;
 	msk->first_pending = NULL;
 	msk->wforward_alloc = 0;
+	msk->tx_pending_data = 0;
+	msk->size_goal_cache = TCP_BASE_MSS;
 
 	msk->ack_hint = NULL;
 	msk->first = NULL;
@@ -2111,12 +2248,17 @@ static void __mptcp_clear_xmit(struct sock *sk)
 {
 	struct mptcp_sock *msk = mptcp_sk(sk);
 	struct mptcp_data_frag *dtmp, *dfrag;
+	struct sk_buff *skb;
 
 	sk_stop_timer(sk, &msk->sk.icsk_retransmit_timer);
 
 	WRITE_ONCE(msk->first_pending, NULL);
 	list_for_each_entry_safe(dfrag, dtmp, &msk->rtx_queue, list)
 		dfrag_clear(sk, dfrag);
+	while ((skb = __skb_dequeue(&msk->skb_tx_cache)) != NULL) {
+		mptcp_wmem_uncharge(sk, skb->truesize);
+		kfree_skb(skb);
+	}
 }
 
 static void mptcp_cancel_work(struct sock *sk)
@@ -2487,9 +2629,6 @@ static void mptcp_destroy(struct sock *sk)
 {
 	struct mptcp_sock *msk = mptcp_sk(sk);
 
-	if (msk->cached_ext)
-		__skb_ext_put(msk->cached_ext);
-
 	mptcp_destroy_common(msk);
 	sk_sockets_allocated_dec(sk);
 }
net/mptcp/protocol.h +3 -1
index 54d5d117ae28..df8e40aeb011 100644
@@ -240,11 +240,13 @@ struct mptcp_sock {
 	struct sk_buff  *ooo_last_skb;
 	struct rb_root  out_of_order_queue;
 	struct sk_buff_head receive_queue;
+	struct sk_buff_head skb_tx_cache;	/* this is wmem accounted */
+	int		tx_pending_data;
+	int		size_goal_cache;
 	struct list_head conn_list;
 	struct list_head rtx_queue;
 	struct mptcp_data_frag *first_pending;
 	struct list_head join_list;
-	struct skb_ext	*cached_ext;	/* for the next sendmsg */
 	struct socket	*subflow; /* outgoing connect/listener/!mp_capable */
 	struct sock	*first;
 	struct mptcp_pm_data	pm;
Signature · 1 line
2.26.2
author
#9 lore raw

PATCH v2 4/6 mptcp: avoid a few atomic ops in the rx path

In reply to Paolo Abeni

Extending the data_lock scope in mptcp_incoming_option we can use that to protect both snd_una and wnd_end. In the typical case we will have a single atomic op instead of 2

Signed-off-by Paolo Abeni <pabeni(a)redhat.com>
---
 net/mptcp/mptcp_diag.c |  2 +-
 net/mptcp/options.c    | 33 +++++++++++++--------------------
 net/mptcp/protocol.c   | 33 +++++++++++++++------------------
 net/mptcp/protocol.h   |  8 ++++----
 4 files changed, 33 insertions(+), 43 deletions(-)
net/mptcp/mptcp_diag.c +1 -1
index 5f390a97f556..b70ae4ba3000 100644
@@ -140,7 +140,7 @@ static void mptcp_diag_get_info(struct sock *sk, struct inet_diag_msg *r,
 	info->mptcpi_flags = flags;
 	info->mptcpi_token = READ_ONCE(msk->token);
 	info->mptcpi_write_seq = READ_ONCE(msk->write_seq);
-	info->mptcpi_snd_una = atomic64_read(&msk->snd_una);
+	info->mptcpi_snd_una = READ_ONCE(msk->snd_una);
 	info->mptcpi_rcv_nxt = READ_ONCE(msk->ack_seq);
 	unlock_sock_fast(sk, slow);
 }
net/mptcp/options.c +13 -20
index 8a59b3e44599..3986454a0340 100644
@@ -833,15 +833,17 @@ static void ack_update_msk(struct mptcp_sock *msk,
 			   const struct sock *ssk,
 			   struct mptcp_options_received *mp_opt)
 {
-	u64 new_snd_una, snd_una, old_snd_una = atomic64_read(&msk->snd_una);
-	u64 new_wnd_end, wnd_end, old_wnd_end = atomic64_read(&msk->wnd_end);
-	u64 snd_nxt = READ_ONCE(msk->snd_nxt);
+	u64 new_wnd_end, new_snd_una, snd_nxt = READ_ONCE(msk->snd_nxt);
 	struct sock *sk = (struct sock *)msk;
+	u64 old_snd_una;
+
+	mptcp_data_lock(sk);
 
 	/* avoid ack expansion on update conflict, to reduce the risk of
 	 * wrongly expanding to a future ack sequence number, which is way
 	 * more dangerous than missing an ack
 	 */
+	old_snd_una = msk->snd_una;
 	new_snd_una = expand_ack(old_snd_una, mp_opt->data_ack, mp_opt->ack64);
 
 	/* ACK for data not even sent yet? Ignore. */
@@ -850,26 +852,17 @@ static void ack_update_msk(struct mptcp_sock *msk,
 
 	new_wnd_end = new_snd_una + tcp_sk(ssk)->snd_wnd;
 
-	while (after64(new_wnd_end, old_wnd_end)) {
-		wnd_end = old_wnd_end;
-		old_wnd_end = atomic64_cmpxchg(&msk->wnd_end, wnd_end,
-					       new_wnd_end);
-		if (old_wnd_end == wnd_end) {
-			if (mptcp_send_head(sk))
-				mptcp_schedule_work(sk);
-			break;
-		}
+	if (after64(new_wnd_end, msk->wnd_end)) {
+		msk->wnd_end = new_wnd_end;
+		if (mptcp_send_head(sk))
+			mptcp_schedule_work(sk);
 	}
 
-	while (after64(new_snd_una, old_snd_una)) {
-		snd_una = old_snd_una;
-		old_snd_una = atomic64_cmpxchg(&msk->snd_una, snd_una,
-					       new_snd_una);
-		if (old_snd_una == snd_una) {
-			mptcp_data_acked(sk);
-			break;
-		}
+	if (after64(new_snd_una, old_snd_una)) {
+		msk->snd_una = new_snd_una;
+		__mptcp_data_acked(sk);
 	}
+	mptcp_data_unlock(sk);
 }
 
 bool mptcp_update_rcv_data_fin(struct mptcp_sock *msk, u64 data_fin_seq, bool use_64bit)
net/mptcp/protocol.c +15 -18
index 1f5c2b7cf678..2a8ccffd91ec 100644
@@ -72,7 +72,7 @@ static struct socket *__mptcp_nmpc_socket(const struct mptcp_sock *msk)
 /* Returns end sequence number of the receiver's advertised window */
 static u64 mptcp_wnd_end(const struct mptcp_sock *msk)
 {
-	return atomic64_read(&msk->wnd_end);
+	return READ_ONCE(msk->wnd_end);
 }
 
 static bool mptcp_is_tcpsk(struct sock *sk)
@@ -357,7 +357,7 @@ static void mptcp_check_data_fin_ack(struct sock *sk)
 	/* Look for an acknowledged DATA_FIN */
 	if (((1 << sk->sk_state) &
 	     (TCPF_FIN_WAIT1 | TCPF_CLOSING | TCPF_LAST_ACK)) &&
-	    msk->write_seq == atomic64_read(&msk->snd_una)) {
+	    msk->write_seq == READ_ONCE(msk->snd_una)) {
 		mptcp_stop_timer(sk);
 
 		WRITE_ONCE(msk->snd_data_fin_enable, 0);
@@ -756,7 +756,7 @@ bool mptcp_schedule_work(struct sock *sk)
 	return false;
 }
 
-void mptcp_data_acked(struct sock *sk)
+void __mptcp_data_acked(struct sock *sk)
 {
 	mptcp_reset_timer(sk);
 
@@ -910,10 +910,9 @@ static void mptcp_clean_una(struct sock *sk)
 	 * plain TCP
 	 */
 	if (__mptcp_check_fallback(msk))
-		atomic64_set(&msk->snd_una, msk->snd_nxt);
-
-	snd_una = atomic64_read(&msk->snd_una);
+		msk->snd_una = READ_ONCE(msk->snd_nxt);
 
+	snd_una = msk->snd_una;
 	list_for_each_entry_safe(dfrag, dtmp, &msk->rtx_queue, list) {
 		if (after64(dfrag->data_seq + dfrag->data_len, snd_una))
 			break;
@@ -1193,10 +1192,12 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,
 	/* Zero window and all data acked? Probe. */
 	avail_size = mptcp_check_allowed_size(msk, data_seq, avail_size);
 	if (avail_size == 0) {
-		if (skb || atomic64_read(&msk->snd_una) != msk->snd_nxt)
+		u64 snd_una = READ_ONCE(msk->snd_una);
+
+		if (skb || snd_una != msk->snd_nxt)
 			return 0;
 		zero_window_probe = true;
-		data_seq = atomic64_read(&msk->snd_una) - 1;
+		data_seq = snd_una - 1;
 		avail_size = 1;
 	}
 
@@ -1904,12 +1905,8 @@ static void mptcp_retransmit_handler(struct sock *sk)
 {
 	struct mptcp_sock *msk = mptcp_sk(sk);
 
-	if (atomic64_read(&msk->snd_una) == READ_ONCE(msk->snd_nxt)) {
-		mptcp_stop_timer(sk);
-	} else {
-		set_bit(MPTCP_WORK_RTX, &msk->flags);
-		mptcp_schedule_work(sk);
-	}
+	set_bit(MPTCP_WORK_RTX, &msk->flags);
+	mptcp_schedule_work(sk);
 }
 
 static void mptcp_retransmit_timer(struct timer_list *t)
@@ -2527,8 +2524,8 @@ struct sock *mptcp_sk_clone(const struct sock *sk,
 
 	msk->write_seq = subflow_req->idsn + 1;
 	msk->snd_nxt = msk->write_seq;
-	atomic64_set(&msk->snd_una, msk->write_seq);
-	atomic64_set(&msk->wnd_end, msk->snd_nxt + req->rsk_rcv_wnd);
+	msk->snd_una = msk->write_seq;
+	msk->wnd_end = msk->snd_nxt + req->rsk_rcv_wnd;
 
 	if (mp_opt->mp_capable) {
 		msk->can_ack = true;
@@ -2566,7 +2563,7 @@ void mptcp_rcv_space_init(struct mptcp_sock *msk, const struct sock *ssk)
 	if (msk->rcvq_space.space == 0)
 		msk->rcvq_space.space = TCP_INIT_CWND * TCP_MSS_DEFAULT;
 
-	atomic64_set(&msk->wnd_end, msk->snd_nxt + tcp_sk(ssk)->snd_wnd);
+	WRITE_ONCE(msk->wnd_end, msk->snd_nxt + tcp_sk(ssk)->snd_wnd);
 }
 
 static struct sock *mptcp_accept(struct sock *sk, int flags, int *err,
@@ -2822,7 +2819,7 @@ void mptcp_finish_connect(struct sock *ssk)
 	WRITE_ONCE(msk->ack_seq, ack_seq);
 	WRITE_ONCE(msk->rcv_wnd_sent, ack_seq);
 	WRITE_ONCE(msk->can_ack, 1);
-	atomic64_set(&msk->snd_una, msk->write_seq);
+	WRITE_ONCE(msk->snd_una, msk->write_seq);
 
 	mptcp_pm_new_connection(msk, 0);
 
net/mptcp/protocol.h +4 -4
index df8e40aeb011..2da75e0e4ce2 100644
@@ -223,8 +223,8 @@ struct mptcp_sock {
 	struct sock	*last_snd;
 	int		snd_burst;
 	int		old_wspace;
-	atomic64_t	snd_una;
-	atomic64_t	wnd_end;
+	u64		snd_una;
+	u64		wnd_end;
 	unsigned long	timer_ival;
 	u32		token;
 	int		rmem_pending;
@@ -308,7 +308,7 @@ static inline struct mptcp_data_frag *mptcp_rtx_tail(const struct sock *sk)
 {
 	struct mptcp_sock *msk = mptcp_sk(sk);
 
-	if (!before64(msk->snd_nxt, atomic64_read(&msk->snd_una)))
+	if (!before64(msk->snd_nxt, READ_ONCE(msk->snd_una)))
 		return NULL;
 
 	return list_last_entry(&msk->rtx_queue, struct mptcp_data_frag, list);
@@ -482,7 +482,7 @@ void mptcp_rcv_space_init(struct mptcp_sock *msk, const struct sock *ssk);
 void mptcp_data_ready(struct sock *sk, struct sock *ssk);
 bool mptcp_finish_join(struct sock *sk);
 bool mptcp_schedule_work(struct sock *sk);
-void mptcp_data_acked(struct sock *sk);
+void __mptcp_data_acked(struct sock *sk);
 void mptcp_subflow_eof(struct sock *sk);
 bool mptcp_update_rcv_data_fin(struct mptcp_sock *msk, u64 data_fin_seq, bool use_64bit);
 void __mptcp_flush_join_list(struct mptcp_sock *msk);
Signature · 1 line
2.26.2
author
#10 lore raw

PATCH v2 5/6 mptcp: simplify mptcp_nospace

In reply to Paolo Abeni

Currently mptcp_space has to traverse the whole conn_list setting the relevant bit on each subflow, because the client ones will have a different sk_socket.

We can actually leave the NOSPACE always set on all client subflow and make nospace simpler.

Signed-off-by Paolo Abeni <pabeni(a)redhat.com>
---
 net/mptcp/protocol.c | 15 +--------------
 net/mptcp/subflow.c  | 31 ++++++++++++++++++++++---------
 2 files changed, 23 insertions(+), 23 deletions(-)
net/mptcp/protocol.c +1 -14
index 2a8ccffd91ec..db1db0e0fc2f 100644
@@ -1252,23 +1252,9 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,
 
 static void mptcp_nospace(struct mptcp_sock *msk)
 {
-	struct mptcp_subflow_context *subflow;
-
 	set_bit(MPTCP_NOSPACE, &msk->flags);
 	smp_mb__after_atomic(); /* msk->flags is changed by write_space cb */
 
-	mptcp_for_each_subflow(msk, subflow) {
-		struct sock *ssk = mptcp_subflow_tcp_sock(subflow);
-		bool ssk_writeable = sk_stream_is_writeable(ssk);
-		struct socket *sock = READ_ONCE(ssk->sk_socket);
-
-		if (ssk_writeable || !sock)
-			continue;
-
-		/* enables ssk->write_space() callbacks */
-		set_bit(SOCK_NOSPACE, &sock->flags);
-	}
-
 	/* mptcp_data_acked() could run just before we set the NOSPACE bit,
 	 * so explicitly check for snd_una value
 	 */
@@ -3050,6 +3036,7 @@ static int mptcp_stream_accept(struct socket *sock, struct socket *newsock,
 		slowpath = lock_sock_fast(newsk);
 		mptcp_copy_inaddrs(newsk, msk->first);
 		mptcp_rcv_space_init(msk, msk->first);
+		set_bit(SOCK_NOSPACE, &newsock->flags);
 
 		/* set ssk->sk_socket of accept()ed flows to mptcp socket.
 		 * This is needed so NOSPACE flag can be set from tcp stack.
net/mptcp/subflow.c +22 -9
index 02d757e0ddb7..e53428892122 100644
@@ -996,19 +996,28 @@ static void subflow_data_ready(struct sock *sk)
 		mptcp_data_ready(parent, sk);
 }
 
-static void subflow_write_space(struct sock *sk)
+static void subflow_write_space(struct sock *ssk)
 {
-	struct mptcp_subflow_context *subflow = mptcp_subflow_ctx(sk);
-	struct socket *sock = READ_ONCE(sk->sk_socket);
-	struct sock *parent = subflow->conn;
+	struct mptcp_subflow_context *subflow = mptcp_subflow_ctx(ssk);
+	struct sock *sk = subflow->conn;
+	struct socket_wq *wq;
 
-	if (!sk_stream_is_writeable(sk))
+	if (!sk_stream_is_writeable(ssk) || !sk_stream_is_writeable(sk) ||
+	    !sk->sk_socket || !test_bit(MPTCP_NOSPACE, &mptcp_sk(sk)->flags))
 		return;
 
-	if (sock && sk_stream_is_writeable(parent))
-		clear_bit(SOCK_NOSPACE, &sock->flags);
-
-	sk_stream_write_space(parent);
+	/* The following is quite alike sk_stream_write_space, but avoids
+	 * clearing the sk SOCK_NOSPACE bit
+	 */
+	clear_bit(MPTCP_NOSPACE, &mptcp_sk(sk)->flags);
+	rcu_read_lock();
+	wq = rcu_dereference(sk->sk_wq);
+	if (skwq_has_sleeper(wq))
+		wake_up_interruptible_poll(&wq->wait, EPOLLOUT |
+					   EPOLLWRNORM | EPOLLWRBAND);
+	if (wq && wq->fasync_list && !(sk->sk_shutdown & SEND_SHUTDOWN))
+		sock_wake_async(wq, SOCK_WAKE_SPACE, POLL_OUT);
+	rcu_read_unlock();
 }
 
 static struct inet_connection_sock_af_ops *
@@ -1208,6 +1217,10 @@ int mptcp_subflow_create_socket(struct sock *sk, struct socket **new_sock)
 	SOCK_INODE(sf)->i_uid = SOCK_INODE(sk->sk_socket)->i_uid;
 	SOCK_INODE(sf)->i_gid = SOCK_INODE(sk->sk_socket)->i_gid;
 
+	/* subflows will always call into sk_write_space, and subflow_write_space()
+	 * will be responsible of doing the actual wake-up
+	 */
+	set_bit(SOCK_NOSPACE, &sf->flags);
 	subflow = mptcp_subflow_ctx(sf->sk);
 	pr_debug("subflow=%p", subflow);
 
Signature · 1 line
2.26.2
author
#11 lore raw

PATCH v2 6/6 mptcp: use mptcp backlog.

In reply to Paolo Abeni

We have some tasks triggered by the subflow receive path which require to access the msk socket status, specifically:
mptcp_clean_una() and mptcp_push_pending()

We have almost everything in place to use the msk backlog and a standard !owned by user schema for that.

Introduce a dummy per msk sk_buff to trigger into the backlog as needed.

Since the worker is no more used to clean the acked data, for fallback sockets we need to explicitly flush them.

As added bonus we can simplify a lot mptcp_poll() and move the timer update under the data lock.

The worker is now used only to process and send DATA_FIN packets and do the mptcp-level retransmissions.

Signed-off-by: Paolo Abeni <pabeni(a)redhat.com>
--
v1 -> v2
- add tx_pending_data accounting in __mptcp_subflow_push_pending()
- in mptcp_data_acked schedule explicitly schedule the worker if ack is pending - checking sk status is both too liberal and not enough.
- add missing call to __mptcp_check_send_data_fin() in mptcp_worker() this is needed to cope with __mptcp_subflow_push_pending() deferring send ack to the worker
- renamed backlog to bl_entry
- sk_mem_reclaim_partial() -> __mptcp_mem_reclaim_partial() in mptcp_clean_una(). We are both under the data_lock and the socket lock so we can call the more effective mptcp variant
- do not try to push pending frames on wnd update if no pending data (use the new __mptcp_wnd_updated helper for that)
- move write queue cleanup in destroy_common() so that it takes place after that subflows stops touching msk->sk_forward_alloc. and we don't need to acquire the msk socket lock spin lock to reclaim the xmit memory
- explicitly init info->flags in __mptcp_subflow_push_pending(), or horrible corruption will happen later
- explicitly flush pending data for fallback socket on ack reception. This likely fixes an older issue, which was hidden by the worker, spooling the pending data, just slower.

---
 net/mptcp/options.c  |  18 +++-
 net/mptcp/protocol.c | 230 +++++++++++++++++++++++++++----------------
 net/mptcp/protocol.h |   6 ++
 3 files changed, 166 insertions(+), 88 deletions(-)
net/mptcp/options.c +14 -4
index 3986454a0340..5810c639a0cc 100644
@@ -830,7 +830,7 @@ static u64 expand_ack(u64 old_ack, u64 cur_ack, bool use_64bit)
 }
 
 static void ack_update_msk(struct mptcp_sock *msk,
-			   const struct sock *ssk,
+			   struct sock *ssk,
 			   struct mptcp_options_received *mp_opt)
 {
 	u64 new_wnd_end, new_snd_una, snd_nxt = READ_ONCE(msk->snd_nxt);
@@ -854,8 +854,7 @@ static void ack_update_msk(struct mptcp_sock *msk,
 
 	if (after64(new_wnd_end, msk->wnd_end)) {
 		msk->wnd_end = new_wnd_end;
-		if (mptcp_send_head(sk))
-			mptcp_schedule_work(sk);
+		__mptcp_wnd_updated(sk, ssk);
 	}
 
 	if (after64(new_snd_una, old_snd_una)) {
@@ -915,8 +914,19 @@ void mptcp_incoming_options(struct sock *sk, struct sk_buff *skb)
 	struct mptcp_options_received mp_opt;
 	struct mptcp_ext *mpext;
 
-	if (__mptcp_check_fallback(msk))
+	if (__mptcp_check_fallback(msk)) {
+		/* Keep it simple and unconditionally trigger send data cleanup and
+		 * pending queue spooling. We will need to acquire the data lock
+		 * for more accurate checks, and once the lock is acquired, such
+		 * helpers are cheap.
+		 */
+		mptcp_data_lock(subflow->conn);
+		if (mptcp_send_head(subflow->conn))
+		    __mptcp_wnd_updated(subflow->conn, sk);
+		__mptcp_data_acked(subflow->conn);
+		mptcp_data_unlock(subflow->conn);
 		return;
+	}
 
 	mptcp_get_options(skb, &mp_opt);
 	if (!check_fully_established(msk, sk, subflow, skb, &mp_opt))
net/mptcp/protocol.c +146 -84

414 lines not on this page · raw message

net/mptcp/protocol.h +6 -0
index 2da75e0e4ce2..99a6348be345 100644
@@ -92,6 +92,8 @@
 #define MPTCP_FALLBACK_DONE	4
 #define MPTCP_WORK_CLOSE_SUBFLOW 5
 #define MPTCP_WORKER_RUNNING	6
+#define MPTCP_PUSH_PENDING	7
+#define MPTCP_CLEAN_UNA	8
 
 static inline bool before64(__u64 seq1, __u64 seq2)
 {
@@ -249,6 +251,9 @@ struct mptcp_sock {
 	struct list_head join_list;
 	struct socket	*subflow; /* outgoing connect/listener/!mp_capable */
 	struct sock	*first;
+
+	struct sk_buff bl_entry;
+
 	struct mptcp_pm_data	pm;
 	struct {
 		u32	space;	/* bytes copied in last measurement window */
@@ -482,6 +487,7 @@ void mptcp_rcv_space_init(struct mptcp_sock *msk, const struct sock *ssk);
 void mptcp_data_ready(struct sock *sk, struct sock *ssk);
 bool mptcp_finish_join(struct sock *sk);
 bool mptcp_schedule_work(struct sock *sk);
+void __mptcp_wnd_updated(struct sock *sk, struct sock *ssk);
 void __mptcp_data_acked(struct sock *sk);
 void mptcp_subflow_eof(struct sock *sk);
 bool mptcp_update_rcv_data_fin(struct mptcp_sock *msk, u64 data_fin_seq, bool use_64bit);
Signature · 1 line
2.26.2
#12 lore raw

[MPTCP] Re: [PATCH v2 6/6] mptcp: use mptcp backlog.

Quoted 50 lines · Paolo Abeni

We have some tasks triggered by the subflow receive path which require to access the msk socket status, specifically:
mptcp_clean_una() and mptcp_push_pending()

We have almost everything in place to use the msk backlog and a standard !owned by user schema for that.

Introduce a dummy per msk sk_buff to trigger into the backlog as needed.

Since the worker is no more used to clean the acked data, for fallback sockets we need to explicitly flush them.

As added bonus we can simplify a lot mptcp_poll() and move the timer update under the data lock.

The worker is now used only to process and send DATA_FIN packets and do the mptcp-level retransmissions.

Signed-off-by: Paolo Abeni <pabeni(a)redhat.com>
--
v1 -> v2
- add tx_pending_data accounting in __mptcp_subflow_push_pending()
- in mptcp_data_acked schedule explicitly schedule the worker if ack is pending - checking sk status is both too liberal and not enough.
- add missing call to __mptcp_check_send_data_fin() in mptcp_worker() this is needed to cope with __mptcp_subflow_push_pending() deferring send ack to the worker
- renamed backlog to bl_entry
- sk_mem_reclaim_partial() -> __mptcp_mem_reclaim_partial() in mptcp_clean_una(). We are both under the data_lock and the socket lock so we can call the more effective mptcp variant
- do not try to push pending frames on wnd update if no pending data (use the new __mptcp_wnd_updated helper for that)
- move write queue cleanup in destroy_common() so that it takes place after that subflows stops touching msk->sk_forward_alloc. and we don't need to acquire the msk socket lock spin lock to reclaim the xmit memory
- explicitly init info->flags in __mptcp_subflow_push_pending(), or horrible corruption will happen later
- explicitly flush pending data for fallback socket on ack reception. This likely fixes an older issue, which was hidden by the worker, spooling the pending data, just slower.
---

net/mptcp/options.c  |  18 +++-
net/mptcp/protocol.c | 230 +++++++++++++++++++++++++++----------------
net/mptcp/protocol.h |   6 ++
3 files changed, 166 insertions(+), 88 deletions(-)

...

Quoted 4 lines
net/mptcp/protocol.c +0 -0
index db1db0e0fc2f..6fbd394d6701 100644

...

Quoted 37 lines
net/mptcp/protocol.c +33 -0
@@ -2726,6 +2758,50 @@ static int mptcp_getsockopt(struct sock *sk, int level, int optname,
 	return -EOPNOTSUPP;
 }
 
+static void __mptcp_add_to_backlog(struct sock *sk, int event)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+
+	if (!sk->sk_backlog.tail)
+		__sk_add_backlog(sk, &msk->bl_entry);
+
+	set_bit(event, &msk->flags);
+}
+
+void __mptcp_data_acked(struct sock *sk)
+{
+	if (!sock_owned_by_user(sk))
+		__mptcp_clean_una(sk);
+	else
+		__mptcp_add_to_backlog(sk, MPTCP_CLEAN_UNA);
+
+	if (mptcp_pending_data_fin_ack(sk))
+		mptcp_schedule_work(sk);
+}
+
+void __mptcp_wnd_updated(struct sock *sk, struct sock *ssk)
+{
+	if (!mptcp_send_head(sk))
+		return;
+
+	if (!sock_owned_by_user(sk))
+		__mptcp_subflow_push_pending(sk, ssk);
+	else
+		__mptcp_add_to_backlog(sk, MPTCP_PUSH_PENDING);
+}
+
+static int mptcp_do_backlog(struct sock *sk, struct sk_buff *skb)

net/mptcp/protocol.c static int mptcp_getsockopt(struct sock *sk, int level, int optname, · line 2803

Quoted 10 lines
net/mptcp/protocol.c +10 -0
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+
+	if (test_and_clear_bit(MPTCP_CLEAN_UNA, &msk->flags))
+		mptcp_clean_una(sk);
+
+	if (test_and_clear_bit(MPTCP_PUSH_PENDING, &msk->flags))
+		mptcp_push_pending(sk, 0);
+	return 0;
+}

Is there anything happening here in mptcp_do_backlog() that doesn't work in mptcp_release_cb()?

(this goes for this version of the patch as well as the updated one in
https://github.com/pabeni/mptcp/tree/worker_must_die_6.1 )

It looks like mptcp_do_backlog() ends up getting invoked through release_sock(), right before sk->sk_prot->release_cb() is called - which suggests that maybe mptcp_release_cb() could do it all. Is there another code path for MPTCP that calls sk_backlog_rcv() or __release_sock() that I've missed?

The zeroed-out bl_entry skb seems fragile. The current code in the backlog ends up handling it ok today, but that could easily change. Not that I'm trying to veto the idea, but it does need some more discussion :)

--
Mat Martineau
Intel

author
#13 lore raw

[MPTCP] Re: [PATCH v2 6/6] mptcp: use mptcp backlog.

Quoted 104 lines · Mat Martineau
Quoted 50 lines · Paolo Abeni

We have some tasks triggered by the subflow receive path which require to access the msk socket status, specifically:
mptcp_clean_una() and mptcp_push_pending()

We have almost everything in place to use the msk backlog and a standard !owned by user schema for that.

Introduce a dummy per msk sk_buff to trigger into the backlog as needed.

Since the worker is no more used to clean the acked data, for fallback sockets we need to explicitly flush them.

As added bonus we can simplify a lot mptcp_poll() and move the timer update under the data lock.

The worker is now used only to process and send DATA_FIN packets and do the mptcp-level retransmissions.

Signed-off-by: Paolo Abeni <pabeni(a)redhat.com>
--
v1 -> v2
- add tx_pending_data accounting in __mptcp_subflow_push_pending()
- in mptcp_data_acked schedule explicitly schedule the worker if ack is pending - checking sk status is both too liberal and not enough.
- add missing call to __mptcp_check_send_data_fin() in mptcp_worker() this is needed to cope with __mptcp_subflow_push_pending() deferring send ack to the worker
- renamed backlog to bl_entry
- sk_mem_reclaim_partial() -> __mptcp_mem_reclaim_partial() in mptcp_clean_una(). We are both under the data_lock and the socket lock so we can call the more effective mptcp variant
- do not try to push pending frames on wnd update if no pending data (use the new __mptcp_wnd_updated helper for that)
- move write queue cleanup in destroy_common() so that it takes place after that subflows stops touching msk->sk_forward_alloc. and we don't need to acquire the msk socket lock spin lock to reclaim the xmit memory
- explicitly init info->flags in __mptcp_subflow_push_pending(), or horrible corruption will happen later
- explicitly flush pending data for fallback socket on ack reception. This likely fixes an older issue, which was hidden by the worker, spooling the pending data, just slower.
---

net/mptcp/options.c  |  18 +++-
net/mptcp/protocol.c | 230 +++++++++++++++++++++++++++----------------
net/mptcp/protocol.h |   6 ++
3 files changed, 166 insertions(+), 88 deletions(-)

...

Quoted 4 lines
net/mptcp/protocol.c +0 -0
index db1db0e0fc2f..6fbd394d6701 100644

...

Quoted 42 lines
net/mptcp/protocol.c +38 -0
@@ -2726,6 +2758,50 @@ static int mptcp_getsockopt(struct sock *sk, int level, int optname,
 	return -EOPNOTSUPP;
 }
 
+static void __mptcp_add_to_backlog(struct sock *sk, int event)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+
+	if (!sk->sk_backlog.tail)
+		__sk_add_backlog(sk, &msk->bl_entry);
+
+	set_bit(event, &msk->flags);
+}
+
+void __mptcp_data_acked(struct sock *sk)
+{
+	if (!sock_owned_by_user(sk))
+		__mptcp_clean_una(sk);
+	else
+		__mptcp_add_to_backlog(sk, MPTCP_CLEAN_UNA);
+
+	if (mptcp_pending_data_fin_ack(sk))
+		mptcp_schedule_work(sk);
+}
+
+void __mptcp_wnd_updated(struct sock *sk, struct sock *ssk)
+{
+	if (!mptcp_send_head(sk))
+		return;
+
+	if (!sock_owned_by_user(sk))
+		__mptcp_subflow_push_pending(sk, ssk);
+	else
+		__mptcp_add_to_backlog(sk, MPTCP_PUSH_PENDING);
+}
+
+static int mptcp_do_backlog(struct sock *sk, struct sk_buff *skb)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+
+	if (test_and_clear_bit(MPTCP_CLEAN_UNA, &msk->flags))
+		mptcp_clean_una(sk);
Quoted 8 lines

net/mptcp/protocol.c static int mptcp_getsockopt(struct sock *sk, int level, int optname, · line 2803

Quoted 5 lines
net/mptcp/protocol.c +5 -0
+
+	if (test_and_clear_bit(MPTCP_PUSH_PENDING, &msk->flags))
+		mptcp_push_pending(sk, 0);
+	return 0;
+}

Is there anything happening here in mptcp_do_backlog() that doesn't work in mptcp_release_cb()?

Yes, we can't call mptcp_push_pending() in release_cb(), because mptcp_push_pending() requires a non atomic context, while release_cb is invoked under the msk socket spin lock.

We can't use __mptcp_subflow_push_pending() - this latter can be invoked under atomic context - because we don't have an ssk handy. We could look-up a suitable ssk and invoke __mptcp_subflow_push_pending(), but I think is preferrable leveraging mptcp_push_pending().

Quoted 3 lines

The zeroed-out bl_entry skb seems fragile. The current code in the backlog ends up handling it ok today, but that could easily change. Not that I'm trying to veto the idea, but it does need some more discussion :)

I think we can avoid the backlog usage with the above alternative.

There is a downside:

mptcp_push_pending() and __mptcp_subflow_push_pending() may require an skb allocation - even if we try to preallocate a suitable skbs number, the estimate could be wrong and under pressure we fall-back to preallocating a single skb.

Currently memory allocation failures in mptcp_push_pending() and __mptcp_subflow_push_pending() are not handled explicitly - quite alike what TCP does. When that happens, TCP will try again at RTX timeout. We currently don't do that, but we could add that logic. Anyhow I think we should try to avoid such code path as much as possible (workqueue usage and long delay due to RTX).

__mptcp_subflow_push_pending() uses atomic allocation, so it will make the fallback scenario describe above less unlikely, I think. Possibly not a big deal, I think - if we fail to allocate an skb we are really on some edge case and we should get an OOM splat.

I'm wondering about avoiding the workqueue usage to send for data fin/data_fin_ack. Looks like that is doable with release_cb too.

Thanks,

Paolo

author
#14 lore raw

[MPTCP] Re: [PATCH v2 6/6] mptcp: use mptcp backlog.

Quoted 4 lines · Paolo Abeni
Quoted 2 lines · Mat Martineau

Is there anything happening here in mptcp_do_backlog() that doesn't work in mptcp_release_cb()?

Quoted 8 lines

Yes, we can't call mptcp_push_pending() in release_cb(), because mptcp_push_pending() requires a non atomic context, while release_cb is invoked under the msk socket spin lock.

We can't use __mptcp_subflow_push_pending() - this latter can be invoked under atomic context - because we don't have an ssk handy. We could look-up a suitable ssk and invoke __mptcp_subflow_push_pending(), but I think is preferrable leveraging mptcp_push_pending().

Uhm... I was too optimistic about __mptcp_subflow_push_pending() usage:
mptcp_release_cb() will need to acquire the ssk socket lock, but it can't because:
- in atomic context
- there will be a AB-BA deadlock (the rx path acquires ssk lock -> msk spinlock in order, release cb held the msk spinlock and will need to acquire the ssk lock)

Quoted 5 lines
Quoted 3 lines

The zeroed-out bl_entry skb seems fragile. The current code in the backlog ends up handling it ok today, but that could easily change. Not that I'm trying to veto the idea, but it does need some more discussion :)

I think we can avoid the backlog usage with the above alternative.

So the above is not feasible, and backlog usage looks required. Why do you think zeroed bl_entry would be fragile ?!? are you wondering about skb->data / skb_shinfo(skb) access? If so, do you think providing a valid data ptr would be the way to go? (looks a little overkill to me, we can add that later if required)

Cheers,

Paolo

#15 lore raw

[MPTCP] Re: [PATCH v2 6/6] mptcp: use mptcp backlog.

Quoted 14 lines · Paolo Abeni
Quoted 4 lines · Paolo Abeni
Quoted 2 lines · Mat Martineau

Is there anything happening here in mptcp_do_backlog() that doesn't work in mptcp_release_cb()?

Quoted 8 lines

Yes, we can't call mptcp_push_pending() in release_cb(), because mptcp_push_pending() requires a non atomic context, while release_cb is invoked under the msk socket spin lock.

We can't use __mptcp_subflow_push_pending() - this latter can be invoked under atomic context - because we don't have an ssk handy. We could look-up a suitable ssk and invoke __mptcp_subflow_push_pending(), but I think is preferrable leveraging mptcp_push_pending().

Quoted 7 lines

Uhm... I was too optimistic about __mptcp_subflow_push_pending() usage:
mptcp_release_cb() will need to acquire the ssk socket lock, but it can't because:
- in atomic context
- there will be a AB-BA deadlock (the rx path acquires ssk lock -> msk spinlock in order, release cb held the msk spinlock and will need to acquire the ssk lock)

Looking at release_sock(), it invokes __release_sock() under the msk socket spin lock right before release_cb(), so they're running in the same context.

__release_sock() unlocks/relocks the spin lock around the call to sk_backlog_rcv(). Seems like release_cb() could do the same to release the spin lock when calling mptcp_push_pending()?

Quoted 11 lines
Quoted 5 lines
Quoted 3 lines

The zeroed-out bl_entry skb seems fragile. The current code in the backlog ends up handling it ok today, but that could easily change. Not that I'm trying to veto the idea, but it does need some more discussion :)

I think we can avoid the backlog usage with the above alternative.

So the above is not feasible, and backlog usage looks required. Why do you think zeroed bl_entry would be fragile ?!? are you wondering about skb->data / skb_shinfo(skb) access? If so, do you think providing a valid data ptr would be the way to go? (looks a little overkill to me, we can add that later if required)

I assume that a skb pointer is pointing to memory that has been initialized by one of the alloc_skb functions (or copied/cloned/etc) - I figure others would assume the same?

The few skb functions called by __sk_add_backlog() and sk_backlog_rcv() happen to behave today with a zeroed-out skb, and problematic change to those functions would probably break our self tests if run with a suitable kernel config. But it would be preferable to avoid that (and the extra size in the msk struct) *if* we can.

--
Mat Martineau
Intel