Skip to contentSkip to conversation

mptcp: just another complete datapath refactor

PATCH Superseded Paolo Abeni 3 participants 15 messages Nov 10, 2020 – Nov 12, 2020

Subsystems (2)

v1 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Paolo Abeni

author
#0 lore raw

cover mptcp: just another complete datapath refactor

This series refactor once again the data path, with the goal of avoding the worker usage for the fast path.

The worker will still be used to:
- cleanup closed subflows
- handle timeout
- retransmissions
- send the DATA_FIN/DATA_FIN ack.
- fallback eof

The latters 2 use case could be fourther removed, if we accept/consider sending DATA_FIN/ack on a single subflow.

The code is still pretty much unstable - survives very basic tests and don't complete yet a single self-test iteration, anyhow sharing early to discuss a few design points.

- separate memory accunting for write memory in patch 1 is a required pre-req for patch 2, but also brings a bit of additional complexity in the following patches. I would be very happy to find any other alternative.

- in patch 3 we may end-up pre-allocating more-than-strictly-required skbs. Doing more accurate estimate would be nice, but I don't know how without adding even more complexity.
Pre-allocating skbs is quite necessary, otherwise under memory pressure we may end-up copying data to kernel (in the msk write queue) but not being able later to allocate the required skbs at push_pending() time and xmit will be stuck.

- I guess patch 6 could cause quite hard reaction upstream, if they will dare look inside. I like it ;)

Paolo Abeni (6):
mptcp: separate accouting for wmem forward alloc mem
mptcp: protect the rx path with the msk socket spinlock
mptcp: allocate TX skbs in msk context.
mptcp: avoid a few atomic ops in the rx path
mptcp: simplify mptcp_nospace
mptcp: use mptcp backlog.

 net/mptcp/mptcp_diag.c |   2 +-
 net/mptcp/options.c    |  47 ++--
 net/mptcp/protocol.c   | 606 +++++++++++++++++++++++++++++------------
 net/mptcp/protocol.h   |  25 +-
 net/mptcp/subflow.c    |  31 ++-
 5 files changed, 496 insertions(+), 215 deletions(-)
Signature · 1 line
2.26.2
author
#1 lore raw

PATCH 1/6 mptcp: separate accouting for wmem forward alloc mem

Account separatelly the memory forward allocated for write operation. When sendmsg() need more memory it looks at wforward_alloc first and if that is exaused, move more space from sk_forward_alloc.

This will simplify the next patch

Signed-off-by Paolo Abeni <pabeni(a)redhat.com>

---
TODO:
- sk_diag_dump() after calling inet_sk_diag_fill() must modify the newly added nlhdr, changing INET_DIAG_MEMINFO addr and specifically .idiag_fmem to sk_forward_alloc + wforward_alloc

---
 net/mptcp/protocol.c | 54 ++++++++++++++++++++++++++++++++++++++++----
 net/mptcp/protocol.h |  1 +
 2 files changed, 50 insertions(+), 5 deletions(-)
net/mptcp/protocol.c +49 -5
index 0956c7db0fd5..7000560c0cd9 100644
@@ -840,9 +840,50 @@ static bool mptcp_frag_can_collapse_to(const struct mptcp_sock *msk,
 		df->data_seq + df->data_len == msk->write_seq;
 }
 
+static bool mptcp_wmem_alloc(struct sock *sk, int size)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+	int amount;
+	int ret;
+
+	if (msk->wforward_alloc >= size)
+		goto account;
+
+	ret = sk_wmem_schedule(sk, size);
+	if (!ret)
+		return false;
+
+	/* try to keep half fwd alloc memory for each direction */
+	amount = max(size, sk->sk_forward_alloc >> 1);
+	sk->sk_forward_alloc -= amount;
+	msk->wforward_alloc += amount;
+
+account:
+	msk->wforward_alloc -= size;
+	return ret;
+}
+
+static void mptcp_wmem_uncharge(struct sock *sk, int size)
+{
+	mptcp_sk(sk)->wforward_alloc += size;
+}
+
+static void mptcp_mem_reclaim_partial(struct sock *sk)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+
+	sk->sk_forward_alloc += msk->wforward_alloc;
+	msk->wforward_alloc = 0;
+	sk_mem_reclaim_partial(sk);
+
+	/* split the remaining fwd allocated memory between rx and tx */
+	msk->wforward_alloc = sk->sk_forward_alloc >> 1;
+	sk->sk_forward_alloc -= msk->wforward_alloc;
+}
+
 static void dfrag_uncharge(struct sock *sk, int len)
 {
-	sk_mem_uncharge(sk, len);
+	mptcp_wmem_uncharge(sk, len);
 	sk_wmem_queued_add(sk, -len);
 }
 
@@ -897,8 +938,8 @@ static void mptcp_clean_una(struct sock *sk)
 	}
 
 out:
-	if (cleaned)
-		sk_mem_reclaim_partial(sk);
+	if (cleaned && tcp_under_memory_pressure(sk))
+		mptcp_mem_reclaim_partial(sk);
 }
 
 static void mptcp_clean_una_wakeup(struct sock *sk)
@@ -1322,11 +1363,12 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 		offset = dfrag->offset + dfrag->data_len;
 		psize = pfrag->size - offset;
 		psize = min_t(size_t, psize, msg_data_left(msg));
-		if (!sk_wmem_schedule(sk, psize + frag_truesize))
+		if (!mptcp_wmem_alloc(sk, psize + frag_truesize))
 			goto wait_for_memory;
 
 		if (copy_page_from_iter(dfrag->page, offset, psize,
 					&msg->msg_iter) != psize) {
+			mptcp_wmem_uncharge(sk, psize + frag_truesize);
 			ret = -EFAULT;
 			goto out;
 		}
@@ -1342,7 +1384,6 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 		 * Note: we charge such data both to sk and ssk
 		 */
 		sk_wmem_queued_add(sk, frag_truesize);
-		sk->sk_forward_alloc -= frag_truesize;
 		if (!dfrag_collapsed) {
 			get_page(dfrag->page);
 			list_add_tail(&dfrag->list, &msk->rtx_queue);
@@ -1965,6 +2006,7 @@ static int __mptcp_init_sock(struct sock *sk)
 	INIT_WORK(&msk->work, mptcp_worker);
 	msk->out_of_order_queue = RB_ROOT;
 	msk->first_pending = NULL;
+	msk->wforward_alloc = 0;
 
 	msk->ack_hint = NULL;
 	msk->first = NULL;
@@ -2155,6 +2197,8 @@ static void __mptcp_destroy_sock(struct sock *sk)
 
 	sk->sk_prot->destroy(sk);
 
+	sk->sk_forward_alloc += msk->wforward_alloc;
+	msk->wforward_alloc = 0;
 	sk_stream_kill_queues(sk);
 	xfrm_sk_free_policy(sk);
 	sk_refcnt_debug_release(sk);
net/mptcp/protocol.h +1 -0
index 07542f7b349a..2a842d228950 100644
@@ -219,6 +219,7 @@ struct mptcp_sock {
 	u64		ack_seq;
 	u64		rcv_wnd_sent;
 	u64		rcv_data_fin_seq;
+	int		wforward_alloc;
 	struct sock	*last_snd;
 	int		snd_burst;
 	int		old_wspace;
Signature · 1 line
2.26.2
author
#3 lore raw

PATCH 2/6 mptcp: protect the rx path with the msk socket spinlock

In reply to Paolo Abeni

Such spinlock is currently used only to protect the 'owned' flag inside the socket lock itself. With this patch we extend it's scope to protect the whole msk receive path and sk_forward_memory.

Given the above we can always move data into the msk receive queue (and OoO queue) from the subflow.

We leverage the previous commit, so that we need to acquire the spinlock in the tx path only when moving fwd memory to wfwd and vice versa.

recvmsg() must now explicitly acquire the socket spinlock when moving skbs out of sk_receive_queue. We use a snd rx queue and splice the first into the latter to reduce the number locks required.

Signed-off-by <pabeni(a)redhat.com>

---
- add __mptcp_mem_reclaim_partial() variant

---
 net/mptcp/protocol.c | 158 +++++++++++++++++++++++++++++--------------
 net/mptcp/protocol.h |   4 ++
 2 files changed, 113 insertions(+), 49 deletions(-)
net/mptcp/protocol.c +109 -49
index 7000560c0cd9..c6370aa58ecb 100644
@@ -270,7 +270,8 @@ static void mptcp_data_queue_ofo(struct mptcp_sock *msk, struct sk_buff *skb)
 
 end:
 	skb_condense(skb);
-	skb_set_owner_r(skb, sk);
+	atomic_add(skb->truesize, &sk->sk_rmem_alloc);
+	sk_mem_charge(sk, skb->truesize);
 }
 
 static bool __mptcp_move_skb(struct mptcp_sock *msk, struct sock *ssk,
@@ -310,7 +311,8 @@ static bool __mptcp_move_skb(struct mptcp_sock *msk, struct sock *ssk,
 		if (tail && mptcp_try_coalesce(sk, tail, skb))
 			return true;
 
-		skb_set_owner_r(skb, sk);
+		atomic_add(skb->truesize, &sk->sk_rmem_alloc);
+		sk_mem_charge(sk, skb->truesize);
 		__skb_queue_tail(&sk->sk_receive_queue, skb);
 		return true;
 	} else if (after64(MPTCP_SKB_CB(skb)->map_seq, msk->ack_seq)) {
@@ -600,7 +602,7 @@ static bool __mptcp_move_skbs_from_subflow(struct mptcp_sock *msk,
 	return done;
 }
 
-static bool mptcp_ofo_queue(struct mptcp_sock *msk)
+static bool __mptcp_ofo_queue(struct mptcp_sock *msk)
 {
 	struct sock *sk = (struct sock *)msk;
 	struct sk_buff *skb, *tail;
@@ -646,34 +648,24 @@ static bool mptcp_ofo_queue(struct mptcp_sock *msk)
 /* In most cases we will be able to lock the mptcp socket.  If its already
  * owned, we need to defer to the work queue to avoid ABBA deadlock.
  */
-static bool move_skbs_to_msk(struct mptcp_sock *msk, struct sock *ssk)
+static void move_skbs_to_msk(struct mptcp_sock *msk, struct sock *ssk)
 {
 	struct sock *sk = (struct sock *)msk;
 	unsigned int moved = 0;
 
-	if (READ_ONCE(sk->sk_lock.owned))
-		return false;
-
-	if (unlikely(!spin_trylock_bh(&sk->sk_lock.slock)))
-		return false;
+	mptcp_data_lock(sk);
 
-	/* must re-check after taking the lock */
-	if (!READ_ONCE(sk->sk_lock.owned)) {
-		__mptcp_move_skbs_from_subflow(msk, ssk, &moved);
-		mptcp_ofo_queue(msk);
-
-		/* If the moves have caught up with the DATA_FIN sequence number
-		 * it's time to ack the DATA_FIN and change socket state, but
-		 * this is not a good place to change state. Let the workqueue
-		 * do it.
-		 */
-		if (mptcp_pending_data_fin(sk, NULL))
-			mptcp_schedule_work(sk);
-	}
+	__mptcp_move_skbs_from_subflow(msk, ssk, &moved);
+	__mptcp_ofo_queue(msk);
 
-	spin_unlock_bh(&sk->sk_lock.slock);
-
-	return moved > 0;
+	/* If the moves have caught up with the DATA_FIN sequence number
+	 * it's time to ack the DATA_FIN and change socket state, but
+	 * this is not a good place to change state. Let the workqueue
+	 * do it.
+	 */
+	if (mptcp_pending_data_fin(sk, NULL))
+		mptcp_schedule_work(sk);
+	mptcp_data_unlock(sk);
 }
 
 void mptcp_data_ready(struct sock *sk, struct sock *ssk)
@@ -849,14 +841,18 @@ static bool mptcp_wmem_alloc(struct sock *sk, int size)
 	if (msk->wforward_alloc >= size)
 		goto account;
 
+	mptcp_data_lock(sk);
 	ret = sk_wmem_schedule(sk, size);
-	if (!ret)
+	if (!ret) {
+		mptcp_data_unlock(sk);
 		return false;
+	}
 
 	/* try to keep half fwd alloc memory for each direction */
 	amount = max(size, sk->sk_forward_alloc >> 1);
 	sk->sk_forward_alloc -= amount;
 	msk->wforward_alloc += amount;
+	mptcp_data_unlock(sk);
 
 account:
 	msk->wforward_alloc -= size;
@@ -868,12 +864,11 @@ static void mptcp_wmem_uncharge(struct sock *sk, int size)
 	mptcp_sk(sk)->wforward_alloc += size;
 }
 
-static void mptcp_mem_reclaim_partial(struct sock *sk)
+static void __mptcp_mem_reclaim_partial(struct sock *sk)
 {
 	struct mptcp_sock *msk = mptcp_sk(sk);
 
 	sk->sk_forward_alloc += msk->wforward_alloc;
-	msk->wforward_alloc = 0;
 	sk_mem_reclaim_partial(sk);
 
 	/* split the remaining fwd allocated memory between rx and tx */
@@ -881,6 +876,13 @@ static void mptcp_mem_reclaim_partial(struct sock *sk)
 	sk->sk_forward_alloc -= msk->wforward_alloc;
 }
 
+static void mptcp_mem_reclaim_partial(struct sock *sk)
+{
+	mptcp_data_lock(sk);
+	__mptcp_mem_reclaim_partial(sk);
+	mptcp_data_unlock(sk);
+}
+
 static void dfrag_uncharge(struct sock *sk, int len)
 {
 	mptcp_wmem_uncharge(sk, len);
@@ -1432,13 +1434,12 @@ static void mptcp_wait_data(struct sock *sk, long *timeo)
 
 static int __mptcp_recvmsg_mskq(struct mptcp_sock *msk,
 				struct msghdr *msg,
-				size_t len)
+				size_t len, int *released)
 {
-	struct sock *sk = (struct sock *)msk;
 	struct sk_buff *skb;
 	int copied = 0;
 
-	while ((skb = skb_peek(&sk->sk_receive_queue)) != NULL) {
+	while ((skb = skb_peek(&msk->receive_queue)) != NULL) {
 		u32 offset = MPTCP_SKB_CB(skb)->offset;
 		u32 data_len = skb->len - offset;
 		u32 count = min_t(size_t, len - copied, data_len);
@@ -1458,7 +1459,8 @@ static int __mptcp_recvmsg_mskq(struct mptcp_sock *msk,
 			break;
 		}
 
-		__skb_unlink(skb, &sk->sk_receive_queue);
+		*released += skb->truesize;
+		__skb_unlink(skb, &msk->receive_queue);
 		__kfree_skb(skb);
 
 		if (copied >= len)
@@ -1566,25 +1568,49 @@ static void mptcp_rcv_space_adjust(struct mptcp_sock *msk, int copied)
 	msk->rcvq_space.time = mstamp;
 }
 
-static bool __mptcp_move_skbs(struct mptcp_sock *msk, unsigned int rcv)
+static void __mptcp_update_rmem(struct sock *sk, int *released)
+{
+	sk_mem_uncharge(sk, *released);
+	atomic_sub(*released, &sk->sk_rmem_alloc);
+	*released = 0;
+}
+
+static void mptcp_update_rmem(struct sock *sk, int *released)
 {
+	if (!released)
+		return;
+
+	mptcp_data_lock(sk);
+	__mptcp_update_rmem(sk, released);
+	mptcp_data_unlock(sk);
+}
+
+static bool __mptcp_move_skbs(struct mptcp_sock *msk, unsigned int rcv, int *released)
+{
+	struct sock *sk = (struct sock *)msk;
 	unsigned int moved = 0;
-	bool done;
+	bool ret, done;
 
 	/* avoid looping forever below on racing close */
-	if (((struct sock *)msk)->sk_state == TCP_CLOSE)
-		return false;
+	if (sk->sk_state == TCP_CLOSE)
+		goto chk_splice;
 
 	__mptcp_flush_join_list(msk);
 	do {
 		struct sock *ssk = mptcp_subflow_recv_lookup(msk);
 		bool slowpath;
 
-		if (!ssk)
+		/* we can have data pending in the subflows only if the msk
+		 * receive buffer was full at subflow_data_ready() time,
+		 * it is an likely slow path
+		 */
+		if (likely(!ssk))
 			break;
 
 		slowpath = lock_sock_fast(ssk);
+		mptcp_data_lock(sk);
 		done = __mptcp_move_skbs_from_subflow(msk, ssk, &moved);
+		mptcp_data_unlock(sk);
 		if (moved && rcv) {
 			WRITE_ONCE(msk->rmem_pending, min(rcv, moved));
 			tcp_cleanup_rbuf(ssk, 1);
@@ -1593,17 +1619,28 @@ static bool __mptcp_move_skbs(struct mptcp_sock *msk, unsigned int rcv)
 		unlock_sock_fast(ssk, slowpath);
 	} while (!done);
 
-	if (mptcp_ofo_queue(msk) || moved > 0) {
-		mptcp_check_data_fin((struct sock *)msk);
-		return true;
+chk_splice:
+	/* acquire the data lock only if some input data is pending */
+	ret = moved > 0;
+	if (!RB_EMPTY_ROOT(&msk->out_of_order_queue) ||
+	    !skb_queue_empty_lockless(&sk->sk_receive_queue)) {
+		mptcp_data_lock(sk);
+		__mptcp_update_rmem(sk, released);
+		ret |= __mptcp_ofo_queue(msk);
+		skb_queue_splice_tail_init(&sk->sk_receive_queue,
+					   &msk->receive_queue);
+		mptcp_data_unlock(sk);
 	}
-	return false;
+	if (ret)
+		mptcp_check_data_fin((struct sock *)msk);
+	return !skb_queue_empty(&msk->receive_queue);
 }
 
 static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 			 int nonblock, int flags, int *addr_len)
 {
 	struct mptcp_sock *msk = mptcp_sk(sk);
+	int released = 0;
 	int copied = 0;
 	int target;
 	long timeo;
@@ -1621,7 +1658,8 @@ static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 	for (;;) {
 		int bytes_read, old_space;
 
-		bytes_read = __mptcp_recvmsg_mskq(msk, msg, len - copied);
+		bytes_read = __mptcp_recvmsg_mskq(msk, msg, len - copied,
+						  &released);
 		if (unlikely(bytes_read < 0)) {
 			if (!copied)
 				copied = bytes_read;
@@ -1630,8 +1668,8 @@ static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 
 		copied += bytes_read;
 
-		if (skb_queue_empty(&sk->sk_receive_queue) &&
-		    __mptcp_move_skbs(msk, len - copied))
+		if (skb_queue_empty(&msk->receive_queue) &&
+		    __mptcp_move_skbs(msk, len - copied, &released))
 			continue;
 
 		/* be sure to advertize window change */
@@ -1661,8 +1699,15 @@ static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 			if (test_and_clear_bit(MPTCP_WORK_EOF, &msk->flags))
 				mptcp_check_for_eof(msk);
 
-			if (sk->sk_shutdown & RCV_SHUTDOWN)
+			if (sk->sk_shutdown & RCV_SHUTDOWN) {
+				/* race breaker: the shutdown could be after the
+				 * previous receive queue check
+				 */
+				if (__mptcp_move_skbs(msk, len - copied,
+						      &released))
+					continue;
 				break;
+			}
 
 			if (sk->sk_state == TCP_CLOSE) {
 				copied = -ENOTCONN;
@@ -1681,26 +1726,29 @@ static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 		}
 
 		pr_debug("block timeout %ld", timeo);
+		mptcp_update_rmem(sk, &released);
 		mptcp_wait_data(sk, &timeo);
 	}
 
-	if (skb_queue_empty(&sk->sk_receive_queue)) {
+	if (skb_queue_empty_lockless(&sk->sk_receive_queue) &&
+	    skb_queue_empty(&msk->receive_queue)) {
 		/* entire backlog drained, clear DATA_READY. */
 		clear_bit(MPTCP_DATA_READY, &msk->flags);
 
 		/* .. race-breaker: ssk might have gotten new data
 		 * after last __mptcp_move_skbs() returned false.
 		 */
-		if (unlikely(__mptcp_move_skbs(msk, 0)))
+		if (unlikely(__mptcp_move_skbs(msk, 0, &released)))
 			set_bit(MPTCP_DATA_READY, &msk->flags);
 	} else if (unlikely(!test_bit(MPTCP_DATA_READY, &msk->flags))) {
 		/* data to read but mptcp_wait_data() cleared DATA_READY */
 		set_bit(MPTCP_DATA_READY, &msk->flags);
 	}
 out_err:
+	mptcp_update_rmem(sk, &released);
 	pr_debug("msk=%p data_ready=%d rx queue empty=%d copied=%d",
 		 msk, test_bit(MPTCP_DATA_READY, &msk->flags),
-		 skb_queue_empty(&sk->sk_receive_queue), copied);
+		 skb_queue_empty_lockless(&sk->sk_receive_queue), copied);
 	mptcp_rcv_space_adjust(msk, copied);
 
 	release_sock(sk);
@@ -2004,6 +2052,7 @@ static int __mptcp_init_sock(struct sock *sk)
 	INIT_LIST_HEAD(&msk->join_list);
 	INIT_LIST_HEAD(&msk->rtx_queue);
 	INIT_WORK(&msk->work, mptcp_worker);
+	__skb_queue_head_init(&msk->receive_queue);
 	msk->out_of_order_queue = RB_ROOT;
 	msk->first_pending = NULL;
 	msk->wforward_alloc = 0;
@@ -2200,6 +2249,7 @@ static void __mptcp_destroy_sock(struct sock *sk)
 	sk->sk_forward_alloc += msk->wforward_alloc;
 	msk->wforward_alloc = 0;
 	sk_stream_kill_queues(sk);
+
 	xfrm_sk_free_policy(sk);
 	sk_refcnt_debug_release(sk);
 	sock_put(sk);
@@ -2422,7 +2472,17 @@ static struct sock *mptcp_accept(struct sock *sk, int flags, int *err,
 
 void mptcp_destroy_common(struct mptcp_sock *msk)
 {
-	skb_rbtree_purge(&msk->out_of_order_queue);
+	struct sock *sk = (struct sock *)msk;
+	struct sk_buff *skb;
+	int queued = 0;
+
+	skb_queue_splice_tail_init(&sk->sk_receive_queue, &msk->receive_queue);
+	while ((skb = __skb_dequeue(&msk->receive_queue)) != NULL) {
+		queued += skb->truesize;
+		kfree_skb(skb);
+	}
+	queued += skb_rbtree_purge(&msk->out_of_order_queue);
+	__mptcp_update_rmem(sk, &queued);
 	mptcp_token_destroy(msk);
 	mptcp_pm_free_anno_list(msk);
 }
net/mptcp/protocol.h +4 -0
index 2a842d228950..a7107a70badd 100644
@@ -239,6 +239,7 @@ struct mptcp_sock {
 	struct work_struct work;
 	struct sk_buff  *ooo_last_skb;
 	struct rb_root  out_of_order_queue;
+	struct sk_buff_head receive_queue;
 	struct list_head conn_list;
 	struct list_head rtx_queue;
 	struct mptcp_data_frag *first_pending;
@@ -255,6 +256,9 @@ struct mptcp_sock {
 	} rcvq_space;
 };
 
+#define mptcp_data_lock(sk) spin_lock_bh(&sk->sk_lock.slock)
+#define mptcp_data_unlock(sk) spin_unlock_bh(&sk->sk_lock.slock)
+
 #define mptcp_for_each_subflow(__msk, __subflow)			\
 	list_for_each_entry(__subflow, &((__msk)->conn_list), node)
 
Signature · 1 line
2.26.2
author
#6 lore raw

PATCH 3/6 mptcp: allocate TX skbs in msk context.

In reply to Paolo Abeni

Move the TX skbs allocation in mptcp_sendmsg() scope, and tentativelly allocate a skbs based on the goal size on the last used subflow.

Use the ssk tx skb cache to prevent the subflow allocation.

This allows removing the msk skb extension cache and will make possible the later patch.

Signed-off-by Paolo Abeni <pabeni(a)redhat.com>

---
RFC v3 -> v1:
- fixed some allocation bugs

---
 net/mptcp/protocol.c | 193 +++++++++++++++++++++++++++++++++++--------
 net/mptcp/protocol.h |   4 +-
 2 files changed, 160 insertions(+), 37 deletions(-)
net/mptcp/protocol.c +157 -36
index c6370aa58ecb..7fa4c7ff911f 100644
@@ -784,16 +784,6 @@ static void mptcp_check_for_eof(struct mptcp_sock *msk)
 		inet_sk_state_store(sk, TCP_CLOSE);
 }
 
-static bool mptcp_ext_cache_refill(struct mptcp_sock *msk)
-{
-	const struct sock *sk = (const struct sock *)msk;
-
-	if (!msk->cached_ext)
-		msk->cached_ext = __skb_ext_alloc(sk->sk_allocation);
-
-	return !!msk->cached_ext;
-}
-
 static struct sock *mptcp_subflow_recv_lookup(const struct mptcp_sock *msk)
 {
 	struct mptcp_subflow_context *subflow;
@@ -957,19 +947,12 @@ static void mptcp_clean_una_wakeup(struct sock *sk)
 	}
 }
 
-/* ensure we get enough memory for the frag hdr, beyond some minimal amount of
- * data
- */
-static bool mptcp_page_frag_refill(struct sock *sk, struct page_frag *pfrag)
+static void mptcp_enter_memory_pressure(struct sock *sk)
 {
 	struct mptcp_subflow_context *subflow;
 	struct mptcp_sock *msk = mptcp_sk(sk);
 	bool first = true;
 
-	if (likely(skb_page_frag_refill(32U + sizeof(struct mptcp_data_frag),
-					pfrag, sk->sk_allocation)))
-		return true;
-
 	sk_stream_moderate_sndbuf(sk);
 	mptcp_for_each_subflow(msk, subflow) {
 		struct sock *ssk = mptcp_subflow_tcp_sock(subflow);
@@ -979,6 +962,18 @@ static bool mptcp_page_frag_refill(struct sock *sk, struct page_frag *pfrag)
 		sk_stream_moderate_sndbuf(ssk);
 		first = false;
 	}
+}
+
+/* ensure we get enough memory for the frag hdr, beyond some minimal amount of
+ * data
+ */
+static bool mptcp_page_frag_refill(struct sock *sk, struct page_frag *pfrag)
+{
+	if (likely(skb_page_frag_refill(32U + sizeof(struct mptcp_data_frag),
+					pfrag, sk->sk_allocation)))
+		return true;
+
+	mptcp_enter_memory_pressure(sk);
 	return false;
 }
 
@@ -1025,6 +1020,116 @@ static int mptcp_check_allowed_size(struct mptcp_sock *msk, u64 data_seq,
 	return avail_size;
 }
 
+static bool __mptcp_add_ext(struct sk_buff *skb, gfp_t gfp)
+{
+	struct skb_ext *mpext = __skb_ext_alloc(gfp);
+
+	if (!mpext)
+		return false;
+	__skb_ext_set(skb, SKB_EXT_MPTCP, mpext);
+	return true;
+}
+
+static struct sk_buff *__mptcp_do_alloc_tx_skb(struct sock *sk)
+{
+	struct sk_buff *skb;
+
+	skb = alloc_skb_fclone(MAX_TCP_HEADER, sk->sk_allocation);
+	if (likely(skb)) {
+		if (likely(__mptcp_add_ext(skb, sk->sk_allocation))) {
+			skb_reserve(skb, MAX_TCP_HEADER);
+			skb->reserved_tailroom = skb->end - skb->tail;
+			return skb;
+		}
+		__kfree_skb(skb);
+	} else {
+		mptcp_enter_memory_pressure(sk);
+	}
+	return NULL;
+}
+
+static bool mptcp_tx_cache_refill(struct sock *sk, int size)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+	struct sk_buff *skb;
+	int space_needed;
+
+	if (unlikely(tcp_under_memory_pressure(sk)))
+		mptcp_mem_reclaim_partial(sk);
+
+	space_needed = msk->tx_pending_data + size -
+		       msk->skb_tx_cache.qlen * msk->size_goal_cache;
+	while (space_needed > 0) {
+		skb = __mptcp_do_alloc_tx_skb(sk);
+		if (!skb)
+			return false;
+
+		if (!mptcp_wmem_alloc(sk, skb->truesize)) {
+			__kfree_skb(skb);
+			return false;
+		}
+
+		__skb_queue_tail(&msk->skb_tx_cache, skb);
+		space_needed -= msk->size_goal_cache;
+	}
+	return true;
+}
+
+static bool __mptcp_alloc_tx_skb(struct sock *sk, struct sock *ssk)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+	struct sk_buff *skb;
+
+	if (ssk->sk_tx_skb_cache) {
+		skb = ssk->sk_tx_skb_cache;
+		if (unlikely(!skb_ext_find(skb, SKB_EXT_MPTCP) &&
+			     !__mptcp_add_ext(skb, sk->sk_allocation)))
+			return false;
+		return true;
+	}
+
+	skb = skb_peek(&msk->skb_tx_cache);
+	if (skb) {
+		if (likely(sk_wmem_schedule(ssk, skb->truesize))) {
+			skb = __skb_dequeue(&msk->skb_tx_cache);
+			if (WARN_ON_ONCE(!skb))
+				return false;
+
+			mptcp_wmem_uncharge(sk, skb->truesize);
+			ssk->sk_tx_skb_cache = skb;
+			return true;
+		}
+
+		/* over memory limit, no point to try to allocate a new skb */
+		return false;
+	}
+
+	skb = __mptcp_do_alloc_tx_skb(sk);
+	if (!skb)
+		return false;
+
+	if (likely(sk_wmem_schedule(ssk, skb->truesize))) {
+		ssk->sk_tx_skb_cache = skb;
+		return true;
+	}
+	kfree_skb(skb);
+	return false;
+}
+
+static bool mptcp_must_reclaim_memory(struct sock *sk, struct sock *ssk)
+{
+	return !ssk->sk_tx_skb_cache &&
+	       !skb_peek(&mptcp_sk(sk)->skb_tx_cache) &&
+	       tcp_under_memory_pressure(sk);
+}
+
+static bool mptcp_alloc_tx_skb(struct sock *sk, struct sock *ssk)
+{
+	if (unlikely(mptcp_must_reclaim_memory(sk, ssk)))
+		mptcp_mem_reclaim_partial(sk);
+	return __mptcp_alloc_tx_skb(sk, ssk);
+}
+
 static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,
 			      struct mptcp_data_frag *dfrag,
 			      struct mptcp_sendmsg_info *info)
@@ -1036,7 +1141,7 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,
 	struct sk_buff *skb, *tail;
 	bool can_collapse = false;
 	int avail_size;
-	size_t ret;
+	size_t ret = 0;
 
 	pr_debug("msk=%p ssk=%p sending dfrag at seq=%lld len=%d already sent=%d",
 		 msk, ssk, dfrag->data_seq, dfrag->data_len, info->sent);
@@ -1044,6 +1149,7 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,
 	/* compute send limit */
 	info->mss_now = tcp_send_mss(ssk, &info->size_goal, info->flags);
 	avail_size = info->size_goal;
+	msk->size_goal_cache = info->size_goal;
 	skb = tcp_write_queue_tail(ssk);
 	if (skb) {
 		/* Limit the write to the size available in the
@@ -1092,8 +1198,11 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,
 		goto out;
 	}
 
-	mpext = __skb_ext_set(tail, SKB_EXT_MPTCP, msk->cached_ext);
-	msk->cached_ext = NULL;
+	mpext = skb_ext_find(tail, SKB_EXT_MPTCP);
+	if (WARN_ON_ONCE(!mpext)) {
+		/* should never reach here, stream corrupted */
+		return -EINVAL;
+	}
 
 	memset(mpext, 0, sizeof(*mpext));
 	mpext->data_seq = data_seq;
@@ -1113,6 +1222,7 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,
 		tcp_push_pending_frames(ssk);
 	}
 out:
+	msk->tx_pending_data -= ret;
 	mptcp_subflow_ctx(ssk)->rel_write_seq += ret;
 	return ret;
 }
@@ -1166,9 +1276,6 @@ static struct sock *mptcp_subflow_get_send(struct mptcp_sock *msk,
 	sock_owned_by_me((struct sock *)msk);
 
 	*sndbuf = 0;
-	if (!mptcp_ext_cache_refill(msk))
-		return NULL;
-
 	if (__mptcp_check_fallback(msk)) {
 		if (!msk->first)
 			return NULL;
@@ -1277,6 +1384,15 @@ static void mptcp_push_pending(struct sock *sk, unsigned int flags)
 			if (ssk != prev_ssk || !prev_ssk)
 				lock_sock(ssk);
 
+			/* keep it simple and always provide a new skb for the
+			 * subflow, even if we will not use it when collapsing
+			 * on the pending one
+			 */
+			if (!mptcp_alloc_tx_skb(sk, ssk)) {
+				mptcp_push_release(sk, ssk, &info);
+				goto out;
+			}
+
 			ret = mptcp_sendmsg_frag(sk, ssk, dfrag, &info);
 			if (ret <= 0) {
 				mptcp_push_release(sk, ssk, &info);
@@ -1365,6 +1481,9 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 		offset = dfrag->offset + dfrag->data_len;
 		psize = pfrag->size - offset;
 		psize = min_t(size_t, psize, msg_data_left(msg));
+		if (!mptcp_tx_cache_refill(sk, psize))
+			goto wait_for_memory;
+
 		if (!mptcp_wmem_alloc(sk, psize + frag_truesize))
 			goto wait_for_memory;
 
@@ -1396,8 +1515,6 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 			 dfrag->data_seq, dfrag->data_len, dfrag->already_sent,
 			 !dfrag_collapsed);
 
-		if (!mptcp_ext_cache_refill(msk))
-			goto wait_for_memory;
 		continue;
 
 wait_for_memory:
@@ -1409,8 +1526,10 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 			goto out;
 	}
 
-	if (copied)
+	if (copied) {
+		msk->tx_pending_data += copied;
 		mptcp_push_pending(sk, msg->msg_flags);
+	}
 
 out:
 	release_sock(sk);
@@ -2001,9 +2120,6 @@ static void mptcp_worker(struct work_struct *work)
 	if (!dfrag)
 		goto unlock;
 
-	if (!mptcp_ext_cache_refill(msk))
-		goto reset_unlock;
-
 	ssk = mptcp_subflow_get_retrans(msk);
 	if (!ssk)
 		goto reset_unlock;
@@ -2014,6 +2130,9 @@ static void mptcp_worker(struct work_struct *work)
 	info.sent = 0;
 	info.limit = dfrag->already_sent;
 	while (info.sent < dfrag->already_sent) {
+		if (!mptcp_alloc_tx_skb(sk, ssk))
+			break;
+
 		ret = mptcp_sendmsg_frag(sk, ssk, dfrag, &info);
 		if (ret <= 0)
 			break;
@@ -2021,9 +2140,6 @@ static void mptcp_worker(struct work_struct *work)
 		MPTCP_INC_STATS(sock_net(sk), MPTCP_MIB_RETRANSSEGS);
 		copied += ret;
 		info.sent += ret;
-
-		if (!mptcp_ext_cache_refill(msk))
-			break;
 	}
 	if (copied)
 		tcp_push(ssk, 0, info.mss_now, tcp_sk(ssk)->nonagle,
@@ -2053,9 +2169,12 @@ static int __mptcp_init_sock(struct sock *sk)
 	INIT_LIST_HEAD(&msk->rtx_queue);
 	INIT_WORK(&msk->work, mptcp_worker);
 	__skb_queue_head_init(&msk->receive_queue);
+	__skb_queue_head_init(&msk->skb_tx_cache);
 	msk->out_of_order_queue = RB_ROOT;
 	msk->first_pending = NULL;
 	msk->wforward_alloc = 0;
+	msk->tx_pending_data = 0;
+	msk->size_goal_cache = TCP_BASE_MSS;
 
 	msk->ack_hint = NULL;
 	msk->first = NULL;
@@ -2103,12 +2222,17 @@ static void __mptcp_clear_xmit(struct sock *sk)
 {
 	struct mptcp_sock *msk = mptcp_sk(sk);
 	struct mptcp_data_frag *dtmp, *dfrag;
+	struct sk_buff *skb;
 
 	sk_stop_timer(sk, &msk->sk.icsk_retransmit_timer);
 
 	WRITE_ONCE(msk->first_pending, NULL);
 	list_for_each_entry_safe(dfrag, dtmp, &msk->rtx_queue, list)
 		dfrag_clear(sk, dfrag);
+	while ((skb = __skb_dequeue(&msk->skb_tx_cache)) != NULL) {
+		mptcp_wmem_uncharge(sk, skb->truesize);
+		kfree_skb(skb);
+	}
 }
 
 static void mptcp_cancel_work(struct sock *sk)
@@ -2491,9 +2615,6 @@ static void mptcp_destroy(struct sock *sk)
 {
 	struct mptcp_sock *msk = mptcp_sk(sk);
 
-	if (msk->cached_ext)
-		__skb_ext_put(msk->cached_ext);
-
 	mptcp_destroy_common(msk);
 	sk_sockets_allocated_dec(sk);
 }
net/mptcp/protocol.h +3 -1
index a7107a70badd..ff8489161261 100644
@@ -240,11 +240,13 @@ struct mptcp_sock {
 	struct sk_buff  *ooo_last_skb;
 	struct rb_root  out_of_order_queue;
 	struct sk_buff_head receive_queue;
+	struct sk_buff_head skb_tx_cache;	/* this is wmem accounted */
+	int		tx_pending_data;
+	int		size_goal_cache;
 	struct list_head conn_list;
 	struct list_head rtx_queue;
 	struct mptcp_data_frag *first_pending;
 	struct list_head join_list;
-	struct skb_ext	*cached_ext;	/* for the next sendmsg */
 	struct socket	*subflow; /* outgoing connect/listener/!mp_capable */
 	struct sock	*first;
 	struct mptcp_pm_data	pm;
Signature · 1 line
2.26.2
author
#7 lore raw

PATCH 4/6 mptcp: avoid a few atomic ops in the rx path

In reply to Paolo Abeni

Extending the data_lock scope in mptcp_incoming_option we can use that to protect both snd_una and wnd_end. In the typical case we will have a single atomic op instead of 2

Signed-off-by Paolo Abeni <pabeni(a)redhat.com>
---
 net/mptcp/mptcp_diag.c |  2 +-
 net/mptcp/options.c    | 33 +++++++++++++--------------------
 net/mptcp/protocol.c   | 33 +++++++++++++++------------------
 net/mptcp/protocol.h   |  8 ++++----
 4 files changed, 33 insertions(+), 43 deletions(-)
net/mptcp/mptcp_diag.c +1 -1
index 5f390a97f556..b70ae4ba3000 100644
@@ -140,7 +140,7 @@ static void mptcp_diag_get_info(struct sock *sk, struct inet_diag_msg *r,
 	info->mptcpi_flags = flags;
 	info->mptcpi_token = READ_ONCE(msk->token);
 	info->mptcpi_write_seq = READ_ONCE(msk->write_seq);
-	info->mptcpi_snd_una = atomic64_read(&msk->snd_una);
+	info->mptcpi_snd_una = READ_ONCE(msk->snd_una);
 	info->mptcpi_rcv_nxt = READ_ONCE(msk->ack_seq);
 	unlock_sock_fast(sk, slow);
 }
net/mptcp/options.c +13 -20
index 8a59b3e44599..3986454a0340 100644
@@ -833,15 +833,17 @@ static void ack_update_msk(struct mptcp_sock *msk,
 			   const struct sock *ssk,
 			   struct mptcp_options_received *mp_opt)
 {
-	u64 new_snd_una, snd_una, old_snd_una = atomic64_read(&msk->snd_una);
-	u64 new_wnd_end, wnd_end, old_wnd_end = atomic64_read(&msk->wnd_end);
-	u64 snd_nxt = READ_ONCE(msk->snd_nxt);
+	u64 new_wnd_end, new_snd_una, snd_nxt = READ_ONCE(msk->snd_nxt);
 	struct sock *sk = (struct sock *)msk;
+	u64 old_snd_una;
+
+	mptcp_data_lock(sk);
 
 	/* avoid ack expansion on update conflict, to reduce the risk of
 	 * wrongly expanding to a future ack sequence number, which is way
 	 * more dangerous than missing an ack
 	 */
+	old_snd_una = msk->snd_una;
 	new_snd_una = expand_ack(old_snd_una, mp_opt->data_ack, mp_opt->ack64);
 
 	/* ACK for data not even sent yet? Ignore. */
@@ -850,26 +852,17 @@ static void ack_update_msk(struct mptcp_sock *msk,
 
 	new_wnd_end = new_snd_una + tcp_sk(ssk)->snd_wnd;
 
-	while (after64(new_wnd_end, old_wnd_end)) {
-		wnd_end = old_wnd_end;
-		old_wnd_end = atomic64_cmpxchg(&msk->wnd_end, wnd_end,
-					       new_wnd_end);
-		if (old_wnd_end == wnd_end) {
-			if (mptcp_send_head(sk))
-				mptcp_schedule_work(sk);
-			break;
-		}
+	if (after64(new_wnd_end, msk->wnd_end)) {
+		msk->wnd_end = new_wnd_end;
+		if (mptcp_send_head(sk))
+			mptcp_schedule_work(sk);
 	}
 
-	while (after64(new_snd_una, old_snd_una)) {
-		snd_una = old_snd_una;
-		old_snd_una = atomic64_cmpxchg(&msk->snd_una, snd_una,
-					       new_snd_una);
-		if (old_snd_una == snd_una) {
-			mptcp_data_acked(sk);
-			break;
-		}
+	if (after64(new_snd_una, old_snd_una)) {
+		msk->snd_una = new_snd_una;
+		__mptcp_data_acked(sk);
 	}
+	mptcp_data_unlock(sk);
 }
 
 bool mptcp_update_rcv_data_fin(struct mptcp_sock *msk, u64 data_fin_seq, bool use_64bit)
net/mptcp/protocol.c +15 -18
index 7fa4c7ff911f..ccf0ceaf5a1c 100644
@@ -60,7 +60,7 @@ static struct socket *__mptcp_nmpc_socket(const struct mptcp_sock *msk)
 /* Returns end sequence number of the receiver's advertised window */
 static u64 mptcp_wnd_end(const struct mptcp_sock *msk)
 {
-	return atomic64_read(&msk->wnd_end);
+	return READ_ONCE(msk->wnd_end);
 }
 
 static bool mptcp_is_tcpsk(struct sock *sk)
@@ -347,7 +347,7 @@ static void mptcp_check_data_fin_ack(struct sock *sk)
 	/* Look for an acknowledged DATA_FIN */
 	if (((1 << sk->sk_state) &
 	     (TCPF_FIN_WAIT1 | TCPF_CLOSING | TCPF_LAST_ACK)) &&
-	    msk->write_seq == atomic64_read(&msk->snd_una)) {
+	    msk->write_seq == READ_ONCE(msk->snd_una)) {
 		mptcp_stop_timer(sk);
 
 		WRITE_ONCE(msk->snd_data_fin_enable, 0);
@@ -743,7 +743,7 @@ bool mptcp_schedule_work(struct sock *sk)
 	return false;
 }
 
-void mptcp_data_acked(struct sock *sk)
+void __mptcp_data_acked(struct sock *sk)
 {
 	mptcp_reset_timer(sk);
 
@@ -899,10 +899,9 @@ static void mptcp_clean_una(struct sock *sk)
 	 * plain TCP
 	 */
 	if (__mptcp_check_fallback(msk))
-		atomic64_set(&msk->snd_una, msk->snd_nxt);
-
-	snd_una = atomic64_read(&msk->snd_una);
+		msk->snd_una = READ_ONCE(msk->snd_nxt);
 
+	snd_una = msk->snd_una;
 	list_for_each_entry_safe(dfrag, dtmp, &msk->rtx_queue, list) {
 		if (after64(dfrag->data_seq + dfrag->data_len, snd_una))
 			break;
@@ -1170,10 +1169,12 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,
 	/* Zero window and all data acked? Probe. */
 	avail_size = mptcp_check_allowed_size(msk, data_seq, avail_size);
 	if (avail_size == 0) {
-		if (skb || atomic64_read(&msk->snd_una) != msk->snd_nxt)
+		u64 snd_una = READ_ONCE(msk->snd_una);
+
+		if (skb || snd_una != msk->snd_nxt)
 			return 0;
 		zero_window_probe = true;
-		data_seq = atomic64_read(&msk->snd_una) - 1;
+		data_seq = snd_una - 1;
 		avail_size = 1;
 	}
 
@@ -1878,12 +1879,8 @@ static void mptcp_retransmit_handler(struct sock *sk)
 {
 	struct mptcp_sock *msk = mptcp_sk(sk);
 
-	if (atomic64_read(&msk->snd_una) == READ_ONCE(msk->snd_nxt)) {
-		mptcp_stop_timer(sk);
-	} else {
-		set_bit(MPTCP_WORK_RTX, &msk->flags);
-		mptcp_schedule_work(sk);
-	}
+	set_bit(MPTCP_WORK_RTX, &msk->flags);
+	mptcp_schedule_work(sk);
 }
 
 static void mptcp_retransmit_timer(struct timer_list *t)
@@ -2506,8 +2503,8 @@ struct sock *mptcp_sk_clone(const struct sock *sk,
 
 	msk->write_seq = subflow_req->idsn + 1;
 	msk->snd_nxt = msk->write_seq;
-	atomic64_set(&msk->snd_una, msk->write_seq);
-	atomic64_set(&msk->wnd_end, msk->snd_nxt + req->rsk_rcv_wnd);
+	msk->snd_una = msk->write_seq;
+	msk->wnd_end = msk->snd_nxt + req->rsk_rcv_wnd;
 
 	if (mp_opt->mp_capable) {
 		msk->can_ack = true;
@@ -2545,7 +2542,7 @@ void mptcp_rcv_space_init(struct mptcp_sock *msk, const struct sock *ssk)
 	if (msk->rcvq_space.space == 0)
 		msk->rcvq_space.space = TCP_INIT_CWND * TCP_MSS_DEFAULT;
 
-	atomic64_set(&msk->wnd_end, msk->snd_nxt + tcp_sk(ssk)->snd_wnd);
+	WRITE_ONCE(msk->wnd_end, msk->snd_nxt + tcp_sk(ssk)->snd_wnd);
 }
 
 static struct sock *mptcp_accept(struct sock *sk, int flags, int *err,
@@ -2808,7 +2805,7 @@ void mptcp_finish_connect(struct sock *ssk)
 	WRITE_ONCE(msk->ack_seq, ack_seq);
 	WRITE_ONCE(msk->rcv_wnd_sent, ack_seq);
 	WRITE_ONCE(msk->can_ack, 1);
-	atomic64_set(&msk->snd_una, msk->write_seq);
+	WRITE_ONCE(msk->snd_una, msk->write_seq);
 
 	mptcp_pm_new_connection(msk, 0);
 
net/mptcp/protocol.h +4 -4
index ff8489161261..0047295b766f 100644
@@ -223,8 +223,8 @@ struct mptcp_sock {
 	struct sock	*last_snd;
 	int		snd_burst;
 	int		old_wspace;
-	atomic64_t	snd_una;
-	atomic64_t	wnd_end;
+	u64		snd_una;
+	u64		wnd_end;
 	unsigned long	timer_ival;
 	u32		token;
 	int		rmem_pending;
@@ -308,7 +308,7 @@ static inline struct mptcp_data_frag *mptcp_rtx_tail(const struct sock *sk)
 {
 	struct mptcp_sock *msk = mptcp_sk(sk);
 
-	if (!before64(msk->snd_nxt, atomic64_read(&msk->snd_una)))
+	if (!before64(msk->snd_nxt, READ_ONCE(msk->snd_una)))
 		return NULL;
 
 	return list_last_entry(&msk->rtx_queue, struct mptcp_data_frag, list);
@@ -482,7 +482,7 @@ void mptcp_rcv_space_init(struct mptcp_sock *msk, const struct sock *ssk);
 void mptcp_data_ready(struct sock *sk, struct sock *ssk);
 bool mptcp_finish_join(struct sock *sk);
 bool mptcp_schedule_work(struct sock *sk);
-void mptcp_data_acked(struct sock *sk);
+void __mptcp_data_acked(struct sock *sk);
 void mptcp_subflow_eof(struct sock *sk);
 bool mptcp_update_rcv_data_fin(struct mptcp_sock *msk, u64 data_fin_seq, bool use_64bit);
 void __mptcp_flush_join_list(struct mptcp_sock *msk);
Signature · 1 line
2.26.2
author
#8 lore raw

PATCH 5/6 mptcp: simplify mptcp_nospace

In reply to Paolo Abeni

Currently mptcp_space has to traverse the whole conn_list setting the relevant bit on each subflow, because the client ones will have a different sk_socket.

We can actually leave the NOSPACE always set on all client subflow and make nospace simpler.

Signed-off-by Paolo Abeni <pabeni(a)redhat.com>
---
 net/mptcp/protocol.c | 15 +--------------
 net/mptcp/subflow.c  | 31 ++++++++++++++++++++++---------
 2 files changed, 23 insertions(+), 23 deletions(-)
net/mptcp/protocol.c +1 -14
index ccf0ceaf5a1c..f016b5c11154 100644
@@ -1230,23 +1230,9 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,
 
 static void mptcp_nospace(struct mptcp_sock *msk)
 {
-	struct mptcp_subflow_context *subflow;
-
 	set_bit(MPTCP_NOSPACE, &msk->flags);
 	smp_mb__after_atomic(); /* msk->flags is changed by write_space cb */
 
-	mptcp_for_each_subflow(msk, subflow) {
-		struct sock *ssk = mptcp_subflow_tcp_sock(subflow);
-		bool ssk_writeable = sk_stream_is_writeable(ssk);
-		struct socket *sock = READ_ONCE(ssk->sk_socket);
-
-		if (ssk_writeable || !sock)
-			continue;
-
-		/* enables ssk->write_space() callbacks */
-		set_bit(SOCK_NOSPACE, &sock->flags);
-	}
-
 	/* mptcp_data_acked() could run just before we set the NOSPACE bit,
 	 * so explicitly check for snd_una value
 	 */
@@ -3036,6 +3022,7 @@ static int mptcp_stream_accept(struct socket *sock, struct socket *newsock,
 		slowpath = lock_sock_fast(newsk);
 		mptcp_copy_inaddrs(newsk, msk->first);
 		mptcp_rcv_space_init(msk, msk->first);
+		set_bit(SOCK_NOSPACE, &newsock->flags);
 
 		/* set ssk->sk_socket of accept()ed flows to mptcp socket.
 		 * This is needed so NOSPACE flag can be set from tcp stack.
net/mptcp/subflow.c +22 -9
index 02d757e0ddb7..77cddbeaee1a 100644
@@ -996,19 +996,28 @@ static void subflow_data_ready(struct sock *sk)
 		mptcp_data_ready(parent, sk);
 }
 
-static void subflow_write_space(struct sock *sk)
+static void subflow_write_space(struct sock *ssk)
 {
-	struct mptcp_subflow_context *subflow = mptcp_subflow_ctx(sk);
-	struct socket *sock = READ_ONCE(sk->sk_socket);
-	struct sock *parent = subflow->conn;
+	struct mptcp_subflow_context *subflow = mptcp_subflow_ctx(ssk);
+	struct sock *sk = subflow->conn;
+	struct socket_wq *wq;
 
-	if (!sk_stream_is_writeable(sk))
+	if (!sk_stream_is_writeable(ssk) || !sk_stream_is_writeable(sk) ||
+	    !sk->sk_socket || !test_bit(MPTCP_NOSPACE, &mptcp_sk(sk)->flags))
 		return;
 
-	if (sock && sk_stream_is_writeable(parent))
-		clear_bit(SOCK_NOSPACE, &sock->flags);
-
-	sk_stream_write_space(parent);
+	/* The following is quite alike sk_stream_write_space, but avoids
+	 * clearing the sk SOCK_NOSPACE bit
+	 */
+	clear_bit(MPTCP_NOSPACE, &mptcp_sk(sk)->flags);
+	rcu_read_lock();
+	wq = rcu_dereference(sk->sk_wq);
+	if (skwq_has_sleeper(wq))
+		wake_up_interruptible_poll(&wq->wait, EPOLLOUT |
+		                        EPOLLWRNORM | EPOLLWRBAND);
+	if (wq && wq->fasync_list && !(sk->sk_shutdown & SEND_SHUTDOWN))
+		sock_wake_async(wq, SOCK_WAKE_SPACE, POLL_OUT);
+	rcu_read_unlock();
 }
 
 static struct inet_connection_sock_af_ops *
@@ -1208,6 +1217,10 @@ int mptcp_subflow_create_socket(struct sock *sk, struct socket **new_sock)
 	SOCK_INODE(sf)->i_uid = SOCK_INODE(sk->sk_socket)->i_uid;
 	SOCK_INODE(sf)->i_gid = SOCK_INODE(sk->sk_socket)->i_gid;
 
+	/* subflows will always call into sk_write_space, and subflow_write_space()
+	 * will be responsible of doing the actual wake-up
+	 */
+	set_bit(SOCK_NOSPACE, &sf->flags);
 	subflow = mptcp_subflow_ctx(sf->sk);
 	pr_debug("subflow=%p", subflow);
 
Signature · 1 line
2.26.2
author
#9 lore raw

PATCH 6/6 mptcp: use mptcp backlog.

In reply to Paolo Abeni

We have some tasks triggered by the subflow receive path which require to access the msk socket status, specifically:
mptcp_clean_una() and mptcp_push_pending()

We have almost everything in place to use the msk backlog and a standard !owned by user schema for that.

Introduce a dummy per msk sk_buff to trigger into the backlog as needed.

Since the worker is no more used to clean the acked data, for fallback sockets we need to explicitly call flush them.

As added bonus we can simplify a lot mptcp_poll() and move the timer update under the data lock

Signed-off-by: Paolo Abeni <pabeni(a)redhat.com> --
rfc v3-> v2:
- dfrag_release is always invoked under sock spin lock, must use mem_release
- move the mem_reclaim in mptcp_clean_una
- cleanup poll, leveraging the previous patch

---
 net/mptcp/options.c  |  18 +++--
 net/mptcp/protocol.c | 183 +++++++++++++++++++++++++++----------------
 net/mptcp/protocol.h |   8 ++
 3 files changed, 134 insertions(+), 75 deletions(-)
net/mptcp/options.c +12 -6
index 3986454a0340..48dd6622a04c 100644
@@ -189,7 +189,7 @@ static void mptcp_parse_option(const struct sk_buff *skb,
 				ptr += 4;
 			}
 
-			pr_debug("data_ack=%llu", mp_opt->data_ack);
+			pr_debug("64=%d data_ack=%llx", mp_opt->ack64, mp_opt->data_ack);
 		}
 
 		if (mp_opt->use_map) {
@@ -207,7 +207,7 @@ static void mptcp_parse_option(const struct sk_buff *skb,
 			mp_opt->data_len = get_unaligned_be16(ptr);
 			ptr += 2;
 
-			pr_debug("data_seq=%llu subflow_seq=%u data_len=%u",
+			pr_debug("data_seq=%llx subflow_seq=%u data_len=%u",
 				 mp_opt->data_seq, mp_opt->subflow_seq,
 				 mp_opt->data_len);
 		}
@@ -830,7 +830,7 @@ static u64 expand_ack(u64 old_ack, u64 cur_ack, bool use_64bit)
 }
 
 static void ack_update_msk(struct mptcp_sock *msk,
-			   const struct sock *ssk,
+			   struct sock *ssk,
 			   struct mptcp_options_received *mp_opt)
 {
 	u64 new_wnd_end, new_snd_una, snd_nxt = READ_ONCE(msk->snd_nxt);
@@ -854,8 +854,10 @@ static void ack_update_msk(struct mptcp_sock *msk,
 
 	if (after64(new_wnd_end, msk->wnd_end)) {
 		msk->wnd_end = new_wnd_end;
-		if (mptcp_send_head(sk))
-			mptcp_schedule_work(sk);
+		if (!sock_owned_by_user(sk))
+			__mptcp_subflow_push_pending(sk, ssk);
+		else
+			__mptcp_add_to_backlog(sk, MPTCP_PUSH_PENDING);
 	}
 
 	if (after64(new_snd_una, old_snd_una)) {
@@ -915,8 +917,12 @@ void mptcp_incoming_options(struct sock *sk, struct sk_buff *skb)
 	struct mptcp_options_received mp_opt;
 	struct mptcp_ext *mpext;
 
-	if (__mptcp_check_fallback(msk))
+	if (__mptcp_check_fallback(msk)) {
+		mptcp_data_lock(subflow->conn);
+		__mptcp_data_acked(subflow->conn);
+		mptcp_data_unlock(subflow->conn);
 		return;
+	}
 
 	mptcp_get_options(skb, &mp_opt);
 	if (!check_fully_established(msk, sk, subflow, skb, &mp_opt))
net/mptcp/protocol.c +114 -69

335 lines not on this page · raw message

net/mptcp/protocol.h +8 -0
index 0047295b766f..3a7eb6384801 100644
@@ -92,6 +92,8 @@
 #define MPTCP_FALLBACK_DONE	4
 #define MPTCP_WORK_CLOSE_SUBFLOW 5
 #define MPTCP_WORKER_RUNNING	6
+#define MPTCP_PUSH_PENDING	7
+#define MPTCP_CLEAN_UNA	8
 
 static inline bool before64(__u64 seq1, __u64 seq2)
 {
@@ -249,6 +251,9 @@ struct mptcp_sock {
 	struct list_head join_list;
 	struct socket	*subflow; /* outgoing connect/listener/!mp_capable */
 	struct sock	*first;
+
+	struct sk_buff backlog;
+
 	struct mptcp_pm_data	pm;
 	struct {
 		u32	space;	/* bytes copied in last measurement window */
@@ -440,6 +445,9 @@ unsigned int mptcp_get_add_addr_timeout(struct net *net);
 void mptcp_subflow_fully_established(struct mptcp_subflow_context *subflow,
 				     struct mptcp_options_received *mp_opt);
 bool mptcp_subflow_data_available(struct sock *sk);
+void mptcp_push_pending(struct sock *sk, unsigned int flags);
+void __mptcp_subflow_push_pending(struct sock *sk, struct sock *ssk);
+void __mptcp_add_to_backlog(struct sock *sk, int event);
 void __init mptcp_subflow_init(void);
 void mptcp_subflow_shutdown(struct sock *sk, struct sock *ssk, int how);
 void __mptcp_close_ssk(struct sock *sk, struct sock *ssk,
Signature · 1 line
2.26.2
#12 lore raw
In reply to Paolo Abeni
Quoted 10 lines · Paolo Abeni

This series refactor once again the data path, with the goal of avoding the worker usage for the fast path.

The worker will still be used to:
- cleanup closed subflows
- handle timeout
- retransmissions
- send the DATA_FIN/DATA_FIN ack.
- fallback eof

Quoted 2 lines

The latters 2 use case could be fourther removed, if we accept/consider sending DATA_FIN/ack on a single subflow.

Sending DATA_FIN/ack on a single subflow looks ok to me in this case. It would make sense to send that ack on the active subflow where incoming data is being processed. On the workqueue that we didn't know which subflow would be best to use, and didn't want to pick an inactive and possibly-not-working subflow.

Mat

Quoted 29 lines

The code is still pretty much unstable - survives very basic tests and don't complete yet a single self-test iteration, anyhow sharing early to discuss a few design points.

- separate memory accunting for write memory in patch 1 is a required pre-req for patch 2, but also brings a bit of additional complexity in the following patches. I would be very happy to find any other alternative.

- in patch 3 we may end-up pre-allocating more-than-strictly-required skbs. Doing more accurate estimate would be nice, but I don't know how without adding even more complexity.
Pre-allocating skbs is quite necessary, otherwise under memory pressure we may end-up copying data to kernel (in the msk write queue) but not being able later to allocate the required skbs at push_pending() time and xmit will be stuck.

- I guess patch 6 could cause quite hard reaction upstream, if they will dare look inside. I like it ;)

Paolo Abeni (6):
mptcp: separate accouting for wmem forward alloc mem
mptcp: protect the rx path with the msk socket spinlock
mptcp: allocate TX skbs in msk context.
mptcp: avoid a few atomic ops in the rx path
mptcp: simplify mptcp_nospace
mptcp: use mptcp backlog.

Quoted 9 lines
net/mptcp/mptcp_diag.c |   2 +-
net/mptcp/options.c    |  47 ++--
net/mptcp/protocol.c   | 606 +++++++++++++++++++++++++++++------------
net/mptcp/protocol.h   |  25 +-
net/mptcp/subflow.c    |  31 ++-
5 files changed, 496 insertions(+), 215 deletions(-)
Signature · 1 line
2.26.2

--
Mat Martineau
Intel

#2 lore raw

[MPTCP] Re: [PATCH net-next 1/6] mptcp: separate accouting for wmem forward alloc mem

In reply to Paolo Abeni
Quoted 25 lines · Paolo Abeni

Account separatelly the memory forward allocated for write operation. When sendmsg() need more memory it looks at wforward_alloc first and if that is exaused, move more space from sk_forward_alloc.

This will simplify the next patch

Signed-off-by Paolo Abeni <pabeni(a)redhat.com>

---
TODO:
- sk_diag_dump() after calling inet_sk_diag_fill()
must modify the newly added nlhdr, changing INET_DIAG_MEMINFO addr and specifically
.idiag_fmem to sk_forward_alloc + wforward_alloc
---
net/mptcp/protocol.c | 54 ++++++++++++++++++++++++++++++++++++++++---- net/mptcp/protocol.h | 1 +

2 files changed, 50 insertions(+), 5 deletions(-)
net/mptcp/protocol.c +0 -0
index 0956c7db0fd5..7000560c0cd9 100644
@@ -840,9 +840,50 @@ static bool mptcp_frag_can_collapse_to(const struct mptcp_sock *msk,
 		df->data_seq + df->data_len == msk->write_seq;

net/mptcp/protocol.c static bool mptcp_frag_can_collapse_to(const struct mptcp_sock *msk, · line 850

Quoted 10 lines
net/mptcp/protocol.c +8 -0
 }
 
+static bool mptcp_wmem_alloc(struct sock *sk, int size)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+	int amount;
+	int ret;
+
+	if (msk->wforward_alloc >= size)
+		goto account;

'ret' is uninitialized on this 'goto' path.

Mat

Quoted 108 lines
net/mptcp/protocol.c +34 -1
+
+	ret = sk_wmem_schedule(sk, size);
+	if (!ret)
+		return false;
+
+	/* try to keep half fwd alloc memory for each direction */
+	amount = max(size, sk->sk_forward_alloc >> 1);
+	sk->sk_forward_alloc -= amount;
+	msk->wforward_alloc += amount;
+
+account:
+	msk->wforward_alloc -= size;
+	return ret;
+}
+
+static void mptcp_wmem_uncharge(struct sock *sk, int size)
+{
+	mptcp_sk(sk)->wforward_alloc += size;
+}
+
+static void mptcp_mem_reclaim_partial(struct sock *sk)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+
+	sk->sk_forward_alloc += msk->wforward_alloc;
+	msk->wforward_alloc = 0;
+	sk_mem_reclaim_partial(sk);
+
+	/* split the remaining fwd allocated memory between rx and tx */
+	msk->wforward_alloc = sk->sk_forward_alloc >> 1;
+	sk->sk_forward_alloc -= msk->wforward_alloc;
+}
+
 static void dfrag_uncharge(struct sock *sk, int len)
 {
-	sk_mem_uncharge(sk, len);
+	mptcp_wmem_uncharge(sk, len);
 	sk_wmem_queued_add(sk, -len);
 }
 
@@ -897,8 +938,8 @@ static void mptcp_clean_una(struct sock *sk)
 	}
 

out:

net/mptcp/protocol.c +2 -2
-	if (cleaned)
-		sk_mem_reclaim_partial(sk);
+	if (cleaned && tcp_under_memory_pressure(sk))
+		mptcp_mem_reclaim_partial(sk);

}

static void mptcp_clean_una_wakeup(struct sock *sk)

net/mptcp/protocol.c +5 -2
@@ -1322,11 +1363,12 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 		offset = dfrag->offset + dfrag->data_len;
 		psize = pfrag->size - offset;
 		psize = min_t(size_t, psize, msg_data_left(msg));
-		if (!sk_wmem_schedule(sk, psize + frag_truesize))
+		if (!mptcp_wmem_alloc(sk, psize + frag_truesize))
 			goto wait_for_memory;
 
 		if (copy_page_from_iter(dfrag->page, offset, psize,
 					&msg->msg_iter) != psize) {
+			mptcp_wmem_uncharge(sk, psize + frag_truesize);
 			ret = -EFAULT;
 			goto out;
 		}
@@ -1342,7 +1384,6 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 		 * Note: we charge such data both to sk and ssk
 		 */
 		sk_wmem_queued_add(sk, frag_truesize);
-		sk->sk_forward_alloc -= frag_truesize;
 		if (!dfrag_collapsed) {
 			get_page(dfrag->page);
 			list_add_tail(&dfrag->list, &msk->rtx_queue);
@@ -1965,6 +2006,7 @@ static int __mptcp_init_sock(struct sock *sk)
 	INIT_WORK(&msk->work, mptcp_worker);
 	msk->out_of_order_queue = RB_ROOT;
 	msk->first_pending = NULL;
+	msk->wforward_alloc = 0;
 
 	msk->ack_hint = NULL;
 	msk->first = NULL;
@@ -2155,6 +2197,8 @@ static void __mptcp_destroy_sock(struct sock *sk)
 
 	sk->sk_prot->destroy(sk);
 
+	sk->sk_forward_alloc += msk->wforward_alloc;
+	msk->wforward_alloc = 0;
 	sk_stream_kill_queues(sk);
 	xfrm_sk_free_policy(sk);
 	sk_refcnt_debug_release(sk);
net/mptcp/protocol.h +1 -0
index 07542f7b349a..2a842d228950 100644
@@ -219,6 +219,7 @@ struct mptcp_sock {
 	u64		ack_seq;
 	u64		rcv_wnd_sent;
 	u64		rcv_data_fin_seq;
+	int		wforward_alloc;
 	struct sock	*last_snd;
 	int		snd_burst;
 	int		old_wspace;
Signature · 4 lines
2.26.2
_______________________________________________
mptcp mailing list -- mptcp(a)lists.01.org
To unsubscribe send an email to mptcp-leave(a)lists.01.org

--
Mat Martineau
Intel

#4 lore raw

[MPTCP] Re: [PATCH net-next 2/6] mptcp: protect the rx path with the msk socket spinlock

In reply to Paolo Abeni
Quoted 5 lines · Paolo Abeni

Such spinlock is currently used only to protect the 'owned' flag inside the socket lock itself. With this patch we extend it's scope to protect the whole msk receive path and sk_forward_memory.

Is it a problem that lock_sock()/release_sock() and the data path will be competing for this spinlock? Just wondering if the choice to use this lock is driven by not wanting to add a new lock (which is understandable!). Given that the spinlock is only held for short times hopefully the data path is not delaying the regular socket lock calls very much.

Quoted 11 lines

Given the above we can always move data into the msk receive queue (and OoO queue) from the subflow.

We leverage the previous commit, so that we need to acquire the spinlock in the tx path only when moving fwd memory to wfwd and vice versa.

recvmsg() must now explicitly acquire the socket spinlock when moving skbs out of sk_receive_queue. We use a snd rx queue and splice the first into the latter to reduce the number locks required.

To see if I understood this (and the code) correctly:

For the msk, sk_receive_queue is protected only by the mptcp_data_lock() and is where MPTCP-level data is reassembled in-order without holding the msk socket lock.

msk->receive_queue is protected by the socket lock, and is where in-order skbs are moved so they can be copied to userspace.

I still need to take a deeper look at the locking changes but the approach looks ok.

--
Mat Martineau
Intel

#10 lore raw

[MPTCP] Re: [PATCH net-next 6/6] mptcp: use mptcp backlog.

In reply to Paolo Abeni

hello,

Quoted 34 lines · Paolo Abeni

We have some tasks triggered by the subflow receive path which require to access the msk socket status, specifically:
mptcp_clean_una() and mptcp_push_pending()

We have almost everything in place to use the msk backlog and a standard !owned by user schema for that.

Introduce a dummy per msk sk_buff to trigger into the backlog as needed.

Since the worker is no more used to clean the acked data, for fallback sockets we need to explicitly call flush them.

As added bonus we can simplify a lot mptcp_poll() and move the timer update under the data lock

Signed-off-by: Paolo Abeni <pabeni(a)redhat.com> --
rfc v3-> v2:
 - dfrag_release is always invoked under sock spin lock, must use mem_release
 - move the mem_reclaim in mptcp_clean_una
 - cleanup poll, leveraging the previous patch
---

 net/mptcp/options.c  |  18 +++--
 net/mptcp/protocol.c | 183 +++++++++++++++++++++++++++----------------
 net/mptcp/protocol.h |   8 ++
 3 files changed, 134 insertions(+), 75 deletions(-)
net/mptcp/options.c +0 -0
index 3986454a0340..48dd6622a04c 100644

net/mptcp/options.c static void mptcp_parse_option(const struct sk_buff *skb, · line 192

Quoted 5 lines
net/mptcp/options.c +1 -1
@@ -189,7 +189,7 @@ static void mptcp_parse_option(const struct sk_buff *skb,
  				ptr += 4;
  			}
 
-			pr_debug("data_ack=%llu", mp_opt->data_ack);
+			pr_debug("64=%d data_ack=%llx", mp_opt->ack64, mp_opt->data_ack);

why are you changing (here and in hunks below) the format specifier of DACKs from decimal to hex format? tcpdump and wireshark print it as decimal, and we use to compare it with TCP sequence numbers, that are in decimal format too: I'd prefer to keep it %llu.

WDYT?

Signature · 1 line
davide
author
#5 lore raw

[MPTCP] Re: [PATCH net-next 2/6] mptcp: protect the rx path with the msk socket spinlock

In reply to Mat Martineau
Quoted 8 lines · Mat Martineau
Quoted 5 lines · Paolo Abeni

Such spinlock is currently used only to protect the 'owned' flag inside the socket lock itself. With this patch we extend it's scope to protect the whole msk receive path and sk_forward_memory.

Quoted 5 lines

Is it a problem that lock_sock()/release_sock() and the data path will be competing for this spinlock? Just wondering if the choice to use this lock is driven by not wanting to add a new lock (which is understandable!). Given that the spinlock is only held for short times hopefully the data path is not delaying the regular socket lock calls very much.

Path 6/6 will shed more light on this. Using the socket spin lock allows the hack/trick in such patch.

I don't think contention will be a problem, at least in the short term
- if contention will become visibile, we will be doing a lot of pps, which is good ;). Additionally, in the current code, we already have contention on the msk socket lock via trylock.

Quoted 12 lines
Quoted 11 lines

Given the above we can always move data into the msk receive queue (and OoO queue) from the subflow.

We leverage the previous commit, so that we need to acquire the spinlock in the tx path only when moving fwd memory to wfwd and vice versa.

recvmsg() must now explicitly acquire the socket spinlock when moving skbs out of sk_receive_queue. We use a snd rx queue and splice the first into the latter to reduce the number locks required.

Quoted 8 lines

To see if I understood this (and the code) correctly:

For the msk, sk_receive_queue is protected only by the mptcp_data_lock() and is where MPTCP-level data is reassembled in-order without holding the msk socket lock.

msk->receive_queue is protected by the socket lock, and is where in-order skbs are moved so they can be copied to userspace.

Yes, the above is correct. The introduction of msk->receive_queue allows skipping one mptcp_data_lock() in recvmsg() for bulk transfer.

There is a possible follow-up improvement:

Open code lock_sock()/release_sock(), so that we can do mptcp-specific operations - e.g. splicing receive_queue, or fwd allocating memory -
while acquiring the msk socket lock spinlock inside such helper. That
will avoid up to 2 additional lock operations per recvmsg()/sendmsg().

/P

author
#11 lore raw

[MPTCP] Re: [PATCH net-next 6/6] mptcp: use mptcp backlog.

In reply to Davide Caratti
Quoted 40 lines · Davide Caratti

hello,

Quoted 37 lines · Paolo Abeni

We have some tasks triggered by the subflow receive path which require to access the msk socket status, specifically:
mptcp_clean_una() and mptcp_push_pending()

We have almost everything in place to use the msk backlog and a standard !owned by user schema for that.

Introduce a dummy per msk sk_buff to trigger into the backlog as needed.

Since the worker is no more used to clean the acked data, for fallback sockets we need to explicitly call flush them.

As added bonus we can simplify a lot mptcp_poll() and move the timer update under the data lock

Signed-off-by: Paolo Abeni <pabeni(a)redhat.com> --
rfc v3-> v2:
- dfrag_release is always invoked under sock spin lock, must use mem_release
- move the mem_reclaim in mptcp_clean_una
- cleanup poll, leveraging the previous patch

---
 net/mptcp/options.c  |  18 +++--
 net/mptcp/protocol.c | 183 +++++++++++++++++++++++++++----------------
 net/mptcp/protocol.h |   8 ++
 3 files changed, 134 insertions(+), 75 deletions(-)
net/mptcp/options.c +0 -0
index 3986454a0340..48dd6622a04c 100644
@@ -189,7 +189,7 @@ static void mptcp_parse_option(const struct sk_buff *skb,
 				ptr += 4;
 			}
 
Quoted 5 lines

net/mptcp/options.c static void mptcp_parse_option(const struct sk_buff *skb, · line 192

Quoted 2 lines
net/mptcp/options.c +1 -1
-			pr_debug("data_ack=%llu", mp_opt->data_ack);
+			pr_debug("64=%d data_ack=%llx", mp_opt->ack64, mp_opt->data_ack);

why are you changing (here and in hunks below) the format specifier of DACKs from decimal to hex format?

whoops, this is an unneeded/unindended change that slipped in - it's there as debug hint, fits better my personal preference/setup.

Will be dropped in later iteration.

Cheers,

Paolo

author
#13 lore raw
In reply to Mat Martineau
Quoted 12 lines · Mat Martineau
Quoted 10 lines · Paolo Abeni

This series refactor once again the data path, with the goal of avoding the worker usage for the fast path.

The worker will still be used to:
- cleanup closed subflows
- handle timeout
- retransmissions
- send the DATA_FIN/DATA_FIN ack.
- fallback eof

Quoted 4 lines
Quoted 2 lines

The latters 2 use case could be fourther removed, if we accept/consider sending DATA_FIN/ack on a single subflow.

Sending DATA_FIN/ack on a single subflow looks ok to me in this case.

Excellent! does that applies to the data_fin, too?

Quoted 5 lines

It
would make sense to send that ack on the active subflow where incoming data is being processed. On the workqueue that we didn't know which subflow would be best to use, and didn't want to pick an inactive and possibly-not-working subflow.

If we can send the data_fin on the subflow reaching the end of sequence, we will not need anymore the workqueue to send the data_fin. And the same for the data fin ack.

We will need the workqueue only for shutting down the subflows after receiving the data_fin ack.

Cheers,

Paolo

[sorry, this mail got stuck in my mailbox the whole day]

#14 lore raw
Quoted 13 lines · Paolo Abeni
Quoted 12 lines · Mat Martineau
Quoted 10 lines · Paolo Abeni

This series refactor once again the data path, with the goal of avoding the worker usage for the fast path.

The worker will still be used to:
- cleanup closed subflows
- handle timeout
- retransmissions
- send the DATA_FIN/DATA_FIN ack.
- fallback eof

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

The latters 2 use case could be fourther removed, if we accept/consider sending DATA_FIN/ack on a single subflow.

Sending DATA_FIN/ack on a single subflow looks ok to me in this case.

Excellent! does that applies to the data_fin, too?

Sure.

Quoted 6 lines
Quoted 5 lines

It would make sense to send that ack on the active subflow where incoming data is being processed. On the workqueue that we didn't know which subflow would be best to use, and didn't want to pick an inactive and possibly-not-working subflow.

Quoted 7 lines

If we can send the data_fin on the subflow reaching the end of sequence, we will not need anymore the workqueue to send the data_fin. And the same for the data fin ack.

We will need the workqueue only for shutting down the subflows after receiving the data_fin ack.

And retransmission?

--
Mat Martineau
Intel