mptcp: just another complete datapath refactor
Subsystems (2)
Paolo Abeni
cover mptcp: just another complete datapath refactor
This series refactor once again the data path, with the goal of avoding the worker usage for the fast path.
The worker will still be used to:
- cleanup closed subflows
- handle timeout
- retransmissions
- send the DATA_FIN/DATA_FIN ack.
- fallback eof
The latters 2 use case could be fourther removed, if we accept/consider sending DATA_FIN/ack on a single subflow.
The code is still pretty much unstable - survives very basic tests and don't complete yet a single self-test iteration, anyhow sharing early to discuss a few design points.
- separate memory accunting for write memory in patch 1 is a required pre-req for patch 2, but also brings a bit of additional complexity in the following patches. I would be very happy to find any other alternative.
- in patch 3 we may end-up pre-allocating more-than-strictly-required skbs. Doing more accurate estimate would be nice, but I don't know how without adding even more complexity.
Pre-allocating skbs is quite necessary, otherwise under memory pressure we may end-up copying data to kernel (in the msk write queue) but not being able later to allocate the required skbs at push_pending() time and xmit will be stuck.
- I guess patch 6 could cause quite hard reaction upstream, if they will dare look inside. I like it ;)
Paolo Abeni (6):
mptcp: separate accouting for wmem forward alloc mem
mptcp: protect the rx path with the msk socket spinlock
mptcp: allocate TX skbs in msk context.
mptcp: avoid a few atomic ops in the rx path
mptcp: simplify mptcp_nospace
mptcp: use mptcp backlog.
net/mptcp/mptcp_diag.c | 2 +- net/mptcp/options.c | 47 ++-- net/mptcp/protocol.c | 606 +++++++++++++++++++++++++++++------------ net/mptcp/protocol.h | 25 +- net/mptcp/subflow.c | 31 ++- 5 files changed, 496 insertions(+), 215 deletions(-)
Signature · 1 line
2.26.2
Paolo Abeni
PATCH 1/6 mptcp: separate accouting for wmem forward alloc mem
Account separatelly the memory forward allocated for write operation. When sendmsg() need more memory it looks at wforward_alloc first and if that is exaused, move more space from sk_forward_alloc.
This will simplify the next patch
---
TODO:
- sk_diag_dump() after calling inet_sk_diag_fill() must modify the newly added nlhdr, changing INET_DIAG_MEMINFO addr and specifically .idiag_fmem to sk_forward_alloc + wforward_alloc
--- net/mptcp/protocol.c | 54 ++++++++++++++++++++++++++++++++++++++++---- net/mptcp/protocol.h | 1 + 2 files changed, 50 insertions(+), 5 deletions(-)
net/mptcp/protocol.c +49 -5
@@ -840,9 +840,50 @@ static bool mptcp_frag_can_collapse_to(const struct mptcp_sock *msk, df->data_seq + df->data_len == msk->write_seq; } +static bool mptcp_wmem_alloc (struct sock *sk,int size) +{ +struct mptcp_sock *msk =mptcp_sk (sk); +int amount; +int ret; + +if (msk->wforward_alloc >= size) + goto account; + + ret =sk_wmem_schedule (sk, size); +if (!ret) +return false; + +/* try to keep half fwd alloc memory for each direction */ + amount =max (size, sk->sk_forward_alloc >>1 ); + sk->sk_forward_alloc -= amount; + msk->wforward_alloc += amount; + +account: + msk->wforward_alloc -= size; +return ret; +} + +static void mptcp_wmem_uncharge (struct sock *sk,int size) +{ +mptcp_sk (sk)->wforward_alloc += size; +} + +static void mptcp_mem_reclaim_partial (struct sock *sk) +{ +struct mptcp_sock *msk =mptcp_sk (sk); + + sk->sk_forward_alloc += msk->wforward_alloc; + msk->wforward_alloc =0 ; +sk_mem_reclaim_partial (sk); + +/* split the remaining fwd allocated memory between rx and tx */ + msk->wforward_alloc = sk->sk_forward_alloc >>1 ; + sk->sk_forward_alloc -= msk->wforward_alloc; +} +static void dfrag_uncharge (struct sock *sk,int len) { -sk_mem_uncharge (sk, len); +mptcp_wmem_uncharge (sk, len);sk_wmem_queued_add (sk, -len); }
@@ -897,8 +938,8 @@ static void mptcp_clean_una(struct sock *sk) } out: -if (cleaned) -sk_mem_reclaim_partial (sk); +if (cleaned &&tcp_under_memory_pressure (sk)) +mptcp_mem_reclaim_partial (sk); }static void mptcp_clean_una_wakeup (struct sock *sk)
@@ -1322,11 +1363,12 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) offset = dfrag->offset + dfrag->data_len; psize = pfrag->size - offset; psize =min_t (size_t, psize,msg_data_left (msg)); -if (!sk_wmem_schedule (sk, psize + frag_truesize)) +if (!mptcp_wmem_alloc (sk, psize + frag_truesize)) goto wait_for_memory;if (copy_page_from_iter (dfrag->page, offset, psize, &msg->msg_iter) != psize) { +mptcp_wmem_uncharge (sk, psize + frag_truesize); ret = -EFAULT ; goto out; }
@@ -1342,7 +1384,6 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)* Note: we charge such data both to sk and ssk */ sk_wmem_queued_add (sk, frag_truesize); - sk->sk_forward_alloc -= frag_truesize;if (!dfrag_collapsed) {get_page (dfrag->page);list_add_tail (&dfrag->list, &msk->rtx_queue);
@@ -1965,6 +2006,7 @@ static int __mptcp_init_sock(struct sock *sk)INIT_WORK (&msk->work, mptcp_worker); msk->out_of_order_queue =RB_ROOT ; msk->first_pending =NULL ; + msk->wforward_alloc =0 ; msk->ack_hint =NULL ; msk->first =NULL ;
@@ -2155,6 +2197,8 @@ static void __mptcp_destroy_sock(struct sock *sk) sk->sk_prot->destroy (sk); + sk->sk_forward_alloc += msk->wforward_alloc; + msk->wforward_alloc =0 ;sk_stream_kill_queues (sk);xfrm_sk_free_policy (sk);sk_refcnt_debug_release (sk);
net/mptcp/protocol.h +1 -0
@@ -219,6 +219,7 @@ struct mptcp_sock {u64 ack_seq;u64 rcv_wnd_sent;u64 rcv_data_fin_seq; +int wforward_alloc;struct sock *last_snd;int snd_burst;int old_wspace;
Signature · 1 line
2.26.2
Mat Martineau
[MPTCP] Re: [PATCH net-next 1/6] mptcp: separate accouting for wmem forward alloc mem
Quoted 25 lines · Paolo Abeni
Account separatelly the memory forward allocated for write operation. When
sendmsg()need more memory it looks atwforward_allocfirst and if that is exaused, move more space fromsk_forward_alloc.This will simplify the next patch
Signed-off-by Paolo Abeni <pabeni(a)redhat.com>---
TODO:
-sk_diag_dump()after callinginet_sk_diag_fill()
must modify the newly added nlhdr, changingINET_DIAG_MEMINFOaddr and specifically
.idiag_fmem tosk_forward_alloc+wforward_alloc
---net/mptcp/protocol.c| 54 ++++++++++++++++++++++++++++++++++++++++----net/mptcp/protocol.h| 1 +2 files changed, 50 insertions(+), 5 deletions(-)net/mptcp/protocol.c +0
-0@@ -840,9 +840,50 @@ static bool mptcp_frag_can_collapse_to(const struct mptcp_sock *msk, df->data_seq + df->data_len == msk->write_seq;
net/mptcp/protocol.c static bool mptcp_frag_can_collapse_to(const struct mptcp_sock *msk, · line 850
Quoted 10 lines
net/mptcp/protocol.c +8
-0} +static bool mptcp_wmem_alloc (struct sock *sk,int size) +{ +struct mptcp_sock *msk =mptcp_sk (sk); +int amount; +int ret; + +if (msk->wforward_alloc >= size) + goto account;
'ret' is uninitialized on this 'goto' path.
Mat
Quoted 108 lines
net/mptcp/protocol.c +34
-1+ + ret =sk_wmem_schedule (sk, size); +if (!ret) +return false; + +/* try to keep half fwd alloc memory for each direction */ + amount =max (size, sk->sk_forward_alloc >>1 ); + sk->sk_forward_alloc -= amount; + msk->wforward_alloc += amount; + +account: + msk->wforward_alloc -= size; +return ret; +} + +static void mptcp_wmem_uncharge (struct sock *sk,int size) +{ +mptcp_sk (sk)->wforward_alloc += size; +} + +static void mptcp_mem_reclaim_partial (struct sock *sk) +{ +struct mptcp_sock *msk =mptcp_sk (sk); + + sk->sk_forward_alloc += msk->wforward_alloc; + msk->wforward_alloc =0 ; +sk_mem_reclaim_partial (sk); + +/* split the remaining fwd allocated memory between rx and tx */ + msk->wforward_alloc = sk->sk_forward_alloc >>1 ; + sk->sk_forward_alloc -= msk->wforward_alloc; +} +static void dfrag_uncharge (struct sock *sk,int len) { -sk_mem_uncharge (sk, len); +mptcp_wmem_uncharge (sk, len);sk_wmem_queued_add (sk, -len); }@@ -897,8 +938,8 @@ static void mptcp_clean_una(struct sock *sk) }out:
net/mptcp/protocol.c +2
-2-if (cleaned) -sk_mem_reclaim_partial (sk); +if (cleaned &&tcp_under_memory_pressure (sk)) +mptcp_mem_reclaim_partial (sk);}
static void
mptcp_clean_una_wakeup(struct sock *sk)net/mptcp/protocol.c +5
-2@@ -1322,11 +1363,12 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) offset = dfrag->offset + dfrag->data_len; psize = pfrag->size - offset; psize =min_t (size_t, psize,msg_data_left (msg)); -if (!sk_wmem_schedule (sk, psize + frag_truesize)) +if (!mptcp_wmem_alloc (sk, psize + frag_truesize)) goto wait_for_memory;if (copy_page_from_iter (dfrag->page, offset, psize, &msg->msg_iter) != psize) { +mptcp_wmem_uncharge (sk, psize + frag_truesize); ret = -EFAULT ; goto out; }@@ -1342,7 +1384,6 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)* Note: we charge such data both to sk and ssk */ sk_wmem_queued_add (sk, frag_truesize); - sk->sk_forward_alloc -= frag_truesize;if (!dfrag_collapsed) {get_page (dfrag->page);list_add_tail (&dfrag->list, &msk->rtx_queue);@@ -1965,6 +2006,7 @@ static int __mptcp_init_sock(struct sock *sk)INIT_WORK (&msk->work, mptcp_worker); msk->out_of_order_queue =RB_ROOT ; msk->first_pending =NULL ; + msk->wforward_alloc =0 ; msk->ack_hint =NULL ; msk->first =NULL ;@@ -2155,6 +2197,8 @@ static void __mptcp_destroy_sock(struct sock *sk) sk->sk_prot->destroy (sk); + sk->sk_forward_alloc += msk->wforward_alloc; + msk->wforward_alloc =0 ;sk_stream_kill_queues (sk);xfrm_sk_free_policy (sk);sk_refcnt_debug_release (sk);net/mptcp/protocol.h +1
-0@@ -219,6 +219,7 @@ struct mptcp_sock {u64 ack_seq;u64 rcv_wnd_sent;u64 rcv_data_fin_seq; +int wforward_alloc;struct sock *last_snd;int snd_burst;int old_wspace;Signature · 4 lines
2.26.2 _______________________________________________ mptcp mailing list -- mptcp(a)lists.01.org To unsubscribe send an email to mptcp-leave(a)lists.01.org
--
Mat Martineau
Intel
Paolo Abeni
PATCH 2/6 mptcp: protect the rx path with the msk socket spinlock
In reply to Paolo AbeniSuch spinlock is currently used only to protect the 'owned' flag inside the socket lock itself. With this patch we extend it's scope to protect the whole msk receive path and sk_forward_memory.
Given the above we can always move data into the msk receive queue (and OoO queue) from the subflow.
We leverage the previous commit, so that we need to acquire the spinlock in the tx path only when moving fwd memory to wfwd and vice versa.
recvmsg() must now explicitly acquire the socket spinlock when moving skbs out of sk_receive_queue. We use a snd rx queue and splice the first into the latter to reduce the number locks required.
---
- add __mptcp_mem_reclaim_partial() variant
--- net/mptcp/protocol.c | 158 +++++++++++++++++++++++++++++-------------- net/mptcp/protocol.h | 4 ++ 2 files changed, 113 insertions(+), 49 deletions(-)
net/mptcp/protocol.c +109 -49
@@ -270,7 +270,8 @@ static void mptcp_data_queue_ofo(struct mptcp_sock *msk, struct sk_buff *skb) end:skb_condense (skb); -skb_set_owner_r (skb, sk); +atomic_add (skb->truesize, &sk->sk_rmem_alloc); +sk_mem_charge (sk, skb->truesize); } static bool __mptcp_move_skb(struct mptcp_sock *msk, struct sock *ssk,
@@ -310,7 +311,8 @@ static bool __mptcp_move_skb(struct mptcp_sock *msk, struct sock *ssk,if (tail &&mptcp_try_coalesce (sk, tail, skb))return true; -skb_set_owner_r (skb, sk); +atomic_add (skb->truesize, &sk->sk_rmem_alloc); +sk_mem_charge (sk, skb->truesize);__skb_queue_tail (&sk->sk_receive_queue, skb);return true; }else if (after64 (MPTCP_SKB_CB (skb)->map_seq, msk->ack_seq)) {
@@ -600,7 +602,7 @@ static bool __mptcp_move_skbs_from_subflow(struct mptcp_sock *msk,return done; } -static bool mptcp_ofo_queue (struct mptcp_sock *msk) +static bool __mptcp_ofo_queue (struct mptcp_sock *msk) {struct sock *sk = (struct sock *)msk;struct sk_buff *skb, *tail;
@@ -646,34 +648,24 @@ static bool mptcp_ofo_queue(struct mptcp_sock *msk)/* In most cases we will be able to lock the mptcp socket. If its already * owned, we need to defer to the work queue to avoid ABBA deadlock. */ -static bool move_skbs_to_msk (struct mptcp_sock *msk,struct sock *ssk) +static void move_skbs_to_msk (struct mptcp_sock *msk,struct sock *ssk) {struct sock *sk = (struct sock *)msk;unsigned int moved =0 ; -if (READ_ONCE (sk->sk_lock.owned)) -return false; - -if (unlikely (!spin_trylock_bh (&sk->sk_lock.slock))) -return false; +mptcp_data_lock (sk); -/* must re-check after taking the lock */ -if (!READ_ONCE (sk->sk_lock.owned)) { -__mptcp_move_skbs_from_subflow (msk, ssk, &moved); -mptcp_ofo_queue (msk); - -/* If the moves have caught up with the DATA_FIN sequence number -* it's time to ack the DATA_FIN and change socket state, but -* this is not a good place to change state. Let the workqueue -* do it. -*/ -if (mptcp_pending_data_fin (sk,NULL )) -mptcp_schedule_work (sk); - } +__mptcp_move_skbs_from_subflow (msk, ssk, &moved); +__mptcp_ofo_queue (msk); -spin_unlock_bh (&sk->sk_lock.slock); - -return moved >0 ; +/* If the moves have caught up with the DATA_FIN sequence number +* it's time to ack the DATA_FIN and change socket state, but +* this is not a good place to change state. Let the workqueue +* do it. +*/ +if (mptcp_pending_data_fin (sk,NULL )) +mptcp_schedule_work (sk); +mptcp_data_unlock (sk); }void mptcp_data_ready (struct sock *sk,struct sock *ssk)
@@ -849,14 +841,18 @@ static bool mptcp_wmem_alloc(struct sock *sk, int size)if (msk->wforward_alloc >= size) goto account; +mptcp_data_lock (sk); ret =sk_wmem_schedule (sk, size); -if (!ret) +if (!ret) { +mptcp_data_unlock (sk);return false; + }/* try to keep half fwd alloc memory for each direction */ amount =max (size, sk->sk_forward_alloc >>1 ); sk->sk_forward_alloc -= amount; msk->wforward_alloc += amount; +mptcp_data_unlock (sk); account: msk->wforward_alloc -= size;
@@ -868,12 +864,11 @@ static void mptcp_wmem_uncharge(struct sock *sk, int size)mptcp_sk (sk)->wforward_alloc += size; } -static void mptcp_mem_reclaim_partial (struct sock *sk) +static void __mptcp_mem_reclaim_partial (struct sock *sk) {struct mptcp_sock *msk =mptcp_sk (sk); sk->sk_forward_alloc += msk->wforward_alloc; - msk->wforward_alloc =0 ;sk_mem_reclaim_partial (sk);/* split the remaining fwd allocated memory between rx and tx */
@@ -881,6 +876,13 @@ static void mptcp_mem_reclaim_partial(struct sock *sk) sk->sk_forward_alloc -= msk->wforward_alloc; } +static void mptcp_mem_reclaim_partial (struct sock *sk) +{ +mptcp_data_lock (sk); +__mptcp_mem_reclaim_partial (sk); +mptcp_data_unlock (sk); +} +static void dfrag_uncharge (struct sock *sk,int len) {mptcp_wmem_uncharge (sk, len);
@@ -1432,13 +1434,12 @@ static void mptcp_wait_data(struct sock *sk, long *timeo)static int __mptcp_recvmsg_mskq (struct mptcp_sock *msk,struct msghdr *msg, -size_t len) +size_t len,int *released) { -struct sock *sk = (struct sock *)msk;struct sk_buff *skb;int copied =0 ; -while ((skb =skb_peek (&sk->sk_receive_queue)) !=NULL ) { +while ((skb =skb_peek (&msk->receive_queue)) !=NULL ) {u32 offset =MPTCP_SKB_CB (skb)->offset;u32 data_len = skb->len - offset;u32 count =min_t (size_t, len - copied, data_len);
@@ -1458,7 +1459,8 @@ static int __mptcp_recvmsg_mskq(struct mptcp_sock *msk,break ; } -__skb_unlink (skb, &sk->sk_receive_queue); + *released += skb->truesize; +__skb_unlink (skb, &msk->receive_queue);__kfree_skb (skb);if (copied >= len)
@@ -1566,25 +1568,49 @@ static void mptcp_rcv_space_adjust(struct mptcp_sock *msk, int copied) msk->rcvq_space.time = mstamp; } -static bool __mptcp_move_skbs (struct mptcp_sock *msk,unsigned int rcv) +static void __mptcp_update_rmem (struct sock *sk,int *released) +{ +sk_mem_uncharge (sk, *released); +atomic_sub (*released, &sk->sk_rmem_alloc); + *released =0 ; +} + +static void mptcp_update_rmem (struct sock *sk,int *released) { +if (!released) +return ; + +mptcp_data_lock (sk); +__mptcp_update_rmem (sk, released); +mptcp_data_unlock (sk); +} + +static bool __mptcp_move_skbs (struct mptcp_sock *msk,unsigned int rcv,int *released) +{ +struct sock *sk = (struct sock *)msk;unsigned int moved =0 ; -bool done; +bool ret, done;/* avoid looping forever below on racing close */ -if (((struct sock *)msk)->sk_state ==TCP_CLOSE ) -return false; +if (sk->sk_state ==TCP_CLOSE ) + goto chk_splice;__mptcp_flush_join_list (msk); do { struct sock *ssk = mptcp_subflow_recv_lookup(msk); bool slowpath; - if (!ssk) + /* we can have data pending in the subflows only if the msk + * receive buffer was full at subflow_data_ready() time, + * it is an likely slow path + */ + if (likely(!ssk)) break; slowpath = lock_sock_fast(ssk); + mptcp_data_lock(sk); done = __mptcp_move_skbs_from_subflow(msk, ssk, &moved); + mptcp_data_unlock(sk); if (moved && rcv) { WRITE_ONCE(msk->rmem_pending, min(rcv, moved)); tcp_cleanup_rbuf(ssk, 1);
@@ -1593,17 +1619,28 @@ static bool __mptcp_move_skbs(struct mptcp_sock *msk, unsigned int rcv)unlock_sock_fast (ssk, slowpath); }while (!done); -if (mptcp_ofo_queue (msk) || moved >0 ) { -mptcp_check_data_fin ((struct sock *)msk); -return true; +chk_splice: +/* acquire the data lock only if some input data is pending */ + ret = moved >0 ; +if (!RB_EMPTY_ROOT (&msk->out_of_order_queue) || + !skb_queue_empty_lockless (&sk->sk_receive_queue)) { +mptcp_data_lock (sk); +__mptcp_update_rmem (sk, released); + ret |=__mptcp_ofo_queue (msk); +skb_queue_splice_tail_init (&sk->sk_receive_queue, + &msk->receive_queue); +mptcp_data_unlock (sk); } -return false; +if (ret) +mptcp_check_data_fin ((struct sock *)msk); +return !skb_queue_empty (&msk->receive_queue); }static int mptcp_recvmsg (struct sock *sk,struct msghdr *msg,size_t len,int nonblock,int flags,int *addr_len) {struct mptcp_sock *msk =mptcp_sk (sk); +int released =0 ;int copied =0 ;int target;long timeo;
@@ -1621,7 +1658,8 @@ static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,for (;;) {int bytes_read, old_space; - bytes_read =__mptcp_recvmsg_mskq (msk, msg, len - copied); + bytes_read =__mptcp_recvmsg_mskq (msk, msg, len - copied, + &released);if (unlikely (bytes_read <0 )) {if (!copied) copied = bytes_read;
@@ -1630,8 +1668,8 @@ static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, copied += bytes_read; -if (skb_queue_empty (&sk->sk_receive_queue) && -__mptcp_move_skbs (msk, len - copied)) +if (skb_queue_empty (&msk->receive_queue) && +__mptcp_move_skbs (msk, len - copied, &released))continue ;/* be sure to advertize window change */
@@ -1661,8 +1699,15 @@ static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,if (test_and_clear_bit (MPTCP_WORK_EOF , &msk->flags))mptcp_check_for_eof (msk); -if (sk->sk_shutdown &RCV_SHUTDOWN ) +if (sk->sk_shutdown &RCV_SHUTDOWN ) { +/* race breaker: the shutdown could be after the +* previous receive queue check +*/ +if (__mptcp_move_skbs (msk, len - copied, + &released)) +continue ;break ; + }if (sk->sk_state ==TCP_CLOSE ) { copied = -ENOTCONN ;
@@ -1681,26 +1726,29 @@ static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, }pr_debug ("block timeout %ld" , timeo); +mptcp_update_rmem (sk, &released);mptcp_wait_data (sk, &timeo); } -if (skb_queue_empty (&sk->sk_receive_queue)) { +if (skb_queue_empty_lockless (&sk->sk_receive_queue) && +skb_queue_empty (&msk->receive_queue)) {/* entire backlog drained, clear DATA_READY. */ clear_bit (MPTCP_DATA_READY , &msk->flags);/* .. race-breaker: ssk might have gotten new data * after last __mptcp_move_skbs() returned false. */ -if (unlikely (__mptcp_move_skbs (msk,0 ))) +if (unlikely (__mptcp_move_skbs (msk,0 , &released)))set_bit (MPTCP_DATA_READY , &msk->flags); }else if (unlikely (!test_bit (MPTCP_DATA_READY , &msk->flags))) {/* data to read but mptcp_wait_data() cleared DATA_READY */ set_bit (MPTCP_DATA_READY , &msk->flags); } out_err: +mptcp_update_rmem (sk, &released);pr_debug ("msk=%p data_ready=%d rx queue empty=%d copied=%d" , msk,test_bit (MPTCP_DATA_READY , &msk->flags), -skb_queue_empty (&sk->sk_receive_queue), copied); +skb_queue_empty_lockless (&sk->sk_receive_queue), copied);mptcp_rcv_space_adjust (msk, copied);release_sock (sk);
@@ -2004,6 +2052,7 @@ static int __mptcp_init_sock(struct sock *sk)INIT_LIST_HEAD (&msk->join_list);INIT_LIST_HEAD (&msk->rtx_queue);INIT_WORK (&msk->work, mptcp_worker); +__skb_queue_head_init (&msk->receive_queue); msk->out_of_order_queue =RB_ROOT ; msk->first_pending =NULL ; msk->wforward_alloc =0 ;
@@ -2200,6 +2249,7 @@ static void __mptcp_destroy_sock(struct sock *sk) sk->sk_forward_alloc += msk->wforward_alloc; msk->wforward_alloc =0 ;sk_stream_kill_queues (sk); +xfrm_sk_free_policy (sk);sk_refcnt_debug_release (sk);sock_put (sk);
@@ -2422,7 +2472,17 @@ static struct sock *mptcp_accept(struct sock *sk, int flags, int *err,void mptcp_destroy_common (struct mptcp_sock *msk) { -skb_rbtree_purge (&msk->out_of_order_queue); +struct sock *sk = (struct sock *)msk; +struct sk_buff *skb; +int queued =0 ; + +skb_queue_splice_tail_init (&sk->sk_receive_queue, &msk->receive_queue); +while ((skb =__skb_dequeue (&msk->receive_queue)) !=NULL ) { + queued += skb->truesize; +kfree_skb (skb); + } + queued +=skb_rbtree_purge (&msk->out_of_order_queue); +__mptcp_update_rmem (sk, &queued);mptcp_token_destroy (msk);mptcp_pm_free_anno_list (msk); }
net/mptcp/protocol.h +4 -0
@@ -239,6 +239,7 @@ struct mptcp_sock {struct work_struct work;struct sk_buff *ooo_last_skb;struct rb_root out_of_order_queue; +struct sk_buff_head receive_queue;struct list_head conn_list;struct list_head rtx_queue;struct mptcp_data_frag *first_pending;
@@ -255,6 +256,9 @@ struct mptcp_sock { } rcvq_space; }; +#define mptcp_data_lock (sk) spin_lock_bh(&sk->sk_lock.slock) +#define mptcp_data_unlock (sk) spin_unlock_bh(&sk->sk_lock.slock) +#define mptcp_for_each_subflow (__msk, __subflow) \ list_for_each_entry(__subflow, &((__msk)->conn_list), node)
Signature · 1 line
2.26.2
Mat Martineau
[MPTCP] Re: [PATCH net-next 2/6] mptcp: protect the rx path with the msk socket spinlock
Quoted 5 lines · Paolo Abeni
Such spinlock is currently used only to protect the 'owned' flag inside the socket lock itself. With this patch we extend it's scope to protect the whole msk receive path and
sk_forward_memory.
Is it a problem that lock_sock()/release_sock() and the data path will be competing for this spinlock? Just wondering if the choice to use this lock is driven by not wanting to add a new lock (which is understandable!). Given that the spinlock is only held for short times hopefully the data path is not delaying the regular socket lock calls very much.
Quoted 11 lines
Given the above we can always move data into the msk receive queue (and OoO queue) from the subflow.
We leverage the previous commit, so that we need to acquire the spinlock in the tx path only when moving fwd memory to wfwd and vice versa.
recvmsg()must now explicitly acquire the socket spinlock when moving skbs out ofsk_receive_queue. We use a snd rx queue and splice the first into the latter to reduce the number locks required.
To see if I understood this (and the code) correctly:
For the msk, sk_receive_queue is protected only by the mptcp_data_lock() and is where MPTCP-level data is reassembled in-order without holding the msk socket lock.
msk->receive_queue is protected by the socket lock, and is where in-order skbs are moved so they can be copied to userspace.
I still need to take a deeper look at the locking changes but the approach looks ok.
--
Mat Martineau
Intel
Paolo Abeni
[MPTCP] Re: [PATCH net-next 2/6] mptcp: protect the rx path with the msk socket spinlock
Quoted 8 lines · Mat Martineau
Quoted 5 lines · Paolo Abeni
Such spinlock is currently used only to protect the 'owned' flag inside the socket lock itself. With this patch we extend it's scope to protect the whole msk receive path and
sk_forward_memory.
Quoted 5 lines
Is it a problem that lock_sock()/release_sock() and the data path will be competing for this spinlock? Just wondering if the choice to use this lock is driven by not wanting to add a new lock (which is understandable!). Given that the spinlock is only held for short times hopefully the data path is not delaying the regular socket lock calls very much.
Path 6/6 will shed more light on this. Using the socket spin lock allows the hack/trick in such patch.
I don't think contention will be a problem, at least in the short term
- if contention will become visibile, we will be doing a lot of pps, which is good ;). Additionally, in the current code, we already have contention on the msk socket lock via trylock.
Quoted 12 lines
Quoted 11 lines
Given the above we can always move data into the msk receive queue (and OoO queue) from the subflow.
We leverage the previous commit, so that we need to acquire the spinlock in the tx path only when moving fwd memory to wfwd and vice versa.
recvmsg()must now explicitly acquire the socket spinlock when moving skbs out ofsk_receive_queue. We use a snd rx queue and splice the first into the latter to reduce the number locks required.
Quoted 8 lines
To see if I understood this (and the code) correctly:
For the msk,
sk_receive_queueis protected only by themptcp_data_lock()and is where MPTCP-level data is reassembled in-order without holding the msk socket lock.
msk->receive_queueis protected by the socket lock, and is where in-order skbs are moved so they can be copied to userspace.
Yes, the above is correct. The introduction of msk->receive_queue allows skipping one mptcp_data_lock() in recvmsg() for bulk transfer.
There is a possible follow-up improvement:
Open code lock_sock()/release_sock(), so that we can do mptcp-specific operations - e.g. splicing receive_queue, or fwd allocating memory -
while acquiring the msk socket lock spinlock inside such helper. That
will avoid up to 2 additional lock operations per recvmsg()/sendmsg().
/P
Paolo Abeni
PATCH 3/6 mptcp: allocate TX skbs in msk context.
In reply to Paolo AbeniMove the TX skbs allocation in mptcp_sendmsg() scope, and tentativelly allocate a skbs based on the goal size on the last used subflow.
Use the ssk tx skb cache to prevent the subflow allocation.
This allows removing the msk skb extension cache and will make possible the later patch.
---
RFC v3 -> v1:
- fixed some allocation bugs
--- net/mptcp/protocol.c | 193 +++++++++++++++++++++++++++++++++++-------- net/mptcp/protocol.h | 4 +- 2 files changed, 160 insertions(+), 37 deletions(-)
net/mptcp/protocol.c +157 -36
@@ -784,16 +784,6 @@ static void mptcp_check_for_eof(struct mptcp_sock *msk)inet_sk_state_store (sk,TCP_CLOSE ); } -static bool mptcp_ext_cache_refill (struct mptcp_sock *msk) -{ -const struct sock *sk = (const struct sock *)msk; - -if (!msk->cached_ext) - msk->cached_ext =__skb_ext_alloc (sk->sk_allocation); - -return !!msk->cached_ext; -} -static struct sock *mptcp_subflow_recv_lookup (const struct mptcp_sock *msk) {struct mptcp_subflow_context *subflow;
@@ -957,19 +947,12 @@ static void mptcp_clean_una_wakeup(struct sock *sk) } } -/* ensure we get enough memory for the frag hdr, beyond some minimal amount of -* data -*/ -static bool mptcp_page_frag_refill (struct sock *sk,struct page_frag *pfrag) +static void mptcp_enter_memory_pressure (struct sock *sk) {struct mptcp_subflow_context *subflow;struct mptcp_sock *msk =mptcp_sk (sk);bool first = true; -if (likely (skb_page_frag_refill (32U +sizeof (struct mptcp_data_frag ), - pfrag, sk->sk_allocation))) -return true; -sk_stream_moderate_sndbuf (sk);mptcp_for_each_subflow (msk, subflow) {struct sock *ssk =mptcp_subflow_tcp_sock (subflow);
@@ -979,6 +962,18 @@ static bool mptcp_page_frag_refill(struct sock *sk, struct page_frag *pfrag)sk_stream_moderate_sndbuf (ssk); first = false; } +} + +/* ensure we get enough memory for the frag hdr, beyond some minimal amount of +* data +*/ +static bool mptcp_page_frag_refill (struct sock *sk,struct page_frag *pfrag) +{ +if (likely (skb_page_frag_refill (32U +sizeof (struct mptcp_data_frag ), + pfrag, sk->sk_allocation))) +return true; + +mptcp_enter_memory_pressure (sk);return false; }
@@ -1025,6 +1020,116 @@ static int mptcp_check_allowed_size(struct mptcp_sock *msk, u64 data_seq,return avail_size; } +static bool __mptcp_add_ext (struct sk_buff *skb,gfp_t gfp) +{ +struct skb_ext *mpext =__skb_ext_alloc (gfp); + +if (!mpext) +return false; +__skb_ext_set (skb,SKB_EXT_MPTCP , mpext); +return true; +} + +static struct sk_buff *__mptcp_do_alloc_tx_skb (struct sock *sk) +{ +struct sk_buff *skb; + + skb =alloc_skb_fclone (MAX_TCP_HEADER , sk->sk_allocation); +if (likely (skb)) { +if (likely (__mptcp_add_ext (skb, sk->sk_allocation))) { +skb_reserve (skb,MAX_TCP_HEADER ); + skb->reserved_tailroom = skb->end - skb->tail; +return skb; + } +__kfree_skb (skb); + }else { +mptcp_enter_memory_pressure (sk); + } +return NULL ; +} + +static bool mptcp_tx_cache_refill (struct sock *sk,int size) +{ +struct mptcp_sock *msk =mptcp_sk (sk); +struct sk_buff *skb; +int space_needed; + +if (unlikely (tcp_under_memory_pressure (sk))) +mptcp_mem_reclaim_partial (sk); + + space_needed = msk->tx_pending_data + size - + msk->skb_tx_cache.qlen * msk->size_goal_cache; +while (space_needed >0 ) { + skb =__mptcp_do_alloc_tx_skb (sk); +if (!skb) +return false; + +if (!mptcp_wmem_alloc (sk, skb->truesize)) { +__kfree_skb (skb); +return false; + } + +__skb_queue_tail (&msk->skb_tx_cache, skb); + space_needed -= msk->size_goal_cache; + } +return true; +} + +static bool __mptcp_alloc_tx_skb (struct sock *sk,struct sock *ssk) +{ +struct mptcp_sock *msk =mptcp_sk (sk); +struct sk_buff *skb; + +if (ssk->sk_tx_skb_cache) { + skb = ssk->sk_tx_skb_cache; +if (unlikely (!skb_ext_find (skb,SKB_EXT_MPTCP ) && + !__mptcp_add_ext (skb, sk->sk_allocation))) +return false; +return true; + } + + skb =skb_peek (&msk->skb_tx_cache); +if (skb) { +if (likely (sk_wmem_schedule (ssk, skb->truesize))) { + skb =__skb_dequeue (&msk->skb_tx_cache); +if (WARN_ON_ONCE (!skb)) +return false; + +mptcp_wmem_uncharge (sk, skb->truesize); + ssk->sk_tx_skb_cache = skb; +return true; + } + +/* over memory limit, no point to try to allocate a new skb */ +return false; + } + + skb =__mptcp_do_alloc_tx_skb (sk); +if (!skb) +return false; + +if (likely (sk_wmem_schedule (ssk, skb->truesize))) { + ssk->sk_tx_skb_cache = skb; +return true; + } +kfree_skb (skb); +return false; +} + +static bool mptcp_must_reclaim_memory (struct sock *sk,struct sock *ssk) +{ +return !ssk->sk_tx_skb_cache && + !skb_peek (&mptcp_sk (sk)->skb_tx_cache) && +tcp_under_memory_pressure (sk); +} + +static bool mptcp_alloc_tx_skb (struct sock *sk,struct sock *ssk) +{ +if (unlikely (mptcp_must_reclaim_memory (sk, ssk))) +mptcp_mem_reclaim_partial (sk); +return __mptcp_alloc_tx_skb (sk, ssk); +} +static int mptcp_sendmsg_frag (struct sock *sk,struct sock *ssk,struct mptcp_data_frag *dfrag,struct mptcp_sendmsg_info *info)
@@ -1036,7 +1141,7 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,struct sk_buff *skb, *tail;bool can_collapse = false;int avail_size; -size_t ret; +size_t ret =0 ;pr_debug ("msk=%p ssk=%p sending dfrag at seq=%lld len=%d already sent=%d" , msk, ssk, dfrag->data_seq, dfrag->data_len, info->sent);
@@ -1044,6 +1149,7 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,/* compute send limit */ info->mss_now =tcp_send_mss (ssk, &info->size_goal, info->flags); avail_size = info->size_goal; + msk->size_goal_cache = info->size_goal; skb =tcp_write_queue_tail (ssk);if (skb) {/* Limit the write to the size available in the
@@ -1092,8 +1198,11 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk, goto out; } - mpext =__skb_ext_set (tail,SKB_EXT_MPTCP , msk->cached_ext); - msk->cached_ext =NULL ; + mpext =skb_ext_find (tail,SKB_EXT_MPTCP ); +if (WARN_ON_ONCE (!mpext)) { +/* should never reach here, stream corrupted */ +return -EINVAL ; + }memset (mpext,0 ,sizeof (*mpext)); mpext->data_seq = data_seq;
@@ -1113,6 +1222,7 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,tcp_push_pending_frames (ssk); } out: + msk->tx_pending_data -= ret;mptcp_subflow_ctx (ssk)->rel_write_seq += ret;return ret; }
@@ -1166,9 +1276,6 @@ static struct sock *mptcp_subflow_get_send(struct mptcp_sock *msk,sock_owned_by_me ((struct sock *)msk); *sndbuf =0 ; -if (!mptcp_ext_cache_refill (msk)) -return NULL ; -if (__mptcp_check_fallback (msk)) {if (!msk->first)return NULL ;
@@ -1277,6 +1384,15 @@ static void mptcp_push_pending(struct sock *sk, unsigned int flags)if (ssk != prev_ssk || !prev_ssk)lock_sock (ssk); +/* keep it simple and always provide a new skb for the +* subflow, even if we will not use it when collapsing +* on the pending one +*/ +if (!mptcp_alloc_tx_skb (sk, ssk)) { +mptcp_push_release (sk, ssk, &info); + goto out; + } + ret =mptcp_sendmsg_frag (sk, ssk, dfrag, &info);if (ret <=0 ) {mptcp_push_release (sk, ssk, &info);
@@ -1365,6 +1481,9 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) offset = dfrag->offset + dfrag->data_len; psize = pfrag->size - offset; psize =min_t (size_t, psize,msg_data_left (msg)); +if (!mptcp_tx_cache_refill (sk, psize)) + goto wait_for_memory; +if (!mptcp_wmem_alloc (sk, psize + frag_truesize)) goto wait_for_memory;
@@ -1396,8 +1515,6 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) dfrag->data_seq, dfrag->data_len, dfrag->already_sent, !dfrag_collapsed); -if (!mptcp_ext_cache_refill (msk)) - goto wait_for_memory;continue ; wait_for_memory:
@@ -1409,8 +1526,10 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) goto out; } -if (copied) +if (copied) { + msk->tx_pending_data += copied;mptcp_push_pending (sk, msg->msg_flags); + } out:release_sock (sk);
@@ -2001,9 +2120,6 @@ static void mptcp_worker(struct work_struct *work)if (!dfrag) goto unlock; -if (!mptcp_ext_cache_refill (msk)) - goto reset_unlock; - ssk =mptcp_subflow_get_retrans (msk);if (!ssk) goto reset_unlock;
@@ -2014,6 +2130,9 @@ static void mptcp_worker(struct work_struct *work) info.sent =0 ; info.limit = dfrag->already_sent;while (info.sent < dfrag->already_sent) { +if (!mptcp_alloc_tx_skb (sk, ssk)) +break ; + ret =mptcp_sendmsg_frag (sk, ssk, dfrag, &info);if (ret <=0 )break ;
@@ -2021,9 +2140,6 @@ static void mptcp_worker(struct work_struct *work)MPTCP_INC_STATS (sock_net (sk),MPTCP_MIB_RETRANSSEGS ); copied += ret; info.sent += ret; - -if (!mptcp_ext_cache_refill (msk)) -break ; }if (copied) tcp_push(ssk, 0, info.mss_now, tcp_sk(ssk)->nonagle,
@@ -2053,9 +2169,12 @@ static int __mptcp_init_sock(struct sock *sk)INIT_LIST_HEAD (&msk->rtx_queue);INIT_WORK (&msk->work, mptcp_worker);__skb_queue_head_init (&msk->receive_queue); +__skb_queue_head_init (&msk->skb_tx_cache); msk->out_of_order_queue =RB_ROOT ; msk->first_pending =NULL ; msk->wforward_alloc =0 ; + msk->tx_pending_data =0 ; + msk->size_goal_cache =TCP_BASE_MSS ; msk->ack_hint =NULL ; msk->first =NULL ;
@@ -2103,12 +2222,17 @@ static void __mptcp_clear_xmit(struct sock *sk) {struct mptcp_sock *msk =mptcp_sk (sk);struct mptcp_data_frag *dtmp, *dfrag; +struct sk_buff *skb;sk_stop_timer (sk, &msk->sk.icsk_retransmit_timer);WRITE_ONCE (msk->first_pending,NULL );list_for_each_entry_safe (dfrag, dtmp, &msk->rtx_queue, list)dfrag_clear (sk, dfrag); +while ((skb =__skb_dequeue (&msk->skb_tx_cache)) !=NULL ) { +mptcp_wmem_uncharge (sk, skb->truesize); +kfree_skb (skb); + } }static void mptcp_cancel_work (struct sock *sk)
@@ -2491,9 +2615,6 @@ static void mptcp_destroy(struct sock *sk) {struct mptcp_sock *msk =mptcp_sk (sk); -if (msk->cached_ext) -__skb_ext_put (msk->cached_ext); -mptcp_destroy_common (msk);sk_sockets_allocated_dec (sk); }
net/mptcp/protocol.h +3 -1
@@ -240,11 +240,13 @@ struct mptcp_sock {struct sk_buff *ooo_last_skb;struct rb_root out_of_order_queue;struct sk_buff_head receive_queue; +struct sk_buff_head skb_tx_cache;/* this is wmem accounted */ +int tx_pending_data; +int size_goal_cache;struct list_head conn_list;struct list_head rtx_queue;struct mptcp_data_frag *first_pending;struct list_head join_list; -struct skb_ext *cached_ext;/* for the next sendmsg */ struct socket *subflow;/* outgoing connect/listener/!mp_capable */ struct sock *first;struct mptcp_pm_data pm;
Signature · 1 line
2.26.2
Paolo Abeni
PATCH 4/6 mptcp: avoid a few atomic ops in the rx path
In reply to Paolo AbeniExtending the data_lock scope in mptcp_incoming_option we can use that to protect both snd_una and wnd_end. In the typical case we will have a single atomic op instead of 2
--- net/mptcp/mptcp_diag.c | 2 +- net/mptcp/options.c | 33 +++++++++++++-------------------- net/mptcp/protocol.c | 33 +++++++++++++++------------------ net/mptcp/protocol.h | 8 ++++---- 4 files changed, 33 insertions(+), 43 deletions(-)
net/mptcp/mptcp_diag.c +1 -1
@@ -140,7 +140,7 @@ static void mptcp_diag_get_info(struct sock *sk, struct inet_diag_msg *r, info->mptcpi_flags = flags; info->mptcpi_token =READ_ONCE (msk->token); info->mptcpi_write_seq =READ_ONCE (msk->write_seq); - info->mptcpi_snd_una =atomic64_read (&msk->snd_una); + info->mptcpi_snd_una =READ_ONCE (msk->snd_una); info->mptcpi_rcv_nxt =READ_ONCE (msk->ack_seq);unlock_sock_fast (sk, slow); }
net/mptcp/options.c +13 -20
@@ -833,15 +833,17 @@ static void ack_update_msk(struct mptcp_sock *msk,const struct sock *ssk, struct mptcp_options_received *mp_opt) { - u64 new_snd_una, snd_una, old_snd_una = atomic64_read(&msk->snd_una); -u64 new_wnd_end, wnd_end, old_wnd_end =atomic64_read (&msk->wnd_end); -u64 snd_nxt =READ_ONCE (msk->snd_nxt); + u64 new_wnd_end, new_snd_una, snd_nxt = READ_ONCE(msk->snd_nxt);struct sock *sk = (struct sock *)msk; +u64 old_snd_una; + +mptcp_data_lock (sk);/* avoid ack expansion on update conflict, to reduce the risk of * wrongly expanding to a future ack sequence number, which is way * more dangerous than missing an ack */ + old_snd_una = msk->snd_una; new_snd_una =expand_ack (old_snd_una, mp_opt->data_ack, mp_opt->ack64);/* ACK for data not even sent yet? Ignore. */
@@ -850,26 +852,17 @@ static void ack_update_msk(struct mptcp_sock *msk, new_wnd_end = new_snd_una +tcp_sk (ssk)->snd_wnd; -while (after64 (new_wnd_end, old_wnd_end)) { - wnd_end = old_wnd_end; - old_wnd_end =atomic64_cmpxchg (&msk->wnd_end, wnd_end, - new_wnd_end); -if (old_wnd_end == wnd_end) { -if (mptcp_send_head (sk)) -mptcp_schedule_work (sk); -break ; - } +if (after64 (new_wnd_end, msk->wnd_end)) { + msk->wnd_end = new_wnd_end; +if (mptcp_send_head (sk)) +mptcp_schedule_work (sk); } -while (after64 (new_snd_una, old_snd_una)) { - snd_una = old_snd_una; - old_snd_una =atomic64_cmpxchg (&msk->snd_una, snd_una, - new_snd_una); -if (old_snd_una == snd_una) { -mptcp_data_acked (sk); -break ; - } +if (after64 (new_snd_una, old_snd_una)) { + msk->snd_una = new_snd_una; +__mptcp_data_acked (sk); } +mptcp_data_unlock (sk); }bool mptcp_update_rcv_data_fin (struct mptcp_sock *msk,u64 data_fin_seq,bool use_64bit)
net/mptcp/protocol.c +15 -18
@@ -60,7 +60,7 @@ static struct socket *__mptcp_nmpc_socket(const struct mptcp_sock *msk)/* Returns end sequence number of the receiver's advertised window */ static u64 mptcp_wnd_end (const struct mptcp_sock *msk) { -return atomic64_read (&msk->wnd_end); +return READ_ONCE (msk->wnd_end); }static bool mptcp_is_tcpsk (struct sock *sk)
@@ -347,7 +347,7 @@ static void mptcp_check_data_fin_ack(struct sock *sk)/* Look for an acknowledged DATA_FIN */ if (((1 << sk->sk_state) & (TCPF_FIN_WAIT1 |TCPF_CLOSING |TCPF_LAST_ACK )) && - msk->write_seq ==atomic64_read (&msk->snd_una)) { + msk->write_seq ==READ_ONCE (msk->snd_una)) {mptcp_stop_timer (sk);WRITE_ONCE (msk->snd_data_fin_enable,0 );
@@ -743,7 +743,7 @@ bool mptcp_schedule_work(struct sock *sk)return false; } -void mptcp_data_acked (struct sock *sk) +void __mptcp_data_acked (struct sock *sk) {mptcp_reset_timer (sk);
@@ -899,10 +899,9 @@ static void mptcp_clean_una(struct sock *sk)* plain TCP */ if (__mptcp_check_fallback (msk)) -atomic64_set (&msk->snd_una, msk->snd_nxt); - - snd_una =atomic64_read (&msk->snd_una); + msk->snd_una =READ_ONCE (msk->snd_nxt); + snd_una = msk->snd_una;list_for_each_entry_safe (dfrag, dtmp, &msk->rtx_queue, list) {if (after64 (dfrag->data_seq + dfrag->data_len, snd_una))break ;
@@ -1170,10 +1169,12 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,/* Zero window and all data acked? Probe. */ avail_size =mptcp_check_allowed_size (msk, data_seq, avail_size);if (avail_size ==0 ) { -if (skb ||atomic64_read (&msk->snd_una) != msk->snd_nxt) +u64 snd_una =READ_ONCE (msk->snd_una); + +if (skb || snd_una != msk->snd_nxt)return 0 ; zero_window_probe = true; - data_seq =atomic64_read (&msk->snd_una) -1 ; + data_seq = snd_una -1 ; avail_size =1 ; }
@@ -1878,12 +1879,8 @@ static void mptcp_retransmit_handler(struct sock *sk) {struct mptcp_sock *msk =mptcp_sk (sk); -if (atomic64_read (&msk->snd_una) ==READ_ONCE (msk->snd_nxt)) { -mptcp_stop_timer (sk); - }else { -set_bit (MPTCP_WORK_RTX , &msk->flags); -mptcp_schedule_work (sk); - } +set_bit (MPTCP_WORK_RTX , &msk->flags); +mptcp_schedule_work (sk); }static void mptcp_retransmit_timer (struct timer_list *t)
@@ -2506,8 +2503,8 @@ struct sock *mptcp_sk_clone(const struct sock *sk, msk->write_seq = subflow_req->idsn +1 ; msk->snd_nxt = msk->write_seq; -atomic64_set (&msk->snd_una, msk->write_seq); -atomic64_set (&msk->wnd_end, msk->snd_nxt + req->rsk_rcv_wnd); + msk->snd_una = msk->write_seq; + msk->wnd_end = msk->snd_nxt + req->rsk_rcv_wnd;if (mp_opt->mp_capable) { msk->can_ack = true;
@@ -2545,7 +2542,7 @@ void mptcp_rcv_space_init(struct mptcp_sock *msk, const struct sock *ssk)if (msk->rcvq_space.space ==0 ) msk->rcvq_space.space =TCP_INIT_CWND *TCP_MSS_DEFAULT ; -atomic64_set (&msk->wnd_end, msk->snd_nxt +tcp_sk (ssk)->snd_wnd); +WRITE_ONCE (msk->wnd_end, msk->snd_nxt +tcp_sk (ssk)->snd_wnd); } static struct sock *mptcp_accept(struct sock *sk, int flags, int *err,
@@ -2808,7 +2805,7 @@ void mptcp_finish_connect(struct sock *ssk)WRITE_ONCE (msk->ack_seq, ack_seq);WRITE_ONCE (msk->rcv_wnd_sent, ack_seq);WRITE_ONCE (msk->can_ack,1 ); -atomic64_set (&msk->snd_una, msk->write_seq); +WRITE_ONCE (msk->snd_una, msk->write_seq);mptcp_pm_new_connection (msk,0 );
net/mptcp/protocol.h +4 -4
@@ -223,8 +223,8 @@ struct mptcp_sock {struct sock *last_snd;int snd_burst;int old_wspace; -atomic64_t snd_una; -atomic64_t wnd_end; +u64 snd_una; +u64 wnd_end;unsigned long timer_ival;u32 token;int rmem_pending;
@@ -308,7 +308,7 @@ static inline struct mptcp_data_frag *mptcp_rtx_tail(const struct sock *sk) {struct mptcp_sock *msk =mptcp_sk (sk); -if (!before64 (msk->snd_nxt,atomic64_read (&msk->snd_una))) +if (!before64 (msk->snd_nxt,READ_ONCE (msk->snd_una)))return NULL ; return list_last_entry(&msk->rtx_queue, struct mptcp_data_frag, list);
@@ -482,7 +482,7 @@ void mptcp_rcv_space_init(struct mptcp_sock *msk, const struct sock *ssk);void mptcp_data_ready (struct sock *sk,struct sock *ssk);bool mptcp_finish_join (struct sock *sk);bool mptcp_schedule_work (struct sock *sk); -void mptcp_data_acked (struct sock *sk); +void __mptcp_data_acked (struct sock *sk);void mptcp_subflow_eof (struct sock *sk);bool mptcp_update_rcv_data_fin (struct mptcp_sock *msk,u64 data_fin_seq,bool use_64bit);void __mptcp_flush_join_list (struct mptcp_sock *msk);
Signature · 1 line
2.26.2
Paolo Abeni
PATCH 5/6 mptcp: simplify mptcp_nospace
In reply to Paolo AbeniCurrently mptcp_space has to traverse the whole conn_list setting the relevant bit on each subflow, because the client ones will have a different sk_socket.
We can actually leave the NOSPACE always set on all client subflow and make nospace simpler.
--- net/mptcp/protocol.c | 15 +-------------- net/mptcp/subflow.c | 31 ++++++++++++++++++++++--------- 2 files changed, 23 insertions(+), 23 deletions(-)
net/mptcp/protocol.c +1 -14
@@ -1230,23 +1230,9 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,static void mptcp_nospace (struct mptcp_sock *msk) { -struct mptcp_subflow_context *subflow; -set_bit (MPTCP_NOSPACE , &msk->flags);smp_mb__after_atomic ();/* msk->flags is changed by write_space cb */ -mptcp_for_each_subflow (msk, subflow) { -struct sock *ssk =mptcp_subflow_tcp_sock (subflow); -bool ssk_writeable =sk_stream_is_writeable (ssk); -struct socket *sock =READ_ONCE (ssk->sk_socket); - -if (ssk_writeable || !sock) -continue ; - -/* enables ssk->write_space() callbacks */ -set_bit (SOCK_NOSPACE , &sock->flags); - } -/* mptcp_data_acked() could run just before we set the NOSPACE bit, * so explicitly check for snd_una value */
@@ -3036,6 +3022,7 @@ static int mptcp_stream_accept(struct socket *sock, struct socket *newsock, slowpath =lock_sock_fast (newsk);mptcp_copy_inaddrs (newsk, msk->first);mptcp_rcv_space_init (msk, msk->first); +set_bit (SOCK_NOSPACE , &newsock->flags);/* set ssk->sk_socket of accept()ed flows to mptcp socket. * This is needed so NOSPACE flag can be set from tcp stack.
net/mptcp/subflow.c +22 -9
@@ -996,19 +996,28 @@ static void subflow_data_ready(struct sock *sk)mptcp_data_ready (parent, sk); } -static void subflow_write_space (struct sock *sk) +static void subflow_write_space (struct sock *ssk) { -struct mptcp_subflow_context *subflow =mptcp_subflow_ctx (sk); -struct socket *sock =READ_ONCE (sk->sk_socket); -struct sock *parent = subflow->conn; +struct mptcp_subflow_context *subflow =mptcp_subflow_ctx (ssk); +struct sock *sk = subflow->conn; +struct socket_wq *wq; -if (!sk_stream_is_writeable (sk)) +if (!sk_stream_is_writeable (ssk) || !sk_stream_is_writeable (sk) || + !sk->sk_socket || !test_bit (MPTCP_NOSPACE , &mptcp_sk (sk)->flags))return ; -if (sock &&sk_stream_is_writeable (parent)) -clear_bit (SOCK_NOSPACE , &sock->flags); - -sk_stream_write_space (parent); +/* The following is quite alike sk_stream_write_space, but avoids +* clearing the sk SOCK_NOSPACE bit +*/ +clear_bit (MPTCP_NOSPACE , &mptcp_sk (sk)->flags); +rcu_read_lock (); + wq =rcu_dereference (sk->sk_wq); +if (skwq_has_sleeper (wq)) +wake_up_interruptible_poll (&wq->wait,EPOLLOUT | +EPOLLWRNORM |EPOLLWRBAND ); +if (wq && wq->fasync_list && !(sk->sk_shutdown &SEND_SHUTDOWN )) +sock_wake_async (wq,SOCK_WAKE_SPACE ,POLL_OUT ); +rcu_read_unlock (); } static struct inet_connection_sock_af_ops *
@@ -1208,6 +1217,10 @@ int mptcp_subflow_create_socket(struct sock *sk, struct socket **new_sock)SOCK_INODE (sf)->i_uid =SOCK_INODE (sk->sk_socket)->i_uid;SOCK_INODE (sf)->i_gid =SOCK_INODE (sk->sk_socket)->i_gid; +/* subflows will always call into sk_write_space, and subflow_write_space() +* will be responsible of doing the actual wake-up +*/ +set_bit (SOCK_NOSPACE , &sf->flags); subflow =mptcp_subflow_ctx (sf->sk);pr_debug ("subflow=%p" , subflow);
Signature · 1 line
2.26.2
Paolo Abeni
PATCH 6/6 mptcp: use mptcp backlog.
In reply to Paolo AbeniWe have some tasks triggered by the subflow receive path which require to access the msk socket status, specifically:mptcp_clean_una() and mptcp_push_pending()
We have almost everything in place to use the msk backlog and a standard !owned by user schema for that.
Introduce a dummy per msk sk_buff to trigger into the backlog as needed.
Since the worker is no more used to clean the acked data, for fallback sockets we need to explicitly call flush them.
As added bonus we can simplify a lot mptcp_poll() and move the timer update under the data lock
Signed-off-by: Paolo Abeni <pabeni(a)redhat.com> --
rfc v3-> v2:
- dfrag_release is always invoked under sock spin lock, must use mem_release
- move the mem_reclaim in mptcp_clean_una
- cleanup poll, leveraging the previous patch
--- net/mptcp/options.c | 18 +++-- net/mptcp/protocol.c | 183 +++++++++++++++++++++++++++---------------- net/mptcp/protocol.h | 8 ++ 3 files changed, 134 insertions(+), 75 deletions(-)
net/mptcp/options.c +12 -6
@@ -189,7 +189,7 @@ static void mptcp_parse_option(const struct sk_buff *skb, ptr +=4 ; } -pr_debug ("data_ack=%llu" , mp_opt->data_ack); +pr_debug ("64=%d data_ack=%llx" , mp_opt->ack64, mp_opt->data_ack); }if (mp_opt->use_map) {
@@ -207,7 +207,7 @@ static void mptcp_parse_option(const struct sk_buff *skb, mp_opt->data_len =get_unaligned_be16 (ptr); ptr +=2 ; -pr_debug ("data_seq=%llu subflow_seq=%u data_len=%u" , +pr_debug ("data_seq=%llx subflow_seq=%u data_len=%u" , mp_opt->data_seq, mp_opt->subflow_seq, mp_opt->data_len); }
@@ -830,7 +830,7 @@ static u64 expand_ack(u64 old_ack, u64 cur_ack, bool use_64bit) }static void ack_update_msk (struct mptcp_sock *msk, -const struct sock *ssk, +struct sock *ssk,struct mptcp_options_received *mp_opt) {u64 new_wnd_end, new_snd_una, snd_nxt =READ_ONCE (msk->snd_nxt);
@@ -854,8 +854,10 @@ static void ack_update_msk(struct mptcp_sock *msk,if (after64 (new_wnd_end, msk->wnd_end)) { msk->wnd_end = new_wnd_end; -if (mptcp_send_head (sk)) -mptcp_schedule_work (sk); +if (!sock_owned_by_user (sk)) +__mptcp_subflow_push_pending (sk, ssk); +else +__mptcp_add_to_backlog (sk,MPTCP_PUSH_PENDING ); }if (after64 (new_snd_una, old_snd_una)) {
@@ -915,8 +917,12 @@ void mptcp_incoming_options(struct sock *sk, struct sk_buff *skb)struct mptcp_options_received mp_opt;struct mptcp_ext *mpext; -if (__mptcp_check_fallback (msk)) +if (__mptcp_check_fallback (msk)) { +mptcp_data_lock (subflow->conn); +__mptcp_data_acked (subflow->conn); +mptcp_data_unlock (subflow->conn);return ; + }mptcp_get_options (skb, &mp_opt);if (!check_fully_established (msk, sk, subflow, skb, &mp_opt))
net/mptcp/protocol.c +114 -69
335 lines not on this page · raw message
net/mptcp/protocol.h +8 -0
@@ -92,6 +92,8 @@#define MPTCP_FALLBACK_DONE 4#define MPTCP_WORK_CLOSE_SUBFLOW 5#define MPTCP_WORKER_RUNNING 6 +#define MPTCP_PUSH_PENDING 7 +#define MPTCP_CLEAN_UNA 8static inline bool before64 (__u64 seq1,__u64 seq2) {
@@ -249,6 +251,9 @@ struct mptcp_sock {struct list_head join_list;struct socket *subflow;/* outgoing connect/listener/!mp_capable */ struct sock *first; + +struct sk_buff backlog; +struct mptcp_pm_data pm;struct {u32 space;/* bytes copied in last measurement window */
@@ -440,6 +445,9 @@ unsigned int mptcp_get_add_addr_timeout(struct net *net);void mptcp_subflow_fully_established (struct mptcp_subflow_context *subflow,struct mptcp_options_received *mp_opt);bool mptcp_subflow_data_available (struct sock *sk); +void mptcp_push_pending (struct sock *sk,unsigned int flags); +void __mptcp_subflow_push_pending (struct sock *sk,struct sock *ssk); +void __mptcp_add_to_backlog (struct sock *sk,int event);void __init mptcp_subflow_init (void );void mptcp_subflow_shutdown (struct sock *sk,struct sock *ssk,int how); void __mptcp_close_ssk(struct sock *sk, struct sock *ssk,
Signature · 1 line
2.26.2
Davide Caratti
[MPTCP] Re: [PATCH net-next 6/6] mptcp: use mptcp backlog.
hello,
Quoted 34 lines · Paolo Abeni
We have some tasks triggered by the subflow receive path which require to access the msk socket status, specifically:
mptcp_clean_una()andmptcp_push_pending()We have almost everything in place to use the msk backlog and a standard !owned by user schema for that.
Introduce a dummy per msk
sk_buffto trigger into the backlog as needed.Since the worker is no more used to clean the acked data, for fallback sockets we need to explicitly call flush them.
As added bonus we can simplify a lot
mptcp_poll()and move the timer update under the data lockSigned-off-by: Paolo Abeni <pabeni(a)redhat.com> --
rfc v3-> v2:
-dfrag_releaseis always invoked under sock spin lock, must usemem_release
- move themem_reclaiminmptcp_clean_una
- cleanup poll, leveraging the previous patch
---net/mptcp/options.c | 18 +++-- net/mptcp/protocol.c | 183 +++++++++++++++++++++++++++---------------- net/mptcp/protocol.h | 8 ++ 3 files changed, 134 insertions(+), 75 deletions(-)net/mptcp/options.c +0
-0
net/mptcp/options.c static void mptcp_parse_option(const struct sk_buff *skb, · line 192
Quoted 5 lines
net/mptcp/options.c +1
-1@@ -189,7 +189,7 @@ static void mptcp_parse_option(const struct sk_buff *skb, ptr += 4; } -pr_debug ("data_ack=%llu" , mp_opt->data_ack); +pr_debug ("64=%d data_ack=%llx" , mp_opt->ack64, mp_opt->data_ack);
why are you changing (here and in hunks below) the format specifier of DACKs from decimal to hex format? tcpdump and wireshark print it as decimal, and we use to compare it with TCP sequence numbers, that are in decimal format too: I'd prefer to keep it %llu.
WDYT?
Signature · 1 line
davide
Paolo Abeni
[MPTCP] Re: [PATCH net-next 6/6] mptcp: use mptcp backlog.
Quoted 40 lines · Davide Caratti
hello,
Quoted 37 lines · Paolo Abeni
We have some tasks triggered by the subflow receive path which require to access the msk socket status, specifically:
mptcp_clean_una()andmptcp_push_pending()We have almost everything in place to use the msk backlog and a standard !owned by user schema for that.
Introduce a dummy per msk
sk_buffto trigger into the backlog as needed.Since the worker is no more used to clean the acked data, for fallback sockets we need to explicitly call flush them.
As added bonus we can simplify a lot
mptcp_poll()and move the timer update under the data lockSigned-off-by: Paolo Abeni <pabeni(a)redhat.com> --
rfc v3-> v2:
-dfrag_releaseis always invoked under sock spin lock, must usemem_release
- move themem_reclaiminmptcp_clean_una
- cleanup poll, leveraging the previous patch--- net/mptcp/options.c | 18 +++-- net/mptcp/protocol.c | 183 +++++++++++++++++++++++++++---------------- net/mptcp/protocol.h | 8 ++ 3 files changed, 134 insertions(+), 75 deletions(-)net/mptcp/options.c +0
-0@@ -189,7 +189,7 @@ static void mptcp_parse_option(const struct sk_buff *skb, ptr +=4 ; }
Quoted 5 lines
net/mptcp/options.c static void mptcp_parse_option(const struct sk_buff *skb, · line 192
Quoted 2 lines
net/mptcp/options.c +1
-1-pr_debug ("data_ack=%llu" , mp_opt->data_ack); +pr_debug ("64=%d data_ack=%llx" , mp_opt->ack64, mp_opt->data_ack);why are you changing (here and in hunks below) the format specifier of DACKs from decimal to hex format?
whoops, this is an unneeded/unindended change that slipped in - it's there as debug hint, fits better my personal preference/setup.
Will be dropped in later iteration.
Cheers,
Paolo
Mat Martineau
Quoted 10 lines · Paolo Abeni
This series refactor once again the data path, with the goal of avoding the worker usage for the fast path.
The worker will still be used to:
- cleanup closed subflows
- handle timeout
- retransmissions
- send the DATA_FIN/DATA_FIN ack.
- fallback eof
Quoted 2 lines
The latters 2 use case could be fourther removed, if we accept/consider sending DATA_FIN/ack on a single subflow.
Sending DATA_FIN/ack on a single subflow looks ok to me in this case. It would make sense to send that ack on the active subflow where incoming data is being processed. On the workqueue that we didn't know which subflow would be best to use, and didn't want to pick an inactive and possibly-not-working subflow.
Mat
Quoted 29 lines
The code is still pretty much unstable - survives very basic tests and don't complete yet a single self-test iteration, anyhow sharing early to discuss a few design points.
- separate memory accunting for write memory in patch 1 is a required pre-req for patch 2, but also brings a bit of additional complexity in the following patches. I would be very happy to find any other alternative.
- in patch 3 we may end-up pre-allocating more-than-strictly-required skbs. Doing more accurate estimate would be nice, but I don't know how without adding even more complexity.
Pre-allocating skbs is quite necessary, otherwise under memory pressure we may end-up copying data to kernel (in the msk write queue) but not being able later to allocate the required skbs atpush_pending()time and xmit will be stuck.- I guess patch 6 could cause quite hard reaction upstream, if they will dare look inside. I like it ;)
Paolo Abeni (6):
mptcp: separate accouting for wmem forward alloc mem
mptcp: protect the rx path with the msk socket spinlock
mptcp: allocate TX skbs in msk context.
mptcp: avoid a few atomic ops in the rx path
mptcp: simplifymptcp_nospace
mptcp: use mptcp backlog.
Quoted 9 lines
net/mptcp/mptcp_diag.c | 2 +- net/mptcp/options.c | 47 ++-- net/mptcp/protocol.c | 606 +++++++++++++++++++++++++++++------------ net/mptcp/protocol.h | 25 +- net/mptcp/subflow.c | 31 ++- 5 files changed, 496 insertions(+), 215 deletions(-)Signature · 1 line
2.26.2
--
Mat Martineau
Intel
Paolo Abeni
Quoted 12 lines · Mat Martineau
Quoted 10 lines · Paolo Abeni
This series refactor once again the data path, with the goal of avoding the worker usage for the fast path.
The worker will still be used to:
- cleanup closed subflows
- handle timeout
- retransmissions
- send the DATA_FIN/DATA_FIN ack.
- fallback eof
Quoted 4 lines
Quoted 2 lines
The latters 2 use case could be fourther removed, if we accept/consider sending DATA_FIN/ack on a single subflow.
Sending DATA_FIN/ack on a single subflow looks ok to me in this case.
Excellent! does that applies to the data_fin, too?
Quoted 5 lines
It
would make sense to send that ack on the active subflow where incoming data is being processed. On the workqueue that we didn't know which subflow would be best to use, and didn't want to pick an inactive and possibly-not-working subflow.
If we can send the data_fin on the subflow reaching the end of sequence, we will not need anymore the workqueue to send the data_fin. And the same for the data fin ack.
We will need the workqueue only for shutting down the subflows after receiving the data_fin ack.
Cheers,
Paolo
[sorry, this mail got stuck in my mailbox the whole day]
Mat Martineau
Quoted 13 lines · Paolo Abeni
Quoted 12 lines · Mat Martineau
Quoted 10 lines · Paolo Abeni
This series refactor once again the data path, with the goal of avoding the worker usage for the fast path.
The worker will still be used to:
- cleanup closed subflows
- handle timeout
- retransmissions
- send the DATA_FIN/DATA_FIN ack.
- fallback eof
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
The latters 2 use case could be fourther removed, if we accept/consider sending DATA_FIN/ack on a single subflow.
Sending DATA_FIN/ack on a single subflow looks ok to me in this case.
Excellent! does that applies to the
data_fin, too?
Sure.
Quoted 6 lines
Quoted 5 lines
It would make sense to send that ack on the active subflow where incoming data is being processed. On the workqueue that we didn't know which subflow would be best to use, and didn't want to pick an inactive and possibly-not-working subflow.
Quoted 7 lines
If we can send the
data_finon the subflow reaching the end of sequence, we will not need anymore the workqueue to send thedata_fin. And the same for the data fin ack.We will need the workqueue only for shutting down the subflows after receiving the
data_finack.
And retransmission?
--
Mat Martineau
Intel