mptcp: rework mptcp_sendmsg
Subsystems (2)
Florian Westphal
v2 cover mptcp: rework mptcp_sendmsg
This series reworks mptcp_sendmsg logic to avoid blocking on the subflow socket.
This is another iteration, because there were problems with the previous one.
1. Add a new patch 1/7 to stop mptcp from accepting megabytes worth of data when tcp data gets acked at subflow level quickly, but not at mptcp level.
2. drop the patch that cached an skb in sk->sk_tx_skb_cache. One problem with that patch was that it sabotages an extra check that tcp has: when it allocates and skb it will behave slightly differently when the tcp write and retrans queues are empty wrt. wmem schedule.
By preallocating in mptcp layer that code gets bypassed. Moreover, the preallocation doesn't buy anything; the allocation can schedule but it won't sleep forever.
3. for the 3rd patch I moved the initialisation to a single location, also the timer is only started before returning to userspace. As long was we keep looping, the timer doesn't have to be started here as it will be kicked by mptcp_data_acked() for the incoming acks.
4. xmit via lo and MPTCP is SLOOOOOOOOW in 'mmap' mode. However it turned out this is already the case without these patches, so I decided to send these patches out anyway.
I will send another email to followup up on this.
I've preserved Paolos ACKs on patches that had no changes vs. v1.
Florian Westphal (7):
mptcp: break and restart in case mptcp sndbuf is full
mptcp: avoid blocking in tcp_sendpages
mptcp: make sure timer is inited once data is pending for transmission
mptcp: fill skb extension cache outside of mptcp_sendmsg_frag
mptcp: fill skb page frag cache outside of mptcp_sendmsg_frag
mptcp: remove inner wait loop from mptcp_sendmsg_frag
net: allow __skb_ext_alloc to sleep
include/linux/skbuff.h | 2 +- net/core/skbuff.c | 8 ++-- net/mptcp/protocol.c | 116 ++++++++++++++++++++++++++++++++++++------------- 3 files changed, 91 insertions(+), 35 deletions(-)
Florian Westphal
PATCH v2 1/7 mptcp: break and restart in case mptcp sndbuf is full
Its not enough to wait for TCP-level acks (free tcp send space). We also hold on to any transmitted data for mptcp-level retransmits, so we can't continue sending more data when there is no more mptcp send buffer space available.
Therefore add needed check in mptcp_sendmsg, flush tcp data and wait until mptcp snd space becomes available.
--- net/mptcp/protocol.c | 37 ++++++++++++++++++++++++++++++------- 1 file changed, 30 insertions(+), 7 deletions(-)
net/mptcp/protocol.c +30 -7
@@ -653,6 +653,15 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,return ret; } +static void mptcp_nospace (struct mptcp_sock *msk,struct socket *sock) +{ +clear_bit (MPTCP_SEND_SPACE , &msk->flags); +smp_mb__after_atomic ();/* msk->flags is changed by write_space cb */ + +/* enables sk->write_space() callbacks */ +set_bit (SOCK_NOSPACE , &sock->flags); +} +static struct sock *mptcp_subflow_get_send (struct mptcp_sock *msk) {struct mptcp_subflow_context *subflow;
@@ -666,13 +675,8 @@ static struct sock *mptcp_subflow_get_send(struct mptcp_sock *msk)if (!sk_stream_memory_free (ssk)) {struct socket *sock = ssk->sk_socket; -if (sock) { -clear_bit (MPTCP_SEND_SPACE , &msk->flags); -smp_mb__after_atomic (); - -/* enables sk->write_space() callbacks */ -set_bit (SOCK_NOSPACE , &sock->flags); - } +if (sock) +mptcp_nospace (msk, sock);return NULL ; }
@@ -740,6 +744,7 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)mptcp_clean_una (sk); +wait_for_sndbuf:__mptcp_flush_join_list (msk); ssk =mptcp_subflow_get_send (msk);while (!sk_stream_memory_free (sk) || !ssk) {
@@ -777,6 +782,24 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) } copied += ret; + +/* memory is charged to mptcp level socket as well, i.e. +* if msg is very large, mptcp socket may run out of buffer +* space. mptcp_clean_una() will release data that has +* been acked at mptcp level in the mean time, so there is +* a good chance we can continue sending data right away. +*/ +if (unlikely (!sk_stream_memory_free (sk))) { +tcp_push (ssk, msg->msg_flags, mss_now, +tcp_sk (ssk)->nonagle, size_goal); +mptcp_clean_una (sk); +if (!sk_stream_memory_free (sk)) { +/* no luck, can't send more for now */ +mptcp_nospace (msk, sk->sk_socket); +release_sock (ssk); + goto wait_for_sndbuf; + } + } }mptcp_set_timeout (sk, ssk);
Signature · 1 line
2.26.2
Florian Westphal
PATCH v2 2/7 mptcp: avoid blocking in tcp_sendpages
In reply to Florian WestphalThe transmit loop continues to xmit new data until an error is returned or all data was transmitted.
For the blocking i/o case, this means that tcp_sendpages() may block on the subflow until more space becomes available, i.e. we end up sleeping with the mptcp socket lock held.
Instead we should check if a different subflow is ready to be used.
This restarts the subflow sk lookup when the tx operation succeeded and the tcp subflow can't accept more data or if tcp_sendpages indicates -EAGAIN on a blocking mptcp socket.
In that case we also need to set the NOSPACE bit to make sure we get notified once memory becomes available.
In case all subflows are busy, the existing logic will wait until a subflow is ready, releasing the mptcp socket lock while doing so.
The mptcp worker already sets DONTWAIT, so no need to make changes there.
v2:
* set NOSPACE bit
* add a comment to clarify that mptcp-sk sndbuf limits need to be checked as well.
--- net/mptcp/protocol.c | 33 ++++++++++++++++++++++++++++++--- 1 file changed, 30 insertions(+), 3 deletions(-)
net/mptcp/protocol.c +30 -3
@@ -590,7 +590,7 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,* access the skb after the sendpages call */ ret =do_tcp_sendpages (ssk, page, offset, psize, - msg->msg_flags |MSG_SENDPAGE_NOTLAST ); + msg->msg_flags |MSG_SENDPAGE_NOTLAST |MSG_DONTWAIT );if (ret <=0 )return ret;
@@ -718,6 +718,7 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)struct socket *ssock;size_t copied =0 ;struct sock *ssk; +bool tx_ok;long timeo;if (msg->msg_flags & ~(MSG_MORE |MSG_DONTWAIT |MSG_NOSIGNAL ))
@@ -742,6 +743,7 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)return ret >=0 ? ret + copied : (copied ? copied : ret); } +restart:mptcp_clean_una (sk); wait_for_sndbuf:
@@ -764,11 +766,17 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)pr_debug ("conn_list->subflow=%p" , ssk);lock_sock (ssk); -while (msg_data_left (msg)) { + tx_ok =msg_data_left (msg); +while (tx_ok) { ret =mptcp_sendmsg_frag (sk, ssk, msg,NULL , &timeo, &mss_now, &size_goal); -if (ret <0 ) +if (ret <0 ) { +if (ret == -EAGAIN && timeo >0 ) { +release_sock (ssk); + goto restart; + }break ; + }if (ret ==0 &&unlikely (__mptcp_needs_tcp_fallback (msk))) {/* Can happen for passive sockets: * 3WHS negotiated MPTCP, but first packet after is
@@ -783,11 +791,30 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) copied += ret; + tx_ok =msg_data_left (msg); +if (!tx_ok) +break ; + +if (!sk_stream_memory_free (ssk)) { +set_bit (SOCK_NOSPACE , &sk->sk_socket->flags); +tcp_push (ssk, msg->msg_flags, mss_now, +tcp_sk (ssk)->nonagle, size_goal); +release_sock (ssk); + goto restart; + } +/* memory is charged to mptcp level socket as well, i.e. * if msg is very large, mptcp socket may run out of buffer * space. mptcp_clean_una() will release data that has * been acked at mptcp level in the mean time, so there is * a good chance we can continue sending data right away. +* +* Normally, when the tcp subflow can accept more data, then +* so can the MPTCP socket. However, we need to cope with +* peers that might lag behind in their MPTCP-level +* acknowledgements, i.e. data might have been acked at +* tcp level only. So, we must also check the MPTCP socket +* limits before we send more data. */ if (unlikely (!sk_stream_memory_free (sk))) { tcp_push(ssk, msg->msg_flags, mss_now,
Signature · 1 line
2.26.2
Florian Westphal
PATCH v2 3/7 mptcp: make sure timer is inited once data is pending for transmission
In reply to Florian WestphalThe mptcp_connect self-tests with additional '-m mmap' argument triggers a (harmless) WARN_ON_ONCE splat in mptcp_reset_timer() as the msk sockets timer_ival is still 0.
This is because the timer is only set up after the transmit loop, but the splat can be triggered once the first mptcp-level ACK is processed.
--- net/mptcp/protocol.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
net/mptcp/protocol.c +3 -2
@@ -647,6 +647,9 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk, out:if (!retransmission) pfrag->offset += frag_truesize; + +mptcp_set_timeout (sk, ssk); + *write_seq += ret;mptcp_subflow_ctx (ssk)->rel_write_seq += ret;
@@ -829,7 +832,6 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) } } -mptcp_set_timeout (sk, ssk);if (copied) { ret = copied; tcp_push(ssk, msg->msg_flags, mss_now, tcp_sk(ssk)->nonagle,
@@ -1195,7 +1197,6 @@ static void mptcp_worker(struct work_struct *work) dfrag->offset = orig_offset; dfrag->data_len = orig_len; -mptcp_set_timeout (sk, ssk);release_sock (ssk); reset_unlock:
Signature · 1 line
2.26.2
Florian Westphal
PATCH v2 4/7 mptcp: fill skb extension cache outside of mptcp_sendmsg_frag
In reply to Florian WestphalThe mptcp_sendmsg_frag helper contains a loop that will wait on the subflow sk.
It seems preferrable to only wait in mptcp_sendmsg() when blocking io is requested. mptcp_sendmsg already has such a wait loop that is used when no subflow socket is available for transmission.
This is a preparation patch that makes sure we call mptcp_sendmsg_frag only if a skb extension has been allocated.
Moreover, such allocation currently uses GFP_ATOMIC while it could use sleeping allocation instead.
Followup patches will remove the wait loop from mptcp_sendmsg_frag() and will allow to do a sleeping allocation for the extension.
--- net/mptcp/protocol.c | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-)
net/mptcp/protocol.c +14 -4
@@ -672,6 +672,9 @@ static struct sock *mptcp_subflow_get_send(struct mptcp_sock *msk)sock_owned_by_me ((const struct sock *)msk); +if (!mptcp_ext_cache_refill (msk)) +return NULL ; +mptcp_for_each_subflow (msk, subflow) {struct sock *ssk =mptcp_subflow_tcp_sock (subflow);
@@ -798,7 +801,8 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)if (!tx_ok)break ; -if (!sk_stream_memory_free (ssk)) { +if (!sk_stream_memory_free (ssk) || + !mptcp_ext_cache_refill (msk)) {set_bit (SOCK_NOSPACE , &sk->sk_socket->flags);tcp_push (ssk, msg->msg_flags, mss_now,tcp_sk (ssk)->nonagle, size_goal);
@@ -1146,7 +1150,7 @@ static void mptcp_worker(struct work_struct *work) { struct mptcp_sock *msk = container_of(work, struct mptcp_sock, work);struct sock *ssk, *sk = &msk->sk.icsk_inet.sk; -int orig_len, orig_offset, ret, mss_now =0 , size_goal =0 ; +int orig_len, orig_offset, mss_now =0 , size_goal =0 ;struct mptcp_data_frag *dfrag;u64 orig_write_seq;size_t copied =0 ;
@@ -1168,6 +1172,9 @@ static void mptcp_worker(struct work_struct *work)if (!dfrag) goto unlock; +if (!mptcp_ext_cache_refill (msk)) + goto reset_unlock; + ssk =mptcp_subflow_get_retrans (msk);if (!ssk) goto reset_unlock;
@@ -1179,8 +1186,8 @@ static void mptcp_worker(struct work_struct *work) orig_offset = dfrag->offset; orig_write_seq = dfrag->data_seq;while (dfrag->data_len >0 ) { - ret =mptcp_sendmsg_frag (sk, ssk, &msg, dfrag, &timeo, &mss_now, - &size_goal); +int ret =mptcp_sendmsg_frag (sk, ssk, &msg, dfrag, &timeo, + &mss_now, &size_goal);if (ret <0 )break ;
@@ -1188,6 +1195,9 @@ static void mptcp_worker(struct work_struct *work) copied += ret; dfrag->data_len -= ret; dfrag->offset += ret; + +if (!mptcp_ext_cache_refill (msk)) +break ; }if (copied) tcp_push(ssk, msg.msg_flags, mss_now, tcp_sk(ssk)->nonagle,
Signature · 1 line
2.26.2
Florian Westphal
PATCH v2 5/7 mptcp: fill skb page frag cache outside of mptcp_sendmsg_frag
In reply to Florian WestphalThe mptcp_sendmsg_frag helper contains a loop that will wait on the subflow sk.
It seems preferrable to only wait in mptcp_sendmsg() when blocking io is requested. mptcp_sendmsg already has such a wait loop that is used when no subflow socket is available for transmission.
This is another preparation patch that makes sure we call mptcp_sendmsg_frag only if the page frag cache has been refilled.
Followup patch will remove the wait loop from mptcp_sendmsg_frag().
The retransmit worker doesn't need to do this refill as it won't transmit new mptcp-level data.
--- net/mptcp/protocol.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
net/mptcp/protocol.c +6 -1
@@ -721,6 +721,7 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) {int mss_now =0 , size_goal =0 , ret =0 ;struct mptcp_sock *msk =mptcp_sk (sk); +struct page_frag *pfrag;struct socket *ssock;size_t copied =0 ;struct sock *ssk;
@@ -749,13 +750,16 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)return ret >=0 ? ret + copied : (copied ? copied : ret); } + pfrag =sk_page_frag (sk); restart:mptcp_clean_una (sk); wait_for_sndbuf:__mptcp_flush_join_list (msk); ssk =mptcp_subflow_get_send (msk); -while (!sk_stream_memory_free (sk) || !ssk) { +while (!sk_stream_memory_free (sk) || + !ssk || + !mptcp_page_frag_refill (ssk, pfrag)) { ret =sk_stream_wait_memory (sk, &timeo);if (ret) goto out;
@@ -802,6 +806,7 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)break ;if (!sk_stream_memory_free (ssk) || + !mptcp_page_frag_refill (ssk, pfrag) || !mptcp_ext_cache_refill (msk)) {set_bit (SOCK_NOSPACE , &sk->sk_socket->flags); tcp_push(ssk, msg->msg_flags, mss_now,
Signature · 1 line
2.26.2
Florian Westphal
PATCH v2 6/7 mptcp: remove inner wait loop from mptcp_sendmsg_frag
In reply to Florian Westphalprevious patches made sure we only call into this function when these prerequisites are met, so no need to wait on the subflow socket anymore.
--- net/mptcp/protocol.c | 14 -------------- 1 file changed, 14 deletions(-)
net/mptcp/protocol.c +0 -14
@@ -510,20 +510,6 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,* fooled into a warning if we don't init here */ pfrag =sk_page_frag (sk); -while ((!retransmission && !mptcp_page_frag_refill (ssk, pfrag)) || - !mptcp_ext_cache_refill (msk)) { - ret =sk_stream_wait_memory (ssk, timeo); -if (ret) -return ret; - -/* if sk_stream_wait_memory() sleeps snd_una can change -* significantly, refresh the rtx queue -*/ -mptcp_clean_una (sk); - -if (unlikely (__mptcp_needs_tcp_fallback (msk))) -return 0 ; - }if (!retransmission) { write_seq = &msk->write_seq; page = pfrag->page;
Signature · 1 line
2.26.2
Florian Westphal
PATCH v2 7/7 net: allow __skb_ext_alloc to sleep
In reply to Florian Westphalmptcp calls this from the transmit side, from process context. Allow a sleeping allocation instead of unconditional GFP_ATOMIC.
--- include/linux/skbuff.h | 2 +- net/core/skbuff.c | 8 +++++--- net/mptcp/protocol.c | 4 +++- 3 files changed, 9 insertions(+), 5 deletions(-)
include/linux/skbuff.h +1 -1
@@ -4165,7 +4165,7 @@ struct skb_ext {char data[]__aligned(8) ; }; -struct skb_ext *__skb_ext_alloc (void ); +struct skb_ext *__skb_ext_alloc (gfp_t flags);void *__skb_ext_set (struct sk_buff *skb,enum skb_ext_id id,struct skb_ext *ext);void *skb_ext_add (struct sk_buff *skb,enum skb_ext_id id);
net/core/skbuff.c +5 -3
@@ -6087,13 +6087,15 @@ static void *skb_ext_get_ptr(struct skb_ext *ext, enum skb_ext_id id)/** * __skb_ext_alloc - allocate a new skb extensions storage * +* @flags: See kmalloc(). +* * Returns the newly allocated pointer. The pointer can later attached to a * skb via __skb_ext_set(). * Note: caller must handle the skb_ext as an opaque data. */ -struct skb_ext *__skb_ext_alloc (void ) +struct skb_ext *__skb_ext_alloc (gfp_t flags) { -struct skb_ext *new =kmem_cache_alloc (skbuff_ext_cache,GFP_ATOMIC ); +struct skb_ext *new =kmem_cache_alloc (skbuff_ext_cache, flags);if (new) {memset (new->offset,0 ,sizeof (new->offset));
@@ -6188,7 +6190,7 @@ void *skb_ext_add(struct sk_buff *skb, enum skb_ext_id id) }else { newoff =SKB_EXT_CHUNKSIZEOF (*new); - new =__skb_ext_alloc (); + new =__skb_ext_alloc (GFP_ATOMIC );if (!new)return NULL ; }
net/mptcp/protocol.c +3 -1
@@ -367,8 +367,10 @@ static void mptcp_stop_timer(struct sock *sk)static bool mptcp_ext_cache_refill (struct mptcp_sock *msk) { +const struct sock *sk = (const struct sock *)msk; +if (!msk->cached_ext) - msk->cached_ext =__skb_ext_alloc (); + msk->cached_ext =__skb_ext_alloc (sk->sk_allocation);return !!msk->cached_ext; }
Signature · 1 line
2.26.2
Paolo Abeni
[MPTCP] Re: [PATCH net-next 1/7] mptcp: break and restart in case mptcp sndbuf is full
In reply to Florian WestphalQuoted 19 lines · Florian Westphal
Its not enough to wait for TCP-level acks (free tcp send space). We also hold on to any transmitted data for mptcp-level retransmits, so we can't continue sending more data when there is no more mptcp send buffer space available.
Therefore add needed check in
mptcp_sendmsg, flush tcp data and wait until mptcp snd space becomes available.Signed-off-by Florian Westphal <fw(a)strlen.de>--- net/mptcp/protocol.c | 37 ++++++++++++++++++++++++++++++------- 1 file changed, 30 insertions(+), 7 deletions(-)net/mptcp/protocol.c +0
-0@@ -653,6 +653,15 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,return ret;
net/mptcp/protocol.c static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk, · line 663
Quoted 10 lines
net/mptcp/protocol.c +8
-0} +static void mptcp_nospace (struct mptcp_sock *msk,struct socket *sock) +{ +clear_bit (MPTCP_SEND_SPACE , &msk->flags); +smp_mb__after_atomic ();/* msk->flags is changed by write_space cb */ + +/* enables sk->write_space() callbacks */ +set_bit (SOCK_NOSPACE , &sock->flags); +}
(sharing on the ML a topic already discussed privatelly) I think we can use this helper also in ssk_check_wmem().
Other than this, series LGTM!
Thanks Floerian,
/P
Florian Westphal
[MPTCP] Re: [PATCH net-next 1/7] mptcp: break and restart in case mptcp sndbuf is full
Quoted 22 lines · Paolo Abeni
Quoted 21 lines · Florian Westphal
Its not enough to wait for TCP-level acks (free tcp send space). We also hold on to any transmitted data for mptcp-level retransmits, so we can't continue sending more data when there is no more mptcp send buffer space available.
Therefore add needed check in
mptcp_sendmsg, flush tcp data and wait until mptcp snd space becomes available.Signed-off-by Florian Westphal <fw(a)strlen.de>--- net/mptcp/protocol.c | 37 ++++++++++++++++++++++++++++++------- 1 file changed, 30 insertions(+), 7 deletions(-)net/mptcp/protocol.c +0
-0@@ -653,6 +653,15 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,return ret; }
Quoted 11 lines
net/mptcp/protocol.c static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk, · line 663
Quoted 8 lines
net/mptcp/protocol.c +8
-0+static void mptcp_nospace (struct mptcp_sock *msk,struct socket *sock) +{ +clear_bit (MPTCP_SEND_SPACE , &msk->flags); +smp_mb__after_atomic ();/* msk->flags is changed by write_space cb */ + +/* enables sk->write_space() callbacks */ +set_bit (SOCK_NOSPACE , &sock->flags); +}(sharing on the ML a topic already discussed privatelly) I think we can use this helper also in
ssk_check_wmem().
Thanks for pointing this out, I've split this into a new patch to reduce size of this one.