mptcp: rework poll+nospace handling
Subsystems (2)
Florian Westphal
cover mptcp: rework poll+nospace handling
A while back I had a discussion with Paolo on how mptcp handles the 'is mptcp socket writeable' handling in the sendmsg path, rather than the poll one.
There are only two cases:
1. Blocking IO, where userspace calls send/read.
In this scenario, maintaining the MPTCP_SEND_SPACE bit in sendmsg path is useless extra work -- userspace will block at send time if no more wmem is available.
2. Nonblocking IO.
If userspace calls sendmsg without checking poll first, it has to cope with -EAGAIN.
If userspace does call poll first, then mptcp can do the 'is mptcp socket writeable' check in mptcp_poll.
Consequently, patch #2 removes ssk_check_wmem(). A similar test is added to mptcp_poll().
Furthermore, the MPTCP_SEND_SPACE bit has its meaning inverted and gets renamed to MPTCP_NOSPACE. Its only function is to let the mptcp_data_acked() path know if the mptcp worker needs to be scheduled or not.
IOW, when sendmsg path is going to sleep, or when poll path finds socket isn't writeable, this bit makes sure that the next data ack will wake userspace.
For the unlikely case that the mptcp socket is writeable but subflow sockets are not, this will still enable tcp-level callbacks via the SOCK_NOSPACE machinery.
Sending as RFC for now as I'm not sure others consider this approach better or easier to follow then what we currently have.
First patch might be useful in its own right even if #2 is rejected.
Florian Westphal
RFC 1/3 mptcp: split mptcp_clean_una function
mptcp_clean_una() will wake writes in case memory could be reclaimed. However, when called from mptcp_sendmsg the wakeup isn't needed.
Split the helper and then call the 'lighter' function when we do not need a wakeup.
--- net/mptcp/protocol.c | 34 ++++++++++++++++++++-------------- 1 file changed, 20 insertions(+), 14 deletions(-)
net/mptcp/protocol.c +20 -14
@@ -791,7 +791,7 @@ static bool mptcp_is_writeable(struct mptcp_sock *msk)return false; } -static void mptcp_clean_una (struct sock *sk) +static void __mptcp_clean_una (struct sock *sk) {struct mptcp_sock *msk =mptcp_sk (sk);struct mptcp_data_frag *dtmp, *dfrag;
@@ -829,19 +829,25 @@ static void mptcp_clean_una(struct sock *sk) } out: -if (cleaned) { +if (cleaned)sk_mem_reclaim_partial (sk); +} -/* Only wake up writers if a subflow is ready */ -if (mptcp_is_writeable (msk)) { -set_bit (MPTCP_SEND_SPACE , &mptcp_sk (sk)->flags); -smp_mb__after_atomic (); +static void mptcp_clean_una (struct sock *sk) +{ +struct mptcp_sock *msk =mptcp_sk (sk); -/* set SEND_SPACE before sk_stream_write_space clears -* NOSPACE -*/ -sk_stream_write_space (sk); - } +__mptcp_clean_una (sk); + +/* Only wake up writers if a subflow is ready */ +if (mptcp_is_writeable (msk)) { +set_bit (MPTCP_SEND_SPACE , &msk->flags); +smp_mb__after_atomic (); + +/* set SEND_SPACE before sk_stream_write_space clears +* NOSPACE +*/ +sk_stream_write_space (sk); } }
@@ -1186,7 +1192,7 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) pfrag =sk_page_frag (sk); restart: -mptcp_clean_una (sk); +__mptcp_clean_una (sk);if (sk->sk_err || (sk->sk_shutdown &SEND_SHUTDOWN )) { ret = -EPIPE ;
@@ -1216,7 +1222,7 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)if (ret) goto out; -mptcp_clean_una (sk); +__mptcp_clean_una (sk); ssk =mptcp_subflow_get_send (msk, &sndbuf);if (list_empty (&msk->conn_list)) {
@@ -1282,7 +1288,7 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)if (unlikely (!sk_stream_memory_free (sk))) {tcp_push (ssk, msg->msg_flags, mss_now,tcp_sk (ssk)->nonagle, size_goal); -mptcp_clean_una (sk); +__mptcp_clean_una (sk);if (!sk_stream_memory_free (sk)) {/* can't send more for now, need to wait for * MPTCP-level ACKs from peer.
Signature · 1 line
2.26.2
Florian Westphal
RFC 2/3 mptcp: rework poll+nospace handling
In reply to Florian WestphalAt this time, MPTCP maintains a status bit, MPTCP_SEND_SPACE, that is set when at least one subflow and the mptcp socket itself are writeable.
poll path set EPOLLOUT if the bit is set.
sendmsg path makes sure MPTCP_SEND_SPACE gets cleared when last write has used up all subflows or the mptcp socket wmem.
This reworks nospace handling as follows:
MPTCP_SEND_SPACE is replaced with MPTCP_NOSPACE. This bit is set when the mptcp socket is not writeable anymore and makes the mptcp-level ack path schedule the mptcp worker.
This is needed to wake up userspace processes that wait for a POLLOUT event.
sendmsg will set MPTCP_NOSPACE only when it has to wait for more wmem (blocking I/O case).
poll path will also set MPTCP_NOSPACE in case the mptcp socket is not writeable.
Normal tcp-level notification (SOCK_NOSPACE) is only enabled in case the subflow socket has no available wmem.
--- net/mptcp/protocol.c | 98 ++++++++++++++++++++++++++++++-------------- net/mptcp/protocol.h | 2 +- net/mptcp/subflow.c | 11 +++-- 3 files changed, 74 insertions(+), 37 deletions(-)
net/mptcp/protocol.c +68 -30
@@ -680,10 +680,12 @@ static void mptcp_reset_timer(struct sock *sk)void mptcp_data_acked (struct sock *sk) { +struct mptcp_sock *msk =mptcp_sk (sk); +mptcp_reset_timer (sk); -if ((!test_bit (MPTCP_SEND_SPACE , &mptcp_sk (sk)->flags) || - (inet_sk_state_load (sk) !=TCP_ESTABLISHED )) && +if ((test_bit (MPTCP_NOSPACE , &msk->flags) || + (inet_sk_state_load (sk) !=TCP_ESTABLISHED )) &&schedule_work (&mptcp_sk (sk)->work))sock_hold (sk); }
@@ -841,12 +843,7 @@ static void mptcp_clean_una(struct sock *sk)/* Only wake up writers if a subflow is ready */ if (mptcp_is_writeable (msk)) { -set_bit (MPTCP_SEND_SPACE , &msk->flags); -smp_mb__after_atomic (); - -/* set SEND_SPACE before sk_stream_write_space clears -* NOSPACE -*/ +clear_bit (MPTCP_NOSPACE , &msk->flags);sk_stream_write_space (sk); } }
@@ -1038,21 +1035,37 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,return ret; } -static void mptcp_nospace (struct mptcp_sock *msk) +/* set SOCK_NOSPACE for any subflow that has no more wmem. +* +* Returns true if at least one subflow is writeable. +*/ +static bool mptcp_set_subflow_nospace (struct mptcp_sock *msk) {struct mptcp_subflow_context *subflow; - -clear_bit (MPTCP_SEND_SPACE , &msk->flags); -smp_mb__after_atomic ();/* msk->flags is changed by write_space cb */ +bool ssk_any_writeable = false;mptcp_for_each_subflow (msk, subflow) {struct sock *ssk =mptcp_subflow_tcp_sock (subflow); +bool ssk_writeable =sk_stream_is_writeable (ssk);struct socket *sock =READ_ONCE (ssk->sk_socket); +if (ssk_writeable) { + ssk_any_writeable = true; +continue ; + } + +if (!sock) +continue ; +/* enables ssk->write_space() callbacks */ -if (sock) -set_bit (SOCK_NOSPACE , &sock->flags); +set_bit (SOCK_NOSPACE , &sock->flags); + + ssk_writeable =sk_stream_is_writeable (ssk); +if (ssk_writeable) + ssk_any_writeable = true; } + +return ssk_any_writeable; }static bool mptcp_subflow_active (struct mptcp_subflow_context *subflow)
@@ -1160,10 +1173,28 @@ static struct sock *mptcp_subflow_get_send(struct mptcp_sock *msk,return free ? msk->last_snd :NULL ; } -static void ssk_check_wmem (struct mptcp_sock *msk) +static bool __mptcp_check_writeable (struct mptcp_sock *msk) { -if (unlikely (!mptcp_is_writeable (msk))) -mptcp_nospace (msk); +struct sock *sk = (struct sock *)msk; +bool mptcp_writable; + +__mptcp_clean_una (sk); + mptcp_writable =sk_stream_is_writeable (sk); +if (!mptcp_writable) { +set_bit (MPTCP_NOSPACE , &msk->flags); +smp_mb__after_atomic (); +__mptcp_clean_una (sk); + + mptcp_writable =sk_stream_is_writeable (sk); +if (likely (!mptcp_writable)) +return false; + +/* MPTCP ack after MPTCP_NOSPACE was set */ +clear_bit (MPTCP_NOSPACE , &msk->flags); + } + +/* mptcp socket is writeable, now check subflows */ +return mptcp_set_subflow_nospace (msk); }static int mptcp_sendmsg (struct sock *sk,struct msghdr *msg,size_t len)
@@ -1217,7 +1248,8 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)mptcp_reset_timer (sk); } -mptcp_nospace (msk); +__mptcp_check_writeable (msk); + ret =sk_stream_wait_memory (sk, &timeo);if (ret) goto out;
@@ -1314,7 +1346,6 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)release_sock (ssk); out: -ssk_check_wmem (msk);release_sock (sk);return copied ? : ret; }
@@ -1811,7 +1842,6 @@ static int __mptcp_init_sock(struct sock *sk)INIT_LIST_HEAD (&msk->conn_list);INIT_LIST_HEAD (&msk->join_list);INIT_LIST_HEAD (&msk->rtx_queue); -__set_bit (MPTCP_SEND_SPACE , &msk->flags);INIT_WORK (&msk->work, mptcp_worker); msk->out_of_order_queue =RB_ROOT ;
@@ -2422,13 +2452,6 @@ bool mptcp_finish_join(struct sock *sk)return true; } -static bool mptcp_memory_free (const struct sock *sk,int wake) -{ -struct mptcp_sock *msk =mptcp_sk (sk); - -return wake ?test_bit (MPTCP_SEND_SPACE , &msk->flags) : true; -} -static struct proto mptcp_prot = { .name ="MPTCP" , .owner =THIS_MODULE ,
@@ -2449,7 +2472,6 @@ static struct proto mptcp_prot = { .sockets_allocated = &mptcp_sockets_allocated, .memory_allocated = &tcp_memory_allocated, .memory_pressure = &tcp_memory_pressure, - .stream_memory_free = mptcp_memory_free, .sysctl_wmem_offset = offsetof(struct net, ipv4.sysctl_tcp_wmem), .sysctl_mem = sysctl_tcp_mem, .obj_size = sizeof(struct mptcp_sock),
@@ -2622,6 +2644,23 @@ static __poll_t mptcp_check_readable(struct mptcp_sock *msk)0 ; } +static __poll_t mptcp_check_writeable (struct mptcp_sock *msk) +{ +struct sock *sk = (struct sock *)msk; +__poll_t ret =0 ; +bool slow; + +if (sk->sk_shutdown &SEND_SHUTDOWN ) +return 0 ; + + slow =lock_sock_fast (sk); +if (__mptcp_check_writeable (msk)) + ret =EPOLLOUT |EPOLLWRNORM ; + +unlock_sock_fast (sk, slow); +return ret; +} +static __poll_t mptcp_poll (struct file *file,struct socket *sock,struct poll_table_struct *wait) {
@@ -2640,8 +2679,7 @@ static __poll_t mptcp_poll(struct file *file, struct socket *sock,if (state !=TCP_SYN_SENT && state !=TCP_SYN_RECV ) { mask |=mptcp_check_readable (msk); -if (test_bit (MPTCP_SEND_SPACE , &msk->flags)) - mask |=EPOLLOUT |EPOLLWRNORM ; + mask |=mptcp_check_writeable (msk); }if (sk->sk_shutdown &RCV_SHUTDOWN ) mask |=EPOLLIN |EPOLLRDNORM |EPOLLRDHUP ;
net/mptcp/protocol.h +1 -1
@@ -86,7 +86,7 @@/* MPTCP socket flags */ #define MPTCP_DATA_READY 0 -#define MPTCP_SEND_SPACE 1 +#define MPTCP_NOSPACE 1#define MPTCP_WORK_RTX 2#define MPTCP_WORK_EOF 3#define MPTCP_FALLBACK_DONE 4
net/mptcp/subflow.c +5 -6
@@ -972,17 +972,16 @@ static void subflow_data_ready(struct sock *sk)static void subflow_write_space (struct sock *sk) {struct mptcp_subflow_context *subflow =mptcp_subflow_ctx (sk); +struct socket *sock =READ_ONCE (sk->sk_socket);struct sock *parent = subflow->conn;if (!sk_stream_is_writeable (sk))return ; -if (sk_stream_is_writeable (parent)) { -set_bit (MPTCP_SEND_SPACE , &mptcp_sk (parent)->flags); -smp_mb__after_atomic (); -/* set SEND_SPACE before sk_stream_write_space clears NOSPACE */ -sk_stream_write_space (parent); - } +if (sock) +clear_bit (SOCK_NOSPACE , &sock->flags); + +sk_stream_write_space (parent); } static struct inet_connection_sock_af_ops *
Signature · 1 line
2.26.2
Paolo Abeni
Quoted 74 lines · Florian Westphal
At this time, MPTCP maintains a status bit,
MPTCP_SEND_SPACE, that is set when at least one subflow and the mptcp socket itself are writeable.poll path set EPOLLOUT if the bit is set.
sendmsg path makes sureMPTCP_SEND_SPACEgets cleared when last write has used up all subflows or the mptcp socket wmem.This reworks nospace handling as follows:
MPTCP_SEND_SPACEis replaced withMPTCP_NOSPACE. This bit is set when the mptcp socket is not writeable anymore and makes the mptcp-level ack path schedule the mptcp worker.This is needed to wake up userspace processes that wait for a POLLOUT event.
sendmsg will set
MPTCP_NOSPACEonly when it has to wait for more wmem (blocking I/O case).poll path will also set
MPTCP_NOSPACEin case the mptcp socket is not writeable.Normal tcp-level notification (
SOCK_NOSPACE) is only enabled in case the subflow socket has no available wmem.Signed-off-by Florian Westphal <fw(a)strlen.de>--- net/mptcp/protocol.c | 98 ++++++++++++++++++++++++++++++-------------- net/mptcp/protocol.h | 2 +- net/mptcp/subflow.c | 11 +++-- 3 files changed, 74 insertions(+), 37 deletions(-)net/mptcp/protocol.c +8
-9@@ -680,10 +680,12 @@ static void mptcp_reset_timer(struct sock *sk)void mptcp_data_acked (struct sock *sk) { +struct mptcp_sock *msk =mptcp_sk (sk); +mptcp_reset_timer (sk); -if ((!test_bit (MPTCP_SEND_SPACE , &mptcp_sk (sk)->flags) || - (inet_sk_state_load (sk) !=TCP_ESTABLISHED )) && +if ((test_bit (MPTCP_NOSPACE , &msk->flags) || + (inet_sk_state_load (sk) !=TCP_ESTABLISHED )) &&schedule_work (&mptcp_sk (sk)->work))sock_hold (sk); }@@ -841,12 +843,7 @@ static void mptcp_clean_una(struct sock *sk)/* Only wake up writers if a subflow is ready */ if (mptcp_is_writeable (msk)) { -set_bit (MPTCP_SEND_SPACE , &msk->flags); -smp_mb__after_atomic (); - -/* set SEND_SPACE before sk_stream_write_space clears -* NOSPACE -*/ +clear_bit (MPTCP_NOSPACE , &msk->flags);sk_stream_write_space (sk); } }@@ -1038,21 +1035,37 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,return ret; } -static void mptcp_nospace (struct mptcp_sock *msk) +/* set SOCK_NOSPACE for any subflow that has no more wmem. +* +* Returns true if at least one subflow is writeable.
net/mptcp/protocol.c static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk, · line 1047
Quoted 10 lines
net/mptcp/protocol.c +3
-3+*/ +static bool mptcp_set_subflow_nospace (struct mptcp_sock *msk) {struct mptcp_subflow_context *subflow; - -clear_bit (MPTCP_SEND_SPACE , &msk->flags); -smp_mb__after_atomic ();/* msk->flags is changed by write_space cb */ +bool ssk_any_writeable = false;mptcp_for_each_subflow (msk, subflow) {
Not strictly related to this patch, but perhpas we need a __mptcp_flush_join_list() call before traversing the subflow list?!?
Quoted 59 lines
net/mptcp/protocol.c +37
-5struct sock *ssk =mptcp_subflow_tcp_sock (subflow); +bool ssk_writeable =sk_stream_is_writeable (ssk);struct socket *sock =READ_ONCE (ssk->sk_socket); +if (ssk_writeable) { + ssk_any_writeable = true; +continue ; + } + +if (!sock) +continue ; +/* enables ssk->write_space() callbacks */ -if (sock) -set_bit (SOCK_NOSPACE , &sock->flags); +set_bit (SOCK_NOSPACE , &sock->flags); + + ssk_writeable =sk_stream_is_writeable (ssk); +if (ssk_writeable) + ssk_any_writeable = true; } + +return ssk_any_writeable; }static bool mptcp_subflow_active (struct mptcp_subflow_context *subflow)@@ -1160,10 +1173,28 @@ static struct sock *mptcp_subflow_get_send(struct mptcp_sock *msk,return free ? msk->last_snd :NULL ; } -static void ssk_check_wmem (struct mptcp_sock *msk) +static bool __mptcp_check_writeable (struct mptcp_sock *msk) { -if (unlikely (!mptcp_is_writeable (msk))) -mptcp_nospace (msk); +struct sock *sk = (struct sock *)msk; +bool mptcp_writable; + +__mptcp_clean_una (sk); + mptcp_writable =sk_stream_is_writeable (sk); +if (!mptcp_writable) { +set_bit (MPTCP_NOSPACE , &msk->flags); +smp_mb__after_atomic (); +__mptcp_clean_una (sk); + + mptcp_writable =sk_stream_is_writeable (sk); +if (likely (!mptcp_writable)) +return false; + +/* MPTCP ack after MPTCP_NOSPACE was set */ +clear_bit (MPTCP_NOSPACE , &msk->flags); + } + +/* mptcp socket is writeable, now check subflows */ +return mptcp_set_subflow_nospace (msk); }static int mptcp_sendmsg (struct sock *sk,struct msghdr *msg,size_t len)
net/mptcp/protocol.c static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) · line 1252
Quoted 6 lines
net/mptcp/protocol.c +2
-1@@ -1217,7 +1248,8 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)mptcp_reset_timer (sk); } -mptcp_nospace (msk); +__mptcp_check_writeable (msk); +
Again not strictily related to this patch, but currently we start sleeping/blocking if msk/ssk are not writeable.
I think we should do that instead when !sk_stream_memory_free() ?!?
Quoted 49 lines
net/mptcp/protocol.c +4
-10ret =sk_stream_wait_memory (sk, &timeo);if (ret) goto out;@@ -1314,7 +1346,6 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)release_sock (ssk); out: -ssk_check_wmem (msk);release_sock (sk);return copied ? : ret; }@@ -1811,7 +1842,6 @@ static int __mptcp_init_sock(struct sock *sk)INIT_LIST_HEAD (&msk->conn_list);INIT_LIST_HEAD (&msk->join_list);INIT_LIST_HEAD (&msk->rtx_queue); -__set_bit (MPTCP_SEND_SPACE , &msk->flags);INIT_WORK (&msk->work, mptcp_worker); msk->out_of_order_queue =RB_ROOT ;@@ -2422,13 +2452,6 @@ bool mptcp_finish_join(struct sock *sk)return true; } -static bool mptcp_memory_free (const struct sock *sk,int wake) -{ -struct mptcp_sock *msk =mptcp_sk (sk); - -return wake ?test_bit (MPTCP_SEND_SPACE , &msk->flags) : true; -} -static struct proto mptcp_prot = { .name ="MPTCP" , .owner =THIS_MODULE ,@@ -2449,7 +2472,6 @@ static struct proto mptcp_prot = { .sockets_allocated = &mptcp_sockets_allocated, .memory_allocated = &tcp_memory_allocated, .memory_pressure = &tcp_memory_pressure, - .stream_memory_free = mptcp_memory_free, .sysctl_wmem_offset = offsetof(struct net, ipv4.sysctl_tcp_wmem), .sysctl_mem = sysctl_tcp_mem, .obj_size = sizeof(struct mptcp_sock),@@ -2622,6 +2644,23 @@ static __poll_t mptcp_check_readable(struct mptcp_sock *msk)0 ; } +static __poll_t mptcp_check_writeable (struct mptcp_sock *msk) +{ +struct sock *sk = (struct sock *)msk; +__poll_t ret =0 ;
net/mptcp/protocol.c static __poll_t mptcp_check_readable(struct mptcp_sock *msk) · line 2660
Quoted 10 lines
net/mptcp/protocol.c +10
-0+bool slow; + +if (sk->sk_shutdown &SEND_SHUTDOWN ) +return 0 ; + + slow =lock_sock_fast (sk); +if (__mptcp_check_writeable (msk)) + ret =EPOLLOUT |EPOLLWRNORM ; + +unlock_sock_fast (sk, slow);
It's a pity we need to acquire the lock in mptcp_poll()... perhaps we could try to acquire it only if sk_stream_is_writeable(msk)?
Quoted 41 lines
net/mptcp/protocol.c +4
-2+return ret; +} +static __poll_t mptcp_poll (struct file *file,struct socket *sock,struct poll_table_struct *wait) {@@ -2640,8 +2679,7 @@ static __poll_t mptcp_poll(struct file *file, struct socket *sock,if (state !=TCP_SYN_SENT && state !=TCP_SYN_RECV ) { mask |=mptcp_check_readable (msk); -if (test_bit (MPTCP_SEND_SPACE , &msk->flags)) - mask |=EPOLLOUT |EPOLLWRNORM ; + mask |=mptcp_check_writeable (msk); }if (sk->sk_shutdown &RCV_SHUTDOWN ) mask |=EPOLLIN |EPOLLRDNORM |EPOLLRDHUP ;net/mptcp/protocol.h +1
-1@@ -86,7 +86,7 @@/* MPTCP socket flags */ #define MPTCP_DATA_READY 0 -#define MPTCP_SEND_SPACE 1 +#define MPTCP_NOSPACE 1#define MPTCP_WORK_RTX 2#define MPTCP_WORK_EOF 3#define MPTCP_FALLBACK_DONE 4net/mptcp/subflow.c +1
-0@@ -972,17 +972,16 @@ static void subflow_data_ready(struct sock *sk)static void subflow_write_space (struct sock *sk) {struct mptcp_subflow_context *subflow =mptcp_subflow_ctx (sk); +struct socket *sock =READ_ONCE (sk->sk_socket);struct sock *parent = subflow->conn;if (!sk_stream_is_writeable (sk))
net/mptcp/subflow.c static void subflow_data_ready(struct sock *sk) · line 982
Quoted 10 lines
net/mptcp/subflow.c +2
-6return ; -if (sk_stream_is_writeable (parent)) { -set_bit (MPTCP_SEND_SPACE , &mptcp_sk (parent)->flags); -smp_mb__after_atomic (); -/* set SEND_SPACE before sk_stream_write_space clears NOSPACE */ -sk_stream_write_space (parent); - } +if (sock) +clear_bit (SOCK_NOSPACE , &sock->flags);
for server socket 'sock' is 'shared' among all the subflows (ssk-
Quoted 1 line
sk_socketpoints to the same data). The above is disabling write space
notification on all subflows, is that intended/correct?
net/mptcp/subflow.c static void subflow_data_ready(struct sock *sk) · line 984
Quoted 2 lines
net/mptcp/subflow.c +2
-0+ +sk_stream_write_space (parent);
If I read correctly, this means that the msk could be woken-up even if not writeable (a subflow is writeable, but msk does not have enough wirte space), am I correct?
Thanks!
Paolo
Paolo Abeni
Quoted 10 lines · Paolo Abeni
Quoted 9 lines · Florian Westphal
net/mptcp/protocol.c +9
-0+static __poll_t mptcp_check_writeable (struct mptcp_sock *msk) +{ +struct sock *sk = (struct sock *)msk; +__poll_t ret =0 ; +bool slow; + +if (sk->sk_shutdown &SEND_SHUTDOWN ) +return 0 ; +
Quoted 8 lines
net/mptcp/protocol.c static __poll_t mptcp_check_readable(struct mptcp_sock *msk) · line 2660
Quoted 5 lines
net/mptcp/protocol.c +5
-0+ slow =lock_sock_fast (sk); +if (__mptcp_check_writeable (msk)) + ret =EPOLLOUT |EPOLLWRNORM ; + +unlock_sock_fast (sk, slow);It's a pity we need to acquire the lock in
mptcp_poll()... perhaps we could try to acquire it only ifsk_stream_is_writeable(msk)?
I'm also wondering if checking sk_stream_is_writeable(msk) could be enough??? perhaps with a different sndbuf auto-tuning strategy.
e.g. if msk sndbuf <= max (ssk sndbuf), I *think*
that sk_stream_is_writeable(msk) implies at least a ssk is writeable, too
/P
Florian Westphal
Quoted 20 lines · Paolo Abeni
Quoted 10 lines · Paolo Abeni
Quoted 9 lines · Florian Westphal
net/mptcp/protocol.c +9
-0+static __poll_t mptcp_check_writeable (struct mptcp_sock *msk) +{ +struct sock *sk = (struct sock *)msk; +__poll_t ret =0 ; +bool slow; + +if (sk->sk_shutdown &SEND_SHUTDOWN ) +return 0 ; +Quoted 8 lines
net/mptcp/protocol.c static __poll_t mptcp_check_readable(struct mptcp_sock *msk) · line 2660
Quoted 5 lines
net/mptcp/protocol.c +5
-0+ slow =lock_sock_fast (sk); +if (__mptcp_check_writeable (msk)) + ret =EPOLLOUT |EPOLLWRNORM ; + +unlock_sock_fast (sk, slow);It's a pity we need to acquire the lock in
mptcp_poll()... perhaps we could try to acquire it only ifsk_stream_is_writeable(msk)?
Quoted 2 lines
I'm also wondering if checking
sk_stream_is_writeable(msk)could be enough??? perhaps with a different sndbuf auto-tuning strategy.
Not right now. But perhaps this is worth exploring -- we would need to change mptcp_sendmsg to append data to the rtx queue in case no ssk can be found.
I'd give this a try unless you or anyone else has objections.
Quoted 3 lines
e.g. if msk sndbuf <= max (ssk sndbuf), I *think*
thatsk_stream_is_writeable(msk)implies at least a ssk is writeable, too
You mean mptcp_is_writeable()?
[ Question above still stands though ]
Mat Martineau
Quoted 21 lines · Florian Westphal
Quoted 20 lines · Paolo Abeni
Quoted 18 lines · Paolo Abeni
Quoted 4 lines · Florian Westphal
net/mptcp/protocol.c +4
-0+static __poll_t mptcp_check_writeable (struct mptcp_sock *msk) +{ +struct sock *sk = (struct sock *)msk; +__poll_t ret =0 ;net/mptcp/protocol.c static __poll_t mptcp_check_readable(struct mptcp_sock *msk) · line 2660
Quoted 10 lines
net/mptcp/protocol.c +10
-0+bool slow; + +if (sk->sk_shutdown &SEND_SHUTDOWN ) +return 0 ; + + slow =lock_sock_fast (sk); +if (__mptcp_check_writeable (msk)) + ret =EPOLLOUT |EPOLLWRNORM ; + +unlock_sock_fast (sk, slow);It's a pity we need to acquire the lock in
mptcp_poll()... perhaps we could try to acquire it only ifsk_stream_is_writeable(msk)?
Quoted 8 lines
Quoted 2 lines
I'm also wondering if checking
sk_stream_is_writeable(msk)could be enough??? perhaps with a different sndbuf auto-tuning strategy.Not right now. But perhaps this is worth exploring -- we would need to change
mptcp_sendmsgto append data to the rtx queue in case no ssk can be found.I'd give this a try unless you or anyone else has objections.
No objection here, I think it would be good to explore - might fit the "easier to follow" criteria you mention in the cover letter. Having mptcp_poll() not only locking, but also doing the __mptcp_clean_una() work, seems kind of heavyweight for a poll operation.
Quoted 7 lines
Quoted 4 lines
e.g. if msk sndbuf <= max (ssk sndbuf), I *think*
thatsk_stream_is_writeable(msk)implies at least a ssk is writeable, tooYou mean
mptcp_is_writeable()?
[ Question above still stands though ]
--
Mat Martineau
Intel
Florian Westphal
Quoted 84 lines · Paolo Abeni
Quoted 83 lines · Florian Westphal
At this time, MPTCP maintains a status bit,
MPTCP_SEND_SPACE, that is set when at least one subflow and the mptcp socket itself are writeable.poll path set EPOLLOUT if the bit is set.
sendmsg path makes sureMPTCP_SEND_SPACEgets cleared when last write has used up all subflows or the mptcp socket wmem.This reworks nospace handling as follows:
MPTCP_SEND_SPACEis replaced withMPTCP_NOSPACE. This bit is set when the mptcp socket is not writeable anymore and makes the mptcp-level ack path schedule the mptcp worker.This is needed to wake up userspace processes that wait for a POLLOUT event.
sendmsg will set
MPTCP_NOSPACEonly when it has to wait for more wmem (blocking I/O case).poll path will also set
MPTCP_NOSPACEin case the mptcp socket is not writeable.Normal tcp-level notification (
SOCK_NOSPACE) is only enabled in case the subflow socket has no available wmem.Signed-off-by Florian Westphal <fw(a)strlen.de>--- net/mptcp/protocol.c | 98 ++++++++++++++++++++++++++++++-------------- net/mptcp/protocol.h | 2 +- net/mptcp/subflow.c | 11 +++-- 3 files changed, 74 insertions(+), 37 deletions(-)net/mptcp/protocol.c +11
-12@@ -680,10 +680,12 @@ static void mptcp_reset_timer(struct sock *sk)void mptcp_data_acked (struct sock *sk) { +struct mptcp_sock *msk =mptcp_sk (sk); +mptcp_reset_timer (sk); -if ((!test_bit (MPTCP_SEND_SPACE , &mptcp_sk (sk)->flags) || - (inet_sk_state_load (sk) !=TCP_ESTABLISHED )) && +if ((test_bit (MPTCP_NOSPACE , &msk->flags) || + (inet_sk_state_load (sk) !=TCP_ESTABLISHED )) &&schedule_work (&mptcp_sk (sk)->work))sock_hold (sk); }@@ -841,12 +843,7 @@ static void mptcp_clean_una(struct sock *sk)/* Only wake up writers if a subflow is ready */ if (mptcp_is_writeable (msk)) { -set_bit (MPTCP_SEND_SPACE , &msk->flags); -smp_mb__after_atomic (); - -/* set SEND_SPACE before sk_stream_write_space clears -* NOSPACE -*/ +clear_bit (MPTCP_NOSPACE , &msk->flags);sk_stream_write_space (sk); } }@@ -1038,21 +1035,37 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,return ret; } -static void mptcp_nospace (struct mptcp_sock *msk) +/* set SOCK_NOSPACE for any subflow that has no more wmem. +* +* Returns true if at least one subflow is writeable. +*/ +static bool mptcp_set_subflow_nospace (struct mptcp_sock *msk) {struct mptcp_subflow_context *subflow; - -clear_bit (MPTCP_SEND_SPACE , &msk->flags); -smp_mb__after_atomic ();/* msk->flags is changed by write_space cb */ +bool ssk_any_writeable = false;
Quoted 4 lines
net/mptcp/protocol.c static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk, · line 1047
Quoted 1 line
net/mptcp/protocol.c +0
-0mptcp_for_each_subflow (msk, subflow) {Not strictly related to this patch, but perhpas we need a
__mptcp_flush_join_list()call before traversing the subflow list?!?
Yes, I think it makes sense to update the list here.
Quoted 8 lines
net/mptcp/protocol.c static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) · line 1252
Quoted 3 lines
net/mptcp/protocol.c +2
-1-mptcp_nospace (msk); +__mptcp_check_writeable (msk); +Again not strictily related to this patch, but currently we start sleeping/blocking if msk/ssk are not writeable.
I think we should do that instead when !sk_stream_memory_free() ?!?
Do what? Sleep only if mptcp socket has not enough wspace? We can do that, requires change to mptcp_sendmsg to append to rtx queue (and make sure retrans timer is on), also see my other reply.
Is that what you meant?
Quoted 8 lines
net/mptcp/protocol.c static __poll_t mptcp_check_readable(struct mptcp_sock *msk) · line 2660
Quoted 5 lines
net/mptcp/protocol.c +5
-0+ slow =lock_sock_fast (sk); +if (__mptcp_check_writeable (msk)) + ret =EPOLLOUT |EPOLLWRNORM ; + +unlock_sock_fast (sk, slow);It's a pity we need to acquire the lock in
mptcp_poll()... perhaps we could try to acquire it only ifsk_stream_is_writeable(msk)?
I think we need to determine desired behaviour of send path when no ssk is ready (but mptcp sk has wspace). I think we should change it to queue the data even if no ssk is available (but wmem permits it). Then we could simplify poll too, as ssk state becomes irrelevant.
Quoted 6 lines
net/mptcp/protocol.c static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk, · line 1056
Quoted 2 lines
net/mptcp/protocol.c +2
-0+if (sock) +clear_bit (SOCK_NOSPACE , &sock->flags);for server socket 'sock' is 'shared' among all the subflows (ssk-
Quoted 1 line
sk_socketpoints to the same data). The above is disabling write spacenotification on all subflows, is that intended/correct?
Yes, current sendmsg path accepts data if mptcp socket has wmem and one subflow is ready, so it only sleeps if mptcp wmem is also exhausted.
And for that, the MPTCP_NOSPACE bit has been toggled so next data-ack wakes a blocking poll or blocking sendmsg.
Quoted 6 lines
net/mptcp/protocol.c
Quoted 2 lines
net/mptcp/protocol.c +2
-0+ +sk_stream_write_space (parent);If I read correctly, this means that the msk could be woken-up even if not writeable (a subflow is writeable, but msk does not have enough wirte space), am I correct?
No, this is in a mptcp_is_writeable() conditional so msk has enough wmem. My main takeaway from your response is that I should rework this towards making the mptcp write buffer space the only relevant criteria wrt. when to signal/wake userspace.
Does that sound good to you?
Florian Westphal
RFC 3/3 mptcp: try to clean data inplace if possible
In reply to Florian WestphalIn most cases the worker won't be scheduled because socket has enough space and is established.
If the mptcp retransmit queue needs to cleaned up its possible to try to acquire the mptcp socket lock anyway. This allows to avoid the work queue unless the socket is currently owned.
--- net/mptcp/protocol.c | 25 ++++++++++++++++++++++--- 1 file changed, 22 insertions(+), 3 deletions(-)
net/mptcp/protocol.c +22 -3
@@ -41,6 +41,8 @@ struct mptcp_skb_cb {static struct percpu_counter mptcp_sockets_allocated; +static void mptcp_clean_una (struct sock *sk); +/* If msk has an initial subflow socket, and the MP_CAPABLE handshake has not * completed yet or has failed, return the subflow socket. * Otherwise return NULL.
@@ -681,13 +683,30 @@ static void mptcp_reset_timer(struct sock *sk)void mptcp_data_acked (struct sock *sk) {struct mptcp_sock *msk =mptcp_sk (sk); +bool fast = false;mptcp_reset_timer (sk);if ((test_bit (MPTCP_NOSPACE , &msk->flags) || - (inet_sk_state_load (sk) !=TCP_ESTABLISHED )) && -schedule_work (&mptcp_sk (sk)->work)) -sock_hold (sk); + (inet_sk_state_load (sk) !=TCP_ESTABLISHED ))) { +if (READ_ONCE (sk->sk_lock.owned)) + goto slowpath; + +if (unlikely (!spin_trylock_bh (&sk->sk_lock.slock))) + goto slowpath; + +if (!READ_ONCE (sk->sk_lock.owned)) { +mptcp_clean_una (sk); + fast = true; + } + +spin_unlock_bh (&sk->sk_lock.slock); +if (fast) +return ; + slowpath: +if (schedule_work (&mptcp_sk (sk)->work)) +sock_hold (sk); + } }void mptcp_subflow_eof (struct sock *sk)
Signature · 1 line
2.26.2