four small cleanups and fixes
Subsystems (2)
Florian Westphal
cover four small cleanups and fixes
1. cleanup: mptcp_subflow_get_map_offset arg can be const.
2. fix: don't advance snd_una past the write sequence.
3. cleanup: use existing helper rather than open-coding it
4. fix: avoid blocking kworker forever when polling mptcp socket.
All paches can be squashed, I've added squashto hints.
The following changes since commit 512f0321ae8a2679c15915b68f8a5cd98f190288:
subflow: wake parent mptcp socket on subflow state change (2019-11-05 02:27:34 +0000)
are available in the Git repository at:
git://git.breakpoint.cc/fw/mptcp-next.git mptcp_fixes_04
for you to fetch changes up to 50a570bb03f61cb5e2344e1d3f8d959c40d7b520:
poll: release lock before waiting (2019-11-05 17:58:28 +0100)
----------------------------------------------------------------
Florian Westphal (4):
protocol.h: mptcp_subflow_get_map_offset arg can be const
options: ignore mptcp-level ack that is ahead of write_seq
protocol: use sk_wait_event helper
poll: release lock before waiting
net/mptcp/options.c | 6 ++++++ net/mptcp/protocol.c | 16 ++++------------ net/mptcp/protocol.h | 4 ++-- 3 files changed, 12 insertions(+), 14 deletions(-)
Florian Westphal
PATCH 1/4 protocol.h: mptcp_subflow_get_map_offset arg can be const
squashto: mptcp: Implement MPTCP receive path
--- net/mptcp/protocol.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
net/mptcp/protocol.h +2 -2
@@ -248,7 +248,7 @@ mptcp_subflow_tcp_socket(const struct mptcp_subflow_context *subflow) }static inline u64 -mptcp_subflow_get_map_offset (struct mptcp_subflow_context *subflow) +mptcp_subflow_get_map_offset (const struct mptcp_subflow_context *subflow) {return tcp_sk (mptcp_subflow_tcp_socket (subflow)->sk)->copied_seq - subflow->ssn_offset -
@@ -256,7 +256,7 @@ mptcp_subflow_get_map_offset(struct mptcp_subflow_context *subflow) }static inline u64 -mptcp_subflow_get_mapped_dsn (struct mptcp_subflow_context *subflow) +mptcp_subflow_get_mapped_dsn (const struct mptcp_subflow_context *subflow) {return subflow->map_seq +mptcp_subflow_get_map_offset (subflow); }
Signature · 1 line
2.23.0
Paolo Abeni
[MPTCP] Re: [PATCH mptcp 1/4] protocol.h: mptcp_subflow_get_map_offset arg can be const
Quoted 21 lines · Florian Westphal
squashto: mptcp: Implement MPTCP receive path
Signed-off-by Florian Westphal <fw(a)strlen.de>--- net/mptcp/protocol.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)net/mptcp/protocol.h +1
-1@@ -248,7 +248,7 @@ mptcp_subflow_tcp_socket(const struct mptcp_subflow_context *subflow) }static inline u64 -mptcp_subflow_get_map_offset (struct mptcp_subflow_context *subflow) +mptcp_subflow_get_map_offset (const struct mptcp_subflow_context *subflow) {return tcp_sk (mptcp_subflow_tcp_socket (subflow)->sk)->copied_seq - subflow->ssn_offset -
net/mptcp/protocol.h mptcp_subflow_get_map_offset(struct mptcp_subflow_context *subflow) · line 262
Quoted 8 lines
net/mptcp/protocol.h +1
-1@@ -256,7 +256,7 @@ mptcp_subflow_get_map_offset(struct mptcp_subflow_context *subflow) }static inline u64 -mptcp_subflow_get_mapped_dsn (struct mptcp_subflow_context *subflow) +mptcp_subflow_get_mapped_dsn (const struct mptcp_subflow_context *subflow) {return subflow->map_seq +mptcp_subflow_get_map_offset (subflow); }
All the 4 patches LGTM, thanks Florian!
/P
Florian Westphal
PATCH 2/4 options: ignore mptcp-level ack that is ahead of write_seq
In reply to Florian WestphalWe should not allow a peer to ack data that we haven't even sent yet, so test incoming una vs. write_seq.
squashto: mptcp: update per unacked sequence on pkt reception
--- net/mptcp/options.c | 6 ++++++ 1 file changed, 6 insertions(+)
net/mptcp/options.c +6 -0
@@ -566,12 +566,18 @@ static void update_una(struct mptcp_sock *msk, struct mptcp_options_received *mp_opt) {u64 new_snd_una, snd_una, old_snd_una =atomic64_read (&msk->snd_una); +u64 write_seq =READ_ONCE (msk->write_seq);/* avoid ack expansion on update conflict, to reduce the risk of * wrongly expanding to a future ack sequence number, which is way * more dangerous than missing an ack */ new_snd_una =expand_ack (old_snd_una, mp_opt->data_ack, mp_opt->ack64); + +/* ACK for data not even sent yet? Ignore. */ +if (after64 (new_snd_una, write_seq)) + new_snd_una = old_snd_una; +while (after64 (new_snd_una, old_snd_una)) { snd_una = old_snd_una; old_snd_una = atomic64_cmpxchg(&msk->snd_una, snd_una,
Signature · 1 line
2.23.0
Florian Westphal
PATCH 3/4 protocol: use sk_wait_event helper
In reply to Florian WestphalAvoids a bit of copy&paste code.
squashto: mptcp: recvmsg() can drain data from multiple subflows
--- net/mptcp/protocol.c | 14 ++------------ 1 file changed, 2 insertions(+), 12 deletions(-)
net/mptcp/protocol.c +2 -12
@@ -443,22 +443,12 @@ static void mptcp_wait_data(struct sock *sk, long *timeo) {DEFINE_WAIT_FUNC (wait, woken_wake_function);struct mptcp_sock *msk =mptcp_sk (sk); -int data_ready;add_wait_queue (sk_sleep (sk), &wait);sk_set_bit (SOCKWQ_ASYNC_WAITDATA , sk); -release_sock (sk); - -smp_mb__before_atomic (); - data_ready =test_and_clear_bit (MPTCP_DATA_READY , &msk->flags); -smp_mb__after_atomic (); - -if (!data_ready) - *timeo =wait_woken (&wait,TASK_INTERRUPTIBLE , *timeo); - -sched_annotate_sleep (); -lock_sock (sk); +sk_wait_event (sk, timeo, +test_and_clear_bit (MPTCP_DATA_READY , &msk->flags), &wait);sk_clear_bit (SOCKWQ_ASYNC_WAITDATA , sk);remove_wait_queue (sk_sleep (sk), &wait);
Signature · 1 line
2.23.0
Paolo Abeni
[MPTCP] Re: [PATCH mptcp 3/4] protocol: use sk_wait_event helper
Quoted 27 lines · Florian Westphal
Avoids a bit of copy&paste code.
squashto: mptcp:recvmsg()can drain data from multiple subflowsSigned-off-by Florian Westphal <fw(a)strlen.de>--- net/mptcp/protocol.c | 14 ++------------ 1 file changed, 2 insertions(+), 12 deletions(-)net/mptcp/protocol.c +0
-7@@ -443,22 +443,12 @@ static void mptcp_wait_data(struct sock *sk, long *timeo) {DEFINE_WAIT_FUNC (wait, woken_wake_function);struct mptcp_sock *msk =mptcp_sk (sk); -int data_ready;add_wait_queue (sk_sleep (sk), &wait);sk_set_bit (SOCKWQ_ASYNC_WAITDATA , sk); -release_sock (sk); - -smp_mb__before_atomic (); - data_ready =test_and_clear_bit (MPTCP_DATA_READY , &msk->flags); -smp_mb__after_atomic (); -
net/mptcp/protocol.c static void mptcp_wait_data(struct sock *sk, long *timeo) · line 454
Quoted 10 lines
net/mptcp/protocol.c +2
-5-if (!data_ready) - *timeo =wait_woken (&wait,TASK_INTERRUPTIBLE , *timeo); - -sched_annotate_sleep (); -lock_sock (sk); +sk_wait_event (sk, timeo, +test_and_clear_bit (MPTCP_DATA_READY , &msk->flags), &wait);sk_clear_bit (SOCKWQ_ASYNC_WAITDATA , sk);remove_wait_queue (sk_sleep (sk), &wait);
IIRC, sk_wait_event() was intentionally avoided to allow placing the smp_mb__{before,after}_atomic() pair around test_and_clear_bit(), which I thought was required.
Reading again the relevant documentation:
https://elixir.bootlin.com/linux/latest/source/Documentation/core-api/atomic_ops.rst#L485
it looks like test_and_clear_bit() provides already the required barriers (also grepping inside the kernel sources suggests that).
So the above should be correct, I'm sorry for being cyclothymic ;)
/P
Mat Martineau
[MPTCP] Re: [PATCH mptcp 3/4] protocol: use sk_wait_event helper
Quoted 43 lines · Paolo Abeni
Quoted 27 lines · Florian Westphal
Avoids a bit of copy&paste code.
squashto: mptcp:recvmsg()can drain data from multiple subflowsSigned-off-by Florian Westphal <fw(a)strlen.de>--- net/mptcp/protocol.c | 14 ++------------ 1 file changed, 2 insertions(+), 12 deletions(-)net/mptcp/protocol.c +0
-7@@ -443,22 +443,12 @@ static void mptcp_wait_data(struct sock *sk, long *timeo) {DEFINE_WAIT_FUNC (wait, woken_wake_function);struct mptcp_sock *msk =mptcp_sk (sk); -int data_ready;add_wait_queue (sk_sleep (sk), &wait);sk_set_bit (SOCKWQ_ASYNC_WAITDATA , sk); -release_sock (sk); - -smp_mb__before_atomic (); - data_ready =test_and_clear_bit (MPTCP_DATA_READY , &msk->flags); -smp_mb__after_atomic (); -net/mptcp/protocol.c static void mptcp_wait_data(struct sock *sk, long *timeo) · line 454
Quoted 10 lines
net/mptcp/protocol.c +2
-5-if (!data_ready) - *timeo =wait_woken (&wait,TASK_INTERRUPTIBLE , *timeo); - -sched_annotate_sleep (); -lock_sock (sk); +sk_wait_event (sk, timeo, +test_and_clear_bit (MPTCP_DATA_READY , &msk->flags), &wait);sk_clear_bit (SOCKWQ_ASYNC_WAITDATA , sk);remove_wait_queue (sk_sleep (sk), &wait);IIRC,
sk_wait_event()was intentionally avoided to allow placing the smp_mb__{before,after}_atomic() pair aroundtest_and_clear_bit(), which I thought was required.
Quoted 8 lines
Reading again the relevant documentation:
https://elixir.bootlin.com/linux/latest/source/Documentation/core-api/atomic_ops.rst#L485
it looks like
test_and_clear_bit()provides already the required barriers (also grepping inside the kernel sources suggests that).So the above should be correct, I'm sorry for being cyclothymic ;)
I agree that the barriers can be removed.
sk_wait_event is a macro that evaluates its third arg twice (but we've discussed this before:
https://lists.01.org/hyperkitty/list/mptcp(a)lists.01.org/thread/7HAWM43BLQVRBNIPV5II7C2WGTARAMDR/#7S54V7TAZUXZEUXVANQZ3OETQMPQGK6J),
but in this case the second check of the flag is likely to be an inexpensive READ_ONCE() on a hot cache line. So, the extra code was there a reason but I'm ok with merging this for maintainability purposes.
--
Mat Martineau
Intel
Florian Westphal
[MPTCP] Re: [PATCH mptcp 3/4] protocol: use sk_wait_event helper
Quoted 5 lines · Mat Martineau
sk_wait_eventis a macro that evaluates its third arg twice (but we've
discussed this before: https://lists.01.org/hyperkitty/list/mptcp(a)lists.01.org/thread/7HAWM43BLQVRBNIPV5II7C2WGTARAMDR/#7S54V7TAZUXZEUXVANQZ3OETQMPQGK6J),
but in this case the second check of the flag is likely to be an inexpensiveREAD_ONCE()on a hot cache line. So, the extra code was there a reason but
I'm ok with merging this for maintainability purposes.
We can also drop it. Its likely this needs to be rewritten again in the future.
I'd like to get rid of __tcp_poll() and make mptcp_poll standalone.
For that to work reliably its needed that the mptcp sk has a reliable indicator that there is data available.
Right now this isn't the case because the bit is cleared immediately on mptcp_recvmsg entry.
Removing the clear_bit could get us in a state where the bit is still set, but the last recvmsg did drain the last byte.
Checking the ssk read queue and clearing the bit on revmsg exit doesn't work either because we could have data available in another subflow, but walking the subflow list isn't exactly ideal.
Matthieu Baerts
[MPTCP] Re: [PATCH mptcp 3/4] protocol: use sk_wait_event helper
Hi Florian, Mat, Paolo,
Quoted 18 lines · Florian Westphal
Quoted 5 lines · Mat Martineau
sk_wait_eventis a macro that evaluates its third arg twice (but we've
discussed this before: https://lists.01.org/hyperkitty/list/mptcp(a)lists.01.org/thread/7HAWM43BLQVRBNIPV5II7C2WGTARAMDR/#7S54V7TAZUXZEUXVANQZ3OETQMPQGK6J),
but in this case the second check of the flag is likely to be an inexpensiveREAD_ONCE()on a hot cache line. So, the extra code was there a reason but
I'm ok with merging this for maintainability purposes.We can also drop it. Its likely this needs to be rewritten again in the future.
I'd like to get rid of
__tcp_poll()and makemptcp_pollstandalone.For that to work reliably its needed that the mptcp sk has a reliable indicator that there is data available.
Right now this isn't the case because the bit is cleared immediately on
mptcp_recvmsgentry.
Quoted 6 lines
Removing the
clear_bitcould get us in a state where the bit is still set, but the last recvmsg did drain the last byte.Checking the ssk read queue and clearing the bit on revmsg exit doesn't work either because we could have data available in another subflow, but walking the subflow list isn't exactly ideal.
Thank you for the patches and the reviews.
Is it then OK if I apply this series as it is?
Cheers,
Matt
Signature · 5 lines
Matthieu Baerts | R&D Engineer matthieu.baerts(a)tessares.net Tessares SA | Hybrid Access Solutions www.tessares.net 1 Avenue Jean Monnet, 1348 Louvain-la-Neuve, Belgium
Florian Westphal
[MPTCP] Re: [PATCH mptcp 3/4] protocol: use sk_wait_event helper
Quoted 2 lines · Matthieu Baerts
Thank you for the patches and the reviews.
Is it then OK if I apply this series as it is?
I think so.
Paolo Abeni
[MPTCP] Re: [PATCH mptcp 3/4] protocol: use sk_wait_event helper
In reply to Matthieu BaertsQuoted 2 lines · Matthieu Baerts
Thank you for the patches and the reviews.
Is it then OK if I apply this series as it is?
yes, I agree with that.
Cheers,
Paolo
Florian Westphal
PATCH 4/4 poll: release lock before waiting
In reply to Florian Westphalelse, this can block kworker thread forever -- it needs to lock the mptcp socket, so we should not hold this lock when waiting for events.
squashto: mptcp: add mptcp_poll
--- net/mptcp/protocol.c | 2 ++ 1 file changed, 2 insertions(+)
net/mptcp/protocol.c +2 -0
@@ -1229,7 +1229,9 @@ static __poll_t mptcp_poll(struct file *file, struct socket *sock,return ret; } +release_sock (sk);sock_poll_wait (file, sock, wait); +lock_sock (sk);mptcp_for_each_subflow (msk, subflow) {struct socket *tcp_sock;
Signature · 1 line
2.23.0
Matthieu Baerts
Hi Florian, Paolo,
Quoted 17 lines · Florian Westphal
1. cleanup:
mptcp_subflow_get_map_offsetarg can be const.
2. fix: don't advancesnd_unapast the write sequence.
3. cleanup: use existing helper rather than open-coding it
4. fix: avoid blocking kworker forever when polling mptcp socket.All paches can be squashed, I've added squashto hints.
The following changes since commit
512f0321ae8a2679c15915b68f8a5cd98f190288:subflow: wake parent mptcp socket on subflow state change (2019-11-05 02:27:34 +0000)
are available in the Git repository at:
git://git.breakpoint.cc/fw/mptcp-next.gitmptcp_fixes_04for you to fetch changes up to
50a570bb03f61cb5e2344e1d3f8d959c40d7b520:
Quoted 8 lines
poll: release lock before waiting (2019-11-05 17:58:28 +0100)
----------------------------------------------------------------
Florian Westphal (4):
protocol.h:mptcp_subflow_get_map_offsetarg can be const
options: ignore mptcp-level ack that is ahead ofwrite_seq
protocol: usesk_wait_eventhelper
poll: release lock before waiting
Thank you for the patches and the reviews!
And sorry for the delay!
- 1bf1608f7df8: "squashed" patch 1/4 in "mptcp: Implement MPTCP receive path"
- signed-off already there
- 127eee209f8e: "squashed" patch 2/4 in "mptcp: update per unacked sequence on pkt reception"
- ede08521c3e3: "Signed-off-by" + "Co-developed-by"
- a6260f7b8cab..ece8d606ffaf: result
- 8f7fc74888c8: "squashed" patch 3/4 in "mptcp: recvmsg() can drain data from multiple subflows"
- d8bfdd33c229: "Signed-off-by" + "Co-developed-by"
- ece8d606ffaf..4c99cd06d954: result
- 230846c72398: "squashed" patch 4/4 in "mptcp: add mptcp_poll"
- signed-off already there
- 4c99cd06d954..6b3052b58e78: result
Tests are in progress. The export of the branch will be done in max ~10 minutes.
(Note that some intermediate commits don't compile but I will check that!)
Cheers,
Matt
Signature · 5 lines
Matthieu Baerts | R&D Engineer matthieu.baerts(a)tessares.net Tessares SA | Hybrid Access Solutions www.tessares.net 1 Avenue Jean Monnet, 1348 Louvain-la-Neuve, Belgium