Skip to contentSkip to conversation

four small cleanups and fixes

PATCH In review Florian Westphal 4 participants 13 messages Nov 5, 2019 – Nov 8, 2019

Subsystems (2)
lore
Share & tools
lore

Florian Westphal

author
#0 lore raw

cover four small cleanups and fixes

1. cleanup: mptcp_subflow_get_map_offset arg can be const.
2. fix: don't advance snd_una past the write sequence.
3. cleanup: use existing helper rather than open-coding it
4. fix: avoid blocking kworker forever when polling mptcp socket.

All paches can be squashed, I've added squashto hints.

The following changes since commit 512f0321ae8a2679c15915b68f8a5cd98f190288:

subflow: wake parent mptcp socket on subflow state change (2019-11-05 02:27:34 +0000)

are available in the Git repository at:

git://git.breakpoint.cc/fw/mptcp-next.git mptcp_fixes_04

for you to fetch changes up to 50a570bb03f61cb5e2344e1d3f8d959c40d7b520:

poll: release lock before waiting (2019-11-05 17:58:28 +0100)

----------------------------------------------------------------
Florian Westphal (4):
protocol.h: mptcp_subflow_get_map_offset arg can be const
options: ignore mptcp-level ack that is ahead of write_seq
protocol: use sk_wait_event helper
poll: release lock before waiting

 net/mptcp/options.c  |  6 ++++++
 net/mptcp/protocol.c | 16 ++++------------
 net/mptcp/protocol.h |  4 ++--
 3 files changed, 12 insertions(+), 14 deletions(-)
author
#1 lore raw

PATCH 1/4 protocol.h: mptcp_subflow_get_map_offset arg can be const

squashto: mptcp: Implement MPTCP receive path

Signed-off-by Florian Westphal <fw(a)strlen.de>
---
 net/mptcp/protocol.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
net/mptcp/protocol.h +2 -2
index 8fe5f9383d38..46f7becd9a45 100644
@@ -248,7 +248,7 @@ mptcp_subflow_tcp_socket(const struct mptcp_subflow_context *subflow)
 }
 
 static inline u64
-mptcp_subflow_get_map_offset(struct mptcp_subflow_context *subflow)
+mptcp_subflow_get_map_offset(const struct mptcp_subflow_context *subflow)
 {
 	return tcp_sk(mptcp_subflow_tcp_socket(subflow)->sk)->copied_seq -
 		      subflow->ssn_offset -
@@ -256,7 +256,7 @@ mptcp_subflow_get_map_offset(struct mptcp_subflow_context *subflow)
 }
 
 static inline u64
-mptcp_subflow_get_mapped_dsn(struct mptcp_subflow_context *subflow)
+mptcp_subflow_get_mapped_dsn(const struct mptcp_subflow_context *subflow)
 {
 	return subflow->map_seq + mptcp_subflow_get_map_offset(subflow);
 }
Signature · 1 line
2.23.0
#2 lore raw

[MPTCP] Re: [PATCH mptcp 1/4] protocol.h: mptcp_subflow_get_map_offset arg can be const

Quoted 21 lines · Florian Westphal

squashto: mptcp: Implement MPTCP receive path

Signed-off-by Florian Westphal <fw(a)strlen.de>
---
 net/mptcp/protocol.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
net/mptcp/protocol.h +1 -1
index 8fe5f9383d38..46f7becd9a45 100644
@@ -248,7 +248,7 @@ mptcp_subflow_tcp_socket(const struct mptcp_subflow_context *subflow)
 }
 
 static inline u64
-mptcp_subflow_get_map_offset(struct mptcp_subflow_context *subflow)
+mptcp_subflow_get_map_offset(const struct mptcp_subflow_context *subflow)
 {
 	return tcp_sk(mptcp_subflow_tcp_socket(subflow)->sk)->copied_seq -
 		      subflow->ssn_offset -

net/mptcp/protocol.h mptcp_subflow_get_map_offset(struct mptcp_subflow_context *subflow) · line 262

Quoted 8 lines
net/mptcp/protocol.h +1 -1
@@ -256,7 +256,7 @@ mptcp_subflow_get_map_offset(struct mptcp_subflow_context *subflow)
 }
 
 static inline u64
-mptcp_subflow_get_mapped_dsn(struct mptcp_subflow_context *subflow)
+mptcp_subflow_get_mapped_dsn(const struct mptcp_subflow_context *subflow)
 {
 	return subflow->map_seq + mptcp_subflow_get_map_offset(subflow);
 }

All the 4 patches LGTM, thanks Florian!

/P

author
#3 lore raw

PATCH 2/4 options: ignore mptcp-level ack that is ahead of write_seq

In reply to Florian Westphal

We should not allow a peer to ack data that we haven't even sent yet, so test incoming una vs. write_seq.

squashto: mptcp: update per unacked sequence on pkt reception

Signed-off-by Florian Westphal <fw(a)strlen.de>
---
 net/mptcp/options.c | 6 ++++++
 1 file changed, 6 insertions(+)
net/mptcp/options.c +6 -0
index 1b08e1193991..80dbe7662cea 100644
@@ -566,12 +566,18 @@ static void update_una(struct mptcp_sock *msk,
 		       struct mptcp_options_received *mp_opt)
 {
 	u64 new_snd_una, snd_una, old_snd_una = atomic64_read(&msk->snd_una);
+	u64 write_seq = READ_ONCE(msk->write_seq);
 
 	/* avoid ack expansion on update conflict, to reduce the risk of
 	 * wrongly expanding to a future ack sequence number, which is way
 	 * more dangerous than missing an ack
 	 */
 	new_snd_una = expand_ack(old_snd_una, mp_opt->data_ack, mp_opt->ack64);
+
+	/* ACK for data not even sent yet? Ignore. */
+	if (after64(new_snd_una, write_seq))
+		new_snd_una = old_snd_una;
+
 	while (after64(new_snd_una, old_snd_una)) {
 		snd_una = old_snd_una;
 		old_snd_una = atomic64_cmpxchg(&msk->snd_una, snd_una,
Signature · 1 line
2.23.0
author
#4 lore raw

PATCH 3/4 protocol: use sk_wait_event helper

In reply to Florian Westphal

Avoids a bit of copy&paste code.
squashto: mptcp: recvmsg() can drain data from multiple subflows

Signed-off-by Florian Westphal <fw(a)strlen.de>
---
 net/mptcp/protocol.c | 14 ++------------
 1 file changed, 2 insertions(+), 12 deletions(-)
net/mptcp/protocol.c +2 -12
index 3af5f121af9e..8cc9cc5675c2 100644
@@ -443,22 +443,12 @@ static void mptcp_wait_data(struct sock *sk, long *timeo)
 {
 	DEFINE_WAIT_FUNC(wait, woken_wake_function);
 	struct mptcp_sock *msk = mptcp_sk(sk);
-	int data_ready;
 
 	add_wait_queue(sk_sleep(sk), &wait);
 	sk_set_bit(SOCKWQ_ASYNC_WAITDATA, sk);
 
-	release_sock(sk);
-
-	smp_mb__before_atomic();
-	data_ready = test_and_clear_bit(MPTCP_DATA_READY, &msk->flags);
-	smp_mb__after_atomic();
-
-	if (!data_ready)
-		*timeo = wait_woken(&wait, TASK_INTERRUPTIBLE, *timeo);
-
-	sched_annotate_sleep();
-	lock_sock(sk);
+	sk_wait_event(sk, timeo,
+		      test_and_clear_bit(MPTCP_DATA_READY, &msk->flags), &wait);
 
 	sk_clear_bit(SOCKWQ_ASYNC_WAITDATA, sk);
 	remove_wait_queue(sk_sleep(sk), &wait);
Signature · 1 line
2.23.0
#5 lore raw

[MPTCP] Re: [PATCH mptcp 3/4] protocol: use sk_wait_event helper

Quoted 27 lines · Florian Westphal

Avoids a bit of copy&paste code.
squashto: mptcp: recvmsg() can drain data from multiple subflows

Signed-off-by Florian Westphal <fw(a)strlen.de>
---
 net/mptcp/protocol.c | 14 ++------------
 1 file changed, 2 insertions(+), 12 deletions(-)
net/mptcp/protocol.c +0 -7
index 3af5f121af9e..8cc9cc5675c2 100644
@@ -443,22 +443,12 @@ static void mptcp_wait_data(struct sock *sk, long *timeo)
 {
 	DEFINE_WAIT_FUNC(wait, woken_wake_function);
 	struct mptcp_sock *msk = mptcp_sk(sk);
-	int data_ready;
 
 	add_wait_queue(sk_sleep(sk), &wait);
 	sk_set_bit(SOCKWQ_ASYNC_WAITDATA, sk);
 
-	release_sock(sk);
-
-	smp_mb__before_atomic();
-	data_ready = test_and_clear_bit(MPTCP_DATA_READY, &msk->flags);
-	smp_mb__after_atomic();
-

net/mptcp/protocol.c static void mptcp_wait_data(struct sock *sk, long *timeo) · line 454

Quoted 10 lines
net/mptcp/protocol.c +2 -5
-	if (!data_ready)
-		*timeo = wait_woken(&wait, TASK_INTERRUPTIBLE, *timeo);
-
-	sched_annotate_sleep();
-	lock_sock(sk);
+	sk_wait_event(sk, timeo,
+		      test_and_clear_bit(MPTCP_DATA_READY, &msk->flags), &wait);
 
 	sk_clear_bit(SOCKWQ_ASYNC_WAITDATA, sk);
 	remove_wait_queue(sk_sleep(sk), &wait);

IIRC, sk_wait_event() was intentionally avoided to allow placing the smp_mb__{before,after}_atomic() pair around test_and_clear_bit(), which I thought was required.

Reading again the relevant documentation:

https://elixir.bootlin.com/linux/latest/source/Documentation/core-api/atomic_ops.rst#L485

it looks like test_and_clear_bit() provides already the required barriers (also grepping inside the kernel sources suggests that).

So the above should be correct, I'm sorry for being cyclothymic ;)

/P

#6 lore raw

[MPTCP] Re: [PATCH mptcp 3/4] protocol: use sk_wait_event helper

Quoted 43 lines · Paolo Abeni
Quoted 27 lines · Florian Westphal

Avoids a bit of copy&paste code.
squashto: mptcp: recvmsg() can drain data from multiple subflows

Signed-off-by Florian Westphal <fw(a)strlen.de>
---
 net/mptcp/protocol.c | 14 ++------------
 1 file changed, 2 insertions(+), 12 deletions(-)
net/mptcp/protocol.c +0 -7
index 3af5f121af9e..8cc9cc5675c2 100644
@@ -443,22 +443,12 @@ static void mptcp_wait_data(struct sock *sk, long *timeo)
 {
 	DEFINE_WAIT_FUNC(wait, woken_wake_function);
 	struct mptcp_sock *msk = mptcp_sk(sk);
-	int data_ready;
 
 	add_wait_queue(sk_sleep(sk), &wait);
 	sk_set_bit(SOCKWQ_ASYNC_WAITDATA, sk);
 
-	release_sock(sk);
-
-	smp_mb__before_atomic();
-	data_ready = test_and_clear_bit(MPTCP_DATA_READY, &msk->flags);
-	smp_mb__after_atomic();
-

net/mptcp/protocol.c static void mptcp_wait_data(struct sock *sk, long *timeo) · line 454

Quoted 10 lines
net/mptcp/protocol.c +2 -5
-	if (!data_ready)
-		*timeo = wait_woken(&wait, TASK_INTERRUPTIBLE, *timeo);
-
-	sched_annotate_sleep();
-	lock_sock(sk);
+	sk_wait_event(sk, timeo,
+		      test_and_clear_bit(MPTCP_DATA_READY, &msk->flags), &wait);
 
 	sk_clear_bit(SOCKWQ_ASYNC_WAITDATA, sk);
 	remove_wait_queue(sk_sleep(sk), &wait);

IIRC, sk_wait_event() was intentionally avoided to allow placing the smp_mb__{before,after}_atomic() pair around test_and_clear_bit(), which I thought was required.

Quoted 8 lines

Reading again the relevant documentation:

https://elixir.bootlin.com/linux/latest/source/Documentation/core-api/atomic_ops.rst#L485

it looks like test_and_clear_bit() provides already the required barriers (also grepping inside the kernel sources suggests that).

So the above should be correct, I'm sorry for being cyclothymic ;)

I agree that the barriers can be removed.

sk_wait_event is a macro that evaluates its third arg twice (but we've discussed this before:
https://lists.01.org/hyperkitty/list/mptcp(a)lists.01.org/thread/7HAWM43BLQVRBNIPV5II7C2WGTARAMDR/#7S54V7TAZUXZEUXVANQZ3OETQMPQGK6J),
but in this case the second check of the flag is likely to be an inexpensive READ_ONCE() on a hot cache line. So, the extra code was there a reason but I'm ok with merging this for maintainability purposes.

--
Mat Martineau
Intel

author
#7 lore raw

[MPTCP] Re: [PATCH mptcp 3/4] protocol: use sk_wait_event helper

Quoted 5 lines · Mat Martineau

sk_wait_event is a macro that evaluates its third arg twice (but we've
discussed this before: https://lists.01.org/hyperkitty/list/mptcp(a)lists.01.org/thread/7HAWM43BLQVRBNIPV5II7C2WGTARAMDR/#7S54V7TAZUXZEUXVANQZ3OETQMPQGK6J),
but in this case the second check of the flag is likely to be an inexpensive
READ_ONCE() on a hot cache line. So, the extra code was there a reason but
I'm ok with merging this for maintainability purposes.

We can also drop it. Its likely this needs to be rewritten again in the future.

I'd like to get rid of __tcp_poll() and make mptcp_poll standalone.

For that to work reliably its needed that the mptcp sk has a reliable indicator that there is data available.

Right now this isn't the case because the bit is cleared immediately on mptcp_recvmsg entry.

Removing the clear_bit could get us in a state where the bit is still set, but the last recvmsg did drain the last byte.

Checking the ssk read queue and clearing the bit on revmsg exit doesn't work either because we could have data available in another subflow, but walking the subflow list isn't exactly ideal.

#8 lore raw

[MPTCP] Re: [PATCH mptcp 3/4] protocol: use sk_wait_event helper

Hi Florian, Mat, Paolo,

Quoted 18 lines · Florian Westphal
Quoted 5 lines · Mat Martineau

sk_wait_event is a macro that evaluates its third arg twice (but we've
discussed this before: https://lists.01.org/hyperkitty/list/mptcp(a)lists.01.org/thread/7HAWM43BLQVRBNIPV5II7C2WGTARAMDR/#7S54V7TAZUXZEUXVANQZ3OETQMPQGK6J),
but in this case the second check of the flag is likely to be an inexpensive
READ_ONCE() on a hot cache line. So, the extra code was there a reason but
I'm ok with merging this for maintainability purposes.

We can also drop it. Its likely this needs to be rewritten again in the future.

I'd like to get rid of __tcp_poll() and make mptcp_poll standalone.

For that to work reliably its needed that the mptcp sk has a reliable indicator that there is data available.

Right now this isn't the case because the bit is cleared immediately on mptcp_recvmsg entry.

Quoted 6 lines

Removing the clear_bit could get us in a state where the bit is still set, but the last recvmsg did drain the last byte.

Checking the ssk read queue and clearing the bit on revmsg exit doesn't work either because we could have data available in another subflow, but walking the subflow list isn't exactly ideal.

Thank you for the patches and the reviews.
Is it then OK if I apply this series as it is?

Cheers,
Matt

Signature · 5 lines
Matthieu Baerts | R&D Engineer
matthieu.baerts(a)tessares.net
Tessares SA | Hybrid Access Solutions
www.tessares.net
1 Avenue Jean Monnet, 1348 Louvain-la-Neuve, Belgium
author
#9 lore raw

[MPTCP] Re: [PATCH mptcp 3/4] protocol: use sk_wait_event helper

Quoted 2 lines · Matthieu Baerts

Thank you for the patches and the reviews.
Is it then OK if I apply this series as it is?

I think so.

#10 lore raw

[MPTCP] Re: [PATCH mptcp 3/4] protocol: use sk_wait_event helper

In reply to Matthieu Baerts
Quoted 2 lines · Matthieu Baerts

Thank you for the patches and the reviews.
Is it then OK if I apply this series as it is?

yes, I agree with that.

Cheers,

Paolo

author
#11 lore raw

PATCH 4/4 poll: release lock before waiting

In reply to Florian Westphal

else, this can block kworker thread forever -- it needs to lock the mptcp socket, so we should not hold this lock when waiting for events.

squashto: mptcp: add mptcp_poll

Signed-off-by Florian Westphal <fw(a)strlen.de>
---
 net/mptcp/protocol.c | 2 ++
 1 file changed, 2 insertions(+)
net/mptcp/protocol.c +2 -0
index 8cc9cc5675c2..53b1b015d9d4 100644
@@ -1229,7 +1229,9 @@ static __poll_t mptcp_poll(struct file *file, struct socket *sock,
 		return ret;
 	}
 
+	release_sock(sk);
 	sock_poll_wait(file, sock, wait);
+	lock_sock(sk);
 
 	mptcp_for_each_subflow(msk, subflow) {
 		struct socket *tcp_sock;
Signature · 1 line
2.23.0
#12 lore raw
In reply to Florian Westphal

Hi Florian, Paolo,

Quoted 17 lines · Florian Westphal

1. cleanup: mptcp_subflow_get_map_offset arg can be const.
2. fix: don't advance snd_una past the write sequence.
3. cleanup: use existing helper rather than open-coding it
4. fix: avoid blocking kworker forever when polling mptcp socket.

All paches can be squashed, I've added squashto hints.

The following changes since commit 512f0321ae8a2679c15915b68f8a5cd98f190288:

subflow: wake parent mptcp socket on subflow state change (2019-11-05 02:27:34 +0000)

are available in the Git repository at:

git://git.breakpoint.cc/fw/mptcp-next.git mptcp_fixes_04

for you to fetch changes up to 50a570bb03f61cb5e2344e1d3f8d959c40d7b520:

Quoted 8 lines

poll: release lock before waiting (2019-11-05 17:58:28 +0100)

----------------------------------------------------------------
Florian Westphal (4):
protocol.h: mptcp_subflow_get_map_offset arg can be const
options: ignore mptcp-level ack that is ahead of write_seq
protocol: use sk_wait_event helper
poll: release lock before waiting

Thank you for the patches and the reviews!
And sorry for the delay!

- 1bf1608f7df8: "squashed" patch 1/4 in "mptcp: Implement MPTCP receive path"
- signed-off already there
- 127eee209f8e: "squashed" patch 2/4 in "mptcp: update per unacked sequence on pkt reception"
- ede08521c3e3: "Signed-off-by" + "Co-developed-by"
- a6260f7b8cab..ece8d606ffaf: result
- 8f7fc74888c8: "squashed" patch 3/4 in "mptcp: recvmsg() can drain data from multiple subflows"
- d8bfdd33c229: "Signed-off-by" + "Co-developed-by"
- ece8d606ffaf..4c99cd06d954: result
- 230846c72398: "squashed" patch 4/4 in "mptcp: add mptcp_poll"
- signed-off already there
- 4c99cd06d954..6b3052b58e78: result

Tests are in progress. The export of the branch will be done in max ~10 minutes.
(Note that some intermediate commits don't compile but I will check that!)

Cheers,
Matt

Signature · 5 lines
Matthieu Baerts | R&D Engineer
matthieu.baerts(a)tessares.net
Tessares SA | Hybrid Access Solutions
www.tessares.net
1 Avenue Jean Monnet, 1348 Louvain-la-Neuve, Belgium