Skip to contentSkip to conversation

mptcp: cope better with tcp_sendpage allocation

RFC PATCH In review Paolo Abeni 2 participants 6 messages Oct 6, 2020 – Oct 7, 2020

Subsystems (3)
lore
Share & tools
lore

Paolo Abeni

author
#0 lore raw

cover mptcp: cope better with tcp_sendpage allocation

The current export branch suffers of the following issue:

https://github.com/multipath-tcp/mptcp_net-next/issues/97

To fix the above, MPTCP needs to be able to cope with memory allocation failure in do_tcp_sendpages().

In this attempt, the relevant code is factored out of do_tcp_sendpages(), in a smaller helper, so that mptcp_sendmsg_frag() can avoid touching ssk->sk_socket.

There are other possible alternatives:
- add an explicit check in do_tcp_sendpages() for NULL sk->sk_socket
pro: very simple and little invasive
cons: hackysh?!?
- inline/open-code the relevant part of do_tcp_sendpages in mptcp_sendmsg_frag()
pro: no need to change core TCP code
cons: code duplication, need to change the scope for a couple of TCP internal
helpers (removing the 'static' keyword)

As a follow-up I would try to pre-allocate skbs and extensions, so that push_pending could be called in BH context and we could avoid using the work queue.

I'm looking for feedback on the less ugly/more upstreamable approach

Paolo Abeni (2):
tcp: factor out do_tcp_sendfrag()
mptcp: use do_tcp_sendfrag()

 include/net/tcp.h    |   3 ++
 net/ipv4/tcp.c       | 120 ++++++++++++++++++++++++-------------------
 net/mptcp/protocol.c |  21 ++++----
 3 files changed, 80 insertions(+), 64 deletions(-)
Signature · 1 line
2.26.2
author
#1 lore raw

RFC PATCH 1/2 tcp: factor out do_tcp_sendfrag()

Will be needed by the next patch, as MPTCP needs to handle directly the error/memory-allocation-needed patch.

No functional changes intented.

Signed-off-by Paolo Abeni <pabeni(a)redhat.com>
---
 include/net/tcp.h |   3 ++
 net/ipv4/tcp.c    | 120 ++++++++++++++++++++++++++--------------------
 2 files changed, 71 insertions(+), 52 deletions(-)
include/net/tcp.h +3 -0
index f2a0ff60da25..9d62635e7167 100644
@@ -322,6 +322,7 @@ void tcp_shutdown(struct sock *sk, int how);
 int tcp_v4_early_demux(struct sk_buff *skb);
 int tcp_v4_rcv(struct sk_buff *skb);
 
+void tcp_remove_empty_skb(struct sock *sk, struct sk_buff *skb);
 int tcp_v4_tw_remember_stamp(struct inet_timewait_sock *tw);
 int tcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t size);
 int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size);
@@ -329,6 +330,8 @@ int tcp_sendpage(struct sock *sk, struct page *page, int offset, size_t size,
 		 int flags);
 int tcp_sendpage_locked(struct sock *sk, struct page *page, int offset,
 			size_t size, int flags);
+int do_tcp_sendfrag(struct sock *sk, struct page *page, int offset,
+		    size_t size, int size_goal, int flags);
 ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset,
 		 size_t size, int flags);
 int tcp_send_mss(struct sock *sk, int *size_goal, int flags);
net/ipv4/tcp.c +68 -52

141 lines not on this page · raw message

Signature · 1 line
2.26.2
author
#2 lore raw

RFC PATCH 2/2 mptcp: use do_tcp_sendfrag()

In reply to Paolo Abeni

mptcp_push_pending() is called even on orphaned msk (and orphaned subflows), if there is outstanding data at close() time.

To cope with the above MPTCP needs to handle explicitly the allocation failure on xmit. The newly introduced do_tcp_sendfrag() allows that, just plug it.

We can additionally drop a couple of sanity checks, duplicate in the TCP code.

Signed-off-by Paolo Abeni <pabeni(a)redhat.com>
---
 net/mptcp/protocol.c | 21 +++++++++------------
 1 file changed, 9 insertions(+), 12 deletions(-)
net/mptcp/protocol.c +9 -12
index 7d5905968fac..21b56835f151 100644
@@ -958,7 +958,6 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,
 	struct sk_buff *skb, *tail;
 	bool can_collapse = false;
 	int avail_size, ret;
-	size_t psize;
 
 	pr_debug("msk=%p ssk=%p sending dfrag at seq=%lld len=%d already sent=%d",
 		 msk, ssk, dfrag->data_seq, dfrag->data_len, info->sent);
@@ -997,23 +996,21 @@ static int mptcp_sendmsg_frag(struct sock *sk, struct sock *ssk,
 			 info->limit > dfrag->data_len))
 		return 0;
 
-	psize = min_t(size_t, info->limit - info->sent, avail_size);
-
 	/* tell the TCP stack to delay the push so that we can safely
 	 * access the skb after the sendpages call
 	 */
-	ret = do_tcp_sendpages(ssk, dfrag->page, dfrag->offset + info->sent,
-			       psize, info->flags | MSG_SENDPAGE_NOTLAST | MSG_DONTWAIT);
-	if (ret <= 0)
-		return ret;
+	ret = do_tcp_sendfrag(ssk, dfrag->page, dfrag->offset + info->sent,
+			      info->limit - info->sent, avail_size, info->flags);
+	if (ret == -ENOMEM) {
+		tcp_remove_empty_skb(sk, tcp_write_queue_tail(sk));
+		return 0;
+	}
 
-	/* if the tail skb extension is still the cached one, collapsing
-	 * really happened. Note: we can't check for 'same skb' as the sk_buff
-	 * hdr on tail can be transmitted, freed and re-allocated by the
-	 * do_tcp_sendpages() call
+	/* if the tail skb is still the cached one, collapsing
+	 * really happened.
 	 */
 	tail = tcp_write_queue_tail(ssk);
-	if (mpext && tail && mpext == skb_ext_find(tail, SKB_EXT_MPTCP)) {
+	if (tail == skb) {
 		WARN_ON_ONCE(!can_collapse);
 		mpext->data_len += ret;
 		WARN_ON_ONCE(zero_window_probe);
Signature · 1 line
2.26.2
#3 lore raw
In reply to Paolo Abeni
Quoted 20 lines · Paolo Abeni

The current export branch suffers of the following issue:

https://github.com/multipath-tcp/mptcp_net-next/issues/97

To fix the above, MPTCP needs to be able to cope with memory allocation failure in do_tcp_sendpages().

In this attempt, the relevant code is factored out of do_tcp_sendpages(), in a smaller helper, so that mptcp_sendmsg_frag() can avoid touching ssk->sk_socket.

There are other possible alternatives:
- add an explicit check in do_tcp_sendpages() for NULL sk->sk_socket
pro: very simple and little invasive
cons: hackysh?!?
- inline/open-code the relevant part of do_tcp_sendpages in mptcp_sendmsg_frag()
pro: no need to change core TCP code
cons: code duplication, need to change the scope for a couple of TCP internal
helpers (removing the 'static' keyword)

Quoted 5 lines

As a follow-up I would try to pre-allocate skbs and extensions, so that push_pending could be called in BH context and we could avoid using the work queue.

I'm looking for feedback on the less ugly/more upstreamable approach

I wouldn't object to the explicit sk->sk_socket check, it is simpler and handling that in the existing do_error path in do_tcp_sendpages() does make sense. It is, after all, an error that can be detected and handled.

I think factoring out do_tcp_sendfrag() is "nice" enough - while it seems like such a change might be preferred earlier in the net-next development cycle it doesn't seem too invasive. The drawback I see is that there are not likely to be other users of do_tcp_sendfrag().

Would like to avoid open-coding part of the do_tcp_sendpages() code in mptcp and having to maintain that.

Mat

Quoted 16 lines

Paolo Abeni (2):
tcp: factor out do_tcp_sendfrag()
mptcp: use do_tcp_sendfrag()

include/net/tcp.h    |   3 ++
net/ipv4/tcp.c       | 120 ++++++++++++++++++++++++-------------------
net/mptcp/protocol.c |  21 ++++----
3 files changed, 80 insertions(+), 64 deletions(-)
Signature · 4 lines
2.26.2
_______________________________________________
mptcp mailing list -- mptcp(a)lists.01.org
To unsubscribe send an email to mptcp-leave(a)lists.01.org

--
Mat Martineau
Intel

author
#4 lore raw
Quoted 28 lines · Mat Martineau
Quoted 20 lines · Paolo Abeni

The current export branch suffers of the following issue:

https://github.com/multipath-tcp/mptcp_net-next/issues/97

To fix the above, MPTCP needs to be able to cope with memory allocation failure in do_tcp_sendpages().

In this attempt, the relevant code is factored out of do_tcp_sendpages(), in a smaller helper, so that mptcp_sendmsg_frag() can avoid touching ssk->sk_socket.

There are other possible alternatives:
- add an explicit check in do_tcp_sendpages() for NULL sk->sk_socket
pro: very simple and little invasive
cons: hackysh?!?
- inline/open-code the relevant part of do_tcp_sendpages in mptcp_sendmsg_frag()
pro: no need to change core TCP code
cons: code duplication, need to change the scope for a couple of TCP internal
helpers (removing the 'static' keyword)

Quoted 5 lines

As a follow-up I would try to pre-allocate skbs and extensions, so that push_pending could be called in BH context and we could avoid using the work queue.

I'm looking for feedback on the less ugly/more upstreamable approach

Quoted 8 lines

I wouldn't object to the explicit sk->sk_socket check, it is simpler and handling that in the existing do_error path in do_tcp_sendpages() does make sense. It is, after all, an error that can be detected and handled.

I think factoring out do_tcp_sendfrag() is "nice" enough - while it seems like such a change might be preferred earlier in the net-next development cycle it doesn't seem too invasive. The drawback I see is that there are not likely to be other users of do_tcp_sendfrag().

Thank you for the review and feedback!

I would avoid adding mptcp-specific test in the TCP code, if possible, so my proposal is as follow:

I'll rebase the patches to land on the "bottom" of the export branch - that is, just on top of the net-next branch, so that we could send them upstream ASAP - still on the next wnd, likely. Perhpas I can send them as RFC upstream while net-next will be closed.

I'm wondering about changing do_tcp_sendfrag() signature to:

struct sk_buff *
do_tcp_sendfrag(struct sock *sk, int size_goal, int flags,

		struct page *page, int offset, int *size);

that is, first socket related fields, than buffer related fields. It will return the used skb, so that the caller will not need the additional tcp_write_queue_tail(), and will use the 'size' argument as an in-out one. On return will contains the copied bytes value - this later change to avoid increasing the number of arguments for this function.

Overall it should produce smaller code in both the TCP and MPTCP protocol, WDYT?

Quoted 2 lines

Would like to avoid open-coding part of the do_tcp_sendpages() code in mptcp and having to maintain that.

Agreed ;)

Thanks!

Paolo

#5 lore raw
Quoted 61 lines · Paolo Abeni
Quoted 28 lines · Mat Martineau
Quoted 20 lines · Paolo Abeni

The current export branch suffers of the following issue:

https://github.com/multipath-tcp/mptcp_net-next/issues/97

To fix the above, MPTCP needs to be able to cope with memory allocation failure in do_tcp_sendpages().

In this attempt, the relevant code is factored out of do_tcp_sendpages(), in a smaller helper, so that mptcp_sendmsg_frag() can avoid touching ssk->sk_socket.

There are other possible alternatives:
- add an explicit check in do_tcp_sendpages() for NULL sk->sk_socket
pro: very simple and little invasive
cons: hackysh?!?
- inline/open-code the relevant part of do_tcp_sendpages in mptcp_sendmsg_frag()
pro: no need to change core TCP code
cons: code duplication, need to change the scope for a couple of TCP internal
helpers (removing the 'static' keyword)

Quoted 5 lines

As a follow-up I would try to pre-allocate skbs and extensions, so that push_pending could be called in BH context and we could avoid using the work queue.

I'm looking for feedback on the less ugly/more upstreamable approach

Quoted 8 lines

I wouldn't object to the explicit sk->sk_socket check, it is simpler and handling that in the existing do_error path in do_tcp_sendpages() does make sense. It is, after all, an error that can be detected and handled.

I think factoring out do_tcp_sendfrag() is "nice" enough - while it seems like such a change might be preferred earlier in the net-next development cycle it doesn't seem too invasive. The drawback I see is that there are not likely to be other users of do_tcp_sendfrag().

Thank you for the review and feedback!

I would avoid adding mptcp-specific test in the TCP code, if possible, so my proposal is as follow:

I'll rebase the patches to land on the "bottom" of the export branch - that is, just on top of the net-next branch, so that we could send them upstream ASAP - still on the next wnd, likely. Perhpas I can send them as RFC upstream while net-next will be closed.

I'm wondering about changing do_tcp_sendfrag() signature to:

struct sk_buff *
do_tcp_sendfrag(struct sock *sk, int size_goal, int flags,

		struct page *page, int offset, int *size);

that is, first socket related fields, than buffer related fields. It will return the used skb, so that the caller will not need the additional tcp_write_queue_tail(), and will use the 'size' argument as an in-out one. On return will contains the copied bytes value - this later change to avoid increasing the number of arguments for this function.

Quoted 2 lines

Overall it should produce smaller code in both the TCP and MPTCP protocol, WDYT?

That has some similarities with what I wanted to do in the early drafts of the sendmsg code, but I was being very (overly?) paranoid about not touching TCP code. A good proposal in my opinion!

--
Mat Martineau
Intel