Skip to contentSkip to conversation

MPTCP: improve fallback to TCP

PATCH In review Davide Caratti 3 participants 14 messages Jun 29, 2020 – Jun 30, 2020

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover MPTCP: improve fallback to TCP

there are situations where MPTCP sockets should fall-back to regular TCP:
this series reworks the fallback code to pursue the following goals:

1) cleanup the non fallback code, removing most of 'if (<fallback>)' in the data path
2) improve performance for non-fallback sockets, avoiding locks in poll()

further work will also leverage on this changes to achieve:

a) more consistent behavior of gestockopt()/setsockopt() on passive sockets after fallback
b) support for "infinite maps" as per RFC8684, section 3.7

the series is made of the following items:

- patch 1 lets sendmsg() / recvmsg() / poll() use the main socket also after fallback
- patch 2 fixes 'simultaneous connect' scenario after fallback. The problem was present also before the rework, but the fix is much easier to implement after patch 1
- patch 3, 4, 5 are clean-ups for code that is no more needed after the fallback rework
- patch 6 fixes a race condition between close() and poll(). The problem was theoretically present before the rework, but it became almost systematic after patch 1

Davide Caratti (2):
net: mptcp: improve fallback to TCP
mptcp: fallback in case of simultaneous connect

Paolo Abeni (4):
mptcp: check for plain TCP sock at accept time
mptcp: create first subflow at msk creation time
mptcp: __mptcp_tcp_fallback() returns a struct sock
mptcp: close poll() races

 net/mptcp/options.c  |   9 +-
 net/mptcp/protocol.c | 267 ++++++++++++++-----------------------------
 net/mptcp/protocol.h |  43 +++++++
 net/mptcp/subflow.c  |  57 ++++++---
 4 files changed, 175 insertions(+), 201 deletions(-)
Signature · 1 line
2.26.2
#1 lore raw

PATCH 1/6 net: mptcp: improve fallback to TCP

Keep using MPTCP sockets and a use "dummy mapping" in case of fallback to regular TCP. When fallback is triggered, skip addition of the MPTCP option on send.

Co-developed-by Paolo Abeni <pabeni(a)redhat.com>
Signed-off-by Paolo Abeni <pabeni(a)redhat.com>
Signed-off-by Davide Caratti <dcaratti(a)redhat.com>
---
 net/mptcp/options.c  |  9 +++-
 net/mptcp/protocol.c | 98 ++++++++++++--------------------------------
 net/mptcp/protocol.h | 33 +++++++++++++++
 net/mptcp/subflow.c  | 47 +++++++++++++--------
 4 files changed, 98 insertions(+), 89 deletions(-)
net/mptcp/options.c +8 -1
index df9a51425c6f..b96d3660562f 100644
@@ -624,6 +624,9 @@ bool mptcp_established_options(struct sock *sk, struct sk_buff *skb,
 
 	opts->suboptions = 0;
 
+	if (unlikely(mptcp_check_fallback(sk)))
+		return false;
+
 	if (mptcp_established_options_mp(sk, skb, &opt_size, remaining, opts))
 		ret = true;
 	else if (mptcp_established_options_dss(sk, skb, &opt_size, remaining,
@@ -714,7 +717,8 @@ static bool check_fully_established(struct mptcp_sock *msk, struct sock *sk,
 	 */
 	if (!mp_opt->mp_capable) {
 		subflow->mp_capable = 0;
-		tcp_sk(sk)->is_mptcp = 0;
+		pr_fallback(msk);
+		__mptcp_do_fallback(msk);
 		return false;
 	}
 
@@ -814,6 +818,9 @@ void mptcp_incoming_options(struct sock *sk, struct sk_buff *skb,
 	struct mptcp_options_received mp_opt;
 	struct mptcp_ext *mpext;
 
+	if (__mptcp_check_fallback(msk))
+		return;
+
 	mptcp_get_options(skb, &mp_opt);
 	if (!check_fully_established(msk, sk, subflow, skb, &mp_opt))
 		return;
net/mptcp/protocol.c +26 -72

210 lines not on this page · raw message

net/mptcp/protocol.h +33 -0
index c05552e5fa23..a709df659ae0 100644
@@ -89,6 +89,7 @@
 #define MPTCP_SEND_SPACE	1
 #define MPTCP_WORK_RTX		2
 #define MPTCP_WORK_EOF		3
+#define MPTCP_FALLBACK_DONE	4
 
 struct mptcp_options_received {
 	u64	sndr_key;
@@ -457,4 +458,36 @@ static inline bool before64(__u64 seq1, __u64 seq2)
 
 void mptcp_diag_subflow_init(struct tcp_ulp_ops *ops);
 
+static inline bool __mptcp_check_fallback(struct mptcp_sock *msk)
+{
+	return test_bit(MPTCP_FALLBACK_DONE, &msk->flags);
+}
+
+static inline bool mptcp_check_fallback(struct sock *sk)
+{
+	struct mptcp_subflow_context *subflow = mptcp_subflow_ctx(sk);
+	struct mptcp_sock *msk = mptcp_sk(subflow->conn);
+
+	return __mptcp_check_fallback(msk);
+}
+
+static inline void __mptcp_do_fallback(struct mptcp_sock *msk)
+{
+	if (test_bit(MPTCP_FALLBACK_DONE, &msk->flags)) {
+		pr_debug("TCP fallback already done (msk=%p)", msk);
+		return;
+	}
+	set_bit(MPTCP_FALLBACK_DONE, &msk->flags);
+}
+
+static inline void mptcp_do_fallback(struct sock *sk)
+{
+	struct mptcp_subflow_context *subflow = mptcp_subflow_ctx(sk);
+	struct mptcp_sock *msk = mptcp_sk(subflow->conn);
+
+	__mptcp_do_fallback(msk);
+}
+
+#define pr_fallback(a) pr_debug("%s:fallback to TCP (msk=%p)", __func__, a)
+
 #endif /* __MPTCP_PROTOCOL_H */
net/mptcp/subflow.c +31 -16

114 lines not on this page · raw message

Signature · 1 line
2.26.2
Reviewed-by on v1 1/6
#2 lore raw

Re: [PATCH net-next 1/6] net: mptcp: improve fallback to TCP

Quoted 9 lines · Davide Caratti

Keep using MPTCP sockets and a use "dummy mapping" in case of fallback to regular TCP. When fallback is triggered, skip addition of the MPTCP option on send.

Co-developed-by Paolo Abeni <pabeni(a)redhat.com>
Signed-off-by Paolo Abeni <pabeni(a)redhat.com>
Signed-off-by Davide Caratti <dcaratti(a)redhat.com>
Quoted 7 lines

---

net/mptcp/options.c  |  9 +++-
net/mptcp/protocol.c | 98 ++++++++++++--------------------------------
net/mptcp/protocol.h | 33 +++++++++++++++
net/mptcp/subflow.c  | 47 +++++++++++++--------
4 files changed, 98 insertions(+), 89 deletions(-)

Thanks Davide and Paolo! Appreciate the simplification of the fallback code.

Reviewed-by Mat Martineau <mathew.j.martineau(a)linux.intel.com>

--
Mat Martineau
Intel

#3 lore raw

PATCH 2/6 mptcp: fallback in case of simultaneous connect

In reply to Davide Caratti

when a MPTCP client tries to connect to itself, tcp_finish_connect() is never reached. Because of this, depending on the socket current state, multiple faulty behaviours can be observed:

1) a WARN_ON() in subflow_data_ready() is hit

 WARNING: CPU: 2 PID: 882 at net/mptcp/subflow.c:911 subflow_data_ready+0x18b/0x230
 [...]
 CPU: 2 PID: 882 Comm: gh35 Not tainted 5.7.0+ #187
 [...]
 RIP: 0010:subflow_data_ready+0x18b/0x230
 [...]
 Call Trace:
  tcp_data_queue+0xd2f/0x4250
  tcp_rcv_state_process+0xb1c/0x49d3
  tcp_v4_do_rcv+0x2bc/0x790
  __release_sock+0x153/0x2d0
  release_sock+0x4f/0x170
  mptcp_shutdown+0x167/0x4e0
  __sys_shutdown+0xe6/0x180
  __x64_sys_shutdown+0x50/0x70
  do_syscall_64+0x9a/0x370
  entry_SYSCALL_64_after_hwframe+0x44/0xa9

2) client is stuck forever in mptcp_sendmsg() because the socket is not TCP_ESTABLISHED

crash> bt 4847
PID: 4847 TASK: ffff88814b2fb100 CPU: 1 COMMAND: "gh35"
#0 [ffff8881376ff680] __schedule at ffffffff97248da4
#1 [ffff8881376ff778] schedule at ffffffff9724a34f
#2 [ffff8881376ff7a0] schedule_timeout at ffffffff97252ba0 #3 [ffff8881376ff8a8] wait_woken at ffffffff958ab4ba
#4 [ffff8881376ff940] sk_stream_wait_connect at ffffffff96c2d859 #5 [ffff8881376ffa28] mptcp_sendmsg at ffffffff97207fca
#6 [ffff8881376ffbc0] sock_sendmsg at ffffffff96be1b5b
#7 [ffff8881376ffbe8] sock_write_iter at ffffffff96be1daa
#8 [ffff8881376ffce8] new_sync_write at ffffffff95e5cb52
#9 [ffff8881376ffe50] vfs_write at ffffffff95e6547f
#10 [ffff8881376ffe90] ksys_write at ffffffff95e65d26
#11 [ffff8881376fff28] do_syscall_64 at ffffffff956088ba
#12 [ffff8881376fff50] entry_SYSCALL_64_after_hwframe at ffffffff9740008c

     RIP: 00007f126f6956ed  RSP: 00007ffc2a320278  RFLAGS: 00000217
     RAX: ffffffffffffffda  RBX: 0000000020000044  RCX: 00007f126f6956ed
     RDX: 0000000000000004  RSI: 00000000004007b8  RDI: 0000000000000003
     RBP: 00007ffc2a3202a0   R8: 0000000000400720   R9: 0000000000400720
     R10: 0000000000400720  R11: 0000000000000217  R12: 00000000004004b0
     R13: 00007ffc2a320380  R14: 0000000000000000  R15: 0000000000000000
     ORIG_RAX: 0000000000000001  CS: 0033  SS: 002b

3) tcpdump captures show that DSS is exchanged even when MP_CAPABLE handshake didn't complete.

 $ tcpdump -tnnr bad.pcap
 IP 127.0.0.1.20000 > 127.0.0.1.20000: Flags [S], seq 3208913911, win 65483, options [mss 65495,sackOK,TS val 3291706876 ecr 3291694721,nop,wscale 7,mptcp capable v1], length 0
 IP 127.0.0.1.20000 > 127.0.0.1.20000: Flags [S.], seq 3208913911, ack 3208913912, win 65483, options [mss 65495,sackOK,TS val 3291706876 ecr 3291706876,nop,wscale 7,mptcp capable v1], length 0
 IP 127.0.0.1.20000 > 127.0.0.1.20000: Flags [.], ack 1, win 512, options [nop,nop,TS val 3291706876 ecr 3291706876], length 0
 IP 127.0.0.1.20000 > 127.0.0.1.20000: Flags [F.], seq 1, ack 1, win 512, options [nop,nop,TS val 3291707876 ecr 3291706876,mptcp dss fin seq 0 subseq 0 len 1,nop,nop], length 0
 IP 127.0.0.1.20000 > 127.0.0.1.20000: Flags [.], ack 2, win 512, options [nop,nop,TS val 3291707876 ecr 3291707876], length 0

force a fallback to TCP in these cases, and adjust the main socket state to avoid hanging in mptcp_sendmsg().

Reported-by Christoph Paasch <cpaasch(a)apple.com>
Suggested-by Paolo Abeni <pabeni(a)redhat.com>
Signed-off-by Davide Caratti <dcaratti(a)redhat.com>
---
 net/mptcp/protocol.h | 10 ++++++++++
 net/mptcp/subflow.c  | 10 ++++++++++
 2 files changed, 20 insertions(+)
net/mptcp/protocol.h +10 -0
index a709df659ae0..1d05d9841b5c 100644
@@ -490,4 +490,14 @@ static inline void mptcp_do_fallback(struct sock *sk)
 
 #define pr_fallback(a) pr_debug("%s:fallback to TCP (msk=%p)", __func__, a)
 
+static inline bool subflow_simultaneous_connect(struct sock *sk)
+{
+	struct mptcp_subflow_context *subflow = mptcp_subflow_ctx(sk);
+	struct sock *parent = subflow->conn;
+
+	return sk->sk_state == TCP_ESTABLISHED &&
+	       !mptcp_sk(parent)->pm.server_side &&
+	       !subflow->conn_finished;
+}
+
 #endif /* __MPTCP_PROTOCOL_H */
net/mptcp/subflow.c +10 -0
index cb8a42ff4646..548f9e347ff5 100644
@@ -1128,6 +1128,16 @@ static void subflow_state_change(struct sock *sk)
 
 	__subflow_state_change(sk);
 
+	if (subflow_simultaneous_connect(sk)) {
+		mptcp_do_fallback(sk);
+		pr_fallback(mptcp_sk(parent));
+		subflow->conn_finished = 1;
+		if (inet_sk_state_load(parent) == TCP_SYN_SENT) {
+			inet_sk_state_store(parent, TCP_ESTABLISHED);
+			parent->sk_state_change(parent);
+		}
+	}
+
 	/* as recvmsg() does not acquire the subflow socket for ssk selection
 	 * a fin packet carrying a DSS can be unnoticed if we don't trigger
 	 * the data available machinery here.
Signature · 1 line
2.26.2
Reviewed-by on v1 2/6
#4 lore raw

Re: [PATCH net-next 2/6] mptcp: fallback in case of simultaneous connect

Quoted 62 lines · Davide Caratti

when a MPTCP client tries to connect to itself, tcp_finish_connect() is never reached. Because of this, depending on the socket current state, multiple faulty behaviours can be observed:

1) a WARN_ON() in subflow_data_ready() is hit

WARNING: CPU: 2 PID: 882 at net/mptcp/subflow.c:911 subflow_data_ready+0x18b/0x230
[...]
CPU: 2 PID: 882 Comm: gh35 Not tainted 5.7.0+ #187
[...]
RIP: 0010:subflow_data_ready+0x18b/0x230
[...]
Call Trace:
 tcp_data_queue+0xd2f/0x4250
 tcp_rcv_state_process+0xb1c/0x49d3
 tcp_v4_do_rcv+0x2bc/0x790
 __release_sock+0x153/0x2d0
 release_sock+0x4f/0x170
 mptcp_shutdown+0x167/0x4e0
 __sys_shutdown+0xe6/0x180
 __x64_sys_shutdown+0x50/0x70
 do_syscall_64+0x9a/0x370
 entry_SYSCALL_64_after_hwframe+0x44/0xa9

2) client is stuck forever in mptcp_sendmsg() because the socket is not TCP_ESTABLISHED

crash> bt 4847
PID: 4847 TASK: ffff88814b2fb100 CPU: 1 COMMAND: "gh35"
#0 [ffff8881376ff680] __schedule at ffffffff97248da4
#1 [ffff8881376ff778] schedule at ffffffff9724a34f
#2 [ffff8881376ff7a0] schedule_timeout at ffffffff97252ba0 #3 [ffff8881376ff8a8] wait_woken at ffffffff958ab4ba
#4 [ffff8881376ff940] sk_stream_wait_connect at ffffffff96c2d859 #5 [ffff8881376ffa28] mptcp_sendmsg at ffffffff97207fca
#6 [ffff8881376ffbc0] sock_sendmsg at ffffffff96be1b5b
#7 [ffff8881376ffbe8] sock_write_iter at ffffffff96be1daa
#8 [ffff8881376ffce8] new_sync_write at ffffffff95e5cb52
#9 [ffff8881376ffe50] vfs_write at ffffffff95e6547f
#10 [ffff8881376ffe90] ksys_write at ffffffff95e65d26
#11 [ffff8881376fff28] do_syscall_64 at ffffffff956088ba
#12 [ffff8881376fff50] entry_SYSCALL_64_after_hwframe at ffffffff9740008c

    RIP: 00007f126f6956ed  RSP: 00007ffc2a320278  RFLAGS: 00000217
    RAX: ffffffffffffffda  RBX: 0000000020000044  RCX: 00007f126f6956ed
    RDX: 0000000000000004  RSI: 00000000004007b8  RDI: 0000000000000003
    RBP: 00007ffc2a3202a0   R8: 0000000000400720   R9: 0000000000400720
    R10: 0000000000400720  R11: 0000000000000217  R12: 00000000004004b0
    R13: 00007ffc2a320380  R14: 0000000000000000  R15: 0000000000000000
    ORIG_RAX: 0000000000000001  CS: 0033  SS: 002b

3) tcpdump captures show that DSS is exchanged even when MP_CAPABLE handshake didn't complete.

$ tcpdump -tnnr bad.pcap
IP 127.0.0.1.20000 > 127.0.0.1.20000: Flags [S], seq 3208913911, win 65483, options [mss 65495,sackOK,TS val 3291706876 ecr 3291694721,nop,wscale 7,mptcp capable v1], length 0
IP 127.0.0.1.20000 > 127.0.0.1.20000: Flags [S.], seq 3208913911, ack 3208913912, win 65483, options [mss 65495,sackOK,TS val 3291706876 ecr 3291706876,nop,wscale 7,mptcp capable v1], length 0
IP 127.0.0.1.20000 > 127.0.0.1.20000: Flags [.], ack 1, win 512, options [nop,nop,TS val 3291706876 ecr 3291706876], length 0
IP 127.0.0.1.20000 > 127.0.0.1.20000: Flags [F.], seq 1, ack 1, win 512, options [nop,nop,TS val 3291707876 ecr 3291706876,mptcp dss fin seq 0 subseq 0 len 1,nop,nop], length 0
IP 127.0.0.1.20000 > 127.0.0.1.20000: Flags [.], ack 2, win 512, options [nop,nop,TS val 3291707876 ecr 3291707876], length 0

force a fallback to TCP in these cases, and adjust the main socket state to avoid hanging in mptcp_sendmsg().

Quoted 8 lines
Reported-by Christoph Paasch <cpaasch(a)apple.com>
Suggested-by Paolo Abeni <pabeni(a)redhat.com>
Signed-off-by Davide Caratti <dcaratti(a)redhat.com>

---
net/mptcp/protocol.h | 10 ++++++++++
net/mptcp/subflow.c | 10 ++++++++++

2 files changed, 20 insertions(+)
Reviewed-by Mat Martineau <mathew.j.martineau(a)linux.intel.com>

--
Mat Martineau
Intel

#5 lore raw

PATCH 3/6 mptcp: check for plain TCP sock at accept time

In reply to Davide Caratti

From: Paolo Abeni <pabeni(a)redhat.com>

This cleanup the code a bit and avoid corrupted states on weird syscall sequence (accept(), connect()).

Signed-off-by Paolo Abeni <pabeni(a)redhat.com>
Signed-off-by Davide Caratti <dcaratti(a)redhat.com>
---
 net/mptcp/protocol.c | 69 +++++---------------------------------------
 1 file changed, 7 insertions(+), 62 deletions(-)
net/mptcp/protocol.c +7 -62
index 84ae96be9837..dbeb6fe374f5 100644
@@ -52,13 +52,10 @@ static struct socket *__mptcp_nmpc_socket(const struct mptcp_sock *msk)
 	return msk->subflow;
 }
 
-static struct socket *mptcp_is_tcpsk(struct sock *sk)
+static bool mptcp_is_tcpsk(struct sock *sk)
 {
 	struct socket *sock = sk->sk_socket;
 
-	if (sock->sk != sk)
-		return NULL;
-
 	if (unlikely(sk->sk_prot == &tcp_prot)) {
 		/* we are being invoked after mptcp_accept() has
 		 * accepted a non-mp-capable flow: sk is a tcp_sk,
@@ -68,27 +65,21 @@ static struct socket *mptcp_is_tcpsk(struct sock *sk)
 		 * bypass mptcp.
 		 */
 		sock->ops = &inet_stream_ops;
-		return sock;
+		return true;
 #if IS_ENABLED(CONFIG_MPTCP_IPV6)
 	} else if (unlikely(sk->sk_prot == &tcpv6_prot)) {
 		sock->ops = &inet6_stream_ops;
-		return sock;
+		return true;
 #endif
 	}
 
-	return NULL;
+	return false;
 }
 
 static struct socket *__mptcp_tcp_fallback(struct mptcp_sock *msk)
 {
-	struct socket *sock;
-
 	sock_owned_by_me((const struct sock *)msk);
 
-	sock = mptcp_is_tcpsk((struct sock *)msk);
-	if (unlikely(sock))
-		return sock;
-
 	if (likely(!__mptcp_check_fallback(msk)))
 		return NULL;
 
@@ -1466,7 +1457,6 @@ static struct sock *mptcp_accept(struct sock *sk, int flags, int *err,
 		return NULL;
 
 	pr_debug("msk=%p, subflow is mptcp=%d", msk, sk_is_mptcp(newsk));
-
 	if (sk_is_mptcp(newsk)) {
 		struct mptcp_subflow_context *subflow;
 		struct sock *new_mptcp_sock;
@@ -1821,42 +1811,6 @@ static int mptcp_stream_connect(struct socket *sock, struct sockaddr *uaddr,
 	return err;
 }
 
-static int mptcp_v4_getname(struct socket *sock, struct sockaddr *uaddr,
-			    int peer)
-{
-	if (sock->sk->sk_prot == &tcp_prot) {
-		/* we are being invoked from __sys_accept4, after
-		 * mptcp_accept() has just accepted a non-mp-capable
-		 * flow: sk is a tcp_sk, not an mptcp one.
-		 *
-		 * Hand the socket over to tcp so all further socket ops
-		 * bypass mptcp.
-		 */
-		sock->ops = &inet_stream_ops;
-	}
-
-	return inet_getname(sock, uaddr, peer);
-}
-
-#if IS_ENABLED(CONFIG_MPTCP_IPV6)
-static int mptcp_v6_getname(struct socket *sock, struct sockaddr *uaddr,
-			    int peer)
-{
-	if (sock->sk->sk_prot == &tcpv6_prot) {
-		/* we are being invoked from __sys_accept4 after
-		 * mptcp_accept() has accepted a non-mp-capable
-		 * subflow: sk is a tcp_sk, not mptcp.
-		 *
-		 * Hand the socket over to tcp so all further
-		 * socket ops bypass mptcp.
-		 */
-		sock->ops = &inet6_stream_ops;
-	}
-
-	return inet6_getname(sock, uaddr, peer);
-}
-#endif
-
 static int mptcp_listen(struct socket *sock, int backlog)
 {
 	struct mptcp_sock *msk = mptcp_sk(sock->sk);
@@ -1885,15 +1839,6 @@ static int mptcp_listen(struct socket *sock, int backlog)
 	return err;
 }
 
-static bool is_tcp_proto(const struct proto *p)
-{
-#if IS_ENABLED(CONFIG_MPTCP_IPV6)
-	return p == &tcp_prot || p == &tcpv6_prot;
-#else
-	return p == &tcp_prot;
-#endif
-}
-
 static int mptcp_stream_accept(struct socket *sock, struct socket *newsock,
 			       int flags, bool kern)
 {
@@ -1915,7 +1860,7 @@ static int mptcp_stream_accept(struct socket *sock, struct socket *newsock,
 	release_sock(sock->sk);
 
 	err = ssock->ops->accept(sock, newsock, flags, kern);
-	if (err == 0 && !is_tcp_proto(newsock->sk->sk_prot)) {
+	if (err == 0 && !mptcp_is_tcpsk(newsock->sk)) {
 		struct mptcp_sock *msk = mptcp_sk(newsock->sk);
 		struct mptcp_subflow_context *subflow;
 
@@ -2011,7 +1956,7 @@ static const struct proto_ops mptcp_stream_ops = {
 	.connect	   = mptcp_stream_connect,
 	.socketpair	   = sock_no_socketpair,
 	.accept		   = mptcp_stream_accept,
-	.getname	   = mptcp_v4_getname,
+	.getname	   = inet_getname,
 	.poll		   = mptcp_poll,
 	.ioctl		   = inet_ioctl,
 	.gettstamp	   = sock_gettstamp,
@@ -2065,7 +2010,7 @@ static const struct proto_ops mptcp_v6_stream_ops = {
 	.connect	   = mptcp_stream_connect,
 	.socketpair	   = sock_no_socketpair,
 	.accept		   = mptcp_stream_accept,
-	.getname	   = mptcp_v6_getname,
+	.getname	   = inet6_getname,
 	.poll		   = mptcp_poll,
 	.ioctl		   = inet6_ioctl,
 	.gettstamp	   = sock_gettstamp,
Signature · 1 line
2.26.2
Reviewed-by on v1 3/6
#6 lore raw

Re: [PATCH net-next 3/6] mptcp: check for plain TCP sock at accept time

Quoted 11 lines · Davide Caratti

From: Paolo Abeni <pabeni(a)redhat.com>

This cleanup the code a bit and avoid corrupted states on weird syscall sequence (accept(), connect()).

Signed-off-by Paolo Abeni <pabeni(a)redhat.com>
Signed-off-by Davide Caratti <dcaratti(a)redhat.com>

---
net/mptcp/protocol.c | 69 +++++---------------------------------------

1 file changed, 7 insertions(+), 62 deletions(-)
Reviewed-by Mat Martineau <mathew.j.martineau(a)linux.intel.com>

--
Mat Martineau
Intel

#7 lore raw

PATCH 4/6 mptcp: create first subflow at msk creation time

In reply to Davide Caratti

From: Paolo Abeni <pabeni(a)redhat.com>

This cleans the code a bit and makes the behavior more consistent.

Signed-off-by Paolo Abeni <pabeni(a)redhat.com>
Signed-off-by Davide Caratti <dcaratti(a)redhat.com>
---
 net/mptcp/protocol.c | 53 +++++++++++++++++---------------------------
 1 file changed, 20 insertions(+), 33 deletions(-)
net/mptcp/protocol.c +20 -33
index dbeb6fe374f5..ad619bda71cc 100644
@@ -86,32 +86,16 @@ static struct socket *__mptcp_tcp_fallback(struct mptcp_sock *msk)
 	return msk->subflow;
 }
 
-static bool __mptcp_can_create_subflow(const struct mptcp_sock *msk)
-{
-	return !msk->first;
-}
-
-static struct socket *__mptcp_socket_create(struct mptcp_sock *msk, int state)
+static int __mptcp_socket_create(struct mptcp_sock *msk)
 {
 	struct mptcp_subflow_context *subflow;
 	struct sock *sk = (struct sock *)msk;
 	struct socket *ssock;
 	int err;
 
-	ssock = __mptcp_tcp_fallback(msk);
-	if (unlikely(ssock))
-		return ssock;
-
-	ssock = __mptcp_nmpc_socket(msk);
-	if (ssock)
-		goto set_state;
-
-	if (!__mptcp_can_create_subflow(msk))
-		return ERR_PTR(-EINVAL);
-
 	err = mptcp_subflow_create_socket(sk, &ssock);
 	if (err)
-		return ERR_PTR(err);
+		return err;
 
 	msk->first = ssock->sk;
 	msk->subflow = ssock;
@@ -124,10 +108,7 @@ static struct socket *__mptcp_socket_create(struct mptcp_sock *msk, int state)
 	 */
 	RCU_INIT_POINTER(msk->first->sk_wq, &sk->sk_socket->wq);
 
-set_state:
-	if (state != MPTCP_SAME_STATE)
-		inet_sk_state_store(sk, state);
-	return ssock;
+	return 0;
 }
 
 static void __mptcp_move_skb(struct mptcp_sock *msk, struct sock *ssk,
@@ -1255,6 +1236,10 @@ static int mptcp_init_sock(struct sock *sk)
 	if (ret)
 		return ret;
 
+	ret = __mptcp_socket_create(mptcp_sk(sk));
+	if (ret)
+		return ret;
+
 	sk_sockets_allocated_inc(sk);
 	sk->sk_sndbuf = sock_net(sk)->ipv4.sysctl_tcp_wmem[2];
 
@@ -1744,9 +1729,9 @@ static int mptcp_bind(struct socket *sock, struct sockaddr *uaddr, int addr_len)
 	int err;
 
 	lock_sock(sock->sk);
-	ssock = __mptcp_socket_create(msk, MPTCP_SAME_STATE);
-	if (IS_ERR(ssock)) {
-		err = PTR_ERR(ssock);
+	ssock = __mptcp_nmpc_socket(msk);
+	if (!ssock) {
+		err = -EINVAL;
 		goto unlock;
 	}
 
@@ -1776,13 +1761,14 @@ static int mptcp_stream_connect(struct socket *sock, struct sockaddr *uaddr,
 		goto do_connect;
 	}
 
-	mptcp_token_destroy(msk);
-	ssock = __mptcp_socket_create(msk, TCP_SYN_SENT);
-	if (IS_ERR(ssock)) {
-		err = PTR_ERR(ssock);
+	ssock = __mptcp_nmpc_socket(msk);
+	if (!ssock) {
+		err = -EINVAL;
 		goto unlock;
 	}
 
+	mptcp_token_destroy(msk);
+	inet_sk_state_store(sock->sk, TCP_SYN_SENT);
 	subflow = mptcp_subflow_ctx(ssock->sk);
 #ifdef CONFIG_TCP_MD5SIG
 	/* no MPTCP if MD5SIG is enabled on this socket or we may run out of
@@ -1820,13 +1806,14 @@ static int mptcp_listen(struct socket *sock, int backlog)
 	pr_debug("msk=%p", msk);
 
 	lock_sock(sock->sk);
-	mptcp_token_destroy(msk);
-	ssock = __mptcp_socket_create(msk, TCP_LISTEN);
-	if (IS_ERR(ssock)) {
-		err = PTR_ERR(ssock);
+	ssock = __mptcp_nmpc_socket(msk);
+	if (!ssock) {
+		err = -EINVAL;
 		goto unlock;
 	}
 
+	mptcp_token_destroy(msk);
+	inet_sk_state_store(sock->sk, TCP_LISTEN);
 	sock_set_flag(sock->sk, SOCK_RCU_FREE);
 
 	err = ssock->ops->listen(ssock, backlog);
Signature · 1 line
2.26.2
Reviewed-by on v1 4/6
#8 lore raw

Re: [PATCH net-next 4/6] mptcp: create first subflow at msk creation time

Quoted 10 lines · Davide Caratti

From: Paolo Abeni <pabeni(a)redhat.com>

This cleans the code a bit and makes the behavior more consistent.

Signed-off-by Paolo Abeni <pabeni(a)redhat.com>
Signed-off-by Davide Caratti <dcaratti(a)redhat.com>

---
net/mptcp/protocol.c | 53 +++++++++++++++++---------------------------

1 file changed, 20 insertions(+), 33 deletions(-)
Reviewed-by Mat Martineau <mathew.j.martineau(a)linux.intel.com>

--
Mat Martineau
Intel

#9 lore raw

PATCH 5/6 mptcp: __mptcp_tcp_fallback() returns a struct sock

In reply to Davide Caratti

From: Paolo Abeni <pabeni(a)redhat.com>

Currently __mptcp_tcp_fallback() always return NULL on incoming connections, because MPTCP does not create the additional socket for the first subflow.
Since the previous commit no __mptcp_tcp_fallback() caller needs a struct socket, so let __mptcp_tcp_fallback() return the first subflow sock and cope correctly even with incoming connections.

Signed-off-by Paolo Abeni <pabeni(a)redhat.com>
Signed-off-by Davide Caratti <dcaratti(a)redhat.com>
---
 net/mptcp/protocol.c | 22 ++++++++++------------
 1 file changed, 10 insertions(+), 12 deletions(-)
net/mptcp/protocol.c +10 -12
index ad619bda71cc..f2b2bd37e371 100644
@@ -76,14 +76,14 @@ static bool mptcp_is_tcpsk(struct sock *sk)
 	return false;
 }
 
-static struct socket *__mptcp_tcp_fallback(struct mptcp_sock *msk)
+static struct sock *__mptcp_tcp_fallback(struct mptcp_sock *msk)
 {
 	sock_owned_by_me((const struct sock *)msk);
 
 	if (likely(!__mptcp_check_fallback(msk)))
 		return NULL;
 
-	return msk->subflow;
+	return msk->first;
 }
 
 static int __mptcp_socket_create(struct mptcp_sock *msk)
@@ -1498,7 +1498,7 @@ static int mptcp_setsockopt(struct sock *sk, int level, int optname,
 			    char __user *optval, unsigned int optlen)
 {
 	struct mptcp_sock *msk = mptcp_sk(sk);
-	struct socket *ssock;
+	struct sock *ssk;
 
 	pr_debug("msk=%p", msk);
 
@@ -1509,11 +1509,10 @@ static int mptcp_setsockopt(struct sock *sk, int level, int optname,
 	 * to the one remaining subflow.
 	 */
 	lock_sock(sk);
-	ssock = __mptcp_tcp_fallback(msk);
+	ssk = __mptcp_tcp_fallback(msk);
 	release_sock(sk);
-	if (ssock)
-		return tcp_setsockopt(ssock->sk, level, optname, optval,
-				      optlen);
+	if (ssk)
+		return tcp_setsockopt(ssk, level, optname, optval, optlen);
 
 	return -EOPNOTSUPP;
 }
@@ -1522,7 +1521,7 @@ static int mptcp_getsockopt(struct sock *sk, int level, int optname,
 			    char __user *optval, int __user *option)
 {
 	struct mptcp_sock *msk = mptcp_sk(sk);
-	struct socket *ssock;
+	struct sock *ssk;
 
 	pr_debug("msk=%p", msk);
 
@@ -1533,11 +1532,10 @@ static int mptcp_getsockopt(struct sock *sk, int level, int optname,
 	 * to the one remaining subflow.
 	 */
 	lock_sock(sk);
-	ssock = __mptcp_tcp_fallback(msk);
+	ssk = __mptcp_tcp_fallback(msk);
 	release_sock(sk);
-	if (ssock)
-		return tcp_getsockopt(ssock->sk, level, optname, optval,
-				      option);
+	if (ssk)
+		return tcp_getsockopt(ssk, level, optname, optval, option);
 
 	return -EOPNOTSUPP;
 }
Signature · 1 line
2.26.2
Reviewed-by on v1 5/6
#10 lore raw

Re: [PATCH net-next 5/6] mptcp: __mptcp_tcp_fallback() returns a struct sock

Quoted 10 lines · Davide Caratti

From: Paolo Abeni <pabeni(a)redhat.com>

Currently __mptcp_tcp_fallback() always return NULL on incoming connections, because MPTCP does not create the additional socket for the first subflow.
Since the previous commit no __mptcp_tcp_fallback() caller needs a struct socket, so let __mptcp_tcp_fallback() return the first subflow sock and cope correctly even with incoming connections.

Quoted 6 lines
Signed-off-by Paolo Abeni <pabeni(a)redhat.com>
Signed-off-by Davide Caratti <dcaratti(a)redhat.com>

---
net/mptcp/protocol.c | 22 ++++++++++------------

1 file changed, 10 insertions(+), 12 deletions(-)
Reviewed-by Mat Martineau <mathew.j.martineau(a)linux.intel.com>

--
Mat Martineau
Intel

#11 lore raw

PATCH 6/6 mptcp: close poll() races

In reply to Davide Caratti

From: Paolo Abeni <pabeni(a)redhat.com>

mptcp_poll always return POLLOUT for unblocking connect(), ensure that the socket is a suitable state.
The MPTCP_DATA_READY bit is never cleared on accept:
ensure we don't leave mptcp_accept() with an empty accept queue and such bit set.

Signed-off-by Paolo Abeni <pabeni(a)redhat.com>
Signed-off-by Davide Caratti <dcaratti(a)redhat.com>
---
 net/mptcp/protocol.c | 25 ++++++++++++++++++++-----
 1 file changed, 20 insertions(+), 5 deletions(-)
net/mptcp/protocol.c +20 -5
index f2b2bd37e371..28ec26d97f96 100644
@@ -1841,6 +1841,7 @@ static int mptcp_stream_accept(struct socket *sock, struct socket *newsock,
 	if (!ssock)
 		goto unlock_fail;
 
+	clear_bit(MPTCP_DATA_READY, &msk->flags);
 	sock_hold(ssock->sk);
 	release_sock(sock->sk);
 
@@ -1861,6 +1862,8 @@ static int mptcp_stream_accept(struct socket *sock, struct socket *newsock,
 		}
 	}
 
+	if (inet_csk_listen_poll(ssock->sk))
+		set_bit(MPTCP_DATA_READY, &msk->flags);
 	sock_put(ssock->sk);
 	return err;
 
@@ -1869,21 +1872,33 @@ static int mptcp_stream_accept(struct socket *sock, struct socket *newsock,
 	return -EINVAL;
 }
 
+static __poll_t mptcp_check_readable(struct mptcp_sock *msk)
+{
+	return test_bit(MPTCP_DATA_READY, &msk->flags) ? EPOLLIN | EPOLLRDNORM :
+	       0;
+}
+
 static __poll_t mptcp_poll(struct file *file, struct socket *sock,
 			   struct poll_table_struct *wait)
 {
 	struct sock *sk = sock->sk;
 	struct mptcp_sock *msk;
 	__poll_t mask = 0;
+	int state;
 
 	msk = mptcp_sk(sk);
 	sock_poll_wait(file, sock, wait);
 
-	if (test_bit(MPTCP_DATA_READY, &msk->flags))
-		mask = EPOLLIN | EPOLLRDNORM;
-	if (sk_stream_is_writeable(sk) &&
-	    test_bit(MPTCP_SEND_SPACE, &msk->flags))
-		mask |= EPOLLOUT | EPOLLWRNORM;
+	state = inet_sk_state_load(sk);
+	if (state == TCP_LISTEN)
+		return mptcp_check_readable(msk);
+
+	if (state != TCP_SYN_SENT && state != TCP_SYN_RECV) {
+		mask |= mptcp_check_readable(msk);
+		if (sk_stream_is_writeable(sk) &&
+		    test_bit(MPTCP_SEND_SPACE, &msk->flags))
+			mask |= EPOLLOUT | EPOLLWRNORM;
+	}
 	if (sk->sk_shutdown & RCV_SHUTDOWN)
 		mask |= EPOLLIN | EPOLLRDNORM | EPOLLRDHUP;
 
Signature · 1 line
2.26.2
Reviewed-by on v1 6/6
#12 lore raw

Re: [PATCH net-next 6/6] mptcp: close poll() races

Quoted 9 lines · Davide Caratti

From: Paolo Abeni <pabeni(a)redhat.com>

mptcp_poll always return POLLOUT for unblocking connect(), ensure that the socket is a suitable state.
The MPTCP_DATA_READY bit is never cleared on accept:
ensure we don't leave mptcp_accept() with an empty accept queue and such bit set.

Quoted 6 lines
Signed-off-by Paolo Abeni <pabeni(a)redhat.com>
Signed-off-by Davide Caratti <dcaratti(a)redhat.com>

---
net/mptcp/protocol.c | 25 ++++++++++++++++++++-----

1 file changed, 20 insertions(+), 5 deletions(-)
Reviewed-by Mat Martineau <mathew.j.martineau(a)linux.intel.com>

--
Mat Martineau
Intel

maintainer
#13 lore raw
In reply to Davide Caratti

From: Davide Caratti <dcaratti(a)redhat.com>
Date: Mon, 29 Jun 2020 22:26:19 +0200

Quoted 15 lines

there are situations where MPTCP sockets should fall-back to regular TCP:
this series reworks the fallback code to pursue the following goals:

1) cleanup the non fallback code, removing most of 'if (<fallback>)' in the data path
2) improve performance for non-fallback sockets, avoiding locks in poll()

further work will also leverage on this changes to achieve:

a) more consistent behavior of gestockopt()/setsockopt() on passive sockets after fallback
b) support for "infinite maps" as per RFC8684, section 3.7

the series is made of the following items:

Quoted 10 lines

- patch 1 lets sendmsg() / recvmsg() / poll() use the main socket also after fallback
- patch 2 fixes 'simultaneous connect' scenario after fallback. The problem was present also before the rework, but the fix is much easier to implement after patch 1
- patch 3, 4, 5 are clean-ups for code that is no more needed after the fallback rework
- patch 6 fixes a race condition between close() and poll(). The problem was theoretically present before the rework, but it became almost systematic after patch 1

Series applied to net-next, thank you.