Skip to contentSkip to conversation

improve subflow join handling

PATCH In review Florian Westphal 3 participants 20 messages Nov 25, 2019 – Nov 26, 2019

Subsystems (3)
lore
Share & tools
lore

Florian Westphal

author
#0 lore raw

cover improve subflow join handling

This set aims to improve handling of multiple subflows.
There are multiple remaining problems, see below, but I wanted to send what I have now rather than keep this back.

The first 3 patches are fixes to handle refcount leaks found while working on this, I've added the commit name of where they could be squashed to.

The remaining part of this work is to make the msk->conn_list contain tcp_sock structs without an outer 'struct socket'.

The problem is that for incoming join requests (which are not subject to accept/request socket queue), there is no 'struct socket', i.e. sk->sk_socket is NULL.

As tcp stack will assume that such tcp sk is already detached from userspace, SOCK_NOSPACE wakeups do not work for them.

The solution here (suggested by Paolo) is to have sk->sk_socket refer to the mptcp parent socket structure. This assignment is done by a private function similar to sock_graft().

For incoming connects, tcp-level connectivity works, i.e. mptcp-next can accept a few subflows from the peer and on exit of the test program all resources are cleaned up correctly.

MPTCP-Level connectivity will probably not work, as nothing sets up ssn_offset for joins (both incoming or outgoing).

mptcp.org kernel will not consider the joins as established, as the "4th Ack" is missing.
A simple "tcp_send_ack" is enough, but there are two problems with this:
1. It can be lost
2. Given we are not expecting to receive any data on such flows this doesn't seem to be a problem.

During testing I saw that sometimes mptcp.org kernel will emit further join requests to mptcp-next but those trigger no response (and those packets do not even show up in perfs network drop monitor, sigh).

For outgoing connects, there is one remaining memory leak that I have not located so far.

author
#1 lore raw

PATCH 1/7 subflow: fix join error handling

squashto: mptcp: Add key generation and token tree

Invalid join token gives this splat:

WARNING: CPU: 3 PID: 0 at net/ipv4/inet_connection_sock.c:833 inet_csk_destroy_sock+0x367/0x460
CPU: 3 PID: 0 Comm: swapper/3 Not tainted 5.4.0-rc7+ #1
RIP: 0010:inet_csk_destroy_sock+0x367/0x460
Code: 20 02 00 00 00 00 00 00 e9 dc fe ff ff e8 41 21 e7 fe 0f 0b e9 06 fd ff ff e8 35 21 e7 fe 0f 0b e9 45 fd ff ff e8 29 21 e7 fe <0f> 0b e9 6a fd ff ff e8 1d 21 e7 fe be 03 00 00 00 4c 89 f7 e8 f0
RSP: 0018:ffff88811b5893a0 EFLAGS: 00010206
RAX: ffff88811abddd40 RBX: 0000000000000001 RCX: ffffffff8250ca2f
RDX: 0000000000000100 RSI: ffffffff8250ccf7 RDI: ffff88810c2ccd30
RBP: ffff88811b5893c8 R08: ffff88811abddd40 R09: ffffed10218599a5
R10: ffffed10218599a4 R11: ffff88810c2ccd27 R12: ffff88810c2cccc0
R13: ffff88810c2ccd20 R14: ffff888114bf6840 R15: ffffffff8328f4a0
FS:  0000000000000000(0000) GS:ffff88811b580000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000731e24 CR3: 000000010aa04003 CR4: 0000000000160ea0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 subflow_syn_recv_sock+0x593/0x6c0
 tcp_check_req+0x6ae/0x19e0

The warn is caused by:

        /* It cannot be in hash table! */
        WARN_ON(!sk_unhashed(sk));

so handle this via tcp_done instead, it will call ->sk_unhash() before calling inet_csk_destroy_sock() internally.

While at it, try to notify the peer immediately via reset.

Signed-off-by Florian Westphal <fw(a)strlen.de>
---
 net/mptcp/subflow.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
net/mptcp/subflow.c +3 -3
index 191d7bb0f0a9..ffbe0a4a04be 100644
@@ -361,9 +361,9 @@ static struct sock *subflow_syn_recv_sock(const struct sock *sk,
 
 close_child:
 	pr_debug("closing child socket");
-	inet_sk_set_state(child, TCP_CLOSE);
-	sock_set_flag(child, SOCK_DEAD);
-	inet_csk_destroy_sock(child);
+	tcp_send_active_reset(child, GFP_ATOMIC);
+	inet_csk_prepare_forced_close(child);
+	tcp_done(child);
 	return NULL;
 }
 
Signature · 1 line
2.23.0
author
#2 lore raw

PATCH 2/7 mptcp: must drop reqsk as well

In reply to Florian Westphal

squashto: mptcp: Add handling of incoming MP_JOIN requests

For incoming join requests we need to also drop the request socket, it holds a reference on the listener sk.

Without this, each sucessful join causes a refcount leak on mptcp->subflow listener socket.

---
 net/ipv4/tcp_minisocks.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/ipv4/tcp_minisocks.c +1 -1
index c2d9a47da9c2..65c41894f3dd 100644
@@ -773,8 +773,8 @@ struct sock *tcp_check_req(struct sock *sk, struct sk_buff *skb,
 		goto listen_overflow;
 
 	if (own_req && sk_is_mptcp(child) && mptcp_sk_is_subflow(child)) {
-		inet_csk_reqsk_queue_drop(sk, req);
 		reqsk_queue_removed(&inet_csk(sk)->icsk_accept_queue, req);
+		inet_csk_reqsk_queue_drop_and_put(sk, req);
 		return child;
 	}
 
Signature · 1 line
2.23.0
author
#3 lore raw

PATCH 3/7 remove second conditional

In reply to Florian Westphal

squashto: mptcp: Handle MP_CAPABLE options for outgoing connections

... there is no reason for this second conditional, one is enough.

Signed-off-by Florian Westphal <fw(a)strlen.de>
---
 net/mptcp/protocol.c | 8 +-------
 1 file changed, 1 insertion(+), 7 deletions(-)
net/mptcp/protocol.c +1 -7
index f65c48a0069a..72e6950fbbc2 100644
@@ -901,7 +901,6 @@ static void mptcp_close(struct sock *sk, long timeout)
 {
 	struct mptcp_subflow_context *subflow, *tmp;
 	struct mptcp_sock *msk = mptcp_sk(sk);
-	struct socket *ssk = NULL;
 
 	mptcp_token_destroy(msk->token);
 	inet_sk_state_store(sk, TCP_CLOSE);
@@ -909,15 +908,10 @@ static void mptcp_close(struct sock *sk, long timeout)
 	lock_sock(sk);
 
 	if (msk->subflow) {
-		ssk = msk->subflow;
+		sock_release(msk->subflow);
 		msk->subflow = NULL;
 	}
 
-	if (ssk) {
-		pr_debug("subflow=%p", ssk->sk);
-		sock_release(ssk);
-	}
-
 	list_for_each_entry_safe(subflow, tmp, &msk->conn_list, node) {
 		pr_debug("conn_list->subflow=%p", subflow);
 		sock_release(mptcp_subflow_tcp_socket(subflow));
Signature · 1 line
2.23.0
author
#4 lore raw

RFC 4/7 copy connection id from first subflow to mptcp socket

In reply to Florian Westphal

This allows to call inet(6)_getname on the parent socket and it will return the right result.

At this time it only works because the first incoming subflow has a 'struct socket' allocated.

This will change in followup patch, passive subflows will not have a socket struct anymore.

Signed-off-by Florian Westphal <fw(a)strlen.de>
---
 net/mptcp/pm.c       | 18 ++++----
 net/mptcp/protocol.c | 98 ++++++++++++++++++++++++--------------------
 net/mptcp/protocol.h |  1 -
 3 files changed, 63 insertions(+), 54 deletions(-)
net/mptcp/pm.c +10 -8
index 7e493d81afef..e474bd9dc05d 100644
@@ -81,6 +81,7 @@ int mptcp_pm_create_subflow(u32 token, u8 remote_id, struct in_addr *addr)
 	struct mptcp_sock *msk = mptcp_token_get_sock(token);
 	struct sockaddr_in remote;
 	struct sockaddr_in local;
+	struct sock *sk;
 	int err;
 
 	if (!msk)
@@ -88,6 +89,7 @@ int mptcp_pm_create_subflow(u32 token, u8 remote_id, struct in_addr *addr)
 
 	pr_debug("msk=%p", msk);
 
+	sk = (struct sock *)msk;
 	if (!msk->pm.remote_valid || remote_id != msk->pm.remote_id) {
 		err = -EBADR;
 		goto create_put;
@@ -101,15 +103,14 @@ int mptcp_pm_create_subflow(u32 token, u8 remote_id, struct in_addr *addr)
 		local.sin_addr.s_addr = htonl(INADDR_ANY);
 
 	remote.sin_family = msk->pm.remote_family;
-	remote.sin_port = htons(msk->dport);
+	remote.sin_port = inet_sk(sk)->inet_dport;
 	remote.sin_addr = msk->pm.remote_addr;
 
-	err = mptcp_subflow_connect((struct sock *)msk,
-				    (struct sockaddr *)&local,
+	err = mptcp_subflow_connect(sk, (struct sockaddr *)&local,
 				    (struct sockaddr *)&remote, remote_id);
 
 create_put:
-	sock_put((struct sock *)msk);
+	sock_put(sk);
 	return err;
 }
 
@@ -119,12 +120,14 @@ int mptcp_pm_create_subflow6(u32 token, u8 remote_id, struct in6_addr *addr)
 	struct mptcp_sock *msk = mptcp_token_get_sock(token);
 	struct sockaddr_in6 remote;
 	struct sockaddr_in6 local;
+	struct sock *sk;
 	int err;
 
 	if (!msk)
 		return -EINVAL;
 
 	pr_debug("msk=%p", msk);
+	sk = (struct sock *)msk;
 
 	if (!msk->pm.remote_valid || remote_id != msk->pm.remote_id) {
 		err = -EBADR;
@@ -139,15 +142,14 @@ int mptcp_pm_create_subflow6(u32 token, u8 remote_id, struct in6_addr *addr)
 		local.sin6_addr = in6addr_any;
 
 	remote.sin6_family = msk->pm.remote_family;
-	remote.sin6_port = htons(msk->dport);
+	remote.sin6_port = inet_sk(sk)->inet_dport;
 	remote.sin6_addr = msk->pm.remote_addr6;
 
-	err = mptcp_subflow_connect((struct sock *)msk,
-				    (struct sockaddr *)&local,
+	err = mptcp_subflow_connect(sk, (struct sockaddr *)&local,
 				    (struct sockaddr *)&remote, remote_id);
 
 create_put:
-	sock_put((struct sock *)msk);
+	sock_put(sk);
 	return err;
 }
 #endif
net/mptcp/protocol.c +53 -45

186 lines not on this page · raw message

net/mptcp/protocol.h +0 -1
index acad61c70de9..237c5efd0ba7 100644
@@ -134,7 +134,6 @@ struct mptcp_sock {
 	unsigned long	timer_ival;
 	u32		token;
 	unsigned long	flags;
-	u16		dport;
 	struct work_struct rtx_work;
 	struct list_head conn_list;
 	struct list_head rtx_queue;
Signature · 1 line
2.23.0
author
#5 lore raw

RFC 5/7 store tcp_sk, not socket

In reply to Florian Westphal

preparation patch to avoid outer 'struct socket'.

Signed-off-by Florian Westphal <fw(a)strlen.de>
---
 net/mptcp/protocol.c | 23 ++++++++++++-----------
 net/mptcp/protocol.h |  8 ++++----
 net/mptcp/subflow.c  |  8 +++-----
 3 files changed, 19 insertions(+), 20 deletions(-)
net/mptcp/protocol.c +12 -11
index 720c0ab46cfb..68ae6796359f 100644
@@ -101,7 +101,7 @@ static struct sock *mptcp_subflow_recv_lookup(const struct mptcp_sock *msk)
 
 	mptcp_for_each_subflow(msk, subflow) {
 		if (subflow->data_avail)
-			return mptcp_subflow_tcp_socket(subflow)->sk;
+			return mptcp_subflow_tcp_sock(subflow);
 
 		receivers += !subflow->rx_eof;
 	}
@@ -382,7 +382,7 @@ static struct sock *mptcp_subflow_get_send(struct mptcp_sock *msk)
 	sock_owned_by_me((const struct sock *)msk);
 
 	mptcp_for_each_subflow(msk, subflow) {
-		struct sock *ssk = mptcp_subflow_tcp_socket(subflow)->sk;
+		struct sock *ssk = mptcp_subflow_tcp_sock(subflow);
 
 		if (!sk_stream_memory_free(ssk)) {
 			struct socket *sock = ssk->sk_socket;
@@ -745,7 +745,7 @@ static struct sock *mptcp_subflow_get_retrans(const struct mptcp_sock *msk)
 	sock_owned_by_me((const struct sock *)msk);
 
 	mptcp_for_each_subflow(msk, subflow) {
-		struct sock *ssk = mptcp_subflow_tcp_socket(subflow)->sk;
+		struct sock *ssk = mptcp_subflow_tcp_sock(subflow);
 
 		/* still data outstanding at TCP level?  Don't retransmit. */
 		if (!tcp_write_queue_empty(ssk))
@@ -900,8 +900,11 @@ static void mptcp_close(struct sock *sk, long timeout)
 	}
 
 	list_for_each_entry_safe(subflow, tmp, &msk->conn_list, node) {
+		struct sock *ssk = mptcp_subflow_tcp_sock(subflow);
+
 		pr_debug("conn_list->subflow=%p", subflow);
-		sock_release(mptcp_subflow_tcp_socket(subflow));
+		list_del(&subflow->node);
+		sock_release(ssk->sk_socket);
 	}
 
 	__mptcp_clear_xmit(sk);
@@ -1015,10 +1018,10 @@ static struct sock *mptcp_accept(struct sock *sk, int flags, int *err,
 		subflow->map_seq = ack_seq;
 		subflow->map_subflow_seq = 1;
 		subflow->rel_write_seq = 1;
-		subflow->tcp_sock = new_sock;
+		subflow->tcp_sock = ssk;
 		subflow->conn = new_mptcp_sock;
-		if (unlikely(!skb_queue_empty(&new_sock->sk->sk_receive_queue)))
-			mptcp_subflow_data_available(new_sock->sk);
+		if (unlikely(!skb_queue_empty(&ssk->sk_receive_queue)))
+			mptcp_subflow_data_available(ssk);
 		release_sock(ssk);
 	} else {
 		newsk = new_sock->sk;
@@ -1475,11 +1478,9 @@ static int mptcp_shutdown(struct socket *sock, int how)
 	}
 
 	mptcp_for_each_subflow(msk, subflow) {
-		struct socket *tcp_socket;
+		struct sock *tcp_sk = mptcp_subflow_tcp_sock(subflow);
 
-		tcp_socket = mptcp_subflow_tcp_socket(subflow);
-		pr_debug("conn_list->subflow=%p", subflow);
-		ret = kernel_sock_shutdown(tcp_socket, how);
+		ret = kernel_sock_shutdown(tcp_sk->sk_socket, how);
 	}
 	release_sock(sock->sk);
 
net/mptcp/protocol.h +4 -4
index 237c5efd0ba7..015102402432 100644
@@ -226,7 +226,7 @@ struct mptcp_subflow_context {
 	u8	local_id;
 	u8	remote_id;
 
-	struct	socket *tcp_sock;   /* underlying tcp_sock */
+	struct	sock *tcp_sock;	    /* tcp sk backpointer */
 	struct	sock *conn;	    /* parent mptcp_sock */
 	const	struct inet_connection_sock_af_ops *icsk_af_ops;
 	void	(*tcp_sk_data_ready)(struct sock *sk);
@@ -242,8 +242,8 @@ mptcp_subflow_ctx(const struct sock *sk)
 	return (__force struct mptcp_subflow_context *)icsk->icsk_ulp_data;
 }
 
-static inline struct socket *
-mptcp_subflow_tcp_socket(const struct mptcp_subflow_context *subflow)
+static inline struct sock *
+mptcp_subflow_tcp_sock(const struct mptcp_subflow_context *subflow)
 {
 	return subflow->tcp_sock;
 }
@@ -251,7 +251,7 @@ mptcp_subflow_tcp_socket(const struct mptcp_subflow_context *subflow)
 static inline u64
 mptcp_subflow_get_map_offset(const struct mptcp_subflow_context *subflow)
 {
-	return tcp_sk(mptcp_subflow_tcp_socket(subflow)->sk)->copied_seq -
+	return tcp_sk(mptcp_subflow_tcp_sock(subflow))->copied_seq -
 		      subflow->ssn_offset -
 		      subflow->map_subflow_seq;
 }
net/mptcp/subflow.c +3 -5
index ffbe0a4a04be..e6a512bf6230 100644
@@ -769,7 +769,6 @@ int mptcp_subflow_create_socket(struct sock *sk, struct socket **new_sock)
 }
 
 static struct mptcp_subflow_context *subflow_create_ctx(struct sock *sk,
-							struct socket *sock,
 							gfp_t priority)
 {
 	struct inet_connection_sock *icsk = inet_csk(sk);
@@ -782,8 +781,7 @@ static struct mptcp_subflow_context *subflow_create_ctx(struct sock *sk,
 
 	pr_debug("subflow=%p", ctx);
 
-	/* might be NULL */
-	ctx->tcp_sock = sock;
+	ctx->tcp_sock = sk;
 
 	return ctx;
 }
@@ -825,7 +823,7 @@ static int subflow_ulp_init(struct sock *sk)
 		goto out;
 	}
 
-	ctx = subflow_create_ctx(sk, sk->sk_socket, GFP_KERNEL);
+	ctx = subflow_create_ctx(sk, GFP_KERNEL);
 	if (!ctx) {
 		err = -ENOMEM;
 		goto out;
@@ -870,7 +868,7 @@ static void subflow_ulp_clone(const struct request_sock *req,
 	struct mptcp_subflow_context *new_ctx;
 
 	/* newsk->sk_socket is NULL at this point */
-	new_ctx = subflow_create_ctx(newsk, NULL, priority);
+	new_ctx = subflow_create_ctx(newsk, priority);
 	if (!new_ctx)
 		return;
 
Signature · 1 line
2.23.0
author
#6 lore raw

RFC 6/7 add mptcp_subflow_shutdown

In reply to Florian Westphal

soon tcp_sk->sk_socket may be pointing at the parent mptcp socket. Add a private shutdown function and use it instead.

Signed-off-by Florian Westphal <fw(a)strlen.de>
---
 net/mptcp/protocol.c | 42 +++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 41 insertions(+), 1 deletion(-)
net/mptcp/protocol.c +41 -1
index 68ae6796359f..924e63b354ef 100644
@@ -884,6 +884,29 @@ static void mptcp_cancel_work(struct sock *sk)
 		sock_put(sk);
 }
 
+static void mptcp_subflow_shutdown(struct sock *ssk, int how)
+{
+	lock_sock(ssk);
+
+	switch (ssk->sk_state) {
+	case TCP_LISTEN:
+		if (!(how & RCV_SHUTDOWN))
+			break;
+		/* fall through */
+	case TCP_SYN_SENT:
+		tcp_disconnect(ssk, O_NONBLOCK);
+		break;
+	default:
+		ssk->sk_shutdown |= how;
+		tcp_shutdown(ssk, how);
+		break;
+	}
+
+	/* Wake up anyone sleeping in poll. */
+	ssk->sk_state_change(ssk);
+	release_sock(ssk);
+}
+
 static void mptcp_close(struct sock *sk, long timeout)
 {
 	struct mptcp_subflow_context *subflow, *tmp;
@@ -1477,11 +1500,28 @@ static int mptcp_shutdown(struct socket *sock, int how)
 		return ret;
 	}
 
+	how++;
+
+	if ((how & ~SHUTDOWN_MASK) || !how) {
+		ret = -EINVAL;
+		goto out_unlock;
+	}
+
+	if (sock->state == SS_CONNECTING) {
+		if ((1 << sock->sk->sk_state) &
+		    (TCPF_SYN_SENT | TCPF_SYN_RECV | TCPF_CLOSE))
+			sock->state = SS_DISCONNECTING;
+		else
+			sock->state = SS_CONNECTED;
+	}
+
 	mptcp_for_each_subflow(msk, subflow) {
 		struct sock *tcp_sk = mptcp_subflow_tcp_sock(subflow);
 
-		ret = kernel_sock_shutdown(tcp_sk->sk_socket, how);
+		mptcp_subflow_shutdown(tcp_sk, how);
 	}
+
+out_unlock:
 	release_sock(sock->sk);
 
 	return ret;
Signature · 1 line
2.23.0
#7 lore raw

[MPTCP] Re: [RFC 6/7] add mptcp_subflow_shutdown

Quoted 12 lines · Florian Westphal
net/mptcp/protocol.c +12 -0
+static void mptcp_subflow_shutdown(struct sock *ssk, int how)
+{
+	lock_sock(ssk);
+
+	switch (ssk->sk_state) {
+	case TCP_LISTEN:
+		if (!(how & RCV_SHUTDOWN))
+			break;
+		/* fall through */
+	case TCP_SYN_SENT:
+		tcp_disconnect(ssk, O_NONBLOCK);
+		break;

net/mptcp/protocol.c static void mptcp_cancel_work(struct sock *sk) · line 908

Quoted 10 lines
net/mptcp/protocol.c +10 -0
+	default:
+		ssk->sk_shutdown |= how;
+		tcp_shutdown(ssk, how);
+		break;
+	}
+
+	/* Wake up anyone sleeping in poll. */
+	ssk->sk_state_change(ssk);
+	release_sock(ssk);
+}

I think we can move this one can be moved to subflow.c.

Yep, it will lose the 'static' scope, but protocol.c is becoming quite big...

WDYT?

Thanks,

Paolo

author
#8 lore raw

[MPTCP] Re: [RFC 6/7] add mptcp_subflow_shutdown

Quoted 26 lines · Paolo Abeni
Quoted 12 lines · Florian Westphal
net/mptcp/protocol.c +12 -0
+static void mptcp_subflow_shutdown(struct sock *ssk, int how)
+{
+	lock_sock(ssk);
+
+	switch (ssk->sk_state) {
+	case TCP_LISTEN:
+		if (!(how & RCV_SHUTDOWN))
+			break;
+		/* fall through */
+	case TCP_SYN_SENT:
+		tcp_disconnect(ssk, O_NONBLOCK);
+		break;

net/mptcp/protocol.c static void mptcp_cancel_work(struct sock *sk) · line 908

Quoted 10 lines
net/mptcp/protocol.c +10 -0
+	default:
+		ssk->sk_shutdown |= how;
+		tcp_shutdown(ssk, how);
+		break;
+	}
+
+	/* Wake up anyone sleeping in poll. */
+	ssk->sk_state_change(ssk);
+	release_sock(ssk);
+}

I think we can move this one can be moved to subflow.c.

Quoted 2 lines

Yep, it will lose the 'static' scope, but protocol.c is becoming quite big...

Yes, but its still small compared to e.g. tcp_input.c.

Quoted 1 line

WDYT?

I like static scope. For every visible symbol one needs to check where else it might be used before making changes to it...

#9 lore raw

[MPTCP] Re: [RFC 6/7] add mptcp_subflow_shutdown

Quoted 30 lines · Florian Westphal
Quoted 26 lines · Paolo Abeni
Quoted 12 lines · Florian Westphal
net/mptcp/protocol.c +12 -0
+static void mptcp_subflow_shutdown(struct sock *ssk, int how)
+{
+	lock_sock(ssk);
+
+	switch (ssk->sk_state) {
+	case TCP_LISTEN:
+		if (!(how & RCV_SHUTDOWN))
+			break;
+		/* fall through */
+	case TCP_SYN_SENT:
+		tcp_disconnect(ssk, O_NONBLOCK);
+		break;

net/mptcp/protocol.c static void mptcp_cancel_work(struct sock *sk) · line 908

Quoted 10 lines
net/mptcp/protocol.c +10 -0
+	default:
+		ssk->sk_shutdown |= how;
+		tcp_shutdown(ssk, how);
+		break;
+	}
+
+	/* Wake up anyone sleeping in poll. */
+	ssk->sk_state_change(ssk);
+	release_sock(ssk);
+}

I think we can move this one can be moved to subflow.c.

Quoted 2 lines

Yep, it will lose the 'static' scope, but protocol.c is becoming quite big...

Quoted 6 lines

Yes, but its still small compared to e.g. tcp_input.c.

Quoted 1 line

WDYT?

I like static scope. For every visible symbol one needs to check where else it might be used before making changes to it...

Oks, no objections on my side.

Cheers,

Paolo

author
#10 lore raw

RFC 7/7 make accept not allocate kernel socket struct

In reply to Florian Westphal

Once lower-level accept returns, walk the subflow list (there should be one subflow) and make sk->sk_socket point to the new mptcp socket.

This is needed to make NOSPACE flag work: tcp stack can't signal availability of new data unless ->sk_socket points to a "struct socket".

If sk_socket is NULL, then from TCP stack p.o.v. the socket has been detached from the userspace program already, so nothing to wake up in first place.

mptcp_close needs to handle both cases: connect()ing sockets have a 'struct socket' allocated, while incoming ones do not.

Signed-off-by Florian Westphal <fw(a)strlen.de>
---
 net/mptcp/protocol.c | 66 +++++++++++++++++++++++++++++++++++---------
 1 file changed, 53 insertions(+), 13 deletions(-)
net/mptcp/protocol.c +53 -13
index 924e63b354ef..bd96d58beccc 100644
@@ -924,10 +924,17 @@ static void mptcp_close(struct sock *sk, long timeout)
 
 	list_for_each_entry_safe(subflow, tmp, &msk->conn_list, node) {
 		struct sock *ssk = mptcp_subflow_tcp_sock(subflow);
+		struct socket *sock = READ_ONCE(ssk->sk_socket);
 
 		pr_debug("conn_list->subflow=%p", subflow);
 		list_del(&subflow->node);
-		sock_release(ssk->sk_socket);
+
+		if (sock && sock != sk->sk_socket) {
+			sock_release(sock);
+		} else {
+			sock_orphan(ssk);
+			tcp_close(ssk, timeout);
+		}
 	}
 
 	__mptcp_clear_xmit(sk);
@@ -975,23 +982,22 @@ static struct sock *mptcp_accept(struct sock *sk, int flags, int *err,
 {
 	struct mptcp_sock *msk = mptcp_sk(sk);
 	struct mptcp_subflow_context *subflow;
-	struct socket *new_sock;
 	struct socket *listener;
 	struct sock *newsk;
 
 	listener = msk->subflow;
 
 	pr_debug("msk=%p, listener=%p", msk, mptcp_subflow_ctx(listener->sk));
-	*err = kernel_accept(listener, &new_sock, flags);
-	if (*err < 0)
+	newsk = inet_csk_accept(listener->sk, flags, err, kern);
+	if (!newsk)
 		return NULL;
 
-	subflow = mptcp_subflow_ctx(new_sock->sk);
+	subflow = mptcp_subflow_ctx(newsk);
 	pr_debug("msk=%p, new subflow=%p, ", msk, subflow);
 
 	if (subflow->mp_capable) {
-		struct sock *ssk = new_sock->sk;
 		struct sock *new_mptcp_sock;
+		struct sock *ssk = newsk;
 		u64 ack_seq;
 
 		lock_sock(sk);
@@ -1002,8 +1008,8 @@ static struct sock *mptcp_accept(struct sock *sk, int flags, int *err,
 			*err = -ENOBUFS;
 			local_bh_enable();
 			release_sock(sk);
-			kernel_sock_shutdown(new_sock, SHUT_RDWR);
-			sock_release(new_sock);
+			mptcp_subflow_shutdown(newsk, SHUT_RDWR + 1);
+			tcp_close(newsk, 0);
 			return NULL;
 		}
 
@@ -1014,7 +1020,7 @@ static struct sock *mptcp_accept(struct sock *sk, int flags, int *err,
 		msk->local_key = subflow->local_key;
 		msk->token = subflow->token;
 
-		mptcp_token_update_accept(new_sock->sk, new_mptcp_sock);
+		mptcp_token_update_accept(newsk, new_mptcp_sock);
 		msk->subflow = NULL;
 
 		mptcp_pm_new_connection(msk, 1);
@@ -1047,10 +1053,7 @@ static struct sock *mptcp_accept(struct sock *sk, int flags, int *err,
 			mptcp_subflow_data_available(ssk);
 		release_sock(ssk);
 	} else {
-		newsk = new_sock->sk;
 		tcp_sk(newsk)->is_mptcp = 0;
-		new_sock->sk = NULL;
-		sock_release(new_sock);
 
 		MPTCP_INC_STATS(sock_net(sk),
 				MPTCP_MIB_MPCAPABLEPASSIVEFALLBACK);
@@ -1213,15 +1216,27 @@ void mptcp_finish_connect(struct sock *sk, int mp_capable)
 	inet_sk_state_store(sk, TCP_ESTABLISHED);
 }
 
+static void mptcp_sock_graft(struct sock *sk, struct socket *parent)
+{
+	write_lock_bh(&sk->sk_callback_lock);
+	rcu_assign_pointer(sk->sk_wq, &parent->wq);
+	sk_set_socket(sk, parent);
+	sk->sk_uid = SOCK_INODE(parent)->i_uid;
+	write_unlock_bh(&sk->sk_callback_lock);
+}
+
 void mptcp_finish_join(struct sock *sk)
 {
 	struct mptcp_subflow_context *subflow = mptcp_subflow_ctx(sk);
 	struct mptcp_sock *msk = mptcp_sk(subflow->conn);
+	struct sock *parent = (void *)msk;
 
 	pr_debug("msk=%p, subflow=%p", msk, subflow);
 
 	local_bh_disable();
 	bh_lock_sock_nested(subflow->conn);
+	if (!sk->sk_socket)
+		mptcp_sock_graft(sk, parent->sk_socket);
 	list_add_tail(&subflow->node, &msk->conn_list);
 	bh_unlock_sock(subflow->conn);
 	local_bh_enable();
@@ -1393,6 +1408,8 @@ static int mptcp_v4_getname(struct socket *sock, struct sockaddr *uaddr,
 }
 
 #if IS_ENABLED(CONFIG_MPTCP_IPV6)
+static struct proto mptcp_v6_prot;
+
 static int mptcp_v6_getname(struct socket *sock, struct sockaddr *uaddr,
 			    int peer)
 {
@@ -1410,6 +1427,16 @@ static int mptcp_v6_getname(struct socket *sock, struct sockaddr *uaddr,
 
 	return mptcp_getname(sock, uaddr, peer, AF_INET6);
 }
+
+static bool is_mptcp_v6(const struct socket *sock)
+{
+	return sock->sk->sk_prot == &mptcp_v6_prot;
+}
+#else
+static bool is_mptcp_v6(const struct socket *sock)
+{
+	return false;
+}
 #endif
 
 static int mptcp_listen(struct socket *sock, int backlog)
@@ -1443,6 +1470,20 @@ static int mptcp_stream_accept(struct socket *sock, struct socket *newsock,
 		return -EINVAL;
 
 	err = ssock->ops->accept(sock, newsock, flags, kern);
+	if (err == 0 && (newsock->sk->sk_prot == &mptcp_prot ||
+			 is_mptcp_v6(newsock))) {
+		struct mptcp_sock *msk = mptcp_sk(newsock->sk);
+		struct mptcp_subflow_context *subflow;
+
+		/* Hack: Need to set ssk->sk_socket of accept()ed flows to mptcp socket. */
+		list_for_each_entry(subflow, &msk->conn_list, node) {
+			struct sock *ssk = mptcp_subflow_tcp_sock(subflow);
+
+			if (!ssk->sk_socket)
+				mptcp_sock_graft(ssk, newsock);
+		}
+	}
+
 	sock_put(ssock->sk);
 	return err;
 }
@@ -1563,7 +1604,6 @@ void mptcp_proto_init(void)
 
 #if IS_ENABLED(CONFIG_MPTCP_IPV6)
 static struct proto_ops mptcp_v6_stream_ops;
-static struct proto mptcp_v6_prot;
 
 static void mptcp_v6_destroy(struct sock *sk)
 {
Signature · 1 line
2.23.0
#11 lore raw

[MPTCP] Re: [RFC 7/7] make accept not allocate kernel socket struct

net/mptcp/protocol.c static void mptcp_close(struct sock *sk, long timeout) · line 933

Quoted 12 lines · Florian Westphal
net/mptcp/protocol.c +4 -1
@@ -924,10 +924,17 @@ static void mptcp_close(struct sock *sk, long timeout)
 
 	list_for_each_entry_safe(subflow, tmp, &msk->conn_list, node) {
 		struct sock *ssk = mptcp_subflow_tcp_sock(subflow);
+		struct socket *sock = READ_ONCE(ssk->sk_socket);
 
 		pr_debug("conn_list->subflow=%p", subflow);
 		list_del(&subflow->node);
-		sock_release(ssk->sk_socket);
+
+		if (sock && sock != sk->sk_socket) {
+			sock_release(sock);

Double checking I read the above correctly: the condition 'sock && sock != sk->sk_socket' only for the first subflow of a client socket, right? If so, can we use the msk socket even for that one?

net/mptcp/protocol.c static int mptcp_stream_accept(struct socket *sock, struct socket *newsock, · line 1474

Quoted 6 lines
net/mptcp/protocol.c +2 -0
@@ -1443,6 +1470,20 @@ static int mptcp_stream_accept(struct socket *sock, struct socket *newsock,
 		return -EINVAL;
 
 	err = ssock->ops->accept(sock, newsock, flags, kern);
+	if (err == 0 && (newsock->sk->sk_prot == &mptcp_prot ||
+			 is_mptcp_v6(newsock))) {

Is not clear to me why/how we can hit the condition '!(newsock->sk-

Quoted 1 line

sk_prot == &mptcp_prot || is_mptcp_v6(newsock))' ... Can you please

explain?

Thanks!

Paolo

author
#12 lore raw

[MPTCP] Re: [RFC 7/7] make accept not allocate kernel socket struct

Quoted 7 lines · Paolo Abeni
Quoted 6 lines · Florian Westphal
net/mptcp/protocol.c +1 -0
@@ -924,10 +924,17 @@ static void mptcp_close(struct sock *sk, long timeout)
 
 	list_for_each_entry_safe(subflow, tmp, &msk->conn_list, node) {
 		struct sock *ssk = mptcp_subflow_tcp_sock(subflow);
+		struct socket *sock = READ_ONCE(ssk->sk_socket);
 
Quoted 10 lines

net/mptcp/protocol.c static void mptcp_close(struct sock *sk, long timeout) · line 933

Quoted 6 lines
net/mptcp/protocol.c +3 -1
 		pr_debug("conn_list->subflow=%p", subflow);
 		list_del(&subflow->node);
-		sock_release(ssk->sk_socket);
+
+		if (sock && sock != sk->sk_socket) {
+			sock_release(sock);

Double checking I read the above correctly: the condition 'sock && sock != sk->sk_socket' only for the first subflow of a client socket, right? If so, can we use the msk socket even for that one?

No, its also for outgoing connections, we use in kernel sockets for those.

Quoted 10 lines

net/mptcp/protocol.c static int mptcp_stream_accept(struct socket *sock, struct socket *newsock, · line 1474

Quoted 6 lines
net/mptcp/protocol.c +2 -0
@@ -1443,6 +1470,20 @@ static int mptcp_stream_accept(struct socket *sock, struct socket *newsock,
 		return -EINVAL;
 
 	err = ssock->ops->accept(sock, newsock, flags, kern);
+	if (err == 0 && (newsock->sk->sk_prot == &mptcp_prot ||
+			 is_mptcp_v6(newsock))) {

Is not clear to me why/how we can hit the condition '!(newsock->sk-

Quoted 1 line

sk_prot == &mptcp_prot || is_mptcp_v6(newsock))' ... Can you please

explain?

We will hit it when we get connection from non-mptcp peer, i.e. the !mp_capable part of mptcp_accept().

In that case sk_prot is &tcp_prot.

It might make sense to change this to

    err == 0 && newsock->sk->sk_prot != &tcp_prot

perhaps that would clarify this a bit. WDYT?

#13 lore raw

[MPTCP] Re: [RFC 7/7] make accept not allocate kernel socket struct

Quoted 19 lines · Florian Westphal
Quoted 7 lines · Paolo Abeni
Quoted 6 lines · Florian Westphal
net/mptcp/protocol.c +1 -0
@@ -924,10 +924,17 @@ static void mptcp_close(struct sock *sk, long timeout)
 
 	list_for_each_entry_safe(subflow, tmp, &msk->conn_list, node) {
 		struct sock *ssk = mptcp_subflow_tcp_sock(subflow);
+		struct socket *sock = READ_ONCE(ssk->sk_socket);
 
Quoted 10 lines

net/mptcp/protocol.c static void mptcp_close(struct sock *sk, long timeout) · line 933

Quoted 6 lines
net/mptcp/protocol.c +3 -1
 		pr_debug("conn_list->subflow=%p", subflow);
 		list_del(&subflow->node);
-		sock_release(ssk->sk_socket);
+
+		if (sock && sock != sk->sk_socket) {
+			sock_release(sock);

Double checking I read the above correctly: the condition 'sock && sock != sk->sk_socket' only for the first subflow of a client socket, right? If so, can we use the msk socket even for that one?

Quoted 2 lines

No, its also for outgoing connections, we use in kernel sockets for those.

Uhm... I see. Everything created via mptcp_subflow_create_socket(), right? I though of always using msk's socket also/mainly to reduce the memory usage, but server side we should have very few mptcp_subflow_create_socket() instances, so I guess it's no necessary optimize them.

Quoted 14 lines
Quoted 10 lines

net/mptcp/protocol.c static int mptcp_stream_accept(struct socket *sock, struct socket *newsock, · line 1474

Quoted 6 lines
net/mptcp/protocol.c +2 -0
@@ -1443,6 +1470,20 @@ static int mptcp_stream_accept(struct socket *sock, struct socket *newsock,
 		return -EINVAL;
 
 	err = ssock->ops->accept(sock, newsock, flags, kern);
+	if (err == 0 && (newsock->sk->sk_prot == &mptcp_prot ||
+			 is_mptcp_v6(newsock))) {

Is not clear to me why/how we can hit the condition '!(newsock->sk-

Quoted 1 line

sk_prot == &mptcp_prot || is_mptcp_v6(newsock))' ... Can you please

explain?

We will hit it when we get connection from non-mptcp peer, i.e. the !mp_capable part of mptcp_accept().

Quoted 6 lines

In that case sk_prot is &tcp_prot.

It might make sense to change this to

    err == 0 && newsock->sk->sk_prot != &tcp_prot

perhaps that would clarify this a bit. WDYT?

Yep, I like this latter code more!

Thank you,

Paolo

author
#14 lore raw

[MPTCP] Re: [RFC 7/7] make accept not allocate kernel socket struct

Quoted 8 lines · Paolo Abeni
Quoted 2 lines

No, its also for outgoing connections, we use in kernel sockets for those.

Uhm... I see. Everything created via mptcp_subflow_create_socket(), right? I though of always using msk's socket also/mainly to reduce the memory usage, but server side we should have very few mptcp_subflow_create_socket() instances, so I guess it's no necessary optimize them.

FWIW it might make sense to not have those in kernel sockets either in the future, but this might need a bit more changes to how outgoing connections work in mptcp-next.

I would have a look at it at some later date, I don't think its a high priority.

Quoted 14 lines
Quoted 14 lines
Quoted 10 lines

net/mptcp/protocol.c static int mptcp_stream_accept(struct socket *sock, struct socket *newsock, · line 1474

Quoted 6 lines
net/mptcp/protocol.c +2 -0
@@ -1443,6 +1470,20 @@ static int mptcp_stream_accept(struct socket *sock, struct socket *newsock,
 		return -EINVAL;
 
 	err = ssock->ops->accept(sock, newsock, flags, kern);
+	if (err == 0 && (newsock->sk->sk_prot == &mptcp_prot ||
+			 is_mptcp_v6(newsock))) {

Is not clear to me why/how we can hit the condition '!(newsock->sk-

Quoted 1 line

sk_prot == &mptcp_prot || is_mptcp_v6(newsock))' ... Can you please

explain?

We will hit it when we get connection from non-mptcp peer, i.e. the !mp_capable part of mptcp_accept().

Quoted 8 lines
Quoted 6 lines

In that case sk_prot is &tcp_prot.

It might make sense to change this to

    err == 0 && newsock->sk->sk_prot != &tcp_prot

perhaps that would clarify this a bit. WDYT?

Yep, I like this latter code more!

Ok, I've flagged the patches 'RFC' in patchwork already. I will make this change and will resubmit that part later.

Thanks,
Florian

#15 lore raw
In reply to Florian Westphal
Quoted 39 lines · Florian Westphal

This set aims to improve handling of multiple subflows.
There are multiple remaining problems, see below, but I wanted to send what I have now rather than keep this back.

The first 3 patches are fixes to handle refcount leaks found while working on this, I've added the commit name of where they could be squashed to.

The remaining part of this work is to make the msk->conn_list contain tcp_sock structs without an outer 'struct socket'.

The problem is that for incoming join requests (which are not subject to accept/request socket queue), there is no 'struct socket', i.e. sk->sk_socket is NULL.

As tcp stack will assume that such tcp sk is already detached from userspace, SOCK_NOSPACE wakeups do not work for them.

The solution here (suggested by Paolo) is to have sk->sk_socket refer to the mptcp parent socket structure. This assignment is done by a private function similar to sock_graft().

For incoming connects, tcp-level connectivity works, i.e. mptcp-next can accept a few subflows from the peer and on exit of the test program all resources are cleaned up correctly.

MPTCP-Level connectivity will probably not work, as nothing sets up ssn_offset for joins (both incoming or outgoing).

mptcp.org kernel will not consider the joins as established, as the "4th Ack" is missing.
A simple "tcp_send_ack" is enough, but there are two problems with this:
1. It can be lost
2. Given we are not expecting to receive any data on such flows this doesn't seem to be a problem.

During testing I saw that sometimes mptcp.org kernel will emit further join requests to mptcp-next but those trigger no response (and those packets do not even show up in perfs network drop monitor, sigh).

Quoted 5 lines

For outgoing connects, there is one remaining memory leak that I have not located so far.
_______________________________________________
mptcp mailing list -- mptcp(a)lists.01.org
To unsubscribe send an email to mptcp-leave(a)lists.01.org

LGTM! I think the firs 3 patches can be squashed as-is. I have only a couple of minor question on the last 2 patches, see later individual replies...

Great work, BTW ;)

Thanks,

Paolo

#16 lore raw

Hi Florian, Paolo,

Quoted 7 lines · Paolo Abeni
Quoted 6 lines · Florian Westphal

This set aims to improve handling of multiple subflows.
There are multiple remaining problems, see below, but I wanted to send what I have now rather than keep this back.

The first 3 patches are fixes to handle refcount leaks found while working on this, I've added the commit name of where they could be squashed to.

[...]

Quoted 1 line

LGTM! I think the firs 3 patches can be squashed as-is. [...]

Thank you for the nice work and the review!

- b6f0623fedf6: "squashed" patch 1/7 in "mptcp: Add key generation and token tree"
- 07ec6803478e: "squashed" patch 2/7 in "mptcp: Add handling of incoming MP_JOIN requests"
- 1be614c6efe0: "Signed-off-by" + "Co-developed-by"
- 3af80e92c8cd..d5e48af1fc7f: result
- 4291db56b807: "squashed" patch 3/7 in "mptcp: Handle MP_CAPABLE options for outgoing connections"
- 8c236d0f5ad2: "Signed-off-by" + "Co-developed-by"
- d5e48af1fc7f..722fd4571d61: results

Tests are still OK, each commit can still compile.

Cheers,
Matt

Signature · 5 lines
Matthieu Baerts | R&D Engineer
matthieu.baerts(a)tessares.net
Tessares SA | Hybrid Access Solutions
www.tessares.net
1 Avenue Jean Monnet, 1348 Louvain-la-Neuve, Belgium
author
#17 lore raw
In reply to Florian Westphal
Quoted 6 lines · Florian Westphal

This set aims to improve handling of multiple subflows.
There are multiple remaining problems, see below, but I wanted to send what I have now rather than keep this back.

The first 3 patches are fixes to handle refcount leaks found while working on this, I've added the commit name of where they could be squashed to.

I see those have been marked 'awaiting upstream', what does that mean in mptcp context?

#18 lore raw
Quoted 10 lines · Florian Westphal
Quoted 6 lines · Florian Westphal

This set aims to improve handling of multiple subflows.
There are multiple remaining problems, see below, but I wanted to send what I have now rather than keep this back.

The first 3 patches are fixes to handle refcount leaks found while working on this, I've added the commit name of where they could be squashed to.

I see those have been marked 'awaiting upstream', what does that mean in mptcp context?

It means the patches are ready to be merged, waiting for someone (Matt?) to merge them.

It looks correct to me for the first 3 patches.

/P

#19 lore raw

Hi Paolo, Florian,

Quoted 9 lines · Paolo Abeni
Quoted 8 lines · Florian Westphal
Quoted 6 lines · Florian Westphal

This set aims to improve handling of multiple subflows.
There are multiple remaining problems, see below, but I wanted to send what I have now rather than keep this back.

The first 3 patches are fixes to handle refcount leaks found while working on this, I've added the commit name of where they could be squashed to.

Quoted 7 lines
Quoted 2 lines

I see those have been marked 'awaiting upstream', what does that mean in mptcp context?

It means the patches are ready to be merged, waiting for someone (Matt?) to merge them.

It looks correct to me for the first 3 patches.

Sorry, I already merged them this afternoon but I was waiting for the CI to finish validating the new changes to send the email. Everything done now, the "export" branch contains the 3 patches.

Cheers,
Matt

Signature · 5 lines
Matthieu Baerts | R&D Engineer
matthieu.baerts(a)tessares.net
Tessares SA | Hybrid Access Solutions
www.tessares.net
1 Avenue Jean Monnet, 1348 Louvain-la-Neuve, Belgium