Skip to contentSkip to conversation

Fix socket options used by NVMe over MPTCP

PATCH Superseded Geliang Tang 2 participants 8 messages Jul 25

Subsystems (3)

v1 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

Geliang Tang

author reviewer
#0 lore raw

cover Fix socket options used by NVMe over MPTCP

From: Geliang Tang <[email protected]>

NVMe over MPTCP relies on SO_LINGER, SO_PRIORITY, SO_REUSEADDR, TCP_SYNCNT, TCP_NODELAY, IP_TOS, and SO_BINDTODEVICE. This series contains fixes to make all of them work correctly. Patch 4 is a cleanup needed by patch 5. Patch 6 adds a selftest to cover these options.

Geliang Tang (6):
mptcp: copy the subflow's rcv_tos to the msk on accept
mptcp: don't reset dst when setting default 0 tos
mptcp: inherit sk_reuse on subflow creation
mptcp: handle TCP_MAXSEG getsockopt in common case
mptcp: add TCP_SYNCNT setsockopt/getsockopt
selftests: mptcp: sockopt: cover LINGER REUSEADDR PRIORITY NODELAY
SYNCNT

 net/mptcp/protocol.c                          |   2 +
 net/mptcp/sockopt.c                           |  28 ++++-
 .../selftests/net/mptcp/mptcp_sockopt.c       | 116 ++++++++++++++++++
 3 files changed, 141 insertions(+), 5 deletions(-)
Signature · 1 line
2.53.0
author reviewer
#1 lore raw

PATCH 1/6 mptcp: copy the subflow's rcv_tos to the msk on accept

From: Geliang Tang <[email protected]>

The server-side master socket never inherited the tos seen in the peer's SYN. mptcp_sk_clone_init() now copies ssk->rcv_tos onto nsk so inet->rcv_tos on the accepted msk reflects the peer's tos byte (matching plain TCP).

---
 net/mptcp/protocol.c | 2 ++
 1 file changed, 2 insertions(+)
net/mptcp/protocol.c +2 -0
index 954e20bb27de..178cd8de990b 100644
@@ -3699,6 +3699,8 @@ struct sock *mptcp_sk_clone_init(const struct sock *sk,
 #endif
 		mptcp_copy_ip_options(nsk, sk);
 
+	inet_sk(nsk)->rcv_tos = inet_sk(ssk)->rcv_tos;
+
 	msk = mptcp_sk(nsk);
 	WRITE_ONCE(msk->local_key, subflow_req->local_key);
 	WRITE_ONCE(msk->token, subflow_req->token);
Signature · 1 line
2.53.0
author reviewer
#2 lore raw

PATCH 2/6 mptcp: don't reset dst when setting default 0 tos

In reply to Geliang Tang

From: Geliang Tang <[email protected]>

__ip_sock_set_tos() is called unconditionally during setsockopt and on each sync_socket_options() pass, even when inet_sk(sk)->tos is 0 (the default). On SOCK_STREAM the helper masks out the ECN bits and writes sk_tos2prio(), which leaves priority=0 alone but still calls sk_dst_reset() unconditionally, forcing a route lookup on the next packet. Guard both call sites with 'if (val > 0)' so the default-0 case is a no-op.

---
 net/mptcp/sockopt.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
net/mptcp/sockopt.c +4 -2
index fcf6feb2a9eb..1b2487a5ae2f 100644
@@ -762,7 +762,8 @@ static int mptcp_setsockopt_v4_set_tos(struct mptcp_sock *msk, int optname,
 		bool slow;
 
 		slow = lock_sock_fast(ssk);
-		__ip_sock_set_tos(ssk, val);
+		if (val > 0)
+			__ip_sock_set_tos(ssk, val);
 		unlock_sock_fast(ssk, slow);
 	}
 	release_sock(sk);
@@ -1562,7 +1563,8 @@ static void sync_socket_options(struct mptcp_sock *msk, struct sock *ssk)
 	ssk->sk_bound_dev_if = sk->sk_bound_dev_if;
 	ssk->sk_incoming_cpu = sk->sk_incoming_cpu;
 	ssk->sk_ipv6only = sk->sk_ipv6only;
-	__ip_sock_set_tos(ssk, inet_sk(sk)->tos);
+	if (inet_sk(sk)->tos > 0)
+		__ip_sock_set_tos(ssk, inet_sk(sk)->tos);
 
 	if (sk->sk_userlocks & tx_rx_locks) {
 		ssk->sk_userlocks |= sk->sk_userlocks & tx_rx_locks;
Signature · 1 line
2.53.0
author reviewer
#3 lore raw

PATCH 3/6 mptcp: inherit sk_reuse on subflow creation

In reply to Geliang Tang

From: Geliang Tang <[email protected]>

Propagate sk_reuse from the MPTCP master socket to newly-created subflows in sync_socket_options(), mirroring how sk_reuseport and sk_reuse are already handled for the subflow.

Without this, a subflow started after the master toggled SO_REUSEADDR would not inherit the setting, leading to inconsistent reuse semantics between msk and ssk.

---
 net/mptcp/sockopt.c | 2 ++
 1 file changed, 2 insertions(+)
net/mptcp/sockopt.c +2 -0
index 1b2487a5ae2f..594ce3b4b707 100644
@@ -1603,6 +1603,8 @@ static void sync_socket_options(struct mptcp_sock *msk, struct sock *ssk)
 	inet_assign_bit(FREEBIND, ssk, inet_test_bit(FREEBIND, sk));
 	inet_assign_bit(BIND_ADDRESS_NO_PORT, ssk, inet_test_bit(BIND_ADDRESS_NO_PORT, sk));
 	WRITE_ONCE(inet_sk(ssk)->local_port_range, READ_ONCE(inet_sk(sk)->local_port_range));
+
+	ssk->sk_reuse = sk->sk_reuse;
 }
 
 void mptcp_sockopt_sync_locked(struct mptcp_sock *msk, struct sock *ssk)
Signature · 1 line
2.53.0
author reviewer
#4 lore raw

PATCH 4/6 mptcp: handle TCP_MAXSEG getsockopt in common case

In reply to Geliang Tang

From: Geliang Tang <[email protected]>

The TCP_MAXSEG getsockopt() case was a tail-clause returning mptcp_getsockopt_first_sf_only() while every other "first subflow only" option (TCP_FASTOPEN_*) was already grouped in the common case. Move it next to its peers so the dispatch reads uniformly.

No behavioural change; the lookup still returns the cached MSS on the first subflow.

---
 net/mptcp/sockopt.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
net/mptcp/sockopt.c +1 -3
index 594ce3b4b707..e40a5a23d1c9 100644
@@ -1424,6 +1424,7 @@ static int mptcp_getsockopt_sol_tcp(struct mptcp_sock *msk, int optname,
 	case TCP_FASTOPEN_CONNECT:
 	case TCP_FASTOPEN_KEY:
 	case TCP_FASTOPEN_NO_COOKIE:
+	case TCP_MAXSEG:
 		return mptcp_getsockopt_first_sf_only(msk, SOL_TCP, optname,
 						      optval, optlen);
 	case TCP_INQ:
@@ -1448,9 +1449,6 @@ static int mptcp_getsockopt_sol_tcp(struct mptcp_sock *msk, int optname,
 		return mptcp_put_int_option(msk, optval, optlen, msk->notsent_lowat);
 	case TCP_IS_MPTCP:
 		return mptcp_put_int_option(msk, optval, optlen, 1);
-	case TCP_MAXSEG:
-		return mptcp_getsockopt_first_sf_only(msk, SOL_TCP, optname,
-						      optval, optlen);
 	}
 	return -EOPNOTSUPP;
 }
Signature · 1 line
2.53.0
author reviewer
#5 lore raw

PATCH 5/6 mptcp: add TCP_SYNCNT setsockopt/getsockopt

In reply to Geliang Tang

From: Geliang Tang <[email protected]>

Surface TCP_SYNCNT to userspace on the MPTCP master socket.

setsockopt(TCP_SYNCNT) validates the value via __mptcp_setsockopt_set_val() and stores it in inet_csk(sk)->icsk_syn_retries on the msk. getsockopt() reads the same field back.

A newly created subflow picks up the master socket's syn_retries through sync_socket_options(), so the setting takes effect for outgoing SYNs without further plumbing.

---
 net/mptcp/sockopt.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
net/mptcp/sockopt.c +16 -0
index e40a5a23d1c9..50dcb53effe2 100644
@@ -897,6 +897,17 @@ static int mptcp_setsockopt_sol_tcp(struct mptcp_sock *msk, int optname,
 		ret = mptcp_setsockopt_all_sf(msk, SOL_TCP, optname, optval,
 					      optlen);
 		break;
+	case TCP_SYNCNT: {
+		int syncnt = inet_csk(sk)->icsk_syn_retries;
+
+		ret = __mptcp_setsockopt_set_val(msk, MAX_TCP_SYNCNT,
+						 &tcp_sock_set_syncnt,
+						 &syncnt,
+						 val);
+		if (ret == 0)
+			inet_csk(sk)->icsk_syn_retries = syncnt;
+		break;
+	}
 	default:
 		ret = -ENOPROTOOPT;
 	}
@@ -1449,6 +1460,9 @@ static int mptcp_getsockopt_sol_tcp(struct mptcp_sock *msk, int optname,
 		return mptcp_put_int_option(msk, optval, optlen, msk->notsent_lowat);
 	case TCP_IS_MPTCP:
 		return mptcp_put_int_option(msk, optval, optlen, 1);
+	case TCP_SYNCNT:
+		return mptcp_put_int_option(msk, optval, optlen,
+					    inet_csk(sk)->icsk_syn_retries);
 	}
 	return -EOPNOTSUPP;
 }
@@ -1603,6 +1617,8 @@ static void sync_socket_options(struct mptcp_sock *msk, struct sock *ssk)
 	WRITE_ONCE(inet_sk(ssk)->local_port_range, READ_ONCE(inet_sk(sk)->local_port_range));
 
 	ssk->sk_reuse = sk->sk_reuse;
+	if (inet_csk(sk)->icsk_syn_retries > 0)
+		tcp_sock_set_syncnt(ssk, inet_csk(sk)->icsk_syn_retries);
 }
 
 void mptcp_sockopt_sync_locked(struct mptcp_sock *msk, struct sock *ssk)
Signature · 1 line
2.53.0
author reviewer
#6 lore raw

PATCH 6/6 selftests: mptcp: sockopt: cover LINGER REUSEADDR PRIORITY NODELAY SYNCNT

In reply to Geliang Tang

From: Geliang Tang <[email protected]>

Extend mptcp_sockopt.c with tests for SOL_SOCKET (SO_REUSEADDR, SO_LINGER, SO_PRIORITY) and SOL_TCP (TCP_NODELAY, TCP_SYNCNT), each exercising both setsockopt() and getsockopt() on a connected MPTCP socket.

SO_REUSEADDR on the msk reports 1 (SK_CAN_REUSE) once any non-zero value is set; this captures the documented behaviour where the master socket reflects the subflow's reuse state.

TCP_SYNCNT coverage pairs with the kernel side that propagates icsk_syn_retries to the subflow.

---
 .../selftests/net/mptcp/mptcp_sockopt.c       | 116 ++++++++++++++++++
 1 file changed, 116 insertions(+)
tools/testing/selftests/net/mptcp/mptcp_sockopt.c +116 -0
index b6e58d936ebe..b04e24df9dfe 100644
@@ -35,6 +35,9 @@ static int pf = AF_INET;
 #ifndef SOL_MPTCP
 #define SOL_MPTCP 284
 #endif
+#ifndef SOL_TCP
+#define SOL_TCP 6
+#endif
 
 #ifndef MPTCP_INFO
 struct mptcp_info {
@@ -229,6 +232,30 @@ static int sock_listen_mptcp(const char * const listenaddr,
 	return sock;
 }
 
+static void test_so_reuseaddr_sockopt(int fd)
+{
+	int val_in = 1, val_out, expected;
+	socklen_t s;
+	int r;
+
+	r = setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &val_in, sizeof(val_in));
+	if (r != 0)
+		die_perror("setsockopt SO_REUSEADDR");
+
+	s = sizeof(val_out);
+	r = getsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &val_out, &s);
+	if (r != 0)
+		die_perror("getsockopt SO_REUSEADDR");
+
+	/* SO_REUSEADDR on MPTCP msk returns 1 (SK_CAN_REUSE) when set,
+	 * regardless of the input value. The actual reuse semantics are
+	 * applied to the nmpc subflow.
+	 */
+	expected = val_in ? 1 : 0;
+	if (val_out != expected)
+		xerror("SO_REUSEADDR %d != %d\n", expected, val_out);
+}
+
 static int sock_connect_mptcp(const char * const remoteaddr,
 			      const char * const port, int proto)
 {
@@ -247,6 +274,8 @@ static int sock_connect_mptcp(const char * const remoteaddr,
 		if (sock < 0)
 			continue;
 
+		test_so_reuseaddr_sockopt(sock);
+
 		if (connect(sock, a->ai_addr, a->ai_addrlen) == 0)
 			break; /* success */
 
@@ -769,6 +798,89 @@ static void test_ip_tos_sockopt(int fd)
 		xerror("expect socklen_t == -1");
 }
 
+static void test_so_linger_sockopt(int fd)
+{
+	struct linger ling_in, ling_out;
+	socklen_t s;
+	int r;
+
+	ling_in.l_onoff = 1;
+	ling_in.l_linger = 30;
+	r = setsockopt(fd, SOL_SOCKET, SO_LINGER, &ling_in, sizeof(ling_in));
+	if (r != 0)
+		die_perror("setsockopt SO_LINGER");
+
+	s = sizeof(ling_out);
+	memset(&ling_out, 0, sizeof(ling_out));
+	r = getsockopt(fd, SOL_SOCKET, SO_LINGER, &ling_out, &s);
+	if (r != 0)
+		die_perror("getsockopt SO_LINGER");
+
+	if (ling_in.l_onoff != ling_out.l_onoff ||
+	    ling_in.l_linger != ling_out.l_linger)
+		xerror("SO_LINGER %d/%d != %d/%d\n",
+		       ling_in.l_onoff, ling_in.l_linger,
+		       ling_out.l_onoff, ling_out.l_linger);
+}
+
+static void test_so_priority_sockopt(int fd)
+{
+	int prio_in, prio_out;
+	socklen_t s;
+	int r;
+
+	prio_in = 5;
+	r = setsockopt(fd, SOL_SOCKET, SO_PRIORITY, &prio_in, sizeof(prio_in));
+	if (r != 0)
+		die_perror("setsockopt SO_PRIORITY");
+
+	s = sizeof(prio_out);
+	r = getsockopt(fd, SOL_SOCKET, SO_PRIORITY, &prio_out, &s);
+	if (r != 0)
+		die_perror("getsockopt SO_PRIORITY");
+
+	if (prio_in != prio_out)
+		xerror("SO_PRIORITY %d != %d\n", prio_in, prio_out);
+}
+
+static void test_tcp_nodelay_sockopt(int fd)
+{
+	int val_in = 1, val_out;
+	socklen_t s;
+	int r;
+
+	r = setsockopt(fd, SOL_TCP, TCP_NODELAY, &val_in, sizeof(val_in));
+	if (r != 0)
+		die_perror("setsockopt TCP_NODELAY");
+
+	s = sizeof(val_out);
+	r = getsockopt(fd, SOL_TCP, TCP_NODELAY, &val_out, &s);
+	if (r != 0)
+		die_perror("getsockopt TCP_NODELAY");
+
+	if (val_in != val_out)
+		xerror("TCP_NODELAY %d != %d\n", val_in, val_out);
+}
+
+static void test_tcp_syncnt_sockopt(int fd)
+{
+	int val_in = 3, val_out;
+	socklen_t s;
+	int r;
+
+	r = setsockopt(fd, SOL_TCP, TCP_SYNCNT, &val_in, sizeof(val_in));
+	if (r != 0)
+		die_perror("setsockopt TCP_SYNCNT");
+
+	s = sizeof(val_out);
+	r = getsockopt(fd, SOL_TCP, TCP_SYNCNT, &val_out, &s);
+	if (r != 0)
+		die_perror("getsockopt TCP_SYNCNT");
+
+	if (val_in != val_out)
+		xerror("TCP_SYNCNT %d != %d\n", val_in, val_out);
+}
+
 static int client(int pipefd)
 {
 	int fd = -1;
@@ -787,6 +899,10 @@ static int client(int pipefd)
 	}
 
 	test_ip_tos_sockopt(fd);
+	test_so_linger_sockopt(fd);
+	test_so_priority_sockopt(fd);
+	test_tcp_nodelay_sockopt(fd);
+	test_tcp_syncnt_sockopt(fd);
 
 	connect_one_server(fd, pipefd);
 
Signature · 1 line
2.53.0
#7 lore raw
In reply to Geliang Tang

Hi Geliang,

Thank you for your modifications, that's great!

Our CI did some validations and here is its report:

- KVM Validation: normal (except selftest_mptcp_join): Unstable: 1 failed test(s): selftest_simult_flows ⚠️
- KVM Validation: normal (only selftest_mptcp_join): Success! ✅
- KVM Validation: debug (except selftest_mptcp_join): Success! ✅
- KVM Validation: debug (only selftest_mptcp_join): Success! ✅
- KVM Validation: btf-normal (only bpftest_all): Success! ✅
- KVM Validation: btf-debug (only bpftest_all): Success! ✅
- Task: https://github.com/multipath-tcp/mptcp_net-next/actions/runs/30160012728

Initiator: Patchew Applier
Commits: https://github.com/multipath-tcp/mptcp_net-next/commits/bf15e67a2e42
Patchwork: https://patchwork.kernel.org/project/mptcp/list/?series=1134397

If there are some issues, you can reproduce them using the same environment as the one used by the CI thanks to a docker image, e.g.:

    $ cd [kernel source code]
    $ docker run -v "${PWD}:${PWD}:rw" -w "${PWD}" --privileged --rm -it \
        --pull always mptcp/mptcp-upstream-virtme-docker:latest \
        auto-normal

For more details:

https://github.com/multipath-tcp/mptcp-upstream-virtme-docker

Please note that despite all the efforts that have been already done to have a stable tests suite when executed on a public CI like here, it is possible some reported issues are not due to your modifications. Still, do not hesitate to help us improve that ;-)

Cheers,
MPTCP GH Action bot
Bot operated by Matthieu Baerts (NGI0 Core)