Fix socket options used by NVMe over MPTCP
Subsystems (3)
Geliang Tang
reviewercover Fix socket options used by NVMe over MPTCP
From: Geliang Tang <[email protected]>
NVMe over MPTCP relies on SO_LINGER, SO_PRIORITY, SO_REUSEADDR, TCP_SYNCNT, TCP_NODELAY, IP_TOS, and SO_BINDTODEVICE. This series contains fixes to make all of them work correctly. Patch 4 is a cleanup needed by patch 5. Patch 6 adds a selftest to cover these options.
Geliang Tang (6):
mptcp: copy the subflow's rcv_tos to the msk on accept
mptcp: don't reset dst when setting default 0 tos
mptcp: inherit sk_reuse on subflow creation
mptcp: handle TCP_MAXSEG getsockopt in common case
mptcp: add TCP_SYNCNT setsockopt/getsockopt
selftests: mptcp: sockopt: cover LINGER REUSEADDR PRIORITY NODELAY
SYNCNT
net/mptcp/protocol.c | 2 + net/mptcp/sockopt.c | 28 ++++- .../selftests/net/mptcp/mptcp_sockopt.c | 116 ++++++++++++++++++ 3 files changed, 141 insertions(+), 5 deletions(-)
Signature · 1 line
2.53.0
Geliang Tang
reviewerPATCH 1/6 mptcp: copy the subflow's rcv_tos to the msk on accept
From: Geliang Tang <[email protected]>
The server-side master socket never inherited the tos seen in the peer's SYN. mptcp_sk_clone_init() now copies ssk->rcv_tos onto nsk so inet->rcv_tos on the accepted msk reflects the peer's tos byte (matching plain TCP).
--- net/mptcp/protocol.c | 2 ++ 1 file changed, 2 insertions(+)
net/mptcp/protocol.c +2 -0
@@ -3699,6 +3699,8 @@ struct sock *mptcp_sk_clone_init(const struct sock *sk,#endif mptcp_copy_ip_options (nsk, sk); +inet_sk (nsk)->rcv_tos =inet_sk (ssk)->rcv_tos; + msk =mptcp_sk (nsk);WRITE_ONCE (msk->local_key, subflow_req->local_key);WRITE_ONCE (msk->token, subflow_req->token);
Signature · 1 line
2.53.0
Geliang Tang
reviewerPATCH 2/6 mptcp: don't reset dst when setting default 0 tos
In reply to Geliang TangFrom: Geliang Tang <[email protected]>
__ip_sock_set_tos() is called unconditionally during setsockopt and on each sync_socket_options() pass, even when inet_sk(sk)->tos is 0 (the default). On SOCK_STREAM the helper masks out the ECN bits and writes sk_tos2prio(), which leaves priority=0 alone but still calls sk_dst_reset() unconditionally, forcing a route lookup on the next packet. Guard both call sites with 'if (val > 0)' so the default-0 case is a no-op.
--- net/mptcp/sockopt.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
net/mptcp/sockopt.c +4 -2
@@ -762,7 +762,8 @@ static int mptcp_setsockopt_v4_set_tos(struct mptcp_sock *msk, int optname,bool slow; slow =lock_sock_fast (ssk); -__ip_sock_set_tos (ssk, val); +if (val >0 ) +__ip_sock_set_tos (ssk, val);unlock_sock_fast (ssk, slow); }release_sock (sk);
@@ -1562,7 +1563,8 @@ static void sync_socket_options(struct mptcp_sock *msk, struct sock *ssk) ssk->sk_bound_dev_if = sk->sk_bound_dev_if; ssk->sk_incoming_cpu = sk->sk_incoming_cpu; ssk->sk_ipv6only = sk->sk_ipv6only; -__ip_sock_set_tos (ssk,inet_sk (sk)->tos); +if (inet_sk (sk)->tos >0 ) +__ip_sock_set_tos (ssk,inet_sk (sk)->tos);if (sk->sk_userlocks & tx_rx_locks) { ssk->sk_userlocks |= sk->sk_userlocks & tx_rx_locks;
Signature · 1 line
2.53.0
Geliang Tang
reviewerPATCH 3/6 mptcp: inherit sk_reuse on subflow creation
In reply to Geliang TangFrom: Geliang Tang <[email protected]>
Propagate sk_reuse from the MPTCP master socket to newly-created subflows in sync_socket_options(), mirroring how sk_reuseport and sk_reuse are already handled for the subflow.
Without this, a subflow started after the master toggled SO_REUSEADDR would not inherit the setting, leading to inconsistent reuse semantics between msk and ssk.
--- net/mptcp/sockopt.c | 2 ++ 1 file changed, 2 insertions(+)
net/mptcp/sockopt.c +2 -0
@@ -1603,6 +1603,8 @@ static void sync_socket_options(struct mptcp_sock *msk, struct sock *ssk)inet_assign_bit (FREEBIND , ssk,inet_test_bit (FREEBIND , sk));inet_assign_bit (BIND_ADDRESS_NO_PORT , ssk,inet_test_bit (BIND_ADDRESS_NO_PORT , sk));WRITE_ONCE (inet_sk (ssk)->local_port_range,READ_ONCE (inet_sk (sk)->local_port_range)); + + ssk->sk_reuse = sk->sk_reuse; }void mptcp_sockopt_sync_locked (struct mptcp_sock *msk,struct sock *ssk)
Signature · 1 line
2.53.0
Geliang Tang
reviewerPATCH 4/6 mptcp: handle TCP_MAXSEG getsockopt in common case
In reply to Geliang TangFrom: Geliang Tang <[email protected]>
The TCP_MAXSEG getsockopt() case was a tail-clause returning mptcp_getsockopt_first_sf_only() while every other "first subflow only" option (TCP_FASTOPEN_*) was already grouped in the common case. Move it next to its peers so the dispatch reads uniformly.
No behavioural change; the lookup still returns the cached MSS on the first subflow.
--- net/mptcp/sockopt.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
net/mptcp/sockopt.c +1 -3
@@ -1424,6 +1424,7 @@ static int mptcp_getsockopt_sol_tcp(struct mptcp_sock *msk, int optname,case TCP_FASTOPEN_CONNECT :case TCP_FASTOPEN_KEY :case TCP_FASTOPEN_NO_COOKIE : +case TCP_MAXSEG :return mptcp_getsockopt_first_sf_only (msk,SOL_TCP , optname, optval, optlen);case TCP_INQ :
@@ -1448,9 +1449,6 @@ static int mptcp_getsockopt_sol_tcp(struct mptcp_sock *msk, int optname,return mptcp_put_int_option (msk, optval, optlen, msk->notsent_lowat);case TCP_IS_MPTCP :return mptcp_put_int_option (msk, optval, optlen,1 ); -case TCP_MAXSEG : -return mptcp_getsockopt_first_sf_only (msk,SOL_TCP , optname, - optval, optlen); }return -EOPNOTSUPP ; }
Signature · 1 line
2.53.0
Geliang Tang
reviewerPATCH 5/6 mptcp: add TCP_SYNCNT setsockopt/getsockopt
In reply to Geliang TangFrom: Geliang Tang <[email protected]>
Surface TCP_SYNCNT to userspace on the MPTCP master socket.
setsockopt(TCP_SYNCNT) validates the value via __mptcp_setsockopt_set_val() and stores it in inet_csk(sk)->icsk_syn_retries on the msk. getsockopt() reads the same field back.
A newly created subflow picks up the master socket's syn_retries through sync_socket_options(), so the setting takes effect for outgoing SYNs without further plumbing.
--- net/mptcp/sockopt.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
net/mptcp/sockopt.c +16 -0
@@ -897,6 +897,17 @@ static int mptcp_setsockopt_sol_tcp(struct mptcp_sock *msk, int optname, ret =mptcp_setsockopt_all_sf (msk,SOL_TCP , optname, optval, optlen);break ; +case TCP_SYNCNT : { +int syncnt =inet_csk (sk)->icsk_syn_retries; + + ret =__mptcp_setsockopt_set_val (msk,MAX_TCP_SYNCNT , + &tcp_sock_set_syncnt, + &syncnt, + val); +if (ret ==0 ) +inet_csk (sk)->icsk_syn_retries = syncnt; +break ; + }default : ret = -ENOPROTOOPT ; }
@@ -1449,6 +1460,9 @@ static int mptcp_getsockopt_sol_tcp(struct mptcp_sock *msk, int optname,return mptcp_put_int_option (msk, optval, optlen, msk->notsent_lowat);case TCP_IS_MPTCP :return mptcp_put_int_option (msk, optval, optlen,1 ); +case TCP_SYNCNT : +return mptcp_put_int_option (msk, optval, optlen, +inet_csk (sk)->icsk_syn_retries); }return -EOPNOTSUPP ; }
@@ -1603,6 +1617,8 @@ static void sync_socket_options(struct mptcp_sock *msk, struct sock *ssk)WRITE_ONCE (inet_sk (ssk)->local_port_range,READ_ONCE (inet_sk (sk)->local_port_range)); ssk->sk_reuse = sk->sk_reuse; +if (inet_csk (sk)->icsk_syn_retries >0 ) +tcp_sock_set_syncnt (ssk,inet_csk (sk)->icsk_syn_retries); }void mptcp_sockopt_sync_locked (struct mptcp_sock *msk,struct sock *ssk)
Signature · 1 line
2.53.0
Geliang Tang
reviewerPATCH 6/6 selftests: mptcp: sockopt: cover LINGER REUSEADDR PRIORITY NODELAY SYNCNT
In reply to Geliang TangFrom: Geliang Tang <[email protected]>
Extend mptcp_sockopt.c with tests for SOL_SOCKET (SO_REUSEADDR, SO_LINGER, SO_PRIORITY) and SOL_TCP (TCP_NODELAY, TCP_SYNCNT), each exercising both setsockopt() and getsockopt() on a connected MPTCP socket.
SO_REUSEADDR on the msk reports 1 (SK_CAN_REUSE) once any non-zero value is set; this captures the documented behaviour where the master socket reflects the subflow's reuse state.
TCP_SYNCNT coverage pairs with the kernel side that propagates icsk_syn_retries to the subflow.
--- .../selftests/net/mptcp/mptcp_sockopt.c | 116 ++++++++++++++++++ 1 file changed, 116 insertions(+)
tools/testing/selftests/net/mptcp/mptcp_sockopt.c +116 -0
@@ -35,6 +35,9 @@ static int pf = AF_INET;#ifndef SOL_MPTCP #define SOL_MPTCP 284#endif +#ifndef SOL_TCP +#define SOL_TCP 6 +#endif #ifndef MPTCP_INFO struct mptcp_info {
@@ -229,6 +232,30 @@ static int sock_listen_mptcp(const char * const listenaddr,return sock; } +static void test_so_reuseaddr_sockopt (int fd) +{ +int val_in =1 , val_out, expected; +socklen_t s; +int r; + + r =setsockopt (fd,SOL_SOCKET ,SO_REUSEADDR , &val_in,sizeof (val_in)); +if (r !=0 ) +die_perror ("setsockopt SO_REUSEADDR" ); + + s =sizeof (val_out); + r =getsockopt (fd,SOL_SOCKET ,SO_REUSEADDR , &val_out, &s); +if (r !=0 ) +die_perror ("getsockopt SO_REUSEADDR" ); + +/* SO_REUSEADDR on MPTCP msk returns 1 (SK_CAN_REUSE) when set, +* regardless of the input value. The actual reuse semantics are +* applied to the nmpc subflow. +*/ + expected = val_in ?1 :0 ; +if (val_out != expected) +xerror ("SO_REUSEADDR %d != %d\n" , expected, val_out); +} +static int sock_connect_mptcp (const char *const remoteaddr,const char *const port,int proto) {
@@ -247,6 +274,8 @@ static int sock_connect_mptcp(const char * const remoteaddr,if (sock <0 )continue ; +test_so_reuseaddr_sockopt (sock); +if (connect (sock, a->ai_addr, a->ai_addrlen) ==0 )break ;/* success */
@@ -769,6 +798,89 @@ static void test_ip_tos_sockopt(int fd)xerror ("expect socklen_t == -1" ); } +static void test_so_linger_sockopt (int fd) +{ +struct linger ling_in, ling_out; +socklen_t s; +int r; + + ling_in.l_onoff =1 ; + ling_in.l_linger =30 ; + r =setsockopt (fd,SOL_SOCKET ,SO_LINGER , &ling_in,sizeof (ling_in)); +if (r !=0 ) +die_perror ("setsockopt SO_LINGER" ); + + s =sizeof (ling_out); +memset (&ling_out,0 ,sizeof (ling_out)); + r =getsockopt (fd,SOL_SOCKET ,SO_LINGER , &ling_out, &s); +if (r !=0 ) +die_perror ("getsockopt SO_LINGER" ); + +if (ling_in.l_onoff != ling_out.l_onoff || + ling_in.l_linger != ling_out.l_linger) +xerror ("SO_LINGER %d/%d != %d/%d\n" , + ling_in.l_onoff, ling_in.l_linger, + ling_out.l_onoff, ling_out.l_linger); +} + +static void test_so_priority_sockopt (int fd) +{ +int prio_in, prio_out; +socklen_t s; +int r; + + prio_in =5 ; + r =setsockopt (fd,SOL_SOCKET ,SO_PRIORITY , &prio_in,sizeof (prio_in)); +if (r !=0 ) +die_perror ("setsockopt SO_PRIORITY" ); + + s =sizeof (prio_out); + r =getsockopt (fd,SOL_SOCKET ,SO_PRIORITY , &prio_out, &s); +if (r !=0 ) +die_perror ("getsockopt SO_PRIORITY" ); + +if (prio_in != prio_out) +xerror ("SO_PRIORITY %d != %d\n" , prio_in, prio_out); +} + +static void test_tcp_nodelay_sockopt (int fd) +{ +int val_in =1 , val_out; +socklen_t s; +int r; + + r =setsockopt (fd,SOL_TCP ,TCP_NODELAY , &val_in,sizeof (val_in)); +if (r !=0 ) +die_perror ("setsockopt TCP_NODELAY" ); + + s =sizeof (val_out); + r =getsockopt (fd,SOL_TCP ,TCP_NODELAY , &val_out, &s); +if (r !=0 ) +die_perror ("getsockopt TCP_NODELAY" ); + +if (val_in != val_out) +xerror ("TCP_NODELAY %d != %d\n" , val_in, val_out); +} + +static void test_tcp_syncnt_sockopt (int fd) +{ +int val_in =3 , val_out; +socklen_t s; +int r; + + r =setsockopt (fd,SOL_TCP ,TCP_SYNCNT , &val_in,sizeof (val_in)); +if (r !=0 ) +die_perror ("setsockopt TCP_SYNCNT" ); + + s =sizeof (val_out); + r =getsockopt (fd,SOL_TCP ,TCP_SYNCNT , &val_out, &s); +if (r !=0 ) +die_perror ("getsockopt TCP_SYNCNT" ); + +if (val_in != val_out) +xerror ("TCP_SYNCNT %d != %d\n" , val_in, val_out); +} +static int client (int pipefd) {int fd =-1 ;
@@ -787,6 +899,10 @@ static int client(int pipefd) }test_ip_tos_sockopt (fd); +test_so_linger_sockopt (fd); +test_so_priority_sockopt (fd); +test_tcp_nodelay_sockopt (fd); +test_tcp_syncnt_sockopt (fd);connect_one_server (fd, pipefd);
Signature · 1 line
2.53.0
Hi Geliang,
Thank you for your modifications, that's great!
Our CI did some validations and here is its report:
- KVM Validation: normal (except selftest_mptcp_join): Unstable: 1 failed test(s): selftest_simult_flows ⚠️
- KVM Validation: normal (only selftest_mptcp_join): Success! ✅
- KVM Validation: debug (except selftest_mptcp_join): Success! ✅
- KVM Validation: debug (only selftest_mptcp_join): Success! ✅
- KVM Validation: btf-normal (only bpftest_all): Success! ✅
- KVM Validation: btf-debug (only bpftest_all): Success! ✅
- Task: https://github.com/multipath-tcp/mptcp_net-next/actions/runs/30160012728
Initiator: Patchew Applier
Commits: https://github.com/multipath-tcp/mptcp_net-next/commits/bf15e67a2e42
Patchwork: https://patchwork.kernel.org/project/mptcp/list/?series=1134397
If there are some issues, you can reproduce them using the same environment as the one used by the CI thanks to a docker image, e.g.:
$ cd [kernel source code]
$ docker run -v "${PWD}:${PWD}:rw" -w "${PWD}" --privileged --rm -it \
--pull always mptcp/mptcp-upstream-virtme-docker:latest \
auto-normalFor more details:
https://github.com/multipath-tcp/mptcp-upstream-virtme-docker
Please note that despite all the efforts that have been already done to have a stable tests suite when executed on a public CI like here, it is possible some reported issues are not due to your modifications. Still, do not hesitate to help us improve that ;-)
Cheers,
MPTCP GH Action bot
Bot operated by Matthieu Baerts (NGI0 Core)