mptcp: fix another deadlock issue
Subsystems (3)
Paolo Abeni
maintainerv3 cover mptcp: fix another deadlock issue
Eric reported and diagnosed another possible deadlock problem with the MPTCP diag code. The first patch address the issue, and the second adds self-test coverage for the relevant codepath, to get lockdep help to catch future similar issues.
v2 -> v3:
- cope with very slow env
v1 -> v2:
- fix (avoid) feature detection in patch 2/2 (Matttbe)
Paolo Abeni (2):
mptcp: fix possible deadlock in subflow diag
selftests: mptcp: explicitly trigger the listener diag code-path
net/mptcp/diag.c | 3 +++ tools/testing/selftests/net/mptcp/diag.sh | 30 ++++++++++++++++++++++- 2 files changed, 32 insertions(+), 1 deletion(-)
Signature · 1 line
2.43.0
Paolo Abeni
maintainerPATCH v3 1/2 mptcp: fix possible deadlock in subflow diag
Syzbot and Eric reported a lockdep splat in the subflow diag:
WARNING: possible circular locking dependency detected 6.8.0-rc4-syzkaller-00212-g40b9385dd8e6 #0 Not tainted
syz-executor.2/24141 is trying to acquire lock:
ffff888045870130 (k-sk_lock-AF_INET6){+.+.}-{0:0}, at:
tcp_diag_put_ulp net/ipv4/tcp_diag.c:100 [inline]
ffff888045870130 (k-sk_lock-AF_INET6){+.+.}-{0:0}, at:
tcp_diag_get_aux+0x738/0x830 net/ipv4/tcp_diag.c:137 but task is already holding lock:
ffffc9000135e488 (&h->lhash2[i].lock){+.+.}-{2:2}, at: spin_lock include/linux/spinlock.h:351 [inline] ffffc9000135e488 (&h->lhash2[i].lock){+.+.}-{2:2}, at:
inet_diag_dump_icsk+0x39f/0x1f80 net/ipv4/inet_diag.c:1038
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #1 (&h->lhash2[i].lock){+.+.}-{2:2}:
lock_acquire+0x1e3/0x530 kernel/locking/lockdep.c:5754 __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline] _raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154 spin_lock include/linux/spinlock.h:351 [inline] __inet_hash+0x335/0xbe0 net/ipv4/inet_hashtables.c:743 inet_csk_listen_start+0x23a/0x320 net/ipv4/inet_connection_sock.c:1261 __inet_listen_sk+0x2a2/0x770 net/ipv4/af_inet.c:217 inet_listen+0xa3/0x110 net/ipv4/af_inet.c:239 rds_tcp_listen_init+0x3fd/0x5a0 net/rds/tcp_listen.c:316 rds_tcp_init_net+0x141/0x320 net/rds/tcp.c:577 ops_init+0x352/0x610 net/core/net_namespace.c:136 __register_pernet_operations net/core/net_namespace.c:1214 [inline] register_pernet_operations+0x2cb/0x660 net/core/net_namespace.c:1283 register_pernet_device+0x33/0x80 net/core/net_namespace.c:1370 rds_tcp_init+0x62/0xd0 net/rds/tcp.c:735 do_one_initcall+0x238/0x830 init/main.c:1236 do_initcall_level+0x157/0x210 init/main.c:1298 do_initcalls+0x3f/0x80 init/main.c:1314 kernel_init_freeable+0x42f/0x5d0 init/main.c:1551 kernel_init+0x1d/0x2a0 init/main.c:1441 ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147 ret_from_fork_asm+0x1b/0x30 arch/x86/entry/entry_64.S:242
-> #0 (k-sk_lock-AF_INET6){+.+.}-{0:0}:
check_prev_add kernel/locking/lockdep.c:3134 [inline] check_prevs_add kernel/locking/lockdep.c:3253 [inline] validate_chain+0x18ca/0x58e0 kernel/locking/lockdep.c:3869 __lock_acquire+0x1345/0x1fd0 kernel/locking/lockdep.c:5137 lock_acquire+0x1e3/0x530 kernel/locking/lockdep.c:5754 lock_sock_fast include/net/sock.h:1723 [inline] subflow_get_info+0x166/0xd20 net/mptcp/diag.c:28 tcp_diag_put_ulp net/ipv4/tcp_diag.c:100 [inline] tcp_diag_get_aux+0x738/0x830 net/ipv4/tcp_diag.c:137 inet_sk_diag_fill+0x10ed/0x1e00 net/ipv4/inet_diag.c:345 inet_diag_dump_icsk+0x55b/0x1f80 net/ipv4/inet_diag.c:1061 __inet_diag_dump+0x211/0x3a0 net/ipv4/inet_diag.c:1263 inet_diag_dump_compat+0x1c1/0x2d0 net/ipv4/inet_diag.c:1371 netlink_dump+0x59b/0xc80 net/netlink/af_netlink.c:2264 __netlink_dump_start+0x5df/0x790 net/netlink/af_netlink.c:2370 netlink_dump_start include/linux/netlink.h:338 [inline] inet_diag_rcv_msg_compat+0x209/0x4c0 net/ipv4/inet_diag.c:1405 sock_diag_rcv_msg+0xe7/0x410 netlink_rcv_skb+0x1e3/0x430 net/netlink/af_netlink.c:2543 sock_diag_rcv+0x2a/0x40 net/core/sock_diag.c:280 netlink_unicast_kernel net/netlink/af_netlink.c:1341 [inline] netlink_unicast+0x7ea/0x980 net/netlink/af_netlink.c:1367 netlink_sendmsg+0xa3b/0xd70 net/netlink/af_netlink.c:1908 sock_sendmsg_nosec net/socket.c:730 [inline] __sock_sendmsg+0x221/0x270 net/socket.c:745 ____sys_sendmsg+0x525/0x7d0 net/socket.c:2584 ___sys_sendmsg net/socket.c:2638 [inline] __sys_sendmsg+0x2b0/0x3a0 net/socket.c:2667 do_syscall_64+0xf9/0x240 entry_SYSCALL_64_after_hwframe+0x6f/0x77
As noted by Eric we can break the lock dependency chain avoid dumping any extended info for the mptcp subflow listener:
nothing actually useful is presented there.
--- net/mptcp/diag.c | 3 +++ 1 file changed, 3 insertions(+)
net/mptcp/diag.c +3 -0
@@ -21,6 +21,9 @@ static int subflow_get_info(struct sock *sk, struct sk_buff *skb)bool slow;int err; +if (inet_sk_state_load (sk) ==TCP_LISTEN ) +return 0 ; + start =nla_nest_start_noflag (skb,INET_ULP_INFO_MPTCP );if (!start)return -EMSGSIZE ;
Signature · 1 line
2.43.0
Paolo Abeni
maintainerPATCH v3 2/2 selftests: mptcp: explicitly trigger the listener diag code-path
In reply to Paolo AbeniThe mptcp diag interface already experienced a few locking bugs that lockdep and appropriate coverage have detected in advance.
Let's add a test-case triggering the relevant code path, to prevent similar issues in the future.
Be careful to cope with very slow environments.
Note that we don't need an explicit timeout on the mptcp_connect subprocess to cope with eventual bug/hang-up as the final cleanup terminating the child processes will take care of that.
---
v2 -> v3:
- longer wait timeout
- run all subprocesses in background
- wait for tcp ports after starting all listeners all the above should produce a lot of resilience wrt slow env.
--- tools/testing/selftests/net/mptcp/diag.sh | 30 ++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-)
tools/testing/selftests/net/mptcp/diag.sh +29 -1
@@ -20,7 +20,7 @@ flush_pids() ip netns pids "${ns}" | xargs --no-run-if-empty kill -SIGUSR1 &>/dev/null - for _ in $(seq 10); do + for _ in $(seq $((timeout_poll * 10))); do [ -z "$(ip netns pids "${ns}")" ] && break sleep 0.1 done
@@ -81,6 +81,15 @@ chk_msk_nr() __chk_msk_nr "grep -c token:" "$@" } +chk_listener_nr() +{ + local expected=$1 + local msg="$2" + + __chk_nr "ss -inmlHMON $ns | wc -l" "$expected" "$msg - mptcp" 0 + __chk_nr "ss -inmlHtON $ns | wc -l" "$expected" "$msg - subflows" +} + wait_msk_nr() { local condition="grep -c token:"
@@ -279,5 +288,24 @@ flush_pids chk_msk_inuse 0 "many->0" chk_msk_cestab 0 "many->0" +chk_listener_nr 0 "no listener sockets" +NR_SERVERS=100 +for I in $(seq 1 $NR_SERVERS); do + ip netns exec $ns ./mptcp_connect -p $((I + 20001)) \ + -t ${timeout_poll} -l 0.0.0.0 2>&1 >/dev/null & +done + +for I in $(seq 1 $NR_SERVERS); do + mptcp_lib_wait_local_port_listen $ns $((I + 20001)) +done + +chk_listener_nr $NR_SERVERS "many listener sockets" + +# gracefull termination +for I in $(seq 1 $NR_SERVERS); do + echo a | ip netns exec $ns ./mptcp_connect -p $((I + 20001)) 127.0.0.1 2>&1 >/dev/null & +done +flush_pids + mptcp_lib_result_print_all_tap exit $ret
Signature · 1 line
2.43.0
Re: selftests: mptcp: explicitly trigger the listener diag code-path: Tests Results
Hi Paolo,
Thank you for your modifications, that's great!
Our CI (GitHub Action) did some validations and here is its report:
- KVM Validation: normal:
- Success! ✅:
- Task: https://github.com/multipath-tcp/mptcp_net-next/actions/runs/7993934084
Initiator: Patchew Applier
Commits: https://github.com/multipath-tcp/mptcp_net-next/commits/282733d05eba
If there are some issues, you can reproduce them using the same environment as the one used by the CI thanks to a docker image, e.g.:
$ cd [kernel source code]
$ docker run -v "${PWD}:${PWD}:rw" -w "${PWD}" --privileged --rm -it \
--pull always mptcp/mptcp-upstream-virtme-docker:latest \
auto-normalFor more details:
https://github.com/multipath-tcp/mptcp-upstream-virtme-docker
Please note that despite all the efforts that have been already done to have a stable tests suite when executed on a public CI like here, it is possible some reported issues are not due to your modifications. Still, do not hesitate to help us improve that ;-)
Cheers,
MPTCP GH Action bot
Bot operated by Matthieu Baerts (NGI0 Core)
Hi Paolo,
Quoted 7 lines · Paolo Abeni
Eric reported and diagnosed another possible deadlock problem with the MPTCP diag code. The first patch address the issue, and the second adds self-test coverage for the relevant codepath, to get lockdep help to catch future similar issues.
v2 -> v3:
- cope with very slow env
Thank you for the new version, and having mentioned why the timeout is not needed for this new case!
Cheers,
Matt
Signature · 1 line
Sponsored by the NGI0 Core fund.
Re: selftests: mptcp: explicitly trigger the listener diag code-path: Tests Results
In reply to Paolo AbeniHi Paolo,
Thank you for your modifications, that's great!
Our CI (GitHub Action) did some validations and here is its report:
- KVM Validation: normal:
- Success! ✅:
- Task: https://github.com/multipath-tcp/mptcp_net-next/actions/runs/8002395034
Initiator: Patchew Applier
Commits: https://github.com/multipath-tcp/mptcp_net-next/commits/4fe5b5cc4615
If there are some issues, you can reproduce them using the same environment as the one used by the CI thanks to a docker image, e.g.:
$ cd [kernel source code]
$ docker run -v "${PWD}:${PWD}:rw" -w "${PWD}" --privileged --rm -it \
--pull always mptcp/mptcp-upstream-virtme-docker:latest \
auto-normalFor more details:
https://github.com/multipath-tcp/mptcp-upstream-virtme-docker
Please note that despite all the efforts that have been already done to have a stable tests suite when executed on a public CI like here, it is possible some reported issues are not due to your modifications. Still, do not hesitate to help us improve that ;-)
Cheers,
MPTCP GH Action bot
Bot operated by Matthieu Baerts (NGI0 Core)
Matthieu Baerts
maintainerHi Paolo,
Quoted 4 lines · Paolo Abeni
Eric reported and diagnosed another possible deadlock problem with the MPTCP diag code. The first patch address the issue, and the second adds self-test coverage for the relevant codepath, to get lockdep help to catch future similar issues.
Thank you for the patches!
Now in our tree (fixes for -net), with 2 small modifications to satisfy CheckPatch [1]:
New patches for t/upstream-net and t/upstream:
- 5ca0931fda75: mptcp: fix possible deadlock in subflow diag
- 560736ee5851: selftests: mptcp: explicitly trigger the listener diag code-path
- Results: b35846da7660..8f414384d341 (export-net)
- Results: df010bcea0cb..a542d2d60003 (export)
[1] https://github.com/multipath-tcp/mptcp_net-next/actions/runs/8002395030
Cheers,
Matt
Signature · 1 line
Sponsored by the NGI0 Core fund.