Skip to contentSkip to conversation

mptcp: pm: fix stale ADD_ADDR entry on id 0 removal

PATCH Superseded Kalpan Jani 5 participants 9 messages Jun 22 – Jun 25

Subsystems (3)

v4 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

v4 cover mptcp: pm: fix stale ADD_ADDR entry on id 0 removal

Hi Tao, Matt,

Thank you Tao for identifying the port mismatch in v3 and providing a deterministic reproducer — both are incorporated in this series.

The problem with v3: mptcp_remove_anno_list_by_saddr() compares addresses with use_port=true. Signal endpoints added without an explicit port are stored in the anno_list with port 0, but msk_local carries the connection's actual local port. So the v3 lookup always missed the stale entry, making the patch a no-op. Confirmed with a temporary printk showing msk_local.port=5000 and announced=0 on v3, and announced=1 after clearing the port.

This series:

Patch 1 fixes mptcp_nl_remove_id_zero_address() to tear down the pending ADD_ADDR symmetrically with the non-zero id path: copy msk_local, clear the port, call mptcp_remove_anno_list_by_saddr(), and decrement add_addr_signaled if announced.

Patch 2 adds the regression test based on Tao's reproducer. Verified:
- unpatched kernel: WARNING: net/mptcp/pm fires, test reports FAIL
- patched kernel: no WARNING, test reports PASS

Tested with mptcp/mptcp-upstream-virtme-docker auto-normal. All existing selftests pass (packetdrill_add_addr failure is pre-existing and unrelated to this series).

Kalpan Jani (2):
mptcp: pm: drop pending ADD_ADDR when removing id 0 endpoint
selftests: mptcp: add regression test for stale ADD_ADDR on id 0
removal

 net/mptcp/pm_kernel.c                         |  8 ++
 .../net/mptcp/mptcp_id0_stale_anno.sh         | 85 +++++++++++++++++++
 2 files changed, 93 insertions(+)
 create mode 100755 tools/testing/selftests/net/mptcp/mptcp_id0_stale_anno.sh
Signature · 1 line
2.43.0
#1 lore raw

PATCH v4 1/2 mptcp: pm: drop pending ADD_ADDR when removing id 0 endpoint

syzkaller hit the WARN_ON_ONCE() in mptcp_pm_alloc_anno_list() with the in-kernel path manager.

When a signal endpoint is removed, the pending ADD_ADDR has to be
cancelled: its retransmit timer stopped and the anno_list entry unlinked and freed. For a non-zero id endpoint this is done via mptcp_nl_remove_subflow_and_signal_addr() -> mptcp_pm_remove_anno_addr() -> mptcp_remove_anno_list_by_saddr().

The id 0 removal path, mptcp_nl_remove_id_zero_address(), does not do
this: it only queues a RM_ADDR and marks the id available again, but leaves any pending anno_list entry and its armed retransmit timer alive.

So when the id 0 endpoint is removed and re-added while its previously sent ADD_ADDR is still awaiting the echo, the stale entry survives. The kernel PM reselects id 0, reaches mptcp_pm_alloc_anno_list() a second time, finds the stale entry and hits the WARN.

Make the id 0 removal path symmetric with the non-zero one: drop the pending ADD_ADDR before queuing the RM_ADDR, and decrement add_addr_signaled if the address had been announced. This closes the race at its source, so the WARN_ON_ONCE() stays a valid assertion.

Signal endpoints added without an explicit port are stored in the anno_list with port 0. mptcp_remove_anno_list_by_saddr() compares addresses with use_port=true, and msk_local carries the connection's actual local port, so passing msk_local directly misses the entry. Clear the port before the lookup to match the stored key.

Fixes 740d798e8767 ("mptcp: remove id 0 address")
Reported-by [email protected]
---
 net/mptcp/pm_kernel.c | 8 ++++++++
 1 file changed, 8 insertions(+)
net/mptcp/pm_kernel.c +8 -0
index fc818b63752e..4ab1a339e3e9 100644
@@ -1126,6 +1126,8 @@ static int mptcp_nl_remove_id_zero_address(struct net *net,
 	while ((msk = mptcp_token_iter_next(net, &s_slot, &s_num)) != NULL) {
 		struct sock *sk = (struct sock *)msk;
 		struct mptcp_addr_info msk_local;
+		struct mptcp_addr_info anno_addr;
+		bool announced;
 
 		if (list_empty(&msk->conn_list) || mptcp_pm_is_userspace(msk))
 			goto next;
@@ -1135,7 +1137,13 @@ static int mptcp_nl_remove_id_zero_address(struct net *net,
 			goto next;
 
 		lock_sock(sk);
+		/* Drop a possibly pending ADD_ADDR for this address. */
+		anno_addr = msk_local;
+		anno_addr.port = 0;
+		announced = mptcp_remove_anno_list_by_saddr(msk, &anno_addr);
 		spin_lock_bh(&msk->pm.lock);
+		if (announced)
+			msk->pm.add_addr_signaled--;
 		mptcp_pm_remove_addr(msk, &list);
 		mptcp_pm_rm_subflow(msk, &list);
 		__mark_subflow_endp_available(msk, 0);
Signature · 1 line
2.43.0
#2 lore raw

PATCH v4 2/2 selftests: mptcp: add regression test for stale ADD_ADDR on id 0 removal

In reply to Kalpan Jani

Add a kselftest that reproduces the bug fixed in the previous patch:
removing the id 0 endpoint while a pending ADD_ADDR echo is outstanding left a stale anno_list entry alive, and a subsequent PM reselection re-announced that address and tripped the WARN_ON_ONCE() in mptcp_pm_alloc_anno_list().

The sequence is deterministic:
1. Establish a fully-established MPTCP connection kept alive by a bidirectional /dev/zero stream.
2. Signal 10.0.2.1 so the peer joins: the second subflow keeps the connection alive across the id 0 removal.
3. Signal the MPC address 10.0.1.1: an anno_list entry is created.
4. Delete id 0 (10.0.1.1): on an unfixed kernel the stale entry survives.
5. Signal 10.0.3.1 to force a PM reselection: on an unfixed kernel this hits the stale entry and fires the WARN.

The test counts WARNING: net/mptcp/pm lines in dmesg before and after the sequence and fails if new ones appear.

---
 .../net/mptcp/mptcp_id0_stale_anno.sh         | 85 +++++++++++++++++++
 1 file changed, 85 insertions(+)
 create mode 100755 tools/testing/selftests/net/mptcp/mptcp_id0_stale_anno.sh
tools/testing/selftests/net/mptcp/mptcp_id0_stale_anno.sh new file +85 -0
new file mode 100755
index 000000000000..96c0d2256064
@@ -0,0 +1,85 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Regression test for stale ADD_ADDR anno_list entry on id 0 removal
+
+. "$(dirname "${0}")/mptcp_lib.sh"
+
+ret=0
+ns1=""
+ns2=""
+err=$(mktemp)
+timeout_poll=30
+port=50000
+
+cleanup()
+{
+	rm -f "${err}"
+	mptcp_lib_ns_exit "${ns1}" "${ns2}"
+}
+
+mptcp_lib_check_mptcp
+mptcp_lib_check_tools ip
+
+trap cleanup EXIT
+
+mptcp_lib_ns_init ns1 ns2
+
+ip link add ns1eth1 netns "${ns1}" type veth peer name ns2eth1 netns "${ns2}"
+ip -net "${ns1}" link set lo up
+ip -net "${ns2}" link set lo up
+ip -net "${ns1}" link set ns1eth1 up
+ip -net "${ns2}" link set ns2eth1 up
+ip -net "${ns1}" addr add 10.0.1.1/24 dev ns1eth1
+ip -net "${ns1}" addr add 10.0.2.1/24 dev ns1eth1
+ip -net "${ns1}" addr add 10.0.3.1/24 dev ns1eth1
+ip -net "${ns2}" addr add 10.0.1.2/24 dev ns2eth1
+ip -net "${ns2}" addr add 10.0.2.2/24 dev ns2eth1
+ip -net "${ns2}" addr add 10.0.3.2/24 dev ns2eth1
+
+mptcp_lib_pm_nl_set_limits "${ns1}" 8 8
+mptcp_lib_pm_nl_set_limits "${ns2}" 8 8
+
+ip netns exec "${ns1}" ./mptcp_connect -t "${timeout_poll}" -l -p "${port}" \
+	0.0.0.0 < /dev/zero > /dev/null 2>"${err}" &
+spid=$!
+mptcp_lib_wait_local_port_listen "${ns1}" "${port}"
+ip netns exec "${ns2}" ./mptcp_connect -t "${timeout_poll}" -p "${port}" \
+	10.0.1.1 < /dev/zero > /dev/null 2>"${err}" &
+cpid=$!
+
+sleep 2
+
+warn_before=$(dmesg | grep -c "WARNING: net/mptcp/pm")
+
+# 1. signal 10.0.2.1: peer joins, second subflow keeps connection alive
+mptcp_lib_pm_nl_add_endpoint "${ns1}" 10.0.2.1 flags signal
+sleep 2
+
+# 2. signal MPC address 10.0.1.1: anno_list entry created for id 0
+mptcp_lib_pm_nl_add_endpoint "${ns1}" 10.0.1.1 flags signal
+sleep 1
+
+# 3. remove id 0: stale entry survives on unfixed kernels
+mptcp_lib_pm_nl_del_endpoint "${ns1}" 0 10.0.1.1
+sleep 1
+
+# 4. force PM reselection: hits stale entry on unfixed kernels
+mptcp_lib_pm_nl_add_endpoint "${ns1}" 10.0.3.1 flags signal
+sleep 2
+
+warn_after=$(dmesg | grep -c "WARNING: net/mptcp/pm")
+
+kill "${cpid}" "${spid}" 2>/dev/null
+wait "${cpid}" 2>/dev/null
+wait "${spid}" 2>/dev/null
+
+if [ "${warn_after}" -gt "${warn_before}" ]; then
+	mptcp_lib_result_fail "stale ADD_ADDR warning triggered on id 0 removal"
+	ret=1
+else
+	mptcp_lib_result_pass "no stale ADD_ADDR warning on id 0 removal"
+fi
+
+mptcp_lib_result_print_all_tap
+
+exit ${ret}
Signature · 1 line
2.43.0
maintainer
#3 lore raw

Re: [PATCH net v4 2/2] selftests: mptcp: add regression test for stale ADD_ADDR on id 0 removal

Hi Kalpan,

Quoted 6 lines · Kalpan Jani

Add a kselftest that reproduces the bug fixed in the previous patch:
removing the id 0 endpoint while a pending ADD_ADDR echo is outstanding left a stale anno_list entry alive, and a subsequent PM reselection re-announced that address and tripped the WARN_ON_ONCE() in mptcp_pm_alloc_anno_list().

Quoted 10 lines

The sequence is deterministic:
1. Establish a fully-established MPTCP connection kept alive by a bidirectional /dev/zero stream.
2. Signal 10.0.2.1 so the peer joins: the second subflow keeps the connection alive across the id 0 removal.
3. Signal the MPC address 10.0.1.1: an anno_list entry is created.
4. Delete id 0 (10.0.1.1): on an unfixed kernel the stale entry survives.
5. Signal 10.0.3.1 to force a PM reselection: on an unfixed kernel this hits the stale entry and fires the WARN.

Thank you for having added a reproducer, but in general:

- We try to avoid new "main" tests for maintainability reasons.

- If possible, please prefer packetdrill [1] for the reproducers: that's easier to read (shorter, focussed on the essential to reproducer an issue), faster to execute, and easier to maintain. Please try to use it, and do not hesitate to ask for help if needed. Mastering packetdrill is really helpful in this area of the kernel.

- If not, please check if adding a subtest to an existing selftest is not possible, e.g. mptcp_join.sh

- We try to avoid using "sleep": it is either waiting for a too long time, or not enough depending on the kernel config and test environment. Prefer waiting for an action instead, e.g. checking if a connection has been established, the number of subflows, etc.

[1] https://github.com/multipath-tcp/packetdrill/

Quoted 2 lines

The test counts WARNING: net/mptcp/pm lines in dmesg before and after the sequence and fails if new ones appear.

That's not needed, the executor (i.e. CIs) will monitor that.

A few more requests about this series, please:

- base your patches on top of the export branch (or "for-review" or "t/upstream" branches), but not the "export-net" one: the CI will always try to apply them on top of the "export" branch, and that's easier for me when I have to resolve conflicts. (Note that the "export" branch contains both "net-next" and "net", plus all patches applied in our tree.)

- ideally, avoid renaming the patches, or at least the cover-letter, between versions, so it is easier for reviewers and bots to follow. Here for example, you switched to one patch to multiple ones: it would be easier if the cover-letter has same title as your previous single patch. If renaming is required, please add a note in the patch/cover-letter. Note that b4 automatically adds a link to the previous version, that's also helpful.

- use "mptcp-ne(x)t" prefixes instead of "ne(x)t" when you send patches only to the MPTCP ML (which is what we prefer).

Cheers,
Matt

Signature · 1 line
Sponsored by the NGI0 Core fund.
#4 lore raw

Re: [PATCH net v4 2/2] selftests: mptcp: add regression test for stale ADD_ADDR on id 0 removal

In reply to Kalpan Jani
kernel test robot noticed the following build errors:

Hi Kalpan,

kernel test robot noticed the following build errors:

[auto build test ERROR on net/main]

url:    https://github.com/intel-lab-lkp/linux/commits/Kalpan-Jani/mptcp-pm-drop-pending-ADD_ADDR-when-removing-id-0-endpoint/20260623-033836
base:   net/main
patch link:    https://lore.kernel.org/r/20260622092838.1267134-3-kalpan.jani%40mpiricsoftware.com
patch subject: [PATCH net v4 2/2] selftests: mptcp: add regression test for stale ADD_ADDR on id 0 removal
config: x86_64-rhel-9.4 (https://download.01.org/0day-ci/archive/20260623/[email protected]/config)
compiler: gcc-14 (Debian 14.2.0-19) 14.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260623/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

net/mptcp/pm_kernel.c: In function 'mptcp_nl_remove_id_zero_address':

>> net/mptcp/pm_kernel.c:1154:29: error: implicit declaration of function 'mptcp_remove_anno_list_by_saddr' [-Wimplicit-function-declaration]
    1154 |                 announced = mptcp_remove_anno_list_by_saddr(msk, &anno_addr);
         |                             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

vim +/mptcp_remove_anno_list_by_saddr +1154 net/mptcp/pm_kernel.c

1127
1128 static int mptcp_nl_remove_id_zero_address(struct net *net,
1129 struct mptcp_addr_info *addr)
1130 {
1131 struct mptcp_rm_list list = { .nr = 0 };
1132 long s_slot = 0, s_num = 0;
1133 struct mptcp_sock *msk;
1134
1135 list.ids[list.nr++] = 0;
1136
1137 while ((msk = mptcp_token_iter_next(net, &s_slot, &s_num)) != NULL) {
1138 struct sock *sk = (struct sock *)msk;
1139 struct mptcp_addr_info msk_local;
1140 struct mptcp_addr_info anno_addr;
1141 bool announced;
1142
1143 if (list_empty(&msk->conn_list) || mptcp_pm_is_userspace(msk))
1144 goto next;
1145
1146 mptcp_local_address((struct sock_common *)msk, &msk_local);
1147 if (!mptcp_addresses_equal(&msk_local, addr, addr->port))
1148 goto next;
1149
1150 lock_sock(sk);
1151 /* Drop a possibly pending ADD_ADDR for this address. */
1152 anno_addr = msk_local;
1153 anno_addr.port = 0;

> 1154			announced = mptcp_remove_anno_list_by_saddr(msk, &anno_addr);

1155 spin_lock_bh(&msk->pm.lock);
1156 if (announced)
1157 msk->pm.add_addr_signaled--;
1158 mptcp_pm_remove_addr(msk, &list);
1159 mptcp_pm_rm_subflow(msk, &list);
1160 __mark_subflow_endp_available(msk, 0);
1161 spin_unlock_bh(&msk->pm.lock);
1162 release_sock(sk);
1163
1164 next:
1165 sock_put(sk);
1166 cond_resched();
1167 }
1168
1169 return 0;
1170 }
1171

--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

#5 lore raw

Re: [PATCH net v4 2/2] selftests: mptcp: add regression test for stale ADD_ADDR on id 0 removal

In reply to Kalpan Jani
kernel test robot noticed the following build errors:

Hi Kalpan,

kernel test robot noticed the following build errors:

[auto build test ERROR on net/main]

url:    https://github.com/intel-lab-lkp/linux/commits/Kalpan-Jani/mptcp-pm-drop-pending-ADD_ADDR-when-removing-id-0-endpoint/20260623-033836
base:   net/main
patch link:    https://lore.kernel.org/r/20260622092838.1267134-3-kalpan.jani%40mpiricsoftware.com
patch subject: [PATCH net v4 2/2] selftests: mptcp: add regression test for stale ADD_ADDR on id 0 removal
config: x86_64-kexec (https://download.01.org/0day-ci/archive/20260623/[email protected]/config)
compiler: clang version 22.1.8 (https://github.com/llvm/llvm-project ca7933e47d3a3451d81e72ac174dcb5aa28b59d1)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260623/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

>> net/mptcp/pm_kernel.c:1154:15: error: call to undeclared function 'mptcp_remove_anno_list_by_saddr'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
    1154 |                 announced = mptcp_remove_anno_list_by_saddr(msk, &anno_addr);
         |                             ^

1 error generated.

vim +/mptcp_remove_anno_list_by_saddr +1154 net/mptcp/pm_kernel.c

1127
1128 static int mptcp_nl_remove_id_zero_address(struct net *net,
1129 struct mptcp_addr_info *addr)
1130 {
1131 struct mptcp_rm_list list = { .nr = 0 };
1132 long s_slot = 0, s_num = 0;
1133 struct mptcp_sock *msk;
1134
1135 list.ids[list.nr++] = 0;
1136
1137 while ((msk = mptcp_token_iter_next(net, &s_slot, &s_num)) != NULL) {
1138 struct sock *sk = (struct sock *)msk;
1139 struct mptcp_addr_info msk_local;
1140 struct mptcp_addr_info anno_addr;
1141 bool announced;
1142
1143 if (list_empty(&msk->conn_list) || mptcp_pm_is_userspace(msk))
1144 goto next;
1145
1146 mptcp_local_address((struct sock_common *)msk, &msk_local);
1147 if (!mptcp_addresses_equal(&msk_local, addr, addr->port))
1148 goto next;
1149
1150 lock_sock(sk);
1151 /* Drop a possibly pending ADD_ADDR for this address. */
1152 anno_addr = msk_local;
1153 anno_addr.port = 0;

> 1154			announced = mptcp_remove_anno_list_by_saddr(msk, &anno_addr);

1155 spin_lock_bh(&msk->pm.lock);
1156 if (announced)
1157 msk->pm.add_addr_signaled--;
1158 mptcp_pm_remove_addr(msk, &list);
1159 mptcp_pm_rm_subflow(msk, &list);
1160 __mark_subflow_endp_available(msk, 0);
1161 spin_unlock_bh(&msk->pm.lock);
1162 release_sock(sk);
1163
1164 next:
1165 sock_put(sk);
1166 cond_resched();
1167 }
1168
1169 return 0;
1170 }
1171

--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

#6 lore raw
In reply to Kalpan Jani

Hi Kalpan,

Thank you for your modifications, that's great!

But sadly, our CI spotted some issues with it when trying to build it.

You can find more details there:

https://github.com/multipath-tcp/mptcp_net-next/actions/runs/27944189360

Status: failure
Initiator: Patchew Applier
Commits: https://github.com/multipath-tcp/mptcp_net-next/commits/279ca4d41984
Patchwork: https://patchwork.kernel.org/project/mptcp/list/?series=1114633

Feel free to reply to this email if you cannot access logs, if you need some support to fix the error, if this doesn't seem to be caused by your modifications or if the error is a false positive one.

Cheers,
MPTCP GH Action bot
Bot operated by Matthieu Baerts (NGI0 Core)

#7 lore raw
In reply to Kalpan Jani

Hi Kalpan,

Thank you for your modifications, that's great!

Our CI did some validations and here is its report:

- KVM Validation: normal (except selftest_mptcp_join): Script error! ❓
- KVM Validation: normal (only selftest_mptcp_join): Script error! ❓
- KVM Validation: debug (except selftest_mptcp_join): Script error! ❓
- KVM Validation: debug (only selftest_mptcp_join): Script error! ❓
- KVM Validation: btf-normal (only bpftest_all): Script error! ❓
- KVM Validation: btf-debug (only bpftest_all): Script error! ❓
- Task: https://github.com/multipath-tcp/mptcp_net-next/actions/runs/27944189431

Initiator: Patchew Applier
Commits: https://github.com/multipath-tcp/mptcp_net-next/commits/279ca4d41984
Patchwork: https://patchwork.kernel.org/project/mptcp/list/?series=1114633

If there are some issues, you can reproduce them using the same environment as the one used by the CI thanks to a docker image, e.g.:

    $ cd [kernel source code]
    $ docker run -v "${PWD}:${PWD}:rw" -w "${PWD}" --privileged --rm -it \
        --pull always mptcp/mptcp-upstream-virtme-docker:latest \
        auto-normal

For more details:

https://github.com/multipath-tcp/mptcp-upstream-virtme-docker

Please note that despite all the efforts that have been already done to have a stable tests suite when executed on a public CI like here, it is possible some reported issues are not due to your modifications. Still, do not hesitate to help us improve that ;-)

Cheers,
MPTCP GH Action bot
Bot operated by Matthieu Baerts (NGI0 Core)

#8 lore raw

PATCH v4 1/2 mptcp: pm: drop pending ADD_ADDR when removing id 0 endpoint

In reply to Kalpan Jani

From: Tao Cui <[email protected]>

Hi Kalpan,

The CI build failure is just a leftover rename: patch 1 calls mptcp_remove_anno_list_by_saddr(), which was renamed to mptcp_pm_announced_remove() in 7d4dacc8ccca and no longer exists in the current tree. One-liner:

  -	announced = mptcp_remove_anno_list_by_saddr(msk, &anno_addr);
  +	announced = mptcp_pm_announced_remove(msk, &anno_addr);

Cheers,
Tao