mptcp: pm: net-next port of the alloc-during-teardown leak fix
Subsystems (2)
cover mptcp: pm: net-next port of the alloc-during-teardown leak fix
This is the net-next counterpart of the net fix "mptcp: pm: fix memory leak from alloc-during-teardown race" (v3):
https://lore.kernel.org/all/[email protected]/
v3 remains the canonical fix for net; it applies as-is on export-net. It does not apply on export, because the userspace local address list free moved into the userspace path manager .release callback in "mptcp: pm:
init and release mptcp_pm_ops". This series ports the fix to net-next:
1/2 squashes the unconditional free into that refactor commit (covering
the disconnect/reuse path).
2/2 is the MPTCP_PM_DESTROYING fence (the alloc-during-teardown race),
rebased onto the renamed mptcp_pm_announced_alloc() / mptcp_pm_destroy().
The net fix was validated with an MPTCP protocol-flow harness (BRF) over multi-hour runs with zero kmemleak reports (reported on the v3 thread). This port is structurally equivalent and is sent so the CI can exercise it on net-next.
Shardul Bankar (2):
Squash to "mptcp: pm: init and release mptcp_pm_ops"
mptcp: pm: fix memory leak from alloc-during-teardown race
net/mptcp/pm.c | 13 +++++++++++++ net/mptcp/pm_userspace.c | 10 ++++------ net/mptcp/protocol.h | 8 ++++++-- 3 files changed, 23 insertions(+), 8 deletions(-)
Signature · 1 line
2.34.1
PATCH 1/2 Squash to "mptcp: pm: init and release mptcp_pm_ops"
The userspace local address list is freed from the userspace path manager .release callback (via mptcp_pm_ops_release()), i.e. only when the socket currently uses the userspace PM. A socket that used the userspace PM can be reused via mptcp_disconnect() and re-selected to a different path manager; an entry appended late would then not be freed by the next teardown running a non-userspace PM, and leaks.
Free msk->pm.userspace_pm_local_addr_list unconditionally in mptcp_pm_destroy() and drop the now-redundant userspace .release. The list is initialised in the generic mptcp_pm_data_init(), so freeing it from the generic teardown is consistent; it is empty for a kernel-PM socket, so this is a no-op there.
--- net/mptcp/pm.c | 6 ++++++ net/mptcp/pm_userspace.c | 6 ------ 2 files changed, 6 insertions(+), 6 deletions(-)
net/mptcp/pm.c +6 -0
@@ -1170,6 +1170,12 @@ void mptcp_pm_destroy(struct mptcp_sock *msk) {mptcp_pm_free_announced_list (msk);mptcp_pm_ops_release (msk); +/* Free the userspace local address list unconditionally: the socket +* can be reused (mptcp_disconnect()) and re-selected to a different +* path manager, so entries queued under the userspace PM must be +* reclaimed regardless of the PM in effect at teardown. +*/ +mptcp_userspace_pm_free_local_addr_list (msk); }void mptcp_pm_data_reset (struct mptcp_sock *msk)
net/mptcp/pm_userspace.c +0 -6
@@ -689,15 +689,9 @@ int mptcp_userspace_pm_get_addr(u8 id, struct mptcp_pm_addr_entry *addr,return ret; } -static void mptcp_pm_userspace_release (struct mptcp_sock *msk) -{ -mptcp_userspace_pm_free_local_addr_list (msk); -} -static struct mptcp_pm_ops mptcp_pm_userspace = { .get_local_id = mptcp_pm_userspace_get_local_id, .get_priority = mptcp_pm_userspace_get_priority, - .release = mptcp_pm_userspace_release, .name ="userspace" , .owner =THIS_MODULE , };
Signature · 1 line
2.34.1
PATCH 2/2 mptcp: pm: fix memory leak from alloc-during-teardown race
In reply to Shardul Bankarmptcp_pm_destroy() empties msk->pm.anno_list and msk->pm.userspace_pm_local_addr_list under msk->pm.lock during socket teardown, dropping the lock between the two.
A concurrent userspace PM genl ANNOUNCE on the same msk holds a sock reference via mptcp_token_get_sock() and, in mptcp_pm_nl_announce_doit(), calls mptcp_userspace_pm_append_new_local_addr() and mptcp_pm_announced_alloc(). Both take msk->pm.lock briefly to add to their respective lists. Because the genl handler holds a sock reference, mptcp_pm_destroy() may run on the same msk via mptcp_disconnect(), which invokes mptcp_destroy_common() without dropping the sock refcount, before the handler completes.
If the lock acquisitions interleave such that mptcp_pm_destroy() empties a list first, the later alloc adds its entry to a list head that nothing else iterates for this msk, and the entry leaks. kmemleak reports both mptcp_pm_add_addr objects (from mptcp_pm_announced_alloc()) and mptcp_pm_addr_entry objects (from mptcp_userspace_pm_append_new_local_addr()) under sustained concurrent ANNOUNCE + close load against the userspace PM.
Add an MPTCP_PM_DESTROYING bit in msk->pm.status, set by mptcp_pm_destroy() under pm.lock before the lists are emptied and checked under pm.lock by the alloc paths. Either the alloc takes pm.lock first, in which case its entry is on the list when mptcp_pm_destroy() frees it; or mptcp_pm_destroy() takes pm.lock first, in which case the later alloc observes the bit and refuses.
Found by an MPTCP protocol-flow harness extending BRF (arXiv:2305.08782).
--- net/mptcp/pm.c | 7 +++++++ net/mptcp/pm_userspace.c | 4 ++++ net/mptcp/protocol.h | 8 ++++++-- 3 files changed, 17 insertions(+), 2 deletions(-)
net/mptcp/pm.c +7 -0
@@ -441,6 +441,9 @@ bool mptcp_pm_announced_alloc(struct mptcp_sock *msk,lockdep_assert_held (&msk->pm.lock); +if (msk->pm.status &BIT (MPTCP_PM_DESTROYING )) +return false; + add_entry =mptcp_pm_announced_lookup (msk, addr);if (add_entry) {if (WARN_ON_ONCE (mptcp_pm_is_kernel (msk)))
@@ -1168,6 +1171,10 @@ static void mptcp_pm_ops_release(struct mptcp_sock *msk)void mptcp_pm_destroy (struct mptcp_sock *msk) { +spin_lock_bh (&msk->pm.lock); + msk->pm.status |=BIT (MPTCP_PM_DESTROYING ); +spin_unlock_bh (&msk->pm.lock); +mptcp_pm_free_announced_list (msk);mptcp_pm_ops_release (msk);/* Free the userspace local address list unconditionally: the socket
net/mptcp/pm_userspace.c +4 -0
@@ -54,6 +54,10 @@ static int mptcp_userspace_pm_append_new_local_addr(struct mptcp_sock *msk,bitmap_zero (id_bitmap,MPTCP_PM_MAX_ADDR_ID +1 );spin_lock_bh (&msk->pm.lock); +if (msk->pm.status &BIT (MPTCP_PM_DESTROYING )) { + ret = -EINVAL ; + goto append_err; + }mptcp_for_each_userspace_pm_addr (msk, e) { addr_match =mptcp_addresses_equal (&e->addr, &entry->addr, true);if (addr_match && entry->addr.id ==0 && needs_id)
net/mptcp/protocol.h +6 -2
@@ -189,8 +189,12 @@ enum mptcp_pm_status {MPTCP_PM_ESTABLISHED ,MPTCP_PM_SUBFLOW_ESTABLISHED ,MPTCP_PM_ALREADY_ESTABLISHED ,/* persistent status, set after ESTABLISHED event */ -MPTCP_PM_MPC_ENDPOINT_ACCOUNTED /* persistent status, set after MPC local address is -* accounted int id_avail_bitmap +MPTCP_PM_MPC_ENDPOINT_ACCOUNTED ,/* persistent status, set after MPC local address is +* accounted int id_avail_bitmap +*/ + MPTCP_PM_DESTROYING, /* set under pm.lock by mptcp_pm_destroy() to fence + * out PM list allocs that would be orphaned + * by the teardown */ };
Signature · 1 line
2.34.1
Matthieu Baerts
maintainerRe: [PATCH mptcp-next 2/2] mptcp: pm: fix memory leak from alloc-during-teardown race
Hi Shardul,
Thank you for the v4!
(Note: please make sure to keep the version in the prefix -- b4 does that for you automatically ;) -- to help with the patch tracking)
Quoted 4 lines · Shardul Bankar
mptcp_pm_destroy()emptiesmsk->pm.anno_listandmsk->pm.userspace_pm_local_addr_listundermsk->pm.lockduring socket teardown, dropping the lock between the two.
Quoted 9 lines
A concurrent userspace PM genl ANNOUNCE on the same msk holds a sock reference via
mptcp_token_get_sock()and, inmptcp_pm_nl_announce_doit(), callsmptcp_userspace_pm_append_new_local_addr()andmptcp_pm_announced_alloc(). Both takemsk->pm.lockbriefly to add to their respective lists. Because the genl handler holds a sock reference,mptcp_pm_destroy()may run on the same msk viamptcp_disconnect(), which invokesmptcp_destroy_common()without dropping the sock refcount, before the handler completes.
Sashiko seems to say [1] that the mptcp_disconnect() path is not covered (when 'shutdown()' then 'connect()' is called from the userspace), but it looks like that's what you are covering it. Just to be sure: is it a false positive? (I didn't check the different code path to see if that's possible)
[1]
https://sashiko.dev/#/patchset/20260620084424.3072634-1-shardul.b%40mpiricsoftware.com?part=2
Cheers,
Matt
Signature · 1 line
Sponsored by the NGI0 Core fund.
Re: [PATCH mptcp-next 2/2] mptcp: pm: fix memory leak from alloc-during-teardown race
Hi Matt,
Thanks for the review, and for flagging Sashiko's note.
Quoted 7 lines · Matthieu Baerts
Hi Shardul,
Thank you for the v4!
(Note: please make sure to keep the version in the prefix -- b4 does that for you automatically ;) -- to help with the patch tracking)
Yes I'll be more careful, thanks for pointing it out.
Quoted 30 lines · Shardul Bankar
Quoted 5 lines
mptcp_pm_destroy()emptiesmsk->pm.anno_listandmsk->pm.userspace_pm_local_addr_listundermsk->pm.lockduring socket
teardown, dropping the lock between the two.Quoted 13 lines
A concurrent userspace PM genl ANNOUNCE on the same msk holds a sock
reference viamptcp_token_get_sock()and, inmptcp_pm_nl_announce_doit(), callsmptcp_userspace_pm_append_new_local_addr()andmptcp_pm_announced_alloc(). Both takemsk->pm.lockbriefly to add to
their respective lists. Because the genl handler holds a sock reference,mptcp_pm_destroy()may run on the same msk viamptcp_disconnect(), which
invokesmptcp_destroy_common()without dropping the sock refcount, before the handler completes.Sashiko seems to say [1] that the
mptcp_disconnect()path is not covered
(when 'shutdown()' then 'connect()' is called from the userspace), but
it looks like that's what you are covering it. Just to be sure: is it a
false positive? (I didn't check the different code path to see if that's
possible)
Quoted 5 lines
[1]
https://sashiko.dev/#/patchset/20260620084424.3072634-1-shardul.b%40mpiricsoftware.com?part=2Cheers,
Matt
It's a false positive for the leak, and your reading is right.
The factual part of the note is true: mptcp_pm_destroy() sets MPTCP_PM_DESTROYING, but mptcp_disconnect() then calls mptcp_pm_data_reset(), whose memset(&pm->reset, ...) clears the bit (status lives in the reset struct_group). So the bit deliberately does not survive a disconnect(). It just doesn't cause a leak, because the two teardown paths are covered by two different mechanisms:
- Final teardown: __mptcp_destroy_sock() sets pm.status = 0 and then calls sk_prot->destroy() -> mptcp_destroy_common() -> mptcp_pm_destroy(), which sets DESTROYING under pm.lock right around the list free. There is no pm_data_reset() afterwards, so the bit persists. A concurrent ANNOUNCE (mptcp_userspace_pm_append_new_local_addr() runs under pm.lock only, before announce_doit() takes lock_sock()) either runs before the free and gets reclaimed, or sees DESTROYING and aborts. This is the race the fence exists for.
- disconnect()/reuse: the userspace list free in mptcp_pm_destroy() is unconditional (patch 1/2) and runs on every teardown. An entry appended after the disconnect-time reset lands on the reused, still-live socket and is reclaimed by the unconditional free at that socket's next teardown. So it is never orphaned. The fence bit being reset here is harmless by design.
Thanks,
Shardul
Hi Shardul,
Thank you for your modifications, that's great!
Our CI did some validations and here is its report:
- KVM Validation: normal (except selftest_mptcp_join): Success! ✅
- KVM Validation: normal (only selftest_mptcp_join): Success! ✅
- KVM Validation: debug (except selftest_mptcp_join): Notice: Call Traces at boot time, rebooted and continued ⚠️
- KVM Validation: debug (only selftest_mptcp_join): Notice: Call Traces at boot time, rebooted and continued ⚠️
- KVM Validation: btf-normal (only bpftest_all): Success! ✅
- KVM Validation: btf-debug (only bpftest_all): Notice: Call Traces at boot time, rebooted and continued ⚠️
- Task: https://github.com/multipath-tcp/mptcp_net-next/actions/runs/27866429471
Initiator: Patchew Applier
Commits: https://github.com/multipath-tcp/mptcp_net-next/commits/971d1ed29b64
Patchwork: https://patchwork.kernel.org/project/mptcp/list/?series=1114146
If there are some issues, you can reproduce them using the same environment as the one used by the CI thanks to a docker image, e.g.:
$ cd [kernel source code]
$ docker run -v "${PWD}:${PWD}:rw" -w "${PWD}" --privileged --rm -it \
--pull always mptcp/mptcp-upstream-virtme-docker:latest \
auto-normalFor more details:
https://github.com/multipath-tcp/mptcp-upstream-virtme-docker
Please note that despite all the efforts that have been already done to have a stable tests suite when executed on a public CI like here, it is possible some reported issues are not due to your modifications. Still, do not hesitate to help us improve that ;-)
Cheers,
MPTCP GH Action bot
Bot operated by Matthieu Baerts (NGI0 Core)
Matthieu Baerts
maintainerHi Shardul,
Quoted 15 lines · Shardul Bankar
This is the net-next counterpart of the net fix "mptcp: pm: fix memory leak from alloc-during-teardown race" (v3):
https://lore.kernel.org/all/[email protected]/v3 remains the canonical fix for net; it applies as-is on export-net. It does not apply on export, because the userspace local address list free moved into the userspace path manager .release callback in "mptcp: pm:
init and releasemptcp_pm_ops". This series ports the fix to net-next:1/2 squashes the unconditional free into that refactor commit (covering
the disconnect/reuse path).
2/2 is theMPTCP_PM_DESTROYINGfence (the alloc-during-teardown race),
rebased onto the renamedmptcp_pm_announced_alloc()/mptcp_pm_destroy().
Quoted 4 lines
The net fix was validated with an MPTCP protocol-flow harness (BRF) over multi-hour runs with zero kmemleak reports (reported on the v3 thread). This port is structurally equivalent and is sent so the CI can exercise it on net-next.
Thank you for the v3, the reply to Sashiko's comment, and sorry for the delay.
Note that I applied the patch from the v2 on top of net, and used the message that was on the v3. I also reduced the comments in the code.
I got a bunch of conflicts, but that was expected.
Now in our tree:
New patches for t/upstream-net and t/upstream:
- 2b2f15275853: mptcp: pm: fix memory leak from alloc-during-teardown race
- f53bd07233a6: conflict in t/mptcp-pm-init-and-release-mptcp_pm_ops
- 919a44ee381d: "squashed" (with conflicts) in "mptcp: pm: init and release mptcp_pm_ops"
- c7d92e52241a: conflict in t/mptcp-pm-add-get_local_id-interface
- 58c196b64826: conflict in t/mptcp-pm-add-get_priority-interface
- Results: 8fa2ab05d71f..a811f9685bfd (export-net)
- Results: 588708fb07a7..4cc6862c4795 (export)
Tests are now in progress:
- export-net:
https://github.com/multipath-tcp/mptcp_net-next/commit/a1feb73d0d2bb6aa9873ac55926f920e6d1209ba/checks
- export:
https://github.com/multipath-tcp/mptcp_net-next/commit/ca43dbbb8a8abda8ead4866bfd62b06bdf1c7d08/checks
Cheers,
Matt
Signature · 1 line
Sponsored by the NGI0 Core fund.