mm/page_alloc,slab: return NULL early from *_nolock() memory allocation APIs in NMI on UP
Subsystems (4)
Harry Yoo (Oracle)
maintainerv2 cover mm/page_alloc,slab: return NULL early from *_nolock() memory allocation APIs in NMI on UP
Due to my mistake, V1 was sent twice w/o proper cover letter and
Cc: stable. Please ignore V1. Apologies for the noise.
Changes since V1:
- used b4 to send patch series (w/ a proper cover letter) instead of my broken git send-email script (Thanks Vlastimil)
- added Cc: stable to patches 1 and 2
On UP kernels (!CONFIG_SMP), spin_trylock() is a no-op that unconditionally succeeds even when the lock is already held. As a result, alloc_frozen_pages_nolock() and kmalloc_nolock() called from an NMI context can successfully re-acquire the lock that the page/slab allocators are already holding (no deadlock because it's trylock, but leads to e.g., allocating the same page/object twice and causing use-after-free).
It was discovered while testing the new kmalloc/kfree_nolock() test case in the slub_kunit test module with CONFIG_DEBUG_SPINLOCK=y on a UP kernel.
Patch 1 fixes alloc_frozen_pages_nolock() and
patch 2 fixes kmalloc_nolock().
Note: As pointed out by Vlastimil Babka [1], in theory a kprobe in a locked section could trigger the same issue. However, fixing that involves a non-trivial rework (e.g., inventing a new spinlock type) or introduces unnecessary overhead for all spinlocks on UP (e.g., let all spinlocks check locked status on UP).
Given that BPF tracing on UP is rare, and it's even more unlikely to trace a function called from the memory allocator within the locked section, this patch series addresses the issue only on NMI contexts (which is rare as well but now covered by the new test case).
[1] https://lore.kernel.org/linux-mm/[email protected]
---
Harry Yoo (Oracle) (2):
mm/page_alloc: return NULL early from alloc_frozen_pages_nolock() in NMI on UP mm/slab: return NULL early from kmalloc_nolock() in NMI on UP
mm/page_alloc.c | 5 +++++ mm/slub.c | 4 ++++ 2 files changed, 9 insertions(+)
---
base-commit: ba24da38a519dfcff8cce3f3f2726d7b159a4d75
change-id: 20260427-nolock-api-fix-bd056911e68e
Best regards,
Signature · 2 lines
Cheers, Harry / Hyeonggon
Harry Yoo (Oracle)
maintainerPATCH v2 1/2 mm/page_alloc: return NULL early from alloc_frozen_pages_nolock() in NMI on UP
On UP kernels (!CONFIG_SMP), spin_trylock() is a no-op that unconditionally succeeds even when the lock is already held. As a result, alloc_frozen_pages_nolock() called from NMI context can re-enter rmqueue() and acquire the zone lock that the interrupted context is already holding, corrupting the freelists.
With CONFIG_DEBUG_SPINLOCK on UP, the following BUG is triggered with the slub_kunit test module:
BUG: spinlock trylock failure on UP on CPU#0, kunit_try_catch/243 [...] Call Trace: <NMI> dump_stack_lvl+0x3f/0x60 do_raw_spin_trylock+0x41/0x50 _raw_spin_trylock+0x24/0x50 rmqueue.isra.0+0x2a9/0xa70 get_page_from_freelist+0xeb/0x450 alloc_frozen_pages_nolock_noprof+0x111/0x1e0 allocate_slab+0x42a/0x500 ___slab_alloc+0xa7/0x4c0 kmalloc_nolock_noprof+0x164/0x310 [...] </NMI>
Fix this by returning NULL early when invoked from NMI on a UP kernel.
--- mm/page_alloc.c | 5 +++++ 1 file changed, 5 insertions(+)
mm/page_alloc.c +5 -0
@@ -7737,6 +7737,11 @@ struct page *alloc_frozen_pages_nolock_noprof(gfp_t gfp_flags, int nid, unsigned*/ if (IS_ENABLED (CONFIG_PREEMPT_RT ) && (in_nmi () ||in_hardirq ()))return NULL ; + +/* On UP, spin_trylock() always succeeds even when it is locked */ +if (!IS_ENABLED (CONFIG_SMP ) &&in_nmi ()) +return NULL ; +if (!pcp_allowed_order (order))return NULL ;
Signature · 1 line
2.43.0
Harry Yoo (Oracle)
maintainerPATCH v2 2/2 mm/slab: return NULL early from kmalloc_nolock() in NMI on UP
In reply to Harry Yoo (Oracle)On UP kernels (!CONFIG_SMP), spin_trylock() is a no-op that unconditionally succeeds even when the lock is already held. As a result, kmalloc_nolock() called from NMI context can re-enter the slab allocator and acquire n->list_lock that the interrupted context is already holding, corrupting slab state.
With CONFIG_DEBUG_SPINLOCK on UP, the following BUG is triggered with the slub_kunit test module:
BUG: spinlock trylock failure on UP on CPU#0, kunit_try_catch/243 [...] Call Trace: <NMI> dump_stack_lvl+0x3f/0x60 do_raw_spin_trylock+0x41/0x50 _raw_spin_trylock+0x24/0x50 get_from_partial_node+0x120/0x4d0 ___slab_alloc+0x8a/0x4c0 kmalloc_nolock_noprof+0x164/0x310 [...] </NMI>
Fix this by returning NULL early when invoked from NMI on a UP kernel.
--- mm/slub.c | 4 ++++ 1 file changed, 4 insertions(+)
mm/slub.c +4 -0
@@ -5339,6 +5339,10 @@ void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node)if (IS_ENABLED (CONFIG_PREEMPT_RT ) && (in_nmi () ||in_hardirq ()))return NULL ; +/* On UP, spin_trylock() always succeeds even when it is locked */ +if (!IS_ENABLED (CONFIG_SMP ) &&in_nmi ()) +return NULL ; + retry:if (unlikely (size >KMALLOC_MAX_CACHE_SIZE ))return NULL ;
Signature · 1 line
2.43.0
Vlastimil Babka (SUSE)
maintainerQuoted 16 lines · Harry Yoo (Oracle)
Due to my mistake, V1 was sent twice w/o proper cover letter and
Cc: stable. Please ignore V1. Apologies for the noise.Changes since V1:
- used b4 to send patch series (w/ a proper cover letter) instead of my broken git send-email script (Thanks Vlastimil)
- added Cc: stable to patches 1 and 2On UP kernels (!CONFIG_SMP),
spin_trylock()is a no-op that unconditionally succeeds even when the lock is already held. As a result,alloc_frozen_pages_nolock()andkmalloc_nolock()called from an NMI context can successfully re-acquire the lock that the page/slab allocators are already holding (no deadlock because it's trylock, but leads to e.g., allocating the same page/object twice and causing use-after-free).
Quoted 6 lines
It was discovered while testing the new kmalloc/kfree_nolock() test case in the
slub_kunittest module withCONFIG_DEBUG_SPINLOCK=yon a UP kernel.Patch 1 fixes
alloc_frozen_pages_nolock()and
patch 2 fixeskmalloc_nolock().
Thanks. Given the problem exposed is in a slab kunit test I think it's better to handle this in the slab tree. The page_alloc change is small and should not cause conflicts. So I've merged both in slab/for-next.
Quoted 31 lines
Note: As pointed out by Vlastimil Babka [1], in theory a kprobe in a locked section could trigger the same issue. However, fixing that involves a non-trivial rework (e.g., inventing a new spinlock type) or introduces unnecessary overhead for all spinlocks on UP (e.g., let all spinlocks check locked status on UP).
Given that BPF tracing on UP is rare, and it's even more unlikely to trace a function called from the memory allocator within the locked section, this patch series addresses the issue only on NMI contexts (which is rare as well but now covered by the new test case).
[1] https://lore.kernel.org/linux-mm/[email protected]
Cc stable.vger.kernel.org---
Harry Yoo (Oracle) (2):
mm/page_alloc: return NULL early fromalloc_frozen_pages_nolock()in NMI on UP mm/slab: return NULL early fromkmalloc_nolock()in NMI on UPmm/page_alloc.c | 5 +++++ mm/slub.c | 4 ++++ 2 files changed, 9 insertions(+)---
base-commit:ba24da38a519dfcff8cce3f3f2726d7b159a4d75
change-id: 20260427-nolock-api-fix-bd056911e68eBest regards,
Signature · 2 lines
Cheers, Harry / Hyeonggon
Harry Yoo (Oracle)
maintainerQuoted 21 lines · Vlastimil Babka (SUSE)
Quoted 20 lines · Harry Yoo (Oracle)
Due to my mistake, V1 was sent twice w/o proper cover letter and
Cc: stable. Please ignore V1. Apologies for the noise.Changes since V1:
- used b4 to send patch series (w/ a proper cover letter) instead of my broken git send-email script (Thanks Vlastimil)
- added Cc: stable to patches 1 and 2On UP kernels (!CONFIG_SMP),
spin_trylock()is a no-op that unconditionally succeeds even when the lock is already held. As a result,alloc_frozen_pages_nolock()andkmalloc_nolock()called from an NMI context can successfully re-acquire the lock that the page/slab allocators are already holding (no deadlock because it's trylock, but leads to e.g., allocating the same page/object twice and causing use-after-free).It was discovered while testing the new kmalloc/kfree_nolock() test case in the
slub_kunittest module withCONFIG_DEBUG_SPINLOCK=yon a UP kernel.
Quoted 5 lines
Quoted 2 lines
Patch 1 fixes
alloc_frozen_pages_nolock()and
patch 2 fixeskmalloc_nolock().Thanks. Given the problem exposed is in a slab kunit test I think it's better to handle this in the slab tree.
Ack, I'm fine either way.
Quoted 2 lines
The
page_allocchange is small and
should not cause conflicts.
Right.
Quoted 1 line
So I've merged both in slab/for-next.
Thanks! (I see it's been added to slab/for-next-fixes)
Signature · 2 lines
Cheers, Harry / Hyeonggon