mm/percpu: Fix possible NOFS/NOIO reclaim recursion
Subsystems (2)
cover mm/percpu: Fix possible NOFS/NOIO reclaim recursion
Commit 9a5b183941b5 ("mm, percpu: do not consider sleepable allocations atomic") allowed GFP_NOFS and GFP_NOIO percpu allocations to use pcpu_alloc_mutex and the chunk creation slow path. This restored the allocation capability that was lost when those constrained allocations were treated as atomic, but it also opens two possible reclaim recursion problems.
The first problem is that the create and populate slow paths do not fully preserve the caller's allocation constraints. pcpu_alloc_noprof() derives pcpu_gfp from the caller supplied GFP mask and passes it to the backing page allocator. However, pcpu_create_chunk() calls pcpu_get_vm_areas(), and population can allocate temporary metadata or page tables while mapping backing pages. Those internal allocations can use GFP_KERNEL. A caller using GFP_NOFS or GFP_NOIO can therefore still enter unconstrained FS or IO reclaim while holding pcpu_alloc_mutex. This defeats the caller's allocation context.
The second problem is a possible pcpu_alloc_mutex recursion from reclaim. If reclaim is entered while pcpu_alloc_mutex is already held, and reclaim reaches a path which allocates percpu memory with GFP_NOFS or GFP_NOIO, the nested allocation can now try to take pcpu_alloc_mutex again because 9a5b183941b5 no longer treats those masks as atomic.
Another possible way to avoid these issues is to revert 9a5b183941b5. However, that would also bring back the premature allocation failures for sleepable GFP_NOFS/GFP_NOIO percpu users that 9a5b183941b5 was intended to fix.
Kaitao Cheng (2):
mm/percpu: Preserve NOFS/NOIO scope during chunk create and populate mm/percpu: Avoid pcpu_alloc_mutex recursion from reclaim
mm/percpu.c | 39 +++++++++++++++++++++++++++++++++++---- 1 file changed, 35 insertions(+), 4 deletions(-)
Signature · 1 line
2.50.1 (Apple Git-155)
PATCH 1/2 mm/percpu: Preserve NOFS/NOIO scope during chunk create and populate
From: Kaitao Cheng <[email protected]>
pcpu_alloc_noprof() derives pcpu_gfp from the caller supplied GFP mask and passes it to the backing percpu allocators. This preserves GFP_NOFS and GFP_NOIO for pcpu_alloc_pages() and for the initial pcpu_chunk allocation.
However, the chunk creation and population slow paths also call helpers which do not take a GFP mask and perform internal allocations with GFP_KERNEL. For example, pcpu_create_chunk() calls pcpu_get_vm_areas(), and population can allocate temporary metadata or page tables while mapping backing pages. As a result, a caller which explicitly uses GFP_NOFS or GFP_NOIO can still enter FS or IO reclaim while creating or populating a percpu chunk.
This is problematic for callers which use GFP_NOFS or GFP_NOIO because they are already holding filesystem or IO-path locks. If free chunks are exhausted, the percpu allocation can take pcpu_alloc_mutex and then enter unconstrained reclaim from these internal allocations, defeating the caller's allocation context and potentially recreating reclaim lock dependencies.
Wrap chunk creation and population in a scoped NOIO or NOFS context when pcpu_gfp has the corresponding constraints. Leave ordinary GFP_KERNEL allocations unchanged so they retain full reclaim capability.
--- mm/percpu.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+)
mm/percpu.c +26 -0
@@ -1778,6 +1778,23 @@ static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t s }#endif +static unsigned int pcpu_memalloc_scope_save (gfp_t gfp) +{ +if (!(gfp & __GFP_IO)) +return memalloc_noio_save (); +if (!(gfp & __GFP_FS)) +return memalloc_nofs_save (); +return 0 ; +} + +static void pcpu_memalloc_scope_restore (gfp_t gfp,unsigned int flags) +{ +if (!(gfp & __GFP_IO)) +memalloc_noio_restore (flags); +else if (!(gfp & __GFP_FS)) +memalloc_nofs_restore (flags); +} +/** * pcpu_alloc - the percpu allocator * @size: size of area to allocate in bytes
@@ -1901,7 +1918,12 @@ void __percpu *pcpu_alloc_noprof(size_t size, size_t align, bool reserved,/* No space left. Create a new chunk. */ if (list_empty (&pcpu_chunk_lists[pcpu_free_slot])) { +unsigned int pcpu_scope; + + pcpu_scope =pcpu_memalloc_scope_save (pcpu_gfp); chunk =pcpu_create_chunk (pcpu_gfp); +pcpu_memalloc_scope_restore (pcpu_gfp, pcpu_scope); +if (!chunk) { err ="failed to allocate new chunk" ; goto fail;
@@ -1931,9 +1953,13 @@ void __percpu *pcpu_alloc_noprof(size_t size, size_t align, bool reserved, page_end =PFN_UP (off + size);for_each_clear_bitrange_from (rs, re, chunk->populated, page_end) { +unsigned int pcpu_scope; +WARN_ON (chunk->immutable); + pcpu_scope =pcpu_memalloc_scope_save (pcpu_gfp); ret =pcpu_populate_chunk (chunk, rs, re, pcpu_gfp); +pcpu_memalloc_scope_restore (pcpu_gfp, pcpu_scope);spin_lock_irqsave (&pcpu_lock, flags);if (ret) {
Signature · 1 line
2.50.1 (Apple Git-155)
Re: [PATCH 1/2] mm/percpu: Preserve NOFS/NOIO scope during chunk create and populate
Quoted 21 lines · Kaitao Cheng
From: Kaitao Cheng <[email protected]>
pcpu_alloc_noprof()derivespcpu_gfpfrom the caller supplied GFP mask and passes it to the backing percpu allocators. This preservesGFP_NOFSandGFP_NOIOforpcpu_alloc_pages()and for the initialpcpu_chunkallocation.However, the chunk creation and population slow paths also call helpers which do not take a GFP mask and perform internal allocations with
GFP_KERNEL. For example,pcpu_create_chunk()callspcpu_get_vm_areas(), and population can allocate temporary metadata or page tables while mapping backing pages. As a result, a caller which explicitly usesGFP_NOFSorGFP_NOIOcan still enter FS or IO reclaim while creating or populating a percpu chunk.This is problematic for callers which use
GFP_NOFSorGFP_NOIObecause they are already holding filesystem or IO-path locks. If free chunks are exhausted, the percpu allocation can takepcpu_alloc_mutexand then enter unconstrained reclaim from these internal allocations, defeating the caller's allocation context and potentially recreating reclaim lock dependencies.
Quoted 5 lines
Wrap chunk creation and population in a scoped NOIO or NOFS context when
pcpu_gfphas the corresponding constraints. Leave ordinaryGFP_KERNELallocations unchanged so they retain full reclaim capability.Fixes 9a5b183941b5 ("mm, percpu: do not consider sleepable allocations atomic")
I assume you _did not_ observe this in production? As in no reclaim path should be insane^W daring enough to do pcpu allocations?
Quoted 19 lines
Signed-off-by Kaitao Cheng <[email protected]>--- mm/percpu.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+)mm/percpu.c +6
-0@@ -1778,6 +1778,23 @@ static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t s }#endif +static unsigned int pcpu_memalloc_scope_save (gfp_t gfp) +{ +if (!(gfp & __GFP_IO)) +return memalloc_noio_save (); +if (!(gfp & __GFP_FS)) +return memalloc_nofs_save ();
mm/percpu.c static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t s · line 1796
Quoted 10 lines
mm/percpu.c +10
-0+return 0 ; +} + +static void pcpu_memalloc_scope_restore (gfp_t gfp,unsigned int flags) +{ +if (!(gfp & __GFP_IO)) +memalloc_noio_restore (flags); +else if (!(gfp & __GFP_FS)) +memalloc_nofs_restore (flags); +}
I disagree with this. We already have gfp flags, they're already passed to pcpu_create_chunk() and pcpu_populate_chunk(). It's their job to respect the gfp flags and
Do The Right Thing(tm). Can you fix the problematic places? It seems like it's mostly the vmalloc backend that's problematic.
Quoted 34 lines
mm/percpu.c +10
-0+/** * pcpu_alloc - the percpu allocator * @size: size of area to allocate in bytes @@ -1901,7 +1918,12 @@ void __percpu *pcpu_alloc_noprof(size_t size, size_t align, bool reserved,/* No space left. Create a new chunk. */ if (list_empty (&pcpu_chunk_lists[pcpu_free_slot])) { +unsigned int pcpu_scope; + + pcpu_scope =pcpu_memalloc_scope_save (pcpu_gfp); chunk =pcpu_create_chunk (pcpu_gfp); +pcpu_memalloc_scope_restore (pcpu_gfp, pcpu_scope); +if (!chunk) { err ="failed to allocate new chunk" ; goto fail;@@ -1931,9 +1953,13 @@ void __percpu *pcpu_alloc_noprof(size_t size, size_t align, bool reserved, page_end =PFN_UP (off + size);for_each_clear_bitrange_from (rs, re, chunk->populated, page_end) { +unsigned int pcpu_scope; +WARN_ON (chunk->immutable); + pcpu_scope =pcpu_memalloc_scope_save (pcpu_gfp); ret =pcpu_populate_chunk (chunk, rs, re, pcpu_gfp); +pcpu_memalloc_scope_restore (pcpu_gfp, pcpu_scope);spin_lock_irqsave (&pcpu_lock, flags);if (ret) {Signature · 1 line
2.50.1 (Apple Git-155)
Signature · 1 line
Pedro
Re: [PATCH 1/2] mm/percpu: Preserve NOFS/NOIO scope during chunk create and populate
Quoted 22 lines · Pedro Falcato
Quoted 21 lines · Kaitao Cheng
From: Kaitao Cheng <[email protected]>
pcpu_alloc_noprof()derivespcpu_gfpfrom the caller supplied GFP mask and passes it to the backing percpu allocators. This preservesGFP_NOFSandGFP_NOIOforpcpu_alloc_pages()and for the initialpcpu_chunkallocation.However, the chunk creation and population slow paths also call helpers which do not take a GFP mask and perform internal allocations with
GFP_KERNEL. For example,pcpu_create_chunk()callspcpu_get_vm_areas(), and population can allocate temporary metadata or page tables while mapping backing pages. As a result, a caller which explicitly usesGFP_NOFSorGFP_NOIOcan still enter FS or IO reclaim while creating or populating a percpu chunk.This is problematic for callers which use
GFP_NOFSorGFP_NOIObecause they are already holding filesystem or IO-path locks. If free chunks are exhausted, the percpu allocation can takepcpu_alloc_mutexand then enter unconstrained reclaim from these internal allocations, defeating the caller's allocation context and potentially recreating reclaim lock dependencies.
Quoted 8 lines
Quoted 5 lines
Wrap chunk creation and population in a scoped NOIO or NOFS context when
pcpu_gfphas the corresponding constraints. Leave ordinaryGFP_KERNELallocations unchanged so they retain full reclaim capability.Fixes 9a5b183941b5 ("mm, percpu: do not consider sleepable allocations atomic")I assume you
_didnot_observe this in production? As in no reclaim path should be insane^W daring enough to do pcpu allocations?
Oops, I mixed my issues up. This is purely a GFP flags issue. A quick "git grep alloc_percpu_gfp" shows that the vast majority (all?) callers are using some combination of GFP_KERNEL or GFP_ATOMIC + other GFP flags, but no NOFS or NOIO as far as I can see. So you probably did not observe this?
Quoted 73 lines
Quoted 20 lines
Signed-off-by Kaitao Cheng <[email protected]>--- mm/percpu.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+)mm/percpu.c +6
-0@@ -1778,6 +1778,23 @@ static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t s }#endif +static unsigned int pcpu_memalloc_scope_save (gfp_t gfp) +{ +if (!(gfp & __GFP_IO)) +return memalloc_noio_save (); +if (!(gfp & __GFP_FS)) +return memalloc_nofs_save ();mm/percpu.c static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t s · line 1796
Quoted 10 lines
mm/percpu.c +10
-0+return 0 ; +} + +static void pcpu_memalloc_scope_restore (gfp_t gfp,unsigned int flags) +{ +if (!(gfp & __GFP_IO)) +memalloc_noio_restore (flags); +else if (!(gfp & __GFP_FS)) +memalloc_nofs_restore (flags); +}I disagree with this. We already have gfp flags, they're already passed to
pcpu_create_chunk()andpcpu_populate_chunk(). It's their job to respect the gfp flags and
Do The Right Thing(tm). Can you fix the problematic places? It seems like it's mostly the vmalloc backend that's problematic.Quoted 34 lines
mm/percpu.c +10
-0+/** * pcpu_alloc - the percpu allocator * @size: size of area to allocate in bytes @@ -1901,7 +1918,12 @@ void __percpu *pcpu_alloc_noprof(size_t size, size_t align, bool reserved,/* No space left. Create a new chunk. */ if (list_empty (&pcpu_chunk_lists[pcpu_free_slot])) { +unsigned int pcpu_scope; + + pcpu_scope =pcpu_memalloc_scope_save (pcpu_gfp); chunk =pcpu_create_chunk (pcpu_gfp); +pcpu_memalloc_scope_restore (pcpu_gfp, pcpu_scope); +if (!chunk) { err ="failed to allocate new chunk" ; goto fail;@@ -1931,9 +1953,13 @@ void __percpu *pcpu_alloc_noprof(size_t size, size_t align, bool reserved, page_end =PFN_UP (off + size);for_each_clear_bitrange_from (rs, re, chunk->populated, page_end) { +unsigned int pcpu_scope; +WARN_ON (chunk->immutable); + pcpu_scope =pcpu_memalloc_scope_save (pcpu_gfp); ret =pcpu_populate_chunk (chunk, rs, re, pcpu_gfp); +pcpu_memalloc_scope_restore (pcpu_gfp, pcpu_scope);spin_lock_irqsave (&pcpu_lock, flags);if (ret) {Signature · 1 line
2.50.1 (Apple Git-155)Signature · 1 line
Pedro
Signature · 1 line
Pedro
Re: [PATCH 1/2] mm/percpu: Preserve NOFS/NOIO scope during chunk create and populate
Quoted 29 lines · Pedro Falcato
Quoted 28 lines · Pedro Falcato
Quoted 26 lines · Kaitao Cheng
From: Kaitao Cheng <[email protected]>
pcpu_alloc_noprof()derivespcpu_gfpfrom the caller supplied GFP mask and passes it to the backing percpu allocators. This preservesGFP_NOFSandGFP_NOIOforpcpu_alloc_pages()and for the initialpcpu_chunkallocation.However, the chunk creation and population slow paths also call helpers which do not take a GFP mask and perform internal allocations with
GFP_KERNEL. For example,pcpu_create_chunk()callspcpu_get_vm_areas(), and population can allocate temporary metadata or page tables while mapping backing pages. As a result, a caller which explicitly usesGFP_NOFSorGFP_NOIOcan still enter FS or IO reclaim while creating or populating a percpu chunk.This is problematic for callers which use
GFP_NOFSorGFP_NOIObecause they are already holding filesystem or IO-path locks. If free chunks are exhausted, the percpu allocation can takepcpu_alloc_mutexand then enter unconstrained reclaim from these internal allocations, defeating the caller's allocation context and potentially recreating reclaim lock dependencies.Wrap chunk creation and population in a scoped NOIO or NOFS context when
pcpu_gfphas the corresponding constraints. Leave ordinaryGFP_KERNELallocations unchanged so they retain full reclaim capability.Fixes 9a5b183941b5 ("mm, percpu: do not consider sleepable allocations atomic")
Quoted 7 lines
Quoted 2 lines
I assume you
_didnot_observe this in production? As in no reclaim path should be insane^W daring enough to do pcpu allocations?Oops, I mixed my issues up. This is purely a GFP flags issue. A quick "git grep
alloc_percpu_gfp" shows that the vast majority (all?) callers are using some combination ofGFP_KERNELorGFP_ATOMIC+ other GFP flags, but no NOFS or NOIO as far as I can see. So you probably did not observe this?
Right, this issue has not been observed in production. It came from a question raised by AI code review, and after carefully reading the code, I found that there are indeed some synchronization concerns.
Here is one example of the scenario [PATCH 1/2] is trying to address:
blk-cgroup after 5d726c4dbeed ("blk-cgroup: fix possible deadlock while configuring policy"). blkg_conf_prep() now serializes against blkcg_deactivate_policy() with q->blkcg_mutex, and blkg_alloc() was changed to GFP_NOIO for that reason:
CPU0: blkg_conf_prep()
mutex_lock(q->blkcg_mutex)
blkg_alloc(..., GFP_NOIO)
alloc_percpu_gfp(..., GFP_NOIO)
-> if percpu chunks are exhausted, chunk create/populate may do
internal GFP_KERNEL allocations
-> direct reclaim / writeback can issue IO to this queue
-> IO waits because the queue is frozen CPU1: blkcg_deactivate_policy()
blk_mq_freeze_queue(q)
mutex_lock(q->blkcg_mutex)
-> waits for CPU0
... unfreeze only happens after q->blkcg_mutex is acquired/releasedSo the concern is that the caller deliberately uses GFP_NOIO because it may hold a lock which can be acquired after queue freeze, but the percpu slow path can temporarily lose that allocation context. The failure requires the slow path (new chunk creation or population), so it is not expected to be common.
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Signed-off-by Kaitao Cheng <[email protected]>--- mm/percpu.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+)mm/percpu.c +13
-0@@ -1778,6 +1778,23 @@ static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t s }#endif +static unsigned int pcpu_memalloc_scope_save (gfp_t gfp) +{ +if (!(gfp & __GFP_IO)) +return memalloc_noio_save (); +if (!(gfp & __GFP_FS)) +return memalloc_nofs_save (); +return 0 ; +} + +static void pcpu_memalloc_scope_restore (gfp_t gfp,unsigned int flags) +{ +if (!(gfp & __GFP_IO)) +memalloc_noio_restore (flags);
Quoted 8 lines
Quoted 8 lines
mm/percpu.c static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t s · line 1796
Quoted 3 lines
mm/percpu.c +3
-0+else if (!(gfp & __GFP_FS)) +memalloc_nofs_restore (flags); +}I disagree with this. We already have gfp flags, they're already passed to
pcpu_create_chunk()andpcpu_populate_chunk(). It's their job to respect the gfp flags and
Do The Right Thing(tm). Can you fix the problematic places? It seems like it's mostly the vmalloc backend that's problematic.
I’ll try to do it.
Following your suggestion, including in [PATCH 2/2], I will also try a different approach and fix the issue by reducing the scope of the pcpu_alloc_mutex critical section.
Quoted 33 lines
Quoted 33 lines
Quoted 33 lines
mm/percpu.c +9
-0/** * pcpu_alloc - the percpu allocator * @size: size of area to allocate in bytes @@ -1901,7 +1918,12 @@ void __percpu *pcpu_alloc_noprof(size_t size, size_t align, bool reserved,/* No space left. Create a new chunk. */ if (list_empty (&pcpu_chunk_lists[pcpu_free_slot])) { +unsigned int pcpu_scope; + + pcpu_scope =pcpu_memalloc_scope_save (pcpu_gfp); chunk =pcpu_create_chunk (pcpu_gfp); +pcpu_memalloc_scope_restore (pcpu_gfp, pcpu_scope); +if (!chunk) { err ="failed to allocate new chunk" ; goto fail;@@ -1931,9 +1953,13 @@ void __percpu *pcpu_alloc_noprof(size_t size, size_t align, bool reserved, page_end =PFN_UP (off + size);for_each_clear_bitrange_from (rs, re, chunk->populated, page_end) { +unsigned int pcpu_scope; +WARN_ON (chunk->immutable); + pcpu_scope =pcpu_memalloc_scope_save (pcpu_gfp); ret =pcpu_populate_chunk (chunk, rs, re, pcpu_gfp); +pcpu_memalloc_scope_restore (pcpu_gfp, pcpu_scope);spin_lock_irqsave (&pcpu_lock, flags);if (ret) {Signature · 1 line
2.50.1 (Apple Git-155)
Signature · 2 lines
Thanks Kaitao Cheng
Dennis Zhou
maintainerRe: [PATCH 1/2] mm/percpu: Preserve NOFS/NOIO scope during chunk create and populate
Hello,
Quoted 64 lines · Kaitao Cheng
Quoted 29 lines · Pedro Falcato
Quoted 28 lines · Pedro Falcato
Quoted 26 lines · Kaitao Cheng
From: Kaitao Cheng <[email protected]>
pcpu_alloc_noprof()derivespcpu_gfpfrom the caller supplied GFP mask and passes it to the backing percpu allocators. This preservesGFP_NOFSandGFP_NOIOforpcpu_alloc_pages()and for the initialpcpu_chunkallocation.However, the chunk creation and population slow paths also call helpers which do not take a GFP mask and perform internal allocations with
GFP_KERNEL. For example,pcpu_create_chunk()callspcpu_get_vm_areas(), and population can allocate temporary metadata or page tables while mapping backing pages. As a result, a caller which explicitly usesGFP_NOFSorGFP_NOIOcan still enter FS or IO reclaim while creating or populating a percpu chunk.This is problematic for callers which use
GFP_NOFSorGFP_NOIObecause they are already holding filesystem or IO-path locks. If free chunks are exhausted, the percpu allocation can takepcpu_alloc_mutexand then enter unconstrained reclaim from these internal allocations, defeating the caller's allocation context and potentially recreating reclaim lock dependencies.Wrap chunk creation and population in a scoped NOIO or NOFS context when
pcpu_gfphas the corresponding constraints. Leave ordinaryGFP_KERNELallocations unchanged so they retain full reclaim capability.Fixes 9a5b183941b5 ("mm, percpu: do not consider sleepable allocations atomic")Quoted 7 lines
Quoted 2 lines
I assume you
_didnot_observe this in production? As in no reclaim path should be insane^W daring enough to do pcpu allocations?Oops, I mixed my issues up. This is purely a GFP flags issue. A quick "git grep
alloc_percpu_gfp" shows that the vast majority (all?) callers are using some combination ofGFP_KERNELorGFP_ATOMIC+ other GFP flags, but no NOFS or NOIO as far as I can see. So you probably did not observe this?Right, this issue has not been observed in production. It came from a question raised by AI code review, and after carefully reading the code, I found that there are indeed some synchronization concerns.
Here is one example of the scenario [PATCH 1/2] is trying to address:
blk-cgroup after
5d726c4dbeed("blk-cgroup: fix possible deadlock while configuring policy").blkg_conf_prep()now serializes againstblkcg_deactivate_policy()withq->blkcg_mutex, andblkg_alloc()was changed toGFP_NOIOfor that reason:CPU0:
blkg_conf_prep()mutex_lock(q->blkcg_mutex) blkg_alloc(..., GFP_NOIO) alloc_percpu_gfp(..., GFP_NOIO) -> if percpu chunks are exhausted, chunk create/populate may do internal GFP_KERNEL allocations -> direct reclaim / writeback can issue IO to this queue -> IO waits because the queue is frozenCPU1:
blkcg_deactivate_policy()blk_mq_freeze_queue(q) mutex_lock(q->blkcg_mutex) -> waits for CPU0 ... unfreeze only happens after q->blkcg_mutex is acquired/released
Quoted 5 lines
So the concern is that the caller deliberately uses
GFP_NOIObecause it may hold a lock which can be acquired after queue freeze, but the percpu slow path can temporarily lose that allocation context. The failure requires the slow path (new chunk creation or population), so it is not expected to be common.
This is likely just a miss for [1] where we switching to gfpflags_allow_blocking(),
This seems like it's just a miss from [1] where we switched to a less conservative approach than atomic == !GFP_KERNEL. If anything we just need to allow the additional flags through. Maybe just add GFP_NOIO and GFP_NOFS in pcpu_gfp via the whitelist.
[1] 9a5b183941b ("mm, percpu: do not consider sleepable allocations atomic")
Quoted 36 lines
Quoted 27 lines
Quoted 27 lines
Quoted 27 lines
Signed-off-by Kaitao Cheng <[email protected]>--- mm/percpu.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+)mm/percpu.c +13
-0@@ -1778,6 +1778,23 @@ static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t s }#endif +static unsigned int pcpu_memalloc_scope_save (gfp_t gfp) +{ +if (!(gfp & __GFP_IO)) +return memalloc_noio_save (); +if (!(gfp & __GFP_FS)) +return memalloc_nofs_save (); +return 0 ; +} + +static void pcpu_memalloc_scope_restore (gfp_t gfp,unsigned int flags) +{ +if (!(gfp & __GFP_IO)) +memalloc_noio_restore (flags);Quoted 8 lines
Quoted 8 lines
mm/percpu.c static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t s · line 1796
Quoted 3 lines
mm/percpu.c +3
-0+else if (!(gfp & __GFP_FS)) +memalloc_nofs_restore (flags); +}I disagree with this. We already have gfp flags, they're already passed to
pcpu_create_chunk()andpcpu_populate_chunk(). It's their job to respect the gfp flags and
Do The Right Thing(tm). Can you fix the problematic places? It seems like it's mostly the vmalloc backend that's problematic.
Quoted 6 lines
I’ll try to do it.
Following your suggestion, including in [PATCH 2/2], I will also try a different approach and fix the issue by reducing the scope of the
pcpu_alloc_mutexcritical section.
No please don't. The point of the percpu mutex is to ensure that only one person is ever possibly creating a new chunk. If you drop the mutex, then you have to deal with concurrent callers when available percpu memory is low. Percpu memory is expensive and unmovable so the cost is in the control plane to avoid excess fragmentation.
Thanks,
Dennis
Re: [PATCH 1/2] mm/percpu: Preserve NOFS/NOIO scope during chunk create and populate
Quoted 125 lines · Dennis Zhou
Hello,
Quoted 64 lines · Kaitao Cheng
Quoted 29 lines · Pedro Falcato
Quoted 28 lines · Pedro Falcato
Quoted 26 lines · Kaitao Cheng
From: Kaitao Cheng <[email protected]>
pcpu_alloc_noprof()derivespcpu_gfpfrom the caller supplied GFP mask and passes it to the backing percpu allocators. This preservesGFP_NOFSandGFP_NOIOforpcpu_alloc_pages()and for the initialpcpu_chunkallocation.However, the chunk creation and population slow paths also call helpers which do not take a GFP mask and perform internal allocations with
GFP_KERNEL. For example,pcpu_create_chunk()callspcpu_get_vm_areas(), and population can allocate temporary metadata or page tables while mapping backing pages. As a result, a caller which explicitly usesGFP_NOFSorGFP_NOIOcan still enter FS or IO reclaim while creating or populating a percpu chunk.This is problematic for callers which use
GFP_NOFSorGFP_NOIObecause they are already holding filesystem or IO-path locks. If free chunks are exhausted, the percpu allocation can takepcpu_alloc_mutexand then enter unconstrained reclaim from these internal allocations, defeating the caller's allocation context and potentially recreating reclaim lock dependencies.Wrap chunk creation and population in a scoped NOIO or NOFS context when
pcpu_gfphas the corresponding constraints. Leave ordinaryGFP_KERNELallocations unchanged so they retain full reclaim capability.Fixes 9a5b183941b5 ("mm, percpu: do not consider sleepable allocations atomic")Quoted 7 lines
Quoted 2 lines
I assume you
_didnot_observe this in production? As in no reclaim path should be insane^W daring enough to do pcpu allocations?Oops, I mixed my issues up. This is purely a GFP flags issue. A quick "git grep
alloc_percpu_gfp" shows that the vast majority (all?) callers are using some combination ofGFP_KERNELorGFP_ATOMIC+ other GFP flags, but no NOFS or NOIO as far as I can see. So you probably did not observe this?Right, this issue has not been observed in production. It came from a question raised by AI code review, and after carefully reading the code, I found that there are indeed some synchronization concerns.
Here is one example of the scenario [PATCH 1/2] is trying to address:
blk-cgroup after
5d726c4dbeed("blk-cgroup: fix possible deadlock while configuring policy").blkg_conf_prep()now serializes againstblkcg_deactivate_policy()withq->blkcg_mutex, andblkg_alloc()was changed toGFP_NOIOfor that reason:CPU0:
blkg_conf_prep()mutex_lock(q->blkcg_mutex) blkg_alloc(..., GFP_NOIO) alloc_percpu_gfp(..., GFP_NOIO) -> if percpu chunks are exhausted, chunk create/populate may do internal GFP_KERNEL allocations -> direct reclaim / writeback can issue IO to this queue -> IO waits because the queue is frozenCPU1:
blkcg_deactivate_policy()blk_mq_freeze_queue(q) mutex_lock(q->blkcg_mutex) -> waits for CPU0 ... unfreeze only happens after q->blkcg_mutex is acquired/releasedQuoted 5 lines
So the concern is that the caller deliberately uses
GFP_NOIObecause it may hold a lock which can be acquired after queue freeze, but the percpu slow path can temporarily lose that allocation context. The failure requires the slow path (new chunk creation or population), so it is not expected to be common.This is likely just a miss for [1] where we switching to
gfpflags_allow_blocking(),This seems like it's just a miss from [1] where we switched to a less conservative approach than atomic == !GFP_KERNEL. If anything we just need to allow the additional flags through. Maybe just add
GFP_NOIOandGFP_NOFSinpcpu_gfpvia the whitelist.[1]
9a5b183941b("mm, percpu: do not consider sleepable allocations atomic")Quoted 36 lines
Quoted 27 lines
Quoted 27 lines
Quoted 27 lines
Signed-off-by Kaitao Cheng <[email protected]>--- mm/percpu.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+)mm/percpu.c +13
-0@@ -1778,6 +1778,23 @@ static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t s }#endif +static unsigned int pcpu_memalloc_scope_save (gfp_t gfp) +{ +if (!(gfp & __GFP_IO)) +return memalloc_noio_save (); +if (!(gfp & __GFP_FS)) +return memalloc_nofs_save (); +return 0 ; +} + +static void pcpu_memalloc_scope_restore (gfp_t gfp,unsigned int flags) +{ +if (!(gfp & __GFP_IO)) +memalloc_noio_restore (flags);Quoted 8 lines
Quoted 8 lines
mm/percpu.c static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t s · line 1796
Quoted 3 lines
mm/percpu.c +3
-0+else if (!(gfp & __GFP_FS)) +memalloc_nofs_restore (flags); +}I disagree with this. We already have gfp flags, they're already passed to
pcpu_create_chunk()andpcpu_populate_chunk(). It's their job to respect the gfp flags and
Do The Right Thing(tm). Can you fix the problematic places? It seems like it's mostly the vmalloc backend that's problematic.Quoted 6 lines
I’ll try to do it.
Following your suggestion, including in [PATCH 2/2], I will also try a different approach and fix the issue by reducing the scope of the
pcpu_alloc_mutexcritical section.
Quoted 6 lines
No please don't. The point of the percpu mutex is to ensure that only one person is ever possibly creating a new chunk. If you drop the mutex, then you have to deal with concurrent callers when available percpu memory is low. Percpu memory is expensive and unmovable so the cost is in the control plane to avoid excess fragmentation.
You are right. pcpu_alloc_mutex seems necessary, and there does not appear to be a good way to replace it for now.
The introduction of 9a5b183941b can lead to several synchronization issues. There are already three known cases: the two described in [PATCH 0/2], plus the one raised by AI review.
https://lore.kernel.org/all/[email protected]/
https://sashiko.dev/#/patchset/[email protected]
Since optimizing around pcpu_alloc_mutex is constrained, there does not seem to be a perfect solution that addresses all of them at the same time.
However, if we revert 9a5b183941b, it seems that all of these issues would be resolved. The only downside is that the failure rate of pcpu_alloc_noprof() allocations may increase, which might be acceptable.
I would like to hear what others think.
Signature · 2 lines
Thanks Kaitao Cheng
Re: [PATCH 1/2] mm/percpu: Preserve NOFS/NOIO scope during chunk create and populate
Quoted 3 lines · Kaitao Cheng
However, if we revert
9a5b183941b, it seems that all of these issues would be resolved. The only downside is that the failure rate ofpcpu_alloc_noprof()allocations may increase, which might be acceptable.
That has practical impact on some versions of iscsid which do not have PR_SET_IO_FLUSHER. And maybe some more so I would rather not revert based on a theoretical concerns which I believe is the case here.
Signature · 2 lines
Michal Hocko SUSE Labs
Re: [PATCH 1/2] mm/percpu: Preserve NOFS/NOIO scope during chunk create and populate
Quoted 9 lines · Michal Hocko
Quoted 3 lines · Kaitao Cheng
However, if we revert
9a5b183941b, it seems that all of these issues would be resolved. The only downside is that the failure rate ofpcpu_alloc_noprof()allocations may increase, which might be acceptable.That has practical impact on some versions of iscsid which do not have
PR_SET_IO_FLUSHER. And maybe some more so I would rather not revert based on a theoretical concerns which I believe is the case here.
Based on the previous discussion, I think we have a way to address most of the concurrency issues around percpu allocation.
However, there still seems to be one remaining case that I do not yet have a good way to solve. For example:
Thread A calls pcpu_alloc_noprof() with GFP_KERNEL and takes pcpu_alloc_mutex. Since the internal allocation is not constrained by NOFS, it may enter FS reclaim while still holding pcpu_alloc_mutex, creating a dependency like:
pcpu_alloc_mutex -> fs_reclaim -> FS lock
At the same time, Thread B may already hold an FS lock and then call pcpu_alloc_noprof() with GFP_NOFS. It will try to acquire pcpu_alloc_mutex and block, creating the reverse dependency:
FS lock -> pcpu_alloc_mutex
This can still form a potential deadlock cycle.
Does anyone have a good suggestion for how to handle this remaining case? Or should we simply treat all GFP_KERNEL/GFP_NOFS allocation behavior in pcpu_alloc_noprof() as GFP_NOIO?
If there is no clear solution for now, would it be acceptable to first fix some of the issues introduced by commit 9a5b183941b, and leave this remaining case as a pre-existing historical issue to be handled separately later?
Signature · 2 lines
Thanks Kaitao Cheng
Re: [PATCH 1/2] mm/percpu: Preserve NOFS/NOIO scope during chunk create and populate
Quoted 32 lines · Kaitao Cheng
Quoted 9 lines · Michal Hocko
Quoted 3 lines · Kaitao Cheng
However, if we revert
9a5b183941b, it seems that all of these issues would be resolved. The only downside is that the failure rate ofpcpu_alloc_noprof()allocations may increase, which might be acceptable.That has practical impact on some versions of iscsid which do not have
PR_SET_IO_FLUSHER. And maybe some more so I would rather not revert based on a theoretical concerns which I believe is the case here.Based on the previous discussion, I think we have a way to address most of the concurrency issues around percpu allocation.
However, there still seems to be one remaining case that I do not yet have a good way to solve. For example:
Thread A calls
pcpu_alloc_noprof()withGFP_KERNELand takespcpu_alloc_mutex. Since the internal allocation is not constrained by NOFS, it may enter FS reclaim while still holdingpcpu_alloc_mutex, creating a dependency like:
pcpu_alloc_mutex->fs_reclaim-> FS lock
At the same time, Thread B may already hold an FS lock and then callpcpu_alloc_noprof()withGFP_NOFS. It will try to acquirepcpu_alloc_mutexand block, creating the reverse dependency:FS lock ->
pcpu_alloc_mutex
This can still form a potential deadlock cycle.
Quoted 8 lines
Does anyone have a good suggestion for how to handle this remaining case? Or should we simply treat all GFP_KERNEL/GFP_NOFS allocation behavior in
pcpu_alloc_noprof()asGFP_NOIO?If there is no clear solution for now, would it be acceptable to first fix some of the issues introduced by commit
9a5b183941b, and leave this remaining case as a pre-existing historical issue to be handled separately later?
We don't need to solve any issues that are only theoretical and based on scenarios that nobody sane should be doing, i.e. Pedro already pointed out "As in no reclaim path should be insane^W daring enough to do pcpu allocations?"
If anyone would (start to) do that, we would likely have lockdep reports from the testing bots, which warn that the scenario can now exist, even before it results in an actual deadlock.
Elsewhere Pedro said "The proper way of fixing this would probably be to release pcpu_alloc_mutex (or not have it in the first place!) while you're allocating memory."
Such a refactoring might be worth it (if it's feasible to do cleanly and doesn't come with downsides) just to eliminate these lock dependencies properly for good. Patching over individual theoretical issues is IMHO not worth it.
Re: [PATCH 1/2] mm/percpu: Preserve NOFS/NOIO scope during chunk create and populate
Quoted 37 lines · Vlastimil Babka (SUSE)
Quoted 36 lines · Kaitao Cheng
Quoted 9 lines · Michal Hocko
Quoted 3 lines · Kaitao Cheng
However, if we revert
9a5b183941b, it seems that all of these issues would be resolved. The only downside is that the failure rate ofpcpu_alloc_noprof()allocations may increase, which might be acceptable.That has practical impact on some versions of iscsid which do not have
PR_SET_IO_FLUSHER. And maybe some more so I would rather not revert based on a theoretical concerns which I believe is the case here.Based on the previous discussion, I think we have a way to address most of the concurrency issues around percpu allocation.
However, there still seems to be one remaining case that I do not yet have a good way to solve. For example:
Thread A calls
pcpu_alloc_noprof()withGFP_KERNELand takespcpu_alloc_mutex. Since the internal allocation is not constrained by NOFS, it may enter FS reclaim while still holdingpcpu_alloc_mutex, creating a dependency like:
pcpu_alloc_mutex->fs_reclaim-> FS lock
At the same time, Thread B may already hold an FS lock and then callpcpu_alloc_noprof()withGFP_NOFS. It will try to acquirepcpu_alloc_mutexand block, creating the reverse dependency:FS lock ->
pcpu_alloc_mutex
This can still form a potential deadlock cycle.Does anyone have a good suggestion for how to handle this remaining case? Or should we simply treat all GFP_KERNEL/GFP_NOFS allocation behavior in
pcpu_alloc_noprof()asGFP_NOIO?
Quoted 8 lines
Quoted 4 lines
If there is no clear solution for now, would it be acceptable to first fix some of the issues introduced by commit
9a5b183941b, and leave this remaining case as a pre-existing historical issue to be handled separately later?We don't need to solve any issues that are only theoretical and based on scenarios that nobody sane should be doing, i.e. Pedro already pointed out "As in no reclaim path should be insane^W daring enough to do pcpu allocations?"
Yes, but you do not need to do a pcp allocation from the reclaim path to hit the deadlock. All you need is a NOFS pcp allocation - e.g. one done from NOFS scope. Then you have fs lock <-> pcpu_alloc_mutex dependency and a potential deadlock. It is hard to know whether we have any of those in the kernel but I know for a fact (9a5b183941b) that there are scoped NOIO allocations so I wouldn't be suprised if the same was the case for NOFS. Not only that having NOIO is a weaker reclaim context.
Quoted 3 lines
Elsewhere Pedro said "The proper way of fixing this would probably be to release
pcpu_alloc_mutex(or not have it in the first place!) while you're allocating memory."
I do agree with this. Back then when I was dealing with the NOIO issue I've tried to look at the lock and drop it but it was not really straightforward. Maybe my lack of close understanding of the pcp allocator was an obstacle there. So if there is a path forward like that then it would certainly be the best.
Signature · 2 lines
Michal Hocko SUSE Labs
Re: [PATCH 1/2] mm/percpu: Preserve NOFS/NOIO scope during chunk create and populate
Quoted 47 lines · Michal Hocko
Quoted 37 lines · Vlastimil Babka (SUSE)
Quoted 36 lines · Kaitao Cheng
Quoted 9 lines · Michal Hocko
Quoted 3 lines · Kaitao Cheng
However, if we revert
9a5b183941b, it seems that all of these issues would be resolved. The only downside is that the failure rate ofpcpu_alloc_noprof()allocations may increase, which might be acceptable.That has practical impact on some versions of iscsid which do not have
PR_SET_IO_FLUSHER. And maybe some more so I would rather not revert based on a theoretical concerns which I believe is the case here.Based on the previous discussion, I think we have a way to address most of the concurrency issues around percpu allocation.
However, there still seems to be one remaining case that I do not yet have a good way to solve. For example:
Thread A calls
pcpu_alloc_noprof()withGFP_KERNELand takespcpu_alloc_mutex. Since the internal allocation is not constrained by NOFS, it may enter FS reclaim while still holdingpcpu_alloc_mutex, creating a dependency like:
pcpu_alloc_mutex->fs_reclaim-> FS lock
At the same time, Thread B may already hold an FS lock and then callpcpu_alloc_noprof()withGFP_NOFS. It will try to acquirepcpu_alloc_mutexand block, creating the reverse dependency:FS lock ->
pcpu_alloc_mutex
This can still form a potential deadlock cycle.Does anyone have a good suggestion for how to handle this remaining case? Or should we simply treat all GFP_KERNEL/GFP_NOFS allocation behavior in
pcpu_alloc_noprof()asGFP_NOIO?Quoted 8 lines
Quoted 4 lines
If there is no clear solution for now, would it be acceptable to first fix some of the issues introduced by commit
9a5b183941b, and leave this remaining case as a pre-existing historical issue to be handled separately later?We don't need to solve any issues that are only theoretical and based on scenarios that nobody sane should be doing, i.e. Pedro already pointed out "As in no reclaim path should be insane^W daring enough to do pcpu allocations?"
Quoted 7 lines
Yes, but you do not need to do a pcp allocation from the reclaim path to hit the deadlock. All you need is a NOFS pcp allocation - e.g. one done from NOFS scope. Then you have fs lock <->
pcpu_alloc_mutexdependency and a potential deadlock. It is hard to know whether we have any of those in the kernel but I know for a fact (9a5b183941b) that there are scoped NOIO allocations so I wouldn't be suprised if the same was the case for NOFS. Not only that having NOIO is a weaker reclaim context.
Besides the case mentioned by Michal Hocko, in my previous email I listed a scenario where blkg_conf_prep races with blkcg_deactivate_policy. This is also something that can actually happen in the existing code, and we have discussed some possible solutions.
The concurrent situation Pedro pointed out, "As in no reclaim path should be insane^W daring enough to do pcpu allocations?", does exist at the theoretical level. However, some other issues do genuinely exist. The reason they have not been observed so far may be that the reproduction probability is very low, or that they have already happened but no one has reported them to the community.
Quoted 9 lines
Quoted 3 lines
Elsewhere Pedro said "The proper way of fixing this would probably be to release
pcpu_alloc_mutex(or not have it in the first place!) while you're allocating memory."I do agree with this. Back then when I was dealing with the NOIO issue I've tried to look at the lock and drop it but it was not really straightforward. Maybe my lack of close understanding of the pcp allocator was an obstacle there. So if there is a path forward like that then it would certainly be the best.
That is indeed the optimal solution, but after thinking about it briefly, this optimization may not be easy to implement, or it may require a large amount of changes. I would really appreciate hearing any concrete suggestions people may have on how to optimize pcpu_alloc_mutex. Otherwise, this issue will remain blocked here.
Signature · 2 lines
Thanks Kaitao Cheng
Re: [PATCH 1/2] mm/percpu: Preserve NOFS/NOIO scope during chunk create and populate
In reply to Kaitao ChengQuoted 24 lines · Kaitao Cheng
Quoted 9 lines · Michal Hocko
Quoted 3 lines · Kaitao Cheng
However, if we revert
9a5b183941b, it seems that all of these issues would be resolved. The only downside is that the failure rate ofpcpu_alloc_noprof()allocations may increase, which might be acceptable.That has practical impact on some versions of iscsid which do not have
PR_SET_IO_FLUSHER. And maybe some more so I would rather not revert based on a theoretical concerns which I believe is the case here.Based on the previous discussion, I think we have a way to address most of the concurrency issues around percpu allocation.
However, there still seems to be one remaining case that I do not yet have a good way to solve. For example:
Thread A calls
pcpu_alloc_noprof()withGFP_KERNELand takespcpu_alloc_mutex. Since the internal allocation is not constrained by NOFS, it may enter FS reclaim while still holdingpcpu_alloc_mutex, creating a dependency like:
Quoted 7 lines
pcpu_alloc_mutex->fs_reclaim-> FS lock
At the same time, Thread B may already hold an FS lock and then callpcpu_alloc_noprof()withGFP_NOFS. It will try to acquirepcpu_alloc_mutexand block, creating the reverse dependency:FS lock ->
pcpu_alloc_mutex
This can still form a potential deadlock cycle.
Correct.
Quoted 3 lines
Does anyone have a good suggestion for how to handle this remaining case? Or should we simply treat all GFP_KERNEL/GFP_NOFS allocation behavior in
pcpu_alloc_noprof()asGFP_NOIO?
Yes, weakening the reclaim context would work around these dependencies. This seems like a viable option as long as pcp allocations are not in latency sensitive paths and stalling them under memory pressure is acceptable. My insight into pcp allocator users is very limited so I cannot really make any judgment call here.
Signature · 2 lines
Michal Hocko SUSE Labs
Re: [PATCH 1/2] mm/percpu: Preserve NOFS/NOIO scope during chunk create and populate
In reply to Dennis ZhouOn Sat, May 30, 2026 at 03:32:19PM +0200, Dennis Zhou wrote:
<snip>
Quoted 8 lines
Quoted 7 lines
I’ll try to do it.
Following your suggestion, including in [PATCH 2/2], I will also try a different approach and fix the issue by reducing the scope of the
pcpu_alloc_mutexcritical section.
Quoted 5 lines
No please don't. The point of the percpu mutex is to ensure that only one person is ever possibly creating a new chunk. If you drop the mutex, then you have to deal with concurrent callers when available percpu memory is low. Percpu memory is expensive and unmovable so the cost is in the control plane to avoid excess fragmentation.
And so is slab memory, yet this is not a problem. We shouldn't keep problematic locks just because they may aid fragmentation slightly.
Signature · 1 line
Pedro
PATCH 2/2 mm/percpu: Avoid pcpu_alloc_mutex recursion from reclaim
In reply to Kaitao ChengFrom: Kaitao Cheng <[email protected]>
pcpu_alloc_noprof() takes pcpu_alloc_mutex for sleepable allocations so that it can create chunks and populate backing pages. If reclaim is entered while that mutex is already held, and reclaim reaches a path which allocates percpu memory, the nested allocation can try to take pcpu_alloc_mutex again.
That creates a reclaim recursion dependency:
pcpu_alloc_noprof(GFP_KERNEL)
mutex_lock(&pcpu_alloc_mutex)
reclaim
pcpu_alloc_noprof(GFP_NOIO/GFP_NOFS)
mutex_lock(&pcpu_alloc_mutex)
Avoid this by treating percpu allocations from reclaim context as atomic. Such allocations may still be served from already available and populated areas, but they must not enter the mutex-protected slow path or create new chunks. If no space is available, fail the allocation and let the normal balance work handle replenishment outside reclaim.
Update the function comment to describe that reclaim context allocations are atomic regardless of whether the supplied GFP mask would otherwise allow blocking.
This patch is a preventive fix. There may not currently be any path that calls pcpu_alloc_noprof(GFP_NOIO/GFP_NOFS) from direct reclaim context.
--- mm/percpu.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-)
mm/percpu.c +9 -4
@@ -1803,9 +1803,9 @@ static void pcpu_memalloc_scope_restore(gfp_t gfp, unsigned int flags)* @gfp: allocation flags * * Allocate percpu area of @size bytes aligned at @align. If @gfp doesn't -* contain %GFP_KERNEL, the allocation is atomic. If @gfp has __GFP_NOWARN -* then no warning will be triggered on invalid or failed allocation -* requests. +* allow blocking, or if allocation is requested from reclaim context, the +* allocation is atomic. If @gfp has __GFP_NOWARN then no warning will be +* triggered on invalid or failed allocation requests. * * RETURNS: * Percpu pointer to the allocated area on success, NULL on failure.
@@ -1828,7 +1828,12 @@ void __percpu *pcpu_alloc_noprof(size_t size, size_t align, bool reserved, gfp =current_gfp_context (gfp);/* whitelisted flags that can be passed to the backing allocators */ pcpu_gfp = gfp & (GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN); - is_atomic = !gfpflags_allow_blocking (gfp); +/* +* Reclaim can be entered while pcpu_alloc_mutex is already held by +* another percpu allocation. Avoid recursing back into the mutex from +* reclaim; best-effort allocations from already populated areas are OK. +*/ + is_atomic = !gfpflags_allow_blocking (gfp) || current->reclaim_state; do_warn = !(gfp & __GFP_NOWARN);/*
Signature · 1 line
2.50.1 (Apple Git-155)
Re: [PATCH 2/2] mm/percpu: Avoid pcpu_alloc_mutex recursion from reclaim
Quoted 22 lines · Kaitao Cheng
From: Kaitao Cheng <[email protected]>
pcpu_alloc_noprof()takespcpu_alloc_mutexfor sleepable allocations so that it can create chunks and populate backing pages. If reclaim is entered while that mutex is already held, and reclaim reaches a path which allocates percpu memory, the nested allocation can try to takepcpu_alloc_mutexagain.That creates a reclaim recursion dependency:
pcpu_alloc_noprof(GFP_KERNEL)
mutex_lock(&pcpu_alloc_mutex)
reclaim
pcpu_alloc_noprof(GFP_NOIO/GFP_NOFS)
mutex_lock(&pcpu_alloc_mutex)Avoid this by treating percpu allocations from reclaim context as atomic. Such allocations may still be served from already available and populated areas, but they must not enter the mutex-protected slow path or create new chunks. If no space is available, fail the allocation and let the normal balance work handle replenishment outside reclaim.
Quoted 6 lines
Update the function comment to describe that reclaim context allocations are atomic regardless of whether the supplied GFP mask would otherwise allow blocking.
This patch is a preventive fix. There may not currently be any path that calls
pcpu_alloc_noprof(GFP_NOIO/GFP_NOFS)from direct reclaim context.
I don't like this. The proper way of fixing this would probably be to release pcpu_alloc_mutex (or not have it in the first place!) while you're allocating memory.
Quoted 25 lines
Fixes 9a5b183941b5 ("mm, percpu: do not consider sleepable allocations atomic")Signed-off-by Kaitao Cheng <[email protected]>--- mm/percpu.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-)mm/percpu.c +3
-3@@ -1803,9 +1803,9 @@ static void pcpu_memalloc_scope_restore(gfp_t gfp, unsigned int flags)* @gfp: allocation flags * * Allocate percpu area of @size bytes aligned at @align. If @gfp doesn't -* contain %GFP_KERNEL, the allocation is atomic. If @gfp has __GFP_NOWARN -* then no warning will be triggered on invalid or failed allocation -* requests. +* allow blocking, or if allocation is requested from reclaim context, the +* allocation is atomic. If @gfp has __GFP_NOWARN then no warning will be +* triggered on invalid or failed allocation requests. * * RETURNS: * Percpu pointer to the allocated area on success, NULL on failure.
mm/percpu.c void __percpu *pcpu_alloc_noprof(size_t size, size_t align, bool reserved, · line 1835
Quoted 9 lines
mm/percpu.c +5
-1@@ -1828,7 +1828,12 @@ void __percpu *pcpu_alloc_noprof(size_t size, size_t align, bool reserved, gfp =current_gfp_context (gfp);/* whitelisted flags that can be passed to the backing allocators */ pcpu_gfp = gfp & (GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN); - is_atomic = !gfpflags_allow_blocking (gfp); +/* +* Reclaim can be entered while pcpu_alloc_mutex is already held by +* another percpu allocation. Avoid recursing back into the mutex from +* reclaim; best-effort allocations from already populated areas are OK. +*/
since this is an entirely theoretical issue:
/* Reclaim paths should not be hitting the percpu allocator, for now */ if (WARN_ON_ONCE(current->reclaim_state)) return NULL;
But that's just my 2c.
Quoted 7 lines
mm/percpu.c +1
-0+ is_atomic = !gfpflags_allow_blocking (gfp) || current->reclaim_state; do_warn = !(gfp & __GFP_NOWARN);/* Signature · 1 line
2.50.1 (Apple Git-155)
Signature · 1 line
Pedro
Andrew Morton
maintainerQuoted 23 lines · On Thu, 28 May 2026 21:29:15 +0800 Kaitao Cheng <[email protected]> wrote:
Commit
9a5b183941b5("mm, percpu: do not consider sleepable allocations atomic") allowedGFP_NOFSandGFP_NOIOpercpu allocations to usepcpu_alloc_mutexand the chunk creation slow path. This restored the allocation capability that was lost when those constrained allocations were treated as atomic, but it also opens two possible reclaim recursion problems.The first problem is that the create and populate slow paths do not fully preserve the caller's allocation constraints.
pcpu_alloc_noprof()derivespcpu_gfpfrom the caller supplied GFP mask and passes it to the backing page allocator. However,pcpu_create_chunk()callspcpu_get_vm_areas(), and population can allocate temporary metadata or page tables while mapping backing pages. Those internal allocations can useGFP_KERNEL. A caller usingGFP_NOFSorGFP_NOIOcan therefore still enter unconstrained FS or IO reclaim while holdingpcpu_alloc_mutex. This defeats the caller's allocation context.The second problem is a possible
pcpu_alloc_mutexrecursion from reclaim. If reclaim is entered whilepcpu_alloc_mutexis already held, and reclaim reaches a path which allocates percpu memory withGFP_NOFSorGFP_NOIO, the nested allocation can now try to takepcpu_alloc_mutexagain because9a5b183941b5no longer treats those masks as atomic.
Quoted 4 lines
Another possible way to avoid these issues is to revert
9a5b183941b5. However, that would also bring back the premature allocation failures for sleepable GFP_NOFS/GFP_NOIO percpu users that9a5b183941b5was intended to fix.
Thanks.
9a5b183941b5 has been in there for a year. How are you observing/triggering this bug and what are the userspace-visible effects?
We might choose to backport fixes into -stable kernels, but this additional info is needed to make that determination.
Andrew Morton
maintainerQuoted 23 lines · On Thu, 28 May 2026 21:29:15 +0800 Kaitao Cheng <[email protected]> wrote:
Commit
9a5b183941b5("mm, percpu: do not consider sleepable allocations atomic") allowedGFP_NOFSandGFP_NOIOpercpu allocations to usepcpu_alloc_mutexand the chunk creation slow path. This restored the allocation capability that was lost when those constrained allocations were treated as atomic, but it also opens two possible reclaim recursion problems.The first problem is that the create and populate slow paths do not fully preserve the caller's allocation constraints.
pcpu_alloc_noprof()derivespcpu_gfpfrom the caller supplied GFP mask and passes it to the backing page allocator. However,pcpu_create_chunk()callspcpu_get_vm_areas(), and population can allocate temporary metadata or page tables while mapping backing pages. Those internal allocations can useGFP_KERNEL. A caller usingGFP_NOFSorGFP_NOIOcan therefore still enter unconstrained FS or IO reclaim while holdingpcpu_alloc_mutex. This defeats the caller's allocation context.The second problem is a possible
pcpu_alloc_mutexrecursion from reclaim. If reclaim is entered whilepcpu_alloc_mutexis already held, and reclaim reaches a path which allocates percpu memory withGFP_NOFSorGFP_NOIO, the nested allocation can now try to takepcpu_alloc_mutexagain because9a5b183941b5no longer treats those masks as atomic.
Quoted 4 lines
Another possible way to avoid these issues is to revert
9a5b183941b5. However, that would also bring back the premature allocation failures for sleepable GFP_NOFS/GFP_NOIO percpu users that9a5b183941b5was intended to fix.
AI review might have found a couple of pre-existing issues:
https://sashiko.dev/#/patchset/[email protected]