Skip to contentSkip to conversation

mm/percpu: Fix possible NOFS/NOIO reclaim recursion

PATCH Superseded Kaitao Cheng 6 participants 18 messages May 28 – Jun 2

Subsystems (2)

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

cover mm/percpu: Fix possible NOFS/NOIO reclaim recursion

Commit 9a5b183941b5 ("mm, percpu: do not consider sleepable allocations atomic") allowed GFP_NOFS and GFP_NOIO percpu allocations to use pcpu_alloc_mutex and the chunk creation slow path. This restored the allocation capability that was lost when those constrained allocations were treated as atomic, but it also opens two possible reclaim recursion problems.

The first problem is that the create and populate slow paths do not fully preserve the caller's allocation constraints. pcpu_alloc_noprof() derives pcpu_gfp from the caller supplied GFP mask and passes it to the backing page allocator. However, pcpu_create_chunk() calls pcpu_get_vm_areas(), and population can allocate temporary metadata or page tables while mapping backing pages. Those internal allocations can use GFP_KERNEL. A caller using GFP_NOFS or GFP_NOIO can therefore still enter unconstrained FS or IO reclaim while holding pcpu_alloc_mutex. This defeats the caller's allocation context.

The second problem is a possible pcpu_alloc_mutex recursion from reclaim. If reclaim is entered while pcpu_alloc_mutex is already held, and reclaim reaches a path which allocates percpu memory with GFP_NOFS or GFP_NOIO, the nested allocation can now try to take pcpu_alloc_mutex again because 9a5b183941b5 no longer treats those masks as atomic.

Another possible way to avoid these issues is to revert 9a5b183941b5. However, that would also bring back the premature allocation failures for sleepable GFP_NOFS/GFP_NOIO percpu users that 9a5b183941b5 was intended to fix.

Kaitao Cheng (2):
mm/percpu: Preserve NOFS/NOIO scope during chunk create and populate mm/percpu: Avoid pcpu_alloc_mutex recursion from reclaim

 mm/percpu.c | 39 +++++++++++++++++++++++++++++++++++----
 1 file changed, 35 insertions(+), 4 deletions(-)
Signature · 1 line
2.50.1 (Apple Git-155)
#1 lore raw

PATCH 1/2 mm/percpu: Preserve NOFS/NOIO scope during chunk create and populate

From: Kaitao Cheng <[email protected]>

pcpu_alloc_noprof() derives pcpu_gfp from the caller supplied GFP mask and passes it to the backing percpu allocators. This preserves GFP_NOFS and GFP_NOIO for pcpu_alloc_pages() and for the initial pcpu_chunk allocation.

However, the chunk creation and population slow paths also call helpers which do not take a GFP mask and perform internal allocations with GFP_KERNEL. For example, pcpu_create_chunk() calls pcpu_get_vm_areas(), and population can allocate temporary metadata or page tables while mapping backing pages. As a result, a caller which explicitly uses GFP_NOFS or GFP_NOIO can still enter FS or IO reclaim while creating or populating a percpu chunk.

This is problematic for callers which use GFP_NOFS or GFP_NOIO because they are already holding filesystem or IO-path locks. If free chunks are exhausted, the percpu allocation can take pcpu_alloc_mutex and then enter unconstrained reclaim from these internal allocations, defeating the caller's allocation context and potentially recreating reclaim lock dependencies.

Wrap chunk creation and population in a scoped NOIO or NOFS context when pcpu_gfp has the corresponding constraints. Leave ordinary GFP_KERNEL allocations unchanged so they retain full reclaim capability.

Fixes 9a5b183941b5 ("mm, percpu: do not consider sleepable allocations atomic")
---
 mm/percpu.c | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)
mm/percpu.c +26 -0
index 71a85d7245c7..1bb38467390b 100644
@@ -1778,6 +1778,23 @@ static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t s
 }
 #endif
 
+static unsigned int pcpu_memalloc_scope_save(gfp_t gfp)
+{
+	if (!(gfp & __GFP_IO))
+		return memalloc_noio_save();
+	if (!(gfp & __GFP_FS))
+		return memalloc_nofs_save();
+	return 0;
+}
+
+static void pcpu_memalloc_scope_restore(gfp_t gfp, unsigned int flags)
+{
+	if (!(gfp & __GFP_IO))
+		memalloc_noio_restore(flags);
+	else if (!(gfp & __GFP_FS))
+		memalloc_nofs_restore(flags);
+}
+
 /**
  * pcpu_alloc - the percpu allocator
  * @size: size of area to allocate in bytes
@@ -1901,7 +1918,12 @@ void __percpu *pcpu_alloc_noprof(size_t size, size_t align, bool reserved,
 
 	/* No space left.  Create a new chunk. */
 	if (list_empty(&pcpu_chunk_lists[pcpu_free_slot])) {
+		unsigned int pcpu_scope;
+
+		pcpu_scope = pcpu_memalloc_scope_save(pcpu_gfp);
 		chunk = pcpu_create_chunk(pcpu_gfp);
+		pcpu_memalloc_scope_restore(pcpu_gfp, pcpu_scope);
+
 		if (!chunk) {
 			err = "failed to allocate new chunk";
 			goto fail;
@@ -1931,9 +1953,13 @@ void __percpu *pcpu_alloc_noprof(size_t size, size_t align, bool reserved,
 		page_end = PFN_UP(off + size);
 
 		for_each_clear_bitrange_from(rs, re, chunk->populated, page_end) {
+			unsigned int pcpu_scope;
+
 			WARN_ON(chunk->immutable);
 
+			pcpu_scope = pcpu_memalloc_scope_save(pcpu_gfp);
 			ret = pcpu_populate_chunk(chunk, rs, re, pcpu_gfp);
+			pcpu_memalloc_scope_restore(pcpu_gfp, pcpu_scope);
 
 			spin_lock_irqsave(&pcpu_lock, flags);
 			if (ret) {
Signature · 1 line
2.50.1 (Apple Git-155)
#2 lore raw

Re: [PATCH 1/2] mm/percpu: Preserve NOFS/NOIO scope during chunk create and populate

Quoted 21 lines · Kaitao Cheng

From: Kaitao Cheng <[email protected]>

pcpu_alloc_noprof() derives pcpu_gfp from the caller supplied GFP mask and passes it to the backing percpu allocators. This preserves GFP_NOFS and GFP_NOIO for pcpu_alloc_pages() and for the initial pcpu_chunk allocation.

However, the chunk creation and population slow paths also call helpers which do not take a GFP mask and perform internal allocations with GFP_KERNEL. For example, pcpu_create_chunk() calls pcpu_get_vm_areas(), and population can allocate temporary metadata or page tables while mapping backing pages. As a result, a caller which explicitly uses GFP_NOFS or GFP_NOIO can still enter FS or IO reclaim while creating or populating a percpu chunk.

This is problematic for callers which use GFP_NOFS or GFP_NOIO because they are already holding filesystem or IO-path locks. If free chunks are exhausted, the percpu allocation can take pcpu_alloc_mutex and then enter unconstrained reclaim from these internal allocations, defeating the caller's allocation context and potentially recreating reclaim lock dependencies.

Quoted 5 lines

Wrap chunk creation and population in a scoped NOIO or NOFS context when pcpu_gfp has the corresponding constraints. Leave ordinary GFP_KERNEL allocations unchanged so they retain full reclaim capability.

Fixes 9a5b183941b5 ("mm, percpu: do not consider sleepable allocations atomic")

I assume you _did not_ observe this in production? As in no reclaim path should be insane^W daring enough to do pcpu allocations?

Quoted 19 lines
---
 mm/percpu.c | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)
mm/percpu.c +6 -0
index 71a85d7245c7..1bb38467390b 100644
@@ -1778,6 +1778,23 @@ static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t s
 }
 #endif
 
+static unsigned int pcpu_memalloc_scope_save(gfp_t gfp)
+{
+	if (!(gfp & __GFP_IO))
+		return memalloc_noio_save();
+	if (!(gfp & __GFP_FS))
+		return memalloc_nofs_save();

mm/percpu.c static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t s · line 1796

Quoted 10 lines
mm/percpu.c +10 -0
+	return 0;
+}
+
+static void pcpu_memalloc_scope_restore(gfp_t gfp, unsigned int flags)
+{
+	if (!(gfp & __GFP_IO))
+		memalloc_noio_restore(flags);
+	else if (!(gfp & __GFP_FS))
+		memalloc_nofs_restore(flags);
+}

I disagree with this. We already have gfp flags, they're already passed to pcpu_create_chunk() and pcpu_populate_chunk(). It's their job to respect the gfp flags and
Do The Right Thing(tm). Can you fix the problematic places? It seems like it's mostly the vmalloc backend that's problematic.

Quoted 34 lines
mm/percpu.c +10 -0
+
 /**
  * pcpu_alloc - the percpu allocator
  * @size: size of area to allocate in bytes
@@ -1901,7 +1918,12 @@ void __percpu *pcpu_alloc_noprof(size_t size, size_t align, bool reserved,
 
 	/* No space left.  Create a new chunk. */
 	if (list_empty(&pcpu_chunk_lists[pcpu_free_slot])) {
+		unsigned int pcpu_scope;
+
+		pcpu_scope = pcpu_memalloc_scope_save(pcpu_gfp);
 		chunk = pcpu_create_chunk(pcpu_gfp);
+		pcpu_memalloc_scope_restore(pcpu_gfp, pcpu_scope);
+
 		if (!chunk) {
 			err = "failed to allocate new chunk";
 			goto fail;
@@ -1931,9 +1953,13 @@ void __percpu *pcpu_alloc_noprof(size_t size, size_t align, bool reserved,
 		page_end = PFN_UP(off + size);
 
 		for_each_clear_bitrange_from(rs, re, chunk->populated, page_end) {
+			unsigned int pcpu_scope;
+
 			WARN_ON(chunk->immutable);
 
+			pcpu_scope = pcpu_memalloc_scope_save(pcpu_gfp);
 			ret = pcpu_populate_chunk(chunk, rs, re, pcpu_gfp);
+			pcpu_memalloc_scope_restore(pcpu_gfp, pcpu_scope);
 
 			spin_lock_irqsave(&pcpu_lock, flags);
 			if (ret) {
Signature · 1 line
2.50.1 (Apple Git-155)
Signature · 1 line
Pedro
#3 lore raw

Re: [PATCH 1/2] mm/percpu: Preserve NOFS/NOIO scope during chunk create and populate

Quoted 22 lines · Pedro Falcato
Quoted 21 lines · Kaitao Cheng

From: Kaitao Cheng <[email protected]>

pcpu_alloc_noprof() derives pcpu_gfp from the caller supplied GFP mask and passes it to the backing percpu allocators. This preserves GFP_NOFS and GFP_NOIO for pcpu_alloc_pages() and for the initial pcpu_chunk allocation.

However, the chunk creation and population slow paths also call helpers which do not take a GFP mask and perform internal allocations with GFP_KERNEL. For example, pcpu_create_chunk() calls pcpu_get_vm_areas(), and population can allocate temporary metadata or page tables while mapping backing pages. As a result, a caller which explicitly uses GFP_NOFS or GFP_NOIO can still enter FS or IO reclaim while creating or populating a percpu chunk.

This is problematic for callers which use GFP_NOFS or GFP_NOIO because they are already holding filesystem or IO-path locks. If free chunks are exhausted, the percpu allocation can take pcpu_alloc_mutex and then enter unconstrained reclaim from these internal allocations, defeating the caller's allocation context and potentially recreating reclaim lock dependencies.

Quoted 8 lines
Quoted 5 lines

Wrap chunk creation and population in a scoped NOIO or NOFS context when pcpu_gfp has the corresponding constraints. Leave ordinary GFP_KERNEL allocations unchanged so they retain full reclaim capability.

Fixes 9a5b183941b5 ("mm, percpu: do not consider sleepable allocations atomic")

I assume you _did not_ observe this in production? As in no reclaim path should be insane^W daring enough to do pcpu allocations?

Oops, I mixed my issues up. This is purely a GFP flags issue. A quick "git grep alloc_percpu_gfp" shows that the vast majority (all?) callers are using some combination of GFP_KERNEL or GFP_ATOMIC + other GFP flags, but no NOFS or NOIO as far as I can see. So you probably did not observe this?

Quoted 73 lines
Quoted 20 lines
---
 mm/percpu.c | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)
mm/percpu.c +6 -0
index 71a85d7245c7..1bb38467390b 100644
@@ -1778,6 +1778,23 @@ static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t s
 }
 #endif
 
+static unsigned int pcpu_memalloc_scope_save(gfp_t gfp)
+{
+	if (!(gfp & __GFP_IO))
+		return memalloc_noio_save();
+	if (!(gfp & __GFP_FS))
+		return memalloc_nofs_save();

mm/percpu.c static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t s · line 1796

Quoted 10 lines
mm/percpu.c +10 -0
+	return 0;
+}
+
+static void pcpu_memalloc_scope_restore(gfp_t gfp, unsigned int flags)
+{
+	if (!(gfp & __GFP_IO))
+		memalloc_noio_restore(flags);
+	else if (!(gfp & __GFP_FS))
+		memalloc_nofs_restore(flags);
+}

I disagree with this. We already have gfp flags, they're already passed to pcpu_create_chunk() and pcpu_populate_chunk(). It's their job to respect the gfp flags and
Do The Right Thing(tm). Can you fix the problematic places? It seems like it's mostly the vmalloc backend that's problematic.

Quoted 34 lines
mm/percpu.c +10 -0
+
 /**
  * pcpu_alloc - the percpu allocator
  * @size: size of area to allocate in bytes
@@ -1901,7 +1918,12 @@ void __percpu *pcpu_alloc_noprof(size_t size, size_t align, bool reserved,
 
 	/* No space left.  Create a new chunk. */
 	if (list_empty(&pcpu_chunk_lists[pcpu_free_slot])) {
+		unsigned int pcpu_scope;
+
+		pcpu_scope = pcpu_memalloc_scope_save(pcpu_gfp);
 		chunk = pcpu_create_chunk(pcpu_gfp);
+		pcpu_memalloc_scope_restore(pcpu_gfp, pcpu_scope);
+
 		if (!chunk) {
 			err = "failed to allocate new chunk";
 			goto fail;
@@ -1931,9 +1953,13 @@ void __percpu *pcpu_alloc_noprof(size_t size, size_t align, bool reserved,
 		page_end = PFN_UP(off + size);
 
 		for_each_clear_bitrange_from(rs, re, chunk->populated, page_end) {
+			unsigned int pcpu_scope;
+
 			WARN_ON(chunk->immutable);
 
+			pcpu_scope = pcpu_memalloc_scope_save(pcpu_gfp);
 			ret = pcpu_populate_chunk(chunk, rs, re, pcpu_gfp);
+			pcpu_memalloc_scope_restore(pcpu_gfp, pcpu_scope);
 
 			spin_lock_irqsave(&pcpu_lock, flags);
 			if (ret) {
Signature · 1 line
2.50.1 (Apple Git-155)
Signature · 1 line
Pedro
Signature · 1 line
Pedro
#4 lore raw

Re: [PATCH 1/2] mm/percpu: Preserve NOFS/NOIO scope during chunk create and populate

Quoted 29 lines · Pedro Falcato
Quoted 28 lines · Pedro Falcato
Quoted 26 lines · Kaitao Cheng

From: Kaitao Cheng <[email protected]>

pcpu_alloc_noprof() derives pcpu_gfp from the caller supplied GFP mask and passes it to the backing percpu allocators. This preserves GFP_NOFS and GFP_NOIO for pcpu_alloc_pages() and for the initial pcpu_chunk allocation.

However, the chunk creation and population slow paths also call helpers which do not take a GFP mask and perform internal allocations with GFP_KERNEL. For example, pcpu_create_chunk() calls pcpu_get_vm_areas(), and population can allocate temporary metadata or page tables while mapping backing pages. As a result, a caller which explicitly uses GFP_NOFS or GFP_NOIO can still enter FS or IO reclaim while creating or populating a percpu chunk.

This is problematic for callers which use GFP_NOFS or GFP_NOIO because they are already holding filesystem or IO-path locks. If free chunks are exhausted, the percpu allocation can take pcpu_alloc_mutex and then enter unconstrained reclaim from these internal allocations, defeating the caller's allocation context and potentially recreating reclaim lock dependencies.

Wrap chunk creation and population in a scoped NOIO or NOFS context when pcpu_gfp has the corresponding constraints. Leave ordinary GFP_KERNEL allocations unchanged so they retain full reclaim capability.

Fixes 9a5b183941b5 ("mm, percpu: do not consider sleepable allocations atomic")
Quoted 7 lines
Quoted 2 lines

I assume you _did not_ observe this in production? As in no reclaim path should be insane^W daring enough to do pcpu allocations?

Oops, I mixed my issues up. This is purely a GFP flags issue. A quick "git grep alloc_percpu_gfp" shows that the vast majority (all?) callers are using some combination of GFP_KERNEL or GFP_ATOMIC + other GFP flags, but no NOFS or NOIO as far as I can see. So you probably did not observe this?

Right, this issue has not been observed in production. It came from a question raised by AI code review, and after carefully reading the code, I found that there are indeed some synchronization concerns.

Here is one example of the scenario [PATCH 1/2] is trying to address:

blk-cgroup after 5d726c4dbeed ("blk-cgroup: fix possible deadlock while configuring policy"). blkg_conf_prep() now serializes against blkcg_deactivate_policy() with q->blkcg_mutex, and blkg_alloc() was changed to GFP_NOIO for that reason:

CPU0: blkg_conf_prep()

    mutex_lock(q->blkcg_mutex)
    blkg_alloc(..., GFP_NOIO)
      alloc_percpu_gfp(..., GFP_NOIO)
        -> if percpu chunks are exhausted, chunk create/populate may do
           internal GFP_KERNEL allocations
        -> direct reclaim / writeback can issue IO to this queue
        -> IO waits because the queue is frozen

CPU1: blkcg_deactivate_policy()

    blk_mq_freeze_queue(q)
    mutex_lock(q->blkcg_mutex)
      -> waits for CPU0
    ... unfreeze only happens after q->blkcg_mutex is acquired/released

So the concern is that the caller deliberately uses GFP_NOIO because it may hold a lock which can be acquired after queue freeze, but the percpu slow path can temporarily lose that allocation context. The failure requires the slow path (new chunk creation or population), so it is not expected to be common.

Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
---
 mm/percpu.c | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)
mm/percpu.c +13 -0
index 71a85d7245c7..1bb38467390b 100644
@@ -1778,6 +1778,23 @@ static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t s
 }
 #endif
 
+static unsigned int pcpu_memalloc_scope_save(gfp_t gfp)
+{
+	if (!(gfp & __GFP_IO))
+		return memalloc_noio_save();
+	if (!(gfp & __GFP_FS))
+		return memalloc_nofs_save();
+	return 0;
+}
+
+static void pcpu_memalloc_scope_restore(gfp_t gfp, unsigned int flags)
+{
+	if (!(gfp & __GFP_IO))
+		memalloc_noio_restore(flags);
Quoted 8 lines
Quoted 8 lines

mm/percpu.c static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t s · line 1796

Quoted 3 lines
mm/percpu.c +3 -0
+	else if (!(gfp & __GFP_FS))
+		memalloc_nofs_restore(flags);
+}

I disagree with this. We already have gfp flags, they're already passed to pcpu_create_chunk() and pcpu_populate_chunk(). It's their job to respect the gfp flags and
Do The Right Thing(tm). Can you fix the problematic places? It seems like it's mostly the vmalloc backend that's problematic.

I’ll try to do it.

Following your suggestion, including in [PATCH 2/2], I will also try a different approach and fix the issue by reducing the scope of the pcpu_alloc_mutex critical section.

Quoted 33 lines
Quoted 33 lines
Quoted 33 lines
mm/percpu.c +9 -0
 /**
  * pcpu_alloc - the percpu allocator
  * @size: size of area to allocate in bytes
@@ -1901,7 +1918,12 @@ void __percpu *pcpu_alloc_noprof(size_t size, size_t align, bool reserved,
 
 	/* No space left.  Create a new chunk. */
 	if (list_empty(&pcpu_chunk_lists[pcpu_free_slot])) {
+		unsigned int pcpu_scope;
+
+		pcpu_scope = pcpu_memalloc_scope_save(pcpu_gfp);
 		chunk = pcpu_create_chunk(pcpu_gfp);
+		pcpu_memalloc_scope_restore(pcpu_gfp, pcpu_scope);
+
 		if (!chunk) {
 			err = "failed to allocate new chunk";
 			goto fail;
@@ -1931,9 +1953,13 @@ void __percpu *pcpu_alloc_noprof(size_t size, size_t align, bool reserved,
 		page_end = PFN_UP(off + size);
 
 		for_each_clear_bitrange_from(rs, re, chunk->populated, page_end) {
+			unsigned int pcpu_scope;
+
 			WARN_ON(chunk->immutable);
 
+			pcpu_scope = pcpu_memalloc_scope_save(pcpu_gfp);
 			ret = pcpu_populate_chunk(chunk, rs, re, pcpu_gfp);
+			pcpu_memalloc_scope_restore(pcpu_gfp, pcpu_scope);
 
 			spin_lock_irqsave(&pcpu_lock, flags);
 			if (ret) {
Signature · 1 line
2.50.1 (Apple Git-155)
Signature · 2 lines
Thanks
Kaitao Cheng
maintainer
#5 lore raw

Re: [PATCH 1/2] mm/percpu: Preserve NOFS/NOIO scope during chunk create and populate

Hello,

Quoted 64 lines · Kaitao Cheng
Quoted 29 lines · Pedro Falcato
Quoted 28 lines · Pedro Falcato
Quoted 26 lines · Kaitao Cheng

From: Kaitao Cheng <[email protected]>

pcpu_alloc_noprof() derives pcpu_gfp from the caller supplied GFP mask and passes it to the backing percpu allocators. This preserves GFP_NOFS and GFP_NOIO for pcpu_alloc_pages() and for the initial pcpu_chunk allocation.

However, the chunk creation and population slow paths also call helpers which do not take a GFP mask and perform internal allocations with GFP_KERNEL. For example, pcpu_create_chunk() calls pcpu_get_vm_areas(), and population can allocate temporary metadata or page tables while mapping backing pages. As a result, a caller which explicitly uses GFP_NOFS or GFP_NOIO can still enter FS or IO reclaim while creating or populating a percpu chunk.

This is problematic for callers which use GFP_NOFS or GFP_NOIO because they are already holding filesystem or IO-path locks. If free chunks are exhausted, the percpu allocation can take pcpu_alloc_mutex and then enter unconstrained reclaim from these internal allocations, defeating the caller's allocation context and potentially recreating reclaim lock dependencies.

Wrap chunk creation and population in a scoped NOIO or NOFS context when pcpu_gfp has the corresponding constraints. Leave ordinary GFP_KERNEL allocations unchanged so they retain full reclaim capability.

Fixes 9a5b183941b5 ("mm, percpu: do not consider sleepable allocations atomic")
Quoted 7 lines
Quoted 2 lines

I assume you _did not_ observe this in production? As in no reclaim path should be insane^W daring enough to do pcpu allocations?

Oops, I mixed my issues up. This is purely a GFP flags issue. A quick "git grep alloc_percpu_gfp" shows that the vast majority (all?) callers are using some combination of GFP_KERNEL or GFP_ATOMIC + other GFP flags, but no NOFS or NOIO as far as I can see. So you probably did not observe this?

Right, this issue has not been observed in production. It came from a question raised by AI code review, and after carefully reading the code, I found that there are indeed some synchronization concerns.

Here is one example of the scenario [PATCH 1/2] is trying to address:

blk-cgroup after 5d726c4dbeed ("blk-cgroup: fix possible deadlock while configuring policy"). blkg_conf_prep() now serializes against blkcg_deactivate_policy() with q->blkcg_mutex, and blkg_alloc() was changed to GFP_NOIO for that reason:

CPU0: blkg_conf_prep()

    mutex_lock(q->blkcg_mutex)
    blkg_alloc(..., GFP_NOIO)
      alloc_percpu_gfp(..., GFP_NOIO)
        -> if percpu chunks are exhausted, chunk create/populate may do
           internal GFP_KERNEL allocations
        -> direct reclaim / writeback can issue IO to this queue
        -> IO waits because the queue is frozen

CPU1: blkcg_deactivate_policy()

    blk_mq_freeze_queue(q)
    mutex_lock(q->blkcg_mutex)
      -> waits for CPU0
    ... unfreeze only happens after q->blkcg_mutex is acquired/released
Quoted 5 lines

So the concern is that the caller deliberately uses GFP_NOIO because it may hold a lock which can be acquired after queue freeze, but the percpu slow path can temporarily lose that allocation context. The failure requires the slow path (new chunk creation or population), so it is not expected to be common.

This is likely just a miss for [1] where we switching to gfpflags_allow_blocking(),

This seems like it's just a miss from [1] where we switched to a less conservative approach than atomic == !GFP_KERNEL. If anything we just need to allow the additional flags through. Maybe just add GFP_NOIO and GFP_NOFS in pcpu_gfp via the whitelist.

[1] 9a5b183941b ("mm, percpu: do not consider sleepable allocations atomic")

Quoted 36 lines
Quoted 27 lines
Quoted 27 lines
Quoted 27 lines
---
 mm/percpu.c | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)
mm/percpu.c +13 -0
index 71a85d7245c7..1bb38467390b 100644
@@ -1778,6 +1778,23 @@ static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t s
 }
 #endif
 
+static unsigned int pcpu_memalloc_scope_save(gfp_t gfp)
+{
+	if (!(gfp & __GFP_IO))
+		return memalloc_noio_save();
+	if (!(gfp & __GFP_FS))
+		return memalloc_nofs_save();
+	return 0;
+}
+
+static void pcpu_memalloc_scope_restore(gfp_t gfp, unsigned int flags)
+{
+	if (!(gfp & __GFP_IO))
+		memalloc_noio_restore(flags);
Quoted 8 lines
Quoted 8 lines

mm/percpu.c static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t s · line 1796

Quoted 3 lines
mm/percpu.c +3 -0
+	else if (!(gfp & __GFP_FS))
+		memalloc_nofs_restore(flags);
+}

I disagree with this. We already have gfp flags, they're already passed to pcpu_create_chunk() and pcpu_populate_chunk(). It's their job to respect the gfp flags and
Do The Right Thing(tm). Can you fix the problematic places? It seems like it's mostly the vmalloc backend that's problematic.

Quoted 6 lines

I’ll try to do it.

Following your suggestion, including in [PATCH 2/2], I will also try a different approach and fix the issue by reducing the scope of the pcpu_alloc_mutex critical section.

No please don't. The point of the percpu mutex is to ensure that only one person is ever possibly creating a new chunk. If you drop the mutex, then you have to deal with concurrent callers when available percpu memory is low. Percpu memory is expensive and unmovable so the cost is in the control plane to avoid excess fragmentation.

Thanks,
Dennis

#6 lore raw

Re: [PATCH 1/2] mm/percpu: Preserve NOFS/NOIO scope during chunk create and populate

Quoted 125 lines · Dennis Zhou

Hello,

Quoted 64 lines · Kaitao Cheng
Quoted 29 lines · Pedro Falcato
Quoted 28 lines · Pedro Falcato
Quoted 26 lines · Kaitao Cheng

From: Kaitao Cheng <[email protected]>

pcpu_alloc_noprof() derives pcpu_gfp from the caller supplied GFP mask and passes it to the backing percpu allocators. This preserves GFP_NOFS and GFP_NOIO for pcpu_alloc_pages() and for the initial pcpu_chunk allocation.

However, the chunk creation and population slow paths also call helpers which do not take a GFP mask and perform internal allocations with GFP_KERNEL. For example, pcpu_create_chunk() calls pcpu_get_vm_areas(), and population can allocate temporary metadata or page tables while mapping backing pages. As a result, a caller which explicitly uses GFP_NOFS or GFP_NOIO can still enter FS or IO reclaim while creating or populating a percpu chunk.

This is problematic for callers which use GFP_NOFS or GFP_NOIO because they are already holding filesystem or IO-path locks. If free chunks are exhausted, the percpu allocation can take pcpu_alloc_mutex and then enter unconstrained reclaim from these internal allocations, defeating the caller's allocation context and potentially recreating reclaim lock dependencies.

Wrap chunk creation and population in a scoped NOIO or NOFS context when pcpu_gfp has the corresponding constraints. Leave ordinary GFP_KERNEL allocations unchanged so they retain full reclaim capability.

Fixes 9a5b183941b5 ("mm, percpu: do not consider sleepable allocations atomic")
Quoted 7 lines
Quoted 2 lines

I assume you _did not_ observe this in production? As in no reclaim path should be insane^W daring enough to do pcpu allocations?

Oops, I mixed my issues up. This is purely a GFP flags issue. A quick "git grep alloc_percpu_gfp" shows that the vast majority (all?) callers are using some combination of GFP_KERNEL or GFP_ATOMIC + other GFP flags, but no NOFS or NOIO as far as I can see. So you probably did not observe this?

Right, this issue has not been observed in production. It came from a question raised by AI code review, and after carefully reading the code, I found that there are indeed some synchronization concerns.

Here is one example of the scenario [PATCH 1/2] is trying to address:

blk-cgroup after 5d726c4dbeed ("blk-cgroup: fix possible deadlock while configuring policy"). blkg_conf_prep() now serializes against blkcg_deactivate_policy() with q->blkcg_mutex, and blkg_alloc() was changed to GFP_NOIO for that reason:

CPU0: blkg_conf_prep()

    mutex_lock(q->blkcg_mutex)
    blkg_alloc(..., GFP_NOIO)
      alloc_percpu_gfp(..., GFP_NOIO)
        -> if percpu chunks are exhausted, chunk create/populate may do
           internal GFP_KERNEL allocations
        -> direct reclaim / writeback can issue IO to this queue
        -> IO waits because the queue is frozen

CPU1: blkcg_deactivate_policy()

    blk_mq_freeze_queue(q)
    mutex_lock(q->blkcg_mutex)
      -> waits for CPU0
    ... unfreeze only happens after q->blkcg_mutex is acquired/released
Quoted 5 lines

So the concern is that the caller deliberately uses GFP_NOIO because it may hold a lock which can be acquired after queue freeze, but the percpu slow path can temporarily lose that allocation context. The failure requires the slow path (new chunk creation or population), so it is not expected to be common.

This is likely just a miss for [1] where we switching to gfpflags_allow_blocking(),

This seems like it's just a miss from [1] where we switched to a less conservative approach than atomic == !GFP_KERNEL. If anything we just need to allow the additional flags through. Maybe just add GFP_NOIO and GFP_NOFS in pcpu_gfp via the whitelist.

[1] 9a5b183941b ("mm, percpu: do not consider sleepable allocations atomic")

Quoted 36 lines
Quoted 27 lines
Quoted 27 lines
Quoted 27 lines
---
 mm/percpu.c | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)
mm/percpu.c +13 -0
index 71a85d7245c7..1bb38467390b 100644
@@ -1778,6 +1778,23 @@ static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t s
 }
 #endif
 
+static unsigned int pcpu_memalloc_scope_save(gfp_t gfp)
+{
+	if (!(gfp & __GFP_IO))
+		return memalloc_noio_save();
+	if (!(gfp & __GFP_FS))
+		return memalloc_nofs_save();
+	return 0;
+}
+
+static void pcpu_memalloc_scope_restore(gfp_t gfp, unsigned int flags)
+{
+	if (!(gfp & __GFP_IO))
+		memalloc_noio_restore(flags);
Quoted 8 lines
Quoted 8 lines

mm/percpu.c static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t s · line 1796

Quoted 3 lines
mm/percpu.c +3 -0
+	else if (!(gfp & __GFP_FS))
+		memalloc_nofs_restore(flags);
+}

I disagree with this. We already have gfp flags, they're already passed to pcpu_create_chunk() and pcpu_populate_chunk(). It's their job to respect the gfp flags and
Do The Right Thing(tm). Can you fix the problematic places? It seems like it's mostly the vmalloc backend that's problematic.

Quoted 6 lines

I’ll try to do it.

Following your suggestion, including in [PATCH 2/2], I will also try a different approach and fix the issue by reducing the scope of the pcpu_alloc_mutex critical section.

Quoted 6 lines

No please don't. The point of the percpu mutex is to ensure that only one person is ever possibly creating a new chunk. If you drop the mutex, then you have to deal with concurrent callers when available percpu memory is low. Percpu memory is expensive and unmovable so the cost is in the control plane to avoid excess fragmentation.

You are right. pcpu_alloc_mutex seems necessary, and there does not appear to be a good way to replace it for now.

The introduction of 9a5b183941b can lead to several synchronization issues. There are already three known cases: the two described in [PATCH 0/2], plus the one raised by AI review.

https://lore.kernel.org/all/[email protected]/
https://sashiko.dev/#/patchset/[email protected]

Since optimizing around pcpu_alloc_mutex is constrained, there does not seem to be a perfect solution that addresses all of them at the same time.

However, if we revert 9a5b183941b, it seems that all of these issues would be resolved. The only downside is that the failure rate of pcpu_alloc_noprof() allocations may increase, which might be acceptable.

I would like to hear what others think.

Signature · 2 lines
Thanks
Kaitao Cheng
#7 lore raw

Re: [PATCH 1/2] mm/percpu: Preserve NOFS/NOIO scope during chunk create and populate

Quoted 3 lines · Kaitao Cheng

However, if we revert 9a5b183941b, it seems that all of these issues would be resolved. The only downside is that the failure rate of pcpu_alloc_noprof() allocations may increase, which might be acceptable.

That has practical impact on some versions of iscsid which do not have PR_SET_IO_FLUSHER. And maybe some more so I would rather not revert based on a theoretical concerns which I believe is the case here.

Signature · 2 lines
Michal Hocko
SUSE Labs
#8 lore raw

Re: [PATCH 1/2] mm/percpu: Preserve NOFS/NOIO scope during chunk create and populate

Quoted 9 lines · Michal Hocko
Quoted 3 lines · Kaitao Cheng

However, if we revert 9a5b183941b, it seems that all of these issues would be resolved. The only downside is that the failure rate of pcpu_alloc_noprof() allocations may increase, which might be acceptable.

That has practical impact on some versions of iscsid which do not have PR_SET_IO_FLUSHER. And maybe some more so I would rather not revert based on a theoretical concerns which I believe is the case here.

Based on the previous discussion, I think we have a way to address most of the concurrency issues around percpu allocation.

However, there still seems to be one remaining case that I do not yet have a good way to solve. For example:

Thread A calls pcpu_alloc_noprof() with GFP_KERNEL and takes pcpu_alloc_mutex. Since the internal allocation is not constrained by NOFS, it may enter FS reclaim while still holding pcpu_alloc_mutex, creating a dependency like:

pcpu_alloc_mutex -> fs_reclaim -> FS lock
At the same time, Thread B may already hold an FS lock and then call pcpu_alloc_noprof() with GFP_NOFS. It will try to acquire pcpu_alloc_mutex and block, creating the reverse dependency:

FS lock -> pcpu_alloc_mutex
This can still form a potential deadlock cycle.

Does anyone have a good suggestion for how to handle this remaining case? Or should we simply treat all GFP_KERNEL/GFP_NOFS allocation behavior in pcpu_alloc_noprof() as GFP_NOIO?

If there is no clear solution for now, would it be acceptable to first fix some of the issues introduced by commit 9a5b183941b, and leave this remaining case as a pre-existing historical issue to be handled separately later?

Signature · 2 lines
Thanks
Kaitao Cheng
#9 lore raw

Re: [PATCH 1/2] mm/percpu: Preserve NOFS/NOIO scope during chunk create and populate

Quoted 32 lines · Kaitao Cheng
Quoted 9 lines · Michal Hocko
Quoted 3 lines · Kaitao Cheng

However, if we revert 9a5b183941b, it seems that all of these issues would be resolved. The only downside is that the failure rate of pcpu_alloc_noprof() allocations may increase, which might be acceptable.

That has practical impact on some versions of iscsid which do not have PR_SET_IO_FLUSHER. And maybe some more so I would rather not revert based on a theoretical concerns which I believe is the case here.

Based on the previous discussion, I think we have a way to address most of the concurrency issues around percpu allocation.

However, there still seems to be one remaining case that I do not yet have a good way to solve. For example:

Thread A calls pcpu_alloc_noprof() with GFP_KERNEL and takes pcpu_alloc_mutex. Since the internal allocation is not constrained by NOFS, it may enter FS reclaim while still holding pcpu_alloc_mutex, creating a dependency like:

pcpu_alloc_mutex -> fs_reclaim -> FS lock
At the same time, Thread B may already hold an FS lock and then call pcpu_alloc_noprof() with GFP_NOFS. It will try to acquire pcpu_alloc_mutex and block, creating the reverse dependency:

FS lock -> pcpu_alloc_mutex
This can still form a potential deadlock cycle.

Quoted 8 lines

Does anyone have a good suggestion for how to handle this remaining case? Or should we simply treat all GFP_KERNEL/GFP_NOFS allocation behavior in pcpu_alloc_noprof() as GFP_NOIO?

If there is no clear solution for now, would it be acceptable to first fix some of the issues introduced by commit 9a5b183941b, and leave this remaining case as a pre-existing historical issue to be handled separately later?

We don't need to solve any issues that are only theoretical and based on scenarios that nobody sane should be doing, i.e. Pedro already pointed out "As in no reclaim path should be insane^W daring enough to do pcpu allocations?"

If anyone would (start to) do that, we would likely have lockdep reports from the testing bots, which warn that the scenario can now exist, even before it results in an actual deadlock.

Elsewhere Pedro said "The proper way of fixing this would probably be to release pcpu_alloc_mutex (or not have it in the first place!) while you're allocating memory."

Such a refactoring might be worth it (if it's feasible to do cleanly and doesn't come with downsides) just to eliminate these lock dependencies properly for good. Patching over individual theoretical issues is IMHO not worth it.

#10 lore raw

Re: [PATCH 1/2] mm/percpu: Preserve NOFS/NOIO scope during chunk create and populate

Quoted 37 lines · Vlastimil Babka (SUSE)
Quoted 36 lines · Kaitao Cheng
Quoted 9 lines · Michal Hocko
Quoted 3 lines · Kaitao Cheng

However, if we revert 9a5b183941b, it seems that all of these issues would be resolved. The only downside is that the failure rate of pcpu_alloc_noprof() allocations may increase, which might be acceptable.

That has practical impact on some versions of iscsid which do not have PR_SET_IO_FLUSHER. And maybe some more so I would rather not revert based on a theoretical concerns which I believe is the case here.

Based on the previous discussion, I think we have a way to address most of the concurrency issues around percpu allocation.

However, there still seems to be one remaining case that I do not yet have a good way to solve. For example:

Thread A calls pcpu_alloc_noprof() with GFP_KERNEL and takes pcpu_alloc_mutex. Since the internal allocation is not constrained by NOFS, it may enter FS reclaim while still holding pcpu_alloc_mutex, creating a dependency like:

pcpu_alloc_mutex -> fs_reclaim -> FS lock
At the same time, Thread B may already hold an FS lock and then call pcpu_alloc_noprof() with GFP_NOFS. It will try to acquire pcpu_alloc_mutex and block, creating the reverse dependency:

FS lock -> pcpu_alloc_mutex
This can still form a potential deadlock cycle.

Does anyone have a good suggestion for how to handle this remaining case? Or should we simply treat all GFP_KERNEL/GFP_NOFS allocation behavior in pcpu_alloc_noprof() as GFP_NOIO?

Quoted 8 lines
Quoted 4 lines

If there is no clear solution for now, would it be acceptable to first fix some of the issues introduced by commit 9a5b183941b, and leave this remaining case as a pre-existing historical issue to be handled separately later?

We don't need to solve any issues that are only theoretical and based on scenarios that nobody sane should be doing, i.e. Pedro already pointed out "As in no reclaim path should be insane^W daring enough to do pcpu allocations?"

Yes, but you do not need to do a pcp allocation from the reclaim path to hit the deadlock. All you need is a NOFS pcp allocation - e.g. one done from NOFS scope. Then you have fs lock <-> pcpu_alloc_mutex dependency and a potential deadlock. It is hard to know whether we have any of those in the kernel but I know for a fact (9a5b183941b) that there are scoped NOIO allocations so I wouldn't be suprised if the same was the case for NOFS. Not only that having NOIO is a weaker reclaim context.

Quoted 3 lines

Elsewhere Pedro said "The proper way of fixing this would probably be to release pcpu_alloc_mutex (or not have it in the first place!) while you're allocating memory."

I do agree with this. Back then when I was dealing with the NOIO issue I've tried to look at the lock and drop it but it was not really straightforward. Maybe my lack of close understanding of the pcp allocator was an obstacle there. So if there is a path forward like that then it would certainly be the best.

Signature · 2 lines
Michal Hocko
SUSE Labs
#11 lore raw

Re: [PATCH 1/2] mm/percpu: Preserve NOFS/NOIO scope during chunk create and populate

Quoted 47 lines · Michal Hocko
Quoted 37 lines · Vlastimil Babka (SUSE)
Quoted 36 lines · Kaitao Cheng
Quoted 9 lines · Michal Hocko
Quoted 3 lines · Kaitao Cheng

However, if we revert 9a5b183941b, it seems that all of these issues would be resolved. The only downside is that the failure rate of pcpu_alloc_noprof() allocations may increase, which might be acceptable.

That has practical impact on some versions of iscsid which do not have PR_SET_IO_FLUSHER. And maybe some more so I would rather not revert based on a theoretical concerns which I believe is the case here.

Based on the previous discussion, I think we have a way to address most of the concurrency issues around percpu allocation.

However, there still seems to be one remaining case that I do not yet have a good way to solve. For example:

Thread A calls pcpu_alloc_noprof() with GFP_KERNEL and takes pcpu_alloc_mutex. Since the internal allocation is not constrained by NOFS, it may enter FS reclaim while still holding pcpu_alloc_mutex, creating a dependency like:

pcpu_alloc_mutex -> fs_reclaim -> FS lock
At the same time, Thread B may already hold an FS lock and then call pcpu_alloc_noprof() with GFP_NOFS. It will try to acquire pcpu_alloc_mutex and block, creating the reverse dependency:

FS lock -> pcpu_alloc_mutex
This can still form a potential deadlock cycle.

Does anyone have a good suggestion for how to handle this remaining case? Or should we simply treat all GFP_KERNEL/GFP_NOFS allocation behavior in pcpu_alloc_noprof() as GFP_NOIO?

Quoted 8 lines
Quoted 4 lines

If there is no clear solution for now, would it be acceptable to first fix some of the issues introduced by commit 9a5b183941b, and leave this remaining case as a pre-existing historical issue to be handled separately later?

We don't need to solve any issues that are only theoretical and based on scenarios that nobody sane should be doing, i.e. Pedro already pointed out "As in no reclaim path should be insane^W daring enough to do pcpu allocations?"

Quoted 7 lines

Yes, but you do not need to do a pcp allocation from the reclaim path to hit the deadlock. All you need is a NOFS pcp allocation - e.g. one done from NOFS scope. Then you have fs lock <-> pcpu_alloc_mutex dependency and a potential deadlock. It is hard to know whether we have any of those in the kernel but I know for a fact (9a5b183941b) that there are scoped NOIO allocations so I wouldn't be suprised if the same was the case for NOFS. Not only that having NOIO is a weaker reclaim context.

Besides the case mentioned by Michal Hocko, in my previous email I listed a scenario where blkg_conf_prep races with blkcg_deactivate_policy. This is also something that can actually happen in the existing code, and we have discussed some possible solutions.

The concurrent situation Pedro pointed out, "As in no reclaim path should be insane^W daring enough to do pcpu allocations?", does exist at the theoretical level. However, some other issues do genuinely exist. The reason they have not been observed so far may be that the reproduction probability is very low, or that they have already happened but no one has reported them to the community.

Quoted 9 lines
Quoted 3 lines

Elsewhere Pedro said "The proper way of fixing this would probably be to release pcpu_alloc_mutex (or not have it in the first place!) while you're allocating memory."

I do agree with this. Back then when I was dealing with the NOIO issue I've tried to look at the lock and drop it but it was not really straightforward. Maybe my lack of close understanding of the pcp allocator was an obstacle there. So if there is a path forward like that then it would certainly be the best.

That is indeed the optimal solution, but after thinking about it briefly, this optimization may not be easy to implement, or it may require a large amount of changes. I would really appreciate hearing any concrete suggestions people may have on how to optimize pcpu_alloc_mutex. Otherwise, this issue will remain blocked here.

Signature · 2 lines
Thanks
Kaitao Cheng
#12 lore raw

Re: [PATCH 1/2] mm/percpu: Preserve NOFS/NOIO scope during chunk create and populate

In reply to Kaitao Cheng
Quoted 24 lines · Kaitao Cheng
Quoted 9 lines · Michal Hocko
Quoted 3 lines · Kaitao Cheng

However, if we revert 9a5b183941b, it seems that all of these issues would be resolved. The only downside is that the failure rate of pcpu_alloc_noprof() allocations may increase, which might be acceptable.

That has practical impact on some versions of iscsid which do not have PR_SET_IO_FLUSHER. And maybe some more so I would rather not revert based on a theoretical concerns which I believe is the case here.

Based on the previous discussion, I think we have a way to address most of the concurrency issues around percpu allocation.

However, there still seems to be one remaining case that I do not yet have a good way to solve. For example:

Thread A calls pcpu_alloc_noprof() with GFP_KERNEL and takes pcpu_alloc_mutex. Since the internal allocation is not constrained by NOFS, it may enter FS reclaim while still holding pcpu_alloc_mutex, creating a dependency like:

Quoted 7 lines

pcpu_alloc_mutex -> fs_reclaim -> FS lock
At the same time, Thread B may already hold an FS lock and then call pcpu_alloc_noprof() with GFP_NOFS. It will try to acquire pcpu_alloc_mutex and block, creating the reverse dependency:

FS lock -> pcpu_alloc_mutex
This can still form a potential deadlock cycle.

Correct.

Quoted 3 lines

Does anyone have a good suggestion for how to handle this remaining case? Or should we simply treat all GFP_KERNEL/GFP_NOFS allocation behavior in pcpu_alloc_noprof() as GFP_NOIO?

Yes, weakening the reclaim context would work around these dependencies. This seems like a viable option as long as pcp allocations are not in latency sensitive paths and stalling them under memory pressure is acceptable. My insight into pcp allocator users is very limited so I cannot really make any judgment call here.

Signature · 2 lines
Michal Hocko
SUSE Labs
#13 lore raw

Re: [PATCH 1/2] mm/percpu: Preserve NOFS/NOIO scope during chunk create and populate

In reply to Dennis Zhou

On Sat, May 30, 2026 at 03:32:19PM +0200, Dennis Zhou wrote:
<snip>

Quoted 8 lines
Quoted 7 lines

I’ll try to do it.

Following your suggestion, including in [PATCH 2/2], I will also try a different approach and fix the issue by reducing the scope of the pcpu_alloc_mutex critical section.

Quoted 5 lines

No please don't. The point of the percpu mutex is to ensure that only one person is ever possibly creating a new chunk. If you drop the mutex, then you have to deal with concurrent callers when available percpu memory is low. Percpu memory is expensive and unmovable so the cost is in the control plane to avoid excess fragmentation.

And so is slab memory, yet this is not a problem. We shouldn't keep problematic locks just because they may aid fragmentation slightly.

Signature · 1 line
Pedro
#14 lore raw

PATCH 2/2 mm/percpu: Avoid pcpu_alloc_mutex recursion from reclaim

In reply to Kaitao Cheng

From: Kaitao Cheng <[email protected]>

pcpu_alloc_noprof() takes pcpu_alloc_mutex for sleepable allocations so that it can create chunks and populate backing pages. If reclaim is entered while that mutex is already held, and reclaim reaches a path which allocates percpu memory, the nested allocation can try to take pcpu_alloc_mutex again.

That creates a reclaim recursion dependency:

pcpu_alloc_noprof(GFP_KERNEL)
mutex_lock(&pcpu_alloc_mutex)
reclaim
pcpu_alloc_noprof(GFP_NOIO/GFP_NOFS)
mutex_lock(&pcpu_alloc_mutex)

Avoid this by treating percpu allocations from reclaim context as atomic. Such allocations may still be served from already available and populated areas, but they must not enter the mutex-protected slow path or create new chunks. If no space is available, fail the allocation and let the normal balance work handle replenishment outside reclaim.

Update the function comment to describe that reclaim context allocations are atomic regardless of whether the supplied GFP mask would otherwise allow blocking.

This patch is a preventive fix. There may not currently be any path that calls pcpu_alloc_noprof(GFP_NOIO/GFP_NOFS) from direct reclaim context.

Fixes 9a5b183941b5 ("mm, percpu: do not consider sleepable allocations atomic")
---
 mm/percpu.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)
mm/percpu.c +9 -4
index 1bb38467390b..9c30e5897813 100644
@@ -1803,9 +1803,9 @@ static void pcpu_memalloc_scope_restore(gfp_t gfp, unsigned int flags)
  * @gfp: allocation flags
  *
  * Allocate percpu area of @size bytes aligned at @align.  If @gfp doesn't
- * contain %GFP_KERNEL, the allocation is atomic. If @gfp has __GFP_NOWARN
- * then no warning will be triggered on invalid or failed allocation
- * requests.
+ * allow blocking, or if allocation is requested from reclaim context, the
+ * allocation is atomic. If @gfp has __GFP_NOWARN then no warning will be
+ * triggered on invalid or failed allocation requests.
  *
  * RETURNS:
  * Percpu pointer to the allocated area on success, NULL on failure.
@@ -1828,7 +1828,12 @@ void __percpu *pcpu_alloc_noprof(size_t size, size_t align, bool reserved,
 	gfp = current_gfp_context(gfp);
 	/* whitelisted flags that can be passed to the backing allocators */
 	pcpu_gfp = gfp & (GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN);
-	is_atomic = !gfpflags_allow_blocking(gfp);
+	/*
+	 * Reclaim can be entered while pcpu_alloc_mutex is already held by
+	 * another percpu allocation. Avoid recursing back into the mutex from
+	 * reclaim; best-effort allocations from already populated areas are OK.
+	 */
+	is_atomic = !gfpflags_allow_blocking(gfp) || current->reclaim_state;
 	do_warn = !(gfp & __GFP_NOWARN);
 
 	/*
Signature · 1 line
2.50.1 (Apple Git-155)
#15 lore raw

Re: [PATCH 2/2] mm/percpu: Avoid pcpu_alloc_mutex recursion from reclaim

Quoted 22 lines · Kaitao Cheng

From: Kaitao Cheng <[email protected]>

pcpu_alloc_noprof() takes pcpu_alloc_mutex for sleepable allocations so that it can create chunks and populate backing pages. If reclaim is entered while that mutex is already held, and reclaim reaches a path which allocates percpu memory, the nested allocation can try to take pcpu_alloc_mutex again.

That creates a reclaim recursion dependency:

pcpu_alloc_noprof(GFP_KERNEL)
mutex_lock(&pcpu_alloc_mutex)
reclaim
pcpu_alloc_noprof(GFP_NOIO/GFP_NOFS)
mutex_lock(&pcpu_alloc_mutex)

Avoid this by treating percpu allocations from reclaim context as atomic. Such allocations may still be served from already available and populated areas, but they must not enter the mutex-protected slow path or create new chunks. If no space is available, fail the allocation and let the normal balance work handle replenishment outside reclaim.

Quoted 6 lines

Update the function comment to describe that reclaim context allocations are atomic regardless of whether the supplied GFP mask would otherwise allow blocking.

This patch is a preventive fix. There may not currently be any path that calls pcpu_alloc_noprof(GFP_NOIO/GFP_NOFS) from direct reclaim context.

I don't like this. The proper way of fixing this would probably be to release pcpu_alloc_mutex (or not have it in the first place!) while you're allocating memory.

Quoted 25 lines
Fixes 9a5b183941b5 ("mm, percpu: do not consider sleepable allocations atomic")
---
 mm/percpu.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)
mm/percpu.c +3 -3
index 1bb38467390b..9c30e5897813 100644
@@ -1803,9 +1803,9 @@ static void pcpu_memalloc_scope_restore(gfp_t gfp, unsigned int flags)
  * @gfp: allocation flags
  *
  * Allocate percpu area of @size bytes aligned at @align.  If @gfp doesn't
- * contain %GFP_KERNEL, the allocation is atomic. If @gfp has __GFP_NOWARN
- * then no warning will be triggered on invalid or failed allocation
- * requests.
+ * allow blocking, or if allocation is requested from reclaim context, the
+ * allocation is atomic. If @gfp has __GFP_NOWARN then no warning will be
+ * triggered on invalid or failed allocation requests.
  *
  * RETURNS:
  * Percpu pointer to the allocated area on success, NULL on failure.

mm/percpu.c void __percpu *pcpu_alloc_noprof(size_t size, size_t align, bool reserved, · line 1835

Quoted 9 lines
mm/percpu.c +5 -1
@@ -1828,7 +1828,12 @@ void __percpu *pcpu_alloc_noprof(size_t size, size_t align, bool reserved,
 	gfp = current_gfp_context(gfp);
 	/* whitelisted flags that can be passed to the backing allocators */
 	pcpu_gfp = gfp & (GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN);
-	is_atomic = !gfpflags_allow_blocking(gfp);
+	/*
+	 * Reclaim can be entered while pcpu_alloc_mutex is already held by
+	 * another percpu allocation. Avoid recursing back into the mutex from
+	 * reclaim; best-effort allocations from already populated areas are OK.
+	 */

since this is an entirely theoretical issue:

	/* Reclaim paths should not be hitting the percpu allocator, for now */
	if (WARN_ON_ONCE(current->reclaim_state))
		return NULL;

But that's just my 2c.

Quoted 7 lines
mm/percpu.c +1 -0
+	is_atomic = !gfpflags_allow_blocking(gfp) || current->reclaim_state;
 	do_warn = !(gfp & __GFP_NOWARN);
 
 	/*
Signature · 1 line
2.50.1 (Apple Git-155)
Signature · 1 line
Pedro
maintainer
#16 lore raw
In reply to Kaitao Cheng
Quoted 23 lines · On Thu, 28 May 2026 21:29:15 +0800 Kaitao Cheng <[email protected]> wrote:

Commit 9a5b183941b5 ("mm, percpu: do not consider sleepable allocations atomic") allowed GFP_NOFS and GFP_NOIO percpu allocations to use pcpu_alloc_mutex and the chunk creation slow path. This restored the allocation capability that was lost when those constrained allocations were treated as atomic, but it also opens two possible reclaim recursion problems.

The first problem is that the create and populate slow paths do not fully preserve the caller's allocation constraints. pcpu_alloc_noprof() derives pcpu_gfp from the caller supplied GFP mask and passes it to the backing page allocator. However, pcpu_create_chunk() calls pcpu_get_vm_areas(), and population can allocate temporary metadata or page tables while mapping backing pages. Those internal allocations can use GFP_KERNEL. A caller using GFP_NOFS or GFP_NOIO can therefore still enter unconstrained FS or IO reclaim while holding pcpu_alloc_mutex. This defeats the caller's allocation context.

The second problem is a possible pcpu_alloc_mutex recursion from reclaim. If reclaim is entered while pcpu_alloc_mutex is already held, and reclaim reaches a path which allocates percpu memory with GFP_NOFS or GFP_NOIO, the nested allocation can now try to take pcpu_alloc_mutex again because 9a5b183941b5 no longer treats those masks as atomic.

Quoted 4 lines

Another possible way to avoid these issues is to revert 9a5b183941b5. However, that would also bring back the premature allocation failures for sleepable GFP_NOFS/GFP_NOIO percpu users that 9a5b183941b5 was intended to fix.

Thanks.

9a5b183941b5 has been in there for a year. How are you observing/triggering this bug and what are the userspace-visible effects?

We might choose to backport fixes into -stable kernels, but this additional info is needed to make that determination.

maintainer
#17 lore raw
In reply to Kaitao Cheng
Quoted 23 lines · On Thu, 28 May 2026 21:29:15 +0800 Kaitao Cheng <[email protected]> wrote:

Commit 9a5b183941b5 ("mm, percpu: do not consider sleepable allocations atomic") allowed GFP_NOFS and GFP_NOIO percpu allocations to use pcpu_alloc_mutex and the chunk creation slow path. This restored the allocation capability that was lost when those constrained allocations were treated as atomic, but it also opens two possible reclaim recursion problems.

The first problem is that the create and populate slow paths do not fully preserve the caller's allocation constraints. pcpu_alloc_noprof() derives pcpu_gfp from the caller supplied GFP mask and passes it to the backing page allocator. However, pcpu_create_chunk() calls pcpu_get_vm_areas(), and population can allocate temporary metadata or page tables while mapping backing pages. Those internal allocations can use GFP_KERNEL. A caller using GFP_NOFS or GFP_NOIO can therefore still enter unconstrained FS or IO reclaim while holding pcpu_alloc_mutex. This defeats the caller's allocation context.

The second problem is a possible pcpu_alloc_mutex recursion from reclaim. If reclaim is entered while pcpu_alloc_mutex is already held, and reclaim reaches a path which allocates percpu memory with GFP_NOFS or GFP_NOIO, the nested allocation can now try to take pcpu_alloc_mutex again because 9a5b183941b5 no longer treats those masks as atomic.

Quoted 4 lines

Another possible way to avoid these issues is to revert 9a5b183941b5. However, that would also bring back the premature allocation failures for sleepable GFP_NOFS/GFP_NOIO percpu users that 9a5b183941b5 was intended to fix.

AI review might have found a couple of pre-existing issues:

	https://sashiko.dev/#/patchset/[email protected]