Skip to contentSkip to conversation

mm: percpu: increase PERCPU_MODULE_RESERVE to avoid allocation failure

PATCH gaoxu 5 participants 6 messages Apr 22, 2025 – Apr 23, 2025

Subsystems (2)
lore
Share & tools
lore

gaoxu

author
#0 lore raw

In android16-6.12, enabling CONFIG_MEM_ALLOC_PROFILING causes some modules to fail to load during boot because of failed percpu memory allocation.

[811:modprobe]percpu: allocation failed, size=5200 align=8 atomic=0, alloc from reserved chunk failed
[811:modprobe]Call trace:
[811:modprobe] dump_backtrace+0xfc/0x17c
[811:modprobe] show_stack+0x18/0x28
[811:modprobe] dump_stack_lvl+0x40/0xc0
[811:modprobe] dump_stack+0x18/0x24
[811:modprobe] pcpu_alloc_noprof+0x96c/0xb58
[811:modprobe] percpu_modalloc+0x50/0xec
[811:modprobe] load_module+0x1158/0x153c
[811:modprobe] __arm64_sys_finit_module+0x23c/0x340 [811:modprobe] invoke_syscall+0x58/0x10c
[811:modprobe] el0_svc_common+0xa8/0xdc
[811:modprobe] do_el0_svc+0x1c/0x28
[811:modprobe] el0_svc+0x40/0x90
[811:modprobe] el0t_64_sync_handler+0x70/0xbc
[811:modprobe] el0t_64_sync+0x1a8/0x1ac
[811:modprobe]ipam: Could not allocate 5200 bytes percpu data

Increase PERCPU_MODULE_RESERVE to resolve this issue.

---
 include/linux/percpu.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/percpu.h +1 -1
index 52b5ea663..765c7f562 100644
@@ -16,7 +16,7 @@
 /* enough to cover all DEFINE_PER_CPUs in modules */
 #ifdef CONFIG_MODULES
 #ifdef CONFIG_MEM_ALLOC_PROFILING
-#define PERCPU_MODULE_RESERVE		(8 << 13)
+#define PERCPU_MODULE_RESERVE		(8 << 14)
 #else
 #define PERCPU_MODULE_RESERVE		(8 << 10)
 #endif
Signature · 1 line
2.17.1
maintainer
#1 lore raw
Quoted 2 lines · On Tue, 22 Apr 2025 11:39:30 +0000 gaoxu <[email protected]> wrote:

In android16-6.12, enabling CONFIG_MEM_ALLOC_PROFILING causes some modules to fail to load during boot because of failed percpu memory allocation.

Which modules? If they're in-tree modules then we should fix this issue in -stable kernels also.

If they're out-of-tree modules then what argument is there for altering the mainline kernel?

Quoted 26 lines

[811:modprobe]percpu: allocation failed, size=5200 align=8 atomic=0, alloc from reserved chunk failed
[811:modprobe]Call trace:
[811:modprobe] dump_backtrace+0xfc/0x17c
[811:modprobe] show_stack+0x18/0x28
[811:modprobe] dump_stack_lvl+0x40/0xc0
[811:modprobe] dump_stack+0x18/0x24
[811:modprobe] pcpu_alloc_noprof+0x96c/0xb58
[811:modprobe] percpu_modalloc+0x50/0xec
[811:modprobe] load_module+0x1158/0x153c
[811:modprobe] __arm64_sys_finit_module+0x23c/0x340 [811:modprobe] invoke_syscall+0x58/0x10c
[811:modprobe] el0_svc_common+0xa8/0xdc
[811:modprobe] do_el0_svc+0x1c/0x28
[811:modprobe] el0_svc+0x40/0x90
[811:modprobe] el0t_64_sync_handler+0x70/0xbc
[811:modprobe] el0t_64_sync+0x1a8/0x1ac
[811:modprobe]ipam: Could not allocate 5200 bytes percpu data

Increase PERCPU_MODULE_RESERVE to resolve this issue.

...

include/linux/percpu.h line 22

Quoted 8 lines
include/linux/percpu.h +1 -1
@@ -16,7 +16,7 @@
 /* enough to cover all DEFINE_PER_CPUs in modules */
 #ifdef CONFIG_MODULES
 #ifdef CONFIG_MEM_ALLOC_PROFILING
-#define PERCPU_MODULE_RESERVE		(8 << 13)
+#define PERCPU_MODULE_RESERVE		(8 << 14)
 #else
 #define PERCPU_MODULE_RESERVE		(8 << 10)
 #endif

PERCPU_MODULE_RESERVE is a pretty unpleasant thing. It appears that it gives us the choice between either wasting memory or failing module loading. But I expect that something more dynamic would be a ton of work.

#2 lore raw
Quoted 11 lines · Andrew Morton
Quoted 2 lines · On Tue, 22 Apr 2025 11:39:30 +0000 gaoxu <[email protected]> wrote:

In android16-6.12, enabling CONFIG_MEM_ALLOC_PROFILING causes some modules to fail to load during boot because of failed percpu memory allocation.

Which modules? If they're in-tree modules then we should fix this issue in -stable kernels also.

If they're out-of-tree modules then what argument is there for altering the mainline kernel?

These are most likely out-of-tree modules from an Android partner.

Quoted 24 lines
Quoted 24 lines

[811:modprobe]percpu: allocation failed, size=5200 align=8 atomic=0, alloc from reserved chunk failed
[811:modprobe]Call trace:
[811:modprobe] dump_backtrace+0xfc/0x17c
[811:modprobe] show_stack+0x18/0x28
[811:modprobe] dump_stack_lvl+0x40/0xc0
[811:modprobe] dump_stack+0x18/0x24
[811:modprobe] pcpu_alloc_noprof+0x96c/0xb58
[811:modprobe] percpu_modalloc+0x50/0xec
[811:modprobe] load_module+0x1158/0x153c
[811:modprobe] __arm64_sys_finit_module+0x23c/0x340 [811:modprobe] invoke_syscall+0x58/0x10c
[811:modprobe] el0_svc_common+0xa8/0xdc
[811:modprobe] do_el0_svc+0x1c/0x28
[811:modprobe] el0_svc+0x40/0x90
[811:modprobe] el0t_64_sync_handler+0x70/0xbc
[811:modprobe] el0t_64_sync+0x1a8/0x1ac
[811:modprobe]ipam: Could not allocate 5200 bytes percpu data

Increase PERCPU_MODULE_RESERVE to resolve this issue.

...

Quoted 15 lines

include/linux/percpu.h line 22

Quoted 11 lines
include/linux/percpu.h +1 -1
@@ -16,7 +16,7 @@
 /* enough to cover all DEFINE_PER_CPUs in modules */
 #ifdef CONFIG_MODULES
 #ifdef CONFIG_MEM_ALLOC_PROFILING
-#define PERCPU_MODULE_RESERVE                (8 << 13)
+#define PERCPU_MODULE_RESERVE                (8 << 14)
 #else
 #define PERCPU_MODULE_RESERVE                (8 << 10)
 #endif

PERCPU_MODULE_RESERVE is a pretty unpleasant thing. It appears that it gives us the choice between either wasting memory or failing module loading. But I expect that something more dynamic would be a ton of work.

Allocating this reserved area dynamically would be ideal. OTOH this change increases the area size from 64kb to 128kb. Don't know how much effort we should put into it.

maintainer
#3 lore raw

On Tue, Apr 22, 2025 at 05:19:31PM -0700, Suren Baghdasaryan wrote:
...

Quoted 3 lines

Allocating this reserved area dynamically would be ideal. OTOH this change increases the area size from 64kb to 128kb. Don't know how much effort we should put into it.

The easiest solution would be switching the modules to use alloc_percpu() instead of declaring per-cpu variables statically. I couldn't think of a better way to support static percpu variables in modules and still can't, but there aren't noticeable downsides to using dynamically allocated percpu variables, so if you have several bytes here and there, sure, declare them statically, but for anything chunky, please use dynamic allocations.

Thanks.

Signature · 1 line
tejun
#4 lore raw
Quoted 7 lines · Tejun Heo

On Tue, Apr 22, 2025 at 05:19:31PM -0700, Suren Baghdasaryan wrote:
...

Quoted 3 lines

Allocating this reserved area dynamically would be ideal. OTOH this change increases the area size from 64kb to 128kb. Don't know how much effort we should put into it.

Quoted 6 lines

The easiest solution would be switching the modules to use alloc_percpu() instead of declaring per-cpu variables statically. I couldn't think of a better way to support static percpu variables in modules and still can't, but there aren't noticeable downsides to using dynamically allocated percpu variables, so if you have several bytes here and there, sure, declare them statically, but for anything chunky, please use dynamic allocations.

In case of allocation tags, we are trying to minimize performance overhead as much as possible and allocating their per-cpu counters at compile time is in line with that goal. I'll check how much overhead dynamic allocation would add but it won't be zero.

Quoted 5 lines

Thanks.

--
tejun

maintainer
#5 lore raw
In reply to Andrew Morton

Hi Andrew,

Quoted 34 lines · Andrew Morton
Quoted 2 lines · On Tue, 22 Apr 2025 11:39:30 +0000 gaoxu <[email protected]> wrote:

In android16-6.12, enabling CONFIG_MEM_ALLOC_PROFILING causes some modules to fail to load during boot because of failed percpu memory allocation.

Which modules? If they're in-tree modules then we should fix this issue in -stable kernels also.

If they're out-of-tree modules then what argument is there for altering the mainline kernel?

Quoted 23 lines

[811:modprobe]percpu: allocation failed, size=5200 align=8 atomic=0, alloc from reserved chunk failed
[811:modprobe]Call trace:
[811:modprobe] dump_backtrace+0xfc/0x17c
[811:modprobe] show_stack+0x18/0x28
[811:modprobe] dump_stack_lvl+0x40/0xc0
[811:modprobe] dump_stack+0x18/0x24
[811:modprobe] pcpu_alloc_noprof+0x96c/0xb58
[811:modprobe] percpu_modalloc+0x50/0xec
[811:modprobe] load_module+0x1158/0x153c
[811:modprobe] __arm64_sys_finit_module+0x23c/0x340 [811:modprobe] invoke_syscall+0x58/0x10c
[811:modprobe] el0_svc_common+0xa8/0xdc
[811:modprobe] do_el0_svc+0x1c/0x28
[811:modprobe] el0_svc+0x40/0x90
[811:modprobe] el0t_64_sync_handler+0x70/0xbc
[811:modprobe] el0t_64_sync+0x1a8/0x1ac
[811:modprobe]ipam: Could not allocate 5200 bytes percpu data

Increase PERCPU_MODULE_RESERVE to resolve this issue.

...

Quoted 15 lines

include/linux/percpu.h line 22

Quoted 11 lines
include/linux/percpu.h +1 -1
@@ -16,7 +16,7 @@
 /* enough to cover all DEFINE_PER_CPUs in modules */
 #ifdef CONFIG_MODULES
 #ifdef CONFIG_MEM_ALLOC_PROFILING
-#define PERCPU_MODULE_RESERVE		(8 << 13)
+#define PERCPU_MODULE_RESERVE		(8 << 14)
 #else
 #define PERCPU_MODULE_RESERVE		(8 << 10)
 #endif

PERCPU_MODULE_RESERVE is a pretty unpleasant thing. It appears that it gives us the choice between either wasting memory or failing module loading. But I expect that something more dynamic would be a ton of work.

From Tj's commit back in 2009... 6b19b0c24004

+/*
+ * On x86_64 symbols referenced from code should be reachable using
+ * 32bit relocations. Reserve space for static percpu variables in
+ * modules so that they are always served from the first chunk which
+ * is located at the percpu segment base. On x86_32, anything can
+ * address anywhere. No need to reserve space in the first chunk.
+ */

I'm not too sure where our x86_64 32 bit support is. If that is no longer true then we can likely fold the reserved region back into the dynamic region.

Given the above, there's not really an opportunity to do this after the system has booted hence why it's baked into the first chunk.

Thanks,
Dennis