2d76e79315: Kernel_panic-not_syncing:Fatal_exception
Subsystems (4)
kernel test robot
build botHello,
for this change, we reported
"[linux-next:master] [lib/test_vmalloc.c] 7fc85b92db: Mem-Info" in
https://lore.kernel.org/all/[email protected]/
at that time, we made some tests with x86_64 config which runs well.
now we noticed the commit is in mainline now.
the config still has expected diff with parent:
pkg/linux/x86_64-randconfig-161-20250614/gcc-12/2d76e79315e403aab595d4c8830b7a46c19f0f3b/.config 2025-06-17 14:41:18.448543738 +0800 renamed from pkg/linux/x86_64-randconfig-161-20250614/gcc-12/7a73348e5d4715b5565a53f21c01ea7b54e46cbd/.config 2025-06-17 14:40:29.481052101 +0800 +1 -1
@@ -7551,7 +7551,7 @@ CONFIG_TEST_IDA=m CONFIG_TEST_MISC_MINOR=m # CONFIG_TEST_LKM is not set CONFIG_TEST_BITOPS=m -CONFIG_TEST_VMALLOC=m +CONFIG_TEST_VMALLOC=y # CONFIG_TEST_BPF is not set CONFIG_FIND_BIT_BENCHMARK=m # CONFIG_TEST_FIRMWARE is not set
then we noticed similar random issue with x86_64 randconfig this time.
7a73348e5d4715b5 2d76e79315e403aab595d4c8830
---------------- ---------------------------
fail:runs %reproduction fail:runs
| | |
:199 34% 67:200 dmesg.KASAN:null-ptr-deref_in_range[#-#]
:199 34% 67:200 dmesg.Kernel_panic-not_syncing:Fatal_exception
:199 34% 67:200 dmesg.Mem-Info
:199 34% 67:200 dmesg.Oops:general_protection_fault,probably_for_non-canonical_address#:#[##]SMP_KASAN
:199 34% 67:200 dmesg.RIP:down_read_trylock
we don't have enough knowledge to understand the relationship between code change and the random issues. just report what we obsverved in our tests FYI.
below is full report.
kernel test robot noticed "Kernel_panic-not_syncing:Fatal_exception" on:
commit: 2d76e79315e403aab595d4c8830b7a46c19f0f3b ("lib/test_vmalloc.c: allow built-in execution")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
[test failed on linus/master e04c78d86a9699d136910cfc0bdcf01087e3267e]
[test failed on linux-next/master 050f8ad7b58d9079455af171ac279c4b9b828c11]
in testcase: boot
config: x86_64-randconfig-161-20250614
compiler: gcc-12
test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G
(please refer to attached dmesg/kmsg for entire log/backtrace)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-lkp/[email protected]
[ 36.902716][ T60] vmalloc_node_range for size 8192 failed: Address range restricted to 0xffffc90000000000 - 0xffffe8ffffffffff [ 36.903981][ T60] vmalloc_test/0: vmalloc error: size 4096, vm_struct allocation failed, mode:0xdc0(GFP_KERNEL|__GFP_ZERO), nodemask=(null) [ 36.905195][ T60] CPU: 1 UID: 0 PID: 60 Comm: vmalloc_test/0 Not tainted 6.15.0-rc6-00142-g2d76e79315e4 #1 VOLUNTARY [ 36.905201][ T60] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 [ 36.905203][ T60] Call Trace: [ 36.905206][ T60] <TASK> [ 36.905209][ T60] dump_stack_lvl+0x87/0xd6 [ 36.905223][ T60] warn_alloc+0x15e/0x291 [ 36.905230][ T60] ? has_managed_dma+0x37/0x37 [ 36.905237][ T60] ? __get_vm_area_node+0x33a/0x3c0 [ 36.905244][ T60] ? __get_vm_area_node+0x33a/0x3c0 [ 36.905250][ T60] __vmalloc_node_range_noprof+0x170/0x306 [ 36.905255][ T60] ? __vmalloc_area_node+0x460/0x460 [ 36.905260][ T60] ? test_func+0x2ae/0x469 [ 36.905264][ T60] __vmalloc_node_noprof+0xb8/0xd9 [ 36.905267][ T60] ? test_func+0x2ae/0x469 [ 36.905272][ T60] align_shift_alloc_test+0xa8/0x165 [ 36.905277][ T60] test_func+0x2ae/0x469 [ 36.905281][ T60] ? pcpu_alloc_test+0x31b/0x31b [ 36.905286][ T60] ? __kthread_parkme+0xcb/0x1a3 [ 36.905293][ T60] ? pcpu_alloc_test+0x31b/0x31b [ 36.905297][ T60] kthread+0x452/0x464 [ 36.905301][ T60] ? kthread_is_per_cpu+0x51/0x51 [ 36.905304][ T60] ? _raw_spin_unlock_irq+0x23/0x35 [ 36.905308][ T60] ? kthread_is_per_cpu+0x51/0x51 [ 36.905311][ T60] ? kthread_is_per_cpu (kbuild/obj/consumer/x86_64-randconfig-161-20250614/kernel/kthread.c:413) [ 36.905314][ T60] ret_from_fork (kbuild/obj/consumer/x86_64-randconfig-161-20250614/arch/x86/kernel/process.c:153) [ 36.905318][ T60] ? kthread_is_per_cpu (kbuild/obj/consumer/x86_64-randconfig-161-20250614/kernel/kthread.c:413) [ 36.905321][ T60] ret_from_fork_asm (kbuild/obj/consumer/x86_64-randconfig-161-20250614/arch/x86/entry/entry_64.S:255) [ 36.905330][ T60] </TASK> [ 36.905332][ T60] Mem-Info: [ 36.919941][ T60] active_anon:0 inactive_anon:0 isolated_anon:0 [ 36.919941][ T60] active_file:0 inactive_file:0 isolated_file:0 [ 36.919941][ T60] unevictable:41612 dirty:0 writeback:0 [ 36.919941][ T60] slab_reclaimable:7429 slab_unreclaimable:145259 [ 36.919941][ T60] mapped:0 shmem:0 pagetables:145 [ 36.919941][ T60] sec_pagetables:0 bounce:0 [ 36.919941][ T60] kernel_misc_reclaimable:0 [ 36.919941][ T60] free:3233392 free_pcp:1185 free_cma:0 [ 36.923830][ T60] Node 0 active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:166448kB isolated(anon):0kB isolated(file):0kB mapped:0kB dirty:0kB writeback:0kB shmem:0kB writeback_tmp:0kB kernel_stack:1952kB pagetables:580kB sec_pagetables:0kB all_unreclaimable? no Balloon:0kB [ 36.926265][ T60] DMA free:15360kB boost:0kB min:16kB low:28kB high:40kB reserved_highatomic:0KB free_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [ 36.928855][ T60] lowmem_reserve[]: 0 2991 13741 13741 [ 36.929411][ T60] DMA32 free:3060560kB boost:0kB min:3224kB low:6244kB high:9264kB reserved_highatomic:0KB free_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:3129216kB managed:3063680kB mlocked:0kB bounce:0kB free_pcp:3120kB local_pcp:3120kB free_cma:0kB [ 36.932080][ T60] lowmem_reserve[]: 0 0 10749 10749 [ 36.932604][ T60] Normal free:9857648kB boost:0kB min:11744kB low:22748kB high:33752kB reserved_highatomic:0KB free_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:166448kB writepending:0kB present:13631488kB managed:11007884kB mlocked:0kB bounce:0kB free_pcp:1620kB local_pcp:740kB free_cma:0kB [ 36.935336][ T60] lowmem_reserve[]: 0 0 0 0 [ 36.935802][ T60] DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (U) 3*4096kB (M) = 15360kB [ 36.936931][ T60] DMA32: 0*4kB 0*8kB 1*16kB (M) 2*32kB (M) 2*64kB (M) 1*128kB (M) 2*256kB (M) 2*512kB (M) 1*1024kB (M) 1*2048kB (M) 746*4096kB (M) = 3060560kB [ 36.938318][ T60] Normal: 6*4kB (ME) 2*8kB (ME) 7*16kB (UME) 5*32kB (M) 3*64kB (ME) 4*128kB (M) 6*256kB (UME) 2*512kB (M) 1*1024kB (M) 3*2048kB (UME) 2404*4096kB (M) = 9857528kB [ 36.939849][ T60] 41618 total pagecache pages [ 36.940324][ T60] 4194174 pages RAM [ 36.940721][ T60] 0 pages HighMem/MovableOnly [ 36.941188][ T60] 672443 pages reserved [ 36.941626][ T60] Oops: general protection fault, probably for non-canonical address 0xdffffc000000001b: 0000 [#1] SMP KASAN [ 36.942185][ T60] KASAN: null-ptr-deref in range [0x00000000000000d8-0x00000000000000df] [ 36.942185][ T60] CPU: 1 UID: 0 PID: 60 Comm: vmalloc_test/0 Not tainted 6.15.0-rc6-00142-g2d76e79315e4 #1 VOLUNTARY [ 36.942185][ T60] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 [ 36.942185][ T60] RIP: 0010:down_read_trylock+0xa7/0x2b9 [ 36.942185][ T60] Code: b0 ef 25 91 e8 57 16 40 00 83 3d 9c e6 a7 09 00 0f 85 2c 01 00 00 48 8d 6b 68 b8 ff ff 37 00 48 89 ea 48 c1 e0 2a 48 c1 ea 03 <80> 3c 02 00 74 08 48 89 ef e8 3c 16 40 00 48 3b 5b 68 0f 84 00 01 [ 36.942185][ T60] RSP: 0000:ffff88814657f848 EFLAGS: 00010206 [ 36.942185][ T60] RAX: dffffc0000000000 RBX: 0000000000000070 RCX: 1ffffffff224bdf6 [ 36.942185][ T60] RDX: 000000000000001b RSI: 000000000000000a RDI: 0000000000000070 [ 36.942185][ T60] RBP: 00000000000000d8 R08: 0000000000000000 R09: 0000000000000000 [ 36.942185][ T60] R10: 0000000000000000 R11: 0000000000000000 R12: 1ffff11028caff0a [ 36.942185][ T60] R13: ffff88814657fa30 R14: dffffc0000000000 R15: 0000000000000000 [ 36.942185][ T60] FS: 0000000000000000(0000) GS:ffff88841c1f0000(0000) knlGS:0000000000000000 [ 36.942185][ T60] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 36.942185][ T60] CR2: 0000000000000000 CR3: 00000001636e0000 CR4: 00000000000406b0 [ 36.942185][ T60] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 36.942185][ T60] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 36.942185][ T60] Call Trace: [ 36.942185][ T60] <TASK> [ 36.942185][ T60] ? clear_nonspinnable+0x32/0x32 [ 36.942185][ T60] ? vprintk_emit+0x165/0x194 [ 36.942185][ T60] codetag_trylock_module_list+0xd/0x19 [ 36.942185][ T60] alloc_tag_top_users+0x95/0x216 [ 36.942185][ T60] ? _printk+0xad/0xdf [ 36.942185][ T60] ? reserve_module_tags+0x308/0x308 [ 36.942185][ T60] __show_mem+0x167/0x54b [ 36.942185][ T60] ? _printk+0xad/0xdf [ 36.942185][ T60] ? printk_get_console_flush_type+0x272/0x272 [ 36.942185][ T60] ? show_free_areas+0x115d/0x115d [ 36.942185][ T60] ? tracer_hardirqs_on+0x1b/0x28d [ 36.942185][ T60] ? dump_stack_lvl+0x91/0xd6 [ 36.942185][ T60] ? warn_alloc+0x251/0x291 [ 36.942185][ T60] warn_alloc+0x251/0x291 [ 36.942185][ T60] ? has_managed_dma+0x37/0x37 [ 36.942185][ T60] ? __get_vm_area_node+0x33a/0x3c0 [ 36.942185][ T60] __vmalloc_node_range_noprof+0x170/0x306 [ 36.942185][ T60] ? __vmalloc_area_node+0x460/0x460 [ 36.942185][ T60] ? test_func+0x2ae/0x469 [ 36.942185][ T60] __vmalloc_node_noprof+0xb8/0xd9 [ 36.942185][ T60] ? test_func+0x2ae/0x469 [ 36.942185][ T60] align_shift_alloc_test+0xa8/0x165 [ 36.942185][ T60] test_func+0x2ae/0x469 [ 36.942185][ T60] ? pcpu_alloc_test+0x31b/0x31b [ 36.942185][ T60] ? __kthread_parkme+0xcb/0x1a3 [ 36.942185][ T60] ? pcpu_alloc_test+0x31b/0x31b [ 36.942185][ T60] kthread+0x452/0x464 [ 36.942185][ T60] ? kthread_is_per_cpu+0x51/0x51 [ 36.942185][ T60] ? _raw_spin_unlock_irq+0x23/0x35 [ 36.942185][ T60] ? kthread_is_per_cpu+0x51/0x51 [ 36.942185][ T60] ret_from_fork+0x20/0x54 [ 36.942185][ T60] ? kthread_is_per_cpu+0x51/0x51 [ 36.942185][ T60] ret_from_fork_asm+0x11/0x20 [ 36.942185][ T60] </TASK> [ 36.942185][ T60] Modules linked in: [ 37.000652][ T60] ---[ end trace 0000000000000000 ]--- [ 37.001188][ T60] RIP: 0010:down_read_trylock+0xa7/0x2b9 [ 37.001731][ T60] Code: b0 ef 25 91 e8 57 16 40 00 83 3d 9c e6 a7 09 00 0f 85 2c 01 00 00 48 8d 6b 68 b8 ff ff 37 00 48 89 ea 48 c1 e0 2a 48 c1 ea 03 <80> 3c 02 00 74 08 48 89 ef e8 3c 16 40 00 48 3b 5b 68 0f 84 00 01 [ 37.003488][ T60] RSP: 0000:ffff88814657f848 EFLAGS: 00010206 [ 37.004072][ T60] RAX: dffffc0000000000 RBX: 0000000000000070 RCX: 1ffffffff224bdf6 [ 37.004848][ T60] RDX: 000000000000001b RSI: 000000000000000a RDI: 0000000000000070 [ 37.005610][ T60] RBP: 00000000000000d8 R08: 0000000000000000 R09: 0000000000000000 [ 37.006381][ T60] R10: 0000000000000000 R11: 0000000000000000 R12: 1ffff11028caff0a [ 37.007178][ T60] R13: ffff88814657fa30 R14: dffffc0000000000 R15: 0000000000000000 [ 37.007940][ T60] FS: 0000000000000000(0000) GS:ffff88841c1f0000(0000) knlGS:0000000000000000 [ 37.008792][ T60] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 37.009411][ T60] CR2: 0000000000000000 CR3: 00000001636e0000 CR4: 00000000000406b0 [ 37.010175][ T60] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 37.010950][ T60] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 37.011716][ T60] Kernel panic - not syncing: Fatal exception [ 37.012397][ T60] Kernel Offset: 0x6200000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20250618/[email protected]
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Kernel crash due to alloc_tag_top_users() being called when !mem_profiling_support?
Quoted 11 lines · kernel test robot
Hello,
for this change, we reported
"[linux-next:master] [lib/test_vmalloc.c]7fc85b92db: Mem-Info" in
https://lore.kernel.org/all/[email protected]/at that time, we made some tests with
x86_64config which runs well.now we noticed the commit is in mainline now.
(Re-sending due to not Ccing people and the list...)
Hi, I'm facing the same error on my testing environment.
I think this is related to memory allocation profiling & code tagging subsystems rather than vmalloc, so let's add related folks to Cc.
After a quick skimming of the code, it seems the condition to trigger this is that on 1) MEM_ALLOC_PROFILING is compiled but
2) not enabled by default. and 3) allocation somehow failed, calling alloc_tag_top_users().
I see "Memory allocation profiling is not supported!" in the dmesg, which means it did not alloc & inititialize alloc_tag_cttype properly, but alloc_tag_top_users() tries to acquire the semaphore.
I think the kernel should not call alloc_tag_top_users() at all (or it should return an error) if mem_profiling_support == false?
Does the following work on your testing environment?
(Only did very light testing on my QEMU, but seems to fix the issue for me.)
lib/alloc_tag.c +3 -1
@@ -134,7 +134,9 @@ size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_slstruct codetag_bytes n;unsigned int i, nr =0 ; -if (can_sleep) +if (!mem_profiling_support) +return 0 ; +else if (can_sleep)codetag_lock_module_list (alloc_tag_cttype, true);else if (!codetag_trylock_module_list (alloc_tag_cttype))return 0 ;
Quoted 193 lines
the config still has expected diff with parent:
pkg/linux/x86_64-randconfig-161-20250614/gcc-12/2d76e79315e403aab595d4c8830b7a46c19f0f3b/.config 2025-06-17 14:41:18.448543738 +0800 renamed from pkg/linux/x86_64-randconfig-161-20250614/gcc-12/7a73348e5d4715b5565a53f21c01ea7b54e46cbd/.config 2025-06-17 14:40:29.481052101 +0800 +1
-1@@ -7551,7 +7551,7 @@ CONFIG_TEST_IDA=m CONFIG_TEST_MISC_MINOR=m # CONFIG_TEST_LKM is not set CONFIG_TEST_BITOPS=m -CONFIG_TEST_VMALLOC=m +CONFIG_TEST_VMALLOC=y # CONFIG_TEST_BPF is not set CONFIG_FIND_BIT_BENCHMARK=m # CONFIG_TEST_FIRMWARE is not setthen we noticed similar random issue with
x86_64randconfig this time.
7a73348e5d4715b52d76e79315e403aab595d4c8830
---------------- ---------------------------
fail:runs %reproduction fail:runs
| | |
:199 34% 67:200 dmesg.KASAN:null-ptr-deref_in_range[#-#]
:199 34% 67:200 dmesg.Kernel_panic-not_syncing:Fatal_exception
:199 34% 67:200 dmesg.Mem-Info
:199 34% 67:200 dmesg.Oops:general_protection_fault,probably_for_non-canonical_address#:#[##]SMP_KASAN
:199 34% 67:200 dmesg.RIP:down_read_trylockwe don't have enough knowledge to understand the relationship between code change and the random issues. just report what we obsverved in our tests FYI.
below is full report.
kernel test robot noticed "Kernel_panic-not_syncing:Fatal_exception" on:
commit:
2d76e79315e403aab595d4c8830b7a46c19f0f3b("lib/test_vmalloc.c: allow built-in execution")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master[test failed on linus/master
e04c78d86a9699d136910cfc0bdcf01087e3267e]
[test failed on linux-next/master050f8ad7b58d9079455af171ac279c4b9b828c11]in testcase: boot
config: x86_64-randconfig-161-20250614
compiler: gcc-12
test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G(please refer to attached dmesg/kmsg for entire log/backtrace)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-lkp/[email protected][ 36.902716][ T60] vmalloc_node_range for size 8192 failed: Address range restricted to 0xffffc90000000000 - 0xffffe8ffffffffff [ 36.903981][ T60] vmalloc_test/0: vmalloc error: size 4096, vm_struct allocation failed, mode:0xdc0(GFP_KERNEL|__GFP_ZERO), nodemask=(null) [ 36.905195][ T60] CPU: 1 UID: 0 PID: 60 Comm: vmalloc_test/0 Not tainted 6.15.0-rc6-00142-g2d76e79315e4 #1 VOLUNTARY [ 36.905201][ T60] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 [ 36.905203][ T60] Call Trace: [ 36.905206][ T60] <TASK> [ 36.905209][ T60] dump_stack_lvl+0x87/0xd6 [ 36.905223][ T60] warn_alloc+0x15e/0x291 [ 36.905230][ T60] ? has_managed_dma+0x37/0x37 [ 36.905237][ T60] ? __get_vm_area_node+0x33a/0x3c0 [ 36.905244][ T60] ? __get_vm_area_node+0x33a/0x3c0 [ 36.905250][ T60] __vmalloc_node_range_noprof+0x170/0x306 [ 36.905255][ T60] ? __vmalloc_area_node+0x460/0x460 [ 36.905260][ T60] ? test_func+0x2ae/0x469 [ 36.905264][ T60] __vmalloc_node_noprof+0xb8/0xd9 [ 36.905267][ T60] ? test_func+0x2ae/0x469 [ 36.905272][ T60] align_shift_alloc_test+0xa8/0x165 [ 36.905277][ T60] test_func+0x2ae/0x469 [ 36.905281][ T60] ? pcpu_alloc_test+0x31b/0x31b [ 36.905286][ T60] ? __kthread_parkme+0xcb/0x1a3 [ 36.905293][ T60] ? pcpu_alloc_test+0x31b/0x31b [ 36.905297][ T60] kthread+0x452/0x464 [ 36.905301][ T60] ? kthread_is_per_cpu+0x51/0x51 [ 36.905304][ T60] ? _raw_spin_unlock_irq+0x23/0x35 [ 36.905308][ T60] ? kthread_is_per_cpu+0x51/0x51 [ 36.905311][ T60] ? kthread_is_per_cpu (kbuild/obj/consumer/x86_64-randconfig-161-20250614/kernel/kthread.c:413) [ 36.905314][ T60] ret_from_fork (kbuild/obj/consumer/x86_64-randconfig-161-20250614/arch/x86/kernel/process.c:153) [ 36.905318][ T60] ? kthread_is_per_cpu (kbuild/obj/consumer/x86_64-randconfig-161-20250614/kernel/kthread.c:413) [ 36.905321][ T60] ret_from_fork_asm (kbuild/obj/consumer/x86_64-randconfig-161-20250614/arch/x86/entry/entry_64.S:255) [ 36.905330][ T60] </TASK> [ 36.905332][ T60] Mem-Info: [ 36.919941][ T60] active_anon:0 inactive_anon:0 isolated_anon:0 [ 36.919941][ T60] active_file:0 inactive_file:0 isolated_file:0 [ 36.919941][ T60] unevictable:41612 dirty:0 writeback:0 [ 36.919941][ T60] slab_reclaimable:7429 slab_unreclaimable:145259 [ 36.919941][ T60] mapped:0 shmem:0 pagetables:145 [ 36.919941][ T60] sec_pagetables:0 bounce:0 [ 36.919941][ T60] kernel_misc_reclaimable:0 [ 36.919941][ T60] free:3233392 free_pcp:1185 free_cma:0 [ 36.923830][ T60] Node 0 active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:166448kB isolated(anon):0kB isolated(file):0kB mapped:0kB dirty:0kB writeback:0kB shmem:0kB writeback_tmp:0kB kernel_stack:1952kB pagetables:580kB sec_pagetables:0kB all_unreclaimable? no Balloon:0kB [ 36.926265][ T60] DMA free:15360kB boost:0kB min:16kB low:28kB high:40kB reserved_highatomic:0KB free_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [ 36.928855][ T60] lowmem_reserve[]: 0 2991 13741 13741 [ 36.929411][ T60] DMA32 free:3060560kB boost:0kB min:3224kB low:6244kB high:9264kB reserved_highatomic:0KB free_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:3129216kB managed:3063680kB mlocked:0kB bounce:0kB free_pcp:3120kB local_pcp:3120kB free_cma:0kB [ 36.932080][ T60] lowmem_reserve[]: 0 0 10749 10749 [ 36.932604][ T60] Normal free:9857648kB boost:0kB min:11744kB low:22748kB high:33752kB reserved_highatomic:0KB free_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:166448kB writepending:0kB present:13631488kB managed:11007884kB mlocked:0kB bounce:0kB free_pcp:1620kB local_pcp:740kB free_cma:0kB [ 36.935336][ T60] lowmem_reserve[]: 0 0 0 0 [ 36.935802][ T60] DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (U) 3*4096kB (M) = 15360kB [ 36.936931][ T60] DMA32: 0*4kB 0*8kB 1*16kB (M) 2*32kB (M) 2*64kB (M) 1*128kB (M) 2*256kB (M) 2*512kB (M) 1*1024kB (M) 1*2048kB (M) 746*4096kB (M) = 3060560kB [ 36.938318][ T60] Normal: 6*4kB (ME) 2*8kB (ME) 7*16kB (UME) 5*32kB (M) 3*64kB (ME) 4*128kB (M) 6*256kB (UME) 2*512kB (M) 1*1024kB (M) 3*2048kB (UME) 2404*4096kB (M) = 9857528kB [ 36.939849][ T60] 41618 total pagecache pages [ 36.940324][ T60] 4194174 pages RAM [ 36.940721][ T60] 0 pages HighMem/MovableOnly [ 36.941188][ T60] 672443 pages reserved [ 36.941626][ T60] Oops: general protection fault, probably for non-canonical address 0xdffffc000000001b: 0000 [#1] SMP KASAN [ 36.942185][ T60] KASAN: null-ptr-deref in range [0x00000000000000d8-0x00000000000000df] [ 36.942185][ T60] CPU: 1 UID: 0 PID: 60 Comm: vmalloc_test/0 Not tainted 6.15.0-rc6-00142-g2d76e79315e4 #1 VOLUNTARY [ 36.942185][ T60] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 [ 36.942185][ T60] RIP: 0010:down_read_trylock+0xa7/0x2b9 [ 36.942185][ T60] Code: b0 ef 25 91 e8 57 16 40 00 83 3d 9c e6 a7 09 00 0f 85 2c 01 00 00 48 8d 6b 68 b8 ff ff 37 00 48 89 ea 48 c1 e0 2a 48 c1 ea 03 <80> 3c 02 00 74 08 48 89 ef e8 3c 16 40 00 48 3b 5b 68 0f 84 00 01 [ 36.942185][ T60] RSP: 0000:ffff88814657f848 EFLAGS: 00010206 [ 36.942185][ T60] RAX: dffffc0000000000 RBX: 0000000000000070 RCX: 1ffffffff224bdf6 [ 36.942185][ T60] RDX: 000000000000001b RSI: 000000000000000a RDI: 0000000000000070 [ 36.942185][ T60] RBP: 00000000000000d8 R08: 0000000000000000 R09: 0000000000000000 [ 36.942185][ T60] R10: 0000000000000000 R11: 0000000000000000 R12: 1ffff11028caff0a [ 36.942185][ T60] R13: ffff88814657fa30 R14: dffffc0000000000 R15: 0000000000000000 [ 36.942185][ T60] FS: 0000000000000000(0000) GS:ffff88841c1f0000(0000) knlGS:0000000000000000 [ 36.942185][ T60] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 36.942185][ T60] CR2: 0000000000000000 CR3: 00000001636e0000 CR4: 00000000000406b0 [ 36.942185][ T60] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 36.942185][ T60] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 36.942185][ T60] Call Trace: [ 36.942185][ T60] <TASK> [ 36.942185][ T60] ? clear_nonspinnable+0x32/0x32 [ 36.942185][ T60] ? vprintk_emit+0x165/0x194 [ 36.942185][ T60] codetag_trylock_module_list+0xd/0x19 [ 36.942185][ T60] alloc_tag_top_users+0x95/0x216 [ 36.942185][ T60] ? _printk+0xad/0xdf [ 36.942185][ T60] ? reserve_module_tags+0x308/0x308 [ 36.942185][ T60] __show_mem+0x167/0x54b [ 36.942185][ T60] ? _printk+0xad/0xdf [ 36.942185][ T60] ? printk_get_console_flush_type+0x272/0x272 [ 36.942185][ T60] ? show_free_areas+0x115d/0x115d [ 36.942185][ T60] ? tracer_hardirqs_on+0x1b/0x28d [ 36.942185][ T60] ? dump_stack_lvl+0x91/0xd6 [ 36.942185][ T60] ? warn_alloc+0x251/0x291 [ 36.942185][ T60] warn_alloc+0x251/0x291 [ 36.942185][ T60] ? has_managed_dma+0x37/0x37 [ 36.942185][ T60] ? __get_vm_area_node+0x33a/0x3c0 [ 36.942185][ T60] __vmalloc_node_range_noprof+0x170/0x306 [ 36.942185][ T60] ? __vmalloc_area_node+0x460/0x460 [ 36.942185][ T60] ? test_func+0x2ae/0x469 [ 36.942185][ T60] __vmalloc_node_noprof+0xb8/0xd9 [ 36.942185][ T60] ? test_func+0x2ae/0x469 [ 36.942185][ T60] align_shift_alloc_test+0xa8/0x165 [ 36.942185][ T60] test_func+0x2ae/0x469 [ 36.942185][ T60] ? pcpu_alloc_test+0x31b/0x31b [ 36.942185][ T60] ? __kthread_parkme+0xcb/0x1a3 [ 36.942185][ T60] ? pcpu_alloc_test+0x31b/0x31b [ 36.942185][ T60] kthread+0x452/0x464 [ 36.942185][ T60] ? kthread_is_per_cpu+0x51/0x51 [ 36.942185][ T60] ? _raw_spin_unlock_irq+0x23/0x35 [ 36.942185][ T60] ? kthread_is_per_cpu+0x51/0x51 [ 36.942185][ T60] ret_from_fork+0x20/0x54 [ 36.942185][ T60] ? kthread_is_per_cpu+0x51/0x51 [ 36.942185][ T60] ret_from_fork_asm+0x11/0x20 [ 36.942185][ T60] </TASK> [ 36.942185][ T60] Modules linked in: [ 37.000652][ T60] ---[ end trace 0000000000000000 ]--- [ 37.001188][ T60] RIP: 0010:down_read_trylock+0xa7/0x2b9 [ 37.001731][ T60] Code: b0 ef 25 91 e8 57 16 40 00 83 3d 9c e6 a7 09 00 0f 85 2c 01 00 00 48 8d 6b 68 b8 ff ff 37 00 48 89 ea 48 c1 e0 2a 48 c1 ea 03 <80> 3c 02 00 74 08 48 89 ef e8 3c 16 40 00 48 3b 5b 68 0f 84 00 01 [ 37.003488][ T60] RSP: 0000:ffff88814657f848 EFLAGS: 00010206 [ 37.004072][ T60] RAX: dffffc0000000000 RBX: 0000000000000070 RCX: 1ffffffff224bdf6 [ 37.004848][ T60] RDX: 000000000000001b RSI: 000000000000000a RDI: 0000000000000070 [ 37.005610][ T60] RBP: 00000000000000d8 R08: 0000000000000000 R09: 0000000000000000 [ 37.006381][ T60] R10: 0000000000000000 R11: 0000000000000000 R12: 1ffff11028caff0a [ 37.007178][ T60] R13: ffff88814657fa30 R14: dffffc0000000000 R15: 0000000000000000 [ 37.007940][ T60] FS: 0000000000000000(0000) GS:ffff88841c1f0000(0000) knlGS:0000000000000000 [ 37.008792][ T60] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 37.009411][ T60] CR2: 0000000000000000 CR3: 00000001636e0000 CR4: 00000000000406b0 [ 37.010175][ T60] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 37.010950][ T60] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 37.011716][ T60] Kernel panic - not syncing: Fatal exception [ 37.012397][ T60] Kernel Offset: 0x6200000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20250618/[email protected]Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Signature · 2 lines
Cheers, Harry / Hyeonggon
Re: Kernel crash due to alloc_tag_top_users() being called when !mem_profiling_support?
Quoted 9 lines · Harry Yoo
Quoted 8 lines · kernel test robot
Hello,
for this change, we reported
"[linux-next:master] [lib/test_vmalloc.c]7fc85b92db: Mem-Info" in
https://lore.kernel.org/all/[email protected]/
Quoted 7 lines
Quoted 3 lines
at that time, we made some tests with
x86_64config which runs well.now we noticed the commit is in mainline now.
(Re-sending due to not Ccing people and the list...)
Hi, I'm facing the same error on my testing environment.
I should have clarified that the reason the kernel failed to allocate memory on my machine was due to running out of memory, not because of the vmalloc test module.
But based on the fact that the test case (align_shift_alloc_test) is expected to fail, the issue here is not memory allocation failure itself, but rather that the kernel crashes when the allocation fails.
So I expect the fix below will work for you as well.
Quoted 232 lines
I think this is related to memory allocation profiling & code tagging subsystems rather than vmalloc, so let's add related folks to Cc.
After a quick skimming of the code, it seems the condition to trigger this is that on 1)
MEM_ALLOC_PROFILINGis compiled but
2) not enabled by default. and 3) allocation somehow failed, callingalloc_tag_top_users().I see "Memory allocation profiling is not supported!" in the dmesg, which means it did not alloc & inititialize
alloc_tag_cttypeproperly, butalloc_tag_top_users()tries to acquire the semaphore.I think the kernel should not call
alloc_tag_top_users()at all (or it should return an error) ifmem_profiling_support== false?Does the following work on your testing environment?
(Only did very light testing on my QEMU, but seems to fix the issue for me.)
lib/alloc_tag.c +3
-1@@ -134,7 +134,9 @@ size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_slstruct codetag_bytes n;unsigned int i, nr =0 ; -if (can_sleep) +if (!mem_profiling_support) +return 0 ; +else if (can_sleep)codetag_lock_module_list (alloc_tag_cttype, true);else if (!codetag_trylock_module_list (alloc_tag_cttype))return 0 ;Quoted 194 lines
the config still has expected diff with parent:
pkg/linux/x86_64-randconfig-161-20250614/gcc-12/2d76e79315e403aab595d4c8830b7a46c19f0f3b/.config 2025-06-17 14:41:18.448543738 +0800 renamed from pkg/linux/x86_64-randconfig-161-20250614/gcc-12/7a73348e5d4715b5565a53f21c01ea7b54e46cbd/.config 2025-06-17 14:40:29.481052101 +0800 +1
-1@@ -7551,7 +7551,7 @@ CONFIG_TEST_IDA=m CONFIG_TEST_MISC_MINOR=m # CONFIG_TEST_LKM is not set CONFIG_TEST_BITOPS=m -CONFIG_TEST_VMALLOC=m +CONFIG_TEST_VMALLOC=y # CONFIG_TEST_BPF is not set CONFIG_FIND_BIT_BENCHMARK=m # CONFIG_TEST_FIRMWARE is not setthen we noticed similar random issue with
x86_64randconfig this time.
7a73348e5d4715b52d76e79315e403aab595d4c8830
---------------- ---------------------------
fail:runs %reproduction fail:runs
| | |
:199 34% 67:200 dmesg.KASAN:null-ptr-deref_in_range[#-#]
:199 34% 67:200 dmesg.Kernel_panic-not_syncing:Fatal_exception
:199 34% 67:200 dmesg.Mem-Info
:199 34% 67:200 dmesg.Oops:general_protection_fault,probably_for_non-canonical_address#:#[##]SMP_KASAN
:199 34% 67:200 dmesg.RIP:down_read_trylockwe don't have enough knowledge to understand the relationship between code change and the random issues. just report what we obsverved in our tests FYI.
below is full report.
kernel test robot noticed "Kernel_panic-not_syncing:Fatal_exception" on:
commit:
2d76e79315e403aab595d4c8830b7a46c19f0f3b("lib/test_vmalloc.c: allow built-in execution")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master[test failed on linus/master
e04c78d86a9699d136910cfc0bdcf01087e3267e]
[test failed on linux-next/master050f8ad7b58d9079455af171ac279c4b9b828c11]in testcase: boot
config: x86_64-randconfig-161-20250614
compiler: gcc-12
test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G(please refer to attached dmesg/kmsg for entire log/backtrace)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-lkp/[email protected][ 36.902716][ T60] vmalloc_node_range for size 8192 failed: Address range restricted to 0xffffc90000000000 - 0xffffe8ffffffffff [ 36.903981][ T60] vmalloc_test/0: vmalloc error: size 4096, vm_struct allocation failed, mode:0xdc0(GFP_KERNEL|__GFP_ZERO), nodemask=(null) [ 36.905195][ T60] CPU: 1 UID: 0 PID: 60 Comm: vmalloc_test/0 Not tainted 6.15.0-rc6-00142-g2d76e79315e4 #1 VOLUNTARY [ 36.905201][ T60] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 [ 36.905203][ T60] Call Trace: [ 36.905206][ T60] <TASK> [ 36.905209][ T60] dump_stack_lvl+0x87/0xd6 [ 36.905223][ T60] warn_alloc+0x15e/0x291 [ 36.905230][ T60] ? has_managed_dma+0x37/0x37 [ 36.905237][ T60] ? __get_vm_area_node+0x33a/0x3c0 [ 36.905244][ T60] ? __get_vm_area_node+0x33a/0x3c0 [ 36.905250][ T60] __vmalloc_node_range_noprof+0x170/0x306 [ 36.905255][ T60] ? __vmalloc_area_node+0x460/0x460 [ 36.905260][ T60] ? test_func+0x2ae/0x469 [ 36.905264][ T60] __vmalloc_node_noprof+0xb8/0xd9 [ 36.905267][ T60] ? test_func+0x2ae/0x469 [ 36.905272][ T60] align_shift_alloc_test+0xa8/0x165 [ 36.905277][ T60] test_func+0x2ae/0x469 [ 36.905281][ T60] ? pcpu_alloc_test+0x31b/0x31b [ 36.905286][ T60] ? __kthread_parkme+0xcb/0x1a3 [ 36.905293][ T60] ? pcpu_alloc_test+0x31b/0x31b [ 36.905297][ T60] kthread+0x452/0x464 [ 36.905301][ T60] ? kthread_is_per_cpu+0x51/0x51 [ 36.905304][ T60] ? _raw_spin_unlock_irq+0x23/0x35 [ 36.905308][ T60] ? kthread_is_per_cpu+0x51/0x51 [ 36.905311][ T60] ? kthread_is_per_cpu (kbuild/obj/consumer/x86_64-randconfig-161-20250614/kernel/kthread.c:413) [ 36.905314][ T60] ret_from_fork (kbuild/obj/consumer/x86_64-randconfig-161-20250614/arch/x86/kernel/process.c:153) [ 36.905318][ T60] ? kthread_is_per_cpu (kbuild/obj/consumer/x86_64-randconfig-161-20250614/kernel/kthread.c:413) [ 36.905321][ T60] ret_from_fork_asm (kbuild/obj/consumer/x86_64-randconfig-161-20250614/arch/x86/entry/entry_64.S:255) [ 36.905330][ T60] </TASK> [ 36.905332][ T60] Mem-Info: [ 36.919941][ T60] active_anon:0 inactive_anon:0 isolated_anon:0 [ 36.919941][ T60] active_file:0 inactive_file:0 isolated_file:0 [ 36.919941][ T60] unevictable:41612 dirty:0 writeback:0 [ 36.919941][ T60] slab_reclaimable:7429 slab_unreclaimable:145259 [ 36.919941][ T60] mapped:0 shmem:0 pagetables:145 [ 36.919941][ T60] sec_pagetables:0 bounce:0 [ 36.919941][ T60] kernel_misc_reclaimable:0 [ 36.919941][ T60] free:3233392 free_pcp:1185 free_cma:0 [ 36.923830][ T60] Node 0 active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:166448kB isolated(anon):0kB isolated(file):0kB mapped:0kB dirty:0kB writeback:0kB shmem:0kB writeback_tmp:0kB kernel_stack:1952kB pagetables:580kB sec_pagetables:0kB all_unreclaimable? no Balloon:0kB [ 36.926265][ T60] DMA free:15360kB boost:0kB min:16kB low:28kB high:40kB reserved_highatomic:0KB free_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [ 36.928855][ T60] lowmem_reserve[]: 0 2991 13741 13741 [ 36.929411][ T60] DMA32 free:3060560kB boost:0kB min:3224kB low:6244kB high:9264kB reserved_highatomic:0KB free_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:3129216kB managed:3063680kB mlocked:0kB bounce:0kB free_pcp:3120kB local_pcp:3120kB free_cma:0kB [ 36.932080][ T60] lowmem_reserve[]: 0 0 10749 10749 [ 36.932604][ T60] Normal free:9857648kB boost:0kB min:11744kB low:22748kB high:33752kB reserved_highatomic:0KB free_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:166448kB writepending:0kB present:13631488kB managed:11007884kB mlocked:0kB bounce:0kB free_pcp:1620kB local_pcp:740kB free_cma:0kB [ 36.935336][ T60] lowmem_reserve[]: 0 0 0 0 [ 36.935802][ T60] DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (U) 3*4096kB (M) = 15360kB [ 36.936931][ T60] DMA32: 0*4kB 0*8kB 1*16kB (M) 2*32kB (M) 2*64kB (M) 1*128kB (M) 2*256kB (M) 2*512kB (M) 1*1024kB (M) 1*2048kB (M) 746*4096kB (M) = 3060560kB [ 36.938318][ T60] Normal: 6*4kB (ME) 2*8kB (ME) 7*16kB (UME) 5*32kB (M) 3*64kB (ME) 4*128kB (M) 6*256kB (UME) 2*512kB (M) 1*1024kB (M) 3*2048kB (UME) 2404*4096kB (M) = 9857528kB [ 36.939849][ T60] 41618 total pagecache pages [ 36.940324][ T60] 4194174 pages RAM [ 36.940721][ T60] 0 pages HighMem/MovableOnly [ 36.941188][ T60] 672443 pages reserved [ 36.941626][ T60] Oops: general protection fault, probably for non-canonical address 0xdffffc000000001b: 0000 [#1] SMP KASAN [ 36.942185][ T60] KASAN: null-ptr-deref in range [0x00000000000000d8-0x00000000000000df] [ 36.942185][ T60] CPU: 1 UID: 0 PID: 60 Comm: vmalloc_test/0 Not tainted 6.15.0-rc6-00142-g2d76e79315e4 #1 VOLUNTARY [ 36.942185][ T60] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 [ 36.942185][ T60] RIP: 0010:down_read_trylock+0xa7/0x2b9 [ 36.942185][ T60] Code: b0 ef 25 91 e8 57 16 40 00 83 3d 9c e6 a7 09 00 0f 85 2c 01 00 00 48 8d 6b 68 b8 ff ff 37 00 48 89 ea 48 c1 e0 2a 48 c1 ea 03 <80> 3c 02 00 74 08 48 89 ef e8 3c 16 40 00 48 3b 5b 68 0f 84 00 01 [ 36.942185][ T60] RSP: 0000:ffff88814657f848 EFLAGS: 00010206 [ 36.942185][ T60] RAX: dffffc0000000000 RBX: 0000000000000070 RCX: 1ffffffff224bdf6 [ 36.942185][ T60] RDX: 000000000000001b RSI: 000000000000000a RDI: 0000000000000070 [ 36.942185][ T60] RBP: 00000000000000d8 R08: 0000000000000000 R09: 0000000000000000 [ 36.942185][ T60] R10: 0000000000000000 R11: 0000000000000000 R12: 1ffff11028caff0a [ 36.942185][ T60] R13: ffff88814657fa30 R14: dffffc0000000000 R15: 0000000000000000 [ 36.942185][ T60] FS: 0000000000000000(0000) GS:ffff88841c1f0000(0000) knlGS:0000000000000000 [ 36.942185][ T60] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 36.942185][ T60] CR2: 0000000000000000 CR3: 00000001636e0000 CR4: 00000000000406b0 [ 36.942185][ T60] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 36.942185][ T60] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 36.942185][ T60] Call Trace: [ 36.942185][ T60] <TASK> [ 36.942185][ T60] ? clear_nonspinnable+0x32/0x32 [ 36.942185][ T60] ? vprintk_emit+0x165/0x194 [ 36.942185][ T60] codetag_trylock_module_list+0xd/0x19 [ 36.942185][ T60] alloc_tag_top_users+0x95/0x216 [ 36.942185][ T60] ? _printk+0xad/0xdf [ 36.942185][ T60] ? reserve_module_tags+0x308/0x308 [ 36.942185][ T60] __show_mem+0x167/0x54b [ 36.942185][ T60] ? _printk+0xad/0xdf [ 36.942185][ T60] ? printk_get_console_flush_type+0x272/0x272 [ 36.942185][ T60] ? show_free_areas+0x115d/0x115d [ 36.942185][ T60] ? tracer_hardirqs_on+0x1b/0x28d [ 36.942185][ T60] ? dump_stack_lvl+0x91/0xd6 [ 36.942185][ T60] ? warn_alloc+0x251/0x291 [ 36.942185][ T60] warn_alloc+0x251/0x291 [ 36.942185][ T60] ? has_managed_dma+0x37/0x37 [ 36.942185][ T60] ? __get_vm_area_node+0x33a/0x3c0 [ 36.942185][ T60] __vmalloc_node_range_noprof+0x170/0x306 [ 36.942185][ T60] ? __vmalloc_area_node+0x460/0x460 [ 36.942185][ T60] ? test_func+0x2ae/0x469 [ 36.942185][ T60] __vmalloc_node_noprof+0xb8/0xd9 [ 36.942185][ T60] ? test_func+0x2ae/0x469 [ 36.942185][ T60] align_shift_alloc_test+0xa8/0x165 [ 36.942185][ T60] test_func+0x2ae/0x469 [ 36.942185][ T60] ? pcpu_alloc_test+0x31b/0x31b [ 36.942185][ T60] ? __kthread_parkme+0xcb/0x1a3 [ 36.942185][ T60] ? pcpu_alloc_test+0x31b/0x31b [ 36.942185][ T60] kthread+0x452/0x464 [ 36.942185][ T60] ? kthread_is_per_cpu+0x51/0x51 [ 36.942185][ T60] ? _raw_spin_unlock_irq+0x23/0x35 [ 36.942185][ T60] ? kthread_is_per_cpu+0x51/0x51 [ 36.942185][ T60] ret_from_fork+0x20/0x54 [ 36.942185][ T60] ? kthread_is_per_cpu+0x51/0x51 [ 36.942185][ T60] ret_from_fork_asm+0x11/0x20 [ 36.942185][ T60] </TASK> [ 36.942185][ T60] Modules linked in: [ 37.000652][ T60] ---[ end trace 0000000000000000 ]--- [ 37.001188][ T60] RIP: 0010:down_read_trylock+0xa7/0x2b9 [ 37.001731][ T60] Code: b0 ef 25 91 e8 57 16 40 00 83 3d 9c e6 a7 09 00 0f 85 2c 01 00 00 48 8d 6b 68 b8 ff ff 37 00 48 89 ea 48 c1 e0 2a 48 c1 ea 03 <80> 3c 02 00 74 08 48 89 ef e8 3c 16 40 00 48 3b 5b 68 0f 84 00 01 [ 37.003488][ T60] RSP: 0000:ffff88814657f848 EFLAGS: 00010206 [ 37.004072][ T60] RAX: dffffc0000000000 RBX: 0000000000000070 RCX: 1ffffffff224bdf6 [ 37.004848][ T60] RDX: 000000000000001b RSI: 000000000000000a RDI: 0000000000000070 [ 37.005610][ T60] RBP: 00000000000000d8 R08: 0000000000000000 R09: 0000000000000000 [ 37.006381][ T60] R10: 0000000000000000 R11: 0000000000000000 R12: 1ffff11028caff0a [ 37.007178][ T60] R13: ffff88814657fa30 R14: dffffc0000000000 R15: 0000000000000000 [ 37.007940][ T60] FS: 0000000000000000(0000) GS:ffff88841c1f0000(0000) knlGS:0000000000000000 [ 37.008792][ T60] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 37.009411][ T60] CR2: 0000000000000000 CR3: 00000001636e0000 CR4: 00000000000406b0 [ 37.010175][ T60] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 37.010950][ T60] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 37.011716][ T60] Kernel panic - not syncing: Fatal exception [ 37.012397][ T60] Kernel Offset: 0x6200000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20250618/[email protected]Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wikiSignature · 2 lines
Cheers, Harry / Hyeonggon
Signature · 2 lines
Cheers, Harry / Hyeonggon
Uladzislau Rezki
maintainerRe: Kernel crash due to alloc_tag_top_users() being called when !mem_profiling_support?
Quoted 18 lines · Harry Yoo
Quoted 9 lines · Harry Yoo
Quoted 8 lines · kernel test robot
Hello,
for this change, we reported
"[linux-next:master] [lib/test_vmalloc.c]7fc85b92db: Mem-Info" in
https://lore.kernel.org/all/[email protected]/Quoted 7 lines
Quoted 3 lines
at that time, we made some tests with
x86_64config which runs well.now we noticed the commit is in mainline now.
(Re-sending due to not Ccing people and the list...)
Hi, I'm facing the same error on my testing environment.
Quoted 8 lines
I should have clarified that the reason the kernel failed to allocate memory on my machine was due to running out of memory, not because of the vmalloc test module.
But based on the fact that the test case (
align_shift_alloc_test) is expected to fail, the issue here is not memory allocation failure itself, but rather that the kernel crashes when the allocation fails.
It looks someone tries to test the CONFIG_TEST_VMALLOC=y as built-in approach test-cases. Yes, it will trigger a lot of warnings as some use cases are supposed to be failed. This will trigger a lot of kernel warnings which can be considered by test-robot or people as problem.
In this case i can exclude those use cases or even not run at all unless boot-parameters properly sets if built-in.
--
Uladzislau Rezki
Re: Kernel crash due to alloc_tag_top_users() being called when !mem_profiling_support?
Quoted 28 lines · Uladzislau Rezki
Quoted 18 lines · Harry Yoo
Quoted 9 lines · Harry Yoo
Quoted 8 lines · kernel test robot
Hello,
for this change, we reported
"[linux-next:master] [lib/test_vmalloc.c]7fc85b92db: Mem-Info" in
https://lore.kernel.org/all/[email protected]/Quoted 7 lines
Quoted 3 lines
at that time, we made some tests with
x86_64config which runs well.now we noticed the commit is in mainline now.
(Re-sending due to not Ccing people and the list...)
Hi, I'm facing the same error on my testing environment.
Quoted 8 lines
I should have clarified that the reason the kernel failed to allocate memory on my machine was due to running out of memory, not because of the vmalloc test module.
But based on the fact that the test case (
align_shift_alloc_test) is expected to fail, the issue here is not memory allocation failure itself, but rather that the kernel crashes when the allocation fails.
Quoted 7 lines
It looks someone tries to test the
CONFIG_TEST_VMALLOC=yas built-in approach test-cases. Yes, it will trigger a lot of warnings as some use cases are supposed to be failed. This will trigger a lot of kernel warnings which can be considered by test-robot or people as problem.In this case i can exclude those use cases or even not run at all unless boot-parameters properly sets if built-in.
Sorry, I'm catching up on my email backlog. IIUC
https://lore.kernel.org/all/[email protected]/
addresses this issue. Is my understanding correct?
Quoted 3 lines
--
Uladzislau Rezki
Uladzislau Rezki
maintainerRe: Kernel crash due to alloc_tag_top_users() being called when !mem_profiling_support?
Quoted 34 lines · Suren Baghdasaryan
Quoted 33 lines · Uladzislau Rezki
Quoted 18 lines · Harry Yoo
Quoted 9 lines · Harry Yoo
Quoted 8 lines · kernel test robot
Hello,
for this change, we reported
"[linux-next:master] [lib/test_vmalloc.c]7fc85b92db: Mem-Info" in
https://lore.kernel.org/all/[email protected]/Quoted 7 lines
Quoted 3 lines
at that time, we made some tests with
x86_64config which runs well.now we noticed the commit is in mainline now.
(Re-sending due to not Ccing people and the list...)
Hi, I'm facing the same error on my testing environment.
Quoted 8 lines
I should have clarified that the reason the kernel failed to allocate memory on my machine was due to running out of memory, not because of the vmalloc test module.
But based on the fact that the test case (
align_shift_alloc_test) is expected to fail, the issue here is not memory allocation failure itself, but rather that the kernel crashes when the allocation fails.It looks someone tries to test the
CONFIG_TEST_VMALLOC=yas built-in approach test-cases. Yes, it will trigger a lot of warnings as some use cases are supposed to be failed. This will trigger a lot of kernel warnings which can be considered by test-robot or people as problem.
Quoted 7 lines
Quoted 2 lines
In this case i can exclude those use cases or even not run at all unless boot-parameters properly sets if built-in.
Sorry, I'm catching up on my email backlog. IIUC
https://lore.kernel.org/all/[email protected]/
addresses this issue. Is my understanding correct?
I checked/tested the .config from the test-robot in order to reproduce the kernel crash. Unfortunately i can not trigger this. But, people from the another thread already confirmed that it solves the crash.
--
Uladzislau Rezki
Re: Kernel crash due to alloc_tag_top_users() being called when !mem_profiling_support?
In reply to Harry YooQuoted 41 lines · kernel test robot
Quoted 11 lines
Hello,
for this change, we reported
"[linux-next:master] [lib/test_vmalloc.c]7fc85b92db: Mem-Info" in
https://lore.kernel.org/all/[email protected]/at that time, we made some tests with
x86_64config which runs well.now we noticed the commit is in mainline now.
(Re-sending due to not Ccing people and the list...)
Hi, I'm facing the same error on my testing environment.
I think this is related to memory allocation profiling & code tagging subsystems rather than vmalloc, so let's add related folks to Cc.
After a quick skimming of the code, it seems the condition to trigger this is that on 1)
MEM_ALLOC_PROFILINGis compiled but
2) not enabled by default. and 3) allocation somehow failed, callingalloc_tag_top_users().I see "Memory allocation profiling is not supported!" in the dmesg, which means it did not alloc & inititialize
alloc_tag_cttypeproperly, butalloc_tag_top_users()tries to acquire the semaphore.I think the kernel should not call
alloc_tag_top_users()at all (or it should return an error) ifmem_profiling_support== false?Does the following work on your testing environment?
(Only did very light testing on my QEMU, but seems to fix the issue for me.)
lib/alloc_tag.c +0
-0
lib/alloc_tag.c size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_sl · line 142
Quoted 10 lines
lib/alloc_tag.c +3
-1@@ -134,7 +134,9 @@ size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_slstruct codetag_bytes n;unsigned int i, nr =0 ; -if (can_sleep) +if (!mem_profiling_support) +return 0 ; +else if (can_sleep)codetag_lock_module_list (alloc_tag_cttype, true);else if (!codetag_trylock_module_list (alloc_tag_cttype))return 0 ;
I think you are correct, this was introduced/exposed by
commit 780138b1 ("alloc_tag: check mem_profiling_support in alloc_tag_init") (Before the commit, the BUG would only be triggered when alloc_tag_init failed)
David
Re: Kernel crash due to alloc_tag_top_users() being called when !mem_profiling_support?
Quoted 44 lines · David Wang
Quoted 44 lines · kernel test robot
Quoted 11 lines
Hello,
for this change, we reported
"[linux-next:master] [lib/test_vmalloc.c]7fc85b92db: Mem-Info" in
https://lore.kernel.org/all/[email protected]/at that time, we made some tests with
x86_64config which runs well.now we noticed the commit is in mainline now.
(Re-sending due to not Ccing people and the list...)
Hi, I'm facing the same error on my testing environment.
I think this is related to memory allocation profiling & code tagging subsystems rather than vmalloc, so let's add related folks to Cc.
After a quick skimming of the code, it seems the condition to trigger this is that on 1)
MEM_ALLOC_PROFILINGis compiled but
2) not enabled by default. and 3) allocation somehow failed, callingalloc_tag_top_users().I see "Memory allocation profiling is not supported!" in the dmesg, which means it did not alloc & inititialize
alloc_tag_cttypeproperly, butalloc_tag_top_users()tries to acquire the semaphore.I think the kernel should not call
alloc_tag_top_users()at all (or it should return an error) ifmem_profiling_support== false?Does the following work on your testing environment?
(Only did very light testing on my QEMU, but seems to fix the issue for me.)
lib/alloc_tag.c +0
-0@@ -134,7 +134,9 @@ size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_slstruct codetag_bytes n;unsigned int i, nr =0 ;
Quoted 10 lines
lib/alloc_tag.c size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_sl · line 142
Quoted 7 lines
lib/alloc_tag.c +3
-1-if (can_sleep) +if (!mem_profiling_support) +return 0 ; +else if (can_sleep)codetag_lock_module_list (alloc_tag_cttype, true);else if (!codetag_trylock_module_list (alloc_tag_cttype))return 0 ;I think you are correct, this was introduced/exposed by
commit780138b1("alloc_tag: checkmem_profiling_supportinalloc_tag_init")
Oh, I wasn't aware of that commit.
Thanks for pointing it out!
Indeed, prior to 780138b1, it was unconditionally allocated, so it shouldn't have been a problem unless the allocation fails.
I've sent a formal patch to help testing.
Quoted 1 line
(Before the commit, the BUG would only be triggered when
alloc_tag_initfailed)
That is nearly impossible to trigger as the allocation size is too small to fail, and the allocation is done at boot step, so it shouldn't fail in practice.
Or should we be more paranoid and fix it in v6.12 stable?
Signature · 2 lines
Cheers, Harry / Hyeonggon
PATCH 1/1 lib/alloc_tag: do not acquire nonexistent lock when mem profiling is disabled
In reply to kernel test robotalloc_tag_top_users() attempts to acquire alloc_tag_ctype->mod_lock even when memory allocation profiling feature is disabled at runtime. If the feature is compiled in but not enabled at boot, alloc_tag_init() does not properly allocate and initialize the alloc_tag_cttype variable.
This leads to a crash on memory allocation failure by attempting to acquire a semaphore that does not exist:
Oops: general protection fault, probably for non-canonical address 0xdffffc000000001b: 0000 [#3] SMP KASAN NOPTI KASAN: null-ptr-deref in range [0x00000000000000d8-0x00000000000000df] CPU: 2 UID: 0 PID: 1 Comm: systemd Tainted: G D 6.16.0-rc2 #1 VOLUNTARY Tainted: [D]=DIE Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 RIP: 0010:down_read_trylock+0xaa/0x3b0 Code: d0 7c 08 84 d2 0f 85 a0 02 00 00 8b 0d df 31 dd 04 85 c9 75 29 48 b8 00 00 00 00 00 fc ff df 48 8d 6b 68 48 89 ea 48 c1 ea 03 <80> 3c 02 00 0f 85 88 02 00 00 48 3b 5b 68 0f 85 53 01 00 00 65 ff RSP: 0000:ffff8881002ce9b8 EFLAGS: 00010016 RAX: dffffc0000000000 RBX: 0000000000000070 RCX: 0000000000000000 RDX: 000000000000001b RSI: 000000000000000a RDI: 0000000000000070 RBP: 00000000000000d8 R08: 0000000000000001 R09: ffffed107dde49d1 R10: ffff8883eef24e8b R11: ffff8881002cec20 R12: 1ffff11020059d37 R13: 00000000003fff7b R14: ffff8881002cec20 R15: dffffc0000000000 FS: 00007f963f21d940(0000) GS:ffff888458ca6000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f963f5edf71 CR3: 000000010672c000 CR4: 0000000000350ef0 Call Trace: <TASK> codetag_trylock_module_list+0xd/0x20 alloc_tag_top_users+0x369/0x4b0 __show_mem+0x1cd/0x6e0 warn_alloc+0x2b1/0x390 __alloc_frozen_pages_noprof+0x12b9/0x21a0 alloc_pages_mpol+0x135/0x3e0 alloc_slab_page+0x82/0xe0 new_slab+0x212/0x240 ___slab_alloc+0x82a/0xe00 </TASK>
As David Wang points out, this issue was introduced by commit 780138b12381 ("alloc_tag: check mem_profiling_support in alloc_tag_init"). Before the commit, alloc tagging subsystem unconditionally allocates the semaphore.
After the commit, alloc_tag_top_users() must check whether it was actually initialized. Fix it by adding the appropriate check in alloc_tag_top_users().
---
I manually confirmed that the crash in the vmalloc test module no longer occurs with this patch when the memory profiling feature is compiled but not enabled at boot.
No Cc: stable because the offending commit was introduced in v6.16-rc1.
lib/alloc_tag.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
lib/alloc_tag.c +3 -1
@@ -124,7 +124,9 @@ size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_slstruct codetag_bytes n;unsigned int i, nr =0 ; -if (can_sleep) +if (!mem_profiling_support) +return 0 ; +else if (can_sleep)codetag_lock_module_list (alloc_tag_cttype, true);else if (!codetag_trylock_module_list (alloc_tag_cttype))return 0 ;
Signature · 1 line
2.43.0
Re:[PATCH] lib/alloc_tag: do not acquire nonexistent lock when mem profiling is disabled
Quoted 42 lines · Harry Yoo
alloc_tag_top_users()attempts to acquirealloc_tag_ctype->mod_lockeven when memory allocation profiling feature is disabled at runtime. If the feature is compiled in but not enabled at boot,alloc_tag_init()does not properly allocate and initialize thealloc_tag_cttypevariable.This leads to a crash on memory allocation failure by attempting to acquire a semaphore that does not exist:
Oops: general protection fault, probably for non-canonical address 0xdffffc000000001b: 0000 [#3] SMP KASAN NOPTI KASAN: null-ptr-deref in range [0x00000000000000d8-0x00000000000000df] CPU: 2 UID: 0 PID: 1 Comm: systemd Tainted: G D 6.16.0-rc2 #1 VOLUNTARY Tainted: [D]=DIE Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 RIP: 0010:down_read_trylock+0xaa/0x3b0 Code: d0 7c 08 84 d2 0f 85 a0 02 00 00 8b 0d df 31 dd 04 85 c9 75 29 48 b8 00 00 00 00 00 fc ff df 48 8d 6b 68 48 89 ea 48 c1 ea 03 <80> 3c 02 00 0f 85 88 02 00 00 48 3b 5b 68 0f 85 53 01 00 00 65 ff RSP: 0000:ffff8881002ce9b8 EFLAGS: 00010016 RAX: dffffc0000000000 RBX: 0000000000000070 RCX: 0000000000000000 RDX: 000000000000001b RSI: 000000000000000a RDI: 0000000000000070 RBP: 00000000000000d8 R08: 0000000000000001 R09: ffffed107dde49d1 R10: ffff8883eef24e8b R11: ffff8881002cec20 R12: 1ffff11020059d37 R13: 00000000003fff7b R14: ffff8881002cec20 R15: dffffc0000000000 FS: 00007f963f21d940(0000) GS:ffff888458ca6000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f963f5edf71 CR3: 000000010672c000 CR4: 0000000000350ef0 Call Trace: <TASK> codetag_trylock_module_list+0xd/0x20 alloc_tag_top_users+0x369/0x4b0 __show_mem+0x1cd/0x6e0 warn_alloc+0x2b1/0x390 __alloc_frozen_pages_noprof+0x12b9/0x21a0 alloc_pages_mpol+0x135/0x3e0 alloc_slab_page+0x82/0xe0 new_slab+0x212/0x240 ___slab_alloc+0x82a/0xe00 </TASK>As David Wang points out, this issue was introduced by commit
780138b12381("alloc_tag: checkmem_profiling_supportinalloc_tag_init"). Before the commit, alloc tagging subsystem unconditionally allocates the semaphore.
Quoted 6 lines
After the commit,
alloc_tag_top_users()must check whether it was actually initialized. Fix it by adding the appropriate check inalloc_tag_top_users().
I am not quite sure this can be closed, according to the config file
https://download.01.org/0day-ci/archive/20250618/[email protected]/config-6.15.0-rc6-00142-g2d76e79315e4
CONFIG_MEM_ALLOC_PROFILING=y CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=y <--- CONFIG_MEM_ALLOC_PROFILING_DEBUG=y
mem_profiling_support is true on boot, and alloc_tag_ctype is properly initialized.
Maybe there is other issue lurking somewhere....
Quoted 1 line
This one should not be closed, because "# CONFIG_MEM_ALLOC_PROFILING is not set".
https://download.01.org/0day-ci/archive/20250507/[email protected]/config-6.15.0-rc2-00491-g7fc85b92db96
Quoted 30 lines
Fixes 780138b12381 ("alloc_tag: check mem_profiling_support in alloc_tag_init")Signed-off-by Harry Yoo <[email protected]>---
I manually confirmed that the crash in the vmalloc test module no longer occurs with this patch when the memory profiling feature is compiled but not enabled at boot.
No Cc: stable because the offending commit was introduced in v6.16-rc1.
lib/alloc_tag.c| 4 +++-1 file changed, 3 insertions(+), 1 deletion(-)lib/alloc_tag.c +3
-1@@ -124,7 +124,9 @@ size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_slstruct codetag_bytes n;unsigned int i, nr =0 ; -if (can_sleep) +if (!mem_profiling_support) +return 0 ; +else if (can_sleep)codetag_lock_module_list (alloc_tag_cttype, true);else if (!codetag_trylock_module_list (alloc_tag_cttype))return 0 ;Signature · 1 line
2.43.0
Re: [PATCH] lib/alloc_tag: do not acquire nonexistent lock when mem profiling is disabled
Quoted 55 lines · David Wang
Quoted 42 lines · Harry Yoo
alloc_tag_top_users()attempts to acquirealloc_tag_ctype->mod_lockeven when memory allocation profiling feature is disabled at runtime. If the feature is compiled in but not enabled at boot,alloc_tag_init()does not properly allocate and initialize thealloc_tag_cttypevariable.This leads to a crash on memory allocation failure by attempting to acquire a semaphore that does not exist:
Oops: general protection fault, probably for non-canonical address 0xdffffc000000001b: 0000 [#3] SMP KASAN NOPTI KASAN: null-ptr-deref in range [0x00000000000000d8-0x00000000000000df] CPU: 2 UID: 0 PID: 1 Comm: systemd Tainted: G D 6.16.0-rc2 #1 VOLUNTARY Tainted: [D]=DIE Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 RIP: 0010:down_read_trylock+0xaa/0x3b0 Code: d0 7c 08 84 d2 0f 85 a0 02 00 00 8b 0d df 31 dd 04 85 c9 75 29 48 b8 00 00 00 00 00 fc ff df 48 8d 6b 68 48 89 ea 48 c1 ea 03 <80> 3c 02 00 0f 85 88 02 00 00 48 3b 5b 68 0f 85 53 01 00 00 65 ff RSP: 0000:ffff8881002ce9b8 EFLAGS: 00010016 RAX: dffffc0000000000 RBX: 0000000000000070 RCX: 0000000000000000 RDX: 000000000000001b RSI: 000000000000000a RDI: 0000000000000070 RBP: 00000000000000d8 R08: 0000000000000001 R09: ffffed107dde49d1 R10: ffff8883eef24e8b R11: ffff8881002cec20 R12: 1ffff11020059d37 R13: 00000000003fff7b R14: ffff8881002cec20 R15: dffffc0000000000 FS: 00007f963f21d940(0000) GS:ffff888458ca6000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f963f5edf71 CR3: 000000010672c000 CR4: 0000000000350ef0 Call Trace: <TASK> codetag_trylock_module_list+0xd/0x20 alloc_tag_top_users+0x369/0x4b0 __show_mem+0x1cd/0x6e0 warn_alloc+0x2b1/0x390 __alloc_frozen_pages_noprof+0x12b9/0x21a0 alloc_pages_mpol+0x135/0x3e0 alloc_slab_page+0x82/0xe0 new_slab+0x212/0x240 ___slab_alloc+0x82a/0xe00 </TASK>As David Wang points out, this issue was introduced by commit
780138b12381("alloc_tag: checkmem_profiling_supportinalloc_tag_init"). Before the commit, alloc tagging subsystem unconditionally allocates the semaphore.Quoted 6 lines
After the commit,
alloc_tag_top_users()must check whether it was actually initialized. Fix it by adding the appropriate check inalloc_tag_top_users().I am not quite sure this can be closed, according to the config file
https://download.01.org/0day-ci/archive/20250618/[email protected]/config-6.15.0-rc6-00142-g2d76e79315e4
Quoted 7 lines
CONFIG_MEM_ALLOC_PROFILING=yCONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=y<---CONFIG_MEM_ALLOC_PROFILING_DEBUG=y
mem_profiling_supportis true on boot, andalloc_tag_ctypeis properly initialized.Maybe there is other issue lurking somewhere....
Oops, I thought they are all the same issues.
I should have been more thorough and checked the config. Thank you for pointing it out!
I think you're right. mem_profiling_support == true doesn't necessarily mean it's allocated and initialized, as you demonstrated it in the other email.
I think it'd be more robust to set mem_profiling_support to false, disable mem_alloc_profiling_key at boot and enable it later when it is properly allocated.
Quoted 4 lines
Quoted 1 line
This one should not be closed, because "#
CONFIG_MEM_ALLOC_PROFILINGis not set".
https://download.01.org/0day-ci/archive/20250507/[email protected]/config-6.15.0-rc2-00491-g7fc85b92db96
I assumed it was mem profiling that caused the crash, since it happened while printing memory info. Pretty weird coincidence...
I'll try to reproduce it and figure out why it crashed.
Quoted 30 lines
Quoted 30 lines
Fixes 780138b12381 ("alloc_tag: check mem_profiling_support in alloc_tag_init")Signed-off-by Harry Yoo <[email protected]>---
I manually confirmed that the crash in the vmalloc test module no longer occurs with this patch when the memory profiling feature is compiled but not enabled at boot.
No Cc: stable because the offending commit was introduced in v6.16-rc1.
lib/alloc_tag.c| 4 +++-1 file changed, 3 insertions(+), 1 deletion(-)lib/alloc_tag.c +3
-1@@ -124,7 +124,9 @@ size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_slstruct codetag_bytes n;unsigned int i, nr =0 ; -if (can_sleep) +if (!mem_profiling_support) +return 0 ; +else if (can_sleep)codetag_lock_module_list (alloc_tag_cttype, true);else if (!codetag_trylock_module_list (alloc_tag_cttype))return 0 ;Signature · 1 line
2.43.0
Signature · 2 lines
Cheers, Harry / Hyeonggon
Re: [PATCH] lib/alloc_tag: do not acquire nonexistent lock when mem profiling is disabled
Quoted 68 lines · Harry Yoo
Quoted 55 lines · David Wang
Quoted 42 lines · Harry Yoo
alloc_tag_top_users()attempts to acquirealloc_tag_ctype->mod_lockeven when memory allocation profiling feature is disabled at runtime. If the feature is compiled in but not enabled at boot,alloc_tag_init()does not properly allocate and initialize thealloc_tag_cttypevariable.This leads to a crash on memory allocation failure by attempting to acquire a semaphore that does not exist:
Oops: general protection fault, probably for non-canonical address 0xdffffc000000001b: 0000 [#3] SMP KASAN NOPTI KASAN: null-ptr-deref in range [0x00000000000000d8-0x00000000000000df] CPU: 2 UID: 0 PID: 1 Comm: systemd Tainted: G D 6.16.0-rc2 #1 VOLUNTARY Tainted: [D]=DIE Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 RIP: 0010:down_read_trylock+0xaa/0x3b0 Code: d0 7c 08 84 d2 0f 85 a0 02 00 00 8b 0d df 31 dd 04 85 c9 75 29 48 b8 00 00 00 00 00 fc ff df 48 8d 6b 68 48 89 ea 48 c1 ea 03 <80> 3c 02 00 0f 85 88 02 00 00 48 3b 5b 68 0f 85 53 01 00 00 65 ff RSP: 0000:ffff8881002ce9b8 EFLAGS: 00010016 RAX: dffffc0000000000 RBX: 0000000000000070 RCX: 0000000000000000 RDX: 000000000000001b RSI: 000000000000000a RDI: 0000000000000070 RBP: 00000000000000d8 R08: 0000000000000001 R09: ffffed107dde49d1 R10: ffff8883eef24e8b R11: ffff8881002cec20 R12: 1ffff11020059d37 R13: 00000000003fff7b R14: ffff8881002cec20 R15: dffffc0000000000 FS: 00007f963f21d940(0000) GS:ffff888458ca6000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f963f5edf71 CR3: 000000010672c000 CR4: 0000000000350ef0 Call Trace: <TASK> codetag_trylock_module_list+0xd/0x20 alloc_tag_top_users+0x369/0x4b0 __show_mem+0x1cd/0x6e0 warn_alloc+0x2b1/0x390 __alloc_frozen_pages_noprof+0x12b9/0x21a0 alloc_pages_mpol+0x135/0x3e0 alloc_slab_page+0x82/0xe0 new_slab+0x212/0x240 ___slab_alloc+0x82a/0xe00 </TASK>As David Wang points out, this issue was introduced by commit
780138b12381("alloc_tag: checkmem_profiling_supportinalloc_tag_init"). Before the commit, alloc tagging subsystem unconditionally allocates the semaphore.Quoted 6 lines
After the commit,
alloc_tag_top_users()must check whether it was actually initialized. Fix it by adding the appropriate check inalloc_tag_top_users().I am not quite sure this can be closed, according to the config file
https://download.01.org/0day-ci/archive/20250618/[email protected]/config-6.15.0-rc6-00142-g2d76e79315e4Quoted 7 lines
CONFIG_MEM_ALLOC_PROFILING=yCONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=y<---CONFIG_MEM_ALLOC_PROFILING_DEBUG=y
mem_profiling_supportis true on boot, andalloc_tag_ctypeis properly initialized.Maybe there is other issue lurking somewhere....
Oops, I thought they are all the same issues.
I should have been more thorough and checked the config. Thank you for pointing it out!
Quoted 7 lines
I think you're right.
mem_profiling_support== true doesn't necessarily mean it's allocated and initialized, as you demonstrated it in the other email.I think it'd be more robust to set
mem_profiling_supportto false, disablemem_alloc_profiling_keyat boot and enable it later when it is properly allocated.
Actually, we need something a bit more sophiscated than that. IIUC memory allocation is accounted even before alloc_tag_init(), and the logic depends on mem_alloc_profiling_key being enabled. If we change that, some allocations during early boot stage won't be accounted.
I think we need to introduce a separate variable to indicate whether alloc_tag_init() has completed its initialization and check that in alloc_tag_top_users().
Quoted 45 lines
Quoted 4 lines
Quoted 1 line
This one should not be closed, because "#
CONFIG_MEM_ALLOC_PROFILINGis not set".
https://download.01.org/0day-ci/archive/20250507/[email protected]/config-6.15.0-rc2-00491-g7fc85b92db96I assumed it was mem profiling that caused the crash, since it happened while printing memory info. Pretty weird coincidence...
I'll try to reproduce it and figure out why it crashed.
Quoted 30 lines
Quoted 30 lines
Fixes 780138b12381 ("alloc_tag: check mem_profiling_support in alloc_tag_init")Signed-off-by Harry Yoo <[email protected]>---
I manually confirmed that the crash in the vmalloc test module no longer occurs with this patch when the memory profiling feature is compiled but not enabled at boot.
No Cc: stable because the offending commit was introduced in v6.16-rc1.
lib/alloc_tag.c| 4 +++-1 file changed, 3 insertions(+), 1 deletion(-)lib/alloc_tag.c +3
-1@@ -124,7 +124,9 @@ size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_slstruct codetag_bytes n;unsigned int i, nr =0 ; -if (can_sleep) +if (!mem_profiling_support) +return 0 ; +else if (can_sleep)codetag_lock_module_list (alloc_tag_cttype, true);else if (!codetag_trylock_module_list (alloc_tag_cttype))return 0 ;Signature · 1 line
2.43.0Signature · 2 lines
Cheers, Harry / Hyeonggon
Signature · 2 lines
Cheers, Harry / Hyeonggon
Re: [PATCH] lib/alloc_tag: do not acquire nonexistent lock when mem profiling is disabled
Quoted 83 lines · Harry Yoo
Quoted 68 lines · Harry Yoo
Quoted 55 lines · David Wang
Quoted 42 lines · Harry Yoo
alloc_tag_top_users()attempts to acquirealloc_tag_ctype->mod_lockeven when memory allocation profiling feature is disabled at runtime. If the feature is compiled in but not enabled at boot,alloc_tag_init()does not properly allocate and initialize thealloc_tag_cttypevariable.This leads to a crash on memory allocation failure by attempting to acquire a semaphore that does not exist:
Oops: general protection fault, probably for non-canonical address 0xdffffc000000001b: 0000 [#3] SMP KASAN NOPTI KASAN: null-ptr-deref in range [0x00000000000000d8-0x00000000000000df] CPU: 2 UID: 0 PID: 1 Comm: systemd Tainted: G D 6.16.0-rc2 #1 VOLUNTARY Tainted: [D]=DIE Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 RIP: 0010:down_read_trylock+0xaa/0x3b0 Code: d0 7c 08 84 d2 0f 85 a0 02 00 00 8b 0d df 31 dd 04 85 c9 75 29 48 b8 00 00 00 00 00 fc ff df 48 8d 6b 68 48 89 ea 48 c1 ea 03 <80> 3c 02 00 0f 85 88 02 00 00 48 3b 5b 68 0f 85 53 01 00 00 65 ff RSP: 0000:ffff8881002ce9b8 EFLAGS: 00010016 RAX: dffffc0000000000 RBX: 0000000000000070 RCX: 0000000000000000 RDX: 000000000000001b RSI: 000000000000000a RDI: 0000000000000070 RBP: 00000000000000d8 R08: 0000000000000001 R09: ffffed107dde49d1 R10: ffff8883eef24e8b R11: ffff8881002cec20 R12: 1ffff11020059d37 R13: 00000000003fff7b R14: ffff8881002cec20 R15: dffffc0000000000 FS: 00007f963f21d940(0000) GS:ffff888458ca6000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f963f5edf71 CR3: 000000010672c000 CR4: 0000000000350ef0 Call Trace: <TASK> codetag_trylock_module_list+0xd/0x20 alloc_tag_top_users+0x369/0x4b0 __show_mem+0x1cd/0x6e0 warn_alloc+0x2b1/0x390 __alloc_frozen_pages_noprof+0x12b9/0x21a0 alloc_pages_mpol+0x135/0x3e0 alloc_slab_page+0x82/0xe0 new_slab+0x212/0x240 ___slab_alloc+0x82a/0xe00 </TASK>As David Wang points out, this issue was introduced by commit
780138b12381("alloc_tag: checkmem_profiling_supportinalloc_tag_init"). Before the commit, alloc tagging subsystem unconditionally allocates the semaphore.Quoted 6 lines
After the commit,
alloc_tag_top_users()must check whether it was actually initialized. Fix it by adding the appropriate check inalloc_tag_top_users().I am not quite sure this can be closed, according to the config file
https://download.01.org/0day-ci/archive/20250618/[email protected]/config-6.15.0-rc6-00142-g2d76e79315e4Quoted 7 lines
CONFIG_MEM_ALLOC_PROFILING=yCONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=y<---CONFIG_MEM_ALLOC_PROFILING_DEBUG=y
mem_profiling_supportis true on boot, andalloc_tag_ctypeis properly initialized.Maybe there is other issue lurking somewhere....
Oops, I thought they are all the same issues.
I should have been more thorough and checked the config. Thank you for pointing it out!Quoted 7 lines
I think you're right.
mem_profiling_support== true doesn't necessarily mean it's allocated and initialized, as you demonstrated it in the other email.I think it'd be more robust to set
mem_profiling_supportto false, disablemem_alloc_profiling_keyat boot and enable it later when it is properly allocated.Actually, we need something a bit more sophiscated than that. IIUC memory allocation is accounted even before
alloc_tag_init(), and the logic depends onmem_alloc_profiling_keybeing enabled. If we change that, some allocations during early boot stage won't be accounted.
Quoted 3 lines
I think we need to introduce a separate variable to indicate whether
alloc_tag_init()has completed its initialization and check that inalloc_tag_top_users().
An easy out would be demote vmalloc_test_init to late_initcall, I feel that is reasonable for a test module
(I am preparing a patch for that, will send it out after some tests) Together with this patch, we would be in good shape in most cases. About OOM(mem_show()) during boot before alloc_tag_init,
not sure whether that should raise concern or not, OOM during boot put system in bad shape already.....
Quoted 50 lines
Quoted 46 lines
Quoted 5 lines
Quoted 2 lines
This one should not be closed, because "#
CONFIG_MEM_ALLOC_PROFILINGis not set".
https://download.01.org/0day-ci/archive/20250507/[email protected]/config-6.15.0-rc2-00491-g7fc85b92db96I assumed it was mem profiling that caused the crash, since it happened while printing memory info. Pretty weird coincidence...
I'll try to reproduce it and figure out why it crashed.
Quoted 30 lines
Quoted 30 lines
Fixes 780138b12381 ("alloc_tag: check mem_profiling_support in alloc_tag_init")Signed-off-by Harry Yoo <[email protected]>---
I manually confirmed that the crash in the vmalloc test module no longer occurs with this patch when the memory profiling feature is compiled but not enabled at boot.
No Cc: stable because the offending commit was introduced in v6.16-rc1.
lib/alloc_tag.c| 4 +++-1 file changed, 3 insertions(+), 1 deletion(-)lib/alloc_tag.c +3
-1@@ -124,7 +124,9 @@ size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_slstruct codetag_bytes n;unsigned int i, nr =0 ; -if (can_sleep) +if (!mem_profiling_support) +return 0 ; +else if (can_sleep)codetag_lock_module_list (alloc_tag_cttype, true);else if (!codetag_trylock_module_list (alloc_tag_cttype))return 0 ;Signature · 1 line
2.43.0Signature · 2 lines
Cheers, Harry / HyeonggonSignature · 2 lines
Cheers, Harry / Hyeonggon
Re: [PATCH] lib/alloc_tag: do not acquire nonexistent lock when mem profiling is disabled
In reply to Harry YooQuoted 78 lines · Harry Yoo
Quoted 55 lines · David Wang
Quoted 42 lines · Harry Yoo
alloc_tag_top_users()attempts to acquirealloc_tag_ctype->mod_lockeven when memory allocation profiling feature is disabled at runtime. If the feature is compiled in but not enabled at boot,alloc_tag_init()does not properly allocate and initialize thealloc_tag_cttypevariable.This leads to a crash on memory allocation failure by attempting to acquire a semaphore that does not exist:
Oops: general protection fault, probably for non-canonical address 0xdffffc000000001b: 0000 [#3] SMP KASAN NOPTI KASAN: null-ptr-deref in range [0x00000000000000d8-0x00000000000000df] CPU: 2 UID: 0 PID: 1 Comm: systemd Tainted: G D 6.16.0-rc2 #1 VOLUNTARY Tainted: [D]=DIE Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 RIP: 0010:down_read_trylock+0xaa/0x3b0 Code: d0 7c 08 84 d2 0f 85 a0 02 00 00 8b 0d df 31 dd 04 85 c9 75 29 48 b8 00 00 00 00 00 fc ff df 48 8d 6b 68 48 89 ea 48 c1 ea 03 <80> 3c 02 00 0f 85 88 02 00 00 48 3b 5b 68 0f 85 53 01 00 00 65 ff RSP: 0000:ffff8881002ce9b8 EFLAGS: 00010016 RAX: dffffc0000000000 RBX: 0000000000000070 RCX: 0000000000000000 RDX: 000000000000001b RSI: 000000000000000a RDI: 0000000000000070 RBP: 00000000000000d8 R08: 0000000000000001 R09: ffffed107dde49d1 R10: ffff8883eef24e8b R11: ffff8881002cec20 R12: 1ffff11020059d37 R13: 00000000003fff7b R14: ffff8881002cec20 R15: dffffc0000000000 FS: 00007f963f21d940(0000) GS:ffff888458ca6000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f963f5edf71 CR3: 000000010672c000 CR4: 0000000000350ef0 Call Trace: <TASK> codetag_trylock_module_list+0xd/0x20 alloc_tag_top_users+0x369/0x4b0 __show_mem+0x1cd/0x6e0 warn_alloc+0x2b1/0x390 __alloc_frozen_pages_noprof+0x12b9/0x21a0 alloc_pages_mpol+0x135/0x3e0 alloc_slab_page+0x82/0xe0 new_slab+0x212/0x240 ___slab_alloc+0x82a/0xe00 </TASK>As David Wang points out, this issue was introduced by commit
780138b12381("alloc_tag: checkmem_profiling_supportinalloc_tag_init"). Before the commit, alloc tagging subsystem unconditionally allocates the semaphore.Quoted 6 lines
After the commit,
alloc_tag_top_users()must check whether it was actually initialized. Fix it by adding the appropriate check inalloc_tag_top_users().I am not quite sure this can be closed, according to the config file
https://download.01.org/0day-ci/archive/20250618/[email protected]/config-6.15.0-rc6-00142-g2d76e79315e4Quoted 7 lines
CONFIG_MEM_ALLOC_PROFILING=yCONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=y<---CONFIG_MEM_ALLOC_PROFILING_DEBUG=y
mem_profiling_supportis true on boot, andalloc_tag_ctypeis properly initialized.Maybe there is other issue lurking somewhere....
Oops, I thought they are all the same issues.
I should have been more thorough and checked the config. Thank you for pointing it out!I think you're right.
mem_profiling_support== true doesn't necessarily mean it's allocated and initialized, as you demonstrated it in the other email.I think it'd be more robust to set
mem_profiling_supportto false, disablemem_alloc_profiling_keyat boot and enable it later when it is properly allocated.Quoted 2 lines
Quoted 1 line
Quoted 7 lines
Quoted 2 lines
This one should not be closed, because "#
CONFIG_MEM_ALLOC_PROFILINGis not set".
https://download.01.org/0day-ci/archive/20250507/[email protected]/config-6.15.0-rc2-00491-g7fc85b92db96I assumed it was mem profiling that caused the crash, since it happened while printing memory info. Pretty weird coincidence...
I'll try to reproduce it and figure out why it crashed.
I think this one is not a bug :)
I reproduced it with the config provided and it just takes ~900 seconds to boot because the vmalloc test module does performance testing... and I think the boot test just timed out.
Signature · 2 lines
Cheers, Harry / Hyeonggon
Quoted 35 lines
Quoted 30 lines
Quoted 30 lines
Fixes 780138b12381 ("alloc_tag: check mem_profiling_support in alloc_tag_init")Signed-off-by Harry Yoo <[email protected]>---
I manually confirmed that the crash in the vmalloc test module no longer occurs with this patch when the memory profiling feature is compiled but not enabled at boot.
No Cc: stable because the offending commit was introduced in v6.16-rc1.
lib/alloc_tag.c| 4 +++-1 file changed, 3 insertions(+), 1 deletion(-)lib/alloc_tag.c +3
-1@@ -124,7 +124,9 @@ size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_slstruct codetag_bytes n;unsigned int i, nr =0 ; -if (can_sleep) +if (!mem_profiling_support) +return 0 ; +else if (can_sleep)codetag_lock_module_list (alloc_tag_cttype, true);else if (!codetag_trylock_module_list (alloc_tag_cttype))return 0 ;Signature · 1 line
2.43.0Signature · 2 lines
Cheers, Harry / Hyeonggon
CONFIG_TEST_VMALLOC=y conflict/race with alloc_tag_init
In reply to kernel test robotQuoted 39 lines · kernel test robot
Hello,
for this change, we reported
"[linux-next:master] [lib/test_vmalloc.c]7fc85b92db: Mem-Info" in
https://lore.kernel.org/all/[email protected]/at that time, we made some tests with
x86_64config which runs well.now we noticed the commit is in mainline now.
the config still has expected diff with parent:
pkg/linux/x86_64-randconfig-161-20250614/gcc-12/2d76e79315e403aab595d4c8830b7a46c19f0f3b/.config 2025-06-17 14:41:18.448543738 +0800 renamed from pkg/linux/x86_64-randconfig-161-20250614/gcc-12/7a73348e5d4715b5565a53f21c01ea7b54e46cbd/.config 2025-06-17 14:40:29.481052101 +0800 +1
-1@@ -7551,7 +7551,7 @@ CONFIG_TEST_IDA=m CONFIG_TEST_MISC_MINOR=m # CONFIG_TEST_LKM is not set CONFIG_TEST_BITOPS=m -CONFIG_TEST_VMALLOC=m +CONFIG_TEST_VMALLOC=y # CONFIG_TEST_BPF is not set CONFIG_FIND_BIT_BENCHMARK=m # CONFIG_TEST_FIRMWARE is not setthen we noticed similar random issue with
x86_64randconfig this time.
7a73348e5d4715b52d76e79315e403aab595d4c8830
---------------- ---------------------------
fail:runs %reproduction fail:runs
| | |
:199 34% 67:200 dmesg.KASAN:null-ptr-deref_in_range[#-#]
:199 34% 67:200 dmesg.Kernel_panic-not_syncing:Fatal_exception
:199 34% 67:200 dmesg.Mem-Info
:199 34% 67:200 dmesg.Oops:general_protection_fault,probably_for_non-canonical_address#:#[##]SMP_KASAN
:199 34% 67:200 dmesg.RIP:down_read_trylock
Quoted 3 lines
we don't have enough knowledge to understand the relationship between code change and the random issues. just report what we obsverved in our tests FYI.
I think this is caused by a race between vmalloc_test_init and alloc_tag_init.
vmalloc_test actually depends on alloc_tag via alloc_tag_top_users, because when memory allocation fails show_mem() would invoke alloc_tag_top_users.
With following configuration:
CONFIG_TEST_VMALLOC=yCONFIG_MEM_ALLOC_PROFILING=yCONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=yCONFIG_MEM_ALLOC_PROFILING_DEBUG=y
If vmalloc_test_init starts before alloc_tag_init, show_mem() would cause a NULL deference because alloc_tag_cttype was not init yet.
I add some debug to confirm this theory
lib/alloc_tag.c +3 -0
@@ -133,6 +133,8 @@ size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_slstruct codetag *ct;struct codetag_bytes n;unsigned int i, nr =0 ; +pr_info ("memory profiling alloc top %d: %llx\n" , mem_profiling_support, (long long )alloc_tag_cttype); +return 0 ;if (can_sleep)codetag_lock_module_list (alloc_tag_cttype, true);
@@ -831,6 +833,7 @@ static int __init alloc_tag_init(void)shutdown_mem_profiling (true);return PTR_ERR (alloc_tag_cttype); } +pr_info ("memory profiling ready %d: %llx\n" , mem_profiling_support, (long long )alloc_tag_cttype);return 0 ; }
When bootup the kernel, the log shows:
$ sudo dmesg -T | grep profiling [Fri Jun 20 17:29:35 2025] memory profiling alloc top 1: 0 <--- alloc_tag_cttype == NULL [Fri Jun 20 17:30:24 2025] memory profiling ready 1: ffff9b1641aa06c0
vmalloc_test_init should happened after alloc_tag_init if CONFIG_TEST_VMALLOC=y, or mem_show() should check whether alloc_tag is done initialized when calling alloc_tag_top_users
David
Re: CONFIG_TEST_VMALLOC=y conflict/race with alloc_tag_init
Quoted 85 lines · David Wang
Quoted 39 lines · kernel test robot
Hello,
for this change, we reported
"[linux-next:master] [lib/test_vmalloc.c]7fc85b92db: Mem-Info" in
https://lore.kernel.org/all/[email protected]/at that time, we made some tests with
x86_64config which runs well.now we noticed the commit is in mainline now.
the config still has expected diff with parent:
pkg/linux/x86_64-randconfig-161-20250614/gcc-12/2d76e79315e403aab595d4c8830b7a46c19f0f3b/.config 2025-06-17 14:41:18.448543738 +0800 renamed from pkg/linux/x86_64-randconfig-161-20250614/gcc-12/7a73348e5d4715b5565a53f21c01ea7b54e46cbd/.config 2025-06-17 14:40:29.481052101 +0800 +1
-1@@ -7551,7 +7551,7 @@ CONFIG_TEST_IDA=m CONFIG_TEST_MISC_MINOR=m # CONFIG_TEST_LKM is not set CONFIG_TEST_BITOPS=m -CONFIG_TEST_VMALLOC=m +CONFIG_TEST_VMALLOC=y # CONFIG_TEST_BPF is not set CONFIG_FIND_BIT_BENCHMARK=m # CONFIG_TEST_FIRMWARE is not setthen we noticed similar random issue with
x86_64randconfig this time.
7a73348e5d4715b52d76e79315e403aab595d4c8830
---------------- ---------------------------
fail:runs %reproduction fail:runs
| | |
:199 34% 67:200 dmesg.KASAN:null-ptr-deref_in_range[#-#]
:199 34% 67:200 dmesg.Kernel_panic-not_syncing:Fatal_exception
:199 34% 67:200 dmesg.Mem-Info
:199 34% 67:200 dmesg.Oops:general_protection_fault,probably_for_non-canonical_address#:#[##]SMP_KASAN
:199 34% 67:200 dmesg.RIP:down_read_trylockQuoted 3 lines
we don't have enough knowledge to understand the relationship between code change and the random issues. just report what we obsverved in our tests FYI.
I think this is caused by a race between
vmalloc_test_initandalloc_tag_init.
vmalloc_testactually depends onalloc_tagviaalloc_tag_top_users, because when memory allocation failsshow_mem()would invokealloc_tag_top_users.With following configuration:
CONFIG_TEST_VMALLOC=yCONFIG_MEM_ALLOC_PROFILING=yCONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=yCONFIG_MEM_ALLOC_PROFILING_DEBUG=yIf
vmalloc_test_initstarts beforealloc_tag_init,show_mem()would cause a NULL deference becausealloc_tag_cttypewas not init yet.I add some debug to confirm this theory
lib/alloc_tag.c +3
-0@@ -133,6 +133,8 @@ size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_slstruct codetag *ct;struct codetag_bytes n;unsigned int i, nr =0 ; +pr_info ("memory profiling alloc top %d: %llx\n" , mem_profiling_support, (long long )alloc_tag_cttype); +return 0 ;if (can_sleep)codetag_lock_module_list (alloc_tag_cttype, true);@@ -831,6 +833,7 @@ static int __init alloc_tag_init(void)shutdown_mem_profiling (true);return PTR_ERR (alloc_tag_cttype); } +pr_info ("memory profiling ready %d: %llx\n" , mem_profiling_support, (long long )alloc_tag_cttype);return 0 ; }When bootup the kernel, the log shows:
Quoted 8 lines
$ sudo dmesg -T | grep profiling [Fri Jun 20 17:29:35 2025] memory profiling alloc top 1: 0 <--- alloc_tag_cttype == NULL [Fri Jun 20 17:30:24 2025] memory profiling ready 1: ffff9b1641aa06c0
vmalloc_test_initshould happened afteralloc_tag_initifCONFIG_TEST_VMALLOC=y, ormem_show()should check whetheralloc_tagis done initialized when callingalloc_tag_top_users
Thanks for reporting!
So, IIUC https://lore.kernel.org/all/[email protected]/
will address this issue as well. Is that correct?
Quoted 5 lines
David
Re: CONFIG_TEST_VMALLOC=y conflict/race with alloc_tag_init
Quoted 91 lines · Suren Baghdasaryan
Quoted 90 lines · David Wang
Quoted 39 lines · kernel test robot
Hello,
for this change, we reported
"[linux-next:master] [lib/test_vmalloc.c]7fc85b92db: Mem-Info" in
https://urldefense.com/v3/__https://lore.kernel.org/all/[email protected]/__;!!ACWV5N9M2RV99hQ!LY3bHD8lW73pDdoyiPE87NlpBt6nrJCqoSCm7mxOX2M5tOiT__0NF9Hs2Qm0otnk8D6kx9-OrbpZWVI$at that time, we made some tests with
x86_64config which runs well.now we noticed the commit is in mainline now.
the config still has expected diff with parent:
pkg/linux/x86_64-randconfig-161-20250614/gcc-12/2d76e79315e403aab595d4c8830b7a46c19f0f3b/.config 2025-06-17 14:41:18.448543738 +0800 renamed from pkg/linux/x86_64-randconfig-161-20250614/gcc-12/7a73348e5d4715b5565a53f21c01ea7b54e46cbd/.config 2025-06-17 14:40:29.481052101 +0800 +1
-1@@ -7551,7 +7551,7 @@ CONFIG_TEST_IDA=m CONFIG_TEST_MISC_MINOR=m # CONFIG_TEST_LKM is not set CONFIG_TEST_BITOPS=m -CONFIG_TEST_VMALLOC=m +CONFIG_TEST_VMALLOC=y # CONFIG_TEST_BPF is not set CONFIG_FIND_BIT_BENCHMARK=m # CONFIG_TEST_FIRMWARE is not setthen we noticed similar random issue with
x86_64randconfig this time.
7a73348e5d4715b52d76e79315e403aab595d4c8830
---------------- ---------------------------
fail:runs %reproduction fail:runs
| | |
:199 34% 67:200 dmesg.KASAN:null-ptr-deref_in_range[#-#]
:199 34% 67:200 dmesg.Kernel_panic-not_syncing:Fatal_exception
:199 34% 67:200 dmesg.Mem-Info
:199 34% 67:200 dmesg.Oops:general_protection_fault,probably_for_non-canonical_address#:#[##]SMP_KASAN
:199 34% 67:200 dmesg.RIP:down_read_trylockQuoted 3 lines
we don't have enough knowledge to understand the relationship between code change and the random issues. just report what we obsverved in our tests FYI.
I think this is caused by a race between
vmalloc_test_initandalloc_tag_init.
vmalloc_testactually depends onalloc_tagviaalloc_tag_top_users, because when memory allocation failsshow_mem()would invokealloc_tag_top_users.With following configuration:
CONFIG_TEST_VMALLOC=yCONFIG_MEM_ALLOC_PROFILING=yCONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=yCONFIG_MEM_ALLOC_PROFILING_DEBUG=yIf
vmalloc_test_initstarts beforealloc_tag_init,show_mem()would cause a NULL deference becausealloc_tag_cttypewas not init yet.I add some debug to confirm this theory
lib/alloc_tag.c +3
-0@@ -133,6 +133,8 @@ size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_slstruct codetag *ct;struct codetag_bytes n;unsigned int i, nr =0 ; +pr_info ("memory profiling alloc top %d: %llx\n" , mem_profiling_support, (long long )alloc_tag_cttype); +return 0 ;if (can_sleep)codetag_lock_module_list (alloc_tag_cttype, true);@@ -831,6 +833,7 @@ static int __init alloc_tag_init(void)shutdown_mem_profiling (true);return PTR_ERR (alloc_tag_cttype); } +pr_info ("memory profiling ready %d: %llx\n" , mem_profiling_support, (long long )alloc_tag_cttype);return 0 ; }When bootup the kernel, the log shows:
$ sudo dmesg -T | grep profiling [Fri Jun 20 17:29:35 2025] memory profiling alloc top 1: 0 <--- alloc_tag_cttype == NULL [Fri Jun 20 17:30:24 2025] memory profiling ready 1: ffff9b1641aa06c0
Quoted 7 lines
Quoted 3 lines
vmalloc_test_initshould happened afteralloc_tag_initifCONFIG_TEST_VMALLOC=y, ormem_show()should check whetheralloc_tagis done initialized when callingalloc_tag_top_usersThanks for reporting!
So, IIUC https://lore.kernel.org/all/[email protected]/
will address this issue as well. Is that correct?
Yes, I verified that it addresses this issue.
Quoted 3 lines
Quoted 3 lines
David
Signature · 2 lines
Cheers, Harry / Hyeonggon
Re: CONFIG_TEST_VMALLOC=y conflict/race with alloc_tag_init
In reply to Suren BaghdasaryanQuoted 91 lines · Suren Baghdasaryan
Quoted 90 lines · David Wang
Quoted 39 lines · kernel test robot
Hello,
for this change, we reported
"[linux-next:master] [lib/test_vmalloc.c]7fc85b92db: Mem-Info" in
https://lore.kernel.org/all/[email protected]/at that time, we made some tests with
x86_64config which runs well.now we noticed the commit is in mainline now.
the config still has expected diff with parent:
pkg/linux/x86_64-randconfig-161-20250614/gcc-12/2d76e79315e403aab595d4c8830b7a46c19f0f3b/.config 2025-06-17 14:41:18.448543738 +0800 renamed from pkg/linux/x86_64-randconfig-161-20250614/gcc-12/7a73348e5d4715b5565a53f21c01ea7b54e46cbd/.config 2025-06-17 14:40:29.481052101 +0800 +1
-1@@ -7551,7 +7551,7 @@ CONFIG_TEST_IDA=m CONFIG_TEST_MISC_MINOR=m # CONFIG_TEST_LKM is not set CONFIG_TEST_BITOPS=m -CONFIG_TEST_VMALLOC=m +CONFIG_TEST_VMALLOC=y # CONFIG_TEST_BPF is not set CONFIG_FIND_BIT_BENCHMARK=m # CONFIG_TEST_FIRMWARE is not setthen we noticed similar random issue with
x86_64randconfig this time.
7a73348e5d4715b52d76e79315e403aab595d4c8830
---------------- ---------------------------
fail:runs %reproduction fail:runs
| | |
:199 34% 67:200 dmesg.KASAN:null-ptr-deref_in_range[#-#]
:199 34% 67:200 dmesg.Kernel_panic-not_syncing:Fatal_exception
:199 34% 67:200 dmesg.Mem-Info
:199 34% 67:200 dmesg.Oops:general_protection_fault,probably_for_non-canonical_address#:#[##]SMP_KASAN
:199 34% 67:200 dmesg.RIP:down_read_trylockQuoted 3 lines
we don't have enough knowledge to understand the relationship between code change and the random issues. just report what we obsverved in our tests FYI.
I think this is caused by a race between
vmalloc_test_initandalloc_tag_init.
vmalloc_testactually depends onalloc_tagviaalloc_tag_top_users, because when memory allocation failsshow_mem()would invokealloc_tag_top_users.With following configuration:
CONFIG_TEST_VMALLOC=yCONFIG_MEM_ALLOC_PROFILING=yCONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=yCONFIG_MEM_ALLOC_PROFILING_DEBUG=yIf
vmalloc_test_initstarts beforealloc_tag_init,show_mem()would cause a NULL deference becausealloc_tag_cttypewas not init yet.I add some debug to confirm this theory
lib/alloc_tag.c +3
-0@@ -133,6 +133,8 @@ size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_slstruct codetag *ct;struct codetag_bytes n;unsigned int i, nr =0 ; +pr_info ("memory profiling alloc top %d: %llx\n" , mem_profiling_support, (long long )alloc_tag_cttype); +return 0 ;if (can_sleep)codetag_lock_module_list (alloc_tag_cttype, true);@@ -831,6 +833,7 @@ static int __init alloc_tag_init(void)shutdown_mem_profiling (true);return PTR_ERR (alloc_tag_cttype); } +pr_info ("memory profiling ready %d: %llx\n" , mem_profiling_support, (long long )alloc_tag_cttype);return 0 ; }When bootup the kernel, the log shows:
$ sudo dmesg -T | grep profiling [Fri Jun 20 17:29:35 2025] memory profiling alloc top 1: 0 <--- alloc_tag_cttype == NULL [Fri Jun 20 17:30:24 2025] memory profiling ready 1: ffff9b1641aa06c0
Quoted 7 lines
Quoted 3 lines
vmalloc_test_initshould happened afteralloc_tag_initifCONFIG_TEST_VMALLOC=y, ormem_show()should check whetheralloc_tagis done initialized when callingalloc_tag_top_usersThanks for reporting!
So, IIUC https://lore.kernel.org/all/[email protected]/
will address this issue as well. Is that correct?
Yes, the panic can be fix by that patch.
I still feel it better to delay vmalloc_test_init, make it happen after alloc_tag_init. Or, maybe we can promote alloc_tag_init to some early init? I remember reporting some allocation not registered by memory profiling during boot,
https://lore.kernel.org/all/[email protected]/
I will make some tests, and update later
David
Quoted 6 lines
Quoted 6 lines
David
Re: CONFIG_TEST_VMALLOC=y conflict/race with alloc_tag_init
Quoted 101 lines · David Wang
Quoted 91 lines · Suren Baghdasaryan
Quoted 90 lines · David Wang
Quoted 39 lines · kernel test robot
Hello,
for this change, we reported
"[linux-next:master] [lib/test_vmalloc.c]7fc85b92db: Mem-Info" in
https://lore.kernel.org/all/[email protected]/at that time, we made some tests with
x86_64config which runs well.now we noticed the commit is in mainline now.
the config still has expected diff with parent:
pkg/linux/x86_64-randconfig-161-20250614/gcc-12/2d76e79315e403aab595d4c8830b7a46c19f0f3b/.config 2025-06-17 14:41:18.448543738 +0800 renamed from pkg/linux/x86_64-randconfig-161-20250614/gcc-12/7a73348e5d4715b5565a53f21c01ea7b54e46cbd/.config 2025-06-17 14:40:29.481052101 +0800 +1
-1@@ -7551,7 +7551,7 @@ CONFIG_TEST_IDA=m CONFIG_TEST_MISC_MINOR=m # CONFIG_TEST_LKM is not set CONFIG_TEST_BITOPS=m -CONFIG_TEST_VMALLOC=m +CONFIG_TEST_VMALLOC=y # CONFIG_TEST_BPF is not set CONFIG_FIND_BIT_BENCHMARK=m # CONFIG_TEST_FIRMWARE is not setthen we noticed similar random issue with
x86_64randconfig this time.
7a73348e5d4715b52d76e79315e403aab595d4c8830
---------------- ---------------------------
fail:runs %reproduction fail:runs
| | |
:199 34% 67:200 dmesg.KASAN:null-ptr-deref_in_range[#-#]
:199 34% 67:200 dmesg.Kernel_panic-not_syncing:Fatal_exception
:199 34% 67:200 dmesg.Mem-Info
:199 34% 67:200 dmesg.Oops:general_protection_fault,probably_for_non-canonical_address#:#[##]SMP_KASAN
:199 34% 67:200 dmesg.RIP:down_read_trylockQuoted 3 lines
we don't have enough knowledge to understand the relationship between code change and the random issues. just report what we obsverved in our tests FYI.
I think this is caused by a race between
vmalloc_test_initandalloc_tag_init.
vmalloc_testactually depends onalloc_tagviaalloc_tag_top_users, because when memory allocation failsshow_mem()would invokealloc_tag_top_users.With following configuration:
CONFIG_TEST_VMALLOC=yCONFIG_MEM_ALLOC_PROFILING=yCONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=yCONFIG_MEM_ALLOC_PROFILING_DEBUG=yIf
vmalloc_test_initstarts beforealloc_tag_init,show_mem()would cause a NULL deference becausealloc_tag_cttypewas not init yet.I add some debug to confirm this theory
lib/alloc_tag.c +3
-0@@ -133,6 +133,8 @@ size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_slstruct codetag *ct;struct codetag_bytes n;unsigned int i, nr =0 ; +pr_info ("memory profiling alloc top %d: %llx\n" , mem_profiling_support, (long long )alloc_tag_cttype); +return 0 ;if (can_sleep)codetag_lock_module_list (alloc_tag_cttype, true);@@ -831,6 +833,7 @@ static int __init alloc_tag_init(void)shutdown_mem_profiling (true);return PTR_ERR (alloc_tag_cttype); } +pr_info ("memory profiling ready %d: %llx\n" , mem_profiling_support, (long long )alloc_tag_cttype);return 0 ; }When bootup the kernel, the log shows:
$ sudo dmesg -T | grep profiling [Fri Jun 20 17:29:35 2025] memory profiling alloc top 1: 0 <--- alloc_tag_cttype == NULL [Fri Jun 20 17:30:24 2025] memory profiling ready 1: ffff9b1641aa06c0Quoted 7 lines
Quoted 3 lines
vmalloc_test_initshould happened afteralloc_tag_initifCONFIG_TEST_VMALLOC=y, ormem_show()should check whetheralloc_tagis done initialized when callingalloc_tag_top_usersThanks for reporting!
So, IIUC https://lore.kernel.org/all/[email protected]/
will address this issue as well. Is that correct?
Quoted 8 lines
Yes, the panic can be fix by that patch.
I still feel it better to delay
vmalloc_test_init, make it happen afteralloc_tag_init. Or, maybe we can promotealloc_tag_initto some early init? I remember reporting some allocation not registered by memory profiling during boot,
https://lore.kernel.org/all/[email protected]/I will make some tests, and update later
The memory allocations in sched_init_domains happened quite early, maybe it is core_initcall, while
alloc_tag_init needs rootfs, it needs to be after rootfs_initcall, so no reasonable place to promote.......
But I think this explain why some allocation counter missed during boot: the allocation happened before alloc_tag_init
Thanks
David
Quoted 11 lines
David
Quoted 6 lines
Quoted 6 lines
David
Re: CONFIG_TEST_VMALLOC=y conflict/race with alloc_tag_init
Quoted 110 lines · David Wang
Quoted 108 lines · David Wang
Quoted 91 lines · Suren Baghdasaryan
Quoted 90 lines · David Wang
Quoted 39 lines · kernel test robot
Hello,
for this change, we reported
"[linux-next:master] [lib/test_vmalloc.c]7fc85b92db: Mem-Info" in
https://lore.kernel.org/all/[email protected]/at that time, we made some tests with
x86_64config which runs well.now we noticed the commit is in mainline now.
the config still has expected diff with parent:
pkg/linux/x86_64-randconfig-161-20250614/gcc-12/2d76e79315e403aab595d4c8830b7a46c19f0f3b/.config 2025-06-17 14:41:18.448543738 +0800 renamed from pkg/linux/x86_64-randconfig-161-20250614/gcc-12/7a73348e5d4715b5565a53f21c01ea7b54e46cbd/.config 2025-06-17 14:40:29.481052101 +0800 +1
-1@@ -7551,7 +7551,7 @@ CONFIG_TEST_IDA=m CONFIG_TEST_MISC_MINOR=m # CONFIG_TEST_LKM is not set CONFIG_TEST_BITOPS=m -CONFIG_TEST_VMALLOC=m +CONFIG_TEST_VMALLOC=y # CONFIG_TEST_BPF is not set CONFIG_FIND_BIT_BENCHMARK=m # CONFIG_TEST_FIRMWARE is not setthen we noticed similar random issue with
x86_64randconfig this time.
7a73348e5d4715b52d76e79315e403aab595d4c8830
---------------- ---------------------------
fail:runs %reproduction fail:runs
| | |
:199 34% 67:200 dmesg.KASAN:null-ptr-deref_in_range[#-#]
:199 34% 67:200 dmesg.Kernel_panic-not_syncing:Fatal_exception
:199 34% 67:200 dmesg.Mem-Info
:199 34% 67:200 dmesg.Oops:general_protection_fault,probably_for_non-canonical_address#:#[##]SMP_KASAN
:199 34% 67:200 dmesg.RIP:down_read_trylockQuoted 3 lines
we don't have enough knowledge to understand the relationship between code change and the random issues. just report what we obsverved in our tests FYI.
I think this is caused by a race between
vmalloc_test_initandalloc_tag_init.
vmalloc_testactually depends onalloc_tagviaalloc_tag_top_users, because when memory allocation failsshow_mem()would invokealloc_tag_top_users.With following configuration:
CONFIG_TEST_VMALLOC=yCONFIG_MEM_ALLOC_PROFILING=yCONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=yCONFIG_MEM_ALLOC_PROFILING_DEBUG=yIf
vmalloc_test_initstarts beforealloc_tag_init,show_mem()would cause a NULL deference becausealloc_tag_cttypewas not init yet.I add some debug to confirm this theory
lib/alloc_tag.c +3
-0@@ -133,6 +133,8 @@ size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_slstruct codetag *ct;struct codetag_bytes n;unsigned int i, nr =0 ; +pr_info ("memory profiling alloc top %d: %llx\n" , mem_profiling_support, (long long )alloc_tag_cttype); +return 0 ;if (can_sleep)codetag_lock_module_list (alloc_tag_cttype, true);@@ -831,6 +833,7 @@ static int __init alloc_tag_init(void)shutdown_mem_profiling (true);return PTR_ERR (alloc_tag_cttype); } +pr_info ("memory profiling ready %d: %llx\n" , mem_profiling_support, (long long )alloc_tag_cttype);return 0 ; }When bootup the kernel, the log shows:
$ sudo dmesg -T | grep profiling [Fri Jun 20 17:29:35 2025] memory profiling alloc top 1: 0 <--- alloc_tag_cttype == NULL [Fri Jun 20 17:30:24 2025] memory profiling ready 1: ffff9b1641aa06c0Quoted 7 lines
Quoted 3 lines
vmalloc_test_initshould happened afteralloc_tag_initifCONFIG_TEST_VMALLOC=y, ormem_show()should check whetheralloc_tagis done initialized when callingalloc_tag_top_usersThanks for reporting!
So, IIUC https://lore.kernel.org/all/[email protected]/
will address this issue as well. Is that correct?Yes, the panic can be fix by that patch.
I still feel it better to delay
vmalloc_test_init, make it happen afteralloc_tag_init. Or, maybe we can promotealloc_tag_initto some early init? I remember reporting some allocation not registered by memory profiling during boot,
https://lore.kernel.org/all/[email protected]/
Quoted 5 lines
Quoted 1 line
I will make some tests, and update later
The memory allocations in
sched_init_domainshappened quite early, maybe it iscore_initcall, whilealloc_tag_initneeds rootfs, it needs to be afterrootfs_initcall, so no reasonable place to promote....... But I think this explain why some allocation counter missed during boot: the allocation happened beforealloc_tag_init
..... Sorry, I think I was wrong..... The counters does not need alloc_tag_init...
sorry for bothering, please ignore my mumbo jumbo.
David
Quoted 16 lines
Thanks
DavidQuoted 11 lines
David
Quoted 6 lines
Quoted 6 lines
David
Uladzislau Rezki
maintainerRe: CONFIG_TEST_VMALLOC=y conflict/race with alloc_tag_init
In reply to David WangQuoted 101 lines · David Wang
Quoted 91 lines · Suren Baghdasaryan
Quoted 90 lines · David Wang
Quoted 39 lines · kernel test robot
Hello,
for this change, we reported
"[linux-next:master] [lib/test_vmalloc.c]7fc85b92db: Mem-Info" in
https://lore.kernel.org/all/[email protected]/at that time, we made some tests with
x86_64config which runs well.now we noticed the commit is in mainline now.
the config still has expected diff with parent:
pkg/linux/x86_64-randconfig-161-20250614/gcc-12/2d76e79315e403aab595d4c8830b7a46c19f0f3b/.config 2025-06-17 14:41:18.448543738 +0800 renamed from pkg/linux/x86_64-randconfig-161-20250614/gcc-12/7a73348e5d4715b5565a53f21c01ea7b54e46cbd/.config 2025-06-17 14:40:29.481052101 +0800 +1
-1@@ -7551,7 +7551,7 @@ CONFIG_TEST_IDA=m CONFIG_TEST_MISC_MINOR=m # CONFIG_TEST_LKM is not set CONFIG_TEST_BITOPS=m -CONFIG_TEST_VMALLOC=m +CONFIG_TEST_VMALLOC=y # CONFIG_TEST_BPF is not set CONFIG_FIND_BIT_BENCHMARK=m # CONFIG_TEST_FIRMWARE is not setthen we noticed similar random issue with
x86_64randconfig this time.
7a73348e5d4715b52d76e79315e403aab595d4c8830
---------------- ---------------------------
fail:runs %reproduction fail:runs
| | |
:199 34% 67:200 dmesg.KASAN:null-ptr-deref_in_range[#-#]
:199 34% 67:200 dmesg.Kernel_panic-not_syncing:Fatal_exception
:199 34% 67:200 dmesg.Mem-Info
:199 34% 67:200 dmesg.Oops:general_protection_fault,probably_for_non-canonical_address#:#[##]SMP_KASAN
:199 34% 67:200 dmesg.RIP:down_read_trylockQuoted 3 lines
we don't have enough knowledge to understand the relationship between code change and the random issues. just report what we obsverved in our tests FYI.
I think this is caused by a race between
vmalloc_test_initandalloc_tag_init.
vmalloc_testactually depends onalloc_tagviaalloc_tag_top_users, because when memory allocation failsshow_mem()would invokealloc_tag_top_users.With following configuration:
CONFIG_TEST_VMALLOC=yCONFIG_MEM_ALLOC_PROFILING=yCONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=yCONFIG_MEM_ALLOC_PROFILING_DEBUG=yIf
vmalloc_test_initstarts beforealloc_tag_init,show_mem()would cause a NULL deference becausealloc_tag_cttypewas not init yet.I add some debug to confirm this theory
lib/alloc_tag.c +3
-0@@ -133,6 +133,8 @@ size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_slstruct codetag *ct;struct codetag_bytes n;unsigned int i, nr =0 ; +pr_info ("memory profiling alloc top %d: %llx\n" , mem_profiling_support, (long long )alloc_tag_cttype); +return 0 ;if (can_sleep)codetag_lock_module_list (alloc_tag_cttype, true);@@ -831,6 +833,7 @@ static int __init alloc_tag_init(void)shutdown_mem_profiling (true);return PTR_ERR (alloc_tag_cttype); } +pr_info ("memory profiling ready %d: %llx\n" , mem_profiling_support, (long long )alloc_tag_cttype);return 0 ; }When bootup the kernel, the log shows:
$ sudo dmesg -T | grep profiling [Fri Jun 20 17:29:35 2025] memory profiling alloc top 1: 0 <--- alloc_tag_cttype == NULL [Fri Jun 20 17:30:24 2025] memory profiling ready 1: ffff9b1641aa06c0Quoted 7 lines
Quoted 3 lines
vmalloc_test_initshould happened afteralloc_tag_initifCONFIG_TEST_VMALLOC=y, ormem_show()should check whetheralloc_tagis done initialized when callingalloc_tag_top_usersThanks for reporting!
So, IIUC https://lore.kernel.org/all/[email protected]/
will address this issue as well. Is that correct?
Quoted 4 lines
Yes, the panic can be fix by that patch.
I still feel it better to delay
vmalloc_test_init, make it happen afteralloc_tag_init.
We can, but then we would not notice the bag that is in question :)
At least we should, i think, to exclude the tests which trigger warnings when the test-suite is run with default configurations, i.e. run the tests which are not supposed to fail.
--
Uladzislau Rezki
Re: CONFIG_TEST_VMALLOC=y conflict/race with alloc_tag_init
Quoted 102 lines · Uladzislau Rezki
Quoted 101 lines · David Wang
Quoted 98 lines · Suren Baghdasaryan
Quoted 85 lines · David Wang
Quoted 39 lines · kernel test robot
Hello,
for this change, we reported
"[linux-next:master] [lib/test_vmalloc.c]7fc85b92db: Mem-Info" in
https://lore.kernel.org/all/[email protected]/at that time, we made some tests with
x86_64config which runs well.now we noticed the commit is in mainline now.
the config still has expected diff with parent:
pkg/linux/x86_64-randconfig-161-20250614/gcc-12/2d76e79315e403aab595d4c8830b7a46c19f0f3b/.config 2025-06-17 14:41:18.448543738 +0800 renamed from pkg/linux/x86_64-randconfig-161-20250614/gcc-12/7a73348e5d4715b5565a53f21c01ea7b54e46cbd/.config 2025-06-17 14:40:29.481052101 +0800 +1
-1@@ -7551,7 +7551,7 @@ CONFIG_TEST_IDA=m CONFIG_TEST_MISC_MINOR=m # CONFIG_TEST_LKM is not set CONFIG_TEST_BITOPS=m -CONFIG_TEST_VMALLOC=m +CONFIG_TEST_VMALLOC=y # CONFIG_TEST_BPF is not set CONFIG_FIND_BIT_BENCHMARK=m # CONFIG_TEST_FIRMWARE is not setthen we noticed similar random issue with
x86_64randconfig this time.
7a73348e5d4715b52d76e79315e403aab595d4c8830
---------------- ---------------------------
fail:runs %reproduction fail:runs
| | |
:199 34% 67:200 dmesg.KASAN:null-ptr-deref_in_range[#-#]
:199 34% 67:200 dmesg.Kernel_panic-not_syncing:Fatal_exception
:199 34% 67:200 dmesg.Mem-Info
:199 34% 67:200 dmesg.Oops:general_protection_fault,probably_for_non-canonical_address#:#[##]SMP_KASAN
:199 34% 67:200 dmesg.RIP:down_read_trylockQuoted 3 lines
we don't have enough knowledge to understand the relationship between code change and the random issues. just report what we obsverved in our tests FYI.
I think this is caused by a race between
vmalloc_test_initandalloc_tag_init.
vmalloc_testactually depends onalloc_tagviaalloc_tag_top_users, because when memory allocation failsshow_mem()would invokealloc_tag_top_users.With following configuration:
CONFIG_TEST_VMALLOC=yCONFIG_MEM_ALLOC_PROFILING=yCONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=yCONFIG_MEM_ALLOC_PROFILING_DEBUG=yIf
vmalloc_test_initstarts beforealloc_tag_init,show_mem()would cause a NULL deference becausealloc_tag_cttypewas not init yet.I add some debug to confirm this theory
lib/alloc_tag.c +3
-0@@ -133,6 +133,8 @@ size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_slstruct codetag *ct;struct codetag_bytes n;unsigned int i, nr =0 ; +pr_info ("memory profiling alloc top %d: %llx\n" , mem_profiling_support, (long long )alloc_tag_cttype); +return 0 ;if (can_sleep)codetag_lock_module_list (alloc_tag_cttype, true);@@ -831,6 +833,7 @@ static int __init alloc_tag_init(void)shutdown_mem_profiling (true);return PTR_ERR (alloc_tag_cttype); } +pr_info ("memory profiling ready %d: %llx\n" , mem_profiling_support, (long long )alloc_tag_cttype);return 0 ; }When bootup the kernel, the log shows:
Quoted 8 lines
$ sudo dmesg -T | grep profiling [Fri Jun 20 17:29:35 2025] memory profiling alloc top 1: 0 <--- alloc_tag_cttype == NULL [Fri Jun 20 17:30:24 2025] memory profiling ready 1: ffff9b1641aa06c0
vmalloc_test_initshould happened afteralloc_tag_initifCONFIG_TEST_VMALLOC=y, ormem_show()should check whetheralloc_tagis done initialized when callingalloc_tag_top_usersThanks for reporting!
So, IIUC https://lore.kernel.org/all/[email protected]/
will address this issue as well. Is that correct?
Quoted 5 lines
Quoted 4 lines
Yes, the panic can be fix by that patch.
I still feel it better to delay
vmalloc_test_init, make it happen afteralloc_tag_init.We can, but then we would not notice the bag that is in question :)
Yes, strangely lucky here~ :)
I was thinking, if some vmalloc tests fail, is alloc_tag_top_users helpful for debug? Considering this bug has already been caught, if alloc_tag_top_users is helpful for vmalloc test analysis, maybe it is still reasonable to delay vmalloc_test_init?... ☺︎
Quoted 10 lines
At least we should, i think, to exclude the tests which trigger warnings when the test-suite is run with default configurations, i.e. run the tests which are not supposed to fail.
--
Uladzislau Rezki
PATCH 1/1 lib/test_vmalloc.c: demote vmalloc_test_init to late_initcall
In reply to kernel test robotCommit 2d76e79315e4 ("lib/test_vmalloc.c: allow built-in execution") enable test_vmalloc module to be built into kernel directly, but vmalloc_test_init depends on alloc_tag module via alloc_tag_top_users().
When a kernel build with following config:
CONFIG_TEST_VMALLOC=yCONFIG_MEM_ALLOC_PROFILING=yCONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=yCONFIG_MEM_ALLOC_PROFILING_DEBUG=y
If vmalloc_test_init() run before alloc_tag_init(), memory failure tests would invoke alloc_tag_top_users() which is not ready to use and cause kernel BUG:
[ 135.116045] BUG: kernel NULL pointer dereference, address: 0000000000000030 [ 135.116063] #PF: supervisor read access in kernel mode [ 135.116074] #PF: error_code(0x0000) - not-present page [ 135.116085] PGD 0 P4D 0 [ 135.116094] Oops: Oops: 0000 [#1] SMP NOPTI [ 135.116123] Tainted: [E]=UNSIGNED_MODULE [ 135.116132] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 [ 135.116148] RIP: 0010:down_read_trylock+0x1d/0x80 [ 135.116188] RSP: 0000:ffffb5e481a9b8f8 EFLAGS: 00010246 [ 135.116200] RAX: ffff93dc8a5ac700 RBX: 0000000000000030 RCX: 8000000000000007 [ 135.116214] RDX: 0000000000000001 RSI: 000000000000000a RDI: ffffffff93d2e733 [ 135.116228] RBP: ffffb5e481a9b9a0 R08: 0000000000000000 R09: 0000000000000003 [ 135.116241] R10: ffffb5e481a9b860 R11: ffffffff94ec6328 R12: ffffb5e481a9b9b0 [ 135.116255] R13: 0000000000000003 R14: 0000000000000001 R15: ffffffff94e0c580 [ 135.116271] FS: 00007fd41947e540(0000) GS:ffff93dd6654a000(0000) knlGS:0000000000000000 [ 135.116286] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 135.116298] CR2: 0000000000000030 CR3: 00000001099f8000 CR4: 0000000000350ef0 [ 135.116314] Call Trace: [ 135.116321] <TASK> [ 135.116328] codetag_trylock_module_list+0x9/0x20 [ 135.116342] alloc_tag_top_users+0x153/0x1b0 [ 135.116354] ? srso_return_thunk+0x5/0x5f [ 135.116365] ? _printk+0x57/0x80 [ 135.116378] __show_mem+0xeb/0x210 [ 135.116394] ? dump_header+0x2ce/0x3e0 [ 135.116405] dump_header+0x2ce/0x3e0
Demote vmalloc_test_init to late_initcall can make sure alloc_tag module got initialized before test_vmalloc module.
--- lib/test_vmalloc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
lib/test_vmalloc.c +1 -1
@@ -598,7 +598,7 @@ static int __init vmalloc_test_init(void)return IS_BUILTIN (CONFIG_TEST_VMALLOC ) ?0 :-EAGAIN ; } -module_init(vmalloc_test_init) +late_initcall(vmalloc_test_init) MODULE_LICENSE("GPL");MODULE_AUTHOR ("Uladzislau Rezki" );
Signature · 1 line
2.39.2
Re: [PATCH] lib/test_vmalloc.c: demote vmalloc_test_init to late_initcall
Quoted 42 lines · David Wang
Commit
2d76e79315e4("lib/test_vmalloc.c: allow built-in execution") enabletest_vmallocmodule to be built into kernel directly, butvmalloc_test_initdepends onalloc_tagmodule viaalloc_tag_top_users().When a kernel build with following config:
CONFIG_TEST_VMALLOC=yCONFIG_MEM_ALLOC_PROFILING=yCONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=yCONFIG_MEM_ALLOC_PROFILING_DEBUG=yIf
vmalloc_test_init()run beforealloc_tag_init(), memory failure tests would invokealloc_tag_top_users()which is not ready to use and cause kernel BUG:[ 135.116045] BUG: kernel NULL pointer dereference, address: 0000000000000030 [ 135.116063] #PF: supervisor read access in kernel mode [ 135.116074] #PF: error_code(0x0000) - not-present page [ 135.116085] PGD 0 P4D 0 [ 135.116094] Oops: Oops: 0000 [#1] SMP NOPTI [ 135.116123] Tainted: [E]=UNSIGNED_MODULE [ 135.116132] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 [ 135.116148] RIP: 0010:down_read_trylock+0x1d/0x80 [ 135.116188] RSP: 0000:ffffb5e481a9b8f8 EFLAGS: 00010246 [ 135.116200] RAX: ffff93dc8a5ac700 RBX: 0000000000000030 RCX: 8000000000000007 [ 135.116214] RDX: 0000000000000001 RSI: 000000000000000a RDI: ffffffff93d2e733 [ 135.116228] RBP: ffffb5e481a9b9a0 R08: 0000000000000000 R09: 0000000000000003 [ 135.116241] R10: ffffb5e481a9b860 R11: ffffffff94ec6328 R12: ffffb5e481a9b9b0 [ 135.116255] R13: 0000000000000003 R14: 0000000000000001 R15: ffffffff94e0c580 [ 135.116271] FS: 00007fd41947e540(0000) GS:ffff93dd6654a000(0000) knlGS:0000000000000000 [ 135.116286] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 135.116298] CR2: 0000000000000030 CR3: 00000001099f8000 CR4: 0000000000350ef0 [ 135.116314] Call Trace: [ 135.116321] <TASK> [ 135.116328] codetag_trylock_module_list+0x9/0x20 [ 135.116342] alloc_tag_top_users+0x153/0x1b0 [ 135.116354] ? srso_return_thunk+0x5/0x5f [ 135.116365] ? _printk+0x57/0x80 [ 135.116378] __show_mem+0xeb/0x210 [ 135.116394] ? dump_header+0x2ce/0x3e0 [ 135.116405] dump_header+0x2ce/0x3e0
Quoted 2 lines
Demote
vmalloc_test_inittolate_initcallcan make surealloc_tagmodule got initialized beforetest_vmallocmodule.
I'm not sure this is the right place to fix it.
The bug can be triggered by any early memory allocation failure, before alloc_tag_init() is called (yeah, that's not that likely).
There is nothing specific to vmalloc that triggers the bug.
Signature · 2 lines
Cheers, Harry / Hyeonggon
Quoted 26 lines
Reported-by kernel test robot <[email protected]>Fixes 2d76e79315e4 ("lib/test_vmalloc.c: allow built-in execution")Signed-off-by David Wang <[email protected]>--- lib/test_vmalloc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)lib/test_vmalloc.c +1
-1@@ -598,7 +598,7 @@ static int __init vmalloc_test_init(void)return IS_BUILTIN (CONFIG_TEST_VMALLOC ) ?0 :-EAGAIN ; } -module_init(vmalloc_test_init) +late_initcall(vmalloc_test_init) MODULE_LICENSE("GPL");MODULE_AUTHOR ("Uladzislau Rezki" );Signature · 1 line
2.39.2
PATCH v2 1/1 lib/alloc_tag: do not acquire non-existent lock in alloc_tag_top_users()
In reply to kernel test robotalloc_tag_top_users() attempts to lock alloc_tag_cttype->mod_lock even when the alloc_tag_cttype is not allocated because:
1) alloc tagging is disabled because mem profiling is disabled
(!alloc_tag_cttype)
2) alloc tagging is enabled, but not yet initialized (!alloc_tag_cttype)
3) alloc tagging is enabled, but failed initialization (!alloc_tag_cttype or IS_ERR(alloc_tag_cttype))
In all cases, alloc_tag_cttype is not allocated, and therefore alloc_tag_top_users() should not attempt to acquire the semaphore.
This leads to a crash on memory allocation failure by attempting to acquire a non-existent semaphore:
Oops: general protection fault, probably for non-canonical address 0xdffffc000000001b: 0000 [#3] SMP KASAN NOPTI KASAN: null-ptr-deref in range [0x00000000000000d8-0x00000000000000df] CPU: 2 UID: 0 PID: 1 Comm: systemd Tainted: G D 6.16.0-rc2 #1 VOLUNTARY Tainted: [D]=DIE Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 RIP: 0010:down_read_trylock+0xaa/0x3b0 Code: d0 7c 08 84 d2 0f 85 a0 02 00 00 8b 0d df 31 dd 04 85 c9 75 29 48 b8 00 00 00 00 00 fc ff df 48 8d 6b 68 48 89 ea 48 c1 ea 03 <80> 3c 02 00 0f 85 88 02 00 00 48 3b 5b 68 0f 85 53 01 00 00 65 ff RSP: 0000:ffff8881002ce9b8 EFLAGS: 00010016 RAX: dffffc0000000000 RBX: 0000000000000070 RCX: 0000000000000000 RDX: 000000000000001b RSI: 000000000000000a RDI: 0000000000000070 RBP: 00000000000000d8 R08: 0000000000000001 R09: ffffed107dde49d1 R10: ffff8883eef24e8b R11: ffff8881002cec20 R12: 1ffff11020059d37 R13: 00000000003fff7b R14: ffff8881002cec20 R15: dffffc0000000000 FS: 00007f963f21d940(0000) GS:ffff888458ca6000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f963f5edf71 CR3: 000000010672c000 CR4: 0000000000350ef0 Call Trace: <TASK> codetag_trylock_module_list+0xd/0x20 alloc_tag_top_users+0x369/0x4b0 __show_mem+0x1cd/0x6e0 warn_alloc+0x2b1/0x390 __alloc_frozen_pages_noprof+0x12b9/0x21a0 alloc_pages_mpol+0x135/0x3e0 alloc_slab_page+0x82/0xe0 new_slab+0x212/0x240 ___slab_alloc+0x82a/0xe00 </TASK>
As David Wang points out, this issue became easier to trigger after commit 780138b12381 ("alloc_tag: check mem_profiling_support in alloc_tag_init").
Before the commit, the issue occurred only when it failed to allocate and initialize alloc_tag_cttype or if a memory allocation fails before alloc_tag_init() is called. After the commit, it can be easily triggered when memory profiling is compiled but disabled at boot.
To properly determine whether alloc_tag_init() has been called and its data structures initialized, verify that alloc_tag_cttype is a valid pointer before acquiring the semaphore. If the variable is NULL or an error value, it has not been properly initialized. In such a case, just skip and do not attempt acquire the semaphore.
---
v1 -> v2:
- v1 fixed the bug only when MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=n.
v2 now fixes the bug even when MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=y. I didn't expect alloc_tag_cttype to be NULL when mem_profiling_support is true, but as David points out (Thanks David!) if a memory allocation fails before alloc_tag_init(), it can be NULL.
So instead of indirectly checking mem_profiling_support, just directly check if alloc_tag_cttype is allocated.
- Closes: https://lore.kernel.org/oe-lkp/[email protected]
tag was removed because it was not a crash and not relevant to this patch.
- Added Cc: stable because, if an allocation fails before alloc_tag_init(), it can be triggered even prior-780138b12381. I verified that the bug can be triggered in v6.12 and fixed by this patch.
It should be quite difficult to trigger in practice, though. Maybe I'm a bit paranoid?
lib/alloc_tag.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
lib/alloc_tag.c +3 -1
@@ -124,7 +124,9 @@ size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_slstruct codetag_bytes n;unsigned int i, nr =0 ; -if (can_sleep) +if (IS_ERR_OR_NULL (alloc_tag_cttype)) +return 0 ; +else if (can_sleep)codetag_lock_module_list (alloc_tag_cttype, true);else if (!codetag_trylock_module_list (alloc_tag_cttype))return 0 ;
Signature · 1 line
2.43.0
Re:[PATCH v2] lib/alloc_tag: do not acquire non-existent lock in alloc_tag_top_users()
Quoted 58 lines · Harry Yoo
alloc_tag_top_users()attempts to lockalloc_tag_cttype->mod_lockeven when thealloc_tag_cttypeis not allocated because:1) alloc tagging is disabled because mem profiling is disabled
(!alloc_tag_cttype)2) alloc tagging is enabled, but not yet initialized (!alloc_tag_cttype)
3) alloc tagging is enabled, but failed initialization (!alloc_tag_cttype orIS_ERR(alloc_tag_cttype))In all cases,
alloc_tag_cttypeis not allocated, and thereforealloc_tag_top_users()should not attempt to acquire the semaphore.This leads to a crash on memory allocation failure by attempting to acquire a non-existent semaphore:
Oops: general protection fault, probably for non-canonical address 0xdffffc000000001b: 0000 [#3] SMP KASAN NOPTI KASAN: null-ptr-deref in range [0x00000000000000d8-0x00000000000000df] CPU: 2 UID: 0 PID: 1 Comm: systemd Tainted: G D 6.16.0-rc2 #1 VOLUNTARY Tainted: [D]=DIE Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 RIP: 0010:down_read_trylock+0xaa/0x3b0 Code: d0 7c 08 84 d2 0f 85 a0 02 00 00 8b 0d df 31 dd 04 85 c9 75 29 48 b8 00 00 00 00 00 fc ff df 48 8d 6b 68 48 89 ea 48 c1 ea 03 <80> 3c 02 00 0f 85 88 02 00 00 48 3b 5b 68 0f 85 53 01 00 00 65 ff RSP: 0000:ffff8881002ce9b8 EFLAGS: 00010016 RAX: dffffc0000000000 RBX: 0000000000000070 RCX: 0000000000000000 RDX: 000000000000001b RSI: 000000000000000a RDI: 0000000000000070 RBP: 00000000000000d8 R08: 0000000000000001 R09: ffffed107dde49d1 R10: ffff8883eef24e8b R11: ffff8881002cec20 R12: 1ffff11020059d37 R13: 00000000003fff7b R14: ffff8881002cec20 R15: dffffc0000000000 FS: 00007f963f21d940(0000) GS:ffff888458ca6000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f963f5edf71 CR3: 000000010672c000 CR4: 0000000000350ef0 Call Trace: <TASK> codetag_trylock_module_list+0xd/0x20 alloc_tag_top_users+0x369/0x4b0 __show_mem+0x1cd/0x6e0 warn_alloc+0x2b1/0x390 __alloc_frozen_pages_noprof+0x12b9/0x21a0 alloc_pages_mpol+0x135/0x3e0 alloc_slab_page+0x82/0xe0 new_slab+0x212/0x240 ___slab_alloc+0x82a/0xe00 </TASK>As David Wang points out, this issue became easier to trigger after commit
780138b12381("alloc_tag: checkmem_profiling_supportinalloc_tag_init").Before the commit, the issue occurred only when it failed to allocate and initialize
alloc_tag_cttypeor if a memory allocation fails beforealloc_tag_init()is called. After the commit, it can be easily triggered when memory profiling is compiled but disabled at boot.To properly determine whether
alloc_tag_init()has been called and its data structures initialized, verify thatalloc_tag_cttypeis a valid pointer before acquiring the semaphore. If the variable is NULL or an error value, it has not been properly initialized. In such a case, just skip and do not attempt acquire the semaphore.
Quoted 6 lines
Reported-by kernel test robot <[email protected]>Fixes 780138b12381 ("alloc_tag: check mem_profiling_support in alloc_tag_init")Fixes 1438d349d16b ("lib: add memory allocations report in show_mem()")Signed-off-by Harry Yoo <[email protected]>
Just notice another thread can be closed as well:
https://lore.kernel.org/all/[email protected]/
This coincide with scenario #1, where OOM happened with CONFIG_MEM_ALLOC_PROFILING=y
# CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT is not set # CONFIG_MEM_ALLOC_PROFILING_DEBUG is not set
Quoted 46 lines
---
v1 -> v2:
- v1 fixed the bug only when MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=n.
v2 now fixes the bug even when MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=y. I didn't expect
alloc_tag_cttypeto be NULL whenmem_profiling_supportis true, but as David points out (Thanks David!) if a memory allocation fails beforealloc_tag_init(), it can be NULL.So instead of indirectly checking
mem_profiling_support, just directly check ifalloc_tag_cttypeis allocated.- Closes: https://lore.kernel.org/oe-lkp/[email protected]
tag was removed because it was not a crash and not relevant to this patch.- Added Cc: stable because, if an allocation fails before
alloc_tag_init(), it can be triggered even prior-780138b12381. I verified that the bug can be triggered in v6.12 and fixed by this patch.It should be quite difficult to trigger in practice, though. Maybe I'm a bit paranoid?
lib/alloc_tag.c| 4 +++-1 file changed, 3 insertions(+), 1 deletion(-)lib/alloc_tag.c +3
-1@@ -124,7 +124,9 @@ size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_slstruct codetag_bytes n;unsigned int i, nr =0 ; -if (can_sleep) +if (IS_ERR_OR_NULL (alloc_tag_cttype)) +return 0 ; +else if (can_sleep)codetag_lock_module_list (alloc_tag_cttype, true);else if (!codetag_trylock_module_list (alloc_tag_cttype))return 0 ;Signature · 1 line
2.43.0
Re: [PATCH v2] lib/alloc_tag: do not acquire non-existent lock in alloc_tag_top_users()
Quoted 47 lines · David Wang
Quoted 44 lines · Harry Yoo
alloc_tag_top_users()attempts to lockalloc_tag_cttype->mod_lockeven when thealloc_tag_cttypeis not allocated because:1) alloc tagging is disabled because mem profiling is disabled
(!alloc_tag_cttype)2) alloc tagging is enabled, but not yet initialized (!alloc_tag_cttype)
3) alloc tagging is enabled, but failed initialization (!alloc_tag_cttype orIS_ERR(alloc_tag_cttype))In all cases,
alloc_tag_cttypeis not allocated, and thereforealloc_tag_top_users()should not attempt to acquire the semaphore.This leads to a crash on memory allocation failure by attempting to acquire a non-existent semaphore:
Oops: general protection fault, probably for non-canonical address 0xdffffc000000001b: 0000 [#3] SMP KASAN NOPTI KASAN: null-ptr-deref in range [0x00000000000000d8-0x00000000000000df] CPU: 2 UID: 0 PID: 1 Comm: systemd Tainted: G D 6.16.0-rc2 #1 VOLUNTARY Tainted: [D]=DIE Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 RIP: 0010:down_read_trylock+0xaa/0x3b0 Code: d0 7c 08 84 d2 0f 85 a0 02 00 00 8b 0d df 31 dd 04 85 c9 75 29 48 b8 00 00 00 00 00 fc ff df 48 8d 6b 68 48 89 ea 48 c1 ea 03 <80> 3c 02 00 0f 85 88 02 00 00 48 3b 5b 68 0f 85 53 01 00 00 65 ff RSP: 0000:ffff8881002ce9b8 EFLAGS: 00010016 RAX: dffffc0000000000 RBX: 0000000000000070 RCX: 0000000000000000 RDX: 000000000000001b RSI: 000000000000000a RDI: 0000000000000070 RBP: 00000000000000d8 R08: 0000000000000001 R09: ffffed107dde49d1 R10: ffff8883eef24e8b R11: ffff8881002cec20 R12: 1ffff11020059d37 R13: 00000000003fff7b R14: ffff8881002cec20 R15: dffffc0000000000 FS: 00007f963f21d940(0000) GS:ffff888458ca6000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f963f5edf71 CR3: 000000010672c000 CR4: 0000000000350ef0 Call Trace: <TASK> codetag_trylock_module_list+0xd/0x20 alloc_tag_top_users+0x369/0x4b0 __show_mem+0x1cd/0x6e0 warn_alloc+0x2b1/0x390 __alloc_frozen_pages_noprof+0x12b9/0x21a0 alloc_pages_mpol+0x135/0x3e0 alloc_slab_page+0x82/0xe0 new_slab+0x212/0x240 ___slab_alloc+0x82a/0xe00 </TASK>
Quoted 7 lines
Quoted 7 lines
As David Wang points out, this issue became easier to trigger after commit
780138b12381("alloc_tag: checkmem_profiling_supportinalloc_tag_init").Before the commit, the issue occurred only when it failed to allocate and initialize
alloc_tag_cttypeor if a memory allocation fails beforealloc_tag_init()is called. After the commit, it can be easily triggered when memory profiling is compiled but disabled at boot.
Thanks for the fix and sorry about the delay with reviewing it.
Quoted 6 lines
Quoted 6 lines
To properly determine whether
alloc_tag_init()has been called and its data structures initialized, verify thatalloc_tag_cttypeis a valid pointer before acquiring the semaphore. If the variable is NULL or an error value, it has not been properly initialized. In such a case, just skip and do not attempt acquire the semaphore.
nit: s/attempt acquire/attempt to acquire
Quoted 52 lines
Quoted 7 lines
Reported-by kernel test robot <[email protected]>Fixes 780138b12381 ("alloc_tag: check mem_profiling_support in alloc_tag_init")Fixes 1438d349d16b ("lib: add memory allocations report in show_mem()")Signed-off-by Harry Yoo <[email protected]>Just notice another thread can be closed as well:
https://lore.kernel.org/all/[email protected]/
This coincide with scenario #1, where OOM happened withCONFIG_MEM_ALLOC_PROFILING=y
#CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULTis not set #CONFIG_MEM_ALLOC_PROFILING_DEBUGis not setQuoted 37 lines
---
v1 -> v2:
- v1 fixed the bug only when MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=n.
v2 now fixes the bug even when MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=y. I didn't expect
alloc_tag_cttypeto be NULL whenmem_profiling_supportis true, but as David points out (Thanks David!) if a memory allocation fails beforealloc_tag_init(), it can be NULL.So instead of indirectly checking
mem_profiling_support, just directly check ifalloc_tag_cttypeis allocated.- Closes: https://lore.kernel.org/oe-lkp/[email protected]
tag was removed because it was not a crash and not relevant to this patch.- Added Cc: stable because, if an allocation fails before
alloc_tag_init(), it can be triggered even prior-780138b12381. I verified that the bug can be triggered in v6.12 and fixed by this patch.It should be quite difficult to trigger in practice, though. Maybe I'm a bit paranoid?
lib/alloc_tag.c| 4 +++-1 file changed, 3 insertions(+), 1 deletion(-)lib/alloc_tag.c +0
-0@@ -124,7 +124,9 @@ size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_slstruct codetag_bytes n;unsigned int i, nr =0 ;
Quoted 3 lines
lib/alloc_tag.c size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_sl · line 128
Quoted 3 lines
lib/alloc_tag.c +2
-1-if (can_sleep) +if (IS_ERR_OR_NULL (alloc_tag_cttype)) +return 0 ;
So, AFAIKT alloc_tag_cttype will be NULL when memory profiling is disabled and it will be ENOMEM if codetag_register_type() fails. I think it would be good to add a pr_warn() in the alloc_tag_init() when codetag_register_type() fails so that the user can determine the reason why show_mem() report is missing allocation tag information.
Quoted 1 line
lib/alloc_tag.c size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_sl · line 129
Quoted 1 line
lib/alloc_tag.c +1
-0+else if (can_sleep)
nit: the above extra "else" is not really needed. The following should work just fine, is more readable and produces less churn:
+ if (IS_ERR_OR_NULL(alloc_tag_cttype))
+ return 0;
+
if (can_sleep)
codetag_lock_module_list(alloc_tag_cttype, true);
else if (!codetag_trylock_module_list(alloc_tag_cttype))
return 0;Quoted 5 lines
Quoted 5 lines
lib/alloc_tag.c +0
-1codetag_lock_module_list (alloc_tag_cttype, true); else if (!codetag_trylock_module_list(alloc_tag_cttype)) return 0; --2.43.0
Re: [PATCH v2] lib/alloc_tag: do not acquire non-existent lock in alloc_tag_top_users()
Quoted 51 lines · Suren Baghdasaryan
Quoted 50 lines · David Wang
Quoted 47 lines · Harry Yoo
alloc_tag_top_users()attempts to lockalloc_tag_cttype->mod_lockeven when thealloc_tag_cttypeis not allocated because:1) alloc tagging is disabled because mem profiling is disabled
(!alloc_tag_cttype)2) alloc tagging is enabled, but not yet initialized (!alloc_tag_cttype)
3) alloc tagging is enabled, but failed initialization (!alloc_tag_cttype orIS_ERR(alloc_tag_cttype))In all cases,
alloc_tag_cttypeis not allocated, and thereforealloc_tag_top_users()should not attempt to acquire the semaphore.This leads to a crash on memory allocation failure by attempting to acquire a non-existent semaphore:
Oops: general protection fault, probably for non-canonical address 0xdffffc000000001b: 0000 [#3] SMP KASAN NOPTI KASAN: null-ptr-deref in range [0x00000000000000d8-0x00000000000000df] CPU: 2 UID: 0 PID: 1 Comm: systemd Tainted: G D 6.16.0-rc2 #1 VOLUNTARY Tainted: [D]=DIE Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 RIP: 0010:down_read_trylock+0xaa/0x3b0 Code: d0 7c 08 84 d2 0f 85 a0 02 00 00 8b 0d df 31 dd 04 85 c9 75 29 48 b8 00 00 00 00 00 fc ff df 48 8d 6b 68 48 89 ea 48 c1 ea 03 <80> 3c 02 00 0f 85 88 02 00 00 48 3b 5b 68 0f 85 53 01 00 00 65 ff RSP: 0000:ffff8881002ce9b8 EFLAGS: 00010016 RAX: dffffc0000000000 RBX: 0000000000000070 RCX: 0000000000000000 RDX: 000000000000001b RSI: 000000000000000a RDI: 0000000000000070 RBP: 00000000000000d8 R08: 0000000000000001 R09: ffffed107dde49d1 R10: ffff8883eef24e8b R11: ffff8881002cec20 R12: 1ffff11020059d37 R13: 00000000003fff7b R14: ffff8881002cec20 R15: dffffc0000000000 FS: 00007f963f21d940(0000) GS:ffff888458ca6000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f963f5edf71 CR3: 000000010672c000 CR4: 0000000000350ef0 Call Trace: <TASK> codetag_trylock_module_list+0xd/0x20 alloc_tag_top_users+0x369/0x4b0 __show_mem+0x1cd/0x6e0 warn_alloc+0x2b1/0x390 __alloc_frozen_pages_noprof+0x12b9/0x21a0 alloc_pages_mpol+0x135/0x3e0 alloc_slab_page+0x82/0xe0 new_slab+0x212/0x240 ___slab_alloc+0x82a/0xe00 </TASK>As David Wang points out, this issue became easier to trigger after commit
780138b12381("alloc_tag: checkmem_profiling_supportinalloc_tag_init").
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Before the commit, the issue occurred only when it failed to allocate and initialize
alloc_tag_cttypeor if a memory allocation fails beforealloc_tag_init()is called. After the commit, it can be easily triggered when memory profiling is compiled but disabled at boot.Thanks for the fix and sorry about the delay with reviewing it.
No problem ;)
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
To properly determine whether
alloc_tag_init()has been called and its data structures initialized, verify thatalloc_tag_cttypeis a valid pointer before acquiring the semaphore. If the variable is NULL or an error value, it has not been properly initialized. In such a case, just skip and do not attempt acquire the semaphore.nit: s/attempt acquire/attempt to acquire
Will fix the typo.
Quoted 52 lines
Quoted 52 lines
Quoted 7 lines
Reported-by kernel test robot <[email protected]>Fixes 780138b12381 ("alloc_tag: check mem_profiling_support in alloc_tag_init")Fixes 1438d349d16b ("lib: add memory allocations report in show_mem()")Signed-off-by Harry Yoo <[email protected]>Just notice another thread can be closed as well:
https://urldefense.com/v3/__https://lore.kernel.org/all/[email protected]/__;!!ACWV5N9M2RV99hQ!NZv9w8rtFb5ni1zqQs7y8loVNvbrbW3d1pBi4bA_f_Tfh-pegcni0iK5642QuK6FqCBCaOUfSGgkKj0$
This coincide with scenario #1, where OOM happened withCONFIG_MEM_ALLOC_PROFILING=y
#CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULTis not set #CONFIG_MEM_ALLOC_PROFILING_DEBUGis not setQuoted 37 lines
---
v1 -> v2:
- v1 fixed the bug only when MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=n.
v2 now fixes the bug even when MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=y. I didn't expect
alloc_tag_cttypeto be NULL whenmem_profiling_supportis true, but as David points out (Thanks David!) if a memory allocation fails beforealloc_tag_init(), it can be NULL.So instead of indirectly checking
mem_profiling_support, just directly check ifalloc_tag_cttypeis allocated.- Closes: https://urldefense.com/v3/__https://lore.kernel.org/oe-lkp/[email protected]__;!!ACWV5N9M2RV99hQ!NZv9w8rtFb5ni1zqQs7y8loVNvbrbW3d1pBi4bA_f_Tfh-pegcni0iK5642QuK6FqCBCaOUfwfwsQlE$
tag was removed because it was not a crash and not relevant to this
patch.- Added Cc: stable because, if an allocation fails before
alloc_tag_init(), it can be triggered even prior-780138b12381. I verified that the bug can be triggered in v6.12 and fixed by this patch.It should be quite difficult to trigger in practice, though. Maybe I'm a bit paranoid?
lib/alloc_tag.c| 4 +++-1 file changed, 3 insertions(+), 1 deletion(-)lib/alloc_tag.c +0
-0@@ -124,7 +124,9 @@ size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_slstruct codetag_bytes n;unsigned int i, nr =0 ;
Quoted 6 lines
Quoted 3 lines
lib/alloc_tag.c size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_sl · line 128
Quoted 3 lines
lib/alloc_tag.c +2
-1-if (can_sleep) +if (IS_ERR_OR_NULL (alloc_tag_cttype)) +return 0 ;So, AFAIKT
alloc_tag_cttypewill be NULL when memory profiling is disabled and it will be ENOMEM ifcodetag_register_type()fails.
Yes.
Or when memory profiling is enabled, but a memory allocation fails before alloc_tag_init().
Quoted 3 lines
I think it would be good to add a
pr_warn()in thealloc_tag_init()whencodetag_register_type()fails so that the user can determine the reason whyshow_mem()report is missing allocation tag information.
Will do.
lib/alloc_tag.c size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_sl · line 132
Quoted 12 lines
Quoted 1 line
lib/alloc_tag.c size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_sl · line 129
Quoted 1 line
lib/alloc_tag.c +1
-0+else if (can_sleep)nit: the above extra "else" is not really needed. The following should work just fine, is more readable and produces less churn:
lib/alloc_tag.c +3
-0+if (IS_ERR_OR_NULL (alloc_tag_cttype)) +return 0 ; +if (can_sleep)codetag_lock_module_list (alloc_tag_cttype, true);else if (!codetag_trylock_module_list (alloc_tag_cttype))return 0 ;
Will do, thanks!
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
lib/alloc_tag.c +0
-1codetag_lock_module_list (alloc_tag_cttype, true); else if (!codetag_trylock_module_list(alloc_tag_cttype)) return 0; --2.43.0
Signature · 2 lines
Cheers, Harry / Hyeonggon