BISECTED: 'alloc_tag: populate memory for module tags as needed' crashes on boot.
Hello Suren,
My system crashes on bootup, and I bisected to this commit.
0f9b685626daa2f8e19a9788625c9b624c223e45 is the first bad commit commit 0f9b685626daa2f8e19a9788625c9b624c223e45
Author: Suren Baghdasaryan <[email protected]> Date: Wed Oct 23 10:07:57 2024 -0700
alloc_tag: populate memory for module tags as needed
The memory reserved for module tags does not need to be backed by physical pages until there are tags to store there. Change the way we reserve this memory to allocate only virtual area for the tags and populate it with physical pages as needed when we load a module.
The crash looks like this:
BUG: unable to handle page fault for address: fffffbfff4041000 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 44d0e7067 P4D 44d0e7067 PUD 44d0e3067 PMD 10bb38067 PTE 0 Oops: Oops: 0000 [#1] PREEMPT SMP KASAN CPU: 0 UID: 0 PID: 319 Comm: systemd-udevd Not tainted 6.12.0-rc6+ #21 Hardware name: Default string Default string/SKYBAY, BIOS 5.12 02/15/2023 RIP: 0010:kasan_check_range+0xa5/0x190 Code: 8d 5a 07 4c 0f 49 da 49 c1 fb 03 45 85 db 0f 84 ce 00 00 00 45 89 db 4a 8d 14 d8 eb 0d 48 83 c0 08 48 39 d0 0f 84 b29 RSP: 0018:ffff88812c26f980 EFLAGS: 00010206 RAX: fffffbfff4041000 RBX: fffffbfff404101e RCX: ffffffff814ec29b [ OK DX: fffffbfff4041018 RSI: 00000000000000f0 RDI: ffffffffa0208000 0m] Finished BP: fffffbfff4041000 R08: 0000000000000001 R09: fffffbfff404101d ;1;39msystemd-udR10: ffffffffa02080ef R11: 0000000000000003 R12: ffffffffa0208000 ev-trig…e R13: ffffc90000dac7c8 R14: ffffc90000dac7e8 R15: dffffc0000000000 - Coldplug All uFS: 00007fe869216b40(0000) GS:ffff88841da00000(0000) knlGS:0000000000000000 dev Devices. CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: fffffbfff4041000 CR3: 0000000121e86002 CR4: 00000000003706f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: <TASK> [ OK ? __die+0x1f/0x60 0m] Reached targ ? page_fault_oops+0x258/0x910 et sysi ? dump_pagetable+0x690/0x690 nit.target - ? search_bpf_extables+0x22/0x250 System Initiali ? trace_page_fault_kernel+0x120/0x120 zation. ? search_bpf_extables+0x164/0x250 ? kasan_check_range+0xa5/0x190 ? fixup_exception+0x4d/0xc70 ? exc_page_fault+0xe1/0xf0 [ OK ? asm_exc_page_fault+0x22/0x30 0m] Reached targ ? load_module+0x3d7b/0x7560 et netw ? kasan_check_range+0xa5/0x190 ork.target - __asan_memcpy+0x38/0x60 Network. load_module+0x3d7b/0x7560 ? module_frob_arch_sections+0x30/0x30 ? lockdep_lock+0xbe/0x1b0 ? rw_verify_area+0x18d/0x5e0 ? kernel_read_file+0x246/0x870 ? __x64_sys_fspick+0x290/0x290 ? init_module_from_file+0xd1/0x130 init_module_from_file+0xd1/0x130 ? __ia32_sys_init_module+0xa0/0xa0 ? lock_acquire+0x2d/0xb0 ? idempotent_init_module+0x116/0x790 ? do_raw_spin_unlock+0x54/0x220 idempotent_init_module+0x226/0x790 ? init_module_from_file+0x130/0x130 ? vm_mmap_pgoff+0x203/0x2e0 __x64_sys_finit_module+0xba/0x130 do_syscall_64+0x69/0x160 entry_SYSCALL_64_after_hwframe+0x4b/0x53 RIP: 0033:0x7fe869de327d Code: 5d c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 248 RSP: 002b:00007ffe34a828d8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 RAX: ffffffffffffffda RBX: 0000557fa8f3f3f0 RCX: 00007fe869de327d RDX: 0000000000000000 RSI: 00007fe869f4943c RDI: 0000000000000006 RBP: 00007fe869f4943c R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000006 R11: 0000000000000246 R12: 0000000000020000 R13: 0000557fa8f3f030 R14: 0000000000000000 R15: 0000557fa8f3d110 </TASK> Modules linked in: CR2: fffffbfff4041000 ---[ end trace 0000000000000000 ]---
I suspect you only hit this with an unlucky amount of debugging enabled. The kernel config I used is found here:
http://www.candelatech.com/downloads/cfg-kasan-crash-regression.config
I will be happy to test fixes.
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 88 lines · Ben Greear
Hello Suren,
My system crashes on bootup, and I bisected to this commit.
0f9b685626daa2f8e19a9788625c9b624c223e45is the first bad commit commit0f9b685626daa2f8e19a9788625c9b624c223e45Author: Suren Baghdasaryan <[email protected]> Date: Wed Oct 23 10:07:57 2024 -0700
alloc_tag: populate memory for module tags as neededThe memory reserved for module tags does not need to be backed by physical pages until there are tags to store there. Change the way we reserve this memory to allocate only virtual area for the tags and populate it with physical pages as needed when we load a module.
The crash looks like this:
BUG: unable to handle page fault for address: fffffbfff4041000 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 44d0e7067 P4D 44d0e7067 PUD 44d0e3067 PMD 10bb38067 PTE 0 Oops: Oops: 0000 [#1] PREEMPT SMP KASAN CPU: 0 UID: 0 PID: 319 Comm: systemd-udevd Not tainted 6.12.0-rc6+ #21 Hardware name: Default string Default string/SKYBAY, BIOS 5.12 02/15/2023 RIP: 0010:kasan_check_range+0xa5/0x190 Code: 8d 5a 07 4c 0f 49 da 49 c1 fb 03 45 85 db 0f 84 ce 00 00 00 45 89 db 4a 8d 14 d8 eb 0d 48 83 c0 08 48 39 d0 0f 84 b29 RSP: 0018:ffff88812c26f980 EFLAGS: 00010206 RAX: fffffbfff4041000 RBX: fffffbfff404101e RCX: ffffffff814ec29b [ OK DX: fffffbfff4041018 RSI: 00000000000000f0 RDI: ffffffffa0208000 0m] Finished BP: fffffbfff4041000 R08: 0000000000000001 R09: fffffbfff404101d ;1;39msystemd-udR10: ffffffffa02080ef R11: 0000000000000003 R12: ffffffffa0208000 ev-trig…e R13: ffffc90000dac7c8 R14: ffffc90000dac7e8 R15: dffffc0000000000 - Coldplug All uFS: 00007fe869216b40(0000) GS:ffff88841da00000(0000) knlGS:0000000000000000 dev Devices. CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: fffffbfff4041000 CR3: 0000000121e86002 CR4: 00000000003706f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: <TASK> [ OK ? __die+0x1f/0x60 0m] Reached targ ? page_fault_oops+0x258/0x910 et sysi ? dump_pagetable+0x690/0x690 nit.target - ? search_bpf_extables+0x22/0x250 System Initiali ? trace_page_fault_kernel+0x120/0x120 zation. ? search_bpf_extables+0x164/0x250 ? kasan_check_range+0xa5/0x190 ? fixup_exception+0x4d/0xc70 ? exc_page_fault+0xe1/0xf0 [ OK ? asm_exc_page_fault+0x22/0x30 0m] Reached targ ? load_module+0x3d7b/0x7560 et netw ? kasan_check_range+0xa5/0x190 ork.target - __asan_memcpy+0x38/0x60 Network. load_module+0x3d7b/0x7560 ? module_frob_arch_sections+0x30/0x30 ? lockdep_lock+0xbe/0x1b0 ? rw_verify_area+0x18d/0x5e0 ? kernel_read_file+0x246/0x870 ? __x64_sys_fspick+0x290/0x290 ? init_module_from_file+0xd1/0x130 init_module_from_file+0xd1/0x130 ? __ia32_sys_init_module+0xa0/0xa0 ? lock_acquire+0x2d/0xb0 ? idempotent_init_module+0x116/0x790 ? do_raw_spin_unlock+0x54/0x220 idempotent_init_module+0x226/0x790 ? init_module_from_file+0x130/0x130 ? vm_mmap_pgoff+0x203/0x2e0 __x64_sys_finit_module+0xba/0x130 do_syscall_64+0x69/0x160 entry_SYSCALL_64_after_hwframe+0x4b/0x53 RIP: 0033:0x7fe869de327d Code: 5d c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 248 RSP: 002b:00007ffe34a828d8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 RAX: ffffffffffffffda RBX: 0000557fa8f3f3f0 RCX: 00007fe869de327d RDX: 0000000000000000 RSI: 00007fe869f4943c RDI: 0000000000000006 RBP: 00007fe869f4943c R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000006 R11: 0000000000000246 R12: 0000000000020000 R13: 0000557fa8f3f030 R14: 0000000000000000 R15: 0000557fa8f3d110 </TASK> Modules linked in: CR2: fffffbfff4041000 ---[ end trace 0000000000000000 ]---
Quoted 6 lines
I suspect you only hit this with an unlucky amount of debugging enabled. The kernel config I used is found here:
http://www.candelatech.com/downloads/cfg-kasan-crash-regression.config
I will be happy to test fixes.
Hi Ben,
Thanks for reporting the issue. Do you have these recent fixes in your tree:
https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/all/[email protected]/
If not, couple you please apply them and see if the issue is still happening? Thanks,
Suren.
Quoted 8 lines
Thanks,
Ben--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
Quoted 99 lines · Suren Baghdasaryan
Quoted 88 lines · Ben Greear
Hello Suren,
My system crashes on bootup, and I bisected to this commit.
0f9b685626daa2f8e19a9788625c9b624c223e45is the first bad commit commit0f9b685626daa2f8e19a9788625c9b624c223e45Author: Suren Baghdasaryan <[email protected]> Date: Wed Oct 23 10:07:57 2024 -0700
alloc_tag: populate memory for module tags as neededThe memory reserved for module tags does not need to be backed by physical pages until there are tags to store there. Change the way we reserve this memory to allocate only virtual area for the tags and populate it with physical pages as needed when we load a module.
The crash looks like this:
BUG: unable to handle page fault for address: fffffbfff4041000 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 44d0e7067 P4D 44d0e7067 PUD 44d0e3067 PMD 10bb38067 PTE 0 Oops: Oops: 0000 [#1] PREEMPT SMP KASAN CPU: 0 UID: 0 PID: 319 Comm: systemd-udevd Not tainted 6.12.0-rc6+ #21 Hardware name: Default string Default string/SKYBAY, BIOS 5.12 02/15/2023 RIP: 0010:kasan_check_range+0xa5/0x190 Code: 8d 5a 07 4c 0f 49 da 49 c1 fb 03 45 85 db 0f 84 ce 00 00 00 45 89 db 4a 8d 14 d8 eb 0d 48 83 c0 08 48 39 d0 0f 84 b29 RSP: 0018:ffff88812c26f980 EFLAGS: 00010206 RAX: fffffbfff4041000 RBX: fffffbfff404101e RCX: ffffffff814ec29b [ OK DX: fffffbfff4041018 RSI: 00000000000000f0 RDI: ffffffffa0208000 0m] Finished BP: fffffbfff4041000 R08: 0000000000000001 R09: fffffbfff404101d ;1;39msystemd-udR10: ffffffffa02080ef R11: 0000000000000003 R12: ffffffffa0208000 ev-trig…e R13: ffffc90000dac7c8 R14: ffffc90000dac7e8 R15: dffffc0000000000 - Coldplug All uFS: 00007fe869216b40(0000) GS:ffff88841da00000(0000) knlGS:0000000000000000 dev Devices. CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: fffffbfff4041000 CR3: 0000000121e86002 CR4: 00000000003706f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: <TASK> [ OK ? __die+0x1f/0x60 0m] Reached targ ? page_fault_oops+0x258/0x910 et sysi ? dump_pagetable+0x690/0x690 nit.target - ? search_bpf_extables+0x22/0x250 System Initiali ? trace_page_fault_kernel+0x120/0x120 zation. ? search_bpf_extables+0x164/0x250 ? kasan_check_range+0xa5/0x190 ? fixup_exception+0x4d/0xc70 ? exc_page_fault+0xe1/0xf0 [ OK ? asm_exc_page_fault+0x22/0x30 0m] Reached targ ? load_module+0x3d7b/0x7560 et netw ? kasan_check_range+0xa5/0x190 ork.target - __asan_memcpy+0x38/0x60 Network. load_module+0x3d7b/0x7560 ? module_frob_arch_sections+0x30/0x30 ? lockdep_lock+0xbe/0x1b0 ? rw_verify_area+0x18d/0x5e0 ? kernel_read_file+0x246/0x870 ? __x64_sys_fspick+0x290/0x290 ? init_module_from_file+0xd1/0x130 init_module_from_file+0xd1/0x130 ? __ia32_sys_init_module+0xa0/0xa0 ? lock_acquire+0x2d/0xb0 ? idempotent_init_module+0x116/0x790 ? do_raw_spin_unlock+0x54/0x220 idempotent_init_module+0x226/0x790 ? init_module_from_file+0x130/0x130 ? vm_mmap_pgoff+0x203/0x2e0 __x64_sys_finit_module+0xba/0x130 do_syscall_64+0x69/0x160 entry_SYSCALL_64_after_hwframe+0x4b/0x53 RIP: 0033:0x7fe869de327d Code: 5d c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 248 RSP: 002b:00007ffe34a828d8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 RAX: ffffffffffffffda RBX: 0000557fa8f3f3f0 RCX: 00007fe869de327d RDX: 0000000000000000 RSI: 00007fe869f4943c RDI: 0000000000000006 RBP: 00007fe869f4943c R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000006 R11: 0000000000000246 R12: 0000000000020000 R13: 0000557fa8f3f030 R14: 0000000000000000 R15: 0000557fa8f3d110 </TASK> Modules linked in: CR2: fffffbfff4041000 ---[ end trace 0000000000000000 ]---Quoted 6 lines
I suspect you only hit this with an unlucky amount of debugging enabled. The kernel config I used is found here:
http://www.candelatech.com/downloads/cfg-kasan-crash-regression.config
I will be happy to test fixes.
Hi Ben,
Thanks for reporting the issue. Do you have these recent fixes in your tree:
Quoted 6 lines
https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/all/[email protected]/If not, couple you please apply them and see if the issue is still happening? Thanks,
Suren.
Hello Suren,
Thanks for the quick response. The first patch is already in latest Linus tree, and I applied the second one manually. The kernel still crashes:
(gdb) l *(load_module+0x3c46)0xffffffff814f30d6 is in load_module (/home/greearb/git/linux-2.6/kernel/module/main.c:2624).
2619 if (i == info->index.mod &&
2620 (WARN_ON_ONCE(shdr->sh_size != sizeof(struct module)))) {
2621 ret = -ENOEXEC;
2622 goto out_err;
2623 }
2624 memcpy(dest, (void *)shdr->sh_addr, shdr->sh_size);
2625 }
2626 /*
2627 * Update the userspace copy's ELF section address to point to
2628 * our newly allocated memory as a pure convenience so that
(gdb)
(gdb) l *(__asan_memcpy+0x38)0xffffffff81b39268 is in __asan_memcpy (/home/greearb/git/linux-2.6/mm/kasan/shadow.c:105).
100 EXPORT_SYMBOL(__asan_memmove);
101 #endif
102
103 void *__asan_memcpy(void *dest, const void *src, ssize_t len)
104 {
105 if (!kasan_check_range(src, len, false, _RET_IP_) ||
106 !kasan_check_range(dest, len, true, _RET_IP_))
107 return NULL;
108
109 return __memcpy(dest, src, len);
(gdb)
(gdb) l *(kasan_check_range+0xa5)0xffffffff81b386f5 is in kasan_check_range (/home/greearb/git/linux-2.6/mm/kasan/generic.c:116).
111 start += prefix;
112 }
113
114 words = (end - start) / 8;
115 while (words) {
116 if (unlikely(*(u64 *)start))
117 return bytes_is_nonzero(start, 8);
118 start += 8;
119 words--;
120 }
(gdb)
BUG: unable to handle page fault for address: fffffbfff4041000 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 44d0e7067 P4D 44d0e7067 PUD 44d0e3067 PMD 10bb3d067 PTE 0 Oops: Oops: 0000 [#1] PREEMPT SMP KASAN CPU: 7 UID: 0 PID: 319 Comm: systemd-udevd Not tainted 6.13.0-rc1+ #24 Hardware name: Default string Default string/SKYBAY, BIOS 5.12 02/15/2023 RIP: 0010:kasan_check_range+0xa5/0x190 Code: 8d 5a 07 4c 0f 49 da 49 c1 fb 03 45 85 db 0f 84 ce 00 00 00 45 89 db 4a 8d 14 d8 eb 0d 48 83 c0 08 48 39 d0 0f 84 b29 RSP: 0018:ffff88812d3ef980 EFLAGS: 00010206 RAX: fffffbfff4041000 RBX: fffffbfff404101e RCX: ffffffff814f30d6 RDX: fffffbfff4041018 RSI: 00000000000000f0 RDI: ffffffffa0208000 [ OK BP: fffffbfff4041000 R08: 0000000000000001 R09: fffffbfff404101d 0m] Finished 10: ffffffffa02080ef R11: 0000000000000003 R12: ffffffffa0208000 ;1;39msystemd-udR13: ffffc90000dac930 R14: ffffc90000dac950 R15: dffffc0000000000 ev-trig…e FS: 00007fc25423bb40(0000) GS:ffff88841dd80000(0000) knlGS:0000000000000000 - Coldplug All uCS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: fffffbfff4041000 CR3: 0000000128ef4003 CR4: 00000000003706f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: <TASK> ? __die+0x1f/0x60 ? page_fault_oops+0x258/0x910 ? dump_pagetable+0x690/0x690 ? search_bpf_extables+0x22/0x250 ? show_ldttss+0x230/0x230 ? search_bpf_extables+0x164/0x250 ? kasan_check_range+0xa5/0x190 ? fixup_exception+0x4d/0xc70 ? exc_page_fault+0xe1/0xf0 ? asm_exc_page_fault+0x22/0x30 ? load_module+0x3c46/0x8680 ? kasan_check_range+0xa5/0x190 __asan_memcpy+0x38/0x60 load_module+0x3c46/0x8680 ? __kernel_read+0x270/0x9f0 ? module_frob_arch_sections+0x30/0x30 ? lockdep_lock+0xbe/0x1b0 ? rw_verify_area+0x18d/0x5e0 ? kernel_read_file+0x246/0x870 ? __x64_sys_fspick+0x290/0x290 ? init_module_from_file+0xd1/0x130 init_module_from_file+0xd1/0x130 ? __ia32_sys_init_module+0xa0/0xa0 ? lock_acquire+0x2d/0xb0 ? idempotent_init_module+0x10d/0x780 ? do_raw_spin_unlock+0x54/0x220 idempotent_init_module+0x21d/0x780 ? init_module_from_file+0x130/0x130 ? __fget_files+0x1a5/0x2d0 __x64_sys_finit_module+0xbe/0x130 do_syscall_64+0x69/0x160 entry_SYSCALL_64_after_hwframe+0x4b/0x53 RIP: 0033:0x7fc254e0827d Code: 5d c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 248 RSP: 002b:00007ffe1750a718 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 RAX: ffffffffffffffda RBX: 000055da274c5500 RCX: 00007fc254e0827d RDX: 0000000000000000 RSI: 00007fc254f6e43c RDI: 0000000000000006 RBP: 00007fc254f6e43c R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000006 R11: 0000000000000246 R12: 0000000000020000 R13: 000055da2749cbb0 R14: 0000000000000000 R15: 000055da274c44a0 </TASK> Modules linked in: CR2: fffffbfff4041000 ---[ end trace 0000000000000000 ]---
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 104 lines · Ben Greear
Quoted 102 lines · Suren Baghdasaryan
Quoted 88 lines · Ben Greear
Hello Suren,
My system crashes on bootup, and I bisected to this commit.
0f9b685626daa2f8e19a9788625c9b624c223e45is the first bad commit commit0f9b685626daa2f8e19a9788625c9b624c223e45Author: Suren Baghdasaryan <[email protected]> Date: Wed Oct 23 10:07:57 2024 -0700
alloc_tag: populate memory for module tags as neededThe memory reserved for module tags does not need to be backed by physical pages until there are tags to store there. Change the way we reserve this memory to allocate only virtual area for the tags and populate it with physical pages as needed when we load a module.
The crash looks like this:
BUG: unable to handle page fault for address: fffffbfff4041000 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 44d0e7067 P4D 44d0e7067 PUD 44d0e3067 PMD 10bb38067 PTE 0 Oops: Oops: 0000 [#1] PREEMPT SMP KASAN CPU: 0 UID: 0 PID: 319 Comm: systemd-udevd Not tainted 6.12.0-rc6+ #21 Hardware name: Default string Default string/SKYBAY, BIOS 5.12 02/15/2023 RIP: 0010:kasan_check_range+0xa5/0x190 Code: 8d 5a 07 4c 0f 49 da 49 c1 fb 03 45 85 db 0f 84 ce 00 00 00 45 89 db 4a 8d 14 d8 eb 0d 48 83 c0 08 48 39 d0 0f 84 b29 RSP: 0018:ffff88812c26f980 EFLAGS: 00010206 RAX: fffffbfff4041000 RBX: fffffbfff404101e RCX: ffffffff814ec29b [ OK DX: fffffbfff4041018 RSI: 00000000000000f0 RDI: ffffffffa0208000 0m] Finished BP: fffffbfff4041000 R08: 0000000000000001 R09: fffffbfff404101d ;1;39msystemd-udR10: ffffffffa02080ef R11: 0000000000000003 R12: ffffffffa0208000 ev-trig…e R13: ffffc90000dac7c8 R14: ffffc90000dac7e8 R15: dffffc0000000000 - Coldplug All uFS: 00007fe869216b40(0000) GS:ffff88841da00000(0000) knlGS:0000000000000000 dev Devices. CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: fffffbfff4041000 CR3: 0000000121e86002 CR4: 00000000003706f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: <TASK> [ OK ? __die+0x1f/0x60 0m] Reached targ ? page_fault_oops+0x258/0x910 et sysi ? dump_pagetable+0x690/0x690 nit.target - ? search_bpf_extables+0x22/0x250 System Initiali ? trace_page_fault_kernel+0x120/0x120 zation. ? search_bpf_extables+0x164/0x250 ? kasan_check_range+0xa5/0x190 ? fixup_exception+0x4d/0xc70 ? exc_page_fault+0xe1/0xf0 [ OK ? asm_exc_page_fault+0x22/0x30 0m] Reached targ ? load_module+0x3d7b/0x7560 et netw ? kasan_check_range+0xa5/0x190 ork.target - __asan_memcpy+0x38/0x60 Network. load_module+0x3d7b/0x7560 ? module_frob_arch_sections+0x30/0x30 ? lockdep_lock+0xbe/0x1b0 ? rw_verify_area+0x18d/0x5e0 ? kernel_read_file+0x246/0x870 ? __x64_sys_fspick+0x290/0x290 ? init_module_from_file+0xd1/0x130 init_module_from_file+0xd1/0x130 ? __ia32_sys_init_module+0xa0/0xa0 ? lock_acquire+0x2d/0xb0 ? idempotent_init_module+0x116/0x790 ? do_raw_spin_unlock+0x54/0x220 idempotent_init_module+0x226/0x790 ? init_module_from_file+0x130/0x130 ? vm_mmap_pgoff+0x203/0x2e0 __x64_sys_finit_module+0xba/0x130 do_syscall_64+0x69/0x160 entry_SYSCALL_64_after_hwframe+0x4b/0x53 RIP: 0033:0x7fe869de327d Code: 5d c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 248 RSP: 002b:00007ffe34a828d8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 RAX: ffffffffffffffda RBX: 0000557fa8f3f3f0 RCX: 00007fe869de327d RDX: 0000000000000000 RSI: 00007fe869f4943c RDI: 0000000000000006 RBP: 00007fe869f4943c R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000006 R11: 0000000000000246 R12: 0000000000020000 R13: 0000557fa8f3f030 R14: 0000000000000000 R15: 0000557fa8f3d110 </TASK> Modules linked in: CR2: fffffbfff4041000 ---[ end trace 0000000000000000 ]---Quoted 6 lines
I suspect you only hit this with an unlucky amount of debugging enabled. The kernel config I used is found here:
http://www.candelatech.com/downloads/cfg-kasan-crash-regression.config
I will be happy to test fixes.
Hi Ben,
Thanks for reporting the issue. Do you have these recent fixes in your tree:https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/all/[email protected]/
Quoted 8 lines
Quoted 3 lines
If not, couple you please apply them and see if the issue is still happening? Thanks,
Suren.Hello Suren,
Thanks for the quick response. The first patch is already in latest Linus tree, and I applied the second one manually. The kernel still crashes:
Ok, I'll try to reproduce and troubleshoot it. Thanks!
Quoted 119 lines
(gdb) l *(load_module+0x3c46)
0xffffffff814f30d6is inload_module(/home/greearb/git/linux-2.6/kernel/module/main.c:2624).
2619 if (i ==info->index.mod&&
2620 (WARN_ON_ONCE(shdr->sh_size != sizeof(struct module)))) {
2621 ret =-ENOEXEC;
2622 gotoout_err;
2623 }
2624memcpy(dest, (void *)shdr->sh_addr, shdr->sh_size);
2625 }
2626 /*
2627 * Update the userspace copy's ELF section address to point to
2628 * our newly allocated memory as a pure convenience so that
(gdb)(gdb) l *(__asan_memcpy+0x38)
0xffffffff81b39268is in__asan_memcpy(/home/greearb/git/linux-2.6/mm/kasan/shadow.c:105).
100EXPORT_SYMBOL(__asan_memmove);
101 #endif
102
103 void *__asan_memcpy(void *dest, const void *src,ssize_tlen)
104 {
105 if (!kasan_check_range(src, len, false, _RET_IP_) ||
106 !kasan_check_range(dest, len, true, _RET_IP_))
107 return NULL;
108
109 return__memcpy(dest, src, len);
(gdb)(gdb) l *(kasan_check_range+0xa5)
0xffffffff81b386f5is inkasan_check_range(/home/greearb/git/linux-2.6/mm/kasan/generic.c:116).
111 start += prefix;
112 }
113
114 words = (end - start) / 8;
115 while (words) {
116 if (unlikely(*(u64 *)start))
117 returnbytes_is_nonzero(start, 8);
118 start += 8;
119 words--;
120 }
(gdb)BUG: unable to handle page fault for address: fffffbfff4041000 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 44d0e7067 P4D 44d0e7067 PUD 44d0e3067 PMD 10bb3d067 PTE 0 Oops: Oops: 0000 [#1] PREEMPT SMP KASAN CPU: 7 UID: 0 PID: 319 Comm: systemd-udevd Not tainted 6.13.0-rc1+ #24 Hardware name: Default string Default string/SKYBAY, BIOS 5.12 02/15/2023 RIP: 0010:kasan_check_range+0xa5/0x190 Code: 8d 5a 07 4c 0f 49 da 49 c1 fb 03 45 85 db 0f 84 ce 00 00 00 45 89 db 4a 8d 14 d8 eb 0d 48 83 c0 08 48 39 d0 0f 84 b29 RSP: 0018:ffff88812d3ef980 EFLAGS: 00010206 RAX: fffffbfff4041000 RBX: fffffbfff404101e RCX: ffffffff814f30d6 RDX: fffffbfff4041018 RSI: 00000000000000f0 RDI: ffffffffa0208000 [ OK BP: fffffbfff4041000 R08: 0000000000000001 R09: fffffbfff404101d 0m] Finished 10: ffffffffa02080ef R11: 0000000000000003 R12: ffffffffa0208000 ;1;39msystemd-udR13: ffffc90000dac930 R14: ffffc90000dac950 R15: dffffc0000000000 ev-trig…e FS: 00007fc25423bb40(0000) GS:ffff88841dd80000(0000) knlGS:0000000000000000 - Coldplug All uCS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: fffffbfff4041000 CR3: 0000000128ef4003 CR4: 00000000003706f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: <TASK> ? __die+0x1f/0x60 ? page_fault_oops+0x258/0x910 ? dump_pagetable+0x690/0x690 ? search_bpf_extables+0x22/0x250 ? show_ldttss+0x230/0x230 ? search_bpf_extables+0x164/0x250 ? kasan_check_range+0xa5/0x190 ? fixup_exception+0x4d/0xc70 ? exc_page_fault+0xe1/0xf0 ? asm_exc_page_fault+0x22/0x30 ? load_module+0x3c46/0x8680 ? kasan_check_range+0xa5/0x190 __asan_memcpy+0x38/0x60 load_module+0x3c46/0x8680 ? __kernel_read+0x270/0x9f0 ? module_frob_arch_sections+0x30/0x30 ? lockdep_lock+0xbe/0x1b0 ? rw_verify_area+0x18d/0x5e0 ? kernel_read_file+0x246/0x870 ? __x64_sys_fspick+0x290/0x290 ? init_module_from_file+0xd1/0x130 init_module_from_file+0xd1/0x130 ? __ia32_sys_init_module+0xa0/0xa0 ? lock_acquire+0x2d/0xb0 ? idempotent_init_module+0x10d/0x780 ? do_raw_spin_unlock+0x54/0x220 idempotent_init_module+0x21d/0x780 ? init_module_from_file+0x130/0x130 ? __fget_files+0x1a5/0x2d0 __x64_sys_finit_module+0xbe/0x130 do_syscall_64+0x69/0x160 entry_SYSCALL_64_after_hwframe+0x4b/0x53 RIP: 0033:0x7fc254e0827d Code: 5d c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 248 RSP: 002b:00007ffe1750a718 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 RAX: ffffffffffffffda RBX: 000055da274c5500 RCX: 00007fc254e0827d RDX: 0000000000000000 RSI: 00007fc254f6e43c RDI: 0000000000000006 RBP: 00007fc254f6e43c R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000006 R11: 0000000000000246 R12: 0000000000020000 R13: 000055da2749cbb0 R14: 0000000000000000 R15: 000055da274c44a0 </TASK> Modules linked in: CR2: fffffbfff4041000 ---[ end trace 0000000000000000 ]---Thanks,
Ben--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
Quoted 106 lines · Suren Baghdasaryan
Quoted 104 lines · Ben Greear
Quoted 102 lines · Suren Baghdasaryan
Quoted 88 lines · Ben Greear
Hello Suren,
My system crashes on bootup, and I bisected to this commit.
0f9b685626daa2f8e19a9788625c9b624c223e45is the first bad commit commit0f9b685626daa2f8e19a9788625c9b624c223e45Author: Suren Baghdasaryan <[email protected]> Date: Wed Oct 23 10:07:57 2024 -0700
alloc_tag: populate memory for module tags as neededThe memory reserved for module tags does not need to be backed by physical pages until there are tags to store there. Change the way we reserve this memory to allocate only virtual area for the tags and populate it with physical pages as needed when we load a module.
The crash looks like this:
BUG: unable to handle page fault for address: fffffbfff4041000 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 44d0e7067 P4D 44d0e7067 PUD 44d0e3067 PMD 10bb38067 PTE 0 Oops: Oops: 0000 [#1] PREEMPT SMP KASAN CPU: 0 UID: 0 PID: 319 Comm: systemd-udevd Not tainted 6.12.0-rc6+ #21 Hardware name: Default string Default string/SKYBAY, BIOS 5.12 02/15/2023 RIP: 0010:kasan_check_range+0xa5/0x190 Code: 8d 5a 07 4c 0f 49 da 49 c1 fb 03 45 85 db 0f 84 ce 00 00 00 45 89 db 4a 8d 14 d8 eb 0d 48 83 c0 08 48 39 d0 0f 84 b29 RSP: 0018:ffff88812c26f980 EFLAGS: 00010206 RAX: fffffbfff4041000 RBX: fffffbfff404101e RCX: ffffffff814ec29b [ OK DX: fffffbfff4041018 RSI: 00000000000000f0 RDI: ffffffffa0208000 0m] Finished BP: fffffbfff4041000 R08: 0000000000000001 R09: fffffbfff404101d ;1;39msystemd-udR10: ffffffffa02080ef R11: 0000000000000003 R12: ffffffffa0208000 ev-trig…e R13: ffffc90000dac7c8 R14: ffffc90000dac7e8 R15: dffffc0000000000 - Coldplug All uFS: 00007fe869216b40(0000) GS:ffff88841da00000(0000) knlGS:0000000000000000 dev Devices. CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: fffffbfff4041000 CR3: 0000000121e86002 CR4: 00000000003706f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: <TASK> [ OK ? __die+0x1f/0x60 0m] Reached targ ? page_fault_oops+0x258/0x910 et sysi ? dump_pagetable+0x690/0x690 nit.target - ? search_bpf_extables+0x22/0x250 System Initiali ? trace_page_fault_kernel+0x120/0x120 zation. ? search_bpf_extables+0x164/0x250 ? kasan_check_range+0xa5/0x190 ? fixup_exception+0x4d/0xc70 ? exc_page_fault+0xe1/0xf0 [ OK ? asm_exc_page_fault+0x22/0x30 0m] Reached targ ? load_module+0x3d7b/0x7560 et netw ? kasan_check_range+0xa5/0x190 ork.target - __asan_memcpy+0x38/0x60 Network. load_module+0x3d7b/0x7560 ? module_frob_arch_sections+0x30/0x30 ? lockdep_lock+0xbe/0x1b0 ? rw_verify_area+0x18d/0x5e0 ? kernel_read_file+0x246/0x870 ? __x64_sys_fspick+0x290/0x290 ? init_module_from_file+0xd1/0x130 init_module_from_file+0xd1/0x130 ? __ia32_sys_init_module+0xa0/0xa0 ? lock_acquire+0x2d/0xb0 ? idempotent_init_module+0x116/0x790 ? do_raw_spin_unlock+0x54/0x220 idempotent_init_module+0x226/0x790 ? init_module_from_file+0x130/0x130 ? vm_mmap_pgoff+0x203/0x2e0 __x64_sys_finit_module+0xba/0x130 do_syscall_64+0x69/0x160 entry_SYSCALL_64_after_hwframe+0x4b/0x53 RIP: 0033:0x7fe869de327d Code: 5d c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 248 RSP: 002b:00007ffe34a828d8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 RAX: ffffffffffffffda RBX: 0000557fa8f3f3f0 RCX: 00007fe869de327d RDX: 0000000000000000 RSI: 00007fe869f4943c RDI: 0000000000000006 RBP: 00007fe869f4943c R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000006 R11: 0000000000000246 R12: 0000000000020000 R13: 0000557fa8f3f030 R14: 0000000000000000 R15: 0000557fa8f3d110 </TASK> Modules linked in: CR2: fffffbfff4041000 ---[ end trace 0000000000000000 ]---Quoted 6 lines
I suspect you only hit this with an unlucky amount of debugging enabled. The kernel config I used is found here:
http://www.candelatech.com/downloads/cfg-kasan-crash-regression.config
I will be happy to test fixes.
Hi Ben,
Thanks for reporting the issue. Do you have these recent fixes in your tree:https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/all/[email protected]/
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
If not, couple you please apply them and see if the issue is still happening? Thanks,
Suren.Hello Suren,
Thanks for the quick response. The first patch is already in latest Linus tree,
Hmm. Could you please double-check which tree you are using? I don't see the first patch
(https://lore.kernel.org/all/[email protected]/)
in Linus' tree. Maybe you are using linux-next?
Quoted 123 lines
Quoted 1 line
and I applied the second one manually. The kernel still crashes:
Ok, I'll try to reproduce and troubleshoot it. Thanks!
Quoted 119 lines
(gdb) l *(load_module+0x3c46)
0xffffffff814f30d6is inload_module(/home/greearb/git/linux-2.6/kernel/module/main.c:2624).
2619 if (i ==info->index.mod&&
2620 (WARN_ON_ONCE(shdr->sh_size != sizeof(struct module)))) {
2621 ret =-ENOEXEC;
2622 gotoout_err;
2623 }
2624memcpy(dest, (void *)shdr->sh_addr, shdr->sh_size);
2625 }
2626 /*
2627 * Update the userspace copy's ELF section address to point to
2628 * our newly allocated memory as a pure convenience so that
(gdb)(gdb) l *(__asan_memcpy+0x38)
0xffffffff81b39268is in__asan_memcpy(/home/greearb/git/linux-2.6/mm/kasan/shadow.c:105).
100EXPORT_SYMBOL(__asan_memmove);
101 #endif
102
103 void *__asan_memcpy(void *dest, const void *src,ssize_tlen)
104 {
105 if (!kasan_check_range(src, len, false, _RET_IP_) ||
106 !kasan_check_range(dest, len, true, _RET_IP_))
107 return NULL;
108
109 return__memcpy(dest, src, len);
(gdb)(gdb) l *(kasan_check_range+0xa5)
0xffffffff81b386f5is inkasan_check_range(/home/greearb/git/linux-2.6/mm/kasan/generic.c:116).
111 start += prefix;
112 }
113
114 words = (end - start) / 8;
115 while (words) {
116 if (unlikely(*(u64 *)start))
117 returnbytes_is_nonzero(start, 8);
118 start += 8;
119 words--;
120 }
(gdb)BUG: unable to handle page fault for address: fffffbfff4041000 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 44d0e7067 P4D 44d0e7067 PUD 44d0e3067 PMD 10bb3d067 PTE 0 Oops: Oops: 0000 [#1] PREEMPT SMP KASAN CPU: 7 UID: 0 PID: 319 Comm: systemd-udevd Not tainted 6.13.0-rc1+ #24 Hardware name: Default string Default string/SKYBAY, BIOS 5.12 02/15/2023 RIP: 0010:kasan_check_range+0xa5/0x190 Code: 8d 5a 07 4c 0f 49 da 49 c1 fb 03 45 85 db 0f 84 ce 00 00 00 45 89 db 4a 8d 14 d8 eb 0d 48 83 c0 08 48 39 d0 0f 84 b29 RSP: 0018:ffff88812d3ef980 EFLAGS: 00010206 RAX: fffffbfff4041000 RBX: fffffbfff404101e RCX: ffffffff814f30d6 RDX: fffffbfff4041018 RSI: 00000000000000f0 RDI: ffffffffa0208000 [ OK BP: fffffbfff4041000 R08: 0000000000000001 R09: fffffbfff404101d 0m] Finished 10: ffffffffa02080ef R11: 0000000000000003 R12: ffffffffa0208000 ;1;39msystemd-udR13: ffffc90000dac930 R14: ffffc90000dac950 R15: dffffc0000000000 ev-trig…e FS: 00007fc25423bb40(0000) GS:ffff88841dd80000(0000) knlGS:0000000000000000 - Coldplug All uCS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: fffffbfff4041000 CR3: 0000000128ef4003 CR4: 00000000003706f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: <TASK> ? __die+0x1f/0x60 ? page_fault_oops+0x258/0x910 ? dump_pagetable+0x690/0x690 ? search_bpf_extables+0x22/0x250 ? show_ldttss+0x230/0x230 ? search_bpf_extables+0x164/0x250 ? kasan_check_range+0xa5/0x190 ? fixup_exception+0x4d/0xc70 ? exc_page_fault+0xe1/0xf0 ? asm_exc_page_fault+0x22/0x30 ? load_module+0x3c46/0x8680 ? kasan_check_range+0xa5/0x190 __asan_memcpy+0x38/0x60 load_module+0x3c46/0x8680 ? __kernel_read+0x270/0x9f0 ? module_frob_arch_sections+0x30/0x30 ? lockdep_lock+0xbe/0x1b0 ? rw_verify_area+0x18d/0x5e0 ? kernel_read_file+0x246/0x870 ? __x64_sys_fspick+0x290/0x290 ? init_module_from_file+0xd1/0x130 init_module_from_file+0xd1/0x130 ? __ia32_sys_init_module+0xa0/0xa0 ? lock_acquire+0x2d/0xb0 ? idempotent_init_module+0x10d/0x780 ? do_raw_spin_unlock+0x54/0x220 idempotent_init_module+0x21d/0x780 ? init_module_from_file+0x130/0x130 ? __fget_files+0x1a5/0x2d0 __x64_sys_finit_module+0xbe/0x130 do_syscall_64+0x69/0x160 entry_SYSCALL_64_after_hwframe+0x4b/0x53 RIP: 0033:0x7fc254e0827d Code: 5d c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 248 RSP: 002b:00007ffe1750a718 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 RAX: ffffffffffffffda RBX: 000055da274c5500 RCX: 00007fc254e0827d RDX: 0000000000000000 RSI: 00007fc254f6e43c RDI: 0000000000000006 RBP: 00007fc254f6e43c R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000006 R11: 0000000000000246 R12: 0000000000020000 R13: 000055da2749cbb0 R14: 0000000000000000 R15: 000055da274c44a0 </TASK> Modules linked in: CR2: fffffbfff4041000 ---[ end trace 0000000000000000 ]---Thanks,
Ben--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
Quoted 111 lines · Suren Baghdasaryan
Quoted 110 lines · Suren Baghdasaryan
Quoted 108 lines · Ben Greear
Quoted 105 lines · Suren Baghdasaryan
Quoted 88 lines · Ben Greear
Hello Suren,
My system crashes on bootup, and I bisected to this commit.
0f9b685626daa2f8e19a9788625c9b624c223e45is the first bad commit commit0f9b685626daa2f8e19a9788625c9b624c223e45Author: Suren Baghdasaryan <[email protected]> Date: Wed Oct 23 10:07:57 2024 -0700
alloc_tag: populate memory for module tags as neededThe memory reserved for module tags does not need to be backed by physical pages until there are tags to store there. Change the way we reserve this memory to allocate only virtual area for the tags and populate it with physical pages as needed when we load a module.
The crash looks like this:
BUG: unable to handle page fault for address: fffffbfff4041000 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 44d0e7067 P4D 44d0e7067 PUD 44d0e3067 PMD 10bb38067 PTE 0 Oops: Oops: 0000 [#1] PREEMPT SMP KASAN CPU: 0 UID: 0 PID: 319 Comm: systemd-udevd Not tainted 6.12.0-rc6+ #21 Hardware name: Default string Default string/SKYBAY, BIOS 5.12 02/15/2023 RIP: 0010:kasan_check_range+0xa5/0x190 Code: 8d 5a 07 4c 0f 49 da 49 c1 fb 03 45 85 db 0f 84 ce 00 00 00 45 89 db 4a 8d 14 d8 eb 0d 48 83 c0 08 48 39 d0 0f 84 b29 RSP: 0018:ffff88812c26f980 EFLAGS: 00010206 RAX: fffffbfff4041000 RBX: fffffbfff404101e RCX: ffffffff814ec29b [ OK DX: fffffbfff4041018 RSI: 00000000000000f0 RDI: ffffffffa0208000 0m] Finished BP: fffffbfff4041000 R08: 0000000000000001 R09: fffffbfff404101d ;1;39msystemd-udR10: ffffffffa02080ef R11: 0000000000000003 R12: ffffffffa0208000 ev-trig…e R13: ffffc90000dac7c8 R14: ffffc90000dac7e8 R15: dffffc0000000000 - Coldplug All uFS: 00007fe869216b40(0000) GS:ffff88841da00000(0000) knlGS:0000000000000000 dev Devices. CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: fffffbfff4041000 CR3: 0000000121e86002 CR4: 00000000003706f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: <TASK> [ OK ? __die+0x1f/0x60 0m] Reached targ ? page_fault_oops+0x258/0x910 et sysi ? dump_pagetable+0x690/0x690 nit.target - ? search_bpf_extables+0x22/0x250 System Initiali ? trace_page_fault_kernel+0x120/0x120 zation. ? search_bpf_extables+0x164/0x250 ? kasan_check_range+0xa5/0x190 ? fixup_exception+0x4d/0xc70 ? exc_page_fault+0xe1/0xf0 [ OK ? asm_exc_page_fault+0x22/0x30 0m] Reached targ ? load_module+0x3d7b/0x7560 et netw ? kasan_check_range+0xa5/0x190 ork.target - __asan_memcpy+0x38/0x60 Network. load_module+0x3d7b/0x7560 ? module_frob_arch_sections+0x30/0x30 ? lockdep_lock+0xbe/0x1b0 ? rw_verify_area+0x18d/0x5e0 ? kernel_read_file+0x246/0x870 ? __x64_sys_fspick+0x290/0x290 ? init_module_from_file+0xd1/0x130 init_module_from_file+0xd1/0x130 ? __ia32_sys_init_module+0xa0/0xa0 ? lock_acquire+0x2d/0xb0 ? idempotent_init_module+0x116/0x790 ? do_raw_spin_unlock+0x54/0x220 idempotent_init_module+0x226/0x790 ? init_module_from_file+0x130/0x130 ? vm_mmap_pgoff+0x203/0x2e0 __x64_sys_finit_module+0xba/0x130 do_syscall_64+0x69/0x160 entry_SYSCALL_64_after_hwframe+0x4b/0x53 RIP: 0033:0x7fe869de327d Code: 5d c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 248 RSP: 002b:00007ffe34a828d8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 RAX: ffffffffffffffda RBX: 0000557fa8f3f3f0 RCX: 00007fe869de327d RDX: 0000000000000000 RSI: 00007fe869f4943c RDI: 0000000000000006 RBP: 00007fe869f4943c R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000006 R11: 0000000000000246 R12: 0000000000020000 R13: 0000557fa8f3f030 R14: 0000000000000000 R15: 0000557fa8f3d110 </TASK> Modules linked in: CR2: fffffbfff4041000 ---[ end trace 0000000000000000 ]---Quoted 6 lines
I suspect you only hit this with an unlucky amount of debugging enabled. The kernel config I used is found here:
http://www.candelatech.com/downloads/cfg-kasan-crash-regression.config
I will be happy to test fixes.
Hi Ben,
Thanks for reporting the issue. Do you have these recent fixes in your tree:https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/all/[email protected]/If not, couple you please apply them and see if the issue is still happening? Thanks,
Suren.
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Hello Suren,
Thanks for the quick response. The first patch is already in latest Linus tree,
Hmm. Could you please double-check which tree you are using? I don't see the first patch
(https://lore.kernel.org/all/[email protected]/)
in Linus' tree. Maybe you are using linux-next?
Sorry, you are correct. I must have mangled something when trying to apply the patch and I didn't look hard enough when patch said changes were already applied.
I can re-test this next week...and for reference, kernel boots fine when you disable KASAN and other debugging.
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 122 lines · Ben Greear
Quoted 111 lines · Suren Baghdasaryan
Quoted 110 lines · Suren Baghdasaryan
Quoted 108 lines · Ben Greear
Quoted 105 lines · Suren Baghdasaryan
Quoted 88 lines · Ben Greear
Hello Suren,
My system crashes on bootup, and I bisected to this commit.
0f9b685626daa2f8e19a9788625c9b624c223e45is the first bad commit commit0f9b685626daa2f8e19a9788625c9b624c223e45Author: Suren Baghdasaryan <[email protected]> Date: Wed Oct 23 10:07:57 2024 -0700
alloc_tag: populate memory for module tags as neededThe memory reserved for module tags does not need to be backed by physical pages until there are tags to store there. Change the way we reserve this memory to allocate only virtual area for the tags and populate it with physical pages as needed when we load a module.
The crash looks like this:
BUG: unable to handle page fault for address: fffffbfff4041000 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 44d0e7067 P4D 44d0e7067 PUD 44d0e3067 PMD 10bb38067 PTE 0 Oops: Oops: 0000 [#1] PREEMPT SMP KASAN CPU: 0 UID: 0 PID: 319 Comm: systemd-udevd Not tainted 6.12.0-rc6+ #21 Hardware name: Default string Default string/SKYBAY, BIOS 5.12 02/15/2023 RIP: 0010:kasan_check_range+0xa5/0x190 Code: 8d 5a 07 4c 0f 49 da 49 c1 fb 03 45 85 db 0f 84 ce 00 00 00 45 89 db 4a 8d 14 d8 eb 0d 48 83 c0 08 48 39 d0 0f 84 b29 RSP: 0018:ffff88812c26f980 EFLAGS: 00010206 RAX: fffffbfff4041000 RBX: fffffbfff404101e RCX: ffffffff814ec29b [ OK DX: fffffbfff4041018 RSI: 00000000000000f0 RDI: ffffffffa0208000 0m] Finished BP: fffffbfff4041000 R08: 0000000000000001 R09: fffffbfff404101d ;1;39msystemd-udR10: ffffffffa02080ef R11: 0000000000000003 R12: ffffffffa0208000 ev-trig…e R13: ffffc90000dac7c8 R14: ffffc90000dac7e8 R15: dffffc0000000000 - Coldplug All uFS: 00007fe869216b40(0000) GS:ffff88841da00000(0000) knlGS:0000000000000000 dev Devices. CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: fffffbfff4041000 CR3: 0000000121e86002 CR4: 00000000003706f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: <TASK> [ OK ? __die+0x1f/0x60 0m] Reached targ ? page_fault_oops+0x258/0x910 et sysi ? dump_pagetable+0x690/0x690 nit.target - ? search_bpf_extables+0x22/0x250 System Initiali ? trace_page_fault_kernel+0x120/0x120 zation. ? search_bpf_extables+0x164/0x250 ? kasan_check_range+0xa5/0x190 ? fixup_exception+0x4d/0xc70 ? exc_page_fault+0xe1/0xf0 [ OK ? asm_exc_page_fault+0x22/0x30 0m] Reached targ ? load_module+0x3d7b/0x7560 et netw ? kasan_check_range+0xa5/0x190 ork.target - __asan_memcpy+0x38/0x60 Network. load_module+0x3d7b/0x7560 ? module_frob_arch_sections+0x30/0x30 ? lockdep_lock+0xbe/0x1b0 ? rw_verify_area+0x18d/0x5e0 ? kernel_read_file+0x246/0x870 ? __x64_sys_fspick+0x290/0x290 ? init_module_from_file+0xd1/0x130 init_module_from_file+0xd1/0x130 ? __ia32_sys_init_module+0xa0/0xa0 ? lock_acquire+0x2d/0xb0 ? idempotent_init_module+0x116/0x790 ? do_raw_spin_unlock+0x54/0x220 idempotent_init_module+0x226/0x790 ? init_module_from_file+0x130/0x130 ? vm_mmap_pgoff+0x203/0x2e0 __x64_sys_finit_module+0xba/0x130 do_syscall_64+0x69/0x160 entry_SYSCALL_64_after_hwframe+0x4b/0x53 RIP: 0033:0x7fe869de327d Code: 5d c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 248 RSP: 002b:00007ffe34a828d8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 RAX: ffffffffffffffda RBX: 0000557fa8f3f3f0 RCX: 00007fe869de327d RDX: 0000000000000000 RSI: 00007fe869f4943c RDI: 0000000000000006 RBP: 00007fe869f4943c R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000006 R11: 0000000000000246 R12: 0000000000020000 R13: 0000557fa8f3f030 R14: 0000000000000000 R15: 0000557fa8f3d110 </TASK> Modules linked in: CR2: fffffbfff4041000 ---[ end trace 0000000000000000 ]---Quoted 6 lines
I suspect you only hit this with an unlucky amount of debugging enabled. The kernel config I used is found here:
http://www.candelatech.com/downloads/cfg-kasan-crash-regression.config
I will be happy to test fixes.
Hi Ben,
Thanks for reporting the issue. Do you have these recent fixes in your tree:https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/all/[email protected]/If not, couple you please apply them and see if the issue is still happening? Thanks,
Suren.Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Hello Suren,
Thanks for the quick response. The first patch is already in latest Linus tree,
Hmm. Could you please double-check which tree you are using? I don't see the first patch
(https://lore.kernel.org/all/[email protected]/)
in Linus' tree. Maybe you are using linux-next?
Quoted 5 lines
Sorry, you are correct. I must have mangled something when trying to apply the patch and I didn't look hard enough when patch said changes were already applied.
I can re-test this next week...and for reference, kernel boots fine when you disable KASAN and other debugging.
Thanks! Please retest with this patch and let me know if you are still having issues.
Suren.
Quoted 10 lines
Thanks,
Ben--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
Hi Suren
Quoted 115 lines · Suren Baghdasaryan
Quoted 114 lines · Ben Greear
Quoted 113 lines · Suren Baghdasaryan
Quoted 102 lines · Suren Baghdasaryan
Quoted 101 lines · Ben Greear
Quoted 100 lines · Suren Baghdasaryan
Quoted 87 lines · Ben Greear
Hello Suren,
My system crashes on bootup, and I bisected to this commit.
0f9b685626daa2f8e19a9788625c9b624c223e45is the first bad commit commit0f9b685626daa2f8e19a9788625c9b624c223e45Author: Suren Baghdasaryan <[email protected]> Date: Wed Oct 23 10:07:57 2024 -0700
alloc_tag: populate memory for module tags as neededThe memory reserved for module tags does not need to be backed by physical pages until there are tags to store there. Change the way we reserve this memory to allocate only virtual area for the tags and populate it with physical pages as needed when we load a module.
The crash looks like this:
BUG: unable to handle page fault for address: fffffbfff4041000 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 44d0e7067 P4D 44d0e7067 PUD 44d0e3067 PMD 10bb38067 PTE 0 Oops: Oops: 0000 [#1] PREEMPT SMP KASAN CPU: 0 UID: 0 PID: 319 Comm: systemd-udevd Not tainted 6.12.0-rc6+ #21 Hardware name: Default string Default string/SKYBAY, BIOS 5.12 02/15/2023 RIP: 0010:kasan_check_range+0xa5/0x190 Code: 8d 5a 07 4c 0f 49 da 49 c1 fb 03 45 85 db 0f 84 ce 00 00 00 45 89 db 4a 8d 14 d8 eb 0d 48 83 c0 08 48 39 d0 0f 84 b29 RSP: 0018:ffff88812c26f980 EFLAGS: 00010206 RAX: fffffbfff4041000 RBX: fffffbfff404101e RCX: ffffffff814ec29b [ OK DX: fffffbfff4041018 RSI: 00000000000000f0 RDI: ffffffffa0208000 0m] Finished BP: fffffbfff4041000 R08: 0000000000000001 R09: fffffbfff404101d ;1;39msystemd-udR10: ffffffffa02080ef R11: 0000000000000003 R12: ffffffffa0208000 ev-trig…e R13: ffffc90000dac7c8 R14: ffffc90000dac7e8 R15: dffffc0000000000 - Coldplug All uFS: 00007fe869216b40(0000) GS:ffff88841da00000(0000) knlGS:0000000000000000 dev Devices. CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: fffffbfff4041000 CR3: 0000000121e86002 CR4: 00000000003706f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: <TASK> [ OK ? __die+0x1f/0x60 0m] Reached targ ? page_fault_oops+0x258/0x910 et sysi ? dump_pagetable+0x690/0x690 nit.target - ? search_bpf_extables+0x22/0x250 System Initiali ? trace_page_fault_kernel+0x120/0x120 zation. ? search_bpf_extables+0x164/0x250 ? kasan_check_range+0xa5/0x190 ? fixup_exception+0x4d/0xc70 ? exc_page_fault+0xe1/0xf0 [ OK ? asm_exc_page_fault+0x22/0x30 0m] Reached targ ? load_module+0x3d7b/0x7560 et netw ? kasan_check_range+0xa5/0x190 ork.target - __asan_memcpy+0x38/0x60 Network. load_module+0x3d7b/0x7560 ? module_frob_arch_sections+0x30/0x30 ? lockdep_lock+0xbe/0x1b0 ? rw_verify_area+0x18d/0x5e0 ? kernel_read_file+0x246/0x870 ? __x64_sys_fspick+0x290/0x290 ? init_module_from_file+0xd1/0x130 init_module_from_file+0xd1/0x130 ? __ia32_sys_init_module+0xa0/0xa0 ? lock_acquire+0x2d/0xb0 ? idempotent_init_module+0x116/0x790 ? do_raw_spin_unlock+0x54/0x220 idempotent_init_module+0x226/0x790 ? init_module_from_file+0x130/0x130 ? vm_mmap_pgoff+0x203/0x2e0 __x64_sys_finit_module+0xba/0x130 do_syscall_64+0x69/0x160 entry_SYSCALL_64_after_hwframe+0x4b/0x53 RIP: 0033:0x7fe869de327d Code: 5d c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 248 RSP: 002b:00007ffe34a828d8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 RAX: ffffffffffffffda RBX: 0000557fa8f3f3f0 RCX: 00007fe869de327d RDX: 0000000000000000 RSI: 00007fe869f4943c RDI: 0000000000000006 RBP: 00007fe869f4943c R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000006 R11: 0000000000000246 R12: 0000000000020000 R13: 0000557fa8f3f030 R14: 0000000000000000 R15: 0000557fa8f3d110 </TASK> Modules linked in: CR2: fffffbfff4041000 ---[ end trace 0000000000000000 ]---Quoted 6 lines
I suspect you only hit this with an unlucky amount of debugging enabled. The kernel config I used is found here:
http://www.candelatech.com/downloads/cfg-kasan-crash-regression.config
I will be happy to test fixes.
Hi Ben,
Thanks for reporting the issue. Do you have these recent fixes in your tree:https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/all/[email protected]/Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
If not, couple you please apply them and see if the issue is still happening? Thanks,
Suren.Hello Suren,
Thanks for the quick response. The first patch is already in latest Linus tree,
Hmm. Could you please double-check which tree you are using? I don't see the first patch
(https://lore.kernel.org/all/[email protected]/)
in Linus' tree. Maybe you are using linux-next?
Quoted 8 lines
Quoted 5 lines
Sorry, you are correct. I must have mangled something when trying to apply the patch and I didn't look hard enough when patch said changes were already applied.
I can re-test this next week...and for reference, kernel boots fine when you disable KASAN and other debugging.
Thanks! Please retest with this patch and let me know if you are still having issues.
Suren.
Indeed, this is a bug that still exists in another context, namely when CONFIG_KASAN_VMALLOC is not enabled.
We may need to look into this scenario next.
Thanks
Best Regards
Hao
Quoted 9 lines
Quoted 9 lines
Thanks,
Ben--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
Quoted 123 lines · Hao Ge
Hi Suren
Quoted 118 lines · Suren Baghdasaryan
Quoted 117 lines · Ben Greear
Quoted 106 lines · Suren Baghdasaryan
Quoted 105 lines · Suren Baghdasaryan
Quoted 104 lines · Ben Greear
Quoted 103 lines · Suren Baghdasaryan
Quoted 87 lines · Ben Greear
Hello Suren,
My system crashes on bootup, and I bisected to this commit.
0f9b685626daa2f8e19a9788625c9b624c223e45is the first bad commit commit0f9b685626daa2f8e19a9788625c9b624c223e45Author: Suren Baghdasaryan <[email protected]> Date: Wed Oct 23 10:07:57 2024 -0700
alloc_tag: populate memory for module tags as neededThe memory reserved for module tags does not need to be backed by physical pages until there are tags to store there. Change the way we reserve this memory to allocate only virtual area for the tags and populate it with physical pages as needed when we load a module.
The crash looks like this:
BUG: unable to handle page fault for address: fffffbfff4041000 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 44d0e7067 P4D 44d0e7067 PUD 44d0e3067 PMD 10bb38067 PTE 0 Oops: Oops: 0000 [#1] PREEMPT SMP KASAN CPU: 0 UID: 0 PID: 319 Comm: systemd-udevd Not tainted 6.12.0-rc6+ #21 Hardware name: Default string Default string/SKYBAY, BIOS 5.12 02/15/2023 RIP: 0010:kasan_check_range+0xa5/0x190 Code: 8d 5a 07 4c 0f 49 da 49 c1 fb 03 45 85 db 0f 84 ce 00 00 00 45 89 db 4a 8d 14 d8 eb 0d 48 83 c0 08 48 39 d0 0f 84 b29 RSP: 0018:ffff88812c26f980 EFLAGS: 00010206 RAX: fffffbfff4041000 RBX: fffffbfff404101e RCX: ffffffff814ec29b [ OK DX: fffffbfff4041018 RSI: 00000000000000f0 RDI: ffffffffa0208000 0m] Finished BP: fffffbfff4041000 R08: 0000000000000001 R09: fffffbfff404101d ;1;39msystemd-udR10: ffffffffa02080ef R11: 0000000000000003 R12: ffffffffa0208000 ev-trig…e R13: ffffc90000dac7c8 R14: ffffc90000dac7e8 R15: dffffc0000000000 - Coldplug All uFS: 00007fe869216b40(0000) GS:ffff88841da00000(0000) knlGS:0000000000000000 dev Devices. CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: fffffbfff4041000 CR3: 0000000121e86002 CR4: 00000000003706f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: <TASK> [ OK ? __die+0x1f/0x60 0m] Reached targ ? page_fault_oops+0x258/0x910 et sysi ? dump_pagetable+0x690/0x690 nit.target - ? search_bpf_extables+0x22/0x250 System Initiali ? trace_page_fault_kernel+0x120/0x120 zation. ? search_bpf_extables+0x164/0x250 ? kasan_check_range+0xa5/0x190 ? fixup_exception+0x4d/0xc70 ? exc_page_fault+0xe1/0xf0 [ OK ? asm_exc_page_fault+0x22/0x30 0m] Reached targ ? load_module+0x3d7b/0x7560 et netw ? kasan_check_range+0xa5/0x190 ork.target - __asan_memcpy+0x38/0x60 Network. load_module+0x3d7b/0x7560 ? module_frob_arch_sections+0x30/0x30 ? lockdep_lock+0xbe/0x1b0 ? rw_verify_area+0x18d/0x5e0 ? kernel_read_file+0x246/0x870 ? __x64_sys_fspick+0x290/0x290 ? init_module_from_file+0xd1/0x130 init_module_from_file+0xd1/0x130 ? __ia32_sys_init_module+0xa0/0xa0 ? lock_acquire+0x2d/0xb0 ? idempotent_init_module+0x116/0x790 ? do_raw_spin_unlock+0x54/0x220 idempotent_init_module+0x226/0x790 ? init_module_from_file+0x130/0x130 ? vm_mmap_pgoff+0x203/0x2e0 __x64_sys_finit_module+0xba/0x130 do_syscall_64+0x69/0x160 entry_SYSCALL_64_after_hwframe+0x4b/0x53 RIP: 0033:0x7fe869de327d Code: 5d c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 248 RSP: 002b:00007ffe34a828d8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 RAX: ffffffffffffffda RBX: 0000557fa8f3f3f0 RCX: 00007fe869de327d RDX: 0000000000000000 RSI: 00007fe869f4943c RDI: 0000000000000006 RBP: 00007fe869f4943c R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000006 R11: 0000000000000246 R12: 0000000000020000 R13: 0000557fa8f3f030 R14: 0000000000000000 R15: 0000557fa8f3d110 </TASK> Modules linked in: CR2: fffffbfff4041000 ---[ end trace 0000000000000000 ]---Quoted 6 lines
I suspect you only hit this with an unlucky amount of debugging enabled. The kernel config I used is found here:
http://www.candelatech.com/downloads/cfg-kasan-crash-regression.config
I will be happy to test fixes.
Hi Ben,
Thanks for reporting the issue. Do you have these recent fixes in your tree:https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/all/[email protected]/If not, couple you please apply them and see if the issue is still happening? Thanks,
Suren.Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Hello Suren,
Thanks for the quick response. The first patch is already in latest Linus tree,
Hmm. Could you please double-check which tree you are using? I don't see the first patch
(https://lore.kernel.org/all/[email protected]/)
in Linus' tree. Maybe you are using linux-next?Sorry, you are correct. I must have mangled something when trying to apply the patch and I didn't look hard enough when patch said changes were already applied.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
I can re-test this next week...and for reference, kernel boots fine when you disable KASAN and other debugging.
Thanks! Please retest with this patch and let me know if you are still having issues.
Suren.Indeed, this is a bug that still exists in another context, namely when
CONFIG_KASAN_VMALLOCis not enabled.
Hmm. Are you able to reproduce this issue with all the fixes we had?
Quoted 18 lines
We may need to look into this scenario next.
Thanks
Best Regards
Hao
Quoted 9 lines
Quoted 9 lines
Thanks,
Ben--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
Hi Suren and Ben
Quoted 123 lines · Suren Baghdasaryan
Quoted 122 lines · Hao Ge
Hi Suren
Quoted 118 lines · Suren Baghdasaryan
Quoted 117 lines · Ben Greear
Quoted 106 lines · Suren Baghdasaryan
Quoted 105 lines · Suren Baghdasaryan
Quoted 104 lines · Ben Greear
Quoted 103 lines · Suren Baghdasaryan
Quoted 87 lines · Ben Greear
Hello Suren,
My system crashes on bootup, and I bisected to this commit.
0f9b685626daa2f8e19a9788625c9b624c223e45is the first bad commit commit0f9b685626daa2f8e19a9788625c9b624c223e45Author: Suren Baghdasaryan <[email protected]> Date: Wed Oct 23 10:07:57 2024 -0700
alloc_tag: populate memory for module tags as neededThe memory reserved for module tags does not need to be backed by physical pages until there are tags to store there. Change the way we reserve this memory to allocate only virtual area for the tags and populate it with physical pages as needed when we load a module.
The crash looks like this:
BUG: unable to handle page fault for address: fffffbfff4041000 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 44d0e7067 P4D 44d0e7067 PUD 44d0e3067 PMD 10bb38067 PTE 0 Oops: Oops: 0000 [#1] PREEMPT SMP KASAN CPU: 0 UID: 0 PID: 319 Comm: systemd-udevd Not tainted 6.12.0-rc6+ #21 Hardware name: Default string Default string/SKYBAY, BIOS 5.12 02/15/2023 RIP: 0010:kasan_check_range+0xa5/0x190 Code: 8d 5a 07 4c 0f 49 da 49 c1 fb 03 45 85 db 0f 84 ce 00 00 00 45 89 db 4a 8d 14 d8 eb 0d 48 83 c0 08 48 39 d0 0f 84 b29 RSP: 0018:ffff88812c26f980 EFLAGS: 00010206 RAX: fffffbfff4041000 RBX: fffffbfff404101e RCX: ffffffff814ec29b [ OK DX: fffffbfff4041018 RSI: 00000000000000f0 RDI: ffffffffa0208000 0m] Finished BP: fffffbfff4041000 R08: 0000000000000001 R09: fffffbfff404101d ;1;39msystemd-udR10: ffffffffa02080ef R11: 0000000000000003 R12: ffffffffa0208000 ev-trig…e R13: ffffc90000dac7c8 R14: ffffc90000dac7e8 R15: dffffc0000000000 - Coldplug All uFS: 00007fe869216b40(0000) GS:ffff88841da00000(0000) knlGS:0000000000000000 dev Devices. CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: fffffbfff4041000 CR3: 0000000121e86002 CR4: 00000000003706f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: <TASK> [ OK ? __die+0x1f/0x60 0m] Reached targ ? page_fault_oops+0x258/0x910 et sysi ? dump_pagetable+0x690/0x690 nit.target - ? search_bpf_extables+0x22/0x250 System Initiali ? trace_page_fault_kernel+0x120/0x120 zation. ? search_bpf_extables+0x164/0x250 ? kasan_check_range+0xa5/0x190 ? fixup_exception+0x4d/0xc70 ? exc_page_fault+0xe1/0xf0 [ OK ? asm_exc_page_fault+0x22/0x30 0m] Reached targ ? load_module+0x3d7b/0x7560 et netw ? kasan_check_range+0xa5/0x190 ork.target - __asan_memcpy+0x38/0x60 Network. load_module+0x3d7b/0x7560 ? module_frob_arch_sections+0x30/0x30 ? lockdep_lock+0xbe/0x1b0 ? rw_verify_area+0x18d/0x5e0 ? kernel_read_file+0x246/0x870 ? __x64_sys_fspick+0x290/0x290 ? init_module_from_file+0xd1/0x130 init_module_from_file+0xd1/0x130 ? __ia32_sys_init_module+0xa0/0xa0 ? lock_acquire+0x2d/0xb0 ? idempotent_init_module+0x116/0x790 ? do_raw_spin_unlock+0x54/0x220 idempotent_init_module+0x226/0x790 ? init_module_from_file+0x130/0x130 ? vm_mmap_pgoff+0x203/0x2e0 __x64_sys_finit_module+0xba/0x130 do_syscall_64+0x69/0x160 entry_SYSCALL_64_after_hwframe+0x4b/0x53 RIP: 0033:0x7fe869de327d Code: 5d c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 248 RSP: 002b:00007ffe34a828d8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 RAX: ffffffffffffffda RBX: 0000557fa8f3f3f0 RCX: 00007fe869de327d RDX: 0000000000000000 RSI: 00007fe869f4943c RDI: 0000000000000006 RBP: 00007fe869f4943c R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000006 R11: 0000000000000246 R12: 0000000000020000 R13: 0000557fa8f3f030 R14: 0000000000000000 R15: 0000557fa8f3d110 </TASK> Modules linked in: CR2: fffffbfff4041000 ---[ end trace 0000000000000000 ]---Quoted 6 lines
I suspect you only hit this with an unlucky amount of debugging enabled. The kernel config I used is found here:
http://www.candelatech.com/downloads/cfg-kasan-crash-regression.config
I will be happy to test fixes.
Hi Ben,
Thanks for reporting the issue. Do you have these recent fixes in your tree:https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/all/[email protected]/If not, couple you please apply them and see if the issue is still happening? Thanks,
Suren.Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Hello Suren,
Thanks for the quick response. The first patch is already in latest Linus tree,
Hmm. Could you please double-check which tree you are using? I don't see the first patch
(https://lore.kernel.org/all/[email protected]/)
in Linus' tree. Maybe you are using linux-next?Sorry, you are correct. I must have mangled something when trying to apply the patch and I didn't look hard enough when patch said changes were already applied.
Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
I can re-test this next week...and for reference, kernel boots fine when you disable KASAN and other debugging.
Thanks! Please retest with this patch and let me know if you are still having issues.
Suren.Indeed, this is a bug that still exists in another context, namely when
CONFIG_KASAN_VMALLOCis not enabled.Hmm. Are you able to reproduce this issue with all the fixes we had?
Yes, I set up an x86 virtual machine and after porting both of our patches, I encountered a reproduction of the issue.
I have also submitted a patch to fix this issue.
https://lore.kernel.org/all/[email protected]/
I verified it locally.
Hi Ben
Can you test with this patch?
Thanks
Best Regards
Hao
Quoted 18 lines
Quoted 18 lines
We may need to look into this scenario next.
Thanks
Best Regards
Hao
Quoted 9 lines
Quoted 9 lines
Thanks,
Ben--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
Quoted 138 lines · Hao Ge
Hi Suren and Ben
Quoted 123 lines · Suren Baghdasaryan
Quoted 122 lines · Hao Ge
Hi Suren
Quoted 118 lines · Suren Baghdasaryan
Quoted 117 lines · Ben Greear
Quoted 106 lines · Suren Baghdasaryan
Quoted 105 lines · Suren Baghdasaryan
Quoted 104 lines · Ben Greear
Quoted 103 lines · Suren Baghdasaryan
Quoted 87 lines · Ben Greear
Hello Suren,
My system crashes on bootup, and I bisected to this commit.
0f9b685626daa2f8e19a9788625c9b624c223e45is the first bad commit commit0f9b685626daa2f8e19a9788625c9b624c223e45Author: Suren Baghdasaryan <[email protected]> Date: Wed Oct 23 10:07:57 2024 -0700
alloc_tag: populate memory for module tags as neededThe memory reserved for module tags does not need to be backed by physical pages until there are tags to store there. Change the way we reserve this memory to allocate only virtual area for the tags and populate it with physical pages as needed when we load a module.
The crash looks like this:
BUG: unable to handle page fault for address: fffffbfff4041000 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 44d0e7067 P4D 44d0e7067 PUD 44d0e3067 PMD 10bb38067 PTE 0 Oops: Oops: 0000 [#1] PREEMPT SMP KASAN CPU: 0 UID: 0 PID: 319 Comm: systemd-udevd Not tainted 6.12.0-rc6+ #21 Hardware name: Default string Default string/SKYBAY, BIOS 5.12 02/15/2023 RIP: 0010:kasan_check_range+0xa5/0x190 Code: 8d 5a 07 4c 0f 49 da 49 c1 fb 03 45 85 db 0f 84 ce 00 00 00 45 89 db 4a 8d 14 d8 eb 0d 48 83 c0 08 48 39 d0 0f 84 b29 RSP: 0018:ffff88812c26f980 EFLAGS: 00010206 RAX: fffffbfff4041000 RBX: fffffbfff404101e RCX: ffffffff814ec29b [ OK DX: fffffbfff4041018 RSI: 00000000000000f0 RDI: ffffffffa0208000 0m] Finished BP: fffffbfff4041000 R08: 0000000000000001 R09: fffffbfff404101d ;1;39msystemd-udR10: ffffffffa02080ef R11: 0000000000000003 R12: ffffffffa0208000 ev-trig…e R13: ffffc90000dac7c8 R14: ffffc90000dac7e8 R15: dffffc0000000000 - Coldplug All uFS: 00007fe869216b40(0000) GS:ffff88841da00000(0000) knlGS:0000000000000000 dev Devices. CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: fffffbfff4041000 CR3: 0000000121e86002 CR4: 00000000003706f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: <TASK> [ OK ? __die+0x1f/0x60 0m] Reached targ ? page_fault_oops+0x258/0x910 et sysi ? dump_pagetable+0x690/0x690 nit.target - ? search_bpf_extables+0x22/0x250 System Initiali ? trace_page_fault_kernel+0x120/0x120 zation. ? search_bpf_extables+0x164/0x250 ? kasan_check_range+0xa5/0x190 ? fixup_exception+0x4d/0xc70 ? exc_page_fault+0xe1/0xf0 [ OK ? asm_exc_page_fault+0x22/0x30 0m] Reached targ ? load_module+0x3d7b/0x7560 et netw ? kasan_check_range+0xa5/0x190 ork.target - __asan_memcpy+0x38/0x60 Network. load_module+0x3d7b/0x7560 ? module_frob_arch_sections+0x30/0x30 ? lockdep_lock+0xbe/0x1b0 ? rw_verify_area+0x18d/0x5e0 ? kernel_read_file+0x246/0x870 ? __x64_sys_fspick+0x290/0x290 ? init_module_from_file+0xd1/0x130 init_module_from_file+0xd1/0x130 ? __ia32_sys_init_module+0xa0/0xa0 ? lock_acquire+0x2d/0xb0 ? idempotent_init_module+0x116/0x790 ? do_raw_spin_unlock+0x54/0x220 idempotent_init_module+0x226/0x790 ? init_module_from_file+0x130/0x130 ? vm_mmap_pgoff+0x203/0x2e0 __x64_sys_finit_module+0xba/0x130 do_syscall_64+0x69/0x160 entry_SYSCALL_64_after_hwframe+0x4b/0x53 RIP: 0033:0x7fe869de327d Code: 5d c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 248 RSP: 002b:00007ffe34a828d8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 RAX: ffffffffffffffda RBX: 0000557fa8f3f3f0 RCX: 00007fe869de327d RDX: 0000000000000000 RSI: 00007fe869f4943c RDI: 0000000000000006 RBP: 00007fe869f4943c R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000006 R11: 0000000000000246 R12: 0000000000020000 R13: 0000557fa8f3f030 R14: 0000000000000000 R15: 0000557fa8f3d110 </TASK> Modules linked in: CR2: fffffbfff4041000 ---[ end trace 0000000000000000 ]---Quoted 6 lines
I suspect you only hit this with an unlucky amount of debugging enabled. The kernel config I used is found here:
http://www.candelatech.com/downloads/cfg-kasan-crash-regression.config
I will be happy to test fixes.
Hi Ben,
Thanks for reporting the issue. Do you have these recent fixes in your tree:https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/all/[email protected]/If not, couple you please apply them and see if the issue is still happening? Thanks,
Suren.Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Hello Suren,
Thanks for the quick response. The first patch is already in latest Linus tree,
Hmm. Could you please double-check which tree you are using? I don't see the first patch
(https://lore.kernel.org/all/[email protected]/)
in Linus' tree. Maybe you are using linux-next?Sorry, you are correct. I must have mangled something when trying to apply the patch and I didn't look hard enough when patch said changes were already applied.
Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
I can re-test this next week...and for reference, kernel boots fine when you disable KASAN and other debugging.
Thanks! Please retest with this patch and let me know if you are still having issues.
Suren.Indeed, this is a bug that still exists in another context, namely when
CONFIG_KASAN_VMALLOCis not enabled.Hmm. Are you able to reproduce this issue with all the fixes we had?
Quoted 6 lines
Yes, I set up an x86 virtual machine and after porting both of our patches, I encountered a reproduction of the issue.
I have also submitted a patch to fix this issue.
Thanks! I'll post my comments there.
Quoted 33 lines
I verified it locally.
Hi Ben
Can you test with this patch?
Thanks
Best Regards
Hao
Quoted 18 lines
Quoted 18 lines
We may need to look into this scenario next.
Thanks
Best Regards
Hao
Quoted 9 lines
Quoted 9 lines
Thanks,
Ben--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com