22465bbac5: BUG:KASAN:slab-use-after-free_in__cpuhp_state_add_instance_cpuslocked
kernel test robot
build botHello,
kernel test robot noticed "BUG:KASAN:slab-use-after-free_in__cpuhp_state_add_instance_cpuslocked" on:
commit: 22465bbac53c821319089016f268a2437de9b00a ("blk-mq: move cpuhp callback registering out of q->sysfs_lock")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
[test failed on linus/master 231825b2e1ff6ba799c5eaf396d3ab2354e37c6b]
[test failed on linux-next/master 3e42dc9229c5950e84b1ed705f94ed75ed208228]
in testcase: blktests
version: blktests-x86_64-3617edd-1_20241105
with following parameters:
disk: 1SSD
test: block-group-01
config: x86_64-rhel-9.4-func
compiler: gcc-12
test machine: 4 threads Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz (Skylake) with 32G memory
(please refer to attached dmesg/kmsg for entire log/backtrace)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-lkp/[email protected]
[ 232.596698][ T3545] BUG: KASAN: slab-use-after-free in __cpuhp_state_add_instance_cpuslocked (include/linux/list.h:1026 kernel/cpu.c:2446) [ 232.606144][ T3545] Write of size 8 at addr ffff88880ca5d968 by task check/3545 [ 232.613424][ T3545] [ 232.615602][ T3545] CPU: 3 UID: 0 PID: 3545 Comm: check Not tainted 6.13.0-rc1-00018-g22465bbac53c #1 [ 232.624789][ T3545] Hardware name: Dell Inc. OptiPlex 7040/0Y7WYT, BIOS 1.8.1 12/05/2017 [ 232.632859][ T3545] Call Trace: [ 232.635994][ T3545] <TASK> [ 232.638783][ T3545] dump_stack_lvl (lib/dump_stack.c:123 (discriminator 1)) [ 232.643131][ T3545] print_address_description+0x2c/0x3a0 [ 232.649563][ T3545] ? __cpuhp_state_add_instance_cpuslocked (include/linux/list.h:1026 kernel/cpu.c:2446) [ 232.656244][ T3545] print_report (mm/kasan/report.c:490) [ 232.660497][ T3545] ? kasan_addr_to_slab (mm/kasan/common.c:37) [ 232.665273][ T3545] ? __cpuhp_state_add_instance_cpuslocked (include/linux/list.h:1026 kernel/cpu.c:2446) [ 232.671947][ T3545] kasan_report (mm/kasan/report.c:604) [ 232.676118][ T3545] ? __cpuhp_state_add_instance_cpuslocked (include/linux/list.h:1026 kernel/cpu.c:2446) [ 232.682795][ T3545] __cpuhp_state_add_instance_cpuslocked (include/linux/list.h:1026 kernel/cpu.c:2446) [ 232.689297][ T3545] ? __pfx_mutex_lock (kernel/locking/mutex.c:257) [ 232.693984][ T3545] __cpuhp_state_add_instance (kernel/cpu.c:2458) [ 232.699364][ T3545] blk_mq_realloc_hw_ctxs (include/linux/cpuhotplug.h:402 block/blk-mq.c:3774 block/blk-mq.c:3826 block/blk-mq.c:4497) [ 232.704571][ T3545] ? kfree (mm/slub.c:4598 mm/slub.c:4746) [ 232.708393][ T3545] ? __pfx_blk_mq_realloc_hw_ctxs (block/blk-mq.c:4453) [ 232.714117][ T3545] ? null_map_queues (drivers/block/null_blk/main.c:1502 (discriminator 2)) null_blk [ 232.719852][ T3545] __blk_mq_update_nr_hw_queues (block/blk-mq.c:5021) [ 232.725578][ T3545] ? __mark_inode_dirty (fs/fs-writeback.c:2619) [ 232.730610][ T3545] ? __pfx_rmqueue+0x10/0x10 [ 232.736092][ T3545] ? kasan_unpoison (mm/kasan/shadow.c:156 mm/kasan/shadow.c:182) [ 232.740605][ T3545] ? __kasan_unpoison_pages (mm/kasan/common.c:136) [ 232.745810][ T3545] ? post_alloc_hook (mm/page_alloc.c:1539) [ 232.750497][ T3545] ? arch_stack_walk (arch/x86/kernel/stacktrace.c:24) [ 232.755098][ T3545] ? __pfx___blk_mq_update_nr_hw_queues (block/blk-mq.c:4980) [ 232.761344][ T3545] ? mutex_lock (arch/x86/include/asm/atomic64_64.h:101 include/linux/atomic/atomic-arch-fallback.h:4296 include/linux/atomic/atomic-long.h:1482 include/linux/atomic/atomic-instrumented.h:4458 kernel/locking/mutex.c:146 kernel/locking/mutex.c:260) [ 232.765512][ T3545] ? __pfx_mutex_lock (kernel/locking/mutex.c:257) [ 232.770200][ T3545] blk_mq_update_nr_hw_queues (block/blk-mq.c:5064) [ 232.775580][ T3545] nullb_update_nr_hw_queues (drivers/block/null_blk/main.c:406) null_blk [ 232.782002][ T3545] nullb_device_submit_queues_store (drivers/block/null_blk/main.c:424 (discriminator 3) drivers/block/null_blk/main.c:443 (discriminator 3)) null_blk [ 232.788944][ T3545] ? __pfx_nullb_device_submit_queues_store (drivers/block/null_blk/main.c:443) null_blk [ 232.796490][ T3545] ? kmem_cache_free (mm/slub.c:2303 mm/slub.c:4598 mm/slub.c:4700) [ 232.801262][ T3545] ? syscall_exit_to_user_mode (include/linux/resume_user_mode.h:50 kernel/entry/common.c:114 include/linux/entry-common.h:329 kernel/entry/common.c:207 kernel/entry/common.c:218) [ 232.806900][ T3545] configfs_write_iter (fs/configfs/file.c:207 fs/configfs/file.c:229) [ 232.811845][ T3545] ? __pfx_configfs_write_iter (fs/configfs/file.c:221) [ 232.817311][ T3545] vfs_write (fs/read_write.c:586 fs/read_write.c:679) [ 232.821393][ T3545] ? get_close_on_exec (fs/file.c:1222) [ 232.826167][ T3545] ? __pfx_vfs_write (fs/read_write.c:660) [ 232.830767][ T3545] ? folio_xchg_last_cpupid (mm/mmzone.c:109 (discriminator 14)) [ 232.836061][ T3545] ? __pfx_ptep_set_access_flags (arch/x86/mm/pgtable.c:505) [ 232.841697][ T3545] ? __pfx_pte_mkwrite (arch/x86/mm/pgtable.c:903) [ 232.846472][ T3545] ? fdget_pos (arch/x86/include/asm/atomic64_64.h:15 include/linux/atomic/atomic-arch-fallback.h:2583 include/linux/atomic/atomic-long.h:38 include/linux/atomic/atomic-instrumented.h:3189 include/linux/file_ref.h:171 fs/file.c:1181 fs/file.c:1189) [ 232.850726][ T3545] ? do_wp_page (include/linux/vmstat.h:75 mm/memory.c:3277 mm/memory.c:3745) [ 232.855156][ T3545] ksys_write (fs/read_write.c:731) [ 232.859238][ T3545] ? __pfx_ksys_write (fs/read_write.c:721) [ 232.863927][ T3545] ? __pfx_handle_pte_fault (mm/memory.c:5758) [ 232.869141][ T3545] ? __pfx_expand_files (fs/file.c:270) [ 232.874007][ T3545] do_syscall_64 (arch/x86/entry/common.c:52 arch/x86/entry/common.c:83) [ 232.878357][ T3545] ? __handle_mm_fault (mm/memory.c:5944) [ 232.883396][ T3545] ? __pfx___handle_mm_fault (mm/memory.c:5853) [ 232.888702][ T3545] ? __count_memcg_events (mm/memcontrol.c:583 mm/memcontrol.c:857) [ 232.893913][ T3545] ? handle_mm_fault (mm/memory.c:5986 mm/memory.c:6138) [ 232.898699][ T3545] ? do_user_addr_fault (include/linux/rcupdate.h:882 include/linux/mm.h:741 arch/x86/mm/fault.c:1340) [ 232.903736][ T3545] ? exc_page_fault (arch/x86/include/asm/irqflags.h:37 arch/x86/include/asm/irqflags.h:92 arch/x86/mm/fault.c:1489 arch/x86/mm/fault.c:1539) [ 232.908250][ T3545] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:130) [ 232.913976][ T3545] RIP: 0033:0x7ffb9ce86240 [ 232.918232][ T3545] Code: 40 00 48 8b 15 c1 9b 0d 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 80 3d a1 23 0e 00 00 74 17 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 58 c3 0f 1f 80 00 00 00 00 48 83 ec 28 48 89 All code ======== 0: 40 00 48 8b rex add %cl,-0x75(%rax) 4: 15 c1 9b 0d 00 adc $0xd9bc1,%eax 9: f7 d8 neg %eax b: 64 89 02 mov %eax,%fs:(%rdx) e: 48 c7 c0 ff ff ff ff mov $0xffffffffffffffff,%rax 15: eb b7 jmp 0xffffffffffffffce 17: 0f 1f 00 nopl (%rax) 1a: 80 3d a1 23 0e 00 00 cmpb $0x0,0xe23a1(%rip) # 0xe23c2 21: 74 17 je 0x3a 23: b8 01 00 00 00 mov $0x1,%eax 28: 0f 05 syscall 2a:* 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax <-- trapping instruction 30: 77 58 ja 0x8a 32: c3 ret 33: 0f 1f 80 00 00 00 00 nopl 0x0(%rax) 3a: 48 83 ec 28 sub $0x28,%rsp 3e: 48 rex.W 3f: 89 .byte 0x89
Code starting with the faulting instruction
===========================================
0: 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax 6: 77 58 ja 0x60 8: c3 ret 9: 0f 1f 80 00 00 00 00 nopl 0x0(%rax) 10: 48 83 ec 28 sub $0x28,%rsp 14: 48 rex.W 15: 89 .byte 0x89 [ 232.937637][ T3545] RSP: 002b:00007fffc21695e8 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ 232.945886][ T3545] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007ffb9ce86240 [ 232.953684][ T3545] RDX: 0000000000000002 RSI: 000055dbf9920c00 RDI: 0000000000000001 [ 232.961480][ T3545] RBP: 000055dbf9920c00 R08: 0000000000000007 R09: 0000000000000073 [ 232.969279][ T3545] R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000002 [ 232.977076][ T3545] R13: 00007ffb9cf61760 R14: 0000000000000002 R15: 00007ffb9cf5c9e0 [ 232.984876][ T3545] </TASK> [ 232.987746][ T3545] [ 232.989925][ T3545] Allocated by task 3545: [ 232.994092][ T3545] kasan_save_stack (mm/kasan/common.c:48) [ 232.998604][ T3545] kasan_save_track (arch/x86/include/asm/current.h:49 mm/kasan/common.c:60 mm/kasan/common.c:69) [ 233.003120][ T3545] __kasan_kmalloc (mm/kasan/common.c:377 mm/kasan/common.c:394) [ 233.007546][ T3545] blk_mq_alloc_hctx (include/linux/slab.h:924 block/blk-mq.c:3937) [ 233.012233][ T3545] blk_mq_alloc_and_init_hctx (block/blk-mq.c:4436) [ 233.017782][ T3545] blk_mq_realloc_hw_ctxs (block/blk-mq.c:4469) [ 233.022989][ T3545] blk_mq_init_allocated_queue (block/blk-mq.c:4524) [ 233.028714][ T3545] blk_mq_alloc_queue (block/blk-mq.c:4345) [ 233.033573][ T3545] __blk_mq_alloc_disk (block/blk-mq.c:4388) [ 233.038347][ T3545] null_add_dev (drivers/block/null_blk/main.c:1941) null_blk [ 233.043731][ T3545] nullb_device_power_store (include/linux/instrumented.h:82 include/asm-generic/bitops/instrumented-atomic.h:41 drivers/block/null_blk/main.c:496) null_blk [ 233.050064][ T3545] configfs_write_iter (fs/configfs/file.c:207 fs/configfs/file.c:229) [ 233.055010][ T3545] vfs_write (fs/read_write.c:586 fs/read_write.c:679) [ 233.059092][ T3545] ksys_write (fs/read_write.c:731) [ 233.063172][ T3545] do_syscall_64 (arch/x86/entry/common.c:52 arch/x86/entry/common.c:83) [ 233.067513][ T3545] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:130) [ 233.073237][ T3545] [ 233.075414][ T3545] Freed by task 3545: [ 233.079234][ T3545] kasan_save_stack (mm/kasan/common.c:48) [ 233.083750][ T3545] kasan_save_track (arch/x86/include/asm/current.h:49 mm/kasan/common.c:60 mm/kasan/common.c:69) [ 233.088261][ T3545] kasan_save_free_info (mm/kasan/generic.c:585) [ 233.093122][ T3545] __kasan_slab_free (mm/kasan/common.c:271) [ 233.097721][ T3545] kfree (mm/slub.c:4598 mm/slub.c:4746) [ 233.101370][ T3545] kobject_cleanup (lib/kobject.c:689) [ 233.105979][ T3545] blk_mq_alloc_and_init_hctx (block/blk-mq.c:4446) [ 233.111539][ T3545] blk_mq_realloc_hw_ctxs (block/blk-mq.c:4469) [ 233.116754][ T3545] __blk_mq_update_nr_hw_queues (block/blk-mq.c:5021) [ 233.122486][ T3545] blk_mq_update_nr_hw_queues (block/blk-mq.c:5064) [ 233.127876][ T3545] nullb_update_nr_hw_queues (drivers/block/null_blk/main.c:406) null_blk [ 233.134304][ T3545] nullb_device_submit_queues_store (drivers/block/null_blk/main.c:424 (discriminator 3) drivers/block/null_blk/main.c:443 (discriminator 3)) null_blk [ 233.141245][ T3545] configfs_write_iter (fs/configfs/file.c:207 fs/configfs/file.c:229) [ 233.146194][ T3545] vfs_write (fs/read_write.c:586 fs/read_write.c:679) [ 233.150274][ T3545] ksys_write (fs/read_write.c:731) [ 233.154355][ T3545] do_syscall_64 (arch/x86/entry/common.c:52 arch/x86/entry/common.c:83) [ 233.158708][ T3545] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:130) [ 233.164431][ T3545] [ 233.166607][ T3545] The buggy address belongs to the object at ffff88880ca5d800 [ 233.166607][ T3545] which belongs to the cache kmalloc-512 of size 512 [ 233.180477][ T3545] The buggy address is located 360 bytes inside of [ 233.180477][ T3545] freed 512-byte region [ffff88880ca5d800, ffff88880ca5da00) [ 233.194071][ T3545] [ 233.196248][ T3545] The buggy address belongs to the physical page: [ 233.202492][ T3545] page: refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x80ca5c [ 233.211156][ T3545] head: order:2 mapcount:0 entire_mapcount:0 nr_pages_mapped:0 pincount:0 [ 233.219471][ T3545] flags: 0x17ffffc0000040(head|node=0|zone=2|lastcpupid=0x1fffff) [ 233.227097][ T3545] page_type: f5(slab) [ 233.230921][ T3545] raw: 0017ffffc0000040 ffff88810c842c80 ffffea001ffdb900 dead000000000002 [ 233.239324][ T3545] raw: 0000000000000000 0000000000100010 00000001f5000000 0000000000000000 [ 233.247742][ T3545] head: 0017ffffc0000040 ffff88810c842c80 ffffea001ffdb900 dead000000000002 [ 233.256233][ T3545] head: 0000000000000000 0000000000100010 00000001f5000000 0000000000000000
The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20241217/[email protected]
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Quoted 29 lines · kernel test robot
Hello,
kernel test robot noticed "BUG:KASAN:slab-use-after-free_in__cpuhp_state_add_instance_cpuslocked" on:
commit:
22465bbac53c821319089016f268a2437de9b00a("blk-mq: move cpuhp callback registering out ofq->sysfs_lock")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master[test failed on linus/master
231825b2e1ff6ba799c5eaf396d3ab2354e37c6b]
[test failed on linux-next/master3e42dc9229c5950e84b1ed705f94ed75ed208228]in testcase: blktests
version: blktests-x86_64-3617edd-1_20241105
with following parameters:disk: 1SSD
test: block-group-01config: x86_64-rhel-9.4-func
compiler: gcc-12
test machine: 4 threads Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz (Skylake) with 32G memory(please refer to attached dmesg/kmsg for entire log/backtrace)
Quoted 7 lines
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-lkp/[email protected][ 232.596698][ T3545] BUG: KASAN: slab-use-after-free in __cpuhp_state_add_instance_cpuslocked (include/linux/list.h:1026 kernel/cpu.c:2446)
Hello,
Thanks for the report!
Unfortunately I can't reproduce it in my test VM by running 'blktests block/030' with:
- two numa nodes
- enable CONFIG_BLK_DEV_NULL_BLK_FAULT_INJECTION
But just figured out that one freed hctx still may stay in cpuhp cb list, can you test the following patch?
block/blk-mq.c +2 -1
@@ -4421,7 +4421,8 @@ static struct blk_mq_hw_ctx *blk_mq_alloc_and_init_hctx(/* reuse dead hctx first */ spin_lock (&q->unused_hctx_lock);list_for_each_entry (tmp, &q->unused_hctx_list, hctx_list) { -if (tmp->numa_node == node) { +if (tmp->numa_node == node && +hlist_unhashed (&tmp->cpuhp_online)) { hctx = tmp;break ; }
thanks,
Ming
Oliver Sang
Tested-byhi, Ming,
Quoted 45 lines · Ming Lei
Quoted 29 lines · kernel test robot
Hello,
kernel test robot noticed "BUG:KASAN:slab-use-after-free_in__cpuhp_state_add_instance_cpuslocked" on:
commit:
22465bbac53c821319089016f268a2437de9b00a("blk-mq: move cpuhp callback registering out ofq->sysfs_lock")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master[test failed on linus/master
231825b2e1ff6ba799c5eaf396d3ab2354e37c6b]
[test failed on linux-next/master3e42dc9229c5950e84b1ed705f94ed75ed208228]in testcase: blktests
version: blktests-x86_64-3617edd-1_20241105
with following parameters:disk: 1SSD
test: block-group-01config: x86_64-rhel-9.4-func
compiler: gcc-12
test machine: 4 threads Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz (Skylake) with 32G memory(please refer to attached dmesg/kmsg for entire log/backtrace)
Quoted 7 lines
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-lkp/[email protected][ 232.596698][ T3545] BUG: KASAN: slab-use-after-free in __cpuhp_state_add_instance_cpuslocked (include/linux/list.h:1026 kernel/cpu.c:2446)Hello,
Thanks for the report!
Unfortunately I can't reproduce it in my test VM by running 'blktests block/030' with:
Quoted 6 lines
- two numa nodes
- enableCONFIG_BLK_DEV_NULL_BLK_FAULT_INJECTIONBut just figured out that one freed hctx still may stay in cpuhp cb list, can you test the following patch?
yes, your patch fixed the issue we reported, and 'blktests block/030' can pass now. thanks!
BTW, the config built with the patch is same as for 22465bbac5 when we made original report, so still with CONFIG_BLK_DEV_NULL_BLK_FAULT_INJECTION.
Quoted 21 lines
block/blk-mq.c +2
-1@@ -4421,7 +4421,8 @@ static struct blk_mq_hw_ctx *blk_mq_alloc_and_init_hctx(/* reuse dead hctx first */ spin_lock (&q->unused_hctx_lock);list_for_each_entry (tmp, &q->unused_hctx_list, hctx_list) { -if (tmp->numa_node == node) { +if (tmp->numa_node == node && +hlist_unhashed (&tmp->cpuhp_online)) { hctx = tmp;break ; }thanks,
Ming