Skip to contentSkip to conversation

22465bbac5: BUG:KASAN:slab-use-after-free_in__cpuhp_state_add_instance_cpuslocked

linus:master kernel test robot 2 participants 3 messages Dec 17, 2024 – Dec 18, 2024 Block layer

lore
Share & tools
lore

kernel test robot

author build bot
#0 lore raw

Hello,

kernel test robot noticed "BUG:KASAN:slab-use-after-free_in__cpuhp_state_add_instance_cpuslocked" on:

commit: 22465bbac53c821319089016f268a2437de9b00a ("blk-mq: move cpuhp callback registering out of q->sysfs_lock")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

[test failed on linus/master 231825b2e1ff6ba799c5eaf396d3ab2354e37c6b]
[test failed on linux-next/master 3e42dc9229c5950e84b1ed705f94ed75ed208228]

in testcase: blktests
version: blktests-x86_64-3617edd-1_20241105
with following parameters:

disk: 1SSD
test: block-group-01

config: x86_64-rhel-9.4-func
compiler: gcc-12
test machine: 4 threads Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz (Skylake) with 32G memory

(please refer to attached dmesg/kmsg for entire log/backtrace)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-lkp/[email protected]

[ 232.596698][ T3545] BUG: KASAN: slab-use-after-free in __cpuhp_state_add_instance_cpuslocked (include/linux/list.h:1026 kernel/cpu.c:2446)
[  232.606144][ T3545] Write of size 8 at addr ffff88880ca5d968 by task check/3545
[  232.613424][ T3545]
[  232.615602][ T3545] CPU: 3 UID: 0 PID: 3545 Comm: check Not tainted 6.13.0-rc1-00018-g22465bbac53c #1
[  232.624789][ T3545] Hardware name: Dell Inc. OptiPlex 7040/0Y7WYT, BIOS 1.8.1 12/05/2017
[  232.632859][ T3545] Call Trace:
[  232.635994][ T3545]  <TASK>
[ 232.638783][ T3545] dump_stack_lvl (lib/dump_stack.c:123 (discriminator 1))
[ 232.643131][ T3545] print_address_description+0x2c/0x3a0
[ 232.649563][ T3545] ? __cpuhp_state_add_instance_cpuslocked (include/linux/list.h:1026 kernel/cpu.c:2446)
[ 232.656244][ T3545] print_report (mm/kasan/report.c:490)
[ 232.660497][ T3545] ? kasan_addr_to_slab (mm/kasan/common.c:37)
[ 232.665273][ T3545] ? __cpuhp_state_add_instance_cpuslocked (include/linux/list.h:1026 kernel/cpu.c:2446)
[ 232.671947][ T3545] kasan_report (mm/kasan/report.c:604)
[ 232.676118][ T3545] ? __cpuhp_state_add_instance_cpuslocked (include/linux/list.h:1026 kernel/cpu.c:2446)
[ 232.682795][ T3545] __cpuhp_state_add_instance_cpuslocked (include/linux/list.h:1026 kernel/cpu.c:2446)
[ 232.689297][ T3545] ? __pfx_mutex_lock (kernel/locking/mutex.c:257)
[ 232.693984][ T3545] __cpuhp_state_add_instance (kernel/cpu.c:2458)
[ 232.699364][ T3545] blk_mq_realloc_hw_ctxs (include/linux/cpuhotplug.h:402 block/blk-mq.c:3774 block/blk-mq.c:3826 block/blk-mq.c:4497)
[ 232.704571][ T3545] ? kfree (mm/slub.c:4598 mm/slub.c:4746)
[ 232.708393][ T3545] ? __pfx_blk_mq_realloc_hw_ctxs (block/blk-mq.c:4453)
[ 232.714117][ T3545] ? null_map_queues (drivers/block/null_blk/main.c:1502 (discriminator 2)) null_blk
[ 232.719852][ T3545] __blk_mq_update_nr_hw_queues (block/blk-mq.c:5021)
[ 232.725578][ T3545] ? __mark_inode_dirty (fs/fs-writeback.c:2619)
[ 232.730610][ T3545] ? __pfx_rmqueue+0x10/0x10
[ 232.736092][ T3545] ? kasan_unpoison (mm/kasan/shadow.c:156 mm/kasan/shadow.c:182)
[ 232.740605][ T3545] ? __kasan_unpoison_pages (mm/kasan/common.c:136)
[ 232.745810][ T3545] ? post_alloc_hook (mm/page_alloc.c:1539)
[ 232.750497][ T3545] ? arch_stack_walk (arch/x86/kernel/stacktrace.c:24)
[ 232.755098][ T3545] ? __pfx___blk_mq_update_nr_hw_queues (block/blk-mq.c:4980)
[ 232.761344][ T3545] ? mutex_lock (arch/x86/include/asm/atomic64_64.h:101 include/linux/atomic/atomic-arch-fallback.h:4296 include/linux/atomic/atomic-long.h:1482 include/linux/atomic/atomic-instrumented.h:4458 kernel/locking/mutex.c:146 kernel/locking/mutex.c:260)
[ 232.765512][ T3545] ? __pfx_mutex_lock (kernel/locking/mutex.c:257)
[ 232.770200][ T3545] blk_mq_update_nr_hw_queues (block/blk-mq.c:5064)
[ 232.775580][ T3545] nullb_update_nr_hw_queues (drivers/block/null_blk/main.c:406) null_blk
[ 232.782002][ T3545] nullb_device_submit_queues_store (drivers/block/null_blk/main.c:424 (discriminator 3) drivers/block/null_blk/main.c:443 (discriminator 3)) null_blk
[ 232.788944][ T3545] ? __pfx_nullb_device_submit_queues_store (drivers/block/null_blk/main.c:443) null_blk
[ 232.796490][ T3545] ? kmem_cache_free (mm/slub.c:2303 mm/slub.c:4598 mm/slub.c:4700)
[ 232.801262][ T3545] ? syscall_exit_to_user_mode (include/linux/resume_user_mode.h:50 kernel/entry/common.c:114 include/linux/entry-common.h:329 kernel/entry/common.c:207 kernel/entry/common.c:218)
[ 232.806900][ T3545] configfs_write_iter (fs/configfs/file.c:207 fs/configfs/file.c:229)
[ 232.811845][ T3545] ? __pfx_configfs_write_iter (fs/configfs/file.c:221)
[ 232.817311][ T3545] vfs_write (fs/read_write.c:586 fs/read_write.c:679)
[ 232.821393][ T3545] ? get_close_on_exec (fs/file.c:1222)
[ 232.826167][ T3545] ? __pfx_vfs_write (fs/read_write.c:660)
[ 232.830767][ T3545] ? folio_xchg_last_cpupid (mm/mmzone.c:109 (discriminator 14))
[ 232.836061][ T3545] ? __pfx_ptep_set_access_flags (arch/x86/mm/pgtable.c:505)
[ 232.841697][ T3545] ? __pfx_pte_mkwrite (arch/x86/mm/pgtable.c:903)
[ 232.846472][ T3545] ? fdget_pos (arch/x86/include/asm/atomic64_64.h:15 include/linux/atomic/atomic-arch-fallback.h:2583 include/linux/atomic/atomic-long.h:38 include/linux/atomic/atomic-instrumented.h:3189 include/linux/file_ref.h:171 fs/file.c:1181 fs/file.c:1189)
[ 232.850726][ T3545] ? do_wp_page (include/linux/vmstat.h:75 mm/memory.c:3277 mm/memory.c:3745)
[ 232.855156][ T3545] ksys_write (fs/read_write.c:731)
[ 232.859238][ T3545] ? __pfx_ksys_write (fs/read_write.c:721)
[ 232.863927][ T3545] ? __pfx_handle_pte_fault (mm/memory.c:5758)
[ 232.869141][ T3545] ? __pfx_expand_files (fs/file.c:270)
[ 232.874007][ T3545] do_syscall_64 (arch/x86/entry/common.c:52 arch/x86/entry/common.c:83)
[ 232.878357][ T3545] ? __handle_mm_fault (mm/memory.c:5944)
[ 232.883396][ T3545] ? __pfx___handle_mm_fault (mm/memory.c:5853)
[ 232.888702][ T3545] ? __count_memcg_events (mm/memcontrol.c:583 mm/memcontrol.c:857)
[ 232.893913][ T3545] ? handle_mm_fault (mm/memory.c:5986 mm/memory.c:6138)
[ 232.898699][ T3545] ? do_user_addr_fault (include/linux/rcupdate.h:882 include/linux/mm.h:741 arch/x86/mm/fault.c:1340)
[ 232.903736][ T3545] ? exc_page_fault (arch/x86/include/asm/irqflags.h:37 arch/x86/include/asm/irqflags.h:92 arch/x86/mm/fault.c:1489 arch/x86/mm/fault.c:1539)
[ 232.908250][ T3545] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:130)
[  232.913976][ T3545] RIP: 0033:0x7ffb9ce86240
[ 232.918232][ T3545] Code: 40 00 48 8b 15 c1 9b 0d 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 80 3d a1 23 0e 00 00 74 17 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 58 c3 0f 1f 80 00 00 00 00 48 83 ec 28 48 89
All code
========
   0:	40 00 48 8b          	rex add %cl,-0x75(%rax)
   4:	15 c1 9b 0d 00       	adc    $0xd9bc1,%eax
   9:	f7 d8                	neg    %eax
   b:	64 89 02             	mov    %eax,%fs:(%rdx)
   e:	48 c7 c0 ff ff ff ff 	mov    $0xffffffffffffffff,%rax
  15:	eb b7                	jmp    0xffffffffffffffce
  17:	0f 1f 00             	nopl   (%rax)
  1a:	80 3d a1 23 0e 00 00 	cmpb   $0x0,0xe23a1(%rip)        # 0xe23c2
  21:	74 17                	je     0x3a
  23:	b8 01 00 00 00       	mov    $0x1,%eax
  28:	0f 05                	syscall
  2a:*	48 3d 00 f0 ff ff    	cmp    $0xfffffffffffff000,%rax		<-- trapping instruction
  30:	77 58                	ja     0x8a
  32:	c3                   	ret
  33:	0f 1f 80 00 00 00 00 	nopl   0x0(%rax)
  3a:	48 83 ec 28          	sub    $0x28,%rsp
  3e:	48                   	rex.W
  3f:	89                   	.byte 0x89

Code starting with the faulting instruction
===========================================

   0:	48 3d 00 f0 ff ff    	cmp    $0xfffffffffffff000,%rax
   6:	77 58                	ja     0x60
   8:	c3                   	ret
   9:	0f 1f 80 00 00 00 00 	nopl   0x0(%rax)
  10:	48 83 ec 28          	sub    $0x28,%rsp
  14:	48                   	rex.W
  15:	89                   	.byte 0x89
[  232.937637][ T3545] RSP: 002b:00007fffc21695e8 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[  232.945886][ T3545] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007ffb9ce86240
[  232.953684][ T3545] RDX: 0000000000000002 RSI: 000055dbf9920c00 RDI: 0000000000000001
[  232.961480][ T3545] RBP: 000055dbf9920c00 R08: 0000000000000007 R09: 0000000000000073
[  232.969279][ T3545] R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000002
[  232.977076][ T3545] R13: 00007ffb9cf61760 R14: 0000000000000002 R15: 00007ffb9cf5c9e0
[  232.984876][ T3545]  </TASK>
[  232.987746][ T3545]
[  232.989925][ T3545] Allocated by task 3545:
[ 232.994092][ T3545] kasan_save_stack (mm/kasan/common.c:48)
[ 232.998604][ T3545] kasan_save_track (arch/x86/include/asm/current.h:49 mm/kasan/common.c:60 mm/kasan/common.c:69)
[ 233.003120][ T3545] __kasan_kmalloc (mm/kasan/common.c:377 mm/kasan/common.c:394)
[ 233.007546][ T3545] blk_mq_alloc_hctx (include/linux/slab.h:924 block/blk-mq.c:3937)
[ 233.012233][ T3545] blk_mq_alloc_and_init_hctx (block/blk-mq.c:4436)
[ 233.017782][ T3545] blk_mq_realloc_hw_ctxs (block/blk-mq.c:4469)
[ 233.022989][ T3545] blk_mq_init_allocated_queue (block/blk-mq.c:4524)
[ 233.028714][ T3545] blk_mq_alloc_queue (block/blk-mq.c:4345)
[ 233.033573][ T3545] __blk_mq_alloc_disk (block/blk-mq.c:4388)
[ 233.038347][ T3545] null_add_dev (drivers/block/null_blk/main.c:1941) null_blk
[ 233.043731][ T3545] nullb_device_power_store (include/linux/instrumented.h:82 include/asm-generic/bitops/instrumented-atomic.h:41 drivers/block/null_blk/main.c:496) null_blk
[ 233.050064][ T3545] configfs_write_iter (fs/configfs/file.c:207 fs/configfs/file.c:229)
[ 233.055010][ T3545] vfs_write (fs/read_write.c:586 fs/read_write.c:679)
[ 233.059092][ T3545] ksys_write (fs/read_write.c:731)
[ 233.063172][ T3545] do_syscall_64 (arch/x86/entry/common.c:52 arch/x86/entry/common.c:83)
[ 233.067513][ T3545] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:130)
[  233.073237][ T3545]
[  233.075414][ T3545] Freed by task 3545:
[ 233.079234][ T3545] kasan_save_stack (mm/kasan/common.c:48)
[ 233.083750][ T3545] kasan_save_track (arch/x86/include/asm/current.h:49 mm/kasan/common.c:60 mm/kasan/common.c:69)
[ 233.088261][ T3545] kasan_save_free_info (mm/kasan/generic.c:585)
[ 233.093122][ T3545] __kasan_slab_free (mm/kasan/common.c:271)
[ 233.097721][ T3545] kfree (mm/slub.c:4598 mm/slub.c:4746)
[ 233.101370][ T3545] kobject_cleanup (lib/kobject.c:689)
[ 233.105979][ T3545] blk_mq_alloc_and_init_hctx (block/blk-mq.c:4446)
[ 233.111539][ T3545] blk_mq_realloc_hw_ctxs (block/blk-mq.c:4469)
[ 233.116754][ T3545] __blk_mq_update_nr_hw_queues (block/blk-mq.c:5021)
[ 233.122486][ T3545] blk_mq_update_nr_hw_queues (block/blk-mq.c:5064)
[ 233.127876][ T3545] nullb_update_nr_hw_queues (drivers/block/null_blk/main.c:406) null_blk
[ 233.134304][ T3545] nullb_device_submit_queues_store (drivers/block/null_blk/main.c:424 (discriminator 3) drivers/block/null_blk/main.c:443 (discriminator 3)) null_blk
[ 233.141245][ T3545] configfs_write_iter (fs/configfs/file.c:207 fs/configfs/file.c:229)
[ 233.146194][ T3545] vfs_write (fs/read_write.c:586 fs/read_write.c:679)
[ 233.150274][ T3545] ksys_write (fs/read_write.c:731)
[ 233.154355][ T3545] do_syscall_64 (arch/x86/entry/common.c:52 arch/x86/entry/common.c:83)
[ 233.158708][ T3545] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:130)
[  233.164431][ T3545]
[  233.166607][ T3545] The buggy address belongs to the object at ffff88880ca5d800
[  233.166607][ T3545]  which belongs to the cache kmalloc-512 of size 512
[  233.180477][ T3545] The buggy address is located 360 bytes inside of
[  233.180477][ T3545]  freed 512-byte region [ffff88880ca5d800, ffff88880ca5da00)
[  233.194071][ T3545]
[  233.196248][ T3545] The buggy address belongs to the physical page:
[  233.202492][ T3545] page: refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x80ca5c
[  233.211156][ T3545] head: order:2 mapcount:0 entire_mapcount:0 nr_pages_mapped:0 pincount:0
[  233.219471][ T3545] flags: 0x17ffffc0000040(head|node=0|zone=2|lastcpupid=0x1fffff)
[  233.227097][ T3545] page_type: f5(slab)
[  233.230921][ T3545] raw: 0017ffffc0000040 ffff88810c842c80 ffffea001ffdb900 dead000000000002
[  233.239324][ T3545] raw: 0000000000000000 0000000000100010 00000001f5000000 0000000000000000
[  233.247742][ T3545] head: 0017ffffc0000040 ffff88810c842c80 ffffea001ffdb900 dead000000000002
[  233.256233][ T3545] head: 0000000000000000 0000000000100010 00000001f5000000 0000000000000000

The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20241217/[email protected]

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#1 lore raw
Quoted 29 lines · kernel test robot

Hello,

kernel test robot noticed "BUG:KASAN:slab-use-after-free_in__cpuhp_state_add_instance_cpuslocked" on:

commit: 22465bbac53c821319089016f268a2437de9b00a ("blk-mq: move cpuhp callback registering out of q->sysfs_lock")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

[test failed on linus/master 231825b2e1ff6ba799c5eaf396d3ab2354e37c6b]
[test failed on linux-next/master 3e42dc9229c5950e84b1ed705f94ed75ed208228]

in testcase: blktests
version: blktests-x86_64-3617edd-1_20241105
with following parameters:

disk: 1SSD
test: block-group-01

config: x86_64-rhel-9.4-func
compiler: gcc-12
test machine: 4 threads Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz (Skylake) with 32G memory

(please refer to attached dmesg/kmsg for entire log/backtrace)

Quoted 7 lines

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-lkp/[email protected]

[ 232.596698][ T3545] BUG: KASAN: slab-use-after-free in __cpuhp_state_add_instance_cpuslocked (include/linux/list.h:1026 kernel/cpu.c:2446)

Hello,

Thanks for the report!

Unfortunately I can't reproduce it in my test VM by running 'blktests block/030' with:

- two numa nodes
- enable CONFIG_BLK_DEV_NULL_BLK_FAULT_INJECTION

But just figured out that one freed hctx still may stay in cpuhp cb list, can you test the following patch?

block/blk-mq.c +2 -1
index 92e8ddf34575..f655b34efffe 100644
@@ -4421,7 +4421,8 @@ static struct blk_mq_hw_ctx *blk_mq_alloc_and_init_hctx(
 	/* reuse dead hctx first */
 	spin_lock(&q->unused_hctx_lock);
 	list_for_each_entry(tmp, &q->unused_hctx_list, hctx_list) {
-		if (tmp->numa_node == node) {
+		if (tmp->numa_node == node &&
+				hlist_unhashed(&tmp->cpuhp_online)) {
 			hctx = tmp;
 			break;
 		}

thanks,
Ming

author Tested-by
#2 lore raw

hi, Ming,

Quoted 45 lines · Ming Lei
Quoted 29 lines · kernel test robot

Hello,

kernel test robot noticed "BUG:KASAN:slab-use-after-free_in__cpuhp_state_add_instance_cpuslocked" on:

commit: 22465bbac53c821319089016f268a2437de9b00a ("blk-mq: move cpuhp callback registering out of q->sysfs_lock")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

[test failed on linus/master 231825b2e1ff6ba799c5eaf396d3ab2354e37c6b]
[test failed on linux-next/master 3e42dc9229c5950e84b1ed705f94ed75ed208228]

in testcase: blktests
version: blktests-x86_64-3617edd-1_20241105
with following parameters:

disk: 1SSD
test: block-group-01

config: x86_64-rhel-9.4-func
compiler: gcc-12
test machine: 4 threads Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz (Skylake) with 32G memory

(please refer to attached dmesg/kmsg for entire log/backtrace)

Quoted 7 lines

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-lkp/[email protected]

[ 232.596698][ T3545] BUG: KASAN: slab-use-after-free in __cpuhp_state_add_instance_cpuslocked (include/linux/list.h:1026 kernel/cpu.c:2446)

Hello,

Thanks for the report!

Unfortunately I can't reproduce it in my test VM by running 'blktests block/030' with:

Quoted 6 lines

- two numa nodes
- enable CONFIG_BLK_DEV_NULL_BLK_FAULT_INJECTION

But just figured out that one freed hctx still may stay in cpuhp cb list, can you test the following patch?

yes, your patch fixed the issue we reported, and 'blktests block/030' can pass now. thanks!

BTW, the config built with the patch is same as for 22465bbac5 when we made original report, so still with CONFIG_BLK_DEV_NULL_BLK_FAULT_INJECTION.

Quoted 21 lines
block/blk-mq.c +2 -1
index 92e8ddf34575..f655b34efffe 100644
@@ -4421,7 +4421,8 @@ static struct blk_mq_hw_ctx *blk_mq_alloc_and_init_hctx(
 	/* reuse dead hctx first */
 	spin_lock(&q->unused_hctx_lock);
 	list_for_each_entry(tmp, &q->unused_hctx_list, hctx_list) {
-		if (tmp->numa_node == node) {
+		if (tmp->numa_node == node &&
+				hlist_unhashed(&tmp->cpuhp_online)) {
 			hctx = tmp;
 			break;
 		}

thanks,
Ming