Skip to contentSkip to conversation

There is perf_event_open triggered soft lockup BUG in v6.5-rc5 kernel

Syzkaller & bisect Pengfei Xu 1 participant 1 message Aug 13, 2023 1 message not in the archive Media (V4L/DVB)

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): ZNhVUnPCs/[email protected]

Pengfei Xu

author
#1 lore raw

++ linux-media emial list.

Quoted 209 lines · Pengfei Xu

Hi Paul,

Greeting!

There is perf_event_open triggered soft lockup BUG in v6.5-rc5 kernel.

All detailed info link: https://github.com/xupengfe/syzkaller_logs/tree/main/230812_084142_BUG_soft_lockup
Syzkaller reproduced code: https://github.com/xupengfe/syzkaller_logs/blob/main/230812_084142_BUG_soft_lockup/repro.c
Syzkaller reproduced step: https://github.com/xupengfe/syzkaller_logs/blob/main/230812_084142_BUG_soft_lockup/repro.prog
Syzkaller report analysis: https://github.com/xupengfe/syzkaller_logs/blob/main/230812_084142_BUG_soft_lockup/repro.report
Kconfig(Need "make olddefconfig" to generate target kconfig): https://github.com/xupengfe/syzkaller_logs/blob/main/230812_084142_BUG_soft_lockup/kconfig_origin
Bisect info: https://github.com/xupengfe/syzkaller_logs/blob/main/230812_084142_BUG_soft_lockup/bisect_info.log
Issue dmesg: https://github.com/xupengfe/syzkaller_logs/blob/main/230812_084142_BUG_soft_lockup/52a93d39b17dc7eb98b6aa3edb93943248e03b2f_dmesg.log
v6.5-rc5 bzImage: https://github.com/xupengfe/syzkaller_logs/raw/main/230812_084142_BUG_soft_lockup/bzImage_52a93d39b17dc7eb98b6aa3edb93943248e03b2f.tar.gz

This issue could be reproduced in 253s.

Bisected and found first bad commit is:
f042b08b833d media: sun6i-mipi-csi2: Require both pads to be connected for streaming

"

[   28.083677] cgroup: Unknown subsys name 'rlimit'
[   73.568711] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [repro:1253]
[   73.573792] Kernel panic - not syncing: softlockup: hung tasks
[   73.574833] CPU: 0 PID: 1253 Comm: repro Tainted: G             L     6.5.0-rc5-52a93d39b17d #1
[   73.576363] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
[   73.578364] Call Trace:
[   73.578824]  <IRQ>
[   73.579210]  dump_stack_lvl+0xa9/0x110
[   73.579938]  dump_stack+0x19/0x20
[   73.580562]  panic+0x660/0x6d0
[   73.581165]  ? __pfx_panic+0x10/0x10
[   73.581866]  ? watchdog_timer_fn+0x4ab/0x5c0
[   73.582674]  watchdog_timer_fn+0x4c9/0x5c0
[   73.583421]  ? __pfx_watchdog_timer_fn+0x10/0x10
[   73.584258]  __hrtimer_run_queues+0x61a/0xba0
[   73.585068]  ? __pfx___hrtimer_run_queues+0x10/0x10
[   73.585970]  ? ktime_get_update_offsets_now+0x25b/0x340
[   73.586932]  ? write_comp_data+0x2f/0x90
[   73.587678]  hrtimer_interrupt+0x326/0x7b0
[   73.588453]  __sysvec_apic_timer_interrupt+0x14f/0x3f0
[   73.589411]  sysvec_apic_timer_interrupt+0x48/0xd0
[   73.590308]  asm_sysvec_apic_timer_interrupt+0x1f/0x30
[   73.591252] RIP: 0010:is_bpf_text_address+0x139/0x1c0
[   73.592184] Code: ff e8 7b 3b e7 ff 48 c7 c2 80 05 93 85 be ef 02 00 00 48 c7 c7 e0 05 93 85 c6 05 ca 5f fa 05 01 e8 dc 6e c5 ff e9 22 ff ff ff <e8> 52 3b e7 ff e8 5d b2 ce ff 31 ff 89 c3 89 c6 e8 a2 3c e7 ff 84
[   73.595436] RSP: 0018:ffff88806ca096e0 EFLAGS: 00000202
[   73.596374] RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffffffff81772d6f
[   73.597646] RDX: 0000000000000000 RSI: ffff88800ff7a500 RDI: 0000000000000003
[   73.598921] RBP: ffff88806ca096f0 R08: fffffbfff133afd9 R09: fffffbfff133afd9
[   73.600185] R10: fffffbfff133afd8 R11: ffffffff899d7ec7 R12: 0000000000000000
[   73.601454] R13: 0000000000000000 R14: ffff88800ff7a500 R15: 0000000000000000
[   73.602740]  ? is_bpf_text_address+0x7f/0x1c0
[   73.603557]  kernel_text_address+0xd5/0xe0
[   73.604313]  __kernel_text_address+0x16/0x50
[   73.605109]  unwind_get_return_address+0x6e/0xc0
[   73.605979]  ? __pfx_stack_trace_consume_entry+0x10/0x10
[   73.606951]  arch_stack_walk+0xa5/0x100
[   73.607671]  stack_trace_save+0x8b/0xc0
[   73.608385]  ? __pfx_stack_trace_save+0x10/0x10
[   73.609237]  kasan_save_stack+0x2a/0x50
[   73.609964]  ? kasan_save_stack+0x2a/0x50
[   73.610707]  ? kasan_set_track+0x29/0x40
[   73.611458]  ? kasan_save_alloc_info+0x22/0x30
[   73.612274]  ? __kasan_slab_alloc+0x64/0x80
[   73.613046]  ? slab_post_alloc_hook+0x84/0x3b0
[   73.613872]  ? kmem_cache_alloc_node+0x18a/0x2e0
[   73.614725]  ? __alloc_skb+0x2b6/0x360
[   73.615435]  ? __neigh_notify+0x91/0x170
[   73.616171]  ? neigh_timer_handler+0x2b8/0xfa0
[   73.616981]  ? call_timer_fn+0x1a5/0x570
[   73.617712]  ? run_timer_softirq+0x714/0x14d0
[   73.618511]  ? __do_softirq+0x1b7/0x77c
[   73.619222]  ? irq_exit_rcu+0xc1/0x110
[   73.619928]  ? sysvec_apic_timer_interrupt+0xa1/0xd0
[   73.620828]  ? asm_sysvec_apic_timer_interrupt+0x1f/0x30
[   73.621797]  ? generic_exec_single+0x278/0x540
[   73.622619]  ? smp_call_function_single+0x193/0x460
[   73.623506]  ? task_function_call+0x10e/0x1a0
[   73.624318]  ? perf_install_in_context+0x2f9/0x5b0
[   73.625191]  ? __do_sys_perf_event_open+0x1a9a/0x2a50
[   73.626130]  ? __x64_sys_perf_event_open+0xc7/0x160
[   73.627024]  ? do_syscall_64+0x3b/0x90
[   73.627721]  ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[   73.628676]  ? write_comp_data+0x2f/0x90
[   73.629422]  ? debug_smp_processor_id+0x20/0x30
[   73.630266]  ? __sanitizer_cov_trace_pc+0x25/0x60
[   73.631131]  ? write_comp_data+0x2f/0x90
[   73.631871]  ? mark_held_locks+0xb7/0x140
[   73.632608]  ? irqentry_exit+0x3e/0xa0
[   73.633311]  ? sysvec_irq_work+0x58/0xd0
[   73.634065]  ? __this_cpu_preempt_check+0x20/0x30
[   73.634928]  ? lockdep_hardirqs_on+0x8a/0x110
[   73.635731]  ? irqentry_exit+0x3e/0xa0
[   73.636433]  ? trace_hardirqs_on+0x26/0x120
[   73.637208]  kasan_set_track+0x29/0x40
[   73.637939]  kasan_save_alloc_info+0x22/0x30
[   73.638726]  __kasan_slab_alloc+0x64/0x80
[   73.639487]  slab_post_alloc_hook+0x84/0x3b0
[   73.640283]  kmem_cache_alloc_node+0x18a/0x2e0
[   73.641101]  ? __alloc_skb+0x2b6/0x360
[   73.641817]  __alloc_skb+0x2b6/0x360
[   73.642494]  ? __pfx___alloc_skb+0x10/0x10
[   73.643255]  ? asm_sysvec_irq_work+0x1f/0x30
[   73.644063]  __neigh_notify+0x91/0x170
[   73.644768]  neigh_timer_handler+0x2b8/0xfa0
[   73.645572]  ? __pfx_neigh_timer_handler+0x10/0x10
[   73.646448]  call_timer_fn+0x1a5/0x570
[   73.647137]  ? asm_sysvec_irq_work+0x1f/0x30
[   73.647929]  ? __pfx_call_timer_fn+0x10/0x10
[   73.648725]  ? __pfx_neigh_timer_handler+0x10/0x10
[   73.649611]  run_timer_softirq+0x714/0x14d0
[   73.650401]  ? __pfx_run_timer_softirq+0x10/0x10
[   73.651240]  ? asm_sysvec_irq_work+0x1f/0x30
[   73.652040]  ? __pfx_run_timer_softirq+0x10/0x10
[   73.652890]  __do_softirq+0x1b7/0x77c
[   73.653593]  irq_exit_rcu+0xc1/0x110
[   73.654261]  sysvec_apic_timer_interrupt+0xa1/0xd0
[   73.655132]  </IRQ>
[   73.655532]  <TASK>
[   73.655943]  asm_sysvec_apic_timer_interrupt+0x1f/0x30
[   73.656889] RIP: 0010:generic_exec_single+0x278/0x540
[   73.657832] Code: e8 0d 85 0b 00 4c 89 e6 31 ff 31 db e8 11 87 0b 00 4d 85 e4 0f 84 3e fe ff ff e8 f3 84 0b 00 e8 8e f0 17 00 fb 0f 1f 44 00 00 <e9> 29 fe ff ff e8 de 84 0b 00 65 8b 1d a3 6a b1 7e 89 db be 08 00
[   73.661070] RSP: 0018:ffff88800fa97ac8 EFLAGS: 00000202
[   73.662013] RAX: 0000000000000485 RBX: 0000000000000000 RCX: 1ffffffff1343c28
[   73.663275] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000
[   73.664538] RBP: ffff88800fa97af0 R08: 0000000000000001 R09: fffffbfff133afee
[   73.665808] R10: 0000000000000001 R11: ffffffff899d7f6f R12: 0000000000000200
[   73.667071] R13: ffffffff818e9ad0 R14: ffff88800fa97c20 R15: ffff88800fa97b48
[   73.668333]  ? __pfx_remote_function+0x10/0x10
[   73.669163]  ? generic_exec_single+0x272/0x540
[   73.670005]  smp_call_function_single+0x193/0x460
[   73.670866]  ? write_comp_data+0x2f/0x90
[   73.671590]  ? __pfx_remote_function+0x10/0x10
[   73.672406]  ? __pfx_smp_call_function_single+0x10/0x10
[   73.673371]  ? __pfx_remote_function+0x10/0x10
[   73.674209]  ? __sanitizer_cov_trace_pc+0x25/0x60
[   73.675081]  task_function_call+0x10e/0x1a0
[   73.675855]  ? __pfx_task_function_call+0x10/0x10
[   73.676720]  ? __pfx___perf_install_in_context+0x10/0x10
[   73.677693]  ? exclusive_event_installable+0x25b/0x320
[   73.678626]  ? write_comp_data+0x2f/0x90
[   73.679358]  perf_install_in_context+0x2f9/0x5b0
[   73.680206]  ? __pfx_perf_install_in_context+0x10/0x10
[   73.681145]  ? write_comp_data+0x2f/0x90
[   73.681897]  ? __sanitizer_cov_trace_pc+0x25/0x60
[   73.682767]  __do_sys_perf_event_open+0x1a9a/0x2a50
[   73.683672]  ? __pfx___do_sys_perf_event_open+0x10/0x10
[   73.684649]  __x64_sys_perf_event_open+0xc7/0x160
[   73.685526]  ? syscall_enter_from_user_mode+0x51/0x60
[   73.686449]  do_syscall_64+0x3b/0x90
[   73.687114]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[   73.688035] RIP: 0033:0x7f904803ee5d
[   73.688694] Code: ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 93 af 1b 00 f7 d8 64 89 01 48
[   73.691934] RSP: 002b:00007ffcb1c151a8 EFLAGS: 00000206 ORIG_RAX: 000000000000012a
[   73.693269] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f904803ee5d
[   73.694545] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000020000280
[   73.695804] RBP: 00007ffcb1c151b0 R08: 0000000000000000 R09: 00007ffcb1c15200
[   73.697064] R10: 00000000ffffffff R11: 0000000000000206 R12: 00007ffcb1c15348
[   73.698337] R13: 0000000000407f0f R14: 000000000040adf8 R15: 00007f9048381000
[   73.699619]  </TASK>
[   73.700900] Kernel Offset: disabled
[   73.701602] ---[ end Kernel panic - not syncing: softlockup: hung tasks ]---

"

---

If you don't need the following environment to reproduce the problem or if you already have one, please ignore the following information.

How to reproduce:
git clone https://gitlab.com/xupengfe/repro_vm_env.git
cd repro_vm_env
tar -xvf repro_vm_env.tar.gz
cd repro_vm_env; ./start3.sh // it needs qemu-system-x86_64 and I used v7.1.0
// start3.sh will load bzImage_2241ab53cbb5cdb08a6b2d4688feb13971058f65 v6.2-rc5 kernel
// You could change the bzImage_xxx as you want
// Maybe you need to remove line "-drive if=pflash,format=raw,readonly=on,file=./OVMF_CODE.fd \" for different qemu version
You could use below command to log in, there is no password for root.
ssh -p 10023 root@localhost

After login vm(virtual machine) successfully, you could transfer reproduced binary to the vm by below way, and reproduce the problem in vm:
gcc -pthread -o repro repro.c
scp -P 10023 repro root@localhost:/root/

Get the bzImage for target kernel:
Please use target kconfig and copy it to kernel_src/.config make olddefconfig
make -jx bzImage //x should equal or less than cpu num your pc has

Fill the bzImage file name into above start3.sh to load the target kernel in vm.

Tips:
If you already have qemu-system-x86_64, please ignore below info.
If you want to install qemu v7.1.0 version:
git clone https://github.com/qemu/qemu.git
cd qemu
git checkout -f v7.1.0
mkdir build
cd build
yum install -y ninja-build.x86_64
yum -y install libslirp-devel.x86_64
../configure --target-list=x86_64-softmmu --enable-kvm --enable-vnc --enable-gtk --enable-sdl --enable-usb-redir --enable-slirp make
make install

Best Regards,
Thanks!