Skip to contentSkip to conversation

New: kernel goes unresponsive if single-stepping over an instruction which writes to an address for which a hardware read/write watchpoint has been set

Bug 218339 [email protected] 2 participants 8 messages Jan 4, 2024 – Mar 5, 2024 KVM

lore
Share & tools
lore
#0 lore raw

https://bugzilla.kernel.org/show_bug.cgi?id=218339

Bug ID: 218339
Summary: kernel goes unresponsive if single-stepping over an
instruction which writes to an address for which a hardware read/write watchpoint has been set
Product: Virtualization
Version: unspecified
Hardware: All
OS: Linux
Status: NEW
Severity: normal
Priority: P3
Component: kvm
Assignee: [email protected]
Reporter: [email protected]
Regression: No

In a debian QEMU/KVM virtual machine, run gdb on any executable (e.g. /usr/bin/ls). Run the program by typing starti. Proceed to _dl_start (i.e. break _dl_start, continue). When you get there disassemble the function (i.e. disas). Find an instruction that's going to be executed for which you can compute the address in memory it will write to. Run the program to that instruction (i.e. break *0xINSN, continue). When you're on that instruction, set a read/write watchpoint on the address it will write to, then single-step (i.e. stepi) and the kernel will go unresponsive.

Quoted 7 lines

(gdb) x/1i $pc
=> 0x7ffff7fe6510 <_dl_start+48>: mov %rdi,-0x88(%rbp)
(gdb) x/1wx $rbp-0x88
0x7fffffffec28: 0x00000000
(gdb) awatch *0x7fffffffec28
Hardware access (read/write) watchpoint 2: *0x7fffffffec28 (gdb) stepi

Looking with journalctl, I cannot find anything printed to dmesg.

The kernel of the guest inside the virtual machine is Debian 6.1.0-15-amd64. The kernel of the host running qemu-system-x86_64 is Archlinux 6.6.7-arch1-1. gdb is version 13.1.

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
#1 lore raw
Quoted 18 lines · On Thu, Jan 04, 2024, [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=218339

Bug ID: 218339
Summary: kernel goes unresponsive if single-stepping over an
instruction which writes to an address for which a hardware read/write watchpoint has been set
Product: Virtualization
Version: unspecified
Hardware: All
OS: Linux
Status: NEW
Severity: normal
Priority: P3
Component: kvm
Assignee: [email protected]
Reporter: [email protected]
Regression: No

Quoted 8 lines

In a debian QEMU/KVM virtual machine, run gdb on any executable (e.g. /usr/bin/ls). Run the program by typing starti. Proceed to _dl_start (i.e. break _dl_start, continue). When you get there disassemble the function (i.e. disas). Find an instruction that's going to be executed for which you can compute the address in memory it will write to. Run the program to that instruction (i.e. break *0xINSN, continue). When you're on that instruction, set a read/write watchpoint on the address it will write to, then single-step (i.e. stepi) and the kernel will go unresponsive.

By "the kernel", I assume you mean the guest kernel?

Quoted 9 lines
Quoted 7 lines

(gdb) x/1i $pc
=> 0x7ffff7fe6510 <_dl_start+48>: mov %rdi,-0x88(%rbp)
(gdb) x/1wx $rbp-0x88
0x7fffffffec28: 0x00000000
(gdb) awatch *0x7fffffffec28
Hardware access (read/write) watchpoint 2: *0x7fffffffec28 (gdb) stepi

Quoted 5 lines

Looking with journalctl, I cannot find anything printed to dmesg.

The kernel of the guest inside the virtual machine is Debian 6.1.0-15-amd64. The kernel of the host running qemu-system-x86_64 is Archlinux 6.6.7-arch1-1. gdb is version 13.1.

Is this a regression or something that has always been broken? I.e. did this work on previous host kernels?

#2 lore raw

[Bug 218339] kernel goes unresponsive if single-stepping over an instruction which writes to an address for which a hardware read/write watchpoint has been set

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=218339

--- Comment #1 from Sean Christopherson ([email protected]) ---

Quoted 18 lines · On Thu, Jan 04, 2024, [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=218339

Bug ID: 218339
Summary: kernel goes unresponsive if single-stepping over an
instruction which writes to an address for which a hardware read/write watchpoint has been set
Product: Virtualization
Version: unspecified
Hardware: All
OS: Linux
Status: NEW
Severity: normal
Priority: P3
Component: kvm
Assignee: [email protected]
Reporter: [email protected]
Regression: No

Quoted 9 lines

In a debian QEMU/KVM virtual machine, run gdb on any executable (e.g. /usr/bin/ls). Run the program by typing starti. Proceed to _dl_start (i.e. break _dl_start, continue). When you get there disassemble the function (i.e. disas). Find an instruction that's going to be executed for which you can compute the address in memory it will write to. Run the program to that instruction (i.e. break *0xINSN, continue). When you're on that instruction, set a read/write watchpoint on the address it will write to, then
single-step (i.e. stepi) and the kernel will go unresponsive.

By "the kernel", I assume you mean the guest kernel?

Quoted 9 lines
Quoted 7 lines

(gdb) x/1i $pc
=> 0x7ffff7fe6510 <_dl_start+48>: mov %rdi,-0x88(%rbp)
(gdb) x/1wx $rbp-0x88
0x7fffffffec28: 0x00000000
(gdb) awatch *0x7fffffffec28
Hardware access (read/write) watchpoint 2: *0x7fffffffec28 (gdb) stepi

Quoted 5 lines

Looking with journalctl, I cannot find anything printed to dmesg.

The kernel of the guest inside the virtual machine is Debian 6.1.0-15-amd64. The kernel of the host running qemu-system-x86_64 is Archlinux 6.6.7-arch1-1. gdb is version 13.1.

Is this a regression or something that has always been broken? I.e. did this work
on previous host kernels?

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
#3 lore raw

[Bug 218339] kernel goes unresponsive if single-stepping over an instruction which writes to an address for which a hardware read/write watchpoint has been set

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=218339

--- Comment #2 from Anthony L. Eden ([email protected]) ---

Quoted 1 line

By "the kernel", I assume you mean the guest kernel?

Yes, the guest kernel. I can no longer interact with the VM via the serial console. It is unresponsive.

I attached a debugger to qemu-system-x86_64 to see if qemu itself was in an infinite loop or something but the stacktraces all looked normal.

Quoted 2 lines

Is this a regression or something that has always been broken? I.e. did this work on previous host kernels?

I do not know whether this has always been broken or not.

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
#4 lore raw

[Bug 218339] kernel goes unresponsive if single-stepping over an instruction which writes to an address for which a hardware read/write watchpoint has been set

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=218339

Yao Yuan ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #3 from Yao Yuan ([email protected]) --- Hi,

I tried on my side but can't reproducce it, logs below. Any steps I missed ?

(gdb) b *0x00007ffff7fe4048
Breakpoint 1 at 0x7ffff7fe4048: file ./elf/rtld.c, line 527. (gdb) c
Continuing.

Breakpoint 1, 0x00007ffff7fe4048 in _dl_start (arg=0x7fffffffe510) at ./elf/rtld.c:527
527 in ./elf/rtld.c
(gdb) disassemble
Dump of assembler code for function _dl_start:

   0x00007ffff7fe4030 <+0>:     endbr64                                         
   0x00007ffff7fe4034 <+4>:     push   %rbp                                     
   0x00007ffff7fe4035 <+5>:     mov    %rsp,%rbp                                
   0x00007ffff7fe4038 <+8>:     push   %r15                                     
   0x00007ffff7fe403a <+10>:    push   %r14                                     
   0x00007ffff7fe403c <+12>:    push   %r13                                     
   0x00007ffff7fe403e <+14>:    push   %r12                                     
   0x00007ffff7fe4040 <+16>:    push   %rbx                                     
   0x00007ffff7fe4041 <+17>:    sub    $0x88,%rsp                               
=> 0x00007ffff7fe4048 <+24>:    mov    %rdi,-0x78(%rbp)                         
   0x00007ffff7fe404c <+28>:    rdtsc                                           
(gdb) x/16xb $rbp-0x78                                                          
0x7fffffffe488: 0x00    0x00    0x00    0x00    0x00    0x00    0x00    0x00    
0x7fffffffe490: 0x00    0x00    0x00    0x00    0x00    0x00    0x00    0x00    

(gdb) awatch *0x7fffffffe488
Hardware access (read/write) watchpoint 2: *0x7fffffffe488 (gdb) stepi

Hardware access (read/write) watchpoint 2: *0x7fffffffe488

Old value = 0
New value = -6896
0x00007ffff7fe404c in rtld_timer_start (var=0x7ffff7ffcaa0 <start_time>) at ./elf/rtld.c:85
85 in ./elf/rtld.c

the guest kernel runs properly after above steps inside guest.

My configure:
Host: stable kernel v6.6.8 commit 4c9646a796d66a2d81871a694e88e19a38b115a7
QEMU: v8.1.1 commit 6bb4a8a47a43f35a345f107227fcd6abed59e62c Guest kernel: kvm tree tags/kvm-6.8-1 commit 1c6d984f523f67ecfad1083bb04c55d91977bb15

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
#5 lore raw

[Bug 218339] kernel goes unresponsive if single-stepping over an instruction which writes to an address for which a hardware read/write watchpoint has been set

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=218339

--- Comment #4 from Anthony L. Eden ([email protected]) ---

Quoted 1 line
Quoted 1 line

I tried on my side but can't reproducce it, logs below. Any steps I missed ?

Nope, it looks like you did everything right.

I spent a little more time investigating this, since for me it's trivial to reproduce. I was able to get the guest kernel vmlinux *with* debugging information from the linux-image-6.1.0-15-amd64-dbg debian package.

After entering the final stepi within gdb, which is when the guest goes totally unresponsive, in htop I see qemu-system-x86_64 is taking up 100% CPU. Like I said, the thread call stacks in the qemu process look typical.

I used the qemu monitor command 'dump-guest-memory -p /root/linux.core' three separate times after the guest went unresponsive, and all three of the core file backtraces look like this:

#0 pv_native_set_debugreg (regno=7, val=0) at arch/x86/include/asm/debugreg.h:92
#1 0xffffffff81a21533 in set_debugreg (reg=7, val=0) at arch/x86/include/asm/paravirt.h:129
#2 local_db_save () at arch/x86/include/asm/debugreg.h:127 #3 exc_debug_kernel (dr6=0, regs=0xfffffe0000010f58) at arch/x86/kernel/traps.c:1038
#4 exc_debug (regs=0xfffffe0000010f58) at arch/x86/kernel/traps.c:1175 #5 0xffffffff81c00c6a in asm_exc_debug () at /build/reproducible-path/linux-6.1.66/arch/x86/include/asm/idtentry.h:606 #6 0x0000000000000000 in ?? ()

My VM was in a self-contained folder with its own run script on the host so I made a tarball of it. It is available for download here (~9 GB):

https://drive.google.com/file/d/1r3tlrw8kG17vFwXzP6ETv76ptNhbLYjt/view?usp=sharing

Usage:

$ tar xvSf deb-vm-x86_64.tar
$ cd deb-vm-x86_64/
$ ./run.sh

In another terminal,

$ screen /dev/pts/23 115200
$ login as user 'root' with password 'root'

Once inside,

$ gdb /usr/bin/ls
$ starti
...

Oh and by the way, the version of qemu-system-x86_64 on my host is 7.2.7 (Debian 1:7.2+dfsg-7+deb12u3).

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
#6 lore raw

[Bug 218339] kernel goes unresponsive if single-stepping over an instruction which writes to an address for which a hardware read/write watchpoint has been set

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=218339

--- Comment #5 from Anthony L. Eden ([email protected]) --- Actually upon closer inspection I'm seeing two distinct call stacks appear in the core files.

#0 pv_native_set_debugreg (regno=7, val=0) at arch/x86/include/asm/debugreg.h:92
#1 0xffffffff81a21533 in set_debugreg (reg=7, val=0) at arch/x86/include/asm/paravirt.h:129
#2 local_db_save () at arch/x86/include/asm/debugreg.h:127 #3 exc_debug_kernel (dr6=0, regs=0xfffffe0000010f58) at arch/x86/kernel/traps.c:1038
#4 exc_debug (regs=0xfffffe0000010f58) at arch/x86/kernel/traps.c:1175 #5 0xffffffff81c00c6a in asm_exc_debug () at /build/reproducible-path/linux-6.1.66/arch/x86/include/asm/idtentry.h:606 #6 0x0000000000000000 in ?? ()

#0 pv_native_set_debugreg (regno=7, val=983554) at arch/x86/include/asm/debugreg.h:92
#1 0xffffffff81a21509 in set_debugreg (reg=7, val=983554) at arch/x86/include/asm/paravirt.h:129
#2 local_db_restore (dr7=983554) at arch/x86/include/asm/debugreg.h:147 #3 exc_debug_kernel (dr6=<optimized out>, regs=0xfffffe0000010f58) at arch/x86/kernel/traps.c:1095
#4 exc_debug (regs=0xfffffe0000010f58) at arch/x86/kernel/traps.c:1175 #5 0xffffffff81c00c6a in asm_exc_debug () at /build/reproducible-path/linux-6.1.66/arch/x86/include/asm/idtentry.h:606 #6 0x0000000000000000 in ?? ()

They are quite similar except in one of them frame #2 is local_db_save() and in the other trace frame #2 is local_db_restore().

By the way this time I ran the VM under a different, newer version of qemu-system-x86_64 (8.2.0), and it appears to have made no difference.

Also, concerning the VM in that tarball I linked to, if the run.sh is run as it is you will be able to ssh into the running guest with 'ssh -p 10024 root@localhost', furthermore the path to the kernel image *with* debug info is located at /usr/lib/debug/vmlinux-6.1.0-15-amd64.

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
#7 lore raw

[Bug 218339] kernel goes unresponsive if single-stepping over an instruction which writes to an address for which a hardware read/write watchpoint has been set

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=218339

Kishen Maloor ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #6 from Kishen Maloor ([email protected]) --- (In reply to Anthony L. Eden from comment #0)

Quoted 19 lines

In a debian QEMU/KVM virtual machine, run gdb on any executable (e.g. /usr/bin/ls). Run the program by typing starti. Proceed to _dl_start (i.e. break _dl_start, continue). When you get there disassemble the function (i.e. disas). Find an instruction that's going to be executed for which you can compute the address in memory it will write to. Run the program to that instruction (i.e. break *0xINSN, continue). When you're on that instruction, set a read/write watchpoint on the address it will write to, then single-step (i.e. stepi) and the kernel will go unresponsive.

Quoted 7 lines

(gdb) x/1i $pc
=> 0x7ffff7fe6510 <_dl_start+48>: mov %rdi,-0x88(%rbp)
(gdb) x/1wx $rbp-0x88
0x7fffffffec28: 0x00000000
(gdb) awatch *0x7fffffffec28
Hardware access (read/write) watchpoint 2: *0x7fffffffec28 (gdb) stepi

I can reproduce the behavior you describe. But it seems that you're not invoking KVM at all, because when I add '-accel kvm' or '-enable-kvm' to your qemu command line the problem goes away.

There may be an issue specifically in the handling of hardware watchpoints on the qemu emulation. If I disable hardware watchpoints in gdb using 'set can-use-hw-watchpoints 0' and then use 'watch *<ADDR>', that works.

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.