Skip to contentSkip to conversation

KVM: X86: Also reload the debug registers before kvm_x86->run() when the host is using them

PATCH Applied Lai Jiangshan 5 participants 10 messages Jun 29, 2021 – Jul 9, 2021

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 KVM: X86: Also reload the debug registers before kvm_x86->run() when the host is using them

From: Lai Jiangshan <[email protected]>

When the host is using debug registers but the guest is not using them nor is the guest in guest-debug state, the kvm code does not reset the host debug registers before kvm_x86->run(). Rather, it relies on the hardware vmentry instruction to automatically reset the dr7 registers which ensures that the host breakpoints do not affect the guest.

But there are still problems:
o The addresses of the host breakpoints can leak into the guest and the guest may use these information to attack the host.

o It violates the non-instrumentable nature around VM entry and exit. For example, when a host breakpoint is set on vcpu->arch.cr2, #DB will hit aftr kvm_guest_enter_irqoff().

Beside the problems, the logic is not consistent either. When the guest debug registers are active, the host breakpoints are reset before kvm_x86->run(). But when the guest debug registers are inactive, the host breakpoints are delayed to be disabled. The host tracing tools may see different results depending on there is any guest running or not.

To fix the problems, we also reload the debug registers before kvm_x86->run() when the host is using them whenever the guest is using them or not.

---
 arch/x86/kvm/x86.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kvm/x86.c +1 -1
index b594275d49b5..cce316655d3c 100644
@@ -9320,7 +9320,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 	if (test_thread_flag(TIF_NEED_FPU_LOAD))
 		switch_fpu_return();
 
-	if (unlikely(vcpu->arch.switch_db_regs)) {
+	if (unlikely(vcpu->arch.switch_db_regs || hw_breakpoint_active())) {
 		set_debugreg(0, 7);
 		set_debugreg(vcpu->arch.eff_db[0], 0);
 		set_debugreg(vcpu->arch.eff_db[1], 1);
Signature · 1 line
2.19.1.6.gb485710b
maintainer Applied
#1 lore raw
Quoted 11 lines · Lai Jiangshan

From: Lai Jiangshan <[email protected]>

When the host is using debug registers but the guest is not using them nor is the guest in guest-debug state, the kvm code does not reset the host debug registers before kvm_x86->run(). Rather, it relies on the hardware vmentry instruction to automatically reset the dr7 registers which ensures that the host breakpoints do not affect the guest.

But there are still problems:
o The addresses of the host breakpoints can leak into the guest and the guest may use these information to attack the host.

I don't think this is true, because DRn reads would exit (if they don't, switch_db_regs would be nonzero). But otherwise it makes sense to do at least the DR7 write, and we might as well do all of them.

Quoted 9 lines

o It violates the non-instrumentable nature around VM entry and exit. For example, when a host breakpoint is set on vcpu->arch.cr2, #DB will hit aftr kvm_guest_enter_irqoff().

Beside the problems, the logic is not consistent either. When the guest debug registers are active, the host breakpoints are reset before kvm_x86->run(). But when the guest debug registers are inactive, the host breakpoints are delayed to be disabled. The host tracing tools may see different results depending on there is any guest running or not.

More precisely, the host tracing tools may see different results depending on what the guest is doing.

Queued (with fixed commit message), thanks!

Paolo

Quoted 23 lines

To fix the problems, we also reload the debug registers before kvm_x86->run() when the host is using them whenever the guest is using them or not.

---
  arch/x86/kvm/x86.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kvm/x86.c +1 -1
index b594275d49b5..cce316655d3c 100644
@@ -9320,7 +9320,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
  	if (test_thread_flag(TIF_NEED_FPU_LOAD))
  		switch_fpu_return();
 
-	if (unlikely(vcpu->arch.switch_db_regs)) {
+	if (unlikely(vcpu->arch.switch_db_regs || hw_breakpoint_active())) {
  		set_debugreg(0, 7);
  		set_debugreg(vcpu->arch.eff_db[0], 0);
  		set_debugreg(vcpu->arch.eff_db[1], 1);
#2 lore raw
Quoted 9 lines · Paolo Bonzini
Quoted 8 lines · Lai Jiangshan

From: Lai Jiangshan <[email protected]>

When the host is using debug registers but the guest is not using them nor is the guest in guest-debug state, the kvm code does not reset the host debug registers before kvm_x86->run().  Rather, it relies on the hardware vmentry instruction to automatically reset the dr7 registers which ensures that the host breakpoints do not affect the guest.

Quoted 5 lines
Quoted 3 lines

But there are still problems:
    o The addresses of the host breakpoints can leak into the guest and the guest may use these information to attack the host.

I don't think this is true, because DRn reads would exit (if they don't, switch_db_regs would be nonzero).  But otherwise it makes sense to do at least the DR7 write, and we might as well do all of them.

Ahh.... you are right.

Quoted 11 lines
Quoted 10 lines

    o It violates the non-instrumentable nature around VM entry and exit.  For example, when a host breakpoint is set on vcpu->arch.cr2, #DB will hit aftr kvm_guest_enter_irqoff().

Beside the problems, the logic is not consistent either. When the guest debug registers are active, the host breakpoints are reset before kvm_x86->run(). But when the guest debug registers are inactive, the host breakpoints are delayed to be disabled.  The host tracing tools may see different results depending on there is any guest running or not.

Quoted 5 lines

More precisely, the host tracing tools may see different results depending on what the guest is doing.

Queued (with fixed commit message), thanks!

Paolo

I just noticed that emulation.c fails to emulate with DBn. Is there any problem around it?

For code breakpoint, if the instruction didn't cause vm-exit, (for example, the 2nd instruction when kvm emulates instructions back to back) emulation.c fails to emulate with DBn.

For code breakpoint, if the instruction just caused vm-exit. It is difficult to analyze this case due to the complex priorities between vectored events and fault-like vm-exit.
Anyway, if it is an instruction that vm-exit has priority over #DB, emulation.c fails to emulate with DBn.

For data breakpoint, a #DB must be delivered to guest or to VMM (when guest-debug) after the instruction. But emulation.c doesn't do so.

And the existence of both of effective DBn (guest debug) and guest DBn complicates the problem when we try to emulate them.

Thanks.
Lai.

maintainer
#3 lore raw
Quoted 2 lines · Lai Jiangshan

I just noticed that emulation.c fails to emulate with DBn. Is there any problem around it?

Just what you said, it's not easy and the needs are limited. I implemented kvm_vcpu_check_breakpoint because I was interested in using hardware breakpoints from gdb, even with unrestricted_guest=0 and invalid guest state, but that's it.

Paolo

Quoted 15 lines

For code breakpoint, if the instruction didn't cause vm-exit, (for example, the 2nd instruction when kvm emulates instructions back to back) emulation.c fails to emulate with DBn.

For code breakpoint, if the instruction just caused vm-exit. It is difficult to analyze this case due to the complex priorities between vectored events and fault-like vm-exit.
Anyway, if it is an instruction that vm-exit has priority over #DB, emulation.c fails to emulate with DBn.

For data breakpoint, a #DB must be delivered to guest or to VMM (when guest-debug) after the instruction. But emulation.c doesn't do so.

And the existence of both of effective DBn (guest debug) and guest DBn complicates the problem when we try to emulate them.

#4 lore raw
Quoted 6 lines · Paolo Bonzini
Quoted 2 lines · Lai Jiangshan

I just noticed that emulation.c fails to emulate with DBn. Is there any problem around it?

Just what you said, it's not easy and the needs are limited.  I implemented kvm_vcpu_check_breakpoint because I was interested in using hardware breakpoints from gdb, even with unrestricted_guest=0 and invalid guest state, but that's it.

Hello Paolo

I just remembered I once came across the patch, but I forgot it when I wrote the mail.

It seems kvm_vcpu_check_breakpoint() handles only for code breakpoint and doesn't handle for data breakpoints.

And no code handles DR7_GD bit when the emulation is not resulted from vm-exit. (for example, the non-first instruction when kvm emulates instructions back to back and the instruction accesses to DBn).

Thanks
Lai

Quoted 17 lines

Paolo

Quoted 15 lines

For code breakpoint, if the instruction didn't cause vm-exit, (for example, the 2nd instruction when kvm emulates instructions back to back) emulation.c fails to emulate with DBn.

For code breakpoint, if the instruction just caused vm-exit. It is difficult to analyze this case due to the complex priorities between vectored events and fault-like vm-exit.
Anyway, if it is an instruction that vm-exit has priority over #DB, emulation.c fails to emulate with DBn.

For data breakpoint, a #DB must be delivered to guest or to VMM (when guest-debug) after the instruction. But emulation.c doesn't do so.

And the existence of both of effective DBn (guest debug) and guest DBn complicates the problem when we try to emulate them.

maintainer
#5 lore raw
Quoted 11 lines · Lai Jiangshan
Quoted 5 lines · Paolo Bonzini
Quoted 2 lines · Lai Jiangshan

I just noticed that emulation.c fails to emulate with DBn. Is there any problem around it?

Just what you said, it's not easy and the needs are limited.  I implemented kvm_vcpu_check_breakpoint because I was interested in using hardware breakpoints from gdb, even with unrestricted_guest=0 and invalid guest state, but that's it.

It seems kvm_vcpu_check_breakpoint() handles only for code breakpoint and doesn't handle for data breakpoints.

Correct, there's a comment above the call. But data breakpoint are much harder and relatively less useful.

Quoted 3 lines

And no code handles DR7_GD bit when the emulation is not resulted from vm-exit. (for example, the non-first instruction when kvm emulates instructions back to back and the instruction accesses to DBn).

Good point, that should be fixed too.

Paolo

#6 lore raw
Quoted 14 lines · Paolo Bonzini
Quoted 12 lines · Lai Jiangshan
Quoted 8 lines · Paolo Bonzini
Quoted 2 lines · Lai Jiangshan

I just noticed that emulation.c fails to emulate with DBn. Is there any problem around it?

Just what you said, it's not easy and the needs are limited. I implemented kvm_vcpu_check_breakpoint because I was interested in using hardware breakpoints from gdb, even with unrestricted_guest=0 and invalid guest state, but that's it.

Quoted 5 lines
Quoted 2 lines

It seems kvm_vcpu_check_breakpoint() handles only for code breakpoint and doesn't handle for data breakpoints.

Correct, there's a comment above the call. But data breakpoint are much harder and relatively less useful.

Data breakpoints are actually quite useful. I/O breakpoints not so much.

Quoted 8 lines
Quoted 3 lines

And no code handles DR7_GD bit when the emulation is not resulted from vm-exit. (for example, the non-first instruction when kvm emulates instructions back to back and the instruction accesses to DBn).

Good point, that should be fixed too.

Paolo

maintainer
#7 lore raw
Quoted 10 lines · Jim Mattson
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

Just what you said, it's not easy and the needs are limited. I implemented kvm_vcpu_check_breakpoint because I was interested in using hardware breakpoints from gdb, even with unrestricted_guest=0 and invalid guest state, but that's it.

It seems kvm_vcpu_check_breakpoint() handles only for code breakpoint and doesn't handle for data breakpoints.

Correct, there's a comment above the call. But data breakpoint are much harder and relatively less useful.

Data breakpoints are actually quite useful. I/O breakpoints not so much.

Normally yes; much less for the specific case of debugging invalid-guest-state or other invocations of the emulator.

Paolo

#8 lore raw
Quoted 8 lines · Paolo Bonzini
Quoted 6 lines · Jim Mattson
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

Just what you said, it's not easy and the needs are limited. I implemented kvm_vcpu_check_breakpoint because I was interested in using hardware breakpoints from gdb, even with unrestricted_guest=0 and invalid guest state, but that's it.

It seems kvm_vcpu_check_breakpoint() handles only for code breakpoint and doesn't handle for data breakpoints.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Correct, there's a comment above the call. But data breakpoint are much harder and relatively less useful.

Data breakpoints are actually quite useful. I/O breakpoints not so much.

Normally yes; much less for the specific case of debugging invalid-guest-state or other invocations of the emulator.

Agreed. But if they don't actually work in-guest (because the emulator ignores them), then their normal usefulness is curtailed.

#9 lore raw
In reply to Paolo Bonzini
Quoted 43 lines · Paolo Bonzini
Quoted 11 lines · Lai Jiangshan

From: Lai Jiangshan <[email protected]>

When the host is using debug registers but the guest is not using them nor is the guest in guest-debug state, the kvm code does not reset the host debug registers before kvm_x86->run(). Rather, it relies on the hardware vmentry instruction to automatically reset the dr7 registers which ensures that the host breakpoints do not affect the guest.

But there are still problems:
o The addresses of the host breakpoints can leak into the guest and the guest may use these information to attack the host.

I don't think this is true, because DRn reads would exit (if they don't, switch_db_regs would be nonzero). But otherwise it makes sense to do at least the DR7 write, and we might as well do all of them.

Quoted 9 lines

o It violates the non-instrumentable nature around VM entry and exit. For example, when a host breakpoint is set on vcpu->arch.cr2, #DB will hit aftr kvm_guest_enter_irqoff().

Beside the problems, the logic is not consistent either. When the guest debug registers are active, the host breakpoints are reset before kvm_x86->run(). But when the guest debug registers are inactive, the host breakpoints are delayed to be disabled. The host tracing tools may see different results depending on there is any guest running or not.

More precisely, the host tracing tools may see different results depending on what the guest is doing.

Queued (with fixed commit message), thanks!

Paolo

Quoted 9 lines

To fix the problems, we also reload the debug registers before kvm_x86->run() when the host is using them whenever the guest is using them or not.

---
  arch/x86/kvm/x86.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 10 lines

arch/x86/kvm/x86.c static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · line 9323

Quoted 10 lines
arch/x86/kvm/x86.c +1 -1
index b594275d49b5..cce316655d3c 100644
@@ -9320,7 +9320,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
  	if (test_thread_flag(TIF_NEED_FPU_LOAD))
  		switch_fpu_return();
-	if (unlikely(vcpu->arch.switch_db_regs)) {
+	if (unlikely(vcpu->arch.switch_db_regs || hw_breakpoint_active())) {
  		set_debugreg(0, 7);

I would prefer zero only dr7, e.g.

	if (unlikely(vcpu->arch.switch_db_regs)) {
		...
	} else if (hw_breakpoint_active()) {
		set_debugreg(0, 7);
	}

Stuffing all DRs isn't a bug because hw_breakpoint_restore() will restore all DRs, but loading stale state into DRs is weird.

Quoted 4 lines
Quoted 3 lines
arch/x86/kvm/x86.c +0 -0
  		set_debugreg(vcpu->arch.eff_db[0], 0);
  		set_debugreg(vcpu->arch.eff_db[1], 1);