No kernel interface to reset a VCPU
Hi all,
looks to me like there is no way to properly reset the boot processor. The current pattern used by qemu[-kvm] is to reload all registers with their reset values. But that does not affect the internal VCPU states the KVM keeps in the kernel. They are only reset during VCPU setup or after receiving a SIPI (and the latter only helps with secondary CPUs).
So the only way around it with the current kernel interface is to destroy/recreate the BSP on reset, right? /me is looking into such an approach now.
We stumbled over inconsistent VCPU states with our internal qemu-kvm tree. We have a legacy watchdog emulation here that triggered but failed to bring up the system again. I wasn't able to create a similar case with a standard environment yet, but I think it is not unrealistic for qemu-kvm as well. Hacking kvm_arch_vcpu_reset() into KVM that triggers on the right register values "solved" the issue here.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux
Quoted 8 lines · Jan Kiszka
Hi all,
looks to me like there is no way to properly reset the boot processor. The current pattern used by qemu[-kvm] is to reload all registers with their reset values. But that does not affect the internal VCPU states the KVM keeps in the kernel. They are only reset during VCPU setup or after receiving a SIPI (and the latter only helps with secondary CPUs).
Userspace should have access to internal VCPU states too, otherwise migration will not work.
Quoted 3 lines
So the only way around it with the current kernel interface is to destroy/recreate the BSP on reset, right? /me is looking into such an approach now.
I don't think destroying/recreating vcpu will work. I don't remember exact details though.
Quoted 8 lines
We stumbled over inconsistent VCPU states with our internal qemu-kvm tree. We have a legacy watchdog emulation here that triggered but failed to bring up the system again. I wasn't able to create a similar case with a standard environment yet, but I think it is not unrealistic for qemu-kvm as well. Hacking
kvm_arch_vcpu_reset()into KVM that triggers on the right register values "solved" the issue here.
Can you find the root cause of the problem? As I said above qemu should have full access to vcpu state for proper migration support. Not that kvm_vcpu_reset()/kvm_apic_reset()/kvm_ioapic_reset()/kvm_pit_reset() is bad idea. Actually I want to add them one day.
--
Gleb.
Quoted 11 lines · Gleb Natapov
Quoted 8 lines · Jan Kiszka
Hi all,
looks to me like there is no way to properly reset the boot processor. The current pattern used by qemu[-kvm] is to reload all registers with their reset values. But that does not affect the internal VCPU states the KVM keeps in the kernel. They are only reset during VCPU setup or after receiving a SIPI (and the latter only helps with secondary CPUs).
Userspace should have access to internal VCPU states too, otherwise migration will not work.
Good point.
Quoted 14 lines
Quoted 4 lines
So the only way around it with the current kernel interface is to destroy/recreate the BSP on reset, right? /me is looking into such an approach now.
I don't think destroying/recreating vcpu will work. I don't remember exact details though.
Quoted 7 lines
We stumbled over inconsistent VCPU states with our internal qemu-kvm tree. We have a legacy watchdog emulation here that triggered but failed to bring up the system again. I wasn't able to create a similar case with a standard environment yet, but I think it is not unrealistic for qemu-kvm as well. Hacking
kvm_arch_vcpu_reset()into KVM that triggers on the right register values "solved" the issue here.
Quoted 2 lines
Can you find the root cause of the problem? As I said above qemu should have full access to vcpu state for proper migration support. Not that
I just had a closer look again and found our problem: arch.nmi_pending.
I think the risk to be bitten by this on standard OSes is rather low. The reset issue we see is widely related to the special NMI-based watchdog here. The probability to see the pattern NMI-> guest handler -> NMI -> system-reset on ordinary systems is fairly low. Besides this hidden state may cause lost NMI events during migration or save/restore. Again a corner case.
But it should be fixed. Will check where we could add this bit for userland read-out.
Quoted 5 lines
kvm_vcpu_reset()/kvm_apic_reset()/kvm_ioapic_reset()/kvm_pit_reset() is bad idea. Actually I want to add them one day.
--
Gleb.
Jan