external module sched_in event

Hello,

[ I already sent it once as [email protected] but it didn't go through
for whatever reason, trying again from private email, hope there
won't be dups ]

My worst longstanding problem with KVM is that as the uptime of my host system increased, my opensuse guest images started to destabilize and lockup at boot. The weird thing was that fresh after boot everything was always perfectly ok, so I thought it was rmmod/insmod or some other sticky effect on the CPU after restarting the guest a few times that triggered the crash. Furthermore if I loaded the cpu a lot (like with a while :; do true;done), the crash would magically disappear. Decreasing cpu frequency and timings didn't help. Debugging wasn't trivial because it required a certain uptime and it didn't always crash.

So I once debugged this more aggressively I figured out KVM was ok, it was the guest that crashed in the tsc clocksource because tsc wasn't monotone. guest was looping in an infinite loop with irq disabled. So I tried to pass "notsc" and that fixed the crash just fine.

Initially I thought it was the tsc_offset logic being wrong but then I figured out that the vcpu_put/load wasn't always executed, this bugcheck triggers with current git and so I recommend to apply this to kvm.git to avoid similar nasty hard-to-detect bugs in the future (Avi says vmx would crash hard in such a condition, svm is much simpler and it somewhat survives the lack of sched_in and only crashes the guest due to not monotone tsc):

index ac876ec..26372fa 100644
@@ -742,6 +742,7 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
 
 void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
 {
+	WARN_ON(vcpu->cpu != smp_processor_id());
 	kvm_x86_ops->vcpu_put(vcpu);
 	kvm_put_guest_fpu(vcpu);
 }

So trying to understand why the ->cpu was wrong, I looked into the preempt notifiers emulation, and it looked quite fragile without a real sched_in hook. I figured out I could provide a real sched_in hook by loading the proper values in the tsk->thread.debugreg[0/7]. Initially I got the hooking points out of objdump -d vmlinux, but Avi preferred no dependency on the vmlinux and he suggested to try to find the sched_in hook in the stack. So that's what I implemented now and this should provide real robustness to the out of tree module compiled against binary kernel images with CONFIG_KVM=n. I tried to be compatible with all kernels down to 2.6.5 but only 2.6.2x host is tested and only on 64bit and only on SVM (no vmx system around here at all).

This fixes my longstanding KVM instability and "-smp 2" now works flawlessy with svm too! -smp 2 -snapshot crashes in qemu userland but that's not kernel related, must be some thread mutex lock recursion or lock inversion in the qcow cow code. Removing -snapshot make -smp 2 stable. Multiple guests UP and SMP seems stable too.

To reproduce my crash easily without waiting ages for the two tsc to deviate with an error larger than the number of cycles it takes for a CPU migration, run write_tsc(0,0) in kernel mode (like in the svm.c init function and then insmod kvm-amd; rmmod kvm-amd and then remove write_tsc and recompile kvm-amd).

#include <stdio.h>
main()

{
	unsigned long x1, x2;
	unsigned int a, d;
	asm volatile("rdtsc" : "=a" (a), "=d" (d));
	x1 = ((unsigned long)d << 32) | a;
	for (;;) {
		asm volatile("rdtsc" : "=a" (a), "=d" (d));
		x2 = ((unsigned long)d << 32) | a;
		if (x2 < x1)
			printf("error %Ld\n", x1-x2);
		else
			printf("good  %Ld\n", x1-x2);
		x1 = x2;
	}
}

(the "good.." printf can be commented out if you run this on the host, but it better stay if you run this in the guest because it helps rescheduling X with SDL that increases the frequency of the CPU-switches of the kvm task)

So in short with the below fix applied, after a write_tsc(0,0), the UP guest never return any error anymore. Previously it would return frequent errors because sched_in wasn't properly invoked by svm.c and it would crash at boot every single time after a write_tsc(0,0).

The SMP guest of course still returns TSC errors but that's ok, the smp host also return TSC errors, that's ok, it's only the UP guest that is forbidden to have a not monotone TSC or the guest would crash like it happened to me.

I'm unsure if special_reload_db7 is needed at all, but it certainly can't hurt so it's the only hack I left.

Finally I can enjoy KVM stability too ;). If you always compiled your host kernel with CONFIG_KVM=y on a recent kernels including the preempt-notifiers, you could never run into this. If you compile your host kernel with CONFIG_KVM=n please try to test this.

index 7993aa2..5187c96 100644
@@ -24,32 +24,6 @@
     printf("MODULE_INFO(version, \"%s\");\n", version)
 }
 
-/^static unsigned long vmcs_readl/ {
-    in_vmcs_read = 1
-}
-
-/ASM_VMX_VMREAD_RDX_RAX/ && in_vmcs_read {
-    printf("\tstart_special_insn();\n")
-}
-
-/return/ && in_vmcs_read {
-    printf("\tend_special_insn();\n");
-    in_vmcs_read = 0
-}
-
-/^static void vmcs_writel/ {
-    in_vmcs_write = 1
-}
-
-/ASM_VMX_VMWRITE_RAX_RDX/ && in_vmcs_write {
-    printf("\tstart_special_insn();\n")
-}
-
-/if/ && in_vmcs_write {
-    printf("\tend_special_insn();\n");
-    in_vmcs_write = 0
-}
-
 /^static void vmx_load_host_state/ {
     vmx_load_host_state = 1
 }

@@ -74,15 +48,6 @@
     print "\tspecial_reload_dr7();"
 }
 
-/static void vcpu_put|static int __vcpu_run|static struct kvm_vcpu \*vmx_create_vcpu/ {
-    in_tricky_func = 1
-}
-
-/preempt_disable|get_cpu/ && in_tricky_func {
-    printf("\tin_special_section();\n");
-    in_tricky_func = 0
-}
-
 /unsigned long flags;/ &&  vmx_load_host_state {
     print "\tunsigned long gsbase;"
 }

@@ -90,4 +55,3 @@
 /local_irq_save/ &&  vmx_load_host_state {
     print "\t\tgsbase = vmcs_readl(HOST_GS_BASE);"
 }
-

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

oh, it was sent to the list, dont trust (in case you did) the source forge site for the mails
inside this list, gmane is much better...

you mean that without -snapshot, the userspace not hang at the sigwait() in the qcow code?

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

But this time I received it in my kvm-devel folder... previously I didn't, so it had to be blocked my some spamfilter in the other account sorry for the dup but there was definitely a glitch somewhere and there was no glitch this time ;). Thanks for the gmane tip!

Yes.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

for some reason the list is configured with the mailman option for :

Receive your own posts to the list? no

so you will only see it you post it with another account as the one you have subscribed as you find out, or if waiting long enough for the mail archive to catch up.

and it doesn't have spam filter.

Carlo

PS. I agree that this setup is confusing and would rather see it changed if that is an open possibility.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

It's just a default; users can change it if they wish. I changed the default to send a copy of own posts, but that only applied to new subscribers.

Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

The problem is probably exaggerated by -snapshot, which needs much more metadata activity than without -snapshot.

Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/