Skip to contentSkip to conversation

guest panic fixed

kvm-ppc-devel Hollis Blanchard 2 participants 5 messages Mar 19, 2008 – Mar 20, 2008

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Thanks to some excellent debugging by Christian, we found the cause of our annoying guest panics. We had not implemented exception priorities (my bad), so a userspace ITLB miss was being delivered immediately after entry to the decrementer interrupt vector. Patch follows,
http://penguinppc.org/~hollisb/kvm/ updated.

NFS root with virtio-net still doesn't work for me, but it doesn't crash. :)

arch/powerpc/kvm/powerpc.c +47 -7
@@ -68,6 +68,44 @@ static const u32 interrupt_msr_mask[16] 
 	[BOOKE_INTERRUPT_DEBUG]         = MSR_ME,
 };
 
+const unsigned char exception_priority[] = {
+	[BOOKE_INTERRUPT_DATA_STORAGE] = 0,
+	[BOOKE_INTERRUPT_INST_STORAGE] = 1,
+	[BOOKE_INTERRUPT_ALIGNMENT] = 2,
+	[BOOKE_INTERRUPT_PROGRAM] = 3,
+	[BOOKE_INTERRUPT_FP_UNAVAIL] = 4,
+	[BOOKE_INTERRUPT_SYSCALL] = 5,
+	[BOOKE_INTERRUPT_AP_UNAVAIL] = 6,
+	[BOOKE_INTERRUPT_DTLB_MISS] = 7,
+	[BOOKE_INTERRUPT_ITLB_MISS] = 8,
+	[BOOKE_INTERRUPT_MACHINE_CHECK] = 9,
+	[BOOKE_INTERRUPT_DEBUG] = 10,
+	[BOOKE_INTERRUPT_CRITICAL] = 11,
+	[BOOKE_INTERRUPT_WATCHDOG] = 12,
+	[BOOKE_INTERRUPT_EXTERNAL] = 13,
+	[BOOKE_INTERRUPT_FIT] = 14,
+	[BOOKE_INTERRUPT_DECREMENTER] = 15,
+};
+
+const unsigned char priority_exception[] = {
+	BOOKE_INTERRUPT_DATA_STORAGE,
+	BOOKE_INTERRUPT_INST_STORAGE,
+	BOOKE_INTERRUPT_ALIGNMENT,
+	BOOKE_INTERRUPT_PROGRAM,
+	BOOKE_INTERRUPT_FP_UNAVAIL,
+	BOOKE_INTERRUPT_SYSCALL,
+	BOOKE_INTERRUPT_AP_UNAVAIL,
+	BOOKE_INTERRUPT_DTLB_MISS,
+	BOOKE_INTERRUPT_ITLB_MISS,
+	BOOKE_INTERRUPT_MACHINE_CHECK,
+	BOOKE_INTERRUPT_DEBUG,
+	BOOKE_INTERRUPT_CRITICAL,
+	BOOKE_INTERRUPT_WATCHDOG,
+	BOOKE_INTERRUPT_EXTERNAL,
+	BOOKE_INTERRUPT_FIT,
+	BOOKE_INTERRUPT_DECREMENTER,
+};
+
 
 gfn_t unalias_gfn(struct kvm *kvm, gfn_t gfn)
 {
@@ -183,19 +221,21 @@ static void kvmppc_check_and_deliver_int
 static void kvmppc_check_and_deliver_interrupts(struct kvm_vcpu *vcpu)
 {
 	unsigned long *pending = &vcpu->arch.pending_exceptions;
-	int exception;
-
-	exception = find_first_bit(pending, BITS_PER_BYTE * sizeof(*pending));
-	while (exception < BOOKE_MAX_INTERRUPT) {
+	unsigned int exception;
+	unsigned int priority;
+
+	priority = find_first_bit(pending, BITS_PER_BYTE * sizeof(*pending));
+	while (priority <= BOOKE_MAX_INTERRUPT) {
+		exception = priority_exception[priority];
 		if (kvmppc_can_deliver_interrupt(vcpu, exception)) {
 			kvmppc_clear_exception(vcpu, exception);
 			kvmppc_deliver_interrupt(vcpu, exception);
 			break;
 		}
 
-		exception = find_next_bit(pending,
-		                          BITS_PER_BYTE * sizeof(*pending),
-		                          exception + 1);
+		priority = find_next_bit(pending,
+		                         BITS_PER_BYTE * sizeof(*pending),
+		                         priority + 1);
 	}
 }
 
include/asm-powerpc/kvm_ppc.h +7 -2
@@ -41,6 +41,9 @@ enum emulation_result {
 	EMULATE_FAIL,         /* can't emulate this instruction */
 };
 
+extern const unsigned char exception_priority[];
+extern const unsigned char priority_exception[];
+
 extern int __kvmppc_vcpu_run(struct kvm_run *kvm_run, struct kvm_vcpu *vcpu);
 extern char kvmppc_handlers_start[];
 extern unsigned long kvmppc_handler_len;
@@ -65,12 +68,14 @@ extern void kvmppc_44x_tlb_trace(int act
 
 static inline void kvmppc_queue_exception(struct kvm_vcpu *vcpu, int exception)
 {
-	set_bit(exception, &vcpu->arch.pending_exceptions);
+	unsigned int priority = exception_priority[exception];
+	set_bit(priority, &vcpu->arch.pending_exceptions);
 }
 
 static inline void kvmppc_clear_exception(struct kvm_vcpu *vcpu, int exception)
 {
-	clear_bit(exception, &vcpu->arch.pending_exceptions);
+	unsigned int priority = exception_priority[exception];
+	clear_bit(priority, &vcpu->arch.pending_exceptions);
 }
 
 static inline void kvmppc_set_msr(struct kvm_vcpu *vcpu, u32 new_msr)
Signature · 12 lines
Hollis Blanchard
IBM Linux Technology Center


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
kvm-ppc-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-ppc-devel
#1 lore raw
Quoted 10 lines · Hollis Blanchard

Thanks to some excellent debugging by Christian, we found the cause of our annoying guest panics. We had not implemented exception priorities (my bad), so a userspace ITLB miss was being delivered immediately after entry to the decrementer interrupt vector. Patch follows,
http://penguinppc.org/~hollisb/kvm/ updated.

NFS root with virtio-net still doesn't work for me, but it doesn't crash. :)

include/asm-powerpc/kvm_ppc.h extern void kvmppc_44x_tlb_trace(int act · line 72

Quoted 6 lines
include/asm-powerpc/kvm_ppc.h +2 -1
@@ -65,12 +68,14 @@ extern void kvmppc_44x_tlb_trace(int act
 
 static inline void kvmppc_queue_exception(struct kvm_vcpu *vcpu, int exception)
 {
-	set_bit(exception, &vcpu->arch.pending_exceptions);
+	unsigned int priority = exception_priority[exception];
+	set_bit(priority, &vcpu->arch.pending_exceptions);

saves a line and a local var on irq path (well the compiler might optimize that anyway)

	set_bit(exception_priority[exception], &vcpu->arch.pending_exceptions);

include/asm-powerpc/kvm_ppc.h extern void kvmppc_44x_tlb_trace(int act · line 78

Quoted 7 lines
include/asm-powerpc/kvm_ppc.h +2 -1
 }
 
 static inline void kvmppc_clear_exception(struct kvm_vcpu *vcpu, int exception)
 {
-	clear_bit(exception, &vcpu->arch.pending_exceptions);
+	unsigned int priority = exception_priority[exception];
+	clear_bit(priority, &vcpu->arch.pending_exceptions);

same as above

Quoted 3 lines
include/asm-powerpc/kvm_ppc.h +0 -0
 }
 
 static inline void kvmppc_set_msr(struct kvm_vcpu *vcpu, u32 new_msr)
Signature · 12 lines
Grüsse / regards, 
Christian Ehrhardt
IBM Linux Technology Center, Open Virtualization

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
kvm-ppc-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-ppc-devel
#2 lore raw
In reply to Hollis Blanchard
Quoted 9 lines · Hollis Blanchard

Thanks to some excellent debugging by Christian, we found the cause of our annoying guest panics. We had not implemented exception priorities (my bad), so a userspace ITLB miss was being delivered immediately after entry to the decrementer interrupt vector. Patch follows,
http://penguinppc.org/~hollisb/kvm/ updated.

NFS root with virtio-net still doesn't work for me, but it doesn't crash. :)

my network works fine (at least initially) see the attached log:

Sending DHCP requests .., OK
IP-Config: Got DHCP answer from 192.168.1.2, my address is 192.168.1.10
IP-Config: Complete:

     device=eth0, addr2.168.1.10, mask%5.255.255.0, gw2.168.1.2,
     host2.168.1.10, domain=local-devnet, nis-domain=(none),
     bootserver2.168.1.2, rootserver2.168.1.2, rootpathLooking up port of RPC 100003/2 on 192.168.1.2

Looking up port of RPC 100005/1 on 192.168.1.2
VFS: Mounted root (nfs filesystem).
Freeing unused kernel memory: 116k init
modprobe: FATAL: Could not load /lib/modules/2.6.25-rc3-dirty/modules.dep: No such file or directory
modprobe: FATAL: Could not load /lib/modules/2.6.25-rc3-dirty/modules.dep: No such file or directory
INIT: version 2.85 booting
nfs: server 192.168.1.2 not responding, still trying
nfs: server 192.168.1.2 not responding, still trying
nfs: server 192.168.1.2 not responding, still trying

INFO: task rc.sysinit:679 blocked for more than 120 seconds.

"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

After that I run into those lockup's for init:1 and rc.sysinit:679.
The network is working fine for a while and then that I get these nfs timeouts you see above.
I think thats the track I will continue - find out whats wrong there e.g. virtio mangles the packets, too high latency, lost virtio interrupts, ... whatever.

@Hollis let me know if I can help you with your nfs root setup somehow?

Signature · 12 lines
Grüsse / regards, 
Christian Ehrhardt
IBM Linux Technology Center, Open Virtualization

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
kvm-ppc-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-ppc-devel
#3 lore raw
In reply to Hollis Blanchard
Quoted 32 lines · Christian Ehrhardt
Quoted 9 lines · Hollis Blanchard

Thanks to some excellent debugging by Christian, we found the cause of our annoying guest panics. We had not implemented exception priorities (my bad), so a userspace ITLB miss was being delivered immediately after entry to the decrementer interrupt vector. Patch follows,
http://penguinppc.org/~hollisb/kvm/ updated.

NFS root with virtio-net still doesn't work for me, but it doesn't crash. :)

my network works fine (at least initially) see the attached log:

Sending DHCP requests .., OK
IP-Config: Got DHCP answer from 192.168.1.2, my address is 192.168.1.10
IP-Config: Complete:

     device=eth0, addr2.168.1.10, mask%5.255.255.0, gw2.168.1.2,
     host2.168.1.10, domain=local-devnet, nis-domain=(none),
     bootserver2.168.1.2, rootserver2.168.1.2, rootpath> Looking up port of RPC 100003/2 on 192.168.1.2

Looking up port of RPC 100005/1 on 192.168.1.2
VFS: Mounted root (nfs filesystem).
Freeing unused kernel memory: 116k init
modprobe: FATAL: Could not load /lib/modules/2.6.25-rc3-dirty/modules.dep: No such file or directory
modprobe: FATAL: Could not load /lib/modules/2.6.25-rc3-dirty/modules.dep: No such file or directory
INIT: version 2.85 booting
nfs: server 192.168.1.2 not responding, still trying
nfs: server 192.168.1.2 not responding, still trying
nfs: server 192.168.1.2 not responding, still trying

INFO: task rc.sysinit:679 blocked for more than 120 seconds.

"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Quoted 8 lines

After that I run into those lockup's for init:1 and rc.sysinit:679. The network is working fine for a while and then that I get these nfs timeouts you see above.
I think thats the track I will continue - find out whats wrong there e.g. virtio mangles the packets, too high latency, lost virtio interrupts, ... whatever.

@Hollis let me know if I can help you with your nfs root setup somehow?

No, that's pretty much exactly what I see, but my timeouts appear before modprobe and the INIT message.

Signature · 12 lines
Hollis Blanchard
IBM Linux Technology Center


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
kvm-ppc-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-ppc-devel
#4 lore raw
In reply to Hollis Blanchard
Quoted 12 lines · Christian Ehrhardt
Quoted 11 lines · Hollis Blanchard

Thanks to some excellent debugging by Christian, we found the cause of our annoying guest panics. We had not implemented exception priorities (my bad), so a userspace ITLB miss was being delivered immediately after entry to the decrementer interrupt vector. Patch follows,
http://penguinppc.org/~hollisb/kvm/ updated.

NFS root with virtio-net still doesn't work for me, but it doesn't crash. :)

(quoted diff) +0 -0
@@ -65,12 +68,14 @@ extern void kvmppc_44x_tlb_trace(int act
 
Quoted 8 lines

include/asm-powerpc/kvm_ppc.h extern void kvmppc_44x_tlb_trace(int act · line 72

Quoted 5 lines
include/asm-powerpc/kvm_ppc.h +2 -1
 static inline void kvmppc_queue_exception(struct kvm_vcpu *vcpu, int exception)
 {
-	set_bit(exception, &vcpu->arch.pending_exceptions);
+	unsigned int priority = exception_priority[exception];
+	set_bit(priority, &vcpu->arch.pending_exceptions);

saves a line and a local var on irq path (well the compiler might optimize that anyway)

	set_bit(exception_priority[exception], &vcpu->arch.pending_exceptions);

The compiler should handle it; you can examine the disassembly if you're worried.

Signature · 12 lines
Hollis Blanchard
IBM Linux Technology Center


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
kvm-ppc-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-ppc-devel