Skip to contentSkip to conversation

Guests oops when trying to mount initramfs

kvm-ppc-devel Christian Ehrhardt 2 participants 6 messages Mar 12, 2008 – Mar 17, 2008

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi,
I already chatted with Hollis a bit about that, but wanted to ask you about it. I also put kvm-ppc-devel on cc because it seem to be a common issue. I as well as Hollis saw that kind of bug in the past occasionally but never in a reproducible way to be able to debug it.
So the first question is - is your scenario running into that bug to ~>95% ?

And if it reproducible in your case we can use it to find the reason for it, here a short summary from my past discussions about that issue:
- effectively what you see is that the guest has an itlb miss for it's kernel mapping.
- we only deliver that to the guest if we can't find that mapping in the guest tlb
- this should never happen because the guest should never remove it's own kernel mapping and therefor we should be able to cover all itlb misses in the host by using the guest tlb
- but it happens so the question is when&why is the guest kernel mapping removed from the guest tlb
- afaik we have only one place that might remove that mapping which is the function kvmppc_emul_tlbwe
- we also know the kernel mapping which is usually a single 16mb mapping we initialize for the guest in kvm_arch_vcpu_setup (I might be wrong here - Hollis?) => we could track that tlbwe that should never happen by inserting a warn_on into kvmppc_emul_tlbwe if it overwrites a guest tlb kernel mapping and starting from there debug whats going on

Any comments or other suggestions are welcome

Quoted 10 lines · Jerone Young

I'm looking into this. But got a guest to load without the device tree mainpulation code. I have an cuImage with a prefilled device tree built in & initramfs integrated.

It oopses at Decrementor()

Does not mount the initramfs

Here is the trace:

[...]

Quoted 28 lines

RPC: Registered udp transport module.
RPC: Registered tcp transport module.
Freeing unused kernel memory: 1500k init

Unable to handle kernel paging request for instruction fetch
Faulting instruction address: 0xc0000ac0
Oops: Kernel access of bad area, sig: 1 [#1]
Bamboo
Modules linked in:
NIP: c0000ac0 LR: 48003fdc CTR: 00000000
REGS: bfcfe910 TRAP: 0400   Not tainted  (2.6.25-rc3-dirty)
MSR: 00021000 <ME>  CR: 42000028  XER: 00000000

TASK = c8810400[1] 'init' THREAD: c881e000
GPR00: 00000001 bfcfe9c0 00000000 480177b0 4801795c 00000001 00000000 48011f34
GPR08: 0002d002 48026e64 00000000 bfcfe9c0 c881e000 00000000 00000000 00000000
GPR16: 00000000 48018700 00000001 c0240000 c0240000 bfcfea48 00000007 10000034
GPR24: 00000000 6fffff40 48017000 48027f4c 00000000 480177b0 480272e4 bfcfe9c0
NIP [c0000ac0] Decrementer+0x0/0xc0
LR [48003fdc] 0x48003fdc

Call Trace:

Instruction dump:
914b00b0 3d400002 614a1002 512a0420 4800c6b5 c000b378 c000d924 60000000 60000000 60000000 60000000 60000000 <7d5043a6> 7d7143a6 7c3443a6 7d400026

---[ end trace 59dad6c3a6e54fa0 ]---
Signature · 12 lines
Grüsse / regards, 
Christian Ehrhardt
IBM Linux Technology Center, Open Virtualization

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
kvm-ppc-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-ppc-devel
#1 lore raw

Yup, this is definitely the right direction. I actually hadn't looked at tlbwe yet, but that's probably where the issue is.

I'm also looking at adding some "relay" debugfs code to export all of KVM's TLB operations to userspace for analysis. There are a *lot* of them, and printk is not suitable for this data. :) This will be of more general use down the road when we consider more sophisticated shadow TLB eviction policies (i.e. not round robin)...

Signature · 2 lines
Hollis Blanchard
IBM Linux Technology Center
Quoted 21 lines · Christian Ehrhardt

Hi,
I already chatted with Hollis a bit about that, but wanted to ask you about it. I also put kvm-ppc-devel on cc because it seem to be a common issue. I as well as Hollis saw that kind of bug in the past occasionally but never in a reproducible way to be able to debug it.
So the first question is - is your scenario running into that bug to ~>95% ?

And if it reproducible in your case we can use it to find the reason for it, here a short summary from my past discussions about that issue:
- effectively what you see is that the guest has an itlb miss for it's kernel mapping.
- we only deliver that to the guest if we can't find that mapping in the guest tlb
- this should never happen because the guest should never remove it's own kernel mapping and therefor we should be able to cover all itlb misses in the host by using the guest tlb
- but it happens so the question is when&why is the guest kernel mapping removed from the guest tlb
- afaik we have only one place that might remove that mapping which is the function kvmppc_emul_tlbwe
- we also know the kernel mapping which is usually a single 16mb mapping we initialize for the guest in kvm_arch_vcpu_setup (I might be wrong here - Hollis?) => we could track that tlbwe that should never happen by inserting a warn_on into kvmppc_emul_tlbwe if it overwrites a guest tlb kernel mapping and starting from there debug whats going on

Any comments or other suggestions are welcome

Quoted 4 lines · Jerone Young

I'm looking into this. But got a guest to load without the device tree mainpulation code. I have an cuImage with a prefilled device tree built in & initramfs integrated.

Quoted 36 lines
Quoted 6 lines

It oopses at Decrementor()

Does not mount the initramfs

Here is the trace:

[...]

Quoted 28 lines

RPC: Registered udp transport module.
RPC: Registered tcp transport module.
Freeing unused kernel memory: 1500k init

Unable to handle kernel paging request for instruction fetch
Faulting instruction address: 0xc0000ac0
Oops: Kernel access of bad area, sig: 1 [#1]
Bamboo
Modules linked in:
NIP: c0000ac0 LR: 48003fdc CTR: 00000000
REGS: bfcfe910 TRAP: 0400   Not tainted  (2.6.25-rc3-dirty)
MSR: 00021000 <ME>  CR: 42000028  XER: 00000000

TASK = c8810400[1] 'init' THREAD: c881e000
GPR00: 00000001 bfcfe9c0 00000000 480177b0 4801795c 00000001 00000000 48011f34
GPR08: 0002d002 48026e64 00000000 bfcfe9c0 c881e000 00000000 00000000 00000000
GPR16: 00000000 48018700 00000001 c0240000 c0240000 bfcfea48 00000007 10000034
GPR24: 00000000 6fffff40 48017000 48027f4c 00000000 480177b0 480272e4 bfcfe9c0
NIP [c0000ac0] Decrementer+0x0/0xc0
LR [48003fdc] 0x48003fdc

Call Trace:

Instruction dump:
914b00b0 3d400002 614a1002 512a0420 4800c6b5 c000b378 c000d924 60000000 60000000 60000000 60000000 60000000 <7d5043a6> 7d7143a6 7c3443a6 7d400026

---[ end trace 59dad6c3a6e54fa0 ]---

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
kvm-ppc-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-ppc-devel

#2 lore raw
In reply to Christian Ehrhardt

Christian Ehrhardt wrote:
[...]

Quoted 9 lines

And if it reproducible in your case we can use it to find the reason for it, here a short summary from my past discussions about that issue:
- effectively what you see is that the guest has an itlb miss for it's kernel mapping.
- we only deliver that to the guest if we can't find that mapping in the guest tlb
- this should never happen because the guest should never remove it's own kernel mapping and therefor we should be able to cover all itlb misses in the host by using the guest tlb
- but it happens so the question is when&why is the guest kernel mapping removed from the guest tlb
- afaik we have only one place that might remove that mapping which is the function kvmppc_emul_tlbwe
- we also know the kernel mapping which is usually a single 16mb mapping we initialize for the guest in kvm_arch_vcpu_setup (I might be wrong here - Hollis?) => we could track that tlbwe that should never happen by inserting a warn_on into kvmppc_emul_tlbwe if it overwrites a guest tlb kernel mapping and starting from there debug whats going on

ok for Hollis to continue and anyone else that might have comments
-> tracking any eviction of the guest kernel mapping (=0xC..) did not trigger
-> tracking the delivery of a itlb/dtlb miss with that address did not trigger
-> I inserted a BUG statement in the guests report of "Unable to handle kernel paging request ..."
That triggers a emulation of a trap in the host and there I have a dump_vcpu&dump_tlb
-> for the case that someone wants to reproduce that I attached debug patches for host and guest kernel

Here is the output when hitting the trap:

Looking up port of RPC 100005/1 on 192.168.1.2
VFS: Mounted root (nfs filesystem). Freeing unused kernel memory: 116k init

Unable to handle kernel paging request for instruction fetch
Faulting instruction address: 0xc0000ac0
trap!
pc:   c000fcbc msr:  00021002
lr:   c000fcbc ctr:  c012dc54
srr0: c012d638 srr1: 00021002
exceptions: 00000000
gpr00: c000fcbc bff2aaf0 c8810400 0000002c
gpr04: 00000001 00000001 00000000 00000004
gpr08: 00000001 c025ca00 00001250 c0260000
gpr12: 65930e5e 1001f2f8 00000000 00000000
gpr16: 00000000 00000000 00000000 c01f0000
gpr20: c0240000 c0240000 00000000 00000000
gpr24: c0240000 c01f0000 00000001 48026e60
gpr28: 48027a10 bff2af04 00000001 bff2ab10
vcpu 0 TLB dump:
| nr |     tid  |    word0 |    word1 |    word2 |
G 1 | 00000001 | 0FF3F210 | 08FAB000 | 08FA016D |
G 2 | 00000001 | 0FEAD210 | 08FDC000 | 08FD0149 |
G 3 | 00000001 | 48026210 | 08FFB000 | 08FF0349 |
G 5 | 00000001 | 10003210 | 0022D000 | 0022016D |
G 6 | 00000000 | D1012210 | EF600000 | EF600703 |
G 7 | 00000001 | 4800C210 | 003D7000 | 003D016D |
G 8 | 00000001 | 10000210 | 00227000 | 00220149 |
G 9 | 00000001 | 0FFE8210 | 08FE6000 | 08FE0349 |
G11 | 00000001 | 0FEC5210 | 08FD4000 | 08FD016D |
G13 | 00000001 | 0FFEA210 | 08FCD000 | 08FC036F |
G14 | 00000001 | 0FFEC210 | 08FE5000 | 08FE035B |
G15 | 00000001 | 0FFEF210 | 08FDE000 | 08FD035B |
G16 | 00000000 | D1021210 | E8001000 | E8000703 |
G18 | 00000001 | 0FF70210 | 003EB000 | 003E016D |
G20 | 00000001 | 0FF20210 | 003F8000 | 003F016D |
G22 | 00000001 | 0FF1F210 | 003F9000 | 003F016D |
G23 | 00000001 | 0FFE7210 | 08FCB000 | 08FC0349 |
G25 | 00000001 | 0FF17210 | 08F85000 | 08F8016D |
G26 | 00000001 | 48013210 | 00235000 | 0023016D |
G28 | 00000001 | 10001210 | 00228000 | 0022016D |
G31 | 00000001 | 10017210 | 00242000 | 0024037F |
G32 | 00000001 | 48012210 | 00234000 | 0023016D |
G33 | 00000001 | 0FEA9210 | 003DC000 | 003D0149 |
G34 | 00000001 | 0FEB3210 | 08FEB000 | 08FE0149 |
G35 | 00000001 | 0FEB9210 | 08FEC000 | 08FE0149 |
G36 | 00000001 | 0FEBA210 | 08FC2000 | 08FC0149 |
G37 | 00000001 | 48009210 | 08FFA000 | 08FF016D |
G39 | 00000001 | 0FEDD210 | 08F8A000 | 08F8016D |
G40 | 00000001 | 0FFED210 | 08FCA000 | 08FC035B |
G42 | 00000001 | 10005210 | 0022F000 | 0022016D |
G43 | 00000001 | 10006210 | 00230000 | 0023016D |
G45 | 00000001 | 0FEDA210 | 08F8D000 | 08F8016D |
G46 | 00000001 | 4801C210 | 08FC9000 | 08FC035B |
G47 | 00000001 | 0FEB4210 | 08FC1000 | 08FC0149 |
G48 | 00000001 | 0FEAB210 | 08FD7000 | 08FD0149 |
G49 | 00000001 | 0FEB0210 | 08FD9000 | 08FD0149 |
G50 | 00000001 | 0FEB7210 | 08FEE000 | 08FE0149 |
G51 | 00000001 | 0FEAF210 | 08FDA000 | 08FD0149 |
G52 | 00000001 | 0FEB6210 | 08FEF000 | 08FE0149 |
G53 | 00000001 | 0FEAA210 | 003DB000 | 003D0149 |
G55 | 00000001 | 0FF61210 | 08F97000 | 08F9016D |
G56 | 00000001 | 0FEB2210 | 08FE7000 | 08FE0149 |
G57 | 00000001 | 0FEB8210 | 08FED000 | 08FE0149 |
G58 | 00000001 | 0FEAC210 | 08FDD000 | 08FD0149 |
G59 | 00000001 | 0FEB5210 | 08FC0000 | 08FC0149 |
G60 | 00000001 | 48008210 | 08FFC000 | 08FF016D |
G61 | 00000001 | 48027210 | 00241000 | 0024037F |
G62 | 00000001 | BFF2A210 | 00243000 | 0024035B |
G63 | 00000000 | C0000290 | 00000000 | 00000107 |
S17 | 00000001 | 10003310 | 0E3B8000 | 0000002F |
S18 | 00000001 | BFF2A310 | 0E3CE000 | 0000001F |
S19 | 00000001 | 10017310 | 0E3CD000 | 0000003F |
S20 | 00000000 | C0000310 | 0E38B000 | 0000003F |
S21 | 00000000 | C0246310 | 0E3D1000 | 0000003F |
S22 | 00000000 | C000C310 | 0E3E7000 | 0000003F |
S23 | 00000000 | C8810310 | 0E9BA000 | 0000003F |
S24 | 00000000 | C000D310 | 0E3E8000 | 0000003F |
S25 | 00000000 | C000F310 | 0E3EA000 | 0000003F |
S26 | 00000000 | C0264310 | 0DC31000 | 0000003F |
S27 | 00000000 | C0037310 | 0DCF2000 | 0000003F |
S28 | 00000000 | C0106310 | 0E9D1000 | 0000003F |
S29 | 00000000 | C0223310 | 0E3AE000 | 0000003F |
S30 | 00000000 | C0047310 | 0DD02000 | 0000003F |
S31 | 00000000 | C024A310 | 0E3D5000 | 0000003F |
S32 | 00000000 | C0024310 | 0DC1F000 | 0000003F |
S33 | 00000000 | C0248310 | 0E3D3000 | 0000003F |
S34 | 00000000 | C010D310 | 0E9D8000 | 0000003F |
S35 | 00000000 | C010C310 | 0E9D7000 | 0000003F |
S36 | 00000000 | C01F6310 | 0E201000 | 0000003F |
S37 | 00000000 | C026B310 | 0DC38000 | 0000003F |
S38 | 00000000 | C026C310 | 0DC39000 | 0000003F |
S39 | 00000000 | C0023310 | 0DC1E000 | 0000003F |
S40 | 00000000 | C0257310 | 0DC22000 | 0000003F |
S41 | 00000000 | C012D310 | 0E0B8000 | 0000003F |
S42 | 00000000 | C004A310 | 0DD05000 | 0000003F |
S43 | 00000000 | C001E310 | 0DC19000 | 0000003F |
S44 | 00000000 | C001D310 | 0DC18000 | 0000003F |
S45 | 00000000 | C000A310 | 0E3E5000 | 0000003F |
S46 | 00000000 | C0262310 | 0DC2D000 | 0000003F |
S47 | 00000000 | C027C310 | 0DC69000 | 0000003F |
S48 | 00000000 | C01D2310 | 0E13D000 | 0000003F |
S49 | 00000000 | C8835310 | 0E06B000 | 0000003F |
S50 | 00000000 | C0001310 | 0E38C000 | 0000003F |
S51 | 00000000 | C025C310 | 0DC27000 | 0000003F |
S52 | 00000000 | C027D310 | 0DC6A000 | 0000003F |
S53 | 00000000 | C012A310 | 0E0B5000 | 0000003F |
S54 | 00000000 | C0130310 | 0E0BB000 | 0000003F |
S55 | 00000000 | C01D0310 | 0E13B000 | 0000003F |
S56 | 00000000 | C0256310 | 0DC21000 | 0000003F |
Oops: Exception in kernel mode, sig: 4 [#1]
Bamboo
Modules linked in:
NIP: c000fcbc LR: c000fcbc CTR: c012dc54
REGS: bff2aa40 TRAP: 0700   Not tainted  (2.6.25-rc3)
MSR: 00021002 <ME>  CR: 24002022  XER: 00000000

TASK = c8810400[1] 'init' THREAD: c881e000
GPR00: c000fcbc bff2aaf0 c8810400 0000002c 00000001 00000001 00000000 00000004
GPR08: 00000001 c025ca00 00001250 c0260000 65930e5e 1001f2f8 00000000 00000000
GPR16: 00000000 00000000 00000000 c01f0000 c0240000 c0240000 00000000 00000000
GPR24: c0240000 c01f0000 00000001 48026e60 48027a10 bff2af04 00000001 bff2ab10 NIP [c000fcbc] bad_page_fault+0x7c/0xb4
LR [c000fcbc] bad_page_fault+0x7c/0xb4

Call Trace:
Instruction dump:
2b800380 419d0034 2f800300 40be003c 3c60c01f 809f00a4 38636580 48015271
809f0080 3c60c01f 3863664c 48015261 <0fe00000> 48000000 2f800400 419e001c
---[ end trace 9c05eabdb79d9d2c ]---
Kernel panic - not syncing: Attempted to kill init!

Rebooting in 180 seconds..

Signature · 2 lines
Grüsse / regards, Christian Ehrhardt
IBM Linux Technology Center, Open Virtualization
arch/powerpc/mm/fault.c +3 -0
@@ -432,5 +432,8 @@ void bad_page_fault(struct pt_regs *regs
 	printk(KERN_ALERT "Faulting instruction address: 0x%08lx\n",
 		regs->nip);
 
+	// to trigger kvmppc host trap!
+	BUG();
+
 	die("Kernel access of bad area", regs, sig);
 }
arch/powerpc/kvm/44x_tlb.c +31 -0
@@ -21,6 +21,7 @@
 #include <linux/string.h>
 #include <linux/kvm_host.h>
 #include <linux/highmem.h>
+#include <linux/delay.h>
 #include <asm/mmu-44x.h>
 
 #include "44x_tlb.h"
@@ -29,6 +30,36 @@
 #define PPC44x_TLB_SUPER_PERM_MASK (PPC44x_TLB_SX|PPC44x_TLB_SR|PPC44x_TLB_SW)
 
 static unsigned int kvmppc_tlb_44x_pos;
+
+void kvmppc_dump_tlb(struct kvm_vcpu *vcpu)
+{
+	struct tlbe *tlbe;
+	int i;
+
+	printk("vcpu %d TLB dump:\n", vcpu->vcpu_id);
+	printk("| %2s | %8s | %8s | %8s | %8s |\n",
+		"nr", " tid ", "word0", "word1", "word2");
+
+	for (i = 0; i < PPC44x_TLB_SIZE; i++)
+	{
+		tlbe = &vcpu->arch.guest_tlb[i];
+		if (tlbe->word0 & PPC44x_TLB_VALID)
+			printk("G%2d | %08X | %08X | %08X | %08X |\n",
+			i, tlbe->tid, tlbe->word0, tlbe->word1,	tlbe->word2);
+	}
+
+	msleep(500);
+
+	for (i = 0; i < PPC44x_TLB_SIZE; i++)
+	{
+		tlbe = &vcpu->arch.shadow_tlb[i];
+		if (tlbe->word0 & PPC44x_TLB_VALID)
+			printk("S%2d | %08X | %08X | %08X | %08X |\n",
+			i, tlbe->tid, tlbe->word0, tlbe->word1,	tlbe->word2);
+	}
+
+	msleep(500);
+}
 
 static u32 kvmppc_44x_tlb_shadow_attrib(u32 attrib, int usermode)
 {
arch/powerpc/kvm/44x_tlb.h +2 -0
@@ -27,6 +27,8 @@ extern int kvmppc_44x_tlb_index(struct k
                                 unsigned int pid, unsigned int as);
 extern struct tlbe *kvmppc_44x_dtlb_search(struct kvm_vcpu *vcpu, gva_t eaddr);
 extern struct tlbe *kvmppc_44x_itlb_search(struct kvm_vcpu *vcpu, gva_t eaddr);
+
+extern void kvmppc_dump_tlb(struct kvm_vcpu *vcpu);
 
 /* TLB helper functions */
 static inline unsigned int get_tlb_size(const struct tlbe *tlbe)
arch/powerpc/kvm/emulate.c +11 -0
@@ -129,6 +129,7 @@ static int kvmppc_emul_tlbwe(struct kvm_
 	if (index > PPC44x_TLB_SIZE) {
 		printk("%s: index %d\n", __func__, index);
 		kvmppc_dump_vcpu(vcpu);
+		kvmppc_dump_tlb(vcpu);
 		return EMULATE_FAIL;
 	}
 
@@ -138,6 +139,14 @@ static int kvmppc_emul_tlbwe(struct kvm_
 #endif
 
 	tlbe = &vcpu->arch.guest_tlb[index];
+
+	if ((get_tlb_eaddr(tlbe) >> 30) == 0xc) {
+		printk("evicting %02d: %08x %08x %08x %08x\n", index,
+		tlbe->tid, tlbe->word0, tlbe->word1, tlbe->word2);
+		kvmppc_dump_vcpu(vcpu);
+		WARN_ON(1);
+	}
+
 
 	/* Invalidate shadow mappings for the about-to-be-clobbered TLBE. */
 	if (tlbe->word0 & PPC44x_TLB_VALID) {
@@ -250,6 +259,8 @@ int kvmppc_emulate_instruction(struct kv
 	switch (get_op(inst)) {
 	case 3:                                                 /* trap */
 		printk("trap!\n");
+		kvmppc_dump_vcpu(vcpu);
+		kvmppc_dump_tlb(vcpu);
 		kvmppc_queue_exception(vcpu, BOOKE_INTERRUPT_PROGRAM);
 		advance = 0;
 		break;
arch/powerpc/kvm/powerpc.c +13 -0
@@ -218,6 +218,7 @@ int kvmppc_handle_exit(struct kvm_run *r
 	case BOOKE_INTERRUPT_MACHINE_CHECK:
 		printk("MACHINE CHECK: %lx\n", mfspr(SPRN_MCSR));
 		kvmppc_dump_vcpu(vcpu);
+		kvmppc_dump_tlb(vcpu);
 		r = RESUME_HOST;
 		break;
 
@@ -310,6 +311,12 @@ int kvmppc_handle_exit(struct kvm_run *r
 		gtlbe = kvmppc_44x_dtlb_search(vcpu, eaddr);
 		if (!gtlbe) {
 			/* The guest didn't have a mapping for it. */
+			if ((eaddr >> 30) == 0xc) {
+				printk("Guest mapping f0r 0xc not found!\n");
+				kvmppc_dump_vcpu(vcpu);
+				kvmppc_dump_tlb(vcpu);
+			}
+
 			kvmppc_queue_exception(vcpu, exit_nr);
 			vcpu->arch.dear = vcpu->arch.fault_dear;
 			vcpu->arch.esr = vcpu->arch.fault_esr;
@@ -354,6 +361,12 @@ int kvmppc_handle_exit(struct kvm_run *r
 		gtlbe = kvmppc_44x_itlb_search(vcpu, eaddr);
 		if (!gtlbe) {
 			/* The guest didn't have a mapping for it. */
+			if ((eaddr >> 30) == 0xc) {
+				printk("Guest mapping f0r 0xc not found!\n");
+				kvmppc_dump_vcpu(vcpu);
+				kvmppc_dump_tlb(vcpu);
+			}
+
 			kvmppc_queue_exception(vcpu, exit_nr);
 			r = RESUME_GUEST;
 			break;

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

_______________________________________________
kvm-ppc-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-ppc-devel

#3 lore raw
In reply to Christian Ehrhardt
Quoted 10 lines · Christian Ehrhardt

ok for Hollis to continue and anyone else that might have comments -> tracking any eviction of the guest kernel mapping (=0xC..) did not trigger
-> tracking the delivery of a itlb/dtlb miss with that address did not trigger
-> I inserted a BUG statement in the guests report of "Unable to handle kernel paging request ..."
That triggers a emulation of a trap in the host and there I have a
dump_vcpu&dump_tlb

I think this just means our debug tests in the host kernel aren't working. The *only* way the guest interrupt vectors are invoked is by the host kernel...

Signature · 12 lines
Hollis Blanchard
IBM Linux Technology Center


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
kvm-ppc-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-ppc-devel
#4 lore raw
In reply to Christian Ehrhardt
Quoted 11 lines · Hollis Blanchard
Quoted 10 lines · Christian Ehrhardt

ok for Hollis to continue and anyone else that might have comments -> tracking any eviction of the guest kernel mapping (=0xC..) did not trigger
-> tracking the delivery of a itlb/dtlb miss with that address did not trigger
-> I inserted a BUG statement in the guests report of "Unable to handle kernel paging request ..."
That triggers a emulation of a trap in the host and there I have a
dump_vcpu&dump_tlb

Quoted 4 lines

I think this just means our debug tests in the host kernel aren't working. The *only* way the guest interrupt vectors are invoked is by the host kernel...

right - I checked and changed the eaddr matching and now get useful triggers.
And you now can see the initial programming of the Kernel mapping done by the guest (3x tlbwe). I add the output here for reference, but unfortunately now that I have triggers that are known to work the actual issue does no more occur :-(. I run into some soft lockup all the time now (12/12 tests). Maybe we need to fix that first (if is not related anyway)?
I attach an updated version of the debug patch for the host kernel.

I only post vcpu (all zero gpr lines removed) and tlb dump, not the stack trace because this tlb programming is expected:

kvmppc_emul_tlbwe - writing or evicting guest kernel mapping 63: tid 00000000 w0 00000000 w1 00000000 w2 00000000,

inst: ra 00000000 rs 00000003 ws 00000000
guest address: 0x0
pc:   000000a4 msr:  00000040
lr:   0000003c ctr:  00000000
srr0: 00000000 srr1: 00000000
dear: 00000000 esr:  00000000
exceptions: 00000000
gpr00: 0000003f 00fffff8 00000000 c0000290
gpr04: 00000000 00000107 00000000 00000000
gpr28: 00000000 00000000 00000000 00263100
vcpu 0 TLB dump:
| nr |     tid  |    word0 |    word1 |    word2 |
G 0 | 00000000 | 00000270 | 00000000 | 00000007 |
S42 | 00000000 | 00000310 | 0E622000 | 0000003F |
[...]
kvmppc_emul_tlbwe - writing or evicting guest kernel mapping
63: tid 00000000 w0 c0000290 w1 00000000 w2 00000000,
inst: ra 00000000 rs 00000004 ws 00000001
pc:   000000a8 msr:  00000040
lr:   0000003c ctr:  00000000
srr0: 00000000 srr1: 00000000
dear: 00000000 esr:  00000000
exceptions: 00000000
gpr00: 0000003f 00fffff8 00000000 c0000290
gpr04: 00000000 00000107 00000000 00000000
gpr28: 00000000 00000000 00000000 00263100
vcpu 0 TLB dump:
| nr |     tid  |    word0 |    word1 |    word2 |
G 0 | 00000000 | 00000270 | 00000000 | 00000007 |
G63 | 00000000 | C0000290 | 00000000 | 00000000 |
S42 | 00000000 | 00000310 | 0E622000 | 0000003F |
S43 | 00000000 | C0000310 | 0E622000 | 00000007 |
[...]
kvmppc_emul_tlbwe - writing or evicting guest kernel mapping
63: tid 00000000 w0 c0000290 w1 00000000 w2 00000000,
inst: ra 00000000 rs 00000005 ws 00000002
pc:   000000ac msr:  00000040
lr:   0000003c ctr:  00000000
srr0: 00000000 srr1: 00000000
dear: 00000000 esr:  00000000
exceptions: 00000000
gpr00: 0000003f 00fffff8 00000000 c0000290
gpr04: 00000000 00000107 00000000 00000000
gpr28: 00000000 00000000 00000000 00263100
vcpu 0 TLB dump:
| nr |     tid  |    word0 |    word1 |    word2 |
G 0 | 00000000 | 00000270 | 00000000 | 00000007 |
G63 | 00000000 | C0000290 | 00000000 | 00000000 |
S42 | 00000000 | 00000310 | 0E622000 | 0000003F |
S44 | 00000000 | C0000310 | 0E622000 | 00000007 |

And later only the soft lockup in 12/12 guest test runs (without that debug code, even with the non triggering one, at least 1/3 triggered the tlb issue). Heisenberg affects bugs as it does with quantum effects ;-)

BUG: soft lockup - CPU#0 stuck for 61s! [modprobe:677]
NIP: c0000ac0 LR: 48008904 CTR: 00001fe6
REGS: bf896750 TRAP: 0901   Not tainted  (2.6.25-rc3)
MSR: 00021000 <ME>  CR: 44004028  XER: 20000000

TASK = c89de000[677] 'modprobe' THREAD: c89b0000
GPR00: 00000003 bf896800 00000000 0feb86a2 0feb5d45 00000073 0000005f feff0000
GPR08: 00000000 00000000 00000000 0fea919c 42000024
NIP [c0000ac0] Decrementer+0x0/0xc0
LR [48008904] 0x48008904

Call Trace:

Instruction dump:
914b00b0 3d400002 614a1002 512a0420 4800c471 c000ba60 c000d6e0 60000000 60000000 60000000 60000000 60000000 <7d5043a6> 7d7143a6 7c3443a6 7d400026

Signature · 2 lines
Grüsse / regards, Christian Ehrhardt
IBM Linux Technology Center, Open Virtualization
arch/powerpc/kvm/44x_tlb.c +31 -0
@@ -21,6 +21,7 @@
 #include <linux/string.h>
 #include <linux/kvm_host.h>
 #include <linux/highmem.h>
+#include <linux/delay.h>
 #include <asm/mmu-44x.h>
 
 #include "44x_tlb.h"
@@ -29,6 +30,36 @@
 #define PPC44x_TLB_SUPER_PERM_MASK (PPC44x_TLB_SX|PPC44x_TLB_SR|PPC44x_TLB_SW)
 
 static unsigned int kvmppc_tlb_44x_pos;
+
+void kvmppc_dump_tlb(struct kvm_vcpu *vcpu)
+{
+	struct tlbe *tlbe;
+	int i;
+
+	printk("vcpu %d TLB dump:\n", vcpu->vcpu_id);
+	printk("| %2s | %8s | %8s | %8s | %8s |\n",
+		"nr", " tid ", "word0", "word1", "word2");
+
+	for (i = 0; i < PPC44x_TLB_SIZE; i++)
+	{
+		tlbe = &vcpu->arch.guest_tlb[i];
+		if (tlbe->word0 & PPC44x_TLB_VALID)
+			printk("G%2d | %08X | %08X | %08X | %08X |\n",
+			i, tlbe->tid, tlbe->word0, tlbe->word1,	tlbe->word2);
+	}
+
+	msleep(500);
+
+	for (i = 0; i < PPC44x_TLB_SIZE; i++)
+	{
+		tlbe = &vcpu->arch.shadow_tlb[i];
+		if (tlbe->word0 & PPC44x_TLB_VALID)
+			printk("S%2d | %08X | %08X | %08X | %08X |\n",
+			i, tlbe->tid, tlbe->word0, tlbe->word1,	tlbe->word2);
+	}
+
+	msleep(500);
+}
 
 static u32 kvmppc_44x_tlb_shadow_attrib(u32 attrib, int usermode)
 {
arch/powerpc/kvm/44x_tlb.h +2 -0
@@ -27,6 +27,8 @@ extern int kvmppc_44x_tlb_index(struct k
                                 unsigned int pid, unsigned int as);
 extern struct tlbe *kvmppc_44x_dtlb_search(struct kvm_vcpu *vcpu, gva_t eaddr);
 extern struct tlbe *kvmppc_44x_itlb_search(struct kvm_vcpu *vcpu, gva_t eaddr);
+
+extern void kvmppc_dump_tlb(struct kvm_vcpu *vcpu);
 
 /* TLB helper functions */
 static inline unsigned int get_tlb_size(const struct tlbe *tlbe)
arch/powerpc/kvm/emulate.c +23 -0
@@ -21,6 +21,7 @@
 #include <linux/timer.h>
 #include <linux/types.h>
 #include <linux/string.h>
+#include <linux/delay.h>
 #include <linux/kvm_host.h>
 
 #include <asm/dcr.h>
@@ -129,6 +130,7 @@ static int kvmppc_emul_tlbwe(struct kvm_
 	if (index > PPC44x_TLB_SIZE) {
 		printk("%s: index %d\n", __func__, index);
 		kvmppc_dump_vcpu(vcpu);
+		kvmppc_dump_tlb(vcpu);
 		return EMULATE_FAIL;
 	}
 
@@ -138,6 +140,25 @@ static int kvmppc_emul_tlbwe(struct kvm_
 #endif
 
 	tlbe = &vcpu->arch.guest_tlb[index];
+
+	if (	((get_tlb_eaddr(tlbe) & 0xf0000000) == 0xc0000000)
+		 ||
+		((vcpu->arch.gpr[rs] & 0xf0000000) == 0xc0000000) ) {
+		printk("%s - writing or evicting guest kernel mapping"
+			"\n %02d: tid %08x w0 %08x w1 %08x w2 %08x,"
+			"\n inst: ra %08x rs %08x ws %08x \n",
+		__func__, index, 
+		tlbe->tid, tlbe->word0, tlbe->word1, tlbe->word2,
+		ra, rs, ws);
+		msleep(500);
+		kvmppc_dump_vcpu(vcpu);
+		msleep(500);
+		kvmppc_dump_tlb(vcpu);
+		msleep(500);
+		WARN_ON(1);
+		msleep(500);
+	}
+
 
 	/* Invalidate shadow mappings for the about-to-be-clobbered TLBE. */
 	if (tlbe->word0 & PPC44x_TLB_VALID) {
@@ -250,6 +271,8 @@ int kvmppc_emulate_instruction(struct kv
 	switch (get_op(inst)) {
 	case 3:                                                 /* trap */
 		printk("trap!\n");
+		kvmppc_dump_vcpu(vcpu);
+		kvmppc_dump_tlb(vcpu);
 		kvmppc_queue_exception(vcpu, BOOKE_INTERRUPT_PROGRAM);
 		advance = 0;
 		break;
arch/powerpc/kvm/powerpc.c +15 -1
@@ -70,6 +70,7 @@ void kvmppc_dump_vcpu(struct kvm_vcpu *v
 	printk("pc:   %08x msr:  %08x\n", vcpu->arch.pc, vcpu->arch.msr);
 	printk("lr:   %08x ctr:  %08x\n", vcpu->arch.lr, vcpu->arch.ctr);
 	printk("srr0: %08x srr1: %08x\n", vcpu->arch.srr0, vcpu->arch.srr1);
+	printk("dear: %08x esr:  %08x\n", vcpu->arch.dear, vcpu->arch.esr);
 
 	printk("exceptions: %08lx\n", vcpu->arch.pending_exceptions);
 
@@ -218,6 +219,7 @@ int kvmppc_handle_exit(struct kvm_run *r
 	case BOOKE_INTERRUPT_MACHINE_CHECK:
 		printk("MACHINE CHECK: %lx\n", mfspr(SPRN_MCSR));
 		kvmppc_dump_vcpu(vcpu);
+		kvmppc_dump_tlb(vcpu);
 		r = RESUME_HOST;
 		break;
 
@@ -309,7 +311,13 @@ int kvmppc_handle_exit(struct kvm_run *r
 		/* Check the guest TLB. */
 		gtlbe = kvmppc_44x_dtlb_search(vcpu, eaddr);
 		if (!gtlbe) {
-			/* The guest didn't have a mapping for it. */
+		/* The guest didn't have a mapping for it. */
+			if ((eaddr & 0xf0000000) == 0xc0000000) {
+				printk("DTLBMiss - Guest mapping f0r 0xc not found!\n");
+				kvmppc_dump_vcpu(vcpu);
+				kvmppc_dump_tlb(vcpu);
+			}
+
 			kvmppc_queue_exception(vcpu, exit_nr);
 			vcpu->arch.dear = vcpu->arch.fault_dear;
 			vcpu->arch.esr = vcpu->arch.fault_esr;
@@ -354,6 +362,12 @@ int kvmppc_handle_exit(struct kvm_run *r
 		gtlbe = kvmppc_44x_itlb_search(vcpu, eaddr);
 		if (!gtlbe) {
 			/* The guest didn't have a mapping for it. */
+			if ((eaddr & 0xf0000000) == 0xc0000000) {
+				printk("ITLBMiss - Guest mapping f0r 0xc not found!\n");
+				kvmppc_dump_vcpu(vcpu);
+				kvmppc_dump_tlb(vcpu);
+			}
+
 			kvmppc_queue_exception(vcpu, exit_nr);
 			r = RESUME_GUEST;
 			break;

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

_______________________________________________
kvm-ppc-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-ppc-devel

#5 lore raw
In reply to Christian Ehrhardt
Quoted 6 lines · Christian Ehrhardt

And later only the soft lockup in 12/12 guest test runs (without that debug code, even with the non triggering one, at least 1/3 triggered the tlb issue).
Heisenberg affects bugs as it does with quantum effects ;-)

To update this thread and keep it's completeness - with a bit luck I got the issue again while running with the debug code and can now confirm that we are not delivering a ITLB Miss for the guest kernel mapping prior to the issue.

Signature · 12 lines
Grüsse / regards, 
Christian Ehrhardt
IBM Linux Technology Center, Open Virtualization

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
kvm-ppc-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-ppc-devel