Guests oops when trying to mount initramfs
Subsystems (2)
Hi,
I already chatted with Hollis a bit about that, but wanted to ask you about it. I also put kvm-ppc-devel on cc because it seem to be a common issue. I as well as Hollis saw that kind of bug in the past occasionally but never in a reproducible way to be able to debug it.
So the first question is - is your scenario running into that bug to ~>95% ?
And if it reproducible in your case we can use it to find the reason for it, here a short summary from my past discussions about that issue:
- effectively what you see is that the guest has an itlb miss for it's kernel mapping.
- we only deliver that to the guest if we can't find that mapping in the guest tlb
- this should never happen because the guest should never remove it's own kernel mapping and therefor we should be able to cover all itlb misses in the host by using the guest tlb
- but it happens so the question is when&why is the guest kernel mapping removed from the guest tlb
- afaik we have only one place that might remove that mapping which is the function kvmppc_emul_tlbwe
- we also know the kernel mapping which is usually a single 16mb mapping we initialize for the guest in kvm_arch_vcpu_setup (I might be wrong here - Hollis?) => we could track that tlbwe that should never happen by inserting a warn_on into kvmppc_emul_tlbwe if it overwrites a guest tlb kernel mapping and starting from there debug whats going on
Any comments or other suggestions are welcome
Quoted 10 lines · Jerone Young
I'm looking into this. But got a guest to load without the device tree mainpulation code. I have an cuImage with a prefilled device tree built in & initramfs integrated.
It oopses at
Decrementor()Does not mount the initramfs
Here is the trace:
[...]
Quoted 28 lines
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
Freeing unused kernel memory: 1500k initUnable to handle kernel paging request for instruction fetch Faulting instruction address: 0xc0000ac0 Oops: Kernel access of bad area, sig: 1 [#1] Bamboo Modules linked in: NIP: c0000ac0 LR: 48003fdc CTR: 00000000 REGS: bfcfe910 TRAP: 0400 Not tainted (2.6.25-rc3-dirty) MSR: 00021000 <ME> CR: 42000028 XER: 00000000TASK = c8810400[1] 'init' THREAD:
c881e000
GPR00: 00000001bfcfe9c000000000480177b04801795c00000001 0000000048011f34
GPR08:0002d00248026e6400000000bfcfe9c0c881e00000000000 00000000 00000000
GPR16: 00000000 48018700 00000001c0240000c0240000bfcfea4800000007 10000034
GPR24: 000000006fffff404801700048027f4c00000000480177b0480272e4bfcfe9c0
NIP [c0000ac0]Decrementer+0x0/0xc0
LR [48003fdc]0x48003fdcCall Trace:Instruction dump:
914b00b03d400002614a1002512a04204800c6b5c000b378c000d92460000000 60000000 60000000 60000000 60000000 <7d5043a6>7d7143a67c3443a67d400026---[ end trace 59dad6c3a6e54fa0 ]---
Signature · 12 lines
Grüsse / regards, Christian Ehrhardt IBM Linux Technology Center, Open Virtualization ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ kvm-ppc-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/kvm-ppc-devel
Yup, this is definitely the right direction. I actually hadn't looked at tlbwe yet, but that's probably where the issue is.
I'm also looking at adding some "relay" debugfs code to export all of KVM's TLB operations to userspace for analysis. There are a *lot* of them, and printk is not suitable for this data. :) This will be of more general use down the road when we consider more sophisticated shadow TLB eviction policies (i.e. not round robin)...
Signature · 2 lines
Hollis Blanchard IBM Linux Technology Center
Quoted 21 lines · Christian Ehrhardt
Hi,
I already chatted with Hollis a bit about that, but wanted to ask you about it. I also put kvm-ppc-devel on cc because it seem to be a common issue. I as well as Hollis saw that kind of bug in the past occasionally but never in a reproducible way to be able to debug it.
So the first question is - is your scenario running into that bug to ~>95% ?And if it reproducible in your case we can use it to find the reason for it, here a short summary from my past discussions about that issue:
- effectively what you see is that the guest has an itlb miss for it's kernel mapping.
- we only deliver that to the guest if we can't find that mapping in the guest tlb
- this should never happen because the guest should never remove it's own kernel mapping and therefor we should be able to cover all itlb misses in the host by using the guest tlb
- but it happens so the question is when&why is the guest kernel mapping removed from the guest tlb
- afaik we have only one place that might remove that mapping which is the functionkvmppc_emul_tlbwe
- we also know the kernel mapping which is usually a single 16mb mapping we initialize for the guest inkvm_arch_vcpu_setup(I might be wrong here - Hollis?) => we could track that tlbwe that should never happen by inserting awarn_onintokvmppc_emul_tlbweif it overwrites a guest tlb kernel mapping and starting from there debug whats going onAny comments or other suggestions are welcome
Quoted 4 lines · Jerone Young
I'm looking into this. But got a guest to load without the device tree mainpulation code. I have an cuImage with a prefilled device tree built in & initramfs integrated.
Quoted 36 lines
Quoted 6 lines
It oopses at
Decrementor()Does not mount the initramfs
Here is the trace:
[...]
Quoted 28 lines
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
Freeing unused kernel memory: 1500k initUnable to handle kernel paging request for instruction fetch Faulting instruction address: 0xc0000ac0 Oops: Kernel access of bad area, sig: 1 [#1] Bamboo Modules linked in: NIP: c0000ac0 LR: 48003fdc CTR: 00000000 REGS: bfcfe910 TRAP: 0400 Not tainted (2.6.25-rc3-dirty) MSR: 00021000 <ME> CR: 42000028 XER: 00000000TASK = c8810400[1] 'init' THREAD:
c881e000
GPR00: 00000001bfcfe9c000000000480177b04801795c00000001 0000000048011f34
GPR08:0002d00248026e6400000000bfcfe9c0c881e00000000000 00000000 00000000
GPR16: 00000000 48018700 00000001c0240000c0240000bfcfea4800000007 10000034
GPR24: 000000006fffff404801700048027f4c00000000480177b0480272e4bfcfe9c0
NIP [c0000ac0]Decrementer+0x0/0xc0
LR [48003fdc]0x48003fdcCall Trace:Instruction dump:
914b00b03d400002614a1002512a04204800c6b5c000b378c000d92460000000 60000000 60000000 60000000 60000000 <7d5043a6>7d7143a67c3443a67d400026---[ end trace 59dad6c3a6e54fa0 ]---
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
kvm-ppc-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-ppc-devel
Christian Ehrhardt wrote:
[...]
Quoted 9 lines
And if it reproducible in your case we can use it to find the reason for it, here a short summary from my past discussions about that issue:
- effectively what you see is that the guest has an itlb miss for it's kernel mapping.
- we only deliver that to the guest if we can't find that mapping in the guest tlb
- this should never happen because the guest should never remove it's own kernel mapping and therefor we should be able to cover all itlb misses in the host by using the guest tlb
- but it happens so the question is when&why is the guest kernel mapping removed from the guest tlb
- afaik we have only one place that might remove that mapping which is the functionkvmppc_emul_tlbwe
- we also know the kernel mapping which is usually a single 16mb mapping we initialize for the guest inkvm_arch_vcpu_setup(I might be wrong here - Hollis?) => we could track that tlbwe that should never happen by inserting awarn_onintokvmppc_emul_tlbweif it overwrites a guest tlb kernel mapping and starting from there debug whats going on
ok for Hollis to continue and anyone else that might have comments
-> tracking any eviction of the guest kernel mapping (=0xC..) did not trigger
-> tracking the delivery of a itlb/dtlb miss with that address did not trigger
-> I inserted a BUG statement in the guests report of "Unable to handle kernel paging request ..."
That triggers a emulation of a trap in the host and there I have a dump_vcpu&dump_tlb
-> for the case that someone wants to reproduce that I attached debug patches for host and guest kernel
Here is the output when hitting the trap:
Looking up port of RPC 100005/1 on 192.168.1.2
VFS: Mounted root (nfs filesystem). Freeing unused kernel memory: 116k init
Unable to handle kernel paging request for instruction fetch Faulting instruction address: 0xc0000ac0 trap! pc: c000fcbc msr: 00021002 lr: c000fcbc ctr: c012dc54 srr0: c012d638 srr1: 00021002 exceptions: 00000000 gpr00: c000fcbc bff2aaf0 c8810400 0000002c gpr04: 00000001 00000001 00000000 00000004 gpr08: 00000001 c025ca00 00001250 c0260000 gpr12: 65930e5e 1001f2f8 00000000 00000000 gpr16: 00000000 00000000 00000000 c01f0000 gpr20: c0240000 c0240000 00000000 00000000 gpr24: c0240000 c01f0000 00000001 48026e60 gpr28: 48027a10 bff2af04 00000001 bff2ab10 vcpu 0 TLB dump: | nr | tid | word0 | word1 | word2 | G 1 | 00000001 | 0FF3F210 | 08FAB000 | 08FA016D | G 2 | 00000001 | 0FEAD210 | 08FDC000 | 08FD0149 | G 3 | 00000001 | 48026210 | 08FFB000 | 08FF0349 | G 5 | 00000001 | 10003210 | 0022D000 | 0022016D | G 6 | 00000000 | D1012210 | EF600000 | EF600703 | G 7 | 00000001 | 4800C210 | 003D7000 | 003D016D | G 8 | 00000001 | 10000210 | 00227000 | 00220149 | G 9 | 00000001 | 0FFE8210 | 08FE6000 | 08FE0349 | G11 | 00000001 | 0FEC5210 | 08FD4000 | 08FD016D | G13 | 00000001 | 0FFEA210 | 08FCD000 | 08FC036F | G14 | 00000001 | 0FFEC210 | 08FE5000 | 08FE035B | G15 | 00000001 | 0FFEF210 | 08FDE000 | 08FD035B | G16 | 00000000 | D1021210 | E8001000 | E8000703 | G18 | 00000001 | 0FF70210 | 003EB000 | 003E016D | G20 | 00000001 | 0FF20210 | 003F8000 | 003F016D | G22 | 00000001 | 0FF1F210 | 003F9000 | 003F016D | G23 | 00000001 | 0FFE7210 | 08FCB000 | 08FC0349 | G25 | 00000001 | 0FF17210 | 08F85000 | 08F8016D | G26 | 00000001 | 48013210 | 00235000 | 0023016D | G28 | 00000001 | 10001210 | 00228000 | 0022016D | G31 | 00000001 | 10017210 | 00242000 | 0024037F | G32 | 00000001 | 48012210 | 00234000 | 0023016D | G33 | 00000001 | 0FEA9210 | 003DC000 | 003D0149 | G34 | 00000001 | 0FEB3210 | 08FEB000 | 08FE0149 | G35 | 00000001 | 0FEB9210 | 08FEC000 | 08FE0149 | G36 | 00000001 | 0FEBA210 | 08FC2000 | 08FC0149 | G37 | 00000001 | 48009210 | 08FFA000 | 08FF016D | G39 | 00000001 | 0FEDD210 | 08F8A000 | 08F8016D | G40 | 00000001 | 0FFED210 | 08FCA000 | 08FC035B | G42 | 00000001 | 10005210 | 0022F000 | 0022016D | G43 | 00000001 | 10006210 | 00230000 | 0023016D | G45 | 00000001 | 0FEDA210 | 08F8D000 | 08F8016D | G46 | 00000001 | 4801C210 | 08FC9000 | 08FC035B | G47 | 00000001 | 0FEB4210 | 08FC1000 | 08FC0149 | G48 | 00000001 | 0FEAB210 | 08FD7000 | 08FD0149 | G49 | 00000001 | 0FEB0210 | 08FD9000 | 08FD0149 | G50 | 00000001 | 0FEB7210 | 08FEE000 | 08FE0149 | G51 | 00000001 | 0FEAF210 | 08FDA000 | 08FD0149 | G52 | 00000001 | 0FEB6210 | 08FEF000 | 08FE0149 | G53 | 00000001 | 0FEAA210 | 003DB000 | 003D0149 | G55 | 00000001 | 0FF61210 | 08F97000 | 08F9016D | G56 | 00000001 | 0FEB2210 | 08FE7000 | 08FE0149 | G57 | 00000001 | 0FEB8210 | 08FED000 | 08FE0149 | G58 | 00000001 | 0FEAC210 | 08FDD000 | 08FD0149 | G59 | 00000001 | 0FEB5210 | 08FC0000 | 08FC0149 | G60 | 00000001 | 48008210 | 08FFC000 | 08FF016D | G61 | 00000001 | 48027210 | 00241000 | 0024037F | G62 | 00000001 | BFF2A210 | 00243000 | 0024035B | G63 | 00000000 | C0000290 | 00000000 | 00000107 | S17 | 00000001 | 10003310 | 0E3B8000 | 0000002F | S18 | 00000001 | BFF2A310 | 0E3CE000 | 0000001F | S19 | 00000001 | 10017310 | 0E3CD000 | 0000003F | S20 | 00000000 | C0000310 | 0E38B000 | 0000003F | S21 | 00000000 | C0246310 | 0E3D1000 | 0000003F | S22 | 00000000 | C000C310 | 0E3E7000 | 0000003F | S23 | 00000000 | C8810310 | 0E9BA000 | 0000003F | S24 | 00000000 | C000D310 | 0E3E8000 | 0000003F | S25 | 00000000 | C000F310 | 0E3EA000 | 0000003F | S26 | 00000000 | C0264310 | 0DC31000 | 0000003F | S27 | 00000000 | C0037310 | 0DCF2000 | 0000003F | S28 | 00000000 | C0106310 | 0E9D1000 | 0000003F | S29 | 00000000 | C0223310 | 0E3AE000 | 0000003F | S30 | 00000000 | C0047310 | 0DD02000 | 0000003F | S31 | 00000000 | C024A310 | 0E3D5000 | 0000003F | S32 | 00000000 | C0024310 | 0DC1F000 | 0000003F | S33 | 00000000 | C0248310 | 0E3D3000 | 0000003F | S34 | 00000000 | C010D310 | 0E9D8000 | 0000003F | S35 | 00000000 | C010C310 | 0E9D7000 | 0000003F | S36 | 00000000 | C01F6310 | 0E201000 | 0000003F | S37 | 00000000 | C026B310 | 0DC38000 | 0000003F | S38 | 00000000 | C026C310 | 0DC39000 | 0000003F | S39 | 00000000 | C0023310 | 0DC1E000 | 0000003F | S40 | 00000000 | C0257310 | 0DC22000 | 0000003F | S41 | 00000000 | C012D310 | 0E0B8000 | 0000003F | S42 | 00000000 | C004A310 | 0DD05000 | 0000003F | S43 | 00000000 | C001E310 | 0DC19000 | 0000003F | S44 | 00000000 | C001D310 | 0DC18000 | 0000003F | S45 | 00000000 | C000A310 | 0E3E5000 | 0000003F | S46 | 00000000 | C0262310 | 0DC2D000 | 0000003F | S47 | 00000000 | C027C310 | 0DC69000 | 0000003F | S48 | 00000000 | C01D2310 | 0E13D000 | 0000003F | S49 | 00000000 | C8835310 | 0E06B000 | 0000003F | S50 | 00000000 | C0001310 | 0E38C000 | 0000003F | S51 | 00000000 | C025C310 | 0DC27000 | 0000003F | S52 | 00000000 | C027D310 | 0DC6A000 | 0000003F | S53 | 00000000 | C012A310 | 0E0B5000 | 0000003F | S54 | 00000000 | C0130310 | 0E0BB000 | 0000003F | S55 | 00000000 | C01D0310 | 0E13B000 | 0000003F | S56 | 00000000 | C0256310 | 0DC21000 | 0000003F | Oops: Exception in kernel mode, sig: 4 [#1] Bamboo Modules linked in: NIP: c000fcbc LR: c000fcbc CTR: c012dc54 REGS: bff2aa40 TRAP: 0700 Not tainted (2.6.25-rc3) MSR: 00021002 <ME> CR: 24002022 XER: 00000000
TASK = c8810400[1] 'init' THREAD: c881e000
GPR00: c000fcbc bff2aaf0 c8810400 0000002c 00000001 00000001 00000000 00000004
GPR08: 00000001 c025ca00 00001250 c0260000 65930e5e 1001f2f8 00000000 00000000
GPR16: 00000000 00000000 00000000 c01f0000 c0240000 c0240000 00000000 00000000
GPR24: c0240000 c01f0000 00000001 48026e60 48027a10 bff2af04 00000001 bff2ab10 NIP [c000fcbc] bad_page_fault+0x7c/0xb4
LR [c000fcbc] bad_page_fault+0x7c/0xb4
Call Trace: Instruction dump: 2b800380 419d0034 2f800300 40be003c 3c60c01f 809f00a4 38636580 48015271 809f0080 3c60c01f 3863664c 48015261 <0fe00000> 48000000 2f800400 419e001c ---[ end trace 9c05eabdb79d9d2c ]--- Kernel panic - not syncing: Attempted to kill init!
Rebooting in 180 seconds..
Signature · 2 lines
Grüsse / regards, Christian Ehrhardt IBM Linux Technology Center, Open Virtualization
arch/powerpc/mm/fault.c +3 -0
@@ -432,5 +432,8 @@ void bad_page_fault(struct pt_regs *regsprintk (KERN_ALERT "Faulting instruction address: 0x%08lx\n" , regs->nip); +// to trigger kvmppc host trap! +BUG (); +die ("Kernel access of bad area" , regs, sig); }
arch/powerpc/kvm/44x_tlb.c +31 -0
@@ -21,6 +21,7 @@#include <linux/string.h> #include <linux/kvm_host.h> #include <linux/highmem.h> +#include <linux/delay.h> #include <asm/mmu-44x.h> #include "44x_tlb.h"
@@ -29,6 +30,36 @@#define PPC44x_TLB_SUPER_PERM_MASK (PPC44x_TLB_SX|PPC44x_TLB_SR|PPC44x_TLB_SW)static unsigned int kvmppc_tlb_44x_pos; + +void kvmppc_dump_tlb (struct kvm_vcpu *vcpu) +{ +struct tlbe *tlbe; +int i; + +printk ("vcpu %d TLB dump:\n" , vcpu->vcpu_id); +printk ("| %2s | %8s | %8s | %8s | %8s |\n" , +"nr" ," tid " ,"word0" ,"word1" ,"word2" ); + +for (i =0 ; i < PPC44x_TLB_SIZE; i++) + { + tlbe = &vcpu->arch.guest_tlb[i]; +if (tlbe->word0 & PPC44x_TLB_VALID) +printk ("G%2d | %08X | %08X | %08X | %08X |\n" , + i, tlbe->tid, tlbe->word0, tlbe->word1, tlbe->word2); + } + +msleep (500 ); + +for (i =0 ; i < PPC44x_TLB_SIZE; i++) + { + tlbe = &vcpu->arch.shadow_tlb[i]; +if (tlbe->word0 & PPC44x_TLB_VALID) +printk ("S%2d | %08X | %08X | %08X | %08X |\n" , + i, tlbe->tid, tlbe->word0, tlbe->word1, tlbe->word2); + } + +msleep (500 ); +}static u32 kvmppc_44x_tlb_shadow_attrib (u32 attrib,int usermode) {
arch/powerpc/kvm/44x_tlb.h +2 -0
@@ -27,6 +27,8 @@ extern int kvmppc_44x_tlb_index(struct k unsigned int pid, unsigned int as);extern struct tlbe *kvmppc_44x_dtlb_search (struct kvm_vcpu *vcpu,gva_t eaddr);extern struct tlbe *kvmppc_44x_itlb_search (struct kvm_vcpu *vcpu,gva_t eaddr); + +extern void kvmppc_dump_tlb (struct kvm_vcpu *vcpu);/* TLB helper functions */ static inline unsigned int get_tlb_size (const struct tlbe *tlbe)
arch/powerpc/kvm/emulate.c +11 -0
@@ -129,6 +129,7 @@ static int kvmppc_emul_tlbwe(struct kvm_if (index > PPC44x_TLB_SIZE) {printk ("%s: index %d\n" , __func__, index);kvmppc_dump_vcpu (vcpu); +kvmppc_dump_tlb (vcpu);return EMULATE_FAIL ; }
@@ -138,6 +139,14 @@ static int kvmppc_emul_tlbwe(struct kvm_#endif tlbe = &vcpu->arch.guest_tlb[index]; + +if ((get_tlb_eaddr (tlbe) >>30 ) ==0xc ) { +printk ("evicting %02d: %08x %08x %08x %08x\n" , index, + tlbe->tid, tlbe->word0, tlbe->word1, tlbe->word2); +kvmppc_dump_vcpu (vcpu); +WARN_ON (1 ); + } +/* Invalidate shadow mappings for the about-to-be-clobbered TLBE. */ if (tlbe->word0 & PPC44x_TLB_VALID) {
@@ -250,6 +259,8 @@ int kvmppc_emulate_instruction(struct kvswitch (get_op (inst)) {case 3 :/* trap */ printk ("trap!\n" ); +kvmppc_dump_vcpu (vcpu); +kvmppc_dump_tlb (vcpu);kvmppc_queue_exception (vcpu,BOOKE_INTERRUPT_PROGRAM ); advance =0 ;break ;
arch/powerpc/kvm/powerpc.c +13 -0
@@ -218,6 +218,7 @@ int kvmppc_handle_exit(struct kvm_run *rcase BOOKE_INTERRUPT_MACHINE_CHECK :printk ("MACHINE CHECK: %lx\n" ,mfspr (SPRN_MCSR ));kvmppc_dump_vcpu (vcpu); +kvmppc_dump_tlb (vcpu); r =RESUME_HOST ;break ;
@@ -310,6 +311,12 @@ int kvmppc_handle_exit(struct kvm_run *r gtlbe =kvmppc_44x_dtlb_search (vcpu, eaddr);if (!gtlbe) {/* The guest didn't have a mapping for it. */ +if ((eaddr >>30 ) ==0xc ) { +printk ("Guest mapping f0r 0xc not found!\n" ); +kvmppc_dump_vcpu (vcpu); +kvmppc_dump_tlb (vcpu); + } +kvmppc_queue_exception (vcpu, exit_nr); vcpu->arch.dear = vcpu->arch.fault_dear; vcpu->arch.esr = vcpu->arch.fault_esr;
@@ -354,6 +361,12 @@ int kvmppc_handle_exit(struct kvm_run *r gtlbe =kvmppc_44x_itlb_search (vcpu, eaddr);if (!gtlbe) {/* The guest didn't have a mapping for it. */ +if ((eaddr >>30 ) ==0xc ) { +printk ("Guest mapping f0r 0xc not found!\n" ); +kvmppc_dump_vcpu (vcpu); +kvmppc_dump_tlb (vcpu); + } +kvmppc_queue_exception (vcpu, exit_nr); r =RESUME_GUEST ;break ;
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
kvm-ppc-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-ppc-devel
Quoted 10 lines · Christian Ehrhardt
ok for Hollis to continue and anyone else that might have comments -> tracking any eviction of the guest kernel mapping (=0xC..) did not trigger
-> tracking the delivery of a itlb/dtlb miss with that address did not trigger
-> I inserted a BUG statement in the guests report of "Unable to handle kernel paging request ..."
That triggers a emulation of a trap in the host and there I have a
dump_vcpu&dump_tlb
I think this just means our debug tests in the host kernel aren't working. The *only* way the guest interrupt vectors are invoked is by the host kernel...
Signature · 12 lines
Hollis Blanchard IBM Linux Technology Center ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ kvm-ppc-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/kvm-ppc-devel
Quoted 11 lines · Hollis Blanchard
Quoted 10 lines · Christian Ehrhardt
ok for Hollis to continue and anyone else that might have comments -> tracking any eviction of the guest kernel mapping (=0xC..) did not trigger
-> tracking the delivery of a itlb/dtlb miss with that address did not trigger
-> I inserted a BUG statement in the guests report of "Unable to handle kernel paging request ..."
That triggers a emulation of a trap in the host and there I have a
dump_vcpu&dump_tlb
Quoted 4 lines
I think this just means our debug tests in the host kernel aren't working. The *only* way the guest interrupt vectors are invoked is by the host kernel...
right - I checked and changed the eaddr matching and now get useful triggers.
And you now can see the initial programming of the Kernel mapping done by the guest (3x tlbwe). I add the output here for reference, but unfortunately now that I have triggers that are known to work the actual issue does no more occur :-(. I run into some soft lockup all the time now (12/12 tests). Maybe we need to fix that first (if is not related anyway)?
I attach an updated version of the debug patch for the host kernel.
I only post vcpu (all zero gpr lines removed) and tlb dump, not the stack trace because this tlb programming is expected:
kvmppc_emul_tlbwe - writing or evicting guest kernel mapping 63: tid 00000000 w0 00000000 w1 00000000 w2 00000000,
inst: ra 00000000 rs 00000003 ws 00000000 guest address: 0x0 pc: 000000a4 msr: 00000040 lr: 0000003c ctr: 00000000 srr0: 00000000 srr1: 00000000 dear: 00000000 esr: 00000000 exceptions: 00000000 gpr00: 0000003f 00fffff8 00000000 c0000290 gpr04: 00000000 00000107 00000000 00000000 gpr28: 00000000 00000000 00000000 00263100 vcpu 0 TLB dump: | nr | tid | word0 | word1 | word2 | G 0 | 00000000 | 00000270 | 00000000 | 00000007 | S42 | 00000000 | 00000310 | 0E622000 | 0000003F | [...] kvmppc_emul_tlbwe - writing or evicting guest kernel mapping 63: tid 00000000 w0 c0000290 w1 00000000 w2 00000000, inst: ra 00000000 rs 00000004 ws 00000001 pc: 000000a8 msr: 00000040 lr: 0000003c ctr: 00000000 srr0: 00000000 srr1: 00000000 dear: 00000000 esr: 00000000 exceptions: 00000000 gpr00: 0000003f 00fffff8 00000000 c0000290 gpr04: 00000000 00000107 00000000 00000000 gpr28: 00000000 00000000 00000000 00263100 vcpu 0 TLB dump: | nr | tid | word0 | word1 | word2 | G 0 | 00000000 | 00000270 | 00000000 | 00000007 | G63 | 00000000 | C0000290 | 00000000 | 00000000 | S42 | 00000000 | 00000310 | 0E622000 | 0000003F | S43 | 00000000 | C0000310 | 0E622000 | 00000007 | [...] kvmppc_emul_tlbwe - writing or evicting guest kernel mapping 63: tid 00000000 w0 c0000290 w1 00000000 w2 00000000, inst: ra 00000000 rs 00000005 ws 00000002 pc: 000000ac msr: 00000040 lr: 0000003c ctr: 00000000 srr0: 00000000 srr1: 00000000 dear: 00000000 esr: 00000000 exceptions: 00000000 gpr00: 0000003f 00fffff8 00000000 c0000290 gpr04: 00000000 00000107 00000000 00000000 gpr28: 00000000 00000000 00000000 00263100 vcpu 0 TLB dump: | nr | tid | word0 | word1 | word2 | G 0 | 00000000 | 00000270 | 00000000 | 00000007 | G63 | 00000000 | C0000290 | 00000000 | 00000000 | S42 | 00000000 | 00000310 | 0E622000 | 0000003F | S44 | 00000000 | C0000310 | 0E622000 | 00000007 |
And later only the soft lockup in 12/12 guest test runs (without that debug code, even with the non triggering one, at least 1/3 triggered the tlb issue). Heisenberg affects bugs as it does with quantum effects ;-)
BUG: soft lockup - CPU#0 stuck for 61s! [modprobe:677] NIP: c0000ac0 LR: 48008904 CTR: 00001fe6 REGS: bf896750 TRAP: 0901 Not tainted (2.6.25-rc3) MSR: 00021000 <ME> CR: 44004028 XER: 20000000
TASK = c89de000[677] 'modprobe' THREAD: c89b0000
GPR00: 00000003 bf896800 00000000 0feb86a2 0feb5d45 00000073 0000005f feff0000
GPR08: 00000000 00000000 00000000 0fea919c 42000024
NIP [c0000ac0] Decrementer+0x0/0xc0
LR [48008904] 0x48008904
Call Trace:
Instruction dump:914b00b0 3d400002 614a1002 512a0420 4800c471 c000ba60 c000d6e0 60000000 60000000 60000000 60000000 60000000 <7d5043a6> 7d7143a6 7c3443a6 7d400026
Signature · 2 lines
Grüsse / regards, Christian Ehrhardt IBM Linux Technology Center, Open Virtualization
arch/powerpc/kvm/44x_tlb.c +31 -0
@@ -21,6 +21,7 @@#include <linux/string.h> #include <linux/kvm_host.h> #include <linux/highmem.h> +#include <linux/delay.h> #include <asm/mmu-44x.h> #include "44x_tlb.h"
@@ -29,6 +30,36 @@#define PPC44x_TLB_SUPER_PERM_MASK (PPC44x_TLB_SX|PPC44x_TLB_SR|PPC44x_TLB_SW)static unsigned int kvmppc_tlb_44x_pos; + +void kvmppc_dump_tlb (struct kvm_vcpu *vcpu) +{ +struct tlbe *tlbe; +int i; + +printk ("vcpu %d TLB dump:\n" , vcpu->vcpu_id); +printk ("| %2s | %8s | %8s | %8s | %8s |\n" , +"nr" ," tid " ,"word0" ,"word1" ,"word2" ); + +for (i =0 ; i < PPC44x_TLB_SIZE; i++) + { + tlbe = &vcpu->arch.guest_tlb[i]; +if (tlbe->word0 & PPC44x_TLB_VALID) +printk ("G%2d | %08X | %08X | %08X | %08X |\n" , + i, tlbe->tid, tlbe->word0, tlbe->word1, tlbe->word2); + } + +msleep (500 ); + +for (i =0 ; i < PPC44x_TLB_SIZE; i++) + { + tlbe = &vcpu->arch.shadow_tlb[i]; +if (tlbe->word0 & PPC44x_TLB_VALID) +printk ("S%2d | %08X | %08X | %08X | %08X |\n" , + i, tlbe->tid, tlbe->word0, tlbe->word1, tlbe->word2); + } + +msleep (500 ); +}static u32 kvmppc_44x_tlb_shadow_attrib (u32 attrib,int usermode) {
arch/powerpc/kvm/44x_tlb.h +2 -0
@@ -27,6 +27,8 @@ extern int kvmppc_44x_tlb_index(struct k unsigned int pid, unsigned int as);extern struct tlbe *kvmppc_44x_dtlb_search (struct kvm_vcpu *vcpu,gva_t eaddr);extern struct tlbe *kvmppc_44x_itlb_search (struct kvm_vcpu *vcpu,gva_t eaddr); + +extern void kvmppc_dump_tlb (struct kvm_vcpu *vcpu);/* TLB helper functions */ static inline unsigned int get_tlb_size (const struct tlbe *tlbe)
arch/powerpc/kvm/emulate.c +23 -0
@@ -21,6 +21,7 @@#include <linux/timer.h> #include <linux/types.h> #include <linux/string.h> +#include <linux/delay.h> #include <linux/kvm_host.h> #include <asm/dcr.h>
@@ -129,6 +130,7 @@ static int kvmppc_emul_tlbwe(struct kvm_if (index > PPC44x_TLB_SIZE) {printk ("%s: index %d\n" , __func__, index);kvmppc_dump_vcpu (vcpu); +kvmppc_dump_tlb (vcpu);return EMULATE_FAIL ; }
@@ -138,6 +140,25 @@ static int kvmppc_emul_tlbwe(struct kvm_#endif tlbe = &vcpu->arch.guest_tlb[index]; + +if ( ((get_tlb_eaddr (tlbe) &0xf0000000 ) ==0xc0000000 ) + || + ((vcpu->arch.gpr[rs] &0xf0000000 ) ==0xc0000000 ) ) { +printk ("%s - writing or evicting guest kernel mapping" +"\n %02d: tid %08x w0 %08x w1 %08x w2 %08x," +"\n inst: ra %08x rs %08x ws %08x \n" , + __func__, index, + tlbe->tid, tlbe->word0, tlbe->word1, tlbe->word2, + ra, rs, ws); +msleep (500 ); +kvmppc_dump_vcpu (vcpu); +msleep (500 ); +kvmppc_dump_tlb (vcpu); +msleep (500 ); +WARN_ON (1 ); +msleep (500 ); + } +/* Invalidate shadow mappings for the about-to-be-clobbered TLBE. */ if (tlbe->word0 & PPC44x_TLB_VALID) {
@@ -250,6 +271,8 @@ int kvmppc_emulate_instruction(struct kvswitch (get_op (inst)) {case 3 :/* trap */ printk ("trap!\n" ); +kvmppc_dump_vcpu (vcpu); +kvmppc_dump_tlb (vcpu);kvmppc_queue_exception (vcpu,BOOKE_INTERRUPT_PROGRAM ); advance =0 ;break ;
arch/powerpc/kvm/powerpc.c +15 -1
@@ -70,6 +70,7 @@ void kvmppc_dump_vcpu(struct kvm_vcpu *vprintk ("pc: %08x msr: %08x\n" , vcpu->arch.pc, vcpu->arch.msr);printk ("lr: %08x ctr: %08x\n" , vcpu->arch.lr, vcpu->arch.ctr);printk ("srr0: %08x srr1: %08x\n" , vcpu->arch.srr0, vcpu->arch.srr1); +printk ("dear: %08x esr: %08x\n" , vcpu->arch.dear, vcpu->arch.esr);printk ("exceptions: %08lx\n" , vcpu->arch.pending_exceptions);
@@ -218,6 +219,7 @@ int kvmppc_handle_exit(struct kvm_run *rcase BOOKE_INTERRUPT_MACHINE_CHECK :printk ("MACHINE CHECK: %lx\n" ,mfspr (SPRN_MCSR ));kvmppc_dump_vcpu (vcpu); +kvmppc_dump_tlb (vcpu); r =RESUME_HOST ;break ;
@@ -309,7 +311,13 @@ int kvmppc_handle_exit(struct kvm_run *r/* Check the guest TLB. */ gtlbe =kvmppc_44x_dtlb_search (vcpu, eaddr);if (!gtlbe) { -/* The guest didn't have a mapping for it. */ +/* The guest didn't have a mapping for it. */ +if ((eaddr &0xf0000000 ) ==0xc0000000 ) { +printk ("DTLBMiss - Guest mapping f0r 0xc not found!\n" ); +kvmppc_dump_vcpu (vcpu); +kvmppc_dump_tlb (vcpu); + } +kvmppc_queue_exception (vcpu, exit_nr); vcpu->arch.dear = vcpu->arch.fault_dear; vcpu->arch.esr = vcpu->arch.fault_esr;
@@ -354,6 +362,12 @@ int kvmppc_handle_exit(struct kvm_run *r gtlbe =kvmppc_44x_itlb_search (vcpu, eaddr);if (!gtlbe) {/* The guest didn't have a mapping for it. */ +if ((eaddr &0xf0000000 ) ==0xc0000000 ) { +printk ("ITLBMiss - Guest mapping f0r 0xc not found!\n" ); +kvmppc_dump_vcpu (vcpu); +kvmppc_dump_tlb (vcpu); + } +kvmppc_queue_exception (vcpu, exit_nr); r =RESUME_GUEST ;break ;
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
kvm-ppc-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-ppc-devel
Quoted 6 lines · Christian Ehrhardt
And later only the soft lockup in 12/12 guest test runs (without that debug code, even with the non triggering one, at least 1/3 triggered the tlb issue).
Heisenberg affects bugs as it does with quantum effects ;-)
To update this thread and keep it's completeness - with a bit luck I got the issue again while running with the debug code and can now confirm that we are not delivering a ITLB Miss for the guest kernel mapping prior to the issue.
Signature · 12 lines
Grüsse / regards, Christian Ehrhardt IBM Linux Technology Center, Open Virtualization ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ kvm-ppc-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/kvm-ppc-devel