KVM internal error with kvm-pr on ppc64le
Hi guys,
I sent this email to the qemu-ppc ML, but in fact it's more a kvm-ppc issue, so I resend my email to the correct ML.
I think that some of you will received the email twice, sorry for that!
Signature · 88 lines
Loic Devulder - Senior QA Engineer Container & Storage Solutions Quality Assurance team (qa-css) SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton HRB, 21284 (AG Nuernberg) -------- Forwarded Message -------- Subject: Re: [Qemu-ppc] KVM internal error with kvm-pr on ppc64le Date: Thu, 31 May 2018 15:02:06 +0200 From: Loic Devulder <[email protected]> Organization: SUSE To: [email protected] CC: [email protected] Hi, After some test I find a way to workaround this using 2 graphics cards in the VM. Boot with this: # /usr/bin/qemu-system-ppc64 -serial mon:stdio -m 1024 -cpu host -device virtio-scsi-pci,id=scsi0 -device scsi-cd,drive�0,bus=scsi0.0 -drive media�rom,if=none,id�0,format=raw,file=/opt/openSUSE-Leap-42.3-NET-ppc64le-Build0130-Media.iso -smp 1,threads=1 -enable-kvm -device nec-usb-xhci -device usb-tablet -device usb-kbd -device virtio-vga -device virtio-gpu-pci -nodefaults And do this in the SLOF shell (be careful, order is important but I don't know why...): 0 > dev /pci/vga 0 > s" device_type" delete-property 0 > dev /pci/display 0 > s" display" device-type 0 > boot And now OS is booting using the 'virtio-gpu-pci' and not the 'virtio-vga', no crash anymore. I also tried the same but with '-vga std' instead of '-device virtio-vga', in that case I need to first add the 'device_type' property for 'virtio-gpuc-pci' and *after* remove it for the VGA card (again, I don't know why). But even with this I still have a 'KVM Internal error', a new one with a more weird suberror: "Suberror: -2130704892". Here the trace: Trying to load: from: /pci@800000020000000/scsi@1/disk@100000000000000 ... Successfully loaded SUSE Linux #1 SMP Fri Jun 3KVM internal error. Suberror: -2130704892 extra data[0]: 1e1e1e1e extra data[1]: 100000004 extra data[2]: 2fff0000 extra data[3]: 3000000002001000 extra data[4]: b000000002001033 extra data[5]: 42000000 extra data[6]: 8000000000002933 extra data[7]: c000000000fd0874 extra data[8]: 2fff0000 extra data[9]: c000000000030ed0 extra data[10]: 0 extra data[11]: 0 [...] extra data[8190]: 0 extra data[8191]: 0 NIP c0000000000264b0 LR d0000000020b7298 CTR 00000000000c0000 XER 0000000020000000 CPU#0 MSR b000000000009033 HID0 0000000000000000 HF 8000000000000000 iidx 3 didx 3 TB 00000000 00000000 DECR 00000000 GPR00 d0000000020b6ee4 c00000003a0fef80 d0000000020c9230 d000080080400000 GPR04 0000000000000000 0000000000300000 0800200bc00003a1 c00000003e8af1b0 GPR08 00000000000c0000 d000080080400000 0000000000000000 d0000000020bd818 GPR12 c000000000026440 c00000000fa00000 0000000000000020 d00000000211de66 GPR16 d00000000211e760 0000000000000002 c00000003a0ffdfc d000000002150000 GPR20 c00000003a5fb800 c00000003a0e7888 c00000003a9df2b8 0000000000000000 GPR24 c00000003a9df280 c00000003a0ff1f0 c00000003a0e79b8 0000000000000000 GPR28 c00000003a0e78a8 c00000003a0ff1f0 c00000003a0e7888 0000000000000000 CR 24228824 [ E G E E L L E G ] RES ffffffffffffffff FPR00 0000000000000000 0000000000000000 0000000000000000 0000000000000000 FPR04 0000000000000000 0000000000000000 414e564544006b63 45440034706f6f6c FPR08 554e514553006b73 4954494e495f4345 0000000000000000 0000000000000000 FPR12 0000000000000000 0000000000000000 0000000000000000 0000000000000000 FPR16 0000000000000000 0000000000000000 0000000000000000 0000000000000000 FPR20 0000000000000000 0000000000000000 0000000000000000 0000000000000000 FPR24 0000000000000000 0000000000000000 0000000000000000 0000000000000000 FPR28 0000000000000000 0000000000000000 0000000000000000 0000000000000000 FPSCR 0000000000000000 SRR0 c0000000000264b0 SRR1 b000000000009033 PVR 00000000004b0201 VRSAVE 0000000000000000 SPRG0 0000000000000000 SPRG1 c00000000fa00000 SPRG2 c00000000fa00000 SPRG3 0000000000000000 SPRG4 0000000000000000 SPRG5 0000000000000000 SPRG6 0000000000000000 SPRG7 0000000000000000 HSRR0 0000000000000000 HSRR1 0000000000000000 CFAR 0000000000000000 LPCR 0000000002000000 SDR1 0000000000000000 DAR d000080080400000 DSISR 0000000042000000 I don't know if this could be useful, but if someone else has the issue he can find a workaround now :)
Quoted 216 lines · Loic Devulder
Hi guys!
I tried to start a Linux CD (I tried openSUSE Leap 42.3) to install my ppc64le VM with kvm-pr (yes kvm-pr!) on a Power8 host and I have some issues...
My host system is an openSUSE Tumbleweed 20180525 with qemu 2.11 and kernel 4.16.11.
I tried different version of qemu without any success but I tried with openSUSE Leap 42.3 as a host (and qemu 2.11 and 2.12) with success. Kernel is 4.4 on Leap 42.3, and if I update the kernel to a more recent version I have the same issue! So it seems to be a kvm problem, not qemu.
I also tried without KVM and it works (but it is of course very slow!).
I tried to understand why and it seems to be related to the virtio-vga. Here the different tries I made with openSUSE Tumbleweed 20180525 as host.
- With only virtio-vga and boot/messages on VGA console:
/usr/bin/qemu-system-ppc64-m 1024 -cpu host -device virtio-scsi-pci,id=scsi0 -device scsi-cd,drive�0,bus=scsi0.0 -drive media�rom,if=none,id�0,format=raw,file=/opt/openSUSE-Leap-42.3-NET-ppc64le-Build0130-Media.iso -smp 1,threads=1 -enable-kvm -device nec-usb-xhci -device usb-tablet -device usb-kbd -serial stdio:mon -device virtio-vga -nodefaultsTrying to load: from: /pci@800000020000000/scsi@0/disk@100000000000000 ... Successfully loaded
SUSE Linux
#1 SMP Fri Jun 3KVM internal error. Suberror: 61568
NIPc0000000005359f4LRc0000000005356ccCTR 0000000000000032 XER 0000000000000000 CPU#0
MSRb000000000009033HID0 0000000000000000 HF 8000000000000000 iidx 3 didx 3 TB 00000000 00000000 DECR 00000000
GPR00 0000000000006400c00000003e49f440c000000000f3ba000000000000000190 GPR04d0000800803ce8000000000000000000 0000000000000000 0000000000000000 GPR08 0000000000000040d0000800803ce8000000000000000032 ffffffffffffffff GPR12c000000000535950c00000000fa00000c00000000000bf380000000000000000 GPR16 0000000000000064 0000000000000025 0000000000000064c0000000012d5a98GPR20 0000000000000000 0000000000000320 0000000000000000 0000000000000000 GPR24 0000000000000006 0000000000000008 0000000000000000 0000000000006400 GPR28c000000000535950c00000003e7198000000000000000000d0000800803ce800
CR 44022084 [ G G - E E - L G ] RES ffffffffffffffff
FPR00 0000000000000000 0000000000000000 0000000000000000 0000000000000000 FPR04 0000000000000000 0000000000000000 0000000000000000 0000000000000000 FPR08 0000000000000000 0000000000000000 0000000000000000 0000000000000000 FPR12 0000000000000000 0000000000000000 0000000000000000 0000000000000000 FPR16 0000000000000000 0000000000000000 0000000000000000 0000000000000000 FPR20 0000000000000000 0000000000000000 0000000000000000 0000000000000000 FPR24 0000000000000000 0000000000000000 0000000000000000 0000000000000000 FPR28 0000000000000000 0000000000000000 0000000000000000 0000000000000000 FPSCR 0000000000000000
SRR0c0000000005359f4SRR1b000000000009033PVR00000000004b0201VRSAVE 0000000000000000SPRG0 0000000000000000 SPRG1 c00000000fa00000 SPRG2 c00000000fa00000 SPRG3 0000000000000000 SPRG4 0000000000000000 SPRG5 0000000000000000 SPRG6 0000000000000000 SPRG7 0000000000000000HSRR0 0000000000000000 HSRR1 0000000000000000
CFAR 0000000000000000
LPCR 0000000002000000
SDR1 0000000000000000 DARd0000800803ce800DSISR 0000000042000000I have the "Loading kernel..." to "Booting Linux via
__start()..." on the VGA console and after the crash on the serial console. "Suberror: 61568" is weird...Same if I boot with "-vga std" or "-vga virtio" instead of "-device virtio-vga" (I know that "-vga virtio" and "-device virtio-vga" are the same).
With the help of Alexander Graf (thanks!) I was able to have some debug informations:
qemu-system-ppc-9944 [006] .... 83419.417302:kvm_book3s_reenter: reentry r=1 | pc=0xc000000000089930qemu-system-ppc-9944 [006] .... 83419.417304: kvm_exit: exit=PROGRAM | pc=0xc000000000009958 | msr=0xb000000000001031 | dar=0x7fff67356200 | srr1=0x800000000004d033 | last_inst=0x4c000024 qemu-system-ppc-9944 [006] .... 83419.417305: kvm_ppc_instr: inst 1275068452 pc 0xc0000000005359f4 emulate 0 qemu-system-ppc-9944 [006] .... 83419.417305: kvm_book3s_reenter: reentry r=1 | pc=0xc0000000005359f4 qemu-system-ppc-9944 [006] .... 83419.417307: kvm_exit: exit�TA_SEGMENT | pc=0xc0000000005359f4 | msr=0xb000000000009033 | dar=0xd0000800803ce800 | srr1=0x800000000000d033 | last_inst=0x4c000024 qemu-system-ppc-9944 [006] .... 83419.417307: kvm_book3s_slb_found: 18000033deaf8008 -> 61c0f87e5e qemu-system-ppc-9944 [006] .... 83419.417308: kvm_book3s_slbmte: 61c0f87e5e990, d000080088000003 qemu-system-ppc-9944 [006] .... 83419.417308: kvm_book3s_reenter: reentry r=0 | pc=0xc0000000005359f4 qemu-system-ppc-9944 [006] .... 83419.417310: kvm_exit: exit�TA_STORAGE | pc=0xc0000000005359f4 | msr=0xb000000000009033 | dar=0xd0000800803ce800 | srr1=0x800000000000d033 | last_inst=0xf8c90000 qemu-system-ppc-9944 [006] .... 83419.417315: kvm_book3s_slb_found: 18000033deaf8008 -> 61c0f87e5e qemu-system-ppc-9944 [006] .... 83419.417317: kvm_book3s_reenter: reentry r=2 | pc=0xc0000000005359f4 qemu-system-ppc-9944 [006] .... 83419.417319: kvm_userspace_exit: reason KVM_EXIT_INTERNAL_ERROR (17) qemu-system-ppc-9941 [004] .... 83419.447328: kvm_book3s_mmu_flush: Flush 5473 pPTEs: 200080000000 - 200081000000 gdbus-9946 [004] .... 83465.282674: kvm_book3s_mmu_flush: Flush 5473 PTEs: 0 - 0 gdbus-9946 [004] .... 83465.282684: kvm_book3s_mmu_invalidate: Flush: hvaac1b7cf37f7f0pfnv77e ea�0000002f7f0000 vp�1b217002f7f0 ra/7f0000 [6]
- With only virtio-vga and boot/messages on hvterm console (I removed the USB part to force the console output):
/usr/bin/qemu-system-ppc64-serial mon:stdio -m 1024 -cpu host -device virtio-scsi-pci,id=scsi0 -device scsi-cd,drive�0,bus=scsi0.0 -drive media�rom,if=none,id�0,format=raw,file=/opt/openSUSE-Leap-42.3-NET-ppc64le-Build0130-Media.iso -smp 1,threads=1 -enable-kvm -device virtio-vga -nodefaultsLoading kernel ...
Loading initial ramdisk ...
OF stdout device is: /vdevice/vty@71000000
Preparing to boot Linux version 4.4.74-1-default (geeko@buildhost) (gcc version 4.8.5 (SUSE Linux) ) #1 SMP Fri Jun 30 19:11:19 UTC 2017 (f0a86d9)
Detected machine type: 0000000000000101
Max number of cores passed to firmware: 2048 (NR_CPUS= 2048)
Calling ibm,client-architecture-support... done
command line: BOOT_IMAGE=/boot/ppc64le/linux
memory layout at init:
memory_limit: 0000000000000000 (16 MB aligned)
alloc_bottom: 0000000006380000
alloc_top: 0000000030000000
alloc_top_hi: 0000000040000000
rmo_top: 0000000030000000
ram_top: 0000000040000000
found display : /pci@800000020000000/vga@1, opening... done
instantiating rtas at0x000000002fff0000... doneprom_hold_cpus: skipped
copying OF device tree...
Building dt strings...
Building dt structure...
Device tree strings0x0000000006390000->0x0000000006390abe
Device tree struct0x00000000063a0000->0x00000000063b0000
Quiescing Open Firmware ...
Booting Linux via__start()...
Loading kernel ...
Loading initial ramdisk ...
OF stdout device is: /vdevice/vty@71000000
Preparing to boot Linux version 4.4.74-1-default (geeko@buildhost) (gcc version 4.8.5 (SUSE Linux) ) #1 SMP Fri Jun 30 19:11:19 UTC 2017 (f0a86d9)
Detected machine type: 0000000000000101
Max number of cores passed to firmware: 2048 (NR_CPUS= 2048)
Calling ibm,client-architecture-support... done
command line: BOOT_IMAGE=/boot/ppc64le/linux
[...]
io scheduler cfq registered (default)pci_hotplug: PCI Hot Plug PCI Core version: 0.5
pciehp: PCI Express Hot Plug Controller Driver version: 0.4
Using unsupported 800x600 vga at 200080800000, depth2, pitch200
KVM internal error. Suberror: 88
NIPc0000000005359f4LRc0000000005356ccCTR 0000000000000032 XER 0000000000000000 CPU#0
MSRb000000000009033HID0 0000000000000000 HF 8000000000000000 iidx 3 didx 3
TB 00000000 00000000 DECR 00000000
GPR00 0000000000006400c00000003e49f440c000000000f3ba000000000000000190
GPR04d0000800801ce8000000000000000000 0000000000000000 0000000000000000
GPR08 0000000000000040d0000800801ce8000000000000000032 ffffffffffffffff
GPR12c000000000535950c00000000fa00000c00000000000bf380000000000000000
GPR16 0000000000000064 0000000000000025 0000000000000064c0000000012d5a98
GPR20 0000000000000000 0000000000000320 0000000000000000 0000000000000000
GPR24 0000000000000006 0000000000000008 0000000000000000 0000000000006400
GPR28c000000000535950c00000003e7198000000000000000000d0000800801ce800
CR 44022084 [ G G - E E - L G ] RES ffffffffffffffff
FPR00 0000000000000000 0000000000000000 0000000000000000 0000000000000000
FPR04 0000000000000000 0000000000000000 0000000000000000 0000000000000000
FPR08 0000000000000000 0000000000000000 0000000000000000 0000000000000000
FPR12 0000000000000000 0000000000000000 0000000000000000 0000000000000000
FPR16 0000000000000000 0000000000000000 0000000000000000 0000000000000000
FPR20 0000000000000000 0000000000000000 0000000000000000 0000000000000000
FPR24 0000000000000000 0000000000000000 0000000000000000 0000000000000000
FPR28 0000000000000000 0000000000000000 0000000000000000 0000000000000000
FPSCR 0000000000000000
SRR0c0000000005359f4SRR1b000000000009033PVR00000000004b0201VRSAVE 0000000000000000SPRG0 0000000000000000 SPRG1 c00000000fa00000 SPRG2 c00000000fa00000 SPRG3 0000000000000000 SPRG4 0000000000000000 SPRG5 0000000000000000 SPRG6 0000000000000000 SPRG7 0000000000000000HSRR0 0000000000000000 HSRR1 0000000000000000
CFAR 0000000000000000
LPCR 0000000002000000
SDR1 0000000000000000 DARd0000800801ce800DSISR 0000000042000000It's a little bit better, I think "Suberror: 88" is more comprehensible (at least I hope!).
Same if I boot with "-vga std" or "-vga virtio" instead of "-device virtio-vga".
Some debug informations:
qemu-system-ppc-10016 [004] .... 83794.482587:kvm_book3s_reenter: reentry r=1 | pc=0xc000000000089930
qemu-system-ppc-10016 [004] .... 83794.482588:kvm_exit: exit=PROGRAM | pc=0xc000000000009958 |
msr=0xb000000000001031 | dar=0x7fff3e357200 | srr1=0x800000000004d033 | last_inst=0x4c000024
qemu-system-ppc-10016 [004] .... 83794.482589:kvm_ppc_instr: inst 1275068452 pc0xc0000000005359f4
emulate 0
qemu-system-ppc-10016 [004] .... 83794.482590:kvm_book3s_reenter: reentry r=1 | pc=0xc0000000005359f4 qemu-system-ppc-10016 [004] .... 83794.482591:kvm_exit: exit�TA_SEGMENT | pc=0xc0000000005359f4
| msr=0xb000000000009033 | dar=0xd0000800801ce800 | srr1=0x800000000000d033 | last_inst=0x4c000024
qemu-system-ppc-10016 [004] .... 83794.482592:kvm_book3s_slb_found:18000033deaf8008->61c1b7cf37qemu-system-ppc-10016 [004] .... 83794.482592:kvm_book3s_slbmte:61c1b7cf37990,d000080088000003qemu-system-ppc-10016 [004] .... 83794.482592:kvm_book3s_reenter: reentry r=0 | pc=0xc0000000005359f4 qemu-system-ppc-10016 [004] .... 83794.482594:kvm_exit: exit�TA_STORAGE | pc=0xc0000000005359f4
| msr=0xb000000000009033 | dar=0xd0000800801ce800 | srr1=0x800000000000d033 | last_inst=0xf8c90000
qemu-system-ppc-10016 [004] .... 83794.482599:kvm_book3s_slb_found:18000033deaf8008->61c1b7cf37qemu-system-ppc-10016 [004] .... 83794.482602:kvm_book3s_reenter: reentry r=2 | pc=0xc0000000005359f4 qemu-system-ppc-10016 [004] .... 83794.482602:kvm_userspace_exit: reasonKVM_EXIT_INTERNAL_ERROR(17) qemu-system-ppc-10013 [005] .... 83794.495264:kvm_book3s_mmu_flush: Flush 5468 pPTEs: 200080000000
- 200081000000
qemu-system-ppc-10013 [001] .... 83800.464923:kvm_book3s_mmu_flush: Flush 5468 PTEs: 0 - 0 qemu-system-ppc-10013 [001] .... 83800.464933:kvm_book3s_mmu_invalidate: Flush: hvaac0392d85f7f0
pfnub7e ea�0000002f7f0000 vp�1b217002f7f0 ra/7f0000 [6]
qemu-system-ppc-10013 [001] .... 83800.464938:kvm_book3s_mmu_invalidate: Flush: hvaac0392d85f7e0
pfnub7d ea�0000002f7e0000 vp�1b217002f7e0 ra/7e0000 [6]- With only virtio-gpu and boot/messages on hvterm/VGA console:
/usr/bin/qemu-system-ppc64-serial mon:stdio -m 1024 -cpu host -device virtio-scsi-pci,id=scsi0 -device scsi-cd,drive�0,bus=scsi0.0 -drive media�rom,if=none,id�0,format=raw,file=/opt/openSUSE-Leap-42.3-NET-ppc64le-Build0130-Media.iso -smp 1,threads=1 -enable-kvm -device virtio-gpu -nodefaultsNo problem, the kernel boot with default console on hvterm and some messages on VGA console. And the kernel boot also if I set the console to VGA (console=tty on Linux).
- With only virtio-gpu-pci and boot/messages on hvterm:
/usr/bin/qemu-system-ppc64-serial mon:stdio -m 1024 -cpu host -device virtio-scsi-pci,id=scsi0 -device scsi-cd,drive�0,bus=scsi0.0 -drive media�rom,if=none,id�0,format=raw,file=/opt/openSUSE-Leap-42.3-NET-ppc64le-Build0130-Media.iso -smp 1,threads=1 -enable-kvm -device virtio-gpu-pci -nodefaultsSame as above. It seems like if virtio-gpu and virtio-gpu-pci are the same.
QT_X11_NO_MITSHM=1
/usr/bin/qemu-system-ppc64-serial mon:stdio -m 1024 -cpu host -device virtio-scsi-pci,id=scsi0 -device scsi-cd,drive�0,bus=scsi0.0 -drive media�rom,if=none,id�0,format=raw,file=/opt/openSUSE-Leap-42.3-NET-ppc64le-Build0130-Media.iso -smp 1,threads=1 -enable-kvm -device nec-usb-xhci -device usb-tablet -device usb-kbd -device virtio-vga -device virtio-gpu-pci -nodefaultsSo, is anybody already saw this? May I did something wrong? How we can debug this? (And sorry for this long email, I try to add all the needed information!).
Signature · 8 lines
Loic Devulder - Senior QA Engineer Container & Storage Solutions Quality Assurance team (qa-css) SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton HRB, 21284 (AG Nuernberg) -- To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html