New: kvm_spurious_fault in L1 when running a nested kvm instance on AMD i686-pae

https://bugzilla.kernel.org/show_bug.cgi?id=203979

Bug ID: 203979
Summary: kvm_spurious_fault in L1 when running a nested kvm
instance on AMD i686-pae
Product: Virtualization
Version: unspecified
Kernel Version: 5.2-rc6
Hardware: All
OS: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: kvm
Assignee: [email protected]
Reporter: [email protected]
Regression: No

Hello,

when I try to run a freedos nested in linux kvm guest on kernel 5.2-rc6, I get this message in L1 guest:

debian login: [ 13.291265] FS-Cache: Loaded

[   13.295627] 9p: Installing v9fs 9p2000 file system support
[   13.296946] FS-Cache: Netfs '9p' registered for caching
[   19.200271] ------------[ cut here ]------------
[   19.201265] kernel BUG at arch/x86/kvm/x86.c:358!
[   19.202069] invalid opcode: 0000 [#1] SMP NOPTI
[   19.202850] CPU: 0 PID: 270 Comm: qemu-system-i38 Not tainted
5.2.0-rc6-bughunt+ #1
[   19.203568] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
1.12.0-1 04/01/2014
[   19.203568] EIP: kvm_spurious_fault+0x8/0x10 [kvm]
[   19.203568] Code: 08 ff 75 18 ff 75 14 ff 75 10 ff 75 0c 57 56 e8 5e 09 3d
fa 83 c4 30 8d 65 f8 5e 5f 5d c3 8d 74 26 00 0f 1f 44 00 00 55 89 e5 <0f> 0b 8d
b6 00 00 00 00 0f 1f 44 00 00 55 89 e5 57 56 53 83 ec 04
[   19.203568] EAX: 0004c000 EBX: 00000000 ECX: 00000000 EDX: 00000663
[   19.203568] ESI: 00000000 EDI: 00000000 EBP: c0037da4 ESP: c0037da4
[   19.203568] DS: 007b ES: 007b FS: 0000 GS: 00e0 SS: 0068 EFLAGS: 00210246
[   19.203568] CR0: 80050033 CR2: 00cd813b CR3: 04e7f820 CR4: 000006f0
[   19.203568] Call Trace:
[   19.203568]  svm_vcpu_run+0x1a5/0x650 [kvm_amd]
[   19.203568]  ? kvm_ioapic_scan_entry+0x62/0xe0 [kvm]
[   19.203568]  ? kvm_arch_vcpu_ioctl_run+0x596/0x1a80 [kvm]
[   19.203568]  ? _cond_resched+0x17/0x30
[   19.203568]  ? kvm_vcpu_ioctl+0x214/0x590 [kvm]
[   19.203568]  ? kvm_vcpu_ioctl+0x214/0x590 [kvm]
[   19.203568]  ? __bpf_trace_kvm_async_pf_nopresent_ready+0x20/0x20 [kvm]
[   19.203568]  ? do_vfs_ioctl+0x9a/0x6c0
[   19.203568]  ? tomoyo_path_chmod+0x20/0x20
[   19.203568]  ? tomoyo_file_ioctl+0x19/0x20
[   19.203568]  ? security_file_ioctl+0x30/0x50
[   19.203568]  ? ksys_ioctl+0x56/0x80
[   19.203568]  ? sys_ioctl+0x16/0x20
[   19.203568]  ? do_fast_syscall_32+0x87/0x1df
[   19.203568]  ? entry_SYSENTER_32+0x6b/0xbe
[   19.203568] Modules linked in: 9p fscache ppdev edac_mce_amd kvm_amd

9pnet_virtio kvm bochs_drm 9pnet irqbypass ttm snd_pcm snd_timer snd drm_kms_helper soundcore joydev pcspkr evdev serio_raw drm sg parport_pc parport button ip_tables x_tables autofs4 ext4 crc32c_generic crc16 mbcache jbd2 sr_mod cdrom sd_mod ata_generic ata_piix libata psmouse virtio_pci virtio_ring virtio e1000 scsi_mod i2c_piix4 floppy

[   19.203568] ---[ end trace c0c327b925400cd6 ]---
[   19.203568] EIP: kvm_spurious_fault+0x8/0x10 [kvm]
[   19.203568] Code: 08 ff 75 18 ff 75 14 ff 75 10 ff 75 0c 57 56 e8 5e 09 3d
fa 83 c4 30 8d 65 f8 5e 5f 5d c3 8d 74 26 00 0f 1f 44 00 00 55 89 e5 <0f> 0b 8d
b6 00 00 00 00 0f 1f 44 00 00 55 89 e5 57 56 53 83 ec 04
[   19.203568] EAX: 0004c000 EBX: 00000000 ECX: 00000000 EDX: 00000663
[   19.203568] ESI: 00000000 EDI: 00000000 EBP: c0037da4 ESP: c3a71dfc
[   19.203568] DS: 007b ES: 007b FS: 0000 GS: 00e0 SS: 0068 EFLAGS: 00210246
[   19.203568] CR0: 80050033 CR2: 00cd813b CR3: 04e7f820 CR4: 000006f0

svm_vcpu_run+0x1a5 is the vmsave instruction. The bug seems to be only dependent on the L0 host, where version 5.1 works, but 5.2 fails. Linux version of the L1 guest doesn't seem to matter. (I'm running a pae system but didn't actually test if it has anything to do with pae)

Any ideas about what could have caused it?

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=203979

Jiri Palecek ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #1 from Jiri Palecek ([email protected]) --- So, I have bisected this and found the culprit to be:

commit 8c5fbf1a723107814c20c3f4d6343ab9d694a705 (refs/bisect/bad)

Author: KarimAllah Ahmed <[email protected]>
Date:   Thu Jan 31 21:24:40 2019 +0100

KVM/nSVM: Use the new mapping API for mapping guest memory

Use the new mapping API for mapping guest memory to avoid depending on "struct page".

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=203979

--- Comment #2 from Jiri Palecek ([email protected]) --- Created attachment 283501
--> https://bugzilla.kernel.org/attachment.cgi?id=283501&action=edit
Patch that allows me to run nested kvm instances again

After much looking on the offending commit, which doesn't seem to change any functionality, but just changes the API, I found out the gfns used inside __kvm_map_gfn were in fact far off. Further looking into it I found that these gfns were the result of gfn_to_gpa in some (but not all) places that used the new api. However, what was clearly meant was gpa_to_gfn. So I prepared this patch and tested that it allows to run nested linux instances in kvm successfully on my AMD system.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=203979

Jiri Palecek ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------

           Hardware|All                         |i386
         Regression|No                          |Yes

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=203979

Jiri Palecek ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------

             Status|NEW                         |RESOLVED
         Resolution|---                         |CODE_FIX

--- Comment #3 from Jiri Palecek ([email protected]) --- Already fixed by 8f38302c0be2d

You are receiving this mail because:
You are watching the assignee of the bug.