Skip to contentSkip to conversation

KVM: x86: Starting Windows guest triggers UBSAN: array-index-out-of-bounds

REGRESSION 7.2, BISECTED SimonP 3 participants 9 messages Oct 6 – Oct 8

Subsystems (3)
lore
Share & tools
lore

SimonP

author
#0 lore raw

Hello,

See subject. Happens in permission_fault() on line:

	fault = (mmu->permissions[index] >> pte_access) & 1;

Some output from a printk I added, not spammed but keeps happening:

kvm: permission_fault(): pfec = 70, index = 35, not_smap = 0
kvm: permission_fault(): pfec = 70, index = 35, not_smap = 0
kvm: permission_fault(): pfec = 70, index = 35, not_smap = 0
kvm: permission_fault(): pfec = 70, index = 35, not_smap = 0
kvm: permission_fault(): pfec = 68, index = 34, not_smap = 0
kvm: permission_fault(): pfec = 68, index = 34, not_smap = 0
kvm: permission_fault(): pfec = 70, index = 35, not_smap = 0
kvm: permission_fault(): pfec = 68, index = 34, not_smap = 0
kvm: permission_fault(): pfec = 68, index = 34, not_smap = 0
kvm: permission_fault(): pfec = 70, index = 35, not_smap = 0
kvm: permission_fault(): pfec = 70, index = 35, not_smap = 0
kvm: permission_fault(): pfec = 70, index = 35, not_smap = 0
kvm: permission_fault(): pfec = 70, index = 35, not_smap = 0
kvm: permission_fault(): pfec = 70, index = 35, not_smap = 0
kvm: permission_fault(): pfec = 68, index = 34, not_smap = 0
kvm: permission_fault(): pfec = 70, index = 35, not_smap = 0
kvm: permission_fault(): pfec = 68, index = 34, not_smap = 0
kvm: permission_fault(): pfec = 70, index = 39, not_smap = 1
kvm: permission_fault(): pfec = 70, index = 39, not_smap = 1
kvm: permission_fault(): pfec = 70, index = 39, not_smap = 1
kvm: permission_fault(): pfec = 70, index = 39, not_smap = 1
kvm: permission_fault(): pfec = 70, index = 39, not_smap = 1
kvm: permission_fault(): pfec = 70, index = 39, not_smap = 1
kvm: permission_fault(): pfec = 70, index = 39, not_smap = 1
kvm: permission_fault(): pfec = 70, index = 39, not_smap = 1
kvm: permission_fault(): pfec = 70, index = 39, not_smap = 1
kvm: permission_fault(): pfec = 70, index = 39, not_smap = 1
kvm: permission_fault(): pfec = 70, index = 39, not_smap = 1
kvm: permission_fault(): pfec = 70, index = 39, not_smap = 1
kvm: permission_fault(): pfec = 70, index = 39, not_smap = 1
kvm: permission_fault(): pfec = 70, index = 39, not_smap = 1

Full dmesg spew below. The guest works but this seems not great. Not fixed as of 7.2.9.

I bisected the bad commit to 687ee95c1b6d7509a25bbc661e62801aa06cc2ae but as this is just an enablement commit it seems more likely that something is incomplete or went wrong in other commits in the series.

Regards,
Simon

dmesg:

kernel: UBSAN: array-index-out-of-bounds in arch/x86/kvm/mmu.h:225:27
kernel: index 34 is out of range for type 'u16 [16]'
kernel: CPU: 7 UID: 64055 PID: 13695 Comm: CPU 12/KVM Not tainted 7.2.9+deb14-amd64-simonp #1 PREEMPT(lazy)  Debian 7.2.9-1sp 
kernel: Hardware name: To Be Filled By O.E.M. X570 Taichi/X570 Taichi, BIOS P5.80 03/24/2026
kernel: Call Trace:
kernel:  <TASK>
kernel:  dump_stack_lvl+0x4d/0x70
kernel:  ubsan_epilogue+0x5/0x2b
kernel:  __ubsan_handle_out_of_bounds.cold+0x4e/0x58
kernel:  paging64_walk_addr_generic+0xc24/0xce0 [kvm]
kernel:  paging64_page_fault+0x84/0x9a0 [kvm]
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? __pfx_paging64_page_fault+0x10/0x10 [kvm]
kernel:  kvm_mmu_do_page_fault+0x140/0x240 [kvm]
kernel:  kvm_mmu_page_fault+0x9c/0x880 [kvm]
kernel:  ? kvm_vcpu_read_guest+0x52/0xa0 [kvm]
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? nested_svm_merge_msrpm+0xc8/0x1c0 [kvm_amd]
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? __svm_vcpu_run+0x145/0x240 [kvm_amd]
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? rcu_note_context_switch+0x57/0x530
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  npf_interception+0xb2/0x2c0 [kvm_amd]
kernel:  kvm_arch_vcpu_ioctl_run+0x69b/0x20e0 [kvm]
kernel:  kvm_vcpu_ioctl+0x367/0xac0 [kvm]
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? select_task_rq_fair+0x9b5/0x2670
kernel:  __x64_sys_ioctl+0xb9/0x100
kernel:  ? kvm_vcpu_ioctl+0x2ed/0xac0 [kvm]
kernel:  do_syscall_64+0xdd/0x5f0
kernel:  ? __smp_call_single_queue+0xd6/0x190
kernel:  ? kvm_vcpu_ioctl+0x2ed/0xac0 [kvm]
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? ttwu_queue_wakelist+0x144/0x230
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? try_to_wake_up+0x300/0x8b0
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? kvm_on_user_return+0x1e/0xe0 [kvm]
kernel:  ? __x64_sys_ioctl+0xd4/0x100
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? wake_up_q+0x4c/0xa0
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? futex_wake+0x117/0x270
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? __x64_sys_futex+0x12a/0x220
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? do_syscall_64+0x116/0x5f0
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? kvm_vcpu_ioctl+0x2ed/0xac0 [kvm]
kernel:  ? __x64_sys_futex+0x12a/0x220
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? __x64_sys_futex+0x12a/0x220
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? __seccomp_filter+0x86/0x6e0
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? do_syscall_64+0x31e/0x5f0
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? write_ibpb+0x21/0x50
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? do_syscall_64+0x94/0x5f0
kernel:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
kernel: RIP: 0033:0x7f38d6202f5b
kernel: Code: 00 48 89 44 24 18 31 c0 48 8d 44 24 60 c7 04 24 10 00 00 00 48 89 44 24 08 48 8d 44 24 20 48 89 44 24 10 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 1c 48 8b 44 24 18 64 48 2b 04 25 28 00 00
kernel: RSP: 002b:00007f359d9fe280 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
kernel: RAX: ffffffffffffffda RBX: 000000000000ae80 RCX: 00007f38d6202f5b
kernel: RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 000000000000003a
kernel: RBP: 00005575017a2550 R08: 0000000000000000 R09: 0000000000000000
kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
kernel: R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
kernel:  </TASK>
kernel: ---[ end trace ]---
kernel: ------------[ cut here ]------------
kernel: WARNING: arch/x86/kvm/mmu.h:227 at paging64_walk_addr_generic+0x8e4/0xce0 [kvm], CPU#7: CPU 12/KVM/13695
kernel: Modules linked in: vhost_net vhost vhost_iotlb tap tun nft_masq nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_tftp nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 bridge nft_fib stp llc nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 snd_seq_dummy nft_reject snd_hrtimer nft_ct snd_seq snd_seq_device nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables uinput btrfs libblake2b nfsd auth_rpcgss nls_ascii nfs_acl nls_cp437 lockd raid6_pq xor vfat grace fat binfmt_misc xfs sunrpc snd_hda_codec_alc882 amdgpu snd_hda_codec_realtek_lib snd_hda_codec_generic snd_hda_codec_atihdmi snd_hda_codec_hdmi snd_hda_intel amd_atl intel_rapl_msr intel_rapl_common drm_buddy vfio_pci amdxcp vfio_pci_core drm_ttm_helper edac_mce_amd vfio_iommu_type1 ttm jc42 vfio drm_exec snd_intel_dspcfg drm_panel_backlight_quirks kvm_amd snd_hda_codec gpu_sched drm_suballoc_helper ee1004 wmi_bmof snd_hda_core joydev kvm evdev igb drm_display_helper snd_hwdep irqbypass i2c_algo_bit snd_pcm cec dca rapl sg pcspkr k10temp
kernel:  snd_timer video snd soundcore i2c_piix4 button wmi i2c_smbus nvme_fabrics efi_pstore configfs ext4 crc16 mbcache jbd2 crc32c_cryptoapi dm_crypt hid_generic sd_mod usbhid hid ahci libahci xhci_pci nvme libata xhci_hcd nvme_core nvme_keyring scsi_mod usbcore nvme_auth sp5100_tco scsi_common ccp usb_common watchdog dm_mirror dm_region_hash dm_log dm_mod nct6775 nct6775_core ntsync hwmon_vid i2c_dev msr efivarfs autofs4 aesni_intel gf128mul
kernel: CPU: 7 UID: 64055 PID: 13695 Comm: CPU 12/KVM Not tainted 7.2.9+deb14-amd64-simonp #1 PREEMPT(lazy)  Debian 7.2.9-1sp 
kernel: Hardware name: To Be Filled By O.E.M. X570 Taichi/X570 Taichi, BIOS P5.80 03/24/2026
kernel: RIP: 0010:paging64_walk_addr_generic+0x8e4/0xce0 [kvm]
kernel: Code: f9 1f 0f 87 d6 ad 01 00 44 89 c2 d3 ea 23 54 24 18 83 e2 03 89 d1 f7 d9 83 e1 20 83 c9 01 85 d2 0f 95 c2 09 d0 e9 3f fe ff ff <0f> 0b e9 26 fe ff ff 8b 44 24 20 45 8b 7e 04 4c 89 64 24 18 4c 89
kernel: RSP: 0018:ffffc98d85b9b5f0 EFLAGS: 00010202
kernel: RAX: 0000000000000000 RBX: 60000001274000e3 RCX: 000000000000000b
kernel: RDX: 0000000000000022 RSI: 0000000000000001 RDI: ffffffffa5ad965d
kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: ffff892f725bd5f0
kernel: R10: 00000000274000e3 R11: e000000100000023 R12: ffff892f725bd1d8
kernel: R13: ffff892f725bd760 R14: ffffc98d85b9b6b8 R15: 0000000000118c71
kernel: FS:  00007f359d9ff6c0(0000) GS:ffff893b97d6b000(0000) knlGS:0000000000000000
kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: CR2: 0000000000000000 CR3: 00000001c382b000 CR4: 0000000000f50ef0
kernel: PKRU: 55555554
kernel: Call Trace:
kernel:  <TASK>
kernel:  paging64_page_fault+0x84/0x9a0 [kvm]
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? __pfx_paging64_page_fault+0x10/0x10 [kvm]
kernel:  kvm_mmu_do_page_fault+0x140/0x240 [kvm]
kernel:  kvm_mmu_page_fault+0x9c/0x880 [kvm]
kernel:  ? kvm_vcpu_read_guest+0x52/0xa0 [kvm]
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? nested_svm_merge_msrpm+0xc8/0x1c0 [kvm_amd]
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? __svm_vcpu_run+0x145/0x240 [kvm_amd]
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? rcu_note_context_switch+0x57/0x530
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  npf_interception+0xb2/0x2c0 [kvm_amd]
kernel:  kvm_arch_vcpu_ioctl_run+0x69b/0x20e0 [kvm]
kernel:  kvm_vcpu_ioctl+0x367/0xac0 [kvm]
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? select_task_rq_fair+0x9b5/0x2670
kernel:  __x64_sys_ioctl+0xb9/0x100
kernel:  ? kvm_vcpu_ioctl+0x2ed/0xac0 [kvm]
kernel:  do_syscall_64+0xdd/0x5f0
kernel:  ? __smp_call_single_queue+0xd6/0x190
kernel:  ? kvm_vcpu_ioctl+0x2ed/0xac0 [kvm]
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? ttwu_queue_wakelist+0x144/0x230
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? try_to_wake_up+0x300/0x8b0
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? kvm_on_user_return+0x1e/0xe0 [kvm]
kernel:  ? __x64_sys_ioctl+0xd4/0x100
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? wake_up_q+0x4c/0xa0
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? futex_wake+0x117/0x270
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? __x64_sys_futex+0x12a/0x220
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? do_syscall_64+0x116/0x5f0
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? kvm_vcpu_ioctl+0x2ed/0xac0 [kvm]
kernel:  ? __x64_sys_futex+0x12a/0x220
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? __x64_sys_futex+0x12a/0x220
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? __seccomp_filter+0x86/0x6e0
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? do_syscall_64+0x31e/0x5f0
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? write_ibpb+0x21/0x50
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? do_syscall_64+0x94/0x5f0
kernel:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
kernel: RIP: 0033:0x7f38d6202f5b
kernel: Code: 00 48 89 44 24 18 31 c0 48 8d 44 24 60 c7 04 24 10 00 00 00 48 89 44 24 08 48 8d 44 24 20 48 89 44 24 10 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 1c 48 8b 44 24 18 64 48 2b 04 25 28 00 00
kernel: RSP: 002b:00007f359d9fe280 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
kernel: RAX: ffffffffffffffda RBX: 000000000000ae80 RCX: 00007f38d6202f5b
kernel: RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 000000000000003a
kernel: RBP: 00005575017a2550 R08: 0000000000000000 R09: 0000000000000000
kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
kernel: R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
kernel:  </TASK>
maintainer
#1 lore raw
Quoted 9 lines · SimonP

Hello,

See subject. Happens in permission_fault() on line:

        fault = (mmu->permissions[index] >> pte_access) & 1;

Some output from a printk I added, not spammed but keeps happening:

kvm: permission_fault(): pfec = 70, index = 35, not_smap = 0

Can you confirm you are using nested virtualization, and if so what is the L1 hypervisor and L2 guest combination? Also what is the processor model?

The issue is weird; it is caused by the SS bit which is apparently set in the #NPF exitcode; but it shouldn't be unless bit 4 is set in the misc_ctl field, and KVM never sets it. Can you check if the attached patch fixes it (modulo the fact that it shouldn't happen in the first place)?

Thanks,

Paolo

Quoted 205 lines

kvm: permission_fault(): pfec = 70, index = 35, not_smap = 0
kvm: permission_fault(): pfec = 70, index = 35, not_smap = 0
kvm: permission_fault(): pfec = 70, index = 35, not_smap = 0
kvm: permission_fault(): pfec = 68, index = 34, not_smap = 0
kvm: permission_fault(): pfec = 68, index = 34, not_smap = 0
kvm: permission_fault(): pfec = 70, index = 35, not_smap = 0
kvm: permission_fault(): pfec = 68, index = 34, not_smap = 0
kvm: permission_fault(): pfec = 68, index = 34, not_smap = 0
kvm: permission_fault(): pfec = 70, index = 35, not_smap = 0
kvm: permission_fault(): pfec = 70, index = 35, not_smap = 0
kvm: permission_fault(): pfec = 70, index = 35, not_smap = 0
kvm: permission_fault(): pfec = 70, index = 35, not_smap = 0
kvm: permission_fault(): pfec = 70, index = 35, not_smap = 0
kvm: permission_fault(): pfec = 68, index = 34, not_smap = 0
kvm: permission_fault(): pfec = 70, index = 35, not_smap = 0
kvm: permission_fault(): pfec = 68, index = 34, not_smap = 0
kvm: permission_fault(): pfec = 70, index = 39, not_smap = 1
kvm: permission_fault(): pfec = 70, index = 39, not_smap = 1
kvm: permission_fault(): pfec = 70, index = 39, not_smap = 1
kvm: permission_fault(): pfec = 70, index = 39, not_smap = 1
kvm: permission_fault(): pfec = 70, index = 39, not_smap = 1
kvm: permission_fault(): pfec = 70, index = 39, not_smap = 1
kvm: permission_fault(): pfec = 70, index = 39, not_smap = 1
kvm: permission_fault(): pfec = 70, index = 39, not_smap = 1
kvm: permission_fault(): pfec = 70, index = 39, not_smap = 1
kvm: permission_fault(): pfec = 70, index = 39, not_smap = 1
kvm: permission_fault(): pfec = 70, index = 39, not_smap = 1
kvm: permission_fault(): pfec = 70, index = 39, not_smap = 1
kvm: permission_fault(): pfec = 70, index = 39, not_smap = 1
kvm: permission_fault(): pfec = 70, index = 39, not_smap = 1

Full dmesg spew below. The guest works but this seems not great. Not fixed as of 7.2.9.

I bisected the bad commit to 687ee95c1b6d7509a25bbc661e62801aa06cc2ae but as this is just an enablement commit it seems more likely that something is incomplete or went wrong in other commits in the series.

Regards,
Simon

dmesg:

kernel: UBSAN: array-index-out-of-bounds in arch/x86/kvm/mmu.h:225:27
kernel: index 34 is out of range for type 'u16 [16]'
kernel: CPU: 7 UID: 64055 PID: 13695 Comm: CPU 12/KVM Not tainted 7.2.9+deb14-amd64-simonp #1 PREEMPT(lazy)  Debian 7.2.9-1sp
kernel: Hardware name: To Be Filled By O.E.M. X570 Taichi/X570 Taichi, BIOS P5.80 03/24/2026
kernel: Call Trace:
kernel:  <TASK>
kernel:  dump_stack_lvl+0x4d/0x70
kernel:  ubsan_epilogue+0x5/0x2b
kernel:  __ubsan_handle_out_of_bounds.cold+0x4e/0x58
kernel:  paging64_walk_addr_generic+0xc24/0xce0 [kvm]
kernel:  paging64_page_fault+0x84/0x9a0 [kvm]
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? __pfx_paging64_page_fault+0x10/0x10 [kvm]
kernel:  kvm_mmu_do_page_fault+0x140/0x240 [kvm]
kernel:  kvm_mmu_page_fault+0x9c/0x880 [kvm]
kernel:  ? kvm_vcpu_read_guest+0x52/0xa0 [kvm]
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? nested_svm_merge_msrpm+0xc8/0x1c0 [kvm_amd]
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? __svm_vcpu_run+0x145/0x240 [kvm_amd]
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? rcu_note_context_switch+0x57/0x530
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  npf_interception+0xb2/0x2c0 [kvm_amd]
kernel:  kvm_arch_vcpu_ioctl_run+0x69b/0x20e0 [kvm]
kernel:  kvm_vcpu_ioctl+0x367/0xac0 [kvm]
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? select_task_rq_fair+0x9b5/0x2670
kernel:  __x64_sys_ioctl+0xb9/0x100
kernel:  ? kvm_vcpu_ioctl+0x2ed/0xac0 [kvm]
kernel:  do_syscall_64+0xdd/0x5f0
kernel:  ? __smp_call_single_queue+0xd6/0x190
kernel:  ? kvm_vcpu_ioctl+0x2ed/0xac0 [kvm]
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? ttwu_queue_wakelist+0x144/0x230
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? try_to_wake_up+0x300/0x8b0
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? kvm_on_user_return+0x1e/0xe0 [kvm]
kernel:  ? __x64_sys_ioctl+0xd4/0x100
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? wake_up_q+0x4c/0xa0
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? futex_wake+0x117/0x270
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? __x64_sys_futex+0x12a/0x220
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? do_syscall_64+0x116/0x5f0
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? kvm_vcpu_ioctl+0x2ed/0xac0 [kvm]
kernel:  ? __x64_sys_futex+0x12a/0x220
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? __x64_sys_futex+0x12a/0x220
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? __seccomp_filter+0x86/0x6e0
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? do_syscall_64+0x31e/0x5f0
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? write_ibpb+0x21/0x50
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? do_syscall_64+0x94/0x5f0
kernel:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
kernel: RIP: 0033:0x7f38d6202f5b
kernel: Code: 00 48 89 44 24 18 31 c0 48 8d 44 24 60 c7 04 24 10 00 00 00 48 89 44 24 08 48 8d 44 24 20 48 89 44 24 10 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 1c 48 8b 44 24 18 64 48 2b 04 25 28 00 00
kernel: RSP: 002b:00007f359d9fe280 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
kernel: RAX: ffffffffffffffda RBX: 000000000000ae80 RCX: 00007f38d6202f5b
kernel: RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 000000000000003a
kernel: RBP: 00005575017a2550 R08: 0000000000000000 R09: 0000000000000000
kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
kernel: R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
kernel:  </TASK>
kernel: ---[ end trace ]---
kernel: ------------[ cut here ]------------
kernel: WARNING: arch/x86/kvm/mmu.h:227 at paging64_walk_addr_generic+0x8e4/0xce0 [kvm], CPU#7: CPU 12/KVM/13695
kernel: Modules linked in: vhost_net vhost vhost_iotlb tap tun nft_masq nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_tftp nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 bridge nft_fib stp llc nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 snd_seq_dummy nft_reject snd_hrtimer nft_ct snd_seq snd_seq_device nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables uinput btrfs libblake2b nfsd auth_rpcgss nls_ascii nfs_acl nls_cp437 lockd raid6_pq xor vfat grace fat binfmt_misc xfs sunrpc snd_hda_codec_alc882 amdgpu snd_hda_codec_realtek_lib snd_hda_codec_generic snd_hda_codec_atihdmi snd_hda_codec_hdmi snd_hda_intel amd_atl intel_rapl_msr intel_rapl_common drm_buddy vfio_pci amdxcp vfio_pci_core drm_ttm_helper edac_mce_amd vfio_iommu_type1 ttm jc42 vfio drm_exec snd_intel_dspcfg drm_panel_backlight_quirks kvm_amd snd_hda_codec gpu_sched drm_suballoc_helper ee1004 wmi_bmof snd_hda_core joydev kvm evdev igb drm_display_helper snd_hwdep irqbypass i2c_algo_bit snd_pcm cec dca rapl sg pcspkr k10temp
kernel:  snd_timer video snd soundcore i2c_piix4 button wmi i2c_smbus nvme_fabrics efi_pstore configfs ext4 crc16 mbcache jbd2 crc32c_cryptoapi dm_crypt hid_generic sd_mod usbhid hid ahci libahci xhci_pci nvme libata xhci_hcd nvme_core nvme_keyring scsi_mod usbcore nvme_auth sp5100_tco scsi_common ccp usb_common watchdog dm_mirror dm_region_hash dm_log dm_mod nct6775 nct6775_core ntsync hwmon_vid i2c_dev msr efivarfs autofs4 aesni_intel gf128mul
kernel: CPU: 7 UID: 64055 PID: 13695 Comm: CPU 12/KVM Not tainted 7.2.9+deb14-amd64-simonp #1 PREEMPT(lazy)  Debian 7.2.9-1sp
kernel: Hardware name: To Be Filled By O.E.M. X570 Taichi/X570 Taichi, BIOS P5.80 03/24/2026
kernel: RIP: 0010:paging64_walk_addr_generic+0x8e4/0xce0 [kvm]
kernel: Code: f9 1f 0f 87 d6 ad 01 00 44 89 c2 d3 ea 23 54 24 18 83 e2 03 89 d1 f7 d9 83 e1 20 83 c9 01 85 d2 0f 95 c2 09 d0 e9 3f fe ff ff <0f> 0b e9 26 fe ff ff 8b 44 24 20 45 8b 7e 04 4c 89 64 24 18 4c 89
kernel: RSP: 0018:ffffc98d85b9b5f0 EFLAGS: 00010202
kernel: RAX: 0000000000000000 RBX: 60000001274000e3 RCX: 000000000000000b
kernel: RDX: 0000000000000022 RSI: 0000000000000001 RDI: ffffffffa5ad965d
kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: ffff892f725bd5f0
kernel: R10: 00000000274000e3 R11: e000000100000023 R12: ffff892f725bd1d8
kernel: R13: ffff892f725bd760 R14: ffffc98d85b9b6b8 R15: 0000000000118c71
kernel: FS:  00007f359d9ff6c0(0000) GS:ffff893b97d6b000(0000) knlGS:0000000000000000
kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: CR2: 0000000000000000 CR3: 00000001c382b000 CR4: 0000000000f50ef0
kernel: PKRU: 55555554
kernel: Call Trace:
kernel:  <TASK>
kernel:  paging64_page_fault+0x84/0x9a0 [kvm]
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? __pfx_paging64_page_fault+0x10/0x10 [kvm]
kernel:  kvm_mmu_do_page_fault+0x140/0x240 [kvm]
kernel:  kvm_mmu_page_fault+0x9c/0x880 [kvm]
kernel:  ? kvm_vcpu_read_guest+0x52/0xa0 [kvm]
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? nested_svm_merge_msrpm+0xc8/0x1c0 [kvm_amd]
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? __svm_vcpu_run+0x145/0x240 [kvm_amd]
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? rcu_note_context_switch+0x57/0x530
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  npf_interception+0xb2/0x2c0 [kvm_amd]
kernel:  kvm_arch_vcpu_ioctl_run+0x69b/0x20e0 [kvm]
kernel:  kvm_vcpu_ioctl+0x367/0xac0 [kvm]
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? select_task_rq_fair+0x9b5/0x2670
kernel:  __x64_sys_ioctl+0xb9/0x100
kernel:  ? kvm_vcpu_ioctl+0x2ed/0xac0 [kvm]
kernel:  do_syscall_64+0xdd/0x5f0
kernel:  ? __smp_call_single_queue+0xd6/0x190
kernel:  ? kvm_vcpu_ioctl+0x2ed/0xac0 [kvm]
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? ttwu_queue_wakelist+0x144/0x230
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? try_to_wake_up+0x300/0x8b0
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? kvm_on_user_return+0x1e/0xe0 [kvm]
kernel:  ? __x64_sys_ioctl+0xd4/0x100
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? wake_up_q+0x4c/0xa0
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? futex_wake+0x117/0x270
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? __x64_sys_futex+0x12a/0x220
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? do_syscall_64+0x116/0x5f0
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? kvm_vcpu_ioctl+0x2ed/0xac0 [kvm]
kernel:  ? __x64_sys_futex+0x12a/0x220
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? __x64_sys_futex+0x12a/0x220
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? __seccomp_filter+0x86/0x6e0
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? do_syscall_64+0x31e/0x5f0
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? write_ibpb+0x21/0x50
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? do_syscall_64+0x94/0x5f0
kernel:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
kernel: RIP: 0033:0x7f38d6202f5b
kernel: Code: 00 48 89 44 24 18 31 c0 48 8d 44 24 60 c7 04 24 10 00 00 00 48 89 44 24 08 48 8d 44 24 20 48 89 44 24 10 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 1c 48 8b 44 24 18 64 48 2b 04 25 28 00 00
kernel: RSP: 002b:00007f359d9fe280 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
kernel: RAX: ffffffffffffffda RBX: 000000000000ae80 RCX: 00007f38d6202f5b
kernel: RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 000000000000003a
kernel: RBP: 00005575017a2550 R08: 0000000000000000 R09: 0000000000000000
kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
kernel: R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
kernel:  </TASK>
maintainer
#2 lore raw
Quoted 16 lines · Paolo Bonzini
Quoted 9 lines · SimonP

Hello,

See subject. Happens in permission_fault() on line:

        fault = (mmu->permissions[index] >> pte_access) & 1;

Some output from a printk I added, not spammed but keeps happening:

kvm: permission_fault(): pfec = 70, index = 35, not_smap = 0

Can you confirm you are using nested virtualization, and if so what is the L1 hypervisor and L2 guest combination? Also what is the processor model?

Quoted 5 lines

The issue is weird; it is caused by the SS bit which is apparently set in the #NPF exitcode; but it shouldn't be unless bit 4 is set in the misc_ctl field, and KVM never sets it. Can you check if the attached patch fixes it (modulo the fact that it shouldn't happen in the first place)?

Attachment added now.

Paolo

arch/x86/kvm/mmu.h +12 -4
index 40dcbcbae173..eee7edbd9c6f 100644
@@ -275,8 +275,18 @@ static inline u8 permission_fault(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w,
 				  unsigned pte_access, unsigned pte_pkey,
 				  u64 access)
 {
-	/* strip nested paging fault error codes */
-	unsigned int pfec = access;
+	/*
+	 * Of the error codes that do not contribute to the index in
+	 * fmt->permission, PK is not included in EPT violation bits and
+	 * not supported by shadow paging, and RSVD faults are handled
+	 * elsewhere.  SS however is included in the NPT exit code.
+	 */
+	unsigned int pfec = access & ~PFERR_SS_MASK;
+	if (WARN_ON_ONCE(pfec & ~(PFERR_PRESENT_MASK | PFERR_WRITE_MASK |
+				  PFERR_USER_MASK | PFERR_FETCH_MASK)))
+		pfec &= (PFERR_PRESENT_MASK | PFERR_WRITE_MASK |
+			 PFERR_USER_MASK | PFERR_FETCH_MASK);
+
 	unsigned long rflags = kvm_x86_call(get_rflags)(vcpu);
 
 	/*
@@ -301,8 +311,6 @@ static inline u8 permission_fault(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w,
 	kvm_mmu_refresh_passthrough_bits(vcpu, w);
 
 	fault = (fmt->permissions[index] >> pte_access) & 1;
-
-	WARN_ON_ONCE(pfec & (PFERR_PK_MASK | PFERR_SS_MASK | PFERR_RSVD_MASK));
 	if (unlikely(fmt->pkru_mask)) {
 		u32 pkru_bits, offset;
 
arch/x86/kvm/mmu/paging_tmpl.h +1 -1
index dc0155ed1cbf..a1afe2ae28ca 100644
@@ -504,7 +504,7 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker,
 	return 1;
 
 error:
-	errcode |= write_fault | user_fault;
+	errcode |= access & (PFERR_WRITE_MASK | PFERR_USER_MASK | PFERR_SS_MASK);
 	if (fetch_fault && has_pferr_fetch(w))
 		errcode |= PFERR_FETCH_MASK;
 
author
#3 lore raw
Quoted 17 lines · Paolo Bonzini
Quoted 16 lines · Paolo Bonzini
Quoted 9 lines · SimonP

Hello,

See subject. Happens in permission_fault() on line:

        fault = (mmu->permissions[index] >> pte_access) & 1;

Some output from a printk I added, not spammed but keeps happening:

kvm: permission_fault(): pfec = 70, index = 35, not_smap = 0

Can you confirm you are using nested virtualization, and if so what is the L1 hypervisor and L2 guest combination? Also what is the processor model?

Quoted 7 lines
Quoted 5 lines

The issue is weird; it is caused by the SS bit which is apparently set in the #NPF exitcode; but it shouldn't be unless bit 4 is set in the misc_ctl field, and KVM never sets it. Can you check if the attached patch fixes it (modulo the fact that it shouldn't happen in the first place)?

Attachment added now.

Patch is good. UBSAN doesn't complain anymore, no WARN is hit, and the guest works.

Regards,
Simon

Quoted 2 lines

Paolo

author
#4 lore raw
In reply to Paolo Bonzini
Quoted 21 lines · Paolo Bonzini
Quoted 9 lines · SimonP

Hello,

See subject. Happens in permission_fault() on line:

        fault = (mmu->permissions[index] >> pte_access) & 1;

Some output from a printk I added, not spammed but keeps happening:

kvm: permission_fault(): pfec = 70, index = 35, not_smap = 0

Can you confirm you are using nested virtualization, and if so what is the L1 hypervisor and L2 guest combination? Also what is the processor model?

The issue is weird; it is caused by the SS bit which is apparently set in the #NPF exitcode; but it shouldn't be unless bit 4 is set in the misc_ctl field, and KVM never sets it. Can you check if the attached patch fixes it (modulo the fact that it shouldn't happen in the first place)?

Quoted 4 lines

Thanks,

Paolo

Will test the patch when I can but for now increment the weird counter because there is no nesting, this is on a bare-metal hypervisor. The guest is Windows 11. $(lscpu) is below.

Regards,
Simon

Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Address sizes:                           48 bits physical, 48 bits virtual
Byte Order:                              Little Endian
CPU(s):                                  32
On-line CPU(s) list:                     0-31
Vendor ID:                               AuthenticAMD
Model name:                              AMD Ryzen 9 5950X 16-Core Processor
CPU family:                              25
Model:                                   33
Thread(s) per core:                      2
Core(s) per socket:                      16
Socket(s):                               1
Stepping:                                0
Microcode version:                       0xa201030
Frequency boost:                         disabled
CPU(s) scaling MHz:                      86%
CPU max MHz:                             5086.1812
CPU min MHz:                             582.1540
BogoMIPS:                                6799.84
Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpclmulqdq rdpid overflow_recov succor smca fsrm debug_swap
Virtualization:                          AMD-V
L1d cache:                               512 KiB (16 instances)
L1i cache:                               512 KiB (16 instances)
L2 cache:                                8 MiB (16 instances)
L3 cache:                                64 MiB (2 instances)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-31
Vulnerability Gather data sampling:      Not affected
Vulnerability Ghostwrite:                Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Old microcode:             Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Mitigation; Safe RET
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Mitigation; Clear CPU buffers
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Mitigation; IBPB before exit to userspace
maintainer
#5 lore raw
Quoted 23 lines · SimonP
Quoted 15 lines · Paolo Bonzini
Quoted 9 lines · SimonP

Hello,

See subject. Happens in permission_fault() on line:

        fault = (mmu->permissions[index] >> pte_access) & 1;

Some output from a printk I added, not spammed but keeps happening:

kvm: permission_fault(): pfec = 70, index = 35, not_smap = 0

Can you confirm you are using nested virtualization, and if so what is the L1 hypervisor and L2 guest combination? Also what is the processor model?

Quoted 5 lines

The issue is weird; it is caused by the SS bit which is apparently set in the #NPF exitcode; but it shouldn't be unless bit 4 is set in the misc_ctl field, and KVM never sets it. Can you check if the attached patch fixes it (modulo the fact that it shouldn't happen in the first place)?

Quoted 3 lines

Will test the patch when I can but for now increment the weird counter because there is no nesting, this is on a bare-metal hypervisor. The guest is Windows 11. $(lscpu) is below.

No, I mean nesting another hypervisor *inside* KVM; do you have Hyper-V enabled in your Windows VM? The stack trace has both npf_interception and paging64_page_fault, which doesn't really happen without nesting.

If you are using QEMU, you can use "info stats vcpu guest_mode" from the QEMU monitor to tell you more. If you're using libvirt, likewise, that would be

virsh qemu-monitor-command ID --hmp 'info stats vcpu guest_mode

Thanks,

Paolo

author
#6 lore raw
Quoted 33 lines · Paolo Bonzini
Quoted 23 lines · SimonP
Quoted 15 lines · Paolo Bonzini
Quoted 9 lines · SimonP

Hello,

See subject. Happens in permission_fault() on line:

        fault = (mmu->permissions[index] >> pte_access) & 1;

Some output from a printk I added, not spammed but keeps happening:

kvm: permission_fault(): pfec = 70, index = 35, not_smap = 0

Can you confirm you are using nested virtualization, and if so what is the L1 hypervisor and L2 guest combination? Also what is the processor model?

Quoted 5 lines

The issue is weird; it is caused by the SS bit which is apparently set in the #NPF exitcode; but it shouldn't be unless bit 4 is set in the misc_ctl field, and KVM never sets it. Can you check if the attached patch fixes it (modulo the fact that it shouldn't happen in the first place)?

Quoted 3 lines

Will test the patch when I can but for now increment the weird counter because there is no nesting, this is on a bare-metal hypervisor. The guest is Windows 11. $(lscpu) is below.

No, I mean nesting another hypervisor *inside* KVM; do you have Hyper-V enabled in your Windows VM? The stack trace has both npf_interception and paging64_page_fault, which doesn't really happen without nesting.

Quoted 6 lines

If you are using QEMU, you can use "info stats vcpu guest_mode" from the QEMU monitor to tell you more. If you're using libvirt, likewise, that would be

virsh qemu-monitor-command ID --hmp 'info stats vcpu guest_mode

Sorry, misunderstood. Yes, I use WSL2 in the Windows VM.

$ virsh qemu-monitor-command win11 --hmp 'info stats vcpu guest_mode'
provider: kvm
    guest_mode (instant, boolean): no

Regards,
Simon

Quoted 4 lines

Thanks,

Paolo

#7 lore raw
In reply to Paolo Bonzini
Quoted 20 lines · Paolo Bonzini
Quoted 9 lines · SimonP

Hello,

See subject. Happens in permission_fault() on line:

        fault = (mmu->permissions[index] >> pte_access) & 1;

Some output from a printk I added, not spammed but keeps happening:

kvm: permission_fault(): pfec = 70, index = 35, not_smap = 0

Can you confirm you are using nested virtualization, and if so what is the L1 hypervisor and L2 guest combination? Also what is the processor model?

The issue is weird; it is caused by the SS bit which is apparently set in the #NPF exitcode; but it shouldn't be unless bit 4 is set in the misc_ctl field, and KVM never sets it. Can you check if the attached patch fixes it (modulo the fact that it shouldn't happen in the first place)?

...

Quoted 5 lines
Quoted 5 lines

kvm: permission_fault(): pfec = 70, index = 35, not_smap = 0
kvm: permission_fault(): pfec = 70, index = 35, not_smap = 0
kvm: permission_fault(): pfec = 70, index = 35, not_smap = 0
kvm: permission_fault(): pfec = 68, index = 34, not_smap = 0
kvm: permission_fault(): pfec = 68, index = 34, not_smap = 0

It's not just SS, it's also PK (which shouldn't happen either) and RSVD, which *really* shouldn't happen. In the page fault path, KVM explicitly clears RSVD from error code that becomes @access and then pfec, so I'm completely baffled. Nothing in walk_addr_generic() writes to @access, so the only thing I can think of is that we're clobbering the stack?

	r = FNAME(walk_addr)(&walker, vcpu, fault->addr,
			     fault->error_code & ~PFERR_RSVD_MASK);
#8 lore raw
Quoted 30 lines · Sean Christopherson
Quoted 20 lines · Paolo Bonzini
Quoted 9 lines · SimonP

Hello,

See subject. Happens in permission_fault() on line:

        fault = (mmu->permissions[index] >> pte_access) & 1;

Some output from a printk I added, not spammed but keeps happening:

kvm: permission_fault(): pfec = 70, index = 35, not_smap = 0

Can you confirm you are using nested virtualization, and if so what is the L1 hypervisor and L2 guest combination? Also what is the processor model?

The issue is weird; it is caused by the SS bit which is apparently set in the #NPF exitcode; but it shouldn't be unless bit 4 is set in the misc_ctl field, and KVM never sets it. Can you check if the attached patch fixes it (modulo the fact that it shouldn't happen in the first place)?

...

Quoted 5 lines
Quoted 5 lines

kvm: permission_fault(): pfec = 70, index = 35, not_smap = 0
kvm: permission_fault(): pfec = 70, index = 35, not_smap = 0
kvm: permission_fault(): pfec = 70, index = 35, not_smap = 0
kvm: permission_fault(): pfec = 68, index = 34, not_smap = 0
kvm: permission_fault(): pfec = 68, index = 34, not_smap = 0

Quoted 8 lines

It's not just SS, it's also PK (which shouldn't happen either) and RSVD, which *really* shouldn't happen. In the page fault path, KVM explicitly clears RSVD from error code that becomes @access and then pfec, so I'm completely baffled. Nothing in walk_addr_generic() writes to @access, so the only thing I can think of is that we're clobbering the stack?

	r = FNAME(walk_addr)(&walker, vcpu, fault->addr,
			     fault->error_code & ~PFERR_RSVD_MASK);

Paolo pointed out offlist that the printks are in decimal, not hex. *sigh*

So yeah, 70 = 0x46 = SS | USER | WRITE. Which is way more sane than SS | PK | RSVD. :-D

While I'm here, the theory is that hardware is setting the SS bit because the faulting access really was a userspace shadow stack access. Which was obviously unexpected by us, but might be working as intended?