Bad performance since 5.9-rc1
Hi,
in my initial report (https://marc.info/?l=kvm&m=160502183220080&w=2 - now fixed by c887c9b9ca62c051d339b1c7b796edf2724029ed) I saw degraded performance going back somewhere between v5.8 - v5.9-rc1.
OpenBSD 6.8 (GENERIC.MP) guest performance (time ./test-build.sh)
good: 0m13.54s real 0m10.51s user 0m10.96s system
bad : 6m20.07s real 11m42.93s user 0m13.57s system
bisected to first bad commit: 6b82ef2c9cf18a48726e4bb359aa9014632f6466
git bisect log:
# bad: [e47c4aee5bde03e7018f4fde45ba21028a8f8438] KVM: x86/mmu: Rename page_header() to to_shadow_page() # good:
[01c3b2b5cdae39af8dfcf6e40fdf484ae0e812c5] KVM: SVM: Rename svm_nested_virtualize_tpr() to nested_svm_virtualize_tpr() git bisect start 'e47c4aee5bde' '01c3b2b5cdae' # bad:
[ebdb292dac7993425c8e31e2c21c9978e914a676] KVM: x86/mmu: Batch zap MMU pages when shrinking the slab git bisect bad ebdb292dac7993425c8e31e2c21c9978e914a676 # good:
[fb58a9c345f645f1774dcf6a36fda169253008ae] KVM: x86/mmu: Optimize MMU page cache lookup for fully direct MMUs git bisect good fb58a9c345f645f1774dcf6a36fda169253008ae # bad:
[6b82ef2c9cf18a48726e4bb359aa9014632f6466] KVM: x86/mmu: Batch zap MMU pages when recycling oldest pages git bisect bad 6b82ef2c9cf18a48726e4bb359aa9014632f6466 # good:
[f95eec9bed76d42194c23153cb1cc8f186bf91cb] KVM: x86/mmu: Don't put invalid SPs back on the list of active pages git bisect good f95eec9bed76d42194c23153cb1cc8f186bf91cb # first bad commit:
[6b82ef2c9cf18a48726e4bb359aa9014632f6466] KVM: x86/mmu: Batch zap MMU pages when recycling oldest pages
Host machine is old Intel Core2 without EPT (TDP).
TIA, Z.
Quoted 33 lines · On Thu, 19 Nov 2020 04:05:26 +0100 Zdenek Kaspar <[email protected]> wrote:
Hi,
in my initial report (https://marc.info/?l=kvm&m=160502183220080&w=2 - now fixed by
c887c9b9ca62c051d339b1c7b796edf2724029ed) I saw degraded performance going back somewhere between v5.8 - v5.9-rc1.OpenBSD 6.8 (GENERIC.MP) guest performance (time ./test-build.sh)
good: 0m13.54s real 0m10.51s user 0m10.96s system
bad : 6m20.07s real 11m42.93s user 0m13.57s systembisected to first bad commit:
6b82ef2c9cf18a48726e4bb359aa9014632f6466git bisect log:
# bad: [e47c4aee5bde03e7018f4fde45ba21028a8f8438] KVM: x86/mmu: Renamepage_header()toto_shadow_page()# good:
[01c3b2b5cdae39af8dfcf6e40fdf484ae0e812c5] KVM: SVM: Renamesvm_nested_virtualize_tpr()tonested_svm_virtualize_tpr()git bisect start 'e47c4aee5bde' '01c3b2b5cdae' # bad:
[ebdb292dac7993425c8e31e2c21c9978e914a676] KVM: x86/mmu: Batch zap MMU pages when shrinking the slab git bisect badebdb292dac7993425c8e31e2c21c9978e914a676# good:
[fb58a9c345f645f1774dcf6a36fda169253008ae] KVM: x86/mmu: Optimize MMU page cache lookup for fully direct MMUs git bisect goodfb58a9c345f645f1774dcf6a36fda169253008ae# bad:
[6b82ef2c9cf18a48726e4bb359aa9014632f6466] KVM: x86/mmu: Batch zap MMU pages when recycling oldest pages git bisect bad6b82ef2c9cf18a48726e4bb359aa9014632f6466# good:
[f95eec9bed76d42194c23153cb1cc8f186bf91cb] KVM: x86/mmu: Don't put invalid SPs back on the list of active pages git bisect goodf95eec9bed76d42194c23153cb1cc8f186bf91cb# first bad commit:
[6b82ef2c9cf18a48726e4bb359aa9014632f6466] KVM: x86/mmu: Batch zap MMU pages when recycling oldest pages
Quoted 3 lines
Host machine is old Intel Core2 without EPT (TDP).
TIA, Z.
Hi, with v5.10-rc6:get_mmio_spte: detect reserved bits on spte, addr 0xfe00d000, dump hierarchy:
------ spte 0x8000030e level 3.
------ spte 0xaf82027 level 2.
------ spte 0x2038001ffe00d407 level 1.
------------[ cut here ]------------ WARNING: CPU: 1 PID: 355 at kvm_mmu_page_fault.cold+0x42/0x4f [kvm] ... CPU: 1 PID: 355 Comm: qemu-build Not tainted 5.10.0-rc6-amd64 #1 Hardware name: /DG35EC, BIOS ECG3510M.86A.0118.2010.0113.1426 01/13/2010 RIP: 0010:kvm_mmu_page_fault.cold+0x42/0x4f [kvm] Code: e2 ec 44 8b 04 24 8b 5c 24 0c 44 89 c5 89 da 83 eb 01 48 c7 c7 20 b2 65 c0 48 63 c3 48 8b 74 c4 30 e8 dd 74 e2 ec 39 dd 7e e3 <0f> 0b 41 b8 ea ff ff ff e9 27 99 ff ff 0f 0b 48 8b 54 24 10 48 83 RSP: 0018:ffffb67400653d30 EFLAGS: 00010202 RAX: 0000000000000027 RBX: 0000000000000000 RCX: ffffa271ff2976f8 RDX: 00000000ffffffd8 RSI: 0000000000000027 RDI: ffffa271ff2976f0 RBP: 0000000000000001 R08: ffffffffadd02ae8 R09: 0000000000000003 R10: 00000000ffffe000 R11: 3fffffffffffffff R12: 00000000fe00d000 R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000001 FS: 00007fc10ae3d640(0000) GS:ffffa271ff280000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 0000000002dc2000 CR4: 00000000000026e0 Call Trace: kvm_arch_vcpu_ioctl_run+0xbaf/0x18f0 [kvm] ? do_futex+0x7c4/0xb80 kvm_vcpu_ioctl+0x203/0x520 [kvm] ? set_next_entity+0x5b/0x80 ? __switch_to_asm+0x32/0x60 ? finish_task_switch+0x70/0x260 __x64_sys_ioctl+0x338/0x720 ? __x64_sys_futex+0x120/0x190 do_syscall_64+0x33/0x40 entry_SYSCALL_64_after_hwframe+0x44/0xa9 RIP: 0033:0x7fc10c389f6b Code: 89 d8 49 8d 3c 1c 48 f7 d8 49 39 c4 72 b5 e8 1c ff ff ff 85 c0 78 ba 4c 89 e0 5b 5d 41 5c c3 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d d5 ae 0c 00 f7 d8 64 89 01 48 RSP: 002b:00007fc10ae3c628 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 RAX: ffffffffffffffda RBX: 000000000000ae80 RCX: 00007fc10c389f6b RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000012 RBP: 000055ad3767baf0 R08: 000055ad36be4850 R09: 00000000000000ff R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000 R13: 000055ad371d9800 R14: 0000000000000001 R15: 0000000000000002 ---[ end trace c5f7ae690f5abcc4 ]---
without: kvm: x86/mmu: Fix get_mmio_spte() on CPUs supporting 5-level PT I can run guest again, but with degraded performance as before.
Z.
Quoted 84 lines · On Tue, 1 Dec 2020 07:35:37 +0100 Zdenek Kaspar <[email protected]> wrote:
Quoted 35 lines · On Thu, 19 Nov 2020 04:05:26 +0100 Zdenek Kaspar <[email protected]> wrote:
Hi,
in my initial report
(https://marc.info/?l=kvm&m=160502183220080&w=2 - now fixed byc887c9b9ca62c051d339b1c7b796edf2724029ed) I saw degraded performance going back somewhere between v5.8 - v5.9-rc1.OpenBSD 6.8 (GENERIC.MP) guest performance (time ./test-build.sh)
good: 0m13.54s real 0m10.51s user 0m10.96s system
bad : 6m20.07s real 11m42.93s user 0m13.57s systembisected to first bad commit:
6b82ef2c9cf18a48726e4bb359aa9014632f6466git bisect log:
# bad: [e47c4aee5bde03e7018f4fde45ba21028a8f8438] KVM: x86/mmu:
Renamepage_header()toto_shadow_page()# good:
[01c3b2b5cdae39af8dfcf6e40fdf484ae0e812c5] KVM: SVM: Renamesvm_nested_virtualize_tpr()tonested_svm_virtualize_tpr()git bisect start 'e47c4aee5bde' '01c3b2b5cdae' # bad:
[ebdb292dac7993425c8e31e2c21c9978e914a676] KVM: x86/mmu: Batch zap MMU pages when shrinking the slab git bisect badebdb292dac7993425c8e31e2c21c9978e914a676# good:
[fb58a9c345f645f1774dcf6a36fda169253008ae] KVM: x86/mmu: Optimize MMU page cache lookup for fully direct MMUs git bisect goodfb58a9c345f645f1774dcf6a36fda169253008ae# bad:
[6b82ef2c9cf18a48726e4bb359aa9014632f6466] KVM: x86/mmu: Batch zap MMU pages when recycling oldest pages git bisect bad6b82ef2c9cf18a48726e4bb359aa9014632f6466# good:
[f95eec9bed76d42194c23153cb1cc8f186bf91cb] KVM: x86/mmu: Don't put invalid SPs back on the list of active pages git bisect goodf95eec9bed76d42194c23153cb1cc8f186bf91cb# first bad commit:
[6b82ef2c9cf18a48726e4bb359aa9014632f6466] KVM: x86/mmu: Batch zap MMU pages when recycling oldest pagesQuoted 3 lines
Host machine is old Intel Core2 without EPT (TDP).
TIA, Z.
Hi, with v5.10-rc6:
get_mmio_spte: detect reserved bits on spte, addr0xfe00d000, dump
hierarchy: ------ spte0x8000030elevel 3.
------ spte0xaf82027level 2.
------ spte0x2038001ffe00d407level 1.------------[ cut here ]------------ WARNING: CPU: 1 PID: 355 at kvm_mmu_page_fault.cold+0x42/0x4f [kvm] ... CPU: 1 PID: 355 Comm: qemu-build Not tainted 5.10.0-rc6-amd64 #1 Hardware name: /DG35EC, BIOS ECG3510M.86A.0118.2010.0113.1426 01/13/2010 RIP: 0010:kvm_mmu_page_fault.cold+0x42/0x4f [kvm] Code: e2 ec 44 8b 04 24 8b 5c 24 0c 44 89 c5 89 da 83 eb 01 48 c7 c720 b2 65 c0 48 63 c3 48 8b 74 c4 30 e8 dd 74 e2 ec 39 dd 7e e3 <0f> 0b 41 b8 ea ff ff ff e9 27 99 ff ff 0f 0b 48 8b 54 24 10 48 83 RSP:
0018:ffffb67400653d30 EFLAGS: 00010202 RAX: 0000000000000027 RBX:
0000000000000000 RCX:ffffa271ff2976f8RDX:00000000ffffffd8RSI:
0000000000000027 RDI:ffffa271ff2976f0RBP: 0000000000000001 R08:ffffffffadd02ae8R09: 0000000000000003 R10:00000000ffffe000R11:3fffffffffffffffR12:00000000fe00d000R13: 0000000000000000 R14:
0000000000000000 R15: 0000000000000001 FS: 00007fc10ae3d640(0000) GS:ffffa271ff280000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000
ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3:0000000002dc2000CR4:00000000000026e0Call Trace:kvm_arch_vcpu_ioctl_run+0xbaf/0x18f0 [kvm] ? do_futex+0x7c4/0xb80 kvm_vcpu_ioctl+0x203/0x520 [kvm] ? set_next_entity+0x5b/0x80 ? __switch_to_asm+0x32/0x60 ? finish_task_switch+0x70/0x260 __x64_sys_ioctl+0x338/0x720 ? __x64_sys_futex+0x120/0x190 do_syscall_64+0x33/0x40 entry_SYSCALL_64_after_hwframe+0x44/0xa9 RIP: 0033:0x7fc10c389f6b Code: 89 d8 49 8d 3c 1c 48 f7 d8 49 39 c4 72 b5 e8 1c ff ff ff 85 c078 ba 4c 89 e0 5b 5d 41 5c c3 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d d5 ae 0c 00 f7 d8 64 89 01 48 RSP:
002b:00007fc10ae3c628 EFLAGS: 00000246ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX:000000000000ae80RCX:00007fc10c389f6b
RDX: 0000000000000000 RSI:000000000000ae80RDI: 0000000000000012
RBP:000055ad3767baf0R08:000055ad36be4850R09:00000000000000ff
R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
R13:000055ad371d9800R14: 0000000000000001 R15: 0000000000000002
Quoted 6 lines
---[ end trace c5f7ae690f5abcc4 ]---without: kvm: x86/mmu: Fix
get_mmio_spte()on CPUs supporting 5-level PT I can run guest again, but with degraded performance as before.Z.
With: KVM: x86/mmu: Bug fixes and cleanups in get_mmio_spte() series I can run guest again and performance is slightly better:
v5.8: 0m13.54s real 0m10.51s user 0m10.96s system v5.9: 6m20.07s real 11m42.93s user 0m13.57s system
v5.10+fixes: 5m50.77s real 10m38.29s user 0m15.96s system
perf top from host when guest (openbsd) is compiling:
26.85% [kernel] [k] queued_spin_lock_slowpath 8.49% [kvm] [k] mmu_page_zap_pte 7.47% [kvm] [k] __kvm_mmu_prepare_zap_page 3.61% [kernel] [k] clear_page_rep 2.43% [kernel] [k] page_counter_uncharge 2.30% [kvm] [k] paging64_page_fault 2.03% [kvm_intel] [k] vmx_vcpu_run 2.02% [kvm] [k] kvm_vcpu_gfn_to_memslot 1.95% [kernel] [k] internal_get_user_pages_fast 1.64% [kvm] [k] kvm_mmu_get_page 1.55% [kernel] [k] page_counter_try_charge 1.33% [kernel] [k] propagate_protected_usage 1.29% [kvm] [k] kvm_arch_vcpu_ioctl_run 1.13% [kernel] [k] get_page_from_freelist 1.01% [kvm] [k] paging64_walk_addr_generic 0.83% [kernel] [k] ___slab_alloc.constprop.0 0.83% [kernel] [k] kmem_cache_free 0.82% [kvm] [k] __pte_list_remove 0.77% [kernel] [k] try_grab_compound_head 0.76% [kvm_intel] [k] 0x000000000001cfa0 0.74% [kvm] [k] pte_list_add
HTH, Z.
Sean Christopherson
maintainerQuoted 6 lines · Zdenek Kaspar
Quoted 4 lines
without: kvm: x86/mmu: Fix
get_mmio_spte()on CPUs supporting 5-level PT I can run guest again, but with degraded performance as before.Z.
With: KVM: x86/mmu: Bug fixes and cleanups in
get_mmio_spte()series
Apologies, I completely missed your bug report for the get_mmio_spte() bugs.
Quoted 20 lines
I can run guest again and performance is slightly better:
v5.8: 0m13.54s real 0m10.51s user 0m10.96s system v5.9: 6m20.07s real 11m42.93s user 0m13.57s systemv5.10+fixes: 5m50.77s real 10m38.29s user 0m15.96s system
perf top from host when guest (openbsd) is compiling:
26.85% [kernel] [k] queued_spin_lock_slowpath 8.49% [kvm] [k] mmu_page_zap_pte 7.47% [kvm] [k] __kvm_mmu_prepare_zap_page 3.61% [kernel] [k] clear_page_rep 2.43% [kernel] [k] page_counter_uncharge 2.30% [kvm] [k] paging64_page_fault 2.03% [kvm_intel] [k] vmx_vcpu_run 2.02% [kvm] [k] kvm_vcpu_gfn_to_memslot 1.95% [kernel] [k] internal_get_user_pages_fast 1.64% [kvm] [k] kvm_mmu_get_page 1.55% [kernel] [k] page_counter_try_charge 1.33% [kernel] [k] propagate_protected_usage 1.29% [kvm] [k] kvm_arch_vcpu_ioctl_run
Quoted 8 lines
1.13% [kernel] [k] get_page_from_freelist 1.01% [kvm] [k] paging64_walk_addr_generic 0.83% [kernel] [k] ___slab_alloc.constprop.0 0.83% [kernel] [k] kmem_cache_free 0.82% [kvm] [k] __pte_list_remove 0.77% [kernel] [k] try_grab_compound_head 0.76% [kvm_intel] [k] 0x000000000001cfa0 0.74% [kvm] [k] pte_list_add
Can you try running with this debug hack to understand what is causing KVM to zap shadow pages? The expected behavior is that you'll get backtraces for the first five cases where KVM zaps valid shadow pages. Compile tested only.
arch/x86/kvm/mmu/mmu.c +4 -0
@@ -2404,6 +2404,8 @@ static void kvm_mmu_commit_zap_page(struct kvm *kvm, } } +static unsigned long zapped_warns; +static unsigned long kvm_mmu_zap_oldest_mmu_pages (struct kvm *kvm,unsigned long nr_to_zap) {
@@ -2435,6 +2437,8 @@ static unsigned long kvm_mmu_zap_oldest_mmu_pages(struct kvm *kvm, goto restart; } +WARN_ON (total_zapped && zapped_warns++ <5 ); +kvm_mmu_commit_zap_page (kvm, &invalid_list); kvm->stat.mmu_recycled += total_zapped;
Quoted 61 lines · On Mon, 21 Dec 2020 11:41:44 -0800 Sean Christopherson <[email protected]>…
Quoted 7 lines · Zdenek Kaspar
Quoted 5 lines
without: kvm: x86/mmu: Fix
get_mmio_spte()on CPUs supporting 5-level PT I can run guest again, but with degraded performance as before.Z.
With: KVM: x86/mmu: Bug fixes and cleanups in
get_mmio_spte()seriesApologies, I completely missed your bug report for the
get_mmio_spte()bugs.Quoted 20 lines
I can run guest again and performance is slightly better:
v5.8: 0m13.54s real 0m10.51s user 0m10.96s system v5.9: 6m20.07s real 11m42.93s user 0m13.57s systemv5.10+fixes: 5m50.77s real 10m38.29s user 0m15.96s system
perf top from host when guest (openbsd) is compiling:
26.85% [kernel] [k] queued_spin_lock_slowpath 8.49% [kvm] [k] mmu_page_zap_pte 7.47% [kvm] [k] __kvm_mmu_prepare_zap_page 3.61% [kernel] [k] clear_page_rep 2.43% [kernel] [k] page_counter_uncharge 2.30% [kvm] [k] paging64_page_fault 2.03% [kvm_intel] [k] vmx_vcpu_run 2.02% [kvm] [k] kvm_vcpu_gfn_to_memslot 1.95% [kernel] [k] internal_get_user_pages_fast 1.64% [kvm] [k] kvm_mmu_get_page 1.55% [kernel] [k] page_counter_try_charge 1.33% [kernel] [k] propagate_protected_usage 1.29% [kvm] [k] kvm_arch_vcpu_ioctl_runQuoted 8 lines
1.13% [kernel] [k] get_page_from_freelist 1.01% [kvm] [k] paging64_walk_addr_generic 0.83% [kernel] [k] ___slab_alloc.constprop.0 0.83% [kernel] [k] kmem_cache_free 0.82% [kvm] [k] __pte_list_remove 0.77% [kernel] [k] try_grab_compound_head 0.76% [kvm_intel] [k] 0x000000000001cfa0 0.74% [kvm] [k] pte_list_addCan you try running with this debug hack to understand what is causing KVM to zap shadow pages? The expected behavior is that you'll get backtraces for the first five cases where KVM zaps valid shadow pages. Compile tested only.
arch/x86/kvm/mmu/mmu.c +2
-0@@ -2404,6 +2404,8 @@ static void kvm_mmu_commit_zap_page(struct kvm *kvm, } } +static unsigned long zapped_warns; +static unsigned long kvm_mmu_zap_oldest_mmu_pages (struct kvm *kvm,unsigned long nr_to_zap) {
arch/x86/kvm/mmu/mmu.c static unsigned long · line 2444
Quoted 8 lines
arch/x86/kvm/mmu/mmu.c +2
-0@@ -2435,6 +2437,8 @@ static unsigned long kvm_mmu_zap_oldest_mmu_pages(struct kvm *kvm, goto restart; } +WARN_ON (total_zapped && zapped_warns++ <5 ); +kvm_mmu_commit_zap_page (kvm, &invalid_list); kvm->stat.mmu_recycled += total_zapped;
[ 179.364234] ------------[ cut here ]------------ [ 179.364305] WARNING: CPU: 0 PID: 369 at kvm_mmu_zap_oldest_mmu_pages+0xd1/0xe0 [kvm] [ 179.364347] Modules linked in: vhost_net vhost vhost_iotlb tun auth_rpcgss nfsv4 dns_resolver nfs lockd grace sunrpc nfs_ssc lzo zram zsmalloc cpufreq_powersave i915 kvm_intel video intel_gtt iosf_mbi kvm i2c_algo_bit drm_kms_helper bridge stp iTCO_wdt e1000e syscopyarea irqbypass sysfillrect 8250 evdev 8250_base serial_core sysimgblt lpc_ich mfd_core llc button acpi_cpufreq fb_sys_fops processor drm i2c_core sch_fq_codel backlight ip_tables x_tables ipv6 autofs4 btrfs blake2b_generic libcrc32c crc32c_generic xor zstd_decompress zstd_compress lzo_compress lzo_decompress raid6_pq ecb xts dm_crypt dm_mod sd_mod t10_pi hid_generic usbhid hid uhci_hcd ahci libahci pata_jmicron ehci_pci ehci_hcd sata_sil24 usbcore usb_common [ 179.364818] CPU: 0 PID: 369 Comm: qemu-build Not tainted 5.10.2-1-amd64 #1 [ 179.364857] Hardware name: /DG35EC, BIOS ECG3510M.86A.0118.2010.0113.1426 01/13/2010 [ 179.364923] RIP: 0010:kvm_mmu_zap_oldest_mmu_pages+0xd1/0xe0 [kvm] [ 179.364959] Code: 48 83 c4 18 4c 89 f0 5b 5d 41 5c 41 5d 41 5e 41 5f c3 48 8b 05 a0 6b 03 00 48 8d 50 01 48 83 f8 04 48 89 15 91 6b 03 00 77 b9 <0f> 0b eb b5 45 31 f6 eb cd 66 0f 1f 44 00 00 41 57 48 c7 c7 e0 23 [ 179.365065] RSP: 0018:ffffab7e8069bb10 EFLAGS: 00010297 [ 179.365097] RAX: 0000000000000000 RBX: ffff8fd62c589c78 RCX: 0000000000000000 [ 179.365138] RDX: 0000000000000001 RSI: ffffffffffffffff RDI: 00003ba800a00478 [ 179.365179] RBP: ffffab7e8073da78 R08: ffff8fd608d06800 R09: 000000000000000a [ 179.365218] R10: ffff8fd608d06800 R11: 000000000000000a R12: ffffab7e80735000 [ 179.365257] R13: 0000000000000015 R14: 0000000000000015 R15: ffffab7e8069bb18 [ 179.365299] FS: 00007f22e4b6a640(0000) GS:ffff8fd67f200000(0000) knlGS:0000000000000000 [ 179.365343] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 179.365376] CR2: 0000098452a94c00 CR3: 000000000a3c0000 CR4: 00000000000026f0 [ 179.365415] Call Trace: [ 179.365443] paging64_page_fault+0x244/0x8e0 [kvm] [ 179.365482] ? kvm_mmu_pte_write+0x161/0x410 [kvm] [ 179.365521] ? write_emulate+0x36/0x50 [kvm] [ 179.365558] ? kvm_fetch_guest_virt+0x7c/0xb0 [kvm] [ 179.365596] kvm_mmu_page_fault+0x376/0x550 [kvm] [ 179.365628] ? vmx_vmexit+0x1d/0x40 [kvm_intel] [ 179.365655] ? vmx_vmexit+0x11/0x40 [kvm_intel] [ 179.365683] ? vmx_vcpu_enter_exit+0x5c/0x90 [kvm_intel] [ 179.365725] kvm_arch_vcpu_ioctl_run+0xbaf/0x18f0 [kvm] [ 179.365772] kvm_vcpu_ioctl+0x203/0x520 [kvm] [ 179.365801] ? tick_sched_timer+0x69/0xf0 [ 179.365823] ? tick_nohz_handler+0xf0/0xf0 [ 179.365848] ? timerqueue_add+0x96/0xb0 [ 179.365870] ? __hrtimer_run_queues+0x151/0x1b0 [ 179.365895] ? recalibrate_cpu_khz+0x10/0x10 [ 179.365918] ? ktime_get+0x33/0x90 [ 179.365938] __x64_sys_ioctl+0x338/0x720 [ 179.365963] ? fire_user_return_notifiers+0x3c/0x60 [ 179.365992] do_syscall_64+0x33/0x40 [ 179.366013] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 179.366041] RIP: 0033:0x7f22e60cef6b [ 179.366061] Code: 89 d8 49 8d 3c 1c 48 f7 d8 49 39 c4 72 b5 e8 1c ff ff ff 85 c0 78 ba 4c 89 e0 5b 5d 41 5c c3 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d d5 ae 0c 00 f7 d8 64 89 01 48 [ 179.366167] RSP: 002b:00007f22e4b69608 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [ 179.366210] RAX: ffffffffffffffda RBX: 000000000000ae80 RCX: 00007f22e60cef6b [ 179.366248] RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000012 [ 179.366287] RBP: 000055a81d11cf70 R08: 000055a81ac42ad8 R09: 00000000000000ff [ 179.366326] R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000 [ 179.366365] R13: 00007f22e6c35001 R14: 0000000000000064 R15: 0000000000000000 [ 179.366405] ---[ end trace 63d1ba11f1bc6180 ]--- [ 179.367537] ------------[ cut here ]------------ [ 179.367583] WARNING: CPU: 0 PID: 369 at kvm_mmu_zap_oldest_mmu_pages+0xd1/0xe0 [kvm] [ 179.367625] Modules linked in: vhost_net vhost vhost_iotlb tun auth_rpcgss nfsv4 dns_resolver nfs lockd grace sunrpc nfs_ssc lzo zram zsmalloc cpufreq_powersave i915 kvm_intel video intel_gtt iosf_mbi kvm i2c_algo_bit drm_kms_helper bridge stp iTCO_wdt e1000e syscopyarea irqbypass sysfillrect 8250 evdev 8250_base serial_core sysimgblt lpc_ich mfd_core llc button acpi_cpufreq fb_sys_fops processor drm i2c_core sch_fq_codel backlight ip_tables x_tables ipv6 autofs4 btrfs blake2b_generic libcrc32c crc32c_generic xor zstd_decompress zstd_compress lzo_compress lzo_decompress raid6_pq ecb xts dm_crypt dm_mod sd_mod t10_pi hid_generic usbhid hid uhci_hcd ahci libahci pata_jmicron ehci_pci ehci_hcd sata_sil24 usbcore usb_common [ 179.375035] CPU: 0 PID: 369 Comm: qemu-build Tainted: G W 5.10.2-1-amd64 #1 [ 179.377470] Hardware name: /DG35EC, BIOS ECG3510M.86A.0118.2010.0113.1426 01/13/2010 [ 179.379948] RIP: 0010:kvm_mmu_zap_oldest_mmu_pages+0xd1/0xe0 [kvm] [ 179.382436] Code: 48 83 c4 18 4c 89 f0 5b 5d 41 5c 41 5d 41 5e 41 5f c3 48 8b 05 a0 6b 03 00 48 8d 50 01 48 83 f8 04 48 89 15 91 6b 03 00 77 b9 <0f> 0b eb b5 45 31 f6 eb cd 66 0f 1f 44 00 00 41 57 48 c7 c7 e0 23 [ 179.387680] RSP: 0018:ffffab7e8069bb10 EFLAGS: 00010293 [ 179.390307] RAX: 0000000000000001 RBX: ffff8fd62c589c78 RCX: 0000000000000000 [ 179.392890] RDX: 0000000000000002 RSI: ffffffffffffffff RDI: 00003ba800a00478 [ 179.395420] RBP: ffffab7e8073da78 R08: ffff8fd608d06800 R09: 000000000000000a [ 179.397901] R10: ffff8fd608d06800 R11: 000000000000000a R12: ffffab7e80735000 [ 179.400339] R13: 0000000000000015 R14: 0000000000000015 R15: ffffab7e8069bb18 [ 179.402715] FS: 00007f22e4b6a640(0000) GS:ffff8fd67f200000(0000) knlGS:0000000000000000 [ 179.405097] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 179.407466] CR2: 000009837e65c200 CR3: 000000000a3c0000 CR4: 00000000000026f0 [ 179.409839] Call Trace: [ 179.412140] paging64_page_fault+0x244/0x8e0 [kvm] [ 179.414377] ? kvm_mmu_pte_write+0x161/0x410 [kvm] [ 179.416608] ? write_emulate+0x36/0x50 [kvm] [ 179.418821] ? kvm_fetch_guest_virt+0x7c/0xb0 [kvm] [ 179.421031] kvm_mmu_page_fault+0x376/0x550 [kvm] [ 179.423220] ? vmx_vmexit+0x1d/0x40 [kvm_intel] [ 179.425402] ? vmx_vmexit+0x11/0x40 [kvm_intel] [ 179.427556] ? vmx_vcpu_enter_exit+0x5c/0x90 [kvm_intel] [ 179.429707] kvm_arch_vcpu_ioctl_run+0xbaf/0x18f0 [kvm] [ 179.431838] kvm_vcpu_ioctl+0x203/0x520 [kvm] [ 179.433944] ? tick_sched_timer+0x69/0xf0 [ 179.436029] ? tick_nohz_handler+0xf0/0xf0 [ 179.438108] ? timerqueue_add+0x96/0xb0 [ 179.440185] ? __hrtimer_run_queues+0x151/0x1b0 [ 179.442263] ? recalibrate_cpu_khz+0x10/0x10 [ 179.444339] ? ktime_get+0x33/0x90 [ 179.446406] __x64_sys_ioctl+0x338/0x720 [ 179.448470] ? fire_user_return_notifiers+0x3c/0x60 [ 179.450558] do_syscall_64+0x33/0x40 [ 179.452639] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 179.454728] RIP: 0033:0x7f22e60cef6b [ 179.456812] Code: 89 d8 49 8d 3c 1c 48 f7 d8 49 39 c4 72 b5 e8 1c ff ff ff 85 c0 78 ba 4c 89 e0 5b 5d 41 5c c3 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d d5 ae 0c 00 f7 d8 64 89 01 48 [ 179.461281] RSP: 002b:00007f22e4b69608 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [ 179.463572] RAX: ffffffffffffffda RBX: 000000000000ae80 RCX: 00007f22e60cef6b [ 179.465875] RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000012 [ 179.468185] RBP: 000055a81d11cf70 R08: 000055a81ac42ad8 R09: 00000000000000ff [ 179.470499] R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000 [ 179.472829] R13: 00007f22e6c35001 R14: 0000000000000064 R15: 0000000000000000 [ 179.475154] ---[ end trace 63d1ba11f1bc6181 ]--- [ 179.478464] ------------[ cut here ]------------ [ 179.480804] WARNING: CPU: 0 PID: 369 at kvm_mmu_zap_oldest_mmu_pages+0xd1/0xe0 [kvm] [ 179.483160] Modules linked in: vhost_net vhost vhost_iotlb tun auth_rpcgss nfsv4 dns_resolver nfs lockd grace sunrpc nfs_ssc lzo zram zsmalloc cpufreq_powersave i915 kvm_intel video intel_gtt iosf_mbi kvm i2c_algo_bit drm_kms_helper bridge stp iTCO_wdt e1000e syscopyarea irqbypass sysfillrect 8250 evdev 8250_base serial_core sysimgblt lpc_ich mfd_core llc button acpi_cpufreq fb_sys_fops processor drm i2c_core sch_fq_codel backlight ip_tables x_tables ipv6 autofs4 btrfs blake2b_generic libcrc32c crc32c_generic xor zstd_decompress zstd_compress lzo_compress lzo_decompress raid6_pq ecb xts dm_crypt dm_mod sd_mod t10_pi hid_generic usbhid hid uhci_hcd ahci libahci pata_jmicron ehci_pci ehci_hcd sata_sil24 usbcore usb_common [ 179.495262] CPU: 0 PID: 369 Comm: qemu-build Tainted: G W 5.10.2-1-amd64 #1 [ 179.497745] Hardware name: /DG35EC, BIOS ECG3510M.86A.0118.2010.0113.1426 01/13/2010 [ 179.500251] RIP: 0010:kvm_mmu_zap_oldest_mmu_pages+0xd1/0xe0 [kvm] [ 179.502761] Code: 48 83 c4 18 4c 89 f0 5b 5d 41 5c 41 5d 41 5e 41 5f c3 48 8b 05 a0 6b 03 00 48 8d 50 01 48 83 f8 04 48 89 15 91 6b 03 00 77 b9 <0f> 0b eb b5 45 31 f6 eb cd 66 0f 1f 44 00 00 41 57 48 c7 c7 e0 23 [ 179.508004] RSP: 0018:ffffab7e8069bb10 EFLAGS: 00010293 [ 179.510628] RAX: 0000000000000002 RBX: ffff8fd62c589c78 RCX: 0000000000000000 [ 179.513214] RDX: 0000000000000003 RSI: ffffffffffffffff RDI: 00003ba800a00478 [ 179.515747] RBP: ffffab7e8073da78 R08: ffff8fd608d06800 R09: 000000000000000a [ 179.518229] R10: ffff8fd608d06800 R11: 000000000000000a R12: ffffab7e80735000 [ 179.520670] R13: 0000000000000015 R14: 0000000000000015 R15: ffffab7e8069bb18 [ 179.523049] FS: 00007f22e4b6a640(0000) GS:ffff8fd67f200000(0000) knlGS:0000000000000000 [ 179.525431] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 179.527799] CR2: 000009844140de00 CR3: 000000000a3c0000 CR4: 00000000000026f0 [ 179.530172] Call Trace: [ 179.532471] paging64_page_fault+0x244/0x8e0 [kvm] [ 179.534709] ? kvm_fetch_guest_virt+0x7c/0xb0 [kvm] [ 179.536946] kvm_mmu_page_fault+0x376/0x550 [kvm] [ 179.539160] ? vmx_vmexit+0x1d/0x40 [kvm_intel] [ 179.541366] ? vmx_vmexit+0x11/0x40 [kvm_intel] [ 179.543544] ? vmx_vcpu_enter_exit+0x5c/0x90 [kvm_intel] [ 179.545720] kvm_arch_vcpu_ioctl_run+0xbaf/0x18f0 [kvm] [ 179.547890] kvm_vcpu_ioctl+0x203/0x520 [kvm] [ 179.550033] ? tick_sched_timer+0x69/0xf0 [ 179.552153] ? tick_nohz_handler+0xf0/0xf0 [ 179.554250] ? timerqueue_add+0x96/0xb0 [ 179.556329] ? __hrtimer_run_queues+0x151/0x1b0 [ 179.558407] ? recalibrate_cpu_khz+0x10/0x10 [ 179.560486] ? ktime_get+0x33/0x90 [ 179.562557] __x64_sys_ioctl+0x338/0x720 [ 179.564628] ? fire_user_return_notifiers+0x3c/0x60 [ 179.566702] do_syscall_64+0x33/0x40 [ 179.568772] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 179.570867] RIP: 0033:0x7f22e60cef6b [ 179.572957] Code: 89 d8 49 8d 3c 1c 48 f7 d8 49 39 c4 72 b5 e8 1c ff ff ff 85 c0 78 ba 4c 89 e0 5b 5d 41 5c c3 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d d5 ae 0c 00 f7 d8 64 89 01 48 [ 179.577419] RSP: 002b:00007f22e4b69608 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [ 179.579699] RAX: ffffffffffffffda RBX: 000000000000ae80 RCX: 00007f22e60cef6b [ 179.582000] RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000012 [ 179.584312] RBP: 000055a81d11cf70 R08: 000055a81ac42ad8 R09: 00000000000000ff [ 179.586632] R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000 [ 179.588955] R13: 00007f22e6c35001 R14: 0000000000000064 R15: 0000000000000000 [ 179.591283] ---[ end trace 63d1ba11f1bc6182 ]--- [ 179.595542] ------------[ cut here ]------------ [ 179.597890] WARNING: CPU: 0 PID: 369 at kvm_mmu_zap_oldest_mmu_pages+0xd1/0xe0 [kvm] [ 179.600253] Modules linked in: vhost_net vhost vhost_iotlb tun auth_rpcgss nfsv4 dns_resolver nfs lockd grace sunrpc nfs_ssc lzo zram zsmalloc cpufreq_powersave i915 kvm_intel video intel_gtt iosf_mbi kvm i2c_algo_bit drm_kms_helper bridge stp iTCO_wdt e1000e syscopyarea irqbypass sysfillrect 8250 evdev 8250_base serial_core sysimgblt lpc_ich mfd_core llc button acpi_cpufreq fb_sys_fops processor drm i2c_core sch_fq_codel backlight ip_tables x_tables ipv6 autofs4 btrfs blake2b_generic libcrc32c crc32c_generic xor zstd_decompress zstd_compress lzo_compress lzo_decompress raid6_pq ecb xts dm_crypt dm_mod sd_mod t10_pi hid_generic usbhid hid uhci_hcd ahci libahci pata_jmicron ehci_pci ehci_hcd sata_sil24 usbcore usb_common [ 179.612747] CPU: 0 PID: 369 Comm: qemu-build Tainted: G W 5.10.2-1-amd64 #1 [ 179.615250] Hardware name: /DG35EC, BIOS ECG3510M.86A.0118.2010.0113.1426 01/13/2010 [ 179.617769] RIP: 0010:kvm_mmu_zap_oldest_mmu_pages+0xd1/0xe0 [kvm] [ 179.620283] Code: 48 83 c4 18 4c 89 f0 5b 5d 41 5c 41 5d 41 5e 41 5f c3 48 8b 05 a0 6b 03 00 48 8d 50 01 48 83 f8 04 48 89 15 91 6b 03 00 77 b9 <0f> 0b eb b5 45 31 f6 eb cd 66 0f 1f 44 00 00 41 57 48 c7 c7 e0 23 [ 179.625553] RSP: 0018:ffffab7e8069bb10 EFLAGS: 00010297 [ 179.628194] RAX: 0000000000000003 RBX: ffff8fd62c589c78 RCX: 0000000000000000 [ 179.630866] RDX: 0000000000000004 RSI: ffffffffffffffff RDI: 00003ba800a00478 [ 179.633522] RBP: ffffab7e8073da78 R08: ffff8fd608d06800 R09: 000000000000000a [ 179.636131] R10: ffff8fd608d06800 R11: 000000000000000a R12: ffffab7e80735000 [ 179.638696] R13: 0000000000000015 R14: 0000000000000015 R15: ffffab7e8069bb18 [ 179.641201] FS: 00007f22e4b6a640(0000) GS:ffff8fd67f200000(0000) knlGS:0000000000000000 [ 179.643671] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 179.646085] CR2: 00000983af128a00 CR3: 000000000a3c0000 CR4: 00000000000026f0 [ 179.648504] Call Trace: [ 179.650901] paging64_page_fault+0x244/0x8e0 [kvm] [ 179.653291] ? kvm_mmu_pte_write+0x161/0x410 [kvm] [ 179.655605] ? write_emulate+0x36/0x50 [kvm] [ 179.657838] ? kvm_fetch_guest_virt+0x7c/0xb0 [kvm] [ 179.660073] kvm_mmu_page_fault+0x376/0x550 [kvm] [ 179.662286] ? vmx_vmexit+0x1d/0x40 [kvm_intel] [ 179.664491] ? vmx_vmexit+0x11/0x40 [kvm_intel] [ 179.666668] ? vmx_vcpu_enter_exit+0x5c/0x90 [kvm_intel] [ 179.668842] kvm_arch_vcpu_ioctl_run+0xbaf/0x18f0 [kvm] [ 179.671013] kvm_vcpu_ioctl+0x203/0x520 [kvm] [ 179.673155] ? tick_sched_timer+0x69/0xf0 [ 179.675276] ? tick_nohz_handler+0xf0/0xf0 [ 179.677373] ? timerqueue_add+0x96/0xb0 [ 179.679455] ? __hrtimer_run_queues+0x151/0x1b0 [ 179.681534] ? recalibrate_cpu_khz+0x10/0x10 [ 179.683611] ? ktime_get+0x33/0x90 [ 179.685676] __x64_sys_ioctl+0x338/0x720 [ 179.687741] ? fire_user_return_notifiers+0x3c/0x60 [ 179.689829] do_syscall_64+0x33/0x40 [ 179.691909] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 179.693999] RIP: 0033:0x7f22e60cef6b [ 179.696081] Code: 89 d8 49 8d 3c 1c 48 f7 d8 49 39 c4 72 b5 e8 1c ff ff ff 85 c0 78 ba 4c 89 e0 5b 5d 41 5c c3 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d d5 ae 0c 00 f7 d8 64 89 01 48 [ 179.700547] RSP: 002b:00007f22e4b69608 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [ 179.702836] RAX: ffffffffffffffda RBX: 000000000000ae80 RCX: 00007f22e60cef6b [ 179.705138] RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000012 [ 179.707447] RBP: 000055a81d11cf70 R08: 000055a81ac42ad8 R09: 00000000000000ff [ 179.709767] R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000 [ 179.712099] R13: 00007f22e6c35001 R14: 0000000000000064 R15: 0000000000000000 [ 179.714422] ---[ end trace 63d1ba11f1bc6183 ]--- [ 179.720536] ------------[ cut here ]------------ [ 179.722911] WARNING: CPU: 0 PID: 369 at kvm_mmu_zap_oldest_mmu_pages+0xd1/0xe0 [kvm] [ 179.725273] Modules linked in: vhost_net vhost vhost_iotlb tun auth_rpcgss nfsv4 dns_resolver nfs lockd grace sunrpc nfs_ssc lzo zram zsmalloc cpufreq_powersave i915 kvm_intel video intel_gtt iosf_mbi kvm i2c_algo_bit drm_kms_helper bridge stp iTCO_wdt e1000e syscopyarea irqbypass sysfillrect 8250 evdev 8250_base serial_core sysimgblt lpc_ich mfd_core llc button acpi_cpufreq fb_sys_fops processor drm i2c_core sch_fq_codel backlight ip_tables x_tables ipv6 autofs4 btrfs blake2b_generic libcrc32c crc32c_generic xor zstd_decompress zstd_compress lzo_compress lzo_decompress raid6_pq ecb xts dm_crypt dm_mod sd_mod t10_pi hid_generic usbhid hid uhci_hcd ahci libahci pata_jmicron ehci_pci ehci_hcd sata_sil24 usbcore usb_common [ 179.737395] CPU: 0 PID: 369 Comm: qemu-build Tainted: G W 5.10.2-1-amd64 #1 [ 179.739880] Hardware name: /DG35EC, BIOS ECG3510M.86A.0118.2010.0113.1426 01/13/2010 [ 179.742391] RIP: 0010:kvm_mmu_zap_oldest_mmu_pages+0xd1/0xe0 [kvm] [ 179.744905] Code: 48 83 c4 18 4c 89 f0 5b 5d 41 5c 41 5d 41 5e 41 5f c3 48 8b 05 a0 6b 03 00 48 8d 50 01 48 83 f8 04 48 89 15 91 6b 03 00 77 b9 <0f> 0b eb b5 45 31 f6 eb cd 66 0f 1f 44 00 00 41 57 48 c7 c7 e0 23 [ 179.750149] RSP: 0018:ffffab7e8069bb10 EFLAGS: 00010246 [ 179.752776] RAX: 0000000000000004 RBX: ffff8fd62c589c78 RCX: 0000000000000000 [ 179.755366] RDX: 0000000000000005 RSI: ffffffffffffffff RDI: 00003ba800a00478 [ 179.757901] RBP: ffffab7e8073da78 R08: ffff8fd608d06800 R09: 000000000000000a [ 179.760388] R10: ffff8fd608d06800 R11: 000000000000000a R12: ffffab7e80735000 [ 179.762832] R13: 0000000000000015 R14: 0000000000000015 R15: ffffab7e8069bb18 [ 179.765216] FS: 00007f22e4b6a640(0000) GS:ffff8fd67f200000(0000) knlGS:ffffffff820e7ff0 [ 179.767602] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 179.769976] CR2: 000009844cfd4620 CR3: 000000000a3c0000 CR4: 00000000000026f0 [ 179.772351] Call Trace: [ 179.774657] paging64_page_fault+0x244/0x8e0 [kvm] [ 179.776900] ? kvm_fetch_guest_virt+0x7c/0xb0 [kvm] [ 179.779139] kvm_mmu_page_fault+0x376/0x550 [kvm] [ 179.781357] ? vmx_vmexit+0x1d/0x40 [kvm_intel] [ 179.783566] ? vmx_vmexit+0x11/0x40 [kvm_intel] [ 179.785748] ? vmx_vcpu_enter_exit+0x5c/0x90 [kvm_intel] [ 179.787928] kvm_arch_vcpu_ioctl_run+0xbaf/0x18f0 [kvm] [ 179.790105] kvm_vcpu_ioctl+0x203/0x520 [kvm] [ 179.792254] ? tick_sched_timer+0x69/0xf0 [ 179.794377] ? tick_nohz_handler+0xf0/0xf0 [ 179.796478] ? timerqueue_add+0x96/0xb0 [ 179.798560] ? __hrtimer_run_queues+0x151/0x1b0 [ 179.800642] ? recalibrate_cpu_khz+0x10/0x10 [ 179.802724] ? ktime_get+0x33/0x90 [ 179.804800] __x64_sys_ioctl+0x338/0x720 [ 179.806872] ? fire_user_return_notifiers+0x3c/0x60 [ 179.808951] do_syscall_64+0x33/0x40 [ 179.811025] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 179.813123] RIP: 0033:0x7f22e60cef6b [ 179.815220] Code: 89 d8 49 8d 3c 1c 48 f7 d8 49 39 c4 72 b5 e8 1c ff ff ff 85 c0 78 ba 4c 89 e0 5b 5d 41 5c c3 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d d5 ae 0c 00 f7 d8 64 89 01 48 [ 179.819687] RSP: 002b:00007f22e4b69608 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [ 179.821972] RAX: ffffffffffffffda RBX: 000000000000ae80 RCX: 00007f22e60cef6b [ 179.824277] RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000012 [ 179.826590] RBP: 000055a81d11cf70 R08: 000055a81ac42ad8 R09: 00000000000000ff [ 179.828911] R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000 [ 179.831236] R13: 00007f22e6c35001 R14: 0000000000000064 R15: 0000000000000000 [ 179.833563] ---[ end trace 63d1ba11f1bc6184 ]---
Sean Christopherson
maintainerQuoted 3 lines · Zdenek Kaspar
[ 179.364305] WARNING: CPU: 0 PID: 369 at kvm_mmu_zap_oldest_mmu_pages+0xd1/0xe0 [kvm] [ 179.365415] Call Trace: [ 179.365443] paging64_page_fault+0x244/0x8e0 [kvm]
This means the shadow page zapping is occuring because KVM is hitting the max number of allowed MMU shadow pages. Can you provide your QEMU command line? I can reproduce the performance degredation, but only by deliberately overriding the max number of MMU pages via -machine kvm-shadow-mem to be an absurdly low value.
Quoted 6 lines
[ 179.365596] kvm_mmu_page_fault+0x376/0x550 [kvm] [ 179.365725] kvm_arch_vcpu_ioctl_run+0xbaf/0x18f0 [kvm] [ 179.365772] kvm_vcpu_ioctl+0x203/0x520 [kvm] [ 179.365938] __x64_sys_ioctl+0x338/0x720 [ 179.365992] do_syscall_64+0x33/0x40 [ 179.366013] entry_SYSCALL_64_after_hwframe+0x44/0xa9
Quoted 5 lines · On Tue, 22 Dec 2020 09:07:39 -0800 Sean Christopherson <[email protected]>…
Quoted 3 lines · Zdenek Kaspar
[ 179.364305] WARNING: CPU: 0 PID: 369 at kvm_mmu_zap_oldest_mmu_pages+0xd1/0xe0 [kvm] [ 179.365415] Call Trace: [ 179.365443] paging64_page_fault+0x244/0x8e0 [kvm]
Quoted 12 lines
This means the shadow page zapping is occuring because KVM is hitting the max number of allowed MMU shadow pages. Can you provide your QEMU command line? I can reproduce the performance degredation, but only by deliberately overriding the max number of MMU pages via
-machine kvm-shadow-memto be an absurdly low value.Quoted 6 lines
[ 179.365596] kvm_mmu_page_fault+0x376/0x550 [kvm] [ 179.365725] kvm_arch_vcpu_ioctl_run+0xbaf/0x18f0 [kvm] [ 179.365772] kvm_vcpu_ioctl+0x203/0x520 [kvm] [ 179.365938] __x64_sys_ioctl+0x338/0x720 [ 179.365992] do_syscall_64+0x33/0x40 [ 179.366013] entry_SYSCALL_64_after_hwframe+0x44/0xa9
It's one long line, added "\" for mail readability:
qemu-system-x86_64 -machine type=q35,accel=kvm \ -cpu host,host-cache-info=on -smp cpus=2,cores=2 \ -m size=1024 -global virtio-pci.disable-legacy=on \ -global virtio-pci.disable-modern=off \ -device virtio-balloon \ -device virtio-net,netdev=tap-build,mac=DE:AD:BE:EF:00:80 \ -object rng-random,filename=/dev/urandom,id=rng0 \ -device virtio-rng,rng=rng0 \ -name build,process=qemu-build \ -drive file=/mnt/data/export/unix/kvm/build/openbsd-amd64.img,if=virtio,cache=none,format=raw,aio=native \ -netdev type=tap,id=tap-build,vhost=on \ -serial none \ -parallel none \ -monitor unix:/dev/shm/kvm-build.sock,server,nowait \
-enable-kvm -daemonize -runas qemu
Z.
Quoted 40 lines · On Tue, 22 Dec 2020 22:26:45 +0100 Zdenek Kaspar <[email protected]> wrote:
Quoted 5 lines · On Tue, 22 Dec 2020 09:07:39 -0800 Sean Christopherson <[email protected]>…
Quoted 3 lines · Zdenek Kaspar
[ 179.364305] WARNING: CPU: 0 PID: 369 at kvm_mmu_zap_oldest_mmu_pages+0xd1/0xe0 [kvm] [ 179.365415] Call Trace: [ 179.365443] paging64_page_fault+0x244/0x8e0 [kvm]Quoted 12 lines
This means the shadow page zapping is occuring because KVM is hitting the max number of allowed MMU shadow pages. Can you provide your QEMU command line? I can reproduce the performance degredation, but only by deliberately overriding the max number of MMU pages via
-machine kvm-shadow-memto be an absurdly low value.Quoted 6 lines
[ 179.365596] kvm_mmu_page_fault+0x376/0x550 [kvm] [ 179.365725] kvm_arch_vcpu_ioctl_run+0xbaf/0x18f0 [kvm] [ 179.365772] kvm_vcpu_ioctl+0x203/0x520 [kvm] [ 179.365938] __x64_sys_ioctl+0x338/0x720 [ 179.365992] do_syscall_64+0x33/0x40 [ 179.366013] entry_SYSCALL_64_after_hwframe+0x44/0xa9It's one long line, added "\" for mail readability:
qemu-system-x86_64 -machine type=q35,accel=kvm \
(quoted diff) +0
-9-cpu host,host-cache-info=on -smp cpus=2,cores=2 \ -m size=1024 -global virtio-pci.disable-legacy=on \ -global virtio-pci.disable-modern=off \ -device virtio-balloon \ -device virtio-net,netdev=tap-build,mac=DE:AD:BE:EF:00:80 \ -object rng-random,filename=/dev/urandom,id=rng0 \ -device virtio-rng,rng=rng0 \ -name build,process=qemu-build \ -drivefile=/mnt/data/export/unix/kvm/build/openbsd-amd64.img,if=virtio,cache=none,format=raw,aio=native
\ -netdev type=tap,id=tap-build,vhost=on \ -serial
none \ -parallel none
\ -monitor
unix:/dev/shm/kvm-build.sock,server,nowait \ -enable-kvm
-daemonize -runas qemu
Quoted 1 line
Z.
BTW, v5.11-rc3 with kvm-shadow-mem=1073741824 it seems OK.
Just curious what v5.8 does, so by any chance is there command for kvm-shadow-mem value via qemu monitor?
Z.
Sean Christopherson
maintainerQuoted 43 lines · Zdenek Kaspar
Quoted 40 lines · On Tue, 22 Dec 2020 22:26:45 +0100 Zdenek Kaspar <[email protected]> wrote:
Quoted 5 lines · On Tue, 22 Dec 2020 09:07:39 -0800 Sean Christopherson <[email protected]>…
Quoted 3 lines · Zdenek Kaspar
[ 179.364305] WARNING: CPU: 0 PID: 369 at kvm_mmu_zap_oldest_mmu_pages+0xd1/0xe0 [kvm] [ 179.365415] Call Trace: [ 179.365443] paging64_page_fault+0x244/0x8e0 [kvm]Quoted 12 lines
This means the shadow page zapping is occuring because KVM is hitting the max number of allowed MMU shadow pages. Can you provide your QEMU command line? I can reproduce the performance degredation, but only by deliberately overriding the max number of MMU pages via
-machine kvm-shadow-memto be an absurdly low value.Quoted 6 lines
[ 179.365596] kvm_mmu_page_fault+0x376/0x550 [kvm] [ 179.365725] kvm_arch_vcpu_ioctl_run+0xbaf/0x18f0 [kvm] [ 179.365772] kvm_vcpu_ioctl+0x203/0x520 [kvm] [ 179.365938] __x64_sys_ioctl+0x338/0x720 [ 179.365992] do_syscall_64+0x33/0x40 [ 179.366013] entry_SYSCALL_64_after_hwframe+0x44/0xa9It's one long line, added "\" for mail readability:
qemu-system-x86_64 -machine type=q35,accel=kvm \
(quoted diff) +0
-9-cpu host,host-cache-info=on -smp cpus=2,cores=2 \ -m size=1024 -global virtio-pci.disable-legacy=on \ -global virtio-pci.disable-modern=off \ -device virtio-balloon \ -device virtio-net,netdev=tap-build,mac=DE:AD:BE:EF:00:80 \ -object rng-random,filename=/dev/urandom,id=rng0 \ -device virtio-rng,rng=rng0 \ -name build,process=qemu-build \ -drivefile=/mnt/data/export/unix/kvm/build/openbsd-amd64.img,if=virtio,cache=none,format=raw,aio=native
\ -netdev type=tap,id=tap-build,vhost=on \ -serial
none \ -parallel none
\ -monitor
unix:/dev/shm/kvm-build.sock,server,nowait \ -enable-kvm
-daemonize -runas qemu
Quoted 5 lines
Quoted 1 line
Z.
BTW, v5.11-rc3 with kvm-shadow-mem=1073741824 it seems OK.
Just curious what v5.8 does
Aha! Figured it out. v5.9 (the commit you bisected to) broke the zapping, that's what it did. The list of MMU pages is a FIFO list, meaning KVM adds entries to the head, not the tail. I botched the zapping flow and used for_each instead of for_each_reverse, which meant KVM would zap the _newest_ pages instead of the _oldest_ pages. So once a VM hit its limit, KVM would constantly zap the shadow pages it just allocated.
This should resolve the performance regression, or at least make it far less painful. It's possible you may still see some performance degredation due to other changes in the the zapping, e.g. more aggressive recursive zapping. If that's the case, I can explore other tweaks, e.g. skip higher levels when possible. I'll get a proper patch posted later today.
arch/x86/kvm/mmu/mmu.c +1 -1
@@ -2417,7 +2417,7 @@ static unsigned long kvm_mmu_zap_oldest_mmu_pages(struct kvm *kvm,return 0 ; restart: -list_for_each_entry_safe (sp, tmp, &kvm->arch.active_mmu_pages, link) { +list_for_each_entry_safe_reverse (sp, tmp, &kvm->arch.active_mmu_pages, link) {/* * Don't zap active root pages, the page itself can't be freed * and zapping it will just force vCPUs to realloc and reload.
Side topic, I still can't figure out how on earth your guest kernel is hitting the max number of default pages. Even with large pages completely disabled, PTI enabled, multiple guest processes running, etc... I hit OOM in the guest before the host's shadow page limit kicks in. I had to force the limit down to 25% of the default to reproduce the bad behavior. All I can figure is that BSD has a substantially different paging scheme than Linux.
Quoted 3 lines
so by any chance is there command for kvm-shadow-mem value via qemu monitor?
Z.
Quoted 84 lines · On Wed, 13 Jan 2021 12:17:19 -0800 Sean Christopherson <[email protected]>…
Quoted 45 lines · Zdenek Kaspar
Quoted 42 lines · On Tue, 22 Dec 2020 22:26:45 +0100 Zdenek Kaspar <[email protected]> wrote:
Quoted 6 lines · On Tue, 22 Dec 2020 09:07:39 -0800 Sean Christopherson <[email protected]>…
Quoted 4 lines · Zdenek Kaspar
[ 179.364305] WARNING: CPU: 0 PID: 369 at kvm_mmu_zap_oldest_mmu_pages+0xd1/0xe0 [kvm] [ 179.365415] Call Trace: [ 179.365443] paging64_page_fault+0x244/0x8e0[kvm]
Quoted 13 lines
This means the shadow page zapping is occuring because KVM is hitting the max number of allowed MMU shadow pages. Can you provide your QEMU command line? I can reproduce the performance degredation, but only by deliberately overriding the max number of MMU pages via
-machine kvm-shadow-memto be an absurdly low value.Quoted 6 lines
[ 179.365596] kvm_mmu_page_fault+0x376/0x550 [kvm] [ 179.365725] kvm_arch_vcpu_ioctl_run+0xbaf/0x18f0 [kvm] [ 179.365772] kvm_vcpu_ioctl+0x203/0x520 [kvm] [ 179.365938] __x64_sys_ioctl+0x338/0x720 [ 179.365992] do_syscall_64+0x33/0x40 [ 179.366013] entry_SYSCALL_64_after_hwframe+0x44/0xa9It's one long line, added "\" for mail readability:
qemu-system-x86_64 -machine type=q35,accel=kvm \
(quoted diff) +0
-9-cpu host,host-cache-info=on -smp cpus=2,cores=2 \ -m size=1024 -global virtio-pci.disable-legacy=on \ -global virtio-pci.disable-modern=off \ -device virtio-balloon \ -device virtio-net,netdev=tap-build,mac=DE:AD:BE:EF:00:80 \ -object rng-random,filename=/dev/urandom,id=rng0 \ -device virtio-rng,rng=rng0 \ -name build,process=qemu-build \ -drivefile=/mnt/data/export/unix/kvm/build/openbsd-amd64.img,if=virtio,cache=none,format=raw,aio=native
\ -netdev type=tap,id=tap-build,vhost=on \(quoted diff) +0
-2-serial none \ -parallel none \ -monitorunix:/dev/shm/kvm-build.sock,server,nowait \ -enable-kvm
-daemonize -runas qemuQuoted 5 lines
Quoted 1 line
Z.
BTW, v5.11-rc3 with kvm-shadow-mem=1073741824 it seems OK.
Just curious what v5.8 does
Aha! Figured it out. v5.9 (the commit you bisected to) broke the zapping, that's what it did. The list of MMU pages is a FIFO list, meaning KVM adds entries to the head, not the tail. I botched the zapping flow and used
for_eachinstead offor_each_reverse, which meant KVM would zap the _newest_ pages instead of the _oldest_ pages.
So once a VM hit its limit, KVM would constantly zap the shadow
pages it just allocated.This should resolve the performance regression, or at least make it far less painful. It's possible you may still see some performance degredation due to other changes in the the zapping, e.g. more aggressive recursive zapping. If that's the case, I can explore other tweaks, e.g. skip higher levels when possible. I'll get a proper patch posted later today.
arch/x86/kvm/mmu/mmu.c +1
-1@@ -2417,7 +2417,7 @@ static unsigned long kvm_mmu_zap_oldest_mmu_pages(struct kvm *kvm, return 0; restart: -list_for_each_entry_safe (sp, tmp, &kvm->arch.active_mmu_pages, link) { + list_for_each_entry_safe_reverse(sp, tmp,
&kvm->arch.active_mmu_pages, link) { /*
* Don't zap active root pages, the page itself can't
be freed
* and zapping it will just force vCPUs to realloc
and reload.
Quoted 12 lines
Side topic, I still can't figure out how on earth your guest kernel is hitting the max number of default pages. Even with large pages completely disabled, PTI enabled, multiple guest processes running, etc... I hit OOM in the guest before the host's shadow page limit kicks in. I had to force the limit down to 25% of the default to reproduce the bad behavior. All I can figure is that BSD has a substantially different paging scheme than Linux.
Quoted 4 lines
so by any chance is there command for kvm-shadow-mem value via qemu monitor?
Z.
Cool, tested by quick compile in guest and it's a good fix!
5.11.0-rc3-amd64 (list_for_each_entry_safe):
- with kvm-shadow-mem=1073741824 (without == unusable)
0m14.86s real 0m10.87s user 0m12.15s system
5.11.0-rc3-2-amd64 (list_for_each_entry_safe_reverse):
0m14.36s real 0m10.50s user 0m12.43s system
Thanks, Z.
Sean Christopherson
maintainerQuoted 11 lines · Zdenek Kaspar
Hi,
in my initial report (https://marc.info/?l=kvm&m=160502183220080&w=2 - now fixed by
c887c9b9ca62c051d339b1c7b796edf2724029ed) I saw degraded performance going back somewhere between v5.8 - v5.9-rc1.OpenBSD 6.8 (GENERIC.MP) guest performance (time ./test-build.sh)
good: 0m13.54s real 0m10.51s user 0m10.96s system
bad : 6m20.07s real 11m42.93s user 0m13.57s systembisected to first bad commit:
6b82ef2c9cf18a48726e4bb359aa9014632f6466
This is working as intended, in the sense that it's expected that guest performance would go down the drain due to KVM being much more aggressive when reclaiming shadow pages. Prior to commit 6b82ef2c9cf1 ("KVM: x86/mmu: Batch zap MMU pages when recycling oldest pages"), the zapping was completely anemic, e.g. a few shadow pages would get zapped each call, without even really making a dent in the memory consumed by KVM for shadow pages.
Any chance you can track down what is triggering KVM reclaim of shadow pages? E.g. is KVM hitting its limit on the number of MMU pages and reclaiming via make_mmu_pages_available()? Or is the host under high memory pressure and reclaiming memory via mmu_shrink_scan()?