New: QEMU guest crash due to random 'general protection fault' since kernel 5.2.5 on i7-3517UE

Bug 206215 [email protected] 3 participants 16 messages Jan 15, 2020 – Jan 17, 2020

Subsystems (3)

x86 KVM KVM x86

lore

Share & tools

lore

[email protected]

author

2020-01-15 21:19 UTC #0 lore raw

https://bugzilla.kernel.org/show_bug.cgi?id=206215

            Bug ID: 206215
           Summary: QEMU guest crash due to random 'general protection
                    fault' since kernel 5.2.5 on i7-3517UE
           Product: Virtualization
           Version: unspecified
    Kernel Version: 5.5.0-0.rc6
          Hardware: x86-64
                OS: Linux
              Tree: Fedora
            Status: NEW
          Severity: blocking
          Priority: P1
         Component: kvm
          Assignee: [email protected]
          Reporter: [email protected]
        Regression: Yes

Created attachment 286831
--> https://bugzilla.kernel.org/attachment.cgi?id=286831&action=edit
relevant logs

Since kernel 5.2.5 any qemu guest fail to start due to "general protection fault"

[  188.533545] traps: gsd-wacom[1855] general protection fault ip:7fed39b5e7b0
sp:7fff3e349620 error:0 in libglib-2.0.so.0.6200.1[7fed39ae3000+83000]
[  192.002357] traps: gvfs-fuse-sub[1560] general protection fault

ip:7f9cd88100b2 sp:7f9cd5db0bf0 error:0 in
libglib-2.0.so.0.6200.1[7f9cd87de000+83000]

Please note that kernel 5.2.4 work fine.

Tested guests with Widows Server 2016/2019 & Fedora 31

Attached logs show the DMESG output of the guests

Attached host files contains a WARNING thrown upong first guest start on the hypervisor:

[   49.533713] WARNING: CPU: 3 PID: 966 at arch/x86/kvm/x86.c:7963
kvm_arch_vcpu_ioctl_run+0x1927/0x1ce0 [kvm]
[   49.533714] Modules linked in: vhost_net vhost tap tun xfrm4_tunnel tunnel4

ipcomp xfrm_ipcomp esp4 ah4 af_key ebtable_filter ebtables ip6table_filter ip6_tables bridge stp llc nf_log_ipv4 nf_log_common xt_LOG ipt_REJECT nf_reject_ipv4 iptable_filter iptable_security iptable_raw xt_state xt_conntrack xt_DSCP xt_multiport iptable_mangle xt_TCPMSS xt_tcpmss xt_policy xt_nat iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel sunrpc kvm vfat fat mei_hdcp mei_wdt snd_hda_codec_hdmi iTCO_wdt irqbypass iTCO_vendor_support snd_hda_codec_realtek snd_hda_codec_generic crct10dif_pclmul crc32_pclmul ledtrig_audio snd_hda_intel ghash_clmulni_intel snd_hda_codec intel_cstate intel_uncore snd_hda_core snd_hwdep intel_rapl_perf snd_seq snd_seq_device snd_pcm i2c_i801 r8169 lpc_ich mei_me snd_timer snd mei e1000e soundcore pcc_cpufreq tcp_bbr sch_fq ip_tables xfs i915 libcrc32c i2c_algo_bit drm_kms_helper crc32c_intel drm

[   49.533760]  serio_raw video
[   49.533764] CPU: 3 PID: 966 Comm: CPU 0/KVM Not tainted
5.2.5-200.fc30.x86_64 #1
[   49.533765] Hardware name: CompuLab 0000000-00000/Intense-PC, BIOS
IPC_2.2.400.5 X64 03/15/2018
[   49.533784] RIP: 0010:kvm_arch_vcpu_ioctl_run+0x1927/0x1ce0 [kvm]
[   49.533786] Code: 4c 89 e7 e8 1b 0b ff ff 4c 89 e7 e8 d3 8c fe ff 41 83 a4
24 e8 36 00 00 fb e9 bd ed ff ff f0 41 80 4c 24 31 10 e9 a5 ee ff ff <0f> 0b e9
74 ed ff ff 49 8b 84 24 c8 02 00 00 a9 00 00 01 00 0f 84
[   49.533787] RSP: 0018:ffffbe4e423ffd30 EFLAGS: 00010002
[   49.533789] RAX: 0000000000004b00 RBX: 0000000000000000 RCX:
ffffa044ce958000
[   49.533790] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000000
[   49.533791] RBP: ffffbe4e423ffdd8 R08: 0000000000000000 R09:
00000000000003e8
[   49.533792] R10: 0000000000000000 R11: 0000000000000000 R12:
ffffa044d38f8000
[   49.533792] R13: 0000000000000000 R14: ffffbe4e41ccf7b8 R15:
0000000000000000
[   49.533794] FS:  00007f117953f700(0000) GS:ffffa044ee2c0000(0000)
knlGS:0000000000000000
[   49.533795] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   49.533796] CR2: 0000000000000000 CR3: 000000040e8c2003 CR4:
00000000001626e0
[   49.533797] Call Trace:
[   49.533817]  kvm_vcpu_ioctl+0x215/0x5c0 [kvm]
[   49.533821]  ? __seccomp_filter+0x7b/0x640
[   49.533824]  ? __switch_to_asm+0x34/0x70
[   49.533826]  ? __switch_to_asm+0x34/0x70
[   49.533827]  ? apic_timer_interrupt+0xa/0x20
[   49.533831]  do_vfs_ioctl+0x405/0x660
[   49.533834]  ksys_ioctl+0x5e/0x90
[   49.533836]  __x64_sys_ioctl+0x16/0x20
[   49.533839]  do_syscall_64+0x5f/0x1a0
[   49.533842]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[   49.533844] RIP: 0033:0x7f117d1fb34b
[   49.533845] Code: 0f 1e fa 48 8b 05 3d 9b 0c 00 64 c7 00 26 00 00 00 48 c7
c0 ff ff ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01
f0 ff ff 73 01 c3 48 8b 0d 0d 9b 0c 00 f7 d8 64 89 01 48
[   49.533846] RSP: 002b:00007f117953e698 EFLAGS: 00000246 ORIG_RAX:
0000000000000010
[   49.533848] RAX: ffffffffffffffda RBX: 0000564f2cb65ba0 RCX:
00007f117d1fb34b
[   49.533849] RDX: 0000000000000000 RSI: 000000000000ae80 RDI:
0000000000000019
[   49.533850] RBP: 00007f1179f20000 R08: 0000564f2b7e5390 R09:
000000000000ffff
[   49.533851] R10: 0000564f2ca7a710 R11: 0000000000000246 R12:
0000000000000001
[   49.533852] R13: 00007f1179f21002 R14: 0000000000000000 R15:
0000564f2bc66e80
[   49.533854] ---[ end trace a562473b18c9b742 ]---

/proc/cpuinfo

processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 58
model name : Intel(R) Core(TM) i7-3517UE CPU @ 1.70GHz
stepping : 9
microcode : 0x1f
cpu MHz : 828.296
cache size : 4096 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 2
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm cpuid_fault epb pti ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds
bogomips : 4389.89
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:

Signature · 2 lines

You are receiving this mail because:
You are watching the assignee of the bug.

Sean Christopherson

2020-01-15 21:52 UTC #1 lore raw

+cc Derek, who is hitting the same thing.

Quoted 36 lines · On Wed, Jan 15, 2020 at 09:18:56PM +0000, [email protected]…

https://bugzilla.kernel.org/show_bug.cgi?id=206215
            Bug ID: 206215
           Summary: QEMU guest crash due to random 'general protection
                    fault' since kernel 5.2.5 on i7-3517UE
           Product: Virtualization
           Version: unspecified
    Kernel Version: 5.5.0-0.rc6
          Hardware: x86-64
                OS: Linux
              Tree: Fedora
            Status: NEW
          Severity: blocking
          Priority: P1
         Component: kvm
          Assignee: [email protected]
          Reporter: [email protected]
        Regression: Yes
Created attachment 286831
--> https://bugzilla.kernel.org/attachment.cgi?id=286831&action=edit
relevant logs
Since kernel 5.2.5 any qemu guest fail to start due to "general protection fault"
[  188.533545] traps: gsd-wacom[1855] general protection fault ip:7fed39b5e7b0
sp:7fff3e349620 error:0 in libglib-2.0.so.0.6200.1[7fed39ae3000+83000]
[  192.002357] traps: gvfs-fuse-sub[1560] general protection fault
ip:7f9cd88100b2 sp:7f9cd5db0bf0 error:0 in
libglib-2.0.so.0.6200.1[7f9cd87de000+83000]
Please note that kernel 5.2.4 work fine.
Tested guests with Widows Server 2016/2019 & Fedora 31

Quoted 7 lines

Attached logs show the DMESG output of the guests
Attached host files contains a WARNING thrown upong first guest start on the hypervisor:
[   49.533713] WARNING: CPU: 3 PID: 966 at arch/x86/kvm/x86.c:7963
kvm_arch_vcpu_ioctl_run+0x1927/0x1ce0 [kvm]

Between the WARN, which is

  WARN_ON_ONCE(test_thread_flag(TIF_NEED_FPU_LOAD));

and the total diff of arch/x86/kvm for 5.2.4 -> 5.2.5 is

arch/x86/kvm/x86.c 2020-01-15 13:37:08.190438719 -0800 renamed from arch/x86/kvm/x86.c 2020-01-15 13:37:05.154445843 -0800 +6 -3

@@ -3264,6 +3264,10 @@
 
        kvm_x86_ops->vcpu_load(vcpu, cpu);
 
+       fpregs_assert_state_consistent();
+       if (test_thread_flag(TIF_NEED_FPU_LOAD))
+               switch_fpu_return();
+
        /* Apply any externally detected TSC adjustments (due to suspend) */
        if (unlikely(vcpu->arch.tsc_offset_adjustment)) {
                adjust_tsc_offset_host(vcpu, vcpu->arch.tsc_offset_adjustment);

@@ -7955,9 +7959,8 @@
                wait_lapic_expire(vcpu);
        guest_enter_irqoff();
 
-       fpregs_assert_state_consistent();
-       if (test_thread_flag(TIF_NEED_FPU_LOAD))
-               switch_fpu_return();
+       /* The preempt notifier should have taken care of the FPU already.  */
+       WARN_ON_ONCE(test_thread_flag(TIF_NEED_FPU_LOAD));
 
        if (unlikely(vcpu->arch.switch_db_regs)) {
                set_debugreg(0, 7);

that's a big smoking gun pointing at commit ca7e6b286333 ("KVM: X86: Fix fpu state crash in kvm guest"), which is commit e751732486eb upstream.

1. Can you verify reverting ca7e6b286333 (or e751732486eb in upstream) solves the issue?

2. Assuming the answer is yes, on a buggy kernel, can you run with the attached patch to try get debug info?

Quoted 94 lines

[   49.533714] Modules linked in: vhost_net vhost tap tun xfrm4_tunnel tunnel4
ipcomp xfrm_ipcomp esp4 ah4 af_key ebtable_filter ebtables ip6table_filter ip6_tables bridge stp llc nf_log_ipv4 nf_log_common xt_LOG ipt_REJECT nf_reject_ipv4 iptable_filter iptable_security iptable_raw xt_state xt_conntrack xt_DSCP xt_multiport iptable_mangle xt_TCPMSS xt_tcpmss xt_policy xt_nat iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel sunrpc kvm vfat fat mei_hdcp mei_wdt snd_hda_codec_hdmi iTCO_wdt irqbypass iTCO_vendor_support snd_hda_codec_realtek snd_hda_codec_generic crct10dif_pclmul crc32_pclmul ledtrig_audio snd_hda_intel ghash_clmulni_intel snd_hda_codec intel_cstate intel_uncore snd_hda_core snd_hwdep intel_rapl_perf snd_seq snd_seq_device snd_pcm i2c_i801 r8169 lpc_ich mei_me snd_timer snd mei e1000e soundcore pcc_cpufreq tcp_bbr sch_fq ip_tables xfs i915 libcrc32c i2c_algo_bit drm_kms_helper crc32c_intel drm
[   49.533760]  serio_raw video
[   49.533764] CPU: 3 PID: 966 Comm: CPU 0/KVM Not tainted
5.2.5-200.fc30.x86_64 #1
[   49.533765] Hardware name: CompuLab 0000000-00000/Intense-PC, BIOS
IPC_2.2.400.5 X64 03/15/2018
[   49.533784] RIP: 0010:kvm_arch_vcpu_ioctl_run+0x1927/0x1ce0 [kvm]
[   49.533786] Code: 4c 89 e7 e8 1b 0b ff ff 4c 89 e7 e8 d3 8c fe ff 41 83 a4
24 e8 36 00 00 fb e9 bd ed ff ff f0 41 80 4c 24 31 10 e9 a5 ee ff ff <0f> 0b e9
74 ed ff ff 49 8b 84 24 c8 02 00 00 a9 00 00 01 00 0f 84
[   49.533787] RSP: 0018:ffffbe4e423ffd30 EFLAGS: 00010002
[   49.533789] RAX: 0000000000004b00 RBX: 0000000000000000 RCX:
ffffa044ce958000
[   49.533790] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000000
[   49.533791] RBP: ffffbe4e423ffdd8 R08: 0000000000000000 R09:
00000000000003e8
[   49.533792] R10: 0000000000000000 R11: 0000000000000000 R12:
ffffa044d38f8000
[   49.533792] R13: 0000000000000000 R14: ffffbe4e41ccf7b8 R15:
0000000000000000
[   49.533794] FS:  00007f117953f700(0000) GS:ffffa044ee2c0000(0000)
knlGS:0000000000000000
[   49.533795] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   49.533796] CR2: 0000000000000000 CR3: 000000040e8c2003 CR4:
00000000001626e0
[   49.533797] Call Trace:
[   49.533817]  kvm_vcpu_ioctl+0x215/0x5c0 [kvm]
[   49.533821]  ? __seccomp_filter+0x7b/0x640
[   49.533824]  ? __switch_to_asm+0x34/0x70
[   49.533826]  ? __switch_to_asm+0x34/0x70
[   49.533827]  ? apic_timer_interrupt+0xa/0x20
[   49.533831]  do_vfs_ioctl+0x405/0x660
[   49.533834]  ksys_ioctl+0x5e/0x90
[   49.533836]  __x64_sys_ioctl+0x16/0x20
[   49.533839]  do_syscall_64+0x5f/0x1a0
[   49.533842]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[   49.533844] RIP: 0033:0x7f117d1fb34b
[   49.533845] Code: 0f 1e fa 48 8b 05 3d 9b 0c 00 64 c7 00 26 00 00 00 48 c7
c0 ff ff ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01
f0 ff ff 73 01 c3 48 8b 0d 0d 9b 0c 00 f7 d8 64 89 01 48
[   49.533846] RSP: 002b:00007f117953e698 EFLAGS: 00000246 ORIG_RAX:
0000000000000010
[   49.533848] RAX: ffffffffffffffda RBX: 0000564f2cb65ba0 RCX:
00007f117d1fb34b
[   49.533849] RDX: 0000000000000000 RSI: 000000000000ae80 RDI:
0000000000000019
[   49.533850] RBP: 00007f1179f20000 R08: 0000564f2b7e5390 R09:
000000000000ffff
[   49.533851] R10: 0000564f2ca7a710 R11: 0000000000000246 R12:
0000000000000001
[   49.533852] R13: 00007f1179f21002 R14: 0000000000000000 R15:
0000564f2bc66e80
[   49.533854] ---[ end trace a562473b18c9b742 ]---
/proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 58
model name : Intel(R) Core(TM) i7-3517UE CPU @ 1.70GHz
stepping : 9
microcode : 0x1f
cpu MHz : 828.296
cache size : 4096 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 2
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c

Quoted 12 lines

rdrand lahf_lm cpuid_fault epb pti ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds
bogomips : 4389.89
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:
Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.

From 6288031dacbe753b84515d330f62c1f8ed31d932 Mon Sep 17 00:00:00 2001
From: Sean Christopherson <[email protected]>
Date: Wed, 20 Nov 2019 10:12:56 -0800
Subject: [PATCH] thread_info: Add a debug hook to detect FPU changes while a
vCPU is loaded

Signed-off-by Sean Christopherson <[email protected]>

---
 arch/x86/include/asm/thread_info.h | 2 ++
 arch/x86/kvm/x86.c                 | 4 ++++
 include/linux/thread_info.h        | 1 +
 3 files changed, 7 insertions(+)

arch/x86/include/asm/thread_info.h +2 -0

index f9453536f9bb..7b697005cc51 100644
@@ -56,6 +56,8 @@ struct task_struct;
 struct thread_info {
 	unsigned long		flags;		/* low level flags */
 	u32			status;		/* thread synchronous flags */
+	bool			vcpu_loaded;
+
 };
 
 #define INIT_THREAD_INFO(tsk)			\

arch/x86/kvm/x86.c +4 -0

index a8ad3a4d86b1..3d9c049e749e 100644
@@ -3303,6 +3303,8 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
 	}
 
 	kvm_make_request(KVM_REQ_STEAL_UPDATE, vcpu);
+
+	current_thread_info()->vcpu_loaded = 1;
 }
 
 static void kvm_steal_time_set_preempted(struct kvm_vcpu *vcpu)

@@ -3322,6 +3324,8 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
 {
 	int idx;
 
+	current_thread_info()->vcpu_loaded = 0;
+
 	if (vcpu->preempted)
 		vcpu->arch.preempted_in_kernel = !kvm_x86_ops->get_cpl(vcpu);

include/linux/thread_info.h +1 -0

index 8d8821b3689a..016c2c887354 100644
@@ -52,6 +52,7 @@ enum {
 
 static inline void set_ti_thread_flag(struct thread_info *ti, int flag)
 {
+	WARN_ON_ONCE(ti->vcpu_loaded && flag == TIF_NEED_FPU_LOAD);
 	set_bit(flag, (unsigned long *)&ti->flags);
 }

Signature · 1 line

2.24.0

Derek Yerger

2020-01-16 01:11 UTC #2 lore raw

Quoted 8 lines · Sean Christopherson

+cc Derek, who is hitting the same thing.
Quoted 1 line · On Wed, Jan 15, 2020 at 09:18:56PM +0000, [email protected]…
https://bugzilla.kernel.org/show_bug.cgi?id=206215
*snip*
that's a big smoking gun pointing at commit ca7e6b286333 ("KVM: X86: Fix fpu state crash in kvm guest"), which is commit e751732486eb upstream.

Quoted 5 lines

1. Can you verify reverting ca7e6b286333 (or e751732486eb in upstream) solves the issue?
2. Assuming the answer is yes, on a buggy kernel, can you run with the attached patch to try get debug info?

I did these out of order since I had 5.3.11 built with the patch, ready to go for weeks now, waiting for an opportunity to test.

Win10 guest immediately BSOD'ed with:

WARNING: CPU: 2 PID: 9296 at include/linux/thread_info.h:55 kernel_fpu_begin+0x6b/0xc0

Then stashed the patch, reverted ca7e6b286333, compile, reboot.

Guest is running stable now on 5.3.11. Did test my CAD under the guest, did not experience the crashes that had me stuck at 5.1.

Sean Christopherson

2020-01-16 15:38 UTC #3 lore raw

Quoted 14 lines · Derek Yerger

Quoted 8 lines · Sean Christopherson
+cc Derek, who is hitting the same thing.
Quoted 1 line · On Wed, Jan 15, 2020 at 09:18:56PM +0000, [email protected]…
https://bugzilla.kernel.org/show_bug.cgi?id=206215
*snip*
that's a big smoking gun pointing at commit ca7e6b286333 ("KVM: X86: Fix fpu state crash in kvm guest"), which is commit e751732486eb upstream.
Quoted 5 lines
1. Can you verify reverting ca7e6b286333 (or e751732486eb in upstream) solves the issue?
2. Assuming the answer is yes, on a buggy kernel, can you run with the attached patch to try get debug info?

Quoted 7 lines

I did these out of order since I had 5.3.11 built with the patch, ready to go for weeks now, waiting for an opportunity to test.
Win10 guest immediately BSOD'ed with:
WARNING: CPU: 2 PID: 9296 at include/linux/thread_info.h:55
kernel_fpu_begin+0x6b/0xc0

Can you provide the full stack trace of the WARN? I'm hoping that will provide a hint as to what's going wrong.

Quoted 4 lines

Then stashed the patch, reverted ca7e6b286333, compile, reboot.
Guest is running stable now on 5.3.11. Did test my CAD under the guest, did not experience the crashes that had me stuck at 5.1.

Sean Christopherson

2020-01-16 18:08 UTC #4 lore raw

Quoted 18 lines · Sean Christopherson

Quoted 17 lines · Derek Yerger
Quoted 8 lines · Sean Christopherson
+cc Derek, who is hitting the same thing.
Quoted 1 line · On Wed, Jan 15, 2020 at 09:18:56PM +0000, [email protected]…
https://bugzilla.kernel.org/show_bug.cgi?id=206215
*snip*
that's a big smoking gun pointing at commit ca7e6b286333 ("KVM: X86: Fix fpu state crash in kvm guest"), which is commit e751732486eb upstream.
Quoted 5 lines
1. Can you verify reverting ca7e6b286333 (or e751732486eb in upstream) solves the issue?
2. Assuming the answer is yes, on a buggy kernel, can you run with the attached patch to try get debug info?
I did these out of order since I had 5.3.11 built with the patch, ready to go for weeks now, waiting for an opportunity to test.

Quoted 7 lines

Quoted 4 lines
Win10 guest immediately BSOD'ed with:
WARNING: CPU: 2 PID: 9296 at include/linux/thread_info.h:55
kernel_fpu_begin+0x6b/0xc0
Can you provide the full stack trace of the WARN? I'm hoping that will provide a hint as to what's going wrong.

Aha! I found at least two cases where TIF_NEED_FPU_LOAD could be set without the vCPU being preempted.

The comment on fpregs_lock() states that softirq can set TIF_NEED_FPU_LOAD, which would not be handled by the preempt notifier.

/*
* Use fpregs_lock() while editing CPU's FPU registers or fpu->state.
* A context switch will (and softirq might) save CPU's FPU registers to

                           ^^^^^^^^^^^^^^^^^^^

* fpu->state and set TIF_NEED_FPU_LOAD leaving CPU's FPU registers in
* a random state.
*/
static inline void fpregs_lock(void)

The other scenario is from a stack trace from commit f775b13eedee ("x86,kvm:
move qemu/guest FPU switching out to vcpu_run"), which clearly shows that kernel_fpu_begin() can be invoked without KVM being preempted.

  __warn+0xcb/0xf0
  warn_slowpath_null+0x1d/0x20
  kernel_fpu_disable+0x3f/0x50
  __kernel_fpu_begin+0x49/0x100
  kernel_fpu_begin+0xe/0x10
  crc32c_pcl_intel_update+0x84/0xb0
  crypto_shash_update+0x3f/0x110
  crc32c+0x63/0x8a [libcrc32c]
  dm_bm_checksum+0x1b/0x20 [dm_persistent_data]
  node_prepare_for_write+0x44/0x70 [dm_persistent_data]
  dm_block_manager_write_callback+0x41/0x50 [dm_persistent_data]
  submit_io+0x170/0x1b0 [dm_bufio]
  __write_dirty_buffer+0x89/0x90 [dm_bufio]
  __make_buffer_clean+0x4f/0x80 [dm_bufio]
  __try_evict_buffer+0x42/0x60 [dm_bufio]
  dm_bufio_shrink_scan+0xc0/0x130 [dm_bufio]
  shrink_slab.part.40+0x1f5/0x420
  shrink_node+0x22c/0x320
  do_try_to_free_pages+0xf5/0x330
  try_to_free_pages+0xe9/0x190
  __alloc_pages_slowpath+0x40f/0xba0
  __alloc_pages_nodemask+0x209/0x260
  alloc_pages_vma+0x1f1/0x250
  do_huge_pmd_anonymous_page+0x123/0x660
  handle_mm_fault+0xfd3/0x1330
  __get_user_pages+0x113/0x640
  get_user_pages+0x4f/0x60
  __gfn_to_pfn_memslot+0x120/0x3f0 [kvm]
  try_async_pf+0x66/0x230 [kvm]
  tdp_page_fault+0x130/0x280 [kvm]
  kvm_mmu_page_fault+0x60/0x120 [kvm]
  handle_ept_violation+0x91/0x170 [kvm_intel]
  vmx_handle_exit+0x1ca/0x1400 [kvm_intel]

Either of the above explains why pre-e751732486eb code waited until IRQs are disabled by vcpu_enter_guest() to do switch_fpu_return().

Properly fixing soley within KVM is going to be somewhat painful. The most common case, vcpu_enter_guest(), which is being hit here, is easy to handle by restoring the switch_fpu_return() that was removed by commit e751732486eb. The other obvious case I see is emulator's access of guest fpu state, which will effectively require reverting commit 6ab0b9feb82a ("x86,kvm: remove KVM emulator get_fpu / put_fpu") along with new implementations of the hooks to handle TIF_NEED_FPU_LOAD.

Quoted 4 lines

Quoted 4 lines
Then stashed the patch, reverted ca7e6b286333, compile, reboot.
Guest is running stable now on 5.3.11. Did test my CAD under the guest, did not experience the crashes that had me stuck at 5.1.

Derek Yerger

2020-01-16 19:21 UTC #5 lore raw

In reply to Sean Christopherson

Quoted 18 lines · Sean Christopherson

Quoted 17 lines · Derek Yerger
Quoted 8 lines · Sean Christopherson
+cc Derek, who is hitting the same thing.
Quoted 1 line · On Wed, Jan 15, 2020 at 09:18:56PM +0000, [email protected]…
https://bugzilla.kernel.org/show_bug.cgi?id=206215
*snip*
that's a big smoking gun pointing at commit ca7e6b286333 ("KVM: X86: Fix fpu state crash in kvm guest"), which is commit e751732486eb upstream.
Quoted 5 lines
1. Can you verify reverting ca7e6b286333 (or e751732486eb in upstream) solves the issue?
2. Assuming the answer is yes, on a buggy kernel, can you run with the attached patch to try get debug info?
I did these out of order since I had 5.3.11 built with the patch, ready to go for weeks now, waiting for an opportunity to test.

Quoted 6 lines

Quoted 4 lines
Win10 guest immediately BSOD'ed with:
WARNING: CPU: 2 PID: 9296 at include/linux/thread_info.h:55
kernel_fpu_begin+0x6b/0xc0
Can you provide the full stack trace of the WARN? I'm hoping that will provide a hint as to what's going wrong.

WARNING: CPU: 2 PID: 9296 at include/linux/thread_info.h:55 kernel_fpu_begin+0x6b/0xc0
Modules linked in: vhost_net(E) vhost(E) macvtap(E) macvlan(E) tap(E) esp4(E) xt_CHECKSUM(E) xt_MASQUERADE(E) tun(E) bridge(E) stp(E) llc(E) ip6t_rpfilter(E) nf_log_ipv6(E) ip6t_REJECT(E) nf_reject_ipv6>
 mei_hdcp(E) kvm(E) intel_cstate(E) intel_uncore(E) intel_rapl_perf(E) eeepc_wmi(E) asus_wmi(E) sparse_keymap(E) rfkill(E) snd_hda_codec_generic(E) pcspkr(E) wmi_bmof(E) ledtrig_audio(E) i2c_i801(E) snd>
CPU: 2 PID: 9296 Comm: CPU 1/KVM Tainted: P           OE     5.3.11+ #16
Hardware name: System manufacturer System Product Name/Z170-K, BIOS 3805 05/16/2018
RIP: 0010:kernel_fpu_begin+0x6b/0xc0
Code: f6 40 26 20 75 08 48 8b 10 80 e6 40 74 16 65 48 c7 05 b5 27 fe 70 00 00 00 00 c3 65 8a 05 a5 27 fe 70 eb c4 80 78 0c 00 74 02 <0f> 0b 48 83 c0 01 f0 80 08 40 65 48 8b 0c 25 c0 6b 01 00 0f 1f 44
RSP: 0018:ffffb42e0014c7f8 EFLAGS: 00010202
RAX: ffff98f1783a1ec0 RBX: 0000000000000038 RCX: 0000000000000048
RDX: 0000000000000020 RSI: ffff98f1d9a5cb00 RDI: ffff98f1d9a5cb00
RBP: ffffb42e0014caa0 R08: ffffb42e0014cab0 R09: ffffb42e0014c860
R10: 0000000000000000 R11: 0000000000000000 R12: 00000000000002ba
R13: ffffb42e0014c860 R14: ffff98f1d36882aa R15: ffff98f1d9a5caa8
FS:  00007f02faffd700(0000) GS:ffff98f286a80000(0000) knlGS:000000f0dd174000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000001f8249cd000 CR3: 000000043d3dc003 CR4: 00000000003626e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <IRQ>
 gcmaes_crypt_by_sg.constprop.12+0x26e/0x660
 ? 0xffffffffc024547d
 ? __qdisc_run+0x83/0x510
 ? __dev_queue_xmit+0x45e/0x990
 ? ip_finish_output2+0x1a8/0x570
 ? fib4_rule_action+0x61/0x70
 ? fib4_rule_action+0x70/0x70
 ? fib_rules_lookup+0x13f/0x1c0
 ? helper_rfc4106_decrypt+0x82/0xa0
 ? crypto_aead_decrypt+0x40/0x70
 ? crypto_aead_decrypt+0x40/0x70
 ? crypto_aead_decrypt+0x40/0x70
 ? esp_output_tail+0x8f4/0xa5a [esp4]
 ? skb_ext_add+0xd3/0x170
 ? xfrm_input+0x7a6/0x12c0
 ? xfrm4_rcv_encap+0xae/0xd0
 ? xfrm4_transport_finish+0x200/0x200
 ? udp_queue_rcv_one_skb+0x1ba/0x460
 ? udp_unicast_rcv_skb.isra.63+0x72/0x90
 ? __udp4_lib_rcv+0x51b/0xb00
 ? ip_protocol_deliver_rcu+0xd2/0x1c0
 ? ip_local_deliver_finish+0x44/0x50
 ? ip_local_deliver+0xe0/0xf0
 ? ip_protocol_deliver_rcu+0x1c0/0x1c0
 ? ip_rcv+0xbc/0xd0
 ? ip_rcv_finish_core.isra.19+0x380/0x380
 ? __netif_receive_skb_one_core+0x7e/0x90
 ? netif_receive_skb_internal+0x3d/0xb0
 ? napi_gro_receive+0xed/0x150
 ? 0xffffffffc0243c77
 ? net_rx_action+0x149/0x3b0
 ? __do_softirq+0xe4/0x2f8
 ? handle_irq_event_percpu+0x6a/0x80
 ? irq_exit+0xe6/0xf0
 ? do_IRQ+0x7f/0xd0
 ? common_interrupt+0xf/0xf
 </IRQ>
 ? irq_entries_start+0x20/0x660
 ? vmx_get_interrupt_shadow+0x2f0/0x710 [kvm_intel]
 ? kvm_set_msr_common+0xfc7/0x2380 [kvm]
 ? recalibrate_cpu_khz+0x10/0x10
 ? ktime_get+0x3a/0xa0
 ? kvm_arch_vcpu_ioctl_run+0x107/0x560 [kvm]
 ? kvm_init+0x6bf/0xd00 [kvm]
 ? __seccomp_filter+0x7a/0x680
 ? do_vfs_ioctl+0xa4/0x630
 ? security_file_ioctl+0x32/0x50
 ? ksys_ioctl+0x60/0x90
 ? __x64_sys_ioctl+0x16/0x20
 ? do_syscall_64+0x5f/0x1a0
 ? entry_SYSCALL_64_after_hwframe+0x44/0xa9
---[ end trace 9564a1ccad733a90 ]---
WARNING: CPU: 2 PID: 9296 at arch/x86/kvm/x86.c:8060 kvm_set_msr_common+0x2230/0x2380 [kvm]
Modules linked in: vhost_net(E) vhost(E) macvtap(E) macvlan(E) tap(E) esp4(E) xt_CHECKSUM(E) xt_MASQUERADE(E) tun(E) bridge(E) stp(E) llc(E) ip6t_rpfilter(E) nf_log_ipv6(E) ip6t_REJECT(E) nf_reject_ipv6>
 mei_hdcp(E) kvm(E) intel_cstate(E) intel_uncore(E) intel_rapl_perf(E) eeepc_wmi(E) asus_wmi(E) sparse_keymap(E) rfkill(E) snd_hda_codec_generic(E) pcspkr(E) wmi_bmof(E) ledtrig_audio(E) i2c_i801(E) snd>
CPU: 2 PID: 9296 Comm: CPU 1/KVM Tainted: P        W  OE     5.3.11+ #16
Hardware name: System manufacturer System Product Name/Z170-K, BIOS 3805 05/16/2018
RIP: 0010:kvm_set_msr_common+0x2230/0x2380 [kvm]
Code: b0 26 00 00 e8 91 9f b5 ce 66 90 bf 06 00 00 00 48 8b b3 88 26 00 00 e8 7e 9f b5 ce 66 90 83 a3 60 26 00 00 fb e9 e9 ec ff ff <0f> 0b e9 d2 ec ff ff f0 80 4b 31 10 e9 32 ee ff ff 48 8b 83 98 02
RSP: 0018:ffffb42e03d17d30 EFLAGS: 00010002
RAX: 0000000080004b20 RBX: ffff98f1783abf40 RCX: ffff98f17757f000
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000
RBP: ffffb42e03d17db0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff98f1783abf70
R13: 0000000000000000 R14: 0000000000000000 R15: ffff98f1d8bc6c00
FS:  00007f02faffd700(0000) GS:ffff98f286a80000(0000) knlGS:000000f0dd174000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000001f8249cd000 CR3: 000000043d3dc003 CR4: 00000000003626e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 ? recalibrate_cpu_khz+0x10/0x10
 ? ktime_get+0x3a/0xa0
 kvm_arch_vcpu_ioctl_run+0x107/0x560 [kvm]
 kvm_init+0x6bf/0xd00 [kvm]
 ? __seccomp_filter+0x7a/0x680
 do_vfs_ioctl+0xa4/0x630
 ? security_file_ioctl+0x32/0x50
 ksys_ioctl+0x60/0x90
 __x64_sys_ioctl+0x16/0x20
 do_syscall_64+0x5f/0x1a0
 entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x7f0302457d4b
Code: 0f 1e fa 48 8b 05 3d b1 0c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 0d b1 0c 00 f7 d8 64 89 01 48
RSP: 002b:00007f02faffc6c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 00007f03005fd001 RCX: 00007f0302457d4b
RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000020
RBP: 0000000000000001 R08: 0000564efe32fa50 R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000000246 R12: 0000564efe3129c0
R13: 0000000000000000 R14: 00007f03005fc000 R15: 0000564f00123300
---[ end trace 9564a1ccad733a91 ]---

Sean Christopherson

2020-01-16 19:32 UTC #6 lore raw

Quoted 25 lines · Derek Yerger

 <IRQ>
 gcmaes_crypt_by_sg.constprop.12+0x26e/0x660
 ? 0xffffffffc024547d
 ? __qdisc_run+0x83/0x510
 ? __dev_queue_xmit+0x45e/0x990
 ? ip_finish_output2+0x1a8/0x570
 ? fib4_rule_action+0x61/0x70
 ? fib4_rule_action+0x70/0x70
 ? fib_rules_lookup+0x13f/0x1c0
 ? helper_rfc4106_decrypt+0x82/0xa0
 ? crypto_aead_decrypt+0x40/0x70
 ? crypto_aead_decrypt+0x40/0x70
 ? crypto_aead_decrypt+0x40/0x70
 ? esp_output_tail+0x8f4/0xa5a [esp4]
 ? skb_ext_add+0xd3/0x170
 ? xfrm_input+0x7a6/0x12c0
 ? xfrm4_rcv_encap+0xae/0xd0
 ? xfrm4_transport_finish+0x200/0x200
 ? udp_queue_rcv_one_skb+0x1ba/0x460
 ? udp_unicast_rcv_skb.isra.63+0x72/0x90
 ? __udp4_lib_rcv+0x51b/0xb00
 ? ip_protocol_deliver_rcu+0xd2/0x1c0
 ? ip_local_deliver_finish+0x44/0x50
 ? ip_local_deliver+0xe0/0xf0
 ? ip_protocol_deliver_rcu+0x1c0/0x1c0

Quoted 8 lines

 ? ip_rcv+0xbc/0xd0
 ? ip_rcv_finish_core.isra.19+0x380/0x380
 ? __netif_receive_skb_one_core+0x7e/0x90
 ? netif_receive_skb_internal+0x3d/0xb0
 ? napi_gro_receive+0xed/0x150
 ? 0xffffffffc0243c77
 ? net_rx_action+0x149/0x3b0
 ? __do_softirq+0xe4/0x2f8

Bingo! Thanks Derek!

Quoted 69 lines

 ? handle_irq_event_percpu+0x6a/0x80
 ? irq_exit+0xe6/0xf0
 ? do_IRQ+0x7f/0xd0
 ? common_interrupt+0xf/0xf
 </IRQ>
 ? irq_entries_start+0x20/0x660
 ? vmx_get_interrupt_shadow+0x2f0/0x710 [kvm_intel]
 ? kvm_set_msr_common+0xfc7/0x2380 [kvm]
 ? recalibrate_cpu_khz+0x10/0x10
 ? ktime_get+0x3a/0xa0
 ? kvm_arch_vcpu_ioctl_run+0x107/0x560 [kvm]
 ? kvm_init+0x6bf/0xd00 [kvm]
 ? __seccomp_filter+0x7a/0x680
 ? do_vfs_ioctl+0xa4/0x630
 ? security_file_ioctl+0x32/0x50
 ? ksys_ioctl+0x60/0x90
 ? __x64_sys_ioctl+0x16/0x20
 ? do_syscall_64+0x5f/0x1a0
 ? entry_SYSCALL_64_after_hwframe+0x44/0xa9
---[ end trace 9564a1ccad733a90 ]---
WARNING: CPU: 2 PID: 9296 at arch/x86/kvm/x86.c:8060
kvm_set_msr_common+0x2230/0x2380 [kvm]
Modules linked in: vhost_net(E) vhost(E) macvtap(E) macvlan(E) tap(E)
esp4(E) xt_CHECKSUM(E) xt_MASQUERADE(E) tun(E) bridge(E) stp(E) llc(E) ip6t_rpfilter(E) nf_log_ipv6(E) ip6t_REJECT(E) nf_reject_ipv6> mei_hdcp(E) kvm(E) intel_cstate(E) intel_uncore(E) intel_rapl_perf(E) eeepc_wmi(E) asus_wmi(E) sparse_keymap(E) rfkill(E) snd_hda_codec_generic(E) pcspkr(E) wmi_bmof(E) ledtrig_audio(E) i2c_i801(E) snd>
CPU: 2 PID: 9296 Comm: CPU 1/KVM Tainted: P        W  OE     5.3.11+ #16
Hardware name: System manufacturer System Product Name/Z170-K, BIOS 3805 05/16/2018
RIP: 0010:kvm_set_msr_common+0x2230/0x2380 [kvm]
Code: b0 26 00 00 e8 91 9f b5 ce 66 90 bf 06 00 00 00 48 8b b3 88 26 00 00
e8 7e 9f b5 ce 66 90 83 a3 60 26 00 00 fb e9 e9 ec ff ff <0f> 0b e9 d2 ec ff ff f0 80 4b 31 10 e9 32 ee ff ff 48 8b 83 98 02
RSP: 0018:ffffb42e03d17d30 EFLAGS: 00010002
RAX: 0000000080004b20 RBX: ffff98f1783abf40 RCX: ffff98f17757f000
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000
RBP: ffffb42e03d17db0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff98f1783abf70
R13: 0000000000000000 R14: 0000000000000000 R15: ffff98f1d8bc6c00 FS: 00007f02faffd700(0000) GS:ffff98f286a80000(0000) knlGS:000000f0dd174000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000001f8249cd000 CR3: 000000043d3dc003 CR4: 00000000003626e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 ? recalibrate_cpu_khz+0x10/0x10
 ? ktime_get+0x3a/0xa0
 kvm_arch_vcpu_ioctl_run+0x107/0x560 [kvm]
 kvm_init+0x6bf/0xd00 [kvm]
 ? __seccomp_filter+0x7a/0x680
 do_vfs_ioctl+0xa4/0x630
 ? security_file_ioctl+0x32/0x50
 ksys_ioctl+0x60/0x90
 __x64_sys_ioctl+0x16/0x20
 do_syscall_64+0x5f/0x1a0
 entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x7f0302457d4b
Code: 0f 1e fa 48 8b 05 3d b1 0c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff
ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 0d b1 0c 00 f7 d8 64 89 01 48
RSP: 002b:00007f02faffc6c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 00007f03005fd001 RCX: 00007f0302457d4b
RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000020
RBP: 0000000000000001 R08: 0000564efe32fa50 R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000000246 R12: 0000564efe3129c0
R13: 0000000000000000 R14: 00007f03005fc000 R15: 0000564f00123300
---[ end trace 9564a1ccad733a91 ]---

[email protected]

author

2020-01-15 21:53 UTC #7 lore raw

[Bug 206215] QEMU guest crash due to random 'general protection fault' since kernel 5.2.5 on i7-3517UE

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=206215

--- Comment #1 from Sean Christopherson ([email protected]) --- +cc Derek, who is hitting the same thing.

Quoted 37 lines · On Wed, Jan 15, 2020 at 09:18:56PM +0000, [email protected]…

https://bugzilla.kernel.org/show_bug.cgi?id=206215
            Bug ID: 206215
           Summary: QEMU guest crash due to random 'general protection
                    fault' since kernel 5.2.5 on i7-3517UE
           Product: Virtualization
           Version: unspecified
    Kernel Version: 5.5.0-0.rc6
          Hardware: x86-64
                OS: Linux
              Tree: Fedora
            Status: NEW
          Severity: blocking
          Priority: P1
         Component: kvm
          Assignee: [email protected]
          Reporter: [email protected]
        Regression: Yes
Created attachment 286831
--> https://bugzilla.kernel.org/attachment.cgi?id=286831&action=edit
relevant logs
Since kernel 5.2.5 any qemu guest fail to start due to "general protection fault"
[  188.533545] traps: gsd-wacom[1855] general protection fault
ip:7fed39b5e7b0
sp:7fff3e349620 error:0 in libglib-2.0.so.0.6200.1[7fed39ae3000+83000]
[  192.002357] traps: gvfs-fuse-sub[1560] general protection fault
ip:7f9cd88100b2 sp:7f9cd5db0bf0 error:0 in
libglib-2.0.so.0.6200.1[7f9cd87de000+83000]
Please note that kernel 5.2.4 work fine.
Tested guests with Widows Server 2016/2019 & Fedora 31

Quoted 7 lines

Attached logs show the DMESG output of the guests
Attached host files contains a WARNING thrown upong first guest start on the hypervisor:
[   49.533713] WARNING: CPU: 3 PID: 966 at arch/x86/kvm/x86.c:7963
kvm_arch_vcpu_ioctl_run+0x1927/0x1ce0 [kvm]

Between the WARN, which is

  WARN_ON_ONCE(test_thread_flag(TIF_NEED_FPU_LOAD));

and the total diff of arch/x86/kvm for 5.2.4 -> 5.2.5 is

arch/x86/kvm/x86.c 2020-01-15 13:37:08.190438719 -0800 renamed from arch/x86/kvm/x86.c 2020-01-15 13:37:05.154445843 -0800 +6 -3

@@ -3264,6 +3264,10 @@
 
        kvm_x86_ops->vcpu_load(vcpu, cpu);
 
+       fpregs_assert_state_consistent();
+       if (test_thread_flag(TIF_NEED_FPU_LOAD))
+               switch_fpu_return();
+
        /* Apply any externally detected TSC adjustments (due to suspend) */
        if (unlikely(vcpu->arch.tsc_offset_adjustment)) {
                adjust_tsc_offset_host(vcpu, vcpu->arch.tsc_offset_adjustment);

@@ -7955,9 +7959,8 @@
                wait_lapic_expire(vcpu);
        guest_enter_irqoff();
 
-       fpregs_assert_state_consistent();
-       if (test_thread_flag(TIF_NEED_FPU_LOAD))
-               switch_fpu_return();
+       /* The preempt notifier should have taken care of the FPU already.  */
+       WARN_ON_ONCE(test_thread_flag(TIF_NEED_FPU_LOAD));
 
        if (unlikely(vcpu->arch.switch_db_regs)) {
                set_debugreg(0, 7);

that's a big smoking gun pointing at commit ca7e6b286333 ("KVM: X86: Fix fpu state crash in kvm guest"), which is commit e751732486eb upstream.

1. Can you verify reverting ca7e6b286333 (or e751732486eb in upstream) solves the issue?

2. Assuming the answer is yes, on a buggy kernel, can you run with the attached patch to try get debug info?

Quoted 116 lines

[   49.533714] Modules linked in: vhost_net vhost tap tun xfrm4_tunnel
tunnel4
ipcomp xfrm_ipcomp esp4 ah4 af_key ebtable_filter ebtables ip6table_filter ip6_tables bridge stp llc nf_log_ipv4 nf_log_common xt_LOG ipt_REJECT nf_reject_ipv4 iptable_filter iptable_security iptable_raw xt_state xt_conntrack xt_DSCP xt_multiport iptable_mangle xt_TCPMSS xt_tcpmss xt_policy
xt_nat iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 intel_rapl
x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel sunrpc kvm vfat fat mei_hdcp mei_wdt snd_hda_codec_hdmi iTCO_wdt irqbypass iTCO_vendor_support snd_hda_codec_realtek snd_hda_codec_generic crct10dif_pclmul crc32_pclmul ledtrig_audio snd_hda_intel ghash_clmulni_intel snd_hda_codec intel_cstate intel_uncore snd_hda_core snd_hwdep intel_rapl_perf snd_seq snd_seq_device snd_pcm i2c_i801 r8169 lpc_ich mei_me snd_timer snd mei e1000e soundcore pcc_cpufreq tcp_bbr sch_fq ip_tables xfs i915 libcrc32c i2c_algo_bit drm_kms_helper crc32c_intel drm
[   49.533760]  serio_raw video
[   49.533764] CPU: 3 PID: 966 Comm: CPU 0/KVM Not tainted
5.2.5-200.fc30.x86_64 #1
[   49.533765] Hardware name: CompuLab 0000000-00000/Intense-PC, BIOS
IPC_2.2.400.5 X64 03/15/2018
[   49.533784] RIP: 0010:kvm_arch_vcpu_ioctl_run+0x1927/0x1ce0 [kvm]
[   49.533786] Code: 4c 89 e7 e8 1b 0b ff ff 4c 89 e7 e8 d3 8c fe ff 41 83 a4
24 e8 36 00 00 fb e9 bd ed ff ff f0 41 80 4c 24 31 10 e9 a5 ee ff ff <0f> 0b
e9
74 ed ff ff 49 8b 84 24 c8 02 00 00 a9 00 00 01 00 0f 84
[   49.533787] RSP: 0018:ffffbe4e423ffd30 EFLAGS: 00010002
[   49.533789] RAX: 0000000000004b00 RBX: 0000000000000000 RCX:
ffffa044ce958000
[   49.533790] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000000
[   49.533791] RBP: ffffbe4e423ffdd8 R08: 0000000000000000 R09:
00000000000003e8
[   49.533792] R10: 0000000000000000 R11: 0000000000000000 R12:
ffffa044d38f8000
[   49.533792] R13: 0000000000000000 R14: ffffbe4e41ccf7b8 R15:
0000000000000000
[   49.533794] FS:  00007f117953f700(0000) GS:ffffa044ee2c0000(0000)
knlGS:0000000000000000
[   49.533795] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   49.533796] CR2: 0000000000000000 CR3: 000000040e8c2003 CR4:
00000000001626e0
[   49.533797] Call Trace:
[   49.533817]  kvm_vcpu_ioctl+0x215/0x5c0 [kvm]
[   49.533821]  ? __seccomp_filter+0x7b/0x640
[   49.533824]  ? __switch_to_asm+0x34/0x70
[   49.533826]  ? __switch_to_asm+0x34/0x70
[   49.533827]  ? apic_timer_interrupt+0xa/0x20
[   49.533831]  do_vfs_ioctl+0x405/0x660
[   49.533834]  ksys_ioctl+0x5e/0x90
[   49.533836]  __x64_sys_ioctl+0x16/0x20
[   49.533839]  do_syscall_64+0x5f/0x1a0
[   49.533842]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[   49.533844] RIP: 0033:0x7f117d1fb34b
[   49.533845] Code: 0f 1e fa 48 8b 05 3d 9b 0c 00 64 c7 00 26 00 00 00 48 c7
c0 ff ff ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d
01
f0 ff ff 73 01 c3 48 8b 0d 0d 9b 0c 00 f7 d8 64 89 01 48
[   49.533846] RSP: 002b:00007f117953e698 EFLAGS: 00000246 ORIG_RAX:
0000000000000010
[   49.533848] RAX: ffffffffffffffda RBX: 0000564f2cb65ba0 RCX:
00007f117d1fb34b
[   49.533849] RDX: 0000000000000000 RSI: 000000000000ae80 RDI:
0000000000000019
[   49.533850] RBP: 00007f1179f20000 R08: 0000564f2b7e5390 R09:
000000000000ffff
[   49.533851] R10: 0000564f2ca7a710 R11: 0000000000000246 R12:
0000000000000001
[   49.533852] R13: 00007f1179f21002 R14: 0000000000000000 R15:
0000564f2bc66e80
[   49.533854] ---[ end trace a562473b18c9b742 ]---
/proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 58
model name : Intel(R) Core(TM) i7-3517UE CPU @ 1.70GHz
stepping : 9
microcode : 0x1f
cpu MHz : 828.296
cache size : 4096 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 2
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm
constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr
pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm cpuid_fault epb pti ibrs ibpb stibp tpr_shadow vnmi flexpriority
ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
mds
bogomips : 4389.89
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:
Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.

Signature · 2 lines

You are receiving this mail because:
You are watching the assignee of the bug.

[email protected]

author

2020-01-15 22:15 UTC #8 lore raw

[Bug 206215] QEMU guest crash due to random 'general protection fault' since kernel 5.2.5 on i7-3517UE

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=206215

--- Comment #2 from [email protected] ---
(In reply to Sean Christopherson from comment #1)

Quoted 201 lines

Created attachment 286833 [details] 0001-thread_info-Add-a-debug-hook-to-detect-FPU-changes-w.patch
+cc Derek, who is hitting the same thing.
Quoted 33 lines · On Wed, Jan 15, 2020 at 09:18:56PM +0000, [email protected]…
https://bugzilla.kernel.org/show_bug.cgi?id=206215
            Bug ID: 206215
           Summary: QEMU guest crash due to random 'general protection
                    fault' since kernel 5.2.5 on i7-3517UE
           Product: Virtualization
           Version: unspecified
    Kernel Version: 5.5.0-0.rc6
          Hardware: x86-64
                OS: Linux
              Tree: Fedora
            Status: NEW
          Severity: blocking
          Priority: P1
         Component: kvm
          Assignee: [email protected]
          Reporter: [email protected]
        Regression: Yes
Created attachment 286831 [details]
--> https://bugzilla.kernel.org/attachment.cgi?id=286831&action=edit
relevant logs
Since kernel 5.2.5 any qemu guest fail to start due to "general protection fault"
[  188.533545] traps: gsd-wacom[1855] general protection fault
ip:7fed39b5e7b0
sp:7fff3e349620 error:0 in libglib-2.0.so.0.6200.1[7fed39ae3000+83000]
[  192.002357] traps: gvfs-fuse-sub[1560] general protection fault
ip:7f9cd88100b2 sp:7f9cd5db0bf0 error:0 in
libglib-2.0.so.0.6200.1[7f9cd87de000+83000]
Quoted 7 lines
Please note that kernel 5.2.4 work fine.
Tested guests with Widows Server 2016/2019 & Fedora 31
Attached logs show the DMESG output of the guests
Attached host files contains a WARNING thrown upong first guest start on
the
Quoted 4 lines
hypervisor:
[   49.533713] WARNING: CPU: 3 PID: 966 at arch/x86/kvm/x86.c:7963
kvm_arch_vcpu_ioctl_run+0x1927/0x1ce0 [kvm]
Between the WARN, which is
  WARN_ON_ONCE(test_thread_flag(TIF_NEED_FPU_LOAD));
and the total diff of arch/x86/kvm for 5.2.4 -> 5.2.5 is
arch/x86/kvm/x86.c 2020-01-15 13:37:08.190438719 -0800 renamed from arch/x86/kvm/x86.c 2020-01-15 13:37:05.154445843 -0800 +4 -0
@@ -3264,6 +3264,10 @@
 
        kvm_x86_ops->vcpu_load(vcpu, cpu);
 
+       fpregs_assert_state_consistent();
+       if (test_thread_flag(TIF_NEED_FPU_LOAD))
+               switch_fpu_return();
+
        /* Apply any externally detected TSC adjustments (due to suspend) */
        if (unlikely(vcpu->arch.tsc_offset_adjustment)) {
                adjust_tsc_offset_host(vcpu,
vcpu->arch.tsc_offset_adjustment);
arch/x86/kvm/x86.c 2020-01-15 13:37:08.190438719 -0800 renamed from arch/x86/kvm/x86.c 2020-01-15 13:37:05.154445843 -0800 +2 -3
@@ -7955,9 +7959,8 @@
                wait_lapic_expire(vcpu);
        guest_enter_irqoff();
 
-       fpregs_assert_state_consistent();
-       if (test_thread_flag(TIF_NEED_FPU_LOAD))
-               switch_fpu_return();
+       /* The preempt notifier should have taken care of the FPU already. 
 */
+       WARN_ON_ONCE(test_thread_flag(TIF_NEED_FPU_LOAD));
 
        if (unlikely(vcpu->arch.switch_db_regs)) {
                set_debugreg(0, 7); 
that's a big smoking gun pointing at commit ca7e6b286333 ("KVM: X86: Fix fpu state crash in kvm guest"), which is commit e751732486eb upstream.
1. Can you verify reverting ca7e6b286333 (or e751732486eb in upstream) solves the issue?
2. Assuming the answer is yes, on a buggy kernel, can you run with the attached patch to try get debug info?
Quoted 10 lines
[   49.533714] Modules linked in: vhost_net vhost tap tun xfrm4_tunnel
tunnel4
ipcomp xfrm_ipcomp esp4 ah4 af_key ebtable_filter ebtables ip6table_filter ip6_tables bridge stp llc nf_log_ipv4 nf_log_common xt_LOG ipt_REJECT nf_reject_ipv4 iptable_filter iptable_security iptable_raw xt_state xt_conntrack xt_DSCP xt_multiport iptable_mangle xt_TCPMSS xt_tcpmss xt_policy
xt_nat iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 intel_rapl
x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel sunrpc kvm vfat
fat
Quoted 7 lines
mei_hdcp mei_wdt snd_hda_codec_hdmi iTCO_wdt irqbypass iTCO_vendor_support snd_hda_codec_realtek snd_hda_codec_generic crct10dif_pclmul crc32_pclmul ledtrig_audio snd_hda_intel ghash_clmulni_intel snd_hda_codec intel_cstate intel_uncore snd_hda_core snd_hwdep intel_rapl_perf snd_seq snd_seq_device snd_pcm i2c_i801 r8169 lpc_ich mei_me snd_timer snd mei e1000e soundcore pcc_cpufreq tcp_bbr sch_fq ip_tables xfs i915 libcrc32c i2c_algo_bit drm_kms_helper crc32c_intel drm
Quoted 7 lines
[   49.533760]  serio_raw video
[   49.533764] CPU: 3 PID: 966 Comm: CPU 0/KVM Not tainted
5.2.5-200.fc30.x86_64 #1
[   49.533765] Hardware name: CompuLab 0000000-00000/Intense-PC, BIOS
IPC_2.2.400.5 X64 03/15/2018
[   49.533784] RIP: 0010:kvm_arch_vcpu_ioctl_run+0x1927/0x1ce0 [kvm]
[   49.533786] Code: 4c 89 e7 e8 1b 0b ff ff 4c 89 e7 e8 d3 8c fe ff 41 83
a4
Quoted 1 line
24 e8 36 00 00 fb e9 bd ed ff ff f0 41 80 4c 24 31 10 e9 a5 ee ff ff <0f>
0b
Quoted 23 lines
e9
74 ed ff ff 49 8b 84 24 c8 02 00 00 a9 00 00 01 00 0f 84
[   49.533787] RSP: 0018:ffffbe4e423ffd30 EFLAGS: 00010002
[   49.533789] RAX: 0000000000004b00 RBX: 0000000000000000 RCX:
ffffa044ce958000
[   49.533790] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000000
[   49.533791] RBP: ffffbe4e423ffdd8 R08: 0000000000000000 R09:
00000000000003e8
[   49.533792] R10: 0000000000000000 R11: 0000000000000000 R12:
ffffa044d38f8000
[   49.533792] R13: 0000000000000000 R14: ffffbe4e41ccf7b8 R15:
0000000000000000
[   49.533794] FS:  00007f117953f700(0000) GS:ffffa044ee2c0000(0000)
knlGS:0000000000000000
[   49.533795] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   49.533796] CR2: 0000000000000000 CR3: 000000040e8c2003 CR4:
00000000001626e0
[   49.533797] Call Trace:
[   49.533817]  kvm_vcpu_ioctl+0x215/0x5c0 [kvm]
[   49.533821]  ? __seccomp_filter+0x7b/0x640
[   49.533824]  ? __switch_to_asm+0x34/0x70
[   49.533826]  ? __switch_to_asm+0x34/0x70
Quoted 8 lines
[   49.533827]  ? apic_timer_interrupt+0xa/0x20
[   49.533831]  do_vfs_ioctl+0x405/0x660
[   49.533834]  ksys_ioctl+0x5e/0x90
[   49.533836]  __x64_sys_ioctl+0x16/0x20
[   49.533839]  do_syscall_64+0x5f/0x1a0
[   49.533842]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[   49.533844] RIP: 0033:0x7f117d1fb34b
[   49.533845] Code: 0f 1e fa 48 8b 05 3d 9b 0c 00 64 c7 00 26 00 00 00 48
c7
Quoted 1 line
c0 ff ff ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48>
3d
Quoted 34 lines
01
f0 ff ff 73 01 c3 48 8b 0d 0d 9b 0c 00 f7 d8 64 89 01 48
[   49.533846] RSP: 002b:00007f117953e698 EFLAGS: 00000246 ORIG_RAX:
0000000000000010
[   49.533848] RAX: ffffffffffffffda RBX: 0000564f2cb65ba0 RCX:
00007f117d1fb34b
[   49.533849] RDX: 0000000000000000 RSI: 000000000000ae80 RDI:
0000000000000019
[   49.533850] RBP: 00007f1179f20000 R08: 0000564f2b7e5390 R09:
000000000000ffff
[   49.533851] R10: 0000564f2ca7a710 R11: 0000000000000246 R12:
0000000000000001
[   49.533852] R13: 00007f1179f21002 R14: 0000000000000000 R15:
0000564f2bc66e80
[   49.533854] ---[ end trace a562473b18c9b742 ]---
/proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 58
model name : Intel(R) Core(TM) i7-3517UE CPU @ 1.70GHz
stepping : 9
microcode : 0x1f
cpu MHz : 828.296
cache size : 4096 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 2
apicid : 0
initial apicid : 0
fpu : yes
Quoted 8 lines
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm
constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc

Quoted 18 lines

cpuid
Quoted 17 lines
aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr
pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm cpuid_fault epb pti ibrs ibpb stibp tpr_shadow vnmi flexpriority
ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
mds
bogomips : 4389.89
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:
Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.

Sean,
Thank you for the quick feedback.
In the zip file I did attach DMESG logs with latest vanilla kernel with same behavior:
5.5.0-0.rc6.git1.1.vanilla.knurd.1.fc31.x86_64

If I'm rebuilding the kernel I'd rather spend time on the most recent one ?

Can you confirm if I apply the patch on kernel 5.5.0-rc6 ?

Signature · 2 lines

You are receiving this mail because:
You are watching the assignee of the bug.

[email protected]

author

2020-01-16 01:15 UTC #9 lore raw

[Bug 206215] QEMU guest crash due to random 'general protection fault' since kernel 5.2.5 on i7-3517UE

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=206215

--- Comment #3 from [email protected] ---

Quoted 9 lines · Sean Christopherson

+cc Derek, who is hitting the same thing.
Quoted 1 line · On Wed, Jan 15, 2020 at 09:18:56PM +0000, [email protected]…
https://bugzilla.kernel.org/show_bug.cgi?id=206215
*snip*
that's a big smoking gun pointing at commit ca7e6b286333 ("KVM: X86: Fix fpu state crash in kvm guest"), which is commit e751732486eb upstream.

Quoted 5 lines

1. Can you verify reverting ca7e6b286333 (or e751732486eb in upstream) solves the issue?
2. Assuming the answer is yes, on a buggy kernel, can you run with the attached patch to try get debug info?

I did these out of order since I had 5.3.11 built with the patch, ready to go for weeks now, waiting for an opportunity to test.

Win10 guest immediately BSOD'ed with:

WARNING: CPU: 2 PID: 9296 at include/linux/thread_info.h:55 
kernel_fpu_begin+0x6b/0xc0

Then stashed the patch, reverted ca7e6b286333, compile, reboot.

Guest is running stable now on 5.3.11. Did test my CAD under the guest, did not experience the crashes that had me stuck at 5.1.

Signature · 2 lines

You are receiving this mail because:
You are watching the assignee of the bug.

[email protected]

author

2020-01-16 01:36 UTC #10 lore raw

[Bug 206215] QEMU guest crash due to random 'general protection fault' since kernel 5.2.5 on i7-3517UE

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=206215

--- Comment #4 from [email protected] ---
(In reply to derek from comment #3)

Quoted 11 lines · Sean Christopherson

Quoted 3 lines
+cc Derek, who is hitting the same thing.
On Wed, Jan 15, 2020 at 09:18:56PM +0000,
[email protected]
Quoted 6 lines
Quoted 1 line · wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=206215
*snip*
that's a big smoking gun pointing at commit ca7e6b286333 ("KVM: X86: Fix fpu state crash in kvm guest"), which is commit e751732486eb upstream.

Quoted 7 lines

Quoted 5 lines
1. Can you verify reverting ca7e6b286333 (or e751732486eb in upstream) solves the issue?
2. Assuming the answer is yes, on a buggy kernel, can you run with the attached patch to try get debug info?
I did these out of order since I had 5.3.11 built with the patch, ready to go

Sean,
I'm not familiar with rebuilding the kernel, nor applying a patch but I'm working on it right now so I can provide feedback.

Quoted 19 lines

for weeks now, waiting for an opportunity to test.
Win10 guest immediately BSOD'ed with:
WARNING: CPU: 2 PID: 9296 at include/linux/thread_info.h:55 
kernel_fpu_begin+0x6b/0xc0
Then stashed the patch, reverted ca7e6b286333, compile, reboot.
Guest is running stable now on 5.3.11. Did test my CAD under the guest, did not
experience the crashes that had me stuck at 5.1.
I did these out of order since I had 5.3.11 built with the patch, ready to go
for weeks now, waiting for an opportunity to test.
Win10 guest immediately BSOD'ed with:

Quoted 8 lines

WARNING: CPU: 2 PID: 9296 at include/linux/thread_info.h:55 
kernel_fpu_begin+0x6b/0xc0
Then stashed the patch, reverted ca7e6b286333, compile, reboot.
Guest is running stable now on 5.3.11. Did test my CAD under the guest, did not
experience the crashes that had me stuck at 5.1.

Derek,

Thanks for the update.

I'm still curious about the hypervisor CPU model you have.

On Windows I did exprecience a different behavior though.
The OS did boot but the spice/VNC screen did freeze randomly. One of my windows VM did end up being corrupted as an update ehich tried to install just failed miserably.

Anyways, I will try to provide debug details asap from the patch that Sean provided...

To be continued ...

Signature · 2 lines

You are receiving this mail because:
You are watching the assignee of the bug.

[email protected]

author

2020-01-16 15:38 UTC #11 lore raw

[Bug 206215] QEMU guest crash due to random 'general protection fault' since kernel 5.2.5 on i7-3517UE

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=206215

--- Comment #5 from Sean Christopherson ([email protected]) ---

Quoted 15 lines · Derek Yerger

Quoted 3 lines · Sean Christopherson
+cc Derek, who is hitting the same thing.
On Wed, Jan 15, 2020 at 09:18:56PM +0000,
Quoted 10 lines · [email protected] wrote:
Quoted 1 line
https://bugzilla.kernel.org/show_bug.cgi?id=206215
*snip*
that's a big smoking gun pointing at commit ca7e6b286333 ("KVM: X86: Fix fpu state crash in kvm guest"), which is commit e751732486eb upstream.
1. Can you verify reverting ca7e6b286333 (or e751732486eb in upstream) solves the issue?
2. Assuming the answer is yes, on a buggy kernel, can you run with the attached patch to try get debug info?

Quoted 7 lines

I did these out of order since I had 5.3.11 built with the patch, ready to go for weeks now, waiting for an opportunity to test.
Win10 guest immediately BSOD'ed with:
WARNING: CPU: 2 PID: 9296 at include/linux/thread_info.h:55
kernel_fpu_begin+0x6b/0xc0

Can you provide the full stack trace of the WARN? I'm hoping that will provide a hint as to what's going wrong.

Quoted 4 lines

Then stashed the patch, reverted ca7e6b286333, compile, reboot.
Guest is running stable now on 5.3.11. Did test my CAD under the guest, did not experience the crashes that had me stuck at 5.1.

Signature · 2 lines

You are receiving this mail because:
You are watching the assignee of the bug.

[email protected]

author

2020-01-16 18:08 UTC #12 lore raw

[Bug 206215] QEMU guest crash due to random 'general protection fault' since kernel 5.2.5 on i7-3517UE

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=206215

--- Comment #6 from Sean Christopherson ([email protected]) ---

Quoted 19 lines · Sean Christopherson

Quoted 4 lines · Derek Yerger
Quoted 3 lines · Sean Christopherson
+cc Derek, who is hitting the same thing.
On Wed, Jan 15, 2020 at 09:18:56PM +0000,
Quoted 13 lines · [email protected] wrote:
Quoted 10 lines
Quoted 1 line
https://bugzilla.kernel.org/show_bug.cgi?id=206215
*snip*
that's a big smoking gun pointing at commit ca7e6b286333 ("KVM: X86: Fix fpu state crash in kvm guest"), which is commit e751732486eb upstream.
1. Can you verify reverting ca7e6b286333 (or e751732486eb in upstream) solves the issue?
2. Assuming the answer is yes, on a buggy kernel, can you run with the attached patch to try get debug info?
I did these out of order since I had 5.3.11 built with the patch, ready to go for weeks now, waiting for an opportunity to test.

Quoted 7 lines

Quoted 4 lines
Win10 guest immediately BSOD'ed with:
WARNING: CPU: 2 PID: 9296 at include/linux/thread_info.h:55
kernel_fpu_begin+0x6b/0xc0
Can you provide the full stack trace of the WARN? I'm hoping that will provide a hint as to what's going wrong.

Aha! I found at least two cases where TIF_NEED_FPU_LOAD could be set without the vCPU being preempted.

The comment on fpregs_lock() states that softirq can set TIF_NEED_FPU_LOAD, which would not be handled by the preempt notifier.

/*
* Use fpregs_lock() while editing CPU's FPU registers or fpu->state.
* A context switch will (and softirq might) save CPU's FPU registers to

                           ^^^^^^^^^^^^^^^^^^^

* fpu->state and set TIF_NEED_FPU_LOAD leaving CPU's FPU registers in
* a random state.
*/
static inline void fpregs_lock(void)

  __warn+0xcb/0xf0
  warn_slowpath_null+0x1d/0x20
  kernel_fpu_disable+0x3f/0x50
  __kernel_fpu_begin+0x49/0x100
  kernel_fpu_begin+0xe/0x10
  crc32c_pcl_intel_update+0x84/0xb0
  crypto_shash_update+0x3f/0x110
  crc32c+0x63/0x8a [libcrc32c]
  dm_bm_checksum+0x1b/0x20 [dm_persistent_data]
  node_prepare_for_write+0x44/0x70 [dm_persistent_data]
  dm_block_manager_write_callback+0x41/0x50 [dm_persistent_data]
  submit_io+0x170/0x1b0 [dm_bufio]
  __write_dirty_buffer+0x89/0x90 [dm_bufio]
  __make_buffer_clean+0x4f/0x80 [dm_bufio]
  __try_evict_buffer+0x42/0x60 [dm_bufio]
  dm_bufio_shrink_scan+0xc0/0x130 [dm_bufio]
  shrink_slab.part.40+0x1f5/0x420
  shrink_node+0x22c/0x320
  do_try_to_free_pages+0xf5/0x330
  try_to_free_pages+0xe9/0x190
  __alloc_pages_slowpath+0x40f/0xba0
  __alloc_pages_nodemask+0x209/0x260
  alloc_pages_vma+0x1f1/0x250
  do_huge_pmd_anonymous_page+0x123/0x660
  handle_mm_fault+0xfd3/0x1330
  __get_user_pages+0x113/0x640
  get_user_pages+0x4f/0x60
  __gfn_to_pfn_memslot+0x120/0x3f0 [kvm]
  try_async_pf+0x66/0x230 [kvm]
  tdp_page_fault+0x130/0x280 [kvm]
  kvm_mmu_page_fault+0x60/0x120 [kvm]
  handle_ept_violation+0x91/0x170 [kvm_intel]
  vmx_handle_exit+0x1ca/0x1400 [kvm_intel]

Either of the above explains why pre-e751732486eb code waited until IRQs are disabled by vcpu_enter_guest() to do switch_fpu_return().

Quoted 4 lines

Quoted 4 lines
Then stashed the patch, reverted ca7e6b286333, compile, reboot.
Guest is running stable now on 5.3.11. Did test my CAD under the guest, did not experience the crashes that had me stuck at 5.1.

Signature · 2 lines

You are receiving this mail because:
You are watching the assignee of the bug.

[email protected]

author

2020-01-16 19:21 UTC #13 lore raw

[Bug 206215] QEMU guest crash due to random 'general protection fault' since kernel 5.2.5 on i7-3517UE

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=206215

--- Comment #7 from [email protected] ---

Quoted 19 lines · Sean Christopherson

Quoted 18 lines · Derek Yerger
Quoted 9 lines · Sean Christopherson
+cc Derek, who is hitting the same thing.
Quoted 1 line · On Wed, Jan 15, 2020 at 09:18:56PM +0000, [email protected]…
https://bugzilla.kernel.org/show_bug.cgi?id=206215
*snip*
that's a big smoking gun pointing at commit ca7e6b286333 ("KVM: X86: Fix fpu state crash in kvm guest"), which is commit e751732486eb upstream.
Quoted 5 lines
1. Can you verify reverting ca7e6b286333 (or e751732486eb in upstream) solves the issue?
2. Assuming the answer is yes, on a buggy kernel, can you run with the attached patch to try get debug info?
I did these out of order since I had 5.3.11 built with the patch, ready to go for weeks now, waiting for an opportunity to test.

Quoted 6 lines

Quoted 4 lines
Win10 guest immediately BSOD'ed with:
WARNING: CPU: 2 PID: 9296 at include/linux/thread_info.h:55
kernel_fpu_begin+0x6b/0xc0
Can you provide the full stack trace of the WARN? I'm hoping that will provide a hint as to what's going wrong.

WARNING: CPU: 2 PID: 9296 at include/linux/thread_info.h:55 
kernel_fpu_begin+0x6b/0xc0
Modules linked in: vhost_net(E) vhost(E) macvtap(E) macvlan(E) tap(E) esp4(E)

xt_CHECKSUM(E) xt_MASQUERADE(E) tun(E) bridge(E) stp(E) llc(E) ip6t_rpfilter(E) nf_log_ipv6(E) ip6t_REJECT(E) nf_reject_ipv6>
mei_hdcp(E) kvm(E) intel_cstate(E) intel_uncore(E) intel_rapl_perf(E)
eeepc_wmi(E) asus_wmi(E) sparse_keymap(E) rfkill(E) snd_hda_codec_generic(E) pcspkr(E) wmi_bmof(E) ledtrig_audio(E) i2c_i801(E) snd>

CPU: 2 PID: 9296 Comm: CPU 1/KVM Tainted: P           OE     5.3.11+ #16
Hardware name: System manufacturer System Product Name/Z170-K, BIOS 3805
05/16/2018
RIP: 0010:kernel_fpu_begin+0x6b/0xc0
Code: f6 40 26 20 75 08 48 8b 10 80 e6 40 74 16 65 48 c7 05 b5 27 fe 70 00 00

00
00 c3 65 8a 05 a5 27 fe 70 eb c4 80 78 0c 00 74 02 <0f> 0b 48 83 c0 01 f0 80 08 40 65 48 8b 0c 25 c0 6b 01 00 0f 1f 44
RSP: 0018:ffffb42e0014c7f8 EFLAGS: 00010202
RAX: ffff98f1783a1ec0 RBX: 0000000000000038 RCX: 0000000000000048
RDX: 0000000000000020 RSI: ffff98f1d9a5cb00 RDI: ffff98f1d9a5cb00
RBP: ffffb42e0014caa0 R08: ffffb42e0014cab0 R09: ffffb42e0014c860
R10: 0000000000000000 R11: 0000000000000000 R12: 00000000000002ba
R13: ffffb42e0014c860 R14: ffff98f1d36882aa R15: ffff98f1d9a5caa8 FS: 00007f02faffd700(0000) GS:ffff98f286a80000(0000) knlGS:000000f0dd174000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000001f8249cd000 CR3: 000000043d3dc003 CR4: 00000000003626e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400

Call Trace:
  <IRQ>
  gcmaes_crypt_by_sg.constprop.12+0x26e/0x660
  ? 0xffffffffc024547d
  ? __qdisc_run+0x83/0x510
  ? __dev_queue_xmit+0x45e/0x990
  ? ip_finish_output2+0x1a8/0x570
  ? fib4_rule_action+0x61/0x70
  ? fib4_rule_action+0x70/0x70
  ? fib_rules_lookup+0x13f/0x1c0
  ? helper_rfc4106_decrypt+0x82/0xa0
  ? crypto_aead_decrypt+0x40/0x70
  ? crypto_aead_decrypt+0x40/0x70
  ? crypto_aead_decrypt+0x40/0x70
  ? esp_output_tail+0x8f4/0xa5a [esp4]
  ? skb_ext_add+0xd3/0x170
  ? xfrm_input+0x7a6/0x12c0
  ? xfrm4_rcv_encap+0xae/0xd0
  ? xfrm4_transport_finish+0x200/0x200
  ? udp_queue_rcv_one_skb+0x1ba/0x460
  ? udp_unicast_rcv_skb.isra.63+0x72/0x90
  ? __udp4_lib_rcv+0x51b/0xb00
  ? ip_protocol_deliver_rcu+0xd2/0x1c0
  ? ip_local_deliver_finish+0x44/0x50
  ? ip_local_deliver+0xe0/0xf0
  ? ip_protocol_deliver_rcu+0x1c0/0x1c0
  ? ip_rcv+0xbc/0xd0
  ? ip_rcv_finish_core.isra.19+0x380/0x380
  ? __netif_receive_skb_one_core+0x7e/0x90
  ? netif_receive_skb_internal+0x3d/0xb0
  ? napi_gro_receive+0xed/0x150
  ? 0xffffffffc0243c77
  ? net_rx_action+0x149/0x3b0
  ? __do_softirq+0xe4/0x2f8
  ? handle_irq_event_percpu+0x6a/0x80
  ? irq_exit+0xe6/0xf0
  ? do_IRQ+0x7f/0xd0
  ? common_interrupt+0xf/0xf
  </IRQ>
  ? irq_entries_start+0x20/0x660
  ? vmx_get_interrupt_shadow+0x2f0/0x710 [kvm_intel]
  ? kvm_set_msr_common+0xfc7/0x2380 [kvm]
  ? recalibrate_cpu_khz+0x10/0x10
  ? ktime_get+0x3a/0xa0
  ? kvm_arch_vcpu_ioctl_run+0x107/0x560 [kvm]
  ? kvm_init+0x6bf/0xd00 [kvm]
  ? __seccomp_filter+0x7a/0x680
  ? do_vfs_ioctl+0xa4/0x630
  ? security_file_ioctl+0x32/0x50
  ? ksys_ioctl+0x60/0x90
  ? __x64_sys_ioctl+0x16/0x20
  ? do_syscall_64+0x5f/0x1a0
  ? entry_SYSCALL_64_after_hwframe+0x44/0xa9
---[ end trace 9564a1ccad733a90 ]---
WARNING: CPU: 2 PID: 9296 at arch/x86/kvm/x86.c:8060 
kvm_set_msr_common+0x2230/0x2380 [kvm]
Modules linked in: vhost_net(E) vhost(E) macvtap(E) macvlan(E) tap(E) esp4(E)

CPU: 2 PID: 9296 Comm: CPU 1/KVM Tainted: P        W  OE     5.3.11+ #16
Hardware name: System manufacturer System Product Name/Z170-K, BIOS 3805
05/16/2018
RIP: 0010:kvm_set_msr_common+0x2230/0x2380 [kvm]
Code: b0 26 00 00 e8 91 9f b5 ce 66 90 bf 06 00 00 00 48 8b b3 88 26 00 00 e8

7e
9f b5 ce 66 90 83 a3 60 26 00 00 fb e9 e9 ec ff ff <0f> 0b e9 d2 ec ff ff f0 80 4b 31 10 e9 32 ee ff ff 48 8b 83 98 02
RSP: 0018:ffffb42e03d17d30 EFLAGS: 00010002
RAX: 0000000080004b20 RBX: ffff98f1783abf40 RCX: ffff98f17757f000
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000
RBP: ffffb42e03d17db0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff98f1783abf70
R13: 0000000000000000 R14: 0000000000000000 R15: ffff98f1d8bc6c00 FS: 00007f02faffd700(0000) GS:ffff98f286a80000(0000) knlGS:000000f0dd174000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000001f8249cd000 CR3: 000000043d3dc003 CR4: 00000000003626e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400

Call Trace:
  ? recalibrate_cpu_khz+0x10/0x10
  ? ktime_get+0x3a/0xa0
  kvm_arch_vcpu_ioctl_run+0x107/0x560 [kvm]
  kvm_init+0x6bf/0xd00 [kvm]
  ? __seccomp_filter+0x7a/0x680
  do_vfs_ioctl+0xa4/0x630
  ? security_file_ioctl+0x32/0x50
  ksys_ioctl+0x60/0x90
  __x64_sys_ioctl+0x16/0x20
  do_syscall_64+0x5f/0x1a0
  entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x7f0302457d4b
Code: 0f 1e fa 48 8b 05 3d b1 0c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff

c3
66 0f 1f 44 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 0d b1 0c 00 f7 d8 64 89 01 48
RSP: 002b:00007f02faffc6c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 00007f03005fd001 RCX: 00007f0302457d4b
RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000020
RBP: 0000000000000001 R08: 0000564efe32fa50 R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000000246 R12: 0000564efe3129c0
R13: 0000000000000000 R14: 00007f03005fc000 R15: 0000564f00123300

---[ end trace 9564a1ccad733a91 ]---

Signature · 2 lines

You are receiving this mail because:
You are watching the assignee of the bug.

[email protected]

author

2020-01-16 19:32 UTC #14 lore raw

[Bug 206215] QEMU guest crash due to random 'general protection fault' since kernel 5.2.5 on i7-3517UE

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=206215

--- Comment #8 from Sean Christopherson ([email protected]) ---

Quoted 25 lines · Derek Yerger

 <IRQ>
 gcmaes_crypt_by_sg.constprop.12+0x26e/0x660
 ? 0xffffffffc024547d
 ? __qdisc_run+0x83/0x510
 ? __dev_queue_xmit+0x45e/0x990
 ? ip_finish_output2+0x1a8/0x570
 ? fib4_rule_action+0x61/0x70
 ? fib4_rule_action+0x70/0x70
 ? fib_rules_lookup+0x13f/0x1c0
 ? helper_rfc4106_decrypt+0x82/0xa0
 ? crypto_aead_decrypt+0x40/0x70
 ? crypto_aead_decrypt+0x40/0x70
 ? crypto_aead_decrypt+0x40/0x70
 ? esp_output_tail+0x8f4/0xa5a [esp4]
 ? skb_ext_add+0xd3/0x170
 ? xfrm_input+0x7a6/0x12c0
 ? xfrm4_rcv_encap+0xae/0xd0
 ? xfrm4_transport_finish+0x200/0x200
 ? udp_queue_rcv_one_skb+0x1ba/0x460
 ? udp_unicast_rcv_skb.isra.63+0x72/0x90
 ? __udp4_lib_rcv+0x51b/0xb00
 ? ip_protocol_deliver_rcu+0xd2/0x1c0
 ? ip_local_deliver_finish+0x44/0x50
 ? ip_local_deliver+0xe0/0xf0
 ? ip_protocol_deliver_rcu+0x1c0/0x1c0

Quoted 8 lines

 ? ip_rcv+0xbc/0xd0
 ? ip_rcv_finish_core.isra.19+0x380/0x380
 ? __netif_receive_skb_one_core+0x7e/0x90
 ? netif_receive_skb_internal+0x3d/0xb0
 ? napi_gro_receive+0xed/0x150
 ? 0xffffffffc0243c77
 ? net_rx_action+0x149/0x3b0
 ? __do_softirq+0xe4/0x2f8

Bingo! Thanks Derek!

Quoted 70 lines

 ? handle_irq_event_percpu+0x6a/0x80
 ? irq_exit+0xe6/0xf0
 ? do_IRQ+0x7f/0xd0
 ? common_interrupt+0xf/0xf
 </IRQ>
 ? irq_entries_start+0x20/0x660
 ? vmx_get_interrupt_shadow+0x2f0/0x710 [kvm_intel]
 ? kvm_set_msr_common+0xfc7/0x2380 [kvm]
 ? recalibrate_cpu_khz+0x10/0x10
 ? ktime_get+0x3a/0xa0
 ? kvm_arch_vcpu_ioctl_run+0x107/0x560 [kvm]
 ? kvm_init+0x6bf/0xd00 [kvm]
 ? __seccomp_filter+0x7a/0x680
 ? do_vfs_ioctl+0xa4/0x630
 ? security_file_ioctl+0x32/0x50
 ? ksys_ioctl+0x60/0x90
 ? __x64_sys_ioctl+0x16/0x20
 ? do_syscall_64+0x5f/0x1a0
 ? entry_SYSCALL_64_after_hwframe+0x44/0xa9
---[ end trace 9564a1ccad733a90 ]---
WARNING: CPU: 2 PID: 9296 at arch/x86/kvm/x86.c:8060
kvm_set_msr_common+0x2230/0x2380 [kvm]
Modules linked in: vhost_net(E) vhost(E) macvtap(E) macvlan(E) tap(E)
esp4(E) xt_CHECKSUM(E) xt_MASQUERADE(E) tun(E) bridge(E) stp(E) llc(E) ip6t_rpfilter(E) nf_log_ipv6(E) ip6t_REJECT(E) nf_reject_ipv6> mei_hdcp(E) kvm(E) intel_cstate(E) intel_uncore(E) intel_rapl_perf(E) eeepc_wmi(E) asus_wmi(E) sparse_keymap(E) rfkill(E) snd_hda_codec_generic(E) pcspkr(E) wmi_bmof(E) ledtrig_audio(E) i2c_i801(E) snd>
CPU: 2 PID: 9296 Comm: CPU 1/KVM Tainted: P        W  OE     5.3.11+ #16
Hardware name: System manufacturer System Product Name/Z170-K, BIOS 3805
05/16/2018
RIP: 0010:kvm_set_msr_common+0x2230/0x2380 [kvm]
Code: b0 26 00 00 e8 91 9f b5 ce 66 90 bf 06 00 00 00 48 8b b3 88 26 00 00
e8 7e 9f b5 ce 66 90 83 a3 60 26 00 00 fb e9 e9 ec ff ff <0f> 0b e9 d2 ec ff ff f0 80 4b 31 10 e9 32 ee ff ff 48 8b 83 98 02
RSP: 0018:ffffb42e03d17d30 EFLAGS: 00010002
RAX: 0000000080004b20 RBX: ffff98f1783abf40 RCX: ffff98f17757f000
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000
RBP: ffffb42e03d17db0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff98f1783abf70
R13: 0000000000000000 R14: 0000000000000000 R15: ffff98f1d8bc6c00 FS: 00007f02faffd700(0000) GS:ffff98f286a80000(0000) knlGS:000000f0dd174000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000001f8249cd000 CR3: 000000043d3dc003 CR4: 00000000003626e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 ? recalibrate_cpu_khz+0x10/0x10
 ? ktime_get+0x3a/0xa0
 kvm_arch_vcpu_ioctl_run+0x107/0x560 [kvm]
 kvm_init+0x6bf/0xd00 [kvm]
 ? __seccomp_filter+0x7a/0x680
 do_vfs_ioctl+0xa4/0x630
 ? security_file_ioctl+0x32/0x50
 ksys_ioctl+0x60/0x90
 __x64_sys_ioctl+0x16/0x20
 do_syscall_64+0x5f/0x1a0
 entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x7f0302457d4b
Code: 0f 1e fa 48 8b 05 3d b1 0c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff
ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 0d b1 0c 00 f7 d8 64 89 01 48
RSP: 002b:00007f02faffc6c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 00007f03005fd001 RCX: 00007f0302457d4b
RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000020
RBP: 0000000000000001 R08: 0000564efe32fa50 R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000000246 R12: 0000564efe3129c0
R13: 0000000000000000 R14: 00007f03005fc000 R15: 0000564f00123300
---[ end trace 9564a1ccad733a91 ]---

Signature · 2 lines

You are receiving this mail because:
You are watching the assignee of the bug.

[email protected]

author

2020-01-17 22:43 UTC #15 lore raw

[Bug 206215] QEMU guest crash due to random 'general protection fault' since kernel 5.2.5 on i7-3517UE

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=206215

--- Comment #9 from [email protected] ---
Sean,

for the record ...

I did:

git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
@f5ae2ea6347a308cfe91f53b53682ce635497d0d

git revert e751732486eb3f159089a64d1901992b1357e7cc

Then I built and installed kernel:
5.5.0-rc6-revert-e751732486eb3f159089a64d1901992b1357e7cc+ #1 SMP Thu Jan 16 13:02:23 EST 2020 x86_64 x86_64 x86_64 GNU/Linux

Guest is stable; no more "general protection fault".

Here is the stack trace with your patch '0001-thread_info-Add-a-debug-hook-to-detect-FPU-changes-w.patch'

[  122.323347] ------------[ cut here ]------------
[  122.323355] WARNING: CPU: 1 PID: 1132 at include/linux/thread_info.h:55
kernel_fpu_begin+0x6b/0xc0
[  122.323356] Modules linked in: vhost_net vhost tap tun xfrm4_tunnel tunnel4

ipcomp xfrm_ipcomp esp4 ah4 af_key ebtable_filter ebtables ip6table_filter ip6_tables bridge stp llc rfkill xt_TCPMSS xt_tcpmss xt_nat iptable_nat nf_nat xt_DSCP iptable_mangle iptable_raw iptable_security nf_log_ipv4 nf_log_common xt_policy xt_LOG xt_multiport ipt_REJECT nf_reject_ipv4 xt_state xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_filter intel_rapl_msr intel_rapl_common snd_hda_codec_hdmi x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_realtek snd_hda_codec_generic coretemp ledtrig_audio kvm_intel snd_hda_intel sunrpc snd_intel_dspcfg kvm snd_hda_codec snd_hda_core irqbypass mei_wdt mei_hdcp snd_hwdep vfat crct10dif_pclmul snd_seq fat crc32_pclmul snd_seq_device ghash_clmulni_intel iTCO_wdt snd_pcm iTCO_vendor_support intel_cstate snd_timer intel_uncore snd pcspkr intel_rapl_perf mei_me soundcore i2c_i801 lpc_ich mei ata_generic pata_acpi tcp_bbr sch_fq ip_tables xfs libcrc32c i915

[  122.323384]  i2c_algo_bit drm_kms_helper crc32c_intel e1000e drm r8169
serio_raw video
[  122.323389] CPU: 1 PID: 1132 Comm: CPU 2/KVM Not tainted
5.5.0-rc6-thread_info-Add-a-debug-hook-to-detect-FPU-changes-w+ #1
[  122.323390] Hardware name: CompuLab 0000000-00000/Intense-PC, BIOS
IPC_2.2.400.5 X64 03/15/2018
[  122.323392] RIP: 0010:kernel_fpu_begin+0x6b/0xc0
[  122.323394] Code: f6 40 26 20 75 08 48 8b 10 80 e6 40 74 16 65 48 c7 05 d5
2b fe 48 00 00 00 00 c3 65 8a 05 c5 2b fe 48 eb c4 80 78 0c 00 74 02 <0f> 0b 48
83 c0 01 f0 80 08 40 65 48 8b 0c 25 c0 8b 01 00 0f 1f 44
[  122.323395] RSP: 0018:ffffa69b80108308 EFLAGS: 00010202
[  122.323396] RAX: ffff8992513ecd00 RBX: 0000000000000088 RCX:
ffffdd96904487c0
[  122.323397] RDX: 0000000000000000 RSI: ffff89925fa99b00 RDI:
ffff89925fa99b00
[  122.323397] RBP: ffffa69b801085b0 R08: ffffa69b801085c0 R09:
ffffa69b80108370
[  122.323398] R10: 0000000000000000 R11: 0000000000000bbe R12:
0000000000000bce
[  122.323399] R13: ffffa69b80108370 R14: 0000000000000000 R15:
ffff89925121fbbe
[  122.323400] FS:  00007ff235b52700(0000) GS:ffff89927e240000(0000)
knlGS:0000000000000000
[  122.323401] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  122.323401] CR2: 00000000ffffffff CR3: 000000040b55e002 CR4:
00000000001626e0
[  122.323402] Call Trace:
[  122.323404]  <IRQ>
[  122.323409]  gcmaes_crypt_by_sg.constprop.0+0x276/0x6c0
[  122.323415]  ? skb_clone_tx_timestamp+0x3c/0xa0
[  122.323419]  ? sch_direct_xmit+0x8b/0x310
[  122.323423]  ? esp4_err+0x120/0x120 [esp4]
[  122.323425]  ? helper_rfc4106_encrypt+0x7c/0xa0
[  122.323428]  ? crypto_aead_encrypt+0x3c/0x60
[  122.323429]  ? crypto_aead_encrypt+0x3c/0x60
[  122.323431]  ? seqiv_aead_encrypt+0x13a/0x1d0
[  122.323434]  ? fib4_rule_action+0x61/0x70
[  122.323436]  ? fib4_rule_action+0x70/0x70
[  122.323438]  ? fib_rules_lookup+0x143/0x1a0
[  122.323440]  ? __fib_lookup+0x6b/0xb0
[  122.323442]  ? ip_route_output_key_hash_rcu+0x562/0x890
[  122.323444]  ? ip_route_output_key_hash+0x5e/0x80
[  122.323445]  ? __xfrm4_dst_lookup.isra.0+0x88/0x90
[  122.323446]  ? xfrm4_dst_lookup+0x2f/0x50
[  122.323447]  ? rt_add_uncached_list+0x4b/0x80
[  122.323449]  ? xfrm4_fill_dst+0xae/0xf0
[  122.323450]  ? crypto_aead_encrypt+0x3c/0x60
[  122.323452]  ? esp_output_tail+0x1e5/0x580 [esp4]
[  122.323454]  ? esp_output+0x116/0x190 [esp4]
[  122.323457]  ? xfrm_output_resume+0x431/0x4f0
[  122.323464]  ? nf_confirm+0xcb/0xf0 [nf_conntrack]
[  122.323466]  ? __xfrm4_output+0x3f/0x70
[  122.323467]  ? xfrm4_output+0x3b/0xd0
[  122.323468]  ? xfrm4_udp_encap_rcv+0x190/0x190
[  122.323470]  ? ip_forward+0x36c/0x470
[  122.323472]  ? ip_defrag.cold+0x37/0x37
[  122.323473]  ? ip_rcv+0xbc/0xd0
[  122.323475]  ? ip_rcv_finish_core.isra.0+0x410/0x410
[  122.323476]  ? __netif_receive_skb_one_core+0x80/0x90
[  122.323478]  ? netif_receive_skb_internal+0x41/0xb0
[  122.323479]  ? nf_hook_slow+0x40/0xb0
[  122.323480]  ? netif_receive_skb+0x18/0xb0
[  122.323486]  ? br_pass_frame_up+0x133/0x150 [bridge]
[  122.323491]  ? br_port_flags_change+0x40/0x40 [bridge]
[  122.323495]  ? br_handle_frame_finish+0x16f/0x430 [bridge]
[  122.323497]  ? enqueue_entity+0x10e/0x650
[  122.323501]  ? br_handle_frame_finish+0x430/0x430 [bridge]
[  122.323505]  ? br_handle_frame+0x247/0x370 [bridge]
[  122.323506]  ? enqueue_task_fair+0x8c/0x4e0
[  122.323508]  ? update_group_capacity+0x25/0x1e0
[  122.323512]  ? br_handle_frame_finish+0x430/0x430 [bridge]
[  122.323513]  ? __netif_receive_skb_core+0x2db/0xf70
[  122.323515]  ? __netif_receive_skb_list_core+0x138/0x2e0
[  122.323517]  ? netif_receive_skb_list_internal+0x1cc/0x300
[  122.323518]  ? enqueue_task_fair+0x8c/0x4e0
[  122.323520]  ? kmem_cache_alloc+0x165/0x220
[  122.323521]  ? gro_normal_list.part.0+0x19/0x40
[  122.323522]  ? napi_complete_done+0x92/0x130
[  122.323526]  ? rtl8169_poll+0x5a9/0x640 [r8169]
[  122.323527]  ? net_rx_action+0x148/0x3c0
[  122.323530]  ? rtl8169_interrupt+0xfd/0x1e0 [r8169]
[  122.323532]  ? __do_softirq+0xee/0x2ff
[  122.323535]  ? irq_exit+0xe9/0xf0
[  122.323536]  ? do_IRQ+0x55/0xe0
[  122.323538]  ? common_interrupt+0xf/0xf
[  122.323539]  </IRQ>
[  122.323541]  ? irq_entries_start+0x30/0x660
[  122.323546]  ? handle_external_interrupt_irqoff+0x7a/0x100 [kvm_intel]
[  122.323568]  ? kvm_arch_vcpu_ioctl_run+0x995/0x1a60 [kvm]
[  122.323570]  ? futex_wake+0x90/0x170
[  122.323581]  ? kvm_vcpu_ioctl+0x218/0x5c0 [kvm]
[  122.323584]  ? __seccomp_filter+0x7b/0x670
[  122.323585]  ? signal_setup_done+0x82/0xa0
[  122.323586]  ? __fpu__restore_sig+0x436/0x500
[  122.323588]  ? do_vfs_ioctl+0x461/0x6d0
[  122.323590]  ? ksys_ioctl+0x5e/0x90
[  122.323591]  ? __x64_sys_ioctl+0x16/0x20
[  122.323593]  ? do_syscall_64+0x5b/0x1c0
[  122.323595]  ? entry_SYSCALL_64_after_hwframe+0x44/0xa9
[  122.323597] ---[ end trace fffe8684d1d1c2f4 ]---
[  122.323626] ------------[ cut here ]------------
[  122.323644] WARNING: CPU: 1 PID: 1132 at arch/x86/kvm/x86.c:8206
kvm_arch_vcpu_ioctl_run+0x163e/0x1a60 [kvm]
[  122.323644] Modules linked in: vhost_net vhost tap tun xfrm4_tunnel tunnel4

[  122.323664]  i2c_algo_bit drm_kms_helper crc32c_intel e1000e drm r8169
serio_raw video
[  122.323669] CPU: 1 PID: 1132 Comm: CPU 2/KVM Tainted: G        W        
5.5.0-rc6-thread_info-Add-a-debug-hook-to-detect-FPU-changes-w+ #1
[  122.323669] Hardware name: CompuLab 0000000-00000/Intense-PC, BIOS
IPC_2.2.400.5 X64 03/15/2018
[  122.323683] RIP: 0010:kvm_arch_vcpu_ioctl_run+0x163e/0x1a60 [kvm]
[  122.323685] Code: a8 f3 fe ff 41 f6 44 24 42 02 75 08 4c 89 e7 e8 e8 f3 fe
ff 4c 89 e7 e8 20 7d fe ff 41 83 a4 24 60 26 00 00 fb e9 cb f2 ff ff <0f> 0b e9
8e f2 ff ff 31 db bf 07 00 00 00 48 89 de e8 cc 11 65 f6
[  122.323685] RSP: 0018:ffffa69b80fcbd40 EFLAGS: 00010002
[  122.323686] RAX: 0000000080004b00 RBX: 0000000000000000 RCX:
ffff8992513ecd00
[  122.323687] RDX: 0000000000000000 RSI: 0000000000000001 RDI:
0000000000000000
[  122.323688] RBP: ffffa69b80fcbde0 R08: 0000000000000001 R09:
0000000000000000
[  122.323688] R10: 0000000000000000 R11: 0000000000000000 R12:
ffff899250a28000
[  122.323689] R13: 0000000000000000 R14: ffff899250a28038 R15:
ffffa69b81042320
[  122.323690] FS:  00007ff235b52700(0000) GS:ffff89927e240000(0000)
knlGS:0000000000000000
[  122.323691] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  122.323691] CR2: 00000000ffffffff CR3: 000000040b55e002 CR4:
00000000001626e0
[  122.323692] Call Trace:
[  122.323695]  ? futex_wake+0x90/0x170
[  122.323707]  kvm_vcpu_ioctl+0x218/0x5c0 [kvm]
[  122.323709]  ? __seccomp_filter+0x7b/0x670
[  122.323710]  ? signal_setup_done+0x82/0xa0
[  122.323711]  ? __fpu__restore_sig+0x436/0x500
[  122.323713]  do_vfs_ioctl+0x461/0x6d0
[  122.323715]  ksys_ioctl+0x5e/0x90
[  122.323716]  __x64_sys_ioctl+0x16/0x20
[  122.323718]  do_syscall_64+0x5b/0x1c0
[  122.323720]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[  122.323722] RIP: 0033:0x7ff2399bf34b
[  122.323723] Code: 0f 1e fa 48 8b 05 3d 9b 0c 00 64 c7 00 26 00 00 00 48 c7
c0 ff ff ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01
f0 ff ff 73 01 c3 48 8b 0d 0d 9b 0c 00 f7 d8 64 89 01 48
[  122.323724] RSP: 002b:00007ff235b51698 EFLAGS: 00000246 ORIG_RAX:
0000000000000010
[  122.323725] RAX: ffffffffffffffda RBX: 000056321fab4f50 RCX:
00007ff2399bf34b
[  122.323726] RDX: 0000000000000000 RSI: 000000000000ae80 RDI:
000000000000001b
[  122.323726] RBP: 00007ff23534f000 R08: 000056321d5ac390 R09:
000056321da50d40
[  122.323727] R10: 000056321f960760 R11: 0000000000000246 R12:
000056321fad7770
[  122.323727] R13: 000056321fab4f50 R14: 00007ffdb09a9730 R15:
000056321da2de80
[  122.323729] ---[ end trace fffe8684d1d1c2f5 ]---

Signature · 2 lines

You are receiving this mail because:
You are watching the assignee of the bug.