Skip to contentSkip to conversation

vmwrite error: reg 401e value a9 (err 1)

4.1.y Greg Edwards 3 participants 4 messages Nov 9, 2016 – Nov 17, 2016 KVM

lore
Share & tools
lore

Greg Edwards

author
#0 lore raw

On current 4.1.y stable kernel (4.1.35) on a Broadwell-EP system, I see the following when shutting down a multiple vcpu VM:

[  758.387722] vmwrite error: reg 401e value a9 (err 1)
[  758.392860] CPU: 33 PID: 14969 Comm: qemu-system-x86 Not tainted 4.1.35 #1
[  758.399897] Hardware name: DDN 14000x/14000, BIOS 0229 09/23/2016
[  758.406156]  0000000000000286 0000000028b15def ffff88202f3fbb38 ffffffff8159de63
[  758.413942]  ffff88402a938000 0000000000000001 ffff88202f3fbb48 ffffffffa060fa1c
[  758.421736]  ffff88202f3fbb58 ffffffffa060fa49 ffff88202f3fbb78 ffffffffa0618fab
[  758.429534] Call Trace:
[  758.432147]  [<ffffffff8159de63>] dump_stack+0x4d/0x63
[  758.437449]  [<ffffffffa060fa1c>] vmwrite_error+0x2c/0x30 [kvm_intel]
[  758.444059]  [<ffffffffa060fa49>] vmcs_writel+0x29/0x30 [kvm_intel]
[  758.450493]  [<ffffffffa0618fab>] vmx_free_vcpu+0xdb/0xf0 [kvm_intel]
[  758.457111]  [<ffffffffa059ddb8>] kvm_arch_vcpu_free+0x48/0x50 [kvm]
[  758.463637]  [<ffffffffa059eb8a>] kvm_arch_destroy_vm+0x10a/0x200 [kvm]
[  758.470418]  [<ffffffff810caff8>] ? synchronize_srcu+0x28/0x30
[  758.476419]  [<ffffffffa05850c5>] kvm_put_kvm+0x105/0x220 [kvm]
[  758.482505]  [<ffffffffa0585218>] kvm_vcpu_release+0x18/0x20 [kvm]
[  758.488853]  [<ffffffff811a143b>] __fput+0xcb/0x1d0
[  758.493899]  [<ffffffff811a158e>] ____fput+0xe/0x10
[  758.498939]  [<ffffffff81098ec4>] task_work_run+0xd4/0xf0
[  758.504497]  [<ffffffff8107d811>] do_exit+0x2a1/0xb40
[  758.509708]  [<ffffffff8107eef7>] do_group_exit+0x47/0xc0
[  758.515269]  [<ffffffff8108adc3>] get_signal+0x1f3/0x6c0
[  758.520743]  [<ffffffff81003517>] do_signal+0x37/0x800
[  758.526042]  [<ffffffff810e9c35>] ? SyS_futex+0x85/0x1a0
[  758.531513]  [<ffffffff81003d50>] do_notify_resume+0x70/0x80
[  758.537334]  [<ffffffff815a4882>] int_signal+0x12/0x17

This started with the inclusion of 6c2ca21665b99ce2f76389c353b985d8195387cc ("KVM: nVMX: Fix memory corruption when using VMCS shadowing") in 4.1.31.

The error is coming out of vmx_disable_pml() when freeing the 2nd and subsequent vcpus, as SECONDARY_EXEC_ENABLE_PML was already cleared from the SECONDARY_VM_EXEC_CONTROL when the first vcpu was freed.

Additionally pulling back a3eaa8649e4c6a6afdafaa04b9114fb230617bb1 ("KVM: VMX:
Fix commit which broke PML") from 4.4 resolves it for me, as it fixes the above condition.

Is this the correct fix for 4.1.y?

Greg

#1 lore raw

Hi Greg,

Thanks for reporting this issue.

I don't have 4.1.y source code tree at hand but after taking a glance looks the commit a3eaa8649e4c6a6afdafaa04b9114fb230617bb1 ("KVM: VMX:
Fix commit which broke PML") fixes this by removing vmwrite to SECONDARY_VM_EXEC_CONTROL in vmx_disable_pml, so yes I think this commit can fix this issue.

But I think you probably need another commit to fix potential vmwrite error when creating vcpu: 4e59516a12a6ef6dcb660cb3a3f70c64bd60cfec (kvm: vmx: ensure VMCS is current while enabling PML). Peter found and fixed this issue, so I also added him to cc-list.

Paolo/Radim, please comment if I made mistake here.

Thanks,
-Kai

Quoted 45 lines · Greg Edwards

On current 4.1.y stable kernel (4.1.35) on a Broadwell-EP system, I see the following when shutting down a multiple vcpu VM:

[  758.387722] vmwrite error: reg 401e value a9 (err 1)
[  758.392860] CPU: 33 PID: 14969 Comm: qemu-system-x86 Not tainted 4.1.35 #1
[  758.399897] Hardware name: DDN 14000x/14000, BIOS 0229 09/23/2016
[  758.406156]  0000000000000286 0000000028b15def ffff88202f3fbb38 ffffffff8159de63
[  758.413942]  ffff88402a938000 0000000000000001 ffff88202f3fbb48 ffffffffa060fa1c
[  758.421736]  ffff88202f3fbb58 ffffffffa060fa49 ffff88202f3fbb78 ffffffffa0618fab
[  758.429534] Call Trace:
[  758.432147]  [<ffffffff8159de63>] dump_stack+0x4d/0x63
[  758.437449]  [<ffffffffa060fa1c>] vmwrite_error+0x2c/0x30 [kvm_intel]
[  758.444059]  [<ffffffffa060fa49>] vmcs_writel+0x29/0x30 [kvm_intel]
[  758.450493]  [<ffffffffa0618fab>] vmx_free_vcpu+0xdb/0xf0 [kvm_intel]
[  758.457111]  [<ffffffffa059ddb8>] kvm_arch_vcpu_free+0x48/0x50 [kvm]
[  758.463637]  [<ffffffffa059eb8a>] kvm_arch_destroy_vm+0x10a/0x200 [kvm]
[  758.470418]  [<ffffffff810caff8>] ? synchronize_srcu+0x28/0x30
[  758.476419]  [<ffffffffa05850c5>] kvm_put_kvm+0x105/0x220 [kvm]
[  758.482505]  [<ffffffffa0585218>] kvm_vcpu_release+0x18/0x20 [kvm]
[  758.488853]  [<ffffffff811a143b>] __fput+0xcb/0x1d0
[  758.493899]  [<ffffffff811a158e>] ____fput+0xe/0x10
[  758.498939]  [<ffffffff81098ec4>] task_work_run+0xd4/0xf0
[  758.504497]  [<ffffffff8107d811>] do_exit+0x2a1/0xb40
[  758.509708]  [<ffffffff8107eef7>] do_group_exit+0x47/0xc0
[  758.515269]  [<ffffffff8108adc3>] get_signal+0x1f3/0x6c0
[  758.520743]  [<ffffffff81003517>] do_signal+0x37/0x800
[  758.526042]  [<ffffffff810e9c35>] ? SyS_futex+0x85/0x1a0
[  758.531513]  [<ffffffff81003d50>] do_notify_resume+0x70/0x80
[  758.537334]  [<ffffffff815a4882>] int_signal+0x12/0x17

This started with the inclusion of 6c2ca21665b99ce2f76389c353b985d8195387cc ("KVM: nVMX: Fix memory corruption when using VMCS shadowing") in 4.1.31.

The error is coming out of vmx_disable_pml() when freeing the 2nd and subsequent vcpus, as SECONDARY_EXEC_ENABLE_PML was already cleared from the SECONDARY_VM_EXEC_CONTROL when the first vcpu was freed.

Additionally pulling back a3eaa8649e4c6a6afdafaa04b9114fb230617bb1 ("KVM: VMX:
Fix commit which broke PML") from 4.4 resolves it for me, as it fixes the above condition.

Is this the correct fix for 4.1.y?

Greg

maintainer
#2 lore raw
Quoted 10 lines · Kai

Hi Greg,

Thanks for reporting this issue.

I don't have 4.1.y source code tree at hand but after taking a glance looks the commit a3eaa8649e4c6a6afdafaa04b9114fb230617bb1 ("KVM: VMX:
Fix commit which broke PML") fixes this by removing vmwrite to SECONDARY_VM_EXEC_CONTROL in vmx_disable_pml, so yes I think this commit can fix this issue.

Quoted 6 lines

But I think you probably need another commit to fix potential vmwrite error when creating vcpu: 4e59516a12a6ef6dcb660cb3a3f70c64bd60cfec (kvm:
vmx: ensure VMCS is current while enabling PML). Peter found and fixed this issue, so I also added him to cc-list.

Paolo/Radim, please comment if I made mistake here.

Yes, we should backport both of them. Greg, can you test it? Then I'll send the two patches to linux-stable.

Paolo

Quoted 58 lines

Thanks,
-Kai

Quoted 54 lines · Greg Edwards

On current 4.1.y stable kernel (4.1.35) on a Broadwell-EP system, I see the
following when shutting down a multiple vcpu VM:

[  758.387722] vmwrite error: reg 401e value a9 (err 1)
[  758.392860] CPU: 33 PID: 14969 Comm: qemu-system-x86 Not tainted
4.1.35 #1
[  758.399897] Hardware name: DDN 14000x/14000, BIOS 0229 09/23/2016
[  758.406156]  0000000000000286 0000000028b15def ffff88202f3fbb38
ffffffff8159de63
[  758.413942]  ffff88402a938000 0000000000000001 ffff88202f3fbb48
ffffffffa060fa1c
[  758.421736]  ffff88202f3fbb58 ffffffffa060fa49 ffff88202f3fbb78
ffffffffa0618fab
[  758.429534] Call Trace:
[  758.432147]  [<ffffffff8159de63>] dump_stack+0x4d/0x63
[  758.437449]  [<ffffffffa060fa1c>] vmwrite_error+0x2c/0x30 [kvm_intel]
[  758.444059]  [<ffffffffa060fa49>] vmcs_writel+0x29/0x30 [kvm_intel]
[  758.450493]  [<ffffffffa0618fab>] vmx_free_vcpu+0xdb/0xf0 [kvm_intel]
[  758.457111]  [<ffffffffa059ddb8>] kvm_arch_vcpu_free+0x48/0x50 [kvm]
[  758.463637]  [<ffffffffa059eb8a>] kvm_arch_destroy_vm+0x10a/0x200
[kvm]
[  758.470418]  [<ffffffff810caff8>] ? synchronize_srcu+0x28/0x30
[  758.476419]  [<ffffffffa05850c5>] kvm_put_kvm+0x105/0x220 [kvm]
[  758.482505]  [<ffffffffa0585218>] kvm_vcpu_release+0x18/0x20 [kvm]
[  758.488853]  [<ffffffff811a143b>] __fput+0xcb/0x1d0
[  758.493899]  [<ffffffff811a158e>] ____fput+0xe/0x10
[  758.498939]  [<ffffffff81098ec4>] task_work_run+0xd4/0xf0
[  758.504497]  [<ffffffff8107d811>] do_exit+0x2a1/0xb40
[  758.509708]  [<ffffffff8107eef7>] do_group_exit+0x47/0xc0
[  758.515269]  [<ffffffff8108adc3>] get_signal+0x1f3/0x6c0
[  758.520743]  [<ffffffff81003517>] do_signal+0x37/0x800
[  758.526042]  [<ffffffff810e9c35>] ? SyS_futex+0x85/0x1a0
[  758.531513]  [<ffffffff81003d50>] do_notify_resume+0x70/0x80
[  758.537334]  [<ffffffff815a4882>] int_signal+0x12/0x17

This started with the inclusion of 6c2ca21665b99ce2f76389c353b985d8195387cc
("KVM: nVMX: Fix memory corruption when using VMCS shadowing") in 4.1.31.

The error is coming out of vmx_disable_pml() when freeing the 2nd and subsequent vcpus, as SECONDARY_EXEC_ENABLE_PML was already cleared from the
SECONDARY_VM_EXEC_CONTROL when the first vcpu was freed.

Additionally pulling back a3eaa8649e4c6a6afdafaa04b9114fb230617bb1 ("KVM: VMX:
Fix commit which broke PML") from 4.4 resolves it for me, as it fixes the above condition.

Is this the correct fix for 4.1.y?

Greg

author
#3 lore raw
Quoted 16 lines · Paolo Bonzini
Quoted 15 lines · Kai

Hi Greg,

Thanks for reporting this issue.

I don't have 4.1.y source code tree at hand but after taking a glance looks the commit a3eaa8649e4c6a6afdafaa04b9114fb230617bb1 ("KVM: VMX:
Fix commit which broke PML") fixes this by removing vmwrite to SECONDARY_VM_EXEC_CONTROL in vmx_disable_pml, so yes I think this commit can fix this issue.

But I think you probably need another commit to fix potential vmwrite error when creating vcpu: 4e59516a12a6ef6dcb660cb3a3f70c64bd60cfec (kvm:
vmx: ensure VMCS is current while enabling PML). Peter found and fixed this issue, so I also added him to cc-list.

Quoted 4 lines
Quoted 1 line

Paolo/Radim, please comment if I made mistake here.

Yes, we should backport both of them. Greg, can you test it? Then I'll send the two patches to linux-stable.

Yes, we've been running with both patches internally without problems for a week or so.

Greg