Skip to contentSkip to conversation

4.9.36-stable review

PATCH In review Greg Kroah-Hartman 4 participants 169 messages Jul 3, 2017 – Jul 13, 2017 1 message not in the archive Page 2 of 2

Subsystems (3)
Stable branch VFS MIPS
lore
Share & tools
lore
author maintainer
#162 lore raw

PATCH 169/172 KVM: x86: fix emulation of RSM and IRET instructions

In reply to Greg Kroah-Hartman

4.9-stable review patch. If anyone has any objections, please let me know.

------------------

From: Ladi Prosek <[email protected]>

commit 6ed071f051e12cf7baa1b69d3becb8f232fdfb7b upstream.

On AMD, the effect of set_nmi_mask called by emulate_iret_real and em_rsm on hflags is reverted later on in x86_emulate_instruction where hflags are overwritten with ctxt->emul_flags (the kvm_set_hflags call). This manifests as a hang when rebooting Windows VMs with QEMU, OVMF, and >1 vcpu.

Instead of trying to merge ctxt->emul_flags into vcpu->arch.hflags after an instruction is emulated, this commit deletes emul_flags altogether and makes the emulator access vcpu->arch.hflags using two new accessors. This way all changes, on the emulator side as well as in functions called from the emulator and accessing vcpu state with emul_to_vcpu, are preserved.

More details on the bug and its manifestation with Windows and OVMF:

It's a KVM bug in the interaction between SMI/SMM and NMI, specific to AMD. I believe that the SMM part explains why we started seeing this only with OVMF.

KVM masks and unmasks NMI when entering and leaving SMM. When KVM emulates the RSM instruction in em_rsm, the set_nmi_mask call doesn't stick because later on in x86_emulate_instruction we overwrite arch.hflags with ctxt->emul_flags, effectively reverting the effect of the set_nmi_mask call. The AMD-specific hflag of interest here is HF_NMI_MASK.

When rebooting the system, Windows sends an NMI IPI to all but the current cpu to shut them down. Only after all of them are parked in HLT will the initiating cpu finish the restart. If NMI is masked, other cpus never get the memo and the initiating cpu spins forever, waiting for hal!HalpInterruptProcessorsStarted to drop. That's the symptom we observe.

Fixes a584539b24b8 ("KVM: x86: pass the whole hflags field to emulator and back")
Signed-off-by Ladi Prosek <[email protected]>
Signed-off-by Paolo Bonzini <[email protected]>
Signed-off-by Greg Kroah-Hartman <[email protected]>
---
 arch/x86/include/asm/kvm_emulate.h |    4 +++-
 arch/x86/kvm/emulate.c             |   16 +++++++++-------
 arch/x86/kvm/x86.c                 |   15 ++++++++++++---
 3 files changed, 24 insertions(+), 11 deletions(-)
arch/x86/include/asm/kvm_emulate.h +3 -1
@@ -221,6 +221,9 @@ struct x86_emulate_ops {
 	void (*get_cpuid)(struct x86_emulate_ctxt *ctxt,
 			  u32 *eax, u32 *ebx, u32 *ecx, u32 *edx);
 	void (*set_nmi_mask)(struct x86_emulate_ctxt *ctxt, bool masked);
+
+	unsigned (*get_hflags)(struct x86_emulate_ctxt *ctxt);
+	void (*set_hflags)(struct x86_emulate_ctxt *ctxt, unsigned hflags);
 };
 
 typedef u32 __attribute__((vector_size(16))) sse128_t;

@@ -290,7 +293,6 @@ struct x86_emulate_ctxt {
 
 	/* interruptibility state, as a result of execution of STI or MOV SS */
 	int interruptibility;
-	int emul_flags;
 
 	bool perm_ok; /* do not check permissions if true */
 	bool ud;	/* inject an #UD if host doesn't support insn */
arch/x86/kvm/emulate.c +9 -7
@@ -2543,7 +2543,7 @@ static int em_rsm(struct x86_emulate_ctx
 	u64 smbase;
 	int ret;
 
-	if ((ctxt->emul_flags & X86EMUL_SMM_MASK) == 0)
+	if ((ctxt->ops->get_hflags(ctxt) & X86EMUL_SMM_MASK) == 0)
 		return emulate_ud(ctxt);
 
 	/*

@@ -2592,11 +2592,11 @@ static int em_rsm(struct x86_emulate_ctx
 		return X86EMUL_UNHANDLEABLE;
 	}
 
-	if ((ctxt->emul_flags & X86EMUL_SMM_INSIDE_NMI_MASK) == 0)
+	if ((ctxt->ops->get_hflags(ctxt) & X86EMUL_SMM_INSIDE_NMI_MASK) == 0)
 		ctxt->ops->set_nmi_mask(ctxt, false);
 
-	ctxt->emul_flags &= ~X86EMUL_SMM_INSIDE_NMI_MASK;
-	ctxt->emul_flags &= ~X86EMUL_SMM_MASK;
+	ctxt->ops->set_hflags(ctxt, ctxt->ops->get_hflags(ctxt) &
+		~(X86EMUL_SMM_INSIDE_NMI_MASK | X86EMUL_SMM_MASK));
 	return X86EMUL_CONTINUE;
 }
 

@@ -5312,6 +5312,7 @@ int x86_emulate_insn(struct x86_emulate_
 	const struct x86_emulate_ops *ops = ctxt->ops;
 	int rc = X86EMUL_CONTINUE;
 	int saved_dst_type = ctxt->dst.type;
+	unsigned emul_flags;
 
 	ctxt->mem_read.pos = 0;
 

@@ -5326,6 +5327,7 @@ int x86_emulate_insn(struct x86_emulate_
 		goto done;
 	}
 
+	emul_flags = ctxt->ops->get_hflags(ctxt);
 	if (unlikely(ctxt->d &
 		     (No64|Undefined|Sse|Mmx|Intercept|CheckPerm|Priv|Prot|String))) {
 		if ((ctxt->mode == X86EMUL_MODE_PROT64 && (ctxt->d & No64)) ||

@@ -5359,7 +5361,7 @@ int x86_emulate_insn(struct x86_emulate_
 				fetch_possible_mmx_operand(ctxt, &ctxt->dst);
 		}
 
-		if (unlikely(ctxt->emul_flags & X86EMUL_GUEST_MASK) && ctxt->intercept) {
+		if (unlikely(emul_flags & X86EMUL_GUEST_MASK) && ctxt->intercept) {
 			rc = emulator_check_intercept(ctxt, ctxt->intercept,
 						      X86_ICPT_PRE_EXCEPT);
 			if (rc != X86EMUL_CONTINUE)

@@ -5388,7 +5390,7 @@ int x86_emulate_insn(struct x86_emulate_
 				goto done;
 		}
 
-		if (unlikely(ctxt->emul_flags & X86EMUL_GUEST_MASK) && (ctxt->d & Intercept)) {
+		if (unlikely(emul_flags & X86EMUL_GUEST_MASK) && (ctxt->d & Intercept)) {
 			rc = emulator_check_intercept(ctxt, ctxt->intercept,
 						      X86_ICPT_POST_EXCEPT);
 			if (rc != X86EMUL_CONTINUE)

@@ -5442,7 +5444,7 @@ int x86_emulate_insn(struct x86_emulate_
 
 special_insn:
 
-	if (unlikely(ctxt->emul_flags & X86EMUL_GUEST_MASK) && (ctxt->d & Intercept)) {
+	if (unlikely(emul_flags & X86EMUL_GUEST_MASK) && (ctxt->d & Intercept)) {
 		rc = emulator_check_intercept(ctxt, ctxt->intercept,
 					      X86_ICPT_POST_MEMACCESS);
 		if (rc != X86EMUL_CONTINUE)
arch/x86/kvm/x86.c +12 -3
@@ -5154,6 +5154,16 @@ static void emulator_set_nmi_mask(struct
 	kvm_x86_ops->set_nmi_mask(emul_to_vcpu(ctxt), masked);
 }
 
+static unsigned emulator_get_hflags(struct x86_emulate_ctxt *ctxt)
+{
+	return emul_to_vcpu(ctxt)->arch.hflags;
+}
+
+static void emulator_set_hflags(struct x86_emulate_ctxt *ctxt, unsigned emul_flags)
+{
+	kvm_set_hflags(emul_to_vcpu(ctxt), emul_flags);
+}
+
 static const struct x86_emulate_ops emulate_ops = {
 	.read_gpr            = emulator_read_gpr,
 	.write_gpr           = emulator_write_gpr,

@@ -5193,6 +5203,8 @@ static const struct x86_emulate_ops emul
 	.intercept           = emulator_intercept,
 	.get_cpuid           = emulator_get_cpuid,
 	.set_nmi_mask        = emulator_set_nmi_mask,
+	.get_hflags          = emulator_get_hflags,
+	.set_hflags          = emulator_set_hflags,
 };
 
 static void toggle_interruptibility(struct kvm_vcpu *vcpu, u32 mask)

@@ -5245,7 +5257,6 @@ static void init_emulate_ctxt(struct kvm
 	BUILD_BUG_ON(HF_GUEST_MASK != X86EMUL_GUEST_MASK);
 	BUILD_BUG_ON(HF_SMM_MASK != X86EMUL_SMM_MASK);
 	BUILD_BUG_ON(HF_SMM_INSIDE_NMI_MASK != X86EMUL_SMM_INSIDE_NMI_MASK);
-	ctxt->emul_flags = vcpu->arch.hflags;
 
 	init_decode_cache(ctxt);
 	vcpu->arch.emulate_regs_need_sync_from_vcpu = false;

@@ -5636,8 +5647,6 @@ restart:
 		unsigned long rflags = kvm_x86_ops->get_rflags(vcpu);
 		toggle_interruptibility(vcpu, ctxt->interruptibility);
 		vcpu->arch.emulate_regs_need_sync_to_vcpu = false;
-		if (vcpu->arch.hflags != ctxt->emul_flags)
-			kvm_set_hflags(vcpu, ctxt->emul_flags);
 		kvm_rip_write(vcpu, ctxt->eip);
 		if (r == EMULATE_DONE)
 			kvm_vcpu_check_singlestep(vcpu, rflags, &r);
#163 lore raw
In reply to Greg Kroah-Hartman
Quoted 8 lines · Greg Kroah-Hartman

This is the start of the stable review cycle for the 4.9.36 release. There are 172 patches in this series, all will be posted as a response to this one. If anyone has any issues with these being applied, please let me know.

Responses should be made by Wed Jul 5 13:33:25 UTC 2017. Anything received after that time might be too late.

Build results:

	total: 145 pass: 145 fail: 0
Qemu test results:
	total: 122 pass: 122 fail: 0

Details are available at http://kerneltests.org/builders.

Guenter

author maintainer
#164 lore raw
Quoted 10 lines · Guenter Roeck
Quoted 8 lines · Greg Kroah-Hartman

This is the start of the stable review cycle for the 4.9.36 release. There are 172 patches in this series, all will be posted as a response to this one. If anyone has any issues with these being applied, please let me know.

Responses should be made by Wed Jul 5 13:33:25 UTC 2017. Anything received after that time might be too late.

Quoted 6 lines

Build results:

	total: 145 pass: 145 fail: 0
Qemu test results:
	total: 122 pass: 122 fail: 0

Details are available at http://kerneltests.org/builders.

Thanks for testing this, and the other trees, and letting me know.

greg k-h

A message that is not in the archive (sent off-list or not yet received): [email protected]

author maintainer
#166 lore raw
Quoted 16 lines · kernelci.org bot

stable-rc/linux-4.9.y boot: 130 boots: 5 failed, 112 passed with 13 offline (v4.9.35-173-g45949a8fd1df)

Full Boot Summary: https://kernelci.org/boot/all/job/stable-rc/branch/linux-4.9.y/kernel/v4.9.35-173-g45949a8fd1df/
Full Build Summary: https://kernelci.org/build/stable-rc/branch/linux-4.9.y/kernel/v4.9.35-173-g45949a8fd1df/

Tree: stable-rc
Branch: linux-4.9.y
Git Describe: v4.9.35-173-g45949a8fd1df
Git Commit: 45949a8fd1dfe62289359ae6e71bbb3fc45afeeb
Git URL: http://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git
Tested: 31 unique boards, 13 SoC families, 28 builds out of 203

Boot Regressions Detected:

arm:

Quoted 13 lines
    exynos_defconfig:
        exynos5800-peach-pi_rootfs:nfs:
            lab-collabora: new failure (last pass: v4.9.34-45-g92905e331aea)

    multi_v7_defconfig:
        imx6q-sabrelite_rootfs:nfs:
            lab-collabora: new failure (last pass: v4.9.34-44-g8041763f609c)
        rk3288-rock2-square_rootfs:nfs:
            lab-collabora: new failure (last pass: v4.9.34)

    mvebu_v5_defconfig:
        kirkwood-openblocks_a7_rootfs:nfs:
            lab-free-electrons: new failure (last pass: v4.9.34)

Any hint as to why these new failures are happening?

thanks,

greg k-h

#167 lore raw
Quoted 27 lines · Greg Kroah-Hartman
Quoted 26 lines · kernelci.org bot

stable-rc/linux-4.9.y boot: 130 boots: 5 failed, 112 passed with 13 offline (v4.9.35-173-g45949a8fd1df)

Full Boot Summary: https://kernelci.org/boot/all/job/stable-rc/branch/linux-4.9.y/kernel/v4.9.35-173-g45949a8fd1df/
Full Build Summary: https://kernelci.org/build/stable-rc/branch/linux-4.9.y/kernel/v4.9.35-173-g45949a8fd1df/

Tree: stable-rc
Branch: linux-4.9.y
Git Describe: v4.9.35-173-g45949a8fd1df
Git Commit: 45949a8fd1dfe62289359ae6e71bbb3fc45afeeb
Git URL: http://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git
Tested: 31 unique boards, 13 SoC families, 28 builds out of 203

Boot Regressions Detected:

arm:

    exynos_defconfig:
        exynos5800-peach-pi_rootfs:nfs:
            lab-collabora: new failure (last pass: v4.9.34-45-g92905e331aea)

    multi_v7_defconfig:
        imx6q-sabrelite_rootfs:nfs:
            lab-collabora: new failure (last pass: v4.9.34-44-g8041763f609c)
        rk3288-rock2-square_rootfs:nfs:
            lab-collabora: new failure (last pass: v4.9.34)
Quoted 6 lines
Quoted 3 lines

mvebu_v5_defconfig:
kirkwood-openblocks_a7_rootfs:nfs:
lab-free-electrons: new failure (last pass: v4.9.34)

Any hint as to why these new failures are happening?

It looks like the nfs rootfs ones are the only failures, while the corresponding on-disk rootfs ones booted ok - perhaps a momentary network outage in collabora lab? Some one from the lab might need to validate this though.

Quoted 3 lines

thanks,

greg k-h

Best,
Sumit.

#168 lore raw
In reply to Greg Kroah-Hartman

Hi Greg,

Quoted 5 lines · Greg Kroah-Hartman

This is the start of the stable review cycle for the 4.9.36 release. There are 172 patches in this series, all will be posted as a response to this one. If anyone has any issues with these being applied, please let me know.

Quoted 9 lines

Responses should be made by Wed Jul 5 13:33:25 UTC 2017. Anything received after that time might be too late.

The whole patch series can be found in one patch at:

        kernel.org/pub/linux/kernel/v4.x/stable-review/patch-4.9.36-rc1.gz

or in the git tree and branch at:
git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git linux-4.9.y
and the diffstat can be found below.

Built with arm64 defconfig, and boot-tested on hikey. No regressions noted.

Quoted 4 lines

thanks,

greg k-h

Best
Sumit.

author maintainer
#169 lore raw
Quoted 11 lines · Sumit Semwal

Hi Greg,

Quoted 8 lines · Greg Kroah-Hartman

This is the start of the stable review cycle for the 4.9.36 release. There are 172 patches in this series, all will be posted as a response to this one. If anyone has any issues with these being applied, please let me know.

Responses should be made by Wed Jul 5 13:33:25 UTC 2017. Anything received after that time might be too late.

Quoted 7 lines
Quoted 6 lines

The whole patch series can be found in one patch at:

        kernel.org/pub/linux/kernel/v4.x/stable-review/patch-4.9.36-rc1.gz

or in the git tree and branch at:
git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git linux-4.9.y
and the diffstat can be found below.

Built with arm64 defconfig, and boot-tested on hikey. No regressions noted.

Great, thanks for testing and letting me know.

greg k-h