Skip to contentSkip to conversation

AMX Support in KVM

PATCH Superseded Yang Zhong 8 participants 80 messages Dec 7, 2021 – Dec 22, 2021

Subsystems (5)
Documentation x86 KVM Kselftest +1 more subsystems

v1 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Yang Zhong

author
#0 lore raw

cover AMX Support in KVM

(send on behalf of Jing who is currently on leave)

This series brings AMX (Advanced Matrix eXtensions) virtualization support to KVM. The three preparation patches in fpu core from Thomas [1] are also included.

A large portion of the changes in this series is to deal with eXtended Feature Disable (XFD) which allows resizing of the fpstate buffer to support dynamically-enabled XSTATE features with large state component (e.g. 8K for AMX).

The support is based on several key changes (design discussions can be found in [2]):

- Guest permissions for dynamically-enabled XSAVE features

Native tasks have to request permission via prctl() before touching a dynamic-resized XSTATE compoenent. Introduce guest permissions for the similar purpose. Userspace VMM is expected to request guest permission only once when the first vCPU is created.

KVM checks guest permission in KVM_SET_CPUID2. Setting XFD in guest cpuid w/o proper permissions fails this operation.

- Extend fpstate reallocation mechanism to cover guest fpu

Unlike native tasks which have reallocation triggered from #NM handler, guest fpstate reallocation is requested by KVM when it detects the guest intention to use dynamically-enabled XSAVE features.

The reallocation request is handled when exiting to userspace VMM. This implies that KVM must break vcpu_run() loop and exit to userspace VMM instead of immediately resuming back to the guest when reallocation is required.

- Detect fpstate reallocation in the emulation code

Because guest #NM is not trapped in KVM (costly), the guest intention of using a dynamically-enabled XSAVE feature[i] can be indirectly represented by guest XCR0[i]=1 and XFD[i]=0. This requires the emulation logic of both WRMSR(IA32_XFD) and XSETBV to check reallocation requirement when one of the two conditions is changed.

- Disable WRMSR interception for IA32_XFD

IA32_XFD can be frequently updated by the guest, as it is part of the task state and swapped in context switch when prev and next have different XFD setting. Always intercepting WRMSR can easily cause non-negligible overhead.

Disable WRMSR interception for IA32_XFD after fpstate reallocation succeeds. After that point the guest direct writes IA32_XFD without causing VM-exits.

However MSR passthrough implies that guest_fpstate::xfd and per-cpu xfd cache might be out of sync with the current IA32_XFD value set by the guest. This suggests KVM needs to re-sync the software state with IA32_XFD before the vCPU thread might be preempted or interrupted.

- Save/restore guest XFD_ERR

When XFD causes an instruction to generate #NM, XFD_ERR contains information about which disabled state components are being accessed. The #NM handler is expected to check this information and then enable the state components by clearing IA32_XFD for the faulting task (if having permission).

#NM can be triggered in both host and guest. It'd be problematic if the XFD_ERR value generated in guest is consumed/clobbered by the host before the guest itself doing so. This may lead to non-XFD- related #NM treated as XFD #NM in host (due to guest XFD_ERR value), or XFD-related #NM treated as non-XFD #NM in guest (XFD_ERR cleared by the host #NM handler).

KVM needs to save the guest XFD_ERR value before this register might be accessed by the host and restore it before entering the guest.

One open remains in this area about when to start saving/restoring guest XFD_ERR. Several options are discussed in patch 15.

- Expose related cpuid bits to guest

The last step is to allow exposing XFD, AMX_TILE, AMX_INT8 and AMX_BF16 in guest cpuid. Adding those bits into kvm_cpu_caps finally activates all previous logics in this series

To verify AMX virtualization overhead on non-AMX usages, we run the Phoronix kernel build test in the guest w/ and w/o AMX in cpuid. The result shows no observable difference between two configurations.

Live migration support is still being worked on. Userspace VMM needs to use the new KVM_{G|S}SET_XSAVE2 ioctl in this series to migrate state for dynamically-enabled XSAVE features.

Thanks Thomas for the thoughts and patches on the KVM FPU and AMX support. Thanks Jun Nakajima for the design suggestions.

[1] git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git x86/fpu-kvm
[2] https://www.spinics.net/lists/kvm/msg259015.html

Thanks,
Yang

---
Jing Liu (13):
kvm: x86: Fix xstate_required_size() to follow XSTATE alignment rule
kvm: x86: Check guest xstate permissions when KVM_SET_CPUID2 x86/fpu: Move xfd initialization out of __fpstate_reset() to the
callers
kvm: x86: Propagate fpstate reallocation error to userspace x86/fpu: Move xfd_update_state() to xstate.c and export symbol
kvm: x86: Prepare reallocation check
kvm: x86: Emulate WRMSR of guest IA32_XFD
kvm: x86: Disable WRMSR interception for IA32_XFD on demand x86/fpu: Prepare for KVM XFD_ERR handling
kvm: x86: Introduce KVM_{G|S}ET_XSAVE2 ioctl
docs: virt: api.rst: Document the new KVM_{G, S}ET_XSAVE2 ioctls
kvm: x86: AMX XCR0 support for guest
kvm: x86: Add AMX CPUIDs support

Thomas Gleixner (4):
x86/fpu: Extend prctl() with guest permissions x86/fpu: Prepare KVM for dynamically enabled states x86/fpu: Add reallocation mechanims for KVM x86/fpu: Prepare KVM for bringing XFD state back in-sync

Yang Zhong (2):
kvm: x86: Check fpstate reallocation in XSETBV emulation
kvm: x86: Save and restore guest XFD_ERR properly

 Documentation/virt/kvm/api.rst     |  47 +++++++
 arch/x86/include/asm/cpufeatures.h |   2 +
 arch/x86/include/asm/fpu/api.h     |  12 ++
 arch/x86/include/asm/fpu/types.h   |  56 +++++++++
 arch/x86/include/asm/fpu/xstate.h  |   2 +
 arch/x86/include/asm/kvm-x86-ops.h |   1 +
 arch/x86/include/asm/kvm_host.h    |   2 +
 arch/x86/include/uapi/asm/kvm.h    |   6 +
 arch/x86/include/uapi/asm/prctl.h  |  26 ++--
 arch/x86/kernel/fpu/core.c         | 109 ++++++++++++++++-
 arch/x86/kernel/fpu/xstate.c       | 119 +++++++++++++++---
 arch/x86/kernel/fpu/xstate.h       |  29 +++--
 arch/x86/kernel/process.c          |   2 +
 arch/x86/kvm/cpuid.c               |  36 +++++-
 arch/x86/kvm/vmx/vmx.c             |  20 +++
 arch/x86/kvm/vmx/vmx.h             |   2 +-
 arch/x86/kvm/x86.c                 | 189 ++++++++++++++++++++++++++++-
 arch/x86/kvm/x86.h                 |   2 +
 include/uapi/linux/kvm.h           |   8 +-
 19 files changed, 607 insertions(+), 63 deletions(-)
author
#1 lore raw

PATCH 1/19 x86/fpu: Extend prctl() with guest permissions

From: Thomas Gleixner <[email protected]>

Add guest permission control for dynamic XSTATE components, including extension to prctl() with two new options (ARCH_GET_XCOMP_GUEST_PERM and ARCH_REQ_XCOMP_GUEST_PERM) and to struct fpu with a new member (guest_perm).

Userspace VMM has to request guest permissions before it exposes any XSAVE feature using dynamic XSTATE components. The permission can be set only once when the first vCPU is created. A new flag FPU_GUEST_PERM_LOCKED is introduced to lock the change for this purpose

Similar to native permissions this doesn't actually enable the permitted feature. KVM is expected to install a larger kernel buffer and enable the feature when detecting the intention from the guest.

---
(To Thomas) We change the definition of xstate_get_guest_group_perm() from xstate.h to api.h since this will be called by KVM.

 arch/x86/include/asm/fpu/api.h    |  2 ++
 arch/x86/include/asm/fpu/types.h  |  9 ++++++
 arch/x86/include/uapi/asm/prctl.h | 26 ++++++++--------
 arch/x86/kernel/fpu/core.c        |  3 ++
 arch/x86/kernel/fpu/xstate.c      | 50 +++++++++++++++++++++++--------
 arch/x86/kernel/fpu/xstate.h      | 13 ++++++--
 arch/x86/kernel/process.c         |  2 ++
 7 files changed, 78 insertions(+), 27 deletions(-)
arch/x86/include/asm/fpu/api.h +2 -0
index 6053674f9132..7532f73c82a6 100644
@@ -138,6 +138,8 @@ static inline void fpstate_free(struct fpu *fpu) { }
 /* fpstate-related functions which are exported to KVM */
 extern void fpstate_clear_xstate_component(struct fpstate *fps, unsigned int xfeature);
 
+extern inline u64 xstate_get_guest_group_perm(void);
+
 /* KVM specific functions */
 extern bool fpu_alloc_guest_fpstate(struct fpu_guest *gfpu);
 extern void fpu_free_guest_fpstate(struct fpu_guest *gfpu);
arch/x86/include/asm/fpu/types.h +9 -0
index 3c06c82ab355..6ddf80637697 100644
@@ -387,6 +387,8 @@ struct fpstate {
 	/* @regs is dynamically sized! Don't add anything after @regs! */
 } __aligned(64);
 
+#define FPU_GUEST_PERM_LOCKED		BIT_ULL(63)
+
 struct fpu_state_perm {
 	/*
 	 * @__state_perm:
@@ -476,6 +478,13 @@ struct fpu {
 	 */
 	struct fpu_state_perm		perm;
 
+	/*
+	 * @guest_perm:
+	 *
+	 * Permission related information for guest pseudo FPUs
+	 */
+	struct fpu_state_perm		guest_perm;
+
 	/*
 	 * @__fpstate:
 	 *
arch/x86/include/uapi/asm/prctl.h +14 -12
index 754a07856817..500b96e71f18 100644
@@ -2,20 +2,22 @@
 #ifndef _ASM_X86_PRCTL_H
 #define _ASM_X86_PRCTL_H
 
-#define ARCH_SET_GS		0x1001
-#define ARCH_SET_FS		0x1002
-#define ARCH_GET_FS		0x1003
-#define ARCH_GET_GS		0x1004
+#define ARCH_SET_GS			0x1001
+#define ARCH_SET_FS			0x1002
+#define ARCH_GET_FS			0x1003
+#define ARCH_GET_GS			0x1004
 
-#define ARCH_GET_CPUID		0x1011
-#define ARCH_SET_CPUID		0x1012
+#define ARCH_GET_CPUID			0x1011
+#define ARCH_SET_CPUID			0x1012
 
-#define ARCH_GET_XCOMP_SUPP	0x1021
-#define ARCH_GET_XCOMP_PERM	0x1022
-#define ARCH_REQ_XCOMP_PERM	0x1023
+#define ARCH_GET_XCOMP_SUPP		0x1021
+#define ARCH_GET_XCOMP_PERM		0x1022
+#define ARCH_REQ_XCOMP_PERM		0x1023
+#define ARCH_GET_XCOMP_GUEST_PERM	0x1024
+#define ARCH_REQ_XCOMP_GUEST_PERM	0x1025
 
-#define ARCH_MAP_VDSO_X32	0x2001
-#define ARCH_MAP_VDSO_32	0x2002
-#define ARCH_MAP_VDSO_64	0x2003
+#define ARCH_MAP_VDSO_X32		0x2001
+#define ARCH_MAP_VDSO_32		0x2002
+#define ARCH_MAP_VDSO_64		0x2003
 
 #endif /* _ASM_X86_PRCTL_H */
arch/x86/kernel/fpu/core.c +3 -0
index 8ea306b1bf8e..ab19b3d8b2f7 100644
@@ -450,6 +450,8 @@ void fpstate_reset(struct fpu *fpu)
 	fpu->perm.__state_perm		= fpu_kernel_cfg.default_features;
 	fpu->perm.__state_size		= fpu_kernel_cfg.default_size;
 	fpu->perm.__user_state_size	= fpu_user_cfg.default_size;
+	/* Same defaults for guests */
+	fpu->guest_perm = fpu->perm;
 }
 
 static inline void fpu_inherit_perms(struct fpu *dst_fpu)
@@ -460,6 +462,7 @@ static inline void fpu_inherit_perms(struct fpu *dst_fpu)
 		spin_lock_irq(&current->sighand->siglock);
 		/* Fork also inherits the permissions of the parent */
 		dst_fpu->perm = src_fpu->perm;
+		dst_fpu->guest_perm = src_fpu->guest_perm;
 		spin_unlock_irq(&current->sighand->siglock);
 	}
 }
arch/x86/kernel/fpu/xstate.c +37 -13

115 lines not on this page · raw message

arch/x86/kernel/fpu/xstate.h +11 -2
index 86ea7c0fa2f6..98a472775c97 100644
@@ -20,10 +20,19 @@ static inline void xstate_init_xcomp_bv(struct xregs_state *xsave, u64 mask)
 		xsave->header.xcomp_bv = mask | XCOMP_BV_COMPACTED_FORMAT;
 }
 
-static inline u64 xstate_get_host_group_perm(void)
+static inline u64 xstate_get_group_perm(bool guest)
 {
+	struct fpu *fpu = &current->group_leader->thread.fpu;
+	struct fpu_state_perm *perm;
+
 	/* Pairs with WRITE_ONCE() in xstate_request_perm() */
-	return READ_ONCE(current->group_leader->thread.fpu.perm.__state_perm);
+	perm = guest ? &fpu->guest_perm : &fpu->perm;
+	return READ_ONCE(perm->__state_perm);
+}
+
+static inline u64 xstate_get_host_group_perm(void)
+{
+	return xstate_get_group_perm(false);
 }
 
 enum xstate_copy_mode {
arch/x86/kernel/process.c +2 -0
index 04143a653a8a..d7bc23589062 100644
@@ -993,6 +993,8 @@ long do_arch_prctl_common(struct task_struct *task, int option,
 	case ARCH_GET_XCOMP_SUPP:
 	case ARCH_GET_XCOMP_PERM:
 	case ARCH_REQ_XCOMP_PERM:
+	case ARCH_GET_XCOMP_GUEST_PERM:
+	case ARCH_REQ_XCOMP_GUEST_PERM:
 		return fpu_xstate_prctl(task, option, arg2);
 	}
 
#2 lore raw

Re: [PATCH 01/19] x86/fpu: Extend prctl() with guest permissions

Quoted 10 lines · Yang Zhong

Similar to native permissions this doesn't actually enable the permitted feature. KVM is expected to install a larger kernel buffer and enable the feature when detecting the intention from the guest.

---
(To Thomas) We change the definition of xstate_get_guest_group_perm() from xstate.h to api.h since this will be called by KVM.

No.

There is absolutely no need for that. After creating a vCPU the permissions are frozen and readily available via vcpu->arch.guest_fpu.perm.

Thanks,

tglx

author
#3 lore raw

PATCH 2/19 x86/fpu: Prepare KVM for dynamically enabled states

In reply to Yang Zhong

From: Thomas Gleixner <[email protected]>

Add more fields for tracking per-vCPU permissions for dynamic XSTATE components:

- user_xfeatures

Track which features are currently enabled for the vCPU

- user_perm

Copied from guest_perm of the group leader thread. The first vCPU which does the copy locks the guest_perm

- realloc_request

KVM sets this field to request dynamically-enabled features which require reallocation of @fpstate

Initialize those fields properly.

---
 arch/x86/include/asm/fpu/types.h | 23 +++++++++++++++++++++++
 arch/x86/kernel/fpu/core.c       | 26 +++++++++++++++++++++++++-
 2 files changed, 48 insertions(+), 1 deletion(-)
arch/x86/include/asm/fpu/types.h +23 -0
index 6ddf80637697..861cffca3209 100644
@@ -504,6 +504,29 @@ struct fpu {
  * Guest pseudo FPU container
  */
 struct fpu_guest {
+	/*
+	 * @user_xfeatures:		xfeature bitmap of features which are
+	 *				currently enabled for the guest vCPU.
+	 */
+	u64				user_xfeatures;
+
+	/*
+	 * @user_perm:			xfeature bitmap of features which are
+	 *				permitted to be enabled for the guest
+	 *				vCPU.
+	 */
+	u64				user_perm;
+
+	/*
+	 * @realloc_request:		xfeature bitmap of features which are
+	 *				requested to be enabled dynamically
+	 *				which requires reallocation of @fpstate
+	 *
+	 *				Set by an intercept handler and
+	 *				evaluated in fpu_swap_kvm_fpstate()
+	 */
+	u64				realloc_request;
+
 	/*
 	 * @fpstate:			Pointer to the allocated guest fpstate
 	 */
arch/x86/kernel/fpu/core.c +25 -1
index ab19b3d8b2f7..fe592799508c 100644
@@ -201,6 +201,26 @@ void fpu_reset_from_exception_fixup(void)
 #if IS_ENABLED(CONFIG_KVM)
 static void __fpstate_reset(struct fpstate *fpstate);
 
+static void fpu_init_guest_permissions(struct fpu_guest *gfpu)
+{
+	struct fpu_state_perm *fpuperm;
+	u64 perm;
+
+	if (!IS_ENABLED(CONFIG_X86_64))
+		return;
+
+	spin_lock_irq(&current->sighand->siglock);
+	fpuperm = &current->group_leader->thread.fpu.guest_perm;
+	perm = fpuperm->__state_perm;
+
+	/* First fpstate allocation locks down permissions. */
+	WRITE_ONCE(fpuperm->__state_perm, perm | FPU_GUEST_PERM_LOCKED);
+
+	spin_unlock_irq(&current->sighand->siglock);
+
+	gfpu->user_perm = perm & ~FPU_GUEST_PERM_LOCKED;
+}
+
 bool fpu_alloc_guest_fpstate(struct fpu_guest *gfpu)
 {
 	struct fpstate *fpstate;
@@ -216,7 +236,11 @@ bool fpu_alloc_guest_fpstate(struct fpu_guest *gfpu)
 	fpstate->is_valloc	= true;
 	fpstate->is_guest	= true;
 
-	gfpu->fpstate = fpstate;
+	gfpu->fpstate		= fpstate;
+	gfpu->user_xfeatures	= fpu_user_cfg.default_features;
+	gfpu->user_perm		= fpu_user_cfg.default_features;
+	fpu_init_guest_permissions(gfpu);
+
 	return true;
 }
 EXPORT_SYMBOL_GPL(fpu_alloc_guest_fpstate);
maintainer
#4 lore raw

Re: [PATCH 02/19] x86/fpu: Prepare KVM for dynamically enabled states

Quoted 3 lines · Yang Zhong

- user_xfeatures

Track which features are currently enabled for the vCPU

Please rename to alloc_xfeatures

Quoted 4 lines

- user_perm

Copied from guest_perm of the group leader thread. The first vCPU which does the copy locks the guest_perm

Please rename to perm_xfeatures.

Quoted 4 lines

- realloc_request

KVM sets this field to request dynamically-enabled features which require reallocation of @fpstate

This field should be in vcpu->arch, and there is no need for fpu_guest_realloc_fpstate. Rename __xfd_enable_feature to fpu_enable_xfd_feature and add it to the public API, then just do

	if (unlikely(vcpu->arch.xfd_realloc_request)) {
		u64 request = vcpu->arch.xfd_realloc_request;
		ret = fpu_enable_xfd(request, enter_guest);
	}

to kvm_put_guest_fpu.

Paolo

#5 lore raw

Re: [PATCH 02/19] x86/fpu: Prepare KVM for dynamically enabled states

Quoted 6 lines · Paolo Bonzini
Quoted 3 lines · Yang Zhong

- user_xfeatures

Track which features are currently enabled for the vCPU

Please rename to alloc_xfeatures

That name makes no sense at all. This has nothing to do with alloc.

Quoted 6 lines
Quoted 4 lines

- user_perm

Copied from guest_perm of the group leader thread. The first vCPU which does the copy locks the guest_perm

Please rename to perm_xfeatures.

All of that is following the naming conventions in the FPU code related to permissions etc.

Quoted 9 lines
Quoted 4 lines

- realloc_request

KVM sets this field to request dynamically-enabled features which require reallocation of @fpstate

This field should be in vcpu->arch, and there is no need for fpu_guest_realloc_fpstate. Rename __xfd_enable_feature to fpu_enable_xfd_feature and add it to the public API, then just do

Quoted 6 lines
	if (unlikely(vcpu->arch.xfd_realloc_request)) {
		u64 request = vcpu->arch.xfd_realloc_request;
		ret = fpu_enable_xfd(request, enter_guest);
	}

to kvm_put_guest_fpu.

Why? Yet another export of FPU internals just because?

Also what clears the reallocation request and what is the @enter_guest argument supposed to help with?

I have no idea what you are trying to achieve.

Thanks,

tglx

maintainer
#6 lore raw

Re: [PATCH 02/19] x86/fpu: Prepare KVM for dynamically enabled states

Quoted 9 lines · Thomas Gleixner
Quoted 6 lines · Paolo Bonzini
Quoted 3 lines · Yang Zhong

- user_xfeatures

Track which features are currently enabled for the vCPU

Please rename to alloc_xfeatures

That name makes no sense at all. This has nothing to do with alloc.

Isn't that the features for which space is currently allocated?

fpstate_realloc does

+ if (guest_fpu) {
+ newfps->is_guest = true;
+ newfps->is_confidential = curfps->is_confidential;
+ guest_fpu->user_xfeatures |= xfeatures;
+ }
+

and kvm_check_guest_realloc_fpstate does

+ if ((guest_fpu->user_xfeatures & request) != request) {
+ vcpu->arch.guest_fpu.realloc_request |= request;
+ return true;
+ }

Reading "user_xfeatures" in there is cryptic, it seems like it's something related to the userspace thread or group that has invoked the KVM ioctl. If it's renamed to alloc_xfeatures, then this:

+ missing = request & ~guest_fpu->alloc_xfeatures;
+ if (missing) {
+ vcpu->arch.guest_fpu.realloc_request |= missing;
+ return true;
+ }

makes it obvious that the allocation is for features that are requested but haven't been allocated in the xstate yet.

Quoted 9 lines
Quoted 6 lines
Quoted 4 lines

- user_perm

Copied from guest_perm of the group leader thread. The first vCPU which does the copy locks the guest_perm

Please rename to perm_xfeatures.

All of that is following the naming conventions in the FPU code related to permissions etc.

perm or guest_perm is just fine as well; but user_perm is not (there's no preexisting reference to user_perm in the code, in fact, as far as I can see).

Quoted 9 lines
Quoted 9 lines
Quoted 4 lines

- realloc_request

KVM sets this field to request dynamically-enabled features which require reallocation of @fpstate

This field should be in vcpu->arch, and there is no need for fpu_guest_realloc_fpstate. Rename __xfd_enable_feature to fpu_enable_xfd_feature and add it to the public API, then just do

Quoted 8 lines
Quoted 6 lines
	if (unlikely(vcpu->arch.xfd_realloc_request)) {
		u64 request = vcpu->arch.xfd_realloc_request;
		ret = fpu_enable_xfd(request, enter_guest);
	}

to kvm_put_guest_fpu.

Why? Yet another export of FPU internals just because?

It's one function more and one field less. I prefer another export of FPU internals, to a write to a random field with undocumented invariants.

For example, why WARN_ON_ONCE if enter_guest == true? If you enter the guest after the host has restored MSR_IA32_XFD with KVM_SET_MSR, the WARN_ON_ONCE can actually fire. It would be just fine to skip the reallocation until enter_guest == false, which is you actually XSAVE into the guest_fpu.

As an aside, realloc_request (if it survives, see below) shouldn't be added until patch 6.

Quoted 2 lines

Also what clears the reallocation request and what is the @enter_guest argument supposed to help with?

Nothing---make it

if (unlikely(vcpu->arch.xfd_realloc_request)) {
u64 request = vcpu->arch.xfd_realloc_request;

 		ret = fpu_enable_xfd(request, &vcpu->arch.guest_fpu);
		if (!ret)
			vcpu->arch.xfd_realloc_request = 0;

}

but in fact, I'm not sure why the request has to be delayed at all. The obvious implementation of a write to XFD, after all the validity checks, is just

	/* This function just calls xfd_enable_feature.  */
	r = fpu_guest_alloc_xfeatures(&vcpu->arch.guest_fpu,
				      vcpu->arch.xcr0 & ~xfd);
	/*
	 * An error means that userspace has screwed up by not doing
	 * arch_prctl(ARCH_REQ_XCOMP_GUEST_PERM).  If we are here coming
	 * from a ioctl fail it, if the guest has done WRMSR or XSETBV
	 * it will inject a #GP.
	 */
	if (r < 0)
		return 1;

	vcpu->arch.xfd = xfd;

with none of the kvm_check_guest_realloc_fpstate or KVM_EXIT_FPU_REALLOC business. No field and a simple, self-contained external API. The same code works for __kvm_set_xcr as well, just with "xcr0 & ~vcpu->arch.xfd" as the second argument instead.

(Maybe I'm missing why KVM_EXIT_FPU_REALLOC is needed, but I'm not ashamed to say that, given that new userspace API was added with documentation or tests. The only comment in the code is:

+ /*
+ * Check if fpstate reallocate is required. If yes, then
+ * let the fpu core do reallocation and update xfd;
+ * otherwise, update xfd here.
+ */
+ if (kvm_check_guest_realloc_fpstate(vcpu, data)) {
+ vcpu->run->exit_reason = KVM_EXIT_FPU_REALLOC;
+ vcpu->arch.complete_userspace_io =
+ kvm_skip_emulated_instruction;
+ return KVM_MSR_RET_USERSPACE;
+ }
+

which has nothing to do with the actual content of either kvm_check_guest_realloc_fpstate or the "then" branch. Userspace can just do ARCH_REQ_XCOMP_GUEST_PERM based on the guest CPUID, before KVM_SET_CPUID2, KVM_SET_MSR or KVM_SET_XCR).

Paolo

#7 lore raw

Re: [PATCH 02/19] x86/fpu: Prepare KVM for dynamically enabled states

Paolo,

Quoted 7 lines · Paolo Bonzini
Quoted 4 lines · Thomas Gleixner
Quoted 1 line · Paolo Bonzini

Please rename to alloc_xfeatures

That name makes no sense at all. This has nothing to do with alloc.

Isn't that the features for which space is currently allocated?

It is, but from the kernel POV this is user. :)

Quoted 10 lines

Reading "user_xfeatures" in there is cryptic, it seems like it's something related to the userspace thread or group that has invoked the KVM ioctl. If it's renamed to alloc_xfeatures, then this:

(quoted diff) +5 -0
+		missing = request & ~guest_fpu->alloc_xfeatures;
+		if (missing) {
+			vcpu->arch.guest_fpu.realloc_request |= missing;
+			return true;
+		}
Quoted 2 lines

makes it obvious that the allocation is for features that are requested but haven't been allocated in the xstate yet.

Let's rename it to xfeatures and perm and be done with it.

Quoted 5 lines
Quoted 1 line

Why? Yet another export of FPU internals just because?

It's one function more and one field less. I prefer another export of FPU internals, to a write to a random field with undocumented invariants.

We want less not more exports. :)

Quoted 2 lines

For example, why WARN_ON_ONCE if enter_guest == true? If you enter the guest after the host has restored MSR_IA32_XFD with KVM_SET_MSR, the

Indeed restoring a guest might require buffer reallocation, I missed that, duh!

On restore the following components are involved:

XCR0, XFD, XSTATE

XCR0 and XFD have to be restored _before_ XSTATE and that needs to be enforced.

But independent of the ordering of XCR0 and XFD restore the following check applies to both the restore and the runtime logic:

int kvm_fpu_realloc(struct kvm_vcpu *vcpu, u64 xcr0, u64 xfd) {
u64 expand, enabled = xcr0 & ~xfd;

        expand = enabled & ~vcpu->arch.guest_fpu.xfeatures;
        if (!expand)
        	return 0;
        
        return fpu_enable_guest_features(&vcpu->arch.guest_fpu, expand);
}

int fpu_enable_guest_features(struct guest_fpu *gfpu, u64 which)

{
        permission_checks();
        ...
        return fpstate_realloc(.....)
}

fpstate_realloc() needs to be careful about flipping the pointers depending on the question whether guest_fpu->fpstate is actually active, i.e.:

        current->thread.fpu.fpstate == gfpu->fpstate

I'm halfways done with that. Will send something soonish.

Thanks,

tglx

author
#8 lore raw

PATCH 3/19 kvm: x86: Fix xstate_required_size() to follow XSTATE alignment rule

In reply to Yang Zhong

From: Jing Liu <[email protected]>

CPUID.0xD.1.EBX enumerates the size of the XSAVE area (in compacted format) required by XSAVES. If CPUID.0xD.i.ECX[1] is set for a state component (i), this state component should be located on the next 64-bytes boundary following the preceding state component in the compacted layout.

Fix xstate_required_size() to follow the alignment rule. AMX is the first state component with 64-bytes alignment to catch this bug.

---
 arch/x86/kvm/cpuid.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/kvm/cpuid.c +2 -1
index 07e9215e911d..148003e26cbb 100644
@@ -42,7 +42,8 @@ static u32 xstate_required_size(u64 xstate_bv, bool compacted)
 		if (xstate_bv & 0x1) {
 		        u32 eax, ebx, ecx, edx, offset;
 		        cpuid_count(0xD, feature_bit, &eax, &ebx, &ecx, &edx);
-			offset = compacted ? ret : ebx;
+			/* ECX[1]: 64B alignment in compacted form */
+			offset = compacted ? ((ecx & 0x2) ? ALIGN(ret, 64) : ret) : ebx;
 			ret = max(ret, offset + eax);
 		}
 
author
#9 lore raw

PATCH 4/19 kvm: x86: Check guest xstate permissions when KVM_SET_CPUID2

In reply to Yang Zhong

From: Jing Liu <[email protected]>

Guest xstate permissions should be set by userspace VMM before vcpu creation. This patch extends KVM to check the guest permissions in KVM_SET_CPUID2 ioctl to avoid permission failure at guest run-time (e.g. when reallocation path is triggered).

---
 arch/x86/kvm/cpuid.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)
arch/x86/kvm/cpuid.c +12 -0
index 148003e26cbb..f3c61205bbf4 100644
@@ -18,6 +18,7 @@
 #include <asm/processor.h>
 #include <asm/user.h>
 #include <asm/fpu/xstate.h>
+#include <asm/fpu/api.h>
 #include <asm/sgx.h>
 #include "cpuid.h"
 #include "lapic.h"
@@ -97,6 +98,17 @@ static int kvm_check_cpuid(struct kvm_cpuid_entry2 *entries, int nent)
 			return -EINVAL;
 	}
 
+	/*
+	 * Check guest permissions for XSTATE features which must
+	 * be enabled dynamically.
+	 */
+	best = cpuid_entry2_find(entries, nent, 7, 0);
+	if (best && cpuid_entry_has(best, X86_FEATURE_AMX_TILE)) {
+		if (!(xstate_get_guest_group_perm() &
+			XFEATURE_MASK_XTILE_DATA))
+			return -EINVAL;
+	}
+
 	return 0;
 }
 
author
#10 lore raw

PATCH 5/19 x86/fpu: Move xfd initialization out of __fpstate_reset() to the callers

In reply to Yang Zhong

From: Jing Liu <[email protected]>

vCPU threads are different from native tasks regarding to the initial xfd value. While all native tasks follow a fixed value (init_fpstate::xfd) defined by fpu core, vCPU threads need to obey the reset value (i.e. ZERO) defined by the spec, to meet the expectation of the guest.

Move xfd initialization out of __fpstate_reset() to the callers for choosing a specific value.

---
 arch/x86/kernel/fpu/core.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/kernel/fpu/core.c +2 -1
index fe592799508c..fae44fa27cdb 100644
@@ -231,6 +231,7 @@ bool fpu_alloc_guest_fpstate(struct fpu_guest *gfpu)
 	if (!fpstate)
 		return false;
 
+	/* Leave xfd to 0 (the reset value defined by spec) */
 	__fpstate_reset(fpstate);
 	fpstate_init_user(fpstate);
 	fpstate->is_valloc	= true;
@@ -461,7 +462,6 @@ static void __fpstate_reset(struct fpstate *fpstate)
 	fpstate->user_size	= fpu_user_cfg.default_size;
 	fpstate->xfeatures	= fpu_kernel_cfg.default_features;
 	fpstate->user_xfeatures	= fpu_user_cfg.default_features;
-	fpstate->xfd		= init_fpstate.xfd;
 }
 
 void fpstate_reset(struct fpu *fpu)
@@ -469,6 +469,7 @@ void fpstate_reset(struct fpu *fpu)
 	/* Set the fpstate pointer to the default fpstate */
 	fpu->fpstate = &fpu->__fpstate;
 	__fpstate_reset(fpu->fpstate);
+	fpu->fpstate->xfd		= init_fpstate.xfd;
 
 	/* Initialize the permission related info in fpu */
 	fpu->perm.__state_perm		= fpu_kernel_cfg.default_features;
#11 lore raw

Re: [PATCH 05/19] x86/fpu: Move xfd initialization out of __fpstate_reset() to the callers

Yang, Jing,

arch/x86/kernel/fpu/core.c bool fpu_alloc_guest_fpstate(struct fpu_guest *gfpu) · line 235

Quoted 10 lines · Yang Zhong
arch/x86/kernel/fpu/core.c +1 -0
index fe592799508c..fae44fa27cdb 100644
@@ -231,6 +231,7 @@ bool fpu_alloc_guest_fpstate(struct fpu_guest *gfpu)
 	if (!fpstate)
 		return false;
 
+	/* Leave xfd to 0 (the reset value defined by spec) */
 	__fpstate_reset(fpstate);

That change makes me a bit wary simply because the comment here is above __fpstate_reset() which makes no sense. It does make sense to you at the time, but does it make sense to you when you look at it 6 month down the road?

So I'd rather make this very obvious what's going. See below.

Thanks,

tglx
---

arch/x86/kernel/fpu/core.c +6 -5
@@ -199,7 +199,7 @@ void fpu_reset_from_exception_fixup(void
 }
 
 #if IS_ENABLED(CONFIG_KVM)
-static void __fpstate_reset(struct fpstate *fpstate);
+static void __fpstate_reset(struct fpstate *fpstate, u64 xfd);
 
 static void fpu_init_guest_permissions(struct fpu_guest *gfpu)
 {
@@ -231,7 +231,8 @@ bool fpu_alloc_guest_fpstate(struct fpu_
 	if (!fpstate)
 		return false;
 
-	__fpstate_reset(fpstate);
+	/* Leave xfd to 0 (the reset value defined by spec) */
+	__fpstate_reset(fpstate, 0);
 	fpstate_init_user(fpstate);
 	fpstate->is_valloc	= true;
 	fpstate->is_guest	= true;
@@ -454,21 +455,21 @@ void fpstate_init_user(struct fpstate *f
 		fpstate_init_fstate(fpstate);
 }
 
-static void __fpstate_reset(struct fpstate *fpstate)
+static void __fpstate_reset(struct fpstate *fpstate, u64 xfd)
 {
 	/* Initialize sizes and feature masks */
 	fpstate->size		= fpu_kernel_cfg.default_size;
 	fpstate->user_size	= fpu_user_cfg.default_size;
 	fpstate->xfeatures	= fpu_kernel_cfg.default_features;
 	fpstate->user_xfeatures	= fpu_user_cfg.default_features;
-	fpstate->xfd		= init_fpstate.xfd;
+	fpstate->xfd		= xfd;
 }
 
 void fpstate_reset(struct fpu *fpu)
 {
 	/* Set the fpstate pointer to the default fpstate */
 	fpu->fpstate = &fpu->__fpstate;
-	__fpstate_reset(fpu->fpstate);
+	__fpstate_reset(fpu->fpstate, init_fpstate.xfd);
 
 	/* Initialize the permission related info in fpu */
 	fpu->perm.__state_perm		= fpu_kernel_cfg.default_features;
author
#12 lore raw

PATCH 6/19 x86/fpu: Add reallocation mechanims for KVM

In reply to Yang Zhong

From: Thomas Gleixner <[email protected]>

Extend fpstate reallocation mechanism to cover guest fpu. Unlike native tasks which have reallocation triggered from #NM handler, guest fpstate reallocation is requested by KVM when detecting the guest intention on using a dynamically-enabled XSAVE feature.

Since KVM currently swaps host/guest fpstate when exiting to userspace VMM (see fpu_swap_kvm_fpstate()), deal with fpstate reallocation also at this point.

The implication - KVM must break vcpu_run() loop to exit to userspace VMM instead of immediately returning back to the guest when fpstate requires reallocation. In this case KVM should set guest_fpu::realloc_request to mark those features in related VM exit handlers.

---
 arch/x86/kernel/fpu/core.c   | 26 +++++++++++++++++++---
 arch/x86/kernel/fpu/xstate.c | 43 ++++++++++++++++++++++++++++++------
 arch/x86/kernel/fpu/xstate.h |  2 ++
 3 files changed, 61 insertions(+), 10 deletions(-)
arch/x86/kernel/fpu/core.c +23 -3
index fae44fa27cdb..7a0436a0cb2c 100644
@@ -261,11 +261,31 @@ void fpu_free_guest_fpstate(struct fpu_guest *gfpu)
 }
 EXPORT_SYMBOL_GPL(fpu_free_guest_fpstate);
 
+static int fpu_guest_realloc_fpstate(struct fpu_guest *guest_fpu,
+				     bool enter_guest)
+{
+	/*
+	 * Reallocation requests can only be handled when
+	 * switching from guest to host mode.
+	 */
+	if (WARN_ON_ONCE(enter_guest || !IS_ENABLED(CONFIG_X86_64))) {
+		guest_fpu->realloc_request = 0;
+		return -EUNATCH;
+	}
+	return xfd_enable_guest_features(guest_fpu);
+}
+
 int fpu_swap_kvm_fpstate(struct fpu_guest *guest_fpu, bool enter_guest)
 {
-	struct fpstate *guest_fps = guest_fpu->fpstate;
+	struct fpstate *guest_fps, *cur_fps;
 	struct fpu *fpu = &current->thread.fpu;
-	struct fpstate *cur_fps = fpu->fpstate;
+	int ret = 0;
+
+	if (unlikely(guest_fpu->realloc_request))
+		ret = fpu_guest_realloc_fpstate(guest_fpu, enter_guest);
+
+	guest_fps = guest_fpu->fpstate;
+	cur_fps = fpu->fpstate;
 
 	fpregs_lock();
 	if (!cur_fps->is_confidential && !test_thread_flag(TIF_NEED_FPU_LOAD))
@@ -298,7 +318,7 @@ int fpu_swap_kvm_fpstate(struct fpu_guest *guest_fpu, bool enter_guest)
 
 	fpregs_mark_activate();
 	fpregs_unlock();
-	return 0;
+	return ret;
 }
 EXPORT_SYMBOL_GPL(fpu_swap_kvm_fpstate);
 
arch/x86/kernel/fpu/xstate.c +36 -7

100 lines not on this page · raw message

arch/x86/kernel/fpu/xstate.h +2 -0
index 98a472775c97..3254e2b5f17f 100644
@@ -55,6 +55,8 @@ extern void fpu__init_system_xstate(unsigned int legacy_size);
 
 extern void *get_xsave_addr(struct xregs_state *xsave, int xfeature_nr);
 
+extern int xfd_enable_guest_features(struct fpu_guest *guest_fpu);
+
 static inline u64 xfeatures_mask_supervisor(void)
 {
 	return fpu_kernel_cfg.max_features & XFEATURE_MASK_SUPERVISOR_SUPPORTED;
author
#13 lore raw

PATCH 7/19 kvm: x86: Propagate fpstate reallocation error to userspace

In reply to Yang Zhong

From: Jing Liu <[email protected]>

fpstate reallocation is handled when the vCPU thread returns to userspace. As reallocation could fail (e.g. lack of memory), this patch extends kvm_put_guest_fpu() to return an integer value to carry error code to userspace VMM. The userspace VMM is expected to handle any error caused by fpstate reallocation.

---
 arch/x86/kvm/x86.c | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)
arch/x86/kvm/x86.c +11 -4
index 0ee1a039b490..05f2cda73d69 100644
@@ -10171,17 +10171,21 @@ static void kvm_load_guest_fpu(struct kvm_vcpu *vcpu)
 }
 
 /* When vcpu_run ends, restore user space FPU context. */
-static void kvm_put_guest_fpu(struct kvm_vcpu *vcpu)
+static int kvm_put_guest_fpu(struct kvm_vcpu *vcpu)
 {
-	fpu_swap_kvm_fpstate(&vcpu->arch.guest_fpu, false);
+	int ret;
+
+	ret = fpu_swap_kvm_fpstate(&vcpu->arch.guest_fpu, false);
 	++vcpu->stat.fpu_reload;
 	trace_kvm_fpu(0);
+
+	return ret;
 }
 
 int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
 {
 	struct kvm_run *kvm_run = vcpu->run;
-	int r;
+	int r, ret;
 
 	vcpu_load(vcpu);
 	kvm_sigset_activate(vcpu);
@@ -10243,7 +10247,10 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
 		r = vcpu_run(vcpu);
 
 out:
-	kvm_put_guest_fpu(vcpu);
+	ret = kvm_put_guest_fpu(vcpu);
+	if ((r >= 0) && (ret < 0))
+		r = ret;
+
 	if (kvm_run->kvm_valid_regs)
 		store_regs(vcpu);
 	post_kvm_run_save(vcpu);
maintainer
#14 lore raw

Re: [PATCH 07/19] kvm: x86: Propagate fpstate reallocation error to userspace

arch/x86/kvm/x86.c int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) · line 10253

Quoted 6 lines · Yang Zhong

out:

arch/x86/kvm/x86.c +4 -1
-	kvm_put_guest_fpu(vcpu);
+	ret = kvm_put_guest_fpu(vcpu);
+	if ((r >= 0) && (ret < 0))
+		r = ret;
+

No extra parentheses.

Paolo

author
#15 lore raw

PATCH 8/19 x86/fpu: Move xfd_update_state() to xstate.c and export symbol

In reply to Yang Zhong

From: Jing Liu <[email protected]>

xfd_update_state() is the interface to update IA32_XFD and its per-cpu cache. All callers of this interface are currently in fpu core. KVM only indirectly triggers IA32_XFD update via a helper function (fpu_swap_kvm_fpstate()) when switching between user fpu and guest fpu.

Supporting AMX in guest now requires KVM to directly update IA32_XFD with the guest value (when emulating WRMSR) so XSAVE/XRSTOR can manage XSTATE components correctly inside guest.

This patch moves xfd_update_state() from fpu/xstate.h to fpu/xstate.c and export it for reference outside of fpu core.

---
 arch/x86/include/asm/fpu/api.h |  2 ++
 arch/x86/kernel/fpu/xstate.c   | 12 ++++++++++++
 arch/x86/kernel/fpu/xstate.h   | 14 +-------------
 3 files changed, 15 insertions(+), 13 deletions(-)
arch/x86/include/asm/fpu/api.h +2 -0
index 7532f73c82a6..999d89026be9 100644
@@ -131,8 +131,10 @@ DECLARE_PER_CPU(struct fpu *, fpu_fpregs_owner_ctx);
 /* Process cleanup */
 #ifdef CONFIG_X86_64
 extern void fpstate_free(struct fpu *fpu);
+extern void xfd_update_state(struct fpstate *fpstate);
 #else
 static inline void fpstate_free(struct fpu *fpu) { }
+static void xfd_update_state(struct fpstate *fpstate) { }
 #endif
 
 /* fpstate-related functions which are exported to KVM */
arch/x86/kernel/fpu/xstate.c +12 -0
index fe3d8ed3db0e..3c39789deeb9 100644
@@ -1750,6 +1750,18 @@ int xfd_enable_guest_features(struct fpu_guest *guest_fpu)
 	return __xfd_enable_feature(xfd_err, guest_fpu);
 }
 
+void xfd_update_state(struct fpstate *fpstate)
+{
+	if (fpu_state_size_dynamic()) {
+		u64 xfd = fpstate->xfd;
+
+		if (__this_cpu_read(xfd_state) != xfd) {
+			wrmsrl(MSR_IA32_XFD, xfd);
+			__this_cpu_write(xfd_state, xfd);
+		}
+	}
+}
+EXPORT_SYMBOL_GPL(xfd_update_state);
 #else /* CONFIG_X86_64 */
 static inline int xstate_request_perm(unsigned long idx, bool guest)
 {
arch/x86/kernel/fpu/xstate.h +1 -13
index 3254e2b5f17f..651bd29977b9 100644
@@ -149,19 +149,7 @@ static inline void xfd_validate_state(struct fpstate *fpstate, u64 mask, bool rs
 #endif
 
 #ifdef CONFIG_X86_64
-static inline void xfd_update_state(struct fpstate *fpstate)
-{
-	if (fpu_state_size_dynamic()) {
-		u64 xfd = fpstate->xfd;
-
-		if (__this_cpu_read(xfd_state) != xfd) {
-			wrmsrl(MSR_IA32_XFD, xfd);
-			__this_cpu_write(xfd_state, xfd);
-		}
-	}
-}
-#else
-static inline void xfd_update_state(struct fpstate *fpstate) { }
+extern void xfd_update_state(struct fpstate *fpstate);
 #endif
 
 /*
#16 lore raw

Re: [PATCH 08/19] x86/fpu: Move xfd_update_state() to xstate.c and export symbol

Quoted 7 lines · Yang Zhong

From: Jing Liu <[email protected]>

xfd_update_state() is the interface to update IA32_XFD and its per-cpu cache. All callers of this interface are currently in fpu core. KVM only indirectly triggers IA32_XFD update via a helper function (fpu_swap_kvm_fpstate()) when switching between user fpu and guest fpu.

Quoted 5 lines

Supporting AMX in guest now requires KVM to directly update IA32_XFD with the guest value (when emulating WRMSR) so XSAVE/XRSTOR can manage XSTATE components correctly inside guest.

This patch moves xfd_update_state() from fpu/xstate.h to fpu/xstate.c

s/This patch moves/Move/

please. See Documentation/process/submitting-patches.rst and search for 'This patch'

Quoted 15 lines

and export it for reference outside of fpu core.

---
 arch/x86/include/asm/fpu/api.h |  2 ++
 arch/x86/kernel/fpu/xstate.c   | 12 ++++++++++++
 arch/x86/kernel/fpu/xstate.h   | 14 +-------------
 3 files changed, 15 insertions(+), 13 deletions(-)
arch/x86/include/asm/fpu/api.h +0 -0
index 7532f73c82a6..999d89026be9 100644

arch/x86/include/asm/fpu/api.h DECLARE_PER_CPU(struct fpu *, fpu_fpregs_owner_ctx); · line 137

Quoted 7 lines
arch/x86/include/asm/fpu/api.h +2 -0
@@ -131,8 +131,10 @@ DECLARE_PER_CPU(struct fpu *, fpu_fpregs_owner_ctx);
 /* Process cleanup */
 #ifdef CONFIG_X86_64
 extern void fpstate_free(struct fpu *fpu);
+extern void xfd_update_state(struct fpstate *fpstate);
 #else
 static inline void fpstate_free(struct fpu *fpu) { }
+static void xfd_update_state(struct fpstate *fpstate) { }

Try a 32bit build to see the warnings this causes. That wants to be 'static inline void' obviously.

Quoted 5 lines
arch/x86/include/asm/fpu/api.h +0 -4
 #ifdef CONFIG_X86_64
-static inline void xfd_update_state(struct fpstate *fpstate)
-{
-	if (fpu_state_size_dynamic()) {
-		u64 xfd = fpstate->xfd;

arch/x86/include/asm/fpu/api.h DECLARE_PER_CPU(struct fpu *, fpu_fpregs_owner_ctx);

Quoted 10 lines
arch/x86/include/asm/fpu/api.h +1 -9
-
-		if (__this_cpu_read(xfd_state) != xfd) {
-			wrmsrl(MSR_IA32_XFD, xfd);
-			__this_cpu_write(xfd_state, xfd);
-		}
-	}
-}
-#else
-static inline void xfd_update_state(struct fpstate *fpstate) { }
+extern void xfd_update_state(struct fpstate *fpstate);

Why? It's already declared in the global header. So all of this has to be simply removed, no?

Thanks,

tglx

author
#17 lore raw

PATCH 9/19 kvm: x86: Prepare reallocation check

In reply to Yang Zhong

From: Jing Liu <[email protected]>

On native fpstate reallocation is triggered by #NM because IA32_XFD is initialized to 1 for all native tasks.

However #NM in guest is not trapped by KVM. Instead, guest enabling of a dynamic extended feature can be captured via emulation of IA32_XFD and XSETBV. Basically having guest XCR0[i]=1 and XFD[i]=0 indicates that the feature[i] is activated by the guest.

This patch provides a helper function for such check, invoked when either XCR0 or XFD is changed in the emulation path.

---
 arch/x86/kvm/x86.c | 24 ++++++++++++++++++++++++
 arch/x86/kvm/x86.h |  1 +
 2 files changed, 25 insertions(+)
arch/x86/kvm/x86.c +24 -0
index 05f2cda73d69..91cc6f69a7ca 100644
@@ -956,6 +956,30 @@ void kvm_load_host_xsave_state(struct kvm_vcpu *vcpu)
 }
 EXPORT_SYMBOL_GPL(kvm_load_host_xsave_state);
 
+bool kvm_check_guest_realloc_fpstate(struct kvm_vcpu *vcpu, u64 xfd)
+{
+	u64 xcr0 = vcpu->arch.xcr0 & XFEATURE_MASK_USER_DYNAMIC;
+
+	/* For any state which is enabled dynamically */
+	if ((xfd & xcr0) != xcr0) {
+		u64 request = (xcr0 ^ xfd) & xcr0;
+		struct fpu_guest *guest_fpu = &vcpu->arch.guest_fpu;
+
+		/*
+		 * If requested features haven't been enabled, update
+		 * the request bitmap and tell the caller to request
+		 * dynamic buffer reallocation.
+		 */
+		if ((guest_fpu->user_xfeatures & request) != request) {
+			vcpu->arch.guest_fpu.realloc_request = request;
+			return true;
+		}
+	}
+
+	return false;
+}
+EXPORT_SYMBOL_GPL(kvm_check_guest_realloc_fpstate);
+
 static int __kvm_set_xcr(struct kvm_vcpu *vcpu, u32 index, u64 xcr)
 {
 	u64 xcr0 = xcr;
arch/x86/kvm/x86.h +1 -0
index 4abcd8d9836d..24a323980146 100644
@@ -445,6 +445,7 @@ static inline void kvm_machine_check(void)
 
 void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu);
 void kvm_load_host_xsave_state(struct kvm_vcpu *vcpu);
+bool kvm_check_guest_realloc_fpstate(struct kvm_vcpu *vcpu, u64 new_xfd);
 int kvm_spec_ctrl_test_value(u64 value);
 bool kvm_is_valid_cr4(struct kvm_vcpu *vcpu, unsigned long cr4);
 int kvm_handle_memory_failure(struct kvm_vcpu *vcpu, int r,
maintainer
#18 lore raw

Re: [PATCH 09/19] kvm: x86: Prepare reallocation check

Quoted 4 lines · Yang Zhong
arch/x86/kvm/x86.c +4 -0
+	u64 xcr0 = vcpu->arch.xcr0 & XFEATURE_MASK_USER_DYNAMIC;
+
+	/* For any state which is enabled dynamically */
+	if ((xfd & xcr0) != xcr0) {

arch/x86/kvm/x86.c void kvm_load_host_xsave_state(struct kvm_vcpu *vcpu) · line 974

Quoted 10 lines
arch/x86/kvm/x86.c +10 -0
+		u64 request = (xcr0 ^ xfd) & xcr0;
+		struct fpu_guest *guest_fpu = &vcpu->arch.guest_fpu;
+
+		/*
+		 * If requested features haven't been enabled, update
+		 * the request bitmap and tell the caller to request
+		 * dynamic buffer reallocation.
+		 */
+		if ((guest_fpu->user_xfeatures & request) != request) {
+			vcpu->arch.guest_fpu.realloc_request = request;

This should be "|=". If you have

	wrmsr(XFD, dynamic-feature-1);
	...
	wrmsr(XFD, dynamic-feature-2);

then the space for both features has to be allocated.

arch/x86/kvm/x86.c void kvm_load_host_xsave_state(struct kvm_vcpu *vcpu) · line 978

Quoted 4 lines
arch/x86/kvm/x86.c +4 -0
+			return true;
+		}
+	}
+

This is just:

	struct fpu_guest *guest_fpu = &vcpu->arch.guest_fpu;
	u64 xcr0 = vcpu->arch.xcr0 & XFEATURE_MASK_USER_DYNAMIC;
	u64 dynamic_enabled = xcr0 & ~xfd;
	if (!(dynamic_enabled & ~guest_fpu->user_xfeatures))
		return false;

	/*
	 * This should actually not be in guest_fpu, see review of
	 * patch 2.  Also see above regarding "=" vs "|=".
	 */
	vcpu->arch.guest_fpu.realloc_request |= dynamic_enabled;
	return true;

But without documentation I'm not sure why this exit-to-userspace step is needed:

- if (dynamic_enabled & ~guest_fpu->user_perm) != 0, then this is a userspace error and you can #GP the guest without any issue. Userspace is buggy

- if (dynamic_enabled & ~guest_fpu->user_xfeatures) != 0, but the feature *is* permitted, then it is okay to just go on and reallocate the guest FPU.

Paolo

#19 lore raw

RE: [PATCH 09/19] kvm: x86: Prepare reallocation check

Quoted 8 lines

From: Paolo Bonzini
Sent: Monday, December 13, 2021 5:16 PM

- if (dynamic_enabled & ~guest_fpu->user_perm) != 0, then this is a userspace error and you can #GP the guest without any issue. Userspace is buggy

Is it a general guideline that an error caused by emulation itself (e.g. due to no memory) can be reflected into the guest as #GP, even when from guest p.o.v there is nothing wrong with its setting?

Thanks
Kevin

maintainer
#20 lore raw

Re: [PATCH 09/19] kvm: x86: Prepare reallocation check

Quoted 7 lines · Kevin
Quoted 3 lines

- if (dynamic_enabled & ~guest_fpu->user_perm) != 0, then this is a userspace error and you can #GP the guest without any issue. Userspace is buggy

Is it a general guideline that an error caused by emulation itself (e.g. due to no memory) can be reflected into the guest as #GP, even when from guest p.o.v there is nothing wrong with its setting?

No memory is a tricky one, if possible it should propagate -ENOMEM up to KVM_RUN or KVM_SET_MSR. But it's basically an impossible case anyway, because even with 8K TILEDATA we're within the limit of PAGE_ALLOC_COSTLY_ORDER.

So, since it's not easy to do it right now, we can look at it later.

Paolo

#21 lore raw

RE: [PATCH 09/19] kvm: x86: Prepare reallocation check

Quoted 10 lines · Paolo Bonzini
Quoted 7 lines · Kevin
Quoted 3 lines

- if (dynamic_enabled & ~guest_fpu->user_perm) != 0, then this is a userspace error and you can #GP the guest without any issue. Userspace is buggy

Is it a general guideline that an error caused by emulation itself (e.g. due to no memory) can be reflected into the guest as #GP, even when from guest p.o.v there is nothing wrong with its setting?

Quoted 6 lines

No memory is a tricky one, if possible it should propagate -ENOMEM up to KVM_RUN or KVM_SET_MSR. But it's basically an impossible case anyway, because even with 8K TILEDATA we're within the limit of PAGE_ALLOC_COSTLY_ORDER.

So, since it's not easy to do it right now, we can look at it later.

For the way handling xcr0 and xfd ioctl failure, xcr0 and xfd have different handlings. Current KVM_SET_XCRS returns -EINVAL to userspace. KVM_SET_MSR is always allowed as the discussion in another thread.

So I'm thinking if reallocation failure in KVM_SET_XCRS and KVM_SET_MSR (may due to NOMEM or EPERM or ENOTSUPP), what is the way we would like to choose?

Thanks,
Jing

Quoted 1 line

Paolo

#22 lore raw

RE: [PATCH 09/19] kvm: x86: Prepare reallocation check

Quoted 21 lines

From: Liu, Jing2 <[email protected]>
Sent: Tuesday, December 14, 2021 10:42 PM

Quoted 10 lines · Paolo Bonzini
Quoted 7 lines · Kevin
Quoted 3 lines

- if (dynamic_enabled & ~guest_fpu->user_perm) != 0, then this is a userspace error and you can #GP the guest without any issue. Userspace is buggy

Is it a general guideline that an error caused by emulation itself (e.g. due to no memory) can be reflected into the guest as #GP, even when from guest p.o.v there is nothing wrong with its setting?

Quoted 6 lines

No memory is a tricky one, if possible it should propagate -ENOMEM up to KVM_RUN or KVM_SET_MSR. But it's basically an impossible case anyway, because even with 8K TILEDATA we're within the limit of PAGE_ALLOC_COSTLY_ORDER.

So, since it's not easy to do it right now, we can look at it later.

Quoted 9 lines

For the way handling xcr0 and xfd ioctl failure, xcr0 and xfd have different handlings. Current KVM_SET_XCRS returns -EINVAL to userspace. KVM_SET_MSR is always allowed as the discussion in another thread.

So I'm thinking if reallocation failure in KVM_SET_XCRS and KVM_SET_MSR (may due to NOMEM or EPERM or ENOTSUPP), what is the way we would like to choose?

KVM_SET_MSRS can definitely accept failure according to msr_io(). I think Paolo's point is more about that the restore path should not inherit any check related to vCPU capability. It's a different matter if the error is caused by other host kernel errors.

Given that we don't need any special handling between the two scenarios (set by guest vs. set by host) in those emulation paths. Just return '1' to indicate error and whatever error policy exists in those scenarios is just applied.

Thanks
Kevin

author
#23 lore raw

PATCH 10/19 kvm: x86: Emulate WRMSR of guest IA32_XFD

In reply to Yang Zhong

From: Jing Liu <[email protected]>

Intel's eXtended Feature Disable (XFD) feature allows the software to dynamically adjust fpstate buffer size for XSAVE features which have large state.

WRMSR to IA32_XFD is intercepted so if the written value enables a dynamic XSAVE feature the emulation code can exit to userspace to trigger fpstate reallocation for the state.

Introduce a new KVM exit reason (KVM_EXIT_FPU_REALLOC) for this purpose. If reallocation succeeds in fpu_swap_kvm_fpstate(), this exit just bounces to userspace and then back. Otherwise the userspace VMM should handle the error properly.

Use a new exit reason (instead of KVM_EXIT_X86_WRMSR) is clearer and can be shared between WRMSR(IA32_XFD) and XSETBV. This also avoids mixing with the userspace MSR machinery which is tied to KVM_EXIT_X86_WRMSR today.

Also introduce a new MSR return type (KVM_MSR_RET_USERSPACE). Currently MSR emulation returns to userspace only upon error or per certain filtering rules via the userspace MSR mechinary. This new return type indicates that emulation of certain MSR has its own specific reason to bounce to userspace.

IA32_XFD is updated in two ways:

- If reallocation is not required, the emulation code directly updates guest_fpu::xfd and then calls xfd_update_state() to update IA32_XFD and per-cpu cache;

- If reallocation is triggered, above updates are completed as

    part of the fpstate reallocation process if succeeds;

RDMSR to IA32_XFD is not intercepted. fpu_swap_kvm_fpstate() ensures the guest XFD value loaded into MSR before re-entering the guest. Just save an unnecessary VM-exit here

---
 arch/x86/kvm/vmx/vmx.c   |  8 +++++++
 arch/x86/kvm/x86.c       | 48 ++++++++++++++++++++++++++++++++++++++++
 arch/x86/kvm/x86.h       |  1 +
 include/uapi/linux/kvm.h |  1 +
 4 files changed, 58 insertions(+)
arch/x86/kvm/vmx/vmx.c +8 -0
index 70d86ffbccf7..971d60980d5b 100644
@@ -7141,6 +7141,11 @@ static void update_intel_pt_cfg(struct kvm_vcpu *vcpu)
 		vmx->pt_desc.ctl_bitmask &= ~(0xfULL << (32 + i * 4));
 }
 
+static void vmx_update_intercept_xfd(struct kvm_vcpu *vcpu)
+{
+	vmx_set_intercept_for_msr(vcpu, MSR_IA32_XFD, MSR_TYPE_R, false);
+}
+
 static void vmx_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu)
 {
 	struct vcpu_vmx *vmx = to_vmx(vcpu);
@@ -7181,6 +7186,9 @@ static void vmx_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu)
 		}
 	}
 
+	if (cpu_feature_enabled(X86_FEATURE_XFD) && guest_cpuid_has(vcpu, X86_FEATURE_XFD))
+		vmx_update_intercept_xfd(vcpu);
+
 	set_cr4_guest_host_mask(vmx);
 
 	vmx_write_encls_bitmap(vcpu, NULL);
arch/x86/kvm/x86.c +48 -0

60 lines not on this page · raw message

arch/x86/kvm/x86.h +1 -0
index 24a323980146..446ffa8c7804 100644
@@ -460,6 +460,7 @@ bool kvm_msr_allowed(struct kvm_vcpu *vcpu, u32 index, u32 type);
  */
 #define  KVM_MSR_RET_INVALID	2	/* in-kernel MSR emulation #GP condition */
 #define  KVM_MSR_RET_FILTERED	3	/* #GP due to userspace MSR filter */
+#define  KVM_MSR_RET_USERSPACE	4	/* Userspace handling */
 
 #define __cr4_reserved_bits(__cpu_has, __c)             \
 ({                                                      \
include/uapi/linux/kvm.h +1 -0
index 1daa45268de2..0c7b301c7254 100644
@@ -270,6 +270,7 @@ struct kvm_xen_exit {
 #define KVM_EXIT_X86_BUS_LOCK     33
 #define KVM_EXIT_XEN              34
 #define KVM_EXIT_RISCV_SBI        35
+#define KVM_EXIT_FPU_REALLOC      36
 
 /* For KVM_EXIT_INTERNAL_ERROR */
 /* Emulate instruction failed. */
maintainer
#24 lore raw

Re: [PATCH 10/19] kvm: x86: Emulate WRMSR of guest IA32_XFD

First, the MSR should be added to msrs_to_save_all and kvm_cpu_cap_has(X86_FEATURE_XFD) should be checked in kvm_init_msr_list.

It seems that RDMSR support is missing, too.

More important, please include:

- documentation for the new KVM_EXIT_* value

- a selftest that explains how userspace should react to it.

This is a strong requirement for any new API (the first has been for years; but the latter is also almost always respected these days). This series should not have been submitted without documentation.

Also:

arch/x86/kvm/x86.c int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · line 3708

Quoted 3 lines · Yang Zhong
arch/x86/kvm/x86.c +2 -0
+		if (!guest_cpuid_has(vcpu, X86_FEATURE_XFD))
+			return 1;

This should allow msr->host_initiated always (even if XFD is not part of CPUID). However, if XFD is nonzero and kvm_check_guest_realloc_fpstate returns true, then it should return 1.

The selftest should also cover using KVM_GET_MSR/KVM_SET_MSR.

arch/x86/kvm/x86.c int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · line 3714

Quoted 5 lines
arch/x86/kvm/x86.c +5 -0
+		/* Setting unsupported bits causes #GP */
+		if (~XFEATURE_MASK_USER_DYNAMIC & data) {
+			kvm_inject_gp(vcpu, 0);
+			break;
+		}

This should check

	if (data & ~(XFEATURE_MASK_USER_DYNAMIC &
		    vcpu->arch.guest_supported_xcr0))

instead.

Paolo

#25 lore raw

RE: [PATCH 10/19] kvm: x86: Emulate WRMSR of guest IA32_XFD

Quoted 10 lines · Paolo Bonzini

Also:

arch/x86/kvm/x86.c int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · line 3708

Quoted 3 lines · Yang Zhong
arch/x86/kvm/x86.c +2 -0
+		if (!guest_cpuid_has(vcpu, X86_FEATURE_XFD))
+			return 1;

This should allow msr->host_initiated always (even if XFD is not part of CPUID).

Thanks Paolo.

msr->host_initiated handling would be added in next version.

I'd like to ask why always allow msr->host_initiated even if XFD is not part of CPUID, although guest doesn't care that MSR? We found some MSRs
(e.g. MSR_AMD64_OSVW_STATUS and MSR_AMD64_OSVW_ID_LENGTH )
are specially handled so would like to know the consideration of allowing msr->host_initiated.

if (!msr_info->host_initiated && !guest_cpuid_has(vcpu, X86_FEATURE_XFD))

        return 1;

However, if XFD is nonzero and kvm_check_guest_realloc_fpstate

Quoted 2 lines

returns true, then it should return 1.

If XFD is nonzero, kvm_check_guest_realloc_fpstate() won't return true. So may not need this check here?

Thanks,
Jing

Quoted 2 lines

Paolo

maintainer
#26 lore raw

Re: [PATCH 10/19] kvm: x86: Emulate WRMSR of guest IA32_XFD

Quoted 15 lines · On 12/13/21 08:51, Liu, Jing2 wrote:
Quoted 10 lines · Paolo Bonzini

Also:

arch/x86/kvm/x86.c int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · line 3708

Quoted 3 lines · Yang Zhong
arch/x86/kvm/x86.c +2 -0
+		if (!guest_cpuid_has(vcpu, X86_FEATURE_XFD))
+			return 1;

This should allow msr->host_initiated always (even if XFD is not part of CPUID).

Thanks Paolo.

msr->host_initiated handling would be added in next version.

Quoted 8 lines

I'd like to ask why always allow msr->host_initiated even if XFD is not part of CPUID, although guest doesn't care that MSR? We found some MSRs
(e.g. MSR_AMD64_OSVW_STATUS and MSR_AMD64_OSVW_ID_LENGTH )
are specially handled so would like to know the consideration of allowing msr->host_initiated.

if (!msr_info->host_initiated && !guest_cpuid_has(vcpu, X86_FEATURE_XFD))

         return 1;

Because it's simpler if userspace can just take the entire list from KVM_GET_MSR_INDEX_LIST and pass it to KVM_GET/SET_MSR. See for example vcpu_save_state and vcpu_load_state in tools/testing/selftests/kvm/lib/x86_64/processor.c.

Quoted 5 lines
Quoted 2 lines

However, if XFD is nonzero and kvm_check_guest_realloc_fpstate
returns true, then it should return 1.

If XFD is nonzero, kvm_check_guest_realloc_fpstate() won't return true. So may not need this check here?

It can't for now, because there's a single dynamic feature, but here:

+ if ((xfd & xcr0) != xcr0) {
+ u64 request = (xcr0 ^ xfd) & xcr0;
+ struct fpu_guest *guest_fpu = &vcpu->arch.guest_fpu;
+
+ /*
+ * If requested features haven't been enabled, update
+ * the request bitmap and tell the caller to request
+ * dynamic buffer reallocation.
+ */
+ if ((guest_fpu->user_xfeatures & request) != request) {
+ vcpu->arch.guest_fpu.realloc_request = request;
+ return true;
+ }
+ }

it is certainly possible to return true with nonzero XFD.

Paolo

author
#27 lore raw

Re: [PATCH 10/19] kvm: x86: Emulate WRMSR of guest IA32_XFD

In reply to Paolo Bonzini
Quoted 28 lines · Paolo Bonzini

First, the MSR should be added to msrs_to_save_all and kvm_cpu_cap_has(X86_FEATURE_XFD) should be checked in kvm_init_msr_list.

It seems that RDMSR support is missing, too.

More important, please include:

- documentation for the new KVM_EXIT_* value

- a selftest that explains how userspace should react to it.

This is a strong requirement for any new API (the first has been for years; but the latter is also almost always respected these days). This series should not have been submitted without documentation.

Also:

arch/x86/kvm/x86.c int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · line 3708

Quoted 3 lines · Yang Zhong
arch/x86/kvm/x86.c +2 -0
+		if (!guest_cpuid_has(vcpu, X86_FEATURE_XFD))
+			return 1;

This should allow msr->host_initiated always (even if XFD is not part of CPUID). However, if XFD is nonzero and kvm_check_guest_realloc_fpstate returns true, then it should return 1.

Quoted 2 lines

The selftest should also cover using KVM_GET_MSR/KVM_SET_MSR.

Paolo, Seems we do not need new KVM_EXIT_* again from below thomas' new patchset:
git://git.kernel.org/pub/scm/linux/kernel/git/people/tglx/devel.git x86/fpu-kvm

So the selftest stll need support KVM_GET_MSR/KVM_SET_MSR for MSR_IA32_XFD and MSR_IA32_XFD_ERR? If yes, we only do some read/write test with vcpu_set_msr()/ vcpu_get_msr() from new selftest tool? or do wrmsr from guest side and check this value from selftest side?

I checked some msr selftest reference code, tsc_msrs_test.c, which maybe better for this reference. If you have better suggestion, please share it to me. thanks!

Yang

maintainer
#28 lore raw

Re: [PATCH 10/19] kvm: x86: Emulate WRMSR of guest IA32_XFD

Quoted 7 lines · Yang Zhong

Paolo, Seems we do not need new KVM_EXIT_* again from below thomas' new patchset:
git://git.kernel.org/pub/scm/linux/kernel/git/people/tglx/devel.git x86/fpu-kvm

So the selftest stll need support KVM_GET_MSR/KVM_SET_MSR for MSR_IA32_XFD and MSR_IA32_XFD_ERR? If yes, we only do some read/write test with vcpu_set_msr()/ vcpu_get_msr() from new selftest tool? or do wrmsr from guest side and check this value from selftest side?

You can write a test similar to state_test.c to cover XCR0, XFD and the new XSAVE extensions. The test can:

- initialize AMX and write a nonzero value to XFD

- load a matrix into TMM0

- check that #NM is delivered (search for vm_install_exception_handler) and that XFD_ERR is correct

- write 0 to XFD

- load again the matrix, and check that #NM is not delivered

- store it back into memory

- compare it with the original data

All of this can be done with a full save&restore after every step (though I suggest that you first get it working without save&restore, the relevant code in state_test.c is easy to identify and comment out).

You will have to modify vcpu_load_state, so that it does first KVM_SET_MSRS, then KVM_SET_XCRS, then KVM_SET_XSAVE. See patch below.

Paolo

Quoted 2 lines

I checked some msr selftest reference code, tsc_msrs_test.c, which maybe better for this reference. If you have better suggestion, please share it to me. thanks!

------------------ 8< -----------------
From: Paolo Bonzini <[email protected]>
Subject: [PATCH] selftest: kvm: Reorder vcpu_load_state steps for AMX

For AMX support it is recommended to load XCR0 after XFD, so that KVM does not see XFD=0, XCR=1 for a save state that will eventually be disabled (which would lead to premature allocation of the space required for that save state).

It is also required to load XSAVE data after XCR0 and XFD, so that KVM can trigger allocation of the extra space required to store AMX state.

Adjust vcpu_load_state to obey these new requirements.

tools/testing/selftests/kvm/lib/x86_64/processor.c +10 -10
index 82c39db91369..d805f63f7203 100644
@@ -1157,16 +1157,6 @@ void vcpu_load_state(struct kvm_vm *vm, uint32_t vcpuid, struct kvm_x86_state *s
 	struct vcpu *vcpu = vcpu_find(vm, vcpuid);
 	int r;
 
-	r = ioctl(vcpu->fd, KVM_SET_XSAVE, &state->xsave);
-        TEST_ASSERT(r == 0, "Unexpected result from KVM_SET_XSAVE, r: %i",
-                r);
-
-	if (kvm_check_cap(KVM_CAP_XCRS)) {
-		r = ioctl(vcpu->fd, KVM_SET_XCRS, &state->xcrs);
-		TEST_ASSERT(r == 0, "Unexpected result from KVM_SET_XCRS, r: %i",
-			    r);
-	}
-
 	r = ioctl(vcpu->fd, KVM_SET_SREGS, &state->sregs);
         TEST_ASSERT(r == 0, "Unexpected result from KVM_SET_SREGS, r: %i",
                 r);
@@ -1175,6 +1165,16 @@ void vcpu_load_state(struct kvm_vm *vm, uint32_t vcpuid, struct kvm_x86_state *s
         TEST_ASSERT(r == state->msrs.nmsrs, "Unexpected result from KVM_SET_MSRS, r: %i (failed at %x)",
                 r, r == state->msrs.nmsrs ? -1 : state->msrs.entries[r].index);
 
+	if (kvm_check_cap(KVM_CAP_XCRS)) {
+		r = ioctl(vcpu->fd, KVM_SET_XCRS, &state->xcrs);
+		TEST_ASSERT(r == 0, "Unexpected result from KVM_SET_XCRS, r: %i",
+			    r);
+	}
+
+	r = ioctl(vcpu->fd, KVM_SET_XSAVE, &state->xsave);
+        TEST_ASSERT(r == 0, "Unexpected result from KVM_SET_XSAVE, r: %i",
+                r);
+
 	r = ioctl(vcpu->fd, KVM_SET_VCPU_EVENTS, &state->events);
         TEST_ASSERT(r == 0, "Unexpected result from KVM_SET_VCPU_EVENTS, r: %i",
                 r);
#29 lore raw

Re: [PATCH 10/19] kvm: x86: Emulate WRMSR of guest IA32_XFD

In reply to Yang Zhong

Review

Quoted 12 lines · Yang Zhong
(quoted diff) +12 -0
+
+		/*
+		 * Update IA32_XFD to the guest value so #NM can be
+		 * raised properly in the guest. Instead of directly
+		 * writing the MSR, call a helper to avoid breaking
+		 * per-cpu cached value in fpu core.
+		 */
+		fpregs_lock();
+		current->thread.fpu.fpstate->xfd = data;
+		xfd_update_state(current->thread.fpu.fpstate);
+		fpregs_unlock();
+		break;

Now looking at the actual callsite the previous patch really should be something like the below. Why?

It preserves the inline which allows the compiler to generate better code in the other hotpathes and it keeps the FPU internals to the core code. Hmm?

Thanks,

tglx

arch/x86/include/asm/fpu/api.h +2 -0
@@ -125,8 +125,10 @@ DECLARE_PER_CPU(struct fpu *, fpu_fpregs
 /* Process cleanup */
 #ifdef CONFIG_X86_64
 extern void fpstate_free(struct fpu *fpu);
+extern void fpu_update_xfd_state(u64 xfd);
 #else
 static inline void fpstate_free(struct fpu *fpu) { }
+static inline void fpu_update_xfd_state(u64 xfd) { }
 #endif
 
 /* fpstate-related functions which are exported to KVM */
arch/x86/kernel/fpu/core.c +13 -0
@@ -322,6 +322,19 @@ int fpu_swap_kvm_fpstate(struct fpu_gues
 }
 EXPORT_SYMBOL_GPL(fpu_swap_kvm_fpstate);
 
+#ifdef CONFIG_X86_64
+void fpu_update_xfd_state(u64 xfd)
+{
+	struct fpstate *fps = current->thread.fpu.fpstate;
+
+	fpregs_lock();
+	fps->xfd = xfd;
+	xfd_update_state(fps);
+	fpregs_unlock();
+}
+EXPORT_SYMBOL_GPL(fpu_update_xfd_state);
+#endif
+
 void fpu_copy_guest_fpstate_to_uabi(struct fpu_guest *gfpu, void *buf,
 				    unsigned int size, u32 pkru)
 {
maintainer
#30 lore raw

Re: [PATCH 10/19] kvm: x86: Emulate WRMSR of guest IA32_XFD

In reply to Yang Zhong

arch/x86/kvm/x86.c int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · line 3738

Quoted 8 lines · Yang Zhong
arch/x86/kvm/x86.c +8 -0
+		/*
+		 * Update IA32_XFD to the guest value so #NM can be
+		 * raised properly in the guest. Instead of directly
+		 * writing the MSR, call a helper to avoid breaking
+		 * per-cpu cached value in fpu core.
+		 */
+		fpregs_lock();
+		current->thread.fpu.fpstate->xfd = data;

This is wrong, it should be written in vcpu->arch.guest_fpu.

arch/x86/kvm/x86.c int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · line 3739

Quoted 1 line
arch/x86/kvm/x86.c +1 -0
+		xfd_update_state(current->thread.fpu.fpstate);

This is okay though, so that KVM_SET_MSR will not write XFD and WRMSR will.

That said, I think xfd_update_state should not have an argument. current->thread.fpu.fpstate->xfd is the only fpstate that should be synced with the xfd_state per-CPU variable.

Paolo

Quoted 1 line
arch/x86/kvm/x86.c +1 -0
+		fpregs_unlock();
#31 lore raw

Re: [PATCH 10/19] kvm: x86: Emulate WRMSR of guest IA32_XFD

Quoted 12 lines · Paolo Bonzini

arch/x86/kvm/x86.c int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · line 3738

Quoted 8 lines · Yang Zhong
arch/x86/kvm/x86.c +8 -0
+		/*
+		 * Update IA32_XFD to the guest value so #NM can be
+		 * raised properly in the guest. Instead of directly
+		 * writing the MSR, call a helper to avoid breaking
+		 * per-cpu cached value in fpu core.
+		 */
+		fpregs_lock();
+		current->thread.fpu.fpstate->xfd = data;

This is wrong, it should be written in vcpu->arch.guest_fpu.

Quoted 8 lines

arch/x86/kvm/x86.c int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · line 3739

Quoted 1 line
arch/x86/kvm/x86.c +1 -0
+		xfd_update_state(current->thread.fpu.fpstate);

This is okay though, so that KVM_SET_MSR will not write XFD and WRMSR will.

That said, I think xfd_update_state should not have an argument. current->thread.fpu.fpstate->xfd is the only fpstate that should be synced with the xfd_state per-CPU variable.

I'm looking into this right now. The whole restore versus runtime thing needs to be handled differently.

Thanks,

tglx

#32 lore raw

Re: [PATCH 10/19] kvm: x86: Emulate WRMSR of guest IA32_XFD

Paolo,

Quoted 7 lines · Thomas Gleixner
Quoted 3 lines · Paolo Bonzini

That said, I think xfd_update_state should not have an argument. current->thread.fpu.fpstate->xfd is the only fpstate that should be synced with the xfd_state per-CPU variable.

I'm looking into this right now. The whole restore versus runtime thing needs to be handled differently.

We need to look at different things here:

1) XFD MSR write emulation

2) XFD MSR synchronization when write emulation is disabled

3) Guest restore

#1 and #2 are in the context of vcpu_run() and

vcpu->arch.guest_fpu.fpstate == current->thread.fpu.fpstate

while #3 has:

vcpu->arch.guest_fpu.fpstate != current->thread.fpu.fpstate

#2 is only updating fpstate->xfd and the per CPU shadow.

So the state synchronization wants to be something like this:

void fpu_sync_guest_xfd_state(void)

{
	struct fpstate *fps = current->thread.fpu.fpstate;

	lockdep_assert_irqs_disabled();
	if (fpu_state_size_dynamic()) {
		rdmsrl(MSR_IA32_XFD, fps->xfd);
		__this_cpu_write(xfd_state, fps->xfd);
	}
}

EXPORT_SYMBOL_GPL(fpu_sync_guest_xfd_state);

No wrmsrl() because the MSR is already up do date. The important part is that fpstate->xfd and the shadow state are updated so that after reenabling preemption the context switch FPU logic works correctly.

#1 and #3 can trigger a reallocation of guest_fpu.fpstate and can fail. But this is also true for XSETBV emulation and XCR0 restore.

For #1 modifying fps->xfd in the KVM code before calling into the FPU code is just _wrong_ because if the guest removes the XFD restriction then it must be ensured that the buffer is sized correctly _before_ this is updated.

For #3 it's not really important, but I still try to wrap my head around the whole picture vs. XCR0.

There are two options:

1) Require strict ordering of XFD and XCR0 update to avoid pointless buffer expansion, i.e. XFD before XCR0.

Because if XCR0 is updated while guest_fpu->fpstate.xfd is still in init state (0) and XCR0 contains extended features, then the buffer would be expanded because XFD does not mask the extended features out. When XFD is restored with a non-zero value, it's too late already.

2) Ignore buffer expansion up to the point where XSTATE restore happens

     and evaluate guest XCR0 and guest_fpu->fpstate.xfd there.

I'm leaning towards #1 because that means we have exactly _ONE_ place where we need to deal with buffer expansion. If Qemu gets the ordering wrong it wastes memory per vCPU, *shrug*.

Thanks,

tglx

#33 lore raw

RE: [PATCH 10/19] kvm: x86: Emulate WRMSR of guest IA32_XFD

Hi, Thomas,

Quoted 14 lines

From: Thomas Gleixner <[email protected]>
Sent: Tuesday, December 14, 2021 5:23 AM

Paolo,

Quoted 7 lines · Thomas Gleixner
Quoted 3 lines · Paolo Bonzini

That said, I think xfd_update_state should not have an argument. current->thread.fpu.fpstate->xfd is the only fpstate that should be synced with the xfd_state per-CPU variable.

I'm looking into this right now. The whole restore versus runtime thing needs to be handled differently.

After looking at your series, I think it missed Paolo's comment about changing xfd_update_state() to accept no argument.

Thanks
Kevin

author
#34 lore raw

PATCH 11/19 kvm: x86: Check fpstate reallocation in XSETBV emulation

In reply to Yang Zhong

XSETBV allows the software to write the extended control register XCR0, thus its emulation handler also needs to check fpstate reallocation when the changed XCR0 value enables certain dynamically-enabled features.

---
 arch/x86/kvm/x86.c | 9 +++++++++
 1 file changed, 9 insertions(+)
arch/x86/kvm/x86.c +9 -0
index c83887cb55ee..b195f4fa888f 100644
@@ -1028,6 +1028,15 @@ int kvm_emulate_xsetbv(struct kvm_vcpu *vcpu)
 		return 1;
 	}
 
+	if (guest_cpuid_has(vcpu, X86_FEATURE_XFD)) {
+		if (kvm_check_guest_realloc_fpstate(vcpu, vcpu->arch.guest_fpu.fpstate->xfd)) {
+			vcpu->run->exit_reason = KVM_EXIT_FPU_REALLOC;
+			vcpu->arch.complete_userspace_io =
+				kvm_skip_emulated_instruction;
+			return 0;
+		}
+	}
+
 	return kvm_skip_emulated_instruction(vcpu);
 }
 EXPORT_SYMBOL_GPL(kvm_emulate_xsetbv);
author
#35 lore raw

PATCH 12/19 x86/fpu: Prepare KVM for bringing XFD state back in-sync

In reply to Yang Zhong

From: Thomas Gleixner <[email protected]>

Guest may toggle IA32_XFD in high frequency as it is part of the fpstate information (features, sizes, xfd) and swapped in task context switch.

To minimize the trap overhead of writes to this MSR, one optimization is to allow guest direct write thus eliminate traps. However MSR passthrough implies that guest_fpstate::xfd and per-cpu xfd cache might be out of sync with the current IA32_XFD value by the guest.

This suggests KVM needs to re-sync guest_fpstate::xfd and per-cpu cache with IA32_XFD before the vCPU thread might be preempted or interrupted.

This patch provides a helper function for the re-sync purpose.

---
(To Thomas): the original name kvm_update_guest_xfd_state() in your sample code is renamed to xfd_sync_state() in this patch. In concept it is a general helper to bring software values in-sync with the MSR value after they become out-of-sync. KVM is just the first out-of-sync usage on this helper, so a neutral name may make more sense. But if you prefer to the original name we can also change back.

 arch/x86/include/asm/fpu/xstate.h |  2 ++
 arch/x86/kernel/fpu/xstate.c      | 14 ++++++++++++++
 2 files changed, 16 insertions(+)
arch/x86/include/asm/fpu/xstate.h +2 -0
index cd3dd170e23a..c8b51d34daab 100644
@@ -129,4 +129,6 @@ static __always_inline __pure bool fpu_state_size_dynamic(void)
 }
 #endif
 
+extern void xfd_sync_state(void);
+
 #endif
arch/x86/kernel/fpu/xstate.c +14 -0
index 3c39789deeb9..a5656237a763 100644
@@ -1762,11 +1762,25 @@ void xfd_update_state(struct fpstate *fpstate)
 	}
 }
 EXPORT_SYMBOL_GPL(xfd_update_state);
+
+/* Bring software state in sync with the current MSR value */
+void xfd_sync_state(void)
+{
+	if (fpu_state_size_dynamic()) {
+		u64 xfd;
+
+		rdmsrl(MSR_IA32_XFD, xfd);
+		current->thread.fpu.fpstate->xfd = xfd;
+		__this_cpu_write(xfd_state, xfd);
+	}
+}
+EXPORT_SYMBOL_GPL(xfd_sync_state);
 #else /* CONFIG_X86_64 */
 static inline int xstate_request_perm(unsigned long idx, bool guest)
 {
 	return -EPERM;
 }
+void xfd_sync_state(void) {}
 #endif  /* !CONFIG_X86_64 */
 
 inline u64 xstate_get_guest_group_perm(void)
#36 lore raw

Re: [PATCH 12/19] x86/fpu: Prepare KVM for bringing XFD state back in-sync

Quoted 10 lines · Yang Zhong

From: Thomas Gleixner <[email protected]>

Guest may toggle IA32_XFD in high frequency as it is part of the fpstate information (features, sizes, xfd) and swapped in task context switch.

To minimize the trap overhead of writes to this MSR, one optimization is to allow guest direct write thus eliminate traps. However MSR passthrough implies that guest_fpstate::xfd and per-cpu xfd cache might be out of sync with the current IA32_XFD value by the guest.

Quoted 4 lines

This suggests KVM needs to re-sync guest_fpstate::xfd and per-cpu cache with IA32_XFD before the vCPU thread might be preempted or interrupted.

This patch provides a helper function for the re-sync purpose.

Provide a ....

Quoted 11 lines

---
(To Thomas): the original name kvm_update_guest_xfd_state() in your sample code is renamed to xfd_sync_state() in this patch. In concept it is a general helper to bring software values in-sync with the MSR value after they become out-of-sync. KVM is just the first out-of-sync usage on this helper, so a neutral name may make more sense. But if you prefer to the original name we can also change back.

There is no need for a general helper, really.

It's KVM specific and should go into KVM section in core.c next to the other thing vs. the XFD update.

Thanks,

tglx

author
#37 lore raw

PATCH 13/19 kvm: x86: Disable WRMSR interception for IA32_XFD on demand

In reply to Yang Zhong

From: Jing Liu <[email protected]>

Always intercepting IA32_XFD causes non-negligible overhead when this register is updated frequently in the guest.

Disable WRMSR interception to IA32_XFD after fpstate reallocation is completed. There are three options for when to disable the interception:

1) When emulating the 1st WRMSR which requires reallocation, disable interception before exiting to userapce with the assumption that the userspace VMM should not bounch back to the kernel if reallocation fails. However it's not good to design kernel based on application behavior. If due to bug the vCPU thread comes back to the kernel after reallocation fails, XFD passthrough may lead to host memory corruption when doing XSAVES for guest fpstate which has a smaller size than what guest XFD allows.

2) Disable interception when coming back from the userspace VMM (for the 1st WRMSR which triggers reallocation). Re-check whether fpstate size can serve the new guest XFD value. Disable interception only when the check succeeds. This requires KVM to store guest XFD value in some place and then compare it to guest_fpu::user_xfeatures in the completion handler.

3) Disable interception at the 2nd WRMSR which enables dynamic XSTATE features. If guest_fpu::user_xfeatures already includes bits for dynamic features set in guest XFD value, disable interception.

Currently 3) is implemented, with a flow like below:

(G) WRMSR(IA32_XFD) which enables AMX for the FIRST time --trap to host--
(HK) Emulate WRMSR and find fpstate size too small (HK) Reallocate fpstate
--exit to userspace--
(HU) do nothing
--back to kernel via kvm_run--
(HK) complete WRMSR emulation
--enter guest--
(G) do something
(G) WRMSR(IA32_XFD) which disables AMX
--trap to host--
(HK) Emulate WRMSR and disable AMX in IA32_XFD --enter guest--
(G) do something
(G) WRMSR(IA32_XFD) which enables AMX for the SECOND time --trap to host--
(HK) Emulate WRMSR and find fpstate size sufficient for AMX (HK) Disable WRMSR interception for IA32_XFD --enter guest--
(G) WRMSR(IA32_XFD)
(G) WRMSR(IA32_XFD)
(G) WRMSR(IA32_XFD)
...

After disabling WRMSR interception, the guest directly updates IA32_XFD which becomes out-of-sync with the host-side software state (guest_fpstate::xfd and per-cpu xfd cache). This requires KVM to call xfd_sync_state() to bring the software state in sync with IA32_XFD register after VM-exit (before preemption happens or exiting to userspace).

p.s. We have confirmed that SDM is being revised to say that when setting IA32_XFD[18] the AMX register state is not guaranteed to be preserved. This clarification avoids adding mess for a creative guest which sets IA32_XFD[18]=1 before saving active AMX state to its own storage.

---
 arch/x86/include/asm/kvm-x86-ops.h |  1 +
 arch/x86/include/asm/kvm_host.h    |  2 ++
 arch/x86/kvm/vmx/vmx.c             | 10 ++++++++++
 arch/x86/kvm/vmx/vmx.h             |  2 +-
 arch/x86/kvm/x86.c                 |  7 +++++++
 5 files changed, 21 insertions(+), 1 deletion(-)
arch/x86/include/asm/kvm-x86-ops.h +1 -0
index cefe1d81e2e8..60c27f9990e9 100644
@@ -30,6 +30,7 @@ KVM_X86_OP(update_exception_bitmap)
 KVM_X86_OP(get_msr)
 KVM_X86_OP(set_msr)
 KVM_X86_OP(get_segment_base)
+KVM_X86_OP_NULL(set_xfd_passthrough)
 KVM_X86_OP(get_segment)
 KVM_X86_OP(get_cpl)
 KVM_X86_OP(set_segment)
arch/x86/include/asm/kvm_host.h +2 -0
index 6ac61f85e07b..7c97cc1fea89 100644
@@ -640,6 +640,7 @@ struct kvm_vcpu_arch {
 	u64 smi_count;
 	bool tpr_access_reporting;
 	bool xsaves_enabled;
+	bool xfd_out_of_sync;
 	u64 ia32_xss;
 	u64 microcode_version;
 	u64 arch_capabilities;
@@ -1328,6 +1329,7 @@ struct kvm_x86_ops {
 	void (*update_exception_bitmap)(struct kvm_vcpu *vcpu);
 	int (*get_msr)(struct kvm_vcpu *vcpu, struct msr_data *msr);
 	int (*set_msr)(struct kvm_vcpu *vcpu, struct msr_data *msr);
+	void (*set_xfd_passthrough)(struct kvm_vcpu *vcpu);
 	u64 (*get_segment_base)(struct kvm_vcpu *vcpu, int seg);
 	void (*get_segment)(struct kvm_vcpu *vcpu,
 			    struct kvm_segment *var, int seg);
arch/x86/kvm/vmx/vmx.c +10 -0
index 971d60980d5b..6198b13c4846 100644
@@ -160,6 +160,7 @@ static u32 vmx_possible_passthrough_msrs[MAX_POSSIBLE_PASSTHROUGH_MSRS] = {
 	MSR_FS_BASE,
 	MSR_GS_BASE,
 	MSR_KERNEL_GS_BASE,
+	MSR_IA32_XFD,
 #endif
 	MSR_IA32_SYSENTER_CS,
 	MSR_IA32_SYSENTER_ESP,
@@ -1924,6 +1925,14 @@ static u64 vcpu_supported_debugctl(struct kvm_vcpu *vcpu)
 	return debugctl;
 }
 
+#ifdef CONFIG_X86_64
+static void vmx_set_xfd_passthrough(struct kvm_vcpu *vcpu)
+{
+	vmx_disable_intercept_for_msr(vcpu, MSR_IA32_XFD, MSR_TYPE_RW);
+	vcpu->arch.xfd_out_of_sync = true;
+}
+#endif
+
 /*
  * Writes msr value into the appropriate "register".
  * Returns 0 on success, non-0 otherwise.
@@ -7657,6 +7666,7 @@ static struct kvm_x86_ops vmx_x86_ops __initdata = {
 #ifdef CONFIG_X86_64
 	.set_hv_timer = vmx_set_hv_timer,
 	.cancel_hv_timer = vmx_cancel_hv_timer,
+	.set_xfd_passthrough = vmx_set_xfd_passthrough,
 #endif
 
 	.setup_mce = vmx_setup_mce,
arch/x86/kvm/vmx/vmx.h +1 -1
index 4df2ac24ffc1..bf9d3051cd6c 100644
@@ -340,7 +340,7 @@ struct vcpu_vmx {
 	struct lbr_desc lbr_desc;
 
 	/* Save desired MSR intercept (read: pass-through) state */
-#define MAX_POSSIBLE_PASSTHROUGH_MSRS	13
+#define MAX_POSSIBLE_PASSTHROUGH_MSRS	14
 	struct {
 		DECLARE_BITMAP(read, MAX_POSSIBLE_PASSTHROUGH_MSRS);
 		DECLARE_BITMAP(write, MAX_POSSIBLE_PASSTHROUGH_MSRS);
arch/x86/kvm/x86.c +7 -0
index b195f4fa888f..d127b229dd29 100644
@@ -974,6 +974,10 @@ bool kvm_check_guest_realloc_fpstate(struct kvm_vcpu *vcpu, u64 xfd)
 			vcpu->arch.guest_fpu.realloc_request = request;
 			return true;
 		}
+
+		/* Disable WRMSR interception if possible */
+		if (kvm_x86_ops.set_xfd_passthrough)
+			static_call(kvm_x86_set_xfd_passthrough)(vcpu);
 	}
 
 	return false;
@@ -10002,6 +10006,9 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 	if (hw_breakpoint_active())
 		hw_breakpoint_restore();
 
+	if (vcpu->arch.xfd_out_of_sync)
+		xfd_sync_state();
+
 	vcpu->arch.last_vmentry_cpu = vcpu->cpu;
 	vcpu->arch.last_guest_tsc = kvm_read_l1_tsc(vcpu, rdtsc());
 
#38 lore raw

RE: [PATCH 13/19] kvm: x86: Disable WRMSR interception for IA32_XFD on demand

Quoted 19 lines · Yang Zhong

From: Jing Liu <[email protected]>

Always intercepting IA32_XFD causes non-negligible overhead when this register is updated frequently in the guest.

Disable WRMSR interception to IA32_XFD after fpstate reallocation is completed. There are three options for when to disable the interception:

1) When emulating the 1st WRMSR which requires reallocation, disable interception before exiting to userapce with the assumption that the userspace VMM should not bounch back to the kernel if reallocation fails. However it's not good to design kernel based on application behavior. If due to bug the vCPU thread comes back to the kernel after reallocation fails, XFD passthrough may lead to host memory corruption when doing XSAVES for guest fpstate which has a smaller size than what guest XFD allows.

Quoted 6 lines

2) Disable interception when coming back from the userspace VMM (for the 1st WRMSR which triggers reallocation). Re-check whether fpstate size can serve the new guest XFD value. Disable interception only when the check succeeds. This requires KVM to store guest XFD value in some place and then compare it to guest_fpu::user_xfeatures in the completion handler.

For option 2), we are considering that fpstate->size can be used to indicate if reallocation is successful. Because once one of the XFD features (today, it's AMX) is enabled, kernel need reallocate full size, otherwise, KVM has no chance to reallocate for other XFD features later since it's non-trapped (to avoid WRMSR VM EXITs due to guest toggling XFD).

Then KVM doesn't need to store guest XFD value in some place. And kernel fpu core may need an API to tell guest permitted size for KVM.

Thanks,
Jing

Quoted 6 lines

3) Disable interception at the 2nd WRMSR which enables dynamic XSTATE features. If guest_fpu::user_xfeatures already includes bits for dynamic features set in guest XFD value, disable interception.

author
#39 lore raw

PATCH 14/19 x86/fpu: Prepare for KVM XFD_ERR handling

In reply to Yang Zhong

From: Jing Liu <[email protected]>

When XFD causes an instruction to generate #NM, IA32_XFD_ERR contains information about which disabled state components are being accessed. The #NM handler is expected to check this information and then enable the state components by clearing IA32_XFD for the faulting task (if having permission).

if the XFD_ERR value generated in guest is consumed/clobbered by the host before the guest itself doing so. This may lead to non-XFD-related #NM treated as XFD #NM in host (due to non-zero value in XFD_ERR), or XFD-related #NM treated as non-XFD #NM in guest (XFD_ERR cleared by the host #NM handler).

This patch provides two helpers to swap the guest XFD_ERR and host XFD_ERR. Where to call them in KVM will be discussed thoroughly in next patch.

The guest XFD_ERR value is saved in fpu_guest::xfd_err. There is no need to save host XFD_ERR because it's always cleared to ZERO by the host #NM handler (which cannot be preempted by a vCPU thread to observe a non-zero value).

The lower two bits in fpu_guest::xfd_err is borrowed for special purposes. The state components (FP and SSE) covered by the two bits are not XSAVE-enabled feature, thus not XFD-enabled either. It's impossible to see hardware setting them in XFD_ERR:

- XFD_ERR_GUEST_DISABLED (bit 0)

Indicate that XFD extension is not exposed to the guest thus no need to save/restore it.

- XFD_ERR_GUEST_SAVED (bit 1)

Indicate fpu_guest::xfd_err already contains a saved value thus no need for duplicated saving (e.g. when the vCPU thread is preempted multiple times before re-enter the guest).

---
 arch/x86/include/asm/fpu/api.h   |  8 ++++++
 arch/x86/include/asm/fpu/types.h | 24 ++++++++++++++++
 arch/x86/kernel/fpu/core.c       | 49 ++++++++++++++++++++++++++++++++
 3 files changed, 81 insertions(+)
arch/x86/include/asm/fpu/api.h +8 -0
index 999d89026be9..c2e8f2172994 100644
@@ -147,6 +147,14 @@ extern bool fpu_alloc_guest_fpstate(struct fpu_guest *gfpu);
 extern void fpu_free_guest_fpstate(struct fpu_guest *gfpu);
 extern int fpu_swap_kvm_fpstate(struct fpu_guest *gfpu, bool enter_guest);
 
+#ifdef CONFIG_X86_64
+extern void fpu_save_guest_xfd_err(struct fpu_guest *guest_fpu);
+extern void fpu_restore_guest_xfd_err(struct fpu_guest *guest_fpu);
+#else
+static inline void fpu_save_guest_xfd_err(struct fpu_guest *guest_fpu) { }
+static inline void fpu_restore_guest_xfd_err(struct fpu_guest *guest_fpu) { }
+#endif
+
 extern void fpu_copy_guest_fpstate_to_uabi(struct fpu_guest *gfpu, void *buf, unsigned int size, u32 pkru);
 extern int fpu_copy_uabi_to_guest_fpstate(struct fpu_guest *gfpu, const void *buf, u64 xcr0, u32 *vpkru);
 
arch/x86/include/asm/fpu/types.h +24 -0
index 861cffca3209..5ee98222c103 100644
@@ -500,6 +500,22 @@ struct fpu {
 	 */
 };
 
+/*
+ * Use @xfd_err:bit0 to indicate whether guest XFD_ERR should be
+ * saved/restored. The x87 state covered by bit 0 is not a
+ * XSAVE-enabled feature, thus is not XFD-enabled either (won't
+ * occur in XFD_ERR).
+ */
+#define XFD_ERR_GUEST_DISABLED		(1 << XFEATURE_FP)
+
+/*
+ * Use @xfd_err:bit1 to indicate the validity of @xfd_err. Used to
+ * avoid duplicated savings in case the vCPU is preempted multiple
+ * times before it re-enters the guest. The SSE state covered by
+ * bit 1 is neither XSAVE-enabled nor XFD-enabled.
+ */
+#define XFD_ERR_GUEST_SAVED		(1 << XFEATURE_SSE)
+
 /*
  * Guest pseudo FPU container
  */
@@ -527,6 +543,14 @@ struct fpu_guest {
 	 */
 	u64				realloc_request;
 
+	/*
+	 * @xfd_err:			save the guest value. bit 0 and bit1
+	 *				have special meaning to indicate the
+	 *				requirement of saving and the validity
+	 *				of the saved value.
+	 */
+	u64				xfd_err;
+
 	/*
 	 * @fpstate:			Pointer to the allocated guest fpstate
 	 */
arch/x86/kernel/fpu/core.c +49 -0

55 lines not on this page · raw message

maintainer
#40 lore raw

Re: [PATCH 14/19] x86/fpu: Prepare for KVM XFD_ERR handling

Quoted 11 lines · Yang Zhong

The guest XFD_ERR value is saved in fpu_guest::xfd_err. There is no need to save host XFD_ERR because it's always cleared to ZERO by the host #NM handler (which cannot be preempted by a vCPU thread to observe a non-zero value).

The lower two bits in fpu_guest::xfd_err is borrowed for special purposes. The state components (FP and SSE) covered by the two bits are not XSAVE-enabled feature, thus not XFD-enabled either. It's impossible to see hardware setting them in XFD_ERR:

Quoted 4 lines

- XFD_ERR_GUEST_DISABLED (bit 0)

Indicate that XFD extension is not exposed to the guest thus no need to save/restore it.

Can this instead just check if xfeatures includes any dynamic-save features?

Paolo

Quoted 5 lines

- XFD_ERR_GUEST_SAVED (bit 1)

Indicate fpu_guest::xfd_err already contains a saved value thus no need for duplicated saving (e.g. when the vCPU thread is preempted multiple times before re-enter the guest).

#41 lore raw

Re: [PATCH 14/19] x86/fpu: Prepare for KVM XFD_ERR handling

In reply to Yang Zhong
Quoted 8 lines · Yang Zhong
arch/x86/kernel/fpu/core.c +2 -0
@@ -322,6 +322,55 @@ int fpu_swap_kvm_fpstate(struct fpu_guest *guest_fpu, bool enter_guest)
 }
 EXPORT_SYMBOL_GPL(fpu_swap_kvm_fpstate);
 
+#ifdef CONFIG_X86_64
+void fpu_save_guest_xfd_err(struct fpu_guest *guest_fpu)

arch/x86/kernel/fpu/core.c int fpu_swap_kvm_fpstate(struct fpu_guest *guest_fpu, bool enter_guest) · line 336

Quoted 10 lines
arch/x86/kernel/fpu/core.c +10 -0
+{
+	if (guest_fpu->xfd_err & XFD_ERR_GUEST_DISABLED)
+		return;
+
+	/* A non-zero value indicates guest XFD_ERR already saved */
+	if (guest_fpu->xfd_err)
+		return;
+
+	/* Guest XFD_ERR must be saved before switching to host fpstate */
+	WARN_ON_ONCE(!current->thread.fpu.fpstate->is_guest);

Warn and proceed?

Quoted 23 lines
arch/x86/kernel/fpu/core.c +23 -0
+	rdmsrl(MSR_IA32_XFD_ERR, guest_fpu->xfd_err);
+
+	/*
+	 * Restore to the host value if guest xfd_err is non-zero.
+	 * Except in #NM handler, all other places in the kernel
+	 * should just see xfd_err=0. So just restore to 0.
+	 */
+	if (guest_fpu->xfd_err)
+		wrmsrl(MSR_IA32_XFD_ERR, 0);
+
+	guest_fpu->xfd_err |= XFD_ERR_GUEST_SAVED;
+}
+EXPORT_SYMBOL_GPL(fpu_save_guest_xfd_err);
+
+void fpu_restore_guest_xfd_err(struct fpu_guest *guest_fpu)
+{
+	u64 xfd_err = guest_fpu->xfd_err;
+
+	if (xfd_err & XFD_ERR_GUEST_DISABLED)
+		return;
+
+	xfd_err &= ~XFD_ERR_GUEST_SAVED;
+

arch/x86/kernel/fpu/core.c int fpu_swap_kvm_fpstate(struct fpu_guest *guest_fpu, bool enter_guest) · line 370

Quoted 10 lines
arch/x86/kernel/fpu/core.c +10 -0
+	/*
+	 * No need to restore a zero value since XFD_ERR
+	 * is always zero outside of #NM handler in the host.
+	 */
+	if (!xfd_err)
+		return;
+
+	wrmsrl(MSR_IA32_XFD_ERR, xfd_err);
+	guest_fpu->xfd_err = 0;
+}

Why should any pf this be in the FPU core?

It's a pure guest issue as all of this is related to struct fpu_guest and not struct fpu or any other core FPU state.

Thanks,

tglx

author
#42 lore raw

PATCH 15/19 kvm: x86: Save and restore guest XFD_ERR properly

In reply to Yang Zhong

KVM needs to save the guest XFD_ERR value before this register might be accessed by the host and restore it before entering the guest.

This implementation saves guest XFD_ERR in two transition points:

- When the vCPU thread exits to the userspace VMM;
- When the vCPU thread is preempted;

XFD_ERR is cleared to ZERO right after saving the previous guest value. Otherwise a stale guest value may confuse the host #NM handler to misinterpret a non-XFD-related #NM as XFD related.

There is no need to save the host XFD_ERR value because the only place where XFD_ERR is consumed outside of KVM is in #NM handler (which can not be preempted by a vCPU thread). XFD_ERR should always be observed as ZER0 outside of #NM hanlder, thus clearing XFD_ERR meets the host expectation here.

The saved guest value is restored to XFD_ERR right before entering the guest (with preemption disabled).

Current implementation still has two opens which we would like to hear suggestions:

1) Will #NM be triggered in host kernel?

Now the code is written assuming above is true, and it's the only reason for saving guest XFD_ERR at preemption time. Otherwise the save is only required when the CPU enters ring-3 (either from the vCPU itself or other threads), by leveraging the "user-return notifier" machinery as suggested by Paolo.

2) When to enable XFD_ERR save/restore?

There are four options on the table:

a) As long as guest cpuid has xfd enabled

XFD_ERR save/restore is enabled in every VM-exit (if preemption or ret-to-userspace happens)

b) When the guest sets IA32_XFD to 1 for the first time

Indicate that guest OS supports XFD features. Because guest OS usually initializes IA32_XFD at boot time, XFD_ERR save/restore is enabled for almost every VM-exit (if preemption or ret-to- userspace happens).

No save/restore for legacy guest OS which doesn't support XFD features at all (thus won't touch IA32_XFD).

c) When the guest sets IA32_XFD to 0 for the first time

Lazily enabling XFD_ERR save/restore until XFD features are used inside guest. However, this option doesn't work because XFD_ERR is set when #NM is raised. An VM-exit could happen between CPU raising #NM and guest #NM handler reading XFD_ERR (before setting XFD to 0). The very first XFD_ERR might be already clobbered by the host due to no save/restore in that small window.

d) When the 1st guest #NM with non-zero XFD_ERR occurs

Lazily enabling XFD_ERR save/restore until XFD features are used inside guest. This requires intercepting guest #NM until non-zero XFD_ERR occurs. If a guest with XFD in cpuid never launches an AMX application, it implies that #NM is always trapped thus adding a constant overhead which may be even higher than doing RDMSR in preemption path in a) and b):

#preempts < #VMEXITS (no #NM trap) < #VMEXITS (#NM trap)

The number of preemptions and ret-to-userspaces should be a small portion of total #VMEXITs in a healthy virtualization environment. Our gut-feeling is that adding at most one MSR read and one MSR write to the preempt/user-ret paths is possibly more efficient than increasing #VMEXITs due to trapping #NM.

For above analysis we plan to go option b), although this version currently implements a). But we would like to hear other suggestions before making this change.

---
 arch/x86/kernel/fpu/core.c | 2 ++
 arch/x86/kvm/cpuid.c       | 5 +++++
 arch/x86/kvm/vmx/vmx.c     | 2 ++
 arch/x86/kvm/vmx/vmx.h     | 2 +-
 arch/x86/kvm/x86.c         | 5 +++++
 5 files changed, 15 insertions(+), 1 deletion(-)
arch/x86/kernel/fpu/core.c +2 -0
index 5089f2e7dc22..9811dc98d550 100644
@@ -238,6 +238,7 @@ bool fpu_alloc_guest_fpstate(struct fpu_guest *gfpu)
 	fpstate->is_guest	= true;
 
 	gfpu->fpstate		= fpstate;
+	gfpu->xfd_err           = XFD_ERR_GUEST_DISABLED;
 	gfpu->user_xfeatures	= fpu_user_cfg.default_features;
 	gfpu->user_perm		= fpu_user_cfg.default_features;
 	fpu_init_guest_permissions(gfpu);
@@ -297,6 +298,7 @@ int fpu_swap_kvm_fpstate(struct fpu_guest *guest_fpu, bool enter_guest)
 		fpu->fpstate = guest_fps;
 		guest_fps->in_use = true;
 	} else {
+		fpu_save_guest_xfd_err(guest_fpu);
 		guest_fps->in_use = false;
 		fpu->fpstate = fpu->__task_fpstate;
 		fpu->__task_fpstate = NULL;
arch/x86/kvm/cpuid.c +5 -0
index f3c61205bbf4..ea51b986ee67 100644
@@ -219,6 +219,11 @@ static void kvm_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu)
 		kvm_apic_set_version(vcpu);
 	}
 
+	/* Enable saving guest XFD_ERR */
+	best = kvm_find_cpuid_entry(vcpu, 7, 0);
+	if (best && cpuid_entry_has(best, X86_FEATURE_AMX_TILE))
+		vcpu->arch.guest_fpu.xfd_err = 0;
+
 	best = kvm_find_cpuid_entry(vcpu, 0xD, 0);
 	if (!best)
 		vcpu->arch.guest_supported_xcr0 = 0;
arch/x86/kvm/vmx/vmx.c +2 -0
index 6198b13c4846..0db8bdf273e2 100644
@@ -161,6 +161,7 @@ static u32 vmx_possible_passthrough_msrs[MAX_POSSIBLE_PASSTHROUGH_MSRS] = {
 	MSR_GS_BASE,
 	MSR_KERNEL_GS_BASE,
 	MSR_IA32_XFD,
+	MSR_IA32_XFD_ERR,
 #endif
 	MSR_IA32_SYSENTER_CS,
 	MSR_IA32_SYSENTER_ESP,
@@ -7153,6 +7154,7 @@ static void update_intel_pt_cfg(struct kvm_vcpu *vcpu)
 static void vmx_update_intercept_xfd(struct kvm_vcpu *vcpu)
 {
 	vmx_set_intercept_for_msr(vcpu, MSR_IA32_XFD, MSR_TYPE_R, false);
+	vmx_set_intercept_for_msr(vcpu, MSR_IA32_XFD_ERR, MSR_TYPE_RW, false);
 }
 
 static void vmx_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu)
arch/x86/kvm/vmx/vmx.h +1 -1
index bf9d3051cd6c..0a00242a91e7 100644
@@ -340,7 +340,7 @@ struct vcpu_vmx {
 	struct lbr_desc lbr_desc;
 
 	/* Save desired MSR intercept (read: pass-through) state */
-#define MAX_POSSIBLE_PASSTHROUGH_MSRS	14
+#define MAX_POSSIBLE_PASSTHROUGH_MSRS	15
 	struct {
 		DECLARE_BITMAP(read, MAX_POSSIBLE_PASSTHROUGH_MSRS);
 		DECLARE_BITMAP(write, MAX_POSSIBLE_PASSTHROUGH_MSRS);
arch/x86/kvm/x86.c +5 -0
index d127b229dd29..8b033c9241d6 100644
@@ -4550,6 +4550,9 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
 		kvm_steal_time_set_preempted(vcpu);
 	srcu_read_unlock(&vcpu->kvm->srcu, idx);
 
+	if (vcpu->preempted)
+		fpu_save_guest_xfd_err(&vcpu->arch.guest_fpu);
+
 	static_call(kvm_x86_vcpu_put)(vcpu);
 	vcpu->arch.last_host_tsc = rdtsc();
 }
@@ -9951,6 +9954,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 	if (test_thread_flag(TIF_NEED_FPU_LOAD))
 		switch_fpu_return();
 
+	fpu_restore_guest_xfd_err(&vcpu->arch.guest_fpu);
+
 	if (unlikely(vcpu->arch.switch_db_regs)) {
 		set_debugreg(0, 7);
 		set_debugreg(vcpu->arch.eff_db[0], 0);
maintainer
#43 lore raw

Re: [PATCH 15/19] kvm: x86: Save and restore guest XFD_ERR properly

arch/x86/kvm/x86.c void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu) · line 4555

Quoted 6 lines · Yang Zhong
  		kvm_steal_time_set_preempted(vcpu);
  	srcu_read_unlock(&vcpu->kvm->srcu, idx);
arch/x86/kvm/x86.c +3 -0
+	if (vcpu->preempted)
+		fpu_save_guest_xfd_err(&vcpu->arch.guest_fpu);
+

Instead of checking vcpu->preempted, can you instead check if the active FPU is the guest FPU? That is, save if current->thread.fpu->fpstate->is_guest?

Paolo

maintainer
#44 lore raw

Re: [PATCH 15/19] kvm: x86: Save and restore guest XFD_ERR properly

In reply to Yang Zhong

arch/x86/kvm/cpuid.c static void kvm_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu) · line 226

Quoted 11 lines · Yang Zhong
arch/x86/kvm/cpuid.c +5 -0
@@ -219,6 +219,11 @@ static void kvm_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu)
  		kvm_apic_set_version(vcpu);
  	}
 
+	/* Enable saving guest XFD_ERR */
+	best = kvm_find_cpuid_entry(vcpu, 7, 0);
+	if (best && cpuid_entry_has(best, X86_FEATURE_AMX_TILE))
+		vcpu->arch.guest_fpu.xfd_err = 0;
+

This is incorrect. Instead it should check whether leaf 0xD includes any dynamic features.

Paolo

author
#45 lore raw

Re: [PATCH 15/19] kvm: x86: Save and restore guest XFD_ERR properly

Quoted 15 lines · Paolo Bonzini

arch/x86/kvm/cpuid.c static void kvm_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu) · line 225

Quoted 10 lines · Yang Zhong
arch/x86/kvm/cpuid.c +5 -0
@@ -219,6 +219,11 @@ static void kvm_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu)
 		kvm_apic_set_version(vcpu);
 	}
+	/* Enable saving guest XFD_ERR */
+	best = kvm_find_cpuid_entry(vcpu, 7, 0);
+	if (best && cpuid_entry_has(best, X86_FEATURE_AMX_TILE))
+		vcpu->arch.guest_fpu.xfd_err = 0;
+

This is incorrect. Instead it should check whether leaf 0xD includes any dynamic features.

Thanks Paolo, So ditto for "[PATCH 04/19] kvm: x86: Check guest xstate permissions when KVM_SET_CPUID2".

Yang

Quoted 1 line

Paolo

#46 lore raw

Re: [PATCH 15/19] kvm: x86: Save and restore guest XFD_ERR properly

In reply to Yang Zhong

arch/x86/kernel/fpu/core.c bool fpu_alloc_guest_fpstate(struct fpu_guest *gfpu) · line 241

Quoted 9 lines · Yang Zhong
arch/x86/kernel/fpu/core.c +1 -0
index 5089f2e7dc22..9811dc98d550 100644
@@ -238,6 +238,7 @@ bool fpu_alloc_guest_fpstate(struct fpu_guest *gfpu)
 	fpstate->is_guest	= true;
 
 	gfpu->fpstate		= fpstate;
+	gfpu->xfd_err           = XFD_ERR_GUEST_DISABLED;

This wants to be part of the previous patch, which introduces the field.

arch/x86/kernel/fpu/core.c int fpu_swap_kvm_fpstate(struct fpu_guest *guest_fpu, bool enter_guest) · line 301

Quoted 8 lines
arch/x86/kernel/fpu/core.c +1 -0
 	gfpu->user_xfeatures	= fpu_user_cfg.default_features;
 	gfpu->user_perm		= fpu_user_cfg.default_features;
 	fpu_init_guest_permissions(gfpu);
@@ -297,6 +298,7 @@ int fpu_swap_kvm_fpstate(struct fpu_guest *guest_fpu, bool enter_guest)
 		fpu->fpstate = guest_fps;
 		guest_fps->in_use = true;
 	} else {
+		fpu_save_guest_xfd_err(guest_fpu);

Hmm. See below.

arch/x86/kernel/fpu/core.c void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu) · line 4554

Quoted 9 lines
arch/x86/kernel/fpu/core.c +2 -0
 		guest_fps->in_use = false;
 		fpu->fpstate = fpu->__task_fpstate;
 		fpu->__task_fpstate = NULL;
@@ -4550,6 +4550,9 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
 		kvm_steal_time_set_preempted(vcpu);
 	srcu_read_unlock(&vcpu->kvm->srcu, idx);
 
+	if (vcpu->preempted)
+		fpu_save_guest_xfd_err(&vcpu->arch.guest_fpu);

I'm not really exited about the thought of an exception cause register in guest clobbered state.

Aside of that I really have to ask the question why all this is needed?

#NM in the guest is slow path, right? So why are you trying to optimize for it?

The straight forward solution to this is:

    1) Trap #NM and MSR_XFD_ERR write

    2) When the guest triggers #NM is takes an VMEXIT and the host
       does:

                rdmsrl(MSR_XFD_ERR, vcpu->arch.guest_fpu.xfd_err);

       injects the #NM and goes on.

    3) When the guest writes to MSR_XFD_ERR it takes an VMEXIT and
       the host does:

           vcpu->arch.guest_fpu.xfd_err = msrval;
           wrmsrl(MSR_XFD_ERR, msrval);

      and goes back.

    4) Before entering the preemption disabled section of the VCPU loop
       do:
       
           if (vcpu->arch.guest_fpu.xfd_err)
                      wrmsrl(MSR_XFD_ERR, vcpu->arch.guest_fpu.xfd_err);

    5) Before leaving the preemption disabled section of the VCPU loop
       do:
       
           if (vcpu->arch.guest_fpu.xfd_err)
                      wrmsrl(MSR_XFD_ERR, 0);

It's really that simple and pretty much 0 overhead for the regular case.

If the guest triggers #NM with a high frequency then taking the VMEXITs is the least of the problems. That's not a realistic use case, really.

Hmm?

Thanks,

tglx

maintainer
#47 lore raw

Re: [PATCH 15/19] kvm: x86: Save and restore guest XFD_ERR properly

Quoted 5 lines · Thomas Gleixner

2) When the guest triggers #NM is takes an VMEXIT and the host does:

rdmsrl(MSR_XFD_ERR, vcpu->arch.guest_fpu.xfd_err);

Quoted 7 lines

injects the #NM and goes on.

3) When the guest writes to MSR_XFD_ERR it takes an VMEXIT and the host does:

vcpu->arch.guest_fpu.xfd_err = msrval; wrmsrl(MSR_XFD_ERR, msrval);

No wrmsrl here I think, the host value is 0 and should stay so. Instead the wrmsrl will happen the next time the VCPU loop is entred.

Paolo

#48 lore raw

RE: [PATCH 15/19] kvm: x86: Save and restore guest XFD_ERR properly

Quoted 11 lines

From: Paolo Bonzini
Sent: Saturday, December 11, 2021 9:32 AM

Quoted 7 lines · Thomas Gleixner

2) When the guest triggers #NM is takes an VMEXIT and the host does:

rdmsrl(MSR_XFD_ERR, vcpu->arch.guest_fpu.xfd_err);

injects the #NM and goes on.

Quoted 9 lines
Quoted 5 lines

3) When the guest writes to MSR_XFD_ERR it takes an VMEXIT and the host does:

vcpu->arch.guest_fpu.xfd_err = msrval; wrmsrl(MSR_XFD_ERR, msrval);

No wrmsrl here I think, the host value is 0 and should stay so. Instead the wrmsrl will happen the next time the VCPU loop is entred.

To elaborate I guess the reason is because MSR_XFD_ERR should always contain host value 0 after preemption is enabled, while WRMSR emulation is called with preemption enabled. Then we just need wait for the next time the vcpu loop is entered to restore the guest value after preemption is disabled. 😊

Thanks
Kevin

#49 lore raw

Re: [PATCH 15/19] kvm: x86: Save and restore guest XFD_ERR properly

In reply to Paolo Bonzini
Quoted 8 lines · Paolo Bonzini
Quoted 7 lines · Thomas Gleixner

2) When the guest triggers #NM is takes an VMEXIT and the host does:

rdmsrl(MSR_XFD_ERR, vcpu->arch.guest_fpu.xfd_err);

injects the #NM and goes on.

Quoted 8 lines
Quoted 5 lines

3) When the guest writes to MSR_XFD_ERR it takes an VMEXIT and the host does:

vcpu->arch.guest_fpu.xfd_err = msrval; wrmsrl(MSR_XFD_ERR, msrval);

No wrmsrl here I think, the host value is 0 and should stay so. Instead the wrmsrl will happen the next time the VCPU loop is entred.

I assumed this can be handled in the fast path, but either way.

#50 lore raw

RE: [PATCH 15/19] kvm: x86: Save and restore guest XFD_ERR properly

In reply to Thomas Gleixner
Quoted 42 lines

From: Thomas Gleixner <[email protected]>
Sent: Saturday, December 11, 2021 8:11 AM

arch/x86/kernel/fpu/core.c bool fpu_alloc_guest_fpstate(struct fpu_guest

Quoted 5 lines · Yang Zhong
arch/x86/kernel/fpu/core.c +0 -0
index 5089f2e7dc22..9811dc98d550 100644
@@ -238,6 +238,7 @@ bool fpu_alloc_guest_fpstate(struct fpu_guest

*gfpu)

arch/x86/kernel/fpu/core.c bool fpu_alloc_guest_fpstate(struct fpu_guest *gfpu) · line 241

Quoted 4 lines
arch/x86/kernel/fpu/core.c +1 -0
 	fpstate->is_guest	= true;
 
 	gfpu->fpstate		= fpstate;
+	gfpu->xfd_err           = XFD_ERR_GUEST_DISABLED;

This wants to be part of the previous patch, which introduces the field.

arch/x86/kernel/fpu/core.c int fpu_swap_kvm_fpstate(struct fpu_guest

Quoted 4 lines
arch/x86/kernel/fpu/core.c +0 -0
 	gfpu->user_xfeatures	= fpu_user_cfg.default_features;
 	gfpu->user_perm		= fpu_user_cfg.default_features;
 	fpu_init_guest_permissions(gfpu);
@@ -297,6 +298,7 @@ int fpu_swap_kvm_fpstate(struct fpu_guest

*guest_fpu, bool enter_guest)

arch/x86/kernel/fpu/core.c int fpu_swap_kvm_fpstate(struct fpu_guest *guest_fpu, bool enter_guest) · line 301

Quoted 4 lines
arch/x86/kernel/fpu/core.c +1 -0
 		fpu->fpstate = guest_fps;
 		guest_fps->in_use = true;
 	} else {
+		fpu_save_guest_xfd_err(guest_fpu);

Hmm. See below.

arch/x86/kernel/fpu/core.c void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu) · line 4554

Quoted 9 lines
arch/x86/kernel/fpu/core.c +2 -0
 		guest_fps->in_use = false;
 		fpu->fpstate = fpu->__task_fpstate;
 		fpu->__task_fpstate = NULL;
@@ -4550,6 +4550,9 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
 		kvm_steal_time_set_preempted(vcpu);
 	srcu_read_unlock(&vcpu->kvm->srcu, idx);
 
+	if (vcpu->preempted)
+		fpu_save_guest_xfd_err(&vcpu->arch.guest_fpu);

I'm not really exited about the thought of an exception cause register in guest clobbered state.

Quoted 4 lines

Aside of that I really have to ask the question why all this is needed?

#NM in the guest is slow path, right? So why are you trying to optimize for it?

This is really good information. The current logic is obviously based on the assumption that #NM is frequently triggered.

Quoted 4 lines

The straight forward solution to this is:

1) Trap #NM and MSR_XFD_ERR write

and #NM vmexit handler should be called in kvm_x86_handle_exit_irqoff() before preemption is enabled, otherwise there is still a small window where MSR_XFD_ERR might be clobbered after preemption enable and before #NM handler is actually called.

Quoted 28 lines
    2) When the guest triggers #NM is takes an VMEXIT and the host
       does:

                rdmsrl(MSR_XFD_ERR, vcpu->arch.guest_fpu.xfd_err);

       injects the #NM and goes on.

    3) When the guest writes to MSR_XFD_ERR it takes an VMEXIT and
       the host does:

           vcpu->arch.guest_fpu.xfd_err = msrval;
           wrmsrl(MSR_XFD_ERR, msrval);

      and goes back.

    4) Before entering the preemption disabled section of the VCPU loop
       do:

           if (vcpu->arch.guest_fpu.xfd_err)
                      wrmsrl(MSR_XFD_ERR, vcpu->arch.guest_fpu.xfd_err);

    5) Before leaving the preemption disabled section of the VCPU loop
       do:

           if (vcpu->arch.guest_fpu.xfd_err)
                      wrmsrl(MSR_XFD_ERR, 0);
Quoted 1 line

It's really that simple and pretty much 0 overhead for the regular case.

Much cleaner.

Quoted 9 lines

If the guest triggers #NM with a high frequency then taking the VMEXITs is the least of the problems. That's not a realistic use case, really.

Hmm?

Thanks,

tglx

Thanks
Kevin

#51 lore raw

RE: [PATCH 15/19] kvm: x86: Save and restore guest XFD_ERR properly

Kevin,

Quoted 6 lines · Kevin Tian
Quoted 3 lines

From: Thomas Gleixner <[email protected]>
#NM in the guest is slow path, right? So why are you trying to optimize for it?

This is really good information. The current logic is obviously based on the assumption that #NM is frequently triggered.

More context.

When an application want's to use AMX, it invokes the prctl() which grants permission. If permission is granted then still the kernel FPU state buffers are default size and XFD is armed.

When a thread of that process issues the first AMX (tile) instruction, then #NM is raised.

The #NM handler does:

1) Read MSR_XFD_ERR. If 0, goto regular #NM

2) Write MSR_XFD_ERR to 0

3) Check whether the process has permission granted. If not, raise SIGILL and return.

4) Allocate and install a larger FPU state buffer for the task. If allocation fails, raise SIGSEGV and return.

5) Disarm XFD for that task

That means one thread takes at max. one AMX/XFD related #NM during its lifetime, which means two VMEXITs.

If there are other XFD controlled facilities in the future, then it will be NR_USED_XFD_CONTROLLED_FACILITIES * 2 VMEXITs per thread which uses them. Not the end of the world either.

Looking at the targeted application space it's pretty unlikely that tasks which utilize AMX are going to be so short lived that the overhead of these VMEXITs really matters.

This of course can be revisited when there is a sane use case, but optimizing for it prematurely does not buy us anything else than pointless complexity.

Quoted 8 lines
Quoted 3 lines

The straight forward solution to this is:

1) Trap #NM and MSR_XFD_ERR write

and #NM vmexit handler should be called in kvm_x86_handle_exit_irqoff() before preemption is enabled, otherwise there is still a small window where MSR_XFD_ERR might be clobbered after preemption enable and before #NM handler is actually called.

Yes.

Thanks,

tglx

#52 lore raw

RE: [PATCH 15/19] kvm: x86: Save and restore guest XFD_ERR properly

Quoted 44 lines

From: Thomas Gleixner <[email protected]>
Sent: Saturday, December 11, 2021 9:29 PM

Kevin,

Quoted 6 lines · Kevin Tian
Quoted 3 lines

From: Thomas Gleixner <[email protected]>
#NM in the guest is slow path, right? So why are you trying to optimize for it?

This is really good information. The current logic is obviously based on the assumption that #NM is frequently triggered.

More context.

When an application want's to use AMX, it invokes the prctl() which grants permission. If permission is granted then still the kernel FPU state buffers are default size and XFD is armed.

When a thread of that process issues the first AMX (tile) instruction, then #NM is raised.

The #NM handler does:

1) Read MSR_XFD_ERR. If 0, goto regular #NM

2) Write MSR_XFD_ERR to 0

3) Check whether the process has permission granted. If not, raise SIGILL and return.

4) Allocate and install a larger FPU state buffer for the task. If allocation fails, raise SIGSEGV and return.

5) Disarm XFD for that task

That means one thread takes at max. one AMX/XFD related #NM during its lifetime, which means two VMEXITs.

If there are other XFD controlled facilities in the future, then it will be NR_USED_XFD_CONTROLLED_FACILITIES * 2 VMEXITs per thread which uses
them. Not the end of the world either.

Quoted 7 lines

Looking at the targeted application space it's pretty unlikely that tasks which utilize AMX are going to be so short lived that the overhead of these VMEXITs really matters.

This of course can be revisited when there is a sane use case, but optimizing for it prematurely does not buy us anything else than pointless complexity.

I get all above.

I guess the original open is also about the frequency of #NM not due to XFD. For Linux guest looks it's not a problem since CR0.TS is not set now when math emulation is not required:

DEFINE_IDTENTRY(exc_device_not_available)

{
	...
	/* This should not happen. */
	if (WARN(cr0 & X86_CR0_TS, "CR0.TS was set")) {
		/* Try to fix it up and carry on. */
		write_cr0(cr0 & ~X86_CR0_TS);
	} else {
		/*
		 * Something terrible happened, and we're better off trying
		 * to kill the task than getting stuck in a never-ending
		 * loop of #NM faults.
		 */
		die("unexpected #NM exception", regs, 0);
	}
}

It may affect guest which still uses CR0.TS to do lazy save. But likely modern OSes all move to eager save approach so always trapping #NM should be fine.

Is this understanding correct?

Thanks
Kevin

maintainer
#53 lore raw

Re: [PATCH 15/19] kvm: x86: Save and restore guest XFD_ERR properly

Quoted 10 lines · Kevin
Quoted 10 lines

If there are other XFD controlled facilities in the future, then it will be NR_USED_XFD_CONTROLLED_FACILITIES * 2 VMEXITs per thread which uses
them. Not the end of the world either.

Looking at the targeted application space it's pretty unlikely that tasks which utilize AMX are going to be so short lived that the overhead of these VMEXITs really matters.

Quoted 6 lines
Quoted 3 lines

This of course can be revisited when there is a sane use case, but optimizing for it prematurely does not buy us anything else than pointless complexity.

It may affect guest which still uses CR0.TS to do lazy save. But likely modern OSes all move to eager save approach so always trapping #NM should be fine.

You also don't need to trap #NM if CPUID includes no dynamic bits, because then XFD will never be nonzero.

Paolo

author
#54 lore raw

PATCH 16/19 kvm: x86: Introduce KVM_{G|S}ET_XSAVE2 ioctl

In reply to Yang Zhong

From: Jing Liu <[email protected]>

When dynamic XSTATE features are supported, the xsave states are beyond 4KB. The current kvm_xsave structure and related KVM_{G, S}ET_XSAVE only allows 4KB which is not enough for full states.

Introduce a new kvm_xsave2 structure and the corresponding KVM_GET_XSAVE2 and KVM_SET_XSAVE2 ioctls so that userspace VMM can get and set the full xsave states.

---
 arch/x86/include/uapi/asm/kvm.h |  6 ++++
 arch/x86/kvm/x86.c              | 62 +++++++++++++++++++++++++++++++++
 include/uapi/linux/kvm.h        |  7 +++-
 3 files changed, 74 insertions(+), 1 deletion(-)
arch/x86/include/uapi/asm/kvm.h +6 -0
index 5a776a08f78c..de42a51e20c3 100644
@@ -47,6 +47,7 @@
 #define __KVM_HAVE_VCPU_EVENTS
 #define __KVM_HAVE_DEBUGREGS
 #define __KVM_HAVE_XSAVE
+#define __KVM_HAVE_XSAVE2
 #define __KVM_HAVE_XCRS
 #define __KVM_HAVE_READONLY_MEM
 
@@ -378,6 +379,11 @@ struct kvm_xsave {
 	__u32 region[1024];
 };
 
+struct kvm_xsave2 {
+	__u32 size;
+	__u8 state[0];
+};
+
 #define KVM_MAX_XCRS	16
 
 struct kvm_xcr {
arch/x86/kvm/x86.c +62 -0

86 lines not on this page · raw message

include/uapi/linux/kvm.h +6 -1
index 0c7b301c7254..603e1ca9ba09 100644
@@ -1132,7 +1132,9 @@ struct kvm_ppc_resize_hpt {
 #define KVM_CAP_EXIT_ON_EMULATION_FAILURE 204
 #define KVM_CAP_ARM_MTE 205
 #define KVM_CAP_VM_MOVE_ENC_CONTEXT_FROM 206
-
+#ifdef __KVM_HAVE_XSAVE2
+#define KVM_CAP_XSAVE2 207
+#endif
 #ifdef KVM_CAP_IRQ_ROUTING
 
 struct kvm_irq_routing_irqchip {
@@ -1679,6 +1681,9 @@ struct kvm_xen_hvm_attr {
 #define KVM_GET_SREGS2             _IOR(KVMIO,  0xcc, struct kvm_sregs2)
 #define KVM_SET_SREGS2             _IOW(KVMIO,  0xcd, struct kvm_sregs2)
 
+#define KVM_GET_XSAVE2		   _IOR(KVMIO,  0xcf, struct kvm_xsave2)
+#define KVM_SET_XSAVE2		   _IOW(KVMIO,  0xd0, struct kvm_xsave2)
+
 struct kvm_xen_vcpu_attr {
 	__u16 type;
 	__u16 pad[3];
maintainer
#55 lore raw

Re: [PATCH 16/19] kvm: x86: Introduce KVM_{G|S}ET_XSAVE2 ioctl

Quoted 154 lines · Yang Zhong

From: Jing Liu <[email protected]>

When dynamic XSTATE features are supported, the xsave states are beyond 4KB. The current kvm_xsave structure and related KVM_{G, S}ET_XSAVE only allows 4KB which is not enough for full states.

Introduce a new kvm_xsave2 structure and the corresponding KVM_GET_XSAVE2 and KVM_SET_XSAVE2 ioctls so that userspace VMM can get and set the full xsave states.

---
  arch/x86/include/uapi/asm/kvm.h |  6 ++++
  arch/x86/kvm/x86.c              | 62 +++++++++++++++++++++++++++++++++
  include/uapi/linux/kvm.h        |  7 +++-
  3 files changed, 74 insertions(+), 1 deletion(-)
arch/x86/include/uapi/asm/kvm.h +6 -0
index 5a776a08f78c..de42a51e20c3 100644
@@ -47,6 +47,7 @@
  #define __KVM_HAVE_VCPU_EVENTS
  #define __KVM_HAVE_DEBUGREGS
  #define __KVM_HAVE_XSAVE
+#define __KVM_HAVE_XSAVE2
  #define __KVM_HAVE_XCRS
  #define __KVM_HAVE_READONLY_MEM
 
@@ -378,6 +379,11 @@ struct kvm_xsave {
  	__u32 region[1024];
  };
 
+struct kvm_xsave2 {
+	__u32 size;
+	__u8 state[0];
+};
+
  #define KVM_MAX_XCRS	16
 
  struct kvm_xcr {
arch/x86/kvm/x86.c +62 -0
index 8b033c9241d6..d212f6d2d39a 100644
@@ -4216,6 +4216,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
  	case KVM_CAP_DEBUGREGS:
  	case KVM_CAP_X86_ROBUST_SINGLESTEP:
  	case KVM_CAP_XSAVE:
+	case KVM_CAP_XSAVE2:
  	case KVM_CAP_ASYNC_PF:
  	case KVM_CAP_ASYNC_PF_INT:
  	case KVM_CAP_GET_TSC_KHZ:
@@ -4940,6 +4941,17 @@ static void kvm_vcpu_ioctl_x86_get_xsave(struct kvm_vcpu *vcpu,
  				       vcpu->arch.pkru);
  }
 
+static void kvm_vcpu_ioctl_x86_get_xsave2(struct kvm_vcpu *vcpu,
+					  u8 *state, u32 size)
+{
+	if (fpstate_is_confidential(&vcpu->arch.guest_fpu))
+		return;
+
+	fpu_copy_guest_fpstate_to_uabi(&vcpu->arch.guest_fpu,
+				       state, size,
+				       vcpu->arch.pkru);
+}
+
  static int kvm_vcpu_ioctl_x86_set_xsave(struct kvm_vcpu *vcpu,
  					struct kvm_xsave *guest_xsave)
  {
@@ -4951,6 +4963,15 @@ static int kvm_vcpu_ioctl_x86_set_xsave(struct kvm_vcpu *vcpu,
  					      supported_xcr0, &vcpu->arch.pkru);
  }
 
+static int kvm_vcpu_ioctl_x86_set_xsave2(struct kvm_vcpu *vcpu, u8 *state)
+{
+	if (fpstate_is_confidential(&vcpu->arch.guest_fpu))
+		return 0;
+
+	return fpu_copy_uabi_to_guest_fpstate(&vcpu->arch.guest_fpu, state,
+					      supported_xcr0, &vcpu->arch.pkru);
+}
+
  static void kvm_vcpu_ioctl_x86_get_xcrs(struct kvm_vcpu *vcpu,
  					struct kvm_xcrs *guest_xcrs)
  {
@@ -5416,6 +5437,47 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
  		r = kvm_vcpu_ioctl_x86_set_xsave(vcpu, u.xsave);
  		break;
  	}
+	case KVM_GET_XSAVE2: {
+		struct kvm_xsave2 __user *xsave2_arg = argp;
+		struct kvm_xsave2 xsave2;
+
+		r = -EFAULT;
+		if (copy_from_user(&xsave2, xsave2_arg, sizeof(struct kvm_xsave2)))
+			break;
+
+		u.buffer = kzalloc(xsave2.size, GFP_KERNEL_ACCOUNT);
+
+		r = -ENOMEM;
+		if (!u.buffer)
+			break;
+
+		kvm_vcpu_ioctl_x86_get_xsave2(vcpu, u.buffer, xsave2.size);
+
+		r = -EFAULT;
+		if (copy_to_user(xsave2_arg->state, u.buffer, xsave2.size))
+			break;
+
+		r = 0;
+		break;
+	}
+	case KVM_SET_XSAVE2: {
+		struct kvm_xsave2 __user *xsave2_arg = argp;
+		struct kvm_xsave2 xsave2;
+
+		r = -EFAULT;
+		if (copy_from_user(&xsave2, xsave2_arg, sizeof(struct kvm_xsave2)))
+			break;
+
+		u.buffer = memdup_user(xsave2_arg->state, xsave2.size);
+
+		if (IS_ERR(u.buffer)) {
+			r = PTR_ERR(u.buffer);
+			goto out_nofree;
+		}
+
+		r = kvm_vcpu_ioctl_x86_set_xsave2(vcpu, u.buffer);
+		break;
+	}
  	case KVM_GET_XCRS: {
  		u.xcrs = kzalloc(sizeof(struct kvm_xcrs), GFP_KERNEL_ACCOUNT);
  		r = -ENOMEM;
include/uapi/linux/kvm.h +3 -1
index 0c7b301c7254..603e1ca9ba09 100644
@@ -1132,7 +1132,9 @@ struct kvm_ppc_resize_hpt {
  #define KVM_CAP_EXIT_ON_EMULATION_FAILURE 204
  #define KVM_CAP_ARM_MTE 205
  #define KVM_CAP_VM_MOVE_ENC_CONTEXT_FROM 206
-
+#ifdef __KVM_HAVE_XSAVE2
+#define KVM_CAP_XSAVE2 207
+#endif
  #ifdef KVM_CAP_IRQ_ROUTING
 
  struct kvm_irq_routing_irqchip {

include/uapi/linux/kvm.h struct kvm_xen_hvm_attr { · line 1689

Quoted 9 lines
include/uapi/linux/kvm.h +3 -0
@@ -1679,6 +1681,9 @@ struct kvm_xen_hvm_attr {
  #define KVM_GET_SREGS2             _IOR(KVMIO,  0xcc, struct kvm_sregs2)
  #define KVM_SET_SREGS2             _IOW(KVMIO,  0xcd, struct kvm_sregs2)
 
+#define KVM_GET_XSAVE2		   _IOR(KVMIO,  0xcf, struct kvm_xsave2)
+#define KVM_SET_XSAVE2		   _IOW(KVMIO,  0xd0, struct kvm_xsave2)
+
  struct kvm_xen_vcpu_attr {
  	__u16 type;
  	__u16 pad[3];

Please also modify KVM_GET/SET_XSAVE to fail with ENOSPC if the requested size is bigger than 4096.

Paolo

maintainer
#56 lore raw

Re: [PATCH 16/19] kvm: x86: Introduce KVM_{G|S}ET_XSAVE2 ioctl

In reply to Yang Zhong
Quoted 17 lines · Yang Zhong
arch/x86/kvm/x86.c +11 -0
+static void kvm_vcpu_ioctl_x86_get_xsave2(struct kvm_vcpu *vcpu,
+					  u8 *state, u32 size)
+{
+	if (fpstate_is_confidential(&vcpu->arch.guest_fpu))
+		return;
+
+	fpu_copy_guest_fpstate_to_uabi(&vcpu->arch.guest_fpu,
+				       state, size,
+				       vcpu->arch.pkru);
+}
+
  static int kvm_vcpu_ioctl_x86_set_xsave(struct kvm_vcpu *vcpu,
  					struct kvm_xsave *guest_xsave)
  {
@@ -4951,6 +4963,15 @@ static int kvm_vcpu_ioctl_x86_set_xsave(struct kvm_vcpu *vcpu,
  					      supported_xcr0, &vcpu->arch.pkru);
  }

arch/x86/kvm/x86.c static int kvm_vcpu_ioctl_x86_set_xsave(struct kvm_vcpu *vcpu, · line 4974

Quoted 10 lines
arch/x86/kvm/x86.c +9 -0
 
+static int kvm_vcpu_ioctl_x86_set_xsave2(struct kvm_vcpu *vcpu, u8 *state)
+{
+	if (fpstate_is_confidential(&vcpu->arch.guest_fpu))
+		return 0;
+
+	return fpu_copy_uabi_to_guest_fpstate(&vcpu->arch.guest_fpu, state,
+					      supported_xcr0, &vcpu->arch.pkru);
+}
+

I think fpu_copy_uabi_to_guest_fpstate (and therefore copy_uabi_from_kernel_to_xstate) needs to check that the size is compatible with the components in the input.

Also, IIUC the size of the AMX state will vary in different processors.
Is this correct? If so, this should be handled already by
KVM_GET/SET_XSAVE2 and therefore should be part of the arch/x86/kernel/fpu APIs. In the future we want to support migrating a "small AMX" host to a "large AMX" host; and also migrating from a "large AMX" host to a "small AMX" host if the guest CPUID is compatible with the destination of the migration.

Paolo

maintainer
#57 lore raw

Re: [PATCH 16/19] kvm: x86: Introduce KVM_{G|S}ET_XSAVE2 ioctl

Quoted 14 lines · Paolo Bonzini

arch/x86/kvm/x86.c static int kvm_vcpu_ioctl_x86_set_xsave(struct kvm_vcpu *vcpu, · line 4974

Quoted 11 lines

+static int kvm_vcpu_ioctl_x86_set_xsave2(struct kvm_vcpu *vcpu, u8 *state)

arch/x86/kvm/x86.c +8 -0
+{
+    if (fpstate_is_confidential(&vcpu->arch.guest_fpu))
+        return 0;
+
+    return fpu_copy_uabi_to_guest_fpstate(&vcpu->arch.guest_fpu, state,
+                          supported_xcr0, &vcpu->arch.pkru);
+}
+

I think fpu_copy_uabi_to_guest_fpstate (and therefore copy_uabi_from_kernel_to_xstate) needs to check that the size is compatible with the components in the input.

Quoted 3 lines

Also, IIUC the size of the AMX state will vary in different processors.
 Is this correct?  If so, this should be handled already by
KVM_GET/SET_XSAVE2 and therefore should be part of the arch/x86/kernel/fpu APIs.  In the future we want to support migrating a "small AMX" host to a "large AMX" host; and also migrating from a "large AMX" host to a "small AMX" host if the guest CPUID is compatible with the destination of the migration.

So, the size of the AMX state will depend on the active "palette" in TILECONFIG, and on the CPUID information. I have a few questions on how Intel intends to handle future extensions to AMX:

- can we assume that, in the future, palette 1 will always have the same value (bytes_per_row=64, max_names=8, max_rows=16), and basically that the only variable value is really the number of palettes?

- how does Intel plan to handle bigger TILEDATA? Will it use more XCR0 bits or will it rather enlarge save state 18?

If it will use more XCR0 bits, I suppose that XCR0 bits will control which palettes can be chosen by LDTILECFG.

If not, on the other hand, this will be a first case of one system's XSAVE data not being XRSTOR-able on another system even if the destination system can set XCR0 to the same value as the source system.

Likewise, if the size and offsets for save state 18 were to vary depending on the selected palette, then this would be novel, in that the save state size and offsets would not be in CPUID anymore. It would be particularly interesting for non-compacted format, where all save states after 18 would also move forward.

So, I hope that save state 18 will be frozen to 8k. In that case, and if palette 1 is frozen to the same values as today, implementing migration will not be a problem; it will be essentially the same as SSE->AVX (horizontal extension of existing registers) and/or AVX->AVX512 (both horizontal and vertical extension).

By the way, I think KVM_SET_XSAVE2 is not needed. Instead:

- KVM_CHECK_EXTENSION(KVM_CAP_XSAVE2) should return the size of the buffer that is passed to KVM_GET_XSAVE2

- KVM_GET_XSAVE2 should fill in the buffer expecting that its size is whatever KVM_CHECK_EXTENSION(KVM_CAP_XSAVE2) passes

- KVM_SET_XSAVE can just expect a buffer that is bigger than 4k if the save states recorded in the header point to offsets larger than 4k.

Paolo

#58 lore raw

RE: [PATCH 16/19] kvm: x86: Introduce KVM_{G|S}ET_XSAVE2 ioctl

Quoted 11 lines · Paolo Bonzini

By the way, I think KVM_SET_XSAVE2 is not needed. Instead:

- KVM_CHECK_EXTENSION(KVM_CAP_XSAVE2) should return the size of the buffer that is passed to KVM_GET_XSAVE2

- KVM_GET_XSAVE2 should fill in the buffer expecting that its size is whatever KVM_CHECK_EXTENSION(KVM_CAP_XSAVE2) passes

- KVM_SET_XSAVE can just expect a buffer that is bigger than 4k if the save states recorded in the header point to offsets larger than 4k.

I think one issue is that KVM_SET_XSAVE works with "struct kvm_xsave" (hardcoded 4KB buffer), including kvm_vcpu_ioctl_x86_set_xsave. The states obtained via KVM_GET_XSAVE2 will be made using "struct kvm_xsave2".

Did you mean that we could add a new code path under KVM_SET_XSAVE to make it work with the new "struct kvm_xsave2"?
e.g.:

(xsave2_enabled below is set when userspace calls to get KVM_CAP_XSAVE2) if (kvm->xsave2_enabled) {
new implementation using "struct kvm_xsave2"
} else {
current implementation using "struct kvm_xsave"
}
(this seems like a new implementation which might deserve a new ioctl)

Thanks,
Wei

maintainer
#59 lore raw

Re: [PATCH 16/19] kvm: x86: Introduce KVM_{G|S}ET_XSAVE2 ioctl

Quoted 13 lines · Wei W
Quoted 11 lines · Paolo Bonzini

By the way, I think KVM_SET_XSAVE2 is not needed. Instead:

- KVM_CHECK_EXTENSION(KVM_CAP_XSAVE2) should return the size of the buffer that is passed to KVM_GET_XSAVE2

- KVM_GET_XSAVE2 should fill in the buffer expecting that its size is whatever KVM_CHECK_EXTENSION(KVM_CAP_XSAVE2) passes

- KVM_SET_XSAVE can just expect a buffer that is bigger than 4k if the save states recorded in the header point to offsets larger than 4k.

Quoted 6 lines

I think one issue is that KVM_SET_XSAVE works with "struct kvm_xsave" (hardcoded 4KB buffer), including kvm_vcpu_ioctl_x86_set_xsave. The states obtained via KVM_GET_XSAVE2 will be made using "struct kvm_xsave2".

Did you mean that we could add a new code path under KVM_SET_XSAVE to make it work with the new "struct kvm_xsave2"?

There is no need for struct kvm_xsave2, because there is no need for a "size" argument.

- KVM_GET_XSAVE2 *is* needed, and it can expect a buffer as big as the return value of KVM_CHECK_EXTENSION(KVM_CAP_XSAVE2)

- but KVM_SET_XSAVE2 is not needed, because KVM_SET_XSAVE can use copy_from_user to read the XSTATE_BV, use it to deduce the size of the buffer, and use copy_from_user to read the full size of the buffer.

For this to work you can redefine struct kvm_xsave to

	struct kvm_xsave {
		__u32 region[1024];

		/*
		 * KVM_GET_XSAVE only uses 4096 bytes and only returns
		 * user save states up to save state 17 (TILECFG).
		 *
		 * For KVM_GET_XSAVE2, the total size of region + extra
		 * must be the size that is communicated by
		 * KVM_CHECK_EXTENSION(KVM_CAP_XSAVE2).
		 *
		 * KVM_SET_XSAVE uses the extra field if the struct was
		 * returned by KVM_GET_XSAVE2.
		 */
		__u32 extra[];
	}

Paolo

#60 lore raw

RE: [PATCH 16/19] kvm: x86: Introduce KVM_{G|S}ET_XSAVE2 ioctl

Quoted 5 lines · Paolo Bonzini

There is no need for struct kvm_xsave2, because there is no need for a "size" argument.

- KVM_GET_XSAVE2 *is* needed, and it can expect a buffer as big as the return value of KVM_CHECK_EXTENSION(KVM_CAP_XSAVE2)

Why would KVM_GET_XSAVE2 still be needed in this case?

I'm thinking it would also be possible to reuse KVM_GET_XSAVE:

- If userspace calls to KVM_CHECK_EXTENSION(KVM_CAP_XSAVE2),
then KVM knows that the userspace is a new version and it works with larger xsave buffer using the "size" that it returns via KVM_CAP_XSAVE2. So we can add a flag "kvm->xsave2_enabled", which gets set upon userspace checks KVM_CAP_XSAVE2.

- On KVM_GET_XSAVE, if "kvm->xsave2_enabled" is set,
then KVM allocates buffer to load xstates and copies the loaded xstates data to the userspace buffer using the "size" that was returned to userspace on KVM_CAP_XSAVE2.
If "kvm->xsave2_enabled" isn't set, using the legacy "4KB" size.

Thanks,
Wei

maintainer
#61 lore raw

Re: [PATCH 16/19] kvm: x86: Introduce KVM_{G|S}ET_XSAVE2 ioctl

Quoted 7 lines · Wei W
Quoted 5 lines · Paolo Bonzini

There is no need for struct kvm_xsave2, because there is no need for a "size" argument.

- KVM_GET_XSAVE2 *is* needed, and it can expect a buffer as big as the return value of KVM_CHECK_EXTENSION(KVM_CAP_XSAVE2)

Quoted 7 lines

Why would KVM_GET_XSAVE2 still be needed in this case?

I'm thinking it would also be possible to reuse KVM_GET_XSAVE:

- If userspace calls to KVM_CHECK_EXTENSION(KVM_CAP_XSAVE2),
then KVM knows that the userspace is a new version and it works with larger xsave buffer using the "size" that it returns via KVM_CAP_XSAVE2. So we can add a flag "kvm->xsave2_enabled", which gets set upon userspace checks KVM_CAP_XSAVE2.

You can use KVM_ENABLE_CAP(KVM_CAP_XSAVE2) for that, yes. In that case you don't need KVM_GET_XSAVE2.

Paolo

Quoted 8 lines

- On KVM_GET_XSAVE, if "kvm->xsave2_enabled" is set,
then KVM allocates buffer to load xstates and copies the loaded xstates data to the userspace buffer using the "size" that was returned to userspace on KVM_CAP_XSAVE2.
If "kvm->xsave2_enabled" isn't set, using the legacy "4KB" size.

Thanks,
Wei

#62 lore raw

RE: [PATCH 16/19] kvm: x86: Introduce KVM_{G|S}ET_XSAVE2 ioctl

Quoted 22 lines · Paolo Bonzini

To: Wang, Wei W <[email protected]>; Zhong, Yang <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Cc: [email protected]; Nakajima, Jun <[email protected]>; Tian, Kevin <[email protected]>; [email protected]; Liu, Jing2 <[email protected]>; Zeng, Guang <[email protected]>
Subject: Re: [PATCH 16/19] kvm: x86: Introduce KVM_{G|S}ET_XSAVE2 ioctl

Quoted 2 lines · Wei W
Quoted 1 line · Paolo Bonzini

There is no need for struct kvm_xsave2, because there is no need for a

"size"

Quoted 9 lines
Quoted 4 lines

argument.

- KVM_GET_XSAVE2 *is* needed, and it can expect a buffer as big as the return value of KVM_CHECK_EXTENSION(KVM_CAP_XSAVE2)

Why would KVM_GET_XSAVE2 still be needed in this case?

I'm thinking it would also be possible to reuse KVM_GET_XSAVE:

Quoted 8 lines
Quoted 2 lines

- If userspace calls to KVM_CHECK_EXTENSION(KVM_CAP_XSAVE2), then KVM knows that the userspace is a new version and it works with

larger xsave buffer using the "size" that it returns via KVM_CAP_XSAVE2.

Quoted 1 line

So we can add a flag "kvm->xsave2_enabled", which gets set upon

userspace checks KVM_CAP_XSAVE2.

You can use KVM_ENABLE_CAP(KVM_CAP_XSAVE2) for that, yes. In that case you don't need KVM_GET_XSAVE2.

On more thing here, what size should KVM_CHECK_EXTENSION(KVM_CAP_XSAVE2) return? If the size still comes from the guest CPUID(0xd, 0)::RCX, would it be better to just return 1? This requires that the QEMU CPUID info has been set to KVM before checking the cap. QEMU already has this CPUID info to get the size (seems no need to inquire KVM for it).

Thanks,
Wei

maintainer
#63 lore raw

Re: [PATCH 16/19] kvm: x86: Introduce KVM_{G|S}ET_XSAVE2 ioctl

Quoted 10 lines · Wei W
Quoted 10 lines
Quoted 6 lines

Why would KVM_GET_XSAVE2 still be needed in this case?

I'm thinking it would also be possible to reuse KVM_GET_XSAVE:

- If userspace calls to KVM_CHECK_EXTENSION(KVM_CAP_XSAVE2), then KVM knows that the userspace is a new version and it works with

larger xsave buffer using the "size" that it returns via KVM_CAP_XSAVE2.

Quoted 1 line

So we can add a flag "kvm->xsave2_enabled", which gets set upon

userspace checks KVM_CAP_XSAVE2.

Quoted 7 lines
Quoted 2 lines

You can use KVM_ENABLE_CAP(KVM_CAP_XSAVE2) for that, yes. In that case you don't need KVM_GET_XSAVE2.

On more thing here, what size should KVM_CHECK_EXTENSION(KVM_CAP_XSAVE2) return? If the size still comes from the guest CPUID(0xd, 0)::RCX, would it be better to just return 1? This requires that the QEMU CPUID info has been set to KVM before checking the cap. QEMU already has this CPUID info to get the size (seems no need to inquire KVM for it).

It's still easier to return the full size of the buffer from KVM_CHECK_EXTENSION(KVM_CAP_XSAVE2). It makes the userspace code a bit easier.

I'm also thinking that I prefer KVM_GET_XSAVE2 to KVM_ENABLE_CAP(KVM_CAP_XSAVE2), after all. Since it would be a backwards-incompatible change to an _old_ ioctl (KVM_GET_XSAVE), I prefer to limit the ways that userspace can shoot itself in the foot.

Paolo

#64 lore raw

RE: [PATCH 16/19] kvm: x86: Introduce KVM_{G|S}ET_XSAVE2 ioctl

Quoted 3 lines · Paolo Bonzini

It's still easier to return the full size of the buffer from KVM_CHECK_EXTENSION(KVM_CAP_XSAVE2). It makes the userspace code a bit easier.

OK. For the "full size" returned to userspace, would you prefer to directly use the value retrieved from guest CPUID(0xd), or get it from guest_fpu (i.e. fpstate->user_size)?
(retrieved from CPUID will be the max size and should work fine as well)

Quoted 5 lines

I'm also thinking that I prefer KVM_GET_XSAVE2 to KVM_ENABLE_CAP(KVM_CAP_XSAVE2), after all. Since it would be a backwards-incompatible change to an _old_ ioctl (KVM_GET_XSAVE), I prefer to limit the ways that userspace can shoot itself in the foot.

OK, we will use KVM_GET_XSAVE2.

Thanks,
Wei

maintainer
#65 lore raw

Re: [PATCH 16/19] kvm: x86: Introduce KVM_{G|S}ET_XSAVE2 ioctl

Quoted 7 lines · Wei W
Quoted 2 lines

It's still easier to return the full size of the buffer from KVM_CHECK_EXTENSION(KVM_CAP_XSAVE2). It makes the userspace code a bit easier.

OK. For the "full size" returned to userspace, would you prefer to directly use the value retrieved from guest CPUID(0xd), or get it from guest_fpu (i.e. fpstate->user_size)? (retrieved from CPUID will be the max size and should work fine as well)

It is okay to reflect only the bits that were enabled in prctl, but please document it in api.rst as well.

Paolo

#66 lore raw

State Component 18 and Palette 1 (Re: [PATCH 16/19] kvm: x86: Introduce KVM_{G|S}ET_XSAVE2 ioctl)

In reply to Paolo Bonzini
Quoted 23 lines · Paolo Bonzini
Quoted 12 lines · Paolo Bonzini

arch/x86/kvm/x86.c static int kvm_vcpu_ioctl_x86_set_xsave(struct kvm_vcpu *vcpu, · line 4974

Quoted 10 lines
arch/x86/kvm/x86.c +9 -0
+static int kvm_vcpu_ioctl_x86_set_xsave2(struct kvm_vcpu *vcpu, u8 *state)
+{
+    if (fpstate_is_confidential(&vcpu->arch.guest_fpu))
+        return 0;
+
+    return fpu_copy_uabi_to_guest_fpstate(&vcpu->arch.guest_fpu, state,
+                          supported_xcr0, &vcpu->arch.pkru);
+}
+

I think fpu_copy_uabi_to_guest_fpstate (and therefore copy_uabi_from_kernel_to_xstate) needs to check that the size is compatible with the components in the input.
Also, IIUC the size of the AMX state will vary in different processors. Is this correct? If so, this should be handled already by KVM_GET/SET_XSAVE2 and therefore should be part of the arch/x86/kernel/fpu APIs. In the future we want to support migrating a "small AMX" host to a "large AMX" host; and also migrating from a "large AMX" host to a "small AMX" host if the guest CPUID is compatible with the destination of the migration.

So, the size of the AMX state will depend on the active "palette" in TILECONFIG, and on the CPUID information. I have a few questions on how Intel intends to handle future extensions to AMX:

- can we assume that, in the future, palette 1 will always have the same value (bytes_per_row=64, max_names=8, max_rows=16), and basically that the only variable value is really the number of palettes?

- how does Intel plan to handle bigger TILEDATA? Will it use more XCR0 bits or will it rather enlarge save state 18?

Quoted 7 lines

If it will use more XCR0 bits, I suppose that XCR0 bits will control which palettes can be chosen by LDTILECFG.

If not, on the other hand, this will be a first case of one system's XSAVE data not being XRSTOR-able on another system even if the destination system can set XCR0 to the same value as the source system.

Likewise, if the size and offsets for save state 18 were to vary depending on the selected palette, then this would be novel, in that the save state size and offsets would not be in CPUID anymore. It would be particularly interesting for non-compacted format, where all save states after 18 would also move forward.

So, I hope that save state 18 will be frozen to 8k. In that case, and if palette 1 is frozen to the same values as today, implementing migration will not be a problem; it will be essentially the same as SSE->AVX (horizontal extension of existing registers) and/or AVX->AVX512 (both horizontal and vertical extension).

Hi Paolo,

I would like to confirm that the state component 18 will remain 8KB and palette 1 will remain the same.

Thanks,
---
Jun

maintainer
#67 lore raw

Re: State Component 18 and Palette 1 (Re: [PATCH 16/19] kvm: x86: Introduce KVM_{G|S}ET_XSAVE2 ioctl)

Quoted 3 lines · Jun

Hi Paolo,

I would like to confirm that the state component 18 will remain 8KB and palette 1 will remain the same.

Great! Can you also confirm that XCR0 bits will control which palettes can be chosen by LDTILECFG?

Thanks,

Paolo

#68 lore raw

Re: State Component 18 and Palette 1 (Re: [PATCH 16/19] kvm: x86: Introduce KVM_{G|S}ET_XSAVE2 ioctl)

Quoted 8 lines · Paolo Bonzini
Quoted 2 lines · Jun

Hi Paolo,
I would like to confirm that the state component 18 will remain 8KB and palette 1 will remain the same.

Great! Can you also confirm that XCR0 bits will control which palettes can be chosen by LDTILECFG?

I need to discuss with the H/W team more and come back. I think this request is plausible; I will suggest it.

Thanks,
---
Jun

#69 lore raw

Re: State Component 18 and Palette 1 (Re: [PATCH 16/19] kvm: x86: Introduce KVM_{G|S}ET_XSAVE2 ioctl)

In reply to Nakajima, Jun
Quoted 9 lines · Jun
Quoted 6 lines

So, I hope that save state 18 will be frozen to 8k. In that case, and if palette 1 is frozen to the same values as today, implementing migration will not be a problem; it will be essentially the same as SSE->AVX (horizontal extension of existing registers) and/or AVX->AVX512 (both horizontal and vertical extension).

I would like to confirm that the state component 18 will remain 8KB and palette 1 will remain the same.

Is that an architectural statement that will soon be making its way into the SDM?

#70 lore raw

Re: State Component 18 and Palette 1 (Re: [PATCH 16/19] kvm: x86: Introduce KVM_{G|S}ET_XSAVE2 ioctl)

Quoted 11 lines · Dave
Quoted 7 lines · Jun
Quoted 6 lines

So, I hope that save state 18 will be frozen to 8k. In that case, and if palette 1 is frozen to the same values as today, implementing migration will not be a problem; it will be essentially the same as SSE->AVX (horizontal extension of existing registers) and/or AVX->AVX512 (both horizontal and vertical extension).

Quoted 5 lines
Quoted 2 lines

I would like to confirm that the state component 18 will remain 8KB and palette 1 will remain the same.

Is that an architectural statement that will soon be making its way into the SDM?

Yes, with the other clarifications (e.g. setting IA32_XFD[18]).

Thanks,
---
Jun

#71 lore raw

Re: [PATCH 16/19] kvm: x86: Introduce KVM_{G|S}ET_XSAVE2 ioctl

In reply to Paolo Bonzini
Quoted 14 lines · Paolo Bonzini

arch/x86/kvm/x86.c static int kvm_vcpu_ioctl_x86_set_xsave(struct kvm_vcpu *vcpu, · line 4974

Quoted 9 lines · Yang Zhong
arch/x86/kvm/x86.c +9 -0
+static int kvm_vcpu_ioctl_x86_set_xsave2(struct kvm_vcpu *vcpu, u8 *state)
+{
+	if (fpstate_is_confidential(&vcpu->arch.guest_fpu))
+		return 0;
+
+	return fpu_copy_uabi_to_guest_fpstate(&vcpu->arch.guest_fpu, state,
+					      supported_xcr0, &vcpu->arch.pkru);
+}
+

I think fpu_copy_uabi_to_guest_fpstate (and therefore copy_uabi_from_kernel_to_xstate) needs to check that the size is compatible with the components in the input.

fpu_copy_uabi_to_guest_fpstate() expects that the input buffer is correctly sized. We surely can add a size check there.

Quoted 7 lines

Also, IIUC the size of the AMX state will vary in different processors.
Is this correct? If so, this should be handled already by
KVM_GET/SET_XSAVE2 and therefore should be part of the arch/x86/kernel/fpu APIs. In the future we want to support migrating a "small AMX" host to a "large AMX" host; and also migrating from a "large AMX" host to a "small AMX" host if the guest CPUID is compatible with the destination of the migration.

How is that supposed to work? If the AMX state size differs then the hosts are not compatible.

Thanks,

tglx

maintainer
#72 lore raw

Re: [PATCH 16/19] kvm: x86: Introduce KVM_{G|S}ET_XSAVE2 ioctl

Quoted 16 lines · Thomas Gleixner
Quoted 14 lines · Paolo Bonzini

Review

Quoted 9 lines · Yang Zhong
(quoted diff) +9 -0
+static int kvm_vcpu_ioctl_x86_set_xsave2(struct kvm_vcpu *vcpu, u8 *state)
+{
+	if (fpstate_is_confidential(&vcpu->arch.guest_fpu))
+		return 0;
+
+	return fpu_copy_uabi_to_guest_fpstate(&vcpu->arch.guest_fpu, state,
+					      supported_xcr0, &vcpu->arch.pkru);
+}
+

I think fpu_copy_uabi_to_guest_fpstate (and therefore copy_uabi_from_kernel_to_xstate) needs to check that the size is compatible with the components in the input.

Quoted 2 lines

fpu_copy_uabi_to_guest_fpstate() expects that the input buffer is correctly sized. We surely can add a size check there.

fpu_copy_guest_fpstate_to_uabi is more problematic because that one writes memory. For fpu_copy_uabi_to_guest_fpstate, we know the input buffer size from the components and we can use it to do a properly-sized memdup_user.

For fpu_copy_guest_fpstate_to_uabi we can just decide that KVM_GET_XSAVE will only save up to the first 4K. Something like the following might actually be good for 5.16-rc; right now, header.xfeatures might lead userspace into reading uninitialized or unmapped memory:

arch/x86/kernel/fpu/xstate.c +16 -2
index d28829403ed0..69609b8c3887 100644
@@ -1138,8 +1138,10 @@ void __copy_xstate_to_uabi_buf(struct membuf to, struct fpstate *fpstate,
 	struct xstate_header header;
 	unsigned int zerofrom;
 	u64 mask;
+	u64 size;
 	int i;
 
+	size = to->left;
 	memset(&header, 0, sizeof(header));
 	header.xfeatures = xsave->header.xfeatures;
 
@@ -1186,7 +1188,20 @@ void __copy_xstate_to_uabi_buf(struct membuf to, struct fpstate *fpstate,
 	/* Copy xsave->i387.sw_reserved */
 	membuf_write(&to, xstate_fx_sw_bytes, sizeof(xsave->i387.sw_reserved));
 
-	/* Copy the user space relevant state of @xsave->header */
+	/*
+	 * Copy the user space relevant state of @xsave->header.
+	 * If not all features fit in the buffer, drop them from the
+	 * saved state so that userspace does not read uninitialized or
+	 * unmapped memory.
+	 */
+	mask = fpstate->user_xfeatures;
+	for_each_extended_xfeature(i, mask) {
+		if (xstate_offsets[i] + xstate_size[i] > size) {
+			header.xfeatures &= BIT(i) - 1;
+			mask &= BIT(i) - 1;
+			break;
+		}
+	}
 	membuf_write(&to, &header, sizeof(header));
 
 	zerofrom = offsetof(struct xregs_state, extended_state_area);
@@ -1197,7 +1212,6 @@ void __copy_xstate_to_uabi_buf(struct membuf to, struct fpstate *fpstate,
 	 * but there is no state to copy from in the compacted
 	 * init_fpstate. The gap tracking will zero these states.
 	 */
-	mask = fpstate->user_xfeatures;
 
 	for_each_extended_xfeature(i, mask) {
 		/*
Quoted 10 lines
Quoted 7 lines

Also, IIUC the size of the AMX state will vary in different processors.
Is this correct? If so, this should be handled already by
KVM_GET/SET_XSAVE2 and therefore should be part of the arch/x86/kernel/fpu APIs. In the future we want to support migrating a "small AMX" host to a "large AMX" host; and also migrating from a "large AMX" host to a "small AMX" host if the guest CPUID is compatible with the destination of the migration.

How is that supposed to work? If the AMX state size differs then the hosts are not compatible.

I replied with some more questions later. Basically it depends on how Intel will define palettes that aren't part of the first implementation of AMX.

Paolo

#73 lore raw

Re: [PATCH 16/19] kvm: x86: Introduce KVM_{G|S}ET_XSAVE2 ioctl

Quoted 14 lines · Paolo Bonzini
Quoted 7 lines · Thomas Gleixner
Quoted 3 lines · Paolo Bonzini

I think fpu_copy_uabi_to_guest_fpstate (and therefore copy_uabi_from_kernel_to_xstate) needs to check that the size is compatible with the components in the input.

fpu_copy_uabi_to_guest_fpstate() expects that the input buffer is correctly sized. We surely can add a size check there.

fpu_copy_guest_fpstate_to_uabi is more problematic because that one writes memory. For fpu_copy_uabi_to_guest_fpstate, we know the input buffer size from the components and we can use it to do a properly-sized memdup_user.

Quoted 4 lines

For fpu_copy_guest_fpstate_to_uabi we can just decide that KVM_GET_XSAVE will only save up to the first 4K. Something like the following might actually be good for 5.16-rc; right now, header.xfeatures might lead userspace into reading uninitialized or unmapped memory:

If user space supplies a 4k buffer and reads beyond the end of the buffer then it's hardly a kernel problem.

That function allows to provide a short buffer and fill it up to the point where the buffer ends with the real information.

Thanks,

tglx

author
#74 lore raw

PATCH 17/19 docs: virt: api.rst: Document the new KVM_{G, S}ET_XSAVE2 ioctls

In reply to Yang Zhong

From: Jing Liu <[email protected]>

Document the detailed information of the new KVM_{G, S}ET_XSAVE2 ioctls.

---
 Documentation/virt/kvm/api.rst | 47 ++++++++++++++++++++++++++++++++++
 1 file changed, 47 insertions(+)
Documentation/virt/kvm/api.rst +47 -0
index aeeb071c7688..39dfd867e429 100644
@@ -1569,6 +1569,8 @@ otherwise it will return EBUSY error.
   };
 
 This ioctl would copy current vcpu's xsave struct to the userspace.
+Application should use KVM_GET_XSAVE2 if xsave states are larger than
+4KB.
 
 
 4.43 KVM_SET_XSAVE
@@ -1588,6 +1590,8 @@ This ioctl would copy current vcpu's xsave struct to the userspace.
   };
 
 This ioctl would copy userspace's xsave struct to the kernel.
+Application should use KVM_SET_XSAVE2 if xsave states are larger than
+4KB.
 
 
 4.44 KVM_GET_XCRS
@@ -7484,3 +7488,46 @@ The argument to KVM_ENABLE_CAP is also a bitmask, and must be a subset
 of the result of KVM_CHECK_EXTENSION.  KVM will forward to userspace
 the hypercalls whose corresponding bit is in the argument, and return
 ENOSYS for the others.
+
+8.35 KVM_GET_XSAVE2
+-------------------
+
+:Capability: KVM_CAP_XSAVE2
+:Architectures: x86
+:Type: vcpu ioctl
+:Parameters: struct kvm_xsave2 (in)
+:Returns: 0 on success, -1 on error
+
+
+::
+
+  struct kvm_xsave2 {
+	__u32 size;
+	__u8 state[0];
+  };
+
+This ioctl is used for copying current vcpu's xsave struct to the
+userspace when xsave state size is larger than 4KB. Application code
+should set the 'size' member which indicates the size of xsave state
+and KVM copies the xsave state into the 'state' region.
+
+8.36 KVM_SET_XSAVE2
+-------------------
+
+:Capability: KVM_CAP_XSAVE2
+:Architectures: x86
+:Type: vcpu ioctl
+:Parameters: struct kvm_xsave2 (out)
+:Returns: 0 on success, -1 on error
+
+
+::
+
+  struct kvm_xsave2 {
+	__u32 size;
+	__u8 state[0];
+  };
+
+This ioctl is used for copying userspace's xsave struct to the kernel
+when xsave size is larger than 4KB. Application code should set the
+'size' member which indicates the size of xsave state.
author
#75 lore raw

PATCH 18/19 kvm: x86: AMX XCR0 support for guest

In reply to Yang Zhong

From: Jing Liu <[email protected]>

Two XCR0 bits are defined for AMX to support XSAVE mechanism. Bit 17 is for tilecfg and bit 18 is for tiledata.

The value of XCR0[17:18] is always either 00b or 11b. Also, SDM recommends that only 64-bit operating systems enable Intel AMX by setting XCR0[18:17].

---
 arch/x86/kvm/x86.c | 19 ++++++++++++++++++-
 1 file changed, 18 insertions(+), 1 deletion(-)
arch/x86/kvm/x86.c +18 -1
index d212f6d2d39a..a9a608c8fa50 100644
@@ -210,7 +210,7 @@ static struct kvm_user_return_msrs __percpu *user_return_msrs;
 #define KVM_SUPPORTED_XCR0     (XFEATURE_MASK_FP | XFEATURE_MASK_SSE \
 				| XFEATURE_MASK_YMM | XFEATURE_MASK_BNDREGS \
 				| XFEATURE_MASK_BNDCSR | XFEATURE_MASK_AVX512 \
-				| XFEATURE_MASK_PKRU)
+				| XFEATURE_MASK_PKRU | XFEATURE_MASK_XTILE)
 
 u64 __read_mostly host_efer;
 EXPORT_SYMBOL_GPL(host_efer);
@@ -1017,6 +1017,23 @@ static int __kvm_set_xcr(struct kvm_vcpu *vcpu, u32 index, u64 xcr)
 		if ((xcr0 & XFEATURE_MASK_AVX512) != XFEATURE_MASK_AVX512)
 			return 1;
 	}
+
+#ifdef CONFIG_X86_64
+	if ((xcr0 & XFEATURE_MASK_XTILE) &&
+	    ((xcr0 & XFEATURE_MASK_XTILE) != XFEATURE_MASK_XTILE))
+		return 1;
+#else
+	/*
+	 * Intel AMX instructions can be executed only in 64-bit mode but
+	 * XSAVE can operate on XTILECFG and XTILEDATA in any mode.
+	 * Since the FPU core follows SDM recommendation to set
+	 * XCR[18:17] only in 64-bit environment, here also prevent any
+	 * guest OS from setting the two bits when host is 32-bit.
+	 *
+	 * XFEATURE_MASK_XTILE cannot be used since it is 0 in this case.
+	 */
+	xcr0 &= ~(XFEATURE_MASK_XTILE_DATA | XFEATURE_MASK_XTILE_CFG);
+#endif
 	vcpu->arch.xcr0 = xcr0;
 
 	if ((xcr0 ^ old_xcr0) & XFEATURE_MASK_EXTEND)
maintainer
#76 lore raw

Re: [PATCH 18/19] kvm: x86: AMX XCR0 support for guest

Quoted 7 lines · Yang Zhong
arch/x86/kvm/x86.c +7 -0
+
+#ifdef CONFIG_X86_64
+	if ((xcr0 & XFEATURE_MASK_XTILE) &&
+	    ((xcr0 & XFEATURE_MASK_XTILE) != XFEATURE_MASK_XTILE))
+		return 1;
+#else
+	/*

arch/x86/kvm/x86.c static int __kvm_set_xcr(struct kvm_vcpu *vcpu, u32 index, u64 xcr) · line 1036

Quoted 10 lines
arch/x86/kvm/x86.c +10 -0
+	 * Intel AMX instructions can be executed only in 64-bit mode but
+	 * XSAVE can operate on XTILECFG and XTILEDATA in any mode.
+	 * Since the FPU core follows SDM recommendation to set
+	 * XCR[18:17] only in 64-bit environment, here also prevent any
+	 * guest OS from setting the two bits when host is 32-bit.
+	 *
+	 * XFEATURE_MASK_XTILE cannot be used since it is 0 in this case.
+	 */
+	xcr0 &= ~(XFEATURE_MASK_XTILE_DATA | XFEATURE_MASK_XTILE_CFG);
+#endif

This should not be necessary, because on a 32-bit system the bits won't be part of supported_xcr0.

Paolo

author
#77 lore raw

PATCH 19/19 kvm: x86: Add AMX CPUIDs support

In reply to Yang Zhong

From: Jing Liu <[email protected]>

Extend CPUID emulation to support XFD, AMX_TILE, AMX_INT8 and AMX_BF16. Adding those bits into kvm_cpu_caps finally activates all previous logics in this series.

---
 arch/x86/include/asm/cpufeatures.h |  2 ++
 arch/x86/kvm/cpuid.c               | 16 +++++++++++++---
 2 files changed, 15 insertions(+), 3 deletions(-)
arch/x86/include/asm/cpufeatures.h +2 -0
index d5b5f2ab87a0..da872b6f8d8b 100644
@@ -299,7 +299,9 @@
 /* Intel-defined CPU features, CPUID level 0x00000007:1 (EAX), word 12 */
 #define X86_FEATURE_AVX_VNNI		(12*32+ 4) /* AVX VNNI instructions */
 #define X86_FEATURE_AVX512_BF16		(12*32+ 5) /* AVX512 BFLOAT16 instructions */
+#define X86_FEATURE_AMX_BF16		(18*32+22) /* AMX bf16 Support */
 #define X86_FEATURE_AMX_TILE		(18*32+24) /* AMX tile Support */
+#define X86_FEATURE_AMX_INT8		(18*32+25) /* AMX int8 Support */
 
 /* AMD-defined CPU features, CPUID level 0x80000008 (EBX), word 13 */
 #define X86_FEATURE_CLZERO		(13*32+ 0) /* CLZERO instruction */
arch/x86/kvm/cpuid.c +13 -3
index ea51b986ee67..7bb56cc89aa7 100644
@@ -510,7 +510,8 @@ void kvm_set_cpu_caps(void)
 		F(AVX512_4VNNIW) | F(AVX512_4FMAPS) | F(SPEC_CTRL) |
 		F(SPEC_CTRL_SSBD) | F(ARCH_CAPABILITIES) | F(INTEL_STIBP) |
 		F(MD_CLEAR) | F(AVX512_VP2INTERSECT) | F(FSRM) |
-		F(SERIALIZE) | F(TSXLDTRK) | F(AVX512_FP16)
+		F(SERIALIZE) | F(TSXLDTRK) | F(AVX512_FP16) |
+		F(AMX_TILE) | F(AMX_INT8) | F(AMX_BF16)
 	);
 
 	/* TSC_ADJUST and ARCH_CAPABILITIES are emulated in software. */
@@ -529,7 +530,7 @@ void kvm_set_cpu_caps(void)
 	);
 
 	kvm_cpu_cap_mask(CPUID_D_1_EAX,
-		F(XSAVEOPT) | F(XSAVEC) | F(XGETBV1) | F(XSAVES)
+		F(XSAVEOPT) | F(XSAVEC) | F(XGETBV1) | F(XSAVES) | F(XFD)
 	);
 
 	kvm_cpu_cap_init_scattered(CPUID_12_EAX,
@@ -655,6 +656,8 @@ static struct kvm_cpuid_entry2 *do_host_cpuid(struct kvm_cpuid_array *array,
 	case 0x14:
 	case 0x17:
 	case 0x18:
+	case 0x1d:
+	case 0x1e:
 	case 0x1f:
 	case 0x8000001d:
 		entry->flags |= KVM_CPUID_FLAG_SIGNIFCANT_INDEX;
@@ -779,6 +782,7 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function)
 		}
 		break;
 	case 9:
+	case 0x1e: /* TMUL information */
 		break;
 	case 0xa: { /* Architectural Performance Monitoring */
 		struct x86_pmu_capability cap;
@@ -914,7 +918,8 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function)
 		break;
 	/* Intel PT */
 	case 0x14:
-		if (!kvm_cpu_cap_has(X86_FEATURE_INTEL_PT)) {
+		if ((function == 0x14 && !kvm_cpu_cap_has(X86_FEATURE_INTEL_PT)) ||
+		    (function == 0x1d && !kvm_cpu_cap_has(X86_FEATURE_AMX_TILE))) {
 			entry->eax = entry->ebx = entry->ecx = entry->edx = 0;
 			break;
 		}
@@ -924,6 +929,11 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function)
 				goto out;
 		}
 		break;
+	/* Intel AMX TILE */
+	case 0x1d:
+		if (!kvm_cpu_cap_has(X86_FEATURE_AMX_TILE))
+			entry->eax = entry->ebx = entry->ecx = entry->edx = 0;
+		break;
 	case KVM_CPUID_SIGNATURE: {
 		const u32 *sigptr = (const u32 *)KVM_SIGNATURE;
 		entry->eax = KVM_CPUID_FEATURES;
maintainer
#78 lore raw

Re: [PATCH 19/19] kvm: x86: Add AMX CPUIDs support

arch/x86/kvm/cpuid.c static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function) · line 922

Quoted 7 lines · Yang Zhong
arch/x86/kvm/cpuid.c +2 -1
@@ -914,7 +918,8 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function)
  		break;
  	/* Intel PT */
  	case 0x14:
-		if (!kvm_cpu_cap_has(X86_FEATURE_INTEL_PT)) {
+		if ((function == 0x14 && !kvm_cpu_cap_has(X86_FEATURE_INTEL_PT)) ||
+		    (function == 0x1d && !kvm_cpu_cap_has(X86_FEATURE_AMX_TILE))) {

This hunk is wrong.

arch/x86/kvm/cpuid.c static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function) · line 936

Quoted 12 lines
arch/x86/kvm/cpuid.c +5 -0
  			entry->eax = entry->ebx = entry->ecx = entry->edx = 0;
  			break;
  		}
@@ -924,6 +929,11 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function)
  				goto out;
  		}
  		break;
+	/* Intel AMX TILE */
+	case 0x1d:
+		if (!kvm_cpu_cap_has(X86_FEATURE_AMX_TILE))
+			entry->eax = entry->ebx = entry->ecx = entry->edx = 0;
+		break;

This also needs a loop similar to the one in case 0x14; so the "break" goes inside the "if" and then you have

                for (i = 1, max_idx = entry->eax; i <= max_idx; ++i) {
                        if (!do_host_cpuid(array, function, i))
                                goto out;
                }

Same for 0x1e, which also needs to be marked conditional.

Paolo

#79 lore raw
In reply to Yang Zhong

Jing, Yang,

Quoted 3 lines · Yang Zhong

Thanks Thomas for the thoughts and patches on the KVM FPU and AMX support.

welcome. The overall impression of that series is that it is heading into the right direction. Keep up the good work!

Thanks,

tglx