Enable Sub-page Write Protection Support

init_spp() must be called before {get, set}_subpage functions, it creates subpage access bitmaps for memory pages and issues a KVM request to setup SPPT root pages.

kvm_mmu_set_subpages() is to enable SPP bit in EPT leaf page and setup corresponding SPPT entries. The mmu_lock
is held before above operation. If it's called in EPT fault and SPPT mis-config induced handler, mmu_lock is acquired outside the function, otherwise, it's acquired inside it.

kvm_mmu_get_subpages() is used to query access bitmap for protected page, it's also used in EPT fault handler to check whether the fault EPT page is SPP protected as well.

Co-developed-by He Chen <[email protected]>

Signed-off-by He Chen <[email protected]>

Co-developed-by Zhang Yi <[email protected]>

Signed-off-by Zhang Yi <[email protected]>

Signed-off-by Yang Weijiang <[email protected]>

---
 arch/x86/include/asm/kvm_host.h |  18 ++++
 arch/x86/include/asm/vmx.h      |   2 +
 arch/x86/kvm/mmu.c              | 160 ++++++++++++++++++++++++++++++++
 arch/x86/kvm/vmx/vmx.c          |  48 ++++++++++
 arch/x86/kvm/x86.c              |  40 ++++++++
 include/linux/kvm_host.h        |   4 +-
 include/uapi/linux/kvm.h        |   9 ++
 7 files changed, 280 insertions(+), 1 deletion(-)

arch/x86/include/asm/kvm_host.h +0 -0

index 44f6e1757861..5c4882015acc 100644

Quoted 34 lines · Yang Weijiang

init_spp() must be called before {get, set}_subpage functions, it creates subpage access bitmaps for memory pages and issues a KVM request to setup SPPT root pages.

kvm_mmu_set_subpages() is to enable SPP bit in EPT leaf page and setup corresponding SPPT entries. The mmu_lock
is held before above operation. If it's called in EPT fault and SPPT mis-config induced handler, mmu_lock is acquired outside the function, otherwise, it's acquired inside it.

kvm_mmu_get_subpages() is used to query access bitmap for protected page, it's also used in EPT fault handler to check whether the fault EPT page is SPP protected as well.

Co-developed-by He Chen <[email protected]>

Signed-off-by He Chen <[email protected]>

Co-developed-by Zhang Yi <[email protected]>

Signed-off-by Zhang Yi <[email protected]>

Signed-off-by Yang Weijiang <[email protected]>

---
 arch/x86/include/asm/kvm_host.h |  18 ++++
 arch/x86/include/asm/vmx.h      |   2 +
 arch/x86/kvm/mmu.c              | 160 ++++++++++++++++++++++++++++++++
 arch/x86/kvm/vmx/vmx.c          |  48 ++++++++++
 arch/x86/kvm/x86.c              |  40 ++++++++
 include/linux/kvm_host.h        |   4 +-
 include/uapi/linux/kvm.h        |   9 ++
 7 files changed, 280 insertions(+), 1 deletion(-)

arch/x86/include/asm/kvm_host.h +0 -0

index 44f6e1757861..5c4882015acc 100644

arch/x86/include/asm/kvm_host.h struct kvm_mmu { · line 407

Quoted 10 lines

arch/x86/include/asm/kvm_host.h +5 -0

@@ -398,8 +398,13 @@ struct kvm_mmu {
 	void (*invlpg)(struct kvm_vcpu *vcpu, gva_t gva, hpa_t root_hpa);
 	void (*update_pte)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp,
 			   u64 *spte, const void *pte);
+	int (*get_subpages)(struct kvm *kvm, struct kvm_subpage *spp_info);
+	int (*set_subpages)(struct kvm *kvm, struct kvm_subpage *spp_info);
+	int (*init_spp)(struct kvm *kvm);
+
 	hpa_t root_hpa;
 	gpa_t root_cr3;
+	hpa_t sppt_root;

(I'm sorry if this was previously discussed, I didn't look into previous submissions).

What happens when we launch a nested guest and switch vcpu->arch.mmu to point at arch.guest_mmu? sppt_root will point to INVALID_PAGE and SPP won't be enabled in VMCS?

(I'm sorry again, I'm likely missing something obvious)

Signature · 1 line

Vitaly

Quoted 34 lines · Yang Weijiang

init_spp() must be called before {get, set}_subpage functions, it creates subpage access bitmaps for memory pages and issues a KVM request to setup SPPT root pages.

kvm_mmu_set_subpages() is to enable SPP bit in EPT leaf page and setup corresponding SPPT entries. The mmu_lock
is held before above operation. If it's called in EPT fault and SPPT mis-config induced handler, mmu_lock is acquired outside the function, otherwise, it's acquired inside it.

kvm_mmu_get_subpages() is used to query access bitmap for protected page, it's also used in EPT fault handler to check whether the fault EPT page is SPP protected as well.

Co-developed-by He Chen <[email protected]>

Signed-off-by He Chen <[email protected]>

Co-developed-by Zhang Yi <[email protected]>

Signed-off-by Zhang Yi <[email protected]>

Signed-off-by Yang Weijiang <[email protected]>

---
 arch/x86/include/asm/kvm_host.h |  18 ++++
 arch/x86/include/asm/vmx.h      |   2 +
 arch/x86/kvm/mmu.c              | 160 ++++++++++++++++++++++++++++++++
 arch/x86/kvm/vmx/vmx.c          |  48 ++++++++++
 arch/x86/kvm/x86.c              |  40 ++++++++
 include/linux/kvm_host.h        |   4 +-
 include/uapi/linux/kvm.h        |   9 ++
 7 files changed, 280 insertions(+), 1 deletion(-)

arch/x86/include/asm/kvm_host.h +0 -0

index 44f6e1757861..5c4882015acc 100644

arch/x86/include/asm/kvm_host.h struct kvm_mmu { · line 407

Quoted 10 lines

arch/x86/include/asm/kvm_host.h +5 -0

@@ -398,8 +398,13 @@ struct kvm_mmu {
 	void (*invlpg)(struct kvm_vcpu *vcpu, gva_t gva, hpa_t root_hpa);
 	void (*update_pte)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp,
 			   u64 *spte, const void *pte);
+	int (*get_subpages)(struct kvm *kvm, struct kvm_subpage *spp_info);
+	int (*set_subpages)(struct kvm *kvm, struct kvm_subpage *spp_info);
+	int (*init_spp)(struct kvm *kvm);
+
 	hpa_t root_hpa;
 	gpa_t root_cr3;
+	hpa_t sppt_root;

(I'm sorry if this was previously discussed, I didn't look into previous submissions).

What happens when we launch a nested guest and switch vcpu->arch.mmu to point at arch.guest_mmu? sppt_root will point to INVALID_PAGE and SPP won't be enabled in VMCS?

(I'm sorry again, I'm likely missing something obvious)

Signature · 1 line

Vitaly

After looked into the issue and others, I feel to make SPP co-existing with nested VM is not good, the major reason is, L1 pages protected by SPP are transparent to L1 VM, if it launches L2 VM, probably the pages would be allocated to L2 VM, and that will bother to L1 and L2. Given the feature is new and I don't see nested VM can benefit from it right now, I would like to make SPP and nested feature mutually exclusive, i.e., detecting if the other part is active before activate one feature,what do you think of it?

Quoted 8 lines · Yang Weijiang

After looked into the issue and others, I feel to make SPP co-existing with nested VM is not good, the major reason is, L1 pages protected by SPP are transparent to L1 VM, if it launches L2 VM, probably the pages would be allocated to L2 VM, and that will bother to L1 and L2. Given the feature is new and I don't see nested VM can benefit from it right now, I would like to make SPP and nested feature mutually exclusive, i.e., detecting if the other part is active before activate one feature,what do you think of it?

I was mostly worried about creating a loophole (if I understand correctly) for guests to defeat SPP protection: just launching a nested guest and giving it a protected page. I don't see a problem if we limit SPP to non-nested guests as step 1: we, however, need to document this side-effect of the ioctl. Also, if you decide to do this enforecement, I'd suggest you forbid VMLAUCH/VMRESUME and not VMXON as kvm module loads in linux guests automatically when the hardware is suitable.

Thanks,

Signature · 1 line

Vitaly

Hi, Vitaly,
After looked into the issue and others, I feel to make SPP co-existing with nested VM is not good, the major reason is, L1 pages protected by SPP are transparent to L1 VM, if it launches L2 VM, probably the pages would be allocated to L2 VM, and that will bother to L1 and L2. Given the feature is new and I don't see nested VM can benefit from it right now, I would like to make SPP and nested feature mutually exclusive, i.e., detecting if the other part is active before activate one feature,what do you think of it?
thanks!

Quoted 10 lines · Yang Weijiang

Hi, Vitaly,
After looked into the issue and others, I feel to make SPP co-existing with nested VM is not good, the major reason is, L1 pages protected by SPP are transparent to L1 VM, if it launches L2 VM, probably the pages would be allocated to L2 VM, and that will bother to L1 and L2. Given the feature is new and I don't see nested VM can benefit from it right now, I would like to make SPP and nested feature mutually exclusive, i.e., detecting if the other part is active before activate one feature,what do you think of it?
thanks!

How do you propose making the features mutually exclusive?

Quoted 13 lines · Jim Mattson

Quoted 10 lines · Yang Weijiang

Hi, Vitaly,
After looked into the issue and others, I feel to make SPP co-existing with nested VM is not good, the major reason is, L1 pages protected by SPP are transparent to L1 VM, if it launches L2 VM, probably the pages would be allocated to L2 VM, and that will bother to L1 and L2. Given the feature is new and I don't see nested VM can benefit from it right now, I would like to make SPP and nested feature mutually exclusive, i.e., detecting if the other part is active before activate one feature,what do you think of it?
thanks!

Quoted 1 line

How do you propose making the features mutually exclusive?

I haven't looked at the details or the end to end flow, but would it make sense to exit to userspace on nested VMLAUNCH/VMRESUME if there are SPP mappings? And have the SPP ioctl() kick vCPUs out of guest.

KVM already exits on SPP violations, so presumably this is something that can be punted to userspace.

Thanks Jim and Sean! Could we add a new flag in kvm to identify if nested VM is on or off? That would make things easier. When VMLAUNCH is trapped,
set the flag, if VMXOFF is trapped, clear the flag.

Quoted 3 lines · Yang Weijiang

Thanks Jim and Sean! Could we add a new flag in kvm to identify if nested VM is on or off? That would make things easier. When VMLAUNCH is trapped,
set the flag, if VMXOFF is trapped, clear the flag.

KVM_GET_NESTED_STATE has the requested information. If data.vmx.vmxon_pa is anything other than -1, then the vCPU is in VMX operation. If (flags & KVM_STATE_NESTED_GUEST_MODE), then L2 is active.

Quoted 4 lines

KVM_GET_NESTED_STATE has the requested information. If data.vmx.vmxon_pa is anything other than -1, then the vCPU is in VMX operation. If (flags & KVM_STATE_NESTED_GUEST_MODE), then L2 is active.

Thanks Jim, I'll reference the code and make necessary change in next SPP patch release.

Quoted 4 lines · Yang Weijiang

Quoted 4 lines

KVM_GET_NESTED_STATE has the requested information. If data.vmx.vmxon_pa is anything other than -1, then the vCPU is in VMX operation. If (flags & KVM_STATE_NESTED_GUEST_MODE), then L2 is active.

Quoted 2 lines

Thanks Jim, I'll reference the code and make necessary change in next SPP patch release.

Since SPP will not be used very much in the beginning, it would be simplest to enable it only if nested==0.

Paolo

arch/x86/kvm/mmu.c +7 -0

+
+int kvm_mmu_get_subpages(struct kvm *kvm, struct kvm_subpage *spp_info,
+			 bool mmu_locked)
+{
+	u32 *access = spp_info->access_map;
+	gfn_t gfn = spp_info->base_gfn;
+	int npages = spp_info->npages;

Quoted 7 lines · Yang Weijiang

arch/x86/kvm/mmu.c +7 -0

+
+int kvm_mmu_get_subpages(struct kvm *kvm, struct kvm_subpage *spp_info,
+			 bool mmu_locked)
+{
+	u32 *access = spp_info->access_map;
+	gfn_t gfn = spp_info->base_gfn;
+	int npages = spp_info->npages;

arch/x86/kvm/mmu.c int kvm_mmu_setup_spp_structure(struct kvm_vcpu *vcpu, · line 4469

Quoted 10 lines

arch/x86/kvm/mmu.c +10 -0

+	struct kvm_memory_slot *slot;
+	int i;
+	int ret;
+
+	if (!kvm->arch.spp_active)
+	      return -ENODEV;
+
+	if (!mmu_locked)
+	      spin_lock(&kvm->mmu_lock);
+

Do not add this argument. Just lock mmu_lock in the callers.

Paolo

init_spp() must be called before {get, set}_subpage functions, it creates subpage access bitmaps for memory pages and issues a KVM request to setup SPPT root pages.

kvm_mmu_set_subpages() is to enable SPP bit in EPT leaf page and setup corresponding SPPT entries. The mmu_lock
is held before above operation. If it's called in EPT fault and SPPT mis-config induced handler, mmu_lock is acquired outside the function, otherwise, it's acquired inside it.

kvm_mmu_get_subpages() is used to query access bitmap for protected page, it's also used in EPT fault handler to check whether the fault EPT page is SPP protected as well.

Co-developed-by He Chen <[email protected]>

Signed-off-by He Chen <[email protected]>

Co-developed-by Zhang Yi <[email protected]>

Signed-off-by Zhang Yi <[email protected]>

Signed-off-by Yang Weijiang <[email protected]>

---
 arch/x86/include/asm/kvm_host.h |  18 ++++
 arch/x86/include/asm/vmx.h      |   2 +
 arch/x86/kvm/mmu.c              | 160 ++++++++++++++++++++++++++++++++
 arch/x86/kvm/vmx/vmx.c          |  48 ++++++++++
 arch/x86/kvm/x86.c              |  40 ++++++++
 include/linux/kvm_host.h        |   4 +-
 include/uapi/linux/kvm.h        |   9 ++
 7 files changed, 280 insertions(+), 1 deletion(-)

arch/x86/include/asm/kvm_host.h +18 -0

index 44f6e1757861..5c4882015acc 100644
@@ -398,8 +398,13 @@ struct kvm_mmu {
 	void (*invlpg)(struct kvm_vcpu *vcpu, gva_t gva, hpa_t root_hpa);
 	void (*update_pte)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp,
 			   u64 *spte, const void *pte);
+	int (*get_subpages)(struct kvm *kvm, struct kvm_subpage *spp_info);
+	int (*set_subpages)(struct kvm *kvm, struct kvm_subpage *spp_info);
+	int (*init_spp)(struct kvm *kvm);
+
 	hpa_t root_hpa;
 	gpa_t root_cr3;
+	hpa_t sppt_root;
 	union kvm_mmu_role mmu_role;
 	u8 root_level;
 	u8 shadow_root_level;

@@ -927,6 +932,8 @@ struct kvm_arch {
 
 	bool guest_can_read_msr_platform_info;
 	bool exception_payload_enabled;
+
+	bool spp_active;
 };
 
 struct kvm_vm_stat {

@@ -1199,6 +1206,11 @@ struct kvm_x86_ops {
 	uint16_t (*nested_get_evmcs_version)(struct kvm_vcpu *vcpu);
 
 	bool (*need_emulation_on_page_fault)(struct kvm_vcpu *vcpu);
+
+	bool (*get_spp_status)(void);
+	int (*get_subpages)(struct kvm *kvm, struct kvm_subpage *spp_info);
+	int (*set_subpages)(struct kvm *kvm, struct kvm_subpage *spp_info);
+	int (*init_spp)(struct kvm *kvm);
 };
 
 struct kvm_arch_async_pf {

@@ -1417,6 +1429,12 @@ void kvm_mmu_invlpg(struct kvm_vcpu *vcpu, gva_t gva);
 void kvm_mmu_invpcid_gva(struct kvm_vcpu *vcpu, gva_t gva, unsigned long pcid);
 void kvm_mmu_new_cr3(struct kvm_vcpu *vcpu, gpa_t new_cr3, bool skip_tlb_flush);
 
+int kvm_mmu_get_subpages(struct kvm *kvm, struct kvm_subpage *spp_info,
+			 bool mmu_locked);
+int kvm_mmu_set_subpages(struct kvm *kvm, struct kvm_subpage *spp_info,
+			 bool mmu_locked);
+int kvm_mmu_init_spp(struct kvm *kvm);
+
 void kvm_enable_tdp(void);
 void kvm_disable_tdp(void);
 

arch/x86/include/asm/vmx.h +2 -0

index a2c9e18e0ad7..6cb05ac07453 100644
@@ -224,6 +224,8 @@ enum vmcs_field {
 	XSS_EXIT_BITMAP_HIGH            = 0x0000202D,
 	ENCLS_EXITING_BITMAP		= 0x0000202E,
 	ENCLS_EXITING_BITMAP_HIGH	= 0x0000202F,
+	SPPT_POINTER			= 0x00002030,
+	SPPT_POINTER_HIGH		= 0x00002031,
 	TSC_MULTIPLIER                  = 0x00002032,
 	TSC_MULTIPLIER_HIGH             = 0x00002033,
 	GUEST_PHYSICAL_ADDRESS          = 0x00002400,

arch/x86/kvm/mmu.c +160 -0

index 16e390fd6e58..16d3ca544b67 100644
@@ -3756,6 +3756,9 @@ void kvm_mmu_free_roots(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu,
 		    (mmu->root_level >= PT64_ROOT_4LEVEL || mmu->direct_map)) {
 			mmu_free_root_page(vcpu->kvm, &mmu->root_hpa,
 					   &invalid_list);
+			if (vcpu->kvm->arch.spp_active)
+				mmu_free_root_page(vcpu->kvm, &mmu->sppt_root,
+						   &invalid_list);
 		} else {
 			for (i = 0; i < 4; ++i)
 				if (mmu->pae_root[i] != 0)

@@ -4414,6 +4417,158 @@ int kvm_mmu_setup_spp_structure(struct kvm_vcpu *vcpu,
 	return ret;
 }
 EXPORT_SYMBOL_GPL(kvm_mmu_setup_spp_structure);
+
+int kvm_mmu_init_spp(struct kvm *kvm)
+{
+	int i, ret;
+	struct kvm_vcpu *vcpu;
+	int root_level;
+	struct kvm_mmu_page *ssp_sp;
+
+
+	if (!kvm_x86_ops->get_spp_status())
+	      return -ENODEV;
+
+	if (kvm->arch.spp_active)
+	      return 0;
+
+	ret = kvm_subpage_create_bitmaps(kvm);
+
+	if (ret)
+	      return ret;
+
+	kvm_for_each_vcpu(i, vcpu, kvm) {
+		/* prepare caches for SPP setup.*/
+		mmu_topup_memory_caches(vcpu);
+		root_level = vcpu->arch.mmu->shadow_root_level;
+		ssp_sp = kvm_mmu_get_spp_page(vcpu, 0, root_level);
+		++ssp_sp->root_count;
+		vcpu->arch.mmu->sppt_root = __pa(ssp_sp->spt);
+		kvm_make_request(KVM_REQ_LOAD_CR3, vcpu);
+	}
+
+	kvm->arch.spp_active = true;
+	return 0;
+}
+
+int kvm_mmu_get_subpages(struct kvm *kvm, struct kvm_subpage *spp_info,
+			 bool mmu_locked)
+{
+	u32 *access = spp_info->access_map;
+	gfn_t gfn = spp_info->base_gfn;
+	int npages = spp_info->npages;
+	struct kvm_memory_slot *slot;
+	int i;
+	int ret;
+
+	if (!kvm->arch.spp_active)
+	      return -ENODEV;
+
+	if (!mmu_locked)
+	      spin_lock(&kvm->mmu_lock);
+
+	for (i = 0; i < npages; i++, gfn++) {
+		slot = gfn_to_memslot(kvm, gfn);
+		if (!slot) {
+			ret = -EFAULT;
+			goto out_unlock;
+		}
+		access[i] = *gfn_to_subpage_wp_info(slot, gfn);
+	}
+
+	ret = i;
+
+out_unlock:
+	if (!mmu_locked)
+	      spin_unlock(&kvm->mmu_lock);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(kvm_mmu_get_subpages);
+
+int kvm_mmu_set_subpages(struct kvm *kvm, struct kvm_subpage *spp_info,
+			 bool mmu_locked)
+{
+	u32 *access = spp_info->access_map;
+	gfn_t gfn = spp_info->base_gfn;
+	int npages = spp_info->npages;
+	struct kvm_memory_slot *slot;
+	struct kvm_vcpu *vcpu;
+	struct kvm_rmap_head *rmap_head;
+	int i, k;
+	u32 *wp_map;
+	int ret = -EFAULT;
+
+	if (!kvm->arch.spp_active)
+		return -ENODEV;
+
+	if (!mmu_locked)
+	      spin_lock(&kvm->mmu_lock);
+
+	for (i = 0; i < npages; i++, gfn++) {
+		slot = gfn_to_memslot(kvm, gfn);
+		if (!slot)
+			goto out_unlock;
+
+		/*
+		 * check whether the target 4KB page exists in EPT leaf
+		 * entries.If it's there, we can setup SPP protection now,
+		 * otherwise, need to defer it to EPT page fault handler.
+		 */
+		rmap_head = __gfn_to_rmap(gfn, PT_PAGE_TABLE_LEVEL, slot);
+
+		if (rmap_head->val) {
+			/*
+			 * if all subpages are not writable, open SPP bit in
+			 * EPT leaf entry to enable SPP protection for
+			 * corresponding page.
+			 */
+			if (access[i] != FULL_SPP_ACCESS) {
+				ret = kvm_mmu_open_subpage_write_protect(kvm,
+						slot, gfn);
+
+				if (ret)
+					goto out_err;
+
+				kvm_for_each_vcpu(k, vcpu, kvm)
+					kvm_mmu_setup_spp_structure(vcpu,
+						access[i], gfn);
+			} else {
+				ret = kvm_mmu_clear_subpage_write_protect(kvm,
+						slot, gfn);
+				if (ret)
+					goto out_err;
+			}
+
+		} else
+			pr_info("%s - No ETP entry, gfn = 0x%llx, access = 0x%x.\n", __func__, gfn, access[i]);
+
+		/* if this function is called in tdp_page_fault() or
+		 * spp_handler(), mmu_locked = true, SPP access bitmap
+		 * is being used, otherwise, it's being stored.
+		 */
+		if (!mmu_locked) {
+			wp_map = gfn_to_subpage_wp_info(slot, gfn);
+			*wp_map = access[i];
+		}
+	}
+
+	ret = i;
+out_err:
+	if (ret < 0)
+	      pr_info("SPP-Error, didn't get the gfn:" \
+		      "%llx from EPT leaf.\n"
+		      "Current we don't support SPP on" \
+		      "huge page.\n"
+		      "Please disable huge page and have" \
+		      "another try.\n", gfn);
+out_unlock:
+	if (!mmu_locked)
+	      spin_unlock(&kvm->mmu_lock);
+
+	return ret;
+}
+
 static void nonpaging_init_context(struct kvm_vcpu *vcpu,
 				   struct kvm_mmu *context)
 {

@@ -5104,6 +5259,9 @@ static void init_kvm_tdp_mmu(struct kvm_vcpu *vcpu)
 	context->get_cr3 = get_cr3;
 	context->get_pdptr = kvm_pdptr_read;
 	context->inject_page_fault = kvm_inject_page_fault;
+	context->get_subpages = kvm_x86_ops->get_subpages;
+	context->set_subpages = kvm_x86_ops->set_subpages;
+	context->init_spp = kvm_x86_ops->init_spp;
 
 	if (!is_paging(vcpu)) {
 		context->nx = false;

@@ -5309,6 +5467,8 @@ void kvm_init_mmu(struct kvm_vcpu *vcpu, bool reset_roots)
 		uint i;
 
 		vcpu->arch.mmu->root_hpa = INVALID_PAGE;
+		if (!vcpu->kvm->arch.spp_active)
+			vcpu->arch.mmu->sppt_root = INVALID_PAGE;
 
 		for (i = 0; i < KVM_MMU_NUM_PREV_ROOTS; i++)
 			vcpu->arch.mmu->prev_roots[i] = KVM_MMU_ROOT_INFO_INVALID;

arch/x86/kvm/vmx/vmx.c +43 -0

index a0753a36155d..855d02dd94c7 100644
@@ -2847,11 +2847,17 @@ u64 construct_eptp(struct kvm_vcpu *vcpu, unsigned long root_hpa)
 	return eptp;
 }
 
+static inline u64 construct_spptp(unsigned long root_hpa)
+{
+	return root_hpa & PAGE_MASK;
+}
+
 void vmx_set_cr3(struct kvm_vcpu *vcpu, unsigned long cr3)
 {
 	struct kvm *kvm = vcpu->kvm;
 	unsigned long guest_cr3;
 	u64 eptp;
+	u64 spptp;
 
 	guest_cr3 = cr3;
 	if (enable_ept) {

@@ -2874,6 +2880,12 @@ void vmx_set_cr3(struct kvm_vcpu *vcpu, unsigned long cr3)
 		ept_load_pdptrs(vcpu);
 	}
 
+	if (kvm->arch.spp_active && VALID_PAGE(vcpu->arch.mmu->sppt_root)) {
+		spptp = construct_spptp(vcpu->arch.mmu->sppt_root);
+		vmcs_write64(SPPT_POINTER, spptp);
+		vmx_flush_tlb(vcpu, true);
+	}
+
 	vmcs_writel(GUEST_CR3, guest_cr3);
 }
 

@@ -5735,6 +5747,9 @@ void dump_vmcs(void)
 		pr_err("PostedIntrVec = 0x%02x\n", vmcs_read16(POSTED_INTR_NV));
 	if ((secondary_exec_control & SECONDARY_EXEC_ENABLE_EPT))
 		pr_err("EPT pointer = 0x%016llx\n", vmcs_read64(EPT_POINTER));
+	if ((secondary_exec_control & SECONDARY_EXEC_ENABLE_SPP))
+		pr_err("SPPT pointer = 0x%016llx\n", vmcs_read64(SPPT_POINTER));
+
 	n = vmcs_read32(CR3_TARGET_COUNT);
 	for (i = 0; i + 1 < n; i += 4)
 		pr_err("CR3 target%u=%016lx target%u=%016lx\n",

@@ -7546,6 +7561,12 @@ static __init int hardware_setup(void)
 		kvm_x86_ops->enable_log_dirty_pt_masked = NULL;
 	}
 
+	if (!spp_supported) {
+		kvm_x86_ops->get_subpages = NULL;
+		kvm_x86_ops->set_subpages = NULL;
+		kvm_x86_ops->init_spp = NULL;
+	}
+
 	if (!cpu_has_vmx_preemption_timer())
 		kvm_x86_ops->request_immediate_exit = __kvm_request_immediate_exit;
 

@@ -7592,6 +7613,28 @@ static __exit void hardware_unsetup(void)
 	free_kvm_area();
 }
 
+static bool vmx_get_spp_status(void)
+{
+	return spp_supported;
+}
+
+static int vmx_get_subpages(struct kvm *kvm,
+			    struct kvm_subpage *spp_info)
+{
+	return kvm_get_subpages(kvm, spp_info);
+}
+
+static int vmx_set_subpages(struct kvm *kvm,
+			    struct kvm_subpage *spp_info)
+{
+	return kvm_set_subpages(kvm, spp_info);
+}
+
+static int vmx_init_spp(struct kvm *kvm)
+{
+	return kvm_init_spp(kvm);
+}
+
 static struct kvm_x86_ops vmx_x86_ops __ro_after_init = {
 	.cpu_has_kvm_support = cpu_has_kvm_support,
 	.disabled_by_bios = vmx_disabled_by_bios,

arch/x86/kvm/vmx/vmx.c +0 -0

 static void vmx_cleanup_l1d_flush(void)

arch/x86/kvm/x86.c +40 -0

index 26e4c574731c..c7cb17941344 100644
@@ -4589,6 +4589,44 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
 	return r;
 }
 
+int kvm_get_subpages(struct kvm *kvm,
+		     struct kvm_subpage *spp_info)
+{
+	int ret;
+
+	mutex_lock(&kvm->slots_lock);
+	ret = kvm_mmu_get_subpages(kvm, spp_info, false);
+	mutex_unlock(&kvm->slots_lock);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(kvm_get_subpages);
+
+int kvm_set_subpages(struct kvm *kvm,
+		     struct kvm_subpage *spp_info)
+{
+	int ret;
+
+	mutex_lock(&kvm->slots_lock);
+	ret = kvm_mmu_set_subpages(kvm, spp_info, false);
+	mutex_unlock(&kvm->slots_lock);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(kvm_set_subpages);
+
+int kvm_init_spp(struct kvm *kvm)
+{
+	int ret;
+
+	mutex_lock(&kvm->slots_lock);
+	ret = kvm_mmu_init_spp(kvm);
+	mutex_unlock(&kvm->slots_lock);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(kvm_init_spp);
+
 long kvm_arch_vm_ioctl(struct file *filp,
 		       unsigned int ioctl, unsigned long arg)
 {

@@ -9349,6 +9387,8 @@ void kvm_arch_free_memslot(struct kvm *kvm, struct kvm_memory_slot *free,
 	}
 
 	kvm_page_track_free_memslot(free, dont);
+	if (kvm->arch.spp_active)
+	      kvm_subpage_free_memslot(free, dont);
 }
 
 int kvm_arch_create_memslot(struct kvm *kvm, struct kvm_memory_slot *slot,

include/linux/kvm_host.h +3 -1

index 7f904d4d788c..da30fcbb2727 100644
@@ -849,7 +849,9 @@ int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu);
 
 struct kvm_mmu_page *kvm_mmu_get_spp_page(struct kvm_vcpu *vcpu,
 			gfn_t gfn, unsigned int level);
-
+int kvm_get_subpages(struct kvm *kvm, struct kvm_subpage *spp_info);
+int kvm_set_subpages(struct kvm *kvm, struct kvm_subpage *spp_info);
+int kvm_init_spp(struct kvm *kvm);
 #ifndef __KVM_HAVE_ARCH_VM_ALLOC
 /*
  * All architectures that want to use vzalloc currently also

include/uapi/linux/kvm.h +9 -0

index 6d4ea4b6c922..2c75a87ab3b5 100644
@@ -102,6 +102,15 @@ struct kvm_userspace_memory_region {
 	__u64 userspace_addr; /* start of the userspace allocated memory */
 };
 
+/* for KVM_SUBPAGES_GET_ACCESS and KVM_SUBPAGES_SET_ACCESS */
+#define SUBPAGE_MAX_BITMAP   64
+struct kvm_subpage {
+	__u64 base_gfn;
+	__u64 npages;
+	 /* sub-page write-access bitmap array */
+	__u32 access_map[SUBPAGE_MAX_BITMAP];
+};
+
 /*
  * The bit 0 ~ bit 15 of kvm_memory_region::flags are visible for userspace,
  * other bits are reserved for kvm internal use which are defined in

Quoted 9 lines · Yang Weijiang

static struct kvm_x86_ops vmx_x86_ops __ro_after_init = {

arch/x86/kvm/vmx/vmx.c +2 -0

 	.cpu_has_kvm_support = cpu_has_kvm_support,
 	.disabled_by_bios = vmx_disabled_by_bios,

@@ -7740,6 +7783,11 @@ static struct kvm_x86_ops vmx_x86_ops __ro_after_init = {
 	.get_vmcs12_pages = NULL,
 	.nested_enable_evmcs = NULL,
 	.need_emulation_on_page_fault = vmx_need_emulation_on_page_fault,
+
+	.get_spp_status = vmx_get_spp_status,

arch/x86/kvm/vmx/vmx.c static struct kvm_x86_ops vmx_x86_ops __ro_after_init = { · line 7791

Quoted 4 lines

arch/x86/kvm/vmx/vmx.c +3 -0

+	.get_subpages = vmx_get_subpages,
+	.set_subpages = vmx_set_subpages,
+	.init_spp = vmx_init_spp,
 };

There's no need for the get_subpages kvm_x86_ops. You do need init_spp of course, but you do not need get_spp_status either; instead you can check if init_spp is NULL (if NULL, SPP is not supported).

In addition, kvm_mmu_set_subpages should not set up the SPP pages. This should be handled entirely by handle_spp when the processor reports an SPPT miss, so you do not need that callback either. You may need a flush_subpages callback, called by kvm_mmu_set_subpages to clear the SPP page tables for a given GPA range. The next access then will cause an SPPT miss.

Finally, please move all SPP-related code to arch/x86/kvm/vmx/{spp.c,spp.h}.

Thanks,

Paolo

arch/x86/kvm/mmu.c +7 -0

+				fault_handled = true;
+				vcpu->run->exit_reason = KVM_EXIT_SPP;
+				vcpu->run->spp.addr = gva;
+				kvm_skip_emulated_instruction(vcpu);
+				break;
+			}
+
 			new_spte |= PT_WRITABLE_MASK;

arch/x86/kvm/mmu.c static bool fast_page_fault(struct kvm_vcpu *vcpu, gva_t gva, int level, · line 3610

Quoted 6 lines

arch/x86/kvm/mmu.c +6 -0

+			/*
+			 * Record write protect fault caused by
+			 * Sub-page Protection, let VMI decide
+			 * the next step.
+			 */
+			if (spte & PT_SPP_MASK) {

Should this be "if (spte & PT_WRITABLE_MASK)" instead? That is, if the page is already writable, the fault must be an SPP fault.

Quoted 8 lines · Paolo Bonzini

arch/x86/kvm/mmu.c static bool fast_page_fault(struct kvm_vcpu *vcpu, gva_t gva, int level, · line 3610

Quoted 6 lines

arch/x86/kvm/mmu.c +6 -0

+			/*
+			 * Record write protect fault caused by
+			 * Sub-page Protection, let VMI decide
+			 * the next step.
+			 */
+			if (spte & PT_SPP_MASK) {

Should this be "if (spte & PT_WRITABLE_MASK)" instead? That is, if the page is already writable, the fault must be an SPP fault.

Hmm, no I forgot how SPP works; still, this is *not* correct. For example, if SPP marks part of a page as read-write, but KVM wants to write-protect the whole page for access or dirty tracking, that should not cause an SPP exit.

So I think that when KVM wants to write-protect the whole page (wrprot_ad_disabled_spte) it must also clear PT_SPP_MASK; for example it could save it in bit 53 (PT64_SECOND_AVAIL_BITS_SHIFT + 1). If the saved bit is set, fast_page_fault must then set PT_SPP_MASK instead of PT_WRITABLE_MASK.

On re-entry this will cause an SPP vmexit;
fast_page_fault should never trigger an SPP userspace exit on its own, all the SPP handling should go through handle_spp.

Paolo

Quoted 15 lines · Paolo Bonzini

Quoted 8 lines · Paolo Bonzini

arch/x86/kvm/mmu.c static bool fast_page_fault(struct kvm_vcpu *vcpu, gva_t gva, int level, · line 3610

Quoted 6 lines

arch/x86/kvm/mmu.c +6 -0

+			/*
+			 * Record write protect fault caused by
+			 * Sub-page Protection, let VMI decide
+			 * the next step.
+			 */
+			if (spte & PT_SPP_MASK) {

Should this be "if (spte & PT_WRITABLE_MASK)" instead? That is, if the page is already writable, the fault must be an SPP fault.

Hmm, no I forgot how SPP works; still, this is *not* correct. For example, if SPP marks part of a page as read-write, but KVM wants to write-protect the whole page for access or dirty tracking, that should not cause an SPP exit.

Quoted 5 lines

So I think that when KVM wants to write-protect the whole page (wrprot_ad_disabled_spte) it must also clear PT_SPP_MASK; for example it could save it in bit 53 (PT64_SECOND_AVAIL_BITS_SHIFT + 1). If the saved bit is set, fast_page_fault must then set PT_SPP_MASK instead of PT_WRITABLE_MASK.

Sure, will change the processing flow.

Quoted 3 lines

On re-entry this will cause an SPP vmexit;
fast_page_fault should never trigger an SPP userspace exit on its own, all the SPP handling should go through handle_spp.

Quoted 1 line

Paolo

Quoted 3 lines · Yang Weijiang

Quoted 2 lines · Paolo Bonzini

fast_page_fault should never trigger an SPP userspace exit on its own, all the SPP handling should go through handle_spp.

Hi, Paolo,
According to the latest SDM(28.2.4), handle_spp only handles SPPT miss and SPPT misconfig(exit_reason==66), subpage write access violation causes EPT violation, so have to deal with the two cases into handlers.

Quoted 5 lines

Quoted 5 lines

So I think that when KVM wants to write-protect the whole page (wrprot_ad_disabled_spte) it must also clear PT_SPP_MASK; for example it could save it in bit 53 (PT64_SECOND_AVAIL_BITS_SHIFT + 1). If the saved bit is set, fast_page_fault must then set PT_SPP_MASK instead of PT_WRITABLE_MASK.

Quoted 8 lines · Yang Weijiang

Quoted 3 lines · Yang Weijiang

Quoted 2 lines · Paolo Bonzini

fast_page_fault should never trigger an SPP userspace exit on its own, all the SPP handling should go through handle_spp.

Hi, Paolo,
According to the latest SDM(28.2.4), handle_spp only handles SPPT miss and SPPT misconfig(exit_reason==66), subpage write access violation causes EPT violation, so have to deal with the two cases into handlers.

Ok, so this part has to remain, though you do have to save/restore PT_SPP_MASK according to the rest of the email.

Paolo

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

So I think that when KVM wants to write-protect the whole page (wrprot_ad_disabled_spte) it must also clear PT_SPP_MASK; for example it could save it in bit 53 (PT64_SECOND_AVAIL_BITS_SHIFT + 1). If the saved bit is set, fast_page_fault must then set PT_SPP_MASK instead of PT_WRITABLE_MASK.

arch/x86/kvm/mmu.c static int tdp_page_fault(struct kvm_vcpu *vcpu, gva_t gpa, u32 error_code, · line 4381

Quoted 4 lines · Yang Weijiang

arch/x86/kvm/mmu.c +4 -0

+
+	if (vcpu->kvm->arch.spp_active && level == PT_PAGE_TABLE_LEVEL)
+		kvm_enable_spp_protection(vcpu->kvm, gfn);
+

This would not enable SPP if the guest is backed by huge pages. Instead, either the PT_PAGE_TABLE_LEVEL level must be forced for all pages covered by SPP ranges, or (better) kvm_enable_spp_protection must be able to cover multiple pages at once.

Paolo

Quoted 11 lines · Paolo Bonzini

arch/x86/kvm/mmu.c static int tdp_page_fault(struct kvm_vcpu *vcpu, gva_t gpa, u32 error_code, · line 4381

Quoted 4 lines · Yang Weijiang

arch/x86/kvm/mmu.c +4 -0

+
+	if (vcpu->kvm->arch.spp_active && level == PT_PAGE_TABLE_LEVEL)
+		kvm_enable_spp_protection(vcpu->kvm, gfn);
+

This would not enable SPP if the guest is backed by huge pages. Instead, either the PT_PAGE_TABLE_LEVEL level must be forced for all pages covered by SPP ranges, or (better) kvm_enable_spp_protection must be able to cover multiple pages at once.

Quoted 1 line

Paolo

OK, I'll figure out how to make it, thanks!

Quoted 5 lines

Quoted 5 lines

This would not enable SPP if the guest is backed by huge pages. Instead, either the PT_PAGE_TABLE_LEVEL level must be forced for all pages covered by SPP ranges, or (better) kvm_enable_spp_protection must be able to cover multiple pages at once.

Quoted 2 lines

Quoted 1 line

Paolo

OK, I'll figure out how to make it, thanks!

Hi, Paolo,
Regarding this change, I have some concerns, splitting EPT huge page entries(e.g., 1GB page)will take long time compared with normal EPT page fault processing, especially for multiple vcpus/pages,so the in-flight time increases, but HW walks EPT for translations in the meantime, would it bring any side effect? or there's a way to mitigate it?

Quoted 5 lines · Yang Weijiang

Quoted 5 lines

Quoted 5 lines

This would not enable SPP if the guest is backed by huge pages. Instead, either the PT_PAGE_TABLE_LEVEL level must be forced for all pages covered by SPP ranges, or (better) kvm_enable_spp_protection must be able to cover multiple pages at once.

Quoted 8 lines

Quoted 2 lines

Quoted 1 line

Paolo

OK, I'll figure out how to make it, thanks!

Hi, Paolo,
Regarding this change, I have some concerns, splitting EPT huge page entries(e.g., 1GB page)will take long time compared with normal EPT page fault processing, especially for multiple vcpus/pages,so the in-flight time increases, but HW walks EPT for translations in the meantime, would it bring any side effect? or there's a way to mitigate it?

Sub-page permissions are only defined on EPT PTEs, not on large pages. Therefore, in order to allow subpage permissions the EPT page tables must already be split.

Paolo

EPT-Based Sub-Page write Protection(SPP)is a HW capability which allows Virtual Machine Monitor(VMM) to specify write-permission for guest physical memory at a sub-page(128 byte) granularity. When this capability is enabled, the CPU enforces write-access check for sub-pages within a 4KB page.

The feature is targeted to provide fine-grained memory protection for usages such as device virtualization, memory check-point and VM introspection etc.

SPP is active when the "sub-page write protection" (bit 23) is 1 in Secondary VM-Execution Controls. The feature is backed with a Sub-Page Permission Table(SPPT), SPPT is referenced via a 64-bit control field called Sub-Page Permission Table Pointer (SPPTP) which contains a 4K-aligned physical address.

Right now, only 4KB physical pages are supported for SPP. To enable SPP for certain physical page, we need to first make the physical page write-protected, then set bit 61 of the corresponding EPT leaf entry. While HW walks EPT, if bit 61 is set, it traverses SPPT with the guset physical address to find out the sub-page permissions at the leaf entry. If the corresponding bit is set, write to sub-page is permitted, otherwise, SPP induced EPT violation is generated.

Quoted 16 lines · Yang Weijiang

EPT-Based Sub-Page write Protection(SPP)is a HW capability which allows Virtual Machine Monitor(VMM) to specify write-permission for guest physical memory at a sub-page(128 byte) granularity. When this capability is enabled, the CPU enforces write-access check for sub-pages within a 4KB page.

The feature is targeted to provide fine-grained memory protection for usages such as device virtualization, memory check-point and VM introspection etc.

SPP is active when the "sub-page write protection" (bit 23) is 1 in Secondary VM-Execution Controls. The feature is backed with a Sub-Page Permission Table(SPPT), SPPT is referenced via a 64-bit control field called Sub-Page Permission Table Pointer (SPPTP) which contains a 4K-aligned physical address.

Quoted 7 lines

Right now, only 4KB physical pages are supported for SPP. To enable SPP for certain physical page, we need to first make the physical page write-protected, then set bit 61 of the corresponding EPT leaf entry. While HW walks EPT, if bit 61 is set, it traverses SPPT with the guset physical address to find out the sub-page permissions at the leaf entry. If the corresponding bit is set, write to sub-page is permitted, otherwise, SPP induced EPT violation is generated.

Still no testcases?

Paolo

Quoted 25 lines · Paolo Bonzini

Quoted 23 lines · Yang Weijiang

EPT-Based Sub-Page write Protection(SPP)is a HW capability which allows Virtual Machine Monitor(VMM) to specify write-permission for guest physical memory at a sub-page(128 byte) granularity. When this capability is enabled, the CPU enforces write-access check for sub-pages within a 4KB page.

The feature is targeted to provide fine-grained memory protection for usages such as device virtualization, memory check-point and VM introspection etc.

SPP is active when the "sub-page write protection" (bit 23) is 1 in Secondary VM-Execution Controls. The feature is backed with a Sub-Page Permission Table(SPPT), SPPT is referenced via a 64-bit control field called Sub-Page Permission Table Pointer (SPPTP) which contains a 4K-aligned physical address.

Right now, only 4KB physical pages are supported for SPP. To enable SPP for certain physical page, we need to first make the physical page write-protected, then set bit 61 of the corresponding EPT leaf entry. While HW walks EPT, if bit 61 is set, it traverses SPPT with the guset physical address to find out the sub-page permissions at the leaf entry. If the corresponding bit is set, write to sub-page is permitted, otherwise, SPP induced EPT violation is generated.

Quoted 3 lines

Still no testcases?

Paolo

Hi, Paolo,
The testcases are included in selftest: https://lkml.org/lkml/2019/6/18/1197