Skip to contentSkip to conversation

KVM: mm: fd-based approach for supporting KVM guest private memory

PATCH Superseded Chao Peng 36 participants 398 messages Jul 6, 2022 – Jul 15, 2023 Page 3 of 4

Subsystems (23)
lore
Share & tools
lore
#253 lore raw

Re: [PATCH v10 3/9] KVM: Extend the memslot to support fd-based private memory

In reply to Chao Peng
Quoted 21 lines · Chao Peng
Quoted 20 lines · Sean Christopherson
Quoted 18 lines · Chao Peng
Quoted 10 lines · Jarkko Sakkinen
Quoted 3 lines · Chao Peng

To make future maintenance easy, internally use a binary compatible alias struct kvm_user_mem_region to handle both the normal and the '_ext' variants.

Feels bit hacky IMHO, and more like a completely new feature than an extension.

Why not just add a new ioctl? The commit message does not address the most essential design here.

Yes, people can always choose to add a new ioctl for this kind of change and the balance point here is we want to also avoid 'too many ioctls' if the functionalities are similar. The '_ext' variant reuses all the existing fields in the 'normal' variant and most importantly KVM internally can reuse most of the code. I certainly can add some words in the commit message to explain this design choice.

Quoted 6 lines
Quoted 4 lines

After seeing the userspace side of this, I agree with Jarkko; overloading KVM_SET_USER_MEMORY_REGION is a hack. E.g. the size validation ends up being bogus, and userspace ends up abusing unions or implementing kvm_user_mem_region itself.

How is the size validation being bogus? I don't quite follow.

The ioctl() magic embeds the size of the payload (struct kvm_userspace_memory_region in this case) in the ioctl() number, and that information is visible to userspace via _IOCTL_SIZE(). Attempting to take a larger size can mess up sanity checks, e.g. KVM selftests get tripped up on this assert if KVM_SET_USER_MEMORY_REGION is passed an "extended" struct.

	#define kvm_do_ioctl(fd, cmd, arg)						\
	({										\
		kvm_static_assert(!_IOC_SIZE(cmd) || sizeof(*arg) == _IOC_SIZE(cmd));	\
		ioctl(fd, cmd, arg);							\
	})
Quoted 2 lines

Then we will use kvm_userspace_memory_region2 as the KVM internal alias, right?

Yep.

Quoted 10 lines

I see similar examples use different functions to handle different versions but it does look easier if we use alias for this function.

Quoted 5 lines

It feels absolutely ridiculous, but I think the best option is to do:

#define KVM_SET_USER_MEMORY_REGION2 _IOW(KVMIO, 0x49, \
struct kvm_userspace_memory_region2)

Just interesting, is 0x49 a safe number we can use?

Yes? So long as its not used by KVM, it's safe. AFAICT, it's unused.

author
#254 lore raw

Re: [PATCH v10 3/9] KVM: Extend the memslot to support fd-based private memory

Quoted 35 lines · Sean Christopherson
Quoted 21 lines · Chao Peng
Quoted 20 lines · Sean Christopherson
Quoted 18 lines · Chao Peng
Quoted 10 lines · Jarkko Sakkinen
Quoted 3 lines · Chao Peng

To make future maintenance easy, internally use a binary compatible alias struct kvm_user_mem_region to handle both the normal and the '_ext' variants.

Feels bit hacky IMHO, and more like a completely new feature than an extension.

Why not just add a new ioctl? The commit message does not address the most essential design here.

Yes, people can always choose to add a new ioctl for this kind of change and the balance point here is we want to also avoid 'too many ioctls' if the functionalities are similar. The '_ext' variant reuses all the existing fields in the 'normal' variant and most importantly KVM internally can reuse most of the code. I certainly can add some words in the commit message to explain this design choice.

Quoted 6 lines
Quoted 4 lines

After seeing the userspace side of this, I agree with Jarkko; overloading KVM_SET_USER_MEMORY_REGION is a hack. E.g. the size validation ends up being bogus, and userspace ends up abusing unions or implementing kvm_user_mem_region itself.

How is the size validation being bogus? I don't quite follow.

The ioctl() magic embeds the size of the payload (struct kvm_userspace_memory_region in this case) in the ioctl() number, and that information is visible to userspace via _IOCTL_SIZE(). Attempting to take a larger size can mess up sanity checks, e.g. KVM selftests get tripped up on this assert if KVM_SET_USER_MEMORY_REGION is passed an "extended" struct.

Quoted 5 lines
	#define kvm_do_ioctl(fd, cmd, arg)						\
	({										\
		kvm_static_assert(!_IOC_SIZE(cmd) || sizeof(*arg) == _IOC_SIZE(cmd));	\
		ioctl(fd, cmd, arg);							\
	})

Got it. Thanks for the explanation.

Chao

#255 lore raw

Re: [PATCH v10 3/9] KVM: Extend the memslot to support fd-based private memory

In reply to Chao Peng
Quoted 53 lines · Chao Peng
Quoted 20 lines · Sean Christopherson
Quoted 12 lines · Chao Peng
Quoted 10 lines · Jarkko Sakkinen
Quoted 3 lines · Chao Peng

To make future maintenance easy, internally use a binary compatible alias struct kvm_user_mem_region to handle both the normal and the '_ext' variants.

Feels bit hacky IMHO, and more like a completely new feature than an extension.

Why not just add a new ioctl? The commit message does not address the most essential design here.

Quoted 6 lines

Yes, people can always choose to add a new ioctl for this kind of change and the balance point here is we want to also avoid 'too many ioctls' if the functionalities are similar. The '_ext' variant reuses all the existing fields in the 'normal' variant and most importantly KVM internally can reuse most of the code. I certainly can add some words in the commit message to explain this design choice.

Quoted 4 lines

After seeing the userspace side of this, I agree with Jarkko; overloading KVM_SET_USER_MEMORY_REGION is a hack. E.g. the size validation ends up being bogus, and userspace ends up abusing unions or implementing kvm_user_mem_region itself.

How is the size validation being bogus? I don't quite follow. Then we will use kvm_userspace_memory_region2 as the KVM internal alias, right? I see similar examples use different functions to handle different versions but it does look easier if we use alias for this function.

Quoted 5 lines

It feels absolutely ridiculous, but I think the best option is to do:

#define KVM_SET_USER_MEMORY_REGION2 _IOW(KVMIO, 0x49, \
struct kvm_userspace_memory_region2)

Just interesting, is 0x49 a safe number we can use?

Quoted 14 lines

/* for KVM_SET_USER_MEMORY_REGION2 */

struct kvm_user_mem_region2 {
	__u32 slot;
	__u32 flags;
	__u64 guest_phys_addr;
	__u64 memory_size;
	__u64 userspace_addr;
	__u64 restricted_offset;
	__u32 restricted_fd;
	__u32 pad1;
	__u64 pad2[14];
}
Quoted 7 lines
Quoted 1 line

And it's consistent with other KVM ioctls(), e.g. KVM_SET_CPUID2.

Okay, agree from KVM userspace API perspective this is more consistent with similar existing examples. I see several of them.

I think we will also need a CAP_KVM_SET_USER_MEMORY_REGION2 for this new ioctl.

The current API in the patch set is trivial for C user space but for any other more "constrained" language such as Rust a new ioctl would be easier to adapt.

Quoted 6 lines
Quoted 5 lines

Regarding the userspace side of things, please include Vishal's selftests in v11, it's impossible to properly review the uAPI changes without seeing the userspace side of things. I'm in the process of reviewing Vishal's v2[*], I'll try to massage it into a set of patches that you can incorporate into your series.

Quoted 7 lines

Previously I included Vishal's selftests in the github repo, but not include them in this patch series. It's OK for me to incorporate them directly into this series and review together if Vishal is fine.

Chao

Quoted 2 lines

[*] https://lore.kernel.org/all/[email protected]

BR, Jarkko

#256 lore raw

Re: [PATCH v10 3/9] KVM: Extend the memslot to support fd-based private memory

In reply to Sean Christopherson
Quoted 43 lines · Sean Christopherson
Quoted 12 lines · Chao Peng
Quoted 10 lines · Jarkko Sakkinen
Quoted 3 lines · Chao Peng

To make future maintenance easy, internally use a binary compatible alias struct kvm_user_mem_region to handle both the normal and the '_ext' variants.

Feels bit hacky IMHO, and more like a completely new feature than an extension.

Why not just add a new ioctl? The commit message does not address the most essential design here.

Quoted 6 lines

Yes, people can always choose to add a new ioctl for this kind of change and the balance point here is we want to also avoid 'too many ioctls' if the functionalities are similar. The '_ext' variant reuses all the existing fields in the 'normal' variant and most importantly KVM internally can reuse most of the code. I certainly can add some words in the commit message to explain this design choice.

After seeing the userspace side of this, I agree with Jarkko; overloading KVM_SET_USER_MEMORY_REGION is a hack. E.g. the size validation ends up being bogus, and userspace ends up abusing unions or implementing kvm_user_mem_region itself.

It feels absolutely ridiculous, but I think the best option is to do:

#define KVM_SET_USER_MEMORY_REGION2 _IOW(KVMIO, 0x49, \
struct kvm_userspace_memory_region2)

/* for KVM_SET_USER_MEMORY_REGION2 */

struct kvm_user_mem_region2 {
	__u32 slot;
	__u32 flags;
	__u64 guest_phys_addr;
	__u64 memory_size;
	__u64 userspace_addr;
	__u64 restricted_offset;
	__u32 restricted_fd;
	__u32 pad1;
	__u64 pad2[14];
}
Quoted 8 lines

And it's consistent with other KVM ioctls(), e.g. KVM_SET_CPUID2.

Regarding the userspace side of things, please include Vishal's selftests in v11, it's impossible to properly review the uAPI changes without seeing the userspace side of things. I'm in the process of reviewing Vishal's v2[*], I'll try to massage it into a set of patches that you can incorporate into your series.

[*] https://lore.kernel.org/all/[email protected]

+1

BR, Jarkko

author
#257 lore raw

PATCH v10 4/9 KVM: Add KVM_EXIT_MEMORY_FAULT exit

In reply to Chao Peng

This new KVM exit allows userspace to handle memory-related errors. It indicates an error happens in KVM at guest memory range [gpa, gpa+size). The flags includes additional information for userspace to handle the error. Currently bit 0 is defined as 'private memory' where '1' indicates error happens due to private memory access and '0' indicates error happens due to shared memory access.

When private memory is enabled, this new exit will be used for KVM to exit to userspace for shared <-> private memory conversion in memory encryption usage. In such usage, typically there are two kind of memory conversions:
- explicit conversion: happens when guest explicitly calls into KVM to map a range (as private or shared), KVM then exits to userspace to perform the map/unmap operations.
- implicit conversion: happens in KVM page fault handler where KVM exits to userspace for an implicit conversion when the page is in a different state than requested (private or shared).

---
 Documentation/virt/kvm/api.rst | 22 ++++++++++++++++++++++
 include/uapi/linux/kvm.h       |  8 ++++++++
 2 files changed, 30 insertions(+)
Documentation/virt/kvm/api.rst +22 -0
index 99352170c130..d9edb14ce30b 100644
@@ -6634,6 +6634,28 @@ array field represents return values. The userspace should update the return
 values of SBI call before resuming the VCPU. For more details on RISC-V SBI
 spec refer, https://github.com/riscv/riscv-sbi-doc.
 
+::
+
+		/* KVM_EXIT_MEMORY_FAULT */
+		struct {
+  #define KVM_MEMORY_EXIT_FLAG_PRIVATE	(1ULL << 0)
+			__u64 flags;
+			__u64 gpa;
+			__u64 size;
+		} memory;
+
+If exit reason is KVM_EXIT_MEMORY_FAULT then it indicates that the VCPU has
+encountered a memory error which is not handled by KVM kernel module and
+userspace may choose to handle it. The 'flags' field indicates the memory
+properties of the exit.
+
+ - KVM_MEMORY_EXIT_FLAG_PRIVATE - indicates the memory error is caused by
+   private memory access when the bit is set. Otherwise the memory error is
+   caused by shared memory access when the bit is clear.
+
+'gpa' and 'size' indicate the memory range the error occurs at. The userspace
+may handle the error and return to KVM to retry the previous memory access.
+
 ::
 
     /* KVM_EXIT_NOTIFY */
include/uapi/linux/kvm.h +8 -0
index 13bff963b8b0..c7e9d375a902 100644
@@ -300,6 +300,7 @@ struct kvm_xen_exit {
 #define KVM_EXIT_RISCV_SBI        35
 #define KVM_EXIT_RISCV_CSR        36
 #define KVM_EXIT_NOTIFY           37
+#define KVM_EXIT_MEMORY_FAULT     38
 
 /* For KVM_EXIT_INTERNAL_ERROR */
 /* Emulate instruction failed. */
@@ -541,6 +542,13 @@ struct kvm_run {
 #define KVM_NOTIFY_CONTEXT_INVALID	(1 << 0)
 			__u32 flags;
 		} notify;
+		/* KVM_EXIT_MEMORY_FAULT */
+		struct {
+#define KVM_MEMORY_EXIT_FLAG_PRIVATE	(1ULL << 0)
+			__u64 flags;
+			__u64 gpa;
+			__u64 size;
+		} memory;
 		/* Fix the size of the union. */
 		char padding[256];
 	};
Signature · 1 line
2.25.1
Reviewed-by Tested-by on v10 4/9
#258 lore raw

Re: [PATCH v10 4/9] KVM: Add KVM_EXIT_MEMORY_FAULT exit

Hi,

Quoted 34 lines · Chao Peng

This new KVM exit allows userspace to handle memory-related errors. It indicates an error happens in KVM at guest memory range [gpa, gpa+size). The flags includes additional information for userspace to handle the error. Currently bit 0 is defined as 'private memory' where '1' indicates error happens due to private memory access and '0' indicates error happens due to shared memory access.

When private memory is enabled, this new exit will be used for KVM to exit to userspace for shared <-> private memory conversion in memory encryption usage. In such usage, typically there are two kind of memory conversions:
- explicit conversion: happens when guest explicitly calls into KVM to map a range (as private or shared), KVM then exits to userspace to perform the map/unmap operations.
- implicit conversion: happens in KVM page fault handler where KVM exits to userspace for an implicit conversion when the page is in a different state than requested (private or shared).

---
 Documentation/virt/kvm/api.rst | 22 ++++++++++++++++++++++
 include/uapi/linux/kvm.h       |  8 ++++++++
 2 files changed, 30 insertions(+)
Documentation/virt/kvm/api.rst +0 -0
index 99352170c130..d9edb14ce30b 100644

Documentation/virt/kvm/api.rst array field represents return values. The userspace should update the return · line 6642

Quoted 9 lines
Documentation/virt/kvm/api.rst +6 -0
@@ -6634,6 +6634,28 @@ array field represents return values. The userspace should update the return
 values of SBI call before resuming the VCPU. For more details on RISC-V SBI
 spec refer, https://github.com/riscv/riscv-sbi-doc.
 
+::
+
+               /* KVM_EXIT_MEMORY_FAULT */
+               struct {
+  #define KVM_MEMORY_EXIT_FLAG_PRIVATE (1ULL << 0)
+                       __u64 flags;

I see you've removed the padding and increased the flag size.

Cheers,
/fuad

Quoted 48 lines
Documentation/virt/kvm/api.rst +16 -0
+                       __u64 gpa;
+                       __u64 size;
+               } memory;
+
+If exit reason is KVM_EXIT_MEMORY_FAULT then it indicates that the VCPU has
+encountered a memory error which is not handled by KVM kernel module and
+userspace may choose to handle it. The 'flags' field indicates the memory
+properties of the exit.
+
+ - KVM_MEMORY_EXIT_FLAG_PRIVATE - indicates the memory error is caused by
+   private memory access when the bit is set. Otherwise the memory error is
+   caused by shared memory access when the bit is clear.
+
+'gpa' and 'size' indicate the memory range the error occurs at. The userspace
+may handle the error and return to KVM to retry the previous memory access.
+
 ::
 
     /* KVM_EXIT_NOTIFY */
include/uapi/linux/kvm.h +8 -1
index 13bff963b8b0..c7e9d375a902 100644
@@ -300,6 +300,7 @@ struct kvm_xen_exit {
 #define KVM_EXIT_RISCV_SBI        35
 #define KVM_EXIT_RISCV_CSR        36
 #define KVM_EXIT_NOTIFY           37
+#define KVM_EXIT_MEMORY_FAULT     38
 
 /* For KVM_EXIT_INTERNAL_ERROR */
 /* Emulate instruction failed. */
@@ -541,6 +542,13 @@ struct kvm_run {
 #define KVM_NOTIFY_CONTEXT_INVALID     (1 << 0)
                        __u32 flags;
                } notify;
+               /* KVM_EXIT_MEMORY_FAULT */
+               struct {
+#define KVM_MEMORY_EXIT_FLAG_PRIVATE   (1ULL << 0)
+                       __u64 flags;
+                       __u64 gpa;
+                       __u64 size;
+               } memory;
                /* Fix the size of the union. */
                char padding[256];
        };
--

2.25.1

author
#259 lore raw

Re: [PATCH v10 4/9] KVM: Add KVM_EXIT_MEMORY_FAULT exit

Quoted 40 lines · Fuad Tabba

Hi,

Quoted 37 lines · Chao Peng

This new KVM exit allows userspace to handle memory-related errors. It indicates an error happens in KVM at guest memory range [gpa, gpa+size). The flags includes additional information for userspace to handle the error. Currently bit 0 is defined as 'private memory' where '1' indicates error happens due to private memory access and '0' indicates error happens due to shared memory access.

When private memory is enabled, this new exit will be used for KVM to exit to userspace for shared <-> private memory conversion in memory encryption usage. In such usage, typically there are two kind of memory conversions:
- explicit conversion: happens when guest explicitly calls into KVM to map a range (as private or shared), KVM then exits to userspace to perform the map/unmap operations.
- implicit conversion: happens in KVM page fault handler where KVM exits to userspace for an implicit conversion when the page is in a different state than requested (private or shared).

---
 Documentation/virt/kvm/api.rst | 22 ++++++++++++++++++++++
 include/uapi/linux/kvm.h       |  8 ++++++++
 2 files changed, 30 insertions(+)
Documentation/virt/kvm/api.rst +0 -0
index 99352170c130..d9edb14ce30b 100644
@@ -6634,6 +6634,28 @@ array field represents return values. The userspace should update the return
 values of SBI call before resuming the VCPU. For more details on RISC-V SBI
 spec refer, https://github.com/riscv/riscv-sbi-doc.
 
Quoted 8 lines

Documentation/virt/kvm/api.rst array field represents return values. The userspace should update the return · line 6642

Quoted 6 lines
Documentation/virt/kvm/api.rst +6 -0
+::
+
+               /* KVM_EXIT_MEMORY_FAULT */
+               struct {
+  #define KVM_MEMORY_EXIT_FLAG_PRIVATE (1ULL << 0)
+                       __u64 flags;

I see you've removed the padding and increased the flag size.

Yes Sean suggested this and also looks good to me.

Chao

Quoted 58 lines

Cheers,
/fuad

Quoted 48 lines
Documentation/virt/kvm/api.rst +16 -0
+                       __u64 gpa;
+                       __u64 size;
+               } memory;
+
+If exit reason is KVM_EXIT_MEMORY_FAULT then it indicates that the VCPU has
+encountered a memory error which is not handled by KVM kernel module and
+userspace may choose to handle it. The 'flags' field indicates the memory
+properties of the exit.
+
+ - KVM_MEMORY_EXIT_FLAG_PRIVATE - indicates the memory error is caused by
+   private memory access when the bit is set. Otherwise the memory error is
+   caused by shared memory access when the bit is clear.
+
+'gpa' and 'size' indicate the memory range the error occurs at. The userspace
+may handle the error and return to KVM to retry the previous memory access.
+
 ::
 
     /* KVM_EXIT_NOTIFY */
include/uapi/linux/kvm.h +8 -1
index 13bff963b8b0..c7e9d375a902 100644
@@ -300,6 +300,7 @@ struct kvm_xen_exit {
 #define KVM_EXIT_RISCV_SBI        35
 #define KVM_EXIT_RISCV_CSR        36
 #define KVM_EXIT_NOTIFY           37
+#define KVM_EXIT_MEMORY_FAULT     38
 
 /* For KVM_EXIT_INTERNAL_ERROR */
 /* Emulate instruction failed. */
@@ -541,6 +542,13 @@ struct kvm_run {
 #define KVM_NOTIFY_CONTEXT_INVALID     (1 << 0)
                        __u32 flags;
                } notify;
+               /* KVM_EXIT_MEMORY_FAULT */
+               struct {
+#define KVM_MEMORY_EXIT_FLAG_PRIVATE   (1ULL << 0)
+                       __u64 flags;
+                       __u64 gpa;
+                       __u64 size;
+               } memory;
                /* Fix the size of the union. */
                char padding[256];
        };
--

2.25.1

#260 lore raw

Re: [PATCH v10 4/9] KVM: Add KVM_EXIT_MEMORY_FAULT exit

In reply to Chao Peng

Documentation/virt/kvm/api.rst array field represents return values. The userspace should update the return · line 6641

Quoted 13 lines · Chao Peng
Documentation/virt/kvm/api.rst +5 -0
index 99352170c130..d9edb14ce30b 100644
@@ -6634,6 +6634,28 @@ array field represents return values. The userspace should update the return
 values of SBI call before resuming the VCPU. For more details on RISC-V SBI
 spec refer, https://github.com/riscv/riscv-sbi-doc.
 
+::
+
+		/* KVM_EXIT_MEMORY_FAULT */
+		struct {
+  #define KVM_MEMORY_EXIT_FLAG_PRIVATE	(1ULL << 0)

Unless there's a reason not to, we should use bit 3 to match the attributes.

author
#261 lore raw

PATCH v10 5/9 KVM: Use gfn instead of hva for mmu_notifier_retry

In reply to Chao Peng

Currently in mmu_notifier invalidate path, hva range is recorded and then checked against by mmu_notifier_retry_hva() in the page fault handling path. However, for the to be introduced private memory, a page fault may not have a hva associated, checking gfn(gpa) makes more sense.

For existing hva based shared memory, gfn is expected to also work. The only downside is when aliasing multiple gfns to a single hva, the current algorithm of checking multiple ranges could result in a much larger range being rejected. Such aliasing should be uncommon, so the impact is expected small.

---
 arch/x86/kvm/mmu/mmu.c   |  8 +++++---
 include/linux/kvm_host.h | 33 +++++++++++++++++++++------------
 virt/kvm/kvm_main.c      | 32 +++++++++++++++++++++++---------
 3 files changed, 49 insertions(+), 24 deletions(-)
arch/x86/kvm/mmu/mmu.c +5 -3
index 4736d7849c60..e2c70b5afa3e 100644
@@ -4259,7 +4259,7 @@ static bool is_page_fault_stale(struct kvm_vcpu *vcpu,
 		return true;
 
 	return fault->slot &&
-	       mmu_invalidate_retry_hva(vcpu->kvm, mmu_seq, fault->hva);
+	       mmu_invalidate_retry_gfn(vcpu->kvm, mmu_seq, fault->gfn);
 }
 
 static int direct_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
@@ -6098,7 +6098,9 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end)
 
 	write_lock(&kvm->mmu_lock);
 
-	kvm_mmu_invalidate_begin(kvm, gfn_start, gfn_end);
+	kvm_mmu_invalidate_begin(kvm);
+
+	kvm_mmu_invalidate_range_add(kvm, gfn_start, gfn_end);
 
 	flush = kvm_rmap_zap_gfn_range(kvm, gfn_start, gfn_end);
 
@@ -6112,7 +6114,7 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end)
 		kvm_flush_remote_tlbs_with_address(kvm, gfn_start,
 						   gfn_end - gfn_start);
 
-	kvm_mmu_invalidate_end(kvm, gfn_start, gfn_end);
+	kvm_mmu_invalidate_end(kvm);
 
 	write_unlock(&kvm->mmu_lock);
 }
include/linux/kvm_host.h +21 -12
index 02347e386ea2..3d69484d2704 100644
@@ -787,8 +787,8 @@ struct kvm {
 	struct mmu_notifier mmu_notifier;
 	unsigned long mmu_invalidate_seq;
 	long mmu_invalidate_in_progress;
-	unsigned long mmu_invalidate_range_start;
-	unsigned long mmu_invalidate_range_end;
+	gfn_t mmu_invalidate_range_start;
+	gfn_t mmu_invalidate_range_end;
 #endif
 	struct list_head devices;
 	u64 manual_dirty_log_protect;
@@ -1389,10 +1389,9 @@ void kvm_mmu_free_memory_cache(struct kvm_mmu_memory_cache *mc);
 void *kvm_mmu_memory_cache_alloc(struct kvm_mmu_memory_cache *mc);
 #endif
 
-void kvm_mmu_invalidate_begin(struct kvm *kvm, unsigned long start,
-			      unsigned long end);
-void kvm_mmu_invalidate_end(struct kvm *kvm, unsigned long start,
-			    unsigned long end);
+void kvm_mmu_invalidate_begin(struct kvm *kvm);
+void kvm_mmu_invalidate_range_add(struct kvm *kvm, gfn_t start, gfn_t end);
+void kvm_mmu_invalidate_end(struct kvm *kvm);
 
 long kvm_arch_dev_ioctl(struct file *filp,
 			unsigned int ioctl, unsigned long arg);
@@ -1963,9 +1962,9 @@ static inline int mmu_invalidate_retry(struct kvm *kvm, unsigned long mmu_seq)
 	return 0;
 }
 
-static inline int mmu_invalidate_retry_hva(struct kvm *kvm,
+static inline int mmu_invalidate_retry_gfn(struct kvm *kvm,
 					   unsigned long mmu_seq,
-					   unsigned long hva)
+					   gfn_t gfn)
 {
 	lockdep_assert_held(&kvm->mmu_lock);
 	/*
@@ -1974,10 +1973,20 @@ static inline int mmu_invalidate_retry_hva(struct kvm *kvm,
 	 * that might be being invalidated. Note that it may include some false
 	 * positives, due to shortcuts when handing concurrent invalidations.
 	 */
-	if (unlikely(kvm->mmu_invalidate_in_progress) &&
-	    hva >= kvm->mmu_invalidate_range_start &&
-	    hva < kvm->mmu_invalidate_range_end)
-		return 1;
+	if (unlikely(kvm->mmu_invalidate_in_progress)) {
+		/*
+		 * Dropping mmu_lock after bumping mmu_invalidate_in_progress
+		 * but before updating the range is a KVM bug.
+		 */
+		if (WARN_ON_ONCE(kvm->mmu_invalidate_range_start == INVALID_GPA ||
+				 kvm->mmu_invalidate_range_end == INVALID_GPA))
+			return 1;
+
+		if (gfn >= kvm->mmu_invalidate_range_start &&
+		    gfn < kvm->mmu_invalidate_range_end)
+			return 1;
+	}
+
 	if (kvm->mmu_invalidate_seq != mmu_seq)
 		return 1;
 	return 0;
virt/kvm/kvm_main.c +23 -9
index b882eb2c76a2..ad55dfbc75d7 100644
@@ -540,9 +540,7 @@ static void kvm_mmu_notifier_invalidate_range(struct mmu_notifier *mn,
 
 typedef bool (*hva_handler_t)(struct kvm *kvm, struct kvm_gfn_range *range);
 
-typedef void (*on_lock_fn_t)(struct kvm *kvm, unsigned long start,
-			     unsigned long end);
-
+typedef void (*on_lock_fn_t)(struct kvm *kvm);
 typedef void (*on_unlock_fn_t)(struct kvm *kvm);
 
 struct kvm_hva_range {
@@ -628,7 +626,8 @@ static __always_inline int __kvm_handle_hva_range(struct kvm *kvm,
 				locked = true;
 				KVM_MMU_LOCK(kvm);
 				if (!IS_KVM_NULL_FN(range->on_lock))
-					range->on_lock(kvm, range->start, range->end);
+					range->on_lock(kvm);
+
 				if (IS_KVM_NULL_FN(range->handler))
 					break;
 			}
@@ -715,8 +714,7 @@ static void kvm_mmu_notifier_change_pte(struct mmu_notifier *mn,
 	kvm_handle_hva_range(mn, address, address + 1, pte, kvm_set_spte_gfn);
 }
 
-void kvm_mmu_invalidate_begin(struct kvm *kvm, unsigned long start,
-			      unsigned long end)
+void kvm_mmu_invalidate_begin(struct kvm *kvm)
 {
 	/*
 	 * The count increase must become visible at unlock time as no
@@ -724,6 +722,17 @@ void kvm_mmu_invalidate_begin(struct kvm *kvm, unsigned long start,
 	 * count is also read inside the mmu_lock critical section.
 	 */
 	kvm->mmu_invalidate_in_progress++;
+
+	if (likely(kvm->mmu_invalidate_in_progress == 1)) {
+		kvm->mmu_invalidate_range_start = INVALID_GPA;
+		kvm->mmu_invalidate_range_end = INVALID_GPA;
+	}
+}
+
+void kvm_mmu_invalidate_range_add(struct kvm *kvm, gfn_t start, gfn_t end)
+{
+	WARN_ON_ONCE(!kvm->mmu_invalidate_in_progress);
+
 	if (likely(kvm->mmu_invalidate_in_progress == 1)) {
 		kvm->mmu_invalidate_range_start = start;
 		kvm->mmu_invalidate_range_end = end;
@@ -744,6 +753,12 @@ void kvm_mmu_invalidate_begin(struct kvm *kvm, unsigned long start,
 	}
 }
 
+static bool kvm_mmu_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
+{
+	kvm_mmu_invalidate_range_add(kvm, range->start, range->end);
+	return kvm_unmap_gfn_range(kvm, range);
+}
+
 static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,
 					const struct mmu_notifier_range *range)
 {
@@ -752,7 +767,7 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,
 		.start		= range->start,
 		.end		= range->end,
 		.pte		= __pte(0),
-		.handler	= kvm_unmap_gfn_range,
+		.handler	= kvm_mmu_unmap_gfn_range,
 		.on_lock	= kvm_mmu_invalidate_begin,
 		.on_unlock	= kvm_arch_guest_memory_reclaimed,
 		.flush_on_ret	= true,
@@ -791,8 +806,7 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,
 	return 0;
 }
 
-void kvm_mmu_invalidate_end(struct kvm *kvm, unsigned long start,
-			    unsigned long end)
+void kvm_mmu_invalidate_end(struct kvm *kvm)
 {
 	/*
 	 * This sequence increase will notify the kvm page fault that
Signature · 1 line
2.25.1
#262 lore raw

Re: [PATCH v10 5/9] KVM: Use gfn instead of hva for mmu_notifier_retry

Hi Chao,

Quoted 95 lines · Chao Peng

Currently in mmu_notifier invalidate path, hva range is recorded and then checked against by mmu_notifier_retry_hva() in the page fault handling path. However, for the to be introduced private memory, a page fault may not have a hva associated, checking gfn(gpa) makes more sense.

For existing hva based shared memory, gfn is expected to also work. The only downside is when aliasing multiple gfns to a single hva, the current algorithm of checking multiple ranges could result in a much larger range being rejected. Such aliasing should be uncommon, so the impact is expected small.

---
 arch/x86/kvm/mmu/mmu.c   |  8 +++++---
 include/linux/kvm_host.h | 33 +++++++++++++++++++++------------
 virt/kvm/kvm_main.c      | 32 +++++++++++++++++++++++---------
 3 files changed, 49 insertions(+), 24 deletions(-)
arch/x86/kvm/mmu/mmu.c +5 -3
index 4736d7849c60..e2c70b5afa3e 100644
@@ -4259,7 +4259,7 @@ static bool is_page_fault_stale(struct kvm_vcpu *vcpu,
                return true;
 
        return fault->slot &&
-              mmu_invalidate_retry_hva(vcpu->kvm, mmu_seq, fault->hva);
+              mmu_invalidate_retry_gfn(vcpu->kvm, mmu_seq, fault->gfn);
 }
 
 static int direct_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
@@ -6098,7 +6098,9 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end)
 
        write_lock(&kvm->mmu_lock);
 
-       kvm_mmu_invalidate_begin(kvm, gfn_start, gfn_end);
+       kvm_mmu_invalidate_begin(kvm);
+
+       kvm_mmu_invalidate_range_add(kvm, gfn_start, gfn_end);
 
        flush = kvm_rmap_zap_gfn_range(kvm, gfn_start, gfn_end);
 
@@ -6112,7 +6114,7 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end)
                kvm_flush_remote_tlbs_with_address(kvm, gfn_start,
                                                   gfn_end - gfn_start);
 
-       kvm_mmu_invalidate_end(kvm, gfn_start, gfn_end);
+       kvm_mmu_invalidate_end(kvm);
 
        write_unlock(&kvm->mmu_lock);
 }
include/linux/kvm_host.h +7 -8
index 02347e386ea2..3d69484d2704 100644
@@ -787,8 +787,8 @@ struct kvm {
        struct mmu_notifier mmu_notifier;
        unsigned long mmu_invalidate_seq;
        long mmu_invalidate_in_progress;
-       unsigned long mmu_invalidate_range_start;
-       unsigned long mmu_invalidate_range_end;
+       gfn_t mmu_invalidate_range_start;
+       gfn_t mmu_invalidate_range_end;
 #endif
        struct list_head devices;
        u64 manual_dirty_log_protect;
@@ -1389,10 +1389,9 @@ void kvm_mmu_free_memory_cache(struct kvm_mmu_memory_cache *mc);
 void *kvm_mmu_memory_cache_alloc(struct kvm_mmu_memory_cache *mc);
 #endif
 
-void kvm_mmu_invalidate_begin(struct kvm *kvm, unsigned long start,
-                             unsigned long end);
-void kvm_mmu_invalidate_end(struct kvm *kvm, unsigned long start,
-                           unsigned long end);
+void kvm_mmu_invalidate_begin(struct kvm *kvm);
+void kvm_mmu_invalidate_range_add(struct kvm *kvm, gfn_t start, gfn_t end);
+void kvm_mmu_invalidate_end(struct kvm *kvm);
 
 long kvm_arch_dev_ioctl(struct file *filp,
                        unsigned int ioctl, unsigned long arg);
@@ -1963,9 +1962,9 @@ static inline int mmu_invalidate_retry(struct kvm *kvm, unsigned long mmu_seq)
        return 0;
 }
 
-static inline int mmu_invalidate_retry_hva(struct kvm *kvm,
+static inline int mmu_invalidate_retry_gfn(struct kvm *kvm,
                                           unsigned long mmu_seq,
-                                          unsigned long hva)
+                                          gfn_t gfn)
 {
        lockdep_assert_held(&kvm->mmu_lock);
        /*

include/linux/kvm_host.h static inline int mmu_invalidate_retry_hva(struct kvm *kvm, · line 1973

Quoted 1 line
include/linux/kvm_host.h +0 -0
@@ -1974,10 +1973,20 @@ static inline int mmu_invalidate_retry_hva(struct kvm *kvm,
         * that might be being invalidated. Note that it may include some false

nit: "might be" (or) "is being"

include/linux/kvm_host.h static inline int mmu_invalidate_retry_hva(struct kvm *kvm, · line 1974

Quoted 1 line
include/linux/kvm_host.h +0 -0
         * positives, due to shortcuts when handing concurrent invalidations.

nit: handling

include/linux/kvm_host.h static inline int mmu_invalidate_retry_hva(struct kvm *kvm, · line 1982

Quoted 12 lines
include/linux/kvm_host.h +7 -4
         */
-       if (unlikely(kvm->mmu_invalidate_in_progress) &&
-           hva >= kvm->mmu_invalidate_range_start &&
-           hva < kvm->mmu_invalidate_range_end)
-               return 1;
+       if (unlikely(kvm->mmu_invalidate_in_progress)) {
+               /*
+                * Dropping mmu_lock after bumping mmu_invalidate_in_progress
+                * but before updating the range is a KVM bug.
+                */
+               if (WARN_ON_ONCE(kvm->mmu_invalidate_range_start == INVALID_GPA ||
+                                kvm->mmu_invalidate_range_end == INVALID_GPA))

INVALID_GPA is an x86-specific define in arch/x86/include/asm/kvm_host.h, so this doesn't build on other architectures. The obvious fix is to move it to include/linux/kvm_host.h.

Cheers,
/fuad

Quoted 98 lines
include/linux/kvm_host.h +7 -0
+                       return 1;
+
+               if (gfn >= kvm->mmu_invalidate_range_start &&
+                   gfn < kvm->mmu_invalidate_range_end)
+                       return 1;
+       }
+
        if (kvm->mmu_invalidate_seq != mmu_seq)
                return 1;
        return 0;
virt/kvm/kvm_main.c +23 -10
index b882eb2c76a2..ad55dfbc75d7 100644
@@ -540,9 +540,7 @@ static void kvm_mmu_notifier_invalidate_range(struct mmu_notifier *mn,
 
 typedef bool (*hva_handler_t)(struct kvm *kvm, struct kvm_gfn_range *range);
 
-typedef void (*on_lock_fn_t)(struct kvm *kvm, unsigned long start,
-                            unsigned long end);
-
+typedef void (*on_lock_fn_t)(struct kvm *kvm);
 typedef void (*on_unlock_fn_t)(struct kvm *kvm);
 
 struct kvm_hva_range {
@@ -628,7 +626,8 @@ static __always_inline int __kvm_handle_hva_range(struct kvm *kvm,
                                locked = true;
                                KVM_MMU_LOCK(kvm);
                                if (!IS_KVM_NULL_FN(range->on_lock))
-                                       range->on_lock(kvm, range->start, range->end);
+                                       range->on_lock(kvm);
+
                                if (IS_KVM_NULL_FN(range->handler))
                                        break;
                        }
@@ -715,8 +714,7 @@ static void kvm_mmu_notifier_change_pte(struct mmu_notifier *mn,
        kvm_handle_hva_range(mn, address, address + 1, pte, kvm_set_spte_gfn);
 }
 
-void kvm_mmu_invalidate_begin(struct kvm *kvm, unsigned long start,
-                             unsigned long end)
+void kvm_mmu_invalidate_begin(struct kvm *kvm)
 {
        /*
         * The count increase must become visible at unlock time as no
@@ -724,6 +722,17 @@ void kvm_mmu_invalidate_begin(struct kvm *kvm, unsigned long start,
         * count is also read inside the mmu_lock critical section.
         */
        kvm->mmu_invalidate_in_progress++;
+
+       if (likely(kvm->mmu_invalidate_in_progress == 1)) {
+               kvm->mmu_invalidate_range_start = INVALID_GPA;
+               kvm->mmu_invalidate_range_end = INVALID_GPA;
+       }
+}
+
+void kvm_mmu_invalidate_range_add(struct kvm *kvm, gfn_t start, gfn_t end)
+{
+       WARN_ON_ONCE(!kvm->mmu_invalidate_in_progress);
+
        if (likely(kvm->mmu_invalidate_in_progress == 1)) {
                kvm->mmu_invalidate_range_start = start;
                kvm->mmu_invalidate_range_end = end;
@@ -744,6 +753,12 @@ void kvm_mmu_invalidate_begin(struct kvm *kvm, unsigned long start,
        }
 }
 
+static bool kvm_mmu_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
+{
+       kvm_mmu_invalidate_range_add(kvm, range->start, range->end);
+       return kvm_unmap_gfn_range(kvm, range);
+}
+
 static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,
                                        const struct mmu_notifier_range *range)
 {
@@ -752,7 +767,7 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,
                .start          = range->start,
                .end            = range->end,
                .pte            = __pte(0),
-               .handler        = kvm_unmap_gfn_range,
+               .handler        = kvm_mmu_unmap_gfn_range,
                .on_lock        = kvm_mmu_invalidate_begin,
                .on_unlock      = kvm_arch_guest_memory_reclaimed,
                .flush_on_ret   = true,
@@ -791,8 +806,7 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,
        return 0;
 }
 
-void kvm_mmu_invalidate_end(struct kvm *kvm, unsigned long start,
-                           unsigned long end)
+void kvm_mmu_invalidate_end(struct kvm *kvm)
 {
        /*
         * This sequence increase will notify the kvm page fault that
--

2.25.1

author
#263 lore raw

Re: [PATCH v10 5/9] KVM: Use gfn instead of hva for mmu_notifier_retry

Quoted 102 lines · Fuad Tabba

Hi Chao,

Quoted 95 lines · Chao Peng

Currently in mmu_notifier invalidate path, hva range is recorded and then checked against by mmu_notifier_retry_hva() in the page fault handling path. However, for the to be introduced private memory, a page fault may not have a hva associated, checking gfn(gpa) makes more sense.

For existing hva based shared memory, gfn is expected to also work. The only downside is when aliasing multiple gfns to a single hva, the current algorithm of checking multiple ranges could result in a much larger range being rejected. Such aliasing should be uncommon, so the impact is expected small.

---
 arch/x86/kvm/mmu/mmu.c   |  8 +++++---
 include/linux/kvm_host.h | 33 +++++++++++++++++++++------------
 virt/kvm/kvm_main.c      | 32 +++++++++++++++++++++++---------
 3 files changed, 49 insertions(+), 24 deletions(-)
arch/x86/kvm/mmu/mmu.c +5 -3
index 4736d7849c60..e2c70b5afa3e 100644
@@ -4259,7 +4259,7 @@ static bool is_page_fault_stale(struct kvm_vcpu *vcpu,
                return true;
 
        return fault->slot &&
-              mmu_invalidate_retry_hva(vcpu->kvm, mmu_seq, fault->hva);
+              mmu_invalidate_retry_gfn(vcpu->kvm, mmu_seq, fault->gfn);
 }
 
 static int direct_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
@@ -6098,7 +6098,9 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end)
 
        write_lock(&kvm->mmu_lock);
 
-       kvm_mmu_invalidate_begin(kvm, gfn_start, gfn_end);
+       kvm_mmu_invalidate_begin(kvm);
+
+       kvm_mmu_invalidate_range_add(kvm, gfn_start, gfn_end);
 
        flush = kvm_rmap_zap_gfn_range(kvm, gfn_start, gfn_end);
 
@@ -6112,7 +6114,7 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end)
                kvm_flush_remote_tlbs_with_address(kvm, gfn_start,
                                                   gfn_end - gfn_start);
 
-       kvm_mmu_invalidate_end(kvm, gfn_start, gfn_end);
+       kvm_mmu_invalidate_end(kvm);
 
        write_unlock(&kvm->mmu_lock);
 }
include/linux/kvm_host.h +7 -8
index 02347e386ea2..3d69484d2704 100644
@@ -787,8 +787,8 @@ struct kvm {
        struct mmu_notifier mmu_notifier;
        unsigned long mmu_invalidate_seq;
        long mmu_invalidate_in_progress;
-       unsigned long mmu_invalidate_range_start;
-       unsigned long mmu_invalidate_range_end;
+       gfn_t mmu_invalidate_range_start;
+       gfn_t mmu_invalidate_range_end;
 #endif
        struct list_head devices;
        u64 manual_dirty_log_protect;
@@ -1389,10 +1389,9 @@ void kvm_mmu_free_memory_cache(struct kvm_mmu_memory_cache *mc);
 void *kvm_mmu_memory_cache_alloc(struct kvm_mmu_memory_cache *mc);
 #endif
 
-void kvm_mmu_invalidate_begin(struct kvm *kvm, unsigned long start,
-                             unsigned long end);
-void kvm_mmu_invalidate_end(struct kvm *kvm, unsigned long start,
-                           unsigned long end);
+void kvm_mmu_invalidate_begin(struct kvm *kvm);
+void kvm_mmu_invalidate_range_add(struct kvm *kvm, gfn_t start, gfn_t end);
+void kvm_mmu_invalidate_end(struct kvm *kvm);
 
 long kvm_arch_dev_ioctl(struct file *filp,
                        unsigned int ioctl, unsigned long arg);
@@ -1963,9 +1962,9 @@ static inline int mmu_invalidate_retry(struct kvm *kvm, unsigned long mmu_seq)
        return 0;
 }
 
-static inline int mmu_invalidate_retry_hva(struct kvm *kvm,
+static inline int mmu_invalidate_retry_gfn(struct kvm *kvm,
                                           unsigned long mmu_seq,
-                                          unsigned long hva)
+                                          gfn_t gfn)
 {
        lockdep_assert_held(&kvm->mmu_lock);
        /*

include/linux/kvm_host.h static inline int mmu_invalidate_retry_hva(struct kvm *kvm, · line 1973

Quoted 1 line
include/linux/kvm_host.h +0 -0
@@ -1974,10 +1973,20 @@ static inline int mmu_invalidate_retry_hva(struct kvm *kvm,
         * that might be being invalidated. Note that it may include some false

nit: "might be" (or) "is being"

Quoted 3 lines

include/linux/kvm_host.h static inline int mmu_invalidate_retry_hva(struct kvm *kvm, · line 1974

Quoted 1 line
include/linux/kvm_host.h +0 -0
         * positives, due to shortcuts when handing concurrent invalidations.

nit: handling

Both are existing code, but I can fix it either.

Quoted 14 lines

include/linux/kvm_host.h static inline int mmu_invalidate_retry_hva(struct kvm *kvm, · line 1982

Quoted 13 lines
include/linux/kvm_host.h +7 -4
         */
-       if (unlikely(kvm->mmu_invalidate_in_progress) &&
-           hva >= kvm->mmu_invalidate_range_start &&
-           hva < kvm->mmu_invalidate_range_end)
-               return 1;
+       if (unlikely(kvm->mmu_invalidate_in_progress)) {
+               /*
+                * Dropping mmu_lock after bumping mmu_invalidate_in_progress
+                * but before updating the range is a KVM bug.
+                */
+               if (WARN_ON_ONCE(kvm->mmu_invalidate_range_start == INVALID_GPA ||
+                                kvm->mmu_invalidate_range_end == INVALID_GPA))
Quoted 4 lines

INVALID_GPA is an x86-specific define in arch/x86/include/asm/kvm_host.h, so this doesn't build on other architectures. The obvious fix is to move it to include/linux/kvm_host.h.

Hmm, INVALID_GPA is defined as ZERO for x86, not 100% confident this is correct choice for other architectures, but after search it has not been used for other architectures, so should be safe to make it common.

Thanks,
Chao

Quoted 102 lines

Cheers,
/fuad

Quoted 98 lines
include/linux/kvm_host.h +7 -0
+                       return 1;
+
+               if (gfn >= kvm->mmu_invalidate_range_start &&
+                   gfn < kvm->mmu_invalidate_range_end)
+                       return 1;
+       }
+
        if (kvm->mmu_invalidate_seq != mmu_seq)
                return 1;
        return 0;
virt/kvm/kvm_main.c +23 -10
index b882eb2c76a2..ad55dfbc75d7 100644
@@ -540,9 +540,7 @@ static void kvm_mmu_notifier_invalidate_range(struct mmu_notifier *mn,
 
 typedef bool (*hva_handler_t)(struct kvm *kvm, struct kvm_gfn_range *range);
 
-typedef void (*on_lock_fn_t)(struct kvm *kvm, unsigned long start,
-                            unsigned long end);
-
+typedef void (*on_lock_fn_t)(struct kvm *kvm);
 typedef void (*on_unlock_fn_t)(struct kvm *kvm);
 
 struct kvm_hva_range {
@@ -628,7 +626,8 @@ static __always_inline int __kvm_handle_hva_range(struct kvm *kvm,
                                locked = true;
                                KVM_MMU_LOCK(kvm);
                                if (!IS_KVM_NULL_FN(range->on_lock))
-                                       range->on_lock(kvm, range->start, range->end);
+                                       range->on_lock(kvm);
+
                                if (IS_KVM_NULL_FN(range->handler))
                                        break;
                        }
@@ -715,8 +714,7 @@ static void kvm_mmu_notifier_change_pte(struct mmu_notifier *mn,
        kvm_handle_hva_range(mn, address, address + 1, pte, kvm_set_spte_gfn);
 }
 
-void kvm_mmu_invalidate_begin(struct kvm *kvm, unsigned long start,
-                             unsigned long end)
+void kvm_mmu_invalidate_begin(struct kvm *kvm)
 {
        /*
         * The count increase must become visible at unlock time as no
@@ -724,6 +722,17 @@ void kvm_mmu_invalidate_begin(struct kvm *kvm, unsigned long start,
         * count is also read inside the mmu_lock critical section.
         */
        kvm->mmu_invalidate_in_progress++;
+
+       if (likely(kvm->mmu_invalidate_in_progress == 1)) {
+               kvm->mmu_invalidate_range_start = INVALID_GPA;
+               kvm->mmu_invalidate_range_end = INVALID_GPA;
+       }
+}
+
+void kvm_mmu_invalidate_range_add(struct kvm *kvm, gfn_t start, gfn_t end)
+{
+       WARN_ON_ONCE(!kvm->mmu_invalidate_in_progress);
+
        if (likely(kvm->mmu_invalidate_in_progress == 1)) {
                kvm->mmu_invalidate_range_start = start;
                kvm->mmu_invalidate_range_end = end;
@@ -744,6 +753,12 @@ void kvm_mmu_invalidate_begin(struct kvm *kvm, unsigned long start,
        }
 }
 
+static bool kvm_mmu_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
+{
+       kvm_mmu_invalidate_range_add(kvm, range->start, range->end);
+       return kvm_unmap_gfn_range(kvm, range);
+}
+
 static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,
                                        const struct mmu_notifier_range *range)
 {
@@ -752,7 +767,7 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,
                .start          = range->start,
                .end            = range->end,
                .pte            = __pte(0),
-               .handler        = kvm_unmap_gfn_range,
+               .handler        = kvm_mmu_unmap_gfn_range,
                .on_lock        = kvm_mmu_invalidate_begin,
                .on_unlock      = kvm_arch_guest_memory_reclaimed,
                .flush_on_ret   = true,
@@ -791,8 +806,7 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,
        return 0;
 }
 
-void kvm_mmu_invalidate_end(struct kvm *kvm, unsigned long start,
-                           unsigned long end)
+void kvm_mmu_invalidate_end(struct kvm *kvm)
 {
        /*
         * This sequence increase will notify the kvm page fault that
--

2.25.1

Reviewed-by Tested-by
#264 lore raw

Re: [PATCH v10 5/9] KVM: Use gfn instead of hva for mmu_notifier_retry

Hi,

Quoted 104 lines · Chao Peng
Quoted 102 lines · Fuad Tabba

Hi Chao,

Quoted 95 lines · Chao Peng

Currently in mmu_notifier invalidate path, hva range is recorded and then checked against by mmu_notifier_retry_hva() in the page fault handling path. However, for the to be introduced private memory, a page fault may not have a hva associated, checking gfn(gpa) makes more sense.

For existing hva based shared memory, gfn is expected to also work. The only downside is when aliasing multiple gfns to a single hva, the current algorithm of checking multiple ranges could result in a much larger range being rejected. Such aliasing should be uncommon, so the impact is expected small.

---
 arch/x86/kvm/mmu/mmu.c   |  8 +++++---
 include/linux/kvm_host.h | 33 +++++++++++++++++++++------------
 virt/kvm/kvm_main.c      | 32 +++++++++++++++++++++++---------
 3 files changed, 49 insertions(+), 24 deletions(-)
arch/x86/kvm/mmu/mmu.c +5 -3
index 4736d7849c60..e2c70b5afa3e 100644
@@ -4259,7 +4259,7 @@ static bool is_page_fault_stale(struct kvm_vcpu *vcpu,
                return true;
 
        return fault->slot &&
-              mmu_invalidate_retry_hva(vcpu->kvm, mmu_seq, fault->hva);
+              mmu_invalidate_retry_gfn(vcpu->kvm, mmu_seq, fault->gfn);
 }
 
 static int direct_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
@@ -6098,7 +6098,9 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end)
 
        write_lock(&kvm->mmu_lock);
 
-       kvm_mmu_invalidate_begin(kvm, gfn_start, gfn_end);
+       kvm_mmu_invalidate_begin(kvm);
+
+       kvm_mmu_invalidate_range_add(kvm, gfn_start, gfn_end);
 
        flush = kvm_rmap_zap_gfn_range(kvm, gfn_start, gfn_end);
 
@@ -6112,7 +6114,7 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end)
                kvm_flush_remote_tlbs_with_address(kvm, gfn_start,
                                                   gfn_end - gfn_start);
 
-       kvm_mmu_invalidate_end(kvm, gfn_start, gfn_end);
+       kvm_mmu_invalidate_end(kvm);
 
        write_unlock(&kvm->mmu_lock);
 }
include/linux/kvm_host.h +7 -8
index 02347e386ea2..3d69484d2704 100644
@@ -787,8 +787,8 @@ struct kvm {
        struct mmu_notifier mmu_notifier;
        unsigned long mmu_invalidate_seq;
        long mmu_invalidate_in_progress;
-       unsigned long mmu_invalidate_range_start;
-       unsigned long mmu_invalidate_range_end;
+       gfn_t mmu_invalidate_range_start;
+       gfn_t mmu_invalidate_range_end;
 #endif
        struct list_head devices;
        u64 manual_dirty_log_protect;
@@ -1389,10 +1389,9 @@ void kvm_mmu_free_memory_cache(struct kvm_mmu_memory_cache *mc);
 void *kvm_mmu_memory_cache_alloc(struct kvm_mmu_memory_cache *mc);
 #endif
 
-void kvm_mmu_invalidate_begin(struct kvm *kvm, unsigned long start,
-                             unsigned long end);
-void kvm_mmu_invalidate_end(struct kvm *kvm, unsigned long start,
-                           unsigned long end);
+void kvm_mmu_invalidate_begin(struct kvm *kvm);
+void kvm_mmu_invalidate_range_add(struct kvm *kvm, gfn_t start, gfn_t end);
+void kvm_mmu_invalidate_end(struct kvm *kvm);
 
 long kvm_arch_dev_ioctl(struct file *filp,
                        unsigned int ioctl, unsigned long arg);
@@ -1963,9 +1962,9 @@ static inline int mmu_invalidate_retry(struct kvm *kvm, unsigned long mmu_seq)
        return 0;
 }
 
-static inline int mmu_invalidate_retry_hva(struct kvm *kvm,
+static inline int mmu_invalidate_retry_gfn(struct kvm *kvm,
                                           unsigned long mmu_seq,
-                                          unsigned long hva)
+                                          gfn_t gfn)
 {
        lockdep_assert_held(&kvm->mmu_lock);
        /*

include/linux/kvm_host.h static inline int mmu_invalidate_retry_hva(struct kvm *kvm, · line 1973

Quoted 1 line
include/linux/kvm_host.h +0 -0
@@ -1974,10 +1973,20 @@ static inline int mmu_invalidate_retry_hva(struct kvm *kvm,
         * that might be being invalidated. Note that it may include some false

nit: "might be" (or) "is being"

Quoted 5 lines
Quoted 3 lines

include/linux/kvm_host.h static inline int mmu_invalidate_retry_hva(struct kvm *kvm, · line 1974

Quoted 1 line
include/linux/kvm_host.h +0 -0
         * positives, due to shortcuts when handing concurrent invalidations.

nit: handling

Both are existing code, but I can fix it either.

That was just a nit, please feel free to ignore it, especially if it might cause headaches in the future with merges.

Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
include/linux/kvm_host.h +7 -4
         */
-       if (unlikely(kvm->mmu_invalidate_in_progress) &&
-           hva >= kvm->mmu_invalidate_range_start &&
-           hva < kvm->mmu_invalidate_range_end)
-               return 1;
+       if (unlikely(kvm->mmu_invalidate_in_progress)) {
+               /*
+                * Dropping mmu_lock after bumping mmu_invalidate_in_progress
+                * but before updating the range is a KVM bug.
+                */
+               if (WARN_ON_ONCE(kvm->mmu_invalidate_range_start == INVALID_GPA ||
+                                kvm->mmu_invalidate_range_end == INVALID_GPA))
Quoted 8 lines
Quoted 4 lines

INVALID_GPA is an x86-specific define in arch/x86/include/asm/kvm_host.h, so this doesn't build on other architectures. The obvious fix is to move it to include/linux/kvm_host.h.

Hmm, INVALID_GPA is defined as ZERO for x86, not 100% confident this is correct choice for other architectures, but after search it has not been used for other architectures, so should be safe to make it common.

With this fixed,

And the necessary work to port to arm64 (on qemu/arm64):

Cheers,
/fuad

Quoted 105 lines

Thanks,
Chao

Quoted 102 lines

Cheers,
/fuad

Quoted 98 lines
include/linux/kvm_host.h +7 -0
+                       return 1;
+
+               if (gfn >= kvm->mmu_invalidate_range_start &&
+                   gfn < kvm->mmu_invalidate_range_end)
+                       return 1;
+       }
+
        if (kvm->mmu_invalidate_seq != mmu_seq)
                return 1;
        return 0;
virt/kvm/kvm_main.c +23 -10
index b882eb2c76a2..ad55dfbc75d7 100644
@@ -540,9 +540,7 @@ static void kvm_mmu_notifier_invalidate_range(struct mmu_notifier *mn,
 
 typedef bool (*hva_handler_t)(struct kvm *kvm, struct kvm_gfn_range *range);
 
-typedef void (*on_lock_fn_t)(struct kvm *kvm, unsigned long start,
-                            unsigned long end);
-
+typedef void (*on_lock_fn_t)(struct kvm *kvm);
 typedef void (*on_unlock_fn_t)(struct kvm *kvm);
 
 struct kvm_hva_range {
@@ -628,7 +626,8 @@ static __always_inline int __kvm_handle_hva_range(struct kvm *kvm,
                                locked = true;
                                KVM_MMU_LOCK(kvm);
                                if (!IS_KVM_NULL_FN(range->on_lock))
-                                       range->on_lock(kvm, range->start, range->end);
+                                       range->on_lock(kvm);
+
                                if (IS_KVM_NULL_FN(range->handler))
                                        break;
                        }
@@ -715,8 +714,7 @@ static void kvm_mmu_notifier_change_pte(struct mmu_notifier *mn,
        kvm_handle_hva_range(mn, address, address + 1, pte, kvm_set_spte_gfn);
 }
 
-void kvm_mmu_invalidate_begin(struct kvm *kvm, unsigned long start,
-                             unsigned long end)
+void kvm_mmu_invalidate_begin(struct kvm *kvm)
 {
        /*
         * The count increase must become visible at unlock time as no
@@ -724,6 +722,17 @@ void kvm_mmu_invalidate_begin(struct kvm *kvm, unsigned long start,
         * count is also read inside the mmu_lock critical section.
         */
        kvm->mmu_invalidate_in_progress++;
+
+       if (likely(kvm->mmu_invalidate_in_progress == 1)) {
+               kvm->mmu_invalidate_range_start = INVALID_GPA;
+               kvm->mmu_invalidate_range_end = INVALID_GPA;
+       }
+}
+
+void kvm_mmu_invalidate_range_add(struct kvm *kvm, gfn_t start, gfn_t end)
+{
+       WARN_ON_ONCE(!kvm->mmu_invalidate_in_progress);
+
        if (likely(kvm->mmu_invalidate_in_progress == 1)) {
                kvm->mmu_invalidate_range_start = start;
                kvm->mmu_invalidate_range_end = end;
@@ -744,6 +753,12 @@ void kvm_mmu_invalidate_begin(struct kvm *kvm, unsigned long start,
        }
 }
 
+static bool kvm_mmu_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
+{
+       kvm_mmu_invalidate_range_add(kvm, range->start, range->end);
+       return kvm_unmap_gfn_range(kvm, range);
+}
+
 static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,
                                        const struct mmu_notifier_range *range)
 {
@@ -752,7 +767,7 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,
                .start          = range->start,
                .end            = range->end,
                .pte            = __pte(0),
-               .handler        = kvm_unmap_gfn_range,
+               .handler        = kvm_mmu_unmap_gfn_range,
                .on_lock        = kvm_mmu_invalidate_begin,
                .on_unlock      = kvm_arch_guest_memory_reclaimed,
                .flush_on_ret   = true,
@@ -791,8 +806,7 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,
        return 0;
 }
 
-void kvm_mmu_invalidate_end(struct kvm *kvm, unsigned long start,
-                           unsigned long end)
+void kvm_mmu_invalidate_end(struct kvm *kvm)
 {
        /*
         * This sequence increase will notify the kvm page fault that
--

2.25.1

author
#265 lore raw

Re: [PATCH v10 5/9] KVM: Use gfn instead of hva for mmu_notifier_retry

On Tue, Dec 06, 2022 at 03:48:50PM +0000, Fuad Tabba wrote:
...

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
         */
include/linux/kvm_host.h +7 -4
-       if (unlikely(kvm->mmu_invalidate_in_progress) &&
-           hva >= kvm->mmu_invalidate_range_start &&
-           hva < kvm->mmu_invalidate_range_end)
-               return 1;
+       if (unlikely(kvm->mmu_invalidate_in_progress)) {
+               /*
+                * Dropping mmu_lock after bumping mmu_invalidate_in_progress
+                * but before updating the range is a KVM bug.
+                */
+               if (WARN_ON_ONCE(kvm->mmu_invalidate_range_start == INVALID_GPA ||
+                                kvm->mmu_invalidate_range_end == INVALID_GPA))
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines

INVALID_GPA is an x86-specific define in arch/x86/include/asm/kvm_host.h, so this doesn't build on other architectures. The obvious fix is to move it to include/linux/kvm_host.h.

Hmm, INVALID_GPA is defined as ZERO for x86, not 100% confident this is correct choice for other architectures, but after search it has not been used for other architectures, so should be safe to make it common.

As Yu posted a patch:
https://lore.kernel.org/all/[email protected]/

There is a GPA_INVALID in include/linux/kvm_types.h and I see ARM has already been using it so sounds that is exactly what I need.

Chao

Quoted 9 lines

With this fixed,

And the necessary work to port to arm64 (on qemu/arm64):

Cheers,
/fuad

#266 lore raw

Re: [PATCH v10 5/9] KVM: Use gfn instead of hva for mmu_notifier_retry

In reply to Chao Peng
Quoted 12 lines · Chao Peng
Quoted 12 lines
Quoted 11 lines
include/linux/kvm_host.h +7 -4
-       if (unlikely(kvm->mmu_invalidate_in_progress) &&
-           hva >= kvm->mmu_invalidate_range_start &&
-           hva < kvm->mmu_invalidate_range_end)
-               return 1;
+       if (unlikely(kvm->mmu_invalidate_in_progress)) {
+               /*
+                * Dropping mmu_lock after bumping mmu_invalidate_in_progress
+                * but before updating the range is a KVM bug.
+                */
+               if (WARN_ON_ONCE(kvm->mmu_invalidate_range_start == INVALID_GPA ||
+                                kvm->mmu_invalidate_range_end == INVALID_GPA))
Quoted 8 lines
Quoted 4 lines

INVALID_GPA is an x86-specific define in arch/x86/include/asm/kvm_host.h, so this doesn't build on other architectures. The obvious fix is to move it to include/linux/kvm_host.h.

Hmm, INVALID_GPA is defined as ZERO for x86, not 100% confident this is correct choice for other architectures, but after search it has not been used for other architectures, so should be safe to make it common.

INVALID_GPA is defined as all bit 1. Please notice "~" (tilde).

#define INVALID_GPA (~(gpa_t)0)

Signature · 1 line
Isaku Yamahata <[email protected]>
author
#267 lore raw

Re: [PATCH v10 5/9] KVM: Use gfn instead of hva for mmu_notifier_retry

Quoted 20 lines · Isaku Yamahata
Quoted 17 lines · Chao Peng
Quoted 16 lines

include/linux/kvm_host.h static inline int mmu_invalidate_retry_hva(struct kvm *kvm, · line 1982

Quoted 11 lines
include/linux/kvm_host.h +7 -4
-       if (unlikely(kvm->mmu_invalidate_in_progress) &&
-           hva >= kvm->mmu_invalidate_range_start &&
-           hva < kvm->mmu_invalidate_range_end)
-               return 1;
+       if (unlikely(kvm->mmu_invalidate_in_progress)) {
+               /*
+                * Dropping mmu_lock after bumping mmu_invalidate_in_progress
+                * but before updating the range is a KVM bug.
+                */
+               if (WARN_ON_ONCE(kvm->mmu_invalidate_range_start == INVALID_GPA ||
+                                kvm->mmu_invalidate_range_end == INVALID_GPA))

INVALID_GPA is an x86-specific define in arch/x86/include/asm/kvm_host.h, so this doesn't build on other architectures. The obvious fix is to move it to include/linux/kvm_host.h.

Quoted 7 lines
Quoted 3 lines

Hmm, INVALID_GPA is defined as ZERO for x86, not 100% confident this is correct choice for other architectures, but after search it has not been used for other architectures, so should be safe to make it common.

INVALID_GPA is defined as all bit 1. Please notice "~" (tilde).

#define INVALID_GPA (~(gpa_t)0)

Thanks for mention. Still looks right moving it to include/linux/kvm_host.h. Chao

Quoted 2 lines
Signature · 1 line
Isaku Yamahata <[email protected]>
author
#268 lore raw

PATCH v10 6/9 KVM: Unmap existing mappings when change the memory attributes

In reply to Chao Peng

Unmap the existing guest mappings when memory attribute is changed between shared and private. This is needed because shared pages and private pages are from different backends, unmapping existing ones gives a chance for page fault handler to re-populate the mappings according to the new attribute.

Only architecture has private memory support needs this and the supported architecture is expected to rewrite the weak kvm_arch_has_private_mem().

Also, during memory attribute changing and the unmapping time frame, page fault handler may happen in the same memory range and can cause incorrect page state, invoke kvm_mmu_invalidate_* helpers to let the page fault handler retry during this time frame.

---
 include/linux/kvm_host.h |   7 +-
 virt/kvm/kvm_main.c      | 168 ++++++++++++++++++++++++++-------------
 2 files changed, 116 insertions(+), 59 deletions(-)
include/linux/kvm_host.h +5 -2
index 3d69484d2704..3331c0c92838 100644
@@ -255,7 +255,6 @@ bool kvm_setup_async_pf(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 int kvm_async_pf_wakeup_all(struct kvm_vcpu *vcpu);
 #endif
 
-#ifdef KVM_ARCH_WANT_MMU_NOTIFIER
 struct kvm_gfn_range {
 	struct kvm_memory_slot *slot;
 	gfn_t start;
@@ -264,6 +263,8 @@ struct kvm_gfn_range {
 	bool may_block;
 };
 bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range);
+
+#ifdef KVM_ARCH_WANT_MMU_NOTIFIER
 bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range);
 bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range);
 bool kvm_set_spte_gfn(struct kvm *kvm, struct kvm_gfn_range *range);
@@ -785,11 +786,12 @@ struct kvm {
 
 #if defined(CONFIG_MMU_NOTIFIER) && defined(KVM_ARCH_WANT_MMU_NOTIFIER)
 	struct mmu_notifier mmu_notifier;
+#endif
 	unsigned long mmu_invalidate_seq;
 	long mmu_invalidate_in_progress;
 	gfn_t mmu_invalidate_range_start;
 	gfn_t mmu_invalidate_range_end;
-#endif
+
 	struct list_head devices;
 	u64 manual_dirty_log_protect;
 	struct dentry *debugfs_dentry;
@@ -1480,6 +1482,7 @@ bool kvm_arch_dy_has_pending_interrupt(struct kvm_vcpu *vcpu);
 int kvm_arch_post_init_vm(struct kvm *kvm);
 void kvm_arch_pre_destroy_vm(struct kvm *kvm);
 int kvm_arch_create_vm_debugfs(struct kvm *kvm);
+bool kvm_arch_has_private_mem(struct kvm *kvm);
 
 #ifndef __KVM_HAVE_ARCH_VM_ALLOC
 /*
virt/kvm/kvm_main.c +111 -57

218 lines not on this page · raw message

Signature · 1 line
2.25.1
#269 lore raw

Re: [PATCH v10 6/9] KVM: Unmap existing mappings when change the memory attributes

Quoted 263 lines · Chao Peng

Unmap the existing guest mappings when memory attribute is changed between shared and private. This is needed because shared pages and private pages are from different backends, unmapping existing ones gives a chance for page fault handler to re-populate the mappings according to the new attribute.

Only architecture has private memory support needs this and the supported architecture is expected to rewrite the weak kvm_arch_has_private_mem().

Also, during memory attribute changing and the unmapping time frame, page fault handler may happen in the same memory range and can cause incorrect page state, invoke kvm_mmu_invalidate_* helpers to let the page fault handler retry during this time frame.

---
 include/linux/kvm_host.h |   7 +-
 virt/kvm/kvm_main.c      | 168 ++++++++++++++++++++++++++-------------
 2 files changed, 116 insertions(+), 59 deletions(-)
include/linux/kvm_host.h +5 -2
index 3d69484d2704..3331c0c92838 100644
@@ -255,7 +255,6 @@ bool kvm_setup_async_pf(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 int kvm_async_pf_wakeup_all(struct kvm_vcpu *vcpu);
 #endif
 
-#ifdef KVM_ARCH_WANT_MMU_NOTIFIER
 struct kvm_gfn_range {
 	struct kvm_memory_slot *slot;
 	gfn_t start;
@@ -264,6 +263,8 @@ struct kvm_gfn_range {
 	bool may_block;
 };
 bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range);
+
+#ifdef KVM_ARCH_WANT_MMU_NOTIFIER
 bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range);
 bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range);
 bool kvm_set_spte_gfn(struct kvm *kvm, struct kvm_gfn_range *range);
@@ -785,11 +786,12 @@ struct kvm {
 
 #if defined(CONFIG_MMU_NOTIFIER) && defined(KVM_ARCH_WANT_MMU_NOTIFIER)
 	struct mmu_notifier mmu_notifier;
+#endif
 	unsigned long mmu_invalidate_seq;
 	long mmu_invalidate_in_progress;
 	gfn_t mmu_invalidate_range_start;
 	gfn_t mmu_invalidate_range_end;
-#endif
+
 	struct list_head devices;
 	u64 manual_dirty_log_protect;
 	struct dentry *debugfs_dentry;
@@ -1480,6 +1482,7 @@ bool kvm_arch_dy_has_pending_interrupt(struct kvm_vcpu *vcpu);
 int kvm_arch_post_init_vm(struct kvm *kvm);
 void kvm_arch_pre_destroy_vm(struct kvm *kvm);
 int kvm_arch_create_vm_debugfs(struct kvm *kvm);
+bool kvm_arch_has_private_mem(struct kvm *kvm);
 
 #ifndef __KVM_HAVE_ARCH_VM_ALLOC
 /*
virt/kvm/kvm_main.c +94 -57
index ad55dfbc75d7..4e1e1e113bf0 100644
@@ -520,6 +520,62 @@ void kvm_destroy_vcpus(struct kvm *kvm)
 }
 EXPORT_SYMBOL_GPL(kvm_destroy_vcpus);
 
+void kvm_mmu_invalidate_begin(struct kvm *kvm)
+{
+	/*
+	 * The count increase must become visible at unlock time as no
+	 * spte can be established without taking the mmu_lock and
+	 * count is also read inside the mmu_lock critical section.
+	 */
+	kvm->mmu_invalidate_in_progress++;
+
+	if (likely(kvm->mmu_invalidate_in_progress == 1)) {
+		kvm->mmu_invalidate_range_start = INVALID_GPA;
+		kvm->mmu_invalidate_range_end = INVALID_GPA;
+	}
+}
+
+void kvm_mmu_invalidate_range_add(struct kvm *kvm, gfn_t start, gfn_t end)
+{
+	WARN_ON_ONCE(!kvm->mmu_invalidate_in_progress);
+
+	if (likely(kvm->mmu_invalidate_in_progress == 1)) {
+		kvm->mmu_invalidate_range_start = start;
+		kvm->mmu_invalidate_range_end = end;
+	} else {
+		/*
+		 * Fully tracking multiple concurrent ranges has diminishing
+		 * returns. Keep things simple and just find the minimal range
+		 * which includes the current and new ranges. As there won't be
+		 * enough information to subtract a range after its invalidate
+		 * completes, any ranges invalidated concurrently will
+		 * accumulate and persist until all outstanding invalidates
+		 * complete.
+		 */
+		kvm->mmu_invalidate_range_start =
+			min(kvm->mmu_invalidate_range_start, start);
+		kvm->mmu_invalidate_range_end =
+			max(kvm->mmu_invalidate_range_end, end);
+	}
+}
+
+void kvm_mmu_invalidate_end(struct kvm *kvm)
+{
+	/*
+	 * This sequence increase will notify the kvm page fault that
+	 * the page that is going to be mapped in the spte could have
+	 * been freed.
+	 */
+	kvm->mmu_invalidate_seq++;
+	smp_wmb();
+	/*
+	 * The above sequence increase must be visible before the
+	 * below count decrease, which is ensured by the smp_wmb above
+	 * in conjunction with the smp_rmb in mmu_invalidate_retry().
+	 */
+	kvm->mmu_invalidate_in_progress--;
+}
+
 #if defined(CONFIG_MMU_NOTIFIER) && defined(KVM_ARCH_WANT_MMU_NOTIFIER)
 static inline struct kvm *mmu_notifier_to_kvm(struct mmu_notifier *mn)
 {
@@ -714,45 +770,6 @@ static void kvm_mmu_notifier_change_pte(struct mmu_notifier *mn,
 	kvm_handle_hva_range(mn, address, address + 1, pte, kvm_set_spte_gfn);
 }
 
-void kvm_mmu_invalidate_begin(struct kvm *kvm)
-{
-	/*
-	 * The count increase must become visible at unlock time as no
-	 * spte can be established without taking the mmu_lock and
-	 * count is also read inside the mmu_lock critical section.
-	 */
-	kvm->mmu_invalidate_in_progress++;
-
-	if (likely(kvm->mmu_invalidate_in_progress == 1)) {
-		kvm->mmu_invalidate_range_start = INVALID_GPA;
-		kvm->mmu_invalidate_range_end = INVALID_GPA;
-	}
-}
-
-void kvm_mmu_invalidate_range_add(struct kvm *kvm, gfn_t start, gfn_t end)
-{
-	WARN_ON_ONCE(!kvm->mmu_invalidate_in_progress);
-
-	if (likely(kvm->mmu_invalidate_in_progress == 1)) {
-		kvm->mmu_invalidate_range_start = start;
-		kvm->mmu_invalidate_range_end = end;
-	} else {
-		/*
-		 * Fully tracking multiple concurrent ranges has diminishing
-		 * returns. Keep things simple and just find the minimal range
-		 * which includes the current and new ranges. As there won't be
-		 * enough information to subtract a range after its invalidate
-		 * completes, any ranges invalidated concurrently will
-		 * accumulate and persist until all outstanding invalidates
-		 * complete.
-		 */
-		kvm->mmu_invalidate_range_start =
-			min(kvm->mmu_invalidate_range_start, start);
-		kvm->mmu_invalidate_range_end =
-			max(kvm->mmu_invalidate_range_end, end);
-	}
-}
-
 static bool kvm_mmu_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
 {
 	kvm_mmu_invalidate_range_add(kvm, range->start, range->end);
@@ -806,23 +823,6 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,
 	return 0;
 }
 
-void kvm_mmu_invalidate_end(struct kvm *kvm)
-{
-	/*
-	 * This sequence increase will notify the kvm page fault that
-	 * the page that is going to be mapped in the spte could have
-	 * been freed.
-	 */
-	kvm->mmu_invalidate_seq++;
-	smp_wmb();
-	/*
-	 * The above sequence increase must be visible before the
-	 * below count decrease, which is ensured by the smp_wmb above
-	 * in conjunction with the smp_rmb in mmu_invalidate_retry().
-	 */
-	kvm->mmu_invalidate_in_progress--;
-}
-
 static void kvm_mmu_notifier_invalidate_range_end(struct mmu_notifier *mn,
 					const struct mmu_notifier_range *range)
 {
@@ -1140,6 +1140,11 @@ int __weak kvm_arch_create_vm_debugfs(struct kvm *kvm)
 	return 0;
 }
 
+bool __weak kvm_arch_has_private_mem(struct kvm *kvm)
+{
+	return false;
+}
+
 static struct kvm *kvm_create_vm(unsigned long type, const char *fdname)
 {
 	struct kvm *kvm = kvm_arch_alloc_vm();
@@ -2349,15 +2354,47 @@ static u64 kvm_supported_mem_attributes(struct kvm *kvm)
 	return 0;
 }
 
+static void kvm_unmap_mem_range(struct kvm *kvm, gfn_t start, gfn_t end)
+{
+	struct kvm_gfn_range gfn_range;
+	struct kvm_memory_slot *slot;
+	struct kvm_memslots *slots;
+	struct kvm_memslot_iter iter;
+	int i;
+	int r = 0;
+
+	gfn_range.pte = __pte(0);
+	gfn_range.may_block = true;
+
+	for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++) {
+		slots = __kvm_memslots(kvm, i);
+
+		kvm_for_each_memslot_in_gfn_range(&iter, slots, start, end) {
+			slot = iter.slot;
+			gfn_range.start = max(start, slot->base_gfn);
+			gfn_range.end = min(end, slot->base_gfn + slot->npages);
+			if (gfn_range.start >= gfn_range.end)
+				continue;
+			gfn_range.slot = slot;
+
+			r |= kvm_unmap_gfn_range(kvm, &gfn_range);
+		}
+	}
+
+	if (r)
+		kvm_flush_remote_tlbs(kvm);
+}
+
 static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
 					   struct kvm_memory_attributes *attrs)
 {
 	gfn_t start, end;
 	unsigned long i;
 	void *entry;
+	int idx;
 	u64 supported_attrs = kvm_supported_mem_attributes(kvm);
 
-	/* flags is currently not used. */
+	/* 'flags' is currently not used. */
 	if (attrs->flags)
 		return -EINVAL;
 	if (attrs->attributes & ~supported_attrs)

virt/kvm/kvm_main.c static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, · line 2415

Quoted 7 lines
virt/kvm/kvm_main.c +4 -0
@@ -2372,6 +2409,13 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
 
 	entry = attrs->attributes ? xa_mk_value(attrs->attributes) : NULL;
 
+	if (kvm_arch_has_private_mem(kvm)) {
+		KVM_MMU_LOCK(kvm);
+		kvm_mmu_invalidate_begin(kvm);
+		kvm_mmu_invalidate_range_add(kvm, start, end);

Nit: this works for KVM_MEMORY_ATTRIBUTE_PRIVATE, but the invalidation should be necessary yet for attribute change of:

KVM_MEMORY_ATTRIBUTE_READ
KVM_MEMORY_ATTRIBUTE_WRITE
KVM_MEMORY_ATTRIBUTE_EXECUTE

Quoted 7 lines
virt/kvm/kvm_main.c +3 -0
+		KVM_MMU_UNLOCK(kvm);
+	}
+
 	mutex_lock(&kvm->lock);
 	for (i = start; i < end; i++)
 		if (xa_err(xa_store(&kvm->mem_attr_array, i, entry,

virt/kvm/kvm_main.c static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, · line 2431

Quoted 9 lines
virt/kvm/kvm_main.c +6 -0
@@ -2379,6 +2423,16 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
 			break;
 	mutex_unlock(&kvm->lock);
 
+	if (kvm_arch_has_private_mem(kvm)) {
+		idx = srcu_read_lock(&kvm->srcu);
+		KVM_MMU_LOCK(kvm);
+		if (i > start)
+			kvm_unmap_mem_range(kvm, start, i);
+		kvm_mmu_invalidate_end(kvm);

Ditto.

Quoted 11 lines
virt/kvm/kvm_main.c +4 -1
+		KVM_MMU_UNLOCK(kvm);
+		srcu_read_unlock(&kvm->srcu, idx);
+	}
+
 	attrs->address = i << PAGE_SHIFT;
 	attrs->size = (end - i) << PAGE_SHIFT;
 
--

2.25.1

author
#270 lore raw

Re: [PATCH v10 6/9] KVM: Unmap existing mappings when change the memory attributes

Quoted 275 lines · Yuan Yao
Quoted 263 lines · Chao Peng

Unmap the existing guest mappings when memory attribute is changed between shared and private. This is needed because shared pages and private pages are from different backends, unmapping existing ones gives a chance for page fault handler to re-populate the mappings according to the new attribute.

Only architecture has private memory support needs this and the supported architecture is expected to rewrite the weak kvm_arch_has_private_mem().

Also, during memory attribute changing and the unmapping time frame, page fault handler may happen in the same memory range and can cause incorrect page state, invoke kvm_mmu_invalidate_* helpers to let the page fault handler retry during this time frame.

---
 include/linux/kvm_host.h |   7 +-
 virt/kvm/kvm_main.c      | 168 ++++++++++++++++++++++++++-------------
 2 files changed, 116 insertions(+), 59 deletions(-)
include/linux/kvm_host.h +5 -2
index 3d69484d2704..3331c0c92838 100644
@@ -255,7 +255,6 @@ bool kvm_setup_async_pf(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 int kvm_async_pf_wakeup_all(struct kvm_vcpu *vcpu);
 #endif
 
-#ifdef KVM_ARCH_WANT_MMU_NOTIFIER
 struct kvm_gfn_range {
 	struct kvm_memory_slot *slot;
 	gfn_t start;
@@ -264,6 +263,8 @@ struct kvm_gfn_range {
 	bool may_block;
 };
 bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range);
+
+#ifdef KVM_ARCH_WANT_MMU_NOTIFIER
 bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range);
 bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range);
 bool kvm_set_spte_gfn(struct kvm *kvm, struct kvm_gfn_range *range);
@@ -785,11 +786,12 @@ struct kvm {
 
 #if defined(CONFIG_MMU_NOTIFIER) && defined(KVM_ARCH_WANT_MMU_NOTIFIER)
 	struct mmu_notifier mmu_notifier;
+#endif
 	unsigned long mmu_invalidate_seq;
 	long mmu_invalidate_in_progress;
 	gfn_t mmu_invalidate_range_start;
 	gfn_t mmu_invalidate_range_end;
-#endif
+
 	struct list_head devices;
 	u64 manual_dirty_log_protect;
 	struct dentry *debugfs_dentry;
@@ -1480,6 +1482,7 @@ bool kvm_arch_dy_has_pending_interrupt(struct kvm_vcpu *vcpu);
 int kvm_arch_post_init_vm(struct kvm *kvm);
 void kvm_arch_pre_destroy_vm(struct kvm *kvm);
 int kvm_arch_create_vm_debugfs(struct kvm *kvm);
+bool kvm_arch_has_private_mem(struct kvm *kvm);
 
 #ifndef __KVM_HAVE_ARCH_VM_ALLOC
 /*
virt/kvm/kvm_main.c +94 -57
index ad55dfbc75d7..4e1e1e113bf0 100644
@@ -520,6 +520,62 @@ void kvm_destroy_vcpus(struct kvm *kvm)
 }
 EXPORT_SYMBOL_GPL(kvm_destroy_vcpus);
 
+void kvm_mmu_invalidate_begin(struct kvm *kvm)
+{
+	/*
+	 * The count increase must become visible at unlock time as no
+	 * spte can be established without taking the mmu_lock and
+	 * count is also read inside the mmu_lock critical section.
+	 */
+	kvm->mmu_invalidate_in_progress++;
+
+	if (likely(kvm->mmu_invalidate_in_progress == 1)) {
+		kvm->mmu_invalidate_range_start = INVALID_GPA;
+		kvm->mmu_invalidate_range_end = INVALID_GPA;
+	}
+}
+
+void kvm_mmu_invalidate_range_add(struct kvm *kvm, gfn_t start, gfn_t end)
+{
+	WARN_ON_ONCE(!kvm->mmu_invalidate_in_progress);
+
+	if (likely(kvm->mmu_invalidate_in_progress == 1)) {
+		kvm->mmu_invalidate_range_start = start;
+		kvm->mmu_invalidate_range_end = end;
+	} else {
+		/*
+		 * Fully tracking multiple concurrent ranges has diminishing
+		 * returns. Keep things simple and just find the minimal range
+		 * which includes the current and new ranges. As there won't be
+		 * enough information to subtract a range after its invalidate
+		 * completes, any ranges invalidated concurrently will
+		 * accumulate and persist until all outstanding invalidates
+		 * complete.
+		 */
+		kvm->mmu_invalidate_range_start =
+			min(kvm->mmu_invalidate_range_start, start);
+		kvm->mmu_invalidate_range_end =
+			max(kvm->mmu_invalidate_range_end, end);
+	}
+}
+
+void kvm_mmu_invalidate_end(struct kvm *kvm)
+{
+	/*
+	 * This sequence increase will notify the kvm page fault that
+	 * the page that is going to be mapped in the spte could have
+	 * been freed.
+	 */
+	kvm->mmu_invalidate_seq++;
+	smp_wmb();
+	/*
+	 * The above sequence increase must be visible before the
+	 * below count decrease, which is ensured by the smp_wmb above
+	 * in conjunction with the smp_rmb in mmu_invalidate_retry().
+	 */
+	kvm->mmu_invalidate_in_progress--;
+}
+
 #if defined(CONFIG_MMU_NOTIFIER) && defined(KVM_ARCH_WANT_MMU_NOTIFIER)
 static inline struct kvm *mmu_notifier_to_kvm(struct mmu_notifier *mn)
 {
@@ -714,45 +770,6 @@ static void kvm_mmu_notifier_change_pte(struct mmu_notifier *mn,
 	kvm_handle_hva_range(mn, address, address + 1, pte, kvm_set_spte_gfn);
 }
 
-void kvm_mmu_invalidate_begin(struct kvm *kvm)
-{
-	/*
-	 * The count increase must become visible at unlock time as no
-	 * spte can be established without taking the mmu_lock and
-	 * count is also read inside the mmu_lock critical section.
-	 */
-	kvm->mmu_invalidate_in_progress++;
-
-	if (likely(kvm->mmu_invalidate_in_progress == 1)) {
-		kvm->mmu_invalidate_range_start = INVALID_GPA;
-		kvm->mmu_invalidate_range_end = INVALID_GPA;
-	}
-}
-
-void kvm_mmu_invalidate_range_add(struct kvm *kvm, gfn_t start, gfn_t end)
-{
-	WARN_ON_ONCE(!kvm->mmu_invalidate_in_progress);
-
-	if (likely(kvm->mmu_invalidate_in_progress == 1)) {
-		kvm->mmu_invalidate_range_start = start;
-		kvm->mmu_invalidate_range_end = end;
-	} else {
-		/*
-		 * Fully tracking multiple concurrent ranges has diminishing
-		 * returns. Keep things simple and just find the minimal range
-		 * which includes the current and new ranges. As there won't be
-		 * enough information to subtract a range after its invalidate
-		 * completes, any ranges invalidated concurrently will
-		 * accumulate and persist until all outstanding invalidates
-		 * complete.
-		 */
-		kvm->mmu_invalidate_range_start =
-			min(kvm->mmu_invalidate_range_start, start);
-		kvm->mmu_invalidate_range_end =
-			max(kvm->mmu_invalidate_range_end, end);
-	}
-}
-
 static bool kvm_mmu_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
 {
 	kvm_mmu_invalidate_range_add(kvm, range->start, range->end);
@@ -806,23 +823,6 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,
 	return 0;
 }
 
-void kvm_mmu_invalidate_end(struct kvm *kvm)
-{
-	/*
-	 * This sequence increase will notify the kvm page fault that
-	 * the page that is going to be mapped in the spte could have
-	 * been freed.
-	 */
-	kvm->mmu_invalidate_seq++;
-	smp_wmb();
-	/*
-	 * The above sequence increase must be visible before the
-	 * below count decrease, which is ensured by the smp_wmb above
-	 * in conjunction with the smp_rmb in mmu_invalidate_retry().
-	 */
-	kvm->mmu_invalidate_in_progress--;
-}
-
 static void kvm_mmu_notifier_invalidate_range_end(struct mmu_notifier *mn,
 					const struct mmu_notifier_range *range)
 {
@@ -1140,6 +1140,11 @@ int __weak kvm_arch_create_vm_debugfs(struct kvm *kvm)
 	return 0;
 }
 
+bool __weak kvm_arch_has_private_mem(struct kvm *kvm)
+{
+	return false;
+}
+
 static struct kvm *kvm_create_vm(unsigned long type, const char *fdname)
 {
 	struct kvm *kvm = kvm_arch_alloc_vm();
@@ -2349,15 +2354,47 @@ static u64 kvm_supported_mem_attributes(struct kvm *kvm)
 	return 0;
 }
 
+static void kvm_unmap_mem_range(struct kvm *kvm, gfn_t start, gfn_t end)
+{
+	struct kvm_gfn_range gfn_range;
+	struct kvm_memory_slot *slot;
+	struct kvm_memslots *slots;
+	struct kvm_memslot_iter iter;
+	int i;
+	int r = 0;
+
+	gfn_range.pte = __pte(0);
+	gfn_range.may_block = true;
+
+	for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++) {
+		slots = __kvm_memslots(kvm, i);
+
+		kvm_for_each_memslot_in_gfn_range(&iter, slots, start, end) {
+			slot = iter.slot;
+			gfn_range.start = max(start, slot->base_gfn);
+			gfn_range.end = min(end, slot->base_gfn + slot->npages);
+			if (gfn_range.start >= gfn_range.end)
+				continue;
+			gfn_range.slot = slot;
+
+			r |= kvm_unmap_gfn_range(kvm, &gfn_range);
+		}
+	}
+
+	if (r)
+		kvm_flush_remote_tlbs(kvm);
+}
+
 static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
 					   struct kvm_memory_attributes *attrs)
 {
 	gfn_t start, end;
 	unsigned long i;
 	void *entry;
+	int idx;
 	u64 supported_attrs = kvm_supported_mem_attributes(kvm);
 
-	/* flags is currently not used. */
+	/* 'flags' is currently not used. */
 	if (attrs->flags)
 		return -EINVAL;
 	if (attrs->attributes & ~supported_attrs)

virt/kvm/kvm_main.c static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, · line 2415

Quoted 7 lines
virt/kvm/kvm_main.c +4 -0
@@ -2372,6 +2409,13 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
 
 	entry = attrs->attributes ? xa_mk_value(attrs->attributes) : NULL;
 
+	if (kvm_arch_has_private_mem(kvm)) {
+		KVM_MMU_LOCK(kvm);
+		kvm_mmu_invalidate_begin(kvm);
+		kvm_mmu_invalidate_range_add(kvm, start, end);

Nit: this works for KVM_MEMORY_ATTRIBUTE_PRIVATE, but the invalidation should be necessary yet for attribute change of:

Quoted 3 lines

KVM_MEMORY_ATTRIBUTE_READ
KVM_MEMORY_ATTRIBUTE_WRITE
KVM_MEMORY_ATTRIBUTE_EXECUTE

The unmapping is only needed for confidential usages which uses KVM_MEMORY_ATTRIBUTE_PRIVATE only and the other flags are defined here for other usages like pKVM. As Fuad commented in a different reply, pKVM supports in-place remapping and unmapping is unnecessary.

Thanks,
Chao

Quoted 31 lines
Quoted 8 lines
virt/kvm/kvm_main.c +3 -0
+		KVM_MMU_UNLOCK(kvm);
+	}
+
 	mutex_lock(&kvm->lock);
 	for (i = start; i < end; i++)
 		if (xa_err(xa_store(&kvm->mem_attr_array, i, entry,

virt/kvm/kvm_main.c static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, · line 2431

Quoted 9 lines
virt/kvm/kvm_main.c +6 -0
@@ -2379,6 +2423,16 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
 			break;
 	mutex_unlock(&kvm->lock);
 
+	if (kvm_arch_has_private_mem(kvm)) {
+		idx = srcu_read_lock(&kvm->srcu);
+		KVM_MMU_LOCK(kvm);
+		if (i > start)
+			kvm_unmap_mem_range(kvm, start, i);
+		kvm_mmu_invalidate_end(kvm);

Ditto.

Quoted 11 lines
virt/kvm/kvm_main.c +4 -1
+		KVM_MMU_UNLOCK(kvm);
+		srcu_read_unlock(&kvm->srcu, idx);
+	}
+
 	attrs->address = i << PAGE_SHIFT;
 	attrs->size = (end - i) << PAGE_SHIFT;
 
--

2.25.1

#271 lore raw

Re: [PATCH v10 6/9] KVM: Unmap existing mappings when change the memory attributes

Quoted 276 lines · Chao Peng
Quoted 275 lines · Yuan Yao
Quoted 263 lines · Chao Peng

Unmap the existing guest mappings when memory attribute is changed between shared and private. This is needed because shared pages and private pages are from different backends, unmapping existing ones gives a chance for page fault handler to re-populate the mappings according to the new attribute.

Only architecture has private memory support needs this and the supported architecture is expected to rewrite the weak kvm_arch_has_private_mem().

Also, during memory attribute changing and the unmapping time frame, page fault handler may happen in the same memory range and can cause incorrect page state, invoke kvm_mmu_invalidate_* helpers to let the page fault handler retry during this time frame.

---
 include/linux/kvm_host.h |   7 +-
 virt/kvm/kvm_main.c      | 168 ++++++++++++++++++++++++++-------------
 2 files changed, 116 insertions(+), 59 deletions(-)
include/linux/kvm_host.h +5 -2
index 3d69484d2704..3331c0c92838 100644
@@ -255,7 +255,6 @@ bool kvm_setup_async_pf(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 int kvm_async_pf_wakeup_all(struct kvm_vcpu *vcpu);
 #endif
 
-#ifdef KVM_ARCH_WANT_MMU_NOTIFIER
 struct kvm_gfn_range {
 	struct kvm_memory_slot *slot;
 	gfn_t start;
@@ -264,6 +263,8 @@ struct kvm_gfn_range {
 	bool may_block;
 };
 bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range);
+
+#ifdef KVM_ARCH_WANT_MMU_NOTIFIER
 bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range);
 bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range);
 bool kvm_set_spte_gfn(struct kvm *kvm, struct kvm_gfn_range *range);
@@ -785,11 +786,12 @@ struct kvm {
 
 #if defined(CONFIG_MMU_NOTIFIER) && defined(KVM_ARCH_WANT_MMU_NOTIFIER)
 	struct mmu_notifier mmu_notifier;
+#endif
 	unsigned long mmu_invalidate_seq;
 	long mmu_invalidate_in_progress;
 	gfn_t mmu_invalidate_range_start;
 	gfn_t mmu_invalidate_range_end;
-#endif
+
 	struct list_head devices;
 	u64 manual_dirty_log_protect;
 	struct dentry *debugfs_dentry;
@@ -1480,6 +1482,7 @@ bool kvm_arch_dy_has_pending_interrupt(struct kvm_vcpu *vcpu);
 int kvm_arch_post_init_vm(struct kvm *kvm);
 void kvm_arch_pre_destroy_vm(struct kvm *kvm);
 int kvm_arch_create_vm_debugfs(struct kvm *kvm);
+bool kvm_arch_has_private_mem(struct kvm *kvm);
 
 #ifndef __KVM_HAVE_ARCH_VM_ALLOC
 /*
virt/kvm/kvm_main.c +94 -57
index ad55dfbc75d7..4e1e1e113bf0 100644
@@ -520,6 +520,62 @@ void kvm_destroy_vcpus(struct kvm *kvm)
 }
 EXPORT_SYMBOL_GPL(kvm_destroy_vcpus);
 
+void kvm_mmu_invalidate_begin(struct kvm *kvm)
+{
+	/*
+	 * The count increase must become visible at unlock time as no
+	 * spte can be established without taking the mmu_lock and
+	 * count is also read inside the mmu_lock critical section.
+	 */
+	kvm->mmu_invalidate_in_progress++;
+
+	if (likely(kvm->mmu_invalidate_in_progress == 1)) {
+		kvm->mmu_invalidate_range_start = INVALID_GPA;
+		kvm->mmu_invalidate_range_end = INVALID_GPA;
+	}
+}
+
+void kvm_mmu_invalidate_range_add(struct kvm *kvm, gfn_t start, gfn_t end)
+{
+	WARN_ON_ONCE(!kvm->mmu_invalidate_in_progress);
+
+	if (likely(kvm->mmu_invalidate_in_progress == 1)) {
+		kvm->mmu_invalidate_range_start = start;
+		kvm->mmu_invalidate_range_end = end;
+	} else {
+		/*
+		 * Fully tracking multiple concurrent ranges has diminishing
+		 * returns. Keep things simple and just find the minimal range
+		 * which includes the current and new ranges. As there won't be
+		 * enough information to subtract a range after its invalidate
+		 * completes, any ranges invalidated concurrently will
+		 * accumulate and persist until all outstanding invalidates
+		 * complete.
+		 */
+		kvm->mmu_invalidate_range_start =
+			min(kvm->mmu_invalidate_range_start, start);
+		kvm->mmu_invalidate_range_end =
+			max(kvm->mmu_invalidate_range_end, end);
+	}
+}
+
+void kvm_mmu_invalidate_end(struct kvm *kvm)
+{
+	/*
+	 * This sequence increase will notify the kvm page fault that
+	 * the page that is going to be mapped in the spte could have
+	 * been freed.
+	 */
+	kvm->mmu_invalidate_seq++;
+	smp_wmb();
+	/*
+	 * The above sequence increase must be visible before the
+	 * below count decrease, which is ensured by the smp_wmb above
+	 * in conjunction with the smp_rmb in mmu_invalidate_retry().
+	 */
+	kvm->mmu_invalidate_in_progress--;
+}
+
 #if defined(CONFIG_MMU_NOTIFIER) && defined(KVM_ARCH_WANT_MMU_NOTIFIER)
 static inline struct kvm *mmu_notifier_to_kvm(struct mmu_notifier *mn)
 {
@@ -714,45 +770,6 @@ static void kvm_mmu_notifier_change_pte(struct mmu_notifier *mn,
 	kvm_handle_hva_range(mn, address, address + 1, pte, kvm_set_spte_gfn);
 }
 
-void kvm_mmu_invalidate_begin(struct kvm *kvm)
-{
-	/*
-	 * The count increase must become visible at unlock time as no
-	 * spte can be established without taking the mmu_lock and
-	 * count is also read inside the mmu_lock critical section.
-	 */
-	kvm->mmu_invalidate_in_progress++;
-
-	if (likely(kvm->mmu_invalidate_in_progress == 1)) {
-		kvm->mmu_invalidate_range_start = INVALID_GPA;
-		kvm->mmu_invalidate_range_end = INVALID_GPA;
-	}
-}
-
-void kvm_mmu_invalidate_range_add(struct kvm *kvm, gfn_t start, gfn_t end)
-{
-	WARN_ON_ONCE(!kvm->mmu_invalidate_in_progress);
-
-	if (likely(kvm->mmu_invalidate_in_progress == 1)) {
-		kvm->mmu_invalidate_range_start = start;
-		kvm->mmu_invalidate_range_end = end;
-	} else {
-		/*
-		 * Fully tracking multiple concurrent ranges has diminishing
-		 * returns. Keep things simple and just find the minimal range
-		 * which includes the current and new ranges. As there won't be
-		 * enough information to subtract a range after its invalidate
-		 * completes, any ranges invalidated concurrently will
-		 * accumulate and persist until all outstanding invalidates
-		 * complete.
-		 */
-		kvm->mmu_invalidate_range_start =
-			min(kvm->mmu_invalidate_range_start, start);
-		kvm->mmu_invalidate_range_end =
-			max(kvm->mmu_invalidate_range_end, end);
-	}
-}
-
 static bool kvm_mmu_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
 {
 	kvm_mmu_invalidate_range_add(kvm, range->start, range->end);
@@ -806,23 +823,6 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,
 	return 0;
 }
 
-void kvm_mmu_invalidate_end(struct kvm *kvm)
-{
-	/*
-	 * This sequence increase will notify the kvm page fault that
-	 * the page that is going to be mapped in the spte could have
-	 * been freed.
-	 */
-	kvm->mmu_invalidate_seq++;
-	smp_wmb();
-	/*
-	 * The above sequence increase must be visible before the
-	 * below count decrease, which is ensured by the smp_wmb above
-	 * in conjunction with the smp_rmb in mmu_invalidate_retry().
-	 */
-	kvm->mmu_invalidate_in_progress--;
-}
-
 static void kvm_mmu_notifier_invalidate_range_end(struct mmu_notifier *mn,
 					const struct mmu_notifier_range *range)
 {
@@ -1140,6 +1140,11 @@ int __weak kvm_arch_create_vm_debugfs(struct kvm *kvm)
 	return 0;
 }
 
+bool __weak kvm_arch_has_private_mem(struct kvm *kvm)
+{
+	return false;
+}
+
 static struct kvm *kvm_create_vm(unsigned long type, const char *fdname)
 {
 	struct kvm *kvm = kvm_arch_alloc_vm();
@@ -2349,15 +2354,47 @@ static u64 kvm_supported_mem_attributes(struct kvm *kvm)
 	return 0;
 }
 
+static void kvm_unmap_mem_range(struct kvm *kvm, gfn_t start, gfn_t end)
+{
+	struct kvm_gfn_range gfn_range;
+	struct kvm_memory_slot *slot;
+	struct kvm_memslots *slots;
+	struct kvm_memslot_iter iter;
+	int i;
+	int r = 0;
+
+	gfn_range.pte = __pte(0);
+	gfn_range.may_block = true;
+
+	for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++) {
+		slots = __kvm_memslots(kvm, i);
+
+		kvm_for_each_memslot_in_gfn_range(&iter, slots, start, end) {
+			slot = iter.slot;
+			gfn_range.start = max(start, slot->base_gfn);
+			gfn_range.end = min(end, slot->base_gfn + slot->npages);
+			if (gfn_range.start >= gfn_range.end)
+				continue;
+			gfn_range.slot = slot;
+
+			r |= kvm_unmap_gfn_range(kvm, &gfn_range);
+		}
+	}
+
+	if (r)
+		kvm_flush_remote_tlbs(kvm);
+}
+
 static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
 					   struct kvm_memory_attributes *attrs)
 {
 	gfn_t start, end;
 	unsigned long i;
 	void *entry;
+	int idx;
 	u64 supported_attrs = kvm_supported_mem_attributes(kvm);
 
-	/* flags is currently not used. */
+	/* 'flags' is currently not used. */
 	if (attrs->flags)
 		return -EINVAL;
 	if (attrs->attributes & ~supported_attrs)

virt/kvm/kvm_main.c static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, · line 2415

Quoted 7 lines
virt/kvm/kvm_main.c +4 -0
@@ -2372,6 +2409,13 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
 
 	entry = attrs->attributes ? xa_mk_value(attrs->attributes) : NULL;
 
+	if (kvm_arch_has_private_mem(kvm)) {
+		KVM_MMU_LOCK(kvm);
+		kvm_mmu_invalidate_begin(kvm);
+		kvm_mmu_invalidate_range_add(kvm, start, end);

Nit: this works for KVM_MEMORY_ATTRIBUTE_PRIVATE, but the invalidation should be necessary yet for attribute change of:

Quoted 8 lines
Quoted 3 lines

KVM_MEMORY_ATTRIBUTE_READ
KVM_MEMORY_ATTRIBUTE_WRITE
KVM_MEMORY_ATTRIBUTE_EXECUTE

The unmapping is only needed for confidential usages which uses KVM_MEMORY_ATTRIBUTE_PRIVATE only and the other flags are defined here for other usages like pKVM. As Fuad commented in a different reply, pKVM supports in-place remapping and unmapping is unnecessary.

Ah, I see. It's fine to me, thanks.

Quoted 34 lines

Thanks,
Chao

Quoted 31 lines
Quoted 8 lines
virt/kvm/kvm_main.c +3 -0
+		KVM_MMU_UNLOCK(kvm);
+	}
+
 	mutex_lock(&kvm->lock);
 	for (i = start; i < end; i++)
 		if (xa_err(xa_store(&kvm->mem_attr_array, i, entry,

virt/kvm/kvm_main.c static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, · line 2431

Quoted 9 lines
virt/kvm/kvm_main.c +6 -0
@@ -2379,6 +2423,16 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
 			break;
 	mutex_unlock(&kvm->lock);
 
+	if (kvm_arch_has_private_mem(kvm)) {
+		idx = srcu_read_lock(&kvm->srcu);
+		KVM_MMU_LOCK(kvm);
+		if (i > start)
+			kvm_unmap_mem_range(kvm, start, i);
+		kvm_mmu_invalidate_end(kvm);

Ditto.

Quoted 11 lines
virt/kvm/kvm_main.c +4 -1
+		KVM_MMU_UNLOCK(kvm);
+		srcu_read_unlock(&kvm->srcu, idx);
+	}
+
 	attrs->address = i << PAGE_SHIFT;
 	attrs->size = (end - i) << PAGE_SHIFT;
 
--

2.25.1

#272 lore raw

Re: [PATCH v10 6/9] KVM: Unmap existing mappings when change the memory attributes

In reply to Chao Peng

Hi,

Quoted 10 lines · Chao Peng

Unmap the existing guest mappings when memory attribute is changed between shared and private. This is needed because shared pages and private pages are from different backends, unmapping existing ones gives a chance for page fault handler to re-populate the mappings according to the new attribute.

Only architecture has private memory support needs this and the supported architecture is expected to rewrite the weak kvm_arch_has_private_mem().

This kind of ties into the discussion of being able to share memory in place. For pKVM for example, shared and private memory would have the same backend, and the unmapping wouldn't be needed.

So I guess that, instead of kvm_arch_has_private_mem(), can the check be done differently, e.g., with a different function, say kvm_arch_private_notify_attribute_change() (but maybe with a more friendly name than what I suggested :) )?

Thanks,
/fuad

Quoted 287 lines

Also, during memory attribute changing and the unmapping time frame, page fault handler may happen in the same memory range and can cause incorrect page state, invoke kvm_mmu_invalidate_* helpers to let the page fault handler retry during this time frame.

---
 include/linux/kvm_host.h |   7 +-
 virt/kvm/kvm_main.c      | 168 ++++++++++++++++++++++++++-------------
 2 files changed, 116 insertions(+), 59 deletions(-)
include/linux/kvm_host.h +5 -2
index 3d69484d2704..3331c0c92838 100644
@@ -255,7 +255,6 @@ bool kvm_setup_async_pf(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 int kvm_async_pf_wakeup_all(struct kvm_vcpu *vcpu);
 #endif
 
-#ifdef KVM_ARCH_WANT_MMU_NOTIFIER
 struct kvm_gfn_range {
        struct kvm_memory_slot *slot;
        gfn_t start;
@@ -264,6 +263,8 @@ struct kvm_gfn_range {
        bool may_block;
 };
 bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range);
+
+#ifdef KVM_ARCH_WANT_MMU_NOTIFIER
 bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range);
 bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range);
 bool kvm_set_spte_gfn(struct kvm *kvm, struct kvm_gfn_range *range);
@@ -785,11 +786,12 @@ struct kvm {
 
 #if defined(CONFIG_MMU_NOTIFIER) && defined(KVM_ARCH_WANT_MMU_NOTIFIER)
        struct mmu_notifier mmu_notifier;
+#endif
        unsigned long mmu_invalidate_seq;
        long mmu_invalidate_in_progress;
        gfn_t mmu_invalidate_range_start;
        gfn_t mmu_invalidate_range_end;
-#endif
+
        struct list_head devices;
        u64 manual_dirty_log_protect;
        struct dentry *debugfs_dentry;
@@ -1480,6 +1482,7 @@ bool kvm_arch_dy_has_pending_interrupt(struct kvm_vcpu *vcpu);
 int kvm_arch_post_init_vm(struct kvm *kvm);
 void kvm_arch_pre_destroy_vm(struct kvm *kvm);
 int kvm_arch_create_vm_debugfs(struct kvm *kvm);
+bool kvm_arch_has_private_mem(struct kvm *kvm);
 
 #ifndef __KVM_HAVE_ARCH_VM_ALLOC
 /*
virt/kvm/kvm_main.c +111 -58
index ad55dfbc75d7..4e1e1e113bf0 100644
@@ -520,6 +520,62 @@ void kvm_destroy_vcpus(struct kvm *kvm)
 }
 EXPORT_SYMBOL_GPL(kvm_destroy_vcpus);
 
+void kvm_mmu_invalidate_begin(struct kvm *kvm)
+{
+       /*
+        * The count increase must become visible at unlock time as no
+        * spte can be established without taking the mmu_lock and
+        * count is also read inside the mmu_lock critical section.
+        */
+       kvm->mmu_invalidate_in_progress++;
+
+       if (likely(kvm->mmu_invalidate_in_progress == 1)) {
+               kvm->mmu_invalidate_range_start = INVALID_GPA;
+               kvm->mmu_invalidate_range_end = INVALID_GPA;
+       }
+}
+
+void kvm_mmu_invalidate_range_add(struct kvm *kvm, gfn_t start, gfn_t end)
+{
+       WARN_ON_ONCE(!kvm->mmu_invalidate_in_progress);
+
+       if (likely(kvm->mmu_invalidate_in_progress == 1)) {
+               kvm->mmu_invalidate_range_start = start;
+               kvm->mmu_invalidate_range_end = end;
+       } else {
+               /*
+                * Fully tracking multiple concurrent ranges has diminishing
+                * returns. Keep things simple and just find the minimal range
+                * which includes the current and new ranges. As there won't be
+                * enough information to subtract a range after its invalidate
+                * completes, any ranges invalidated concurrently will
+                * accumulate and persist until all outstanding invalidates
+                * complete.
+                */
+               kvm->mmu_invalidate_range_start =
+                       min(kvm->mmu_invalidate_range_start, start);
+               kvm->mmu_invalidate_range_end =
+                       max(kvm->mmu_invalidate_range_end, end);
+       }
+}
+
+void kvm_mmu_invalidate_end(struct kvm *kvm)
+{
+       /*
+        * This sequence increase will notify the kvm page fault that
+        * the page that is going to be mapped in the spte could have
+        * been freed.
+        */
+       kvm->mmu_invalidate_seq++;
+       smp_wmb();
+       /*
+        * The above sequence increase must be visible before the
+        * below count decrease, which is ensured by the smp_wmb above
+        * in conjunction with the smp_rmb in mmu_invalidate_retry().
+        */
+       kvm->mmu_invalidate_in_progress--;
+}
+
 #if defined(CONFIG_MMU_NOTIFIER) && defined(KVM_ARCH_WANT_MMU_NOTIFIER)
 static inline struct kvm *mmu_notifier_to_kvm(struct mmu_notifier *mn)
 {
@@ -714,45 +770,6 @@ static void kvm_mmu_notifier_change_pte(struct mmu_notifier *mn,
        kvm_handle_hva_range(mn, address, address + 1, pte, kvm_set_spte_gfn);
 }
 
-void kvm_mmu_invalidate_begin(struct kvm *kvm)
-{
-       /*
-        * The count increase must become visible at unlock time as no
-        * spte can be established without taking the mmu_lock and
-        * count is also read inside the mmu_lock critical section.
-        */
-       kvm->mmu_invalidate_in_progress++;
-
-       if (likely(kvm->mmu_invalidate_in_progress == 1)) {
-               kvm->mmu_invalidate_range_start = INVALID_GPA;
-               kvm->mmu_invalidate_range_end = INVALID_GPA;
-       }
-}
-
-void kvm_mmu_invalidate_range_add(struct kvm *kvm, gfn_t start, gfn_t end)
-{
-       WARN_ON_ONCE(!kvm->mmu_invalidate_in_progress);
-
-       if (likely(kvm->mmu_invalidate_in_progress == 1)) {
-               kvm->mmu_invalidate_range_start = start;
-               kvm->mmu_invalidate_range_end = end;
-       } else {
-               /*
-                * Fully tracking multiple concurrent ranges has diminishing
-                * returns. Keep things simple and just find the minimal range
-                * which includes the current and new ranges. As there won't be
-                * enough information to subtract a range after its invalidate
-                * completes, any ranges invalidated concurrently will
-                * accumulate and persist until all outstanding invalidates
-                * complete.
-                */
-               kvm->mmu_invalidate_range_start =
-                       min(kvm->mmu_invalidate_range_start, start);
-               kvm->mmu_invalidate_range_end =
-                       max(kvm->mmu_invalidate_range_end, end);
-       }
-}
-
 static bool kvm_mmu_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
 {
        kvm_mmu_invalidate_range_add(kvm, range->start, range->end);
@@ -806,23 +823,6 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,
        return 0;
 }
 
-void kvm_mmu_invalidate_end(struct kvm *kvm)
-{
-       /*
-        * This sequence increase will notify the kvm page fault that
-        * the page that is going to be mapped in the spte could have
-        * been freed.
-        */
-       kvm->mmu_invalidate_seq++;
-       smp_wmb();
-       /*
-        * The above sequence increase must be visible before the
-        * below count decrease, which is ensured by the smp_wmb above
-        * in conjunction with the smp_rmb in mmu_invalidate_retry().
-        */
-       kvm->mmu_invalidate_in_progress--;
-}
-
 static void kvm_mmu_notifier_invalidate_range_end(struct mmu_notifier *mn,
                                        const struct mmu_notifier_range *range)
 {
@@ -1140,6 +1140,11 @@ int __weak kvm_arch_create_vm_debugfs(struct kvm *kvm)
        return 0;
 }
 
+bool __weak kvm_arch_has_private_mem(struct kvm *kvm)
+{
+       return false;
+}
+
 static struct kvm *kvm_create_vm(unsigned long type, const char *fdname)
 {
        struct kvm *kvm = kvm_arch_alloc_vm();
@@ -2349,15 +2354,47 @@ static u64 kvm_supported_mem_attributes(struct kvm *kvm)
        return 0;
 }
 
+static void kvm_unmap_mem_range(struct kvm *kvm, gfn_t start, gfn_t end)
+{
+       struct kvm_gfn_range gfn_range;
+       struct kvm_memory_slot *slot;
+       struct kvm_memslots *slots;
+       struct kvm_memslot_iter iter;
+       int i;
+       int r = 0;
+
+       gfn_range.pte = __pte(0);
+       gfn_range.may_block = true;
+
+       for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++) {
+               slots = __kvm_memslots(kvm, i);
+
+               kvm_for_each_memslot_in_gfn_range(&iter, slots, start, end) {
+                       slot = iter.slot;
+                       gfn_range.start = max(start, slot->base_gfn);
+                       gfn_range.end = min(end, slot->base_gfn + slot->npages);
+                       if (gfn_range.start >= gfn_range.end)
+                               continue;
+                       gfn_range.slot = slot;
+
+                       r |= kvm_unmap_gfn_range(kvm, &gfn_range);
+               }
+       }
+
+       if (r)
+               kvm_flush_remote_tlbs(kvm);
+}
+
 static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
                                           struct kvm_memory_attributes *attrs)
 {
        gfn_t start, end;
        unsigned long i;
        void *entry;
+       int idx;
        u64 supported_attrs = kvm_supported_mem_attributes(kvm);
 
-       /* flags is currently not used. */
+       /* 'flags' is currently not used. */
        if (attrs->flags)
                return -EINVAL;
        if (attrs->attributes & ~supported_attrs)
@@ -2372,6 +2409,13 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
 
        entry = attrs->attributes ? xa_mk_value(attrs->attributes) : NULL;
 
+       if (kvm_arch_has_private_mem(kvm)) {
+               KVM_MMU_LOCK(kvm);
+               kvm_mmu_invalidate_begin(kvm);
+               kvm_mmu_invalidate_range_add(kvm, start, end);
+               KVM_MMU_UNLOCK(kvm);
+       }
+
        mutex_lock(&kvm->lock);
        for (i = start; i < end; i++)
                if (xa_err(xa_store(&kvm->mem_attr_array, i, entry,
@@ -2379,6 +2423,16 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
                        break;
        mutex_unlock(&kvm->lock);
 
+       if (kvm_arch_has_private_mem(kvm)) {
+               idx = srcu_read_lock(&kvm->srcu);
+               KVM_MMU_LOCK(kvm);
+               if (i > start)
+                       kvm_unmap_mem_range(kvm, start, i);
+               kvm_mmu_invalidate_end(kvm);
+               KVM_MMU_UNLOCK(kvm);
+               srcu_read_unlock(&kvm->srcu, idx);
+       }
+
        attrs->address = i << PAGE_SHIFT;
        attrs->size = (end - i) << PAGE_SHIFT;
 
--

2.25.1

author
#273 lore raw

Re: [PATCH v10 6/9] KVM: Unmap existing mappings when change the memory attributes

Quoted 14 lines · Fuad Tabba

Hi,

Quoted 10 lines · Chao Peng

Unmap the existing guest mappings when memory attribute is changed between shared and private. This is needed because shared pages and private pages are from different backends, unmapping existing ones gives a chance for page fault handler to re-populate the mappings according to the new attribute.

Only architecture has private memory support needs this and the supported architecture is expected to rewrite the weak kvm_arch_has_private_mem().

Quoted 8 lines

This kind of ties into the discussion of being able to share memory in place. For pKVM for example, shared and private memory would have the same backend, and the unmapping wouldn't be needed.

So I guess that, instead of kvm_arch_has_private_mem(), can the check be done differently, e.g., with a different function, say kvm_arch_private_notify_attribute_change() (but maybe with a more friendly name than what I suggested :) )?

Besides controlling the unmapping here, kvm_arch_has_private_mem() is also used to gate the memslot KVM_MEM_PRIVATE flag in patch09. I know unmapping is confirmed unnecessary for pKVM, but how about KVM_MEM_PRIVATE? Will pKVM add its own flag or reuse KVM_MEM_PRIVATE? If the answer is the latter, then yes we should use a different check which only works for confidential usages here.

Thanks,
Chao

Quoted 291 lines

Thanks,
/fuad

Quoted 287 lines

Also, during memory attribute changing and the unmapping time frame, page fault handler may happen in the same memory range and can cause incorrect page state, invoke kvm_mmu_invalidate_* helpers to let the page fault handler retry during this time frame.

---
 include/linux/kvm_host.h |   7 +-
 virt/kvm/kvm_main.c      | 168 ++++++++++++++++++++++++++-------------
 2 files changed, 116 insertions(+), 59 deletions(-)
include/linux/kvm_host.h +5 -2
index 3d69484d2704..3331c0c92838 100644
@@ -255,7 +255,6 @@ bool kvm_setup_async_pf(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 int kvm_async_pf_wakeup_all(struct kvm_vcpu *vcpu);
 #endif
 
-#ifdef KVM_ARCH_WANT_MMU_NOTIFIER
 struct kvm_gfn_range {
        struct kvm_memory_slot *slot;
        gfn_t start;
@@ -264,6 +263,8 @@ struct kvm_gfn_range {
        bool may_block;
 };
 bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range);
+
+#ifdef KVM_ARCH_WANT_MMU_NOTIFIER
 bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range);
 bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range);
 bool kvm_set_spte_gfn(struct kvm *kvm, struct kvm_gfn_range *range);
@@ -785,11 +786,12 @@ struct kvm {
 
 #if defined(CONFIG_MMU_NOTIFIER) && defined(KVM_ARCH_WANT_MMU_NOTIFIER)
        struct mmu_notifier mmu_notifier;
+#endif
        unsigned long mmu_invalidate_seq;
        long mmu_invalidate_in_progress;
        gfn_t mmu_invalidate_range_start;
        gfn_t mmu_invalidate_range_end;
-#endif
+
        struct list_head devices;
        u64 manual_dirty_log_protect;
        struct dentry *debugfs_dentry;
@@ -1480,6 +1482,7 @@ bool kvm_arch_dy_has_pending_interrupt(struct kvm_vcpu *vcpu);
 int kvm_arch_post_init_vm(struct kvm *kvm);
 void kvm_arch_pre_destroy_vm(struct kvm *kvm);
 int kvm_arch_create_vm_debugfs(struct kvm *kvm);
+bool kvm_arch_has_private_mem(struct kvm *kvm);
 
 #ifndef __KVM_HAVE_ARCH_VM_ALLOC
 /*
virt/kvm/kvm_main.c +111 -58
index ad55dfbc75d7..4e1e1e113bf0 100644
@@ -520,6 +520,62 @@ void kvm_destroy_vcpus(struct kvm *kvm)
 }
 EXPORT_SYMBOL_GPL(kvm_destroy_vcpus);
 
+void kvm_mmu_invalidate_begin(struct kvm *kvm)
+{
+       /*
+        * The count increase must become visible at unlock time as no
+        * spte can be established without taking the mmu_lock and
+        * count is also read inside the mmu_lock critical section.
+        */
+       kvm->mmu_invalidate_in_progress++;
+
+       if (likely(kvm->mmu_invalidate_in_progress == 1)) {
+               kvm->mmu_invalidate_range_start = INVALID_GPA;
+               kvm->mmu_invalidate_range_end = INVALID_GPA;
+       }
+}
+
+void kvm_mmu_invalidate_range_add(struct kvm *kvm, gfn_t start, gfn_t end)
+{
+       WARN_ON_ONCE(!kvm->mmu_invalidate_in_progress);
+
+       if (likely(kvm->mmu_invalidate_in_progress == 1)) {
+               kvm->mmu_invalidate_range_start = start;
+               kvm->mmu_invalidate_range_end = end;
+       } else {
+               /*
+                * Fully tracking multiple concurrent ranges has diminishing
+                * returns. Keep things simple and just find the minimal range
+                * which includes the current and new ranges. As there won't be
+                * enough information to subtract a range after its invalidate
+                * completes, any ranges invalidated concurrently will
+                * accumulate and persist until all outstanding invalidates
+                * complete.
+                */
+               kvm->mmu_invalidate_range_start =
+                       min(kvm->mmu_invalidate_range_start, start);
+               kvm->mmu_invalidate_range_end =
+                       max(kvm->mmu_invalidate_range_end, end);
+       }
+}
+
+void kvm_mmu_invalidate_end(struct kvm *kvm)
+{
+       /*
+        * This sequence increase will notify the kvm page fault that
+        * the page that is going to be mapped in the spte could have
+        * been freed.
+        */
+       kvm->mmu_invalidate_seq++;
+       smp_wmb();
+       /*
+        * The above sequence increase must be visible before the
+        * below count decrease, which is ensured by the smp_wmb above
+        * in conjunction with the smp_rmb in mmu_invalidate_retry().
+        */
+       kvm->mmu_invalidate_in_progress--;
+}
+
 #if defined(CONFIG_MMU_NOTIFIER) && defined(KVM_ARCH_WANT_MMU_NOTIFIER)
 static inline struct kvm *mmu_notifier_to_kvm(struct mmu_notifier *mn)
 {
@@ -714,45 +770,6 @@ static void kvm_mmu_notifier_change_pte(struct mmu_notifier *mn,
        kvm_handle_hva_range(mn, address, address + 1, pte, kvm_set_spte_gfn);
 }
 
-void kvm_mmu_invalidate_begin(struct kvm *kvm)
-{
-       /*
-        * The count increase must become visible at unlock time as no
-        * spte can be established without taking the mmu_lock and
-        * count is also read inside the mmu_lock critical section.
-        */
-       kvm->mmu_invalidate_in_progress++;
-
-       if (likely(kvm->mmu_invalidate_in_progress == 1)) {
-               kvm->mmu_invalidate_range_start = INVALID_GPA;
-               kvm->mmu_invalidate_range_end = INVALID_GPA;
-       }
-}
-
-void kvm_mmu_invalidate_range_add(struct kvm *kvm, gfn_t start, gfn_t end)
-{
-       WARN_ON_ONCE(!kvm->mmu_invalidate_in_progress);
-
-       if (likely(kvm->mmu_invalidate_in_progress == 1)) {
-               kvm->mmu_invalidate_range_start = start;
-               kvm->mmu_invalidate_range_end = end;
-       } else {
-               /*
-                * Fully tracking multiple concurrent ranges has diminishing
-                * returns. Keep things simple and just find the minimal range
-                * which includes the current and new ranges. As there won't be
-                * enough information to subtract a range after its invalidate
-                * completes, any ranges invalidated concurrently will
-                * accumulate and persist until all outstanding invalidates
-                * complete.
-                */
-               kvm->mmu_invalidate_range_start =
-                       min(kvm->mmu_invalidate_range_start, start);
-               kvm->mmu_invalidate_range_end =
-                       max(kvm->mmu_invalidate_range_end, end);
-       }
-}
-
 static bool kvm_mmu_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
 {
        kvm_mmu_invalidate_range_add(kvm, range->start, range->end);
@@ -806,23 +823,6 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,
        return 0;
 }
 
-void kvm_mmu_invalidate_end(struct kvm *kvm)
-{
-       /*
-        * This sequence increase will notify the kvm page fault that
-        * the page that is going to be mapped in the spte could have
-        * been freed.
-        */
-       kvm->mmu_invalidate_seq++;
-       smp_wmb();
-       /*
-        * The above sequence increase must be visible before the
-        * below count decrease, which is ensured by the smp_wmb above
-        * in conjunction with the smp_rmb in mmu_invalidate_retry().
-        */
-       kvm->mmu_invalidate_in_progress--;
-}
-
 static void kvm_mmu_notifier_invalidate_range_end(struct mmu_notifier *mn,
                                        const struct mmu_notifier_range *range)
 {
@@ -1140,6 +1140,11 @@ int __weak kvm_arch_create_vm_debugfs(struct kvm *kvm)
        return 0;
 }
 
+bool __weak kvm_arch_has_private_mem(struct kvm *kvm)
+{
+       return false;
+}
+
 static struct kvm *kvm_create_vm(unsigned long type, const char *fdname)
 {
        struct kvm *kvm = kvm_arch_alloc_vm();
@@ -2349,15 +2354,47 @@ static u64 kvm_supported_mem_attributes(struct kvm *kvm)
        return 0;
 }
 
+static void kvm_unmap_mem_range(struct kvm *kvm, gfn_t start, gfn_t end)
+{
+       struct kvm_gfn_range gfn_range;
+       struct kvm_memory_slot *slot;
+       struct kvm_memslots *slots;
+       struct kvm_memslot_iter iter;
+       int i;
+       int r = 0;
+
+       gfn_range.pte = __pte(0);
+       gfn_range.may_block = true;
+
+       for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++) {
+               slots = __kvm_memslots(kvm, i);
+
+               kvm_for_each_memslot_in_gfn_range(&iter, slots, start, end) {
+                       slot = iter.slot;
+                       gfn_range.start = max(start, slot->base_gfn);
+                       gfn_range.end = min(end, slot->base_gfn + slot->npages);
+                       if (gfn_range.start >= gfn_range.end)
+                               continue;
+                       gfn_range.slot = slot;
+
+                       r |= kvm_unmap_gfn_range(kvm, &gfn_range);
+               }
+       }
+
+       if (r)
+               kvm_flush_remote_tlbs(kvm);
+}
+
 static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
                                           struct kvm_memory_attributes *attrs)
 {
        gfn_t start, end;
        unsigned long i;
        void *entry;
+       int idx;
        u64 supported_attrs = kvm_supported_mem_attributes(kvm);
 
-       /* flags is currently not used. */
+       /* 'flags' is currently not used. */
        if (attrs->flags)
                return -EINVAL;
        if (attrs->attributes & ~supported_attrs)
@@ -2372,6 +2409,13 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
 
        entry = attrs->attributes ? xa_mk_value(attrs->attributes) : NULL;
 
+       if (kvm_arch_has_private_mem(kvm)) {
+               KVM_MMU_LOCK(kvm);
+               kvm_mmu_invalidate_begin(kvm);
+               kvm_mmu_invalidate_range_add(kvm, start, end);
+               KVM_MMU_UNLOCK(kvm);
+       }
+
        mutex_lock(&kvm->lock);
        for (i = start; i < end; i++)
                if (xa_err(xa_store(&kvm->mem_attr_array, i, entry,
@@ -2379,6 +2423,16 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
                        break;
        mutex_unlock(&kvm->lock);
 
+       if (kvm_arch_has_private_mem(kvm)) {
+               idx = srcu_read_lock(&kvm->srcu);
+               KVM_MMU_LOCK(kvm);
+               if (i > start)
+                       kvm_unmap_mem_range(kvm, start, i);
+               kvm_mmu_invalidate_end(kvm);
+               KVM_MMU_UNLOCK(kvm);
+               srcu_read_unlock(&kvm->srcu, idx);
+       }
+
        attrs->address = i << PAGE_SHIFT;
        attrs->size = (end - i) << PAGE_SHIFT;
 
--

2.25.1

#274 lore raw

Re: [PATCH v10 6/9] KVM: Unmap existing mappings when change the memory attributes

Hi,

Quoted 25 lines · Chao Peng
Quoted 14 lines · Fuad Tabba

Hi,

Quoted 10 lines · Chao Peng

Unmap the existing guest mappings when memory attribute is changed between shared and private. This is needed because shared pages and private pages are from different backends, unmapping existing ones gives a chance for page fault handler to re-populate the mappings according to the new attribute.

Only architecture has private memory support needs this and the supported architecture is expected to rewrite the weak kvm_arch_has_private_mem().

Quoted 8 lines

This kind of ties into the discussion of being able to share memory in place. For pKVM for example, shared and private memory would have the same backend, and the unmapping wouldn't be needed.

So I guess that, instead of kvm_arch_has_private_mem(), can the check be done differently, e.g., with a different function, say kvm_arch_private_notify_attribute_change() (but maybe with a more friendly name than what I suggested :) )?

Quoted 6 lines

Besides controlling the unmapping here, kvm_arch_has_private_mem() is also used to gate the memslot KVM_MEM_PRIVATE flag in patch09. I know unmapping is confirmed unnecessary for pKVM, but how about KVM_MEM_PRIVATE? Will pKVM add its own flag or reuse KVM_MEM_PRIVATE? If the answer is the latter, then yes we should use a different check which only works for confidential usages here.

I think it makes sense for pKVM to use the same flag (KVM_MEM_PRIVATE) and not to add another one.

Thank you,
/fuad

Quoted 294 lines

Thanks,
Chao

Quoted 291 lines

Thanks,
/fuad

Quoted 287 lines

Also, during memory attribute changing and the unmapping time frame, page fault handler may happen in the same memory range and can cause incorrect page state, invoke kvm_mmu_invalidate_* helpers to let the page fault handler retry during this time frame.

---
 include/linux/kvm_host.h |   7 +-
 virt/kvm/kvm_main.c      | 168 ++++++++++++++++++++++++++-------------
 2 files changed, 116 insertions(+), 59 deletions(-)
include/linux/kvm_host.h +5 -2
index 3d69484d2704..3331c0c92838 100644
@@ -255,7 +255,6 @@ bool kvm_setup_async_pf(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 int kvm_async_pf_wakeup_all(struct kvm_vcpu *vcpu);
 #endif
 
-#ifdef KVM_ARCH_WANT_MMU_NOTIFIER
 struct kvm_gfn_range {
        struct kvm_memory_slot *slot;
        gfn_t start;
@@ -264,6 +263,8 @@ struct kvm_gfn_range {
        bool may_block;
 };
 bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range);
+
+#ifdef KVM_ARCH_WANT_MMU_NOTIFIER
 bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range);
 bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range);
 bool kvm_set_spte_gfn(struct kvm *kvm, struct kvm_gfn_range *range);
@@ -785,11 +786,12 @@ struct kvm {
 
 #if defined(CONFIG_MMU_NOTIFIER) && defined(KVM_ARCH_WANT_MMU_NOTIFIER)
        struct mmu_notifier mmu_notifier;
+#endif
        unsigned long mmu_invalidate_seq;
        long mmu_invalidate_in_progress;
        gfn_t mmu_invalidate_range_start;
        gfn_t mmu_invalidate_range_end;
-#endif
+
        struct list_head devices;
        u64 manual_dirty_log_protect;
        struct dentry *debugfs_dentry;
@@ -1480,6 +1482,7 @@ bool kvm_arch_dy_has_pending_interrupt(struct kvm_vcpu *vcpu);
 int kvm_arch_post_init_vm(struct kvm *kvm);
 void kvm_arch_pre_destroy_vm(struct kvm *kvm);
 int kvm_arch_create_vm_debugfs(struct kvm *kvm);
+bool kvm_arch_has_private_mem(struct kvm *kvm);
 
 #ifndef __KVM_HAVE_ARCH_VM_ALLOC
 /*
virt/kvm/kvm_main.c +111 -58
index ad55dfbc75d7..4e1e1e113bf0 100644
@@ -520,6 +520,62 @@ void kvm_destroy_vcpus(struct kvm *kvm)
 }
 EXPORT_SYMBOL_GPL(kvm_destroy_vcpus);
 
+void kvm_mmu_invalidate_begin(struct kvm *kvm)
+{
+       /*
+        * The count increase must become visible at unlock time as no
+        * spte can be established without taking the mmu_lock and
+        * count is also read inside the mmu_lock critical section.
+        */
+       kvm->mmu_invalidate_in_progress++;
+
+       if (likely(kvm->mmu_invalidate_in_progress == 1)) {
+               kvm->mmu_invalidate_range_start = INVALID_GPA;
+               kvm->mmu_invalidate_range_end = INVALID_GPA;
+       }
+}
+
+void kvm_mmu_invalidate_range_add(struct kvm *kvm, gfn_t start, gfn_t end)
+{
+       WARN_ON_ONCE(!kvm->mmu_invalidate_in_progress);
+
+       if (likely(kvm->mmu_invalidate_in_progress == 1)) {
+               kvm->mmu_invalidate_range_start = start;
+               kvm->mmu_invalidate_range_end = end;
+       } else {
+               /*
+                * Fully tracking multiple concurrent ranges has diminishing
+                * returns. Keep things simple and just find the minimal range
+                * which includes the current and new ranges. As there won't be
+                * enough information to subtract a range after its invalidate
+                * completes, any ranges invalidated concurrently will
+                * accumulate and persist until all outstanding invalidates
+                * complete.
+                */
+               kvm->mmu_invalidate_range_start =
+                       min(kvm->mmu_invalidate_range_start, start);
+               kvm->mmu_invalidate_range_end =
+                       max(kvm->mmu_invalidate_range_end, end);
+       }
+}
+
+void kvm_mmu_invalidate_end(struct kvm *kvm)
+{
+       /*
+        * This sequence increase will notify the kvm page fault that
+        * the page that is going to be mapped in the spte could have
+        * been freed.
+        */
+       kvm->mmu_invalidate_seq++;
+       smp_wmb();
+       /*
+        * The above sequence increase must be visible before the
+        * below count decrease, which is ensured by the smp_wmb above
+        * in conjunction with the smp_rmb in mmu_invalidate_retry().
+        */
+       kvm->mmu_invalidate_in_progress--;
+}
+
 #if defined(CONFIG_MMU_NOTIFIER) && defined(KVM_ARCH_WANT_MMU_NOTIFIER)
 static inline struct kvm *mmu_notifier_to_kvm(struct mmu_notifier *mn)
 {
@@ -714,45 +770,6 @@ static void kvm_mmu_notifier_change_pte(struct mmu_notifier *mn,
        kvm_handle_hva_range(mn, address, address + 1, pte, kvm_set_spte_gfn);
 }
 
-void kvm_mmu_invalidate_begin(struct kvm *kvm)
-{
-       /*
-        * The count increase must become visible at unlock time as no
-        * spte can be established without taking the mmu_lock and
-        * count is also read inside the mmu_lock critical section.
-        */
-       kvm->mmu_invalidate_in_progress++;
-
-       if (likely(kvm->mmu_invalidate_in_progress == 1)) {
-               kvm->mmu_invalidate_range_start = INVALID_GPA;
-               kvm->mmu_invalidate_range_end = INVALID_GPA;
-       }
-}
-
-void kvm_mmu_invalidate_range_add(struct kvm *kvm, gfn_t start, gfn_t end)
-{
-       WARN_ON_ONCE(!kvm->mmu_invalidate_in_progress);
-
-       if (likely(kvm->mmu_invalidate_in_progress == 1)) {
-               kvm->mmu_invalidate_range_start = start;
-               kvm->mmu_invalidate_range_end = end;
-       } else {
-               /*
-                * Fully tracking multiple concurrent ranges has diminishing
-                * returns. Keep things simple and just find the minimal range
-                * which includes the current and new ranges. As there won't be
-                * enough information to subtract a range after its invalidate
-                * completes, any ranges invalidated concurrently will
-                * accumulate and persist until all outstanding invalidates
-                * complete.
-                */
-               kvm->mmu_invalidate_range_start =
-                       min(kvm->mmu_invalidate_range_start, start);
-               kvm->mmu_invalidate_range_end =
-                       max(kvm->mmu_invalidate_range_end, end);
-       }
-}
-
 static bool kvm_mmu_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
 {
        kvm_mmu_invalidate_range_add(kvm, range->start, range->end);
@@ -806,23 +823,6 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,
        return 0;
 }
 
-void kvm_mmu_invalidate_end(struct kvm *kvm)
-{
-       /*
-        * This sequence increase will notify the kvm page fault that
-        * the page that is going to be mapped in the spte could have
-        * been freed.
-        */
-       kvm->mmu_invalidate_seq++;
-       smp_wmb();
-       /*
-        * The above sequence increase must be visible before the
-        * below count decrease, which is ensured by the smp_wmb above
-        * in conjunction with the smp_rmb in mmu_invalidate_retry().
-        */
-       kvm->mmu_invalidate_in_progress--;
-}
-
 static void kvm_mmu_notifier_invalidate_range_end(struct mmu_notifier *mn,
                                        const struct mmu_notifier_range *range)
 {
@@ -1140,6 +1140,11 @@ int __weak kvm_arch_create_vm_debugfs(struct kvm *kvm)
        return 0;
 }
 
+bool __weak kvm_arch_has_private_mem(struct kvm *kvm)
+{
+       return false;
+}
+
 static struct kvm *kvm_create_vm(unsigned long type, const char *fdname)
 {
        struct kvm *kvm = kvm_arch_alloc_vm();
@@ -2349,15 +2354,47 @@ static u64 kvm_supported_mem_attributes(struct kvm *kvm)
        return 0;
 }
 
+static void kvm_unmap_mem_range(struct kvm *kvm, gfn_t start, gfn_t end)
+{
+       struct kvm_gfn_range gfn_range;
+       struct kvm_memory_slot *slot;
+       struct kvm_memslots *slots;
+       struct kvm_memslot_iter iter;
+       int i;
+       int r = 0;
+
+       gfn_range.pte = __pte(0);
+       gfn_range.may_block = true;
+
+       for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++) {
+               slots = __kvm_memslots(kvm, i);
+
+               kvm_for_each_memslot_in_gfn_range(&iter, slots, start, end) {
+                       slot = iter.slot;
+                       gfn_range.start = max(start, slot->base_gfn);
+                       gfn_range.end = min(end, slot->base_gfn + slot->npages);
+                       if (gfn_range.start >= gfn_range.end)
+                               continue;
+                       gfn_range.slot = slot;
+
+                       r |= kvm_unmap_gfn_range(kvm, &gfn_range);
+               }
+       }
+
+       if (r)
+               kvm_flush_remote_tlbs(kvm);
+}
+
 static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
                                           struct kvm_memory_attributes *attrs)
 {
        gfn_t start, end;
        unsigned long i;
        void *entry;
+       int idx;
        u64 supported_attrs = kvm_supported_mem_attributes(kvm);
 
-       /* flags is currently not used. */
+       /* 'flags' is currently not used. */
        if (attrs->flags)
                return -EINVAL;
        if (attrs->attributes & ~supported_attrs)
@@ -2372,6 +2409,13 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
 
        entry = attrs->attributes ? xa_mk_value(attrs->attributes) : NULL;
 
+       if (kvm_arch_has_private_mem(kvm)) {
+               KVM_MMU_LOCK(kvm);
+               kvm_mmu_invalidate_begin(kvm);
+               kvm_mmu_invalidate_range_add(kvm, start, end);
+               KVM_MMU_UNLOCK(kvm);
+       }
+
        mutex_lock(&kvm->lock);
        for (i = start; i < end; i++)
                if (xa_err(xa_store(&kvm->mem_attr_array, i, entry,
@@ -2379,6 +2423,16 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
                        break;
        mutex_unlock(&kvm->lock);
 
+       if (kvm_arch_has_private_mem(kvm)) {
+               idx = srcu_read_lock(&kvm->srcu);
+               KVM_MMU_LOCK(kvm);
+               if (i > start)
+                       kvm_unmap_mem_range(kvm, start, i);
+               kvm_mmu_invalidate_end(kvm);
+               KVM_MMU_UNLOCK(kvm);
+               srcu_read_unlock(&kvm->srcu, idx);
+       }
+
        attrs->address = i << PAGE_SHIFT;
        attrs->size = (end - i) << PAGE_SHIFT;
 
--

2.25.1

author
#275 lore raw

Re: [PATCH v10 6/9] KVM: Unmap existing mappings when change the memory attributes

Quoted 35 lines · Fuad Tabba

Hi,

Quoted 25 lines · Chao Peng
Quoted 14 lines · Fuad Tabba

Hi,

Quoted 10 lines · Chao Peng

Unmap the existing guest mappings when memory attribute is changed between shared and private. This is needed because shared pages and private pages are from different backends, unmapping existing ones gives a chance for page fault handler to re-populate the mappings according to the new attribute.

Only architecture has private memory support needs this and the supported architecture is expected to rewrite the weak kvm_arch_has_private_mem().

Quoted 8 lines

This kind of ties into the discussion of being able to share memory in place. For pKVM for example, shared and private memory would have the same backend, and the unmapping wouldn't be needed.

So I guess that, instead of kvm_arch_has_private_mem(), can the check be done differently, e.g., with a different function, say kvm_arch_private_notify_attribute_change() (but maybe with a more friendly name than what I suggested :) )?

Quoted 6 lines

Besides controlling the unmapping here, kvm_arch_has_private_mem() is also used to gate the memslot KVM_MEM_PRIVATE flag in patch09. I know unmapping is confirmed unnecessary for pKVM, but how about KVM_MEM_PRIVATE? Will pKVM add its own flag or reuse KVM_MEM_PRIVATE? If the answer is the latter, then yes we should use a different check which only works for confidential usages here.

Quoted 2 lines

I think it makes sense for pKVM to use the same flag (KVM_MEM_PRIVATE) and not to add another one.

Thanks for the reply.
Chao

Quoted 300 lines

Thank you,
/fuad

Quoted 294 lines

Thanks,
Chao

Quoted 291 lines

Thanks,
/fuad

Quoted 287 lines

Also, during memory attribute changing and the unmapping time frame, page fault handler may happen in the same memory range and can cause incorrect page state, invoke kvm_mmu_invalidate_* helpers to let the page fault handler retry during this time frame.

---
 include/linux/kvm_host.h |   7 +-
 virt/kvm/kvm_main.c      | 168 ++++++++++++++++++++++++++-------------
 2 files changed, 116 insertions(+), 59 deletions(-)
include/linux/kvm_host.h +5 -2
index 3d69484d2704..3331c0c92838 100644
@@ -255,7 +255,6 @@ bool kvm_setup_async_pf(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 int kvm_async_pf_wakeup_all(struct kvm_vcpu *vcpu);
 #endif
 
-#ifdef KVM_ARCH_WANT_MMU_NOTIFIER
 struct kvm_gfn_range {
        struct kvm_memory_slot *slot;
        gfn_t start;
@@ -264,6 +263,8 @@ struct kvm_gfn_range {
        bool may_block;
 };
 bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range);
+
+#ifdef KVM_ARCH_WANT_MMU_NOTIFIER
 bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range);
 bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range);
 bool kvm_set_spte_gfn(struct kvm *kvm, struct kvm_gfn_range *range);
@@ -785,11 +786,12 @@ struct kvm {
 
 #if defined(CONFIG_MMU_NOTIFIER) && defined(KVM_ARCH_WANT_MMU_NOTIFIER)
        struct mmu_notifier mmu_notifier;
+#endif
        unsigned long mmu_invalidate_seq;
        long mmu_invalidate_in_progress;
        gfn_t mmu_invalidate_range_start;
        gfn_t mmu_invalidate_range_end;
-#endif
+
        struct list_head devices;
        u64 manual_dirty_log_protect;
        struct dentry *debugfs_dentry;
@@ -1480,6 +1482,7 @@ bool kvm_arch_dy_has_pending_interrupt(struct kvm_vcpu *vcpu);
 int kvm_arch_post_init_vm(struct kvm *kvm);
 void kvm_arch_pre_destroy_vm(struct kvm *kvm);
 int kvm_arch_create_vm_debugfs(struct kvm *kvm);
+bool kvm_arch_has_private_mem(struct kvm *kvm);
 
 #ifndef __KVM_HAVE_ARCH_VM_ALLOC
 /*
virt/kvm/kvm_main.c +111 -58
index ad55dfbc75d7..4e1e1e113bf0 100644
@@ -520,6 +520,62 @@ void kvm_destroy_vcpus(struct kvm *kvm)
 }
 EXPORT_SYMBOL_GPL(kvm_destroy_vcpus);
 
+void kvm_mmu_invalidate_begin(struct kvm *kvm)
+{
+       /*
+        * The count increase must become visible at unlock time as no
+        * spte can be established without taking the mmu_lock and
+        * count is also read inside the mmu_lock critical section.
+        */
+       kvm->mmu_invalidate_in_progress++;
+
+       if (likely(kvm->mmu_invalidate_in_progress == 1)) {
+               kvm->mmu_invalidate_range_start = INVALID_GPA;
+               kvm->mmu_invalidate_range_end = INVALID_GPA;
+       }
+}
+
+void kvm_mmu_invalidate_range_add(struct kvm *kvm, gfn_t start, gfn_t end)
+{
+       WARN_ON_ONCE(!kvm->mmu_invalidate_in_progress);
+
+       if (likely(kvm->mmu_invalidate_in_progress == 1)) {
+               kvm->mmu_invalidate_range_start = start;
+               kvm->mmu_invalidate_range_end = end;
+       } else {
+               /*
+                * Fully tracking multiple concurrent ranges has diminishing
+                * returns. Keep things simple and just find the minimal range
+                * which includes the current and new ranges. As there won't be
+                * enough information to subtract a range after its invalidate
+                * completes, any ranges invalidated concurrently will
+                * accumulate and persist until all outstanding invalidates
+                * complete.
+                */
+               kvm->mmu_invalidate_range_start =
+                       min(kvm->mmu_invalidate_range_start, start);
+               kvm->mmu_invalidate_range_end =
+                       max(kvm->mmu_invalidate_range_end, end);
+       }
+}
+
+void kvm_mmu_invalidate_end(struct kvm *kvm)
+{
+       /*
+        * This sequence increase will notify the kvm page fault that
+        * the page that is going to be mapped in the spte could have
+        * been freed.
+        */
+       kvm->mmu_invalidate_seq++;
+       smp_wmb();
+       /*
+        * The above sequence increase must be visible before the
+        * below count decrease, which is ensured by the smp_wmb above
+        * in conjunction with the smp_rmb in mmu_invalidate_retry().
+        */
+       kvm->mmu_invalidate_in_progress--;
+}
+
 #if defined(CONFIG_MMU_NOTIFIER) && defined(KVM_ARCH_WANT_MMU_NOTIFIER)
 static inline struct kvm *mmu_notifier_to_kvm(struct mmu_notifier *mn)
 {
@@ -714,45 +770,6 @@ static void kvm_mmu_notifier_change_pte(struct mmu_notifier *mn,
        kvm_handle_hva_range(mn, address, address + 1, pte, kvm_set_spte_gfn);
 }
 
-void kvm_mmu_invalidate_begin(struct kvm *kvm)
-{
-       /*
-        * The count increase must become visible at unlock time as no
-        * spte can be established without taking the mmu_lock and
-        * count is also read inside the mmu_lock critical section.
-        */
-       kvm->mmu_invalidate_in_progress++;
-
-       if (likely(kvm->mmu_invalidate_in_progress == 1)) {
-               kvm->mmu_invalidate_range_start = INVALID_GPA;
-               kvm->mmu_invalidate_range_end = INVALID_GPA;
-       }
-}
-
-void kvm_mmu_invalidate_range_add(struct kvm *kvm, gfn_t start, gfn_t end)
-{
-       WARN_ON_ONCE(!kvm->mmu_invalidate_in_progress);
-
-       if (likely(kvm->mmu_invalidate_in_progress == 1)) {
-               kvm->mmu_invalidate_range_start = start;
-               kvm->mmu_invalidate_range_end = end;
-       } else {
-               /*
-                * Fully tracking multiple concurrent ranges has diminishing
-                * returns. Keep things simple and just find the minimal range
-                * which includes the current and new ranges. As there won't be
-                * enough information to subtract a range after its invalidate
-                * completes, any ranges invalidated concurrently will
-                * accumulate and persist until all outstanding invalidates
-                * complete.
-                */
-               kvm->mmu_invalidate_range_start =
-                       min(kvm->mmu_invalidate_range_start, start);
-               kvm->mmu_invalidate_range_end =
-                       max(kvm->mmu_invalidate_range_end, end);
-       }
-}
-
 static bool kvm_mmu_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
 {
        kvm_mmu_invalidate_range_add(kvm, range->start, range->end);
@@ -806,23 +823,6 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,
        return 0;
 }
 
-void kvm_mmu_invalidate_end(struct kvm *kvm)
-{
-       /*
-        * This sequence increase will notify the kvm page fault that
-        * the page that is going to be mapped in the spte could have
-        * been freed.
-        */
-       kvm->mmu_invalidate_seq++;
-       smp_wmb();
-       /*
-        * The above sequence increase must be visible before the
-        * below count decrease, which is ensured by the smp_wmb above
-        * in conjunction with the smp_rmb in mmu_invalidate_retry().
-        */
-       kvm->mmu_invalidate_in_progress--;
-}
-
 static void kvm_mmu_notifier_invalidate_range_end(struct mmu_notifier *mn,
                                        const struct mmu_notifier_range *range)
 {
@@ -1140,6 +1140,11 @@ int __weak kvm_arch_create_vm_debugfs(struct kvm *kvm)
        return 0;
 }
 
+bool __weak kvm_arch_has_private_mem(struct kvm *kvm)
+{
+       return false;
+}
+
 static struct kvm *kvm_create_vm(unsigned long type, const char *fdname)
 {
        struct kvm *kvm = kvm_arch_alloc_vm();
@@ -2349,15 +2354,47 @@ static u64 kvm_supported_mem_attributes(struct kvm *kvm)
        return 0;
 }
 
+static void kvm_unmap_mem_range(struct kvm *kvm, gfn_t start, gfn_t end)
+{
+       struct kvm_gfn_range gfn_range;
+       struct kvm_memory_slot *slot;
+       struct kvm_memslots *slots;
+       struct kvm_memslot_iter iter;
+       int i;
+       int r = 0;
+
+       gfn_range.pte = __pte(0);
+       gfn_range.may_block = true;
+
+       for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++) {
+               slots = __kvm_memslots(kvm, i);
+
+               kvm_for_each_memslot_in_gfn_range(&iter, slots, start, end) {
+                       slot = iter.slot;
+                       gfn_range.start = max(start, slot->base_gfn);
+                       gfn_range.end = min(end, slot->base_gfn + slot->npages);
+                       if (gfn_range.start >= gfn_range.end)
+                               continue;
+                       gfn_range.slot = slot;
+
+                       r |= kvm_unmap_gfn_range(kvm, &gfn_range);
+               }
+       }
+
+       if (r)
+               kvm_flush_remote_tlbs(kvm);
+}
+
 static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
                                           struct kvm_memory_attributes *attrs)
 {
        gfn_t start, end;
        unsigned long i;
        void *entry;
+       int idx;
        u64 supported_attrs = kvm_supported_mem_attributes(kvm);
 
-       /* flags is currently not used. */
+       /* 'flags' is currently not used. */
        if (attrs->flags)
                return -EINVAL;
        if (attrs->attributes & ~supported_attrs)
@@ -2372,6 +2409,13 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
 
        entry = attrs->attributes ? xa_mk_value(attrs->attributes) : NULL;
 
+       if (kvm_arch_has_private_mem(kvm)) {
+               KVM_MMU_LOCK(kvm);
+               kvm_mmu_invalidate_begin(kvm);
+               kvm_mmu_invalidate_range_add(kvm, start, end);
+               KVM_MMU_UNLOCK(kvm);
+       }
+
        mutex_lock(&kvm->lock);
        for (i = start; i < end; i++)
                if (xa_err(xa_store(&kvm->mem_attr_array, i, entry,
@@ -2379,6 +2423,16 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
                        break;
        mutex_unlock(&kvm->lock);
 
+       if (kvm_arch_has_private_mem(kvm)) {
+               idx = srcu_read_lock(&kvm->srcu);
+               KVM_MMU_LOCK(kvm);
+               if (i > start)
+                       kvm_unmap_mem_range(kvm, start, i);
+               kvm_mmu_invalidate_end(kvm);
+               KVM_MMU_UNLOCK(kvm);
+               srcu_read_unlock(&kvm->srcu, idx);
+       }
+
        attrs->address = i << PAGE_SHIFT;
        attrs->size = (end - i) << PAGE_SHIFT;
 
--

2.25.1

#276 lore raw

Re: [PATCH v10 6/9] KVM: Unmap existing mappings when change the memory attributes

In reply to Chao Peng
Quoted 5 lines · Chao Peng

virt/kvm/kvm_main.c +1 -1
-	/* flags is currently not used. */
+	/* 'flags' is currently not used. */

  if (attrs->flags)
  return -EINVAL;

Unintended code change.

author
#277 lore raw

Re: [PATCH v10 6/9] KVM: Unmap existing mappings when change the memory attributes

Quoted 8 lines · Kai
Quoted 5 lines · Chao Peng

virt/kvm/kvm_main.c +1 -1
-	/* flags is currently not used. */
+	/* 'flags' is currently not used. */

  if (attrs->flags)
  return -EINVAL;

Unintended code change.

Yeah!

Chao

#278 lore raw

Re: [PATCH v10 6/9] KVM: Unmap existing mappings when change the memory attributes

In reply to Chao Peng

include/linux/kvm_host.h struct kvm { · line 792

Quoted 10 lines · Chao Peng
include/linux/kvm_host.h +1 -1
@@ -785,11 +786,12 @@ struct kvm {
 
 #if defined(CONFIG_MMU_NOTIFIER) && defined(KVM_ARCH_WANT_MMU_NOTIFIER)
 	struct mmu_notifier mmu_notifier;
+#endif
 	unsigned long mmu_invalidate_seq;
 	long mmu_invalidate_in_progress;
 	gfn_t mmu_invalidate_range_start;
 	gfn_t mmu_invalidate_range_end;
-#endif

Blech. The existing code is a bit ugly, and trying to extend for this use case makes things even worse.

Rather than use the base MMU_NOTIFIER Kconfig and an arbitrary define, I think we should first add a proper Kconfig, e.g. KVM_GENERIC_MMU_NOTIFIER, to replace the combination. E.g

config KVM_GENERIC_MMU_NOTIFIER
select MMU_NOTIFIER
bool

and then all architectures that currently #define KVM_ARCH_WANT_MMU_NOTIFIER can simply select the Kconfig, which is everything except s390. "GENERIC" again because s390 does select MMU_NOTIFER and actually registers its own notifier for s390's version of protected VMs (at least, I think that's what its "pv" stands for).

And then later down the line in this series, when the attributes and private mem needs to tie into the notifiers, we can do:

config KVM_GENERIC_MEMORY_ATTRIBUTES
select KVM_GENERIC_MMU_NOTIFIER
bool

I.e. that way this patch doesn't need to partially expose KVM's notifier stuff and can instead just keep the soon-to-be-existing KVM_GENERIC_MMU_NOTIFIER.

Taking a depending on KVM_GENERIC_MMU_NOTIFIER for KVM_GENERIC_MEMORY_ATTRIBUTES makes sense, because AFAICT, changing any type of attribute, e.g. RWX bits, is going to necessitate unmapping the affected gfn range.

include/linux/kvm_host.h bool kvm_arch_dy_has_pending_interrupt(struct kvm_vcpu *vcpu); · line 1485

Quoted 8 lines
include/linux/kvm_host.h +1 -0
 	struct list_head devices;
 	u64 manual_dirty_log_protect;
 	struct dentry *debugfs_dentry;
@@ -1480,6 +1482,7 @@ bool kvm_arch_dy_has_pending_interrupt(struct kvm_vcpu *vcpu);
 int kvm_arch_post_init_vm(struct kvm *kvm);
 void kvm_arch_pre_destroy_vm(struct kvm *kvm);
 int kvm_arch_create_vm_debugfs(struct kvm *kvm);
+bool kvm_arch_has_private_mem(struct kvm *kvm);

The reference to private memory belongs in a later patch. More below.

virt/kvm/kvm_main.c static u64 kvm_supported_mem_attributes(struct kvm *kvm) · line 2364

Quoted 8 lines
virt/kvm/kvm_main.c +8 -0
+static void kvm_unmap_mem_range(struct kvm *kvm, gfn_t start, gfn_t end)
+{
+	struct kvm_gfn_range gfn_range;
+	struct kvm_memory_slot *slot;
+	struct kvm_memslots *slots;
+	struct kvm_memslot_iter iter;
+	int i;
+	int r = 0;

The return from kvm_unmap_gfn_range() is a bool, this should be:

	bool flush = false;
Quoted 24 lines
virt/kvm/kvm_main.c +23 -0
+
+	gfn_range.pte = __pte(0);
+	gfn_range.may_block = true;
+
+	for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++) {
+		slots = __kvm_memslots(kvm, i);
+
+		kvm_for_each_memslot_in_gfn_range(&iter, slots, start, end) {
+			slot = iter.slot;
+			gfn_range.start = max(start, slot->base_gfn);
+			gfn_range.end = min(end, slot->base_gfn + slot->npages);
+			if (gfn_range.start >= gfn_range.end)
+				continue;
+			gfn_range.slot = slot;
+
+			r |= kvm_unmap_gfn_range(kvm, &gfn_range);
+		}
+	}
+
+	if (r)
+		kvm_flush_remote_tlbs(kvm);
+}
+
 static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,

virt/kvm/kvm_main.c static u64 kvm_supported_mem_attributes(struct kvm *kvm) · line 2397

Quoted 10 lines
virt/kvm/kvm_main.c +2 -1
 					   struct kvm_memory_attributes *attrs)
 {
 	gfn_t start, end;
 	unsigned long i;
 	void *entry;
+	int idx;
 	u64 supported_attrs = kvm_supported_mem_attributes(kvm);
 
-	/* flags is currently not used. */
+	/* 'flags' is currently not used. */

Kind of a spurious change.

virt/kvm/kvm_main.c static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, · line 2412

Quoted 8 lines
virt/kvm/kvm_main.c +1 -0
 	if (attrs->flags)
 		return -EINVAL;
 	if (attrs->attributes & ~supported_attrs)
@@ -2372,6 +2409,13 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
 
 	entry = attrs->attributes ? xa_mk_value(attrs->attributes) : NULL;
 
+	if (kvm_arch_has_private_mem(kvm)) {

I think we should assume that any future attributes will necessitate unmapping and invalidation, i.e. drop the private mem check. That allows introducing kvm_arch_has_private_mem() in a later patch that is more directly related to private memory.

Quoted 10 lines
virt/kvm/kvm_main.c +6 -0
+		KVM_MMU_LOCK(kvm);
+		kvm_mmu_invalidate_begin(kvm);
+		kvm_mmu_invalidate_range_add(kvm, start, end);
+		KVM_MMU_UNLOCK(kvm);
+	}
+
 	mutex_lock(&kvm->lock);
 	for (i = start; i < end; i++)
 		if (xa_err(xa_store(&kvm->mem_attr_array, i, entry,

virt/kvm/kvm_main.c static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, · line 2427

Quoted 5 lines
virt/kvm/kvm_main.c +2 -0
@@ -2379,6 +2423,16 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
 			break;
 	mutex_unlock(&kvm->lock);
 
+	if (kvm_arch_has_private_mem(kvm)) {
+		idx = srcu_read_lock(&kvm->srcu);

Mostly for reference, this goes away if slots_lock is used instead of kvm->lock.

Quoted 14 lines
virt/kvm/kvm_main.c +8 -0
+		KVM_MMU_LOCK(kvm);
+		if (i > start)
+			kvm_unmap_mem_range(kvm, start, i);
+		kvm_mmu_invalidate_end(kvm);
+		KVM_MMU_UNLOCK(kvm);
+		srcu_read_unlock(&kvm->srcu, idx);
+	}
+
 	attrs->address = i << PAGE_SHIFT;
 	attrs->size = (end - i) << PAGE_SHIFT;
 
Signature · 1 line
2.25.1
author
#279 lore raw

PATCH v10 7/9 KVM: Update lpage info when private/shared memory are mixed

In reply to Chao Peng

A large page with mixed private/shared subpages can't be mapped as large page since its sub private/shared pages are from different memory backends and may also treated by architecture differently. When private/shared memory are mixed in a large page, the current lpage_info is not sufficient to decide whether the page can be mapped as large page or not and additional private/shared mixed information is needed.

Tracking this 'mixed' information with the current 'count' like disallow_lpage is a bit challenge so reserve a bit in 'disallow_lpage' to indicate a large page has mixed private/share subpages and update this 'mixed' bit whenever the memory attribute is changed between private and shared.

---
 arch/x86/include/asm/kvm_host.h |   8 ++
 arch/x86/kvm/mmu/mmu.c          | 134 +++++++++++++++++++++++++++++++-
 arch/x86/kvm/x86.c              |   2 +
 include/linux/kvm_host.h        |  19 +++++
 virt/kvm/kvm_main.c             |   9 ++-
 5 files changed, 169 insertions(+), 3 deletions(-)
arch/x86/include/asm/kvm_host.h +8 -0
index 283cbb83d6ae..7772ab37ac89 100644
@@ -38,6 +38,7 @@
 #include <asm/hyperv-tlfs.h>
 
 #define __KVM_HAVE_ARCH_VCPU_DEBUGFS
+#define __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES
 
 #define KVM_MAX_VCPUS 1024
 
@@ -1011,6 +1012,13 @@ struct kvm_vcpu_arch {
 #endif
 };
 
+/*
+ * Use a bit in disallow_lpage to indicate private/shared pages mixed at the
+ * level. The remaining bits are used as a reference count.
+ */
+#define KVM_LPAGE_PRIVATE_SHARED_MIXED		(1U << 31)
+#define KVM_LPAGE_COUNT_MAX			((1U << 31) - 1)
+
 struct kvm_lpage_info {
 	int disallow_lpage;
 };
arch/x86/kvm/mmu/mmu.c +133 -1

147 lines not on this page · raw message

arch/x86/kvm/x86.c +2 -0
index 9a07380f8d3c..5aefcff614d2 100644
@@ -12362,6 +12362,8 @@ static int kvm_alloc_memslot_metadata(struct kvm *kvm,
 		if ((slot->base_gfn + npages) & (KVM_PAGES_PER_HPAGE(level) - 1))
 			linfo[lpages - 1].disallow_lpage = 1;
 		ugfn = slot->userspace_addr >> PAGE_SHIFT;
+		if (kvm_slot_can_be_private(slot))
+			ugfn |= slot->restricted_offset >> PAGE_SHIFT;
 		/*
 		 * If the gfn and userspace address are not aligned wrt each
 		 * other, disable large page support for this slot.
include/linux/kvm_host.h +19 -0
index 3331c0c92838..25099c94e770 100644
@@ -592,6 +592,11 @@ struct kvm_memory_slot {
 	struct restrictedmem_notifier notifier;
 };
 
+static inline bool kvm_slot_can_be_private(const struct kvm_memory_slot *slot)
+{
+	return slot && (slot->flags & KVM_MEM_PRIVATE);
+}
+
 static inline bool kvm_slot_dirty_track_enabled(const struct kvm_memory_slot *slot)
 {
 	return slot->flags & KVM_MEM_LOG_DIRTY_PAGES;
@@ -2316,4 +2321,18 @@ static inline void kvm_account_pgtable_pages(void *virt, int nr)
 /* Max number of entries allowed for each kvm dirty ring */
 #define  KVM_DIRTY_RING_MAX_ENTRIES  65536
 
+#ifdef __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES
+void kvm_arch_set_memory_attributes(struct kvm *kvm,
+				    struct kvm_memory_slot *slot,
+				    unsigned long attrs,
+				    gfn_t start, gfn_t end);
+#else
+static inline void kvm_arch_set_memory_attributes(struct kvm *kvm,
+						  struct kvm_memory_slot *slot,
+						  unsigned long attrs,
+						  gfn_t start, gfn_t end)
+{
+}
+#endif /* __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES */
+
 #endif
virt/kvm/kvm_main.c +7 -2
index 4e1e1e113bf0..e107afea32f0 100644
@@ -2354,7 +2354,8 @@ static u64 kvm_supported_mem_attributes(struct kvm *kvm)
 	return 0;
 }
 
-static void kvm_unmap_mem_range(struct kvm *kvm, gfn_t start, gfn_t end)
+static void kvm_unmap_mem_range(struct kvm *kvm, gfn_t start, gfn_t end,
+				unsigned long attrs)
 {
 	struct kvm_gfn_range gfn_range;
 	struct kvm_memory_slot *slot;
@@ -2378,6 +2379,10 @@ static void kvm_unmap_mem_range(struct kvm *kvm, gfn_t start, gfn_t end)
 			gfn_range.slot = slot;
 
 			r |= kvm_unmap_gfn_range(kvm, &gfn_range);
+
+			kvm_arch_set_memory_attributes(kvm, slot, attrs,
+						       gfn_range.start,
+						       gfn_range.end);
 		}
 	}
 
@@ -2427,7 +2432,7 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
 		idx = srcu_read_lock(&kvm->srcu);
 		KVM_MMU_LOCK(kvm);
 		if (i > start)
-			kvm_unmap_mem_range(kvm, start, i);
+			kvm_unmap_mem_range(kvm, start, i, attrs->attributes);
 		kvm_mmu_invalidate_end(kvm);
 		KVM_MMU_UNLOCK(kvm);
 		srcu_read_unlock(&kvm->srcu, idx);
Signature · 1 line
2.25.1
#280 lore raw

Re: [PATCH v10 7/9] KVM: Update lpage info when private/shared memory are mixed

Quoted 171 lines · Chao Peng

A large page with mixed private/shared subpages can't be mapped as large page since its sub private/shared pages are from different memory backends and may also treated by architecture differently. When private/shared memory are mixed in a large page, the current lpage_info is not sufficient to decide whether the page can be mapped as large page or not and additional private/shared mixed information is needed.

Tracking this 'mixed' information with the current 'count' like disallow_lpage is a bit challenge so reserve a bit in 'disallow_lpage' to indicate a large page has mixed private/share subpages and update this 'mixed' bit whenever the memory attribute is changed between private and shared.

---
 arch/x86/include/asm/kvm_host.h |   8 ++
 arch/x86/kvm/mmu/mmu.c          | 134 +++++++++++++++++++++++++++++++-
 arch/x86/kvm/x86.c              |   2 +
 include/linux/kvm_host.h        |  19 +++++
 virt/kvm/kvm_main.c             |   9 ++-
 5 files changed, 169 insertions(+), 3 deletions(-)
arch/x86/include/asm/kvm_host.h +8 -0
index 283cbb83d6ae..7772ab37ac89 100644
@@ -38,6 +38,7 @@
 #include <asm/hyperv-tlfs.h>
 
 #define __KVM_HAVE_ARCH_VCPU_DEBUGFS
+#define __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES
 
 #define KVM_MAX_VCPUS 1024
 
@@ -1011,6 +1012,13 @@ struct kvm_vcpu_arch {
 #endif
 };
 
+/*
+ * Use a bit in disallow_lpage to indicate private/shared pages mixed at the
+ * level. The remaining bits are used as a reference count.
+ */
+#define KVM_LPAGE_PRIVATE_SHARED_MIXED		(1U << 31)
+#define KVM_LPAGE_COUNT_MAX			((1U << 31) - 1)
+
 struct kvm_lpage_info {
 	int disallow_lpage;
 };
arch/x86/kvm/mmu/mmu.c +103 -1
index e2c70b5afa3e..2190fd8c95c0 100644
@@ -763,11 +763,16 @@ static void update_gfn_disallow_lpage_count(const struct kvm_memory_slot *slot,
 {
 	struct kvm_lpage_info *linfo;
 	int i;
+	int disallow_count;
 
 	for (i = PG_LEVEL_2M; i <= KVM_MAX_HUGEPAGE_LEVEL; ++i) {
 		linfo = lpage_info_slot(gfn, slot, i);
+
+		disallow_count = linfo->disallow_lpage & KVM_LPAGE_COUNT_MAX;
+		WARN_ON(disallow_count + count < 0 ||
+			disallow_count > KVM_LPAGE_COUNT_MAX - count);
+
 		linfo->disallow_lpage += count;
-		WARN_ON(linfo->disallow_lpage < 0);
 	}
 }
 
@@ -6986,3 +6991,130 @@ void kvm_mmu_pre_destroy_vm(struct kvm *kvm)
 	if (kvm->arch.nx_huge_page_recovery_thread)
 		kthread_stop(kvm->arch.nx_huge_page_recovery_thread);
 }
+
+static bool linfo_is_mixed(struct kvm_lpage_info *linfo)
+{
+	return linfo->disallow_lpage & KVM_LPAGE_PRIVATE_SHARED_MIXED;
+}
+
+static void linfo_set_mixed(gfn_t gfn, struct kvm_memory_slot *slot,
+			    int level, bool mixed)
+{
+	struct kvm_lpage_info *linfo = lpage_info_slot(gfn, slot, level);
+
+	if (mixed)
+		linfo->disallow_lpage |= KVM_LPAGE_PRIVATE_SHARED_MIXED;
+	else
+		linfo->disallow_lpage &= ~KVM_LPAGE_PRIVATE_SHARED_MIXED;
+}
+
+static bool is_expected_attr_entry(void *entry, unsigned long expected_attrs)
+{
+	bool expect_private = expected_attrs & KVM_MEMORY_ATTRIBUTE_PRIVATE;
+
+	if (xa_to_value(entry) & KVM_MEMORY_ATTRIBUTE_PRIVATE) {
+		if (!expect_private)
+			return false;
+	} else if (expect_private)
+		return false;
+
+	return true;
+}
+
+static bool mem_attrs_mixed_2m(struct kvm *kvm, unsigned long attrs,
+			       gfn_t start, gfn_t end)
+{
+	XA_STATE(xas, &kvm->mem_attr_array, start);
+	gfn_t gfn = start;
+	void *entry;
+	bool mixed = false;
+
+	rcu_read_lock();
+	entry = xas_load(&xas);
+	while (gfn < end) {
+		if (xas_retry(&xas, entry))
+			continue;
+
+		KVM_BUG_ON(gfn != xas.xa_index, kvm);
+
+		if (!is_expected_attr_entry(entry, attrs)) {
+			mixed = true;
+			break;
+		}
+
+		entry = xas_next(&xas);
+		gfn++;
+	}
+
+	rcu_read_unlock();
+	return mixed;
+}
+
+static bool mem_attrs_mixed(struct kvm *kvm, struct kvm_memory_slot *slot,
+			    int level, unsigned long attrs,
+			    gfn_t start, gfn_t end)
+{
+	unsigned long gfn;
+
+	if (level == PG_LEVEL_2M)
+		return mem_attrs_mixed_2m(kvm, attrs, start, end);
+
+	for (gfn = start; gfn < end; gfn += KVM_PAGES_PER_HPAGE(level - 1))
+		if (linfo_is_mixed(lpage_info_slot(gfn, slot, level - 1)) ||
+		    !is_expected_attr_entry(xa_load(&kvm->mem_attr_array, gfn),
+					    attrs))
+			return true;
+	return false;
+}
+
+static void kvm_update_lpage_private_shared_mixed(struct kvm *kvm,
+						  struct kvm_memory_slot *slot,
+						  unsigned long attrs,
+						  gfn_t start, gfn_t end)
+{
+	unsigned long pages, mask;
+	gfn_t gfn, gfn_end, first, last;
+	int level;
+	bool mixed;
+
+	/*
+	 * The sequence matters here: we set the higher level basing on the
+	 * lower level's scanning result.
+	 */
+	for (level = PG_LEVEL_2M; level <= KVM_MAX_HUGEPAGE_LEVEL; level++) {
+		pages = KVM_PAGES_PER_HPAGE(level);
+		mask = ~(pages - 1);
+		first = start & mask;
+		last = (end - 1) & mask;
+
+		/*

arch/x86/kvm/mmu/mmu.c void kvm_mmu_pre_destroy_vm(struct kvm *kvm) · line 7100

Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +10 -0
+		 * We only need to scan the head and tail page, for middle pages
+		 * we know they will not be mixed.
+		 */
+		gfn = max(first, slot->base_gfn);
+		gfn_end = min(first + pages, slot->base_gfn + slot->npages);
+		mixed = mem_attrs_mixed(kvm, slot, level, attrs, gfn, gfn_end);
+		linfo_set_mixed(gfn, slot, level, mixed);
+
+		if (first == last)
+			return;

continue.

arch/x86/kvm/mmu/mmu.c void kvm_mmu_pre_destroy_vm(struct kvm *kvm) · line 7106

Quoted 6 lines
arch/x86/kvm/mmu/mmu.c +6 -0
+
+		for (gfn = first + pages; gfn < last; gfn += pages)
+			linfo_set_mixed(gfn, slot, level, false);
+
+		gfn = last;
+		gfn_end = min(last + pages, slot->base_gfn + slot->npages);

if (gfn == gfn_end) continue.

Quoted 19 lines
arch/x86/kvm/mmu/mmu.c +14 -0
+		mixed = mem_attrs_mixed(kvm, slot, level, attrs, gfn, gfn_end);
+		linfo_set_mixed(gfn, slot, level, mixed);
+	}
+}
+
+void kvm_arch_set_memory_attributes(struct kvm *kvm,
+				    struct kvm_memory_slot *slot,
+				    unsigned long attrs,
+				    gfn_t start, gfn_t end)
+{
+	if (kvm_slot_can_be_private(slot))
+		kvm_update_lpage_private_shared_mixed(kvm, slot, attrs,
+						      start, end);
+}
arch/x86/kvm/x86.c +0 -0
index 9a07380f8d3c..5aefcff614d2 100644

arch/x86/kvm/x86.c static int kvm_alloc_memslot_metadata(struct kvm *kvm, · line 12366

Quoted 5 lines
arch/x86/kvm/x86.c +2 -0
@@ -12362,6 +12362,8 @@ static int kvm_alloc_memslot_metadata(struct kvm *kvm,
 		if ((slot->base_gfn + npages) & (KVM_PAGES_PER_HPAGE(level) - 1))
 			linfo[lpages - 1].disallow_lpage = 1;
 		ugfn = slot->userspace_addr >> PAGE_SHIFT;
+		if (kvm_slot_can_be_private(slot))
+			ugfn |= slot->restricted_offset >> PAGE_SHIFT;

Is there any alignment restriction? If no, It should be +=. In practice, alignment will hold though.

Thanks,

Quoted 75 lines
arch/x86/kvm/x86.c +0 -0
 		/*
 		 * If the gfn and userspace address are not aligned wrt each
 		 * other, disable large page support for this slot.
include/linux/kvm_host.h +19 -0
index 3331c0c92838..25099c94e770 100644
@@ -592,6 +592,11 @@ struct kvm_memory_slot {
 	struct restrictedmem_notifier notifier;
 };
 
+static inline bool kvm_slot_can_be_private(const struct kvm_memory_slot *slot)
+{
+	return slot && (slot->flags & KVM_MEM_PRIVATE);
+}
+
 static inline bool kvm_slot_dirty_track_enabled(const struct kvm_memory_slot *slot)
 {
 	return slot->flags & KVM_MEM_LOG_DIRTY_PAGES;
@@ -2316,4 +2321,18 @@ static inline void kvm_account_pgtable_pages(void *virt, int nr)
 /* Max number of entries allowed for each kvm dirty ring */
 #define  KVM_DIRTY_RING_MAX_ENTRIES  65536
 
+#ifdef __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES
+void kvm_arch_set_memory_attributes(struct kvm *kvm,
+				    struct kvm_memory_slot *slot,
+				    unsigned long attrs,
+				    gfn_t start, gfn_t end);
+#else
+static inline void kvm_arch_set_memory_attributes(struct kvm *kvm,
+						  struct kvm_memory_slot *slot,
+						  unsigned long attrs,
+						  gfn_t start, gfn_t end)
+{
+}
+#endif /* __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES */
+
 #endif
virt/kvm/kvm_main.c +7 -2
index 4e1e1e113bf0..e107afea32f0 100644
@@ -2354,7 +2354,8 @@ static u64 kvm_supported_mem_attributes(struct kvm *kvm)
 	return 0;
 }
 
-static void kvm_unmap_mem_range(struct kvm *kvm, gfn_t start, gfn_t end)
+static void kvm_unmap_mem_range(struct kvm *kvm, gfn_t start, gfn_t end,
+				unsigned long attrs)
 {
 	struct kvm_gfn_range gfn_range;
 	struct kvm_memory_slot *slot;
@@ -2378,6 +2379,10 @@ static void kvm_unmap_mem_range(struct kvm *kvm, gfn_t start, gfn_t end)
 			gfn_range.slot = slot;
 
 			r |= kvm_unmap_gfn_range(kvm, &gfn_range);
+
+			kvm_arch_set_memory_attributes(kvm, slot, attrs,
+						       gfn_range.start,
+						       gfn_range.end);
 		}
 	}
 
@@ -2427,7 +2432,7 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
 		idx = srcu_read_lock(&kvm->srcu);
 		KVM_MMU_LOCK(kvm);
 		if (i > start)
-			kvm_unmap_mem_range(kvm, start, i);
+			kvm_unmap_mem_range(kvm, start, i, attrs->attributes);
 		kvm_mmu_invalidate_end(kvm);
 		KVM_MMU_UNLOCK(kvm);
 		srcu_read_unlock(&kvm->srcu, idx);
Signature · 1 line
2.25.1
Signature · 1 line
Isaku Yamahata <[email protected]>
author
#281 lore raw

Re: [PATCH v10 7/9] KVM: Update lpage info when private/shared memory are mixed

Quoted 179 lines · Isaku Yamahata
Quoted 176 lines · Chao Peng

A large page with mixed private/shared subpages can't be mapped as large page since its sub private/shared pages are from different memory backends and may also treated by architecture differently. When private/shared memory are mixed in a large page, the current lpage_info is not sufficient to decide whether the page can be mapped as large page or not and additional private/shared mixed information is needed.

Tracking this 'mixed' information with the current 'count' like disallow_lpage is a bit challenge so reserve a bit in 'disallow_lpage' to indicate a large page has mixed private/share subpages and update this 'mixed' bit whenever the memory attribute is changed between private and shared.

---
 arch/x86/include/asm/kvm_host.h |   8 ++
 arch/x86/kvm/mmu/mmu.c          | 134 +++++++++++++++++++++++++++++++-
 arch/x86/kvm/x86.c              |   2 +
 include/linux/kvm_host.h        |  19 +++++
 virt/kvm/kvm_main.c             |   9 ++-
 5 files changed, 169 insertions(+), 3 deletions(-)
arch/x86/include/asm/kvm_host.h +8 -0
index 283cbb83d6ae..7772ab37ac89 100644
@@ -38,6 +38,7 @@
 #include <asm/hyperv-tlfs.h>
 
 #define __KVM_HAVE_ARCH_VCPU_DEBUGFS
+#define __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES
 
 #define KVM_MAX_VCPUS 1024
 
@@ -1011,6 +1012,13 @@ struct kvm_vcpu_arch {
 #endif
 };
 
+/*
+ * Use a bit in disallow_lpage to indicate private/shared pages mixed at the
+ * level. The remaining bits are used as a reference count.
+ */
+#define KVM_LPAGE_PRIVATE_SHARED_MIXED		(1U << 31)
+#define KVM_LPAGE_COUNT_MAX			((1U << 31) - 1)
+
 struct kvm_lpage_info {
 	int disallow_lpage;
 };
arch/x86/kvm/mmu/mmu.c +108 -1
index e2c70b5afa3e..2190fd8c95c0 100644
@@ -763,11 +763,16 @@ static void update_gfn_disallow_lpage_count(const struct kvm_memory_slot *slot,
 {
 	struct kvm_lpage_info *linfo;
 	int i;
+	int disallow_count;
 
 	for (i = PG_LEVEL_2M; i <= KVM_MAX_HUGEPAGE_LEVEL; ++i) {
 		linfo = lpage_info_slot(gfn, slot, i);
+
+		disallow_count = linfo->disallow_lpage & KVM_LPAGE_COUNT_MAX;
+		WARN_ON(disallow_count + count < 0 ||
+			disallow_count > KVM_LPAGE_COUNT_MAX - count);
+
 		linfo->disallow_lpage += count;
-		WARN_ON(linfo->disallow_lpage < 0);
 	}
 }
 
@@ -6986,3 +6991,130 @@ void kvm_mmu_pre_destroy_vm(struct kvm *kvm)
 	if (kvm->arch.nx_huge_page_recovery_thread)
 		kthread_stop(kvm->arch.nx_huge_page_recovery_thread);
 }
+
+static bool linfo_is_mixed(struct kvm_lpage_info *linfo)
+{
+	return linfo->disallow_lpage & KVM_LPAGE_PRIVATE_SHARED_MIXED;
+}
+
+static void linfo_set_mixed(gfn_t gfn, struct kvm_memory_slot *slot,
+			    int level, bool mixed)
+{
+	struct kvm_lpage_info *linfo = lpage_info_slot(gfn, slot, level);
+
+	if (mixed)
+		linfo->disallow_lpage |= KVM_LPAGE_PRIVATE_SHARED_MIXED;
+	else
+		linfo->disallow_lpage &= ~KVM_LPAGE_PRIVATE_SHARED_MIXED;
+}
+
+static bool is_expected_attr_entry(void *entry, unsigned long expected_attrs)
+{
+	bool expect_private = expected_attrs & KVM_MEMORY_ATTRIBUTE_PRIVATE;
+
+	if (xa_to_value(entry) & KVM_MEMORY_ATTRIBUTE_PRIVATE) {
+		if (!expect_private)
+			return false;
+	} else if (expect_private)
+		return false;
+
+	return true;
+}
+
+static bool mem_attrs_mixed_2m(struct kvm *kvm, unsigned long attrs,
+			       gfn_t start, gfn_t end)
+{
+	XA_STATE(xas, &kvm->mem_attr_array, start);
+	gfn_t gfn = start;
+	void *entry;
+	bool mixed = false;
+
+	rcu_read_lock();
+	entry = xas_load(&xas);
+	while (gfn < end) {
+		if (xas_retry(&xas, entry))
+			continue;
+
+		KVM_BUG_ON(gfn != xas.xa_index, kvm);
+
+		if (!is_expected_attr_entry(entry, attrs)) {
+			mixed = true;
+			break;
+		}
+
+		entry = xas_next(&xas);
+		gfn++;
+	}
+
+	rcu_read_unlock();
+	return mixed;
+}
+
+static bool mem_attrs_mixed(struct kvm *kvm, struct kvm_memory_slot *slot,
+			    int level, unsigned long attrs,
+			    gfn_t start, gfn_t end)
+{
+	unsigned long gfn;
+
+	if (level == PG_LEVEL_2M)
+		return mem_attrs_mixed_2m(kvm, attrs, start, end);
+
+	for (gfn = start; gfn < end; gfn += KVM_PAGES_PER_HPAGE(level - 1))
+		if (linfo_is_mixed(lpage_info_slot(gfn, slot, level - 1)) ||
+		    !is_expected_attr_entry(xa_load(&kvm->mem_attr_array, gfn),
+					    attrs))
+			return true;
+	return false;
+}
+
+static void kvm_update_lpage_private_shared_mixed(struct kvm *kvm,
+						  struct kvm_memory_slot *slot,
+						  unsigned long attrs,
+						  gfn_t start, gfn_t end)
+{
+	unsigned long pages, mask;
+	gfn_t gfn, gfn_end, first, last;
+	int level;
+	bool mixed;
+
+	/*
+	 * The sequence matters here: we set the higher level basing on the
+	 * lower level's scanning result.
+	 */
+	for (level = PG_LEVEL_2M; level <= KVM_MAX_HUGEPAGE_LEVEL; level++) {
+		pages = KVM_PAGES_PER_HPAGE(level);
+		mask = ~(pages - 1);
+		first = start & mask;
+		last = (end - 1) & mask;
+
+		/*
+		 * We only need to scan the head and tail page, for middle pages
+		 * we know they will not be mixed.
+		 */
+		gfn = max(first, slot->base_gfn);
+		gfn_end = min(first + pages, slot->base_gfn + slot->npages);
Quoted 8 lines

arch/x86/kvm/mmu/mmu.c void kvm_mmu_pre_destroy_vm(struct kvm *kvm) · line 7100

Quoted 5 lines
arch/x86/kvm/mmu/mmu.c +5 -0
+		mixed = mem_attrs_mixed(kvm, slot, level, attrs, gfn, gfn_end);
+		linfo_set_mixed(gfn, slot, level, mixed);
+
+		if (first == last)
+			return;

continue.

Ya!

Quoted 9 lines

arch/x86/kvm/mmu/mmu.c void kvm_mmu_pre_destroy_vm(struct kvm *kvm) · line 7106

Quoted 7 lines
arch/x86/kvm/mmu/mmu.c +6 -0
+
+		for (gfn = first + pages; gfn < last; gfn += pages)
+			linfo_set_mixed(gfn, slot, level, false);
+
+		gfn = last;
+		gfn_end = min(last + pages, slot->base_gfn + slot->npages);

if (gfn == gfn_end) continue.

Do you see a case where gfn can equal to gfn_end? Though it does not hurt to add a check.

Quoted 21 lines
Quoted 21 lines
arch/x86/kvm/mmu/mmu.c +14 -0
+		mixed = mem_attrs_mixed(kvm, slot, level, attrs, gfn, gfn_end);
+		linfo_set_mixed(gfn, slot, level, mixed);
+	}
+}
+
+void kvm_arch_set_memory_attributes(struct kvm *kvm,
+				    struct kvm_memory_slot *slot,
+				    unsigned long attrs,
+				    gfn_t start, gfn_t end)
+{
+	if (kvm_slot_can_be_private(slot))
+		kvm_update_lpage_private_shared_mixed(kvm, slot, attrs,
+						      start, end);
+}
arch/x86/kvm/x86.c +0 -0
index 9a07380f8d3c..5aefcff614d2 100644
@@ -12362,6 +12362,8 @@ static int kvm_alloc_memslot_metadata(struct kvm *kvm,
Quoted 8 lines

arch/x86/kvm/x86.c static int kvm_alloc_memslot_metadata(struct kvm *kvm, · line 12366

Quoted 5 lines
arch/x86/kvm/x86.c +2 -0
 		if ((slot->base_gfn + npages) & (KVM_PAGES_PER_HPAGE(level) - 1))
 			linfo[lpages - 1].disallow_lpage = 1;
 		ugfn = slot->userspace_addr >> PAGE_SHIFT;
+		if (kvm_slot_can_be_private(slot))
+			ugfn |= slot->restricted_offset >> PAGE_SHIFT;

Is there any alignment restriction? If no, It should be +=. In practice, alignment will hold though.

All we need here is checking whether both userspace_addr and restricted_offset are aligned to HPAGE_SIZE or not. '+=' actually can yield wrong value in cases when userspace_addr + restricted_offset is aligned to HPAGE_SIZE but individually they may not align to HPAGE_SIZE.

Thanks,
Chao

Quoted 81 lines

Thanks,

Quoted 75 lines
arch/x86/kvm/x86.c +0 -0
 		/*
 		 * If the gfn and userspace address are not aligned wrt each
 		 * other, disable large page support for this slot.
include/linux/kvm_host.h +19 -0
index 3331c0c92838..25099c94e770 100644
@@ -592,6 +592,11 @@ struct kvm_memory_slot {
 	struct restrictedmem_notifier notifier;
 };
 
+static inline bool kvm_slot_can_be_private(const struct kvm_memory_slot *slot)
+{
+	return slot && (slot->flags & KVM_MEM_PRIVATE);
+}
+
 static inline bool kvm_slot_dirty_track_enabled(const struct kvm_memory_slot *slot)
 {
 	return slot->flags & KVM_MEM_LOG_DIRTY_PAGES;
@@ -2316,4 +2321,18 @@ static inline void kvm_account_pgtable_pages(void *virt, int nr)
 /* Max number of entries allowed for each kvm dirty ring */
 #define  KVM_DIRTY_RING_MAX_ENTRIES  65536
 
+#ifdef __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES
+void kvm_arch_set_memory_attributes(struct kvm *kvm,
+				    struct kvm_memory_slot *slot,
+				    unsigned long attrs,
+				    gfn_t start, gfn_t end);
+#else
+static inline void kvm_arch_set_memory_attributes(struct kvm *kvm,
+						  struct kvm_memory_slot *slot,
+						  unsigned long attrs,
+						  gfn_t start, gfn_t end)
+{
+}
+#endif /* __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES */
+
 #endif
virt/kvm/kvm_main.c +7 -2
index 4e1e1e113bf0..e107afea32f0 100644
@@ -2354,7 +2354,8 @@ static u64 kvm_supported_mem_attributes(struct kvm *kvm)
 	return 0;
 }
 
-static void kvm_unmap_mem_range(struct kvm *kvm, gfn_t start, gfn_t end)
+static void kvm_unmap_mem_range(struct kvm *kvm, gfn_t start, gfn_t end,
+				unsigned long attrs)
 {
 	struct kvm_gfn_range gfn_range;
 	struct kvm_memory_slot *slot;
@@ -2378,6 +2379,10 @@ static void kvm_unmap_mem_range(struct kvm *kvm, gfn_t start, gfn_t end)
 			gfn_range.slot = slot;
 
 			r |= kvm_unmap_gfn_range(kvm, &gfn_range);
+
+			kvm_arch_set_memory_attributes(kvm, slot, attrs,
+						       gfn_range.start,
+						       gfn_range.end);
 		}
 	}
 
@@ -2427,7 +2432,7 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
 		idx = srcu_read_lock(&kvm->srcu);
 		KVM_MMU_LOCK(kvm);
 		if (i > start)
-			kvm_unmap_mem_range(kvm, start, i);
+			kvm_unmap_mem_range(kvm, start, i, attrs->attributes);
 		kvm_mmu_invalidate_end(kvm);
 		KVM_MMU_UNLOCK(kvm);
 		srcu_read_unlock(&kvm->srcu, idx);
Signature · 1 line
2.25.1
Signature · 1 line
Isaku Yamahata <[email protected]>
#282 lore raw

Re: [PATCH v10 7/9] KVM: Update lpage info when private/shared memory are mixed

Quoted 201 lines · Chao Peng
Quoted 179 lines · Isaku Yamahata
Quoted 176 lines · Chao Peng

A large page with mixed private/shared subpages can't be mapped as large page since its sub private/shared pages are from different memory backends and may also treated by architecture differently. When private/shared memory are mixed in a large page, the current lpage_info is not sufficient to decide whether the page can be mapped as large page or not and additional private/shared mixed information is needed.

Tracking this 'mixed' information with the current 'count' like disallow_lpage is a bit challenge so reserve a bit in 'disallow_lpage' to indicate a large page has mixed private/share subpages and update this 'mixed' bit whenever the memory attribute is changed between private and shared.

---
 arch/x86/include/asm/kvm_host.h |   8 ++
 arch/x86/kvm/mmu/mmu.c          | 134 +++++++++++++++++++++++++++++++-
 arch/x86/kvm/x86.c              |   2 +
 include/linux/kvm_host.h        |  19 +++++
 virt/kvm/kvm_main.c             |   9 ++-
 5 files changed, 169 insertions(+), 3 deletions(-)
arch/x86/include/asm/kvm_host.h +8 -0
index 283cbb83d6ae..7772ab37ac89 100644
@@ -38,6 +38,7 @@
 #include <asm/hyperv-tlfs.h>
 
 #define __KVM_HAVE_ARCH_VCPU_DEBUGFS
+#define __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES
 
 #define KVM_MAX_VCPUS 1024
 
@@ -1011,6 +1012,13 @@ struct kvm_vcpu_arch {
 #endif
 };
 
+/*
+ * Use a bit in disallow_lpage to indicate private/shared pages mixed at the
+ * level. The remaining bits are used as a reference count.
+ */
+#define KVM_LPAGE_PRIVATE_SHARED_MIXED		(1U << 31)
+#define KVM_LPAGE_COUNT_MAX			((1U << 31) - 1)
+
 struct kvm_lpage_info {
 	int disallow_lpage;
 };
arch/x86/kvm/mmu/mmu.c +108 -1
index e2c70b5afa3e..2190fd8c95c0 100644
@@ -763,11 +763,16 @@ static void update_gfn_disallow_lpage_count(const struct kvm_memory_slot *slot,
 {
 	struct kvm_lpage_info *linfo;
 	int i;
+	int disallow_count;
 
 	for (i = PG_LEVEL_2M; i <= KVM_MAX_HUGEPAGE_LEVEL; ++i) {
 		linfo = lpage_info_slot(gfn, slot, i);
+
+		disallow_count = linfo->disallow_lpage & KVM_LPAGE_COUNT_MAX;
+		WARN_ON(disallow_count + count < 0 ||
+			disallow_count > KVM_LPAGE_COUNT_MAX - count);
+
 		linfo->disallow_lpage += count;
-		WARN_ON(linfo->disallow_lpage < 0);
 	}
 }
 
@@ -6986,3 +6991,130 @@ void kvm_mmu_pre_destroy_vm(struct kvm *kvm)
 	if (kvm->arch.nx_huge_page_recovery_thread)
 		kthread_stop(kvm->arch.nx_huge_page_recovery_thread);
 }
+
+static bool linfo_is_mixed(struct kvm_lpage_info *linfo)
+{
+	return linfo->disallow_lpage & KVM_LPAGE_PRIVATE_SHARED_MIXED;
+}
+
+static void linfo_set_mixed(gfn_t gfn, struct kvm_memory_slot *slot,
+			    int level, bool mixed)
+{
+	struct kvm_lpage_info *linfo = lpage_info_slot(gfn, slot, level);
+
+	if (mixed)
+		linfo->disallow_lpage |= KVM_LPAGE_PRIVATE_SHARED_MIXED;
+	else
+		linfo->disallow_lpage &= ~KVM_LPAGE_PRIVATE_SHARED_MIXED;
+}
+
+static bool is_expected_attr_entry(void *entry, unsigned long expected_attrs)
+{
+	bool expect_private = expected_attrs & KVM_MEMORY_ATTRIBUTE_PRIVATE;
+
+	if (xa_to_value(entry) & KVM_MEMORY_ATTRIBUTE_PRIVATE) {
+		if (!expect_private)
+			return false;
+	} else if (expect_private)
+		return false;
+
+	return true;
+}
+
+static bool mem_attrs_mixed_2m(struct kvm *kvm, unsigned long attrs,
+			       gfn_t start, gfn_t end)
+{
+	XA_STATE(xas, &kvm->mem_attr_array, start);
+	gfn_t gfn = start;
+	void *entry;
+	bool mixed = false;
+
+	rcu_read_lock();
+	entry = xas_load(&xas);
+	while (gfn < end) {
+		if (xas_retry(&xas, entry))
+			continue;
+
+		KVM_BUG_ON(gfn != xas.xa_index, kvm);
+
+		if (!is_expected_attr_entry(entry, attrs)) {
+			mixed = true;
+			break;
+		}
+
+		entry = xas_next(&xas);
+		gfn++;
+	}
+
+	rcu_read_unlock();
+	return mixed;
+}
+
+static bool mem_attrs_mixed(struct kvm *kvm, struct kvm_memory_slot *slot,
+			    int level, unsigned long attrs,
+			    gfn_t start, gfn_t end)
+{
+	unsigned long gfn;
+
+	if (level == PG_LEVEL_2M)
+		return mem_attrs_mixed_2m(kvm, attrs, start, end);
+
+	for (gfn = start; gfn < end; gfn += KVM_PAGES_PER_HPAGE(level - 1))
+		if (linfo_is_mixed(lpage_info_slot(gfn, slot, level - 1)) ||
+		    !is_expected_attr_entry(xa_load(&kvm->mem_attr_array, gfn),
+					    attrs))
+			return true;
+	return false;
+}
+
+static void kvm_update_lpage_private_shared_mixed(struct kvm *kvm,
+						  struct kvm_memory_slot *slot,
+						  unsigned long attrs,
+						  gfn_t start, gfn_t end)
+{
+	unsigned long pages, mask;
+	gfn_t gfn, gfn_end, first, last;
+	int level;
+	bool mixed;
+
+	/*
+	 * The sequence matters here: we set the higher level basing on the
+	 * lower level's scanning result.
+	 */
+	for (level = PG_LEVEL_2M; level <= KVM_MAX_HUGEPAGE_LEVEL; level++) {
+		pages = KVM_PAGES_PER_HPAGE(level);
+		mask = ~(pages - 1);
+		first = start & mask;
+		last = (end - 1) & mask;
+
+		/*
+		 * We only need to scan the head and tail page, for middle pages
+		 * we know they will not be mixed.
+		 */
+		gfn = max(first, slot->base_gfn);
+		gfn_end = min(first + pages, slot->base_gfn + slot->npages);
Quoted 8 lines

arch/x86/kvm/mmu/mmu.c void kvm_mmu_pre_destroy_vm(struct kvm *kvm) · line 7100

Quoted 5 lines
arch/x86/kvm/mmu/mmu.c +5 -0
+		mixed = mem_attrs_mixed(kvm, slot, level, attrs, gfn, gfn_end);
+		linfo_set_mixed(gfn, slot, level, mixed);
+
+		if (first == last)
+			return;

continue.

Ya!

Quoted 9 lines

arch/x86/kvm/mmu/mmu.c void kvm_mmu_pre_destroy_vm(struct kvm *kvm) · line 7106

Quoted 7 lines
arch/x86/kvm/mmu/mmu.c +6 -0
+
+		for (gfn = first + pages; gfn < last; gfn += pages)
+			linfo_set_mixed(gfn, slot, level, false);
+
+		gfn = last;
+		gfn_end = min(last + pages, slot->base_gfn + slot->npages);

if (gfn == gfn_end) continue.

Quoted 2 lines

Do you see a case where gfn can equal to gfn_end? Though it does not hurt to add a check.

If last == base_gfn + npages, gfn == gfn_end can occur.

Quoted 14 lines
Quoted 5 lines
Quoted 5 lines
arch/x86/kvm/x86.c +0 -0
index 9a07380f8d3c..5aefcff614d2 100644
@@ -12362,6 +12362,8 @@ static int kvm_alloc_memslot_metadata(struct kvm *kvm,
Quoted 8 lines

arch/x86/kvm/x86.c static int kvm_alloc_memslot_metadata(struct kvm *kvm, · line 12366

Quoted 5 lines
arch/x86/kvm/x86.c +2 -0
 		if ((slot->base_gfn + npages) & (KVM_PAGES_PER_HPAGE(level) - 1))
 			linfo[lpages - 1].disallow_lpage = 1;
 		ugfn = slot->userspace_addr >> PAGE_SHIFT;
+		if (kvm_slot_can_be_private(slot))
+			ugfn |= slot->restricted_offset >> PAGE_SHIFT;

Is there any alignment restriction? If no, It should be +=. In practice, alignment will hold though.

Quoted 4 lines

All we need here is checking whether both userspace_addr and restricted_offset are aligned to HPAGE_SIZE or not. '+=' actually can yield wrong value in cases when userspace_addr + restricted_offset is aligned to HPAGE_SIZE but individually they may not align to HPAGE_SIZE.

Ah, got it. The blow comment explains it.

Quoted 8 lines

Thanks,
Chao

Quoted 6 lines

Thanks,

Quoted 3 lines
arch/x86/kvm/x86.c +0 -0
 		/*
 		 * If the gfn and userspace address are not aligned wrt each
 		 * other, disable large page support for this slot.
Signature · 1 line
Isaku Yamahata <[email protected]>
author
#283 lore raw

Re: [PATCH v10 7/9] KVM: Update lpage info when private/shared memory are mixed

Quoted 204 lines · Isaku Yamahata
Quoted 201 lines · Chao Peng
Quoted 179 lines · Isaku Yamahata
Quoted 176 lines · Chao Peng

A large page with mixed private/shared subpages can't be mapped as large page since its sub private/shared pages are from different memory backends and may also treated by architecture differently. When private/shared memory are mixed in a large page, the current lpage_info is not sufficient to decide whether the page can be mapped as large page or not and additional private/shared mixed information is needed.

Tracking this 'mixed' information with the current 'count' like disallow_lpage is a bit challenge so reserve a bit in 'disallow_lpage' to indicate a large page has mixed private/share subpages and update this 'mixed' bit whenever the memory attribute is changed between private and shared.

---
 arch/x86/include/asm/kvm_host.h |   8 ++
 arch/x86/kvm/mmu/mmu.c          | 134 +++++++++++++++++++++++++++++++-
 arch/x86/kvm/x86.c              |   2 +
 include/linux/kvm_host.h        |  19 +++++
 virt/kvm/kvm_main.c             |   9 ++-
 5 files changed, 169 insertions(+), 3 deletions(-)
arch/x86/include/asm/kvm_host.h +8 -0
index 283cbb83d6ae..7772ab37ac89 100644
@@ -38,6 +38,7 @@
 #include <asm/hyperv-tlfs.h>
 
 #define __KVM_HAVE_ARCH_VCPU_DEBUGFS
+#define __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES
 
 #define KVM_MAX_VCPUS 1024
 
@@ -1011,6 +1012,13 @@ struct kvm_vcpu_arch {
 #endif
 };
 
+/*
+ * Use a bit in disallow_lpage to indicate private/shared pages mixed at the
+ * level. The remaining bits are used as a reference count.
+ */
+#define KVM_LPAGE_PRIVATE_SHARED_MIXED		(1U << 31)
+#define KVM_LPAGE_COUNT_MAX			((1U << 31) - 1)
+
 struct kvm_lpage_info {
 	int disallow_lpage;
 };
arch/x86/kvm/mmu/mmu.c +108 -1
index e2c70b5afa3e..2190fd8c95c0 100644
@@ -763,11 +763,16 @@ static void update_gfn_disallow_lpage_count(const struct kvm_memory_slot *slot,
 {
 	struct kvm_lpage_info *linfo;
 	int i;
+	int disallow_count;
 
 	for (i = PG_LEVEL_2M; i <= KVM_MAX_HUGEPAGE_LEVEL; ++i) {
 		linfo = lpage_info_slot(gfn, slot, i);
+
+		disallow_count = linfo->disallow_lpage & KVM_LPAGE_COUNT_MAX;
+		WARN_ON(disallow_count + count < 0 ||
+			disallow_count > KVM_LPAGE_COUNT_MAX - count);
+
 		linfo->disallow_lpage += count;
-		WARN_ON(linfo->disallow_lpage < 0);
 	}
 }
 
@@ -6986,3 +6991,130 @@ void kvm_mmu_pre_destroy_vm(struct kvm *kvm)
 	if (kvm->arch.nx_huge_page_recovery_thread)
 		kthread_stop(kvm->arch.nx_huge_page_recovery_thread);
 }
+
+static bool linfo_is_mixed(struct kvm_lpage_info *linfo)
+{
+	return linfo->disallow_lpage & KVM_LPAGE_PRIVATE_SHARED_MIXED;
+}
+
+static void linfo_set_mixed(gfn_t gfn, struct kvm_memory_slot *slot,
+			    int level, bool mixed)
+{
+	struct kvm_lpage_info *linfo = lpage_info_slot(gfn, slot, level);
+
+	if (mixed)
+		linfo->disallow_lpage |= KVM_LPAGE_PRIVATE_SHARED_MIXED;
+	else
+		linfo->disallow_lpage &= ~KVM_LPAGE_PRIVATE_SHARED_MIXED;
+}
+
+static bool is_expected_attr_entry(void *entry, unsigned long expected_attrs)
+{
+	bool expect_private = expected_attrs & KVM_MEMORY_ATTRIBUTE_PRIVATE;
+
+	if (xa_to_value(entry) & KVM_MEMORY_ATTRIBUTE_PRIVATE) {
+		if (!expect_private)
+			return false;
+	} else if (expect_private)
+		return false;
+
+	return true;
+}
+
+static bool mem_attrs_mixed_2m(struct kvm *kvm, unsigned long attrs,
+			       gfn_t start, gfn_t end)
+{
+	XA_STATE(xas, &kvm->mem_attr_array, start);
+	gfn_t gfn = start;
+	void *entry;
+	bool mixed = false;
+
+	rcu_read_lock();
+	entry = xas_load(&xas);
+	while (gfn < end) {
+		if (xas_retry(&xas, entry))
+			continue;
+
+		KVM_BUG_ON(gfn != xas.xa_index, kvm);
+
+		if (!is_expected_attr_entry(entry, attrs)) {
+			mixed = true;
+			break;
+		}
+
+		entry = xas_next(&xas);
+		gfn++;
+	}
+
+	rcu_read_unlock();
+	return mixed;
+}
+
+static bool mem_attrs_mixed(struct kvm *kvm, struct kvm_memory_slot *slot,
+			    int level, unsigned long attrs,
+			    gfn_t start, gfn_t end)
+{
+	unsigned long gfn;
+
+	if (level == PG_LEVEL_2M)
+		return mem_attrs_mixed_2m(kvm, attrs, start, end);
+
+	for (gfn = start; gfn < end; gfn += KVM_PAGES_PER_HPAGE(level - 1))
+		if (linfo_is_mixed(lpage_info_slot(gfn, slot, level - 1)) ||
+		    !is_expected_attr_entry(xa_load(&kvm->mem_attr_array, gfn),
+					    attrs))
+			return true;
+	return false;
+}
+
+static void kvm_update_lpage_private_shared_mixed(struct kvm *kvm,
+						  struct kvm_memory_slot *slot,
+						  unsigned long attrs,
+						  gfn_t start, gfn_t end)
+{
+	unsigned long pages, mask;
+	gfn_t gfn, gfn_end, first, last;
+	int level;
+	bool mixed;
+
+	/*
+	 * The sequence matters here: we set the higher level basing on the
+	 * lower level's scanning result.
+	 */
+	for (level = PG_LEVEL_2M; level <= KVM_MAX_HUGEPAGE_LEVEL; level++) {
+		pages = KVM_PAGES_PER_HPAGE(level);
+		mask = ~(pages - 1);
+		first = start & mask;
+		last = (end - 1) & mask;
+
+		/*
+		 * We only need to scan the head and tail page, for middle pages
+		 * we know they will not be mixed.
+		 */
+		gfn = max(first, slot->base_gfn);
+		gfn_end = min(first + pages, slot->base_gfn + slot->npages);
Quoted 8 lines

arch/x86/kvm/mmu/mmu.c void kvm_mmu_pre_destroy_vm(struct kvm *kvm) · line 7100

Quoted 5 lines
arch/x86/kvm/mmu/mmu.c +5 -0
+		mixed = mem_attrs_mixed(kvm, slot, level, attrs, gfn, gfn_end);
+		linfo_set_mixed(gfn, slot, level, mixed);
+
+		if (first == last)
+			return;

continue.

Ya!

Quoted 9 lines

arch/x86/kvm/mmu/mmu.c void kvm_mmu_pre_destroy_vm(struct kvm *kvm) · line 7106

Quoted 7 lines
arch/x86/kvm/mmu/mmu.c +6 -0
+
+		for (gfn = first + pages; gfn < last; gfn += pages)
+			linfo_set_mixed(gfn, slot, level, false);
+
+		gfn = last;
+		gfn_end = min(last + pages, slot->base_gfn + slot->npages);

if (gfn == gfn_end) continue.

Quoted 4 lines
Quoted 2 lines

Do you see a case where gfn can equal to gfn_end? Though it does not hurt to add a check.

If last == base_gfn + npages, gfn == gfn_end can occur.

'end' is guaranteed <= base_gfn + npages in kvm_unmap_mem_range():

	gfn_range.end = min(end, slot->base_gfn + slot->npages);

And 'last' is defined as:

	last = (end - 1) & mask;

Then the math is:

	last = (end - 1) & mask
	     <= end - 1
	     <= base_gfn + npages - 1
	     <  base_gfn + npages

Thanks,
Chao

Quoted 33 lines
Quoted 16 lines
Quoted 15 lines

arch/x86/kvm/x86.c static int kvm_alloc_memslot_metadata(struct kvm *kvm, · line 12366

Quoted 12 lines
arch/x86/kvm/x86.c +2 -0
index 9a07380f8d3c..5aefcff614d2 100644
@@ -12362,6 +12362,8 @@ static int kvm_alloc_memslot_metadata(struct kvm *kvm,
 		if ((slot->base_gfn + npages) & (KVM_PAGES_PER_HPAGE(level) - 1))
 			linfo[lpages - 1].disallow_lpage = 1;
 		ugfn = slot->userspace_addr >> PAGE_SHIFT;
+		if (kvm_slot_can_be_private(slot))
+			ugfn |= slot->restricted_offset >> PAGE_SHIFT;

Is there any alignment restriction? If no, It should be +=. In practice, alignment will hold though.

Quoted 4 lines

All we need here is checking whether both userspace_addr and restricted_offset are aligned to HPAGE_SIZE or not. '+=' actually can yield wrong value in cases when userspace_addr + restricted_offset is aligned to HPAGE_SIZE but individually they may not align to HPAGE_SIZE.

Ah, got it. The blow comment explains it.

Quoted 8 lines

Thanks,
Chao

Quoted 6 lines

Thanks,

Quoted 3 lines
arch/x86/kvm/x86.c +0 -0
 		/*
 		 * If the gfn and userspace address are not aligned wrt each
 		 * other, disable large page support for this slot.
Signature · 1 line
Isaku Yamahata <[email protected]>
#284 lore raw

Re: [PATCH v10 7/9] KVM: Update lpage info when private/shared memory are mixed

In reply to Chao Peng

arch/x86/include/asm/kvm_host.h line 41

Quoted 9 lines · Chao Peng
arch/x86/include/asm/kvm_host.h +1 -0
index 283cbb83d6ae..7772ab37ac89 100644
@@ -38,6 +38,7 @@
 #include <asm/hyperv-tlfs.h>
 
 #define __KVM_HAVE_ARCH_VCPU_DEBUGFS
+#define __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES

No need for this, I think we should just make it mandatory to implement the arch hook when CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES=y. If another arch gains support for mem attributes and doesn't need the hook, then we can simply add a weak helper (or maybe add a #define then if we feel that's the way to go).

arch/x86/include/asm/kvm_host.h struct kvm_vcpu_arch { · line 1019

Quoted 11 lines
arch/x86/include/asm/kvm_host.h +5 -0
 #define KVM_MAX_VCPUS 1024
 
@@ -1011,6 +1012,13 @@ struct kvm_vcpu_arch {
 #endif
 };
 
+/*
+ * Use a bit in disallow_lpage to indicate private/shared pages mixed at the
+ * level. The remaining bits are used as a reference count.
+ */
+#define KVM_LPAGE_PRIVATE_SHARED_MIXED		(1U << 31)

Similar to the need to unmap, I think we should just say "mixed" and ignore the private vs. shared, i.e. make this a flag for all memory attributes.

arch/x86/include/asm/kvm_host.h struct kvm_vcpu_arch { · line 1020

Quoted 1 line
arch/x86/include/asm/kvm_host.h +1 -0
+#define KVM_LPAGE_COUNT_MAX			((1U << 31) - 1)

"MAX" is technically correct, but it's more of a mask. I think we can make it a moot point though. There's no need to mask the count, we just want to assert that adjusting the counting doesn't change the flag.

I would also say throw these defines into mmu.c, at least pending the bug fix for kvm_alloc_memslot_metadata() (more on that below).

Quoted 12 lines
arch/x86/include/asm/kvm_host.h +0 -0
 struct kvm_lpage_info {
 	int disallow_lpage;
 };
arch/x86/kvm/mmu/mmu.c +1 -0
index e2c70b5afa3e..2190fd8c95c0 100644
@@ -763,11 +763,16 @@ static void update_gfn_disallow_lpage_count(const struct kvm_memory_slot *slot,
 {
 	struct kvm_lpage_info *linfo;
 	int i;
+	int disallow_count;

arch/x86/kvm/mmu/mmu.c static void update_gfn_disallow_lpage_count(const struct kvm_memory_slot *slot, · line 770

Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +5 -1
 
 	for (i = PG_LEVEL_2M; i <= KVM_MAX_HUGEPAGE_LEVEL; ++i) {
 		linfo = lpage_info_slot(gfn, slot, i);
+
+		disallow_count = linfo->disallow_lpage & KVM_LPAGE_COUNT_MAX;
+		WARN_ON(disallow_count + count < 0 ||
+			disallow_count > KVM_LPAGE_COUNT_MAX - count);
+
 		linfo->disallow_lpage += count;
-		WARN_ON(linfo->disallow_lpage < 0);

It's been a long week so don't trust my math, but I believe this can simply be:

		old = linfo->disallow_lpage;
		linfo->disallow_lpage += count;

		WARN_ON_ONCE((old ^ linfo->disallow_lpage) & KVM_LPAGE_MIXED_FLAG);
Quoted 23 lines
arch/x86/kvm/mmu/mmu.c +16 -0
 	}
 }
 
@@ -6986,3 +6991,130 @@ void kvm_mmu_pre_destroy_vm(struct kvm *kvm)
 	if (kvm->arch.nx_huge_page_recovery_thread)
 		kthread_stop(kvm->arch.nx_huge_page_recovery_thread);
 }
+
+static bool linfo_is_mixed(struct kvm_lpage_info *linfo)
+{
+	return linfo->disallow_lpage & KVM_LPAGE_PRIVATE_SHARED_MIXED;
+}
+
+static void linfo_set_mixed(gfn_t gfn, struct kvm_memory_slot *slot,
+			    int level, bool mixed)
+{
+	struct kvm_lpage_info *linfo = lpage_info_slot(gfn, slot, level);
+
+	if (mixed)
+		linfo->disallow_lpage |= KVM_LPAGE_PRIVATE_SHARED_MIXED;
+	else
+		linfo->disallow_lpage &= ~KVM_LPAGE_PRIVATE_SHARED_MIXED;
+}

arch/x86/kvm/mmu/mmu.c void kvm_mmu_pre_destroy_vm(struct kvm *kvm) · line 7019

Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +10 -0
+
+static bool is_expected_attr_entry(void *entry, unsigned long expected_attrs)
+{
+	bool expect_private = expected_attrs & KVM_MEMORY_ATTRIBUTE_PRIVATE;
+
+	if (xa_to_value(entry) & KVM_MEMORY_ATTRIBUTE_PRIVATE) {
+		if (!expect_private)
+			return false;
+	} else if (expect_private)
+		return false;

This is messy. If we drop the private vs. shared specifity, this can go away if we add a helper to get attributes

	static inline unsigned long kvm_get_memory_attributes(struct kvm *kvm, gfn_t gfn)
	{
		return xa_to_value(xa_load(&kvm->mem_attr_array, gfn));
	}

and then we can do

		if (KVM_BUG_ON(gfn != xas.xa_index, kvm) ||
		    attrs != kvm_get_memory_attributes(kvm, gfn)) {
			mixed = true;
			break;
		}

and

		if (linfo_is_mixed(lpage_info_slot(gfn, slot, level - 1)) ||
		    attrs != kvm_get_memory_attributes(kvm, gfn))
			return true;
Quoted 9 lines
arch/x86/kvm/mmu/mmu.c +9 -0
+
+	return true;
+}
+
+static bool mem_attrs_mixed_2m(struct kvm *kvm, unsigned long attrs,
+			       gfn_t start, gfn_t end)
+{
+	XA_STATE(xas, &kvm->mem_attr_array, start);
+	gfn_t gfn = start;

arch/x86/kvm/mmu/mmu.c void kvm_mmu_pre_destroy_vm(struct kvm *kvm) · line 7038

Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +10 -0
+	void *entry;
+	bool mixed = false;
+
+	rcu_read_lock();
+	entry = xas_load(&xas);
+	while (gfn < end) {
+		if (xas_retry(&xas, entry))
+			continue;
+
+		KVM_BUG_ON(gfn != xas.xa_index, kvm);

As above, I think it's worth bailing immediately if there's a mismatch.

Quoted 5 lines
arch/x86/kvm/mmu/mmu.c +5 -0
+
+		if (!is_expected_attr_entry(entry, attrs)) {
+			mixed = true;
+			break;
+		}

arch/x86/kvm/mmu/mmu.c void kvm_mmu_pre_destroy_vm(struct kvm *kvm) · line 7053

Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +10 -0
+
+		entry = xas_next(&xas);
+		gfn++;
+	}
+
+	rcu_read_unlock();
+	return mixed;
+}
+
+static bool mem_attrs_mixed(struct kvm *kvm, struct kvm_memory_slot *slot,

s/mem_attrs_mixed/has_mixed_attrs to make it clear this is querying, not setting. And has_mixed_attrs_2m() above.

arch/x86/kvm/mmu/mmu.c void kvm_mmu_pre_destroy_vm(struct kvm *kvm) · line 7062

Quoted 9 lines
arch/x86/kvm/mmu/mmu.c +9 -0
+			    int level, unsigned long attrs,
+			    gfn_t start, gfn_t end)
+{
+	unsigned long gfn;
+
+	if (level == PG_LEVEL_2M)
+		return mem_attrs_mixed_2m(kvm, attrs, start, end);
+
+	for (gfn = start; gfn < end; gfn += KVM_PAGES_PER_HPAGE(level - 1))

Curly braces needed on the for-loop.

Quoted 45 lines
arch/x86/kvm/mmu/mmu.c +45 -0
+		if (linfo_is_mixed(lpage_info_slot(gfn, slot, level - 1)) ||
+		    !is_expected_attr_entry(xa_load(&kvm->mem_attr_array, gfn),
+					    attrs))
+			return true;
+	return false;
+}
+
+static void kvm_update_lpage_private_shared_mixed(struct kvm *kvm,
+						  struct kvm_memory_slot *slot,
+						  unsigned long attrs,
+						  gfn_t start, gfn_t end)
+{
+	unsigned long pages, mask;
+	gfn_t gfn, gfn_end, first, last;
+	int level;
+	bool mixed;
+
+	/*
+	 * The sequence matters here: we set the higher level basing on the
+	 * lower level's scanning result.
+	 */
+	for (level = PG_LEVEL_2M; level <= KVM_MAX_HUGEPAGE_LEVEL; level++) {
+		pages = KVM_PAGES_PER_HPAGE(level);
+		mask = ~(pages - 1);
+		first = start & mask;
+		last = (end - 1) & mask;
+
+		/*
+		 * We only need to scan the head and tail page, for middle pages
+		 * we know they will not be mixed.
+		 */
+		gfn = max(first, slot->base_gfn);
+		gfn_end = min(first + pages, slot->base_gfn + slot->npages);
+		mixed = mem_attrs_mixed(kvm, slot, level, attrs, gfn, gfn_end);
+		linfo_set_mixed(gfn, slot, level, mixed);
+
+		if (first == last)
+			return;
+
+		for (gfn = first + pages; gfn < last; gfn += pages)
+			linfo_set_mixed(gfn, slot, level, false);
+
+		gfn = last;
+		gfn_end = min(last + pages, slot->base_gfn + slot->npages);
+		mixed = mem_attrs_mixed(kvm, slot, level, attrs, gfn, gfn_end);

arch/x86/kvm/mmu/mmu.c void kvm_mmu_pre_destroy_vm(struct kvm *kvm) · line 7117

Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +10 -0
+		linfo_set_mixed(gfn, slot, level, mixed);
+	}
+}
+
+void kvm_arch_set_memory_attributes(struct kvm *kvm,
+				    struct kvm_memory_slot *slot,
+				    unsigned long attrs,
+				    gfn_t start, gfn_t end)
+{
+	if (kvm_slot_can_be_private(slot))

Make this an early return optimization, with a comment explaining that KVM x86 doesn't yet support other attributes.

	/*
	 * KVM x86 currently only supports KVM_MEMORY_ATTRIBUTE_PRIVATE, skip
	 * the slot if the slot will never consume the PRIVATE attribute.
	 */
	if (!kvm_slot_can_be_private(slot))
		return;

arch/x86/kvm/x86.c static int kvm_alloc_memslot_metadata(struct kvm *kvm, · line 12366

Quoted 13 lines
arch/x86/kvm/mmu/mmu.c +3 -0
+		kvm_update_lpage_private_shared_mixed(kvm, slot, attrs,
+						      start, end);
+}
arch/x86/kvm/x86.c +2 -0
index 9a07380f8d3c..5aefcff614d2 100644
@@ -12362,6 +12362,8 @@ static int kvm_alloc_memslot_metadata(struct kvm *kvm,
 		if ((slot->base_gfn + npages) & (KVM_PAGES_PER_HPAGE(level) - 1))
 			linfo[lpages - 1].disallow_lpage = 1;
 		ugfn = slot->userspace_addr >> PAGE_SHIFT;
+		if (kvm_slot_can_be_private(slot))
+			ugfn |= slot->restricted_offset >> PAGE_SHIFT;

I would rather reject memslot if the gfn has lesser alignment than the offset. I'm totally ok with this approach _if_ there's a use case. Until such a use case presents itself, I would rather be conservative from a uAPI perspective.

Quoted 8 lines
arch/x86/kvm/x86.c +0 -0
 		/*
 		 * If the gfn and userspace address are not aligned wrt each
 		 * other, disable large page support for this slot.
include/linux/kvm_host.h +0 -0
index 3331c0c92838..25099c94e770 100644

include/linux/kvm_host.h struct kvm_memory_slot { · line 597

Quoted 6 lines
include/linux/kvm_host.h +3 -0
@@ -592,6 +592,11 @@ struct kvm_memory_slot {
 	struct restrictedmem_notifier notifier;
 };
 
+static inline bool kvm_slot_can_be_private(const struct kvm_memory_slot *slot)
+{
+	return slot && (slot->flags & KVM_MEM_PRIVATE);

KVM_MEM_PRIVATE should really be defined only when private memory is exposed to userspace. For this patch, even though it means we have untestable code, I think it makes sense to "return false".

Quoted 12 lines
include/linux/kvm_host.h +5 -0
+}
+
 static inline bool kvm_slot_dirty_track_enabled(const struct kvm_memory_slot *slot)
 {
 	return slot->flags & KVM_MEM_LOG_DIRTY_PAGES;
@@ -2316,4 +2321,18 @@ static inline void kvm_account_pgtable_pages(void *virt, int nr)
 /* Max number of entries allowed for each kvm dirty ring */
 #define  KVM_DIRTY_RING_MAX_ENTRIES  65536
 
+#ifdef __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES
+void kvm_arch_set_memory_attributes(struct kvm *kvm,
+				    struct kvm_memory_slot *slot,

include/linux/kvm_host.h static inline void kvm_account_pgtable_pages(void *virt, int nr) · line 2336

Quoted 10 lines
include/linux/kvm_host.h +10 -0
+				    unsigned long attrs,
+				    gfn_t start, gfn_t end);
+#else
+static inline void kvm_arch_set_memory_attributes(struct kvm *kvm,
+						  struct kvm_memory_slot *slot,
+						  unsigned long attrs,
+						  gfn_t start, gfn_t end)
+{
+}
+#endif /* __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES */

As above, no stub is necessary.

virt/kvm/kvm_main.c static u64 kvm_supported_mem_attributes(struct kvm *kvm) · line 2357

Quoted 10 lines
include/linux/kvm_host.h +0 -0
 #endif
virt/kvm/kvm_main.c +0 -1
index 4e1e1e113bf0..e107afea32f0 100644
@@ -2354,7 +2354,8 @@ static u64 kvm_supported_mem_attributes(struct kvm *kvm)
 	return 0;
 }
 
-static void kvm_unmap_mem_range(struct kvm *kvm, gfn_t start, gfn_t end)

Feedback for an earlier patch (to avoid churn): this should be kvm_mem_attrs_changed() or so now that this does more than just unmap.

virt/kvm/kvm_main.c static u64 kvm_supported_mem_attributes(struct kvm *kvm) · line 2358

Quoted 2 lines
virt/kvm/kvm_main.c +2 -0
+static void kvm_unmap_mem_range(struct kvm *kvm, gfn_t start, gfn_t end,
+				unsigned long attrs)

Weird nit. I think we should keep the prototypes for kvm_mem_attrs_changed() and kvm_arch_set_memory_attributes() somewhat similar, i.e. squeeze in @attrs before @start.

Quoted 26 lines
virt/kvm/kvm_main.c +5 -1
 {
 	struct kvm_gfn_range gfn_range;
 	struct kvm_memory_slot *slot;
@@ -2378,6 +2379,10 @@ static void kvm_unmap_mem_range(struct kvm *kvm, gfn_t start, gfn_t end)
 			gfn_range.slot = slot;
 
 			r |= kvm_unmap_gfn_range(kvm, &gfn_range);
+
+			kvm_arch_set_memory_attributes(kvm, slot, attrs,
+						       gfn_range.start,
+						       gfn_range.end);
 		}
 	}
 
@@ -2427,7 +2432,7 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
 		idx = srcu_read_lock(&kvm->srcu);
 		KVM_MMU_LOCK(kvm);
 		if (i > start)
-			kvm_unmap_mem_range(kvm, start, i);
+			kvm_unmap_mem_range(kvm, start, i, attrs->attributes);
 		kvm_mmu_invalidate_end(kvm);
 		KVM_MMU_UNLOCK(kvm);
 		srcu_read_unlock(&kvm->srcu, idx);
Signature · 1 line
2.25.1
#285 lore raw

Re: [PATCH v10 7/9] KVM: Update lpage info when private/shared memory are mixed

In reply to Chao Peng

arch/x86/kvm/x86.c static int kvm_alloc_memslot_metadata(struct kvm *kvm, · line 12369

Quoted 13 lines · Chao Peng
arch/x86/kvm/x86.c +2 -0
index 9a07380f8d3c..5aefcff614d2 100644
@@ -12362,6 +12362,8 @@ static int kvm_alloc_memslot_metadata(struct kvm *kvm,
 		if ((slot->base_gfn + npages) & (KVM_PAGES_PER_HPAGE(level) - 1))
 			linfo[lpages - 1].disallow_lpage = 1;
 		ugfn = slot->userspace_addr >> PAGE_SHIFT;
+		if (kvm_slot_can_be_private(slot))
+			ugfn |= slot->restricted_offset >> PAGE_SHIFT;
 		/*
 		 * If the gfn and userspace address are not aligned wrt each
 		 * other, disable large page support for this slot.

Forgot to talk about the bug. This code needs to handle the scenario where a memslot is created with existing, non-uniform attributes. It might be a bit ugly (I didn't even try to write the code), but it's definitely possible, and since memslot updates are already slow I think it's best to handle things here.

In the meantime, I added this so we don't forget to fix it before merging.

#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
	pr_crit_once("FIXME: Walk the memory attributes of the slot and set the mixed status appropriately");

#endif

author
#286 lore raw

Re: [PATCH v10 7/9] KVM: Update lpage info when private/shared memory are mixed

Quoted 20 lines · Sean Christopherson

arch/x86/kvm/x86.c static int kvm_alloc_memslot_metadata(struct kvm *kvm, · line 12369

Quoted 13 lines · Chao Peng
arch/x86/kvm/x86.c +2 -0
index 9a07380f8d3c..5aefcff614d2 100644
@@ -12362,6 +12362,8 @@ static int kvm_alloc_memslot_metadata(struct kvm *kvm,
 		if ((slot->base_gfn + npages) & (KVM_PAGES_PER_HPAGE(level) - 1))
 			linfo[lpages - 1].disallow_lpage = 1;
 		ugfn = slot->userspace_addr >> PAGE_SHIFT;
+		if (kvm_slot_can_be_private(slot))
+			ugfn |= slot->restricted_offset >> PAGE_SHIFT;
 		/*
 		 * If the gfn and userspace address are not aligned wrt each
 		 * other, disable large page support for this slot.

Forgot to talk about the bug. This code needs to handle the scenario where a memslot is created with existing, non-uniform attributes. It might be a bit ugly (I didn't even try to write the code), but it's definitely possible, and since memslot updates are already slow I think it's best to handle things here.

Quoted 5 lines

In the meantime, I added this so we don't forget to fix it before merging.

#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
	pr_crit_once("FIXME: Walk the memory attributes of the slot and set the mixed status appropriately");

#endif

Here is the code to fix (based on your latest github repo).

arch/x86/include/asm/kvm_host.h +5 -0
index e552374f2357..609ff1cba9c5 100644
@@ -2195,4 +2195,9 @@ int memslot_rmap_alloc(struct kvm_memory_slot *slot, unsigned long npages);
 	 KVM_X86_QUIRK_FIX_HYPERCALL_INSN |	\
 	 KVM_X86_QUIRK_MWAIT_NEVER_UD_FAULTS)
 
+#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
+void kvm_memory_attributes_create_memslot(struct kvm *kvm,
+					  struct kvm_memory_slot *slot);
+#endif
+
 #endif /* _ASM_X86_KVM_HOST_H */
arch/x86/kvm/mmu/mmu.c +39 -10

74 lines not on this page · raw message

arch/x86/kvm/x86.c +1 -1
index 268c3d16894d..c1074aecf2d0 100644
@@ -12443,7 +12443,7 @@ static int kvm_alloc_memslot_metadata(struct kvm *kvm,
 	}
 
 #ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
-	pr_crit_once("FIXME: Walk the memory attributes of the slot and set the mixed status appropriately");
+	kvm_memory_attributes_create_memslot(kvm, slot);
 #endif
 
 	if (kvm_page_track_create_memslot(kvm, slot, npages))
author
#287 lore raw

PATCH v10 8/9 KVM: Handle page fault for private memory

In reply to Chao Peng

A KVM_MEM_PRIVATE memslot can include both fd-based private memory and hva-based shared memory. Architecture code (like TDX code) can tell whether the on-going fault is private or not. This patch adds a 'is_private' field to kvm_page_fault to indicate this and architecture code is expected to set it.

To handle page fault for such memslot, the handling logic is different depending on whether the fault is private or shared. KVM checks if 'is_private' matches the host's view of the page (maintained in mem_attr_array).
- For a successful match, private pfn is obtained with restrictedmem_get_page() and shared pfn is obtained with existing get_user_pages().
- For a failed match, KVM causes a KVM_EXIT_MEMORY_FAULT exit to userspace. Userspace then can convert memory between private/shared in host's view and retry the fault.

---
 arch/x86/kvm/mmu/mmu.c          | 63 +++++++++++++++++++++++++++++++--
 arch/x86/kvm/mmu/mmu_internal.h | 14 +++++++-
 arch/x86/kvm/mmu/mmutrace.h     |  1 +
 arch/x86/kvm/mmu/tdp_mmu.c      |  2 +-
 include/linux/kvm_host.h        | 30 ++++++++++++++++
 5 files changed, 105 insertions(+), 5 deletions(-)
arch/x86/kvm/mmu/mmu.c +60 -3
index 2190fd8c95c0..b1953ebc012e 100644
@@ -3058,7 +3058,7 @@ static int host_pfn_mapping_level(struct kvm *kvm, gfn_t gfn,
 
 int kvm_mmu_max_mapping_level(struct kvm *kvm,
 			      const struct kvm_memory_slot *slot, gfn_t gfn,
-			      int max_level)
+			      int max_level, bool is_private)
 {
 	struct kvm_lpage_info *linfo;
 	int host_level;
@@ -3070,6 +3070,9 @@ int kvm_mmu_max_mapping_level(struct kvm *kvm,
 			break;
 	}
 
+	if (is_private)
+		return max_level;
+
 	if (max_level == PG_LEVEL_4K)
 		return PG_LEVEL_4K;
 
@@ -3098,7 +3101,8 @@ void kvm_mmu_hugepage_adjust(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault
 	 * level, which will be used to do precise, accurate accounting.
 	 */
 	fault->req_level = kvm_mmu_max_mapping_level(vcpu->kvm, slot,
-						     fault->gfn, fault->max_level);
+						     fault->gfn, fault->max_level,
+						     fault->is_private);
 	if (fault->req_level == PG_LEVEL_4K || fault->huge_page_disallowed)
 		return;
 
@@ -4178,6 +4182,49 @@ void kvm_arch_async_page_ready(struct kvm_vcpu *vcpu, struct kvm_async_pf *work)
 	kvm_mmu_do_page_fault(vcpu, work->cr2_or_gpa, 0, true);
 }
 
+static inline u8 order_to_level(int order)
+{
+	BUILD_BUG_ON(KVM_MAX_HUGEPAGE_LEVEL > PG_LEVEL_1G);
+
+	if (order >= KVM_HPAGE_GFN_SHIFT(PG_LEVEL_1G))
+		return PG_LEVEL_1G;
+
+	if (order >= KVM_HPAGE_GFN_SHIFT(PG_LEVEL_2M))
+		return PG_LEVEL_2M;
+
+	return PG_LEVEL_4K;
+}
+
+static int kvm_do_memory_fault_exit(struct kvm_vcpu *vcpu,
+				    struct kvm_page_fault *fault)
+{
+	vcpu->run->exit_reason = KVM_EXIT_MEMORY_FAULT;
+	if (fault->is_private)
+		vcpu->run->memory.flags = KVM_MEMORY_EXIT_FLAG_PRIVATE;
+	else
+		vcpu->run->memory.flags = 0;
+	vcpu->run->memory.gpa = fault->gfn << PAGE_SHIFT;
+	vcpu->run->memory.size = PAGE_SIZE;
+	return RET_PF_USER;
+}
+
+static int kvm_faultin_pfn_private(struct kvm_vcpu *vcpu,
+				   struct kvm_page_fault *fault)
+{
+	int order;
+	struct kvm_memory_slot *slot = fault->slot;
+
+	if (!kvm_slot_can_be_private(slot))
+		return kvm_do_memory_fault_exit(vcpu, fault);
+
+	if (kvm_restricted_mem_get_pfn(slot, fault->gfn, &fault->pfn, &order))
+		return RET_PF_RETRY;
+
+	fault->max_level = min(order_to_level(order), fault->max_level);
+	fault->map_writable = !(slot->flags & KVM_MEM_READONLY);
+	return RET_PF_CONTINUE;
+}
+
 static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
 {
 	struct kvm_memory_slot *slot = fault->slot;
@@ -4210,6 +4257,12 @@ static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
 			return RET_PF_EMULATE;
 	}
 
+	if (fault->is_private != kvm_mem_is_private(vcpu->kvm, fault->gfn))
+		return kvm_do_memory_fault_exit(vcpu, fault);
+
+	if (fault->is_private)
+		return kvm_faultin_pfn_private(vcpu, fault);
+
 	async = false;
 	fault->pfn = __gfn_to_pfn_memslot(slot, fault->gfn, false, false, &async,
 					  fault->write, &fault->map_writable,
@@ -5599,6 +5652,9 @@ int noinline kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 err
 			return -EIO;
 	}
 
+	if (r == RET_PF_USER)
+		return 0;
+
 	if (r < 0)
 		return r;
 	if (r != RET_PF_EMULATE)
@@ -6452,7 +6508,8 @@ static bool kvm_mmu_zap_collapsible_spte(struct kvm *kvm,
 		 */
 		if (sp->role.direct &&
 		    sp->role.level < kvm_mmu_max_mapping_level(kvm, slot, sp->gfn,
-							       PG_LEVEL_NUM)) {
+							       PG_LEVEL_NUM,
+							       false)) {
 			kvm_zap_one_rmap_spte(kvm, rmap_head, sptep);
 
 			if (kvm_available_flush_tlb_with_range())
arch/x86/kvm/mmu/mmu_internal.h +13 -1
index dbaf6755c5a7..5ccf08183b00 100644
@@ -189,6 +189,7 @@ struct kvm_page_fault {
 
 	/* Derived from mmu and global state.  */
 	const bool is_tdp;
+	const bool is_private;
 	const bool nx_huge_page_workaround_enabled;
 
 	/*
@@ -237,6 +238,7 @@ int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault);
  * RET_PF_RETRY: let CPU fault again on the address.
  * RET_PF_EMULATE: mmio page fault, emulate the instruction directly.
  * RET_PF_INVALID: the spte is invalid, let the real page fault path update it.
+ * RET_PF_USER: need to exit to userspace to handle this fault.
  * RET_PF_FIXED: The faulting entry has been fixed.
  * RET_PF_SPURIOUS: The faulting entry was already fixed, e.g. by another vCPU.
  *
@@ -253,6 +255,7 @@ enum {
 	RET_PF_RETRY,
 	RET_PF_EMULATE,
 	RET_PF_INVALID,
+	RET_PF_USER,
 	RET_PF_FIXED,
 	RET_PF_SPURIOUS,
 };
@@ -310,7 +313,7 @@ static inline int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 
 int kvm_mmu_max_mapping_level(struct kvm *kvm,
 			      const struct kvm_memory_slot *slot, gfn_t gfn,
-			      int max_level);
+			      int max_level, bool is_private);
 void kvm_mmu_hugepage_adjust(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault);
 void disallowed_hugepage_adjust(struct kvm_page_fault *fault, u64 spte, int cur_level);
 
@@ -319,4 +322,13 @@ void *mmu_memory_cache_alloc(struct kvm_mmu_memory_cache *mc);
 void track_possible_nx_huge_page(struct kvm *kvm, struct kvm_mmu_page *sp);
 void untrack_possible_nx_huge_page(struct kvm *kvm, struct kvm_mmu_page *sp);
 
+#ifndef CONFIG_HAVE_KVM_RESTRICTED_MEM
+static inline int kvm_restricted_mem_get_pfn(struct kvm_memory_slot *slot,
+					gfn_t gfn, kvm_pfn_t *pfn, int *order)
+{
+	WARN_ON_ONCE(1);
+	return -EOPNOTSUPP;
+}
+#endif /* CONFIG_HAVE_KVM_RESTRICTED_MEM */
+
 #endif /* __KVM_X86_MMU_INTERNAL_H */
arch/x86/kvm/mmu/mmutrace.h +1 -0
index ae86820cef69..2d7555381955 100644
@@ -58,6 +58,7 @@ TRACE_DEFINE_ENUM(RET_PF_CONTINUE);
 TRACE_DEFINE_ENUM(RET_PF_RETRY);
 TRACE_DEFINE_ENUM(RET_PF_EMULATE);
 TRACE_DEFINE_ENUM(RET_PF_INVALID);
+TRACE_DEFINE_ENUM(RET_PF_USER);
 TRACE_DEFINE_ENUM(RET_PF_FIXED);
 TRACE_DEFINE_ENUM(RET_PF_SPURIOUS);
 
arch/x86/kvm/mmu/tdp_mmu.c +1 -1
index 771210ce5181..8ba1a4afc546 100644
@@ -1768,7 +1768,7 @@ static void zap_collapsible_spte_range(struct kvm *kvm,
 			continue;
 
 		max_mapping_level = kvm_mmu_max_mapping_level(kvm, slot,
-							      iter.gfn, PG_LEVEL_NUM);
+						iter.gfn, PG_LEVEL_NUM, false);
 		if (max_mapping_level < iter.level)
 			continue;
 
include/linux/kvm_host.h +30 -0
index 25099c94e770..153842bb33df 100644
@@ -2335,4 +2335,34 @@ static inline void kvm_arch_set_memory_attributes(struct kvm *kvm,
 }
 #endif /* __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES */
 
+#ifdef CONFIG_HAVE_KVM_MEMORY_ATTRIBUTES
+static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
+{
+	return xa_to_value(xa_load(&kvm->mem_attr_array, gfn)) &
+	       KVM_MEMORY_ATTRIBUTE_PRIVATE;
+}
+#else
+static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
+{
+	return false;
+}
+
+#endif /* CONFIG_HAVE_KVM_MEMORY_ATTRIBUTES */
+
+#ifdef CONFIG_HAVE_KVM_RESTRICTED_MEM
+static inline int kvm_restricted_mem_get_pfn(struct kvm_memory_slot *slot,
+					gfn_t gfn, kvm_pfn_t *pfn, int *order)
+{
+	int ret;
+	struct page *page;
+	pgoff_t index = gfn - slot->base_gfn +
+			(slot->restricted_offset >> PAGE_SHIFT);
+
+	ret = restrictedmem_get_page(slot->restricted_file, index,
+				     &page, order);
+	*pfn = page_to_pfn(page);
+	return ret;
+}
+#endif /* CONFIG_HAVE_KVM_RESTRICTED_MEM */
+
 #endif
Signature · 1 line
2.25.1
#288 lore raw

Re: [PATCH v10 8/9] KVM: Handle page fault for private memory

Quoted 42 lines · Chao Peng

A KVM_MEM_PRIVATE memslot can include both fd-based private memory and hva-based shared memory. Architecture code (like TDX code) can tell whether the on-going fault is private or not. This patch adds a 'is_private' field to kvm_page_fault to indicate this and architecture code is expected to set it.

To handle page fault for such memslot, the handling logic is different depending on whether the fault is private or shared. KVM checks if 'is_private' matches the host's view of the page (maintained in mem_attr_array).
- For a successful match, private pfn is obtained with restrictedmem_get_page() and shared pfn is obtained with existing get_user_pages().
- For a failed match, KVM causes a KVM_EXIT_MEMORY_FAULT exit to userspace. Userspace then can convert memory between private/shared in host's view and retry the fault.

---
 arch/x86/kvm/mmu/mmu.c          | 63 +++++++++++++++++++++++++++++++--
 arch/x86/kvm/mmu/mmu_internal.h | 14 +++++++-
 arch/x86/kvm/mmu/mmutrace.h     |  1 +
 arch/x86/kvm/mmu/tdp_mmu.c      |  2 +-
 include/linux/kvm_host.h        | 30 ++++++++++++++++
 5 files changed, 105 insertions(+), 5 deletions(-)
arch/x86/kvm/mmu/mmu.c +1 -1
index 2190fd8c95c0..b1953ebc012e 100644
@@ -3058,7 +3058,7 @@ static int host_pfn_mapping_level(struct kvm *kvm, gfn_t gfn,
 
 int kvm_mmu_max_mapping_level(struct kvm *kvm,
 			      const struct kvm_memory_slot *slot, gfn_t gfn,
-			      int max_level)
+			      int max_level, bool is_private)
 {
 	struct kvm_lpage_info *linfo;
 	int host_level;

arch/x86/kvm/mmu/mmu.c int kvm_mmu_max_mapping_level(struct kvm *kvm, · line 3074

Quoted 5 lines
arch/x86/kvm/mmu/mmu.c +2 -0
@@ -3070,6 +3070,9 @@ int kvm_mmu_max_mapping_level(struct kvm *kvm,
 			break;
 	}
 
+	if (is_private)
+		return max_level;

lpage mixed information already saved, so is that possible to query info->disallow_lpage without care 'is_private' ?

Quoted 216 lines
arch/x86/kvm/mmu/mmu.c +57 -2
+
 	if (max_level == PG_LEVEL_4K)
 		return PG_LEVEL_4K;
 
@@ -3098,7 +3101,8 @@ void kvm_mmu_hugepage_adjust(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault
 	 * level, which will be used to do precise, accurate accounting.
 	 */
 	fault->req_level = kvm_mmu_max_mapping_level(vcpu->kvm, slot,
-						     fault->gfn, fault->max_level);
+						     fault->gfn, fault->max_level,
+						     fault->is_private);
 	if (fault->req_level == PG_LEVEL_4K || fault->huge_page_disallowed)
 		return;
 
@@ -4178,6 +4182,49 @@ void kvm_arch_async_page_ready(struct kvm_vcpu *vcpu, struct kvm_async_pf *work)
 	kvm_mmu_do_page_fault(vcpu, work->cr2_or_gpa, 0, true);
 }
 
+static inline u8 order_to_level(int order)
+{
+	BUILD_BUG_ON(KVM_MAX_HUGEPAGE_LEVEL > PG_LEVEL_1G);
+
+	if (order >= KVM_HPAGE_GFN_SHIFT(PG_LEVEL_1G))
+		return PG_LEVEL_1G;
+
+	if (order >= KVM_HPAGE_GFN_SHIFT(PG_LEVEL_2M))
+		return PG_LEVEL_2M;
+
+	return PG_LEVEL_4K;
+}
+
+static int kvm_do_memory_fault_exit(struct kvm_vcpu *vcpu,
+				    struct kvm_page_fault *fault)
+{
+	vcpu->run->exit_reason = KVM_EXIT_MEMORY_FAULT;
+	if (fault->is_private)
+		vcpu->run->memory.flags = KVM_MEMORY_EXIT_FLAG_PRIVATE;
+	else
+		vcpu->run->memory.flags = 0;
+	vcpu->run->memory.gpa = fault->gfn << PAGE_SHIFT;
+	vcpu->run->memory.size = PAGE_SIZE;
+	return RET_PF_USER;
+}
+
+static int kvm_faultin_pfn_private(struct kvm_vcpu *vcpu,
+				   struct kvm_page_fault *fault)
+{
+	int order;
+	struct kvm_memory_slot *slot = fault->slot;
+
+	if (!kvm_slot_can_be_private(slot))
+		return kvm_do_memory_fault_exit(vcpu, fault);
+
+	if (kvm_restricted_mem_get_pfn(slot, fault->gfn, &fault->pfn, &order))
+		return RET_PF_RETRY;
+
+	fault->max_level = min(order_to_level(order), fault->max_level);
+	fault->map_writable = !(slot->flags & KVM_MEM_READONLY);
+	return RET_PF_CONTINUE;
+}
+
 static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
 {
 	struct kvm_memory_slot *slot = fault->slot;
@@ -4210,6 +4257,12 @@ static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
 			return RET_PF_EMULATE;
 	}
 
+	if (fault->is_private != kvm_mem_is_private(vcpu->kvm, fault->gfn))
+		return kvm_do_memory_fault_exit(vcpu, fault);
+
+	if (fault->is_private)
+		return kvm_faultin_pfn_private(vcpu, fault);
+
 	async = false;
 	fault->pfn = __gfn_to_pfn_memslot(slot, fault->gfn, false, false, &async,
 					  fault->write, &fault->map_writable,
@@ -5599,6 +5652,9 @@ int noinline kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 err
 			return -EIO;
 	}
 
+	if (r == RET_PF_USER)
+		return 0;
+
 	if (r < 0)
 		return r;
 	if (r != RET_PF_EMULATE)
@@ -6452,7 +6508,8 @@ static bool kvm_mmu_zap_collapsible_spte(struct kvm *kvm,
 		 */
 		if (sp->role.direct &&
 		    sp->role.level < kvm_mmu_max_mapping_level(kvm, slot, sp->gfn,
-							       PG_LEVEL_NUM)) {
+							       PG_LEVEL_NUM,
+							       false)) {
 			kvm_zap_one_rmap_spte(kvm, rmap_head, sptep);
 
 			if (kvm_available_flush_tlb_with_range())
arch/x86/kvm/mmu/mmu_internal.h +13 -1
index dbaf6755c5a7..5ccf08183b00 100644
@@ -189,6 +189,7 @@ struct kvm_page_fault {
 
 	/* Derived from mmu and global state.  */
 	const bool is_tdp;
+	const bool is_private;
 	const bool nx_huge_page_workaround_enabled;
 
 	/*
@@ -237,6 +238,7 @@ int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault);
  * RET_PF_RETRY: let CPU fault again on the address.
  * RET_PF_EMULATE: mmio page fault, emulate the instruction directly.
  * RET_PF_INVALID: the spte is invalid, let the real page fault path update it.
+ * RET_PF_USER: need to exit to userspace to handle this fault.
  * RET_PF_FIXED: The faulting entry has been fixed.
  * RET_PF_SPURIOUS: The faulting entry was already fixed, e.g. by another vCPU.
  *
@@ -253,6 +255,7 @@ enum {
 	RET_PF_RETRY,
 	RET_PF_EMULATE,
 	RET_PF_INVALID,
+	RET_PF_USER,
 	RET_PF_FIXED,
 	RET_PF_SPURIOUS,
 };
@@ -310,7 +313,7 @@ static inline int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 
 int kvm_mmu_max_mapping_level(struct kvm *kvm,
 			      const struct kvm_memory_slot *slot, gfn_t gfn,
-			      int max_level);
+			      int max_level, bool is_private);
 void kvm_mmu_hugepage_adjust(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault);
 void disallowed_hugepage_adjust(struct kvm_page_fault *fault, u64 spte, int cur_level);
 
@@ -319,4 +322,13 @@ void *mmu_memory_cache_alloc(struct kvm_mmu_memory_cache *mc);
 void track_possible_nx_huge_page(struct kvm *kvm, struct kvm_mmu_page *sp);
 void untrack_possible_nx_huge_page(struct kvm *kvm, struct kvm_mmu_page *sp);
 
+#ifndef CONFIG_HAVE_KVM_RESTRICTED_MEM
+static inline int kvm_restricted_mem_get_pfn(struct kvm_memory_slot *slot,
+					gfn_t gfn, kvm_pfn_t *pfn, int *order)
+{
+	WARN_ON_ONCE(1);
+	return -EOPNOTSUPP;
+}
+#endif /* CONFIG_HAVE_KVM_RESTRICTED_MEM */
+
 #endif /* __KVM_X86_MMU_INTERNAL_H */
arch/x86/kvm/mmu/mmutrace.h +1 -0
index ae86820cef69..2d7555381955 100644
@@ -58,6 +58,7 @@ TRACE_DEFINE_ENUM(RET_PF_CONTINUE);
 TRACE_DEFINE_ENUM(RET_PF_RETRY);
 TRACE_DEFINE_ENUM(RET_PF_EMULATE);
 TRACE_DEFINE_ENUM(RET_PF_INVALID);
+TRACE_DEFINE_ENUM(RET_PF_USER);
 TRACE_DEFINE_ENUM(RET_PF_FIXED);
 TRACE_DEFINE_ENUM(RET_PF_SPURIOUS);
 
arch/x86/kvm/mmu/tdp_mmu.c +1 -1
index 771210ce5181..8ba1a4afc546 100644
@@ -1768,7 +1768,7 @@ static void zap_collapsible_spte_range(struct kvm *kvm,
 			continue;
 
 		max_mapping_level = kvm_mmu_max_mapping_level(kvm, slot,
-							      iter.gfn, PG_LEVEL_NUM);
+						iter.gfn, PG_LEVEL_NUM, false);
 		if (max_mapping_level < iter.level)
 			continue;
 
include/linux/kvm_host.h +30 -1
index 25099c94e770..153842bb33df 100644
@@ -2335,4 +2335,34 @@ static inline void kvm_arch_set_memory_attributes(struct kvm *kvm,
 }
 #endif /* __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES */
 
+#ifdef CONFIG_HAVE_KVM_MEMORY_ATTRIBUTES
+static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
+{
+	return xa_to_value(xa_load(&kvm->mem_attr_array, gfn)) &
+	       KVM_MEMORY_ATTRIBUTE_PRIVATE;
+}
+#else
+static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
+{
+	return false;
+}
+
+#endif /* CONFIG_HAVE_KVM_MEMORY_ATTRIBUTES */
+
+#ifdef CONFIG_HAVE_KVM_RESTRICTED_MEM
+static inline int kvm_restricted_mem_get_pfn(struct kvm_memory_slot *slot,
+					gfn_t gfn, kvm_pfn_t *pfn, int *order)
+{
+	int ret;
+	struct page *page;
+	pgoff_t index = gfn - slot->base_gfn +
+			(slot->restricted_offset >> PAGE_SHIFT);
+
+	ret = restrictedmem_get_page(slot->restricted_file, index,
+				     &page, order);
+	*pfn = page_to_pfn(page);
+	return ret;
+}
+#endif /* CONFIG_HAVE_KVM_RESTRICTED_MEM */
+
 #endif
--

2.25.1

author
#289 lore raw

Re: [PATCH v10 8/9] KVM: Handle page fault for private memory

Quoted 46 lines · Yuan Yao
Quoted 45 lines · Chao Peng

A KVM_MEM_PRIVATE memslot can include both fd-based private memory and hva-based shared memory. Architecture code (like TDX code) can tell whether the on-going fault is private or not. This patch adds a 'is_private' field to kvm_page_fault to indicate this and architecture code is expected to set it.

To handle page fault for such memslot, the handling logic is different depending on whether the fault is private or shared. KVM checks if 'is_private' matches the host's view of the page (maintained in mem_attr_array).
- For a successful match, private pfn is obtained with restrictedmem_get_page() and shared pfn is obtained with existing get_user_pages().
- For a failed match, KVM causes a KVM_EXIT_MEMORY_FAULT exit to userspace. Userspace then can convert memory between private/shared in host's view and retry the fault.

---
 arch/x86/kvm/mmu/mmu.c          | 63 +++++++++++++++++++++++++++++++--
 arch/x86/kvm/mmu/mmu_internal.h | 14 +++++++-
 arch/x86/kvm/mmu/mmutrace.h     |  1 +
 arch/x86/kvm/mmu/tdp_mmu.c      |  2 +-
 include/linux/kvm_host.h        | 30 ++++++++++++++++
 5 files changed, 105 insertions(+), 5 deletions(-)
arch/x86/kvm/mmu/mmu.c +1 -1
index 2190fd8c95c0..b1953ebc012e 100644
@@ -3058,7 +3058,7 @@ static int host_pfn_mapping_level(struct kvm *kvm, gfn_t gfn,
 
 int kvm_mmu_max_mapping_level(struct kvm *kvm,
 			      const struct kvm_memory_slot *slot, gfn_t gfn,
-			      int max_level)
+			      int max_level, bool is_private)
 {
 	struct kvm_lpage_info *linfo;
 	int host_level;
@@ -3070,6 +3070,9 @@ int kvm_mmu_max_mapping_level(struct kvm *kvm,
 			break;
 	}
 
Quoted 5 lines

arch/x86/kvm/mmu/mmu.c int kvm_mmu_max_mapping_level(struct kvm *kvm, · line 3074

Quoted 2 lines
arch/x86/kvm/mmu/mmu.c +2 -0
+	if (is_private)
+		return max_level;

lpage mixed information already saved, so is that possible to query info->disallow_lpage without care 'is_private' ?

Actually we already queried info->disallow_lpage just before this sentence. The check is needed because later in the function we call host_pfn_mapping_level() which is shared memory specific.

Thanks,
Chao

Quoted 217 lines
Quoted 217 lines
arch/x86/kvm/mmu/mmu.c +57 -2
+
 	if (max_level == PG_LEVEL_4K)
 		return PG_LEVEL_4K;
 
@@ -3098,7 +3101,8 @@ void kvm_mmu_hugepage_adjust(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault
 	 * level, which will be used to do precise, accurate accounting.
 	 */
 	fault->req_level = kvm_mmu_max_mapping_level(vcpu->kvm, slot,
-						     fault->gfn, fault->max_level);
+						     fault->gfn, fault->max_level,
+						     fault->is_private);
 	if (fault->req_level == PG_LEVEL_4K || fault->huge_page_disallowed)
 		return;
 
@@ -4178,6 +4182,49 @@ void kvm_arch_async_page_ready(struct kvm_vcpu *vcpu, struct kvm_async_pf *work)
 	kvm_mmu_do_page_fault(vcpu, work->cr2_or_gpa, 0, true);
 }
 
+static inline u8 order_to_level(int order)
+{
+	BUILD_BUG_ON(KVM_MAX_HUGEPAGE_LEVEL > PG_LEVEL_1G);
+
+	if (order >= KVM_HPAGE_GFN_SHIFT(PG_LEVEL_1G))
+		return PG_LEVEL_1G;
+
+	if (order >= KVM_HPAGE_GFN_SHIFT(PG_LEVEL_2M))
+		return PG_LEVEL_2M;
+
+	return PG_LEVEL_4K;
+}
+
+static int kvm_do_memory_fault_exit(struct kvm_vcpu *vcpu,
+				    struct kvm_page_fault *fault)
+{
+	vcpu->run->exit_reason = KVM_EXIT_MEMORY_FAULT;
+	if (fault->is_private)
+		vcpu->run->memory.flags = KVM_MEMORY_EXIT_FLAG_PRIVATE;
+	else
+		vcpu->run->memory.flags = 0;
+	vcpu->run->memory.gpa = fault->gfn << PAGE_SHIFT;
+	vcpu->run->memory.size = PAGE_SIZE;
+	return RET_PF_USER;
+}
+
+static int kvm_faultin_pfn_private(struct kvm_vcpu *vcpu,
+				   struct kvm_page_fault *fault)
+{
+	int order;
+	struct kvm_memory_slot *slot = fault->slot;
+
+	if (!kvm_slot_can_be_private(slot))
+		return kvm_do_memory_fault_exit(vcpu, fault);
+
+	if (kvm_restricted_mem_get_pfn(slot, fault->gfn, &fault->pfn, &order))
+		return RET_PF_RETRY;
+
+	fault->max_level = min(order_to_level(order), fault->max_level);
+	fault->map_writable = !(slot->flags & KVM_MEM_READONLY);
+	return RET_PF_CONTINUE;
+}
+
 static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
 {
 	struct kvm_memory_slot *slot = fault->slot;
@@ -4210,6 +4257,12 @@ static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
 			return RET_PF_EMULATE;
 	}
 
+	if (fault->is_private != kvm_mem_is_private(vcpu->kvm, fault->gfn))
+		return kvm_do_memory_fault_exit(vcpu, fault);
+
+	if (fault->is_private)
+		return kvm_faultin_pfn_private(vcpu, fault);
+
 	async = false;
 	fault->pfn = __gfn_to_pfn_memslot(slot, fault->gfn, false, false, &async,
 					  fault->write, &fault->map_writable,
@@ -5599,6 +5652,9 @@ int noinline kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 err
 			return -EIO;
 	}
 
+	if (r == RET_PF_USER)
+		return 0;
+
 	if (r < 0)
 		return r;
 	if (r != RET_PF_EMULATE)
@@ -6452,7 +6508,8 @@ static bool kvm_mmu_zap_collapsible_spte(struct kvm *kvm,
 		 */
 		if (sp->role.direct &&
 		    sp->role.level < kvm_mmu_max_mapping_level(kvm, slot, sp->gfn,
-							       PG_LEVEL_NUM)) {
+							       PG_LEVEL_NUM,
+							       false)) {
 			kvm_zap_one_rmap_spte(kvm, rmap_head, sptep);
 
 			if (kvm_available_flush_tlb_with_range())
arch/x86/kvm/mmu/mmu_internal.h +13 -1
index dbaf6755c5a7..5ccf08183b00 100644
@@ -189,6 +189,7 @@ struct kvm_page_fault {
 
 	/* Derived from mmu and global state.  */
 	const bool is_tdp;
+	const bool is_private;
 	const bool nx_huge_page_workaround_enabled;
 
 	/*
@@ -237,6 +238,7 @@ int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault);
  * RET_PF_RETRY: let CPU fault again on the address.
  * RET_PF_EMULATE: mmio page fault, emulate the instruction directly.
  * RET_PF_INVALID: the spte is invalid, let the real page fault path update it.
+ * RET_PF_USER: need to exit to userspace to handle this fault.
  * RET_PF_FIXED: The faulting entry has been fixed.
  * RET_PF_SPURIOUS: The faulting entry was already fixed, e.g. by another vCPU.
  *
@@ -253,6 +255,7 @@ enum {
 	RET_PF_RETRY,
 	RET_PF_EMULATE,
 	RET_PF_INVALID,
+	RET_PF_USER,
 	RET_PF_FIXED,
 	RET_PF_SPURIOUS,
 };
@@ -310,7 +313,7 @@ static inline int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 
 int kvm_mmu_max_mapping_level(struct kvm *kvm,
 			      const struct kvm_memory_slot *slot, gfn_t gfn,
-			      int max_level);
+			      int max_level, bool is_private);
 void kvm_mmu_hugepage_adjust(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault);
 void disallowed_hugepage_adjust(struct kvm_page_fault *fault, u64 spte, int cur_level);
 
@@ -319,4 +322,13 @@ void *mmu_memory_cache_alloc(struct kvm_mmu_memory_cache *mc);
 void track_possible_nx_huge_page(struct kvm *kvm, struct kvm_mmu_page *sp);
 void untrack_possible_nx_huge_page(struct kvm *kvm, struct kvm_mmu_page *sp);
 
+#ifndef CONFIG_HAVE_KVM_RESTRICTED_MEM
+static inline int kvm_restricted_mem_get_pfn(struct kvm_memory_slot *slot,
+					gfn_t gfn, kvm_pfn_t *pfn, int *order)
+{
+	WARN_ON_ONCE(1);
+	return -EOPNOTSUPP;
+}
+#endif /* CONFIG_HAVE_KVM_RESTRICTED_MEM */
+
 #endif /* __KVM_X86_MMU_INTERNAL_H */
arch/x86/kvm/mmu/mmutrace.h +1 -0
index ae86820cef69..2d7555381955 100644
@@ -58,6 +58,7 @@ TRACE_DEFINE_ENUM(RET_PF_CONTINUE);
 TRACE_DEFINE_ENUM(RET_PF_RETRY);
 TRACE_DEFINE_ENUM(RET_PF_EMULATE);
 TRACE_DEFINE_ENUM(RET_PF_INVALID);
+TRACE_DEFINE_ENUM(RET_PF_USER);
 TRACE_DEFINE_ENUM(RET_PF_FIXED);
 TRACE_DEFINE_ENUM(RET_PF_SPURIOUS);
 
arch/x86/kvm/mmu/tdp_mmu.c +1 -1
index 771210ce5181..8ba1a4afc546 100644
@@ -1768,7 +1768,7 @@ static void zap_collapsible_spte_range(struct kvm *kvm,
 			continue;
 
 		max_mapping_level = kvm_mmu_max_mapping_level(kvm, slot,
-							      iter.gfn, PG_LEVEL_NUM);
+						iter.gfn, PG_LEVEL_NUM, false);
 		if (max_mapping_level < iter.level)
 			continue;
 
include/linux/kvm_host.h +30 -1
index 25099c94e770..153842bb33df 100644
@@ -2335,4 +2335,34 @@ static inline void kvm_arch_set_memory_attributes(struct kvm *kvm,
 }
 #endif /* __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES */
 
+#ifdef CONFIG_HAVE_KVM_MEMORY_ATTRIBUTES
+static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
+{
+	return xa_to_value(xa_load(&kvm->mem_attr_array, gfn)) &
+	       KVM_MEMORY_ATTRIBUTE_PRIVATE;
+}
+#else
+static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
+{
+	return false;
+}
+
+#endif /* CONFIG_HAVE_KVM_MEMORY_ATTRIBUTES */
+
+#ifdef CONFIG_HAVE_KVM_RESTRICTED_MEM
+static inline int kvm_restricted_mem_get_pfn(struct kvm_memory_slot *slot,
+					gfn_t gfn, kvm_pfn_t *pfn, int *order)
+{
+	int ret;
+	struct page *page;
+	pgoff_t index = gfn - slot->base_gfn +
+			(slot->restricted_offset >> PAGE_SHIFT);
+
+	ret = restrictedmem_get_page(slot->restricted_file, index,
+				     &page, order);
+	*pfn = page_to_pfn(page);
+	return ret;
+}
+#endif /* CONFIG_HAVE_KVM_RESTRICTED_MEM */
+
 #endif
--

2.25.1

#290 lore raw

Re: [PATCH v10 8/9] KVM: Handle page fault for private memory

Quoted 53 lines · Chao Peng
Quoted 46 lines · Yuan Yao
Quoted 45 lines · Chao Peng

A KVM_MEM_PRIVATE memslot can include both fd-based private memory and hva-based shared memory. Architecture code (like TDX code) can tell whether the on-going fault is private or not. This patch adds a 'is_private' field to kvm_page_fault to indicate this and architecture code is expected to set it.

To handle page fault for such memslot, the handling logic is different depending on whether the fault is private or shared. KVM checks if 'is_private' matches the host's view of the page (maintained in mem_attr_array).
- For a successful match, private pfn is obtained with restrictedmem_get_page() and shared pfn is obtained with existing get_user_pages().
- For a failed match, KVM causes a KVM_EXIT_MEMORY_FAULT exit to userspace. Userspace then can convert memory between private/shared in host's view and retry the fault.

---
 arch/x86/kvm/mmu/mmu.c          | 63 +++++++++++++++++++++++++++++++--
 arch/x86/kvm/mmu/mmu_internal.h | 14 +++++++-
 arch/x86/kvm/mmu/mmutrace.h     |  1 +
 arch/x86/kvm/mmu/tdp_mmu.c      |  2 +-
 include/linux/kvm_host.h        | 30 ++++++++++++++++
 5 files changed, 105 insertions(+), 5 deletions(-)
arch/x86/kvm/mmu/mmu.c +1 -1
index 2190fd8c95c0..b1953ebc012e 100644
@@ -3058,7 +3058,7 @@ static int host_pfn_mapping_level(struct kvm *kvm, gfn_t gfn,
 
 int kvm_mmu_max_mapping_level(struct kvm *kvm,
 			      const struct kvm_memory_slot *slot, gfn_t gfn,
-			      int max_level)
+			      int max_level, bool is_private)
 {
 	struct kvm_lpage_info *linfo;
 	int host_level;
@@ -3070,6 +3070,9 @@ int kvm_mmu_max_mapping_level(struct kvm *kvm,
 			break;
 	}
 
Quoted 5 lines

arch/x86/kvm/mmu/mmu.c int kvm_mmu_max_mapping_level(struct kvm *kvm, · line 3074

Quoted 2 lines
arch/x86/kvm/mmu/mmu.c +2 -0
+	if (is_private)
+		return max_level;

lpage mixed information already saved, so is that possible to query info->disallow_lpage without care 'is_private' ?

Quoted 3 lines

Actually we already queried info->disallow_lpage just before this sentence. The check is needed because later in the function we call host_pfn_mapping_level() which is shared memory specific.

You're right. We can't get mapping level info for private page from host_pfn_mapping_level().

Quoted 220 lines

Thanks,
Chao

Quoted 217 lines
Quoted 217 lines
arch/x86/kvm/mmu/mmu.c +57 -2
+
 	if (max_level == PG_LEVEL_4K)
 		return PG_LEVEL_4K;
 
@@ -3098,7 +3101,8 @@ void kvm_mmu_hugepage_adjust(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault
 	 * level, which will be used to do precise, accurate accounting.
 	 */
 	fault->req_level = kvm_mmu_max_mapping_level(vcpu->kvm, slot,
-						     fault->gfn, fault->max_level);
+						     fault->gfn, fault->max_level,
+						     fault->is_private);
 	if (fault->req_level == PG_LEVEL_4K || fault->huge_page_disallowed)
 		return;
 
@@ -4178,6 +4182,49 @@ void kvm_arch_async_page_ready(struct kvm_vcpu *vcpu, struct kvm_async_pf *work)
 	kvm_mmu_do_page_fault(vcpu, work->cr2_or_gpa, 0, true);
 }
 
+static inline u8 order_to_level(int order)
+{
+	BUILD_BUG_ON(KVM_MAX_HUGEPAGE_LEVEL > PG_LEVEL_1G);
+
+	if (order >= KVM_HPAGE_GFN_SHIFT(PG_LEVEL_1G))
+		return PG_LEVEL_1G;
+
+	if (order >= KVM_HPAGE_GFN_SHIFT(PG_LEVEL_2M))
+		return PG_LEVEL_2M;
+
+	return PG_LEVEL_4K;
+}
+
+static int kvm_do_memory_fault_exit(struct kvm_vcpu *vcpu,
+				    struct kvm_page_fault *fault)
+{
+	vcpu->run->exit_reason = KVM_EXIT_MEMORY_FAULT;
+	if (fault->is_private)
+		vcpu->run->memory.flags = KVM_MEMORY_EXIT_FLAG_PRIVATE;
+	else
+		vcpu->run->memory.flags = 0;
+	vcpu->run->memory.gpa = fault->gfn << PAGE_SHIFT;
+	vcpu->run->memory.size = PAGE_SIZE;
+	return RET_PF_USER;
+}
+
+static int kvm_faultin_pfn_private(struct kvm_vcpu *vcpu,
+				   struct kvm_page_fault *fault)
+{
+	int order;
+	struct kvm_memory_slot *slot = fault->slot;
+
+	if (!kvm_slot_can_be_private(slot))
+		return kvm_do_memory_fault_exit(vcpu, fault);
+
+	if (kvm_restricted_mem_get_pfn(slot, fault->gfn, &fault->pfn, &order))
+		return RET_PF_RETRY;
+
+	fault->max_level = min(order_to_level(order), fault->max_level);
+	fault->map_writable = !(slot->flags & KVM_MEM_READONLY);
+	return RET_PF_CONTINUE;
+}
+
 static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
 {
 	struct kvm_memory_slot *slot = fault->slot;
@@ -4210,6 +4257,12 @@ static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
 			return RET_PF_EMULATE;
 	}
 
+	if (fault->is_private != kvm_mem_is_private(vcpu->kvm, fault->gfn))
+		return kvm_do_memory_fault_exit(vcpu, fault);
+
+	if (fault->is_private)
+		return kvm_faultin_pfn_private(vcpu, fault);
+
 	async = false;
 	fault->pfn = __gfn_to_pfn_memslot(slot, fault->gfn, false, false, &async,
 					  fault->write, &fault->map_writable,
@@ -5599,6 +5652,9 @@ int noinline kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 err
 			return -EIO;
 	}
 
+	if (r == RET_PF_USER)
+		return 0;
+
 	if (r < 0)
 		return r;
 	if (r != RET_PF_EMULATE)
@@ -6452,7 +6508,8 @@ static bool kvm_mmu_zap_collapsible_spte(struct kvm *kvm,
 		 */
 		if (sp->role.direct &&
 		    sp->role.level < kvm_mmu_max_mapping_level(kvm, slot, sp->gfn,
-							       PG_LEVEL_NUM)) {
+							       PG_LEVEL_NUM,
+							       false)) {
 			kvm_zap_one_rmap_spte(kvm, rmap_head, sptep);
 
 			if (kvm_available_flush_tlb_with_range())
arch/x86/kvm/mmu/mmu_internal.h +13 -1
index dbaf6755c5a7..5ccf08183b00 100644
@@ -189,6 +189,7 @@ struct kvm_page_fault {
 
 	/* Derived from mmu and global state.  */
 	const bool is_tdp;
+	const bool is_private;
 	const bool nx_huge_page_workaround_enabled;
 
 	/*
@@ -237,6 +238,7 @@ int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault);
  * RET_PF_RETRY: let CPU fault again on the address.
  * RET_PF_EMULATE: mmio page fault, emulate the instruction directly.
  * RET_PF_INVALID: the spte is invalid, let the real page fault path update it.
+ * RET_PF_USER: need to exit to userspace to handle this fault.
  * RET_PF_FIXED: The faulting entry has been fixed.
  * RET_PF_SPURIOUS: The faulting entry was already fixed, e.g. by another vCPU.
  *
@@ -253,6 +255,7 @@ enum {
 	RET_PF_RETRY,
 	RET_PF_EMULATE,
 	RET_PF_INVALID,
+	RET_PF_USER,
 	RET_PF_FIXED,
 	RET_PF_SPURIOUS,
 };
@@ -310,7 +313,7 @@ static inline int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 
 int kvm_mmu_max_mapping_level(struct kvm *kvm,
 			      const struct kvm_memory_slot *slot, gfn_t gfn,
-			      int max_level);
+			      int max_level, bool is_private);
 void kvm_mmu_hugepage_adjust(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault);
 void disallowed_hugepage_adjust(struct kvm_page_fault *fault, u64 spte, int cur_level);
 
@@ -319,4 +322,13 @@ void *mmu_memory_cache_alloc(struct kvm_mmu_memory_cache *mc);
 void track_possible_nx_huge_page(struct kvm *kvm, struct kvm_mmu_page *sp);
 void untrack_possible_nx_huge_page(struct kvm *kvm, struct kvm_mmu_page *sp);
 
+#ifndef CONFIG_HAVE_KVM_RESTRICTED_MEM
+static inline int kvm_restricted_mem_get_pfn(struct kvm_memory_slot *slot,
+					gfn_t gfn, kvm_pfn_t *pfn, int *order)
+{
+	WARN_ON_ONCE(1);
+	return -EOPNOTSUPP;
+}
+#endif /* CONFIG_HAVE_KVM_RESTRICTED_MEM */
+
 #endif /* __KVM_X86_MMU_INTERNAL_H */
arch/x86/kvm/mmu/mmutrace.h +1 -0
index ae86820cef69..2d7555381955 100644
@@ -58,6 +58,7 @@ TRACE_DEFINE_ENUM(RET_PF_CONTINUE);
 TRACE_DEFINE_ENUM(RET_PF_RETRY);
 TRACE_DEFINE_ENUM(RET_PF_EMULATE);
 TRACE_DEFINE_ENUM(RET_PF_INVALID);
+TRACE_DEFINE_ENUM(RET_PF_USER);
 TRACE_DEFINE_ENUM(RET_PF_FIXED);
 TRACE_DEFINE_ENUM(RET_PF_SPURIOUS);
 
arch/x86/kvm/mmu/tdp_mmu.c +1 -1
index 771210ce5181..8ba1a4afc546 100644
@@ -1768,7 +1768,7 @@ static void zap_collapsible_spte_range(struct kvm *kvm,
 			continue;
 
 		max_mapping_level = kvm_mmu_max_mapping_level(kvm, slot,
-							      iter.gfn, PG_LEVEL_NUM);
+						iter.gfn, PG_LEVEL_NUM, false);
 		if (max_mapping_level < iter.level)
 			continue;
 
include/linux/kvm_host.h +30 -1
index 25099c94e770..153842bb33df 100644
@@ -2335,4 +2335,34 @@ static inline void kvm_arch_set_memory_attributes(struct kvm *kvm,
 }
 #endif /* __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES */
 
+#ifdef CONFIG_HAVE_KVM_MEMORY_ATTRIBUTES
+static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
+{
+	return xa_to_value(xa_load(&kvm->mem_attr_array, gfn)) &
+	       KVM_MEMORY_ATTRIBUTE_PRIVATE;
+}
+#else
+static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
+{
+	return false;
+}
+
+#endif /* CONFIG_HAVE_KVM_MEMORY_ATTRIBUTES */
+
+#ifdef CONFIG_HAVE_KVM_RESTRICTED_MEM
+static inline int kvm_restricted_mem_get_pfn(struct kvm_memory_slot *slot,
+					gfn_t gfn, kvm_pfn_t *pfn, int *order)
+{
+	int ret;
+	struct page *page;
+	pgoff_t index = gfn - slot->base_gfn +
+			(slot->restricted_offset >> PAGE_SHIFT);
+
+	ret = restrictedmem_get_page(slot->restricted_file, index,
+				     &page, order);
+	*pfn = page_to_pfn(page);
+	return ret;
+}
+#endif /* CONFIG_HAVE_KVM_RESTRICTED_MEM */
+
 #endif
--

2.25.1

Reviewed-by Tested-by on v10 8/9
#291 lore raw

Re: [PATCH v10 8/9] KVM: Handle page fault for private memory

In reply to Chao Peng

Hi,

Quoted 78 lines · Chao Peng

A KVM_MEM_PRIVATE memslot can include both fd-based private memory and hva-based shared memory. Architecture code (like TDX code) can tell whether the on-going fault is private or not. This patch adds a 'is_private' field to kvm_page_fault to indicate this and architecture code is expected to set it.

To handle page fault for such memslot, the handling logic is different depending on whether the fault is private or shared. KVM checks if 'is_private' matches the host's view of the page (maintained in mem_attr_array).
- For a successful match, private pfn is obtained with restrictedmem_get_page() and shared pfn is obtained with existing get_user_pages().
- For a failed match, KVM causes a KVM_EXIT_MEMORY_FAULT exit to userspace. Userspace then can convert memory between private/shared in host's view and retry the fault.

---
 arch/x86/kvm/mmu/mmu.c          | 63 +++++++++++++++++++++++++++++++--
 arch/x86/kvm/mmu/mmu_internal.h | 14 +++++++-
 arch/x86/kvm/mmu/mmutrace.h     |  1 +
 arch/x86/kvm/mmu/tdp_mmu.c      |  2 +-
 include/linux/kvm_host.h        | 30 ++++++++++++++++
 5 files changed, 105 insertions(+), 5 deletions(-)
arch/x86/kvm/mmu/mmu.c +18 -2
index 2190fd8c95c0..b1953ebc012e 100644
@@ -3058,7 +3058,7 @@ static int host_pfn_mapping_level(struct kvm *kvm, gfn_t gfn,
 
 int kvm_mmu_max_mapping_level(struct kvm *kvm,
                              const struct kvm_memory_slot *slot, gfn_t gfn,
-                             int max_level)
+                             int max_level, bool is_private)
 {
        struct kvm_lpage_info *linfo;
        int host_level;
@@ -3070,6 +3070,9 @@ int kvm_mmu_max_mapping_level(struct kvm *kvm,
                        break;
        }
 
+       if (is_private)
+               return max_level;
+
        if (max_level == PG_LEVEL_4K)
                return PG_LEVEL_4K;
 
@@ -3098,7 +3101,8 @@ void kvm_mmu_hugepage_adjust(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault
         * level, which will be used to do precise, accurate accounting.
         */
        fault->req_level = kvm_mmu_max_mapping_level(vcpu->kvm, slot,
-                                                    fault->gfn, fault->max_level);
+                                                    fault->gfn, fault->max_level,
+                                                    fault->is_private);
        if (fault->req_level == PG_LEVEL_4K || fault->huge_page_disallowed)
                return;
 
@@ -4178,6 +4182,49 @@ void kvm_arch_async_page_ready(struct kvm_vcpu *vcpu, struct kvm_async_pf *work)
        kvm_mmu_do_page_fault(vcpu, work->cr2_or_gpa, 0, true);
 }
 
+static inline u8 order_to_level(int order)
+{
+       BUILD_BUG_ON(KVM_MAX_HUGEPAGE_LEVEL > PG_LEVEL_1G);
+
+       if (order >= KVM_HPAGE_GFN_SHIFT(PG_LEVEL_1G))
+               return PG_LEVEL_1G;
+
+       if (order >= KVM_HPAGE_GFN_SHIFT(PG_LEVEL_2M))
+               return PG_LEVEL_2M;
+
+       return PG_LEVEL_4K;
+}

arch/x86/kvm/mmu/mmu.c void kvm_arch_async_page_ready(struct kvm_vcpu *vcpu, struct kvm_async_pf *work) · line 4206

Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +10 -0
+
+static int kvm_do_memory_fault_exit(struct kvm_vcpu *vcpu,
+                                   struct kvm_page_fault *fault)
+{
+       vcpu->run->exit_reason = KVM_EXIT_MEMORY_FAULT;
+       if (fault->is_private)
+               vcpu->run->memory.flags = KVM_MEMORY_EXIT_FLAG_PRIVATE;
+       else
+               vcpu->run->memory.flags = 0;
+       vcpu->run->memory.gpa = fault->gfn << PAGE_SHIFT;

nit: As in previous patches, use helpers (for this and other similar shifts in this patch)?

Quoted 173 lines
arch/x86/kvm/mmu/mmu.c +32 -1
+       vcpu->run->memory.size = PAGE_SIZE;
+       return RET_PF_USER;
+}
+
+static int kvm_faultin_pfn_private(struct kvm_vcpu *vcpu,
+                                  struct kvm_page_fault *fault)
+{
+       int order;
+       struct kvm_memory_slot *slot = fault->slot;
+
+       if (!kvm_slot_can_be_private(slot))
+               return kvm_do_memory_fault_exit(vcpu, fault);
+
+       if (kvm_restricted_mem_get_pfn(slot, fault->gfn, &fault->pfn, &order))
+               return RET_PF_RETRY;
+
+       fault->max_level = min(order_to_level(order), fault->max_level);
+       fault->map_writable = !(slot->flags & KVM_MEM_READONLY);
+       return RET_PF_CONTINUE;
+}
+
 static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
 {
        struct kvm_memory_slot *slot = fault->slot;
@@ -4210,6 +4257,12 @@ static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
                        return RET_PF_EMULATE;
        }
 
+       if (fault->is_private != kvm_mem_is_private(vcpu->kvm, fault->gfn))
+               return kvm_do_memory_fault_exit(vcpu, fault);
+
+       if (fault->is_private)
+               return kvm_faultin_pfn_private(vcpu, fault);
+
        async = false;
        fault->pfn = __gfn_to_pfn_memslot(slot, fault->gfn, false, false, &async,
                                          fault->write, &fault->map_writable,
@@ -5599,6 +5652,9 @@ int noinline kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 err
                        return -EIO;
        }
 
+       if (r == RET_PF_USER)
+               return 0;
+
        if (r < 0)
                return r;
        if (r != RET_PF_EMULATE)
@@ -6452,7 +6508,8 @@ static bool kvm_mmu_zap_collapsible_spte(struct kvm *kvm,
                 */
                if (sp->role.direct &&
                    sp->role.level < kvm_mmu_max_mapping_level(kvm, slot, sp->gfn,
-                                                              PG_LEVEL_NUM)) {
+                                                              PG_LEVEL_NUM,
+                                                              false)) {
                        kvm_zap_one_rmap_spte(kvm, rmap_head, sptep);
 
                        if (kvm_available_flush_tlb_with_range())
arch/x86/kvm/mmu/mmu_internal.h +13 -1
index dbaf6755c5a7..5ccf08183b00 100644
@@ -189,6 +189,7 @@ struct kvm_page_fault {
 
        /* Derived from mmu and global state.  */
        const bool is_tdp;
+       const bool is_private;
        const bool nx_huge_page_workaround_enabled;
 
        /*
@@ -237,6 +238,7 @@ int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault);
  * RET_PF_RETRY: let CPU fault again on the address.
  * RET_PF_EMULATE: mmio page fault, emulate the instruction directly.
  * RET_PF_INVALID: the spte is invalid, let the real page fault path update it.
+ * RET_PF_USER: need to exit to userspace to handle this fault.
  * RET_PF_FIXED: The faulting entry has been fixed.
  * RET_PF_SPURIOUS: The faulting entry was already fixed, e.g. by another vCPU.
  *
@@ -253,6 +255,7 @@ enum {
        RET_PF_RETRY,
        RET_PF_EMULATE,
        RET_PF_INVALID,
+       RET_PF_USER,
        RET_PF_FIXED,
        RET_PF_SPURIOUS,
 };
@@ -310,7 +313,7 @@ static inline int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 
 int kvm_mmu_max_mapping_level(struct kvm *kvm,
                              const struct kvm_memory_slot *slot, gfn_t gfn,
-                             int max_level);
+                             int max_level, bool is_private);
 void kvm_mmu_hugepage_adjust(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault);
 void disallowed_hugepage_adjust(struct kvm_page_fault *fault, u64 spte, int cur_level);
 
@@ -319,4 +322,13 @@ void *mmu_memory_cache_alloc(struct kvm_mmu_memory_cache *mc);
 void track_possible_nx_huge_page(struct kvm *kvm, struct kvm_mmu_page *sp);
 void untrack_possible_nx_huge_page(struct kvm *kvm, struct kvm_mmu_page *sp);
 
+#ifndef CONFIG_HAVE_KVM_RESTRICTED_MEM
+static inline int kvm_restricted_mem_get_pfn(struct kvm_memory_slot *slot,
+                                       gfn_t gfn, kvm_pfn_t *pfn, int *order)
+{
+       WARN_ON_ONCE(1);
+       return -EOPNOTSUPP;
+}
+#endif /* CONFIG_HAVE_KVM_RESTRICTED_MEM */
+
 #endif /* __KVM_X86_MMU_INTERNAL_H */
arch/x86/kvm/mmu/mmutrace.h +1 -0
index ae86820cef69..2d7555381955 100644
@@ -58,6 +58,7 @@ TRACE_DEFINE_ENUM(RET_PF_CONTINUE);
 TRACE_DEFINE_ENUM(RET_PF_RETRY);
 TRACE_DEFINE_ENUM(RET_PF_EMULATE);
 TRACE_DEFINE_ENUM(RET_PF_INVALID);
+TRACE_DEFINE_ENUM(RET_PF_USER);
 TRACE_DEFINE_ENUM(RET_PF_FIXED);
 TRACE_DEFINE_ENUM(RET_PF_SPURIOUS);
 
arch/x86/kvm/mmu/tdp_mmu.c +1 -1
index 771210ce5181..8ba1a4afc546 100644
@@ -1768,7 +1768,7 @@ static void zap_collapsible_spte_range(struct kvm *kvm,
                        continue;
 
                max_mapping_level = kvm_mmu_max_mapping_level(kvm, slot,
-                                                             iter.gfn, PG_LEVEL_NUM);
+                                               iter.gfn, PG_LEVEL_NUM, false);
                if (max_mapping_level < iter.level)
                        continue;
 
include/linux/kvm_host.h +30 -1
index 25099c94e770..153842bb33df 100644
@@ -2335,4 +2335,34 @@ static inline void kvm_arch_set_memory_attributes(struct kvm *kvm,
 }
 #endif /* __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES */
 
+#ifdef CONFIG_HAVE_KVM_MEMORY_ATTRIBUTES
+static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
+{
+       return xa_to_value(xa_load(&kvm->mem_attr_array, gfn)) &
+              KVM_MEMORY_ATTRIBUTE_PRIVATE;
+}
+#else
+static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
+{
+       return false;
+}
+
+#endif /* CONFIG_HAVE_KVM_MEMORY_ATTRIBUTES */
+
+#ifdef CONFIG_HAVE_KVM_RESTRICTED_MEM
+static inline int kvm_restricted_mem_get_pfn(struct kvm_memory_slot *slot,
+                                       gfn_t gfn, kvm_pfn_t *pfn, int *order)
+{
+       int ret;
+       struct page *page;
+       pgoff_t index = gfn - slot->base_gfn +
+                       (slot->restricted_offset >> PAGE_SHIFT);
+
+       ret = restrictedmem_get_page(slot->restricted_file, index,
+                                    &page, order);
+       *pfn = page_to_pfn(page);
+       return ret;
+}
+#endif /* CONFIG_HAVE_KVM_RESTRICTED_MEM */
+
 #endif
--
Quoted 2 lines

2.25.1

With my limited understanding of x86 code:

The common code in kvm_host.h was used in the port to arm64, and the x86 fault handling code was used as a guide to how it should be done in pKVM (with similar code added there). So with these caveats in mind:

Cheers,
/fuad

author
#292 lore raw

Re: [PATCH v10 8/9] KVM: Handle page fault for private memory

Quoted 86 lines · Fuad Tabba

Hi,

Quoted 83 lines · Chao Peng

A KVM_MEM_PRIVATE memslot can include both fd-based private memory and hva-based shared memory. Architecture code (like TDX code) can tell whether the on-going fault is private or not. This patch adds a 'is_private' field to kvm_page_fault to indicate this and architecture code is expected to set it.

To handle page fault for such memslot, the handling logic is different depending on whether the fault is private or shared. KVM checks if 'is_private' matches the host's view of the page (maintained in mem_attr_array).
- For a successful match, private pfn is obtained with restrictedmem_get_page() and shared pfn is obtained with existing get_user_pages().
- For a failed match, KVM causes a KVM_EXIT_MEMORY_FAULT exit to userspace. Userspace then can convert memory between private/shared in host's view and retry the fault.

---
 arch/x86/kvm/mmu/mmu.c          | 63 +++++++++++++++++++++++++++++++--
 arch/x86/kvm/mmu/mmu_internal.h | 14 +++++++-
 arch/x86/kvm/mmu/mmutrace.h     |  1 +
 arch/x86/kvm/mmu/tdp_mmu.c      |  2 +-
 include/linux/kvm_host.h        | 30 ++++++++++++++++
 5 files changed, 105 insertions(+), 5 deletions(-)
arch/x86/kvm/mmu/mmu.c +23 -2
index 2190fd8c95c0..b1953ebc012e 100644
@@ -3058,7 +3058,7 @@ static int host_pfn_mapping_level(struct kvm *kvm, gfn_t gfn,
 
 int kvm_mmu_max_mapping_level(struct kvm *kvm,
                              const struct kvm_memory_slot *slot, gfn_t gfn,
-                             int max_level)
+                             int max_level, bool is_private)
 {
        struct kvm_lpage_info *linfo;
        int host_level;
@@ -3070,6 +3070,9 @@ int kvm_mmu_max_mapping_level(struct kvm *kvm,
                        break;
        }
 
+       if (is_private)
+               return max_level;
+
        if (max_level == PG_LEVEL_4K)
                return PG_LEVEL_4K;
 
@@ -3098,7 +3101,8 @@ void kvm_mmu_hugepage_adjust(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault
         * level, which will be used to do precise, accurate accounting.
         */
        fault->req_level = kvm_mmu_max_mapping_level(vcpu->kvm, slot,
-                                                    fault->gfn, fault->max_level);
+                                                    fault->gfn, fault->max_level,
+                                                    fault->is_private);
        if (fault->req_level == PG_LEVEL_4K || fault->huge_page_disallowed)
                return;
 
@@ -4178,6 +4182,49 @@ void kvm_arch_async_page_ready(struct kvm_vcpu *vcpu, struct kvm_async_pf *work)
        kvm_mmu_do_page_fault(vcpu, work->cr2_or_gpa, 0, true);
 }
 
+static inline u8 order_to_level(int order)
+{
+       BUILD_BUG_ON(KVM_MAX_HUGEPAGE_LEVEL > PG_LEVEL_1G);
+
+       if (order >= KVM_HPAGE_GFN_SHIFT(PG_LEVEL_1G))
+               return PG_LEVEL_1G;
+
+       if (order >= KVM_HPAGE_GFN_SHIFT(PG_LEVEL_2M))
+               return PG_LEVEL_2M;
+
+       return PG_LEVEL_4K;
+}
+
+static int kvm_do_memory_fault_exit(struct kvm_vcpu *vcpu,
+                                   struct kvm_page_fault *fault)
+{
+       vcpu->run->exit_reason = KVM_EXIT_MEMORY_FAULT;
Quoted 8 lines

arch/x86/kvm/mmu/mmu.c void kvm_arch_async_page_ready(struct kvm_vcpu *vcpu, struct kvm_async_pf *work) · line 4206

Quoted 5 lines
arch/x86/kvm/mmu/mmu.c +5 -0
+       if (fault->is_private)
+               vcpu->run->memory.flags = KVM_MEMORY_EXIT_FLAG_PRIVATE;
+       else
+               vcpu->run->memory.flags = 0;
+       vcpu->run->memory.gpa = fault->gfn << PAGE_SHIFT;

nit: As in previous patches, use helpers (for this and other similar shifts in this patch)?

Agreed.

Quoted 188 lines
Quoted 174 lines
arch/x86/kvm/mmu/mmu.c +32 -1
+       vcpu->run->memory.size = PAGE_SIZE;
+       return RET_PF_USER;
+}
+
+static int kvm_faultin_pfn_private(struct kvm_vcpu *vcpu,
+                                  struct kvm_page_fault *fault)
+{
+       int order;
+       struct kvm_memory_slot *slot = fault->slot;
+
+       if (!kvm_slot_can_be_private(slot))
+               return kvm_do_memory_fault_exit(vcpu, fault);
+
+       if (kvm_restricted_mem_get_pfn(slot, fault->gfn, &fault->pfn, &order))
+               return RET_PF_RETRY;
+
+       fault->max_level = min(order_to_level(order), fault->max_level);
+       fault->map_writable = !(slot->flags & KVM_MEM_READONLY);
+       return RET_PF_CONTINUE;
+}
+
 static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
 {
        struct kvm_memory_slot *slot = fault->slot;
@@ -4210,6 +4257,12 @@ static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
                        return RET_PF_EMULATE;
        }
 
+       if (fault->is_private != kvm_mem_is_private(vcpu->kvm, fault->gfn))
+               return kvm_do_memory_fault_exit(vcpu, fault);
+
+       if (fault->is_private)
+               return kvm_faultin_pfn_private(vcpu, fault);
+
        async = false;
        fault->pfn = __gfn_to_pfn_memslot(slot, fault->gfn, false, false, &async,
                                          fault->write, &fault->map_writable,
@@ -5599,6 +5652,9 @@ int noinline kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 err
                        return -EIO;
        }
 
+       if (r == RET_PF_USER)
+               return 0;
+
        if (r < 0)
                return r;
        if (r != RET_PF_EMULATE)
@@ -6452,7 +6508,8 @@ static bool kvm_mmu_zap_collapsible_spte(struct kvm *kvm,
                 */
                if (sp->role.direct &&
                    sp->role.level < kvm_mmu_max_mapping_level(kvm, slot, sp->gfn,
-                                                              PG_LEVEL_NUM)) {
+                                                              PG_LEVEL_NUM,
+                                                              false)) {
                        kvm_zap_one_rmap_spte(kvm, rmap_head, sptep);
 
                        if (kvm_available_flush_tlb_with_range())
arch/x86/kvm/mmu/mmu_internal.h +13 -1
index dbaf6755c5a7..5ccf08183b00 100644
@@ -189,6 +189,7 @@ struct kvm_page_fault {
 
        /* Derived from mmu and global state.  */
        const bool is_tdp;
+       const bool is_private;
        const bool nx_huge_page_workaround_enabled;
 
        /*
@@ -237,6 +238,7 @@ int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault);
  * RET_PF_RETRY: let CPU fault again on the address.
  * RET_PF_EMULATE: mmio page fault, emulate the instruction directly.
  * RET_PF_INVALID: the spte is invalid, let the real page fault path update it.
+ * RET_PF_USER: need to exit to userspace to handle this fault.
  * RET_PF_FIXED: The faulting entry has been fixed.
  * RET_PF_SPURIOUS: The faulting entry was already fixed, e.g. by another vCPU.
  *
@@ -253,6 +255,7 @@ enum {
        RET_PF_RETRY,
        RET_PF_EMULATE,
        RET_PF_INVALID,
+       RET_PF_USER,
        RET_PF_FIXED,
        RET_PF_SPURIOUS,
 };
@@ -310,7 +313,7 @@ static inline int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 
 int kvm_mmu_max_mapping_level(struct kvm *kvm,
                              const struct kvm_memory_slot *slot, gfn_t gfn,
-                             int max_level);
+                             int max_level, bool is_private);
 void kvm_mmu_hugepage_adjust(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault);
 void disallowed_hugepage_adjust(struct kvm_page_fault *fault, u64 spte, int cur_level);
 
@@ -319,4 +322,13 @@ void *mmu_memory_cache_alloc(struct kvm_mmu_memory_cache *mc);
 void track_possible_nx_huge_page(struct kvm *kvm, struct kvm_mmu_page *sp);
 void untrack_possible_nx_huge_page(struct kvm *kvm, struct kvm_mmu_page *sp);
 
+#ifndef CONFIG_HAVE_KVM_RESTRICTED_MEM
+static inline int kvm_restricted_mem_get_pfn(struct kvm_memory_slot *slot,
+                                       gfn_t gfn, kvm_pfn_t *pfn, int *order)
+{
+       WARN_ON_ONCE(1);
+       return -EOPNOTSUPP;
+}
+#endif /* CONFIG_HAVE_KVM_RESTRICTED_MEM */
+
 #endif /* __KVM_X86_MMU_INTERNAL_H */
arch/x86/kvm/mmu/mmutrace.h +1 -0
index ae86820cef69..2d7555381955 100644
@@ -58,6 +58,7 @@ TRACE_DEFINE_ENUM(RET_PF_CONTINUE);
 TRACE_DEFINE_ENUM(RET_PF_RETRY);
 TRACE_DEFINE_ENUM(RET_PF_EMULATE);
 TRACE_DEFINE_ENUM(RET_PF_INVALID);
+TRACE_DEFINE_ENUM(RET_PF_USER);
 TRACE_DEFINE_ENUM(RET_PF_FIXED);
 TRACE_DEFINE_ENUM(RET_PF_SPURIOUS);
 
arch/x86/kvm/mmu/tdp_mmu.c +1 -1
index 771210ce5181..8ba1a4afc546 100644
@@ -1768,7 +1768,7 @@ static void zap_collapsible_spte_range(struct kvm *kvm,
                        continue;
 
                max_mapping_level = kvm_mmu_max_mapping_level(kvm, slot,
-                                                             iter.gfn, PG_LEVEL_NUM);
+                                               iter.gfn, PG_LEVEL_NUM, false);
                if (max_mapping_level < iter.level)
                        continue;
 
include/linux/kvm_host.h +30 -1
index 25099c94e770..153842bb33df 100644
@@ -2335,4 +2335,34 @@ static inline void kvm_arch_set_memory_attributes(struct kvm *kvm,
 }
 #endif /* __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES */
 
+#ifdef CONFIG_HAVE_KVM_MEMORY_ATTRIBUTES
+static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
+{
+       return xa_to_value(xa_load(&kvm->mem_attr_array, gfn)) &
+              KVM_MEMORY_ATTRIBUTE_PRIVATE;
+}
+#else
+static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
+{
+       return false;
+}
+
+#endif /* CONFIG_HAVE_KVM_MEMORY_ATTRIBUTES */
+
+#ifdef CONFIG_HAVE_KVM_RESTRICTED_MEM
+static inline int kvm_restricted_mem_get_pfn(struct kvm_memory_slot *slot,
+                                       gfn_t gfn, kvm_pfn_t *pfn, int *order)
+{
+       int ret;
+       struct page *page;
+       pgoff_t index = gfn - slot->base_gfn +
+                       (slot->restricted_offset >> PAGE_SHIFT);
+
+       ret = restrictedmem_get_page(slot->restricted_file, index,
+                                    &page, order);
+       *pfn = page_to_pfn(page);
+       return ret;
+}
+#endif /* CONFIG_HAVE_KVM_RESTRICTED_MEM */
+
 #endif
--
Quoted 2 lines

2.25.1

With my limited understanding of x86 code:

The common code in kvm_host.h was used in the port to arm64, and the x86 fault handling code was used as a guide to how it should be done in pKVM (with similar code added there). So with these caveats in mind:

Cheers,
/fuad

#293 lore raw

Re: [PATCH v10 8/9] KVM: Handle page fault for private memory

In reply to Chao Peng
Quoted 11 lines · Chao Peng
arch/x86/kvm/mmu/mmu.c +3 -0
@@ -5599,6 +5652,9 @@ int noinline kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 err
 			return -EIO;
 	}
 
+	if (r == RET_PF_USER)
+		return 0;
+
 	if (r < 0)
 		return r;
 	if (r != RET_PF_EMULATE)

arch/x86/kvm/mmu/mmu.c static bool kvm_mmu_zap_collapsible_spte(struct kvm *kvm, · line 6512

Quoted 6 lines
arch/x86/kvm/mmu/mmu.c +2 -1
@@ -6452,7 +6508,8 @@ static bool kvm_mmu_zap_collapsible_spte(struct kvm *kvm,
 		 */
 		if (sp->role.direct &&
 		    sp->role.level < kvm_mmu_max_mapping_level(kvm, slot, sp->gfn,
-							       PG_LEVEL_NUM)) {
+							       PG_LEVEL_NUM,
+							       false)) {

Passing %false is incorrect. It might not cause problems because KVM currently doesn't allowing modifying private memslots (that likely needs to change to allow dirty logging), but it's wrong since nothing guarantees KVM is operating on SPTEs for shared memory.

One option would be to take the patches from the TDX series that add a "private" flag to the shadow page role, but I'd rather not add the role until it's truly necessary.

For now, I think we can do this without impacting performance of guests that don't support private memory.

int kvm_mmu_max_mapping_level(struct kvm *kvm,
const struct kvm_memory_slot *slot, gfn_t gfn, int max_level)

{
	bool is_private = kvm_slot_can_be_private(slot) &&
			  kvm_mem_is_private(kvm, gfn);

	return __kvm_mmu_max_mapping_level(kvm, slot, gfn, max_level, is_private);
}

include/linux/kvm_host.h static inline void kvm_arch_set_memory_attributes(struct kvm *kvm, · line 2340

Quoted 11 lines
include/linux/kvm_host.h +3 -0
index 25099c94e770..153842bb33df 100644
@@ -2335,4 +2335,34 @@ static inline void kvm_arch_set_memory_attributes(struct kvm *kvm,
 }
 #endif /* __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES */
 
+#ifdef CONFIG_HAVE_KVM_MEMORY_ATTRIBUTES
+static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
+{

This code, i.e. the generic KVM changes, belongs in a separate patch. It'll be small, but I want to separate x86's page fault changes from the restrictedmem support adding to common KVM.

This should also short-circuit based on CONFIG_HAVE_KVM_RESTRICTED_MEM, though I would name that CONFIG_KVM_PRIVATE_MEMORY since in KVM's world, it's all about private vs. shared at this time.

Quoted 12 lines
include/linux/kvm_host.h +12 -0
+	return xa_to_value(xa_load(&kvm->mem_attr_array, gfn)) &
+	       KVM_MEMORY_ATTRIBUTE_PRIVATE;
+}
+#else
+static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
+{
+	return false;
+}
+
+#endif /* CONFIG_HAVE_KVM_MEMORY_ATTRIBUTES */
+
+#ifdef CONFIG_HAVE_KVM_RESTRICTED_MEM

include/linux/kvm_host.h static inline void kvm_arch_set_memory_attributes(struct kvm *kvm, · line 2362

Quoted 10 lines
include/linux/kvm_host.h +10 -0
+static inline int kvm_restricted_mem_get_pfn(struct kvm_memory_slot *slot,
+					gfn_t gfn, kvm_pfn_t *pfn, int *order)
+{
+	int ret;
+	struct page *page;
+	pgoff_t index = gfn - slot->base_gfn +
+			(slot->restricted_offset >> PAGE_SHIFT);
+
+	ret = restrictedmem_get_page(slot->restricted_file, index,
+				     &page, order);

This needs handle errors. If "ret" is non-zero, "page" is garbage.

Quoted 9 lines
include/linux/kvm_host.h +5 -0
+	*pfn = page_to_pfn(page);
+	return ret;
+}
+#endif /* CONFIG_HAVE_KVM_RESTRICTED_MEM */
+
 #endif
Signature · 1 line
2.25.1
author
#294 lore raw

PATCH v10 9/9 KVM: Enable and expose KVM_MEM_PRIVATE

In reply to Chao Peng

Register/unregister private memslot to fd-based memory backing store restrictedmem and implement the callbacks for restrictedmem_notifier:
- invalidate_start()/invalidate_end() to zap the existing memory mappings in the KVM page table.
- error() to request KVM_REQ_MEMORY_MCE and later exit to userspace with KVM_EXIT_SHUTDOWN.

Expose KVM_MEM_PRIVATE for memslot and KVM_MEMORY_ATTRIBUTE_PRIVATE for KVM_GET_SUPPORTED_MEMORY_ATTRIBUTES to userspace but either are controlled by kvm_arch_has_private_mem() which should be rewritten by architecture code.

---
 arch/x86/include/asm/kvm_host.h |   1 +
 arch/x86/kvm/x86.c              |  13 +++
 include/linux/kvm_host.h        |   3 +
 virt/kvm/kvm_main.c             | 179 +++++++++++++++++++++++++++++++-
 4 files changed, 191 insertions(+), 5 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -0
index 7772ab37ac89..27ef31133352 100644
@@ -114,6 +114,7 @@
 	KVM_ARCH_REQ_FLAGS(31, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
 #define KVM_REQ_HV_TLB_FLUSH \
 	KVM_ARCH_REQ_FLAGS(32, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
+#define KVM_REQ_MEMORY_MCE		KVM_ARCH_REQ(33)
 
 #define CR0_RESERVED_BITS                                               \
 	(~(unsigned long)(X86_CR0_PE | X86_CR0_MP | X86_CR0_EM | X86_CR0_TS \
arch/x86/kvm/x86.c +13 -0
index 5aefcff614d2..c67e22f3e2ee 100644
@@ -6587,6 +6587,13 @@ int kvm_arch_pm_notifier(struct kvm *kvm, unsigned long state)
 }
 #endif /* CONFIG_HAVE_KVM_PM_NOTIFIER */
 
+#ifdef CONFIG_HAVE_KVM_RESTRICTED_MEM
+void kvm_arch_memory_mce(struct kvm *kvm)
+{
+	kvm_make_all_cpus_request(kvm, KVM_REQ_MEMORY_MCE);
+}
+#endif
+
 static int kvm_vm_ioctl_get_clock(struct kvm *kvm, void __user *argp)
 {
 	struct kvm_clock_data data = { 0 };
@@ -10357,6 +10364,12 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 
 		if (kvm_check_request(KVM_REQ_UPDATE_CPU_DIRTY_LOGGING, vcpu))
 			static_call(kvm_x86_update_cpu_dirty_logging)(vcpu);
+
+		if (kvm_check_request(KVM_REQ_MEMORY_MCE, vcpu)) {
+			vcpu->run->exit_reason = KVM_EXIT_SHUTDOWN;
+			r = 0;
+			goto out;
+		}
 	}
 
 	if (kvm_check_request(KVM_REQ_EVENT, vcpu) || req_int_win ||
include/linux/kvm_host.h +3 -0
index 153842bb33df..f032d878e034 100644
@@ -590,6 +590,7 @@ struct kvm_memory_slot {
 	struct file *restricted_file;
 	loff_t restricted_offset;
 	struct restrictedmem_notifier notifier;
+	struct kvm *kvm;
 };
 
 static inline bool kvm_slot_can_be_private(const struct kvm_memory_slot *slot)
@@ -2363,6 +2364,8 @@ static inline int kvm_restricted_mem_get_pfn(struct kvm_memory_slot *slot,
 	*pfn = page_to_pfn(page);
 	return ret;
 }
+
+void kvm_arch_memory_mce(struct kvm *kvm);
 #endif /* CONFIG_HAVE_KVM_RESTRICTED_MEM */
 
 #endif
virt/kvm/kvm_main.c +174 -5

259 lines not on this page · raw message

Signature · 1 line
2.25.1
Tested-by on v10 9/9
#295 lore raw

Re: [PATCH v10 9/9] KVM: Enable and expose KVM_MEM_PRIVATE

Hi,

Quoted 13 lines · Chao Peng

Register/unregister private memslot to fd-based memory backing store restrictedmem and implement the callbacks for restrictedmem_notifier:
- invalidate_start()/invalidate_end() to zap the existing memory mappings in the KVM page table.
- error() to request KVM_REQ_MEMORY_MCE and later exit to userspace with KVM_EXIT_SHUTDOWN.

Expose KVM_MEM_PRIVATE for memslot and KVM_MEMORY_ATTRIBUTE_PRIVATE for KVM_GET_SUPPORTED_MEMORY_ATTRIBUTES to userspace but either are controlled by kvm_arch_has_private_mem() which should be rewritten by architecture code.

Quoted 4 lines

With the code to port it to pKVM/arm64:

Cheers,
/fuad

Quoted 348 lines
---
 arch/x86/include/asm/kvm_host.h |   1 +
 arch/x86/kvm/x86.c              |  13 +++
 include/linux/kvm_host.h        |   3 +
 virt/kvm/kvm_main.c             | 179 +++++++++++++++++++++++++++++++-
 4 files changed, 191 insertions(+), 5 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -0
index 7772ab37ac89..27ef31133352 100644
@@ -114,6 +114,7 @@
        KVM_ARCH_REQ_FLAGS(31, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
 #define KVM_REQ_HV_TLB_FLUSH \
        KVM_ARCH_REQ_FLAGS(32, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
+#define KVM_REQ_MEMORY_MCE             KVM_ARCH_REQ(33)
 
 #define CR0_RESERVED_BITS                                               \
        (~(unsigned long)(X86_CR0_PE | X86_CR0_MP | X86_CR0_EM | X86_CR0_TS \
arch/x86/kvm/x86.c +13 -0
index 5aefcff614d2..c67e22f3e2ee 100644
@@ -6587,6 +6587,13 @@ int kvm_arch_pm_notifier(struct kvm *kvm, unsigned long state)
 }
 #endif /* CONFIG_HAVE_KVM_PM_NOTIFIER */
 
+#ifdef CONFIG_HAVE_KVM_RESTRICTED_MEM
+void kvm_arch_memory_mce(struct kvm *kvm)
+{
+       kvm_make_all_cpus_request(kvm, KVM_REQ_MEMORY_MCE);
+}
+#endif
+
 static int kvm_vm_ioctl_get_clock(struct kvm *kvm, void __user *argp)
 {
        struct kvm_clock_data data = { 0 };
@@ -10357,6 +10364,12 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 
                if (kvm_check_request(KVM_REQ_UPDATE_CPU_DIRTY_LOGGING, vcpu))
                        static_call(kvm_x86_update_cpu_dirty_logging)(vcpu);
+
+               if (kvm_check_request(KVM_REQ_MEMORY_MCE, vcpu)) {
+                       vcpu->run->exit_reason = KVM_EXIT_SHUTDOWN;
+                       r = 0;
+                       goto out;
+               }
        }
 
        if (kvm_check_request(KVM_REQ_EVENT, vcpu) || req_int_win ||
include/linux/kvm_host.h +3 -0
index 153842bb33df..f032d878e034 100644
@@ -590,6 +590,7 @@ struct kvm_memory_slot {
        struct file *restricted_file;
        loff_t restricted_offset;
        struct restrictedmem_notifier notifier;
+       struct kvm *kvm;
 };
 
 static inline bool kvm_slot_can_be_private(const struct kvm_memory_slot *slot)
@@ -2363,6 +2364,8 @@ static inline int kvm_restricted_mem_get_pfn(struct kvm_memory_slot *slot,
        *pfn = page_to_pfn(page);
        return ret;
 }
+
+void kvm_arch_memory_mce(struct kvm *kvm);
 #endif /* CONFIG_HAVE_KVM_RESTRICTED_MEM */
 
 #endif
virt/kvm/kvm_main.c +174 -6
index e107afea32f0..ac835fc77273 100644
@@ -936,6 +936,121 @@ static int kvm_init_mmu_notifier(struct kvm *kvm)
 
 #endif /* CONFIG_MMU_NOTIFIER && KVM_ARCH_WANT_MMU_NOTIFIER */
 
+#ifdef CONFIG_HAVE_KVM_RESTRICTED_MEM
+static bool restrictedmem_range_is_valid(struct kvm_memory_slot *slot,
+                                        pgoff_t start, pgoff_t end,
+                                        gfn_t *gfn_start, gfn_t *gfn_end)
+{
+       unsigned long base_pgoff = slot->restricted_offset >> PAGE_SHIFT;
+
+       if (start > base_pgoff)
+               *gfn_start = slot->base_gfn + start - base_pgoff;
+       else
+               *gfn_start = slot->base_gfn;
+
+       if (end < base_pgoff + slot->npages)
+               *gfn_end = slot->base_gfn + end - base_pgoff;
+       else
+               *gfn_end = slot->base_gfn + slot->npages;
+
+       if (*gfn_start >= *gfn_end)
+               return false;
+
+       return true;
+}
+
+static void kvm_restrictedmem_invalidate_begin(struct restrictedmem_notifier *notifier,
+                                              pgoff_t start, pgoff_t end)
+{
+       struct kvm_memory_slot *slot = container_of(notifier,
+                                                   struct kvm_memory_slot,
+                                                   notifier);
+       struct kvm *kvm = slot->kvm;
+       gfn_t gfn_start, gfn_end;
+       struct kvm_gfn_range gfn_range;
+       int idx;
+
+       if (!restrictedmem_range_is_valid(slot, start, end,
+                                         &gfn_start, &gfn_end))
+               return;
+
+       gfn_range.start = gfn_start;
+       gfn_range.end = gfn_end;
+       gfn_range.slot = slot;
+       gfn_range.pte = __pte(0);
+       gfn_range.may_block = true;
+
+       idx = srcu_read_lock(&kvm->srcu);
+       KVM_MMU_LOCK(kvm);
+
+       kvm_mmu_invalidate_begin(kvm);
+       kvm_mmu_invalidate_range_add(kvm, gfn_start, gfn_end);
+       if (kvm_unmap_gfn_range(kvm, &gfn_range))
+               kvm_flush_remote_tlbs(kvm);
+
+       KVM_MMU_UNLOCK(kvm);
+       srcu_read_unlock(&kvm->srcu, idx);
+}
+
+static void kvm_restrictedmem_invalidate_end(struct restrictedmem_notifier *notifier,
+                                            pgoff_t start, pgoff_t end)
+{
+       struct kvm_memory_slot *slot = container_of(notifier,
+                                                   struct kvm_memory_slot,
+                                                   notifier);
+       struct kvm *kvm = slot->kvm;
+       gfn_t gfn_start, gfn_end;
+
+       if (!restrictedmem_range_is_valid(slot, start, end,
+                                         &gfn_start, &gfn_end))
+               return;
+
+       KVM_MMU_LOCK(kvm);
+       kvm_mmu_invalidate_end(kvm);
+       KVM_MMU_UNLOCK(kvm);
+}
+
+static void kvm_restrictedmem_error(struct restrictedmem_notifier *notifier,
+                                   pgoff_t start, pgoff_t end)
+{
+       struct kvm_memory_slot *slot = container_of(notifier,
+                                                   struct kvm_memory_slot,
+                                                   notifier);
+       kvm_arch_memory_mce(slot->kvm);
+}
+
+static struct restrictedmem_notifier_ops kvm_restrictedmem_notifier_ops = {
+       .invalidate_start = kvm_restrictedmem_invalidate_begin,
+       .invalidate_end = kvm_restrictedmem_invalidate_end,
+       .error = kvm_restrictedmem_error,
+};
+
+static inline void kvm_restrictedmem_register(struct kvm_memory_slot *slot)
+{
+       slot->notifier.ops = &kvm_restrictedmem_notifier_ops;
+       restrictedmem_register_notifier(slot->restricted_file, &slot->notifier);
+}
+
+static inline void kvm_restrictedmem_unregister(struct kvm_memory_slot *slot)
+{
+       restrictedmem_unregister_notifier(slot->restricted_file,
+                                         &slot->notifier);
+}
+
+#else /* !CONFIG_HAVE_KVM_RESTRICTED_MEM */
+
+static inline void kvm_restrictedmem_register(struct kvm_memory_slot *slot)
+{
+       WARN_ON_ONCE(1);
+}
+
+static inline void kvm_restrictedmem_unregister(struct kvm_memory_slot *slot)
+{
+       WARN_ON_ONCE(1);
+}
+
+#endif /* CONFIG_HAVE_KVM_RESTRICTED_MEM */
+
 #ifdef CONFIG_HAVE_KVM_PM_NOTIFIER
 static int kvm_pm_notifier_call(struct notifier_block *bl,
                                unsigned long state,
@@ -980,6 +1095,11 @@ static void kvm_destroy_dirty_bitmap(struct kvm_memory_slot *memslot)
 /* This does not remove the slot from struct kvm_memslots data structures */
 static void kvm_free_memslot(struct kvm *kvm, struct kvm_memory_slot *slot)
 {
+       if (slot->flags & KVM_MEM_PRIVATE) {
+               kvm_restrictedmem_unregister(slot);
+               fput(slot->restricted_file);
+       }
+
        kvm_destroy_dirty_bitmap(slot);
 
        kvm_arch_free_memslot(kvm, slot);
@@ -1551,10 +1671,14 @@ static void kvm_replace_memslot(struct kvm *kvm,
        }
 }
 
-static int check_memory_region_flags(const struct kvm_user_mem_region *mem)
+static int check_memory_region_flags(struct kvm *kvm,
+                                    const struct kvm_user_mem_region *mem)
 {
        u32 valid_flags = KVM_MEM_LOG_DIRTY_PAGES;
 
+       if (kvm_arch_has_private_mem(kvm))
+               valid_flags |= KVM_MEM_PRIVATE;
+
 #ifdef __KVM_HAVE_READONLY_MEM
        valid_flags |= KVM_MEM_READONLY;
 #endif
@@ -1630,6 +1754,9 @@ static int kvm_prepare_memory_region(struct kvm *kvm,
 {
        int r;
 
+       if (change == KVM_MR_CREATE && new->flags & KVM_MEM_PRIVATE)
+               kvm_restrictedmem_register(new);
+
        /*
         * If dirty logging is disabled, nullify the bitmap; the old bitmap
         * will be freed on "commit".  If logging is enabled in both old and
@@ -1658,6 +1785,9 @@ static int kvm_prepare_memory_region(struct kvm *kvm,
        if (r && new && new->dirty_bitmap && (!old || !old->dirty_bitmap))
                kvm_destroy_dirty_bitmap(new);
 
+       if (r && change == KVM_MR_CREATE && new->flags & KVM_MEM_PRIVATE)
+               kvm_restrictedmem_unregister(new);
+
        return r;
 }
 
@@ -1963,7 +2093,7 @@ int __kvm_set_memory_region(struct kvm *kvm,
        int as_id, id;
        int r;
 
-       r = check_memory_region_flags(mem);
+       r = check_memory_region_flags(kvm, mem);
        if (r)
                return r;
 
@@ -1982,6 +2112,10 @@ int __kvm_set_memory_region(struct kvm *kvm,
             !access_ok((void __user *)(unsigned long)mem->userspace_addr,
                        mem->memory_size))
                return -EINVAL;
+       if (mem->flags & KVM_MEM_PRIVATE &&
+               (mem->restricted_offset & (PAGE_SIZE - 1) ||
+                mem->restricted_offset > U64_MAX - mem->memory_size))
+               return -EINVAL;
        if (as_id >= KVM_ADDRESS_SPACE_NUM || id >= KVM_MEM_SLOTS_NUM)
                return -EINVAL;
        if (mem->guest_phys_addr + mem->memory_size < mem->guest_phys_addr)
@@ -2020,6 +2154,9 @@ int __kvm_set_memory_region(struct kvm *kvm,
                if ((kvm->nr_memslot_pages + npages) < kvm->nr_memslot_pages)
                        return -EINVAL;
        } else { /* Modify an existing slot. */
+               /* Private memslots are immutable, they can only be deleted. */
+               if (mem->flags & KVM_MEM_PRIVATE)
+                       return -EINVAL;
                if ((mem->userspace_addr != old->userspace_addr) ||
                    (npages != old->npages) ||
                    ((mem->flags ^ old->flags) & KVM_MEM_READONLY))
@@ -2048,10 +2185,28 @@ int __kvm_set_memory_region(struct kvm *kvm,
        new->npages = npages;
        new->flags = mem->flags;
        new->userspace_addr = mem->userspace_addr;
+       if (mem->flags & KVM_MEM_PRIVATE) {
+               new->restricted_file = fget(mem->restricted_fd);
+               if (!new->restricted_file ||
+                   !file_is_restrictedmem(new->restricted_file)) {
+                       r = -EINVAL;
+                       goto out;
+               }
+               new->restricted_offset = mem->restricted_offset;
+       }
+
+       new->kvm = kvm;
 
        r = kvm_set_memslot(kvm, old, new, change);
        if (r)
-               kfree(new);
+               goto out;
+
+       return 0;
+
+out:
+       if (new->restricted_file)
+               fput(new->restricted_file);
+       kfree(new);
        return r;
 }
 EXPORT_SYMBOL_GPL(__kvm_set_memory_region);
@@ -2351,6 +2506,8 @@ static int kvm_vm_ioctl_clear_dirty_log(struct kvm *kvm,
 #ifdef CONFIG_HAVE_KVM_MEMORY_ATTRIBUTES
 static u64 kvm_supported_mem_attributes(struct kvm *kvm)
 {
+       if (kvm_arch_has_private_mem(kvm))
+               return KVM_MEMORY_ATTRIBUTE_PRIVATE;
        return 0;
 }
 
@@ -4822,16 +4979,28 @@ static long kvm_vm_ioctl(struct file *filp,
        }
        case KVM_SET_USER_MEMORY_REGION: {
                struct kvm_user_mem_region mem;
-               unsigned long size = sizeof(struct kvm_userspace_memory_region);
+               unsigned int flags_offset = offsetof(typeof(mem), flags);
+               unsigned long size;
+               u32 flags;
 
                kvm_sanity_check_user_mem_region_alias();
 
+               memset(&mem, 0, sizeof(mem));
+
                r = -EFAULT;
+               if (get_user(flags, (u32 __user *)(argp + flags_offset)))
+                       goto out;
+
+               if (flags & KVM_MEM_PRIVATE)
+                       size = sizeof(struct kvm_userspace_memory_region_ext);
+               else
+                       size = sizeof(struct kvm_userspace_memory_region);
+
                if (copy_from_user(&mem, argp, size))
                        goto out;
 
                r = -EINVAL;
-               if (mem.flags & KVM_MEM_PRIVATE)
+               if ((flags ^ mem.flags) & KVM_MEM_PRIVATE)
                        goto out;
 
                r = kvm_vm_ioctl_set_memory_region(kvm, &mem);
--

2.25.1

#296 lore raw

Re: [PATCH v10 9/9] KVM: Enable and expose KVM_MEM_PRIVATE

In reply to Chao Peng

virt/kvm/kvm_main.c static int kvm_init_mmu_notifier(struct kvm *kvm) · line 947

Quoted 10 lines · Chao Peng
virt/kvm/kvm_main.c +9 -0
+#ifdef CONFIG_HAVE_KVM_RESTRICTED_MEM
+static bool restrictedmem_range_is_valid(struct kvm_memory_slot *slot,
+                                        pgoff_t start, pgoff_t end,
+                                        gfn_t *gfn_start, gfn_t *gfn_end)
+{
+       unsigned long base_pgoff = slot->restricted_offset >> PAGE_SHIFT;
+
+       if (start > base_pgoff)
+               *gfn_start = slot->base_gfn + start - base_pgoff;

There should be a check for overflow here in case start is a very big value. Additional check can look like:
if (start >= base_pgoff + slot->npages)

       return false;

virt/kvm/kvm_main.c static int kvm_init_mmu_notifier(struct kvm *kvm) · line 952

Quoted 5 lines
virt/kvm/kvm_main.c +5 -0
+       else
+               *gfn_start = slot->base_gfn;
+
+       if (end < base_pgoff + slot->npages)
+               *gfn_end = slot->base_gfn + end - base_pgoff;

If "end" is smaller than base_pgoff, this can cause overflow and return the range as valid. There should be additional check:
if (end < base_pgoff)

         return false;
Quoted 9 lines
virt/kvm/kvm_main.c +9 -0
+       else
+               *gfn_end = slot->base_gfn + slot->npages;
+
+       if (*gfn_start >= *gfn_end)
+               return false;
+
+       return true;
+}
+
author
#297 lore raw

Re: [PATCH v10 9/9] KVM: Enable and expose KVM_MEM_PRIVATE

Quoted 23 lines · Vishal Annapurve

virt/kvm/kvm_main.c static int kvm_init_mmu_notifier(struct kvm *kvm) · line 947

Quoted 10 lines · Chao Peng
virt/kvm/kvm_main.c +9 -0
+#ifdef CONFIG_HAVE_KVM_RESTRICTED_MEM
+static bool restrictedmem_range_is_valid(struct kvm_memory_slot *slot,
+                                        pgoff_t start, pgoff_t end,
+                                        gfn_t *gfn_start, gfn_t *gfn_end)
+{
+       unsigned long base_pgoff = slot->restricted_offset >> PAGE_SHIFT;
+
+       if (start > base_pgoff)
+               *gfn_start = slot->base_gfn + start - base_pgoff;

There should be a check for overflow here in case start is a very big value. Additional check can look like:
if (start >= base_pgoff + slot->npages)

       return false;

virt/kvm/kvm_main.c static int kvm_init_mmu_notifier(struct kvm *kvm) · line 952

Quoted 5 lines
virt/kvm/kvm_main.c +5 -0
+       else
+               *gfn_start = slot->base_gfn;
+
+       if (end < base_pgoff + slot->npages)
+               *gfn_end = slot->base_gfn + end - base_pgoff;
Quoted 4 lines

If "end" is smaller than base_pgoff, this can cause overflow and return the range as valid. There should be additional check:
if (end < base_pgoff)

         return false;

Thanks! Both are good catches. The improved code:

static bool restrictedmem_range_is_valid(struct kvm_memory_slot *slot,
pgoff_t start, pgoff_t end, gfn_t *gfn_start, gfn_t *gfn_end)

{
	unsigned long base_pgoff = slot->restricted_offset >> PAGE_SHIFT;

	if (start >= base_pgoff + slot->npages)
		return false;
	else if (start <= base_pgoff)
		*gfn_start = slot->base_gfn;
	else
		*gfn_start = start - base_pgoff + slot->base_gfn;

	if (end <= base_pgoff)
		return false;
	else if (end >= base_pgoff + slot->npages)
		*gfn_end = slot->base_gfn + slot->npages;
	else
		*gfn_end = end - base_pgoff + slot->base_gfn;

	if (*gfn_start >= *gfn_end)
		return false;

	return true;
}

Thanks,
Chao

Quoted 11 lines
Quoted 11 lines
virt/kvm/kvm_main.c +9 -0
+       else
+               *gfn_end = slot->base_gfn + slot->npages;
+
+       if (*gfn_start >= *gfn_end)
+               return false;
+
+       return true;
+}
+
#298 lore raw

Re: [PATCH v10 9/9] KVM: Enable and expose KVM_MEM_PRIVATE

In reply to Chao Peng

arch/x86/kvm/x86.c static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · line 10369

Quoted 7 lines · Chao Peng
arch/x86/kvm/x86.c +3 -0
@@ -10357,6 +10364,12 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 
 		if (kvm_check_request(KVM_REQ_UPDATE_CPU_DIRTY_LOGGING, vcpu))
 			static_call(kvm_x86_update_cpu_dirty_logging)(vcpu);
+
+		if (kvm_check_request(KVM_REQ_MEMORY_MCE, vcpu)) {
+			vcpu->run->exit_reason = KVM_EXIT_SHUTDOWN;

Synthesizing triple fault shutdown is not the right approach. Even with TDX's MCE "architecture" (heavy sarcasm), it's possible that host userspace and the guest have a paravirt interface for handling memory errors without killing the host.

arch/x86/kvm/x86.c int __kvm_set_memory_region(struct kvm *kvm, · line 2116

Quoted 12 lines
arch/x86/kvm/x86.c +5 -0
+			r = 0;
+			goto out;
+		}
 	}
 
 
@@ -1982,6 +2112,10 @@ int __kvm_set_memory_region(struct kvm *kvm,
 	     !access_ok((void __user *)(unsigned long)mem->userspace_addr,
 			mem->memory_size))
 		return -EINVAL;
+	if (mem->flags & KVM_MEM_PRIVATE &&
+		(mem->restricted_offset & (PAGE_SIZE - 1) ||

Align indentation.

virt/kvm/kvm_main.c int __kvm_set_memory_region(struct kvm *kvm, · line 2117

Quoted 1 line
virt/kvm/kvm_main.c +1 -0
+		 mem->restricted_offset > U64_MAX - mem->memory_size))

Strongly prefer to use similar logic to existing code that detects wraps:

		mem->restricted_offset + mem->memory_size < mem->restricted_offset

This is also where I'd like to add the "gfn is aligned to offset" check, though my brain is too fried to figure that out right now.

virt/kvm/kvm_main.c int __kvm_set_memory_region(struct kvm *kvm, · line 2157

Quoted 9 lines
virt/kvm/kvm_main.c +2 -0
+		return -EINVAL;
 	if (as_id >= KVM_ADDRESS_SPACE_NUM || id >= KVM_MEM_SLOTS_NUM)
 		return -EINVAL;
 	if (mem->guest_phys_addr + mem->memory_size < mem->guest_phys_addr)
@@ -2020,6 +2154,9 @@ int __kvm_set_memory_region(struct kvm *kvm,
 		if ((kvm->nr_memslot_pages + npages) < kvm->nr_memslot_pages)
 			return -EINVAL;
 	} else { /* Modify an existing slot. */
+		/* Private memslots are immutable, they can only be deleted. */

I'm 99% certain I suggested this, but if we're going to make these memslots immutable, then we should straight up disallow dirty logging, otherwise we'll end up with a bizarre uAPI.

Quoted 10 lines
virt/kvm/kvm_main.c +3 -0
+		if (mem->flags & KVM_MEM_PRIVATE)
+			return -EINVAL;
 		if ((mem->userspace_addr != old->userspace_addr) ||
 		    (npages != old->npages) ||
 		    ((mem->flags ^ old->flags) & KVM_MEM_READONLY))
@@ -2048,10 +2185,28 @@ int __kvm_set_memory_region(struct kvm *kvm,
 	new->npages = npages;
 	new->flags = mem->flags;
 	new->userspace_addr = mem->userspace_addr;
+	if (mem->flags & KVM_MEM_PRIVATE) {

virt/kvm/kvm_main.c int __kvm_set_memory_region(struct kvm *kvm, · line 2198

Quoted 10 lines
virt/kvm/kvm_main.c +10 -0
+		new->restricted_file = fget(mem->restricted_fd);
+		if (!new->restricted_file ||
+		    !file_is_restrictedmem(new->restricted_file)) {
+			r = -EINVAL;
+			goto out;
+		}
+		new->restricted_offset = mem->restricted_offset;
+	}
+
+	new->kvm = kvm;

Set this above, just so that the code flows better.

author
#299 lore raw

Re: [PATCH v10 9/9] KVM: Enable and expose KVM_MEM_PRIVATE

Quoted 13 lines · Sean Christopherson

Review static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · line 10369

Quoted 7 lines · Chao Peng
(quoted diff) +3 -0
@@ -10357,6 +10364,12 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 
 		if (kvm_check_request(KVM_REQ_UPDATE_CPU_DIRTY_LOGGING, vcpu))
 			static_call(kvm_x86_update_cpu_dirty_logging)(vcpu);
+
+		if (kvm_check_request(KVM_REQ_MEMORY_MCE, vcpu)) {
+			vcpu->run->exit_reason = KVM_EXIT_SHUTDOWN;

Synthesizing triple fault shutdown is not the right approach. Even with TDX's MCE "architecture" (heavy sarcasm), it's possible that host userspace and the guest have a paravirt interface for handling memory errors without killing the host.

Agree shutdown is not the correct choice. I see you made below change:

send_sig_mceerr(BUS_MCEERR_AR, (void __user *)hva, PAGE_SHIFT, current)

The MCE may happen in any thread than KVM thread, sending siginal to 'current' thread may not be the expected behavior. Also how userspace can tell is the MCE on the shared page or private page? Do we care?

Quoted 25 lines

Review int __kvm_set_memory_region(struct kvm *kvm, · line 2116

Quoted 13 lines
(quoted diff) +5 -0
+			r = 0;
+			goto out;
+		}
 	}
 
 
@@ -1982,6 +2112,10 @@ int __kvm_set_memory_region(struct kvm *kvm,
 	     !access_ok((void __user *)(unsigned long)mem->userspace_addr,
 			mem->memory_size))
 		return -EINVAL;
+	if (mem->flags & KVM_MEM_PRIVATE &&
+		(mem->restricted_offset & (PAGE_SIZE - 1) ||

Align indentation.

Review int __kvm_set_memory_region(struct kvm *kvm, · line 2117

Quoted 1 line
(quoted diff) +1 -0
+		 mem->restricted_offset > U64_MAX - mem->memory_size))

Strongly prefer to use similar logic to existing code that detects wraps:

		mem->restricted_offset + mem->memory_size < mem->restricted_offset

This is also where I'd like to add the "gfn is aligned to offset" check, though my brain is too fried to figure that out right now.

Quoted 13 lines

Review int __kvm_set_memory_region(struct kvm *kvm, · line 2157

Quoted 9 lines
(quoted diff) +2 -0
+		return -EINVAL;
 	if (as_id >= KVM_ADDRESS_SPACE_NUM || id >= KVM_MEM_SLOTS_NUM)
 		return -EINVAL;
 	if (mem->guest_phys_addr + mem->memory_size < mem->guest_phys_addr)
@@ -2020,6 +2154,9 @@ int __kvm_set_memory_region(struct kvm *kvm,
 		if ((kvm->nr_memslot_pages + npages) < kvm->nr_memslot_pages)
 			return -EINVAL;
 	} else { /* Modify an existing slot. */
+		/* Private memslots are immutable, they can only be deleted. */

I'm 99% certain I suggested this, but if we're going to make these memslots immutable, then we should straight up disallow dirty logging, otherwise we'll end up with a bizarre uAPI.

But in my mind dirty logging will be needed in the very short time, when live migration gets supported?

Quoted 10 lines
Quoted 10 lines
(quoted diff) +2 -0
+		if (mem->flags & KVM_MEM_PRIVATE)
+			return -EINVAL;
 		if ((mem->userspace_addr != old->userspace_addr) ||
 		    (npages != old->npages) ||
 		    ((mem->flags ^ old->flags) & KVM_MEM_READONLY))
@@ -2048,10 +2185,28 @@ int __kvm_set_memory_region(struct kvm *kvm,
 	new->npages = npages;
 	new->flags = mem->flags;
 	new->userspace_addr = mem->userspace_addr;
Quoted 8 lines

Review int __kvm_set_memory_region(struct kvm *kvm, · line 2195

Quoted 8 lines
(quoted diff) +8 -0
+	if (mem->flags & KVM_MEM_PRIVATE) {
+		new->restricted_file = fget(mem->restricted_fd);
+		if (!new->restricted_file ||
+		    !file_is_restrictedmem(new->restricted_file)) {
+			r = -EINVAL;
+			goto out;
+		}
+		new->restricted_offset = mem->restricted_offset;

I see you changed slot->restricted_offset type from loff_t to gfn_t and used pgoff_t when doing the restrictedmem_bind/unbind(). Using page index is reasonable KVM internally and sounds simpler than loff_t. But we also need initialize it to page index here as well as changes in another two cases. This is needed when restricted_offset != 0.

include/linux/kvm_host.h +1 -2
index 547b92215002..49e375e78f30 100644
@@ -2364,8 +2364,7 @@ static inline int kvm_restricted_mem_get_pfn(struct kvm_memory_slot *slot,
                                             gfn_t gfn, kvm_pfn_t *pfn,
                                             int *order)
 {
-       pgoff_t index = gfn - slot->base_gfn +
-                       (slot->restricted_offset >> PAGE_SHIFT);
+       pgoff_t index = gfn - slot->base_gfn + slot->restricted_offset;
        struct page *page;
        int ret;
 
virt/kvm/kvm_main.c +2 -2
index 01db35ddd5b3..7439bdcb0d04 100644
@@ -935,7 +935,7 @@ static bool restrictedmem_range_is_valid(struct kvm_memory_slot *slot,
                                         pgoff_t start, pgoff_t end,
                                         gfn_t *gfn_start, gfn_t *gfn_end)
 {
-       unsigned long base_pgoff = slot->restricted_offset >> PAGE_SHIFT;
+       unsigned long base_pgoff = slot->restricted_offset;
 
        if (start > base_pgoff)
                *gfn_start = slot->base_gfn + start - base_pgoff;
@@ -2275,7 +2275,7 @@ int __kvm_set_memory_region(struct kvm *kvm,
                        r = -EINVAL;
                        goto out;
                }
-               new->restricted_offset = mem->restricted_offset;
+               new->restricted_offset = mem->restricted_offset >> PAGE_SHIFT;
        }
 
        r = kvm_set_memslot(kvm, old, new, change);

Chao

Quoted 5 lines

Review int __kvm_set_memory_region(struct kvm *kvm, · line 2198

Quoted 3 lines
(quoted diff) +3 -0
+	}
+
+	new->kvm = kvm;

Set this above, just so that the code flows better.

#300 lore raw

Re: [PATCH v10 9/9] KVM: Enable and expose KVM_MEM_PRIVATE

Quoted 15 lines · Chao Peng
Quoted 13 lines · Sean Christopherson

Review static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · line 10369

Quoted 7 lines · Chao Peng
(quoted diff) +3 -0
@@ -10357,6 +10364,12 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 
 		if (kvm_check_request(KVM_REQ_UPDATE_CPU_DIRTY_LOGGING, vcpu))
 			static_call(kvm_x86_update_cpu_dirty_logging)(vcpu);
+
+		if (kvm_check_request(KVM_REQ_MEMORY_MCE, vcpu)) {
+			vcpu->run->exit_reason = KVM_EXIT_SHUTDOWN;

Synthesizing triple fault shutdown is not the right approach. Even with TDX's MCE "architecture" (heavy sarcasm), it's possible that host userspace and the guest have a paravirt interface for handling memory errors without killing the host.

Quoted 6 lines

Agree shutdown is not the correct choice. I see you made below change:

send_sig_mceerr(BUS_MCEERR_AR, (void __user *)hva, PAGE_SHIFT, current)

The MCE may happen in any thread than KVM thread, sending siginal to 'current' thread may not be the expected behavior.

This is already true today, e.g. a #MC in memory that is mapped into the guest can be triggered by a host access. Hrm, but in this case we actually have a KVM instance, and we know that the #MC is relevant to the KVM instance, so I agree that signaling 'current' is kludgy.

Quoted 2 lines

Also how userspace can tell is the MCE on the shared page or private page? Do we care?

We care. I was originally thinking we could require userspace to keep track of things, but that's quite prescriptive and flawed, e.g. could race with conversions.

One option would be to KVM_EXIT_MEMORY_FAULT, and then wire up a generic (not x86 specific) KVM request to exit to userspace, e.g.

		/* KVM_EXIT_MEMORY_FAULT */
		struct {
#define KVM_MEMORY_EXIT_FLAG_PRIVATE	(1ULL << 3)
#define KVM_MEMORY_EXIT_FLAG_HW_ERROR	(1ULL << 4)
			__u64 flags;
			__u64 gpa;
			__u64 size;
		} memory;

But I'm not sure that's the correct approach. It kinda feels like we're reinventing the wheel. It seems like restrictedmem_get_page() _must_ be able to reject attempts to get a poisoned page, i.e. restrictedmem_get_page() should yield KVM_PFN_ERR_HWPOISON. Assuming that's the case, then I believe KVM simply needs to zap SPTEs in response to an error notification in order to force vCPUs to fault on the poisoned page.

Quoted 14 lines
Quoted 5 lines
Quoted 5 lines
(quoted diff) +1 -0
+		return -EINVAL;
 	if (as_id >= KVM_ADDRESS_SPACE_NUM || id >= KVM_MEM_SLOTS_NUM)
 		return -EINVAL;
 	if (mem->guest_phys_addr + mem->memory_size < mem->guest_phys_addr)
@@ -2020,6 +2154,9 @@ int __kvm_set_memory_region(struct kvm *kvm,
Quoted 8 lines

Review int __kvm_set_memory_region(struct kvm *kvm, · line 2157

Quoted 4 lines
(quoted diff) +1 -0
 		if ((kvm->nr_memslot_pages + npages) < kvm->nr_memslot_pages)
 			return -EINVAL;
 	} else { /* Modify an existing slot. */
+		/* Private memslots are immutable, they can only be deleted. */

I'm 99% certain I suggested this, but if we're going to make these memslots immutable, then we should straight up disallow dirty logging, otherwise we'll end up with a bizarre uAPI.

Quoted 2 lines

But in my mind dirty logging will be needed in the very short time, when live migration gets supported?

Ya, but if/when live migration support is added, private memslots will no longer be immutable as userspace will want to enable dirty logging only when a VM is being migrated, i.e. something will need to change.

Given that it looks like we have clear line of sight to SEV+UPM guests, my preference would be to allow toggling dirty logging from the get-go. It doesn't necessarily have to be in the first patch, e.g. KVM could initially reject KVM_MEM_LOG_DIRTY_PAGES + KVM_MEM_PRIVATE and then add support separately to make the series easier to review, test, and bisect.

static int check_memory_region_flags(struct kvm *kvm,
const struct kvm_userspace_memory_region2 *mem)

{
	u32 valid_flags = KVM_MEM_LOG_DIRTY_PAGES;

	if (kvm_arch_has_private_mem(kvm) &&
	    ~(mem->flags & KVM_MEM_LOG_DIRTY_PAGES))
		valid_flags |= KVM_MEM_PRIVATE;


	...
}
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
(quoted diff) +8 -0
+		if (mem->flags & KVM_MEM_PRIVATE)
+			return -EINVAL;
 		if ((mem->userspace_addr != old->userspace_addr) ||
 		    (npages != old->npages) ||
 		    ((mem->flags ^ old->flags) & KVM_MEM_READONLY))
@@ -2048,10 +2185,28 @@ int __kvm_set_memory_region(struct kvm *kvm,
 	new->npages = npages;
 	new->flags = mem->flags;
 	new->userspace_addr = mem->userspace_addr;
+	if (mem->flags & KVM_MEM_PRIVATE) {
+		new->restricted_file = fget(mem->restricted_fd);
+		if (!new->restricted_file ||
+		    !file_is_restrictedmem(new->restricted_file)) {
+			r = -EINVAL;
+			goto out;
Quoted 8 lines
Quoted 2 lines

Review int __kvm_set_memory_region(struct kvm *kvm, · line 2195

Quoted 2 lines
(quoted diff) +2 -0
+		}
+		new->restricted_offset = mem->restricted_offset;

I see you changed slot->restricted_offset type from loff_t to gfn_t and used pgoff_t when doing the restrictedmem_bind/unbind(). Using page index is reasonable KVM internally and sounds simpler than loff_t. But we also need initialize it to page index here as well as changes in another two cases. This is needed when restricted_offset != 0.

Oof. I'm pretty sure I completely missed that loff_t is used for byte offsets, whereas pgoff_t is a frame index.

Given that the restrictmem APIs take pgoff_t, I definitely think it makes sense to the index, but I'm very tempted to store pgoff_t instead of gfn_t, and name the field "index" to help connect the dots to the rest of kernel, where "pgoff_t index" is quite common.

And looking at those bits again, we should wrap all of the restrictedmem fields with CONFIG_KVM_PRIVATE_MEM. It'll require minor tweaks to __kvm_set_memory_region(), but I think will yield cleaner code (and internal APIs) overall.

And wrap the three fields in an anonymous struct? E.g. this is a little more versbose (restrictedmem instead restricted), but at first glance it doesn't seem to cause widespared line length issues.

#ifdef CONFIG_KVM_PRIVATE_MEM
	struct {
		struct file *file;
		pgoff_t index;
		struct restrictedmem_notifier notifier;
	} restrictedmem;

#endif

Quoted 43 lines
include/linux/kvm_host.h +1 -2
index 547b92215002..49e375e78f30 100644
@@ -2364,8 +2364,7 @@ static inline int kvm_restricted_mem_get_pfn(struct kvm_memory_slot *slot,
                                             gfn_t gfn, kvm_pfn_t *pfn,
                                             int *order)
 {
-       pgoff_t index = gfn - slot->base_gfn +
-                       (slot->restricted_offset >> PAGE_SHIFT);
+       pgoff_t index = gfn - slot->base_gfn + slot->restricted_offset;
        struct page *page;
        int ret;
 
virt/kvm/kvm_main.c +2 -2
index 01db35ddd5b3..7439bdcb0d04 100644
@@ -935,7 +935,7 @@ static bool restrictedmem_range_is_valid(struct kvm_memory_slot *slot,
                                         pgoff_t start, pgoff_t end,
                                         gfn_t *gfn_start, gfn_t *gfn_end)
 {
-       unsigned long base_pgoff = slot->restricted_offset >> PAGE_SHIFT;
+       unsigned long base_pgoff = slot->restricted_offset;
 
        if (start > base_pgoff)
                *gfn_start = slot->base_gfn + start - base_pgoff;
@@ -2275,7 +2275,7 @@ int __kvm_set_memory_region(struct kvm *kvm,
                        r = -EINVAL;
                        goto out;
                }
-               new->restricted_offset = mem->restricted_offset;
+               new->restricted_offset = mem->restricted_offset >> PAGE_SHIFT;
        }
 
        r = kvm_set_memslot(kvm, old, new, change);

Chao

Quoted 5 lines

Review int __kvm_set_memory_region(struct kvm *kvm, · line 2198

Quoted 3 lines
(quoted diff) +3 -0
+	}
+
+	new->kvm = kvm;

Set this above, just so that the code flows better.

author
#301 lore raw

Re: [PATCH v10 9/9] KVM: Enable and expose KVM_MEM_PRIVATE

Quoted 46 lines · Sean Christopherson
Quoted 15 lines · Chao Peng
Quoted 13 lines · Sean Christopherson

Review static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · line 10369

Quoted 7 lines · Chao Peng
(quoted diff) +3 -0
@@ -10357,6 +10364,12 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 
 		if (kvm_check_request(KVM_REQ_UPDATE_CPU_DIRTY_LOGGING, vcpu))
 			static_call(kvm_x86_update_cpu_dirty_logging)(vcpu);
+
+		if (kvm_check_request(KVM_REQ_MEMORY_MCE, vcpu)) {
+			vcpu->run->exit_reason = KVM_EXIT_SHUTDOWN;

Synthesizing triple fault shutdown is not the right approach. Even with TDX's MCE "architecture" (heavy sarcasm), it's possible that host userspace and the guest have a paravirt interface for handling memory errors without killing the host.

Quoted 6 lines

Agree shutdown is not the correct choice. I see you made below change:

send_sig_mceerr(BUS_MCEERR_AR, (void __user *)hva, PAGE_SHIFT, current)

The MCE may happen in any thread than KVM thread, sending siginal to 'current' thread may not be the expected behavior.

This is already true today, e.g. a #MC in memory that is mapped into the guest can be triggered by a host access. Hrm, but in this case we actually have a KVM instance, and we know that the #MC is relevant to the KVM instance, so I agree that signaling 'current' is kludgy.

Quoted 2 lines

Also how userspace can tell is the MCE on the shared page or private page? Do we care?

We care. I was originally thinking we could require userspace to keep track of things, but that's quite prescriptive and flawed, e.g. could race with conversions.

One option would be to KVM_EXIT_MEMORY_FAULT, and then wire up a generic (not x86 specific) KVM request to exit to userspace, e.g.

		/* KVM_EXIT_MEMORY_FAULT */
		struct {
#define KVM_MEMORY_EXIT_FLAG_PRIVATE	(1ULL << 3)
#define KVM_MEMORY_EXIT_FLAG_HW_ERROR	(1ULL << 4)
			__u64 flags;
			__u64 gpa;
			__u64 size;
		} memory;
Quoted 3 lines

But I'm not sure that's the correct approach. It kinda feels like we're reinventing the wheel. It seems like restrictedmem_get_page() _must_ be able to reject attempts to get a poisoned page, i.e. restrictedmem_get_page() should yield KVM_PFN_ERR_HWPOISON.

Yes, I see there is -EHWPOISON handling for hva_to_pfn() for shared memory. It makes sense doing similar for private page.

Quoted 2 lines

Assuming that's the case, then I believe KVM simply needs to zap SPTEs in response to an error notification in order to force vCPUs to fault on the poisoned page.

Agree, this is waht we should do anyway.

Quoted 28 lines
Quoted 15 lines
Quoted 14 lines

Review int __kvm_set_memory_region(struct kvm *kvm, · line 2157

Quoted 10 lines
(quoted diff) +2 -0
+		return -EINVAL;
 	if (as_id >= KVM_ADDRESS_SPACE_NUM || id >= KVM_MEM_SLOTS_NUM)
 		return -EINVAL;
 	if (mem->guest_phys_addr + mem->memory_size < mem->guest_phys_addr)
@@ -2020,6 +2154,9 @@ int __kvm_set_memory_region(struct kvm *kvm,
 		if ((kvm->nr_memslot_pages + npages) < kvm->nr_memslot_pages)
 			return -EINVAL;
 	} else { /* Modify an existing slot. */
+		/* Private memslots are immutable, they can only be deleted. */

I'm 99% certain I suggested this, but if we're going to make these memslots immutable, then we should straight up disallow dirty logging, otherwise we'll end up with a bizarre uAPI.

Quoted 2 lines

But in my mind dirty logging will be needed in the very short time, when live migration gets supported?

Ya, but if/when live migration support is added, private memslots will no longer be immutable as userspace will want to enable dirty logging only when a VM is being migrated, i.e. something will need to change.

Given that it looks like we have clear line of sight to SEV+UPM guests, my preference would be to allow toggling dirty logging from the get-go. It doesn't necessarily have to be in the first patch, e.g. KVM could initially reject KVM_MEM_LOG_DIRTY_PAGES + KVM_MEM_PRIVATE and then add support separately to make the series easier to review, test, and bisect.

Quoted 8 lines

static int check_memory_region_flags(struct kvm *kvm,
const struct kvm_userspace_memory_region2 *mem)

{
	u32 valid_flags = KVM_MEM_LOG_DIRTY_PAGES;

	if (kvm_arch_has_private_mem(kvm) &&
	    ~(mem->flags & KVM_MEM_LOG_DIRTY_PAGES))
		valid_flags |= KVM_MEM_PRIVATE;

Adding this limitation is OK to me. It's not too hard to remove it when live migration gets added.

Quoted 45 lines

	...
}
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
(quoted diff) +8 -0
+		if (mem->flags & KVM_MEM_PRIVATE)
+			return -EINVAL;
 		if ((mem->userspace_addr != old->userspace_addr) ||
 		    (npages != old->npages) ||
 		    ((mem->flags ^ old->flags) & KVM_MEM_READONLY))
@@ -2048,10 +2185,28 @@ int __kvm_set_memory_region(struct kvm *kvm,
 	new->npages = npages;
 	new->flags = mem->flags;
 	new->userspace_addr = mem->userspace_addr;
+	if (mem->flags & KVM_MEM_PRIVATE) {
+		new->restricted_file = fget(mem->restricted_fd);
+		if (!new->restricted_file ||
+		    !file_is_restrictedmem(new->restricted_file)) {
+			r = -EINVAL;
+			goto out;
Quoted 8 lines
Quoted 2 lines

Review int __kvm_set_memory_region(struct kvm *kvm, · line 2195

Quoted 2 lines
(quoted diff) +2 -0
+		}
+		new->restricted_offset = mem->restricted_offset;

I see you changed slot->restricted_offset type from loff_t to gfn_t and used pgoff_t when doing the restrictedmem_bind/unbind(). Using page index is reasonable KVM internally and sounds simpler than loff_t. But we also need initialize it to page index here as well as changes in another two cases. This is needed when restricted_offset != 0.

Oof. I'm pretty sure I completely missed that loff_t is used for byte offsets, whereas pgoff_t is a frame index.

Given that the restrictmem APIs take pgoff_t, I definitely think it makes sense to the index, but I'm very tempted to store pgoff_t instead of gfn_t, and name the field "index" to help connect the dots to the rest of kernel, where "pgoff_t index" is quite common.

And looking at those bits again, we should wrap all of the restrictedmem fields with CONFIG_KVM_PRIVATE_MEM. It'll require minor tweaks to __kvm_set_memory_region(), but I think will yield cleaner code (and internal APIs) overall.

And wrap the three fields in an anonymous struct? E.g. this is a little more versbose (restrictedmem instead restricted), but at first glance it doesn't seem to cause widespared line length issues.

Quoted 7 lines
#ifdef CONFIG_KVM_PRIVATE_MEM
	struct {
		struct file *file;
		pgoff_t index;
		struct restrictedmem_notifier notifier;
	} restrictedmem;

#endif

Looks better.

Thanks,
Chao

Quoted 44 lines
Quoted 44 lines
include/linux/kvm_host.h +1 -2
index 547b92215002..49e375e78f30 100644
@@ -2364,8 +2364,7 @@ static inline int kvm_restricted_mem_get_pfn(struct kvm_memory_slot *slot,
                                             gfn_t gfn, kvm_pfn_t *pfn,
                                             int *order)
 {
-       pgoff_t index = gfn - slot->base_gfn +
-                       (slot->restricted_offset >> PAGE_SHIFT);
+       pgoff_t index = gfn - slot->base_gfn + slot->restricted_offset;
        struct page *page;
        int ret;
 
virt/kvm/kvm_main.c +2 -2
index 01db35ddd5b3..7439bdcb0d04 100644
@@ -935,7 +935,7 @@ static bool restrictedmem_range_is_valid(struct kvm_memory_slot *slot,
                                         pgoff_t start, pgoff_t end,
                                         gfn_t *gfn_start, gfn_t *gfn_end)
 {
-       unsigned long base_pgoff = slot->restricted_offset >> PAGE_SHIFT;
+       unsigned long base_pgoff = slot->restricted_offset;
 
        if (start > base_pgoff)
                *gfn_start = slot->base_gfn + start - base_pgoff;
@@ -2275,7 +2275,7 @@ int __kvm_set_memory_region(struct kvm *kvm,
                        r = -EINVAL;
                        goto out;
                }
-               new->restricted_offset = mem->restricted_offset;
+               new->restricted_offset = mem->restricted_offset >> PAGE_SHIFT;
        }
 
        r = kvm_set_memslot(kvm, old, new, change);

Chao

Quoted 5 lines

Review int __kvm_set_memory_region(struct kvm *kvm, · line 2198

Quoted 3 lines
(quoted diff) +3 -0
+	}
+
+	new->kvm = kvm;

Set this above, just so that the code flows better.

author
#302 lore raw

Re: [PATCH v10 9/9] KVM: Enable and expose KVM_MEM_PRIVATE

In reply to Sean Christopherson
Quoted 1 line · Sean Christopherson

On Fri, Dec 02, 2022, Chao Peng wrote:

...

Quoted 6 lines

Strongly prefer to use similar logic to existing code that detects wraps:

		mem->restricted_offset + mem->memory_size < mem->restricted_offset

This is also where I'd like to add the "gfn is aligned to offset" check, though my brain is too fried to figure that out right now.

Used count_trailing_zeros() for this TODO, unsure we have other better approach.

virt/kvm/kvm_main.c +16 -1
index afc8c26fa652..fd34c5f7cd2f 100644
@@ -56,6 +56,7 @@
 #include <asm/processor.h>
 #include <asm/ioctl.h>
 #include <linux/uaccess.h>
+#include <linux/count_zeros.h>
 
 #include "coalesced_mmio.h"
 #include "async_pf.h"
@@ -2087,6 +2088,19 @@ static bool kvm_check_memslot_overlap(struct kvm_memslots *slots, int id,
 	return false;
 }
 
+/*
+ * Return true when ALIGNMENT(offset) >= ALIGNMENT(gpa).
+ */
+static bool kvm_check_rmem_offset_alignment(u64 offset, u64 gpa)
+{
+	if (!offset)
+		return true;
+	if (!gpa)
+		return false;
+
+	return !!(count_trailing_zeros(offset) >= count_trailing_zeros(gpa));
+}
+
 /*
  * Allocate some memory and give it an address in the guest physical address
  * space.
@@ -2128,7 +2142,8 @@ int __kvm_set_memory_region(struct kvm *kvm,
 	if (mem->flags & KVM_MEM_PRIVATE &&
 	    (mem->restrictedmem_offset & (PAGE_SIZE - 1) ||
 	     mem->restrictedmem_offset + mem->memory_size < mem->restrictedmem_offset ||
-	     0 /* TODO: require gfn be aligned with restricted offset */))
+	     !kvm_check_rmem_offset_alignment(mem->restrictedmem_offset,
+					      mem->guest_phys_addr)))
 		return -EINVAL;
 	if (as_id >= kvm_arch_nr_memslot_as_ids(kvm) || id >= KVM_MEM_SLOTS_NUM)
 		return -EINVAL;
#303 lore raw

Re: [PATCH v10 9/9] KVM: Enable and expose KVM_MEM_PRIVATE

Quoted 31 lines · Chao Peng
Quoted 1 line · Sean Christopherson

On Fri, Dec 02, 2022, Chao Peng wrote:

...

Quoted 6 lines

Strongly prefer to use similar logic to existing code that detects wraps:

		mem->restricted_offset + mem->memory_size < mem->restricted_offset

This is also where I'd like to add the "gfn is aligned to offset" check, though my brain is too fried to figure that out right now.

Used count_trailing_zeros() for this TODO, unsure we have other better approach.

virt/kvm/kvm_main.c +2 -0
index afc8c26fa652..fd34c5f7cd2f 100644
@@ -56,6 +56,7 @@
  #include <asm/processor.h>
  #include <asm/ioctl.h>
  #include <linux/uaccess.h>
+#include <linux/count_zeros.h>
 
  #include "coalesced_mmio.h"
  #include "async_pf.h"
@@ -2087,6 +2088,19 @@ static bool kvm_check_memslot_overlap(struct  
 kvm_memslots *slots, int id,
  	return false;
  }
 
+/*

virt/kvm/kvm_main.c static bool kvm_check_memslot_overlap(struct · line 2102

Quoted 10 lines
virt/kvm/kvm_main.c +10 -0
+ * Return true when ALIGNMENT(offset) >= ALIGNMENT(gpa).
+ */
+static bool kvm_check_rmem_offset_alignment(u64 offset, u64 gpa)
+{
+	if (!offset)
+		return true;
+	if (!gpa)
+		return false;
+
+	return !!(count_trailing_zeros(offset) >= count_trailing_zeros(gpa));

Perhaps we could do something like

#define lowest_set_bit(val) (val & -val)

and use

return lowest_set_bit(offset) >= lowest_set_bit(gpa);

Please help me to understand: why must ALIGNMENT(offset) >= ALIGNMENT(gpa)? Why is it not sufficient to have both gpa and offset be aligned to PAGE_SIZE?

Quoted 17 lines
virt/kvm/kvm_main.c +2 -0
+}
+
  /*
   * Allocate some memory and give it an address in the guest physical  

address
* space.

virt/kvm/kvm_main.c +0 -0
@@ -2128,7 +2142,8 @@ int __kvm_set_memory_region(struct kvm *kvm,
  	if (mem->flags & KVM_MEM_PRIVATE &&
  	    (mem->restrictedmem_offset & (PAGE_SIZE - 1) ||
  	     mem->restrictedmem_offset + mem->memory_size <  

mem->restrictedmem_offset ||

virt/kvm/kvm_main.c +2 -1
-	     0 /* TODO: require gfn be aligned with restricted offset */))
+	     !kvm_check_rmem_offset_alignment(mem->restrictedmem_offset,
+					      mem->guest_phys_addr)))
  		return -EINVAL;
  	if (as_id >= kvm_arch_nr_memslot_as_ids(kvm) || id >= KVM_MEM_SLOTS_NUM)
  		return -EINVAL;
author
#304 lore raw

Re: [PATCH v10 9/9] KVM: Enable and expose KVM_MEM_PRIVATE

Quoted 47 lines · Ackerley Tng
Quoted 32 lines · Chao Peng
Quoted 1 line · Sean Christopherson

On Fri, Dec 02, 2022, Chao Peng wrote:

...

Quoted 7 lines

Strongly prefer to use similar logic to existing code that detects wraps:

		mem->restricted_offset + mem->memory_size < mem->restricted_offset

This is also where I'd like to add the "gfn is aligned to offset" check, though
my brain is too fried to figure that out right now.

Used count_trailing_zeros() for this TODO, unsure we have other better approach.

virt/kvm/kvm_main.c +2 -0
index afc8c26fa652..fd34c5f7cd2f 100644
@@ -56,6 +56,7 @@
  #include <asm/processor.h>
  #include <asm/ioctl.h>
  #include <linux/uaccess.h>
+#include <linux/count_zeros.h>
 
  #include "coalesced_mmio.h"
  #include "async_pf.h"
@@ -2087,6 +2088,19 @@ static bool kvm_check_memslot_overlap(struct
 kvm_memslots *slots, int id,
  	return false;
  }
 
+/*

virt/kvm/kvm_main.c static bool kvm_check_memslot_overlap(struct · line 2102

Quoted 10 lines
virt/kvm/kvm_main.c +10 -0
+ * Return true when ALIGNMENT(offset) >= ALIGNMENT(gpa).
+ */
+static bool kvm_check_rmem_offset_alignment(u64 offset, u64 gpa)
+{
+	if (!offset)
+		return true;
+	if (!gpa)
+		return false;
+
+	return !!(count_trailing_zeros(offset) >= count_trailing_zeros(gpa));

Perhaps we could do something like

Quoted 5 lines

#define lowest_set_bit(val) (val & -val)

and use

return lowest_set_bit(offset) >= lowest_set_bit(gpa);

I see kernel already has fls64(), that looks what we need ;)

Quoted 4 lines

Please help me to understand: why must ALIGNMENT(offset) >= ALIGNMENT(gpa)? Why is it not sufficient to have both gpa and offset be aligned to PAGE_SIZE?

Yes, it's sufficient. Here we just want to be conservative on the uAPI as Sean explained this at [1]:

I would rather reject memslot if the gfn has lesser alignment than the offset. I'm totally ok with this approach _if_ there's a use case. Until such a use case presents itself, I would rather be conservative from a uAPI perspective.

[1] https://lore.kernel.org/all/[email protected]/

Chao

Quoted 18 lines
Quoted 18 lines
virt/kvm/kvm_main.c +2 -0
+}
+
  /*
   * Allocate some memory and give it an address in the guest physical

address
* space.

virt/kvm/kvm_main.c +0 -0
@@ -2128,7 +2142,8 @@ int __kvm_set_memory_region(struct kvm *kvm,
  	if (mem->flags & KVM_MEM_PRIVATE &&
  	    (mem->restrictedmem_offset & (PAGE_SIZE - 1) ||
  	     mem->restrictedmem_offset + mem->memory_size <

mem->restrictedmem_offset ||

virt/kvm/kvm_main.c +2 -1
-	     0 /* TODO: require gfn be aligned with restricted offset */))
+	     !kvm_check_rmem_offset_alignment(mem->restrictedmem_offset,
+					      mem->guest_phys_addr)))
  		return -EINVAL;
  	if (as_id >= kvm_arch_nr_memslot_as_ids(kvm) || id >= KVM_MEM_SLOTS_NUM)
  		return -EINVAL;
#305 lore raw

Re: [PATCH v10 9/9] KVM: Enable and expose KVM_MEM_PRIVATE

Quoted 34 lines · Chao Peng
Quoted 33 lines · Ackerley Tng
Quoted 30 lines · Chao Peng
Quoted 1 line · Sean Christopherson

On Fri, Dec 02, 2022, Chao Peng wrote:

...

Quoted 7 lines

Strongly prefer to use similar logic to existing code that detects wraps:

		mem->restricted_offset + mem->memory_size < mem->restricted_offset

This is also where I'd like to add the "gfn is aligned to offset" check, though
my brain is too fried to figure that out right now.

Used count_trailing_zeros() for this TODO, unsure we have other better approach.

virt/kvm/kvm_main.c +1 -0
index afc8c26fa652..fd34c5f7cd2f 100644
@@ -56,6 +56,7 @@
  #include <asm/processor.h>
  #include <asm/ioctl.h>
  #include <linux/uaccess.h>
+#include <linux/count_zeros.h>
 
  #include "coalesced_mmio.h"
  #include "async_pf.h"
@@ -2087,6 +2088,19 @@ static bool kvm_check_memslot_overlap(struct
 kvm_memslots *slots, int id,
  	return false;
  }
Quoted 11 lines
Quoted 11 lines

virt/kvm/kvm_main.c static bool kvm_check_memslot_overlap(struct · line 2101

Quoted 11 lines
virt/kvm/kvm_main.c +11 -0
+/*
+ * Return true when ALIGNMENT(offset) >= ALIGNMENT(gpa).
+ */
+static bool kvm_check_rmem_offset_alignment(u64 offset, u64 gpa)
+{
+	if (!offset)
+		return true;
+	if (!gpa)
+		return false;
+
+	return !!(count_trailing_zeros(offset) >= count_trailing_zeros(gpa));

This check doesn't work expected. For example, offset = 2GB, gpa=4GB this check fails.
I come up with the following.

From ec87e25082f0497431b732702fae82c6a05071bf Mon Sep 17 00:00:00 2001
Message-Id: <ec87e25082f0497431b732702fae82c6a05071bf.1679531995.git.isaku.yamahata@intel.com>
From: Isaku Yamahata <[email protected]>
Date: Wed, 22 Mar 2023 15:32:56 -0700
Subject: [PATCH] KVM: Relax alignment check for restricted mem

kvm_check_rmem_offset_alignment() only checks based on offset alignment and GPA alignment. However, the actual alignment for offset depends on architecture. For x86 case, it can be 1G, 2M or 4K. So even if GPA is aligned for 1G+, only 1G-alignment is required for offset.

Without this patch, gpa=4G, offset=2G results in failure of memory slot creation.

Fixes edc8814b2c77 ("KVM: Require gfn be aligned with restricted offset")
---
 arch/x86/include/asm/kvm_host.h | 15 +++++++++++++++
 virt/kvm/kvm_main.c             |  9 ++++++++-
 2 files changed, 23 insertions(+), 1 deletion(-)
arch/x86/include/asm/kvm_host.h +15 -0
index 88e11dd3afde..03af44650f24 100644
@@ -16,6 +16,7 @@
 #include <linux/irq_work.h>
 #include <linux/irq.h>
 #include <linux/workqueue.h>
+#include <linux/count_zeros.h>
 
 #include <linux/kvm.h>
 #include <linux/kvm_para.h>
@@ -143,6 +144,20 @@
 #define KVM_HPAGE_MASK(x)	(~(KVM_HPAGE_SIZE(x) - 1))
 #define KVM_PAGES_PER_HPAGE(x)	(KVM_HPAGE_SIZE(x) / PAGE_SIZE)
 
+#define kvm_arch_required_alignment	kvm_arch_required_alignment
+static inline int kvm_arch_required_alignment(u64 gpa)
+{
+	int zeros = count_trailing_zeros(gpa);
+
+	WARN_ON_ONCE(!PAGE_ALIGNED(gpa));
+	if (zeros >= KVM_HPAGE_SHIFT(PG_LEVEL_1G))
+		return KVM_HPAGE_SHIFT(PG_LEVEL_1G);
+	else if (zeros >= KVM_HPAGE_SHIFT(PG_LEVEL_2M))
+		return KVM_HPAGE_SHIFT(PG_LEVEL_2M);
+
+	return PAGE_SHIFT;
+}
+
 #define KVM_MEMSLOT_PAGES_TO_MMU_PAGES_RATIO 50
 #define KVM_MIN_ALLOC_MMU_PAGES 64UL
 #define KVM_MMU_HASH_SHIFT 12
virt/kvm/kvm_main.c +8 -1
index c9c4eef457b0..f4ff96171d24 100644
@@ -2113,6 +2113,13 @@ static bool kvm_check_memslot_overlap(struct kvm_memslots *slots, int id,
 	return false;
 }
 
+#ifndef kvm_arch_required_alignment
+__weak int kvm_arch_required_alignment(u64 gpa)
+{
+	return PAGE_SHIFT
+}
+#endif
+
 /*
  * Return true when ALIGNMENT(offset) >= ALIGNMENT(gpa).
  */
@@ -2123,7 +2130,7 @@ static bool kvm_check_rmem_offset_alignment(u64 offset, u64 gpa)
 	if (!gpa)
 		return false;
 
-	return !!(count_trailing_zeros(offset) >= count_trailing_zeros(gpa));
+	return !!(count_trailing_zeros(offset) >= kvm_arch_required_alignment(gpa));
 }
 
 /*
Signature · 6 lines
2.25.1



-- 
Isaku Yamahata <[email protected]>
author
#306 lore raw

Re: [PATCH v10 9/9] KVM: Enable and expose KVM_MEM_PRIVATE

Quoted 37 lines · Isaku Yamahata
Quoted 34 lines · Chao Peng
Quoted 33 lines · Ackerley Tng
Quoted 30 lines · Chao Peng
Quoted 1 line · Sean Christopherson

On Fri, Dec 02, 2022, Chao Peng wrote:

...

Quoted 7 lines

Strongly prefer to use similar logic to existing code that detects wraps:

		mem->restricted_offset + mem->memory_size < mem->restricted_offset

This is also where I'd like to add the "gfn is aligned to offset" check, though
my brain is too fried to figure that out right now.

Used count_trailing_zeros() for this TODO, unsure we have other better approach.

virt/kvm/kvm_main.c +1 -0
index afc8c26fa652..fd34c5f7cd2f 100644
@@ -56,6 +56,7 @@
  #include <asm/processor.h>
  #include <asm/ioctl.h>
  #include <linux/uaccess.h>
+#include <linux/count_zeros.h>
 
  #include "coalesced_mmio.h"
  #include "async_pf.h"
@@ -2087,6 +2088,19 @@ static bool kvm_check_memslot_overlap(struct
 kvm_memslots *slots, int id,
  	return false;
  }
Quoted 14 lines
Quoted 11 lines
Quoted 11 lines

virt/kvm/kvm_main.c static bool kvm_check_memslot_overlap(struct · line 2101

Quoted 11 lines
virt/kvm/kvm_main.c +11 -0
+/*
+ * Return true when ALIGNMENT(offset) >= ALIGNMENT(gpa).
+ */
+static bool kvm_check_rmem_offset_alignment(u64 offset, u64 gpa)
+{
+	if (!offset)
+		return true;
+	if (!gpa)
+		return false;
+
+	return !!(count_trailing_zeros(offset) >= count_trailing_zeros(gpa));

This check doesn't work expected. For example, offset = 2GB, gpa=4GB this check fails.

This case is expected to fail as Sean initially suggested[*]:
I would rather reject memslot if the gfn has lesser alignment than the offset. I'm totally ok with this approach _if_ there's a use case. Until such a use case presents itself, I would rather be conservative from a uAPI perspective.

I understand that we put tighter restriction on this but if you see such restriction is really a big issue for real usage, instead of a theoretical problem, then we can loosen the check here. But at that time below code is kind of x86 specific and may need improve.

BTW, in latest code, I replaced count_trailing_zeros() with fls64():
return !!(fls64(offset) >= fls64(gpa));

[*] https://lore.kernel.org/all/[email protected]/

Chao

Quoted 90 lines

I come up with the following.

Quoted 1 line

From ec87e25082f0497431b732702fae82c6a05071bf Mon Sep 17 00:00:00 2001

Message-Id: <ec87e25082f0497431b732702fae82c6a05071bf.1679531995.git.isaku.yamahata@intel.com>
From: Isaku Yamahata <[email protected]>
Date: Wed, 22 Mar 2023 15:32:56 -0700
Subject: [PATCH] KVM: Relax alignment check for restricted mem

kvm_check_rmem_offset_alignment() only checks based on offset alignment and GPA alignment. However, the actual alignment for offset depends on architecture. For x86 case, it can be 1G, 2M or 4K. So even if GPA is aligned for 1G+, only 1G-alignment is required for offset.

Without this patch, gpa=4G, offset=2G results in failure of memory slot creation.

Fixes edc8814b2c77 ("KVM: Require gfn be aligned with restricted offset")
---
 arch/x86/include/asm/kvm_host.h | 15 +++++++++++++++
 virt/kvm/kvm_main.c             |  9 ++++++++-
 2 files changed, 23 insertions(+), 1 deletion(-)
arch/x86/include/asm/kvm_host.h +15 -0
index 88e11dd3afde..03af44650f24 100644
@@ -16,6 +16,7 @@
 #include <linux/irq_work.h>
 #include <linux/irq.h>
 #include <linux/workqueue.h>
+#include <linux/count_zeros.h>
 
 #include <linux/kvm.h>
 #include <linux/kvm_para.h>
@@ -143,6 +144,20 @@
 #define KVM_HPAGE_MASK(x)	(~(KVM_HPAGE_SIZE(x) - 1))
 #define KVM_PAGES_PER_HPAGE(x)	(KVM_HPAGE_SIZE(x) / PAGE_SIZE)
 
+#define kvm_arch_required_alignment	kvm_arch_required_alignment
+static inline int kvm_arch_required_alignment(u64 gpa)
+{
+	int zeros = count_trailing_zeros(gpa);
+
+	WARN_ON_ONCE(!PAGE_ALIGNED(gpa));
+	if (zeros >= KVM_HPAGE_SHIFT(PG_LEVEL_1G))
+		return KVM_HPAGE_SHIFT(PG_LEVEL_1G);
+	else if (zeros >= KVM_HPAGE_SHIFT(PG_LEVEL_2M))
+		return KVM_HPAGE_SHIFT(PG_LEVEL_2M);
+
+	return PAGE_SHIFT;
+}
+
 #define KVM_MEMSLOT_PAGES_TO_MMU_PAGES_RATIO 50
 #define KVM_MIN_ALLOC_MMU_PAGES 64UL
 #define KVM_MMU_HASH_SHIFT 12
virt/kvm/kvm_main.c +8 -1
index c9c4eef457b0..f4ff96171d24 100644
@@ -2113,6 +2113,13 @@ static bool kvm_check_memslot_overlap(struct kvm_memslots *slots, int id,
 	return false;
 }
 
+#ifndef kvm_arch_required_alignment
+__weak int kvm_arch_required_alignment(u64 gpa)
+{
+	return PAGE_SHIFT
+}
+#endif
+
 /*
  * Return true when ALIGNMENT(offset) >= ALIGNMENT(gpa).
  */
@@ -2123,7 +2130,7 @@ static bool kvm_check_rmem_offset_alignment(u64 offset, u64 gpa)
 	if (!gpa)
 		return false;
 
-	return !!(count_trailing_zeros(offset) >= count_trailing_zeros(gpa));
+	return !!(count_trailing_zeros(offset) >= kvm_arch_required_alignment(gpa));
 }
 
 /*
Signature · 6 lines
2.25.1



-- 
Isaku Yamahata <[email protected]>
#307 lore raw

Re: [PATCH v10 9/9] KVM: Enable and expose KVM_MEM_PRIVATE

Quoted 59 lines · Chao Peng
Quoted 37 lines · Isaku Yamahata
Quoted 34 lines · Chao Peng
Quoted 33 lines · Ackerley Tng
Quoted 30 lines · Chao Peng
Quoted 1 line · Sean Christopherson

On Fri, Dec 02, 2022, Chao Peng wrote:

...

Quoted 7 lines

Strongly prefer to use similar logic to existing code that detects wraps:

		mem->restricted_offset + mem->memory_size < mem->restricted_offset

This is also where I'd like to add the "gfn is aligned to offset" check, though
my brain is too fried to figure that out right now.

Used count_trailing_zeros() for this TODO, unsure we have other better approach.

virt/kvm/kvm_main.c +1 -0
index afc8c26fa652..fd34c5f7cd2f 100644
@@ -56,6 +56,7 @@
   #include <asm/processor.h>
   #include <asm/ioctl.h>
   #include <linux/uaccess.h>
+#include <linux/count_zeros.h>
 
   #include "coalesced_mmio.h"
   #include "async_pf.h"
@@ -2087,6 +2088,19 @@ static bool kvm_check_memslot_overlap(struct
 kvm_memslots *slots, int id,
   	return false;
   }
Quoted 14 lines
Quoted 11 lines
Quoted 11 lines

virt/kvm/kvm_main.c static bool kvm_check_memslot_overlap(struct · line 2101

Quoted 11 lines
virt/kvm/kvm_main.c +11 -0
+/*
+ * Return true when ALIGNMENT(offset) >= ALIGNMENT(gpa).
+ */
+static bool kvm_check_rmem_offset_alignment(u64 offset, u64 gpa)
+{
+	if (!offset)
+		return true;
+	if (!gpa)
+		return false;
+
+	return !!(count_trailing_zeros(offset) >= count_trailing_zeros(gpa));

This check doesn't work expected. For example, offset = 2GB, gpa=4GB this check fails.

This case is expected to fail as Sean initially suggested[*]:
I would rather reject memslot if the gfn has lesser alignment than the offset. I'm totally ok with this approach _if_ there's a use case. Until such a use case presents itself, I would rather be conservative from a uAPI perspective.

Quoted 7 lines

I understand that we put tighter restriction on this but if you see such restriction is really a big issue for real usage, instead of a theoretical problem, then we can loosen the check here. But at that time below code is kind of x86 specific and may need improve.

BTW, in latest code, I replaced count_trailing_zeros() with fls64():

   return !!(fls64(offset) >= fls64(gpa));

wouldn't it be !!(ffs64(offset) <= ffs64(gpa)) ?

Quoted 93 lines

[*] https://lore.kernel.org/all/[email protected]/

Chao

Quoted 90 lines

I come up with the following.

Quoted 1 line

From ec87e25082f0497431b732702fae82c6a05071bf Mon Sep 17 00:00:00 2001

Message-Id: <ec87e25082f0497431b732702fae82c6a05071bf.1679531995.git.isaku.yamahata@intel.com>
From: Isaku Yamahata <[email protected]>
Date: Wed, 22 Mar 2023 15:32:56 -0700
Subject: [PATCH] KVM: Relax alignment check for restricted mem

kvm_check_rmem_offset_alignment() only checks based on offset alignment and GPA alignment. However, the actual alignment for offset depends on architecture. For x86 case, it can be 1G, 2M or 4K. So even if GPA is aligned for 1G+, only 1G-alignment is required for offset.

Without this patch, gpa=4G, offset=2G results in failure of memory slot creation.

Fixes edc8814b2c77 ("KVM: Require gfn be aligned with restricted offset")
---
  arch/x86/include/asm/kvm_host.h | 15 +++++++++++++++
  virt/kvm/kvm_main.c             |  9 ++++++++-
  2 files changed, 23 insertions(+), 1 deletion(-)
arch/x86/include/asm/kvm_host.h +15 -0
index 88e11dd3afde..03af44650f24 100644
@@ -16,6 +16,7 @@
  #include <linux/irq_work.h>
  #include <linux/irq.h>
  #include <linux/workqueue.h>
+#include <linux/count_zeros.h>
 
  #include <linux/kvm.h>
  #include <linux/kvm_para.h>
@@ -143,6 +144,20 @@
  #define KVM_HPAGE_MASK(x)	(~(KVM_HPAGE_SIZE(x) - 1))
  #define KVM_PAGES_PER_HPAGE(x)	(KVM_HPAGE_SIZE(x) / PAGE_SIZE)
 
+#define kvm_arch_required_alignment	kvm_arch_required_alignment
+static inline int kvm_arch_required_alignment(u64 gpa)
+{
+	int zeros = count_trailing_zeros(gpa);
+
+	WARN_ON_ONCE(!PAGE_ALIGNED(gpa));
+	if (zeros >= KVM_HPAGE_SHIFT(PG_LEVEL_1G))
+		return KVM_HPAGE_SHIFT(PG_LEVEL_1G);
+	else if (zeros >= KVM_HPAGE_SHIFT(PG_LEVEL_2M))
+		return KVM_HPAGE_SHIFT(PG_LEVEL_2M);
+
+	return PAGE_SHIFT;
+}
+
  #define KVM_MEMSLOT_PAGES_TO_MMU_PAGES_RATIO 50
  #define KVM_MIN_ALLOC_MMU_PAGES 64UL
  #define KVM_MMU_HASH_SHIFT 12
virt/kvm/kvm_main.c +8 -1
index c9c4eef457b0..f4ff96171d24 100644
@@ -2113,6 +2113,13 @@ static bool kvm_check_memslot_overlap(struct kvm_memslots *slots, int id,
  	return false;
  }
 
+#ifndef kvm_arch_required_alignment
+__weak int kvm_arch_required_alignment(u64 gpa)
+{
+	return PAGE_SHIFT
+}
+#endif
+
  /*
   * Return true when ALIGNMENT(offset) >= ALIGNMENT(gpa).
   */
@@ -2123,7 +2130,7 @@ static bool kvm_check_rmem_offset_alignment(u64 offset, u64 gpa)
  	if (!gpa)
  		return false;
 
-	return !!(count_trailing_zeros(offset) >= count_trailing_zeros(gpa));
+	return !!(count_trailing_zeros(offset) >= kvm_arch_required_alignment(gpa));
  }
 
  /*
Signature · 6 lines
2.25.1



-- 
Isaku Yamahata <[email protected]>
author
#308 lore raw

Re: [PATCH v10 9/9] KVM: Enable and expose KVM_MEM_PRIVATE

Quoted 65 lines · Xiaoyao Li
Quoted 64 lines · Chao Peng
Quoted 37 lines · Isaku Yamahata
Quoted 34 lines · Chao Peng
Quoted 33 lines · Ackerley Tng
Quoted 30 lines · Chao Peng
Quoted 1 line · Sean Christopherson

On Fri, Dec 02, 2022, Chao Peng wrote:

...

Quoted 7 lines

Strongly prefer to use similar logic to existing code that detects wraps:

		mem->restricted_offset + mem->memory_size < mem->restricted_offset

This is also where I'd like to add the "gfn is aligned to offset" check, though
my brain is too fried to figure that out right now.

Used count_trailing_zeros() for this TODO, unsure we have other better approach.

virt/kvm/kvm_main.c +1 -0
index afc8c26fa652..fd34c5f7cd2f 100644
@@ -56,6 +56,7 @@
   #include <asm/processor.h>
   #include <asm/ioctl.h>
   #include <linux/uaccess.h>
+#include <linux/count_zeros.h>
 
   #include "coalesced_mmio.h"
   #include "async_pf.h"
@@ -2087,6 +2088,19 @@ static bool kvm_check_memslot_overlap(struct
 kvm_memslots *slots, int id,
   	return false;
   }
Quoted 14 lines
Quoted 11 lines
Quoted 11 lines

virt/kvm/kvm_main.c static bool kvm_check_memslot_overlap(struct · line 2101

Quoted 11 lines
virt/kvm/kvm_main.c +11 -0
+/*
+ * Return true when ALIGNMENT(offset) >= ALIGNMENT(gpa).
+ */
+static bool kvm_check_rmem_offset_alignment(u64 offset, u64 gpa)
+{
+	if (!offset)
+		return true;
+	if (!gpa)
+		return false;
+
+	return !!(count_trailing_zeros(offset) >= count_trailing_zeros(gpa));

This check doesn't work expected. For example, offset = 2GB, gpa=4GB this check fails.

This case is expected to fail as Sean initially suggested[*]:
I would rather reject memslot if the gfn has lesser alignment than the offset. I'm totally ok with this approach _if_ there's a use case. Until such a use case presents itself, I would rather be conservative from a uAPI perspective.

I understand that we put tighter restriction on this but if you see such restriction is really a big issue for real usage, instead of a theoretical problem, then we can loosen the check here. But at that time below code is kind of x86 specific and may need improve.

Quoted 4 lines
Quoted 2 lines

BTW, in latest code, I replaced count_trailing_zeros() with fls64():

   return !!(fls64(offset) >= fls64(gpa));

wouldn't it be !!(ffs64(offset) <= ffs64(gpa)) ?

As the function document explains, here we want to return true when ALIGNMENT(offset) >= ALIGNMENT(gpa), so '>=' is what we need.

It's worthy clarifying that in Sean's original suggestion he actually mentioned the opposite. He said 'reject memslot if the gfn has lesser alignment than the offset', but I wonder this is his purpose, since if ALIGNMENT(offset) < ALIGNMENT(gpa), we wouldn't be possible to map the page as largepage. Consider we have below config:

gpa=2M, offset=1M

In this case KVM tries to map gpa at 2M as 2M hugepage but the physical page at the offset(1M) in private_fd cannot provide the 2M page due to misalignment.

But as we discussed in the off-list thread, here we do find a real use case indicating this check is too strict. i.e. QEMU immediately fails when launch a guest > 2G memory. For this case QEMU splits guest memory space into two slots:

Slot#1(ram_below_4G): gpa=0x0, offset=0x0, size=2G Slot#2(ram_above_4G): gpa=4G, offset=2G, size=totalsize-2G

This strict alignment check fails for slot#2 because offset(2G) has less alignment than gpa(4G). To allow this, one solution can revert to my previous change in kvm_alloc_memslot_metadata() to disallow hugepage only when the offset/gpa are not aligned to related page size.

Sean, How do you think?

Chao

Quoted 89 lines
Quoted 89 lines

[*] https://lore.kernel.org/all/[email protected]/

Chao

Quoted 85 lines

I come up with the following.

Quoted 1 line

From ec87e25082f0497431b732702fae82c6a05071bf Mon Sep 17 00:00:00 2001

Message-Id: <ec87e25082f0497431b732702fae82c6a05071bf.1679531995.git.isaku.yamahata@intel.com>
From: Isaku Yamahata <[email protected]>
Date: Wed, 22 Mar 2023 15:32:56 -0700
Subject: [PATCH] KVM: Relax alignment check for restricted mem

kvm_check_rmem_offset_alignment() only checks based on offset alignment and GPA alignment. However, the actual alignment for offset depends on architecture. For x86 case, it can be 1G, 2M or 4K. So even if GPA is aligned for 1G+, only 1G-alignment is required for offset.

Without this patch, gpa=4G, offset=2G results in failure of memory slot creation.

Fixes edc8814b2c77 ("KVM: Require gfn be aligned with restricted offset")
---
  arch/x86/include/asm/kvm_host.h | 15 +++++++++++++++
  virt/kvm/kvm_main.c             |  9 ++++++++-
  2 files changed, 23 insertions(+), 1 deletion(-)
arch/x86/include/asm/kvm_host.h +15 -0
index 88e11dd3afde..03af44650f24 100644
@@ -16,6 +16,7 @@
  #include <linux/irq_work.h>
  #include <linux/irq.h>
  #include <linux/workqueue.h>
+#include <linux/count_zeros.h>
  #include <linux/kvm.h>
  #include <linux/kvm_para.h>
@@ -143,6 +144,20 @@
  #define KVM_HPAGE_MASK(x)	(~(KVM_HPAGE_SIZE(x) - 1))
  #define KVM_PAGES_PER_HPAGE(x)	(KVM_HPAGE_SIZE(x) / PAGE_SIZE)
+#define kvm_arch_required_alignment	kvm_arch_required_alignment
+static inline int kvm_arch_required_alignment(u64 gpa)
+{
+	int zeros = count_trailing_zeros(gpa);
+
+	WARN_ON_ONCE(!PAGE_ALIGNED(gpa));
+	if (zeros >= KVM_HPAGE_SHIFT(PG_LEVEL_1G))
+		return KVM_HPAGE_SHIFT(PG_LEVEL_1G);
+	else if (zeros >= KVM_HPAGE_SHIFT(PG_LEVEL_2M))
+		return KVM_HPAGE_SHIFT(PG_LEVEL_2M);
+
+	return PAGE_SHIFT;
+}
+
  #define KVM_MEMSLOT_PAGES_TO_MMU_PAGES_RATIO 50
  #define KVM_MIN_ALLOC_MMU_PAGES 64UL
  #define KVM_MMU_HASH_SHIFT 12
virt/kvm/kvm_main.c +8 -1
index c9c4eef457b0..f4ff96171d24 100644
@@ -2113,6 +2113,13 @@ static bool kvm_check_memslot_overlap(struct kvm_memslots *slots, int id,
  	return false;
  }
+#ifndef kvm_arch_required_alignment
+__weak int kvm_arch_required_alignment(u64 gpa)
+{
+	return PAGE_SHIFT
+}
+#endif
+
  /*
   * Return true when ALIGNMENT(offset) >= ALIGNMENT(gpa).
   */
@@ -2123,7 +2130,7 @@ static bool kvm_check_rmem_offset_alignment(u64 offset, u64 gpa)
  	if (!gpa)
  		return false;
-	return !!(count_trailing_zeros(offset) >= count_trailing_zeros(gpa));
+	return !!(count_trailing_zeros(offset) >= kvm_arch_required_alignment(gpa));
  }
  /*
Signature · 6 lines
2.25.1



-- 
Isaku Yamahata <[email protected]>
#309 lore raw

Re: [PATCH v10 9/9] KVM: Enable and expose KVM_MEM_PRIVATE

Quoted 62 lines · Chao Peng
Quoted 33 lines · Xiaoyao Li
Quoted 32 lines · Chao Peng
Quoted 8 lines · Isaku Yamahata
Quoted 5 lines · Chao Peng
Quoted 4 lines · Ackerley Tng
Quoted 2 lines · Chao Peng
Quoted 1 line · Sean Christopherson

On Fri, Dec 02, 2022, Chao Peng wrote:

Quoted 11 lines
Quoted 8 lines
Quoted 8 lines

Review

Quoted 8 lines
(quoted diff) +8 -0
+static bool kvm_check_rmem_offset_alignment(u64 offset, u64 gpa)
+{
+	if (!offset)
+		return true;
+	if (!gpa)
+		return false;
+
+	return !!(count_trailing_zeros(offset) >= count_trailing_zeros(gpa));

This check doesn't work expected. For example, offset = 2GB, gpa=4GB this check fails.

This case is expected to fail as Sean initially suggested[*]:
I would rather reject memslot if the gfn has lesser alignment than the offset. I'm totally ok with this approach _if_ there's a use case. Until such a use case presents itself, I would rather be conservative from a uAPI perspective.

I understand that we put tighter restriction on this but if you see such restriction is really a big issue for real usage, instead of a theoretical problem, then we can loosen the check here. But at that time below code is kind of x86 specific and may need improve.

Quoted 4 lines
Quoted 2 lines

BTW, in latest code, I replaced count_trailing_zeros() with fls64():

   return !!(fls64(offset) >= fls64(gpa));

wouldn't it be !!(ffs64(offset) <= ffs64(gpa)) ?

As the function document explains, here we want to return true when ALIGNMENT(offset) >= ALIGNMENT(gpa), so '>=' is what we need.

It's worthy clarifying that in Sean's original suggestion he actually mentioned the opposite. He said 'reject memslot if the gfn has lesser alignment than the offset', but I wonder this is his purpose, since if ALIGNMENT(offset) < ALIGNMENT(gpa), we wouldn't be possible to map the page as largepage. Consider we have below config:

gpa=2M, offset=1M

In this case KVM tries to map gpa at 2M as 2M hugepage but the physical page at the offset(1M) in private_fd cannot provide the 2M page due to misalignment.

But as we discussed in the off-list thread, here we do find a real use case indicating this check is too strict. i.e. QEMU immediately fails when launch a guest > 2G memory. For this case QEMU splits guest memory space into two slots:

Slot#1(ram_below_4G): gpa=0x0, offset=0x0, size=2G Slot#2(ram_above_4G): gpa=4G, offset=2G, size=totalsize-2G

Quoted 6 lines

This strict alignment check fails for slot#2 because offset(2G) has less alignment than gpa(4G). To allow this, one solution can revert to my previous change in kvm_alloc_memslot_metadata() to disallow hugepage only when the offset/gpa are not aligned to related page size.

Sean, How do you think?

I agree, a pure alignment check is too restrictive, and not really what I intended despite past me literally saying that's what I wanted :-) I think I may have also inverted the "less alignment" statement, but luckily I believe that ends up being a moot point.

The goal is to avoid having to juggle scenarios where KVM wants to create a hugepage, but restrictedmem can't provide one because of a misaligned file offset. I think the rule we want is that the offset must be aligned to the largest page size allowed by the memslot _size_. E.g. on x86, if the memslot size is >=1GiB then the offset must be 1GiB or beter, ditto for >=2MiB and >=4KiB (ignoring that 4KiB is already a requirement).

We could loosen that to say the largest size allowed by the memslot, but I don't think that's worth the effort unless it's trivially easy to implement in code, e.g. KVM could technically allow a 4KiB aligned offset if the memslot is 2MiB sized but only 4KiB aligned on the GPA. I doubt there's a real use case for such a memslot, so I want to disallow that unless it's super easy to implement.

#310 lore raw

Re: [PATCH v10 9/9] KVM: Enable and expose KVM_MEM_PRIVATE

Quoted 90 lines · Sean Christopherson
Quoted 70 lines · Chao Peng
Quoted 9 lines · Xiaoyao Li
Quoted 8 lines · Chao Peng
Quoted 7 lines · Isaku Yamahata
Quoted 4 lines · Chao Peng
Quoted 3 lines · Ackerley Tng
Quoted 1 line · Chao Peng

On Sat, Jan 14, 2023 at 12:01:01AM +0000, Sean

Quoted 2 lines · Christopherson
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line · Chao Peng

+static bool kvm_check_rmem_offset_alignment(u64 offset, u64

gpa)

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Review

Quoted 7 lines
(quoted diff) +7 -0
+{
+	if (!offset)
+		return true;
+	if (!gpa)
+		return false;
+
+	return !!(count_trailing_zeros(offset) >=  

count_trailing_zeros(gpa));

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

This check doesn't work expected. For example, offset = 2GB,

gpa=4GB

Quoted 5 lines
Quoted 5 lines
Quoted 1 line

this check fails.

This case is expected to fail as Sean initially suggested[*]:
I would rather reject memslot if the gfn has lesser alignment than the offset. I'm totally ok with this approach _if_ there's a use

case.

Quoted 1 line
Quoted 1 line

Until such a use case presents itself, I would rather be

conservative

Quoted 3 lines
Quoted 3 lines

from a uAPI perspective.

I understand that we put tighter restriction on this but if you see

such

Quoted 2 lines
Quoted 2 lines

restriction is really a big issue for real usage, instead of a theoretical problem, then we can loosen the check here. But at that

time

Quoted 6 lines
Quoted 4 lines

below code is kind of x86 specific and may need improve.

BTW, in latest code, I replaced count_trailing_zeros() with fls64():

   return !!(fls64(offset) >= fls64(gpa));

wouldn't it be !!(ffs64(offset) <= ffs64(gpa)) ?

As the function document explains, here we want to return true when ALIGNMENT(offset) >= ALIGNMENT(gpa), so '>=' is what we need.

It's worthy clarifying that in Sean's original suggestion he actually mentioned the opposite. He said 'reject memslot if the gfn has lesser alignment than the offset', but I wonder this is his purpose, since if ALIGNMENT(offset) < ALIGNMENT(gpa), we wouldn't be possible to map the page as largepage. Consider we have below config:

   gpa=2M, offset=1M

In this case KVM tries to map gpa at 2M as 2M hugepage but the physical page at the offset(1M) in private_fd cannot provide the 2M page due to misalignment.

But as we discussed in the off-list thread, here we do find a real use case indicating this check is too strict. i.e. QEMU immediately fails when launch a guest > 2G memory. For this case QEMU splits guest memory space into two slots:

   Slot#1(ram_below_4G): gpa=0x0, offset=0x0, size=2G
   Slot#2(ram_above_4G): gpa=4G,  offset=2G,  size=totalsize-2G
Quoted 6 lines

This strict alignment check fails for slot#2 because offset(2G) has less alignment than gpa(4G). To allow this, one solution can revert to my previous change in kvm_alloc_memslot_metadata() to disallow hugepage only when the offset/gpa are not aligned to related page size.

Sean, How do you think?

I agree, a pure alignment check is too restrictive, and not really what I intended despite past me literally saying that's what I wanted :-) I think I may have also inverted the "less alignment" statement, but luckily I believe that ends up being a moot point.

The goal is to avoid having to juggle scenarios where KVM wants to create a hugepage, but restrictedmem can't provide one because of a misaligned file offset. I think the rule we want is that the offset must be aligned to the largest page size allowed by the memslot _size_. E.g. on x86, if the memslot size is >=1GiB then the offset must be 1GiB or beter, ditto for >=2MiB and >=4KiB (ignoring that 4KiB is already a requirement).

Quoted 5 lines

We could loosen that to say the largest size allowed by the memslot, but I don't think that's worth the effort unless it's trivially easy to implement in code, e.g. KVM could technically allow a 4KiB aligned offset if the memslot is 2MiB sized but only 4KiB aligned on the GPA. I doubt there's a real use case for such a memslot, so I want to disallow that unless it's super easy to implement.

Checking my understanding here about why we need this alignment check:

When KVM requests a page from restrictedmem, KVM will provide an offset into the file in terms of 4K pages.

When shmem is configured to use hugepages, shmem_get_folio() will round the requested offset down to the nearest hugepage-aligned boundary in shmem_alloc_hugefolio().

Example of problematic configuration provided to
KVM_SET_USER_MEMORY_REGION2:

+ shmem configured to use 1GB pages
+ restrictedmem_offset provided to KVM_SET_USER_MEMORY_REGION2: 0x4000
+ memory_size provided in KVM_SET_USER_MEMORY_REGION2: 1GB
+ KVM requests offset (pgoff_t) 0x8, which translates to offset 0x8000

restrictedmem_get_page() and shmem_get_folio() returns the page for offset 0x0 in the file, since rounding down 0x8000 to the nearest 1GB is 0x0. This is allocating outside the range that KVM is supposed to use, since the parameters provided in KVM_SET_USER_MEMORY_REGION2 is only supposed to be offset 0x4000 to (0x4000 + 1GB = 0x40004000) in the file.

IIUC shmem will actually just round down (0x4000 rounded down to nearest 1GB will be 0x0) and allocate without checking bounds, so if offset 0x0 to 0x4000 in the file were supposed to be used by something else, there might be issues.

Hence, this alignment check ensures that rounding down of any offsets provided by KVM (based on page size configured in the backing file provided) to restrictedmem_get_page() must not go below the offset provided to KVM_SET_USER_MEMORY_REGION2.

Enforcing alignment of restrictedmem_offset based on the currently-set page size in the backing file (i.e. shmem) may not be effective, since the size of the pages in the backing file can be adjusted to a larger size after KVM_SET_USER_MEMORY_REGION2 succeeds. With that, we may still end up allocating outside the range that KVM was provided with.

Hence, to be safe, we should check alignment to the max page size across all backing filesystems, so the constraint is

rounding down restrictedmem_offset to
min(max page size across all backing filesystems,
max page size that fits in memory_size) == restrictedmem_offset

which is the same check as

restrictedmem_offset must be aligned to min(max page size across all backing filesystems, max page size that fits in memory_size)

which can safely reduce to

restrictedmem_offset must be aligned to max page size that fits in memory_size

since "max page size that fits in memory_size" is probably <= to "max page size across all backing filesystems", and if it's larger, it'll just be a tighter constraint.

If the above understanding is correct:

+ We must enforce this in the KVM_SET_USER_MEMORY_REGION2 handler, since IIUC shmem will just round down and allocate without checking bounds.

+ I think this is okay because holes in the restrictedmem file (in terms of offset) made to accommodate this constraint don't cost us anything anyway(?) Are they just arbitrary offsets in a file? In our case, this file is usually a new and empty file.

+ In the case of migration of a restrictedmem file between two KVM VMs, this constraint would cause a problem is if the largest possible page size on the destination machine is larger than that of the source machine. In that case, we might have to move the data in the file to a different offset (a separate problem).

+ On this note, it seems like there is no check for when the range is smaller than the allocated page? Like if the range provided is 4KB in size, but shmem is then configured to use a 1GB page, will we end up allocating past the end of the range?

#311 lore raw

Re: [PATCH v10 9/9] KVM: Enable and expose KVM_MEM_PRIVATE

Quoted 71 lines · Ackerley Tng
Quoted 10 lines · Sean Christopherson

I agree, a pure alignment check is too restrictive, and not really what I intended despite past me literally saying that's what I wanted :-) I think I may have also inverted the "less alignment" statement, but luckily I believe that ends up being a moot point.

The goal is to avoid having to juggle scenarios where KVM wants to create a hugepage, but restrictedmem can't provide one because of a misaligned file offset. I think the rule we want is that the offset must be aligned to the largest page size allowed by the memslot _size_. E.g. on x86, if the memslot size is >=1GiB then the offset must be 1GiB or beter, ditto for

Quoted 8 lines
Quoted 1 line

=2MiB and >=4KiB (ignoring that 4KiB is already a requirement).

We could loosen that to say the largest size allowed by the memslot, but I don't think that's worth the effort unless it's trivially easy to implement in code, e.g. KVM could technically allow a 4KiB aligned offset if the memslot is 2MiB sized but only 4KiB aligned on the GPA. I doubt there's a real use case for such a memslot, so I want to disallow that unless it's super easy to implement.

Checking my understanding here about why we need this alignment check:

When KVM requests a page from restrictedmem, KVM will provide an offset into the file in terms of 4K pages.

When shmem is configured to use hugepages, shmem_get_folio() will round the requested offset down to the nearest hugepage-aligned boundary in shmem_alloc_hugefolio().

Example of problematic configuration provided to
KVM_SET_USER_MEMORY_REGION2:

+ shmem configured to use 1GB pages
+ restrictedmem_offset provided to KVM_SET_USER_MEMORY_REGION2: 0x4000
+ memory_size provided in KVM_SET_USER_MEMORY_REGION2: 1GB
+ KVM requests offset (pgoff_t) 0x8, which translates to offset 0x8000

restrictedmem_get_page() and shmem_get_folio() returns the page for offset 0x0 in the file, since rounding down 0x8000 to the nearest 1GB is 0x0. This is allocating outside the range that KVM is supposed to use, since the parameters provided in KVM_SET_USER_MEMORY_REGION2 is only supposed to be offset 0x4000 to (0x4000 + 1GB = 0x40004000) in the file.

IIUC shmem will actually just round down (0x4000 rounded down to nearest 1GB will be 0x0) and allocate without checking bounds, so if offset 0x0 to 0x4000 in the file were supposed to be used by something else, there might be issues.

Hence, this alignment check ensures that rounding down of any offsets provided by KVM (based on page size configured in the backing file provided) to restrictedmem_get_page() must not go below the offset provided to KVM_SET_USER_MEMORY_REGION2.

Enforcing alignment of restrictedmem_offset based on the currently-set page size in the backing file (i.e. shmem) may not be effective, since the size of the pages in the backing file can be adjusted to a larger size after KVM_SET_USER_MEMORY_REGION2 succeeds. With that, we may still end up allocating outside the range that KVM was provided with.

Hence, to be safe, we should check alignment to the max page size across all backing filesystems, so the constraint is

rounding down restrictedmem_offset to
min(max page size across all backing filesystems,
max page size that fits in memory_size) == restrictedmem_offset

which is the same check as

restrictedmem_offset must be aligned to min(max page size across all backing filesystems, max page size that fits in memory_size)

Quoted 8 lines

which can safely reduce to

restrictedmem_offset must be aligned to max page size that fits in memory_size

since "max page size that fits in memory_size" is probably <= to "max page size across all backing filesystems", and if it's larger, it'll just be a tighter constraint.

Yes? The alignment check isn't strictly required, KVM _could_ deal with the above scenario, it's just a lot simpler and safer for KVM if the file offset needs to be sanely aligned.

Quoted 10 lines

If the above understanding is correct:

+ We must enforce this in the KVM_SET_USER_MEMORY_REGION2 handler, since IIUC shmem will just round down and allocate without checking bounds.

+ I think this is okay because holes in the restrictedmem file (in terms of offset) made to accommodate this constraint don't cost us anything anyway(?) Are they just arbitrary offsets in a file? In our case, this file is usually a new and empty file.

Quoted 5 lines

+ In the case of migration of a restrictedmem file between two KVM VMs, this constraint would cause a problem is if the largest possible page size on the destination machine is larger than that of the source machine. In that case, we might have to move the data in the file to a different offset (a separate problem).

Hmm, I was thinking this would be a non-issue because the check would be tied to the max page _possible_ page size irrespective of hardware support, but that would be problematic if KVM ever supports 512GiB pages. I'm not sure that speculatively requiring super huge memslots to be 512GiB aligned is sensible.

Aha! If we go with a KVM ioctl(), a clean way around this is tie the alignment requirement to the memfd flags, e.g. if userspace requests the memfd to be backed by PMD hugepages, then the memslot offset needs to be 2MiB aligned on x86. That will continue to work if (big if) KVM supports 512GiB pages because the "legacy" memfd would still be capped at 2MiB pages.

Architectures that support variable hugepage sizes might need to do something else, but I don't think that possibility affects what x86 can/can't do.

Quoted 4 lines

+ On this note, it seems like there is no check for when the range is smaller than the allocated page? Like if the range provided is 4KB in size, but shmem is then configured to use a 1GB page, will we end up allocating past the end of the range?

No, KVM already gracefully handles situations like this. Well, x86 does, I assume other architectures do too :-)

As above, the intent of the extra restriction is so that KVM doen't need even more weird code (read: math) to gracefully handle the new edge cases that would come with fd-only memslots.

#312 lore raw

Re: [PATCH v10 9/9] KVM: Enable and expose KVM_MEM_PRIVATE

In reply to Chao Peng
Quoted 11 lines · Chao Peng

Register/unregister private memslot to fd-based memory backing store restrictedmem and implement the callbacks for restrictedmem_notifier:
- invalidate_start()/invalidate_end() to zap the existing memory mappings in the KVM page table.
- error() to request KVM_REQ_MEMORY_MCE and later exit to userspace with KVM_EXIT_SHUTDOWN.

Expose KVM_MEM_PRIVATE for memslot and KVM_MEMORY_ATTRIBUTE_PRIVATE for KVM_GET_SUPPORTED_MEMORY_ATTRIBUTES to userspace but either are controlled by kvm_arch_has_private_mem() which should be rewritten by architecture code.

Could we perhaps rename KVM_MEM_PRIVATE to KVM_MEM_PROTECTED, to be in line with KVM_X86_PROTECTED_VM?

I feel that a memslot that has the KVM_MEM_PRIVATE flag need not always be private; It can sometimes be providing memory that is shared and also accessible from the host.

KVM_MEMORY_ATTRIBUTE_PRIVATE is fine as-is because this flag is set when the guest memory is meant to be backed by private memory.

KVM_MEMORY_EXIT_FLAG_PRIVATE is also okay because the flag is used to indicate when the memory error is caused by a private access (as opposed to a shared access).

kvm_slot_can_be_private() could perhaps be renamed kvm_is_protected_slot()?

Quoted 368 lines
---
  arch/x86/include/asm/kvm_host.h |   1 +
  arch/x86/kvm/x86.c              |  13 +++
  include/linux/kvm_host.h        |   3 +
  virt/kvm/kvm_main.c             | 179 +++++++++++++++++++++++++++++++-
  4 files changed, 191 insertions(+), 5 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -0
index 7772ab37ac89..27ef31133352 100644
@@ -114,6 +114,7 @@
  	KVM_ARCH_REQ_FLAGS(31, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
  #define KVM_REQ_HV_TLB_FLUSH \
  	KVM_ARCH_REQ_FLAGS(32, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
+#define KVM_REQ_MEMORY_MCE		KVM_ARCH_REQ(33)
 
  #define CR0_RESERVED_BITS                                               \
  	(~(unsigned long)(X86_CR0_PE | X86_CR0_MP | X86_CR0_EM | X86_CR0_TS \
arch/x86/kvm/x86.c +7 -0
index 5aefcff614d2..c67e22f3e2ee 100644
@@ -6587,6 +6587,13 @@ int kvm_arch_pm_notifier(struct kvm *kvm, unsigned  
 long state)
  }
  #endif /* CONFIG_HAVE_KVM_PM_NOTIFIER */
 
+#ifdef CONFIG_HAVE_KVM_RESTRICTED_MEM
+void kvm_arch_memory_mce(struct kvm *kvm)
+{
+	kvm_make_all_cpus_request(kvm, KVM_REQ_MEMORY_MCE);
+}
+#endif
+
  static int kvm_vm_ioctl_get_clock(struct kvm *kvm, void __user *argp)
  {
  	struct kvm_clock_data data = { 0 };
arch/x86/kvm/x86.c +6 -0
@@ -10357,6 +10364,12 @@ static int vcpu_enter_guest(struct kvm_vcpu  
 *vcpu)
 
  		if (kvm_check_request(KVM_REQ_UPDATE_CPU_DIRTY_LOGGING, vcpu))
  			static_call(kvm_x86_update_cpu_dirty_logging)(vcpu);
+
+		if (kvm_check_request(KVM_REQ_MEMORY_MCE, vcpu)) {
+			vcpu->run->exit_reason = KVM_EXIT_SHUTDOWN;
+			r = 0;
+			goto out;
+		}
  	}
 
  	if (kvm_check_request(KVM_REQ_EVENT, vcpu) || req_int_win ||
include/linux/kvm_host.h +1 -0
index 153842bb33df..f032d878e034 100644
@@ -590,6 +590,7 @@ struct kvm_memory_slot {
  	struct file *restricted_file;
  	loff_t restricted_offset;
  	struct restrictedmem_notifier notifier;
+	struct kvm *kvm;
  };
 
  static inline bool kvm_slot_can_be_private(const struct kvm_memory_slot  

*slot)

include/linux/kvm_host.h +2 -0
@@ -2363,6 +2364,8 @@ static inline int kvm_restricted_mem_get_pfn(struct  
 kvm_memory_slot *slot,
  	*pfn = page_to_pfn(page);
  	return ret;
  }
+
+void kvm_arch_memory_mce(struct kvm *kvm);
  #endif /* CONFIG_HAVE_KVM_RESTRICTED_MEM */
 
  #endif
virt/kvm/kvm_main.c +84 -0
index e107afea32f0..ac835fc77273 100644
@@ -936,6 +936,121 @@ static int kvm_init_mmu_notifier(struct kvm *kvm)
 
  #endif /* CONFIG_MMU_NOTIFIER && KVM_ARCH_WANT_MMU_NOTIFIER */
 
+#ifdef CONFIG_HAVE_KVM_RESTRICTED_MEM
+static bool restrictedmem_range_is_valid(struct kvm_memory_slot *slot,
+					 pgoff_t start, pgoff_t end,
+					 gfn_t *gfn_start, gfn_t *gfn_end)
+{
+	unsigned long base_pgoff = slot->restricted_offset >> PAGE_SHIFT;
+
+	if (start > base_pgoff)
+		*gfn_start = slot->base_gfn + start - base_pgoff;
+	else
+		*gfn_start = slot->base_gfn;
+
+	if (end < base_pgoff + slot->npages)
+		*gfn_end = slot->base_gfn + end - base_pgoff;
+	else
+		*gfn_end = slot->base_gfn + slot->npages;
+
+	if (*gfn_start >= *gfn_end)
+		return false;
+
+	return true;
+}
+
+static void kvm_restrictedmem_invalidate_begin(struct  
 restrictedmem_notifier *notifier,
+					       pgoff_t start, pgoff_t end)
+{
+	struct kvm_memory_slot *slot = container_of(notifier,
+						    struct kvm_memory_slot,
+						    notifier);
+	struct kvm *kvm = slot->kvm;
+	gfn_t gfn_start, gfn_end;
+	struct kvm_gfn_range gfn_range;
+	int idx;
+
+	if (!restrictedmem_range_is_valid(slot, start, end,
+					  &gfn_start, &gfn_end))
+		return;
+
+	gfn_range.start = gfn_start;
+	gfn_range.end = gfn_end;
+	gfn_range.slot = slot;
+	gfn_range.pte = __pte(0);
+	gfn_range.may_block = true;
+
+	idx = srcu_read_lock(&kvm->srcu);
+	KVM_MMU_LOCK(kvm);
+
+	kvm_mmu_invalidate_begin(kvm);
+	kvm_mmu_invalidate_range_add(kvm, gfn_start, gfn_end);
+	if (kvm_unmap_gfn_range(kvm, &gfn_range))
+		kvm_flush_remote_tlbs(kvm);
+
+	KVM_MMU_UNLOCK(kvm);
+	srcu_read_unlock(&kvm->srcu, idx);
+}
+
+static void kvm_restrictedmem_invalidate_end(struct  
 restrictedmem_notifier *notifier,
+					     pgoff_t start, pgoff_t end)
+{
+	struct kvm_memory_slot *slot = container_of(notifier,
+						    struct kvm_memory_slot,
+						    notifier);
+	struct kvm *kvm = slot->kvm;
+	gfn_t gfn_start, gfn_end;
+
+	if (!restrictedmem_range_is_valid(slot, start, end,
+					  &gfn_start, &gfn_end))
+		return;
+
+	KVM_MMU_LOCK(kvm);
+	kvm_mmu_invalidate_end(kvm);
+	KVM_MMU_UNLOCK(kvm);
+}
+
+static void kvm_restrictedmem_error(struct restrictedmem_notifier  
 *notifier,
+				    pgoff_t start, pgoff_t end)
+{
+	struct kvm_memory_slot *slot = container_of(notifier,
+						    struct kvm_memory_slot,
+						    notifier);
+	kvm_arch_memory_mce(slot->kvm);
+}
+
+static struct restrictedmem_notifier_ops kvm_restrictedmem_notifier_ops  

= {

virt/kvm/kvm_main.c +6 -0
+	.invalidate_start = kvm_restrictedmem_invalidate_begin,
+	.invalidate_end = kvm_restrictedmem_invalidate_end,
+	.error = kvm_restrictedmem_error,
+};
+
+static inline void kvm_restrictedmem_register(struct kvm_memory_slot  

*slot)

virt/kvm/kvm_main.c +6 -0
+{
+	slot->notifier.ops = &kvm_restrictedmem_notifier_ops;
+	restrictedmem_register_notifier(slot->restricted_file, &slot->notifier);
+}
+
+static inline void kvm_restrictedmem_unregister(struct kvm_memory_slot  

*slot)

virt/kvm/kvm_main.c +8 -0
+{
+	restrictedmem_unregister_notifier(slot->restricted_file,
+					  &slot->notifier);
+}
+
+#else /* !CONFIG_HAVE_KVM_RESTRICTED_MEM */
+
+static inline void kvm_restrictedmem_register(struct kvm_memory_slot  

*slot)

virt/kvm/kvm_main.c +5 -0
+{
+	WARN_ON_ONCE(1);
+}
+
+static inline void kvm_restrictedmem_unregister(struct kvm_memory_slot  

*slot)

virt/kvm/kvm_main.c +6 -0
+{
+	WARN_ON_ONCE(1);
+}
+
+#endif /* CONFIG_HAVE_KVM_RESTRICTED_MEM */
+
  #ifdef CONFIG_HAVE_KVM_PM_NOTIFIER
  static int kvm_pm_notifier_call(struct notifier_block *bl,
  				unsigned long state,
@@ -980,6 +1095,11 @@ static void kvm_destroy_dirty_bitmap(struct  
 kvm_memory_slot *memslot)
  /* This does not remove the slot from struct kvm_memslots data  

structures */
static void kvm_free_memslot(struct kvm *kvm, struct kvm_memory_slot
*slot)
{

virt/kvm/kvm_main.c +10 -1
+	if (slot->flags & KVM_MEM_PRIVATE) {
+		kvm_restrictedmem_unregister(slot);
+		fput(slot->restricted_file);
+	}
+
  	kvm_destroy_dirty_bitmap(slot);
 
  	kvm_arch_free_memslot(kvm, slot);
@@ -1551,10 +1671,14 @@ static void kvm_replace_memslot(struct kvm *kvm,
  	}
  }
 
-static int check_memory_region_flags(const struct kvm_user_mem_region  
 *mem)
+static int check_memory_region_flags(struct kvm *kvm,
+				     const struct kvm_user_mem_region *mem)
  {
  	u32 valid_flags = KVM_MEM_LOG_DIRTY_PAGES;
 
+	if (kvm_arch_has_private_mem(kvm))
+		valid_flags |= KVM_MEM_PRIVATE;
+
  #ifdef __KVM_HAVE_READONLY_MEM
  	valid_flags |= KVM_MEM_READONLY;
  #endif
virt/kvm/kvm_main.c +3 -0
@@ -1630,6 +1754,9 @@ static int kvm_prepare_memory_region(struct kvm  
 *kvm,
  {
  	int r;
 
+	if (change == KVM_MR_CREATE && new->flags & KVM_MEM_PRIVATE)
+		kvm_restrictedmem_register(new);
+
  	/*
  	 * If dirty logging is disabled, nullify the bitmap; the old bitmap

* will be freed on "commit". If logging is enabled in both old and

virt/kvm/kvm_main.c +46 -4
@@ -1658,6 +1785,9 @@ static int kvm_prepare_memory_region(struct kvm  
 *kvm,
  	if (r && new && new->dirty_bitmap && (!old || !old->dirty_bitmap))
  		kvm_destroy_dirty_bitmap(new);
 
+	if (r && change == KVM_MR_CREATE && new->flags & KVM_MEM_PRIVATE)
+		kvm_restrictedmem_unregister(new);
+
  	return r;
  }
@@ -1963,7 +2093,7 @@ int __kvm_set_memory_region(struct kvm *kvm,
  	int as_id, id;
  	int r;
 
-	r = check_memory_region_flags(mem);
+	r = check_memory_region_flags(kvm, mem);
  	if (r)
  		return r;
 
@@ -1982,6 +2112,10 @@ int __kvm_set_memory_region(struct kvm *kvm,
  	     !access_ok((void __user *)(unsigned long)mem->userspace_addr,
  			mem->memory_size))
  		return -EINVAL;
+	if (mem->flags & KVM_MEM_PRIVATE &&
+		(mem->restricted_offset & (PAGE_SIZE - 1) ||
+		 mem->restricted_offset > U64_MAX - mem->memory_size))
+		return -EINVAL;
  	if (as_id >= KVM_ADDRESS_SPACE_NUM || id >= KVM_MEM_SLOTS_NUM)
  		return -EINVAL;
  	if (mem->guest_phys_addr + mem->memory_size < mem->guest_phys_addr)
@@ -2020,6 +2154,9 @@ int __kvm_set_memory_region(struct kvm *kvm,
  		if ((kvm->nr_memslot_pages + npages) < kvm->nr_memslot_pages)
  			return -EINVAL;
  	} else { /* Modify an existing slot. */
+		/* Private memslots are immutable, they can only be deleted. */
+		if (mem->flags & KVM_MEM_PRIVATE)
+			return -EINVAL;
  		if ((mem->userspace_addr != old->userspace_addr) ||
  		    (npages != old->npages) ||
  		    ((mem->flags ^ old->flags) & KVM_MEM_READONLY))
@@ -2048,10 +2185,28 @@ int __kvm_set_memory_region(struct kvm *kvm,
  	new->npages = npages;
  	new->flags = mem->flags;
  	new->userspace_addr = mem->userspace_addr;
+	if (mem->flags & KVM_MEM_PRIVATE) {
+		new->restricted_file = fget(mem->restricted_fd);
+		if (!new->restricted_file ||
+		    !file_is_restrictedmem(new->restricted_file)) {
+			r = -EINVAL;
+			goto out;
+		}
+		new->restricted_offset = mem->restricted_offset;
+	}
+
+	new->kvm = kvm;
 
  	r = kvm_set_memslot(kvm, old, new, change);
  	if (r)
-		kfree(new);
+		goto out;
+
+	return 0;
+
+out:
+	if (new->restricted_file)
+		fput(new->restricted_file);
+	kfree(new);
  	return r;
  }
  EXPORT_SYMBOL_GPL(__kvm_set_memory_region);
@@ -2351,6 +2506,8 @@ static int kvm_vm_ioctl_clear_dirty_log(struct kvm  
 *kvm,
  #ifdef CONFIG_HAVE_KVM_MEMORY_ATTRIBUTES
  static u64 kvm_supported_mem_attributes(struct kvm *kvm)
  {
+	if (kvm_arch_has_private_mem(kvm))
+		return KVM_MEMORY_ATTRIBUTE_PRIVATE;
  	return 0;
  }
@@ -4822,16 +4979,28 @@ static long kvm_vm_ioctl(struct file *filp,
  	}
  	case KVM_SET_USER_MEMORY_REGION: {
  		struct kvm_user_mem_region mem;
-		unsigned long size = sizeof(struct kvm_userspace_memory_region);
+		unsigned int flags_offset = offsetof(typeof(mem), flags);
+		unsigned long size;
+		u32 flags;
 
  		kvm_sanity_check_user_mem_region_alias();
 
+		memset(&mem, 0, sizeof(mem));
+
  		r = -EFAULT;
+		if (get_user(flags, (u32 __user *)(argp + flags_offset)))
+			goto out;
+
+		if (flags & KVM_MEM_PRIVATE)
+			size = sizeof(struct kvm_userspace_memory_region_ext);
+		else
+			size = sizeof(struct kvm_userspace_memory_region);
+
  		if (copy_from_user(&mem, argp, size))
  			goto out;
 
  		r = -EINVAL;
-		if (mem.flags & KVM_MEM_PRIVATE)
+		if ((flags ^ mem.flags) & KVM_MEM_PRIVATE)
  			goto out;
 
  		r = kvm_vm_ioctl_set_memory_region(kvm, &mem);
#313 lore raw

Re: [PATCH v10 9/9] KVM: Enable and expose KVM_MEM_PRIVATE

Please trim your replies so that readers don't need to scan through a hundred or so lines of quotes just to confirm there's nothing there.

Quoted 21 lines · Ackerley Tng
Quoted 11 lines · Chao Peng

Register/unregister private memslot to fd-based memory backing store restrictedmem and implement the callbacks for restrictedmem_notifier:
- invalidate_start()/invalidate_end() to zap the existing memory mappings in the KVM page table.
- error() to request KVM_REQ_MEMORY_MCE and later exit to userspace with KVM_EXIT_SHUTDOWN.

Expose KVM_MEM_PRIVATE for memslot and KVM_MEMORY_ATTRIBUTE_PRIVATE for KVM_GET_SUPPORTED_MEMORY_ATTRIBUTES to userspace but either are controlled by kvm_arch_has_private_mem() which should be rewritten by architecture code.

Could we perhaps rename KVM_MEM_PRIVATE to KVM_MEM_PROTECTED, to be in line with KVM_X86_PROTECTED_VM?

I feel that a memslot that has the KVM_MEM_PRIVATE flag need not always be private; It can sometimes be providing memory that is shared and also accessible from the host.

Quoted 8 lines

KVM_MEMORY_ATTRIBUTE_PRIVATE is fine as-is because this flag is set when the guest memory is meant to be backed by private memory.

KVM_MEMORY_EXIT_FLAG_PRIVATE is also okay because the flag is used to indicate when the memory error is caused by a private access (as opposed to a shared access).

kvm_slot_can_be_private() could perhaps be renamed kvm_is_protected_slot()?

No to this suggestion. I agree that KVM_MEM_PRIVATE is a bad name, but kvm_is_protected_slot() is just as wrong. The _only_ thing that the flag controls is whether whether or not the memslot has an fd that is bound to restricted memory. The memslot itself is not protected in any way, and if the entire memslot is mapped shared, then the data backed by the memslot isn't protected either.

What about KVM_MEM_CAN_BE_PRIVATE? KVM_MEM_PRIVATIZABLE is more succinct, but AFAICT that's a made up word, and IMO is unnecessarily fancy.

#314 lore raw

Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM

In reply to Chao Peng
Quoted 10 lines · Chao Peng

This patch series implements KVM guest private memory for confidential computing scenarios like Intel TDX[1]. If a TDX host accesses TDX-protected guest memory, machine check can happen which can further crash the running host system, this is terrible for multi-tenant configurations. The host accesses include those from KVM userspace like QEMU. This series addresses KVM userspace induced crash by introducing new mm and KVM interfaces so KVM userspace can still manage guest memory via a fd-based approach, but it can never access the guest memory content.

Quoted 5 lines

The patch series touches both core mm and KVM code. I appreciate Andrew/Hugh and Paolo/Sean can review and pick these patches. Any other reviews are always welcome.
- 01: mm change, target for mm tree
- 02-09: KVM change, target for KVM tree

A version with all of my feedback, plus reworked versions of Vishal's selftest, is available here:

[email protected]:sean-jc/linux.git x86/upm_base_support

It compiles and passes the selftest, but it's otherwise barely tested. There are a few todos (2 I think?) and many of the commits need changelogs, i.e. it's still a WIP.

As for next steps, can you (handwaving all of the TDX folks) take a look at what I pushed and see if there's anything horrifically broken, and that it still works for TDX?

Fuad (and pKVM folks) same ask for you with respect to pKVM. Absolutely no rush (and I mean that).

On my side, the two things on my mind are (a) tests and (b) downstream dependencies (SEV and TDX). For tests, I want to build a lists of tests that are required for merging so that the criteria for merging are clear, and so that if the list is large (haven't thought much yet), the work of writing and running tests can be distributed.

Regarding downstream dependencies, before this lands, I want to pull in all the TDX and SNP series and see how everything fits together. Specifically, I want to make sure that we don't end up with a uAPI that necessitates ugly code, and that we don't miss an opportunity to make things simpler. The patches in the SNP series to add "legacy" SEV support for UPM in particular made me slightly rethink some minor details. Nothing remotely major, but something that needs attention since it'll be uAPI.

I'm off Monday, so it'll be at least Tuesday before I make any more progress on my side.

Thanks!

#315 lore raw

Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM

Quoted 22 lines · Sean Christopherson
Quoted 10 lines · Chao Peng

This patch series implements KVM guest private memory for confidential computing scenarios like Intel TDX[1]. If a TDX host accesses TDX-protected guest memory, machine check can happen which can further crash the running host system, this is terrible for multi-tenant configurations. The host accesses include those from KVM userspace like QEMU. This series addresses KVM userspace induced crash by introducing new mm and KVM interfaces so KVM userspace can still manage guest memory via a fd-based approach, but it can never access the guest memory content.

Quoted 5 lines

The patch series touches both core mm and KVM code. I appreciate Andrew/Hugh and Paolo/Sean can review and pick these patches. Any other reviews are always welcome.
- 01: mm change, target for mm tree
- 02-09: KVM change, target for KVM tree

A version with all of my feedback, plus reworked versions of Vishal's selftest, is available here:

[email protected]:sean-jc/linux.git x86/upm_base_support

Quoted 7 lines

It compiles and passes the selftest, but it's otherwise barely tested. There are a few todos (2 I think?) and many of the commits need changelogs, i.e. it's still a WIP.

As for next steps, can you (handwaving all of the TDX folks) take a look at what I pushed and see if there's anything horrifically broken, and that it still works for TDX?

Minor build fix:

arch/x86/kvm/mmu/mmu.c +2 -2
index 6eb5336ccc65..4a9e9fa2552a 100644
@@ -7211,8 +7211,8 @@ void kvm_arch_set_memory_attributes(struct kvm *kvm,
 	int level;
 	bool mixed;
 
-	lockdep_assert_held_write(kvm->mmu_lock);
-	lockdep_assert_held(kvm->slots_lock);
+	lockdep_assert_held_write(&kvm->mmu_lock);
+	lockdep_assert_held(&kvm->slots_lock);
 
 	/*
 	 * KVM x86 currently only supports KVM_MEMORY_ATTRIBUTE_PRIVATE, skip
include/linux/kvm_host.h +1 -1
index 467916943c73..4ef60ba7eb1d 100644
@@ -2304,7 +2304,7 @@ static inline void kvm_account_pgtable_pages(void *virt, int nr)
 #ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
 static inline unsigned long kvm_get_memory_attributes(struct kvm *kvm, gfn_t gfn)
 {
-	lockdep_assert_held(kvm->mmu_lock);
+	lockdep_assert_held(&kvm->mmu_lock);
 
 	return xa_to_value(xa_load(&kvm->mem_attr_array, gfn));
 }
Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
author
#316 lore raw

Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM

In reply to Sean Christopherson
Quoted 17 lines · Sean Christopherson
Quoted 10 lines · Chao Peng

This patch series implements KVM guest private memory for confidential computing scenarios like Intel TDX[1]. If a TDX host accesses TDX-protected guest memory, machine check can happen which can further crash the running host system, this is terrible for multi-tenant configurations. The host accesses include those from KVM userspace like QEMU. This series addresses KVM userspace induced crash by introducing new mm and KVM interfaces so KVM userspace can still manage guest memory via a fd-based approach, but it can never access the guest memory content.

Quoted 5 lines

The patch series touches both core mm and KVM code. I appreciate Andrew/Hugh and Paolo/Sean can review and pick these patches. Any other reviews are always welcome.
- 01: mm change, target for mm tree
- 02-09: KVM change, target for KVM tree

Quoted 8 lines

A version with all of my feedback, plus reworked versions of Vishal's selftest, is available here:

[email protected]:sean-jc/linux.git x86/upm_base_support

It compiles and passes the selftest, but it's otherwise barely tested. There are a few todos (2 I think?) and many of the commits need changelogs, i.e. it's still a WIP.

Thanks very much for doing this. Almost all of your comments are well received, except for two cases that need more discussions which have replied individually.

Quoted 4 lines

As for next steps, can you (handwaving all of the TDX folks) take a look at what I pushed and see if there's anything horrifically broken, and that it still works for TDX?

I have integrated this into my local TDX repo, with some changes (as I replied individually), the new code basically still works with TDX.

I have also asked other TDX folks to take a look.

Quoted 17 lines

Fuad (and pKVM folks) same ask for you with respect to pKVM. Absolutely no rush (and I mean that).

On my side, the two things on my mind are (a) tests and (b) downstream dependencies (SEV and TDX). For tests, I want to build a lists of tests that are required for merging so that the criteria for merging are clear, and so that if the list is large (haven't thought much yet), the work of writing and running tests can be distributed.

Regarding downstream dependencies, before this lands, I want to pull in all the TDX and SNP series and see how everything fits together. Specifically, I want to make sure that we don't end up with a uAPI that necessitates ugly code, and that we don't miss an opportunity to make things simpler. The patches in the SNP series to add "legacy" SEV support for UPM in particular made me slightly rethink some minor details. Nothing remotely major, but something that needs attention since it'll be uAPI.

Quoted 2 lines

I'm off Monday, so it'll be at least Tuesday before I make any more progress on my side.

Appreciate your effort. As for the next steps, if you see something we can do parallel, feel free to let me know.

Thanks,
Chao

#317 lore raw

Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM

In reply to Sean Christopherson

Hi Sean,

Quoted 27 lines · Sean Christopherson
Quoted 10 lines · Chao Peng

This patch series implements KVM guest private memory for confidential computing scenarios like Intel TDX[1]. If a TDX host accesses TDX-protected guest memory, machine check can happen which can further crash the running host system, this is terrible for multi-tenant configurations. The host accesses include those from KVM userspace like QEMU. This series addresses KVM userspace induced crash by introducing new mm and KVM interfaces so KVM userspace can still manage guest memory via a fd-based approach, but it can never access the guest memory content.

Quoted 5 lines

The patch series touches both core mm and KVM code. I appreciate Andrew/Hugh and Paolo/Sean can review and pick these patches. Any other reviews are always welcome.
- 01: mm change, target for mm tree
- 02-09: KVM change, target for KVM tree

A version with all of my feedback, plus reworked versions of Vishal's selftest, is available here:

[email protected]:sean-jc/linux.git x86/upm_base_support

It compiles and passes the selftest, but it's otherwise barely tested. There are a few todos (2 I think?) and many of the commits need changelogs, i.e. it's still a WIP.

Quoted 6 lines

As for next steps, can you (handwaving all of the TDX folks) take a look at what I pushed and see if there's anything horrifically broken, and that it still works for TDX?

Fuad (and pKVM folks) same ask for you with respect to pKVM. Absolutely no rush (and I mean that).

Thanks for sharing this. I've had a look at the patches, and have ported them to work with pKVM. At a high level, the new interface seems fine and it works with the arm64/pKVM port. I have a couple of comments regarding some of the details, but they can wait until v11 is posted.

Cheers,
/fuad

Quoted 17 lines

On my side, the two things on my mind are (a) tests and (b) downstream dependencies (SEV and TDX). For tests, I want to build a lists of tests that are required for merging so that the criteria for merging are clear, and so that if the list is large (haven't thought much yet), the work of writing and running tests can be distributed.

Regarding downstream dependencies, before this lands, I want to pull in all the TDX and SNP series and see how everything fits together. Specifically, I want to make sure that we don't end up with a uAPI that necessitates ugly code, and that we don't miss an opportunity to make things simpler. The patches in the SNP series to add "legacy" SEV support for UPM in particular made me slightly rethink some minor details. Nothing remotely major, but something that needs attention since it'll be uAPI.

I'm off Monday, so it'll be at least Tuesday before I make any more progress on my side.

Thanks!

#318 lore raw

Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM

In reply to Sean Christopherson
Quoted 38 lines · Sean Christopherson
Quoted 10 lines · Chao Peng

This patch series implements KVM guest private memory for confidential computing scenarios like Intel TDX[1]. If a TDX host accesses TDX-protected guest memory, machine check can happen which can further crash the running host system, this is terrible for multi-tenant configurations. The host accesses include those from KVM userspace like QEMU. This series addresses KVM userspace induced crash by introducing new mm and KVM interfaces so KVM userspace can still manage guest memory via a fd-based approach, but it can never access the guest memory content.

Quoted 5 lines

The patch series touches both core mm and KVM code. I appreciate Andrew/Hugh and Paolo/Sean can review and pick these patches. Any other reviews are always welcome.
- 01: mm change, target for mm tree
- 02-09: KVM change, target for KVM tree

A version with all of my feedback, plus reworked versions of Vishal's selftest, is available here:

[email protected]:sean-jc/linux.git x86/upm_base_support

It compiles and passes the selftest, but it's otherwise barely tested. There are a few todos (2 I think?) and many of the commits need changelogs, i.e. it's still a WIP.

As for next steps, can you (handwaving all of the TDX folks) take a look at what I pushed and see if there's anything horrifically broken, and that it still works for TDX?

Fuad (and pKVM folks) same ask for you with respect to pKVM. Absolutely no rush (and I mean that).

On my side, the two things on my mind are (a) tests and (b) downstream dependencies (SEV and TDX). For tests, I want to build a lists of tests that are required for merging so that the criteria for merging are clear, and so that if the list is large (haven't thought much yet), the work of writing and running tests can be distributed.

Quoted 7 lines

Regarding downstream dependencies, before this lands, I want to pull in all the TDX and SNP series and see how everything fits together. Specifically, I want to make sure that we don't end up with a uAPI that necessitates ugly code, and that we don't miss an opportunity to make things simpler. The patches in the SNP series to add "legacy" SEV support for UPM in particular made me slightly rethink some minor details. Nothing remotely major, but something that needs attention since it'll be uAPI.

Although I'm still debuging with TDX KVM, I needed the following. kvm_faultin_pfn() is called without mmu_lock held. the race to change private/shared is handled by mmu_seq. Maybe dedicated function only for kvm_faultin_pfn().

include/linux/kvm_host.h +1 -1
index 02be5e1cba1e..38699ca75ab8 100644
@@ -2322,7 +2322,7 @@ static inline void kvm_account_pgtable_pages(void *virt, int nr)
 #ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
 static inline unsigned long kvm_get_memory_attributes(struct kvm *kvm, gfn_t gfn)
 {
-       lockdep_assert_held(&kvm->mmu_lock);
+       // lockdep_assert_held(&kvm->mmu_lock);
 
        return xa_to_value(xa_load(&kvm->mem_attr_array, gfn));
 }
Signature · 1 line
Isaku Yamahata <[email protected]>
#319 lore raw

Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM

Quoted 49 lines · Isaku Yamahata
Quoted 38 lines · Sean Christopherson
Quoted 10 lines · Chao Peng

This patch series implements KVM guest private memory for confidential computing scenarios like Intel TDX[1]. If a TDX host accesses TDX-protected guest memory, machine check can happen which can further crash the running host system, this is terrible for multi-tenant configurations. The host accesses include those from KVM userspace like QEMU. This series addresses KVM userspace induced crash by introducing new mm and KVM interfaces so KVM userspace can still manage guest memory via a fd-based approach, but it can never access the guest memory content.

Quoted 5 lines

The patch series touches both core mm and KVM code. I appreciate Andrew/Hugh and Paolo/Sean can review and pick these patches. Any other reviews are always welcome.
- 01: mm change, target for mm tree
- 02-09: KVM change, target for KVM tree

A version with all of my feedback, plus reworked versions of Vishal's selftest, is available here:

[email protected]:sean-jc/linux.git x86/upm_base_support

It compiles and passes the selftest, but it's otherwise barely tested. There are a few todos (2 I think?) and many of the commits need changelogs, i.e. it's still a WIP.

As for next steps, can you (handwaving all of the TDX folks) take a look at what I pushed and see if there's anything horrifically broken, and that it still works for TDX?

Fuad (and pKVM folks) same ask for you with respect to pKVM. Absolutely no rush (and I mean that).

On my side, the two things on my mind are (a) tests and (b) downstream dependencies (SEV and TDX). For tests, I want to build a lists of tests that are required for merging so that the criteria for merging are clear, and so that if the list is large (haven't thought much yet), the work of writing and running tests can be distributed.

Quoted 7 lines

Regarding downstream dependencies, before this lands, I want to pull in all the TDX and SNP series and see how everything fits together. Specifically, I want to make sure that we don't end up with a uAPI that necessitates ugly code, and that we don't miss an opportunity to make things simpler. The patches in the SNP series to add "legacy" SEV support for UPM in particular made me slightly rethink some minor details. Nothing remotely major, but something that needs attention since it'll be uAPI.

Quoted 4 lines

Although I'm still debuging with TDX KVM, I needed the following. kvm_faultin_pfn() is called without mmu_lock held. the race to change private/shared is handled by mmu_seq. Maybe dedicated function only for kvm_faultin_pfn().

Gah, you're not on the other thread where this was discussed[*]. Simply deleting the lockdep assertion is safe, for guest types that rely on the attributes to define shared vs. private, KVM rechecks the attributes under the protection of mmu_seq.

I'll get a fixed version pushed out today.

[*] https://lore.kernel.org/all/[email protected]

#320 lore raw

Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM

Quoted 55 lines · Sean Christopherson
Quoted 49 lines · Isaku Yamahata
Quoted 38 lines · Sean Christopherson
Quoted 10 lines · Chao Peng

This patch series implements KVM guest private memory for confidential computing scenarios like Intel TDX[1]. If a TDX host accesses TDX-protected guest memory, machine check can happen which can further crash the running host system, this is terrible for multi-tenant configurations. The host accesses include those from KVM userspace like QEMU. This series addresses KVM userspace induced crash by introducing new mm and KVM interfaces so KVM userspace can still manage guest memory via a fd-based approach, but it can never access the guest memory content.

Quoted 5 lines

The patch series touches both core mm and KVM code. I appreciate Andrew/Hugh and Paolo/Sean can review and pick these patches. Any other reviews are always welcome.
- 01: mm change, target for mm tree
- 02-09: KVM change, target for KVM tree

A version with all of my feedback, plus reworked versions of Vishal's selftest, is available here:

[email protected]:sean-jc/linux.git x86/upm_base_support

It compiles and passes the selftest, but it's otherwise barely tested. There are a few todos (2 I think?) and many of the commits need changelogs, i.e. it's still a WIP.

As for next steps, can you (handwaving all of the TDX folks) take a look at what I pushed and see if there's anything horrifically broken, and that it still works for TDX?

Fuad (and pKVM folks) same ask for you with respect to pKVM. Absolutely no rush (and I mean that).

On my side, the two things on my mind are (a) tests and (b) downstream dependencies (SEV and TDX). For tests, I want to build a lists of tests that are required for merging so that the criteria for merging are clear, and so that if the list is large (haven't thought much yet), the work of writing and running tests can be distributed.

Quoted 7 lines

Regarding downstream dependencies, before this lands, I want to pull in all the TDX and SNP series and see how everything fits together. Specifically, I want to make sure that we don't end up with a uAPI that necessitates ugly code, and that we don't miss an opportunity to make things simpler. The patches in the SNP series to add "legacy" SEV support for UPM in particular made me slightly rethink some minor details. Nothing remotely major, but something that needs attention since it'll be uAPI.

Quoted 4 lines

Although I'm still debuging with TDX KVM, I needed the following. kvm_faultin_pfn() is called without mmu_lock held. the race to change private/shared is handled by mmu_seq. Maybe dedicated function only for kvm_faultin_pfn().

Quoted 8 lines

Gah, you're not on the other thread where this was discussed[*]. Simply deleting the lockdep assertion is safe, for guest types that rely on the attributes to define shared vs. private, KVM rechecks the attributes under the protection of mmu_seq.

I'll get a fixed version pushed out today.

[*] https://lore.kernel.org/all/[email protected]

Now I have tdx kvm working. I've uploaded at the followings. It's rebased to v6.2-rc3.
[email protected]:yamahata/linux.git tdx/upm [email protected]:yamahata/qemu.git tdx/upm

kvm_mmu_do_page_fault() needs the following change. kvm_mem_is_private() queries mem_attr_array. kvm_faultin_pfn() also uses kvm_mem_is_private(). So the shared-private check in kvm_faultin_pfn() doesn't make sense. This change would belong to TDX KVM patches, though.

arch/x86/kvm/mmu/mmu_internal.h +1 -1
index 72b0da8e27e0..f45ac438bbf4 100644
@@ -430,7 +430,7 @@ static inline int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
                .max_level = vcpu->kvm->arch.tdp_max_page_level,
                .req_level = PG_LEVEL_4K,
                .goal_level = PG_LEVEL_4K,
-               .is_private = kvm_mem_is_private(vcpu->kvm, cr2_or_gpa >> PAGE_SHIFT),
+               .is_private = kvm_is_private_gpa(vcpu->kvm, cr2_or_gpa),
        };
        int r;
 
Signature · 1 line
Isaku Yamahata <[email protected]>
#321 lore raw

Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM

Quoted 63 lines · Isaku Yamahata
Quoted 60 lines · Sean Christopherson
Quoted 49 lines · Isaku Yamahata
Quoted 38 lines · Sean Christopherson
Quoted 10 lines · Chao Peng

This patch series implements KVM guest private memory for confidential computing scenarios like Intel TDX[1]. If a TDX host accesses TDX-protected guest memory, machine check can happen which can further crash the running host system, this is terrible for multi-tenant configurations. The host accesses include those from KVM userspace like QEMU. This series addresses KVM userspace induced crash by introducing new mm and KVM interfaces so KVM userspace can still manage guest memory via a fd-based approach, but it can never access the guest memory content.

Quoted 5 lines

The patch series touches both core mm and KVM code. I appreciate Andrew/Hugh and Paolo/Sean can review and pick these patches. Any other reviews are always welcome.
- 01: mm change, target for mm tree
- 02-09: KVM change, target for KVM tree

A version with all of my feedback, plus reworked versions of Vishal's selftest, is available here:

[email protected]:sean-jc/linux.git x86/upm_base_support

It compiles and passes the selftest, but it's otherwise barely tested. There are a few todos (2 I think?) and many of the commits need changelogs, i.e. it's still a WIP.

As for next steps, can you (handwaving all of the TDX folks) take a look at what I pushed and see if there's anything horrifically broken, and that it still works for TDX?

Fuad (and pKVM folks) same ask for you with respect to pKVM. Absolutely no rush (and I mean that).

On my side, the two things on my mind are (a) tests and (b) downstream dependencies (SEV and TDX). For tests, I want to build a lists of tests that are required for merging so that the criteria for merging are clear, and so that if the list is large (haven't thought much yet), the work of writing and running tests can be distributed.

Quoted 7 lines

Regarding downstream dependencies, before this lands, I want to pull in all the TDX and SNP series and see how everything fits together. Specifically, I want to make sure that we don't end up with a uAPI that necessitates ugly code, and that we don't miss an opportunity to make things simpler. The patches in the SNP series to add "legacy" SEV support for UPM in particular made me slightly rethink some minor details. Nothing remotely major, but something that needs attention since it'll be uAPI.

Quoted 4 lines

Although I'm still debuging with TDX KVM, I needed the following. kvm_faultin_pfn() is called without mmu_lock held. the race to change private/shared is handled by mmu_seq. Maybe dedicated function only for kvm_faultin_pfn().

Gah, you're not on the other thread where this was discussed[*]. Simply deleting the lockdep assertion is safe, for guest types that rely on the attributes to define shared vs. private, KVM rechecks the attributes under the protection of mmu_seq.

Quoted 8 lines
Quoted 3 lines

I'll get a fixed version pushed out today.

[*] https://lore.kernel.org/all/[email protected]

Now I have tdx kvm working. I've uploaded at the followings. It's rebased to v6.2-rc3.
[email protected]:yamahata/linux.git tdx/upm [email protected]:yamahata/qemu.git tdx/upm

And I finally got a working, building version updated and pushed out (again to):

[email protected]:sean-jc/linux.git x86/upm_base_support

Took longer than expected to get the memslot restrictions sussed out. I'm done working on the code for now, my plan is to come back to it+TDX+SNP in 2-3 weeks to resolves any remaining todos (that no one else tackles) and to do the whole "merge the world" excersise.

Quoted 4 lines

kvm_mmu_do_page_fault() needs the following change. kvm_mem_is_private() queries mem_attr_array. kvm_faultin_pfn() also uses kvm_mem_is_private(). So the shared-private check in kvm_faultin_pfn() doesn't make sense. This change would belong to TDX KVM patches, though.

Yeah, SNP needs similar treatment. Sorting that out is high up on the todo list.

#322 lore raw

Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM

Quoted 68 lines · Sean Christopherson
Quoted 67 lines · Isaku Yamahata
Quoted 63 lines · Sean Christopherson
Quoted 49 lines · Isaku Yamahata
Quoted 38 lines · Sean Christopherson
Quoted 10 lines · Chao Peng

This patch series implements KVM guest private memory for confidential computing scenarios like Intel TDX[1]. If a TDX host accesses TDX-protected guest memory, machine check can happen which can further crash the running host system, this is terrible for multi-tenant configurations. The host accesses include those from KVM userspace like QEMU. This series addresses KVM userspace induced crash by introducing new mm and KVM interfaces so KVM userspace can still manage guest memory via a fd-based approach, but it can never access the guest memory content.

Quoted 5 lines

The patch series touches both core mm and KVM code. I appreciate Andrew/Hugh and Paolo/Sean can review and pick these patches. Any other reviews are always welcome.
- 01: mm change, target for mm tree
- 02-09: KVM change, target for KVM tree

A version with all of my feedback, plus reworked versions of Vishal's selftest, is available here:

[email protected]:sean-jc/linux.git x86/upm_base_support

It compiles and passes the selftest, but it's otherwise barely tested. There are a few todos (2 I think?) and many of the commits need changelogs, i.e. it's still a WIP.

As for next steps, can you (handwaving all of the TDX folks) take a look at what I pushed and see if there's anything horrifically broken, and that it still works for TDX?

Fuad (and pKVM folks) same ask for you with respect to pKVM. Absolutely no rush (and I mean that).

On my side, the two things on my mind are (a) tests and (b) downstream dependencies (SEV and TDX). For tests, I want to build a lists of tests that are required for merging so that the criteria for merging are clear, and so that if the list is large (haven't thought much yet), the work of writing and running tests can be distributed.

Quoted 7 lines

Regarding downstream dependencies, before this lands, I want to pull in all the TDX and SNP series and see how everything fits together. Specifically, I want to make sure that we don't end up with a uAPI that necessitates ugly code, and that we don't miss an opportunity to make things simpler. The patches in the SNP series to add "legacy" SEV support for UPM in particular made me slightly rethink some minor details. Nothing remotely major, but something that needs attention since it'll be uAPI.

Quoted 4 lines

Although I'm still debuging with TDX KVM, I needed the following. kvm_faultin_pfn() is called without mmu_lock held. the race to change private/shared is handled by mmu_seq. Maybe dedicated function only for kvm_faultin_pfn().

Gah, you're not on the other thread where this was discussed[*]. Simply deleting the lockdep assertion is safe, for guest types that rely on the attributes to define shared vs. private, KVM rechecks the attributes under the protection of mmu_seq.

I'll get a fixed version pushed out today.

[*] https://lore.kernel.org/all/[email protected]

Quoted 9 lines
Quoted 4 lines

Now I have tdx kvm working. I've uploaded at the followings. It's rebased to v6.2-rc3.
[email protected]:yamahata/linux.git tdx/upm [email protected]:yamahata/qemu.git tdx/upm

And I finally got a working, building version updated and pushed out (again to):

[email protected]:sean-jc/linux.git x86/upm_base_support

Ok, I rebased TDX part to the updated branch.
[email protected]:yamahata/linux.git tdx/upm
[email protected]:yamahata/qemu.git tdx/upm

Now it's v6.2-rc7 based.
qemu needs more patches to avoid registering memory slot for SMM.

Signature · 1 line
Isaku Yamahata <[email protected]>
#323 lore raw

Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM

In reply to Sean Christopherson
Quoted 82 lines · Sean Christopherson
Quoted 63 lines · Isaku Yamahata
Quoted 60 lines · Sean Christopherson
Quoted 49 lines · Isaku Yamahata
Quoted 38 lines · Sean Christopherson
Quoted 10 lines · Chao Peng

This patch series implements KVM guest private memory for confidential computing scenarios like Intel TDX[1]. If a TDX host accesses TDX-protected guest memory, machine check can happen which can further crash the running host system, this is terrible for multi-tenant configurations. The host accesses include those from KVM userspace like QEMU. This series addresses KVM userspace induced crash by introducing new mm and KVM interfaces so KVM userspace can still manage guest memory via a fd-based approach, but it can never access the guest memory content.

Quoted 5 lines

The patch series touches both core mm and KVM code. I appreciate Andrew/Hugh and Paolo/Sean can review and pick these patches. Any other reviews are always welcome.
- 01: mm change, target for mm tree
- 02-09: KVM change, target for KVM tree

A version with all of my feedback, plus reworked versions of Vishal's selftest, is available here:

[email protected]:sean-jc/linux.git x86/upm_base_support

It compiles and passes the selftest, but it's otherwise barely tested. There are a few todos (2 I think?) and many of the commits need changelogs, i.e. it's still a WIP.

As for next steps, can you (handwaving all of the TDX folks) take a look at what I pushed and see if there's anything horrifically broken, and that it still works for TDX?

Fuad (and pKVM folks) same ask for you with respect to pKVM. Absolutely no rush (and I mean that).

On my side, the two things on my mind are (a) tests and (b) downstream dependencies (SEV and TDX). For tests, I want to build a lists of tests that are required for merging so that the criteria for merging are clear, and so that if the list is large (haven't thought much yet), the work of writing and running tests can be distributed.

Quoted 7 lines

Regarding downstream dependencies, before this lands, I want to pull in all the TDX and SNP series and see how everything fits together. Specifically, I want to make sure that we don't end up with a uAPI that necessitates ugly code, and that we don't miss an opportunity to make things simpler. The patches in the SNP series to add "legacy" SEV support for UPM in particular made me slightly rethink some minor details. Nothing remotely major, but something that needs attention since it'll be uAPI.

Quoted 4 lines

Although I'm still debuging with TDX KVM, I needed the following. kvm_faultin_pfn() is called without mmu_lock held. the race to change private/shared is handled by mmu_seq. Maybe dedicated function only for kvm_faultin_pfn().

Gah, you're not on the other thread where this was discussed[*]. Simply deleting the lockdep assertion is safe, for guest types that rely on the attributes to define shared vs. private, KVM rechecks the attributes under the protection of mmu_seq.

Quoted 8 lines
Quoted 3 lines

I'll get a fixed version pushed out today.

[*] https://lore.kernel.org/all/[email protected]

Now I have tdx kvm working. I've uploaded at the followings. It's rebased to v6.2-rc3.
[email protected]:yamahata/linux.git tdx/upm [email protected]:yamahata/qemu.git tdx/upm

And I finally got a working, building version updated and pushed out (again to):

[email protected]:sean-jc/linux.git x86/upm_base_support

Took longer than expected to get the memslot restrictions sussed out. I'm done working on the code for now, my plan is to come back to it+TDX+SNP in 2-3 weeks to resolves any remaining todos (that no one else tackles) and to do the whole "merge the world" excersise.

Quoted 6 lines
Quoted 4 lines

kvm_mmu_do_page_fault() needs the following change. kvm_mem_is_private() queries mem_attr_array. kvm_faultin_pfn() also uses kvm_mem_is_private(). So the shared-private check in kvm_faultin_pfn() doesn't make sense. This change would belong to TDX KVM patches, though.

Yeah, SNP needs similar treatment. Sorting that out is high up on the todo list.

Hi Sean,

We've rebased the SEV+SNP support onto your updated UPM base support tree and things seem to be working okay, but we needed some fixups on top of the base support get things working, along with 1 workaround for an issue that hasn't been root-caused yet:

https://github.com/mdroth/linux/commits/upmv10b-host-snp-v8-wip

*stash (upm_base_support): mm: restrictedmem: Kirill's pinning implementation *workaround (use_base_support): mm: restrictedmem: loosen exclusivity check *fixup (upm_base_support): KVM: use inclusive ranges for restrictedmem binding/unbinding *fixup (upm_base_support): mm: restrictedmem: use inclusive ranges for issuing invalidations *fixup (upm_base_support): KVM: fix restrictedmem GFN range calculations *fixup (upm_base_support): KVM: selftests: CoCo compilation fixes

We plan to post an updated RFC for v8 soon, but also wanted to share the staging tree in case you end up looking at the UPM integration aspects before then.

-Mike

author
#324 lore raw

Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM

Quoted 11 lines

Hi Sean,

We've rebased the SEV+SNP support onto your updated UPM base support tree and things seem to be working okay, but we needed some fixups on top of the base support get things working, along with 1 workaround for an issue that hasn't been root-caused yet:

https://github.com/mdroth/linux/commits/upmv10b-host-snp-v8-wip

*stash (upm_base_support): mm: restrictedmem: Kirill's pinning implementation *workaround (use_base_support): mm: restrictedmem: loosen exclusivity check

What I'm seeing is Slot#3 gets added first and then deleted. When it's gets added, Slot#0 already has the same range bound to restrictedmem so trigger the exclusive check. This check is exactly the current code for.

Quoted 2 lines

*fixup (upm_base_support): KVM: use inclusive ranges for restrictedmem binding/unbinding *fixup (upm_base_support): mm: restrictedmem: use inclusive ranges for issuing invalidations

As many kernel APIs treat 'end' as exclusive, I would rather keep using exclusive 'end' for these APIs(restrictedmem_bind/restrictedmem_unbind and notifier callbacks) but fix it internally in the restrictedmem. E.g. all the places where xarray API needs a 'last'/'max' we use 'end - 1'. See below for the change.

Quoted 1 line

*fixup (upm_base_support): KVM: fix restrictedmem GFN range calculations

Subtracting slot->restrictedmem.index for start/end in restrictedmem_get_gfn_range() is the correct fix.

Quoted 7 lines

*fixup (upm_base_support): KVM: selftests: CoCo compilation fixes

We plan to post an updated RFC for v8 soon, but also wanted to share the staging tree in case you end up looking at the UPM integration aspects before then.

-Mike

This is the restrictedmem fix to solve 'end' being stored and checked in xarray:

mm/restrictedmem.c +7 -6
@@ -46,12 +46,12 @@ static long restrictedmem_punch_hole(struct restrictedmem *rm, int mode,
         */
        down_read(&rm->lock);
 
-       xa_for_each_range(&rm->bindings, index, notifier, start, end)
+       xa_for_each_range(&rm->bindings, index, notifier, start, end - 1)
                notifier->ops->invalidate_start(notifier, start, end);
 
        ret = memfd->f_op->fallocate(memfd, mode, offset, len);
 
-       xa_for_each_range(&rm->bindings, index, notifier, start, end)
+       xa_for_each_range(&rm->bindings, index, notifier, start, end - 1)
                notifier->ops->invalidate_end(notifier, start, end);
 
        up_read(&rm->lock);
@@ -224,7 +224,7 @@ static int restricted_error_remove_page(struct address_space *mapping,
                }
                spin_unlock(&inode->i_lock);
 
-               xa_for_each_range(&rm->bindings, index, notifier, start, end)
+               xa_for_each_range(&rm->bindings, index, notifier, start, end - 1)
                        notifier->ops->error(notifier, start, end);
                break;
        }
@@ -301,11 +301,12 @@ int restrictedmem_bind(struct file *file, pgoff_t start, pgoff_t end,
                if (exclusive != rm->exclusive)
                        goto out_unlock;
 
-               if (exclusive && xa_find(&rm->bindings, &start, end, XA_PRESENT))
+               if (exclusive &&
+                   xa_find(&rm->bindings, &start, end - 1, XA_PRESENT))
                        goto out_unlock;
        }
 
-       xa_store_range(&rm->bindings, start, end, notifier, GFP_KERNEL);
+       xa_store_range(&rm->bindings, start, end - 1, notifier, GFP_KERNEL);
        rm->exclusive = exclusive;
        ret = 0;
 out_unlock:
@@ -320,7 +321,7 @@ void restrictedmem_unbind(struct file *file, pgoff_t start, pgoff_t end,
        struct restrictedmem *rm = file->f_mapping->private_data;
 
        down_write(&rm->lock);
-       xa_store_range(&rm->bindings, start, end, NULL, GFP_KERNEL);
+       xa_store_range(&rm->bindings, start, end - 1, NULL, GFP_KERNEL);
        synchronize_rcu();
        up_write(&rm->lock);
 }
#325 lore raw

Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM

Quoted 7 lines · Chao Peng
Quoted 7 lines

Hi Sean,

We've rebased the SEV+SNP support onto your updated UPM base support tree and things seem to be working okay, but we needed some fixups on top of the base support get things working, along with 1 workaround for an issue that hasn't been root-caused yet:

Quoted 8 lines
Quoted 4 lines

https://github.com/mdroth/linux/commits/upmv10b-host-snp-v8-wip

*stash (upm_base_support): mm: restrictedmem: Kirill's pinning implementation *workaround (use_base_support): mm: restrictedmem: loosen exclusivity check

What I'm seeing is Slot#3 gets added first and then deleted. When it's gets added, Slot#0 already has the same range bound to restrictedmem so trigger the exclusive check. This check is exactly the current code for.

With the following change in QEMU, we no longer trigger this check:

diff --git a/hw/pci-host/q35.c b/hw/pci-host/q35.c
index 20da121374..849b5de469 100644
--- a/hw/pci-host/q35.c
+++ b/hw/pci-host/q35.c
@@ -588,9 +588,9 @@ static void mch_realize(PCIDevice *d, Error **errp)

       memory_region_init_alias(&mch->open_high_smram, OBJECT(mch), "smram-open-high",
                                mch->ram_memory, MCH_HOST_BRIDGE_SMRAM_C_BASE,
                                MCH_HOST_BRIDGE_SMRAM_C_SIZE);
  +    memory_region_set_enabled(&mch->open_high_smram, false);
       memory_region_add_subregion_overlap(mch->system_memory, 0xfeda0000,
                                           &mch->open_high_smram, 1);

- memory_region_set_enabled(&mch->open_high_smram, false);

I'm not sure if QEMU is actually doing something wrong here though or if this check is putting tighter restrictions on userspace than what was expected before. Will look into it more.

Quoted 9 lines
Quoted 3 lines

*fixup (upm_base_support): KVM: use inclusive ranges for restrictedmem binding/unbinding *fixup (upm_base_support): mm: restrictedmem: use inclusive ranges for issuing invalidations

As many kernel APIs treat 'end' as exclusive, I would rather keep using exclusive 'end' for these APIs(restrictedmem_bind/restrictedmem_unbind and notifier callbacks) but fix it internally in the restrictedmem. E.g. all the places where xarray API needs a 'last'/'max' we use 'end - 1'. See below for the change.

Yes I did feel like I was fighting the kernel a bit on that; your suggestion seems like it would be a better fit.

Quoted 8 lines
Quoted 2 lines

*fixup (upm_base_support): KVM: fix restrictedmem GFN range calculations

Subtracting slot->restrictedmem.index for start/end in restrictedmem_get_gfn_range() is the correct fix.

Quoted 2 lines

*fixup (upm_base_support): KVM: selftests: CoCo compilation fixes

Quoted 7 lines
Quoted 5 lines

We plan to post an updated RFC for v8 soon, but also wanted to share the staging tree in case you end up looking at the UPM integration aspects before then.

-Mike

This is the restrictedmem fix to solve 'end' being stored and checked in xarray:

Looks good.

Thanks!

-Mike

Quoted 51 lines
mm/restrictedmem.c +7 -6
@@ -46,12 +46,12 @@ static long restrictedmem_punch_hole(struct restrictedmem *rm, int mode,
         */
        down_read(&rm->lock);
 
-       xa_for_each_range(&rm->bindings, index, notifier, start, end)
+       xa_for_each_range(&rm->bindings, index, notifier, start, end - 1)
                notifier->ops->invalidate_start(notifier, start, end);
 
        ret = memfd->f_op->fallocate(memfd, mode, offset, len);
 
-       xa_for_each_range(&rm->bindings, index, notifier, start, end)
+       xa_for_each_range(&rm->bindings, index, notifier, start, end - 1)
                notifier->ops->invalidate_end(notifier, start, end);
 
        up_read(&rm->lock);
@@ -224,7 +224,7 @@ static int restricted_error_remove_page(struct address_space *mapping,
                }
                spin_unlock(&inode->i_lock);
 
-               xa_for_each_range(&rm->bindings, index, notifier, start, end)
+               xa_for_each_range(&rm->bindings, index, notifier, start, end - 1)
                        notifier->ops->error(notifier, start, end);
                break;
        }
@@ -301,11 +301,12 @@ int restrictedmem_bind(struct file *file, pgoff_t start, pgoff_t end,
                if (exclusive != rm->exclusive)
                        goto out_unlock;
 
-               if (exclusive && xa_find(&rm->bindings, &start, end, XA_PRESENT))
+               if (exclusive &&
+                   xa_find(&rm->bindings, &start, end - 1, XA_PRESENT))
                        goto out_unlock;
        }
 
-       xa_store_range(&rm->bindings, start, end, notifier, GFP_KERNEL);
+       xa_store_range(&rm->bindings, start, end - 1, notifier, GFP_KERNEL);
        rm->exclusive = exclusive;
        ret = 0;
 out_unlock:
@@ -320,7 +321,7 @@ void restrictedmem_unbind(struct file *file, pgoff_t start, pgoff_t end,
        struct restrictedmem *rm = file->f_mapping->private_data;
 
        down_write(&rm->lock);
-       xa_store_range(&rm->bindings, start, end, NULL, GFP_KERNEL);
+       xa_store_range(&rm->bindings, start, end - 1, NULL, GFP_KERNEL);
        synchronize_rcu();
        up_write(&rm->lock);
 }
author
#326 lore raw

Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM

Quoted 30 lines · Michael Roth
Quoted 7 lines · Chao Peng
Quoted 7 lines

Hi Sean,

We've rebased the SEV+SNP support onto your updated UPM base support tree and things seem to be working okay, but we needed some fixups on top of the base support get things working, along with 1 workaround for an issue that hasn't been root-caused yet:

Quoted 8 lines
Quoted 4 lines

https://github.com/mdroth/linux/commits/upmv10b-host-snp-v8-wip

*stash (upm_base_support): mm: restrictedmem: Kirill's pinning implementation *workaround (use_base_support): mm: restrictedmem: loosen exclusivity check

What I'm seeing is Slot#3 gets added first and then deleted. When it's gets added, Slot#0 already has the same range bound to restrictedmem so trigger the exclusive check. This check is exactly the current code for.

With the following change in QEMU, we no longer trigger this check:

diff --git a/hw/pci-host/q35.c b/hw/pci-host/q35.c
index 20da121374..849b5de469 100644
--- a/hw/pci-host/q35.c
+++ b/hw/pci-host/q35.c
@@ -588,9 +588,9 @@ static void mch_realize(PCIDevice *d, Error **errp)

       memory_region_init_alias(&mch->open_high_smram, OBJECT(mch), "smram-open-high",
                                mch->ram_memory, MCH_HOST_BRIDGE_SMRAM_C_BASE,
                                MCH_HOST_BRIDGE_SMRAM_C_SIZE);
  +    memory_region_set_enabled(&mch->open_high_smram, false);
       memory_region_add_subregion_overlap(mch->system_memory, 0xfeda0000,
                                           &mch->open_high_smram, 1);
Quoted 5 lines

- memory_region_set_enabled(&mch->open_high_smram, false);

I'm not sure if QEMU is actually doing something wrong here though or if this check is putting tighter restrictions on userspace than what was expected before. Will look into it more.

I don't think above QEMU change is upstream acceptable. It may break functionality for 'normal' VMs.

The UPM check does putting tighter restriction, the restriction is that you can't bind the same fd range to more than one memslot. For SMRAM in QEMU however, it violates this restriction. The right 'fix' is disabling SMM in QEMU for UPM usages rather than trying to work around it. There is more discussion in below link:

https://lore.kernel.org/all/[email protected]/

Chao

Quoted 87 lines
Quoted 10 lines
Quoted 4 lines

*fixup (upm_base_support): KVM: use inclusive ranges for restrictedmem binding/unbinding *fixup (upm_base_support): mm: restrictedmem: use inclusive ranges for issuing invalidations

As many kernel APIs treat 'end' as exclusive, I would rather keep using exclusive 'end' for these APIs(restrictedmem_bind/restrictedmem_unbind and notifier callbacks) but fix it internally in the restrictedmem. E.g. all the places where xarray API needs a 'last'/'max' we use 'end - 1'. See below for the change.

Yes I did feel like I was fighting the kernel a bit on that; your suggestion seems like it would be a better fit.

Quoted 8 lines
Quoted 2 lines

*fixup (upm_base_support): KVM: fix restrictedmem GFN range calculations

Subtracting slot->restrictedmem.index for start/end in restrictedmem_get_gfn_range() is the correct fix.

Quoted 2 lines

*fixup (upm_base_support): KVM: selftests: CoCo compilation fixes

Quoted 7 lines
Quoted 5 lines

We plan to post an updated RFC for v8 soon, but also wanted to share the staging tree in case you end up looking at the UPM integration aspects before then.

-Mike

This is the restrictedmem fix to solve 'end' being stored and checked in xarray:

Looks good.

Thanks!

-Mike

Quoted 51 lines
mm/restrictedmem.c +7 -6
@@ -46,12 +46,12 @@ static long restrictedmem_punch_hole(struct restrictedmem *rm, int mode,
         */
        down_read(&rm->lock);
 
-       xa_for_each_range(&rm->bindings, index, notifier, start, end)
+       xa_for_each_range(&rm->bindings, index, notifier, start, end - 1)
                notifier->ops->invalidate_start(notifier, start, end);
 
        ret = memfd->f_op->fallocate(memfd, mode, offset, len);
 
-       xa_for_each_range(&rm->bindings, index, notifier, start, end)
+       xa_for_each_range(&rm->bindings, index, notifier, start, end - 1)
                notifier->ops->invalidate_end(notifier, start, end);
 
        up_read(&rm->lock);
@@ -224,7 +224,7 @@ static int restricted_error_remove_page(struct address_space *mapping,
                }
                spin_unlock(&inode->i_lock);
 
-               xa_for_each_range(&rm->bindings, index, notifier, start, end)
+               xa_for_each_range(&rm->bindings, index, notifier, start, end - 1)
                        notifier->ops->error(notifier, start, end);
                break;
        }
@@ -301,11 +301,12 @@ int restrictedmem_bind(struct file *file, pgoff_t start, pgoff_t end,
                if (exclusive != rm->exclusive)
                        goto out_unlock;
 
-               if (exclusive && xa_find(&rm->bindings, &start, end, XA_PRESENT))
+               if (exclusive &&
+                   xa_find(&rm->bindings, &start, end - 1, XA_PRESENT))
                        goto out_unlock;
        }
 
-       xa_store_range(&rm->bindings, start, end, notifier, GFP_KERNEL);
+       xa_store_range(&rm->bindings, start, end - 1, notifier, GFP_KERNEL);
        rm->exclusive = exclusive;
        ret = 0;
 out_unlock:
@@ -320,7 +321,7 @@ void restrictedmem_unbind(struct file *file, pgoff_t start, pgoff_t end,
        struct restrictedmem *rm = file->f_mapping->private_data;
 
        down_write(&rm->lock);
-       xa_store_range(&rm->bindings, start, end, NULL, GFP_KERNEL);
+       xa_store_range(&rm->bindings, start, end - 1, NULL, GFP_KERNEL);
        synchronize_rcu();
        up_write(&rm->lock);
 }
#327 lore raw

Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM

In reply to Michael Roth
Quoted 4 lines · Michael Roth
Quoted 3 lines · Chao Peng
Quoted 2 lines

*fixup (upm_base_support): KVM: use inclusive ranges for restrictedmem binding/unbinding *fixup (upm_base_support): mm: restrictedmem: use inclusive ranges for issuing invalidations

Quoted 8 lines
Quoted 5 lines

As many kernel APIs treat 'end' as exclusive, I would rather keep using exclusive 'end' for these APIs(restrictedmem_bind/restrictedmem_unbind and notifier callbacks) but fix it internally in the restrictedmem. E.g. all the places where xarray API needs a 'last'/'max' we use 'end - 1'. See below for the change.

Yes I did feel like I was fighting the kernel a bit on that; your suggestion seems like it would be a better fit.

Comically belated +1, XArray is the odd one here.

author
#328 lore raw

Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM

In reply to Sean Christopherson
Quoted 73 lines · Sean Christopherson
Quoted 63 lines · Isaku Yamahata
Quoted 60 lines · Sean Christopherson
Quoted 49 lines · Isaku Yamahata
Quoted 38 lines · Sean Christopherson
Quoted 10 lines · Chao Peng

This patch series implements KVM guest private memory for confidential computing scenarios like Intel TDX[1]. If a TDX host accesses TDX-protected guest memory, machine check can happen which can further crash the running host system, this is terrible for multi-tenant configurations. The host accesses include those from KVM userspace like QEMU. This series addresses KVM userspace induced crash by introducing new mm and KVM interfaces so KVM userspace can still manage guest memory via a fd-based approach, but it can never access the guest memory content.

Quoted 5 lines

The patch series touches both core mm and KVM code. I appreciate Andrew/Hugh and Paolo/Sean can review and pick these patches. Any other reviews are always welcome.
- 01: mm change, target for mm tree
- 02-09: KVM change, target for KVM tree

A version with all of my feedback, plus reworked versions of Vishal's selftest, is available here:

[email protected]:sean-jc/linux.git x86/upm_base_support

It compiles and passes the selftest, but it's otherwise barely tested. There are a few todos (2 I think?) and many of the commits need changelogs, i.e. it's still a WIP.

As for next steps, can you (handwaving all of the TDX folks) take a look at what I pushed and see if there's anything horrifically broken, and that it still works for TDX?

Fuad (and pKVM folks) same ask for you with respect to pKVM. Absolutely no rush (and I mean that).

On my side, the two things on my mind are (a) tests and (b) downstream dependencies (SEV and TDX). For tests, I want to build a lists of tests that are required for merging so that the criteria for merging are clear, and so that if the list is large (haven't thought much yet), the work of writing and running tests can be distributed.

Quoted 7 lines

Regarding downstream dependencies, before this lands, I want to pull in all the TDX and SNP series and see how everything fits together. Specifically, I want to make sure that we don't end up with a uAPI that necessitates ugly code, and that we don't miss an opportunity to make things simpler. The patches in the SNP series to add "legacy" SEV support for UPM in particular made me slightly rethink some minor details. Nothing remotely major, but something that needs attention since it'll be uAPI.

Quoted 4 lines

Although I'm still debuging with TDX KVM, I needed the following. kvm_faultin_pfn() is called without mmu_lock held. the race to change private/shared is handled by mmu_seq. Maybe dedicated function only for kvm_faultin_pfn().

Gah, you're not on the other thread where this was discussed[*]. Simply deleting the lockdep assertion is safe, for guest types that rely on the attributes to define shared vs. private, KVM rechecks the attributes under the protection of mmu_seq.

Quoted 8 lines
Quoted 3 lines

I'll get a fixed version pushed out today.

[*] https://lore.kernel.org/all/[email protected]

Now I have tdx kvm working. I've uploaded at the followings. It's rebased to v6.2-rc3.
[email protected]:yamahata/linux.git tdx/upm [email protected]:yamahata/qemu.git tdx/upm

Quoted 8 lines

And I finally got a working, building version updated and pushed out (again to):

[email protected]:sean-jc/linux.git x86/upm_base_support

Took longer than expected to get the memslot restrictions sussed out. I'm done working on the code for now, my plan is to come back to it+TDX+SNP in 2-3 weeks to resolves any remaining todos (that no one else tackles) and to do the whole "merge the world" excersise.

Hi Sean,

In case you started working on the code again, I have a branch [1] originally planned as v11 candidate which I believe I addressed all the discussions we had for v10 except the very latest one [2] and integrated all the newly added selftests from Ackerley and myself. The branch was based on your original upm_base_support and then rebased to your kvm-x86/mmu head. Feel free to take anything you think useful( most of them are trivial things but also some fixes for bugs).

[1] https://github.com/chao-p/linux/commits/privmem-v11.6
[2] https://lore.kernel.org/all/[email protected]/

Chao

Quoted 7 lines
Quoted 5 lines

kvm_mmu_do_page_fault() needs the following change. kvm_mem_is_private() queries mem_attr_array. kvm_faultin_pfn() also uses kvm_mem_is_private(). So the shared-private check in kvm_faultin_pfn() doesn't make sense. This change would belong to TDX KVM patches, though.

Yeah, SNP needs similar treatment. Sorting that out is high up on the todo list.

#329 lore raw

Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM

Quoted 7 lines · Chao Peng

In case you started working on the code again, I have a branch [1] originally planned as v11 candidate which I believe I addressed all the discussions we had for v10 except the very latest one [2] and integrated all the newly added selftests from Ackerley and myself. The branch was based on your original upm_base_support and then rebased to your kvm-x86/mmu head. Feel free to take anything you think useful( most of them are trivial things but also some fixes for bugs).

Nice! I am going to work on splicing together the various series this week, I'll make sure to grab your work.

Thanks much!

#330 lore raw

Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM

In reply to Sean Christopherson
Quoted 17 lines · Sean Christopherson
Quoted 10 lines · Chao Peng

This patch series implements KVM guest private memory for confidential computing scenarios like Intel TDX[1]. If a TDX host accesses TDX-protected guest memory, machine check can happen which can further crash the running host system, this is terrible for multi-tenant configurations. The host accesses include those from KVM userspace like QEMU. This series addresses KVM userspace induced crash by introducing new mm and KVM interfaces so KVM userspace can still manage guest memory via a fd-based approach, but it can never access the guest memory content.

Quoted 5 lines

The patch series touches both core mm and KVM code. I appreciate Andrew/Hugh and Paolo/Sean can review and pick these patches. Any other reviews are always welcome.
- 01: mm change, target for mm tree
- 02-09: KVM change, target for KVM tree

Quoted 9 lines

A version with all of my feedback, plus reworked versions of Vishal's selftest, is available here:

[email protected]:sean-jc/linux.git x86/upm_base_support

It compiles and passes the selftest, but it's otherwise barely tested. There are a few todos (2 I think?) and many of the commits need changelogs, i.e. it's still a WIP.

When running LTP (https://github.com/linux-test-project/ltp) on the v10 bits (and also with Sean's branch above) I encounter the following NULL pointer dereference with testcases/kernel/syscalls/madvise/madvise01 (100% reproducible).

It appears that in restrictedmem_error_page() inode->i_mapping->private_data is NULL in the list_for_each_entry_safe(inode, next, &sb->s_inodes, i_sb_list) but I don't know why.

[ 5365.177168] BUG: kernel NULL pointer dereference, address: 0000000000000028
[ 5365.178881] #PF: supervisor read access in kernel mode
[ 5365.180006] #PF: error_code(0x0000) - not-present page
[ 5365.181322] PGD 8000000109dad067 P4D 8000000109dad067 PUD 107707067 PMD 0
[ 5365.183474] Oops: 0000 [#1] PREEMPT SMP PTI
[ 5365.184792] CPU: 0 PID: 22086 Comm: madvise01 Not tainted 6.1.0-1.el8.seanjcupm.x86_64 #1
[ 5365.186572] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.5.1 06/16/2021
[ 5365.188816] RIP: 0010:restrictedmem_error_page+0xc7/0x1b0
[ 5365.190081] Code: 99 00 48 8b 55 00 48 8b 02 48 8d 8a e8 fe ff ff 48 2d 18 01 00 00 48 39 d5 0f 84 8a 00 00 00 48 8b 51 30 48 8b 92 b8 00 00 00 <48> 8b 4a 28 4c 39 b1 d8 00 00 00 74 22 48 8b 88 18 01 00 00 48 8d
[ 5365.193984] RSP: 0018:ffff9b7343c07d80 EFLAGS: 00010206
[ 5365.195142] RAX: ffff8e5b410cfc70 RBX: 0000000000000001 RCX: ffff8e5b4048e580
[ 5365.196888] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 5365.198399] RBP: ffff8e5b410cfd88 R08: 0000000000000000 R09: 0000000000000000
[ 5365.200200] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[ 5365.201843] R13: ffff8e5b410cfd80 R14: ffff8e5b47cc7618 R15: ffffd49d44c05080
[ 5365.203472] FS:  00007fc96de9b5c0(0000) GS:ffff8e5deda00000(0000) knlGS:0000000000000000
[ 5365.205485] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 5365.206791] CR2: 0000000000000028 CR3: 000000012047e002 CR4: 0000000000170ef0
[ 5365.208131] Call Trace:
[ 5365.208752]  <TASK>
[ 5365.209229]  me_pagecache_clean+0x58/0x100
[ 5365.210196]  identify_page_state+0x84/0xd0
[ 5365.211180]  memory_failure+0x231/0x8b0
[ 5365.212148]  madvise_inject_error.cold+0x8d/0xa4
[ 5365.213317]  do_madvise+0x363/0x3a0
[ 5365.214177]  __x64_sys_madvise+0x2c/0x40
[ 5365.215159]  do_syscall_64+0x3f/0xa0
[ 5365.216016]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[ 5365.217130] RIP: 0033:0x7fc96d8399ab
[ 5365.217953] Code: 73 01 c3 48 8b 0d dd 54 38 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa b8 1c 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d ad 54 38 00 f7 d8 64 89 01 48
[ 5365.222323] RSP: 002b:00007fff62a99b18 EFLAGS: 00000206 ORIG_RAX: 000000000000001c
[ 5365.224026] RAX: ffffffffffffffda RBX: 000000000041ce00 RCX: 00007fc96d8399ab
[ 5365.225375] RDX: 0000000000000064 RSI: 000000000000a000 RDI: 00007fc96de8e000
[ 5365.226999] RBP: 00007fc96de9b540 R08: 0000000000000001 R09: 0000000000415c80
[ 5365.228641] R10: 0000000000000001 R11: 0000000000000206 R12: 0000000000000008
[ 5365.230074] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000

Regards,
Liam

Quoted 25 lines

As for next steps, can you (handwaving all of the TDX folks) take a look at what I pushed and see if there's anything horrifically broken, and that it still works for TDX?

Fuad (and pKVM folks) same ask for you with respect to pKVM. Absolutely no rush (and I mean that).

On my side, the two things on my mind are (a) tests and (b) downstream dependencies (SEV and TDX). For tests, I want to build a lists of tests that are required for merging so that the criteria for merging are clear, and so that if the list is large (haven't thought much yet), the work of writing and running tests can be distributed.

Regarding downstream dependencies, before this lands, I want to pull in all the TDX and SNP series and see how everything fits together. Specifically, I want to make sure that we don't end up with a uAPI that necessitates ugly code, and that we don't miss an opportunity to make things simpler. The patches in the SNP series to add "legacy" SEV support for UPM in particular made me slightly rethink some minor details. Nothing remotely major, but something that needs attention since it'll be uAPI.

I'm off Monday, so it'll be at least Tuesday before I make any more progress on my side.

Thanks!

#331 lore raw

Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM

Quoted 33 lines · Liam Merwick
Quoted 17 lines · Sean Christopherson
Quoted 10 lines · Chao Peng

This patch series implements KVM guest private memory for confidential computing scenarios like Intel TDX[1]. If a TDX host accesses TDX-protected guest memory, machine check can happen which can further crash the running host system, this is terrible for multi-tenant configurations. The host accesses include those from KVM userspace like QEMU. This series addresses KVM userspace induced crash by introducing new mm and KVM interfaces so KVM userspace can still manage guest memory via a fd-based approach, but it can never access the guest memory content.

Quoted 5 lines

The patch series touches both core mm and KVM code. I appreciate Andrew/Hugh and Paolo/Sean can review and pick these patches. Any other reviews are always welcome.
- 01: mm change, target for mm tree
- 02-09: KVM change, target for KVM tree

Quoted 9 lines

A version with all of my feedback, plus reworked versions of Vishal's selftest, is available here:

[email protected]:sean-jc/linux.git x86/upm_base_support

It compiles and passes the selftest, but it's otherwise barely tested. There are a few todos (2 I think?) and many of the commits need changelogs, i.e. it's still a WIP.

When running LTP (https://github.com/linux-test-project/ltp) on the v10 bits (and also with Sean's branch above) I encounter the following NULL pointer dereference with testcases/kernel/syscalls/madvise/madvise01 (100% reproducible).

Quoted 4 lines

It appears that in restrictedmem_error_page() inode->i_mapping->private_data is NULL
in the list_for_each_entry_safe(inode, next, &sb->s_inodes, i_sb_list) but I don't know why.

Kirill, can you take a look? Or pass the buck to someone who can? :-)

#332 lore raw

Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM

Quoted 34 lines · Sean Christopherson
Quoted 33 lines · Liam Merwick
Quoted 17 lines · Sean Christopherson
Quoted 10 lines · Chao Peng

This patch series implements KVM guest private memory for confidential computing scenarios like Intel TDX[1]. If a TDX host accesses TDX-protected guest memory, machine check can happen which can further crash the running host system, this is terrible for multi-tenant configurations. The host accesses include those from KVM userspace like QEMU. This series addresses KVM userspace induced crash by introducing new mm and KVM interfaces so KVM userspace can still manage guest memory via a fd-based approach, but it can never access the guest memory content.

Quoted 5 lines

The patch series touches both core mm and KVM code. I appreciate Andrew/Hugh and Paolo/Sean can review and pick these patches. Any other reviews are always welcome.
- 01: mm change, target for mm tree
- 02-09: KVM change, target for KVM tree

Quoted 9 lines

A version with all of my feedback, plus reworked versions of Vishal's selftest, is available here:

[email protected]:sean-jc/linux.git x86/upm_base_support

It compiles and passes the selftest, but it's otherwise barely tested. There are a few todos (2 I think?) and many of the commits need changelogs, i.e. it's still a WIP.

When running LTP (https://github.com/linux-test-project/ltp) on the v10 bits (and also with Sean's branch above) I encounter the following NULL pointer dereference with testcases/kernel/syscalls/madvise/madvise01 (100% reproducible).

Quoted 6 lines
Quoted 4 lines

It appears that in restrictedmem_error_page() inode->i_mapping->private_data is NULL
in the list_for_each_entry_safe(inode, next, &sb->s_inodes, i_sb_list) but I don't know why.

Kirill, can you take a look? Or pass the buck to someone who can? :-)

The patch below should help.

mm/restrictedmem.c +18 -3
index 15c52301eeb9..39ada985c7c0 100644
@@ -307,14 +307,29 @@ void restrictedmem_error_page(struct page *page, struct address_space *mapping)
 
 	spin_lock(&sb->s_inode_list_lock);
 	list_for_each_entry_safe(inode, next, &sb->s_inodes, i_sb_list) {
-		struct restrictedmem *rm = inode->i_mapping->private_data;
 		struct restrictedmem_notifier *notifier;
-		struct file *memfd = rm->memfd;
+		struct restrictedmem *rm;
 		unsigned long index;
+		struct file *memfd;
 
-		if (memfd->f_mapping != mapping)
+		if (atomic_read(&inode->i_count))
 			continue;
 
+		spin_lock(&inode->i_lock);
+		if (inode->i_state & (I_NEW | I_FREEING | I_WILL_FREE)) {
+			spin_unlock(&inode->i_lock);
+			continue;
+		}
+
+		rm = inode->i_mapping->private_data;
+		memfd = rm->memfd;
+
+		if (memfd->f_mapping != mapping) {
+			spin_unlock(&inode->i_lock);
+			continue;
+		}
+		spin_unlock(&inode->i_lock);
+
 		xa_for_each_range(&rm->bindings, index, notifier, start, end)
 			notifier->ops->error(notifier, start, end);
 		break;
Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
#333 lore raw

Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM

Quoted 4 lines · Kirill A. Shutemov
Quoted 3 lines · Sean Christopherson
Quoted 2 lines · Liam Merwick
Quoted 1 line · Sean Christopherson

On Fri, Dec 02, 2022, Chao Peng wrote:

...

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

When running LTP (https://github.com/linux-test-project/ltp) on the v10 bits (and also with Sean's branch above) I encounter the following NULL pointer dereference with testcases/kernel/syscalls/madvise/madvise01 (100% reproducible).

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

It appears that in restrictedmem_error_page() inode->i_mapping->private_data is NULL
in the list_for_each_entry_safe(inode, next, &sb->s_inodes, i_sb_list) but I don't know why.

Kirill, can you take a look? Or pass the buck to someone who can? :-)

The patch below should help.

Thanks, this works for me.

Regards,
Liam

Quoted 38 lines
mm/restrictedmem.c +18 -3
index 15c52301eeb9..39ada985c7c0 100644
@@ -307,14 +307,29 @@ void restrictedmem_error_page(struct page *page, struct address_space *mapping)
 
  	spin_lock(&sb->s_inode_list_lock);
  	list_for_each_entry_safe(inode, next, &sb->s_inodes, i_sb_list) {
-		struct restrictedmem *rm = inode->i_mapping->private_data;
  		struct restrictedmem_notifier *notifier;
-		struct file *memfd = rm->memfd;
+		struct restrictedmem *rm;
  		unsigned long index;
+		struct file *memfd;
 
-		if (memfd->f_mapping != mapping)
+		if (atomic_read(&inode->i_count))
  			continue;
 
+		spin_lock(&inode->i_lock);
+		if (inode->i_state & (I_NEW | I_FREEING | I_WILL_FREE)) {
+			spin_unlock(&inode->i_lock);
+			continue;
+		}
+
+		rm = inode->i_mapping->private_data;
+		memfd = rm->memfd;
+
+		if (memfd->f_mapping != mapping) {
+			spin_unlock(&inode->i_lock);
+			continue;
+		}
+		spin_unlock(&inode->i_lock);
+
  		xa_for_each_range(&rm->bindings, index, notifier, start, end)
  			notifier->ops->error(notifier, start, end);
  		break;
#334 lore raw

Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM

In reply to Kirill A. Shutemov
Quoted 51 lines · Kirill A. Shutemov
Quoted 34 lines · Sean Christopherson
Quoted 33 lines · Liam Merwick
Quoted 17 lines · Sean Christopherson
Quoted 10 lines · Chao Peng

This patch series implements KVM guest private memory for confidential computing scenarios like Intel TDX[1]. If a TDX host accesses TDX-protected guest memory, machine check can happen which can further crash the running host system, this is terrible for multi-tenant configurations. The host accesses include those from KVM userspace like QEMU. This series addresses KVM userspace induced crash by introducing new mm and KVM interfaces so KVM userspace can still manage guest memory via a fd-based approach, but it can never access the guest memory content.

Quoted 5 lines

The patch series touches both core mm and KVM code. I appreciate Andrew/Hugh and Paolo/Sean can review and pick these patches. Any other reviews are always welcome.
- 01: mm change, target for mm tree
- 02-09: KVM change, target for KVM tree

Quoted 9 lines

A version with all of my feedback, plus reworked versions of Vishal's selftest, is available here:

[email protected]:sean-jc/linux.git x86/upm_base_support

It compiles and passes the selftest, but it's otherwise barely tested. There are a few todos (2 I think?) and many of the commits need changelogs, i.e. it's still a WIP.

When running LTP (https://github.com/linux-test-project/ltp) on the v10 bits (and also with Sean's branch above) I encounter the following NULL pointer dereference with testcases/kernel/syscalls/madvise/madvise01 (100% reproducible).

Quoted 6 lines
Quoted 4 lines

It appears that in restrictedmem_error_page() inode->i_mapping->private_data is NULL in the list_for_each_entry_safe(inode, next, &sb->s_inodes, i_sb_list) but I don't know why.

Kirill, can you take a look? Or pass the buck to someone who can? :-)

The patch below should help.

mm/restrictedmem.c +0 -0
index 15c52301eeb9..39ada985c7c0 100644
@@ -307,14 +307,29 @@ void restrictedmem_error_page(struct page *page, struct address_space *mapping)
 
 	spin_lock(&sb->s_inode_list_lock);

mm/restrictedmem.c void restrictedmem_error_page(struct page *page, struct address_space *mapping) · line 315

Quoted 10 lines
mm/restrictedmem.c +3 -3
 	list_for_each_entry_safe(inode, next, &sb->s_inodes, i_sb_list) {
-		struct restrictedmem *rm = inode->i_mapping->private_data;
 		struct restrictedmem_notifier *notifier;
-		struct file *memfd = rm->memfd;
+		struct restrictedmem *rm;
 		unsigned long index;
+		struct file *memfd;
 
-		if (memfd->f_mapping != mapping)
+		if (atomic_read(&inode->i_count))

Kirill, should this be

		if (!atomic_read(&inode->i_count))
			continue;

i.e. skip unreferenced inodes, not skip referenced inodes?

#335 lore raw

Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM

Quoted 65 lines · Sean Christopherson
Quoted 51 lines · Kirill A. Shutemov
Quoted 34 lines · Sean Christopherson
Quoted 33 lines · Liam Merwick
Quoted 17 lines · Sean Christopherson
Quoted 10 lines · Chao Peng

This patch series implements KVM guest private memory for confidential computing scenarios like Intel TDX[1]. If a TDX host accesses TDX-protected guest memory, machine check can happen which can further crash the running host system, this is terrible for multi-tenant configurations. The host accesses include those from KVM userspace like QEMU. This series addresses KVM userspace induced crash by introducing new mm and KVM interfaces so KVM userspace can still manage guest memory via a fd-based approach, but it can never access the guest memory content.

Quoted 5 lines

The patch series touches both core mm and KVM code. I appreciate Andrew/Hugh and Paolo/Sean can review and pick these patches. Any other reviews are always welcome.
- 01: mm change, target for mm tree
- 02-09: KVM change, target for KVM tree

Quoted 9 lines

A version with all of my feedback, plus reworked versions of Vishal's selftest, is available here:

[email protected]:sean-jc/linux.git x86/upm_base_support

It compiles and passes the selftest, but it's otherwise barely tested. There are a few todos (2 I think?) and many of the commits need changelogs, i.e. it's still a WIP.

When running LTP (https://github.com/linux-test-project/ltp) on the v10 bits (and also with Sean's branch above) I encounter the following NULL pointer dereference with testcases/kernel/syscalls/madvise/madvise01 (100% reproducible).

Quoted 6 lines
Quoted 4 lines

It appears that in restrictedmem_error_page() inode->i_mapping->private_data is NULL in the list_for_each_entry_safe(inode, next, &sb->s_inodes, i_sb_list) but I don't know why.

Kirill, can you take a look? Or pass the buck to someone who can? :-)

The patch below should help.

mm/restrictedmem.c +0 -0
index 15c52301eeb9..39ada985c7c0 100644
@@ -307,14 +307,29 @@ void restrictedmem_error_page(struct page *page, struct address_space *mapping)
 
 	spin_lock(&sb->s_inode_list_lock);

mm/restrictedmem.c void restrictedmem_error_page(struct page *page, struct address_space *mapping) · line 315

Quoted 10 lines
mm/restrictedmem.c +3 -3
 	list_for_each_entry_safe(inode, next, &sb->s_inodes, i_sb_list) {
-		struct restrictedmem *rm = inode->i_mapping->private_data;
 		struct restrictedmem_notifier *notifier;
-		struct file *memfd = rm->memfd;
+		struct restrictedmem *rm;
 		unsigned long index;
+		struct file *memfd;
 
-		if (memfd->f_mapping != mapping)
+		if (atomic_read(&inode->i_count))

Kirill, should this be

Quoted 4 lines
		if (!atomic_read(&inode->i_count))
			continue;

i.e. skip unreferenced inodes, not skip referenced inodes?

Ouch. Yes.

But looking at other instances of s_inodes usage, I think we can drop the check altogether. inode cannot be completely free until it is removed from s_inodes list.

While there, replace list_for_each_entry_safe() with list_for_each_entry() as we don't remove anything from the list.

mm/restrictedmem.c +2 -5
index 55e99e6c09a1..8e8a4420d3d1 100644
@@ -194,22 +194,19 @@ static int restricted_error_remove_page(struct address_space *mapping,
 					struct page *page)
 {
 	struct super_block *sb = restrictedmem_mnt->mnt_sb;
-	struct inode *inode, *next;
+	struct inode *inode;
 	pgoff_t start, end;
 
 	start = page->index;
 	end = start + thp_nr_pages(page);
 
 	spin_lock(&sb->s_inode_list_lock);
-	list_for_each_entry_safe(inode, next, &sb->s_inodes, i_sb_list) {
+	list_for_each_entry(inode, &sb->s_inodes, i_sb_list) {
 		struct restrictedmem_notifier *notifier;
 		struct restrictedmem *rm;
 		unsigned long index;
 		struct file *memfd;
 
-		if (atomic_read(&inode->i_count))
-			continue;
-
 		spin_lock(&inode->i_lock);
 		if (inode->i_state & (I_NEW | I_FREEING | I_WILL_FREE)) {
 			spin_unlock(&inode->i_lock);
Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
reviewer
#336 lore raw

Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM

In reply to Chao Peng

Hi,

Quoted 9 lines · Chao Peng

This patch series implements KVM guest private memory for confidential computing scenarios like Intel TDX[1]. If a TDX host accesses TDX-protected guest memory, machine check can happen which can further crash the running host system, this is terrible for multi-tenant configurations. The host accesses include those from KVM userspace like QEMU. This series addresses KVM userspace induced crash by introducing new mm and KVM interfaces so KVM userspace can still manage guest memory via a fd-based approach, but it can never access the guest memory content.

Sorry for jumping late.

Unless I'm missing something, hibernation will also cause an machine check when there is TDX-protected memory in the system. When the hibernation creates memory snapshot it essentially walks all physical pages and saves their contents, so for TDX memory this will trigger machine check, right?

Quoted 26 lines
 Documentation/virt/kvm/api.rst         | 125 ++++++-
 arch/x86/entry/syscalls/syscall_32.tbl |   1 +
 arch/x86/entry/syscalls/syscall_64.tbl |   1 +
 arch/x86/include/asm/kvm_host.h        |   9 +
 arch/x86/kvm/Kconfig                   |   3 +
 arch/x86/kvm/mmu/mmu.c                 | 205 ++++++++++-
 arch/x86/kvm/mmu/mmu_internal.h        |  14 +-
 arch/x86/kvm/mmu/mmutrace.h            |   1 +
 arch/x86/kvm/mmu/tdp_mmu.c             |   2 +-
 arch/x86/kvm/x86.c                     |  17 +-
 include/linux/kvm_host.h               | 103 +++++-
 include/linux/restrictedmem.h          |  71 ++++
 include/linux/syscalls.h               |   1 +
 include/uapi/asm-generic/unistd.h      |   5 +-
 include/uapi/linux/kvm.h               |  53 +++
 include/uapi/linux/magic.h             |   1 +
 kernel/sys_ni.c                        |   3 +
 mm/Kconfig                             |   4 +
 mm/Makefile                            |   1 +
 mm/memory-failure.c                    |   3 +
 mm/restrictedmem.c                     | 318 +++++++++++++++++
 virt/kvm/Kconfig                       |   6 +
 virt/kvm/kvm_main.c                    | 469 +++++++++++++++++++++----
 23 files changed, 1323 insertions(+), 93 deletions(-)
 create mode 100644 include/linux/restrictedmem.h
 create mode 100644 mm/restrictedmem.c
Signature · 2 lines
Sincerely yours,
Mike.
#337 lore raw

Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM

Quoted 13 lines · Mike Rapoport

Hi,

Quoted 9 lines · Chao Peng

This patch series implements KVM guest private memory for confidential computing scenarios like Intel TDX[1]. If a TDX host accesses TDX-protected guest memory, machine check can happen which can further crash the running host system, this is terrible for multi-tenant configurations. The host accesses include those from KVM userspace like QEMU. This series addresses KVM userspace induced crash by introducing new mm and KVM interfaces so KVM userspace can still manage guest memory via a fd-based approach, but it can never access the guest memory content.

Quoted 6 lines

Sorry for jumping late.

Unless I'm missing something, hibernation will also cause an machine check when there is TDX-protected memory in the system. When the hibernation creates memory snapshot it essentially walks all physical pages and saves their contents, so for TDX memory this will trigger machine check, right?

I recall bringing that up in the past (also memory access due to kdump, /prov/kcore) and was told that the main focus for now is preventing unprivileged users from crashing the system, that is, not mapping such memory into user space (e.g., QEMU). In the long run, we'll want to handle such pages also properly in the other events where the kernel might access them.

Signature · 3 lines
Thanks,

David / dhildenb
#338 lore raw

Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM

Quoted 14 lines · David Hildenbrand
Quoted 13 lines · Mike Rapoport

Hi,

Quoted 9 lines · Chao Peng

This patch series implements KVM guest private memory for confidential computing scenarios like Intel TDX[1]. If a TDX host accesses TDX-protected guest memory, machine check can happen which can further crash the running host system, this is terrible for multi-tenant configurations. The host accesses include those from KVM userspace like QEMU. This series addresses KVM userspace induced crash by introducing new mm and KVM interfaces so KVM userspace can still manage guest memory via a fd-based approach, but it can never access the guest memory content.

Quoted 6 lines
Quoted 6 lines

Sorry for jumping late.

Unless I'm missing something, hibernation will also cause an machine check when there is TDX-protected memory in the system. When the hibernation creates memory snapshot it essentially walks all physical pages and saves their contents, so for TDX memory this will trigger machine check, right?

For hibernation specifically, I think that should be handled elsewhere as hibernation is simply incompatible with TDX, SNP, pKVM, etc. without paravirtualizing the guest, as none of those technologies support auto-export a la s390. I suspect the right approach is to disallow hibernation if KVM is running any protected guests.

Quoted 6 lines

I recall bringing that up in the past (also memory access due to kdump, /prov/kcore) and was told that the main focus for now is preventing unprivileged users from crashing the system, that is, not mapping such memory into user space (e.g., QEMU). In the long run, we'll want to handle such pages also properly in the other events where the kernel might access them.

Ya, unless someone strongly objects, the plan is to essentially treat "attacks" from privileged users as out of to scope for initial support, and then iterate as needed to fix/enable more features.

FWIW, read accesses, e.g. kdump, should be ok for TDX and SNP as they both play nice with "bad" reads. pKVM is a different beast though as I believe any access to guest private memory will fault. But my understanding is that this series would be a big step forward for pKVM, which currently doesn't have any safeguards.

#339 lore raw

Rename restrictedmem => guardedmem? (was: Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM)

In reply to Chao Peng

What do y'all think about renaming "restrictedmem" to "guardedmem"?

I want to start referring to the code/patches by its syscall/implementation name instead of "UPM", as "UPM" is (a) very KVM centric, (b) refers to the broader effort and not just the non-KVM code, and (c) will likely be confusing for future reviewers since there's nothing in the code that mentions "UPM" in any way.

But typing out restrictedmem is quite tedious, and git grep shows that "rmem" is already used to refer to "reserved memory".

Renaming the syscall to "guardedmem"...

1. Allows for a shorthand and namespace, "gmem", that isn't already in use by the kernel (see "reserved memory above").

2. Provides a stronger hint as to its purpose. "Restricted" conveys that the allocated memory is limited in some way, but doesn't capture how the memory is restricted, e.g. "restricted" could just as easily mean that the allocation can be restricted to certain types of backing stores or something. "Guarded" on the other hand captures that the memory has extra defenses of some form.

3. Is shorter to type and speak. Work smart, not hard :-)

4. Isn't totally wrong for the KVM use case if someone assumes the "g" means "guest" when reading mail and whatnot.

P.S. I trimmed the Cc/To substantially for this particular discussion to avoid
spamming folks that don't (yet) care about this stuff with another potentially lengthy thread. Feel free to add (back) any people/lists.

#340 lore raw

Re: Rename restrictedmem => guardedmem? (was: Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM)

Quoted 1 line · Sean Christopherson

What do y'all think about renaming "restrictedmem" to "guardedmem"?

Yeay, let's add more confusion :D

If we're at renaming, I'd appreciate if we could find a terminology that does look/sound less horrible.

Quoted 10 lines

I want to start referring to the code/patches by its syscall/implementation name instead of "UPM", as "UPM" is (a) very KVM centric, (b) refers to the broader effort and not just the non-KVM code, and (c) will likely be confusing for future reviewers since there's nothing in the code that mentions "UPM" in any way.

But typing out restrictedmem is quite tedious, and git grep shows that "rmem" is already used to refer to "reserved memory".

Renaming the syscall to "guardedmem"...

restrictedmem, guardedmem, ... all fairly "suboptimal" if you'd ask me ...

Signature · 3 lines
Thanks,

David / dhildenb
#341 lore raw

Re: Rename restrictedmem => guardedmem? (was: Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM)

Quoted 6 lines · David Hildenbrand
Quoted 5 lines · Sean Christopherson

I want to start referring to the code/patches by its syscall/implementation name instead of "UPM", as "UPM" is (a) very KVM centric, (b) refers to the broader effort and not just the non-KVM code, and (c) will likely be confusing for future reviewers since there's nothing in the code that mentions "UPM" in any way.

Quoted 6 lines
Quoted 4 lines

But typing out restrictedmem is quite tedious, and git grep shows that "rmem" is already used to refer to "reserved memory".

Renaming the syscall to "guardedmem"...

restrictedmem, guardedmem, ... all fairly "suboptimal" if you'd ask me ...

I'm definitely open to other suggestions, but I suspect it's going to be difficult to be more precise than something like "guarded".

E.g. we discussed "unmappable" at one point, but the memory can still be mapped, just not via mmap(). And it's not just about mappings, e.g. read() and its many variants are all disallowed too, despite the kernel direct map still being live (modulo SNP requirements).

#342 lore raw

Re: Rename restrictedmem => guardedmem? (was: Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM)

Quoted 10 lines · Sean Christopherson
Quoted 9 lines · David Hildenbrand
Quoted 8 lines · Sean Christopherson

I want to start referring to the code/patches by its syscall/implementation name instead of "UPM", as "UPM" is (a) very KVM centric, (b) refers to the broader effort and not just the non-KVM code, and (c) will likely be confusing for future reviewers since there's nothing in the code that mentions "UPM" in any way.

But typing out restrictedmem is quite tedious, and git grep shows that "rmem" is already used to refer to "reserved memory".

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Renaming the syscall to "guardedmem"...

restrictedmem, guardedmem, ... all fairly "suboptimal" if you'd ask me ...

I'm definitely open to other suggestions, but I suspect it's going to be difficult to be more precise than something like "guarded".

Guardedmem is just as bad as restrictedmem IMHO, sorry.

Restricted: what's restricted? how does the restriction manifest? secretmem also has it's restrictions/limitations (pinning), why does that one not fall under the same category?

Make a stranger guess what "restrictedmem" is and I can guarantee that it has nothing to do with the concept we're introducing here.

Guarded: what's guarded? From whom? For which purpose? How does the "guarding" manifest?

Again, make a stranger guess what "guardedmem" is and I can guarantee that it has nothing to do with the concept we're introducing here.

If, at all, the guess might be "guarded storage" [1] on s390x, which, of course, has nothing to do with the concept here. (storage on s390x is just the dinosaur slang for memory)

Often, if we fail to find a good name, the concept is either unclear or not well defined.

So what are the characteristics we want to generalize under that new name? We want to have an fd, that

(a) cannot be mapped into user space (mmap)
(b) cannot be accessed using ordinary system calls (read/write)
(c) can still be managed like other fds (fallocate, future NUMA policies?)
(d) can be consumed by some special entities that are allowed to read/write/map.

So the fd content is inaccessible using the ordinary POSIX syscalls. It's only accessible by special entities (e.g., KVM).

Most probably I am forgetting something. But maybe that will help to find a more expressive name. Maybe :)

Quoted 6 lines

E.g. we discussed "unmappable" at one point, but the memory can still be mapped, just not via mmap(). And it's not just about mappings, e.g. read() and its many variants are all disallowed too, despite the kernel direct map still being live (modulo SNP requirements).

[1] https://man.archlinux.org/man/s390_guarded_storage.2.en

Signature · 3 lines
Thanks,

David / dhildenb
#343 lore raw

Re: Rename restrictedmem => guardedmem? (was: Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM)

Quoted 25 lines · David Hildenbrand
Quoted 10 lines · Sean Christopherson
Quoted 9 lines · David Hildenbrand
Quoted 8 lines · Sean Christopherson

I want to start referring to the code/patches by its syscall/implementation name instead of "UPM", as "UPM" is (a) very KVM centric, (b) refers to the broader effort and not just the non-KVM code, and (c) will likely be confusing for future reviewers since there's nothing in the code that mentions "UPM" in any way.

But typing out restrictedmem is quite tedious, and git grep shows that "rmem" is already used to refer to "reserved memory".

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Renaming the syscall to "guardedmem"...

restrictedmem, guardedmem, ... all fairly "suboptimal" if you'd ask me ...

I'm definitely open to other suggestions, but I suspect it's going to be difficult to be more precise than something like "guarded".

Guardedmem is just as bad as restrictedmem IMHO, sorry.

Restricted: what's restricted? how does the restriction manifest? secretmem also has it's restrictions/limitations (pinning), why does that one not fall under the same category?

Quoted 6 lines

Make a stranger guess what "restrictedmem" is and I can guarantee that it has nothing to do with the concept we're introducing here.

Guarded: what's guarded? From whom? For which purpose? How does the "guarding" manifest?

I completely agree that "guarded" lacks precision, but as above, I've pretty much given up hope of being super precise. I actually like "restricted", I just don't like that I can't shorten the name.

Hmm, maybe that won't be a huge problem in practice. I can't say I've ever heard any use "rmem" in verbale or written communication, it's primarily just "rmem" in code that we can't use, and I don't mind having to use restrictedmem for the namespace. So maybe we can use "rmem", just not in code?

Or, we could pretend we're pirates and call it arrrmem!, which is definitely going to be how I refer to it in my internal dialogue if we keep "restricted" :-)

Quoted 5 lines

Again, make a stranger guess what "guardedmem" is and I can guarantee that it has nothing to do with the concept we're introducing here.

If, at all, the guess might be "guarded storage" [1] on s390x, which, of course, has nothing to do with the concept here.

Oof, and guarded storage is even documented in Documentation/virt/kvm/api.rst.

Quoted 16 lines

(storage on s390x is just the dinosaur slang for memory)

Often, if we fail to find a good name, the concept is either unclear or not well defined.

So what are the characteristics we want to generalize under that new name? We want to have an fd, that

(a) cannot be mapped into user space (mmap)
(b) cannot be accessed using ordinary system calls (read/write)
(c) can still be managed like other fds (fallocate, future NUMA policies?)
(d) can be consumed by some special entities that are allowed to read/write/map.

Quoted 5 lines

So the fd content is inaccessible using the ordinary POSIX syscalls. It's only accessible by special entities (e.g., KVM).

Most probably I am forgetting something. But maybe that will help to find a more expressive name. Maybe :)

Hidden/Concealed/etc - Too close to secretmem, suffers the "hidden from whom" problem, and depending on the use case, the memory may not actually be concealed from the user that controls the VMM.

Restricted - "rmem" collides with "reserved memory" in code.

Guarded - Conflicts with s390's "guarded storage", has the "from whom" problem.

Inaccessible - Many of the same problems as "hidden".

Unmappable - Doesn't cover things like read/write, and is wrong in the sense that the memory is still mappable, just not via mmap().

Secured - I'm not getting anywhere near this one :-)

#344 lore raw

Re: Rename restrictedmem => guardedmem? (was: Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM)

On Mon, Apr 17, 2023 at 8:16 PM Sean Christopherson <[email protected]> wrote:
....

Quoted 12 lines
Quoted 5 lines

So the fd content is inaccessible using the ordinary POSIX syscalls. It's only accessible by special entities (e.g., KVM).

Most probably I am forgetting something. But maybe that will help to find a more expressive name. Maybe :)

Hidden/Concealed/etc - Too close to secretmem, suffers the "hidden from whom" problem, and depending on the use case, the memory may not actually be concealed from the user that controls the VMM.

Restricted - "rmem" collides with "reserved memory" in code.

Quoted 8 lines

Guarded - Conflicts with s390's "guarded storage", has the "from whom" problem.

Inaccessible - Many of the same problems as "hidden".

Unmappable - Doesn't cover things like read/write, and is wrong in the sense that the memory is still mappable, just not via mmap().

Secured - I'm not getting anywhere near this one :-)

How about "protected" ;)? _ducks_

To me the name doesn't matter much, but fwiw I have developed a liking to "restricted", more than the previous "private", since of all of the one-word suggestions I think it captures most of what it's trying to do.

Cheers,
/fuad

#345 lore raw

Re: Rename restrictedmem => guardedmem? (was: Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM)

In reply to Sean Christopherson
Quoted 37 lines · Sean Christopherson
Quoted 25 lines · David Hildenbrand
Quoted 10 lines · Sean Christopherson
Quoted 9 lines · David Hildenbrand
Quoted 8 lines · Sean Christopherson

I want to start referring to the code/patches by its syscall/implementation name instead of "UPM", as "UPM" is (a) very KVM centric, (b) refers to the broader effort and not just the non-KVM code, and (c) will likely be confusing for future reviewers since there's nothing in the code that mentions "UPM" in any way.

But typing out restrictedmem is quite tedious, and git grep shows that "rmem" is already used to refer to "reserved memory".

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Renaming the syscall to "guardedmem"...

restrictedmem, guardedmem, ... all fairly "suboptimal" if you'd ask me ...

I'm definitely open to other suggestions, but I suspect it's going to be difficult to be more precise than something like "guarded".

Guardedmem is just as bad as restrictedmem IMHO, sorry.

Restricted: what's restricted? how does the restriction manifest? secretmem also has it's restrictions/limitations (pinning), why does that one not fall under the same category?

Quoted 6 lines

Make a stranger guess what "restrictedmem" is and I can guarantee that it has nothing to do with the concept we're introducing here.

Guarded: what's guarded? From whom? For which purpose? How does the "guarding" manifest?

I completely agree that "guarded" lacks precision, but as above, I've pretty much given up hope of being super precise. I actually like "restricted", I just don't like that I can't shorten the name.

Quoted 7 lines

Hmm, maybe that won't be a huge problem in practice. I can't say I've ever heard any use "rmem" in verbale or written communication, it's primarily just "rmem" in code that we can't use, and I don't mind having to use restrictedmem for the namespace. So maybe we can use "rmem", just not in code?

Or, we could pretend we're pirates and call it arrrmem!, which is definitely going to be how I refer to it in my internal dialogue if we keep "restricted" :-)

:)

Quoted 37 lines
Quoted 6 lines

Again, make a stranger guess what "guardedmem" is and I can guarantee that it has nothing to do with the concept we're introducing here.

If, at all, the guess might be "guarded storage" [1] on s390x, which, of course, has nothing to do with the concept here.

Oof, and guarded storage is even documented in Documentation/virt/kvm/api.rst.

Quoted 16 lines

(storage on s390x is just the dinosaur slang for memory)

Often, if we fail to find a good name, the concept is either unclear or not well defined.

So what are the characteristics we want to generalize under that new name? We want to have an fd, that

(a) cannot be mapped into user space (mmap)
(b) cannot be accessed using ordinary system calls (read/write)
(c) can still be managed like other fds (fallocate, future NUMA policies?)
(d) can be consumed by some special entities that are allowed to read/write/map.

Quoted 5 lines

So the fd content is inaccessible using the ordinary POSIX syscalls. It's only accessible by special entities (e.g., KVM).

Most probably I am forgetting something. But maybe that will help to find a more expressive name. Maybe :)

Hidden/Concealed/etc - Too close to secretmem, suffers the "hidden from whom" problem, and depending on the use case, the memory may not actually be concealed from the user that controls the VMM.

Restricted - "rmem" collides with "reserved memory" in code.

Quoted 8 lines

Guarded - Conflicts with s390's "guarded storage", has the "from whom" problem.

Inaccessible - Many of the same problems as "hidden".

Unmappable - Doesn't cover things like read/write, and is wrong in the sense that the memory is still mappable, just not via mmap().

Secured - I'm not getting anywhere near this one :-)

The think about "secretmem" that I kind-of like (a little) is that it explains what it's used for: storing secrets. We don't call it "unmapped" memory because we unmap it from the directmap or "unpinnable" memory or "inaccessible" memory ... or even "restricted" because it has restrictions ... how the secrets are protected is kind of an implementation detail.

So instead of describing *why*/*how* restrictedmem is the weird kid (restricted/guarded/hidden/restricted/inaccessible/ ...), maybe rather describe what it's used for?

I know, I know, "there are other use cases where it will be used outside of VM context". I really don't care. "memfd_vm" / "vm_mem" would be sooo (feel free to add some more o's here) much easier to get. It's a special fd to be used to back VM memory. Depending on the VM type (encrypted/protected/whatever), restrictions might apply (not able to mmap, not able to read/write ...). For example, there really is no need to disallow mmap/read/write when using that memory to back a simple VM where all we want to do is avoid user-space page tables.

Signature · 3 lines
Thanks,

David / dhildenb
#346 lore raw

Re: Rename restrictedmem => guardedmem? (was: Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM)

Quoted 22 lines · David Hildenbrand
Quoted 6 lines · Sean Christopherson

Hidden/Concealed/etc - Too close to secretmem, suffers the "hidden from whom" problem, and depending on the use case, the memory may not actually be concealed from the user that controls the VMM.

Restricted - "rmem" collides with "reserved memory" in code.

Quoted 8 lines

Guarded - Conflicts with s390's "guarded storage", has the "from whom" problem.

Inaccessible - Many of the same problems as "hidden".

Unmappable - Doesn't cover things like read/write, and is wrong in the sense that the memory is still mappable, just not via mmap().

Secured - I'm not getting anywhere near this one :-)

The think about "secretmem" that I kind-of like (a little) is that it explains what it's used for: storing secrets. We don't call it "unmapped" memory because we unmap it from the directmap or "unpinnable" memory or "inaccessible" memory ... or even "restricted" because it has restrictions ... how the secrets are protected is kind of an implementation detail.

Quoted 6 lines

So instead of describing *why*/*how* restrictedmem is the weird kid (restricted/guarded/hidden/restricted/inaccessible/ ...), maybe rather describe what it's used for?

I know, I know, "there are other use cases where it will be used outside of VM context". I really don't care.

Heh, we originally proposed F_SEAL_GUEST, but that was also sub-optimal[1] ;-)

Quoted 6 lines

"memfd_vm" / "vm_mem" would be sooo (feel free to add some more o's here) much easier to get. It's a special fd to be used to back VM memory. Depending on the VM type (encrypted/protected/whatever), restrictions might apply (not able to mmap, not able to read/write ...). For example, there really is no need to disallow mmap/read/write when using that memory to back a simple VM where all we want to do is avoid user-space page tables.

In seriousness, I do agree with Jason's very explicit objection[2] against naming a non-KVM uAPI "guest", or any variation thereof.

An alternative that we haven't considered since the very early days is making the uAPI a KVM ioctl() instead of a memfd() flag or dedicated syscall. Looking at the code for "pure shim" implementation[3], that's actually not that crazy of an idea.

I don't know that I love the idea of burying this in KVM, but there are benefits to coupling restrictedmem to KVM (aside from getting out from behind this bikeshed that I created).

The big benefit is that the layer of indirection goes away. That simplifies things like enhancing restrictedmem to allow host userspace access for debug purposes, batching TLB flushes if a PUNCH_HOLE spans multiple memslots, enforcing exclusive access, likely the whole "share with a device" story if/when we get there, etc.

The obvious downsides are that (a) maintenance falls under the KVM umbrella, but that's likely to be true in practice regardless of where the code lands, and
(b) if another use case comes along, e.g. the Gunyah hypervisor[4][5], we risk someone reinventing a similar solution.

If we can get Gunyah on board and they don't need substantial changes to the restrictedmem implementation, then I'm all for continuing on the path we're on. But if Gunyah wants to do their own thing, and the lightweight shim approach is viable, then it's awfully tempting to put this all behind a KVM ioctl().

[1] https://lore.kernel.org/all/[email protected]
[2] https://lore.kernel.org/all/[email protected]
[3] https://lore.kernel.org/all/ZDiCG%[email protected]
[4] https://lore.kernel.org/all/Y%[email protected]
[5] https://lore.kernel.org/all/[email protected]

#347 lore raw

Re: Rename restrictedmem => guardedmem? (was: Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM)

Quoted 39 lines · Sean Christopherson
Quoted 22 lines · David Hildenbrand
Quoted 6 lines · Sean Christopherson

Hidden/Concealed/etc - Too close to secretmem, suffers the "hidden from whom" problem, and depending on the use case, the memory may not actually be concealed from the user that controls the VMM.

Restricted - "rmem" collides with "reserved memory" in code.

Quoted 8 lines

Guarded - Conflicts with s390's "guarded storage", has the "from whom" problem.

Inaccessible - Many of the same problems as "hidden".

Unmappable - Doesn't cover things like read/write, and is wrong in the sense that the memory is still mappable, just not via mmap().

Secured - I'm not getting anywhere near this one :-)

The think about "secretmem" that I kind-of like (a little) is that it explains what it's used for: storing secrets. We don't call it "unmapped" memory because we unmap it from the directmap or "unpinnable" memory or "inaccessible" memory ... or even "restricted" because it has restrictions ... how the secrets are protected is kind of an implementation detail.

Quoted 6 lines

So instead of describing *why*/*how* restrictedmem is the weird kid (restricted/guarded/hidden/restricted/inaccessible/ ...), maybe rather describe what it's used for?

I know, I know, "there are other use cases where it will be used outside of VM context". I really don't care.

Heh, we originally proposed F_SEAL_GUEST, but that was also sub-optimal[1] ;-)

Quoted 6 lines

"memfd_vm" / "vm_mem" would be sooo (feel free to add some more o's here) much easier to get. It's a special fd to be used to back VM memory. Depending on the VM type (encrypted/protected/whatever), restrictions might apply (not able to mmap, not able to read/write ...). For example, there really is no need to disallow mmap/read/write when using that memory to back a simple VM where all we want to do is avoid user-space page tables.

Quoted 2 lines

In seriousness, I do agree with Jason's very explicit objection[2] against naming a non-KVM uAPI "guest", or any variation thereof.

While I agree, it's all better than the naming we use right now ...

Let me throw "tee_mem" / "memfd_tee" into the picture. That could eventually catch what we want to have.

Or "coco_mem" / "memfd_coco".

Of course, both expect that people know the terminology (just like what "vm" stands for), but it's IMHO significantly better than restricted/guarded/opaque/whatsoever.

Again, expresses what it's used for, not why it behaves in weird ways.

Quoted 4 lines

An alternative that we haven't considered since the very early days is making the uAPI a KVM ioctl() instead of a memfd() flag or dedicated syscall. Looking at the code for "pure shim" implementation[3], that's actually not that crazy of an idea.

Yes.

Quoted 4 lines

I don't know that I love the idea of burying this in KVM, but there are benefits to coupling restrictedmem to KVM (aside from getting out from behind this bikeshed that I created).

Yes, it's all better than jumping through hoops to come up with a bad name like "restrictedmem".

Quoted 8 lines

The big benefit is that the layer of indirection goes away. That simplifies things like enhancing restrictedmem to allow host userspace access for debug purposes, batching TLB flushes if a PUNCH_HOLE spans multiple memslots, enforcing exclusive access, likely the whole "share with a device" story if/when we get there, etc.

The obvious downsides are that (a) maintenance falls under the KVM umbrella, but that's likely to be true in practice regardless of where the code lands, and

Yes.

Quoted 2 lines

(b) if another use case comes along, e.g. the Gunyah hypervisor[4][5], we risk someone reinventing a similar solution.

I agree. But if it's as simple as providing an ioctl for that hypervisor that simply wires up the existing implementation, it's not too bad.

Quoted 5 lines

If we can get Gunyah on board and they don't need substantial changes to the restrictedmem implementation, then I'm all for continuing on the path we're on. But if Gunyah wants to do their own thing, and the lightweight shim approach is viable, then it's awfully tempting to put this all behind a KVM ioctl().

Right. Or we still succeed in finding a name that's not as bad as what we had so far.

Signature · 3 lines
Thanks,

David / dhildenb
#348 lore raw

Re: Rename restrictedmem => guardedmem? (was: Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM)

Quoted 18 lines · David Hildenbrand
Quoted 10 lines · Sean Christopherson
Quoted 6 lines · David Hildenbrand

"memfd_vm" / "vm_mem" would be sooo (feel free to add some more o's here) much easier to get. It's a special fd to be used to back VM memory. Depending on the VM type (encrypted/protected/whatever), restrictions might apply (not able to mmap, not able to read/write ...). For example, there really is no need to disallow mmap/read/write when using that memory to back a simple VM where all we want to do is avoid user-space page tables.

In seriousness, I do agree with Jason's very explicit objection[2] against naming a non-KVM uAPI "guest", or any variation thereof.

While I agree, it's all better than the naming we use right now ...

Let me throw "tee_mem" / "memfd_tee" into the picture. That could eventually catch what we want to have.

Quoted 7 lines

Or "coco_mem" / "memfd_coco".

Of course, both expect that people know the terminology (just like what "vm" stands for), but it's IMHO significantly better than restricted/guarded/opaque/whatsoever.

Again, expresses what it's used for, not why it behaves in weird ways.

I don't want to explicitly tie this to trusted execution or confidential compute, as there is value in backing "normal" guests with memory that cannot be accessed by the host userspace without jumping through a few extra hoops, e.g. to add a layer of protection against data corruption due to host userspace bugs.

Quoted 5 lines
Quoted 2 lines

(b) if another use case comes along, e.g. the Gunyah hypervisor[4][5], we risk someone reinventing a similar solution.

I agree. But if it's as simple as providing an ioctl for that hypervisor that simply wires up the existing implementation, it's not too bad.

Yeah, my mind was wandering in this direction too. The absolute worst case scenario seems to be that we do end up creating a generic syscall that is a superset of KVM's functionality, in which case KVM would end up with an ioctl() that is just a redirect/wrapper.

#349 lore raw

Re: Rename restrictedmem => guardedmem? (was: Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM)

Quoted 25 lines · Sean Christopherson
Quoted 24 lines · David Hildenbrand
Quoted 10 lines · Sean Christopherson
Quoted 6 lines · David Hildenbrand

"memfd_vm" / "vm_mem" would be sooo (feel free to add some more o's here) much easier to get. It's a special fd to be used to back VM memory. Depending on the VM type (encrypted/protected/whatever), restrictions might apply (not able to mmap, not able to read/write ...). For example, there really is no need to disallow mmap/read/write when using that memory to back a simple VM where all we want to do is avoid user-space page tables.

In seriousness, I do agree with Jason's very explicit objection[2] against naming a non-KVM uAPI "guest", or any variation thereof.

While I agree, it's all better than the naming we use right now ...

Let me throw "tee_mem" / "memfd_tee" into the picture. That could eventually catch what we want to have.

Or "coco_mem" / "memfd_coco".

Of course, both expect that people know the terminology (just like what "vm" stands for), but it's IMHO significantly better than restricted/guarded/opaque/whatsoever.

Quoted 6 lines
Quoted 1 line

Again, expresses what it's used for, not why it behaves in weird ways.

I don't want to explicitly tie this to trusted execution or confidential compute, as there is value in backing "normal" guests with memory that cannot be accessed by the host userspace without jumping through a few extra hoops, e.g. to add a layer of protection against data corruption due to host userspace bugs.

Nothing speaks against using tee_mem for the same purpose I guess. I like the sound of it after all. :)

Signature · 3 lines
Thanks,

David / dhildenb
#350 lore raw

Re: Rename restrictedmem => guardedmem? (was: Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM)

In reply to David Hildenbrand

+Christian and Hugh

Quoted 33 lines · David Hildenbrand
Quoted 3 lines · Sean Christopherson

An alternative that we haven't considered since the very early days is making the uAPI a KVM ioctl() instead of a memfd() flag or dedicated syscall. Looking at the code for "pure shim" implementation[3], that's actually not that crazy of an idea.

Yes.

Quoted 4 lines

I don't know that I love the idea of burying this in KVM, but there are benefits to coupling restrictedmem to KVM (aside from getting out from behind this bikeshed that I created).

Yes, it's all better than jumping through hoops to come up with a bad name like "restrictedmem".

Quoted 8 lines

The big benefit is that the layer of indirection goes away. That simplifies things like enhancing restrictedmem to allow host userspace access for debug purposes, batching TLB flushes if a PUNCH_HOLE spans multiple memslots, enforcing exclusive access, likely the whole "share with a device" story if/when we get there, etc.

The obvious downsides are that (a) maintenance falls under the KVM umbrella, but that's likely to be true in practice regardless of where the code lands, and

Yes.

Quoted 2 lines

(b) if another use case comes along, e.g. the Gunyah hypervisor[4][5], we risk someone reinventing a similar solution.

I agree. But if it's as simple as providing an ioctl for that hypervisor that simply wires up the existing implementation, it's not too bad.

Quoted 1 line
Quoted 7 lines
Quoted 4 lines

If we can get Gunyah on board and they don't need substantial changes to the restrictedmem implementation, then I'm all for continuing on the path we're on. But if Gunyah wants to do their own thing, and the lightweight shim approach is viable, then it's awfully tempting to put this all behind a KVM ioctl().

Right. Or we still succeed in finding a name that's not as bad as what we had so far.

Okie dokie. So as not to bury the lede:

I think we should provide a KVM ioctl() and have KVM provide its own low-level implementation, i.e. not wrap shmem. As much as I don't want to have KVM serving up memory to userspace, by trying to keep KVM out of the memory management business, we're actually making things *more* complex and harder to maintain (and merge).

Hugh said something to this effect quite early on[1], it just unfortunately took me a long time to understand the benefits.

: QEMU gets fd from opening /dev/kvm_something, uses ioctls (or perhaps : the fallocate syscall interface itself) to allocate and free the memory, : ioctl for initializing some of it too. KVM in control of whether that : fd can be read or written or mmap'ed or whatever, no need to prevent it : in shmem.c, no need for flags, seals, notifications to and fro because : KVM is already in control and knows the history. If shmem actually has : value, call into it underneath - somewhat like SysV SHM, and /dev/zero : mmap, and i915/gem make use of it underneath. If shmem has nothing to : add, just allocate and free kernel memory directly, recorded in your : own xarray.

Christian also suggested that we stop trying to be lazy and create a proper API[2].

: I find this hard to navigate tbh and it feels like taking a shortcut to : avoid building a proper api. If you only care about a specific set of : operations specific to memfd restricte that needs to be available to : in-kernel consumers, I wonder if you shouldn't just go one step further : then your proposal below and build a dedicated minimal ops api. Idk, : sketching like a madman on a drawning board here with no claim to : feasibility from a mm perspective whatsoever

The key point from Hugh is that, except for a few minor things that are trivially easy to replicate, the things that make shmem "shmem" don't provide any value for the KVM use case:

- We have no plans to support swap, and migration support is dubious at best. Swap in particular brings in a lot of complexity for no benefit (to us). That complexity doesn't mean depending on shmem is inherently bad, but it does mean rolling our own implementation is highly unlikely to result in reinventing shmem's wheel.

- Sharing a backing file between arbitrary process is _unwanted_ for the initial use cases. There may come a time when mutually trusted VMs can share "private" data, but (a) that's a distant future problem and (b) it'll likely require even more KVM control over the memory.

- All of the interfaces for read/write/mmap/etc. are dead weight from our perspective. Even worse, we have to actively avoid those interfaces. That can kinda sorta be done without polluting shmem code by using a shim, but that has problems of its own (see below).

And Christian pointed out several flaws with wrapping shmem:

- Implementing a partial overlay filesystem leads to inconsistencies because only some of the ops are changed, e.g. poking at the inode_operations or super_operations will show shmem stuff, whereas address_space_operations and file_operations will show restrictedmem stuff.

- Usurping just f_ops and a_ops without creating a full blown filesystem avoids the partial overlay issues, but partially overriding ops isn't really supported (because it's weird and brittle), e.g. blindly forwarding a fallocate() after restrictedmem does it's thing "works", but only because we very carefully and deliberately designed restrictedmem on top of shmem.

On top of the points raised by Hugh and Christian, wrapping shmem isn't really any simpler, just different. E.g. there's a very subtle bug in the shim variant:
by passing SGP_WRITE, shmem skips zeroing the page because restrictemem is telling shmem that it (restrictedmem) will immediately write the page. For TDX and SNP, that's a "feature" because in those cases trusted firmware will zero (or init) memory when it's assigned to the guest, but it's a nasty flaw for other use cases.

I'm not saying that we'll magically avoid such bugs by avoiding shmem, just pointing out that using shmem requires understanding exactly how shmem works, i.e. using shmem isn't necessarily any easier than building directly on filemap and/or folio APIs. And I gotta imagine it will be a similar story if/when we want to add hugetlbfs support. Building on filemap/folio will definitely have its own challenges, but after prototyping an implementation I can confidently say that none of the problems will be insurmountable. KVM has _way_ more complex code in its memslot and MMU implementations.

And another benefit of building on filemap and/or folio APIs is that, because we would be reinventing the wheel to some extent, when we do inevitablly run into problems, it will be easier to get help solving those problems because (a) we won't be doing weird things no one wants to deal with and (b) because the problems will likely be things others have already dealt with.

The other angle I've been looking at is whether or not having KVM provide its own implementation will lead to maintenance problems in the future, specifically if we get to the point where we want to support "fancy" things like swap and migration. For migration, I'm quite confident that a dedicated KVM ioctl() versus wrapping shmem would be at worst a wash, and more than likely simpler if KVM owns everything. E.g. migrating pages under TDX and SNP requires invoking magic instructions, and so we'd be overriding ->migrate_folio() no matter what.

As for swap, I think we should put a stake in the ground and say that KVM will never support swap for KVM's ioctl(). Sooo much of the infrastructure around swap/reclaim is tied to userspace mappings, e.g. knowing which pages are LRU and/or cold. I poked around a bit to see how we could avoid reinventing all of that infrastructure for fd-only memory, and the best idea I could come up with is basically a rehash of Kirill's very original "KVM protected memory" RFC[3], i.e. allow "mapping" fd-only memory, but ensure that memory is never actually present from hardware's perspective.

But on top of the various problems with that approach, the only use cases I can think of for using fd-only to back non-confidential VMs is to guard against spurious writes/reads to guest memory and/or avoid memory overhead for mapping guest memory into the user page tables. Avoiding memory overhead is completely defeated if the page tables are populated PROT_NONE, which just leaves the "harden against guest data corruption use case". And for that specific use case, _if_ swap is desirable, it would be far, far easier to teach the kernel to allow KVM to follow PROT_NONE (as Kirill's series did), as opposed to trying to teach the kernel and/or KVM how to swap fd-only memory.

In other words, fd-only memory is purely for slice-of-hardware VMs. If someone wants to overcommit VMs, then they use KVM's traditional API for mapping memory into the guest.

Regarding putting this in KVM, as mentioned (way) above in the quote, the worst case scenario of making this a KVM ioctl() seems to be that KVM will end up with an ioctl() that redirects to common kernel code. On the flip side, implementing this in KVM gives us a much clearer path to getting all of this merged. There will be a few small non-KVM patches, but they will either be trivial, e.g. exporting APIs (that aren't contentious) or not strictly required, e.g. adding a flag to mark an address space as completely unmigratable (for improved performance). I.e. the non-KVM patches will be small and be very actionable for their maintainers, and other than that we control our own destiny.

And on the topic of control, making this a KVM ioctl() and implementation gives KVM a _lot_ of control. E.g. we can make the file size immutable to simplify the implementation, bind the fd to a single VM at creation time, add KVM-defined flags for controlling hugepage behavior, etc.

Last but not least, I think forcing us KVM folks to get our hands at least a little dirty with MM and FS stuff would be a *good* thing. KVM has had more than a few bugs that we missed in no small part because most of the people that work on KVM have almost zero knowledge of MM and FS, and especially at the boundaries between those two.

Note, I implemented the POC on top of the filemap APIs because that was much faster and less error prone than re-implementing xarray management myself. We may or may not want to actually make the kernel at large aware of these allocations, i.e. it may be better to follow Hugh's suggestion and use the folio APIs directly instead of piggybacking filemap+folios. E.g. I _think_ migration becomes a complete non-issue if the pages are "raw" allocations and never get marked as LRU-friendly. What I'm not so sure about is if there is anything substantial that we'd lose out on by not reusing the filemap stuff.

The POC also doesn't play nice with Ackerley's patches to allocate files on a user-defined mount. AIUI, that's largely a non-issue as something like fbind() would provide a superset of that functionality and more than one maintainer has expressed (handwavy) support for a generic fbind().

Code is available here if folks want to take a look before any kind of formal posting:

https://github.com/sean-jc/linux.git x86/kvm_gmem_solo

[1] https://lore.kernel.org/all/[email protected]
[2] https://lore.kernel.org/all/20230418-anfallen-irdisch-6993a61be10b@brauner
[3] https://lore.kernel.org/linux-mm/[email protected]

#351 lore raw

Re: Rename restrictedmem => guardedmem? (was: Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM)

Hi Sean,

Thanks for implementing this POC!

I’ve started porting the selftests (both Chao’s and those I added [1]).

guest mem seems to cover the use cases that have been discussed and proposed so far, but I still need to figure out how gmem can work with

+ hugetlbfs
+ specification of/storing memory policy (for NUMA node bindings)
+ memory accounting - we may need to account for memory used separately, so that guest mem shows up separately on /proc/meminfo and similar places.

One issue I’ve found so far is that the pointer to kvm (gmem->kvm) is not cleaned up, and hence it is possible to crash the host kernel in the following way

1. Create a KVM VM
2. Create a guest mem fd on that VM
3. Create a memslot with the guest mem fd (hence binding the fd to the VM)
4. Close/destroy the KVM VM
5. Call fallocate(PUNCH_HOLE) on the guest mem fd, which uses gmem->kvm when it tries to do invalidation.

I then tried to clean up the gmem->kvm pointer during unbinding when the KVM VM is destroyed.

That works, but then I realized there’s a simpler way to use the pointer after freeing:

1. Create a KVM VM
2. Create a guest mem fd on that VM
3. Close/destroy the KVM VM
4. Call fallocate(PUNCH_HOLE) on the guest mem fd, which uses gmem->kvm when it tries to do invalidation.

Perhaps binding should mean setting the gmem->kvm pointer in addition to gmem->bindings. This makes binding and unbinding symmetric and avoids the use-after-frees described above.

This also means that creating a guest mem fd is no longer dependent on the VM. Perhaps we can make creating a gmem fd a system ioctl (like KVM_GET_API_VERSION and KVM_CREATE_VM) instead of a vm ioctl?

[1] https://lore.kernel.org/all/[email protected]/T/

Ackerley

#352 lore raw

Re: Rename restrictedmem => guardedmem? (was: Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM)

Quoted 40 lines · Ackerley Tng

Hi Sean,

Thanks for implementing this POC!

I’ve started porting the selftests (both Chao’s and those I added [1]).

guest mem seems to cover the use cases that have been discussed and proposed so far, but I still need to figure out how gmem can work with

+ hugetlbfs
+ specification of/storing memory policy (for NUMA node bindings)
+ memory accounting - we may need to account for memory used separately, so that guest mem shows up separately on /proc/meminfo and similar places.

One issue I’ve found so far is that the pointer to kvm (gmem->kvm) is not cleaned up, and hence it is possible to crash the host kernel in the following way

1. Create a KVM VM
2. Create a guest mem fd on that VM
3. Create a memslot with the guest mem fd (hence binding the fd to the VM)
4. Close/destroy the KVM VM
5. Call fallocate(PUNCH_HOLE) on the guest mem fd, which uses gmem->kvm when it tries to do invalidation.

I then tried to clean up the gmem->kvm pointer during unbinding when the KVM VM is destroyed.

That works, but then I realized there’s a simpler way to use the pointer after freeing:

1. Create a KVM VM
2. Create a guest mem fd on that VM
3. Close/destroy the KVM VM
4. Call fallocate(PUNCH_HOLE) on the guest mem fd, which uses gmem->kvm when it tries to do invalidation.

Quoted 3 lines

Perhaps binding should mean setting the gmem->kvm pointer in addition to gmem->bindings. This makes binding and unbinding symmetric and avoids the use-after-frees described above.

Hrm, that would work, though it's a bit convoluted, e.g. would require detecting when the last binding is being removed. A similar (also ugly) solution would be to nullify gmem->kvm when KVM dies.

I don't love either approach idea because it means a file created in the context of a VM can outlive the VM itself, and then userspace ends up with a file descriptor that it can't do anything with except close(). I doubt that matters in practice though, e.g. when the VM dies, all memory can be freed so that the file ends up being little more than a shell. And if we go that route, there's no need to grab a reference to the file during bind, KVM can just grab a longterm reference when the file is initially created and then drop it when KVM dies (and nullifies gmem->kvm).

Blech, another wart is that I believe gmem would need to do __module_get() during file creation to prevent kvm.ko from being unloaded after the last VM dies. Ah, but that'd also be true if we went with a system-scoped KVM ioctl(), so I suppose it's not _that_ ugly.

Exchanging references (at binding or at creation) doesn't work, because that creates a circular dependency, i.e. gmem and KVM would pin each other.

A "proper" refcounting approach, where the file pins KVM and not vice versa, gets nasty because of how KVM's memslots work. The least awful approach I can think of would be to delete the associated memslot(s) when the file is released, possibly via deferred work to avoid deadlock issues. Not the prettiest thing ever and in some ways that'd yield an even worse ABI.

Side topic, there's a second bug (and probably more lurking): kvm_swap_active_memslots()'s call to synchronize_srcu_expedited() is done _before_ the call to kvm_gmem_unbind(), i.e. doesn't wait for readers in kvm_gmem_invalidate_begin() to go away. The easy solution for that one is to add another synchronize_srcu_expedited() after unbinding.

Quoted 3 lines

This also means that creating a guest mem fd is no longer dependent on the VM. Perhaps we can make creating a gmem fd a system ioctl (like KVM_GET_API_VERSION and KVM_CREATE_VM) instead of a vm ioctl?

My preference is to make it a VM-scoped ioctl(), if it ends up being a KVM ioctl() and not a common syscall. If the file isn't tightly coupled to a single VM, then punching a hole is further complicated by needing to deal with invalidating multiple regions that are bound to different @kvm instances. It's not super complex, but AFAICT having the ioctl() be system-scoped doesn't add value, e.g. I don't think having one VM own the memory will complicate even if/when we get to the point where VMs can share "private" memory, and the gmem code would still need to deal with grabbing a module reference.

#353 lore raw

Re: Rename restrictedmem => guardedmem? (was: Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM)

Quoted 10 lines · Sean Christopherson

...
My preference is to make it a VM-scoped ioctl(), if it ends up being a KVM ioctl() and not a common syscall. If the file isn't tightly coupled to a single VM, then punching a hole is further complicated by needing to deal with invalidating multiple regions that are bound to different @kvm instances. It's not super complex, but AFAICT having the ioctl() be system-scoped doesn't add value, e.g. I don't think having one VM own the memory will complicate even if/when we get to the point where VMs can share "private" memory, and the gmem code would still need to deal with grabbing a module reference.

Copyless migration would be a scenario where "private" memory may need to be shared between source and target VMs depending on how migration support is implemented.

Regards,
Vishal