KVM: mm: fd-based approach for supporting KVM guest private memory
Subsystems (23)
Sean Christopherson
maintainerRe: [PATCH v10 3/9] KVM: Extend the memslot to support fd-based private memory
In reply to Chao PengQuoted 21 lines · Chao Peng
Quoted 20 lines · Sean Christopherson
Quoted 18 lines · Chao Peng
Quoted 10 lines · Jarkko Sakkinen
Quoted 3 lines · Chao Peng
To make future maintenance easy, internally use a binary compatible alias struct
kvm_user_mem_regionto handle both the normal and the '_ext' variants.Feels bit hacky IMHO, and more like a completely new feature than an extension.
Why not just add a new ioctl? The commit message does not address the most essential design here.
Yes, people can always choose to add a new ioctl for this kind of change and the balance point here is we want to also avoid 'too many ioctls' if the functionalities are similar. The '
_ext' variant reuses all the existing fields in the 'normal' variant and most importantly KVM internally can reuse most of the code. I certainly can add some words in the commit message to explain this design choice.
Quoted 6 lines
Quoted 4 lines
After seeing the userspace side of this, I agree with Jarkko; overloading
KVM_SET_USER_MEMORY_REGIONis a hack. E.g. the size validation ends up being bogus, and userspace ends up abusing unions or implementingkvm_user_mem_regionitself.How is the size validation being bogus? I don't quite follow.
The ioctl() magic embeds the size of the payload (struct kvm_userspace_memory_region in this case) in the ioctl() number, and that information is visible to userspace via _IOCTL_SIZE(). Attempting to take a larger size can mess up sanity checks, e.g. KVM selftests get tripped up on this assert if KVM_SET_USER_MEMORY_REGION is passed an "extended" struct.
#define kvm_do_ioctl(fd, cmd, arg) \
({ \
kvm_static_assert(!_IOC_SIZE(cmd) || sizeof(*arg) == _IOC_SIZE(cmd)); \
ioctl(fd, cmd, arg); \
})Quoted 2 lines
Then we will use
kvm_userspace_memory_region2as the KVM internal alias, right?
Yep.
Quoted 10 lines
I see similar examples use different functions to handle different versions but it does look easier if we use alias for this function.
Quoted 5 lines
It feels absolutely ridiculous, but I think the best option is to do:
#define
KVM_SET_USER_MEMORY_REGION2_IOW(KVMIO,0x49, \
structkvm_userspace_memory_region2)Just interesting, is
0x49a safe number we can use?
Yes? So long as its not used by KVM, it's safe. AFAICT, it's unused.
Re: [PATCH v10 3/9] KVM: Extend the memslot to support fd-based private memory
Quoted 35 lines · Sean Christopherson
Quoted 21 lines · Chao Peng
Quoted 20 lines · Sean Christopherson
Quoted 18 lines · Chao Peng
Quoted 10 lines · Jarkko Sakkinen
Quoted 3 lines · Chao Peng
To make future maintenance easy, internally use a binary compatible alias struct
kvm_user_mem_regionto handle both the normal and the '_ext' variants.Feels bit hacky IMHO, and more like a completely new feature than an extension.
Why not just add a new ioctl? The commit message does not address the most essential design here.
Yes, people can always choose to add a new ioctl for this kind of change and the balance point here is we want to also avoid 'too many ioctls' if the functionalities are similar. The '
_ext' variant reuses all the existing fields in the 'normal' variant and most importantly KVM internally can reuse most of the code. I certainly can add some words in the commit message to explain this design choice.Quoted 6 lines
Quoted 4 lines
After seeing the userspace side of this, I agree with Jarkko; overloading
KVM_SET_USER_MEMORY_REGIONis a hack. E.g. the size validation ends up being bogus, and userspace ends up abusing unions or implementingkvm_user_mem_regionitself.How is the size validation being bogus? I don't quite follow.
The
ioctl()magic embeds the size of the payload (structkvm_userspace_memory_regionin this case) in theioctl()number, and that information is visible to userspace via_IOCTL_SIZE(). Attempting to take a larger size can mess up sanity checks, e.g. KVM selftests get tripped up on this assert ifKVM_SET_USER_MEMORY_REGIONis passed an "extended" struct.
Quoted 5 lines
#define kvm_do_ioctl(fd, cmd, arg) \
({ \
kvm_static_assert(!_IOC_SIZE(cmd) || sizeof(*arg) == _IOC_SIZE(cmd)); \
ioctl(fd, cmd, arg); \
})Got it. Thanks for the explanation.
Chao
Re: [PATCH v10 3/9] KVM: Extend the memslot to support fd-based private memory
In reply to Chao PengQuoted 53 lines · Chao Peng
Quoted 20 lines · Sean Christopherson
Quoted 12 lines · Chao Peng
Quoted 10 lines · Jarkko Sakkinen
Quoted 3 lines · Chao Peng
To make future maintenance easy, internally use a binary compatible alias struct
kvm_user_mem_regionto handle both the normal and the '_ext' variants.Feels bit hacky IMHO, and more like a completely new feature than an extension.
Why not just add a new ioctl? The commit message does not address the most essential design here.
Quoted 6 lines
Yes, people can always choose to add a new ioctl for this kind of change and the balance point here is we want to also avoid 'too many ioctls' if the functionalities are similar. The '
_ext' variant reuses all the existing fields in the 'normal' variant and most importantly KVM internally can reuse most of the code. I certainly can add some words in the commit message to explain this design choice.Quoted 4 lines
After seeing the userspace side of this, I agree with Jarkko; overloading
KVM_SET_USER_MEMORY_REGIONis a hack. E.g. the size validation ends up being bogus, and userspace ends up abusing unions or implementingkvm_user_mem_regionitself.How is the size validation being bogus? I don't quite follow. Then we will use
kvm_userspace_memory_region2as the KVM internal alias, right? I see similar examples use different functions to handle different versions but it does look easier if we use alias for this function.Quoted 5 lines
It feels absolutely ridiculous, but I think the best option is to do:
#define
KVM_SET_USER_MEMORY_REGION2_IOW(KVMIO,0x49, \
structkvm_userspace_memory_region2)Just interesting, is
0x49a safe number we can use?Quoted 14 lines
/* for
KVM_SET_USER_MEMORY_REGION2*/struct kvm_user_mem_region2 { __u32 slot; __u32 flags; __u64 guest_phys_addr; __u64 memory_size; __u64 userspace_addr; __u64 restricted_offset; __u32 restricted_fd; __u32 pad1; __u64 pad2[14]; }
Quoted 7 lines
Quoted 1 line
And it's consistent with other KVM
ioctls(), e.g.KVM_SET_CPUID2.Okay, agree from KVM userspace API perspective this is more consistent with similar existing examples. I see several of them.
I think we will also need a
CAP_KVM_SET_USER_MEMORY_REGION2for this new ioctl.
The current API in the patch set is trivial for C user space but for any other more "constrained" language such as Rust a new ioctl would be easier to adapt.
Quoted 6 lines
Quoted 5 lines
Regarding the userspace side of things, please include Vishal's selftests in v11, it's impossible to properly review the uAPI changes without seeing the userspace side of things. I'm in the process of reviewing Vishal's v2[*], I'll try to massage it into a set of patches that you can incorporate into your series.
Quoted 7 lines
Previously I included Vishal's selftests in the github repo, but not include them in this patch series. It's OK for me to incorporate them directly into this series and review together if Vishal is fine.
Chao
Quoted 2 lines
BR, Jarkko
Re: [PATCH v10 3/9] KVM: Extend the memslot to support fd-based private memory
In reply to Sean ChristophersonQuoted 43 lines · Sean Christopherson
Quoted 12 lines · Chao Peng
Quoted 10 lines · Jarkko Sakkinen
Quoted 3 lines · Chao Peng
To make future maintenance easy, internally use a binary compatible alias struct
kvm_user_mem_regionto handle both the normal and the '_ext' variants.Feels bit hacky IMHO, and more like a completely new feature than an extension.
Why not just add a new ioctl? The commit message does not address the most essential design here.
Quoted 6 lines
Yes, people can always choose to add a new ioctl for this kind of change and the balance point here is we want to also avoid 'too many ioctls' if the functionalities are similar. The '
_ext' variant reuses all the existing fields in the 'normal' variant and most importantly KVM internally can reuse most of the code. I certainly can add some words in the commit message to explain this design choice.After seeing the userspace side of this, I agree with Jarkko; overloading
KVM_SET_USER_MEMORY_REGIONis a hack. E.g. the size validation ends up being bogus, and userspace ends up abusing unions or implementingkvm_user_mem_regionitself.It feels absolutely ridiculous, but I think the best option is to do:
#define
KVM_SET_USER_MEMORY_REGION2_IOW(KVMIO,0x49, \
structkvm_userspace_memory_region2)/* for
KVM_SET_USER_MEMORY_REGION2*/struct kvm_user_mem_region2 { __u32 slot; __u32 flags; __u64 guest_phys_addr; __u64 memory_size; __u64 userspace_addr; __u64 restricted_offset; __u32 restricted_fd; __u32 pad1; __u64 pad2[14]; }
Quoted 8 lines
And it's consistent with other KVM
ioctls(), e.g.KVM_SET_CPUID2.Regarding the userspace side of things, please include Vishal's selftests in v11, it's impossible to properly review the uAPI changes without seeing the userspace side of things. I'm in the process of reviewing Vishal's v2[*], I'll try to massage it into a set of patches that you can incorporate into your series.
+1
BR, Jarkko
PATCH v10 4/9 KVM: Add KVM_EXIT_MEMORY_FAULT exit
In reply to Chao PengThis new KVM exit allows userspace to handle memory-related errors. It indicates an error happens in KVM at guest memory range [gpa, gpa+size). The flags includes additional information for userspace to handle the error. Currently bit 0 is defined as 'private memory' where '1' indicates error happens due to private memory access and '0' indicates error happens due to shared memory access.
When private memory is enabled, this new exit will be used for KVM to exit to userspace for shared <-> private memory conversion in memory encryption usage. In such usage, typically there are two kind of memory conversions:
- explicit conversion: happens when guest explicitly calls into KVM to map a range (as private or shared), KVM then exits to userspace to perform the map/unmap operations.
- implicit conversion: happens in KVM page fault handler where KVM exits to userspace for an implicit conversion when the page is in a different state than requested (private or shared).
--- Documentation/virt/kvm/api.rst | 22 ++++++++++++++++++++++ include/uapi/linux/kvm.h | 8 ++++++++ 2 files changed, 30 insertions(+)
Documentation/virt/kvm/api.rst +22 -0
@@ -6634,6 +6634,28 @@ array field represents return values. The userspace should update the return values of SBI call before resuming the VCPU. For more details on RISC-V SBI spec refer, https://github.com/riscv/riscv-sbi-doc. +:: + + /* KVM_EXIT_MEMORY_FAULT */ + struct { + #define KVM_MEMORY_EXIT_FLAG_PRIVATE (1ULL << 0) + __u64 flags; + __u64 gpa; + __u64 size; + } memory; + +If exit reason is KVM_EXIT_MEMORY_FAULT then it indicates that the VCPU has +encountered a memory error which is not handled by KVM kernel module and +userspace may choose to handle it. The 'flags' field indicates the memory +properties of the exit. + + - KVM_MEMORY_EXIT_FLAG_PRIVATE - indicates the memory error is caused by + private memory access when the bit is set. Otherwise the memory error is + caused by shared memory access when the bit is clear. + +'gpa' and 'size' indicate the memory range the error occurs at. The userspace +may handle the error and return to KVM to retry the previous memory access. + :: /* KVM_EXIT_NOTIFY */
include/uapi/linux/kvm.h +8 -0
@@ -300,6 +300,7 @@ struct kvm_xen_exit {#define KVM_EXIT_RISCV_SBI 35#define KVM_EXIT_RISCV_CSR 36#define KVM_EXIT_NOTIFY 37 +#define KVM_EXIT_MEMORY_FAULT 38/* For KVM_EXIT_INTERNAL_ERROR */ /* Emulate instruction failed. */
@@ -541,6 +542,13 @@ struct kvm_run {#define KVM_NOTIFY_CONTEXT_INVALID (1 << 0)__u32 flags; } notify; +/* KVM_EXIT_MEMORY_FAULT */ +struct { +#define KVM_MEMORY_EXIT_FLAG_PRIVATE (1ULL << 0) +__u64 flags; +__u64 gpa; +__u64 size; + } memory;/* Fix the size of the union. */ char padding[256 ]; };
Signature · 1 line
2.25.1
Re: [PATCH v10 4/9] KVM: Add KVM_EXIT_MEMORY_FAULT exit
Hi,
Quoted 34 lines · Chao Peng
This new KVM exit allows userspace to handle memory-related errors. It indicates an error happens in KVM at guest memory range [gpa, gpa+size). The flags includes additional information for userspace to handle the error. Currently bit 0 is defined as 'private memory' where '1' indicates error happens due to private memory access and '0' indicates error happens due to shared memory access.
When private memory is enabled, this new exit will be used for KVM to exit to userspace for shared <-> private memory conversion in memory encryption usage. In such usage, typically there are two kind of memory conversions:
- explicit conversion: happens when guest explicitly calls into KVM to map a range (as private or shared), KVM then exits to userspace to perform the map/unmap operations.
- implicit conversion: happens in KVM page fault handler where KVM exits to userspace for an implicit conversion when the page is in a different state than requested (private or shared).Suggested-by Sean Christopherson <[email protected]>Co-developed-by Yu Zhang <[email protected]>Signed-off-by Yu Zhang <[email protected]>Signed-off-by Chao Peng <[email protected]>Reviewed-by Fuad Tabba <[email protected]>--- Documentation/virt/kvm/api.rst | 22 ++++++++++++++++++++++ include/uapi/linux/kvm.h | 8 ++++++++ 2 files changed, 30 insertions(+)Documentation/virt/kvm/api.rst +0
-0
Documentation/virt/kvm/api.rst array field represents return values. The userspace should update the return · line 6642
Quoted 9 lines
Documentation/virt/kvm/api.rst +6
-0@@ -6634,6 +6634,28 @@ array field represents return values. The userspace should update the return values of SBI call before resuming the VCPU. For more details on RISC-V SBI spec refer, https://github.com/riscv/riscv-sbi-doc. +:: + + /* KVM_EXIT_MEMORY_FAULT */ + struct { + #define KVM_MEMORY_EXIT_FLAG_PRIVATE (1ULL << 0) + __u64 flags;
I see you've removed the padding and increased the flag size.
Cheers,
/fuad
Quoted 48 lines
Documentation/virt/kvm/api.rst +16
-0+ __u64 gpa; + __u64 size; + } memory; + +If exit reason is KVM_EXIT_MEMORY_FAULT then it indicates that the VCPU has +encountered a memory error which is not handled by KVM kernel module and +userspace may choose to handle it. The 'flags' field indicates the memory +properties of the exit. + + - KVM_MEMORY_EXIT_FLAG_PRIVATE - indicates the memory error is caused by + private memory access when the bit is set. Otherwise the memory error is + caused by shared memory access when the bit is clear. + +'gpa' and 'size' indicate the memory range the error occurs at. The userspace +may handle the error and return to KVM to retry the previous memory access. + :: /* KVM_EXIT_NOTIFY */include/uapi/linux/kvm.h +8
-1@@ -300,6 +300,7 @@ struct kvm_xen_exit {#define KVM_EXIT_RISCV_SBI 35#define KVM_EXIT_RISCV_CSR 36#define KVM_EXIT_NOTIFY 37 +#define KVM_EXIT_MEMORY_FAULT 38/* For KVM_EXIT_INTERNAL_ERROR */ /* Emulate instruction failed. */ @@ -541,6 +542,13 @@ struct kvm_run {#define KVM_NOTIFY_CONTEXT_INVALID (1 << 0)__u32 flags; } notify; +/* KVM_EXIT_MEMORY_FAULT */ +struct { +#define KVM_MEMORY_EXIT_FLAG_PRIVATE (1ULL << 0) +__u64 flags; +__u64 gpa; +__u64 size; + } memory;/* Fix the size of the union. */ char padding[256 ]; }; --2.25.1
Re: [PATCH v10 4/9] KVM: Add KVM_EXIT_MEMORY_FAULT exit
Quoted 40 lines · Fuad Tabba
Hi,
Quoted 37 lines · Chao Peng
This new KVM exit allows userspace to handle memory-related errors. It indicates an error happens in KVM at guest memory range [gpa, gpa+size). The flags includes additional information for userspace to handle the error. Currently bit 0 is defined as 'private memory' where '1' indicates error happens due to private memory access and '0' indicates error happens due to shared memory access.
When private memory is enabled, this new exit will be used for KVM to exit to userspace for shared <-> private memory conversion in memory encryption usage. In such usage, typically there are two kind of memory conversions:
- explicit conversion: happens when guest explicitly calls into KVM to map a range (as private or shared), KVM then exits to userspace to perform the map/unmap operations.
- implicit conversion: happens in KVM page fault handler where KVM exits to userspace for an implicit conversion when the page is in a different state than requested (private or shared).Suggested-by Sean Christopherson <[email protected]>Co-developed-by Yu Zhang <[email protected]>Signed-off-by Yu Zhang <[email protected]>Signed-off-by Chao Peng <[email protected]>Reviewed-by Fuad Tabba <[email protected]>--- Documentation/virt/kvm/api.rst | 22 ++++++++++++++++++++++ include/uapi/linux/kvm.h | 8 ++++++++ 2 files changed, 30 insertions(+)Documentation/virt/kvm/api.rst +0
-0@@ -6634,6 +6634,28 @@ array field represents return values. The userspace should update the return values of SBI call before resuming the VCPU. For more details on RISC-V SBI spec refer, https://github.com/riscv/riscv-sbi-doc.
Quoted 8 lines
Documentation/virt/kvm/api.rst array field represents return values. The userspace should update the return · line 6642
Quoted 6 lines
Documentation/virt/kvm/api.rst +6
-0+:: + + /* KVM_EXIT_MEMORY_FAULT */ + struct { + #define KVM_MEMORY_EXIT_FLAG_PRIVATE (1ULL << 0) + __u64 flags;I see you've removed the padding and increased the flag size.
Yes Sean suggested this and also looks good to me.
Chao
Quoted 58 lines
Reviewed-by Fuad Tabba <[email protected]>Tested-by Fuad Tabba <[email protected]>Cheers,
/fuadQuoted 48 lines
Documentation/virt/kvm/api.rst +16
-0+ __u64 gpa; + __u64 size; + } memory; + +If exit reason is KVM_EXIT_MEMORY_FAULT then it indicates that the VCPU has +encountered a memory error which is not handled by KVM kernel module and +userspace may choose to handle it. The 'flags' field indicates the memory +properties of the exit. + + - KVM_MEMORY_EXIT_FLAG_PRIVATE - indicates the memory error is caused by + private memory access when the bit is set. Otherwise the memory error is + caused by shared memory access when the bit is clear. + +'gpa' and 'size' indicate the memory range the error occurs at. The userspace +may handle the error and return to KVM to retry the previous memory access. + :: /* KVM_EXIT_NOTIFY */include/uapi/linux/kvm.h +8
-1@@ -300,6 +300,7 @@ struct kvm_xen_exit {#define KVM_EXIT_RISCV_SBI 35#define KVM_EXIT_RISCV_CSR 36#define KVM_EXIT_NOTIFY 37 +#define KVM_EXIT_MEMORY_FAULT 38/* For KVM_EXIT_INTERNAL_ERROR */ /* Emulate instruction failed. */ @@ -541,6 +542,13 @@ struct kvm_run {#define KVM_NOTIFY_CONTEXT_INVALID (1 << 0)__u32 flags; } notify; +/* KVM_EXIT_MEMORY_FAULT */ +struct { +#define KVM_MEMORY_EXIT_FLAG_PRIVATE (1ULL << 0) +__u64 flags; +__u64 gpa; +__u64 size; + } memory;/* Fix the size of the union. */ char padding[256 ]; }; --2.25.1
Sean Christopherson
maintainerRe: [PATCH v10 4/9] KVM: Add KVM_EXIT_MEMORY_FAULT exit
In reply to Chao PengDocumentation/virt/kvm/api.rst array field represents return values. The userspace should update the return · line 6641
Quoted 13 lines · Chao Peng
Documentation/virt/kvm/api.rst +5
-0@@ -6634,6 +6634,28 @@ array field represents return values. The userspace should update the return values of SBI call before resuming the VCPU. For more details on RISC-V SBI spec refer, https://github.com/riscv/riscv-sbi-doc. +:: + + /* KVM_EXIT_MEMORY_FAULT */ + struct { + #define KVM_MEMORY_EXIT_FLAG_PRIVATE (1ULL << 0)
Unless there's a reason not to, we should use bit 3 to match the attributes.
PATCH v10 5/9 KVM: Use gfn instead of hva for mmu_notifier_retry
In reply to Chao PengCurrently in mmu_notifier invalidate path, hva range is recorded and then checked against by mmu_notifier_retry_hva() in the page fault handling path. However, for the to be introduced private memory, a page fault may not have a hva associated, checking gfn(gpa) makes more sense.
For existing hva based shared memory, gfn is expected to also work. The only downside is when aliasing multiple gfns to a single hva, the current algorithm of checking multiple ranges could result in a much larger range being rejected. Such aliasing should be uncommon, so the impact is expected small.
--- arch/x86/kvm/mmu/mmu.c | 8 +++++--- include/linux/kvm_host.h | 33 +++++++++++++++++++++------------ virt/kvm/kvm_main.c | 32 +++++++++++++++++++++++--------- 3 files changed, 49 insertions(+), 24 deletions(-)
arch/x86/kvm/mmu/mmu.c +5 -3
@@ -4259,7 +4259,7 @@ static bool is_page_fault_stale(struct kvm_vcpu *vcpu,return true;return fault->slot && -mmu_invalidate_retry_hva (vcpu->kvm, mmu_seq, fault->hva); +mmu_invalidate_retry_gfn (vcpu->kvm, mmu_seq, fault->gfn); }static int direct_page_fault (struct kvm_vcpu *vcpu,struct kvm_page_fault *fault)
@@ -6098,7 +6098,9 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end)write_lock (&kvm->mmu_lock); -kvm_mmu_invalidate_begin (kvm, gfn_start, gfn_end); +kvm_mmu_invalidate_begin (kvm); + +kvm_mmu_invalidate_range_add (kvm, gfn_start, gfn_end); flush =kvm_rmap_zap_gfn_range (kvm, gfn_start, gfn_end);
@@ -6112,7 +6114,7 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end)kvm_flush_remote_tlbs_with_address (kvm, gfn_start, gfn_end - gfn_start); -kvm_mmu_invalidate_end (kvm, gfn_start, gfn_end); +kvm_mmu_invalidate_end (kvm);write_unlock (&kvm->mmu_lock); }
include/linux/kvm_host.h +21 -12
@@ -787,8 +787,8 @@ struct kvm {struct mmu_notifier mmu_notifier;unsigned long mmu_invalidate_seq;long mmu_invalidate_in_progress; -unsigned long mmu_invalidate_range_start; -unsigned long mmu_invalidate_range_end; +gfn_t mmu_invalidate_range_start; +gfn_t mmu_invalidate_range_end;#endif struct list_head devices;u64 manual_dirty_log_protect;
@@ -1389,10 +1389,9 @@ void kvm_mmu_free_memory_cache(struct kvm_mmu_memory_cache *mc);void *kvm_mmu_memory_cache_alloc (struct kvm_mmu_memory_cache *mc);#endif -void kvm_mmu_invalidate_begin (struct kvm *kvm,unsigned long start, -unsigned long end); -void kvm_mmu_invalidate_end (struct kvm *kvm,unsigned long start, -unsigned long end); +void kvm_mmu_invalidate_begin (struct kvm *kvm); +void kvm_mmu_invalidate_range_add (struct kvm *kvm,gfn_t start,gfn_t end); +void kvm_mmu_invalidate_end (struct kvm *kvm);long kvm_arch_dev_ioctl (struct file *filp,unsigned int ioctl,unsigned long arg);
@@ -1963,9 +1962,9 @@ static inline int mmu_invalidate_retry(struct kvm *kvm, unsigned long mmu_seq)return 0 ; } -static inline int mmu_invalidate_retry_hva (struct kvm *kvm, +static inline int mmu_invalidate_retry_gfn (struct kvm *kvm,unsigned long mmu_seq, -unsigned long hva) +gfn_t gfn) {lockdep_assert_held (&kvm->mmu_lock);/*
@@ -1974,10 +1973,20 @@ static inline int mmu_invalidate_retry_hva(struct kvm *kvm,* that might be being invalidated. Note that it may include some false * positives, due to shortcuts when handing concurrent invalidations. */ -if (unlikely (kvm->mmu_invalidate_in_progress) && - hva >= kvm->mmu_invalidate_range_start && - hva < kvm->mmu_invalidate_range_end) -return 1 ; +if (unlikely (kvm->mmu_invalidate_in_progress)) { +/* +* Dropping mmu_lock after bumping mmu_invalidate_in_progress +* but before updating the range is a KVM bug. +*/ +if (WARN_ON_ONCE (kvm->mmu_invalidate_range_start ==INVALID_GPA || + kvm->mmu_invalidate_range_end ==INVALID_GPA )) +return 1 ; + +if (gfn >= kvm->mmu_invalidate_range_start && + gfn < kvm->mmu_invalidate_range_end) +return 1 ; + } +if (kvm->mmu_invalidate_seq != mmu_seq)return 1 ;return 0 ;
virt/kvm/kvm_main.c +23 -9
@@ -540,9 +540,7 @@ static void kvm_mmu_notifier_invalidate_range(struct mmu_notifier *mn,typedef bool (*hva_handler_t )(struct kvm *kvm,struct kvm_gfn_range *range); -typedef void (*on_lock_fn_t )(struct kvm *kvm,unsigned long start, -unsigned long end); - +typedef void (*on_lock_fn_t )(struct kvm *kvm);typedef void (*on_unlock_fn_t )(struct kvm *kvm);struct kvm_hva_range {
@@ -628,7 +626,8 @@ static __always_inline int __kvm_handle_hva_range(struct kvm *kvm, locked = true;KVM_MMU_LOCK (kvm);if (!IS_KVM_NULL_FN (range->on_lock)) - range->on_lock (kvm, range->start, range->end); + range->on_lock (kvm); +if (IS_KVM_NULL_FN (range->handler))break ; }
@@ -715,8 +714,7 @@ static void kvm_mmu_notifier_change_pte(struct mmu_notifier *mn,kvm_handle_hva_range (mn, address, address +1 , pte, kvm_set_spte_gfn); } -void kvm_mmu_invalidate_begin (struct kvm *kvm,unsigned long start, -unsigned long end) +void kvm_mmu_invalidate_begin (struct kvm *kvm) {/* * The count increase must become visible at unlock time as no
@@ -724,6 +722,17 @@ void kvm_mmu_invalidate_begin(struct kvm *kvm, unsigned long start,* count is also read inside the mmu_lock critical section. */ kvm->mmu_invalidate_in_progress++; + +if (likely (kvm->mmu_invalidate_in_progress ==1 )) { + kvm->mmu_invalidate_range_start =INVALID_GPA ; + kvm->mmu_invalidate_range_end =INVALID_GPA ; + } +} + +void kvm_mmu_invalidate_range_add (struct kvm *kvm,gfn_t start,gfn_t end) +{ +WARN_ON_ONCE (!kvm->mmu_invalidate_in_progress); +if (likely (kvm->mmu_invalidate_in_progress ==1 )) { kvm->mmu_invalidate_range_start = start; kvm->mmu_invalidate_range_end = end;
@@ -744,6 +753,12 @@ void kvm_mmu_invalidate_begin(struct kvm *kvm, unsigned long start, } } +static bool kvm_mmu_unmap_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range) +{ +kvm_mmu_invalidate_range_add (kvm, range->start, range->end); +return kvm_unmap_gfn_range (kvm, range); +} +static int kvm_mmu_notifier_invalidate_range_start (struct mmu_notifier *mn,const struct mmu_notifier_range *range) {
@@ -752,7 +767,7 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn, .start = range->start, .end = range->end, .pte = __pte(0), - .handler = kvm_unmap_gfn_range, + .handler = kvm_mmu_unmap_gfn_range, .on_lock = kvm_mmu_invalidate_begin, .on_unlock = kvm_arch_guest_memory_reclaimed, .flush_on_ret = true,
@@ -791,8 +806,7 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,return 0 ; } -void kvm_mmu_invalidate_end (struct kvm *kvm,unsigned long start, -unsigned long end) +void kvm_mmu_invalidate_end (struct kvm *kvm) {/* * This sequence increase will notify the kvm page fault that
Signature · 1 line
2.25.1
Re: [PATCH v10 5/9] KVM: Use gfn instead of hva for mmu_notifier_retry
Hi Chao,
Quoted 95 lines · Chao Peng
Currently in
mmu_notifierinvalidate path, hva range is recorded and then checked against bymmu_notifier_retry_hva()in the page fault handling path. However, for the to be introduced private memory, a page fault may not have a hva associated, checking gfn(gpa) makes more sense.For existing hva based shared memory, gfn is expected to also work. The only downside is when aliasing multiple gfns to a single hva, the current algorithm of checking multiple ranges could result in a much larger range being rejected. Such aliasing should be uncommon, so the impact is expected small.
Suggested-by Sean Christopherson <[email protected]>Signed-off-by Chao Peng <[email protected]>--- arch/x86/kvm/mmu/mmu.c | 8 +++++--- include/linux/kvm_host.h | 33 +++++++++++++++++++++------------ virt/kvm/kvm_main.c | 32 +++++++++++++++++++++++--------- 3 files changed, 49 insertions(+), 24 deletions(-)arch/x86/kvm/mmu/mmu.c +5
-3@@ -4259,7 +4259,7 @@ static bool is_page_fault_stale(struct kvm_vcpu *vcpu,return true;return fault->slot && -mmu_invalidate_retry_hva (vcpu->kvm, mmu_seq, fault->hva); +mmu_invalidate_retry_gfn (vcpu->kvm, mmu_seq, fault->gfn); }static int direct_page_fault (struct kvm_vcpu *vcpu,struct kvm_page_fault *fault)@@ -6098,7 +6098,9 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end)write_lock (&kvm->mmu_lock); -kvm_mmu_invalidate_begin (kvm, gfn_start, gfn_end); +kvm_mmu_invalidate_begin (kvm); + +kvm_mmu_invalidate_range_add (kvm, gfn_start, gfn_end); flush =kvm_rmap_zap_gfn_range (kvm, gfn_start, gfn_end);@@ -6112,7 +6114,7 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end)kvm_flush_remote_tlbs_with_address (kvm, gfn_start, gfn_end - gfn_start); -kvm_mmu_invalidate_end (kvm, gfn_start, gfn_end); +kvm_mmu_invalidate_end (kvm);write_unlock (&kvm->mmu_lock); }include/linux/kvm_host.h +7
-8@@ -787,8 +787,8 @@ struct kvm {struct mmu_notifier mmu_notifier;unsigned long mmu_invalidate_seq;long mmu_invalidate_in_progress; -unsigned long mmu_invalidate_range_start; -unsigned long mmu_invalidate_range_end; +gfn_t mmu_invalidate_range_start; +gfn_t mmu_invalidate_range_end;#endif struct list_head devices;u64 manual_dirty_log_protect;@@ -1389,10 +1389,9 @@ void kvm_mmu_free_memory_cache(struct kvm_mmu_memory_cache *mc);void *kvm_mmu_memory_cache_alloc (struct kvm_mmu_memory_cache *mc);#endif -void kvm_mmu_invalidate_begin (struct kvm *kvm,unsigned long start, -unsigned long end); -void kvm_mmu_invalidate_end (struct kvm *kvm,unsigned long start, -unsigned long end); +void kvm_mmu_invalidate_begin (struct kvm *kvm); +void kvm_mmu_invalidate_range_add (struct kvm *kvm,gfn_t start,gfn_t end); +void kvm_mmu_invalidate_end (struct kvm *kvm);long kvm_arch_dev_ioctl (struct file *filp,unsigned int ioctl,unsigned long arg);@@ -1963,9 +1962,9 @@ static inline int mmu_invalidate_retry(struct kvm *kvm, unsigned long mmu_seq)return 0 ; } -static inline int mmu_invalidate_retry_hva (struct kvm *kvm, +static inline int mmu_invalidate_retry_gfn (struct kvm *kvm,unsigned long mmu_seq, -unsigned long hva) +gfn_t gfn) {lockdep_assert_held (&kvm->mmu_lock);/*
include/linux/kvm_host.h static inline int mmu_invalidate_retry_hva(struct kvm *kvm, · line 1973
Quoted 1 line
include/linux/kvm_host.h +0
-0@@ -1974,10 +1973,20 @@ static inline int mmu_invalidate_retry_hva(struct kvm *kvm,* that might be being invalidated. Note that it may include some false
nit: "might be" (or) "is being"
include/linux/kvm_host.h static inline int mmu_invalidate_retry_hva(struct kvm *kvm, · line 1974
Quoted 1 line
include/linux/kvm_host.h +0
-0* positives, due to shortcuts when handing concurrent invalidations.
nit: handling
include/linux/kvm_host.h static inline int mmu_invalidate_retry_hva(struct kvm *kvm, · line 1982
Quoted 12 lines
include/linux/kvm_host.h +7
-4*/ -if (unlikely (kvm->mmu_invalidate_in_progress) && - hva >= kvm->mmu_invalidate_range_start && - hva < kvm->mmu_invalidate_range_end) -return 1 ; +if (unlikely (kvm->mmu_invalidate_in_progress)) { +/* +* Dropping mmu_lock after bumping mmu_invalidate_in_progress +* but before updating the range is a KVM bug. +*/ +if (WARN_ON_ONCE (kvm->mmu_invalidate_range_start ==INVALID_GPA || + kvm->mmu_invalidate_range_end ==INVALID_GPA ))
INVALID_GPA is an x86-specific define in arch/x86/include/asm/kvm_host.h, so this doesn't build on other architectures. The obvious fix is to move it to include/linux/kvm_host.h.
Cheers,
/fuad
Quoted 98 lines
include/linux/kvm_host.h +7
-0+return 1 ; + +if (gfn >= kvm->mmu_invalidate_range_start && + gfn < kvm->mmu_invalidate_range_end) +return 1 ; + } +if (kvm->mmu_invalidate_seq != mmu_seq)return 1 ;return 0 ;virt/kvm/kvm_main.c +23
-10@@ -540,9 +540,7 @@ static void kvm_mmu_notifier_invalidate_range(struct mmu_notifier *mn,typedef bool (*hva_handler_t )(struct kvm *kvm,struct kvm_gfn_range *range); -typedef void (*on_lock_fn_t )(struct kvm *kvm,unsigned long start, -unsigned long end); - +typedef void (*on_lock_fn_t )(struct kvm *kvm);typedef void (*on_unlock_fn_t )(struct kvm *kvm);struct kvm_hva_range {@@ -628,7 +626,8 @@ static __always_inline int __kvm_handle_hva_range(struct kvm *kvm, locked = true;KVM_MMU_LOCK (kvm);if (!IS_KVM_NULL_FN (range->on_lock)) - range->on_lock (kvm, range->start, range->end); + range->on_lock (kvm); +if (IS_KVM_NULL_FN (range->handler))break ; }@@ -715,8 +714,7 @@ static void kvm_mmu_notifier_change_pte(struct mmu_notifier *mn,kvm_handle_hva_range (mn, address, address +1 , pte, kvm_set_spte_gfn); } -void kvm_mmu_invalidate_begin (struct kvm *kvm,unsigned long start, -unsigned long end) +void kvm_mmu_invalidate_begin (struct kvm *kvm) {/* * The count increase must become visible at unlock time as no @@ -724,6 +722,17 @@ void kvm_mmu_invalidate_begin(struct kvm *kvm, unsigned long start,* count is also read inside the mmu_lock critical section. */ kvm->mmu_invalidate_in_progress++; + +if (likely (kvm->mmu_invalidate_in_progress ==1 )) { + kvm->mmu_invalidate_range_start =INVALID_GPA ; + kvm->mmu_invalidate_range_end =INVALID_GPA ; + } +} + +void kvm_mmu_invalidate_range_add (struct kvm *kvm,gfn_t start,gfn_t end) +{ +WARN_ON_ONCE (!kvm->mmu_invalidate_in_progress); +if (likely (kvm->mmu_invalidate_in_progress ==1 )) { kvm->mmu_invalidate_range_start = start; kvm->mmu_invalidate_range_end = end;@@ -744,6 +753,12 @@ void kvm_mmu_invalidate_begin(struct kvm *kvm, unsigned long start, } } +static bool kvm_mmu_unmap_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range) +{ +kvm_mmu_invalidate_range_add (kvm, range->start, range->end); +return kvm_unmap_gfn_range (kvm, range); +} +static int kvm_mmu_notifier_invalidate_range_start (struct mmu_notifier *mn,const struct mmu_notifier_range *range) {@@ -752,7 +767,7 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn, .start = range->start, .end = range->end, .pte = __pte(0), - .handler = kvm_unmap_gfn_range, + .handler = kvm_mmu_unmap_gfn_range, .on_lock = kvm_mmu_invalidate_begin, .on_unlock = kvm_arch_guest_memory_reclaimed, .flush_on_ret = true,@@ -791,8 +806,7 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,return 0 ; } -void kvm_mmu_invalidate_end (struct kvm *kvm,unsigned long start, -unsigned long end) +void kvm_mmu_invalidate_end (struct kvm *kvm) {/* * This sequence increase will notify the kvm page fault that -- 2.25.1
Re: [PATCH v10 5/9] KVM: Use gfn instead of hva for mmu_notifier_retry
Quoted 102 lines · Fuad Tabba
Hi Chao,
Quoted 95 lines · Chao Peng
Currently in
mmu_notifierinvalidate path, hva range is recorded and then checked against bymmu_notifier_retry_hva()in the page fault handling path. However, for the to be introduced private memory, a page fault may not have a hva associated, checking gfn(gpa) makes more sense.For existing hva based shared memory, gfn is expected to also work. The only downside is when aliasing multiple gfns to a single hva, the current algorithm of checking multiple ranges could result in a much larger range being rejected. Such aliasing should be uncommon, so the impact is expected small.
Suggested-by Sean Christopherson <[email protected]>Signed-off-by Chao Peng <[email protected]>--- arch/x86/kvm/mmu/mmu.c | 8 +++++--- include/linux/kvm_host.h | 33 +++++++++++++++++++++------------ virt/kvm/kvm_main.c | 32 +++++++++++++++++++++++--------- 3 files changed, 49 insertions(+), 24 deletions(-)arch/x86/kvm/mmu/mmu.c +5
-3@@ -4259,7 +4259,7 @@ static bool is_page_fault_stale(struct kvm_vcpu *vcpu,return true;return fault->slot && -mmu_invalidate_retry_hva (vcpu->kvm, mmu_seq, fault->hva); +mmu_invalidate_retry_gfn (vcpu->kvm, mmu_seq, fault->gfn); }static int direct_page_fault (struct kvm_vcpu *vcpu,struct kvm_page_fault *fault)@@ -6098,7 +6098,9 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end)write_lock (&kvm->mmu_lock); -kvm_mmu_invalidate_begin (kvm, gfn_start, gfn_end); +kvm_mmu_invalidate_begin (kvm); + +kvm_mmu_invalidate_range_add (kvm, gfn_start, gfn_end); flush =kvm_rmap_zap_gfn_range (kvm, gfn_start, gfn_end);@@ -6112,7 +6114,7 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end)kvm_flush_remote_tlbs_with_address (kvm, gfn_start, gfn_end - gfn_start); -kvm_mmu_invalidate_end (kvm, gfn_start, gfn_end); +kvm_mmu_invalidate_end (kvm);write_unlock (&kvm->mmu_lock); }include/linux/kvm_host.h +7
-8@@ -787,8 +787,8 @@ struct kvm {struct mmu_notifier mmu_notifier;unsigned long mmu_invalidate_seq;long mmu_invalidate_in_progress; -unsigned long mmu_invalidate_range_start; -unsigned long mmu_invalidate_range_end; +gfn_t mmu_invalidate_range_start; +gfn_t mmu_invalidate_range_end;#endif struct list_head devices;u64 manual_dirty_log_protect;@@ -1389,10 +1389,9 @@ void kvm_mmu_free_memory_cache(struct kvm_mmu_memory_cache *mc);void *kvm_mmu_memory_cache_alloc (struct kvm_mmu_memory_cache *mc);#endif -void kvm_mmu_invalidate_begin (struct kvm *kvm,unsigned long start, -unsigned long end); -void kvm_mmu_invalidate_end (struct kvm *kvm,unsigned long start, -unsigned long end); +void kvm_mmu_invalidate_begin (struct kvm *kvm); +void kvm_mmu_invalidate_range_add (struct kvm *kvm,gfn_t start,gfn_t end); +void kvm_mmu_invalidate_end (struct kvm *kvm);long kvm_arch_dev_ioctl (struct file *filp,unsigned int ioctl,unsigned long arg);@@ -1963,9 +1962,9 @@ static inline int mmu_invalidate_retry(struct kvm *kvm, unsigned long mmu_seq)return 0 ; } -static inline int mmu_invalidate_retry_hva (struct kvm *kvm, +static inline int mmu_invalidate_retry_gfn (struct kvm *kvm,unsigned long mmu_seq, -unsigned long hva) +gfn_t gfn) {lockdep_assert_held (&kvm->mmu_lock);/* include/linux/kvm_host.h static inline int mmu_invalidate_retry_hva(struct kvm *kvm, · line 1973
Quoted 1 line
include/linux/kvm_host.h +0
-0@@ -1974,10 +1973,20 @@ static inline int mmu_invalidate_retry_hva(struct kvm *kvm,* that might be being invalidated. Note that it may include some false nit: "might be" (or) "is being"
Quoted 3 lines
include/linux/kvm_host.h static inline int mmu_invalidate_retry_hva(struct kvm *kvm, · line 1974
Quoted 1 line
include/linux/kvm_host.h +0
-0* positives, due to shortcuts when handing concurrent invalidations. nit: handling
Both are existing code, but I can fix it either.
Quoted 14 lines
include/linux/kvm_host.h static inline int mmu_invalidate_retry_hva(struct kvm *kvm, · line 1982
Quoted 13 lines
include/linux/kvm_host.h +7
-4*/ -if (unlikely (kvm->mmu_invalidate_in_progress) && - hva >= kvm->mmu_invalidate_range_start && - hva < kvm->mmu_invalidate_range_end) -return 1 ; +if (unlikely (kvm->mmu_invalidate_in_progress)) { +/* +* Dropping mmu_lock after bumping mmu_invalidate_in_progress +* but before updating the range is a KVM bug. +*/ +if (WARN_ON_ONCE (kvm->mmu_invalidate_range_start ==INVALID_GPA || + kvm->mmu_invalidate_range_end ==INVALID_GPA ))
Quoted 4 lines
INVALID_GPAis an x86-specific define inarch/x86/include/asm/kvm_host.h, so this doesn't build on other architectures. The obvious fix is to move it toinclude/linux/kvm_host.h.
Hmm, INVALID_GPA is defined as ZERO for x86, not 100% confident this is correct choice for other architectures, but after search it has not been used for other architectures, so should be safe to make it common.
Thanks,
Chao
Quoted 102 lines
Cheers,
/fuadQuoted 98 lines
include/linux/kvm_host.h +7
-0+return 1 ; + +if (gfn >= kvm->mmu_invalidate_range_start && + gfn < kvm->mmu_invalidate_range_end) +return 1 ; + } +if (kvm->mmu_invalidate_seq != mmu_seq)return 1 ;return 0 ;virt/kvm/kvm_main.c +23
-10@@ -540,9 +540,7 @@ static void kvm_mmu_notifier_invalidate_range(struct mmu_notifier *mn,typedef bool (*hva_handler_t )(struct kvm *kvm,struct kvm_gfn_range *range); -typedef void (*on_lock_fn_t )(struct kvm *kvm,unsigned long start, -unsigned long end); - +typedef void (*on_lock_fn_t )(struct kvm *kvm);typedef void (*on_unlock_fn_t )(struct kvm *kvm);struct kvm_hva_range {@@ -628,7 +626,8 @@ static __always_inline int __kvm_handle_hva_range(struct kvm *kvm, locked = true;KVM_MMU_LOCK (kvm);if (!IS_KVM_NULL_FN (range->on_lock)) - range->on_lock (kvm, range->start, range->end); + range->on_lock (kvm); +if (IS_KVM_NULL_FN (range->handler))break ; }@@ -715,8 +714,7 @@ static void kvm_mmu_notifier_change_pte(struct mmu_notifier *mn,kvm_handle_hva_range (mn, address, address +1 , pte, kvm_set_spte_gfn); } -void kvm_mmu_invalidate_begin (struct kvm *kvm,unsigned long start, -unsigned long end) +void kvm_mmu_invalidate_begin (struct kvm *kvm) {/* * The count increase must become visible at unlock time as no @@ -724,6 +722,17 @@ void kvm_mmu_invalidate_begin(struct kvm *kvm, unsigned long start,* count is also read inside the mmu_lock critical section. */ kvm->mmu_invalidate_in_progress++; + +if (likely (kvm->mmu_invalidate_in_progress ==1 )) { + kvm->mmu_invalidate_range_start =INVALID_GPA ; + kvm->mmu_invalidate_range_end =INVALID_GPA ; + } +} + +void kvm_mmu_invalidate_range_add (struct kvm *kvm,gfn_t start,gfn_t end) +{ +WARN_ON_ONCE (!kvm->mmu_invalidate_in_progress); +if (likely (kvm->mmu_invalidate_in_progress ==1 )) { kvm->mmu_invalidate_range_start = start; kvm->mmu_invalidate_range_end = end;@@ -744,6 +753,12 @@ void kvm_mmu_invalidate_begin(struct kvm *kvm, unsigned long start, } } +static bool kvm_mmu_unmap_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range) +{ +kvm_mmu_invalidate_range_add (kvm, range->start, range->end); +return kvm_unmap_gfn_range (kvm, range); +} +static int kvm_mmu_notifier_invalidate_range_start (struct mmu_notifier *mn,const struct mmu_notifier_range *range) {@@ -752,7 +767,7 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn, .start = range->start, .end = range->end, .pte = __pte(0), - .handler = kvm_unmap_gfn_range, + .handler = kvm_mmu_unmap_gfn_range, .on_lock = kvm_mmu_invalidate_begin, .on_unlock = kvm_arch_guest_memory_reclaimed, .flush_on_ret = true,@@ -791,8 +806,7 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,return 0 ; } -void kvm_mmu_invalidate_end (struct kvm *kvm,unsigned long start, -unsigned long end) +void kvm_mmu_invalidate_end (struct kvm *kvm) {/* * This sequence increase will notify the kvm page fault that -- 2.25.1
Re: [PATCH v10 5/9] KVM: Use gfn instead of hva for mmu_notifier_retry
Hi,
Quoted 104 lines · Chao Peng
Quoted 102 lines · Fuad Tabba
Hi Chao,
Quoted 95 lines · Chao Peng
Currently in
mmu_notifierinvalidate path, hva range is recorded and then checked against bymmu_notifier_retry_hva()in the page fault handling path. However, for the to be introduced private memory, a page fault may not have a hva associated, checking gfn(gpa) makes more sense.For existing hva based shared memory, gfn is expected to also work. The only downside is when aliasing multiple gfns to a single hva, the current algorithm of checking multiple ranges could result in a much larger range being rejected. Such aliasing should be uncommon, so the impact is expected small.
Suggested-by Sean Christopherson <[email protected]>Signed-off-by Chao Peng <[email protected]>--- arch/x86/kvm/mmu/mmu.c | 8 +++++--- include/linux/kvm_host.h | 33 +++++++++++++++++++++------------ virt/kvm/kvm_main.c | 32 +++++++++++++++++++++++--------- 3 files changed, 49 insertions(+), 24 deletions(-)arch/x86/kvm/mmu/mmu.c +5
-3@@ -4259,7 +4259,7 @@ static bool is_page_fault_stale(struct kvm_vcpu *vcpu,return true;return fault->slot && -mmu_invalidate_retry_hva (vcpu->kvm, mmu_seq, fault->hva); +mmu_invalidate_retry_gfn (vcpu->kvm, mmu_seq, fault->gfn); }static int direct_page_fault (struct kvm_vcpu *vcpu,struct kvm_page_fault *fault)@@ -6098,7 +6098,9 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end)write_lock (&kvm->mmu_lock); -kvm_mmu_invalidate_begin (kvm, gfn_start, gfn_end); +kvm_mmu_invalidate_begin (kvm); + +kvm_mmu_invalidate_range_add (kvm, gfn_start, gfn_end); flush =kvm_rmap_zap_gfn_range (kvm, gfn_start, gfn_end);@@ -6112,7 +6114,7 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end)kvm_flush_remote_tlbs_with_address (kvm, gfn_start, gfn_end - gfn_start); -kvm_mmu_invalidate_end (kvm, gfn_start, gfn_end); +kvm_mmu_invalidate_end (kvm);write_unlock (&kvm->mmu_lock); }include/linux/kvm_host.h +7
-8@@ -787,8 +787,8 @@ struct kvm {struct mmu_notifier mmu_notifier;unsigned long mmu_invalidate_seq;long mmu_invalidate_in_progress; -unsigned long mmu_invalidate_range_start; -unsigned long mmu_invalidate_range_end; +gfn_t mmu_invalidate_range_start; +gfn_t mmu_invalidate_range_end;#endif struct list_head devices;u64 manual_dirty_log_protect;@@ -1389,10 +1389,9 @@ void kvm_mmu_free_memory_cache(struct kvm_mmu_memory_cache *mc);void *kvm_mmu_memory_cache_alloc (struct kvm_mmu_memory_cache *mc);#endif -void kvm_mmu_invalidate_begin (struct kvm *kvm,unsigned long start, -unsigned long end); -void kvm_mmu_invalidate_end (struct kvm *kvm,unsigned long start, -unsigned long end); +void kvm_mmu_invalidate_begin (struct kvm *kvm); +void kvm_mmu_invalidate_range_add (struct kvm *kvm,gfn_t start,gfn_t end); +void kvm_mmu_invalidate_end (struct kvm *kvm);long kvm_arch_dev_ioctl (struct file *filp,unsigned int ioctl,unsigned long arg);@@ -1963,9 +1962,9 @@ static inline int mmu_invalidate_retry(struct kvm *kvm, unsigned long mmu_seq)return 0 ; } -static inline int mmu_invalidate_retry_hva (struct kvm *kvm, +static inline int mmu_invalidate_retry_gfn (struct kvm *kvm,unsigned long mmu_seq, -unsigned long hva) +gfn_t gfn) {lockdep_assert_held (&kvm->mmu_lock);/* include/linux/kvm_host.h static inline int mmu_invalidate_retry_hva(struct kvm *kvm, · line 1973
Quoted 1 line
include/linux/kvm_host.h +0
-0@@ -1974,10 +1973,20 @@ static inline int mmu_invalidate_retry_hva(struct kvm *kvm,* that might be being invalidated. Note that it may include some false nit: "might be" (or) "is being"
Quoted 5 lines
Quoted 3 lines
include/linux/kvm_host.h static inline int mmu_invalidate_retry_hva(struct kvm *kvm, · line 1974
Quoted 1 line
include/linux/kvm_host.h +0
-0* positives, due to shortcuts when handing concurrent invalidations. nit: handling
Both are existing code, but I can fix it either.
That was just a nit, please feel free to ignore it, especially if it might cause headaches in the future with merges.
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
include/linux/kvm_host.h +7
-4*/ -if (unlikely (kvm->mmu_invalidate_in_progress) && - hva >= kvm->mmu_invalidate_range_start && - hva < kvm->mmu_invalidate_range_end) -return 1 ; +if (unlikely (kvm->mmu_invalidate_in_progress)) { +/* +* Dropping mmu_lock after bumping mmu_invalidate_in_progress +* but before updating the range is a KVM bug. +*/ +if (WARN_ON_ONCE (kvm->mmu_invalidate_range_start ==INVALID_GPA || + kvm->mmu_invalidate_range_end ==INVALID_GPA ))
Quoted 8 lines
Quoted 4 lines
INVALID_GPAis an x86-specific define inarch/x86/include/asm/kvm_host.h, so this doesn't build on other architectures. The obvious fix is to move it toinclude/linux/kvm_host.h.Hmm,
INVALID_GPAis defined as ZERO for x86, not 100% confident this is correct choice for other architectures, but after search it has not been used for other architectures, so should be safe to make it common.
With this fixed,
And the necessary work to port to arm64 (on qemu/arm64):
Cheers,
/fuad
Quoted 105 lines
Thanks,
ChaoQuoted 102 lines
Cheers,
/fuadQuoted 98 lines
include/linux/kvm_host.h +7
-0+return 1 ; + +if (gfn >= kvm->mmu_invalidate_range_start && + gfn < kvm->mmu_invalidate_range_end) +return 1 ; + } +if (kvm->mmu_invalidate_seq != mmu_seq)return 1 ;return 0 ;virt/kvm/kvm_main.c +23
-10@@ -540,9 +540,7 @@ static void kvm_mmu_notifier_invalidate_range(struct mmu_notifier *mn,typedef bool (*hva_handler_t )(struct kvm *kvm,struct kvm_gfn_range *range); -typedef void (*on_lock_fn_t )(struct kvm *kvm,unsigned long start, -unsigned long end); - +typedef void (*on_lock_fn_t )(struct kvm *kvm);typedef void (*on_unlock_fn_t )(struct kvm *kvm);struct kvm_hva_range {@@ -628,7 +626,8 @@ static __always_inline int __kvm_handle_hva_range(struct kvm *kvm, locked = true;KVM_MMU_LOCK (kvm);if (!IS_KVM_NULL_FN (range->on_lock)) - range->on_lock (kvm, range->start, range->end); + range->on_lock (kvm); +if (IS_KVM_NULL_FN (range->handler))break ; }@@ -715,8 +714,7 @@ static void kvm_mmu_notifier_change_pte(struct mmu_notifier *mn,kvm_handle_hva_range (mn, address, address +1 , pte, kvm_set_spte_gfn); } -void kvm_mmu_invalidate_begin (struct kvm *kvm,unsigned long start, -unsigned long end) +void kvm_mmu_invalidate_begin (struct kvm *kvm) {/* * The count increase must become visible at unlock time as no @@ -724,6 +722,17 @@ void kvm_mmu_invalidate_begin(struct kvm *kvm, unsigned long start,* count is also read inside the mmu_lock critical section. */ kvm->mmu_invalidate_in_progress++; + +if (likely (kvm->mmu_invalidate_in_progress ==1 )) { + kvm->mmu_invalidate_range_start =INVALID_GPA ; + kvm->mmu_invalidate_range_end =INVALID_GPA ; + } +} + +void kvm_mmu_invalidate_range_add (struct kvm *kvm,gfn_t start,gfn_t end) +{ +WARN_ON_ONCE (!kvm->mmu_invalidate_in_progress); +if (likely (kvm->mmu_invalidate_in_progress ==1 )) { kvm->mmu_invalidate_range_start = start; kvm->mmu_invalidate_range_end = end;@@ -744,6 +753,12 @@ void kvm_mmu_invalidate_begin(struct kvm *kvm, unsigned long start, } } +static bool kvm_mmu_unmap_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range) +{ +kvm_mmu_invalidate_range_add (kvm, range->start, range->end); +return kvm_unmap_gfn_range (kvm, range); +} +static int kvm_mmu_notifier_invalidate_range_start (struct mmu_notifier *mn,const struct mmu_notifier_range *range) {@@ -752,7 +767,7 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn, .start = range->start, .end = range->end, .pte = __pte(0), - .handler = kvm_unmap_gfn_range, + .handler = kvm_mmu_unmap_gfn_range, .on_lock = kvm_mmu_invalidate_begin, .on_unlock = kvm_arch_guest_memory_reclaimed, .flush_on_ret = true,@@ -791,8 +806,7 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,return 0 ; } -void kvm_mmu_invalidate_end (struct kvm *kvm,unsigned long start, -unsigned long end) +void kvm_mmu_invalidate_end (struct kvm *kvm) {/* * This sequence increase will notify the kvm page fault that -- 2.25.1
Re: [PATCH v10 5/9] KVM: Use gfn instead of hva for mmu_notifier_retry
On Tue, Dec 06, 2022 at 03:48:50PM +0000, Fuad Tabba wrote:
...
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
*/include/linux/kvm_host.h +7
-4-if (unlikely (kvm->mmu_invalidate_in_progress) && - hva >= kvm->mmu_invalidate_range_start && - hva < kvm->mmu_invalidate_range_end) -return 1 ; +if (unlikely (kvm->mmu_invalidate_in_progress)) { +/* +* Dropping mmu_lock after bumping mmu_invalidate_in_progress +* but before updating the range is a KVM bug. +*/ +if (WARN_ON_ONCE (kvm->mmu_invalidate_range_start ==INVALID_GPA || + kvm->mmu_invalidate_range_end ==INVALID_GPA ))
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
INVALID_GPAis an x86-specific define inarch/x86/include/asm/kvm_host.h, so this doesn't build on other architectures. The obvious fix is to move it toinclude/linux/kvm_host.h.Hmm,
INVALID_GPAis defined as ZERO for x86, not 100% confident this is correct choice for other architectures, but after search it has not been used for other architectures, so should be safe to make it common.
As Yu posted a patch:
https://lore.kernel.org/all/[email protected]/
There is a GPA_INVALID in include/linux/kvm_types.h and I see ARM has already been using it so sounds that is exactly what I need.
Chao
Quoted 9 lines
With this fixed,
Reviewed-by Fuad Tabba <[email protected]>And the necessary work to port to arm64 (on qemu/arm64):
Tested-by Fuad Tabba <[email protected]>Cheers,
/fuad
Re: [PATCH v10 5/9] KVM: Use gfn instead of hva for mmu_notifier_retry
In reply to Chao PengQuoted 12 lines · Chao Peng
Quoted 12 lines
Quoted 11 lines
include/linux/kvm_host.h +7
-4-if (unlikely (kvm->mmu_invalidate_in_progress) && - hva >= kvm->mmu_invalidate_range_start && - hva < kvm->mmu_invalidate_range_end) -return 1 ; +if (unlikely (kvm->mmu_invalidate_in_progress)) { +/* +* Dropping mmu_lock after bumping mmu_invalidate_in_progress +* but before updating the range is a KVM bug. +*/ +if (WARN_ON_ONCE (kvm->mmu_invalidate_range_start ==INVALID_GPA || + kvm->mmu_invalidate_range_end ==INVALID_GPA ))
Quoted 8 lines
Quoted 4 lines
INVALID_GPAis an x86-specific define inarch/x86/include/asm/kvm_host.h, so this doesn't build on other architectures. The obvious fix is to move it toinclude/linux/kvm_host.h.Hmm,
INVALID_GPAis defined as ZERO for x86, not 100% confident this is correct choice for other architectures, but after search it has not been used for other architectures, so should be safe to make it common.
INVALID_GPA is defined as all bit 1. Please notice "~" (tilde).
#define INVALID_GPA (~(gpa_t)0)
Signature · 1 line
Isaku Yamahata <[email protected]>
Re: [PATCH v10 5/9] KVM: Use gfn instead of hva for mmu_notifier_retry
Quoted 20 lines · Isaku Yamahata
Quoted 17 lines · Chao Peng
Quoted 16 lines
include/linux/kvm_host.h static inline int mmu_invalidate_retry_hva(struct kvm *kvm, · line 1982
Quoted 11 lines
include/linux/kvm_host.h +7
-4-if (unlikely (kvm->mmu_invalidate_in_progress) && - hva >= kvm->mmu_invalidate_range_start && - hva < kvm->mmu_invalidate_range_end) -return 1 ; +if (unlikely (kvm->mmu_invalidate_in_progress)) { +/* +* Dropping mmu_lock after bumping mmu_invalidate_in_progress +* but before updating the range is a KVM bug. +*/ +if (WARN_ON_ONCE (kvm->mmu_invalidate_range_start ==INVALID_GPA || + kvm->mmu_invalidate_range_end ==INVALID_GPA ))
INVALID_GPAis an x86-specific define inarch/x86/include/asm/kvm_host.h, so this doesn't build on other architectures. The obvious fix is to move it toinclude/linux/kvm_host.h.
Quoted 7 lines
Quoted 3 lines
Hmm,
INVALID_GPAis defined as ZERO for x86, not 100% confident this is correct choice for other architectures, but after search it has not been used for other architectures, so should be safe to make it common.
INVALID_GPAis defined as all bit 1. Please notice "~" (tilde).#define
INVALID_GPA(~(gpa_t)0)
Thanks for mention. Still looks right moving it to include/linux/kvm_host.h. Chao
Quoted 2 lines
Signature · 1 line
Isaku Yamahata <[email protected]>
PATCH v10 6/9 KVM: Unmap existing mappings when change the memory attributes
In reply to Chao PengUnmap the existing guest mappings when memory attribute is changed between shared and private. This is needed because shared pages and private pages are from different backends, unmapping existing ones gives a chance for page fault handler to re-populate the mappings according to the new attribute.
Only architecture has private memory support needs this and the supported architecture is expected to rewrite the weak kvm_arch_has_private_mem().
Also, during memory attribute changing and the unmapping time frame, page fault handler may happen in the same memory range and can cause incorrect page state, invoke kvm_mmu_invalidate_* helpers to let the page fault handler retry during this time frame.
--- include/linux/kvm_host.h | 7 +- virt/kvm/kvm_main.c | 168 ++++++++++++++++++++++++++------------- 2 files changed, 116 insertions(+), 59 deletions(-)
include/linux/kvm_host.h +5 -2
@@ -255,7 +255,6 @@ bool kvm_setup_async_pf(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,int kvm_async_pf_wakeup_all (struct kvm_vcpu *vcpu);#endif -#ifdef KVM_ARCH_WANT_MMU_NOTIFIERstruct kvm_gfn_range {struct kvm_memory_slot *slot;gfn_t start;
@@ -264,6 +263,8 @@ struct kvm_gfn_range {bool may_block; };bool kvm_unmap_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range); + +#ifdef KVM_ARCH_WANT_MMU_NOTIFIER bool kvm_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range);bool kvm_test_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range);bool kvm_set_spte_gfn (struct kvm *kvm,struct kvm_gfn_range *range);
@@ -785,11 +786,12 @@ struct kvm {#if defined(CONFIG_MMU_NOTIFIER ) && defined(KVM_ARCH_WANT_MMU_NOTIFIER )struct mmu_notifier mmu_notifier; +#endif unsigned long mmu_invalidate_seq;long mmu_invalidate_in_progress;gfn_t mmu_invalidate_range_start;gfn_t mmu_invalidate_range_end; -#endif +struct list_head devices;u64 manual_dirty_log_protect;struct dentry *debugfs_dentry;
@@ -1480,6 +1482,7 @@ bool kvm_arch_dy_has_pending_interrupt(struct kvm_vcpu *vcpu);int kvm_arch_post_init_vm (struct kvm *kvm);void kvm_arch_pre_destroy_vm (struct kvm *kvm);int kvm_arch_create_vm_debugfs (struct kvm *kvm); +bool kvm_arch_has_private_mem (struct kvm *kvm);#ifndef __KVM_HAVE_ARCH_VM_ALLOC/*
virt/kvm/kvm_main.c +111 -57
218 lines not on this page · raw message
Signature · 1 line
2.25.1
Re: [PATCH v10 6/9] KVM: Unmap existing mappings when change the memory attributes
Quoted 263 lines · Chao Peng
Unmap the existing guest mappings when memory attribute is changed between shared and private. This is needed because shared pages and private pages are from different backends, unmapping existing ones gives a chance for page fault handler to re-populate the mappings according to the new attribute.
Only architecture has private memory support needs this and the supported architecture is expected to rewrite the weak
kvm_arch_has_private_mem().Also, during memory attribute changing and the unmapping time frame, page fault handler may happen in the same memory range and can cause incorrect page state, invoke kvm_mmu_invalidate_* helpers to let the page fault handler retry during this time frame.
Signed-off-by Chao Peng <[email protected]>--- include/linux/kvm_host.h | 7 +- virt/kvm/kvm_main.c | 168 ++++++++++++++++++++++++++------------- 2 files changed, 116 insertions(+), 59 deletions(-)include/linux/kvm_host.h +5
-2@@ -255,7 +255,6 @@ bool kvm_setup_async_pf(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,int kvm_async_pf_wakeup_all (struct kvm_vcpu *vcpu);#endif -#ifdef KVM_ARCH_WANT_MMU_NOTIFIERstruct kvm_gfn_range {struct kvm_memory_slot *slot;gfn_t start;@@ -264,6 +263,8 @@ struct kvm_gfn_range {bool may_block; };bool kvm_unmap_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range); + +#ifdef KVM_ARCH_WANT_MMU_NOTIFIER bool kvm_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range);bool kvm_test_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range);bool kvm_set_spte_gfn (struct kvm *kvm,struct kvm_gfn_range *range);@@ -785,11 +786,12 @@ struct kvm {#if defined(CONFIG_MMU_NOTIFIER ) && defined(KVM_ARCH_WANT_MMU_NOTIFIER )struct mmu_notifier mmu_notifier; +#endif unsigned long mmu_invalidate_seq;long mmu_invalidate_in_progress;gfn_t mmu_invalidate_range_start;gfn_t mmu_invalidate_range_end; -#endif +struct list_head devices;u64 manual_dirty_log_protect;struct dentry *debugfs_dentry;@@ -1480,6 +1482,7 @@ bool kvm_arch_dy_has_pending_interrupt(struct kvm_vcpu *vcpu);int kvm_arch_post_init_vm (struct kvm *kvm);void kvm_arch_pre_destroy_vm (struct kvm *kvm);int kvm_arch_create_vm_debugfs (struct kvm *kvm); +bool kvm_arch_has_private_mem (struct kvm *kvm);#ifndef __KVM_HAVE_ARCH_VM_ALLOC/* virt/kvm/kvm_main.c +94
-57@@ -520,6 +520,62 @@ void kvm_destroy_vcpus(struct kvm *kvm) }EXPORT_SYMBOL_GPL (kvm_destroy_vcpus); +void kvm_mmu_invalidate_begin (struct kvm *kvm) +{ +/* +* The count increase must become visible at unlock time as no +* spte can be established without taking the mmu_lock and +* count is also read inside the mmu_lock critical section. +*/ + kvm->mmu_invalidate_in_progress++; + +if (likely (kvm->mmu_invalidate_in_progress ==1 )) { + kvm->mmu_invalidate_range_start =INVALID_GPA ; + kvm->mmu_invalidate_range_end =INVALID_GPA ; + } +} + +void kvm_mmu_invalidate_range_add (struct kvm *kvm,gfn_t start,gfn_t end) +{ +WARN_ON_ONCE (!kvm->mmu_invalidate_in_progress); + +if (likely (kvm->mmu_invalidate_in_progress ==1 )) { + kvm->mmu_invalidate_range_start = start; + kvm->mmu_invalidate_range_end = end; + }else { +/* +* Fully tracking multiple concurrent ranges has diminishing +* returns. Keep things simple and just find the minimal range +* which includes the current and new ranges. As there won't be +* enough information to subtract a range after its invalidate +* completes, any ranges invalidated concurrently will +* accumulate and persist until all outstanding invalidates +* complete. +*/ + kvm->mmu_invalidate_range_start = +min (kvm->mmu_invalidate_range_start, start); + kvm->mmu_invalidate_range_end = +max (kvm->mmu_invalidate_range_end, end); + } +} + +void kvm_mmu_invalidate_end (struct kvm *kvm) +{ +/* +* This sequence increase will notify the kvm page fault that +* the page that is going to be mapped in the spte could have +* been freed. +*/ + kvm->mmu_invalidate_seq++; +smp_wmb (); +/* +* The above sequence increase must be visible before the +* below count decrease, which is ensured by the smp_wmb above +* in conjunction with the smp_rmb in mmu_invalidate_retry(). +*/ + kvm->mmu_invalidate_in_progress--; +} +#if defined(CONFIG_MMU_NOTIFIER ) && defined(KVM_ARCH_WANT_MMU_NOTIFIER )static inline struct kvm *mmu_notifier_to_kvm (struct mmu_notifier *mn) {@@ -714,45 +770,6 @@ static void kvm_mmu_notifier_change_pte(struct mmu_notifier *mn,kvm_handle_hva_range (mn, address, address +1 , pte, kvm_set_spte_gfn); } -void kvm_mmu_invalidate_begin (struct kvm *kvm) -{ -/* -* The count increase must become visible at unlock time as no -* spte can be established without taking the mmu_lock and -* count is also read inside the mmu_lock critical section. -*/ - kvm->mmu_invalidate_in_progress++; - -if (likely (kvm->mmu_invalidate_in_progress ==1 )) { - kvm->mmu_invalidate_range_start =INVALID_GPA ; - kvm->mmu_invalidate_range_end =INVALID_GPA ; - } -} - -void kvm_mmu_invalidate_range_add (struct kvm *kvm,gfn_t start,gfn_t end) -{ -WARN_ON_ONCE (!kvm->mmu_invalidate_in_progress); - -if (likely (kvm->mmu_invalidate_in_progress ==1 )) { - kvm->mmu_invalidate_range_start = start; - kvm->mmu_invalidate_range_end = end; - }else { -/* -* Fully tracking multiple concurrent ranges has diminishing -* returns. Keep things simple and just find the minimal range -* which includes the current and new ranges. As there won't be -* enough information to subtract a range after its invalidate -* completes, any ranges invalidated concurrently will -* accumulate and persist until all outstanding invalidates -* complete. -*/ - kvm->mmu_invalidate_range_start = -min (kvm->mmu_invalidate_range_start, start); - kvm->mmu_invalidate_range_end = -max (kvm->mmu_invalidate_range_end, end); - } -} -static bool kvm_mmu_unmap_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range) {kvm_mmu_invalidate_range_add (kvm, range->start, range->end);@@ -806,23 +823,6 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,return 0 ; } -void kvm_mmu_invalidate_end (struct kvm *kvm) -{ -/* -* This sequence increase will notify the kvm page fault that -* the page that is going to be mapped in the spte could have -* been freed. -*/ - kvm->mmu_invalidate_seq++; -smp_wmb (); -/* -* The above sequence increase must be visible before the -* below count decrease, which is ensured by the smp_wmb above -* in conjunction with the smp_rmb in mmu_invalidate_retry(). -*/ - kvm->mmu_invalidate_in_progress--; -} -static void kvm_mmu_notifier_invalidate_range_end (struct mmu_notifier *mn,const struct mmu_notifier_range *range) {@@ -1140,6 +1140,11 @@ int __weak kvm_arch_create_vm_debugfs(struct kvm *kvm)return 0 ; } +bool __weak kvm_arch_has_private_mem (struct kvm *kvm) +{ +return false; +} +static struct kvm *kvm_create_vm (unsigned long type,const char *fdname) {struct kvm *kvm =kvm_arch_alloc_vm ();@@ -2349,15 +2354,47 @@ static u64 kvm_supported_mem_attributes(struct kvm *kvm)return 0 ; } +static void kvm_unmap_mem_range (struct kvm *kvm,gfn_t start,gfn_t end) +{ +struct kvm_gfn_range gfn_range; +struct kvm_memory_slot *slot; +struct kvm_memslots *slots; +struct kvm_memslot_iter iter; +int i; +int r =0 ; + + gfn_range.pte =__pte (0 ); + gfn_range.may_block = true; + +for (i =0 ; i <KVM_ADDRESS_SPACE_NUM ; i++) { + slots =__kvm_memslots (kvm, i); + +kvm_for_each_memslot_in_gfn_range (&iter, slots, start, end) { + slot = iter.slot; + gfn_range.start =max (start, slot->base_gfn); + gfn_range.end =min (end, slot->base_gfn + slot->npages); +if (gfn_range.start >= gfn_range.end) +continue ; + gfn_range.slot = slot; + + r |=kvm_unmap_gfn_range (kvm, &gfn_range); + } + } + +if (r) +kvm_flush_remote_tlbs (kvm); +} +static int kvm_vm_ioctl_set_mem_attributes (struct kvm *kvm,struct kvm_memory_attributes *attrs) {gfn_t start, end;unsigned long i;void *entry; +int idx;u64 supported_attrs =kvm_supported_mem_attributes (kvm); -/* flags is currently not used. */ +/* 'flags' is currently not used. */ if (attrs->flags)return -EINVAL ;if (attrs->attributes & ~supported_attrs)
virt/kvm/kvm_main.c static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, · line 2415
Quoted 7 lines
virt/kvm/kvm_main.c +4
-0@@ -2372,6 +2409,13 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, entry = attrs->attributes ?xa_mk_value (attrs->attributes) :NULL ; +if (kvm_arch_has_private_mem (kvm)) { +KVM_MMU_LOCK (kvm); +kvm_mmu_invalidate_begin (kvm); +kvm_mmu_invalidate_range_add (kvm, start, end);
Nit: this works for KVM_MEMORY_ATTRIBUTE_PRIVATE, but the invalidation should be necessary yet for attribute change of:
KVM_MEMORY_ATTRIBUTE_READKVM_MEMORY_ATTRIBUTE_WRITEKVM_MEMORY_ATTRIBUTE_EXECUTE
Quoted 7 lines
virt/kvm/kvm_main.c +3
-0+KVM_MMU_UNLOCK (kvm); + } +mutex_lock (&kvm->lock);for (i = start; i < end; i++) if (xa_err(xa_store(&kvm->mem_attr_array, i, entry,
virt/kvm/kvm_main.c static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, · line 2431
Quoted 9 lines
virt/kvm/kvm_main.c +6
-0@@ -2379,6 +2423,16 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,break ;mutex_unlock (&kvm->lock); +if (kvm_arch_has_private_mem (kvm)) { + idx =srcu_read_lock (&kvm->srcu); +KVM_MMU_LOCK (kvm); +if (i > start) +kvm_unmap_mem_range (kvm, start, i); +kvm_mmu_invalidate_end (kvm);
Ditto.
Quoted 11 lines
virt/kvm/kvm_main.c +4
-1+KVM_MMU_UNLOCK (kvm); +srcu_read_unlock (&kvm->srcu, idx); + } + attrs->address = i <<PAGE_SHIFT ; attrs->size = (end - i) <<PAGE_SHIFT ; --2.25.1
Re: [PATCH v10 6/9] KVM: Unmap existing mappings when change the memory attributes
Quoted 275 lines · Yuan Yao
Quoted 263 lines · Chao Peng
Unmap the existing guest mappings when memory attribute is changed between shared and private. This is needed because shared pages and private pages are from different backends, unmapping existing ones gives a chance for page fault handler to re-populate the mappings according to the new attribute.
Only architecture has private memory support needs this and the supported architecture is expected to rewrite the weak
kvm_arch_has_private_mem().Also, during memory attribute changing and the unmapping time frame, page fault handler may happen in the same memory range and can cause incorrect page state, invoke kvm_mmu_invalidate_* helpers to let the page fault handler retry during this time frame.
Signed-off-by Chao Peng <[email protected]>--- include/linux/kvm_host.h | 7 +- virt/kvm/kvm_main.c | 168 ++++++++++++++++++++++++++------------- 2 files changed, 116 insertions(+), 59 deletions(-)include/linux/kvm_host.h +5
-2@@ -255,7 +255,6 @@ bool kvm_setup_async_pf(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,int kvm_async_pf_wakeup_all (struct kvm_vcpu *vcpu);#endif -#ifdef KVM_ARCH_WANT_MMU_NOTIFIERstruct kvm_gfn_range {struct kvm_memory_slot *slot;gfn_t start;@@ -264,6 +263,8 @@ struct kvm_gfn_range {bool may_block; };bool kvm_unmap_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range); + +#ifdef KVM_ARCH_WANT_MMU_NOTIFIER bool kvm_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range);bool kvm_test_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range);bool kvm_set_spte_gfn (struct kvm *kvm,struct kvm_gfn_range *range);@@ -785,11 +786,12 @@ struct kvm {#if defined(CONFIG_MMU_NOTIFIER ) && defined(KVM_ARCH_WANT_MMU_NOTIFIER )struct mmu_notifier mmu_notifier; +#endif unsigned long mmu_invalidate_seq;long mmu_invalidate_in_progress;gfn_t mmu_invalidate_range_start;gfn_t mmu_invalidate_range_end; -#endif +struct list_head devices;u64 manual_dirty_log_protect;struct dentry *debugfs_dentry;@@ -1480,6 +1482,7 @@ bool kvm_arch_dy_has_pending_interrupt(struct kvm_vcpu *vcpu);int kvm_arch_post_init_vm (struct kvm *kvm);void kvm_arch_pre_destroy_vm (struct kvm *kvm);int kvm_arch_create_vm_debugfs (struct kvm *kvm); +bool kvm_arch_has_private_mem (struct kvm *kvm);#ifndef __KVM_HAVE_ARCH_VM_ALLOC/* virt/kvm/kvm_main.c +94
-57@@ -520,6 +520,62 @@ void kvm_destroy_vcpus(struct kvm *kvm) }EXPORT_SYMBOL_GPL (kvm_destroy_vcpus); +void kvm_mmu_invalidate_begin (struct kvm *kvm) +{ +/* +* The count increase must become visible at unlock time as no +* spte can be established without taking the mmu_lock and +* count is also read inside the mmu_lock critical section. +*/ + kvm->mmu_invalidate_in_progress++; + +if (likely (kvm->mmu_invalidate_in_progress ==1 )) { + kvm->mmu_invalidate_range_start =INVALID_GPA ; + kvm->mmu_invalidate_range_end =INVALID_GPA ; + } +} + +void kvm_mmu_invalidate_range_add (struct kvm *kvm,gfn_t start,gfn_t end) +{ +WARN_ON_ONCE (!kvm->mmu_invalidate_in_progress); + +if (likely (kvm->mmu_invalidate_in_progress ==1 )) { + kvm->mmu_invalidate_range_start = start; + kvm->mmu_invalidate_range_end = end; + }else { +/* +* Fully tracking multiple concurrent ranges has diminishing +* returns. Keep things simple and just find the minimal range +* which includes the current and new ranges. As there won't be +* enough information to subtract a range after its invalidate +* completes, any ranges invalidated concurrently will +* accumulate and persist until all outstanding invalidates +* complete. +*/ + kvm->mmu_invalidate_range_start = +min (kvm->mmu_invalidate_range_start, start); + kvm->mmu_invalidate_range_end = +max (kvm->mmu_invalidate_range_end, end); + } +} + +void kvm_mmu_invalidate_end (struct kvm *kvm) +{ +/* +* This sequence increase will notify the kvm page fault that +* the page that is going to be mapped in the spte could have +* been freed. +*/ + kvm->mmu_invalidate_seq++; +smp_wmb (); +/* +* The above sequence increase must be visible before the +* below count decrease, which is ensured by the smp_wmb above +* in conjunction with the smp_rmb in mmu_invalidate_retry(). +*/ + kvm->mmu_invalidate_in_progress--; +} +#if defined(CONFIG_MMU_NOTIFIER ) && defined(KVM_ARCH_WANT_MMU_NOTIFIER )static inline struct kvm *mmu_notifier_to_kvm (struct mmu_notifier *mn) {@@ -714,45 +770,6 @@ static void kvm_mmu_notifier_change_pte(struct mmu_notifier *mn,kvm_handle_hva_range (mn, address, address +1 , pte, kvm_set_spte_gfn); } -void kvm_mmu_invalidate_begin (struct kvm *kvm) -{ -/* -* The count increase must become visible at unlock time as no -* spte can be established without taking the mmu_lock and -* count is also read inside the mmu_lock critical section. -*/ - kvm->mmu_invalidate_in_progress++; - -if (likely (kvm->mmu_invalidate_in_progress ==1 )) { - kvm->mmu_invalidate_range_start =INVALID_GPA ; - kvm->mmu_invalidate_range_end =INVALID_GPA ; - } -} - -void kvm_mmu_invalidate_range_add (struct kvm *kvm,gfn_t start,gfn_t end) -{ -WARN_ON_ONCE (!kvm->mmu_invalidate_in_progress); - -if (likely (kvm->mmu_invalidate_in_progress ==1 )) { - kvm->mmu_invalidate_range_start = start; - kvm->mmu_invalidate_range_end = end; - }else { -/* -* Fully tracking multiple concurrent ranges has diminishing -* returns. Keep things simple and just find the minimal range -* which includes the current and new ranges. As there won't be -* enough information to subtract a range after its invalidate -* completes, any ranges invalidated concurrently will -* accumulate and persist until all outstanding invalidates -* complete. -*/ - kvm->mmu_invalidate_range_start = -min (kvm->mmu_invalidate_range_start, start); - kvm->mmu_invalidate_range_end = -max (kvm->mmu_invalidate_range_end, end); - } -} -static bool kvm_mmu_unmap_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range) {kvm_mmu_invalidate_range_add (kvm, range->start, range->end);@@ -806,23 +823,6 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,return 0 ; } -void kvm_mmu_invalidate_end (struct kvm *kvm) -{ -/* -* This sequence increase will notify the kvm page fault that -* the page that is going to be mapped in the spte could have -* been freed. -*/ - kvm->mmu_invalidate_seq++; -smp_wmb (); -/* -* The above sequence increase must be visible before the -* below count decrease, which is ensured by the smp_wmb above -* in conjunction with the smp_rmb in mmu_invalidate_retry(). -*/ - kvm->mmu_invalidate_in_progress--; -} -static void kvm_mmu_notifier_invalidate_range_end (struct mmu_notifier *mn,const struct mmu_notifier_range *range) {@@ -1140,6 +1140,11 @@ int __weak kvm_arch_create_vm_debugfs(struct kvm *kvm)return 0 ; } +bool __weak kvm_arch_has_private_mem (struct kvm *kvm) +{ +return false; +} +static struct kvm *kvm_create_vm (unsigned long type,const char *fdname) {struct kvm *kvm =kvm_arch_alloc_vm ();@@ -2349,15 +2354,47 @@ static u64 kvm_supported_mem_attributes(struct kvm *kvm)return 0 ; } +static void kvm_unmap_mem_range (struct kvm *kvm,gfn_t start,gfn_t end) +{ +struct kvm_gfn_range gfn_range; +struct kvm_memory_slot *slot; +struct kvm_memslots *slots; +struct kvm_memslot_iter iter; +int i; +int r =0 ; + + gfn_range.pte =__pte (0 ); + gfn_range.may_block = true; + +for (i =0 ; i <KVM_ADDRESS_SPACE_NUM ; i++) { + slots =__kvm_memslots (kvm, i); + +kvm_for_each_memslot_in_gfn_range (&iter, slots, start, end) { + slot = iter.slot; + gfn_range.start =max (start, slot->base_gfn); + gfn_range.end =min (end, slot->base_gfn + slot->npages); +if (gfn_range.start >= gfn_range.end) +continue ; + gfn_range.slot = slot; + + r |=kvm_unmap_gfn_range (kvm, &gfn_range); + } + } + +if (r) +kvm_flush_remote_tlbs (kvm); +} +static int kvm_vm_ioctl_set_mem_attributes (struct kvm *kvm,struct kvm_memory_attributes *attrs) {gfn_t start, end;unsigned long i;void *entry; +int idx;u64 supported_attrs =kvm_supported_mem_attributes (kvm); -/* flags is currently not used. */ +/* 'flags' is currently not used. */ if (attrs->flags)return -EINVAL ;if (attrs->attributes & ~supported_attrs)virt/kvm/kvm_main.c static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, · line 2415
Quoted 7 lines
virt/kvm/kvm_main.c +4
-0@@ -2372,6 +2409,13 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, entry = attrs->attributes ?xa_mk_value (attrs->attributes) :NULL ; +if (kvm_arch_has_private_mem (kvm)) { +KVM_MMU_LOCK (kvm); +kvm_mmu_invalidate_begin (kvm); +kvm_mmu_invalidate_range_add (kvm, start, end);Nit: this works for
KVM_MEMORY_ATTRIBUTE_PRIVATE, but the invalidation should be necessary yet for attribute change of:
Quoted 3 lines
KVM_MEMORY_ATTRIBUTE_READKVM_MEMORY_ATTRIBUTE_WRITEKVM_MEMORY_ATTRIBUTE_EXECUTE
The unmapping is only needed for confidential usages which uses KVM_MEMORY_ATTRIBUTE_PRIVATE only and the other flags are defined here for other usages like pKVM. As Fuad commented in a different reply, pKVM supports in-place remapping and unmapping is unnecessary.
Thanks,
Chao
Quoted 31 lines
Quoted 8 lines
virt/kvm/kvm_main.c +3
-0+KVM_MMU_UNLOCK (kvm); + } +mutex_lock (&kvm->lock);for (i = start; i < end; i++) if (xa_err(xa_store(&kvm->mem_attr_array, i, entry,virt/kvm/kvm_main.c static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, · line 2431
Quoted 9 lines
virt/kvm/kvm_main.c +6
-0@@ -2379,6 +2423,16 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,break ;mutex_unlock (&kvm->lock); +if (kvm_arch_has_private_mem (kvm)) { + idx =srcu_read_lock (&kvm->srcu); +KVM_MMU_LOCK (kvm); +if (i > start) +kvm_unmap_mem_range (kvm, start, i); +kvm_mmu_invalidate_end (kvm);Ditto.
Quoted 11 lines
virt/kvm/kvm_main.c +4
-1+KVM_MMU_UNLOCK (kvm); +srcu_read_unlock (&kvm->srcu, idx); + } + attrs->address = i <<PAGE_SHIFT ; attrs->size = (end - i) <<PAGE_SHIFT ; --2.25.1
Re: [PATCH v10 6/9] KVM: Unmap existing mappings when change the memory attributes
Quoted 276 lines · Chao Peng
Quoted 275 lines · Yuan Yao
Quoted 263 lines · Chao Peng
Unmap the existing guest mappings when memory attribute is changed between shared and private. This is needed because shared pages and private pages are from different backends, unmapping existing ones gives a chance for page fault handler to re-populate the mappings according to the new attribute.
Only architecture has private memory support needs this and the supported architecture is expected to rewrite the weak
kvm_arch_has_private_mem().Also, during memory attribute changing and the unmapping time frame, page fault handler may happen in the same memory range and can cause incorrect page state, invoke kvm_mmu_invalidate_* helpers to let the page fault handler retry during this time frame.
Signed-off-by Chao Peng <[email protected]>--- include/linux/kvm_host.h | 7 +- virt/kvm/kvm_main.c | 168 ++++++++++++++++++++++++++------------- 2 files changed, 116 insertions(+), 59 deletions(-)include/linux/kvm_host.h +5
-2@@ -255,7 +255,6 @@ bool kvm_setup_async_pf(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,int kvm_async_pf_wakeup_all (struct kvm_vcpu *vcpu);#endif -#ifdef KVM_ARCH_WANT_MMU_NOTIFIERstruct kvm_gfn_range {struct kvm_memory_slot *slot;gfn_t start;@@ -264,6 +263,8 @@ struct kvm_gfn_range {bool may_block; };bool kvm_unmap_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range); + +#ifdef KVM_ARCH_WANT_MMU_NOTIFIER bool kvm_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range);bool kvm_test_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range);bool kvm_set_spte_gfn (struct kvm *kvm,struct kvm_gfn_range *range);@@ -785,11 +786,12 @@ struct kvm {#if defined(CONFIG_MMU_NOTIFIER ) && defined(KVM_ARCH_WANT_MMU_NOTIFIER )struct mmu_notifier mmu_notifier; +#endif unsigned long mmu_invalidate_seq;long mmu_invalidate_in_progress;gfn_t mmu_invalidate_range_start;gfn_t mmu_invalidate_range_end; -#endif +struct list_head devices;u64 manual_dirty_log_protect;struct dentry *debugfs_dentry;@@ -1480,6 +1482,7 @@ bool kvm_arch_dy_has_pending_interrupt(struct kvm_vcpu *vcpu);int kvm_arch_post_init_vm (struct kvm *kvm);void kvm_arch_pre_destroy_vm (struct kvm *kvm);int kvm_arch_create_vm_debugfs (struct kvm *kvm); +bool kvm_arch_has_private_mem (struct kvm *kvm);#ifndef __KVM_HAVE_ARCH_VM_ALLOC/* virt/kvm/kvm_main.c +94
-57@@ -520,6 +520,62 @@ void kvm_destroy_vcpus(struct kvm *kvm) }EXPORT_SYMBOL_GPL (kvm_destroy_vcpus); +void kvm_mmu_invalidate_begin (struct kvm *kvm) +{ +/* +* The count increase must become visible at unlock time as no +* spte can be established without taking the mmu_lock and +* count is also read inside the mmu_lock critical section. +*/ + kvm->mmu_invalidate_in_progress++; + +if (likely (kvm->mmu_invalidate_in_progress ==1 )) { + kvm->mmu_invalidate_range_start =INVALID_GPA ; + kvm->mmu_invalidate_range_end =INVALID_GPA ; + } +} + +void kvm_mmu_invalidate_range_add (struct kvm *kvm,gfn_t start,gfn_t end) +{ +WARN_ON_ONCE (!kvm->mmu_invalidate_in_progress); + +if (likely (kvm->mmu_invalidate_in_progress ==1 )) { + kvm->mmu_invalidate_range_start = start; + kvm->mmu_invalidate_range_end = end; + }else { +/* +* Fully tracking multiple concurrent ranges has diminishing +* returns. Keep things simple and just find the minimal range +* which includes the current and new ranges. As there won't be +* enough information to subtract a range after its invalidate +* completes, any ranges invalidated concurrently will +* accumulate and persist until all outstanding invalidates +* complete. +*/ + kvm->mmu_invalidate_range_start = +min (kvm->mmu_invalidate_range_start, start); + kvm->mmu_invalidate_range_end = +max (kvm->mmu_invalidate_range_end, end); + } +} + +void kvm_mmu_invalidate_end (struct kvm *kvm) +{ +/* +* This sequence increase will notify the kvm page fault that +* the page that is going to be mapped in the spte could have +* been freed. +*/ + kvm->mmu_invalidate_seq++; +smp_wmb (); +/* +* The above sequence increase must be visible before the +* below count decrease, which is ensured by the smp_wmb above +* in conjunction with the smp_rmb in mmu_invalidate_retry(). +*/ + kvm->mmu_invalidate_in_progress--; +} +#if defined(CONFIG_MMU_NOTIFIER ) && defined(KVM_ARCH_WANT_MMU_NOTIFIER )static inline struct kvm *mmu_notifier_to_kvm (struct mmu_notifier *mn) {@@ -714,45 +770,6 @@ static void kvm_mmu_notifier_change_pte(struct mmu_notifier *mn,kvm_handle_hva_range (mn, address, address +1 , pte, kvm_set_spte_gfn); } -void kvm_mmu_invalidate_begin (struct kvm *kvm) -{ -/* -* The count increase must become visible at unlock time as no -* spte can be established without taking the mmu_lock and -* count is also read inside the mmu_lock critical section. -*/ - kvm->mmu_invalidate_in_progress++; - -if (likely (kvm->mmu_invalidate_in_progress ==1 )) { - kvm->mmu_invalidate_range_start =INVALID_GPA ; - kvm->mmu_invalidate_range_end =INVALID_GPA ; - } -} - -void kvm_mmu_invalidate_range_add (struct kvm *kvm,gfn_t start,gfn_t end) -{ -WARN_ON_ONCE (!kvm->mmu_invalidate_in_progress); - -if (likely (kvm->mmu_invalidate_in_progress ==1 )) { - kvm->mmu_invalidate_range_start = start; - kvm->mmu_invalidate_range_end = end; - }else { -/* -* Fully tracking multiple concurrent ranges has diminishing -* returns. Keep things simple and just find the minimal range -* which includes the current and new ranges. As there won't be -* enough information to subtract a range after its invalidate -* completes, any ranges invalidated concurrently will -* accumulate and persist until all outstanding invalidates -* complete. -*/ - kvm->mmu_invalidate_range_start = -min (kvm->mmu_invalidate_range_start, start); - kvm->mmu_invalidate_range_end = -max (kvm->mmu_invalidate_range_end, end); - } -} -static bool kvm_mmu_unmap_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range) {kvm_mmu_invalidate_range_add (kvm, range->start, range->end);@@ -806,23 +823,6 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,return 0 ; } -void kvm_mmu_invalidate_end (struct kvm *kvm) -{ -/* -* This sequence increase will notify the kvm page fault that -* the page that is going to be mapped in the spte could have -* been freed. -*/ - kvm->mmu_invalidate_seq++; -smp_wmb (); -/* -* The above sequence increase must be visible before the -* below count decrease, which is ensured by the smp_wmb above -* in conjunction with the smp_rmb in mmu_invalidate_retry(). -*/ - kvm->mmu_invalidate_in_progress--; -} -static void kvm_mmu_notifier_invalidate_range_end (struct mmu_notifier *mn,const struct mmu_notifier_range *range) {@@ -1140,6 +1140,11 @@ int __weak kvm_arch_create_vm_debugfs(struct kvm *kvm)return 0 ; } +bool __weak kvm_arch_has_private_mem (struct kvm *kvm) +{ +return false; +} +static struct kvm *kvm_create_vm (unsigned long type,const char *fdname) {struct kvm *kvm =kvm_arch_alloc_vm ();@@ -2349,15 +2354,47 @@ static u64 kvm_supported_mem_attributes(struct kvm *kvm)return 0 ; } +static void kvm_unmap_mem_range (struct kvm *kvm,gfn_t start,gfn_t end) +{ +struct kvm_gfn_range gfn_range; +struct kvm_memory_slot *slot; +struct kvm_memslots *slots; +struct kvm_memslot_iter iter; +int i; +int r =0 ; + + gfn_range.pte =__pte (0 ); + gfn_range.may_block = true; + +for (i =0 ; i <KVM_ADDRESS_SPACE_NUM ; i++) { + slots =__kvm_memslots (kvm, i); + +kvm_for_each_memslot_in_gfn_range (&iter, slots, start, end) { + slot = iter.slot; + gfn_range.start =max (start, slot->base_gfn); + gfn_range.end =min (end, slot->base_gfn + slot->npages); +if (gfn_range.start >= gfn_range.end) +continue ; + gfn_range.slot = slot; + + r |=kvm_unmap_gfn_range (kvm, &gfn_range); + } + } + +if (r) +kvm_flush_remote_tlbs (kvm); +} +static int kvm_vm_ioctl_set_mem_attributes (struct kvm *kvm,struct kvm_memory_attributes *attrs) {gfn_t start, end;unsigned long i;void *entry; +int idx;u64 supported_attrs =kvm_supported_mem_attributes (kvm); -/* flags is currently not used. */ +/* 'flags' is currently not used. */ if (attrs->flags)return -EINVAL ;if (attrs->attributes & ~supported_attrs)virt/kvm/kvm_main.c static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, · line 2415
Quoted 7 lines
virt/kvm/kvm_main.c +4
-0@@ -2372,6 +2409,13 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, entry = attrs->attributes ?xa_mk_value (attrs->attributes) :NULL ; +if (kvm_arch_has_private_mem (kvm)) { +KVM_MMU_LOCK (kvm); +kvm_mmu_invalidate_begin (kvm); +kvm_mmu_invalidate_range_add (kvm, start, end);Nit: this works for
KVM_MEMORY_ATTRIBUTE_PRIVATE, but the invalidation should be necessary yet for attribute change of:
Quoted 8 lines
Quoted 3 lines
KVM_MEMORY_ATTRIBUTE_READKVM_MEMORY_ATTRIBUTE_WRITEKVM_MEMORY_ATTRIBUTE_EXECUTEThe unmapping is only needed for confidential usages which uses
KVM_MEMORY_ATTRIBUTE_PRIVATEonly and the other flags are defined here for other usages like pKVM. As Fuad commented in a different reply, pKVM supports in-place remapping and unmapping is unnecessary.
Ah, I see. It's fine to me, thanks.
Quoted 34 lines
Thanks,
ChaoQuoted 31 lines
Quoted 8 lines
virt/kvm/kvm_main.c +3
-0+KVM_MMU_UNLOCK (kvm); + } +mutex_lock (&kvm->lock);for (i = start; i < end; i++) if (xa_err(xa_store(&kvm->mem_attr_array, i, entry,virt/kvm/kvm_main.c static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, · line 2431
Quoted 9 lines
virt/kvm/kvm_main.c +6
-0@@ -2379,6 +2423,16 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,break ;mutex_unlock (&kvm->lock); +if (kvm_arch_has_private_mem (kvm)) { + idx =srcu_read_lock (&kvm->srcu); +KVM_MMU_LOCK (kvm); +if (i > start) +kvm_unmap_mem_range (kvm, start, i); +kvm_mmu_invalidate_end (kvm);Ditto.
Quoted 11 lines
virt/kvm/kvm_main.c +4
-1+KVM_MMU_UNLOCK (kvm); +srcu_read_unlock (&kvm->srcu, idx); + } + attrs->address = i <<PAGE_SHIFT ; attrs->size = (end - i) <<PAGE_SHIFT ; --2.25.1
Re: [PATCH v10 6/9] KVM: Unmap existing mappings when change the memory attributes
In reply to Chao PengHi,
Quoted 10 lines · Chao Peng
Unmap the existing guest mappings when memory attribute is changed between shared and private. This is needed because shared pages and private pages are from different backends, unmapping existing ones gives a chance for page fault handler to re-populate the mappings according to the new attribute.
Only architecture has private memory support needs this and the supported architecture is expected to rewrite the weak
kvm_arch_has_private_mem().
This kind of ties into the discussion of being able to share memory in place. For pKVM for example, shared and private memory would have the same backend, and the unmapping wouldn't be needed.
So I guess that, instead of kvm_arch_has_private_mem(), can the check be done differently, e.g., with a different function, say kvm_arch_private_notify_attribute_change() (but maybe with a more friendly name than what I suggested :) )?
Thanks,
/fuad
Quoted 287 lines
Also, during memory attribute changing and the unmapping time frame, page fault handler may happen in the same memory range and can cause incorrect page state, invoke kvm_mmu_invalidate_* helpers to let the page fault handler retry during this time frame.
Signed-off-by Chao Peng <[email protected]>--- include/linux/kvm_host.h | 7 +- virt/kvm/kvm_main.c | 168 ++++++++++++++++++++++++++------------- 2 files changed, 116 insertions(+), 59 deletions(-)include/linux/kvm_host.h +5
-2@@ -255,7 +255,6 @@ bool kvm_setup_async_pf(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,int kvm_async_pf_wakeup_all (struct kvm_vcpu *vcpu);#endif -#ifdef KVM_ARCH_WANT_MMU_NOTIFIERstruct kvm_gfn_range {struct kvm_memory_slot *slot;gfn_t start;@@ -264,6 +263,8 @@ struct kvm_gfn_range {bool may_block; };bool kvm_unmap_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range); + +#ifdef KVM_ARCH_WANT_MMU_NOTIFIER bool kvm_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range);bool kvm_test_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range);bool kvm_set_spte_gfn (struct kvm *kvm,struct kvm_gfn_range *range);@@ -785,11 +786,12 @@ struct kvm {#if defined(CONFIG_MMU_NOTIFIER ) && defined(KVM_ARCH_WANT_MMU_NOTIFIER )struct mmu_notifier mmu_notifier; +#endif unsigned long mmu_invalidate_seq;long mmu_invalidate_in_progress;gfn_t mmu_invalidate_range_start;gfn_t mmu_invalidate_range_end; -#endif +struct list_head devices;u64 manual_dirty_log_protect;struct dentry *debugfs_dentry;@@ -1480,6 +1482,7 @@ bool kvm_arch_dy_has_pending_interrupt(struct kvm_vcpu *vcpu);int kvm_arch_post_init_vm (struct kvm *kvm);void kvm_arch_pre_destroy_vm (struct kvm *kvm);int kvm_arch_create_vm_debugfs (struct kvm *kvm); +bool kvm_arch_has_private_mem (struct kvm *kvm);#ifndef __KVM_HAVE_ARCH_VM_ALLOC/* virt/kvm/kvm_main.c +111
-58@@ -520,6 +520,62 @@ void kvm_destroy_vcpus(struct kvm *kvm) }EXPORT_SYMBOL_GPL (kvm_destroy_vcpus); +void kvm_mmu_invalidate_begin (struct kvm *kvm) +{ +/* +* The count increase must become visible at unlock time as no +* spte can be established without taking the mmu_lock and +* count is also read inside the mmu_lock critical section. +*/ + kvm->mmu_invalidate_in_progress++; + +if (likely (kvm->mmu_invalidate_in_progress ==1 )) { + kvm->mmu_invalidate_range_start =INVALID_GPA ; + kvm->mmu_invalidate_range_end =INVALID_GPA ; + } +} + +void kvm_mmu_invalidate_range_add (struct kvm *kvm,gfn_t start,gfn_t end) +{ +WARN_ON_ONCE (!kvm->mmu_invalidate_in_progress); + +if (likely (kvm->mmu_invalidate_in_progress ==1 )) { + kvm->mmu_invalidate_range_start = start; + kvm->mmu_invalidate_range_end = end; + }else { +/* +* Fully tracking multiple concurrent ranges has diminishing +* returns. Keep things simple and just find the minimal range +* which includes the current and new ranges. As there won't be +* enough information to subtract a range after its invalidate +* completes, any ranges invalidated concurrently will +* accumulate and persist until all outstanding invalidates +* complete. +*/ + kvm->mmu_invalidate_range_start = +min (kvm->mmu_invalidate_range_start, start); + kvm->mmu_invalidate_range_end = +max (kvm->mmu_invalidate_range_end, end); + } +} + +void kvm_mmu_invalidate_end (struct kvm *kvm) +{ +/* +* This sequence increase will notify the kvm page fault that +* the page that is going to be mapped in the spte could have +* been freed. +*/ + kvm->mmu_invalidate_seq++; +smp_wmb (); +/* +* The above sequence increase must be visible before the +* below count decrease, which is ensured by the smp_wmb above +* in conjunction with the smp_rmb in mmu_invalidate_retry(). +*/ + kvm->mmu_invalidate_in_progress--; +} +#if defined(CONFIG_MMU_NOTIFIER ) && defined(KVM_ARCH_WANT_MMU_NOTIFIER )static inline struct kvm *mmu_notifier_to_kvm (struct mmu_notifier *mn) {@@ -714,45 +770,6 @@ static void kvm_mmu_notifier_change_pte(struct mmu_notifier *mn,kvm_handle_hva_range (mn, address, address +1 , pte, kvm_set_spte_gfn); } -void kvm_mmu_invalidate_begin (struct kvm *kvm) -{ -/* -* The count increase must become visible at unlock time as no -* spte can be established without taking the mmu_lock and -* count is also read inside the mmu_lock critical section. -*/ - kvm->mmu_invalidate_in_progress++; - -if (likely (kvm->mmu_invalidate_in_progress ==1 )) { - kvm->mmu_invalidate_range_start =INVALID_GPA ; - kvm->mmu_invalidate_range_end =INVALID_GPA ; - } -} - -void kvm_mmu_invalidate_range_add (struct kvm *kvm,gfn_t start,gfn_t end) -{ -WARN_ON_ONCE (!kvm->mmu_invalidate_in_progress); - -if (likely (kvm->mmu_invalidate_in_progress ==1 )) { - kvm->mmu_invalidate_range_start = start; - kvm->mmu_invalidate_range_end = end; - }else { -/* -* Fully tracking multiple concurrent ranges has diminishing -* returns. Keep things simple and just find the minimal range -* which includes the current and new ranges. As there won't be -* enough information to subtract a range after its invalidate -* completes, any ranges invalidated concurrently will -* accumulate and persist until all outstanding invalidates -* complete. -*/ - kvm->mmu_invalidate_range_start = -min (kvm->mmu_invalidate_range_start, start); - kvm->mmu_invalidate_range_end = -max (kvm->mmu_invalidate_range_end, end); - } -} -static bool kvm_mmu_unmap_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range) {kvm_mmu_invalidate_range_add (kvm, range->start, range->end);@@ -806,23 +823,6 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,return 0 ; } -void kvm_mmu_invalidate_end (struct kvm *kvm) -{ -/* -* This sequence increase will notify the kvm page fault that -* the page that is going to be mapped in the spte could have -* been freed. -*/ - kvm->mmu_invalidate_seq++; -smp_wmb (); -/* -* The above sequence increase must be visible before the -* below count decrease, which is ensured by the smp_wmb above -* in conjunction with the smp_rmb in mmu_invalidate_retry(). -*/ - kvm->mmu_invalidate_in_progress--; -} -static void kvm_mmu_notifier_invalidate_range_end (struct mmu_notifier *mn,const struct mmu_notifier_range *range) {@@ -1140,6 +1140,11 @@ int __weak kvm_arch_create_vm_debugfs(struct kvm *kvm)return 0 ; } +bool __weak kvm_arch_has_private_mem (struct kvm *kvm) +{ +return false; +} +static struct kvm *kvm_create_vm (unsigned long type,const char *fdname) {struct kvm *kvm =kvm_arch_alloc_vm ();@@ -2349,15 +2354,47 @@ static u64 kvm_supported_mem_attributes(struct kvm *kvm)return 0 ; } +static void kvm_unmap_mem_range (struct kvm *kvm,gfn_t start,gfn_t end) +{ +struct kvm_gfn_range gfn_range; +struct kvm_memory_slot *slot; +struct kvm_memslots *slots; +struct kvm_memslot_iter iter; +int i; +int r =0 ; + + gfn_range.pte =__pte (0 ); + gfn_range.may_block = true; + +for (i =0 ; i <KVM_ADDRESS_SPACE_NUM ; i++) { + slots =__kvm_memslots (kvm, i); + +kvm_for_each_memslot_in_gfn_range (&iter, slots, start, end) { + slot = iter.slot; + gfn_range.start =max (start, slot->base_gfn); + gfn_range.end =min (end, slot->base_gfn + slot->npages); +if (gfn_range.start >= gfn_range.end) +continue ; + gfn_range.slot = slot; + + r |=kvm_unmap_gfn_range (kvm, &gfn_range); + } + } + +if (r) +kvm_flush_remote_tlbs (kvm); +} +static int kvm_vm_ioctl_set_mem_attributes (struct kvm *kvm,struct kvm_memory_attributes *attrs) {gfn_t start, end;unsigned long i;void *entry; +int idx;u64 supported_attrs =kvm_supported_mem_attributes (kvm); -/* flags is currently not used. */ +/* 'flags' is currently not used. */ if (attrs->flags)return -EINVAL ;if (attrs->attributes & ~supported_attrs)@@ -2372,6 +2409,13 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, entry = attrs->attributes ?xa_mk_value (attrs->attributes) :NULL ; +if (kvm_arch_has_private_mem (kvm)) { +KVM_MMU_LOCK (kvm); +kvm_mmu_invalidate_begin (kvm); +kvm_mmu_invalidate_range_add (kvm, start, end); +KVM_MMU_UNLOCK (kvm); + } +mutex_lock (&kvm->lock);for (i = start; i < end; i++) if (xa_err(xa_store(&kvm->mem_attr_array, i, entry,@@ -2379,6 +2423,16 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,break ;mutex_unlock (&kvm->lock); +if (kvm_arch_has_private_mem (kvm)) { + idx =srcu_read_lock (&kvm->srcu); +KVM_MMU_LOCK (kvm); +if (i > start) +kvm_unmap_mem_range (kvm, start, i); +kvm_mmu_invalidate_end (kvm); +KVM_MMU_UNLOCK (kvm); +srcu_read_unlock (&kvm->srcu, idx); + } + attrs->address = i <<PAGE_SHIFT ; attrs->size = (end - i) <<PAGE_SHIFT ; --2.25.1
Re: [PATCH v10 6/9] KVM: Unmap existing mappings when change the memory attributes
Quoted 14 lines · Fuad Tabba
Hi,
Quoted 10 lines · Chao Peng
Unmap the existing guest mappings when memory attribute is changed between shared and private. This is needed because shared pages and private pages are from different backends, unmapping existing ones gives a chance for page fault handler to re-populate the mappings according to the new attribute.
Only architecture has private memory support needs this and the supported architecture is expected to rewrite the weak
kvm_arch_has_private_mem().
Quoted 8 lines
This kind of ties into the discussion of being able to share memory in place. For pKVM for example, shared and private memory would have the same backend, and the unmapping wouldn't be needed.
So I guess that, instead of
kvm_arch_has_private_mem(), can the check be done differently, e.g., with a different function, saykvm_arch_private_notify_attribute_change()(but maybe with a more friendly name than what I suggested :) )?
Besides controlling the unmapping here, kvm_arch_has_private_mem() is also used to gate the memslot KVM_MEM_PRIVATE flag in patch09. I know unmapping is confirmed unnecessary for pKVM, but how about KVM_MEM_PRIVATE? Will pKVM add its own flag or reuse KVM_MEM_PRIVATE? If the answer is the latter, then yes we should use a different check which only works for confidential usages here.
Thanks,
Chao
Quoted 291 lines
Thanks,
/fuadQuoted 287 lines
Also, during memory attribute changing and the unmapping time frame, page fault handler may happen in the same memory range and can cause incorrect page state, invoke kvm_mmu_invalidate_* helpers to let the page fault handler retry during this time frame.
Signed-off-by Chao Peng <[email protected]>--- include/linux/kvm_host.h | 7 +- virt/kvm/kvm_main.c | 168 ++++++++++++++++++++++++++------------- 2 files changed, 116 insertions(+), 59 deletions(-)include/linux/kvm_host.h +5
-2@@ -255,7 +255,6 @@ bool kvm_setup_async_pf(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,int kvm_async_pf_wakeup_all (struct kvm_vcpu *vcpu);#endif -#ifdef KVM_ARCH_WANT_MMU_NOTIFIERstruct kvm_gfn_range {struct kvm_memory_slot *slot;gfn_t start;@@ -264,6 +263,8 @@ struct kvm_gfn_range {bool may_block; };bool kvm_unmap_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range); + +#ifdef KVM_ARCH_WANT_MMU_NOTIFIER bool kvm_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range);bool kvm_test_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range);bool kvm_set_spte_gfn (struct kvm *kvm,struct kvm_gfn_range *range);@@ -785,11 +786,12 @@ struct kvm {#if defined(CONFIG_MMU_NOTIFIER ) && defined(KVM_ARCH_WANT_MMU_NOTIFIER )struct mmu_notifier mmu_notifier; +#endif unsigned long mmu_invalidate_seq;long mmu_invalidate_in_progress;gfn_t mmu_invalidate_range_start;gfn_t mmu_invalidate_range_end; -#endif +struct list_head devices;u64 manual_dirty_log_protect;struct dentry *debugfs_dentry;@@ -1480,6 +1482,7 @@ bool kvm_arch_dy_has_pending_interrupt(struct kvm_vcpu *vcpu);int kvm_arch_post_init_vm (struct kvm *kvm);void kvm_arch_pre_destroy_vm (struct kvm *kvm);int kvm_arch_create_vm_debugfs (struct kvm *kvm); +bool kvm_arch_has_private_mem (struct kvm *kvm);#ifndef __KVM_HAVE_ARCH_VM_ALLOC/* virt/kvm/kvm_main.c +111
-58@@ -520,6 +520,62 @@ void kvm_destroy_vcpus(struct kvm *kvm) }EXPORT_SYMBOL_GPL (kvm_destroy_vcpus); +void kvm_mmu_invalidate_begin (struct kvm *kvm) +{ +/* +* The count increase must become visible at unlock time as no +* spte can be established without taking the mmu_lock and +* count is also read inside the mmu_lock critical section. +*/ + kvm->mmu_invalidate_in_progress++; + +if (likely (kvm->mmu_invalidate_in_progress ==1 )) { + kvm->mmu_invalidate_range_start =INVALID_GPA ; + kvm->mmu_invalidate_range_end =INVALID_GPA ; + } +} + +void kvm_mmu_invalidate_range_add (struct kvm *kvm,gfn_t start,gfn_t end) +{ +WARN_ON_ONCE (!kvm->mmu_invalidate_in_progress); + +if (likely (kvm->mmu_invalidate_in_progress ==1 )) { + kvm->mmu_invalidate_range_start = start; + kvm->mmu_invalidate_range_end = end; + }else { +/* +* Fully tracking multiple concurrent ranges has diminishing +* returns. Keep things simple and just find the minimal range +* which includes the current and new ranges. As there won't be +* enough information to subtract a range after its invalidate +* completes, any ranges invalidated concurrently will +* accumulate and persist until all outstanding invalidates +* complete. +*/ + kvm->mmu_invalidate_range_start = +min (kvm->mmu_invalidate_range_start, start); + kvm->mmu_invalidate_range_end = +max (kvm->mmu_invalidate_range_end, end); + } +} + +void kvm_mmu_invalidate_end (struct kvm *kvm) +{ +/* +* This sequence increase will notify the kvm page fault that +* the page that is going to be mapped in the spte could have +* been freed. +*/ + kvm->mmu_invalidate_seq++; +smp_wmb (); +/* +* The above sequence increase must be visible before the +* below count decrease, which is ensured by the smp_wmb above +* in conjunction with the smp_rmb in mmu_invalidate_retry(). +*/ + kvm->mmu_invalidate_in_progress--; +} +#if defined(CONFIG_MMU_NOTIFIER ) && defined(KVM_ARCH_WANT_MMU_NOTIFIER )static inline struct kvm *mmu_notifier_to_kvm (struct mmu_notifier *mn) {@@ -714,45 +770,6 @@ static void kvm_mmu_notifier_change_pte(struct mmu_notifier *mn,kvm_handle_hva_range (mn, address, address +1 , pte, kvm_set_spte_gfn); } -void kvm_mmu_invalidate_begin (struct kvm *kvm) -{ -/* -* The count increase must become visible at unlock time as no -* spte can be established without taking the mmu_lock and -* count is also read inside the mmu_lock critical section. -*/ - kvm->mmu_invalidate_in_progress++; - -if (likely (kvm->mmu_invalidate_in_progress ==1 )) { - kvm->mmu_invalidate_range_start =INVALID_GPA ; - kvm->mmu_invalidate_range_end =INVALID_GPA ; - } -} - -void kvm_mmu_invalidate_range_add (struct kvm *kvm,gfn_t start,gfn_t end) -{ -WARN_ON_ONCE (!kvm->mmu_invalidate_in_progress); - -if (likely (kvm->mmu_invalidate_in_progress ==1 )) { - kvm->mmu_invalidate_range_start = start; - kvm->mmu_invalidate_range_end = end; - }else { -/* -* Fully tracking multiple concurrent ranges has diminishing -* returns. Keep things simple and just find the minimal range -* which includes the current and new ranges. As there won't be -* enough information to subtract a range after its invalidate -* completes, any ranges invalidated concurrently will -* accumulate and persist until all outstanding invalidates -* complete. -*/ - kvm->mmu_invalidate_range_start = -min (kvm->mmu_invalidate_range_start, start); - kvm->mmu_invalidate_range_end = -max (kvm->mmu_invalidate_range_end, end); - } -} -static bool kvm_mmu_unmap_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range) {kvm_mmu_invalidate_range_add (kvm, range->start, range->end);@@ -806,23 +823,6 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,return 0 ; } -void kvm_mmu_invalidate_end (struct kvm *kvm) -{ -/* -* This sequence increase will notify the kvm page fault that -* the page that is going to be mapped in the spte could have -* been freed. -*/ - kvm->mmu_invalidate_seq++; -smp_wmb (); -/* -* The above sequence increase must be visible before the -* below count decrease, which is ensured by the smp_wmb above -* in conjunction with the smp_rmb in mmu_invalidate_retry(). -*/ - kvm->mmu_invalidate_in_progress--; -} -static void kvm_mmu_notifier_invalidate_range_end (struct mmu_notifier *mn,const struct mmu_notifier_range *range) {@@ -1140,6 +1140,11 @@ int __weak kvm_arch_create_vm_debugfs(struct kvm *kvm)return 0 ; } +bool __weak kvm_arch_has_private_mem (struct kvm *kvm) +{ +return false; +} +static struct kvm *kvm_create_vm (unsigned long type,const char *fdname) {struct kvm *kvm =kvm_arch_alloc_vm ();@@ -2349,15 +2354,47 @@ static u64 kvm_supported_mem_attributes(struct kvm *kvm)return 0 ; } +static void kvm_unmap_mem_range (struct kvm *kvm,gfn_t start,gfn_t end) +{ +struct kvm_gfn_range gfn_range; +struct kvm_memory_slot *slot; +struct kvm_memslots *slots; +struct kvm_memslot_iter iter; +int i; +int r =0 ; + + gfn_range.pte =__pte (0 ); + gfn_range.may_block = true; + +for (i =0 ; i <KVM_ADDRESS_SPACE_NUM ; i++) { + slots =__kvm_memslots (kvm, i); + +kvm_for_each_memslot_in_gfn_range (&iter, slots, start, end) { + slot = iter.slot; + gfn_range.start =max (start, slot->base_gfn); + gfn_range.end =min (end, slot->base_gfn + slot->npages); +if (gfn_range.start >= gfn_range.end) +continue ; + gfn_range.slot = slot; + + r |=kvm_unmap_gfn_range (kvm, &gfn_range); + } + } + +if (r) +kvm_flush_remote_tlbs (kvm); +} +static int kvm_vm_ioctl_set_mem_attributes (struct kvm *kvm,struct kvm_memory_attributes *attrs) {gfn_t start, end;unsigned long i;void *entry; +int idx;u64 supported_attrs =kvm_supported_mem_attributes (kvm); -/* flags is currently not used. */ +/* 'flags' is currently not used. */ if (attrs->flags)return -EINVAL ;if (attrs->attributes & ~supported_attrs)@@ -2372,6 +2409,13 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, entry = attrs->attributes ?xa_mk_value (attrs->attributes) :NULL ; +if (kvm_arch_has_private_mem (kvm)) { +KVM_MMU_LOCK (kvm); +kvm_mmu_invalidate_begin (kvm); +kvm_mmu_invalidate_range_add (kvm, start, end); +KVM_MMU_UNLOCK (kvm); + } +mutex_lock (&kvm->lock);for (i = start; i < end; i++) if (xa_err(xa_store(&kvm->mem_attr_array, i, entry,@@ -2379,6 +2423,16 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,break ;mutex_unlock (&kvm->lock); +if (kvm_arch_has_private_mem (kvm)) { + idx =srcu_read_lock (&kvm->srcu); +KVM_MMU_LOCK (kvm); +if (i > start) +kvm_unmap_mem_range (kvm, start, i); +kvm_mmu_invalidate_end (kvm); +KVM_MMU_UNLOCK (kvm); +srcu_read_unlock (&kvm->srcu, idx); + } + attrs->address = i <<PAGE_SHIFT ; attrs->size = (end - i) <<PAGE_SHIFT ; --2.25.1
Re: [PATCH v10 6/9] KVM: Unmap existing mappings when change the memory attributes
Hi,
Quoted 25 lines · Chao Peng
Quoted 14 lines · Fuad Tabba
Hi,
Quoted 10 lines · Chao Peng
Unmap the existing guest mappings when memory attribute is changed between shared and private. This is needed because shared pages and private pages are from different backends, unmapping existing ones gives a chance for page fault handler to re-populate the mappings according to the new attribute.
Only architecture has private memory support needs this and the supported architecture is expected to rewrite the weak
kvm_arch_has_private_mem().Quoted 8 lines
This kind of ties into the discussion of being able to share memory in place. For pKVM for example, shared and private memory would have the same backend, and the unmapping wouldn't be needed.
So I guess that, instead of
kvm_arch_has_private_mem(), can the check be done differently, e.g., with a different function, saykvm_arch_private_notify_attribute_change()(but maybe with a more friendly name than what I suggested :) )?
Quoted 6 lines
Besides controlling the unmapping here,
kvm_arch_has_private_mem()is also used to gate the memslotKVM_MEM_PRIVATEflag in patch09. I know unmapping is confirmed unnecessary for pKVM, but how aboutKVM_MEM_PRIVATE? Will pKVM add its own flag or reuseKVM_MEM_PRIVATE? If the answer is the latter, then yes we should use a different check which only works for confidential usages here.
I think it makes sense for pKVM to use the same flag (KVM_MEM_PRIVATE) and not to add another one.
Thank you,
/fuad
Quoted 294 lines
Thanks,
ChaoQuoted 291 lines
Thanks,
/fuadQuoted 287 lines
Also, during memory attribute changing and the unmapping time frame, page fault handler may happen in the same memory range and can cause incorrect page state, invoke kvm_mmu_invalidate_* helpers to let the page fault handler retry during this time frame.
Signed-off-by Chao Peng <[email protected]>--- include/linux/kvm_host.h | 7 +- virt/kvm/kvm_main.c | 168 ++++++++++++++++++++++++++------------- 2 files changed, 116 insertions(+), 59 deletions(-)include/linux/kvm_host.h +5
-2@@ -255,7 +255,6 @@ bool kvm_setup_async_pf(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,int kvm_async_pf_wakeup_all (struct kvm_vcpu *vcpu);#endif -#ifdef KVM_ARCH_WANT_MMU_NOTIFIERstruct kvm_gfn_range {struct kvm_memory_slot *slot;gfn_t start;@@ -264,6 +263,8 @@ struct kvm_gfn_range {bool may_block; };bool kvm_unmap_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range); + +#ifdef KVM_ARCH_WANT_MMU_NOTIFIER bool kvm_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range);bool kvm_test_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range);bool kvm_set_spte_gfn (struct kvm *kvm,struct kvm_gfn_range *range);@@ -785,11 +786,12 @@ struct kvm {#if defined(CONFIG_MMU_NOTIFIER ) && defined(KVM_ARCH_WANT_MMU_NOTIFIER )struct mmu_notifier mmu_notifier; +#endif unsigned long mmu_invalidate_seq;long mmu_invalidate_in_progress;gfn_t mmu_invalidate_range_start;gfn_t mmu_invalidate_range_end; -#endif +struct list_head devices;u64 manual_dirty_log_protect;struct dentry *debugfs_dentry;@@ -1480,6 +1482,7 @@ bool kvm_arch_dy_has_pending_interrupt(struct kvm_vcpu *vcpu);int kvm_arch_post_init_vm (struct kvm *kvm);void kvm_arch_pre_destroy_vm (struct kvm *kvm);int kvm_arch_create_vm_debugfs (struct kvm *kvm); +bool kvm_arch_has_private_mem (struct kvm *kvm);#ifndef __KVM_HAVE_ARCH_VM_ALLOC/* virt/kvm/kvm_main.c +111
-58@@ -520,6 +520,62 @@ void kvm_destroy_vcpus(struct kvm *kvm) }EXPORT_SYMBOL_GPL (kvm_destroy_vcpus); +void kvm_mmu_invalidate_begin (struct kvm *kvm) +{ +/* +* The count increase must become visible at unlock time as no +* spte can be established without taking the mmu_lock and +* count is also read inside the mmu_lock critical section. +*/ + kvm->mmu_invalidate_in_progress++; + +if (likely (kvm->mmu_invalidate_in_progress ==1 )) { + kvm->mmu_invalidate_range_start =INVALID_GPA ; + kvm->mmu_invalidate_range_end =INVALID_GPA ; + } +} + +void kvm_mmu_invalidate_range_add (struct kvm *kvm,gfn_t start,gfn_t end) +{ +WARN_ON_ONCE (!kvm->mmu_invalidate_in_progress); + +if (likely (kvm->mmu_invalidate_in_progress ==1 )) { + kvm->mmu_invalidate_range_start = start; + kvm->mmu_invalidate_range_end = end; + }else { +/* +* Fully tracking multiple concurrent ranges has diminishing +* returns. Keep things simple and just find the minimal range +* which includes the current and new ranges. As there won't be +* enough information to subtract a range after its invalidate +* completes, any ranges invalidated concurrently will +* accumulate and persist until all outstanding invalidates +* complete. +*/ + kvm->mmu_invalidate_range_start = +min (kvm->mmu_invalidate_range_start, start); + kvm->mmu_invalidate_range_end = +max (kvm->mmu_invalidate_range_end, end); + } +} + +void kvm_mmu_invalidate_end (struct kvm *kvm) +{ +/* +* This sequence increase will notify the kvm page fault that +* the page that is going to be mapped in the spte could have +* been freed. +*/ + kvm->mmu_invalidate_seq++; +smp_wmb (); +/* +* The above sequence increase must be visible before the +* below count decrease, which is ensured by the smp_wmb above +* in conjunction with the smp_rmb in mmu_invalidate_retry(). +*/ + kvm->mmu_invalidate_in_progress--; +} +#if defined(CONFIG_MMU_NOTIFIER ) && defined(KVM_ARCH_WANT_MMU_NOTIFIER )static inline struct kvm *mmu_notifier_to_kvm (struct mmu_notifier *mn) {@@ -714,45 +770,6 @@ static void kvm_mmu_notifier_change_pte(struct mmu_notifier *mn,kvm_handle_hva_range (mn, address, address +1 , pte, kvm_set_spte_gfn); } -void kvm_mmu_invalidate_begin (struct kvm *kvm) -{ -/* -* The count increase must become visible at unlock time as no -* spte can be established without taking the mmu_lock and -* count is also read inside the mmu_lock critical section. -*/ - kvm->mmu_invalidate_in_progress++; - -if (likely (kvm->mmu_invalidate_in_progress ==1 )) { - kvm->mmu_invalidate_range_start =INVALID_GPA ; - kvm->mmu_invalidate_range_end =INVALID_GPA ; - } -} - -void kvm_mmu_invalidate_range_add (struct kvm *kvm,gfn_t start,gfn_t end) -{ -WARN_ON_ONCE (!kvm->mmu_invalidate_in_progress); - -if (likely (kvm->mmu_invalidate_in_progress ==1 )) { - kvm->mmu_invalidate_range_start = start; - kvm->mmu_invalidate_range_end = end; - }else { -/* -* Fully tracking multiple concurrent ranges has diminishing -* returns. Keep things simple and just find the minimal range -* which includes the current and new ranges. As there won't be -* enough information to subtract a range after its invalidate -* completes, any ranges invalidated concurrently will -* accumulate and persist until all outstanding invalidates -* complete. -*/ - kvm->mmu_invalidate_range_start = -min (kvm->mmu_invalidate_range_start, start); - kvm->mmu_invalidate_range_end = -max (kvm->mmu_invalidate_range_end, end); - } -} -static bool kvm_mmu_unmap_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range) {kvm_mmu_invalidate_range_add (kvm, range->start, range->end);@@ -806,23 +823,6 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,return 0 ; } -void kvm_mmu_invalidate_end (struct kvm *kvm) -{ -/* -* This sequence increase will notify the kvm page fault that -* the page that is going to be mapped in the spte could have -* been freed. -*/ - kvm->mmu_invalidate_seq++; -smp_wmb (); -/* -* The above sequence increase must be visible before the -* below count decrease, which is ensured by the smp_wmb above -* in conjunction with the smp_rmb in mmu_invalidate_retry(). -*/ - kvm->mmu_invalidate_in_progress--; -} -static void kvm_mmu_notifier_invalidate_range_end (struct mmu_notifier *mn,const struct mmu_notifier_range *range) {@@ -1140,6 +1140,11 @@ int __weak kvm_arch_create_vm_debugfs(struct kvm *kvm)return 0 ; } +bool __weak kvm_arch_has_private_mem (struct kvm *kvm) +{ +return false; +} +static struct kvm *kvm_create_vm (unsigned long type,const char *fdname) {struct kvm *kvm =kvm_arch_alloc_vm ();@@ -2349,15 +2354,47 @@ static u64 kvm_supported_mem_attributes(struct kvm *kvm)return 0 ; } +static void kvm_unmap_mem_range (struct kvm *kvm,gfn_t start,gfn_t end) +{ +struct kvm_gfn_range gfn_range; +struct kvm_memory_slot *slot; +struct kvm_memslots *slots; +struct kvm_memslot_iter iter; +int i; +int r =0 ; + + gfn_range.pte =__pte (0 ); + gfn_range.may_block = true; + +for (i =0 ; i <KVM_ADDRESS_SPACE_NUM ; i++) { + slots =__kvm_memslots (kvm, i); + +kvm_for_each_memslot_in_gfn_range (&iter, slots, start, end) { + slot = iter.slot; + gfn_range.start =max (start, slot->base_gfn); + gfn_range.end =min (end, slot->base_gfn + slot->npages); +if (gfn_range.start >= gfn_range.end) +continue ; + gfn_range.slot = slot; + + r |=kvm_unmap_gfn_range (kvm, &gfn_range); + } + } + +if (r) +kvm_flush_remote_tlbs (kvm); +} +static int kvm_vm_ioctl_set_mem_attributes (struct kvm *kvm,struct kvm_memory_attributes *attrs) {gfn_t start, end;unsigned long i;void *entry; +int idx;u64 supported_attrs =kvm_supported_mem_attributes (kvm); -/* flags is currently not used. */ +/* 'flags' is currently not used. */ if (attrs->flags)return -EINVAL ;if (attrs->attributes & ~supported_attrs)@@ -2372,6 +2409,13 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, entry = attrs->attributes ?xa_mk_value (attrs->attributes) :NULL ; +if (kvm_arch_has_private_mem (kvm)) { +KVM_MMU_LOCK (kvm); +kvm_mmu_invalidate_begin (kvm); +kvm_mmu_invalidate_range_add (kvm, start, end); +KVM_MMU_UNLOCK (kvm); + } +mutex_lock (&kvm->lock);for (i = start; i < end; i++) if (xa_err(xa_store(&kvm->mem_attr_array, i, entry,@@ -2379,6 +2423,16 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,break ;mutex_unlock (&kvm->lock); +if (kvm_arch_has_private_mem (kvm)) { + idx =srcu_read_lock (&kvm->srcu); +KVM_MMU_LOCK (kvm); +if (i > start) +kvm_unmap_mem_range (kvm, start, i); +kvm_mmu_invalidate_end (kvm); +KVM_MMU_UNLOCK (kvm); +srcu_read_unlock (&kvm->srcu, idx); + } + attrs->address = i <<PAGE_SHIFT ; attrs->size = (end - i) <<PAGE_SHIFT ; --2.25.1
Re: [PATCH v10 6/9] KVM: Unmap existing mappings when change the memory attributes
Quoted 35 lines · Fuad Tabba
Hi,
Quoted 25 lines · Chao Peng
Quoted 14 lines · Fuad Tabba
Hi,
Quoted 10 lines · Chao Peng
Unmap the existing guest mappings when memory attribute is changed between shared and private. This is needed because shared pages and private pages are from different backends, unmapping existing ones gives a chance for page fault handler to re-populate the mappings according to the new attribute.
Only architecture has private memory support needs this and the supported architecture is expected to rewrite the weak
kvm_arch_has_private_mem().Quoted 8 lines
This kind of ties into the discussion of being able to share memory in place. For pKVM for example, shared and private memory would have the same backend, and the unmapping wouldn't be needed.
So I guess that, instead of
kvm_arch_has_private_mem(), can the check be done differently, e.g., with a different function, saykvm_arch_private_notify_attribute_change()(but maybe with a more friendly name than what I suggested :) )?Quoted 6 lines
Besides controlling the unmapping here,
kvm_arch_has_private_mem()is also used to gate the memslotKVM_MEM_PRIVATEflag in patch09. I know unmapping is confirmed unnecessary for pKVM, but how aboutKVM_MEM_PRIVATE? Will pKVM add its own flag or reuseKVM_MEM_PRIVATE? If the answer is the latter, then yes we should use a different check which only works for confidential usages here.
Quoted 2 lines
I think it makes sense for pKVM to use the same flag (
KVM_MEM_PRIVATE) and not to add another one.
Thanks for the reply.
Chao
Quoted 300 lines
Thank you,
/fuadQuoted 294 lines
Thanks,
ChaoQuoted 291 lines
Thanks,
/fuadQuoted 287 lines
Also, during memory attribute changing and the unmapping time frame, page fault handler may happen in the same memory range and can cause incorrect page state, invoke kvm_mmu_invalidate_* helpers to let the page fault handler retry during this time frame.
Signed-off-by Chao Peng <[email protected]>--- include/linux/kvm_host.h | 7 +- virt/kvm/kvm_main.c | 168 ++++++++++++++++++++++++++------------- 2 files changed, 116 insertions(+), 59 deletions(-)include/linux/kvm_host.h +5
-2@@ -255,7 +255,6 @@ bool kvm_setup_async_pf(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,int kvm_async_pf_wakeup_all (struct kvm_vcpu *vcpu);#endif -#ifdef KVM_ARCH_WANT_MMU_NOTIFIERstruct kvm_gfn_range {struct kvm_memory_slot *slot;gfn_t start;@@ -264,6 +263,8 @@ struct kvm_gfn_range {bool may_block; };bool kvm_unmap_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range); + +#ifdef KVM_ARCH_WANT_MMU_NOTIFIER bool kvm_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range);bool kvm_test_age_gfn (struct kvm *kvm,struct kvm_gfn_range *range);bool kvm_set_spte_gfn (struct kvm *kvm,struct kvm_gfn_range *range);@@ -785,11 +786,12 @@ struct kvm {#if defined(CONFIG_MMU_NOTIFIER ) && defined(KVM_ARCH_WANT_MMU_NOTIFIER )struct mmu_notifier mmu_notifier; +#endif unsigned long mmu_invalidate_seq;long mmu_invalidate_in_progress;gfn_t mmu_invalidate_range_start;gfn_t mmu_invalidate_range_end; -#endif +struct list_head devices;u64 manual_dirty_log_protect;struct dentry *debugfs_dentry;@@ -1480,6 +1482,7 @@ bool kvm_arch_dy_has_pending_interrupt(struct kvm_vcpu *vcpu);int kvm_arch_post_init_vm (struct kvm *kvm);void kvm_arch_pre_destroy_vm (struct kvm *kvm);int kvm_arch_create_vm_debugfs (struct kvm *kvm); +bool kvm_arch_has_private_mem (struct kvm *kvm);#ifndef __KVM_HAVE_ARCH_VM_ALLOC/* virt/kvm/kvm_main.c +111
-58@@ -520,6 +520,62 @@ void kvm_destroy_vcpus(struct kvm *kvm) }EXPORT_SYMBOL_GPL (kvm_destroy_vcpus); +void kvm_mmu_invalidate_begin (struct kvm *kvm) +{ +/* +* The count increase must become visible at unlock time as no +* spte can be established without taking the mmu_lock and +* count is also read inside the mmu_lock critical section. +*/ + kvm->mmu_invalidate_in_progress++; + +if (likely (kvm->mmu_invalidate_in_progress ==1 )) { + kvm->mmu_invalidate_range_start =INVALID_GPA ; + kvm->mmu_invalidate_range_end =INVALID_GPA ; + } +} + +void kvm_mmu_invalidate_range_add (struct kvm *kvm,gfn_t start,gfn_t end) +{ +WARN_ON_ONCE (!kvm->mmu_invalidate_in_progress); + +if (likely (kvm->mmu_invalidate_in_progress ==1 )) { + kvm->mmu_invalidate_range_start = start; + kvm->mmu_invalidate_range_end = end; + }else { +/* +* Fully tracking multiple concurrent ranges has diminishing +* returns. Keep things simple and just find the minimal range +* which includes the current and new ranges. As there won't be +* enough information to subtract a range after its invalidate +* completes, any ranges invalidated concurrently will +* accumulate and persist until all outstanding invalidates +* complete. +*/ + kvm->mmu_invalidate_range_start = +min (kvm->mmu_invalidate_range_start, start); + kvm->mmu_invalidate_range_end = +max (kvm->mmu_invalidate_range_end, end); + } +} + +void kvm_mmu_invalidate_end (struct kvm *kvm) +{ +/* +* This sequence increase will notify the kvm page fault that +* the page that is going to be mapped in the spte could have +* been freed. +*/ + kvm->mmu_invalidate_seq++; +smp_wmb (); +/* +* The above sequence increase must be visible before the +* below count decrease, which is ensured by the smp_wmb above +* in conjunction with the smp_rmb in mmu_invalidate_retry(). +*/ + kvm->mmu_invalidate_in_progress--; +} +#if defined(CONFIG_MMU_NOTIFIER ) && defined(KVM_ARCH_WANT_MMU_NOTIFIER )static inline struct kvm *mmu_notifier_to_kvm (struct mmu_notifier *mn) {@@ -714,45 +770,6 @@ static void kvm_mmu_notifier_change_pte(struct mmu_notifier *mn,kvm_handle_hva_range (mn, address, address +1 , pte, kvm_set_spte_gfn); } -void kvm_mmu_invalidate_begin (struct kvm *kvm) -{ -/* -* The count increase must become visible at unlock time as no -* spte can be established without taking the mmu_lock and -* count is also read inside the mmu_lock critical section. -*/ - kvm->mmu_invalidate_in_progress++; - -if (likely (kvm->mmu_invalidate_in_progress ==1 )) { - kvm->mmu_invalidate_range_start =INVALID_GPA ; - kvm->mmu_invalidate_range_end =INVALID_GPA ; - } -} - -void kvm_mmu_invalidate_range_add (struct kvm *kvm,gfn_t start,gfn_t end) -{ -WARN_ON_ONCE (!kvm->mmu_invalidate_in_progress); - -if (likely (kvm->mmu_invalidate_in_progress ==1 )) { - kvm->mmu_invalidate_range_start = start; - kvm->mmu_invalidate_range_end = end; - }else { -/* -* Fully tracking multiple concurrent ranges has diminishing -* returns. Keep things simple and just find the minimal range -* which includes the current and new ranges. As there won't be -* enough information to subtract a range after its invalidate -* completes, any ranges invalidated concurrently will -* accumulate and persist until all outstanding invalidates -* complete. -*/ - kvm->mmu_invalidate_range_start = -min (kvm->mmu_invalidate_range_start, start); - kvm->mmu_invalidate_range_end = -max (kvm->mmu_invalidate_range_end, end); - } -} -static bool kvm_mmu_unmap_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range) {kvm_mmu_invalidate_range_add (kvm, range->start, range->end);@@ -806,23 +823,6 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,return 0 ; } -void kvm_mmu_invalidate_end (struct kvm *kvm) -{ -/* -* This sequence increase will notify the kvm page fault that -* the page that is going to be mapped in the spte could have -* been freed. -*/ - kvm->mmu_invalidate_seq++; -smp_wmb (); -/* -* The above sequence increase must be visible before the -* below count decrease, which is ensured by the smp_wmb above -* in conjunction with the smp_rmb in mmu_invalidate_retry(). -*/ - kvm->mmu_invalidate_in_progress--; -} -static void kvm_mmu_notifier_invalidate_range_end (struct mmu_notifier *mn,const struct mmu_notifier_range *range) {@@ -1140,6 +1140,11 @@ int __weak kvm_arch_create_vm_debugfs(struct kvm *kvm)return 0 ; } +bool __weak kvm_arch_has_private_mem (struct kvm *kvm) +{ +return false; +} +static struct kvm *kvm_create_vm (unsigned long type,const char *fdname) {struct kvm *kvm =kvm_arch_alloc_vm ();@@ -2349,15 +2354,47 @@ static u64 kvm_supported_mem_attributes(struct kvm *kvm)return 0 ; } +static void kvm_unmap_mem_range (struct kvm *kvm,gfn_t start,gfn_t end) +{ +struct kvm_gfn_range gfn_range; +struct kvm_memory_slot *slot; +struct kvm_memslots *slots; +struct kvm_memslot_iter iter; +int i; +int r =0 ; + + gfn_range.pte =__pte (0 ); + gfn_range.may_block = true; + +for (i =0 ; i <KVM_ADDRESS_SPACE_NUM ; i++) { + slots =__kvm_memslots (kvm, i); + +kvm_for_each_memslot_in_gfn_range (&iter, slots, start, end) { + slot = iter.slot; + gfn_range.start =max (start, slot->base_gfn); + gfn_range.end =min (end, slot->base_gfn + slot->npages); +if (gfn_range.start >= gfn_range.end) +continue ; + gfn_range.slot = slot; + + r |=kvm_unmap_gfn_range (kvm, &gfn_range); + } + } + +if (r) +kvm_flush_remote_tlbs (kvm); +} +static int kvm_vm_ioctl_set_mem_attributes (struct kvm *kvm,struct kvm_memory_attributes *attrs) {gfn_t start, end;unsigned long i;void *entry; +int idx;u64 supported_attrs =kvm_supported_mem_attributes (kvm); -/* flags is currently not used. */ +/* 'flags' is currently not used. */ if (attrs->flags)return -EINVAL ;if (attrs->attributes & ~supported_attrs)@@ -2372,6 +2409,13 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, entry = attrs->attributes ?xa_mk_value (attrs->attributes) :NULL ; +if (kvm_arch_has_private_mem (kvm)) { +KVM_MMU_LOCK (kvm); +kvm_mmu_invalidate_begin (kvm); +kvm_mmu_invalidate_range_add (kvm, start, end); +KVM_MMU_UNLOCK (kvm); + } +mutex_lock (&kvm->lock);for (i = start; i < end; i++) if (xa_err(xa_store(&kvm->mem_attr_array, i, entry,@@ -2379,6 +2423,16 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,break ;mutex_unlock (&kvm->lock); +if (kvm_arch_has_private_mem (kvm)) { + idx =srcu_read_lock (&kvm->srcu); +KVM_MMU_LOCK (kvm); +if (i > start) +kvm_unmap_mem_range (kvm, start, i); +kvm_mmu_invalidate_end (kvm); +KVM_MMU_UNLOCK (kvm); +srcu_read_unlock (&kvm->srcu, idx); + } + attrs->address = i <<PAGE_SHIFT ; attrs->size = (end - i) <<PAGE_SHIFT ; --2.25.1
Re: [PATCH v10 6/9] KVM: Unmap existing mappings when change the memory attributes
In reply to Chao PengQuoted 5 lines · Chao Peng
virt/kvm/kvm_main.c +1
-1-/* flags is currently not used. */ +/* 'flags' is currently not used. */ if (
attrs->flags)
return-EINVAL;
Unintended code change.
Re: [PATCH v10 6/9] KVM: Unmap existing mappings when change the memory attributes
Quoted 8 lines · Kai
Quoted 5 lines · Chao Peng
virt/kvm/kvm_main.c +1
-1-/* flags is currently not used. */ +/* 'flags' is currently not used. */ if (
attrs->flags)
return-EINVAL;Unintended code change.
Yeah!
Chao
Sean Christopherson
maintainerRe: [PATCH v10 6/9] KVM: Unmap existing mappings when change the memory attributes
In reply to Chao Penginclude/linux/kvm_host.h struct kvm { · line 792
Quoted 10 lines · Chao Peng
include/linux/kvm_host.h +1
-1@@ -785,11 +786,12 @@ struct kvm {#if defined(CONFIG_MMU_NOTIFIER ) && defined(KVM_ARCH_WANT_MMU_NOTIFIER )struct mmu_notifier mmu_notifier; +#endif unsigned long mmu_invalidate_seq;long mmu_invalidate_in_progress;gfn_t mmu_invalidate_range_start;gfn_t mmu_invalidate_range_end; -#endif
Blech. The existing code is a bit ugly, and trying to extend for this use case makes things even worse.
Rather than use the base MMU_NOTIFIER Kconfig and an arbitrary define, I think we should first add a proper Kconfig, e.g. KVM_GENERIC_MMU_NOTIFIER, to replace the combination. E.g
config KVM_GENERIC_MMU_NOTIFIER
select MMU_NOTIFIER
bool
and then all architectures that currently #define KVM_ARCH_WANT_MMU_NOTIFIER can simply select the Kconfig, which is everything except s390. "GENERIC" again because s390 does select MMU_NOTIFER and actually registers its own notifier for s390's version of protected VMs (at least, I think that's what its "pv" stands for).
And then later down the line in this series, when the attributes and private mem needs to tie into the notifiers, we can do:
config KVM_GENERIC_MEMORY_ATTRIBUTES
select KVM_GENERIC_MMU_NOTIFIER
bool
I.e. that way this patch doesn't need to partially expose KVM's notifier stuff and can instead just keep the soon-to-be-existing KVM_GENERIC_MMU_NOTIFIER.
Taking a depending on KVM_GENERIC_MMU_NOTIFIER for KVM_GENERIC_MEMORY_ATTRIBUTES makes sense, because AFAICT, changing any type of attribute, e.g. RWX bits, is going to necessitate unmapping the affected gfn range.
include/linux/kvm_host.h bool kvm_arch_dy_has_pending_interrupt(struct kvm_vcpu *vcpu); · line 1485
Quoted 8 lines
include/linux/kvm_host.h +1
-0struct list_head devices;u64 manual_dirty_log_protect;struct dentry *debugfs_dentry;@@ -1480,6 +1482,7 @@ bool kvm_arch_dy_has_pending_interrupt(struct kvm_vcpu *vcpu);int kvm_arch_post_init_vm (struct kvm *kvm);void kvm_arch_pre_destroy_vm (struct kvm *kvm);int kvm_arch_create_vm_debugfs (struct kvm *kvm); +bool kvm_arch_has_private_mem (struct kvm *kvm);
The reference to private memory belongs in a later patch. More below.
virt/kvm/kvm_main.c static u64 kvm_supported_mem_attributes(struct kvm *kvm) · line 2364
Quoted 8 lines
virt/kvm/kvm_main.c +8
-0+static void kvm_unmap_mem_range (struct kvm *kvm,gfn_t start,gfn_t end) +{ +struct kvm_gfn_range gfn_range; +struct kvm_memory_slot *slot; +struct kvm_memslots *slots; +struct kvm_memslot_iter iter; +int i; +int r =0 ;
The return from kvm_unmap_gfn_range() is a bool, this should be:
bool flush = false;
Quoted 24 lines
virt/kvm/kvm_main.c +23
-0+ + gfn_range.pte =__pte (0 ); + gfn_range.may_block = true; + +for (i =0 ; i <KVM_ADDRESS_SPACE_NUM ; i++) { + slots =__kvm_memslots (kvm, i); + +kvm_for_each_memslot_in_gfn_range (&iter, slots, start, end) { + slot = iter.slot; + gfn_range.start =max (start, slot->base_gfn); + gfn_range.end =min (end, slot->base_gfn + slot->npages); +if (gfn_range.start >= gfn_range.end) +continue ; + gfn_range.slot = slot; + + r |=kvm_unmap_gfn_range (kvm, &gfn_range); + } + } + +if (r) +kvm_flush_remote_tlbs (kvm); +} + static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
virt/kvm/kvm_main.c static u64 kvm_supported_mem_attributes(struct kvm *kvm) · line 2397
Quoted 10 lines
virt/kvm/kvm_main.c +2
-1struct kvm_memory_attributes *attrs) {gfn_t start, end;unsigned long i;void *entry; +int idx;u64 supported_attrs =kvm_supported_mem_attributes (kvm); -/* flags is currently not used. */ +/* 'flags' is currently not used. */
Kind of a spurious change.
virt/kvm/kvm_main.c static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, · line 2412
Quoted 8 lines
virt/kvm/kvm_main.c +1
-0if (attrs->flags)return -EINVAL ;if (attrs->attributes & ~supported_attrs)@@ -2372,6 +2409,13 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, entry = attrs->attributes ?xa_mk_value (attrs->attributes) :NULL ; +if (kvm_arch_has_private_mem (kvm)) {
I think we should assume that any future attributes will necessitate unmapping and invalidation, i.e. drop the private mem check. That allows introducing kvm_arch_has_private_mem() in a later patch that is more directly related to private memory.
Quoted 10 lines
virt/kvm/kvm_main.c +6
-0+KVM_MMU_LOCK (kvm); +kvm_mmu_invalidate_begin (kvm); +kvm_mmu_invalidate_range_add (kvm, start, end); +KVM_MMU_UNLOCK (kvm); + } +mutex_lock (&kvm->lock);for (i = start; i < end; i++) if (xa_err(xa_store(&kvm->mem_attr_array, i, entry,
virt/kvm/kvm_main.c static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, · line 2427
Quoted 5 lines
virt/kvm/kvm_main.c +2
-0@@ -2379,6 +2423,16 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,break ;mutex_unlock (&kvm->lock); +if (kvm_arch_has_private_mem (kvm)) { + idx =srcu_read_lock (&kvm->srcu);
Mostly for reference, this goes away if slots_lock is used instead of kvm->lock.
Quoted 14 lines
virt/kvm/kvm_main.c +8
-0+KVM_MMU_LOCK (kvm); +if (i > start) +kvm_unmap_mem_range (kvm, start, i); +kvm_mmu_invalidate_end (kvm); +KVM_MMU_UNLOCK (kvm); +srcu_read_unlock (&kvm->srcu, idx); + } + attrs->address = i <<PAGE_SHIFT ; attrs->size = (end - i) <<PAGE_SHIFT ;Signature · 1 line
2.25.1
PATCH v10 7/9 KVM: Update lpage info when private/shared memory are mixed
In reply to Chao PengA large page with mixed private/shared subpages can't be mapped as large page since its sub private/shared pages are from different memory backends and may also treated by architecture differently. When private/shared memory are mixed in a large page, the current lpage_info is not sufficient to decide whether the page can be mapped as large page or not and additional private/shared mixed information is needed.
Tracking this 'mixed' information with the current 'count' like disallow_lpage is a bit challenge so reserve a bit in 'disallow_lpage' to indicate a large page has mixed private/share subpages and update this 'mixed' bit whenever the memory attribute is changed between private and shared.
--- arch/x86/include/asm/kvm_host.h | 8 ++ arch/x86/kvm/mmu/mmu.c | 134 +++++++++++++++++++++++++++++++- arch/x86/kvm/x86.c | 2 + include/linux/kvm_host.h | 19 +++++ virt/kvm/kvm_main.c | 9 ++- 5 files changed, 169 insertions(+), 3 deletions(-)
arch/x86/include/asm/kvm_host.h +8 -0
@@ -38,6 +38,7 @@#include <asm/hyperv-tlfs.h> #define __KVM_HAVE_ARCH_VCPU_DEBUGFS +#define __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES#define KVM_MAX_VCPUS 1024
@@ -1011,6 +1012,13 @@ struct kvm_vcpu_arch {#endif }; +/* +* Use a bit in disallow_lpage to indicate private/shared pages mixed at the +* level. The remaining bits are used as a reference count. +*/ +#define KVM_LPAGE_PRIVATE_SHARED_MIXED (1U << 31) +#define KVM_LPAGE_COUNT_MAX ((1U << 31) - 1) +struct kvm_lpage_info {int disallow_lpage; };
arch/x86/kvm/mmu/mmu.c +133 -1
147 lines not on this page · raw message
arch/x86/kvm/x86.c +2 -0
@@ -12362,6 +12362,8 @@ static int kvm_alloc_memslot_metadata(struct kvm *kvm,if ((slot->base_gfn + npages) & (KVM_PAGES_PER_HPAGE (level) -1 )) linfo[lpages -1 ].disallow_lpage =1 ; ugfn = slot->userspace_addr >>PAGE_SHIFT ; +if (kvm_slot_can_be_private (slot)) + ugfn |= slot->restricted_offset >>PAGE_SHIFT ;/* * If the gfn and userspace address are not aligned wrt each * other, disable large page support for this slot.
include/linux/kvm_host.h +19 -0
@@ -592,6 +592,11 @@ struct kvm_memory_slot {struct restrictedmem_notifier notifier; }; +static inline bool kvm_slot_can_be_private (const struct kvm_memory_slot *slot) +{ +return slot && (slot->flags &KVM_MEM_PRIVATE ); +} +static inline bool kvm_slot_dirty_track_enabled (const struct kvm_memory_slot *slot) {return slot->flags &KVM_MEM_LOG_DIRTY_PAGES ;
@@ -2316,4 +2321,18 @@ static inline void kvm_account_pgtable_pages(void *virt, int nr)/* Max number of entries allowed for each kvm dirty ring */ #define KVM_DIRTY_RING_MAX_ENTRIES 65536 +#ifdef __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES +void kvm_arch_set_memory_attributes (struct kvm *kvm, +struct kvm_memory_slot *slot, +unsigned long attrs, +gfn_t start,gfn_t end); +#else +static inline void kvm_arch_set_memory_attributes (struct kvm *kvm, +struct kvm_memory_slot *slot, +unsigned long attrs, +gfn_t start,gfn_t end) +{ +} +#endif /* __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES */ +#endif
virt/kvm/kvm_main.c +7 -2
@@ -2354,7 +2354,8 @@ static u64 kvm_supported_mem_attributes(struct kvm *kvm)return 0 ; } -static void kvm_unmap_mem_range (struct kvm *kvm,gfn_t start,gfn_t end) +static void kvm_unmap_mem_range (struct kvm *kvm,gfn_t start,gfn_t end, +unsigned long attrs) {struct kvm_gfn_range gfn_range;struct kvm_memory_slot *slot;
@@ -2378,6 +2379,10 @@ static void kvm_unmap_mem_range(struct kvm *kvm, gfn_t start, gfn_t end) gfn_range.slot = slot; r |=kvm_unmap_gfn_range (kvm, &gfn_range); + +kvm_arch_set_memory_attributes (kvm, slot, attrs, + gfn_range.start, + gfn_range.end); } }
@@ -2427,7 +2432,7 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, idx =srcu_read_lock (&kvm->srcu);KVM_MMU_LOCK (kvm);if (i > start) -kvm_unmap_mem_range (kvm, start, i); +kvm_unmap_mem_range (kvm, start, i, attrs->attributes);kvm_mmu_invalidate_end (kvm);KVM_MMU_UNLOCK (kvm);srcu_read_unlock (&kvm->srcu, idx);
Signature · 1 line
2.25.1
Re: [PATCH v10 7/9] KVM: Update lpage info when private/shared memory are mixed
Quoted 171 lines · Chao Peng
A large page with mixed private/shared subpages can't be mapped as large page since its sub private/shared pages are from different memory backends and may also treated by architecture differently. When private/shared memory are mixed in a large page, the current
lpage_infois not sufficient to decide whether the page can be mapped as large page or not and additional private/shared mixed information is needed.Tracking this 'mixed' information with the current 'count' like
disallow_lpageis a bit challenge so reserve a bit in 'disallow_lpage' to indicate a large page has mixed private/share subpages and update this 'mixed' bit whenever the memory attribute is changed between private and shared.Signed-off-by Chao Peng <[email protected]>--- arch/x86/include/asm/kvm_host.h | 8 ++ arch/x86/kvm/mmu/mmu.c | 134 +++++++++++++++++++++++++++++++- arch/x86/kvm/x86.c | 2 + include/linux/kvm_host.h | 19 +++++ virt/kvm/kvm_main.c | 9 ++- 5 files changed, 169 insertions(+), 3 deletions(-)arch/x86/include/asm/kvm_host.h +8
-0@@ -38,6 +38,7 @@#include <asm/hyperv-tlfs.h> #define __KVM_HAVE_ARCH_VCPU_DEBUGFS +#define __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES#define KVM_MAX_VCPUS 1024@@ -1011,6 +1012,13 @@ struct kvm_vcpu_arch {#endif }; +/* +* Use a bit in disallow_lpage to indicate private/shared pages mixed at the +* level. The remaining bits are used as a reference count. +*/ +#define KVM_LPAGE_PRIVATE_SHARED_MIXED (1U << 31) +#define KVM_LPAGE_COUNT_MAX ((1U << 31) - 1) +struct kvm_lpage_info {int disallow_lpage; };arch/x86/kvm/mmu/mmu.c +103
-1@@ -763,11 +763,16 @@ static void update_gfn_disallow_lpage_count(const struct kvm_memory_slot *slot, {struct kvm_lpage_info *linfo;int i; +int disallow_count;for (i =PG_LEVEL_2M ; i <=KVM_MAX_HUGEPAGE_LEVEL ; ++i) { linfo =lpage_info_slot (gfn, slot, i); + + disallow_count = linfo->disallow_lpage &KVM_LPAGE_COUNT_MAX ; +WARN_ON (disallow_count + count <0 || + disallow_count >KVM_LPAGE_COUNT_MAX - count); + linfo->disallow_lpage += count; -WARN_ON (linfo->disallow_lpage <0 ); } }@@ -6986,3 +6991,130 @@ void kvm_mmu_pre_destroy_vm(struct kvm *kvm)if (kvm->arch.nx_huge_page_recovery_thread)kthread_stop (kvm->arch.nx_huge_page_recovery_thread); } + +static bool linfo_is_mixed (struct kvm_lpage_info *linfo) +{ +return linfo->disallow_lpage &KVM_LPAGE_PRIVATE_SHARED_MIXED ; +} + +static void linfo_set_mixed (gfn_t gfn,struct kvm_memory_slot *slot, +int level,bool mixed) +{ +struct kvm_lpage_info *linfo =lpage_info_slot (gfn, slot, level); + +if (mixed) + linfo->disallow_lpage |=KVM_LPAGE_PRIVATE_SHARED_MIXED ; +else + linfo->disallow_lpage &= ~KVM_LPAGE_PRIVATE_SHARED_MIXED ; +} + +static bool is_expected_attr_entry (void *entry,unsigned long expected_attrs) +{ +bool expect_private = expected_attrs &KVM_MEMORY_ATTRIBUTE_PRIVATE ; + +if (xa_to_value (entry) &KVM_MEMORY_ATTRIBUTE_PRIVATE ) { +if (!expect_private) +return false; + }else if (expect_private) +return false; + +return true; +} + +static bool mem_attrs_mixed_2m (struct kvm *kvm,unsigned long attrs, +gfn_t start,gfn_t end) +{ +XA_STATE (xas, &kvm->mem_attr_array, start); +gfn_t gfn = start; +void *entry; +bool mixed = false; + +rcu_read_lock (); + entry =xas_load (&xas); +while (gfn < end) { +if (xas_retry (&xas, entry)) +continue ; + +KVM_BUG_ON (gfn != xas.xa_index, kvm); + +if (!is_expected_attr_entry (entry, attrs)) { + mixed = true; +break ; + } + + entry =xas_next (&xas); + gfn++; + } + +rcu_read_unlock (); +return mixed; +} + +static bool mem_attrs_mixed (struct kvm *kvm,struct kvm_memory_slot *slot, +int level,unsigned long attrs, +gfn_t start,gfn_t end) +{ +unsigned long gfn; + +if (level ==PG_LEVEL_2M ) +return mem_attrs_mixed_2m (kvm, attrs, start, end); + +for (gfn = start; gfn < end; gfn +=KVM_PAGES_PER_HPAGE (level -1 )) +if (linfo_is_mixed (lpage_info_slot (gfn, slot, level -1 )) || + !is_expected_attr_entry (xa_load (&kvm->mem_attr_array, gfn), + attrs)) +return true; +return false; +} + +static void kvm_update_lpage_private_shared_mixed (struct kvm *kvm, +struct kvm_memory_slot *slot, +unsigned long attrs, +gfn_t start,gfn_t end) +{ +unsigned long pages, mask; +gfn_t gfn, gfn_end, first, last; +int level; +bool mixed; + +/* +* The sequence matters here: we set the higher level basing on the +* lower level's scanning result. +*/ +for (level =PG_LEVEL_2M ; level <=KVM_MAX_HUGEPAGE_LEVEL ; level++) { + pages =KVM_PAGES_PER_HPAGE (level); + mask = ~(pages -1 ); + first = start & mask; + last = (end -1 ) & mask; + +/*
arch/x86/kvm/mmu/mmu.c void kvm_mmu_pre_destroy_vm(struct kvm *kvm) · line 7100
Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +10
-0+* We only need to scan the head and tail page, for middle pages +* we know they will not be mixed. +*/ + gfn =max (first, slot->base_gfn); + gfn_end =min (first + pages, slot->base_gfn + slot->npages); + mixed =mem_attrs_mixed (kvm, slot, level, attrs, gfn, gfn_end); +linfo_set_mixed (gfn, slot, level, mixed); + +if (first == last) +return ;
continue.
arch/x86/kvm/mmu/mmu.c void kvm_mmu_pre_destroy_vm(struct kvm *kvm) · line 7106
Quoted 6 lines
arch/x86/kvm/mmu/mmu.c +6
-0+ +for (gfn = first + pages; gfn < last; gfn += pages) +linfo_set_mixed (gfn, slot, level, false); + + gfn = last; + gfn_end =min (last + pages, slot->base_gfn + slot->npages);
if (gfn == gfn_end) continue.
Quoted 19 lines
arch/x86/kvm/mmu/mmu.c +14
-0+ mixed =mem_attrs_mixed (kvm, slot, level, attrs, gfn, gfn_end); +linfo_set_mixed (gfn, slot, level, mixed); + } +} + +void kvm_arch_set_memory_attributes (struct kvm *kvm, +struct kvm_memory_slot *slot, +unsigned long attrs, +gfn_t start,gfn_t end) +{ +if (kvm_slot_can_be_private (slot)) +kvm_update_lpage_private_shared_mixed (kvm, slot, attrs, + start, end); +}arch/x86/kvm/x86.c +0
-0
arch/x86/kvm/x86.c static int kvm_alloc_memslot_metadata(struct kvm *kvm, · line 12366
Quoted 5 lines
arch/x86/kvm/x86.c +2
-0@@ -12362,6 +12362,8 @@ static int kvm_alloc_memslot_metadata(struct kvm *kvm,if ((slot->base_gfn + npages) & (KVM_PAGES_PER_HPAGE (level) -1 )) linfo[lpages -1 ].disallow_lpage =1 ; ugfn = slot->userspace_addr >>PAGE_SHIFT ; +if (kvm_slot_can_be_private (slot)) + ugfn |= slot->restricted_offset >>PAGE_SHIFT ;
Is there any alignment restriction? If no, It should be +=. In practice, alignment will hold though.
Thanks,
Quoted 75 lines
arch/x86/kvm/x86.c +0
-0/* * If the gfn and userspace address are not aligned wrt each * other, disable large page support for this slot. include/linux/kvm_host.h +19
-0@@ -592,6 +592,11 @@ struct kvm_memory_slot {struct restrictedmem_notifier notifier; }; +static inline bool kvm_slot_can_be_private (const struct kvm_memory_slot *slot) +{ +return slot && (slot->flags &KVM_MEM_PRIVATE ); +} +static inline bool kvm_slot_dirty_track_enabled (const struct kvm_memory_slot *slot) {return slot->flags &KVM_MEM_LOG_DIRTY_PAGES ;@@ -2316,4 +2321,18 @@ static inline void kvm_account_pgtable_pages(void *virt, int nr)/* Max number of entries allowed for each kvm dirty ring */ #define KVM_DIRTY_RING_MAX_ENTRIES 65536 +#ifdef __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES +void kvm_arch_set_memory_attributes (struct kvm *kvm, +struct kvm_memory_slot *slot, +unsigned long attrs, +gfn_t start,gfn_t end); +#else +static inline void kvm_arch_set_memory_attributes (struct kvm *kvm, +struct kvm_memory_slot *slot, +unsigned long attrs, +gfn_t start,gfn_t end) +{ +} +#endif /* __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES */ +#endif virt/kvm/kvm_main.c +7
-2@@ -2354,7 +2354,8 @@ static u64 kvm_supported_mem_attributes(struct kvm *kvm)return 0 ; } -static void kvm_unmap_mem_range (struct kvm *kvm,gfn_t start,gfn_t end) +static void kvm_unmap_mem_range (struct kvm *kvm,gfn_t start,gfn_t end, +unsigned long attrs) {struct kvm_gfn_range gfn_range;struct kvm_memory_slot *slot;@@ -2378,6 +2379,10 @@ static void kvm_unmap_mem_range(struct kvm *kvm, gfn_t start, gfn_t end) gfn_range.slot = slot; r |=kvm_unmap_gfn_range (kvm, &gfn_range); + +kvm_arch_set_memory_attributes (kvm, slot, attrs, + gfn_range.start, + gfn_range.end); } }@@ -2427,7 +2432,7 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, idx =srcu_read_lock (&kvm->srcu);KVM_MMU_LOCK (kvm);if (i > start) -kvm_unmap_mem_range (kvm, start, i); +kvm_unmap_mem_range (kvm, start, i, attrs->attributes);kvm_mmu_invalidate_end (kvm);KVM_MMU_UNLOCK (kvm);srcu_read_unlock (&kvm->srcu, idx);Signature · 1 line
2.25.1
Signature · 1 line
Isaku Yamahata <[email protected]>
Re: [PATCH v10 7/9] KVM: Update lpage info when private/shared memory are mixed
Quoted 179 lines · Isaku Yamahata
Quoted 176 lines · Chao Peng
A large page with mixed private/shared subpages can't be mapped as large page since its sub private/shared pages are from different memory backends and may also treated by architecture differently. When private/shared memory are mixed in a large page, the current
lpage_infois not sufficient to decide whether the page can be mapped as large page or not and additional private/shared mixed information is needed.Tracking this 'mixed' information with the current 'count' like
disallow_lpageis a bit challenge so reserve a bit in 'disallow_lpage' to indicate a large page has mixed private/share subpages and update this 'mixed' bit whenever the memory attribute is changed between private and shared.Signed-off-by Chao Peng <[email protected]>--- arch/x86/include/asm/kvm_host.h | 8 ++ arch/x86/kvm/mmu/mmu.c | 134 +++++++++++++++++++++++++++++++- arch/x86/kvm/x86.c | 2 + include/linux/kvm_host.h | 19 +++++ virt/kvm/kvm_main.c | 9 ++- 5 files changed, 169 insertions(+), 3 deletions(-)arch/x86/include/asm/kvm_host.h +8
-0@@ -38,6 +38,7 @@#include <asm/hyperv-tlfs.h> #define __KVM_HAVE_ARCH_VCPU_DEBUGFS +#define __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES#define KVM_MAX_VCPUS 1024@@ -1011,6 +1012,13 @@ struct kvm_vcpu_arch {#endif }; +/* +* Use a bit in disallow_lpage to indicate private/shared pages mixed at the +* level. The remaining bits are used as a reference count. +*/ +#define KVM_LPAGE_PRIVATE_SHARED_MIXED (1U << 31) +#define KVM_LPAGE_COUNT_MAX ((1U << 31) - 1) +struct kvm_lpage_info {int disallow_lpage; };arch/x86/kvm/mmu/mmu.c +108
-1@@ -763,11 +763,16 @@ static void update_gfn_disallow_lpage_count(const struct kvm_memory_slot *slot, {struct kvm_lpage_info *linfo;int i; +int disallow_count;for (i =PG_LEVEL_2M ; i <=KVM_MAX_HUGEPAGE_LEVEL ; ++i) { linfo =lpage_info_slot (gfn, slot, i); + + disallow_count = linfo->disallow_lpage &KVM_LPAGE_COUNT_MAX ; +WARN_ON (disallow_count + count <0 || + disallow_count >KVM_LPAGE_COUNT_MAX - count); + linfo->disallow_lpage += count; -WARN_ON (linfo->disallow_lpage <0 ); } }@@ -6986,3 +6991,130 @@ void kvm_mmu_pre_destroy_vm(struct kvm *kvm)if (kvm->arch.nx_huge_page_recovery_thread)kthread_stop (kvm->arch.nx_huge_page_recovery_thread); } + +static bool linfo_is_mixed (struct kvm_lpage_info *linfo) +{ +return linfo->disallow_lpage &KVM_LPAGE_PRIVATE_SHARED_MIXED ; +} + +static void linfo_set_mixed (gfn_t gfn,struct kvm_memory_slot *slot, +int level,bool mixed) +{ +struct kvm_lpage_info *linfo =lpage_info_slot (gfn, slot, level); + +if (mixed) + linfo->disallow_lpage |=KVM_LPAGE_PRIVATE_SHARED_MIXED ; +else + linfo->disallow_lpage &= ~KVM_LPAGE_PRIVATE_SHARED_MIXED ; +} + +static bool is_expected_attr_entry (void *entry,unsigned long expected_attrs) +{ +bool expect_private = expected_attrs &KVM_MEMORY_ATTRIBUTE_PRIVATE ; + +if (xa_to_value (entry) &KVM_MEMORY_ATTRIBUTE_PRIVATE ) { +if (!expect_private) +return false; + }else if (expect_private) +return false; + +return true; +} + +static bool mem_attrs_mixed_2m (struct kvm *kvm,unsigned long attrs, +gfn_t start,gfn_t end) +{ +XA_STATE (xas, &kvm->mem_attr_array, start); +gfn_t gfn = start; +void *entry; +bool mixed = false; + +rcu_read_lock (); + entry =xas_load (&xas); +while (gfn < end) { +if (xas_retry (&xas, entry)) +continue ; + +KVM_BUG_ON (gfn != xas.xa_index, kvm); + +if (!is_expected_attr_entry (entry, attrs)) { + mixed = true; +break ; + } + + entry =xas_next (&xas); + gfn++; + } + +rcu_read_unlock (); +return mixed; +} + +static bool mem_attrs_mixed (struct kvm *kvm,struct kvm_memory_slot *slot, +int level,unsigned long attrs, +gfn_t start,gfn_t end) +{ +unsigned long gfn; + +if (level ==PG_LEVEL_2M ) +return mem_attrs_mixed_2m (kvm, attrs, start, end); + +for (gfn = start; gfn < end; gfn +=KVM_PAGES_PER_HPAGE (level -1 )) +if (linfo_is_mixed (lpage_info_slot (gfn, slot, level -1 )) || + !is_expected_attr_entry (xa_load (&kvm->mem_attr_array, gfn), + attrs)) +return true; +return false; +} + +static void kvm_update_lpage_private_shared_mixed (struct kvm *kvm, +struct kvm_memory_slot *slot, +unsigned long attrs, +gfn_t start,gfn_t end) +{ +unsigned long pages, mask; +gfn_t gfn, gfn_end, first, last; +int level; +bool mixed; + +/* +* The sequence matters here: we set the higher level basing on the +* lower level's scanning result. +*/ +for (level =PG_LEVEL_2M ; level <=KVM_MAX_HUGEPAGE_LEVEL ; level++) { + pages =KVM_PAGES_PER_HPAGE (level); + mask = ~(pages -1 ); + first = start & mask; + last = (end -1 ) & mask; + +/* +* We only need to scan the head and tail page, for middle pages +* we know they will not be mixed. +*/ + gfn =max (first, slot->base_gfn); + gfn_end =min (first + pages, slot->base_gfn + slot->npages);
Quoted 8 lines
arch/x86/kvm/mmu/mmu.c void kvm_mmu_pre_destroy_vm(struct kvm *kvm) · line 7100
Quoted 5 lines
arch/x86/kvm/mmu/mmu.c +5
-0+ mixed =mem_attrs_mixed (kvm, slot, level, attrs, gfn, gfn_end); +linfo_set_mixed (gfn, slot, level, mixed); + +if (first == last) +return ;continue.
Ya!
Quoted 9 lines
arch/x86/kvm/mmu/mmu.c void kvm_mmu_pre_destroy_vm(struct kvm *kvm) · line 7106
Quoted 7 lines
arch/x86/kvm/mmu/mmu.c +6
-0+ +for (gfn = first + pages; gfn < last; gfn += pages) +linfo_set_mixed (gfn, slot, level, false); + + gfn = last; + gfn_end =min (last + pages, slot->base_gfn + slot->npages);if (gfn ==
gfn_end) continue.
Do you see a case where gfn can equal to gfn_end? Though it does not hurt to add a check.
Quoted 21 lines
Quoted 21 lines
arch/x86/kvm/mmu/mmu.c +14
-0+ mixed =mem_attrs_mixed (kvm, slot, level, attrs, gfn, gfn_end); +linfo_set_mixed (gfn, slot, level, mixed); + } +} + +void kvm_arch_set_memory_attributes (struct kvm *kvm, +struct kvm_memory_slot *slot, +unsigned long attrs, +gfn_t start,gfn_t end) +{ +if (kvm_slot_can_be_private (slot)) +kvm_update_lpage_private_shared_mixed (kvm, slot, attrs, + start, end); +}arch/x86/kvm/x86.c +0
-0@@ -12362,6 +12362,8 @@ static int kvm_alloc_memslot_metadata(struct kvm *kvm,
Quoted 8 lines
arch/x86/kvm/x86.c static int kvm_alloc_memslot_metadata(struct kvm *kvm, · line 12366
Quoted 5 lines
arch/x86/kvm/x86.c +2
-0if ((slot->base_gfn + npages) & (KVM_PAGES_PER_HPAGE (level) -1 )) linfo[lpages -1 ].disallow_lpage =1 ; ugfn = slot->userspace_addr >>PAGE_SHIFT ; +if (kvm_slot_can_be_private (slot)) + ugfn |= slot->restricted_offset >>PAGE_SHIFT ;Is there any alignment restriction? If no, It should be +=. In practice, alignment will hold though.
All we need here is checking whether both userspace_addr and restricted_offset are aligned to HPAGE_SIZE or not. '+=' actually can yield wrong value in cases when userspace_addr + restricted_offset is aligned to HPAGE_SIZE but individually they may not align to HPAGE_SIZE.
Thanks,
Chao
Quoted 81 lines
Thanks,
Quoted 75 lines
arch/x86/kvm/x86.c +0
-0/* * If the gfn and userspace address are not aligned wrt each * other, disable large page support for this slot. include/linux/kvm_host.h +19
-0@@ -592,6 +592,11 @@ struct kvm_memory_slot {struct restrictedmem_notifier notifier; }; +static inline bool kvm_slot_can_be_private (const struct kvm_memory_slot *slot) +{ +return slot && (slot->flags &KVM_MEM_PRIVATE ); +} +static inline bool kvm_slot_dirty_track_enabled (const struct kvm_memory_slot *slot) {return slot->flags &KVM_MEM_LOG_DIRTY_PAGES ;@@ -2316,4 +2321,18 @@ static inline void kvm_account_pgtable_pages(void *virt, int nr)/* Max number of entries allowed for each kvm dirty ring */ #define KVM_DIRTY_RING_MAX_ENTRIES 65536 +#ifdef __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES +void kvm_arch_set_memory_attributes (struct kvm *kvm, +struct kvm_memory_slot *slot, +unsigned long attrs, +gfn_t start,gfn_t end); +#else +static inline void kvm_arch_set_memory_attributes (struct kvm *kvm, +struct kvm_memory_slot *slot, +unsigned long attrs, +gfn_t start,gfn_t end) +{ +} +#endif /* __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES */ +#endif virt/kvm/kvm_main.c +7
-2@@ -2354,7 +2354,8 @@ static u64 kvm_supported_mem_attributes(struct kvm *kvm)return 0 ; } -static void kvm_unmap_mem_range (struct kvm *kvm,gfn_t start,gfn_t end) +static void kvm_unmap_mem_range (struct kvm *kvm,gfn_t start,gfn_t end, +unsigned long attrs) {struct kvm_gfn_range gfn_range;struct kvm_memory_slot *slot;@@ -2378,6 +2379,10 @@ static void kvm_unmap_mem_range(struct kvm *kvm, gfn_t start, gfn_t end) gfn_range.slot = slot; r |=kvm_unmap_gfn_range (kvm, &gfn_range); + +kvm_arch_set_memory_attributes (kvm, slot, attrs, + gfn_range.start, + gfn_range.end); } }@@ -2427,7 +2432,7 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, idx =srcu_read_lock (&kvm->srcu);KVM_MMU_LOCK (kvm);if (i > start) -kvm_unmap_mem_range (kvm, start, i); +kvm_unmap_mem_range (kvm, start, i, attrs->attributes);kvm_mmu_invalidate_end (kvm);KVM_MMU_UNLOCK (kvm);srcu_read_unlock (&kvm->srcu, idx);Signature · 1 line
2.25.1Signature · 1 line
Isaku Yamahata <[email protected]>
Re: [PATCH v10 7/9] KVM: Update lpage info when private/shared memory are mixed
Quoted 201 lines · Chao Peng
Quoted 179 lines · Isaku Yamahata
Quoted 176 lines · Chao Peng
A large page with mixed private/shared subpages can't be mapped as large page since its sub private/shared pages are from different memory backends and may also treated by architecture differently. When private/shared memory are mixed in a large page, the current
lpage_infois not sufficient to decide whether the page can be mapped as large page or not and additional private/shared mixed information is needed.Tracking this 'mixed' information with the current 'count' like
disallow_lpageis a bit challenge so reserve a bit in 'disallow_lpage' to indicate a large page has mixed private/share subpages and update this 'mixed' bit whenever the memory attribute is changed between private and shared.Signed-off-by Chao Peng <[email protected]>--- arch/x86/include/asm/kvm_host.h | 8 ++ arch/x86/kvm/mmu/mmu.c | 134 +++++++++++++++++++++++++++++++- arch/x86/kvm/x86.c | 2 + include/linux/kvm_host.h | 19 +++++ virt/kvm/kvm_main.c | 9 ++- 5 files changed, 169 insertions(+), 3 deletions(-)arch/x86/include/asm/kvm_host.h +8
-0@@ -38,6 +38,7 @@#include <asm/hyperv-tlfs.h> #define __KVM_HAVE_ARCH_VCPU_DEBUGFS +#define __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES#define KVM_MAX_VCPUS 1024@@ -1011,6 +1012,13 @@ struct kvm_vcpu_arch {#endif }; +/* +* Use a bit in disallow_lpage to indicate private/shared pages mixed at the +* level. The remaining bits are used as a reference count. +*/ +#define KVM_LPAGE_PRIVATE_SHARED_MIXED (1U << 31) +#define KVM_LPAGE_COUNT_MAX ((1U << 31) - 1) +struct kvm_lpage_info {int disallow_lpage; };arch/x86/kvm/mmu/mmu.c +108
-1@@ -763,11 +763,16 @@ static void update_gfn_disallow_lpage_count(const struct kvm_memory_slot *slot, {struct kvm_lpage_info *linfo;int i; +int disallow_count;for (i =PG_LEVEL_2M ; i <=KVM_MAX_HUGEPAGE_LEVEL ; ++i) { linfo =lpage_info_slot (gfn, slot, i); + + disallow_count = linfo->disallow_lpage &KVM_LPAGE_COUNT_MAX ; +WARN_ON (disallow_count + count <0 || + disallow_count >KVM_LPAGE_COUNT_MAX - count); + linfo->disallow_lpage += count; -WARN_ON (linfo->disallow_lpage <0 ); } }@@ -6986,3 +6991,130 @@ void kvm_mmu_pre_destroy_vm(struct kvm *kvm)if (kvm->arch.nx_huge_page_recovery_thread)kthread_stop (kvm->arch.nx_huge_page_recovery_thread); } + +static bool linfo_is_mixed (struct kvm_lpage_info *linfo) +{ +return linfo->disallow_lpage &KVM_LPAGE_PRIVATE_SHARED_MIXED ; +} + +static void linfo_set_mixed (gfn_t gfn,struct kvm_memory_slot *slot, +int level,bool mixed) +{ +struct kvm_lpage_info *linfo =lpage_info_slot (gfn, slot, level); + +if (mixed) + linfo->disallow_lpage |=KVM_LPAGE_PRIVATE_SHARED_MIXED ; +else + linfo->disallow_lpage &= ~KVM_LPAGE_PRIVATE_SHARED_MIXED ; +} + +static bool is_expected_attr_entry (void *entry,unsigned long expected_attrs) +{ +bool expect_private = expected_attrs &KVM_MEMORY_ATTRIBUTE_PRIVATE ; + +if (xa_to_value (entry) &KVM_MEMORY_ATTRIBUTE_PRIVATE ) { +if (!expect_private) +return false; + }else if (expect_private) +return false; + +return true; +} + +static bool mem_attrs_mixed_2m (struct kvm *kvm,unsigned long attrs, +gfn_t start,gfn_t end) +{ +XA_STATE (xas, &kvm->mem_attr_array, start); +gfn_t gfn = start; +void *entry; +bool mixed = false; + +rcu_read_lock (); + entry =xas_load (&xas); +while (gfn < end) { +if (xas_retry (&xas, entry)) +continue ; + +KVM_BUG_ON (gfn != xas.xa_index, kvm); + +if (!is_expected_attr_entry (entry, attrs)) { + mixed = true; +break ; + } + + entry =xas_next (&xas); + gfn++; + } + +rcu_read_unlock (); +return mixed; +} + +static bool mem_attrs_mixed (struct kvm *kvm,struct kvm_memory_slot *slot, +int level,unsigned long attrs, +gfn_t start,gfn_t end) +{ +unsigned long gfn; + +if (level ==PG_LEVEL_2M ) +return mem_attrs_mixed_2m (kvm, attrs, start, end); + +for (gfn = start; gfn < end; gfn +=KVM_PAGES_PER_HPAGE (level -1 )) +if (linfo_is_mixed (lpage_info_slot (gfn, slot, level -1 )) || + !is_expected_attr_entry (xa_load (&kvm->mem_attr_array, gfn), + attrs)) +return true; +return false; +} + +static void kvm_update_lpage_private_shared_mixed (struct kvm *kvm, +struct kvm_memory_slot *slot, +unsigned long attrs, +gfn_t start,gfn_t end) +{ +unsigned long pages, mask; +gfn_t gfn, gfn_end, first, last; +int level; +bool mixed; + +/* +* The sequence matters here: we set the higher level basing on the +* lower level's scanning result. +*/ +for (level =PG_LEVEL_2M ; level <=KVM_MAX_HUGEPAGE_LEVEL ; level++) { + pages =KVM_PAGES_PER_HPAGE (level); + mask = ~(pages -1 ); + first = start & mask; + last = (end -1 ) & mask; + +/* +* We only need to scan the head and tail page, for middle pages +* we know they will not be mixed. +*/ + gfn =max (first, slot->base_gfn); + gfn_end =min (first + pages, slot->base_gfn + slot->npages);Quoted 8 lines
arch/x86/kvm/mmu/mmu.c void kvm_mmu_pre_destroy_vm(struct kvm *kvm) · line 7100
Quoted 5 lines
arch/x86/kvm/mmu/mmu.c +5
-0+ mixed =mem_attrs_mixed (kvm, slot, level, attrs, gfn, gfn_end); +linfo_set_mixed (gfn, slot, level, mixed); + +if (first == last) +return ;continue.
Ya!
Quoted 9 lines
arch/x86/kvm/mmu/mmu.c void kvm_mmu_pre_destroy_vm(struct kvm *kvm) · line 7106
Quoted 7 lines
arch/x86/kvm/mmu/mmu.c +6
-0+ +for (gfn = first + pages; gfn < last; gfn += pages) +linfo_set_mixed (gfn, slot, level, false); + + gfn = last; + gfn_end =min (last + pages, slot->base_gfn + slot->npages);if (gfn ==
gfn_end) continue.
Quoted 2 lines
Do you see a case where gfn can equal to
gfn_end? Though it does not hurt to add a check.
If last == base_gfn + npages, gfn == gfn_end can occur.
Quoted 14 lines
Quoted 5 lines
Quoted 5 lines
arch/x86/kvm/x86.c +0
-0@@ -12362,6 +12362,8 @@ static int kvm_alloc_memslot_metadata(struct kvm *kvm,Quoted 8 lines
arch/x86/kvm/x86.c static int kvm_alloc_memslot_metadata(struct kvm *kvm, · line 12366
Quoted 5 lines
arch/x86/kvm/x86.c +2
-0if ((slot->base_gfn + npages) & (KVM_PAGES_PER_HPAGE (level) -1 )) linfo[lpages -1 ].disallow_lpage =1 ; ugfn = slot->userspace_addr >>PAGE_SHIFT ; +if (kvm_slot_can_be_private (slot)) + ugfn |= slot->restricted_offset >>PAGE_SHIFT ;Is there any alignment restriction? If no, It should be +=. In practice, alignment will hold though.
Quoted 4 lines
All we need here is checking whether both
userspace_addrandrestricted_offsetare aligned toHPAGE_SIZEor not. '+=' actually can yield wrong value in cases whenuserspace_addr+restricted_offsetis aligned toHPAGE_SIZEbut individually they may not align toHPAGE_SIZE.
Ah, got it. The blow comment explains it.
Quoted 8 lines
Thanks,
ChaoQuoted 6 lines
Thanks,
Quoted 3 lines
arch/x86/kvm/x86.c +0
-0/* * If the gfn and userspace address are not aligned wrt each * other, disable large page support for this slot.
Signature · 1 line
Isaku Yamahata <[email protected]>
Re: [PATCH v10 7/9] KVM: Update lpage info when private/shared memory are mixed
Quoted 204 lines · Isaku Yamahata
Quoted 201 lines · Chao Peng
Quoted 179 lines · Isaku Yamahata
Quoted 176 lines · Chao Peng
A large page with mixed private/shared subpages can't be mapped as large page since its sub private/shared pages are from different memory backends and may also treated by architecture differently. When private/shared memory are mixed in a large page, the current
lpage_infois not sufficient to decide whether the page can be mapped as large page or not and additional private/shared mixed information is needed.Tracking this 'mixed' information with the current 'count' like
disallow_lpageis a bit challenge so reserve a bit in 'disallow_lpage' to indicate a large page has mixed private/share subpages and update this 'mixed' bit whenever the memory attribute is changed between private and shared.Signed-off-by Chao Peng <[email protected]>--- arch/x86/include/asm/kvm_host.h | 8 ++ arch/x86/kvm/mmu/mmu.c | 134 +++++++++++++++++++++++++++++++- arch/x86/kvm/x86.c | 2 + include/linux/kvm_host.h | 19 +++++ virt/kvm/kvm_main.c | 9 ++- 5 files changed, 169 insertions(+), 3 deletions(-)arch/x86/include/asm/kvm_host.h +8
-0@@ -38,6 +38,7 @@#include <asm/hyperv-tlfs.h> #define __KVM_HAVE_ARCH_VCPU_DEBUGFS +#define __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES#define KVM_MAX_VCPUS 1024@@ -1011,6 +1012,13 @@ struct kvm_vcpu_arch {#endif }; +/* +* Use a bit in disallow_lpage to indicate private/shared pages mixed at the +* level. The remaining bits are used as a reference count. +*/ +#define KVM_LPAGE_PRIVATE_SHARED_MIXED (1U << 31) +#define KVM_LPAGE_COUNT_MAX ((1U << 31) - 1) +struct kvm_lpage_info {int disallow_lpage; };arch/x86/kvm/mmu/mmu.c +108
-1@@ -763,11 +763,16 @@ static void update_gfn_disallow_lpage_count(const struct kvm_memory_slot *slot, {struct kvm_lpage_info *linfo;int i; +int disallow_count;for (i =PG_LEVEL_2M ; i <=KVM_MAX_HUGEPAGE_LEVEL ; ++i) { linfo =lpage_info_slot (gfn, slot, i); + + disallow_count = linfo->disallow_lpage &KVM_LPAGE_COUNT_MAX ; +WARN_ON (disallow_count + count <0 || + disallow_count >KVM_LPAGE_COUNT_MAX - count); + linfo->disallow_lpage += count; -WARN_ON (linfo->disallow_lpage <0 ); } }@@ -6986,3 +6991,130 @@ void kvm_mmu_pre_destroy_vm(struct kvm *kvm)if (kvm->arch.nx_huge_page_recovery_thread)kthread_stop (kvm->arch.nx_huge_page_recovery_thread); } + +static bool linfo_is_mixed (struct kvm_lpage_info *linfo) +{ +return linfo->disallow_lpage &KVM_LPAGE_PRIVATE_SHARED_MIXED ; +} + +static void linfo_set_mixed (gfn_t gfn,struct kvm_memory_slot *slot, +int level,bool mixed) +{ +struct kvm_lpage_info *linfo =lpage_info_slot (gfn, slot, level); + +if (mixed) + linfo->disallow_lpage |=KVM_LPAGE_PRIVATE_SHARED_MIXED ; +else + linfo->disallow_lpage &= ~KVM_LPAGE_PRIVATE_SHARED_MIXED ; +} + +static bool is_expected_attr_entry (void *entry,unsigned long expected_attrs) +{ +bool expect_private = expected_attrs &KVM_MEMORY_ATTRIBUTE_PRIVATE ; + +if (xa_to_value (entry) &KVM_MEMORY_ATTRIBUTE_PRIVATE ) { +if (!expect_private) +return false; + }else if (expect_private) +return false; + +return true; +} + +static bool mem_attrs_mixed_2m (struct kvm *kvm,unsigned long attrs, +gfn_t start,gfn_t end) +{ +XA_STATE (xas, &kvm->mem_attr_array, start); +gfn_t gfn = start; +void *entry; +bool mixed = false; + +rcu_read_lock (); + entry =xas_load (&xas); +while (gfn < end) { +if (xas_retry (&xas, entry)) +continue ; + +KVM_BUG_ON (gfn != xas.xa_index, kvm); + +if (!is_expected_attr_entry (entry, attrs)) { + mixed = true; +break ; + } + + entry =xas_next (&xas); + gfn++; + } + +rcu_read_unlock (); +return mixed; +} + +static bool mem_attrs_mixed (struct kvm *kvm,struct kvm_memory_slot *slot, +int level,unsigned long attrs, +gfn_t start,gfn_t end) +{ +unsigned long gfn; + +if (level ==PG_LEVEL_2M ) +return mem_attrs_mixed_2m (kvm, attrs, start, end); + +for (gfn = start; gfn < end; gfn +=KVM_PAGES_PER_HPAGE (level -1 )) +if (linfo_is_mixed (lpage_info_slot (gfn, slot, level -1 )) || + !is_expected_attr_entry (xa_load (&kvm->mem_attr_array, gfn), + attrs)) +return true; +return false; +} + +static void kvm_update_lpage_private_shared_mixed (struct kvm *kvm, +struct kvm_memory_slot *slot, +unsigned long attrs, +gfn_t start,gfn_t end) +{ +unsigned long pages, mask; +gfn_t gfn, gfn_end, first, last; +int level; +bool mixed; + +/* +* The sequence matters here: we set the higher level basing on the +* lower level's scanning result. +*/ +for (level =PG_LEVEL_2M ; level <=KVM_MAX_HUGEPAGE_LEVEL ; level++) { + pages =KVM_PAGES_PER_HPAGE (level); + mask = ~(pages -1 ); + first = start & mask; + last = (end -1 ) & mask; + +/* +* We only need to scan the head and tail page, for middle pages +* we know they will not be mixed. +*/ + gfn =max (first, slot->base_gfn); + gfn_end =min (first + pages, slot->base_gfn + slot->npages);Quoted 8 lines
arch/x86/kvm/mmu/mmu.c void kvm_mmu_pre_destroy_vm(struct kvm *kvm) · line 7100
Quoted 5 lines
arch/x86/kvm/mmu/mmu.c +5
-0+ mixed =mem_attrs_mixed (kvm, slot, level, attrs, gfn, gfn_end); +linfo_set_mixed (gfn, slot, level, mixed); + +if (first == last) +return ;continue.
Ya!
Quoted 9 lines
arch/x86/kvm/mmu/mmu.c void kvm_mmu_pre_destroy_vm(struct kvm *kvm) · line 7106
Quoted 7 lines
arch/x86/kvm/mmu/mmu.c +6
-0+ +for (gfn = first + pages; gfn < last; gfn += pages) +linfo_set_mixed (gfn, slot, level, false); + + gfn = last; + gfn_end =min (last + pages, slot->base_gfn + slot->npages);if (gfn ==
gfn_end) continue.
Quoted 4 lines
Quoted 2 lines
Do you see a case where gfn can equal to
gfn_end? Though it does not hurt to add a check.If last ==
base_gfn+ npages, gfn ==gfn_endcan occur.
'end' is guaranteed <= base_gfn + npages in kvm_unmap_mem_range():
gfn_range.end = min(end, slot->base_gfn + slot->npages);
And 'last' is defined as:
last = (end - 1) & mask;
Then the math is:
last = (end - 1) & mask <= end - 1 <= base_gfn + npages - 1 < base_gfn + npages
Thanks,
Chao
Quoted 33 lines
Quoted 16 lines
Quoted 15 lines
arch/x86/kvm/x86.c static int kvm_alloc_memslot_metadata(struct kvm *kvm, · line 12366
Quoted 12 lines
arch/x86/kvm/x86.c +2
-0@@ -12362,6 +12362,8 @@ static int kvm_alloc_memslot_metadata(struct kvm *kvm,if ((slot->base_gfn + npages) & (KVM_PAGES_PER_HPAGE (level) -1 )) linfo[lpages -1 ].disallow_lpage =1 ; ugfn = slot->userspace_addr >>PAGE_SHIFT ; +if (kvm_slot_can_be_private (slot)) + ugfn |= slot->restricted_offset >>PAGE_SHIFT ;Is there any alignment restriction? If no, It should be +=. In practice, alignment will hold though.
Quoted 4 lines
All we need here is checking whether both
userspace_addrandrestricted_offsetare aligned toHPAGE_SIZEor not. '+=' actually can yield wrong value in cases whenuserspace_addr+restricted_offsetis aligned toHPAGE_SIZEbut individually they may not align toHPAGE_SIZE.Ah, got it. The blow comment explains it.
Quoted 8 lines
Thanks,
ChaoQuoted 6 lines
Thanks,
Quoted 3 lines
arch/x86/kvm/x86.c +0
-0/* * If the gfn and userspace address are not aligned wrt each * other, disable large page support for this slot. Signature · 1 line
Isaku Yamahata <[email protected]>
Sean Christopherson
maintainerRe: [PATCH v10 7/9] KVM: Update lpage info when private/shared memory are mixed
In reply to Chao Pengarch/x86/include/asm/kvm_host.h line 41
Quoted 9 lines · Chao Peng
arch/x86/include/asm/kvm_host.h +1
-0@@ -38,6 +38,7 @@#include <asm/hyperv-tlfs.h> #define __KVM_HAVE_ARCH_VCPU_DEBUGFS +#define __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES
No need for this, I think we should just make it mandatory to implement the arch hook when CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES=y. If another arch gains support for mem attributes and doesn't need the hook, then we can simply add a weak helper (or maybe add a #define then if we feel that's the way to go).
arch/x86/include/asm/kvm_host.h struct kvm_vcpu_arch { · line 1019
Quoted 11 lines
arch/x86/include/asm/kvm_host.h +5
-0#define KVM_MAX_VCPUS 1024@@ -1011,6 +1012,13 @@ struct kvm_vcpu_arch {#endif }; +/* +* Use a bit in disallow_lpage to indicate private/shared pages mixed at the +* level. The remaining bits are used as a reference count. +*/ +#define KVM_LPAGE_PRIVATE_SHARED_MIXED (1U << 31)
Similar to the need to unmap, I think we should just say "mixed" and ignore the private vs. shared, i.e. make this a flag for all memory attributes.
arch/x86/include/asm/kvm_host.h struct kvm_vcpu_arch { · line 1020
Quoted 1 line
arch/x86/include/asm/kvm_host.h +1
-0+#define KVM_LPAGE_COUNT_MAX ((1U << 31) - 1)
"MAX" is technically correct, but it's more of a mask. I think we can make it a moot point though. There's no need to mask the count, we just want to assert that adjusting the counting doesn't change the flag.
I would also say throw these defines into mmu.c, at least pending the bug fix for kvm_alloc_memslot_metadata() (more on that below).
Quoted 12 lines
arch/x86/include/asm/kvm_host.h +0
-0struct kvm_lpage_info {int disallow_lpage; };arch/x86/kvm/mmu/mmu.c +1
-0@@ -763,11 +763,16 @@ static void update_gfn_disallow_lpage_count(const struct kvm_memory_slot *slot, {struct kvm_lpage_info *linfo;int i; +int disallow_count;
arch/x86/kvm/mmu/mmu.c static void update_gfn_disallow_lpage_count(const struct kvm_memory_slot *slot, · line 770
Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +5
-1for (i =PG_LEVEL_2M ; i <=KVM_MAX_HUGEPAGE_LEVEL ; ++i) { linfo =lpage_info_slot (gfn, slot, i); + + disallow_count = linfo->disallow_lpage &KVM_LPAGE_COUNT_MAX ; +WARN_ON (disallow_count + count <0 || + disallow_count >KVM_LPAGE_COUNT_MAX - count); + linfo->disallow_lpage += count; -WARN_ON (linfo->disallow_lpage <0 );
It's been a long week so don't trust my math, but I believe this can simply be:
old = linfo->disallow_lpage; linfo->disallow_lpage += count; WARN_ON_ONCE((old ^ linfo->disallow_lpage) & KVM_LPAGE_MIXED_FLAG);
Quoted 23 lines
arch/x86/kvm/mmu/mmu.c +16
-0} }@@ -6986,3 +6991,130 @@ void kvm_mmu_pre_destroy_vm(struct kvm *kvm)if (kvm->arch.nx_huge_page_recovery_thread)kthread_stop (kvm->arch.nx_huge_page_recovery_thread); } + +static bool linfo_is_mixed (struct kvm_lpage_info *linfo) +{ +return linfo->disallow_lpage &KVM_LPAGE_PRIVATE_SHARED_MIXED ; +} + +static void linfo_set_mixed (gfn_t gfn,struct kvm_memory_slot *slot, +int level,bool mixed) +{ +struct kvm_lpage_info *linfo =lpage_info_slot (gfn, slot, level); + +if (mixed) + linfo->disallow_lpage |=KVM_LPAGE_PRIVATE_SHARED_MIXED ; +else + linfo->disallow_lpage &= ~KVM_LPAGE_PRIVATE_SHARED_MIXED ; +}
arch/x86/kvm/mmu/mmu.c void kvm_mmu_pre_destroy_vm(struct kvm *kvm) · line 7019
Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +10
-0+ +static bool is_expected_attr_entry (void *entry,unsigned long expected_attrs) +{ +bool expect_private = expected_attrs &KVM_MEMORY_ATTRIBUTE_PRIVATE ; + +if (xa_to_value (entry) &KVM_MEMORY_ATTRIBUTE_PRIVATE ) { +if (!expect_private) +return false; + }else if (expect_private) +return false;
This is messy. If we drop the private vs. shared specifity, this can go away if we add a helper to get attributes
static inline unsigned long kvm_get_memory_attributes(struct kvm *kvm, gfn_t gfn)
{
return xa_to_value(xa_load(&kvm->mem_attr_array, gfn));
}and then we can do
if (KVM_BUG_ON(gfn != xas.xa_index, kvm) ||
attrs != kvm_get_memory_attributes(kvm, gfn)) {
mixed = true;
break;
}and
if (linfo_is_mixed(lpage_info_slot(gfn, slot, level - 1)) || attrs != kvm_get_memory_attributes(kvm, gfn)) return true;
Quoted 9 lines
arch/x86/kvm/mmu/mmu.c +9
-0+ +return true; +} + +static bool mem_attrs_mixed_2m (struct kvm *kvm,unsigned long attrs, +gfn_t start,gfn_t end) +{ +XA_STATE (xas, &kvm->mem_attr_array, start); +gfn_t gfn = start;
arch/x86/kvm/mmu/mmu.c void kvm_mmu_pre_destroy_vm(struct kvm *kvm) · line 7038
Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +10
-0+void *entry; +bool mixed = false; + +rcu_read_lock (); + entry =xas_load (&xas); +while (gfn < end) { +if (xas_retry (&xas, entry)) +continue ; + +KVM_BUG_ON (gfn != xas.xa_index, kvm);
As above, I think it's worth bailing immediately if there's a mismatch.
Quoted 5 lines
arch/x86/kvm/mmu/mmu.c +5
-0+ +if (!is_expected_attr_entry (entry, attrs)) { + mixed = true; +break ; + }
arch/x86/kvm/mmu/mmu.c void kvm_mmu_pre_destroy_vm(struct kvm *kvm) · line 7053
Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +10
-0+ + entry =xas_next (&xas); + gfn++; + } + +rcu_read_unlock (); +return mixed; +} + +static bool mem_attrs_mixed(struct kvm *kvm, struct kvm_memory_slot *slot,
s/mem_attrs_mixed/has_mixed_attrs to make it clear this is querying, not setting. And has_mixed_attrs_2m() above.
arch/x86/kvm/mmu/mmu.c void kvm_mmu_pre_destroy_vm(struct kvm *kvm) · line 7062
Quoted 9 lines
arch/x86/kvm/mmu/mmu.c +9
-0+ int level, unsigned long attrs, + gfn_t start, gfn_t end) +{ + unsigned long gfn; + +if (level ==PG_LEVEL_2M ) +return mem_attrs_mixed_2m (kvm, attrs, start, end); + +for (gfn = start; gfn < end; gfn +=KVM_PAGES_PER_HPAGE (level -1 ))
Curly braces needed on the for-loop.
Quoted 45 lines
arch/x86/kvm/mmu/mmu.c +45
-0+if (linfo_is_mixed (lpage_info_slot (gfn, slot, level -1 )) || + !is_expected_attr_entry (xa_load (&kvm->mem_attr_array, gfn), + attrs)) +return true; +return false; +} + +static void kvm_update_lpage_private_shared_mixed (struct kvm *kvm, +struct kvm_memory_slot *slot, +unsigned long attrs, +gfn_t start,gfn_t end) +{ +unsigned long pages, mask; +gfn_t gfn, gfn_end, first, last; +int level; +bool mixed; + +/* +* The sequence matters here: we set the higher level basing on the +* lower level's scanning result. +*/ +for (level =PG_LEVEL_2M ; level <=KVM_MAX_HUGEPAGE_LEVEL ; level++) { + pages =KVM_PAGES_PER_HPAGE (level); + mask = ~(pages -1 ); + first = start & mask; + last = (end -1 ) & mask; + +/* +* We only need to scan the head and tail page, for middle pages +* we know they will not be mixed. +*/ + gfn =max (first, slot->base_gfn); + gfn_end =min (first + pages, slot->base_gfn + slot->npages); + mixed =mem_attrs_mixed (kvm, slot, level, attrs, gfn, gfn_end); +linfo_set_mixed (gfn, slot, level, mixed); + +if (first == last) +return ; + +for (gfn = first + pages; gfn < last; gfn += pages) +linfo_set_mixed (gfn, slot, level, false); + + gfn = last; + gfn_end =min (last + pages, slot->base_gfn + slot->npages); + mixed =mem_attrs_mixed (kvm, slot, level, attrs, gfn, gfn_end);
arch/x86/kvm/mmu/mmu.c void kvm_mmu_pre_destroy_vm(struct kvm *kvm) · line 7117
Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +10
-0+linfo_set_mixed (gfn, slot, level, mixed); + } +} + +void kvm_arch_set_memory_attributes (struct kvm *kvm, +struct kvm_memory_slot *slot, +unsigned long attrs, +gfn_t start,gfn_t end) +{ +if (kvm_slot_can_be_private (slot))
Make this an early return optimization, with a comment explaining that KVM x86 doesn't yet support other attributes.
/* * KVM x86 currently only supports KVM_MEMORY_ATTRIBUTE_PRIVATE, skip * the slot if the slot will never consume the PRIVATE attribute. */ if (!kvm_slot_can_be_private(slot)) return;
arch/x86/kvm/x86.c static int kvm_alloc_memslot_metadata(struct kvm *kvm, · line 12366
Quoted 13 lines
arch/x86/kvm/mmu/mmu.c +3
-0+kvm_update_lpage_private_shared_mixed (kvm, slot, attrs, + start, end); +}arch/x86/kvm/x86.c +2
-0@@ -12362,6 +12362,8 @@ static int kvm_alloc_memslot_metadata(struct kvm *kvm,if ((slot->base_gfn + npages) & (KVM_PAGES_PER_HPAGE (level) -1 )) linfo[lpages -1 ].disallow_lpage =1 ; ugfn = slot->userspace_addr >>PAGE_SHIFT ; +if (kvm_slot_can_be_private (slot)) + ugfn |= slot->restricted_offset >>PAGE_SHIFT ;
I would rather reject memslot if the gfn has lesser alignment than the offset. I'm totally ok with this approach _if_ there's a use case. Until such a use case presents itself, I would rather be conservative from a uAPI perspective.
Quoted 8 lines
arch/x86/kvm/x86.c +0
-0/* * If the gfn and userspace address are not aligned wrt each * other, disable large page support for this slot. include/linux/kvm_host.h +0
-0
include/linux/kvm_host.h struct kvm_memory_slot { · line 597
Quoted 6 lines
include/linux/kvm_host.h +3
-0@@ -592,6 +592,11 @@ struct kvm_memory_slot {struct restrictedmem_notifier notifier; }; +static inline bool kvm_slot_can_be_private (const struct kvm_memory_slot *slot) +{ +return slot && (slot->flags &KVM_MEM_PRIVATE );
KVM_MEM_PRIVATE should really be defined only when private memory is exposed to userspace. For this patch, even though it means we have untestable code, I think it makes sense to "return false".
Quoted 12 lines
include/linux/kvm_host.h +5
-0+} +static inline bool kvm_slot_dirty_track_enabled (const struct kvm_memory_slot *slot) {return slot->flags &KVM_MEM_LOG_DIRTY_PAGES ;@@ -2316,4 +2321,18 @@ static inline void kvm_account_pgtable_pages(void *virt, int nr)/* Max number of entries allowed for each kvm dirty ring */ #define KVM_DIRTY_RING_MAX_ENTRIES 65536 +#ifdef __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES +void kvm_arch_set_memory_attributes(struct kvm *kvm, + struct kvm_memory_slot *slot,
include/linux/kvm_host.h static inline void kvm_account_pgtable_pages(void *virt, int nr) · line 2336
Quoted 10 lines
include/linux/kvm_host.h +10
-0+unsigned long attrs, + gfn_t start, gfn_t end); +#else +static inline void kvm_arch_set_memory_attributes (struct kvm *kvm, +struct kvm_memory_slot *slot, +unsigned long attrs, +gfn_t start,gfn_t end) +{ +} +#endif /* __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES */
As above, no stub is necessary.
virt/kvm/kvm_main.c static u64 kvm_supported_mem_attributes(struct kvm *kvm) · line 2357
Quoted 10 lines
include/linux/kvm_host.h +0
-0#endif virt/kvm/kvm_main.c +0
-1@@ -2354,7 +2354,8 @@ static u64 kvm_supported_mem_attributes(struct kvm *kvm)return 0 ; } -static void kvm_unmap_mem_range (struct kvm *kvm,gfn_t start,gfn_t end)
Feedback for an earlier patch (to avoid churn): this should be kvm_mem_attrs_changed() or so now that this does more than just unmap.
virt/kvm/kvm_main.c static u64 kvm_supported_mem_attributes(struct kvm *kvm) · line 2358
Quoted 2 lines
virt/kvm/kvm_main.c +2
-0+static void kvm_unmap_mem_range (struct kvm *kvm,gfn_t start,gfn_t end, +unsigned long attrs)
Weird nit. I think we should keep the prototypes for kvm_mem_attrs_changed() and kvm_arch_set_memory_attributes() somewhat similar, i.e. squeeze in @attrs before @start.
Quoted 26 lines
virt/kvm/kvm_main.c +5
-1{struct kvm_gfn_range gfn_range;struct kvm_memory_slot *slot;@@ -2378,6 +2379,10 @@ static void kvm_unmap_mem_range(struct kvm *kvm, gfn_t start, gfn_t end) gfn_range.slot = slot; r |=kvm_unmap_gfn_range (kvm, &gfn_range); + +kvm_arch_set_memory_attributes (kvm, slot, attrs, + gfn_range.start, + gfn_range.end); } }@@ -2427,7 +2432,7 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, idx =srcu_read_lock (&kvm->srcu);KVM_MMU_LOCK (kvm);if (i > start) -kvm_unmap_mem_range (kvm, start, i); +kvm_unmap_mem_range (kvm, start, i, attrs->attributes);kvm_mmu_invalidate_end (kvm);KVM_MMU_UNLOCK (kvm);srcu_read_unlock (&kvm->srcu, idx);Signature · 1 line
2.25.1
Sean Christopherson
maintainerRe: [PATCH v10 7/9] KVM: Update lpage info when private/shared memory are mixed
In reply to Chao Pengarch/x86/kvm/x86.c static int kvm_alloc_memslot_metadata(struct kvm *kvm, · line 12369
Quoted 13 lines · Chao Peng
arch/x86/kvm/x86.c +2
-0@@ -12362,6 +12362,8 @@ static int kvm_alloc_memslot_metadata(struct kvm *kvm,if ((slot->base_gfn + npages) & (KVM_PAGES_PER_HPAGE (level) -1 )) linfo[lpages -1 ].disallow_lpage =1 ; ugfn = slot->userspace_addr >>PAGE_SHIFT ; +if (kvm_slot_can_be_private (slot)) + ugfn |= slot->restricted_offset >>PAGE_SHIFT ;/* * If the gfn and userspace address are not aligned wrt each * other, disable large page support for this slot.
Forgot to talk about the bug. This code needs to handle the scenario where a memslot is created with existing, non-uniform attributes. It might be a bit ugly (I didn't even try to write the code), but it's definitely possible, and since memslot updates are already slow I think it's best to handle things here.
In the meantime, I added this so we don't forget to fix it before merging.
#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
pr_crit_once("FIXME: Walk the memory attributes of the slot and set the mixed status appropriately");#endif
Re: [PATCH v10 7/9] KVM: Update lpage info when private/shared memory are mixed
Quoted 20 lines · Sean Christopherson
arch/x86/kvm/x86.c static int kvm_alloc_memslot_metadata(struct kvm *kvm, · line 12369
Quoted 13 lines · Chao Peng
arch/x86/kvm/x86.c +2
-0@@ -12362,6 +12362,8 @@ static int kvm_alloc_memslot_metadata(struct kvm *kvm,if ((slot->base_gfn + npages) & (KVM_PAGES_PER_HPAGE (level) -1 )) linfo[lpages -1 ].disallow_lpage =1 ; ugfn = slot->userspace_addr >>PAGE_SHIFT ; +if (kvm_slot_can_be_private (slot)) + ugfn |= slot->restricted_offset >>PAGE_SHIFT ;/* * If the gfn and userspace address are not aligned wrt each * other, disable large page support for this slot. Forgot to talk about the bug. This code needs to handle the scenario where a memslot is created with existing, non-uniform attributes. It might be a bit ugly (I didn't even try to write the code), but it's definitely possible, and since memslot updates are already slow I think it's best to handle things here.
Quoted 5 lines
In the meantime, I added this so we don't forget to fix it before merging.
#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES pr_crit_once("FIXME: Walk the memory attributes of the slot and set the mixed status appropriately");#endif
Here is the code to fix (based on your latest github repo).
arch/x86/include/asm/kvm_host.h +5 -0
@@ -2195,4 +2195,9 @@ int memslot_rmap_alloc(struct kvm_memory_slot *slot, unsigned long npages); KVM_X86_QUIRK_FIX_HYPERCALL_INSN | \ KVM_X86_QUIRK_MWAIT_NEVER_UD_FAULTS) +#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES +void kvm_memory_attributes_create_memslot (struct kvm *kvm, +struct kvm_memory_slot *slot); +#endif +#endif /* _ASM_X86_KVM_HOST_H */
arch/x86/kvm/mmu/mmu.c +39 -10
74 lines not on this page · raw message
arch/x86/kvm/x86.c +1 -1
@@ -12443,7 +12443,7 @@ static int kvm_alloc_memslot_metadata(struct kvm *kvm, }#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES -pr_crit_once ("FIXME: Walk the memory attributes of the slot and set the mixed status appropriately" ); +kvm_memory_attributes_create_memslot (kvm, slot);#endif if (kvm_page_track_create_memslot (kvm, slot, npages))
PATCH v10 8/9 KVM: Handle page fault for private memory
In reply to Chao PengA KVM_MEM_PRIVATE memslot can include both fd-based private memory and hva-based shared memory. Architecture code (like TDX code) can tell whether the on-going fault is private or not. This patch adds a 'is_private' field to kvm_page_fault to indicate this and architecture code is expected to set it.
To handle page fault for such memslot, the handling logic is different depending on whether the fault is private or shared. KVM checks if 'is_private' matches the host's view of the page (maintained in mem_attr_array).
- For a successful match, private pfn is obtained with restrictedmem_get_page() and shared pfn is obtained with existing get_user_pages().
- For a failed match, KVM causes a KVM_EXIT_MEMORY_FAULT exit to userspace. Userspace then can convert memory between private/shared in host's view and retry the fault.
--- arch/x86/kvm/mmu/mmu.c | 63 +++++++++++++++++++++++++++++++-- arch/x86/kvm/mmu/mmu_internal.h | 14 +++++++- arch/x86/kvm/mmu/mmutrace.h | 1 + arch/x86/kvm/mmu/tdp_mmu.c | 2 +- include/linux/kvm_host.h | 30 ++++++++++++++++ 5 files changed, 105 insertions(+), 5 deletions(-)
arch/x86/kvm/mmu/mmu.c +60 -3
@@ -3058,7 +3058,7 @@ static int host_pfn_mapping_level(struct kvm *kvm, gfn_t gfn,int kvm_mmu_max_mapping_level (struct kvm *kvm,const struct kvm_memory_slot *slot,gfn_t gfn, -int max_level) +int max_level,bool is_private) {struct kvm_lpage_info *linfo;int host_level;
@@ -3070,6 +3070,9 @@ int kvm_mmu_max_mapping_level(struct kvm *kvm,break ; } +if (is_private) +return max_level; +if (max_level ==PG_LEVEL_4K )return PG_LEVEL_4K ;
@@ -3098,7 +3101,8 @@ void kvm_mmu_hugepage_adjust(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault* level, which will be used to do precise, accurate accounting. */ fault->req_level =kvm_mmu_max_mapping_level (vcpu->kvm, slot, - fault->gfn, fault->max_level); + fault->gfn, fault->max_level, + fault->is_private);if (fault->req_level ==PG_LEVEL_4K || fault->huge_page_disallowed)return ;
@@ -4178,6 +4182,49 @@ void kvm_arch_async_page_ready(struct kvm_vcpu *vcpu, struct kvm_async_pf *work)kvm_mmu_do_page_fault (vcpu, work->cr2_or_gpa,0 , true); } +static inline u8 order_to_level (int order) +{ +BUILD_BUG_ON (KVM_MAX_HUGEPAGE_LEVEL >PG_LEVEL_1G ); + +if (order >=KVM_HPAGE_GFN_SHIFT (PG_LEVEL_1G )) +return PG_LEVEL_1G ; + +if (order >=KVM_HPAGE_GFN_SHIFT (PG_LEVEL_2M )) +return PG_LEVEL_2M ; + +return PG_LEVEL_4K ; +} + +static int kvm_do_memory_fault_exit (struct kvm_vcpu *vcpu, +struct kvm_page_fault *fault) +{ + vcpu->run->exit_reason =KVM_EXIT_MEMORY_FAULT ; +if (fault->is_private) + vcpu->run->memory.flags =KVM_MEMORY_EXIT_FLAG_PRIVATE ; +else + vcpu->run->memory.flags =0 ; + vcpu->run->memory.gpa = fault->gfn <<PAGE_SHIFT ; + vcpu->run->memory.size =PAGE_SIZE ; +return RET_PF_USER ; +} + +static int kvm_faultin_pfn_private (struct kvm_vcpu *vcpu, +struct kvm_page_fault *fault) +{ +int order; +struct kvm_memory_slot *slot = fault->slot; + +if (!kvm_slot_can_be_private (slot)) +return kvm_do_memory_fault_exit (vcpu, fault); + +if (kvm_restricted_mem_get_pfn (slot, fault->gfn, &fault->pfn, &order)) +return RET_PF_RETRY ; + + fault->max_level =min (order_to_level (order), fault->max_level); + fault->map_writable = !(slot->flags &KVM_MEM_READONLY ); +return RET_PF_CONTINUE ; +} +static int kvm_faultin_pfn (struct kvm_vcpu *vcpu,struct kvm_page_fault *fault) {struct kvm_memory_slot *slot = fault->slot;
@@ -4210,6 +4257,12 @@ static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)return RET_PF_EMULATE ; } +if (fault->is_private !=kvm_mem_is_private (vcpu->kvm, fault->gfn)) +return kvm_do_memory_fault_exit (vcpu, fault); + +if (fault->is_private) +return kvm_faultin_pfn_private (vcpu, fault); + async = false; fault->pfn = __gfn_to_pfn_memslot(slot, fault->gfn, false, false, &async, fault->write, &fault->map_writable,
@@ -5599,6 +5652,9 @@ int noinline kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 errreturn -EIO ; } +if (r ==RET_PF_USER ) +return 0 ; +if (r <0 )return r;if (r !=RET_PF_EMULATE )
@@ -6452,7 +6508,8 @@ static bool kvm_mmu_zap_collapsible_spte(struct kvm *kvm,*/ if (sp->role.direct && sp->role.level <kvm_mmu_max_mapping_level (kvm, slot, sp->gfn, -PG_LEVEL_NUM )) { +PG_LEVEL_NUM , + false)) {kvm_zap_one_rmap_spte (kvm, rmap_head, sptep);if (kvm_available_flush_tlb_with_range ())
arch/x86/kvm/mmu/mmu_internal.h +13 -1
@@ -189,6 +189,7 @@ struct kvm_page_fault {/* Derived from mmu and global state. */ const bool is_tdp; +const bool is_private;const bool nx_huge_page_workaround_enabled;/*
@@ -237,6 +238,7 @@ int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault);* RET_PF_RETRY: let CPU fault again on the address. * RET_PF_EMULATE: mmio page fault, emulate the instruction directly. * RET_PF_INVALID: the spte is invalid, let the real page fault path update it. +* RET_PF_USER: need to exit to userspace to handle this fault. * RET_PF_FIXED: The faulting entry has been fixed. * RET_PF_SPURIOUS: The faulting entry was already fixed, e.g. by another vCPU. *
@@ -253,6 +255,7 @@ enum {RET_PF_RETRY ,RET_PF_EMULATE ,RET_PF_INVALID , +RET_PF_USER ,RET_PF_FIXED , RET_PF_SPURIOUS, };
@@ -310,7 +313,7 @@ static inline int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,int kvm_mmu_max_mapping_level (struct kvm *kvm,const struct kvm_memory_slot *slot,gfn_t gfn, -int max_level); +int max_level,bool is_private);void kvm_mmu_hugepage_adjust (struct kvm_vcpu *vcpu,struct kvm_page_fault *fault);void disallowed_hugepage_adjust (struct kvm_page_fault *fault,u64 spte,int cur_level);
@@ -319,4 +322,13 @@ void *mmu_memory_cache_alloc(struct kvm_mmu_memory_cache *mc);void track_possible_nx_huge_page (struct kvm *kvm,struct kvm_mmu_page *sp);void untrack_possible_nx_huge_page (struct kvm *kvm,struct kvm_mmu_page *sp); +#ifndef CONFIG_HAVE_KVM_RESTRICTED_MEM +static inline int kvm_restricted_mem_get_pfn (struct kvm_memory_slot *slot, +gfn_t gfn,kvm_pfn_t *pfn,int *order) +{ +WARN_ON_ONCE (1 ); +return -EOPNOTSUPP ; +} +#endif /* CONFIG_HAVE_KVM_RESTRICTED_MEM */ +#endif /* __KVM_X86_MMU_INTERNAL_H */
arch/x86/kvm/mmu/mmutrace.h +1 -0
@@ -58,6 +58,7 @@ TRACE_DEFINE_ENUM(RET_PF_CONTINUE);TRACE_DEFINE_ENUM (RET_PF_RETRY );TRACE_DEFINE_ENUM (RET_PF_EMULATE );TRACE_DEFINE_ENUM (RET_PF_INVALID ); +TRACE_DEFINE_ENUM (RET_PF_USER );TRACE_DEFINE_ENUM (RET_PF_FIXED );TRACE_DEFINE_ENUM (RET_PF_SPURIOUS );
arch/x86/kvm/mmu/tdp_mmu.c +1 -1
@@ -1768,7 +1768,7 @@ static void zap_collapsible_spte_range(struct kvm *kvm,continue ; max_mapping_level =kvm_mmu_max_mapping_level (kvm, slot, - iter.gfn,PG_LEVEL_NUM ); + iter.gfn,PG_LEVEL_NUM , false);if (max_mapping_level < iter.level)continue ;
include/linux/kvm_host.h +30 -0
@@ -2335,4 +2335,34 @@ static inline void kvm_arch_set_memory_attributes(struct kvm *kvm, }#endif /* __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES */ +#ifdef CONFIG_HAVE_KVM_MEMORY_ATTRIBUTES +static inline bool kvm_mem_is_private (struct kvm *kvm,gfn_t gfn) +{ +return xa_to_value (xa_load (&kvm->mem_attr_array, gfn)) & +KVM_MEMORY_ATTRIBUTE_PRIVATE ; +} +#else +static inline bool kvm_mem_is_private (struct kvm *kvm,gfn_t gfn) +{ +return false; +} + +#endif /* CONFIG_HAVE_KVM_MEMORY_ATTRIBUTES */ + +#ifdef CONFIG_HAVE_KVM_RESTRICTED_MEM +static inline int kvm_restricted_mem_get_pfn (struct kvm_memory_slot *slot, +gfn_t gfn,kvm_pfn_t *pfn,int *order) +{ +int ret; +struct page *page; +pgoff_t index = gfn - slot->base_gfn + + (slot->restricted_offset >>PAGE_SHIFT ); + + ret =restrictedmem_get_page (slot->restricted_file, index, + &page, order); + *pfn =page_to_pfn (page); +return ret; +} +#endif /* CONFIG_HAVE_KVM_RESTRICTED_MEM */ +#endif
Signature · 1 line
2.25.1
Re: [PATCH v10 8/9] KVM: Handle page fault for private memory
Quoted 42 lines · Chao Peng
A
KVM_MEM_PRIVATEmemslot can include both fd-based private memory and hva-based shared memory. Architecture code (like TDX code) can tell whether the on-going fault is private or not. This patch adds a 'is_private' field tokvm_page_faultto indicate this and architecture code is expected to set it.To handle page fault for such memslot, the handling logic is different depending on whether the fault is private or shared. KVM checks if '
is_private' matches the host's view of the page (maintained inmem_attr_array).
- For a successful match, private pfn is obtained withrestrictedmem_get_page()and shared pfn is obtained with existingget_user_pages().
- For a failed match, KVM causes aKVM_EXIT_MEMORY_FAULTexit to userspace. Userspace then can convert memory between private/shared in host's view and retry the fault.Co-developed-by Yu Zhang <[email protected]>Signed-off-by Yu Zhang <[email protected]>Signed-off-by Chao Peng <[email protected]>--- arch/x86/kvm/mmu/mmu.c | 63 +++++++++++++++++++++++++++++++-- arch/x86/kvm/mmu/mmu_internal.h | 14 +++++++- arch/x86/kvm/mmu/mmutrace.h | 1 + arch/x86/kvm/mmu/tdp_mmu.c | 2 +- include/linux/kvm_host.h | 30 ++++++++++++++++ 5 files changed, 105 insertions(+), 5 deletions(-)arch/x86/kvm/mmu/mmu.c +1
-1@@ -3058,7 +3058,7 @@ static int host_pfn_mapping_level(struct kvm *kvm, gfn_t gfn,int kvm_mmu_max_mapping_level (struct kvm *kvm,const struct kvm_memory_slot *slot,gfn_t gfn, -int max_level) +int max_level,bool is_private) {struct kvm_lpage_info *linfo;int host_level;
arch/x86/kvm/mmu/mmu.c int kvm_mmu_max_mapping_level(struct kvm *kvm, · line 3074
Quoted 5 lines
arch/x86/kvm/mmu/mmu.c +2
-0@@ -3070,6 +3070,9 @@ int kvm_mmu_max_mapping_level(struct kvm *kvm,break ; } +if (is_private) +return max_level;
lpage mixed information already saved, so is that possible to query info->disallow_lpage without care 'is_private' ?
Quoted 216 lines
arch/x86/kvm/mmu/mmu.c +57
-2+if (max_level ==PG_LEVEL_4K )return PG_LEVEL_4K ;@@ -3098,7 +3101,8 @@ void kvm_mmu_hugepage_adjust(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault* level, which will be used to do precise, accurate accounting. */ fault->req_level =kvm_mmu_max_mapping_level (vcpu->kvm, slot, - fault->gfn, fault->max_level); + fault->gfn, fault->max_level, + fault->is_private);if (fault->req_level ==PG_LEVEL_4K || fault->huge_page_disallowed)return ;@@ -4178,6 +4182,49 @@ void kvm_arch_async_page_ready(struct kvm_vcpu *vcpu, struct kvm_async_pf *work)kvm_mmu_do_page_fault (vcpu, work->cr2_or_gpa,0 , true); } +static inline u8 order_to_level (int order) +{ +BUILD_BUG_ON (KVM_MAX_HUGEPAGE_LEVEL >PG_LEVEL_1G ); + +if (order >=KVM_HPAGE_GFN_SHIFT (PG_LEVEL_1G )) +return PG_LEVEL_1G ; + +if (order >=KVM_HPAGE_GFN_SHIFT (PG_LEVEL_2M )) +return PG_LEVEL_2M ; + +return PG_LEVEL_4K ; +} + +static int kvm_do_memory_fault_exit (struct kvm_vcpu *vcpu, +struct kvm_page_fault *fault) +{ + vcpu->run->exit_reason =KVM_EXIT_MEMORY_FAULT ; +if (fault->is_private) + vcpu->run->memory.flags =KVM_MEMORY_EXIT_FLAG_PRIVATE ; +else + vcpu->run->memory.flags =0 ; + vcpu->run->memory.gpa = fault->gfn <<PAGE_SHIFT ; + vcpu->run->memory.size =PAGE_SIZE ; +return RET_PF_USER ; +} + +static int kvm_faultin_pfn_private (struct kvm_vcpu *vcpu, +struct kvm_page_fault *fault) +{ +int order; +struct kvm_memory_slot *slot = fault->slot; + +if (!kvm_slot_can_be_private (slot)) +return kvm_do_memory_fault_exit (vcpu, fault); + +if (kvm_restricted_mem_get_pfn (slot, fault->gfn, &fault->pfn, &order)) +return RET_PF_RETRY ; + + fault->max_level =min (order_to_level (order), fault->max_level); + fault->map_writable = !(slot->flags &KVM_MEM_READONLY ); +return RET_PF_CONTINUE ; +} +static int kvm_faultin_pfn (struct kvm_vcpu *vcpu,struct kvm_page_fault *fault) {struct kvm_memory_slot *slot = fault->slot;@@ -4210,6 +4257,12 @@ static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)return RET_PF_EMULATE ; } +if (fault->is_private !=kvm_mem_is_private (vcpu->kvm, fault->gfn)) +return kvm_do_memory_fault_exit (vcpu, fault); + +if (fault->is_private) +return kvm_faultin_pfn_private (vcpu, fault); + async = false; fault->pfn = __gfn_to_pfn_memslot(slot, fault->gfn, false, false, &async, fault->write, &fault->map_writable,@@ -5599,6 +5652,9 @@ int noinline kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 errreturn -EIO ; } +if (r ==RET_PF_USER ) +return 0 ; +if (r <0 )return r;if (r !=RET_PF_EMULATE )@@ -6452,7 +6508,8 @@ static bool kvm_mmu_zap_collapsible_spte(struct kvm *kvm,*/ if (sp->role.direct && sp->role.level <kvm_mmu_max_mapping_level (kvm, slot, sp->gfn, -PG_LEVEL_NUM )) { +PG_LEVEL_NUM , + false)) {kvm_zap_one_rmap_spte (kvm, rmap_head, sptep);if (kvm_available_flush_tlb_with_range ())arch/x86/kvm/mmu/mmu_internal.h +13
-1@@ -189,6 +189,7 @@ struct kvm_page_fault {/* Derived from mmu and global state. */ const bool is_tdp; +const bool is_private;const bool nx_huge_page_workaround_enabled;/* @@ -237,6 +238,7 @@ int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault);* RET_PF_RETRY: let CPU fault again on the address. * RET_PF_EMULATE: mmio page fault, emulate the instruction directly. * RET_PF_INVALID: the spte is invalid, let the real page fault path update it. +* RET_PF_USER: need to exit to userspace to handle this fault. * RET_PF_FIXED: The faulting entry has been fixed. * RET_PF_SPURIOUS: The faulting entry was already fixed, e.g. by another vCPU. * @@ -253,6 +255,7 @@ enum {RET_PF_RETRY ,RET_PF_EMULATE ,RET_PF_INVALID , +RET_PF_USER ,RET_PF_FIXED , RET_PF_SPURIOUS, };@@ -310,7 +313,7 @@ static inline int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,int kvm_mmu_max_mapping_level (struct kvm *kvm,const struct kvm_memory_slot *slot,gfn_t gfn, -int max_level); +int max_level,bool is_private);void kvm_mmu_hugepage_adjust (struct kvm_vcpu *vcpu,struct kvm_page_fault *fault);void disallowed_hugepage_adjust (struct kvm_page_fault *fault,u64 spte,int cur_level);@@ -319,4 +322,13 @@ void *mmu_memory_cache_alloc(struct kvm_mmu_memory_cache *mc);void track_possible_nx_huge_page (struct kvm *kvm,struct kvm_mmu_page *sp);void untrack_possible_nx_huge_page (struct kvm *kvm,struct kvm_mmu_page *sp); +#ifndef CONFIG_HAVE_KVM_RESTRICTED_MEM +static inline int kvm_restricted_mem_get_pfn (struct kvm_memory_slot *slot, +gfn_t gfn,kvm_pfn_t *pfn,int *order) +{ +WARN_ON_ONCE (1 ); +return -EOPNOTSUPP ; +} +#endif /* CONFIG_HAVE_KVM_RESTRICTED_MEM */ +#endif /* __KVM_X86_MMU_INTERNAL_H */ arch/x86/kvm/mmu/mmutrace.h +1
-0@@ -58,6 +58,7 @@ TRACE_DEFINE_ENUM(RET_PF_CONTINUE);TRACE_DEFINE_ENUM (RET_PF_RETRY );TRACE_DEFINE_ENUM (RET_PF_EMULATE );TRACE_DEFINE_ENUM (RET_PF_INVALID ); +TRACE_DEFINE_ENUM (RET_PF_USER );TRACE_DEFINE_ENUM (RET_PF_FIXED );TRACE_DEFINE_ENUM (RET_PF_SPURIOUS );arch/x86/kvm/mmu/tdp_mmu.c +1
-1@@ -1768,7 +1768,7 @@ static void zap_collapsible_spte_range(struct kvm *kvm,continue ; max_mapping_level =kvm_mmu_max_mapping_level (kvm, slot, - iter.gfn,PG_LEVEL_NUM ); + iter.gfn,PG_LEVEL_NUM , false);if (max_mapping_level < iter.level)continue ;include/linux/kvm_host.h +30
-1@@ -2335,4 +2335,34 @@ static inline void kvm_arch_set_memory_attributes(struct kvm *kvm, }#endif /* __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES */ +#ifdef CONFIG_HAVE_KVM_MEMORY_ATTRIBUTES +static inline bool kvm_mem_is_private (struct kvm *kvm,gfn_t gfn) +{ +return xa_to_value (xa_load (&kvm->mem_attr_array, gfn)) & +KVM_MEMORY_ATTRIBUTE_PRIVATE ; +} +#else +static inline bool kvm_mem_is_private (struct kvm *kvm,gfn_t gfn) +{ +return false; +} + +#endif /* CONFIG_HAVE_KVM_MEMORY_ATTRIBUTES */ + +#ifdef CONFIG_HAVE_KVM_RESTRICTED_MEM +static inline int kvm_restricted_mem_get_pfn (struct kvm_memory_slot *slot, +gfn_t gfn,kvm_pfn_t *pfn,int *order) +{ +int ret; +struct page *page; +pgoff_t index = gfn - slot->base_gfn + + (slot->restricted_offset >>PAGE_SHIFT ); + + ret =restrictedmem_get_page (slot->restricted_file, index, + &page, order); + *pfn =page_to_pfn (page); +return ret; +} +#endif /* CONFIG_HAVE_KVM_RESTRICTED_MEM */ +#endif --2.25.1
Re: [PATCH v10 8/9] KVM: Handle page fault for private memory
Quoted 46 lines · Yuan Yao
Quoted 45 lines · Chao Peng
A
KVM_MEM_PRIVATEmemslot can include both fd-based private memory and hva-based shared memory. Architecture code (like TDX code) can tell whether the on-going fault is private or not. This patch adds a 'is_private' field tokvm_page_faultto indicate this and architecture code is expected to set it.To handle page fault for such memslot, the handling logic is different depending on whether the fault is private or shared. KVM checks if '
is_private' matches the host's view of the page (maintained inmem_attr_array).
- For a successful match, private pfn is obtained withrestrictedmem_get_page()and shared pfn is obtained with existingget_user_pages().
- For a failed match, KVM causes aKVM_EXIT_MEMORY_FAULTexit to userspace. Userspace then can convert memory between private/shared in host's view and retry the fault.Co-developed-by Yu Zhang <[email protected]>Signed-off-by Yu Zhang <[email protected]>Signed-off-by Chao Peng <[email protected]>--- arch/x86/kvm/mmu/mmu.c | 63 +++++++++++++++++++++++++++++++-- arch/x86/kvm/mmu/mmu_internal.h | 14 +++++++- arch/x86/kvm/mmu/mmutrace.h | 1 + arch/x86/kvm/mmu/tdp_mmu.c | 2 +- include/linux/kvm_host.h | 30 ++++++++++++++++ 5 files changed, 105 insertions(+), 5 deletions(-)arch/x86/kvm/mmu/mmu.c +1
-1@@ -3058,7 +3058,7 @@ static int host_pfn_mapping_level(struct kvm *kvm, gfn_t gfn,int kvm_mmu_max_mapping_level (struct kvm *kvm,const struct kvm_memory_slot *slot,gfn_t gfn, -int max_level) +int max_level,bool is_private) {struct kvm_lpage_info *linfo;int host_level;@@ -3070,6 +3070,9 @@ int kvm_mmu_max_mapping_level(struct kvm *kvm,break ; }
Quoted 5 lines
arch/x86/kvm/mmu/mmu.c int kvm_mmu_max_mapping_level(struct kvm *kvm, · line 3074
Quoted 2 lines
arch/x86/kvm/mmu/mmu.c +2
-0+if (is_private) +return max_level;lpage mixed information already saved, so is that possible to query
info->disallow_lpagewithout care 'is_private' ?
Actually we already queried info->disallow_lpage just before this sentence. The check is needed because later in the function we call host_pfn_mapping_level() which is shared memory specific.
Thanks,
Chao
Quoted 217 lines
Quoted 217 lines
arch/x86/kvm/mmu/mmu.c +57
-2+if (max_level ==PG_LEVEL_4K )return PG_LEVEL_4K ;@@ -3098,7 +3101,8 @@ void kvm_mmu_hugepage_adjust(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault* level, which will be used to do precise, accurate accounting. */ fault->req_level =kvm_mmu_max_mapping_level (vcpu->kvm, slot, - fault->gfn, fault->max_level); + fault->gfn, fault->max_level, + fault->is_private);if (fault->req_level ==PG_LEVEL_4K || fault->huge_page_disallowed)return ;@@ -4178,6 +4182,49 @@ void kvm_arch_async_page_ready(struct kvm_vcpu *vcpu, struct kvm_async_pf *work)kvm_mmu_do_page_fault (vcpu, work->cr2_or_gpa,0 , true); } +static inline u8 order_to_level (int order) +{ +BUILD_BUG_ON (KVM_MAX_HUGEPAGE_LEVEL >PG_LEVEL_1G ); + +if (order >=KVM_HPAGE_GFN_SHIFT (PG_LEVEL_1G )) +return PG_LEVEL_1G ; + +if (order >=KVM_HPAGE_GFN_SHIFT (PG_LEVEL_2M )) +return PG_LEVEL_2M ; + +return PG_LEVEL_4K ; +} + +static int kvm_do_memory_fault_exit (struct kvm_vcpu *vcpu, +struct kvm_page_fault *fault) +{ + vcpu->run->exit_reason =KVM_EXIT_MEMORY_FAULT ; +if (fault->is_private) + vcpu->run->memory.flags =KVM_MEMORY_EXIT_FLAG_PRIVATE ; +else + vcpu->run->memory.flags =0 ; + vcpu->run->memory.gpa = fault->gfn <<PAGE_SHIFT ; + vcpu->run->memory.size =PAGE_SIZE ; +return RET_PF_USER ; +} + +static int kvm_faultin_pfn_private (struct kvm_vcpu *vcpu, +struct kvm_page_fault *fault) +{ +int order; +struct kvm_memory_slot *slot = fault->slot; + +if (!kvm_slot_can_be_private (slot)) +return kvm_do_memory_fault_exit (vcpu, fault); + +if (kvm_restricted_mem_get_pfn (slot, fault->gfn, &fault->pfn, &order)) +return RET_PF_RETRY ; + + fault->max_level =min (order_to_level (order), fault->max_level); + fault->map_writable = !(slot->flags &KVM_MEM_READONLY ); +return RET_PF_CONTINUE ; +} +static int kvm_faultin_pfn (struct kvm_vcpu *vcpu,struct kvm_page_fault *fault) {struct kvm_memory_slot *slot = fault->slot;@@ -4210,6 +4257,12 @@ static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)return RET_PF_EMULATE ; } +if (fault->is_private !=kvm_mem_is_private (vcpu->kvm, fault->gfn)) +return kvm_do_memory_fault_exit (vcpu, fault); + +if (fault->is_private) +return kvm_faultin_pfn_private (vcpu, fault); + async = false; fault->pfn = __gfn_to_pfn_memslot(slot, fault->gfn, false, false, &async, fault->write, &fault->map_writable,@@ -5599,6 +5652,9 @@ int noinline kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 errreturn -EIO ; } +if (r ==RET_PF_USER ) +return 0 ; +if (r <0 )return r;if (r !=RET_PF_EMULATE )@@ -6452,7 +6508,8 @@ static bool kvm_mmu_zap_collapsible_spte(struct kvm *kvm,*/ if (sp->role.direct && sp->role.level <kvm_mmu_max_mapping_level (kvm, slot, sp->gfn, -PG_LEVEL_NUM )) { +PG_LEVEL_NUM , + false)) {kvm_zap_one_rmap_spte (kvm, rmap_head, sptep);if (kvm_available_flush_tlb_with_range ())arch/x86/kvm/mmu/mmu_internal.h +13
-1@@ -189,6 +189,7 @@ struct kvm_page_fault {/* Derived from mmu and global state. */ const bool is_tdp; +const bool is_private;const bool nx_huge_page_workaround_enabled;/* @@ -237,6 +238,7 @@ int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault);* RET_PF_RETRY: let CPU fault again on the address. * RET_PF_EMULATE: mmio page fault, emulate the instruction directly. * RET_PF_INVALID: the spte is invalid, let the real page fault path update it. +* RET_PF_USER: need to exit to userspace to handle this fault. * RET_PF_FIXED: The faulting entry has been fixed. * RET_PF_SPURIOUS: The faulting entry was already fixed, e.g. by another vCPU. * @@ -253,6 +255,7 @@ enum {RET_PF_RETRY ,RET_PF_EMULATE ,RET_PF_INVALID , +RET_PF_USER ,RET_PF_FIXED , RET_PF_SPURIOUS, };@@ -310,7 +313,7 @@ static inline int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,int kvm_mmu_max_mapping_level (struct kvm *kvm,const struct kvm_memory_slot *slot,gfn_t gfn, -int max_level); +int max_level,bool is_private);void kvm_mmu_hugepage_adjust (struct kvm_vcpu *vcpu,struct kvm_page_fault *fault);void disallowed_hugepage_adjust (struct kvm_page_fault *fault,u64 spte,int cur_level);@@ -319,4 +322,13 @@ void *mmu_memory_cache_alloc(struct kvm_mmu_memory_cache *mc);void track_possible_nx_huge_page (struct kvm *kvm,struct kvm_mmu_page *sp);void untrack_possible_nx_huge_page (struct kvm *kvm,struct kvm_mmu_page *sp); +#ifndef CONFIG_HAVE_KVM_RESTRICTED_MEM +static inline int kvm_restricted_mem_get_pfn (struct kvm_memory_slot *slot, +gfn_t gfn,kvm_pfn_t *pfn,int *order) +{ +WARN_ON_ONCE (1 ); +return -EOPNOTSUPP ; +} +#endif /* CONFIG_HAVE_KVM_RESTRICTED_MEM */ +#endif /* __KVM_X86_MMU_INTERNAL_H */ arch/x86/kvm/mmu/mmutrace.h +1
-0@@ -58,6 +58,7 @@ TRACE_DEFINE_ENUM(RET_PF_CONTINUE);TRACE_DEFINE_ENUM (RET_PF_RETRY );TRACE_DEFINE_ENUM (RET_PF_EMULATE );TRACE_DEFINE_ENUM (RET_PF_INVALID ); +TRACE_DEFINE_ENUM (RET_PF_USER );TRACE_DEFINE_ENUM (RET_PF_FIXED );TRACE_DEFINE_ENUM (RET_PF_SPURIOUS );arch/x86/kvm/mmu/tdp_mmu.c +1
-1@@ -1768,7 +1768,7 @@ static void zap_collapsible_spte_range(struct kvm *kvm,continue ; max_mapping_level =kvm_mmu_max_mapping_level (kvm, slot, - iter.gfn,PG_LEVEL_NUM ); + iter.gfn,PG_LEVEL_NUM , false);if (max_mapping_level < iter.level)continue ;include/linux/kvm_host.h +30
-1@@ -2335,4 +2335,34 @@ static inline void kvm_arch_set_memory_attributes(struct kvm *kvm, }#endif /* __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES */ +#ifdef CONFIG_HAVE_KVM_MEMORY_ATTRIBUTES +static inline bool kvm_mem_is_private (struct kvm *kvm,gfn_t gfn) +{ +return xa_to_value (xa_load (&kvm->mem_attr_array, gfn)) & +KVM_MEMORY_ATTRIBUTE_PRIVATE ; +} +#else +static inline bool kvm_mem_is_private (struct kvm *kvm,gfn_t gfn) +{ +return false; +} + +#endif /* CONFIG_HAVE_KVM_MEMORY_ATTRIBUTES */ + +#ifdef CONFIG_HAVE_KVM_RESTRICTED_MEM +static inline int kvm_restricted_mem_get_pfn (struct kvm_memory_slot *slot, +gfn_t gfn,kvm_pfn_t *pfn,int *order) +{ +int ret; +struct page *page; +pgoff_t index = gfn - slot->base_gfn + + (slot->restricted_offset >>PAGE_SHIFT ); + + ret =restrictedmem_get_page (slot->restricted_file, index, + &page, order); + *pfn =page_to_pfn (page); +return ret; +} +#endif /* CONFIG_HAVE_KVM_RESTRICTED_MEM */ +#endif --2.25.1
Re: [PATCH v10 8/9] KVM: Handle page fault for private memory
Quoted 53 lines · Chao Peng
Quoted 46 lines · Yuan Yao
Quoted 45 lines · Chao Peng
A
KVM_MEM_PRIVATEmemslot can include both fd-based private memory and hva-based shared memory. Architecture code (like TDX code) can tell whether the on-going fault is private or not. This patch adds a 'is_private' field tokvm_page_faultto indicate this and architecture code is expected to set it.To handle page fault for such memslot, the handling logic is different depending on whether the fault is private or shared. KVM checks if '
is_private' matches the host's view of the page (maintained inmem_attr_array).
- For a successful match, private pfn is obtained withrestrictedmem_get_page()and shared pfn is obtained with existingget_user_pages().
- For a failed match, KVM causes aKVM_EXIT_MEMORY_FAULTexit to userspace. Userspace then can convert memory between private/shared in host's view and retry the fault.Co-developed-by Yu Zhang <[email protected]>Signed-off-by Yu Zhang <[email protected]>Signed-off-by Chao Peng <[email protected]>--- arch/x86/kvm/mmu/mmu.c | 63 +++++++++++++++++++++++++++++++-- arch/x86/kvm/mmu/mmu_internal.h | 14 +++++++- arch/x86/kvm/mmu/mmutrace.h | 1 + arch/x86/kvm/mmu/tdp_mmu.c | 2 +- include/linux/kvm_host.h | 30 ++++++++++++++++ 5 files changed, 105 insertions(+), 5 deletions(-)arch/x86/kvm/mmu/mmu.c +1
-1@@ -3058,7 +3058,7 @@ static int host_pfn_mapping_level(struct kvm *kvm, gfn_t gfn,int kvm_mmu_max_mapping_level (struct kvm *kvm,const struct kvm_memory_slot *slot,gfn_t gfn, -int max_level) +int max_level,bool is_private) {struct kvm_lpage_info *linfo;int host_level;@@ -3070,6 +3070,9 @@ int kvm_mmu_max_mapping_level(struct kvm *kvm,break ; }Quoted 5 lines
arch/x86/kvm/mmu/mmu.c int kvm_mmu_max_mapping_level(struct kvm *kvm, · line 3074
Quoted 2 lines
arch/x86/kvm/mmu/mmu.c +2
-0+if (is_private) +return max_level;lpage mixed information already saved, so is that possible to query
info->disallow_lpagewithout care 'is_private' ?
Quoted 3 lines
Actually we already queried
info->disallow_lpagejust before this sentence. The check is needed because later in the function we callhost_pfn_mapping_level()which is shared memory specific.
You're right. We can't get mapping level info for private page from host_pfn_mapping_level().
Quoted 220 lines
Thanks,
ChaoQuoted 217 lines
Quoted 217 lines
arch/x86/kvm/mmu/mmu.c +57
-2+if (max_level ==PG_LEVEL_4K )return PG_LEVEL_4K ;@@ -3098,7 +3101,8 @@ void kvm_mmu_hugepage_adjust(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault* level, which will be used to do precise, accurate accounting. */ fault->req_level =kvm_mmu_max_mapping_level (vcpu->kvm, slot, - fault->gfn, fault->max_level); + fault->gfn, fault->max_level, + fault->is_private);if (fault->req_level ==PG_LEVEL_4K || fault->huge_page_disallowed)return ;@@ -4178,6 +4182,49 @@ void kvm_arch_async_page_ready(struct kvm_vcpu *vcpu, struct kvm_async_pf *work)kvm_mmu_do_page_fault (vcpu, work->cr2_or_gpa,0 , true); } +static inline u8 order_to_level (int order) +{ +BUILD_BUG_ON (KVM_MAX_HUGEPAGE_LEVEL >PG_LEVEL_1G ); + +if (order >=KVM_HPAGE_GFN_SHIFT (PG_LEVEL_1G )) +return PG_LEVEL_1G ; + +if (order >=KVM_HPAGE_GFN_SHIFT (PG_LEVEL_2M )) +return PG_LEVEL_2M ; + +return PG_LEVEL_4K ; +} + +static int kvm_do_memory_fault_exit (struct kvm_vcpu *vcpu, +struct kvm_page_fault *fault) +{ + vcpu->run->exit_reason =KVM_EXIT_MEMORY_FAULT ; +if (fault->is_private) + vcpu->run->memory.flags =KVM_MEMORY_EXIT_FLAG_PRIVATE ; +else + vcpu->run->memory.flags =0 ; + vcpu->run->memory.gpa = fault->gfn <<PAGE_SHIFT ; + vcpu->run->memory.size =PAGE_SIZE ; +return RET_PF_USER ; +} + +static int kvm_faultin_pfn_private (struct kvm_vcpu *vcpu, +struct kvm_page_fault *fault) +{ +int order; +struct kvm_memory_slot *slot = fault->slot; + +if (!kvm_slot_can_be_private (slot)) +return kvm_do_memory_fault_exit (vcpu, fault); + +if (kvm_restricted_mem_get_pfn (slot, fault->gfn, &fault->pfn, &order)) +return RET_PF_RETRY ; + + fault->max_level =min (order_to_level (order), fault->max_level); + fault->map_writable = !(slot->flags &KVM_MEM_READONLY ); +return RET_PF_CONTINUE ; +} +static int kvm_faultin_pfn (struct kvm_vcpu *vcpu,struct kvm_page_fault *fault) {struct kvm_memory_slot *slot = fault->slot;@@ -4210,6 +4257,12 @@ static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)return RET_PF_EMULATE ; } +if (fault->is_private !=kvm_mem_is_private (vcpu->kvm, fault->gfn)) +return kvm_do_memory_fault_exit (vcpu, fault); + +if (fault->is_private) +return kvm_faultin_pfn_private (vcpu, fault); + async = false; fault->pfn = __gfn_to_pfn_memslot(slot, fault->gfn, false, false, &async, fault->write, &fault->map_writable,@@ -5599,6 +5652,9 @@ int noinline kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 errreturn -EIO ; } +if (r ==RET_PF_USER ) +return 0 ; +if (r <0 )return r;if (r !=RET_PF_EMULATE )@@ -6452,7 +6508,8 @@ static bool kvm_mmu_zap_collapsible_spte(struct kvm *kvm,*/ if (sp->role.direct && sp->role.level <kvm_mmu_max_mapping_level (kvm, slot, sp->gfn, -PG_LEVEL_NUM )) { +PG_LEVEL_NUM , + false)) {kvm_zap_one_rmap_spte (kvm, rmap_head, sptep);if (kvm_available_flush_tlb_with_range ())arch/x86/kvm/mmu/mmu_internal.h +13
-1@@ -189,6 +189,7 @@ struct kvm_page_fault {/* Derived from mmu and global state. */ const bool is_tdp; +const bool is_private;const bool nx_huge_page_workaround_enabled;/* @@ -237,6 +238,7 @@ int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault);* RET_PF_RETRY: let CPU fault again on the address. * RET_PF_EMULATE: mmio page fault, emulate the instruction directly. * RET_PF_INVALID: the spte is invalid, let the real page fault path update it. +* RET_PF_USER: need to exit to userspace to handle this fault. * RET_PF_FIXED: The faulting entry has been fixed. * RET_PF_SPURIOUS: The faulting entry was already fixed, e.g. by another vCPU. * @@ -253,6 +255,7 @@ enum {RET_PF_RETRY ,RET_PF_EMULATE ,RET_PF_INVALID , +RET_PF_USER ,RET_PF_FIXED , RET_PF_SPURIOUS, };@@ -310,7 +313,7 @@ static inline int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,int kvm_mmu_max_mapping_level (struct kvm *kvm,const struct kvm_memory_slot *slot,gfn_t gfn, -int max_level); +int max_level,bool is_private);void kvm_mmu_hugepage_adjust (struct kvm_vcpu *vcpu,struct kvm_page_fault *fault);void disallowed_hugepage_adjust (struct kvm_page_fault *fault,u64 spte,int cur_level);@@ -319,4 +322,13 @@ void *mmu_memory_cache_alloc(struct kvm_mmu_memory_cache *mc);void track_possible_nx_huge_page (struct kvm *kvm,struct kvm_mmu_page *sp);void untrack_possible_nx_huge_page (struct kvm *kvm,struct kvm_mmu_page *sp); +#ifndef CONFIG_HAVE_KVM_RESTRICTED_MEM +static inline int kvm_restricted_mem_get_pfn (struct kvm_memory_slot *slot, +gfn_t gfn,kvm_pfn_t *pfn,int *order) +{ +WARN_ON_ONCE (1 ); +return -EOPNOTSUPP ; +} +#endif /* CONFIG_HAVE_KVM_RESTRICTED_MEM */ +#endif /* __KVM_X86_MMU_INTERNAL_H */ arch/x86/kvm/mmu/mmutrace.h +1
-0@@ -58,6 +58,7 @@ TRACE_DEFINE_ENUM(RET_PF_CONTINUE);TRACE_DEFINE_ENUM (RET_PF_RETRY );TRACE_DEFINE_ENUM (RET_PF_EMULATE );TRACE_DEFINE_ENUM (RET_PF_INVALID ); +TRACE_DEFINE_ENUM (RET_PF_USER );TRACE_DEFINE_ENUM (RET_PF_FIXED );TRACE_DEFINE_ENUM (RET_PF_SPURIOUS );arch/x86/kvm/mmu/tdp_mmu.c +1
-1@@ -1768,7 +1768,7 @@ static void zap_collapsible_spte_range(struct kvm *kvm,continue ; max_mapping_level =kvm_mmu_max_mapping_level (kvm, slot, - iter.gfn,PG_LEVEL_NUM ); + iter.gfn,PG_LEVEL_NUM , false);if (max_mapping_level < iter.level)continue ;include/linux/kvm_host.h +30
-1@@ -2335,4 +2335,34 @@ static inline void kvm_arch_set_memory_attributes(struct kvm *kvm, }#endif /* __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES */ +#ifdef CONFIG_HAVE_KVM_MEMORY_ATTRIBUTES +static inline bool kvm_mem_is_private (struct kvm *kvm,gfn_t gfn) +{ +return xa_to_value (xa_load (&kvm->mem_attr_array, gfn)) & +KVM_MEMORY_ATTRIBUTE_PRIVATE ; +} +#else +static inline bool kvm_mem_is_private (struct kvm *kvm,gfn_t gfn) +{ +return false; +} + +#endif /* CONFIG_HAVE_KVM_MEMORY_ATTRIBUTES */ + +#ifdef CONFIG_HAVE_KVM_RESTRICTED_MEM +static inline int kvm_restricted_mem_get_pfn (struct kvm_memory_slot *slot, +gfn_t gfn,kvm_pfn_t *pfn,int *order) +{ +int ret; +struct page *page; +pgoff_t index = gfn - slot->base_gfn + + (slot->restricted_offset >>PAGE_SHIFT ); + + ret =restrictedmem_get_page (slot->restricted_file, index, + &page, order); + *pfn =page_to_pfn (page); +return ret; +} +#endif /* CONFIG_HAVE_KVM_RESTRICTED_MEM */ +#endif --2.25.1
Re: [PATCH v10 8/9] KVM: Handle page fault for private memory
In reply to Chao PengHi,
Quoted 78 lines · Chao Peng
A
KVM_MEM_PRIVATEmemslot can include both fd-based private memory and hva-based shared memory. Architecture code (like TDX code) can tell whether the on-going fault is private or not. This patch adds a 'is_private' field tokvm_page_faultto indicate this and architecture code is expected to set it.To handle page fault for such memslot, the handling logic is different depending on whether the fault is private or shared. KVM checks if '
is_private' matches the host's view of the page (maintained inmem_attr_array).
- For a successful match, private pfn is obtained withrestrictedmem_get_page()and shared pfn is obtained with existingget_user_pages().
- For a failed match, KVM causes aKVM_EXIT_MEMORY_FAULTexit to userspace. Userspace then can convert memory between private/shared in host's view and retry the fault.Co-developed-by Yu Zhang <[email protected]>Signed-off-by Yu Zhang <[email protected]>Signed-off-by Chao Peng <[email protected]>--- arch/x86/kvm/mmu/mmu.c | 63 +++++++++++++++++++++++++++++++-- arch/x86/kvm/mmu/mmu_internal.h | 14 +++++++- arch/x86/kvm/mmu/mmutrace.h | 1 + arch/x86/kvm/mmu/tdp_mmu.c | 2 +- include/linux/kvm_host.h | 30 ++++++++++++++++ 5 files changed, 105 insertions(+), 5 deletions(-)arch/x86/kvm/mmu/mmu.c +18
-2@@ -3058,7 +3058,7 @@ static int host_pfn_mapping_level(struct kvm *kvm, gfn_t gfn,int kvm_mmu_max_mapping_level (struct kvm *kvm,const struct kvm_memory_slot *slot,gfn_t gfn, -int max_level) +int max_level,bool is_private) {struct kvm_lpage_info *linfo;int host_level;@@ -3070,6 +3070,9 @@ int kvm_mmu_max_mapping_level(struct kvm *kvm,break ; } +if (is_private) +return max_level; +if (max_level ==PG_LEVEL_4K )return PG_LEVEL_4K ;@@ -3098,7 +3101,8 @@ void kvm_mmu_hugepage_adjust(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault* level, which will be used to do precise, accurate accounting. */ fault->req_level =kvm_mmu_max_mapping_level (vcpu->kvm, slot, - fault->gfn, fault->max_level); + fault->gfn, fault->max_level, + fault->is_private);if (fault->req_level ==PG_LEVEL_4K || fault->huge_page_disallowed)return ;@@ -4178,6 +4182,49 @@ void kvm_arch_async_page_ready(struct kvm_vcpu *vcpu, struct kvm_async_pf *work)kvm_mmu_do_page_fault (vcpu, work->cr2_or_gpa,0 , true); } +static inline u8 order_to_level (int order) +{ +BUILD_BUG_ON (KVM_MAX_HUGEPAGE_LEVEL >PG_LEVEL_1G ); + +if (order >=KVM_HPAGE_GFN_SHIFT (PG_LEVEL_1G )) +return PG_LEVEL_1G ; + +if (order >=KVM_HPAGE_GFN_SHIFT (PG_LEVEL_2M )) +return PG_LEVEL_2M ; + +return PG_LEVEL_4K ; +}
arch/x86/kvm/mmu/mmu.c void kvm_arch_async_page_ready(struct kvm_vcpu *vcpu, struct kvm_async_pf *work) · line 4206
Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +10
-0+ +static int kvm_do_memory_fault_exit (struct kvm_vcpu *vcpu, +struct kvm_page_fault *fault) +{ + vcpu->run->exit_reason =KVM_EXIT_MEMORY_FAULT ; +if (fault->is_private) + vcpu->run->memory.flags =KVM_MEMORY_EXIT_FLAG_PRIVATE ; +else + vcpu->run->memory.flags =0 ; + vcpu->run->memory.gpa = fault->gfn <<PAGE_SHIFT ;
nit: As in previous patches, use helpers (for this and other similar shifts in this patch)?
Quoted 173 lines
arch/x86/kvm/mmu/mmu.c +32
-1+ vcpu->run->memory.size =PAGE_SIZE ; +return RET_PF_USER ; +} + +static int kvm_faultin_pfn_private (struct kvm_vcpu *vcpu, +struct kvm_page_fault *fault) +{ +int order; +struct kvm_memory_slot *slot = fault->slot; + +if (!kvm_slot_can_be_private (slot)) +return kvm_do_memory_fault_exit (vcpu, fault); + +if (kvm_restricted_mem_get_pfn (slot, fault->gfn, &fault->pfn, &order)) +return RET_PF_RETRY ; + + fault->max_level =min (order_to_level (order), fault->max_level); + fault->map_writable = !(slot->flags &KVM_MEM_READONLY ); +return RET_PF_CONTINUE ; +} +static int kvm_faultin_pfn (struct kvm_vcpu *vcpu,struct kvm_page_fault *fault) {struct kvm_memory_slot *slot = fault->slot;@@ -4210,6 +4257,12 @@ static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)return RET_PF_EMULATE ; } +if (fault->is_private !=kvm_mem_is_private (vcpu->kvm, fault->gfn)) +return kvm_do_memory_fault_exit (vcpu, fault); + +if (fault->is_private) +return kvm_faultin_pfn_private (vcpu, fault); + async = false; fault->pfn = __gfn_to_pfn_memslot(slot, fault->gfn, false, false, &async, fault->write, &fault->map_writable,@@ -5599,6 +5652,9 @@ int noinline kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 errreturn -EIO ; } +if (r ==RET_PF_USER ) +return 0 ; +if (r <0 )return r;if (r !=RET_PF_EMULATE )@@ -6452,7 +6508,8 @@ static bool kvm_mmu_zap_collapsible_spte(struct kvm *kvm,*/ if (sp->role.direct && sp->role.level <kvm_mmu_max_mapping_level (kvm, slot, sp->gfn, -PG_LEVEL_NUM )) { +PG_LEVEL_NUM , + false)) {kvm_zap_one_rmap_spte (kvm, rmap_head, sptep);if (kvm_available_flush_tlb_with_range ())arch/x86/kvm/mmu/mmu_internal.h +13
-1@@ -189,6 +189,7 @@ struct kvm_page_fault {/* Derived from mmu and global state. */ const bool is_tdp; +const bool is_private;const bool nx_huge_page_workaround_enabled;/* @@ -237,6 +238,7 @@ int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault);* RET_PF_RETRY: let CPU fault again on the address. * RET_PF_EMULATE: mmio page fault, emulate the instruction directly. * RET_PF_INVALID: the spte is invalid, let the real page fault path update it. +* RET_PF_USER: need to exit to userspace to handle this fault. * RET_PF_FIXED: The faulting entry has been fixed. * RET_PF_SPURIOUS: The faulting entry was already fixed, e.g. by another vCPU. * @@ -253,6 +255,7 @@ enum {RET_PF_RETRY ,RET_PF_EMULATE ,RET_PF_INVALID , +RET_PF_USER ,RET_PF_FIXED , RET_PF_SPURIOUS, };@@ -310,7 +313,7 @@ static inline int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,int kvm_mmu_max_mapping_level (struct kvm *kvm,const struct kvm_memory_slot *slot,gfn_t gfn, -int max_level); +int max_level,bool is_private);void kvm_mmu_hugepage_adjust (struct kvm_vcpu *vcpu,struct kvm_page_fault *fault);void disallowed_hugepage_adjust (struct kvm_page_fault *fault,u64 spte,int cur_level);@@ -319,4 +322,13 @@ void *mmu_memory_cache_alloc(struct kvm_mmu_memory_cache *mc);void track_possible_nx_huge_page (struct kvm *kvm,struct kvm_mmu_page *sp);void untrack_possible_nx_huge_page (struct kvm *kvm,struct kvm_mmu_page *sp); +#ifndef CONFIG_HAVE_KVM_RESTRICTED_MEM +static inline int kvm_restricted_mem_get_pfn (struct kvm_memory_slot *slot, +gfn_t gfn,kvm_pfn_t *pfn,int *order) +{ +WARN_ON_ONCE (1 ); +return -EOPNOTSUPP ; +} +#endif /* CONFIG_HAVE_KVM_RESTRICTED_MEM */ +#endif /* __KVM_X86_MMU_INTERNAL_H */ arch/x86/kvm/mmu/mmutrace.h +1
-0@@ -58,6 +58,7 @@ TRACE_DEFINE_ENUM(RET_PF_CONTINUE);TRACE_DEFINE_ENUM (RET_PF_RETRY );TRACE_DEFINE_ENUM (RET_PF_EMULATE );TRACE_DEFINE_ENUM (RET_PF_INVALID ); +TRACE_DEFINE_ENUM (RET_PF_USER );TRACE_DEFINE_ENUM (RET_PF_FIXED );TRACE_DEFINE_ENUM (RET_PF_SPURIOUS );arch/x86/kvm/mmu/tdp_mmu.c +1
-1@@ -1768,7 +1768,7 @@ static void zap_collapsible_spte_range(struct kvm *kvm,continue ; max_mapping_level =kvm_mmu_max_mapping_level (kvm, slot, - iter.gfn,PG_LEVEL_NUM ); + iter.gfn,PG_LEVEL_NUM , false);if (max_mapping_level < iter.level)continue ;include/linux/kvm_host.h +30
-1@@ -2335,4 +2335,34 @@ static inline void kvm_arch_set_memory_attributes(struct kvm *kvm, }#endif /* __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES */ +#ifdef CONFIG_HAVE_KVM_MEMORY_ATTRIBUTES +static inline bool kvm_mem_is_private (struct kvm *kvm,gfn_t gfn) +{ +return xa_to_value (xa_load (&kvm->mem_attr_array, gfn)) & +KVM_MEMORY_ATTRIBUTE_PRIVATE ; +} +#else +static inline bool kvm_mem_is_private (struct kvm *kvm,gfn_t gfn) +{ +return false; +} + +#endif /* CONFIG_HAVE_KVM_MEMORY_ATTRIBUTES */ + +#ifdef CONFIG_HAVE_KVM_RESTRICTED_MEM +static inline int kvm_restricted_mem_get_pfn (struct kvm_memory_slot *slot, +gfn_t gfn,kvm_pfn_t *pfn,int *order) +{ +int ret; +struct page *page; +pgoff_t index = gfn - slot->base_gfn + + (slot->restricted_offset >>PAGE_SHIFT ); + + ret =restrictedmem_get_page (slot->restricted_file, index, + &page, order); + *pfn =page_to_pfn (page); +return ret; +} +#endif /* CONFIG_HAVE_KVM_RESTRICTED_MEM */ +#endif --
Quoted 2 lines
2.25.1
With my limited understanding of x86 code:
The common code in kvm_host.h was used in the port to arm64, and the x86 fault handling code was used as a guide to how it should be done in pKVM (with similar code added there). So with these caveats in mind:
Cheers,
/fuad
Re: [PATCH v10 8/9] KVM: Handle page fault for private memory
Quoted 86 lines · Fuad Tabba
Hi,
Quoted 83 lines · Chao Peng
A
KVM_MEM_PRIVATEmemslot can include both fd-based private memory and hva-based shared memory. Architecture code (like TDX code) can tell whether the on-going fault is private or not. This patch adds a 'is_private' field tokvm_page_faultto indicate this and architecture code is expected to set it.To handle page fault for such memslot, the handling logic is different depending on whether the fault is private or shared. KVM checks if '
is_private' matches the host's view of the page (maintained inmem_attr_array).
- For a successful match, private pfn is obtained withrestrictedmem_get_page()and shared pfn is obtained with existingget_user_pages().
- For a failed match, KVM causes aKVM_EXIT_MEMORY_FAULTexit to userspace. Userspace then can convert memory between private/shared in host's view and retry the fault.Co-developed-by Yu Zhang <[email protected]>Signed-off-by Yu Zhang <[email protected]>Signed-off-by Chao Peng <[email protected]>--- arch/x86/kvm/mmu/mmu.c | 63 +++++++++++++++++++++++++++++++-- arch/x86/kvm/mmu/mmu_internal.h | 14 +++++++- arch/x86/kvm/mmu/mmutrace.h | 1 + arch/x86/kvm/mmu/tdp_mmu.c | 2 +- include/linux/kvm_host.h | 30 ++++++++++++++++ 5 files changed, 105 insertions(+), 5 deletions(-)arch/x86/kvm/mmu/mmu.c +23
-2@@ -3058,7 +3058,7 @@ static int host_pfn_mapping_level(struct kvm *kvm, gfn_t gfn,int kvm_mmu_max_mapping_level (struct kvm *kvm,const struct kvm_memory_slot *slot,gfn_t gfn, -int max_level) +int max_level,bool is_private) {struct kvm_lpage_info *linfo;int host_level;@@ -3070,6 +3070,9 @@ int kvm_mmu_max_mapping_level(struct kvm *kvm,break ; } +if (is_private) +return max_level; +if (max_level ==PG_LEVEL_4K )return PG_LEVEL_4K ;@@ -3098,7 +3101,8 @@ void kvm_mmu_hugepage_adjust(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault* level, which will be used to do precise, accurate accounting. */ fault->req_level =kvm_mmu_max_mapping_level (vcpu->kvm, slot, - fault->gfn, fault->max_level); + fault->gfn, fault->max_level, + fault->is_private);if (fault->req_level ==PG_LEVEL_4K || fault->huge_page_disallowed)return ;@@ -4178,6 +4182,49 @@ void kvm_arch_async_page_ready(struct kvm_vcpu *vcpu, struct kvm_async_pf *work)kvm_mmu_do_page_fault (vcpu, work->cr2_or_gpa,0 , true); } +static inline u8 order_to_level (int order) +{ +BUILD_BUG_ON (KVM_MAX_HUGEPAGE_LEVEL >PG_LEVEL_1G ); + +if (order >=KVM_HPAGE_GFN_SHIFT (PG_LEVEL_1G )) +return PG_LEVEL_1G ; + +if (order >=KVM_HPAGE_GFN_SHIFT (PG_LEVEL_2M )) +return PG_LEVEL_2M ; + +return PG_LEVEL_4K ; +} + +static int kvm_do_memory_fault_exit (struct kvm_vcpu *vcpu, +struct kvm_page_fault *fault) +{ + vcpu->run->exit_reason =KVM_EXIT_MEMORY_FAULT ;
Quoted 8 lines
arch/x86/kvm/mmu/mmu.c void kvm_arch_async_page_ready(struct kvm_vcpu *vcpu, struct kvm_async_pf *work) · line 4206
Quoted 5 lines
arch/x86/kvm/mmu/mmu.c +5
-0+if (fault->is_private) + vcpu->run->memory.flags =KVM_MEMORY_EXIT_FLAG_PRIVATE ; +else + vcpu->run->memory.flags =0 ; + vcpu->run->memory.gpa = fault->gfn <<PAGE_SHIFT ;nit: As in previous patches, use helpers (for this and other similar shifts in this patch)?
Agreed.
Quoted 188 lines
Quoted 174 lines
arch/x86/kvm/mmu/mmu.c +32
-1+ vcpu->run->memory.size =PAGE_SIZE ; +return RET_PF_USER ; +} + +static int kvm_faultin_pfn_private (struct kvm_vcpu *vcpu, +struct kvm_page_fault *fault) +{ +int order; +struct kvm_memory_slot *slot = fault->slot; + +if (!kvm_slot_can_be_private (slot)) +return kvm_do_memory_fault_exit (vcpu, fault); + +if (kvm_restricted_mem_get_pfn (slot, fault->gfn, &fault->pfn, &order)) +return RET_PF_RETRY ; + + fault->max_level =min (order_to_level (order), fault->max_level); + fault->map_writable = !(slot->flags &KVM_MEM_READONLY ); +return RET_PF_CONTINUE ; +} +static int kvm_faultin_pfn (struct kvm_vcpu *vcpu,struct kvm_page_fault *fault) {struct kvm_memory_slot *slot = fault->slot;@@ -4210,6 +4257,12 @@ static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)return RET_PF_EMULATE ; } +if (fault->is_private !=kvm_mem_is_private (vcpu->kvm, fault->gfn)) +return kvm_do_memory_fault_exit (vcpu, fault); + +if (fault->is_private) +return kvm_faultin_pfn_private (vcpu, fault); + async = false; fault->pfn = __gfn_to_pfn_memslot(slot, fault->gfn, false, false, &async, fault->write, &fault->map_writable,@@ -5599,6 +5652,9 @@ int noinline kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 errreturn -EIO ; } +if (r ==RET_PF_USER ) +return 0 ; +if (r <0 )return r;if (r !=RET_PF_EMULATE )@@ -6452,7 +6508,8 @@ static bool kvm_mmu_zap_collapsible_spte(struct kvm *kvm,*/ if (sp->role.direct && sp->role.level <kvm_mmu_max_mapping_level (kvm, slot, sp->gfn, -PG_LEVEL_NUM )) { +PG_LEVEL_NUM , + false)) {kvm_zap_one_rmap_spte (kvm, rmap_head, sptep);if (kvm_available_flush_tlb_with_range ())arch/x86/kvm/mmu/mmu_internal.h +13
-1@@ -189,6 +189,7 @@ struct kvm_page_fault {/* Derived from mmu and global state. */ const bool is_tdp; +const bool is_private;const bool nx_huge_page_workaround_enabled;/* @@ -237,6 +238,7 @@ int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault);* RET_PF_RETRY: let CPU fault again on the address. * RET_PF_EMULATE: mmio page fault, emulate the instruction directly. * RET_PF_INVALID: the spte is invalid, let the real page fault path update it. +* RET_PF_USER: need to exit to userspace to handle this fault. * RET_PF_FIXED: The faulting entry has been fixed. * RET_PF_SPURIOUS: The faulting entry was already fixed, e.g. by another vCPU. * @@ -253,6 +255,7 @@ enum {RET_PF_RETRY ,RET_PF_EMULATE ,RET_PF_INVALID , +RET_PF_USER ,RET_PF_FIXED , RET_PF_SPURIOUS, };@@ -310,7 +313,7 @@ static inline int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,int kvm_mmu_max_mapping_level (struct kvm *kvm,const struct kvm_memory_slot *slot,gfn_t gfn, -int max_level); +int max_level,bool is_private);void kvm_mmu_hugepage_adjust (struct kvm_vcpu *vcpu,struct kvm_page_fault *fault);void disallowed_hugepage_adjust (struct kvm_page_fault *fault,u64 spte,int cur_level);@@ -319,4 +322,13 @@ void *mmu_memory_cache_alloc(struct kvm_mmu_memory_cache *mc);void track_possible_nx_huge_page (struct kvm *kvm,struct kvm_mmu_page *sp);void untrack_possible_nx_huge_page (struct kvm *kvm,struct kvm_mmu_page *sp); +#ifndef CONFIG_HAVE_KVM_RESTRICTED_MEM +static inline int kvm_restricted_mem_get_pfn (struct kvm_memory_slot *slot, +gfn_t gfn,kvm_pfn_t *pfn,int *order) +{ +WARN_ON_ONCE (1 ); +return -EOPNOTSUPP ; +} +#endif /* CONFIG_HAVE_KVM_RESTRICTED_MEM */ +#endif /* __KVM_X86_MMU_INTERNAL_H */ arch/x86/kvm/mmu/mmutrace.h +1
-0@@ -58,6 +58,7 @@ TRACE_DEFINE_ENUM(RET_PF_CONTINUE);TRACE_DEFINE_ENUM (RET_PF_RETRY );TRACE_DEFINE_ENUM (RET_PF_EMULATE );TRACE_DEFINE_ENUM (RET_PF_INVALID ); +TRACE_DEFINE_ENUM (RET_PF_USER );TRACE_DEFINE_ENUM (RET_PF_FIXED );TRACE_DEFINE_ENUM (RET_PF_SPURIOUS );arch/x86/kvm/mmu/tdp_mmu.c +1
-1@@ -1768,7 +1768,7 @@ static void zap_collapsible_spte_range(struct kvm *kvm,continue ; max_mapping_level =kvm_mmu_max_mapping_level (kvm, slot, - iter.gfn,PG_LEVEL_NUM ); + iter.gfn,PG_LEVEL_NUM , false);if (max_mapping_level < iter.level)continue ;include/linux/kvm_host.h +30
-1@@ -2335,4 +2335,34 @@ static inline void kvm_arch_set_memory_attributes(struct kvm *kvm, }#endif /* __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES */ +#ifdef CONFIG_HAVE_KVM_MEMORY_ATTRIBUTES +static inline bool kvm_mem_is_private (struct kvm *kvm,gfn_t gfn) +{ +return xa_to_value (xa_load (&kvm->mem_attr_array, gfn)) & +KVM_MEMORY_ATTRIBUTE_PRIVATE ; +} +#else +static inline bool kvm_mem_is_private (struct kvm *kvm,gfn_t gfn) +{ +return false; +} + +#endif /* CONFIG_HAVE_KVM_MEMORY_ATTRIBUTES */ + +#ifdef CONFIG_HAVE_KVM_RESTRICTED_MEM +static inline int kvm_restricted_mem_get_pfn (struct kvm_memory_slot *slot, +gfn_t gfn,kvm_pfn_t *pfn,int *order) +{ +int ret; +struct page *page; +pgoff_t index = gfn - slot->base_gfn + + (slot->restricted_offset >>PAGE_SHIFT ); + + ret =restrictedmem_get_page (slot->restricted_file, index, + &page, order); + *pfn =page_to_pfn (page); +return ret; +} +#endif /* CONFIG_HAVE_KVM_RESTRICTED_MEM */ +#endif --Quoted 2 lines
2.25.1
With my limited understanding of x86 code:
Reviewed-by Fuad Tabba <[email protected]>The common code in
kvm_host.hwas used in the port to arm64, and the x86 fault handling code was used as a guide to how it should be done in pKVM (with similar code added there). So with these caveats in mind:Tested-by Fuad Tabba <[email protected]>Cheers,
/fuad
Sean Christopherson
maintainerRe: [PATCH v10 8/9] KVM: Handle page fault for private memory
In reply to Chao PengQuoted 11 lines · Chao Peng
arch/x86/kvm/mmu/mmu.c +3
-0@@ -5599,6 +5652,9 @@ int noinline kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 errreturn -EIO ; } +if (r ==RET_PF_USER ) +return 0 ; +if (r <0 )return r;if (r !=RET_PF_EMULATE )
arch/x86/kvm/mmu/mmu.c static bool kvm_mmu_zap_collapsible_spte(struct kvm *kvm, · line 6512
Quoted 6 lines
arch/x86/kvm/mmu/mmu.c +2
-1@@ -6452,7 +6508,8 @@ static bool kvm_mmu_zap_collapsible_spte(struct kvm *kvm,*/ if (sp->role.direct && sp->role.level <kvm_mmu_max_mapping_level (kvm, slot, sp->gfn, -PG_LEVEL_NUM )) { +PG_LEVEL_NUM , + false)) {
Passing %false is incorrect. It might not cause problems because KVM currently doesn't allowing modifying private memslots (that likely needs to change to allow dirty logging), but it's wrong since nothing guarantees KVM is operating on SPTEs for shared memory.
One option would be to take the patches from the TDX series that add a "private" flag to the shadow page role, but I'd rather not add the role until it's truly necessary.
For now, I think we can do this without impacting performance of guests that don't support private memory.
int kvm_mmu_max_mapping_level(struct kvm *kvm,
const struct kvm_memory_slot *slot, gfn_t gfn, int max_level)
{
bool is_private = kvm_slot_can_be_private(slot) &&
kvm_mem_is_private(kvm, gfn);
return __kvm_mmu_max_mapping_level(kvm, slot, gfn, max_level, is_private);
}include/linux/kvm_host.h static inline void kvm_arch_set_memory_attributes(struct kvm *kvm, · line 2340
Quoted 11 lines
include/linux/kvm_host.h +3
-0@@ -2335,4 +2335,34 @@ static inline void kvm_arch_set_memory_attributes(struct kvm *kvm, }#endif /* __KVM_HAVE_ARCH_SET_MEMORY_ATTRIBUTES */ +#ifdef CONFIG_HAVE_KVM_MEMORY_ATTRIBUTES +static inline bool kvm_mem_is_private (struct kvm *kvm,gfn_t gfn) +{
This code, i.e. the generic KVM changes, belongs in a separate patch. It'll be small, but I want to separate x86's page fault changes from the restrictedmem support adding to common KVM.
This should also short-circuit based on CONFIG_HAVE_KVM_RESTRICTED_MEM, though I would name that CONFIG_KVM_PRIVATE_MEMORY since in KVM's world, it's all about private vs. shared at this time.
Quoted 12 lines
include/linux/kvm_host.h +12
-0+return xa_to_value (xa_load (&kvm->mem_attr_array, gfn)) & + KVM_MEMORY_ATTRIBUTE_PRIVATE; +} +#else +static inline bool kvm_mem_is_private (struct kvm *kvm,gfn_t gfn) +{ +return false; +} + +#endif /* CONFIG_HAVE_KVM_MEMORY_ATTRIBUTES */ + +#ifdef CONFIG_HAVE_KVM_RESTRICTED_MEM
include/linux/kvm_host.h static inline void kvm_arch_set_memory_attributes(struct kvm *kvm, · line 2362
Quoted 10 lines
include/linux/kvm_host.h +10
-0+static inline int kvm_restricted_mem_get_pfn (struct kvm_memory_slot *slot, +gfn_t gfn,kvm_pfn_t *pfn,int *order) +{ +int ret; +struct page *page; +pgoff_t index = gfn - slot->base_gfn + + (slot->restricted_offset >>PAGE_SHIFT ); + + ret =restrictedmem_get_page (slot->restricted_file, index, + &page, order);
This needs handle errors. If "ret" is non-zero, "page" is garbage.
Quoted 9 lines
include/linux/kvm_host.h +5
-0+ *pfn =page_to_pfn (page); +return ret; +} +#endif /* CONFIG_HAVE_KVM_RESTRICTED_MEM */ +#endif Signature · 1 line
2.25.1
PATCH v10 9/9 KVM: Enable and expose KVM_MEM_PRIVATE
In reply to Chao PengRegister/unregister private memslot to fd-based memory backing store restrictedmem and implement the callbacks for restrictedmem_notifier:
- invalidate_start()/invalidate_end() to zap the existing memory mappings in the KVM page table.
- error() to request KVM_REQ_MEMORY_MCE and later exit to userspace with KVM_EXIT_SHUTDOWN.
Expose KVM_MEM_PRIVATE for memslot and KVM_MEMORY_ATTRIBUTE_PRIVATE for KVM_GET_SUPPORTED_MEMORY_ATTRIBUTES to userspace but either are controlled by kvm_arch_has_private_mem() which should be rewritten by architecture code.
--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/x86.c | 13 +++ include/linux/kvm_host.h | 3 + virt/kvm/kvm_main.c | 179 +++++++++++++++++++++++++++++++- 4 files changed, 191 insertions(+), 5 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -0
@@ -114,6 +114,7 @@ KVM_ARCH_REQ_FLAGS(31, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)#define KVM_REQ_HV_TLB_FLUSH \ KVM_ARCH_REQ_FLAGS(32, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP) +#define KVM_REQ_MEMORY_MCE KVM_ARCH_REQ(33)#define CR0_RESERVED_BITS \ (~(unsigned long)(X86_CR0_PE | X86_CR0_MP | X86_CR0_EM | X86_CR0_TS \
arch/x86/kvm/x86.c +13 -0
@@ -6587,6 +6587,13 @@ int kvm_arch_pm_notifier(struct kvm *kvm, unsigned long state) }#endif /* CONFIG_HAVE_KVM_PM_NOTIFIER */ +#ifdef CONFIG_HAVE_KVM_RESTRICTED_MEM +void kvm_arch_memory_mce (struct kvm *kvm) +{ +kvm_make_all_cpus_request (kvm,KVM_REQ_MEMORY_MCE ); +} +#endif +static int kvm_vm_ioctl_get_clock (struct kvm *kvm,void __user *argp) {struct kvm_clock_data data = {0 };
@@ -10357,6 +10364,12 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)if (kvm_check_request (KVM_REQ_UPDATE_CPU_DIRTY_LOGGING , vcpu))static_call (kvm_x86_update_cpu_dirty_logging)(vcpu); + +if (kvm_check_request (KVM_REQ_MEMORY_MCE , vcpu)) { + vcpu->run->exit_reason =KVM_EXIT_SHUTDOWN ; + r =0 ; + goto out; + } } if (kvm_check_request(KVM_REQ_EVENT, vcpu) || req_int_win ||
include/linux/kvm_host.h +3 -0
@@ -590,6 +590,7 @@ struct kvm_memory_slot {struct file *restricted_file;loff_t restricted_offset;struct restrictedmem_notifier notifier; +struct kvm *kvm; };static inline bool kvm_slot_can_be_private (const struct kvm_memory_slot *slot)
@@ -2363,6 +2364,8 @@ static inline int kvm_restricted_mem_get_pfn(struct kvm_memory_slot *slot, *pfn =page_to_pfn (page);return ret; } + +void kvm_arch_memory_mce (struct kvm *kvm);#endif /* CONFIG_HAVE_KVM_RESTRICTED_MEM */ #endif
virt/kvm/kvm_main.c +174 -5
259 lines not on this page · raw message
Signature · 1 line
2.25.1
Re: [PATCH v10 9/9] KVM: Enable and expose KVM_MEM_PRIVATE
Hi,
Quoted 13 lines · Chao Peng
Register/unregister private memslot to fd-based memory backing store restrictedmem and implement the callbacks for
restrictedmem_notifier:
- invalidate_start()/invalidate_end() to zap the existing memory mappings in the KVM page table.
-error()to requestKVM_REQ_MEMORY_MCEand later exit to userspace withKVM_EXIT_SHUTDOWN.Expose
KVM_MEM_PRIVATEfor memslot andKVM_MEMORY_ATTRIBUTE_PRIVATEforKVM_GET_SUPPORTED_MEMORY_ATTRIBUTESto userspace but either are controlled bykvm_arch_has_private_mem()which should be rewritten by architecture code.
Quoted 4 lines
Co-developed-by Yu Zhang <[email protected]>Signed-off-by Yu Zhang <[email protected]>Signed-off-by Chao Peng <[email protected]>Reviewed-by Fuad Tabba <[email protected]>
With the code to port it to pKVM/arm64:
Cheers,
/fuad
Quoted 348 lines
--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/x86.c | 13 +++ include/linux/kvm_host.h | 3 + virt/kvm/kvm_main.c | 179 +++++++++++++++++++++++++++++++- 4 files changed, 191 insertions(+), 5 deletions(-)arch/x86/include/asm/kvm_host.h +1
-0@@ -114,6 +114,7 @@ KVM_ARCH_REQ_FLAGS(31, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)#define KVM_REQ_HV_TLB_FLUSH \ KVM_ARCH_REQ_FLAGS(32, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP) +#define KVM_REQ_MEMORY_MCE KVM_ARCH_REQ(33)#define CR0_RESERVED_BITS \ (~(unsigned long)(X86_CR0_PE | X86_CR0_MP | X86_CR0_EM | X86_CR0_TS \arch/x86/kvm/x86.c +13
-0@@ -6587,6 +6587,13 @@ int kvm_arch_pm_notifier(struct kvm *kvm, unsigned long state) }#endif /* CONFIG_HAVE_KVM_PM_NOTIFIER */ +#ifdef CONFIG_HAVE_KVM_RESTRICTED_MEM +void kvm_arch_memory_mce (struct kvm *kvm) +{ +kvm_make_all_cpus_request (kvm,KVM_REQ_MEMORY_MCE ); +} +#endif +static int kvm_vm_ioctl_get_clock (struct kvm *kvm,void __user *argp) {struct kvm_clock_data data = {0 };@@ -10357,6 +10364,12 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)if (kvm_check_request (KVM_REQ_UPDATE_CPU_DIRTY_LOGGING , vcpu))static_call (kvm_x86_update_cpu_dirty_logging)(vcpu); + +if (kvm_check_request (KVM_REQ_MEMORY_MCE , vcpu)) { + vcpu->run->exit_reason =KVM_EXIT_SHUTDOWN ; + r =0 ; + goto out; + } } if (kvm_check_request(KVM_REQ_EVENT, vcpu) || req_int_win ||include/linux/kvm_host.h +3
-0@@ -590,6 +590,7 @@ struct kvm_memory_slot {struct file *restricted_file;loff_t restricted_offset;struct restrictedmem_notifier notifier; +struct kvm *kvm; };static inline bool kvm_slot_can_be_private (const struct kvm_memory_slot *slot)@@ -2363,6 +2364,8 @@ static inline int kvm_restricted_mem_get_pfn(struct kvm_memory_slot *slot, *pfn =page_to_pfn (page);return ret; } + +void kvm_arch_memory_mce (struct kvm *kvm);#endif /* CONFIG_HAVE_KVM_RESTRICTED_MEM */ #endif virt/kvm/kvm_main.c +174
-6@@ -936,6 +936,121 @@ static int kvm_init_mmu_notifier(struct kvm *kvm)#endif /* CONFIG_MMU_NOTIFIER && KVM_ARCH_WANT_MMU_NOTIFIER */ +#ifdef CONFIG_HAVE_KVM_RESTRICTED_MEM +static bool restrictedmem_range_is_valid (struct kvm_memory_slot *slot, +pgoff_t start,pgoff_t end, +gfn_t *gfn_start,gfn_t *gfn_end) +{ +unsigned long base_pgoff = slot->restricted_offset >>PAGE_SHIFT ; + +if (start > base_pgoff) + *gfn_start = slot->base_gfn + start - base_pgoff; +else + *gfn_start = slot->base_gfn; + +if (end < base_pgoff + slot->npages) + *gfn_end = slot->base_gfn + end - base_pgoff; +else + *gfn_end = slot->base_gfn + slot->npages; + +if (*gfn_start >= *gfn_end) +return false; + +return true; +} + +static void kvm_restrictedmem_invalidate_begin (struct restrictedmem_notifier *notifier, +pgoff_t start,pgoff_t end) +{ +struct kvm_memory_slot *slot =container_of (notifier, + struct kvm_memory_slot, + notifier); +struct kvm *kvm = slot->kvm; +gfn_t gfn_start, gfn_end; +struct kvm_gfn_range gfn_range; +int idx; + +if (!restrictedmem_range_is_valid (slot, start, end, + &gfn_start, &gfn_end)) +return ; + + gfn_range.start = gfn_start; + gfn_range.end = gfn_end; + gfn_range.slot = slot; + gfn_range.pte =__pte (0 ); + gfn_range.may_block = true; + + idx =srcu_read_lock (&kvm->srcu); +KVM_MMU_LOCK (kvm); + +kvm_mmu_invalidate_begin (kvm); +kvm_mmu_invalidate_range_add (kvm, gfn_start, gfn_end); +if (kvm_unmap_gfn_range (kvm, &gfn_range)) +kvm_flush_remote_tlbs (kvm); + +KVM_MMU_UNLOCK (kvm); +srcu_read_unlock (&kvm->srcu, idx); +} + +static void kvm_restrictedmem_invalidate_end (struct restrictedmem_notifier *notifier, +pgoff_t start,pgoff_t end) +{ +struct kvm_memory_slot *slot =container_of (notifier, + struct kvm_memory_slot, + notifier); +struct kvm *kvm = slot->kvm; +gfn_t gfn_start, gfn_end; + +if (!restrictedmem_range_is_valid (slot, start, end, + &gfn_start, &gfn_end)) +return ; + +KVM_MMU_LOCK (kvm); +kvm_mmu_invalidate_end (kvm); +KVM_MMU_UNLOCK (kvm); +} + +static void kvm_restrictedmem_error (struct restrictedmem_notifier *notifier, +pgoff_t start,pgoff_t end) +{ +struct kvm_memory_slot *slot =container_of (notifier, + struct kvm_memory_slot, + notifier); +kvm_arch_memory_mce (slot->kvm); +} + +static struct restrictedmem_notifier_ops kvm_restrictedmem_notifier_ops = { + .invalidate_start = kvm_restrictedmem_invalidate_begin, + .invalidate_end = kvm_restrictedmem_invalidate_end, + .error = kvm_restrictedmem_error, +}; + +static inline void kvm_restrictedmem_register (struct kvm_memory_slot *slot) +{ + slot->notifier.ops = &kvm_restrictedmem_notifier_ops; +restrictedmem_register_notifier (slot->restricted_file, &slot->notifier); +} + +static inline void kvm_restrictedmem_unregister (struct kvm_memory_slot *slot) +{ +restrictedmem_unregister_notifier (slot->restricted_file, + &slot->notifier); +} + +#else /* !CONFIG_HAVE_KVM_RESTRICTED_MEM */ + +static inline void kvm_restrictedmem_register (struct kvm_memory_slot *slot) +{ +WARN_ON_ONCE (1 ); +} + +static inline void kvm_restrictedmem_unregister (struct kvm_memory_slot *slot) +{ +WARN_ON_ONCE (1 ); +} + +#endif /* CONFIG_HAVE_KVM_RESTRICTED_MEM */ +#ifdef CONFIG_HAVE_KVM_PM_NOTIFIER static int kvm_pm_notifier_call(struct notifier_block *bl, unsigned long state,@@ -980,6 +1095,11 @@ static void kvm_destroy_dirty_bitmap(struct kvm_memory_slot *memslot)/* This does not remove the slot from struct kvm_memslots data structures */ static void kvm_free_memslot (struct kvm *kvm,struct kvm_memory_slot *slot) { +if (slot->flags &KVM_MEM_PRIVATE ) { +kvm_restrictedmem_unregister (slot); +fput (slot->restricted_file); + } +kvm_destroy_dirty_bitmap (slot);kvm_arch_free_memslot (kvm, slot);@@ -1551,10 +1671,14 @@ static void kvm_replace_memslot(struct kvm *kvm, } } -static int check_memory_region_flags (const struct kvm_user_mem_region *mem) +static int check_memory_region_flags (struct kvm *kvm, +const struct kvm_user_mem_region *mem) {u32 valid_flags =KVM_MEM_LOG_DIRTY_PAGES ; +if (kvm_arch_has_private_mem (kvm)) + valid_flags |=KVM_MEM_PRIVATE ; +#ifdef __KVM_HAVE_READONLY_MEM valid_flags |=KVM_MEM_READONLY ;#endif @@ -1630,6 +1754,9 @@ static int kvm_prepare_memory_region(struct kvm *kvm, {int r; +if (change ==KVM_MR_CREATE && new->flags &KVM_MEM_PRIVATE ) +kvm_restrictedmem_register (new); +/* * If dirty logging is disabled, nullify the bitmap; the old bitmap * will be freed on "commit". If logging is enabled in both old and @@ -1658,6 +1785,9 @@ static int kvm_prepare_memory_region(struct kvm *kvm,if (r && new && new->dirty_bitmap && (!old || !old->dirty_bitmap))kvm_destroy_dirty_bitmap (new); +if (r && change ==KVM_MR_CREATE && new->flags &KVM_MEM_PRIVATE ) +kvm_restrictedmem_unregister (new); +return r; }@@ -1963,7 +2093,7 @@ int __kvm_set_memory_region(struct kvm *kvm,int as_id, id;int r; - r =check_memory_region_flags (mem); + r =check_memory_region_flags (kvm, mem);if (r)return r;@@ -1982,6 +2112,10 @@ int __kvm_set_memory_region(struct kvm *kvm, !access_ok ((void __user *)(unsigned long )mem->userspace_addr, mem->memory_size)) return -EINVAL; +if (mem->flags &KVM_MEM_PRIVATE && + (mem->restricted_offset & (PAGE_SIZE -1 ) || + mem->restricted_offset >U64_MAX - mem->memory_size)) +return -EINVAL ;if (as_id >=KVM_ADDRESS_SPACE_NUM || id >=KVM_MEM_SLOTS_NUM )return -EINVAL ;if (mem->guest_phys_addr + mem->memory_size < mem->guest_phys_addr)@@ -2020,6 +2154,9 @@ int __kvm_set_memory_region(struct kvm *kvm,if ((kvm->nr_memslot_pages + npages) < kvm->nr_memslot_pages)return -EINVAL ; }else {/* Modify an existing slot. */ +/* Private memslots are immutable, they can only be deleted. */ +if (mem->flags &KVM_MEM_PRIVATE ) +return -EINVAL ;if ((mem->userspace_addr != old->userspace_addr) || (npages != old->npages) || ((mem->flags ^ old->flags) &KVM_MEM_READONLY ))@@ -2048,10 +2185,28 @@ int __kvm_set_memory_region(struct kvm *kvm, new->npages = npages; new->flags = mem->flags; new->userspace_addr = mem->userspace_addr; +if (mem->flags &KVM_MEM_PRIVATE ) { + new->restricted_file =fget (mem->restricted_fd); +if (!new->restricted_file || + !file_is_restrictedmem (new->restricted_file)) { + r = -EINVAL ; + goto out; + } + new->restricted_offset = mem->restricted_offset; + } + + new->kvm = kvm; r =kvm_set_memslot (kvm, old, new, change);if (r) -kfree (new); + goto out; + +return 0 ; + +out: +if (new->restricted_file) +fput (new->restricted_file); +kfree (new);return r; }EXPORT_SYMBOL_GPL (__kvm_set_memory_region);@@ -2351,6 +2506,8 @@ static int kvm_vm_ioctl_clear_dirty_log(struct kvm *kvm,#ifdef CONFIG_HAVE_KVM_MEMORY_ATTRIBUTES static u64 kvm_supported_mem_attributes (struct kvm *kvm) { +if (kvm_arch_has_private_mem (kvm)) +return KVM_MEMORY_ATTRIBUTE_PRIVATE ;return 0 ; }@@ -4822,16 +4979,28 @@ static long kvm_vm_ioctl(struct file *filp, }case KVM_SET_USER_MEMORY_REGION : {struct kvm_user_mem_region mem; -unsigned long size =sizeof (struct kvm_userspace_memory_region ); +unsigned int flags_offset = offsetof(typeof (mem ), flags); +unsigned long size; +u32 flags;kvm_sanity_check_user_mem_region_alias (); +memset (&mem,0 ,sizeof (mem)); + r = -EFAULT ; +if (get_user (flags, (u32 __user *)(argp + flags_offset))) + goto out; + +if (flags &KVM_MEM_PRIVATE ) + size =sizeof (struct kvm_userspace_memory_region_ext ); +else + size =sizeof (struct kvm_userspace_memory_region ); +if (copy_from_user (&mem, argp, size)) goto out; r = -EINVAL ; -if (mem.flags &KVM_MEM_PRIVATE ) +if ((flags ^ mem.flags) &KVM_MEM_PRIVATE ) goto out; r =kvm_vm_ioctl_set_memory_region (kvm, &mem); --2.25.1
Re: [PATCH v10 9/9] KVM: Enable and expose KVM_MEM_PRIVATE
In reply to Chao Pengvirt/kvm/kvm_main.c static int kvm_init_mmu_notifier(struct kvm *kvm) · line 947
Quoted 10 lines · Chao Peng
virt/kvm/kvm_main.c +9
-0+#ifdef CONFIG_HAVE_KVM_RESTRICTED_MEM +static bool restrictedmem_range_is_valid (struct kvm_memory_slot *slot, +pgoff_t start,pgoff_t end, +gfn_t *gfn_start,gfn_t *gfn_end) +{ +unsigned long base_pgoff = slot->restricted_offset >>PAGE_SHIFT ; + +if (start > base_pgoff) + *gfn_start = slot->base_gfn + start - base_pgoff;
There should be a check for overflow here in case start is a very big value. Additional check can look like:
if (start >= base_pgoff + slot->npages)
return false;
virt/kvm/kvm_main.c static int kvm_init_mmu_notifier(struct kvm *kvm) · line 952
Quoted 5 lines
virt/kvm/kvm_main.c +5
-0+else + *gfn_start = slot->base_gfn; + +if (end < base_pgoff + slot->npages) + *gfn_end = slot->base_gfn + end - base_pgoff;
If "end" is smaller than base_pgoff, this can cause overflow and return the range as valid. There should be additional check:
if (end < base_pgoff)
return false;
Quoted 9 lines
virt/kvm/kvm_main.c +9
-0+else + *gfn_end = slot->base_gfn + slot->npages; + +if (*gfn_start >= *gfn_end) +return false; + +return true; +} +
Re: [PATCH v10 9/9] KVM: Enable and expose KVM_MEM_PRIVATE
Quoted 23 lines · Vishal Annapurve
virt/kvm/kvm_main.c static int kvm_init_mmu_notifier(struct kvm *kvm) · line 947
Quoted 10 lines · Chao Peng
virt/kvm/kvm_main.c +9
-0+#ifdef CONFIG_HAVE_KVM_RESTRICTED_MEM +static bool restrictedmem_range_is_valid (struct kvm_memory_slot *slot, +pgoff_t start,pgoff_t end, +gfn_t *gfn_start,gfn_t *gfn_end) +{ +unsigned long base_pgoff = slot->restricted_offset >>PAGE_SHIFT ; + +if (start > base_pgoff) + *gfn_start = slot->base_gfn + start - base_pgoff;There should be a check for overflow here in case start is a very big value. Additional check can look like:
if (start >=base_pgoff+slot->npages)return false;virt/kvm/kvm_main.c static int kvm_init_mmu_notifier(struct kvm *kvm) · line 952
Quoted 5 lines
virt/kvm/kvm_main.c +5
-0+else + *gfn_start = slot->base_gfn; + +if (end < base_pgoff + slot->npages) + *gfn_end = slot->base_gfn + end - base_pgoff;
Quoted 4 lines
If "end" is smaller than
base_pgoff, this can cause overflow and return the range as valid. There should be additional check:
if (end <base_pgoff)return false;
Thanks! Both are good catches. The improved code:
static bool restrictedmem_range_is_valid(struct kvm_memory_slot *slot,
pgoff_t start, pgoff_t end, gfn_t *gfn_start, gfn_t *gfn_end)
{
unsigned long base_pgoff = slot->restricted_offset >> PAGE_SHIFT;
if (start >= base_pgoff + slot->npages)
return false;
else if (start <= base_pgoff)
*gfn_start = slot->base_gfn;
else
*gfn_start = start - base_pgoff + slot->base_gfn;
if (end <= base_pgoff)
return false;
else if (end >= base_pgoff + slot->npages)
*gfn_end = slot->base_gfn + slot->npages;
else
*gfn_end = end - base_pgoff + slot->base_gfn;
if (*gfn_start >= *gfn_end)
return false;
return true;
}Thanks,
Chao
Quoted 11 lines
Quoted 11 lines
virt/kvm/kvm_main.c +9
-0+else + *gfn_end = slot->base_gfn + slot->npages; + +if (*gfn_start >= *gfn_end) +return false; + +return true; +} +
Sean Christopherson
maintainerRe: [PATCH v10 9/9] KVM: Enable and expose KVM_MEM_PRIVATE
In reply to Chao Pengarch/x86/kvm/x86.c static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · line 10369
Quoted 7 lines · Chao Peng
arch/x86/kvm/x86.c +3
-0@@ -10357,6 +10364,12 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)if (kvm_check_request (KVM_REQ_UPDATE_CPU_DIRTY_LOGGING , vcpu))static_call (kvm_x86_update_cpu_dirty_logging)(vcpu); + +if (kvm_check_request (KVM_REQ_MEMORY_MCE , vcpu)) { + vcpu->run->exit_reason =KVM_EXIT_SHUTDOWN ;
Synthesizing triple fault shutdown is not the right approach. Even with TDX's MCE "architecture" (heavy sarcasm), it's possible that host userspace and the guest have a paravirt interface for handling memory errors without killing the host.
arch/x86/kvm/x86.c int __kvm_set_memory_region(struct kvm *kvm, · line 2116
Quoted 12 lines
arch/x86/kvm/x86.c +5
-0+ r =0 ; + goto out; + } }@@ -1982,6 +2112,10 @@ int __kvm_set_memory_region(struct kvm *kvm, !access_ok ((void __user *)(unsigned long )mem->userspace_addr, mem->memory_size)) return -EINVAL; + if (mem->flags & KVM_MEM_PRIVATE && + (mem->restricted_offset & (PAGE_SIZE - 1) ||
Align indentation.
virt/kvm/kvm_main.c int __kvm_set_memory_region(struct kvm *kvm, · line 2117
Quoted 1 line
virt/kvm/kvm_main.c +1
-0+ mem->restricted_offset > U64_MAX - mem->memory_size))
Strongly prefer to use similar logic to existing code that detects wraps:
mem->restricted_offset + mem->memory_size < mem->restricted_offset
This is also where I'd like to add the "gfn is aligned to offset" check, though my brain is too fried to figure that out right now.
virt/kvm/kvm_main.c int __kvm_set_memory_region(struct kvm *kvm, · line 2157
Quoted 9 lines
virt/kvm/kvm_main.c +2
-0+return -EINVAL ;if (as_id >=KVM_ADDRESS_SPACE_NUM || id >=KVM_MEM_SLOTS_NUM )return -EINVAL ;if (mem->guest_phys_addr + mem->memory_size < mem->guest_phys_addr)@@ -2020,6 +2154,9 @@ int __kvm_set_memory_region(struct kvm *kvm,if ((kvm->nr_memslot_pages + npages) < kvm->nr_memslot_pages)return -EINVAL ; }else {/* Modify an existing slot. */ +/* Private memslots are immutable, they can only be deleted. */
I'm 99% certain I suggested this, but if we're going to make these memslots immutable, then we should straight up disallow dirty logging, otherwise we'll end up with a bizarre uAPI.
Quoted 10 lines
virt/kvm/kvm_main.c +3
-0+if (mem->flags &KVM_MEM_PRIVATE ) +return -EINVAL ;if ((mem->userspace_addr != old->userspace_addr) || (npages != old->npages) || ((mem->flags ^ old->flags) &KVM_MEM_READONLY ))@@ -2048,10 +2185,28 @@ int __kvm_set_memory_region(struct kvm *kvm, new->npages = npages; new->flags = mem->flags; new->userspace_addr = mem->userspace_addr; +if (mem->flags &KVM_MEM_PRIVATE ) {
virt/kvm/kvm_main.c int __kvm_set_memory_region(struct kvm *kvm, · line 2198
Quoted 10 lines
virt/kvm/kvm_main.c +10
-0+ new->restricted_file =fget (mem->restricted_fd); +if (!new->restricted_file || + !file_is_restrictedmem (new->restricted_file)) { + r = -EINVAL ; + goto out; + } + new->restricted_offset = mem->restricted_offset; + } + + new->kvm = kvm;
Set this above, just so that the code flows better.
Re: [PATCH v10 9/9] KVM: Enable and expose KVM_MEM_PRIVATE
Quoted 13 lines · Sean Christopherson
Review static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · line 10369
Quoted 7 lines · Chao Peng
(quoted diff) +3
-0@@ -10357,6 +10364,12 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) if (kvm_check_request(KVM_REQ_UPDATE_CPU_DIRTY_LOGGING, vcpu)) static_call(kvm_x86_update_cpu_dirty_logging)(vcpu); + + if (kvm_check_request(KVM_REQ_MEMORY_MCE, vcpu)) { + vcpu->run->exit_reason = KVM_EXIT_SHUTDOWN;Synthesizing triple fault shutdown is not the right approach. Even with TDX's MCE "architecture" (heavy sarcasm), it's possible that host userspace and the guest have a paravirt interface for handling memory errors without killing the host.
Agree shutdown is not the correct choice. I see you made below change:
send_sig_mceerr(BUS_MCEERR_AR, (void __user *)hva, PAGE_SHIFT, current)
The MCE may happen in any thread than KVM thread, sending siginal to 'current' thread may not be the expected behavior. Also how userspace can tell is the MCE on the shared page or private page? Do we care?
Quoted 25 lines
Review int __kvm_set_memory_region(struct kvm *kvm, · line 2116
Quoted 13 lines
(quoted diff) +5
-0+ r = 0; + goto out; + } }@@ -1982,6 +2112,10 @@ int __kvm_set_memory_region(struct kvm *kvm, !access_ok((void __user *)(unsigned long)mem->userspace_addr, mem->memory_size)) return -EINVAL; + if (mem->flags & KVM_MEM_PRIVATE && + (mem->restricted_offset & (PAGE_SIZE - 1) ||Align indentation.
Review int __kvm_set_memory_region(struct kvm *kvm, · line 2117
Quoted 1 line
(quoted diff) +1
-0+ mem->restricted_offset > U64_MAX - mem->memory_size))Strongly prefer to use similar logic to existing code that detects wraps:
mem->restricted_offset + mem->memory_size < mem->restricted_offsetThis is also where I'd like to add the "gfn is aligned to offset" check, though my brain is too fried to figure that out right now.
Quoted 13 lines
Review int __kvm_set_memory_region(struct kvm *kvm, · line 2157
Quoted 9 lines
(quoted diff) +2
-0+ return -EINVAL; if (as_id >= KVM_ADDRESS_SPACE_NUM || id >= KVM_MEM_SLOTS_NUM) return -EINVAL; if (mem->guest_phys_addr + mem->memory_size < mem->guest_phys_addr)@@ -2020,6 +2154,9 @@ int __kvm_set_memory_region(struct kvm *kvm, if ((kvm->nr_memslot_pages + npages) < kvm->nr_memslot_pages) return -EINVAL; } else { /* Modify an existing slot. */ + /* Private memslots are immutable, they can only be deleted. */I'm 99% certain I suggested this, but if we're going to make these memslots immutable, then we should straight up disallow dirty logging, otherwise we'll end up with a bizarre uAPI.
But in my mind dirty logging will be needed in the very short time, when live migration gets supported?
Quoted 10 lines
Quoted 10 lines
(quoted diff) +2
-0+ if (mem->flags & KVM_MEM_PRIVATE) + return -EINVAL; if ((mem->userspace_addr != old->userspace_addr) || (npages != old->npages) || ((mem->flags ^ old->flags) & KVM_MEM_READONLY))@@ -2048,10 +2185,28 @@ int __kvm_set_memory_region(struct kvm *kvm, new->npages = npages; new->flags = mem->flags; new->userspace_addr = mem->userspace_addr;
Quoted 8 lines
Review int __kvm_set_memory_region(struct kvm *kvm, · line 2195
Quoted 8 lines
(quoted diff) +8
-0+ if (mem->flags & KVM_MEM_PRIVATE) { + new->restricted_file = fget(mem->restricted_fd); + if (!new->restricted_file || + !file_is_restrictedmem(new->restricted_file)) { + r = -EINVAL; + goto out; + } + new->restricted_offset = mem->restricted_offset;
I see you changed slot->restricted_offset type from loff_t to gfn_t and used pgoff_t when doing the restrictedmem_bind/unbind(). Using page index is reasonable KVM internally and sounds simpler than loff_t. But we also need initialize it to page index here as well as changes in another two cases. This is needed when restricted_offset != 0.
include/linux/kvm_host.h +1 -2
@@ -2364,8 +2364,7 @@ static inline int kvm_restricted_mem_get_pfn(struct kvm_memory_slot *slot, gfn_t gfn, kvm_pfn_t *pfn, int *order) { - pgoff_t index = gfn - slot->base_gfn + - (slot->restricted_offset >>PAGE_SHIFT ); + pgoff_t index = gfn - slot->base_gfn + slot->restricted_offset;struct page *page;int ret;
virt/kvm/kvm_main.c +2 -2
@@ -935,7 +935,7 @@ static bool restrictedmem_range_is_valid(struct kvm_memory_slot *slot, pgoff_t start, pgoff_t end, gfn_t *gfn_start, gfn_t *gfn_end) { - unsigned long base_pgoff = slot->restricted_offset >> PAGE_SHIFT; + unsigned long base_pgoff = slot->restricted_offset;if (start > base_pgoff) *gfn_start = slot->base_gfn + start - base_pgoff;
@@ -2275,7 +2275,7 @@ int __kvm_set_memory_region(struct kvm *kvm, r = -EINVAL ; goto out; } - new->restricted_offset = mem->restricted_offset; + new->restricted_offset = mem->restricted_offset >>PAGE_SHIFT ; } r =kvm_set_memslot (kvm, old, new, change);
Chao
Quoted 5 lines
Review int __kvm_set_memory_region(struct kvm *kvm, · line 2198
Quoted 3 lines
(quoted diff) +3
-0+ } + + new->kvm = kvm;Set this above, just so that the code flows better.
Sean Christopherson
maintainerRe: [PATCH v10 9/9] KVM: Enable and expose KVM_MEM_PRIVATE
Quoted 15 lines · Chao Peng
Quoted 13 lines · Sean Christopherson
Review static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · line 10369
Quoted 7 lines · Chao Peng
(quoted diff) +3
-0@@ -10357,6 +10364,12 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) if (kvm_check_request(KVM_REQ_UPDATE_CPU_DIRTY_LOGGING, vcpu)) static_call(kvm_x86_update_cpu_dirty_logging)(vcpu); + + if (kvm_check_request(KVM_REQ_MEMORY_MCE, vcpu)) { + vcpu->run->exit_reason = KVM_EXIT_SHUTDOWN;Synthesizing triple fault shutdown is not the right approach. Even with TDX's MCE "architecture" (heavy sarcasm), it's possible that host userspace and the guest have a paravirt interface for handling memory errors without killing the host.
Quoted 6 lines
Agree shutdown is not the correct choice. I see you made below change:
send_sig_mceerr(BUS_MCEERR_AR, (void __user *)hva, PAGE_SHIFT, current)The MCE may happen in any thread than KVM thread, sending siginal to 'current' thread may not be the expected behavior.
This is already true today, e.g. a #MC in memory that is mapped into the guest can be triggered by a host access. Hrm, but in this case we actually have a KVM instance, and we know that the #MC is relevant to the KVM instance, so I agree that signaling 'current' is kludgy.
Quoted 2 lines
Also how userspace can tell is the MCE on the shared page or private page? Do we care?
We care. I was originally thinking we could require userspace to keep track of things, but that's quite prescriptive and flawed, e.g. could race with conversions.
One option would be to KVM_EXIT_MEMORY_FAULT, and then wire up a generic (not x86 specific) KVM request to exit to userspace, e.g.
/* KVM_EXIT_MEMORY_FAULT */
struct {
#define KVM_MEMORY_EXIT_FLAG_PRIVATE (1ULL << 3)
#define KVM_MEMORY_EXIT_FLAG_HW_ERROR (1ULL << 4)
__u64 flags;
__u64 gpa;
__u64 size;
} memory;But I'm not sure that's the correct approach. It kinda feels like we're reinventing the wheel. It seems like restrictedmem_get_page() _must_ be able to reject attempts to get a poisoned page, i.e. restrictedmem_get_page() should yield KVM_PFN_ERR_HWPOISON. Assuming that's the case, then I believe KVM simply needs to zap SPTEs in response to an error notification in order to force vCPUs to fault on the poisoned page.
Quoted 14 lines
Quoted 5 lines
Quoted 5 lines
(quoted diff) +1
-0+ return -EINVAL; if (as_id >= KVM_ADDRESS_SPACE_NUM || id >= KVM_MEM_SLOTS_NUM) return -EINVAL; if (mem->guest_phys_addr + mem->memory_size < mem->guest_phys_addr)@@ -2020,6 +2154,9 @@ int __kvm_set_memory_region(struct kvm *kvm,Quoted 8 lines
Review int __kvm_set_memory_region(struct kvm *kvm, · line 2157
Quoted 4 lines
(quoted diff) +1
-0if ((kvm->nr_memslot_pages + npages) < kvm->nr_memslot_pages) return -EINVAL; } else { /* Modify an existing slot. */ + /* Private memslots are immutable, they can only be deleted. */I'm 99% certain I suggested this, but if we're going to make these memslots immutable, then we should straight up disallow dirty logging, otherwise we'll end up with a bizarre uAPI.
Quoted 2 lines
But in my mind dirty logging will be needed in the very short time, when live migration gets supported?
Ya, but if/when live migration support is added, private memslots will no longer be immutable as userspace will want to enable dirty logging only when a VM is being migrated, i.e. something will need to change.
Given that it looks like we have clear line of sight to SEV+UPM guests, my preference would be to allow toggling dirty logging from the get-go. It doesn't necessarily have to be in the first patch, e.g. KVM could initially reject KVM_MEM_LOG_DIRTY_PAGES + KVM_MEM_PRIVATE and then add support separately to make the series easier to review, test, and bisect.
static int check_memory_region_flags(struct kvm *kvm,
const struct kvm_userspace_memory_region2 *mem)
{
u32 valid_flags = KVM_MEM_LOG_DIRTY_PAGES;
if (kvm_arch_has_private_mem(kvm) &&
~(mem->flags & KVM_MEM_LOG_DIRTY_PAGES))
valid_flags |= KVM_MEM_PRIVATE;
...
}Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
(quoted diff) +8
-0+ if (mem->flags & KVM_MEM_PRIVATE) + return -EINVAL; if ((mem->userspace_addr != old->userspace_addr) || (npages != old->npages) || ((mem->flags ^ old->flags) & KVM_MEM_READONLY))@@ -2048,10 +2185,28 @@ int __kvm_set_memory_region(struct kvm *kvm, new->npages = npages; new->flags = mem->flags; new->userspace_addr = mem->userspace_addr; + if (mem->flags & KVM_MEM_PRIVATE) { + new->restricted_file = fget(mem->restricted_fd); + if (!new->restricted_file || + !file_is_restrictedmem(new->restricted_file)) { + r = -EINVAL; + goto out;
Quoted 8 lines
Quoted 2 lines
Review int __kvm_set_memory_region(struct kvm *kvm, · line 2195
Quoted 2 lines
(quoted diff) +2
-0+ } + new->restricted_offset = mem->restricted_offset;I see you changed
slot->restricted_offsettype fromloff_ttogfn_tand usedpgoff_twhen doing the restrictedmem_bind/unbind(). Using page index is reasonable KVM internally and sounds simpler thanloff_t. But we also need initialize it to page index here as well as changes in another two cases. This is needed whenrestricted_offset!= 0.
Oof. I'm pretty sure I completely missed that loff_t is used for byte offsets, whereas pgoff_t is a frame index.
Given that the restrictmem APIs take pgoff_t, I definitely think it makes sense to the index, but I'm very tempted to store pgoff_t instead of gfn_t, and name the field "index" to help connect the dots to the rest of kernel, where "pgoff_t index" is quite common.
And looking at those bits again, we should wrap all of the restrictedmem fields with CONFIG_KVM_PRIVATE_MEM. It'll require minor tweaks to __kvm_set_memory_region(), but I think will yield cleaner code (and internal APIs) overall.
And wrap the three fields in an anonymous struct? E.g. this is a little more versbose (restrictedmem instead restricted), but at first glance it doesn't seem to cause widespared line length issues.
#ifdef CONFIG_KVM_PRIVATE_MEM
struct {
struct file *file;
pgoff_t index;
struct restrictedmem_notifier notifier;
} restrictedmem;#endif
Quoted 43 lines
include/linux/kvm_host.h +1
-2@@ -2364,8 +2364,7 @@ static inline int kvm_restricted_mem_get_pfn(struct kvm_memory_slot *slot, gfn_t gfn, kvm_pfn_t *pfn, int *order) { - pgoff_t index = gfn - slot->base_gfn + - (slot->restricted_offset >>PAGE_SHIFT ); + pgoff_t index = gfn - slot->base_gfn + slot->restricted_offset;struct page *page;int ret;virt/kvm/kvm_main.c +2
-2@@ -935,7 +935,7 @@ static bool restrictedmem_range_is_valid(struct kvm_memory_slot *slot, pgoff_t start, pgoff_t end, gfn_t *gfn_start, gfn_t *gfn_end) { - unsigned long base_pgoff = slot->restricted_offset >> PAGE_SHIFT; + unsigned long base_pgoff = slot->restricted_offset;if (start > base_pgoff) *gfn_start = slot->base_gfn + start - base_pgoff;@@ -2275,7 +2275,7 @@ int __kvm_set_memory_region(struct kvm *kvm, r = -EINVAL ; goto out; } - new->restricted_offset = mem->restricted_offset; + new->restricted_offset = mem->restricted_offset >>PAGE_SHIFT ; } r =kvm_set_memslot (kvm, old, new, change);Chao
Quoted 5 lines
Review int __kvm_set_memory_region(struct kvm *kvm, · line 2198
Quoted 3 lines
(quoted diff) +3
-0+ } + + new->kvm = kvm;Set this above, just so that the code flows better.
Re: [PATCH v10 9/9] KVM: Enable and expose KVM_MEM_PRIVATE
Quoted 46 lines · Sean Christopherson
Quoted 15 lines · Chao Peng
Quoted 13 lines · Sean Christopherson
Review static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · line 10369
Quoted 7 lines · Chao Peng
(quoted diff) +3
-0@@ -10357,6 +10364,12 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) if (kvm_check_request(KVM_REQ_UPDATE_CPU_DIRTY_LOGGING, vcpu)) static_call(kvm_x86_update_cpu_dirty_logging)(vcpu); + + if (kvm_check_request(KVM_REQ_MEMORY_MCE, vcpu)) { + vcpu->run->exit_reason = KVM_EXIT_SHUTDOWN;Synthesizing triple fault shutdown is not the right approach. Even with TDX's MCE "architecture" (heavy sarcasm), it's possible that host userspace and the guest have a paravirt interface for handling memory errors without killing the host.
Quoted 6 lines
Agree shutdown is not the correct choice. I see you made below change:
send_sig_mceerr(BUS_MCEERR_AR, (void __user *)hva, PAGE_SHIFT, current)The MCE may happen in any thread than KVM thread, sending siginal to 'current' thread may not be the expected behavior.
This is already true today, e.g. a #MC in memory that is mapped into the guest can be triggered by a host access. Hrm, but in this case we actually have a KVM instance, and we know that the #MC is relevant to the KVM instance, so I agree that signaling 'current' is kludgy.
Quoted 2 lines
Also how userspace can tell is the MCE on the shared page or private page? Do we care?
We care. I was originally thinking we could require userspace to keep track of things, but that's quite prescriptive and flawed, e.g. could race with conversions.
One option would be to
KVM_EXIT_MEMORY_FAULT, and then wire up a generic (not x86 specific) KVM request to exit to userspace, e.g./* KVM_EXIT_MEMORY_FAULT */ struct { #define KVM_MEMORY_EXIT_FLAG_PRIVATE (1ULL << 3) #define KVM_MEMORY_EXIT_FLAG_HW_ERROR (1ULL << 4) __u64 flags; __u64 gpa; __u64 size; } memory;
Quoted 3 lines
But I'm not sure that's the correct approach. It kinda feels like we're reinventing the wheel. It seems like
restrictedmem_get_page()_must_ be able to reject attempts to get a poisoned page, i.e.restrictedmem_get_page()should yieldKVM_PFN_ERR_HWPOISON.
Yes, I see there is -EHWPOISON handling for hva_to_pfn() for shared memory. It makes sense doing similar for private page.
Quoted 2 lines
Assuming that's the case, then I believe KVM simply needs to zap SPTEs in response to an error notification in order to force vCPUs to fault on the poisoned page.
Agree, this is waht we should do anyway.
Quoted 28 lines
Quoted 15 lines
Quoted 14 lines
Review int __kvm_set_memory_region(struct kvm *kvm, · line 2157
Quoted 10 lines
(quoted diff) +2
-0+ return -EINVAL; if (as_id >= KVM_ADDRESS_SPACE_NUM || id >= KVM_MEM_SLOTS_NUM) return -EINVAL; if (mem->guest_phys_addr + mem->memory_size < mem->guest_phys_addr)@@ -2020,6 +2154,9 @@ int __kvm_set_memory_region(struct kvm *kvm, if ((kvm->nr_memslot_pages + npages) < kvm->nr_memslot_pages) return -EINVAL; } else { /* Modify an existing slot. */ + /* Private memslots are immutable, they can only be deleted. */I'm 99% certain I suggested this, but if we're going to make these memslots immutable, then we should straight up disallow dirty logging, otherwise we'll end up with a bizarre uAPI.
Quoted 2 lines
But in my mind dirty logging will be needed in the very short time, when live migration gets supported?
Ya, but if/when live migration support is added, private memslots will no longer be immutable as userspace will want to enable dirty logging only when a VM is being migrated, i.e. something will need to change.
Given that it looks like we have clear line of sight to SEV+UPM guests, my preference would be to allow toggling dirty logging from the get-go. It doesn't necessarily have to be in the first patch, e.g. KVM could initially reject
KVM_MEM_LOG_DIRTY_PAGES+KVM_MEM_PRIVATEand then add support separately to make the series easier to review, test, and bisect.
Quoted 8 lines
static int check_memory_region_flags(struct kvm *kvm,
const structkvm_userspace_memory_region2*mem){ u32 valid_flags = KVM_MEM_LOG_DIRTY_PAGES; if (kvm_arch_has_private_mem(kvm) && ~(mem->flags & KVM_MEM_LOG_DIRTY_PAGES)) valid_flags |= KVM_MEM_PRIVATE;
Adding this limitation is OK to me. It's not too hard to remove it when live migration gets added.
Quoted 45 lines
... }Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
(quoted diff) +8
-0+ if (mem->flags & KVM_MEM_PRIVATE) + return -EINVAL; if ((mem->userspace_addr != old->userspace_addr) || (npages != old->npages) || ((mem->flags ^ old->flags) & KVM_MEM_READONLY))@@ -2048,10 +2185,28 @@ int __kvm_set_memory_region(struct kvm *kvm, new->npages = npages; new->flags = mem->flags; new->userspace_addr = mem->userspace_addr; + if (mem->flags & KVM_MEM_PRIVATE) { + new->restricted_file = fget(mem->restricted_fd); + if (!new->restricted_file || + !file_is_restrictedmem(new->restricted_file)) { + r = -EINVAL; + goto out;Quoted 8 lines
Quoted 2 lines
Review int __kvm_set_memory_region(struct kvm *kvm, · line 2195
Quoted 2 lines
(quoted diff) +2
-0+ } + new->restricted_offset = mem->restricted_offset;I see you changed
slot->restricted_offsettype fromloff_ttogfn_tand usedpgoff_twhen doing the restrictedmem_bind/unbind(). Using page index is reasonable KVM internally and sounds simpler thanloff_t. But we also need initialize it to page index here as well as changes in another two cases. This is needed whenrestricted_offset!= 0.Oof. I'm pretty sure I completely missed that
loff_tis used for byte offsets, whereaspgoff_tis a frame index.Given that the restrictmem APIs take
pgoff_t, I definitely think it makes sense to the index, but I'm very tempted to storepgoff_tinstead ofgfn_t, and name the field "index" to help connect the dots to the rest of kernel, where "pgoff_tindex" is quite common.And looking at those bits again, we should wrap all of the restrictedmem fields with
CONFIG_KVM_PRIVATE_MEM. It'll require minor tweaks to__kvm_set_memory_region(), but I think will yield cleaner code (and internal APIs) overall.And wrap the three fields in an anonymous struct? E.g. this is a little more versbose (restrictedmem instead restricted), but at first glance it doesn't seem to cause widespared line length issues.
Quoted 7 lines
#ifdef CONFIG_KVM_PRIVATE_MEM struct { struct file *file; pgoff_t index; struct restrictedmem_notifier notifier; } restrictedmem;#endif
Looks better.
Thanks,
Chao
Quoted 44 lines
Quoted 44 lines
include/linux/kvm_host.h +1
-2@@ -2364,8 +2364,7 @@ static inline int kvm_restricted_mem_get_pfn(struct kvm_memory_slot *slot, gfn_t gfn, kvm_pfn_t *pfn, int *order) { - pgoff_t index = gfn - slot->base_gfn + - (slot->restricted_offset >>PAGE_SHIFT ); + pgoff_t index = gfn - slot->base_gfn + slot->restricted_offset;struct page *page;int ret;virt/kvm/kvm_main.c +2
-2@@ -935,7 +935,7 @@ static bool restrictedmem_range_is_valid(struct kvm_memory_slot *slot, pgoff_t start, pgoff_t end, gfn_t *gfn_start, gfn_t *gfn_end) { - unsigned long base_pgoff = slot->restricted_offset >> PAGE_SHIFT; + unsigned long base_pgoff = slot->restricted_offset;if (start > base_pgoff) *gfn_start = slot->base_gfn + start - base_pgoff;@@ -2275,7 +2275,7 @@ int __kvm_set_memory_region(struct kvm *kvm, r = -EINVAL ; goto out; } - new->restricted_offset = mem->restricted_offset; + new->restricted_offset = mem->restricted_offset >>PAGE_SHIFT ; } r =kvm_set_memslot (kvm, old, new, change);Chao
Quoted 5 lines
Review int __kvm_set_memory_region(struct kvm *kvm, · line 2198
Quoted 3 lines
(quoted diff) +3
-0+ } + + new->kvm = kvm;Set this above, just so that the code flows better.
Re: [PATCH v10 9/9] KVM: Enable and expose KVM_MEM_PRIVATE
In reply to Sean ChristophersonQuoted 1 line · Sean Christopherson
On Fri, Dec 02, 2022, Chao Peng wrote:
...
Quoted 6 lines
Strongly prefer to use similar logic to existing code that detects wraps:
mem->restricted_offset + mem->memory_size < mem->restricted_offsetThis is also where I'd like to add the "gfn is aligned to offset" check, though my brain is too fried to figure that out right now.
Used count_trailing_zeros() for this TODO, unsure we have other better approach.
virt/kvm/kvm_main.c +16 -1
@@ -56,6 +56,7 @@#include <asm/processor.h> #include <asm/ioctl.h> #include <linux/uaccess.h> +#include <linux/count_zeros.h> #include "coalesced_mmio.h" #include "async_pf.h"
@@ -2087,6 +2088,19 @@ static bool kvm_check_memslot_overlap(struct kvm_memslots *slots, int id,return false; } +/* +* Return true when ALIGNMENT(offset) >= ALIGNMENT(gpa). +*/ +static bool kvm_check_rmem_offset_alignment (u64 offset,u64 gpa) +{ +if (!offset) +return true; +if (!gpa) +return false; + +return !!(count_trailing_zeros (offset) >=count_trailing_zeros (gpa)); +} +/* * Allocate some memory and give it an address in the guest physical address * space.
@@ -2128,7 +2142,8 @@ int __kvm_set_memory_region(struct kvm *kvm,if (mem->flags &KVM_MEM_PRIVATE && (mem->restrictedmem_offset & (PAGE_SIZE -1 ) || mem->restrictedmem_offset + mem->memory_size < mem->restrictedmem_offset || -0 /* TODO: require gfn be aligned with restricted offset */ )) + !kvm_check_rmem_offset_alignment (mem->restrictedmem_offset, + mem->guest_phys_addr)))return -EINVAL ;if (as_id >=kvm_arch_nr_memslot_as_ids (kvm) || id >=KVM_MEM_SLOTS_NUM )return -EINVAL ;
Re: [PATCH v10 9/9] KVM: Enable and expose KVM_MEM_PRIVATE
Quoted 31 lines · Chao Peng
Quoted 1 line · Sean Christopherson
On Fri, Dec 02, 2022, Chao Peng wrote:
...
Quoted 6 lines
Strongly prefer to use similar logic to existing code that detects wraps:
mem->restricted_offset + mem->memory_size < mem->restricted_offsetThis is also where I'd like to add the "gfn is aligned to offset" check, though my brain is too fried to figure that out right now.
Used
count_trailing_zeros()for this TODO, unsure we have other better approach.virt/kvm/kvm_main.c +2
-0@@ -56,6 +56,7 @@#include <asm/processor.h> #include <asm/ioctl.h> #include <linux/uaccess.h> +#include <linux/count_zeros.h> #include "coalesced_mmio.h" #include "async_pf.h" @@ -2087,6 +2088,19 @@ static bool kvm_check_memslot_overlap(struct kvm_memslots *slots, int id, return false; } +/*
virt/kvm/kvm_main.c static bool kvm_check_memslot_overlap(struct · line 2102
Quoted 10 lines
virt/kvm/kvm_main.c +10
-0+* Return true when ALIGNMENT(offset) >= ALIGNMENT(gpa). +*/ +static bool kvm_check_rmem_offset_alignment (u64 offset,u64 gpa) +{ +if (!offset) +return true; +if (!gpa) +return false; + +return !!(count_trailing_zeros (offset) >=count_trailing_zeros (gpa));
Perhaps we could do something like
#define lowest_set_bit(val) (val & -val)
and use
return lowest_set_bit(offset) >= lowest_set_bit(gpa);
Please help me to understand: why must ALIGNMENT(offset) >= ALIGNMENT(gpa)? Why is it not sufficient to have both gpa and offset be aligned to PAGE_SIZE?
Quoted 17 lines
virt/kvm/kvm_main.c +2
-0+} +/* * Allocate some memory and give it an address in the guest physical address
* space.virt/kvm/kvm_main.c +0
-0@@ -2128,7 +2142,8 @@ int __kvm_set_memory_region(struct kvm *kvm, if (mem->flags & KVM_MEM_PRIVATE && (mem->restrictedmem_offset & (PAGE_SIZE - 1) || mem->restrictedmem_offset + mem->memory_size <
mem->restrictedmem_offset||virt/kvm/kvm_main.c +2
-1- 0 /* TODO: require gfn be aligned with restricted offset */)) + !kvm_check_rmem_offset_alignment (mem->restrictedmem_offset, + mem->guest_phys_addr))) return -EINVAL;if (as_id >=kvm_arch_nr_memslot_as_ids (kvm) || id >=KVM_MEM_SLOTS_NUM )return -EINVAL ;
Re: [PATCH v10 9/9] KVM: Enable and expose KVM_MEM_PRIVATE
Quoted 47 lines · Ackerley Tng
Quoted 32 lines · Chao Peng
Quoted 1 line · Sean Christopherson
On Fri, Dec 02, 2022, Chao Peng wrote:
...
Quoted 7 lines
Strongly prefer to use similar logic to existing code that detects wraps:
mem->restricted_offset + mem->memory_size < mem->restricted_offsetThis is also where I'd like to add the "gfn is aligned to offset" check, though
my brain is too fried to figure that out right now.Used
count_trailing_zeros()for this TODO, unsure we have other better approach.virt/kvm/kvm_main.c +2
-0@@ -56,6 +56,7 @@#include <asm/processor.h> #include <asm/ioctl.h> #include <linux/uaccess.h> +#include <linux/count_zeros.h> #include "coalesced_mmio.h" #include "async_pf.h" @@ -2087,6 +2088,19 @@ static bool kvm_check_memslot_overlap(struct kvm_memslots *slots, int id, return false; } +/* virt/kvm/kvm_main.c static bool kvm_check_memslot_overlap(struct · line 2102
Quoted 10 lines
virt/kvm/kvm_main.c +10
-0+* Return true when ALIGNMENT(offset) >= ALIGNMENT(gpa). +*/ +static bool kvm_check_rmem_offset_alignment (u64 offset,u64 gpa) +{ +if (!offset) +return true; +if (!gpa) +return false; + +return !!(count_trailing_zeros (offset) >=count_trailing_zeros (gpa));Perhaps we could do something like
Quoted 5 lines
#define
lowest_set_bit(val)(val & -val)and use
return
lowest_set_bit(offset)>=lowest_set_bit(gpa);
I see kernel already has fls64(), that looks what we need ;)
Quoted 4 lines
Please help me to understand: why must ALIGNMENT(offset) >= ALIGNMENT(gpa)? Why is it not sufficient to have both gpa and offset be aligned to
PAGE_SIZE?
Yes, it's sufficient. Here we just want to be conservative on the uAPI as Sean explained this at [1]:
I would rather reject memslot if the gfn has lesser alignment than the offset. I'm totally ok with this approach _if_ there's a use case. Until such a use case presents itself, I would rather be conservative from a uAPI perspective.
[1] https://lore.kernel.org/all/[email protected]/
Chao
Quoted 18 lines
Quoted 18 lines
virt/kvm/kvm_main.c +2
-0+} +/* * Allocate some memory and give it an address in the guest physical address
* space.virt/kvm/kvm_main.c +0
-0@@ -2128,7 +2142,8 @@ int __kvm_set_memory_region(struct kvm *kvm, if (mem->flags & KVM_MEM_PRIVATE && (mem->restrictedmem_offset & (PAGE_SIZE - 1) || mem->restrictedmem_offset + mem->memory_size <
mem->restrictedmem_offset||virt/kvm/kvm_main.c +2
-1- 0 /* TODO: require gfn be aligned with restricted offset */)) + !kvm_check_rmem_offset_alignment (mem->restrictedmem_offset, + mem->guest_phys_addr))) return -EINVAL;if (as_id >=kvm_arch_nr_memslot_as_ids (kvm) || id >=KVM_MEM_SLOTS_NUM )return -EINVAL ;
Re: [PATCH v10 9/9] KVM: Enable and expose KVM_MEM_PRIVATE
Quoted 34 lines · Chao Peng
Quoted 33 lines · Ackerley Tng
Quoted 30 lines · Chao Peng
Quoted 1 line · Sean Christopherson
On Fri, Dec 02, 2022, Chao Peng wrote:
...
Quoted 7 lines
Strongly prefer to use similar logic to existing code that detects wraps:
mem->restricted_offset + mem->memory_size < mem->restricted_offsetThis is also where I'd like to add the "gfn is aligned to offset" check, though
my brain is too fried to figure that out right now.Used
count_trailing_zeros()for this TODO, unsure we have other better approach.virt/kvm/kvm_main.c +1
-0@@ -56,6 +56,7 @@#include <asm/processor.h> #include <asm/ioctl.h> #include <linux/uaccess.h> +#include <linux/count_zeros.h> #include "coalesced_mmio.h" #include "async_pf.h" @@ -2087,6 +2088,19 @@ static bool kvm_check_memslot_overlap(struct kvm_memslots *slots, int id, return false; }
Quoted 11 lines
Quoted 11 lines
virt/kvm/kvm_main.c static bool kvm_check_memslot_overlap(struct · line 2101
Quoted 11 lines
virt/kvm/kvm_main.c +11
-0+/* +* Return true when ALIGNMENT(offset) >= ALIGNMENT(gpa). +*/ +static bool kvm_check_rmem_offset_alignment (u64 offset,u64 gpa) +{ +if (!offset) +return true; +if (!gpa) +return false; + +return !!(count_trailing_zeros (offset) >=count_trailing_zeros (gpa));
This check doesn't work expected. For example, offset = 2GB, gpa=4GB this check fails.
I come up with the following.
From ec87e25082f0497431b732702fae82c6a05071bf Mon Sep 17 00:00:00 2001
Message-Id: <ec87e25082f0497431b732702fae82c6a05071bf.1679531995.git.isaku.yamahata@intel.com>
From: Isaku Yamahata <[email protected]>
Date: Wed, 22 Mar 2023 15:32:56 -0700
Subject: [PATCH] KVM: Relax alignment check for restricted mem
kvm_check_rmem_offset_alignment() only checks based on offset alignment and GPA alignment. However, the actual alignment for offset depends on architecture. For x86 case, it can be 1G, 2M or 4K. So even if GPA is aligned for 1G+, only 1G-alignment is required for offset.
Without this patch, gpa=4G, offset=2G results in failure of memory slot creation.
--- arch/x86/include/asm/kvm_host.h | 15 +++++++++++++++ virt/kvm/kvm_main.c | 9 ++++++++- 2 files changed, 23 insertions(+), 1 deletion(-)
arch/x86/include/asm/kvm_host.h +15 -0
@@ -16,6 +16,7 @@#include <linux/irq_work.h> #include <linux/irq.h> #include <linux/workqueue.h> +#include <linux/count_zeros.h> #include <linux/kvm.h> #include <linux/kvm_para.h>
@@ -143,6 +144,20 @@#define KVM_HPAGE_MASK (x) (~(KVM_HPAGE_SIZE(x) - 1))#define KVM_PAGES_PER_HPAGE (x) (KVM_HPAGE_SIZE(x) / PAGE_SIZE) +#define kvm_arch_required_alignment kvm_arch_required_alignment +static inline int kvm_arch_required_alignment (u64 gpa) +{ +int zeros =count_trailing_zeros (gpa); + +WARN_ON_ONCE (!PAGE_ALIGNED (gpa)); +if (zeros >=KVM_HPAGE_SHIFT (PG_LEVEL_1G )) +return KVM_HPAGE_SHIFT (PG_LEVEL_1G ); +else if (zeros >=KVM_HPAGE_SHIFT (PG_LEVEL_2M )) +return KVM_HPAGE_SHIFT (PG_LEVEL_2M ); + +return PAGE_SHIFT ; +} +#define KVM_MEMSLOT_PAGES_TO_MMU_PAGES_RATIO 50#define KVM_MIN_ALLOC_MMU_PAGES 64UL#define KVM_MMU_HASH_SHIFT 12
virt/kvm/kvm_main.c +8 -1
@@ -2113,6 +2113,13 @@ static bool kvm_check_memslot_overlap(struct kvm_memslots *slots, int id,return false; } +#ifndef kvm_arch_required_alignment +__weak int kvm_arch_required_alignment (u64 gpa) +{ + return PAGE_SHIFT +} +#endif +/* * Return true when ALIGNMENT(offset) >= ALIGNMENT(gpa). */
@@ -2123,7 +2130,7 @@ static bool kvm_check_rmem_offset_alignment(u64 offset, u64 gpa)if (!gpa)return false; -return !!(count_trailing_zeros (offset) >=count_trailing_zeros (gpa)); +return !!(count_trailing_zeros (offset) >=kvm_arch_required_alignment (gpa)); }/*
Signature · 6 lines
2.25.1 -- Isaku Yamahata <[email protected]>
Re: [PATCH v10 9/9] KVM: Enable and expose KVM_MEM_PRIVATE
Quoted 37 lines · Isaku Yamahata
Quoted 34 lines · Chao Peng
Quoted 33 lines · Ackerley Tng
Quoted 30 lines · Chao Peng
Quoted 1 line · Sean Christopherson
On Fri, Dec 02, 2022, Chao Peng wrote:
...
Quoted 7 lines
Strongly prefer to use similar logic to existing code that detects wraps:
mem->restricted_offset + mem->memory_size < mem->restricted_offsetThis is also where I'd like to add the "gfn is aligned to offset" check, though
my brain is too fried to figure that out right now.Used
count_trailing_zeros()for this TODO, unsure we have other better approach.virt/kvm/kvm_main.c +1
-0@@ -56,6 +56,7 @@#include <asm/processor.h> #include <asm/ioctl.h> #include <linux/uaccess.h> +#include <linux/count_zeros.h> #include "coalesced_mmio.h" #include "async_pf.h" @@ -2087,6 +2088,19 @@ static bool kvm_check_memslot_overlap(struct kvm_memslots *slots, int id, return false; }
Quoted 14 lines
Quoted 11 lines
Quoted 11 lines
virt/kvm/kvm_main.c static bool kvm_check_memslot_overlap(struct · line 2101
Quoted 11 lines
virt/kvm/kvm_main.c +11
-0+/* +* Return true when ALIGNMENT(offset) >= ALIGNMENT(gpa). +*/ +static bool kvm_check_rmem_offset_alignment (u64 offset,u64 gpa) +{ +if (!offset) +return true; +if (!gpa) +return false; + +return !!(count_trailing_zeros (offset) >=count_trailing_zeros (gpa));This check doesn't work expected. For example, offset = 2GB, gpa=4GB this check fails.
This case is expected to fail as Sean initially suggested[*]:
I would rather reject memslot if the gfn has lesser alignment than the offset. I'm totally ok with this approach _if_ there's a use case. Until such a use case presents itself, I would rather be conservative from a uAPI perspective.
I understand that we put tighter restriction on this but if you see such restriction is really a big issue for real usage, instead of a theoretical problem, then we can loosen the check here. But at that time below code is kind of x86 specific and may need improve.
BTW, in latest code, I replaced count_trailing_zeros() with fls64():
return !!(fls64(offset) >= fls64(gpa));
[*] https://lore.kernel.org/all/[email protected]/
Chao
Quoted 90 lines
I come up with the following.
Quoted 1 line
From
ec87e25082f0497431b732702fae82c6a05071bfMon Sep 17 00:00:00 2001Message-Id: <ec87e25082f0497431b732702fae82c6a05071bf.1679531995.git.isaku.yamahata@intel.com>
From: Isaku Yamahata <[email protected]>
Date: Wed, 22 Mar 2023 15:32:56 -0700
Subject: [PATCH] KVM: Relax alignment check for restricted mem
kvm_check_rmem_offset_alignment()only checks based on offset alignment and GPA alignment. However, the actual alignment for offset depends on architecture. For x86 case, it can be 1G, 2M or 4K. So even if GPA is aligned for 1G+, only 1G-alignment is required for offset.Without this patch, gpa=4G, offset=2G results in failure of memory slot creation.
Fixes edc8814b2c77 ("KVM: Require gfn be aligned with restricted offset")Signed-off-by Isaku Yamahata <[email protected]>--- arch/x86/include/asm/kvm_host.h | 15 +++++++++++++++ virt/kvm/kvm_main.c | 9 ++++++++- 2 files changed, 23 insertions(+), 1 deletion(-)arch/x86/include/asm/kvm_host.h +15
-0@@ -16,6 +16,7 @@#include <linux/irq_work.h> #include <linux/irq.h> #include <linux/workqueue.h> +#include <linux/count_zeros.h> #include <linux/kvm.h> #include <linux/kvm_para.h> @@ -143,6 +144,20 @@#define KVM_HPAGE_MASK (x) (~(KVM_HPAGE_SIZE(x) - 1))#define KVM_PAGES_PER_HPAGE (x) (KVM_HPAGE_SIZE(x) / PAGE_SIZE) +#define kvm_arch_required_alignment kvm_arch_required_alignment +static inline int kvm_arch_required_alignment (u64 gpa) +{ +int zeros =count_trailing_zeros (gpa); + +WARN_ON_ONCE (!PAGE_ALIGNED (gpa)); +if (zeros >=KVM_HPAGE_SHIFT (PG_LEVEL_1G )) +return KVM_HPAGE_SHIFT (PG_LEVEL_1G ); +else if (zeros >=KVM_HPAGE_SHIFT (PG_LEVEL_2M )) +return KVM_HPAGE_SHIFT (PG_LEVEL_2M ); + +return PAGE_SHIFT ; +} +#define KVM_MEMSLOT_PAGES_TO_MMU_PAGES_RATIO 50#define KVM_MIN_ALLOC_MMU_PAGES 64UL#define KVM_MMU_HASH_SHIFT 12virt/kvm/kvm_main.c +8
-1@@ -2113,6 +2113,13 @@ static bool kvm_check_memslot_overlap(struct kvm_memslots *slots, int id,return false; } +#ifndef kvm_arch_required_alignment +__weak int kvm_arch_required_alignment (u64 gpa) +{ + return PAGE_SHIFT +} +#endif +/* * Return true when ALIGNMENT(offset) >= ALIGNMENT(gpa). */ @@ -2123,7 +2130,7 @@ static bool kvm_check_rmem_offset_alignment(u64 offset, u64 gpa)if (!gpa)return false; -return !!(count_trailing_zeros (offset) >=count_trailing_zeros (gpa)); +return !!(count_trailing_zeros (offset) >=kvm_arch_required_alignment (gpa)); }/* Signature · 6 lines
2.25.1 -- Isaku Yamahata <[email protected]>
Re: [PATCH v10 9/9] KVM: Enable and expose KVM_MEM_PRIVATE
Quoted 59 lines · Chao Peng
Quoted 37 lines · Isaku Yamahata
Quoted 34 lines · Chao Peng
Quoted 33 lines · Ackerley Tng
Quoted 30 lines · Chao Peng
Quoted 1 line · Sean Christopherson
On Fri, Dec 02, 2022, Chao Peng wrote:
...
Quoted 7 lines
Strongly prefer to use similar logic to existing code that detects wraps:
mem->restricted_offset + mem->memory_size < mem->restricted_offsetThis is also where I'd like to add the "gfn is aligned to offset" check, though
my brain is too fried to figure that out right now.Used
count_trailing_zeros()for this TODO, unsure we have other better approach.virt/kvm/kvm_main.c +1
-0@@ -56,6 +56,7 @@#include <asm/processor.h> #include <asm/ioctl.h> #include <linux/uaccess.h> +#include <linux/count_zeros.h> #include "coalesced_mmio.h" #include "async_pf.h" @@ -2087,6 +2088,19 @@ static bool kvm_check_memslot_overlap(struct kvm_memslots *slots, int id, return false; }Quoted 14 lines
Quoted 11 lines
Quoted 11 lines
virt/kvm/kvm_main.c static bool kvm_check_memslot_overlap(struct · line 2101
Quoted 11 lines
virt/kvm/kvm_main.c +11
-0+/* +* Return true when ALIGNMENT(offset) >= ALIGNMENT(gpa). +*/ +static bool kvm_check_rmem_offset_alignment (u64 offset,u64 gpa) +{ +if (!offset) +return true; +if (!gpa) +return false; + +return !!(count_trailing_zeros (offset) >=count_trailing_zeros (gpa));This check doesn't work expected. For example, offset = 2GB, gpa=4GB this check fails.
This case is expected to fail as Sean initially suggested[*]:
I would rather reject memslot if the gfn has lesser alignment than the offset. I'm totally ok with this approach _if_ there's a use case. Until such a use case presents itself, I would rather be conservative from a uAPI perspective.
Quoted 7 lines
I understand that we put tighter restriction on this but if you see such restriction is really a big issue for real usage, instead of a theoretical problem, then we can loosen the check here. But at that time below code is kind of x86 specific and may need improve.
BTW, in latest code, I replaced
count_trailing_zeros()withfls64():return !!(fls64(offset) >= fls64(gpa));
wouldn't it be !!(ffs64(offset) <= ffs64(gpa)) ?
Quoted 93 lines
[*] https://lore.kernel.org/all/[email protected]/
Chao
Quoted 90 lines
I come up with the following.
Quoted 1 line
From
ec87e25082f0497431b732702fae82c6a05071bfMon Sep 17 00:00:00 2001Message-Id: <ec87e25082f0497431b732702fae82c6a05071bf.1679531995.git.isaku.yamahata@intel.com>
From: Isaku Yamahata <[email protected]>
Date: Wed, 22 Mar 2023 15:32:56 -0700
Subject: [PATCH] KVM: Relax alignment check for restricted mem
kvm_check_rmem_offset_alignment()only checks based on offset alignment and GPA alignment. However, the actual alignment for offset depends on architecture. For x86 case, it can be 1G, 2M or 4K. So even if GPA is aligned for 1G+, only 1G-alignment is required for offset.Without this patch, gpa=4G, offset=2G results in failure of memory slot creation.
Fixes edc8814b2c77 ("KVM: Require gfn be aligned with restricted offset")Signed-off-by Isaku Yamahata <[email protected]>--- arch/x86/include/asm/kvm_host.h | 15 +++++++++++++++ virt/kvm/kvm_main.c | 9 ++++++++- 2 files changed, 23 insertions(+), 1 deletion(-)arch/x86/include/asm/kvm_host.h +15
-0@@ -16,6 +16,7 @@#include <linux/irq_work.h> #include <linux/irq.h> #include <linux/workqueue.h> +#include <linux/count_zeros.h> #include <linux/kvm.h> #include <linux/kvm_para.h> @@ -143,6 +144,20 @@#define KVM_HPAGE_MASK (x) (~(KVM_HPAGE_SIZE(x) - 1))#define KVM_PAGES_PER_HPAGE (x) (KVM_HPAGE_SIZE(x) / PAGE_SIZE) +#define kvm_arch_required_alignment kvm_arch_required_alignment +static inline int kvm_arch_required_alignment (u64 gpa) +{ +int zeros =count_trailing_zeros (gpa); + +WARN_ON_ONCE (!PAGE_ALIGNED (gpa)); +if (zeros >=KVM_HPAGE_SHIFT (PG_LEVEL_1G )) +return KVM_HPAGE_SHIFT (PG_LEVEL_1G ); +else if (zeros >=KVM_HPAGE_SHIFT (PG_LEVEL_2M )) +return KVM_HPAGE_SHIFT (PG_LEVEL_2M ); + +return PAGE_SHIFT ; +} +#define KVM_MEMSLOT_PAGES_TO_MMU_PAGES_RATIO 50#define KVM_MIN_ALLOC_MMU_PAGES 64UL#define KVM_MMU_HASH_SHIFT 12virt/kvm/kvm_main.c +8
-1@@ -2113,6 +2113,13 @@ static bool kvm_check_memslot_overlap(struct kvm_memslots *slots, int id,return false; } +#ifndef kvm_arch_required_alignment +__weak int kvm_arch_required_alignment (u64 gpa) +{ + return PAGE_SHIFT +} +#endif +/* * Return true when ALIGNMENT(offset) >= ALIGNMENT(gpa). */ @@ -2123,7 +2130,7 @@ static bool kvm_check_rmem_offset_alignment(u64 offset, u64 gpa)if (!gpa)return false; -return !!(count_trailing_zeros (offset) >=count_trailing_zeros (gpa)); +return !!(count_trailing_zeros (offset) >=kvm_arch_required_alignment (gpa)); }/* Signature · 6 lines
2.25.1 -- Isaku Yamahata <[email protected]>
Re: [PATCH v10 9/9] KVM: Enable and expose KVM_MEM_PRIVATE
Quoted 65 lines · Xiaoyao Li
Quoted 64 lines · Chao Peng
Quoted 37 lines · Isaku Yamahata
Quoted 34 lines · Chao Peng
Quoted 33 lines · Ackerley Tng
Quoted 30 lines · Chao Peng
Quoted 1 line · Sean Christopherson
On Fri, Dec 02, 2022, Chao Peng wrote:
...
Quoted 7 lines
Strongly prefer to use similar logic to existing code that detects wraps:
mem->restricted_offset + mem->memory_size < mem->restricted_offsetThis is also where I'd like to add the "gfn is aligned to offset" check, though
my brain is too fried to figure that out right now.Used
count_trailing_zeros()for this TODO, unsure we have other better approach.virt/kvm/kvm_main.c +1
-0@@ -56,6 +56,7 @@#include <asm/processor.h> #include <asm/ioctl.h> #include <linux/uaccess.h> +#include <linux/count_zeros.h> #include "coalesced_mmio.h" #include "async_pf.h" @@ -2087,6 +2088,19 @@ static bool kvm_check_memslot_overlap(struct kvm_memslots *slots, int id, return false; }Quoted 14 lines
Quoted 11 lines
Quoted 11 lines
virt/kvm/kvm_main.c static bool kvm_check_memslot_overlap(struct · line 2101
Quoted 11 lines
virt/kvm/kvm_main.c +11
-0+/* +* Return true when ALIGNMENT(offset) >= ALIGNMENT(gpa). +*/ +static bool kvm_check_rmem_offset_alignment (u64 offset,u64 gpa) +{ +if (!offset) +return true; +if (!gpa) +return false; + +return !!(count_trailing_zeros (offset) >=count_trailing_zeros (gpa));This check doesn't work expected. For example, offset = 2GB, gpa=4GB this check fails.
This case is expected to fail as Sean initially suggested[*]:
I would rather reject memslot if the gfn has lesser alignment than the offset. I'm totally ok with this approach _if_ there's a use case. Until such a use case presents itself, I would rather be conservative from a uAPI perspective.I understand that we put tighter restriction on this but if you see such restriction is really a big issue for real usage, instead of a theoretical problem, then we can loosen the check here. But at that time below code is kind of x86 specific and may need improve.
Quoted 4 lines
Quoted 2 lines
BTW, in latest code, I replaced
count_trailing_zeros()withfls64():return !!(fls64(offset) >= fls64(gpa));wouldn't it be !!(ffs64(offset) <= ffs64(gpa)) ?
As the function document explains, here we want to return true when ALIGNMENT(offset) >= ALIGNMENT(gpa), so '>=' is what we need.
It's worthy clarifying that in Sean's original suggestion he actually mentioned the opposite. He said 'reject memslot if the gfn has lesser alignment than the offset', but I wonder this is his purpose, since if ALIGNMENT(offset) < ALIGNMENT(gpa), we wouldn't be possible to map the page as largepage. Consider we have below config:
gpa=2M, offset=1M
In this case KVM tries to map gpa at 2M as 2M hugepage but the physical page at the offset(1M) in private_fd cannot provide the 2M page due to misalignment.
But as we discussed in the off-list thread, here we do find a real use case indicating this check is too strict. i.e. QEMU immediately fails when launch a guest > 2G memory. For this case QEMU splits guest memory space into two slots:
Slot#1(ram_below_4G): gpa=0x0, offset=0x0, size=2G Slot#2(ram_above_4G): gpa=4G, offset=2G, size=totalsize-2G
This strict alignment check fails for slot#2 because offset(2G) has less alignment than gpa(4G). To allow this, one solution can revert to my previous change in kvm_alloc_memslot_metadata() to disallow hugepage only when the offset/gpa are not aligned to related page size.
Sean, How do you think?
Chao
Quoted 89 lines
Quoted 89 lines
[*] https://lore.kernel.org/all/[email protected]/
Chao
Quoted 85 lines
I come up with the following.
Quoted 1 line
From
ec87e25082f0497431b732702fae82c6a05071bfMon Sep 17 00:00:00 2001Message-Id: <ec87e25082f0497431b732702fae82c6a05071bf.1679531995.git.isaku.yamahata@intel.com>
From: Isaku Yamahata <[email protected]>
Date: Wed, 22 Mar 2023 15:32:56 -0700
Subject: [PATCH] KVM: Relax alignment check for restricted mem
kvm_check_rmem_offset_alignment()only checks based on offset alignment and GPA alignment. However, the actual alignment for offset depends on architecture. For x86 case, it can be 1G, 2M or 4K. So even if GPA is aligned for 1G+, only 1G-alignment is required for offset.Without this patch, gpa=4G, offset=2G results in failure of memory slot creation.
Fixes edc8814b2c77 ("KVM: Require gfn be aligned with restricted offset")Signed-off-by Isaku Yamahata <[email protected]>--- arch/x86/include/asm/kvm_host.h | 15 +++++++++++++++ virt/kvm/kvm_main.c | 9 ++++++++- 2 files changed, 23 insertions(+), 1 deletion(-)arch/x86/include/asm/kvm_host.h +15
-0@@ -16,6 +16,7 @@#include <linux/irq_work.h> #include <linux/irq.h> #include <linux/workqueue.h> +#include <linux/count_zeros.h> #include <linux/kvm.h> #include <linux/kvm_para.h> @@ -143,6 +144,20 @@#define KVM_HPAGE_MASK (x) (~(KVM_HPAGE_SIZE(x) - 1))#define KVM_PAGES_PER_HPAGE (x) (KVM_HPAGE_SIZE(x) / PAGE_SIZE) +#define kvm_arch_required_alignment kvm_arch_required_alignment +static inline int kvm_arch_required_alignment (u64 gpa) +{ +int zeros =count_trailing_zeros (gpa); + +WARN_ON_ONCE (!PAGE_ALIGNED (gpa)); +if (zeros >=KVM_HPAGE_SHIFT (PG_LEVEL_1G )) +return KVM_HPAGE_SHIFT (PG_LEVEL_1G ); +else if (zeros >=KVM_HPAGE_SHIFT (PG_LEVEL_2M )) +return KVM_HPAGE_SHIFT (PG_LEVEL_2M ); + +return PAGE_SHIFT ; +} +#define KVM_MEMSLOT_PAGES_TO_MMU_PAGES_RATIO 50#define KVM_MIN_ALLOC_MMU_PAGES 64UL#define KVM_MMU_HASH_SHIFT 12virt/kvm/kvm_main.c +8
-1@@ -2113,6 +2113,13 @@ static bool kvm_check_memslot_overlap(struct kvm_memslots *slots, int id,return false; } +#ifndef kvm_arch_required_alignment +__weak int kvm_arch_required_alignment (u64 gpa) +{ + return PAGE_SHIFT +} +#endif +/* * Return true when ALIGNMENT(offset) >= ALIGNMENT(gpa). */ @@ -2123,7 +2130,7 @@ static bool kvm_check_rmem_offset_alignment(u64 offset, u64 gpa)if (!gpa)return false; -return !!(count_trailing_zeros (offset) >=count_trailing_zeros (gpa)); +return !!(count_trailing_zeros (offset) >=kvm_arch_required_alignment (gpa)); }/* Signature · 6 lines
2.25.1 -- Isaku Yamahata <[email protected]>
Sean Christopherson
maintainerRe: [PATCH v10 9/9] KVM: Enable and expose KVM_MEM_PRIVATE
Quoted 62 lines · Chao Peng
Quoted 33 lines · Xiaoyao Li
Quoted 32 lines · Chao Peng
Quoted 8 lines · Isaku Yamahata
Quoted 5 lines · Chao Peng
Quoted 4 lines · Ackerley Tng
Quoted 2 lines · Chao Peng
Quoted 1 line · Sean Christopherson
On Fri, Dec 02, 2022, Chao Peng wrote:
Quoted 11 lines
Quoted 8 lines
Quoted 8 lines
Review
Quoted 8 lines
(quoted diff) +8
-0+static bool kvm_check_rmem_offset_alignment(u64 offset, u64 gpa) +{ + if (!offset) + return true; + if (!gpa) + return false; + + return !!(count_trailing_zeros(offset) >= count_trailing_zeros(gpa));This check doesn't work expected. For example, offset = 2GB, gpa=4GB this check fails.
This case is expected to fail as Sean initially suggested[*]:
I would rather reject memslot if the gfn has lesser alignment than the offset. I'm totally ok with this approach _if_ there's a use case. Until such a use case presents itself, I would rather be conservative from a uAPI perspective.I understand that we put tighter restriction on this but if you see such restriction is really a big issue for real usage, instead of a theoretical problem, then we can loosen the check here. But at that time below code is kind of x86 specific and may need improve.
Quoted 4 lines
Quoted 2 lines
BTW, in latest code, I replaced
count_trailing_zeros()withfls64():return !!(fls64(offset) >= fls64(gpa));wouldn't it be !!(ffs64(offset) <= ffs64(gpa)) ?
As the function document explains, here we want to return true when ALIGNMENT(offset) >= ALIGNMENT(gpa), so '>=' is what we need.
It's worthy clarifying that in Sean's original suggestion he actually mentioned the opposite. He said 'reject memslot if the gfn has lesser alignment than the offset', but I wonder this is his purpose, since if ALIGNMENT(offset) < ALIGNMENT(gpa), we wouldn't be possible to map the page as largepage. Consider we have below config:
gpa=2M, offset=1M
In this case KVM tries to map gpa at 2M as 2M hugepage but the physical page at the offset(1M) in
private_fdcannot provide the 2M page due to misalignment.But as we discussed in the off-list thread, here we do find a real use case indicating this check is too strict. i.e. QEMU immediately fails when launch a guest > 2G memory. For this case QEMU splits guest memory space into two slots:
Slot#1(ram_below_4G): gpa=0x0, offset=0x0, size=2G Slot#2(ram_above_4G): gpa=4G, offset=2G, size=totalsize-2G
Quoted 6 lines
This strict alignment check fails for slot#2 because offset(2G) has less alignment than gpa(4G). To allow this, one solution can revert to my previous change in
kvm_alloc_memslot_metadata()to disallow hugepage only when the offset/gpa are not aligned to related page size.Sean, How do you think?
I agree, a pure alignment check is too restrictive, and not really what I intended despite past me literally saying that's what I wanted :-) I think I may have also inverted the "less alignment" statement, but luckily I believe that ends up being a moot point.
The goal is to avoid having to juggle scenarios where KVM wants to create a hugepage, but restrictedmem can't provide one because of a misaligned file offset. I think the rule we want is that the offset must be aligned to the largest page size allowed by the memslot _size_. E.g. on x86, if the memslot size is >=1GiB then the offset must be 1GiB or beter, ditto for >=2MiB and >=4KiB (ignoring that 4KiB is already a requirement).
We could loosen that to say the largest size allowed by the memslot, but I don't think that's worth the effort unless it's trivially easy to implement in code, e.g. KVM could technically allow a 4KiB aligned offset if the memslot is 2MiB sized but only 4KiB aligned on the GPA. I doubt there's a real use case for such a memslot, so I want to disallow that unless it's super easy to implement.
Re: [PATCH v10 9/9] KVM: Enable and expose KVM_MEM_PRIVATE
Quoted 90 lines · Sean Christopherson
Quoted 70 lines · Chao Peng
Quoted 9 lines · Xiaoyao Li
Quoted 8 lines · Chao Peng
Quoted 7 lines · Isaku Yamahata
Quoted 4 lines · Chao Peng
Quoted 3 lines · Ackerley Tng
Quoted 1 line · Chao Peng
On Sat, Jan 14, 2023 at 12:01:01AM +0000, Sean
Quoted 2 lines · Christopherson
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line · Chao Peng
+static bool kvm_check_rmem_offset_alignment(u64 offset, u64
gpa)
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Review
Quoted 7 lines
(quoted diff) +7
-0+{ + if (!offset) + return true; + if (!gpa) + return false; + + return !!(count_trailing_zeros(offset) >=
count_trailing_zeros(gpa));Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
This check doesn't work expected. For example, offset = 2GB,
gpa=4GB
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
this check fails.
This case is expected to fail as Sean initially suggested[*]:
I would rather reject memslot if the gfn has lesser alignment than the offset. I'm totally ok with this approach _if_ there's a usecase.
Quoted 1 line
Quoted 1 line
Until such a use case presents itself, I would rather be
conservative
Quoted 3 lines
Quoted 3 lines
from a uAPI perspective.
I understand that we put tighter restriction on this but if you see
such
Quoted 2 lines
Quoted 2 lines
restriction is really a big issue for real usage, instead of a theoretical problem, then we can loosen the check here. But at that
time
Quoted 6 lines
Quoted 4 lines
below code is kind of x86 specific and may need improve.
BTW, in latest code, I replaced
count_trailing_zeros()withfls64():return !!(fls64(offset) >= fls64(gpa));wouldn't it be !!(ffs64(offset) <= ffs64(gpa)) ?
As the function document explains, here we want to return true when ALIGNMENT(offset) >= ALIGNMENT(gpa), so '>=' is what we need.
It's worthy clarifying that in Sean's original suggestion he actually mentioned the opposite. He said 'reject memslot if the gfn has lesser alignment than the offset', but I wonder this is his purpose, since if ALIGNMENT(offset) < ALIGNMENT(gpa), we wouldn't be possible to map the page as largepage. Consider we have below config:
gpa=2M, offset=1MIn this case KVM tries to map gpa at 2M as 2M hugepage but the physical page at the offset(1M) in
private_fdcannot provide the 2M page due to misalignment.But as we discussed in the off-list thread, here we do find a real use case indicating this check is too strict. i.e. QEMU immediately fails when launch a guest > 2G memory. For this case QEMU splits guest memory space into two slots:
Slot#1(ram_below_4G): gpa=0x0, offset=0x0, size=2G Slot#2(ram_above_4G): gpa=4G, offset=2G, size=totalsize-2GQuoted 6 lines
This strict alignment check fails for slot#2 because offset(2G) has less alignment than gpa(4G). To allow this, one solution can revert to my previous change in
kvm_alloc_memslot_metadata()to disallow hugepage only when the offset/gpa are not aligned to related page size.Sean, How do you think?
I agree, a pure alignment check is too restrictive, and not really what I intended despite past me literally saying that's what I wanted :-) I think I may have also inverted the "less alignment" statement, but luckily I believe that ends up being a moot point.
The goal is to avoid having to juggle scenarios where KVM wants to create a hugepage, but restrictedmem can't provide one because of a misaligned file offset. I think the rule we want is that the offset must be aligned to the largest page size allowed by the memslot _size_. E.g. on x86, if the memslot size is >=1GiB then the offset must be 1GiB or beter, ditto for >=2MiB and >=4KiB (ignoring that 4KiB is already a requirement).
Quoted 5 lines
We could loosen that to say the largest size allowed by the memslot, but I don't think that's worth the effort unless it's trivially easy to implement in code, e.g. KVM could technically allow a 4KiB aligned offset if the memslot is 2MiB sized but only 4KiB aligned on the GPA. I doubt there's a real use case for such a memslot, so I want to disallow that unless it's super easy to implement.
Checking my understanding here about why we need this alignment check:
When KVM requests a page from restrictedmem, KVM will provide an offset into the file in terms of 4K pages.
When shmem is configured to use hugepages, shmem_get_folio() will round the requested offset down to the nearest hugepage-aligned boundary in shmem_alloc_hugefolio().
Example of problematic configuration provided toKVM_SET_USER_MEMORY_REGION2:
+ shmem configured to use 1GB pages
+ restrictedmem_offset provided to KVM_SET_USER_MEMORY_REGION2: 0x4000
+ memory_size provided in KVM_SET_USER_MEMORY_REGION2: 1GB
+ KVM requests offset (pgoff_t) 0x8, which translates to offset 0x8000
restrictedmem_get_page() and shmem_get_folio() returns the page for offset 0x0 in the file, since rounding down 0x8000 to the nearest 1GB is 0x0. This is allocating outside the range that KVM is supposed to use, since the parameters provided in KVM_SET_USER_MEMORY_REGION2 is only supposed to be offset 0x4000 to (0x4000 + 1GB = 0x40004000) in the file.
IIUC shmem will actually just round down (0x4000 rounded down to nearest 1GB will be 0x0) and allocate without checking bounds, so if offset 0x0 to 0x4000 in the file were supposed to be used by something else, there might be issues.
Hence, this alignment check ensures that rounding down of any offsets provided by KVM (based on page size configured in the backing file provided) to restrictedmem_get_page() must not go below the offset provided to KVM_SET_USER_MEMORY_REGION2.
Enforcing alignment of restrictedmem_offset based on the currently-set page size in the backing file (i.e. shmem) may not be effective, since the size of the pages in the backing file can be adjusted to a larger size after KVM_SET_USER_MEMORY_REGION2 succeeds. With that, we may still end up allocating outside the range that KVM was provided with.
Hence, to be safe, we should check alignment to the max page size across all backing filesystems, so the constraint is
rounding down restrictedmem_offset to
min(max page size across all backing filesystems,
max page size that fits in memory_size) == restrictedmem_offset
which is the same check as
restrictedmem_offset must be aligned to min(max page size across all backing filesystems, max page size that fits in memory_size)
which can safely reduce to
restrictedmem_offset must be aligned to max page size that fits in memory_size
since "max page size that fits in memory_size" is probably <= to "max page size across all backing filesystems", and if it's larger, it'll just be a tighter constraint.
If the above understanding is correct:
+ We must enforce this in the KVM_SET_USER_MEMORY_REGION2 handler, since IIUC shmem will just round down and allocate without checking bounds.
+ I think this is okay because holes in the restrictedmem file (in terms of offset) made to accommodate this constraint don't cost us anything anyway(?) Are they just arbitrary offsets in a file? In our case, this file is usually a new and empty file.
+ In the case of migration of a restrictedmem file between two KVM VMs, this constraint would cause a problem is if the largest possible page size on the destination machine is larger than that of the source machine. In that case, we might have to move the data in the file to a different offset (a separate problem).
+ On this note, it seems like there is no check for when the range is smaller than the allocated page? Like if the range provided is 4KB in size, but shmem is then configured to use a 1GB page, will we end up allocating past the end of the range?
Sean Christopherson
maintainerRe: [PATCH v10 9/9] KVM: Enable and expose KVM_MEM_PRIVATE
Quoted 71 lines · Ackerley Tng
Quoted 10 lines · Sean Christopherson
I agree, a pure alignment check is too restrictive, and not really what I intended despite past me literally saying that's what I wanted :-) I think I may have also inverted the "less alignment" statement, but luckily I believe that ends up being a moot point.
The goal is to avoid having to juggle scenarios where KVM wants to create a hugepage, but restrictedmem can't provide one because of a misaligned file offset. I think the rule we want is that the offset must be aligned to the largest page size allowed by the memslot _size_. E.g. on x86, if the memslot size is >=1GiB then the offset must be 1GiB or beter, ditto for
Quoted 8 lines
Quoted 1 line
=2MiB and >=4KiB (ignoring that 4KiB is already a requirement).
We could loosen that to say the largest size allowed by the memslot, but I don't think that's worth the effort unless it's trivially easy to implement in code, e.g. KVM could technically allow a 4KiB aligned offset if the memslot is 2MiB sized but only 4KiB aligned on the GPA. I doubt there's a real use case for such a memslot, so I want to disallow that unless it's super easy to implement.
Checking my understanding here about why we need this alignment check:
When KVM requests a page from restrictedmem, KVM will provide an offset into the file in terms of 4K pages.
When shmem is configured to use hugepages,
shmem_get_folio()will round the requested offset down to the nearest hugepage-aligned boundary inshmem_alloc_hugefolio().Example of problematic configuration provided to
KVM_SET_USER_MEMORY_REGION2:+ shmem configured to use 1GB pages
+restrictedmem_offsetprovided toKVM_SET_USER_MEMORY_REGION2:0x4000
+memory_sizeprovided inKVM_SET_USER_MEMORY_REGION2: 1GB
+ KVM requests offset (pgoff_t)0x8, which translates to offset0x8000
restrictedmem_get_page()andshmem_get_folio()returns the page for offset0x0in the file, since rounding down0x8000to the nearest 1GB is0x0. This is allocating outside the range that KVM is supposed to use, since the parameters provided inKVM_SET_USER_MEMORY_REGION2is only supposed to be offset0x4000to (0x4000+ 1GB =0x40004000) in the file.IIUC shmem will actually just round down (
0x4000rounded down to nearest 1GB will be0x0) and allocate without checking bounds, so if offset0x0to0x4000in the file were supposed to be used by something else, there might be issues.Hence, this alignment check ensures that rounding down of any offsets provided by KVM (based on page size configured in the backing file provided) to
restrictedmem_get_page()must not go below the offset provided toKVM_SET_USER_MEMORY_REGION2.Enforcing alignment of
restrictedmem_offsetbased on the currently-set page size in the backing file (i.e. shmem) may not be effective, since the size of the pages in the backing file can be adjusted to a larger size afterKVM_SET_USER_MEMORY_REGION2succeeds. With that, we may still end up allocating outside the range that KVM was provided with.Hence, to be safe, we should check alignment to the max page size across all backing filesystems, so the constraint is
rounding down
restrictedmem_offsetto
min(max page size across all backing filesystems,
max page size that fits inmemory_size) ==restrictedmem_offsetwhich is the same check as
restrictedmem_offsetmust be aligned tomin(max page size across all backing filesystems, max page size that fits in memory_size)
Quoted 8 lines
which can safely reduce to
restrictedmem_offsetmust be aligned to max page size that fits inmemory_sizesince "max page size that fits in
memory_size" is probably <= to "max page size across all backing filesystems", and if it's larger, it'll just be a tighter constraint.
Yes? The alignment check isn't strictly required, KVM _could_ deal with the above scenario, it's just a lot simpler and safer for KVM if the file offset needs to be sanely aligned.
Quoted 10 lines
If the above understanding is correct:
+ We must enforce this in the
KVM_SET_USER_MEMORY_REGION2handler, since IIUC shmem will just round down and allocate without checking bounds.+ I think this is okay because holes in the restrictedmem file (in terms of offset) made to accommodate this constraint don't cost us anything anyway(?) Are they just arbitrary offsets in a file? In our case, this file is usually a new and empty file.
Quoted 5 lines
+ In the case of migration of a restrictedmem file between two KVM VMs, this constraint would cause a problem is if the largest possible page size on the destination machine is larger than that of the source machine. In that case, we might have to move the data in the file to a different offset (a separate problem).
Hmm, I was thinking this would be a non-issue because the check would be tied to the max page _possible_ page size irrespective of hardware support, but that would be problematic if KVM ever supports 512GiB pages. I'm not sure that speculatively requiring super huge memslots to be 512GiB aligned is sensible.
Aha! If we go with a KVM ioctl(), a clean way around this is tie the alignment requirement to the memfd flags, e.g. if userspace requests the memfd to be backed by PMD hugepages, then the memslot offset needs to be 2MiB aligned on x86. That will continue to work if (big if) KVM supports 512GiB pages because the "legacy" memfd would still be capped at 2MiB pages.
Architectures that support variable hugepage sizes might need to do something else, but I don't think that possibility affects what x86 can/can't do.
Quoted 4 lines
+ On this note, it seems like there is no check for when the range is smaller than the allocated page? Like if the range provided is 4KB in size, but shmem is then configured to use a 1GB page, will we end up allocating past the end of the range?
No, KVM already gracefully handles situations like this. Well, x86 does, I assume other architectures do too :-)
As above, the intent of the extra restriction is so that KVM doen't need even more weird code (read: math) to gracefully handle the new edge cases that would come with fd-only memslots.
Re: [PATCH v10 9/9] KVM: Enable and expose KVM_MEM_PRIVATE
In reply to Chao PengQuoted 11 lines · Chao Peng
Register/unregister private memslot to fd-based memory backing store restrictedmem and implement the callbacks for
restrictedmem_notifier:
- invalidate_start()/invalidate_end() to zap the existing memory mappings in the KVM page table.
-error()to requestKVM_REQ_MEMORY_MCEand later exit to userspace withKVM_EXIT_SHUTDOWN.Expose
KVM_MEM_PRIVATEfor memslot andKVM_MEMORY_ATTRIBUTE_PRIVATEforKVM_GET_SUPPORTED_MEMORY_ATTRIBUTESto userspace but either are controlled bykvm_arch_has_private_mem()which should be rewritten by architecture code.
Could we perhaps rename KVM_MEM_PRIVATE to KVM_MEM_PROTECTED, to be in line with KVM_X86_PROTECTED_VM?
I feel that a memslot that has the KVM_MEM_PRIVATE flag need not always be private; It can sometimes be providing memory that is shared and also accessible from the host.
KVM_MEMORY_ATTRIBUTE_PRIVATE is fine as-is because this flag is set when the guest memory is meant to be backed by private memory.
KVM_MEMORY_EXIT_FLAG_PRIVATE is also okay because the flag is used to indicate when the memory error is caused by a private access (as opposed to a shared access).
kvm_slot_can_be_private() could perhaps be renamed kvm_is_protected_slot()?
Quoted 368 lines
Co-developed-by Yu Zhang <[email protected]>Signed-off-by Yu Zhang <[email protected]>Signed-off-by Chao Peng <[email protected]>Reviewed-by Fuad Tabba <[email protected]>--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/x86.c | 13 +++ include/linux/kvm_host.h | 3 + virt/kvm/kvm_main.c | 179 +++++++++++++++++++++++++++++++- 4 files changed, 191 insertions(+), 5 deletions(-)arch/x86/include/asm/kvm_host.h +1
-0@@ -114,6 +114,7 @@ KVM_ARCH_REQ_FLAGS(31, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)#define KVM_REQ_HV_TLB_FLUSH \ KVM_ARCH_REQ_FLAGS(32, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP) +#define KVM_REQ_MEMORY_MCE KVM_ARCH_REQ(33)#define CR0_RESERVED_BITS \ (~(unsigned long)(X86_CR0_PE | X86_CR0_MP | X86_CR0_EM | X86_CR0_TS \arch/x86/kvm/x86.c +7
-0@@ -6587,6 +6587,13 @@ int kvm_arch_pm_notifier(struct kvm *kvm, unsigned long state) }#endif /* CONFIG_HAVE_KVM_PM_NOTIFIER */ +#ifdef CONFIG_HAVE_KVM_RESTRICTED_MEM +void kvm_arch_memory_mce (struct kvm *kvm) +{ +kvm_make_all_cpus_request (kvm,KVM_REQ_MEMORY_MCE ); +} +#endif +static int kvm_vm_ioctl_get_clock (struct kvm *kvm,void __user *argp) {struct kvm_clock_data data = { 0 };arch/x86/kvm/x86.c +6
-0@@ -10357,6 +10364,12 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)if (kvm_check_request (KVM_REQ_UPDATE_CPU_DIRTY_LOGGING , vcpu))static_call (kvm_x86_update_cpu_dirty_logging)(vcpu); + +if (kvm_check_request (KVM_REQ_MEMORY_MCE , vcpu)) { + vcpu->run->exit_reason =KVM_EXIT_SHUTDOWN ; + r =0 ; + goto out; + } }if (kvm_check_request(KVM_REQ_EVENT, vcpu) || req_int_win ||include/linux/kvm_host.h +1
-0@@ -590,6 +590,7 @@ struct kvm_memory_slot {struct file *restricted_file;loff_t restricted_offset;struct restrictedmem_notifier notifier; +struct kvm *kvm; }; static inline bool kvm_slot_can_be_private(const struct kvm_memory_slot*slot)
include/linux/kvm_host.h +2
-0@@ -2363,6 +2364,8 @@ static inline int kvm_restricted_mem_get_pfn(structkvm_memory_slot *slot, *pfn =page_to_pfn (page); return ret; } + +void kvm_arch_memory_mce (struct kvm *kvm);#endif /* CONFIG_HAVE_KVM_RESTRICTED_MEM */ #endifvirt/kvm/kvm_main.c +84
-0@@ -936,6 +936,121 @@ static int kvm_init_mmu_notifier(struct kvm *kvm)#endif /* CONFIG_MMU_NOTIFIER && KVM_ARCH_WANT_MMU_NOTIFIER */ +#ifdef CONFIG_HAVE_KVM_RESTRICTED_MEM +static bool restrictedmem_range_is_valid (struct kvm_memory_slot *slot, +pgoff_t start,pgoff_t end, +gfn_t *gfn_start,gfn_t *gfn_end) +{ +unsigned long base_pgoff = slot->restricted_offset >>PAGE_SHIFT ; + +if (start > base_pgoff) + *gfn_start = slot->base_gfn + start - base_pgoff; +else + *gfn_start = slot->base_gfn; + +if (end < base_pgoff + slot->npages) + *gfn_end = slot->base_gfn + end - base_pgoff; +else + *gfn_end = slot->base_gfn + slot->npages; + +if (*gfn_start >= *gfn_end) +return false; + +return true; +} + +static void kvm_restrictedmem_invalidate_begin (struct restrictedmem_notifier *notifier, +pgoff_t start,pgoff_t end) +{ +struct kvm_memory_slot *slot =container_of (notifier, + struct kvm_memory_slot, + notifier); +struct kvm *kvm = slot->kvm; +gfn_t gfn_start, gfn_end; +struct kvm_gfn_range gfn_range; +int idx; + +if (!restrictedmem_range_is_valid (slot, start, end, + &gfn_start, &gfn_end)) +return ; + + gfn_range.start = gfn_start; + gfn_range.end = gfn_end; + gfn_range.slot = slot; + gfn_range.pte =__pte (0 ); + gfn_range.may_block = true; + + idx =srcu_read_lock (&kvm->srcu); +KVM_MMU_LOCK (kvm); + +kvm_mmu_invalidate_begin (kvm); +kvm_mmu_invalidate_range_add (kvm, gfn_start, gfn_end); +if (kvm_unmap_gfn_range (kvm, &gfn_range)) +kvm_flush_remote_tlbs (kvm); + +KVM_MMU_UNLOCK (kvm); +srcu_read_unlock (&kvm->srcu, idx); +} + +static void kvm_restrictedmem_invalidate_end (struct restrictedmem_notifier *notifier, +pgoff_t start,pgoff_t end) +{ +struct kvm_memory_slot *slot =container_of (notifier, + struct kvm_memory_slot, + notifier); +struct kvm *kvm = slot->kvm; +gfn_t gfn_start, gfn_end; + +if (!restrictedmem_range_is_valid (slot, start, end, + &gfn_start, &gfn_end)) +return ; + +KVM_MMU_LOCK (kvm); +kvm_mmu_invalidate_end (kvm); +KVM_MMU_UNLOCK (kvm); +} + +static void kvm_restrictedmem_error (struct restrictedmem_notifier *notifier, +pgoff_t start,pgoff_t end) +{ +struct kvm_memory_slot *slot =container_of (notifier, + struct kvm_memory_slot, + notifier); +kvm_arch_memory_mce (slot->kvm); +} + +static struct restrictedmem_notifier_ops kvm_restrictedmem_notifier_ops= {
virt/kvm/kvm_main.c +6
-0+ .invalidate_start = kvm_restrictedmem_invalidate_begin, + .invalidate_end = kvm_restrictedmem_invalidate_end, + .error = kvm_restrictedmem_error, +}; + +static inline void kvm_restrictedmem_register(struct kvm_memory_slot*slot)
virt/kvm/kvm_main.c +6
-0+{ + slot->notifier.ops = &kvm_restrictedmem_notifier_ops; +restrictedmem_register_notifier (slot->restricted_file, &slot->notifier); +} + +static inline void kvm_restrictedmem_unregister(struct kvm_memory_slot*slot)
virt/kvm/kvm_main.c +8
-0+{ +restrictedmem_unregister_notifier (slot->restricted_file, + &slot->notifier); +} + +#else /* !CONFIG_HAVE_KVM_RESTRICTED_MEM */ + +static inline void kvm_restrictedmem_register(struct kvm_memory_slot*slot)
virt/kvm/kvm_main.c +5
-0+{ +WARN_ON_ONCE (1 ); +} + +static inline void kvm_restrictedmem_unregister(struct kvm_memory_slot*slot)
virt/kvm/kvm_main.c +6
-0+{ +WARN_ON_ONCE (1 ); +} + +#endif /* CONFIG_HAVE_KVM_RESTRICTED_MEM */ +#ifdef CONFIG_HAVE_KVM_PM_NOTIFIER static int kvm_pm_notifier_call(struct notifier_block *bl, unsigned long state,@@ -980,6 +1095,11 @@ static void kvm_destroy_dirty_bitmap(struct kvm_memory_slot *memslot)/* This does not remove the slot from struct kvm_memslots data structures */
static void kvm_free_memslot(struct kvm *kvm, structkvm_memory_slot
*slot)
{virt/kvm/kvm_main.c +10
-1+if (slot->flags &KVM_MEM_PRIVATE ) { +kvm_restrictedmem_unregister (slot); +fput (slot->restricted_file); + } +kvm_destroy_dirty_bitmap (slot);kvm_arch_free_memslot (kvm, slot);@@ -1551,10 +1671,14 @@ static void kvm_replace_memslot(struct kvm *kvm, } } -static int check_memory_region_flags (const struct kvm_user_mem_region *mem) +static int check_memory_region_flags (struct kvm *kvm, +const struct kvm_user_mem_region *mem) {u32 valid_flags =KVM_MEM_LOG_DIRTY_PAGES ; +if (kvm_arch_has_private_mem (kvm)) + valid_flags |=KVM_MEM_PRIVATE ; +#ifdef __KVM_HAVE_READONLY_MEM valid_flags |=KVM_MEM_READONLY ;#endifvirt/kvm/kvm_main.c +3
-0@@ -1630,6 +1754,9 @@ static int kvm_prepare_memory_region(struct kvm *kvm, {int r; +if (change ==KVM_MR_CREATE && new->flags &KVM_MEM_PRIVATE ) +kvm_restrictedmem_register (new); +/* * If dirty logging is disabled, nullify the bitmap; the old bitmap * will be freed on "commit". If logging is enabled in both old and
virt/kvm/kvm_main.c +46
-4@@ -1658,6 +1785,9 @@ static int kvm_prepare_memory_region(struct kvm *kvm, if (r && new && new->dirty_bitmap && (!old || !old->dirty_bitmap))kvm_destroy_dirty_bitmap (new); +if (r && change ==KVM_MR_CREATE && new->flags &KVM_MEM_PRIVATE ) +kvm_restrictedmem_unregister (new); +return r; }@@ -1963,7 +2093,7 @@ int __kvm_set_memory_region(struct kvm *kvm,int as_id, id;int r; - r =check_memory_region_flags (mem); + r =check_memory_region_flags (kvm, mem);if (r)return r;@@ -1982,6 +2112,10 @@ int __kvm_set_memory_region(struct kvm *kvm, !access_ok ((void __user *)(unsigned long )mem->userspace_addr, mem->memory_size)) return -EINVAL; +if (mem->flags &KVM_MEM_PRIVATE && + (mem->restricted_offset & (PAGE_SIZE -1 ) || + mem->restricted_offset >U64_MAX - mem->memory_size)) +return -EINVAL ;if (as_id >=KVM_ADDRESS_SPACE_NUM || id >=KVM_MEM_SLOTS_NUM )return -EINVAL ;if (mem->guest_phys_addr + mem->memory_size < mem->guest_phys_addr)@@ -2020,6 +2154,9 @@ int __kvm_set_memory_region(struct kvm *kvm,if ((kvm->nr_memslot_pages + npages) < kvm->nr_memslot_pages)return -EINVAL ; }else {/* Modify an existing slot. */ +/* Private memslots are immutable, they can only be deleted. */ +if (mem->flags &KVM_MEM_PRIVATE ) +return -EINVAL ;if ((mem->userspace_addr != old->userspace_addr) || (npages != old->npages) || ((mem->flags ^ old->flags) &KVM_MEM_READONLY ))@@ -2048,10 +2185,28 @@ int __kvm_set_memory_region(struct kvm *kvm, new->npages = npages; new->flags = mem->flags; new->userspace_addr = mem->userspace_addr; +if (mem->flags &KVM_MEM_PRIVATE ) { + new->restricted_file =fget (mem->restricted_fd); +if (!new->restricted_file || + !file_is_restrictedmem (new->restricted_file)) { + r = -EINVAL ; + goto out; + } + new->restricted_offset = mem->restricted_offset; + } + + new->kvm = kvm; r =kvm_set_memslot (kvm, old, new, change);if (r) -kfree (new); + goto out; + +return 0 ; + +out: +if (new->restricted_file) +fput (new->restricted_file); +kfree (new);return r; }EXPORT_SYMBOL_GPL (__kvm_set_memory_region);@@ -2351,6 +2506,8 @@ static int kvm_vm_ioctl_clear_dirty_log(struct kvm *kvm,#ifdef CONFIG_HAVE_KVM_MEMORY_ATTRIBUTES static u64 kvm_supported_mem_attributes (struct kvm *kvm) { +if (kvm_arch_has_private_mem (kvm)) +return KVM_MEMORY_ATTRIBUTE_PRIVATE ;return 0 ; }@@ -4822,16 +4979,28 @@ static long kvm_vm_ioctl(struct file *filp, }case KVM_SET_USER_MEMORY_REGION : {struct kvm_user_mem_region mem; -unsigned long size =sizeof (struct kvm_userspace_memory_region ); +unsigned int flags_offset = offsetof(typeof (mem ), flags); +unsigned long size; +u32 flags;kvm_sanity_check_user_mem_region_alias (); +memset (&mem,0 ,sizeof (mem)); + r = -EFAULT ; +if (get_user (flags, (u32 __user *)(argp + flags_offset))) + goto out; + +if (flags &KVM_MEM_PRIVATE ) + size =sizeof (struct kvm_userspace_memory_region_ext ); +else + size =sizeof (struct kvm_userspace_memory_region ); +if (copy_from_user (&mem, argp, size)) goto out; r = -EINVAL ; -if (mem.flags &KVM_MEM_PRIVATE ) +if ((flags ^ mem.flags) &KVM_MEM_PRIVATE ) goto out; r =kvm_vm_ioctl_set_memory_region (kvm, &mem);
Sean Christopherson
maintainerRe: [PATCH v10 9/9] KVM: Enable and expose KVM_MEM_PRIVATE
Please trim your replies so that readers don't need to scan through a hundred or so lines of quotes just to confirm there's nothing there.
Quoted 21 lines · Ackerley Tng
Quoted 11 lines · Chao Peng
Register/unregister private memslot to fd-based memory backing store restrictedmem and implement the callbacks for
restrictedmem_notifier:
- invalidate_start()/invalidate_end() to zap the existing memory mappings in the KVM page table.
-error()to requestKVM_REQ_MEMORY_MCEand later exit to userspace withKVM_EXIT_SHUTDOWN.Expose
KVM_MEM_PRIVATEfor memslot andKVM_MEMORY_ATTRIBUTE_PRIVATEforKVM_GET_SUPPORTED_MEMORY_ATTRIBUTESto userspace but either are controlled bykvm_arch_has_private_mem()which should be rewritten by architecture code.Could we perhaps rename
KVM_MEM_PRIVATEtoKVM_MEM_PROTECTED, to be in line withKVM_X86_PROTECTED_VM?I feel that a memslot that has the
KVM_MEM_PRIVATEflag need not always be private; It can sometimes be providing memory that is shared and also accessible from the host.
Quoted 8 lines
KVM_MEMORY_ATTRIBUTE_PRIVATEis fine as-is because this flag is set when the guest memory is meant to be backed by private memory.
KVM_MEMORY_EXIT_FLAG_PRIVATEis also okay because the flag is used to indicate when the memory error is caused by a private access (as opposed to a shared access).
kvm_slot_can_be_private()could perhaps be renamedkvm_is_protected_slot()?
No to this suggestion. I agree that KVM_MEM_PRIVATE is a bad name, but kvm_is_protected_slot() is just as wrong. The _only_ thing that the flag controls is whether whether or not the memslot has an fd that is bound to restricted memory. The memslot itself is not protected in any way, and if the entire memslot is mapped shared, then the data backed by the memslot isn't protected either.
What about KVM_MEM_CAN_BE_PRIVATE? KVM_MEM_PRIVATIZABLE is more succinct, but AFAICT that's a made up word, and IMO is unnecessarily fancy.
Sean Christopherson
maintainerRe: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM
In reply to Chao PengQuoted 10 lines · Chao Peng
This patch series implements KVM guest private memory for confidential computing scenarios like Intel TDX[1]. If a TDX host accesses TDX-protected guest memory, machine check can happen which can further crash the running host system, this is terrible for multi-tenant configurations. The host accesses include those from KVM userspace like QEMU. This series addresses KVM userspace induced crash by introducing new mm and KVM interfaces so KVM userspace can still manage guest memory via a fd-based approach, but it can never access the guest memory content.
Quoted 5 lines
The patch series touches both core mm and KVM code. I appreciate Andrew/Hugh and Paolo/Sean can review and pick these patches. Any other reviews are always welcome.
- 01: mm change, target for mm tree
- 02-09: KVM change, target for KVM tree
A version with all of my feedback, plus reworked versions of Vishal's selftest, is available here:
[email protected]:sean-jc/linux.git x86/upm_base_support
It compiles and passes the selftest, but it's otherwise barely tested. There are a few todos (2 I think?) and many of the commits need changelogs, i.e. it's still a WIP.
As for next steps, can you (handwaving all of the TDX folks) take a look at what I pushed and see if there's anything horrifically broken, and that it still works for TDX?
Fuad (and pKVM folks) same ask for you with respect to pKVM. Absolutely no rush (and I mean that).
On my side, the two things on my mind are (a) tests and (b) downstream dependencies (SEV and TDX). For tests, I want to build a lists of tests that are required for merging so that the criteria for merging are clear, and so that if the list is large (haven't thought much yet), the work of writing and running tests can be distributed.
Regarding downstream dependencies, before this lands, I want to pull in all the TDX and SNP series and see how everything fits together. Specifically, I want to make sure that we don't end up with a uAPI that necessitates ugly code, and that we don't miss an opportunity to make things simpler. The patches in the SNP series to add "legacy" SEV support for UPM in particular made me slightly rethink some minor details. Nothing remotely major, but something that needs attention since it'll be uAPI.
I'm off Monday, so it'll be at least Tuesday before I make any more progress on my side.
Thanks!
Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM
Quoted 22 lines · Sean Christopherson
Quoted 10 lines · Chao Peng
This patch series implements KVM guest private memory for confidential computing scenarios like Intel TDX[1]. If a TDX host accesses TDX-protected guest memory, machine check can happen which can further crash the running host system, this is terrible for multi-tenant configurations. The host accesses include those from KVM userspace like QEMU. This series addresses KVM userspace induced crash by introducing new mm and KVM interfaces so KVM userspace can still manage guest memory via a fd-based approach, but it can never access the guest memory content.
Quoted 5 lines
The patch series touches both core mm and KVM code. I appreciate Andrew/Hugh and Paolo/Sean can review and pick these patches. Any other reviews are always welcome.
- 01: mm change, target for mm tree
- 02-09: KVM change, target for KVM treeA version with all of my feedback, plus reworked versions of Vishal's selftest, is available here:
[email protected]:sean-jc/linux.git x86/upm_base_support
Quoted 7 lines
It compiles and passes the selftest, but it's otherwise barely tested. There are a few todos (2 I think?) and many of the commits need changelogs, i.e. it's still a WIP.
As for next steps, can you (handwaving all of the TDX folks) take a look at what I pushed and see if there's anything horrifically broken, and that it still works for TDX?
Minor build fix:
arch/x86/kvm/mmu/mmu.c +2 -2
@@ -7211,8 +7211,8 @@ void kvm_arch_set_memory_attributes(struct kvm *kvm,int level;bool mixed; -lockdep_assert_held_write (kvm->mmu_lock); -lockdep_assert_held (kvm->slots_lock); +lockdep_assert_held_write (&kvm->mmu_lock); +lockdep_assert_held (&kvm->slots_lock);/* * KVM x86 currently only supports KVM_MEMORY_ATTRIBUTE_PRIVATE, skip
include/linux/kvm_host.h +1 -1
@@ -2304,7 +2304,7 @@ static inline void kvm_account_pgtable_pages(void *virt, int nr)#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES static inline unsigned long kvm_get_memory_attributes (struct kvm *kvm,gfn_t gfn) { -lockdep_assert_held (kvm->mmu_lock); +lockdep_assert_held (&kvm->mmu_lock);return xa_to_value (xa_load (&kvm->mem_attr_array, gfn)); }
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM
In reply to Sean ChristophersonQuoted 17 lines · Sean Christopherson
Quoted 10 lines · Chao Peng
This patch series implements KVM guest private memory for confidential computing scenarios like Intel TDX[1]. If a TDX host accesses TDX-protected guest memory, machine check can happen which can further crash the running host system, this is terrible for multi-tenant configurations. The host accesses include those from KVM userspace like QEMU. This series addresses KVM userspace induced crash by introducing new mm and KVM interfaces so KVM userspace can still manage guest memory via a fd-based approach, but it can never access the guest memory content.
Quoted 5 lines
The patch series touches both core mm and KVM code. I appreciate Andrew/Hugh and Paolo/Sean can review and pick these patches. Any other reviews are always welcome.
- 01: mm change, target for mm tree
- 02-09: KVM change, target for KVM tree
Quoted 8 lines
A version with all of my feedback, plus reworked versions of Vishal's selftest, is available here:
[email protected]:sean-jc/linux.git x86/upm_base_support
It compiles and passes the selftest, but it's otherwise barely tested. There are a few todos (2 I think?) and many of the commits need changelogs, i.e. it's still a WIP.
Thanks very much for doing this. Almost all of your comments are well received, except for two cases that need more discussions which have replied individually.
Quoted 4 lines
As for next steps, can you (handwaving all of the TDX folks) take a look at what I pushed and see if there's anything horrifically broken, and that it still works for TDX?
I have integrated this into my local TDX repo, with some changes (as I replied individually), the new code basically still works with TDX.
I have also asked other TDX folks to take a look.
Quoted 17 lines
Fuad (and pKVM folks) same ask for you with respect to pKVM. Absolutely no rush (and I mean that).
On my side, the two things on my mind are (a) tests and (b) downstream dependencies (SEV and TDX). For tests, I want to build a lists of tests that are required for merging so that the criteria for merging are clear, and so that if the list is large (haven't thought much yet), the work of writing and running tests can be distributed.
Regarding downstream dependencies, before this lands, I want to pull in all the TDX and SNP series and see how everything fits together. Specifically, I want to make sure that we don't end up with a uAPI that necessitates ugly code, and that we don't miss an opportunity to make things simpler. The patches in the SNP series to add "legacy" SEV support for UPM in particular made me slightly rethink some minor details. Nothing remotely major, but something that needs attention since it'll be uAPI.
Quoted 2 lines
I'm off Monday, so it'll be at least Tuesday before I make any more progress on my side.
Appreciate your effort. As for the next steps, if you see something we can do parallel, feel free to let me know.
Thanks,
Chao
Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM
In reply to Sean ChristophersonHi Sean,
Quoted 27 lines · Sean Christopherson
Quoted 10 lines · Chao Peng
This patch series implements KVM guest private memory for confidential computing scenarios like Intel TDX[1]. If a TDX host accesses TDX-protected guest memory, machine check can happen which can further crash the running host system, this is terrible for multi-tenant configurations. The host accesses include those from KVM userspace like QEMU. This series addresses KVM userspace induced crash by introducing new mm and KVM interfaces so KVM userspace can still manage guest memory via a fd-based approach, but it can never access the guest memory content.
Quoted 5 lines
The patch series touches both core mm and KVM code. I appreciate Andrew/Hugh and Paolo/Sean can review and pick these patches. Any other reviews are always welcome.
- 01: mm change, target for mm tree
- 02-09: KVM change, target for KVM treeA version with all of my feedback, plus reworked versions of Vishal's selftest, is available here:
[email protected]:sean-jc/linux.git x86/upm_base_support
It compiles and passes the selftest, but it's otherwise barely tested. There are a few todos (2 I think?) and many of the commits need changelogs, i.e. it's still a WIP.
Quoted 6 lines
As for next steps, can you (handwaving all of the TDX folks) take a look at what I pushed and see if there's anything horrifically broken, and that it still works for TDX?
Fuad (and pKVM folks) same ask for you with respect to pKVM. Absolutely no rush (and I mean that).
Thanks for sharing this. I've had a look at the patches, and have ported them to work with pKVM. At a high level, the new interface seems fine and it works with the arm64/pKVM port. I have a couple of comments regarding some of the details, but they can wait until v11 is posted.
Cheers,
/fuad
Quoted 17 lines
On my side, the two things on my mind are (a) tests and (b) downstream dependencies (SEV and TDX). For tests, I want to build a lists of tests that are required for merging so that the criteria for merging are clear, and so that if the list is large (haven't thought much yet), the work of writing and running tests can be distributed.
Regarding downstream dependencies, before this lands, I want to pull in all the TDX and SNP series and see how everything fits together. Specifically, I want to make sure that we don't end up with a uAPI that necessitates ugly code, and that we don't miss an opportunity to make things simpler. The patches in the SNP series to add "legacy" SEV support for UPM in particular made me slightly rethink some minor details. Nothing remotely major, but something that needs attention since it'll be uAPI.
I'm off Monday, so it'll be at least Tuesday before I make any more progress on my side.
Thanks!
Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM
In reply to Sean ChristophersonQuoted 38 lines · Sean Christopherson
Quoted 10 lines · Chao Peng
This patch series implements KVM guest private memory for confidential computing scenarios like Intel TDX[1]. If a TDX host accesses TDX-protected guest memory, machine check can happen which can further crash the running host system, this is terrible for multi-tenant configurations. The host accesses include those from KVM userspace like QEMU. This series addresses KVM userspace induced crash by introducing new mm and KVM interfaces so KVM userspace can still manage guest memory via a fd-based approach, but it can never access the guest memory content.
Quoted 5 lines
The patch series touches both core mm and KVM code. I appreciate Andrew/Hugh and Paolo/Sean can review and pick these patches. Any other reviews are always welcome.
- 01: mm change, target for mm tree
- 02-09: KVM change, target for KVM treeA version with all of my feedback, plus reworked versions of Vishal's selftest, is available here:
[email protected]:sean-jc/linux.git x86/upm_base_support
It compiles and passes the selftest, but it's otherwise barely tested. There are a few todos (2 I think?) and many of the commits need changelogs, i.e. it's still a WIP.
As for next steps, can you (handwaving all of the TDX folks) take a look at what I pushed and see if there's anything horrifically broken, and that it still works for TDX?
Fuad (and pKVM folks) same ask for you with respect to pKVM. Absolutely no rush (and I mean that).
On my side, the two things on my mind are (a) tests and (b) downstream dependencies (SEV and TDX). For tests, I want to build a lists of tests that are required for merging so that the criteria for merging are clear, and so that if the list is large (haven't thought much yet), the work of writing and running tests can be distributed.
Quoted 7 lines
Regarding downstream dependencies, before this lands, I want to pull in all the TDX and SNP series and see how everything fits together. Specifically, I want to make sure that we don't end up with a uAPI that necessitates ugly code, and that we don't miss an opportunity to make things simpler. The patches in the SNP series to add "legacy" SEV support for UPM in particular made me slightly rethink some minor details. Nothing remotely major, but something that needs attention since it'll be uAPI.
Although I'm still debuging with TDX KVM, I needed the following. kvm_faultin_pfn() is called without mmu_lock held. the race to change private/shared is handled by mmu_seq. Maybe dedicated function only for kvm_faultin_pfn().
include/linux/kvm_host.h +1 -1
@@ -2322,7 +2322,7 @@ static inline void kvm_account_pgtable_pages(void *virt, int nr)#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES static inline unsigned long kvm_get_memory_attributes (struct kvm *kvm,gfn_t gfn) { -lockdep_assert_held (&kvm->mmu_lock); +// lockdep_assert_held(&kvm->mmu_lock); return xa_to_value (xa_load (&kvm->mem_attr_array, gfn)); }
Signature · 1 line
Isaku Yamahata <[email protected]>
Sean Christopherson
maintainerRe: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM
Quoted 49 lines · Isaku Yamahata
Quoted 38 lines · Sean Christopherson
Quoted 10 lines · Chao Peng
This patch series implements KVM guest private memory for confidential computing scenarios like Intel TDX[1]. If a TDX host accesses TDX-protected guest memory, machine check can happen which can further crash the running host system, this is terrible for multi-tenant configurations. The host accesses include those from KVM userspace like QEMU. This series addresses KVM userspace induced crash by introducing new mm and KVM interfaces so KVM userspace can still manage guest memory via a fd-based approach, but it can never access the guest memory content.
Quoted 5 lines
The patch series touches both core mm and KVM code. I appreciate Andrew/Hugh and Paolo/Sean can review and pick these patches. Any other reviews are always welcome.
- 01: mm change, target for mm tree
- 02-09: KVM change, target for KVM treeA version with all of my feedback, plus reworked versions of Vishal's selftest, is available here:
[email protected]:sean-jc/linux.git x86/upm_base_support
It compiles and passes the selftest, but it's otherwise barely tested. There are a few todos (2 I think?) and many of the commits need changelogs, i.e. it's still a WIP.
As for next steps, can you (handwaving all of the TDX folks) take a look at what I pushed and see if there's anything horrifically broken, and that it still works for TDX?
Fuad (and pKVM folks) same ask for you with respect to pKVM. Absolutely no rush (and I mean that).
On my side, the two things on my mind are (a) tests and (b) downstream dependencies (SEV and TDX). For tests, I want to build a lists of tests that are required for merging so that the criteria for merging are clear, and so that if the list is large (haven't thought much yet), the work of writing and running tests can be distributed.
Quoted 7 lines
Regarding downstream dependencies, before this lands, I want to pull in all the TDX and SNP series and see how everything fits together. Specifically, I want to make sure that we don't end up with a uAPI that necessitates ugly code, and that we don't miss an opportunity to make things simpler. The patches in the SNP series to add "legacy" SEV support for UPM in particular made me slightly rethink some minor details. Nothing remotely major, but something that needs attention since it'll be uAPI.
Quoted 4 lines
Although I'm still debuging with TDX KVM, I needed the following.
kvm_faultin_pfn()is called withoutmmu_lockheld. the race to change private/shared is handled bymmu_seq. Maybe dedicated function only forkvm_faultin_pfn().
Gah, you're not on the other thread where this was discussed[*]. Simply deleting the lockdep assertion is safe, for guest types that rely on the attributes to define shared vs. private, KVM rechecks the attributes under the protection of mmu_seq.
I'll get a fixed version pushed out today.
Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM
Quoted 55 lines · Sean Christopherson
Quoted 49 lines · Isaku Yamahata
Quoted 38 lines · Sean Christopherson
Quoted 10 lines · Chao Peng
This patch series implements KVM guest private memory for confidential computing scenarios like Intel TDX[1]. If a TDX host accesses TDX-protected guest memory, machine check can happen which can further crash the running host system, this is terrible for multi-tenant configurations. The host accesses include those from KVM userspace like QEMU. This series addresses KVM userspace induced crash by introducing new mm and KVM interfaces so KVM userspace can still manage guest memory via a fd-based approach, but it can never access the guest memory content.
Quoted 5 lines
The patch series touches both core mm and KVM code. I appreciate Andrew/Hugh and Paolo/Sean can review and pick these patches. Any other reviews are always welcome.
- 01: mm change, target for mm tree
- 02-09: KVM change, target for KVM treeA version with all of my feedback, plus reworked versions of Vishal's selftest, is available here:
[email protected]:sean-jc/linux.git x86/upm_base_support
It compiles and passes the selftest, but it's otherwise barely tested. There are a few todos (2 I think?) and many of the commits need changelogs, i.e. it's still a WIP.
As for next steps, can you (handwaving all of the TDX folks) take a look at what I pushed and see if there's anything horrifically broken, and that it still works for TDX?
Fuad (and pKVM folks) same ask for you with respect to pKVM. Absolutely no rush (and I mean that).
On my side, the two things on my mind are (a) tests and (b) downstream dependencies (SEV and TDX). For tests, I want to build a lists of tests that are required for merging so that the criteria for merging are clear, and so that if the list is large (haven't thought much yet), the work of writing and running tests can be distributed.
Quoted 7 lines
Regarding downstream dependencies, before this lands, I want to pull in all the TDX and SNP series and see how everything fits together. Specifically, I want to make sure that we don't end up with a uAPI that necessitates ugly code, and that we don't miss an opportunity to make things simpler. The patches in the SNP series to add "legacy" SEV support for UPM in particular made me slightly rethink some minor details. Nothing remotely major, but something that needs attention since it'll be uAPI.
Quoted 4 lines
Although I'm still debuging with TDX KVM, I needed the following.
kvm_faultin_pfn()is called withoutmmu_lockheld. the race to change private/shared is handled bymmu_seq. Maybe dedicated function only forkvm_faultin_pfn().
Quoted 8 lines
Gah, you're not on the other thread where this was discussed[*]. Simply deleting the lockdep assertion is safe, for guest types that rely on the attributes to define shared vs. private, KVM rechecks the attributes under the protection of
mmu_seq.I'll get a fixed version pushed out today.
Now I have tdx kvm working. I've uploaded at the followings. It's rebased to v6.2-rc3.
[email protected]:yamahata/linux.git tdx/upm [email protected]:yamahata/qemu.git tdx/upm
kvm_mmu_do_page_fault() needs the following change. kvm_mem_is_private() queries mem_attr_array. kvm_faultin_pfn() also uses kvm_mem_is_private(). So the shared-private check in kvm_faultin_pfn() doesn't make sense. This change would belong to TDX KVM patches, though.
arch/x86/kvm/mmu/mmu_internal.h +1 -1
@@ -430,7 +430,7 @@ static inline int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, .max_level = vcpu->kvm->arch.tdp_max_page_level, .req_level = PG_LEVEL_4K, .goal_level = PG_LEVEL_4K, - .is_private = kvm_mem_is_private(vcpu->kvm, cr2_or_gpa >> PAGE_SHIFT), + .is_private = kvm_is_private_gpa(vcpu->kvm, cr2_or_gpa), };int r;
Signature · 1 line
Isaku Yamahata <[email protected]>
Sean Christopherson
maintainerRe: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM
Quoted 63 lines · Isaku Yamahata
Quoted 60 lines · Sean Christopherson
Quoted 49 lines · Isaku Yamahata
Quoted 38 lines · Sean Christopherson
Quoted 10 lines · Chao Peng
This patch series implements KVM guest private memory for confidential computing scenarios like Intel TDX[1]. If a TDX host accesses TDX-protected guest memory, machine check can happen which can further crash the running host system, this is terrible for multi-tenant configurations. The host accesses include those from KVM userspace like QEMU. This series addresses KVM userspace induced crash by introducing new mm and KVM interfaces so KVM userspace can still manage guest memory via a fd-based approach, but it can never access the guest memory content.
Quoted 5 lines
The patch series touches both core mm and KVM code. I appreciate Andrew/Hugh and Paolo/Sean can review and pick these patches. Any other reviews are always welcome.
- 01: mm change, target for mm tree
- 02-09: KVM change, target for KVM treeA version with all of my feedback, plus reworked versions of Vishal's selftest, is available here:
[email protected]:sean-jc/linux.git x86/upm_base_support
It compiles and passes the selftest, but it's otherwise barely tested. There are a few todos (2 I think?) and many of the commits need changelogs, i.e. it's still a WIP.
As for next steps, can you (handwaving all of the TDX folks) take a look at what I pushed and see if there's anything horrifically broken, and that it still works for TDX?
Fuad (and pKVM folks) same ask for you with respect to pKVM. Absolutely no rush (and I mean that).
On my side, the two things on my mind are (a) tests and (b) downstream dependencies (SEV and TDX). For tests, I want to build a lists of tests that are required for merging so that the criteria for merging are clear, and so that if the list is large (haven't thought much yet), the work of writing and running tests can be distributed.
Quoted 7 lines
Regarding downstream dependencies, before this lands, I want to pull in all the TDX and SNP series and see how everything fits together. Specifically, I want to make sure that we don't end up with a uAPI that necessitates ugly code, and that we don't miss an opportunity to make things simpler. The patches in the SNP series to add "legacy" SEV support for UPM in particular made me slightly rethink some minor details. Nothing remotely major, but something that needs attention since it'll be uAPI.
Quoted 4 lines
Although I'm still debuging with TDX KVM, I needed the following.
kvm_faultin_pfn()is called withoutmmu_lockheld. the race to change private/shared is handled bymmu_seq. Maybe dedicated function only forkvm_faultin_pfn().Gah, you're not on the other thread where this was discussed[*]. Simply deleting the lockdep assertion is safe, for guest types that rely on the attributes to define shared vs. private, KVM rechecks the attributes under the protection of
mmu_seq.
Quoted 8 lines
Quoted 3 lines
I'll get a fixed version pushed out today.
Now I have tdx kvm working. I've uploaded at the followings. It's rebased to v6.2-rc3.
[email protected]:yamahata/linux.git tdx/upm [email protected]:yamahata/qemu.git tdx/upm
And I finally got a working, building version updated and pushed out (again to):
[email protected]:sean-jc/linux.git x86/upm_base_support
Took longer than expected to get the memslot restrictions sussed out. I'm done working on the code for now, my plan is to come back to it+TDX+SNP in 2-3 weeks to resolves any remaining todos (that no one else tackles) and to do the whole "merge the world" excersise.
Quoted 4 lines
kvm_mmu_do_page_fault()needs the following change.kvm_mem_is_private()queriesmem_attr_array.kvm_faultin_pfn()also useskvm_mem_is_private(). So the shared-private check inkvm_faultin_pfn()doesn't make sense. This change would belong to TDX KVM patches, though.
Yeah, SNP needs similar treatment. Sorting that out is high up on the todo list.
Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM
Quoted 68 lines · Sean Christopherson
Quoted 67 lines · Isaku Yamahata
Quoted 63 lines · Sean Christopherson
Quoted 49 lines · Isaku Yamahata
Quoted 38 lines · Sean Christopherson
Quoted 10 lines · Chao Peng
This patch series implements KVM guest private memory for confidential computing scenarios like Intel TDX[1]. If a TDX host accesses TDX-protected guest memory, machine check can happen which can further crash the running host system, this is terrible for multi-tenant configurations. The host accesses include those from KVM userspace like QEMU. This series addresses KVM userspace induced crash by introducing new mm and KVM interfaces so KVM userspace can still manage guest memory via a fd-based approach, but it can never access the guest memory content.
Quoted 5 lines
The patch series touches both core mm and KVM code. I appreciate Andrew/Hugh and Paolo/Sean can review and pick these patches. Any other reviews are always welcome.
- 01: mm change, target for mm tree
- 02-09: KVM change, target for KVM treeA version with all of my feedback, plus reworked versions of Vishal's selftest, is available here:
[email protected]:sean-jc/linux.git x86/upm_base_support
It compiles and passes the selftest, but it's otherwise barely tested. There are a few todos (2 I think?) and many of the commits need changelogs, i.e. it's still a WIP.
As for next steps, can you (handwaving all of the TDX folks) take a look at what I pushed and see if there's anything horrifically broken, and that it still works for TDX?
Fuad (and pKVM folks) same ask for you with respect to pKVM. Absolutely no rush (and I mean that).
On my side, the two things on my mind are (a) tests and (b) downstream dependencies (SEV and TDX). For tests, I want to build a lists of tests that are required for merging so that the criteria for merging are clear, and so that if the list is large (haven't thought much yet), the work of writing and running tests can be distributed.
Quoted 7 lines
Regarding downstream dependencies, before this lands, I want to pull in all the TDX and SNP series and see how everything fits together. Specifically, I want to make sure that we don't end up with a uAPI that necessitates ugly code, and that we don't miss an opportunity to make things simpler. The patches in the SNP series to add "legacy" SEV support for UPM in particular made me slightly rethink some minor details. Nothing remotely major, but something that needs attention since it'll be uAPI.
Quoted 4 lines
Although I'm still debuging with TDX KVM, I needed the following.
kvm_faultin_pfn()is called withoutmmu_lockheld. the race to change private/shared is handled bymmu_seq. Maybe dedicated function only forkvm_faultin_pfn().Gah, you're not on the other thread where this was discussed[*]. Simply deleting the lockdep assertion is safe, for guest types that rely on the attributes to define shared vs. private, KVM rechecks the attributes under the protection of
mmu_seq.I'll get a fixed version pushed out today.
Quoted 9 lines
Quoted 4 lines
Now I have tdx kvm working. I've uploaded at the followings. It's rebased to v6.2-rc3.
[email protected]:yamahata/linux.git tdx/upm [email protected]:yamahata/qemu.git tdx/upmAnd I finally got a working, building version updated and pushed out (again to):
[email protected]:sean-jc/linux.git x86/upm_base_support
Ok, I rebased TDX part to the updated branch.
[email protected]:yamahata/linux.git tdx/upm
[email protected]:yamahata/qemu.git tdx/upm
Now it's v6.2-rc7 based.
qemu needs more patches to avoid registering memory slot for SMM.
Signature · 1 line
Isaku Yamahata <[email protected]>
Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM
In reply to Sean ChristophersonQuoted 82 lines · Sean Christopherson
Quoted 63 lines · Isaku Yamahata
Quoted 60 lines · Sean Christopherson
Quoted 49 lines · Isaku Yamahata
Quoted 38 lines · Sean Christopherson
Quoted 10 lines · Chao Peng
This patch series implements KVM guest private memory for confidential computing scenarios like Intel TDX[1]. If a TDX host accesses TDX-protected guest memory, machine check can happen which can further crash the running host system, this is terrible for multi-tenant configurations. The host accesses include those from KVM userspace like QEMU. This series addresses KVM userspace induced crash by introducing new mm and KVM interfaces so KVM userspace can still manage guest memory via a fd-based approach, but it can never access the guest memory content.
Quoted 5 lines
The patch series touches both core mm and KVM code. I appreciate Andrew/Hugh and Paolo/Sean can review and pick these patches. Any other reviews are always welcome.
- 01: mm change, target for mm tree
- 02-09: KVM change, target for KVM treeA version with all of my feedback, plus reworked versions of Vishal's selftest, is available here:
[email protected]:sean-jc/linux.git x86/upm_base_support
It compiles and passes the selftest, but it's otherwise barely tested. There are a few todos (2 I think?) and many of the commits need changelogs, i.e. it's still a WIP.
As for next steps, can you (handwaving all of the TDX folks) take a look at what I pushed and see if there's anything horrifically broken, and that it still works for TDX?
Fuad (and pKVM folks) same ask for you with respect to pKVM. Absolutely no rush (and I mean that).
On my side, the two things on my mind are (a) tests and (b) downstream dependencies (SEV and TDX). For tests, I want to build a lists of tests that are required for merging so that the criteria for merging are clear, and so that if the list is large (haven't thought much yet), the work of writing and running tests can be distributed.
Quoted 7 lines
Regarding downstream dependencies, before this lands, I want to pull in all the TDX and SNP series and see how everything fits together. Specifically, I want to make sure that we don't end up with a uAPI that necessitates ugly code, and that we don't miss an opportunity to make things simpler. The patches in the SNP series to add "legacy" SEV support for UPM in particular made me slightly rethink some minor details. Nothing remotely major, but something that needs attention since it'll be uAPI.
Quoted 4 lines
Although I'm still debuging with TDX KVM, I needed the following.
kvm_faultin_pfn()is called withoutmmu_lockheld. the race to change private/shared is handled bymmu_seq. Maybe dedicated function only forkvm_faultin_pfn().Gah, you're not on the other thread where this was discussed[*]. Simply deleting the lockdep assertion is safe, for guest types that rely on the attributes to define shared vs. private, KVM rechecks the attributes under the protection of
mmu_seq.Quoted 8 lines
Quoted 3 lines
I'll get a fixed version pushed out today.
Now I have tdx kvm working. I've uploaded at the followings. It's rebased to v6.2-rc3.
[email protected]:yamahata/linux.git tdx/upm [email protected]:yamahata/qemu.git tdx/upmAnd I finally got a working, building version updated and pushed out (again to):
[email protected]:sean-jc/linux.git x86/upm_base_support
Took longer than expected to get the memslot restrictions sussed out. I'm done working on the code for now, my plan is to come back to it+TDX+SNP in 2-3 weeks to resolves any remaining todos (that no one else tackles) and to do the whole "merge the world" excersise.
Quoted 6 lines
Quoted 4 lines
kvm_mmu_do_page_fault()needs the following change.kvm_mem_is_private()queriesmem_attr_array.kvm_faultin_pfn()also useskvm_mem_is_private(). So the shared-private check inkvm_faultin_pfn()doesn't make sense. This change would belong to TDX KVM patches, though.Yeah, SNP needs similar treatment. Sorting that out is high up on the todo list.
Hi Sean,
We've rebased the SEV+SNP support onto your updated UPM base support tree and things seem to be working okay, but we needed some fixups on top of the base support get things working, along with 1 workaround for an issue that hasn't been root-caused yet:
https://github.com/mdroth/linux/commits/upmv10b-host-snp-v8-wip
*stash (upm_base_support): mm: restrictedmem: Kirill's pinning implementation *workaround (use_base_support): mm: restrictedmem: loosen exclusivity check *fixup (upm_base_support): KVM: use inclusive ranges for restrictedmem binding/unbinding *fixup (upm_base_support): mm: restrictedmem: use inclusive ranges for issuing invalidations *fixup (upm_base_support): KVM: fix restrictedmem GFN range calculations *fixup (upm_base_support): KVM: selftests: CoCo compilation fixes
We plan to post an updated RFC for v8 soon, but also wanted to share the staging tree in case you end up looking at the UPM integration aspects before then.
-Mike
Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM
Quoted 11 lines
Hi Sean,
We've rebased the SEV+SNP support onto your updated UPM base support tree and things seem to be working okay, but we needed some fixups on top of the base support get things working, along with 1 workaround for an issue that hasn't been root-caused yet:
https://github.com/mdroth/linux/commits/upmv10b-host-snp-v8-wip
*stash (
upm_base_support): mm: restrictedmem: Kirill's pinning implementation *workaround (use_base_support): mm: restrictedmem: loosen exclusivity check
What I'm seeing is Slot#3 gets added first and then deleted. When it's gets added, Slot#0 already has the same range bound to restrictedmem so trigger the exclusive check. This check is exactly the current code for.
Quoted 2 lines
*fixup (
upm_base_support): KVM: use inclusive ranges for restrictedmem binding/unbinding *fixup (upm_base_support): mm: restrictedmem: use inclusive ranges for issuing invalidations
As many kernel APIs treat 'end' as exclusive, I would rather keep using exclusive 'end' for these APIs(restrictedmem_bind/restrictedmem_unbind and notifier callbacks) but fix it internally in the restrictedmem. E.g. all the places where xarray API needs a 'last'/'max' we use 'end - 1'. See below for the change.
Quoted 1 line
*fixup (
upm_base_support): KVM: fix restrictedmem GFN range calculations
Subtracting slot->restrictedmem.index for start/end in restrictedmem_get_gfn_range() is the correct fix.
Quoted 7 lines
*fixup (
upm_base_support): KVM: selftests: CoCo compilation fixesWe plan to post an updated RFC for v8 soon, but also wanted to share the staging tree in case you end up looking at the UPM integration aspects before then.
-Mike
This is the restrictedmem fix to solve 'end' being stored and checked in xarray:
mm/restrictedmem.c +7 -6
@@ -46,12 +46,12 @@ static long restrictedmem_punch_hole(struct restrictedmem *rm, int mode,*/ down_read (&rm->lock); -xa_for_each_range (&rm->bindings, index, notifier, start, end) +xa_for_each_range (&rm->bindings, index, notifier, start, end -1 ) notifier->ops->invalidate_start (notifier, start, end); ret = memfd->f_op->fallocate (memfd, mode, offset, len); -xa_for_each_range (&rm->bindings, index, notifier, start, end) +xa_for_each_range (&rm->bindings, index, notifier, start, end -1 ) notifier->ops->invalidate_end (notifier, start, end);up_read (&rm->lock);
@@ -224,7 +224,7 @@ static int restricted_error_remove_page(struct address_space *mapping, }spin_unlock (&inode->i_lock); -xa_for_each_range (&rm->bindings, index, notifier, start, end) +xa_for_each_range (&rm->bindings, index, notifier, start, end -1 ) notifier->ops->error (notifier, start, end);break ; }
@@ -301,11 +301,12 @@ int restrictedmem_bind(struct file *file, pgoff_t start, pgoff_t end,if (exclusive != rm->exclusive) goto out_unlock; -if (exclusive &&xa_find (&rm->bindings, &start, end,XA_PRESENT )) +if (exclusive && +xa_find (&rm->bindings, &start, end -1 ,XA_PRESENT )) goto out_unlock; } -xa_store_range (&rm->bindings, start, end, notifier,GFP_KERNEL ); +xa_store_range (&rm->bindings, start, end -1 , notifier,GFP_KERNEL ); rm->exclusive = exclusive; ret =0 ; out_unlock:
@@ -320,7 +321,7 @@ void restrictedmem_unbind(struct file *file, pgoff_t start, pgoff_t end,struct restrictedmem *rm = file->f_mapping->private_data;down_write (&rm->lock); -xa_store_range (&rm->bindings, start, end,NULL ,GFP_KERNEL ); +xa_store_range (&rm->bindings, start, end -1 ,NULL ,GFP_KERNEL );synchronize_rcu ();up_write (&rm->lock); }
Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM
Quoted 7 lines · Chao Peng
Quoted 7 lines
Hi Sean,
We've rebased the SEV+SNP support onto your updated UPM base support tree and things seem to be working okay, but we needed some fixups on top of the base support get things working, along with 1 workaround for an issue that hasn't been root-caused yet:
Quoted 8 lines
Quoted 4 lines
https://github.com/mdroth/linux/commits/upmv10b-host-snp-v8-wip
*stash (
upm_base_support): mm: restrictedmem: Kirill's pinning implementation *workaround (use_base_support): mm: restrictedmem: loosen exclusivity checkWhat I'm seeing is Slot#3 gets added first and then deleted. When it's gets added, Slot#0 already has the same range bound to restrictedmem so trigger the exclusive check. This check is exactly the current code for.
With the following change in QEMU, we no longer trigger this check:
diff --git a/hw/pci-host/q35.c b/hw/pci-host/q35.c
index 20da121374..849b5de469 100644
--- a/hw/pci-host/q35.c
+++ b/hw/pci-host/q35.c
@@ -588,9 +588,9 @@ static void mch_realize(PCIDevice *d, Error **errp)
memory_region_init_alias(&mch->open_high_smram, OBJECT(mch), "smram-open-high",
mch->ram_memory, MCH_HOST_BRIDGE_SMRAM_C_BASE,
MCH_HOST_BRIDGE_SMRAM_C_SIZE);
+ memory_region_set_enabled(&mch->open_high_smram, false);
memory_region_add_subregion_overlap(mch->system_memory, 0xfeda0000,
&mch->open_high_smram, 1); - memory_region_set_enabled(&mch->open_high_smram, false);
I'm not sure if QEMU is actually doing something wrong here though or if this check is putting tighter restrictions on userspace than what was expected before. Will look into it more.
Quoted 9 lines
Quoted 3 lines
*fixup (
upm_base_support): KVM: use inclusive ranges for restrictedmem binding/unbinding *fixup (upm_base_support): mm: restrictedmem: use inclusive ranges for issuing invalidationsAs many kernel APIs treat 'end' as exclusive, I would rather keep using exclusive 'end' for these
APIs(restrictedmem_bind/restrictedmem_unbind and notifier callbacks)but fix it internally in the restrictedmem. E.g. all the places where xarray API needs a 'last'/'max' we use 'end - 1'. See below for the change.
Yes I did feel like I was fighting the kernel a bit on that; your suggestion seems like it would be a better fit.
Quoted 8 lines
Quoted 2 lines
*fixup (
upm_base_support): KVM: fix restrictedmem GFN range calculationsSubtracting
slot->restrictedmem.indexfor start/end inrestrictedmem_get_gfn_range()is the correct fix.Quoted 2 lines
*fixup (
upm_base_support): KVM: selftests: CoCo compilation fixes
Quoted 7 lines
Quoted 5 lines
We plan to post an updated RFC for v8 soon, but also wanted to share the staging tree in case you end up looking at the UPM integration aspects before then.
-Mike
This is the restrictedmem fix to solve 'end' being stored and checked in xarray:
Looks good.
Thanks!
-Mike
Quoted 51 lines
mm/restrictedmem.c +7
-6@@ -46,12 +46,12 @@ static long restrictedmem_punch_hole(struct restrictedmem *rm, int mode,*/ down_read (&rm->lock); -xa_for_each_range (&rm->bindings, index, notifier, start, end) +xa_for_each_range (&rm->bindings, index, notifier, start, end -1 ) notifier->ops->invalidate_start (notifier, start, end); ret = memfd->f_op->fallocate (memfd, mode, offset, len); -xa_for_each_range (&rm->bindings, index, notifier, start, end) +xa_for_each_range (&rm->bindings, index, notifier, start, end -1 ) notifier->ops->invalidate_end (notifier, start, end);up_read (&rm->lock);@@ -224,7 +224,7 @@ static int restricted_error_remove_page(struct address_space *mapping, }spin_unlock (&inode->i_lock); -xa_for_each_range (&rm->bindings, index, notifier, start, end) +xa_for_each_range (&rm->bindings, index, notifier, start, end -1 ) notifier->ops->error (notifier, start, end);break ; }@@ -301,11 +301,12 @@ int restrictedmem_bind(struct file *file, pgoff_t start, pgoff_t end,if (exclusive != rm->exclusive) goto out_unlock; -if (exclusive &&xa_find (&rm->bindings, &start, end,XA_PRESENT )) +if (exclusive && +xa_find (&rm->bindings, &start, end -1 ,XA_PRESENT )) goto out_unlock; } -xa_store_range (&rm->bindings, start, end, notifier,GFP_KERNEL ); +xa_store_range (&rm->bindings, start, end -1 , notifier,GFP_KERNEL ); rm->exclusive = exclusive; ret =0 ; out_unlock:@@ -320,7 +321,7 @@ void restrictedmem_unbind(struct file *file, pgoff_t start, pgoff_t end,struct restrictedmem *rm = file->f_mapping->private_data;down_write (&rm->lock); -xa_store_range (&rm->bindings, start, end,NULL ,GFP_KERNEL ); +xa_store_range (&rm->bindings, start, end -1 ,NULL ,GFP_KERNEL );synchronize_rcu ();up_write (&rm->lock); }
Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM
Quoted 30 lines · Michael Roth
Quoted 7 lines · Chao Peng
Quoted 7 lines
Hi Sean,
We've rebased the SEV+SNP support onto your updated UPM base support tree and things seem to be working okay, but we needed some fixups on top of the base support get things working, along with 1 workaround for an issue that hasn't been root-caused yet:
Quoted 8 lines
Quoted 4 lines
https://github.com/mdroth/linux/commits/upmv10b-host-snp-v8-wip
*stash (
upm_base_support): mm: restrictedmem: Kirill's pinning implementation *workaround (use_base_support): mm: restrictedmem: loosen exclusivity checkWhat I'm seeing is Slot#3 gets added first and then deleted. When it's gets added, Slot#0 already has the same range bound to restrictedmem so trigger the exclusive check. This check is exactly the current code for.
With the following change in QEMU, we no longer trigger this check:
diff --git
a/hw/pci-host/q35.cb/hw/pci-host/q35.c
index20da121374..849b5de469100644
---a/hw/pci-host/q35.c
+++b/hw/pci-host/q35.c
@@ -588,9 +588,9 @@ static voidmch_realize(PCIDevice *d, Error **errp)memory_region_init_alias(&mch->open_high_smram, OBJECT(mch), "smram-open-high", mch->ram_memory, MCH_HOST_BRIDGE_SMRAM_C_BASE, MCH_HOST_BRIDGE_SMRAM_C_SIZE); + memory_region_set_enabled(&mch->open_high_smram, false); memory_region_add_subregion_overlap(mch->system_memory, 0xfeda0000, &mch->open_high_smram, 1);
Quoted 5 lines
-
memory_region_set_enabled(&mch->open_high_smram, false);I'm not sure if QEMU is actually doing something wrong here though or if this check is putting tighter restrictions on userspace than what was expected before. Will look into it more.
I don't think above QEMU change is upstream acceptable. It may break functionality for 'normal' VMs.
The UPM check does putting tighter restriction, the restriction is that you can't bind the same fd range to more than one memslot. For SMRAM in QEMU however, it violates this restriction. The right 'fix' is disabling SMM in QEMU for UPM usages rather than trying to work around it. There is more discussion in below link:
https://lore.kernel.org/all/[email protected]/
Chao
Quoted 87 lines
Quoted 10 lines
Quoted 4 lines
*fixup (
upm_base_support): KVM: use inclusive ranges for restrictedmem binding/unbinding *fixup (upm_base_support): mm: restrictedmem: use inclusive ranges for issuing invalidationsAs many kernel APIs treat 'end' as exclusive, I would rather keep using exclusive 'end' for these
APIs(restrictedmem_bind/restrictedmem_unbind and notifier callbacks)but fix it internally in the restrictedmem. E.g. all the places where xarray API needs a 'last'/'max' we use 'end - 1'. See below for the change.Yes I did feel like I was fighting the kernel a bit on that; your suggestion seems like it would be a better fit.
Quoted 8 lines
Quoted 2 lines
*fixup (
upm_base_support): KVM: fix restrictedmem GFN range calculationsSubtracting
slot->restrictedmem.indexfor start/end inrestrictedmem_get_gfn_range()is the correct fix.Quoted 2 lines
*fixup (
upm_base_support): KVM: selftests: CoCo compilation fixesQuoted 7 lines
Quoted 5 lines
We plan to post an updated RFC for v8 soon, but also wanted to share the staging tree in case you end up looking at the UPM integration aspects before then.
-Mike
This is the restrictedmem fix to solve 'end' being stored and checked in xarray:
Looks good.
Thanks!
-Mike
Quoted 51 lines
mm/restrictedmem.c +7
-6@@ -46,12 +46,12 @@ static long restrictedmem_punch_hole(struct restrictedmem *rm, int mode,*/ down_read (&rm->lock); -xa_for_each_range (&rm->bindings, index, notifier, start, end) +xa_for_each_range (&rm->bindings, index, notifier, start, end -1 ) notifier->ops->invalidate_start (notifier, start, end); ret = memfd->f_op->fallocate (memfd, mode, offset, len); -xa_for_each_range (&rm->bindings, index, notifier, start, end) +xa_for_each_range (&rm->bindings, index, notifier, start, end -1 ) notifier->ops->invalidate_end (notifier, start, end);up_read (&rm->lock);@@ -224,7 +224,7 @@ static int restricted_error_remove_page(struct address_space *mapping, }spin_unlock (&inode->i_lock); -xa_for_each_range (&rm->bindings, index, notifier, start, end) +xa_for_each_range (&rm->bindings, index, notifier, start, end -1 ) notifier->ops->error (notifier, start, end);break ; }@@ -301,11 +301,12 @@ int restrictedmem_bind(struct file *file, pgoff_t start, pgoff_t end,if (exclusive != rm->exclusive) goto out_unlock; -if (exclusive &&xa_find (&rm->bindings, &start, end,XA_PRESENT )) +if (exclusive && +xa_find (&rm->bindings, &start, end -1 ,XA_PRESENT )) goto out_unlock; } -xa_store_range (&rm->bindings, start, end, notifier,GFP_KERNEL ); +xa_store_range (&rm->bindings, start, end -1 , notifier,GFP_KERNEL ); rm->exclusive = exclusive; ret =0 ; out_unlock:@@ -320,7 +321,7 @@ void restrictedmem_unbind(struct file *file, pgoff_t start, pgoff_t end,struct restrictedmem *rm = file->f_mapping->private_data;down_write (&rm->lock); -xa_store_range (&rm->bindings, start, end,NULL ,GFP_KERNEL ); +xa_store_range (&rm->bindings, start, end -1 ,NULL ,GFP_KERNEL );synchronize_rcu ();up_write (&rm->lock); }
Sean Christopherson
maintainerRe: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM
In reply to Michael RothQuoted 4 lines · Michael Roth
Quoted 3 lines · Chao Peng
Quoted 2 lines
*fixup (
upm_base_support): KVM: use inclusive ranges for restrictedmem binding/unbinding *fixup (upm_base_support): mm: restrictedmem: use inclusive ranges for issuing invalidations
Quoted 8 lines
Quoted 5 lines
As many kernel APIs treat 'end' as exclusive, I would rather keep using exclusive 'end' for these
APIs(restrictedmem_bind/restrictedmem_unbind and notifier callbacks)but fix it internally in the restrictedmem. E.g. all the places where xarray API needs a 'last'/'max' we use 'end - 1'. See below for the change.Yes I did feel like I was fighting the kernel a bit on that; your suggestion seems like it would be a better fit.
Comically belated +1, XArray is the odd one here.
Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM
In reply to Sean ChristophersonQuoted 73 lines · Sean Christopherson
Quoted 63 lines · Isaku Yamahata
Quoted 60 lines · Sean Christopherson
Quoted 49 lines · Isaku Yamahata
Quoted 38 lines · Sean Christopherson
Quoted 10 lines · Chao Peng
This patch series implements KVM guest private memory for confidential computing scenarios like Intel TDX[1]. If a TDX host accesses TDX-protected guest memory, machine check can happen which can further crash the running host system, this is terrible for multi-tenant configurations. The host accesses include those from KVM userspace like QEMU. This series addresses KVM userspace induced crash by introducing new mm and KVM interfaces so KVM userspace can still manage guest memory via a fd-based approach, but it can never access the guest memory content.
Quoted 5 lines
The patch series touches both core mm and KVM code. I appreciate Andrew/Hugh and Paolo/Sean can review and pick these patches. Any other reviews are always welcome.
- 01: mm change, target for mm tree
- 02-09: KVM change, target for KVM treeA version with all of my feedback, plus reworked versions of Vishal's selftest, is available here:
[email protected]:sean-jc/linux.git x86/upm_base_support
It compiles and passes the selftest, but it's otherwise barely tested. There are a few todos (2 I think?) and many of the commits need changelogs, i.e. it's still a WIP.
As for next steps, can you (handwaving all of the TDX folks) take a look at what I pushed and see if there's anything horrifically broken, and that it still works for TDX?
Fuad (and pKVM folks) same ask for you with respect to pKVM. Absolutely no rush (and I mean that).
On my side, the two things on my mind are (a) tests and (b) downstream dependencies (SEV and TDX). For tests, I want to build a lists of tests that are required for merging so that the criteria for merging are clear, and so that if the list is large (haven't thought much yet), the work of writing and running tests can be distributed.
Quoted 7 lines
Regarding downstream dependencies, before this lands, I want to pull in all the TDX and SNP series and see how everything fits together. Specifically, I want to make sure that we don't end up with a uAPI that necessitates ugly code, and that we don't miss an opportunity to make things simpler. The patches in the SNP series to add "legacy" SEV support for UPM in particular made me slightly rethink some minor details. Nothing remotely major, but something that needs attention since it'll be uAPI.
Quoted 4 lines
Although I'm still debuging with TDX KVM, I needed the following.
kvm_faultin_pfn()is called withoutmmu_lockheld. the race to change private/shared is handled bymmu_seq. Maybe dedicated function only forkvm_faultin_pfn().Gah, you're not on the other thread where this was discussed[*]. Simply deleting the lockdep assertion is safe, for guest types that rely on the attributes to define shared vs. private, KVM rechecks the attributes under the protection of
mmu_seq.Quoted 8 lines
Quoted 3 lines
I'll get a fixed version pushed out today.
Now I have tdx kvm working. I've uploaded at the followings. It's rebased to v6.2-rc3.
[email protected]:yamahata/linux.git tdx/upm [email protected]:yamahata/qemu.git tdx/upm
Quoted 8 lines
And I finally got a working, building version updated and pushed out (again to):
[email protected]:sean-jc/linux.git x86/upm_base_support
Took longer than expected to get the memslot restrictions sussed out. I'm done working on the code for now, my plan is to come back to it+TDX+SNP in 2-3 weeks to resolves any remaining todos (that no one else tackles) and to do the whole "merge the world" excersise.
Hi Sean,
In case you started working on the code again, I have a branch [1] originally planned as v11 candidate which I believe I addressed all the discussions we had for v10 except the very latest one [2] and integrated all the newly added selftests from Ackerley and myself. The branch was based on your original upm_base_support and then rebased to your kvm-x86/mmu head. Feel free to take anything you think useful( most of them are trivial things but also some fixes for bugs).
[1] https://github.com/chao-p/linux/commits/privmem-v11.6
[2] https://lore.kernel.org/all/[email protected]/
Chao
Quoted 7 lines
Quoted 5 lines
kvm_mmu_do_page_fault()needs the following change.kvm_mem_is_private()queriesmem_attr_array.kvm_faultin_pfn()also useskvm_mem_is_private(). So the shared-private check inkvm_faultin_pfn()doesn't make sense. This change would belong to TDX KVM patches, though.Yeah, SNP needs similar treatment. Sorting that out is high up on the todo list.
Sean Christopherson
maintainerRe: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM
Quoted 7 lines · Chao Peng
In case you started working on the code again, I have a branch [1] originally planned as v11 candidate which I believe I addressed all the discussions we had for v10 except the very latest one [2] and integrated all the newly added selftests from Ackerley and myself. The branch was based on your original
upm_base_supportand then rebased to your kvm-x86/mmu head. Feel free to take anything you think useful( most of them are trivial things but also some fixes for bugs).
Nice! I am going to work on splicing together the various series this week, I'll make sure to grab your work.
Thanks much!
Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM
In reply to Sean ChristophersonQuoted 17 lines · Sean Christopherson
Quoted 10 lines · Chao Peng
This patch series implements KVM guest private memory for confidential computing scenarios like Intel TDX[1]. If a TDX host accesses TDX-protected guest memory, machine check can happen which can further crash the running host system, this is terrible for multi-tenant configurations. The host accesses include those from KVM userspace like QEMU. This series addresses KVM userspace induced crash by introducing new mm and KVM interfaces so KVM userspace can still manage guest memory via a fd-based approach, but it can never access the guest memory content.
Quoted 5 lines
The patch series touches both core mm and KVM code. I appreciate Andrew/Hugh and Paolo/Sean can review and pick these patches. Any other reviews are always welcome.
- 01: mm change, target for mm tree
- 02-09: KVM change, target for KVM tree
Quoted 9 lines
A version with all of my feedback, plus reworked versions of Vishal's selftest, is available here:
[email protected]:sean-jc/linux.git x86/upm_base_support
It compiles and passes the selftest, but it's otherwise barely tested. There are a few todos (2 I think?) and many of the commits need changelogs, i.e. it's still a WIP.
When running LTP (https://github.com/linux-test-project/ltp) on the v10 bits (and also with Sean's branch above) I encounter the following NULL pointer dereference with testcases/kernel/syscalls/madvise/madvise01 (100% reproducible).
It appears that in restrictedmem_error_page() inode->i_mapping->private_data is NULL in the list_for_each_entry_safe(inode, next, &sb->s_inodes, i_sb_list) but I don't know why.
[ 5365.177168] BUG: kernel NULL pointer dereference, address: 0000000000000028 [ 5365.178881] #PF: supervisor read access in kernel mode [ 5365.180006] #PF: error_code(0x0000) - not-present page [ 5365.181322] PGD 8000000109dad067 P4D 8000000109dad067 PUD 107707067 PMD 0 [ 5365.183474] Oops: 0000 [#1] PREEMPT SMP PTI [ 5365.184792] CPU: 0 PID: 22086 Comm: madvise01 Not tainted 6.1.0-1.el8.seanjcupm.x86_64 #1 [ 5365.186572] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.5.1 06/16/2021 [ 5365.188816] RIP: 0010:restrictedmem_error_page+0xc7/0x1b0 [ 5365.190081] Code: 99 00 48 8b 55 00 48 8b 02 48 8d 8a e8 fe ff ff 48 2d 18 01 00 00 48 39 d5 0f 84 8a 00 00 00 48 8b 51 30 48 8b 92 b8 00 00 00 <48> 8b 4a 28 4c 39 b1 d8 00 00 00 74 22 48 8b 88 18 01 00 00 48 8d [ 5365.193984] RSP: 0018:ffff9b7343c07d80 EFLAGS: 00010206 [ 5365.195142] RAX: ffff8e5b410cfc70 RBX: 0000000000000001 RCX: ffff8e5b4048e580 [ 5365.196888] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 [ 5365.198399] RBP: ffff8e5b410cfd88 R08: 0000000000000000 R09: 0000000000000000 [ 5365.200200] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 5365.201843] R13: ffff8e5b410cfd80 R14: ffff8e5b47cc7618 R15: ffffd49d44c05080 [ 5365.203472] FS: 00007fc96de9b5c0(0000) GS:ffff8e5deda00000(0000) knlGS:0000000000000000 [ 5365.205485] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5365.206791] CR2: 0000000000000028 CR3: 000000012047e002 CR4: 0000000000170ef0 [ 5365.208131] Call Trace: [ 5365.208752] <TASK> [ 5365.209229] me_pagecache_clean+0x58/0x100 [ 5365.210196] identify_page_state+0x84/0xd0 [ 5365.211180] memory_failure+0x231/0x8b0 [ 5365.212148] madvise_inject_error.cold+0x8d/0xa4 [ 5365.213317] do_madvise+0x363/0x3a0 [ 5365.214177] __x64_sys_madvise+0x2c/0x40 [ 5365.215159] do_syscall_64+0x3f/0xa0 [ 5365.216016] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 5365.217130] RIP: 0033:0x7fc96d8399ab [ 5365.217953] Code: 73 01 c3 48 8b 0d dd 54 38 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa b8 1c 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d ad 54 38 00 f7 d8 64 89 01 48 [ 5365.222323] RSP: 002b:00007fff62a99b18 EFLAGS: 00000206 ORIG_RAX: 000000000000001c [ 5365.224026] RAX: ffffffffffffffda RBX: 000000000041ce00 RCX: 00007fc96d8399ab [ 5365.225375] RDX: 0000000000000064 RSI: 000000000000a000 RDI: 00007fc96de8e000 [ 5365.226999] RBP: 00007fc96de9b540 R08: 0000000000000001 R09: 0000000000415c80 [ 5365.228641] R10: 0000000000000001 R11: 0000000000000206 R12: 0000000000000008 [ 5365.230074] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Regards,
Liam
Quoted 25 lines
As for next steps, can you (handwaving all of the TDX folks) take a look at what I pushed and see if there's anything horrifically broken, and that it still works for TDX?
Fuad (and pKVM folks) same ask for you with respect to pKVM. Absolutely no rush (and I mean that).
On my side, the two things on my mind are (a) tests and (b) downstream dependencies (SEV and TDX). For tests, I want to build a lists of tests that are required for merging so that the criteria for merging are clear, and so that if the list is large (haven't thought much yet), the work of writing and running tests can be distributed.
Regarding downstream dependencies, before this lands, I want to pull in all the TDX and SNP series and see how everything fits together. Specifically, I want to make sure that we don't end up with a uAPI that necessitates ugly code, and that we don't miss an opportunity to make things simpler. The patches in the SNP series to add "legacy" SEV support for UPM in particular made me slightly rethink some minor details. Nothing remotely major, but something that needs attention since it'll be uAPI.
I'm off Monday, so it'll be at least Tuesday before I make any more progress on my side.
Thanks!
Sean Christopherson
maintainerRe: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM
Quoted 33 lines · Liam Merwick
Quoted 17 lines · Sean Christopherson
Quoted 10 lines · Chao Peng
This patch series implements KVM guest private memory for confidential computing scenarios like Intel TDX[1]. If a TDX host accesses TDX-protected guest memory, machine check can happen which can further crash the running host system, this is terrible for multi-tenant configurations. The host accesses include those from KVM userspace like QEMU. This series addresses KVM userspace induced crash by introducing new mm and KVM interfaces so KVM userspace can still manage guest memory via a fd-based approach, but it can never access the guest memory content.
Quoted 5 lines
The patch series touches both core mm and KVM code. I appreciate Andrew/Hugh and Paolo/Sean can review and pick these patches. Any other reviews are always welcome.
- 01: mm change, target for mm tree
- 02-09: KVM change, target for KVM treeQuoted 9 lines
A version with all of my feedback, plus reworked versions of Vishal's selftest, is available here:
[email protected]:sean-jc/linux.git x86/upm_base_support
It compiles and passes the selftest, but it's otherwise barely tested. There are a few todos (2 I think?) and many of the commits need changelogs, i.e. it's still a WIP.
When running LTP (https://github.com/linux-test-project/ltp) on the v10 bits (and also with Sean's branch above) I encounter the following NULL pointer dereference with testcases/kernel/syscalls/madvise/madvise01 (100% reproducible).
Quoted 4 lines
It appears that in
restrictedmem_error_page()inode->i_mapping->private_datais NULL
in thelist_for_each_entry_safe(inode, next, &sb->s_inodes, i_sb_list)but I don't know why.
Kirill, can you take a look? Or pass the buck to someone who can? :-)
Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM
Quoted 34 lines · Sean Christopherson
Quoted 33 lines · Liam Merwick
Quoted 17 lines · Sean Christopherson
Quoted 10 lines · Chao Peng
This patch series implements KVM guest private memory for confidential computing scenarios like Intel TDX[1]. If a TDX host accesses TDX-protected guest memory, machine check can happen which can further crash the running host system, this is terrible for multi-tenant configurations. The host accesses include those from KVM userspace like QEMU. This series addresses KVM userspace induced crash by introducing new mm and KVM interfaces so KVM userspace can still manage guest memory via a fd-based approach, but it can never access the guest memory content.
Quoted 5 lines
The patch series touches both core mm and KVM code. I appreciate Andrew/Hugh and Paolo/Sean can review and pick these patches. Any other reviews are always welcome.
- 01: mm change, target for mm tree
- 02-09: KVM change, target for KVM treeQuoted 9 lines
A version with all of my feedback, plus reworked versions of Vishal's selftest, is available here:
[email protected]:sean-jc/linux.git x86/upm_base_support
It compiles and passes the selftest, but it's otherwise barely tested. There are a few todos (2 I think?) and many of the commits need changelogs, i.e. it's still a WIP.
When running LTP (https://github.com/linux-test-project/ltp) on the v10 bits (and also with Sean's branch above) I encounter the following NULL pointer dereference with testcases/kernel/syscalls/madvise/madvise01 (100% reproducible).
Quoted 6 lines
Quoted 4 lines
It appears that in
restrictedmem_error_page()inode->i_mapping->private_datais NULL
in thelist_for_each_entry_safe(inode, next, &sb->s_inodes, i_sb_list)but I don't know why.Kirill, can you take a look? Or pass the buck to someone who can? :-)
The patch below should help.
mm/restrictedmem.c +18 -3
@@ -307,14 +307,29 @@ void restrictedmem_error_page(struct page *page, struct address_space *mapping)spin_lock (&sb->s_inode_list_lock);list_for_each_entry_safe (inode, next, &sb->s_inodes, i_sb_list) { -struct restrictedmem *rm = inode->i_mapping->private_data;struct restrictedmem_notifier *notifier; -struct file *memfd = rm->memfd; +struct restrictedmem *rm;unsigned long index; +struct file *memfd; -if (memfd->f_mapping != mapping) +if (atomic_read (&inode->i_count))continue ; +spin_lock (&inode->i_lock); +if (inode->i_state & (I_NEW |I_FREEING |I_WILL_FREE )) { +spin_unlock (&inode->i_lock); +continue ; + } + + rm = inode->i_mapping->private_data; + memfd = rm->memfd; + +if (memfd->f_mapping != mapping) { +spin_unlock (&inode->i_lock); +continue ; + } +spin_unlock (&inode->i_lock); +xa_for_each_range (&rm->bindings, index, notifier, start, end) notifier->ops->error (notifier, start, end);break ;
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM
Quoted 4 lines · Kirill A. Shutemov
Quoted 3 lines · Sean Christopherson
Quoted 2 lines · Liam Merwick
Quoted 1 line · Sean Christopherson
On Fri, Dec 02, 2022, Chao Peng wrote:
...
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
When running LTP (https://github.com/linux-test-project/ltp) on the v10 bits (and also with Sean's branch above) I encounter the following NULL pointer dereference with testcases/kernel/syscalls/madvise/madvise01 (100% reproducible).
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
It appears that in
restrictedmem_error_page()inode->i_mapping->private_datais NULL
in thelist_for_each_entry_safe(inode, next, &sb->s_inodes, i_sb_list)but I don't know why.Kirill, can you take a look? Or pass the buck to someone who can? :-)
The patch below should help.
Thanks, this works for me.
Regards,
Liam
Quoted 38 lines
mm/restrictedmem.c +18
-3@@ -307,14 +307,29 @@ void restrictedmem_error_page(struct page *page, struct address_space *mapping)spin_lock (&sb->s_inode_list_lock);list_for_each_entry_safe (inode, next, &sb->s_inodes, i_sb_list) { -struct restrictedmem *rm = inode->i_mapping->private_data;struct restrictedmem_notifier *notifier; -struct file *memfd = rm->memfd; +struct restrictedmem *rm;unsigned long index; +struct file *memfd; -if (memfd->f_mapping != mapping) +if (atomic_read (&inode->i_count))continue ; +spin_lock (&inode->i_lock); +if (inode->i_state & (I_NEW |I_FREEING |I_WILL_FREE )) { +spin_unlock (&inode->i_lock); +continue ; + } + + rm = inode->i_mapping->private_data; + memfd = rm->memfd; + +if (memfd->f_mapping != mapping) { +spin_unlock (&inode->i_lock); +continue ; + } +spin_unlock (&inode->i_lock); +xa_for_each_range (&rm->bindings, index, notifier, start, end) notifier->ops->error (notifier, start, end);break ;
Sean Christopherson
maintainerRe: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM
In reply to Kirill A. ShutemovQuoted 51 lines · Kirill A. Shutemov
Quoted 34 lines · Sean Christopherson
Quoted 33 lines · Liam Merwick
Quoted 17 lines · Sean Christopherson
Quoted 10 lines · Chao Peng
This patch series implements KVM guest private memory for confidential computing scenarios like Intel TDX[1]. If a TDX host accesses TDX-protected guest memory, machine check can happen which can further crash the running host system, this is terrible for multi-tenant configurations. The host accesses include those from KVM userspace like QEMU. This series addresses KVM userspace induced crash by introducing new mm and KVM interfaces so KVM userspace can still manage guest memory via a fd-based approach, but it can never access the guest memory content.
Quoted 5 lines
The patch series touches both core mm and KVM code. I appreciate Andrew/Hugh and Paolo/Sean can review and pick these patches. Any other reviews are always welcome.
- 01: mm change, target for mm tree
- 02-09: KVM change, target for KVM treeQuoted 9 lines
A version with all of my feedback, plus reworked versions of Vishal's selftest, is available here:
[email protected]:sean-jc/linux.git x86/upm_base_support
It compiles and passes the selftest, but it's otherwise barely tested. There are a few todos (2 I think?) and many of the commits need changelogs, i.e. it's still a WIP.
When running LTP (https://github.com/linux-test-project/ltp) on the v10 bits (and also with Sean's branch above) I encounter the following NULL pointer dereference with testcases/kernel/syscalls/madvise/madvise01 (100% reproducible).
Quoted 6 lines
Quoted 4 lines
It appears that in
restrictedmem_error_page()inode->i_mapping->private_datais NULL in thelist_for_each_entry_safe(inode, next, &sb->s_inodes, i_sb_list)but I don't know why.Kirill, can you take a look? Or pass the buck to someone who can? :-)
The patch below should help.
mm/restrictedmem.c +0
-0@@ -307,14 +307,29 @@ void restrictedmem_error_page(struct page *page, struct address_space *mapping)spin_lock (&sb->s_inode_list_lock);
mm/restrictedmem.c void restrictedmem_error_page(struct page *page, struct address_space *mapping) · line 315
Quoted 10 lines
mm/restrictedmem.c +3
-3list_for_each_entry_safe (inode, next, &sb->s_inodes, i_sb_list) { -struct restrictedmem *rm = inode->i_mapping->private_data;struct restrictedmem_notifier *notifier; -struct file *memfd = rm->memfd; +struct restrictedmem *rm;unsigned long index; +struct file *memfd; -if (memfd->f_mapping != mapping) +if (atomic_read (&inode->i_count))
Kirill, should this be
if (!atomic_read(&inode->i_count)) continue;
i.e. skip unreferenced inodes, not skip referenced inodes?
Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM
Quoted 65 lines · Sean Christopherson
Quoted 51 lines · Kirill A. Shutemov
Quoted 34 lines · Sean Christopherson
Quoted 33 lines · Liam Merwick
Quoted 17 lines · Sean Christopherson
Quoted 10 lines · Chao Peng
This patch series implements KVM guest private memory for confidential computing scenarios like Intel TDX[1]. If a TDX host accesses TDX-protected guest memory, machine check can happen which can further crash the running host system, this is terrible for multi-tenant configurations. The host accesses include those from KVM userspace like QEMU. This series addresses KVM userspace induced crash by introducing new mm and KVM interfaces so KVM userspace can still manage guest memory via a fd-based approach, but it can never access the guest memory content.
Quoted 5 lines
The patch series touches both core mm and KVM code. I appreciate Andrew/Hugh and Paolo/Sean can review and pick these patches. Any other reviews are always welcome.
- 01: mm change, target for mm tree
- 02-09: KVM change, target for KVM treeQuoted 9 lines
A version with all of my feedback, plus reworked versions of Vishal's selftest, is available here:
[email protected]:sean-jc/linux.git x86/upm_base_support
It compiles and passes the selftest, but it's otherwise barely tested. There are a few todos (2 I think?) and many of the commits need changelogs, i.e. it's still a WIP.
When running LTP (https://github.com/linux-test-project/ltp) on the v10 bits (and also with Sean's branch above) I encounter the following NULL pointer dereference with testcases/kernel/syscalls/madvise/madvise01 (100% reproducible).
Quoted 6 lines
Quoted 4 lines
It appears that in
restrictedmem_error_page()inode->i_mapping->private_datais NULL in thelist_for_each_entry_safe(inode, next, &sb->s_inodes, i_sb_list)but I don't know why.Kirill, can you take a look? Or pass the buck to someone who can? :-)
The patch below should help.
mm/restrictedmem.c +0
-0@@ -307,14 +307,29 @@ void restrictedmem_error_page(struct page *page, struct address_space *mapping)spin_lock (&sb->s_inode_list_lock);mm/restrictedmem.c void restrictedmem_error_page(struct page *page, struct address_space *mapping) · line 315
Quoted 10 lines
mm/restrictedmem.c +3
-3list_for_each_entry_safe (inode, next, &sb->s_inodes, i_sb_list) { -struct restrictedmem *rm = inode->i_mapping->private_data;struct restrictedmem_notifier *notifier; -struct file *memfd = rm->memfd; +struct restrictedmem *rm;unsigned long index; +struct file *memfd; -if (memfd->f_mapping != mapping) +if (atomic_read (&inode->i_count))Kirill, should this be
Quoted 4 lines
if (!atomic_read(&inode->i_count)) continue;i.e. skip unreferenced inodes, not skip referenced inodes?
Ouch. Yes.
But looking at other instances of s_inodes usage, I think we can drop the check altogether. inode cannot be completely free until it is removed from s_inodes list.
While there, replace list_for_each_entry_safe() with list_for_each_entry() as we don't remove anything from the list.
mm/restrictedmem.c +2 -5
@@ -194,22 +194,19 @@ static int restricted_error_remove_page(struct address_space *mapping, struct page *page) {struct super_block *sb = restrictedmem_mnt->mnt_sb; -struct inode *inode, *next; +struct inode *inode;pgoff_t start, end; start = page->index; end = start +thp_nr_pages (page);spin_lock (&sb->s_inode_list_lock); -list_for_each_entry_safe (inode, next, &sb->s_inodes, i_sb_list) { +list_for_each_entry (inode, &sb->s_inodes, i_sb_list) {struct restrictedmem_notifier *notifier;struct restrictedmem *rm;unsigned long index;struct file *memfd; -if (atomic_read (&inode->i_count)) -continue ; -spin_lock (&inode->i_lock);if (inode->i_state & (I_NEW |I_FREEING |I_WILL_FREE )) {spin_unlock (&inode->i_lock);
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Mike Rapoport
reviewerRe: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM
In reply to Chao PengHi,
Quoted 9 lines · Chao Peng
This patch series implements KVM guest private memory for confidential computing scenarios like Intel TDX[1]. If a TDX host accesses TDX-protected guest memory, machine check can happen which can further crash the running host system, this is terrible for multi-tenant configurations. The host accesses include those from KVM userspace like QEMU. This series addresses KVM userspace induced crash by introducing new mm and KVM interfaces so KVM userspace can still manage guest memory via a fd-based approach, but it can never access the guest memory content.
Sorry for jumping late.
Unless I'm missing something, hibernation will also cause an machine check when there is TDX-protected memory in the system. When the hibernation creates memory snapshot it essentially walks all physical pages and saves their contents, so for TDX memory this will trigger machine check, right?
Quoted 26 lines
Documentation/virt/kvm/api.rst | 125 ++++++- arch/x86/entry/syscalls/syscall_32.tbl | 1 + arch/x86/entry/syscalls/syscall_64.tbl | 1 + arch/x86/include/asm/kvm_host.h | 9 + arch/x86/kvm/Kconfig | 3 + arch/x86/kvm/mmu/mmu.c | 205 ++++++++++- arch/x86/kvm/mmu/mmu_internal.h | 14 +- arch/x86/kvm/mmu/mmutrace.h | 1 + arch/x86/kvm/mmu/tdp_mmu.c | 2 +- arch/x86/kvm/x86.c | 17 +- include/linux/kvm_host.h | 103 +++++- include/linux/restrictedmem.h | 71 ++++ include/linux/syscalls.h | 1 + include/uapi/asm-generic/unistd.h | 5 +- include/uapi/linux/kvm.h | 53 +++ include/uapi/linux/magic.h | 1 + kernel/sys_ni.c | 3 + mm/Kconfig | 4 + mm/Makefile | 1 + mm/memory-failure.c | 3 + mm/restrictedmem.c | 318 +++++++++++++++++ virt/kvm/Kconfig | 6 + virt/kvm/kvm_main.c | 469 +++++++++++++++++++++---- 23 files changed, 1323 insertions(+), 93 deletions(-) create mode 100644 include/linux/restrictedmem.h create mode 100644 mm/restrictedmem.c
Signature · 2 lines
Sincerely yours, Mike.
Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM
Quoted 13 lines · Mike Rapoport
Hi,
Quoted 9 lines · Chao Peng
This patch series implements KVM guest private memory for confidential computing scenarios like Intel TDX[1]. If a TDX host accesses TDX-protected guest memory, machine check can happen which can further crash the running host system, this is terrible for multi-tenant configurations. The host accesses include those from KVM userspace like QEMU. This series addresses KVM userspace induced crash by introducing new mm and KVM interfaces so KVM userspace can still manage guest memory via a fd-based approach, but it can never access the guest memory content.
Quoted 6 lines
Sorry for jumping late.
Unless I'm missing something, hibernation will also cause an machine check when there is TDX-protected memory in the system. When the hibernation creates memory snapshot it essentially walks all physical pages and saves their contents, so for TDX memory this will trigger machine check, right?
I recall bringing that up in the past (also memory access due to kdump, /prov/kcore) and was told that the main focus for now is preventing unprivileged users from crashing the system, that is, not mapping such memory into user space (e.g., QEMU). In the long run, we'll want to handle such pages also properly in the other events where the kernel might access them.
Signature · 3 lines
Thanks, David / dhildenb
Sean Christopherson
maintainerRe: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM
Quoted 14 lines · David Hildenbrand
Quoted 13 lines · Mike Rapoport
Hi,
Quoted 9 lines · Chao Peng
This patch series implements KVM guest private memory for confidential computing scenarios like Intel TDX[1]. If a TDX host accesses TDX-protected guest memory, machine check can happen which can further crash the running host system, this is terrible for multi-tenant configurations. The host accesses include those from KVM userspace like QEMU. This series addresses KVM userspace induced crash by introducing new mm and KVM interfaces so KVM userspace can still manage guest memory via a fd-based approach, but it can never access the guest memory content.
Quoted 6 lines
Quoted 6 lines
Sorry for jumping late.
Unless I'm missing something, hibernation will also cause an machine check when there is TDX-protected memory in the system. When the hibernation creates memory snapshot it essentially walks all physical pages and saves their contents, so for TDX memory this will trigger machine check, right?
For hibernation specifically, I think that should be handled elsewhere as hibernation is simply incompatible with TDX, SNP, pKVM, etc. without paravirtualizing the guest, as none of those technologies support auto-export a la s390. I suspect the right approach is to disallow hibernation if KVM is running any protected guests.
Quoted 6 lines
I recall bringing that up in the past (also memory access due to kdump, /prov/kcore) and was told that the main focus for now is preventing unprivileged users from crashing the system, that is, not mapping such memory into user space (e.g., QEMU). In the long run, we'll want to handle such pages also properly in the other events where the kernel might access them.
Ya, unless someone strongly objects, the plan is to essentially treat "attacks" from privileged users as out of to scope for initial support, and then iterate as needed to fix/enable more features.
FWIW, read accesses, e.g. kdump, should be ok for TDX and SNP as they both play nice with "bad" reads. pKVM is a different beast though as I believe any access to guest private memory will fault. But my understanding is that this series would be a big step forward for pKVM, which currently doesn't have any safeguards.
Sean Christopherson
maintainerRename restrictedmem => guardedmem? (was: Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM)
In reply to Chao PengWhat do y'all think about renaming "restrictedmem" to "guardedmem"?
I want to start referring to the code/patches by its syscall/implementation name instead of "UPM", as "UPM" is (a) very KVM centric, (b) refers to the broader effort and not just the non-KVM code, and (c) will likely be confusing for future reviewers since there's nothing in the code that mentions "UPM" in any way.
But typing out restrictedmem is quite tedious, and git grep shows that "rmem" is already used to refer to "reserved memory".
Renaming the syscall to "guardedmem"...
1. Allows for a shorthand and namespace, "gmem", that isn't already in use by the kernel (see "reserved memory above").
2. Provides a stronger hint as to its purpose. "Restricted" conveys that the allocated memory is limited in some way, but doesn't capture how the memory is restricted, e.g. "restricted" could just as easily mean that the allocation can be restricted to certain types of backing stores or something. "Guarded" on the other hand captures that the memory has extra defenses of some form.
3. Is shorter to type and speak. Work smart, not hard :-)
4. Isn't totally wrong for the KVM use case if someone assumes the "g" means "guest" when reading mail and whatnot.
P.S. I trimmed the Cc/To substantially for this particular discussion to avoid
spamming folks that don't (yet) care about this stuff with another potentially lengthy thread. Feel free to add (back) any people/lists.
Re: Rename restrictedmem => guardedmem? (was: Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM)
Quoted 1 line · Sean Christopherson
What do y'all think about renaming "restrictedmem" to "guardedmem"?
Yeay, let's add more confusion :D
If we're at renaming, I'd appreciate if we could find a terminology that does look/sound less horrible.
Quoted 10 lines
I want to start referring to the code/patches by its syscall/implementation name instead of "UPM", as "UPM" is (a) very KVM centric, (b) refers to the broader effort and not just the non-KVM code, and (c) will likely be confusing for future reviewers since there's nothing in the code that mentions "UPM" in any way.
But typing out restrictedmem is quite tedious, and git grep shows that "rmem" is already used to refer to "reserved memory".
Renaming the syscall to "guardedmem"...
restrictedmem, guardedmem, ... all fairly "suboptimal" if you'd ask me ...
Signature · 3 lines
Thanks, David / dhildenb
Sean Christopherson
maintainerRe: Rename restrictedmem => guardedmem? (was: Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM)
Quoted 6 lines · David Hildenbrand
Quoted 5 lines · Sean Christopherson
I want to start referring to the code/patches by its syscall/implementation name instead of "UPM", as "UPM" is (a) very KVM centric, (b) refers to the broader effort and not just the non-KVM code, and (c) will likely be confusing for future reviewers since there's nothing in the code that mentions "UPM" in any way.
Quoted 6 lines
Quoted 4 lines
But typing out restrictedmem is quite tedious, and git grep shows that "rmem" is already used to refer to "reserved memory".
Renaming the syscall to "guardedmem"...
restrictedmem, guardedmem, ... all fairly "suboptimal" if you'd ask me ...
I'm definitely open to other suggestions, but I suspect it's going to be difficult to be more precise than something like "guarded".
E.g. we discussed "unmappable" at one point, but the memory can still be mapped, just not via mmap(). And it's not just about mappings, e.g. read() and its many variants are all disallowed too, despite the kernel direct map still being live (modulo SNP requirements).
Re: Rename restrictedmem => guardedmem? (was: Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM)
Quoted 10 lines · Sean Christopherson
Quoted 9 lines · David Hildenbrand
Quoted 8 lines · Sean Christopherson
I want to start referring to the code/patches by its syscall/implementation name instead of "UPM", as "UPM" is (a) very KVM centric, (b) refers to the broader effort and not just the non-KVM code, and (c) will likely be confusing for future reviewers since there's nothing in the code that mentions "UPM" in any way.
But typing out restrictedmem is quite tedious, and git grep shows that "rmem" is already used to refer to "reserved memory".
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Renaming the syscall to "guardedmem"...
restrictedmem, guardedmem, ... all fairly "suboptimal" if you'd ask me ...
I'm definitely open to other suggestions, but I suspect it's going to be difficult to be more precise than something like "guarded".
Guardedmem is just as bad as restrictedmem IMHO, sorry.
Restricted: what's restricted? how does the restriction manifest? secretmem also has it's restrictions/limitations (pinning), why does that one not fall under the same category?
Make a stranger guess what "restrictedmem" is and I can guarantee that it has nothing to do with the concept we're introducing here.
Guarded: what's guarded? From whom? For which purpose? How does the "guarding" manifest?
Again, make a stranger guess what "guardedmem" is and I can guarantee that it has nothing to do with the concept we're introducing here.
If, at all, the guess might be "guarded storage" [1] on s390x, which, of course, has nothing to do with the concept here. (storage on s390x is just the dinosaur slang for memory)
Often, if we fail to find a good name, the concept is either unclear or not well defined.
So what are the characteristics we want to generalize under that new name? We want to have an fd, that
(a) cannot be mapped into user space (mmap)
(b) cannot be accessed using ordinary system calls (read/write)
(c) can still be managed like other fds (fallocate, future NUMA policies?)
(d) can be consumed by some special entities that are allowed to read/write/map.
So the fd content is inaccessible using the ordinary POSIX syscalls. It's only accessible by special entities (e.g., KVM).
Most probably I am forgetting something. But maybe that will help to find a more expressive name. Maybe :)
Quoted 6 lines
E.g. we discussed "unmappable" at one point, but the memory can still be mapped, just not via
mmap(). And it's not just about mappings, e.g.read()and its many variants are all disallowed too, despite the kernel direct map still being live (modulo SNP requirements).
[1] https://man.archlinux.org/man/s390_guarded_storage.2.en
Signature · 3 lines
Thanks, David / dhildenb
Sean Christopherson
maintainerRe: Rename restrictedmem => guardedmem? (was: Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM)
Quoted 25 lines · David Hildenbrand
Quoted 10 lines · Sean Christopherson
Quoted 9 lines · David Hildenbrand
Quoted 8 lines · Sean Christopherson
I want to start referring to the code/patches by its syscall/implementation name instead of "UPM", as "UPM" is (a) very KVM centric, (b) refers to the broader effort and not just the non-KVM code, and (c) will likely be confusing for future reviewers since there's nothing in the code that mentions "UPM" in any way.
But typing out restrictedmem is quite tedious, and git grep shows that "rmem" is already used to refer to "reserved memory".
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Renaming the syscall to "guardedmem"...
restrictedmem, guardedmem, ... all fairly "suboptimal" if you'd ask me ...
I'm definitely open to other suggestions, but I suspect it's going to be difficult to be more precise than something like "guarded".
Guardedmem is just as bad as restrictedmem IMHO, sorry.
Restricted: what's restricted? how does the restriction manifest? secretmem also has it's restrictions/limitations (pinning), why does that one not fall under the same category?
Quoted 6 lines
Make a stranger guess what "restrictedmem" is and I can guarantee that it has nothing to do with the concept we're introducing here.
Guarded: what's guarded? From whom? For which purpose? How does the "guarding" manifest?
I completely agree that "guarded" lacks precision, but as above, I've pretty much given up hope of being super precise. I actually like "restricted", I just don't like that I can't shorten the name.
Hmm, maybe that won't be a huge problem in practice. I can't say I've ever heard any use "rmem" in verbale or written communication, it's primarily just "rmem" in code that we can't use, and I don't mind having to use restrictedmem for the namespace. So maybe we can use "rmem", just not in code?
Or, we could pretend we're pirates and call it arrrmem!, which is definitely going to be how I refer to it in my internal dialogue if we keep "restricted" :-)
Quoted 5 lines
Again, make a stranger guess what "guardedmem" is and I can guarantee that it has nothing to do with the concept we're introducing here.
If, at all, the guess might be "guarded storage" [1] on s390x, which, of course, has nothing to do with the concept here.
Oof, and guarded storage is even documented in Documentation/virt/kvm/api.rst.
Quoted 16 lines
(storage on s390x is just the dinosaur slang for memory)
Often, if we fail to find a good name, the concept is either unclear or not well defined.
So what are the characteristics we want to generalize under that new name? We want to have an fd, that
(a) cannot be mapped into user space (mmap)
(b) cannot be accessed using ordinary system calls (read/write)
(c) can still be managed like other fds (fallocate, future NUMA policies?)
(d) can be consumed by some special entities that are allowed to read/write/map.
Quoted 5 lines
So the fd content is inaccessible using the ordinary POSIX syscalls. It's only accessible by special entities (e.g., KVM).
Most probably I am forgetting something. But maybe that will help to find a more expressive name. Maybe :)
Hidden/Concealed/etc - Too close to secretmem, suffers the "hidden from whom" problem, and depending on the use case, the memory may not actually be concealed from the user that controls the VMM.
Restricted - "rmem" collides with "reserved memory" in code.
Guarded - Conflicts with s390's "guarded storage", has the "from whom" problem.
Inaccessible - Many of the same problems as "hidden".
Unmappable - Doesn't cover things like read/write, and is wrong in the sense that the memory is still mappable, just not via mmap().
Secured - I'm not getting anywhere near this one :-)
Re: Rename restrictedmem => guardedmem? (was: Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM)
On Mon, Apr 17, 2023 at 8:16 PM Sean Christopherson <[email protected]> wrote:
....
Quoted 12 lines
Quoted 5 lines
So the fd content is inaccessible using the ordinary POSIX syscalls. It's only accessible by special entities (e.g., KVM).
Most probably I am forgetting something. But maybe that will help to find a more expressive name. Maybe :)
Hidden/Concealed/etc - Too close to secretmem, suffers the "hidden from whom" problem, and depending on the use case, the memory may not actually be concealed from the user that controls the VMM.
Restricted - "rmem" collides with "reserved memory" in code.
Quoted 8 lines
Guarded - Conflicts with s390's "guarded storage", has the "from whom" problem.
Inaccessible - Many of the same problems as "hidden".
Unmappable - Doesn't cover things like read/write, and is wrong in the sense that the memory is still mappable, just not via
mmap().Secured - I'm not getting anywhere near this one :-)
How about "protected" ;)? _ducks_
To me the name doesn't matter much, but fwiw I have developed a liking to "restricted", more than the previous "private", since of all of the one-word suggestions I think it captures most of what it's trying to do.
Cheers,
/fuad
Re: Rename restrictedmem => guardedmem? (was: Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM)
In reply to Sean ChristophersonQuoted 37 lines · Sean Christopherson
Quoted 25 lines · David Hildenbrand
Quoted 10 lines · Sean Christopherson
Quoted 9 lines · David Hildenbrand
Quoted 8 lines · Sean Christopherson
I want to start referring to the code/patches by its syscall/implementation name instead of "UPM", as "UPM" is (a) very KVM centric, (b) refers to the broader effort and not just the non-KVM code, and (c) will likely be confusing for future reviewers since there's nothing in the code that mentions "UPM" in any way.
But typing out restrictedmem is quite tedious, and git grep shows that "rmem" is already used to refer to "reserved memory".
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Renaming the syscall to "guardedmem"...
restrictedmem, guardedmem, ... all fairly "suboptimal" if you'd ask me ...
I'm definitely open to other suggestions, but I suspect it's going to be difficult to be more precise than something like "guarded".
Guardedmem is just as bad as restrictedmem IMHO, sorry.
Restricted: what's restricted? how does the restriction manifest? secretmem also has it's restrictions/limitations (pinning), why does that one not fall under the same category?
Quoted 6 lines
Make a stranger guess what "restrictedmem" is and I can guarantee that it has nothing to do with the concept we're introducing here.
Guarded: what's guarded? From whom? For which purpose? How does the "guarding" manifest?
I completely agree that "guarded" lacks precision, but as above, I've pretty much given up hope of being super precise. I actually like "restricted", I just don't like that I can't shorten the name.
Quoted 7 lines
Hmm, maybe that won't be a huge problem in practice. I can't say I've ever heard any use "rmem" in verbale or written communication, it's primarily just "rmem" in code that we can't use, and I don't mind having to use restrictedmem for the namespace. So maybe we can use "rmem", just not in code?
Or, we could pretend we're pirates and call it arrrmem!, which is definitely going to be how I refer to it in my internal dialogue if we keep "restricted" :-)
:)
Quoted 37 lines
Quoted 6 lines
Again, make a stranger guess what "guardedmem" is and I can guarantee that it has nothing to do with the concept we're introducing here.
If, at all, the guess might be "guarded storage" [1] on s390x, which, of course, has nothing to do with the concept here.
Oof, and guarded storage is even documented in
Documentation/virt/kvm/api.rst.Quoted 16 lines
(storage on s390x is just the dinosaur slang for memory)
Often, if we fail to find a good name, the concept is either unclear or not well defined.
So what are the characteristics we want to generalize under that new name? We want to have an fd, that
(a) cannot be mapped into user space (mmap)
(b) cannot be accessed using ordinary system calls (read/write)
(c) can still be managed like other fds (fallocate, future NUMA policies?)
(d) can be consumed by some special entities that are allowed to read/write/map.Quoted 5 lines
So the fd content is inaccessible using the ordinary POSIX syscalls. It's only accessible by special entities (e.g., KVM).
Most probably I am forgetting something. But maybe that will help to find a more expressive name. Maybe :)
Hidden/Concealed/etc - Too close to secretmem, suffers the "hidden from whom" problem, and depending on the use case, the memory may not actually be concealed from the user that controls the VMM.
Restricted - "rmem" collides with "reserved memory" in code.
Quoted 8 lines
Guarded - Conflicts with s390's "guarded storage", has the "from whom" problem.
Inaccessible - Many of the same problems as "hidden".
Unmappable - Doesn't cover things like read/write, and is wrong in the sense that the memory is still mappable, just not via
mmap().Secured - I'm not getting anywhere near this one :-)
The think about "secretmem" that I kind-of like (a little) is that it explains what it's used for: storing secrets. We don't call it "unmapped" memory because we unmap it from the directmap or "unpinnable" memory or "inaccessible" memory ... or even "restricted" because it has restrictions ... how the secrets are protected is kind of an implementation detail.
So instead of describing *why*/*how* restrictedmem is the weird kid (restricted/guarded/hidden/restricted/inaccessible/ ...), maybe rather describe what it's used for?
I know, I know, "there are other use cases where it will be used outside of VM context". I really don't care. "memfd_vm" / "vm_mem" would be sooo (feel free to add some more o's here) much easier to get. It's a special fd to be used to back VM memory. Depending on the VM type (encrypted/protected/whatever), restrictions might apply (not able to mmap, not able to read/write ...). For example, there really is no need to disallow mmap/read/write when using that memory to back a simple VM where all we want to do is avoid user-space page tables.
Signature · 3 lines
Thanks, David / dhildenb
Sean Christopherson
maintainerRe: Rename restrictedmem => guardedmem? (was: Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM)
Quoted 22 lines · David Hildenbrand
Quoted 6 lines · Sean Christopherson
Hidden/Concealed/etc - Too close to secretmem, suffers the "hidden from whom" problem, and depending on the use case, the memory may not actually be concealed from the user that controls the VMM.
Restricted - "rmem" collides with "reserved memory" in code.
Quoted 8 lines
Guarded - Conflicts with s390's "guarded storage", has the "from whom" problem.
Inaccessible - Many of the same problems as "hidden".
Unmappable - Doesn't cover things like read/write, and is wrong in the sense that the memory is still mappable, just not via
mmap().Secured - I'm not getting anywhere near this one :-)
The think about "secretmem" that I kind-of like (a little) is that it explains what it's used for: storing secrets. We don't call it "unmapped" memory because we unmap it from the directmap or "unpinnable" memory or "inaccessible" memory ... or even "restricted" because it has restrictions ... how the secrets are protected is kind of an implementation detail.
Quoted 6 lines
So instead of describing *why*/*how* restrictedmem is the weird kid (restricted/guarded/hidden/restricted/inaccessible/ ...), maybe rather describe what it's used for?
I know, I know, "there are other use cases where it will be used outside of VM context". I really don't care.
Heh, we originally proposed F_SEAL_GUEST, but that was also sub-optimal[1] ;-)
Quoted 6 lines
"
memfd_vm" / "vm_mem" would be sooo (feel free to add some more o's here) much easier to get. It's a special fd to be used to back VM memory. Depending on the VM type (encrypted/protected/whatever), restrictions might apply (not able to mmap, not able to read/write ...). For example, there really is no need to disallow mmap/read/write when using that memory to back a simple VM where all we want to do is avoid user-space page tables.
In seriousness, I do agree with Jason's very explicit objection[2] against naming a non-KVM uAPI "guest", or any variation thereof.
An alternative that we haven't considered since the very early days is making the uAPI a KVM ioctl() instead of a memfd() flag or dedicated syscall. Looking at the code for "pure shim" implementation[3], that's actually not that crazy of an idea.
I don't know that I love the idea of burying this in KVM, but there are benefits to coupling restrictedmem to KVM (aside from getting out from behind this bikeshed that I created).
The big benefit is that the layer of indirection goes away. That simplifies things like enhancing restrictedmem to allow host userspace access for debug purposes, batching TLB flushes if a PUNCH_HOLE spans multiple memslots, enforcing exclusive access, likely the whole "share with a device" story if/when we get there, etc.
The obvious downsides are that (a) maintenance falls under the KVM umbrella, but that's likely to be true in practice regardless of where the code lands, and
(b) if another use case comes along, e.g. the Gunyah hypervisor[4][5], we risk someone reinventing a similar solution.
If we can get Gunyah on board and they don't need substantial changes to the restrictedmem implementation, then I'm all for continuing on the path we're on. But if Gunyah wants to do their own thing, and the lightweight shim approach is viable, then it's awfully tempting to put this all behind a KVM ioctl().
[1] https://lore.kernel.org/all/[email protected]
[2] https://lore.kernel.org/all/[email protected]
[3] https://lore.kernel.org/all/ZDiCG%[email protected]
[4] https://lore.kernel.org/all/Y%[email protected]
[5] https://lore.kernel.org/all/[email protected]
Re: Rename restrictedmem => guardedmem? (was: Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM)
Quoted 39 lines · Sean Christopherson
Quoted 22 lines · David Hildenbrand
Quoted 6 lines · Sean Christopherson
Hidden/Concealed/etc - Too close to secretmem, suffers the "hidden from whom" problem, and depending on the use case, the memory may not actually be concealed from the user that controls the VMM.
Restricted - "rmem" collides with "reserved memory" in code.
Quoted 8 lines
Guarded - Conflicts with s390's "guarded storage", has the "from whom" problem.
Inaccessible - Many of the same problems as "hidden".
Unmappable - Doesn't cover things like read/write, and is wrong in the sense that the memory is still mappable, just not via
mmap().Secured - I'm not getting anywhere near this one :-)
The think about "secretmem" that I kind-of like (a little) is that it explains what it's used for: storing secrets. We don't call it "unmapped" memory because we unmap it from the directmap or "unpinnable" memory or "inaccessible" memory ... or even "restricted" because it has restrictions ... how the secrets are protected is kind of an implementation detail.
Quoted 6 lines
So instead of describing *why*/*how* restrictedmem is the weird kid (restricted/guarded/hidden/restricted/inaccessible/ ...), maybe rather describe what it's used for?
I know, I know, "there are other use cases where it will be used outside of VM context". I really don't care.
Heh, we originally proposed
F_SEAL_GUEST, but that was also sub-optimal[1] ;-)Quoted 6 lines
"
memfd_vm" / "vm_mem" would be sooo (feel free to add some more o's here) much easier to get. It's a special fd to be used to back VM memory. Depending on the VM type (encrypted/protected/whatever), restrictions might apply (not able to mmap, not able to read/write ...). For example, there really is no need to disallow mmap/read/write when using that memory to back a simple VM where all we want to do is avoid user-space page tables.
Quoted 2 lines
In seriousness, I do agree with Jason's very explicit objection[2] against naming a non-KVM uAPI "guest", or any variation thereof.
While I agree, it's all better than the naming we use right now ...
Let me throw "tee_mem" / "memfd_tee" into the picture. That could eventually catch what we want to have.
Or "coco_mem" / "memfd_coco".
Of course, both expect that people know the terminology (just like what "vm" stands for), but it's IMHO significantly better than restricted/guarded/opaque/whatsoever.
Again, expresses what it's used for, not why it behaves in weird ways.
Quoted 4 lines
An alternative that we haven't considered since the very early days is making the uAPI a KVM
ioctl()instead of amemfd()flag or dedicated syscall. Looking at the code for "pure shim" implementation[3], that's actually not that crazy of an idea.
Yes.
Quoted 4 lines
I don't know that I love the idea of burying this in KVM, but there are benefits to coupling restrictedmem to KVM (aside from getting out from behind this bikeshed that I created).
Yes, it's all better than jumping through hoops to come up with a bad name like "restrictedmem".
Quoted 8 lines
The big benefit is that the layer of indirection goes away. That simplifies things like enhancing restrictedmem to allow host userspace access for debug purposes, batching TLB flushes if a
PUNCH_HOLEspans multiple memslots, enforcing exclusive access, likely the whole "share with a device" story if/when we get there, etc.The obvious downsides are that (a) maintenance falls under the KVM umbrella, but that's likely to be true in practice regardless of where the code lands, and
Yes.
Quoted 2 lines
(b) if another use case comes along, e.g. the Gunyah hypervisor[4][5], we risk someone reinventing a similar solution.
I agree. But if it's as simple as providing an ioctl for that hypervisor that simply wires up the existing implementation, it's not too bad.
Quoted 5 lines
If we can get Gunyah on board and they don't need substantial changes to the restrictedmem implementation, then I'm all for continuing on the path we're on. But if Gunyah wants to do their own thing, and the lightweight shim approach is viable, then it's awfully tempting to put this all behind a KVM
ioctl().
Right. Or we still succeed in finding a name that's not as bad as what we had so far.
Signature · 3 lines
Thanks, David / dhildenb
Sean Christopherson
maintainerRe: Rename restrictedmem => guardedmem? (was: Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM)
Quoted 18 lines · David Hildenbrand
Quoted 10 lines · Sean Christopherson
Quoted 6 lines · David Hildenbrand
"
memfd_vm" / "vm_mem" would be sooo (feel free to add some more o's here) much easier to get. It's a special fd to be used to back VM memory. Depending on the VM type (encrypted/protected/whatever), restrictions might apply (not able to mmap, not able to read/write ...). For example, there really is no need to disallow mmap/read/write when using that memory to back a simple VM where all we want to do is avoid user-space page tables.In seriousness, I do agree with Jason's very explicit objection[2] against naming a non-KVM uAPI "guest", or any variation thereof.
While I agree, it's all better than the naming we use right now ...
Let me throw "
tee_mem" / "memfd_tee" into the picture. That could eventually catch what we want to have.
Quoted 7 lines
Or "
coco_mem" / "memfd_coco".Of course, both expect that people know the terminology (just like what "vm" stands for), but it's IMHO significantly better than restricted/guarded/opaque/whatsoever.
Again, expresses what it's used for, not why it behaves in weird ways.
I don't want to explicitly tie this to trusted execution or confidential compute, as there is value in backing "normal" guests with memory that cannot be accessed by the host userspace without jumping through a few extra hoops, e.g. to add a layer of protection against data corruption due to host userspace bugs.
Quoted 5 lines
Quoted 2 lines
(b) if another use case comes along, e.g. the Gunyah hypervisor[4][5], we risk someone reinventing a similar solution.
I agree. But if it's as simple as providing an ioctl for that hypervisor that simply wires up the existing implementation, it's not too bad.
Yeah, my mind was wandering in this direction too. The absolute worst case scenario seems to be that we do end up creating a generic syscall that is a superset of KVM's functionality, in which case KVM would end up with an ioctl() that is just a redirect/wrapper.
Re: Rename restrictedmem => guardedmem? (was: Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM)
Quoted 25 lines · Sean Christopherson
Quoted 24 lines · David Hildenbrand
Quoted 10 lines · Sean Christopherson
Quoted 6 lines · David Hildenbrand
"
memfd_vm" / "vm_mem" would be sooo (feel free to add some more o's here) much easier to get. It's a special fd to be used to back VM memory. Depending on the VM type (encrypted/protected/whatever), restrictions might apply (not able to mmap, not able to read/write ...). For example, there really is no need to disallow mmap/read/write when using that memory to back a simple VM where all we want to do is avoid user-space page tables.In seriousness, I do agree with Jason's very explicit objection[2] against naming a non-KVM uAPI "guest", or any variation thereof.
While I agree, it's all better than the naming we use right now ...
Let me throw "
tee_mem" / "memfd_tee" into the picture. That could eventually catch what we want to have.Or "
coco_mem" / "memfd_coco".Of course, both expect that people know the terminology (just like what "vm" stands for), but it's IMHO significantly better than restricted/guarded/opaque/whatsoever.
Quoted 6 lines
Quoted 1 line
Again, expresses what it's used for, not why it behaves in weird ways.
I don't want to explicitly tie this to trusted execution or confidential compute, as there is value in backing "normal" guests with memory that cannot be accessed by the host userspace without jumping through a few extra hoops, e.g. to add a layer of protection against data corruption due to host userspace bugs.
Nothing speaks against using tee_mem for the same purpose I guess. I like the sound of it after all. :)
Signature · 3 lines
Thanks, David / dhildenb
Sean Christopherson
maintainerRe: Rename restrictedmem => guardedmem? (was: Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM)
In reply to David Hildenbrand+Christian and Hugh
Quoted 33 lines · David Hildenbrand
Quoted 3 lines · Sean Christopherson
An alternative that we haven't considered since the very early days is making the uAPI a KVM
ioctl()instead of amemfd()flag or dedicated syscall. Looking at the code for "pure shim" implementation[3], that's actually not that crazy of an idea.Yes.
Quoted 4 lines
I don't know that I love the idea of burying this in KVM, but there are benefits to coupling restrictedmem to KVM (aside from getting out from behind this bikeshed that I created).
Yes, it's all better than jumping through hoops to come up with a bad name like "restrictedmem".
Quoted 8 lines
The big benefit is that the layer of indirection goes away. That simplifies things like enhancing restrictedmem to allow host userspace access for debug purposes, batching TLB flushes if a
PUNCH_HOLEspans multiple memslots, enforcing exclusive access, likely the whole "share with a device" story if/when we get there, etc.The obvious downsides are that (a) maintenance falls under the KVM umbrella, but that's likely to be true in practice regardless of where the code lands, and
Yes.
Quoted 2 lines
(b) if another use case comes along, e.g. the Gunyah hypervisor[4][5], we risk someone reinventing a similar solution.
I agree. But if it's as simple as providing an ioctl for that hypervisor that simply wires up the existing implementation, it's not too bad.
Quoted 1 line
Quoted 7 lines
Quoted 4 lines
If we can get Gunyah on board and they don't need substantial changes to the restrictedmem implementation, then I'm all for continuing on the path we're on. But if Gunyah wants to do their own thing, and the lightweight shim approach is viable, then it's awfully tempting to put this all behind a KVM
ioctl().Right. Or we still succeed in finding a name that's not as bad as what we had so far.
Okie dokie. So as not to bury the lede:
I think we should provide a KVM ioctl() and have KVM provide its own low-level implementation, i.e. not wrap shmem. As much as I don't want to have KVM serving up memory to userspace, by trying to keep KVM out of the memory management business, we're actually making things *more* complex and harder to maintain (and merge).
Hugh said something to this effect quite early on[1], it just unfortunately took me a long time to understand the benefits.
: QEMU gets fd from opening /dev/kvm_something, uses ioctls (or perhaps : the fallocate syscall interface itself) to allocate and free the memory, : ioctl for initializing some of it too. KVM in control of whether that : fd can be read or written or mmap'ed or whatever, no need to prevent it : in shmem.c, no need for flags, seals, notifications to and fro because : KVM is already in control and knows the history. If shmem actually has : value, call into it underneath - somewhat like SysV SHM, and /dev/zero : mmap, and i915/gem make use of it underneath. If shmem has nothing to : add, just allocate and free kernel memory directly, recorded in your : own xarray.
Christian also suggested that we stop trying to be lazy and create a proper API[2].
: I find this hard to navigate tbh and it feels like taking a shortcut to : avoid building a proper api. If you only care about a specific set of : operations specific to memfd restricte that needs to be available to : in-kernel consumers, I wonder if you shouldn't just go one step further : then your proposal below and build a dedicated minimal ops api. Idk, : sketching like a madman on a drawning board here with no claim to : feasibility from a mm perspective whatsoever
The key point from Hugh is that, except for a few minor things that are trivially easy to replicate, the things that make shmem "shmem" don't provide any value for the KVM use case:
- We have no plans to support swap, and migration support is dubious at best. Swap in particular brings in a lot of complexity for no benefit (to us). That complexity doesn't mean depending on shmem is inherently bad, but it does mean rolling our own implementation is highly unlikely to result in reinventing shmem's wheel.
- Sharing a backing file between arbitrary process is _unwanted_ for the initial use cases. There may come a time when mutually trusted VMs can share "private" data, but (a) that's a distant future problem and (b) it'll likely require even more KVM control over the memory.
- All of the interfaces for read/write/mmap/etc. are dead weight from our perspective. Even worse, we have to actively avoid those interfaces. That can kinda sorta be done without polluting shmem code by using a shim, but that has problems of its own (see below).
And Christian pointed out several flaws with wrapping shmem:
- Implementing a partial overlay filesystem leads to inconsistencies because only some of the ops are changed, e.g. poking at the inode_operations or super_operations will show shmem stuff, whereas address_space_operations and file_operations will show restrictedmem stuff.
- Usurping just f_ops and a_ops without creating a full blown filesystem avoids the partial overlay issues, but partially overriding ops isn't really supported (because it's weird and brittle), e.g. blindly forwarding a fallocate() after restrictedmem does it's thing "works", but only because we very carefully and deliberately designed restrictedmem on top of shmem.
On top of the points raised by Hugh and Christian, wrapping shmem isn't really any simpler, just different. E.g. there's a very subtle bug in the shim variant:
by passing SGP_WRITE, shmem skips zeroing the page because restrictemem is telling shmem that it (restrictedmem) will immediately write the page. For TDX and SNP, that's a "feature" because in those cases trusted firmware will zero (or init) memory when it's assigned to the guest, but it's a nasty flaw for other use cases.
I'm not saying that we'll magically avoid such bugs by avoiding shmem, just pointing out that using shmem requires understanding exactly how shmem works, i.e. using shmem isn't necessarily any easier than building directly on filemap and/or folio APIs. And I gotta imagine it will be a similar story if/when we want to add hugetlbfs support. Building on filemap/folio will definitely have its own challenges, but after prototyping an implementation I can confidently say that none of the problems will be insurmountable. KVM has _way_ more complex code in its memslot and MMU implementations.
And another benefit of building on filemap and/or folio APIs is that, because we would be reinventing the wheel to some extent, when we do inevitablly run into problems, it will be easier to get help solving those problems because (a) we won't be doing weird things no one wants to deal with and (b) because the problems will likely be things others have already dealt with.
The other angle I've been looking at is whether or not having KVM provide its own implementation will lead to maintenance problems in the future, specifically if we get to the point where we want to support "fancy" things like swap and migration. For migration, I'm quite confident that a dedicated KVM ioctl() versus wrapping shmem would be at worst a wash, and more than likely simpler if KVM owns everything. E.g. migrating pages under TDX and SNP requires invoking magic instructions, and so we'd be overriding ->migrate_folio() no matter what.
As for swap, I think we should put a stake in the ground and say that KVM will never support swap for KVM's ioctl(). Sooo much of the infrastructure around swap/reclaim is tied to userspace mappings, e.g. knowing which pages are LRU and/or cold. I poked around a bit to see how we could avoid reinventing all of that infrastructure for fd-only memory, and the best idea I could come up with is basically a rehash of Kirill's very original "KVM protected memory" RFC[3], i.e. allow "mapping" fd-only memory, but ensure that memory is never actually present from hardware's perspective.
But on top of the various problems with that approach, the only use cases I can think of for using fd-only to back non-confidential VMs is to guard against spurious writes/reads to guest memory and/or avoid memory overhead for mapping guest memory into the user page tables. Avoiding memory overhead is completely defeated if the page tables are populated PROT_NONE, which just leaves the "harden against guest data corruption use case". And for that specific use case, _if_ swap is desirable, it would be far, far easier to teach the kernel to allow KVM to follow PROT_NONE (as Kirill's series did), as opposed to trying to teach the kernel and/or KVM how to swap fd-only memory.
In other words, fd-only memory is purely for slice-of-hardware VMs. If someone wants to overcommit VMs, then they use KVM's traditional API for mapping memory into the guest.
Regarding putting this in KVM, as mentioned (way) above in the quote, the worst case scenario of making this a KVM ioctl() seems to be that KVM will end up with an ioctl() that redirects to common kernel code. On the flip side, implementing this in KVM gives us a much clearer path to getting all of this merged. There will be a few small non-KVM patches, but they will either be trivial, e.g. exporting APIs (that aren't contentious) or not strictly required, e.g. adding a flag to mark an address space as completely unmigratable (for improved performance). I.e. the non-KVM patches will be small and be very actionable for their maintainers, and other than that we control our own destiny.
And on the topic of control, making this a KVM ioctl() and implementation gives KVM a _lot_ of control. E.g. we can make the file size immutable to simplify the implementation, bind the fd to a single VM at creation time, add KVM-defined flags for controlling hugepage behavior, etc.
Last but not least, I think forcing us KVM folks to get our hands at least a little dirty with MM and FS stuff would be a *good* thing. KVM has had more than a few bugs that we missed in no small part because most of the people that work on KVM have almost zero knowledge of MM and FS, and especially at the boundaries between those two.
Note, I implemented the POC on top of the filemap APIs because that was much faster and less error prone than re-implementing xarray management myself. We may or may not want to actually make the kernel at large aware of these allocations, i.e. it may be better to follow Hugh's suggestion and use the folio APIs directly instead of piggybacking filemap+folios. E.g. I _think_ migration becomes a complete non-issue if the pages are "raw" allocations and never get marked as LRU-friendly. What I'm not so sure about is if there is anything substantial that we'd lose out on by not reusing the filemap stuff.
The POC also doesn't play nice with Ackerley's patches to allocate files on a user-defined mount. AIUI, that's largely a non-issue as something like fbind() would provide a superset of that functionality and more than one maintainer has expressed (handwavy) support for a generic fbind().
Code is available here if folks want to take a look before any kind of formal posting:
https://github.com/sean-jc/linux.git x86/kvm_gmem_solo
[1] https://lore.kernel.org/all/[email protected]
[2] https://lore.kernel.org/all/20230418-anfallen-irdisch-6993a61be10b@brauner
[3] https://lore.kernel.org/linux-mm/[email protected]
Re: Rename restrictedmem => guardedmem? (was: Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM)
Hi Sean,
Thanks for implementing this POC!
I’ve started porting the selftests (both Chao’s and those I added [1]).
guest mem seems to cover the use cases that have been discussed and proposed so far, but I still need to figure out how gmem can work with
+ hugetlbfs
+ specification of/storing memory policy (for NUMA node bindings)
+ memory accounting - we may need to account for memory used separately, so that guest mem shows up separately on /proc/meminfo and similar places.
One issue I’ve found so far is that the pointer to kvm (gmem->kvm) is not cleaned up, and hence it is possible to crash the host kernel in the following way
1. Create a KVM VM
2. Create a guest mem fd on that VM
3. Create a memslot with the guest mem fd (hence binding the fd to the VM)
4. Close/destroy the KVM VM
5. Call fallocate(PUNCH_HOLE) on the guest mem fd, which uses gmem->kvm when it tries to do invalidation.
I then tried to clean up the gmem->kvm pointer during unbinding when the KVM VM is destroyed.
That works, but then I realized there’s a simpler way to use the pointer after freeing:
1. Create a KVM VM
2. Create a guest mem fd on that VM
3. Close/destroy the KVM VM
4. Call fallocate(PUNCH_HOLE) on the guest mem fd, which uses gmem->kvm when it tries to do invalidation.
Perhaps binding should mean setting the gmem->kvm pointer in addition to gmem->bindings. This makes binding and unbinding symmetric and avoids the use-after-frees described above.
This also means that creating a guest mem fd is no longer dependent on the VM. Perhaps we can make creating a gmem fd a system ioctl (like KVM_GET_API_VERSION and KVM_CREATE_VM) instead of a vm ioctl?
[1] https://lore.kernel.org/all/[email protected]/T/
Ackerley
Sean Christopherson
maintainerRe: Rename restrictedmem => guardedmem? (was: Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM)
Quoted 40 lines · Ackerley Tng
Hi Sean,
Thanks for implementing this POC!
I’ve started porting the selftests (both Chao’s and those I added [1]).
guest mem seems to cover the use cases that have been discussed and proposed so far, but I still need to figure out how gmem can work with
+ hugetlbfs
+ specification of/storing memory policy (for NUMA node bindings)
+ memory accounting - we may need to account for memory used separately, so that guest mem shows up separately on/proc/meminfoand similar places.One issue I’ve found so far is that the pointer to kvm (
gmem->kvm) is not cleaned up, and hence it is possible to crash the host kernel in the following way1. Create a KVM VM
2. Create a guest mem fd on that VM
3. Create a memslot with the guest mem fd (hence binding the fd to the VM)
4. Close/destroy the KVM VM
5. Callfallocate(PUNCH_HOLE)on the guest mem fd, which usesgmem->kvmwhen it tries to do invalidation.I then tried to clean up the
gmem->kvmpointer during unbinding when the KVM VM is destroyed.That works, but then I realized there’s a simpler way to use the pointer after freeing:
1. Create a KVM VM
2. Create a guest mem fd on that VM
3. Close/destroy the KVM VM
4. Callfallocate(PUNCH_HOLE)on the guest mem fd, which usesgmem->kvmwhen it tries to do invalidation.
Quoted 3 lines
Perhaps binding should mean setting the
gmem->kvmpointer in addition togmem->bindings. This makes binding and unbinding symmetric and avoids the use-after-frees described above.
Hrm, that would work, though it's a bit convoluted, e.g. would require detecting when the last binding is being removed. A similar (also ugly) solution would be to nullify gmem->kvm when KVM dies.
I don't love either approach idea because it means a file created in the context of a VM can outlive the VM itself, and then userspace ends up with a file descriptor that it can't do anything with except close(). I doubt that matters in practice though, e.g. when the VM dies, all memory can be freed so that the file ends up being little more than a shell. And if we go that route, there's no need to grab a reference to the file during bind, KVM can just grab a longterm reference when the file is initially created and then drop it when KVM dies (and nullifies gmem->kvm).
Blech, another wart is that I believe gmem would need to do __module_get() during file creation to prevent kvm.ko from being unloaded after the last VM dies. Ah, but that'd also be true if we went with a system-scoped KVM ioctl(), so I suppose it's not _that_ ugly.
Exchanging references (at binding or at creation) doesn't work, because that creates a circular dependency, i.e. gmem and KVM would pin each other.
A "proper" refcounting approach, where the file pins KVM and not vice versa, gets nasty because of how KVM's memslots work. The least awful approach I can think of would be to delete the associated memslot(s) when the file is released, possibly via deferred work to avoid deadlock issues. Not the prettiest thing ever and in some ways that'd yield an even worse ABI.
Side topic, there's a second bug (and probably more lurking): kvm_swap_active_memslots()'s call to synchronize_srcu_expedited() is done _before_ the call to kvm_gmem_unbind(), i.e. doesn't wait for readers in kvm_gmem_invalidate_begin() to go away. The easy solution for that one is to add another synchronize_srcu_expedited() after unbinding.
Quoted 3 lines
This also means that creating a guest mem fd is no longer dependent on the VM. Perhaps we can make creating a gmem fd a system ioctl (like
KVM_GET_API_VERSIONandKVM_CREATE_VM) instead of a vm ioctl?
My preference is to make it a VM-scoped ioctl(), if it ends up being a KVM ioctl() and not a common syscall. If the file isn't tightly coupled to a single VM, then punching a hole is further complicated by needing to deal with invalidating multiple regions that are bound to different @kvm instances. It's not super complex, but AFAICT having the ioctl() be system-scoped doesn't add value, e.g. I don't think having one VM own the memory will complicate even if/when we get to the point where VMs can share "private" memory, and the gmem code would still need to deal with grabbing a module reference.
Re: Rename restrictedmem => guardedmem? (was: Re: [PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM)
Quoted 10 lines · Sean Christopherson
...
My preference is to make it a VM-scopedioctl(), if it ends up being a KVMioctl()and not a common syscall. If the file isn't tightly coupled to a single VM, then punching a hole is further complicated by needing to deal with invalidating multiple regions that are bound to different @kvm instances. It's not super complex, but AFAICT having theioctl()be system-scoped doesn't add value, e.g. I don't think having one VM own the memory will complicate even if/when we get to the point where VMs can share "private" memory, and the gmem code would still need to deal with grabbing a module reference.
Copyless migration would be a scenario where "private" memory may need to be shared between source and target VMs depending on how migration support is implemented.
Regards,
Vishal