QEMU gmem implemention
Subsystems (3)
v2 cover QEMU gmem implemention
It's the v2 RFC of enabling KVM gmem[1] as the backend for private memory.
For confidential-computing, KVM provides gmem/guest_mem interfaces for userspace, like QEMU, to allocate user-unaccesible private memory. This series aims to add gmem support in QEMU's RAMBlock so that each RAM can have both hva-based shared memory and gmem_fd based private memory. QEMU does the shared-private conversion on KVM_MEMORY_EXIT and discards the memory.
It chooses the design that adds "private" property to hostmeory backend. If "private" property is set, QEMU will allocate/create KVM gmem when initialize the RAMbloch of the memory backend.
This sereis also introduces the first user of kvm gmem, KVM_X86_SW_PROTECTED_VM. A KVM_X86_SW_PROTECTED_VM with private KVM gmem can be created with
$qemu -object sw-protected-vm,id=sp-vm0 \ -object memory-backend-ram,id=mem0,size=1G,private=on \ -machine q35,kernel_irqchip=split,confidential-guest-support=sp-vm0,memory-backend=mem0 \ ...
Unfortunately this patch series fails the boot of OVMF at very early stage due to triple fault, because KVM doesn't support emulating string IO to private memory.
This version still leave some opens to be discussed:
1. whether we need "private" propery to be user-settable?
It seems unnecessary because vm-type is determined. If the VM is confidential-guest, then the RAM of the guest must be able to be mapped as private, i.e., have kvm gmem backend. So QEMU can determine the value of "private" property automatiacally based on vm type.
This also aligns with the board internal MemoryRegion that needs to have kvm gmem backend, e.g., TDX requires OVMF to act as private memory so bios memory region needs to have kvm gmem fd associated. QEMU no doubt will do it internally automatically.
2. hugepage support.
KVM gmem can be allocated from hugetlbfs. How does QEMU determine when to allocate KVM gmem with KVM_GUEST_MEMFD_ALLOW_HUGEPAGE. The easiest solution is create KVM gmem with KVM_GUEST_MEMFD_ALLOW_HUGEPAGE only when memory backend is HostMemoryBackendFile of hugetlbfs.
3. What is KVM_X86_SW_PROTECTED_VM going to look like? and do we need it?
This series implements KVM_X86_SW_PROTECTED_VM because it's introduced with gmem together on KVM side and it's supposed to be the first user who requires KVM gmem. However the implementation is incomplete and there lacks the definition of how KVM_X86_SW_PROTECTED_VM works.
Any other idea/open/question is welcomed.
Beside, TDX QEMU implemetation is based on this series to provide private gmem for TD private memory, which can be found at [2]. And it can work corresponding KVM [3] to boot TDX guest.
[1] https://lore.kernel.org/all/[email protected]/
[2] https://github.com/intel/qemu-tdx/tree/tdx-qemu-upstream
[3] https://github.com/intel/tdx/tree/kvm-upstream-2023.07.27-v6.5-rc2-workaround
===
Changes since rfc v1:
- Implement KVM_X86_SW_PROTECTED_VM with confidential-guest-support interface;
- rename memory_region_can_be_private() to memory_region_has_gmem_fd();
- allocate kvm gmem fd when creating/initializing the memory backend by introducing the RAM_KVM_GMEM flag;
Chao Peng (3):
RAMBlock: Add support of KVM private gmem
kvm: Enable KVM_SET_USER_MEMORY_REGION2 for memslot
kvm: handle KVM_EXIT_MEMORY_FAULT
Isaku Yamahata (4):
HostMem: Add private property and associate it with RAM_KVM_GMEM trace/kvm: Add trace for page convertion between shared and private pci-host/q35: Move PAM initialization above SMRAM initialization
q35: Introduce smm_ranges property for q35-pci-host
Xiaoyao Li (14):
*** HACK *** linux-headers: Update headers to pull in gmem APIs
memory: Introduce memory_region_has_gmem_fd()
i386: Add support for sw-protected-vm object
i386/pc: Drop pc_machine_kvm_type()
target/i386: Implement mc->kvm_type() to get VM type target/i386: Introduce kvm_confidential_guest_init() i386/kvm: Implement kvm_sw_protected_vm_init() for sw-protcted-vm
specific functions
kvm: Introduce support for memory_attributes
kvm/memory: Introduce the infrastructure to set the default
shared/private value
i386/kvm: Set memory to default private for KVM_X86_SW_PROTECTED_VM
physmem: replace function name with __func__ in
ram_block_discard_range()
physmem: extract ram_block_discard_range_fd() from
ram_block_discard_range()
physmem: Introduce ram_block_convert_range()
i386: Disable SMM mode for X86_SW_PROTECTED_VM
accel/kvm/kvm-all.c | 180 ++++++++++++++++++++- accel/kvm/trace-events | 4 +- backends/hostmem-file.c | 1 + backends/hostmem-memfd.c | 1 + backends/hostmem-ram.c | 1 + backends/hostmem.c | 18 +++ hw/i386/pc.c | 5 - hw/i386/pc_q35.c | 3 +- hw/i386/x86.c | 12 ++ hw/pci-host/q35.c | 61 ++++--- include/exec/cpu-common.h | 2 + include/exec/memory.h | 20 +++ include/exec/ramblock.h | 1 + include/hw/i386/pc.h | 4 +- include/hw/i386/x86.h | 1 + include/hw/pci-host/q35.h | 1 + include/sysemu/hostmem.h | 2 +- include/sysemu/kvm.h | 5 + include/sysemu/kvm_int.h | 2 + linux-headers/asm-x86/kvm.h | 3 + linux-headers/linux/kvm.h | 50 ++++++ qapi/qom.json | 5 + softmmu/memory.c | 18 +++ softmmu/physmem.c | 256 ++++++++++++++++++------------ target/i386/kvm/kvm.c | 43 ++++- target/i386/kvm/kvm_i386.h | 1 + target/i386/kvm/meson.build | 1 + target/i386/kvm/sw-protected-vm.c | 71 +++++++++ target/i386/kvm/sw-protected-vm.h | 19 +++ target/i386/sev.c | 1 - target/i386/sev.h | 2 + 31 files changed, 648 insertions(+), 146 deletions(-) create mode 100644 target/i386/kvm/sw-protected-vm.c create mode 100644 target/i386/kvm/sw-protected-vm.h
Signature · 1 line
2.34.1
RFC PATCH v2 1/21 *** HACK *** linux-headers: Update headers to pull in gmem APIs
This patch needs to be updated by script
scripts/update-linux-headers.sh
once gmem fd support is upstreamed in Linux kernel.
--- linux-headers/asm-x86/kvm.h | 3 +++ linux-headers/linux/kvm.h | 50 +++++++++++++++++++++++++++++++++++++ 2 files changed, 53 insertions(+)
linux-headers/asm-x86/kvm.h +3 -0
@@ -560,4 +560,7 @@ struct kvm_pmu_event_filter {/* x86-specific KVM_EXIT_HYPERCALL flags. */ #define KVM_EXIT_HYPERCALL_LONG_MODE BIT(0) +#define KVM_X86_DEFAULT_VM 0 +#define KVM_X86_SW_PROTECTED_VM 1 +#endif /* _ASM_X86_KVM_H */
linux-headers/linux/kvm.h +50 -0
90 lines not on this page · raw message
Signature · 1 line
2.34.1
RFC PATCH v2 2/21 RAMBlock: Add support of KVM private gmem
In reply to Xiaoyao LiFrom: Chao Peng <[email protected]>
Add KVM gmem support to RAMBlock so both normal hva based memory and kvm gmem fd based private memory can be associated in one RAMBlock.
Introduce new flag RAM_KVM_GMEM. It calls KVM ioctl to create private gmem for the RAMBlock when it's set.
--- accel/kvm/kvm-all.c | 17 +++++++++++++++++ include/exec/memory.h | 3 +++ include/exec/ramblock.h | 1 + include/sysemu/kvm.h | 2 ++ softmmu/physmem.c | 18 +++++++++++++++--- 5 files changed, 38 insertions(+), 3 deletions(-)
accel/kvm/kvm-all.c +17 -0
@@ -4225,3 +4225,20 @@ void query_stats_schemas_cb(StatsSchemaList **result, Error **errp)query_stats_schema_vcpu (first_cpu, &stats_args); } } + +int kvm_create_guest_memfd (uint64_t size,uint64_t flags,Error **errp) +{ +int fd; +struct kvm_create_guest_memfd gmem = { + .size = size, +/* TODO: to decide whether KVM_GUEST_MEMFD_ALLOW_HUGEPAGE is supported */ + .flags = flags, + }; + + fd =kvm_vm_ioctl (kvm_state,KVM_CREATE_GUEST_MEMFD , &gmem); +if (fd <0 ) { +error_setg_errno (errp, errno,"%s: error creating kvm gmem\n" , __func__); + } + +return fd; +}
include/exec/memory.h +3 -0
@@ -235,6 +235,9 @@ typedef struct IOMMUTLBEvent {/* RAM is an mmap-ed named file */ #define RAM_NAMED_FILE (1 << 9) +/* RAM can be private that has kvm gmem backend */ +#define RAM_KVM_GMEM (1 << 10) + static inline void iommu_notifier_init(IOMMUNotifier *n, IOMMUNotify fn, IOMMUNotifierFlag flags, hwaddr start, hwaddr end,
include/exec/ramblock.h +1 -0
@@ -41,6 +41,7 @@ struct RAMBlock { QLIST_HEAD(, RAMBlockNotifier) ramblock_notifiers;int fd;uint64_t fd_offset; +int gmem_fd;size_t page_size;/* dirty bitmap used during migration */ unsigned long *bmap;
include/sysemu/kvm.h +2 -0
@@ -580,4 +580,6 @@ bool kvm_arch_cpu_check_are_resettable(void);bool kvm_dirty_ring_enabled (void );uint32_t kvm_dirty_ring_size (void ); + +int kvm_create_guest_memfd (uint64_t size,uint64_t flags,Error **errp);#endif
softmmu/physmem.c +15 -3
@@ -1824,6 +1824,16 @@ static void ram_block_add(RAMBlock *new_block, Error **errp) } } +if (kvm_enabled () && new_block->flags &RAM_KVM_GMEM && + new_block->gmem_fd <0 ) { + new_block->gmem_fd =kvm_create_guest_memfd (new_block->max_length, +0 , errp); +if (new_block->gmem_fd <0 ) { +qemu_mutex_unlock_ramlist (); +return ; + } + } + new_ram_size =MAX (old_ram_size, (new_block->offset + new_block->max_length) >>TARGET_PAGE_BITS );if (new_ram_size > old_ram_size) {
@@ -1885,7 +1895,7 @@ RAMBlock *qemu_ram_alloc_from_fd(ram_addr_t size, MemoryRegion *mr,/* Just support these ram flags by now. */ assert ((ram_flags & ~(RAM_SHARED |RAM_PMEM |RAM_NORESERVE | -RAM_PROTECTED |RAM_NAMED_FILE )) ==0 ); +RAM_PROTECTED |RAM_NAMED_FILE |RAM_KVM_GMEM )) ==0 );if (xen_enabled ()) {error_setg (errp,"-mem-path not supported with Xen" );
@@ -1920,6 +1930,7 @@ RAMBlock *qemu_ram_alloc_from_fd(ram_addr_t size, MemoryRegion *mr, new_block->used_length = size; new_block->max_length = size; new_block->flags = ram_flags; + new_block->gmem_fd =-1 ; new_block->host =file_ram_alloc (new_block, size, fd, readonly, !file_size, offset, errp);if (!new_block->host) {
@@ -1978,7 +1989,7 @@ RAMBlock *qemu_ram_alloc_internal(ram_addr_t size, ram_addr_t max_size,Error *local_err =NULL ;assert ((ram_flags & ~(RAM_SHARED |RAM_RESIZEABLE |RAM_PREALLOC | -RAM_NORESERVE )) ==0 ); +RAM_NORESERVE |RAM_KVM_GMEM )) ==0 );assert (!host ^ (ram_flags &RAM_PREALLOC )); size =HOST_PAGE_ALIGN (size);
@@ -1990,6 +2001,7 @@ RAMBlock *qemu_ram_alloc_internal(ram_addr_t size, ram_addr_t max_size, new_block->max_length = max_size;assert (max_size >= size); new_block->fd =-1 ; + new_block->gmem_fd =-1 ; new_block->page_size =qemu_real_host_page_size (); new_block->host = host; new_block->flags = ram_flags;
@@ -2012,7 +2024,7 @@ RAMBlock *qemu_ram_alloc_from_ptr(ram_addr_t size, void *host,RAMBlock *qemu_ram_alloc (ram_addr_t size,uint32_t ram_flags,MemoryRegion *mr,Error **errp) { -assert ((ram_flags & ~(RAM_SHARED |RAM_NORESERVE )) ==0 ); +assert ((ram_flags & ~(RAM_SHARED |RAM_NORESERVE |RAM_KVM_GMEM )) ==0 );return qemu_ram_alloc_internal (size, size,NULL ,NULL , ram_flags, mr, errp); }
Signature · 1 line
2.34.1
Re: [RFC PATCH v2 02/21] RAMBlock: Add support of KVM private gmem
Quoted 9 lines · Xiaoyao Li
From: Chao Peng <[email protected]>
Add KVM gmem support to RAMBlock so both normal hva based memory and kvm gmem fd based private memory can be associated in one RAMBlock.
Introduce new flag
RAM_KVM_GMEM. It calls KVM ioctl to create private gmem for the RAMBlock when it's set.Signed-off-by Xiaoyao Li <[email protected]>
Kindly reminding the author's Signed-off-by is missing.
Quoted 134 lines
--- accel/kvm/kvm-all.c | 17 +++++++++++++++++ include/exec/memory.h | 3 +++ include/exec/ramblock.h | 1 + include/sysemu/kvm.h | 2 ++ softmmu/physmem.c | 18 +++++++++++++++--- 5 files changed, 38 insertions(+), 3 deletions(-)accel/kvm/kvm-all.c +17
-0@@ -4225,3 +4225,20 @@ void query_stats_schemas_cb(StatsSchemaList **result, Error **errp)query_stats_schema_vcpu (first_cpu, &stats_args); } } + +int kvm_create_guest_memfd (uint64_t size,uint64_t flags,Error **errp) +{ +int fd; +struct kvm_create_guest_memfd gmem = { + .size = size, +/* TODO: to decide whether KVM_GUEST_MEMFD_ALLOW_HUGEPAGE is supported */ + .flags = flags, + }; + + fd =kvm_vm_ioctl (kvm_state,KVM_CREATE_GUEST_MEMFD , &gmem); +if (fd <0 ) { +error_setg_errno (errp, errno,"%s: error creating kvm gmem\n" , __func__); + } + +return fd; +}include/exec/memory.h +3
-0@@ -235,6 +235,9 @@ typedef struct IOMMUTLBEvent {/* RAM is an mmap-ed named file */ #define RAM_NAMED_FILE (1 << 9) +/* RAM can be private that has kvm gmem backend */ +#define RAM_KVM_GMEM (1 << 10) + static inline void iommu_notifier_init(IOMMUNotifier *n, IOMMUNotify fn, IOMMUNotifierFlag flags, hwaddr start, hwaddr end,include/exec/ramblock.h +1
-0@@ -41,6 +41,7 @@ struct RAMBlock { QLIST_HEAD(, RAMBlockNotifier) ramblock_notifiers;int fd;uint64_t fd_offset; +int gmem_fd;size_t page_size;/* dirty bitmap used during migration */ unsigned long *bmap;include/sysemu/kvm.h +2
-0@@ -580,4 +580,6 @@ bool kvm_arch_cpu_check_are_resettable(void);bool kvm_dirty_ring_enabled (void );uint32_t kvm_dirty_ring_size (void ); + +int kvm_create_guest_memfd (uint64_t size,uint64_t flags,Error **errp);#endif softmmu/physmem.c +15
-3@@ -1824,6 +1824,16 @@ static void ram_block_add(RAMBlock *new_block, Error **errp) } } +if (kvm_enabled () && new_block->flags &RAM_KVM_GMEM && + new_block->gmem_fd <0 ) { + new_block->gmem_fd =kvm_create_guest_memfd (new_block->max_length, +0 , errp); +if (new_block->gmem_fd <0 ) { +qemu_mutex_unlock_ramlist (); +return ; + } + } + new_ram_size =MAX (old_ram_size, (new_block->offset + new_block->max_length) >>TARGET_PAGE_BITS );if (new_ram_size > old_ram_size) {@@ -1885,7 +1895,7 @@ RAMBlock *qemu_ram_alloc_from_fd(ram_addr_t size, MemoryRegion *mr,/* Just support these ram flags by now. */ assert ((ram_flags & ~(RAM_SHARED |RAM_PMEM |RAM_NORESERVE | -RAM_PROTECTED |RAM_NAMED_FILE )) ==0 ); +RAM_PROTECTED |RAM_NAMED_FILE |RAM_KVM_GMEM )) ==0 );if (xen_enabled ()) {error_setg (errp,"-mem-path not supported with Xen" );@@ -1920,6 +1930,7 @@ RAMBlock *qemu_ram_alloc_from_fd(ram_addr_t size, MemoryRegion *mr, new_block->used_length = size; new_block->max_length = size; new_block->flags = ram_flags; + new_block->gmem_fd =-1 ; new_block->host =file_ram_alloc (new_block, size, fd, readonly, !file_size, offset, errp);if (!new_block->host) {@@ -1978,7 +1989,7 @@ RAMBlock *qemu_ram_alloc_internal(ram_addr_t size, ram_addr_t max_size,Error *local_err =NULL ;assert ((ram_flags & ~(RAM_SHARED |RAM_RESIZEABLE |RAM_PREALLOC | -RAM_NORESERVE )) ==0 ); +RAM_NORESERVE |RAM_KVM_GMEM )) ==0 );assert (!host ^ (ram_flags &RAM_PREALLOC )); size =HOST_PAGE_ALIGN (size);@@ -1990,6 +2001,7 @@ RAMBlock *qemu_ram_alloc_internal(ram_addr_t size, ram_addr_t max_size, new_block->max_length = max_size;assert (max_size >= size); new_block->fd =-1 ; + new_block->gmem_fd =-1 ; new_block->page_size =qemu_real_host_page_size (); new_block->host = host; new_block->flags = ram_flags;@@ -2012,7 +2024,7 @@ RAMBlock *qemu_ram_alloc_from_ptr(ram_addr_t size, void *host,RAMBlock *qemu_ram_alloc (ram_addr_t size,uint32_t ram_flags,MemoryRegion *mr,Error **errp) { -assert ((ram_flags & ~(RAM_SHARED |RAM_NORESERVE )) ==0 ); +assert ((ram_flags & ~(RAM_SHARED |RAM_NORESERVE |RAM_KVM_GMEM )) ==0 );return qemu_ram_alloc_internal (size, size,NULL ,NULL , ram_flags, mr, errp); }
Re: [RFC PATCH v2 02/21] RAMBlock: Add support of KVM private gmem
Quoted 11 lines · Lei
Quoted 9 lines · Xiaoyao Li
From: Chao Peng<[email protected]>
Add KVM gmem support to RAMBlock so both normal hva based memory and kvm gmem fd based private memory can be associated in one RAMBlock.
Introduce new flag
RAM_KVM_GMEM. It calls KVM ioctl to create private gmem for the RAMBlock when it's set.Signed-off-by Xiaoyao Li<[email protected]>Kindly reminding the author's Signed-off-by is missing.
I will fix it.
Thanks!
Re: [RFC PATCH v2 02/21] RAMBlock: Add support of KVM private gmem
In reply to Xiaoyao LiQuoted 7 lines · Xiaoyao Li
From: Chao Peng <[email protected]>
Add KVM gmem support to RAMBlock so both normal hva based memory and kvm gmem fd based private memory can be associated in one RAMBlock.
Introduce new flag
RAM_KVM_GMEM. It calls KVM ioctl to create private gmem for the RAMBlock when it's set.
But who sets RAM_KVM_GMEM and when? Don't we simply allocate it for all RAMBlocks under such special VMs? What's the downside of doing that?
Signature · 3 lines
Cheers, David / dhildenb
Re: [RFC PATCH v2 02/21] RAMBlock: Add support of KVM private gmem
Quoted 11 lines · David Hildenbrand
Quoted 7 lines · Xiaoyao Li
From: Chao Peng <[email protected]>
Add KVM gmem support to RAMBlock so both normal hva based memory and kvm gmem fd based private memory can be associated in one RAMBlock.
Introduce new flag
RAM_KVM_GMEM. It calls KVM ioctl to create private gmem for the RAMBlock when it's set.But who sets
RAM_KVM_GMEMand when?
The answer is in the next patch. When private property of memory backend is set to true, it will pass RAM_KVM_GMEM flag to memory_region_init_ram_*()
Quoted 1 line
Don't we simply allocate it for all RAMBlocks under such special VMs?
yes, this is the direction after your comments.
I'll try to figure out how to achieve it.
Quoted 1 line
What's the downside of doing that?
As far as I see, for TDX, no downside.
Re: [RFC PATCH v2 02/21] RAMBlock: Add support of KVM private gmem
Quoted 11 lines · Xiaoyao Li
Quoted 10 lines · David Hildenbrand
Quoted 7 lines · Xiaoyao Li
From: Chao Peng <[email protected]>
Add KVM gmem support to RAMBlock so both normal hva based memory and kvm gmem fd based private memory can be associated in one RAMBlock.
Introduce new flag
RAM_KVM_GMEM. It calls KVM ioctl to create private gmem for the RAMBlock when it's set.
Quoted 5 lines
Quoted 1 line
But who sets
RAM_KVM_GMEMand when?The answer is in the next patch. When
privateproperty of memory backend is set to true, it will passRAM_KVM_GMEMflag to memory_region_init_ram_*()
Okay, assuming that patch (and property) will go away, I assume this flag can also go away, right?
Signature · 3 lines
Cheers, David / dhildenb
Re: [RFC PATCH v2 02/21] RAMBlock: Add support of KVM private gmem
Quoted 12 lines · David Hildenbrand
Quoted 11 lines · Xiaoyao Li
Quoted 10 lines · David Hildenbrand
Quoted 7 lines · Xiaoyao Li
From: Chao Peng <[email protected]>
Add KVM gmem support to RAMBlock so both normal hva based memory and kvm gmem fd based private memory can be associated in one RAMBlock.
Introduce new flag
RAM_KVM_GMEM. It calls KVM ioctl to create private gmem for the RAMBlock when it's set.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
But who sets
RAM_KVM_GMEMand when?The answer is in the next patch. When
privateproperty of memory backend is set to true, it will passRAM_KVM_GMEMflag to memory_region_init_ram_*()Okay, assuming that patch (and property) will go away, I assume this flag can also go away, right?
If dropping the flag RAM_KVM_GMEM, it seems we need go back to the approach of rfc v1[1][2], that allocating gmem inside .region_add() callback. Is it accepted by you?
Another option is allocating gmem inside ram_block_add() by checking the vm_type (it looks hacky for me). What's your opinion on this option?
One more option is, we keep the RAM_KVM_GMEM as this patch, and change "private" property of memory backend into "need_kvm_gmem" field (make it not user settable) and "need_kvm_gmem" field will be set to true in tdx_kvm_init() specific cgs initialization function.
[1] https://lore.kernel.org/qemu-devel/[email protected]/
[2] https://lore.kernel.org/qemu-devel/[email protected]/
Re: [RFC PATCH v2 02/21] RAMBlock: Add support of KVM private gmem
In reply to Xiaoyao LiQuoted 61 lines · Xiaoyao Li
From: Chao Peng <[email protected]>
Add KVM gmem support to RAMBlock so both normal hva based memory and kvm gmem fd based private memory can be associated in one RAMBlock.
Introduce new flag
RAM_KVM_GMEM. It calls KVM ioctl to create private gmem for the RAMBlock when it's set.Signed-off-by Xiaoyao Li <[email protected]>--- accel/kvm/kvm-all.c | 17 +++++++++++++++++ include/exec/memory.h | 3 +++ include/exec/ramblock.h | 1 + include/sysemu/kvm.h | 2 ++ softmmu/physmem.c | 18 +++++++++++++++--- 5 files changed, 38 insertions(+), 3 deletions(-)accel/kvm/kvm-all.c +17
-0@@ -4225,3 +4225,20 @@ void query_stats_schemas_cb(StatsSchemaList **result, Error **errp)query_stats_schema_vcpu (first_cpu, &stats_args); } } + +int kvm_create_guest_memfd (uint64_t size,uint64_t flags,Error **errp) +{ +int fd; +struct kvm_create_guest_memfd gmem = { + .size = size, +/* TODO: to decide whether KVM_GUEST_MEMFD_ALLOW_HUGEPAGE is supported */ + .flags = flags, + }; + + fd =kvm_vm_ioctl (kvm_state,KVM_CREATE_GUEST_MEMFD , &gmem); +if (fd <0 ) { +error_setg_errno (errp, errno,"%s: error creating kvm gmem\n" , __func__); + } + +return fd; +}include/exec/memory.h +3
-0@@ -235,6 +235,9 @@ typedef struct IOMMUTLBEvent {/* RAM is an mmap-ed named file */ #define RAM_NAMED_FILE (1 << 9) +/* RAM can be private that has kvm gmem backend */ +#define RAM_KVM_GMEM (1 << 10) + static inline void iommu_notifier_init(IOMMUNotifier *n, IOMMUNotify fn, IOMMUNotifierFlag flags, hwaddr start, hwaddr end,include/exec/ramblock.h +0
-0
include/exec/ramblock.h struct RAMBlock { · line 47
Quoted 7 lines
include/exec/ramblock.h +1
-0@@ -41,6 +41,7 @@ struct RAMBlock { QLIST_HEAD(, RAMBlockNotifier) ramblock_notifiers;int fd;uint64_t fd_offset; +int gmem_fd;size_t page_size;/* dirty bitmap used during migration */ unsigned long *bmap;
You're adding a file descriptor to RAMBlock, but I don't see anything in this patch that ever calls close(gmem_fd) when the RAMBlock is released. Presuambly reclaim_ramblock() needs to deal with this ?
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
RFC PATCH v2 3/21 HostMem: Add private property and associate it with RAM_KVM_GMEM
In reply to Xiaoyao LiFrom: Isaku Yamahata <[email protected]>
Add a new property "private" to memory backends. When it's set to true, it indicates the RAMblock of the backend also requires kvm gmem.
--- backends/hostmem-file.c | 1 + backends/hostmem-memfd.c | 1 + backends/hostmem-ram.c | 1 + backends/hostmem.c | 18 ++++++++++++++++++ include/sysemu/hostmem.h | 2 +- qapi/qom.json | 4 ++++ 6 files changed, 26 insertions(+), 1 deletion(-)
backends/hostmem-file.c +1 -0
@@ -56,6 +56,7 @@ file_backend_memory_alloc(HostMemoryBackend *backend, Error **errp) name =host_memory_backend_get_name (backend); ram_flags = backend->share ?RAM_SHARED :0 ; ram_flags |= backend->reserve ?0 :RAM_NORESERVE ; + ram_flags |= backend->private ?RAM_KVM_GMEM :0 ; ram_flags |= fb->is_pmem ?RAM_PMEM :0 ; ram_flags |=RAM_NAMED_FILE ; memory_region_init_ram_from_file(&backend->mr, OBJECT(backend), name,
backends/hostmem-memfd.c +1 -0
@@ -55,6 +55,7 @@ memfd_backend_memory_alloc(HostMemoryBackend *backend, Error **errp) name =host_memory_backend_get_name (backend); ram_flags = backend->share ?RAM_SHARED :0 ; ram_flags |= backend->reserve ?0 :RAM_NORESERVE ; + ram_flags |= backend->private ?RAM_KVM_GMEM :0 ;memory_region_init_ram_from_fd (&backend->mr,OBJECT (backend), name, backend->size, ram_flags, fd,0 , errp);g_free (name);
backends/hostmem-ram.c +1 -0
@@ -30,6 +30,7 @@ ram_backend_memory_alloc(HostMemoryBackend *backend, Error **errp) name =host_memory_backend_get_name (backend); ram_flags = backend->share ?RAM_SHARED :0 ; ram_flags |= backend->reserve ?0 :RAM_NORESERVE ; + ram_flags |= backend->private ?RAM_KVM_GMEM :0 ;memory_region_init_ram_flags_nomigrate (&backend->mr,OBJECT (backend), name, backend->size, ram_flags, errp);g_free (name);
backends/hostmem.c +18 -0
@@ -461,6 +461,20 @@ static void host_memory_backend_set_reserve(Object *o, bool value, Error **errp) } backend->reserve = value; } + +static bool host_memory_backend_get_private (Object *o,Error **errp) +{ +HostMemoryBackend *backend =MEMORY_BACKEND (o); + +return backend->private; +} + +static void host_memory_backend_set_private (Object *o,bool value,Error **errp) +{ +HostMemoryBackend *backend =MEMORY_BACKEND (o); + + backend->private = value; +}#endif /* CONFIG_LINUX */ static bool
@@ -541,6 +555,10 @@ host_memory_backend_class_init(ObjectClass *oc, void *data) host_memory_backend_get_reserve, host_memory_backend_set_reserve);object_class_property_set_description (oc,"reserve" ,"Reserve swap space (or huge pages) if applicable" ); +object_class_property_add_bool (oc,"private" , + host_memory_backend_get_private, host_memory_backend_set_private); +object_class_property_set_description (oc,"private" , +"Use KVM gmem private memory" );#endif /* CONFIG_LINUX */ /* * Do not delete/rename option. This option must be considered stable
include/sysemu/hostmem.h +1 -1
@@ -65,7 +65,7 @@ struct HostMemoryBackend {/* protected */ uint64_t size;bool merge, dump, use_canonical_path; -bool prealloc, is_mapped, share, reserve; +bool prealloc, is_mapped, share, reserve, private;uint32_t prealloc_threads;ThreadContext *prealloc_context;DECLARE_BITMAP (host_nodes,MAX_NODES +1 );
qapi/qom.json +4 -0
@@ -605,6 +605,9 @@ # @reserve: if true, reserve swap space (or huge pages) if applicable # (default: true) (since 6.1) # +# @private: if true, use KVM gmem private memory (default: false) +# (since 8.2) +# # @size: size of the memory region in bytes # # @x-use-canonical-path-for-ramblock-id: if true, the canonical path
@@ -631,6 +634,7 @@ '*prealloc-context': 'str', '*share': 'bool', '*reserve': 'bool', + '*private': 'bool', 'size': 'size', '*x-use-canonical-path-for-ramblock-id': 'bool' } }
Signature · 1 line
2.34.1
Re: [RFC PATCH v2 03/21] HostMem: Add private property and associate it with RAM_KVM_GMEM
Quoted 4 lines · Xiaoyao Li
From: Isaku Yamahata <[email protected]>
Add a new property "private" to memory backends. When it's set to true, it indicates the RAMblock of the backend also requires kvm gmem.
Can you add a brief explanation why you need the property?
Quoted 2 lines
Signed-off-by Isaku Yamahata <[email protected]>Signed-off-by Xiaoyao Li <[email protected]>
[...]
Quoted 21 lines
qapi/qom.json +4
-0@@ -605,6 +605,9 @@ # @reserve: if true, reserve swap space (or huge pages) if applicable # (default: true) (since 6.1) # +# @private: if true, use KVM gmem private memory (default: false) +# (since 8.2) +# # @size: size of the memory region in bytes # # @x-use-canonical-path-for-ramblock-id: if true, the canonical path@@ -631,6 +634,7 @@ '*prealloc-context': 'str', '*share': 'bool', '*reserve': 'bool', + '*private': 'bool', 'size': 'size', '*x-use-canonical-path-for-ramblock-id': 'bool' } }
Re: [RFC PATCH v2 03/21] HostMem: Add private property and associate it with RAM_KVM_GMEM
Quoted 8 lines · Markus Armbruster
Quoted 4 lines · Xiaoyao Li
From: Isaku Yamahata <[email protected]>
Add a new property "private" to memory backends. When it's set to true, it indicates the RAMblock of the backend also requires kvm gmem.
Can you add a brief explanation why you need the property?
It provides a mechanism for user to specify whether the memory can serve as private memory (need request kvm gmem).
Quoted 27 lines
Quoted 2 lines
Signed-off-by Isaku Yamahata <[email protected]>Signed-off-by Xiaoyao Li <[email protected]>[...]
Quoted 21 lines
qapi/qom.json +4
-0@@ -605,6 +605,9 @@ # @reserve: if true, reserve swap space (or huge pages) if applicable # (default: true) (since 6.1) # +# @private: if true, use KVM gmem private memory (default: false) +# (since 8.2) +# # @size: size of the memory region in bytes # # @x-use-canonical-path-for-ramblock-id: if true, the canonical path@@ -631,6 +634,7 @@ '*prealloc-context': 'str', '*share': 'bool', '*reserve': 'bool', + '*private': 'bool', 'size': 'size', '*x-use-canonical-path-for-ramblock-id': 'bool' } }
Re: [RFC PATCH v2 03/21] HostMem: Add private property and associate it with RAM_KVM_GMEM
Quoted 10 lines · Xiaoyao Li
Quoted 7 lines · Markus Armbruster
Quoted 4 lines · Xiaoyao Li
From: Isaku Yamahata <[email protected]>
Add a new property "private" to memory backends. When it's set to true, it indicates the RAMblock of the backend also requires kvm gmem.
Can you add a brief explanation why you need the property?
It provides a mechanism for user to specify whether the memory can serve as private memory (need request kvm gmem).
Yes, but why would a user want such memory?
Re: [RFC PATCH v2 03/21] HostMem: Add private property and associate it with RAM_KVM_GMEM
Quoted 7 lines · Markus Armbruster
Quoted 5 lines · Xiaoyao Li
Quoted 4 lines · Markus Armbruster
Quoted 2 lines · Xiaoyao Li
From: Isaku Yamahata <[email protected]>
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines
Add a new property "private" to memory backends. When it's set to true, it indicates the RAMblock of the backend also requires kvm gmem.
Can you add a brief explanation why you need the property?
It provides a mechanism for user to specify whether the memory can serve as private memory (need request kvm gmem).
Yes, but why would a user want such memory?
Because KVM demands it for confidential guest, e.g., TDX guest. KVM demands that the mem slot needs to have KVM_MEM_PRIVATE set and has valid gmem associated if the guest accesses it as private memory.
Re: [RFC PATCH v2 03/21] HostMem: Add private property and associate it with RAM_KVM_GMEM
Quoted 12 lines · Xiaoyao Li
Quoted 11 lines · Markus Armbruster
Quoted 9 lines · Xiaoyao Li
Quoted 7 lines · Markus Armbruster
Quoted 4 lines · Xiaoyao Li
From: Isaku Yamahata <[email protected]>
Add a new property "private" to memory backends. When it's set to true, it indicates the RAMblock of the backend also requires kvm gmem.
Can you add a brief explanation why you need the property?
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
It provides a mechanism for user to specify whether the memory can serve as private memory (need request kvm gmem).
Yes, but why would a user want such memory?
Because KVM demands it for confidential guest, e.g., TDX guest. KVM demands that the mem slot needs to have
KVM_MEM_PRIVATEset and has valid gmem associated if the guest accesses it as private memory.
I think as long as there is no demand to have a TDX guest with this property be set to "off", then just don't add it.
With a TDX VM, it will can be implicitly active. If we ever have to disable it for selective memory backends, we can add the property and have something like on/off/auto. For now it would be "auto".
Signature · 3 lines
Cheers, David / dhildenb
Re: [RFC PATCH v2 03/21] HostMem: Add private property and associate it with RAM_KVM_GMEM
Quoted 13 lines · David Hildenbrand
Quoted 12 lines · Xiaoyao Li
Quoted 11 lines · Markus Armbruster
Quoted 9 lines · Xiaoyao Li
Quoted 7 lines · Markus Armbruster
Quoted 4 lines · Xiaoyao Li
From: Isaku Yamahata <[email protected]>
Add a new property "private" to memory backends. When it's set to true, it indicates the RAMblock of the backend also requires kvm gmem.
Can you add a brief explanation why you need the property?
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
It provides a mechanism for user to specify whether the memory can serve as private memory (need request kvm gmem).
Yes, but why would a user want such memory?
Because KVM demands it for confidential guest, e.g., TDX guest. KVM demands that the mem slot needs to have
KVM_MEM_PRIVATEset and has valid gmem associated if the guest accesses it as private memory.
Commit messages should explain why we want the patch. Documenting "why" is at least as important as "what". If "what" is missing, I can read the patch to find out. If "why" is missing, I'm reduced to guesswork.
Quoted 3 lines
I think as long as there is no demand to have a TDX guest with this property be set to "off", then just don't add it.
With a TDX VM, it will can be implicitly active. If we ever have to disable it for selective memory backends, we can add the property and have something like on/off/auto. For now it would be "auto".
Makes sense to me.
Re: [RFC PATCH v2 03/21] HostMem: Add private property and associate it with RAM_KVM_GMEM
Quoted 19 lines · Markus Armbruster
Quoted 17 lines · David Hildenbrand
Quoted 16 lines · Xiaoyao Li
Quoted 15 lines · Markus Armbruster
Quoted 10 lines · Xiaoyao Li
Quoted 7 lines · Markus Armbruster
Quoted 4 lines · Xiaoyao Li
From: Isaku Yamahata <[email protected]>
Add a new property "private" to memory backends. When it's set to true, it indicates the RAMblock of the backend also requires kvm gmem.
Can you add a brief explanation why you need the property?
It provides a mechanism for user to specify whether the memory can serve as private memory (need request kvm gmem).
Yes, but why would a user want such memory?
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Because KVM demands it for confidential guest, e.g., TDX guest. KVM demands that the mem slot needs to have
KVM_MEM_PRIVATEset and has valid gmem associated if the guest accesses it as private memory.Commit messages should explain why we want the patch. Documenting "why" is at least as important as "what". If "what" is missing, I can read the patch to find out. If "why" is missing, I'm reduced to guesswork.
I'll try best to improve the commit message of this patch, and all other patches.
Quoted 5 lines
Quoted 3 lines
I think as long as there is no demand to have a TDX guest with this property be set to "off", then just don't add it.
With a TDX VM, it will can be implicitly active. If we ever have to disable it for selective memory backends, we can add the property and have something like on/off/auto. For now it would be "auto".
Makes sense to me.
OK. I think I get the answer of open #1 in cover letter.
If no other voice, I'll drop this patch and allocate gmem RAM when vm_type is TDX.
Re: [RFC PATCH v2 03/21] HostMem: Add private property and associate it with RAM_KVM_GMEM
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
I think as long as there is no demand to have a TDX guest with this property be set to "off", then just don't add it.
With a TDX VM, it will can be implicitly active. If we ever have to disable it for selective memory backends, we can add the property and have something like on/off/auto. For now it would be "auto".
Makes sense to me.
OK. I think I get the answer of open #1 in cover letter.
Yes.
Quoted 2 lines
If no other voice, I'll drop this patch and allocate gmem RAM when
vm_typeis TDX.
Good!
Signature · 3 lines
Cheers, David / dhildenb
RFC PATCH v2 4/21 memory: Introduce memory_region_has_gmem_fd()
In reply to Xiaoyao LiIntroduce memory_region_has_gmem_fd() to query if the MemoryRegion has KVM gmem fd allocated.
--- include/exec/memory.h | 10 ++++++++++ softmmu/memory.c | 5 +++++ 2 files changed, 15 insertions(+)
include/exec/memory.h +10 -0
@@ -1674,6 +1674,16 @@ static inline bool memory_region_is_romd(MemoryRegion *mr)*/ bool memory_region_is_protected (MemoryRegion *mr); +/** +* memory_region_has_gmem_fd: check whether a memory region has KVM gmem fd +* associated +* +* Returns %true if a memory region's ram_block has valid gmem fd assigned. +* +* @mr: the memory region being queried +*/ +bool memory_region_has_gmem_fd (MemoryRegion *mr); +/** * memory_region_get_iommu: check whether a memory region is an iommu *
softmmu/memory.c +5 -0
@@ -1846,6 +1846,11 @@ bool memory_region_is_protected(MemoryRegion *mr)return mr->ram && (mr->ram_block->flags &RAM_PROTECTED ); } +bool memory_region_has_gmem_fd (MemoryRegion *mr) +{ +return mr->ram_block && mr->ram_block->gmem_fd >=0 ; +} +uint8_t memory_region_get_dirty_log_mask (MemoryRegion *mr) {uint8_t mask = mr->dirty_log_mask;
Signature · 1 line
2.34.1
Re: [RFC PATCH v2 04/21] memory: Introduce memory_region_has_gmem_fd()
Quoted 2 lines · Xiaoyao Li
Introduce
memory_region_has_gmem_fd()to query if the MemoryRegion has KVM gmem fd allocated.
*probably* best to just squash that into patch #2.
Signature · 3 lines
Cheers, David / dhildenb
Re: [RFC PATCH v2 04/21] memory: Introduce memory_region_has_gmem_fd()
Quoted 5 lines · David Hildenbrand
Quoted 2 lines · Xiaoyao Li
Introduce
memory_region_has_gmem_fd()to query if the MemoryRegion has KVM gmem fd allocated.*probably* best to just squash that into patch #2.
Sure, I will do it.
RFC PATCH v2 5/21 kvm: Enable KVM_SET_USER_MEMORY_REGION2 for memslot
In reply to Xiaoyao LiFrom: Chao Peng <[email protected]>
Switch to KVM_SET_USER_MEMORY_REGION2 when supported by KVM.
With KVM_SET_USER_MEMORY_REGION2, QEMU can set up memory region that backend'ed both by hva-based shared memory and gmem fd based private memory.
--- accel/kvm/kvm-all.c | 55 ++++++++++++++++++++++++++++++++++------ accel/kvm/trace-events | 2 +- include/sysemu/kvm_int.h | 2 ++ 3 files changed, 50 insertions(+), 9 deletions(-)
accel/kvm/kvm-all.c +47 -8
@@ -288,35 +288,68 @@ int kvm_physical_memory_addr_from_host(KVMState *s, void *ram,static int kvm_set_user_memory_region (KVMMemoryListener *kml,KVMSlot *slot,bool new) {KVMState *s = kvm_state; -struct kvm_userspace_memory_region mem; +struct kvm_userspace_memory_region2 mem; +static int cap_user_memory2 =-1 ;int ret; +if (cap_user_memory2 ==-1 ) { + cap_user_memory2 =kvm_check_extension (s,KVM_CAP_USER_MEMORY2 ); + } + +if (!cap_user_memory2 && slot->gmem_fd >=0 ) { +error_report ("%s, KVM doesn't support gmem!" , __func__); +exit (1 ); + } + mem.slot = slot->slot | (kml->as_id <<16 ); mem.guest_phys_addr = slot->start_addr; mem.userspace_addr = (unsigned long )slot->ram; mem.flags = slot->flags; + mem.gmem_fd = slot->gmem_fd; + mem.gmem_offset = slot->ofs;if (slot->memory_size && !new && (mem.flags ^ slot->old_flags) &KVM_MEM_READONLY ) {/* Set the slot size to 0 before setting the slot to the desired * value. This is needed based on KVM commit 75d61fbc. */ mem.memory_size =0 ; - ret =kvm_vm_ioctl (s,KVM_SET_USER_MEMORY_REGION , &mem); + +if (cap_user_memory2) { + ret =kvm_vm_ioctl (s,KVM_SET_USER_MEMORY_REGION2 , &mem); + }else { + ret =kvm_vm_ioctl (s,KVM_SET_USER_MEMORY_REGION , &mem); + }if (ret <0 ) { goto err; } } mem.memory_size = slot->memory_size; - ret =kvm_vm_ioctl (s,KVM_SET_USER_MEMORY_REGION , &mem); +if (cap_user_memory2) { + ret =kvm_vm_ioctl (s,KVM_SET_USER_MEMORY_REGION2 , &mem); + }else { + ret =kvm_vm_ioctl (s,KVM_SET_USER_MEMORY_REGION , &mem); + } slot->old_flags = mem.flags; err:trace_kvm_set_user_memory (mem.slot >>16 , (uint16_t )mem.slot, mem.flags, mem.guest_phys_addr, mem.memory_size, - mem.userspace_addr, ret); + mem.userspace_addr, mem.gmem_fd, + mem.gmem_offset, ret);if (ret <0 ) { -error_report ("%s: KVM_SET_USER_MEMORY_REGION failed, slot=%d," -" start=0x%" PRIx64", size=0x%" PRIx64": %s" , - __func__, mem.slot, slot->start_addr, - (uint64_t )mem.memory_size,strerror (errno)); +if (cap_user_memory2) { +error_report ("%s: KVM_SET_USER_MEMORY_REGION2 failed, slot=%d," +" start=0x%" PRIx64", size=0x%" PRIx64"," +" flags=0x%" PRIx32"," +" gmem_fd=%" PRId32", gmem_offset=0x%" PRIx64": %s" , + __func__, mem.slot, slot->start_addr, + (uint64_t )mem.memory_size, mem.flags, + mem.gmem_fd, (uint64_t )mem.gmem_offset, +strerror (errno)); + }else { +error_report ("%s: KVM_SET_USER_MEMORY_REGION failed, slot=%d," +" start=0x%" PRIx64", size=0x%" PRIx64": %s" , + __func__, mem.slot, slot->start_addr, + (uint64_t )mem.memory_size,strerror (errno)); + } }return ret; }
@@ -472,6 +505,9 @@ static int kvm_mem_flags(MemoryRegion *mr)if (readonly && kvm_readonly_mem_allowed) { flags |=KVM_MEM_READONLY ; } +if (memory_region_has_gmem_fd (mr)) { + flags |=KVM_MEM_PRIVATE ; + }return flags; }
@@ -1402,6 +1438,9 @@ static void kvm_set_phys_mem(KVMMemoryListener *kml, mem->ram_start_offset = ram_start_offset; mem->ram = ram; mem->flags =kvm_mem_flags (mr); + mem->gmem_fd = mr->ram_block->gmem_fd; + mem->ofs = (uint8_t *)ram - mr->ram_block->host; +kvm_slot_init_dirty_bitmap (mem); err =kvm_set_user_memory_region (kml, mem, true);if (err) {
accel/kvm/trace-events +1 -1
@@ -15,7 +15,7 @@ kvm_irqchip_update_msi_route(int virq) "Updating MSI route virq=%d" kvm_irqchip_release_virq(int virq) "virq %d" kvm_set_ioeventfd_mmio(int fd, uint64_t addr, uint32_t val, bool assign, uint32_t size, bool datamatch) "fd: %d @0x%" PRIx64 " val=0x%x assign: %d size: %d match: %d" kvm_set_ioeventfd_pio(int fd, uint16_t addr, uint32_t val, bool assign, uint32_t size, bool datamatch) "fd: %d @0x%x val=0x%x assign: %d size: %d match: %d" -kvm_set_user_memory(uint16_t as, uint16_t slot, uint32_t flags, uint64_t guest_phys_addr, uint64_t memory_size, uint64_t userspace_addr, int ret) "AddrSpace#%d Slot#%d flags=0x%x gpa=0x%"PRIx64 " size=0x%"PRIx64 " ua=0x%"PRIx64 " ret=%d" +kvm_set_user_memory(uint16_t as, uint16_t slot, uint32_t flags, uint64_t guest_phys_addr, uint64_t memory_size, uint64_t userspace_addr, uint32_t fd, uint64_t fd_offset, int ret) "AddrSpace#%d Slot#%d flags=0x%x gpa=0x%"PRIx64 " size=0x%"PRIx64 " ua=0x%"PRIx64 " gmem_fd=%d" " gmem_fd_offset=0x%" PRIx64 " ret=%d" kvm_clear_dirty_log(uint32_t slot, uint64_t start, uint32_t size) "slot#%"PRId32" start 0x%"PRIx64" size 0x%"PRIx32 kvm_resample_fd_notify(int gsi) "gsi %d" kvm_dirty_ring_full(int id) "vcpu %d"
include/sysemu/kvm_int.h +2 -0
@@ -30,6 +30,8 @@ typedef struct KVMSlotint as_id;/* Cache of the offset in ram address space */ ram_addr_t ram_start_offset; +int gmem_fd; +hwaddr ofs; } KVMSlot; typedef struct KVMMemoryUpdate {
Signature · 1 line
2.34.1
Re: [RFC PATCH v2 05/21] kvm: Enable KVM_SET_USER_MEMORY_REGION2 for memslot
Quoted 10 lines · Xiaoyao Li
From: Chao Peng <[email protected]>
Switch to
KVM_SET_USER_MEMORY_REGION2when supported by KVM.With
KVM_SET_USER_MEMORY_REGION2, QEMU can set up memory region that backend'ed both by hva-based shared memory and gmem fd based private memory.Signed-off-by Chao Peng <[email protected]>Codeveloped-by Xiaoyao Li <[email protected]>
"Co-developed-by".
Signature · 3 lines
Cheers, David / dhildenb
Re: [RFC PATCH v2 05/21] kvm: Enable KVM_SET_USER_MEMORY_REGION2 for memslot
Quoted 5 lines · David Hildenbrand
Quoted 4 lines · Xiaoyao Li
From: Chao Peng <[email protected]>
Switch to
KVM_SET_USER_MEMORY_REGION2when supported by KVM.
Quoted 8 lines
Quoted 6 lines
With
KVM_SET_USER_MEMORY_REGION2, QEMU can set up memory region that backend'ed both by hva-based shared memory and gmem fd based private memory.Signed-off-by Chao Peng <[email protected]>Codeveloped-by Xiaoyao Li <[email protected]>"Co-developed-by".
I will fix it, thanks!
RFC PATCH v2 6/21 i386: Add support for sw-protected-vm object
In reply to Xiaoyao LiIntroduce sw-protected-vm object which implements the interface of CONFIDENTIAL_GUEST_SUPPORT, and will be used to create X86_SW_PROTECTED_VM via
$qemu -machine ...,confidential-guest-support=sp-vm0 \
-object sw-protected-vm,id=sp-vm0--- qapi/qom.json | 1 + target/i386/kvm/meson.build | 1 + target/i386/kvm/sw-protected-vm.c | 35 +++++++++++++++++++++++++++++++ target/i386/kvm/sw-protected-vm.h | 17 +++++++++++++++ 4 files changed, 54 insertions(+) create mode 100644 target/i386/kvm/sw-protected-vm.c create mode 100644 target/i386/kvm/sw-protected-vm.h
qapi/qom.json +1 -0
@@ -944,6 +944,7 @@ 'if': 'CONFIG_SECRET_KEYRING' }, 'sev-guest', 'thread-context', + 'sw-protected-vm', 's390-pv-guest', 'throttle-group', 'tls-creds-anon',
target/i386/kvm/meson.build +1 -0
@@ -5,6 +5,7 @@ i386_softmmu_kvm_ss = ss.source_set() i386_softmmu_kvm_ss.add(files( 'kvm.c', 'kvm-cpu.c', + 'sw-protected-vm.c', )) i386_softmmu_kvm_ss.add(when: 'CONFIG_XEN_EMU', if_true: files('xen-emu.c'))
target/i386/kvm/sw-protected-vm.c new file +35 -0
@@ -0,0 +1,35 @@ +/* +* QEMU X86_SW_PROTECTED_VM SUPPORT +* +* Author: +* Xiaoyao Li <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2 or later. +* See the COPYING file in the top-level directory +* +*/ + +#include "qemu/osdep.h" +#include "qom/object_interfaces.h" + +#include "sw-protected-vm.h" + +/* x86-sw-protected-vm */ +OBJECT_DEFINE_TYPE_WITH_INTERFACES (SwProtectedVm, + sw_protected_vm, +SW_PROTECTED_VM , +CONFIDENTIAL_GUEST_SUPPORT , + { TYPE_USER_CREATABLE }, + { NULL }) + +static void sw_protected_vm_init (Object *obj) +{ +} + +static void sw_protected_vm_finalize (Object *obj) +{ +} + +static void sw_protected_vm_class_init (ObjectClass *oc,void *data) +{ +}
target/i386/kvm/sw-protected-vm.h new file +17 -0
@@ -0,0 +1,17 @@ +#ifndef QEMU_I386_SW_PROTECTED_VM_H +#define QEMU_I386_SW_PROTECTED_VM_H + +#include "exec/confidential-guest-support.h" + +#define TYPE_SW_PROTECTED_VM "sw-protected-vm" +#define SW_PROTECTED_VM (obj) OBJECT_CHECK(SwProtectedVm, (obj), TYPE_SW_PROTECTED_VM) + +typedef struct SwProtectedVmClass { +ConfidentialGuestSupportClass parent_class; +}SwProtectedVmClass ; + +typedef struct SwProtectedVm { +ConfidentialGuestSupport parent_obj; +}SwProtectedVm ; + +#endif /* QEMU_I386_SW_PROTECTED_VM_H */
Signature · 1 line
2.34.1
RFC PATCH v2 7/21 i386/pc: Drop pc_machine_kvm_type()
In reply to Xiaoyao Lipc_machine_kvm_type() was introduced by commit e21be724eaf5 ("i386/xen:
add pc_machine_kvm_type to initialize XEN_EMULATE mode") to do Xen specific initialization by utilizing kvm_type method.
commit eeedfe6c6316 ("hw/xen: Simplify emulated Xen platform init") moves the Xen specific initialization to pc_basic_device_init().
There is no need to keep the PC specific kvm_type() implementation anymore. On the other hand, later patch will implement kvm_type() method for all x86/i386 machines to support KVM_X86_SW_PROTECTED_VM.
--- hw/i386/pc.c | 5 ----- include/hw/i386/pc.h | 3 --- 2 files changed, 8 deletions(-)
hw/i386/pc.c +0 -5
@@ -1794,11 +1794,6 @@ static void pc_machine_initfn(Object *obj)cxl_machine_init (obj, &pcms->cxl_devices_state); } -int pc_machine_kvm_type (MachineState *machine,const char *kvm_type) -{ -return 0 ; -} -static void pc_machine_reset (MachineState *machine,ShutdownCause reason) {CPUState *cs;
include/hw/i386/pc.h +0 -3
@@ -296,15 +296,12 @@ extern const size_t pc_compat_1_5_len;extern GlobalProperty pc_compat_1_4[];extern const size_t pc_compat_1_4_len; -int pc_machine_kvm_type (MachineState *machine,const char *vm_type); -#define DEFINE_PC_MACHINE (suffix, namestr, initfn, optsfn) \ static void pc_machine_##suffix##_class_init(ObjectClass *oc, void *data) \ { \ MachineClass *mc = MACHINE_CLASS(oc); \ optsfn(mc); \ mc->init = initfn; \ - mc->kvm_type = pc_machine_kvm_type; \ } \ static const TypeInfo pc_machine_type_##suffix = { \ .name = namestr TYPE_MACHINE_SUFFIX, \
Signature · 1 line
2.34.1
Re: [RFC PATCH v2 07/21] i386/pc: Drop pc_machine_kvm_type()
Quoted 9 lines · Xiaoyao Li
pc_machine_kvm_type()was introduced by commite21be724eaf5("i386/xen:
addpc_machine_kvm_typeto initializeXEN_EMULATEmode") to do Xen specific initialization by utilizingkvm_typemethod.commit
eeedfe6c6316("hw/xen: Simplify emulated Xen platform init") moves the Xen specific initialization topc_basic_device_init().There is no need to keep the PC specific
kvm_type()implementation anymore.
So we'll fallback to kvm_arch_get_default_type(), which simply returns 0.
Quoted 3 lines
On the other hand, later patch will implement
kvm_type()method for all x86/i386 machines to supportKVM_X86_SW_PROTECTED_VM.
^ I suggest dropping that and merging that patch ahead-of-time as a simple cleanup.
Signature · 3 lines
Cheers, David / dhildenb
Re: [RFC PATCH v2 07/21] i386/pc: Drop pc_machine_kvm_type()
Quoted 11 lines · David Hildenbrand
Quoted 9 lines · Xiaoyao Li
pc_machine_kvm_type()was introduced by commite21be724eaf5("i386/xen:
addpc_machine_kvm_typeto initializeXEN_EMULATEmode") to do Xen specific initialization by utilizingkvm_typemethod.commit
eeedfe6c6316("hw/xen: Simplify emulated Xen platform init") moves the Xen specific initialization topc_basic_device_init().There is no need to keep the PC specific
kvm_type()implementation anymore.
Quoted 7 lines
So we'll fallback to
kvm_arch_get_default_type(), which simply returns 0.Quoted 3 lines
On the other hand, later patch will implement
kvm_type()method for all x86/i386 machines to supportKVM_X86_SW_PROTECTED_VM.^ I suggest dropping that and merging that patch ahead-of-time as a simple cleanup.
I suppose the "that" here means "this patch", right?
If so, I can submit this patch separately.
Quoted 2 lines
Reviewed-by David Hildenbrand <[email protected]>
Re: [RFC PATCH v2 07/21] i386/pc: Drop pc_machine_kvm_type()
Quoted 14 lines · Xiaoyao Li
Quoted 13 lines · David Hildenbrand
Quoted 9 lines · Xiaoyao Li
pc_machine_kvm_type()was introduced by commite21be724eaf5("i386/xen:
addpc_machine_kvm_typeto initializeXEN_EMULATEmode") to do Xen specific initialization by utilizingkvm_typemethod.commit
eeedfe6c6316("hw/xen: Simplify emulated Xen platform init") moves the Xen specific initialization topc_basic_device_init().There is no need to keep the PC specific
kvm_type()implementation anymore.So we'll fallback to
kvm_arch_get_default_type(), which simply returns 0.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
On the other hand, later patch will implement
kvm_type()method for all x86/i386 machines to supportKVM_X86_SW_PROTECTED_VM.^ I suggest dropping that and merging that patch ahead-of-time as a simple cleanup.
I suppose the "that" here means "this patch", right?
With that I meant that paragraph "On the other hand, later patch ...".
Signature · 3 lines
Cheers, David / dhildenb
Re: [RFC PATCH v2 07/21] i386/pc: Drop pc_machine_kvm_type()
In reply to Xiaoyao LiQuoted 8 lines · Xiaoyao Li
pc_machine_kvm_type()was introduced by commite21be724eaf5("i386/xen:
addpc_machine_kvm_typeto initializeXEN_EMULATEmode") to do Xen specific initialization by utilizingkvm_typemethod.commit
eeedfe6c6316("hw/xen: Simplify emulated Xen platform init") moves the Xen specific initialization topc_basic_device_init().
Quoted 6 lines
There is no need to keep the PC specific
kvm_type()implementation anymore. On the other hand, later patch will implementkvm_type()method for all x86/i386 machines to supportKVM_X86_SW_PROTECTED_VM.Signed-off-by Xiaoyao Li <[email protected]>Reviewed-by Isaku Yamahata <[email protected]>
Indeed, I added it and then later ripped everything out of it and left it empty, as you nicely describe (thanks) in your commit message. I have no designs on using it again, so
RFC PATCH v2 8/21 target/i386: Implement mc->kvm_type() to get VM type
In reply to Xiaoyao LiImplement mc->kvm_type() for i386 machines. It provides a way for user to create SW_PROTECTE_VM.
Also store the vm_type in machinestate to other code to query what the VM type is.
--- hw/i386/x86.c | 12 ++++++++++++ include/hw/i386/x86.h | 1 + target/i386/kvm/kvm.c | 30 ++++++++++++++++++++++++++++++ target/i386/kvm/kvm_i386.h | 1 + 4 files changed, 44 insertions(+)
hw/i386/x86.c +12 -0
@@ -1382,6 +1382,17 @@ static void machine_set_sgx_epc(Object *obj, Visitor *v, const char *name,qapi_free_SgxEPCList (list); } +static int x86_kvm_type (MachineState *ms,const char *vm_type) +{ +X86MachineState *x86ms =X86_MACHINE (ms); +int kvm_type; + + kvm_type =kvm_get_vm_type (ms, vm_type); + x86ms->vm_type = kvm_type; + +return kvm_type; +} +static void x86_machine_initfn (Object *obj) {X86MachineState *x86ms =X86_MACHINE (obj);
@@ -1406,6 +1417,7 @@ static void x86_machine_class_init(ObjectClass *oc, void *data) mc->cpu_index_to_instance_props = x86_cpu_index_to_props; mc->get_default_cpu_node_id = x86_get_default_cpu_node_id; mc->possible_cpu_arch_ids = x86_possible_cpu_arch_ids; + mc->kvm_type = x86_kvm_type; x86mc->save_tsc_khz = true; x86mc->fwcfg_dma_enabled = true; nc->nmi_monitor_handler = x86_nmi;
include/hw/i386/x86.h +1 -0
@@ -41,6 +41,7 @@ struct X86MachineState {MachineState parent;/*< public >*/ +unsigned int vm_type;/* Pointers to devices and objects: */ ISADevice *rtc;
target/i386/kvm/kvm.c +30 -0
@@ -32,6 +32,7 @@#include "sysemu/runstate.h" #include "kvm_i386.h" #include "sev.h" +#include "sw-protected-vm.h" #include "xen-emu.h" #include "hyperv.h" #include "hyperv-proto.h"
@@ -154,6 +155,35 @@ static KVMMSRHandlers msr_handlers[KVM_MSR_FILTER_MAX_RANGES];static RateLimit bus_lock_ratelimit_ctrl;static int kvm_get_one_msr (X86CPU *cpu,int index,uint64_t *value); +static const char * vm_type_name[] = { + [KVM_X86_DEFAULT_VM ] ="default" , + [KVM_X86_SW_PROTECTED_VM ] ="sw-protected-vm" , +}; + +int kvm_get_vm_type (MachineState *ms,const char *vm_type) +{ +int kvm_type =KVM_X86_DEFAULT_VM ; + +if (ms->cgs &&object_dynamic_cast (OBJECT (ms->cgs),TYPE_SW_PROTECTED_VM )) { + kvm_type =KVM_X86_SW_PROTECTED_VM ; + } + +/* +* old KVM doesn't support KVM_CAP_VM_TYPES and KVM_X86_DEFAULT_VM +* is always supported +*/ +if (kvm_type ==KVM_X86_DEFAULT_VM ) { +return kvm_type; + } + +if (!(kvm_check_extension (KVM_STATE (ms->accelerator),KVM_CAP_VM_TYPES ) &BIT (kvm_type))) { +error_report ("vm-type %s not supported by KVM" , vm_type_name[kvm_type]); +exit (1 ); + } + +return kvm_type; +} +int kvm_has_pit_state2 (void ) {return has_pit_state2;
target/i386/kvm/kvm_i386.h +1 -0
@@ -37,6 +37,7 @@ bool kvm_has_adjust_clock(void);bool kvm_has_adjust_clock_stable (void );bool kvm_has_exception_payload (void );void kvm_synchronize_all_tsc (void ); +int kvm_get_vm_type (MachineState *ms,const char *vm_type);void kvm_arch_reset_vcpu (X86CPU *cs);void kvm_arch_after_reset_vcpu (X86CPU *cpu);void kvm_arch_do_init_vcpu (X86CPU *cs);
Signature · 1 line
2.34.1
RFC PATCH v2 9/21 target/i386: Introduce kvm_confidential_guest_init()
In reply to Xiaoyao LiIntroduce a separate function kvm_confidential_guest_init(), which dispatches specific confidential guest initialization function by ms->cgs type.
--- target/i386/kvm/kvm.c | 11 ++++++++++- target/i386/sev.c | 1 - target/i386/sev.h | 2 ++ 3 files changed, 12 insertions(+), 2 deletions(-)
target/i386/kvm/kvm.c +10 -1
@@ -2583,6 +2583,15 @@ static void register_smram_listener(Notifier *n, void *unused) &smram_address_space, 1, "kvm-smram"); } +static int kvm_confidential_guest_init (MachineState *ms,Error **errp) +{ +if (object_dynamic_cast (OBJECT (ms->cgs),TYPE_SEV_GUEST )) { +return sev_kvm_init (ms->cgs, errp); + } + +return 0 ; +} +int kvm_arch_init (MachineState *ms,KVMState *s) {uint64_t identity_base =0xfffbc000 ;
@@ -2603,7 +2612,7 @@ int kvm_arch_init(MachineState *ms, KVMState *s)* mechanisms are supported in future (e.g. TDX), they'll need * their own initialization either here or elsewhere. */ - ret =sev_kvm_init (ms->cgs, &local_err); + ret =kvm_confidential_guest_init (ms, &local_err);if (ret <0 ) {error_report_err (local_err);return ret;
target/i386/sev.c +0 -1
@@ -39,7 +39,6 @@#include "hw/i386/pc.h" #include "exec/address-spaces.h" -#define TYPE_SEV_GUEST "sev-guest"OBJECT_DECLARE_SIMPLE_TYPE (SevGuestState,SEV_GUEST )
target/i386/sev.h +2 -0
@@ -20,6 +20,8 @@#include "exec/confidential-guest-support.h" +#define TYPE_SEV_GUEST "sev-guest" +#define SEV_POLICY_NODBG 0x1#define SEV_POLICY_NOKS 0x2#define SEV_POLICY_ES 0x4
Signature · 1 line
2.34.1
RFC PATCH v2 10/21 i386/kvm: Implement kvm_sw_protected_vm_init() for sw-protcted-vm specific functions
In reply to Xiaoyao Li--- target/i386/kvm/kvm.c | 2 ++ target/i386/kvm/sw-protected-vm.c | 10 ++++++++++ target/i386/kvm/sw-protected-vm.h | 2 ++ 3 files changed, 14 insertions(+)
target/i386/kvm/kvm.c +2 -0
@@ -2587,6 +2587,8 @@ static int kvm_confidential_guest_init(MachineState *ms, Error **errp) {if (object_dynamic_cast (OBJECT (ms->cgs),TYPE_SEV_GUEST )) {return sev_kvm_init (ms->cgs, errp); + }else if (object_dynamic_cast (OBJECT (ms->cgs),TYPE_SW_PROTECTED_VM )) { +return sw_protected_vm_kvm_init (ms, errp); }return 0 ;
target/i386/kvm/sw-protected-vm.c +10 -0
@@ -10,10 +10,20 @@*/ #include "qemu/osdep.h" +#include "qapi/error.h" #include "qom/object_interfaces.h" +#include "hw/i386/x86.h" #include "sw-protected-vm.h" +int sw_protected_vm_kvm_init (MachineState *ms,Error **errp) +{ +SwProtectedVm *spvm =SW_PROTECTED_VM (OBJECT (ms->cgs)); + + spvm->parent_obj.ready = true; +return 0 ; +} +/* x86-sw-protected-vm */ OBJECT_DEFINE_TYPE_WITH_INTERFACES(SwProtectedVm, sw_protected_vm,
target/i386/kvm/sw-protected-vm.h +2 -0
@@ -14,4 +14,6 @@ typedef struct SwProtectedVm { ConfidentialGuestSupport parent_obj; } SwProtectedVm; +int sw_protected_vm_kvm_init (MachineState *ms,Error **errp); +#endif /* QEMU_I386_SW_PROTECTED_VM_H */
Signature · 1 line
2.34.1
RFC PATCH v2 11/21 kvm: Introduce support for memory_attributes
In reply to Xiaoyao LiIntrocude the helper functions to set the attributes of a range of memory to private and shared.
--- accel/kvm/kvm-all.c | 43 +++++++++++++++++++++++++++++++++++++++++++ include/sysemu/kvm.h | 3 +++ 2 files changed, 46 insertions(+)
accel/kvm/kvm-all.c +43 -0
@@ -105,6 +105,7 @@ bool kvm_msi_use_devid;bool kvm_has_guest_debug;static int kvm_sstep_flags;static bool kvm_immediate_exit; +static uint64_t kvm_supported_memory_attributes;static hwaddr kvm_max_slot_size = ~0 ;static const KVMCapabilityInfo kvm_required_capabilites[] = {
@@ -1343,6 +1344,44 @@ void kvm_set_max_memslot_size(hwaddr max_slot_size) kvm_max_slot_size = max_slot_size; } +static int kvm_set_memory_attributes (hwaddr start,hwaddr size,uint64_t attr) +{ +struct kvm_memory_attributes attrs; +int r; + + attrs.attributes = attr; + attrs.address = start; + attrs.size = size; + attrs.flags =0 ; + + r =kvm_vm_ioctl (kvm_state,KVM_SET_MEMORY_ATTRIBUTES , &attrs); +if (r) { +warn_report ("%s: failed to set memory (0x%lx+%#zx) with attr 0x%lx error '%s'" , + __func__, start, size, attr,strerror (errno)); + } +return r; +} + +int kvm_set_memory_attributes_private (hwaddr start,hwaddr size) +{ +if (!(kvm_supported_memory_attributes &KVM_MEMORY_ATTRIBUTE_PRIVATE )) { +error_report ("KVM doesn't support PRIVATE memory attribute\n" ); +return -EINVAL ; + } + +return kvm_set_memory_attributes (start, size,KVM_MEMORY_ATTRIBUTE_PRIVATE ); +} + +int kvm_set_memory_attributes_shared (hwaddr start,hwaddr size) +{ +if (!(kvm_supported_memory_attributes &KVM_MEMORY_ATTRIBUTE_PRIVATE )) { +error_report ("KVM doesn't support PRIVATE memory attribute\n" ); +return -EINVAL ; + } + +return kvm_set_memory_attributes (start, size,0 ); +} +/* Called with KVMMemoryListener.slots_lock held */ static void kvm_set_phys_mem (KVMMemoryListener *kml,MemoryRegionSection *section,bool add)
@@ -2556,6 +2595,10 @@ static int kvm_init(MachineState *ms) } s->as = g_new0(struct KVMAs, s->nr_as); +if (kvm_check_extension (s,KVM_CAP_MEMORY_ATTRIBUTES )) { + kvm_supported_memory_attributes =kvm_ioctl (s,KVM_GET_SUPPORTED_MEMORY_ATTRIBUTES ,0 ); + } +if (object_property_find (OBJECT (current_machine),"kvm-type" )) { g_autofree char *kvm_type = object_property_get_str(OBJECT(current_machine), "kvm-type",
include/sysemu/kvm.h +3 -0
@@ -582,4 +582,7 @@ bool kvm_dirty_ring_enabled(void);uint32_t kvm_dirty_ring_size (void );int kvm_create_guest_memfd (uint64_t size,uint64_t flags,Error **errp); + +int kvm_set_memory_attributes_private (hwaddr start,hwaddr size); +int kvm_set_memory_attributes_shared (hwaddr start,hwaddr size);#endif
Signature · 1 line
2.34.1
RFC PATCH v2 12/21 kvm/memory: Introduce the infrastructure to set the default shared/private value
In reply to Xiaoyao LiIntroduce new flag RAM_DEFAULT_PRIVATE for RAMBlock. It's used to indicate the default attribute, private or not.
Set the RAM range to private explicitly when it's default private.
Originated-from: Isaku Yamahata <[email protected]>
--- accel/kvm/kvm-all.c | 10 ++++++++++ include/exec/memory.h | 6 ++++++ softmmu/memory.c | 13 +++++++++++++ 3 files changed, 29 insertions(+)
accel/kvm/kvm-all.c +10 -0
@@ -1487,6 +1487,16 @@ static void kvm_set_phys_mem(KVMMemoryListener *kml, strerror(-err));abort (); } + +if (memory_region_is_default_private (mr)) { + err =kvm_set_memory_attributes_private (start_addr, slot_size); +if (err) { +error_report ("%s: failed to set memory attribute private: %s\n" , + __func__,strerror (-err)); +exit (1 ); + } + } + start_addr += slot_size; ram_start_offset += slot_size; ram += slot_size;
include/exec/memory.h +6 -0
@@ -238,6 +238,9 @@ typedef struct IOMMUTLBEvent {/* RAM can be private that has kvm gmem backend */ #define RAM_KVM_GMEM (1 << 10) +/* RAM is default private */ +#define RAM_DEFAULT_PRIVATE (1 << 11) + static inline void iommu_notifier_init(IOMMUNotifier *n, IOMMUNotify fn, IOMMUNotifierFlag flags, hwaddr start, hwaddr end,
@@ -1684,6 +1687,9 @@ bool memory_region_is_protected(MemoryRegion *mr);*/ bool memory_region_has_gmem_fd (MemoryRegion *mr); +void memory_region_set_default_private (MemoryRegion *mr); +bool memory_region_is_default_private (MemoryRegion *mr); +/** * memory_region_get_iommu: check whether a memory region is an iommu *
softmmu/memory.c +13 -0
@@ -1851,6 +1851,19 @@ bool memory_region_has_gmem_fd(MemoryRegion *mr)return mr->ram_block && mr->ram_block->gmem_fd >=0 ; } +bool memory_region_is_default_private (MemoryRegion *mr) +{ +return memory_region_has_gmem_fd (mr) && + (mr->ram_block->flags &RAM_DEFAULT_PRIVATE ); +} + +void memory_region_set_default_private (MemoryRegion *mr) +{ +if (memory_region_has_gmem_fd (mr)) { + mr->ram_block->flags |=RAM_DEFAULT_PRIVATE ; + } +} +uint8_t memory_region_get_dirty_log_mask (MemoryRegion *mr) {uint8_t mask = mr->dirty_log_mask;
Signature · 1 line
2.34.1
RFC PATCH v2 13/21 i386/kvm: Set memory to default private for KVM_X86_SW_PROTECTED_VM
In reply to Xiaoyao LiRegister a memory listener for KVM_X86_SW_PROVTED_VM. It set RAM to private by default.
--- include/exec/memory.h | 1 + target/i386/kvm/sw-protected-vm.c | 18 ++++++++++++++++++ 2 files changed, 19 insertions(+)
include/exec/memory.h +1 -0
@@ -820,6 +820,7 @@ struct IOMMUMemoryRegion {#define MEMORY_LISTENER_PRIORITY_MIN 0#define MEMORY_LISTENER_PRIORITY_ACCEL 10#define MEMORY_LISTENER_PRIORITY_DEV_BACKEND 10 +#define MEMORY_LISTENER_PRIORITY_ACCEL_HIGH 20/** * struct MemoryListener: callbacks structure for updates to the physical memory map
target/i386/kvm/sw-protected-vm.c +18 -0
@@ -12,14 +12,32 @@#include "qemu/osdep.h" #include "qapi/error.h" #include "qom/object_interfaces.h" +#include "exec/address-spaces.h" +#include "sysemu/kvm.h" #include "hw/i386/x86.h" #include "sw-protected-vm.h" +static void kvm_x86_sw_protected_vm_region_add (MemoryListener *listenr, +MemoryRegionSection *section) +{ +memory_region_set_default_private (section->mr); +} + +static MemoryListener kvm_x86_sw_protected_vm_memory_listener = { + .name ="kvm_x86_sw_protected_vm_memory_listener" , + .region_add = kvm_x86_sw_protected_vm_region_add, +/* Higher than KVM memory listener = 10. */ + .priority =MEMORY_LISTENER_PRIORITY_ACCEL_HIGH , +}; +int sw_protected_vm_kvm_init (MachineState *ms,Error **errp) {SwProtectedVm *spvm =SW_PROTECTED_VM (OBJECT (ms->cgs)); +memory_listener_register (&kvm_x86_sw_protected_vm_memory_listener, + &address_space_memory); + spvm->parent_obj.ready = true;return 0 ; }
Signature · 1 line
2.34.1
RFC PATCH v2 14/21 physmem: replace function name with __func__ in ram_block_discard_range()
In reply to Xiaoyao Li--- softmmu/physmem.c | 34 +++++++++++++++------------------- 1 file changed, 15 insertions(+), 19 deletions(-)
softmmu/physmem.c +15 -19
@@ -3440,16 +3440,15 @@ int ram_block_discard_range(RAMBlock *rb, uint64_t start, size_t length)uint8_t *host_startaddr = rb->host + start;if (!QEMU_PTR_IS_ALIGNED (host_startaddr, rb->page_size)) { -error_report ("ram_block_discard_range: Unaligned start address: %p" , - host_startaddr); +error_report ("%s: Unaligned start address: %p" , + __func__, host_startaddr); goto err; }if ((start + length) <= rb->max_length) {bool need_madvise, need_fallocate;if (!QEMU_IS_ALIGNED (length, rb->page_size)) { -error_report ("ram_block_discard_range: Unaligned length: %zx" , - length); +error_report ("%s: Unaligned length: %zx" , __func__, length); goto err; }
@@ -3479,27 +3478,26 @@ int ram_block_discard_range(RAMBlock *rb, uint64_t start, size_t length)* file. */ if (!qemu_ram_is_shared (rb)) { -warn_report_once ("ram_block_discard_range: Discarding RAM" +warn_report_once ("%s: Discarding RAM" " in private file mappings is possibly" " dangerous, because it will modify the" " underlying file and will affect other" -" users of the file" ); +" users of the file" , __func__); } ret =fallocate (rb->fd,FALLOC_FL_PUNCH_HOLE |FALLOC_FL_KEEP_SIZE , start, length);if (ret) { ret = -errno; -error_report ("ram_block_discard_range: Failed to fallocate " -"%s:%" PRIx64" +%zx (%d)" , - rb->idstr, start, length, ret); +error_report ("%s: Failed to fallocate %s:%" PRIx64" +%zx (%d)" , + __func__, rb->idstr, start, length, ret); goto err; }#else ret = -ENOSYS ; -error_report ("ram_block_discard_range: fallocate not available/file" +error_report ("%s: fallocate not available/file" "%s:%" PRIx64" +%zx (%d)" , - rb->idstr, start, length, ret); + __func__, rb->idstr, start, length, ret); goto err;#endif }
@@ -3517,25 +3515,23 @@ int ram_block_discard_range(RAMBlock *rb, uint64_t start, size_t length) }if (ret) { ret = -errno; -error_report ("ram_block_discard_range: Failed to discard range " +error_report ("%s: Failed to discard range " "%s:%" PRIx64" +%zx (%d)" , - rb->idstr, start, length, ret); + __func__, rb->idstr, start, length, ret); goto err; }#else ret = -ENOSYS ; -error_report ("ram_block_discard_range: MADVISE not available" -"%s:%" PRIx64" +%zx (%d)" , - rb->idstr, start, length, ret); +error_report ("%s: MADVISE not available %s:%" PRIx64" +%zx (%d)" , + __func__, rb->idstr, start, length, ret); goto err;#endif }trace_ram_block_discard_range (rb->idstr, host_startaddr, length, need_madvise, need_fallocate, ret); }else { -error_report ("ram_block_discard_range: Overrun block '%s' (%" PRIu64 -"/%zx/" RAM_ADDR_FMT ")" , - rb->idstr, start, length, rb->max_length); +error_report ("%s: Overrun block '%s' (%" PRIu64"/%zx/" RAM_ADDR_FMT ")" , + __func__, rb->idstr, start, length, rb->max_length); } err:
Signature · 1 line
2.34.1
RFC PATCH v2 15/21 physmem: extract ram_block_discard_range_fd() from ram_block_discard_range()
In reply to Xiaoyao LiExtract the alignment check and sanity check out from ram_block_discard_range() into a seperate function ram_block_discard_range_fd(), which can be passed with an explicit fd as input parameter.
ram_block_discard_range_fd() can be used to discard private memory range from gmem fd with later patch. When doing private memory <-> shared memory conversion, it requires 4KB alignment instead of RamBlock.page_size.
--- softmmu/physmem.c | 192 ++++++++++++++++++++++++---------------------- 1 file changed, 100 insertions(+), 92 deletions(-)
softmmu/physmem.c +100 -92
@@ -3425,117 +3425,125 @@ int qemu_ram_foreach_block(RAMBlockIterFunc func, void *opaque)return ret; } +static int ram_block_discard_range_fd (RAMBlock *rb,uint64_t start, +size_t length,int fd) +{ +uint8_t *host_startaddr = rb->host + start; +bool need_madvise, need_fallocate; +int ret =-1 ; + + errno =ENOTSUP ;/* If we are missing MADVISE etc */ + +/* The logic here is messy; +* madvise DONTNEED fails for hugepages +* fallocate works on hugepages and shmem +* shared anonymous memory requires madvise REMOVE +*/ + need_madvise = (rb->page_size == qemu_host_page_size) && (rb->fd == fd); + need_fallocate = fd !=-1 ; + +if (need_fallocate) { +/* For a file, this causes the area of the file to be zero'd +* if read, and for hugetlbfs also causes it to be unmapped +* so a userfault will trigger. +*/ +#ifdef CONFIG_FALLOCATE_PUNCH_HOLE +/* +* We'll discard data from the actual file, even though we only +* have a MAP_PRIVATE mapping, possibly messing with other +* MAP_PRIVATE/MAP_SHARED mappings. There is no easy way to +* change that behavior whithout violating the promised +* semantics of ram_block_discard_range(). +* +* Only warn, because it works as long as nobody else uses that +* file. +*/ +if (!qemu_ram_is_shared (rb)) { +warn_report_once ("%s: Discarding RAM" +" in private file mappings is possibly" +" dangerous, because it will modify the" +" underlying file and will affect other" +" users of the file" , __func__); + } + + ret =fallocate (fd,FALLOC_FL_PUNCH_HOLE |FALLOC_FL_KEEP_SIZE , + start, length); +if (ret) { + ret = -errno; +error_report ("%s: Failed to fallocate %s:%" PRIx64" +%zx (%d)" , + __func__, rb->idstr, start, length, ret); +return ret; + } +#else + ret = -ENOSYS ; +error_report ("%s: fallocate not available/file " +"%s:%" PRIx64" +%zx (%d)" , + __func__, rb->idstr, start, length, ret); +return ret; +#endif + } + +if (need_madvise) { +/* For normal RAM this causes it to be unmapped, +* for shared memory it causes the local mapping to disappear +* and to fall back on the file contents (which we just +* fallocate'd away). +*/ +#if defined(CONFIG_MADVISE ) +if (qemu_ram_is_shared (rb) && fd <0 ) { + ret =madvise (host_startaddr, length,QEMU_MADV_REMOVE ); + }else { + ret =madvise (host_startaddr, length,QEMU_MADV_DONTNEED ); + } +if (ret) { + ret = -errno; +error_report ("%s: Failed to discard range %s:%" PRIx64" +%zx (%d)" , + __func__, rb->idstr, start, length, ret); +return ret; + } +#else + ret = -ENOSYS ; +error_report ("%s: MADVISE not available %s:%" PRIx64" +%zx (%d)" , + __func__, rb->idstr, start, length, ret); +return ret; +#endif + } + +trace_ram_block_discard_range (rb->idstr, host_startaddr, length, + need_madvise, need_fallocate, ret); +return ret; +} +/* * Unmap pages of memory from start to start+length such that * they a) read as 0, b) Trigger whatever fault mechanism * the OS provides for postcopy. +* * The pages must be unmapped by the end of the function. -* Returns: 0 on success, none-0 on failure -* +* Returns: 0 on success, none-0 on failure. */ int ram_block_discard_range (RAMBlock *rb,uint64_t start,size_t length) { -int ret =-1 ; -uint8_t *host_startaddr = rb->host + start;if (!QEMU_PTR_IS_ALIGNED (host_startaddr, rb->page_size)) {error_report ("%s: Unaligned start address: %p" , __func__, host_startaddr); - goto err; +return -1 ; } -if ((start + length) <= rb->max_length) { -bool need_madvise, need_fallocate; -if (!QEMU_IS_ALIGNED (length, rb->page_size)) { -error_report ("%s: Unaligned length: %zx" , __func__, length); - goto err; - } - - errno =ENOTSUP ;/* If we are missing MADVISE etc */ - -/* The logic here is messy; -* madvise DONTNEED fails for hugepages -* fallocate works on hugepages and shmem -* shared anonymous memory requires madvise REMOVE -*/ - need_madvise = (rb->page_size == qemu_host_page_size); - need_fallocate = rb->fd !=-1 ; -if (need_fallocate) { -/* For a file, this causes the area of the file to be zero'd -* if read, and for hugetlbfs also causes it to be unmapped -* so a userfault will trigger. -*/ -#ifdef CONFIG_FALLOCATE_PUNCH_HOLE -/* -* We'll discard data from the actual file, even though we only -* have a MAP_PRIVATE mapping, possibly messing with other -* MAP_PRIVATE/MAP_SHARED mappings. There is no easy way to -* change that behavior whithout violating the promised -* semantics of ram_block_discard_range(). -* -* Only warn, because it works as long as nobody else uses that -* file. -*/ -if (!qemu_ram_is_shared (rb)) { -warn_report_once ("%s: Discarding RAM" -" in private file mappings is possibly" -" dangerous, because it will modify the" -" underlying file and will affect other" -" users of the file" , __func__); - } - - ret =fallocate (rb->fd,FALLOC_FL_PUNCH_HOLE |FALLOC_FL_KEEP_SIZE , - start, length); -if (ret) { - ret = -errno; -error_report ("%s: Failed to fallocate %s:%" PRIx64" +%zx (%d)" , - __func__, rb->idstr, start, length, ret); - goto err; - } -#else - ret = -ENOSYS ; -error_report ("%s: fallocate not available/file" -"%s:%" PRIx64" +%zx (%d)" , - __func__, rb->idstr, start, length, ret); - goto err; -#endif - } -if (need_madvise) { -/* For normal RAM this causes it to be unmapped, -* for shared memory it causes the local mapping to disappear -* and to fall back on the file contents (which we just -* fallocate'd away). -*/ -#if defined(CONFIG_MADVISE ) -if (qemu_ram_is_shared (rb) && rb->fd <0 ) { - ret =madvise (host_startaddr, length,QEMU_MADV_REMOVE ); - }else { - ret =madvise (host_startaddr, length,QEMU_MADV_DONTNEED ); - } -if (ret) { - ret = -errno; -error_report ("%s: Failed to discard range " -"%s:%" PRIx64" +%zx (%d)" , - __func__, rb->idstr, start, length, ret); - goto err; - } -#else - ret = -ENOSYS ; -error_report ("%s: MADVISE not available %s:%" PRIx64" +%zx (%d)" , - __func__, rb->idstr, start, length, ret); - goto err; -#endif - } -trace_ram_block_discard_range (rb->idstr, host_startaddr, length, - need_madvise, need_fallocate, ret); - }else { +if ((start + length) > rb->max_length) {error_report ("%s: Overrun block '%s' (%" PRIu64"/%zx/" RAM_ADDR_FMT ")" , __func__, rb->idstr, start, length, rb->max_length); +return -1 ; } -err: -return ret; +if (!QEMU_IS_ALIGNED (length, rb->page_size)) { +error_report ("%s: Unaligned length: %zx" , __func__, length); +return -1 ; + } + +return ram_block_discard_range_fd (rb, start, length, rb->fd); }bool ramblock_is_pmem (RAMBlock *rb)
Signature · 1 line
2.34.1
RFC PATCH v2 16/21 physmem: Introduce ram_block_convert_range()
In reply to Xiaoyao LiIt's used for discarding oppsite memory after memory conversion to shared/private.
Note, private-shared page conversion is done at 4KB granularity. Don't check alignment with rb->page_size, instead qemu_host_page_size, which is 4K.
Originally-from: Isaku Yamahata <[email protected]>
--- include/exec/cpu-common.h | 2 ++ softmmu/physmem.c | 36 ++++++++++++++++++++++++++++++++++++ 2 files changed, 38 insertions(+)
include/exec/cpu-common.h +2 -0
@@ -157,6 +157,8 @@ typedef int (RAMBlockIterFunc)(RAMBlock *rb, void *opaque);int qemu_ram_foreach_block (RAMBlockIterFunc func,void *opaque);int ram_block_discard_range (RAMBlock *rb,uint64_t start,size_t length); +int ram_block_convert_range (RAMBlock *rb,uint64_t start,size_t length, +bool shared_to_private);#endif
softmmu/physmem.c +36 -0
@@ -3733,3 +3733,39 @@ bool ram_block_discard_is_required(void)return qatomic_read (&ram_block_discard_required_cnt) ||qatomic_read (&ram_block_coordinated_discard_required_cnt); } + +int ram_block_convert_range (RAMBlock *rb,uint64_t start,size_t length, +bool shared_to_private) +{ +int fd; + +if (!rb || rb->gmem_fd <0 ) { +return -1 ; + } + +if (!QEMU_PTR_IS_ALIGNED (start, qemu_host_page_size) || + !QEMU_PTR_IS_ALIGNED (length, qemu_host_page_size)) { +return -1 ; + } + +if (!length) { +return -1 ; + } + +if (start + length > rb->max_length) { +return -1 ; + } + +if (shared_to_private) { +void *host_startaddr = rb->host + start; + +if (!QEMU_PTR_IS_ALIGNED (host_startaddr, qemu_host_page_size)) { +return -1 ; + } + fd = rb->fd; + }else { + fd = rb->gmem_fd; + } + +return ram_block_discard_range_fd (rb, start, length, fd); +}
Signature · 1 line
2.34.1
RFC PATCH v2 17/21 kvm: handle KVM_EXIT_MEMORY_FAULT
In reply to Xiaoyao LiFrom: Chao Peng <[email protected]>
Currently only KVM_MEMORY_EXIT_FLAG_PRIVATE in flags is valid when KVM_EXIT_MEMORY_FAULT happens. It indicates userspace needs to do the memory conversion on the RAMBlock to turn the memory into desired attribute, i.e., private/shared.
Note, KVM_EXIT_MEMORY_FAULT makes sense only when the RAMBlock has gmem memory backend.
--- accel/kvm/kvm-all.c | 54 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+)
accel/kvm/kvm-all.c +54 -0
@@ -3040,6 +3040,50 @@ static void kvm_eat_signals(CPUState *cpu) }while (sigismember (&chkset,SIG_IPI )); } +static int kvm_convert_memory (hwaddr start,hwaddr size,bool to_private) +{ +MemoryRegionSection section; +ram_addr_t offset; +RAMBlock *rb; +void *addr; +int ret =-1 ; + + section =memory_region_find (get_system_memory (), start, size); +if (!section.mr) { +return ret; + } + +if (memory_region_has_gmem_fd (section.mr)) { +if (to_private) { + ret =kvm_set_memory_attributes_private (start, size); + }else { + ret =kvm_set_memory_attributes_shared (start, size); + } + +if (ret) { +memory_region_unref (section.mr); +return ret; + } + + addr =memory_region_get_ram_ptr (section.mr) + + section.offset_within_region; + rb =qemu_ram_block_from_host (addr, false, &offset); +/* +* With KVM_SET_MEMORY_ATTRIBUTES by kvm_set_memory_attributes(), +* operation on underlying file descriptor is only for releasing +* unnecessary pages. +*/ +ram_block_convert_range (rb, offset, size, to_private); + }else { +warn_report ("Convert non guest-memfd backed memory region " +"(0x%" HWADDR_PRIx" ,+ 0x%" HWADDR_PRIx") to %s" , + start, size, to_private ?"private" :"shared" ); + } + +memory_region_unref (section.mr); +return ret; +} +int kvm_cpu_exec (CPUState *cpu) {struct kvm_run *run = cpu->kvm_run;
@@ -3198,6 +3242,16 @@ int kvm_cpu_exec(CPUState *cpu)break ; }break ; +case KVM_EXIT_MEMORY_FAULT : +if (run->memory.flags & ~KVM_MEMORY_EXIT_FLAG_PRIVATE ) { +error_report ("KVM_EXIT_MEMORY_FAULT: Unknown flag 0x%" PRIx64, + (uint64_t )run->memory.flags); + ret =-1 ; +break ; + } + ret =kvm_convert_memory (run->memory.gpa, run->memory.size, + run->memory.flags &KVM_MEMORY_EXIT_FLAG_PRIVATE ); +break ;default :DPRINTF ("kvm_arch_handle_exit\n" ); ret =kvm_arch_handle_exit (cpu, run);
Signature · 1 line
2.34.1
RFC PATCH v2 18/21 trace/kvm: Add trace for page convertion between shared and private
In reply to Xiaoyao LiFrom: Isaku Yamahata <[email protected]>
--- accel/kvm/kvm-all.c | 1 + accel/kvm/trace-events | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-)
accel/kvm/kvm-all.c +1 -0
@@ -3048,6 +3048,7 @@ static int kvm_convert_memory(hwaddr start, hwaddr size, bool to_private)void *addr;int ret =-1 ; +trace_kvm_convert_memory (start, size, to_private ?"shared_to_private" :"private_to_shared" ); section =memory_region_find (get_system_memory (), start, size);if (!section.mr) {return ret;
accel/kvm/trace-events +1 -1
@@ -25,4 +25,4 @@ kvm_dirty_ring_reaper(const char *s) "%s" kvm_dirty_ring_reap(uint64_t count, int64_t t) "reaped %"PRIu64" pages (took %"PRIi64" us)" kvm_dirty_ring_reaper_kick(const char *reason) "%s" kvm_dirty_ring_flush(int finished) "%d" - +kvm_convert_memory(uint64_t start, uint64_t size, const char *msg) "start 0x%" PRIx64 " size 0x%" PRIx64 " %s"
Signature · 1 line
2.34.1
RFC PATCH v2 19/21 pci-host/q35: Move PAM initialization above SMRAM initialization
In reply to Xiaoyao LiFrom: Isaku Yamahata <[email protected]>
In mch_realize(), process PAM initialization before SMRAM initialization so that later patch can skill all the SMRAM related with a single check.
--- hw/pci-host/q35.c | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-)
hw/pci-host/q35.c +10 -9
@@ -575,6 +575,16 @@ static void mch_realize(PCIDevice *d, Error **errp)/* setup pci memory mapping */ pc_pci_as_mapping_init (mch->system_memory, mch->pci_address_space); +/* PAM */ +init_pam (&mch->pam_regions[0 ],OBJECT (mch), mch->ram_memory, + mch->system_memory, mch->pci_address_space, +PAM_BIOS_BASE ,PAM_BIOS_SIZE ); +for (i =0 ; i <ARRAY_SIZE (mch->pam_regions) -1 ; ++i) { +init_pam (&mch->pam_regions[i +1 ],OBJECT (mch), mch->ram_memory, + mch->system_memory, mch->pci_address_space, +PAM_EXPAN_BASE + i *PAM_EXPAN_SIZE ,PAM_EXPAN_SIZE ); + } +/* if *disabled* show SMRAM to all CPUs */ memory_region_init_alias(&mch->smram_region, OBJECT(mch), "smram-region", mch->pci_address_space, MCH_HOST_BRIDGE_SMRAM_C_BASE,
@@ -641,15 +651,6 @@ static void mch_realize(PCIDevice *d, Error **errp)object_property_add_const_link (qdev_get_machine (),"smram" ,OBJECT (&mch->smram)); - -init_pam (&mch->pam_regions[0 ],OBJECT (mch), mch->ram_memory, - mch->system_memory, mch->pci_address_space, -PAM_BIOS_BASE ,PAM_BIOS_SIZE ); -for (i =0 ; i <ARRAY_SIZE (mch->pam_regions) -1 ; ++i) { -init_pam (&mch->pam_regions[i +1 ],OBJECT (mch), mch->ram_memory, - mch->system_memory, mch->pci_address_space, -PAM_EXPAN_BASE + i *PAM_EXPAN_SIZE ,PAM_EXPAN_SIZE ); - } }uint64_t mch_mcfg_base (void )
Signature · 1 line
2.34.1
RFC PATCH v2 20/21 q35: Introduce smm_ranges property for q35-pci-host
In reply to Xiaoyao LiFrom: Isaku Yamahata <[email protected]>
Add a q35 property to check whether or not SMM ranges, e.g. SMRAM, TSEG, etc... exist for the target platform. TDX doesn't support SMM and doesn't play nice with QEMU modifying related guest memory ranges.
--- hw/i386/pc_q35.c | 3 ++- hw/pci-host/q35.c | 42 +++++++++++++++++++++++++++------------ include/hw/i386/pc.h | 1 + include/hw/pci-host/q35.h | 1 + 4 files changed, 33 insertions(+), 14 deletions(-)
hw/i386/pc_q35.c +2 -1
@@ -232,8 +232,9 @@ static void pc_q35_init(MachineState *machine) x86ms->above_4g_mem_size, NULL);object_property_set_bool (phb,PCI_HOST_BYPASS_IOMMU , pcms->default_bus_bypass_iommu,NULL ); +object_property_set_bool (phb,PCI_HOST_PROP_SMM_RANGES , +x86_machine_is_smm_enabled (x86ms),NULL );sysbus_realize_and_unref (SYS_BUS_DEVICE (phb), &error_fatal); -/* pci */ host_bus =PCI_BUS (qdev_get_child_bus (DEVICE (phb),"pcie.0" )); pcms->bus = host_bus;
hw/pci-host/q35.c +29 -13
82 lines not on this page · raw message
include/hw/i386/pc.h +1 -0
@@ -156,6 +156,7 @@ void pc_guest_info_init(PCMachineState *pcms);#define PCI_HOST_PROP_PCI_HOLE64_SIZE "pci-hole64-size"#define PCI_HOST_BELOW_4G_MEM_SIZE "below-4g-mem-size"#define PCI_HOST_ABOVE_4G_MEM_SIZE "above-4g-mem-size" +#define PCI_HOST_PROP_SMM_RANGES "smm-ranges" void pc_pci_as_mapping_init(MemoryRegion *system_memory,
include/hw/pci-host/q35.h +1 -0
@@ -50,6 +50,7 @@ struct MCHPCIState {MemoryRegion tseg_blackhole, tseg_window;MemoryRegion smbase_blackhole, smbase_window;bool has_smram_at_smbase; +bool has_smm_ranges;Range pci_hole;uint64_t below_4g_mem_size;uint64_t above_4g_mem_size;
Signature · 1 line
2.34.1
RFC PATCH v2 21/21 i386: Disable SMM mode for X86_SW_PROTECTED_VM
In reply to Xiaoyao Li--- target/i386/kvm/sw-protected-vm.c | 8 ++++++++ 1 file changed, 8 insertions(+)
target/i386/kvm/sw-protected-vm.c +8 -0
@@ -34,10 +34,18 @@ static MemoryListener kvm_x86_sw_protected_vm_memory_listener = {int sw_protected_vm_kvm_init (MachineState *ms,Error **errp) {SwProtectedVm *spvm =SW_PROTECTED_VM (OBJECT (ms->cgs)); +X86MachineState *x86ms =X86_MACHINE (ms);memory_listener_register (&kvm_x86_sw_protected_vm_memory_listener, &address_space_memory); +if (x86ms->smm ==ON_OFF_AUTO_AUTO ) { + x86ms->smm =ON_OFF_AUTO_OFF ; + }else if (x86ms->smm ==ON_OFF_AUTO_ON ) { +error_setg (errp,"X86_SW_PROTECTED_VM doesn't support SMM" ); +return -EINVAL ; + } + spvm->parent_obj.ready = true;return 0 ; }
Signature · 1 line
2.34.1
Quoted 18 lines · Xiaoyao Li
It's the v2 RFC of enabling KVM gmem[1] as the backend for private memory.
For confidential-computing, KVM provides gmem/guest_mem interfaces for userspace, like QEMU, to allocate user-unaccesible private memory. This series aims to add gmem support in QEMU's RAMBlock so that each RAM can have both hva-based shared memory and
gmem_fdbased private memory. QEMU does the shared-private conversion onKVM_MEMORY_EXITand discards the memory.It chooses the design that adds "private" property to hostmeory backend. If "private" property is set, QEMU will allocate/create KVM gmem when initialize the RAMbloch of the memory backend.
This sereis also introduces the first user of kvm gmem,
KVM_X86_SW_PROTECTED_VM. AKVM_X86_SW_PROTECTED_VMwith private KVM gmem can be created with
Quoted 8 lines
$qemu -object sw-protected-vm,id=sp-vm0 \ -object memory-backend-ram,id=mem0,size=1G,private=on \ -machine q35,kernel_irqchip=split,confidential-guest-support=sp-vm0,memory-backend=mem0 \ ...Unfortunately this patch series fails the boot of OVMF at very early stage due to triple fault, because KVM doesn't support emulating string IO to private memory.
Is support being added? Or have we figured out what it would take to make it work?
How does this interact with other features (memory ballooning, virtiofs, vfio/mdev/...)?
Quoted 10 lines
This version still leave some opens to be discussed:
1. whether we need "private" propery to be user-settable?It seems unnecessary because vm-type is determined. If the VM is confidential-guest, then the RAM of the guest must be able to be mapped as private, i.e., have kvm gmem backend. So QEMU can determine the value of "private" property automatiacally based on vm type.
Quoted 4 lines
This also aligns with the board internal MemoryRegion that needs to have kvm gmem backend, e.g., TDX requires OVMF to act as private memory so bios memory region needs to have kvm gmem fd associated. QEMU no doubt will do it internally automatically.
Would it make sense to have some regions without "pivate" semantics? Like NVDIMMs?
Quoted 7 lines
2. hugepage support.
KVM gmem can be allocated from hugetlbfs. How does QEMU determine when to allocate KVM gmem with
KVM_GUEST_MEMFD_ALLOW_HUGEPAGE. The easiest solution is create KVM gmem withKVM_GUEST_MEMFD_ALLOW_HUGEPAGEonly when memory backend is HostMemoryBackendFile of hugetlbfs.
Good question.
Probably "if the memory backend uses huge pages, also use huge pages for the private gmem" makes sense.
... but it becomes a mess with preallocation ... which is what people should actually be using with hugetlb. Andeventual double memory-consumption ... but maybe that's all been taken care of already?
Probably it's best to leave hugetlb support as future work and start with something minimal.
Quoted 3 lines
3. What is
KVM_X86_SW_PROTECTED_VMgoing to look like? and do we need it?
Why implement it when you have to ask others for a motivation? ;)
Personally, I'm not sure if it is really useful, especially in this state.
Quoted 4 lines
This series implements
KVM_X86_SW_PROTECTED_VMbecause it's introduced with gmem together on KVM side and it's supposed to be the first user who requires KVM gmem. However the implementation is incomplete and there lacks the definition of howKVM_X86_SW_PROTECTED_VMworks.
Then it should not be included in this series such that you can make progress with the gmem implementation for TDX guests instead?
Signature · 3 lines
Cheers, David / dhildenb
Sean Christopherson
reviewerQuoted 24 lines · David Hildenbrand
Quoted 23 lines · Xiaoyao Li
It's the v2 RFC of enabling KVM gmem[1] as the backend for private memory.
For confidential-computing, KVM provides gmem/guest_mem interfaces for userspace, like QEMU, to allocate user-unaccesible private memory. This series aims to add gmem support in QEMU's RAMBlock so that each RAM can have both hva-based shared memory and
gmem_fdbased private memory. QEMU does the shared-private conversion onKVM_MEMORY_EXITand discards the memory.It chooses the design that adds "private" property to hostmeory backend. If "private" property is set, QEMU will allocate/create KVM gmem when initialize the RAMbloch of the memory backend.
This sereis also introduces the first user of kvm gmem,
KVM_X86_SW_PROTECTED_VM. AKVM_X86_SW_PROTECTED_VMwith private KVM gmem can be created with$qemu -object sw-protected-vm,id=sp-vm0 \ -object memory-backend-ram,id=mem0,size=1G,private=on \ -machine q35,kernel_irqchip=split,confidential-guest-support=sp-vm0,memory-backend=mem0 \ ...
Quoted 6 lines
Quoted 3 lines
Unfortunately this patch series fails the boot of OVMF at very early stage due to triple fault, because KVM doesn't support emulating string IO to private memory.
Is support being added? Or have we figured out what it would take to make it work?
Hrm, this isn't something I've thought deeply about. The issue is that anything that reaches any form of copy_{from,to}_user() will go kablooie because KVM will always try to read/write the shared mappings. The best case scenario is that the shared mapping is invalid and the uaccess faults. The worst case scenario is that KVM read/writes the wrong memory and sends the guest into the weeds. Eww.
And we (well, at least I) definitely want to support this so that gmem can be used for "regular" VMs, i.e. for VMs where userspace is in the TCB, but for which userspace doesn't have access to guest memory by default.
It shouldn't be too hard to support. It's easy enough to wire up the hook (thankfully that aren't _that_ many sites), and gmem only supports struct page at the moment so we go straight to kmap. E.g. something like this
virt/kvm/kvm_main.c +8 -6
@@ -3291,12 +3291,15 @@ static int next_segment(unsigned long len, int offset)return len; } -static int __kvm_read_guest_page (struct kvm_memory_slot *slot,gfn_t gfn, -void *data,int offset,int len) +static int __kvm_read_guest_page (struct kvm *kvm,struct kvm_memory_slot *slot, +gfn_t gfn,void *data,int offset,int len) {int r;unsigned long addr; +if (kvm_mem_is_private (kvm, gfn)) +return kvm_gmem_read (slot, gfn, data, offset, len); + addr =gfn_to_hva_memslot_prot (slot, gfn,NULL );if (kvm_is_error_hva (addr))return -EFAULT ;
@@ -3309,9 +3312,8 @@ static int __kvm_read_guest_page(struct kvm_memory_slot *slot, gfn_t gfn,int kvm_read_guest_page (struct kvm *kvm,gfn_t gfn,void *data,int offset,int len) { -struct kvm_memory_slot *slot =gfn_to_memslot (kvm, gfn); - -return __kvm_read_guest_page (slot, gfn, data, offset, len); +return __kvm_read_guest_page (kvm,gfn_to_memslot (kvm, gfn), gfn, data, + offset, len); }EXPORT_SYMBOL_GPL (kvm_read_guest_page);
@@ -3320,7 +3322,7 @@ int kvm_vcpu_read_guest_page(struct kvm_vcpu *vcpu, gfn_t gfn, void *data, {struct kvm_memory_slot *slot =kvm_vcpu_gfn_to_memslot (vcpu, gfn); -return __kvm_read_guest_page (slot, gfn, data, offset, len); +return __kvm_read_guest_page (vcpu->kvm, slot, gfn, data, offset, len); }EXPORT_SYMBOL_GPL (kvm_vcpu_read_guest_page);
Quoted 3 lines
Quoted 3 lines
2. hugepage support.
KVM gmem can be allocated from hugetlbfs. How does QEMU determine
Not yet it can't. gmem only supports THP, hugetlbfs is a future thing, if it's ever supported. I wouldn't be at all surprised if we end up going down a slightly different route and don't use hugetlbfs directly.
Quoted 17 lines
Quoted 3 lines
when to allocate KVM gmem with
KVM_GUEST_MEMFD_ALLOW_HUGEPAGE. The easiest solution is create KVM gmem withKVM_GUEST_MEMFD_ALLOW_HUGEPAGEonly when memory backend is HostMemoryBackendFile of hugetlbfs.Good question.
Probably "if the memory backend uses huge pages, also use huge pages for the private gmem" makes sense.
... but it becomes a mess with preallocation ... which is what people should actually be using with hugetlb. Andeventual double memory-consumption ... but maybe that's all been taken care of already?
Probably it's best to leave hugetlb support as future work and start with something minimal.
Quoted 1 line
Quoted 6 lines
Quoted 2 lines
3. What is
KVM_X86_SW_PROTECTED_VMgoing to look like? and do we need it?Why implement it when you have to ask others for a motivation? ;)
Personally, I'm not sure if it is really useful, especially in this state.
Yeah, as of today, KVM_X86_SW_PROTECTED_VM is mainly a development vehicle, e.g. so that testing gmem doesn't require TDX/SNP hardware, debugging gmem guests isn't brutally painful, etc.
Longer term, I have aspirations of being able to back most VMs with gmem, but that's going to require quite a bit more work, e.g. gmem needs to be mappable (when hardware allows it) so that gmem doesn't all but require double mapping, KVM obviously needs to be able to read/write gmem, etc.
The value proposition is that having a guest-first memory type will allow KVM to optimize and harden gmem in ways that wouldn't be feasible for a more generic memory implementation. E.g. memory isn't mapped into host userspace by default (makes it harder to accidentally corrupt the guest), the guest can have *larger* mappings than host userspace, guest memory can be served from a dedicated pool (similar-ish to hugetlb), the pool can be omitted from the direct map, etc.
Quoted 7 lines
Quoted 4 lines
This series implements
KVM_X86_SW_PROTECTED_VMbecause it's introduced with gmem together on KVM side and it's supposed to be the first user who requires KVM gmem. However the implementation is incomplete and there lacks the definition of howKVM_X86_SW_PROTECTED_VMworks.Then it should not be included in this series such that you can make progress with the gmem implementation for TDX guests instead?
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
2. hugepage support.
KVM gmem can be allocated from hugetlbfs. How does QEMU determine
Not yet it can't. gmem only supports THP, hugetlbfs is a future thing, if it's ever supported. I wouldn't be at all surprised if we end up going down a slightly different route and don't use hugetlbfs directly.
Agreed. Certainly future work.
Quoted 34 lines
Quoted 18 lines
Quoted 4 lines
when to allocate KVM gmem with
KVM_GUEST_MEMFD_ALLOW_HUGEPAGE. The easiest solution is create KVM gmem withKVM_GUEST_MEMFD_ALLOW_HUGEPAGEonly when memory backend is HostMemoryBackendFile of hugetlbfs.Good question.
Probably "if the memory backend uses huge pages, also use huge pages for the private gmem" makes sense.
... but it becomes a mess with preallocation ... which is what people should actually be using with hugetlb. Andeventual double memory-consumption ... but maybe that's all been taken care of already?
Probably it's best to leave hugetlb support as future work and start with something minimal.
Quoted 1 line
Quoted 6 lines
Quoted 2 lines
3. What is
KVM_X86_SW_PROTECTED_VMgoing to look like? and do we need it?Why implement it when you have to ask others for a motivation? ;)
Personally, I'm not sure if it is really useful, especially in this state.
Yeah, as of today,
KVM_X86_SW_PROTECTED_VMis mainly a development vehicle, e.g. so that testing gmem doesn't require TDX/SNP hardware, debugging gmem guests isn't brutally painful, etc.Longer term, I have aspirations of being able to back most VMs with gmem, but that's going to require quite a bit more work, e.g. gmem needs to be mappable (when hardware allows it) so that gmem doesn't all but require double mapping, KVM obviously needs to be able to read/write gmem, etc.
Quoted 7 lines
The value proposition is that having a guest-first memory type will allow KVM to optimize and harden gmem in ways that wouldn't be feasible for a more generic memory implementation. E.g. memory isn't mapped into host userspace by default (makes it harder to accidentally corrupt the guest), the guest can have *larger* mappings than host userspace, guest memory can be served from a dedicated pool (similar-ish to hugetlb), the pool can be omitted from the direct map, etc.
Thanks for that information. Personally, I don't believe "to back most VMs with gmem", but that's a different discussion.
As a development vehicle to get TDX up and running it might be very valuable indeed. But it doesn't necessarily have to be merged in QEMU for that case -- especially in a semi-finished form.
Signature · 3 lines
Cheers, David / dhildenb
Quoted 23 lines · David Hildenbrand
Quoted 16 lines
Quoted 6 lines
Quoted 2 lines
3. What is
KVM_X86_SW_PROTECTED_VMgoing to look like? and do we need it?Why implement it when you have to ask others for a motivation? 😉
Personally, I'm not sure if it is really useful, especially in this state.
Yeah, as of today,
KVM_X86_SW_PROTECTED_VMis mainly a development vehicle, e.g. so that testing gmem doesn't require TDX/SNP hardware, debugging gmem guests isn't brutally painful, etc.Longer term, I have aspirations of being able to back most VMs with gmem, but that's going to require quite a bit more work, e.g. gmem needs to be mappable (when hardware allows it) so that gmem doesn't all but require double mapping, KVM obviously needs to be able to read/write gmem, etc.
Quoted 7 lines
The value proposition is that having a guest-first memory type will allow KVM to optimize and harden gmem in ways that wouldn't be feasible for a more generic memory implementation. E.g. memory isn't mapped into host userspace by default (makes it harder to accidentally corrupt the guest), the guest can have *larger* mappings than host userspace, guest memory can be served from a dedicated pool (similar-ish to hugetlb), the pool can be omitted from the direct map, etc.
Quoted 3 lines
Thanks for that information. Personally, I don't believe "to back most VMs with gmem", but that's a different discussion.
As a development vehicle to get TDX up and running it might be very valuable indeed. But it doesn't necessarily have to be merged in QEMU for that case -- especially in a semi-finished form.
It's true and I agree with it. I'll drop the KVM_X86_SW_PROTECTED_VM part in next version.
How would you like this series to proceed in next version? only the patches of gmem support without a user? or together with next QEMU TDX series?
Quoted 44 lines · Xiaoyao Li
Quoted 36 lines · David Hildenbrand
Quoted 23 lines
Quoted 8 lines
Quoted 3 lines
3. What is
KVM_X86_SW_PROTECTED_VMgoing to look like? and do we need it?Why implement it when you have to ask others for a motivation? 😉
Personally, I'm not sure if it is really useful, especially in this state.
Yeah, as of today,
KVM_X86_SW_PROTECTED_VMis mainly a development vehicle,
e.g. so that testing gmem doesn't require TDX/SNP hardware, debugging gmem guests
isn't brutally painful, etc.Longer term, I have aspirations of being able to back most VMs with gmem, but
that's going to require quite a bit more work, e.g. gmem needs to be mappable
(when hardware allows it) so that gmem doesn't all but require double mapping,
KVM obviously needs to be able to read/write gmem, etc.Quoted 13 lines
The value proposition is that having a guest-first memory type will allow KVM to
optimize and harden gmem in ways that wouldn't be feasible for a more generic
memory implementation. E.g. memory isn't mapped into host userspace by default
(makes it harder to accidentally corrupt the guest), the guest can have *larger*
mappings than host userspace, guest memory can be served from a dedicated pool
(similar-ish to hugetlb), the pool can be omitted from the direct map, etc.Quoted 6 lines
Thanks for that information. Personally, I don't believe "to back most VMs with gmem", but that's a different discussion.
As a development vehicle to get TDX up and running it might be very valuable indeed. But it doesn't necessarily have to be merged in QEMU for that case -- especially in a semi-finished form.
Quoted 6 lines
It's true and I agree with it. I'll drop the
KVM_X86_SW_PROTECTED_VMpart in next version.How would you like this series to proceed in next version? only the patches of gmem support without a user? or together with next QEMU TDX series?
Makes sense to me. GMEM series can be a prereq for QEMU TDX.
Signature · 3 lines
Cheers, David / dhildenb
Quoted 24 lines · David Hildenbrand
Quoted 23 lines · Xiaoyao Li
It's the v2 RFC of enabling KVM gmem[1] as the backend for private memory.
For confidential-computing, KVM provides gmem/guest_mem interfaces for userspace, like QEMU, to allocate user-unaccesible private memory. This series aims to add gmem support in QEMU's RAMBlock so that each RAM can have both hva-based shared memory and
gmem_fdbased private memory. QEMU does the shared-private conversion onKVM_MEMORY_EXITand discards the memory.It chooses the design that adds "private" property to hostmeory backend. If "private" property is set, QEMU will allocate/create KVM gmem when initialize the RAMbloch of the memory backend.
This sereis also introduces the first user of kvm gmem,
KVM_X86_SW_PROTECTED_VM. AKVM_X86_SW_PROTECTED_VMwith private KVM gmem can be created with$qemu -object sw-protected-vm,id=sp-vm0 \
-object memory-backend-ram,id=mem0,size=1G,private=on \
-machine q35,kernel_irqchip=split,confidential-guest-support=sp-vm0,memory-backend=mem0 \
...
Quoted 5 lines
Quoted 3 lines
Unfortunately this patch series fails the boot of OVMF at very early stage due to triple fault, because KVM doesn't support emulating string IO to private memory.
Is support being added? Or have we figured out what it would take to make it work?
Hi David,
I only reply the questions that werrn't covered by Sean's reply.
Quoted 1 line
How does this interact with other features (memory ballooning, virtiofs, vfio/mdev/...)?
I need time to learn them before I can answer it.
Quoted 10 lines
Quoted 10 lines
This version still leave some opens to be discussed:
1. whether we need "private" propery to be user-settable?It seems unnecessary because vm-type is determined. If the VM is confidential-guest, then the RAM of the guest must be able to be mapped as private, i.e., have kvm gmem backend. So QEMU can determine the value of "private" property automatiacally based on vm type.
Quoted 6 lines
Quoted 4 lines
This also aligns with the board internal MemoryRegion that needs to have kvm gmem backend, e.g., TDX requires OVMF to act as private memory so bios memory region needs to have kvm gmem fd associated. QEMU no doubt will do it internally automatically.
Would it make sense to have some regions without "pivate" semantics? Like NVDIMMs?
Of course it can have regions without "private" semantics.
Whether a region needs "private" backend depends on the definition of VM type. E.g., for TDX,
- all the RAM needs to able to mapped as private. So it needs private
gmem.
- TDVF(OVMF) code must be mapped as private. So it needs private gmem.
- MMIO region needs to be shared for TDX 1.0, and it doesn't need
private gmem;
Quoted 8 lines
Quoted 7 lines
2. hugepage support.
KVM gmem can be allocated from hugetlbfs. How does QEMU determine when to allocate KVM gmem with
KVM_GUEST_MEMFD_ALLOW_HUGEPAGE. The easiest solution is create KVM gmem withKVM_GUEST_MEMFD_ALLOW_HUGEPAGEonly when memory backend is HostMemoryBackendFile of hugetlbfs.
Quoted 8 lines
Good question.
Probably "if the memory backend uses huge pages, also use huge pages for the private gmem" makes sense.
... but it becomes a mess with preallocation ... which is what people should actually be using with hugetlb. Andeventual double memory-consumption ... but maybe that's all been taken care of already?
Probably it's best to leave hugetlb support as future work and start with something minimal.
As Sean replied, I had some misunderstanding of KVM_GUEST_MEMFD_ALLOW_HUGEPAGE. If it's for THP, I think we can allow it for every gmem.
As for hugetlb, we can leave it as future work.
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
This version still leave some opens to be discussed:
1. whether we need "private" propery to be user-settable?It seems unnecessary because vm-type is determined. If the VM is confidential-guest, then the RAM of the guest must be able to be mapped as private, i.e., have kvm gmem backend. So QEMU can determine the value of "private" property automatiacally based on vm type.
This also aligns with the board internal MemoryRegion that needs to have kvm gmem backend, e.g., TDX requires OVMF to act as private memory so bios memory region needs to have kvm gmem fd associated. QEMU no doubt will do it internally automatically.
Quoted 4 lines
Quoted 2 lines
Would it make sense to have some regions without "pivate" semantics? Like NVDIMMs?
Of course it can have regions without "private" semantics.
I meant "RAM memory regions on such a special VM". Does it make sense to have !private there? I assume "for now, not".
Quoted 18 lines
Quoted 18 lines
Quoted 8 lines
2. hugepage support.
KVM gmem can be allocated from hugetlbfs. How does QEMU determine when to allocate KVM gmem with
KVM_GUEST_MEMFD_ALLOW_HUGEPAGE. The easiest solution is create KVM gmem withKVM_GUEST_MEMFD_ALLOW_HUGEPAGE
only when memory backend is HostMemoryBackendFile of hugetlbfs.Good question.
Probably "if the memory backend uses huge pages, also use huge pages for the private gmem" makes sense.
... but it becomes a mess with preallocation ... which is what people should actually be using with hugetlb. Andeventual double memory-consumption ... but maybe that's all been taken care of already?
Quoted 7 lines
Quoted 3 lines
Probably it's best to leave hugetlb support as future work and start with something minimal.
As Sean replied, I had some misunderstanding of
KVM_GUEST_MEMFD_ALLOW_HUGEPAGE. If it's for THP, I think we can allow it for every gmem.
Right, just like we do a MADV_HUGEPAGE rather blindly on all memory.
Signature · 3 lines
Cheers, David / dhildenb