Skip to contentSkip to conversation

QEMU gmem implemention

RFC PATCH Superseded Xiaoyao Li 7 participants 52 messages Sep 14, 2023 – Oct 8, 2023

Subsystems (3)

v2 of 9 Superseded Latest is v9 →

lore
Share & tools
lore

Xiaoyao Li

author
#0 lore raw

v2 cover QEMU gmem implemention

It's the v2 RFC of enabling KVM gmem[1] as the backend for private memory.

For confidential-computing, KVM provides gmem/guest_mem interfaces for userspace, like QEMU, to allocate user-unaccesible private memory. This series aims to add gmem support in QEMU's RAMBlock so that each RAM can have both hva-based shared memory and gmem_fd based private memory. QEMU does the shared-private conversion on KVM_MEMORY_EXIT and discards the memory.

It chooses the design that adds "private" property to hostmeory backend. If "private" property is set, QEMU will allocate/create KVM gmem when initialize the RAMbloch of the memory backend.

This sereis also introduces the first user of kvm gmem, KVM_X86_SW_PROTECTED_VM. A KVM_X86_SW_PROTECTED_VM with private KVM gmem can be created with

  $qemu -object sw-protected-vm,id=sp-vm0 \
	-object memory-backend-ram,id=mem0,size=1G,private=on \
	-machine q35,kernel_irqchip=split,confidential-guest-support=sp-vm0,memory-backend=mem0 \
	...

Unfortunately this patch series fails the boot of OVMF at very early stage due to triple fault, because KVM doesn't support emulating string IO to private memory.

This version still leave some opens to be discussed:
1. whether we need "private" propery to be user-settable?

It seems unnecessary because vm-type is determined. If the VM is confidential-guest, then the RAM of the guest must be able to be mapped as private, i.e., have kvm gmem backend. So QEMU can determine the value of "private" property automatiacally based on vm type.

This also aligns with the board internal MemoryRegion that needs to have kvm gmem backend, e.g., TDX requires OVMF to act as private memory so bios memory region needs to have kvm gmem fd associated. QEMU no doubt will do it internally automatically.

2. hugepage support.

KVM gmem can be allocated from hugetlbfs. How does QEMU determine when to allocate KVM gmem with KVM_GUEST_MEMFD_ALLOW_HUGEPAGE. The easiest solution is create KVM gmem with KVM_GUEST_MEMFD_ALLOW_HUGEPAGE only when memory backend is HostMemoryBackendFile of hugetlbfs.

3. What is KVM_X86_SW_PROTECTED_VM going to look like? and do we need it?

This series implements KVM_X86_SW_PROTECTED_VM because it's introduced with gmem together on KVM side and it's supposed to be the first user who requires KVM gmem. However the implementation is incomplete and there lacks the definition of how KVM_X86_SW_PROTECTED_VM works.

Any other idea/open/question is welcomed.

Beside, TDX QEMU implemetation is based on this series to provide private gmem for TD private memory, which can be found at [2]. And it can work corresponding KVM [3] to boot TDX guest.

[1] https://lore.kernel.org/all/[email protected]/
[2] https://github.com/intel/qemu-tdx/tree/tdx-qemu-upstream
[3] https://github.com/intel/tdx/tree/kvm-upstream-2023.07.27-v6.5-rc2-workaround

===
Changes since rfc v1:
- Implement KVM_X86_SW_PROTECTED_VM with confidential-guest-support interface;
- rename memory_region_can_be_private() to memory_region_has_gmem_fd();
- allocate kvm gmem fd when creating/initializing the memory backend by introducing the RAM_KVM_GMEM flag;

Chao Peng (3):
RAMBlock: Add support of KVM private gmem
kvm: Enable KVM_SET_USER_MEMORY_REGION2 for memslot
kvm: handle KVM_EXIT_MEMORY_FAULT

Isaku Yamahata (4):
HostMem: Add private property and associate it with RAM_KVM_GMEM trace/kvm: Add trace for page convertion between shared and private pci-host/q35: Move PAM initialization above SMRAM initialization
q35: Introduce smm_ranges property for q35-pci-host

Xiaoyao Li (14):
*** HACK *** linux-headers: Update headers to pull in gmem APIs
memory: Introduce memory_region_has_gmem_fd()
i386: Add support for sw-protected-vm object
i386/pc: Drop pc_machine_kvm_type()
target/i386: Implement mc->kvm_type() to get VM type target/i386: Introduce kvm_confidential_guest_init() i386/kvm: Implement kvm_sw_protected_vm_init() for sw-protcted-vm
specific functions
kvm: Introduce support for memory_attributes
kvm/memory: Introduce the infrastructure to set the default
shared/private value
i386/kvm: Set memory to default private for KVM_X86_SW_PROTECTED_VM
physmem: replace function name with __func__ in
ram_block_discard_range()
physmem: extract ram_block_discard_range_fd() from
ram_block_discard_range()
physmem: Introduce ram_block_convert_range()
i386: Disable SMM mode for X86_SW_PROTECTED_VM

 accel/kvm/kvm-all.c               | 180 ++++++++++++++++++++-
 accel/kvm/trace-events            |   4 +-
 backends/hostmem-file.c           |   1 +
 backends/hostmem-memfd.c          |   1 +
 backends/hostmem-ram.c            |   1 +
 backends/hostmem.c                |  18 +++
 hw/i386/pc.c                      |   5 -
 hw/i386/pc_q35.c                  |   3 +-
 hw/i386/x86.c                     |  12 ++
 hw/pci-host/q35.c                 |  61 ++++---
 include/exec/cpu-common.h         |   2 +
 include/exec/memory.h             |  20 +++
 include/exec/ramblock.h           |   1 +
 include/hw/i386/pc.h              |   4 +-
 include/hw/i386/x86.h             |   1 +
 include/hw/pci-host/q35.h         |   1 +
 include/sysemu/hostmem.h          |   2 +-
 include/sysemu/kvm.h              |   5 +
 include/sysemu/kvm_int.h          |   2 +
 linux-headers/asm-x86/kvm.h       |   3 +
 linux-headers/linux/kvm.h         |  50 ++++++
 qapi/qom.json                     |   5 +
 softmmu/memory.c                  |  18 +++
 softmmu/physmem.c                 | 256 ++++++++++++++++++------------
 target/i386/kvm/kvm.c             |  43 ++++-
 target/i386/kvm/kvm_i386.h        |   1 +
 target/i386/kvm/meson.build       |   1 +
 target/i386/kvm/sw-protected-vm.c |  71 +++++++++
 target/i386/kvm/sw-protected-vm.h |  19 +++
 target/i386/sev.c                 |   1 -
 target/i386/sev.h                 |   2 +
 31 files changed, 648 insertions(+), 146 deletions(-)
 create mode 100644 target/i386/kvm/sw-protected-vm.c
 create mode 100644 target/i386/kvm/sw-protected-vm.h
Signature · 1 line
2.34.1
author
#1 lore raw

RFC PATCH v2 1/21 *** HACK *** linux-headers: Update headers to pull in gmem APIs

This patch needs to be updated by script

scripts/update-linux-headers.sh

once gmem fd support is upstreamed in Linux kernel.

---
 linux-headers/asm-x86/kvm.h |  3 +++
 linux-headers/linux/kvm.h   | 50 +++++++++++++++++++++++++++++++++++++
 2 files changed, 53 insertions(+)
linux-headers/asm-x86/kvm.h +3 -0
index 2b3a8f7bd2c0..003fb745347c 100644
@@ -560,4 +560,7 @@ struct kvm_pmu_event_filter {
 /* x86-specific KVM_EXIT_HYPERCALL flags. */
 #define KVM_EXIT_HYPERCALL_LONG_MODE	BIT(0)
 
+#define KVM_X86_DEFAULT_VM	0
+#define KVM_X86_SW_PROTECTED_VM	1
+
 #endif /* _ASM_X86_KVM_H */
linux-headers/linux/kvm.h +50 -0

90 lines not on this page · raw message

Signature · 1 line
2.34.1
author
#2 lore raw

RFC PATCH v2 2/21 RAMBlock: Add support of KVM private gmem

In reply to Xiaoyao Li

From: Chao Peng <[email protected]>

Add KVM gmem support to RAMBlock so both normal hva based memory and kvm gmem fd based private memory can be associated in one RAMBlock.

Introduce new flag RAM_KVM_GMEM. It calls KVM ioctl to create private gmem for the RAMBlock when it's set.

---
 accel/kvm/kvm-all.c     | 17 +++++++++++++++++
 include/exec/memory.h   |  3 +++
 include/exec/ramblock.h |  1 +
 include/sysemu/kvm.h    |  2 ++
 softmmu/physmem.c       | 18 +++++++++++++++---
 5 files changed, 38 insertions(+), 3 deletions(-)
accel/kvm/kvm-all.c +17 -0
index 60aacd925393..185ae16d9620 100644
@@ -4225,3 +4225,20 @@ void query_stats_schemas_cb(StatsSchemaList **result, Error **errp)
         query_stats_schema_vcpu(first_cpu, &stats_args);
     }
 }
+
+int kvm_create_guest_memfd(uint64_t size, uint64_t flags, Error **errp)
+{
+    int fd;
+    struct kvm_create_guest_memfd gmem = {
+        .size = size,
+        /* TODO: to decide whether KVM_GUEST_MEMFD_ALLOW_HUGEPAGE is supported */
+        .flags = flags,
+    };
+
+    fd = kvm_vm_ioctl(kvm_state, KVM_CREATE_GUEST_MEMFD, &gmem);
+    if (fd < 0) {
+        error_setg_errno(errp, errno, "%s: error creating kvm gmem\n", __func__);
+    }
+
+    return fd;
+}
include/exec/memory.h +3 -0
index 68284428f87c..227cb2578e95 100644
@@ -235,6 +235,9 @@ typedef struct IOMMUTLBEvent {
 /* RAM is an mmap-ed named file */
 #define RAM_NAMED_FILE (1 << 9)
 
+/* RAM can be private that has kvm gmem backend */
+#define RAM_KVM_GMEM    (1 << 10)
+
 static inline void iommu_notifier_init(IOMMUNotifier *n, IOMMUNotify fn,
                                        IOMMUNotifierFlag flags,
                                        hwaddr start, hwaddr end,
include/exec/ramblock.h +1 -0
index 69c6a5390293..0d158b3909c9 100644
@@ -41,6 +41,7 @@ struct RAMBlock {
     QLIST_HEAD(, RAMBlockNotifier) ramblock_notifiers;
     int fd;
     uint64_t fd_offset;
+    int gmem_fd;
     size_t page_size;
     /* dirty bitmap used during migration */
     unsigned long *bmap;
include/sysemu/kvm.h +2 -0
index 115f0cca79d1..f5b74c8dd8c5 100644
@@ -580,4 +580,6 @@ bool kvm_arch_cpu_check_are_resettable(void);
 bool kvm_dirty_ring_enabled(void);
 
 uint32_t kvm_dirty_ring_size(void);
+
+int kvm_create_guest_memfd(uint64_t size, uint64_t flags, Error **errp);
 #endif
softmmu/physmem.c +15 -3
index 3df73542e1fe..2d98a88f41f0 100644
@@ -1824,6 +1824,16 @@ static void ram_block_add(RAMBlock *new_block, Error **errp)
         }
     }
 
+    if (kvm_enabled() && new_block->flags & RAM_KVM_GMEM &&
+        new_block->gmem_fd < 0) {
+        new_block->gmem_fd = kvm_create_guest_memfd(new_block->max_length,
+                                                    0, errp);
+        if (new_block->gmem_fd < 0) {
+            qemu_mutex_unlock_ramlist();
+            return;
+        }
+    }
+
     new_ram_size = MAX(old_ram_size,
               (new_block->offset + new_block->max_length) >> TARGET_PAGE_BITS);
     if (new_ram_size > old_ram_size) {
@@ -1885,7 +1895,7 @@ RAMBlock *qemu_ram_alloc_from_fd(ram_addr_t size, MemoryRegion *mr,
 
     /* Just support these ram flags by now. */
     assert((ram_flags & ~(RAM_SHARED | RAM_PMEM | RAM_NORESERVE |
-                          RAM_PROTECTED | RAM_NAMED_FILE)) == 0);
+                          RAM_PROTECTED | RAM_NAMED_FILE | RAM_KVM_GMEM)) == 0);
 
     if (xen_enabled()) {
         error_setg(errp, "-mem-path not supported with Xen");
@@ -1920,6 +1930,7 @@ RAMBlock *qemu_ram_alloc_from_fd(ram_addr_t size, MemoryRegion *mr,
     new_block->used_length = size;
     new_block->max_length = size;
     new_block->flags = ram_flags;
+    new_block->gmem_fd = -1;
     new_block->host = file_ram_alloc(new_block, size, fd, readonly,
                                      !file_size, offset, errp);
     if (!new_block->host) {
@@ -1978,7 +1989,7 @@ RAMBlock *qemu_ram_alloc_internal(ram_addr_t size, ram_addr_t max_size,
     Error *local_err = NULL;
 
     assert((ram_flags & ~(RAM_SHARED | RAM_RESIZEABLE | RAM_PREALLOC |
-                          RAM_NORESERVE)) == 0);
+                          RAM_NORESERVE| RAM_KVM_GMEM)) == 0);
     assert(!host ^ (ram_flags & RAM_PREALLOC));
 
     size = HOST_PAGE_ALIGN(size);
@@ -1990,6 +2001,7 @@ RAMBlock *qemu_ram_alloc_internal(ram_addr_t size, ram_addr_t max_size,
     new_block->max_length = max_size;
     assert(max_size >= size);
     new_block->fd = -1;
+    new_block->gmem_fd = -1;
     new_block->page_size = qemu_real_host_page_size();
     new_block->host = host;
     new_block->flags = ram_flags;
@@ -2012,7 +2024,7 @@ RAMBlock *qemu_ram_alloc_from_ptr(ram_addr_t size, void *host,
 RAMBlock *qemu_ram_alloc(ram_addr_t size, uint32_t ram_flags,
                          MemoryRegion *mr, Error **errp)
 {
-    assert((ram_flags & ~(RAM_SHARED | RAM_NORESERVE)) == 0);
+    assert((ram_flags & ~(RAM_SHARED | RAM_NORESERVE | RAM_KVM_GMEM)) == 0);
     return qemu_ram_alloc_internal(size, size, NULL, NULL, ram_flags, mr, errp);
 }
 
Signature · 1 line
2.34.1
#3 lore raw

Re: [RFC PATCH v2 02/21] RAMBlock: Add support of KVM private gmem

Quoted 9 lines · Xiaoyao Li

From: Chao Peng <[email protected]>

Add KVM gmem support to RAMBlock so both normal hva based memory and kvm gmem fd based private memory can be associated in one RAMBlock.

Introduce new flag RAM_KVM_GMEM. It calls KVM ioctl to create private gmem for the RAMBlock when it's set.

Kindly reminding the author's Signed-off-by is missing.

Quoted 134 lines
---
 accel/kvm/kvm-all.c     | 17 +++++++++++++++++
 include/exec/memory.h   |  3 +++
 include/exec/ramblock.h |  1 +
 include/sysemu/kvm.h    |  2 ++
 softmmu/physmem.c       | 18 +++++++++++++++---
 5 files changed, 38 insertions(+), 3 deletions(-)
accel/kvm/kvm-all.c +17 -0
index 60aacd925393..185ae16d9620 100644
@@ -4225,3 +4225,20 @@ void query_stats_schemas_cb(StatsSchemaList **result, Error **errp)
         query_stats_schema_vcpu(first_cpu, &stats_args);
     }
 }
+
+int kvm_create_guest_memfd(uint64_t size, uint64_t flags, Error **errp)
+{
+    int fd;
+    struct kvm_create_guest_memfd gmem = {
+        .size = size,
+        /* TODO: to decide whether KVM_GUEST_MEMFD_ALLOW_HUGEPAGE is supported */
+        .flags = flags,
+    };
+
+    fd = kvm_vm_ioctl(kvm_state, KVM_CREATE_GUEST_MEMFD, &gmem);
+    if (fd < 0) {
+        error_setg_errno(errp, errno, "%s: error creating kvm gmem\n", __func__);
+    }
+
+    return fd;
+}
include/exec/memory.h +3 -0
index 68284428f87c..227cb2578e95 100644
@@ -235,6 +235,9 @@ typedef struct IOMMUTLBEvent {
 /* RAM is an mmap-ed named file */
 #define RAM_NAMED_FILE (1 << 9)
 
+/* RAM can be private that has kvm gmem backend */
+#define RAM_KVM_GMEM    (1 << 10)
+
 static inline void iommu_notifier_init(IOMMUNotifier *n, IOMMUNotify fn,
                                        IOMMUNotifierFlag flags,
                                        hwaddr start, hwaddr end,
include/exec/ramblock.h +1 -0
index 69c6a5390293..0d158b3909c9 100644
@@ -41,6 +41,7 @@ struct RAMBlock {
     QLIST_HEAD(, RAMBlockNotifier) ramblock_notifiers;
     int fd;
     uint64_t fd_offset;
+    int gmem_fd;
     size_t page_size;
     /* dirty bitmap used during migration */
     unsigned long *bmap;
include/sysemu/kvm.h +2 -0
index 115f0cca79d1..f5b74c8dd8c5 100644
@@ -580,4 +580,6 @@ bool kvm_arch_cpu_check_are_resettable(void);
 bool kvm_dirty_ring_enabled(void);
 
 uint32_t kvm_dirty_ring_size(void);
+
+int kvm_create_guest_memfd(uint64_t size, uint64_t flags, Error **errp);
 #endif
softmmu/physmem.c +15 -3
index 3df73542e1fe..2d98a88f41f0 100644
@@ -1824,6 +1824,16 @@ static void ram_block_add(RAMBlock *new_block, Error **errp)
         }
     }
 
+    if (kvm_enabled() && new_block->flags & RAM_KVM_GMEM &&
+        new_block->gmem_fd < 0) {
+        new_block->gmem_fd = kvm_create_guest_memfd(new_block->max_length,
+                                                    0, errp);
+        if (new_block->gmem_fd < 0) {
+            qemu_mutex_unlock_ramlist();
+            return;
+        }
+    }
+
     new_ram_size = MAX(old_ram_size,
               (new_block->offset + new_block->max_length) >> TARGET_PAGE_BITS);
     if (new_ram_size > old_ram_size) {
@@ -1885,7 +1895,7 @@ RAMBlock *qemu_ram_alloc_from_fd(ram_addr_t size, MemoryRegion *mr,
 
     /* Just support these ram flags by now. */
     assert((ram_flags & ~(RAM_SHARED | RAM_PMEM | RAM_NORESERVE |
-                          RAM_PROTECTED | RAM_NAMED_FILE)) == 0);
+                          RAM_PROTECTED | RAM_NAMED_FILE | RAM_KVM_GMEM)) == 0);
 
     if (xen_enabled()) {
         error_setg(errp, "-mem-path not supported with Xen");
@@ -1920,6 +1930,7 @@ RAMBlock *qemu_ram_alloc_from_fd(ram_addr_t size, MemoryRegion *mr,
     new_block->used_length = size;
     new_block->max_length = size;
     new_block->flags = ram_flags;
+    new_block->gmem_fd = -1;
     new_block->host = file_ram_alloc(new_block, size, fd, readonly,
                                      !file_size, offset, errp);
     if (!new_block->host) {
@@ -1978,7 +1989,7 @@ RAMBlock *qemu_ram_alloc_internal(ram_addr_t size, ram_addr_t max_size,
     Error *local_err = NULL;
 
     assert((ram_flags & ~(RAM_SHARED | RAM_RESIZEABLE | RAM_PREALLOC |
-                          RAM_NORESERVE)) == 0);
+                          RAM_NORESERVE| RAM_KVM_GMEM)) == 0);
     assert(!host ^ (ram_flags & RAM_PREALLOC));
 
     size = HOST_PAGE_ALIGN(size);
@@ -1990,6 +2001,7 @@ RAMBlock *qemu_ram_alloc_internal(ram_addr_t size, ram_addr_t max_size,
     new_block->max_length = max_size;
     assert(max_size >= size);
     new_block->fd = -1;
+    new_block->gmem_fd = -1;
     new_block->page_size = qemu_real_host_page_size();
     new_block->host = host;
     new_block->flags = ram_flags;
@@ -2012,7 +2024,7 @@ RAMBlock *qemu_ram_alloc_from_ptr(ram_addr_t size, void *host,
 RAMBlock *qemu_ram_alloc(ram_addr_t size, uint32_t ram_flags,
                          MemoryRegion *mr, Error **errp)
 {
-    assert((ram_flags & ~(RAM_SHARED | RAM_NORESERVE)) == 0);
+    assert((ram_flags & ~(RAM_SHARED | RAM_NORESERVE | RAM_KVM_GMEM)) == 0);
     return qemu_ram_alloc_internal(size, size, NULL, NULL, ram_flags, mr, errp);
 }
 
author
#4 lore raw

Re: [RFC PATCH v2 02/21] RAMBlock: Add support of KVM private gmem

Quoted 11 lines · Lei
Quoted 9 lines · Xiaoyao Li

From: Chao Peng<[email protected]>

Add KVM gmem support to RAMBlock so both normal hva based memory and kvm gmem fd based private memory can be associated in one RAMBlock.

Introduce new flag RAM_KVM_GMEM. It calls KVM ioctl to create private gmem for the RAMBlock when it's set.

Kindly reminding the author's Signed-off-by is missing.

I will fix it.
Thanks!

#5 lore raw

Re: [RFC PATCH v2 02/21] RAMBlock: Add support of KVM private gmem

In reply to Xiaoyao Li
Quoted 7 lines · Xiaoyao Li

From: Chao Peng <[email protected]>

Add KVM gmem support to RAMBlock so both normal hva based memory and kvm gmem fd based private memory can be associated in one RAMBlock.

Introduce new flag RAM_KVM_GMEM. It calls KVM ioctl to create private gmem for the RAMBlock when it's set.

But who sets RAM_KVM_GMEM and when? Don't we simply allocate it for all RAMBlocks under such special VMs? What's the downside of doing that?

Signature · 3 lines
Cheers,

David / dhildenb
author
#6 lore raw

Re: [RFC PATCH v2 02/21] RAMBlock: Add support of KVM private gmem

Quoted 11 lines · David Hildenbrand
Quoted 7 lines · Xiaoyao Li

From: Chao Peng <[email protected]>

Add KVM gmem support to RAMBlock so both normal hva based memory and kvm gmem fd based private memory can be associated in one RAMBlock.

Introduce new flag RAM_KVM_GMEM. It calls KVM ioctl to create private gmem for the RAMBlock when it's set.

But who sets RAM_KVM_GMEM and when?

The answer is in the next patch. When private property of memory backend is set to true, it will pass RAM_KVM_GMEM flag to memory_region_init_ram_*()

Quoted 1 line

Don't we simply allocate it for all RAMBlocks under such special VMs?

yes, this is the direction after your comments.

I'll try to figure out how to achieve it.

Quoted 1 line

What's the downside of doing that?

As far as I see, for TDX, no downside.

#7 lore raw

Re: [RFC PATCH v2 02/21] RAMBlock: Add support of KVM private gmem

Quoted 11 lines · Xiaoyao Li
Quoted 10 lines · David Hildenbrand
Quoted 7 lines · Xiaoyao Li

From: Chao Peng <[email protected]>

Add KVM gmem support to RAMBlock so both normal hva based memory and kvm gmem fd based private memory can be associated in one RAMBlock.

Introduce new flag RAM_KVM_GMEM. It calls KVM ioctl to create private gmem for the RAMBlock when it's set.

Quoted 5 lines
Quoted 1 line

But who sets RAM_KVM_GMEM and when?

The answer is in the next patch. When private property of memory backend is set to true, it will pass RAM_KVM_GMEM flag to memory_region_init_ram_*()

Okay, assuming that patch (and property) will go away, I assume this flag can also go away, right?

Signature · 3 lines
Cheers,

David / dhildenb
author
#8 lore raw

Re: [RFC PATCH v2 02/21] RAMBlock: Add support of KVM private gmem

Quoted 12 lines · David Hildenbrand
Quoted 11 lines · Xiaoyao Li
Quoted 10 lines · David Hildenbrand
Quoted 7 lines · Xiaoyao Li

From: Chao Peng <[email protected]>

Add KVM gmem support to RAMBlock so both normal hva based memory and kvm gmem fd based private memory can be associated in one RAMBlock.

Introduce new flag RAM_KVM_GMEM. It calls KVM ioctl to create private gmem for the RAMBlock when it's set.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

But who sets RAM_KVM_GMEM and when?

The answer is in the next patch. When private property of memory backend is set to true, it will pass RAM_KVM_GMEM flag to memory_region_init_ram_*()

Okay, assuming that patch (and property) will go away, I assume this flag can also go away, right?

If dropping the flag RAM_KVM_GMEM, it seems we need go back to the approach of rfc v1[1][2], that allocating gmem inside .region_add() callback. Is it accepted by you?

Another option is allocating gmem inside ram_block_add() by checking the vm_type (it looks hacky for me). What's your opinion on this option?

One more option is, we keep the RAM_KVM_GMEM as this patch, and change "private" property of memory backend into "need_kvm_gmem" field (make it not user settable) and "need_kvm_gmem" field will be set to true in tdx_kvm_init() specific cgs initialization function.

[1] https://lore.kernel.org/qemu-devel/[email protected]/
[2] https://lore.kernel.org/qemu-devel/[email protected]/

#9 lore raw

Re: [RFC PATCH v2 02/21] RAMBlock: Add support of KVM private gmem

In reply to Xiaoyao Li
Quoted 61 lines · Xiaoyao Li

From: Chao Peng <[email protected]>

Add KVM gmem support to RAMBlock so both normal hva based memory and kvm gmem fd based private memory can be associated in one RAMBlock.

Introduce new flag RAM_KVM_GMEM. It calls KVM ioctl to create private gmem for the RAMBlock when it's set.

---
 accel/kvm/kvm-all.c     | 17 +++++++++++++++++
 include/exec/memory.h   |  3 +++
 include/exec/ramblock.h |  1 +
 include/sysemu/kvm.h    |  2 ++
 softmmu/physmem.c       | 18 +++++++++++++++---
 5 files changed, 38 insertions(+), 3 deletions(-)
accel/kvm/kvm-all.c +17 -0
index 60aacd925393..185ae16d9620 100644
@@ -4225,3 +4225,20 @@ void query_stats_schemas_cb(StatsSchemaList **result, Error **errp)
         query_stats_schema_vcpu(first_cpu, &stats_args);
     }
 }
+
+int kvm_create_guest_memfd(uint64_t size, uint64_t flags, Error **errp)
+{
+    int fd;
+    struct kvm_create_guest_memfd gmem = {
+        .size = size,
+        /* TODO: to decide whether KVM_GUEST_MEMFD_ALLOW_HUGEPAGE is supported */
+        .flags = flags,
+    };
+
+    fd = kvm_vm_ioctl(kvm_state, KVM_CREATE_GUEST_MEMFD, &gmem);
+    if (fd < 0) {
+        error_setg_errno(errp, errno, "%s: error creating kvm gmem\n", __func__);
+    }
+
+    return fd;
+}
include/exec/memory.h +3 -0
index 68284428f87c..227cb2578e95 100644
@@ -235,6 +235,9 @@ typedef struct IOMMUTLBEvent {
 /* RAM is an mmap-ed named file */
 #define RAM_NAMED_FILE (1 << 9)
 
+/* RAM can be private that has kvm gmem backend */
+#define RAM_KVM_GMEM    (1 << 10)
+
 static inline void iommu_notifier_init(IOMMUNotifier *n, IOMMUNotify fn,
                                        IOMMUNotifierFlag flags,
                                        hwaddr start, hwaddr end,
include/exec/ramblock.h +0 -0
index 69c6a5390293..0d158b3909c9 100644

include/exec/ramblock.h struct RAMBlock { · line 47

Quoted 7 lines
include/exec/ramblock.h +1 -0
@@ -41,6 +41,7 @@ struct RAMBlock {
     QLIST_HEAD(, RAMBlockNotifier) ramblock_notifiers;
     int fd;
     uint64_t fd_offset;
+    int gmem_fd;
     size_t page_size;
     /* dirty bitmap used during migration */
     unsigned long *bmap;

You're adding a file descriptor to RAMBlock, but I don't see anything in this patch that ever calls close(gmem_fd) when the RAMBlock is released. Presuambly reclaim_ramblock() needs to deal with this ?

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
author
#10 lore raw

RFC PATCH v2 3/21 HostMem: Add private property and associate it with RAM_KVM_GMEM

In reply to Xiaoyao Li

From: Isaku Yamahata <[email protected]>

Add a new property "private" to memory backends. When it's set to true, it indicates the RAMblock of the backend also requires kvm gmem.

---
 backends/hostmem-file.c  |  1 +
 backends/hostmem-memfd.c |  1 +
 backends/hostmem-ram.c   |  1 +
 backends/hostmem.c       | 18 ++++++++++++++++++
 include/sysemu/hostmem.h |  2 +-
 qapi/qom.json            |  4 ++++
 6 files changed, 26 insertions(+), 1 deletion(-)
backends/hostmem-file.c +1 -0
index b4335a80e6da..861f76f2de8a 100644
@@ -56,6 +56,7 @@ file_backend_memory_alloc(HostMemoryBackend *backend, Error **errp)
     name = host_memory_backend_get_name(backend);
     ram_flags = backend->share ? RAM_SHARED : 0;
     ram_flags |= backend->reserve ? 0 : RAM_NORESERVE;
+    ram_flags |= backend->private ? RAM_KVM_GMEM : 0;
     ram_flags |= fb->is_pmem ? RAM_PMEM : 0;
     ram_flags |= RAM_NAMED_FILE;
     memory_region_init_ram_from_file(&backend->mr, OBJECT(backend), name,
backends/hostmem-memfd.c +1 -0
index 3fc85c3db81b..f49990ce3bbd 100644
@@ -55,6 +55,7 @@ memfd_backend_memory_alloc(HostMemoryBackend *backend, Error **errp)
     name = host_memory_backend_get_name(backend);
     ram_flags = backend->share ? RAM_SHARED : 0;
     ram_flags |= backend->reserve ? 0 : RAM_NORESERVE;
+    ram_flags |= backend->private ? RAM_KVM_GMEM : 0;
     memory_region_init_ram_from_fd(&backend->mr, OBJECT(backend), name,
                                    backend->size, ram_flags, fd, 0, errp);
     g_free(name);
backends/hostmem-ram.c +1 -0
index b8e55cdbd0f8..d6c46250dcfd 100644
@@ -30,6 +30,7 @@ ram_backend_memory_alloc(HostMemoryBackend *backend, Error **errp)
     name = host_memory_backend_get_name(backend);
     ram_flags = backend->share ? RAM_SHARED : 0;
     ram_flags |= backend->reserve ? 0 : RAM_NORESERVE;
+    ram_flags |= backend->private ? RAM_KVM_GMEM : 0;
     memory_region_init_ram_flags_nomigrate(&backend->mr, OBJECT(backend), name,
                                            backend->size, ram_flags, errp);
     g_free(name);
backends/hostmem.c +18 -0
index 747e7838c031..dbdbb0aafd45 100644
@@ -461,6 +461,20 @@ static void host_memory_backend_set_reserve(Object *o, bool value, Error **errp)
     }
     backend->reserve = value;
 }
+
+static bool host_memory_backend_get_private(Object *o, Error **errp)
+{
+    HostMemoryBackend *backend = MEMORY_BACKEND(o);
+
+    return backend->private;
+}
+
+static void host_memory_backend_set_private(Object *o, bool value, Error **errp)
+{
+    HostMemoryBackend *backend = MEMORY_BACKEND(o);
+
+    backend->private = value;
+}
 #endif /* CONFIG_LINUX */
 
 static bool
@@ -541,6 +555,10 @@ host_memory_backend_class_init(ObjectClass *oc, void *data)
         host_memory_backend_get_reserve, host_memory_backend_set_reserve);
     object_class_property_set_description(oc, "reserve",
         "Reserve swap space (or huge pages) if applicable");
+    object_class_property_add_bool(oc, "private",
+        host_memory_backend_get_private, host_memory_backend_set_private);
+    object_class_property_set_description(oc, "private",
+        "Use KVM gmem private memory");
 #endif /* CONFIG_LINUX */
     /*
      * Do not delete/rename option. This option must be considered stable
include/sysemu/hostmem.h +1 -1
index 39326f1d4f9c..d88970395618 100644
@@ -65,7 +65,7 @@ struct HostMemoryBackend {
     /* protected */
     uint64_t size;
     bool merge, dump, use_canonical_path;
-    bool prealloc, is_mapped, share, reserve;
+    bool prealloc, is_mapped, share, reserve, private;
     uint32_t prealloc_threads;
     ThreadContext *prealloc_context;
     DECLARE_BITMAP(host_nodes, MAX_NODES + 1);
qapi/qom.json +4 -0
index fa3e88c8e6ab..d28c5403bc0f 100644
@@ -605,6 +605,9 @@
 # @reserve: if true, reserve swap space (or huge pages) if applicable
 #     (default: true) (since 6.1)
 #
+# @private: if true, use KVM gmem private memory (default: false)
+#     (since 8.2)
+#
 # @size: size of the memory region in bytes
 #
 # @x-use-canonical-path-for-ramblock-id: if true, the canonical path
@@ -631,6 +634,7 @@
             '*prealloc-context': 'str',
             '*share': 'bool',
             '*reserve': 'bool',
+            '*private': 'bool',
             'size': 'size',
             '*x-use-canonical-path-for-ramblock-id': 'bool' } }
 
Signature · 1 line
2.34.1
#11 lore raw

Re: [RFC PATCH v2 03/21] HostMem: Add private property and associate it with RAM_KVM_GMEM

Quoted 4 lines · Xiaoyao Li

From: Isaku Yamahata <[email protected]>

Add a new property "private" to memory backends. When it's set to true, it indicates the RAMblock of the backend also requires kvm gmem.

Can you add a brief explanation why you need the property?

Quoted 2 lines

[...]

Quoted 21 lines
qapi/qom.json +4 -0
index fa3e88c8e6ab..d28c5403bc0f 100644
@@ -605,6 +605,9 @@
 # @reserve: if true, reserve swap space (or huge pages) if applicable
 #     (default: true) (since 6.1)
 #
+# @private: if true, use KVM gmem private memory (default: false)
+#     (since 8.2)
+#
 # @size: size of the memory region in bytes
 #
 # @x-use-canonical-path-for-ramblock-id: if true, the canonical path
@@ -631,6 +634,7 @@
             '*prealloc-context': 'str',
             '*share': 'bool',
             '*reserve': 'bool',
+            '*private': 'bool',
             'size': 'size',
             '*x-use-canonical-path-for-ramblock-id': 'bool' } }
author
#12 lore raw

Re: [RFC PATCH v2 03/21] HostMem: Add private property and associate it with RAM_KVM_GMEM

Quoted 8 lines · Markus Armbruster
Quoted 4 lines · Xiaoyao Li

From: Isaku Yamahata <[email protected]>

Add a new property "private" to memory backends. When it's set to true, it indicates the RAMblock of the backend also requires kvm gmem.

Can you add a brief explanation why you need the property?

It provides a mechanism for user to specify whether the memory can serve as private memory (need request kvm gmem).

Quoted 27 lines
Quoted 2 lines

[...]

Quoted 21 lines
qapi/qom.json +4 -0
index fa3e88c8e6ab..d28c5403bc0f 100644
@@ -605,6 +605,9 @@
  # @reserve: if true, reserve swap space (or huge pages) if applicable
  #     (default: true) (since 6.1)
  #
+# @private: if true, use KVM gmem private memory (default: false)
+#     (since 8.2)
+#
  # @size: size of the memory region in bytes
  #
  # @x-use-canonical-path-for-ramblock-id: if true, the canonical path
@@ -631,6 +634,7 @@
              '*prealloc-context': 'str',
              '*share': 'bool',
              '*reserve': 'bool',
+            '*private': 'bool',
              'size': 'size',
              '*x-use-canonical-path-for-ramblock-id': 'bool' } }
#13 lore raw

Re: [RFC PATCH v2 03/21] HostMem: Add private property and associate it with RAM_KVM_GMEM

Quoted 10 lines · Xiaoyao Li
Quoted 7 lines · Markus Armbruster
Quoted 4 lines · Xiaoyao Li

From: Isaku Yamahata <[email protected]>

Add a new property "private" to memory backends. When it's set to true, it indicates the RAMblock of the backend also requires kvm gmem.

Can you add a brief explanation why you need the property?

It provides a mechanism for user to specify whether the memory can serve as private memory (need request kvm gmem).

Yes, but why would a user want such memory?

author
#14 lore raw

Re: [RFC PATCH v2 03/21] HostMem: Add private property and associate it with RAM_KVM_GMEM

Quoted 7 lines · Markus Armbruster
Quoted 5 lines · Xiaoyao Li
Quoted 4 lines · Markus Armbruster
Quoted 2 lines · Xiaoyao Li

From: Isaku Yamahata <[email protected]>

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines

Add a new property "private" to memory backends. When it's set to true, it indicates the RAMblock of the backend also requires kvm gmem.

Can you add a brief explanation why you need the property?

It provides a mechanism for user to specify whether the memory can serve as private memory (need request kvm gmem).

Yes, but why would a user want such memory?

Because KVM demands it for confidential guest, e.g., TDX guest. KVM demands that the mem slot needs to have KVM_MEM_PRIVATE set and has valid gmem associated if the guest accesses it as private memory.

#15 lore raw

Re: [RFC PATCH v2 03/21] HostMem: Add private property and associate it with RAM_KVM_GMEM

Quoted 12 lines · Xiaoyao Li
Quoted 11 lines · Markus Armbruster
Quoted 9 lines · Xiaoyao Li
Quoted 7 lines · Markus Armbruster
Quoted 4 lines · Xiaoyao Li

From: Isaku Yamahata <[email protected]>

Add a new property "private" to memory backends. When it's set to true, it indicates the RAMblock of the backend also requires kvm gmem.

Can you add a brief explanation why you need the property?

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

It provides a mechanism for user to specify whether the memory can serve as private memory (need request kvm gmem).

Yes, but why would a user want such memory?

Because KVM demands it for confidential guest, e.g., TDX guest. KVM demands that the mem slot needs to have KVM_MEM_PRIVATE set and has valid gmem associated if the guest accesses it as private memory.

I think as long as there is no demand to have a TDX guest with this property be set to "off", then just don't add it.

With a TDX VM, it will can be implicitly active. If we ever have to disable it for selective memory backends, we can add the property and have something like on/off/auto. For now it would be "auto".

Signature · 3 lines
Cheers,

David / dhildenb
#16 lore raw

Re: [RFC PATCH v2 03/21] HostMem: Add private property and associate it with RAM_KVM_GMEM

Quoted 13 lines · David Hildenbrand
Quoted 12 lines · Xiaoyao Li
Quoted 11 lines · Markus Armbruster
Quoted 9 lines · Xiaoyao Li
Quoted 7 lines · Markus Armbruster
Quoted 4 lines · Xiaoyao Li

From: Isaku Yamahata <[email protected]>

Add a new property "private" to memory backends. When it's set to true, it indicates the RAMblock of the backend also requires kvm gmem.

Can you add a brief explanation why you need the property?

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 1 line

It provides a mechanism for user to specify whether the memory can serve as private memory (need request kvm gmem).

Yes, but why would a user want such memory?

Because KVM demands it for confidential guest, e.g., TDX guest. KVM demands that the mem slot needs to have KVM_MEM_PRIVATE set and has valid gmem associated if the guest accesses it as private memory.

Commit messages should explain why we want the patch. Documenting "why" is at least as important as "what". If "what" is missing, I can read the patch to find out. If "why" is missing, I'm reduced to guesswork.

Quoted 3 lines

I think as long as there is no demand to have a TDX guest with this property be set to "off", then just don't add it.

With a TDX VM, it will can be implicitly active. If we ever have to disable it for selective memory backends, we can add the property and have something like on/off/auto. For now it would be "auto".

Makes sense to me.

author
#17 lore raw

Re: [RFC PATCH v2 03/21] HostMem: Add private property and associate it with RAM_KVM_GMEM

Quoted 19 lines · Markus Armbruster
Quoted 17 lines · David Hildenbrand
Quoted 16 lines · Xiaoyao Li
Quoted 15 lines · Markus Armbruster
Quoted 10 lines · Xiaoyao Li
Quoted 7 lines · Markus Armbruster
Quoted 4 lines · Xiaoyao Li

From: Isaku Yamahata <[email protected]>

Add a new property "private" to memory backends. When it's set to true, it indicates the RAMblock of the backend also requires kvm gmem.

Can you add a brief explanation why you need the property?

It provides a mechanism for user to specify whether the memory can serve as private memory (need request kvm gmem).

Yes, but why would a user want such memory?

Quoted 7 lines
Quoted 3 lines
Quoted 3 lines

Because KVM demands it for confidential guest, e.g., TDX guest. KVM demands that the mem slot needs to have KVM_MEM_PRIVATE set and has valid gmem associated if the guest accesses it as private memory.

Commit messages should explain why we want the patch. Documenting "why" is at least as important as "what". If "what" is missing, I can read the patch to find out. If "why" is missing, I'm reduced to guesswork.

I'll try best to improve the commit message of this patch, and all other patches.

Quoted 5 lines
Quoted 3 lines

I think as long as there is no demand to have a TDX guest with this property be set to "off", then just don't add it.

With a TDX VM, it will can be implicitly active. If we ever have to disable it for selective memory backends, we can add the property and have something like on/off/auto. For now it would be "auto".

Makes sense to me.

OK. I think I get the answer of open #1 in cover letter.

If no other voice, I'll drop this patch and allocate gmem RAM when vm_type is TDX.

#18 lore raw

Re: [RFC PATCH v2 03/21] HostMem: Add private property and associate it with RAM_KVM_GMEM

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

I think as long as there is no demand to have a TDX guest with this property be set to "off", then just don't add it.

With a TDX VM, it will can be implicitly active. If we ever have to disable it for selective memory backends, we can add the property and have something like on/off/auto. For now it would be "auto".

Makes sense to me.

OK. I think I get the answer of open #1 in cover letter.

Yes.

Quoted 2 lines

If no other voice, I'll drop this patch and allocate gmem RAM when vm_type is TDX.

Good!

Signature · 3 lines
Cheers,

David / dhildenb
author
#19 lore raw

RFC PATCH v2 4/21 memory: Introduce memory_region_has_gmem_fd()

In reply to Xiaoyao Li

Introduce memory_region_has_gmem_fd() to query if the MemoryRegion has KVM gmem fd allocated.

---
 include/exec/memory.h | 10 ++++++++++
 softmmu/memory.c      |  5 +++++
 2 files changed, 15 insertions(+)
include/exec/memory.h +10 -0
index 227cb2578e95..4b8486ca3632 100644
@@ -1674,6 +1674,16 @@ static inline bool memory_region_is_romd(MemoryRegion *mr)
  */
 bool memory_region_is_protected(MemoryRegion *mr);
 
+/**
+ * memory_region_has_gmem_fd: check whether a memory region has KVM gmem fd
+ *     associated
+ *
+ * Returns %true if a memory region's ram_block has valid gmem fd assigned.
+ *
+ * @mr: the memory region being queried
+ */
+bool memory_region_has_gmem_fd(MemoryRegion *mr);
+
 /**
  * memory_region_get_iommu: check whether a memory region is an iommu
  *
softmmu/memory.c +5 -0
index 7d9494ce7028..e69a5f96d5d1 100644
@@ -1846,6 +1846,11 @@ bool memory_region_is_protected(MemoryRegion *mr)
     return mr->ram && (mr->ram_block->flags & RAM_PROTECTED);
 }
 
+bool memory_region_has_gmem_fd(MemoryRegion *mr)
+{
+    return mr->ram_block && mr->ram_block->gmem_fd >= 0;
+}
+
 uint8_t memory_region_get_dirty_log_mask(MemoryRegion *mr)
 {
     uint8_t mask = mr->dirty_log_mask;
Signature · 1 line
2.34.1
#20 lore raw

Re: [RFC PATCH v2 04/21] memory: Introduce memory_region_has_gmem_fd()

Quoted 2 lines · Xiaoyao Li

Introduce memory_region_has_gmem_fd() to query if the MemoryRegion has KVM gmem fd allocated.

*probably* best to just squash that into patch #2.

Signature · 3 lines
Cheers,

David / dhildenb
author
#21 lore raw

Re: [RFC PATCH v2 04/21] memory: Introduce memory_region_has_gmem_fd()

Quoted 5 lines · David Hildenbrand
Quoted 2 lines · Xiaoyao Li

Introduce memory_region_has_gmem_fd() to query if the MemoryRegion has KVM gmem fd allocated.

*probably* best to just squash that into patch #2.

Sure, I will do it.

author
#22 lore raw

RFC PATCH v2 5/21 kvm: Enable KVM_SET_USER_MEMORY_REGION2 for memslot

In reply to Xiaoyao Li

From: Chao Peng <[email protected]>

Switch to KVM_SET_USER_MEMORY_REGION2 when supported by KVM.

With KVM_SET_USER_MEMORY_REGION2, QEMU can set up memory region that backend'ed both by hva-based shared memory and gmem fd based private memory.

---
 accel/kvm/kvm-all.c      | 55 ++++++++++++++++++++++++++++++++++------
 accel/kvm/trace-events   |  2 +-
 include/sysemu/kvm_int.h |  2 ++
 3 files changed, 50 insertions(+), 9 deletions(-)
accel/kvm/kvm-all.c +47 -8
index 185ae16d9620..91cee0878366 100644
@@ -288,35 +288,68 @@ int kvm_physical_memory_addr_from_host(KVMState *s, void *ram,
 static int kvm_set_user_memory_region(KVMMemoryListener *kml, KVMSlot *slot, bool new)
 {
     KVMState *s = kvm_state;
-    struct kvm_userspace_memory_region mem;
+    struct kvm_userspace_memory_region2 mem;
+    static int cap_user_memory2 = -1;
     int ret;
 
+    if (cap_user_memory2 == -1) {
+        cap_user_memory2 = kvm_check_extension(s, KVM_CAP_USER_MEMORY2);
+    }
+
+    if (!cap_user_memory2 && slot->gmem_fd >= 0) {
+        error_report("%s, KVM doesn't support gmem!", __func__);
+        exit(1);
+    }
+
     mem.slot = slot->slot | (kml->as_id << 16);
     mem.guest_phys_addr = slot->start_addr;
     mem.userspace_addr = (unsigned long)slot->ram;
     mem.flags = slot->flags;
+    mem.gmem_fd = slot->gmem_fd;
+    mem.gmem_offset = slot->ofs;
 
     if (slot->memory_size && !new && (mem.flags ^ slot->old_flags) & KVM_MEM_READONLY) {
         /* Set the slot size to 0 before setting the slot to the desired
          * value. This is needed based on KVM commit 75d61fbc. */
         mem.memory_size = 0;
-        ret = kvm_vm_ioctl(s, KVM_SET_USER_MEMORY_REGION, &mem);
+
+        if (cap_user_memory2) {
+            ret = kvm_vm_ioctl(s, KVM_SET_USER_MEMORY_REGION2, &mem);
+        } else {
+            ret = kvm_vm_ioctl(s, KVM_SET_USER_MEMORY_REGION, &mem);
+	    }
         if (ret < 0) {
             goto err;
         }
     }
     mem.memory_size = slot->memory_size;
-    ret = kvm_vm_ioctl(s, KVM_SET_USER_MEMORY_REGION, &mem);
+    if (cap_user_memory2) {
+        ret = kvm_vm_ioctl(s, KVM_SET_USER_MEMORY_REGION2, &mem);
+    } else {
+        ret = kvm_vm_ioctl(s, KVM_SET_USER_MEMORY_REGION, &mem);
+    }
     slot->old_flags = mem.flags;
 err:
     trace_kvm_set_user_memory(mem.slot >> 16, (uint16_t)mem.slot, mem.flags,
                               mem.guest_phys_addr, mem.memory_size,
-                              mem.userspace_addr, ret);
+                              mem.userspace_addr, mem.gmem_fd,
+			      mem.gmem_offset, ret);
     if (ret < 0) {
-        error_report("%s: KVM_SET_USER_MEMORY_REGION failed, slot=%d,"
-                     " start=0x%" PRIx64 ", size=0x%" PRIx64 ": %s",
-                     __func__, mem.slot, slot->start_addr,
-                     (uint64_t)mem.memory_size, strerror(errno));
+        if (cap_user_memory2) {
+                error_report("%s: KVM_SET_USER_MEMORY_REGION2 failed, slot=%d,"
+                        " start=0x%" PRIx64 ", size=0x%" PRIx64 ","
+                        " flags=0x%" PRIx32 ","
+                        " gmem_fd=%" PRId32 ", gmem_offset=0x%" PRIx64 ": %s",
+                        __func__, mem.slot, slot->start_addr,
+                        (uint64_t)mem.memory_size, mem.flags,
+                        mem.gmem_fd, (uint64_t)mem.gmem_offset,
+                        strerror(errno));
+        } else {
+                error_report("%s: KVM_SET_USER_MEMORY_REGION failed, slot=%d,"
+                            " start=0x%" PRIx64 ", size=0x%" PRIx64 ": %s",
+                            __func__, mem.slot, slot->start_addr,
+                            (uint64_t)mem.memory_size, strerror(errno));
+        }
     }
     return ret;
 }
@@ -472,6 +505,9 @@ static int kvm_mem_flags(MemoryRegion *mr)
     if (readonly && kvm_readonly_mem_allowed) {
         flags |= KVM_MEM_READONLY;
     }
+    if (memory_region_has_gmem_fd(mr)) {
+        flags |= KVM_MEM_PRIVATE;
+    }
     return flags;
 }
 
@@ -1402,6 +1438,9 @@ static void kvm_set_phys_mem(KVMMemoryListener *kml,
         mem->ram_start_offset = ram_start_offset;
         mem->ram = ram;
         mem->flags = kvm_mem_flags(mr);
+        mem->gmem_fd = mr->ram_block->gmem_fd;
+        mem->ofs = (uint8_t*)ram - mr->ram_block->host;
+
         kvm_slot_init_dirty_bitmap(mem);
         err = kvm_set_user_memory_region(kml, mem, true);
         if (err) {
accel/kvm/trace-events +1 -1
index 14ebfa1b991c..80694683acea 100644
@@ -15,7 +15,7 @@ kvm_irqchip_update_msi_route(int virq) "Updating MSI route virq=%d"
 kvm_irqchip_release_virq(int virq) "virq %d"
 kvm_set_ioeventfd_mmio(int fd, uint64_t addr, uint32_t val, bool assign, uint32_t size, bool datamatch) "fd: %d @0x%" PRIx64 " val=0x%x assign: %d size: %d match: %d"
 kvm_set_ioeventfd_pio(int fd, uint16_t addr, uint32_t val, bool assign, uint32_t size, bool datamatch) "fd: %d @0x%x val=0x%x assign: %d size: %d match: %d"
-kvm_set_user_memory(uint16_t as, uint16_t slot, uint32_t flags, uint64_t guest_phys_addr, uint64_t memory_size, uint64_t userspace_addr, int ret) "AddrSpace#%d Slot#%d flags=0x%x gpa=0x%"PRIx64 " size=0x%"PRIx64 " ua=0x%"PRIx64 " ret=%d"
+kvm_set_user_memory(uint16_t as, uint16_t slot, uint32_t flags, uint64_t guest_phys_addr, uint64_t memory_size, uint64_t userspace_addr, uint32_t fd, uint64_t fd_offset, int ret) "AddrSpace#%d Slot#%d flags=0x%x gpa=0x%"PRIx64 " size=0x%"PRIx64 " ua=0x%"PRIx64 " gmem_fd=%d" " gmem_fd_offset=0x%" PRIx64 " ret=%d"
 kvm_clear_dirty_log(uint32_t slot, uint64_t start, uint32_t size) "slot#%"PRId32" start 0x%"PRIx64" size 0x%"PRIx32
 kvm_resample_fd_notify(int gsi) "gsi %d"
 kvm_dirty_ring_full(int id) "vcpu %d"
include/sysemu/kvm_int.h +2 -0
index 511b42bde5c4..9990889d3494 100644
@@ -30,6 +30,8 @@ typedef struct KVMSlot
     int as_id;
     /* Cache of the offset in ram address space */
     ram_addr_t ram_start_offset;
+    int gmem_fd;
+    hwaddr ofs;
 } KVMSlot;
 
 typedef struct KVMMemoryUpdate {
Signature · 1 line
2.34.1
#23 lore raw

Re: [RFC PATCH v2 05/21] kvm: Enable KVM_SET_USER_MEMORY_REGION2 for memslot

Quoted 10 lines · Xiaoyao Li

From: Chao Peng <[email protected]>

Switch to KVM_SET_USER_MEMORY_REGION2 when supported by KVM.

With KVM_SET_USER_MEMORY_REGION2, QEMU can set up memory region that backend'ed both by hva-based shared memory and gmem fd based private memory.

"Co-developed-by".

Signature · 3 lines
Cheers,

David / dhildenb
author
#24 lore raw

Re: [RFC PATCH v2 05/21] kvm: Enable KVM_SET_USER_MEMORY_REGION2 for memslot

Quoted 5 lines · David Hildenbrand
Quoted 4 lines · Xiaoyao Li

From: Chao Peng <[email protected]>

Switch to KVM_SET_USER_MEMORY_REGION2 when supported by KVM.

Quoted 8 lines
Quoted 6 lines

With KVM_SET_USER_MEMORY_REGION2, QEMU can set up memory region that backend'ed both by hva-based shared memory and gmem fd based private memory.

"Co-developed-by".

I will fix it, thanks!

author
#25 lore raw

RFC PATCH v2 6/21 i386: Add support for sw-protected-vm object

In reply to Xiaoyao Li

Introduce sw-protected-vm object which implements the interface of CONFIDENTIAL_GUEST_SUPPORT, and will be used to create X86_SW_PROTECTED_VM via

  $qemu -machine ...,confidential-guest-support=sp-vm0	\
        -object sw-protected-vm,id=sp-vm0
---
 qapi/qom.json                     |  1 +
 target/i386/kvm/meson.build       |  1 +
 target/i386/kvm/sw-protected-vm.c | 35 +++++++++++++++++++++++++++++++
 target/i386/kvm/sw-protected-vm.h | 17 +++++++++++++++
 4 files changed, 54 insertions(+)
 create mode 100644 target/i386/kvm/sw-protected-vm.c
 create mode 100644 target/i386/kvm/sw-protected-vm.h
qapi/qom.json +1 -0
index d28c5403bc0f..be054ee2f348 100644
@@ -944,6 +944,7 @@
       'if': 'CONFIG_SECRET_KEYRING' },
     'sev-guest',
     'thread-context',
+    'sw-protected-vm',
     's390-pv-guest',
     'throttle-group',
     'tls-creds-anon',
target/i386/kvm/meson.build +1 -0
index 40fbde96cac6..a31e760b3f19 100644
@@ -5,6 +5,7 @@ i386_softmmu_kvm_ss = ss.source_set()
 i386_softmmu_kvm_ss.add(files(
   'kvm.c',
   'kvm-cpu.c',
+  'sw-protected-vm.c',
 ))
 
 i386_softmmu_kvm_ss.add(when: 'CONFIG_XEN_EMU', if_true: files('xen-emu.c'))
target/i386/kvm/sw-protected-vm.c new file +35 -0
new file mode 100644
index 000000000000..62a1d3d5d3fe
@@ -0,0 +1,35 @@
+/*
+ * QEMU X86_SW_PROTECTED_VM SUPPORT
+ *
+ * Author:
+ *      Xiaoyao Li <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory
+ *
+ */
+
+#include "qemu/osdep.h"
+#include "qom/object_interfaces.h"
+
+#include "sw-protected-vm.h"
+
+/* x86-sw-protected-vm */
+OBJECT_DEFINE_TYPE_WITH_INTERFACES(SwProtectedVm,
+                                   sw_protected_vm,
+                                   SW_PROTECTED_VM,
+                                   CONFIDENTIAL_GUEST_SUPPORT,
+                                   { TYPE_USER_CREATABLE },
+                                   { NULL })
+
+static void sw_protected_vm_init(Object *obj)
+{
+}
+
+static void sw_protected_vm_finalize(Object *obj)
+{
+}
+
+static void sw_protected_vm_class_init(ObjectClass *oc, void *data)
+{
+}
target/i386/kvm/sw-protected-vm.h new file +17 -0
new file mode 100644
index 000000000000..db192a81c75e
@@ -0,0 +1,17 @@
+#ifndef QEMU_I386_SW_PROTECTED_VM_H
+#define QEMU_I386_SW_PROTECTED_VM_H
+
+#include "exec/confidential-guest-support.h"
+
+#define TYPE_SW_PROTECTED_VM    "sw-protected-vm"
+#define SW_PROTECTED_VM(obj)    OBJECT_CHECK(SwProtectedVm, (obj), TYPE_SW_PROTECTED_VM)
+
+typedef struct SwProtectedVmClass {
+    ConfidentialGuestSupportClass parent_class;
+} SwProtectedVmClass;
+
+typedef struct SwProtectedVm {
+    ConfidentialGuestSupport parent_obj;
+} SwProtectedVm;
+
+#endif /* QEMU_I386_SW_PROTECTED_VM_H */
Signature · 1 line
2.34.1
author
#26 lore raw

RFC PATCH v2 7/21 i386/pc: Drop pc_machine_kvm_type()

In reply to Xiaoyao Li

pc_machine_kvm_type() was introduced by commit e21be724eaf5 ("i386/xen:
add pc_machine_kvm_type to initialize XEN_EMULATE mode") to do Xen specific initialization by utilizing kvm_type method.

commit eeedfe6c6316 ("hw/xen: Simplify emulated Xen platform init") moves the Xen specific initialization to pc_basic_device_init().

There is no need to keep the PC specific kvm_type() implementation anymore. On the other hand, later patch will implement kvm_type() method for all x86/i386 machines to support KVM_X86_SW_PROTECTED_VM.

---
 hw/i386/pc.c         | 5 -----
 include/hw/i386/pc.h | 3 ---
 2 files changed, 8 deletions(-)
hw/i386/pc.c +0 -5
index 3109d5e0e035..abeadd903827 100644
@@ -1794,11 +1794,6 @@ static void pc_machine_initfn(Object *obj)
     cxl_machine_init(obj, &pcms->cxl_devices_state);
 }
 
-int pc_machine_kvm_type(MachineState *machine, const char *kvm_type)
-{
-    return 0;
-}
-
 static void pc_machine_reset(MachineState *machine, ShutdownCause reason)
 {
     CPUState *cs;
include/hw/i386/pc.h +0 -3
index d54e8b1101e4..c98d628a76f3 100644
@@ -296,15 +296,12 @@ extern const size_t pc_compat_1_5_len;
 extern GlobalProperty pc_compat_1_4[];
 extern const size_t pc_compat_1_4_len;
 
-int pc_machine_kvm_type(MachineState *machine, const char *vm_type);
-
 #define DEFINE_PC_MACHINE(suffix, namestr, initfn, optsfn) \
     static void pc_machine_##suffix##_class_init(ObjectClass *oc, void *data) \
     { \
         MachineClass *mc = MACHINE_CLASS(oc); \
         optsfn(mc); \
         mc->init = initfn; \
-        mc->kvm_type = pc_machine_kvm_type; \
     } \
     static const TypeInfo pc_machine_type_##suffix = { \
         .name       = namestr TYPE_MACHINE_SUFFIX, \
Signature · 1 line
2.34.1
Reviewed-by on v2 7/21
#27 lore raw

Re: [RFC PATCH v2 07/21] i386/pc: Drop pc_machine_kvm_type()

Quoted 9 lines · Xiaoyao Li

pc_machine_kvm_type() was introduced by commit e21be724eaf5 ("i386/xen:
add pc_machine_kvm_type to initialize XEN_EMULATE mode") to do Xen specific initialization by utilizing kvm_type method.

commit eeedfe6c6316 ("hw/xen: Simplify emulated Xen platform init") moves the Xen specific initialization to pc_basic_device_init().

There is no need to keep the PC specific kvm_type() implementation anymore.

So we'll fallback to kvm_arch_get_default_type(), which simply returns 0.

Quoted 3 lines

On the other hand, later patch will implement kvm_type() method for all x86/i386 machines to support KVM_X86_SW_PROTECTED_VM.

^ I suggest dropping that and merging that patch ahead-of-time as a simple cleanup.

Signature · 3 lines
Cheers,

David / dhildenb
author
#28 lore raw

Re: [RFC PATCH v2 07/21] i386/pc: Drop pc_machine_kvm_type()

Quoted 11 lines · David Hildenbrand
Quoted 9 lines · Xiaoyao Li

pc_machine_kvm_type() was introduced by commit e21be724eaf5 ("i386/xen:
add pc_machine_kvm_type to initialize XEN_EMULATE mode") to do Xen specific initialization by utilizing kvm_type method.

commit eeedfe6c6316 ("hw/xen: Simplify emulated Xen platform init") moves the Xen specific initialization to pc_basic_device_init().

There is no need to keep the PC specific kvm_type() implementation anymore.

Quoted 7 lines

So we'll fallback to kvm_arch_get_default_type(), which simply returns 0.

Quoted 3 lines

On the other hand, later patch will implement kvm_type() method for all x86/i386 machines to support KVM_X86_SW_PROTECTED_VM.

^ I suggest dropping that and merging that patch ahead-of-time as a simple cleanup.

I suppose the "that" here means "this patch", right?

If so, I can submit this patch separately.

Quoted 2 lines
#29 lore raw

Re: [RFC PATCH v2 07/21] i386/pc: Drop pc_machine_kvm_type()

Quoted 14 lines · Xiaoyao Li
Quoted 13 lines · David Hildenbrand
Quoted 9 lines · Xiaoyao Li

pc_machine_kvm_type() was introduced by commit e21be724eaf5 ("i386/xen:
add pc_machine_kvm_type to initialize XEN_EMULATE mode") to do Xen specific initialization by utilizing kvm_type method.

commit eeedfe6c6316 ("hw/xen: Simplify emulated Xen platform init") moves the Xen specific initialization to pc_basic_device_init().

There is no need to keep the PC specific kvm_type() implementation anymore.

So we'll fallback to kvm_arch_get_default_type(), which simply returns 0.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

On the other hand, later patch will implement kvm_type() method for all x86/i386 machines to support KVM_X86_SW_PROTECTED_VM.

^ I suggest dropping that and merging that patch ahead-of-time as a simple cleanup.

I suppose the "that" here means "this patch", right?

With that I meant that paragraph "On the other hand, later patch ...".

Signature · 3 lines
Cheers,

David / dhildenb
Acked-by on v2 7/21
#30 lore raw

Re: [RFC PATCH v2 07/21] i386/pc: Drop pc_machine_kvm_type()

In reply to Xiaoyao Li
Quoted 8 lines · Xiaoyao Li

pc_machine_kvm_type() was introduced by commit e21be724eaf5 ("i386/xen:
add pc_machine_kvm_type to initialize XEN_EMULATE mode") to do Xen specific initialization by utilizing kvm_type method.

commit eeedfe6c6316 ("hw/xen: Simplify emulated Xen platform init") moves the Xen specific initialization to pc_basic_device_init().

Quoted 6 lines

There is no need to keep the PC specific kvm_type() implementation anymore. On the other hand, later patch will implement kvm_type() method for all x86/i386 machines to support KVM_X86_SW_PROTECTED_VM.

Indeed, I added it and then later ripped everything out of it and left it empty, as you nicely describe (thanks) in your commit message. I have no designs on using it again, so

author
#31 lore raw

RFC PATCH v2 8/21 target/i386: Implement mc->kvm_type() to get VM type

In reply to Xiaoyao Li

Implement mc->kvm_type() for i386 machines. It provides a way for user to create SW_PROTECTE_VM.

Also store the vm_type in machinestate to other code to query what the VM type is.

---
 hw/i386/x86.c              | 12 ++++++++++++
 include/hw/i386/x86.h      |  1 +
 target/i386/kvm/kvm.c      | 30 ++++++++++++++++++++++++++++++
 target/i386/kvm/kvm_i386.h |  1 +
 4 files changed, 44 insertions(+)
hw/i386/x86.c +12 -0
index a88a126123be..660f83935315 100644
@@ -1382,6 +1382,17 @@ static void machine_set_sgx_epc(Object *obj, Visitor *v, const char *name,
     qapi_free_SgxEPCList(list);
 }
 
+static int x86_kvm_type(MachineState *ms, const char *vm_type)
+{
+    X86MachineState *x86ms = X86_MACHINE(ms);
+    int kvm_type;
+
+    kvm_type = kvm_get_vm_type(ms, vm_type);
+    x86ms->vm_type = kvm_type;
+
+    return kvm_type;
+}
+
 static void x86_machine_initfn(Object *obj)
 {
     X86MachineState *x86ms = X86_MACHINE(obj);
@@ -1406,6 +1417,7 @@ static void x86_machine_class_init(ObjectClass *oc, void *data)
     mc->cpu_index_to_instance_props = x86_cpu_index_to_props;
     mc->get_default_cpu_node_id = x86_get_default_cpu_node_id;
     mc->possible_cpu_arch_ids = x86_possible_cpu_arch_ids;
+    mc->kvm_type = x86_kvm_type;
     x86mc->save_tsc_khz = true;
     x86mc->fwcfg_dma_enabled = true;
     nc->nmi_monitor_handler = x86_nmi;
include/hw/i386/x86.h +1 -0
index da19ae15463a..ab1d38569019 100644
@@ -41,6 +41,7 @@ struct X86MachineState {
     MachineState parent;
 
     /*< public >*/
+    unsigned int vm_type;
 
     /* Pointers to devices and objects: */
     ISADevice *rtc;
target/i386/kvm/kvm.c +30 -0
index f8cc8eb1fe70..d1cf6c1f63b3 100644
@@ -32,6 +32,7 @@
 #include "sysemu/runstate.h"
 #include "kvm_i386.h"
 #include "sev.h"
+#include "sw-protected-vm.h"
 #include "xen-emu.h"
 #include "hyperv.h"
 #include "hyperv-proto.h"
@@ -154,6 +155,35 @@ static KVMMSRHandlers msr_handlers[KVM_MSR_FILTER_MAX_RANGES];
 static RateLimit bus_lock_ratelimit_ctrl;
 static int kvm_get_one_msr(X86CPU *cpu, int index, uint64_t *value);
 
+static const char* vm_type_name[] = {
+    [KVM_X86_DEFAULT_VM] = "default",
+    [KVM_X86_SW_PROTECTED_VM] = "sw-protected-vm",
+};
+
+int kvm_get_vm_type(MachineState *ms, const char *vm_type)
+{
+    int kvm_type = KVM_X86_DEFAULT_VM;
+
+    if (ms->cgs && object_dynamic_cast(OBJECT(ms->cgs), TYPE_SW_PROTECTED_VM)) {
+        kvm_type = KVM_X86_SW_PROTECTED_VM;
+    }
+
+    /*
+     * old KVM doesn't support KVM_CAP_VM_TYPES and KVM_X86_DEFAULT_VM
+     * is always supported
+     */
+    if (kvm_type == KVM_X86_DEFAULT_VM) {
+        return kvm_type;
+    }
+
+    if (!(kvm_check_extension(KVM_STATE(ms->accelerator), KVM_CAP_VM_TYPES) & BIT(kvm_type))) {
+        error_report("vm-type %s not supported by KVM", vm_type_name[kvm_type]);
+        exit(1);
+    }
+
+    return kvm_type;
+}
+
 int kvm_has_pit_state2(void)
 {
     return has_pit_state2;
target/i386/kvm/kvm_i386.h +1 -0
index e24753abfe6a..ea3a5b174ac0 100644
@@ -37,6 +37,7 @@ bool kvm_has_adjust_clock(void);
 bool kvm_has_adjust_clock_stable(void);
 bool kvm_has_exception_payload(void);
 void kvm_synchronize_all_tsc(void);
+int kvm_get_vm_type(MachineState *ms, const char *vm_type);
 void kvm_arch_reset_vcpu(X86CPU *cs);
 void kvm_arch_after_reset_vcpu(X86CPU *cpu);
 void kvm_arch_do_init_vcpu(X86CPU *cs);
Signature · 1 line
2.34.1
author
#32 lore raw

RFC PATCH v2 9/21 target/i386: Introduce kvm_confidential_guest_init()

In reply to Xiaoyao Li

Introduce a separate function kvm_confidential_guest_init(), which dispatches specific confidential guest initialization function by ms->cgs type.

---
 target/i386/kvm/kvm.c | 11 ++++++++++-
 target/i386/sev.c     |  1 -
 target/i386/sev.h     |  2 ++
 3 files changed, 12 insertions(+), 2 deletions(-)
target/i386/kvm/kvm.c +10 -1
index d1cf6c1f63b3..fb1be16471b4 100644
@@ -2583,6 +2583,15 @@ static void register_smram_listener(Notifier *n, void *unused)
                                  &smram_address_space, 1, "kvm-smram");
 }
 
+static int kvm_confidential_guest_init(MachineState *ms, Error **errp)
+{
+    if (object_dynamic_cast(OBJECT(ms->cgs), TYPE_SEV_GUEST)) {
+        return sev_kvm_init(ms->cgs, errp);
+    }
+
+    return 0;
+}
+
 int kvm_arch_init(MachineState *ms, KVMState *s)
 {
     uint64_t identity_base = 0xfffbc000;
@@ -2603,7 +2612,7 @@ int kvm_arch_init(MachineState *ms, KVMState *s)
      * mechanisms are supported in future (e.g. TDX), they'll need
      * their own initialization either here or elsewhere.
      */
-    ret = sev_kvm_init(ms->cgs, &local_err);
+    ret = kvm_confidential_guest_init(ms, &local_err);
     if (ret < 0) {
         error_report_err(local_err);
         return ret;
target/i386/sev.c +0 -1
index fe2144c0388b..5aa04863846d 100644
@@ -39,7 +39,6 @@
 #include "hw/i386/pc.h"
 #include "exec/address-spaces.h"
 
-#define TYPE_SEV_GUEST "sev-guest"
 OBJECT_DECLARE_SIMPLE_TYPE(SevGuestState, SEV_GUEST)
 
 
target/i386/sev.h +2 -0
index 7b1528248a54..64fbf186dbd2 100644
@@ -20,6 +20,8 @@
 
 #include "exec/confidential-guest-support.h"
 
+#define TYPE_SEV_GUEST "sev-guest"
+
 #define SEV_POLICY_NODBG        0x1
 #define SEV_POLICY_NOKS         0x2
 #define SEV_POLICY_ES           0x4
Signature · 1 line
2.34.1
author
#33 lore raw

RFC PATCH v2 10/21 i386/kvm: Implement kvm_sw_protected_vm_init() for sw-protcted-vm specific functions

In reply to Xiaoyao Li
---
 target/i386/kvm/kvm.c             |  2 ++
 target/i386/kvm/sw-protected-vm.c | 10 ++++++++++
 target/i386/kvm/sw-protected-vm.h |  2 ++
 3 files changed, 14 insertions(+)
target/i386/kvm/kvm.c +2 -0
index fb1be16471b4..e126bf4e7ddd 100644
@@ -2587,6 +2587,8 @@ static int kvm_confidential_guest_init(MachineState *ms, Error **errp)
 {
     if (object_dynamic_cast(OBJECT(ms->cgs), TYPE_SEV_GUEST)) {
         return sev_kvm_init(ms->cgs, errp);
+    } else if (object_dynamic_cast(OBJECT(ms->cgs), TYPE_SW_PROTECTED_VM)) {
+        return sw_protected_vm_kvm_init(ms, errp);
     }
 
     return 0;
target/i386/kvm/sw-protected-vm.c +10 -0
index 62a1d3d5d3fe..3cfcc89202a6 100644
@@ -10,10 +10,20 @@
  */
 
 #include "qemu/osdep.h"
+#include "qapi/error.h"
 #include "qom/object_interfaces.h"
 
+#include "hw/i386/x86.h"
 #include "sw-protected-vm.h"
 
+int sw_protected_vm_kvm_init(MachineState *ms, Error **errp)
+{
+    SwProtectedVm *spvm = SW_PROTECTED_VM(OBJECT(ms->cgs));
+
+    spvm->parent_obj.ready = true;
+    return 0;
+}
+
 /* x86-sw-protected-vm */
 OBJECT_DEFINE_TYPE_WITH_INTERFACES(SwProtectedVm,
                                    sw_protected_vm,
target/i386/kvm/sw-protected-vm.h +2 -0
index db192a81c75e..15f63bfc7c60 100644
@@ -14,4 +14,6 @@ typedef struct SwProtectedVm {
     ConfidentialGuestSupport parent_obj;
 } SwProtectedVm;
 
+int sw_protected_vm_kvm_init(MachineState *ms, Error **errp);
+
 #endif /* QEMU_I386_SW_PROTECTED_VM_H */
Signature · 1 line
2.34.1
author
#34 lore raw

RFC PATCH v2 11/21 kvm: Introduce support for memory_attributes

In reply to Xiaoyao Li

Introcude the helper functions to set the attributes of a range of memory to private and shared.

---
 accel/kvm/kvm-all.c  | 43 +++++++++++++++++++++++++++++++++++++++++++
 include/sysemu/kvm.h |  3 +++
 2 files changed, 46 insertions(+)
accel/kvm/kvm-all.c +43 -0
index 91cee0878366..eeccc6317fa9 100644
@@ -105,6 +105,7 @@ bool kvm_msi_use_devid;
 bool kvm_has_guest_debug;
 static int kvm_sstep_flags;
 static bool kvm_immediate_exit;
+static uint64_t kvm_supported_memory_attributes;
 static hwaddr kvm_max_slot_size = ~0;
 
 static const KVMCapabilityInfo kvm_required_capabilites[] = {
@@ -1343,6 +1344,44 @@ void kvm_set_max_memslot_size(hwaddr max_slot_size)
     kvm_max_slot_size = max_slot_size;
 }
 
+static int kvm_set_memory_attributes(hwaddr start, hwaddr size, uint64_t attr)
+{
+    struct kvm_memory_attributes attrs;
+    int r;
+
+    attrs.attributes = attr;
+    attrs.address = start;
+    attrs.size = size;
+    attrs.flags = 0;
+
+    r = kvm_vm_ioctl(kvm_state, KVM_SET_MEMORY_ATTRIBUTES, &attrs);
+    if (r) {
+        warn_report("%s: failed to set memory (0x%lx+%#zx) with attr 0x%lx error '%s'",
+                     __func__, start, size, attr, strerror(errno));
+    }
+    return r;
+}
+
+int kvm_set_memory_attributes_private(hwaddr start, hwaddr size)
+{
+    if (!(kvm_supported_memory_attributes & KVM_MEMORY_ATTRIBUTE_PRIVATE)) {
+        error_report("KVM doesn't support PRIVATE memory attribute\n");
+        return -EINVAL;
+    }
+
+    return kvm_set_memory_attributes(start, size, KVM_MEMORY_ATTRIBUTE_PRIVATE);
+}
+
+int kvm_set_memory_attributes_shared(hwaddr start, hwaddr size)
+{
+    if (!(kvm_supported_memory_attributes & KVM_MEMORY_ATTRIBUTE_PRIVATE)) {
+        error_report("KVM doesn't support PRIVATE memory attribute\n");
+        return -EINVAL;
+    }
+
+    return kvm_set_memory_attributes(start, size, 0);
+}
+
 /* Called with KVMMemoryListener.slots_lock held */
 static void kvm_set_phys_mem(KVMMemoryListener *kml,
                              MemoryRegionSection *section, bool add)
@@ -2556,6 +2595,10 @@ static int kvm_init(MachineState *ms)
     }
     s->as = g_new0(struct KVMAs, s->nr_as);
 
+    if (kvm_check_extension(s, KVM_CAP_MEMORY_ATTRIBUTES)) {
+        kvm_supported_memory_attributes = kvm_ioctl(s, KVM_GET_SUPPORTED_MEMORY_ATTRIBUTES, 0);
+    }
+
     if (object_property_find(OBJECT(current_machine), "kvm-type")) {
         g_autofree char *kvm_type = object_property_get_str(OBJECT(current_machine),
                                                             "kvm-type",
include/sysemu/kvm.h +3 -0
index f5b74c8dd8c5..0f78f1246c7f 100644
@@ -582,4 +582,7 @@ bool kvm_dirty_ring_enabled(void);
 uint32_t kvm_dirty_ring_size(void);
 
 int kvm_create_guest_memfd(uint64_t size, uint64_t flags, Error **errp);
+
+int kvm_set_memory_attributes_private(hwaddr start, hwaddr size);
+int kvm_set_memory_attributes_shared(hwaddr start, hwaddr size);
 #endif
Signature · 1 line
2.34.1
author
#35 lore raw

RFC PATCH v2 12/21 kvm/memory: Introduce the infrastructure to set the default shared/private value

In reply to Xiaoyao Li

Introduce new flag RAM_DEFAULT_PRIVATE for RAMBlock. It's used to indicate the default attribute, private or not.

Set the RAM range to private explicitly when it's default private.

Originated-from: Isaku Yamahata <[email protected]>

---
 accel/kvm/kvm-all.c   | 10 ++++++++++
 include/exec/memory.h |  6 ++++++
 softmmu/memory.c      | 13 +++++++++++++
 3 files changed, 29 insertions(+)
accel/kvm/kvm-all.c +10 -0
index eeccc6317fa9..7e32ee83b258 100644
@@ -1487,6 +1487,16 @@ static void kvm_set_phys_mem(KVMMemoryListener *kml,
                     strerror(-err));
             abort();
         }
+
+        if (memory_region_is_default_private(mr)) {
+            err = kvm_set_memory_attributes_private(start_addr, slot_size);
+            if (err) {
+                error_report("%s: failed to set memory attribute private: %s\n",
+                             __func__, strerror(-err));
+                exit(1);
+            }
+        }
+
         start_addr += slot_size;
         ram_start_offset += slot_size;
         ram += slot_size;
include/exec/memory.h +6 -0
index 4b8486ca3632..2c738b5dc420 100644
@@ -238,6 +238,9 @@ typedef struct IOMMUTLBEvent {
 /* RAM can be private that has kvm gmem backend */
 #define RAM_KVM_GMEM    (1 << 10)
 
+/* RAM is default private */
+#define RAM_DEFAULT_PRIVATE     (1 << 11)
+
 static inline void iommu_notifier_init(IOMMUNotifier *n, IOMMUNotify fn,
                                        IOMMUNotifierFlag flags,
                                        hwaddr start, hwaddr end,
@@ -1684,6 +1687,9 @@ bool memory_region_is_protected(MemoryRegion *mr);
  */
 bool memory_region_has_gmem_fd(MemoryRegion *mr);
 
+void memory_region_set_default_private(MemoryRegion *mr);
+bool memory_region_is_default_private(MemoryRegion *mr);
+
 /**
  * memory_region_get_iommu: check whether a memory region is an iommu
  *
softmmu/memory.c +13 -0
index e69a5f96d5d1..dc5d0d7703b5 100644
@@ -1851,6 +1851,19 @@ bool memory_region_has_gmem_fd(MemoryRegion *mr)
     return mr->ram_block && mr->ram_block->gmem_fd >= 0;
 }
 
+bool memory_region_is_default_private(MemoryRegion *mr)
+{
+    return memory_region_has_gmem_fd(mr) &&
+           (mr->ram_block->flags & RAM_DEFAULT_PRIVATE);
+}
+
+void memory_region_set_default_private(MemoryRegion *mr)
+{
+    if (memory_region_has_gmem_fd(mr)) {
+        mr->ram_block->flags |= RAM_DEFAULT_PRIVATE;
+    }
+}
+
 uint8_t memory_region_get_dirty_log_mask(MemoryRegion *mr)
 {
     uint8_t mask = mr->dirty_log_mask;
Signature · 1 line
2.34.1
author
#36 lore raw

RFC PATCH v2 13/21 i386/kvm: Set memory to default private for KVM_X86_SW_PROTECTED_VM

In reply to Xiaoyao Li

Register a memory listener for KVM_X86_SW_PROVTED_VM. It set RAM to private by default.

---
 include/exec/memory.h             |  1 +
 target/i386/kvm/sw-protected-vm.c | 18 ++++++++++++++++++
 2 files changed, 19 insertions(+)
include/exec/memory.h +1 -0
index 2c738b5dc420..a2602b783a38 100644
@@ -820,6 +820,7 @@ struct IOMMUMemoryRegion {
 #define MEMORY_LISTENER_PRIORITY_MIN            0
 #define MEMORY_LISTENER_PRIORITY_ACCEL          10
 #define MEMORY_LISTENER_PRIORITY_DEV_BACKEND    10
+#define MEMORY_LISTENER_PRIORITY_ACCEL_HIGH     20
 
 /**
  * struct MemoryListener: callbacks structure for updates to the physical memory map
target/i386/kvm/sw-protected-vm.c +18 -0
index 3cfcc89202a6..f47ac383e1dd 100644
@@ -12,14 +12,32 @@
 #include "qemu/osdep.h"
 #include "qapi/error.h"
 #include "qom/object_interfaces.h"
+#include "exec/address-spaces.h"
+#include "sysemu/kvm.h"
 
 #include "hw/i386/x86.h"
 #include "sw-protected-vm.h"
 
+static void kvm_x86_sw_protected_vm_region_add(MemoryListener *listenr,
+                                               MemoryRegionSection *section)
+{
+    memory_region_set_default_private(section->mr);
+}
+
+static MemoryListener kvm_x86_sw_protected_vm_memory_listener = {
+    .name = "kvm_x86_sw_protected_vm_memory_listener",
+    .region_add = kvm_x86_sw_protected_vm_region_add,
+    /* Higher than KVM memory listener = 10. */
+    .priority = MEMORY_LISTENER_PRIORITY_ACCEL_HIGH,
+};
+
 int sw_protected_vm_kvm_init(MachineState *ms, Error **errp)
 {
     SwProtectedVm *spvm = SW_PROTECTED_VM(OBJECT(ms->cgs));
 
+    memory_listener_register(&kvm_x86_sw_protected_vm_memory_listener,
+                             &address_space_memory);
+
     spvm->parent_obj.ready = true;
     return 0;
 }
Signature · 1 line
2.34.1
author
#37 lore raw

RFC PATCH v2 14/21 physmem: replace function name with __func__ in ram_block_discard_range()

In reply to Xiaoyao Li
---
 softmmu/physmem.c | 34 +++++++++++++++-------------------
 1 file changed, 15 insertions(+), 19 deletions(-)
softmmu/physmem.c +15 -19
index 2d98a88f41f0..34d580ec0d39 100644
@@ -3440,16 +3440,15 @@ int ram_block_discard_range(RAMBlock *rb, uint64_t start, size_t length)
     uint8_t *host_startaddr = rb->host + start;
 
     if (!QEMU_PTR_IS_ALIGNED(host_startaddr, rb->page_size)) {
-        error_report("ram_block_discard_range: Unaligned start address: %p",
-                     host_startaddr);
+        error_report("%s: Unaligned start address: %p",
+                     __func__, host_startaddr);
         goto err;
     }
 
     if ((start + length) <= rb->max_length) {
         bool need_madvise, need_fallocate;
         if (!QEMU_IS_ALIGNED(length, rb->page_size)) {
-            error_report("ram_block_discard_range: Unaligned length: %zx",
-                         length);
+            error_report("%s: Unaligned length: %zx", __func__, length);
             goto err;
         }
 
@@ -3479,27 +3478,26 @@ int ram_block_discard_range(RAMBlock *rb, uint64_t start, size_t length)
              * file.
              */
             if (!qemu_ram_is_shared(rb)) {
-                warn_report_once("ram_block_discard_range: Discarding RAM"
+                warn_report_once("%s: Discarding RAM"
                                  " in private file mappings is possibly"
                                  " dangerous, because it will modify the"
                                  " underlying file and will affect other"
-                                 " users of the file");
+                                 " users of the file", __func__);
             }
 
             ret = fallocate(rb->fd, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE,
                             start, length);
             if (ret) {
                 ret = -errno;
-                error_report("ram_block_discard_range: Failed to fallocate "
-                             "%s:%" PRIx64 " +%zx (%d)",
-                             rb->idstr, start, length, ret);
+                error_report("%s: Failed to fallocate %s:%" PRIx64 " +%zx (%d)",
+                             __func__, rb->idstr, start, length, ret);
                 goto err;
             }
 #else
             ret = -ENOSYS;
-            error_report("ram_block_discard_range: fallocate not available/file"
+            error_report("%s: fallocate not available/file"
                          "%s:%" PRIx64 " +%zx (%d)",
-                         rb->idstr, start, length, ret);
+                         __func__, rb->idstr, start, length, ret);
             goto err;
 #endif
         }
@@ -3517,25 +3515,23 @@ int ram_block_discard_range(RAMBlock *rb, uint64_t start, size_t length)
             }
             if (ret) {
                 ret = -errno;
-                error_report("ram_block_discard_range: Failed to discard range "
+                error_report("%s: Failed to discard range "
                              "%s:%" PRIx64 " +%zx (%d)",
-                             rb->idstr, start, length, ret);
+                             __func__, rb->idstr, start, length, ret);
                 goto err;
             }
 #else
             ret = -ENOSYS;
-            error_report("ram_block_discard_range: MADVISE not available"
-                         "%s:%" PRIx64 " +%zx (%d)",
-                         rb->idstr, start, length, ret);
+            error_report("%s: MADVISE not available %s:%" PRIx64 " +%zx (%d)",
+                         __func__, rb->idstr, start, length, ret);
             goto err;
 #endif
         }
         trace_ram_block_discard_range(rb->idstr, host_startaddr, length,
                                       need_madvise, need_fallocate, ret);
     } else {
-        error_report("ram_block_discard_range: Overrun block '%s' (%" PRIu64
-                     "/%zx/" RAM_ADDR_FMT")",
-                     rb->idstr, start, length, rb->max_length);
+        error_report("%s: Overrun block '%s' (%" PRIu64 "/%zx/" RAM_ADDR_FMT")",
+                     __func__, rb->idstr, start, length, rb->max_length);
     }
 
 err:
Signature · 1 line
2.34.1
author
#38 lore raw

RFC PATCH v2 15/21 physmem: extract ram_block_discard_range_fd() from ram_block_discard_range()

In reply to Xiaoyao Li

Extract the alignment check and sanity check out from ram_block_discard_range() into a seperate function ram_block_discard_range_fd(), which can be passed with an explicit fd as input parameter.

ram_block_discard_range_fd() can be used to discard private memory range from gmem fd with later patch. When doing private memory <-> shared memory conversion, it requires 4KB alignment instead of RamBlock.page_size.

---
 softmmu/physmem.c | 192 ++++++++++++++++++++++++----------------------
 1 file changed, 100 insertions(+), 92 deletions(-)
softmmu/physmem.c +100 -92
index 34d580ec0d39..6ee6bc794f44 100644
@@ -3425,117 +3425,125 @@ int qemu_ram_foreach_block(RAMBlockIterFunc func, void *opaque)
     return ret;
 }
 
+static int ram_block_discard_range_fd(RAMBlock *rb, uint64_t start,
+                                      size_t length, int fd)
+{
+    uint8_t *host_startaddr = rb->host + start;
+    bool need_madvise, need_fallocate;
+    int ret = -1;
+
+    errno = ENOTSUP; /* If we are missing MADVISE etc */
+
+    /* The logic here is messy;
+     *    madvise DONTNEED fails for hugepages
+     *    fallocate works on hugepages and shmem
+     *    shared anonymous memory requires madvise REMOVE
+     */
+    need_madvise = (rb->page_size == qemu_host_page_size) && (rb->fd == fd);
+    need_fallocate = fd != -1;
+
+    if (need_fallocate) {
+        /* For a file, this causes the area of the file to be zero'd
+         * if read, and for hugetlbfs also causes it to be unmapped
+         * so a userfault will trigger.
+         */
+#ifdef CONFIG_FALLOCATE_PUNCH_HOLE
+        /*
+         * We'll discard data from the actual file, even though we only
+         * have a MAP_PRIVATE mapping, possibly messing with other
+         * MAP_PRIVATE/MAP_SHARED mappings. There is no easy way to
+         * change that behavior whithout violating the promised
+         * semantics of ram_block_discard_range().
+         *
+         * Only warn, because it works as long as nobody else uses that
+         * file.
+         */
+        if (!qemu_ram_is_shared(rb)) {
+            warn_report_once("%s: Discarding RAM"
+                                " in private file mappings is possibly"
+                                " dangerous, because it will modify the"
+                                " underlying file and will affect other"
+                                " users of the file", __func__);
+        }
+
+        ret = fallocate(fd, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE,
+                        start, length);
+        if (ret) {
+            ret = -errno;
+            error_report("%s: Failed to fallocate %s:%" PRIx64 " +%zx (%d)",
+                            __func__, rb->idstr, start, length, ret);
+            return ret;
+        }
+#else
+        ret = -ENOSYS;
+        error_report("%s: fallocate not available/file "
+                     "%s:%" PRIx64 " +%zx (%d)",
+                     __func__, rb->idstr, start, length, ret);
+        return ret;
+#endif
+    }
+
+    if (need_madvise) {
+        /* For normal RAM this causes it to be unmapped,
+         * for shared memory it causes the local mapping to disappear
+         * and to fall back on the file contents (which we just
+         * fallocate'd away).
+         */
+#if defined(CONFIG_MADVISE)
+        if (qemu_ram_is_shared(rb) && fd < 0) {
+            ret = madvise(host_startaddr, length, QEMU_MADV_REMOVE);
+        } else {
+            ret = madvise(host_startaddr, length, QEMU_MADV_DONTNEED);
+        }
+        if (ret) {
+            ret = -errno;
+            error_report("%s: Failed to discard range %s:%" PRIx64 " +%zx (%d)",
+                         __func__, rb->idstr, start, length, ret);
+            return ret;
+        }
+#else
+        ret = -ENOSYS;
+        error_report("%s: MADVISE not available %s:%" PRIx64 " +%zx (%d)",
+                        __func__, rb->idstr, start, length, ret);
+        return ret;
+#endif
+    }
+
+    trace_ram_block_discard_range(rb->idstr, host_startaddr, length,
+                                  need_madvise, need_fallocate, ret);
+    return ret;
+}
+
 /*
  * Unmap pages of memory from start to start+length such that
  * they a) read as 0, b) Trigger whatever fault mechanism
  * the OS provides for postcopy.
+ *
  * The pages must be unmapped by the end of the function.
- * Returns: 0 on success, none-0 on failure
- *
+ * Returns: 0 on success, none-0 on failure.
  */
 int ram_block_discard_range(RAMBlock *rb, uint64_t start, size_t length)
 {
-    int ret = -1;
-
     uint8_t *host_startaddr = rb->host + start;
 
     if (!QEMU_PTR_IS_ALIGNED(host_startaddr, rb->page_size)) {
         error_report("%s: Unaligned start address: %p",
                      __func__, host_startaddr);
-        goto err;
+        return -1;
     }
 
-    if ((start + length) <= rb->max_length) {
-        bool need_madvise, need_fallocate;
-        if (!QEMU_IS_ALIGNED(length, rb->page_size)) {
-            error_report("%s: Unaligned length: %zx", __func__, length);
-            goto err;
-        }
-
-        errno = ENOTSUP; /* If we are missing MADVISE etc */
-
-        /* The logic here is messy;
-         *    madvise DONTNEED fails for hugepages
-         *    fallocate works on hugepages and shmem
-         *    shared anonymous memory requires madvise REMOVE
-         */
-        need_madvise = (rb->page_size == qemu_host_page_size);
-        need_fallocate = rb->fd != -1;
-        if (need_fallocate) {
-            /* For a file, this causes the area of the file to be zero'd
-             * if read, and for hugetlbfs also causes it to be unmapped
-             * so a userfault will trigger.
-             */
-#ifdef CONFIG_FALLOCATE_PUNCH_HOLE
-            /*
-             * We'll discard data from the actual file, even though we only
-             * have a MAP_PRIVATE mapping, possibly messing with other
-             * MAP_PRIVATE/MAP_SHARED mappings. There is no easy way to
-             * change that behavior whithout violating the promised
-             * semantics of ram_block_discard_range().
-             *
-             * Only warn, because it works as long as nobody else uses that
-             * file.
-             */
-            if (!qemu_ram_is_shared(rb)) {
-                warn_report_once("%s: Discarding RAM"
-                                 " in private file mappings is possibly"
-                                 " dangerous, because it will modify the"
-                                 " underlying file and will affect other"
-                                 " users of the file", __func__);
-            }
-
-            ret = fallocate(rb->fd, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE,
-                            start, length);
-            if (ret) {
-                ret = -errno;
-                error_report("%s: Failed to fallocate %s:%" PRIx64 " +%zx (%d)",
-                             __func__, rb->idstr, start, length, ret);
-                goto err;
-            }
-#else
-            ret = -ENOSYS;
-            error_report("%s: fallocate not available/file"
-                         "%s:%" PRIx64 " +%zx (%d)",
-                         __func__, rb->idstr, start, length, ret);
-            goto err;
-#endif
-        }
-        if (need_madvise) {
-            /* For normal RAM this causes it to be unmapped,
-             * for shared memory it causes the local mapping to disappear
-             * and to fall back on the file contents (which we just
-             * fallocate'd away).
-             */
-#if defined(CONFIG_MADVISE)
-            if (qemu_ram_is_shared(rb) && rb->fd < 0) {
-                ret = madvise(host_startaddr, length, QEMU_MADV_REMOVE);
-            } else {
-                ret = madvise(host_startaddr, length, QEMU_MADV_DONTNEED);
-            }
-            if (ret) {
-                ret = -errno;
-                error_report("%s: Failed to discard range "
-                             "%s:%" PRIx64 " +%zx (%d)",
-                             __func__, rb->idstr, start, length, ret);
-                goto err;
-            }
-#else
-            ret = -ENOSYS;
-            error_report("%s: MADVISE not available %s:%" PRIx64 " +%zx (%d)",
-                         __func__, rb->idstr, start, length, ret);
-            goto err;
-#endif
-        }
-        trace_ram_block_discard_range(rb->idstr, host_startaddr, length,
-                                      need_madvise, need_fallocate, ret);
-    } else {
+    if ((start + length) > rb->max_length) {
         error_report("%s: Overrun block '%s' (%" PRIu64 "/%zx/" RAM_ADDR_FMT")",
                      __func__, rb->idstr, start, length, rb->max_length);
+        return -1;
     }
 
-err:
-    return ret;
+    if (!QEMU_IS_ALIGNED(length, rb->page_size)) {
+        error_report("%s: Unaligned length: %zx", __func__, length);
+        return -1;
+    }
+
+    return ram_block_discard_range_fd(rb, start, length, rb->fd);
 }
 
 bool ramblock_is_pmem(RAMBlock *rb)
Signature · 1 line
2.34.1
author
#39 lore raw

RFC PATCH v2 16/21 physmem: Introduce ram_block_convert_range()

In reply to Xiaoyao Li

It's used for discarding oppsite memory after memory conversion to shared/private.

Note, private-shared page conversion is done at 4KB granularity. Don't check alignment with rb->page_size, instead qemu_host_page_size, which is 4K.

Originally-from: Isaku Yamahata <[email protected]>

---
 include/exec/cpu-common.h |  2 ++
 softmmu/physmem.c         | 36 ++++++++++++++++++++++++++++++++++++
 2 files changed, 38 insertions(+)
include/exec/cpu-common.h +2 -0
index 87dc9a752c9a..558684b9f246 100644
@@ -157,6 +157,8 @@ typedef int (RAMBlockIterFunc)(RAMBlock *rb, void *opaque);
 
 int qemu_ram_foreach_block(RAMBlockIterFunc func, void *opaque);
 int ram_block_discard_range(RAMBlock *rb, uint64_t start, size_t length);
+int ram_block_convert_range(RAMBlock *rb, uint64_t start, size_t length,
+                            bool shared_to_private);
 
 #endif
 
softmmu/physmem.c +36 -0
index 6ee6bc794f44..dab3247d461c 100644
@@ -3733,3 +3733,39 @@ bool ram_block_discard_is_required(void)
     return qatomic_read(&ram_block_discard_required_cnt) ||
            qatomic_read(&ram_block_coordinated_discard_required_cnt);
 }
+
+int ram_block_convert_range(RAMBlock *rb, uint64_t start, size_t length,
+                            bool shared_to_private)
+{
+    int fd;
+
+    if (!rb || rb->gmem_fd < 0) {
+        return -1;
+    }
+
+    if (!QEMU_PTR_IS_ALIGNED(start, qemu_host_page_size) ||
+        !QEMU_PTR_IS_ALIGNED(length, qemu_host_page_size)) {
+        return -1;
+    }
+
+    if (!length) {
+        return -1;
+    }
+
+    if (start + length > rb->max_length) {
+        return -1;
+    }
+
+    if (shared_to_private) {
+        void *host_startaddr = rb->host + start;
+
+        if (!QEMU_PTR_IS_ALIGNED(host_startaddr, qemu_host_page_size)) {
+            return -1;
+        }
+        fd = rb->fd;
+    } else {
+        fd = rb->gmem_fd;
+    }
+
+    return ram_block_discard_range_fd(rb, start, length, fd);
+}
Signature · 1 line
2.34.1
author
#40 lore raw

RFC PATCH v2 17/21 kvm: handle KVM_EXIT_MEMORY_FAULT

In reply to Xiaoyao Li

From: Chao Peng <[email protected]>

Currently only KVM_MEMORY_EXIT_FLAG_PRIVATE in flags is valid when KVM_EXIT_MEMORY_FAULT happens. It indicates userspace needs to do the memory conversion on the RAMBlock to turn the memory into desired attribute, i.e., private/shared.

Note, KVM_EXIT_MEMORY_FAULT makes sense only when the RAMBlock has gmem memory backend.

---
 accel/kvm/kvm-all.c | 54 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 54 insertions(+)
accel/kvm/kvm-all.c +54 -0
index 7e32ee83b258..c67aa66b0559 100644
@@ -3040,6 +3040,50 @@ static void kvm_eat_signals(CPUState *cpu)
     } while (sigismember(&chkset, SIG_IPI));
 }
 
+static int kvm_convert_memory(hwaddr start, hwaddr size, bool to_private)
+{
+    MemoryRegionSection section;
+    ram_addr_t offset;
+    RAMBlock *rb;
+    void *addr;
+    int ret = -1;
+
+    section = memory_region_find(get_system_memory(), start, size);
+    if (!section.mr) {
+        return ret;
+    }
+
+    if (memory_region_has_gmem_fd(section.mr)) {
+        if (to_private) {
+            ret = kvm_set_memory_attributes_private(start, size);
+        } else {
+            ret = kvm_set_memory_attributes_shared(start, size);
+        }
+
+        if (ret) {
+            memory_region_unref(section.mr);
+            return ret;
+        }
+
+        addr = memory_region_get_ram_ptr(section.mr) +
+               section.offset_within_region;
+        rb = qemu_ram_block_from_host(addr, false, &offset);
+        /*
+         * With KVM_SET_MEMORY_ATTRIBUTES by kvm_set_memory_attributes(),
+         * operation on underlying file descriptor is only for releasing
+         * unnecessary pages.
+         */
+        ram_block_convert_range(rb, offset, size, to_private);
+    } else {
+        warn_report("Convert non guest-memfd backed memory region "
+                    "(0x%"HWADDR_PRIx" ,+ 0x%"HWADDR_PRIx") to %s",
+                    start, size, to_private ? "private" : "shared");
+    }
+
+    memory_region_unref(section.mr);
+    return ret;
+}
+
 int kvm_cpu_exec(CPUState *cpu)
 {
     struct kvm_run *run = cpu->kvm_run;
@@ -3198,6 +3242,16 @@ int kvm_cpu_exec(CPUState *cpu)
                 break;
             }
             break;
+        case KVM_EXIT_MEMORY_FAULT:
+            if (run->memory.flags & ~KVM_MEMORY_EXIT_FLAG_PRIVATE) {
+                error_report("KVM_EXIT_MEMORY_FAULT: Unknown flag 0x%" PRIx64,
+                             (uint64_t)run->memory.flags);
+                ret = -1;
+                break;
+            }
+            ret = kvm_convert_memory(run->memory.gpa, run->memory.size,
+                                     run->memory.flags & KVM_MEMORY_EXIT_FLAG_PRIVATE);
+            break;
         default:
             DPRINTF("kvm_arch_handle_exit\n");
             ret = kvm_arch_handle_exit(cpu, run);
Signature · 1 line
2.34.1
author
#41 lore raw

RFC PATCH v2 18/21 trace/kvm: Add trace for page convertion between shared and private

In reply to Xiaoyao Li

From: Isaku Yamahata <[email protected]>

---
 accel/kvm/kvm-all.c    | 1 +
 accel/kvm/trace-events | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
accel/kvm/kvm-all.c +1 -0
index c67aa66b0559..229b7038a4c2 100644
@@ -3048,6 +3048,7 @@ static int kvm_convert_memory(hwaddr start, hwaddr size, bool to_private)
     void *addr;
     int ret = -1;
 
+    trace_kvm_convert_memory(start, size, to_private ? "shared_to_private" : "private_to_shared");
     section = memory_region_find(get_system_memory(), start, size);
     if (!section.mr) {
         return ret;
accel/kvm/trace-events +1 -1
index 80694683acea..4935c9c5cf0b 100644
@@ -25,4 +25,4 @@ kvm_dirty_ring_reaper(const char *s) "%s"
 kvm_dirty_ring_reap(uint64_t count, int64_t t) "reaped %"PRIu64" pages (took %"PRIi64" us)"
 kvm_dirty_ring_reaper_kick(const char *reason) "%s"
 kvm_dirty_ring_flush(int finished) "%d"
-
+kvm_convert_memory(uint64_t start, uint64_t size, const char *msg) "start 0x%" PRIx64 " size 0x%" PRIx64 " %s"
Signature · 1 line
2.34.1
author
#42 lore raw

RFC PATCH v2 19/21 pci-host/q35: Move PAM initialization above SMRAM initialization

In reply to Xiaoyao Li

From: Isaku Yamahata <[email protected]>

In mch_realize(), process PAM initialization before SMRAM initialization so that later patch can skill all the SMRAM related with a single check.

---
 hw/pci-host/q35.c | 19 ++++++++++---------
 1 file changed, 10 insertions(+), 9 deletions(-)
hw/pci-host/q35.c +10 -9
index 91c46df9ae31..ac1518a94ee4 100644
@@ -575,6 +575,16 @@ static void mch_realize(PCIDevice *d, Error **errp)
     /* setup pci memory mapping */
     pc_pci_as_mapping_init(mch->system_memory, mch->pci_address_space);
 
+    /* PAM */
+    init_pam(&mch->pam_regions[0], OBJECT(mch), mch->ram_memory,
+             mch->system_memory, mch->pci_address_space,
+             PAM_BIOS_BASE, PAM_BIOS_SIZE);
+    for (i = 0; i < ARRAY_SIZE(mch->pam_regions) - 1; ++i) {
+        init_pam(&mch->pam_regions[i + 1], OBJECT(mch), mch->ram_memory,
+                 mch->system_memory, mch->pci_address_space,
+                 PAM_EXPAN_BASE + i * PAM_EXPAN_SIZE, PAM_EXPAN_SIZE);
+    }
+
     /* if *disabled* show SMRAM to all CPUs */
     memory_region_init_alias(&mch->smram_region, OBJECT(mch), "smram-region",
                              mch->pci_address_space, MCH_HOST_BRIDGE_SMRAM_C_BASE,
@@ -641,15 +651,6 @@ static void mch_realize(PCIDevice *d, Error **errp)
 
     object_property_add_const_link(qdev_get_machine(), "smram",
                                    OBJECT(&mch->smram));
-
-    init_pam(&mch->pam_regions[0], OBJECT(mch), mch->ram_memory,
-             mch->system_memory, mch->pci_address_space,
-             PAM_BIOS_BASE, PAM_BIOS_SIZE);
-    for (i = 0; i < ARRAY_SIZE(mch->pam_regions) - 1; ++i) {
-        init_pam(&mch->pam_regions[i + 1], OBJECT(mch), mch->ram_memory,
-                 mch->system_memory, mch->pci_address_space,
-                 PAM_EXPAN_BASE + i * PAM_EXPAN_SIZE, PAM_EXPAN_SIZE);
-    }
 }
 
 uint64_t mch_mcfg_base(void)
Signature · 1 line
2.34.1
author
#43 lore raw

RFC PATCH v2 20/21 q35: Introduce smm_ranges property for q35-pci-host

In reply to Xiaoyao Li

From: Isaku Yamahata <[email protected]>

Add a q35 property to check whether or not SMM ranges, e.g. SMRAM, TSEG, etc... exist for the target platform. TDX doesn't support SMM and doesn't play nice with QEMU modifying related guest memory ranges.

---
 hw/i386/pc_q35.c          |  3 ++-
 hw/pci-host/q35.c         | 42 +++++++++++++++++++++++++++------------
 include/hw/i386/pc.h      |  1 +
 include/hw/pci-host/q35.h |  1 +
 4 files changed, 33 insertions(+), 14 deletions(-)
hw/i386/pc_q35.c +2 -1
index dc27a9e223a2..73eb3bc5e826 100644
@@ -232,8 +232,9 @@ static void pc_q35_init(MachineState *machine)
                             x86ms->above_4g_mem_size, NULL);
     object_property_set_bool(phb, PCI_HOST_BYPASS_IOMMU,
                              pcms->default_bus_bypass_iommu, NULL);
+    object_property_set_bool(phb, PCI_HOST_PROP_SMM_RANGES,
+                             x86_machine_is_smm_enabled(x86ms), NULL);
     sysbus_realize_and_unref(SYS_BUS_DEVICE(phb), &error_fatal);
-
     /* pci */
     host_bus = PCI_BUS(qdev_get_child_bus(DEVICE(phb), "pcie.0"));
     pcms->bus = host_bus;
hw/pci-host/q35.c +29 -13

82 lines not on this page · raw message

include/hw/i386/pc.h +1 -0
index c98d628a76f3..3c9ced22cea9 100644
@@ -156,6 +156,7 @@ void pc_guest_info_init(PCMachineState *pcms);
 #define PCI_HOST_PROP_PCI_HOLE64_SIZE  "pci-hole64-size"
 #define PCI_HOST_BELOW_4G_MEM_SIZE     "below-4g-mem-size"
 #define PCI_HOST_ABOVE_4G_MEM_SIZE     "above-4g-mem-size"
+#define PCI_HOST_PROP_SMM_RANGES       "smm-ranges"
 
 
 void pc_pci_as_mapping_init(MemoryRegion *system_memory,
include/hw/pci-host/q35.h +1 -0
index 1d98bbfe0dd2..930af3870e8a 100644
@@ -50,6 +50,7 @@ struct MCHPCIState {
     MemoryRegion tseg_blackhole, tseg_window;
     MemoryRegion smbase_blackhole, smbase_window;
     bool has_smram_at_smbase;
+    bool has_smm_ranges;
     Range pci_hole;
     uint64_t below_4g_mem_size;
     uint64_t above_4g_mem_size;
Signature · 1 line
2.34.1
author
#44 lore raw

RFC PATCH v2 21/21 i386: Disable SMM mode for X86_SW_PROTECTED_VM

In reply to Xiaoyao Li
---
 target/i386/kvm/sw-protected-vm.c | 8 ++++++++
 1 file changed, 8 insertions(+)
target/i386/kvm/sw-protected-vm.c +8 -0
index f47ac383e1dd..65347067aa03 100644
@@ -34,10 +34,18 @@ static MemoryListener kvm_x86_sw_protected_vm_memory_listener = {
 int sw_protected_vm_kvm_init(MachineState *ms, Error **errp)
 {
     SwProtectedVm *spvm = SW_PROTECTED_VM(OBJECT(ms->cgs));
+    X86MachineState *x86ms = X86_MACHINE(ms);
 
     memory_listener_register(&kvm_x86_sw_protected_vm_memory_listener,
                              &address_space_memory);
 
+    if (x86ms->smm == ON_OFF_AUTO_AUTO) {
+        x86ms->smm = ON_OFF_AUTO_OFF;
+    } else if (x86ms->smm == ON_OFF_AUTO_ON) {
+        error_setg(errp, "X86_SW_PROTECTED_VM doesn't support SMM");
+        return -EINVAL;
+    }
+
     spvm->parent_obj.ready = true;
     return 0;
 }
Signature · 1 line
2.34.1
#45 lore raw
In reply to Xiaoyao Li
Quoted 18 lines · Xiaoyao Li

It's the v2 RFC of enabling KVM gmem[1] as the backend for private memory.

For confidential-computing, KVM provides gmem/guest_mem interfaces for userspace, like QEMU, to allocate user-unaccesible private memory. This series aims to add gmem support in QEMU's RAMBlock so that each RAM can have both hva-based shared memory and gmem_fd based private memory. QEMU does the shared-private conversion on KVM_MEMORY_EXIT and discards the memory.

It chooses the design that adds "private" property to hostmeory backend. If "private" property is set, QEMU will allocate/create KVM gmem when initialize the RAMbloch of the memory backend.

This sereis also introduces the first user of kvm gmem, KVM_X86_SW_PROTECTED_VM. A KVM_X86_SW_PROTECTED_VM with private KVM gmem can be created with

Quoted 8 lines
   $qemu -object sw-protected-vm,id=sp-vm0 \
	-object memory-backend-ram,id=mem0,size=1G,private=on \
	-machine q35,kernel_irqchip=split,confidential-guest-support=sp-vm0,memory-backend=mem0 \
	...

Unfortunately this patch series fails the boot of OVMF at very early stage due to triple fault, because KVM doesn't support emulating string IO to private memory.

Is support being added? Or have we figured out what it would take to make it work?

How does this interact with other features (memory ballooning, virtiofs, vfio/mdev/...)?

Quoted 10 lines

This version still leave some opens to be discussed:
1. whether we need "private" propery to be user-settable?

It seems unnecessary because vm-type is determined. If the VM is confidential-guest, then the RAM of the guest must be able to be mapped as private, i.e., have kvm gmem backend. So QEMU can determine the value of "private" property automatiacally based on vm type.

Quoted 4 lines

This also aligns with the board internal MemoryRegion that needs to have kvm gmem backend, e.g., TDX requires OVMF to act as private memory so bios memory region needs to have kvm gmem fd associated. QEMU no doubt will do it internally automatically.

Would it make sense to have some regions without "pivate" semantics? Like NVDIMMs?

Quoted 7 lines

2. hugepage support.

KVM gmem can be allocated from hugetlbfs. How does QEMU determine when to allocate KVM gmem with KVM_GUEST_MEMFD_ALLOW_HUGEPAGE. The easiest solution is create KVM gmem with KVM_GUEST_MEMFD_ALLOW_HUGEPAGE only when memory backend is HostMemoryBackendFile of hugetlbfs.

Good question.

Probably "if the memory backend uses huge pages, also use huge pages for the private gmem" makes sense.

... but it becomes a mess with preallocation ... which is what people should actually be using with hugetlb. Andeventual double memory-consumption ... but maybe that's all been taken care of already?

Probably it's best to leave hugetlb support as future work and start with something minimal.

Quoted 3 lines

3. What is KVM_X86_SW_PROTECTED_VM going to look like? and do we need it?

Why implement it when you have to ask others for a motivation? ;)

Personally, I'm not sure if it is really useful, especially in this state.

Quoted 4 lines

This series implements KVM_X86_SW_PROTECTED_VM because it's introduced with gmem together on KVM side and it's supposed to be the first user who requires KVM gmem. However the implementation is incomplete and there lacks the definition of how KVM_X86_SW_PROTECTED_VM works.

Then it should not be included in this series such that you can make progress with the gmem implementation for TDX guests instead?

Signature · 3 lines
Cheers,

David / dhildenb
#46 lore raw
Quoted 24 lines · David Hildenbrand
Quoted 23 lines · Xiaoyao Li

It's the v2 RFC of enabling KVM gmem[1] as the backend for private memory.

For confidential-computing, KVM provides gmem/guest_mem interfaces for userspace, like QEMU, to allocate user-unaccesible private memory. This series aims to add gmem support in QEMU's RAMBlock so that each RAM can have both hva-based shared memory and gmem_fd based private memory. QEMU does the shared-private conversion on KVM_MEMORY_EXIT and discards the memory.

It chooses the design that adds "private" property to hostmeory backend. If "private" property is set, QEMU will allocate/create KVM gmem when initialize the RAMbloch of the memory backend.

This sereis also introduces the first user of kvm gmem, KVM_X86_SW_PROTECTED_VM. A KVM_X86_SW_PROTECTED_VM with private KVM gmem can be created with

   $qemu -object sw-protected-vm,id=sp-vm0 \
	-object memory-backend-ram,id=mem0,size=1G,private=on \
	-machine q35,kernel_irqchip=split,confidential-guest-support=sp-vm0,memory-backend=mem0 \
	...
Quoted 6 lines
Quoted 3 lines

Unfortunately this patch series fails the boot of OVMF at very early stage due to triple fault, because KVM doesn't support emulating string IO to private memory.

Is support being added? Or have we figured out what it would take to make it work?

Hrm, this isn't something I've thought deeply about. The issue is that anything that reaches any form of copy_{from,to}_user() will go kablooie because KVM will always try to read/write the shared mappings. The best case scenario is that the shared mapping is invalid and the uaccess faults. The worst case scenario is that KVM read/writes the wrong memory and sends the guest into the weeds. Eww.

And we (well, at least I) definitely want to support this so that gmem can be used for "regular" VMs, i.e. for VMs where userspace is in the TCB, but for which userspace doesn't have access to guest memory by default.

It shouldn't be too hard to support. It's easy enough to wire up the hook (thankfully that aren't _that_ many sites), and gmem only supports struct page at the moment so we go straight to kmap. E.g. something like this

virt/kvm/kvm_main.c +8 -6
index 54480655bcce..b500b0ce5ce3 100644
@@ -3291,12 +3291,15 @@ static int next_segment(unsigned long len, int offset)
                return len;
 }
 
-static int __kvm_read_guest_page(struct kvm_memory_slot *slot, gfn_t gfn,
-                                void *data, int offset, int len)
+static int __kvm_read_guest_page(struct kvm *kvm, struct kvm_memory_slot *slot,
+                                gfn_t gfn, void *data, int offset, int len)
 {
        int r;
        unsigned long addr;
 
+       if (kvm_mem_is_private(kvm, gfn))
+               return kvm_gmem_read(slot, gfn, data, offset, len);
+
        addr = gfn_to_hva_memslot_prot(slot, gfn, NULL);
        if (kvm_is_error_hva(addr))
                return -EFAULT;
@@ -3309,9 +3312,8 @@ static int __kvm_read_guest_page(struct kvm_memory_slot *slot, gfn_t gfn,
 int kvm_read_guest_page(struct kvm *kvm, gfn_t gfn, void *data, int offset,
                        int len)
 {
-       struct kvm_memory_slot *slot = gfn_to_memslot(kvm, gfn);
-
-       return __kvm_read_guest_page(slot, gfn, data, offset, len);
+       return __kvm_read_guest_page(kvm, gfn_to_memslot(kvm, gfn), gfn, data,
+                                    offset, len);
 }
 EXPORT_SYMBOL_GPL(kvm_read_guest_page);
 
@@ -3320,7 +3322,7 @@ int kvm_vcpu_read_guest_page(struct kvm_vcpu *vcpu, gfn_t gfn, void *data,
 {
        struct kvm_memory_slot *slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn);
 
-       return __kvm_read_guest_page(slot, gfn, data, offset, len);
+       return __kvm_read_guest_page(vcpu->kvm, slot, gfn, data, offset, len);
 }
 EXPORT_SYMBOL_GPL(kvm_vcpu_read_guest_page);
 
Quoted 3 lines
Quoted 3 lines

2. hugepage support.

KVM gmem can be allocated from hugetlbfs. How does QEMU determine

Not yet it can't. gmem only supports THP, hugetlbfs is a future thing, if it's ever supported. I wouldn't be at all surprised if we end up going down a slightly different route and don't use hugetlbfs directly.

Quoted 17 lines
Quoted 3 lines

when to allocate KVM gmem with KVM_GUEST_MEMFD_ALLOW_HUGEPAGE. The easiest solution is create KVM gmem with KVM_GUEST_MEMFD_ALLOW_HUGEPAGE only when memory backend is HostMemoryBackendFile of hugetlbfs.

Good question.

Probably "if the memory backend uses huge pages, also use huge pages for the private gmem" makes sense.

... but it becomes a mess with preallocation ... which is what people should actually be using with hugetlb. Andeventual double memory-consumption ... but maybe that's all been taken care of already?

Probably it's best to leave hugetlb support as future work and start with something minimal.

Quoted 1 line
Quoted 6 lines
Quoted 2 lines

3. What is KVM_X86_SW_PROTECTED_VM going to look like? and do we need it?

Why implement it when you have to ask others for a motivation? ;)

Personally, I'm not sure if it is really useful, especially in this state.

Yeah, as of today, KVM_X86_SW_PROTECTED_VM is mainly a development vehicle, e.g. so that testing gmem doesn't require TDX/SNP hardware, debugging gmem guests isn't brutally painful, etc.

Longer term, I have aspirations of being able to back most VMs with gmem, but that's going to require quite a bit more work, e.g. gmem needs to be mappable (when hardware allows it) so that gmem doesn't all but require double mapping, KVM obviously needs to be able to read/write gmem, etc.

The value proposition is that having a guest-first memory type will allow KVM to optimize and harden gmem in ways that wouldn't be feasible for a more generic memory implementation. E.g. memory isn't mapped into host userspace by default (makes it harder to accidentally corrupt the guest), the guest can have *larger* mappings than host userspace, guest memory can be served from a dedicated pool (similar-ish to hugetlb), the pool can be omitted from the direct map, etc.

Quoted 7 lines
Quoted 4 lines

This series implements KVM_X86_SW_PROTECTED_VM because it's introduced with gmem together on KVM side and it's supposed to be the first user who requires KVM gmem. However the implementation is incomplete and there lacks the definition of how KVM_X86_SW_PROTECTED_VM works.

Then it should not be included in this series such that you can make progress with the gmem implementation for TDX guests instead?

#47 lore raw
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines

2. hugepage support.

KVM gmem can be allocated from hugetlbfs. How does QEMU determine

Not yet it can't. gmem only supports THP, hugetlbfs is a future thing, if it's ever supported. I wouldn't be at all surprised if we end up going down a slightly different route and don't use hugetlbfs directly.

Agreed. Certainly future work.

Quoted 34 lines
Quoted 18 lines
Quoted 4 lines

when to allocate KVM gmem with KVM_GUEST_MEMFD_ALLOW_HUGEPAGE. The easiest solution is create KVM gmem with KVM_GUEST_MEMFD_ALLOW_HUGEPAGE only when memory backend is HostMemoryBackendFile of hugetlbfs.

Good question.

Probably "if the memory backend uses huge pages, also use huge pages for the private gmem" makes sense.

... but it becomes a mess with preallocation ... which is what people should actually be using with hugetlb. Andeventual double memory-consumption ... but maybe that's all been taken care of already?

Probably it's best to leave hugetlb support as future work and start with something minimal.

Quoted 1 line
Quoted 6 lines
Quoted 2 lines

3. What is KVM_X86_SW_PROTECTED_VM going to look like? and do we need it?

Why implement it when you have to ask others for a motivation? ;)

Personally, I'm not sure if it is really useful, especially in this state.

Yeah, as of today, KVM_X86_SW_PROTECTED_VM is mainly a development vehicle, e.g. so that testing gmem doesn't require TDX/SNP hardware, debugging gmem guests isn't brutally painful, etc.

Longer term, I have aspirations of being able to back most VMs with gmem, but that's going to require quite a bit more work, e.g. gmem needs to be mappable (when hardware allows it) so that gmem doesn't all but require double mapping, KVM obviously needs to be able to read/write gmem, etc.

Quoted 7 lines

The value proposition is that having a guest-first memory type will allow KVM to optimize and harden gmem in ways that wouldn't be feasible for a more generic memory implementation. E.g. memory isn't mapped into host userspace by default (makes it harder to accidentally corrupt the guest), the guest can have *larger* mappings than host userspace, guest memory can be served from a dedicated pool (similar-ish to hugetlb), the pool can be omitted from the direct map, etc.

Thanks for that information. Personally, I don't believe "to back most VMs with gmem", but that's a different discussion.

As a development vehicle to get TDX up and running it might be very valuable indeed. But it doesn't necessarily have to be merged in QEMU for that case -- especially in a semi-finished form.

Signature · 3 lines
Cheers,

David / dhildenb
author
#48 lore raw
Quoted 23 lines · David Hildenbrand
Quoted 16 lines
Quoted 6 lines
Quoted 2 lines

3. What is KVM_X86_SW_PROTECTED_VM going to look like? and do we need it?

Why implement it when you have to ask others for a motivation? 😉

Personally, I'm not sure if it is really useful, especially in this state.

Yeah, as of today, KVM_X86_SW_PROTECTED_VM is mainly a development vehicle, e.g. so that testing gmem doesn't require TDX/SNP hardware, debugging gmem guests isn't brutally painful, etc.

Longer term, I have aspirations of being able to back most VMs with gmem, but that's going to require quite a bit more work, e.g. gmem needs to be mappable (when hardware allows it) so that gmem doesn't all but require double mapping, KVM obviously needs to be able to read/write gmem, etc.

Quoted 7 lines

The value proposition is that having a guest-first memory type will allow KVM to optimize and harden gmem in ways that wouldn't be feasible for a more generic memory implementation.  E.g. memory isn't mapped into host userspace by default (makes it harder to accidentally corrupt the guest), the guest can have *larger* mappings than host userspace, guest memory can be served from a dedicated pool (similar-ish to hugetlb), the pool can be omitted from the direct map, etc.

Quoted 3 lines

Thanks for that information. Personally, I don't believe "to back most VMs with gmem", but that's a different discussion.

As a development vehicle to get TDX up and running it might be very valuable indeed. But it doesn't necessarily have to be merged in QEMU for that case -- especially in a semi-finished form.

It's true and I agree with it. I'll drop the KVM_X86_SW_PROTECTED_VM part in next version.

How would you like this series to proceed in next version? only the patches of gmem support without a user? or together with next QEMU TDX series?

#49 lore raw
Quoted 44 lines · Xiaoyao Li
Quoted 36 lines · David Hildenbrand
Quoted 23 lines
Quoted 8 lines
Quoted 3 lines

3. What is KVM_X86_SW_PROTECTED_VM going to look like? and do we need it?

Why implement it when you have to ask others for a motivation? 😉

Personally, I'm not sure if it is really useful, especially in this state.

Yeah, as of today, KVM_X86_SW_PROTECTED_VM is mainly a development vehicle,
e.g. so that testing gmem doesn't require TDX/SNP hardware, debugging gmem guests
isn't brutally painful, etc.

Longer term, I have aspirations of being able to back most VMs with gmem, but
that's going to require quite a bit more work, e.g. gmem needs to be mappable
(when hardware allows it) so that gmem doesn't all but require double mapping,
KVM obviously needs to be able to read/write gmem, etc.

Quoted 13 lines

The value proposition is that having a guest-first memory type will allow KVM to
optimize and harden gmem in ways that wouldn't be feasible for a more generic
memory implementation.  E.g. memory isn't mapped into host userspace by default
(makes it harder to accidentally corrupt the guest), the guest can have *larger*
mappings than host userspace, guest memory can be served from a dedicated pool
(similar-ish to hugetlb), the pool can be omitted from the direct map, etc.

Quoted 6 lines

Thanks for that information. Personally, I don't believe "to back most VMs with gmem", but that's a different discussion.

As a development vehicle to get TDX up and running it might be very valuable indeed. But it doesn't necessarily have to be merged in QEMU for that case -- especially in a semi-finished form.

Quoted 6 lines

It's true and I agree with it. I'll drop the KVM_X86_SW_PROTECTED_VM part in next version.

How would you like this series to proceed in next version? only the patches of gmem support without a user? or together with next QEMU TDX series?

Makes sense to me. GMEM series can be a prereq for QEMU TDX.

Signature · 3 lines
Cheers,

David / dhildenb
author
#50 lore raw
In reply to David Hildenbrand
Quoted 24 lines · David Hildenbrand
Quoted 23 lines · Xiaoyao Li

It's the v2 RFC of enabling KVM gmem[1] as the backend for private memory.

For confidential-computing, KVM provides gmem/guest_mem interfaces for userspace, like QEMU, to allocate user-unaccesible private memory. This series aims to add gmem support in QEMU's RAMBlock so that each RAM can have both hva-based shared memory and gmem_fd based private memory. QEMU does the shared-private conversion on KVM_MEMORY_EXIT and discards the memory.

It chooses the design that adds "private" property to hostmeory backend. If "private" property is set, QEMU will allocate/create KVM gmem when initialize the RAMbloch of the memory backend.

This sereis also introduces the first user of kvm gmem, KVM_X86_SW_PROTECTED_VM. A KVM_X86_SW_PROTECTED_VM with private KVM gmem can be created with

   $qemu -object sw-protected-vm,id=sp-vm0 \
    -object memory-backend-ram,id=mem0,size=1G,private=on \
    -machine q35,kernel_irqchip=split,confidential-guest-support=sp-vm0,memory-backend=mem0 \
    ...

Quoted 5 lines
Quoted 3 lines

Unfortunately this patch series fails the boot of OVMF at very early stage due to triple fault, because KVM doesn't support emulating string IO to private memory.

Is support being added? Or have we figured out what it would take to make it work?

Hi David,

I only reply the questions that werrn't covered by Sean's reply.

Quoted 1 line

How does this interact with other features (memory ballooning, virtiofs, vfio/mdev/...)?

I need time to learn them before I can answer it.

Quoted 10 lines
Quoted 10 lines

This version still leave some opens to be discussed:
1. whether we need "private" propery to be user-settable?

    It seems unnecessary because vm-type is determined. If the VM is confidential-guest, then the RAM of the guest must be able to be mapped as private, i.e., have kvm gmem backend. So QEMU can determine the value of "private" property automatiacally based on vm type.

Quoted 6 lines
Quoted 4 lines

    This also aligns with the board internal MemoryRegion that needs to have kvm gmem backend, e.g., TDX requires OVMF to act as private memory so bios memory region needs to have kvm gmem fd associated. QEMU no doubt will do it internally automatically.

Would it make sense to have some regions without "pivate" semantics? Like NVDIMMs?

Of course it can have regions without "private" semantics.

Whether a region needs "private" backend depends on the definition of VM type. E.g., for TDX,
- all the RAM needs to able to mapped as private. So it needs private
gmem.
- TDVF(OVMF) code must be mapped as private. So it needs private gmem.
- MMIO region needs to be shared for TDX 1.0, and it doesn't need
private gmem;

Quoted 8 lines
Quoted 7 lines

2. hugepage support.

    KVM gmem can be allocated from hugetlbfs. How does QEMU determine when to allocate KVM gmem with KVM_GUEST_MEMFD_ALLOW_HUGEPAGE. The easiest solution is create KVM gmem with KVM_GUEST_MEMFD_ALLOW_HUGEPAGE only when memory backend is HostMemoryBackendFile of hugetlbfs.

Quoted 8 lines

Good question.

Probably "if the memory backend uses huge pages, also use huge pages for the private gmem" makes sense.

... but it becomes a mess with preallocation ... which is what people should actually be using with hugetlb. Andeventual double memory-consumption ... but maybe that's all been taken care of already?

Probably it's best to leave hugetlb support as future work and start with something minimal.

As Sean replied, I had some misunderstanding of KVM_GUEST_MEMFD_ALLOW_HUGEPAGE. If it's for THP, I think we can allow it for every gmem.

As for hugetlb, we can leave it as future work.

#51 lore raw
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines

This version still leave some opens to be discussed:
1. whether we need "private" propery to be user-settable?

    It seems unnecessary because vm-type is determined. If the VM is confidential-guest, then the RAM of the guest must be able to be mapped as private, i.e., have kvm gmem backend. So QEMU can determine the value of "private" property automatiacally based on vm type.

    This also aligns with the board internal MemoryRegion that needs to have kvm gmem backend, e.g., TDX requires OVMF to act as private memory so bios memory region needs to have kvm gmem fd associated. QEMU no doubt will do it internally automatically.

Quoted 4 lines
Quoted 2 lines

Would it make sense to have some regions without "pivate" semantics? Like NVDIMMs?

Of course it can have regions without "private" semantics.

I meant "RAM memory regions on such a special VM". Does it make sense to have !private there? I assume "for now, not".

Quoted 18 lines
Quoted 18 lines
Quoted 8 lines

2. hugepage support.

    KVM gmem can be allocated from hugetlbfs. How does QEMU determine when to allocate KVM gmem with KVM_GUEST_MEMFD_ALLOW_HUGEPAGE. The easiest solution is create KVM gmem with
KVM_GUEST_MEMFD_ALLOW_HUGEPAGE
    only when memory backend is HostMemoryBackendFile of hugetlbfs.

Good question.

Probably "if the memory backend uses huge pages, also use huge pages for the private gmem" makes sense.

... but it becomes a mess with preallocation ... which is what people should actually be using with hugetlb. Andeventual double memory-consumption ... but maybe that's all been taken care of already?

Quoted 7 lines
Quoted 3 lines

Probably it's best to leave hugetlb support as future work and start with something minimal.

As Sean replied, I had some misunderstanding of KVM_GUEST_MEMFD_ALLOW_HUGEPAGE. If it's for THP, I think we can allow it for every gmem.

Right, just like we do a MADV_HUGEPAGE rather blindly on all memory.

Signature · 3 lines
Cheers,

David / dhildenb