Enable shared device assignment
Subsystems (2)
cover Enable shared device assignment
Commit 852f0048f3 ("RAMBlock: make guest_memfd require uncoordinated discard") effectively disables device assignment when using guest_memfd. This poses a significant challenge as guest_memfd is essential for confidential guests, thereby blocking device assignment to these VMs. The initial rationale for disabling device assignment was due to stale IOMMU mappings (see Problem section) and the assumption that TEE I/O (SEV-TIO, TDX Connect, COVE-IO, etc.) would solve the device-assignment problem for confidential guests [1]. However, this assumption has proven to be incorrect. TEE I/O relies on the ability to operate devices against "shared" or untrusted memory, which is crucial for device initialization and error recovery scenarios. As a result, the current implementation does not adequately support device assignment for confidential guests, necessitating a reevaluation of the approach to ensure compatibility and functionality.
This series enables shared device assignment by notifying VFIO of page conversions using an existing framework named RamDiscardListener. Additionally, there is an ongoing patch set [2] that aims to add 1G page support for guest_memfd. This patch set introduces in-place page conversion, where private and shared memory share the same physical pages as the backend. This development may impact our solution.
We presented our solution in the guest_memfd meeting to discuss its compatibility with the new changes and potential future directions (see [3] for more details). The conclusion was that, although our solution may not be the most elegant (see the Limitation section), it is sufficient for now and can be easily adapted to future changes.
We are re-posting the patch series with some cleanup and have removed the RFC label for the main enabling patches (1-6). The newly-added patch 7 is still marked as RFC as it tries to resolve some extension concerns related to RamDiscardManager for future usage.
The overview of the patches:
- Patch 1: Export a helper to get intersection of a MemoryRegionSection with a given range.
- Patch 2-6: Introduce a new object to manage the guest-memfd with RamDiscardManager, and notify the shared/private state change during conversion.
- Patch 7: Try to resolve a semantics concern related to RamDiscardManager i.e. RamDiscardManager is used to manage memory plug/unplug state instead of shared/private state. It would affect future users of RamDiscardManger in confidential VMs. Attach it behind as a RFC patch[4].
Changes since last version:
- Add a patch to export some generic helper functions from virtio-mem code.
- Change the bitmap in guest_memfd_manager from default shared to default private. This keeps alignment with virtio-mem that 1-setting in bitmap represents the populated state and may help to export more generic code if necessary.
- Add the helpers to initialize/uninitialize the guest_memfd_manager instance to make it more clear.
- Add a patch to distinguish between the shared/private state change and the memory plug/unplug state change in RamDiscardManager.
- RFC: https://lore.kernel.org/qemu-devel/[email protected]/
---
Background
==========
Confidential VMs have two classes of memory: shared and private memory. Shared memory is accessible from the host/VMM while private memory is not. Confidential VMs can decide which memory is shared/private and convert memory between shared/private at runtime.
"guest_memfd" is a new kind of fd whose primary goal is to serve guest private memory. The key differences between guest_memfd and normal memfd are that guest_memfd is spawned by a KVM ioctl, bound to its owner VM and cannot be mapped, read or written by userspace.
In QEMU's implementation, shared memory is allocated with normal methods (e.g. mmap or fallocate) while private memory is allocated from guest_memfd. When a VM performs memory conversions, QEMU frees pages via madvise() or via PUNCH_HOLE on memfd or guest_memfd from one side and allocates new pages from the other side.
Problem
=======
Device assignment in QEMU is implemented via VFIO system. In the normal VM, VM memory is pinned at the beginning of time by VFIO. In the confidential VM, the VM can convert memory and when that happens nothing currently tells VFIO that its mappings are stale. This means that page conversion leaks memory and leaves stale IOMMU mappings. For example, sequence like the following can result in stale IOMMU mappings:
1. allocate shared page
2. convert page shared->private
3. discard shared page
4. convert page private->shared
5. allocate shared page
6. issue DMA operations against that shared page
After step 3, VFIO is still pinning the page. However, DMA operations in step 6 will hit the old mapping that was allocated in step 1, which causes the device to access the invalid data.
Solution
========
The key to enable shared device assignment is to update the IOMMU mappings on page conversion.
Given the constraints and assumptions here is a solution that satisfied the use cases. RamDiscardManager, an existing interface currently utilized by virtio-mem, offers a means to modify IOMMU mappings in accordance with VM page assignment. Page conversion is similar to hot-removing a page in one mode and adding it back in the other.
This series implements a RamDiscardManager for confidential VMs and utilizes its infrastructure to notify VFIO of page conversions.
Another possible attempt [5] was to not discard shared pages in step 3 above. This was an incomplete band-aid because guests would consume twice the memory since shared pages wouldn't be freed even after they were converted to private.
w/ in-place page conversion
===========================
To support 1G page support for guest_memfd, the current direction is to allow mmap() of guest_memfd to userspace so that both private and shared memory can use the same physical pages as the backend. This in-place page conversion design eliminates the need to discard pages during shared/private conversions. However, device assignment will still be blocked because the in-place page conversion will reject the conversion when the page is pinned by VFIO.
To address this, the key difference lies in the sequence of VFIO map/unmap operations and the page conversion. This series can be adjusted to achieve unmap-before-conversion-to-private and map-after-conversion-to-shared, ensuring compatibility with guest_memfd.
Additionally, with in-place page conversion, the previously mentioned solution to disable the discard of shared pages is not feasible because shared and private memory share the same backend, and no discard operation is performed. Retaining the old mappings in the IOMMU would result in unsafe DMA access to protected memory.
Limitation
==========
One limitation (also discussed in the guest_memfd meeting) is that VFIO expects the DMA mapping for a specific IOVA to be mapped and unmapped with the same granularity. The guest may perform partial conversions, such as converting a small region within a larger region. To prevent such invalid cases, all operations are performed with 4K granularity. The possible solutions we can think of are either to enable VFIO to support partial unmap or to implement an enlightened guest to avoid partial conversion. The former requires complex changes in VFIO, while the latter requires the page conversion to be a guest-enlightened behavior. It is still uncertain which option is a preferred one.
Testing
=======
This patch series is tested with the KVM/QEMU branch:
KVM: https://github.com/intel/tdx/tree/tdx_kvm_dev-2024-11-20
QEMU: https://github.com/intel-staging/qemu-tdx/tree/tdx-upstream-snapshot-2024-12-13
To facilitate shared device assignment with the NIC, employ the legacy type1 VFIO with the QEMU command:
qemu-system-x86_64 [...]
-device vfio-pci,host=XX:XX.X
The parameter of dma_entry_limit needs to be adjusted. For example, a 16GB guest needs to adjust the parameter like vfio_iommu_type1.dma_entry_limit=4194304.
If use the iommufd-backed VFIO with the qemu command:
qemu-system-x86_64 [...]
-object iommufd,id=iommufd0 \
-device vfio-pci,host=XX:XX.X,iommufd=iommufd0No additional adjustment required.
Following the bootup of the TD guest, the guest's IP address becomes visible, and iperf is able to successfully send and receive data.
Related link
============
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://docs.google.com/document/d/1M6766BzdY1Lhk7LiR5IqVR8B8mG3cr-cxTxOrAosPOk/edit?tab=t.0#heading=h.jr4csfgw1uql
[4] https://lore.kernel.org/qemu-devel/[email protected]/
[5] https://lore.kernel.org/all/[email protected]/
Chenyi Qiang (7):
memory: Export a helper to get intersection of a MemoryRegionSection
with a given range
guest_memfd: Introduce an object to manage the guest-memfd with
RamDiscardManager
guest_memfd: Introduce a callback to notify the shared/private state
change
KVM: Notify the state change event during shared/private conversion
memory: Register the RamDiscardManager instance upon guest_memfd
creation
RAMBlock: make guest_memfd require coordinate discard
memory: Add a new argument to indicate the request attribute in
RamDismcardManager helpers
accel/kvm/kvm-all.c | 4 + hw/vfio/common.c | 22 +- hw/virtio/virtio-mem.c | 55 ++-- include/exec/memory.h | 36 ++- include/sysemu/guest-memfd-manager.h | 91 ++++++ migration/ram.c | 14 +- system/guest-memfd-manager.c | 456 +++++++++++++++++++++++++++ system/memory.c | 30 +- system/memory_mapping.c | 4 +- system/meson.build | 1 + system/physmem.c | 9 +- 11 files changed, 659 insertions(+), 63 deletions(-) create mode 100644 include/sysemu/guest-memfd-manager.h create mode 100644 system/guest-memfd-manager.c
Signature · 1 line
2.43.5
PATCH 1/7 memory: Export a helper to get intersection of a MemoryRegionSection with a given range
Rename the helper to memory_region_section_intersect_range() to make it more generic.
--- hw/virtio/virtio-mem.c | 32 +++++--------------------------- include/exec/memory.h | 13 +++++++++++++ system/memory.c | 17 +++++++++++++++++ 3 files changed, 35 insertions(+), 27 deletions(-)
hw/virtio/virtio-mem.c +5 -27
@@ -242,28 +242,6 @@ static int virtio_mem_for_each_plugged_range(VirtIOMEM *vmem, void *arg,return ret; } -/* -* Adjust the memory section to cover the intersection with the given range. -* -* Returns false if the intersection is empty, otherwise returns true. -*/ -static bool virtio_mem_intersect_memory_section (MemoryRegionSection *s, -uint64_t offset,uint64_t size) -{ -uint64_t start =MAX (s->offset_within_region, offset); -uint64_t end =MIN (s->offset_within_region +int128_get64 (s->size), - offset + size); - -if (end <= start) { -return false; - } - - s->offset_within_address_space += start - s->offset_within_region; - s->offset_within_region = start; - s->size =int128_make64 (end - start); -return true; -} -typedef int (*virtio_mem_section_cb )(MemoryRegionSection *s,void *arg); static int virtio_mem_for_each_plugged_section(const VirtIOMEM *vmem,
@@ -285,7 +263,7 @@ static int virtio_mem_for_each_plugged_section(const VirtIOMEM *vmem, first_bit + 1) - 1; size = (last_bit - first_bit +1 ) * vmem->block_size; -if (!virtio_mem_intersect_memory_section (&tmp, offset, size)) { +if (!memory_region_section_intersect_range (&tmp, offset, size)) {break ; } ret =cb (&tmp, arg);
@@ -317,7 +295,7 @@ static int virtio_mem_for_each_unplugged_section(const VirtIOMEM *vmem, first_bit + 1) - 1; size = (last_bit - first_bit +1 ) * vmem->block_size; -if (!virtio_mem_intersect_memory_section (&tmp, offset, size)) { +if (!memory_region_section_intersect_range (&tmp, offset, size)) {break ; } ret =cb (&tmp, arg);
@@ -353,7 +331,7 @@ static void virtio_mem_notify_unplug(VirtIOMEM *vmem, uint64_t offset, QLIST_FOREACH(rdl, &vmem->rdl_list, next) {MemoryRegionSection tmp = *rdl->section; -if (!virtio_mem_intersect_memory_section (&tmp, offset, size)) { +if (!memory_region_section_intersect_range (&tmp, offset, size)) {continue ; } rdl->notify_discard (rdl, &tmp);
@@ -369,7 +347,7 @@ static int virtio_mem_notify_plug(VirtIOMEM *vmem, uint64_t offset, QLIST_FOREACH(rdl, &vmem->rdl_list, next) {MemoryRegionSection tmp = *rdl->section; -if (!virtio_mem_intersect_memory_section (&tmp, offset, size)) { +if (!memory_region_section_intersect_range (&tmp, offset, size)) {continue ; } ret = rdl->notify_populate (rdl, &tmp);
@@ -386,7 +364,7 @@ static int virtio_mem_notify_plug(VirtIOMEM *vmem, uint64_t offset,if (rdl2 == rdl) {break ; } -if (!virtio_mem_intersect_memory_section (&tmp, offset, size)) { +if (!memory_region_section_intersect_range (&tmp, offset, size)) {continue ; } rdl2->notify_discard (rdl2, &tmp);
include/exec/memory.h +13 -0
@@ -1196,6 +1196,19 @@ MemoryRegionSection *memory_region_section_new_copy(MemoryRegionSection *s);*/ void memory_region_section_free_copy (MemoryRegionSection *s); +/** +* memory_region_section_intersect_range: Adjust the memory section to cover +* the intersection with the given range. +* +* @s: the #MemoryRegionSection to be adjusted +* @offset: the offset of the given range in the memory region +* @size: the size of the given range +* +* Returns false if the intersection is empty, otherwise returns true. +*/ +bool memory_region_section_intersect_range (MemoryRegionSection *s, +uint64_t offset,uint64_t size); +/** * memory_region_init: Initialize a memory region *
system/memory.c +17 -0
@@ -2898,6 +2898,23 @@ void memory_region_section_free_copy(MemoryRegionSection *s)g_free (s); } +bool memory_region_section_intersect_range (MemoryRegionSection *s, +uint64_t offset,uint64_t size) +{ +uint64_t start =MAX (s->offset_within_region, offset); +uint64_t end =MIN (s->offset_within_region +int128_get64 (s->size), + offset + size); + +if (end <= start) { +return false; + } + + s->offset_within_address_space += start - s->offset_within_region; + s->offset_within_region = start; + s->size =int128_make64 (end - start); +return true; +} +bool memory_region_present (MemoryRegion *container,hwaddr addr) {MemoryRegion *mr;
Signature · 1 line
2.43.5
Re: [PATCH 1/7] memory: Export a helper to get intersection of a MemoryRegionSection with a given range
Quoted 101 lines · Chenyi Qiang
Rename the helper to
memory_region_section_intersect_range()to make it more generic.Signed-off-by Chenyi Qiang <[email protected]>--- hw/virtio/virtio-mem.c | 32 +++++--------------------------- include/exec/memory.h | 13 +++++++++++++ system/memory.c | 17 +++++++++++++++++ 3 files changed, 35 insertions(+), 27 deletions(-)hw/virtio/virtio-mem.c +5
-27@@ -242,28 +242,6 @@ static int virtio_mem_for_each_plugged_range(VirtIOMEM *vmem, void *arg,return ret; } -/* -* Adjust the memory section to cover the intersection with the given range. -* -* Returns false if the intersection is empty, otherwise returns true. -*/ -static bool virtio_mem_intersect_memory_section (MemoryRegionSection *s, -uint64_t offset,uint64_t size) -{ -uint64_t start =MAX (s->offset_within_region, offset); -uint64_t end =MIN (s->offset_within_region +int128_get64 (s->size), - offset + size); - -if (end <= start) { -return false; - } - - s->offset_within_address_space += start - s->offset_within_region; - s->offset_within_region = start; - s->size =int128_make64 (end - start); -return true; -} -typedef int (*virtio_mem_section_cb )(MemoryRegionSection *s,void *arg); static int virtio_mem_for_each_plugged_section(const VirtIOMEM *vmem,@@ -285,7 +263,7 @@ static int virtio_mem_for_each_plugged_section(const VirtIOMEM *vmem, first_bit + 1) - 1; size = (last_bit - first_bit +1 ) * vmem->block_size; -if (!virtio_mem_intersect_memory_section (&tmp, offset, size)) { +if (!memory_region_section_intersect_range (&tmp, offset, size)) {break ; } ret =cb (&tmp, arg);@@ -317,7 +295,7 @@ static int virtio_mem_for_each_unplugged_section(const VirtIOMEM *vmem, first_bit + 1) - 1; size = (last_bit - first_bit +1 ) * vmem->block_size; -if (!virtio_mem_intersect_memory_section (&tmp, offset, size)) { +if (!memory_region_section_intersect_range (&tmp, offset, size)) {break ; } ret =cb (&tmp, arg);@@ -353,7 +331,7 @@ static void virtio_mem_notify_unplug(VirtIOMEM *vmem, uint64_t offset, QLIST_FOREACH(rdl, &vmem->rdl_list, next) {MemoryRegionSection tmp = *rdl->section; -if (!virtio_mem_intersect_memory_section (&tmp, offset, size)) { +if (!memory_region_section_intersect_range (&tmp, offset, size)) {continue ; } rdl->notify_discard (rdl, &tmp);@@ -369,7 +347,7 @@ static int virtio_mem_notify_plug(VirtIOMEM *vmem, uint64_t offset, QLIST_FOREACH(rdl, &vmem->rdl_list, next) {MemoryRegionSection tmp = *rdl->section; -if (!virtio_mem_intersect_memory_section (&tmp, offset, size)) { +if (!memory_region_section_intersect_range (&tmp, offset, size)) {continue ; } ret = rdl->notify_populate (rdl, &tmp);@@ -386,7 +364,7 @@ static int virtio_mem_notify_plug(VirtIOMEM *vmem, uint64_t offset,if (rdl2 == rdl) {break ; } -if (!virtio_mem_intersect_memory_section (&tmp, offset, size)) { +if (!memory_region_section_intersect_range (&tmp, offset, size)) {continue ; } rdl2->notify_discard (rdl2, &tmp);include/exec/memory.h +5
-0@@ -1196,6 +1196,19 @@ MemoryRegionSection *memory_region_section_new_copy(MemoryRegionSection *s);*/ void memory_region_section_free_copy (MemoryRegionSection *s); +/** +* memory_region_section_intersect_range: Adjust the memory section to cover +* the intersection with the given range. +* +* @s: the #MemoryRegionSection to be adjusted
include/exec/memory.h MemoryRegionSection *memory_region_section_new_copy(MemoryRegionSection *s); · line 1213
Quoted 10 lines
include/exec/memory.h +8
-0+* @offset: the offset of the given range in the memory region +* @size: the size of the given range +* +* Returns false if the intersection is empty, otherwise returns true. +*/ +bool memory_region_section_intersect_range (MemoryRegionSection *s, +uint64_t offset,uint64_t size); +/** * memory_region_init: Initialize a memory region
Maybe it could simply be an inline function. In any case, LGTM:
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH 1/7] memory: Export a helper to get intersection of a MemoryRegionSection with a given range
In reply to Chenyi QiangQuoted 117 lines · Chenyi Qiang
Rename the helper to
memory_region_section_intersect_range()to make it more generic.Signed-off-by Chenyi Qiang <[email protected]>--- hw/virtio/virtio-mem.c | 32 +++++--------------------------- include/exec/memory.h | 13 +++++++++++++ system/memory.c | 17 +++++++++++++++++ 3 files changed, 35 insertions(+), 27 deletions(-)hw/virtio/virtio-mem.c +5
-27@@ -242,28 +242,6 @@ static int virtio_mem_for_each_plugged_range(VirtIOMEM *vmem, void *arg,return ret; } -/* -* Adjust the memory section to cover the intersection with the given range. -* -* Returns false if the intersection is empty, otherwise returns true. -*/ -static bool virtio_mem_intersect_memory_section (MemoryRegionSection *s, -uint64_t offset,uint64_t size) -{ -uint64_t start =MAX (s->offset_within_region, offset); -uint64_t end =MIN (s->offset_within_region +int128_get64 (s->size), - offset + size); - -if (end <= start) { -return false; - } - - s->offset_within_address_space += start - s->offset_within_region; - s->offset_within_region = start; - s->size =int128_make64 (end - start); -return true; -} -typedef int (*virtio_mem_section_cb )(MemoryRegionSection *s,void *arg); static int virtio_mem_for_each_plugged_section(const VirtIOMEM *vmem,@@ -285,7 +263,7 @@ static int virtio_mem_for_each_plugged_section(const VirtIOMEM *vmem, first_bit + 1) - 1; size = (last_bit - first_bit +1 ) * vmem->block_size; -if (!virtio_mem_intersect_memory_section (&tmp, offset, size)) { +if (!memory_region_section_intersect_range (&tmp, offset, size)) {break ; } ret =cb (&tmp, arg);@@ -317,7 +295,7 @@ static int virtio_mem_for_each_unplugged_section(const VirtIOMEM *vmem, first_bit + 1) - 1; size = (last_bit - first_bit +1 ) * vmem->block_size; -if (!virtio_mem_intersect_memory_section (&tmp, offset, size)) { +if (!memory_region_section_intersect_range (&tmp, offset, size)) {break ; } ret =cb (&tmp, arg);@@ -353,7 +331,7 @@ static void virtio_mem_notify_unplug(VirtIOMEM *vmem, uint64_t offset, QLIST_FOREACH(rdl, &vmem->rdl_list, next) {MemoryRegionSection tmp = *rdl->section; -if (!virtio_mem_intersect_memory_section (&tmp, offset, size)) { +if (!memory_region_section_intersect_range (&tmp, offset, size)) {continue ; } rdl->notify_discard (rdl, &tmp);@@ -369,7 +347,7 @@ static int virtio_mem_notify_plug(VirtIOMEM *vmem, uint64_t offset, QLIST_FOREACH(rdl, &vmem->rdl_list, next) {MemoryRegionSection tmp = *rdl->section; -if (!virtio_mem_intersect_memory_section (&tmp, offset, size)) { +if (!memory_region_section_intersect_range (&tmp, offset, size)) {continue ; } ret = rdl->notify_populate (rdl, &tmp);@@ -386,7 +364,7 @@ static int virtio_mem_notify_plug(VirtIOMEM *vmem, uint64_t offset,if (rdl2 == rdl) {break ; } -if (!virtio_mem_intersect_memory_section (&tmp, offset, size)) { +if (!memory_region_section_intersect_range (&tmp, offset, size)) {continue ; } rdl2->notify_discard (rdl2, &tmp);include/exec/memory.h +13
-0@@ -1196,6 +1196,19 @@ MemoryRegionSection *memory_region_section_new_copy(MemoryRegionSection *s);*/ void memory_region_section_free_copy (MemoryRegionSection *s); +/** +* memory_region_section_intersect_range: Adjust the memory section to cover +* the intersection with the given range. +* +* @s: the #MemoryRegionSection to be adjusted +* @offset: the offset of the given range in the memory region +* @size: the size of the given range +* +* Returns false if the intersection is empty, otherwise returns true. +*/ +bool memory_region_section_intersect_range (MemoryRegionSection *s, +uint64_t offset,uint64_t size); +/** * memory_region_init: Initialize a memory region * system/memory.c +0
-0
system/memory.c void memory_region_section_free_copy(MemoryRegionSection *s) · line 2906
Quoted 9 lines
system/memory.c +6
-0@@ -2898,6 +2898,23 @@ void memory_region_section_free_copy(MemoryRegionSection *s)g_free (s); } +bool memory_region_section_intersect_range (MemoryRegionSection *s, +uint64_t offset,uint64_t size) +{ +uint64_t start =MAX (s->offset_within_region, offset); +uint64_t end =MIN (s->offset_within_region +int128_get64 (s->size), + offset + size);
imho @end needs to be Int128 and s/MIN/int128_min/, etc to be totally correct (although it is going to look horrendous). May be it was alright when it was just virtio but now it is a wider API. I understand this is cut-n-paste and unlikely scenario of offset+size crossing 1<<64 but still. Thanks,
Quoted 14 lines
system/memory.c +11
-0+ +if (end <= start) { +return false; + } + + s->offset_within_address_space += start - s->offset_within_region; + s->offset_within_region = start; + s->size =int128_make64 (end - start); +return true; +} +bool memory_region_present (MemoryRegion *container,hwaddr addr) {MemoryRegion *mr;
Signature · 1 line
Alexey
Re: [PATCH 1/7] memory: Export a helper to get intersection of a MemoryRegionSection with a given range
Quoted 141 lines · Alexey Kardashevskiy
Quoted 140 lines · Chenyi Qiang
Rename the helper to
memory_region_section_intersect_range()to make it more generic.Signed-off-by Chenyi Qiang <[email protected]>---
hw/virtio/virtio-mem.c| 32 +++++--------------------------- include/exec/memory.h | 13 +++++++++++++ system/memory.c | 17 +++++++++++++++++3 files changed, 35 insertions(+), 27 deletions(-)hw/virtio/virtio-mem.c +0
-0@@ -242,28 +242,6 @@ static int virtio_mem_for_each_plugged_range(VirtIOMEM *vmem, void *arg, return ret; }-/*
- * Adjust the memory section to cover the intersection with the given range.
- *
- * Returns false if the intersection is empty, otherwise returns true.
- */
-static bool virtio_mem_intersect_memory_section(MemoryRegionSection *s, -uint64_toffset,uint64_tsize)hw/virtio/virtio-mem.c +0
-15-{ - uint64_t start = MAX(s->offset_within_region, offset); - uint64_t end = MIN(s->offset_within_region + int128_get64(s->size), - offset + size); - - if (end <= start) { - return false; - } - - s->offset_within_address_space += start - s->offset_within_region; - s->offset_within_region = start; - s->size = int128_make64(end - start); - return true; -} -typedef int (*virtio_mem_section_cb)(MemoryRegionSection *s, void *arg);
static int virtio_mem_for_each_plugged_section(const VirtIOMEM *vmem,hw/virtio/virtio-mem.c +1
-0@@ -285,7 +263,7 @@ static int virtio_mem_for_each_plugged_section(const VirtIOMEM *vmem, first_bit + 1) - 1; size = (last_bit - first_bit + 1) * vmem->block_size; - if (!virtio_mem_intersect_memory_section(&tmp, offset, size)) { + if (!memory_region_section_intersect_range(&tmp, offset, size)) {break;
}
ret =cb(&tmp, arg);hw/virtio/virtio-mem.c +1
-0@@ -317,7 +295,7 @@ static int virtio_mem_for_each_unplugged_section(const VirtIOMEM *vmem, first_bit + 1) - 1; size = (last_bit - first_bit + 1) * vmem->block_size; - if (!virtio_mem_intersect_memory_section(&tmp, offset, size)) { + if (!memory_region_section_intersect_range(&tmp, offset, size)) {break;
}
ret =cb(&tmp, arg);hw/virtio/virtio-mem.c +1
-0@@ -353,7 +331,7 @@ static void virtio_mem_notify_unplug(VirtIOMEM *vmem, uint64_t offset, QLIST_FOREACH(rdl, &vmem->rdl_list, next) { MemoryRegionSection tmp = *rdl->section; - if (!virtio_mem_intersect_memory_section(&tmp, offset, size)) { + if (!memory_region_section_intersect_range(&tmp, offset, size)) {continue;
}
rdl->notify_discard(rdl, &tmp);hw/virtio/virtio-mem.c +1
-0@@ -369,7 +347,7 @@ static int virtio_mem_notify_plug(VirtIOMEM *vmem,uint64_t offset, QLIST_FOREACH(rdl, &vmem->rdl_list, next) { MemoryRegionSection tmp = *rdl->section; - if (!virtio_mem_intersect_memory_section(&tmp, offset, size)) { + if (!memory_region_section_intersect_range(&tmp, offset, size)) {continue;
}
ret =rdl->notify_populate(rdl, &tmp);hw/virtio/virtio-mem.c +1
-1@@ -386,7 +364,7 @@ static int virtio_mem_notify_plug(VirtIOMEM *vmem, uint64_t offset, if (rdl2 == rdl) { break; } - if (!virtio_mem_intersect_memory_section(&tmp, offset, size)) { + if (!memory_region_section_intersect_range(&tmp, offset, size)) {continue;
}
rdl2->notify_discard(rdl2, &tmp);include/exec/memory.h +0
-0@@ -1196,6 +1196,19 @@ MemoryRegionSection*memory_region_section_new_copy(MemoryRegionSection *s); */ void memory_region_section_free_copy(MemoryRegionSection *s);+/**
+ *memory_region_section_intersect_range: Adjust the memory section to cover
+ * the intersection with the given range.
+ *
+ * @s: the #MemoryRegionSection to be adjusted
+ * @offset: the offset of the given range in the memory region
+ * @size: the size of the given range
+ *
+ * Returns false if the intersection is empty, otherwise returns true.
+ */
+bool memory_region_section_intersect_range(MemoryRegionSection *s, +uint64_toffset,uint64_tsize);
+
/**
*memory_region_init: Initialize a memory region
*system/memory.c +3
-0@@ -2898,6 +2898,23 @@ void memory_region_section_free_copy(MemoryRegionSection *s) g_free(s); } +bool memory_region_section_intersect_range(MemoryRegionSection *s, + uint64_t offset, uint64_t size) +{ + uint64_t start = MAX(s->offset_within_region, offset);
Quoted 8 lines
system/memory.c void memory_region_section_free_copy(MemoryRegionSection *s) · line 2906
Quoted 2 lines
system/memory.c +2
-0+ uint64_t end = MIN(s->offset_within_region + int128_get64(s->size), + offset + size);imho @end needs to be Int128 and s/MIN/int128_min/, etc to be totally correct (although it is going to look horrendous). May be it was alright when it was just virtio but now it is a wider API. I understand this is cut-n-paste and unlikely scenario of offset+size crossing 1<<64 but still. Thanks,
Make sense. I'll change it in next version.
Quoted 17 lines
Quoted 16 lines
system/memory.c +11
-0+ + if (end <= start) { + return false; + } + + s->offset_within_address_space += start - s->offset_within_region; + s->offset_within_region = start; + s->size = int128_make64(end - start); + return true; +} + bool memory_region_present(MemoryRegion *container, hwaddr addr){
MemoryRegion *mr;
Re: [PATCH 1/7] memory: Export a helper to get intersection of a MemoryRegionSection with a given range
In reply to Alexey KardashevskiyQuoted 121 lines · Alexey Kardashevskiy
Quoted 120 lines · Chenyi Qiang
Rename the helper to
memory_region_section_intersect_range()to make it more generic.Signed-off-by Chenyi Qiang <[email protected]>--- hw/virtio/virtio-mem.c | 32 +++++--------------------------- include/exec/memory.h | 13 +++++++++++++ system/memory.c | 17 +++++++++++++++++ 3 files changed, 35 insertions(+), 27 deletions(-)hw/virtio/virtio-mem.c +5
-27@@ -242,28 +242,6 @@ static int virtio_mem_for_each_plugged_range(VirtIOMEM *vmem, void *arg,return ret; } -/* -* Adjust the memory section to cover the intersection with the given range. -* -* Returns false if the intersection is empty, otherwise returns true. -*/ -static bool virtio_mem_intersect_memory_section (MemoryRegionSection *s, -uint64_t offset,uint64_t size) -{ -uint64_t start =MAX (s->offset_within_region, offset); -uint64_t end =MIN (s->offset_within_region +int128_get64 (s->size), - offset + size); - -if (end <= start) { -return false; - } - - s->offset_within_address_space += start - s->offset_within_region; - s->offset_within_region = start; - s->size =int128_make64 (end - start); -return true; -} -typedef int (*virtio_mem_section_cb )(MemoryRegionSection *s,void *arg); static int virtio_mem_for_each_plugged_section(const VirtIOMEM *vmem,@@ -285,7 +263,7 @@ static int virtio_mem_for_each_plugged_section(const VirtIOMEM *vmem, first_bit + 1) - 1; size = (last_bit - first_bit +1 ) * vmem->block_size; -if (!virtio_mem_intersect_memory_section (&tmp, offset, size)) { +if (!memory_region_section_intersect_range (&tmp, offset, size)) {break ; } ret =cb (&tmp, arg);@@ -317,7 +295,7 @@ static int virtio_mem_for_each_unplugged_section(const VirtIOMEM *vmem, first_bit + 1) - 1; size = (last_bit - first_bit +1 ) * vmem->block_size; -if (!virtio_mem_intersect_memory_section (&tmp, offset, size)) { +if (!memory_region_section_intersect_range (&tmp, offset, size)) {break ; } ret =cb (&tmp, arg);@@ -353,7 +331,7 @@ static void virtio_mem_notify_unplug(VirtIOMEM *vmem, uint64_t offset, QLIST_FOREACH(rdl, &vmem->rdl_list, next) {MemoryRegionSection tmp = *rdl->section; -if (!virtio_mem_intersect_memory_section (&tmp, offset, size)) { +if (!memory_region_section_intersect_range (&tmp, offset, size)) {continue ; } rdl->notify_discard (rdl, &tmp);@@ -369,7 +347,7 @@ static int virtio_mem_notify_plug(VirtIOMEM *vmem, uint64_t offset, QLIST_FOREACH(rdl, &vmem->rdl_list, next) {MemoryRegionSection tmp = *rdl->section; -if (!virtio_mem_intersect_memory_section (&tmp, offset, size)) { +if (!memory_region_section_intersect_range (&tmp, offset, size)) {continue ; } ret = rdl->notify_populate (rdl, &tmp);@@ -386,7 +364,7 @@ static int virtio_mem_notify_plug(VirtIOMEM *vmem, uint64_t offset,if (rdl2 == rdl) {break ; } -if (!virtio_mem_intersect_memory_section (&tmp, offset, size)) { +if (!memory_region_section_intersect_range (&tmp, offset, size)) {continue ; } rdl2->notify_discard (rdl2, &tmp);include/exec/memory.h +13
-0@@ -1196,6 +1196,19 @@ MemoryRegionSection *memory_region_section_new_copy(MemoryRegionSection *s);*/ void memory_region_section_free_copy (MemoryRegionSection *s); +/** +* memory_region_section_intersect_range: Adjust the memory section to cover +* the intersection with the given range. +* +* @s: the #MemoryRegionSection to be adjusted +* @offset: the offset of the given range in the memory region +* @size: the size of the given range +* +* Returns false if the intersection is empty, otherwise returns true. +*/ +bool memory_region_section_intersect_range (MemoryRegionSection *s, +uint64_t offset,uint64_t size); +/** * memory_region_init: Initialize a memory region * system/memory.c +0
-0@@ -2898,6 +2898,23 @@ void memory_region_section_free_copy(MemoryRegionSection *s)g_free (s); }
Quoted 10 lines
system/memory.c void memory_region_section_free_copy(MemoryRegionSection *s) · line 2906
Quoted 6 lines
system/memory.c +6
-0+bool memory_region_section_intersect_range (MemoryRegionSection *s, +uint64_t offset,uint64_t size) +{ +uint64_t start =MAX (s->offset_within_region, offset); +uint64_t end =MIN (s->offset_within_region +int128_get64 (s->size), + offset + size);imho @end needs to be Int128 and s/MIN/int128_min/, etc to be totally correct (although it is going to look horrendous). May be it was alright when it was just virtio but now it is a wider API.
Yes, virtio-mem operates on RAM regions only, so it does not exceed 64bit.
I understand this is
Quoted 2 lines
cut-n-paste and unlikely scenario of offset+size crossing 1<<64 but still.
It's been bugging me for a while that size is 128bit butoffset_within_address_space and friends are 64bit. Likely we'll never get offset+size > 64bit here as of now, but it is indeed cleaner to just handle it using the correct type.
Signature · 3 lines
Cheers, David / dhildenb
PATCH 2/7 guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
In reply to Chenyi QiangAs the commit 852f0048f3 ("RAMBlock: make guest_memfd require uncoordinated discard") highlighted, some subsystems like VFIO might disable ram block discard. However, guest_memfd relies on the discard operation to perform page conversion between private and shared memory. This can lead to stale IOMMU mapping issue when assigning a hardware device to a confidential VM via shared memory (unprotected memory pages). Blocking shared page discard can solve this problem, but it could cause guests to consume twice the memory with VFIO, which is not acceptable in some cases. An alternative solution is to convey other systems like VFIO to refresh its outdated IOMMU mappings.
RamDiscardManager is an existing concept (used by virtio-mem) to adjust VFIO mappings in relation to VM page assignment. Effectively page conversion is similar to hot-removing a page in one mode and adding it back in the other, so the similar work that needs to happen in response to virtio-mem changes needs to happen for page conversion events. Introduce the RamDiscardManager to guest_memfd to achieve it.
However, guest_memfd is not an object so it cannot directly implement the RamDiscardManager interface.
One solution is to implement the interface in HostMemoryBackend. Any guest_memfd-backed host memory backend can register itself in the target MemoryRegion. However, this solution doesn't cover the scenario where a guest_memfd MemoryRegion doesn't belong to the HostMemoryBackend, e.g. the virtual BIOS MemoryRegion.
Thus, choose the second option, i.e. define an object type named guest_memfd_manager with RamDiscardManager interface. Upon creation of guest_memfd, a new guest_memfd_manager object can be instantiated and registered to the managed guest_memfd MemoryRegion to handle the page conversion events.
In the context of guest_memfd, the discarded state signifies that the page is private, while the populated state indicated that the page is shared. The state of the memory is tracked at the granularity of the host page size (i.e. block_size), as the minimum conversion size can be one page per request.
In addition, VFIO expects the DMA mapping for a specific iova to be mapped and unmapped with the same granularity. However, the confidential VMs may do partial conversion, e.g. conversion happens on a small region within a large region. To prevent such invalid cases and before any potential optimization comes out, all operations are performed with 4K granularity.
--- include/sysemu/guest-memfd-manager.h | 46 +++++ system/guest-memfd-manager.c | 250 +++++++++++++++++++++++++++ system/meson.build | 1 + 3 files changed, 297 insertions(+) create mode 100644 include/sysemu/guest-memfd-manager.h create mode 100644 system/guest-memfd-manager.c
include/sysemu/guest-memfd-manager.h new file +46 -0
@@ -0,0 +1,46 @@ +/* +* QEMU guest memfd manager +* +* Copyright Intel +* +* Author: +* Chenyi Qiang <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2 or later. +* See the COPYING file in the top-level directory +* +*/ + +#ifndef SYSEMU_GUEST_MEMFD_MANAGER_H +#define SYSEMU_GUEST_MEMFD_MANAGER_H + +#include "sysemu/hostmem.h" + +#define TYPE_GUEST_MEMFD_MANAGER "guest-memfd-manager" + +OBJECT_DECLARE_TYPE(GuestMemfdManager, GuestMemfdManagerClass, GUEST_MEMFD_MANAGER) + +struct GuestMemfdManager { +Object parent; + +/* Managed memory region. */ +MemoryRegion *mr; + +/* +* 1-setting of the bit represents the memory is populated (shared). +*/ +int32_t bitmap_size; +unsigned long *bitmap; + +/* block size and alignment */ +uint64_t block_size; + +/* listeners to notify on populate/discard activity. */ + QLIST_HEAD(, RamDiscardListener) rdl_list; +}; + +struct GuestMemfdManagerClass { +ObjectClass parent_class; +}; + +#endif
system/guest-memfd-manager.c new file +250 -0
250 lines not on this page · raw message
system/meson.build +1 -0
@@ -15,6 +15,7 @@ system_ss.add(files( 'dirtylimit.c', 'dma-helpers.c', 'globals.c', + 'guest-memfd-manager.c', 'memory_mapping.c', 'qdev-monitor.c', 'qtest.c',
Signature · 1 line
2.43.5
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 127 lines · Chenyi Qiang
As the commit
852f0048f3("RAMBlock: makeguest_memfdrequire uncoordinated discard") highlighted, some subsystems like VFIO might disable ram block discard. However,guest_memfdrelies on the discard operation to perform page conversion between private and shared memory. This can lead to stale IOMMU mapping issue when assigning a hardware device to a confidential VM via shared memory (unprotected memory pages). Blocking shared page discard can solve this problem, but it could cause guests to consume twice the memory with VFIO, which is not acceptable in some cases. An alternative solution is to convey other systems like VFIO to refresh its outdated IOMMU mappings.RamDiscardManager is an existing concept (used by virtio-mem) to adjust VFIO mappings in relation to VM page assignment. Effectively page conversion is similar to hot-removing a page in one mode and adding it back in the other, so the similar work that needs to happen in response to virtio-mem changes needs to happen for page conversion events. Introduce the RamDiscardManager to
guest_memfdto achieve it.However,
guest_memfdis not an object so it cannot directly implement the RamDiscardManager interface.One solution is to implement the interface in HostMemoryBackend. Any guest_memfd-backed host memory backend can register itself in the target MemoryRegion. However, this solution doesn't cover the scenario where a
guest_memfdMemoryRegion doesn't belong to the HostMemoryBackend, e.g. the virtual BIOS MemoryRegion.Thus, choose the second option, i.e. define an object type named
guest_memfd_managerwith RamDiscardManager interface. Upon creation ofguest_memfd, a newguest_memfd_managerobject can be instantiated and registered to the managedguest_memfdMemoryRegion to handle the page conversion events.In the context of
guest_memfd, the discarded state signifies that the page is private, while the populated state indicated that the page is shared. The state of the memory is tracked at the granularity of the host page size (i.e.block_size), as the minimum conversion size can be one page per request.In addition, VFIO expects the DMA mapping for a specific iova to be mapped and unmapped with the same granularity. However, the confidential VMs may do partial conversion, e.g. conversion happens on a small region within a large region. To prevent such invalid cases and before any potential optimization comes out, all operations are performed with 4K granularity.
Signed-off-by Chenyi Qiang <[email protected]>--- include/sysemu/guest-memfd-manager.h | 46 +++++ system/guest-memfd-manager.c | 250 +++++++++++++++++++++++++++ system/meson.build | 1 + 3 files changed, 297 insertions(+) create mode 100644 include/sysemu/guest-memfd-manager.h create mode 100644 system/guest-memfd-manager.cinclude/sysemu/guest-memfd-manager.h new file +46
-0@@ -0,0 +1,46 @@ +/* +* QEMU guest memfd manager +* +* Copyright Intel +* +* Author: +* Chenyi Qiang <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2 or later. +* See the COPYING file in the top-level directory +* +*/ + +#ifndef SYSEMU_GUEST_MEMFD_MANAGER_H +#define SYSEMU_GUEST_MEMFD_MANAGER_H + +#include "sysemu/hostmem.h" + +#define TYPE_GUEST_MEMFD_MANAGER "guest-memfd-manager" + +OBJECT_DECLARE_TYPE(GuestMemfdManager, GuestMemfdManagerClass, GUEST_MEMFD_MANAGER) + +struct GuestMemfdManager { +Object parent; + +/* Managed memory region. */ +MemoryRegion *mr; + +/* +* 1-setting of the bit represents the memory is populated (shared). +*/ +int32_t bitmap_size; +unsigned long *bitmap; + +/* block size and alignment */ +uint64_t block_size; + +/* listeners to notify on populate/discard activity. */ + QLIST_HEAD(, RamDiscardListener) rdl_list; +}; + +struct GuestMemfdManagerClass { +ObjectClass parent_class; +}; + +#endif system/guest-memfd-manager.c new file +14
-0@@ -0,0 +1,250 @@ +/* +* QEMU guest memfd manager +* +* Copyright Intel +* +* Author: +* Chenyi Qiang <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2 or later. +* See the COPYING file in the top-level directory +* +*/ + +#include "qemu/osdep.h"
system/guest-memfd-manager.c line 24
Quoted 10 lines
system/guest-memfd-manager.c new file +10
-0+#include "qemu/error-report.h" +#include "sysemu/guest-memfd-manager.h" + +OBJECT_DEFINE_SIMPLE_TYPE_WITH_INTERFACES (GuestMemfdManager, + guest_memfd_manager, +GUEST_MEMFD_MANAGER , +OBJECT , + { TYPE_RAM_DISCARD_MANAGER }, + { }) +
Fixup: Use OBJECT_DEFINE_TYPE_WITH_INTERFACES() instead of OBJECT_DEFINE_SIMPLE_TYPE_WITH_INTERFACES() as we define a class struct.
system/guest-memfd-manager.c +6 -6
@@ -15,12 +15,12 @@#include "qemu/error-report.h" #include "sysemu/guest-memfd-manager.h" -OBJECT_DEFINE_SIMPLE_TYPE_WITH_INTERFACES (GuestMemfdManager, - guest_memfd_manager, -GUEST_MEMFD_MANAGER , -OBJECT , - { TYPE_RAM_DISCARD_MANAGER }, - { }) +OBJECT_DEFINE_TYPE_WITH_INTERFACES (GuestMemfdManager, + guest_memfd_manager, +GUEST_MEMFD_MANAGER , +OBJECT , + { TYPE_RAM_DISCARD_MANAGER }, + { }) static bool guest_memfd_rdm_is_populated(const RamDiscardManager *rdm,
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
In reply to Chenyi QiangQuoted 18 lines · Chenyi Qiang
As the commit
852f0048f3("RAMBlock: makeguest_memfdrequire uncoordinated discard") highlighted, some subsystems like VFIO might disable ram block discard. However,guest_memfdrelies on the discard operation to perform page conversion between private and shared memory. This can lead to stale IOMMU mapping issue when assigning a hardware device to a confidential VM via shared memory (unprotected memory pages). Blocking shared page discard can solve this problem, but it could cause guests to consume twice the memory with VFIO, which is not acceptable in some cases. An alternative solution is to convey other systems like VFIO to refresh its outdated IOMMU mappings.RamDiscardManager is an existing concept (used by virtio-mem) to adjust VFIO mappings in relation to VM page assignment. Effectively page conversion is similar to hot-removing a page in one mode and adding it back in the other, so the similar work that needs to happen in response to virtio-mem changes needs to happen for page conversion events. Introduce the RamDiscardManager to
guest_memfdto achieve it.
Quoted 4 lines
However,
guest_memfdis not an object so it cannot directly implement the RamDiscardManager interface.One solution is to implement the interface in HostMemoryBackend. Any
This sounds about right.
Quoted 4 lines
guest_memfd-backed host memory backend can register itself in the target MemoryRegion. However, this solution doesn't cover the scenario where a
guest_memfdMemoryRegion doesn't belong to the HostMemoryBackend, e.g. the virtual BIOS MemoryRegion.
What is this virtual BIOS MemoryRegion exactly? What does it look like in "info mtree -f"? Do we really want this memory to be DMAable?
Quoted 50 lines
Thus, choose the second option, i.e. define an object type named
guest_memfd_managerwith RamDiscardManager interface. Upon creation ofguest_memfd, a newguest_memfd_managerobject can be instantiated and registered to the managedguest_memfdMemoryRegion to handle the page conversion events.In the context of
guest_memfd, the discarded state signifies that the page is private, while the populated state indicated that the page is shared. The state of the memory is tracked at the granularity of the host page size (i.e.block_size), as the minimum conversion size can be one page per request.In addition, VFIO expects the DMA mapping for a specific iova to be mapped and unmapped with the same granularity. However, the confidential VMs may do partial conversion, e.g. conversion happens on a small region within a large region. To prevent such invalid cases and before any potential optimization comes out, all operations are performed with 4K granularity.
Signed-off-by Chenyi Qiang <[email protected]>--- include/sysemu/guest-memfd-manager.h | 46 +++++ system/guest-memfd-manager.c | 250 +++++++++++++++++++++++++++ system/meson.build | 1 + 3 files changed, 297 insertions(+)create mode 100644
include/sysemu/guest-memfd-manager.hcreate mode 100644system/guest-memfd-manager.cinclude/sysemu/guest-memfd-manager.h new file +16
-0@@ -0,0 +1,46 @@ +/* +* QEMU guest memfd manager +* +* Copyright Intel +* +* Author: +* Chenyi Qiang <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2 or later. +* See the COPYING file in the top-level directory +* +*/ + +#ifndef SYSEMU_GUEST_MEMFD_MANAGER_H +#define SYSEMU_GUEST_MEMFD_MANAGER_H +
include/sysemu/guest-memfd-manager.h line 26
Quoted 10 lines
include/sysemu/guest-memfd-manager.h new file +10
-0+#include "sysemu/hostmem.h" + +#define TYPE_GUEST_MEMFD_MANAGER "guest-memfd-manager" + +OBJECT_DECLARE_TYPE(GuestMemfdManager, GuestMemfdManagerClass, GUEST_MEMFD_MANAGER) + +struct GuestMemfdManager { +Object parent; + +/* Managed memory region. */
Do not need this comment. And the period.
include/sysemu/guest-memfd-manager.h line 31
Quoted 5 lines
include/sysemu/guest-memfd-manager.h new file +5
-0+MemoryRegion *mr; + +/* +* 1-setting of the bit represents the memory is populated (shared). +*/
Could be 1 line comment.
include/sysemu/guest-memfd-manager.h line 32
Quoted 1 line
include/sysemu/guest-memfd-manager.h new file +1
-0+int32_t bitmap_size;
int or unsigned
include/sysemu/guest-memfd-manager.h line 36
Quoted 4 lines
include/sysemu/guest-memfd-manager.h new file +4
-0+unsigned long *bitmap; + +/* block size and alignment */ +uint64_t block_size;
unsigned?
(u)int(32|64)_t make sense for migrations which is not the case (yet?). Thanks,
include/sysemu/guest-memfd-manager.h line 38
Quoted 2 lines
include/sysemu/guest-memfd-manager.h new file +2
-0+ +/* listeners to notify on populate/discard activity. */
Do not really need this comment either imho.
Quoted 77 lines
include/sysemu/guest-memfd-manager.h new file +8
-0+ QLIST_HEAD(, RamDiscardListener) rdl_list; +}; + +struct GuestMemfdManagerClass { +ObjectClass parent_class; +}; + +#endif system/guest-memfd-manager.c new file +63
-0@@ -0,0 +1,250 @@ +/* +* QEMU guest memfd manager +* +* Copyright Intel +* +* Author: +* Chenyi Qiang <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2 or later. +* See the COPYING file in the top-level directory +* +*/ + +#include "qemu/osdep.h" +#include "qemu/error-report.h" +#include "sysemu/guest-memfd-manager.h" + +OBJECT_DEFINE_SIMPLE_TYPE_WITH_INTERFACES (GuestMemfdManager, + guest_memfd_manager, +GUEST_MEMFD_MANAGER , +OBJECT , + { TYPE_RAM_DISCARD_MANAGER }, + { }) + +static bool guest_memfd_rdm_is_populated (const RamDiscardManager *rdm, +const MemoryRegionSection *section) +{ +const GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER (rdm); +uint64_t first_bit = section->offset_within_region / gmm->block_size; +uint64_t last_bit = first_bit +int128_get64 (section->size) / gmm->block_size -1 ; +unsigned long first_discard_bit; + + first_discard_bit =find_next_zero_bit (gmm->bitmap, last_bit +1 , first_bit); +return first_discard_bit > last_bit; +} + +typedef int (*guest_memfd_section_cb )(MemoryRegionSection *s,void *arg); + +static int guest_memfd_notify_populate_cb (MemoryRegionSection *section,void *arg) +{ +RamDiscardListener *rdl = arg; + +return rdl->notify_populate (rdl, section); +} + +static int guest_memfd_notify_discard_cb (MemoryRegionSection *section,void *arg) +{ +RamDiscardListener *rdl = arg; + + rdl->notify_discard (rdl, section); + +return 0 ; +} + +static int guest_memfd_for_each_populated_section (const GuestMemfdManager *gmm, +MemoryRegionSection *section, +void *arg, +guest_memfd_section_cb cb) +{ +unsigned long first_one_bit, last_one_bit; +uint64_t offset, size; +int ret =0 ; +
system/guest-memfd-manager.c line 73
Quoted 10 lines
system/guest-memfd-manager.c new file +10
-0+ first_one_bit = section->offset_within_region / gmm->block_size; + first_one_bit =find_next_bit (gmm->bitmap, gmm->bitmap_size, first_one_bit); + +while (first_one_bit < gmm->bitmap_size) { +MemoryRegionSection tmp = *section; + + offset = first_one_bit * gmm->block_size; + last_one_bit =find_next_zero_bit (gmm->bitmap, gmm->bitmap_size, + first_one_bit +1 ) -1 ; + size = (last_one_bit - first_one_bit +1 ) * gmm->block_size;
This tries calling cb() on bigger chunks even though we say from the beginning that only page size is supported?
May be simplify this for now and extend if/when VFIO learns to split mappings, or just drop it when we get in-place page state convertion (which will make this all irrelevant)?
Quoted 97 lines
system/guest-memfd-manager.c new file +97
-0+ +if (!memory_region_section_intersect_range (&tmp, offset, size)) { +break ; + } + + ret =cb (&tmp, arg); +if (ret) { +break ; + } + + first_one_bit =find_next_bit (gmm->bitmap, gmm->bitmap_size, + last_one_bit +2 ); + } + +return ret; +} + +static int guest_memfd_for_each_discarded_section (const GuestMemfdManager *gmm, +MemoryRegionSection *section, +void *arg, +guest_memfd_section_cb cb) +{ +unsigned long first_zero_bit, last_zero_bit; +uint64_t offset, size; +int ret =0 ; + + first_zero_bit = section->offset_within_region / gmm->block_size; + first_zero_bit =find_next_zero_bit (gmm->bitmap, gmm->bitmap_size, + first_zero_bit); + +while (first_zero_bit < gmm->bitmap_size) { +MemoryRegionSection tmp = *section; + + offset = first_zero_bit * gmm->block_size; + last_zero_bit =find_next_bit (gmm->bitmap, gmm->bitmap_size, + first_zero_bit +1 ) -1 ; + size = (last_zero_bit - first_zero_bit +1 ) * gmm->block_size; + +if (!memory_region_section_intersect_range (&tmp, offset, size)) { +break ; + } + + ret =cb (&tmp, arg); +if (ret) { +break ; + } + + first_zero_bit =find_next_zero_bit (gmm->bitmap, gmm->bitmap_size, + last_zero_bit +2 ); + } + +return ret; +} + +static uint64_t guest_memfd_rdm_get_min_granularity (const RamDiscardManager *rdm, +const MemoryRegion *mr) +{ +GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER (rdm); + +g_assert (mr == gmm->mr); +return gmm->block_size; +} + +static void guest_memfd_rdm_register_listener (RamDiscardManager *rdm, +RamDiscardListener *rdl, +MemoryRegionSection *section) +{ +GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER (rdm); +int ret; + +g_assert (section->mr == gmm->mr); + rdl->section =memory_region_section_new_copy (section); + +QLIST_INSERT_HEAD (&gmm->rdl_list, rdl, next); + + ret =guest_memfd_for_each_populated_section (gmm, section, rdl, + guest_memfd_notify_populate_cb); +if (ret) { +error_report ("%s: Failed to register RAM discard listener: %s" , __func__, +strerror (-ret)); + } +} + +static void guest_memfd_rdm_unregister_listener (RamDiscardManager *rdm, +RamDiscardListener *rdl) +{ +GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER (rdm); +int ret; + +g_assert (rdl->section); +g_assert (rdl->section->mr == gmm->mr); + + ret =guest_memfd_for_each_populated_section (gmm, rdl->section, rdl, + guest_memfd_notify_discard_cb); +if (ret) { +error_report ("%s: Failed to unregister RAM discard listener: %s" , __func__, +strerror (-ret));
system/guest-memfd-manager.c line 180
Quoted 10 lines
system/guest-memfd-manager.c new file +10
-0+ } + +memory_region_section_free_copy (rdl->section); + rdl->section =NULL ; +QLIST_REMOVE (rdl, next); + +} + +typedef struct GuestMemfdReplayData { + void *fn;
s/void */ReplayRamPopulate/
system/guest-memfd-manager.c line 186
Quoted 6 lines
system/guest-memfd-manager.c new file +6
-0+void *opaque; +} GuestMemfdReplayData; + +static int guest_memfd_rdm_replay_populated_cb (MemoryRegionSection *section,void *arg) +{ +struct GuestMemfdReplayData *data = arg;
Drop "struct" here and below.
Quoted 14 lines
system/guest-memfd-manager.c new file +14
-0+ReplayRamPopulate replay_fn = data->fn; + +return replay_fn (section, data->opaque); +} + +static int guest_memfd_rdm_replay_populated (const RamDiscardManager *rdm, +MemoryRegionSection *section, +ReplayRamPopulate replay_fn, +void *opaque) +{ +GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER (rdm); +struct GuestMemfdReplayData data = { .fn = replay_fn, .opaque = opaque }; + +g_assert (section->mr == gmm->mr);
system/guest-memfd-manager.c line 210
Quoted 10 lines
system/guest-memfd-manager.c new file +10
-0+return guest_memfd_for_each_populated_section (gmm, section, &data, + guest_memfd_rdm_replay_populated_cb); +} + +static int guest_memfd_rdm_replay_discarded_cb (MemoryRegionSection *section,void *arg) +{ +struct GuestMemfdReplayData *data = arg; +ReplayRamDiscard replay_fn = data->fn; + +replay_fn (section, data->opaque);
guest_memfd_rdm_replay_populated_cb() checks for errors though.
Quoted 17 lines
system/guest-memfd-manager.c new file +17
-0+ +return 0 ; +} + +static void guest_memfd_rdm_replay_discarded (const RamDiscardManager *rdm, +MemoryRegionSection *section, +ReplayRamDiscard replay_fn, +void *opaque) +{ +GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER (rdm); +struct GuestMemfdReplayData data = { .fn = replay_fn, .opaque = opaque }; + +g_assert (section->mr == gmm->mr); +guest_memfd_for_each_discarded_section (gmm, section, &data, + guest_memfd_rdm_replay_discarded_cb); +} +
system/guest-memfd-manager.c line 237
Quoted 10 lines
system/guest-memfd-manager.c new file +10
-0+static void guest_memfd_manager_init (Object *obj) +{ +GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER (obj); + +QLIST_INIT (&gmm->rdl_list); +} + +static void guest_memfd_manager_finalize (Object *obj) +{ +g_free (GUEST_MEMFD_MANAGER (obj)->bitmap);
bitmap is not allocated though. And 5/7 removes this anyway. Thanks,
Quoted 25 lines
system/guest-memfd-manager.c new file +13
-0+} + +static void guest_memfd_manager_class_init (ObjectClass *oc,void *data) +{ +RamDiscardManagerClass *rdmc =RAM_DISCARD_MANAGER_CLASS (oc); + + rdmc->get_min_granularity = guest_memfd_rdm_get_min_granularity; + rdmc->register_listener = guest_memfd_rdm_register_listener; + rdmc->unregister_listener = guest_memfd_rdm_unregister_listener; + rdmc->is_populated = guest_memfd_rdm_is_populated; + rdmc->replay_populated = guest_memfd_rdm_replay_populated; + rdmc->replay_discarded = guest_memfd_rdm_replay_discarded; +}system/meson.build +1
-0@@ -15,6 +15,7 @@ system_ss.add(files( 'dirtylimit.c', 'dma-helpers.c', 'globals.c', + 'guest-memfd-manager.c', 'memory_mapping.c', 'qdev-monitor.c', 'qtest.c',
Signature · 1 line
Alexey
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 26 lines · Alexey Kardashevskiy
Quoted 18 lines · Chenyi Qiang
As the commit
852f0048f3("RAMBlock: makeguest_memfdrequire uncoordinated discard") highlighted, some subsystems like VFIO might disable ram block discard. However,guest_memfdrelies on the discard operation to perform page conversion between private and shared memory. This can lead to stale IOMMU mapping issue when assigning a hardware device to a confidential VM via shared memory (unprotected memory pages). Blocking shared page discard can solve this problem, but it could cause guests to consume twice the memory with VFIO, which is not acceptable in some cases. An alternative solution is to convey other systems like VFIO to refresh its outdated IOMMU mappings.RamDiscardManager is an existing concept (used by virtio-mem) to adjust VFIO mappings in relation to VM page assignment. Effectively page conversion is similar to hot-removing a page in one mode and adding it back in the other, so the similar work that needs to happen in response to virtio-mem changes needs to happen for page conversion events. Introduce the RamDiscardManager to
guest_memfdto achieve it.Quoted 4 lines
However,
guest_memfdis not an object so it cannot directly implement the RamDiscardManager interface.One solution is to implement the interface in HostMemoryBackend. Any
This sounds about right.
Quoted 7 lines
Quoted 4 lines
guest_memfd-backed host memory backend can register itself in the target MemoryRegion. However, this solution doesn't cover the scenario where a
guest_memfdMemoryRegion doesn't belong to the HostMemoryBackend, e.g. the virtual BIOS MemoryRegion.What is this virtual BIOS MemoryRegion exactly? What does it look like in "info mtree -f"? Do we really want this memory to be DMAable?
virtual BIOS shows in a separate region:
Root memory region: system
0000000000000000-000000007fffffff (prio 0, ram): pc.ram KVM
...
00000000ffc00000-00000000ffffffff (prio 0, ram): pc.bios KVM 0000000100000000-000000017fffffff (prio 0, ram): pc.ram
@0000000080000000 KVM
We also consider to implement the interface in HostMemoryBackend, but maybe implement with guest_memfd region is more general. We don't know if any DMAable memory would belong to HostMemoryBackend although at present it is.
If it is more appropriate to implement it with HostMemoryBackend, I can change to this way.
Quoted 67 lines
Quoted 60 lines
Thus, choose the second option, i.e. define an object type named
guest_memfd_managerwith RamDiscardManager interface. Upon creation ofguest_memfd, a newguest_memfd_managerobject can be instantiated and registered to the managedguest_memfdMemoryRegion to handle the page conversion events.In the context of
guest_memfd, the discarded state signifies that the page is private, while the populated state indicated that the page is shared. The state of the memory is tracked at the granularity of the host page size (i.e.block_size), as the minimum conversion size can be one page per request.In addition, VFIO expects the DMA mapping for a specific iova to be mapped and unmapped with the same granularity. However, the confidential VMs may do partial conversion, e.g. conversion happens on a small region within a large region. To prevent such invalid cases and before any potential optimization comes out, all operations are performed with 4K granularity.
Signed-off-by Chenyi Qiang <[email protected]>---
include/sysemu/guest-memfd-manager.h| 46 +++++ system/guest-memfd-manager.c | 250 +++++++++++++++++++++++++++ system/meson.build | 1 +3 files changed, 297 insertions(+)create mode 100644
include/sysemu/guest-memfd-manager.hcreate mode 100644system/guest-memfd-manager.cinclude/sysemu/ renamed from include/sysemu/guest-memfd-manager.h +0
-0
guest-memfd-manager.h
new file mode 100644
index 0000000000..ba4a99b614include/sysemu/guest-memfd-manager.h new file +9
-0@@ -0,0 +1,46 @@ +/* +* QEMU guest memfd manager +* +* Copyright Intel +* +* Author: +* Chenyi Qiang <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory
+ *
+ */include/sysemu/guest-memfd-manager.h new file +9
-0+ +#ifndef SYSEMU_GUEST_MEMFD_MANAGER_H +#define SYSEMU_GUEST_MEMFD_MANAGER_H + +#include "sysemu/hostmem.h" + +#define TYPE_GUEST_MEMFD_MANAGER "guest-memfd-manager" + +OBJECT_DECLARE_TYPE(GuestMemfdManager, GuestMemfdManagerClass,
GUEST_MEMFD_MANAGER)include/sysemu/guest-memfd-manager.h line 26
Quoted 5 lines
include/sysemu/guest-memfd-manager.h new file +5
-0+ +struct GuestMemfdManager { + Object parent; + + /* Managed memory region. */Do not need this comment. And the period.
[...]
Quoted 6 lines
Quoted 6 lines
+ MemoryRegion *mr;
+
+ /*
+ * 1-setting of the bit represents the memory is populated (shared). + */
Will fix it.
Quoted 12 lines
Could be 1 line comment.
Quoted 1 line
+
int32_tbitmap_size;int or unsigned
Quoted 4 lines
+ unsigned long *bitmap;
+
+ /* block size and alignment */
+uint64_tblock_size;
Quoted 4 lines
unsigned?
(u)int(32|64)_t make sense for migrations which is not the case (yet?). Thanks,
I think these fields would be helpful for future migration support. Maybe defining as this way is more straightforward.
Quoted 6 lines
Quoted 3 lines
+
+ /* listeners to notify on populate/discard activity. */Do not really need this comment either imho.
I prefer to provide the comment for each field as virtio-mem do. If it is not necessary, I would remove those obvious ones.
Quoted 8 lines
include/sysemu/guest-memfd-manager.h line 46
Quoted 8 lines
+
QLIST_HEAD(, RamDiscardListener)rdl_list;include/sysemu/guest-memfd-manager.h new file +7
-0+}; + +struct GuestMemfdManagerClass { + ObjectClass parent_class; +}; + +#endif
[...]
void *arg,
Quoted 22 lines
Quoted 10 lines
+
guest_memfd_section_cbcb)include/sysemu/guest-memfd-manager.h new file +7
-0+{ + unsigned long first_one_bit, last_one_bit; + uint64_t offset, size; + int ret = 0; + + first_one_bit = section->offset_within_region / gmm->block_size; + first_one_bit = find_next_bit(gmm->bitmap, gmm->bitmap_size,
first_one_bit);include/sysemu/guest-memfd-manager.h
Quoted 8 lines
include/sysemu/guest-memfd-manager.h new file +8
-0+ + while (first_one_bit < gmm->bitmap_size) { + MemoryRegionSection tmp = *section; + + offset = first_one_bit * gmm->block_size; + last_one_bit = find_next_zero_bit(gmm->bitmap, gmm->bitmap_size, + first_one_bit + 1) - 1; + size = (last_one_bit - first_one_bit + 1) * gmm->block_size;This tries calling
cb()on bigger chunks even though we say from the beginning that only page size is supported?
Quoted 3 lines
May be simplify this for now and extend if/when VFIO learns to split mappings, or just drop it when we get in-place page state convertion (which will make this all irrelevant)?
The cb() will call with big chunks but actually it do the split with the granularity of block_size in the cb(). See the vfio_ram_discard_notify_populate(), which do the DMA_MAP with granularity size.
Quoted 124 lines
Quoted 113 lines
+
+ if (!memory_region_section_intersect_range(&tmp, offset, size)) {
+ break;
+ }include/sysemu/guest-memfd-manager.h new file +14
-0+ + ret = cb(&tmp, arg); + if (ret) { + break; + } + + first_one_bit = find_next_bit(gmm->bitmap, gmm->bitmap_size, + last_one_bit + 2); + } + + return ret; +} + +static int guest_memfd_for_each_discarded_section(constGuestMemfdManager *gmm,
+ MemoryRegionSection *section,
+ void *arg, +guest_memfd_section_cbcb)include/sysemu/guest-memfd-manager.h new file +18
-0+{ + unsigned long first_zero_bit, last_zero_bit; + uint64_t offset, size; + int ret = 0; + + first_zero_bit = section->offset_within_region / gmm->block_size; + first_zero_bit = find_next_zero_bit(gmm->bitmap, gmm->bitmap_size, + first_zero_bit); + + while (first_zero_bit < gmm->bitmap_size) { + MemoryRegionSection tmp = *section; + + offset = first_zero_bit * gmm->block_size; + last_zero_bit = find_next_bit(gmm->bitmap, gmm->bitmap_size, + first_zero_bit + 1) - 1; + size = (last_zero_bit - first_zero_bit + 1) * gmm->block_size; + + if (!memory_region_section_intersect_range(&tmp, offset,size)) {
+ break;
+ }include/sysemu/guest-memfd-manager.h new file +7
-0+ + ret = cb(&tmp, arg); + if (ret) { + break; + } + + first_zero_bit = find_next_zero_bit(gmm->bitmap, gmm-Quoted 1 line
bitmap_size,+
last_zero_bit+ 2); + }include/sysemu/guest-memfd-manager.h new file +5
-0+ + return ret; +} + +static uint64_t guest_memfd_rdm_get_min_granularity(constRamDiscardManager *rdm,
+ const MemoryRegion *mr)include/sysemu/guest-memfd-manager.h new file +10
-0+{ + GuestMemfdManager *gmm = GUEST_MEMFD_MANAGER(rdm); + + g_assert(mr == gmm->mr); + return gmm->block_size; +} + +static void guest_memfd_rdm_register_listener (RamDiscardManager *rdm, + RamDiscardListener *rdl, + MemoryRegionSection*section)
include/sysemu/guest-memfd-manager.h new file +11
-0+{ + GuestMemfdManager *gmm = GUEST_MEMFD_MANAGER(rdm); + int ret; + + g_assert(section->mr == gmm->mr); + rdl->section = memory_region_section_new_copy(section); + + QLIST_INSERT_HEAD(&gmm->rdl_list, rdl, next); + + ret = guest_memfd_for_each_populated_section(gmm, section, rdl, +
guest_memfd_notify_populate_cb);
+ if (ret) {
+ error_report("%s: Failed to register RAM discard listener:
%s",__func__,
+ strerror(-ret));
+ }include/sysemu/guest-memfd-manager.h new file +13
-0+} + +static void guest_memfd_rdm_unregister_listener (RamDiscardManager *rdm, + RamDiscardListener *rdl) +{ + GuestMemfdManager *gmm = GUEST_MEMFD_MANAGER(rdm); + int ret; + + g_assert(rdl->section); + g_assert(rdl->section->mr == gmm->mr); + + ret = guest_memfd_for_each_populated_section(gmm, rdl->section, rdl, +
guest_memfd_notify_discard_cb);
+ if (ret) {
+ error_report("%s: Failed to unregister RAM discard listener:
%s",__func__,
+ strerror(-ret));
+ }include/sysemu/guest-memfd-manager.h
Quoted 9 lines
include/sysemu/guest-memfd-manager.h new file +9
-0+ + memory_region_section_free_copy(rdl->section); + rdl->section = NULL; + QLIST_REMOVE(rdl, next); + +} + +typedef struct GuestMemfdReplayData { + void *fn;s/void */ReplayRamPopulate/
[...]
Quoted 10 lines
Quoted 8 lines
+ void *opaque;
include/sysemu/guest-memfd-manager.h new file +3
-0+} GuestMemfdReplayData; + +static int guest_memfd_rdm_replay_populated_cb(MemoryRegionSection*section, void *arg)
+{
+ struct GuestMemfdReplayData *data = arg;Drop "struct" here and below.
Fixed. Thanks!
Quoted 25 lines
Quoted 25 lines
+ ReplayRamPopulate
replay_fn=data->fn;include/sysemu/guest-memfd-manager.h new file +5
-0+ + return replay_fn(section, data->opaque); +} + +static int guest_memfd_rdm_replay_populated(const RamDiscardManager*rdm,
+ MemoryRegionSection *section,
+ ReplayRamPopulatereplay_fn, + void *opaque)
+{
+ GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER(rdm);
+ struct GuestMemfdReplayData data = { .fn =replay_fn, .opaque = opaque };
+
+g_assert(section->mr == gmm->mr);
+ return guest_memfd_for_each_populated_section(gmm, section, &data, +guest_memfd_rdm_replay_populated_cb);include/sysemu/guest-memfd-manager.h new file +3
-0+} + +static int guest_memfd_rdm_replay_discarded_cb(MemoryRegionSection*section, void *arg)
Quoted 8 lines
include/sysemu/guest-memfd-manager.h
Quoted 5 lines
include/sysemu/guest-memfd-manager.h new file +5
-0+{ + struct GuestMemfdReplayData *data = arg; + ReplayRamDiscard replay_fn = data->fn; + + replay_fn(section, data->opaque);
guest_memfd_rdm_replay_populated_cb()checks for errors though.
It follows current definiton of ReplayRamDiscard() and ReplayRamPopulate() where replay_discard() doesn't return errors and replay_populate() returns errors.
Quoted 27 lines
Quoted 27 lines
include/sysemu/guest-memfd-manager.h new file +5
-0+ + return 0; +} + +static void guest_memfd_rdm_replay_discarded(const RamDiscardManager*rdm,
+ MemoryRegionSection *section,
+ ReplayRamDiscardreplay_fn, + void *opaque) +{
+ GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER(rdm);
+ struct GuestMemfdReplayData data = { .fn =replay_fn, .opaque = opaque };
+
+g_assert(section->mr == gmm->mr);
+ guest_memfd_for_each_discarded_section(gmm, section, &data, +guest_memfd_rdm_replay_discarded_cb);include/sysemu/guest-memfd-manager.h new file +7
-0+} + +static void guest_memfd_manager_init (Object *obj) +{ + GuestMemfdManager *gmm = GUEST_MEMFD_MANAGER(obj); + + QLIST_INIT(&gmm->rdl_list);
Quoted 8 lines
include/sysemu/guest-memfd-manager.h
Quoted 5 lines
include/sysemu/guest-memfd-manager.h new file +5
-0+} + +static void guest_memfd_manager_finalize (Object *obj) +{ + g_free(GUEST_MEMFD_MANAGER(obj)->bitmap);bitmap is not allocated though. And 5/7 removes this anyway. Thanks,
Will remove it. Thanks.
Quoted 28 lines
Quoted 27 lines
include/sysemu/guest-memfd-manager.h new file +13
-0+} + +static void guest_memfd_manager_class_init (ObjectClass *oc,void *data) +{ + RamDiscardManagerClass *rdmc = RAM_DISCARD_MANAGER_CLASS(oc); + + rdmc->get_min_granularity = guest_memfd_rdm_get_min_granularity; + rdmc->register_listener = guest_memfd_rdm_register_listener; + rdmc->unregister_listener = guest_memfd_rdm_unregister_listener; + rdmc->is_populated = guest_memfd_rdm_is_populated; + rdmc->replay_populated = guest_memfd_rdm_replay_populated; + rdmc->replay_discarded = guest_memfd_rdm_replay_discarded; +}system/meson.build +1
-0@@ -15,6 +15,7 @@ system_ss.add(files( 'dirtylimit.c', 'dma-helpers.c', 'globals.c', + 'guest-memfd-manager.c', 'memory_mapping.c', 'qdev-monitor.c', 'qtest.c',
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 37 lines · Chenyi Qiang
Quoted 26 lines · Alexey Kardashevskiy
Quoted 18 lines · Chenyi Qiang
As the commit
852f0048f3("RAMBlock: makeguest_memfdrequire uncoordinated discard") highlighted, some subsystems like VFIO might disable ram block discard. However,guest_memfdrelies on the discard operation to perform page conversion between private and shared memory. This can lead to stale IOMMU mapping issue when assigning a hardware device to a confidential VM via shared memory (unprotected memory pages). Blocking shared page discard can solve this problem, but it could cause guests to consume twice the memory with VFIO, which is not acceptable in some cases. An alternative solution is to convey other systems like VFIO to refresh its outdated IOMMU mappings.RamDiscardManager is an existing concept (used by virtio-mem) to adjust VFIO mappings in relation to VM page assignment. Effectively page conversion is similar to hot-removing a page in one mode and adding it back in the other, so the similar work that needs to happen in response to virtio-mem changes needs to happen for page conversion events. Introduce the RamDiscardManager to
guest_memfdto achieve it.Quoted 4 lines
However,
guest_memfdis not an object so it cannot directly implement the RamDiscardManager interface.One solution is to implement the interface in HostMemoryBackend. Any
This sounds about right.
Quoted 7 lines
Quoted 4 lines
guest_memfd-backed host memory backend can register itself in the target MemoryRegion. However, this solution doesn't cover the scenario where a
guest_memfdMemoryRegion doesn't belong to the HostMemoryBackend, e.g. the virtual BIOS MemoryRegion.What is this virtual BIOS MemoryRegion exactly? What does it look like in "info mtree -f"? Do we really want this memory to be DMAable?
Quoted 6 lines
virtual BIOS shows in a separate region:
Root memory region: system
0000000000000000-000000007fffffff (prio 0, ram): pc.ram KVM
...
00000000ffc00000-00000000ffffffff (prio 0, ram): pc.bios KVM
Looks like a normal MR which can be backed by guest_memfd.
Quoted 2 lines
0000000100000000-000000017fffffff (prio 0, ram): pc.ram@0000000080000000 KVM
Anyway if there is no guest_memfd backing it and memory_region_has_ram_discard_manager() returns false, then the MR is just going to be mapped for VFIO as usual which seems... alright, right?
Quoted 7 lines
We also consider to implement the interface in HostMemoryBackend, but maybe implement with
guest_memfdregion is more general. We don't know if any DMAable memory would belong to HostMemoryBackend although at present it is.If it is more appropriate to implement it with HostMemoryBackend, I can change to this way.
Seems cleaner imho.
Quoted 100 lines
Quoted 67 lines
Quoted 60 lines
Thus, choose the second option, i.e. define an object type named
guest_memfd_managerwith RamDiscardManager interface. Upon creation ofguest_memfd, a newguest_memfd_managerobject can be instantiated and registered to the managedguest_memfdMemoryRegion to handle the page conversion events.In the context of
guest_memfd, the discarded state signifies that the page is private, while the populated state indicated that the page is shared. The state of the memory is tracked at the granularity of the host page size (i.e.block_size), as the minimum conversion size can be one page per request.In addition, VFIO expects the DMA mapping for a specific iova to be mapped and unmapped with the same granularity. However, the confidential VMs may do partial conversion, e.g. conversion happens on a small region within a large region. To prevent such invalid cases and before any potential optimization comes out, all operations are performed with 4K granularity.
Signed-off-by Chenyi Qiang <[email protected]>--- include/sysemu/guest-memfd-manager.h | 46 +++++ system/guest-memfd-manager.c | 250 +++++++++++++++++++++++++++ system/meson.build | 1 + 3 files changed, 297 insertions(+)create mode 100644
include/sysemu/guest-memfd-manager.hcreate mode 100644system/guest-memfd-manager.cinclude/sysemu/ renamed from include/sysemu/guest-memfd-manager.h +0
-0
guest-memfd-manager.h
new file mode 100644
index 0000000000..ba4a99b614include/sysemu/guest-memfd-manager.h new file +9
-0@@ -0,0 +1,46 @@ +/* +* QEMU guest memfd manager +* +* Copyright Intel +* +* Author: +* Chenyi Qiang <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory
+ *
+ */include/sysemu/guest-memfd-manager.h new file +9
-0+ +#ifndef SYSEMU_GUEST_MEMFD_MANAGER_H +#define SYSEMU_GUEST_MEMFD_MANAGER_H + +#include "sysemu/hostmem.h" + +#define TYPE_GUEST_MEMFD_MANAGER "guest-memfd-manager" + +OBJECT_DECLARE_TYPE(GuestMemfdManager, GuestMemfdManagerClass,
GUEST_MEMFD_MANAGER)include/sysemu/guest-memfd-manager.h line 26
Quoted 5 lines
include/sysemu/guest-memfd-manager.h new file +5
-0+ +struct GuestMemfdManager { + Object parent; + + /* Managed memory region. */Do not need this comment. And the period.
[...]
Quoted 6 lines
Quoted 6 lines
+ MemoryRegion *mr;
+
+ /*
+ * 1-setting of the bit represents the memory is populated (shared). + */Will fix it.
Quoted 12 lines
Could be 1 line comment.
Quoted 1 line
+
int32_tbitmap_size;int or unsigned
Quoted 4 lines
+ unsigned long *bitmap;
+
+ /* block size and alignment */
+uint64_tblock_size;Quoted 4 lines
unsigned?
(u)int(32|64)_t make sense for migrations which is not the case (yet?). Thanks,
I think these fields would be helpful for future migration support. Maybe defining as this way is more straightforward.
Quoted 1 line
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
+
+ /* listeners to notify on populate/discard activity. */Do not really need this comment either imho.
I prefer to provide the comment for each field as virtio-mem do. If it is not necessary, I would remove those obvious ones.
[bikeshedding on] But the "RamDiscardListener" word says that already, why repeating? :) It should add information, not duplicate. Like the block_size comment which mentions "alignment" [bikeshedding off]
Quoted 34 lines
Quoted 8 lines
include/sysemu/guest-memfd-manager.h line 46
Quoted 8 lines
+
QLIST_HEAD(, RamDiscardListener)rdl_list;include/sysemu/guest-memfd-manager.h new file +7
-0+}; + +struct GuestMemfdManagerClass { + ObjectClass parent_class; +}; + +#endif [...]
void *arg,
Quoted 22 lines
Quoted 10 lines
+
guest_memfd_section_cbcb)include/sysemu/guest-memfd-manager.h new file +7
-0+{ + unsigned long first_one_bit, last_one_bit; + uint64_t offset, size; + int ret = 0; + + first_one_bit = section->offset_within_region / gmm->block_size; + first_one_bit = find_next_bit(gmm->bitmap, gmm->bitmap_size,
first_one_bit);include/sysemu/guest-memfd-manager.h
Quoted 8 lines
include/sysemu/guest-memfd-manager.h new file +8
-0+ + while (first_one_bit < gmm->bitmap_size) { + MemoryRegionSection tmp = *section; + + offset = first_one_bit * gmm->block_size; + last_one_bit = find_next_zero_bit(gmm->bitmap, gmm->bitmap_size, + first_one_bit + 1) - 1; + size = (last_one_bit - first_one_bit + 1) * gmm->block_size;This tries calling
cb()on bigger chunks even though we say from the beginning that only page size is supported?
Quoted 8 lines
Quoted 3 lines
May be simplify this for now and extend if/when VFIO learns to split mappings, or just drop it when we get in-place page state convertion (which will make this all irrelevant)?
The
cb()will call with big chunks but actually it do the split with the granularity ofblock_sizein thecb(). See thevfio_ram_discard_notify_populate(), which do theDMA_MAPwith granularity size.
Right, and this all happens inside QEMU - first the code finds bigger chunks and then it splits them anyway to call the VFIO driver. Seems pointless to bother about bigger chunks here.
Quoted 175 lines
Quoted 125 lines
Quoted 114 lines
+
+ if (!memory_region_section_intersect_range(&tmp, offset, size)) {
+ break;
+ }include/sysemu/guest-memfd-manager.h new file +14
-0+ + ret = cb(&tmp, arg); + if (ret) { + break; + } + + first_one_bit = find_next_bit(gmm->bitmap, gmm->bitmap_size, + last_one_bit + 2); + } + + return ret; +} + +static int guest_memfd_for_each_discarded_section(constGuestMemfdManager *gmm,
+ MemoryRegionSection *section,
+ void *arg, +guest_memfd_section_cbcb)include/sysemu/guest-memfd-manager.h new file +18
-0+{ + unsigned long first_zero_bit, last_zero_bit; + uint64_t offset, size; + int ret = 0; + + first_zero_bit = section->offset_within_region / gmm->block_size; + first_zero_bit = find_next_zero_bit(gmm->bitmap, gmm->bitmap_size, + first_zero_bit); + + while (first_zero_bit < gmm->bitmap_size) { + MemoryRegionSection tmp = *section; + + offset = first_zero_bit * gmm->block_size; + last_zero_bit = find_next_bit(gmm->bitmap, gmm->bitmap_size, + first_zero_bit + 1) - 1; + size = (last_zero_bit - first_zero_bit + 1) * gmm->block_size; + + if (!memory_region_section_intersect_range(&tmp, offset,size)) {
+ break;
+ }include/sysemu/guest-memfd-manager.h new file +7
-0+ + ret = cb(&tmp, arg); + if (ret) { + break; + } + + first_zero_bit = find_next_zero_bit(gmm->bitmap, gmm-Quoted 1 line
bitmap_size,+
last_zero_bit+ 2); + }include/sysemu/guest-memfd-manager.h new file +5
-0+ + return ret; +} + +static uint64_t guest_memfd_rdm_get_min_granularity(constRamDiscardManager *rdm,
+ const MemoryRegion *mr)include/sysemu/guest-memfd-manager.h new file +10
-0+{ + GuestMemfdManager *gmm = GUEST_MEMFD_MANAGER(rdm); + + g_assert(mr == gmm->mr); + return gmm->block_size; +} + +static void guest_memfd_rdm_register_listener (RamDiscardManager *rdm, + RamDiscardListener *rdl, + MemoryRegionSection*section)
include/sysemu/guest-memfd-manager.h new file +11
-0+{ + GuestMemfdManager *gmm = GUEST_MEMFD_MANAGER(rdm); + int ret; + + g_assert(section->mr == gmm->mr); + rdl->section = memory_region_section_new_copy(section); + + QLIST_INSERT_HEAD(&gmm->rdl_list, rdl, next); + + ret = guest_memfd_for_each_populated_section(gmm, section, rdl, +
guest_memfd_notify_populate_cb);
+ if (ret) {
+ error_report("%s: Failed to register RAM discard listener:
%s",__func__,
+ strerror(-ret));
+ }include/sysemu/guest-memfd-manager.h new file +13
-0+} + +static void guest_memfd_rdm_unregister_listener (RamDiscardManager *rdm, + RamDiscardListener *rdl) +{ + GuestMemfdManager *gmm = GUEST_MEMFD_MANAGER(rdm); + int ret; + + g_assert(rdl->section); + g_assert(rdl->section->mr == gmm->mr); + + ret = guest_memfd_for_each_populated_section(gmm, rdl->section, rdl, +
guest_memfd_notify_discard_cb);
+ if (ret) {
+ error_report("%s: Failed to unregister RAM discard listener:
%s",__func__,
+ strerror(-ret));
+ }include/sysemu/guest-memfd-manager.h
Quoted 9 lines
include/sysemu/guest-memfd-manager.h new file +9
-0+ + memory_region_section_free_copy(rdl->section); + rdl->section = NULL; + QLIST_REMOVE(rdl, next); + +} + +typedef struct GuestMemfdReplayData { + void *fn;s/void */ReplayRamPopulate/
[...]
Quoted 10 lines
Quoted 8 lines
+ void *opaque;
include/sysemu/guest-memfd-manager.h new file +3
-0+} GuestMemfdReplayData; + +static int guest_memfd_rdm_replay_populated_cb(MemoryRegionSection*section, void *arg)
+{
+ struct GuestMemfdReplayData *data = arg;Drop "struct" here and below.
Fixed. Thanks!
Quoted 25 lines
Quoted 25 lines
+ ReplayRamPopulate
replay_fn=data->fn;include/sysemu/guest-memfd-manager.h new file +5
-0+ + return replay_fn(section, data->opaque); +} + +static int guest_memfd_rdm_replay_populated(const RamDiscardManager*rdm,
+ MemoryRegionSection *section,
+ ReplayRamPopulatereplay_fn, + void *opaque)
+{
+ GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER(rdm);
+ struct GuestMemfdReplayData data = { .fn =replay_fn, .opaque = opaque };include/sysemu/guest-memfd-manager.h new file +4
-0+ + g_assert(section->mr == gmm->mr); + return guest_memfd_for_each_populated_section(gmm, section, &data, +
guest_memfd_rdm_replay_populated_cb);include/sysemu/guest-memfd-manager.h new file +3
-0+} + +static int guest_memfd_rdm_replay_discarded_cb(MemoryRegionSection*section, void *arg)
Quoted 8 lines
include/sysemu/guest-memfd-manager.h
Quoted 5 lines
include/sysemu/guest-memfd-manager.h new file +5
-0+{ + struct GuestMemfdReplayData *data = arg; + ReplayRamDiscard replay_fn = data->fn; + + replay_fn(section, data->opaque);
guest_memfd_rdm_replay_populated_cb()checks for errors though.
Quoted 3 lines
It follows current definiton of
ReplayRamDiscard()andReplayRamPopulate()wherereplay_discard()doesn't return errors andreplay_populate()returns errors.
A trace would be appropriate imho. Thanks,
Quoted 67 lines
Quoted 27 lines
Quoted 27 lines
include/sysemu/guest-memfd-manager.h new file +5
-0+ + return 0; +} + +static void guest_memfd_rdm_replay_discarded(const RamDiscardManager*rdm,
+ MemoryRegionSection *section,
+ ReplayRamDiscardreplay_fn, + void *opaque) +{
+ GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER(rdm);
+ struct GuestMemfdReplayData data = { .fn =replay_fn, .opaque = opaque };include/sysemu/guest-memfd-manager.h new file +4
-0+ + g_assert(section->mr == gmm->mr); + guest_memfd_for_each_discarded_section(gmm, section, &data, +
guest_memfd_rdm_replay_discarded_cb);include/sysemu/guest-memfd-manager.h new file +7
-0+} + +static void guest_memfd_manager_init (Object *obj) +{ + GuestMemfdManager *gmm = GUEST_MEMFD_MANAGER(obj); + + QLIST_INIT(&gmm->rdl_list);Quoted 8 lines
include/sysemu/guest-memfd-manager.h
Quoted 5 lines
include/sysemu/guest-memfd-manager.h new file +5
-0+} + +static void guest_memfd_manager_finalize (Object *obj) +{ + g_free(GUEST_MEMFD_MANAGER(obj)->bitmap);bitmap is not allocated though. And 5/7 removes this anyway. Thanks,
Will remove it. Thanks.
Quoted 28 lines
Quoted 27 lines
include/sysemu/guest-memfd-manager.h new file +13
-0+} + +static void guest_memfd_manager_class_init (ObjectClass *oc,void *data) +{ + RamDiscardManagerClass *rdmc = RAM_DISCARD_MANAGER_CLASS(oc); + + rdmc->get_min_granularity = guest_memfd_rdm_get_min_granularity; + rdmc->register_listener = guest_memfd_rdm_register_listener; + rdmc->unregister_listener = guest_memfd_rdm_unregister_listener; + rdmc->is_populated = guest_memfd_rdm_is_populated; + rdmc->replay_populated = guest_memfd_rdm_replay_populated; + rdmc->replay_discarded = guest_memfd_rdm_replay_discarded; +}system/meson.build +1
-0@@ -15,6 +15,7 @@ system_ss.add(files( 'dirtylimit.c', 'dma-helpers.c', 'globals.c', + 'guest-memfd-manager.c', 'memory_mapping.c', 'qdev-monitor.c', 'qtest.c',
Signature · 1 line
Alexey
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 41 lines · Alexey Kardashevskiy
Quoted 38 lines · Chenyi Qiang
Quoted 34 lines · Alexey Kardashevskiy
Quoted 18 lines · Chenyi Qiang
As the commit
852f0048f3("RAMBlock: makeguest_memfdrequire uncoordinated discard") highlighted, some subsystems like VFIO might disable ram block discard. However,guest_memfdrelies on the discard operation to perform page conversion between private and shared memory. This can lead to stale IOMMU mapping issue when assigning a hardware device to a confidential VM via shared memory (unprotected memory pages). Blocking shared page discard can solve this problem, but it could cause guests to consume twice the memory with VFIO, which is not acceptable in some cases. An alternative solution is to convey other systems like VFIO to refresh its outdated IOMMU mappings.RamDiscardManager is an existing concept (used by virtio-mem) to adjust VFIO mappings in relation to VM page assignment. Effectively page conversion is similar to hot-removing a page in one mode and adding it back in the other, so the similar work that needs to happen in response to virtio-mem changes needs to happen for page conversion events. Introduce the RamDiscardManager to
guest_memfdto achieve it.Quoted 4 lines
However,
guest_memfdis not an object so it cannot directly implement the RamDiscardManager interface.One solution is to implement the interface in HostMemoryBackend. Any
This sounds about right.
Quoted 5 lines
guest_memfd-backed host memory backend can register itself in the target
MemoryRegion. However, this solution doesn't cover the scenario where aguest_memfdMemoryRegion doesn't belong to the HostMemoryBackend, e.g. the virtual BIOS MemoryRegion.What is this virtual BIOS MemoryRegion exactly? What does it look like in "info mtree -f"? Do we really want this memory to be DMAable?
Quoted 8 lines
Quoted 6 lines
virtual BIOS shows in a separate region:
Root memory region: system 0000000000000000-000000007fffffff (prio 0, ram): pc.ram KVM
...
00000000ffc00000-00000000ffffffff (prio 0, ram): pc.bios KVMLooks like a normal MR which can be backed by
guest_memfd.
Yes, virtual BIOS memory region is initialized by memory_region_init_ram_guest_memfd() which will be backed by a guest_memfd.
The tricky thing is, for Intel TDX (not sure about AMD SEV), the virtual BIOS image will be loaded and then copied to private region. After that, the loaded image will be discarded and this region become useless. So I feel like this virtual BIOS should not be backed by guest_memfd?
Quoted 7 lines
Quoted 3 lines
0000000100000000-000000017fffffff (prio 0, ram): pc.ram @0000000080000000 KVM
Anyway if there is no
guest_memfdbacking it andmemory_region_has_ram_discard_manager()returns false, then the MR is just going to be mapped for VFIO as usual which seems... alright, right?
Correct. As the vBIOS is backed by guest_memfd and we implement the RDM for guest_memfd_manager, the vBIOS MR won't be mapped by VFIO.
If we go with the HostMemoryBackend instead of guest_memfd_manager, this MR would be mapped by VFIO. Maybe need to avoid such vBIOS mapping, or just ignore it since the MR is useless (but looks not so good).
Quoted 11 lines
Quoted 9 lines
We also consider to implement the interface in HostMemoryBackend, but maybe implement with
guest_memfdregion is more general. We don't know if any DMAable memory would belong to HostMemoryBackend although at present it is.If it is more appropriate to implement it with HostMemoryBackend, I can change to this way.
Seems cleaner imho.
I can go this way.
Quoted 111 lines
Quoted 111 lines
Quoted 71 lines
Quoted 64 lines
Thus, choose the second option, i.e. define an object type named
guest_memfd_managerwith RamDiscardManager interface. Upon creation ofguest_memfd, a newguest_memfd_managerobject can be instantiated and registered to the managedguest_memfdMemoryRegion to handle the page conversion events.In the context of
guest_memfd, the discarded state signifies that the page is private, while the populated state indicated that the page is shared. The state of the memory is tracked at the granularity of the host page size (i.e.block_size), as the minimum conversion size can be one page per request.In addition, VFIO expects the DMA mapping for a specific iova to be mapped and unmapped with the same granularity. However, the confidential
VMs may do partial conversion, e.g. conversion happens on a small region
within a large region. To prevent such invalid cases and before any potential optimization comes out, all operations are performed with 4K granularity.Signed-off-by Chenyi Qiang <[email protected]>---
include/sysemu/guest-memfd-manager.h| 46 +++++ system/guest-memfd-manager.c | 250 ++++++++++++++++++++++
+++++
system/meson.build | 1 +3 files changed, 297 insertions(+)create mode 100644
include/sysemu/guest-memfd-manager.hcreate mode 100644system/guest-memfd-manager.cinclude/sysemu/ renamed from include/sysemu/guest-memfd-manager.h +0
-0
guest-memfd-manager.h
new file mode 100644
index 0000000000..ba4a99b614include/sysemu/guest-memfd-manager.h new file +9
-0@@ -0,0 +1,46 @@ +/* +* QEMU guest memfd manager +* +* Copyright Intel +* +* Author: +* Chenyi Qiang <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory
+ *
+ */include/sysemu/guest-memfd-manager.h new file +9
-0+ +#ifndef SYSEMU_GUEST_MEMFD_MANAGER_H +#define SYSEMU_GUEST_MEMFD_MANAGER_H + +#include "sysemu/hostmem.h" + +#define TYPE_GUEST_MEMFD_MANAGER "guest-memfd-manager" + +OBJECT_DECLARE_TYPE(GuestMemfdManager, GuestMemfdManagerClass,
GUEST_MEMFD_MANAGER)include/sysemu/guest-memfd-manager.h line 26
Quoted 5 lines
include/sysemu/guest-memfd-manager.h new file +5
-0+ +struct GuestMemfdManager { + Object parent; + + /* Managed memory region. */Do not need this comment. And the period.
[...]
Quoted 7 lines
Quoted 7 lines
+ MemoryRegion *mr;
+
+ /*
+ * 1-setting of the bit represents the memory is populated (shared).
+ */Will fix it.
Quoted 12 lines
Could be 1 line comment.
Quoted 1 line
+
int32_tbitmap_size;int or unsigned
Quoted 4 lines
+ unsigned long *bitmap;
+
+ /* block size and alignment */
+uint64_tblock_size;Quoted 4 lines
unsigned?
(u)int(32|64)_t make sense for migrations which is not the case (yet?). Thanks,
I think these fields would be helpful for future migration support. Maybe defining as this way is more straightforward.
Quoted 6 lines
Quoted 3 lines
+
+ /* listeners to notify on populate/discard activity. */Do not really need this comment either imho.
Quoted 6 lines
Quoted 2 lines
I prefer to provide the comment for each field as virtio-mem do. If it is not necessary, I would remove those obvious ones.
[bikeshedding on] But the "RamDiscardListener" word says that already, why repeating? :) It should add information, not duplicate. Like the
block_sizecomment which mentions "alignment" [bikeshedding off]
Got it. Thanks!
Quoted 229 lines
Quoted 36 lines
Quoted 9 lines
include/sysemu/guest-memfd-manager.h line 46
Quoted 9 lines
+
QLIST_HEAD(, RamDiscardListener)rdl_list;include/sysemu/guest-memfd-manager.h new file +7
-0+}; + +struct GuestMemfdManagerClass { + ObjectClass parent_class; +}; + +#endif [...]
void *arg,
Quoted 23 lines
Quoted 17 lines
+
guest_memfd_section_cbcb)include/sysemu/guest-memfd-manager.h new file +7
-0+{ + unsigned long first_one_bit, last_one_bit; + uint64_t offset, size; + int ret = 0; + + first_one_bit = section->offset_within_region / gmm->block_size; + first_one_bit = find_next_bit(gmm->bitmap, gmm->bitmap_size,
first_one_bit);include/sysemu/guest-memfd-manager.h new file +6
-0+ + while (first_one_bit < gmm->bitmap_size) { + MemoryRegionSection tmp = *section; + + offset = first_one_bit * gmm->block_size; + last_one_bit = find_next_zero_bit(gmm->bitmap, gmm-Quoted 1 line
bitmap_size,Quoted 2 lines
+
first_one_bit+ 1) - 1; + size = (last_one_bit-first_one_bit+ 1) *gmm->block_size;This tries calling
cb()on bigger chunks even though we say from the beginning that only page size is supported?Quoted 8 lines
Quoted 3 lines
May be simplify this for now and extend if/when VFIO learns to split mappings, or just drop it when we get in-place page state convertion (which will make this all irrelevant)?
The
cb()will call with big chunks but actually it do the split with the granularity ofblock_sizein thecb(). See thevfio_ram_discard_notify_populate(), which do theDMA_MAPwith granularity size.Right, and this all happens inside QEMU - first the code finds bigger chunks and then it splits them anyway to call the VFIO driver. Seems pointless to bother about bigger chunks here.
Quoted 179 lines
Quoted 128 lines
Quoted 117 lines
+
+ if (!memory_region_section_intersect_range(&tmp, offset, size)) {
+ break;
+ }include/sysemu/guest-memfd-manager.h new file +14
-0+ + ret = cb(&tmp, arg); + if (ret) { + break; + } + + first_one_bit = find_next_bit(gmm->bitmap, gmm->bitmap_size, + last_one_bit + 2); + } + + return ret; +} + +static int guest_memfd_for_each_discarded_section(constGuestMemfdManager *gmm,
+ MemoryRegionSection *section,
+ void *arg, +guest_memfd_section_cbcb)include/sysemu/guest-memfd-manager.h new file +18
-0+{ + unsigned long first_zero_bit, last_zero_bit; + uint64_t offset, size; + int ret = 0; + + first_zero_bit = section->offset_within_region / gmm->block_size; + first_zero_bit = find_next_zero_bit(gmm->bitmap, gmm->bitmap_size, + first_zero_bit); + + while (first_zero_bit < gmm->bitmap_size) { + MemoryRegionSection tmp = *section; + + offset = first_zero_bit * gmm->block_size; + last_zero_bit = find_next_bit(gmm->bitmap, gmm->bitmap_size, + first_zero_bit + 1) - 1; + size = (last_zero_bit - first_zero_bit + 1) * gmm->block_size; + + if (!memory_region_section_intersect_range(&tmp, offset,size)) {
+ break;
+ }include/sysemu/guest-memfd-manager.h new file +7
-0+ + ret = cb(&tmp, arg); + if (ret) { + break; + } + + first_zero_bit = find_next_zero_bit(gmm->bitmap, gmm-Quoted 1 line
bitmap_size,+
last_zero_bit+ 2); + }include/sysemu/guest-memfd-manager.h new file +5
-0+ + return ret; +} + +static uint64_t guest_memfd_rdm_get_min_granularity(constRamDiscardManager *rdm,
+ const MemoryRegion *mr)include/sysemu/guest-memfd-manager.h new file +10
-0+{ + GuestMemfdManager *gmm = GUEST_MEMFD_MANAGER(rdm); + + g_assert(mr == gmm->mr); + return gmm->block_size; +} + +static void guest_memfd_rdm_register_listener (RamDiscardManager *rdm, + RamDiscardListener *rdl, + MemoryRegionSection*section)
include/sysemu/guest-memfd-manager.h new file +11
-0+{ + GuestMemfdManager *gmm = GUEST_MEMFD_MANAGER(rdm); + int ret; + + g_assert(section->mr == gmm->mr); + rdl->section = memory_region_section_new_copy(section); + + QLIST_INSERT_HEAD(&gmm->rdl_list, rdl, next); + + ret = guest_memfd_for_each_populated_section(gmm, section, rdl, +
guest_memfd_notify_populate_cb);
+ if (ret) {
+ error_report("%s: Failed to register RAM discard listener:
%s",__func__,
+ strerror(-ret));
+ }include/sysemu/guest-memfd-manager.h new file +3
-0+} + +static void guest_memfd_rdm_unregister_listener(RamDiscardManager*rdm,
+ RamDiscardListener *rdl)include/sysemu/guest-memfd-manager.h new file +8
-0+{ + GuestMemfdManager *gmm = GUEST_MEMFD_MANAGER(rdm); + int ret; + + g_assert(rdl->section); + g_assert(rdl->section->mr == gmm->mr); + + ret = guest_memfd_for_each_populated_section(gmm, rdl->section,rdl,
+guest_memfd_notify_discard_cb);
+ if (ret) {
+ error_report("%s: Failed to unregister RAM discard listener:
%s",__func__,
+ strerror(-ret));
+ }include/sysemu/guest-memfd-manager.h
Quoted 9 lines
include/sysemu/guest-memfd-manager.h new file +9
-0+ + memory_region_section_free_copy(rdl->section); + rdl->section = NULL; + QLIST_REMOVE(rdl, next); + +} + +typedef struct GuestMemfdReplayData { + void *fn;s/void */ReplayRamPopulate/
[...]
Quoted 10 lines
Quoted 8 lines
+ void *opaque;
include/sysemu/guest-memfd-manager.h new file +3
-0+} GuestMemfdReplayData; + +static int guest_memfd_rdm_replay_populated_cb(MemoryRegionSection*section, void *arg)
+{
+ struct GuestMemfdReplayData *data = arg;Drop "struct" here and below.
Fixed. Thanks!
Quoted 34 lines
Quoted 26 lines
+ ReplayRamPopulate
replay_fn=data->fn;include/sysemu/guest-memfd-manager.h new file +5
-0+ + return replay_fn(section, data->opaque); +} + +static int guest_memfd_rdm_replay_populated(const RamDiscardManager*rdm,
+ MemoryRegionSection *section,
+ ReplayRamPopulatereplay_fn,
+ void *opaque) +{
+ GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER(rdm);
+ struct GuestMemfdReplayData data = { .fn =replay_fn, .opaque = opaque };include/sysemu/guest-memfd-manager.h new file +4
-0+ + g_assert(section->mr == gmm->mr); + return guest_memfd_for_each_populated_section(gmm, section, &data, +
guest_memfd_rdm_replay_populated_cb);include/sysemu/guest-memfd-manager.h new file +3
-0+} + +static int guest_memfd_rdm_replay_discarded_cb(MemoryRegionSection*section, void *arg)
include/sysemu/guest-memfd-manager.h
Quoted 5 lines
include/sysemu/guest-memfd-manager.h new file +5
-0+{ + struct GuestMemfdReplayData *data = arg; + ReplayRamDiscard replay_fn = data->fn; + + replay_fn(section, data->opaque);
guest_memfd_rdm_replay_populated_cb()checks for errors though.
Quoted 5 lines
Quoted 3 lines
It follows current definiton of
ReplayRamDiscard()andReplayRamPopulate()wherereplay_discard()doesn't return errors andreplay_populate()returns errors.A trace would be appropriate imho. Thanks,
Sorry, can't catch you. What kind of info to be traced? The errors returned by replay_populate()?
Quoted 71 lines
Quoted 70 lines
Quoted 29 lines
Quoted 29 lines
include/sysemu/guest-memfd-manager.h new file +5
-0+ + return 0; +} + +static void guest_memfd_rdm_replay_discarded(const RamDiscardManager*rdm,
+ MemoryRegionSection *section,
+ ReplayRamDiscardreplay_fn,
+ void *opaque) +{
+ GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER(rdm);
+ struct GuestMemfdReplayData data = { .fn =replay_fn, .opaque = opaque };include/sysemu/guest-memfd-manager.h new file +4
-0+ + g_assert(section->mr == gmm->mr); + guest_memfd_for_each_discarded_section(gmm, section, &data, +
guest_memfd_rdm_replay_discarded_cb);include/sysemu/guest-memfd-manager.h new file +7
-0+} + +static void guest_memfd_manager_init (Object *obj) +{ + GuestMemfdManager *gmm = GUEST_MEMFD_MANAGER(obj); + + QLIST_INIT(&gmm->rdl_list);Quoted 8 lines
include/sysemu/guest-memfd-manager.h
Quoted 5 lines
include/sysemu/guest-memfd-manager.h new file +5
-0+} + +static void guest_memfd_manager_finalize (Object *obj) +{ + g_free(GUEST_MEMFD_MANAGER(obj)->bitmap);bitmap is not allocated though. And 5/7 removes this anyway. Thanks,
Will remove it. Thanks.
Quoted 29 lines
Quoted 28 lines
include/sysemu/guest-memfd-manager.h new file +3
-0+} + +static void guest_memfd_manager_class_init(ObjectClass *oc, void*data)
include/sysemu/guest-memfd-manager.h new file +10
-0+{ + RamDiscardManagerClass *rdmc = RAM_DISCARD_MANAGER_CLASS(oc); + + rdmc->get_min_granularity = guest_memfd_rdm_get_min_granularity; + rdmc->register_listener = guest_memfd_rdm_register_listener; + rdmc->unregister_listener = guest_memfd_rdm_unregister_listener; + rdmc->is_populated = guest_memfd_rdm_is_populated; + rdmc->replay_populated = guest_memfd_rdm_replay_populated; + rdmc->replay_discarded = guest_memfd_rdm_replay_discarded; +}system/meson.build +1
-0@@ -15,6 +15,7 @@ system_ss.add(files( 'dirtylimit.c', 'dma-helpers.c', 'globals.c', + 'guest-memfd-manager.c', 'memory_mapping.c', 'qdev-monitor.c', 'qtest.c',
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 53 lines · Chenyi Qiang
Quoted 41 lines · Alexey Kardashevskiy
Quoted 38 lines · Chenyi Qiang
Quoted 34 lines · Alexey Kardashevskiy
Quoted 18 lines · Chenyi Qiang
As the commit
852f0048f3("RAMBlock: makeguest_memfdrequire uncoordinated discard") highlighted, some subsystems like VFIO might disable ram block discard. However,guest_memfdrelies on the discard operation to perform page conversion between private and shared memory. This can lead to stale IOMMU mapping issue when assigning a hardware device to a confidential VM via shared memory (unprotected memory pages). Blocking shared page discard can solve this problem, but it could cause guests to consume twice the memory with VFIO, which is not acceptable in some cases. An alternative solution is to convey other systems like VFIO to refresh its outdated IOMMU mappings.RamDiscardManager is an existing concept (used by virtio-mem) to adjust VFIO mappings in relation to VM page assignment. Effectively page conversion is similar to hot-removing a page in one mode and adding it back in the other, so the similar work that needs to happen in response to virtio-mem changes needs to happen for page conversion events. Introduce the RamDiscardManager to
guest_memfdto achieve it.Quoted 4 lines
However,
guest_memfdis not an object so it cannot directly implement the RamDiscardManager interface.One solution is to implement the interface in HostMemoryBackend. Any
This sounds about right.
Quoted 5 lines
guest_memfd-backed host memory backend can register itself in the target
MemoryRegion. However, this solution doesn't cover the scenario where aguest_memfdMemoryRegion doesn't belong to the HostMemoryBackend, e.g. the virtual BIOS MemoryRegion.What is this virtual BIOS MemoryRegion exactly? What does it look like in "info mtree -f"? Do we really want this memory to be DMAable?
Quoted 8 lines
Quoted 6 lines
virtual BIOS shows in a separate region:
Root memory region: system 0000000000000000-000000007fffffff (prio 0, ram): pc.ram KVM
...
00000000ffc00000-00000000ffffffff (prio 0, ram): pc.bios KVMLooks like a normal MR which can be backed by
guest_memfd.
Quoted 7 lines
Yes, virtual BIOS memory region is initialized by
memory_region_init_ram_guest_memfd()which will be backed by aguest_memfd.The tricky thing is, for Intel TDX (not sure about AMD SEV), the virtual BIOS image will be loaded and then copied to private region. After that,
the loaded image will be discarded and this region become useless.
I'd think it is loaded as "struct Rom" and then copied to the MR-ram_guest_memfd() which does not leave MR useless - we still see "pc.bios" in the list so it is not discarded. What piece of code are you referring to exactly?
Quoted 2 lines
So I
feel like this virtual BIOS should not be backed byguest_memfd?
From the above it sounds like the opposite, i.e. it should :)
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
0000000100000000-000000017fffffff (prio 0, ram): pc.ram
@0000000080000000 KVMAnyway if there is no
guest_memfdbacking it andmemory_region_has_ram_discard_manager()returns false, then the MR is just going to be mapped for VFIO as usual which seems... alright, right?
Quoted 6 lines
Correct. As the vBIOS is backed by
guest_memfdand we implement the RDM forguest_memfd_manager, the vBIOS MR won't be mapped by VFIO.If we go with the HostMemoryBackend instead of
guest_memfd_manager, this MR would be mapped by VFIO. Maybe need to avoid such vBIOS mapping, or just ignore it since the MR is useless (but looks not so good).
Sorry I am missing necessary details here, let's figure out the above.
Quoted 364 lines
Quoted 8 lines
Quoted 8 lines
We also consider to implement the interface in HostMemoryBackend, but maybe implement with
guest_memfdregion is more general. We don't know if any DMAable memory would belong to HostMemoryBackend although at present it is.Quoted 4 lines
Quoted 2 lines
If it is more appropriate to implement it with HostMemoryBackend, I can change to this way.
Seems cleaner imho.
I can go this way.
Quoted 111 lines
Quoted 111 lines
Quoted 71 lines
Quoted 64 lines
Thus, choose the second option, i.e. define an object type named
guest_memfd_managerwith RamDiscardManager interface. Upon creation ofguest_memfd, a newguest_memfd_managerobject can be instantiated and registered to the managedguest_memfdMemoryRegion to handle the page conversion events.In the context of
guest_memfd, the discarded state signifies that the page is private, while the populated state indicated that the page is shared. The state of the memory is tracked at the granularity of the host page size (i.e.block_size), as the minimum conversion size can be one page per request.In addition, VFIO expects the DMA mapping for a specific iova to be mapped and unmapped with the same granularity. However, the confidential
VMs may do partial conversion, e.g. conversion happens on a small region
within a large region. To prevent such invalid cases and before any potential optimization comes out, all operations are performed with 4K granularity.Signed-off-by Chenyi Qiang <[email protected]>--- include/sysemu/guest-memfd-manager.h | 46 +++++ system/guest-memfd-manager.c | 250 ++++++++++++++++++++++ +++++ system/meson.build | 1 + 3 files changed, 297 insertions(+)create mode 100644
include/sysemu/guest-memfd-manager.hcreate mode 100644system/guest-memfd-manager.cinclude/sysemu/ renamed from include/sysemu/guest-memfd-manager.h +0
-0
guest-memfd-manager.h
new file mode 100644
index 0000000000..ba4a99b614include/sysemu/guest-memfd-manager.h new file +9
-0@@ -0,0 +1,46 @@ +/* +* QEMU guest memfd manager +* +* Copyright Intel +* +* Author: +* Chenyi Qiang <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory
+ *
+ */include/sysemu/guest-memfd-manager.h new file +9
-0+ +#ifndef SYSEMU_GUEST_MEMFD_MANAGER_H +#define SYSEMU_GUEST_MEMFD_MANAGER_H + +#include "sysemu/hostmem.h" + +#define TYPE_GUEST_MEMFD_MANAGER "guest-memfd-manager" + +OBJECT_DECLARE_TYPE(GuestMemfdManager, GuestMemfdManagerClass,
GUEST_MEMFD_MANAGER)include/sysemu/guest-memfd-manager.h line 26
Quoted 5 lines
include/sysemu/guest-memfd-manager.h new file +5
-0+ +struct GuestMemfdManager { + Object parent; + + /* Managed memory region. */Do not need this comment. And the period.
[...]
Quoted 7 lines
Quoted 7 lines
+ MemoryRegion *mr;
+
+ /*
+ * 1-setting of the bit represents the memory is populated (shared).
+ */Will fix it.
Quoted 12 lines
Could be 1 line comment.
Quoted 1 line
+
int32_tbitmap_size;int or unsigned
Quoted 4 lines
+ unsigned long *bitmap;
+
+ /* block size and alignment */
+uint64_tblock_size;Quoted 4 lines
unsigned?
(u)int(32|64)_t make sense for migrations which is not the case (yet?). Thanks,
I think these fields would be helpful for future migration support. Maybe defining as this way is more straightforward.
Quoted 6 lines
Quoted 3 lines
+
+ /* listeners to notify on populate/discard activity. */Do not really need this comment either imho.
Quoted 6 lines
Quoted 2 lines
I prefer to provide the comment for each field as virtio-mem do. If it is not necessary, I would remove those obvious ones.
[bikeshedding on] But the "RamDiscardListener" word says that already, why repeating? :) It should add information, not duplicate. Like the
block_sizecomment which mentions "alignment" [bikeshedding off]Got it. Thanks!
Quoted 229 lines
Quoted 36 lines
Quoted 9 lines
include/sysemu/guest-memfd-manager.h line 46
Quoted 9 lines
+
QLIST_HEAD(, RamDiscardListener)rdl_list;include/sysemu/guest-memfd-manager.h new file +7
-0+}; + +struct GuestMemfdManagerClass { + ObjectClass parent_class; +}; + +#endif [...]
void *arg,
Quoted 23 lines
Quoted 17 lines
+
guest_memfd_section_cbcb)include/sysemu/guest-memfd-manager.h new file +7
-0+{ + unsigned long first_one_bit, last_one_bit; + uint64_t offset, size; + int ret = 0; + + first_one_bit = section->offset_within_region / gmm->block_size; + first_one_bit = find_next_bit(gmm->bitmap, gmm->bitmap_size,
first_one_bit);include/sysemu/guest-memfd-manager.h new file +6
-0+ + while (first_one_bit < gmm->bitmap_size) { + MemoryRegionSection tmp = *section; + + offset = first_one_bit * gmm->block_size; + last_one_bit = find_next_zero_bit(gmm->bitmap, gmm-Quoted 1 line
bitmap_size,Quoted 2 lines
+
first_one_bit+ 1) - 1; + size = (last_one_bit-first_one_bit+ 1) *gmm->block_size;This tries calling
cb()on bigger chunks even though we say from the beginning that only page size is supported?Quoted 8 lines
Quoted 3 lines
May be simplify this for now and extend if/when VFIO learns to split mappings, or just drop it when we get in-place page state convertion (which will make this all irrelevant)?
The
cb()will call with big chunks but actually it do the split with the granularity ofblock_sizein thecb(). See thevfio_ram_discard_notify_populate(), which do theDMA_MAPwith granularity size.Right, and this all happens inside QEMU - first the code finds bigger chunks and then it splits them anyway to call the VFIO driver. Seems pointless to bother about bigger chunks here.
Quoted 179 lines
Quoted 128 lines
Quoted 117 lines
+
+ if (!memory_region_section_intersect_range(&tmp, offset, size)) {
+ break;
+ }include/sysemu/guest-memfd-manager.h new file +14
-0+ + ret = cb(&tmp, arg); + if (ret) { + break; + } + + first_one_bit = find_next_bit(gmm->bitmap, gmm->bitmap_size, + last_one_bit + 2); + } + + return ret; +} + +static int guest_memfd_for_each_discarded_section(constGuestMemfdManager *gmm,
+ MemoryRegionSection *section,
+ void *arg, +guest_memfd_section_cbcb)include/sysemu/guest-memfd-manager.h new file +18
-0+{ + unsigned long first_zero_bit, last_zero_bit; + uint64_t offset, size; + int ret = 0; + + first_zero_bit = section->offset_within_region / gmm->block_size; + first_zero_bit = find_next_zero_bit(gmm->bitmap, gmm->bitmap_size, + first_zero_bit); + + while (first_zero_bit < gmm->bitmap_size) { + MemoryRegionSection tmp = *section; + + offset = first_zero_bit * gmm->block_size; + last_zero_bit = find_next_bit(gmm->bitmap, gmm->bitmap_size, + first_zero_bit + 1) - 1; + size = (last_zero_bit - first_zero_bit + 1) * gmm->block_size; + + if (!memory_region_section_intersect_range(&tmp, offset,size)) {
+ break;
+ }include/sysemu/guest-memfd-manager.h new file +7
-0+ + ret = cb(&tmp, arg); + if (ret) { + break; + } + + first_zero_bit = find_next_zero_bit(gmm->bitmap, gmm-Quoted 1 line
bitmap_size,+
last_zero_bit+ 2); + }include/sysemu/guest-memfd-manager.h new file +5
-0+ + return ret; +} + +static uint64_t guest_memfd_rdm_get_min_granularity(constRamDiscardManager *rdm,
+ const MemoryRegion *mr)include/sysemu/guest-memfd-manager.h new file +10
-0+{ + GuestMemfdManager *gmm = GUEST_MEMFD_MANAGER(rdm); + + g_assert(mr == gmm->mr); + return gmm->block_size; +} + +static void guest_memfd_rdm_register_listener (RamDiscardManager *rdm, + RamDiscardListener *rdl, + MemoryRegionSection*section)
include/sysemu/guest-memfd-manager.h new file +11
-0+{ + GuestMemfdManager *gmm = GUEST_MEMFD_MANAGER(rdm); + int ret; + + g_assert(section->mr == gmm->mr); + rdl->section = memory_region_section_new_copy(section); + + QLIST_INSERT_HEAD(&gmm->rdl_list, rdl, next); + + ret = guest_memfd_for_each_populated_section(gmm, section, rdl, +
guest_memfd_notify_populate_cb);
+ if (ret) {
+ error_report("%s: Failed to register RAM discard listener:
%s",__func__,
+ strerror(-ret));
+ }include/sysemu/guest-memfd-manager.h new file +3
-0+} + +static void guest_memfd_rdm_unregister_listener(RamDiscardManager*rdm,
+ RamDiscardListener *rdl)include/sysemu/guest-memfd-manager.h new file +8
-0+{ + GuestMemfdManager *gmm = GUEST_MEMFD_MANAGER(rdm); + int ret; + + g_assert(rdl->section); + g_assert(rdl->section->mr == gmm->mr); + + ret = guest_memfd_for_each_populated_section(gmm, rdl->section,rdl,
+guest_memfd_notify_discard_cb);
+ if (ret) {
+ error_report("%s: Failed to unregister RAM discard listener:
%s",__func__,
+ strerror(-ret));
+ }include/sysemu/guest-memfd-manager.h
Quoted 9 lines
include/sysemu/guest-memfd-manager.h new file +9
-0+ + memory_region_section_free_copy(rdl->section); + rdl->section = NULL; + QLIST_REMOVE(rdl, next); + +} + +typedef struct GuestMemfdReplayData { + void *fn;s/void */ReplayRamPopulate/
[...]
Quoted 10 lines
Quoted 8 lines
+ void *opaque;
include/sysemu/guest-memfd-manager.h new file +3
-0+} GuestMemfdReplayData; + +static int guest_memfd_rdm_replay_populated_cb(MemoryRegionSection*section, void *arg)
+{
+ struct GuestMemfdReplayData *data = arg;Drop "struct" here and below.
Fixed. Thanks!
Quoted 34 lines
Quoted 26 lines
+ ReplayRamPopulate
replay_fn=data->fn;include/sysemu/guest-memfd-manager.h new file +5
-0+ + return replay_fn(section, data->opaque); +} + +static int guest_memfd_rdm_replay_populated(const RamDiscardManager*rdm,
+ MemoryRegionSection *section,
+ ReplayRamPopulatereplay_fn,
+ void *opaque) +{
+ GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER(rdm);
+ struct GuestMemfdReplayData data = { .fn =replay_fn, .opaque = opaque };include/sysemu/guest-memfd-manager.h new file +4
-0+ + g_assert(section->mr == gmm->mr); + return guest_memfd_for_each_populated_section(gmm, section, &data, +
guest_memfd_rdm_replay_populated_cb);include/sysemu/guest-memfd-manager.h new file +3
-0+} + +static int guest_memfd_rdm_replay_discarded_cb(MemoryRegionSection*section, void *arg)
include/sysemu/guest-memfd-manager.h
Quoted 5 lines
include/sysemu/guest-memfd-manager.h new file +5
-0+{ + struct GuestMemfdReplayData *data = arg; + ReplayRamDiscard replay_fn = data->fn; + + replay_fn(section, data->opaque);
guest_memfd_rdm_replay_populated_cb()checks for errors though.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
It follows current definiton of
ReplayRamDiscard()andReplayRamPopulate()wherereplay_discard()doesn't return errors andreplay_populate()returns errors.A trace would be appropriate imho. Thanks,
Sorry, can't catch you. What kind of info to be traced? The errors returned by
replay_populate()?
Yeah. imho these are useful as we expect this part to work in general too, right? Thanks,
Quoted 73 lines
Quoted 72 lines
Quoted 71 lines
Quoted 30 lines
Quoted 30 lines
include/sysemu/guest-memfd-manager.h new file +5
-0+ + return 0; +} + +static void guest_memfd_rdm_replay_discarded(const RamDiscardManager*rdm,
+ MemoryRegionSection *section,
+ ReplayRamDiscardreplay_fn,
+ void *opaque) +{
+ GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER(rdm);
+ struct GuestMemfdReplayData data = { .fn =replay_fn, .opaque = opaque };include/sysemu/guest-memfd-manager.h new file +4
-0+ + g_assert(section->mr == gmm->mr); + guest_memfd_for_each_discarded_section(gmm, section, &data, +
guest_memfd_rdm_replay_discarded_cb);include/sysemu/guest-memfd-manager.h new file +7
-0+} + +static void guest_memfd_manager_init (Object *obj) +{ + GuestMemfdManager *gmm = GUEST_MEMFD_MANAGER(obj); + + QLIST_INIT(&gmm->rdl_list);Quoted 8 lines
include/sysemu/guest-memfd-manager.h
Quoted 5 lines
include/sysemu/guest-memfd-manager.h new file +5
-0+} + +static void guest_memfd_manager_finalize (Object *obj) +{ + g_free(GUEST_MEMFD_MANAGER(obj)->bitmap);bitmap is not allocated though. And 5/7 removes this anyway. Thanks,
Will remove it. Thanks.
Quoted 29 lines
Quoted 28 lines
include/sysemu/guest-memfd-manager.h new file +3
-0+} + +static void guest_memfd_manager_class_init(ObjectClass *oc, void*data)
include/sysemu/guest-memfd-manager.h new file +10
-0+{ + RamDiscardManagerClass *rdmc = RAM_DISCARD_MANAGER_CLASS(oc); + + rdmc->get_min_granularity = guest_memfd_rdm_get_min_granularity; + rdmc->register_listener = guest_memfd_rdm_register_listener; + rdmc->unregister_listener = guest_memfd_rdm_unregister_listener; + rdmc->is_populated = guest_memfd_rdm_is_populated; + rdmc->replay_populated = guest_memfd_rdm_replay_populated; + rdmc->replay_discarded = guest_memfd_rdm_replay_discarded; +}system/meson.build +1
-0@@ -15,6 +15,7 @@ system_ss.add(files( 'dirtylimit.c', 'dma-helpers.c', 'globals.c', + 'guest-memfd-manager.c', 'memory_mapping.c', 'qdev-monitor.c', 'qtest.c',
Signature · 1 line
Alexey
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 72 lines · Alexey Kardashevskiy
Quoted 60 lines · Chenyi Qiang
Quoted 48 lines · Alexey Kardashevskiy
Quoted 45 lines · Chenyi Qiang
Quoted 41 lines · Alexey Kardashevskiy
Quoted 23 lines · Chenyi Qiang
As the commit
852f0048f3("RAMBlock: makeguest_memfdrequire uncoordinated discard") highlighted, some subsystems like VFIO might disable ram block discard. However,guest_memfdrelies on the discard operation to perform page conversion between private and shared memory.
This can lead to stale IOMMU mapping issue when assigning a hardware device to a confidential VM via shared memory (unprotected memory pages). Blocking shared page discard can solve this problem, but it could cause guests to consume twice the memory with VFIO, which is not
acceptable in some cases. An alternative solution is to convey other systems like VFIO to refresh its outdated IOMMU mappings.RamDiscardManager is an existing concept (used by virtio-mem) to adjust
VFIO mappings in relation to VM page assignment. Effectively page conversion is similar to hot-removing a page in one mode and adding it
back in the other, so the similar work that needs to happen in response
to virtio-mem changes needs to happen for page conversion events. Introduce the RamDiscardManager toguest_memfdto achieve it.Quoted 4 lines
However,
guest_memfdis not an object so it cannot directly implement the RamDiscardManager interface.One solution is to implement the interface in HostMemoryBackend. Any
This sounds about right.
Quoted 7 lines
guest_memfd-backed host memory backend can register itself in the target
MemoryRegion. However, this solution doesn't cover the scenario where aguest_memfdMemoryRegion doesn't belong to the HostMemoryBackend, e.g.
the virtual BIOS MemoryRegion.What is this virtual BIOS MemoryRegion exactly? What does it look like in "info mtree -f"? Do we really want this memory to be DMAable?
Quoted 8 lines
Quoted 6 lines
virtual BIOS shows in a separate region:
Root memory region: system 0000000000000000-000000007fffffff (prio 0, ram): pc.ram KVM
...
00000000ffc00000-00000000ffffffff (prio 0, ram): pc.bios KVMLooks like a normal MR which can be backed by
guest_memfd.Quoted 8 lines
Yes, virtual BIOS memory region is initialized by
memory_region_init_ram_guest_memfd()which will be backed by aguest_memfd.The tricky thing is, for Intel TDX (not sure about AMD SEV), the virtual BIOS image will be loaded and then copied to private region. After that,
the loaded image will be discarded and this region become useless.
Quoted 4 lines
I'd think it is loaded as "struct Rom" and then copied to the MR-
ram_guest_memfd()which does not leave MR useless - we still see "pc.bios" in the list so it is not discarded. What piece of code are you referring to exactly?
Sorry for confusion, maybe it is different between TDX and SEV-SNP for the vBIOS handling.
In x86_bios_rom_init(), it initializes a guest_memfd-backed MR and loads the vBIOS image to the shared part of the guest_memfd MR. For TDX, it will copy the image to private region (not the vBIOS guest_memfd MR private part) and discard the shared part. So, although the memory region still exists, it seems useless.
It is different for SEV-SNP, correct? Does SEV-SNP manage the vBIOS in vBIOS guest_memfd private memory?
Quoted 38 lines
Quoted 4 lines
So I
feel like this virtual BIOS should not be backed byguest_memfd?From the above it sounds like the opposite, i.e. it should :)
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
0000000100000000-000000017fffffff (prio 0, ram): pc.ram @0000000080000000 KVM
Anyway if there is no
guest_memfdbacking it andmemory_region_has_ram_discard_manager()returns false, then the MR is just going to be mapped for VFIO as usual which seems... alright, right?Quoted 6 lines
Correct. As the vBIOS is backed by
guest_memfdand we implement the RDM forguest_memfd_manager, the vBIOS MR won't be mapped by VFIO.If we go with the HostMemoryBackend instead of
guest_memfd_manager, this MR would be mapped by VFIO. Maybe need to avoid such vBIOS mapping, or just ignore it since the MR is useless (but looks not so good).Sorry I am missing necessary details here, let's figure out the above.
Quoted 14 lines
Quoted 8 lines
Quoted 8 lines
We also consider to implement the interface in HostMemoryBackend, but maybe implement with
guest_memfdregion is more general. We don't know if any DMAable memory would belong to HostMemoryBackend although at present it is.Quoted 4 lines
Quoted 2 lines
If it is more appropriate to implement it with HostMemoryBackend, I can change to this way.
Seems cleaner imho.
I can go this way.
[...]
Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
Quoted 34 lines
Quoted 22 lines
Quoted 22 lines
(quoted diff) +2
-0+ +static int guest_memfd_rdm_replay_populated(const RamDiscardManager*rdm,
+ MemoryRegionSection *section,
+ ReplayRamPopulatereplay_fn,
+ void *opaque) +{
+ GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER(rdm);
+ struct GuestMemfdReplayData data = { .fn =replay_fn, .opaque = opaque };
+
+g_assert(section->mr == gmm->mr);
+ return guest_memfd_for_each_populated_section(gmm, section, &data,
+guest_memfd_rdm_replay_populated_cb);(quoted diff) +3
-0+} + +static int guest_memfd_rdm_replay_discarded_cb(MemoryRegionSection*section, void *arg)
Quoted 8 lines
Review
Quoted 5 lines
(quoted diff) +5
-0+{ + struct GuestMemfdReplayData *data = arg; + ReplayRamDiscard replay_fn = data->fn; + + replay_fn(section, data->opaque);
guest_memfd_rdm_replay_populated_cb()checks for errors though.It follows current definiton of
ReplayRamDiscard()andReplayRamPopulate()wherereplay_discard()doesn't return errors andreplay_populate()returns errors.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
A trace would be appropriate imho. Thanks,
Sorry, can't catch you. What kind of info to be traced? The errors returned by
replay_populate()?Yeah. imho these are useful as we expect this part to work in general too, right? Thanks,
Something like?
system/guest-memfd-manager.c +0 -0
@@ -185,8 +185,14 @@ static int
guest_memfd_rdm_replay_populated_cb(MemoryRegionSection *section, voi
{
struct GuestMemfdReplayData *data = arg;
ReplayRamPopulate replay_fn = data->fn;+ int ret;
- return replay_fn(section, data->opaque);
+ ret = replay_fn(section, data->opaque);
+ if (ret) {+ trace_guest_memfd_rdm_replay_populated_cb(ret);
+ }
+
+ return ret;
}
How about just adding some error output in guest_memfd_for_each_populated_section()/guest_memfd_for_each_discarded_section() if the cb() (i.e. replay_populate()) returns error?
Quoted 75 lines
Quoted 74 lines
Quoted 73 lines
Quoted 72 lines
Quoted 31 lines
Quoted 31 lines
(quoted diff) +5
-0+ + return 0; +} + +static void guest_memfd_rdm_replay_discarded(const RamDiscardManager*rdm,
+ MemoryRegionSection *section,
+ ReplayRamDiscardreplay_fn,
+ void *opaque) +{
+ GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER(rdm);
+ struct GuestMemfdReplayData data = { .fn =replay_fn, .opaque = opaque };(quoted diff) +4
-0+ + g_assert(section->mr == gmm->mr); + guest_memfd_for_each_discarded_section(gmm, section, &data, +
guest_memfd_rdm_replay_discarded_cb);(quoted diff) +7
-0+} + +static void guest_memfd_manager_init(Object *obj) +{ + GuestMemfdManager *gmm = GUEST_MEMFD_MANAGER(obj); + + QLIST_INIT(&gmm->rdl_list);Quoted 8 lines
Review
Quoted 5 lines
(quoted diff) +5
-0+} + +static void guest_memfd_manager_finalize(Object *obj) +{ + g_free(GUEST_MEMFD_MANAGER(obj)->bitmap);bitmap is not allocated though. And 5/7 removes this anyway. Thanks,
Will remove it. Thanks.
Quoted 29 lines
Quoted 28 lines
(quoted diff) +3
-0+} + +static void guest_memfd_manager_class_init(ObjectClass *oc, void*data)
(quoted diff) +10
-0+{ + RamDiscardManagerClass *rdmc = RAM_DISCARD_MANAGER_CLASS(oc); + + rdmc->get_min_granularity = guest_memfd_rdm_get_min_granularity; + rdmc->register_listener = guest_memfd_rdm_register_listener; + rdmc->unregister_listener = guest_memfd_rdm_unregister_listener; + rdmc->is_populated = guest_memfd_rdm_is_populated; + rdmc->replay_populated = guest_memfd_rdm_replay_populated; + rdmc->replay_discarded = guest_memfd_rdm_replay_discarded; +}system/meson.build +1
-0@@ -15,6 +15,7 @@ system_ss.add(files( 'dirtylimit.c', 'dma-helpers.c', 'globals.c', + 'guest-memfd-manager.c', 'memory_mapping.c', 'qdev-monitor.c', 'qtest.c',
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 39 lines · Chenyi Qiang
Quoted 36 lines · Alexey Kardashevskiy
Quoted 33 lines · Chenyi Qiang
Quoted 30 lines · Alexey Kardashevskiy
Quoted 27 lines · Chenyi Qiang
Quoted 24 lines · Alexey Kardashevskiy
Quoted 23 lines · Chenyi Qiang
As the commit
852f0048f3("RAMBlock: makeguest_memfdrequire uncoordinated discard") highlighted, some subsystems like VFIO might disable ram block discard. However,guest_memfdrelies on the discard operation to perform page conversion between private and shared memory.
This can lead to stale IOMMU mapping issue when assigning a hardware device to a confidential VM via shared memory (unprotected memory pages). Blocking shared page discard can solve this problem, but it could cause guests to consume twice the memory with VFIO, which is not
acceptable in some cases. An alternative solution is to convey other systems like VFIO to refresh its outdated IOMMU mappings.RamDiscardManager is an existing concept (used by virtio-mem) to adjust
VFIO mappings in relation to VM page assignment. Effectively page conversion is similar to hot-removing a page in one mode and adding it
back in the other, so the similar work that needs to happen in response
to virtio-mem changes needs to happen for page conversion events. Introduce the RamDiscardManager toguest_memfdto achieve it.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
However,
guest_memfdis not an object so it cannot directly implement the RamDiscardManager interface.One solution is to implement the interface in HostMemoryBackend. Any
This sounds about right.
btw I am using this for ages:
https://github.com/aik/qemu/commit/3663f889883d4aebbeb0e4422f7be5e357e2ee46
but I am not sure if this ever saw the light of the day, did not it? (ironically I am using it as a base for encrypted DMA :) )
Quoted 38 lines
Quoted 30 lines
Quoted 21 lines
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 8 lines
guest_memfd-backed host memory backend can register itself in the target
MemoryRegion. However, this solution doesn't cover the scenario where aguest_memfdMemoryRegion doesn't belong to the HostMemoryBackend, e.g.
the virtual BIOS MemoryRegion.What is this virtual BIOS MemoryRegion exactly? What does it look like in "info mtree -f"? Do we really want this memory to be DMAable?
Quoted 8 lines
Quoted 6 lines
virtual BIOS shows in a separate region:
Root memory region: system 0000000000000000-000000007fffffff (prio 0, ram): pc.ram KVM
...
00000000ffc00000-00000000ffffffff (prio 0, ram): pc.bios KVMLooks like a normal MR which can be backed by
guest_memfd.Quoted 8 lines
Yes, virtual BIOS memory region is initialized by
memory_region_init_ram_guest_memfd()which will be backed by aguest_memfd.The tricky thing is, for Intel TDX (not sure about AMD SEV), the virtual BIOS image will be loaded and then copied to private region. After that,
the loaded image will be discarded and this region become useless.Quoted 4 lines
I'd think it is loaded as "struct Rom" and then copied to the MR-
ram_guest_memfd()which does not leave MR useless - we still see "pc.bios" in the list so it is not discarded. What piece of code are you referring to exactly?Sorry for confusion, maybe it is different between TDX and SEV-SNP for the vBIOS handling.
Quoted 8 lines
In
x86_bios_rom_init(), it initializes a guest_memfd-backed MR and loads the vBIOS image to the shared part of theguest_memfdMR. For TDX, it
will copy the image to private region (not the vBIOSguest_memfdMR private part) and discard the shared part. So, although the memory region still exists, it seems useless.
It is different for SEV-SNP, correct? Does SEV-SNP manage the vBIOS in vBIOSguest_memfdprivate memory?
This is what it looks like on my SNP VM (which, I suspect, is the same as yours as hw/i386/pc.c does not distinguish Intel/AMD for this matter):
Root memory region: system
0000000000000000-00000000000bffff (prio 0, ram): ram1 KVM gmemfd=20
00000000000c0000-00000000000dffff (prio 1, ram): pc.rom KVM gmemfd=27
00000000000e0000-000000001fffffff (prio 0, ram): ram1
@00000000000e0000 KVM gmemfd=20
...
00000000ffc00000-00000000ffffffff (prio 0, ram): pc.bios KVM gmemfd=26
So the pc.bios MR exists and in use (hence its appearance in "info mtree -f").
I added the gmemfd dumping:
system/memory.c +3 -0
@@ -3446,6 +3446,9 @@ static void mtree_print_flatview(gpointer key, gpointer value, } } } +if (mr->ram_block && mr->ram_block->guest_memfd >=0 ) { +qemu_printf (" gmemfd=%d" , mr->ram_block->guest_memfd); + }
Quoted 106 lines
Quoted 38 lines
Quoted 4 lines
So I
feel like this virtual BIOS should not be backed byguest_memfd?From the above it sounds like the opposite, i.e. it should :)
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
0000000100000000-000000017fffffff (prio 0, ram): pc.ram
@0000000080000000 KVMAnyway if there is no
guest_memfdbacking it andmemory_region_has_ram_discard_manager()returns false, then the MR is just going to be mapped for VFIO as usual which seems... alright, right?Quoted 6 lines
Correct. As the vBIOS is backed by
guest_memfdand we implement the RDM forguest_memfd_manager, the vBIOS MR won't be mapped by VFIO.If we go with the HostMemoryBackend instead of
guest_memfd_manager, this MR would be mapped by VFIO. Maybe need to avoid such vBIOS mapping, or just ignore it since the MR is useless (but looks not so good).Sorry I am missing necessary details here, let's figure out the above.
Quoted 14 lines
Quoted 8 lines
Quoted 8 lines
We also consider to implement the interface in HostMemoryBackend, but maybe implement with
guest_memfdregion is more general. We don't know if any DMAable memory would belong to HostMemoryBackend although at present it is.Quoted 4 lines
Quoted 2 lines
If it is more appropriate to implement it with HostMemoryBackend, I can change to this way.
Seems cleaner imho.
I can go this way.
[...]
Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
Quoted 34 lines
Quoted 22 lines
Quoted 22 lines
(quoted diff) +2
-0+ +static int guest_memfd_rdm_replay_populated(const RamDiscardManager*rdm,
+ MemoryRegionSection *section,
+ ReplayRamPopulatereplay_fn,
+ void *opaque) +{
+ GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER(rdm);
+ struct GuestMemfdReplayData data = { .fn =replay_fn, .opaque = opaque };
+
+g_assert(section->mr == gmm->mr);
+ return guest_memfd_for_each_populated_section(gmm, section, &data,
+guest_memfd_rdm_replay_populated_cb);(quoted diff) +3
-0+} + +static int guest_memfd_rdm_replay_discarded_cb(MemoryRegionSection*section, void *arg)
Quoted 8 lines
Review
Quoted 5 lines
(quoted diff) +5
-0+{ + struct GuestMemfdReplayData *data = arg; + ReplayRamDiscard replay_fn = data->fn; + + replay_fn(section, data->opaque);
guest_memfd_rdm_replay_populated_cb()checks for errors though.It follows current definiton of
ReplayRamDiscard()andReplayRamPopulate()wherereplay_discard()doesn't return errors andreplay_populate()returns errors.Quoted 7 lines
Quoted 4 lines
Quoted 1 line
A trace would be appropriate imho. Thanks,
Sorry, can't catch you. What kind of info to be traced? The errors returned by
replay_populate()?Yeah. imho these are useful as we expect this part to work in general too, right? Thanks,
Something like?
system/guest-memfd-manager.c +0
-0@@ -185,8 +185,14 @@ static intguest_memfd_rdm_replay_populated_cb(MemoryRegionSection *section, voi
{ struct GuestMemfdReplayData *data = arg; ReplayRamPopulate replay_fn = data->fn;system/guest-memfd-manager.c +7
-1+int ret; -return replay_fn (section, data->opaque); + ret =replay_fn (section, data->opaque); +if (ret) { +trace_guest_memfd_rdm_replay_populated_cb (ret); + } + +return ret; }
Quoted 3 lines
How about just adding some error output in guest_memfd_for_each_populated_section()/guest_memfd_for_each_discarded_section() if the
cb()(i.e.replay_populate()) returns error?
this will do too, yes. Thanks,
Quoted 77 lines
Quoted 76 lines
Quoted 75 lines
Quoted 74 lines
Quoted 73 lines
Quoted 32 lines
Quoted 32 lines
(quoted diff) +5
-0+ + return 0; +} + +static void guest_memfd_rdm_replay_discarded(const RamDiscardManager*rdm,
+ MemoryRegionSection *section,
+ ReplayRamDiscardreplay_fn,
+ void *opaque) +{
+ GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER(rdm);
+ struct GuestMemfdReplayData data = { .fn =replay_fn, .opaque = opaque };(quoted diff) +4
-0+ + g_assert(section->mr == gmm->mr); + guest_memfd_for_each_discarded_section(gmm, section, &data, +
guest_memfd_rdm_replay_discarded_cb);(quoted diff) +7
-0+} + +static void guest_memfd_manager_init(Object *obj) +{ + GuestMemfdManager *gmm = GUEST_MEMFD_MANAGER(obj); + + QLIST_INIT(&gmm->rdl_list);Quoted 8 lines
Review
Quoted 5 lines
(quoted diff) +5
-0+} + +static void guest_memfd_manager_finalize(Object *obj) +{ + g_free(GUEST_MEMFD_MANAGER(obj)->bitmap);bitmap is not allocated though. And 5/7 removes this anyway. Thanks,
Will remove it. Thanks.
Quoted 29 lines
Quoted 28 lines
(quoted diff) +3
-0+} + +static void guest_memfd_manager_class_init(ObjectClass *oc, void*data)
(quoted diff) +10
-0+{ + RamDiscardManagerClass *rdmc = RAM_DISCARD_MANAGER_CLASS(oc); + + rdmc->get_min_granularity = guest_memfd_rdm_get_min_granularity; + rdmc->register_listener = guest_memfd_rdm_register_listener; + rdmc->unregister_listener = guest_memfd_rdm_unregister_listener; + rdmc->is_populated = guest_memfd_rdm_is_populated; + rdmc->replay_populated = guest_memfd_rdm_replay_populated; + rdmc->replay_discarded = guest_memfd_rdm_replay_discarded; +}system/meson.build +1
-0@@ -15,6 +15,7 @@ system_ss.add(files( 'dirtylimit.c', 'dma-helpers.c', 'globals.c', + 'guest-memfd-manager.c', 'memory_mapping.c', 'qdev-monitor.c', 'qtest.c',
Signature · 1 line
Alexey
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 53 lines · Alexey Kardashevskiy
Quoted 50 lines · Chenyi Qiang
Quoted 47 lines · Alexey Kardashevskiy
Quoted 44 lines · Chenyi Qiang
Quoted 41 lines · Alexey Kardashevskiy
Quoted 38 lines · Chenyi Qiang
Quoted 35 lines · Alexey Kardashevskiy
Quoted 33 lines · Chenyi Qiang
As the commit
852f0048f3("RAMBlock: makeguest_memfdrequire uncoordinated discard") highlighted, some subsystems like VFIO might
disable ram block discard. However,guest_memfdrelies on the discard
operation to perform page conversion between private and shared memory.
This can lead to stale IOMMU mapping issue when assigning a hardware
device to a confidential VM via shared memory (unprotected memory pages). Blocking shared page discard can solve this problem, but it could cause guests to consume twice the memory with VFIO, which is not
acceptable in some cases. An alternative solution is to convey other
systems like VFIO to refresh its outdated IOMMU mappings.RamDiscardManager is an existing concept (used by virtio-mem) to adjust
VFIO mappings in relation to VM page assignment. Effectively page conversion is similar to hot-removing a page in one mode and adding it
back in the other, so the similar work that needs to happen in response
to virtio-mem changes needs to happen for page conversion events. Introduce the RamDiscardManager toguest_memfdto achieve it.However,
guest_memfdis not an object so it cannot directly implement
the RamDiscardManager interface.One solution is to implement the interface in HostMemoryBackend. Any
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
This sounds about right.
btw I am using this for ages:
https://github.com/aik/qemu/commit/3663f889883d4aebbeb0e4422f7be5e357e2ee46
but I am not sure if this ever saw the light of the day, did not it? (ironically I am using it as a base for encrypted DMA :) )
Yeah, we are doing the same work. I saw a solution from Michael long time ago (when there was still
a dedicated hostmem-memfd-private backend for restrictedmem/gmem)
(https://github.com/AMDESE/qemu/commit/3bf5255fc48d648724d66410485081ace41d8ee6)
For your patch, it only implement the interface for HostMemoryBackendMemfd. Maybe it is more appropriate to implement it for the parent object HostMemoryBackend, because besides the MEMORY_BACKEND_MEMFD, other backend types like MEMORY_BACKEND_RAM and MEMORY_BACKEND_FILE can also be guest_memfd-backed.
Think more about where to implement this interface. It is still uncertain to me. As I mentioned in another mail, maybe ram device memory region would be backed by guest_memfd if we support TEE IO iommufd MMIO in future. Then a specific object is more appropriate. What's your opinion?
Quoted 50 lines
Quoted 41 lines
Quoted 33 lines
Quoted 27 lines
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
Quoted 9 lines
guest_memfd-backed host memory backend can register itself in the target
MemoryRegion. However, this solution doesn't cover the scenario where aguest_memfdMemoryRegion doesn't belong to the HostMemoryBackend, e.g.
the virtual BIOS MemoryRegion.What is this virtual BIOS MemoryRegion exactly? What does it look like
in "info mtree -f"? Do we really want this memory to be DMAable?Quoted 8 lines
Quoted 6 lines
virtual BIOS shows in a separate region:
Root memory region: system 0000000000000000-000000007fffffff (prio 0, ram): pc.ram KVM
...
00000000ffc00000-00000000ffffffff (prio 0, ram): pc.bios KVMLooks like a normal MR which can be backed by
guest_memfd.Yes, virtual BIOS memory region is initialized by
memory_region_init_ram_guest_memfd()which will be backed by aguest_memfd.Quoted 5 lines
The tricky thing is, for Intel TDX (not sure about AMD SEV), the virtual
BIOS image will be loaded and then copied to private region. After that,
the loaded image will be discarded and this region become useless.Quoted 4 lines
I'd think it is loaded as "struct Rom" and then copied to the MR-
ram_guest_memfd()which does not leave MR useless - we still see "pc.bios" in the list so it is not discarded. What piece of code are you referring to exactly?Sorry for confusion, maybe it is different between TDX and SEV-SNP for the vBIOS handling.
Quoted 8 lines
In
x86_bios_rom_init(), it initializes a guest_memfd-backed MR and loads the vBIOS image to the shared part of theguest_memfdMR. For TDX, it
will copy the image to private region (not the vBIOSguest_memfdMR private part) and discard the shared part. So, although the memory region still exists, it seems useless.
It is different for SEV-SNP, correct? Does SEV-SNP manage the vBIOS in vBIOSguest_memfdprivate memory?
Quoted 2 lines
This is what it looks like on my SNP VM (which, I suspect, is the same as yours as
hw/i386/pc.cdoes not distinguish Intel/AMD for this matter):
Yes, the memory region object is created on both TDX and SEV-SNP.
Quoted 18 lines
Root memory region: system 0000000000000000-00000000000bffff (prio 0, ram): ram1 KVM gmemfd=20 00000000000c0000-00000000000dffff (prio 1, ram): pc.rom KVM gmemfd=27 00000000000e0000-000000001fffffff (prio 0, ram): ram1 @00000000000e0000 KVM gmemfd=20
...
00000000ffc00000-00000000ffffffff (prio 0, ram): pc.bios KVM gmemfd=26So the pc.bios MR exists and in use (hence its appearance in "info mtree -f").
I added the gmemfd dumping:
system/memory.c static void mtree_print_flatview(gpointer key, · line 3453
Quoted 8 lines
system/memory.c +3
-0@@ -3446,6 +3446,9 @@ static void mtree_print_flatview(gpointer key,gpointer value, } } } + if (mr->ram_block && mr->ram_block->guest_memfd >= 0) { + qemu_printf(" gmemfd=%d", mr->ram_block->guest_memfd); + }
Then I think the virtual BIOS is another case not belonging to HostMemoryBackend which convince us to implement the interface in a specific object, no?
Quoted 120 lines
Quoted 113 lines
Quoted 43 lines
Quoted 5 lines
So I
feel like this virtual BIOS should not be backed byguest_memfd?From the above it sounds like the opposite, i.e. it should :)
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
0000000100000000-000000017fffffff (prio 0, ram): pc.ram @0000000080000000 KVM
Anyway if there is no
guest_memfdbacking it andmemory_region_has_ram_discard_manager()returns false, then the MR is just going to be mapped for VFIO as usual which seems... alright, right?Quoted 7 lines
Correct. As the vBIOS is backed by
guest_memfdand we implement the RDM forguest_memfd_manager, the vBIOS MR won't be mapped by VFIO.If we go with the HostMemoryBackend instead of
guest_memfd_manager, this
MR would be mapped by VFIO. Maybe need to avoid such vBIOS mapping, or just ignore it since the MR is useless (but looks not so good).Sorry I am missing necessary details here, let's figure out the above.
Quoted 16 lines
Quoted 9 lines
Quoted 9 lines
We also consider to implement the interface in HostMemoryBackend, but maybe implement with
guest_memfdregion is more general. We don't know
if any DMAable memory would belong to HostMemoryBackend although at present it is.Quoted 5 lines
Quoted 3 lines
If it is more appropriate to implement it with HostMemoryBackend, I can
change to this way.Seems cleaner imho.
I can go this way.
[...]
Quoted 37 lines
Quoted 37 lines
Quoted 37 lines
Quoted 36 lines
Quoted 24 lines
Quoted 24 lines
(quoted diff) +2
-0+ +static int guest_memfd_rdm_replay_populated(constRamDiscardManager
*rdm,
+ MemoryRegionSection *section,
+ ReplayRamPopulatereplay_fn,
+ void *opaque) +{
+ GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER(rdm); + struct GuestMemfdReplayData data = { .fn =replay_fn, .opaque =
opaque };
+
+g_assert(section->mr == gmm->mr);
+ return guest_memfd_for_each_populated_section(gmm, section, &data,
+guest_memfd_rdm_replay_populated_cb);(quoted diff) +3
-0+} + +static int guest_memfd_rdm_replay_discarded_cb(MemoryRegionSection*section, void *arg)
Quoted 8 lines
Review
Quoted 5 lines
(quoted diff) +5
-0+{ + struct GuestMemfdReplayData *data = arg; + ReplayRamDiscard replay_fn = data->fn; + + replay_fn(section, data->opaque);
guest_memfd_rdm_replay_populated_cb()checks for errors though.It follows current definiton of
ReplayRamDiscard()andReplayRamPopulate()wherereplay_discard()doesn't return errors andreplay_populate()returns errors.Quoted 7 lines
Quoted 4 lines
Quoted 1 line
A trace would be appropriate imho. Thanks,
Sorry, can't catch you. What kind of info to be traced? The errors returned by
replay_populate()?Yeah. imho these are useful as we expect this part to work in general too, right? Thanks,
Something like?
system/guest-memfd-manager.c +0
-0@@ -185,8 +185,14 @@ static intguest_memfd_rdm_replay_populated_cb(MemoryRegionSection *section, voi {
struct GuestMemfdReplayData *data = arg; ReplayRamPopulatereplay_fn=data->fn;
+ int ret;- return
replay_fn(section, data->opaque);
+ ret =replay_fn(section, data->opaque);
+ if (ret) {
+trace_guest_memfd_rdm_replay_populated_cb(ret); + }
+
+ return ret;
}Quoted 4 lines
How about just adding some error output in guest_memfd_for_each_populated_section()/
guest_memfd_for_each_discarded_section()
if thecb()(i.e.replay_populate()) returns error?this will do too, yes. Thanks,
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 17 lines
Quoted 5 lines
https://github.com/aik/qemu/commit/3663f889883d4aebbeb0e4422f7be5e357e2ee46
but I am not sure if this ever saw the light of the day, did not it? (ironically I am using it as a base for encrypted DMA :) )
Yeah, we are doing the same work. I saw a solution from Michael long time ago (when there was still
a dedicated hostmem-memfd-private backend for restrictedmem/gmem)
(https://github.com/AMDESE/qemu/commit/3bf5255fc48d648724d66410485081ace41d8ee6)For your patch, it only implement the interface for HostMemoryBackendMemfd. Maybe it is more appropriate to implement it for the parent object HostMemoryBackend, because besides the
MEMORY_BACKEND_MEMFD, other backend types likeMEMORY_BACKEND_RAMandMEMORY_BACKEND_FILEcan also be guest_memfd-backed.
Quoted 3 lines
Think more about where to implement this interface. It is still uncertain to me. As I mentioned in another mail, maybe ram device memory region would be backed by
guest_memfdif we support TEE IO iommufd MMIO
It is unlikely an assigned MMIO region would be backed by guest_memfd or be implemented as part of HostMemoryBackend. Nowadays assigned MMIO resource is owned by VFIO types, and I assume it is still true for private MMIO.
But I think with TIO, MMIO regions also need conversion. So I support an object, but maybe not guest_memfd_manager.
Thanks,
Yilun
Quoted 2 lines
in future. Then a specific object is more appropriate. What's your opinion?
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 21 lines · Xu Yilun
Quoted 17 lines
Quoted 5 lines
https://github.com/aik/qemu/commit/3663f889883d4aebbeb0e4422f7be5e357e2ee46
but I am not sure if this ever saw the light of the day, did not it? (ironically I am using it as a base for encrypted DMA :) )
Yeah, we are doing the same work. I saw a solution from Michael long time ago (when there was still
a dedicated hostmem-memfd-private backend for restrictedmem/gmem)
(https://github.com/AMDESE/qemu/commit/3bf5255fc48d648724d66410485081ace41d8ee6)For your patch, it only implement the interface for HostMemoryBackendMemfd. Maybe it is more appropriate to implement it for the parent object HostMemoryBackend, because besides the
MEMORY_BACKEND_MEMFD, other backend types likeMEMORY_BACKEND_RAMandMEMORY_BACKEND_FILEcan also be guest_memfd-backed.Quoted 3 lines
Think more about where to implement this interface. It is still uncertain to me. As I mentioned in another mail, maybe ram device memory region would be backed by
guest_memfdif we support TEE IO iommufd MMIO
Quoted 6 lines
It is unlikely an assigned MMIO region would be backed by
guest_memfdor be implemented as part of HostMemoryBackend. Nowadays assigned MMIO resource is owned by VFIO types, and I assume it is still true for private MMIO.But I think with TIO, MMIO regions also need conversion. So I support an object, but maybe not
guest_memfd_manager.
Sorry, I mean the name only covers private memory, but not private MMIO.
Quoted 7 lines
Thanks,
YilunQuoted 2 lines
in future. Then a specific object is more appropriate. What's your opinion?
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 22 lines · Xu Yilun
Quoted 21 lines · Xu Yilun
Quoted 20 lines
Quoted 5 lines
https://github.com/aik/qemu/commit/3663f889883d4aebbeb0e4422f7be5e357e2ee46
but I am not sure if this ever saw the light of the day, did not it? (ironically I am using it as a base for encrypted DMA :) )
Yeah, we are doing the same work. I saw a solution from Michael long time ago (when there was still
a dedicated hostmem-memfd-private backend for restrictedmem/gmem)
(https://github.com/AMDESE/qemu/commit/3bf5255fc48d648724d66410485081ace41d8ee6)For your patch, it only implement the interface for HostMemoryBackendMemfd. Maybe it is more appropriate to implement it for the parent object HostMemoryBackend, because besides the
MEMORY_BACKEND_MEMFD, other backend types likeMEMORY_BACKEND_RAMandMEMORY_BACKEND_FILEcan also be guest_memfd-backed.Think more about where to implement this interface. It is still uncertain to me. As I mentioned in another mail, maybe ram device memory region would be backed by
guest_memfdif we support TEE IO iommufd MMIO
Quoted 8 lines
Quoted 6 lines
It is unlikely an assigned MMIO region would be backed by
guest_memfdor be implemented as part of HostMemoryBackend. Nowadays assigned MMIO resource is owned by VFIO types, and I assume it is still true for private MMIO.But I think with TIO, MMIO regions also need conversion. So I support an object, but maybe not
guest_memfd_manager.Sorry, I mean the name only covers private memory, but not private MMIO.
So you suggest renaming the object to cover the private MMIO. Then how about page_conversion_manager, or page_attribute_manager?
Quoted 8 lines
Quoted 8 lines
Thanks,
YilunQuoted 2 lines
in future. Then a specific object is more appropriate. What's your opinion?
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 29 lines · Chenyi Qiang
Quoted 26 lines · Xu Yilun
Quoted 25 lines · Xu Yilun
Quoted 17 lines
Quoted 5 lines
https://github.com/aik/qemu/commit/3663f889883d4aebbeb0e4422f7be5e357e2ee46
but I am not sure if this ever saw the light of the day, did not it? (ironically I am using it as a base for encrypted DMA :) )
Yeah, we are doing the same work. I saw a solution from Michael long time ago (when there was still
a dedicated hostmem-memfd-private backend for restrictedmem/gmem)
(https://github.com/AMDESE/qemu/commit/3bf5255fc48d648724d66410485081ace41d8ee6)For your patch, it only implement the interface for HostMemoryBackendMemfd. Maybe it is more appropriate to implement it for the parent object HostMemoryBackend, because besides the
MEMORY_BACKEND_MEMFD, other backend types likeMEMORY_BACKEND_RAMandMEMORY_BACKEND_FILEcan also be guest_memfd-backed.Quoted 3 lines
Think more about where to implement this interface. It is still uncertain to me. As I mentioned in another mail, maybe ram device memory region would be backed by
guest_memfdif we support TEE IO iommufd MMIOIt is unlikely an assigned MMIO region would be backed by
guest_memfdor be implemented as part of HostMemoryBackend. Nowadays assigned MMIO resource is owned by VFIO types, and I assume it is still true for private MMIO.
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
But I think with TIO, MMIO regions also need conversion. So I support an object, but maybe not
guest_memfd_manager.Sorry, I mean the name only covers private memory, but not private MMIO.
So you suggest renaming the object to cover the private MMIO. Then how
Yes.
Quoted 1 line
about
page_conversion_manager, orpage_attribute_manager?
Maybe memory_attribute_manager? Strictly speaking MMIO resource is not backed by pages.
Thanks,
Yilun
Quoted 10 lines
Quoted 9 lines
Quoted 9 lines
Thanks,
YilunQuoted 2 lines
in future. Then a specific object is more appropriate. What's your opinion?
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 35 lines · Xu Yilun
Quoted 34 lines · Chenyi Qiang
Quoted 30 lines · Xu Yilun
Quoted 21 lines · Xu Yilun
Quoted 17 lines
Quoted 5 lines
https://github.com/aik/qemu/commit/3663f889883d4aebbeb0e4422f7be5e357e2ee46
but I am not sure if this ever saw the light of the day, did not it? (ironically I am using it as a base for encrypted DMA :) )
Yeah, we are doing the same work. I saw a solution from Michael long time ago (when there was still
a dedicated hostmem-memfd-private backend for restrictedmem/gmem)
(https://github.com/AMDESE/qemu/commit/3bf5255fc48d648724d66410485081ace41d8ee6)For your patch, it only implement the interface for HostMemoryBackendMemfd. Maybe it is more appropriate to implement it for the parent object HostMemoryBackend, because besides the
MEMORY_BACKEND_MEMFD, other backend types likeMEMORY_BACKEND_RAMandMEMORY_BACKEND_FILEcan also be guest_memfd-backed.Quoted 3 lines
Think more about where to implement this interface. It is still uncertain to me. As I mentioned in another mail, maybe ram device memory region would be backed by
guest_memfdif we support TEE IO iommufd MMIOQuoted 6 lines
It is unlikely an assigned MMIO region would be backed by
guest_memfdor be implemented as part of HostMemoryBackend. Nowadays assigned MMIO resource is owned by VFIO types, and I assume it is still true for private MMIO.But I think with TIO, MMIO regions also need conversion. So I support an object, but maybe not
guest_memfd_manager.Sorry, I mean the name only covers private memory, but not private MMIO.
Quoted 8 lines
Quoted 1 line
So you suggest renaming the object to cover the private MMIO. Then how
Yes.
Quoted 1 line
about
page_conversion_manager, orpage_attribute_manager?Maybe
memory_attribute_manager? Strictly speaking MMIO resource is not backed by pages.
Looks good to me. Thanks!
Quoted 15 lines
Thanks,
YilunQuoted 10 lines
Quoted 9 lines
Quoted 9 lines
Thanks,
YilunQuoted 2 lines
in future. Then a specific object is more appropriate. What's your opinion?
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
In reply to Chenyi QiangQuoted 76 lines · Chenyi Qiang
Quoted 53 lines · Alexey Kardashevskiy
Quoted 50 lines · Chenyi Qiang
Quoted 47 lines · Alexey Kardashevskiy
Quoted 44 lines · Chenyi Qiang
Quoted 41 lines · Alexey Kardashevskiy
Quoted 38 lines · Chenyi Qiang
Quoted 35 lines · Alexey Kardashevskiy
Quoted 33 lines · Chenyi Qiang
As the commit
852f0048f3("RAMBlock: makeguest_memfdrequire uncoordinated discard") highlighted, some subsystems like VFIO might
disable ram block discard. However,guest_memfdrelies on the discard
operation to perform page conversion between private and shared memory.
This can lead to stale IOMMU mapping issue when assigning a hardware
device to a confidential VM via shared memory (unprotected memory pages). Blocking shared page discard can solve this problem, but it could cause guests to consume twice the memory with VFIO, which is not
acceptable in some cases. An alternative solution is to convey other
systems like VFIO to refresh its outdated IOMMU mappings.RamDiscardManager is an existing concept (used by virtio-mem) to adjust
VFIO mappings in relation to VM page assignment. Effectively page conversion is similar to hot-removing a page in one mode and adding it
back in the other, so the similar work that needs to happen in response
to virtio-mem changes needs to happen for page conversion events. Introduce the RamDiscardManager toguest_memfdto achieve it.However,
guest_memfdis not an object so it cannot directly implement
the RamDiscardManager interface.One solution is to implement the interface in HostMemoryBackend. Any
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
This sounds about right.
btw I am using this for ages:
https://github.com/aik/qemu/commit/3663f889883d4aebbeb0e4422f7be5e357e2ee46
but I am not sure if this ever saw the light of the day, did not it? (ironically I am using it as a base for encrypted DMA :) )
Yeah, we are doing the same work. I saw a solution from Michael long time ago (when there was still
a dedicated hostmem-memfd-private backend for restrictedmem/gmem)
(https://github.com/AMDESE/qemu/commit/3bf5255fc48d648724d66410485081ace41d8ee6)For your patch, it only implement the interface for HostMemoryBackendMemfd. Maybe it is more appropriate to implement it for the parent object HostMemoryBackend, because besides the
MEMORY_BACKEND_MEMFD, other backend types likeMEMORY_BACKEND_RAMandMEMORY_BACKEND_FILEcan also be guest_memfd-backed.
Quoted 4 lines
Think more about where to implement this interface. It is still uncertain to me. As I mentioned in another mail, maybe ram device memory region would be backed by
guest_memfdif we support TEE IO iommufd MMIO in future. Then a specific object is more appropriate. What's your opinion?
I do not know about this. Unlike RAM, MMIO can only do "in-place conversion" and the interface to do so is not straight forward and VFIO owns MMIO anyway so the uAPI will be in iommufd, here is a gist of it:
https://github.com/aik/linux/commit/89e45c0404fa5006b2a4de33a4d582adf1ba9831
"guest request" is a communication channel from the VM to the secure FW (AMD's "PSP") to make MMIO allow encrypted access.
Quoted 70 lines
Quoted 50 lines
Quoted 41 lines
Quoted 33 lines
Quoted 27 lines
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
Quoted 9 lines
guest_memfd-backed host memory backend can register itself in the target
MemoryRegion. However, this solution doesn't cover the scenario where aguest_memfdMemoryRegion doesn't belong to the HostMemoryBackend, e.g.
the virtual BIOS MemoryRegion.What is this virtual BIOS MemoryRegion exactly? What does it look like
in "info mtree -f"? Do we really want this memory to be DMAable?Quoted 8 lines
Quoted 6 lines
virtual BIOS shows in a separate region:
Root memory region: system 0000000000000000-000000007fffffff (prio 0, ram): pc.ram KVM
...
00000000ffc00000-00000000ffffffff (prio 0, ram): pc.bios KVMLooks like a normal MR which can be backed by
guest_memfd.Yes, virtual BIOS memory region is initialized by
memory_region_init_ram_guest_memfd()which will be backed by aguest_memfd.Quoted 5 lines
The tricky thing is, for Intel TDX (not sure about AMD SEV), the virtual
BIOS image will be loaded and then copied to private region. After that,
the loaded image will be discarded and this region become useless.Quoted 4 lines
I'd think it is loaded as "struct Rom" and then copied to the MR-
ram_guest_memfd()which does not leave MR useless - we still see "pc.bios" in the list so it is not discarded. What piece of code are you referring to exactly?Sorry for confusion, maybe it is different between TDX and SEV-SNP for the vBIOS handling.
Quoted 8 lines
In
x86_bios_rom_init(), it initializes a guest_memfd-backed MR and loads the vBIOS image to the shared part of theguest_memfdMR. For TDX, it
will copy the image to private region (not the vBIOSguest_memfdMR private part) and discard the shared part. So, although the memory region still exists, it seems useless.
It is different for SEV-SNP, correct? Does SEV-SNP manage the vBIOS in vBIOSguest_memfdprivate memory?Quoted 2 lines
This is what it looks like on my SNP VM (which, I suspect, is the same as yours as
hw/i386/pc.cdoes not distinguish Intel/AMD for this matter):Yes, the memory region object is created on both TDX and SEV-SNP.
Quoted 15 lines
Root memory region: system 0000000000000000-00000000000bffff (prio 0, ram): ram1 KVM gmemfd=20 00000000000c0000-00000000000dffff (prio 1, ram): pc.rom KVM gmemfd=27 00000000000e0000-000000001fffffff (prio 0, ram): ram1
@00000000000e0000 KVM gmemfd=20
...
00000000ffc00000-00000000ffffffff (prio 0, ram): pc.bios KVM gmemfd=26So the pc.bios MR exists and in use (hence its appearance in "info mtree -f").
I added the gmemfd dumping:
Quoted 15 lines
system/memory.c static void mtree_print_flatview(gpointer key, · line 3453
Quoted 11 lines
system/memory.c +3
-0@@ -3446,6 +3446,9 @@ static void mtree_print_flatview(gpointer key,gpointer value, } } } + if (mr->ram_block && mr->ram_block->guest_memfd >= 0) { + qemu_printf(" gmemfd=%d", mr->ram_block->guest_memfd); + }Then I think the virtual BIOS is another case not belonging to HostMemoryBackend which convince us to implement the interface in a specific object, no?
TBH I have no idea why pc.rom and pc.bios are separate memory regions but in any case why do these 2 areas need to be treated any different than the rest of RAM? Thanks,
Quoted 123 lines
Quoted 120 lines
Quoted 113 lines
Quoted 43 lines
Quoted 5 lines
So I
feel like this virtual BIOS should not be backed byguest_memfd?From the above it sounds like the opposite, i.e. it should :)
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
0000000100000000-000000017fffffff (prio 0, ram): pc.ram
@0000000080000000 KVMAnyway if there is no
guest_memfdbacking it andmemory_region_has_ram_discard_manager()returns false, then the MR is just going to be mapped for VFIO as usual which seems... alright, right?Quoted 7 lines
Correct. As the vBIOS is backed by
guest_memfdand we implement the RDM forguest_memfd_manager, the vBIOS MR won't be mapped by VFIO.If we go with the HostMemoryBackend instead of
guest_memfd_manager, this
MR would be mapped by VFIO. Maybe need to avoid such vBIOS mapping, or just ignore it since the MR is useless (but looks not so good).Sorry I am missing necessary details here, let's figure out the above.
Quoted 16 lines
Quoted 9 lines
Quoted 9 lines
We also consider to implement the interface in HostMemoryBackend, but maybe implement with
guest_memfdregion is more general. We don't know
if any DMAable memory would belong to HostMemoryBackend although at present it is.Quoted 5 lines
Quoted 3 lines
If it is more appropriate to implement it with HostMemoryBackend, I can
change to this way.Seems cleaner imho.
I can go this way.
[...]
Quoted 37 lines
Quoted 37 lines
Quoted 37 lines
Quoted 36 lines
Quoted 24 lines
Quoted 24 lines
(quoted diff) +2
-0+ +static int guest_memfd_rdm_replay_populated(constRamDiscardManager
*rdm,
+ MemoryRegionSection *section,
+ ReplayRamPopulatereplay_fn,
+ void *opaque) +{
+ GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER(rdm); + struct GuestMemfdReplayData data = { .fn =replay_fn, .opaque =
opaque };
+
+g_assert(section->mr == gmm->mr);
+ return guest_memfd_for_each_populated_section(gmm, section, &data,
+guest_memfd_rdm_replay_populated_cb);(quoted diff) +3
-0+} + +static int guest_memfd_rdm_replay_discarded_cb(MemoryRegionSection*section, void *arg)
Quoted 8 lines
Review
Quoted 5 lines
(quoted diff) +5
-0+{ + struct GuestMemfdReplayData *data = arg; + ReplayRamDiscard replay_fn = data->fn; + + replay_fn(section, data->opaque);
guest_memfd_rdm_replay_populated_cb()checks for errors though.It follows current definiton of
ReplayRamDiscard()andReplayRamPopulate()wherereplay_discard()doesn't return errors andreplay_populate()returns errors.Quoted 7 lines
Quoted 4 lines
Quoted 1 line
A trace would be appropriate imho. Thanks,
Sorry, can't catch you. What kind of info to be traced? The errors returned by
replay_populate()?Yeah. imho these are useful as we expect this part to work in general too, right? Thanks,
Something like?
system/guest-memfd-manager.c +0
-0@@ -185,8 +185,14 @@ static intguest_memfd_rdm_replay_populated_cb(MemoryRegionSection *section, voi
{
struct GuestMemfdReplayData *data = arg; ReplayRamPopulatereplay_fn=data->fn;
+ int ret;- return
replay_fn(section, data->opaque);
+ ret =replay_fn(section, data->opaque);
+ if (ret) {
+trace_guest_memfd_rdm_replay_populated_cb(ret); + }
+
+ return ret;
}Quoted 4 lines
How about just adding some error output in guest_memfd_for_each_populated_section()/
guest_memfd_for_each_discarded_section()
if thecb()(i.e.replay_populate()) returns error?this will do too, yes. Thanks,
Signature · 1 line
Alexey
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 91 lines · Alexey Kardashevskiy
Quoted 80 lines · Chenyi Qiang
Quoted 57 lines · Alexey Kardashevskiy
Quoted 45 lines · Chenyi Qiang
Quoted 42 lines · Alexey Kardashevskiy
Quoted 39 lines · Chenyi Qiang
Quoted 36 lines · Alexey Kardashevskiy
Quoted 33 lines · Chenyi Qiang
Quoted 30 lines · Alexey Kardashevskiy
Quoted 29 lines · Chenyi Qiang
As the commit
852f0048f3("RAMBlock: makeguest_memfdrequire uncoordinated discard") highlighted, some subsystems like VFIO might
disable ram block discard. However,guest_memfdrelies on the discard
operation to perform page conversion between private and shared memory.
This can lead to stale IOMMU mapping issue when assigning a hardware
device to a confidential VM via shared memory (unprotected memory pages). Blocking shared page discard can solve this problem, but it
could cause guests to consume twice the memory with VFIO, which is
not
acceptable in some cases. An alternative solution is to convey other
systems like VFIO to refresh its outdated IOMMU mappings.RamDiscardManager is an existing concept (used by virtio-mem) to adjust
VFIO mappings in relation to VM page assignment. Effectively page conversion is similar to hot-removing a page in one mode and adding it
back in the other, so the similar work that needs to happen in response
to virtio-mem changes needs to happen for page conversion events. Introduce the RamDiscardManager toguest_memfdto achieve it.Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
However,
guest_memfdis not an object so it cannot directly implement
the RamDiscardManager interface.One solution is to implement the interface in HostMemoryBackend. Any
This sounds about right.
Quoted 7 lines
btw I am using this for ages:
https://github.com/aik/qemu/
commit/3663f889883d4aebbeb0e4422f7be5e357e2ee46but I am not sure if this ever saw the light of the day, did not it? (ironically I am using it as a base for encrypted DMA :) )
Yeah, we are doing the same work. I saw a solution from Michael long time ago (when there was still
a dedicated hostmem-memfd-private backend for restrictedmem/gmem)
(https://github.com/AMDESE/qemu/
commit/3bf5255fc48d648724d66410485081ace41d8ee6)For your patch, it only implement the interface for HostMemoryBackendMemfd. Maybe it is more appropriate to implement it for the parent object HostMemoryBackend, because besides the
MEMORY_BACKEND_MEMFD, other backend types likeMEMORY_BACKEND_RAMandMEMORY_BACKEND_FILEcan also be guest_memfd-backed.Quoted 5 lines
Think more about where to implement this interface. It is still uncertain to me. As I mentioned in another mail, maybe ram device memory region would be backed by
guest_memfdif we support TEE IO iommufd MMIO in future. Then a specific object is more appropriate. What's your opinion?I do not know about this. Unlike RAM, MMIO can only do "in-place conversion" and the interface to do so is not straight forward and VFIO owns MMIO anyway so the uAPI will be in iommufd, here is a gist of it:
Quoted 5 lines
https://github.com/aik/linux/
commit/89e45c0404fa5006b2a4de33a4d582adf1ba9831"guest request" is a communication channel from the VM to the secure FW (AMD's "PSP") to make MMIO allow encrypted access.
It is still uncertain how to implement the private MMIO. Our assumption is the private MMIO would also create a memory region with guest_memfd-like backend. Its mr->ram is true and should be managed by RamdDiscardManager which can skip doing DMA_MAP in VFIO's region_add listener.
Quoted 92 lines
Quoted 76 lines
Quoted 54 lines
Quoted 44 lines
Quoted 35 lines
Quoted 29 lines
Quoted 16 lines
Quoted 16 lines
Quoted 15 lines
Quoted 11 lines
guest_memfd-backed host memory backend can register itself in the target
MemoryRegion. However, this solution doesn't cover the scenario where aguest_memfdMemoryRegion doesn't belong to the HostMemoryBackend, e.g.
the virtual BIOS MemoryRegion.What is this virtual BIOS MemoryRegion exactly? What does it look like
in "info mtree -f"? Do we really want this memory to be DMAable?Quoted 8 lines
Quoted 6 lines
virtual BIOS shows in a separate region:
Root memory region: system 0000000000000000-000000007fffffff (prio 0, ram): pc.ram KVM
...
00000000ffc00000-00000000ffffffff (prio 0, ram): pc.bios KVMLooks like a normal MR which can be backed by
guest_memfd.Yes, virtual BIOS memory region is initialized by
memory_region_init_ram_guest_memfd()which will be backed by aguest_memfd.Quoted 5 lines
The tricky thing is, for Intel TDX (not sure about AMD SEV), the virtual
BIOS image will be loaded and then copied to private region. After that,
the loaded image will be discarded and this region become useless.Quoted 5 lines
I'd think it is loaded as "struct Rom" and then copied to the MR-
ram_guest_memfd()which does not leave MR useless - we still see "pc.bios" in the list so it is not discarded. What piece of code are you
referring to exactly?Sorry for confusion, maybe it is different between TDX and SEV-SNP for the vBIOS handling.
Quoted 9 lines
In
x86_bios_rom_init(), it initializes a guest_memfd-backed MR and loads
the vBIOS image to the shared part of theguest_memfdMR. For TDX, it
will copy the image to private region (not the vBIOSguest_memfdMR private part) and discard the shared part. So, although the memory region still exists, it seems useless.
It is different for SEV-SNP, correct? Does SEV-SNP manage the vBIOS in vBIOSguest_memfdprivate memory?Quoted 3 lines
This is what it looks like on my SNP VM (which, I suspect, is the same as yours as
hw/i386/pc.cdoes not distinguish Intel/AMD for this matter):Yes, the memory region object is created on both TDX and SEV-SNP.
Quoted 16 lines
Root memory region: system 0000000000000000-00000000000bffff (prio 0, ram): ram1 KVM gmemfd=20 00000000000c0000-00000000000dffff (prio 1, ram): pc.rom KVM gmemfd=27 00000000000e0000-000000001fffffff (prio 0, ram): ram1 @00000000000e0000 KVM gmemfd=20
...
00000000ffc00000-00000000ffffffff (prio 0, ram): pc.bios KVM gmemfd=26So the pc.bios MR exists and in use (hence its appearance in "info mtree -f").
I added the gmemfd dumping:
Quoted 15 lines
system/memory.c static void mtree_print_flatview(gpointer key, · line 3453
Quoted 11 lines
system/memory.c +3
-0@@ -3446,6 +3446,9 @@ static void mtree_print_flatview(gpointer key,gpointer value, } } } + if (mr->ram_block && mr->ram_block->guest_memfd >= 0) { + qemu_printf(" gmemfd=%d", mr->ram_block->guest_memfd); + }Then I think the virtual BIOS is another case not belonging to HostMemoryBackend which convince us to implement the interface in a specific object, no?
Quoted 3 lines
TBH I have no idea why pc.rom and pc.bios are separate memory regions but in any case why do these 2 areas need to be treated any different than the rest of RAM? Thanks,
I think no difference. That's why I suggest implementing the RDM interface in a specific object to cover both instead of the only HostMemoryBackend.
Quoted 2 lines
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 97 lines · Chenyi Qiang
Quoted 94 lines · Alexey Kardashevskiy
Quoted 80 lines · Chenyi Qiang
Quoted 57 lines · Alexey Kardashevskiy
Quoted 45 lines · Chenyi Qiang
Quoted 42 lines · Alexey Kardashevskiy
Quoted 39 lines · Chenyi Qiang
Quoted 36 lines · Alexey Kardashevskiy
Quoted 33 lines · Chenyi Qiang
Quoted 30 lines · Alexey Kardashevskiy
Quoted 29 lines · Chenyi Qiang
As the commit
852f0048f3("RAMBlock: makeguest_memfdrequire uncoordinated discard") highlighted, some subsystems like VFIO might
disable ram block discard. However,guest_memfdrelies on the discard
operation to perform page conversion between private and shared memory.
This can lead to stale IOMMU mapping issue when assigning a hardware
device to a confidential VM via shared memory (unprotected memory pages). Blocking shared page discard can solve this problem, but it
could cause guests to consume twice the memory with VFIO, which is
not
acceptable in some cases. An alternative solution is to convey other
systems like VFIO to refresh its outdated IOMMU mappings.RamDiscardManager is an existing concept (used by virtio-mem) to adjust
VFIO mappings in relation to VM page assignment. Effectively page conversion is similar to hot-removing a page in one mode and adding it
back in the other, so the similar work that needs to happen in response
to virtio-mem changes needs to happen for page conversion events. Introduce the RamDiscardManager toguest_memfdto achieve it.Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
However,
guest_memfdis not an object so it cannot directly implement
the RamDiscardManager interface.One solution is to implement the interface in HostMemoryBackend. Any
This sounds about right.
Quoted 7 lines
btw I am using this for ages:
https://github.com/aik/qemu/
commit/3663f889883d4aebbeb0e4422f7be5e357e2ee46but I am not sure if this ever saw the light of the day, did not it? (ironically I am using it as a base for encrypted DMA :) )
Yeah, we are doing the same work. I saw a solution from Michael long time ago (when there was still
a dedicated hostmem-memfd-private backend for restrictedmem/gmem)
(https://github.com/AMDESE/qemu/
commit/3bf5255fc48d648724d66410485081ace41d8ee6)For your patch, it only implement the interface for HostMemoryBackendMemfd. Maybe it is more appropriate to implement it for the parent object HostMemoryBackend, because besides the
MEMORY_BACKEND_MEMFD, other backend types likeMEMORY_BACKEND_RAMandMEMORY_BACKEND_FILEcan also be guest_memfd-backed.Quoted 5 lines
Think more about where to implement this interface. It is still uncertain to me. As I mentioned in another mail, maybe ram device memory region would be backed by
guest_memfdif we support TEE IO iommufd MMIO in future. Then a specific object is more appropriate. What's your opinion?I do not know about this. Unlike RAM, MMIO can only do "in-place conversion" and the interface to do so is not straight forward and VFIO owns MMIO anyway so the uAPI will be in iommufd, here is a gist of it:
https://github.com/aik/linux/
commit/89e45c0404fa5006b2a4de33a4d582adf1ba9831
Quoted 8 lines
Quoted 2 lines
"guest request" is a communication channel from the VM to the secure FW (AMD's "PSP") to make MMIO allow encrypted access.
It is still uncertain how to implement the private MMIO. Our assumption is the private MMIO would also create a memory region with guest_memfd-like backend. Its
mr->ramis true and should be managed by RamdDiscardManager which can skip doingDMA_MAPin VFIO'sregion_addlistener.
My current working approach is to leave it as is in QEMU and VFIO. And then avoid page state changes in KVM when private MMIO fault happens, and treat it just like normal MMIO.
And iommufd does not allow mapping VFIO MMIO anyway, I am getting:
qemu-system-x86_64: warning: IOMMU_IOAS_MAP failed: Bad address, PCI BAR?
I am really hoping for in-place memory conversion to become available sooner than later :)
Quoted 92 lines
Quoted 92 lines
Quoted 91 lines
Quoted 54 lines
Quoted 44 lines
Quoted 35 lines
Quoted 29 lines
Quoted 16 lines
Quoted 16 lines
Quoted 15 lines
Quoted 11 lines
guest_memfd-backed host memory backend can register itself in the target
MemoryRegion. However, this solution doesn't cover the scenario where aguest_memfdMemoryRegion doesn't belong to the HostMemoryBackend, e.g.
the virtual BIOS MemoryRegion.What is this virtual BIOS MemoryRegion exactly? What does it look like
in "info mtree -f"? Do we really want this memory to be DMAable?Quoted 8 lines
Quoted 6 lines
virtual BIOS shows in a separate region:
Root memory region: system 0000000000000000-000000007fffffff (prio 0, ram): pc.ram KVM
...
00000000ffc00000-00000000ffffffff (prio 0, ram): pc.bios KVMLooks like a normal MR which can be backed by
guest_memfd.Yes, virtual BIOS memory region is initialized by
memory_region_init_ram_guest_memfd()which will be backed by aguest_memfd.Quoted 5 lines
The tricky thing is, for Intel TDX (not sure about AMD SEV), the virtual
BIOS image will be loaded and then copied to private region. After that,
the loaded image will be discarded and this region become useless.Quoted 5 lines
I'd think it is loaded as "struct Rom" and then copied to the MR-
ram_guest_memfd()which does not leave MR useless - we still see "pc.bios" in the list so it is not discarded. What piece of code are you
referring to exactly?Sorry for confusion, maybe it is different between TDX and SEV-SNP for the vBIOS handling.
Quoted 9 lines
In
x86_bios_rom_init(), it initializes a guest_memfd-backed MR and loads
the vBIOS image to the shared part of theguest_memfdMR. For TDX, it
will copy the image to private region (not the vBIOSguest_memfdMR private part) and discard the shared part. So, although the memory region still exists, it seems useless.
It is different for SEV-SNP, correct? Does SEV-SNP manage the vBIOS in vBIOSguest_memfdprivate memory?Quoted 3 lines
This is what it looks like on my SNP VM (which, I suspect, is the same as yours as
hw/i386/pc.cdoes not distinguish Intel/AMD for this matter):Yes, the memory region object is created on both TDX and SEV-SNP.
Quoted 19 lines
Root memory region: system 0000000000000000-00000000000bffff (prio 0, ram): ram1 KVM gmemfd=20 00000000000c0000-00000000000dffff (prio 1, ram): pc.rom KVM gmemfd=27 00000000000e0000-000000001fffffff (prio 0, ram): ram1
@00000000000e0000 KVM gmemfd=20
...
00000000ffc00000-00000000ffffffff (prio 0, ram): pc.bios KVM
gmemfd=26So the pc.bios MR exists and in use (hence its appearance in "info mtree -f").
I added the gmemfd dumping:
system/memory.c static void mtree_print_flatview(gpointer key, · line 3453
Quoted 8 lines
system/memory.c +3
-0@@ -3446,6 +3446,9 @@ static void mtree_print_flatview(gpointer key,gpointer value, } } } + if (mr->ram_block && mr->ram_block->guest_memfd >= 0) { + qemu_printf(" gmemfd=%d", mr->ram_block->guest_memfd); + }Then I think the virtual BIOS is another case not belonging to HostMemoryBackend which convince us to implement the interface in a specific object, no?
Quoted 7 lines
Quoted 3 lines
TBH I have no idea why pc.rom and pc.bios are separate memory regions but in any case why do these 2 areas need to be treated any different than the rest of RAM? Thanks,
I think no difference. That's why I suggest implementing the RDM interface in a specific object to cover both instead of the only HostMemoryBackend.
I am still confused. Sounds like nothing prevents doing it either way, just a matter of taste, is that right? Thanks,
Signature · 1 line
Alexey
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 7 lines · Alexey Kardashevskiy
Quoted 5 lines
It is still uncertain how to implement the private MMIO. Our assumption is the private MMIO would also create a memory region with guest_memfd-like backend. Its
mr->ramis true and should be managed by RamdDiscardManager which can skip doingDMA_MAPin VFIO'sregion_addlistener.My current working approach is to leave it as is in QEMU and VFIO.
Agreed. Setting ram=true to even private MMIO sounds hackish, at least currently QEMU heavily rely on that flag for any possible direct accesses. E.g., in memory_access_is_direct().
Signature · 1 line
Peter Xu
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 10 lines · Peter Xu
Quoted 7 lines · Alexey Kardashevskiy
Quoted 5 lines
It is still uncertain how to implement the private MMIO. Our assumption is the private MMIO would also create a memory region with guest_memfd-like backend. Its
mr->ramis true and should be managed by RamdDiscardManager which can skip doingDMA_MAPin VFIO'sregion_addlistener.My current working approach is to leave it as is in QEMU and VFIO.
Agreed. Setting ram=true to even private MMIO sounds hackish, at least
The private MMIO refers to assigned MMIO, not emulated MMIO. IIUC, normal assigned MMIO is always set ram=true,
void memory_region_init_ram_device_ptr(MemoryRegion *mr,
Object *owner, const char *name, uint64_t size, void *ptr)
{
memory_region_init(mr, owner, name, size);
mr->ram = true;So I don't think ram=true is a problem here.
Thanks,
Yilun
Quoted 7 lines
currently QEMU heavily rely on that flag for any possible direct accesses. E.g., in
memory_access_is_direct().Signature · 1 line
Peter Xu
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 20 lines · Xu Yilun
Quoted 10 lines · Peter Xu
Quoted 7 lines · Alexey Kardashevskiy
Quoted 5 lines
It is still uncertain how to implement the private MMIO. Our assumption is the private MMIO would also create a memory region with guest_memfd-like backend. Its
mr->ramis true and should be managed by RamdDiscardManager which can skip doingDMA_MAPin VFIO'sregion_addlistener.My current working approach is to leave it as is in QEMU and VFIO.
Agreed. Setting ram=true to even private MMIO sounds hackish, at least
The private MMIO refers to assigned MMIO, not emulated MMIO. IIUC, normal assigned MMIO is always set ram=true,
void memory_region_init_ram_device_ptr(MemoryRegion *mr,
Object *owner, const char *name,uint64_tsize, void *ptr)
Quoted 6 lines
{ memory_region_init(mr, owner, name, size); mr->ram = true;So I don't think ram=true is a problem here.
I see. If there's always a host pointer then it looks valid. So it means the device private MMIOs are always mappable since the start?
Thanks,
Signature · 1 line
Peter Xu
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 26 lines · Peter Xu
Quoted 25 lines · Xu Yilun
Quoted 10 lines · Peter Xu
Quoted 7 lines · Alexey Kardashevskiy
Quoted 5 lines
It is still uncertain how to implement the private MMIO. Our assumption is the private MMIO would also create a memory region with guest_memfd-like backend. Its
mr->ramis true and should be managed by RamdDiscardManager which can skip doingDMA_MAPin VFIO'sregion_addlistener.My current working approach is to leave it as is in QEMU and VFIO.
Agreed. Setting ram=true to even private MMIO sounds hackish, at least
The private MMIO refers to assigned MMIO, not emulated MMIO. IIUC, normal assigned MMIO is always set ram=true,
void memory_region_init_ram_device_ptr(MemoryRegion *mr,
Object *owner, const char *name,uint64_tsize, void *ptr){ memory_region_init(mr, owner, name, size); mr->ram = true;
Quoted 4 lines
Quoted 1 line
So I don't think ram=true is a problem here.
I see. If there's always a host pointer then it looks valid. So it means the device private MMIOs are always mappable since the start?
Yes. VFIO owns the mapping and does not treat shared/private MMIO any different at the moment. Thanks,
Quoted 3 lines
Thanks,
Signature · 1 line
Alexey
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 29 lines · Alexey Kardashevskiy
Quoted 26 lines · Peter Xu
Quoted 25 lines · Xu Yilun
Quoted 10 lines · Peter Xu
Quoted 7 lines · Alexey Kardashevskiy
Quoted 5 lines
It is still uncertain how to implement the private MMIO. Our assumption is the private MMIO would also create a memory region with guest_memfd-like backend. Its
mr->ramis true and should be managed by RamdDiscardManager which can skip doingDMA_MAPin VFIO'sregion_addlistener.My current working approach is to leave it as is in QEMU and VFIO.
Agreed. Setting ram=true to even private MMIO sounds hackish, at least
The private MMIO refers to assigned MMIO, not emulated MMIO. IIUC, normal assigned MMIO is always set ram=true,
void memory_region_init_ram_device_ptr(MemoryRegion *mr,
Object *owner, const char *name,uint64_tsize, void *ptr){ memory_region_init(mr, owner, name, size); mr->ram = true;
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
So I don't think ram=true is a problem here.
I see. If there's always a host pointer then it looks valid. So it means the device private MMIOs are always mappable since the start?
Yes. VFIO owns the mapping and does not treat shared/private MMIO any different at the moment. Thanks,
mm.. I'm actually expecting private MMIO not have a host pointer, just as private memory do.
But I'm not sure why having host pointer correlates mr->ram == true.
Thanks,
Yilun
Quoted 8 lines
Quoted 4 lines
Thanks,
Signature · 1 line
Alexey
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 17 lines · Xu Yilun
Quoted 16 lines · Alexey Kardashevskiy
Quoted 13 lines · Peter Xu
Quoted 12 lines · Xu Yilun
Quoted 10 lines · Peter Xu
Quoted 7 lines · Alexey Kardashevskiy
Quoted 5 lines
It is still uncertain how to implement the private MMIO. Our assumption is the private MMIO would also create a memory region with guest_memfd-like backend. Its
mr->ramis true and should be managed by RamdDiscardManager which can skip doingDMA_MAPin VFIO'sregion_addlistener.My current working approach is to leave it as is in QEMU and VFIO.
Agreed. Setting ram=true to even private MMIO sounds hackish, at least
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
The private MMIO refers to assigned MMIO, not emulated MMIO. IIUC, normal assigned MMIO is always set ram=true,
void memory_region_init_ram_device_ptr(MemoryRegion *mr,
Object *owner, const char *name,uint64_tsize, void *ptr)
[1]
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 6 lines
{ memory_region_init(mr, owner, name, size); mr->ram = true;So I don't think ram=true is a problem here.
I see. If there's always a host pointer then it looks valid. So it means the device private MMIOs are always mappable since the start?
Quoted 7 lines
Quoted 2 lines
Yes. VFIO owns the mapping and does not treat shared/private MMIO any different at the moment. Thanks,
mm.. I'm actually expecting private MMIO not have a host pointer, just as private memory do.
But I'm not sure why having host pointer correlates
mr->ram== true.
If there is no host pointer, what would you pass into "ptr" as referenced at [1] above when creating the private MMIO memory region?
OTOH, IIUC guest private memory finally can also have a host pointer (aka, mmap()-able), it's just that even if it exists, accessing it may crash QEMU if it's private.
Thanks,
Signature · 1 line
Peter Xu
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 42 lines · Peter Xu
Quoted 17 lines · Xu Yilun
Quoted 16 lines · Alexey Kardashevskiy
Quoted 13 lines · Peter Xu
Quoted 12 lines · Xu Yilun
Quoted 10 lines · Peter Xu
Quoted 7 lines · Alexey Kardashevskiy
Quoted 5 lines
It is still uncertain how to implement the private MMIO. Our assumption is the private MMIO would also create a memory region with guest_memfd-like backend. Its
mr->ramis true and should be managed by RamdDiscardManager which can skip doingDMA_MAPin VFIO'sregion_addlistener.My current working approach is to leave it as is in QEMU and VFIO.
Agreed. Setting ram=true to even private MMIO sounds hackish, at least
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
The private MMIO refers to assigned MMIO, not emulated MMIO. IIUC, normal assigned MMIO is always set ram=true,
void memory_region_init_ram_device_ptr(MemoryRegion *mr,
Object *owner, const char *name,uint64_tsize, void *ptr)[1]
Quoted 13 lines
Quoted 12 lines
Quoted 9 lines
Quoted 6 lines
{ memory_region_init(mr, owner, name, size); mr->ram = true;So I don't think ram=true is a problem here.
I see. If there's always a host pointer then it looks valid. So it means the device private MMIOs are always mappable since the start?
Yes. VFIO owns the mapping and does not treat shared/private MMIO any different at the moment. Thanks,
Quoted 7 lines
Quoted 4 lines
mm.. I'm actually expecting private MMIO not have a host pointer, just as private memory do.
But I'm not sure why having host pointer correlates
mr->ram== true.If there is no host pointer, what would you pass into "ptr" as referenced at [1] above when creating the private MMIO memory region?
Sorry for confusion. I mean existing MMIO region use set mr->ram = true, and unmappable region (gmem) also set mr->ram = true. So don't know why mr->ram = true for private MMIO is hackish.
I think We could add another helper to create memory region for private MMIO.
Quoted 4 lines
OTOH, IIUC guest private memory finally can also have a host pointer (aka, mmap()-able), it's just that even if it exists, accessing it may crash QEMU if it's private.
Not sure if I get it correct: when memory will be converted to private, QEMU should firstly unmap the host ptr, which means host ptr doesn't alway exist.
Thanks,
Yilun
Quoted 6 lines
Thanks,
Signature · 1 line
Peter Xu
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 46 lines · Xu Yilun
Quoted 45 lines · Peter Xu
Quoted 17 lines · Xu Yilun
Quoted 16 lines · Alexey Kardashevskiy
Quoted 13 lines · Peter Xu
Quoted 12 lines · Xu Yilun
Quoted 10 lines · Peter Xu
Quoted 7 lines · Alexey Kardashevskiy
Quoted 5 lines
It is still uncertain how to implement the private MMIO. Our assumption is the private MMIO would also create a memory region with guest_memfd-like backend. Its
mr->ramis true and should be managed by RamdDiscardManager which can skip doingDMA_MAPin VFIO'sregion_addlistener.My current working approach is to leave it as is in QEMU and VFIO.
Agreed. Setting ram=true to even private MMIO sounds hackish, at least
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
The private MMIO refers to assigned MMIO, not emulated MMIO. IIUC, normal assigned MMIO is always set ram=true,
void memory_region_init_ram_device_ptr(MemoryRegion *mr,
Object *owner, const char *name,uint64_tsize, void *ptr)[1]
Quoted 16 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
{ memory_region_init(mr, owner, name, size); mr->ram = true;Quoted 7 lines
Quoted 4 lines
Quoted 1 line
So I don't think ram=true is a problem here.
I see. If there's always a host pointer then it looks valid. So it means the device private MMIOs are always mappable since the start?
Yes. VFIO owns the mapping and does not treat shared/private MMIO any different at the moment. Thanks,
mm.. I'm actually expecting private MMIO not have a host pointer, just as private memory do.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
But I'm not sure why having host pointer correlates
mr->ram== true.If there is no host pointer, what would you pass into "ptr" as referenced at [1] above when creating the private MMIO memory region?
Sorry for confusion. I mean existing MMIO region use set
mr->ram= true, and unmappable region (gmem) also setmr->ram= true. So don't know whymr->ram= true for private MMIO is hackish.
That's exactly what I had on the question in the previous email - please have a look at what QEMU does right now with memory_access_is_direct(). I'm not 100% sure it'll work if the host pointer doesn't exist.
Let's take one user of it to be explicit: flatview_write_continue_step() will try to access the ram pointer if it's direct:
if (!memory_access_is_direct(mr, true)) {
...
} else {
/* RAM case */
uint8_t *ram_ptr = qemu_ram_ptr_length(mr->ram_block, mr_addr, l,
false, true);
memmove(ram_ptr, buf, *l);
invalidate_and_set_dirty(mr, mr_addr, *l);
return MEMTX_OK;
}I don't see how QEMU could work yet if one MR set ram=true but without a host pointer..
As discussed previously, IMHO it's okay that the pointer is not accessible, but still I assume QEMU assumes the pointer at least existed for a ram=on MR. I don't know whether it's suitable to set ram=on if the pointer doesn't ever exist.
Quoted 11 lines
I think We could add another helper to create memory region for private MMIO.
Quoted 4 lines
OTOH, IIUC guest private memory finally can also have a host pointer (aka, mmap()-able), it's just that even if it exists, accessing it may crash QEMU if it's private.
Not sure if I get it correct: when memory will be converted to private, QEMU should firstly unmap the host ptr, which means host ptr doesn't alway exist.
At least current QEMU doesn't unmap it?
kvm_convert_memory() does ram_block_discard_range() indeed, but that's hole punches, not unmap. So the host pointer can always be there.
Even if we could have in-place gmemfd conversions in the future for guest mem, we should also need the host pointer to be around, in which case (per my current understand) it will even avoid hole punching but instead make the page accessible (by being able to be faulted in).
Thanks,
Signature · 1 line
Peter Xu
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 52 lines · Peter Xu
Quoted 51 lines · Xu Yilun
Quoted 49 lines · Peter Xu
Quoted 17 lines · Xu Yilun
Quoted 16 lines · Alexey Kardashevskiy
Quoted 13 lines · Peter Xu
Quoted 12 lines · Xu Yilun
Quoted 10 lines · Peter Xu
Quoted 7 lines · Alexey Kardashevskiy
Quoted 5 lines
It is still uncertain how to implement the private MMIO. Our assumption is the private MMIO would also create a memory region with guest_memfd-like backend. Its
mr->ramis true and should be managed by RamdDiscardManager which can skip doingDMA_MAPin VFIO'sregion_addlistener.My current working approach is to leave it as is in QEMU and VFIO.
Agreed. Setting ram=true to even private MMIO sounds hackish, at least
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
The private MMIO refers to assigned MMIO, not emulated MMIO. IIUC, normal assigned MMIO is always set ram=true,
void memory_region_init_ram_device_ptr(MemoryRegion *mr,
Object *owner, const char *name,uint64_tsize, void *ptr)[1]
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 6 lines
{ memory_region_init(mr, owner, name, size); mr->ram = true;So I don't think ram=true is a problem here.
I see. If there's always a host pointer then it looks valid. So it means the device private MMIOs are always mappable since the start?
Quoted 7 lines
Quoted 2 lines
Yes. VFIO owns the mapping and does not treat shared/private MMIO any different at the moment. Thanks,
mm.. I'm actually expecting private MMIO not have a host pointer, just as private memory do.
But I'm not sure why having host pointer correlates
mr->ram== true.If there is no host pointer, what would you pass into "ptr" as referenced at [1] above when creating the private MMIO memory region?
Quoted 6 lines
Quoted 3 lines
Sorry for confusion. I mean existing MMIO region use set
mr->ram= true, and unmappable region (gmem) also setmr->ram= true. So don't know whymr->ram= true for private MMIO is hackish.That's exactly what I had on the question in the previous email - please have a look at what QEMU does right now with
memory_access_is_direct().
I see memory_access_is_direct() should exclude mr->ram_device == true, which is the case for normal assigned MMIO and for private assigned MMIO. So this func is not a problem.
But I think flatview_access_allowed() is a problem that it doesn't filter out the private memory. When memory is converted to private, the result of host access can't be what you want and should be errored out. IOW, the host ptr is sometimes invalid.
Quoted 18 lines
I'm not 100% sure it'll work if the host pointer doesn't exist.
Let's take one user of it to be explicit:
flatview_write_continue_step()will try to access the ram pointer if it's direct:if (!memory_access_is_direct(mr, true)) { ... } else { /* RAM case */ uint8_t *ram_ptr = qemu_ram_ptr_length(mr->ram_block, mr_addr, l, false, true); memmove(ram_ptr, buf, *l); invalidate_and_set_dirty(mr, mr_addr, *l); return MEMTX_OK; }
Quoted 4 lines
I don't see how QEMU could work yet if one MR set ram=true but without a host pointer..
As discussed previously, IMHO it's okay that the pointer is not accessible,
Maybe I missed something in previous discussion, I assume it is OK cause no address_space_rw is happening on this host ptr when memory is private, is it?
Quoted 3 lines
but still I assume QEMU assumes the pointer at least existed for a ram=on MR. I don't know whether it's suitable to set ram=on if the pointer doesn't ever exist.
In theory, any code logic should not depends on an invalid pointer. I think a NULL pointer would be much better than a invalid pointer, at least you can check whether to access. So if you think an invalid pointer is OK, a NULL pointer should be also OK.
Thanks,
Yilun
Quoted 29 lines
Quoted 6 lines
I think We could add another helper to create memory region for private MMIO.
Quoted 1 line
Quoted 6 lines
Quoted 3 lines
OTOH, IIUC guest private memory finally can also have a host pointer (aka, mmap()-able), it's just that even if it exists, accessing it may crash QEMU if it's private.
Not sure if I get it correct: when memory will be converted to private, QEMU should firstly unmap the host ptr, which means host ptr doesn't alway exist.
At least current QEMU doesn't unmap it?
kvm_convert_memory()doesram_block_discard_range()indeed, but that's hole punches, not unmap. So the host pointer can always be there.Even if we could have in-place gmemfd conversions in the future for guest mem, we should also need the host pointer to be around, in which case (per my current understand) it will even avoid hole punching but instead make the page accessible (by being able to be faulted in).
Thanks,
Signature · 1 line
Peter Xu
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 57 lines · Xu Yilun
Quoted 56 lines · Peter Xu
Quoted 54 lines · Xu Yilun
Quoted 42 lines · Peter Xu
Quoted 17 lines · Xu Yilun
Quoted 16 lines · Alexey Kardashevskiy
Quoted 13 lines · Peter Xu
Quoted 12 lines · Xu Yilun
Quoted 10 lines · Peter Xu
Quoted 7 lines · Alexey Kardashevskiy
Quoted 5 lines
It is still uncertain how to implement the private MMIO. Our assumption is the private MMIO would also create a memory region with guest_memfd-like backend. Its
mr->ramis true and should be managed by RamdDiscardManager which can skip doingDMA_MAPin VFIO'sregion_addlistener.My current working approach is to leave it as is in QEMU and VFIO.
Agreed. Setting ram=true to even private MMIO sounds hackish, at least
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
The private MMIO refers to assigned MMIO, not emulated MMIO. IIUC, normal assigned MMIO is always set ram=true,
void memory_region_init_ram_device_ptr(MemoryRegion *mr,
Object *owner, const char *name,uint64_tsize, void *ptr)[1]
Quoted 13 lines
Quoted 12 lines
Quoted 9 lines
Quoted 6 lines
{ memory_region_init(mr, owner, name, size); mr->ram = true;So I don't think ram=true is a problem here.
I see. If there's always a host pointer then it looks valid. So it means the device private MMIOs are always mappable since the start?
Yes. VFIO owns the mapping and does not treat shared/private MMIO any different at the moment. Thanks,
Quoted 7 lines
Quoted 4 lines
mm.. I'm actually expecting private MMIO not have a host pointer, just as private memory do.
But I'm not sure why having host pointer correlates
mr->ram== true.If there is no host pointer, what would you pass into "ptr" as referenced at [1] above when creating the private MMIO memory region?
Sorry for confusion. I mean existing MMIO region use set
mr->ram= true, and unmappable region (gmem) also setmr->ram= true. So don't know whymr->ram= true for private MMIO is hackish.
Quoted 6 lines
Quoted 2 lines
That's exactly what I had on the question in the previous email - please have a look at what QEMU does right now with
memory_access_is_direct().I see
memory_access_is_direct()should excludemr->ram_device== true, which is the case for normal assigned MMIO and for private assigned MMIO. So this func is not a problem.
I'm not sure even if so.
VFIO's current use case is pretty special - it still has a host pointer, it's just that things like memcpy() might not be always suitable to be applied on MMIO mapped regions. Alex explained the rational in commit 4a2e242bbb3. I mean, the host pointer is valid even if ram_device=true in this case. Even if no direct access allowed (memcpy, etc.) it still operates on the host address using ram_device_mem_ops.
Quoted 24 lines
But I think
flatview_access_allowed()is a problem that it doesn't filter out the private memory. When memory is converted to private, the result of host access can't be what you want and should be errored out. IOW, the host ptr is sometimes invalid.Quoted 18 lines
I'm not 100% sure it'll work if the host pointer doesn't exist.
Let's take one user of it to be explicit:
flatview_write_continue_step()will try to access the ram pointer if it's direct:if (!memory_access_is_direct(mr, true)) { ... } else { /* RAM case */ uint8_t *ram_ptr = qemu_ram_ptr_length(mr->ram_block, mr_addr, l, false, true); memmove(ram_ptr, buf, *l); invalidate_and_set_dirty(mr, mr_addr, *l); return MEMTX_OK; }
Quoted 8 lines
Quoted 4 lines
I don't see how QEMU could work yet if one MR set ram=true but without a host pointer..
As discussed previously, IMHO it's okay that the pointer is not accessible,
Maybe I missed something in previous discussion, I assume it is OK cause no
address_space_rwis happening on this host ptr when memory is private, is it?
Yes, and when there is a mapped host address and someone tries to access an address that is bound to a private page, QEMU should get a SIGBUS. This code is not ready yet for gmem, but I believe it'll work like that when in-place gmem folio conversions will be ready. So far QEMU's gmem works by providing two layers of memory backends, which is IMHO pretty tricky.
Quoted 9 lines
Quoted 4 lines
but still I assume QEMU assumes the pointer at least existed for a ram=on MR. I don't know whether it's suitable to set ram=on if the pointer doesn't ever exist.
In theory, any code logic should not depends on an invalid pointer. I think a NULL pointer would be much better than a invalid pointer, at least you can check whether to access. So if you think an invalid pointer is OK, a NULL pointer should be also OK.
Definitely not suggesting to install an invalid pointer anywhere. The mapped pointer will still be valid for gmem for example, but the fault isn't. We need to differenciate two things (1) virtual address mapping, then (2) permission and accesses on the folios / pages of the mapping. Here I think it's okay if the host pointer is correctly mapped.
For your private MMIO use case, my question is if there's no host pointer to be mapped anyway, then what's the benefit to make the MR to be ram=on? Can we simply make it a normal IO memory region? The only benefit of a ram=on MR is, IMHO, being able to be accessed as RAM-like. If there's no host pointer at all, I don't yet understand how that helps private MMIO from working.
Thanks,
Signature · 1 line
Peter Xu
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 6 lines
Definitely not suggesting to install an invalid pointer anywhere. The mapped pointer will still be valid for gmem for example, but the fault isn't. We need to differenciate two things (1) virtual address mapping, then (2) permission and accesses on the folios / pages of the mapping. Here I think it's okay if the host pointer is correctly mapped.
Quoted 6 lines
For your private MMIO use case, my question is if there's no host pointer to be mapped anyway, then what's the benefit to make the MR to be ram=on? Can we simply make it a normal IO memory region? The only benefit of a ram=on MR is, IMHO, being able to be accessed as RAM-like. If there's no host pointer at all, I don't yet understand how that helps private MMIO from working.
Same here.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
In reply to Peter XuQuoted 9 lines
Definitely not suggesting to install an invalid pointer anywhere. The mapped pointer will still be valid for gmem for example, but the fault isn't. We need to differenciate two things (1) virtual address mapping, then (2) permission and accesses on the folios / pages of the mapping. Here I think it's okay if the host pointer is correctly mapped.
For your private MMIO use case, my question is if there's no host pointer to be mapped anyway, then what's the benefit to make the MR to be ram=on? Can we simply make it a normal IO memory region? The only benefit of a
The guest access to normal IO memory region would be emulated by QEMU, while private assigned MMIO requires guest direct access via Secure EPT.
Seems the existing code doesn't support guest direct access if mr->ram == false:
static void kvm_set_phys_mem(KVMMemoryListener *kml,
MemoryRegionSection *section, bool add)
{
[...]
if (!memory_region_is_ram(mr)) {
if (writable || !kvm_readonly_mem_allowed) {
return;
} else if (!mr->romd_mode) {
/* If the memory device is not in romd_mode, then we actually want
* to remove the kvm memory slot so all accesses will trap. */
add = false;
}
}
[...]
/* register the new slot */
do {
[...]
err = kvm_set_user_memory_region(kml, mem, true);
}
}Quoted 3 lines
ram=on MR is, IMHO, being able to be accessed as RAM-like. If there's no host pointer at all, I don't yet understand how that helps private MMIO from working.
I expect private MMIO not accessible from host, but accessible from guest so has kvm_userspace_memory_region2 set. That means the resolving of its PFN during EPT fault cannot depends on host pointer.
https://lore.kernel.org/all/[email protected]/
Thanks,
Yilun
Quoted 6 lines
Thanks,
Signature · 1 line
Peter Xu
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 10 lines · Xu Yilun
Quoted 9 lines
Definitely not suggesting to install an invalid pointer anywhere. The mapped pointer will still be valid for gmem for example, but the fault isn't. We need to differenciate two things (1) virtual address mapping, then (2) permission and accesses on the folios / pages of the mapping. Here I think it's okay if the host pointer is correctly mapped.
For your private MMIO use case, my question is if there's no host pointer to be mapped anyway, then what's the benefit to make the MR to be ram=on? Can we simply make it a normal IO memory region? The only benefit of a
Quoted 5 lines
The guest access to normal IO memory region would be emulated by QEMU, while private assigned MMIO requires guest direct access via Secure EPT.
Seems the existing code doesn't support guest direct access if
mr->ram== false:
Ah it's about this, ok.
I am not sure what's the best approach, but IMHO it's still better we stick with host pointer always available when ram=on. OTOH, VFIO private regions may be able to provide a special mark somewhere, just like when romd_mode was done previously as below (qemu 235e8982ad39), so that KVM should still apply these MRs even if they're not RAM.
Quoted 31 lines
static void kvm_set_phys_mem(KVMMemoryListener *kml,
MemoryRegionSection *section, bool add){ [...] if (!memory_region_is_ram(mr)) { if (writable || !kvm_readonly_mem_allowed) { return; } else if (!mr->romd_mode) { /* If the memory device is not in romd_mode, then we actually want * to remove the kvm memory slot so all accesses will trap. */ add = false; } } [...] /* register the new slot */ do { [...] err = kvm_set_user_memory_region(kml, mem, true); } }Quoted 3 lines
ram=on MR is, IMHO, being able to be accessed as RAM-like. If there's no host pointer at all, I don't yet understand how that helps private MMIO from working.
Quoted 5 lines
I expect private MMIO not accessible from host, but accessible from guest so has
kvm_userspace_memory_region2set. That means the resolving of its PFN during EPT fault cannot depends on host pointer.
I'll leave this to KVM experts, but I actually didn't follow exactly on why mmu notifier is an issue to make , as I thought that was per-mm anyway, and KVM should logically be able to skip all VFIO private MMIO regions if affected. This is a comment to this part of your commit message:
Rely on userspace mapping also means private MMIO mapping should follow userspace mapping change via mmu_notifier. This conflicts with the current design that mmu_notifier never impacts private mapping. It also makes no sense to support mmu_notifier just for private MMIO, private MMIO mapping should be fixed when CoCo-VM accepts the private MMIO, any following mapping change without guest permission should be invalid.
So I don't yet see a hard-no of reusing userspace mapping even if they're not faultable as of now - what if they can be faultable in the future? I am not sure..
OTOH, I also don't think we need KVM_SET_USER_MEMORY_REGION3 anyway.. The _REGION2 API is already smart enough to leave some reserved fields:
/* for KVM_SET_USER_MEMORY_REGION2 */
struct kvm_userspace_memory_region2 {
__u32 slot;
__u32 flags;
__u64 guest_phys_addr;
__u64 memory_size;
__u64 userspace_addr;
__u64 guest_memfd_offset;
__u32 guest_memfd;
__u32 pad1;
__u64 pad2[14];
};I think we _could_ reuse some pad*? Reusing guest_memfd field sounds error prone to me.
Not sure it could be easier if it's not guest_memfd* but fd + fd_offset since the start. But I guess when introducing _REGION2 we didn't expect MMIO private regions come so soon..
Thanks,
Signature · 1 line
Peter Xu
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 17 lines · Peter Xu
Quoted 10 lines · Xu Yilun
Quoted 9 lines
Definitely not suggesting to install an invalid pointer anywhere. The mapped pointer will still be valid for gmem for example, but the fault isn't. We need to differenciate two things (1) virtual address mapping, then (2) permission and accesses on the folios / pages of the mapping. Here I think it's okay if the host pointer is correctly mapped.
For your private MMIO use case, my question is if there's no host pointer to be mapped anyway, then what's the benefit to make the MR to be ram=on? Can we simply make it a normal IO memory region? The only benefit of a
Quoted 5 lines
The guest access to normal IO memory region would be emulated by QEMU, while private assigned MMIO requires guest direct access via Secure EPT.
Seems the existing code doesn't support guest direct access if
mr->ram== false:
Quoted 7 lines
Ah it's about this, ok.
I am not sure what's the best approach, but IMHO it's still better we stick with host pointer always available when ram=on. OTOH, VFIO private regions may be able to provide a special mark somewhere, just like when
romd_modewas done previously as below (qemu235e8982ad39), so that KVM should still apply these MRs even if they're not RAM.
I agree.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
In reply to Peter XuQuoted 17 lines · Peter Xu
Quoted 10 lines · Xu Yilun
Quoted 9 lines
Definitely not suggesting to install an invalid pointer anywhere. The mapped pointer will still be valid for gmem for example, but the fault isn't. We need to differenciate two things (1) virtual address mapping, then (2) permission and accesses on the folios / pages of the mapping. Here I think it's okay if the host pointer is correctly mapped.
For your private MMIO use case, my question is if there's no host pointer to be mapped anyway, then what's the benefit to make the MR to be ram=on? Can we simply make it a normal IO memory region? The only benefit of a
Quoted 5 lines
The guest access to normal IO memory region would be emulated by QEMU, while private assigned MMIO requires guest direct access via Secure EPT.
Seems the existing code doesn't support guest direct access if
mr->ram== false:
Quoted 7 lines
Ah it's about this, ok.
I am not sure what's the best approach, but IMHO it's still better we stick with host pointer always available when ram=on. OTOH, VFIO private regions may be able to provide a special mark somewhere, just like when
romd_modewas done previously as below (qemu235e8982ad39), so that KVM should still apply these MRs even if they're not RAM.
Also good to me.
Quoted 36 lines
Quoted 36 lines
static void kvm_set_phys_mem(KVMMemoryListener *kml,
MemoryRegionSection *section, bool add){ [...] if (!memory_region_is_ram(mr)) { if (writable || !kvm_readonly_mem_allowed) { return; } else if (!mr->romd_mode) { /* If the memory device is not in romd_mode, then we actually want * to remove the kvm memory slot so all accesses will trap. */ add = false; } } [...] /* register the new slot */ do { [...] err = kvm_set_user_memory_region(kml, mem, true); } }Quoted 3 lines
ram=on MR is, IMHO, being able to be accessed as RAM-like. If there's no host pointer at all, I don't yet understand how that helps private MMIO from working.
I expect private MMIO not accessible from host, but accessible from guest so has
kvm_userspace_memory_region2set. That means the resolving of its PFN during EPT fault cannot depends on host pointer.
Quoted 5 lines
Quoted 1 line
I'll leave this to KVM experts, but I actually didn't follow exactly on why mmu notifier is an issue to make , as I thought that was per-mm anyway, and KVM should logically be able to skip all VFIO private MMIO regions if affected.
I think this creates logical inconsistency. You builds the private MMIO EPT mapping on fault based on the HVA<->HPA mapping, but doesn't follow the HVA<->HPA mapping change. Why KVM believes the mapping on fault time but doesn't on mmu notify time?
Quoted 10 lines
This is a comment to this part of your commit message:
Rely on userspace mapping also means private MMIO mapping should follow userspace mapping change via
mmu_notifier. This conflicts with the current design thatmmu_notifiernever impacts private mapping. It also makes no sense to supportmmu_notifierjust for private MMIO, private MMIO mapping should be fixed when CoCo-VM accepts the private MMIO, any following mapping change without guest permission should be invalid.
Quoted 2 lines
So I don't yet see a hard-no of reusing userspace mapping even if they're not faultable as of now - what if they can be faultable in the future? I
The first commit of guest_memfd emphasize a lot on the benifit of decoupling KVM mapping from host mapping. My understanding is even if guest memfd can be faultable later, KVM should still work in a way without userspace mapping.
Quoted 18 lines
am not sure..
OTOH, I also don't think we need
KVM_SET_USER_MEMORY_REGION3anyway.. The_REGION2API is already smart enough to leave some reserved fields:/* for
KVM_SET_USER_MEMORY_REGION2*/struct kvm_userspace_memory_region2 { __u32 slot; __u32 flags; __u64 guest_phys_addr; __u64 memory_size; __u64 userspace_addr; __u64 guest_memfd_offset; __u32 guest_memfd; __u32 pad1; __u64 pad2[14]; };
Quoted 2 lines
I think we _could_ reuse some pad*? Reusing
guest_memfdfield sounds error prone to me.
It truly is. I'm expecting some suggestions here.
Thanks,
Yilun
Quoted 10 lines
Not sure it could be easier if it's not guest_memfd* but fd +
fd_offsetsince the start. But I guess when introducing_REGION2we didn't expect MMIO private regions come so soon..Thanks,
Signature · 1 line
Peter Xu
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 66 lines · Xu Yilun
Quoted 17 lines · Peter Xu
Quoted 10 lines · Xu Yilun
Quoted 9 lines
Definitely not suggesting to install an invalid pointer anywhere. The mapped pointer will still be valid for gmem for example, but the fault isn't. We need to differenciate two things (1) virtual address mapping, then (2) permission and accesses on the folios / pages of the mapping. Here I think it's okay if the host pointer is correctly mapped.
For your private MMIO use case, my question is if there's no host pointer to be mapped anyway, then what's the benefit to make the MR to be ram=on? Can we simply make it a normal IO memory region? The only benefit of a
Quoted 5 lines
The guest access to normal IO memory region would be emulated by QEMU, while private assigned MMIO requires guest direct access via Secure EPT.
Seems the existing code doesn't support guest direct access if
mr->ram== false:Quoted 7 lines
Ah it's about this, ok.
I am not sure what's the best approach, but IMHO it's still better we stick with host pointer always available when ram=on. OTOH, VFIO private regions may be able to provide a special mark somewhere, just like when
romd_modewas done previously as below (qemu235e8982ad39), so that KVM should still apply these MRs even if they're not RAM.Also good to me.
Quoted 38 lines
Quoted 37 lines
static void kvm_set_phys_mem(KVMMemoryListener *kml,
MemoryRegionSection *section, bool add){ [...] if (!memory_region_is_ram(mr)) { if (writable || !kvm_readonly_mem_allowed) { return; } else if (!mr->romd_mode) { /* If the memory device is not in romd_mode, then we actually want * to remove the kvm memory slot so all accesses will trap. */ add = false; } } [...] /* register the new slot */ do { [...] err = kvm_set_user_memory_region(kml, mem, true); } }Quoted 3 lines
ram=on MR is, IMHO, being able to be accessed as RAM-like. If there's no host pointer at all, I don't yet understand how that helps private MMIO from working.
I expect private MMIO not accessible from host, but accessible from guest so has
kvm_userspace_memory_region2set. That means the resolving of its PFN during EPT fault cannot depends on host pointer.
Quoted 8 lines
Quoted 3 lines
I'll leave this to KVM experts, but I actually didn't follow exactly on why mmu notifier is an issue to make , as I thought that was per-mm anyway, and KVM should logically be able to skip all VFIO private MMIO regions if affected.
I think this creates logical inconsistency. You builds the private MMIO EPT mapping on fault based on the HVA<->HPA mapping, but doesn't follow the HVA<->HPA mapping change. Why KVM believes the mapping on fault time but doesn't on mmu notify time?
IMHO as long as kvm knows it's a private MMIO and there's no mapping under it guaranteed, then KVM can safely skip those ranges to speedup the mmu notifier.
Said that, I'm not suggesting to stick with hvas if there're better alternatives. It's only about the paragraph that confused me a bit.
Quoted 11 lines
Quoted 11 lines
This is a comment to this part of your commit message:
Rely on userspace mapping also means private MMIO mapping should follow userspace mapping change via
mmu_notifier. This conflicts with the current design thatmmu_notifiernever impacts private mapping. It also makes no sense to supportmmu_notifierjust for private MMIO, private MMIO mapping should be fixed when CoCo-VM accepts the private MMIO, any following mapping change without guest permission should be invalid.
Quoted 7 lines
Quoted 2 lines
So I don't yet see a hard-no of reusing userspace mapping even if they're not faultable as of now - what if they can be faultable in the future? I
The first commit of
guest_memfdemphasize a lot on the benifit of decoupling KVM mapping from host mapping. My understanding is even if guest memfd can be faultable later, KVM should still work in a way without userspace mapping.
I could have implied to suggest using hva, not my intention. I agree fd-based API is better too in this case at least as of now.
What I'm not sure is how the whole things evolve with either gmemfd or device fd when they're used with shared and mappable pages. We can leave that for later discussion for sure.
Quoted 19 lines
Quoted 19 lines
am not sure..
OTOH, I also don't think we need
KVM_SET_USER_MEMORY_REGION3anyway.. The_REGION2API is already smart enough to leave some reserved fields:/* for
KVM_SET_USER_MEMORY_REGION2*/struct kvm_userspace_memory_region2 { __u32 slot; __u32 flags; __u64 guest_phys_addr; __u64 memory_size; __u64 userspace_addr; __u64 guest_memfd_offset; __u32 guest_memfd; __u32 pad1; __u64 pad2[14]; };
Quoted 4 lines
Quoted 2 lines
I think we _could_ reuse some pad*? Reusing
guest_memfdfield sounds error prone to me.It truly is. I'm expecting some suggestions here.
Maybe a generic fd+offset pair from pad*? I'm not sure whether at some point that could also support guest-memfd there too, after all it's easy for kvm to check whatever file->f_op it's backing, so logically kvm should allow backing a memslot with whatever file without HVA. Just my 2 cents.
Thanks,
Signature · 1 line
Peter Xu
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
In reply to Chenyi QiangQuoted 80 lines · Chenyi Qiang
Quoted 72 lines · Alexey Kardashevskiy
Quoted 60 lines · Chenyi Qiang
Quoted 48 lines · Alexey Kardashevskiy
Quoted 45 lines · Chenyi Qiang
Quoted 41 lines · Alexey Kardashevskiy
Quoted 23 lines · Chenyi Qiang
As the commit
852f0048f3("RAMBlock: makeguest_memfdrequire uncoordinated discard") highlighted, some subsystems like VFIO might disable ram block discard. However,guest_memfdrelies on the discard operation to perform page conversion between private and shared memory.
This can lead to stale IOMMU mapping issue when assigning a hardware device to a confidential VM via shared memory (unprotected memory pages). Blocking shared page discard can solve this problem, but it could cause guests to consume twice the memory with VFIO, which is not
acceptable in some cases. An alternative solution is to convey other systems like VFIO to refresh its outdated IOMMU mappings.RamDiscardManager is an existing concept (used by virtio-mem) to adjust
VFIO mappings in relation to VM page assignment. Effectively page conversion is similar to hot-removing a page in one mode and adding it
back in the other, so the similar work that needs to happen in response
to virtio-mem changes needs to happen for page conversion events. Introduce the RamDiscardManager toguest_memfdto achieve it.Quoted 4 lines
However,
guest_memfdis not an object so it cannot directly implement the RamDiscardManager interface.One solution is to implement the interface in HostMemoryBackend. Any
This sounds about right.
Quoted 7 lines
guest_memfd-backed host memory backend can register itself in the target
MemoryRegion. However, this solution doesn't cover the scenario where aguest_memfdMemoryRegion doesn't belong to the HostMemoryBackend, e.g.
the virtual BIOS MemoryRegion.What is this virtual BIOS MemoryRegion exactly? What does it look like in "info mtree -f"? Do we really want this memory to be DMAable?
Quoted 8 lines
Quoted 6 lines
virtual BIOS shows in a separate region:
Root memory region: system 0000000000000000-000000007fffffff (prio 0, ram): pc.ram KVM
...
00000000ffc00000-00000000ffffffff (prio 0, ram): pc.bios KVMLooks like a normal MR which can be backed by
guest_memfd.Quoted 8 lines
Yes, virtual BIOS memory region is initialized by
memory_region_init_ram_guest_memfd()which will be backed by aguest_memfd.The tricky thing is, for Intel TDX (not sure about AMD SEV), the virtual BIOS image will be loaded and then copied to private region. After that,
the loaded image will be discarded and this region become useless.Quoted 4 lines
I'd think it is loaded as "struct Rom" and then copied to the MR-
ram_guest_memfd()which does not leave MR useless - we still see "pc.bios" in the list so it is not discarded. What piece of code are you referring to exactly?
Quoted 8 lines
Sorry for confusion, maybe it is different between TDX and SEV-SNP for the vBIOS handling.
In
x86_bios_rom_init(), it initializes a guest_memfd-backed MR and loads the vBIOS image to the shared part of theguest_memfdMR. For TDX, it will copy the image to private region (not the vBIOSguest_memfdMR private part) and discard the shared part. So, although the memory region still exists, it seems useless.
Correct myself. After some discussion internally, I found I misunderstood the vBIOS handling in TDX. The memory region is valid. It copies the vBIOS image to the private region (vBIOS guest_memfd private part). Sorry for confusion.
Quoted 4 lines
It is different for SEV-SNP, correct? Does SEV-SNP manage the vBIOS in vBIOS
guest_memfdprivate memory?
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
In reply to Chenyi QiangQuoted 46 lines · Chenyi Qiang
Quoted 26 lines · Alexey Kardashevskiy
Quoted 18 lines · Chenyi Qiang
As the commit
852f0048f3("RAMBlock: makeguest_memfdrequire uncoordinated discard") highlighted, some subsystems like VFIO might disable ram block discard. However,guest_memfdrelies on the discard operation to perform page conversion between private and shared memory. This can lead to stale IOMMU mapping issue when assigning a hardware device to a confidential VM via shared memory (unprotected memory pages). Blocking shared page discard can solve this problem, but it could cause guests to consume twice the memory with VFIO, which is not acceptable in some cases. An alternative solution is to convey other systems like VFIO to refresh its outdated IOMMU mappings.RamDiscardManager is an existing concept (used by virtio-mem) to adjust VFIO mappings in relation to VM page assignment. Effectively page conversion is similar to hot-removing a page in one mode and adding it back in the other, so the similar work that needs to happen in response to virtio-mem changes needs to happen for page conversion events. Introduce the RamDiscardManager to
guest_memfdto achieve it.Quoted 4 lines
However,
guest_memfdis not an object so it cannot directly implement the RamDiscardManager interface.One solution is to implement the interface in HostMemoryBackend. Any
This sounds about right.
Quoted 7 lines
Quoted 4 lines
guest_memfd-backed host memory backend can register itself in the target MemoryRegion. However, this solution doesn't cover the scenario where a
guest_memfdMemoryRegion doesn't belong to the HostMemoryBackend, e.g. the virtual BIOS MemoryRegion.What is this virtual BIOS MemoryRegion exactly? What does it look like in "info mtree -f"? Do we really want this memory to be DMAable?
virtual BIOS shows in a separate region:
Root memory region: system
0000000000000000-000000007fffffff (prio 0, ram): pc.ram KVM
...
00000000ffc00000-00000000ffffffff (prio 0, ram): pc.bios KVM 0000000100000000-000000017fffffff (prio 0, ram): pc.ram
@0000000080000000 KVM
Quoted 7 lines
We also consider to implement the interface in HostMemoryBackend, but maybe implement with
guest_memfdregion is more general. We don't know if any DMAable memory would belong to HostMemoryBackend although at present it is.If it is more appropriate to implement it with HostMemoryBackend, I can change to this way.
Not sure that's the right place. Isn't it the (cc) machine that controls the state?
It's not really the memory backend, that's just the memory provider.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Thanks David for your review!
Quoted 53 lines · David Hildenbrand
Quoted 52 lines · Chenyi Qiang
Quoted 26 lines · Alexey Kardashevskiy
Quoted 18 lines · Chenyi Qiang
As the commit
852f0048f3("RAMBlock: makeguest_memfdrequire uncoordinated discard") highlighted, some subsystems like VFIO might disable ram block discard. However,guest_memfdrelies on the discard operation to perform page conversion between private and shared memory. This can lead to stale IOMMU mapping issue when assigning a hardware device to a confidential VM via shared memory (unprotected memory pages). Blocking shared page discard can solve this problem, but it could cause guests to consume twice the memory with VFIO, which is not acceptable in some cases. An alternative solution is to convey other systems like VFIO to refresh its outdated IOMMU mappings.RamDiscardManager is an existing concept (used by virtio-mem) to adjust VFIO mappings in relation to VM page assignment. Effectively page conversion is similar to hot-removing a page in one mode and adding it back in the other, so the similar work that needs to happen in response to virtio-mem changes needs to happen for page conversion events. Introduce the RamDiscardManager to
guest_memfdto achieve it.Quoted 4 lines
However,
guest_memfdis not an object so it cannot directly implement the RamDiscardManager interface.One solution is to implement the interface in HostMemoryBackend. Any
This sounds about right.
Quoted 8 lines
Quoted 5 lines
guest_memfd-backed host memory backend can register itself in the target
MemoryRegion. However, this solution doesn't cover the scenario where aguest_memfdMemoryRegion doesn't belong to the HostMemoryBackend, e.g. the virtual BIOS MemoryRegion.What is this virtual BIOS MemoryRegion exactly? What does it look like in "info mtree -f"? Do we really want this memory to be DMAable?
virtual BIOS shows in a separate region:
Root memory region: system 0000000000000000-000000007fffffff (prio 0, ram): pc.ram KVM
...
00000000ffc00000-00000000ffffffff (prio 0, ram): pc.bios KVM 0000000100000000-000000017fffffff (prio 0, ram): pc.ram @0000000080000000 KVMWe also consider to implement the interface in HostMemoryBackend, but maybe implement with
guest_memfdregion is more general. We don't know if any DMAable memory would belong to HostMemoryBackend although at present it is.
Quoted 7 lines
Quoted 2 lines
If it is more appropriate to implement it with HostMemoryBackend, I can change to this way.
Not sure that's the right place. Isn't it the (cc) machine that controls the state?
It's not really the memory backend, that's just the memory provider.
Yes, the cc machine defines the require_guest_memfd. And besides the normal memory, there's also some other memory region requires the state control. See in another thread, For example, private mmio may require the state change notification but not belong to memory backend. So I think it is still better to define a specific object to control the state.
Quoted 1 line
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
In reply to David HildenbrandQuoted 52 lines · David Hildenbrand
Quoted 51 lines · Chenyi Qiang
Quoted 26 lines · Alexey Kardashevskiy
Quoted 18 lines · Chenyi Qiang
As the commit
852f0048f3("RAMBlock: makeguest_memfdrequire uncoordinated discard") highlighted, some subsystems like VFIO might disable ram block discard. However,guest_memfdrelies on the discard operation to perform page conversion between private and shared memory. This can lead to stale IOMMU mapping issue when assigning a hardware device to a confidential VM via shared memory (unprotected memory pages). Blocking shared page discard can solve this problem, but it could cause guests to consume twice the memory with VFIO, which is not acceptable in some cases. An alternative solution is to convey other systems like VFIO to refresh its outdated IOMMU mappings.RamDiscardManager is an existing concept (used by virtio-mem) to adjust VFIO mappings in relation to VM page assignment. Effectively page conversion is similar to hot-removing a page in one mode and adding it back in the other, so the similar work that needs to happen in response to virtio-mem changes needs to happen for page conversion events. Introduce the RamDiscardManager to
guest_memfdto achieve it.Quoted 4 lines
However,
guest_memfdis not an object so it cannot directly implement the RamDiscardManager interface.One solution is to implement the interface in HostMemoryBackend. Any
This sounds about right.
Quoted 7 lines
Quoted 4 lines
guest_memfd-backed host memory backend can register itself in the target MemoryRegion. However, this solution doesn't cover the scenario where a
guest_memfdMemoryRegion doesn't belong to the HostMemoryBackend, e.g. the virtual BIOS MemoryRegion.What is this virtual BIOS MemoryRegion exactly? What does it look like in "info mtree -f"? Do we really want this memory to be DMAable?
virtual BIOS shows in a separate region:
Root memory region: system 0000000000000000-000000007fffffff (prio 0, ram): pc.ram KVM
...
00000000ffc00000-00000000ffffffff (prio 0, ram): pc.bios KVM 0000000100000000-000000017fffffff (prio 0, ram): pc.ram @0000000080000000 KVMWe also consider to implement the interface in HostMemoryBackend, but maybe implement with
guest_memfdregion is more general. We don't know if any DMAable memory would belong to HostMemoryBackend although at present it is.
Quoted 4 lines
Quoted 2 lines
If it is more appropriate to implement it with HostMemoryBackend, I can change to this way.
Not sure that's the right place. Isn't it the (cc) machine that controls the state?
KVM does, via MemoryRegion->RAMBlock->guest_memfd.
Quoted 1 line
It's not really the memory backend, that's just the memory provider.
Sorry but is not "providing memory" the purpose of "memory backend"? :)
Signature · 1 line
Alexey
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Sorry, I was traveling end of last week. I wrote a mail on the train and apparently it was swallowed somehow ...
Quoted 4 lines
Quoted 2 lines
Not sure that's the right place. Isn't it the (cc) machine that controls the state?
KVM does, via
MemoryRegion->RAMBlock->guest_memfd.
Right; I consider KVM part of the machine.
Quoted 4 lines
Quoted 2 lines
It's not really the memory backend, that's just the memory provider.
Sorry but is not "providing memory" the purpose of "memory backend"? :)
Hehe, what I wanted to say is that a memory backend is just something to create a RAMBlock. There are different ways to create a RAMBlock, even guest_memfd ones.
guest_memfd is stored per RAMBlock. I assume the state should be stored per RAMBlock as well, maybe as part of a "guest_memfd state" thing.
Now, the question is, who is the manager?
1) The machine. KVM requests the machine to perform the transition, and the machine takes care of updating the guest_memfd state and notifying any listeners.
2) The RAMBlock. Then we need some other Object to trigger that. Maybe RAMBlock would have to become an object, or we allocate separate objects.
I'm leaning towards 1), but I might be missing something.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 25 lines · David Hildenbrand
Sorry, I was traveling end of last week. I wrote a mail on the train and apparently it was swallowed somehow ...
Quoted 4 lines
Quoted 2 lines
Not sure that's the right place. Isn't it the (cc) machine that controls the state?
KVM does, via
MemoryRegion->RAMBlock->guest_memfd.Right; I consider KVM part of the machine.
Quoted 4 lines
Quoted 2 lines
It's not really the memory backend, that's just the memory provider.
Sorry but is not "providing memory" the purpose of "memory backend"? :)
Hehe, what I wanted to say is that a memory backend is just something to create a RAMBlock. There are different ways to create a RAMBlock, even
guest_memfdones.
guest_memfdis stored per RAMBlock. I assume the state should be stored per RAMBlock as well, maybe as part of a "guest_memfdstate" thing.Now, the question is, who is the manager?
Quoted 8 lines
1) The machine. KVM requests the machine to perform the transition, and the machine takes care of updating the
guest_memfdstate and notifying any listeners.2) The RAMBlock. Then we need some other Object to trigger that. Maybe RAMBlock would have to become an object, or we allocate separate objects.
I'm leaning towards 1), but I might be missing something.
A pure question: how do we process the bios gmemfds? I assume they're shared when VM starts if QEMU needs to load the bios into it, but are they always shared, or can they be converted to private later?
I wonder if it's possible (now, or in the future so it can be >2 fds) that a VM can contain multiple guest_memfds, meanwhile they request different security levels. Then it could be more future proof that such idea be managed per-fd / per-ramblock / .. rather than per-VM. For example, always shared gmemfds can avoid the manager but be treated like normal memories, while some gmemfds can still be confidential to install the manager.
But I'd confess this is pretty much whild guesses as of now.
Thanks,
Signature · 1 line
Peter Xu
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 30 lines · Peter Xu
Quoted 29 lines · David Hildenbrand
Sorry, I was traveling end of last week. I wrote a mail on the train and apparently it was swallowed somehow ...
Quoted 4 lines
Quoted 2 lines
Not sure that's the right place. Isn't it the (cc) machine that controls the state?
KVM does, via
MemoryRegion->RAMBlock->guest_memfd.Right; I consider KVM part of the machine.
Quoted 4 lines
Quoted 2 lines
It's not really the memory backend, that's just the memory provider.
Sorry but is not "providing memory" the purpose of "memory backend"? :)
Hehe, what I wanted to say is that a memory backend is just something to create a RAMBlock. There are different ways to create a RAMBlock, even
guest_memfdones.
guest_memfdis stored per RAMBlock. I assume the state should be stored per RAMBlock as well, maybe as part of a "guest_memfdstate" thing.Now, the question is, who is the manager?
1) The machine. KVM requests the machine to perform the transition, and the machine takes care of updating the
guest_memfdstate and notifying any listeners.
Quoted 8 lines
Quoted 4 lines
2) The RAMBlock. Then we need some other Object to trigger that. Maybe RAMBlock would have to become an object, or we allocate separate objects.
I'm leaning towards 1), but I might be missing something.
A pure question: how do we process the bios gmemfds? I assume they're shared when VM starts if QEMU needs to load the bios into it, but are they always shared, or can they be converted to private later?
You're probably looking for memory_region_init_ram_guest_memfd().
Quoted 7 lines
I wonder if it's possible (now, or in the future so it can be >2 fds) that a VM can contain multiple
guest_memfds, meanwhile they request different security levels. Then it could be more future proof that such idea be managed per-fd / per-ramblock / .. rather than per-VM. For example, always shared gmemfds can avoid the manager but be treated like normal memories, while some gmemfds can still be confidential to install the manager.
I think all of that is possible with whatever design we chose.
The situation is:
* guest_memfd is per RAMBlock (block->guest_memfd set in ram_block_add)
* Some RAMBlocks have a memory backend, others do not. In particular, the ones calling memory_region_init_ram_guest_memfd() do not.
So the *guest_memfd information* (fd, bitmap) really must be stored per RAMBlock.
The question *which object* implements the RamDiscardManager interface to manage the RAMBlocks that have a guest_memfd.
We either need
1) Something attached to the RAMBlock or the RAMBlock itself. This series does it via a new object attached to the RAMBlock.
2) A per-VM entity (e.g., machine, distinct management object)
In case of 1) KVM looks up the RAMBlock->object to trigger the state change. That object will inform all listeners.
In case of 2) KVM calls the per-VM entity (e.g., guest_memfd manager), which looks up the RAMBlock and triggers the state change. It will inform all listeners.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 34 lines · David Hildenbrand
Quoted 33 lines · Peter Xu
Quoted 32 lines · David Hildenbrand
Sorry, I was traveling end of last week. I wrote a mail on the train and apparently it was swallowed somehow ...
Quoted 4 lines
Quoted 2 lines
Not sure that's the right place. Isn't it the (cc) machine that controls the state?
KVM does, via
MemoryRegion->RAMBlock->guest_memfd.Right; I consider KVM part of the machine.
Quoted 4 lines
Quoted 2 lines
It's not really the memory backend, that's just the memory provider.
Sorry but is not "providing memory" the purpose of "memory backend"? :)
Hehe, what I wanted to say is that a memory backend is just something to create a RAMBlock. There are different ways to create a RAMBlock, even
guest_memfdones.
guest_memfdis stored per RAMBlock. I assume the state should be stored per RAMBlock as well, maybe as part of a "guest_memfdstate" thing.Now, the question is, who is the manager?
1) The machine. KVM requests the machine to perform the transition, and the machine takes care of updating the
guest_memfdstate and notifying any listeners.2) The RAMBlock. Then we need some other Object to trigger that. Maybe RAMBlock would have to become an object, or we allocate separate objects.
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
I'm leaning towards 1), but I might be missing something.
A pure question: how do we process the bios gmemfds? I assume they're shared when VM starts if QEMU needs to load the bios into it, but are they always shared, or can they be converted to private later?
You're probably looking for
memory_region_init_ram_guest_memfd().
Yes, but I didn't see whether such gmemfd needs conversions there. I saw an answer though from Chenyi in another email:
https://lore.kernel.org/all/[email protected]/
So I suppose the BIOS region must support private / share conversions too, just like the rest part.
Though in that case, I'm not 100% sure whether that could also be done by reusing the major guest memfd with some specific offset regions.
Quoted 29 lines
Quoted 8 lines
I wonder if it's possible (now, or in the future so it can be >2 fds) that a VM can contain multiple
guest_memfds, meanwhile they request different security levels. Then it could be more future proof that such idea be managed per-fd / per-ramblock / .. rather than per-VM. For example, always shared gmemfds can avoid the manager but be treated like normal memories, while some gmemfds can still be confidential to install the manager.I think all of that is possible with whatever design we chose.
The situation is:
*
guest_memfdis per RAMBlock (block->guest_memfdset inram_block_add)
* Some RAMBlocks have a memory backend, others do not. In particular, the ones callingmemory_region_init_ram_guest_memfd()do not.So the
*guest_memfdinformation* (fd, bitmap) really must be stored per RAMBlock.The question *which object* implements the RamDiscardManager interface to manage the RAMBlocks that have a
guest_memfd.We either need
1) Something attached to the RAMBlock or the RAMBlock itself. This series does it via a new object attached to the RAMBlock.
2) A per-VM entity (e.g., machine, distinct management object)
Quoted 6 lines
In case of 1) KVM looks up the
RAMBlock->objectto trigger the state change. That object will inform all listeners.In case of 2) KVM calls the per-VM entity (e.g.,
guest_memfdmanager), which looks up the RAMBlock and triggers the state change. It will inform all listeners.
(after I finished reading the whole discussion..)
Looks like Yilun raised another point, on how to reuse the same object for device TIO support here (conversions for device MMIOs):
https://lore.kernel.org/r/https://lore.kernel.org/all/Z4RA1vMGFECmYNXp@yilunxu-OptiPlex-7050/
Thanks,
Signature · 1 line
Peter Xu
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 86 lines · Peter Xu
Quoted 34 lines · David Hildenbrand
Quoted 33 lines · Peter Xu
Quoted 32 lines · David Hildenbrand
Sorry, I was traveling end of last week. I wrote a mail on the train and apparently it was swallowed somehow ...
Quoted 4 lines
Quoted 2 lines
Not sure that's the right place. Isn't it the (cc) machine that controls the state?
KVM does, via
MemoryRegion->RAMBlock->guest_memfd.Right; I consider KVM part of the machine.
Quoted 4 lines
Quoted 2 lines
It's not really the memory backend, that's just the memory provider.
Sorry but is not "providing memory" the purpose of "memory backend"? :)
Hehe, what I wanted to say is that a memory backend is just something to create a RAMBlock. There are different ways to create a RAMBlock, even
guest_memfdones.
guest_memfdis stored per RAMBlock. I assume the state should be stored per RAMBlock as well, maybe as part of a "guest_memfdstate" thing.Now, the question is, who is the manager?
1) The machine. KVM requests the machine to perform the transition, and the machine takes care of updating the
guest_memfdstate and notifying any listeners.2) The RAMBlock. Then we need some other Object to trigger that. Maybe RAMBlock would have to become an object, or we allocate separate objects.
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
I'm leaning towards 1), but I might be missing something.
A pure question: how do we process the bios gmemfds? I assume they're shared when VM starts if QEMU needs to load the bios into it, but are they always shared, or can they be converted to private later?
You're probably looking for
memory_region_init_ram_guest_memfd().Yes, but I didn't see whether such gmemfd needs conversions there. I saw an answer though from Chenyi in another email:
https://lore.kernel.org/all/[email protected]/
So I suppose the BIOS region must support private / share conversions too, just like the rest part.
Though in that case, I'm not 100% sure whether that could also be done by reusing the major guest memfd with some specific offset regions.
Quoted 32 lines
Quoted 8 lines
I wonder if it's possible (now, or in the future so it can be >2 fds) that a VM can contain multiple
guest_memfds, meanwhile they request different security levels. Then it could be more future proof that such idea be managed per-fd / per-ramblock / .. rather than per-VM. For example, always shared gmemfds can avoid the manager but be treated like normal memories, while some gmemfds can still be confidential to install the manager.I think all of that is possible with whatever design we chose.
The situation is:
*
guest_memfdis per RAMBlock (block->guest_memfdset inram_block_add)
* Some RAMBlocks have a memory backend, others do not. In particular, the ones callingmemory_region_init_ram_guest_memfd()do not.So the
*guest_memfdinformation* (fd, bitmap) really must be stored per RAMBlock.The question *which object* implements the RamDiscardManager interface to manage the RAMBlocks that have a
guest_memfd.We either need
1) Something attached to the RAMBlock or the RAMBlock itself. This series does it via a new object attached to the RAMBlock.
2) A per-VM entity (e.g., machine, distinct management object)In case of 1) KVM looks up the
RAMBlock->objectto trigger the state change. That object will inform all listeners.
Quoted 8 lines
Quoted 3 lines
In case of 2) KVM calls the per-VM entity (e.g.,
guest_memfdmanager), which looks up the RAMBlock and triggers the state change. It will inform all listeners.(after I finished reading the whole discussion..)
Looks like Yilun raised another point, on how to reuse the same object for device TIO support here (conversions for device MMIOs):
I don't grasp the full picture, but I suspect it would not be RAM (no RAMBlock?)?
"memory_attribute_manager" is weird if it is not memory, but memory-mapped I/O ... :)
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 2 lines · David Hildenbrand
"
memory_attribute_manager" is weird if it is not memory, but memory-mapped I/O ... :)
What you said sounds like a better name already than GuestMemfdManager in this patch.. :) To me it's ok to call MMIO as part of "memory" too, and "attribute" can describe the shareable / private (as an attribute). I'm guessing Yilun and Chenyi will figure that out..
Signature · 1 line
Peter Xu
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 6 lines · Peter Xu
Quoted 2 lines · David Hildenbrand
"
memory_attribute_manager" is weird if it is not memory, but memory-mapped I/O ... :)What you said sounds like a better name already than GuestMemfdManager in this patch..
Agreed.
:) To me it's ok to call MMIO as part of "memory" too, and
Quoted 2 lines
"attribute" can describe the shareable / private (as an attribute). I'm guessing Yilun and Chenyi will figure that out..
Yes, calling it "attributes" popped up during RFC discussion: in theory, disacard vs. populated and shared vs. private could co-exist (maybe in the future with virtio-mem or something similar).
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 3 lines · David Hildenbrand
Yes, calling it "attributes" popped up during RFC discussion: in theory, disacard vs. populated and shared vs. private could co-exist (maybe in the future with virtio-mem or something similar).
Yes makes sense. The attribute then can be easily converted into something like "user_accessible" / ... Then that can equal to "populated (plugged) && shared" ultimately.
Signature · 1 line
Peter Xu
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
In reply to Peter XuQuoted 43 lines · Peter Xu
Quoted 34 lines · David Hildenbrand
Quoted 33 lines · Peter Xu
Quoted 32 lines · David Hildenbrand
Sorry, I was traveling end of last week. I wrote a mail on the train and apparently it was swallowed somehow ...
Quoted 4 lines
Quoted 2 lines
Not sure that's the right place. Isn't it the (cc) machine that controls the state?
KVM does, via
MemoryRegion->RAMBlock->guest_memfd.Right; I consider KVM part of the machine.
Quoted 4 lines
Quoted 2 lines
It's not really the memory backend, that's just the memory provider.
Sorry but is not "providing memory" the purpose of "memory backend"? :)
Hehe, what I wanted to say is that a memory backend is just something to create a RAMBlock. There are different ways to create a RAMBlock, even
guest_memfdones.
guest_memfdis stored per RAMBlock. I assume the state should be stored per RAMBlock as well, maybe as part of a "guest_memfdstate" thing.Now, the question is, who is the manager?
1) The machine. KVM requests the machine to perform the transition, and the machine takes care of updating the
guest_memfdstate and notifying any listeners.2) The RAMBlock. Then we need some other Object to trigger that. Maybe RAMBlock would have to become an object, or we allocate separate objects.
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
I'm leaning towards 1), but I might be missing something.
A pure question: how do we process the bios gmemfds? I assume they're shared when VM starts if QEMU needs to load the bios into it, but are they always shared, or can they be converted to private later?
You're probably looking for
memory_region_init_ram_guest_memfd().
Quoted 7 lines
Yes, but I didn't see whether such gmemfd needs conversions there. I saw an answer though from Chenyi in another email:
https://lore.kernel.org/all/[email protected]/
So I suppose the BIOS region must support private / share conversions too, just like the rest part.
Yes, the BIOS region can support conversion as well. I think guest_memfd backed memory regions all follow the same sequence during setup time:
guest_memfd is shared when the guest_memfd fd is created by kvm_create_guest_memfd() in ram_block_add(), But it will sooner be converted to private just after kvm_set_user_memory_region() in kvm_set_phys_mem(). So at the boot time of cc VM, the default attribute is private. During runtime, the vBIOS can also do the conversion if it wants.
Quoted 3 lines
Though in that case, I'm not 100% sure whether that could also be done by reusing the major guest memfd with some specific offset regions.
Not sure if I understand you clearly. guest_memfd is per-Ramblock. It will have its own slot. So the vBIOS can use its own guest_memfd to get the specific offset regions.
Quoted 46 lines
Quoted 30 lines
Quoted 9 lines
I wonder if it's possible (now, or in the future so it can be >2 fds) that a VM can contain multiple
guest_memfds, meanwhile they request different security levels. Then it could be more future proof that such idea be managed per-fd / per-ramblock / .. rather than per-VM. For example, always shared gmemfds can avoid the manager but be treated like normal memories, while some gmemfds can still be confidential to install the manager.I think all of that is possible with whatever design we chose.
The situation is:
*
guest_memfdis per RAMBlock (block->guest_memfdset inram_block_add)
* Some RAMBlocks have a memory backend, others do not. In particular, the ones callingmemory_region_init_ram_guest_memfd()do not.So the
*guest_memfdinformation* (fd, bitmap) really must be stored per RAMBlock.The question *which object* implements the RamDiscardManager interface to manage the RAMBlocks that have a
guest_memfd.We either need
1) Something attached to the RAMBlock or the RAMBlock itself. This series does it via a new object attached to the RAMBlock.
2) A per-VM entity (e.g., machine, distinct management object)Quoted 6 lines
In case of 1) KVM looks up the
RAMBlock->objectto trigger the state change. That object will inform all listeners.In case of 2) KVM calls the per-VM entity (e.g.,
guest_memfdmanager), which looks up the RAMBlock and triggers the state change. It will inform all listeners.(after I finished reading the whole discussion..)
Looks like Yilun raised another point, on how to reuse the same object for device TIO support here (conversions for device MMIOs):
https://lore.kernel.org/r/https://lore.kernel.org/all/Z4RA1vMGFECmYNXp@yilunxu-OptiPlex-7050/
Thanks,
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 57 lines · Chenyi Qiang
Quoted 43 lines · Peter Xu
Quoted 34 lines · David Hildenbrand
Quoted 33 lines · Peter Xu
Quoted 32 lines · David Hildenbrand
Sorry, I was traveling end of last week. I wrote a mail on the train and apparently it was swallowed somehow ...
Quoted 4 lines
Quoted 2 lines
Not sure that's the right place. Isn't it the (cc) machine that controls the state?
KVM does, via
MemoryRegion->RAMBlock->guest_memfd.Right; I consider KVM part of the machine.
Quoted 4 lines
Quoted 2 lines
It's not really the memory backend, that's just the memory provider.
Sorry but is not "providing memory" the purpose of "memory backend"? :)
Hehe, what I wanted to say is that a memory backend is just something to create a RAMBlock. There are different ways to create a RAMBlock, even
guest_memfdones.
guest_memfdis stored per RAMBlock. I assume the state should be stored per RAMBlock as well, maybe as part of a "guest_memfdstate" thing.Now, the question is, who is the manager?
1) The machine. KVM requests the machine to perform the transition, and the machine takes care of updating the
guest_memfdstate and notifying any listeners.2) The RAMBlock. Then we need some other Object to trigger that. Maybe RAMBlock would have to become an object, or we allocate separate objects.
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
I'm leaning towards 1), but I might be missing something.
A pure question: how do we process the bios gmemfds? I assume they're shared when VM starts if QEMU needs to load the bios into it, but are they always shared, or can they be converted to private later?
You're probably looking for
memory_region_init_ram_guest_memfd().Quoted 7 lines
Yes, but I didn't see whether such gmemfd needs conversions there. I saw an answer though from Chenyi in another email:
https://lore.kernel.org/all/[email protected]/
So I suppose the BIOS region must support private / share conversions too, just like the rest part.
Yes, the BIOS region can support conversion as well. I think
guest_memfdbacked memory regions all follow the same sequence during setup time:
Quoted 6 lines
guest_memfdis shared when theguest_memfdfd is created bykvm_create_guest_memfd()inram_block_add(), But it will sooner be converted to private just afterkvm_set_user_memory_region()inkvm_set_phys_mem(). So at the boot time of cc VM, the default attribute is private. During runtime, the vBIOS can also do the conversion if it wants.
I see.
Quoted 8 lines
Quoted 4 lines
Though in that case, I'm not 100% sure whether that could also be done by reusing the major guest memfd with some specific offset regions.
Not sure if I understand you clearly.
guest_memfdis per-Ramblock. It will have its own slot. So the vBIOS can use its ownguest_memfdto get the specific offset regions.
Sorry to be confusing, please feel free to ignore my previous comment. That came from a very limited mindset that maybe one confidential VM should only have one gmemfd..
Now I see it looks like it's by design open to multiple gmemfds for each VM, then it's definitely ok that bios has its own.
Do you know why the bios needs to be convertable? I wonder whether the VM can copy it over to a private region and do whatever it wants, e.g. attest the bios being valid. However this is also more of a pure question.. and it can be offtopic to this series, so feel free to ignore.
Thanks,
Signature · 1 line
Peter Xu
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 80 lines · Peter Xu
Quoted 57 lines · Chenyi Qiang
Quoted 43 lines · Peter Xu
Quoted 34 lines · David Hildenbrand
Quoted 33 lines · Peter Xu
Quoted 32 lines · David Hildenbrand
Sorry, I was traveling end of last week. I wrote a mail on the train and apparently it was swallowed somehow ...
Quoted 4 lines
Quoted 2 lines
Not sure that's the right place. Isn't it the (cc) machine that controls the state?
KVM does, via
MemoryRegion->RAMBlock->guest_memfd.Right; I consider KVM part of the machine.
Quoted 4 lines
Quoted 2 lines
It's not really the memory backend, that's just the memory provider.
Sorry but is not "providing memory" the purpose of "memory backend"? :)
Hehe, what I wanted to say is that a memory backend is just something to create a RAMBlock. There are different ways to create a RAMBlock, even
guest_memfdones.
guest_memfdis stored per RAMBlock. I assume the state should be stored per RAMBlock as well, maybe as part of a "guest_memfdstate" thing.Now, the question is, who is the manager?
1) The machine. KVM requests the machine to perform the transition, and the machine takes care of updating the
guest_memfdstate and notifying any listeners.2) The RAMBlock. Then we need some other Object to trigger that. Maybe RAMBlock would have to become an object, or we allocate separate objects.
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
I'm leaning towards 1), but I might be missing something.
A pure question: how do we process the bios gmemfds? I assume they're shared when VM starts if QEMU needs to load the bios into it, but are they always shared, or can they be converted to private later?
You're probably looking for
memory_region_init_ram_guest_memfd().Quoted 7 lines
Yes, but I didn't see whether such gmemfd needs conversions there. I saw an answer though from Chenyi in another email:
https://lore.kernel.org/all/[email protected]/
So I suppose the BIOS region must support private / share conversions too, just like the rest part.
Yes, the BIOS region can support conversion as well. I think
guest_memfdbacked memory regions all follow the same sequence during setup time:Quoted 6 lines
guest_memfdis shared when theguest_memfdfd is created bykvm_create_guest_memfd()inram_block_add(), But it will sooner be converted to private just afterkvm_set_user_memory_region()inkvm_set_phys_mem(). So at the boot time of cc VM, the default attribute is private. During runtime, the vBIOS can also do the conversion if it wants.I see.
Quoted 8 lines
Quoted 4 lines
Though in that case, I'm not 100% sure whether that could also be done by reusing the major guest memfd with some specific offset regions.
Not sure if I understand you clearly.
guest_memfdis per-Ramblock. It will have its own slot. So the vBIOS can use its ownguest_memfdto get the specific offset regions.Sorry to be confusing, please feel free to ignore my previous comment. That came from a very limited mindset that maybe one confidential VM should only have one gmemfd..
Quoted 7 lines
Now I see it looks like it's by design open to multiple gmemfds for each VM, then it's definitely ok that bios has its own.
Do you know why the bios needs to be convertable? I wonder whether the VM can copy it over to a private region and do whatever it wants, e.g. attest the bios being valid. However this is also more of a pure question.. and it can be offtopic to this series, so feel free to ignore.
AFAIK, the vBIOS won't do conversion after it is set as private at the beginning. But in theory, the VM can do the conversion at runtime with current implementation. As for why make the vBIOS convertable, I'm also uncertain about it. Maybe convenient for managing the private/shared status by guest_memfd as it's also converted once at the beginning.
Quoted 3 lines
Thanks,
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 91 lines · Chenyi Qiang
Quoted 80 lines · Peter Xu
Quoted 57 lines · Chenyi Qiang
Quoted 43 lines · Peter Xu
Quoted 34 lines · David Hildenbrand
Quoted 33 lines · Peter Xu
Quoted 32 lines · David Hildenbrand
Sorry, I was traveling end of last week. I wrote a mail on the train and apparently it was swallowed somehow ...
Quoted 4 lines
Quoted 2 lines
Not sure that's the right place. Isn't it the (cc) machine that controls the state?
KVM does, via
MemoryRegion->RAMBlock->guest_memfd.Right; I consider KVM part of the machine.
Quoted 4 lines
Quoted 2 lines
It's not really the memory backend, that's just the memory provider.
Sorry but is not "providing memory" the purpose of "memory backend"? :)
Hehe, what I wanted to say is that a memory backend is just something to create a RAMBlock. There are different ways to create a RAMBlock, even
guest_memfdones.
guest_memfdis stored per RAMBlock. I assume the state should be stored per RAMBlock as well, maybe as part of a "guest_memfdstate" thing.Now, the question is, who is the manager?
1) The machine. KVM requests the machine to perform the transition, and the machine takes care of updating the
guest_memfdstate and notifying any listeners.2) The RAMBlock. Then we need some other Object to trigger that. Maybe RAMBlock would have to become an object, or we allocate separate objects.
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
I'm leaning towards 1), but I might be missing something.
A pure question: how do we process the bios gmemfds? I assume they're shared when VM starts if QEMU needs to load the bios into it, but are they always shared, or can they be converted to private later?
You're probably looking for
memory_region_init_ram_guest_memfd().Quoted 7 lines
Yes, but I didn't see whether such gmemfd needs conversions there. I saw an answer though from Chenyi in another email:
https://lore.kernel.org/all/[email protected]/
So I suppose the BIOS region must support private / share conversions too, just like the rest part.
Yes, the BIOS region can support conversion as well. I think
guest_memfdbacked memory regions all follow the same sequence during setup time:Quoted 6 lines
guest_memfdis shared when theguest_memfdfd is created bykvm_create_guest_memfd()inram_block_add(), But it will sooner be converted to private just afterkvm_set_user_memory_region()inkvm_set_phys_mem(). So at the boot time of cc VM, the default attribute is private. During runtime, the vBIOS can also do the conversion if it wants.I see.
Quoted 8 lines
Quoted 4 lines
Though in that case, I'm not 100% sure whether that could also be done by reusing the major guest memfd with some specific offset regions.
Not sure if I understand you clearly.
guest_memfdis per-Ramblock. It will have its own slot. So the vBIOS can use its ownguest_memfdto get the specific offset regions.Sorry to be confusing, please feel free to ignore my previous comment. That came from a very limited mindset that maybe one confidential VM should only have one gmemfd..
Quoted 7 lines
Now I see it looks like it's by design open to multiple gmemfds for each VM, then it's definitely ok that bios has its own.
Do you know why the bios needs to be convertable? I wonder whether the VM can copy it over to a private region and do whatever it wants, e.g. attest the bios being valid. However this is also more of a pure question.. and it can be offtopic to this series, so feel free to ignore.
Quoted 5 lines
AFAIK, the vBIOS won't do conversion after it is set as private at the beginning. But in theory, the VM can do the conversion at runtime with current implementation. As for why make the vBIOS convertable, I'm also uncertain about it. Maybe convenient for managing the private/shared status by
guest_memfdas it's also converted once at the beginning.
The reason is just that we are too lazy to implement a variant of guest memfd for vBIOS that is disallowed to be converted from private to shared.
Quoted 4 lines
Quoted 3 lines
Thanks,
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 92 lines · Xiaoyao Li
Quoted 91 lines · Chenyi Qiang
Quoted 87 lines · Peter Xu
Quoted 57 lines · Chenyi Qiang
Quoted 43 lines · Peter Xu
Quoted 34 lines · David Hildenbrand
Quoted 33 lines · Peter Xu
Quoted 32 lines · David Hildenbrand
Sorry, I was traveling end of last week. I wrote a mail on the train and apparently it was swallowed somehow ...
Quoted 4 lines
Quoted 2 lines
Not sure that's the right place. Isn't it the (cc) machine that controls the state?
KVM does, via
MemoryRegion->RAMBlock->guest_memfd.Right; I consider KVM part of the machine.
Quoted 4 lines
Quoted 2 lines
It's not really the memory backend, that's just the memory provider.
Sorry but is not "providing memory" the purpose of "memory backend"? :)
Hehe, what I wanted to say is that a memory backend is just something to create a RAMBlock. There are different ways to create a RAMBlock, even
guest_memfdones.
guest_memfdis stored per RAMBlock. I assume the state should be stored per RAMBlock as well, maybe as part of a "guest_memfdstate" thing.Now, the question is, who is the manager?
1) The machine. KVM requests the machine to perform the transition, and the machine takes care of updating the
guest_memfdstate and notifying any listeners.2) The RAMBlock. Then we need some other Object to trigger that. Maybe RAMBlock would have to become an object, or we allocate separate objects.
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
I'm leaning towards 1), but I might be missing something.
A pure question: how do we process the bios gmemfds? I assume they're shared when VM starts if QEMU needs to load the bios into it, but are they always shared, or can they be converted to private later?
You're probably looking for
memory_region_init_ram_guest_memfd().Quoted 7 lines
Yes, but I didn't see whether such gmemfd needs conversions there. I saw an answer though from Chenyi in another email:
https://lore.kernel.org/all/[email protected]/
So I suppose the BIOS region must support private / share conversions too, just like the rest part.
Yes, the BIOS region can support conversion as well. I think
guest_memfdbacked memory regions all follow the same sequence during setup time:Quoted 6 lines
guest_memfdis shared when theguest_memfdfd is created bykvm_create_guest_memfd()inram_block_add(), But it will sooner be converted to private just afterkvm_set_user_memory_region()inkvm_set_phys_mem(). So at the boot time of cc VM, the default attribute is private. During runtime, the vBIOS can also do the conversion if it wants.I see.
Quoted 8 lines
Quoted 4 lines
Though in that case, I'm not 100% sure whether that could also be done by reusing the major guest memfd with some specific offset regions.
Not sure if I understand you clearly.
guest_memfdis per-Ramblock. It will have its own slot. So the vBIOS can use its ownguest_memfdto get the specific offset regions.Sorry to be confusing, please feel free to ignore my previous comment. That came from a very limited mindset that maybe one confidential VM should only have one gmemfd..
Now I see it looks like it's by design open to multiple gmemfds for each VM, then it's definitely ok that bios has its own.
Do you know why the bios needs to be convertable? I wonder whether the VM can copy it over to a private region and do whatever it wants, e.g. attest the bios being valid. However this is also more of a pure question.. and it can be offtopic to this series, so feel free to ignore.
Quoted 7 lines
Quoted 5 lines
AFAIK, the vBIOS won't do conversion after it is set as private at the beginning. But in theory, the VM can do the conversion at runtime with current implementation. As for why make the vBIOS convertable, I'm also uncertain about it. Maybe convenient for managing the private/shared status by
guest_memfdas it's also converted once at the beginning.The reason is just that we are too lazy to implement a variant of guest memfd for vBIOS that is disallowed to be converted from private to shared.
What is the point in disallowing such conversion in QEMU? On AMD, a malicious HV can try converting at any time and if the guest did not ask for it, it will continue accessing those pages as private and trigger an RMP fault. But if the guest asked for conversion, then it should be no problem to convert to shared. What do I miss about TDX here? Thanks,
Quoted 6 lines
Quoted 5 lines
Quoted 4 lines
Thanks,
Signature · 1 line
Alexey
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 130 lines · Alexey Kardashevskiy
Quoted 123 lines · Xiaoyao Li
Quoted 116 lines · Chenyi Qiang
Quoted 105 lines · Peter Xu
Quoted 74 lines · Chenyi Qiang
Quoted 60 lines · Peter Xu
Quoted 47 lines · David Hildenbrand
Quoted 46 lines · Peter Xu
Quoted 44 lines · David Hildenbrand
Sorry, I was traveling end of last week. I wrote a mail on the train and
apparently it was swallowed somehow ...Quoted 5 lines
Quoted 3 lines
Not sure that's the right place. Isn't it the (cc) machine that controls
the state?KVM does, via
MemoryRegion->RAMBlock->guest_memfd.Right; I consider KVM part of the machine.
Quoted 6 lines
Quoted 3 lines
It's not really the memory backend, that's just the memory provider.
Sorry but is not "providing memory" the purpose of "memory backend"? :)
Hehe, what I wanted to say is that a memory backend is just something to
create a RAMBlock. There are different ways to create a RAMBlock, evenguest_memfdones.
guest_memfdis stored per RAMBlock. I assume the state should be stored per
RAMBlock as well, maybe as part of a "guest_memfdstate" thing.Now, the question is, who is the manager?
1) The machine. KVM requests the machine to perform the transition, and the
machine takes care of updating theguest_memfdstate and notifying any
listeners.2) The RAMBlock. Then we need some other Object to trigger that. Maybe
RAMBlock would have to become an object, or we allocate separate objects.I'm leaning towards 1), but I might be missing something.
Quoted 7 lines
Quoted 5 lines
A pure question: how do we process the bios gmemfds? I assume they're
shared when VM starts if QEMU needs to load the bios into it, but are they
always shared, or can they be converted to private later?You're probably looking for
memory_region_init_ram_guest_memfd().Yes, but I didn't see whether such gmemfd needs conversions there. I saw
an answer though from Chenyi in another email:Quoted 6 lines
https://lore.kernel.org/all/fc7194ee-ed21-4f6b-
[email protected]/So I suppose the BIOS region must support private / share conversions too,
just like the rest part.Yes, the BIOS region can support conversion as well. I think
guest_memfd
backed memory regions all follow the same sequence during setup time:Quoted 7 lines
guest_memfdis shared when theguest_memfdfd is created bykvm_create_guest_memfd()inram_block_add(), But it will sooner be converted to private just afterkvm_set_user_memory_region()inkvm_set_phys_mem(). So at the boot time of cc VM, the default attribute
is private. During runtime, the vBIOS can also do the conversion if it wants.I see.
Quoted 10 lines
Quoted 5 lines
Though in that case, I'm not 100% sure whether that could also be done by
reusing the major guest memfd with some specific offset regions.Not sure if I understand you clearly.
guest_memfdis per-Ramblock. It will have its own slot. So the vBIOS can use its ownguest_memfdto get
the specific offset regions.Sorry to be confusing, please feel free to ignore my previous comment. That came from a very limited mindset that maybe one confidential VM should
only have one gmemfd..Now I see it looks like it's by design open to multiple gmemfds for each
VM, then it's definitely ok that bios has its own.Quoted 7 lines
Do you know why the bios needs to be convertable? I wonder whether the VM
can copy it over to a private region and do whatever it wants, e.g. attest
the bios being valid. However this is also more of a pure question.. and
it can be offtopic to this series, so feel free to ignore.Quoted 5 lines
AFAIK, the vBIOS won't do conversion after it is set as private at the beginning. But in theory, the VM can do the conversion at runtime with current implementation. As for why make the vBIOS convertable, I'm also uncertain about it. Maybe convenient for managing the private/shared status by
guest_memfdas it's also converted once at the beginning.Quoted 3 lines
The reason is just that we are too lazy to implement a variant of guest memfd for vBIOS that is disallowed to be converted from private to shared.
Quoted 5 lines
What is the point in disallowing such conversion in QEMU? On AMD, a malicious HV can try converting at any time and if the guest did not ask for it, it will continue accessing those pages as private and trigger an RMP fault. But if the guest asked for conversion, then it should be no problem to convert to shared. What do I miss about TDX here? Thanks,
Re-read Peter's question, maybe I misunderstood it a little bit.
I thought Peter asked why the vBIOS need to do page conversion since it would keep private and no need to convert to shared at runtime. So it is not necessary to manage the vBIOS with guest_memfd-backed memory region as it only converts to private once during setup stage. Xiaoyao mentioned no need to implement a variant of guest_memfd to convert from private to shared. As you said, allowing such conversion won't bring security issues.
Now, I assume Peter's real question is, if we can copy the vBIOS to a private region and no need to create a specific guest_memfd-backed memory region for it?
Quoted 9 lines
Quoted 8 lines
Quoted 7 lines
Quoted 6 lines
Thanks,
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 139 lines · Chenyi Qiang
Quoted 130 lines · Alexey Kardashevskiy
Quoted 123 lines · Xiaoyao Li
Quoted 116 lines · Chenyi Qiang
Quoted 105 lines · Peter Xu
Quoted 74 lines · Chenyi Qiang
Quoted 60 lines · Peter Xu
Quoted 47 lines · David Hildenbrand
Quoted 46 lines · Peter Xu
Quoted 44 lines · David Hildenbrand
Sorry, I was traveling end of last week. I wrote a mail on the train and
apparently it was swallowed somehow ...Quoted 5 lines
Quoted 3 lines
Not sure that's the right place. Isn't it the (cc) machine that controls
the state?KVM does, via
MemoryRegion->RAMBlock->guest_memfd.Right; I consider KVM part of the machine.
Quoted 6 lines
Quoted 3 lines
It's not really the memory backend, that's just the memory provider.
Sorry but is not "providing memory" the purpose of "memory backend"? :)
Hehe, what I wanted to say is that a memory backend is just something to
create a RAMBlock. There are different ways to create a RAMBlock, evenguest_memfdones.
guest_memfdis stored per RAMBlock. I assume the state should be stored per
RAMBlock as well, maybe as part of a "guest_memfdstate" thing.Now, the question is, who is the manager?
1) The machine. KVM requests the machine to perform the transition, and the
machine takes care of updating theguest_memfdstate and notifying any
listeners.2) The RAMBlock. Then we need some other Object to trigger that. Maybe
RAMBlock would have to become an object, or we allocate separate objects.I'm leaning towards 1), but I might be missing something.
Quoted 7 lines
Quoted 5 lines
A pure question: how do we process the bios gmemfds? I assume they're
shared when VM starts if QEMU needs to load the bios into it, but are they
always shared, or can they be converted to private later?You're probably looking for
memory_region_init_ram_guest_memfd().Yes, but I didn't see whether such gmemfd needs conversions there. I saw
an answer though from Chenyi in another email:Quoted 6 lines
https://lore.kernel.org/all/fc7194ee-ed21-4f6b-
[email protected]/So I suppose the BIOS region must support private / share conversions too,
just like the rest part.Yes, the BIOS region can support conversion as well. I think
guest_memfd
backed memory regions all follow the same sequence during setup time:Quoted 7 lines
guest_memfdis shared when theguest_memfdfd is created bykvm_create_guest_memfd()inram_block_add(), But it will sooner be converted to private just afterkvm_set_user_memory_region()inkvm_set_phys_mem(). So at the boot time of cc VM, the default attribute
is private. During runtime, the vBIOS can also do the conversion if it wants.I see.
Quoted 10 lines
Quoted 5 lines
Though in that case, I'm not 100% sure whether that could also be done by
reusing the major guest memfd with some specific offset regions.Not sure if I understand you clearly.
guest_memfdis per-Ramblock. It will have its own slot. So the vBIOS can use its ownguest_memfdto get
the specific offset regions.Sorry to be confusing, please feel free to ignore my previous comment. That came from a very limited mindset that maybe one confidential VM should
only have one gmemfd..Now I see it looks like it's by design open to multiple gmemfds for each
VM, then it's definitely ok that bios has its own.Quoted 7 lines
Do you know why the bios needs to be convertable? I wonder whether the VM
can copy it over to a private region and do whatever it wants, e.g. attest
the bios being valid. However this is also more of a pure question.. and
it can be offtopic to this series, so feel free to ignore.Quoted 5 lines
AFAIK, the vBIOS won't do conversion after it is set as private at the beginning. But in theory, the VM can do the conversion at runtime with current implementation. As for why make the vBIOS convertable, I'm also uncertain about it. Maybe convenient for managing the private/shared status by
guest_memfdas it's also converted once at the beginning.Quoted 3 lines
The reason is just that we are too lazy to implement a variant of guest memfd for vBIOS that is disallowed to be converted from private to shared.
Quoted 5 lines
What is the point in disallowing such conversion in QEMU? On AMD, a malicious HV can try converting at any time and if the guest did not ask for it, it will continue accessing those pages as private and trigger an RMP fault. But if the guest asked for conversion, then it should be no problem to convert to shared. What do I miss about TDX here? Thanks,
Quoted 4 lines
Re-read Peter's question, maybe I misunderstood it a little bit.
I thought Peter asked why the vBIOS need to do page conversion since it would keep private and no need to convert to shared at runtime. So it is
I suspect there is no need to convert vBIOS but also there is no need to assume that some memory is never convertable.
Quoted 9 lines
not necessary to manage the vBIOS with guest_memfd-backed memory region as it only converts to private once during setup stage. Xiaoyao mentioned no need to implement a variant of
guest_memfdto convert from private to shared. As you said, allowing such conversion won't bring security issues.Now, I assume Peter's real question is, if we can copy the vBIOS to a private region and no need to create a specific guest_memfd-backed memory region for it?
I guess we can copy it but we have pc.bios and pc.rom in own memory regions for some reason even for legacy non-secure VMs, for ages, so it has little or nothing to do with whether vBIOS is in private or shared memory. Thanks,
Quoted 11 lines
Quoted 10 lines
Quoted 9 lines
Quoted 8 lines
Quoted 7 lines
Thanks,
Signature · 1 line
Alexey
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 3 lines · Alexey Kardashevskiy
Quoted 3 lines
Now, I assume Peter's real question is, if we can copy the vBIOS to a private region and no need to create a specific guest_memfd-backed memory region for it?
Yes.
Quoted 4 lines
I guess we can copy it but we have pc.bios and pc.rom in own memory regions for some reason even for legacy non-secure VMs, for ages, so it has little or nothing to do with whether vBIOS is in private or shared memory. Thanks,
My previous question is whether they are required to be converted to be guest-memfd backed memory regions, irrelevant of whether they're separate or not.
I think I found some answers in the commit logs here (it isn't hiding too deep; I could have tried when asking):
===8<===
commit fc7a69e177e4ba26d11fcf47b853f85115b35a11
Author: Michael Roth <[email protected]> Date: Thu May 30 06:16:40 2024 -0500
hw/i386: Add support for loading BIOS using guest_memfd
When guest_memfd is enabled, the BIOS is generally part of the initial encrypted guest image and will be accessed as private guest memory. Add the necessary changes to set up the associated RAM region with a guest_memfd backend to allow for this.
Current support centers around using -bios to load the BIOS data. Support for loading the BIOS via pflash requires additional enablement since those interfaces rely on the use of ROM memory regions which make use of the KVM_MEM_READONLY memslot flag, which is not supported for guest_memfd-backed memslots.
commit 413a67450750e0459efeffc3db3ba9759c3e381c
Author: Michael Roth <[email protected]> Date: Thu May 30 06:16:39 2024 -0500
hw/i386/sev: Use guest_memfd for legacy ROMs
Current SNP guest kernels will attempt to access these regions with with C-bit set, so guest_memfd is needed to handle that. Otherwise, kvm_convert_memory() will fail when the guest kernel tries to access it and QEMU attempts to call KVM_SET_MEMORY_ATTRIBUTES to set these ranges to private.
Whether guests should actually try to access ROM regions in this way (or need to deal with legacy ROM regions at all), is a separate issue to be addressed on kernel side, but current SNP guest kernels will exhibit this behavior and so this handling is needed to allow QEMU to continue running existing SNP guest kernels.
===8<===
So IIUC the CoCo VMs will assume they're somehow convertable memories or they'll stop working I assume, at least on some existing hardwares.
Thanks,
Signature · 1 line
Peter Xu
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
In reply to Chenyi QiangTwo trivial comments I spot:
include/sysemu/guest-memfd-manager.h line 33
Quoted 11 lines · Chenyi Qiang
include/sysemu/guest-memfd-manager.h new file +11
-0+struct GuestMemfdManager { +Object parent; + +/* Managed memory region. */ +MemoryRegion *mr; + +/* +* 1-setting of the bit represents the memory is populated (shared). +*/ +int32_t bitmap_size; +unsigned long *bitmap;
Might be clearer to name the bitmap directly as what it represents. E.g., shared_bitmap?
include/sysemu/guest-memfd-manager.h line 36
Quoted 3 lines
include/sysemu/guest-memfd-manager.h new file +3
-0+ +/* block size and alignment */ +uint64_t block_size;
Can we always fetch it from the MR/ramblock? If this is needed, better add some comment explaining why.
Quoted 4 lines
include/sysemu/guest-memfd-manager.h new file +4
-0+ +/* listeners to notify on populate/discard activity. */ + QLIST_HEAD(, RamDiscardListener) rdl_list; +};
Signature · 1 line
Peter Xu
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Thanks Peter for your review!
Quoted 17 lines · Peter Xu
Two trivial comments I spot:
include/sysemu/guest-memfd-manager.h line 33
Quoted 11 lines · Chenyi Qiang
include/sysemu/guest-memfd-manager.h new file +11
-0+struct GuestMemfdManager { +Object parent; + +/* Managed memory region. */ +MemoryRegion *mr; + +/* +* 1-setting of the bit represents the memory is populated (shared). +*/ +int32_t bitmap_size; +unsigned long *bitmap;Might be clearer to name the bitmap directly as what it represents. E.g.,
shared_bitmap?
Make sense.
Quoted 7 lines
include/sysemu/guest-memfd-manager.h line 36
Quoted 4 lines
include/sysemu/guest-memfd-manager.h new file +3
-0+ +/* block size and alignment */ +uint64_t block_size;Can we always fetch it from the MR/ramblock? If this is needed, better add some comment explaining why.
The block_size is the granularity used to track the private/shared attribute in the bitmap. It is currently hardcoded to 4K as guest_memfd may manipulate the page conversion in at least 4K size and alignment. I think It is somewhat a variable to cache the size and can avoid many getpagesize() calls.
Quoted 6 lines
Quoted 5 lines
include/sysemu/guest-memfd-manager.h new file +4
-0+ +/* listeners to notify on populate/discard activity. */ + QLIST_HEAD(, RamDiscardListener) rdl_list; +};
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 21 lines · Chenyi Qiang
Thanks Peter for your review!
Quoted 17 lines · Peter Xu
Two trivial comments I spot:
include/sysemu/guest-memfd-manager.h line 33
Quoted 11 lines · Chenyi Qiang
include/sysemu/guest-memfd-manager.h new file +11
-0+struct GuestMemfdManager { +Object parent; + +/* Managed memory region. */ +MemoryRegion *mr; + +/* +* 1-setting of the bit represents the memory is populated (shared). +*/ +int32_t bitmap_size; +unsigned long *bitmap;Might be clearer to name the bitmap directly as what it represents. E.g.,
shared_bitmap?
Quoted 2 lines
Make sense.
BTW, I was wondering if this information should be stored/linked from the RAMBlock, where we already store the guest_memdfd "int guest_memfd;".
For example, having a "struct guest_memfd_state", and either embedding it in the RAMBlock or dynamically allocating and linking it.
Alternatively, it would be such an object that we would simply link from the RAMBlock. (depending on which object will implement the manager interface)
In any case, having all guest_memfd state that belongs to a RAMBlock at a single location might be cleanest.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 33 lines · David Hildenbrand
Quoted 23 lines · Chenyi Qiang
Thanks Peter for your review!
Quoted 19 lines · Peter Xu
Two trivial comments I spot:
Quoted 8 lines · Chenyi Qiang
include/sysemu/guest-memfd-manager.h new file +8
-0+struct GuestMemfdManager { + Object parent; + + /* Managed memory region. */ + MemoryRegion *mr; + + /* + * 1-setting of the bit represents the memory is populatedQuoted 4 lines
(shared).
+ */
+int32_tbitmap_size;
+ unsigned long *bitmap;Might be clearer to name the bitmap directly as what it represents. E.g.,
shared_bitmap?Quoted 2 lines
Make sense.
BTW, I was wondering if this information should be stored/linked from the RAMBlock, where we already store the
guest_memdfd"intguest_memfd;".For example, having a "struct
guest_memfd_state", and either embedding it in the RAMBlock or dynamically allocating and linking it.
Quoted 6 lines
Alternatively, it would be such an object that we would simply link from the RAMBlock. (depending on which object will implement the manager interface)
In any case, having all
guest_memfdstate that belongs to a RAMBlock at a single location might be cleanest.
Good suggestion. Follow the design of this series, we can add link to the guest_memfd_manager object in RAMBlock.
Quoted 1 line
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 40 lines · Chenyi Qiang
Quoted 37 lines · David Hildenbrand
Quoted 23 lines · Chenyi Qiang
Thanks Peter for your review!
Quoted 19 lines · Peter Xu
Two trivial comments I spot:
Quoted 8 lines · Chenyi Qiang
include/sysemu/guest-memfd-manager.h new file +8
-0+struct GuestMemfdManager { + Object parent; + + /* Managed memory region. */ + MemoryRegion *mr; + + /* + * 1-setting of the bit represents the memory is populatedQuoted 4 lines
(shared).
+ */
+int32_tbitmap_size;
+ unsigned long *bitmap;Might be clearer to name the bitmap directly as what it represents. E.g.,
shared_bitmap?Quoted 2 lines
Make sense.
BTW, I was wondering if this information should be stored/linked from the RAMBlock, where we already store the
guest_memdfd"intguest_memfd;".For example, having a "struct
guest_memfd_state", and either embedding it in the RAMBlock or dynamically allocating and linking it.Alternatively, it would be such an object that we would simply link from the RAMBlock. (depending on which object will implement the manager interface)
Quoted 5 lines
Quoted 2 lines
In any case, having all
guest_memfdstate that belongs to a RAMBlock at a single location might be cleanest.Good suggestion. Follow the design of this series, we can add link to the
guest_memfd_managerobject in RAMBlock.
Or we'll move / link that to the RAM memory region, because that's what the object actually controls.
It starts getting a bit blury what should be part of the RAMBlock and what should be part of the "owning" RAM memory region :(
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Quoted 45 lines · David Hildenbrand
Quoted 44 lines · Chenyi Qiang
Quoted 40 lines · David Hildenbrand
Quoted 23 lines · Chenyi Qiang
Thanks Peter for your review!
Quoted 19 lines · Peter Xu
Two trivial comments I spot:
Quoted 8 lines · Chenyi Qiang
include/sysemu/guest-memfd-manager.h new file +8
-0+struct GuestMemfdManager { + Object parent; + + /* Managed memory region. */ + MemoryRegion *mr; + + /* + * 1-setting of the bit represents the memory is populatedQuoted 4 lines
(shared).
+ */
+int32_tbitmap_size;
+ unsigned long *bitmap;Might be clearer to name the bitmap directly as what it represents. E.g.,
shared_bitmap?Quoted 2 lines
Make sense.
BTW, I was wondering if this information should be stored/linked from the RAMBlock, where we already store the
guest_memdfd"intguest_memfd;".For example, having a "struct
guest_memfd_state", and either embedding it in the RAMBlock or dynamically allocating and linking it.Alternatively, it would be such an object that we would simply link from the RAMBlock. (depending on which object will implement the manager interface)
In any case, having all
guest_memfdstate that belongs to a RAMBlock at a single location might be cleanest.
Quoted 8 lines
Quoted 2 lines
Good suggestion. Follow the design of this series, we can add link to the
guest_memfd_managerobject in RAMBlock.Or we'll move / link that to the RAM memory region, because that's what the object actually controls.
It starts getting a bit blury what should be part of the RAMBlock and what should be part of the "owning" RAM memory region :(
Maybe still part of RAMBlock. I think guest_memfd state should go along with "int guest_memfd" as it is only valid when guest_memfd > 0; And guest_memfd is only valid for ram MemoryRegion.
Quoted 1 line
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
In reply to Chenyi QiangQuoted 4 lines · Chenyi Qiang
Quoted 4 lines
Quoted 3 lines
include/sysemu/guest-memfd-manager.h new file +3
-0+ +/* block size and alignment */ +uint64_t block_size;
Quoted 8 lines
Quoted 2 lines
Can we always fetch it from the MR/ramblock? If this is needed, better add some comment explaining why.
The
block_sizeis the granularity used to track the private/shared attribute in the bitmap. It is currently hardcoded to 4K asguest_memfdmay manipulate the page conversion in at least 4K size and alignment. I think It is somewhat a variable to cache the size and can avoid manygetpagesize()calls.
Though qemu does it frequently.. e.g. qemu_real_host_page_size() wraps that. So IIUC that's not a major concern, and if it's a concern maybe we can cache it globally instead.
OTOH, this is not a per-ramblock limitation either, IIUC. So maybe instead of caching it per manager, we could have memory_attr_manager_get_psize() helper (or any better name..):
memory_attr_manager_get_psize(MemoryAttrManager *mgr)
{
/* Due to limitation of ... always notify with host psize */
return qemu_real_host_page_size();
}Then in the future if necessary, switch to:
memory_attr_manager_get_psize(MemoryAttrManager *mgr)
{
return mgr->mr->ramblock->pagesize;
}Thanks,
Signature · 1 line
Peter Xu
Re: [PATCH 2/7] guest_memfd: Introduce an object to manage the guest-memfd with RamDiscardManager
Sorry I missed this mail.
Quoted 28 lines · Peter Xu
Quoted 4 lines · Chenyi Qiang
Quoted 4 lines
Quoted 3 lines
include/sysemu/guest-memfd-manager.h new file +3
-0+ +/* block size and alignment */ +uint64_t block_size;Quoted 8 lines
Quoted 2 lines
Can we always fetch it from the MR/ramblock? If this is needed, better add some comment explaining why.
The
block_sizeis the granularity used to track the private/shared attribute in the bitmap. It is currently hardcoded to 4K asguest_memfdmay manipulate the page conversion in at least 4K size and alignment. I think It is somewhat a variable to cache the size and can avoid manygetpagesize()calls.Though qemu does it frequently.. e.g.
qemu_real_host_page_size()wraps that. So IIUC that's not a major concern, and if it's a concern maybe we can cache it globally instead.OTOH, this is not a per-ramblock limitation either, IIUC. So maybe instead of caching it per manager, we could have
memory_attr_manager_get_psize()helper (or any better name..):
memory_attr_manager_get_psize(MemoryAttrManager *mgr){ /* Due to limitation of ... always notify with host psize */ return qemu_real_host_page_size(); }
Quoted 6 lines
Then in the future if necessary, switch to:
memory_attr_manager_get_psize(MemoryAttrManager *mgr){ return mgr->mr->ramblock->pagesize; }
This looks good to me. I'll change in this way.
Quoted 3 lines
Thanks,
PATCH 3/7 guest_memfd: Introduce a callback to notify the shared/private state change
In reply to Chenyi QiangIntroduce a new state_change() callback in GuestMemfdManagerClass to efficiently notify all registered RamDiscardListeners, including VFIO listeners about the memory conversion events in guest_memfd. The existing VFIO listener can dynamically DMA map/unmap the shared pages based on conversion types:
- For conversions from shared to private, the VFIO system ensures the discarding of shared mapping from the IOMMU.
- For conversions from private to shared, it triggers the population of the shared mapping into the IOMMU.
Additionally, there could be some special conversion requests:
- When a conversion request is made for a page already in the desired state, the helper simply returns success.
- For requests involving a range partially in the desired state, only the necessary segments are converted, ensuring the entire range complies with the request efficiently.
- In scenarios where a conversion request is declined by other systems, such as a failure from VFIO during notify_populate(), the helper will roll back the request, maintaining consistency.
--- include/sysemu/guest-memfd-manager.h | 3 + system/guest-memfd-manager.c | 144 +++++++++++++++++++++++++++ 2 files changed, 147 insertions(+)
include/sysemu/guest-memfd-manager.h +3 -0
@@ -41,6 +41,9 @@ struct GuestMemfdManager {struct GuestMemfdManagerClass {ObjectClass parent_class; + +int (*state_change)(GuestMemfdManager *gmm,uint64_t offset,uint64_t size, +bool shared_to_private); };#endif
system/guest-memfd-manager.c +144 -0
158 lines not on this page · raw message
Signature · 1 line
2.43.5
PATCH 4/7 KVM: Notify the state change event during shared/private conversion
In reply to Chenyi QiangIntroduce a helper to trigger the state_change() callback of the class. Once exit to userspace to convert the page from private to shared or vice versa at runtime, notify the event via the helper so that other registered subsystems like VFIO can be notified.
--- accel/kvm/kvm-all.c | 4 ++++ include/sysemu/guest-memfd-manager.h | 15 +++++++++++++++ 2 files changed, 19 insertions(+)
accel/kvm/kvm-all.c +4 -0
@@ -48,6 +48,7 @@#include "kvm-cpus.h" #include "sysemu/dirtylimit.h" #include "qemu/range.h" +#include "sysemu/guest-memfd-manager.h" #include "hw/boards.h" #include "sysemu/stats.h"
@@ -3080,6 +3081,9 @@ int kvm_convert_memory(hwaddr start, hwaddr size, bool to_private) addr =memory_region_get_ram_ptr (mr) + section.offset_within_region; rb =qemu_ram_block_from_host (addr, false, &offset); +guest_memfd_manager_state_change (GUEST_MEMFD_MANAGER (mr->rdm), offset, + size, to_private); +if (to_private) {if (rb->page_size !=qemu_real_host_page_size ()) {/*
include/sysemu/guest-memfd-manager.h +15 -0
@@ -46,4 +46,19 @@ struct GuestMemfdManagerClass { bool shared_to_private); }; +static inline int guest_memfd_manager_state_change (GuestMemfdManager *gmm,uint64_t offset, +uint64_t size,bool shared_to_private) +{ +GuestMemfdManagerClass *klass; + +g_assert (gmm); + klass =GUEST_MEMFD_MANAGER_GET_CLASS (gmm); + +if (klass->state_change) { +return klass->state_change (gmm, offset, size, shared_to_private); + } + +return 0 ; +} +#endif
Signature · 1 line
2.43.5
PATCH 5/7 memory: Register the RamDiscardManager instance upon guest_memfd creation
In reply to Chenyi QiangIntroduce the realize()/unrealize() callbacks to initialize/uninitialize the new guest_memfd_manager object and register/unregister it in the target MemoryRegion.
Guest_memfd was initially set to shared until the commit bd3bcf6962 ("kvm/memory: Make memory type private by default if it has guest memfd backend"). To align with this change, the default state in guest_memfd_manager is set to private. (The bitmap is cleared to 0). Additionally, setting the default to private can also reduce the overhead of mapping shared pages into IOMMU by VFIO during the bootup stage.
--- include/sysemu/guest-memfd-manager.h | 27 +++++++++++++++++++++++++++ system/guest-memfd-manager.c | 28 +++++++++++++++++++++++++++- system/physmem.c | 7 +++++++ 3 files changed, 61 insertions(+), 1 deletion(-)
include/sysemu/guest-memfd-manager.h +27 -0
@@ -42,6 +42,8 @@ struct GuestMemfdManager {struct GuestMemfdManagerClass {ObjectClass parent_class; +void (*realize)(GuestMemfdManager *gmm,MemoryRegion *mr,uint64_t region_size); +void (*unrealize)(GuestMemfdManager *gmm);int (*state_change)(GuestMemfdManager *gmm,uint64_t offset,uint64_t size,bool shared_to_private); };
@@ -61,4 +63,29 @@ static inline int guest_memfd_manager_state_change(GuestMemfdManager *gmm, uint6 return 0; } +static inline void guest_memfd_manager_realize (GuestMemfdManager *gmm, +MemoryRegion *mr,uint64_t region_size) +{ +GuestMemfdManagerClass *klass; + +g_assert (gmm); + klass =GUEST_MEMFD_MANAGER_GET_CLASS (gmm); + +if (klass->realize) { + klass->realize (gmm, mr, region_size); + } +} + +static inline void guest_memfd_manager_unrealize (GuestMemfdManager *gmm) +{ +GuestMemfdManagerClass *klass; + +g_assert (gmm); + klass =GUEST_MEMFD_MANAGER_GET_CLASS (gmm); + +if (klass->unrealize) { + klass->unrealize (gmm); + } +} +#endif
system/guest-memfd-manager.c +27 -1
@@ -366,6 +366,31 @@ static int guest_memfd_state_change(GuestMemfdManager *gmm, uint64_t offset,return ret; } +static void guest_memfd_manager_realizefn (GuestMemfdManager *gmm,MemoryRegion *mr, +uint64_t region_size) +{ +uint64_t bitmap_size; + + gmm->block_size =qemu_real_host_page_size (); + bitmap_size =ROUND_UP (region_size, gmm->block_size) / gmm->block_size; + + gmm->mr = mr; + gmm->bitmap_size = bitmap_size; + gmm->bitmap =bitmap_new (bitmap_size); + +memory_region_set_ram_discard_manager (gmm->mr,RAM_DISCARD_MANAGER (gmm)); +} + +static void guest_memfd_manager_unrealizefn (GuestMemfdManager *gmm) +{ +memory_region_set_ram_discard_manager (gmm->mr,NULL ); + +g_free (gmm->bitmap); + gmm->bitmap =NULL ; + gmm->bitmap_size =0 ; + gmm->mr =NULL ; +} +static void guest_memfd_manager_init (Object *obj) {GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER (obj);
@@ -375,7 +400,6 @@ static void guest_memfd_manager_init(Object *obj)static void guest_memfd_manager_finalize (Object *obj) { -g_free (GUEST_MEMFD_MANAGER (obj)->bitmap); }static void guest_memfd_manager_class_init (ObjectClass *oc,void *data)
@@ -384,6 +408,8 @@ static void guest_memfd_manager_class_init(ObjectClass *oc, void *data)RamDiscardManagerClass *rdmc =RAM_DISCARD_MANAGER_CLASS (oc); gmmc->state_change = guest_memfd_state_change; + gmmc->realize = guest_memfd_manager_realizefn; + gmmc->unrealize = guest_memfd_manager_unrealizefn; rdmc->get_min_granularity = guest_memfd_rdm_get_min_granularity; rdmc->register_listener = guest_memfd_rdm_register_listener;
system/physmem.c +7 -0
@@ -53,6 +53,7 @@#include "sysemu/hostmem.h" #include "sysemu/hw_accel.h" #include "sysemu/xen-mapcache.h" +#include "sysemu/guest-memfd-manager.h" #include "trace.h" #ifdef CONFIG_FALLOCATE_PUNCH_HOLE
@@ -1885,6 +1886,9 @@ static void ram_block_add(RAMBlock *new_block, Error **errp)qemu_mutex_unlock_ramlist (); goto out_free; } + +GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER (object_new (TYPE_GUEST_MEMFD_MANAGER )); +guest_memfd_manager_realize (gmm, new_block->mr, new_block->mr->size); } ram_size = (new_block->offset + new_block->max_length) >>TARGET_PAGE_BITS ;
@@ -2139,6 +2143,9 @@ static void reclaim_ramblock(RAMBlock *block)if (block->guest_memfd >=0 ) {close (block->guest_memfd); +GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER (block->mr->rdm); +guest_memfd_manager_unrealize (gmm); +object_unref (OBJECT (gmm));ram_block_discard_require (false); }
Signature · 1 line
2.43.5
Re: [PATCH 5/7] memory: Register the RamDiscardManager instance upon guest_memfd creation
Quoted 35 lines · Chenyi Qiang
Introduce the realize()/unrealize() callbacks to initialize/uninitialize the new
guest_memfd_managerobject and register/unregister it in the target MemoryRegion.
Guest_memfdwas initially set to shared until the commitbd3bcf6962("kvm/memory: Make memory type private by default if it has guest memfd backend"). To align with this change, the default state inguest_memfd_manageris set to private. (The bitmap is cleared to 0). Additionally, setting the default to private can also reduce the overhead of mapping shared pages into IOMMU by VFIO during the bootup stage.Signed-off-by Chenyi Qiang <[email protected]>--- include/sysemu/guest-memfd-manager.h | 27 +++++++++++++++++++++++++++ system/guest-memfd-manager.c | 28 +++++++++++++++++++++++++++- system/physmem.c | 7 +++++++ 3 files changed, 61 insertions(+), 1 deletion(-)include/sysemu/guest-memfd-manager.h +2
-0@@ -42,6 +42,8 @@ struct GuestMemfdManager {struct GuestMemfdManagerClass {ObjectClass parent_class; +void (*realize)(GuestMemfdManager *gmm,MemoryRegion *mr,uint64_t region_size); +void (*unrealize)(GuestMemfdManager *gmm);int (*state_change)(GuestMemfdManager *gmm,uint64_t offset,uint64_t size,bool shared_to_private); };@@ -61,4 +63,29 @@ static inline int guest_memfd_manager_state_change(GuestMemfdManager *gmm, uint6return 0 ; }
include/sysemu/guest-memfd-manager.h static inline int guest_memfd_manager_state_change(GuestMemfdManager *gmm, uint6 · line 75
Quoted 10 lines
include/sysemu/guest-memfd-manager.h +10
-0+static inline void guest_memfd_manager_realize (GuestMemfdManager *gmm, +MemoryRegion *mr,uint64_t region_size) +{ +GuestMemfdManagerClass *klass; + +g_assert (gmm); + klass =GUEST_MEMFD_MANAGER_GET_CLASS (gmm); + +if (klass->realize) { + klass->realize (gmm, mr, region_size);
Ditch realize() hook and call guest_memfd_manager_realizefn() directly? Not clear why these new hooks are needed.
Quoted 4 lines
include/sysemu/guest-memfd-manager.h +4
-0+ } +} + +static inline void guest_memfd_manager_unrealize (GuestMemfdManager *gmm)
include/sysemu/guest-memfd-manager.h static inline int guest_memfd_manager_state_change(GuestMemfdManager *gmm, uint6 · line 89
Quoted 10 lines
include/sysemu/guest-memfd-manager.h +10
-0+{ +GuestMemfdManagerClass *klass; + +g_assert (gmm); + klass =GUEST_MEMFD_MANAGER_GET_CLASS (gmm); + +if (klass->unrealize) { + klass->unrealize (gmm); + } +}
guest_memfd_manager_unrealizefn()?
Quoted 7 lines
include/sysemu/guest-memfd-manager.h +1
-0+#endif system/guest-memfd-manager.c +0
-0
system/guest-memfd-manager.c static int guest_memfd_state_change(GuestMemfdManager *gmm, uint64_t offset, · line 375
Quoted 10 lines
system/guest-memfd-manager.c +7
-0@@ -366,6 +366,31 @@ static int guest_memfd_state_change(GuestMemfdManager *gmm, uint64_t offset,return ret; } +static void guest_memfd_manager_realizefn (GuestMemfdManager *gmm,MemoryRegion *mr, +uint64_t region_size) +{ +uint64_t bitmap_size; + + gmm->block_size =qemu_real_host_page_size (); + bitmap_size =ROUND_UP (region_size, gmm->block_size) / gmm->block_size;
imho unaligned region_size should be an assert.
system/guest-memfd-manager.c static int guest_memfd_state_change(GuestMemfdManager *gmm, uint64_t offset, · line 382
Quoted 7 lines
system/guest-memfd-manager.c +7
-0+ + gmm->mr = mr; + gmm->bitmap_size = bitmap_size; + gmm->bitmap =bitmap_new (bitmap_size); + +memory_region_set_ram_discard_manager (gmm->mr,RAM_DISCARD_MANAGER (gmm)); +}
This belongs to 2/7.
system/guest-memfd-manager.c static int guest_memfd_state_change(GuestMemfdManager *gmm, uint64_t offset, · line 391
Quoted 9 lines
system/guest-memfd-manager.c +9
-0+ +static void guest_memfd_manager_unrealizefn (GuestMemfdManager *gmm) +{ +memory_region_set_ram_discard_manager (gmm->mr,NULL ); + +g_free (gmm->bitmap); + gmm->bitmap =NULL ; + gmm->bitmap_size =0 ; + gmm->mr =NULL ;
@gmm is being destroyed here, why bother zeroing?
system/guest-memfd-manager.c static int guest_memfd_state_change(GuestMemfdManager *gmm, uint64_t offset, · line 393
Quoted 2 lines
system/guest-memfd-manager.c +2
-0+} +
This function belongs to 2/7.
Quoted 33 lines
system/guest-memfd-manager.c +2
-1static void guest_memfd_manager_init (Object *obj) {GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER (obj);@@ -375,7 +400,6 @@ static void guest_memfd_manager_init(Object *obj)static void guest_memfd_manager_finalize (Object *obj) { -g_free (GUEST_MEMFD_MANAGER (obj)->bitmap); }static void guest_memfd_manager_class_init (ObjectClass *oc,void *data)@@ -384,6 +408,8 @@ static void guest_memfd_manager_class_init(ObjectClass *oc, void *data)RamDiscardManagerClass *rdmc =RAM_DISCARD_MANAGER_CLASS (oc); gmmc->state_change = guest_memfd_state_change; + gmmc->realize = guest_memfd_manager_realizefn; + gmmc->unrealize = guest_memfd_manager_unrealizefn; rdmc->get_min_granularity = guest_memfd_rdm_get_min_granularity; rdmc->register_listener = guest_memfd_rdm_register_listener;system/physmem.c +1
-0@@ -53,6 +53,7 @@#include "sysemu/hostmem.h" #include "sysemu/hw_accel.h" #include "sysemu/xen-mapcache.h" +#include "sysemu/guest-memfd-manager.h" #include "trace.h" #ifdef CONFIG_FALLOCATE_PUNCH_HOLE
system/physmem.c static void ram_block_add(RAMBlock *new_block, Error **errp) · line 1891
Quoted 6 lines
system/physmem.c +3
-0@@ -1885,6 +1886,9 @@ static void ram_block_add(RAMBlock *new_block, Error **errp)qemu_mutex_unlock_ramlist (); goto out_free; } + +GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER (object_new (TYPE_GUEST_MEMFD_MANAGER )); +guest_memfd_manager_realize (gmm, new_block->mr, new_block->mr->size);
Wow. Quite invasive.
system/physmem.c static void reclaim_ramblock(RAMBlock *block) · line 2148
Quoted 10 lines
system/physmem.c +3
-0} ram_size = (new_block->offset + new_block->max_length) >>TARGET_PAGE_BITS ;@@ -2139,6 +2143,9 @@ static void reclaim_ramblock(RAMBlock *block)if (block->guest_memfd >=0 ) {close (block->guest_memfd); +GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER (block->mr->rdm); +guest_memfd_manager_unrealize (gmm); +object_unref (OBJECT (gmm));
Likely don't matter but I'd do the cleanup before close() or do block->guest_memfd=-1 before the cleanup. Thanks,
Quoted 3 lines
system/physmem.c +0
-0ram_block_discard_require (false); }
Signature · 1 line
Alexey
Re: [PATCH 5/7] memory: Register the RamDiscardManager instance upon guest_memfd creation
Quoted 55 lines · Alexey Kardashevskiy
Quoted 41 lines · Chenyi Qiang
Introduce the realize()/unrealize() callbacks to initialize/uninitialize the new
guest_memfd_managerobject and register/unregister it in the target MemoryRegion.
Guest_memfdwas initially set to shared until the commitbd3bcf6962("kvm/memory: Make memory type private by default if it has guest memfd backend"). To align with this change, the default state inguest_memfd_manageris set to private. (The bitmap is cleared to 0). Additionally, setting the default to private can also reduce the overhead of mapping shared pages into IOMMU by VFIO during the bootup stage.Signed-off-by Chenyi Qiang <[email protected]>---
include/sysemu/guest-memfd-manager.h| 27 +++++++++++++++++++++++++++ system/guest-memfd-manager.c | 28 +++++++++++++++++++++++++++- system/physmem.c | 7 +++++++3 files changed, 61 insertions(+), 1 deletion(-)include/sysemu/ renamed from include/sysemu/guest-memfd-manager.h +0
-0
guest-memfd-manager.h
index9dc4e0346d..d1e7f698e8100644include/sysemu/guest-memfd-manager.h +1
-0@@ -42,6 +42,8 @@ struct GuestMemfdManager { struct GuestMemfdManagerClass { ObjectClass parent_class; + void (*realize)(GuestMemfdManager *gmm, MemoryRegion *mr,uint64_t region_size); + void (*unrealize)(GuestMemfdManager *gmm); int (*state_change)(GuestMemfdManager *gmm, uint64_t offset, uint64_t size,bool
shared_to_private); };include/sysemu/guest-memfd-manager.h +0
-0@@ -61,4 +63,29 @@ static inline intguest_memfd_manager_state_change(GuestMemfdManager *gmm, uint6 return 0;
}
+static inline void guest_memfd_manager_realize(GuestMemfdManager *gmm, + MemoryRegion *mr,uint64_tregion_size)include/sysemu/guest-memfd-manager.h static inline int guest_memfd_manager_state_change(GuestMemfdManager *gmm, uint6 · line 75
Quoted 8 lines
include/sysemu/guest-memfd-manager.h +8
-0+{ + GuestMemfdManagerClass *klass; + + g_assert(gmm); + klass = GUEST_MEMFD_MANAGER_GET_CLASS(gmm); + + if (klass->realize) { + klass->realize(gmm, mr, region_size);Ditch
realize()hook and callguest_memfd_manager_realizefn()directly? Not clear why these new hooks are needed.
Quoted 16 lines
Quoted 4 lines
+ }
include/sysemu/guest-memfd-manager.h +3
-0+} + +static inline void guest_memfd_manager_unrealize (GuestMemfdManager *gmm)include/sysemu/guest-memfd-manager.h static inline int guest_memfd_manager_state_change(GuestMemfdManager *gmm, uint6 · line 89
Quoted 10 lines
include/sysemu/guest-memfd-manager.h +10
-0+{ + GuestMemfdManagerClass *klass; + + g_assert(gmm); + klass = GUEST_MEMFD_MANAGER_GET_CLASS(gmm); + + if (klass->unrealize) { + klass->unrealize(gmm); + } +}
guest_memfd_manager_unrealizefn()?
Agree. Adding these wrappers seem unnecessary.
Quoted 20 lines
Quoted 20 lines
+
#endifsystem/guest-memfd-manager.c +6
-0@@ -366,6 +366,31 @@ static int guest_memfd_state_change(GuestMemfdManager *gmm, uint64_t offset, return ret; } +static void guest_memfd_manager_realizefn(GuestMemfdManager *gmm, MemoryRegion *mr, + uint64_t region_size) +{ + uint64_t bitmap_size; + + gmm->block_size = qemu_real_host_page_size(); + bitmap_size = ROUND_UP(region_size, gmm->block_size) / gmm-
Quoted 3 lines
system/guest-memfd-manager.c static int · line 377
Quoted 1 line
system/guest-memfd-manager.c +0
-0>block_size;imho unaligned
region_sizeshould be an assert.
There's no guarantee the region_size of the MemoryRegion is PAGE_SIZE aligned. So the ROUND_UP() is more appropriate.
Quoted 12 lines
Quoted 9 lines
system/guest-memfd-manager.c +6
-0+ + gmm->mr = mr; + gmm->bitmap_size = bitmap_size; + gmm->bitmap = bitmap_new(bitmap_size); + + memory_region_set_ram_discard_manager(gmm->mr,
RAM_DISCARD_MANAGER(gmm));
+}This belongs to 2/7.
Quoted 11 lines
system/guest-memfd-manager.c static int guest_memfd_state_change(GuestMemfdManager *gmm, uint64_t offset, · line 391
Quoted 9 lines
system/guest-memfd-manager.c +9
-0+ +static void guest_memfd_manager_unrealizefn (GuestMemfdManager *gmm) +{ + memory_region_set_ram_discard_manager(gmm->mr, NULL); + + g_free(gmm->bitmap); + gmm->bitmap = NULL; + gmm->bitmap_size = 0; + gmm->mr = NULL;@gmm is being destroyed here, why bother zeroing?
OK, will remove it.
Quoted 5 lines
system/guest-memfd-manager.c
Quoted 3 lines
system/guest-memfd-manager.c +2
-0+} +This function belongs to 2/7.
Will move both realizefn() and unrealizefn().
Quoted 39 lines
Quoted 39 lines
static void
guest_memfd_manager_init(Object *obj){
GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER(obj);system/guest-memfd-manager.c +0
-1@@ -375,7 +400,6 @@ static void guest_memfd_manager_init(Object *obj) static void guest_memfd_manager_finalize(Object *obj) { - g_free(GUEST_MEMFD_MANAGER(obj)->bitmap); }static void
guest_memfd_manager_class_init(ObjectClass *oc, void *data)system/guest-memfd-manager.c +2
-0@@ -384,6 +408,8 @@ static voidguest_memfd_manager_class_init (ObjectClass *oc, void *data) RamDiscardManagerClass *rdmc = RAM_DISCARD_MANAGER_CLASS(oc); gmmc->state_change = guest_memfd_state_change; + gmmc->realize = guest_memfd_manager_realizefn; + gmmc->unrealize = guest_memfd_manager_unrealizefn; rdmc->get_min_granularity = guest_memfd_rdm_get_min_granularity; rdmc->register_listener = guest_memfd_rdm_register_listener;system/physmem.c +4
-0@@ -53,6 +53,7 @@ #include "sysemu/hostmem.h" #include "sysemu/hw_accel.h" #include "sysemu/xen-mapcache.h" +#include "sysemu/guest-memfd-manager.h" #include "trace.h" #ifdef CONFIG_FALLOCATE_PUNCH_HOLE@@ -1885,6 +1886,9 @@ static void ram_block_add(RAMBlock *new_block, Error **errp) qemu_mutex_unlock_ramlist(); goto out_free; } + + GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER (object_new (TYPE_GUEST_MEMFD_MANAGER )); + guest_memfd_manager_realize(gmm, new_block->mr, new_block-
Quoted 3 lines
system/physmem.c static void ram_block_add(RAMBlock *new_block, · line 1894
Quoted 1 line
system/physmem.c +0
-0>mr->size);Wow. Quite invasive.
Yeah... It creates a manager object no matter whether the user wants to use shared passthru or not. We assume some fields like private/shared bitmap may also be helpful in other scenario for future usage, and if no passthru device, the listener would just return, so it is acceptable.
Quoted 19 lines
system/physmem.c static void reclaim_ramblock(RAMBlock *block) · line 2147
Quoted 10 lines
}
ram_size= (new_block->offset+new_block->max_length) >>TARGET_PAGE_BITS;system/physmem.c +3
-0@@ -2139,6 +2143,9 @@ static void reclaim_ramblock(RAMBlock *block) if (block->guest_memfd >= 0) { close(block->guest_memfd); + GuestMemfdManager *gmm = GUEST_MEMFD_MANAGER(block->mr->rdm); + guest_memfd_manager_unrealize(gmm); + object_unref(OBJECT(gmm));Likely don't matter but I'd do the cleanup before
close()or do block-Quoted 6 lines
system/physmem.c +0
-0guest_memfd=-1 before the cleanup. Thanks, ram_block_discard_require(false);}
Re: [PATCH 5/7] memory: Register the RamDiscardManager instance upon guest_memfd creation
Quoted 97 lines · Chenyi Qiang
Quoted 55 lines · Alexey Kardashevskiy
Quoted 41 lines · Chenyi Qiang
Introduce the realize()/unrealize() callbacks to initialize/uninitialize the new
guest_memfd_managerobject and register/unregister it in the target MemoryRegion.
Guest_memfdwas initially set to shared until the commitbd3bcf6962("kvm/memory: Make memory type private by default if it has guest memfd backend"). To align with this change, the default state inguest_memfd_manageris set to private. (The bitmap is cleared to 0). Additionally, setting the default to private can also reduce the overhead of mapping shared pages into IOMMU by VFIO during the bootup stage.Signed-off-by Chenyi Qiang <[email protected]>--- include/sysemu/guest-memfd-manager.h | 27 +++++++++++++++++++++++++++ system/guest-memfd-manager.c | 28 +++++++++++++++++++++++++++- system/physmem.c | 7 +++++++ 3 files changed, 61 insertions(+), 1 deletion(-)include/sysemu/ renamed from include/sysemu/guest-memfd-manager.h +0
-0
guest-memfd-manager.h
index9dc4e0346d..d1e7f698e8100644include/sysemu/guest-memfd-manager.h +1
-0@@ -42,6 +42,8 @@ struct GuestMemfdManager { struct GuestMemfdManagerClass { ObjectClass parent_class; + void (*realize)(GuestMemfdManager *gmm, MemoryRegion *mr,uint64_t region_size); + void (*unrealize)(GuestMemfdManager *gmm); int (*state_change)(GuestMemfdManager *gmm, uint64_t offset,uint64_t size, bool shared_to_private); };@@ -61,4 +63,29 @@ static inline intguest_memfd_manager_state_change(GuestMemfdManager *gmm, uint6
return 0;
}
+static inline void guest_memfd_manager_realize(GuestMemfdManager *gmm,
+ MemoryRegion *mr,uint64_tregion_size)include/sysemu/guest-memfd-manager.h static inline int guest_memfd_manager_state_change(GuestMemfdManager *gmm, uint6 · line 75
Quoted 8 lines
include/sysemu/guest-memfd-manager.h +8
-0+{ + GuestMemfdManagerClass *klass; + + g_assert(gmm); + klass = GUEST_MEMFD_MANAGER_GET_CLASS(gmm); + + if (klass->realize) { + klass->realize(gmm, mr, region_size);Ditch
realize()hook and callguest_memfd_manager_realizefn()directly? Not clear why these new hooks are needed.Quoted 16 lines
Quoted 4 lines
+ }
include/sysemu/guest-memfd-manager.h +3
-0+} + +static inline void guest_memfd_manager_unrealize (GuestMemfdManager *gmm)include/sysemu/guest-memfd-manager.h static inline int guest_memfd_manager_state_change(GuestMemfdManager *gmm, uint6 · line 89
Quoted 10 lines
include/sysemu/guest-memfd-manager.h +10
-0+{ + GuestMemfdManagerClass *klass; + + g_assert(gmm); + klass = GUEST_MEMFD_MANAGER_GET_CLASS(gmm); + + if (klass->unrealize) { + klass->unrealize(gmm); + } +}
guest_memfd_manager_unrealizefn()?Agree. Adding these wrappers seem unnecessary.
Quoted 20 lines
Quoted 20 lines
include/sysemu/guest-memfd-manager.h +1
-0+ #endifsystem/guest-memfd-manager.c +6
-0@@ -366,6 +366,31 @@ static int guest_memfd_state_change(GuestMemfdManager *gmm, uint64_t offset, return ret; } +static void guest_memfd_manager_realizefn(GuestMemfdManager *gmm, MemoryRegion *mr, + uint64_t region_size) +{ + uint64_t bitmap_size; + + gmm->block_size = qemu_real_host_page_size(); + bitmap_size = ROUND_UP(region_size, gmm->block_size) / gmm-
Quoted 6 lines
Quoted 3 lines
system/guest-memfd-manager.c static int · line 377
Quoted 1 line
system/guest-memfd-manager.c +0
-0> block_size;imho unaligned
region_sizeshould be an assert.There's no guarantee the
region_sizeof the MemoryRegion isPAGE_SIZEaligned. So theROUND_UP()is more appropriate.
It is all about DMA so the smallest you can map is PAGE_SIZE so even if you round up here, it is likely going to fail to DMA-map later anyway (or not?).
Quoted 26 lines
Quoted 11 lines
Quoted 8 lines
system/guest-memfd-manager.c +6
-0+ + gmm->mr = mr; + gmm->bitmap_size = bitmap_size; + gmm->bitmap = bitmap_new(bitmap_size); + + memory_region_set_ram_discard_manager(gmm->mr,
RAM_DISCARD_MANAGER(gmm));
+}This belongs to 2/7.
Quoted 11 lines
system/guest-memfd-manager.c static int guest_memfd_state_change(GuestMemfdManager *gmm, uint64_t offset, · line 391
Quoted 9 lines
system/guest-memfd-manager.c +9
-0+ +static void guest_memfd_manager_unrealizefn (GuestMemfdManager *gmm) +{ + memory_region_set_ram_discard_manager(gmm->mr, NULL); + + g_free(gmm->bitmap); + gmm->bitmap = NULL; + gmm->bitmap_size = 0; + gmm->mr = NULL;@gmm is being destroyed here, why bother zeroing?
OK, will remove it.
Quoted 1 line
Quoted 6 lines
Quoted 4 lines
system/guest-memfd-manager.c
Quoted 2 lines
system/guest-memfd-manager.c +2
-0+} +This function belongs to 2/7.
Will move both
realizefn()andunrealizefn().
Yes.
Quoted 39 lines
Quoted 39 lines
Quoted 39 lines
static void
guest_memfd_manager_init(Object *obj){
GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER(obj);system/guest-memfd-manager.c +2
-1@@ -375,7 +400,6 @@ static void guest_memfd_manager_init(Object *obj) static void guest_memfd_manager_finalize(Object *obj) { - g_free(GUEST_MEMFD_MANAGER(obj)->bitmap); } static void guest_memfd_manager_class_init(ObjectClass *oc, void *data)@@ -384,6 +408,8 @@ static voidguest_memfd_manager_class_init (ObjectClass *oc, void *data) RamDiscardManagerClass *rdmc = RAM_DISCARD_MANAGER_CLASS(oc); gmmc->state_change = guest_memfd_state_change; + gmmc->realize = guest_memfd_manager_realizefn; + gmmc->unrealize = guest_memfd_manager_unrealizefn; rdmc->get_min_granularity = guest_memfd_rdm_get_min_granularity; rdmc->register_listener = guest_memfd_rdm_register_listener;system/physmem.c +4
-0@@ -53,6 +53,7 @@ #include "sysemu/hostmem.h" #include "sysemu/hw_accel.h" #include "sysemu/xen-mapcache.h" +#include "sysemu/guest-memfd-manager.h" #include "trace.h" #ifdef CONFIG_FALLOCATE_PUNCH_HOLE@@ -1885,6 +1886,9 @@ static void ram_block_add(RAMBlock *new_block, Error **errp) qemu_mutex_unlock_ramlist(); goto out_free; } + + GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER (object_new (TYPE_GUEST_MEMFD_MANAGER )); + guest_memfd_manager_realize(gmm, new_block->mr, new_block-
Quoted 8 lines
Quoted 3 lines
system/physmem.c static void ram_block_add(RAMBlock *new_block, · line 1894
Quoted 1 line
system/physmem.c +0
-0> mr->size);Wow. Quite invasive.
Yeah... It creates a manager object no matter whether the user wants to
us e shared passthru or not. We assume some fields like private/shared
bitmap may also be helpful in other scenario for future usage, and if no passthru device, the listener would just return, so it is acceptable.
Explain these other scenarios in the commit log please as otherwise making this an interface of HostMemoryBackendMemfd looks way cleaner. Thanks,
Quoted 20 lines
Quoted 19 lines
system/physmem.c static void reclaim_ramblock(RAMBlock *block) · line 2147
Quoted 10 lines
}
ram_size= (new_block->offset+new_block->max_length) >>TARGET_PAGE_BITS;system/physmem.c +3
-0@@ -2139,6 +2143,9 @@ static void reclaim_ramblock(RAMBlock *block) if (block->guest_memfd >= 0) { close(block->guest_memfd); + GuestMemfdManager *gmm = GUEST_MEMFD_MANAGER(block->mr->rdm); + guest_memfd_manager_unrealize(gmm); + object_unref(OBJECT(gmm));Likely don't matter but I'd do the cleanup before
close()or do block-Quoted 6 lines
system/physmem.c +0
-0guest_memfd=-1 before the cleanup. Thanks, ram_block_discard_require(false);}
Signature · 1 line
Alexey
Re: [PATCH 5/7] memory: Register the RamDiscardManager instance upon guest_memfd creation
Quoted 108 lines · Alexey Kardashevskiy
Quoted 105 lines · Chenyi Qiang
Quoted 68 lines · Alexey Kardashevskiy
Quoted 45 lines · Chenyi Qiang
Introduce the realize()/unrealize() callbacks to initialize/ uninitialize
the newguest_memfd_managerobject and register/unregister it in the target MemoryRegion.
Guest_memfdwas initially set to shared until the commitbd3bcf6962("kvm/memory: Make memory type private by default if it has guest memfd backend"). To align with this change, the default state inguest_memfd_manageris set to private. (The bitmap is cleared to 0). Additionally, setting the default to private can also reduce the overhead of mapping shared pages into IOMMU by VFIO during the bootup stage.Signed-off-by Chenyi Qiang <[email protected]>---
include/sysemu/guest-memfd-manager.h| 27 +++++++++++++++++++++++
++++
system/guest-memfd-manager.c | 28 +++++++++++++++++++++++
++++-
system/physmem.c | 7 +++++++3 files changed, 61 insertions(+), 1 deletion(-)include/sysemu/ renamed from include/sysemu/guest-memfd-manager.h +0
-0
guest-memfd-manager.h
index9dc4e0346d..d1e7f698e8100644include/sysemu/guest-memfd-manager.h +1
-0@@ -42,6 +42,8 @@ struct GuestMemfdManager { struct GuestMemfdManagerClass { ObjectClass parent_class; + void (*realize)(GuestMemfdManager *gmm, MemoryRegion *mr,uint64_t region_size); + void (*unrealize)(GuestMemfdManager *gmm); int (*state_change)(GuestMemfdManager *gmm, uint64_t offset, uint64_t size,bool
shared_to_private); };include/sysemu/guest-memfd-manager.h +0
-0@@ -61,4 +63,29 @@ static inline intguest_memfd_manager_state_change(GuestMemfdManager *gmm, uint6 return 0;
}
+static inline void guest_memfd_manager_realize(GuestMemfdManager *gmm,
+ MemoryRegion *mr,uint64_tregion_size)include/sysemu/guest-memfd-manager.h static inline int guest_memfd_manager_state_change(GuestMemfdManager *gmm, uint6 · line 75
Quoted 8 lines
include/sysemu/guest-memfd-manager.h +8
-0+{ + GuestMemfdManagerClass *klass; + + g_assert(gmm); + klass = GUEST_MEMFD_MANAGER_GET_CLASS(gmm); + + if (klass->realize) { + klass->realize(gmm, mr, region_size);Ditch
realize()hook and callguest_memfd_manager_realizefn()directly? Not clear why these new hooks are needed.Quoted 9 lines
+ }
include/sysemu/guest-memfd-manager.h +3
-0+} + +static inline void guest_memfd_manager_unrealize(GuestMemfdManager*gmm)
include/sysemu/guest-memfd-manager.h +4
-0+{ + GuestMemfdManagerClass *klass; + + g_assert(gmm);Quoted 8 lines
include/sysemu/guest-memfd-manager.h static inline int guest_memfd_manager_state_change(GuestMemfdManager *gmm, uint6 · line 89
Quoted 6 lines
+ klass =
GUEST_MEMFD_MANAGER_GET_CLASS(gmm);include/sysemu/guest-memfd-manager.h +5
-0+ + if (klass->unrealize) { + klass->unrealize(gmm); + } +}
guest_memfd_manager_unrealizefn()?Agree. Adding these wrappers seem unnecessary.
Quoted 23 lines
Quoted 22 lines
+
#endifsystem/guest-memfd- renamed from system/guest-memfd-manager.c +0
-0
manager.c
index6601df5f3f..b6a32f0bfb100644system/guest-memfd-manager.c +6
-0@@ -366,6 +366,31 @@ static int guest_memfd_state_change(GuestMemfdManager *gmm, uint64_t offset, return ret; } +static void guest_memfd_manager_realizefn(GuestMemfdManager *gmm, MemoryRegion *mr, + uint64_t region_size) +{ + uint64_t bitmap_size; + + gmm->block_size = qemu_real_host_page_size(); + bitmap_size = ROUND_UP(region_size, gmm->block_size) / gmm- > block_size;
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
imho unaligned
region_sizeshould be an assert.There's no guarantee the
region_sizeof the MemoryRegion isPAGE_SIZEaligned. So theROUND_UP()is more appropriate.It is all about DMA so the smallest you can map is
PAGE_SIZEso even if you round up here, it is likely going to fail to DMA-map later anyway (or not?).
Checked the handling of VFIO, if the size is less than PAGE_SIZE, it will just return and won't do DMA-map.
Here is a different thing. It tries to calculate the bitmap_size. The bitmap is used to track the private/shared status of the page. So if the size is less than PAGE_SIZE, we still use the one bit to track this small-size range.
Quoted 82 lines
Quoted 28 lines
Quoted 13 lines
Quoted 10 lines
system/guest-memfd-manager.c +6
-0+ + gmm->mr = mr; + gmm->bitmap_size = bitmap_size; + gmm->bitmap = bitmap_new(bitmap_size); + + memory_region_set_ram_discard_manager(gmm->mr,
RAM_DISCARD_MANAGER(gmm));
+}This belongs to 2/7.
Quoted 11 lines
system/guest-memfd-manager.c static int guest_memfd_state_change(GuestMemfdManager *gmm, uint64_t offset, · line 391
Quoted 9 lines
system/guest-memfd-manager.c +9
-0+ +static void guest_memfd_manager_unrealizefn (GuestMemfdManager *gmm) +{ + memory_region_set_ram_discard_manager(gmm->mr, NULL); + + g_free(gmm->bitmap); + gmm->bitmap = NULL; + gmm->bitmap_size = 0; + gmm->mr = NULL;@gmm is being destroyed here, why bother zeroing?
OK, will remove it.
Quoted 1 line
Quoted 6 lines
Quoted 4 lines
system/guest-memfd-manager.c
Quoted 2 lines
system/guest-memfd-manager.c +2
-0+} +This function belongs to 2/7.
Will move both
realizefn()andunrealizefn().Yes.
Quoted 44 lines
Quoted 43 lines
Quoted 32 lines
static void
guest_memfd_manager_init(Object *obj){
GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER(obj);system/guest-memfd-manager.c +0
-1@@ -375,7 +400,6 @@ static void guest_memfd_manager_init(Object *obj) static void guest_memfd_manager_finalize(Object *obj) { - g_free(GUEST_MEMFD_MANAGER(obj)->bitmap); }static void
guest_memfd_manager_class_init(ObjectClass *oc, void *data)system/guest-memfd-manager.c +2
-0@@ -384,6 +408,8 @@ static voidguest_memfd_manager_class_init (ObjectClass *oc, void *data) RamDiscardManagerClass *rdmc = RAM_DISCARD_MANAGER_CLASS(oc); gmmc->state_change = guest_memfd_state_change; + gmmc->realize = guest_memfd_manager_realizefn; + gmmc->unrealize = guest_memfd_manager_unrealizefn;
rdmc->get_min_granularity=guest_memfd_rdm_get_min_granularity;rdmc->register_listener=guest_memfd_rdm_register_listener;system/physmem.c +1
-0@@ -53,6 +53,7 @@ #include "sysemu/hostmem.h" #include "sysemu/hw_accel.h" #include "sysemu/xen-mapcache.h" +#include "sysemu/guest-memfd-manager.h" #include "trace.h" #ifdef CONFIG_FALLOCATE_PUNCH_HOLEsystem/physmem.c static void ram_block_add(RAMBlock *new_block, · line 1894
Quoted 9 lines
system/physmem.c +3
-0@@ -1885,6 +1886,9 @@ static void ram_block_add(RAMBlock *new_block, Error **errp) qemu_mutex_unlock_ramlist(); goto out_free; } + + GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER (object_new (TYPE_GUEST_MEMFD_MANAGER )); + guest_memfd_manager_realize(gmm, new_block->mr, new_block- > mr->size);Wow. Quite invasive.
Quoted 8 lines
Quoted 4 lines
Yeah... It creates a manager object no matter whether the user wants to us e shared passthru or not. We assume some fields like private/shared bitmap may also be helpful in other scenario for future usage, and if no passthru device, the listener would just return, so it is acceptable.
Explain these other scenarios in the commit log please as otherwise making this an interface of HostMemoryBackendMemfd looks way cleaner. Thanks,
Thanks for the suggestion. Until now, I think making this an interface of HostMemoryBackend is cleaner. The potential future usage for non-HostMemoryBackend guest_memfd-backed memory region I can think of is the the TEE I/O for iommufd P2P support? when it tries to initialize RAM device memory region with the attribute of shared/private. But I think it would be a long term story and we are not sure what it will be like in future.
Quoted 22 lines
Quoted 21 lines
Quoted 20 lines
system/physmem.c static void reclaim_ramblock(RAMBlock *block) · line 2147
Quoted 11 lines
}
ram_size= (new_block->offset+new_block->max_length) >>TARGET_PAGE_BITS;system/physmem.c +3
-0@@ -2139,6 +2143,9 @@ static void reclaim_ramblock(RAMBlock *block) if (block->guest_memfd >= 0) { close(block->guest_memfd); + GuestMemfdManager *gmm = GUEST_MEMFD_MANAGER(block->mr->rdm); + guest_memfd_manager_unrealize(gmm); + object_unref(OBJECT(gmm));Likely don't matter but I'd do the cleanup before
close()or do block-Quoted 6 lines
system/physmem.c +0
-0guest_memfd=-1 before the cleanup. Thanks, ram_block_discard_require(false);}
Re: [PATCH 5/7] memory: Register the RamDiscardManager instance upon guest_memfd creation
Quoted 219 lines · Chenyi Qiang
Quoted 108 lines · Alexey Kardashevskiy
Quoted 105 lines · Chenyi Qiang
Quoted 68 lines · Alexey Kardashevskiy
Quoted 45 lines · Chenyi Qiang
Introduce the realize()/unrealize() callbacks to initialize/ uninitialize
the newguest_memfd_managerobject and register/unregister it in the target MemoryRegion.
Guest_memfdwas initially set to shared until the commitbd3bcf6962("kvm/memory: Make memory type private by default if it has guest memfd backend"). To align with this change, the default state inguest_memfd_manageris set to private. (The bitmap is cleared to 0). Additionally, setting the default to private can also reduce the overhead of mapping shared pages into IOMMU by VFIO during the bootup stage.Signed-off-by Chenyi Qiang <[email protected]>--- include/sysemu/guest-memfd-manager.h | 27 +++++++++++++++++++++++ ++++ system/guest-memfd-manager.c | 28 +++++++++++++++++++++++ ++++- system/physmem.c | 7 +++++++ 3 files changed, 61 insertions(+), 1 deletion(-)include/sysemu/ renamed from include/sysemu/guest-memfd-manager.h +0
-0
guest-memfd-manager.h
index9dc4e0346d..d1e7f698e8100644include/sysemu/guest-memfd-manager.h +1
-0@@ -42,6 +42,8 @@ struct GuestMemfdManager { struct GuestMemfdManagerClass { ObjectClass parent_class; + void (*realize)(GuestMemfdManager *gmm, MemoryRegion *mr,uint64_t region_size); + void (*unrealize)(GuestMemfdManager *gmm); int (*state_change)(GuestMemfdManager *gmm, uint64_t offset,uint64_t size, bool shared_to_private); };@@ -61,4 +63,29 @@ static inline intguest_memfd_manager_state_change(GuestMemfdManager *gmm, uint6
return 0;
}
+static inline void guest_memfd_manager_realize(GuestMemfdManager
*gmm,
+ MemoryRegion *mr,uint64_tregion_size)include/sysemu/guest-memfd-manager.h static inline int guest_memfd_manager_state_change(GuestMemfdManager *gmm, uint6 · line 75
Quoted 8 lines
include/sysemu/guest-memfd-manager.h +8
-0+{ + GuestMemfdManagerClass *klass; + + g_assert(gmm); + klass = GUEST_MEMFD_MANAGER_GET_CLASS(gmm); + + if (klass->realize) { + klass->realize(gmm, mr, region_size);Ditch
realize()hook and callguest_memfd_manager_realizefn()directly? Not clear why these new hooks are needed.Quoted 9 lines
+ }
include/sysemu/guest-memfd-manager.h +3
-0+} + +static inline void guest_memfd_manager_unrealize(GuestMemfdManager*gmm)
include/sysemu/guest-memfd-manager.h +4
-0+{ + GuestMemfdManagerClass *klass; + + g_assert(gmm);Quoted 8 lines
include/sysemu/guest-memfd-manager.h static inline int guest_memfd_manager_state_change(GuestMemfdManager *gmm, uint6 · line 89
Quoted 6 lines
+ klass =
GUEST_MEMFD_MANAGER_GET_CLASS(gmm);include/sysemu/guest-memfd-manager.h +5
-0+ + if (klass->unrealize) { + klass->unrealize(gmm); + } +}
guest_memfd_manager_unrealizefn()?Agree. Adding these wrappers seem unnecessary.
Quoted 23 lines
Quoted 22 lines
include/sysemu/guest-memfd-manager.h +1
-0+ #endifsystem/guest-memfd- renamed from system/guest-memfd-manager.c +0
-0
manager.c
index6601df5f3f..b6a32f0bfb100644system/guest-memfd-manager.c +6
-0@@ -366,6 +366,31 @@ static int guest_memfd_state_change(GuestMemfdManager *gmm, uint64_t offset, return ret; } +static void guest_memfd_manager_realizefn(GuestMemfdManager *gmm, MemoryRegion *mr, + uint64_t region_size) +{ + uint64_t bitmap_size; + + gmm->block_size = qemu_real_host_page_size(); + bitmap_size = ROUND_UP(region_size, gmm->block_size) / gmm- > block_size;Quoted 8 lines
Quoted 4 lines
Quoted 1 line
imho unaligned
region_sizeshould be an assert.There's no guarantee the
region_sizeof the MemoryRegion isPAGE_SIZEaligned. So theROUND_UP()is more appropriate.It is all about DMA so the smallest you can map is
PAGE_SIZEso even if you round up here, it is likely going to fail to DMA-map later anyway (or not?).Checked the handling of VFIO, if the size is less than
PAGE_SIZE, it will just return and won't do DMA-map.Here is a different thing. It tries to calculate the
bitmap_size. The bitmap is used to track the private/shared status of the page. So if the size is less thanPAGE_SIZE, we still use the one bit to track this small-size range.Quoted 82 lines
Quoted 28 lines
Quoted 13 lines
Quoted 10 lines
system/guest-memfd-manager.c +6
-0+ + gmm->mr = mr; + gmm->bitmap_size = bitmap_size; + gmm->bitmap = bitmap_new(bitmap_size); + + memory_region_set_ram_discard_manager(gmm->mr,
RAM_DISCARD_MANAGER(gmm));
+}This belongs to 2/7.
Quoted 11 lines
system/guest-memfd-manager.c static int guest_memfd_state_change(GuestMemfdManager *gmm, uint64_t offset, · line 391
Quoted 9 lines
system/guest-memfd-manager.c +9
-0+ +static void guest_memfd_manager_unrealizefn (GuestMemfdManager *gmm) +{ + memory_region_set_ram_discard_manager(gmm->mr, NULL); + + g_free(gmm->bitmap); + gmm->bitmap = NULL; + gmm->bitmap_size = 0; + gmm->mr = NULL;@gmm is being destroyed here, why bother zeroing?
OK, will remove it.
Quoted 1 line
Quoted 6 lines
Quoted 4 lines
system/guest-memfd-manager.c
Quoted 2 lines
system/guest-memfd-manager.c +2
-0+} +This function belongs to 2/7.
Will move both
realizefn()andunrealizefn().Yes.
Quoted 44 lines
Quoted 43 lines
Quoted 32 lines
static void
guest_memfd_manager_init(Object *obj){
GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER(obj);system/guest-memfd-manager.c +2
-1@@ -375,7 +400,6 @@ static void guest_memfd_manager_init(Object *obj) static void guest_memfd_manager_finalize(Object *obj) { - g_free(GUEST_MEMFD_MANAGER(obj)->bitmap); } static void guest_memfd_manager_class_init(ObjectClass *oc, void *data)@@ -384,6 +408,8 @@ static voidguest_memfd_manager_class_init (ObjectClass *oc, void *data) RamDiscardManagerClass *rdmc = RAM_DISCARD_MANAGER_CLASS(oc); gmmc->state_change = guest_memfd_state_change; + gmmc->realize = guest_memfd_manager_realizefn; + gmmc->unrealize = guest_memfd_manager_unrealizefn; rdmc->get_min_granularity = guest_memfd_rdm_get_min_granularity; rdmc->register_listener = guest_memfd_rdm_register_listener;system/physmem.c +1
-0@@ -53,6 +53,7 @@ #include "sysemu/hostmem.h" #include "sysemu/hw_accel.h" #include "sysemu/xen-mapcache.h" +#include "sysemu/guest-memfd-manager.h" #include "trace.h" #ifdef CONFIG_FALLOCATE_PUNCH_HOLEsystem/physmem.c static void ram_block_add(RAMBlock *new_block, · line 1894
Quoted 9 lines
system/physmem.c +3
-0@@ -1885,6 +1886,9 @@ static void ram_block_add(RAMBlock *new_block, Error **errp) qemu_mutex_unlock_ramlist(); goto out_free; } + + GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER (object_new (TYPE_GUEST_MEMFD_MANAGER )); + guest_memfd_manager_realize(gmm, new_block->mr, new_block- > mr->size);Wow. Quite invasive.
Quoted 8 lines
Quoted 4 lines
Yeah... It creates a manager object no matter whether the user wants to us e shared passthru or not. We assume some fields like private/shared bitmap may also be helpful in other scenario for future usage, and if no passthru device, the listener would just return, so it is acceptable.
Explain these other scenarios in the commit log please as otherwise making this an interface of HostMemoryBackendMemfd looks way cleaner. Thanks,
Quoted 7 lines
Thanks for the suggestion. Until now, I think making this an interface of HostMemoryBackend is cleaner. The potential future usage for non-HostMemoryBackend guest_memfd-backed memory region I can think of is the the TEE I/O for iommufd P2P support? when it tries to initialize RAM device memory region with the attribute of shared/private. But I think it would be a long term story and we are not sure what it will be like in future.
As raised in #2, I'm don't think this belongs into HostMemoryBackend. It kind-of belongs to the RAMBlock, but we could have another object (similar to virtio-mem currently managing a single HostMemoryBackend->RAMBlock) that takes care of that for multiple memory backends.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH 5/7] memory: Register the RamDiscardManager instance upon guest_memfd creation
Quoted 228 lines · David Hildenbrand
Quoted 219 lines · Chenyi Qiang
Quoted 108 lines · Alexey Kardashevskiy
Quoted 105 lines · Chenyi Qiang
Quoted 68 lines · Alexey Kardashevskiy
Quoted 45 lines · Chenyi Qiang
Introduce the realize()/unrealize() callbacks to initialize/ uninitialize
the newguest_memfd_managerobject and register/unregister it in the target MemoryRegion.
Guest_memfdwas initially set to shared until the commitbd3bcf6962("kvm/memory: Make memory type private by default if it has guest memfd backend"). To align with this change, the default state inguest_memfd_manageris set to private. (The bitmap is cleared to 0). Additionally, setting the default to private can also reduce the overhead of mapping shared pages into IOMMU by VFIO during the bootup stage.Signed-off-by Chenyi Qiang <[email protected]>---
include/sysemu/guest-memfd-manager.h| 27 +++++++++++++++++++++++
++++
system/guest-memfd-manager.c | 28 +++++++++++++++++++++++
++++-
system/physmem.c | 7 +++++++3 files changed, 61 insertions(+), 1 deletion(-)include/sysemu/ renamed from include/sysemu/guest-memfd-manager.h +0
-0
guest-memfd-manager.h
index9dc4e0346d..d1e7f698e8100644include/sysemu/guest-memfd-manager.h +1
-0@@ -42,6 +42,8 @@ struct GuestMemfdManager { struct GuestMemfdManagerClass { ObjectClass parent_class; + void (*realize)(GuestMemfdManager *gmm, MemoryRegion *mr,uint64_t region_size); + void (*unrealize)(GuestMemfdManager *gmm); int (*state_change)(GuestMemfdManager *gmm, uint64_t offset, uint64_t size,bool
shared_to_private); };include/sysemu/guest-memfd-manager.h +0
-0@@ -61,4 +63,29 @@ static inline intguest_memfd_manager_state_change(GuestMemfdManager *gmm, uint6 return 0;
}
+static inline void guest_memfd_manager_realize(GuestMemfdManager *gmm,
+ MemoryRegion *mr,uint64_tregion_size)include/sysemu/guest-memfd-manager.h static inline int guest_memfd_manager_state_change(GuestMemfdManager *gmm, uint6 · line 75
Quoted 8 lines
include/sysemu/guest-memfd-manager.h +8
-0+{ + GuestMemfdManagerClass *klass; + + g_assert(gmm); + klass = GUEST_MEMFD_MANAGER_GET_CLASS(gmm); + + if (klass->realize) { + klass->realize(gmm, mr, region_size);Ditch
realize()hook and callguest_memfd_manager_realizefn()directly? Not clear why these new hooks are needed.Quoted 9 lines
+ }
include/sysemu/guest-memfd-manager.h +3
-0+} + +static inline void guest_memfd_manager_unrealize(GuestMemfdManager*gmm)
include/sysemu/guest-memfd-manager.h +4
-0+{ + GuestMemfdManagerClass *klass; + + g_assert(gmm);Quoted 8 lines
include/sysemu/guest-memfd-manager.h static inline int guest_memfd_manager_state_change(GuestMemfdManager *gmm, uint6 · line 89
Quoted 6 lines
+ klass =
GUEST_MEMFD_MANAGER_GET_CLASS(gmm);include/sysemu/guest-memfd-manager.h +5
-0+ + if (klass->unrealize) { + klass->unrealize(gmm); + } +}
guest_memfd_manager_unrealizefn()?Agree. Adding these wrappers seem unnecessary.
Quoted 23 lines
Quoted 22 lines
+
#endifsystem/guest-memfd- renamed from system/guest-memfd-manager.c +0
-0
manager.c
index6601df5f3f..b6a32f0bfb100644system/guest-memfd-manager.c +6
-0@@ -366,6 +366,31 @@ static int guest_memfd_state_change(GuestMemfdManager *gmm, uint64_t offset, return ret; } +static void guest_memfd_manager_realizefn(GuestMemfdManager *gmm, MemoryRegion *mr, + uint64_t region_size) +{ + uint64_t bitmap_size; + + gmm->block_size = qemu_real_host_page_size(); + bitmap_size = ROUND_UP(region_size, gmm->block_size) / gmm- > block_size;Quoted 8 lines
Quoted 4 lines
Quoted 1 line
imho unaligned
region_sizeshould be an assert.There's no guarantee the
region_sizeof the MemoryRegion isPAGE_SIZEaligned. So theROUND_UP()is more appropriate.It is all about DMA so the smallest you can map is
PAGE_SIZEso even if you round up here, it is likely going to fail to DMA-map later anyway (or not?).Checked the handling of VFIO, if the size is less than
PAGE_SIZE, it will just return and won't do DMA-map.Here is a different thing. It tries to calculate the
bitmap_size. The bitmap is used to track the private/shared status of the page. So if the size is less thanPAGE_SIZE, we still use the one bit to track this small-size range.Quoted 82 lines
Quoted 28 lines
Quoted 13 lines
Quoted 10 lines
system/guest-memfd-manager.c +6
-0+ + gmm->mr = mr; + gmm->bitmap_size = bitmap_size; + gmm->bitmap = bitmap_new(bitmap_size); + + memory_region_set_ram_discard_manager(gmm->mr,
RAM_DISCARD_MANAGER(gmm));
+}This belongs to 2/7.
Quoted 11 lines
system/guest-memfd-manager.c static int guest_memfd_state_change(GuestMemfdManager *gmm, uint64_t offset, · line 391
Quoted 9 lines
system/guest-memfd-manager.c +9
-0+ +static void guest_memfd_manager_unrealizefn (GuestMemfdManager *gmm) +{ + memory_region_set_ram_discard_manager(gmm->mr, NULL); + + g_free(gmm->bitmap); + gmm->bitmap = NULL; + gmm->bitmap_size = 0; + gmm->mr = NULL;@gmm is being destroyed here, why bother zeroing?
OK, will remove it.
Quoted 1 line
Quoted 6 lines
Quoted 4 lines
system/guest-memfd-manager.c
Quoted 2 lines
system/guest-memfd-manager.c +2
-0+} +This function belongs to 2/7.
Will move both
realizefn()andunrealizefn().Yes.
Quoted 44 lines
Quoted 43 lines
Quoted 32 lines
static void
guest_memfd_manager_init(Object *obj){
GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER(obj);system/guest-memfd-manager.c +0
-1@@ -375,7 +400,6 @@ static void guest_memfd_manager_init(Object *obj) static void guest_memfd_manager_finalize(Object *obj) { - g_free(GUEST_MEMFD_MANAGER(obj)->bitmap); }static void
guest_memfd_manager_class_init(ObjectClass *oc, void *data)system/guest-memfd-manager.c +2
-0@@ -384,6 +408,8 @@ static voidguest_memfd_manager_class_init (ObjectClass *oc, void *data) RamDiscardManagerClass *rdmc = RAM_DISCARD_MANAGER_CLASS(oc); gmmc->state_change = guest_memfd_state_change; + gmmc->realize = guest_memfd_manager_realizefn; + gmmc->unrealize = guest_memfd_manager_unrealizefn;
rdmc->get_min_granularity=guest_memfd_rdm_get_min_granularity;rdmc->register_listener=guest_memfd_rdm_register_listener;system/physmem.c +1
-0@@ -53,6 +53,7 @@ #include "sysemu/hostmem.h" #include "sysemu/hw_accel.h" #include "sysemu/xen-mapcache.h" +#include "sysemu/guest-memfd-manager.h" #include "trace.h" #ifdef CONFIG_FALLOCATE_PUNCH_HOLEsystem/physmem.c static void ram_block_add(RAMBlock *new_block, · line 1894
Quoted 9 lines
system/physmem.c +3
-0@@ -1885,6 +1886,9 @@ static void ram_block_add(RAMBlock *new_block, Error **errp) qemu_mutex_unlock_ramlist(); goto out_free; } + + GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER (object_new (TYPE_GUEST_MEMFD_MANAGER )); + guest_memfd_manager_realize(gmm, new_block->mr, new_block- > mr->size);Wow. Quite invasive.
Quoted 8 lines
Quoted 4 lines
Yeah... It creates a manager object no matter whether the user wants to us e shared passthru or not. We assume some fields like private/shared bitmap may also be helpful in other scenario for future usage, and if no passthru device, the listener would just return, so it is acceptable.
Explain these other scenarios in the commit log please as otherwise making this an interface of HostMemoryBackendMemfd looks way cleaner. Thanks,
Quoted 7 lines
Thanks for the suggestion. Until now, I think making this an interface of HostMemoryBackend is cleaner. The potential future usage for non-HostMemoryBackend guest_memfd-backed memory region I can think of is the the TEE I/O for iommufd P2P support? when it tries to initialize RAM device memory region with the attribute of shared/private. But I think it would be a long term story and we are not sure what it will be like in future.
Quoted 1 line
As raised in #2, I'm don't think this belongs into HostMemoryBackend. It kind-of belongs to the RAMBlock, but we could have another object (similar to virtio-mem currently managing a single
HostMemoryBackend->RAMBlock) that takes care of that for multiple memory backends.
The vBIOS thingy confused me and then I confused others :) There are 2 things:
1) an interface or new subclass of HostMemoryBackendClass which we need to advertise and implement ability to discard pages;
2) RamDiscardManagerClass which is MR/Ramblock and does not really belong to HostMemoryBackend (as it is in what was posted ages ago).
I suggest Chenyi post a new version using the current approach with the comments and commitlogs fixed. Makes sense? Thanks,
Signature · 1 line
Alexey
Re: [PATCH 5/7] memory: Register the RamDiscardManager instance upon guest_memfd creation
Quoted 256 lines · Alexey Kardashevskiy
Quoted 241 lines · David Hildenbrand
Quoted 232 lines · Chenyi Qiang
Quoted 115 lines · Alexey Kardashevskiy
Quoted 112 lines · Chenyi Qiang
Quoted 74 lines · Alexey Kardashevskiy
Quoted 50 lines · Chenyi Qiang
Introduce the realize()/unrealize() callbacks to initialize/ uninitialize
the newguest_memfd_managerobject and register/unregister it in the target MemoryRegion.
Guest_memfdwas initially set to shared until the commitbd3bcf6962("kvm/memory: Make memory type private by default if it has guest memfd
backend"). To align with this change, the default state inguest_memfd_manageris set to private. (The bitmap is cleared to 0). Additionally, setting the default to private can also reduce the overhead of mapping shared pages into IOMMU by VFIO during the bootup
stage.Signed-off-by Chenyi Qiang <[email protected]>---
include/sysemu/guest-memfd-manager.h| 27 +++++++++++++++++++
++++
++++
system/guest-memfd-manager.c | 28 +++++++++++++++++++
++++
++++-
system/physmem.c | 7 +++++++3 files changed, 61 insertions(+), 1 deletion(-)include/sysemu/ renamed from include/sysemu/guest-memfd-manager.h +0
-0
guest-memfd-manager.h
index9dc4e0346d..d1e7f698e8100644include/sysemu/guest-memfd-manager.h +1
-0@@ -42,6 +42,8 @@ struct GuestMemfdManager { struct GuestMemfdManagerClass { ObjectClass parent_class; + void (*realize)(GuestMemfdManager *gmm, MemoryRegion *mr,uint64_t region_size); + void (*unrealize)(GuestMemfdManager *gmm); int (*state_change)(GuestMemfdManager *gmm, uint64_t offset, uint64_t size,bool
shared_to_private); };include/sysemu/guest-memfd-manager.h +0
-0@@ -61,4 +63,29 @@ static inline intguest_memfd_manager_state_change(GuestMemfdManager *gmm, uint6 return 0;
}
+static inline void guest_memfd_manager_realize(GuestMemfdManager
*gmm,
+ MemoryRegion *mr,uint64_tregion_size)include/sysemu/guest-memfd-manager.h
Quoted 8 lines
include/sysemu/guest-memfd-manager.h +8
-0+{ + GuestMemfdManagerClass *klass; + + g_assert(gmm); + klass = GUEST_MEMFD_MANAGER_GET_CLASS(gmm); + + if (klass->realize) { + klass->realize(gmm, mr, region_size);Ditch
realize()hook and callguest_memfd_manager_realizefn()directly?
Not clear why these new hooks are needed.Quoted 9 lines
+ }
include/sysemu/guest-memfd-manager.h +3
-0+} + +static inline void guest_memfd_manager_unrealize(GuestMemfdManager*gmm)
include/sysemu/guest-memfd-manager.h +4
-0+{ + GuestMemfdManagerClass *klass; + + g_assert(gmm);Quoted 8 lines
include/sysemu/guest-memfd-manager.h
Quoted 6 lines
+ klass =
GUEST_MEMFD_MANAGER_GET_CLASS(gmm);include/sysemu/guest-memfd-manager.h +5
-0+ + if (klass->unrealize) { + klass->unrealize(gmm); + } +}
guest_memfd_manager_unrealizefn()?Agree. Adding these wrappers seem unnecessary.
Quoted 24 lines
Quoted 23 lines
+
#endifsystem/guest-memfd- renamed from system/guest-memfd-manager.c +0
-0
manager.c
index6601df5f3f..b6a32f0bfb100644system/guest-memfd-manager.c +0
-0@@ -366,6 +366,31 @@ static int guest_memfd_state_change(GuestMemfdManager *gmm, uint64_t offset, return ret; }+static void guest_memfd_manager_realizefn(GuestMemfdManager *gmm,
MemoryRegion *mr,
+uint64_tregion_size)system/guest-memfd-manager.c +5
-0+{ + uint64_t bitmap_size; + + gmm->block_size = qemu_real_host_page_size(); + bitmap_size = ROUND_UP(region_size, gmm->block_size) / gmm-Quoted 1 line
block_size;Quoted 8 lines
Quoted 4 lines
Quoted 1 line
imho unaligned
region_sizeshould be an assert.There's no guarantee the
region_sizeof the MemoryRegion isPAGE_SIZEaligned. So theROUND_UP()is more appropriate.It is all about DMA so the smallest you can map is
PAGE_SIZEso even if you round up here, it is likely going to fail to DMA-map later anyway (or not?).Checked the handling of VFIO, if the size is less than
PAGE_SIZE, it will just return and won't do DMA-map.Here is a different thing. It tries to calculate the
bitmap_size. The bitmap is used to track the private/shared status of the page. So if the size is less thanPAGE_SIZE, we still use the one bit to track this small-size range.Quoted 93 lines
Quoted 28 lines
Quoted 13 lines
Quoted 10 lines
system/guest-memfd-manager.c +6
-0+ + gmm->mr = mr; + gmm->bitmap_size = bitmap_size; + gmm->bitmap = bitmap_new(bitmap_size); + + memory_region_set_ram_discard_manager(gmm->mr,
RAM_DISCARD_MANAGER(gmm));
+}This belongs to 2/7.
Quoted 11 lines
system/guest-memfd-manager.c
Quoted 9 lines
system/guest-memfd-manager.c +9
-0+ +static void guest_memfd_manager_unrealizefn (GuestMemfdManager *gmm) +{ + memory_region_set_ram_discard_manager(gmm->mr, NULL); + + g_free(gmm->bitmap); + gmm->bitmap = NULL; + gmm->bitmap_size = 0; + gmm->mr = NULL;@gmm is being destroyed here, why bother zeroing?
OK, will remove it.
Quoted 1 line
Quoted 6 lines
Quoted 4 lines
system/guest-memfd-manager.c
Quoted 2 lines
system/guest-memfd-manager.c +2
-0+} +This function belongs to 2/7.
Will move both
realizefn()andunrealizefn().Yes.
Quoted 47 lines
Quoted 43 lines
Quoted 43 lines
static void
guest_memfd_manager_init(Object *obj){
GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER(obj);system/guest-memfd-manager.c +0
-1@@ -375,7 +400,6 @@ static void guest_memfd_manager_init(Object *obj) static void guest_memfd_manager_finalize(Object *obj) { - g_free(GUEST_MEMFD_MANAGER(obj)->bitmap); } static void guest_memfd_manager_class_init(ObjectClass *oc,void
*data)system/guest-memfd-manager.c +0
-0@@ -384,6 +408,8 @@ static voidguest_memfd_manager_class_init (ObjectClass *oc, void *data)RamDiscardManagerClass *rdmc =
RAM_DISCARD_MANAGER_CLASS(oc);
gmmc->state_change=guest_memfd_state_change; +gmmc->realize=guest_memfd_manager_realizefn;
+gmmc->unrealize=guest_memfd_manager_unrealizefn;rdmc->get_min_granularity=guest_memfd_rdm_get_min_granularity;rdmc->register_listener=guest_memfd_rdm_register_listener;system/physmem.c +4
-0@@ -53,6 +53,7 @@ #include "sysemu/hostmem.h" #include "sysemu/hw_accel.h" #include "sysemu/xen-mapcache.h" +#include "sysemu/guest-memfd-manager.h" #include "trace.h" #ifdef CONFIG_FALLOCATE_PUNCH_HOLE@@ -1885,6 +1886,9 @@ static void ram_block_add(RAMBlock *new_block, Error **errp) qemu_mutex_unlock_ramlist(); goto out_free; } + + GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER (object_new (TYPE_GUEST_MEMFD_MANAGER )); + guest_memfd_manager_realize(gmm, new_block->mr, new_block-Quoted 3 lines
system/physmem.c static void ram_block_add(RAMBlock *new_block, · line 1894
Quoted 1 line
system/physmem.c +0
-0> mr->size);Wow. Quite invasive.
Quoted 7 lines
Yeah... It creates a manager object no matter whether the user wants to
us e shared passthru or not. We assume some fields like private/ shared
bitmap may also be helpful in other scenario for future usage, and if no
passthru device, the listener would just return, so it is acceptable.Quoted 3 lines
Explain these other scenarios in the commit log please as otherwise making this an interface of HostMemoryBackendMemfd looks way cleaner. Thanks,
Quoted 7 lines
Thanks for the suggestion. Until now, I think making this an interface of HostMemoryBackend is cleaner. The potential future usage for non-HostMemoryBackend guest_memfd-backed memory region I can think of is the the TEE I/O for iommufd P2P support? when it tries to initialize RAM device memory region with the attribute of shared/private. But I think it would be a long term story and we are not sure what it will be like in future.
Quoted 4 lines
As raised in #2, I'm don't think this belongs into HostMemoryBackend. It kind-of belongs to the RAMBlock, but we could have another object (similar to virtio-mem currently managing a single HostMemoryBackend-
Quoted 1 line
RAMBlock) that takes care of that for multiple memory backends.
The vBIOS thingy confused me and then I confused others :) There are 2 things:
1) an interface or new subclass of HostMemoryBackendClass which we need to advertise and implement ability to discard pages;
2) RamDiscardManagerClass which is MR/Ramblock and does not really belong to HostMemoryBackend (as it is in what was posted ages ago).
Quoted 2 lines
I suggest Chenyi post a new version using the current approach with the comments and commitlogs fixed. Makes sense? Thanks,
Sure, thanks Alexey! BTW, I'm going to have a vacation. Will continue to work on it after I come back :)
Quoted 2 lines
Re: [PATCH 5/7] memory: Register the RamDiscardManager instance upon guest_memfd creation
In reply to Chenyi Qiangsystem/physmem.c static void ram_block_add(RAMBlock *new_block, Error **errp) · line 1891
Quoted 8 lines
#ifdef
CONFIG_FALLOCATE_PUNCH_HOLEsystem/physmem.c +3
-0@@ -1885,6 +1886,9 @@ static void ram_block_add(RAMBlock *new_block, Error **errp)qemu_mutex_unlock_ramlist (); goto out_free; } + +GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER (object_new (TYPE_GUEST_MEMFD_MANAGER )); +guest_memfd_manager_realize (gmm, new_block->mr, new_block->mr->size);
realize & unrealize are usually used for QDev. I think it's not good to use *realize and *unrealize here.
Why about "guest_memfd_manager_attach_ram"?
In addition, it seems the third parameter is unnecessary and we can access MemoryRegion.size directly in guest_memfd_manager_realize().
system/physmem.c static void reclaim_ramblock(RAMBlock *block) · line 2147
Quoted 9 lines
system/physmem.c +2
-0} ram_size = (new_block->offset + new_block->max_length) >>TARGET_PAGE_BITS ;@@ -2139,6 +2143,9 @@ static void reclaim_ramblock(RAMBlock *block)if (block->guest_memfd >=0 ) {close (block->guest_memfd); +GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER (block->mr->rdm); +guest_memfd_manager_unrealize (gmm);
Similiarly, what about "guest_memfd_manager_unattach_ram"?
system/physmem.c static void reclaim_ramblock(RAMBlock *block) · line 2151
Quoted 4 lines
system/physmem.c +1
-0+object_unref (OBJECT (gmm));ram_block_discard_require (false); }
Regards,
Zhao
Re: [PATCH 5/7] memory: Register the RamDiscardManager instance upon guest_memfd creation
Thanks Zhao for your review!
Quoted 12 lines · Zhao Liu
system/physmem.c static void ram_block_add(RAMBlock *new_block, Error **errp) · line 1891
Quoted 8 lines
#ifdef
CONFIG_FALLOCATE_PUNCH_HOLEsystem/physmem.c +3
-0@@ -1885,6 +1886,9 @@ static void ram_block_add(RAMBlock *new_block, Error **errp)qemu_mutex_unlock_ramlist (); goto out_free; } + +GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER (object_new (TYPE_GUEST_MEMFD_MANAGER )); +guest_memfd_manager_realize (gmm, new_block->mr, new_block->mr->size);realize & unrealize are usually used for QDev. I think it's not good to use *realize and *unrealize here.
Quoted 4 lines
Why about "
guest_memfd_manager_attach_ram"?In addition, it seems the third parameter is unnecessary and we can access MemoryRegion.size directly in
guest_memfd_manager_realize().
LGTM. Will follow your suggestion if we still wrap the operations in one function. (We may change to the HostMemoryBackend RDM then unpack the operations).
Quoted 6 lines
Quoted 6 lines
system/physmem.c +0
-0} ram_size = (new_block->offset + new_block->max_length) >>TARGET_PAGE_BITS ;@@ -2139,6 +2143,9 @@ static void reclaim_ramblock(RAMBlock *block)
Quoted 6 lines
system/physmem.c static void reclaim_ramblock(RAMBlock *block) · line 2147
Quoted 4 lines
system/physmem.c +2
-0if (block->guest_memfd >=0 ) {close (block->guest_memfd); +GuestMemfdManager *gmm =GUEST_MEMFD_MANAGER (block->mr->rdm); +guest_memfd_manager_unrealize (gmm);Similiarly, what about "
guest_memfd_manager_unattach_ram"?
Ditto. thanks.
Quoted 9 lines
system/physmem.c static void reclaim_ramblock(RAMBlock *block) · line 2151
Quoted 5 lines
system/physmem.c +1
-0+object_unref (OBJECT (gmm));ram_block_discard_require (false); }Regards,
Zhao
PATCH 6/7 RAMBlock: make guest_memfd require coordinate discard
In reply to Chenyi QiangAs guest_memfd is now managed by guest_memfd_manager with RamDiscardManager, only block uncoordinated discard.
--- system/physmem.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
system/physmem.c +1 -1
@@ -1872,7 +1872,7 @@ static void ram_block_add(RAMBlock *new_block, Error **errp)assert (kvm_enabled ());assert (new_block->guest_memfd <0 ); - ret =ram_block_discard_require (true); + ret =ram_block_coordinated_discard_require (true);if (ret <0 ) {error_setg_errno (errp, -ret,"cannot set up private guest memory: discard currently blocked" );
Signature · 1 line
2.43.5
Re: [PATCH 6/7] RAMBlock: make guest_memfd require coordinate discard
Quoted 13 lines · Chenyi Qiang
As
guest_memfdis now managed byguest_memfd_managerwith RamDiscardManager, only block uncoordinated discard.Signed-off-by Chenyi Qiang <[email protected]>--- system/physmem.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)system/physmem.c +0
-0
system/physmem.c static void ram_block_add(RAMBlock *new_block, Error **errp) · line 1878
Quoted 8 lines
system/physmem.c +1
-1@@ -1872,7 +1872,7 @@ static void ram_block_add(RAMBlock *new_block, Error **errp)assert (kvm_enabled ());assert (new_block->guest_memfd <0 ); - ret =ram_block_discard_require (true); + ret =ram_block_coordinated_discard_require (true);if (ret <0 ) {error_setg_errno (errp, -ret,"cannot set up private guest memory: discard currently blocked" );
Would that also unlock virtio-mem by accident?
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH 6/7] RAMBlock: make guest_memfd require coordinate discard
Quoted 9 lines · David Hildenbrand
Quoted 8 lines · Chenyi Qiang
As
guest_memfdis now managed byguest_memfd_managerwith RamDiscardManager, only block uncoordinated discard.Signed-off-by Chenyi Qiang <[email protected]>---
system/physmem.c| 2 +-1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 16 lines
Quoted 12 lines
system/physmem.c +1
-0@@ -1872,7 +1872,7 @@ static void ram_block_add(RAMBlock *new_block, Error **errp) assert(kvm_enabled()); assert(new_block->guest_memfd < 0); - ret = ram_block_discard_require(true); + ret = ram_block_coordinated_discard_require(true); if (ret < 0) { error_setg_errno(errp, -ret,Quoted 2 lines
"cannot set up private guest memory:
discard currently blocked");Would that also unlock virtio-mem by accident?
Hum, that's true. At present, the rdm in MR can only point to one instance, thus if we unlock virtio-mem and try to use it with guest_memfd, it would trigger assert in memory_region_set_ram_discard_manager().
Maybe we need to add some explicit check in virtio-mem to exclude it with guest_memfd at present?
Quoted 1 line
Re: [PATCH 6/7] RAMBlock: make guest_memfd require coordinate discard
Quoted 29 lines · Chenyi Qiang
Quoted 9 lines · David Hildenbrand
Quoted 8 lines · Chenyi Qiang
As
guest_memfdis now managed byguest_memfd_managerwith RamDiscardManager, only block uncoordinated discard.Signed-off-by Chenyi Qiang <[email protected]>--- system/physmem.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)Quoted 16 lines
Quoted 13 lines
system/physmem.c +1
-0@@ -1872,7 +1872,7 @@ static void ram_block_add(RAMBlock *new_block, Error **errp) assert(kvm_enabled()); assert(new_block->guest_memfd < 0); - ret = ram_block_discard_require(true); + ret = ram_block_coordinated_discard_require(true); if (ret < 0) { error_setg_errno(errp, -ret, "cannot set up private guest memory:Quoted 1 line
discard currently blocked");
Would that also unlock virtio-mem by accident?
Quoted 7 lines
Hum, that's true. At present, the rdm in MR can only point to one instance, thus if we unlock virtio-mem and try to use it with
guest_memfd, it would trigger assert inmemory_region_set_ram_discard_manager().Maybe we need to add some explicit check in virtio-mem to exclude it with
guest_memfdat present?
Likely we should make memory_region_set_ram_discard_manager() fail if there is already something, and handle it in the callers?
In case of virtio-mem, we'd have to undo what we did and fail realize().
In case of CC, we'd have to bail out in a different way.
Then, I think if we see new_block->guest_memfd here, that we can assume that any coordinated discard corresponds to only the guest_memfd one, not to anything else?
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH 6/7] RAMBlock: make guest_memfd require coordinate discard
Quoted 41 lines · David Hildenbrand
Quoted 29 lines · Chenyi Qiang
Quoted 9 lines · David Hildenbrand
Quoted 8 lines · Chenyi Qiang
As
guest_memfdis now managed byguest_memfd_managerwith RamDiscardManager, only block uncoordinated discard.Signed-off-by Chenyi Qiang <[email protected]>---
system/physmem.c| 2 +-1 file changed, 1 insertion(+), 1 deletion(-)Quoted 16 lines
Quoted 12 lines
system/physmem.c +1
-0@@ -1872,7 +1872,7 @@ static void ram_block_add(RAMBlock *new_block, Error **errp) assert(kvm_enabled()); assert(new_block->guest_memfd < 0); - ret = ram_block_discard_require(true); + ret = ram_block_coordinated_discard_require(true); if (ret < 0) { error_setg_errno(errp, -ret,Quoted 2 lines
"cannot set up private guest memory:
discard currently blocked");Would that also unlock virtio-mem by accident?
Quoted 7 lines
Hum, that's true. At present, the rdm in MR can only point to one instance, thus if we unlock virtio-mem and try to use it with
guest_memfd, it would trigger assert inmemory_region_set_ram_discard_manager().Maybe we need to add some explicit check in virtio-mem to exclude it with
guest_memfdat present?Likely we should make
memory_region_set_ram_discard_manager()fail if there is already something, and handle it in the callers?
Quoted 8 lines
In case of virtio-mem, we'd have to undo what we did and fail
realize().In case of CC, we'd have to bail out in a different way.
Then, I think if we see
new_block->guest_memfdhere, that we can assume that any coordinated discard corresponds to only theguest_memfdone, not to anything else?
LGTM. In case of CC, I think we can also check the memory_region_set_ram_discard_manager() failure, undo what we did and make the ram_block_add() fail (set errno).
Quoted 1 line
Re: [PATCH 6/7] RAMBlock: make guest_memfd require coordinate discard
Quoted 49 lines · Chenyi Qiang
Quoted 46 lines · David Hildenbrand
Quoted 29 lines · Chenyi Qiang
Quoted 9 lines · David Hildenbrand
Quoted 8 lines · Chenyi Qiang
As
guest_memfdis now managed byguest_memfd_managerwith RamDiscardManager, only block uncoordinated discard.Signed-off-by Chenyi Qiang <[email protected]>--- system/physmem.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)Quoted 16 lines
Quoted 13 lines
system/physmem.c +1
-0@@ -1872,7 +1872,7 @@ static void ram_block_add(RAMBlock *new_block, Error **errp) assert(kvm_enabled()); assert(new_block->guest_memfd < 0); - ret = ram_block_discard_require(true); + ret = ram_block_coordinated_discard_require(true); if (ret < 0) { error_setg_errno(errp, -ret, "cannot set up private guest memory:Quoted 1 line
discard currently blocked");
Would that also unlock virtio-mem by accident?
Quoted 7 lines
Hum, that's true. At present, the rdm in MR can only point to one instance, thus if we unlock virtio-mem and try to use it with
guest_memfd, it would trigger assert inmemory_region_set_ram_discard_manager().Maybe we need to add some explicit check in virtio-mem to exclude it with
guest_memfdat present?Likely we should make
memory_region_set_ram_discard_manager()fail if there is already something, and handle it in the callers?In case of virtio-mem, we'd have to undo what we did and fail
realize().In case of CC, we'd have to bail out in a different way.
Quoted 7 lines
Quoted 3 lines
Then, I think if we see
new_block->guest_memfdhere, that we can assume that any coordinated discard corresponds to only theguest_memfdone, not to anything else?LGTM. In case of CC, I think we can also check the
memory_region_set_ram_discard_manager()failure, undo what we did and make theram_block_add()fail (set errno).
As we have memory_region_has_ram_discard_manager(), we could also check that instead of failing memory_region_set_ram_discard_manager().
But failing memory_region_set_ram_discard_manager() will force everybody to handle that, so it might be the better choice.
Of course, setting it to "NULL" should be guaranteed to never fail.
Signature · 3 lines
Cheers, David / dhildenb
RFC PATCH 7/7 memory: Add a new argument to indicate the request attribute in RamDismcardManager helpers
In reply to Chenyi QiangFor each ram_discard_manager helper, add a new argument 'is_private' to indicate the request attribute. If is_private is true, the operation targets the private range in the section. For example, replay_populate(true) will replay the populate operation on private part in the MemoryRegionSection, while replay_popuate(false) will replay population on shared part.
This helps to distinguish between the states of private/shared and discarded/populated. It is essential for guest_memfd_manager which uses RamDiscardManager interface but can't treat private memory as discarded memory. This is because it does not align with the expectation of current RamDiscardManager users (e.g. live migration), who expect that discarded memory is hot-removed and can be skipped when processing guest memory. Treating private memory as discarded won't work in the future if live migration needs to handle private memory. For example, live migration needs to migrate private memory.
The user of the helper needs to figure out which attribute to manipulate. For legacy VM case, use is_private=true by default. Private attribute is only valid in a guest_memfd based VM.
Opportunistically rename the guest_memfd_for_each_{discarded, populated}_section() to guest_memfd_for_each_{private, shared)_section() to distinguish between private/shared and discarded/populated at the same time.
--- hw/vfio/common.c | 22 ++++++-- hw/virtio/virtio-mem.c | 23 ++++---- include/exec/memory.h | 23 ++++++-- migration/ram.c | 14 ++--- system/guest-memfd-manager.c | 106 +++++++++++++++++++++++------------ system/memory.c | 13 +++-- system/memory_mapping.c | 4 +- 7 files changed, 135 insertions(+), 70 deletions(-)
hw/vfio/common.c +17 -5
@@ -345,7 +345,8 @@ out: }static void vfio_ram_discard_notify_discard (RamDiscardListener *rdl, -MemoryRegionSection *section) +MemoryRegionSection *section, +bool is_private) {VFIORamDiscardListener *vrdl =container_of (rdl, VFIORamDiscardListener, listener);
@@ -354,6 +355,11 @@ static void vfio_ram_discard_notify_discard(RamDiscardListener *rdl,const hwaddr iova = section->offset_within_address_space;int ret; +if (is_private) { +/* Not support discard private memory yet. */ +return ; + } +/* Unmap with a single call. */ ret =vfio_container_dma_unmap (bcontainer, iova, size ,NULL );if (ret) {
@@ -363,7 +369,8 @@ static void vfio_ram_discard_notify_discard(RamDiscardListener *rdl, }static int vfio_ram_discard_notify_populate (RamDiscardListener *rdl, -MemoryRegionSection *section) +MemoryRegionSection *section, +bool is_private) {VFIORamDiscardListener *vrdl =container_of (rdl, VFIORamDiscardListener, listener);
@@ -374,6 +381,11 @@ static int vfio_ram_discard_notify_populate(RamDiscardListener *rdl,void *vaddr;int ret; +if (is_private) { +/* Not support discard private memory yet. */ +return 0 ; + } +/* * Map in (aligned within memory region) minimum granularity, so we can * unmap in minimum granularity later.
@@ -390,7 +402,7 @@ static int vfio_ram_discard_notify_populate(RamDiscardListener *rdl, vaddr, section->readonly);if (ret) {/* Rollback */ -vfio_ram_discard_notify_discard (rdl, section); +vfio_ram_discard_notify_discard (rdl, section, false);return ret; } }
@@ -1248,7 +1260,7 @@ out: }static int vfio_ram_discard_get_dirty_bitmap (MemoryRegionSection *section, -void *opaque) +bool is_private,void *opaque) {const hwaddr size =int128_get64 (section->size);const hwaddr iova = section->offset_within_address_space;
@@ -1293,7 +1305,7 @@ vfio_sync_ram_discard_listener_dirty_bitmap(VFIOContainerBase *bcontainer,* We only want/can synchronize the bitmap for actually mapped parts - * which correspond to populated parts. Replay all populated parts. */ -return ram_discard_manager_replay_populated (rdm, section, +return ram_discard_manager_replay_populated (rdm, section, false, vfio_ram_discard_get_dirty_bitmap, &vrdl); }
hw/virtio/virtio-mem.c +13 -10
@@ -312,14 +312,14 @@ static int virtio_mem_notify_populate_cb(MemoryRegionSection *s, void *arg) {RamDiscardListener *rdl = arg; -return rdl->notify_populate (rdl, s); +return rdl->notify_populate (rdl, s, false); }static int virtio_mem_notify_discard_cb (MemoryRegionSection *s,void *arg) {RamDiscardListener *rdl = arg; - rdl->notify_discard (rdl, s); + rdl->notify_discard (rdl, s, false);return 0 ; }
@@ -334,7 +334,7 @@ static void virtio_mem_notify_unplug(VirtIOMEM *vmem, uint64_t offset,if (!memory_region_section_intersect_range (&tmp, offset, size)) {continue ; } - rdl->notify_discard (rdl, &tmp); + rdl->notify_discard (rdl, &tmp, false); } }
@@ -350,7 +350,7 @@ static int virtio_mem_notify_plug(VirtIOMEM *vmem, uint64_t offset,if (!memory_region_section_intersect_range (&tmp, offset, size)) {continue ; } - ret = rdl->notify_populate (rdl, &tmp); + ret = rdl->notify_populate (rdl, &tmp, false);if (ret) {break ; }
@@ -367,7 +367,7 @@ static int virtio_mem_notify_plug(VirtIOMEM *vmem, uint64_t offset,if (!memory_region_section_intersect_range (&tmp, offset, size)) {continue ; } - rdl2->notify_discard (rdl2, &tmp); + rdl2->notify_discard (rdl2, &tmp, false); } }return ret;
@@ -383,7 +383,7 @@ static void virtio_mem_notify_unplug_all(VirtIOMEM *vmem) QLIST_FOREACH(rdl, &vmem->rdl_list, next) {if (rdl->double_discard_supported) { - rdl->notify_discard (rdl, rdl->section); + rdl->notify_discard (rdl, rdl->section, false); }else {virtio_mem_for_each_plugged_section (vmem, rdl->section, rdl, virtio_mem_notify_discard_cb);
@@ -1685,7 +1685,8 @@ static uint64_t virtio_mem_rdm_get_min_granularity(const RamDiscardManager *rdm, }static bool virtio_mem_rdm_is_populated (const RamDiscardManager *rdm, -const MemoryRegionSection *s) +const MemoryRegionSection *s, +bool is_private) {const VirtIOMEM *vmem =VIRTIO_MEM (rdm);uint64_t start_gpa = vmem->addr + s->offset_within_region;
@@ -1712,11 +1713,12 @@ static int virtio_mem_rdm_replay_populated_cb(MemoryRegionSection *s, void *arg) {struct VirtIOMEMReplayData *data = arg; -return ((ReplayRamPopulate )data->fn)(s, data->opaque); +return ((ReplayRamPopulate )data->fn)(s, false, data->opaque); }static int virtio_mem_rdm_replay_populated (const RamDiscardManager *rdm,MemoryRegionSection *s, +bool is_private,ReplayRamPopulate replay_fn,void *opaque) {
@@ -1736,12 +1738,13 @@ static int virtio_mem_rdm_replay_discarded_cb(MemoryRegionSection *s, {struct VirtIOMEMReplayData *data = arg; - ((ReplayRamDiscard )data->fn)(s, data->opaque); + ((ReplayRamDiscard )data->fn)(s, false, data->opaque);return 0 ; }static void virtio_mem_rdm_replay_discarded (const RamDiscardManager *rdm,MemoryRegionSection *s, +bool is_private,ReplayRamDiscard replay_fn,void *opaque) {
@@ -1783,7 +1786,7 @@ static void virtio_mem_rdm_unregister_listener(RamDiscardManager *rdm,g_assert (rdl->section->mr == &vmem->memdev->mr);if (vmem->size) {if (rdl->double_discard_supported) { - rdl->notify_discard (rdl, rdl->section); + rdl->notify_discard (rdl, rdl->section, false); }else {virtio_mem_for_each_plugged_section (vmem, rdl->section, rdl, virtio_mem_notify_discard_cb);
include/exec/memory.h +17 -6
@@ -508,9 +508,11 @@ struct IOMMUMemoryRegionClass {typedef struct RamDiscardListener RamDiscardListener ;typedef int (*NotifyRamPopulate )(RamDiscardListener *rdl, -MemoryRegionSection *section); +MemoryRegionSection *section, +bool is_private);typedef void (*NotifyRamDiscard )(RamDiscardListener *rdl, -MemoryRegionSection *section); +MemoryRegionSection *section, +bool is_private);struct RamDiscardListener {/*
@@ -566,8 +568,8 @@ static inline void ram_discard_listener_init(RamDiscardListener *rdl, rdl->double_discard_supported = double_discard_supported; } -typedef int (*ReplayRamPopulate )(MemoryRegionSection *section,void *opaque); -typedef void (*ReplayRamDiscard )(MemoryRegionSection *section,void *opaque); +typedef int (*ReplayRamPopulate )(MemoryRegionSection *section,bool is_private,void *opaque); +typedef void (*ReplayRamDiscard )(MemoryRegionSection *section,bool is_private,void *opaque);/* * RamDiscardManagerClass:
@@ -632,11 +634,13 @@ struct RamDiscardManagerClass {* * @rdm: the #RamDiscardManager * @section: the #MemoryRegionSection +* @is_private: the attribute of the request section * * Returns whether the given range is completely populated. */ bool (*is_populated)(const RamDiscardManager *rdm, -const MemoryRegionSection *section); +const MemoryRegionSection *section, +bool is_private);/** * @replay_populated:
@@ -648,6 +652,7 @@ struct RamDiscardManagerClass {* * @rdm: the #RamDiscardManager * @section: the #MemoryRegionSection +* @is_private: the attribute of the populated parts * @replay_fn: the #ReplayRamPopulate callback * @opaque: pointer to forward to the callback *
@@ -655,6 +660,7 @@ struct RamDiscardManagerClass {*/ int (*replay_populated)(const RamDiscardManager *rdm,MemoryRegionSection *section, +bool is_private,ReplayRamPopulate replay_fn,void *opaque);/**
@@ -665,11 +671,13 @@ struct RamDiscardManagerClass {* * @rdm: the #RamDiscardManager * @section: the #MemoryRegionSection +* @is_private: the attribute of the discarded parts * @replay_fn: the #ReplayRamDiscard callback * @opaque: pointer to forward to the callback */ void (*replay_discarded)(const RamDiscardManager *rdm,MemoryRegionSection *section, +bool is_private,ReplayRamDiscard replay_fn,void *opaque);/**
@@ -709,15 +717,18 @@ uint64_t ram_discard_manager_get_min_granularity(const RamDiscardManager *rdm, const MemoryRegion *mr);bool ram_discard_manager_is_populated (const RamDiscardManager *rdm, -const MemoryRegionSection *section); +const MemoryRegionSection *section, +bool is_private);int ram_discard_manager_replay_populated (const RamDiscardManager *rdm,MemoryRegionSection *section, +bool is_private,ReplayRamPopulate replay_fn,void *opaque);void ram_discard_manager_replay_discarded (const RamDiscardManager *rdm,MemoryRegionSection *section, +bool is_private,ReplayRamDiscard replay_fn,void *opaque);
migration/ram.c +7 -7
@@ -838,7 +838,7 @@ static inline bool migration_bitmap_clear_dirty(RAMState *rs, }static void dirty_bitmap_clear_section (MemoryRegionSection *section, -void *opaque) +bool is_private,void *opaque) {const hwaddr offset = section->offset_within_region;const hwaddr size =int128_get64 (section->size);
@@ -884,7 +884,7 @@ static uint64_t ramblock_dirty_bitmap_clear_discarded_pages(RAMBlock *rb) .size = int128_make64(qemu_ram_get_used_length(rb)), }; -ram_discard_manager_replay_discarded (rdm, §ion, +ram_discard_manager_replay_discarded (rdm, §ion, false, dirty_bitmap_clear_section, &cleared_bits); }
@@ -907,7 +907,7 @@ bool ramblock_page_is_discarded(RAMBlock *rb, ram_addr_t start) .size = int128_make64(qemu_ram_pagesize(rb)), }; -return !ram_discard_manager_is_populated (rdm, §ion); +return !ram_discard_manager_is_populated (rdm, §ion, false); }return false; }
@@ -1539,7 +1539,7 @@ static inline void populate_read_range(RAMBlock *block, ram_addr_t offset, }static inline int populate_read_section (MemoryRegionSection *section, -void *opaque) +bool is_private,void *opaque) {const hwaddr size =int128_get64 (section->size);hwaddr offset = section->offset_within_region;
@@ -1579,7 +1579,7 @@ static void ram_block_populate_read(RAMBlock *rb) .size = rb->mr->size, }; -ram_discard_manager_replay_populated (rdm, §ion, +ram_discard_manager_replay_populated (rdm, §ion, false, populate_read_section,NULL ); }else {populate_read_range (rb,0 , rb->used_length);
@@ -1614,7 +1614,7 @@ void ram_write_tracking_prepare(void) }static inline int uffd_protect_section (MemoryRegionSection *section, -void *opaque) +bool is_private,void *opaque) {const hwaddr size =int128_get64 (section->size);const hwaddr offset = section->offset_within_region;
@@ -1638,7 +1638,7 @@ static int ram_block_uffd_protect(RAMBlock *rb, int uffd_fd) .size = rb->mr->size, }; -return ram_discard_manager_replay_populated (rdm, §ion, +return ram_discard_manager_replay_populated (rdm, §ion, false, uffd_protect_section, (void *)(uintptr_t )uffd_fd); }
system/guest-memfd-manager.c +71 -35
220 lines not on this page · raw message
system/memory.c +8 -5
@@ -2133,34 +2133,37 @@ uint64_t ram_discard_manager_get_min_granularity(const RamDiscardManager *rdm, }bool ram_discard_manager_is_populated (const RamDiscardManager *rdm, -const MemoryRegionSection *section) +const MemoryRegionSection *section, +bool is_private) {RamDiscardManagerClass *rdmc =RAM_DISCARD_MANAGER_GET_CLASS (rdm);g_assert (rdmc->is_populated); -return rdmc->is_populated (rdm, section); +return rdmc->is_populated (rdm, section, is_private); }int ram_discard_manager_replay_populated (const RamDiscardManager *rdm,MemoryRegionSection *section, +bool is_private,ReplayRamPopulate replay_fn,void *opaque) {RamDiscardManagerClass *rdmc =RAM_DISCARD_MANAGER_GET_CLASS (rdm);g_assert (rdmc->replay_populated); -return rdmc->replay_populated (rdm, section, replay_fn, opaque); +return rdmc->replay_populated (rdm, section, is_private, replay_fn, opaque); }void ram_discard_manager_replay_discarded (const RamDiscardManager *rdm,MemoryRegionSection *section, +bool is_private,ReplayRamDiscard replay_fn,void *opaque) {RamDiscardManagerClass *rdmc =RAM_DISCARD_MANAGER_GET_CLASS (rdm);g_assert (rdmc->replay_discarded); - rdmc->replay_discarded (rdm, section, replay_fn, opaque); + rdmc->replay_discarded (rdm, section, is_private, replay_fn, opaque); } void ram_discard_manager_register_listener(RamDiscardManager *rdm,
@@ -2221,7 +2224,7 @@ bool memory_get_xlat_addr(IOMMUTLBEntry *iotlb, void **vaddr,* Disallow that. vmstate priorities make sure any RamDiscardManager * were already restored before IOMMUs are restored. */ -if (!ram_discard_manager_is_populated (rdm, &tmp)) { +if (!ram_discard_manager_is_populated (rdm, &tmp, false)) {error_setg (errp,"iommu map to discarded memory (e.g., unplugged" " via virtio-mem): %" HWADDR_PRIx"" , iotlb->translated_addr);
system/memory_mapping.c +2 -2
@@ -249,7 +249,7 @@ static void guest_phys_block_add_section(GuestPhysListener *g, }static int guest_phys_ram_populate_cb (MemoryRegionSection *section, -void *opaque) +bool is_private,void *opaque) {GuestPhysListener *g = opaque;
@@ -274,7 +274,7 @@ static void guest_phys_blocks_region_add(MemoryListener *listener,RamDiscardManager *rdm; rdm =memory_region_get_ram_discard_manager (section->mr); -ram_discard_manager_replay_populated (rdm, section, +ram_discard_manager_replay_populated (rdm, section, false, guest_phys_ram_populate_cb, g);return ; }
Signature · 1 line
2.43.5
Quoted 64 lines · Chenyi Qiang
Commit
852f0048f3("RAMBlock: makeguest_memfdrequire uncoordinated discard") effectively disables device assignment when usingguest_memfd. This poses a significant challenge asguest_memfdis essential for confidential guests, thereby blocking device assignment to these VMs. The initial rationale for disabling device assignment was due to stale IOMMU mappings (see Problem section) and the assumption that TEE I/O (SEV-TIO, TDX Connect, COVE-IO, etc.) would solve the device-assignment problem for confidential guests [1]. However, this assumption has proven to be incorrect. TEE I/O relies on the ability to operate devices against "shared" or untrusted memory, which is crucial for device initialization and error recovery scenarios. As a result, the current implementation does not adequately support device assignment for confidential guests, necessitating a reevaluation of the approach to ensure compatibility and functionality.This series enables shared device assignment by notifying VFIO of page conversions using an existing framework named RamDiscardListener. Additionally, there is an ongoing patch set [2] that aims to add 1G page support for
guest_memfd. This patch set introduces in-place page conversion, where private and shared memory share the same physical pages as the backend. This development may impact our solution.We presented our solution in the
guest_memfdmeeting to discuss its compatibility with the new changes and potential future directions (see [3] for more details). The conclusion was that, although our solution may not be the most elegant (see the Limitation section), it is sufficient for now and can be easily adapted to future changes.We are re-posting the patch series with some cleanup and have removed the RFC label for the main enabling patches (1-6). The newly-added patch 7 is still marked as RFC as it tries to resolve some extension concerns related to RamDiscardManager for future usage.
The overview of the patches:
- Patch 1: Export a helper to get intersection of a MemoryRegionSection with a given range.
- Patch 2-6: Introduce a new object to manage the guest-memfd with RamDiscardManager, and notify the shared/private state change during conversion.
- Patch 7: Try to resolve a semantics concern related to RamDiscardManager i.e. RamDiscardManager is used to manage memory plug/unplug state instead of shared/private state. It would affect future users of RamDiscardManger in confidential VMs. Attach it behind as a RFC patch[4].Changes since last version:
- Add a patch to export some generic helper functions from virtio-mem code.
- Change the bitmap inguest_memfd_managerfrom default shared to default private. This keeps alignment with virtio-mem that 1-setting in bitmap represents the populated state and may help to export more generic code if necessary.
- Add the helpers to initialize/uninitialize theguest_memfd_managerinstance to make it more clear.
- Add a patch to distinguish between the shared/private state change and the memory plug/unplug state change in RamDiscardManager.
- RFC: https://lore.kernel.org/qemu-devel/[email protected]/---
Background
==========
Confidential VMs have two classes of memory: shared and private memory. Shared memory is accessible from the host/VMM while private memory is not. Confidential VMs can decide which memory is shared/private and convert memory between shared/private at runtime.
Quoted 4 lines
"
guest_memfd" is a new kind of fd whose primary goal is to serve guest private memory. The key differences betweenguest_memfdand normal memfd are thatguest_memfdis spawned by a KVM ioctl, bound to its owner VM and cannot be mapped, read or written by userspace.
The "cannot be mapped" seems to be not true soon anymore (if not already).
https://lore.kernel.org/all/[email protected]/T/
Quoted 70 lines
In QEMU's implementation, shared memory is allocated with normal methods (e.g. mmap or fallocate) while private memory is allocated from
guest_memfd. When a VM performs memory conversions, QEMU frees pages viamadvise()or viaPUNCH_HOLEon memfd orguest_memfdfrom one side and allocates new pages from the other side.Problem
=======
Device assignment in QEMU is implemented via VFIO system. In the normal VM, VM memory is pinned at the beginning of time by VFIO. In the confidential VM, the VM can convert memory and when that happens nothing currently tells VFIO that its mappings are stale. This means that page conversion leaks memory and leaves stale IOMMU mappings. For example, sequence like the following can result in stale IOMMU mappings:1. allocate shared page
2. convert pageshared->private
3. discard shared page
4. convert pageprivate->shared
5. allocate shared page
6. issue DMA operations against that shared pageAfter step 3, VFIO is still pinning the page. However, DMA operations in step 6 will hit the old mapping that was allocated in step 1, which causes the device to access the invalid data.
Solution
========
The key to enable shared device assignment is to update the IOMMU mappings on page conversion.Given the constraints and assumptions here is a solution that satisfied the use cases. RamDiscardManager, an existing interface currently utilized by virtio-mem, offers a means to modify IOMMU mappings in accordance with VM page assignment. Page conversion is similar to hot-removing a page in one mode and adding it back in the other.
This series implements a RamDiscardManager for confidential VMs and utilizes its infrastructure to notify VFIO of page conversions.
Another possible attempt [5] was to not discard shared pages in step 3 above. This was an incomplete band-aid because guests would consume twice the memory since shared pages wouldn't be freed even after they were converted to private.
w/ in-place page conversion
===========================
To support 1G page support forguest_memfd, the current direction is to allowmmap()ofguest_memfdto userspace so that both private and shared memory can use the same physical pages as the backend. This in-place page conversion design eliminates the need to discard pages during shared/private conversions. However, device assignment will still be blocked because the in-place page conversion will reject the conversion when the page is pinned by VFIO.To address this, the key difference lies in the sequence of VFIO map/unmap operations and the page conversion. This series can be adjusted to achieve unmap-before-conversion-to-private and map-after-conversion-to-shared, ensuring compatibility with
guest_memfd.Additionally, with in-place page conversion, the previously mentioned solution to disable the discard of shared pages is not feasible because shared and private memory share the same backend, and no discard operation is performed. Retaining the old mappings in the IOMMU would result in unsafe DMA access to protected memory.
Limitation
==========
Quoted 10 lines
One limitation (also discussed in the
guest_memfdmeeting) is that VFIO expects the DMA mapping for a specific IOVA to be mapped and unmapped with the same granularity. The guest may perform partial conversions, such as converting a small region within a larger region. To prevent such invalid cases, all operations are performed with 4K granularity. The possible solutions we can think of are either to enable VFIO to support partial unmap or to implement an enlightened guest to avoid partial conversion. The former requires complex changes in VFIO, while the latter requires the page conversion to be a guest-enlightened behavior. It is still uncertain which option is a preferred one.
in-place memory conversion is :)
Quoted 6 lines
Testing
=======
This patch series is tested with the KVM/QEMU branch:
KVM: https://github.com/intel/tdx/tree/tdx_kvm_dev-2024-11-20
QEMU: https://github.com/intel-staging/qemu-tdx/tree/tdx-upstream-snapshot-2024-12-13
The branch is gone now? tdx-upstream-snapshot-2024-12-18 seems to have these though. Thanks,
Quoted 59 lines
To facilitate shared device assignment with the NIC, employ the legacy type1 VFIO with the QEMU command:
qemu-system-x86_64 [...]
-device vfio-pci,host=XX:XX.XThe parameter of
dma_entry_limitneeds to be adjusted. For example, a 16GB guest needs to adjust the parameter like vfio_iommu_type1.dma_entry_limit=4194304.If use the iommufd-backed VFIO with the qemu command:
qemu-system-x86_64 [...]
-object iommufd,id=iommufd0 \ -device vfio-pci,host=XX:XX.X,iommufd=iommufd0No additional adjustment required.
Following the bootup of the TD guest, the guest's IP address becomes visible, and iperf is able to successfully send and receive data.
Related link
============
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://docs.google.com/document/d/1M6766BzdY1Lhk7LiR5IqVR8B8mG3cr-cxTxOrAosPOk/edit?tab=t.0#heading=h.jr4csfgw1uql
[4] https://lore.kernel.org/qemu-devel/[email protected]/
[5] https://lore.kernel.org/all/[email protected]/Chenyi Qiang (7):
memory: Export a helper to get intersection of a MemoryRegionSection
with a given range
guest_memfd: Introduce an object to manage the guest-memfd with
RamDiscardManager
guest_memfd: Introduce a callback to notify the shared/private state
change
KVM: Notify the state change event during shared/private conversion
memory: Register the RamDiscardManager instance uponguest_memfd
creation
RAMBlock: makeguest_memfdrequire coordinate discard
memory: Add a new argument to indicate the request attribute in
RamDismcardManager helpersaccel/kvm/kvm-all.c | 4 + hw/vfio/common.c | 22 +- hw/virtio/virtio-mem.c | 55 ++-- include/exec/memory.h | 36 ++- include/sysemu/guest-memfd-manager.h | 91 ++++++ migration/ram.c | 14 +- system/guest-memfd-manager.c | 456 +++++++++++++++++++++++++++ system/memory.c | 30 +- system/memory_mapping.c | 4 +- system/meson.build | 1 + system/physmem.c | 9 +- 11 files changed, 659 insertions(+), 63 deletions(-)create mode 100644
include/sysemu/guest-memfd-manager.hcreate mode 100644system/guest-memfd-manager.c
Signature · 1 line
Alexey
Thanks Alexey for your review!
Quoted 80 lines · Alexey Kardashevskiy
Quoted 79 lines · Chenyi Qiang
Commit
852f0048f3("RAMBlock: makeguest_memfdrequire uncoordinated discard") effectively disables device assignment when usingguest_memfd. This poses a significant challenge asguest_memfdis essential for confidential guests, thereby blocking device assignment to these VMs. The initial rationale for disabling device assignment was due to stale IOMMU mappings (see Problem section) and the assumption that TEE I/O (SEV-TIO, TDX Connect, COVE-IO, etc.) would solve the device-assignment problem for confidential guests [1]. However, this assumption has proven to be incorrect. TEE I/O relies on the ability to operate devices against "shared" or untrusted memory, which is crucial for device initialization and error recovery scenarios. As a result, the current implementation does
not adequately support device assignment for confidential guests, necessitating
a reevaluation of the approach to ensure compatibility and functionality.This series enables shared device assignment by notifying VFIO of page conversions using an existing framework named RamDiscardListener. Additionally, there is an ongoing patch set [2] that aims to add 1G page support for
guest_memfd. This patch set introduces in-place page conversion,
where private and shared memory share the same physical pages as the backend.
This development may impact our solution.We presented our solution in the
guest_memfdmeeting to discuss its compatibility with the new changes and potential future directions (see [3]
for more details). The conclusion was that, although our solution may not be
the most elegant (see the Limitation section), it is sufficient for now and
can be easily adapted to future changes.We are re-posting the patch series with some cleanup and have removed the RFC
label for the main enabling patches (1-6). The newly-added patch 7 is still
marked as RFC as it tries to resolve some extension concerns related to RamDiscardManager for future usage.The overview of the patches:
- Patch 1: Export a helper to get intersection of a MemoryRegionSection with a given range.
- Patch 2-6: Introduce a new object to manage the guest-memfd with RamDiscardManager, and notify the shared/private state change during conversion.
- Patch 7: Try to resolve a semantics concern related to RamDiscardManager
i.e. RamDiscardManager is used to manage memory plug/unplug state instead of shared/private state. It would affect future users of RamDiscardManger in confidential VMs. Attach it behind as a RFC patch[4].Changes since last version:
- Add a patch to export some generic helper functions from virtio-mem code.
- Change the bitmap inguest_memfd_managerfrom default shared to default private. This keeps alignment with virtio-mem that 1-setting in bitmap represents the populated state and may help to export more generic code
if necessary.
- Add the helpers to initialize/uninitialize theguest_memfd_managerinstance
to make it more clear.
- Add a patch to distinguish between the shared/private state change and the memory plug/unplug state change in RamDiscardManager.
- RFC: https://lore.kernel.org/qemu-devel/20240725072118.358923-1-
[email protected]/---
Background
==========
Confidential VMs have two classes of memory: shared and private memory. Shared memory is accessible from the host/VMM while private memory is not. Confidential VMs can decide which memory is shared/private and convert memory between shared/private at runtime.
Quoted 8 lines
Quoted 4 lines
"
guest_memfd" is a new kind of fd whose primary goal is to serve guest private memory. The key differences betweenguest_memfdand normal memfd are thatguest_memfdis spawned by a KVM ioctl, bound to its owner VM and cannot be mapped, read or written by userspace.The "cannot be mapped" seems to be not true soon anymore (if not already).
Exactly, allowing guest_memfd to do mmap is the direction. I mentioned it below with in-place page conversion. Maybe I would move it here to make it more clear.
Quoted 9 lines
Quoted 9 lines
In QEMU's implementation, shared memory is allocated with normal methods (e.g. mmap or fallocate) while private memory is allocated from
guest_memfd. When a VM performs memory conversions, QEMU frees pages viamadvise()or viaPUNCH_HOLEon memfd orguest_memfdfrom one side and allocates new pages from the other side.
[...]
Quoted 27 lines
Quoted 15 lines
One limitation (also discussed in the
guest_memfdmeeting) is that VFIO expects the DMA mapping for a specific IOVA to be mapped and unmapped with
the same granularity. The guest may perform partial conversions, such as converting a small region within a larger region. To prevent such invalid cases, all operations are performed with 4K granularity. The possible solutions we can think of are either to enable VFIO to support partial unmap
or to implement an enlightened guest to avoid partial conversion. The former
requires complex changes in VFIO, while the latter requires the page conversion to be a guest-enlightened behavior. It is still uncertain which
option is a preferred one.in-place memory conversion is :)
Quoted 7 lines
Testing
=======
This patch series is tested with the KVM/QEMU branch:
KVM: https://github.com/intel/tdx/tree/tdx_kvm_dev-2024-11-20
QEMU: https://github.com/intel-staging/qemu-tdx/tree/tdx-upstream-
snapshot-2024-12-13
Quoted 2 lines
The branch is gone now? tdx-upstream-snapshot-2024-12-18 seems to have these though. Thanks,
Thanks for pointing it out. You're right, tdx-upstream-snapshot-2024-12-18 is the latest branch. I added the fixup for patch 1 and forgot to update the change here.
Quoted 24 lines
Quoted 22 lines
To facilitate shared device assignment with the NIC, employ the legacy type1 VFIO with the QEMU command:
qemu-system-x86_64 [...]
-device vfio-pci,host=XX:XX.XThe parameter of
dma_entry_limitneeds to be adjusted. For example, a 16GB guest needs to adjust the parameter like vfio_iommu_type1.dma_entry_limit=4194304.If use the iommufd-backed VFIO with the qemu command:
qemu-system-x86_64 [...]
-object iommufd,id=iommufd0 \
-device vfio-pci,host=XX:XX.X,iommufd=iommufd0No additional adjustment required.
Following the bootup of the TD guest, the guest's IP address becomes visible, and iperf is able to successfully send and receive data.
Quoted 99 lines · Chenyi Qiang
Thanks Alexey for your review!
Quoted 80 lines · Alexey Kardashevskiy
Quoted 79 lines · Chenyi Qiang
Commit
852f0048f3("RAMBlock: makeguest_memfdrequire uncoordinated discard") effectively disables device assignment when usingguest_memfd. This poses a significant challenge asguest_memfdis essential for confidential guests, thereby blocking device assignment to these VMs. The initial rationale for disabling device assignment was due to stale IOMMU mappings (see Problem section) and the assumption that TEE I/O (SEV-TIO, TDX Connect, COVE-IO, etc.) would solve the device-assignment problem for confidential guests [1]. However, this assumption has proven to be incorrect. TEE I/O relies on the ability to operate devices against "shared" or untrusted memory, which is crucial for device initialization and error recovery scenarios. As a result, the current implementation does
not adequately support device assignment for confidential guests, necessitating
a reevaluation of the approach to ensure compatibility and functionality.This series enables shared device assignment by notifying VFIO of page conversions using an existing framework named RamDiscardListener. Additionally, there is an ongoing patch set [2] that aims to add 1G page support for
guest_memfd. This patch set introduces in-place page conversion,
where private and shared memory share the same physical pages as the backend.
This development may impact our solution.We presented our solution in the
guest_memfdmeeting to discuss its compatibility with the new changes and potential future directions (see [3]
for more details). The conclusion was that, although our solution may not be
the most elegant (see the Limitation section), it is sufficient for now and
can be easily adapted to future changes.We are re-posting the patch series with some cleanup and have removed the RFC
label for the main enabling patches (1-6). The newly-added patch 7 is still
marked as RFC as it tries to resolve some extension concerns related to RamDiscardManager for future usage.The overview of the patches:
- Patch 1: Export a helper to get intersection of a MemoryRegionSection with a given range.
- Patch 2-6: Introduce a new object to manage the guest-memfd with RamDiscardManager, and notify the shared/private state change during conversion.
- Patch 7: Try to resolve a semantics concern related to RamDiscardManager
i.e. RamDiscardManager is used to manage memory plug/unplug state instead of shared/private state. It would affect future users of RamDiscardManger in confidential VMs. Attach it behind as a RFC
patch[4].Changes since last version:
- Add a patch to export some generic helper functions from virtio-mem code.
- Change the bitmap inguest_memfd_managerfrom default shared to default private. This keeps alignment with virtio-mem that 1-setting in bitmap represents the populated state and may help to export more generic
code
if necessary.
- Add the helpers to initialize/uninitialize theguest_memfd_managerinstance
to make it more clear.
- Add a patch to distinguish between the shared/private state change and the memory plug/unplug state change in RamDiscardManager.
- RFC: https://lore.kernel.org/qemu-devel/20240725072118.358923-1-
[email protected]/---
Background
==========
Confidential VMs have two classes of memory: shared and private memory. Shared memory is accessible from the host/VMM while private memory is not. Confidential VMs can decide which memory is shared/private and convert memory between shared/private at runtime.Quoted 8 lines
Quoted 4 lines
"
guest_memfd" is a new kind of fd whose primary goal is to serve guest private memory. The key differences betweenguest_memfdand normal memfd are thatguest_memfdis spawned by a KVM ioctl, bound to its owner VM and cannot be mapped, read or written by userspace.The "cannot be mapped" seems to be not true soon anymore (if not already).
Exactly, allowing
guest_memfdto do mmap is the direction. I mentioned it below with in-place page conversion. Maybe I would move it here to make it more clear.Quoted 3 lines
Quoted 3 lines
Quoted 18 lines
Quoted 6 lines
Quoted 6 lines
In QEMU's implementation, shared memory is allocated with normal methods (e.g. mmap or fallocate) while private memory is allocated from
guest_memfd. When a VM performs memory conversions, QEMU frees pages viamadvise()or viaPUNCH_HOLEon memfd orguest_memfdfrom one side and allocates new pages from the other side.[...]
Quoted 9 lines
Quoted 9 lines
One limitation (also discussed in the
guest_memfdmeeting) is that VFIO expects the DMA mapping for a specific IOVA to be mapped and unmapped with
the same granularity. The guest may perform partial conversions, such as converting a small region within a larger region. To prevent such invalid cases, all operations are performed with 4K granularity. The possible solutions we can think of are either to enable VFIO to support partial unmap
btw the old VFIO does not split mappings but iommufd seems to be capable of it - there is iopt_area_split(). What happens if you try unmapping a smaller chunk that does not exactly match any mapped chunk? thanks,
Signature · 1 line
Alexey
Quoted 138 lines · Alexey Kardashevskiy
Quoted 121 lines · Chenyi Qiang
Thanks Alexey for your review!
Quoted 97 lines · Alexey Kardashevskiy
Quoted 89 lines · Chenyi Qiang
Commit
852f0048f3("RAMBlock: makeguest_memfdrequire uncoordinated discard") effectively disables device assignment when usingguest_memfd.
This poses a significant challenge asguest_memfdis essential for confidential guests, thereby blocking device assignment to these VMs. The initial rationale for disabling device assignment was due to stale IOMMU mappings (see Problem section) and the assumption that TEE I/O (SEV-TIO, TDX Connect, COVE-IO, etc.) would solve the device-assignment problem for confidential guests [1]. However, this assumption has proven
to be incorrect. TEE I/O relies on the ability to operate devices against
"shared" or untrusted memory, which is crucial for device initialization
and error recovery scenarios. As a result, the current implementation does
not adequately support device assignment for confidential guests, necessitating
a reevaluation of the approach to ensure compatibility and functionality.This series enables shared device assignment by notifying VFIO of page conversions using an existing framework named RamDiscardListener. Additionally, there is an ongoing patch set [2] that aims to add 1G page
support forguest_memfd. This patch set introduces in-place page conversion,
where private and shared memory share the same physical pages as the backend.
This development may impact our solution.We presented our solution in the
guest_memfdmeeting to discuss its compatibility with the new changes and potential future directions (see [3]
for more details). The conclusion was that, although our solution may not be
the most elegant (see the Limitation section), it is sufficient for now and
can be easily adapted to future changes.We are re-posting the patch series with some cleanup and have removed the RFC
label for the main enabling patches (1-6). The newly-added patch 7 is still
marked as RFC as it tries to resolve some extension concerns related to RamDiscardManager for future usage.The overview of the patches:
- Patch 1: Export a helper to get intersection of a MemoryRegionSection with a given range.
- Patch 2-6: Introduce a new object to manage the guest-memfd with RamDiscardManager, and notify the shared/private state change during
conversion.
- Patch 7: Try to resolve a semantics concern related to RamDiscardManager
i.e. RamDiscardManager is used to manage memory plug/unplug state instead of shared/private state. It would affect future users of RamDiscardManger in confidential VMs. Attach it behind as a RFC patch[4].Changes since last version:
- Add a patch to export some generic helper functions from virtio-mem code.
- Change the bitmap inguest_memfd_managerfrom default shared to default
private. This keeps alignment with virtio-mem that 1-setting in bitmap
represents the populated state and may help to export more generic code
if necessary.
- Add the helpers to initialize/uninitialize theguest_memfd_managerinstance
to make it more clear.
- Add a patch to distinguish between the shared/private state change and
the memory plug/unplug state change in RamDiscardManager.
- RFC: https://lore.kernel.org/qemu-devel/20240725072118.358923-1-
[email protected]/---
Background
==========
Confidential VMs have two classes of memory: shared and private memory. Shared memory is accessible from the host/VMM while private memory is not. Confidential VMs can decide which memory is shared/private and convert memory between shared/private at runtime.Quoted 6 lines
"
guest_memfd" is a new kind of fd whose primary goal is to serve guest private memory. The key differences betweenguest_memfdand normal memfd
are thatguest_memfdis spawned by a KVM ioctl, bound to its owner VM and
cannot be mapped, read or written by userspace.Quoted 4 lines
The "cannot be mapped" seems to be not true soon anymore (if not already).
Exactly, allowing
guest_memfdto do mmap is the direction. I mentioned it below with in-place page conversion. Maybe I would move it here to make it more clear.Quoted 11 lines
Quoted 11 lines
In QEMU's implementation, shared memory is allocated with normal methods
(e.g. mmap or fallocate) while private memory is allocated fromguest_memfd. When a VM performs memory conversions, QEMU frees pages viamadvise()or viaPUNCH_HOLEon memfd orguest_memfdfrom one side and allocates new pages from the other side.Quoted 13 lines
[...]
Quoted 11 lines
Quoted 11 lines
One limitation (also discussed in the
guest_memfdmeeting) is that VFIO expects the DMA mapping for a specific IOVA to be mapped and unmapped with
the same granularity. The guest may perform partial conversions, such as
converting a small region within a larger region. To prevent such invalid
cases, all operations are performed with 4K granularity. The possible solutions we can think of are either to enable VFIO to support partial unmap
Quoted 3 lines
btw the old VFIO does not split mappings but iommufd seems to be capable of it - there is
iopt_area_split(). What happens if you try unmapping a smaller chunk that does not exactly match any mapped chunk? thanks,
iopt_cut_iova() happens in iommufd vfio_compat.c, which is to make iommufd be compatible with old VFIO_TYPE1. IIUC, it happens with disable_large_page=true. That means the large IOPTE is also disabled in IOMMU. So it can do the split easily. See the comment in iommufd_vfio_set_iommu().
iommufd VFIO compatible mode is a transition from legacy VFIO to iommufd. For the normal iommufd, it requires the iova/length must be a superset of a previously mapped range. If not match, will return error.
Quoted 2 lines
Quoted 151 lines · Chenyi Qiang
Quoted 138 lines · Alexey Kardashevskiy
Quoted 121 lines · Chenyi Qiang
Thanks Alexey for your review!
Quoted 97 lines · Alexey Kardashevskiy
Quoted 89 lines · Chenyi Qiang
Commit
852f0048f3("RAMBlock: makeguest_memfdrequire uncoordinated discard") effectively disables device assignment when usingguest_memfd.
This poses a significant challenge asguest_memfdis essential for confidential guests, thereby blocking device assignment to these VMs. The initial rationale for disabling device assignment was due to stale IOMMU mappings (see Problem section) and the assumption that TEE I/O (SEV-TIO, TDX Connect, COVE-IO, etc.) would solve the device-assignment problem for confidential guests [1]. However, this assumption has proven
to be incorrect. TEE I/O relies on the ability to operate devices against
"shared" or untrusted memory, which is crucial for device initialization
and error recovery scenarios. As a result, the current implementation does
not adequately support device assignment for confidential guests, necessitating
a reevaluation of the approach to ensure compatibility and functionality.This series enables shared device assignment by notifying VFIO of page conversions using an existing framework named RamDiscardListener. Additionally, there is an ongoing patch set [2] that aims to add 1G page
support forguest_memfd. This patch set introduces in-place page conversion,
where private and shared memory share the same physical pages as the backend.
This development may impact our solution.We presented our solution in the
guest_memfdmeeting to discuss its compatibility with the new changes and potential future directions (see [3]
for more details). The conclusion was that, although our solution may not be
the most elegant (see the Limitation section), it is sufficient for now and
can be easily adapted to future changes.We are re-posting the patch series with some cleanup and have removed the RFC
label for the main enabling patches (1-6). The newly-added patch 7 is still
marked as RFC as it tries to resolve some extension concerns related to RamDiscardManager for future usage.The overview of the patches:
- Patch 1: Export a helper to get intersection of a MemoryRegionSection with a given range.
- Patch 2-6: Introduce a new object to manage the guest-memfd with RamDiscardManager, and notify the shared/private state change
during
conversion.
- Patch 7: Try to resolve a semantics concern related to RamDiscardManager
i.e. RamDiscardManager is used to manage memory plug/unplug state instead of shared/private state. It would affect future users of RamDiscardManger in confidential VMs. Attach it behind as a RFC
patch[4].Changes since last version:
- Add a patch to export some generic helper functions from virtio-mem code.
- Change the bitmap inguest_memfd_managerfrom default shared to default
private. This keeps alignment with virtio-mem that 1-setting in
bitmap
represents the populated state and may help to export more generic
code
if necessary.
- Add the helpers to initialize/uninitialize theguest_memfd_managerinstance
to make it more clear.
- Add a patch to distinguish between the shared/private state change and
the memory plug/unplug state change in RamDiscardManager.
- RFC: https://lore.kernel.org/qemu-devel/20240725072118.358923-1-
[email protected]/---
Background
==========
Confidential VMs have two classes of memory: shared and private memory. Shared memory is accessible from the host/VMM while private memory is not. Confidential VMs can decide which memory is shared/private and convert memory between shared/private at runtime.Quoted 6 lines
"
guest_memfd" is a new kind of fd whose primary goal is to serve guest private memory. The key differences betweenguest_memfdand normal memfd
are thatguest_memfdis spawned by a KVM ioctl, bound to its owner VM and
cannot be mapped, read or written by userspace.Quoted 4 lines
The "cannot be mapped" seems to be not true soon anymore (if not already).
Exactly, allowing
guest_memfdto do mmap is the direction. I mentioned it below with in-place page conversion. Maybe I would move it here to make it more clear.Quoted 11 lines
Quoted 11 lines
In QEMU's implementation, shared memory is allocated with normal methods
(e.g. mmap or fallocate) while private memory is allocated fromguest_memfd. When a VM performs memory conversions, QEMU frees pages viamadvise()or viaPUNCH_HOLEon memfd orguest_memfdfrom one side and allocates new pages from the other side.Quoted 13 lines
[...]
Quoted 11 lines
Quoted 11 lines
One limitation (also discussed in the
guest_memfdmeeting) is that VFIO expects the DMA mapping for a specific IOVA to be mapped and unmapped with
the same granularity. The guest may perform partial conversions, such as
converting a small region within a larger region. To prevent such invalid
cases, all operations are performed with 4K granularity. The possible solutions we can think of are either to enable VFIO to support partial unmapQuoted 3 lines
btw the old VFIO does not split mappings but iommufd seems to be capable of it - there is
iopt_area_split(). What happens if you try unmapping a smaller chunk that does not exactly match any mapped chunk? thanks,
iopt_cut_iova()happens in iommufdvfio_compat.c, which is to make iommufd be compatible with oldVFIO_TYPE1. IIUC, it happens with disable_large_page=true. That means the large IOPTE is also disabled in IOMMU. So it can do the split easily. See the comment iniommufd_vfio_set_iommu().
Quoted 3 lines
iommufd VFIO compatible mode is a transition from legacy VFIO to iommufd. For the normal iommufd, it requires the iova/length must be a superset of a previously mapped range. If not match, will return error.
This is all true but this also means that "The former requires complex changes in VFIO" is not entirely true - some code is already there. Thanks,
Signature · 1 line
Alexey
Quoted 166 lines · Alexey Kardashevskiy
Quoted 163 lines · Chenyi Qiang
Quoted 150 lines · Alexey Kardashevskiy
Quoted 131 lines · Chenyi Qiang
Thanks Alexey for your review!
Quoted 106 lines · Alexey Kardashevskiy
Quoted 97 lines · Chenyi Qiang
Commit
852f0048f3("RAMBlock: makeguest_memfdrequire uncoordinated discard") effectively disables device assignment when usingguest_memfd.
This poses a significant challenge asguest_memfdis essential for confidential guests, thereby blocking device assignment to these VMs. The initial rationale for disabling device assignment was due to stale
IOMMU mappings (see Problem section) and the assumption that TEE I/O (SEV-TIO, TDX Connect, COVE-IO, etc.) would solve the device- assignment
problem for confidential guests [1]. However, this assumption has proven
to be incorrect. TEE I/O relies on the ability to operate devices against
"shared" or untrusted memory, which is crucial for device initialization
and error recovery scenarios. As a result, the current implementation does
not adequately support device assignment for confidential guests, necessitating
a reevaluation of the approach to ensure compatibility and functionality.This series enables shared device assignment by notifying VFIO of page
conversions using an existing framework named RamDiscardListener. Additionally, there is an ongoing patch set [2] that aims to add 1G page
support forguest_memfd. This patch set introduces in-place page conversion,
where private and shared memory share the same physical pages as the backend.
This development may impact our solution.We presented our solution in the
guest_memfdmeeting to discuss its compatibility with the new changes and potential future directions (see [3]
for more details). The conclusion was that, although our solution may not be
the most elegant (see the Limitation section), it is sufficient for now and
can be easily adapted to future changes.We are re-posting the patch series with some cleanup and have removed the RFC
label for the main enabling patches (1-6). The newly-added patch 7 is still
marked as RFC as it tries to resolve some extension concerns related to
RamDiscardManager for future usage.The overview of the patches:
- Patch 1: Export a helper to get intersection of a MemoryRegionSection
with a given range.
- Patch 2-6: Introduce a new object to manage the guest-memfd with RamDiscardManager, and notify the shared/private state change during
conversion.
- Patch 7: Try to resolve a semantics concern related to RamDiscardManager
i.e. RamDiscardManager is used to manage memory plug/unplug state
instead of shared/private state. It would affect future users of RamDiscardManger in confidential VMs. Attach it behind as a RFC patch[4].Changes since last version:
- Add a patch to export some generic helper functions from virtio-mem code.
- Change the bitmap inguest_memfd_managerfrom default shared to default
private. This keeps alignment with virtio-mem that 1-setting in bitmap
represents the populated state and may help to export more generic
code
if necessary.
- Add the helpers to initialize/uninitialize theguest_memfd_managerinstance
to make it more clear.
- Add a patch to distinguish between the shared/private state change and
the memory plug/unplug state change in RamDiscardManager.
- RFC: https://lore.kernel.org/qemu-devel/20240725072118.358923-1-
[email protected]/---
Background
==========
Confidential VMs have two classes of memory: shared and private memory.
Shared memory is accessible from the host/VMM while private memory is not. Confidential VMs can decide which memory is shared/private and convert memory between shared/private at runtime.Quoted 7 lines
"
guest_memfd" is a new kind of fd whose primary goal is to serve guest
private memory. The key differences betweenguest_memfdand normal memfd
are thatguest_memfdis spawned by a KVM ioctl, bound to its owner VM and
cannot be mapped, read or written by userspace.Quoted 5 lines
The "cannot be mapped" seems to be not true soon anymore (if not already).
https://lore.kernel.org/all/20240801090117.3841080-1-
[email protected]/T/Exactly, allowing
guest_memfdto do mmap is the direction. I mentioned it below with in-place page conversion. Maybe I would move it here to make it more clear.Quoted 11 lines
Quoted 11 lines
In QEMU's implementation, shared memory is allocated with normal methods
(e.g. mmap or fallocate) while private memory is allocated fromguest_memfd. When a VM performs memory conversions, QEMU frees pages viamadvise()or viaPUNCH_HOLEon memfd orguest_memfdfrom one side and allocates new pages from the other side.Quoted 15 lines
[...]
Quoted 13 lines
Quoted 13 lines
One limitation (also discussed in the
guest_memfdmeeting) is that VFIO
expects the DMA mapping for a specific IOVA to be mapped and unmapped with
the same granularity. The guest may perform partial conversions, such as
converting a small region within a larger region. To prevent such invalid
cases, all operations are performed with 4K granularity. The possible solutions we can think of are either to enable VFIO to support partial
unmapQuoted 3 lines
btw the old VFIO does not split mappings but iommufd seems to be capable of it - there is
iopt_area_split(). What happens if you try unmapping a smaller chunk that does not exactly match any mapped chunk? thanks,
iopt_cut_iova()happens in iommufdvfio_compat.c, which is to make iommufd be compatible with oldVFIO_TYPE1. IIUC, it happens with disable_large_page=true. That means the large IOPTE is also disabled in IOMMU. So it can do the split easily. See the comment iniommufd_vfio_set_iommu().
Quoted 7 lines
Quoted 3 lines
iommufd VFIO compatible mode is a transition from legacy VFIO to iommufd. For the normal iommufd, it requires the iova/length must be a superset of a previously mapped range. If not match, will return error.
This is all true but this also means that "The former requires complex changes in VFIO" is not entirely true - some code is already there. Thanks,
Hmm, my statement is a little confusing. The bottleneck is that the IOMMU driver doesn't support the large page split. So if we want to enable large page and want to do partial unmap, it requires complex change.
Quoted 3 lines
Quoted 174 lines · Chenyi Qiang
Quoted 171 lines · Alexey Kardashevskiy
Quoted 166 lines · Chenyi Qiang
Quoted 150 lines · Alexey Kardashevskiy
Quoted 131 lines · Chenyi Qiang
Thanks Alexey for your review!
Quoted 106 lines · Alexey Kardashevskiy
Quoted 97 lines · Chenyi Qiang
Commit
852f0048f3("RAMBlock: makeguest_memfdrequire uncoordinated discard") effectively disables device assignment when usingguest_memfd.
This poses a significant challenge asguest_memfdis essential for confidential guests, thereby blocking device assignment to these VMs. The initial rationale for disabling device assignment was due to stale
IOMMU mappings (see Problem section) and the assumption that TEE I/O (SEV-TIO, TDX Connect, COVE-IO, etc.) would solve the device- assignment
problem for confidential guests [1]. However, this assumption has proven
to be incorrect. TEE I/O relies on the ability to operate devices against
"shared" or untrusted memory, which is crucial for device initialization
and error recovery scenarios. As a result, the current implementation does
not adequately support device assignment for confidential guests, necessitating
a reevaluation of the approach to ensure compatibility and functionality.This series enables shared device assignment by notifying VFIO of page
conversions using an existing framework named RamDiscardListener. Additionally, there is an ongoing patch set [2] that aims to add 1G page
support forguest_memfd. This patch set introduces in-place page conversion,
where private and shared memory share the same physical pages as the backend.
This development may impact our solution.We presented our solution in the
guest_memfdmeeting to discuss its compatibility with the new changes and potential future directions (see [3]
for more details). The conclusion was that, although our solution may not be
the most elegant (see the Limitation section), it is sufficient for now and
can be easily adapted to future changes.We are re-posting the patch series with some cleanup and have removed the RFC
label for the main enabling patches (1-6). The newly-added patch 7 is still
marked as RFC as it tries to resolve some extension concerns related to
RamDiscardManager for future usage.The overview of the patches:
- Patch 1: Export a helper to get intersection of a MemoryRegionSection
with a given range.
- Patch 2-6: Introduce a new object to manage the guest-memfd with RamDiscardManager, and notify the shared/private state change
during
conversion.
- Patch 7: Try to resolve a semantics concern related to RamDiscardManager
i.e. RamDiscardManager is used to manage memory plug/unplug
state
instead of shared/private state. It would affect future users of RamDiscardManger in confidential VMs. Attach it behind as a RFC
patch[4].Changes since last version:
- Add a patch to export some generic helper functions from virtio-mem code.
- Change the bitmap inguest_memfd_managerfrom default shared to default
private. This keeps alignment with virtio-mem that 1-setting in
bitmap
represents the populated state and may help to export more
generic
code
if necessary.
- Add the helpers to initialize/uninitialize theguest_memfd_managerinstance
to make it more clear.
- Add a patch to distinguish between the shared/private state change and
the memory plug/unplug state change in RamDiscardManager.
- RFC: https://lore.kernel.org/qemu-devel/20240725072118.358923-1-
[email protected]/---
Background
==========
Confidential VMs have two classes of memory: shared and private memory.
Shared memory is accessible from the host/VMM while private memory is not. Confidential VMs can decide which memory is shared/private and convert memory between shared/private at runtime.Quoted 7 lines
"
guest_memfd" is a new kind of fd whose primary goal is to serve guest
private memory. The key differences betweenguest_memfdand normal memfd
are thatguest_memfdis spawned by a KVM ioctl, bound to its owner VM and
cannot be mapped, read or written by userspace.Quoted 5 lines
The "cannot be mapped" seems to be not true soon anymore (if not already).
https://lore.kernel.org/all/20240801090117.3841080-1-
[email protected]/T/Exactly, allowing
guest_memfdto do mmap is the direction. I mentioned it below with in-place page conversion. Maybe I would move it here to make it more clear.Quoted 11 lines
Quoted 11 lines
In QEMU's implementation, shared memory is allocated with normal methods
(e.g. mmap or fallocate) while private memory is allocated fromguest_memfd. When a VM performs memory conversions, QEMU frees pages viamadvise()or viaPUNCH_HOLEon memfd orguest_memfdfrom one side and allocates new pages from the other side.Quoted 15 lines
[...]
Quoted 13 lines
Quoted 13 lines
One limitation (also discussed in the
guest_memfdmeeting) is that VFIO
expects the DMA mapping for a specific IOVA to be mapped and unmapped with
the same granularity. The guest may perform partial conversions, such as
converting a small region within a larger region. To prevent such invalid
cases, all operations are performed with 4K granularity. The possible solutions we can think of are either to enable VFIO to support partial
unmapQuoted 3 lines
btw the old VFIO does not split mappings but iommufd seems to be capable of it - there is
iopt_area_split(). What happens if you try unmapping a smaller chunk that does not exactly match any mapped chunk? thanks,
iopt_cut_iova()happens in iommufdvfio_compat.c, which is to make iommufd be compatible with oldVFIO_TYPE1. IIUC, it happens with disable_large_page=true. That means the large IOPTE is also disabled in IOMMU. So it can do the split easily. See the comment iniommufd_vfio_set_iommu().iommufd VFIO compatible mode is a transition from legacy VFIO to iommufd. For the normal iommufd, it requires the iova/length must be a superset of a previously mapped range. If not match, will return error.
Quoted 6 lines
Quoted 2 lines
This is all true but this also means that "The former requires complex changes in VFIO" is not entirely true - some code is already there. Thanks,
Hmm, my statement is a little confusing. The bottleneck is that the IOMMU driver doesn't support the large page split. So if we want to enable large page and want to do partial unmap, it requires complex change.
We won't need to split large pages (if we stick to 4K for now), we need to split large mappings (not large pages) to allow partial unmapping and iopt_area_split() seems to be doing this. Thanks,
Quoted 5 lines
Quoted 4 lines
Signature · 1 line
Alexey
Quoted 203 lines · Alexey Kardashevskiy
Quoted 200 lines · Chenyi Qiang
Quoted 196 lines · Alexey Kardashevskiy
Quoted 185 lines · Chenyi Qiang
Quoted 170 lines · Alexey Kardashevskiy
Quoted 149 lines · Chenyi Qiang
Thanks Alexey for your review!
Quoted 121 lines · Alexey Kardashevskiy
Quoted 112 lines · Chenyi Qiang
Commit
852f0048f3("RAMBlock: makeguest_memfdrequire uncoordinated
discard") effectively disables device assignment when usingguest_memfd.
This poses a significant challenge asguest_memfdis essential for confidential guests, thereby blocking device assignment to these VMs.
The initial rationale for disabling device assignment was due to stale
IOMMU mappings (see Problem section) and the assumption that TEE I/O
(SEV-TIO, TDX Connect, COVE-IO, etc.) would solve the device- assignment
problem for confidential guests [1]. However, this assumption has proven
to be incorrect. TEE I/O relies on the ability to operate devices against
"shared" or untrusted memory, which is crucial for device initialization
and error recovery scenarios. As a result, the current implementation
does
not adequately support device assignment for confidential guests, necessitating
a reevaluation of the approach to ensure compatibility and functionality.This series enables shared device assignment by notifying VFIO of page
conversions using an existing framework named RamDiscardListener. Additionally, there is an ongoing patch set [2] that aims to add 1G page
support forguest_memfd. This patch set introduces in-place page conversion,
where private and shared memory share the same physical pages as the
backend.
This development may impact our solution.We presented our solution in the
guest_memfdmeeting to discuss its compatibility with the new changes and potential future directions (see [3]
for more details). The conclusion was that, although our solution may
not be
the most elegant (see the Limitation section), it is sufficient for now and
can be easily adapted to future changes.We are re-posting the patch series with some cleanup and have removed
the RFC
label for the main enabling patches (1-6). The newly-added patch 7 is
still
marked as RFC as it tries to resolve some extension concerns related to
RamDiscardManager for future usage.The overview of the patches:
- Patch 1: Export a helper to get intersection of a MemoryRegionSection
with a given range.
- Patch 2-6: Introduce a new object to manage the guest-memfd with RamDiscardManager, and notify the shared/private state change during
conversion.
- Patch 7: Try to resolve a semantics concern related to RamDiscardManager
i.e. RamDiscardManager is used to manage memory plug/unplug state
instead of shared/private state. It would affect future users of
RamDiscardManger in confidential VMs. Attach it behind as a RFC
patch[4].Changes since last version:
- Add a patch to export some generic helper functions from virtio-mem
code.
- Change the bitmap inguest_memfd_managerfrom default shared to default
private. This keeps alignment with virtio-mem that 1- setting in
bitmap
represents the populated state and may help to export more generic
code
if necessary.
- Add the helpers to initialize/uninitialize theguest_memfd_manager
instance
to make it more clear.
- Add a patch to distinguish between the shared/private state change
and
the memory plug/unplug state change in RamDiscardManager.
- RFC: https://lore.kernel.org/qemu-devel/20240725072118.358923-1-
[email protected]/---
Background
==========
Confidential VMs have two classes of memory: shared and private memory.
Shared memory is accessible from the host/VMM while private memory is
not. Confidential VMs can decide which memory is shared/private and convert memory between shared/private at runtime.Quoted 7 lines
"
guest_memfd" is a new kind of fd whose primary goal is to serve guest
private memory. The key differences betweenguest_memfdand normal memfd
are thatguest_memfdis spawned by a KVM ioctl, bound to its owner VM and
cannot be mapped, read or written by userspace.Quoted 5 lines
The "cannot be mapped" seems to be not true soon anymore (if not already).
https://lore.kernel.org/all/20240801090117.3841080-1-
[email protected]/T/Exactly, allowing
guest_memfdto do mmap is the direction. I mentioned
it below with in-place page conversion. Maybe I would move it here to make it more clear.Quoted 13 lines
Quoted 13 lines
In QEMU's implementation, shared memory is allocated with normal methods
(e.g. mmap or fallocate) while private memory is allocated fromguest_memfd. When a VM performs memory conversions, QEMU frees pages
viamadvise()or viaPUNCH_HOLEon memfd orguest_memfdfrom one side and
allocates new pages from the other side.Quoted 17 lines
[...]
Quoted 15 lines
Quoted 15 lines
One limitation (also discussed in the
guest_memfdmeeting) is that VFIO
expects the DMA mapping for a specific IOVA to be mapped and unmapped
with
the same granularity. The guest may perform partial conversions, such as
converting a small region within a larger region. To prevent such invalid
cases, all operations are performed with 4K granularity. The possible
solutions we can think of are either to enable VFIO to support partial
unmapQuoted 5 lines
btw the old VFIO does not split mappings but iommufd seems to be capable
of it - there isiopt_area_split(). What happens if you try unmapping a
smaller chunk that does not exactly match any mapped chunk? thanks,
iopt_cut_iova()happens in iommufdvfio_compat.c, which is to make iommufd be compatible with oldVFIO_TYPE1. IIUC, it happens with disable_large_page=true. That means the large IOPTE is also disabled in IOMMU. So it can do the split easily. See the comment iniommufd_vfio_set_iommu().Quoted 3 lines
iommufd VFIO compatible mode is a transition from legacy VFIO to iommufd. For the normal iommufd, it requires the iova/length must be a superset of a previously mapped range. If not match, will return error.
This is all true but this also means that "The former requires complex changes in VFIO" is not entirely true - some code is already there. Thanks,
Quoted 8 lines
Quoted 4 lines
Hmm, my statement is a little confusing. The bottleneck is that the IOMMU driver doesn't support the large page split. So if we want to enable large page and want to do partial unmap, it requires complex change.
We won't need to split large pages (if we stick to 4K for now), we need to split large mappings (not large pages) to allow partial unmapping and
iopt_area_split()seems to be doing this. Thanks,
You mean we can disable large page in iommufd and then VFIO will be able to do partial unmap. Yes, I think it is doable and we can avoid many ioctl context switches overhead.
Quoted 8 lines
Quoted 7 lines
Quoted 6 lines
Quoted 211 lines · Chenyi Qiang
Quoted 208 lines · Alexey Kardashevskiy
Quoted 204 lines · Chenyi Qiang
Quoted 188 lines · Alexey Kardashevskiy
Quoted 185 lines · Chenyi Qiang
Quoted 170 lines · Alexey Kardashevskiy
Quoted 149 lines · Chenyi Qiang
Thanks Alexey for your review!
Quoted 121 lines · Alexey Kardashevskiy
Quoted 112 lines · Chenyi Qiang
Commit
852f0048f3("RAMBlock: makeguest_memfdrequire uncoordinated
discard") effectively disables device assignment when usingguest_memfd.
This poses a significant challenge asguest_memfdis essential for confidential guests, thereby blocking device assignment to these VMs.
The initial rationale for disabling device assignment was due to stale
IOMMU mappings (see Problem section) and the assumption that TEE I/O
(SEV-TIO, TDX Connect, COVE-IO, etc.) would solve the device- assignment
problem for confidential guests [1]. However, this assumption has proven
to be incorrect. TEE I/O relies on the ability to operate devices against
"shared" or untrusted memory, which is crucial for device initialization
and error recovery scenarios. As a result, the current implementation
does
not adequately support device assignment for confidential guests, necessitating
a reevaluation of the approach to ensure compatibility and functionality.This series enables shared device assignment by notifying VFIO of page
conversions using an existing framework named RamDiscardListener. Additionally, there is an ongoing patch set [2] that aims to add 1G page
support forguest_memfd. This patch set introduces in-place page conversion,
where private and shared memory share the same physical pages as the
backend.
This development may impact our solution.We presented our solution in the
guest_memfdmeeting to discuss its compatibility with the new changes and potential future directions (see [3]
for more details). The conclusion was that, although our solution may
not be
the most elegant (see the Limitation section), it is sufficient for now and
can be easily adapted to future changes.We are re-posting the patch series with some cleanup and have removed
the RFC
label for the main enabling patches (1-6). The newly-added patch 7 is
still
marked as RFC as it tries to resolve some extension concerns related to
RamDiscardManager for future usage.The overview of the patches:
- Patch 1: Export a helper to get intersection of a MemoryRegionSection
with a given range.
- Patch 2-6: Introduce a new object to manage the guest-memfd with RamDiscardManager, and notify the shared/private state change
during
conversion.
- Patch 7: Try to resolve a semantics concern related to RamDiscardManager
i.e. RamDiscardManager is used to manage memory plug/unplug
state
instead of shared/private state. It would affect future
users of
RamDiscardManger in confidential VMs. Attach it behind as
a RFC
patch[4].Changes since last version:
- Add a patch to export some generic helper functions from virtio-mem
code.
- Change the bitmap inguest_memfd_managerfrom default shared to default
private. This keeps alignment with virtio-mem that 1-
setting in
bitmap
represents the populated state and may help to export more
generic
code
if necessary.
- Add the helpers to initialize/uninitialize theguest_memfd_manager
instance
to make it more clear.
- Add a patch to distinguish between the shared/private state change
and
the memory plug/unplug state change in RamDiscardManager.
- RFC: https://lore.kernel.org/qemu-devel/20240725072118.358923-1-
[email protected]/---
Background
==========
Confidential VMs have two classes of memory: shared and private memory.
Shared memory is accessible from the host/VMM while private memory is
not. Confidential VMs can decide which memory is shared/private and convert memory between shared/private at runtime.Quoted 7 lines
"
guest_memfd" is a new kind of fd whose primary goal is to serve guest
private memory. The key differences betweenguest_memfdand normal memfd
are thatguest_memfdis spawned by a KVM ioctl, bound to its owner VM and
cannot be mapped, read or written by userspace.Quoted 5 lines
The "cannot be mapped" seems to be not true soon anymore (if not already).
https://lore.kernel.org/all/20240801090117.3841080-1-
[email protected]/T/Exactly, allowing
guest_memfdto do mmap is the direction. I mentioned
it below with in-place page conversion. Maybe I would move it here to make it more clear.Quoted 13 lines
Quoted 13 lines
In QEMU's implementation, shared memory is allocated with normal methods
(e.g. mmap or fallocate) while private memory is allocated fromguest_memfd. When a VM performs memory conversions, QEMU frees pages
viamadvise()or viaPUNCH_HOLEon memfd orguest_memfdfrom one side and
allocates new pages from the other side.Quoted 17 lines
[...]
Quoted 15 lines
Quoted 15 lines
One limitation (also discussed in the
guest_memfdmeeting) is that VFIO
expects the DMA mapping for a specific IOVA to be mapped and unmapped
with
the same granularity. The guest may perform partial conversions, such as
converting a small region within a larger region. To prevent such invalid
cases, all operations are performed with 4K granularity. The possible
solutions we can think of are either to enable VFIO to support partial
unmapQuoted 5 lines
btw the old VFIO does not split mappings but iommufd seems to be capable
of it - there isiopt_area_split(). What happens if you try unmapping a
smaller chunk that does not exactly match any mapped chunk? thanks,
iopt_cut_iova()happens in iommufdvfio_compat.c, which is to make iommufd be compatible with oldVFIO_TYPE1. IIUC, it happens with disable_large_page=true. That means the large IOPTE is also disabled in IOMMU. So it can do the split easily. See the comment iniommufd_vfio_set_iommu().Quoted 8 lines
Quoted 3 lines
iommufd VFIO compatible mode is a transition from legacy VFIO to iommufd. For the normal iommufd, it requires the iova/length must be a superset of a previously mapped range. If not match, will return error.
This is all true but this also means that "The former requires complex changes in VFIO" is not entirely true - some code is already there. Thanks,
Hmm, my statement is a little confusing. The bottleneck is that the IOMMU driver doesn't support the large page split. So if we want to enable large page and want to do partial unmap, it requires complex change.
Quoted 7 lines
Quoted 3 lines
We won't need to split large pages (if we stick to 4K for now), we need to split large mappings (not large pages) to allow partial unmapping and
iopt_area_split()seems to be doing this. Thanks,You mean we can disable large page in iommufd and then VFIO will be able to do partial unmap. Yes, I think it is doable and we can avoid many ioctl context switches overhead.
So I understand this correctly: the disable_large_pages=true will imply that we never have PMD mappings such that we can atomically poke a hole in a mapping, without temporarily having to remove a PMD mapping in the iommu table to insert a PTE table?
batch_iommu_map_small() seems to document that behavior.
It's interesting that that comment points out that this is purely "VFIO compatibility", and that it otherwise violates the iommufd invariant: "pairing map/unmap". So, it is against the real iommufd design ...
Back when working on virtio-mem support (RAMDiscardManager), thought there was not way to reliably do atomic partial unmappings.
Signature · 3 lines
Cheers, David / dhildenb
Quoted 4 lines · David Hildenbrand
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
One limitation (also discussed in the
guest_memfdmeeting) is that VFIO expects the DMA mapping for a specific IOVA to be mapped and unmapped with the same granularity.
Not just same granularity, whatever you map you have to unmap in whole. map/unmap must be perfectly paired by userspace.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
such as converting a small region within a larger region. To prevent such invalid cases, all operations are performed with 4K granularity. The possible solutions we can think of are either to enable VFIO to support partial unmap
Yes, you can do that, but it is aweful for performance everywhere
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
iopt_cut_iova()happens in iommufdvfio_compat.c, which is to make iommufd be compatible with oldVFIO_TYPE1. IIUC, it happens with disable_large_page=true. That means the large IOPTE is also disabled in IOMMU. So it can do the split easily. See the comment iniommufd_vfio_set_iommu().
Yes. But I am working on a project to make this more general purpose and not have the 4k limitation. There are now several use cases for this kind of cut feature.
https://lore.kernel.org/linux-iommu/[email protected]/
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
This is all true but this also means that "The former requires complex changes in VFIO" is not entirely true - some code is already there.
Well, to do it without forcing 4k requires complex changes.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Hmm, my statement is a little confusing. The bottleneck is that the IOMMU driver doesn't support the large page split. So if we want to enable large page and want to do partial unmap, it requires complex change.
Yes, this is what I'm working on.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
We won't need to split large pages (if we stick to 4K for now), we need to split large mappings (not large pages) to allow partial unmapping and
iopt_area_split()seems to be doing this. Thanks,
Correct
Quoted 3 lines
Quoted 3 lines
You mean we can disable large page in iommufd and then VFIO will be able to do partial unmap. Yes, I think it is doable and we can avoid many ioctl context switches overhead.
Right
Quoted 4 lines
So I understand this correctly: the disable_large_pages=true will imply that we never have PMD mappings such that we can atomically poke a hole in a mapping, without temporarily having to remove a PMD mapping in the iommu table to insert a PTE table?
Yes
Quoted 1 line
batch_iommu_map_small()seems to document that behavior.
Yes
Quoted 3 lines
It's interesting that that comment points out that this is purely "VFIO compatibility", and that it otherwise violates the iommufd invariant:
"pairing map/unmap". So, it is against the real iommufd design ...
IIRC you can only trigger split using the VFIO type 1 legacy API. We would need to formalize split as an IOMMUFD native ioctl.
Nobody should use this stuf through the legacy type 1 API!!!!
Quoted 2 lines
Back when working on virtio-mem support (RAMDiscardManager), thought there was not way to reliably do atomic partial unmappings.
Correct
Jason
On 10.01.25 14:20, Jason Gunthorpe wrote:
Thanks for your reply, I knew CCing you would be very helpful :)
Quoted 8 lines · David Hildenbrand
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
One limitation (also discussed in the
guest_memfdmeeting) is that VFIO expects the DMA mapping for a specific IOVA to be mapped and unmapped with the same granularity.Not just same granularity, whatever you map you have to unmap in whole. map/unmap must be perfectly paired by userspace.
Right, that's what virtio-mem ends up doing by mapping each memory block (e.g., 2 MiB) separately that could be unmapped separately.
It adds "overhead", but at least you don't run into "no, you cannot split this region because you would be out of memory/slots" or in the past issues with concurrent ongoing DMA.
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
such as converting a small region within a larger region. To prevent such invalid cases, all operations are performed with 4K granularity. The possible solutions we can think of are either to enable VFIO to support partial unmap
Yes, you can do that, but it is aweful for performance everywhere
Absolutely.
In your commit I read:
"Implement the cut operation to be hitless, changes to the page table
during cutting must cause zero disruption to any ongoing DMA. This is the expectation of the VFIO type 1 uAPI. Hitless requires HW support, it is incompatible with HW requiring break-before-make."
So I guess that would mean that, depending on HW support, one could avoid disabling large pages to still allow for atomic cuts / partial unmaps that don't affect concurrent DMA.
What would be your suggestion here to avoid the "map each 4k page individually so we can unmap it individually" ? I didn't completely grasp that, sorry.
From "IIRC you can only trigger split using the VFIO type 1 legacy API. We would need to formalize split as an IOMMUFD native ioctl. Nobody should use this stuf through the legacy type 1 API!!!!"
I assume you mean that we can only avoid the 4k map/unmap if we add proper support to IOMMUFD native ioctl, and not try making it fly somehow with the legacy type 1 API?
Signature · 3 lines
Cheers, David / dhildenb
Quoted 11 lines · David Hildenbrand
In your commit I read:
"Implement the cut operation to be hitless, changes to the page table during cutting must cause zero disruption to any ongoing DMA. This is the expectation of the VFIO type 1 uAPI. Hitless requires HW support, it is incompatible with HW requiring break-before-make."
So I guess that would mean that, depending on HW support, one could avoid disabling large pages to still allow for atomic cuts / partial unmaps that don't affect concurrent DMA.
Yes. Most x86 server HW will do this, though ARM support is a bit newish.
Quoted 3 lines
What would be your suggestion here to avoid the "map each 4k page individually so we can unmap it individually" ? I didn't completely grasp that, sorry.
Map in large ranges in the VMM, lets say 1G of shared memory as a single mapping (called an iommufd area)
When the guest makes a 2M chunk of it private you do a ioctl to iommufd to split the area into three, leaving the 2M chunk as a seperate area.
The new iommufd ioctl to split areas will go down into the iommu driver and atomically cut the 1G PTEs into smaller PTEs as necessary so that no PTE spans the edges of the 2M area.
Then userspace can unmap the 2M area and leave the remainder of the 1G area mapped.
All of this would be fully hitless to ongoing DMA.
The iommufs code is there to do this assuming the areas are mapped at 4k, what is missing is the iommu driver side to atomically resize large PTEs.
Quoted 7 lines
From "IIRC you can only trigger split using the VFIO type 1 legacy API. We would need to formalize split as an IOMMUFD native ioctl. Nobody should use this stuf through the legacy type 1 API!!!!"
I assume you mean that we can only avoid the 4k map/unmap if we add proper support to IOMMUFD native ioctl, and not try making it fly somehow with the legacy type 1 API?
The thread was talking about the built-in support in iommufd to split mappings. That built-in support is only accessible through legacy APIs and should never be used in new qemu code. To use that built in support in new code we need to build new APIs. The advantage of the built-in support is qemu can map in large regions (which is more efficient) and the kernel will break it down to 4k for the iommu driver.
Mapping 4k at a time through the uAPI would be outrageously inefficient.
Jason
Quoted 55 lines · Jason Gunthorpe
Quoted 11 lines · David Hildenbrand
In your commit I read:
"Implement the cut operation to be hitless, changes to the page table during cutting must cause zero disruption to any ongoing DMA. This is the expectation of the VFIO type 1 uAPI. Hitless requires HW support, it is incompatible with HW requiring break-before-make."
So I guess that would mean that, depending on HW support, one could avoid disabling large pages to still allow for atomic cuts / partial unmaps that don't affect concurrent DMA.
Yes. Most x86 server HW will do this, though ARM support is a bit newish.
Quoted 3 lines
What would be your suggestion here to avoid the "map each 4k page individually so we can unmap it individually" ? I didn't completely grasp that, sorry.
Map in large ranges in the VMM, lets say 1G of shared memory as a single mapping (called an iommufd area)
When the guest makes a 2M chunk of it private you do a ioctl to iommufd to split the area into three, leaving the 2M chunk as a seperate area.
The new iommufd ioctl to split areas will go down into the iommu driver and atomically cut the 1G PTEs into smaller PTEs as necessary so that no PTE spans the edges of the 2M area.
Then userspace can unmap the 2M area and leave the remainder of the 1G area mapped.
All of this would be fully hitless to ongoing DMA.
The iommufs code is there to do this assuming the areas are mapped at 4k, what is missing is the iommu driver side to atomically resize large PTEs.
Quoted 7 lines
From "IIRC you can only trigger split using the VFIO type 1 legacy API. We
would need to formalize split as an IOMMUFD native ioctl. Nobody should use this stuf through the legacy type 1 API!!!!"I assume you mean that we can only avoid the 4k map/unmap if we add proper support to IOMMUFD native ioctl, and not try making it fly somehow with the legacy type 1 API?
The thread was talking about the built-in support in iommufd to split mappings. That built-in support is only accessible through legacy APIs and should never be used in new qemu code. To use that built in support in new code we need to build new APIs. The advantage of the built-in support is qemu can map in large regions (which is more efficient) and the kernel will break it down to 4k for the iommu driver.
Quoted 2 lines
Mapping 4k at a time through the uAPI would be outrageously inefficient.
Got it, makes all sense, thanks!
Signature · 3 lines
Cheers, David / dhildenb
Quoted 43 lines · Jason Gunthorpe
Quoted 11 lines · David Hildenbrand
In your commit I read:
"Implement the cut operation to be hitless, changes to the page table during cutting must cause zero disruption to any ongoing DMA. This is the expectation of the VFIO type 1 uAPI. Hitless requires HW support, it is incompatible with HW requiring break-before-make."
So I guess that would mean that, depending on HW support, one could avoid disabling large pages to still allow for atomic cuts / partial unmaps that don't affect concurrent DMA.
Yes. Most x86 server HW will do this, though ARM support is a bit newish.
Quoted 3 lines
What would be your suggestion here to avoid the "map each 4k page individually so we can unmap it individually" ? I didn't completely grasp that, sorry.
Map in large ranges in the VMM, lets say 1G of shared memory as a single mapping (called an iommufd area)
When the guest makes a 2M chunk of it private you do a ioctl to iommufd to split the area into three, leaving the 2M chunk as a seperate area.
The new iommufd ioctl to split areas will go down into the iommu driver and atomically cut the 1G PTEs into smaller PTEs as necessary so that no PTE spans the edges of the 2M area.
Then userspace can unmap the 2M area and leave the remainder of the 1G area mapped.
All of this would be fully hitless to ongoing DMA.
The iommufs code is there to do this assuming the areas are mapped at 4k, what is missing is the iommu driver side to atomically resize large PTEs.
Quoted 4 lines
From "IIRC you can only trigger split using the VFIO type 1 legacy API. We
would need to formalize split as an IOMMUFD native ioctl. Nobody should use this stuf through the legacy type 1 API!!!!"
Quoted 6 lines
Quoted 3 lines
I assume you mean that we can only avoid the 4k map/unmap if we add proper support to IOMMUFD native ioctl, and not try making it fly somehow with the legacy type 1 API?
The thread was talking about the built-in support in iommufd to split mappings.
Just to clarify - I am talking about splitting only "iommufd areas", not large pages. If all IOMMU PTEs are 4k and areas are bigger than 4K => the hw support is not needed to allow splitting. The comments above and below seem to confuse large pages with large areas (well, I am consufed, at least).
Quoted 3 lines
That built-in support is only accessible through legacy APIs and should never be used in new qemu code. To use that built in support in new code we need to build new APIs.
Why would not IOMMU_IOAS_MAP/UNMAP uAPI work? Thanks,
Quoted 10 lines
The advantage of the
built-in support is qemu can map in large regions (which is more efficient) and the kernel will break it down to 4k for the iommu driver.
Mapping 4k at a time through the uAPI would be outrageously inefficient.Jason
Signature · 1 line
Alexey
Quoted 5 lines · Alexey Kardashevskiy
Quoted 2 lines
The thread was talking about the built-in support in iommufd to split mappings.
Just to clarify - I am talking about splitting only "iommufd areas", not large pages.
In generality it is the same thing as you cannot generally guarantee that an area split doesn't also cross a large page.
Quoted 3 lines
If all IOMMU PTEs are 4k and areas are bigger than 4K => the hw support is not needed to allow splitting. The comments above and below seem to confuse large pages with large areas (well, I am consufed, at least).
Yes, in that special case yes.
Quoted 5 lines
Quoted 3 lines
That built-in support is only accessible through legacy APIs and should never be used in new qemu code. To use that built in support in new code we need to build new APIs.
Why would not IOMMU_IOAS_MAP/UNMAP uAPI work? Thanks,
I don't want to overload those APIs, I prefer to see a new API that is just about splitting areas. Splitting is a special operation that can fail depending on driver support.
Jason
Quoted 20 lines · Jason Gunthorpe
Quoted 5 lines · Alexey Kardashevskiy
Quoted 2 lines
The thread was talking about the built-in support in iommufd to split mappings.
Just to clarify - I am talking about splitting only "iommufd areas", not large pages.
In generality it is the same thing as you cannot generally guarantee that an area split doesn't also cross a large page.
Quoted 3 lines
If all IOMMU PTEs are 4k and areas are bigger than 4K => the hw support is not needed to allow splitting. The comments above and below seem to confuse large pages with large areas (well, I am consufed, at least).
Yes, in that special case yes.
Quoted 4 lines
Quoted 3 lines
That built-in support is only accessible through legacy APIs and should never be used in new qemu code. To use that built in support in new code we need to build new APIs.
Quoted 5 lines
Quoted 1 line
Why would not IOMMU_IOAS_MAP/UNMAP uAPI work? Thanks,
I don't want to overload those APIs, I prefer to see a new API that is just about splitting areas. Splitting is a special operation that can fail depending on driver support.
So we'd just always perform a split-before-unmap. If split fails, we're in trouble, just like we would be when unmap would fail.
If the split succeeded, the unmap will succeed *and* be atomic.
That sounds reasonable to me and virtio-mem could benefit from that as well.
Signature · 3 lines
Cheers, David / dhildenb
Quoted 10 lines · David Hildenbrand
Quoted 3 lines
I don't want to overload those APIs, I prefer to see a new API that is just about splitting areas. Splitting is a special operation that can fail depending on driver support.
So we'd just always perform a split-before-unmap. If split fails, we're in trouble, just like we would be when unmap would fail.
If the split succeeded, the unmap will succeed *and* be atomic.
That sounds reasonable to me and virtio-mem could benefit from that as well.
Yeah, we just went through removing implicit split on unmap behaviors from the code in iommu and I think it was a mistake that existed at all.
Very few places even want/expect to do split. Instead we ended up with this situation where it was hard to tell who was even using it, if at all. Turned out nobody used it.
So I'd like the very special behavior marked out in code. If performance is a concern I'd prefer to see split gain a auto-unmap option.
Jason