VIRTIO-IOMMU/VFIO page size related fixes
cover VIRTIO-IOMMU/VFIO page size related fixes
When assigning a host device and protecting it with the virtio-iommu we may end up with qemu crashing with
qemu-kvm: virtio-iommu page mask 0xfffffffffffff000 is incompatible
with mask 0x20010000
qemu: hardware error: vfio: DMA mapping failed, unable to continueThis happens if the host features a 64kB page size and constraints the physical IOMMU to use a 64kB page size. By default 4kB granule is used by the qemu virtio-iommu device and this latter becomes aware of the 64kB requirement too late, after the machine init, when the vfio device domain is attached. virtio_iommu_set_page_size_mask() fails and this causes vfio_listener_region_add() to end up with hw_error(). Currently the granule is global to all domains.
To work around this issue, despite the IOMMU MR may be bypassed, we transiently enable it on machine init done to get vfio_listener_region_add and virtio_iommu_set_page_size_mask called ealier, before the domain attach. That way the page size requirement can be taken into account before the guest gets started.
Also get benefit of this series to do some cleanups in some traces which may confuse the end user.
Best Regards
Eric
This series can be found at:
https://github.com/eauger/qemu/tree/virtio-iommu-page-size-v1
Eric Auger (2):
virtio-iommu: Fix 64kB host page size VFIO device assignment
virtio-iommu: Rework the trace in virtio_iommu_set_page_size_mask()
include/hw/virtio/virtio-iommu.h | 2 ++ hw/virtio/virtio-iommu.c | 49 +++++++++++++++++++++++--------- hw/virtio/trace-events | 1 + 3 files changed, 38 insertions(+), 14 deletions(-)
Signature · 1 line
2.38.1
PATCH 1/2 virtio-iommu: Fix 64kB host page size VFIO device assignment
When running on a 64kB page size host and protecting a VFIO device with the virtio-iommu, qemu crashes with this kind of message:
qemu-kvm: virtio-iommu page mask 0xfffffffffffff000 is incompatible with mask 0x20010000
qemu: hardware error: vfio: DMA mapping failed, unable to continue
This is due to the fact the IOMMU MR corresponding to the VFIO device is enabled very late on domain attach, after the machine init. The device reports a minimal 64kB page size but it is too late to be applied. virtio_iommu_set_page_size_mask() fails and this causes vfio_listener_region_add() to end up with hw_error();
To work around this issue, we transiently enable the IOMMU MR on machine init to collect the page size requirements and then restore the bypass state.
--- include/hw/virtio/virtio-iommu.h | 2 ++ hw/virtio/virtio-iommu.c | 30 ++++++++++++++++++++++++++++-- hw/virtio/trace-events | 1 + 3 files changed, 31 insertions(+), 2 deletions(-)
include/hw/virtio/virtio-iommu.h +2 -0
@@ -61,6 +61,8 @@ struct VirtIOIOMMU {QemuRecMutex mutex;GTree *endpoints;bool boot_bypass; +Notifier machine_done; + bool granule_frozen; };#endif
hw/virtio/virtio-iommu.c +28 -2
@@ -24,6 +24,7 @@#include "hw/virtio/virtio.h" #include "sysemu/kvm.h" #include "sysemu/reset.h" +#include "sysemu/sysemu.h" #include "qapi/error.h" #include "qemu/error-report.h" #include "trace.h"
@@ -1106,12 +1107,12 @@ static int virtio_iommu_set_page_size_mask(IOMMUMemoryRegion *mr, }/* -* After the machine is finalized, we can't change the mask anymore. If by +* Once the granule is frozen we can't change the mask anymore. If by * chance the hotplugged device supports the same granule, we can still * accept it. Having a different masks is possible but the guest will use * sub-optimal block sizes, so warn about it. */ -if (phase_check (PHASE_MACHINE_READY )) { +if (s->granule_frozen) {int new_granule =ctz64 (new_mask);int cur_granule =ctz64 (cur_mask);
@@ -1146,6 +1147,27 @@ static void virtio_iommu_system_reset(void *opaque) } +static void virtio_iommu_freeze_granule (Notifier *notifier,void *data) +{ +VirtIOIOMMU *s =container_of (notifier, VirtIOIOMMU, machine_done); +bool boot_bypass = s->config.bypass; +int granule; + +/* +* Transient IOMMU MR enable to collect page_size_mask requirement +* through memory_region_iommu_set_page_size_mask() called by +* VFIO region_add() callback +*/ + s->config.bypass =0 ; +virtio_iommu_switch_address_space_all (s); +/* restore default */ + s->config.bypass = boot_bypass; +virtio_iommu_switch_address_space_all (s); + s->granule_frozen = true; + granule =ctz64 (s->config.page_size_mask); +trace_virtio_iommu_freeze_granule (BIT (granule)); +} +static void virtio_iommu_device_realize (DeviceState *dev,Error **errp) {VirtIODevice *vdev =VIRTIO_DEVICE (dev);
@@ -1189,6 +1211,9 @@ static void virtio_iommu_device_realize(DeviceState *dev, Error **errp)error_setg (errp,"VIRTIO-IOMMU is not attached to any PCI bus!" ); } + s->machine_done.notify = virtio_iommu_freeze_granule; +qemu_add_machine_init_done_notifier (&s->machine_done); +qemu_register_reset (virtio_iommu_system_reset, s); }
@@ -1198,6 +1223,7 @@ static void virtio_iommu_device_unrealize(DeviceState *dev)VirtIOIOMMU *s =VIRTIO_IOMMU (dev);qemu_unregister_reset (virtio_iommu_system_reset, s); +qemu_remove_machine_init_done_notifier (&s->machine_done);g_hash_table_destroy (s->as_by_busptr);if (s->domains) {
hw/virtio/trace-events +1 -0
@@ -131,6 +131,7 @@ virtio_iommu_set_page_size_mask(const char *name, uint64_t old, uint64_t new) "m virtio_iommu_notify_flag_add(const char *name) "add notifier to mr %s" virtio_iommu_notify_flag_del(const char *name) "del notifier from mr %s" virtio_iommu_switch_address_space(uint8_t bus, uint8_t slot, uint8_t fn, bool on) "Device %02x:%02x.%x switching address space (iommu enabled=%d)" +virtio_iommu_freeze_granule(uint64_t page_size_mask) "granule set to 0x%"PRIx64 # virtio-mem.c virtio_mem_send_response(uint16_t type) "type=%" PRIu16
Signature · 1 line
2.38.1
RE: [PATCH 1/2] virtio-iommu: Fix 64kB host page size VFIO device assignment
Hi Eric,
Quoted 11 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: Tuesday, July 4, 2023 7:15 PM
Subject: [PATCH 1/2] virtio-iommu: Fix 64kB host page size VFIO device assignmentWhen running on a 64kB page size host and protecting a VFIO device with the virtio-iommu, qemu crashes with this kind of message:
qemu-kvm: virtio-iommu page mask
0xfffffffffffff000is incompatible with mask0x20010000
Does 0x20010000 mean only 512MB and 64KB super page mapping is supported for host iommu hw? 4KB mapping not supported?
There is a check in guest kernel side hint only 4KB is supported, with this patch we force viommu->pgsize_bitmap to 0x20010000
and fail below check? Does this device work in guest? Please correct me if I understand wrong.
static int viommu_domain_finalise(struct viommu_endpoint *vdev,
struct iommu_domain *domain)
{
...
viommu_page_size = 1UL << __ffs(viommu->pgsize_bitmap);
if (viommu_page_size > PAGE_SIZE) {
dev_err(vdev->dev,
"granule 0x%lx larger than system page size 0x%lx\n",
viommu_page_size, PAGE_SIZE);
return -ENODEV;
}Another question is: Presume 0x20010000 does mean only 512MB and 64KB is supported. Is host hva->hpa mapping ensured to be compatible with at least 64KB mapping? If host mmu only support 4KB mapping or other non-compatible with 0x20010000, will vfio_dma_map() fail?
Quoted 78 lines
qemu: hardware error: vfio: DMA mapping failed, unable to continue
This is due to the fact the IOMMU MR corresponding to the VFIO device is enabled very late on domain attach, after the machine init. The device reports a minimal 64kB page size but it is too late to be applied.
virtio_iommu_set_page_size_mask()fails and this causesvfio_listener_region_add()to end up withhw_error();To work around this issue, we transiently enable the IOMMU MR on machine init to collect the page size requirements and then restore the bypass state.
Fixes 90519b9053 ("virtio-iommu: Add bypass mode support to assigneddevice")
Signed-off-by Eric Auger <[email protected]>---
include/hw/virtio/virtio-iommu.h | 2 ++ hw/virtio/virtio-iommu.c | 30 ++++++++++++++++++++++++++++-- hw/virtio/trace-events | 1 + 3 files changed, 31 insertions(+), 2 deletions(-)include/hw/virtio/virtio- renamed from include/hw/virtio/virtio-iommu.h +0
-0
iommu.h
index2ad5ee320b..a93fc5383e100644include/hw/virtio/virtio-iommu.h +2
-0@@ -61,6 +61,8 @@ struct VirtIOIOMMU {QemuRecMutex mutex;GTree *endpoints;bool boot_bypass; +Notifier machine_done; + bool granule_frozen; };#endif hw/virtio/virtio-iommu.c +13
-2@@ -24,6 +24,7 @@#include "hw/virtio/virtio.h" #include "sysemu/kvm.h" #include "sysemu/reset.h" +#include "sysemu/sysemu.h" #include "qapi/error.h" #include "qemu/error-report.h" #include "trace.h" @@ -1106,12 +1107,12 @@ static int virtio_iommu_set_page_size_mask(IOMMUMemoryRegion *mr, }/* -* After the machine is finalized, we can't change the mask anymore. If by +* Once the granule is frozen we can't change the mask anymore. If by * chance the hotplugged device supports the same granule, we can still * accept it. Having a different masks is possible but the guest will use * sub-optimal block sizes, so warn about it. */ -if (phase_check (PHASE_MACHINE_READY )) { +if (s->granule_frozen) {int new_granule =ctz64 (new_mask);int cur_granule =ctz64 (cur_mask);@@ -1146,6 +1147,27 @@ static void virtio_iommu_system_reset(void *opaque) } +static void virtio_iommu_freeze_granule (Notifier *notifier,void *data) +{ +VirtIOIOMMU *s =container_of (notifier, VirtIOIOMMU, machine_done); +bool boot_bypass = s->config.bypass; +int granule; + +/* +* Transient IOMMU MR enable to collect page_size_mask requirement +* through memory_region_iommu_set_page_size_mask() called by +* VFIO region_add() callback
hw/virtio/virtio-iommu.c static void virtio_iommu_system_reset(void · line 1170
Quoted 10 lines
hw/virtio/virtio-iommu.c +10
-0+*/ + s->config.bypass =0 ; +virtio_iommu_switch_address_space_all (s); +/* restore default */ + s->config.bypass = boot_bypass; +virtio_iommu_switch_address_space_all (s); + s->granule_frozen = true; + granule =ctz64 (s->config.page_size_mask); +trace_virtio_iommu_freeze_granule (BIT (granule)); +}
It looks a bit heavy here just in order to get page_size_mask from host side. But maybe this is the only way with current interface.
Thanks
Zhenzhong
Quoted 42 lines
hw/virtio/virtio-iommu.c +1
-0+static void virtio_iommu_device_realize (DeviceState *dev,Error **errp) {VirtIODevice *vdev = VIRTIO_DEVICE(dev);hw/virtio/virtio-iommu.c +4
-0@@ -1189,6 +1211,9 @@ static void virtio_iommu_device_realize(DeviceState *dev, Error **errp)error_setg (errp,"VIRTIO-IOMMU is not attached to any PCI bus!" ); } + s->machine_done.notify = virtio_iommu_freeze_granule; +qemu_add_machine_init_done_notifier (&s->machine_done); +qemu_register_reset (virtio_iommu_system_reset, s); }@@ -1198,6 +1223,7 @@ static void virtio_iommu_device_unrealize(DeviceState *dev)VirtIOIOMMU *s =VIRTIO_IOMMU (dev);qemu_unregister_reset (virtio_iommu_system_reset, s); +qemu_remove_machine_init_done_notifier (&s->machine_done);g_hash_table_destroy (s->as_by_busptr);if (s->domains) {hw/virtio/trace-events +0
-0@@ -131,6 +131,7 @@ virtio_iommu_set_page_size_mask(const char *name,
uint64_told,uint64_tnew) "mvirtio_iommu_notify_flag_add(const char *name)"add notifier to mr %s"virtio_iommu_notify_flag_del(const char *name)"del notifier from mr %s"virtio_iommu_switch_address_space(uint8_t bus, uint8_t slot, uint8_t fn, bool on)"Device %02x:%02x.%x switching address space (iommu enabled=%d)"
+virtio_iommu_freeze_granule(uint64_tpage_size_mask) "granule set to 0x%"PRIx64# virtio-mem.c virtio_mem_send_response(uint16_t type) "type=%" PRIu16 -- 2.38.1
Re: [PATCH 1/2] virtio-iommu: Fix 64kB host page size VFIO device assignment
Hi Zhenzhong,
Quoted 8 lines · Zhenzhong
Hi Eric,
Quoted 6 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: Tuesday, July 4, 2023 7:15 PM
Subject: [PATCH 1/2] virtio-iommu: Fix 64kB host page size VFIO device assignment
Quoted 7 lines
Quoted 5 lines
When running on a 64kB page size host and protecting a VFIO device with the virtio-iommu, qemu crashes with this kind of message:
qemu-kvm: virtio-iommu page mask
0xfffffffffffff000is incompatible with mask0x20010000Does
0x20010000mean only 512MB and 64KB super page mapping is supported for host iommu hw? 4KB mapping not supported?
Yes that's correct. In that case the host has 64kB page and 4kB is not supported.
Quoted 5 lines
There is a check in guest kernel side hint only 4KB is supported, with this patch we force
viommu->pgsize_bitmapto0x20010000
and fail below check? Does this device work in guest? Please correct me if I understand wrong.
my guest also has 64kB so the check hereafter succeeds. effectively in case your host has a larger page size than your guest it fails with [ 2.147031] virtio-pci 0000:00:01.0: granule 0x10000 larger than system page size 0x1000
[ 7.231128] ixgbevf 0000:00:02.0: granule 0x10000 larger than system page size 0x1000
Quoted 13 lines
static int viommu_domain_finalise(struct
viommu_endpoint*vdev,
structiommu_domain*domain)
{
...viommu_page_size = 1UL << __ffs(viommu->pgsize_bitmap); if (viommu_page_size > PAGE_SIZE) { dev_err(vdev->dev, "granule 0x%lx larger than system page size 0x%lx\n", viommu_page_size, PAGE_SIZE); return -ENODEV; }
Quoted 4 lines
Another question is: Presume
0x20010000does mean only 512MB and 64KB is supported. Is hosthva->hpamapping ensured to be compatible with at least 64KB mapping? If host mmu only support 4KB mapping or other non-compatible with0x20010000, willvfio_dma_map()fail?
the page size mask is retrieved with VFIO_IOMMU_GET_INFO uapi which returns host vfio_iommu_type1 vfio_iommu->pgsize_bitmap. This latter is initialized to host PAGE_MASK and later restricted on vfio_iommu_type1_attach_group though the vfio_update_pgsize_bitmap() calls
so yes both IOMMU and CPU page size are garanteed to be compatible.
Quoted 83 lines
Quoted 83 lines
qemu: hardware error: vfio: DMA mapping failed, unable to continue
This is due to the fact the IOMMU MR corresponding to the VFIO device is enabled very late on domain attach, after the machine init. The device reports a minimal 64kB page size but it is too late to be applied.
virtio_iommu_set_page_size_mask()fails and this causesvfio_listener_region_add()to end up withhw_error();To work around this issue, we transiently enable the IOMMU MR on machine init to collect the page size requirements and then restore the bypass state.
Fixes 90519b9053 ("virtio-iommu: Add bypass mode support to assigneddevice")
Signed-off-by Eric Auger <[email protected]>---
include/hw/virtio/virtio-iommu.h | 2 ++ hw/virtio/virtio-iommu.c | 30 ++++++++++++++++++++++++++++-- hw/virtio/trace-events | 1 + 3 files changed, 31 insertions(+), 2 deletions(-)include/hw/virtio/virtio- renamed from include/hw/virtio/virtio-iommu.h +0
-0
iommu.h
index2ad5ee320b..a93fc5383e100644include/hw/virtio/virtio-iommu.h +2
-0@@ -61,6 +61,8 @@ struct VirtIOIOMMU {QemuRecMutex mutex;GTree *endpoints;bool boot_bypass; +Notifier machine_done; + bool granule_frozen; };#endif hw/virtio/virtio-iommu.c +17
-2@@ -24,6 +24,7 @@#include "hw/virtio/virtio.h" #include "sysemu/kvm.h" #include "sysemu/reset.h" +#include "sysemu/sysemu.h" #include "qapi/error.h" #include "qemu/error-report.h" #include "trace.h" @@ -1106,12 +1107,12 @@ static int virtio_iommu_set_page_size_mask(IOMMUMemoryRegion *mr, }/* -* After the machine is finalized, we can't change the mask anymore. If by +* Once the granule is frozen we can't change the mask anymore. If by * chance the hotplugged device supports the same granule, we can still * accept it. Having a different masks is possible but the guest will use * sub-optimal block sizes, so warn about it. */ -if (phase_check (PHASE_MACHINE_READY )) { +if (s->granule_frozen) {int new_granule =ctz64 (new_mask);int cur_granule =ctz64 (cur_mask);@@ -1146,6 +1147,27 @@ static void virtio_iommu_system_reset(void *opaque) } +static void virtio_iommu_freeze_granule (Notifier *notifier,void *data) +{ +VirtIOIOMMU *s =container_of (notifier, VirtIOIOMMU, machine_done); +bool boot_bypass = s->config.bypass; +int granule; + +/* +* Transient IOMMU MR enable to collect page_size_mask requirement +* through memory_region_iommu_set_page_size_mask() called by +* VFIO region_add() callback +*/ + s->config.bypass =0 ; +virtio_iommu_switch_address_space_all (s); +/* restore default */
Quoted 8 lines
hw/virtio/virtio-iommu.c static void virtio_iommu_system_reset(void *opaque) · line 1169
Quoted 6 lines
hw/virtio/virtio-iommu.c +6
-0+ s->config.bypass = boot_bypass; +virtio_iommu_switch_address_space_all (s); + s->granule_frozen = true; + granule =ctz64 (s->config.page_size_mask); +trace_virtio_iommu_freeze_granule (BIT (granule)); +}It looks a bit heavy here just in order to get
page_size_maskfrom host side. But maybe this is the only way with current interface.
the problem comes from the fact the regions are aliased due to the bypass and vfio_listener_region_add() does not get a chance to be called until the actual domain attach. So I do not see any other way to transiently enable the region.
At least I could check if boot bypass is set before doing that dance. I will respin with that.
Thanks
Eric
Quoted 46 lines
Thanks
ZhenzhongQuoted 42 lines
hw/virtio/virtio-iommu.c +1
-0+static void virtio_iommu_device_realize (DeviceState *dev,Error **errp) {VirtIODevice *vdev = VIRTIO_DEVICE(dev);hw/virtio/virtio-iommu.c +4
-0@@ -1189,6 +1211,9 @@ static void virtio_iommu_device_realize(DeviceState *dev, Error **errp)error_setg (errp,"VIRTIO-IOMMU is not attached to any PCI bus!" ); } + s->machine_done.notify = virtio_iommu_freeze_granule; +qemu_add_machine_init_done_notifier (&s->machine_done); +qemu_register_reset (virtio_iommu_system_reset, s); }@@ -1198,6 +1223,7 @@ static void virtio_iommu_device_unrealize(DeviceState *dev)VirtIOIOMMU *s =VIRTIO_IOMMU (dev);qemu_unregister_reset (virtio_iommu_system_reset, s); +qemu_remove_machine_init_done_notifier (&s->machine_done);g_hash_table_destroy (s->as_by_busptr);if (s->domains) {hw/virtio/trace-events +0
-0@@ -131,6 +131,7 @@ virtio_iommu_set_page_size_mask(const char *name,
uint64_told,uint64_tnew) "mvirtio_iommu_notify_flag_add(const char *name)"add notifier to mr %s"virtio_iommu_notify_flag_del(const char *name)"del notifier from mr %s"virtio_iommu_switch_address_space(uint8_t bus, uint8_t slot, uint8_t fn, bool on)"Device %02x:%02x.%x switching address space (iommu enabled=%d)"
+virtio_iommu_freeze_granule(uint64_tpage_size_mask) "granule set to 0x%"PRIx64# virtio-mem.c virtio_mem_send_response(uint16_t type) "type=%" PRIu16 -- 2.38.1
RE: [PATCH 1/2] virtio-iommu: Fix 64kB host page size VFIO device assignment
Quoted 30 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: Wednesday, July 5, 2023 2:20 PM
Subject: Re: [PATCH 1/2] virtio-iommu: Fix 64kB host page size VFIO device assignmentHi Zhenzhong,
Quoted 8 lines · Zhenzhong
Hi Eric,
Quoted 6 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: Tuesday, July 4, 2023 7:15 PM
Subject: [PATCH 1/2] virtio-iommu: Fix 64kB host page size VFIO device assignmentQuoted 7 lines
Quoted 5 lines
When running on a 64kB page size host and protecting a VFIO device with the virtio-iommu, qemu crashes with this kind of message:
qemu-kvm: virtio-iommu page mask
0xfffffffffffff000is incompatible with mask0x20010000Does
0x20010000mean only 512MB and 64KB super page mapping is supported for host iommu hw? 4KB mapping not supported?Yes that's correct. In that case the host has 64kB page and 4kB is not supported.
Quoted 5 lines
There is a check in guest kernel side hint only 4KB is supported, with this patch we force
viommu->pgsize_bitmapto0x20010000
and fail below check? Does this device work in guest? Please correct me if I understand wrong.
Quoted 6 lines
my guest also has 64kB so the check hereafter succeeds. effectively in case your host has a larger page size than your guest it fails with [ 2.147031] virtio-pci 0000:00:01.0: granule
0x10000larger than system page size0x1000
[ 7.231128] ixgbevf 0000:00:02.0: granule0x10000larger than system page size0x1000
Oh, I see, I took PAGE_SIZE as 4KB for granted.
Quoted 17 lines
Quoted 14 lines
static int viommu_domain_finalise(struct
viommu_endpoint*vdev,
structiommu_domain*domain)
{
...viommu_page_size = 1UL << __ffs(viommu->pgsize_bitmap); if (viommu_page_size > PAGE_SIZE) { dev_err(vdev->dev, "granule 0x%lx larger than system page size 0x%lx\n", viommu_page_size, PAGE_SIZE); return -ENODEV; }Quoted 2 lines
Another question is: Presume
0x20010000does mean only 512MB and 64KB is supported. Is hosthva->hpamapping ensured to be compatible with atleast
Quoted 8 lines
Quoted 1 line
64KB mapping? If host mmu only support 4KB mapping or other non-
compatible
Quoted 1 line
with
0x20010000, willvfio_dma_map()fail?the page size mask is retrieved with
VFIO_IOMMU_GET_INFOuapi which returns hostvfio_iommu_type1vfio_iommu->pgsize_bitmap. This latter is initialized to hostPAGE_MASKand later restricted onvfio_iommu_type1_attach_groupthough thevfio_update_pgsize_bitmap()calls
Understood, thanks for your analysis.
Quoted 96 lines
so yes both IOMMU and CPU page size are garanteed to be compatible.
Quoted 52 lines
Quoted 52 lines
qemu: hardware error: vfio: DMA mapping failed, unable to continue
This is due to the fact the IOMMU MR corresponding to the VFIO device is enabled very late on domain attach, after the machine init. The device reports a minimal 64kB page size but it is too late to be applied.
virtio_iommu_set_page_size_mask()fails and this causesvfio_listener_region_add()to end up withhw_error();To work around this issue, we transiently enable the IOMMU MR on machine init to collect the page size requirements and then restore the bypass state.
Fixes 90519b9053 ("virtio-iommu: Add bypass mode support to assigneddevice")
Signed-off-by Eric Auger <[email protected]>---
include/hw/virtio/virtio-iommu.h | 2 ++ hw/virtio/virtio-iommu.c | 30 ++++++++++++++++++++++++++++-- hw/virtio/trace-events | 1 + 3 files changed, 31 insertions(+), 2 deletions(-)include/hw/virtio/virtio- renamed from include/hw/virtio/virtio-iommu.h +0
-0
iommu.h
index2ad5ee320b..a93fc5383e100644include/hw/virtio/virtio-iommu.h +2
-0@@ -61,6 +61,8 @@ struct VirtIOIOMMU {QemuRecMutex mutex;GTree *endpoints;bool boot_bypass; +Notifier machine_done; + bool granule_frozen; };#endif hw/virtio/virtio-iommu.c +1
-0@@ -24,6 +24,7 @@#include "hw/virtio/virtio.h" #include "sysemu/kvm.h" #include "sysemu/reset.h" +#include "sysemu/sysemu.h" #include "qapi/error.h" #include "qemu/error-report.h" #include "trace.h" @@ -1106,12 +1107,12 @@ static int virtio_iommu_set_page_size_mask(IOMMUMemoryRegion *mr, }Quoted 2 lines
hw/virtio/virtio-iommu.c static int · line 1110
Quoted 2 lines
hw/virtio/virtio-iommu.c +0
-1/* -* After the machine is finalized, we can't change the mask anymore. If by
Quoted 29 lines
Quoted 29 lines
hw/virtio/virtio-iommu.c +16
-1+* Once the granule is frozen we can't change the mask anymore. If by * chance the hotplugged device supports the same granule, we can still * accept it. Having a different masks is possible but the guest will use * sub-optimal block sizes, so warn about it. */ -if (phase_check (PHASE_MACHINE_READY )) { +if (s->granule_frozen) {int new_granule =ctz64 (new_mask);int cur_granule =ctz64 (cur_mask);@@ -1146,6 +1147,27 @@ static void virtio_iommu_system_reset(void *opaque) } +static void virtio_iommu_freeze_granule (Notifier *notifier,void *data) +{ +VirtIOIOMMU *s =container_of (notifier, VirtIOIOMMU, machine_done); +bool boot_bypass = s->config.bypass; +int granule; + +/* +* Transient IOMMU MR enable to collect page_size_mask requirement +* through memory_region_iommu_set_page_size_mask() called by +* VFIO region_add() callback +*/ + s->config.bypass =0 ; +virtio_iommu_switch_address_space_all (s); +/* restore default */ Quoted 8 lines
hw/virtio/virtio-iommu.c static void virtio_iommu_system_reset(void *opaque) · line 1169
Quoted 6 lines
hw/virtio/virtio-iommu.c +6
-0+ s->config.bypass = boot_bypass; +virtio_iommu_switch_address_space_all (s); + s->granule_frozen = true; + granule =ctz64 (s->config.page_size_mask); +trace_virtio_iommu_freeze_granule (BIT (granule)); +}It looks a bit heavy here just in order to get
page_size_maskfrom host side. But maybe this is the only way with current interface.
Quoted 4 lines
the problem comes from the fact the regions are aliased due to the bypass and
vfio_listener_region_add()does not get a chance to be called until the actual domain attach. So I do not see any other way to transiently enable the region.
Agree.
Quoted 3 lines
At least I could check if boot bypass is set before doing that dance. I will respin with that.
Make sense.
Thanks
Zhenzhong
Re: [PATCH 1/2] virtio-iommu: Fix 64kB host page size VFIO device assignment
In reply to Duan, ZhenzhongQuoted 8 lines · Zhenzhong
Hi Eric,
Quoted 6 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: Tuesday, July 4, 2023 7:15 PM
Subject: [PATCH 1/2] virtio-iommu: Fix 64kB host page size VFIO device assignment
Quoted 8 lines
Quoted 5 lines
When running on a 64kB page size host and protecting a VFIO device with the virtio-iommu, qemu crashes with this kind of message:
qemu-kvm: virtio-iommu page mask
0xfffffffffffff000is incompatible with mask0x20010000Does
0x20010000mean only 512MB and 64KB super page mapping is supported for host iommu hw? 4KB mapping not supported?
It's not a restriction by the HW IOMMU, but the host kernel. An Arm SMMU can implement 4KB, 16KB and/or 64KB granules, but the host kernel only advertises through VFIO the granule corresponding to host PAGE_SIZE. This restriction is done by arm_lpae_restrict_pgsizes() in order to choose a page size when a device is driven by the host.
Quoted 5 lines
There is a check in guest kernel side hint only 4KB is supported, with this patch we force
viommu->pgsize_bitmapto0x20010000
and fail below check? Does this device work in guest? Please correct me if I understand wrong.
Right, a guest using 4KB pages under a host that uses 64KB is not supported, because if the guest attempts to dma_map a 4K page, the IOMMU cannot create a mapping small enough, the mapping would have to spill over neighbouring guest memory.
One possible solution would be supporting multiple page granules. If we added a page granule negotiation through VFIO and virtio-iommu then the guest could pick the page size it wants. But this requires changes to Linux UAPI so isn't a priority at the moment, because we're trying to enable nesting which would support 64K-host/4K-guest as well.
See also the discussion on the patch that introduced the guest check
https://lore.kernel.org/linux-iommu/[email protected]/
Thanks,
Jean
RE: [PATCH 1/2] virtio-iommu: Fix 64kB host page size VFIO device assignment
Quoted 21 lines
-----Original Message-----
From: Jean-Philippe Brucker <[email protected]>
Sent: Wednesday, July 5, 2023 4:29 PM
Subject: Re: [PATCH 1/2] virtio-iommu: Fix 64kB host page size VFIO device assignmentQuoted 14 lines · Zhenzhong
Hi Eric,
Quoted 11 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: Tuesday, July 4, 2023 7:15 PM
Subject: [PATCH 1/2] virtio-iommu: Fix 64kB host page size VFIO device assignmentWhen running on a 64kB page size host and protecting a VFIO device with the virtio-iommu, qemu crashes with this kind of message:
qemu-kvm: virtio-iommu page mask
0xfffffffffffff000is incompatible with mask0x20010000
Quoted 8 lines
Quoted 2 lines
Does
0x20010000mean only 512MB and 64KB super page mapping is supported for host iommu hw? 4KB mapping not supported?It's not a restriction by the HW IOMMU, but the host kernel. An Arm SMMU can implement 4KB, 16KB and/or 64KB granules, but the host kernel only advertises through VFIO the granule corresponding to host
PAGE_SIZE. This restriction is done byarm_lpae_restrict_pgsizes()in order to choose a page size when a device is driven by the host.
Just curious why not advertises the Arm SMMU implemented granules to VFIO Eg:4KB, 16KB or 64KB granules? But arm_lpae_restrict_pgsizes() restricted ones,
Eg: for SZ_4K, (SZ_4K | SZ_2M | SZ_1G).
(SZ_4K | SZ_2M | SZ_1G) looks not real hardware granules of Arm SMMU.
Quoted 18 lines
Quoted 6 lines
There is a check in guest kernel side hint only 4KB is supported, with this patch we force
viommu->pgsize_bitmapto0x20010000and fail below check? Does this device work in guest?
Please correct me if I understand wrong.Right, a guest using 4KB pages under a host that uses 64KB is not supported, because if the guest attempts to
dma_mapa 4K page, the IOMMU cannot create a mapping small enough, the mapping would have to spill over neighbouring guest memory.One possible solution would be supporting multiple page granules. If we added a page granule negotiation through VFIO and virtio-iommu then the guest could pick the page size it wants. But this requires changes to Linux UAPI so isn't a priority at the moment, because we're trying to enable nesting which would support 64K-host/4K-guest as well.
Quoted 3 lines
See also the discussion on the patch that introduced the guest check
https://lore.kernel.org/linux-iommu/20200318114047.1518048-1-jean-
[email protected]/
Clear, thanks for sharing the history.
Regards
Zhenzhong
Re: [PATCH 1/2] virtio-iommu: Fix 64kB host page size VFIO device assignment
Quoted 24 lines · Zhenzhong
Quoted 24 lines
-----Original Message-----
From: Jean-Philippe Brucker <[email protected]>
Sent: Wednesday, July 5, 2023 4:29 PM
Subject: Re: [PATCH 1/2] virtio-iommu: Fix 64kB host page size VFIO device assignmentQuoted 16 lines · Zhenzhong
Hi Eric,
Quoted 11 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: Tuesday, July 4, 2023 7:15 PM
Subject: [PATCH 1/2] virtio-iommu: Fix 64kB host page size VFIO device assignmentWhen running on a 64kB page size host and protecting a VFIO device with the virtio-iommu, qemu crashes with this kind of message:
qemu-kvm: virtio-iommu page mask
0xfffffffffffff000is incompatible with mask0x20010000Does
0x20010000mean only 512MB and 64KB super page mapping is supported for host iommu hw? 4KB mapping not supported?
Quoted 8 lines
Quoted 5 lines
It's not a restriction by the HW IOMMU, but the host kernel. An Arm SMMU can implement 4KB, 16KB and/or 64KB granules, but the host kernel only advertises through VFIO the granule corresponding to host
PAGE_SIZE. This restriction is done byarm_lpae_restrict_pgsizes()in order to choose a page size when a device is driven by the host.Just curious why not advertises the Arm SMMU implemented granules to VFIO Eg:4KB, 16KB or 64KB granules?
That's possible, but the difficulty is setting up the page table configuration afterwards. At the moment the host installs the HW page tables early, when QEMU sets up the VFIO container. That initializes the page size bitmap because configuring the HW page tables requires picking one of the supported granules (setting TG0 in the SMMU Context Descriptor).
If the guest could pick a granule via an ATTACH request, then QEMU would need to tell the host kernel to install page tables with the desired granule at that point. That would require a new interface in VFIO to reconfigure a live container and replace the existing HW page tables configuration (before ATTACH, the container must already be configured with working page tables in order to implement boot-bypass, I think).
Quoted 3 lines
But
arm_lpae_restrict_pgsizes()restricted ones,
Eg: forSZ_4K, (SZ_4K|SZ_2M|SZ_1G).
(SZ_4K|SZ_2M|SZ_1G) looks not real hardware granules of Arm SMMU.
Yes, the granule here is 4K, and other bits only indicate huge page sizes, so the user can try to optimize large mappings to use fewer TLB entries where possible.
Thanks,
Jean
RE: [PATCH 1/2] virtio-iommu: Fix 64kB host page size VFIO device assignment
Quoted 48 lines
-----Original Message-----
From: Jean-Philippe Brucker <[email protected]>
Sent: Wednesday, July 5, 2023 7:33 PM
Subject: Re: [PATCH 1/2] virtio-iommu: Fix 64kB host page size VFIO device assignmentQuoted 24 lines · Zhenzhong
Quoted 24 lines
-----Original Message-----
From: Jean-Philippe Brucker <[email protected]>
Sent: Wednesday, July 5, 2023 4:29 PM
Subject: Re: [PATCH 1/2] virtio-iommu: Fix 64kB host page size VFIO device assignmentQuoted 16 lines · Zhenzhong
Hi Eric,
Quoted 11 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: Tuesday, July 4, 2023 7:15 PM
Subject: [PATCH 1/2] virtio-iommu: Fix 64kB host page size VFIO device assignmentWhen running on a 64kB page size host and protecting a VFIO device with the virtio-iommu, qemu crashes with this kind of message:
qemu-kvm: virtio-iommu page mask
0xfffffffffffff000is incompatible with mask0x20010000Does
0x20010000mean only 512MB and 64KB super page mapping is supported for host iommu hw? 4KB mapping not supported?Quoted 8 lines
Quoted 5 lines
It's not a restriction by the HW IOMMU, but the host kernel. An Arm SMMU can implement 4KB, 16KB and/or 64KB granules, but the host kernel only advertises through VFIO the granule corresponding to host
PAGE_SIZE. This restriction is done byarm_lpae_restrict_pgsizes()in order to choose a page size when a device is driven by the host.Just curious why not advertises the Arm SMMU implemented granules to VFIO Eg:4KB, 16KB or 64KB granules?
That's possible, but the difficulty is setting up the page table configuration afterwards. At the moment the host installs the HW page tables early, when QEMU sets up the VFIO container. That initializes the page size bitmap because configuring the HW page tables requires picking one of the supported granules (setting TG0 in the SMMU Context Descriptor).
Quoted 6 lines
If the guest could pick a granule via an ATTACH request, then QEMU would need to tell the host kernel to install page tables with the desired granule at that point. That would require a new interface in VFIO to reconfigure a live container and replace the existing HW page tables configuration (before ATTACH, the container must already be configured with working page tables in order to implement boot-bypass, I think).
Thanks, clear, it looks different from x86. For x86 guest, 1GB mapping on guest side may be shadowed with 4KB mappings on host side in my understanding.
Regards
Zhenzhong
PATCH 2/2 virtio-iommu: Rework the trace in virtio_iommu_set_page_size_mask()
In reply to Eric AugerThe current error messages in virtio_iommu_set_page_size_mask() sound quite similar for different situations and miss the IOMMU memory region that causes the issue.
Clarify them and rework the comment.
Also remove the trace when the new page_size_mask is not applied as the current frozen granule is kept. This message is rather confusing for the end user and anyway the current granule would have been used by the driver
--- hw/virtio/virtio-iommu.c | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-)
hw/virtio/virtio-iommu.c +7 -12
@@ -1101,29 +1101,24 @@ static int virtio_iommu_set_page_size_mask(IOMMUMemoryRegion *mr, new_mask);if ((cur_mask & new_mask) ==0 ) { -error_setg (errp,"virtio-iommu page mask 0x%" PRIx64 -" is incompatible with mask 0x%" PRIx64, cur_mask, new_mask); +error_setg (errp,"virtio-iommu %s reports a page size mask 0x%" PRIx64 +" incompatible with currently supported mask 0x%" PRIx64, + mr->parent_obj.name, new_mask, cur_mask);return -1 ; }/* * Once the granule is frozen we can't change the mask anymore. If by * chance the hotplugged device supports the same granule, we can still -* accept it. Having a different masks is possible but the guest will use -* sub-optimal block sizes, so warn about it. +* accept it. */ if (s->granule_frozen) { -int new_granule =ctz64 (new_mask);int cur_granule =ctz64 (cur_mask); -if (new_granule != cur_granule) { -error_setg (errp,"virtio-iommu page mask 0x%" PRIx64 -" is incompatible with mask 0x%" PRIx64, cur_mask, - new_mask); +if (!(BIT (cur_granule) & new_mask)) { +error_setg (errp,"virtio-iommu %s does not support frozen granule 0x%" PRIx64, + mr->parent_obj.name,BIT (cur_granule));return -1 ; - }else if (new_mask != cur_mask) { -warn_report ("virtio-iommu page mask 0x%" PRIx64 -" does not match 0x%" PRIx64, cur_mask, new_mask); }return 0 ; }
Signature · 1 line
2.38.1
RE: [PATCH 2/2] virtio-iommu: Rework the trace in virtio_iommu_set_page_size_mask()
Quoted 52 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: Tuesday, July 4, 2023 7:15 PM
To: [email protected]; [email protected]; qemu- [email protected]; [email protected]; [email protected]; jean- [email protected]; Duan, Zhenzhong <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]
Subject: [PATCH 2/2] virtio-iommu: Rework the trace invirtio_iommu_set_page_size_mask()The current error messages in
virtio_iommu_set_page_size_mask()sound quite similar for different situations and miss the IOMMU memory region that causes the issue.Clarify them and rework the comment.
Also remove the trace when the new
page_size_maskis not applied as the current frozen granule is kept. This message is rather confusing for the end user and anyway the current granule would have been used by the driverSigned-off-by Eric Auger <[email protected]>---
hw/virtio/virtio-iommu.c| 19 +++++++------------1 file changed, 7 insertions(+), 12 deletions(-)hw/virtio/virtio-iommu.c +4
-4@@ -1101,29 +1101,24 @@ static intvirtio_iommu_set_page_size_mask (IOMMUMemoryRegion *mr, new_mask);if ((cur_mask & new_mask) ==0 ) { -error_setg (errp,"virtio-iommu page mask 0x%" PRIx64 -" is incompatible with mask 0x%" PRIx64, cur_mask, new_mask); +error_setg (errp,"virtio-iommu %s reports a page size mask 0x%" PRIx64 +" incompatible with currently supported mask 0x%" PRIx64, + mr->parent_obj.name, new_mask, cur_mask);return -1 ; }/* * Once the granule is frozen we can't change the mask anymore. If by * chance the hotplugged device supports the same granule, we can still -* accept it. Having a different masks is possible but the guest will use -* sub-optimal block sizes, so warn about it. +* accept it.
hw/virtio/virtio-iommu.c static int · line 1119
Quoted 10 lines
hw/virtio/virtio-iommu.c +1
-5*/ if (s->granule_frozen) { -int new_granule =ctz64 (new_mask);int cur_granule =ctz64 (cur_mask); -if (new_granule != cur_granule) { -error_setg (errp,"virtio-iommu page mask 0x%" PRIx64 -" is incompatible with mask 0x%" PRIx64, cur_mask, - new_mask); +if (!(BIT (cur_granule) & new_mask)) {
Good catch.
Thanks
Zhenzhong
Quoted 12 lines
hw/virtio/virtio-iommu.c +2
-3+error_setg (errp,"virtio-iommu %s does not support frozen granule 0x%"PRIx64, + mr->parent_obj.name,BIT (cur_granule));return -1 ; - }else if (new_mask != cur_mask) { -warn_report ("virtio-iommu page mask 0x%" PRIx64 -" does not match 0x%" PRIx64, cur_mask, new_mask); }return 0; }--
2.38.1
RE: [PATCH 2/2] virtio-iommu: Rework the trace in virtio_iommu_set_page_size_mask()
Quoted 66 lines
-----Original Message-----
From: Duan, Zhenzhong
Sent: Wednesday, July 5, 2023 12:56 PM
Subject: RE: [PATCH 2/2] virtio-iommu: Rework the trace invirtio_iommu_set_page_size_mask()Quoted 7 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: Tuesday, July 4, 2023 7:15 PM
To: [email protected]; [email protected]; qemu- [email protected]; [email protected]; [email protected]; jean- [email protected]; Duan, Zhenzhong <[email protected]>
Cc: [email protected]; [email protected];Quoted 48 lines
[email protected]
Subject: [PATCH 2/2] virtio-iommu: Rework the trace invirtio_iommu_set_page_size_mask()The current error messages in
virtio_iommu_set_page_size_mask()sound quite similar for different situations and miss the IOMMU memory region that causes the issue.Clarify them and rework the comment.
Also remove the trace when the new
page_size_maskis not applied as the current frozen granule is kept. This message is rather confusing for the end user and anyway the current granule would have been used by the driverSigned-off-by Eric Auger <[email protected]>---
hw/virtio/virtio-iommu.c| 19 +++++++------------1 file changed, 7 insertions(+), 12 deletions(-)hw/virtio/virtio-iommu.c index renamed from hw/virtio/virtio-iommu.c +0
-0
1eaf81bab5..0d9f7196fe100644hw/virtio/virtio-iommu.c +4
-5@@ -1101,29 +1101,24 @@ static intvirtio_iommu_set_page_size_mask (IOMMUMemoryRegion *mr, new_mask);if ((cur_mask & new_mask) ==0 ) { -error_setg (errp,"virtio-iommu page mask 0x%" PRIx64 -" is incompatible with mask 0x%" PRIx64, cur_mask, new_mask); +error_setg (errp,"virtio-iommu %s reports a page size mask 0x%" PRIx64 +" incompatible with currently supported mask 0x%" PRIx64, + mr->parent_obj.name, new_mask, cur_mask);return -1 ; }/* * Once the granule is frozen we can't change the mask anymore. If by * chance the hotplugged device supports the same granule, we can still -* accept it. Having a different masks is possible but the guest will use -* sub-optimal block sizes, so warn about it. +* accept it. */ if (s->granule_frozen) { -int new_granule =ctz64 (new_mask);int cur_granule =ctz64 (cur_mask);
Quoted 5 lines
hw/virtio/virtio-iommu.c static int virtio_iommu_set_page_size_mask(IOMMUMemoryRegion *mr, · line 1118
Quoted 5 lines
hw/virtio/virtio-iommu.c +1
-4-if (new_granule != cur_granule) { -error_setg (errp,"virtio-iommu page mask 0x%" PRIx64 -" is incompatible with mask 0x%" PRIx64, cur_mask, - new_mask); +if (!(BIT (cur_granule) & new_mask)) {
Sorry, I read this piece code again and got a question, if new_mask has finer granularity than cur_granule, should we allow it to pass even though BIT(cur_granule) is not set?
Thanks
Zhenzhong
Quoted 21 lines
Good catch.
Reviewed-by Zhenzhong Duan <[email protected]>Thanks
ZhenzhongQuoted 13 lines
hw/virtio/virtio-iommu.c +3
-4+error_setg (errp,"virtio-iommu %s does not support frozen + granule 0x%"PRIx64, + mr->parent_obj.name,BIT (cur_granule));return -1 ; - }else if (new_mask != cur_mask) { -warn_report ("virtio-iommu page mask 0x%" PRIx64 -" does not match 0x%" PRIx64, cur_mask, new_mask); }return 0 ; } --2.38.1
Re: [PATCH 2/2] virtio-iommu: Rework the trace in virtio_iommu_set_page_size_mask()
Hi Zhenghong,
Quoted 65 lines · Zhenzhong
Quoted 65 lines
-----Original Message-----
From: Duan, Zhenzhong
Sent: Wednesday, July 5, 2023 12:56 PM
Subject: RE: [PATCH 2/2] virtio-iommu: Rework the trace invirtio_iommu_set_page_size_mask()Quoted 7 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: Tuesday, July 4, 2023 7:15 PM
To: [email protected]; [email protected]; qemu- [email protected]; [email protected]; [email protected]; jean- [email protected]; Duan, Zhenzhong <[email protected]>
Cc: [email protected]; [email protected];Quoted 48 lines
[email protected]
Subject: [PATCH 2/2] virtio-iommu: Rework the trace invirtio_iommu_set_page_size_mask()The current error messages in
virtio_iommu_set_page_size_mask()sound quite similar for different situations and miss the IOMMU memory region that causes the issue.Clarify them and rework the comment.
Also remove the trace when the new
page_size_maskis not applied as the current frozen granule is kept. This message is rather confusing for the end user and anyway the current granule would have been used by the driverSigned-off-by Eric Auger <[email protected]>---
hw/virtio/virtio-iommu.c| 19 +++++++------------1 file changed, 7 insertions(+), 12 deletions(-)hw/virtio/virtio-iommu.c index renamed from hw/virtio/virtio-iommu.c +0
-0
1eaf81bab5..0d9f7196fe100644hw/virtio/virtio-iommu.c +4
-5@@ -1101,29 +1101,24 @@ static intvirtio_iommu_set_page_size_mask (IOMMUMemoryRegion *mr, new_mask);if ((cur_mask & new_mask) ==0 ) { -error_setg (errp,"virtio-iommu page mask 0x%" PRIx64 -" is incompatible with mask 0x%" PRIx64, cur_mask, new_mask); +error_setg (errp,"virtio-iommu %s reports a page size mask 0x%" PRIx64 +" incompatible with currently supported mask 0x%" PRIx64, + mr->parent_obj.name, new_mask, cur_mask);return -1 ; }/* * Once the granule is frozen we can't change the mask anymore. If by * chance the hotplugged device supports the same granule, we can still -* accept it. Having a different masks is possible but the guest will use -* sub-optimal block sizes, so warn about it. +* accept it. */ if (s->granule_frozen) { -int new_granule =ctz64 (new_mask);int cur_granule =ctz64 (cur_mask);
Quoted 8 lines
Quoted 5 lines
hw/virtio/virtio-iommu.c static int virtio_iommu_set_page_size_mask(IOMMUMemoryRegion *mr, · line 1118
Quoted 5 lines
hw/virtio/virtio-iommu.c +1
-4-if (new_granule != cur_granule) { -error_setg (errp,"virtio-iommu page mask 0x%" PRIx64 -" is incompatible with mask 0x%" PRIx64, cur_mask, - new_mask); +if (!(BIT (cur_granule) & new_mask)) {Sorry, I read this piece code again and got a question, if
new_maskhas finer granularity thancur_granule, should we allow it to pass even thoughBIT(cur_granule)is not set?
I think this should work but this is not straightforward to test. virtio-iommu would use the current granule for map/unmap. In map/unmap notifiers, this is split into pow2 ranges and cascaded to VFIO through vfio_dma_map/unmap. The iova and size are aligned with the smaller supported granule.
Jean, do you share this understanding or do I miss something.
Nevertheless the current code would have rejected that case and nobody complained at that point ;-)
thanks
Eric
Quoted 24 lines
Thanks
ZhenzhongQuoted 20 lines
Good catch.
Reviewed-by Zhenzhong Duan <[email protected]>Thanks
ZhenzhongQuoted 13 lines
hw/virtio/virtio-iommu.c +3
-4+error_setg (errp,"virtio-iommu %s does not support frozen + granule 0x%"PRIx64, + mr->parent_obj.name,BIT (cur_granule));return -1 ; - }else if (new_mask != cur_mask) { -warn_report ("virtio-iommu page mask 0x%" PRIx64 -" does not match 0x%" PRIx64, cur_mask, new_mask); }return 0 ; } --2.38.1
RE: [PATCH 2/2] virtio-iommu: Rework the trace in virtio_iommu_set_page_size_mask()
Hi Eric,
Quoted 85 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: Wednesday, July 5, 2023 9:17 PM
Subject: Re: [PATCH 2/2] virtio-iommu: Rework the trace invirtio_iommu_set_page_size_mask()Hi Zhenghong,
Quoted 45 lines · Zhenzhong
Quoted 45 lines
-----Original Message-----
From: Duan, Zhenzhong
Sent: Wednesday, July 5, 2023 12:56 PM
Subject: RE: [PATCH 2/2] virtio-iommu: Rework the trace invirtio_iommu_set_page_size_mask()Quoted 7 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: Tuesday, July 4, 2023 7:15 PM
To: [email protected]; [email protected]; qemu- [email protected]; [email protected]; [email protected]; jean- [email protected]; Duan, Zhenzhong <[email protected]>
Cc: [email protected]; [email protected];Quoted 28 lines
[email protected]
Subject: [PATCH 2/2] virtio-iommu: Rework the trace invirtio_iommu_set_page_size_mask()The current error messages in
virtio_iommu_set_page_size_mask()sound quite similar for different situations and miss the IOMMU memory region that causes the issue.Clarify them and rework the comment.
Also remove the trace when the new
page_size_maskis not applied as the current frozen granule is kept. This message is rather confusing for the end user and anyway the current granule would have been used by the driverSigned-off-by Eric Auger <[email protected]>---
hw/virtio/virtio-iommu.c| 19 +++++++------------1 file changed, 7 insertions(+), 12 deletions(-)hw/virtio/virtio-iommu.c +0
-0@@ -1101,29 +1101,24 @@ static intvirtio_iommu_set_page_size_mask (IOMMUMemoryRegion *mr, new_mask);Quoted 3 lines
Quoted 3 lines
hw/virtio/virtio-iommu.c static int virtio_iommu_set_page_size_mask(IOMMUMemoryRegion *mr, · line 1121
Quoted 3 lines
hw/virtio/virtio-iommu.c +0
-2if ((cur_mask & new_mask) ==0 ) { - error_setg(errp, "virtio-iommu page mask 0x%"PRIx64 - " is incompatible with mask 0x%"PRIx64, cur_mask,
new_mask);Quoted 1 line
Quoted 1 line
hw/virtio/virtio-iommu.c static int · line 1105
Quoted 1 line
hw/virtio/virtio-iommu.c +1
-0+ error_setg(errp, "virtio-iommu %s reports a page size mask0x%"PRIx64
Quoted 21 lines
Quoted 16 lines
Quoted 16 lines
hw/virtio/virtio-iommu.c +3
-3+" incompatible with currently supported mask 0x%" PRIx64, + mr->parent_obj.name, new_mask, cur_mask);return -1 ; }/* * Once the granule is frozen we can't change the mask anymore. If by * chance the hotplugged device supports the same granule, we can still -* accept it. Having a different masks is possible but the guest will use -* sub-optimal block sizes, so warn about it. +* accept it. */ if (s->granule_frozen) { -int new_granule =ctz64 (new_mask);int cur_granule =ctz64 (cur_mask);Quoted 5 lines
hw/virtio/virtio-iommu.c static int virtio_iommu_set_page_size_mask(IOMMUMemoryRegion *mr, · line 1118
Quoted 5 lines
hw/virtio/virtio-iommu.c +1
-4-if (new_granule != cur_granule) { -error_setg (errp,"virtio-iommu page mask 0x%" PRIx64 -" is incompatible with mask 0x%" PRIx64, cur_mask, - new_mask); +if (!(BIT (cur_granule) & new_mask)) {Quoted 4 lines
Sorry, I read this piece code again and got a question, if
new_maskhas finer granularity thancur_granule, should we allow it to pass even thoughBIT(cur_granule)is not set?
Quoted 5 lines
I think this should work but this is not straightforward to test. virtio-iommu would use the current granule for map/unmap. In map/unmap notifiers, this is split into pow2 ranges and cascaded to VFIO through vfio_dma_map/unmap. The iova and size are aligned with the smaller supported granule.
I see, guess you mean malicious guest which may send any mapping request to virtio-iommu backend. A well designed guest may use at least cur_granule as the granularity of its mapping, even with pow2 split, cur_granule is guaranteed at backend.
Thanks
Zhenzhong
Quoted 34 lines
Jean, do you share this understanding or do I miss something.
Nevertheless the current code would have rejected that case and nobody complained at that point ;-)
thanks
Eric
Quoted 24 lines
Thanks
ZhenzhongQuoted 20 lines
Good catch.
Reviewed-by Zhenzhong Duan <[email protected]>Thanks
ZhenzhongQuoted 13 lines
hw/virtio/virtio-iommu.c +3
-4+ error_setg(errp, "virtio-iommu %s does not support + frozen granule 0x%"PRIx64, + mr->parent_obj.name,BIT (cur_granule));return -1 ; - }else if (new_mask != cur_mask) { -warn_report ("virtio-iommu page mask 0x%" PRIx64 -" does not match 0x%" PRIx64, cur_mask, new_mask); }return 0 ; } --2.38.1
Re: [PATCH 2/2] virtio-iommu: Rework the trace in virtio_iommu_set_page_size_mask()
In reply to Eric AugerQuoted 36 lines · Eric Auger
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
hw/virtio/virtio-iommu.c index renamed from hw/virtio/virtio-iommu.c +0
-0
1eaf81bab5..0d9f7196fe100644hw/virtio/virtio-iommu.c +4
-5@@ -1101,29 +1101,24 @@ static intvirtio_iommu_set_page_size_mask (IOMMUMemoryRegion *mr, new_mask);if ((cur_mask & new_mask) ==0 ) { -error_setg (errp,"virtio-iommu page mask 0x%" PRIx64 -" is incompatible with mask 0x%" PRIx64, cur_mask, new_mask); +error_setg (errp,"virtio-iommu %s reports a page size mask 0x%" PRIx64 +" incompatible with currently supported mask 0x%" PRIx64, + mr->parent_obj.name, new_mask, cur_mask);return -1 ; }/* * Once the granule is frozen we can't change the mask anymore. If by * chance the hotplugged device supports the same granule, we can still -* accept it. Having a different masks is possible but the guest will use -* sub-optimal block sizes, so warn about it. +* accept it. */ if (s->granule_frozen) { -int new_granule =ctz64 (new_mask);int cur_granule =ctz64 (cur_mask);Quoted 8 lines
Quoted 5 lines
hw/virtio/virtio-iommu.c static int virtio_iommu_set_page_size_mask(IOMMUMemoryRegion *mr, · line 1118
Quoted 5 lines
hw/virtio/virtio-iommu.c +1
-4-if (new_granule != cur_granule) { -error_setg (errp,"virtio-iommu page mask 0x%" PRIx64 -" is incompatible with mask 0x%" PRIx64, cur_mask, - new_mask); +if (!(BIT (cur_granule) & new_mask)) {Sorry, I read this piece code again and got a question, if
new_maskhas finer granularity thancur_granule, should we allow it to pass even thoughBIT(cur_granule)is not set?
Quoted 7 lines
I think this should work but this is not straightforward to test. virtio-iommu would use the current granule for map/unmap. In map/unmap notifiers, this is split into pow2 ranges and cascaded to VFIO through vfio_dma_map/unmap. The iova and size are aligned with the smaller supported granule.
Jean, do you share this understanding or do I miss something.
Yes, I also think that would work. The guest would only issue mappings with the larger granularity, which can be applied by VFIO with a finer granule. However I doubt we're going to encounter this case, because seeing a cur_granule larger than 4k here means that a VFIO device has already been assigned with a large granule like 64k, and we're trying to add a new device with 4k. This indicates two HW IOMMUs supporting different granules in the same system, which seems unlikely.
Hopefully by the time we actually need this (if ever) we will support per-endpoint probe properties, which allow informing the guest about different hardware properties instead of relying on one global property in the virtio config.
Thanks,
Jean
Re: [PATCH 2/2] virtio-iommu: Rework the trace in virtio_iommu_set_page_size_mask()
Hi Jean,
Quoted 43 lines · Jean-Philippe Brucker
Quoted 42 lines · Eric Auger
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
hw/virtio/virtio-iommu.c index renamed from hw/virtio/virtio-iommu.c +0
-0
1eaf81bab5..0d9f7196fe100644hw/virtio/virtio-iommu.c +4
-5@@ -1101,29 +1101,24 @@ static intvirtio_iommu_set_page_size_mask (IOMMUMemoryRegion *mr, new_mask);if ((cur_mask & new_mask) ==0 ) { -error_setg (errp,"virtio-iommu page mask 0x%" PRIx64 -" is incompatible with mask 0x%" PRIx64, cur_mask, new_mask); +error_setg (errp,"virtio-iommu %s reports a page size mask 0x%" PRIx64 +" incompatible with currently supported mask 0x%" PRIx64, + mr->parent_obj.name, new_mask, cur_mask);return -1 ; }/* * Once the granule is frozen we can't change the mask anymore. If by * chance the hotplugged device supports the same granule, we can still -* accept it. Having a different masks is possible but the guest will use -* sub-optimal block sizes, so warn about it. +* accept it. */ if (s->granule_frozen) { -int new_granule =ctz64 (new_mask);int cur_granule =ctz64 (cur_mask);Quoted 8 lines
Quoted 5 lines
hw/virtio/virtio-iommu.c static int virtio_iommu_set_page_size_mask(IOMMUMemoryRegion *mr, · line 1118
Quoted 5 lines
hw/virtio/virtio-iommu.c +1
-4-if (new_granule != cur_granule) { -error_setg (errp,"virtio-iommu page mask 0x%" PRIx64 -" is incompatible with mask 0x%" PRIx64, cur_mask, - new_mask); +if (!(BIT (cur_granule) & new_mask)) {Sorry, I read this piece code again and got a question, if
new_maskhas finer granularity thancur_granule, should we allow it to pass even thoughBIT(cur_granule)is not set?I think this should work but this is not straightforward to test. virtio-iommu would use the current granule for map/unmap. In map/unmap notifiers, this is split into pow2 ranges and cascaded to VFIO through vfio_dma_map/unmap. The iova and size are aligned with the smaller supported granule.
Quoted 8 lines
Quoted 1 line
Jean, do you share this understanding or do I miss something.
Yes, I also think that would work. The guest would only issue mappings with the larger granularity, which can be applied by VFIO with a finer granule. However I doubt we're going to encounter this case, because seeing a
cur_granulelarger than 4k here means that a VFIO device has already been assigned with a large granule like 64k, and we're trying to add a new device with 4k. This indicates two HW IOMMUs supporting different granules in the same system, which seems unlikely.
agreed
Quoted 5 lines
Hopefully by the time we actually need this (if ever) we will support per-endpoint probe properties, which allow informing the guest about different hardware properties instead of relying on one global property in the virtio config.
yes looking forward to reviewing that.
Thanks
Eric
Quoted 4 lines
Thanks,
Jean
RE: [PATCH 2/2] virtio-iommu: Rework the trace in virtio_iommu_set_page_size_mask()
In reply to Jean-Philippe BruckerQuoted 56 lines
-----Original Message-----
From: Jean-Philippe Brucker <[email protected]>
Sent: Thursday, July 6, 2023 10:36 PM
Subject: Re: [PATCH 2/2] virtio-iommu: Rework the trace invirtio_iommu_set_page_size_mask()Quoted 11 lines · Eric Auger
Quoted 11 lines
Quoted 11 lines
hw/virtio/virtio-iommu.c static int · line 1106
Quoted 11 lines
hw/virtio/virtio-iommu.c +0
-2@@ -1101,29 +1101,24 @@ static intvirtio_iommu_set_page_size_mask (IOMMUMemoryRegion *mr, new_mask);if ((cur_mask & new_mask) ==0 ) { - error_setg(errp, "virtio-iommu page mask 0x%"PRIx64 - " is incompatible with mask 0x%"PRIx64, cur_mask,
new_mask);Quoted 1 line
Quoted 1 line
Quoted 1 line
hw/virtio/virtio-iommu.c static int · line 1105
Quoted 1 line
hw/virtio/virtio-iommu.c +1
-0+ error_setg(errp, "virtio-iommu %s reports a page size mask0x%"PRIx64
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
hw/virtio/virtio-iommu.c static int virtio_iommu_set_page_size_mask(IOMMUMemoryRegion *mr,
Quoted 8 lines
hw/virtio/virtio-iommu.c +2
-0+" incompatible with currently supported mask 0x%" PRIx64, + mr->parent_obj.name, new_mask, cur_mask);return -1 ; }/* * Once the granule is frozen we can't change the mask anymore. If by * chance the hotplugged device supports the same granule, we can still
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
hw/virtio/virtio-iommu.c +2
-7-* accept it. Having a different masks is possible but the guest will use -* sub-optimal block sizes, so warn about it. +* accept it. */ if (s->granule_frozen) { -int new_granule =ctz64 (new_mask);int cur_granule =ctz64 (cur_mask); -if (new_granule != cur_granule) { -error_setg (errp,"virtio-iommu page mask 0x%" PRIx64 -" is incompatible with mask 0x%" PRIx64, cur_mask, - new_mask); +if (!(BIT (cur_granule) & new_mask)) {Quoted 6 lines
Quoted 4 lines
Sorry, I read this piece code again and got a question, if
new_maskhas finer granularity thancur_granule, should we allow it to pass even thoughBIT(cur_granule)is not set?I think this should work but this is not straightforward to test. virtio-iommu would use the current granule for map/unmap. In
map/unmap
Quoted 5 lines
notifiers, this is split into pow2 ranges and cascaded to VFIO through vfio_dma_map/unmap. The iova and size are aligned with the smaller supported granule.
Jean, do you share this understanding or do I miss something.
Quoted 7 lines
Yes, I also think that would work. The guest would only issue mappings with the larger granularity, which can be applied by VFIO with a finer granule. However I doubt we're going to encounter this case, because seeing a
cur_granulelarger than 4k here means that a VFIO device has already been assigned with a large granule like 64k, and we're trying to add a new device with 4k. This indicates two HW IOMMUs supporting different granules in the same system, which seems unlikely.
Indeed. Another scenario I can think of is migration from src with 64k granule to dest with 4k, then hotplug a new device with 4k. Not clear if it's allowed to migrate between host with different granule.
Thanks
Zhenzhong
Re: [PATCH 2/2] virtio-iommu: Rework the trace in virtio_iommu_set_page_size_mask()
In reply to Eric AugerQuoted 76 lines · Eric Auger
Hi Zhenghong,
Quoted 65 lines · Zhenzhong
Quoted 65 lines
-----Original Message-----
From: Duan, Zhenzhong
Sent: Wednesday, July 5, 2023 12:56 PM
Subject: RE: [PATCH 2/2] virtio-iommu: Rework the trace invirtio_iommu_set_page_size_mask()Quoted 7 lines
-----Original Message-----
From: Eric Auger <[email protected]>
Sent: Tuesday, July 4, 2023 7:15 PM
To: [email protected]; [email protected]; qemu- [email protected]; [email protected]; [email protected]; jean- [email protected]; Duan, Zhenzhong <[email protected]>
Cc: [email protected]; [email protected];Quoted 48 lines
[email protected]
Subject: [PATCH 2/2] virtio-iommu: Rework the trace invirtio_iommu_set_page_size_mask()The current error messages in
virtio_iommu_set_page_size_mask()sound quite similar for different situations and miss the IOMMU memory region that causes the issue.Clarify them and rework the comment.
Also remove the trace when the new
page_size_maskis not applied as the current frozen granule is kept. This message is rather confusing for the end user and anyway the current granule would have been used by the driverSigned-off-by Eric Auger <[email protected]>---
hw/virtio/virtio-iommu.c| 19 +++++++------------1 file changed, 7 insertions(+), 12 deletions(-)hw/virtio/virtio-iommu.c index renamed from hw/virtio/virtio-iommu.c +0
-0
1eaf81bab5..0d9f7196fe100644hw/virtio/virtio-iommu.c +4
-5@@ -1101,29 +1101,24 @@ static intvirtio_iommu_set_page_size_mask (IOMMUMemoryRegion *mr, new_mask);if ((cur_mask & new_mask) ==0 ) { -error_setg (errp,"virtio-iommu page mask 0x%" PRIx64 -" is incompatible with mask 0x%" PRIx64, cur_mask, new_mask); +error_setg (errp,"virtio-iommu %s reports a page size mask 0x%" PRIx64 +" incompatible with currently supported mask 0x%" PRIx64, + mr->parent_obj.name, new_mask, cur_mask);return -1 ; }/* * Once the granule is frozen we can't change the mask anymore. If by * chance the hotplugged device supports the same granule, we can still -* accept it. Having a different masks is possible but the guest will use -* sub-optimal block sizes, so warn about it. +* accept it. */ if (s->granule_frozen) { -int new_granule =ctz64 (new_mask);int cur_granule =ctz64 (cur_mask);Quoted 8 lines
Quoted 5 lines
hw/virtio/virtio-iommu.c static int virtio_iommu_set_page_size_mask(IOMMUMemoryRegion *mr, · line 1118
Quoted 5 lines
hw/virtio/virtio-iommu.c +1
-4-if (new_granule != cur_granule) { -error_setg (errp,"virtio-iommu page mask 0x%" PRIx64 -" is incompatible with mask 0x%" PRIx64, cur_mask, - new_mask); +if (!(BIT (cur_granule) & new_mask)) {Sorry, I read this piece code again and got a question, if
new_maskhas finer granularity thancur_granule, should we allow it to pass even thoughBIT(cur_granule)is not set?
Quoted 5 lines
I think this should work but this is not straightforward to test. virtio-iommu would use the current granule for map/unmap. In map/unmap notifiers, this is split into pow2 ranges and cascaded to VFIO through vfio_dma_map/unmap. The iova and size are aligned with the smaller supported granule.
Maybe add a comment down the road. Not urgent.
Quoted 33 lines
Jean, do you share this understanding or do I miss something.
Nevertheless the current code would have rejected that case and nobody complained at that point ;-)
thanks
Eric
Quoted 24 lines
Thanks
ZhenzhongQuoted 20 lines
Good catch.
Reviewed-by Zhenzhong Duan <[email protected]>Thanks
ZhenzhongQuoted 13 lines
hw/virtio/virtio-iommu.c +3
-4+error_setg (errp,"virtio-iommu %s does not support frozen + granule 0x%"PRIx64, + mr->parent_obj.name,BIT (cur_granule));return -1 ; - }else if (new_mask != cur_mask) { -warn_report ("virtio-iommu page mask 0x%" PRIx64 -" does not match 0x%" PRIx64, cur_mask, new_mask); }return 0 ; } --2.38.1