Skip to contentSkip to conversation

virtio-iommu: Use qemu_real_host_page_mask as default page_size_mask

PATCH Superseded Eric Auger 2 participants 4 messages Dec 21, 2023 – Jan 16, 2024

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Eric Auger

author
#0 lore raw

PATCH 1/1 virtio-iommu: Use qemu_real_host_page_mask as default page_size_mask

We used to set default page_size_mask to qemu_target_page_mask() but with VFIO assignment it makes more sense to use the actual host page mask instead.

So from now on qemu_real_host_page_mask() will be used as a default. To be able to migrate older code, we increase the vmstat version_id to 3 and if an older incoming v2 stream is detected we set the previous default value.

The new default is well adapted to configs where host and guest have the same page size. This allows to fix hotplugging VFIO devices on a 64kB guest and a 64kB host. This test case has been failing before and even crashing qemu with hw_error("vfio: DMA mapping failed, unable to continue") in VFIO common). Indeed the hot-attached VFIO device would call memory_region_iommu_set_page_size_mask with 64kB mask whereas after the granule was frozen to 4kB on machine init done. Now this works. However the new default will prevent 4kB guest on 64kB host because the granule will be set to 64kB which would be larger than the guest page size. In that situation, the virtio-iommu driver fails the viommu_domain_finalise() with
"granule 0x10000 larger than system page zie 0x1000".

The current limitation of global granule in the virtio-iommu should be removed and turned into per domain granule. But until we get this upgraded, this new default is probably better because I don't think anyone is currently interested in running a 4kB page size guest with virtio-iommu on a 64kB host. However supporting 64kB guest on 64kB host with virtio-iommu and VFIO looks a more important feature.

---
 hw/virtio/virtio-iommu.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
hw/virtio/virtio-iommu.c +5 -2
index 9d463efc52..b77e3644ea 100644
@@ -1313,7 +1313,7 @@ static void virtio_iommu_device_realize(DeviceState *dev, Error **errp)
      * in vfio realize
      */
     s->config.bypass = s->boot_bypass;
-    s->config.page_size_mask = qemu_target_page_mask();
+    s->config.page_size_mask = qemu_real_host_page_mask();
     s->config.input_range.end = UINT64_MAX;
     s->config.domain_range.end = UINT32_MAX;
     s->config.probe_size = VIOMMU_PROBE_SIZE;
@@ -1491,13 +1491,16 @@ static int iommu_post_load(void *opaque, int version_id)
      * still correct.
      */
     virtio_iommu_switch_address_space_all(s);
+    if (version_id <= 2) {
+        s->config.page_size_mask = qemu_target_page_mask();
+    }
     return 0;
 }
 
 static const VMStateDescription vmstate_virtio_iommu_device = {
     .name = "virtio-iommu-device",
     .minimum_version_id = 2,
-    .version_id = 2,
+    .version_id = 3,
     .post_load = iommu_post_load,
     .fields = (VMStateField[]) {
         VMSTATE_GTREE_DIRECT_KEY_V(domains, VirtIOIOMMU, 2,
Signature · 1 line
2.27.0
author
#1 lore raw

Hi,

Quoted 22 lines · Eric Auger

We used to set default page_size_mask to qemu_target_page_mask() but with VFIO assignment it makes more sense to use the actual host page mask instead.

So from now on qemu_real_host_page_mask() will be used as a default. To be able to migrate older code, we increase the vmstat version_id to 3 and if an older incoming v2 stream is detected we set the previous default value.

The new default is well adapted to configs where host and guest have the same page size. This allows to fix hotplugging VFIO devices on a 64kB guest and a 64kB host. This test case has been failing before and even crashing qemu with hw_error("vfio: DMA mapping failed, unable to continue") in VFIO common). Indeed the hot-attached VFIO device would call memory_region_iommu_set_page_size_mask with 64kB mask whereas after the granule was frozen to 4kB on machine init done. Now this works. However the new default will prevent 4kB guest on 64kB host because the granule will be set to 64kB which would be larger than the guest page size. In that situation, the virtio-iommu driver fails the viommu_domain_finalise() with
"granule 0x10000 larger than system page zie 0x1000".

Quoted 7 lines

The current limitation of global granule in the virtio-iommu should be removed and turned into per domain granule. But until we get this upgraded, this new default is probably better because I don't think anyone is currently interested in running a 4kB page size guest with virtio-iommu on a 64kB host. However supporting 64kB guest on 64kB host with virtio-iommu and VFIO looks a more important feature.

Gentle ping

Thanks

Eric

Quoted 37 lines
---
 hw/virtio/virtio-iommu.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
hw/virtio/virtio-iommu.c +5 -2
index 9d463efc52..b77e3644ea 100644
@@ -1313,7 +1313,7 @@ static void virtio_iommu_device_realize(DeviceState *dev, Error **errp)
      * in vfio realize
      */
     s->config.bypass = s->boot_bypass;
-    s->config.page_size_mask = qemu_target_page_mask();
+    s->config.page_size_mask = qemu_real_host_page_mask();
     s->config.input_range.end = UINT64_MAX;
     s->config.domain_range.end = UINT32_MAX;
     s->config.probe_size = VIOMMU_PROBE_SIZE;
@@ -1491,13 +1491,16 @@ static int iommu_post_load(void *opaque, int version_id)
      * still correct.
      */
     virtio_iommu_switch_address_space_all(s);
+    if (version_id <= 2) {
+        s->config.page_size_mask = qemu_target_page_mask();
+    }
     return 0;
 }
 
 static const VMStateDescription vmstate_virtio_iommu_device = {
     .name = "virtio-iommu-device",
     .minimum_version_id = 2,
-    .version_id = 2,
+    .version_id = 3,
     .post_load = iommu_post_load,
     .fields = (VMStateField[]) {
         VMSTATE_GTREE_DIRECT_KEY_V(domains, VirtIOIOMMU, 2,
Tested-by Reviewed-by on v1 1/1
#2 lore raw
In reply to Eric Auger
Quoted 9 lines · Eric Auger

We used to set default page_size_mask to qemu_target_page_mask() but with VFIO assignment it makes more sense to use the actual host page mask instead.

So from now on qemu_real_host_page_mask() will be used as a default. To be able to migrate older code, we increase the vmstat version_id to 3 and if an older incoming v2 stream is detected we set the previous default value.

Quoted 7 lines

The new default is well adapted to configs where host and guest have the same page size. This allows to fix hotplugging VFIO devices on a 64kB guest and a 64kB host. This test case has been failing before and even crashing qemu with hw_error("vfio: DMA mapping failed, unable to continue") in VFIO common). Indeed the hot-attached VFIO device would call memory_region_iommu_set_page_size_mask with 64kB mask whereas after the granule was frozen to 4kB on machine init done.

I guess TARGET_PAGE_MASK is always 4kB on arm64 CPUs, since it's the smallest supported and the guest configures its page size at runtime. Even if QEMU's software IOMMU can deal with any page size, VFIO can't so passing the host page size seems more accurate than forcing a value of 4kB.

Quoted 5 lines

Now this works. However the new default will prevent 4kB guest on 64kB host because the granule will be set to 64kB which would be larger than the guest page size. In that situation, the virtio-iommu driver fails the viommu_domain_finalise() with
"granule 0x10000 larger than system page zie 0x1000".

"size"
(it could matter if someone searches for this message later)

Quoted 10 lines

The current limitation of global granule in the virtio-iommu should be removed and turned into per domain granule. But until we get this upgraded, this new default is probably better because I don't think anyone is currently interested in running a 4kB page size guest with virtio-iommu on a 64kB host. However supporting 64kB guest on 64kB host with virtio-iommu and VFIO looks a more important feature.

So to summarize the configurations that work for hotplug (tested with QEMU system emulation with SMMU + QEMU VMM with virtio-iommu):

 Host | Guest | virtio-net | IGB passthrough
  4k  | 4k    | Y          | Y
  64k | 64k   | Y          | N -> Y (fixed by this patch)
  64k | 4k    | Y -> N     | N
  4k  | 64k   | Y          | Y

The change is a reasonable trade-off in my opinion. It fixes the more common 64k on 64k case, and for 4k on 64k, the error is now contained to the guest and made clear ("granule 0x10000 larger than system page size 0x1000") instead of crashing the VMM. A guest OS now discovers that the host needs DMA buffers aligned on 64k and could actually support this case (but Linux won't because it can't control the origin of all DMA buffers). Later, support for page tables will enable 4k on 64k for all devices.

Quoted 38 lines
---
 hw/virtio/virtio-iommu.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
hw/virtio/virtio-iommu.c +5 -2
index 9d463efc52..b77e3644ea 100644
@@ -1313,7 +1313,7 @@ static void virtio_iommu_device_realize(DeviceState *dev, Error **errp)
      * in vfio realize
      */
     s->config.bypass = s->boot_bypass;
-    s->config.page_size_mask = qemu_target_page_mask();
+    s->config.page_size_mask = qemu_real_host_page_mask();
     s->config.input_range.end = UINT64_MAX;
     s->config.domain_range.end = UINT32_MAX;
     s->config.probe_size = VIOMMU_PROBE_SIZE;
@@ -1491,13 +1491,16 @@ static int iommu_post_load(void *opaque, int version_id)
      * still correct.
      */
     virtio_iommu_switch_address_space_all(s);
+    if (version_id <= 2) {
+        s->config.page_size_mask = qemu_target_page_mask();
+    }
     return 0;
 }
 
 static const VMStateDescription vmstate_virtio_iommu_device = {
     .name = "virtio-iommu-device",
     .minimum_version_id = 2,
-    .version_id = 2,
+    .version_id = 3,
     .post_load = iommu_post_load,
     .fields = (VMStateField[]) {
         VMSTATE_GTREE_DIRECT_KEY_V(domains, VirtIOIOMMU, 2,
Signature · 1 line
2.27.0
author
#3 lore raw

Hi Jean,

Quoted 17 lines · Jean-Philippe Brucker
Quoted 9 lines · Eric Auger

We used to set default page_size_mask to qemu_target_page_mask() but with VFIO assignment it makes more sense to use the actual host page mask instead.

So from now on qemu_real_host_page_mask() will be used as a default. To be able to migrate older code, we increase the vmstat version_id to 3 and if an older incoming v2 stream is detected we set the previous default value.

Quoted 7 lines

The new default is well adapted to configs where host and guest have the same page size. This allows to fix hotplugging VFIO devices on a 64kB guest and a 64kB host. This test case has been failing before and even crashing qemu with hw_error("vfio: DMA mapping failed, unable to continue") in VFIO common). Indeed the hot-attached VFIO device would call memory_region_iommu_set_page_size_mask with 64kB mask whereas after the granule was frozen to 4kB on machine init done.

Quoted 2 lines

I guess TARGET_PAGE_MASK is always 4kB on arm64 CPUs, since it's the smallest supported and the guest configures its page size at runtime.

correct

Quoted 3 lines

Even if QEMU's software IOMMU can deal with any page size, VFIO can't so passing the host page size seems more accurate than forcing a value of 4kB.

that's my guess too

Quoted 8 lines
Quoted 6 lines

Now this works. However the new default will prevent 4kB guest on 64kB host because the granule will be set to 64kB which would be larger than the guest page size. In that situation, the virtio-iommu driver fails the viommu_domain_finalise() with
"granule 0x10000 larger than system page zie 0x1000".

"size"
(it could matter if someone searches for this message later)

sure I will correct it

Quoted 10 lines
Quoted 10 lines

The current limitation of global granule in the virtio-iommu should be removed and turned into per domain granule. But until we get this upgraded, this new default is probably better because I don't think anyone is currently interested in running a 4kB page size guest with virtio-iommu on a 64kB host. However supporting 64kB guest on 64kB host with virtio-iommu and VFIO looks a more important feature.

Quoted 8 lines

So to summarize the configurations that work for hotplug (tested with QEMU system emulation with SMMU + QEMU VMM with virtio-iommu):

 Host | Guest | virtio-net | IGB passthrough
  4k  | 4k    | Y          | Y
  64k | 64k   | Y          | N -> Y (fixed by this patch)
  64k | 4k    | Y -> N     | N
  4k  | 64k   | Y          | Y

That table is correct

Quoted 11 lines

The change is a reasonable trade-off in my opinion. It fixes the more common 64k on 64k case, and for 4k on 64k, the error is now contained to the guest and made clear ("granule 0x10000 larger than system page size 0x1000") instead of crashing the VMM. A guest OS now discovers that the host needs DMA buffers aligned on 64k and could actually support this case (but Linux won't because it can't control the origin of all DMA buffers). Later, support for page tables will enable 4k on 64k for all devices.

Many thanks!

Eric

Quoted 39 lines
Quoted 39 lines
---
 hw/virtio/virtio-iommu.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
hw/virtio/virtio-iommu.c +5 -2
index 9d463efc52..b77e3644ea 100644
@@ -1313,7 +1313,7 @@ static void virtio_iommu_device_realize(DeviceState *dev, Error **errp)
      * in vfio realize
      */
     s->config.bypass = s->boot_bypass;
-    s->config.page_size_mask = qemu_target_page_mask();
+    s->config.page_size_mask = qemu_real_host_page_mask();
     s->config.input_range.end = UINT64_MAX;
     s->config.domain_range.end = UINT32_MAX;
     s->config.probe_size = VIOMMU_PROBE_SIZE;
@@ -1491,13 +1491,16 @@ static int iommu_post_load(void *opaque, int version_id)
      * still correct.
      */
     virtio_iommu_switch_address_space_all(s);
+    if (version_id <= 2) {
+        s->config.page_size_mask = qemu_target_page_mask();
+    }
     return 0;
 }
 
 static const VMStateDescription vmstate_virtio_iommu_device = {
     .name = "virtio-iommu-device",
     .minimum_version_id = 2,
-    .version_id = 2,
+    .version_id = 3,
     .post_load = iommu_post_load,
     .fields = (VMStateField[]) {
         VMSTATE_GTREE_DIRECT_KEY_V(domains, VirtIOIOMMU, 2,
Signature · 1 line
2.27.0