Skip to contentSkip to conversation

Allow memory_region_register_iommu_notifier() to fail

PATCH Superseded Eric Auger 2 participants 8 messages Sep 19, 2019 – Sep 20, 2019

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Eric Auger

author
#0 lore raw

v2 cover Allow memory_region_register_iommu_notifier() to fail

This series allows the memory_region_register_iommu_notifier() to fail. As of now, when a MAP notifier is attempted to be registered along with SMMUv3, Intel iommu without caching mode or AMD IOMMU, we exit in the IOMMU MR notify_flag_changed() callback. In case of VFIO assigned device hotplug, this could be handled more nicely directly within the VFIO code, simply rejecting the hotplug without exiting. This is what the series achieves by handling the memory_region_register_iommu_notifier() returned value.

Best Regards

Eric

This series can be found at:
https://github.com/eauger/qemu/tree/v4.1.0_register_iommu_notifier_fail_v2

History:

v1 -> v2:
- Intel IOMMU now handles the problem differently with machine init done notifier and machine hotplug allowed hook.
- use assert(!ret)
- message rewording in SMMUv3

Follow-up of "VFIO/SMMUv3: Fail on VFIO/HW nested paging detection"
https://patchew.org/QEMU/[email protected]/

Eric Auger (5):
memory: allow memory_region_register_iommu_notifier() to fail vfio/common: Handle memory_region_register_iommu_notifier() failure
exec: assert on memory_region_register_iommu_notifier() failure
vhost: assert on memory_region_register_iommu_notifier() failure
amd_iommu: Let amdvi_iommu_notify_flag_changed() fail

 exec.c                |  6 ++++--
 hw/arm/smmuv3.c       | 12 +++++++-----
 hw/i386/amd_iommu.c   |  9 +++++----
 hw/i386/intel_iommu.c |  7 ++++---
 hw/ppc/spapr_iommu.c  |  7 ++++---
 hw/vfio/common.c      |  8 ++++++--
 hw/virtio/vhost.c     |  5 +++--
 include/exec/memory.h | 18 +++++++++++++-----
 memory.c              | 28 ++++++++++++++++++----------
 9 files changed, 64 insertions(+), 36 deletions(-)
Signature · 1 line
2.20.1
author
#1 lore raw

PATCH v2 1/5 memory: allow memory_region_register_iommu_notifier() to fail

Currently, when a notifier is attempted to be registered and its flags are not supported (especially the MAP one) by the IOMMU MR, we generally abruptly exit in the IOMMU code. The failure could be handled more nicely in the caller and especially in the VFIO code.

So let's allow memory_region_register_iommu_notifier() to fail as well as notify_flag_changed() callback.

All sites implementing the callback are updated. This patch does not yet remove the exit(1) in the amd_iommu code.

in SMMUv3 we turn the warning message into an error message saying that the assigned device would not work properly.

---

v2 -> v3:
- turn the warning message into an error message

---
 hw/arm/smmuv3.c       | 12 +++++++-----
 hw/i386/amd_iommu.c   |  7 ++++---
 hw/i386/intel_iommu.c |  7 ++++---
 hw/ppc/spapr_iommu.c  |  7 ++++---
 include/exec/memory.h | 18 +++++++++++++-----
 memory.c              | 28 ++++++++++++++++++----------
 6 files changed, 50 insertions(+), 29 deletions(-)
hw/arm/smmuv3.c +7 -5
index db051dcac8..2845aa7ffd 100644
@@ -1469,9 +1469,9 @@ static void smmuv3_class_init(ObjectClass *klass, void *data)
     dc->realize = smmu_realize;
 }
 
-static void smmuv3_notify_flag_changed(IOMMUMemoryRegion *iommu,
-                                       IOMMUNotifierFlag old,
-                                       IOMMUNotifierFlag new)
+static int smmuv3_notify_flag_changed(IOMMUMemoryRegion *iommu,
+                                      IOMMUNotifierFlag old,
+                                      IOMMUNotifierFlag new)
 {
     SMMUDevice *sdev = container_of(iommu, SMMUDevice, iommu);
     SMMUv3State *s3 = sdev->smmu;
@@ -1481,8 +1481,9 @@ static void smmuv3_notify_flag_changed(IOMMUMemoryRegion *iommu,
         int bus_num = pci_bus_num(sdev->bus);
         PCIDevice *pcidev = pci_find_device(sdev->bus, bus_num, sdev->devfn);
 
-        warn_report("SMMUv3 does not support notification on MAP: "
-                     "device %s will not function properly", pcidev->name);
+        error_report("SMMUv3 does not support notification on MAP: "
+                     "device %s would not function properly", pcidev->name);
+        return -EINVAL;
     }
 
     if (old == IOMMU_NOTIFIER_NONE) {
@@ -1492,6 +1493,7 @@ static void smmuv3_notify_flag_changed(IOMMUMemoryRegion *iommu,
         trace_smmuv3_notify_flag_del(iommu->parent_obj.name);
         QLIST_REMOVE(sdev, next);
     }
+    return 0;
 }
 
 static void smmuv3_iommu_memory_region_class_init(ObjectClass *klass,
hw/i386/amd_iommu.c +4 -3
index 08884523e2..137ba367db 100644
@@ -1466,9 +1466,9 @@ static const MemoryRegionOps mmio_mem_ops = {
     }
 };
 
-static void amdvi_iommu_notify_flag_changed(IOMMUMemoryRegion *iommu,
-                                            IOMMUNotifierFlag old,
-                                            IOMMUNotifierFlag new)
+static int amdvi_iommu_notify_flag_changed(IOMMUMemoryRegion *iommu,
+                                           IOMMUNotifierFlag old,
+                                           IOMMUNotifierFlag new)
 {
     AMDVIAddressSpace *as = container_of(iommu, AMDVIAddressSpace, iommu);
 
@@ -1478,6 +1478,7 @@ static void amdvi_iommu_notify_flag_changed(IOMMUMemoryRegion *iommu,
                      PCI_FUNC(as->devfn));
         exit(1);
     }
+    return 0;
 }
 
 static void amdvi_init(AMDVIState *s)
hw/i386/intel_iommu.c +4 -3
index f1de8fdb75..6465a570a1 100644
@@ -2929,9 +2929,9 @@ static IOMMUTLBEntry vtd_iommu_translate(IOMMUMemoryRegion *iommu, hwaddr addr,
     return iotlb;
 }
 
-static void vtd_iommu_notify_flag_changed(IOMMUMemoryRegion *iommu,
-                                          IOMMUNotifierFlag old,
-                                          IOMMUNotifierFlag new)
+static int vtd_iommu_notify_flag_changed(IOMMUMemoryRegion *iommu,
+                                         IOMMUNotifierFlag old,
+                                         IOMMUNotifierFlag new)
 {
     VTDAddressSpace *vtd_as = container_of(iommu, VTDAddressSpace, iommu);
     IntelIOMMUState *s = vtd_as->iommu_state;
@@ -2944,6 +2944,7 @@ static void vtd_iommu_notify_flag_changed(IOMMUMemoryRegion *iommu,
     } else if (new == IOMMU_NOTIFIER_NONE) {
         QLIST_REMOVE(vtd_as, next);
     }
+    return 0;
 }
 
 static int vtd_post_load(void *opaque, int version_id)
hw/ppc/spapr_iommu.c +4 -3
index e87b3d50f7..8d3b38f90b 100644
@@ -205,9 +205,9 @@ static int spapr_tce_get_attr(IOMMUMemoryRegion *iommu,
     return -EINVAL;
 }
 
-static void spapr_tce_notify_flag_changed(IOMMUMemoryRegion *iommu,
-                                          IOMMUNotifierFlag old,
-                                          IOMMUNotifierFlag new)
+static int spapr_tce_notify_flag_changed(IOMMUMemoryRegion *iommu,
+                                         IOMMUNotifierFlag old,
+                                         IOMMUNotifierFlag new)
 {
     struct SpaprTceTable *tbl = container_of(iommu, SpaprTceTable, iommu);
 
@@ -216,6 +216,7 @@ static void spapr_tce_notify_flag_changed(IOMMUMemoryRegion *iommu,
     } else if (old != IOMMU_NOTIFIER_NONE && new == IOMMU_NOTIFIER_NONE) {
         spapr_tce_set_need_vfio(tbl, false);
     }
+    return 0;
 }
 
 static int spapr_tce_table_post_load(void *opaque, int version_id)
include/exec/memory.h +13 -5
index a30245c25a..83257e3dce 100644
@@ -288,10 +288,14 @@ typedef struct IOMMUMemoryRegionClass {
      * @iommu: the IOMMUMemoryRegion
      * @old_flags: events which previously needed to be notified
      * @new_flags: events which now need to be notified
+     *
+     * Returns 0 on success, or a negative errno; in particular
+     * returns -EINVAL if the new flag bitmap is not supported by the
+     * IOMMU memory region.
      */
-    void (*notify_flag_changed)(IOMMUMemoryRegion *iommu,
-                                IOMMUNotifierFlag old_flags,
-                                IOMMUNotifierFlag new_flags);
+    int (*notify_flag_changed)(IOMMUMemoryRegion *iommu,
+                               IOMMUNotifierFlag old_flags,
+                               IOMMUNotifierFlag new_flags);
     /* Called to handle memory_region_iommu_replay().
      *
      * The default implementation of memory_region_iommu_replay() is to
@@ -1067,13 +1071,17 @@ void memory_region_notify_one(IOMMUNotifier *notifier,
  * memory_region_register_iommu_notifier: register a notifier for changes to
  * IOMMU translation entries.
  *
+ * Returns 0 on success, or a negative errno otherwise. In particular,
+ * -EINVAL indicates that at least one of the attributes of the notifier
+ * is not supported (flag/range) by the IOMMU memory region.
+ *
  * @mr: the memory region to observe
  * @n: the IOMMUNotifier to be added; the notify callback receives a
  *     pointer to an #IOMMUTLBEntry as the opaque value; the pointer
  *     ceases to be valid on exit from the notifier.
  */
-void memory_region_register_iommu_notifier(MemoryRegion *mr,
-                                           IOMMUNotifier *n);
+int memory_region_register_iommu_notifier(MemoryRegion *mr,
+                                          IOMMUNotifier *n);
 
 /**
  * memory_region_iommu_replay: replay existing IOMMU translations to
memory.c +18 -10
index b9dd6b94ca..2545d22ca9 100644
@@ -1837,33 +1837,37 @@ bool memory_region_is_logging(MemoryRegion *mr, uint8_t client)
     return memory_region_get_dirty_log_mask(mr) & (1 << client);
 }
 
-static void memory_region_update_iommu_notify_flags(IOMMUMemoryRegion *iommu_mr)
+static int memory_region_update_iommu_notify_flags(IOMMUMemoryRegion *iommu_mr)
 {
     IOMMUNotifierFlag flags = IOMMU_NOTIFIER_NONE;
     IOMMUNotifier *iommu_notifier;
     IOMMUMemoryRegionClass *imrc = IOMMU_MEMORY_REGION_GET_CLASS(iommu_mr);
+    int ret = 0;
 
     IOMMU_NOTIFIER_FOREACH(iommu_notifier, iommu_mr) {
         flags |= iommu_notifier->notifier_flags;
     }
 
     if (flags != iommu_mr->iommu_notify_flags && imrc->notify_flag_changed) {
-        imrc->notify_flag_changed(iommu_mr,
-                                  iommu_mr->iommu_notify_flags,
-                                  flags);
+        ret = imrc->notify_flag_changed(iommu_mr,
+                                        iommu_mr->iommu_notify_flags,
+                                        flags);
     }
 
-    iommu_mr->iommu_notify_flags = flags;
+    if (!ret) {
+        iommu_mr->iommu_notify_flags = flags;
+    }
+    return ret;
 }
 
-void memory_region_register_iommu_notifier(MemoryRegion *mr,
-                                           IOMMUNotifier *n)
+int memory_region_register_iommu_notifier(MemoryRegion *mr,
+                                          IOMMUNotifier *n)
 {
     IOMMUMemoryRegion *iommu_mr;
+    int ret;
 
     if (mr->alias) {
-        memory_region_register_iommu_notifier(mr->alias, n);
-        return;
+        return memory_region_register_iommu_notifier(mr->alias, n);
     }
 
     /* We need to register for at least one bitfield */
@@ -1874,7 +1878,11 @@ void memory_region_register_iommu_notifier(MemoryRegion *mr,
            n->iommu_idx < memory_region_iommu_num_indexes(iommu_mr));
 
     QLIST_INSERT_HEAD(&iommu_mr->iommu_notify, n, node);
-    memory_region_update_iommu_notify_flags(iommu_mr);
+    ret = memory_region_update_iommu_notify_flags(iommu_mr);
+    if (ret) {
+        QLIST_REMOVE(n, node);
+    }
+    return ret;
 }
 
 uint64_t memory_region_iommu_get_min_page_size(IOMMUMemoryRegion *iommu_mr)
Signature · 1 line
2.20.1
author
#2 lore raw

PATCH v2 2/5 vfio/common: Handle memory_region_register_iommu_notifier() failure

In reply to Eric Auger

Now memory_region_register_iommu_notifier() is allowed to fail, let's handle the returned value in vfio_listener_region_add().

This will allow to remove the error handling (exit) in the IOMMUs that implement a notify_flag_changed() that sometimes cannot accept the MAP flag.

---
 hw/vfio/common.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
hw/vfio/common.c +6 -2
index 3e03c495d8..d57d72cfb9 100644
@@ -630,9 +630,13 @@ static void vfio_listener_region_add(MemoryListener *listener,
                             section->offset_within_region,
                             int128_get64(llend),
                             iommu_idx);
-        QLIST_INSERT_HEAD(&container->giommu_list, giommu, giommu_next);
 
-        memory_region_register_iommu_notifier(section->mr, &giommu->n);
+        ret = memory_region_register_iommu_notifier(section->mr, &giommu->n);
+        if (ret) {
+            g_free(giommu);
+            goto fail;
+        }
+        QLIST_INSERT_HEAD(&container->giommu_list, giommu, giommu_next);
         memory_region_iommu_replay(giommu->iommu, &giommu->n);
 
         return;
Signature · 1 line
2.20.1
author
#4 lore raw

PATCH v2 4/5 vhost: assert on memory_region_register_iommu_notifier() failure

In reply to Eric Auger

memory_region_register_iommu_notifier now returns an error in case of failure. Assert in such a case.

---

v1 -> v2:
- assert(!ret)

---
 hw/virtio/vhost.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
hw/virtio/vhost.c +3 -2
index 34accdf615..975cb4c28c 100644
@@ -672,7 +672,7 @@ static void vhost_iommu_region_add(MemoryListener *listener,
                                          iommu_listener);
     struct vhost_iommu *iommu;
     Int128 end;
-    int iommu_idx;
+    int iommu_idx, ret;
     IOMMUMemoryRegion *iommu_mr;
 
     if (!memory_region_is_iommu(section->mr)) {
@@ -696,7 +696,8 @@ static void vhost_iommu_region_add(MemoryListener *listener,
     iommu->iommu_offset = section->offset_within_address_space -
                           section->offset_within_region;
     iommu->hdev = dev;
-    memory_region_register_iommu_notifier(section->mr, &iommu->n);
+    ret = memory_region_register_iommu_notifier(section->mr, &iommu->n);
+    assert(!ret);
     QLIST_INSERT_HEAD(&dev->iommu_list, iommu, iommu_next);
     /* TODO: can replay help performance here? */
 }
Signature · 1 line
2.20.1
author
#3 lore raw

PATCH v2 3/5 exec: assert on memory_region_register_iommu_notifier() failure

In reply to Eric Auger

memory_region_register_iommu_notifier now returns an error in case of failure. Assert in such a case.

---

v1 -> v2:
- add assert(!ret)

---
 exec.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
exec.c +4 -2
index 8b998974f8..5be8db6253 100644
@@ -663,7 +663,7 @@ static void tcg_register_iommu_notifier(CPUState *cpu,
      */
     MemoryRegion *mr = MEMORY_REGION(iommu_mr);
     TCGIOMMUNotifier *notifier;
-    int i;
+    int i, ret;
 
     for (i = 0; i < cpu->iommu_notifiers->len; i++) {
         notifier = g_array_index(cpu->iommu_notifiers, TCGIOMMUNotifier *, i);
@@ -692,7 +692,9 @@ static void tcg_register_iommu_notifier(CPUState *cpu,
                             0,
                             HWADDR_MAX,
                             iommu_idx);
-        memory_region_register_iommu_notifier(notifier->mr, &notifier->n);
+        ret = memory_region_register_iommu_notifier(notifier->mr,
+                                                    &notifier->n);
+        assert(!ret);
     }
 
     if (!notifier->active) {
Signature · 1 line
2.20.1
author
#5 lore raw

PATCH v2 5/5 amd_iommu: Let amdvi_iommu_notify_flag_changed() fail

In reply to Eric Auger

In case a MAP notifier is attempted to be registered, let's simply return an error. This latter now is handled in the VFIO code.

---
 hw/i386/amd_iommu.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
hw/i386/amd_iommu.c +1 -1
index 137ba367db..09dee48fac 100644
@@ -1476,7 +1476,7 @@ static int amdvi_iommu_notify_flag_changed(IOMMUMemoryRegion *iommu,
         error_report("device %02x.%02x.%x requires iommu notifier which is not "
                      "currently supported", as->bus_num, PCI_SLOT(as->devfn),
                      PCI_FUNC(as->devfn));
-        exit(1);
+        return -EINVAL;
     }
     return 0;
 }
Signature · 1 line
2.20.1
#6 lore raw
In reply to Eric Auger
Quoted 9 lines · Eric Auger

This series allows the memory_region_register_iommu_notifier() to fail. As of now, when a MAP notifier is attempted to be registered along with SMMUv3, Intel iommu without caching mode or AMD IOMMU, we exit in the IOMMU MR notify_flag_changed() callback. In case of VFIO assigned device hotplug, this could be handled more nicely directly within the VFIO code, simply rejecting the hotplug without exiting. This is what the series achieves by handling the memory_region_register_iommu_notifier() returned value.

Could I ask what's the code path for ARM when the hot plug failed? Is that vfio_realize() then vfio_connect_container() will fail with this?

    if (container->error) {
        ret = container->error;
        error_setg_errno(errp, -ret,
                         "memory listener initialization failed for container");
        goto listener_release_exit;
    }

If so, I would again suggest you to use Error** in patch 1. IMHO we can let vfio_listener_region_add() to be the first one to use the Error** so that instead of this:

    /*
     * On the initfn path, store the first error in the container so we
     * can gracefully fail.  Runtime, there's not much we can do other
     * than throw a hardware error.
     */
    if (!container->initialized) {
        if (!container->error) {
            container->error = ret;
        }
    } else {
        hw_error("vfio: DMA mapping failed, unable to continue");
    }

We can also cache the Error** into container and return to user if the user is using QMP which should be better than the int number (or again, maybe return both errors?). IIUC error_report() will not work for QMP.

Regards,

Signature · 1 line
Peter Xu
author
#7 lore raw

Hi Peter,

Quoted 14 lines · Peter Xu
Quoted 9 lines · Eric Auger

This series allows the memory_region_register_iommu_notifier() to fail. As of now, when a MAP notifier is attempted to be registered along with SMMUv3, Intel iommu without caching mode or AMD IOMMU, we exit in the IOMMU MR notify_flag_changed() callback. In case of VFIO assigned device hotplug, this could be handled more nicely directly within the VFIO code, simply rejecting the hotplug without exiting. This is what the series achieves by handling the memory_region_register_iommu_notifier() returned value.

Could I ask what's the code path for ARM when the hot plug failed? Is that vfio_realize() then vfio_connect_container() will fail with this?

Quoted 7 lines
    if (container->error) {
        ret = container->error;
        error_setg_errno(errp, -ret,
                         "memory listener initialization failed for container");
        goto listener_release_exit;
    }

Yes that path is exercised.

Quoted 4 lines

If so, I would again suggest you to use Error** in patch 1. IMHO we can let vfio_listener_region_add() to be the first one to use the Error** so that instead of this:

Quoted 12 lines
    /*
     * On the initfn path, store the first error in the container so we
     * can gracefully fail.  Runtime, there's not much we can do other
     * than throw a hardware error.
     */
    if (!container->initialized) {
        if (!container->error) {
            container->error = ret;
        }
    } else {
        hw_error("vfio: DMA mapping failed, unable to continue");
    }

OK. I agree that's better. That's also more code change ;-)

But let's do it!

Thank you for the review!

Best Regards

Eric

Quoted 8 lines

We can also cache the Error** into container and return to user if the user is using QMP which should be better than the int number (or again, maybe return both errors?). IIUC error_report() will not work for QMP.

Regards,