Skip to contentSkip to conversation

VIRTIO-IOMMU probe request support and MSI bypass on ARM

PATCH Superseded Eric Auger 3 participants 14 messages May 7, 2020 – May 8, 2020

v1 of 7 Superseded Latest is v7 →

lore
Share & tools
lore

Eric Auger

author
#0 lore raw

cover VIRTIO-IOMMU probe request support and MSI bypass on ARM

By default the virtio-iommu translates MSI transactions. This behavior is inherited from ARM SMMU. However the virt machine code knows where the MSI doorbells are so we can easily declare those regions as VIRTIO_IOMMU_RESV_MEM_T_MSI. With that setting the guest iommu subsystem will not need to map MSIs. This setup will simplify VFIO integration.

In this series, the ITS doorbell is declared as an MSI region to be bypassed by the VIRTIO-IOMMU.

This paves the way to the x86 integration where the MSI region [0xFEE00000,0xFEEFFFFF] will be exposed by q35 machine. However this is not part of this series.

Best Regards

Eric

This series can be found at:
https://github.com/eauger/qemu/tree/v5.0.0-virtio-iommu-msi-bypass-v1

History:
- Most of those patches were respinned from
[PATCH for-5.0 v11 00/20] VIRTIO-IOMMU device
except the last one which is new

Eric Auger (5):
qdev: Introduce DEFINE_PROP_RESERVED_REGION
virtio-iommu: Implement RESV_MEM probe request
virtio-iommu: Handle reserved regions in the translation process
virtio-iommu-pci: Add array of Interval properties hw/arm/virt: Let the virtio-iommu bypass MSIs

 include/exec/memory.h            |   6 ++
 include/hw/qdev-properties.h     |   3 +
 include/hw/virtio/virtio-iommu.h |   2 +
 include/qemu/typedefs.h          |   1 +
 hw/arm/virt.c                    |   5 ++
 hw/core/qdev-properties.c        |  89 +++++++++++++++++++++++++
 hw/virtio/virtio-iommu-pci.c     |   3 +
 hw/virtio/virtio-iommu.c         | 109 ++++++++++++++++++++++++++++++-
 hw/virtio/trace-events           |   1 +
 9 files changed, 217 insertions(+), 2 deletions(-)
Signature · 1 line
2.20.1
author
#1 lore raw

PATCH 1/5 qdev: Introduce DEFINE_PROP_RESERVED_REGION

Introduce a new property defining a reserved region:
<low address>, <high address>, <type>.

This will be used to encode reserved IOVA regions.

For instance, in virtio-iommu use case, reserved IOVA regions will be passed by the machine code to the virtio-iommu-pci device (an array of those). The type of the reserved region will match the virtio_iommu_probe_resv_mem subtype value:
- VIRTIO_IOMMU_RESV_MEM_T_RESERVED (0)
- VIRTIO_IOMMU_RESV_MEM_T_MSI (1)

on PC/Q35 machine, this will be used to inform the virtio-iommu-pci device it should bypass the MSI region. The reserved region will be: 0xfee00000, 0xfeefffff, 1.

On ARM, we can declare the ITS MSI doorbell as an MSI region to prevent MSIs from being mapped on guest side.

---

v11 -> v12:
- rename into DEFINE_PROP_RESERVED_REGION
- do not use g_strsplit anymore, use endptr instead
- remove 0x references

---
 include/exec/memory.h        |  6 +++
 include/hw/qdev-properties.h |  3 ++
 include/qemu/typedefs.h      |  1 +
 hw/core/qdev-properties.c    | 89 ++++++++++++++++++++++++++++++++++++
 4 files changed, 99 insertions(+)
include/exec/memory.h +6 -0
index e000bd2f97..7e47afabe8 100644
@@ -57,6 +57,12 @@ struct MemoryRegionMmio {
     CPUWriteMemoryFunc *write[3];
 };
 
+struct ReservedRegion {
+    hwaddr low;
+    hwaddr high;
+    unsigned int type;
+};
+
 typedef struct IOMMUTLBEntry IOMMUTLBEntry;
 
 /* See address_space_translate: bit 0 is read, bit 1 is write.  */
include/hw/qdev-properties.h +3 -0
index f161604fb6..03bf850a7e 100644
@@ -19,6 +19,7 @@ extern const PropertyInfo qdev_prop_string;
 extern const PropertyInfo qdev_prop_chr;
 extern const PropertyInfo qdev_prop_tpm;
 extern const PropertyInfo qdev_prop_macaddr;
+extern const PropertyInfo qdev_prop_reserved_region;
 extern const PropertyInfo qdev_prop_on_off_auto;
 extern const PropertyInfo qdev_prop_multifd_compression;
 extern const PropertyInfo qdev_prop_losttickpolicy;
@@ -183,6 +184,8 @@ extern const PropertyInfo qdev_prop_pcie_link_width;
     DEFINE_PROP(_n, _s, _f, qdev_prop_drive_iothread, BlockBackend *)
 #define DEFINE_PROP_MACADDR(_n, _s, _f)         \
     DEFINE_PROP(_n, _s, _f, qdev_prop_macaddr, MACAddr)
+#define DEFINE_PROP_RESERVED_REGION(_n, _s, _f)         \
+    DEFINE_PROP(_n, _s, _f, qdev_prop_reserved_region, ReservedRegion)
 #define DEFINE_PROP_ON_OFF_AUTO(_n, _s, _f, _d) \
     DEFINE_PROP_SIGNED(_n, _s, _f, _d, qdev_prop_on_off_auto, OnOffAuto)
 #define DEFINE_PROP_MULTIFD_COMPRESSION(_n, _s, _f, _d) \
include/qemu/typedefs.h +1 -0
index ecf3cde26c..85c4f891f4 100644
@@ -59,6 +59,7 @@ typedef struct ISABus ISABus;
 typedef struct ISADevice ISADevice;
 typedef struct IsaDma IsaDma;
 typedef struct MACAddr MACAddr;
+typedef struct ReservedRegion ReservedRegion;
 typedef struct MachineClass MachineClass;
 typedef struct MachineState MachineState;
 typedef struct MemoryListener MemoryListener;
hw/core/qdev-properties.c +89 -0

101 lines not on this page · raw message

Signature · 1 line
2.20.1
author
#2 lore raw

PATCH 2/5 virtio-iommu: Implement RESV_MEM probe request

In reply to Eric Auger

This patch implements the PROBE request. At the moment, only THE RESV_MEM property is handled. The first goal is to report iommu wide reserved regions such as the MSI regions set by the machine code. On x86 this will be the IOAPIC MSI region, [0xFEE00000 - 0xFEEFFFFF], on ARM this may be the ITS doorbell.

In the future we may introduce per device reserved regions. This will be useful when protecting host assigned devices which may expose their own reserved regions

---

Previous version was reviewed by Peter and Jean-Philippe. I simply removed the NONE property and memset the free space and reworded the commit message. I dared to keep the R-b.

---
 include/hw/virtio/virtio-iommu.h |  2 +
 hw/virtio/virtio-iommu.c         | 90 +++++++++++++++++++++++++++++++-
 hw/virtio/trace-events           |  1 +
 3 files changed, 91 insertions(+), 2 deletions(-)
include/hw/virtio/virtio-iommu.h +2 -0
index e653004d7c..49eb105cd8 100644
@@ -53,6 +53,8 @@ typedef struct VirtIOIOMMU {
     GHashTable *as_by_busptr;
     IOMMUPciBus *iommu_pcibus_by_bus_num[PCI_BUS_MAX];
     PCIBus *primary_bus;
+    ReservedRegion *reserved_regions;
+    uint32_t nb_reserved_regions;
     GTree *domains;
     QemuMutex mutex;
     GTree *endpoints;
hw/virtio/virtio-iommu.c +88 -2

135 lines not on this page · raw message

hw/virtio/trace-events +1 -0
index e83500bee9..5550475691 100644
@@ -73,3 +73,4 @@ virtio_iommu_get_domain(uint32_t domain_id) "Alloc domain=%d"
 virtio_iommu_put_domain(uint32_t domain_id) "Free domain=%d"
 virtio_iommu_translate_out(uint64_t virt_addr, uint64_t phys_addr, uint32_t sid) "0x%"PRIx64" -> 0x%"PRIx64 " for sid=%d"
 virtio_iommu_report_fault(uint8_t reason, uint32_t flags, uint32_t endpoint, uint64_t addr) "FAULT reason=%d flags=%d endpoint=%d address =0x%"PRIx64
+virtio_iommu_fill_resv_property(uint32_t devid, uint8_t subtype, uint64_t start, uint64_t end) "dev= %d, type=%d start=0x%"PRIx64" end=0x%"PRIx64
Signature · 1 line
2.20.1
#3 lore raw

Re: [PATCH 2/5] virtio-iommu: Implement RESV_MEM probe request

Hi, Eric,

On Thu, May 07, 2020 at 04:31:58PM +0200, Eric Auger wrote:

[...]

Quoted 22 lines
hw/virtio/virtio-iommu.c +15 -0
@@ -452,17 +520,33 @@ static void virtio_iommu_handle_command(VirtIODevice *vdev, VirtQueue *vq)
         case VIRTIO_IOMMU_T_UNMAP:
             tail.status = virtio_iommu_handle_unmap(s, iov, iov_cnt);
             break;
+        case VIRTIO_IOMMU_T_PROBE:
+        {
+            struct virtio_iommu_req_tail *ptail;
+            uint8_t *buf = g_malloc0(s->config.probe_size + sizeof(tail));
+
+            ptail = (struct virtio_iommu_req_tail *)
+                        (buf + s->config.probe_size);
+            ptail->status = virtio_iommu_handle_probe(s, iov, iov_cnt, buf);
+
+            sz = iov_from_buf(elem->in_sg, elem->in_num, 0,
+                              buf, s->config.probe_size + sizeof(tail));
+            g_free(buf);
+            assert(sz == s->config.probe_size + sizeof(tail));
+            goto push;
+        }
         default:
             tail.status = VIRTIO_IOMMU_S_UNSUPP;
         }

hw/virtio/virtio-iommu.c static void virtio_iommu_handle_command(VirtIODevice *vdev, VirtQueue *vq) · line 548

Quoted 10 lines
hw/virtio/virtio-iommu.c +2 -2
-        qemu_mutex_unlock(&s->mutex);
 
 out:
         sz = iov_from_buf(elem->in_sg, elem->in_num, 0,
                           &tail, sizeof(tail));
         assert(sz == sizeof(tail));
 
-        virtqueue_push(vq, elem, sizeof(tail));
+push:
+        qemu_mutex_unlock(&s->mutex);

I think we can't move this unlock to here because otherwise "goto out" could potentially try to unlock it without locked first. Thanks,

Quoted 4 lines
hw/virtio/virtio-iommu.c +1 -0
+        virtqueue_push(vq, elem, sz);
         virtio_notify(vdev, vq);
         g_free(elem);
     }
Signature · 1 line
Peter Xu
author
#4 lore raw

Re: [PATCH 2/5] virtio-iommu: Implement RESV_MEM probe request

Hi Peter,

Quoted 35 lines · Peter Xu

Hi, Eric,

On Thu, May 07, 2020 at 04:31:58PM +0200, Eric Auger wrote:

[...]

Quoted 29 lines
hw/virtio/virtio-iommu.c +15 -1
@@ -452,17 +520,33 @@ static void virtio_iommu_handle_command(VirtIODevice *vdev, VirtQueue *vq)
         case VIRTIO_IOMMU_T_UNMAP:
             tail.status = virtio_iommu_handle_unmap(s, iov, iov_cnt);
             break;
+        case VIRTIO_IOMMU_T_PROBE:
+        {
+            struct virtio_iommu_req_tail *ptail;
+            uint8_t *buf = g_malloc0(s->config.probe_size + sizeof(tail));
+
+            ptail = (struct virtio_iommu_req_tail *)
+                        (buf + s->config.probe_size);
+            ptail->status = virtio_iommu_handle_probe(s, iov, iov_cnt, buf);
+
+            sz = iov_from_buf(elem->in_sg, elem->in_num, 0,
+                              buf, s->config.probe_size + sizeof(tail));
+            g_free(buf);
+            assert(sz == s->config.probe_size + sizeof(tail));
+            goto push;
+        }
         default:
             tail.status = VIRTIO_IOMMU_S_UNSUPP;
         }
-        qemu_mutex_unlock(&s->mutex);
 
 out:
         sz = iov_from_buf(elem->in_sg, elem->in_num, 0,
                           &tail, sizeof(tail));
         assert(sz == sizeof(tail));
 
Quoted 6 lines

hw/virtio/virtio-iommu.c static void virtio_iommu_handle_command(VirtIODevice *vdev, VirtQueue *vq) · line 548

Quoted 3 lines
hw/virtio/virtio-iommu.c +2 -1
-        virtqueue_push(vq, elem, sizeof(tail));
+push:
+        qemu_mutex_unlock(&s->mutex);

I think we can't move this unlock to here because otherwise "goto out" could potentially try to unlock it without locked first. Thanks,

You're right. I will revisit that.

Thanks!

Eric

Quoted 6 lines
Quoted 5 lines
hw/virtio/virtio-iommu.c +1 -0
+        virtqueue_push(vq, elem, sz);
         virtio_notify(vdev, vq);
         g_free(elem);
     }
#5 lore raw

Re: [PATCH 2/5] virtio-iommu: Implement RESV_MEM probe request

In reply to Eric Auger
Quoted 5 lines · Eric Auger
hw/virtio/virtio-iommu.c +5 -0
+static ssize_t virtio_iommu_fill_resv_mem_prop(VirtIOIOMMU *s, uint32_t ep,
+                                               uint8_t *buf, size_t free)
+{
+    struct virtio_iommu_probe_resv_mem prop = {};
+    size_t size = sizeof(prop), length = size - sizeof(prop.head), total;

hw/virtio/virtio-iommu.c static int virtio_iommu_unmap(VirtIOIOMMU *s, · line 396

Quoted 10 lines
hw/virtio/virtio-iommu.c +10 -0
+    int i;
+
+    total = size * s->nb_reserved_regions;
+
+    if (total > free) {
+        return -ENOSPC;
+    }
+
+    for (i = 0; i < s->nb_reserved_regions; i++) {
+        prop.head.type = VIRTIO_IOMMU_PROBE_T_RESV_MEM;

cpu_to_le16

Quoted 1 line

+ prop.head.length = cpu_to_le64(length);

same here

Quoted 1 line

+ prop.subtype = cpu_to_le64(s->reserved_regions[i].type);

and subtype is a byte

Quoted 12 lines
hw/virtio/virtio-iommu.c +12 -0
+        prop.start = cpu_to_le64(s->reserved_regions[i].low);
+        prop.end = cpu_to_le64(s->reserved_regions[i].high);
+
+        memcpy(buf, &prop, size);
+
+        trace_virtio_iommu_fill_resv_property(ep, prop.subtype,
+                                              prop.start, prop.end);
+        buf += size;
+    }
+    return total;
+}
+

hw/virtio/virtio-iommu.c static int virtio_iommu_unmap(VirtIOIOMMU *s, · line 420

Quoted 10 lines
hw/virtio/virtio-iommu.c +10 -0
+/**
+ * virtio_iommu_probe - Fill the probe request buffer with
+ * the properties the device is able to return and add a NONE
+ * property at the end.
+ */
+static int virtio_iommu_probe(VirtIOIOMMU *s,
+                              struct virtio_iommu_req_probe *req,
+                              uint8_t *buf)
+{
+    uint32_t ep_id = le32_to_cpu(req->endpoint);

I think we should check that the endpoint ID is sane even if we're not using it at the moment (it is a SHOULD in the spec, and the page size mask patch will need the ep anyway).

hw/virtio/virtio-iommu.c static int virtio_iommu_unmap(VirtIOIOMMU *s, · line 422

Quoted 2 lines
hw/virtio/virtio-iommu.c +2 -0
+    struct virtio_iommu_probe_property last = {};
+    size_t free = VIOMMU_PROBE_SIZE - sizeof(last);

last isn't needed, you can fill the whole probe buffer with valid properties

hw/virtio/virtio-iommu.c static int virtio_iommu_unmap(VirtIOIOMMU *s, · line 427

Quoted 5 lines
hw/virtio/virtio-iommu.c +5 -0
+    ssize_t count;
+
+    count = virtio_iommu_fill_resv_mem_prop(s, ep_id, buf, free);
+    if (count < 0) {
+            return VIRTIO_IOMMU_S_INVAL;

indentation?

Thanks,
Jean

Quoted 95 lines
hw/virtio/virtio-iommu.c +41 -2
+    }
+    buf += count;
+    free -= count;
+
+    /* Fill the rest with zeroes */
+    memset(buf, 0, free);
+
+    return VIRTIO_IOMMU_S_OK;
+}
+
 static int virtio_iommu_iov_to_req(struct iovec *iov,
                                    unsigned int iov_cnt,
                                    void *req, size_t req_sz)
@@ -407,6 +464,17 @@ virtio_iommu_handle_req(detach)
 virtio_iommu_handle_req(map)
 virtio_iommu_handle_req(unmap)
 
+static int virtio_iommu_handle_probe(VirtIOIOMMU *s,
+                                     struct iovec *iov,
+                                     unsigned int iov_cnt,
+                                     uint8_t *buf)
+{
+    struct virtio_iommu_req_probe req;
+    int ret = virtio_iommu_iov_to_req(iov, iov_cnt, &req, sizeof(req));
+
+    return ret ? ret : virtio_iommu_probe(s, &req, buf);
+}
+
 static void virtio_iommu_handle_command(VirtIODevice *vdev, VirtQueue *vq)
 {
     VirtIOIOMMU *s = VIRTIO_IOMMU(vdev);
@@ -452,17 +520,33 @@ static void virtio_iommu_handle_command(VirtIODevice *vdev, VirtQueue *vq)
         case VIRTIO_IOMMU_T_UNMAP:
             tail.status = virtio_iommu_handle_unmap(s, iov, iov_cnt);
             break;
+        case VIRTIO_IOMMU_T_PROBE:
+        {
+            struct virtio_iommu_req_tail *ptail;
+            uint8_t *buf = g_malloc0(s->config.probe_size + sizeof(tail));
+
+            ptail = (struct virtio_iommu_req_tail *)
+                        (buf + s->config.probe_size);
+            ptail->status = virtio_iommu_handle_probe(s, iov, iov_cnt, buf);
+
+            sz = iov_from_buf(elem->in_sg, elem->in_num, 0,
+                              buf, s->config.probe_size + sizeof(tail));
+            g_free(buf);
+            assert(sz == s->config.probe_size + sizeof(tail));
+            goto push;
+        }
         default:
             tail.status = VIRTIO_IOMMU_S_UNSUPP;
         }
-        qemu_mutex_unlock(&s->mutex);
 
 out:
         sz = iov_from_buf(elem->in_sg, elem->in_num, 0,
                           &tail, sizeof(tail));
         assert(sz == sizeof(tail));
 
-        virtqueue_push(vq, elem, sizeof(tail));
+push:
+        qemu_mutex_unlock(&s->mutex);
+        virtqueue_push(vq, elem, sz);
         virtio_notify(vdev, vq);
         g_free(elem);
     }
@@ -667,6 +751,7 @@ static void virtio_iommu_device_realize(DeviceState *dev, Error **errp)
     s->config.page_size_mask = TARGET_PAGE_MASK;
     s->config.input_range.end = -1UL;
     s->config.domain_range.end = 32;
+    s->config.probe_size = VIOMMU_PROBE_SIZE;
 
     virtio_add_feature(&s->features, VIRTIO_RING_F_EVENT_IDX);
     virtio_add_feature(&s->features, VIRTIO_RING_F_INDIRECT_DESC);
@@ -676,6 +761,7 @@ static void virtio_iommu_device_realize(DeviceState *dev, Error **errp)
     virtio_add_feature(&s->features, VIRTIO_IOMMU_F_MAP_UNMAP);
     virtio_add_feature(&s->features, VIRTIO_IOMMU_F_BYPASS);
     virtio_add_feature(&s->features, VIRTIO_IOMMU_F_MMIO);
+    virtio_add_feature(&s->features, VIRTIO_IOMMU_F_PROBE);
 
     qemu_mutex_init(&s->mutex);
 
hw/virtio/trace-events +1 -0
index e83500bee9..5550475691 100644
@@ -73,3 +73,4 @@ virtio_iommu_get_domain(uint32_t domain_id) "Alloc domain=%d"
 virtio_iommu_put_domain(uint32_t domain_id) "Free domain=%d"
 virtio_iommu_translate_out(uint64_t virt_addr, uint64_t phys_addr, uint32_t sid) "0x%"PRIx64" -> 0x%"PRIx64 " for sid=%d"
 virtio_iommu_report_fault(uint8_t reason, uint32_t flags, uint32_t endpoint, uint64_t addr) "FAULT reason=%d flags=%d endpoint=%d address =0x%"PRIx64
+virtio_iommu_fill_resv_property(uint32_t devid, uint8_t subtype, uint64_t start, uint64_t end) "dev= %d, type=%d start=0x%"PRIx64" end=0x%"PRIx64
Signature · 1 line
2.20.1
author
#6 lore raw

Re: [PATCH 2/5] virtio-iommu: Implement RESV_MEM probe request

Hi Jean,

Quoted 19 lines · Jean-Philippe Brucker
Quoted 5 lines · Eric Auger
hw/virtio/virtio-iommu.c +5 -0
+static ssize_t virtio_iommu_fill_resv_mem_prop(VirtIOIOMMU *s, uint32_t ep,
+                                               uint8_t *buf, size_t free)
+{
+    struct virtio_iommu_probe_resv_mem prop = {};
+    size_t size = sizeof(prop), length = size - sizeof(prop.head), total;

hw/virtio/virtio-iommu.c static int virtio_iommu_unmap(VirtIOIOMMU *s, · line 396

Quoted 10 lines
hw/virtio/virtio-iommu.c +10 -0
+    int i;
+
+    total = size * s->nb_reserved_regions;
+
+    if (total > free) {
+        return -ENOSPC;
+    }
+
+    for (i = 0; i < s->nb_reserved_regions; i++) {
+        prop.head.type = VIRTIO_IOMMU_PROBE_T_RESV_MEM;

cpu_to_le16

Quoted 7 lines
Quoted 1 line

+ prop.head.length = cpu_to_le64(length);

same here

Quoted 1 line

+ prop.subtype = cpu_to_le64(s->reserved_regions[i].type);

and subtype is a byte

All fixed. Thanks for spotting this.

Quoted 19 lines
Quoted 19 lines
hw/virtio/virtio-iommu.c +18 -0
+        prop.start = cpu_to_le64(s->reserved_regions[i].low);
+        prop.end = cpu_to_le64(s->reserved_regions[i].high);
+
+        memcpy(buf, &prop, size);
+
+        trace_virtio_iommu_fill_resv_property(ep, prop.subtype,
+                                              prop.start, prop.end);
+        buf += size;
+    }
+    return total;
+}
+
+/**
+ * virtio_iommu_probe - Fill the probe request buffer with
+ * the properties the device is able to return and add a NONE
+ * property at the end.
+ */
+static int virtio_iommu_probe(VirtIOIOMMU *s,
Quoted 8 lines

hw/virtio/virtio-iommu.c static int virtio_iommu_unmap(VirtIOIOMMU *s, · line 420

Quoted 4 lines
hw/virtio/virtio-iommu.c +4 -0
+                              struct virtio_iommu_req_probe *req,
+                              uint8_t *buf)
+{
+    uint32_t ep_id = le32_to_cpu(req->endpoint);

I think we should check that the endpoint ID is sane even if we're not using it at the moment (it is a SHOULD in the spec, and the page size mask patch will need the ep anyway).

OK

Quoted 6 lines

hw/virtio/virtio-iommu.c static int virtio_iommu_unmap(VirtIOIOMMU *s, · line 422

Quoted 3 lines
hw/virtio/virtio-iommu.c +2 -0
+    struct virtio_iommu_probe_property last = {};
+    size_t free = VIOMMU_PROBE_SIZE - sizeof(last);

last isn't needed, you can fill the whole probe buffer with valid properties

yep. This is a reminder from none prop.

Quoted 8 lines

hw/virtio/virtio-iommu.c static int virtio_iommu_unmap(VirtIOIOMMU *s, · line 427

Quoted 6 lines
hw/virtio/virtio-iommu.c +5 -0
+    ssize_t count;
+
+    count = virtio_iommu_fill_resv_mem_prop(s, ep_id, buf, free);
+    if (count < 0) {
+            return VIRTIO_IOMMU_S_INVAL;

indentation?

yes.

Thank you for the review

Eric

Quoted 100 lines

Thanks,
Jean

Quoted 95 lines
hw/virtio/virtio-iommu.c +41 -2
+    }
+    buf += count;
+    free -= count;
+
+    /* Fill the rest with zeroes */
+    memset(buf, 0, free);
+
+    return VIRTIO_IOMMU_S_OK;
+}
+
 static int virtio_iommu_iov_to_req(struct iovec *iov,
                                    unsigned int iov_cnt,
                                    void *req, size_t req_sz)
@@ -407,6 +464,17 @@ virtio_iommu_handle_req(detach)
 virtio_iommu_handle_req(map)
 virtio_iommu_handle_req(unmap)
 
+static int virtio_iommu_handle_probe(VirtIOIOMMU *s,
+                                     struct iovec *iov,
+                                     unsigned int iov_cnt,
+                                     uint8_t *buf)
+{
+    struct virtio_iommu_req_probe req;
+    int ret = virtio_iommu_iov_to_req(iov, iov_cnt, &req, sizeof(req));
+
+    return ret ? ret : virtio_iommu_probe(s, &req, buf);
+}
+
 static void virtio_iommu_handle_command(VirtIODevice *vdev, VirtQueue *vq)
 {
     VirtIOIOMMU *s = VIRTIO_IOMMU(vdev);
@@ -452,17 +520,33 @@ static void virtio_iommu_handle_command(VirtIODevice *vdev, VirtQueue *vq)
         case VIRTIO_IOMMU_T_UNMAP:
             tail.status = virtio_iommu_handle_unmap(s, iov, iov_cnt);
             break;
+        case VIRTIO_IOMMU_T_PROBE:
+        {
+            struct virtio_iommu_req_tail *ptail;
+            uint8_t *buf = g_malloc0(s->config.probe_size + sizeof(tail));
+
+            ptail = (struct virtio_iommu_req_tail *)
+                        (buf + s->config.probe_size);
+            ptail->status = virtio_iommu_handle_probe(s, iov, iov_cnt, buf);
+
+            sz = iov_from_buf(elem->in_sg, elem->in_num, 0,
+                              buf, s->config.probe_size + sizeof(tail));
+            g_free(buf);
+            assert(sz == s->config.probe_size + sizeof(tail));
+            goto push;
+        }
         default:
             tail.status = VIRTIO_IOMMU_S_UNSUPP;
         }
-        qemu_mutex_unlock(&s->mutex);
 
 out:
         sz = iov_from_buf(elem->in_sg, elem->in_num, 0,
                           &tail, sizeof(tail));
         assert(sz == sizeof(tail));
 
-        virtqueue_push(vq, elem, sizeof(tail));
+push:
+        qemu_mutex_unlock(&s->mutex);
+        virtqueue_push(vq, elem, sz);
         virtio_notify(vdev, vq);
         g_free(elem);
     }
@@ -667,6 +751,7 @@ static void virtio_iommu_device_realize(DeviceState *dev, Error **errp)
     s->config.page_size_mask = TARGET_PAGE_MASK;
     s->config.input_range.end = -1UL;
     s->config.domain_range.end = 32;
+    s->config.probe_size = VIOMMU_PROBE_SIZE;
 
     virtio_add_feature(&s->features, VIRTIO_RING_F_EVENT_IDX);
     virtio_add_feature(&s->features, VIRTIO_RING_F_INDIRECT_DESC);
@@ -676,6 +761,7 @@ static void virtio_iommu_device_realize(DeviceState *dev, Error **errp)
     virtio_add_feature(&s->features, VIRTIO_IOMMU_F_MAP_UNMAP);
     virtio_add_feature(&s->features, VIRTIO_IOMMU_F_BYPASS);
     virtio_add_feature(&s->features, VIRTIO_IOMMU_F_MMIO);
+    virtio_add_feature(&s->features, VIRTIO_IOMMU_F_PROBE);
 
     qemu_mutex_init(&s->mutex);
 
hw/virtio/trace-events +1 -0
index e83500bee9..5550475691 100644
@@ -73,3 +73,4 @@ virtio_iommu_get_domain(uint32_t domain_id) "Alloc domain=%d"
 virtio_iommu_put_domain(uint32_t domain_id) "Free domain=%d"
 virtio_iommu_translate_out(uint64_t virt_addr, uint64_t phys_addr, uint32_t sid) "0x%"PRIx64" -> 0x%"PRIx64 " for sid=%d"
 virtio_iommu_report_fault(uint8_t reason, uint32_t flags, uint32_t endpoint, uint64_t addr) "FAULT reason=%d flags=%d endpoint=%d address =0x%"PRIx64
+virtio_iommu_fill_resv_property(uint32_t devid, uint8_t subtype, uint64_t start, uint64_t end) "dev= %d, type=%d start=0x%"PRIx64" end=0x%"PRIx64
Signature · 1 line
2.20.1
author
#7 lore raw

PATCH 3/5 virtio-iommu: Handle reserved regions in the translation process

In reply to Eric Auger

When translating an address we need to check if it belongs to a reserved virtual address range. If it does, there are 2 cases:

- it belongs to a RESERVED region: the guest should neither use this address in a MAP not instruct the end-point to DMA on them. We report an error

- It belongs to an MSI region: we bypass the translation.

---

v11 -> v12:
- s/Interval/ReservedRegion
- break instead of goto unlock as peter suggested
- added VIRTIO_IOMMU_FAULT_F_ADDRESS flag (jean)

v10 -> v11:
- directly use the reserved_regions properties array

v9 -> v10:
- in case of MSI region, we immediatly return

---
 hw/virtio/virtio-iommu.c | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)
hw/virtio/virtio-iommu.c +19 -0
index 95059eff70..d35fc9522b 100644
@@ -607,6 +607,7 @@ static IOMMUTLBEntry virtio_iommu_translate(IOMMUMemoryRegion *mr, hwaddr addr,
     uint32_t sid, flags;
     bool bypass_allowed;
     bool found;
+    int i;
 
     interval.low = addr;
     interval.high = addr + 1;
@@ -640,6 +641,24 @@ static IOMMUTLBEntry virtio_iommu_translate(IOMMUMemoryRegion *mr, hwaddr addr,
         goto unlock;
     }
 
+    for (i = 0; i < s->nb_reserved_regions; i++) {
+        if (interval.low >= s->reserved_regions[i].low &&
+            interval.low <= s->reserved_regions[i].high) {
+            switch (s->reserved_regions[i].type) {
+            case VIRTIO_IOMMU_RESV_MEM_T_MSI:
+                entry.perm = flag;
+                break;
+            case VIRTIO_IOMMU_RESV_MEM_T_RESERVED:
+            default:
+                virtio_iommu_report_fault(s, VIRTIO_IOMMU_FAULT_R_MAPPING,
+                                          VIRTIO_IOMMU_FAULT_F_ADDRESS,
+                                          sid, addr);
+                break;
+            }
+            goto unlock;
+        }
+    }
+
     if (!ep->domain) {
         if (!bypass_allowed) {
             error_report_once("%s %02x:%02x.%01x not attached to any domain",
Signature · 1 line
2.20.1
#8 lore raw

Re: [PATCH 3/5] virtio-iommu: Handle reserved regions in the translation process

Hi, Eric,

hw/virtio/virtio-iommu.c static IOMMUTLBEntry virtio_iommu_translate(IOMMUMemoryRegion *mr, hwaddr addr, · line 646

Quoted 7 lines · Eric Auger
hw/virtio/virtio-iommu.c +3 -0
@@ -640,6 +641,24 @@ static IOMMUTLBEntry virtio_iommu_translate(IOMMUMemoryRegion *mr, hwaddr addr,
         goto unlock;
     }
 
+    for (i = 0; i < s->nb_reserved_regions; i++) {
+        if (interval.low >= s->reserved_regions[i].low &&
+            interval.low <= s->reserved_regions[i].high) {

Should this be s/low/high/?

For this case (high==low+1) I guess maybe it's also easier to use "addr >= low && addr < high".

Thanks,

Quoted 21 lines
hw/virtio/virtio-iommu.c +15 -0
+            switch (s->reserved_regions[i].type) {
+            case VIRTIO_IOMMU_RESV_MEM_T_MSI:
+                entry.perm = flag;
+                break;
+            case VIRTIO_IOMMU_RESV_MEM_T_RESERVED:
+            default:
+                virtio_iommu_report_fault(s, VIRTIO_IOMMU_FAULT_R_MAPPING,
+                                          VIRTIO_IOMMU_FAULT_F_ADDRESS,
+                                          sid, addr);
+                break;
+            }
+            goto unlock;
+        }
+    }
+
     if (!ep->domain) {
         if (!bypass_allowed) {
             error_report_once("%s %02x:%02x.%01x not attached to any domain",
Signature · 1 line
2.20.1
Signature · 1 line
Peter Xu
author
#9 lore raw

Re: [PATCH 3/5] virtio-iommu: Handle reserved regions in the translation process

Hi Peter,

Quoted 7 lines · Peter Xu

Hi, Eric,

Quoted 4 lines · Eric Auger
hw/virtio/virtio-iommu.c +0 -0
@@ -640,6 +641,24 @@ static IOMMUTLBEntry virtio_iommu_translate(IOMMUMemoryRegion *mr, hwaddr addr,
         goto unlock;
     }
 
Quoted 5 lines

hw/virtio/virtio-iommu.c static IOMMUTLBEntry virtio_iommu_translate(IOMMUMemoryRegion *mr, hwaddr addr, · line 646

Quoted 3 lines
hw/virtio/virtio-iommu.c +3 -0
+    for (i = 0; i < s->nb_reserved_regions; i++) {
+        if (interval.low >= s->reserved_regions[i].low &&
+            interval.low <= s->reserved_regions[i].high) {

Should this be s/low/high/?

I meant addr >= s->reserved_regions[i].low && addr <= s->reserved_regions[i].high

We only compare a single iova against reserved range boundaries and boundaries are inclusive

Quoted 3 lines

For this case (high==low+1) I guess maybe it's also easier to use "addr >= low && addr < high".

Yes using addr directly is definitively more readable ;-)

Thanks

Eric

Quoted 25 lines

Thanks,

Quoted 21 lines
hw/virtio/virtio-iommu.c +15 -0
+            switch (s->reserved_regions[i].type) {
+            case VIRTIO_IOMMU_RESV_MEM_T_MSI:
+                entry.perm = flag;
+                break;
+            case VIRTIO_IOMMU_RESV_MEM_T_RESERVED:
+            default:
+                virtio_iommu_report_fault(s, VIRTIO_IOMMU_FAULT_R_MAPPING,
+                                          VIRTIO_IOMMU_FAULT_F_ADDRESS,
+                                          sid, addr);
+                break;
+            }
+            goto unlock;
+        }
+    }
+
     if (!ep->domain) {
         if (!bypass_allowed) {
             error_report_once("%s %02x:%02x.%01x not attached to any domain",
Signature · 1 line
2.20.1
author
#10 lore raw

PATCH 4/5 virtio-iommu-pci: Add array of Interval properties

In reply to Eric Auger

The machine may need to pass reserved regions to the virtio-iommu-pci device (such as the MSI window on x86). So let's add an array of Interval properties.

---

v12 -> v12:
- added Jean's R-b

---
 hw/virtio/virtio-iommu-pci.c | 3 +++
 1 file changed, 3 insertions(+)
hw/virtio/virtio-iommu-pci.c +3 -0
index 3dfbf55b47..44ae9ebc11 100644
@@ -33,6 +33,9 @@ struct VirtIOIOMMUPCI {
 
 static Property virtio_iommu_pci_properties[] = {
     DEFINE_PROP_UINT32("class", VirtIOPCIProxy, class_code, 0),
+    DEFINE_PROP_ARRAY("reserved-regions", VirtIOIOMMUPCI,
+                      vdev.nb_reserved_regions, vdev.reserved_regions,
+                      qdev_prop_reserved_region, ReservedRegion),
     DEFINE_PROP_END_OF_LIST(),
 };
 
Signature · 1 line
2.20.1
author
#11 lore raw

PATCH 5/5 hw/arm/virt: Let the virtio-iommu bypass MSIs

In reply to Eric Auger

At the moment the virtio-iommu translates MSI transactions. This behavior is inherited from ARM SMMU. The virt machine code knows where the guest MSI doorbells are so we can easily declare those regions as VIRTIO_IOMMU_RESV_MEM_T_MSI. With that setting the guest will not map MSIs through the IOMMU and those transactions will be simply bypassed. The ITS does MSI translation anyway.

---
 hw/arm/virt.c | 5 +++++
 1 file changed, 5 insertions(+)
hw/arm/virt.c +5 -0
index 171e6908ec..6a041e97a5 100644
@@ -2138,6 +2138,11 @@ static void virt_machine_device_pre_plug_cb(HotplugHandler *hotplug_dev,
 {
     if (object_dynamic_cast(OBJECT(dev), TYPE_PC_DIMM)) {
         virt_memory_pre_plug(hotplug_dev, dev, errp);
+    } else if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_IOMMU_PCI)) {
+        /* we declare a VIRTIO_IOMMU_RESV_MEM_T_MSI region */
+        qdev_prop_set_uint32(dev, "len-reserved-regions", 1);
+        qdev_prop_set_string(dev, "reserved-regions[0]",
+                             "0x8090000, 0x809FFFF, 1");
     }
 }
 
Signature · 1 line
2.20.1
#12 lore raw

Re: [PATCH 5/5] hw/arm/virt: Let the virtio-iommu bypass MSIs

Quoted 7 lines · Eric Auger

At the moment the virtio-iommu translates MSI transactions. This behavior is inherited from ARM SMMU. The virt machine code knows where the guest MSI doorbells are so we can easily declare those regions as VIRTIO_IOMMU_RESV_MEM_T_MSI. With that setting the guest will not map MSIs through the IOMMU and those transactions will be simply bypassed. The ITS does MSI translation anyway.

Makes sense. I think we have to check that the ITS is used before declaring the resv region, though. When using gicv2m, the guest will see this MSI region and deduce that it doesn't need to create a SW mapping for MSIs (might be wrong, I haven't tested it yet).

Quoted 10 lines
---
 hw/arm/virt.c | 5 +++++
 1 file changed, 5 insertions(+)
hw/arm/virt.c +0 -0
index 171e6908ec..6a041e97a5 100644

hw/arm/virt.c static void virt_machine_device_pre_plug_cb(HotplugHandler *hotplug_dev, · line 2145

Quoted 8 lines
hw/arm/virt.c +5 -0
@@ -2138,6 +2138,11 @@ static void virt_machine_device_pre_plug_cb(HotplugHandler *hotplug_dev,
 {
     if (object_dynamic_cast(OBJECT(dev), TYPE_PC_DIMM)) {
         virt_memory_pre_plug(hotplug_dev, dev, errp);
+    } else if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_IOMMU_PCI)) {
+        /* we declare a VIRTIO_IOMMU_RESV_MEM_T_MSI region */
+        qdev_prop_set_uint32(dev, "len-reserved-regions", 1);
+        qdev_prop_set_string(dev, "reserved-regions[0]",
+                             "0x8090000, 0x809FFFF, 1");

Maybe worth adding a comment saying this is the GITS_TRANSLATER page?

Thanks,
Jean

author
#13 lore raw

Re: [PATCH 5/5] hw/arm/virt: Let the virtio-iommu bypass MSIs

Hi Jean-Philippe,

Quoted 11 lines · Jean-Philippe Brucker
Quoted 7 lines · Eric Auger

At the moment the virtio-iommu translates MSI transactions. This behavior is inherited from ARM SMMU. The virt machine code knows where the guest MSI doorbells are so we can easily declare those regions as VIRTIO_IOMMU_RESV_MEM_T_MSI. With that setting the guest will not map MSIs through the IOMMU and those transactions will be simply bypassed. The ITS does MSI translation anyway.

Makes sense. I think we have to check that the ITS is used before declaring the resv region, though.

When using gicv2m, the guest will see

Quoted 2 lines

this MSI region and deduce that it doesn't need to create a SW mapping for MSIs (might be wrong, I haven't tested it yet).

Yes you're right. I think any MSI region hides the SW MSI one. So I will test the GIC type beforehand.

Also I will extend the series to handle the GICv2M backdoor

To me the problem is similar and the iommu subsystem will map the GICV2M MSI doorbell as well. From a security pov, there is no difference inbetween the 2 solutions. Anyway the doorbell is reachable by any assigned device, would it be mapped or not. And given the GICv2M does not perform any interrupt translation, an assigned device can trigger MSIs on another userspace driver.

Thanks

Eric

Quoted 25 lines
Quoted 11 lines
---
 hw/arm/virt.c | 5 +++++
 1 file changed, 5 insertions(+)
hw/arm/virt.c +0 -0
index 171e6908ec..6a041e97a5 100644

hw/arm/virt.c static void virt_machine_device_pre_plug_cb(HotplugHandler *hotplug_dev, · line 2145

Quoted 8 lines
hw/arm/virt.c +5 -0
@@ -2138,6 +2138,11 @@ static void virt_machine_device_pre_plug_cb(HotplugHandler *hotplug_dev,
 {
     if (object_dynamic_cast(OBJECT(dev), TYPE_PC_DIMM)) {
         virt_memory_pre_plug(hotplug_dev, dev, errp);
+    } else if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_IOMMU_PCI)) {
+        /* we declare a VIRTIO_IOMMU_RESV_MEM_T_MSI region */
+        qdev_prop_set_uint32(dev, "len-reserved-regions", 1);
+        qdev_prop_set_string(dev, "reserved-regions[0]",
+                             "0x8090000, 0x809FFFF, 1");

Maybe worth adding a comment saying this is the GITS_TRANSLATER page?

Thanks,
Jean