VIRTIO-IOMMU probe request support and MSI bypass on ARM
v7 cover VIRTIO-IOMMU probe request support and MSI bypass on ARM
By default the virtio-iommu translates MSI transactions. This behavior is inherited from ARM SMMU. However the virt machine code knows where the MSI doorbells are, so we can easily declare those regions as VIRTIO_IOMMU_RESV_MEM_T_MSI. With that setting the guest iommu subsystem will not need to map MSIs. This setup will simplify the VFIO integration.
In this series, the ITS or GICV2M doorbells are declared as HW MSI regions to be bypassed by the VIRTIO-IOMMU.
This also paves the way to the x86 integration where the MSI region, [0xFEE00000,0xFEEFFFFF], will be exposed by the q35 machine. However this will be handled in a separate series when not-DT support gets resolved.
Best Regards
Eric
This series can be found at:
https://github.com/eauger/qemu/tree/v5.0.0-virtio-iommu-msi-bypass-v7
History:
v6 -> v7:
- fix the hint message
- put the assert again on probe request
v5 -> v6:
- do not hardcode start/end addresses of doorbells
- check reserved region type on realize()
v4 -> v5:
- Take into account some additional comments from Markus:
- reserved region type becomes an unsigned + some comment/desc rewording
- assert if the type is not RESERVED or MSI
v3 -> v4:
- collected Jean and markus's R-bs
- tool into account all Markus' comments in [1/5] (except removal of goto)
- use ':' as delimitor instead of commas
- add example in 4/5 commit message as suggested by Markus
v2 -> v3:
- Introduce VIRT_MSI_CTRL_NONE in VirtMSIControllerType
- do not fill the remainder of the probe buffer
v1 -> v2:
- check which MSI controller is in use and advertise the corresponding MSI doorbell
- managed for both ITS and GICv2M
- various fixes spotted by Peter and Jean-Philippe, see individual logs
v1: Most of those patches were respinned from
[PATCH for-5.0 v11 00/20] VIRTIO-IOMMU device
except the last one which is new
Eric Auger (5):
qdev: Introduce DEFINE_PROP_RESERVED_REGION
virtio-iommu: Implement RESV_MEM probe request
virtio-iommu: Handle reserved regions in the translation process
virtio-iommu-pci: Add array of Interval properties hw/arm/virt: Let the virtio-iommu bypass MSIs
include/exec/memory.h | 6 ++ include/hw/arm/virt.h | 7 ++ include/hw/qdev-properties.h | 3 + include/hw/virtio/virtio-iommu.h | 2 + include/qemu/typedefs.h | 1 + hw/arm/virt.c | 30 ++++++++ hw/core/qdev-properties.c | 89 ++++++++++++++++++++++++ hw/virtio/virtio-iommu-pci.c | 11 +++ hw/virtio/virtio-iommu.c | 114 +++++++++++++++++++++++++++++-- hw/virtio/trace-events | 1 + 10 files changed, 260 insertions(+), 4 deletions(-)
Signature · 1 line
2.20.1
PATCH v7 1/5 qdev: Introduce DEFINE_PROP_RESERVED_REGION
Introduce a new property defining a reserved region:
<low address>:<high address>:<type>.
This will be used to encode reserved IOVA regions.
For instance, in virtio-iommu use case, reserved IOVA regions will be passed by the machine code to the virtio-iommu-pci device (an array of those). The type of the reserved region will match the virtio_iommu_probe_resv_mem subtype value:
- VIRTIO_IOMMU_RESV_MEM_T_RESERVED (0)
- VIRTIO_IOMMU_RESV_MEM_T_MSI (1)
on PC/Q35 machine, this will be used to inform the virtio-iommu-pci device it should bypass the MSI region. The reserved region will be: 0xfee00000:0xfeefffff:1.
On ARM, we can declare the ITS MSI doorbell as an MSI region to prevent MSIs from being mapped on guest side.
---
v4 -> v5:
- type becomes an unsigned + comment/error rewording
v3 -> v4:
- use ':' instead of commas as separators.
- rearrange error messages
- check snprintf returned value
- dared to keep Markus' R-b despite those changes
--- include/exec/memory.h | 6 +++ include/hw/qdev-properties.h | 3 ++ include/qemu/typedefs.h | 1 + hw/core/qdev-properties.c | 89 ++++++++++++++++++++++++++++++++++++ 4 files changed, 99 insertions(+)
include/exec/memory.h +6 -0
@@ -51,6 +51,12 @@ extern bool global_dirty_log;typedef struct MemoryRegionOps MemoryRegionOps ; +struct ReservedRegion { +hwaddr low; +hwaddr high; +unsigned type; +}; +typedef struct IOMMUTLBEntry IOMMUTLBEntry ;/* See address_space_translate: bit 0 is read, bit 1 is write. */
include/hw/qdev-properties.h +3 -0
@@ -19,6 +19,7 @@ extern const PropertyInfo qdev_prop_string;extern const PropertyInfo qdev_prop_chr;extern const PropertyInfo qdev_prop_tpm;extern const PropertyInfo qdev_prop_macaddr; +extern const PropertyInfo qdev_prop_reserved_region;extern const PropertyInfo qdev_prop_on_off_auto;extern const PropertyInfo qdev_prop_multifd_compression;extern const PropertyInfo qdev_prop_losttickpolicy;
@@ -184,6 +185,8 @@ extern const PropertyInfo qdev_prop_pcie_link_width; DEFINE_PROP(_n, _s, _f, qdev_prop_drive_iothread, BlockBackend *) #define DEFINE_PROP_MACADDR(_n, _s, _f) \DEFINE_PROP (_n, _s, _f, qdev_prop_macaddr, MACAddr) +#define DEFINE_PROP_RESERVED_REGION(_n, _s, _f) \ + DEFINE_PROP(_n, _s, _f, qdev_prop_reserved_region, ReservedRegion)#define DEFINE_PROP_ON_OFF_AUTO (_n, _s, _f, _d) \ DEFINE_PROP_SIGNED(_n, _s, _f, _d, qdev_prop_on_off_auto, OnOffAuto)#define DEFINE_PROP_MULTIFD_COMPRESSION (_n, _s, _f, _d) \
include/qemu/typedefs.h +1 -0
@@ -58,6 +58,7 @@ typedef struct ISABus ISABus;typedef struct ISADevice ISADevice ;typedef struct IsaDma IsaDma ;typedef struct MACAddr MACAddr ; +typedef struct ReservedRegion ReservedRegion ;typedef struct MachineClass MachineClass ;typedef struct MachineState MachineState ;typedef struct MemoryListener MemoryListener ;
hw/core/qdev-properties.c +89 -0
101 lines not on this page · raw message
Signature · 1 line
2.20.1
PATCH v7 2/5 virtio-iommu: Implement RESV_MEM probe request
In reply to Eric AugerThis patch implements the PROBE request. At the moment, only THE RESV_MEM property is handled. The first goal is to report iommu wide reserved regions such as the MSI regions set by the machine code. On x86 this will be the IOAPIC MSI region, [0xFEE00000 - 0xFEEFFFFF], on ARM this may be the ITS doorbell.
In the future we may introduce per device reserved regions. This will be useful when protecting host assigned devices which may expose their own reserved regions
---
v6 -> v7:
- put the assert again to make it clear there is no risk of truncation
v5 -> v6:
- removed validation of s->reserved_regions[i].type in the probe request as it should rather happen in the realize()
v4 -> v5:
- assert if reserved region type is different from RESERVED or MSI
v3 -> v4:
- removed any reference to the NONE property that does not exist anymore.
v2 -> v3:
- on probe, do not fill the reminder of the buffer with zeroes as the buffer was already zero initialized (Bharat)
v1 -> v2:
- move the unlock back to the same place
- remove the push label and factorize the code after the out label
- fix a bunch of cpu_to_leX according to the latest spec revision
- do not remove sizeof(last) from free space
- check the ep exists
--- include/hw/virtio/virtio-iommu.h | 2 + hw/virtio/virtio-iommu.c | 94 ++++++++++++++++++++++++++++++-- hw/virtio/trace-events | 1 + 3 files changed, 93 insertions(+), 4 deletions(-)
include/hw/virtio/virtio-iommu.h +2 -0
@@ -53,6 +53,8 @@ typedef struct VirtIOIOMMU {GHashTable *as_by_busptr;IOMMUPciBus *iommu_pcibus_by_bus_num[PCI_BUS_MAX ];PCIBus *primary_bus; +ReservedRegion *reserved_regions; +uint32_t nb_reserved_regions;GTree *domains;QemuMutex mutex;GTree *endpoints;
hw/virtio/virtio-iommu.c +90 -4
147 lines not on this page · raw message
hw/virtio/trace-events +1 -0
@@ -74,3 +74,4 @@ virtio_iommu_get_domain(uint32_t domain_id) "Alloc domain=%d" virtio_iommu_put_domain(uint32_t domain_id) "Free domain=%d" virtio_iommu_translate_out(uint64_t virt_addr, uint64_t phys_addr, uint32_t sid) "0x%"PRIx64" -> 0x%"PRIx64 " for sid=%d" virtio_iommu_report_fault(uint8_t reason, uint32_t flags, uint32_t endpoint, uint64_t addr) "FAULT reason=%d flags=%d endpoint=%d address =0x%"PRIx64 +virtio_iommu_fill_resv_property(uint32_t devid, uint8_t subtype, uint64_t start, uint64_t end) "dev= %d, type=%d start=0x%"PRIx64" end=0x%"PRIx64
Signature · 1 line
2.20.1
Re: [PATCH v7 2/5] virtio-iommu: Implement RESV_MEM probe request
Quoted 212 lines · Eric Auger
This patch implements the PROBE request. At the moment, only THE
RESV_MEMproperty is handled. The first goal is to report iommu wide reserved regions such as the MSI regions set by the machine code. On x86 this will be the IOAPIC MSI region, [0xFEE00000-0xFEEFFFFF], on ARM this may be the ITS doorbell.In the future we may introduce per device reserved regions. This will be useful when protecting host assigned devices which may expose their own reserved regions
Signed-off-by Eric Auger <[email protected]>Reviewed-by Jean-Philippe Brucker <[email protected]>---
v6 -> v7:
- put the assert again to make it clear there is no risk of truncationv5 -> v6:
- removed validation of s->reserved_regions[i].type in the probe request as it should rather happen in therealize()v4 -> v5:
- assert if reserved region type is different from RESERVED or MSIv3 -> v4:
- removed any reference to the NONE property that does not exist anymore.v2 -> v3:
- on probe, do not fill the reminder of the buffer with zeroes as the buffer was already zero initialized (Bharat)v1 -> v2:
- move the unlock back to the same place
- remove the push label and factorize the code after the out label
- fix a bunch ofcpu_to_leXaccording to the latest spec revision
- do not remove sizeof(last) from free space
- check the ep exists--- include/hw/virtio/virtio-iommu.h | 2 + hw/virtio/virtio-iommu.c | 94 ++++++++++++++++++++++++++++++-- hw/virtio/trace-events | 1 + 3 files changed, 93 insertions(+), 4 deletions(-)include/hw/virtio/virtio-iommu.h +2
-0@@ -53,6 +53,8 @@ typedef struct VirtIOIOMMU {GHashTable *as_by_busptr;IOMMUPciBus *iommu_pcibus_by_bus_num[PCI_BUS_MAX ];PCIBus *primary_bus; +ReservedRegion *reserved_regions; +uint32_t nb_reserved_regions;GTree *domains;QemuMutex mutex;GTree *endpoints;hw/virtio/virtio-iommu.c +89
-4@@ -38,6 +38,7 @@/* Max size */ #define VIOMMU_DEFAULT_QUEUE_SIZE 256 +#define VIOMMU_PROBE_SIZE 512 typedef struct VirtIOIOMMUDomain { uint32_t id;@@ -378,6 +379,65 @@ static int virtio_iommu_unmap(VirtIOIOMMU *s,return ret; } +static ssize_t virtio_iommu_fill_resv_mem_prop (VirtIOIOMMU *s,uint32_t ep, +uint8_t *buf,size_t free) +{ +struct virtio_iommu_probe_resv_mem prop = {}; +size_t size =sizeof (prop), length = size -sizeof (prop.head), total; +int i; + + total = size * s->nb_reserved_regions; + +if (total > free) { +return -ENOSPC ; + } + +for (i =0 ; i < s->nb_reserved_regions; i++) { +unsigned subtype = s->reserved_regions[i].type; + +assert (subtype ==VIRTIO_IOMMU_RESV_MEM_T_RESERVED || + subtype ==VIRTIO_IOMMU_RESV_MEM_T_MSI ); + prop.head.type =cpu_to_le16 (VIRTIO_IOMMU_PROBE_T_RESV_MEM ); + prop.head.length =cpu_to_le16 (length); + prop.subtype = subtype; + prop.start =cpu_to_le64 (s->reserved_regions[i].low); + prop.end =cpu_to_le64 (s->reserved_regions[i].high); + +memcpy (buf, &prop, size); + +trace_virtio_iommu_fill_resv_property (ep, prop.subtype, + prop.start, prop.end); + buf += size; + } +return total; +} + +/** +* virtio_iommu_probe - Fill the probe request buffer with +* the properties the device is able to return +*/ +static int virtio_iommu_probe (VirtIOIOMMU *s, +struct virtio_iommu_req_probe *req, +uint8_t *buf) +{ +uint32_t ep_id =le32_to_cpu (req->endpoint); +size_t free =VIOMMU_PROBE_SIZE ; +ssize_t count; + +if (!virtio_iommu_mr (s, ep_id)) { +return VIRTIO_IOMMU_S_NOENT ; + } + + count =virtio_iommu_fill_resv_mem_prop (s, ep_id, buf, free); +if (count <0 ) { +return VIRTIO_IOMMU_S_INVAL ; + } + buf += count; + free -= count; + +return VIRTIO_IOMMU_S_OK ; +} +static int virtio_iommu_iov_to_req (struct iovec *iov,unsigned int iov_cnt,void *req,size_t req_sz)@@ -407,15 +467,27 @@ virtio_iommu_handle_req(detach)virtio_iommu_handle_req (map)virtio_iommu_handle_req (unmap) +static int virtio_iommu_handle_probe(VirtIOIOMMU *s, + struct iovec *iov, + unsigned int iov_cnt, + uint8_t *buf) +{ +struct virtio_iommu_req_probe req; +int ret =virtio_iommu_iov_to_req (iov, iov_cnt, &req,sizeof (req)); + + return ret ? ret : virtio_iommu_probe(s, &req, buf); +} + static void virtio_iommu_handle_command(VirtIODevice *vdev, VirtQueue *vq) {VirtIOIOMMU *s =VIRTIO_IOMMU (vdev);struct virtio_iommu_req_head head;struct virtio_iommu_req_tail tail = {}; +size_t output_size =sizeof (tail), sz;VirtQueueElement *elem;unsigned int iov_cnt;struct iovec *iov; -size_t sz; +void *buf =NULL ;for (;;) { elem =virtqueue_pop (vq,sizeof (VirtQueueElement));@@ -452,6 +524,17 @@ static void virtio_iommu_handle_command(VirtIODevice *vdev, VirtQueue *vq)case VIRTIO_IOMMU_T_UNMAP : tail.status =virtio_iommu_handle_unmap (s, iov, iov_cnt);break ; +case VIRTIO_IOMMU_T_PROBE : + { +struct virtio_iommu_req_tail *ptail; + + output_size = s->config.probe_size +sizeof (tail); + buf =g_malloc0 (output_size); + + ptail = (struct virtio_iommu_req_tail *) + (buf + s->config.probe_size); + ptail->status =virtio_iommu_handle_probe (s, iov, iov_cnt, buf); + }default : tail.status =VIRTIO_IOMMU_S_UNSUPP ; }@@ -459,12 +542,13 @@ static void virtio_iommu_handle_command(VirtIODevice *vdev, VirtQueue *vq) out: sz =iov_from_buf (elem->in_sg, elem->in_num,0 , - &tail,sizeof (tail)); -assert (sz ==sizeof (tail)); + buf ? buf : &tail, output_size); +assert (sz == output_size); -virtqueue_push (vq, elem,sizeof (tail)); +virtqueue_push (vq, elem, sz);virtio_notify (vdev, vq);g_free (elem); +g_free (buf); } }@@ -667,6 +751,7 @@ static void virtio_iommu_device_realize(DeviceState *dev, Error **errp) s->config.page_size_mask =TARGET_PAGE_MASK ; s->config.input_range.end =-1UL ; s->config.domain_range.end =32 ; + s->config.probe_size =VIOMMU_PROBE_SIZE ;virtio_add_feature (&s->features,VIRTIO_RING_F_EVENT_IDX );virtio_add_feature (&s->features,VIRTIO_RING_F_INDIRECT_DESC );
hw/virtio/virtio-iommu.c static void virtio_iommu_device_realize(DeviceState *dev, Error **errp) · line 765
Quoted 5 lines
hw/virtio/virtio-iommu.c +1
-0@@ -676,6 +761,7 @@ static void virtio_iommu_device_realize(DeviceState *dev, Error **errp)virtio_add_feature (&s->features,VIRTIO_IOMMU_F_MAP_UNMAP );virtio_add_feature (&s->features,VIRTIO_IOMMU_F_BYPASS );virtio_add_feature (&s->features,VIRTIO_IOMMU_F_MMIO ); +virtio_add_feature (&s->features,VIRTIO_IOMMU_F_PROBE );
Don't we need to disable this for existing machine types?
Quoted 13 lines
hw/virtio/virtio-iommu.c +0
-0qemu_mutex_init (&s->mutex);hw/virtio/trace-events +1
-0@@ -74,3 +74,4 @@ virtio_iommu_get_domain(uint32_t domain_id) "Alloc domain=%d" virtio_iommu_put_domain(uint32_t domain_id) "Free domain=%d" virtio_iommu_translate_out(uint64_t virt_addr, uint64_t phys_addr, uint32_t sid) "0x%"PRIx64" -> 0x%"PRIx64 " for sid=%d" virtio_iommu_report_fault(uint8_t reason, uint32_t flags, uint32_t endpoint, uint64_t addr) "FAULT reason=%d flags=%d endpoint=%d address =0x%"PRIx64 +virtio_iommu_fill_resv_property(uint32_t devid, uint8_t subtype, uint64_t start, uint64_t end) "dev= %d, type=%d start=0x%"PRIx64" end=0x%"PRIx64Signature · 1 line
2.20.1
Re: [PATCH v7 2/5] virtio-iommu: Implement RESV_MEM probe request
Pls ignore, my mail is acting up serving me old patches. Sorry about the noise!
Signature · 1 line
MST
PATCH v7 3/5 virtio-iommu: Handle reserved regions in the translation process
In reply to Eric AugerWhen translating an address we need to check if it belongs to a reserved virtual address range. If it does, there are 2 cases:
- it belongs to a RESERVED region: the guest should neither use this address in a MAP not instruct the end-point to DMA on them. We report an error
- It belongs to an MSI region: we bypass the translation.
---
v1 -> v2:
- use addr when testing addr belongs to the reserved region and use a block local variable
--- hw/virtio/virtio-iommu.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)
hw/virtio/virtio-iommu.c +20 -0
@@ -607,6 +607,7 @@ static IOMMUTLBEntry virtio_iommu_translate(IOMMUMemoryRegion *mr, hwaddr addr,uint32_t sid, flags;bool bypass_allowed;bool found; +int i; interval.low = addr; interval.high = addr +1 ;
@@ -640,6 +641,25 @@ static IOMMUTLBEntry virtio_iommu_translate(IOMMUMemoryRegion *mr, hwaddr addr, goto unlock; } +for (i =0 ; i < s->nb_reserved_regions; i++) { +ReservedRegion *reg = &s->reserved_regions[i]; + +if (addr >= reg->low && addr <= reg->high) { +switch (reg->type) { +case VIRTIO_IOMMU_RESV_MEM_T_MSI : + entry.perm = flag; +break ; +case VIRTIO_IOMMU_RESV_MEM_T_RESERVED : +default : +virtio_iommu_report_fault (s,VIRTIO_IOMMU_FAULT_R_MAPPING , +VIRTIO_IOMMU_FAULT_F_ADDRESS , + sid, addr); +break ; + } + goto unlock; + } + } +if (!ep->domain) {if (!bypass_allowed) { error_report_once("%s %02x:%02x.%01x not attached to any domain",
Signature · 1 line
2.20.1
PATCH v7 4/5 virtio-iommu-pci: Add array of Interval properties
In reply to Eric AugerThe machine may need to pass reserved regions to the virtio-iommu-pci device (such as the MSI window on x86 or the MSI doorbells on ARM).
So let's add an array of Interval properties.
Note: if some reserved regions are already set by the machine code - which should be the case in general -, the length of the property array is already set and prevents the end-user from modifying them. For example, attempting to use:
-device virtio-iommu-pci,\
len-reserved-regions=1,reserved-regions[0]=0xfee00000:0xfeefffff:1
would result in the following error message:
qemu-system-aarch64: -device virtio-iommu-pci,addr=0xa, len-reserved-regions=1,reserved-regions[0]=0xfee00000:0xfeefffff:1:
array size property len-reserved-regions may not be set more than once
Otherwise, for example, adding two reserved regions is achieved using the following options:
-device virtio-iommu-pci,addr=0xa,len-reserved-regions=2,\
reserved-regions[0]=0xfee00000:0xfeefffff:1,\
reserved-regions[1]=0x1000000:100ffff:1
---
v6 -> v7:
- fix hint message with correct valid values, ie. 0 and 1
- added Markus' R-b
v5 -> v6:
- check the type value
- removed Jean's R-b
v3 -> v4:
- added examples in the commit message as suggested by Markus
- added Jean's R-b
--- hw/virtio/virtio-iommu-pci.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
hw/virtio/virtio-iommu-pci.c +11 -0
@@ -33,6 +33,9 @@ struct VirtIOIOMMUPCI {static Property virtio_iommu_pci_properties[] = {DEFINE_PROP_UINT32 ("class" , VirtIOPCIProxy, class_code,0 ), +DEFINE_PROP_ARRAY ("reserved-regions" , VirtIOIOMMUPCI, + vdev.nb_reserved_regions, vdev.reserved_regions, + qdev_prop_reserved_region, ReservedRegion),DEFINE_PROP_END_OF_LIST (), };
@@ -40,6 +43,7 @@ static void virtio_iommu_pci_realize(VirtIOPCIProxy *vpci_dev, Error **errp) {VirtIOIOMMUPCI *dev =VIRTIO_IOMMU_PCI (vpci_dev);DeviceState *vdev =DEVICE (&dev->vdev); +VirtIOIOMMU *s =VIRTIO_IOMMU (vdev);if (!qdev_get_machine_hotplug_handler (DEVICE (vpci_dev))) {MachineClass *mc =MACHINE_GET_CLASS (qdev_get_machine ());
@@ -54,6 +58,13 @@ static void virtio_iommu_pci_realize(VirtIOPCIProxy *vpci_dev, Error **errp) "-no-acpi\n");return ; } +for (int i =0 ; i < s->nb_reserved_regions; i++) { +if (s->reserved_regions[i].type !=VIRTIO_IOMMU_RESV_MEM_T_RESERVED && + s->reserved_regions[i].type !=VIRTIO_IOMMU_RESV_MEM_T_MSI ) { +error_setg (errp,"reserved region %d has an invalid type" , i); +error_append_hint (errp,"Valid values are 0 and 1\n" ); + } + }object_property_set_link (OBJECT (dev),OBJECT (pci_get_bus (&vpci_dev->pci_dev)),"primary-bus" , errp);
Signature · 1 line
2.20.1
PATCH v7 5/5 hw/arm/virt: Let the virtio-iommu bypass MSIs
In reply to Eric AugerAt the moment the virtio-iommu translates MSI transactions. This behavior is inherited from ARM SMMU. The virt machine code knows where the guest MSI doorbells are so we can easily declare those regions as VIRTIO_IOMMU_RESV_MEM_T_MSI. With that setting the guest will not map MSIs through the IOMMU and those transactions will be simply bypassed.
Depending on which MSI controller is in use (ITS or GICV2M), we declare either:
- the ITS interrupt translation space (ITS_base + 0x10000), containing the GITS_TRANSLATOR or
- The GICV2M single frame, containing the MSI_SETSP_NS register.
---
v5 -> v6:
- do not hardcode doorbell base and size
- removed Jean's R-b
v3 -> v4:
- use ':' as separators
v2 -> v3:
- Add a new value to VirtMSIControllerType
v1 -> v2:
- Test which MSI controller is instantiated
- If GICV2M is in use, declare its doorbell as an MSI doorbell too
--- include/hw/arm/virt.h | 7 +++++++ hw/arm/virt.c | 30 ++++++++++++++++++++++++++++++ 2 files changed, 37 insertions(+)
include/hw/arm/virt.h +7 -0
@@ -96,6 +96,12 @@ typedef enum VirtIOMMUType {VIRT_IOMMU_VIRTIO , } VirtIOMMUType; +typedef enum VirtMSIControllerType { +VIRT_MSI_CTRL_NONE , +VIRT_MSI_CTRL_GICV2M , +VIRT_MSI_CTRL_ITS , +}VirtMSIControllerType ; + typedef enum VirtGICType { VIRT_GIC_VERSION_MAX, VIRT_GIC_VERSION_HOST,
@@ -136,6 +142,7 @@ typedef struct {OnOffAuto acpi;VirtGICType gic_version;VirtIOMMUType iommu; +VirtMSIControllerType msi_controller;uint16_t virtio_iommu_bdf;struct arm_boot_info bootinfo;MemMapEntry *memmap;
hw/arm/virt.c +30 -0
@@ -602,6 +602,7 @@ static void create_its(VirtMachineState *vms)sysbus_mmio_map (SYS_BUS_DEVICE (dev),0 , vms->memmap[VIRT_GIC_ITS ].base);fdt_add_its_gic_node (vms); + vms->msi_controller =VIRT_MSI_CTRL_ITS ; }static void create_v2m (VirtMachineState *vms)
@@ -622,6 +623,7 @@ static void create_v2m(VirtMachineState *vms) }fdt_add_v2m_gic_node (vms); + vms->msi_controller =VIRT_MSI_CTRL_GICV2M ; }static void create_gic (VirtMachineState *vms)
@@ -2200,8 +2202,36 @@ out:static void virt_machine_device_pre_plug_cb (HotplugHandler *hotplug_dev,DeviceState *dev,Error **errp) { +VirtMachineState *vms =VIRT_MACHINE (hotplug_dev); +if (object_dynamic_cast (OBJECT (dev),TYPE_PC_DIMM )) {virt_memory_pre_plug (hotplug_dev, dev, errp); + }else if (object_dynamic_cast (OBJECT (dev),TYPE_VIRTIO_IOMMU_PCI )) { +hwaddr db_start =0 , db_end =0 ; +char *resv_prop_str; + +switch (vms->msi_controller) { +case VIRT_MSI_CTRL_NONE : +return ; +case VIRT_MSI_CTRL_ITS : +/* GITS_TRANSLATER page */ + db_start = base_memmap[VIRT_GIC_ITS ].base +0x10000 ; + db_end = base_memmap[VIRT_GIC_ITS ].base + + base_memmap[VIRT_GIC_ITS ].size -1 ; +break ; +case VIRT_MSI_CTRL_GICV2M : +/* MSI_SETSPI_NS page */ + db_start = base_memmap[VIRT_GIC_V2M ].base; + db_end = db_start + base_memmap[VIRT_GIC_V2M ].size -1 ; +break ; + } + resv_prop_str =g_strdup_printf ("0x%" PRIx64":0x%" PRIx64":%u" , + db_start, db_end, +VIRTIO_IOMMU_RESV_MEM_T_MSI ); + +qdev_prop_set_uint32 (dev,"len-reserved-regions" ,1 ); +qdev_prop_set_string (dev,"reserved-regions[0]" , resv_prop_str); +g_free (resv_prop_str); } }
Signature · 1 line
2.20.1
Re: [PATCH v7 5/5] hw/arm/virt: Let the virtio-iommu bypass MSIs
Quoted 7 lines · Eric Auger
At the moment the virtio-iommu translates MSI transactions. This behavior is inherited from ARM SMMU. The virt machine code knows where the guest MSI doorbells are so we can easily declare those regions as
VIRTIO_IOMMU_RESV_MEM_T_MSI. With that setting the guest will not map MSIs through the IOMMU and those transactions will be simply bypassed.
Quoted 7 lines
Depending on which MSI controller is in use (ITS or GICV2M), we declare either:
- the ITS interrupt translation space (ITS_base+0x10000), containing theGITS_TRANSLATORor
- The GICV2M single frame, containing theMSI_SETSP_NSregister.Signed-off-by Eric Auger <[email protected]>
This needs ARM ACK.
Quoted 103 lines
---
v5 -> v6:
- do not hardcode doorbell base and size
- removed Jean's R-bv3 -> v4:
- use ':' as separatorsv2 -> v3:
- Add a new value to VirtMSIControllerTypev1 -> v2:
- Test which MSI controller is instantiated
- If GICV2M is in use, declare its doorbell as an MSI doorbell too--- include/hw/arm/virt.h | 7 +++++++ hw/arm/virt.c | 30 ++++++++++++++++++++++++++++++ 2 files changed, 37 insertions(+)include/hw/arm/virt.h +7
-0@@ -96,6 +96,12 @@ typedef enum VirtIOMMUType {VIRT_IOMMU_VIRTIO , } VirtIOMMUType; +typedef enum VirtMSIControllerType { +VIRT_MSI_CTRL_NONE , +VIRT_MSI_CTRL_GICV2M , +VIRT_MSI_CTRL_ITS , +}VirtMSIControllerType ; + typedef enum VirtGICType { VIRT_GIC_VERSION_MAX, VIRT_GIC_VERSION_HOST,@@ -136,6 +142,7 @@ typedef struct {OnOffAuto acpi;VirtGICType gic_version;VirtIOMMUType iommu; +VirtMSIControllerType msi_controller;uint16_t virtio_iommu_bdf;struct arm_boot_info bootinfo;MemMapEntry *memmap;hw/arm/virt.c +30
-0@@ -602,6 +602,7 @@ static void create_its(VirtMachineState *vms)sysbus_mmio_map (SYS_BUS_DEVICE (dev),0 , vms->memmap[VIRT_GIC_ITS ].base);fdt_add_its_gic_node (vms); + vms->msi_controller =VIRT_MSI_CTRL_ITS ; }static void create_v2m (VirtMachineState *vms)@@ -622,6 +623,7 @@ static void create_v2m(VirtMachineState *vms) }fdt_add_v2m_gic_node (vms); + vms->msi_controller =VIRT_MSI_CTRL_GICV2M ; }static void create_gic (VirtMachineState *vms)@@ -2200,8 +2202,36 @@ out:static void virt_machine_device_pre_plug_cb (HotplugHandler *hotplug_dev,DeviceState *dev,Error **errp) { +VirtMachineState *vms =VIRT_MACHINE (hotplug_dev); +if (object_dynamic_cast (OBJECT (dev),TYPE_PC_DIMM )) {virt_memory_pre_plug (hotplug_dev, dev, errp); + }else if (object_dynamic_cast (OBJECT (dev),TYPE_VIRTIO_IOMMU_PCI )) { +hwaddr db_start =0 , db_end =0 ; +char *resv_prop_str; + +switch (vms->msi_controller) { +case VIRT_MSI_CTRL_NONE : +return ; +case VIRT_MSI_CTRL_ITS : +/* GITS_TRANSLATER page */ + db_start = base_memmap[VIRT_GIC_ITS ].base +0x10000 ; + db_end = base_memmap[VIRT_GIC_ITS ].base + + base_memmap[VIRT_GIC_ITS ].size -1 ; +break ; +case VIRT_MSI_CTRL_GICV2M : +/* MSI_SETSPI_NS page */ + db_start = base_memmap[VIRT_GIC_V2M ].base; + db_end = db_start + base_memmap[VIRT_GIC_V2M ].size -1 ; +break ; + } + resv_prop_str =g_strdup_printf ("0x%" PRIx64":0x%" PRIx64":%u" , + db_start, db_end, +VIRTIO_IOMMU_RESV_MEM_T_MSI ); + +qdev_prop_set_uint32 (dev,"len-reserved-regions" ,1 ); +qdev_prop_set_string (dev,"reserved-regions[0]" , resv_prop_str); +g_free (resv_prop_str); } }Signature · 1 line
2.20.1
Hi,
Quoted 15 lines · Eric Auger
By default the virtio-iommu translates MSI transactions. This behavior is inherited from ARM SMMU. However the virt machine code knows where the MSI doorbells are, so we can easily declare those regions as
VIRTIO_IOMMU_RESV_MEM_T_MSI. With that setting the guest iommu subsystem will not need to map MSIs. This setup will simplify the VFIO integration.In this series, the ITS or GICV2M doorbells are declared as HW MSI regions to be bypassed by the VIRTIO-IOMMU.
This also paves the way to the x86 integration where the MSI region, [0xFEE00000,0xFEEFFFFF], will be exposed by the q35 machine. However this will be handled in a separate series when not-DT support gets resolved.
Quoted 6 lines
Best Regards
Eric
This series can be found at:
https://github.com/eauger/qemu/tree/v5.0.0-virtio-iommu-msi-bypass-v7
A gentle reminder to make sure this series does not get forgotten before the freeze.
Best Regards
Eric
Quoted 59 lines
History:
v6 -> v7:
- fix the hint message
- put the assert again on probe requestv5 -> v6:
- do not hardcode start/end addresses of doorbells
- check reserved region type onrealize()v4 -> v5:
- Take into account some additional comments from Markus:
- reserved region type becomes an unsigned + some comment/desc rewording
- assert if the type is not RESERVED or MSIv3 -> v4:
- collected Jean and markus's R-bs
- tool into account all Markus' comments in [1/5] (except removal of goto)
- use ':' as delimitor instead of commas
- add example in 4/5 commit message as suggested by Markusv2 -> v3:
- IntroduceVIRT_MSI_CTRL_NONEin VirtMSIControllerType
- do not fill the remainder of the probe bufferv1 -> v2:
- check which MSI controller is in use and advertise the corresponding MSI doorbell
- managed for both ITS and GICv2M
- various fixes spotted by Peter and Jean-Philippe, see individual logsv1: Most of those patches were respinned from
[PATCH for-5.0 v11 00/20] VIRTIO-IOMMU device
except the last one which is newEric Auger (5):
qdev: IntroduceDEFINE_PROP_RESERVED_REGION
virtio-iommu: ImplementRESV_MEMprobe request
virtio-iommu: Handle reserved regions in the translation process
virtio-iommu-pci: Add array of Interval properties hw/arm/virt: Let the virtio-iommu bypass MSIsinclude/exec/memory.h | 6 ++ include/hw/arm/virt.h | 7 ++ include/hw/qdev-properties.h | 3 + include/hw/virtio/virtio-iommu.h | 2 + include/qemu/typedefs.h | 1 + hw/arm/virt.c | 30 ++++++++ hw/core/qdev-properties.c | 89 ++++++++++++++++++++++++ hw/virtio/virtio-iommu-pci.c | 11 +++ hw/virtio/virtio-iommu.c | 114 +++++++++++++++++++++++++++++-- hw/virtio/trace-events | 1 + 10 files changed, 260 insertions(+), 4 deletions(-)
Quoted 7 lines · Eric Auger
By default the virtio-iommu translates MSI transactions. This behavior is inherited from ARM SMMU. However the virt machine code knows where the MSI doorbells are, so we can easily declare those regions as
VIRTIO_IOMMU_RESV_MEM_T_MSI. With that setting the guest iommu subsystem will not need to map MSIs. This setup will simplify the VFIO integration.
Quoted 7 lines
In this series, the ITS or GICV2M doorbells are declared as HW MSI regions to be bypassed by the VIRTIO-IOMMU.
This also paves the way to the x86 integration where the MSI region, [0xFEE00000,0xFEEFFFFF], will be exposed by the q35 machine. However this will be handled in a separate series when not-DT support gets resolved.
virtio things:
ARM tree makes sense for this, right?
Quoted 67 lines
Best Regards
Eric
This series can be found at:
https://github.com/eauger/qemu/tree/v5.0.0-virtio-iommu-msi-bypass-v7History:
v6 -> v7:
- fix the hint message
- put the assert again on probe requestv5 -> v6:
- do not hardcode start/end addresses of doorbells
- check reserved region type onrealize()v4 -> v5:
- Take into account some additional comments from Markus:
- reserved region type becomes an unsigned + some comment/desc rewording
- assert if the type is not RESERVED or MSIv3 -> v4:
- collected Jean and markus's R-bs
- tool into account all Markus' comments in [1/5] (except removal of goto)
- use ':' as delimitor instead of commas
- add example in 4/5 commit message as suggested by Markusv2 -> v3:
- IntroduceVIRT_MSI_CTRL_NONEin VirtMSIControllerType
- do not fill the remainder of the probe bufferv1 -> v2:
- check which MSI controller is in use and advertise the corresponding MSI doorbell
- managed for both ITS and GICv2M
- various fixes spotted by Peter and Jean-Philippe, see individual logsv1: Most of those patches were respinned from
[PATCH for-5.0 v11 00/20] VIRTIO-IOMMU device
except the last one which is newEric Auger (5):
qdev: IntroduceDEFINE_PROP_RESERVED_REGION
virtio-iommu: ImplementRESV_MEMprobe request
virtio-iommu: Handle reserved regions in the translation process
virtio-iommu-pci: Add array of Interval properties hw/arm/virt: Let the virtio-iommu bypass MSIsinclude/exec/memory.h | 6 ++ include/hw/arm/virt.h | 7 ++ include/hw/qdev-properties.h | 3 + include/hw/virtio/virtio-iommu.h | 2 + include/qemu/typedefs.h | 1 + hw/arm/virt.c | 30 ++++++++ hw/core/qdev-properties.c | 89 ++++++++++++++++++++++++ hw/virtio/virtio-iommu-pci.c | 11 +++ hw/virtio/virtio-iommu.c | 114 +++++++++++++++++++++++++++++-- hw/virtio/trace-events | 1 + 10 files changed, 260 insertions(+), 4 deletions(-)Signature · 1 line
2.20.1
Hi Michael,
Quoted 17 lines · Michael S. Tsirkin
Quoted 7 lines · Eric Auger
By default the virtio-iommu translates MSI transactions. This behavior is inherited from ARM SMMU. However the virt machine code knows where the MSI doorbells are, so we can easily declare those regions as
VIRTIO_IOMMU_RESV_MEM_T_MSI. With that setting the guest iommu subsystem will not need to map MSIs. This setup will simplify the VFIO integration.Quoted 7 lines
In this series, the ITS or GICV2M doorbells are declared as HW MSI regions to be bypassed by the VIRTIO-IOMMU.
This also paves the way to the x86 integration where the MSI region, [0xFEE00000,0xFEEFFFFF], will be exposed by the q35 machine. However this will be handled in a separate series when not-DT support gets resolved.
Quoted 6 lines
virtio things:
Reviewed-by Michael S. Tsirkin <[email protected]>ARM tree makes sense for this, right?
Yes I think so.
Thanks
Eric
Quoted 71 lines
Quoted 69 lines
Best Regards
Eric
This series can be found at:
https://github.com/eauger/qemu/tree/v5.0.0-virtio-iommu-msi-bypass-v7History:
v6 -> v7:
- fix the hint message
- put the assert again on probe requestv5 -> v6:
- do not hardcode start/end addresses of doorbells
- check reserved region type onrealize()v4 -> v5:
- Take into account some additional comments from Markus:
- reserved region type becomes an unsigned + some comment/desc rewording
- assert if the type is not RESERVED or MSIv3 -> v4:
- collected Jean and markus's R-bs
- tool into account all Markus' comments in [1/5] (except removal of goto)
- use ':' as delimitor instead of commas
- add example in 4/5 commit message as suggested by Markusv2 -> v3:
- IntroduceVIRT_MSI_CTRL_NONEin VirtMSIControllerType
- do not fill the remainder of the probe bufferv1 -> v2:
- check which MSI controller is in use and advertise the corresponding MSI doorbell
- managed for both ITS and GICv2M
- various fixes spotted by Peter and Jean-Philippe, see individual logsv1: Most of those patches were respinned from
[PATCH for-5.0 v11 00/20] VIRTIO-IOMMU device
except the last one which is newEric Auger (5):
qdev: IntroduceDEFINE_PROP_RESERVED_REGION
virtio-iommu: ImplementRESV_MEMprobe request
virtio-iommu: Handle reserved regions in the translation process
virtio-iommu-pci: Add array of Interval properties hw/arm/virt: Let the virtio-iommu bypass MSIsinclude/exec/memory.h | 6 ++ include/hw/arm/virt.h | 7 ++ include/hw/qdev-properties.h | 3 + include/hw/virtio/virtio-iommu.h | 2 + include/qemu/typedefs.h | 1 + hw/arm/virt.c | 30 ++++++++ hw/core/qdev-properties.c | 89 ++++++++++++++++++++++++ hw/virtio/virtio-iommu-pci.c | 11 +++ hw/virtio/virtio-iommu.c | 114 +++++++++++++++++++++++++++++-- hw/virtio/trace-events | 1 + 10 files changed, 260 insertions(+), 4 deletions(-)Signature · 1 line
2.20.1
Michael S. Tsirkin
Reviewed-byQuoted 25 lines · Auger Eric
Hi Michael,
Quoted 22 lines · Michael S. Tsirkin
Quoted 7 lines · Eric Auger
By default the virtio-iommu translates MSI transactions. This behavior is inherited from ARM SMMU. However the virt machine code knows where the MSI doorbells are, so we can easily declare those regions as
VIRTIO_IOMMU_RESV_MEM_T_MSI. With that setting the guest iommu subsystem will not need to map MSIs. This setup will simplify the VFIO integration.Quoted 7 lines
In this series, the ITS or GICV2M doorbells are declared as HW MSI regions to be bypassed by the VIRTIO-IOMMU.
This also paves the way to the x86 integration where the MSI region, [0xFEE00000,0xFEEFFFFF], will be exposed by the q35 machine. However this will be handled in a separate series when not-DT support gets resolved.
virtio things:
Reviewed-by Michael S. Tsirkin <[email protected]>
Quoted 7 lines
Quoted 1 line
ARM tree makes sense for this, right?
Yes I think so.
Thanks
Eric
Okay. Feel free to merge through ARM tree pls.
Quoted 71 lines
Quoted 71 lines
Quoted 69 lines
Best Regards
Eric
This series can be found at:
https://github.com/eauger/qemu/tree/v5.0.0-virtio-iommu-msi-bypass-v7History:
v6 -> v7:
- fix the hint message
- put the assert again on probe requestv5 -> v6:
- do not hardcode start/end addresses of doorbells
- check reserved region type onrealize()v4 -> v5:
- Take into account some additional comments from Markus:
- reserved region type becomes an unsigned + some comment/desc rewording
- assert if the type is not RESERVED or MSIv3 -> v4:
- collected Jean and markus's R-bs
- tool into account all Markus' comments in [1/5] (except removal of goto)
- use ':' as delimitor instead of commas
- add example in 4/5 commit message as suggested by Markusv2 -> v3:
- IntroduceVIRT_MSI_CTRL_NONEin VirtMSIControllerType
- do not fill the remainder of the probe bufferv1 -> v2:
- check which MSI controller is in use and advertise the corresponding MSI doorbell
- managed for both ITS and GICv2M
- various fixes spotted by Peter and Jean-Philippe, see individual logsv1: Most of those patches were respinned from
[PATCH for-5.0 v11 00/20] VIRTIO-IOMMU device
except the last one which is newEric Auger (5):
qdev: IntroduceDEFINE_PROP_RESERVED_REGION
virtio-iommu: ImplementRESV_MEMprobe request
virtio-iommu: Handle reserved regions in the translation process
virtio-iommu-pci: Add array of Interval properties hw/arm/virt: Let the virtio-iommu bypass MSIsinclude/exec/memory.h | 6 ++ include/hw/arm/virt.h | 7 ++ include/hw/qdev-properties.h | 3 + include/hw/virtio/virtio-iommu.h | 2 + include/qemu/typedefs.h | 1 + hw/arm/virt.c | 30 ++++++++ hw/core/qdev-properties.c | 89 ++++++++++++++++++++++++ hw/virtio/virtio-iommu-pci.c | 11 +++ hw/virtio/virtio-iommu.c | 114 +++++++++++++++++++++++++++++-- hw/virtio/trace-events | 1 + 10 files changed, 260 insertions(+), 4 deletions(-)Signature · 1 line
2.20.1
Peter Maydell
AppliedQuoted 11 lines · Eric Auger
By default the virtio-iommu translates MSI transactions. This behavior is inherited from ARM SMMU. However the virt machine code knows where the MSI doorbells are, so we can easily declare those regions as
VIRTIO_IOMMU_RESV_MEM_T_MSI. With that setting the guest iommu subsystem will not need to map MSIs. This setup will simplify the VFIO integration.In this series, the ITS or GICV2M doorbells are declared as HW MSI regions to be bypassed by the VIRTIO-IOMMU.
Quoted 8 lines
This also paves the way to the x86 integration where the MSI region, [0xFEE00000,0xFEEFFFFF], will be exposed by the q35 machine. However this will be handled in a separate series when not-DT support gets resolved.
Best Regards
Eric
Applied to target-arm.next, thanks.
-- PMM