hw/arm/virt: Add multiple nested SMMUs
cover hw/arm/virt: Add multiple nested SMMUs
Hi all,
This is a draft solution adding multiple nested SMMU instances to VM. The main goal of the series is to collect opinions, to figure out a reasonable solution that would fit our needs.
I understood that there are concerns regarding this support, from our previous discussion:
https://lore.kernel.org/all/ZEcT%2F7erkhHDaNvD@Asurada-Nvidia/
Yet, some followup discussion in the kernel maillist has shifted the direction of regular SMMU nesting to potentially having multiple vSMMU instances as well:
https://lore.kernel.org/all/[email protected]/
I will summerize all the dots in the following paragraphs:
[ Why do we need multiple nested SMMUs? ]
1, This is a must feature for NVIDIA's Grace SoC to support its CMDQV
(an extension HW for SMMU). It allows to assgin a Command Queue HW dedicatedly to a VM. Then VM controls it via an mmap'd MMIO page:
https://lore.kernel.org/all/f00da8df12a154204e53b343b2439bf31517241f.1712978213.git.nicolinc@nvidia.com/
Each Grace SoC has 5 SMMUs (i.e. 5 CMDQVs), meaning there can be 5 MMIO pages. If QEMU only supports one vSMMU and all passthrough devices attach to one shared vSMMU, it technically cannot mmap 5 MMIO pages, nor assign devices to use corresponding pages.
2, This is optional for nested SMMU, and essentially a design choice
between a single vSMMU design and a multiple-vSMMU design. Here're the pros and cons:
+ Pros for single vSMMU design
a) It is easy and clean, by all means.
- Cons for single vSMMU design
b) It can have complications if underlying pSMMUs are different.
c) Emulated devices might have to be added to the nested SMMU, since "iommu=nested-smmuv3" enables for the whole VM. This means the vSMMU instance has to act at the same time as both a nested SMMU and a para-virt SMMU.
d) IOTLB inefficiency. Since devices behind different pSMMUs are attached to a single vSMMU, the vSMMU code traps invalidation commands in a shared guest CMDQ, meaning it needs to dispatch those commands correctly to pSMMUs, by either a broadcast or walking through a lookup table. Note that if a command is not tied to any hwpt or device, it still has to be broadcasted.
+ Pros for multiple vSMMU design
e) Emulated devices can be isolated from any nested SMMU.
f) Cache invalidation commands will always be forwarded to the corresponding pSMMU, reducing the overhead from vSMMU walking through a lookup table or broadcasting.
g) It will adapt CMDQV very easily.
- Cons for multiple vSMMU diesgn
h) Complications in VIRT and IORT design
i) Difficulty to support device hotplugs
j) Potential of running out of PCI bus number, as QEMU doesn't support multiple PCI domains.
[ How is it implemented with this series? ]
* As an experimental series, this is all done in VIRT and ACPI code.
* Scan iommu sysfs nodes and build an SMMU node list (PATCH-03).
* Create one PCIe Expander Bridge (+ one vSMMU) from the top of bus number (0xFF) with intervals for root-ports (PATCH-05). E.g. host system with three pSMMUs:
[ pcie.0 bus ]
-----------------------------------------------------------------------------
| | | |
----------------- ------------------ ------------------ ------------------
| emulated devs | | smmu_bridge.e5 | | smmu_bridge.ee | | smmu_bridge.f7 |
----------------- ------------------ ------------------ ------------------
* Loop vfio-pci devices against the SMMU node list and assign them
automatically in the VIRT code to the corresponding smmu_bridges,
and then attach them by creating a root port (PATCH-06):
[ pcie.0 bus ]
-----------------------------------------------------------------------------
| | | |
----------------- ------------------ ------------------ ------------------
| emulated devs | | smmu_bridge.e5 | | smmu_bridge.ee | | smmu_bridge.f7 |
----------------- ------------------ ------------------ ------------------
|
---------------- -----------
| root_port.ef |---| PCI dev |
---------------- -----------
* Set the "pcie.0" root bus to iommu bypass, so its entire ID space
will be directed to ITS in IORT. If a vfio-pci device chooses to
bypass 2-stage translation, it can be added to "pcie.0" (PATCH-07):
--------------build_iort: its_idmaps
build_iort_id_mapping: input_base=0x0, id_count=0xe4ff, out_ref=0x30
* Map IDs of smmu_bridges to corresponding vSMMUs (PATCH-09):
--------------build_iort: smmu_idmaps
build_iort_id_mapping: input_base=0xe500, id_count=0x8ff, out_ref=0x48
build_iort_id_mapping: input_base=0xee00, id_count=0x8ff, out_ref=0xa0
build_iort_id_mapping: input_base=0xf700, id_count=0x8ff, out_ref=0xf8 * Finally, "lspci -tv" in the guest looks like this:
-+-[0000:ee]---00.0-[ef]----00.0 [vfio-pci passthrough]
\-[0000:00]-+-00.0 Red Hat, Inc. QEMU PCIe Host bridge
+-01.0 Red Hat, Inc. QEMU PCIe Expander bridge +-02.0 Red Hat, Inc. QEMU PCIe Expander bridge +-03.0 Red Hat, Inc. QEMU PCIe Expander bridge +-04.0 Red Hat, Inc. QEMU NVM Express Controller [emulated] \-05.0 Intel Corporation 82540EM Gigabit Ethernet [emulated]
[ Topics for discussion ]
* Some of the bits can be moved to backends/iommufd.c, e.g. -object iommufd,id=iommufd0,[nesting=smmu3,[max-hotplugs=1]] And I was hoping that the vfio-pci device could take the iommufd BE pointer so it can redirect the PCI bus. Yet, seems to be more complicated than I thought...
* Possiblity of adding nesting support for vfio-pci-nohotplug only? The kernel uAPI (even for nesting cap detection) requires a dev handler. If a VM boots without a vfio-pci and then gets a hotplug after boot-to-console, a vSMMU that has already finished a reset cycle will need to sync the idr/idrr bits and will have to reset again?
This series is on Github:
https://github.com/nicolinc/qemu/commits/iommufd_multi_vsmmu-rfcv1
Thanks!
Nicolin
Eric Auger (1):
hw/arm/virt-acpi-build: Add IORT RMR regions to handle MSI nested
binding
Nicolin Chen (9):
hw/arm/virt: Add iommufd link to virt-machine hw/arm/virt: Get the number of host-level SMMUv3 instances hw/arm/virt: Add an SMMU_IO_LEN macro
hw/arm/virt: Add VIRT_NESTED_SMMU
hw/arm/virt: Assign vfio-pci devices to nested SMMUs hw/arm/virt: Bypass iommu for default PCI bus hw/arm/virt-acpi-build: Handle reserved bus number of pxb buses hw/arm/virt-acpi-build: Build IORT with multiple SMMU nodes hw/arm/virt-acpi-build: Enable ATS for nested SMMUv3
hw/arm/virt-acpi-build.c | 144 ++++++++++++++++---- hw/arm/virt.c | 277 +++++++++++++++++++++++++++++++++++++-- include/hw/arm/virt.h | 63 +++++++++ 3 files changed, 449 insertions(+), 35 deletions(-)
Signature · 1 line
2.43.0
RFC PATCH 1/10 hw/arm/virt-acpi-build: Add IORT RMR regions to handle MSI nested binding
From: Eric Auger <[email protected]>
To handle SMMUv3 nested stage support it is practical to expose the guest with reserved memory regions (RMRs) covering the IOVAs used by the host kernel to map physical MSI doorbells.
Those IOVAs belong to [0x8000000, 0x8100000] matching MSI_IOVA_BASE and MSI_IOVA_LENGTH definitions in kernel arm-smmu-v3 driver. This is the window used to allocate IOVAs matching physical MSI doorbells.
With those RMRs, the guest is forced to use a flat mapping for this range. Hence the assigned device is programmed with one IOVA from this range. Stage 1, owned by the guest has a flat mapping for this IOVA. Stage2, owned by the VMM then enforces a mapping from this IOVA to the physical MSI doorbell.
The creation of those RMR nodes only is relevant if nested stage SMMU is in use, along with VFIO. As VFIO devices can be hotplugged, all RMRs need to be created in advance. Hence the patch introduces a new arm virt "nested-smmuv3" iommu type.
ARM DEN 0049E.b IORT specification also mandates that when RMRs are present, the OS must preserve PCIe configuration performed by the boot FW. So along with the RMR IORT nodes, a _DSM function #5, as defined by PCI FIRMWARE SPECIFICATION EVISION 3.3, chapter 4.6.5 is added to PCIe host bridge and PCIe expander bridge objects.
---
v2 -> v3:
- Comply with IORT E.d spec. RMR node rev = 3; IORT rev = 5 With this spec revision, the restriction on number of Stream IDs that can be associated with memory ranges in an RMR node was removed. So no need anymore to define 1 RMR node per SID!
--- hw/arm/virt-acpi-build.c | 84 +++++++++++++++++++++++++++++++++------- hw/arm/virt.c | 7 +++- include/hw/arm/virt.h | 7 ++++ 3 files changed, 84 insertions(+), 14 deletions(-)
hw/arm/virt-acpi-build.c +71 -13
@@ -132,6 +132,14 @@ static void acpi_dsdt_add_pci(Aml *scope, const MemMapEntry *memmap, .bus = vms->bus, }; +/* +* Nested SMMU requires RMRs for MSI 1-1 mapping, which +* require _DSM for PreservingPCI Boot Configurations +*/ +if (vms->iommu ==VIRT_IOMMU_NESTED_SMMUV3 ) { + cfg.preserve_config = true; + } +if (vms->highmem_mmio) { cfg.mmio64 = memmap[VIRT_HIGH_PCIE_MMIO ]; }
@@ -216,16 +224,16 @@ static void acpi_dsdt_add_tpm(Aml *scope, VirtMachineState *vms)* * Note that @id_count gets internally subtracted by one, following the spec. */ -static void build_iort_id_mapping (GArray *table_data,uint32_t input_base, -uint32_t id_count,uint32_t out_ref) +static void +build_iort_id_mapping (GArray *table_data,uint32_t input_base, +uint32_t id_count,uint32_t out_ref,uint32_t flags) {build_append_int_noprefix (table_data, input_base,4 );/* Input base */ /* Number of IDs - The number of IDs in the range minus one */ build_append_int_noprefix (table_data, id_count -1 ,4 );build_append_int_noprefix (table_data, input_base,4 );/* Output base */ build_append_int_noprefix (table_data, out_ref,4 );/* Output Reference */ -/* Flags */ -build_append_int_noprefix (table_data,0 /* Single mapping (disabled) */ ,4 ); +build_append_int_noprefix (table_data, flags,4 );/* Flags */ }struct AcpiIortIdMapping {
@@ -267,6 +275,48 @@ static int iort_idmap_compare(gconstpointer a, gconstpointer b)return idmap_a->input_base - idmap_b->input_base; } +static void +build_iort_rmr_nodes (GArray *table_data,GArray *smmu_idmaps,int smmu_offset,uint32_t *id) { +AcpiIortIdMapping *range; +int i; + +for (i =0 ; i < smmu_idmaps->len; i++) { + range = &g_array_index (smmu_idmaps, AcpiIortIdMapping, i); +int bdf = range->input_base; + +/* Table 18 Reserved Memory Range Node */ + +build_append_int_noprefix (table_data,6 /* RMR */ ,1 );/* Type */ +/* Length */ +build_append_int_noprefix (table_data,28 +ID_MAPPING_ENTRY_SIZE +20 ,2 ); +build_append_int_noprefix (table_data,3 ,1 );/* Revision */ +build_append_int_noprefix (table_data, *id,4 );/* Identifier */ +/* Number of ID mappings */ +build_append_int_noprefix (table_data,1 ,4 ); +/* Reference to ID Array */ +build_append_int_noprefix (table_data,28 ,4 ); + +/* RMR specific data */ + +/* Flags */ +build_append_int_noprefix (table_data,0 /* Disallow remapping */ ,4 ); +/* Number of Memory Range Descriptors */ +build_append_int_noprefix (table_data,1 ,4 ); +/* Reference to Memory Range Descriptors */ +build_append_int_noprefix (table_data,28 +ID_MAPPING_ENTRY_SIZE ,4 ); +build_iort_id_mapping (table_data, bdf, range->id_count, smmu_offset,1 ); + +/* Table 19 Memory Range Descriptor */ + +/* Physical Range offset */ +build_append_int_noprefix (table_data,0x8000000 ,8 ); +/* Physical Range length */ +build_append_int_noprefix (table_data,0x100000 ,8 ); +build_append_int_noprefix (table_data,0 ,4 );/* Reserved */ + *id +=1 ; + } +} +/* * Input Output Remapping Table (IORT) * Conforms to "IO Remapping Table System Software on ARM Platforms",
@@ -282,17 +332,19 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)GArray *smmu_idmaps =g_array_new (false, true,sizeof (AcpiIortIdMapping));GArray *its_idmaps =g_array_new (false, true,sizeof (AcpiIortIdMapping)); -AcpiTable table = { .sig ="IORT" , .rev =3 , .oem_id = vms->oem_id, +AcpiTable table = { .sig ="IORT" , .rev =5 , .oem_id = vms->oem_id, .oem_table_id = vms->oem_table_id };/* Table 2 The IORT */ acpi_table_begin (&table, table_data); -if (vms->iommu ==VIRT_IOMMU_SMMUV3 ) { +if (virt_has_smmuv3 (vms)) {AcpiIortIdMapping next_range = {0 };object_child_foreach_recursive (object_get_root (), iort_host_bridges, smmu_idmaps); + nb_nodes =3 ;/* RC, ITS, SMMUv3 */ +/* Sort the smmu idmap by input_base */ g_array_sort (smmu_idmaps, iort_idmap_compare);
@@ -309,6 +361,9 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) } next_range.input_base = idmap->input_base + idmap->id_count; +if (vms->iommu ==VIRT_IOMMU_NESTED_SMMUV3 ) { + nb_nodes++; + } }/* Append the last RC -> ITS ID mapping */
@@ -317,7 +372,6 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)g_array_append_val (its_idmaps, next_range); } - nb_nodes =3 ;/* RC, ITS, SMMUv3 */ rc_mapping_count = smmu_idmaps->len + its_idmaps->len; }else { nb_nodes =2 ;/* RC, ITS */
@@ -342,7 +396,7 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)/* GIC ITS Identifier Array */ build_append_int_noprefix (table_data,0 /* MADT translation_id */ ,4 ); -if (vms->iommu ==VIRT_IOMMU_SMMUV3 ) { +if (virt_has_smmuv3 (vms)) {int irq = vms->irqmap[VIRT_SMMU ] +ARM_SPI_BASE ; smmu_offset = table_data->len - table.table_offset;
@@ -372,7 +426,7 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)build_append_int_noprefix (table_data,0 ,4 );/* output IORT node is the ITS group node (the first node) */ -build_iort_id_mapping (table_data,0 ,0x10000 ,IORT_NODE_OFFSET ); +build_iort_id_mapping (table_data,0 ,0x10000 ,IORT_NODE_OFFSET ,0 ); }/* Table 17 Root Complex Node */
@@ -405,7 +459,7 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)build_append_int_noprefix (table_data,0 ,3 );/* Reserved */ /* Output Reference */ -if (vms->iommu ==VIRT_IOMMU_SMMUV3 ) { +if (virt_has_smmuv3 (vms)) {AcpiIortIdMapping *range;/* translated RIDs connect to SMMUv3 node: RC -> SMMUv3 -> ITS */
@@ -413,7 +467,7 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) range = &g_array_index (smmu_idmaps, AcpiIortIdMapping, i);/* output IORT node is the smmuv3 node */ build_iort_id_mapping (table_data, range->input_base, - range->id_count, smmu_offset); + range->id_count, smmu_offset,0 ); }/* bypassed RIDs connect to ITS group node directly: RC -> ITS */
@@ -421,11 +475,15 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) range = &g_array_index (its_idmaps, AcpiIortIdMapping, i);/* output IORT node is the ITS group node (the first node) */ build_iort_id_mapping (table_data, range->input_base, - range->id_count,IORT_NODE_OFFSET ); + range->id_count,IORT_NODE_OFFSET ,0 ); } }else {/* output IORT node is the ITS group node (the first node) */ -build_iort_id_mapping (table_data,0 ,0x10000 ,IORT_NODE_OFFSET ); +build_iort_id_mapping (table_data,0 ,0x10000 ,IORT_NODE_OFFSET ,0 ); + } + +if (vms->iommu ==VIRT_IOMMU_NESTED_SMMUV3 ) { +build_iort_rmr_nodes (table_data, smmu_idmaps, smmu_offset, &id); }acpi_table_end (linker, &table);
hw/arm/virt.c +6 -1
@@ -1396,7 +1396,7 @@ static void create_smmu(const VirtMachineState *vms,DeviceState *dev;MachineState *ms =MACHINE (vms); -if (vms->iommu !=VIRT_IOMMU_SMMUV3 || !vms->iommu_phandle) { +if (!virt_has_smmuv3 (vms) || !vms->iommu_phandle) {return ; }
@@ -1578,6 +1578,7 @@ static void create_pcie(VirtMachineState *vms)switch (vms->iommu) {case VIRT_IOMMU_SMMUV3 : +case VIRT_IOMMU_NESTED_SMMUV3 :create_smmu (vms, vms->bus);qemu_fdt_setprop_cells (ms->fdt, nodename,"iommu-map" ,0x0 , vms->iommu_phandle,0x0 ,0x10000 );
@@ -2653,6 +2654,8 @@ static char *virt_get_iommu(Object *obj, Error **errp)return g_strdup ("none" );case VIRT_IOMMU_SMMUV3 :return g_strdup ("smmuv3" ); +case VIRT_IOMMU_NESTED_SMMUV3 : +return g_strdup ("nested-smmuv3" );default :g_assert_not_reached (); }
@@ -2664,6 +2667,8 @@ static void virt_set_iommu(Object *obj, const char *value, Error **errp)if (!strcmp (value,"smmuv3" )) { vms->iommu =VIRT_IOMMU_SMMUV3 ; + }else if (!strcmp (value,"nested-smmuv3" )) { + vms->iommu =VIRT_IOMMU_NESTED_SMMUV3 ; }else if (!strcmp (value,"none" )) { vms->iommu =VIRT_IOMMU_NONE ; }else {
include/hw/arm/virt.h +7 -0
@@ -89,6 +89,7 @@ enum {typedef enum VirtIOMMUType {VIRT_IOMMU_NONE ,VIRT_IOMMU_SMMUV3 , +VIRT_IOMMU_NESTED_SMMUV3 ,VIRT_IOMMU_VIRTIO , }VirtIOMMUType ;
@@ -209,4 +210,10 @@ static inline int virt_gicv3_redist_region_count(VirtMachineState *vms) vms->highmem_redists) ? 2 : 1; } +static inline bool virt_has_smmuv3 (const VirtMachineState *vms) +{ +return vms->iommu ==VIRT_IOMMU_SMMUV3 || + vms->iommu ==VIRT_IOMMU_NESTED_SMMUV3 ; +} +#endif /* QEMU_ARM_VIRT_H */
Signature · 1 line
2.43.0
RFC PATCH 2/10 hw/arm/virt: Add iommufd link to virt-machine
In reply to Nicolin ChenA nested SMMU must use iommufd ioctls to communicate with the host-level SMMU instance for 2-stage translation support. Add an iommufd link to the ARM virt-machine, allowing QEMU command to pass in an iommufd object.
--- hw/arm/virt.c | 14 ++++++++++++++ include/hw/arm/virt.h | 2 ++ 2 files changed, 16 insertions(+)
hw/arm/virt.c +14 -0
@@ -1404,6 +1404,13 @@ static void create_smmu(const VirtMachineState *vms,object_property_set_link (OBJECT (dev),"primary-bus" ,OBJECT (bus), &error_abort); + +if (vms->iommu ==VIRT_IOMMU_NESTED_SMMUV3 ) { +g_assert (vms->iommufd); +object_property_set_link (OBJECT (dev),"iommufd" ,OBJECT (vms->iommufd), + &error_abort); +object_property_set_bool (OBJECT (dev),"nested" , true, &error_abort); + }sysbus_realize_and_unref (SYS_BUS_DEVICE (dev), &error_fatal);sysbus_mmio_map (SYS_BUS_DEVICE (dev),0 , base);for (i =0 ; i <NUM_SMMU_IRQS ; i++) {
@@ -3114,6 +3121,13 @@ static void virt_machine_class_init(ObjectClass *oc, void *data)"Set GIC version. " "Valid values are 2, 3, 4, host and max"); +object_class_property_add_link (oc,"iommufd" ,TYPE_IOMMUFD_BACKEND , + offsetof(VirtMachineState , iommufd), + object_property_allow_set_link, +OBJ_PROP_LINK_STRONG ); +object_class_property_set_description (oc,"iommufd" , +"Set the IOMMUFD handler from \"-iommufd\"" ); +object_class_property_add_str (oc,"iommu" , virt_get_iommu, virt_set_iommu);object_class_property_set_description (oc,"iommu" , "Set the IOMMU type. "
include/hw/arm/virt.h +2 -0
@@ -36,6 +36,7 @@#include "hw/arm/boot.h" #include "hw/arm/bsa.h" #include "hw/block/flash.h" +#include "sysemu/iommufd.h" #include "sysemu/kvm.h" #include "hw/intc/arm_gicv3_common.h" #include "qom/object.h"
@@ -154,6 +155,7 @@ struct VirtMachineState {bool dtb_randomness;OnOffAuto acpi;VirtGICType gic_version; +IOMMUFDBackend *iommufd;VirtIOMMUType iommu;bool default_bus_bypass_iommu;VirtMSIControllerType msi_controller;
Signature · 1 line
2.43.0
Re: [PATCH RFCv1 02/10] hw/arm/virt: Add iommufd link to virt-machine
Hi Nicolin,
Quoted 3 lines · Nicolin Chen
A nested SMMU must use iommufd ioctls to communicate with the host-level SMMU instance for 2-stage translation support. Add an iommufd link to the ARM virt-machine, allowing QEMU command to pass in an iommufd object.
If I am not wrong vfio devices are allowed to use different iommufd's (although there is no real benefice). So this command line wouldn't match with that option.
Also while reading the commit msg it is not clear with the iommufd is needed in the machine whereas the vfio iommufd BE generally calls those ioctls.
Thanks
Eric
Quoted 60 lines
Signed-off-by Nicolin Chen <[email protected]>--- hw/arm/virt.c | 14 ++++++++++++++ include/hw/arm/virt.h | 2 ++ 2 files changed, 16 insertions(+)hw/arm/virt.c +7
-0@@ -1404,6 +1404,13 @@ static void create_smmu(const VirtMachineState *vms,object_property_set_link (OBJECT (dev),"primary-bus" ,OBJECT (bus), &error_abort); + +if (vms->iommu ==VIRT_IOMMU_NESTED_SMMUV3 ) { +g_assert (vms->iommufd); +object_property_set_link (OBJECT (dev),"iommufd" ,OBJECT (vms->iommufd), + &error_abort); +object_property_set_bool (OBJECT (dev),"nested" , true, &error_abort); + }sysbus_realize_and_unref (SYS_BUS_DEVICE (dev), &error_fatal);sysbus_mmio_map (SYS_BUS_DEVICE (dev),0 , base);for (i = 0; i < NUM_SMMU_IRQS; i++) {hw/arm/virt.c +7
-0@@ -3114,6 +3121,13 @@ static void virt_machine_class_init(ObjectClass *oc, void *data)"Set GIC version. " "Valid values are 2, 3, 4, host and max"); +object_class_property_add_link (oc,"iommufd" ,TYPE_IOMMUFD_BACKEND , + offsetof(VirtMachineState , iommufd), + object_property_allow_set_link, +OBJ_PROP_LINK_STRONG ); +object_class_property_set_description (oc,"iommufd" , +"Set the IOMMUFD handler from \"-iommufd\"" ); +object_class_property_add_str (oc,"iommu" , virt_get_iommu, virt_set_iommu);object_class_property_set_description (oc,"iommu" , "Set the IOMMU type. "include/hw/arm/virt.h +2
-0@@ -36,6 +36,7 @@#include "hw/arm/boot.h" #include "hw/arm/bsa.h" #include "hw/block/flash.h" +#include "sysemu/iommufd.h" #include "sysemu/kvm.h" #include "hw/intc/arm_gicv3_common.h" #include "qom/object.h" @@ -154,6 +155,7 @@ struct VirtMachineState {bool dtb_randomness;OnOffAuto acpi;VirtGICType gic_version; +IOMMUFDBackend *iommufd;VirtIOMMUType iommu;bool default_bus_bypass_iommu;VirtMSIControllerType msi_controller;
Re: [PATCH RFCv1 02/10] hw/arm/virt: Add iommufd link to virt-machine
Hi Eric,
Thanks for the comments!
Quoted 8 lines · Eric Auger
Quoted 3 lines · Nicolin Chen
A nested SMMU must use iommufd ioctls to communicate with the host-level SMMU instance for 2-stage translation support. Add an iommufd link to the ARM virt-machine, allowing QEMU command to pass in an iommufd object.
If I am not wrong vfio devices are allowed to use different iommufd's (although there is no real benefice). So this command line wouldn't match with that option.
I think Jason's remarks highlighted that FD should be one per VM:
https://lore.kernel.org/qemu-devel/[email protected]/
Quoted 3 lines
Also while reading the commit msg it is not clear with the iommufd is needed in the machine whereas the vfio iommufd BE generally calls those ioctls.
I think I forgot to revisit it. Both intel_iommu and smmu-common used to call iommufd_backend_connect() for counting, so there was a need to pass in the same iommufd handler to the viommu driver. For SMMU, since it is created in the virt code, we had to pass in with this patch.
That being said, it looks like intel_iommu had removed that. So, likely we don't need an extra user counting for SMMU too.
Thank you
Nicolin
Re: [PATCH RFCv1 02/10] hw/arm/virt: Add iommufd link to virt-machine
Quoted 6 lines · Nicolin Chen
Hi Eric,
Thanks for the comments!
Quoted 1 line · Eric Auger
On 6/26/24 02:28, Nicolin Chen wrote:
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
A nested SMMU must use iommufd ioctls to communicate with the host-level SMMU instance for 2-stage translation support. Add an iommufd link to the ARM virt-machine, allowing QEMU command to pass in an iommufd object.
If I am not wrong vfio devices are allowed to use different iommufd's (although there is no real benefice). So this command line wouldn't match with that option.
I think Jason's remarks highlighted that FD should be one per VM:
https://lore.kernel.org/qemu-devel/[email protected]/
OK I thought this was still envisionned althought not really meaningful. By the way, please add Yi and Zhenzhong in cc since thre problematics are connected I think.
Quoted 4 lines
Quoted 4 lines
Also while reading the commit msg it is not clear with the iommufd is needed in the machine whereas the vfio iommufd BE generally calls those ioctls.
Quoted 8 lines
I think I forgot to revisit it. Both
intel_iommuand smmu-common used to calliommufd_backend_connect()for counting, so there was a need to pass in the same iommufd handler to the viommu driver. For SMMU, since it is created in the virt code, we had to pass in with this patch.That being said, it looks like
intel_iommuhad removed that. So, likely we don't need an extra user counting for SMMU too.
OK at least it deserves some explanation about the "why"
Thanks
Eric
Quoted 4 lines
Thank you
Nicolin
Re: [PATCH RFCv1 02/10] hw/arm/virt: Add iommufd link to virt-machine
Quoted 10 lines · Eric Auger
Quoted 9 lines · Nicolin Chen
Hi Eric,
Thanks for the comments!
Quoted 4 lines · Eric Auger
Quoted 3 lines · Nicolin Chen
A nested SMMU must use iommufd ioctls to communicate with the host-level SMMU instance for 2-stage translation support. Add an iommufd link to the ARM virt-machine, allowing QEMU command to pass in an iommufd object.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
If I am not wrong vfio devices are allowed to use different iommufd's (although there is no real benefice). So this command line wouldn't match with that option.
I think Jason's remarks highlighted that FD should be one per VM:
https://lore.kernel.org/qemu-devel/[email protected]/OK I thought this was still envisionned althought not really meaningful. By the way, please add Yi and Zhenzhong in cc since thre problematics are connected I think.
Yea.
Yi/Zhenzhong, would you please shed some light on forwarding an iommufd handler to the intel_iommu code? IIRC, we did that at the beginning but removed it later?
Quoted 9 lines
Quoted 9 lines
Quoted 3 lines
Also while reading the commit msg it is not clear with the iommufd is needed in the machine whereas the vfio iommufd BE generally calls those ioctls.
I think I forgot to revisit it. Both
intel_iommuand smmu-common used to calliommufd_backend_connect()for counting, so there was a need to pass in the same iommufd handler to the viommu driver. For SMMU, since it is created in the virt code, we had to pass in with this patch.
Quoted 3 lines
Quoted 2 lines
That being said, it looks like
intel_iommuhad removed that. So, likely we don't need an extra user counting for SMMU too.OK at least it deserves some explanation about the "why"
Yes, I agree that the commit message isn't good enough.
Thanks
Nicolin
RE: [PATCH RFCv1 02/10] hw/arm/virt: Add iommufd link to virt-machine
Hi Nicolin,
Quoted 29 lines
-----Original Message-----
From: Nicolin Chen <[email protected]>
Subject: Re: [PATCH RFCv1 02/10] hw/arm/virt: Add iommufd link to virt- machineQuoted 8 lines · Eric Auger
Quoted 7 lines · Nicolin Chen
Hi Eric,
Thanks for the comments!
Quoted 2 lines · Eric Auger
Quoted 1 line · Nicolin Chen
A nested SMMU must use iommufd ioctls to communicate with the
host-level
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
SMMU instance for 2-stage translation support. Add an iommufd link
to the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
ARM virt-machine, allowing QEMU command to pass in an iommufd
object.
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
If I am not wrong vfio devices are allowed to use different iommufd's (although there is no real benefice). So this command line wouldn't match with that option.
I think Jason's remarks highlighted that FD should be one per VM:
https://lore.kernel.org/qemu-devel/[email protected]/
Quoted 3 lines
OK I thought this was still envisionned althought not really meaningful. By the way, please add Yi and Zhenzhong in cc since thre problematics are connected I think.
Quoted 5 lines
Yea.
Yi/Zhenzhong, would you please shed some light on forwarding an iommufd handler to the
intel_iommucode? IIRC, we did that at the beginning but removed it later?
IOMMUFD/devid/ioas handler is packaged in HostIOMMUDeviceIOMMUFD and passed to vIOMMU, see https://github.com/yiliu1765/qemu/commit/02892a5b452382866e804c3db3bb392c8f8f500f
The whole nesting series is at https://github.com/yiliu1765/qemu/commits/zhenzhong/iommufd_nesting_rfcv2/
We gave the user flexibility to use different iommufd backends in one VM in iommufd cdev series. We want to be backward compatible in nesting series, the code change to support that is also trivial.
Thanks
Zhenzhong
Quoted 19 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Also while reading the commit msg it is not clear with the iommufd is needed in the machine whereas the vfio iommufd BE generally calls
those
Quoted 10 lines
Quoted 9 lines
Quoted 1 line
ioctls.
I think I forgot to revisit it. Both
intel_iommuand smmu-common used to calliommufd_backend_connect()for counting, so there was a need to pass in the same iommufd handler to the viommu driver. For SMMU, since it is created in the virt code, we had to pass in with this patch.That being said, it looks like
intel_iommuhad removed that. So, likely we don't need an extra user counting for SMMU too.OK at least it deserves some explanation about the "why"
Yes, I agree that the commit message isn't good enough.
Thanks
Nicolin
RFC PATCH 3/10 hw/arm/virt: Get the number of host-level SMMUv3 instances
In reply to Nicolin ChenNested SMMUv3 feature requires the support/presence of host-level SMMUv3 instance(s). Add a helper to read the sysfs for the number of instances. Log them in a vms list using a new struct VirtNestedSmmu.
This will be used by a following patch to assign a passthrough device to corresponding nested SMMUv3 instance.
--- hw/arm/virt.c | 35 +++++++++++++++++++++++++++++++++++ include/hw/arm/virt.h | 8 ++++++++ 2 files changed, 43 insertions(+)
hw/arm/virt.c +35 -0
@@ -2668,6 +2668,37 @@ static char *virt_get_iommu(Object *obj, Error **errp) } } +static int virt_get_num_nested_smmus (VirtMachineState *vms,Error **errp) +{ +VirtNestedSmmu *nested_smmu; +struct dirent *dent; +DIR *dir =NULL ; +int num =0 ; + + dir =opendir ("/sys/class/iommu" ); +if (!dir) { +error_setg_errno (errp, errno,"couldn't open /sys/class/iommu" ); +return 0 ; + } + +while ((dent =readdir (dir))) { +if (!strncmp (dent->d_name,"smmu3.0x" ,7 )) { + nested_smmu =g_new0 (VirtNestedSmmu,1 ); + nested_smmu->index = num; + nested_smmu->smmu_node =g_strdup (dent->d_name); +QLIST_INSERT_HEAD (&vms->nested_smmu_list, nested_smmu, next); + num++; + } + } + +if (num ==0 ) { +error_setg_errno (errp, errno, +"couldn't find any SMMUv3 HW to setup nesting" ); + } + +return num; +} +static void virt_set_iommu (Object *obj,const char *value,Error **errp) {VirtMachineState *vms =VIRT_MACHINE (obj);
@@ -2676,6 +2707,7 @@ static void virt_set_iommu(Object *obj, const char *value, Error **errp) vms->iommu =VIRT_IOMMU_SMMUV3 ; }else if (!strcmp (value,"nested-smmuv3" )) { vms->iommu =VIRT_IOMMU_NESTED_SMMUV3 ; + vms->num_nested_smmus =virt_get_num_nested_smmus (vms, errp); }else if (!strcmp (value,"none" )) { vms->iommu =VIRT_IOMMU_NONE ; }else {
@@ -3232,6 +3264,9 @@ static void virt_instance_init(Object *obj)/* The default root bus is attached to iommu by default */ vms->default_bus_bypass_iommu = false; +/* Default disallows nested SMMU instantiation */ + vms->num_nested_smmus =0 ; +/* Default disallows RAS instantiation */ vms->ras = false;
include/hw/arm/virt.h +8 -0
@@ -135,6 +135,12 @@ struct VirtMachineClass {bool no_ns_el2_virt_timer_irq; }; +typedef struct VirtNestedSmmu { +int index; +char *smmu_node; +QLIST_ENTRY (VirtNestedSmmu) next; +}VirtNestedSmmu ; +struct VirtMachineState {MachineState parent;Notifier machine_done;
@@ -153,6 +159,7 @@ struct VirtMachineState {bool ras;bool mte;bool dtb_randomness; +int num_nested_smmus;OnOffAuto acpi;VirtGICType gic_version;IOMMUFDBackend *iommufd;
@@ -178,6 +185,7 @@ struct VirtMachineState {char *oem_id;char *oem_table_id;bool ns_el2_virt_timer_irq; + QLIST_HEAD(, VirtNestedSmmu) nested_smmu_list; };#define VIRT_ECAM_ID (high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
Signature · 1 line
2.43.0
Re: [PATCH RFCv1 03/10] hw/arm/virt: Get the number of host-level SMMUv3 instances
Hi Nicolin,
Quoted 6 lines · Nicolin Chen
Nested SMMUv3 feature requires the support/presence of host-level SMMUv3 instance(s). Add a helper to read the sysfs for the number of instances. Log them in a vms list using a new struct VirtNestedSmmu.
This will be used by a following patch to assign a passthrough device to corresponding nested SMMUv3 instance.
Laterly the HostIOMMUDevice has been introduced to allow, among other things, to pass information related to the physical IOMMU to the virtual IOMMU.
I guess it would be well fitted to associate the viommu with its underlying piommu.
I don't think we have such kind of host introspection in machine type. Generally in can happen in the very device or in libvirt.
Thanks
Eric
Quoted 100 lines
Signed-off-by Nicolin Chen <[email protected]>--- hw/arm/virt.c | 35 +++++++++++++++++++++++++++++++++++ include/hw/arm/virt.h | 8 ++++++++ 2 files changed, 43 insertions(+)hw/arm/virt.c +35
-0@@ -2668,6 +2668,37 @@ static char *virt_get_iommu(Object *obj, Error **errp) } } +static int virt_get_num_nested_smmus (VirtMachineState *vms,Error **errp) +{ +VirtNestedSmmu *nested_smmu; +struct dirent *dent; +DIR *dir =NULL ; +int num =0 ; + + dir =opendir ("/sys/class/iommu" ); +if (!dir) { +error_setg_errno (errp, errno,"couldn't open /sys/class/iommu" ); +return 0 ; + } + +while ((dent =readdir (dir))) { +if (!strncmp (dent->d_name,"smmu3.0x" ,7 )) { + nested_smmu =g_new0 (VirtNestedSmmu,1 ); + nested_smmu->index = num; + nested_smmu->smmu_node =g_strdup (dent->d_name); +QLIST_INSERT_HEAD (&vms->nested_smmu_list, nested_smmu, next); + num++; + } + } + +if (num ==0 ) { +error_setg_errno (errp, errno, +"couldn't find any SMMUv3 HW to setup nesting" ); + } + +return num; +} +static void virt_set_iommu (Object *obj,const char *value,Error **errp) {VirtMachineState *vms =VIRT_MACHINE (obj);@@ -2676,6 +2707,7 @@ static void virt_set_iommu(Object *obj, const char *value, Error **errp) vms->iommu =VIRT_IOMMU_SMMUV3 ; }else if (!strcmp (value,"nested-smmuv3" )) { vms->iommu =VIRT_IOMMU_NESTED_SMMUV3 ; + vms->num_nested_smmus =virt_get_num_nested_smmus (vms, errp); }else if (!strcmp (value,"none" )) { vms->iommu =VIRT_IOMMU_NONE ; }else {@@ -3232,6 +3264,9 @@ static void virt_instance_init(Object *obj)/* The default root bus is attached to iommu by default */ vms->default_bus_bypass_iommu = false; +/* Default disallows nested SMMU instantiation */ + vms->num_nested_smmus =0 ; +/* Default disallows RAS instantiation */ vms->ras = false;include/hw/arm/virt.h +8
-0@@ -135,6 +135,12 @@ struct VirtMachineClass {bool no_ns_el2_virt_timer_irq; }; +typedef struct VirtNestedSmmu { +int index; +char *smmu_node; +QLIST_ENTRY (VirtNestedSmmu) next; +}VirtNestedSmmu ; +struct VirtMachineState {MachineState parent;Notifier machine_done;@@ -153,6 +159,7 @@ struct VirtMachineState {bool ras;bool mte;bool dtb_randomness; +int num_nested_smmus;OnOffAuto acpi;VirtGICType gic_version;IOMMUFDBackend *iommufd;@@ -178,6 +185,7 @@ struct VirtMachineState {char *oem_id;char *oem_table_id;bool ns_el2_virt_timer_irq; + QLIST_HEAD(, VirtNestedSmmu) nested_smmu_list; };#define VIRT_ECAM_ID (high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
Re: [PATCH RFCv1 03/10] hw/arm/virt: Get the number of host-level SMMUv3 instances
Quoted 5 lines · Eric Auger
Quoted 4 lines · Nicolin Chen
Nested SMMUv3 feature requires the support/presence of host-level SMMUv3 instance(s). Add a helper to read the sysfs for the number of instances. Log them in a vms list using a new struct VirtNestedSmmu.
Quoted 8 lines
Quoted 2 lines
This will be used by a following patch to assign a passthrough device to corresponding nested SMMUv3 instance.
Laterly the HostIOMMUDevice has been introduced to allow, among other things, to pass information related to the physical IOMMU to the virtual IOMMU.
I guess it would be well fitted to associate the viommu with its underlying piommu.
Wow, I missed that part -- backends/host_iommu_device. I will see how I can fit these well with that.
Quoted 2 lines
I don't think we have such kind of host introspection in machine type. Generally in can happen in the very device or in libvirt.
I think the biggest reason for having such an introspection in the virt code is because of hotplug, (though it's not properly implemented yet), as we don't know what new devices requiring for nested translation would be joining later. So somebody has to hold a full list.
Would you mind elaborating how the "device" or "libvirt" can handle that?
Thanks!
Nicolin
Re: [PATCH RFCv1 03/10] hw/arm/virt: Get the number of host-level SMMUv3 instances
Quoted 18 lines · Nicolin Chen
Quoted 5 lines · Eric Auger
Quoted 4 lines · Nicolin Chen
Nested SMMUv3 feature requires the support/presence of host-level SMMUv3 instance(s). Add a helper to read the sysfs for the number of instances. Log them in a vms list using a new struct VirtNestedSmmu.
Quoted 7 lines
Quoted 2 lines
This will be used by a following patch to assign a passthrough device to corresponding nested SMMUv3 instance.
Laterly the HostIOMMUDevice has been introduced to allow, among other things, to pass information related to the physical IOMMU to the virtual IOMMU.
I guess it would be well fitted to associate the viommu with its underlying piommu.Wow, I missed that part -- backends/host_iommu_device. I will see how I can fit these well with that.
Quoted 2 lines
I don't think we have such kind of host introspection in machine type. Generally in can happen in the very device or in libvirt.
Quoted 8 lines
I think the biggest reason for having such an introspection in the virt code is because of hotplug, (though it's not properly implemented yet), as we don't know what new devices requiring for nested translation would be joining later. So somebody has to hold a full list.
Would you mind elaborating how the "device" or "libvirt" can handle that?
If you know that on Grace you have 5 SMMU instances, can't you pre-build a PCIe topology with 5 PXB and root ports at libvirt level. Then when you hotplug your device you specify the corresponding slot just as we do normally. But maybe I misunderstood the hotplug problematics.
Eric
Quoted 4 lines
Thanks!
Nicolin
Re: [PATCH RFCv1 03/10] hw/arm/virt: Get the number of host-level SMMUv3 instances
Quoted 25 lines · Eric Auger
Quoted 24 lines · Nicolin Chen
Quoted 5 lines · Eric Auger
Quoted 4 lines · Nicolin Chen
Nested SMMUv3 feature requires the support/presence of host-level SMMUv3 instance(s). Add a helper to read the sysfs for the number of instances. Log them in a vms list using a new struct VirtNestedSmmu.
Quoted 7 lines
Quoted 2 lines
This will be used by a following patch to assign a passthrough device to corresponding nested SMMUv3 instance.
Laterly the HostIOMMUDevice has been introduced to allow, among other things, to pass information related to the physical IOMMU to the virtual IOMMU.
I guess it would be well fitted to associate the viommu with its underlying piommu.Wow, I missed that part -- backends/host_iommu_device. I will see how I can fit these well with that.
Quoted 2 lines
I don't think we have such kind of host introspection in machine type. Generally in can happen in the very device or in libvirt.
I think the biggest reason for having such an introspection in the virt code is because of hotplug, (though it's not properly implemented yet), as we don't know what new devices requiring for nested translation would be joining later. So somebody has to hold a full list.
Quoted 6 lines
Quoted 2 lines
Would you mind elaborating how the "device" or "libvirt" can handle that?
If you know that on Grace you have 5 SMMU instances, can't you pre-build a PCIe topology with 5 PXB and root ports at libvirt level. Then when you hotplug your device you specify the corresponding slot just as we do normally. But maybe I misunderstood the hotplug problematics.
I guess I got your point: basically, the introspection and sysfs node matching for device assigning should happen in libvirt.
Thanks
Nicolin
RFC PATCH 4/10 hw/arm/virt: Add an SMMU_IO_LEN macro
In reply to Nicolin ChenA following patch will add a new MMIO region for nested SMMU instances.
This macro will be repeatedly used to set offsets and MMIO sizes in both virt and virt-acpi-build.
--- hw/arm/virt.c | 2 +- include/hw/arm/virt.h | 3 +++ 2 files changed, 4 insertions(+), 1 deletion(-)
hw/arm/virt.c +1 -1
@@ -170,7 +170,7 @@ static const MemMapEntry base_memmap[] = { [VIRT_FW_CFG] = { 0x09020000, 0x00000018 }, [VIRT_GPIO] = { 0x09030000, 0x00001000 }, [VIRT_SECURE_UART] = { 0x09040000, 0x00001000 }, - [VIRT_SMMU] = { 0x09050000, 0x00020000 }, + [VIRT_SMMU] = { 0x09050000, SMMU_IO_LEN }, [VIRT_PCDIMM_ACPI] = { 0x09070000, MEMORY_HOTPLUG_IO_LEN }, [VIRT_ACPI_GED] = { 0x09080000, ACPI_GED_EVT_SEL_LEN }, [VIRT_NVDIMM_ACPI] = { 0x09090000, NVDIMM_ACPI_IO_LEN},
include/hw/arm/virt.h +3 -0
@@ -48,6 +48,9 @@/* See Linux kernel arch/arm64/include/asm/pvclock-abi.h */ #define PVTIME_SIZE_PER_CPU 64 +/* MMIO region size for SMMUv3 */ +#define SMMU_IO_LEN (0x20000) +enum {VIRT_FLASH ,VIRT_MEM ,
Signature · 1 line
2.43.0
RFC PATCH 5/10 hw/arm/virt: Add VIRT_NESTED_SMMU
In reply to Nicolin ChenVIRT_SMMU can be used as an emulated SMMU, i.e. iommu=smmuv3. However, the MMIO space for VIRT_SMMU isn't large enough to accommodate multiple nested SMMUv3 instances. Add a new VIRT_NESTED_SMMU to separate MMIO space.
Meanwhile, a nested SMMUv3 could only work with a vfio-pci device that is physically associated to a host-level SMMUv3 instance, meaning that such a passthrough deivce must be linked to a nesteed SMMU corresponding to the underlying host-level SMMUv3 instance:
HW SMMU0 <-----------------> vSMMU0 (nested)
^ ^
| |
----> PCI dev0 & PCI dev1 <----
HW SMMU1 <-----------------> vSMMU1 (nested)
^ ^
| |
----> PCI dev2 & PCI dev3 <----Note that, on the other hand, an emulated deivce should not be associated to a nested SMMU.
To prepare for that, create a PCIe Expander Bridge per nested SMMU as a docker for pci-vfio devices. It eases the QEMU code to build ID mappings in the IORT.
--- hw/arm/virt.c | 110 +++++++++++++++++++++++++++++++++++++----- include/hw/arm/virt.h | 17 +++++++ 2 files changed, 116 insertions(+), 11 deletions(-)
hw/arm/virt.c +99 -11
@@ -55,6 +55,7 @@#include "qemu/bitops.h" #include "qemu/error-report.h" #include "qemu/module.h" +#include "hw/pci/pci_bus.h" #include "hw/pci-host/gpex.h" #include "hw/virtio/virtio-pci.h" #include "hw/core/sysbus-fdt.h"
@@ -83,6 +84,7 @@#include "hw/virtio/virtio-md-pci.h" #include "hw/virtio/virtio-iommu.h" #include "hw/char/pl011.h" +#include "qemu/config-file.h" #include "qemu/guest-random.h" static GlobalProperty arm_virt_compat[] = {
@@ -177,6 +179,7 @@ static const MemMapEntry base_memmap[] = { [VIRT_PVTIME] = { 0x090a0000, 0x00010000 }, [VIRT_SECURE_GPIO] = { 0x090b0000, 0x00001000 }, [VIRT_MMIO] = { 0x0a000000, 0x00000200 }, + [VIRT_NESTED_SMMU] = { 0x0b000000, 0x01000000 }, /* ...repeating for a total of NUM_VIRTIO_TRANSPORTS, each of that size */ [VIRT_PLATFORM_BUS] = { 0x0c000000, 0x02000000 }, [VIRT_SECURE_MEM] = { 0x0e000000, 0x01000000 },
@@ -221,7 +224,8 @@ static const int a15irqmap[] = { [VIRT_MMIO] = 16, /* ...to 16 + NUM_VIRTIO_TRANSPORTS - 1 */ [VIRT_GIC_V2M] = 48, /* ...to 48 + NUM_GICV2M_SPIS - 1 */ [VIRT_SMMU] = 74, /* ...to 74 + NUM_SMMU_IRQS - 1 */ - [VIRT_PLATFORM_BUS] = 112, /* ...to 112 + PLATFORM_BUS_NUM_IRQS -1 */ + [VIRT_PLATFORM_BUS] = 112, /* ...to 112 + PLATFORM_BUS_NUM_IRQS -1 (179) */ + [VIRT_NESTED_SMMU] = 200, /* Keep it at the end of list */ };static void create_randomness (MachineState *ms,const char *node)
@@ -1383,21 +1387,19 @@ static void create_pcie_irq_map(const MachineState *ms, 0x7 /* PCI irq */); } -static void create_smmu (const VirtMachineState *vms, -PCIBus *bus) +static DeviceState *_create_smmu (const VirtMachineState *vms,PCIBus *bus, +hwaddr base,hwaddr size,int irq, +uint32_t smmu_phandle) {char *node;const char compat[] ="arm,smmu-v3" ; -int irq = vms->irqmap[VIRT_SMMU ];int i; -hwaddr base = vms->memmap[VIRT_SMMU ].base; -hwaddr size = vms->memmap[VIRT_SMMU ].size;const char irq_names[] ="eventq\0priq\0cmdq-sync\0gerror" ;DeviceState *dev;MachineState *ms =MACHINE (vms); -if (!virt_has_smmuv3 (vms) || !vms->iommu_phandle) { -return ; +if (!virt_has_smmuv3 (vms) || !smmu_phandle) { +return NULL ; } dev =qdev_new (TYPE_ARM_SMMUV3 );
@@ -1436,8 +1438,31 @@ static void create_smmu(const VirtMachineState *vms,qemu_fdt_setprop_cell (ms->fdt, node,"#iommu-cells" ,1 ); -qemu_fdt_setprop_cell (ms->fdt, node,"phandle" , vms->iommu_phandle); +qemu_fdt_setprop_cell (ms->fdt, node,"phandle" , smmu_phandle);g_free (node); + +return dev; +} + +static DeviceState *create_smmu (const VirtMachineState *vms,PCIBus *bus) +{ +hwaddr base = vms->memmap[VIRT_SMMU ].base; +hwaddr size = vms->memmap[VIRT_SMMU ].size; +int irq = vms->irqmap[VIRT_SMMU ]; + +return _create_smmu (vms, bus, base, size, irq, vms->iommu_phandle); +} + +static DeviceState *create_nested_smmu (const VirtMachineState *vms,PCIBus *bus, +int i) +{ +hwaddr base = vms->memmap[VIRT_NESTED_SMMU ].base + i *SMMU_IO_LEN ; +int irq = vms->irqmap[VIRT_SMMU ] + i *NUM_SMMU_IRQS ; +hwaddr size =SMMU_IO_LEN ; +DeviceState *dev; + + dev =_create_smmu (vms, bus, base, size, irq, vms->nested_smmu_phandle[i]); +return dev; }static void create_virtio_iommu_dt_bindings (VirtMachineState *vms)
@@ -1466,6 +1491,48 @@ static void create_virtio_iommu_dt_bindings(VirtMachineState *vms) bdf + 1, vms->iommu_phandle, bdf + 1, 0xffff - bdf); } +/* +* FIXME this is used to reverse for hotplug devices, yet it could result in a +* big waste of PCI bus numbners. +*/ +#define MAX_DEV_PER_NESTED_SMMU (4) + +static PCIBus *create_pcie_expander_bridge (VirtMachineState *vms,uint8_t idx) +{ +/* Total = PXB + MAX_DEV_PER_NESTED_SMMU x (Rootport + device) */ +const uint8_t total_bus_nums =1 +MAX_DEV_PER_NESTED_SMMU *2 ; +uint8_t bus_nr =PCI_BUS_MAX - + (vms->num_nested_smmus - idx) * total_bus_nums; + +VirtNestedSmmu *nested_smmu =find_nested_smmu_by_index (vms, idx); +char *name_pxb =g_strdup_printf ("pxb_for_smmu.%d" , idx); +PCIBus *bus = vms->bus; +DeviceState *dev; + +/* Create an expander bridge */ + dev =qdev_new ("pxb-pcie" ); +if (!qdev_set_id (dev, name_pxb, &error_fatal)) { +return NULL ; + } + +qdev_prop_set_uint8 (dev,"bus_nr" , bus_nr); +qdev_prop_set_uint16 (dev,"numa_node" , idx); +qdev_realize_and_unref (dev,BUS (bus), &error_fatal); + +/* Get the pxb bus */ + QLIST_FOREACH(bus, &bus->child, sibling) { +if (pci_bus_num (bus) == bus_nr) { +break ; + } + } +g_assert (bus &&pci_bus_num (bus) == bus_nr); + nested_smmu->pci_bus = bus; + nested_smmu->reserved_bus_nums = total_bus_nums; + +/* Return the pxb bus */ +return bus; +} +static void create_pcie (VirtMachineState *vms) {hwaddr base_mmio = vms->memmap[VIRT_PCIE_MMIO ].base;
@@ -1580,12 +1647,33 @@ static void create_pcie(VirtMachineState *vms)qemu_fdt_setprop_cell (ms->fdt, nodename,"#interrupt-cells" ,1 );create_pcie_irq_map (ms, vms->gic_phandle, irq, nodename); -if (vms->iommu) { +/* Build PCI Expander Bridge + Root Port from the top of PCI_BUS_MAX */ +if (vms->num_nested_smmus) { +/* VIRT_NESTED_SMMU must hold all vSMMUs */ +g_assert (vms->num_nested_smmus <= + vms->memmap[VIRT_NESTED_SMMU ].size /SMMU_IO_LEN ); + + vms->nested_smmu_phandle =g_new0 (uint32_t, vms->num_nested_smmus); + +for (i =0 ; i < vms->num_nested_smmus; i++) { +DeviceState *smmu_dev; +PCIBus *pxb_bus; + + pxb_bus =create_pcie_expander_bridge (vms, i); +g_assert (pxb_bus); + + vms->nested_smmu_phandle[i] =qemu_fdt_alloc_phandle (ms->fdt); + smmu_dev =create_nested_smmu (vms, pxb_bus, i); +g_assert (smmu_dev); + +qemu_fdt_setprop_cells (ms->fdt, nodename,"iommu-map" ,0x0 , + vms->nested_smmu_phandle[i],0x0 ,0x10000 ); + } + }else if (vms->iommu) { vms->iommu_phandle =qemu_fdt_alloc_phandle (ms->fdt);switch (vms->iommu) {case VIRT_IOMMU_SMMUV3 : -case VIRT_IOMMU_NESTED_SMMUV3 :create_smmu (vms, vms->bus);qemu_fdt_setprop_cells (ms->fdt, nodename,"iommu-map" ,0x0 , vms->iommu_phandle,0x0 ,0x10000 );
include/hw/arm/virt.h +17 -0
@@ -63,6 +63,7 @@ enum {VIRT_GIC_ITS ,VIRT_GIC_REDIST ,VIRT_SMMU , +VIRT_NESTED_SMMU ,VIRT_UART ,VIRT_MMIO , VIRT_RTC,
@@ -141,6 +142,8 @@ struct VirtMachineClass {typedef struct VirtNestedSmmu {int index;char *smmu_node; +PCIBus *pci_bus; +int reserved_bus_nums;QLIST_ENTRY (VirtNestedSmmu) next; }VirtNestedSmmu ;
@@ -179,6 +182,7 @@ struct VirtMachineState {uint32_t gic_phandle;uint32_t msi_phandle;uint32_t iommu_phandle; +uint32_t *nested_smmu_phandle;int psci_conduit;hwaddr highest_gpa;DeviceState *gic;
@@ -229,4 +233,17 @@ static inline bool virt_has_smmuv3(const VirtMachineState *vms) vms->iommu ==VIRT_IOMMU_NESTED_SMMUV3 ; } +static inline VirtNestedSmmu * +find_nested_smmu_by_index (VirtMachineState *vms,int index) +{ +VirtNestedSmmu *nested_smmu; + + QLIST_FOREACH(nested_smmu, &vms->nested_smmu_list, next) { +if (nested_smmu->index == index) { +return nested_smmu; + } + } +return NULL ; +} +#endif /* QEMU_ARM_VIRT_H */
Signature · 1 line
2.43.0
Re: [PATCH RFCv1 05/10] hw/arm/virt: Add VIRT_NESTED_SMMU
Hi Nicolin,
Quoted 193 lines · Nicolin Chen
VIRT_SMMUcan be used as an emulated SMMU, i.e. iommu=smmuv3. However, the MMIO space forVIRT_SMMUisn't large enough to accommodate multiple nested SMMUv3 instances. Add a newVIRT_NESTED_SMMUto separate MMIO space.Meanwhile, a nested SMMUv3 could only work with a vfio-pci device that is physically associated to a host-level SMMUv3 instance, meaning that such a passthrough deivce must be linked to a nesteed SMMU corresponding to the underlying host-level SMMUv3 instance:
HW SMMU0 <-----------------> vSMMU0 (nested) ^ ^ | | ----> PCI dev0 & PCI dev1 <---- HW SMMU1 <-----------------> vSMMU1 (nested) ^ ^ | | ----> PCI dev2 & PCI dev3 <----Note that, on the other hand, an emulated deivce should not be associated to a nested SMMU.
To prepare for that, create a PCIe Expander Bridge per nested SMMU as a docker for pci-vfio devices. It eases the QEMU code to build ID mappings in the IORT.
Signed-off-by Nicolin Chen <[email protected]>--- hw/arm/virt.c | 110 +++++++++++++++++++++++++++++++++++++----- include/hw/arm/virt.h | 17 +++++++ 2 files changed, 116 insertions(+), 11 deletions(-)hw/arm/virt.c +87
-10@@ -55,6 +55,7 @@#include "qemu/bitops.h" #include "qemu/error-report.h" #include "qemu/module.h" +#include "hw/pci/pci_bus.h" #include "hw/pci-host/gpex.h" #include "hw/virtio/virtio-pci.h" #include "hw/core/sysbus-fdt.h" @@ -83,6 +84,7 @@#include "hw/virtio/virtio-md-pci.h" #include "hw/virtio/virtio-iommu.h" #include "hw/char/pl011.h" +#include "qemu/config-file.h" #include "qemu/guest-random.h" static GlobalProperty arm_virt_compat[] = {@@ -177,6 +179,7 @@ static const MemMapEntry base_memmap[] = { [VIRT_PVTIME] = { 0x090a0000, 0x00010000 }, [VIRT_SECURE_GPIO] = { 0x090b0000, 0x00001000 }, [VIRT_MMIO] = { 0x0a000000, 0x00000200 }, + [VIRT_NESTED_SMMU] = { 0x0b000000, 0x01000000 }, /* ...repeating for a total of NUM_VIRTIO_TRANSPORTS, each of that size */ [VIRT_PLATFORM_BUS] = { 0x0c000000, 0x02000000 }, [VIRT_SECURE_MEM] = { 0x0e000000, 0x01000000 },@@ -221,7 +224,8 @@ static const int a15irqmap[] = { [VIRT_MMIO] = 16, /* ...to 16 + NUM_VIRTIO_TRANSPORTS - 1 */ [VIRT_GIC_V2M] = 48, /* ...to 48 + NUM_GICV2M_SPIS - 1 */ [VIRT_SMMU] = 74, /* ...to 74 + NUM_SMMU_IRQS - 1 */ - [VIRT_PLATFORM_BUS] = 112, /* ...to 112 + PLATFORM_BUS_NUM_IRQS -1 */ + [VIRT_PLATFORM_BUS] = 112, /* ...to 112 + PLATFORM_BUS_NUM_IRQS -1 (179) */ + [VIRT_NESTED_SMMU] = 200, /* Keep it at the end of list */ };static void create_randomness (MachineState *ms,const char *node)@@ -1383,21 +1387,19 @@ static void create_pcie_irq_map(const MachineState *ms, 0x7 /* PCI irq */); } -static void create_smmu (const VirtMachineState *vms, -PCIBus *bus) +static DeviceState *_create_smmu (const VirtMachineState *vms,PCIBus *bus, +hwaddr base,hwaddr size,int irq, +uint32_t smmu_phandle) {char *node;const char compat[] ="arm,smmu-v3" ; -int irq = vms->irqmap[VIRT_SMMU ];int i; -hwaddr base = vms->memmap[VIRT_SMMU ].base; -hwaddr size = vms->memmap[VIRT_SMMU ].size;const char irq_names[] ="eventq\0priq\0cmdq-sync\0gerror" ;DeviceState *dev;MachineState *ms =MACHINE (vms); -if (!virt_has_smmuv3 (vms) || !vms->iommu_phandle) { -return ; +if (!virt_has_smmuv3 (vms) || !smmu_phandle) { +return NULL ; } dev =qdev_new (TYPE_ARM_SMMUV3 );@@ -1436,8 +1438,31 @@ static void create_smmu(const VirtMachineState *vms,qemu_fdt_setprop_cell (ms->fdt, node,"#iommu-cells" ,1 ); -qemu_fdt_setprop_cell (ms->fdt, node,"phandle" , vms->iommu_phandle); +qemu_fdt_setprop_cell (ms->fdt, node,"phandle" , smmu_phandle);g_free (node); + +return dev; +} + +static DeviceState *create_smmu (const VirtMachineState *vms,PCIBus *bus) +{ +hwaddr base = vms->memmap[VIRT_SMMU ].base; +hwaddr size = vms->memmap[VIRT_SMMU ].size; +int irq = vms->irqmap[VIRT_SMMU ]; + +return _create_smmu (vms, bus, base, size, irq, vms->iommu_phandle); +} + +static DeviceState *create_nested_smmu (const VirtMachineState *vms,PCIBus *bus, +int i) +{ +hwaddr base = vms->memmap[VIRT_NESTED_SMMU ].base + i *SMMU_IO_LEN ; +int irq = vms->irqmap[VIRT_SMMU ] + i *NUM_SMMU_IRQS ; +hwaddr size =SMMU_IO_LEN ; +DeviceState *dev; + + dev =_create_smmu (vms, bus, base, size, irq, vms->nested_smmu_phandle[i]); +return dev; }static void create_virtio_iommu_dt_bindings (VirtMachineState *vms)@@ -1466,6 +1491,48 @@ static void create_virtio_iommu_dt_bindings(VirtMachineState *vms) bdf + 1, vms->iommu_phandle, bdf + 1, 0xffff - bdf); } +/* +* FIXME this is used to reverse for hotplug devices, yet it could result in a +* big waste of PCI bus numbners. +*/ +#define MAX_DEV_PER_NESTED_SMMU (4) + +static PCIBus *create_pcie_expander_bridge (VirtMachineState *vms,uint8_t idx) +{ +/* Total = PXB + MAX_DEV_PER_NESTED_SMMU x (Rootport + device) */ +const uint8_t total_bus_nums =1 +MAX_DEV_PER_NESTED_SMMU *2 ; +uint8_t bus_nr =PCI_BUS_MAX - + (vms->num_nested_smmus - idx) * total_bus_nums; + +VirtNestedSmmu *nested_smmu =find_nested_smmu_by_index (vms, idx); +char *name_pxb =g_strdup_printf ("pxb_for_smmu.%d" , idx); +PCIBus *bus = vms->bus; +DeviceState *dev; + +/* Create an expander bridge */ + dev =qdev_new ("pxb-pcie" ); +if (!qdev_set_id (dev, name_pxb, &error_fatal)) { +return NULL ; + } + +qdev_prop_set_uint8 (dev,"bus_nr" , bus_nr); +qdev_prop_set_uint16 (dev,"numa_node" , idx); +qdev_realize_and_unref (dev,BUS (bus), &error_fatal); + +/* Get the pxb bus */ + QLIST_FOREACH(bus, &bus->child, sibling) { +if (pci_bus_num (bus) == bus_nr) { +break ; + } + } +g_assert (bus &&pci_bus_num (bus) == bus_nr); + nested_smmu->pci_bus = bus; + nested_smmu->reserved_bus_nums = total_bus_nums; + +/* Return the pxb bus */ +return bus; +} +static void create_pcie (VirtMachineState *vms) {hwaddr base_mmio = vms->memmap[VIRT_PCIE_MMIO ].base;@@ -1580,12 +1647,33 @@ static void create_pcie(VirtMachineState *vms)qemu_fdt_setprop_cell (ms->fdt, nodename,"#interrupt-cells" ,1 );create_pcie_irq_map (ms, vms->gic_phandle, irq, nodename); -if (vms->iommu) { +/* Build PCI Expander Bridge + Root Port from the top of PCI_BUS_MAX */ +if (vms->num_nested_smmus) { +/* VIRT_NESTED_SMMU must hold all vSMMUs */ +g_assert (vms->num_nested_smmus <= + vms->memmap[VIRT_NESTED_SMMU ].size /SMMU_IO_LEN ); + + vms->nested_smmu_phandle =g_new0 (uint32_t, vms->num_nested_smmus); + +for (i =0 ; i < vms->num_nested_smmus; i++) { +DeviceState *smmu_dev; +PCIBus *pxb_bus;
hw/arm/virt.c static void create_pcie(VirtMachineState *vms) · line 1670
Quoted 10 lines
hw/arm/virt.c +10
-0+ + pxb_bus =create_pcie_expander_bridge (vms, i); +g_assert (pxb_bus); + + vms->nested_smmu_phandle[i] =qemu_fdt_alloc_phandle (ms->fdt); + smmu_dev =create_nested_smmu (vms, pxb_bus, i); +g_assert (smmu_dev); + +qemu_fdt_setprop_cells (ms->fdt, nodename,"iommu-map" ,0x0 , + vms->nested_smmu_phandle[i],0x0 ,0x10000 );
I think libvirt is supposed to create the pcie bus topology instead and it shall not be created by qemu behind the scene.
The pcie elements you create here are not visible to libvirt and I guess they may collide with elements explicitly created by libvirt at a given pci bdf.
I think it would make more sense to be able to attach an smmu instance to a given pci root or pxb either by adding an iommu id to a given pxb-pcie option
-device pxb-pcie,bus_nr=100,id=pci.12,numa_node=0,bus=pcie.0,addr=0x3,iommu=<id> or
adding a list of pxb ids to the iommu option. It is unfortunate the iommu option is a machine option. the smmu being a sysbus device the platform bus framework could be considered to dynamically allocate them using the -device option. This has been used along with dt generation but with ACPI this would need to be studied. However at the time the smmu was integrated the machine option was prefered.
Maybe using the 1st option would allow to infer that if there are different iommu ids this implies that several IOMMU instances need to be created.
Adding Andrea in cc
Thanks
Eric
Quoted 57 lines
hw/arm/virt.c +2
-1+ } + }else if (vms->iommu) { vms->iommu_phandle =qemu_fdt_alloc_phandle (ms->fdt);switch (vms->iommu) {case VIRT_IOMMU_SMMUV3 : -case VIRT_IOMMU_NESTED_SMMUV3 :create_smmu (vms, vms->bus);qemu_fdt_setprop_cells (ms->fdt, nodename,"iommu-map" ,0x0 , vms->iommu_phandle,0x0 ,0x10000 );include/hw/arm/virt.h +17
-0@@ -63,6 +63,7 @@ enum {VIRT_GIC_ITS ,VIRT_GIC_REDIST ,VIRT_SMMU , +VIRT_NESTED_SMMU ,VIRT_UART ,VIRT_MMIO , VIRT_RTC,@@ -141,6 +142,8 @@ struct VirtMachineClass {typedef struct VirtNestedSmmu {int index;char *smmu_node; +PCIBus *pci_bus; +int reserved_bus_nums;QLIST_ENTRY (VirtNestedSmmu) next; }VirtNestedSmmu ;@@ -179,6 +182,7 @@ struct VirtMachineState {uint32_t gic_phandle;uint32_t msi_phandle;uint32_t iommu_phandle; +uint32_t *nested_smmu_phandle;int psci_conduit;hwaddr highest_gpa;DeviceState *gic;@@ -229,4 +233,17 @@ static inline bool virt_has_smmuv3(const VirtMachineState *vms) vms->iommu ==VIRT_IOMMU_NESTED_SMMUV3 ; } +static inline VirtNestedSmmu * +find_nested_smmu_by_index (VirtMachineState *vms,int index) +{ +VirtNestedSmmu *nested_smmu; + + QLIST_FOREACH(nested_smmu, &vms->nested_smmu_list, next) { +if (nested_smmu->index == index) { +return nested_smmu; + } + } +return NULL ; +} +#endif /* QEMU_ARM_VIRT_H */
Re: [PATCH RFCv1 05/10] hw/arm/virt: Add VIRT_NESTED_SMMU
Quoted 23 lines · Eric Auger
Quoted 23 lines
hw/arm/virt.c +18
-1@@ -1580,12 +1647,33 @@ static void create_pcie(VirtMachineState *vms)qemu_fdt_setprop_cell (ms->fdt, nodename,"#interrupt-cells" ,1 );create_pcie_irq_map (ms, vms->gic_phandle, irq, nodename); -if (vms->iommu) { +/* Build PCI Expander Bridge + Root Port from the top of PCI_BUS_MAX */ +if (vms->num_nested_smmus) { +/* VIRT_NESTED_SMMU must hold all vSMMUs */ +g_assert (vms->num_nested_smmus <= + vms->memmap[VIRT_NESTED_SMMU ].size /SMMU_IO_LEN ); + + vms->nested_smmu_phandle =g_new0 (uint32_t, vms->num_nested_smmus); + +for (i =0 ; i < vms->num_nested_smmus; i++) { +DeviceState *smmu_dev; +PCIBus *pxb_bus; + + pxb_bus =create_pcie_expander_bridge (vms, i); +g_assert (pxb_bus); + + vms->nested_smmu_phandle[i] =qemu_fdt_alloc_phandle (ms->fdt); + smmu_dev =create_nested_smmu (vms, pxb_bus, i); +g_assert (smmu_dev);
Quoted 8 lines
hw/arm/virt.c static void create_pcie(VirtMachineState *vms) · line 1670
Quoted 3 lines
hw/arm/virt.c +3
-0+ +qemu_fdt_setprop_cells (ms->fdt, nodename,"iommu-map" ,0x0 , + vms->nested_smmu_phandle[i],0x0 ,0x10000 );I think libvirt is supposed to create the pcie bus topology instead and it shall not be created by qemu behind the scene.
The pcie elements you create here are not visible to libvirt and I guess they may collide with elements explicitly created by libvirt at a given pci bdf.
Yea, the bdf conflict is a concern. So I allocated the bdf list from the top of the bus number... one of the reasons of doing this is to ease users so they don't need to deal with the over- complicated topology. I will try libvirt and see how it goes.
Quoted 9 lines
I think it would make more sense to be able to attach an smmu instance to a given pci root or pxb either by adding an iommu id to a given pxb-pcie option
-device pxb-pcie,bus_nr=100,id=pci.12,numa_node=0,bus=pcie.0,addr=0x3,iommu=<id> or
adding a list of pxb ids to the iommu option. It is unfortunate the iommu option is a machine option.
Yes. I had thought about that too, but the virt-machine code creates all the instance at this moment...
Quoted 8 lines
platform bus framework could be considered to dynamically allocate them using the -device option. This has been used along with dt generation but with ACPI this would need to be studied. However at the time the smmu was integrated the machine option was prefered.
Maybe using the 1st option would allow to infer that if there are different iommu ids this implies that several IOMMU instances need to be created.
Yea, I like the idea of creating iommu instance with a "-device" string.
One more question. Let's say we have 2 smmus/pxb-buses:
[ pxb0] <---> vSMMU0/pSMMU0 [ devA, devB, devC ]
[ pxb1] <---> vSMMU1/pSMMU1 [ devD, devE, devF ]
How would a user know that devA/devB should be attached to pxb0 without doing like devA->pxb0 and devB->pxb1? Should QEMU just error out until the user associate them correctly? Or they may rely on libvirt to figure that out, i.e. moving the iommu node matching from QEMU to libvirt?
Thanks
Nicolin
Re: [PATCH RFCv1 05/10] hw/arm/virt: Add VIRT_NESTED_SMMU
Quoted 29 lines · Nicolin Chen
Quoted 19 lines · Eric Auger
Quoted 19 lines
hw/arm/virt.c +19
-0+/* Build PCI Expander Bridge + Root Port from the top of PCI_BUS_MAX */ +if (vms->num_nested_smmus) { +/* VIRT_NESTED_SMMU must hold all vSMMUs */ +g_assert (vms->num_nested_smmus <= + vms->memmap[VIRT_NESTED_SMMU ].size /SMMU_IO_LEN ); + + vms->nested_smmu_phandle =g_new0 (uint32_t, vms->num_nested_smmus); + +for (i =0 ; i < vms->num_nested_smmus; i++) { +DeviceState *smmu_dev; +PCIBus *pxb_bus; + + pxb_bus =create_pcie_expander_bridge (vms, i); +g_assert (pxb_bus); + + vms->nested_smmu_phandle[i] =qemu_fdt_alloc_phandle (ms->fdt); + smmu_dev =create_nested_smmu (vms, pxb_bus, i); +g_assert (smmu_dev); +Quoted 8 lines
hw/arm/virt.c static void create_pcie(VirtMachineState *vms) · line 1670
Quoted 2 lines
hw/arm/virt.c +2
-0+qemu_fdt_setprop_cells (ms->fdt, nodename,"iommu-map" ,0x0 , + vms->nested_smmu_phandle[i],0x0 ,0x10000 );I think libvirt is supposed to create the pcie bus topology instead and it shall not be created by qemu behind the scene.
The pcie elements you create here are not visible to libvirt and I guess they may collide with elements explicitly created by libvirt at a given pci bdf.
Quoted 4 lines
Yea, the bdf conflict is a concern. So I allocated the bdf list from the top of the bus number... one of the reasons of doing this is to ease users so they don't need to deal with the over- complicated topology. I will try libvirt and see how it goes.
libvirt developer here. Eric is absolutely right that creating PCI controllers implicitly will not fly, as libvirt's PCI address allocation algorithm would not know about them and would produce incorrect output as a result.
Quoted 18 lines
Quoted 9 lines
I think it would make more sense to be able to attach an smmu instance to a given pci root or pxb either by adding an iommu id to a given pxb-pcie option
-device pxb-pcie,bus_nr=100,id=pci.12,numa_node=0,bus=pcie.0,addr=0x3,iommu=<id> or
adding a list of pxb ids to the iommu option. It is unfortunate the iommu option is a machine option.Yes. I had thought about that too, but the virt-machine code creates all the instance at this moment...
Quoted 5 lines
platform bus framework could be considered to dynamically allocate them using the -device option. This has been used along with dt generation but with ACPI this would need to be studied. However at the time the smmu was integrated the machine option was prefered.
Quoted 6 lines
Quoted 3 lines
Maybe using the 1st option would allow to infer that if there are different iommu ids this implies that several IOMMU instances need to be created.
Yea, I like the idea of creating iommu instance with a "-device" string.
This sounds like the best option to me as well. Incidentally, it's also how the intel-iommu device works. From libvirt's test suite:
-device '{"driver":"intel-iommu","id":"iommu0"}'
Looks like a sensible model to follow. You could then associate each specific IOMMU instance to a PCI controller using an attribute, as Eric suggested above.
Quoted 8 lines
One more question. Let's say we have 2 smmus/pxb-buses:
[ pxb0] <---> vSMMU0/pSMMU0 [ devA, devB, devC ]
[ pxb1] <---> vSMMU1/pSMMU1 [ devD, devE, devF ]
How would a user know that devA/devB should be attached to pxb0 without doing likedevA->pxb0anddevB->pxb1? Should QEMU just error out until the user associate them correctly? Or they may rely on libvirt to figure that out, i.e. moving the iommu node matching from QEMU to libvirt?
IIUC having devices associated to the "correct" SMMU is only a requirement for optimal performance, not pure functionality, right? In other words, getting the association wrong will make things slower but they will keep working.
Is a 1:1 match between pSMMU and vSMMU mandatory? On a system with 4 SMMUs, could I create a VMs with
vSMMU0 <-> pSMMU0
vSMMU1 <-> pSMMU1
and another one with
vSMMU0 <-> pSMMU2
vSMMU1 <-> pSMMU3
assuming of course that devices are assigned to the correct VM?
How is the association between vSMMU and pSMMU created anyway? Is that something that the user can control, or is it done automatically somehow?
Signature · 1 line
Andrea Bolognani / Red Hat / Virtualization
Re: [PATCH RFCv1 05/10] hw/arm/virt: Add VIRT_NESTED_SMMU
Quoted 4 lines · Andrea Bolognani
IIUC having devices associated to the "correct" SMMU is only a requirement for optimal performance, not pure functionality, right? In other words, getting the association wrong will make things slower but they will keep working.
As part of declaring the iommu as a device libvirt should also specify the iommu capabilities and features.
There are capabilities that require perfect 1:1 pIOMMU to vIOMMU matching, or it can't work.
For instance if the pIOMMU has a direct path for invalidations then the pIOMMU, pPCI, vIOMMU, and vPCI all must be perfectly matched because the vIOMMU will deliver invalidations directly to a single pIOMMU with no possibility to steer them to the correct place in hypervisor SW.
Quoted 10 lines
Is a 1:1 match between pSMMU and vSMMU mandatory? On a system with 4 SMMUs, could I create a VMs with
vSMMU0 <-> pSMMU0
vSMMU1 <-> pSMMU1and another one with
vSMMU0 <-> pSMMU2
vSMMU1 <-> pSMMU3
This is fine
Quoted 1 line
assuming of course that devices are assigned to the correct VM?
But devices on pSMMU0 cannot be assigned to the second VM in above situations.
Quoted 3 lines
How is the association between vSMMU and pSMMU created anyway? Is that something that the user can control, or is it done automatically somehow?
From a kernel side VIOMMU will be created in IOMMUFD and it will get as input a reference to a VFIO device. The VIOMMU will be linked to that VFIO device's pIOMMU.
Also creating a dummy vIOMMU without any devices and thus no way to reach the pIOMMU is definately "tricky", I'm not sure exactly what is possible there.
Jason
RFC PATCH 6/10 hw/arm/virt: Assign vfio-pci devices to nested SMMUs
In reply to Nicolin ChenWith iommu=nested-smmuv3, there could be multiple nested SMMU instances in the vms. A passthrough device must to look up for its iommu handler in its sysfs node, and then link to the nested SMMU instance created for the same iommu handler. This isn't easy to do.
Add an auto-assign piece after all vSMMU backed pxb buses are created. It loops the existing input devices, and sets/replaces their pci bus numbers with a newly created pcie-root-port to the pxb bus.
Note that this is not an ideal solution to handle hot plug device.
--- hw/arm/virt.c | 110 ++++++++++++++++++++++++++++++++++++++++++ include/hw/arm/virt.h | 13 +++++ 2 files changed, 123 insertions(+)
hw/arm/virt.c +110 -0
@@ -38,6 +38,7 @@#include "hw/arm/primecell.h" #include "hw/arm/virt.h" #include "hw/block/flash.h" +#include "hw/vfio/pci.h" #include "hw/vfio/vfio-calxeda-xgmac.h" #include "hw/vfio/vfio-amd-xgbe.h" #include "hw/display/ramfb.h"
@@ -1491,6 +1492,112 @@ static void create_virtio_iommu_dt_bindings(VirtMachineState *vms) bdf + 1, vms->iommu_phandle, bdf + 1, 0xffff - bdf); } +static char *create_new_pcie_port (VirtNestedSmmu *nested_smmu,Error **errp) +{ +uint32_t port_nr = nested_smmu->pci_bus->qbus.num_children; +uint32_t chassis_nr =UINT8_MAX - nested_smmu->index; +uint32_t bus_nr =pci_bus_num (nested_smmu->pci_bus); +DeviceState *dev; +char *name_port; + +/* Create a root port */ + dev =qdev_new ("pcie-root-port" ); + name_port =g_strdup_printf ("smmu_bus0x%x_port%d" , bus_nr, port_nr); + +if (!qdev_set_id (dev, name_port, &error_fatal)) { +/* FIXME retry with a different port num? */ +error_setg (errp,"Could not set pcie-root-port ID %s" , name_port); +g_free (name_port); +g_free (dev); +return NULL ; + } +qdev_prop_set_uint32 (dev,"chassis" , chassis_nr); +qdev_prop_set_uint32 (dev,"slot" , port_nr); +qdev_prop_set_uint64 (dev,"io-reserve" ,0 ); +qdev_realize_and_unref (dev,BUS (nested_smmu->pci_bus), &error_fatal); +return name_port; +} + +static int assign_nested_smmu (void *opaque,QemuOpts *opts,Error **errp) +{ +VirtMachineState *vms = (VirtMachineState *)opaque; +const char *sysfsdev =qemu_opt_get (opts,"sysfsdev" ); +const char *iommufd =qemu_opt_get (opts,"iommufd" ); +const char *driver =qemu_opt_get (opts,"driver" ); +const char *host =qemu_opt_get (opts,"host" ); +const char *bus =qemu_opt_get (opts,"bus" ); +VirtNestedSmmu *nested_smmu; +char *link_iommu; +char *dir_iommu; +char *smmu_node; +char *name_port; +int ret =0 ; + +if (!iommufd || !driver) { +return 0 ; + } +if (!sysfsdev && !host) { +return 0 ; + } +if (strncmp (driver,TYPE_VFIO_PCI ,strlen (TYPE_VFIO_PCI ))) { +return 0 ; + } +/* If the device wants to attach to the default bus, do not reassign it */ +if (bus && !strncmp (bus,"pcie.0" ,strlen (bus))) { +return 0 ; + } + +if (sysfsdev) { + link_iommu =g_strdup_printf ("%s/iommu" , sysfsdev); + }else { + link_iommu =g_strdup_printf ("/sys/bus/pci/devices/%s/iommu" , host); + } + + dir_iommu =realpath (link_iommu,NULL ); +if (!dir_iommu) { +error_setg (errp,"Could not get the real path for iommu link: %s" , + link_iommu); + ret = -EINVAL ; + goto free_link; + } + + smmu_node =g_path_get_basename (dir_iommu); +if (!smmu_node) { +error_setg (errp,"Could not get SMMU node name for iommu at: %s" , + dir_iommu); + ret = -EINVAL ; + goto free_dir; + } + + nested_smmu =find_nested_smmu_by_sysfs (vms, smmu_node); +if (!nested_smmu) { +error_setg (errp,"Could not find any detected SMMU matching node: %s" , + smmu_node); + ret = -EINVAL ; + goto free_node; + } + + name_port =create_new_pcie_port (nested_smmu, errp); +if (!name_port) { + ret = -EBUSY ; + goto free_node; + } + +qemu_opt_set (opts,"bus" , name_port, &error_fatal); +if (bus) { +error_report ("overriding PCI bus %s to %s for device %s [%s]" , + bus, name_port, host, sysfsdev); + } + +free_node: +free (smmu_node); +free_dir: +free (dir_iommu); +free_link: +free (link_iommu); +return ret; +} +/* * FIXME this is used to reverse for hotplug devices, yet it could result in a * big waste of PCI bus numbners.
@@ -1669,6 +1776,9 @@ static void create_pcie(VirtMachineState *vms)qemu_fdt_setprop_cells (ms->fdt, nodename,"iommu-map" ,0x0 , vms->nested_smmu_phandle[i],0x0 ,0x10000 ); } + +qemu_opts_foreach (qemu_find_opts ("device" ), + assign_nested_smmu, vms, &error_fatal); }else if (vms->iommu) { vms->iommu_phandle =qemu_fdt_alloc_phandle (ms->fdt);
include/hw/arm/virt.h +13 -0
@@ -246,4 +246,17 @@ find_nested_smmu_by_index(VirtMachineState *vms, int index)return NULL ; } +static inline VirtNestedSmmu * +find_nested_smmu_by_sysfs (VirtMachineState *vms,char *node) +{ +VirtNestedSmmu *nested_smmu; + + QLIST_FOREACH(nested_smmu, &vms->nested_smmu_list, next) { +if (!strncmp (nested_smmu->smmu_node, node,strlen (node))) { +return nested_smmu; + } + } +return NULL ; +} +#endif /* QEMU_ARM_VIRT_H */
Signature · 1 line
2.43.0
Re: [PATCH RFCv1 06/10] hw/arm/virt: Assign vfio-pci devices to nested SMMUs
Quoted 8 lines · Nicolin Chen
With iommu=nested-smmuv3, there could be multiple nested SMMU instances in the vms. A passthrough device must to look up for its iommu handler in its sysfs node, and then link to the nested SMMU instance created for the same iommu handler. This isn't easy to do.
Add an auto-assign piece after all vSMMU backed pxb buses are created. It loops the existing input devices, and sets/replaces their pci bus numbers with a newly created pcie-root-port to the pxb bus.
Here again I don't think it is acceptable to create such topology under the hood. Libvirt shall master the whole PCIe topology.
Eric
Quoted 166 lines
Note that this is not an ideal solution to handle hot plug device.
Signed-off-by Nicolin Chen <[email protected]>--- hw/arm/virt.c | 110 ++++++++++++++++++++++++++++++++++++++++++ include/hw/arm/virt.h | 13 +++++ 2 files changed, 123 insertions(+)hw/arm/virt.c +110
-0@@ -38,6 +38,7 @@#include "hw/arm/primecell.h" #include "hw/arm/virt.h" #include "hw/block/flash.h" +#include "hw/vfio/pci.h" #include "hw/vfio/vfio-calxeda-xgmac.h" #include "hw/vfio/vfio-amd-xgbe.h" #include "hw/display/ramfb.h" @@ -1491,6 +1492,112 @@ static void create_virtio_iommu_dt_bindings(VirtMachineState *vms) bdf + 1, vms->iommu_phandle, bdf + 1, 0xffff - bdf); } +static char *create_new_pcie_port (VirtNestedSmmu *nested_smmu,Error **errp) +{ +uint32_t port_nr = nested_smmu->pci_bus->qbus.num_children; +uint32_t chassis_nr =UINT8_MAX - nested_smmu->index; +uint32_t bus_nr =pci_bus_num (nested_smmu->pci_bus); +DeviceState *dev; +char *name_port; + +/* Create a root port */ + dev =qdev_new ("pcie-root-port" ); + name_port =g_strdup_printf ("smmu_bus0x%x_port%d" , bus_nr, port_nr); + +if (!qdev_set_id (dev, name_port, &error_fatal)) { +/* FIXME retry with a different port num? */ +error_setg (errp,"Could not set pcie-root-port ID %s" , name_port); +g_free (name_port); +g_free (dev); +return NULL ; + } +qdev_prop_set_uint32 (dev,"chassis" , chassis_nr); +qdev_prop_set_uint32 (dev,"slot" , port_nr); +qdev_prop_set_uint64 (dev,"io-reserve" ,0 ); +qdev_realize_and_unref (dev,BUS (nested_smmu->pci_bus), &error_fatal); +return name_port; +} + +static int assign_nested_smmu (void *opaque,QemuOpts *opts,Error **errp) +{ +VirtMachineState *vms = (VirtMachineState *)opaque; +const char *sysfsdev =qemu_opt_get (opts,"sysfsdev" ); +const char *iommufd =qemu_opt_get (opts,"iommufd" ); +const char *driver =qemu_opt_get (opts,"driver" ); +const char *host =qemu_opt_get (opts,"host" ); +const char *bus =qemu_opt_get (opts,"bus" ); +VirtNestedSmmu *nested_smmu; +char *link_iommu; +char *dir_iommu; +char *smmu_node; +char *name_port; +int ret =0 ; + +if (!iommufd || !driver) { +return 0 ; + } +if (!sysfsdev && !host) { +return 0 ; + } +if (strncmp (driver,TYPE_VFIO_PCI ,strlen (TYPE_VFIO_PCI ))) { +return 0 ; + } +/* If the device wants to attach to the default bus, do not reassign it */ +if (bus && !strncmp (bus,"pcie.0" ,strlen (bus))) { +return 0 ; + } + +if (sysfsdev) { + link_iommu =g_strdup_printf ("%s/iommu" , sysfsdev); + }else { + link_iommu =g_strdup_printf ("/sys/bus/pci/devices/%s/iommu" , host); + } + + dir_iommu =realpath (link_iommu,NULL ); +if (!dir_iommu) { +error_setg (errp,"Could not get the real path for iommu link: %s" , + link_iommu); + ret = -EINVAL ; + goto free_link; + } + + smmu_node =g_path_get_basename (dir_iommu); +if (!smmu_node) { +error_setg (errp,"Could not get SMMU node name for iommu at: %s" , + dir_iommu); + ret = -EINVAL ; + goto free_dir; + } + + nested_smmu =find_nested_smmu_by_sysfs (vms, smmu_node); +if (!nested_smmu) { +error_setg (errp,"Could not find any detected SMMU matching node: %s" , + smmu_node); + ret = -EINVAL ; + goto free_node; + } + + name_port =create_new_pcie_port (nested_smmu, errp); +if (!name_port) { + ret = -EBUSY ; + goto free_node; + } + +qemu_opt_set (opts,"bus" , name_port, &error_fatal); +if (bus) { +error_report ("overriding PCI bus %s to %s for device %s [%s]" , + bus, name_port, host, sysfsdev); + } + +free_node: +free (smmu_node); +free_dir: +free (dir_iommu); +free_link: +free (link_iommu); +return ret; +} +/* * FIXME this is used to reverse for hotplug devices, yet it could result in a * big waste of PCI bus numbners. @@ -1669,6 +1776,9 @@ static void create_pcie(VirtMachineState *vms)qemu_fdt_setprop_cells (ms->fdt, nodename,"iommu-map" ,0x0 , vms->nested_smmu_phandle[i],0x0 ,0x10000 ); } + +qemu_opts_foreach (qemu_find_opts ("device" ), + assign_nested_smmu, vms, &error_fatal); }else if (vms->iommu) { vms->iommu_phandle =qemu_fdt_alloc_phandle (ms->fdt);include/hw/arm/virt.h +13
-0@@ -246,4 +246,17 @@ find_nested_smmu_by_index(VirtMachineState *vms, int index)return NULL ; } +static inline VirtNestedSmmu * +find_nested_smmu_by_sysfs (VirtMachineState *vms,char *node) +{ +VirtNestedSmmu *nested_smmu; + + QLIST_FOREACH(nested_smmu, &vms->nested_smmu_list, next) { +if (!strncmp (nested_smmu->smmu_node, node,strlen (node))) { +return nested_smmu; + } + } +return NULL ; +} +#endif /* QEMU_ARM_VIRT_H */
RFC PATCH 7/10 hw/arm/virt: Bypass iommu for default PCI bus
In reply to Nicolin ChenNow, all passthrough devices that should benefit from the nested SMMUv3 feature are assigned to dedicated pxb buses. So, the default PCI bus can be only used by emulated devices.
In theory, these emualted devices can be still attached to an emualted SMMUv3 instance, yet there is no gain doing that. Set the default PCI bus to bypass iommu, for the maximum performance.
--- hw/arm/virt.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
hw/arm/virt.c +2 -1
@@ -1708,7 +1708,8 @@ static void create_pcie(VirtMachineState *vms) } pci =PCI_HOST_BRIDGE (dev); - pci->bypass_iommu = vms->default_bus_bypass_iommu; +/* Default bus used by emulated devices does not go through nested SMMUs */ + pci->bypass_iommu = vms->default_bus_bypass_iommu || vms->num_nested_smmus; vms->bus = pci->bus;if (vms->bus) {pci_init_nic_devices (pci->bus, mc->default_nic);
Signature · 1 line
2.43.0
RFC PATCH 8/10 hw/arm/virt-acpi-build: Handle reserved bus number of pxb buses
In reply to Nicolin ChenEach pxb bus created for a nested SMMU has a reserved bus number, allowing a hotplug device to attach to the bus in a later stage.
Read it out to apply to the id_count calculation.
--- hw/arm/virt-acpi-build.c | 28 ++++++++++++++++++++++++---- include/hw/arm/virt.h | 13 +++++++++++++ 2 files changed, 37 insertions(+), 4 deletions(-)
hw/arm/virt-acpi-build.c +24 -4
@@ -236,6 +236,12 @@ build_iort_id_mapping(GArray *table_data, uint32_t input_base,build_append_int_noprefix (table_data, flags,4 );/* Flags */ } +struct AcpiIortIdMappingVM { +VirtMachineState *vms; +GArray *smmu_idmaps; +}; +typedef struct AcpiIortIdMappingVM AcpiIortIdMappingVM ; +struct AcpiIortIdMapping {uint32_t input_base;uint32_t id_count;
@@ -246,21 +252,34 @@ typedef struct AcpiIortIdMapping AcpiIortIdMapping;static int iort_host_bridges (Object *obj,void *opaque) { -GArray *idmap_blob = opaque; +AcpiIortIdMappingVM *idmap_vm = opaque; +VirtMachineState *vms = idmap_vm->vms;if (object_dynamic_cast (obj,TYPE_PCI_HOST_BRIDGE )) {PCIBus *bus =PCI_HOST_BRIDGE (obj)->bus;if (bus && !pci_bus_bypass_iommu (bus)) { +VirtNestedSmmu *nested_smmu =find_nested_smmu_by_pci_bus (vms, bus);int min_bus, max_bus; -pci_bus_range (bus, &min_bus, &max_bus); +if (vms->iommu ==VIRT_IOMMU_NESTED_SMMUV3 ) { +/* PCI host bridge hehind a nested SMMU has reserved buses */ +if (nested_smmu) { + min_bus =pci_bus_num (nested_smmu->pci_bus); + max_bus = min_bus + nested_smmu->reserved_bus_nums -1 ; + }else { +/* Not connected to a nested SMMU */ +return 0 ; + } + }else { +pci_bus_range (bus, &min_bus, &max_bus); + }AcpiIortIdMapping idmap = { .input_base = min_bus <<8 , .id_count = (max_bus - min_bus +1 ) <<8 , }; -g_array_append_val (idmap_blob, idmap); +g_array_append_val (idmap_vm->smmu_idmaps, idmap); } }
@@ -331,6 +350,7 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)uint32_t id =0 ;GArray *smmu_idmaps =g_array_new (false, true,sizeof (AcpiIortIdMapping));GArray *its_idmaps =g_array_new (false, true,sizeof (AcpiIortIdMapping)); +AcpiIortIdMappingVM idmap_vm = { .vms = vms, .smmu_idmaps = smmu_idmaps, };AcpiTable table = { .sig ="IORT" , .rev =5 , .oem_id = vms->oem_id, .oem_table_id = vms->oem_table_id };
@@ -341,7 +361,7 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)AcpiIortIdMapping next_range = {0 };object_child_foreach_recursive (object_get_root (), - iort_host_bridges, smmu_idmaps); + iort_host_bridges, &idmap_vm); nb_nodes =3 ;/* RC, ITS, SMMUv3 */
include/hw/arm/virt.h +13 -0
@@ -259,4 +259,17 @@ find_nested_smmu_by_sysfs(VirtMachineState *vms, char *node)return NULL ; } +static inline VirtNestedSmmu * +find_nested_smmu_by_pci_bus (VirtMachineState *vms,PCIBus *pci_bus) +{ +VirtNestedSmmu *nested_smmu; + + QLIST_FOREACH(nested_smmu, &vms->nested_smmu_list, next) { +if (nested_smmu->pci_bus == pci_bus) { +return nested_smmu; + } + } +return NULL ; +} +#endif /* QEMU_ARM_VIRT_H */
Signature · 1 line
2.43.0
RFC PATCH 9/10 hw/arm/virt-acpi-build: Build IORT with multiple SMMU nodes
In reply to Nicolin ChenThere can be multiple PCI buses behind different SMMU nodes. And each pair should be associated in the IORT table too when building the ID mappings.
Create multiple SMMU nodes if needed, store their offsets in an array.
--- hw/arm/virt-acpi-build.c | 36 ++++++++++++++++++++++++++---------- 1 file changed, 26 insertions(+), 10 deletions(-)
hw/arm/virt-acpi-build.c +26 -10
@@ -295,7 +295,7 @@ static int iort_idmap_compare(gconstpointer a, gconstpointer b) }static void -build_iort_rmr_nodes (GArray *table_data,GArray *smmu_idmaps,int smmu_offset,uint32_t *id) { +build_iort_rmr_nodes (GArray *table_data,GArray *smmu_idmaps,size_t *smmu_offset,uint32_t *id) {AcpiIortIdMapping *range;int i;
@@ -323,7 +323,7 @@ build_iort_rmr_nodes(GArray *table_data, GArray *smmu_idmaps, int smmu_offset, ubuild_append_int_noprefix (table_data,1 ,4 );/* Reference to Memory Range Descriptors */ build_append_int_noprefix (table_data,28 +ID_MAPPING_ENTRY_SIZE ,4 ); -build_iort_id_mapping (table_data, bdf, range->id_count, smmu_offset,1 ); +build_iort_id_mapping (table_data, bdf, range->id_count, smmu_offset[i],1 );/* Table 19 Memory Range Descriptor */
@@ -345,25 +345,42 @@ static void build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) {int i, nb_nodes, rc_mapping_count; -size_t node_size, smmu_offset =0 ; +size_t node_size, *smmu_offset;AcpiIortIdMapping *idmap;uint32_t id =0 ;GArray *smmu_idmaps =g_array_new (false, true,sizeof (AcpiIortIdMapping));GArray *its_idmaps =g_array_new (false, true,sizeof (AcpiIortIdMapping));AcpiIortIdMappingVM idmap_vm = { .vms = vms, .smmu_idmaps = smmu_idmaps, }; +int irq_offset =NUM_SMMU_IRQS ; +hwaddr offset =SMMU_IO_LEN ; +int irq, num_smmus =0 ; +hwaddr base;AcpiTable table = { .sig ="IORT" , .rev =5 , .oem_id = vms->oem_id, .oem_table_id = vms->oem_table_id };/* Table 2 The IORT */ acpi_table_begin (&table, table_data); +if (vms->num_nested_smmus) { + irq = vms->irqmap[VIRT_NESTED_SMMU ] +ARM_SPI_BASE ; + base = vms->memmap[VIRT_NESTED_SMMU ].base; + num_smmus = vms->num_nested_smmus; + }else if (virt_has_smmuv3 (vms)) { + irq = vms->irqmap[VIRT_SMMU ] +ARM_SPI_BASE ; + base = vms->memmap[VIRT_SMMU ].base; + num_smmus =1 ; + } + smmu_offset =g_new0 (size_t, num_smmus); + + nb_nodes =2 ;/* RC, ITS */ + nb_nodes += num_smmus;/* SMMU nodes */ +if (virt_has_smmuv3 (vms)) {AcpiIortIdMapping next_range = {0 };object_child_foreach_recursive (object_get_root (), iort_host_bridges, &idmap_vm); - nb_nodes =3 ;/* RC, ITS, SMMUv3 */ /* Sort the smmu idmap by input_base */ g_array_sort (smmu_idmaps, iort_idmap_compare);
@@ -394,7 +411,6 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) rc_mapping_count = smmu_idmaps->len + its_idmaps->len; }else { - nb_nodes =2 ;/* RC, ITS */ rc_mapping_count =1 ; }/* Number of IORT Nodes */
@@ -416,10 +432,9 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)/* GIC ITS Identifier Array */ build_append_int_noprefix (table_data,0 /* MADT translation_id */ ,4 ); -if (virt_has_smmuv3 (vms)) { -int irq = vms->irqmap[VIRT_SMMU ] +ARM_SPI_BASE ; +for (i =0 ; i < num_smmus; i++) { + smmu_offset[i] = table_data->len - table.table_offset; - smmu_offset = table_data->len - table.table_offset;/* Table 9 SMMUv3 Format */ build_append_int_noprefix (table_data,4 /* SMMUv3 */ ,1 );/* Type */ node_size =SMMU_V3_ENTRY_SIZE +ID_MAPPING_ENTRY_SIZE ;
@@ -430,12 +445,13 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)/* Reference to ID Array */ build_append_int_noprefix (table_data,SMMU_V3_ENTRY_SIZE ,4 );/* Base address */ -build_append_int_noprefix (table_data, vms->memmap[VIRT_SMMU ].base,8 ); +build_append_int_noprefix (table_data, base + i * offset,8 );/* Flags */ build_append_int_noprefix (table_data,1 /* COHACC Override */ ,4 );build_append_int_noprefix (table_data,0 ,4 );/* Reserved */ build_append_int_noprefix (table_data,0 ,8 );/* VATOS address */ /* Model */ + irq += irq_offset;build_append_int_noprefix (table_data,0 /* Generic SMMU-v3 */ ,4 );build_append_int_noprefix (table_data, irq,4 );/* Event */ build_append_int_noprefix (table_data, irq +1 ,4 );/* PRI */
@@ -487,7 +503,7 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) range = &g_array_index (smmu_idmaps, AcpiIortIdMapping, i);/* output IORT node is the smmuv3 node */ build_iort_id_mapping (table_data, range->input_base, - range->id_count, smmu_offset,0 ); + range->id_count, smmu_offset[i],0 ); }/* bypassed RIDs connect to ITS group node directly: RC -> ITS */
Signature · 1 line
2.43.0
RFC PATCH 10/10 hw/arm/virt-acpi-build: Enable ATS for nested SMMUv3
In reply to Nicolin ChenFor a nested SMMUv3, the ATS capaiblity is decided by the underlying HW, and then reflected in the IDR0 register of the vSMMU.
The IORT on the other hand could allow it to be always enabled, relying on the guest-level SMMU kernel driver to disable ATS feature if the ATS bit isn't set in IDR0.
--- hw/arm/virt-acpi-build.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
hw/arm/virt-acpi-build.c +5 -1
@@ -485,7 +485,11 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)/* Table 15 Memory Access Flags */ build_append_int_noprefix (table_data,0x3 /* CCA = CPM = DACS = 1 */ ,1 ); -build_append_int_noprefix (table_data,0 ,4 );/* ATS Attribute */ +if (vms->iommu ==VIRT_IOMMU_NESTED_SMMUV3 ) { +build_append_int_noprefix (table_data,1 ,4 );/* ATS Attribute */ + }else { +build_append_int_noprefix (table_data,0 ,4 );/* ATS Attribute */ + }/* MCFG pci_segment */ build_append_int_noprefix (table_data,0 ,4 );/* PCI Segment number */
Signature · 1 line
2.43.0