Skip to contentSkip to conversation

amd_iommu: support up to 2048 MSI vectors per IRT

RFC PATCH In review Sairaj Kodilkar 2 participants 13 messages Nov 18, 2025 – Jan 28 KVM

lore
Share & tools
lore
#0 lore raw

cover amd_iommu: support up to 2048 MSI vectors per IRT

Resending this series with KVM and IOMMU maintainers in CC.

AMD IOMMU can route upto 2048 MSI vectors through a single Interrupt Remapping Table (IRT) entry. This series brings the same capability to the emulated AMD IOMMU in QEMU.

Highlights
----------
* Sets bits [9:8] in Extended-Feature-Register-2 to advertise 2K MSI support to the guest.
* Uses bits [10:0] of the MSI data to select the IRTE when the guest programs MSIs in logical-destination mode.
* Introduces a new IOMMU device property:

        -device amd-iommu,...,numint2k=on

The feature is **opt-in**; guests keep the 512-MSI behaviour unless numint2k=on is supplied.

Passthrough devices
-------------------
When a PCI function is passed through via iommufd the code checks the host’s vendor capabilities. If the host IOMMU has not enabled 2K-MSI support (bits [44:43] set in the control register) the guest feature is disabled even if numint2k=on was requested.

The detection logic relies on the iommufd interface; with the legacy VFIO container the guest always falls back to 512 MSIs.

Example
-------
qemu-system-x86_64 \
-enable-kvm -m 10G -smp cpus=8 \
-kernel /boot/vmlinuz \
-initrd /boot/initrd.img \
-append "console=ttyS0 earlyprintk=serial root=<DEVICE>"
-device amd-iommu,dma-remap=on,numint2k=on \
-object iommufd,id=iommufd0 \
-device vfio-pci,host=<DEVID>,iommufd=iommufd0 \
-global kvm-pit.lost_tick_policy=discard \
-cpu host \
-machine q35,kernel_irqchip=split \
-nographic \
-smbios type=0,version=2.8 \
-blockdev node-name=drive0,driver=qcow2,file.driver=file,file.filename=<IMAGE> \
-device virtio-blk-pci,drive=drive0

Limitations
-----------
This approach works well for features queried after IOMMUFD initialization but cannot handle features needed during early QEMU setup, before IOMMUFD is available.

A key example is EFR2[HTRangeIgnore]. When this bit is set, the physical IOMMU treats HyperTransport (HT) address ranges as regular memory accesses rather than reserved regions. This has important implications for memory layout:

* Without HTRangeIgnore: QEMU must relocate RAM above 4G to above 1T on AMD platforms to avoid HT conflicts
* With HTRangeIgnore: QEMU can safely place RAM immediately above 4G, improving memory utilization

Since RAM layout must be determined before IOMMUFD initialization, QEMU cannot use hwinfo to query EFR2[HTRangeIgnore] feature bit.

Another limitation with using the control register is that, if BIOS enables particular feature (e.g. ControlRegister[GCR3TRPMode) without kernel support QEMU incorrectly assumes that host kernel supports that feature potentially causing guest failure.

Alternative considered
----------------------
We also explored alternate approach which uses KVM capability "KVM_CAP_AMD_NUM_INT_2K_SUP", which user can query to know if host kernel supports 2K MSIs. Similarly, this enables qemu to detect the presence of EFR2[HTRangeIgnore] during RAM initialization.

Although current implementation allows 2K MSI support only with iommufd, it keeps the logic inside the vfio/iommufd and avoids modifying KVM ABI. I am happy to discuss advantages and drawbacks of both approaches.

------------------------------------------------------------------------

The patches are based on top of bc831f37398b (qemu master). Additionally it requires linux kernel with patches[1] which expose control register via IOMMU_GET_HW_INFO ioctl.

[1] https://lore.kernel.org/linux-iommu/[email protected]/

------------------------------------------------------------------------

Sairaj Kodilkar (3):
vfio/iommufd: Add amd specific hardware info struct to vendor
capability
amd_iommu: Add support for extended feature register 2
amd_iommu: Add support for upto 2048 interrupts per IRT

Suravee Suthikulpanit (2):
[DO NOT MERGE] linux-headers: Introduce struct iommu_hw_info_amd
amd-iommu: Add support for set/unset IOMMU for VFIO PCI devices

 hw/i386/acpi-build.c               |   4 +-
 hw/i386/amd_iommu-stub.c           |   5 +
 hw/i386/amd_iommu.c                | 163 +++++++++++++++++++++++++++--
 hw/i386/amd_iommu.h                |  24 +++++
 include/system/host_iommu_device.h |   1 +
 linux-headers/linux/iommufd.h      |  20 ++++
 6 files changed, 207 insertions(+), 10 deletions(-)
Signature · 1 line
2.34.1
#1 lore raw

RFC PATCH 1/5 linux-headers: Introduce struct iommu_hw_info_amd

From: Suravee Suthikulpanit <[email protected]>

---
 linux-headers/linux/iommufd.h | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
linux-headers/linux/iommufd.h +20 -0
index 2105a039551e..c5fcb0396a38 100644
@@ -613,6 +613,24 @@ struct iommu_hw_info_tegra241_cmdqv {
 	__u8 __reserved;
 };
 
+/* struct iommu_hw_info_amd - AMD IOMMU device info
+ *
+ * @efr : Value of AMD IOMMU Extended Feature Register (EFR)
+ * @efr2: Value of AMD IOMMU Extended Feature 2 Register (EFR2)
+ *
+ * Please See description of these registers in the following sections of
+ * the AMD I/O Virtualization Technology (IOMMU) Specification.
+ * (https://www.amd.com/content/dam/amd/en/documents/processor-tech-docs/specifications/48882_IOMMU.pdf)
+ *
+ * - MMIO Offset 0030h IOMMU Extended Feature Register
+ * - MMIO Offset 01A0h IOMMU Extended Feature 2 Register
+ */
+struct iommu_hw_info_amd {
+	__aligned_u64 efr;
+	__aligned_u64 efr2;
+	__aligned_u64 control_register;
+};
+
 /**
  * enum iommu_hw_info_type - IOMMU Hardware Info Types
  * @IOMMU_HW_INFO_TYPE_NONE: Output by the drivers that do not report hardware
@@ -622,6 +640,7 @@ struct iommu_hw_info_tegra241_cmdqv {
  * @IOMMU_HW_INFO_TYPE_ARM_SMMUV3: ARM SMMUv3 iommu info type
  * @IOMMU_HW_INFO_TYPE_TEGRA241_CMDQV: NVIDIA Tegra241 CMDQV (extension for ARM
  *                                     SMMUv3) info type
+ * @IOMMU_HW_INFO_TYPE_AMD: AMD IOMMU info type
  */
 enum iommu_hw_info_type {
 	IOMMU_HW_INFO_TYPE_NONE = 0,
@@ -629,6 +648,7 @@ enum iommu_hw_info_type {
 	IOMMU_HW_INFO_TYPE_INTEL_VTD = 1,
 	IOMMU_HW_INFO_TYPE_ARM_SMMUV3 = 2,
 	IOMMU_HW_INFO_TYPE_TEGRA241_CMDQV = 3,
+	IOMMU_HW_INFO_TYPE_AMD = 4,
 };
 
 /**
Signature · 1 line
2.34.1
#2 lore raw

RFC PATCH 2/5 vfio/iommufd: Add amd specific hardware info struct to vendor capability

In reply to Sairaj Kodilkar

Update the vendor capability structure to have struct iommu_hw_info_amd. AMD vIOMMU can use this to determine hardware features supported by the host IOMMU.

---
 include/system/host_iommu_device.h | 1 +
 1 file changed, 1 insertion(+)
include/system/host_iommu_device.h +1 -0
index ab849a4a82d5..cb1745b97a09 100644
@@ -20,6 +20,7 @@
 typedef union VendorCaps {
     struct iommu_hw_info_vtd vtd;
     struct iommu_hw_info_arm_smmuv3 smmuv3;
+    struct iommu_hw_info_amd amd;
 } VendorCaps;
 
 /**
Signature · 1 line
2.34.1
#3 lore raw

Re: [RFC PATCH RESEND 2/5] vfio/iommufd: Add amd specific hardware info struct to vendor capability

nit: Capitalize AMD on commit subject line?

Quoted 21 lines · Sairaj Kodilkar

Update the vendor capability structure to have struct iommu_hw_info_amd. AMD vIOMMU can use this to determine hardware features supported by the host IOMMU.

---
 include/system/host_iommu_device.h | 1 +
 1 file changed, 1 insertion(+)
include/system/host_iommu_device.h +1 -0
index ab849a4a82d5..cb1745b97a09 100644
@@ -20,6 +20,7 @@
 typedef union VendorCaps {
     struct iommu_hw_info_vtd vtd;
     struct iommu_hw_info_arm_smmuv3 smmuv3;
+    struct iommu_hw_info_amd amd;
 } VendorCaps;
 
 /**
#4 lore raw

RFC PATCH 3/5 amd-iommu: Add support for set/unset IOMMU for VFIO PCI devices

In reply to Sairaj Kodilkar

From: Suravee Suthikulpanit <[email protected]>

"Set" function tracks VFIO devices in the hash table. This is useful when looking up per-device host IOMMU information later on.

---
 hw/i386/amd_iommu.c | 71 +++++++++++++++++++++++++++++++++++++++++++++
 hw/i386/amd_iommu.h |  8 +++++
 2 files changed, 79 insertions(+)
hw/i386/amd_iommu.c +71 -0
index 378e0cb55eab..8b146f4d33d2 100644
@@ -382,6 +382,22 @@ static guint amdvi_uint64_hash(gconstpointer v)
     return (guint)*(const uint64_t *)v;
 }
 
+static guint amdvi_dte_hash(gconstpointer v)
+{
+    const struct AMDVI_dte_key *key = v;
+    guint value = (guint)(uintptr_t)key->bus;
+
+    return (guint)(value << 8 | key->devfn);
+}
+
+static gboolean amdvi_dte_equal(gconstpointer v1, gconstpointer v2)
+{
+    const struct AMDVI_dte_key *key1 = v1;
+    const struct AMDVI_dte_key *key2 = v2;
+
+    return (key1->bus == key2->bus) && (key1->devfn == key2->devfn);
+}
+
 static AMDVIIOTLBEntry *amdvi_iotlb_lookup(AMDVIState *s, hwaddr addr,
                                            uint64_t devid)
 {
@@ -2291,8 +2307,60 @@ static AddressSpace *amdvi_host_dma_iommu(PCIBus *bus, void *opaque, int devfn)
     return &iommu_as[devfn]->as;
 }
 
+static bool amdvi_set_iommu_device(PCIBus *bus, void *opaque, int devfn,
+                                   HostIOMMUDevice *hiod, Error **errp)
+{
+    AMDVIState *s = opaque;
+    struct AMDVI_dte_key *new_key;
+    struct AMDVI_dte_key key = {
+        .bus = bus,
+        .devfn = devfn,
+    };
+
+    assert(hiod);
+    assert(0 <= devfn && devfn < PCI_DEVFN_MAX);
+
+    if (g_hash_table_lookup(s->hiod_hash, &key)) {
+        error_setg(errp, "Host IOMMU device already exist");
+        return false;
+    }
+
+    if (hiod->caps.type != IOMMU_HW_INFO_TYPE_AMD &&
+        hiod->caps.type != IOMMU_HW_INFO_TYPE_DEFAULT) {
+        error_setg(errp, "IOMMU hardware is not compatible");
+        return false;
+    }
+
+    new_key = g_malloc(sizeof(*new_key));
+    new_key->bus = bus;
+    new_key->devfn = devfn;
+
+    object_ref(hiod);
+    g_hash_table_insert(s->hiod_hash, new_key, hiod);
+
+    return true;
+}
+
+static void amdvi_unset_iommu_device(PCIBus *bus, void *opaque,
+                                     int devfn)
+{
+    AMDVIState *s = opaque;
+    struct AMDVI_dte_key key = {
+        .bus = bus,
+        .devfn = devfn,
+    };
+
+    if (!g_hash_table_lookup(s->hiod_hash, &key)) {
+        return;
+    }
+
+    g_hash_table_remove(s->hiod_hash, &key);
+}
+
 static const PCIIOMMUOps amdvi_iommu_ops = {
     .get_address_space = amdvi_host_dma_iommu,
+    .set_iommu_device = amdvi_set_iommu_device,
+    .unset_iommu_device = amdvi_unset_iommu_device,
 };
 
 static const MemoryRegionOps mmio_mem_ops = {
@@ -2510,6 +2578,9 @@ static void amdvi_sysbus_realize(DeviceState *dev, Error **errp)
     s->iotlb = g_hash_table_new_full(amdvi_uint64_hash,
                                      amdvi_uint64_equal, g_free, g_free);
 
+    s->hiod_hash = g_hash_table_new_full(amdvi_dte_hash,
+                                         amdvi_dte_equal, g_free, g_free);
+
     /* set up MMIO */
     memory_region_init_io(&s->mr_mmio, OBJECT(s), &mmio_mem_ops, s,
                           "amdvi-mmio", AMDVI_MMIO_SIZE);
hw/i386/amd_iommu.h +8 -0
index daf82fc85f96..e6f6902fe06d 100644
@@ -358,6 +358,11 @@ struct AMDVIPCIState {
     uint32_t capab_offset;       /* capability offset pointer    */
 };
 
+struct AMDVI_dte_key {
+    PCIBus *bus;
+    uint8_t devfn;
+};
+
 struct AMDVIState {
     X86IOMMUState iommu;        /* IOMMU bus device             */
     AMDVIPCIState *pci;         /* IOMMU PCI device             */
@@ -416,6 +421,9 @@ struct AMDVIState {
     /* IOTLB */
     GHashTable *iotlb;
 
+    /* HostIOMMUDevice hash table*/
+    GHashTable *hiod_hash;
+
     /* Interrupt remapping */
     bool ga_enabled;
     bool xtsup;
Signature · 1 line
2.34.1
#5 lore raw

Re: [RFC PATCH RESEND 3/5] amd-iommu: Add support for set/unset IOMMU for VFIO PCI devices

Hi,

Quoted 48 lines · Sairaj Kodilkar

From: Suravee Suthikulpanit <[email protected]>

"Set" function tracks VFIO devices in the hash table. This is useful when looking up per-device host IOMMU information later on.

---
 hw/i386/amd_iommu.c | 71 +++++++++++++++++++++++++++++++++++++++++++++
 hw/i386/amd_iommu.h |  8 +++++
 2 files changed, 79 insertions(+)
hw/i386/amd_iommu.c +21 -0
index 378e0cb55eab..8b146f4d33d2 100644
@@ -382,6 +382,22 @@ static guint amdvi_uint64_hash(gconstpointer v)
     return (guint)*(const uint64_t *)v;
 }
 
+static guint amdvi_dte_hash(gconstpointer v)
+{
+    const struct AMDVI_dte_key *key = v;
+    guint value = (guint)(uintptr_t)key->bus;
+
+    return (guint)(value << 8 | key->devfn);
+}
+
+static gboolean amdvi_dte_equal(gconstpointer v1, gconstpointer v2)
+{
+    const struct AMDVI_dte_key *key1 = v1;
+    const struct AMDVI_dte_key *key2 = v2;
+
+    return (key1->bus == key2->bus) && (key1->devfn == key2->devfn);
+}
+
 static AMDVIIOTLBEntry *amdvi_iotlb_lookup(AMDVIState *s, hwaddr addr,
                                            uint64_t devid)
 {
@@ -2291,8 +2307,60 @@ static AddressSpace *amdvi_host_dma_iommu(PCIBus *bus, void *opaque, int devfn)
     return &iommu_as[devfn]->as;
 }
 
+static bool amdvi_set_iommu_device(PCIBus *bus, void *opaque, int devfn,
+                                   HostIOMMUDevice *hiod, Error **errp)
+{
+    AMDVIState *s = opaque;
+    struct AMDVI_dte_key *new_key;

hw/i386/amd_iommu.c static AddressSpace *amdvi_host_dma_iommu(PCIBus *bus, void *opaque, int devfn) · line 2324

Quoted 10 lines
hw/i386/amd_iommu.c +10 -0
+    struct AMDVI_dte_key key = {
+        .bus = bus,
+        .devfn = devfn,
+    };
+
+    assert(hiod);
+    assert(0 <= devfn && devfn < PCI_DEVFN_MAX);
+
+    if (g_hash_table_lookup(s->hiod_hash, &key)) {
+        error_setg(errp, "Host IOMMU device already exist");

nit: s/exist/exists/

hw/i386/amd_iommu.c static AddressSpace *amdvi_host_dma_iommu(PCIBus *bus, void *opaque, int devfn) · line 2334

Quoted 10 lines
hw/i386/amd_iommu.c +10 -0
+        return false;
+    }
+
+    if (hiod->caps.type != IOMMU_HW_INFO_TYPE_AMD &&
+        hiod->caps.type != IOMMU_HW_INFO_TYPE_DEFAULT) {
+        error_setg(errp, "IOMMU hardware is not compatible");
+        return false;
+    }
+
+    new_key = g_malloc(sizeof(*new_key));

When allocating the new key, use g_new0() instead of g_malloc(), matches the current code better e.g.

new_key = g_new0(AMDVIHIODKey, 1);

*the AMDVIHIODKey type comes from a suggestion I make later.

Quoted 15 lines
hw/i386/amd_iommu.c +15 -0
+    new_key->bus = bus;
+    new_key->devfn = devfn;
+
+    object_ref(hiod);
+    g_hash_table_insert(s->hiod_hash, new_key, hiod);
+
+    return true;
+}
+
+static void amdvi_unset_iommu_device(PCIBus *bus, void *opaque,
+                                     int devfn)
+{
+    AMDVIState *s = opaque;
+    struct AMDVI_dte_key key = {
+        .bus = bus,

hw/i386/amd_iommu.c static AddressSpace *amdvi_host_dma_iommu(PCIBus *bus, void *opaque, int devfn) · line 2359

Quoted 10 lines
hw/i386/amd_iommu.c +10 -0
+        .devfn = devfn,
+    };
+
+    if (!g_hash_table_lookup(s->hiod_hash, &key)) {
+        return;
+    }
+
+    g_hash_table_remove(s->hiod_hash, &key);
+}
+

I think we have to explicitly decrement the reference count for the hiod object when removing the last entry from s->hiod_hash.

It looks like the best approach is to pass a custom value_destroy_func callback for it when calling g_hash_table_new_full() to create the table. Both the VT-d and virtio IOMMU implementations do it via that method.

Quoted 8 lines
hw/i386/amd_iommu.c +2 -0
 static const PCIIOMMUOps amdvi_iommu_ops = {
     .get_address_space = amdvi_host_dma_iommu,
+    .set_iommu_device = amdvi_set_iommu_device,
+    .unset_iommu_device = amdvi_unset_iommu_device,
 };
 
 static const MemoryRegionOps mmio_mem_ops = {

hw/i386/amd_iommu.c static void amdvi_sysbus_realize(DeviceState *dev, Error **errp) · line 2583

Quoted 6 lines
hw/i386/amd_iommu.c +3 -0
@@ -2510,6 +2578,9 @@ static void amdvi_sysbus_realize(DeviceState *dev, Error **errp)
     s->iotlb = g_hash_table_new_full(amdvi_uint64_hash,
                                      amdvi_uint64_equal, g_free, g_free);
 
+    s->hiod_hash = g_hash_table_new_full(amdvi_dte_hash,
+                                         amdvi_dte_equal, g_free, g_free);
+

As I mentioned above, I think the last parameter to g_hash_table_new_full() should be a custom destroy function with a call to:

object_unref((HostIOMMUDevice *)v);

Quoted 8 lines
hw/i386/amd_iommu.c +0 -0
     /* set up MMIO */
     memory_region_init_io(&s->mr_mmio, OBJECT(s), &mmio_mem_ops, s,
                           "amdvi-mmio", AMDVI_MMIO_SIZE);
hw/i386/amd_iommu.h +0 -0
index daf82fc85f96..e6f6902fe06d 100644

hw/i386/amd_iommu.h struct AMDVIPCIState { · line 365

Quoted 8 lines
hw/i386/amd_iommu.h +5 -0
@@ -358,6 +358,11 @@ struct AMDVIPCIState {
     uint32_t capab_offset;       /* capability offset pointer    */
 };
 
+struct AMDVI_dte_key {
+    PCIBus *bus;
+    uint8_t devfn;
+};
+

For consistency with earlier usage, use a typedef and CamelCase for the new AMDVI_dte_key definition i.e.

typedef struct AMDVIDTEKey {
    PCIBus *bus;
    uint8_t devfn;
} AMDVIDTEKey;

having it in the header file is best I think. I will send a patch moving other definitions to amd_iommu.h as well.

But I am not sure that using "dte" in this case is the best choice. I had this comment written for another section, fits better here:

hiod_hash and amdvi_dte_hash() should probably use a similar naming to signal their relationship. Maybe they can all be 'hiod' based i.e. amdvi_hiod_hash().
This seems to be the choice the VT-d implementation made, and it also signals we are using the same HostIOMMUDevice abstraction/model. I get that the device the HostIOMMUDevice represents is identified by a unique DTE on the host side IOMMU structures, so I am not arguing the naming is incorrect, but since we also have many places in the code that act on the guest DTE (e.g. amdvi_get_dte()), it would be better to avoid overloading 'dte' to avoid confusion.

If the above makes sense, then we should also use typedef struct AMDVIHIODKey instead...

Thank you,
Alejandro

Quoted 13 lines
hw/i386/amd_iommu.h +3 -0
 struct AMDVIState {
     X86IOMMUState iommu;        /* IOMMU bus device             */
     AMDVIPCIState *pci;         /* IOMMU PCI device             */
@@ -416,6 +421,9 @@ struct AMDVIState {
     /* IOTLB */
     GHashTable *iotlb;
 
+    /* HostIOMMUDevice hash table*/
+    GHashTable *hiod_hash;
+
     /* Interrupt remapping */
     bool ga_enabled;
     bool xtsup;
#6 lore raw

Re: [RFC PATCH RESEND 3/5] amd-iommu: Add support for set/unset IOMMU for VFIO PCI devices

Quoted 46 lines · Alejandro Jimenez

Hi,

Quoted 43 lines · Sairaj Kodilkar

From: Suravee Suthikulpanit <[email protected]>

"Set" function tracks VFIO devices in the hash table. This is useful when looking up per-device host IOMMU information later on.

---
  hw/i386/amd_iommu.c | 71 +++++++++++++++++++++++++++++++++++++++++++++
  hw/i386/amd_iommu.h |  8 +++++
  2 files changed, 79 insertions(+)
hw/i386/amd_iommu.c +16 -0
index 378e0cb55eab..8b146f4d33d2 100644
@@ -382,6 +382,22 @@ static guint amdvi_uint64_hash(gconstpointer v)
      return (guint)*(const uint64_t *)v;
  }
 
+static guint amdvi_dte_hash(gconstpointer v)
+{
+    const struct AMDVI_dte_key *key = v;
+    guint value = (guint)(uintptr_t)key->bus;
+
+    return (guint)(value << 8 | key->devfn);
+}
+
+static gboolean amdvi_dte_equal(gconstpointer v1, gconstpointer v2)
+{
+    const struct AMDVI_dte_key *key1 = v1;
+    const struct AMDVI_dte_key *key2 = v2;
+
+    return (key1->bus == key2->bus) && (key1->devfn == key2->devfn);
+}
+
  static AMDVIIOTLBEntry *amdvi_iotlb_lookup(AMDVIState *s, hwaddr addr,
                                             uint64_t devid)
  {
@@ -2291,8 +2307,60 @@ static AddressSpace *amdvi_host_dma_iommu(PCIBus *bus, void *opaque, int devfn)
      return &iommu_as[devfn]->as;
  }
 
Quoted 16 lines
Quoted 5 lines
hw/i386/amd_iommu.c +5 -0
+static bool amdvi_set_iommu_device(PCIBus *bus, void *opaque, int devfn,
+                                   HostIOMMUDevice *hiod, Error **errp)
+{
+    AMDVIState *s = opaque;
+    struct AMDVI_dte_key *new_key;

hw/i386/amd_iommu.c static AddressSpace *amdvi_host_dma_iommu(PCIBus *bus, void *opaque, int devfn) · line 2324

Quoted 10 lines
hw/i386/amd_iommu.c +10 -0
+    struct AMDVI_dte_key key = {
+        .bus = bus,
+        .devfn = devfn,
+    };
+
+    assert(hiod);
+    assert(0 <= devfn && devfn < PCI_DEVFN_MAX);
+
+    if (g_hash_table_lookup(s->hiod_hash, &key)) {
+        error_setg(errp, "Host IOMMU device already exist");

nit: s/exist/exists/

Sure

Quoted 13 lines

hw/i386/amd_iommu.c static AddressSpace *amdvi_host_dma_iommu(PCIBus *bus, void *opaque, int devfn) · line 2334

Quoted 10 lines
hw/i386/amd_iommu.c +10 -0
+        return false;
+    }
+
+    if (hiod->caps.type != IOMMU_HW_INFO_TYPE_AMD &&
+        hiod->caps.type != IOMMU_HW_INFO_TYPE_DEFAULT) {
+        error_setg(errp, "IOMMU hardware is not compatible");
+        return false;
+    }
+
+    new_key = g_malloc(sizeof(*new_key));

When allocating the new key, use g_new0() instead of g_malloc(), matches the current code better e.g.

Quoted 3 lines

new_key = g_new0(AMDVIHIODKey, 1);

*the AMDVIHIODKey type comes from a suggestion I make later.

Right

Quoted 28 lines
Quoted 15 lines
hw/i386/amd_iommu.c +15 -0
+    new_key->bus = bus;
+    new_key->devfn = devfn;
+
+    object_ref(hiod);
+    g_hash_table_insert(s->hiod_hash, new_key, hiod);
+
+    return true;
+}
+
+static void amdvi_unset_iommu_device(PCIBus *bus, void *opaque,
+                                     int devfn)
+{
+    AMDVIState *s = opaque;
+    struct AMDVI_dte_key key = {
+        .bus = bus,

hw/i386/amd_iommu.c static AddressSpace *amdvi_host_dma_iommu(PCIBus *bus, void *opaque, int devfn) · line 2359

Quoted 10 lines
hw/i386/amd_iommu.c +10 -0
+        .devfn = devfn,
+    };
+
+    if (!g_hash_table_lookup(s->hiod_hash, &key)) {
+        return;
+    }
+
+    g_hash_table_remove(s->hiod_hash, &key);
+}
+

I think we have to explicitly decrement the reference count for the hiod object when removing the last entry from s->hiod_hash.

Quoted 3 lines

It looks like the best approach is to pass a custom value_destroy_func callback for it when calling g_hash_table_new_full() to create the table. Both the VT-d and virtio IOMMU implementations do it via that method.

Good catch, Will do this !

Quoted 62 lines
Quoted 9 lines
hw/i386/amd_iommu.c +2 -0
  static const PCIIOMMUOps amdvi_iommu_ops = {
      .get_address_space = amdvi_host_dma_iommu,
+    .set_iommu_device = amdvi_set_iommu_device,
+    .unset_iommu_device = amdvi_unset_iommu_device,
  };
 
  static const MemoryRegionOps mmio_mem_ops = {

hw/i386/amd_iommu.c static void amdvi_sysbus_realize(DeviceState *dev, Error **errp) · line 2583

Quoted 6 lines
hw/i386/amd_iommu.c +3 -0
@@ -2510,6 +2578,9 @@ static void amdvi_sysbus_realize(DeviceState *dev, Error **errp)
      s->iotlb = g_hash_table_new_full(amdvi_uint64_hash,
                                       amdvi_uint64_equal, g_free, g_free);
 
+    s->hiod_hash = g_hash_table_new_full(amdvi_dte_hash,
+                                         amdvi_dte_equal, g_free, g_free);
+

As I mentioned above, I think the last parameter to g_hash_table_new_full() should be a custom destroy function with a call to:

object_unref((HostIOMMUDevice *)v);

Quoted 8 lines
hw/i386/amd_iommu.c +0 -0
      /* set up MMIO */
      memory_region_init_io(&s->mr_mmio, OBJECT(s), &mmio_mem_ops, s,
                            "amdvi-mmio", AMDVI_MMIO_SIZE);
hw/i386/amd_iommu.h +0 -0
index daf82fc85f96..e6f6902fe06d 100644

hw/i386/amd_iommu.h struct AMDVIPCIState { · line 365

Quoted 8 lines
hw/i386/amd_iommu.h +5 -0
@@ -358,6 +358,11 @@ struct AMDVIPCIState {
      uint32_t capab_offset;       /* capability offset pointer    */
  };
 
+struct AMDVI_dte_key {
+    PCIBus *bus;
+    uint8_t devfn;
+};
+

For consistency with earlier usage, use a typedef and CamelCase for the new AMDVI_dte_key definition i.e.

typedef struct AMDVIDTEKey {
     PCIBus *bus;
     uint8_t devfn;
} AMDVIDTEKey;

having it in the header file is best I think. I will send a patch moving other definitions to amd_iommu.h as well.

But I am not sure that using "dte" in this case is the best choice. I had this comment written for another section, fits better here:

hiod_hash and amdvi_dte_hash() should probably use a similar naming to signal their relationship. Maybe they can all be 'hiod' based i.e. amdvi_hiod_hash().
This seems to be the choice the VT-d implementation made, and it also signals we are using the same HostIOMMUDevice abstraction/model. I get that the device the HostIOMMUDevice represents is identified by a unique DTE on the host side IOMMU structures, so I am not arguing the naming is incorrect, but since we also have many places in the code that act on the guest DTE (e.g. amdvi_get_dte()), it would be better to avoid overloading 'dte' to avoid confusion.

Quoted 5 lines

If the above makes sense, then we should also use typedef struct AMDVIHIODKey instead...

Thank you,
Alejandro

Yeah I am in favour of HIOD base naming, will update it in V2

Thanks
Sairaj

Quoted 14 lines
Quoted 14 lines
hw/i386/amd_iommu.h +3 -0
  struct AMDVIState {
      X86IOMMUState iommu;        /* IOMMU bus device             */
      AMDVIPCIState *pci;         /* IOMMU PCI device             */
@@ -416,6 +421,9 @@ struct AMDVIState {
      /* IOTLB */
      GHashTable *iotlb;
 
+    /* HostIOMMUDevice hash table*/
+    GHashTable *hiod_hash;
+
      /* Interrupt remapping */
      bool ga_enabled;
      bool xtsup;
#7 lore raw

RFC PATCH 4/5 amd_iommu: Add support for extended feature register 2

In reply to Sairaj Kodilkar

Extended feature register 2 (EFR2) exposes newer IOMMU features such as NUM_INT_REMAP_SUP. Set MMIO offset 0x01A0 and ACPI table entry to EFR2.

---
 hw/i386/acpi-build.c     |  4 +++-
 hw/i386/amd_iommu-stub.c |  5 +++++
 hw/i386/amd_iommu.c      | 20 +++++++++++++++++---
 hw/i386/amd_iommu.h      |  4 ++++
 4 files changed, 29 insertions(+), 4 deletions(-)
hw/i386/acpi-build.c +3 -1
index 9446a9f862ca..1d4fd064e9a5 100644
@@ -1873,7 +1873,9 @@ build_amd_iommu(GArray *table_data, BIOSLinker *linker, const char *oem_id,
                               amdvi_extended_feature_register(s),
                               8);
     /* EFR Register Image 2 */
-    build_append_int_noprefix(table_data, 0, 8);
+    build_append_int_noprefix(table_data,
+                              amdvi_extended_feature_register2(s),
+                              8);
 
     /* IVHD entries as found above */
     g_array_append_vals(table_data, ivhd_blob->data, ivhd_blob->len);
hw/i386/amd_iommu-stub.c +5 -0
index d62a3732e60f..39b1afc0c751 100644
@@ -24,3 +24,8 @@ uint64_t amdvi_extended_feature_register(AMDVIState *s)
 {
     return AMDVI_DEFAULT_EXT_FEATURES;
 }
+
+uint64_t amdvi_extended_feature_register2(AMDVIState *s)
+{
+    return AMDVI_DEFAULT_EXT_FEATURES2;
+}
hw/i386/amd_iommu.c +17 -3
index 8b146f4d33d2..3221bf5a0303 100644
@@ -114,6 +114,11 @@ uint64_t amdvi_extended_feature_register(AMDVIState *s)
     return feature;
 }
 
+uint64_t amdvi_extended_feature_register2(AMDVIState *s)
+{
+    return AMDVI_DEFAULT_EXT_FEATURES2;
+}
+
 /* configure MMIO registers at startup/reset */
 static void amdvi_set_quad(AMDVIState *s, hwaddr addr, uint64_t val,
                            uint64_t romask, uint64_t w1cmask)
@@ -123,6 +128,16 @@ static void amdvi_set_quad(AMDVIState *s, hwaddr addr, uint64_t val,
     stq_le_p(&s->w1cmask[addr], w1cmask);
 }
 
+static void amdvi_refresh_efrs(struct AMDVIState *s)
+{
+    amdvi_set_quad(s, AMDVI_MMIO_EXT_FEATURES,
+                   amdvi_extended_feature_register(s),
+                   0xffffffffffffffef, 0);
+    amdvi_set_quad(s, AMDVI_MMIO_EXT_FEATURES2,
+                   amdvi_extended_feature_register2(s),
+                   0xffffffffffffffff, 0);
+}
+
 static uint16_t amdvi_readw(AMDVIState *s, hwaddr addr)
 {
     return lduw_le_p(&s->mmior[addr]);
@@ -2307,6 +2322,7 @@ static AddressSpace *amdvi_host_dma_iommu(PCIBus *bus, void *opaque, int devfn)
     return &iommu_as[devfn]->as;
 }
 
+
 static bool amdvi_set_iommu_device(PCIBus *bus, void *opaque, int devfn,
                                    HostIOMMUDevice *hiod, Error **errp)
 {
@@ -2434,9 +2450,7 @@ static void amdvi_init(AMDVIState *s)
 
     /* reset MMIO */
     memset(s->mmior, 0, AMDVI_MMIO_SIZE);
-    amdvi_set_quad(s, AMDVI_MMIO_EXT_FEATURES,
-                   amdvi_extended_feature_register(s),
-                   0xffffffffffffffef, 0);
+    amdvi_refresh_efrs(s);
     amdvi_set_quad(s, AMDVI_MMIO_STATUS, 0, 0x98, 0x67);
 }
 
hw/i386/amd_iommu.h +4 -0
index e6f6902fe06d..c8eaf229b50e 100644
@@ -57,6 +57,7 @@
 #define AMDVI_MMIO_EXCL_BASE          0x0020
 #define AMDVI_MMIO_EXCL_LIMIT         0x0028
 #define AMDVI_MMIO_EXT_FEATURES       0x0030
+#define AMDVI_MMIO_EXT_FEATURES2      0x01A0
 #define AMDVI_MMIO_COMMAND_HEAD       0x2000
 #define AMDVI_MMIO_COMMAND_TAIL       0x2008
 #define AMDVI_MMIO_EVENT_HEAD         0x2010
@@ -229,6 +230,8 @@
         AMDVI_FEATURE_IA | AMDVI_FEATURE_GT | AMDVI_FEATURE_HE | \
         AMDVI_GATS_MODE | AMDVI_HATS_MODE | AMDVI_FEATURE_GA)
 
+#define AMDVI_DEFAULT_EXT_FEATURES2 (0)
+
 /* capabilities header */
 #define AMDVI_CAPAB_FEATURES (AMDVI_CAPAB_FLAT_EXT | \
         AMDVI_CAPAB_FLAG_NPCACHE | AMDVI_CAPAB_FLAG_IOTLBSUP \
@@ -433,5 +436,6 @@ struct AMDVIState {
 };
 
 uint64_t amdvi_extended_feature_register(AMDVIState *s);
+uint64_t amdvi_extended_feature_register2(AMDVIState *s);
 
 #endif
Signature · 1 line
2.34.1
#8 lore raw

RFC PATCH 5/5 amd_iommu: Add support for upto 2048 interrupts per IRT

In reply to Sairaj Kodilkar

AMD IOMMU supports upto 2048 MSIs for a single device function when NUM_INT_REMAP_SUP Extended-Feature-Register-2 bit is set to one. Software can enable this feature by writing one to NUM_INT_REMAP_MODE in the control register. MSI address destination mode (DM) bit decides how many MSI data bits are used by IOMMU to index into IRT. When DM = 0, IOMMU uses bits 8:0 (max 512) for the index, otherwise (DM = 1) IOMMU uses bits 10:0 (max 2048) for IRT index.

This feature can be enabled with flag numint2k=on. In case of passhthrough devices viommu uses control register provided by vendor capabilites to determine if host IOMMU has enabled 2048 MSIs. If host IOMMU has not enabled it then the guest feature is disabled.

example command line
'''
-object iommufd,id=fd0 \
-device amd_iommu,dma-remap=on,numint2k=on \
-device vfio-host,host=<DEVID>,iommufd=fd0 \
'''

NOTE: In case of legacy VFIO container the guest will always fall back to 512 MSIs.

---
 hw/i386/amd_iommu.c | 74 ++++++++++++++++++++++++++++++++++++++++-----
 hw/i386/amd_iommu.h | 12 ++++++++
 2 files changed, 79 insertions(+), 7 deletions(-)
hw/i386/amd_iommu.c +67 -7
index 3221bf5a0303..4f62c4ee3671 100644
@@ -116,7 +116,12 @@ uint64_t amdvi_extended_feature_register(AMDVIState *s)
 
 uint64_t amdvi_extended_feature_register2(AMDVIState *s)
 {
-    return AMDVI_DEFAULT_EXT_FEATURES2;
+    uint64_t feature = AMDVI_DEFAULT_EXT_FEATURES2;
+    if (s->num_int_sup_2k) {
+        feature |= AMDVI_FEATURE_NUM_INT_REMAP_SUP;
+    }
+
+    return feature;
 }
 
 /* configure MMIO registers at startup/reset */
@@ -1538,6 +1543,9 @@ static void amdvi_handle_control_write(AMDVIState *s)
                         AMDVI_MMIO_CONTROL_CMDBUFLEN);
     s->ga_enabled = !!(control & AMDVI_MMIO_CONTROL_GAEN);
 
+    s->num_int_enabled = (control >> AMDVI_MMIO_CONTROL_NUM_INT_REMAP_SHIFT) &
+                         AMDVI_MMIO_CONTROL_NUM_INT_REMAP_MASK;
+
     /* update the flags depending on the control register */
     if (s->cmdbuf_enabled) {
         amdvi_assign_orq(s, AMDVI_MMIO_STATUS, AMDVI_MMIO_STATUS_CMDBUF_RUN);
@@ -2119,6 +2127,25 @@ static int amdvi_int_remap_msi(AMDVIState *iommu,
      * (page 5)
      */
     delivery_mode = (origin->data >> MSI_DATA_DELIVERY_MODE_SHIFT) & 7;
+    /*
+     * The MSI address register bit[2] is used to get the destination
+     * mode. The dest_mode 1 is valid for fixed and arbitrated interrupts
+     * and when IOMMU supports upto 2048 interrupts.
+     */
+    dest_mode = (origin->address >> MSI_ADDR_DEST_MODE_SHIFT) & 1;
+
+    if (dest_mode &&
+        iommu->num_int_enabled == AMDVI_MMIO_CONTROL_NUM_INT_REMAP_2K) {
+
+        trace_amdvi_ir_delivery_mode("2K interrupt mode");
+        ret = __amdvi_int_remap_msi(iommu, origin, translated, dte, &irq, sid);
+        if (ret < 0) {
+            goto remap_fail;
+        }
+        /* Translate IRQ to MSI messages */
+        x86_iommu_irq_to_msi_message(&irq, translated);
+        goto out;
+    }
 
     switch (delivery_mode) {
     case AMDVI_IOAPIC_INT_TYPE_FIXED:
@@ -2159,12 +2186,6 @@ static int amdvi_int_remap_msi(AMDVIState *iommu,
         goto remap_fail;
     }
 
-    /*
-     * The MSI address register bit[2] is used to get the destination
-     * mode. The dest_mode 1 is valid for fixed and arbitrated interrupts
-     * only.
-     */
-    dest_mode = (origin->address >> MSI_ADDR_DEST_MODE_SHIFT) & 1;
     if (dest_mode) {
         trace_amdvi_ir_err("invalid dest_mode");
         ret = -AMDVI_IR_ERR;
@@ -2322,6 +2343,30 @@ static AddressSpace *amdvi_host_dma_iommu(PCIBus *bus, void *opaque, int devfn)
     return &iommu_as[devfn]->as;
 }
 
+static void amdvi_refresh_efrs_hwinfo(struct AMDVIState *s,
+                                      struct iommu_hw_info_amd *hwinfo)
+{
+    /* Check if host OS has enabled 2K interrupts */
+    bool hwinfo_ctrl_2k;
+
+    if (s->num_int_sup_2k && !hwinfo) {
+        warn_report("AMDVI: Disabling 2048 MSI for guest, "
+                    "use IOMMUFD for device passthrough to support it");
+        s->num_int_sup_2k = 0;
+    }
+
+    hwinfo_ctrl_2k = ((hwinfo->control_register
+                       >> AMDVI_MMIO_CONTROL_NUM_INT_REMAP_SHIFT)
+                      & AMDVI_MMIO_CONTROL_NUM_INT_REMAP_2K);
+
+    if (s->num_int_sup_2k && !hwinfo_ctrl_2k) {
+        warn_report("AMDVI: Disabling 2048 MSIs for guest, "
+                    "as host kernel does not support this feature");
+        s->num_int_sup_2k = 0;
+    }
+
+    amdvi_refresh_efrs(s);
+}
 
 static bool amdvi_set_iommu_device(PCIBus *bus, void *opaque, int devfn,
                                    HostIOMMUDevice *hiod, Error **errp)
@@ -2354,6 +2399,20 @@ static bool amdvi_set_iommu_device(PCIBus *bus, void *opaque, int devfn,
     object_ref(hiod);
     g_hash_table_insert(s->hiod_hash, new_key, hiod);
 
+    if (hiod->caps.type == IOMMU_HW_INFO_TYPE_AMD) {
+        /*
+         * Refresh the MMIO efr registers so that changes are visible to the
+         * guest.
+         */
+        amdvi_refresh_efrs_hwinfo(s, &hiod->caps.vendor_caps.amd);
+    } else {
+        /*
+         * Pass NULL hardware registers when we have non-IOMMUFD
+         * passthrough device
+         */
+        amdvi_refresh_efrs_hwinfo(s, NULL);
+    }
+
     return true;
 }
 
@@ -2641,6 +2700,7 @@ static const Property amdvi_properties[] = {
     DEFINE_PROP_BOOL("xtsup", AMDVIState, xtsup, false),
     DEFINE_PROP_STRING("pci-id", AMDVIState, pci_id),
     DEFINE_PROP_BOOL("dma-remap", AMDVIState, dma_remap, false),
+    DEFINE_PROP_BOOL("numint2k", AMDVIState, num_int_sup_2k, false),
 };
 
 static const VMStateDescription vmstate_amdvi_sysbus = {
hw/i386/amd_iommu.h +12 -0
index c8eaf229b50e..588725fe0c25 100644
@@ -107,6 +107,9 @@
 #define AMDVI_MMIO_CONTROL_COMWAITINTEN   (1ULL << 4)
 #define AMDVI_MMIO_CONTROL_CMDBUFLEN      (1ULL << 12)
 #define AMDVI_MMIO_CONTROL_GAEN           (1ULL << 17)
+#define AMDVI_MMIO_CONTROL_NUM_INT_REMAP_MASK        (0x3)
+#define AMDVI_MMIO_CONTROL_NUM_INT_REMAP_SHIFT       (43)
+#define AMDVI_MMIO_CONTROL_NUM_INT_REMAP_2K          (0x1)
 
 /* MMIO status register bits */
 #define AMDVI_MMIO_STATUS_CMDBUF_RUN  (1 << 4)
@@ -160,6 +163,7 @@
 #define AMDVI_PERM_READ             (1 << 0)
 #define AMDVI_PERM_WRITE            (1 << 1)
 
+/* EFR */
 #define AMDVI_FEATURE_PREFETCH            (1ULL << 0) /* page prefetch       */
 #define AMDVI_FEATURE_PPR                 (1ULL << 1) /* PPR Support         */
 #define AMDVI_FEATURE_XT                  (1ULL << 2) /* x2APIC Support      */
@@ -169,6 +173,9 @@
 #define AMDVI_FEATURE_HE                  (1ULL << 8) /* hardware error regs */
 #define AMDVI_FEATURE_PC                  (1ULL << 9) /* Perf counters       */
 
+/* EFR2 */
+#define AMDVI_FEATURE_NUM_INT_REMAP_SUP   (1ULL << 8) /* 2K int support      */
+
 /* reserved DTE bits */
 #define AMDVI_DTE_QUAD0_RESERVED        (GENMASK64(6, 2) | GENMASK64(63, 63))
 #define AMDVI_DTE_QUAD1_RESERVED        0
@@ -380,6 +387,8 @@ struct AMDVIState {
     bool evtlog_enabled;         /* event log enabled            */
     bool excl_enabled;
 
+    uint8_t num_int_enabled;
+
     hwaddr devtab;               /* base address device table    */
     uint64_t devtab_len;         /* device table length          */
 
@@ -433,6 +442,9 @@ struct AMDVIState {
 
     /* DMA address translation */
     bool dma_remap;
+
+    /* upto 2048 interrupt support */
+    bool num_int_sup_2k;
 };
 
 uint64_t amdvi_extended_feature_register(AMDVIState *s);
Signature · 1 line
2.34.1
#9 lore raw

Re: [RFC PATCH RESEND 5/5] amd_iommu: Add support for upto 2048 interrupts per IRT

Quoted 103 lines · Sairaj Kodilkar

AMD IOMMU supports upto 2048 MSIs for a single device function when NUM_INT_REMAP_SUP Extended-Feature-Register-2 bit is set to one. Software can enable this feature by writing one to NUM_INT_REMAP_MODE in the control register. MSI address destination mode (DM) bit decides how many MSI data bits are used by IOMMU to index into IRT. When DM = 0, IOMMU uses bits 8:0 (max 512) for the index, otherwise (DM = 1) IOMMU uses bits 10:0 (max 2048) for IRT index.

This feature can be enabled with flag numint2k=on. In case of passhthrough devices viommu uses control register provided by vendor capabilites to determine if host IOMMU has enabled 2048 MSIs. If host IOMMU has not enabled it then the guest feature is disabled.

example command line
'''

(quoted diff) +0 -3
-object iommufd,id=fd0 \
-device amd_iommu,dma-remap=on,numint2k=on \
-device vfio-host,host=<DEVID>,iommufd=fd0 \

'''

NOTE: In case of legacy VFIO container the guest will always fall back to 512 MSIs.

---
 hw/i386/amd_iommu.c | 74 ++++++++++++++++++++++++++++++++++++++++-----
 hw/i386/amd_iommu.h | 12 ++++++++
 2 files changed, 79 insertions(+), 7 deletions(-)
hw/i386/amd_iommu.c +31 -7
index 3221bf5a0303..4f62c4ee3671 100644
@@ -116,7 +116,12 @@ uint64_t amdvi_extended_feature_register(AMDVIState *s)
 
 uint64_t amdvi_extended_feature_register2(AMDVIState *s)
 {
-    return AMDVI_DEFAULT_EXT_FEATURES2;
+    uint64_t feature = AMDVI_DEFAULT_EXT_FEATURES2;
+    if (s->num_int_sup_2k) {
+        feature |= AMDVI_FEATURE_NUM_INT_REMAP_SUP;
+    }
+
+    return feature;
 }
 
 /* configure MMIO registers at startup/reset */
@@ -1538,6 +1543,9 @@ static void amdvi_handle_control_write(AMDVIState *s)
                         AMDVI_MMIO_CONTROL_CMDBUFLEN);
     s->ga_enabled = !!(control & AMDVI_MMIO_CONTROL_GAEN);
 
+    s->num_int_enabled = (control >> AMDVI_MMIO_CONTROL_NUM_INT_REMAP_SHIFT) &
+                         AMDVI_MMIO_CONTROL_NUM_INT_REMAP_MASK;
+
     /* update the flags depending on the control register */
     if (s->cmdbuf_enabled) {
         amdvi_assign_orq(s, AMDVI_MMIO_STATUS, AMDVI_MMIO_STATUS_CMDBUF_RUN);
@@ -2119,6 +2127,25 @@ static int amdvi_int_remap_msi(AMDVIState *iommu,
      * (page 5)
      */
     delivery_mode = (origin->data >> MSI_DATA_DELIVERY_MODE_SHIFT) & 7;
+    /*
+     * The MSI address register bit[2] is used to get the destination
+     * mode. The dest_mode 1 is valid for fixed and arbitrated interrupts
+     * and when IOMMU supports upto 2048 interrupts.
+     */
+    dest_mode = (origin->address >> MSI_ADDR_DEST_MODE_SHIFT) & 1;
+
+    if (dest_mode &&
+        iommu->num_int_enabled == AMDVI_MMIO_CONTROL_NUM_INT_REMAP_2K) {
+
+        trace_amdvi_ir_delivery_mode("2K interrupt mode");
+        ret = __amdvi_int_remap_msi(iommu, origin, translated, dte, &irq, sid);
+        if (ret < 0) {
+            goto remap_fail;
+        }
+        /* Translate IRQ to MSI messages */
+        x86_iommu_irq_to_msi_message(&irq, translated);
+        goto out;
+    }
 
     switch (delivery_mode) {
     case AMDVI_IOAPIC_INT_TYPE_FIXED:
@@ -2159,12 +2186,6 @@ static int amdvi_int_remap_msi(AMDVIState *iommu,
         goto remap_fail;
     }
 
-    /*
-     * The MSI address register bit[2] is used to get the destination
-     * mode. The dest_mode 1 is valid for fixed and arbitrated interrupts
-     * only.
-     */
-    dest_mode = (origin->address >> MSI_ADDR_DEST_MODE_SHIFT) & 1;
     if (dest_mode) {
         trace_amdvi_ir_err("invalid dest_mode");
         ret = -AMDVI_IR_ERR;
@@ -2322,6 +2343,30 @@ static AddressSpace *amdvi_host_dma_iommu(PCIBus *bus, void *opaque, int devfn)
     return &iommu_as[devfn]->as;
 }
 
+static void amdvi_refresh_efrs_hwinfo(struct AMDVIState *s,
+                                      struct iommu_hw_info_amd *hwinfo)
+{

hw/i386/amd_iommu.c static AddressSpace *amdvi_host_dma_iommu(PCIBus *bus, void *opaque, int devfn) · line 2358

Quoted 10 lines
hw/i386/amd_iommu.c +10 -0
+    /* Check if host OS has enabled 2K interrupts */
+    bool hwinfo_ctrl_2k;
+
+    if (s->num_int_sup_2k && !hwinfo) {
+        warn_report("AMDVI: Disabling 2048 MSI for guest, "
+                    "use IOMMUFD for device passthrough to support it");
+        s->num_int_sup_2k = 0;
+    }
+
+    hwinfo_ctrl_2k = ((hwinfo->control_register

We need to check that hwinfo is a valid pointer before attempting to access any of its fields. The code in the line above causes a segfault in the common case where we are just using the default VFIO legacy backend and no new options.
Even when trying to use the new feature (numint2k=on) and iommufd backend in QEMU, if the host kernel was built with CONFIG_AMD_IOMMU_IOMMUFD=n (which is currently the default), the ioctl IOMMU_GET_HW_INFO will always return NULL data and hwinfo is also NULL at this point, so we crash and burn.

Quoted 20 lines
hw/i386/amd_iommu.c +13 -0
+                       >> AMDVI_MMIO_CONTROL_NUM_INT_REMAP_SHIFT)
+                      & AMDVI_MMIO_CONTROL_NUM_INT_REMAP_2K);
+
+    if (s->num_int_sup_2k && !hwinfo_ctrl_2k) {
+        warn_report("AMDVI: Disabling 2048 MSIs for guest, "
+                    "as host kernel does not support this feature");
+        s->num_int_sup_2k = 0;
+    }
+
+    amdvi_refresh_efrs(s);
+}
 
 static bool amdvi_set_iommu_device(PCIBus *bus, void *opaque, int devfn,
                                    HostIOMMUDevice *hiod, Error **errp)
@@ -2354,6 +2399,20 @@ static bool amdvi_set_iommu_device(PCIBus *bus, void *opaque, int devfn,
     object_ref(hiod);
     g_hash_table_insert(s->hiod_hash, new_key, hiod);
 
+    if (hiod->caps.type == IOMMU_HW_INFO_TYPE_AMD) {
+        /*

hw/i386/amd_iommu.c static bool amdvi_set_iommu_device(PCIBus *bus, void *opaque, int devfn, · line 2413

Quoted 10 lines
hw/i386/amd_iommu.c +10 -0
+         * Refresh the MMIO efr registers so that changes are visible to the
+         * guest.
+         */
+        amdvi_refresh_efrs_hwinfo(s, &hiod->caps.vendor_caps.amd);
+    } else {
+        /*
+         * Pass NULL hardware registers when we have non-IOMMUFD
+         * passthrough device
+         */
+        amdvi_refresh_efrs_hwinfo(s, NULL);

This call with hwinfo = NULL causes a segfault as I mentioned above. The code in amdvi_refresh_efrs_hwinfo() needs to be hardened.

Thank you,
Alejandro

Quoted 64 lines
hw/i386/amd_iommu.c +3 -0
+    }
+
     return true;
 }
 
@@ -2641,6 +2700,7 @@ static const Property amdvi_properties[] = {
     DEFINE_PROP_BOOL("xtsup", AMDVIState, xtsup, false),
     DEFINE_PROP_STRING("pci-id", AMDVIState, pci_id),
     DEFINE_PROP_BOOL("dma-remap", AMDVIState, dma_remap, false),
+    DEFINE_PROP_BOOL("numint2k", AMDVIState, num_int_sup_2k, false),
 };
 
 static const VMStateDescription vmstate_amdvi_sysbus = {
hw/i386/amd_iommu.h +12 -0
index c8eaf229b50e..588725fe0c25 100644
@@ -107,6 +107,9 @@
 #define AMDVI_MMIO_CONTROL_COMWAITINTEN   (1ULL << 4)
 #define AMDVI_MMIO_CONTROL_CMDBUFLEN      (1ULL << 12)
 #define AMDVI_MMIO_CONTROL_GAEN           (1ULL << 17)
+#define AMDVI_MMIO_CONTROL_NUM_INT_REMAP_MASK        (0x3)
+#define AMDVI_MMIO_CONTROL_NUM_INT_REMAP_SHIFT       (43)
+#define AMDVI_MMIO_CONTROL_NUM_INT_REMAP_2K          (0x1)
 
 /* MMIO status register bits */
 #define AMDVI_MMIO_STATUS_CMDBUF_RUN  (1 << 4)
@@ -160,6 +163,7 @@
 #define AMDVI_PERM_READ             (1 << 0)
 #define AMDVI_PERM_WRITE            (1 << 1)
 
+/* EFR */
 #define AMDVI_FEATURE_PREFETCH            (1ULL << 0) /* page prefetch       */
 #define AMDVI_FEATURE_PPR                 (1ULL << 1) /* PPR Support         */
 #define AMDVI_FEATURE_XT                  (1ULL << 2) /* x2APIC Support      */
@@ -169,6 +173,9 @@
 #define AMDVI_FEATURE_HE                  (1ULL << 8) /* hardware error regs */
 #define AMDVI_FEATURE_PC                  (1ULL << 9) /* Perf counters       */
 
+/* EFR2 */
+#define AMDVI_FEATURE_NUM_INT_REMAP_SUP   (1ULL << 8) /* 2K int support      */
+
 /* reserved DTE bits */
 #define AMDVI_DTE_QUAD0_RESERVED        (GENMASK64(6, 2) | GENMASK64(63, 63))
 #define AMDVI_DTE_QUAD1_RESERVED        0
@@ -380,6 +387,8 @@ struct AMDVIState {
     bool evtlog_enabled;         /* event log enabled            */
     bool excl_enabled;
 
+    uint8_t num_int_enabled;
+
     hwaddr devtab;               /* base address device table    */
     uint64_t devtab_len;         /* device table length          */
 
@@ -433,6 +442,9 @@ struct AMDVIState {
 
     /* DMA address translation */
     bool dma_remap;
+
+    /* upto 2048 interrupt support */
+    bool num_int_sup_2k;
 };
 
 uint64_t amdvi_extended_feature_register(AMDVIState *s);
#10 lore raw

Re: [RFC PATCH RESEND 5/5] amd_iommu: Add support for upto 2048 interrupts per IRT

Quoted 157 lines · Alejandro Jimenez
Quoted 103 lines · Sairaj Kodilkar

AMD IOMMU supports upto 2048 MSIs for a single device function when NUM_INT_REMAP_SUP Extended-Feature-Register-2 bit is set to one. Software can enable this feature by writing one to NUM_INT_REMAP_MODE in the control register. MSI address destination mode (DM) bit decides how many MSI data bits are used by IOMMU to index into IRT. When DM = 0, IOMMU uses bits 8:0 (max 512) for the index, otherwise (DM = 1) IOMMU uses bits 10:0 (max 2048) for IRT index.

This feature can be enabled with flag numint2k=on. In case of passhthrough devices viommu uses control register provided by vendor capabilites to determine if host IOMMU has enabled 2048 MSIs. If host IOMMU has not enabled it then the guest feature is disabled.

example command line
'''

(quoted diff) +0 -3
-object iommufd,id=fd0 \
-device amd_iommu,dma-remap=on,numint2k=on \
-device vfio-host,host=<DEVID>,iommufd=fd0 \

'''

NOTE: In case of legacy VFIO container the guest will always fall back to 512 MSIs.

---
  hw/i386/amd_iommu.c | 74 ++++++++++++++++++++++++++++++++++++++++-----
  hw/i386/amd_iommu.h | 12 ++++++++
  2 files changed, 79 insertions(+), 7 deletions(-)
hw/i386/amd_iommu.c +31 -7
index 3221bf5a0303..4f62c4ee3671 100644
@@ -116,7 +116,12 @@ uint64_t amdvi_extended_feature_register(AMDVIState *s)
 
  uint64_t amdvi_extended_feature_register2(AMDVIState *s)
  {
-    return AMDVI_DEFAULT_EXT_FEATURES2;
+    uint64_t feature = AMDVI_DEFAULT_EXT_FEATURES2;
+    if (s->num_int_sup_2k) {
+        feature |= AMDVI_FEATURE_NUM_INT_REMAP_SUP;
+    }
+
+    return feature;
  }
 
  /* configure MMIO registers at startup/reset */
@@ -1538,6 +1543,9 @@ static void amdvi_handle_control_write(AMDVIState *s)
                          AMDVI_MMIO_CONTROL_CMDBUFLEN);
      s->ga_enabled = !!(control & AMDVI_MMIO_CONTROL_GAEN);
 
+    s->num_int_enabled = (control >> AMDVI_MMIO_CONTROL_NUM_INT_REMAP_SHIFT) &
+                         AMDVI_MMIO_CONTROL_NUM_INT_REMAP_MASK;
+
      /* update the flags depending on the control register */
      if (s->cmdbuf_enabled) {
          amdvi_assign_orq(s, AMDVI_MMIO_STATUS, AMDVI_MMIO_STATUS_CMDBUF_RUN);
@@ -2119,6 +2127,25 @@ static int amdvi_int_remap_msi(AMDVIState *iommu,
       * (page 5)
       */
      delivery_mode = (origin->data >> MSI_DATA_DELIVERY_MODE_SHIFT) & 7;
+    /*
+     * The MSI address register bit[2] is used to get the destination
+     * mode. The dest_mode 1 is valid for fixed and arbitrated interrupts
+     * and when IOMMU supports upto 2048 interrupts.
+     */
+    dest_mode = (origin->address >> MSI_ADDR_DEST_MODE_SHIFT) & 1;
+
+    if (dest_mode &&
+        iommu->num_int_enabled == AMDVI_MMIO_CONTROL_NUM_INT_REMAP_2K) {
+
+        trace_amdvi_ir_delivery_mode("2K interrupt mode");
+        ret = __amdvi_int_remap_msi(iommu, origin, translated, dte, &irq, sid);
+        if (ret < 0) {
+            goto remap_fail;
+        }
+        /* Translate IRQ to MSI messages */
+        x86_iommu_irq_to_msi_message(&irq, translated);
+        goto out;
+    }
 
      switch (delivery_mode) {
      case AMDVI_IOAPIC_INT_TYPE_FIXED:
@@ -2159,12 +2186,6 @@ static int amdvi_int_remap_msi(AMDVIState *iommu,
          goto remap_fail;
      }
 
-    /*
-     * The MSI address register bit[2] is used to get the destination
-     * mode. The dest_mode 1 is valid for fixed and arbitrated interrupts
-     * only.
-     */
-    dest_mode = (origin->address >> MSI_ADDR_DEST_MODE_SHIFT) & 1;
      if (dest_mode) {
          trace_amdvi_ir_err("invalid dest_mode");
          ret = -AMDVI_IR_ERR;
@@ -2322,6 +2343,30 @@ static AddressSpace *amdvi_host_dma_iommu(PCIBus *bus, void *opaque, int devfn)
      return &iommu_as[devfn]->as;
  }
 
+static void amdvi_refresh_efrs_hwinfo(struct AMDVIState *s,
+                                      struct iommu_hw_info_amd *hwinfo)
+{

hw/i386/amd_iommu.c static AddressSpace *amdvi_host_dma_iommu(PCIBus *bus, void *opaque, int devfn) · line 2358

Quoted 10 lines
hw/i386/amd_iommu.c +10 -0
+    /* Check if host OS has enabled 2K interrupts */
+    bool hwinfo_ctrl_2k;
+
+    if (s->num_int_sup_2k && !hwinfo) {
+        warn_report("AMDVI: Disabling 2048 MSI for guest, "
+                    "use IOMMUFD for device passthrough to support it");
+        s->num_int_sup_2k = 0;
+    }
+
+    hwinfo_ctrl_2k = ((hwinfo->control_register

We need to check that hwinfo is a valid pointer before attempting to access any of its fields. The code in the line above causes a segfault in the common case where we are just using the default VFIO legacy backend and no new options.
Even when trying to use the new feature (numint2k=on) and iommufd backend in QEMU, if the host kernel was built with CONFIG_AMD_IOMMU_IOMMUFD=n (which is currently the default), the ioctl IOMMU_GET_HW_INFO will always return NULL data and hwinfo is also NULL at this point, so we crash and burn.

Quoted 20 lines
hw/i386/amd_iommu.c +13 -0
+                       >> AMDVI_MMIO_CONTROL_NUM_INT_REMAP_SHIFT)
+                      & AMDVI_MMIO_CONTROL_NUM_INT_REMAP_2K);
+
+    if (s->num_int_sup_2k && !hwinfo_ctrl_2k) {
+        warn_report("AMDVI: Disabling 2048 MSIs for guest, "
+                    "as host kernel does not support this feature");
+        s->num_int_sup_2k = 0;
+    }
+
+    amdvi_refresh_efrs(s);
+}
 
  static bool amdvi_set_iommu_device(PCIBus *bus, void *opaque, int devfn,
                                     HostIOMMUDevice *hiod, Error **errp)
@@ -2354,6 +2399,20 @@ static bool amdvi_set_iommu_device(PCIBus *bus, void *opaque, int devfn,
      object_ref(hiod);
      g_hash_table_insert(s->hiod_hash, new_key, hiod);
 
+    if (hiod->caps.type == IOMMU_HW_INFO_TYPE_AMD) {
+        /*

hw/i386/amd_iommu.c static bool amdvi_set_iommu_device(PCIBus *bus, void *opaque, int devfn, · line 2413

Quoted 10 lines
hw/i386/amd_iommu.c +10 -0
+         * Refresh the MMIO efr registers so that changes are visible to the
+         * guest.
+         */
+        amdvi_refresh_efrs_hwinfo(s, &hiod->caps.vendor_caps.amd);
+    } else {
+        /*
+         * Pass NULL hardware registers when we have non-IOMMUFD
+         * passthrough device
+         */
+        amdvi_refresh_efrs_hwinfo(s, NULL);

This call with hwinfo = NULL causes a segfault as I mentioned above. The code in amdvi_refresh_efrs_hwinfo() needs to be hardened.

Quoted 2 lines

Thank you,
Alejandro

Hi Alejandro

Good catch, I caught this but somehow forgot to fix. Thanks for pointing it out

Thanks
Sairaj

Quoted 64 lines
Quoted 64 lines
hw/i386/amd_iommu.c +3 -0
+    }
+
      return true;
  }
 
@@ -2641,6 +2700,7 @@ static const Property amdvi_properties[] = {
      DEFINE_PROP_BOOL("xtsup", AMDVIState, xtsup, false),
      DEFINE_PROP_STRING("pci-id", AMDVIState, pci_id),
      DEFINE_PROP_BOOL("dma-remap", AMDVIState, dma_remap, false),
+    DEFINE_PROP_BOOL("numint2k", AMDVIState, num_int_sup_2k, false),
  };
 
  static const VMStateDescription vmstate_amdvi_sysbus = {
hw/i386/amd_iommu.h +12 -0
index c8eaf229b50e..588725fe0c25 100644
@@ -107,6 +107,9 @@
  #define AMDVI_MMIO_CONTROL_COMWAITINTEN   (1ULL << 4)
  #define AMDVI_MMIO_CONTROL_CMDBUFLEN      (1ULL << 12)
  #define AMDVI_MMIO_CONTROL_GAEN           (1ULL << 17)
+#define AMDVI_MMIO_CONTROL_NUM_INT_REMAP_MASK        (0x3)
+#define AMDVI_MMIO_CONTROL_NUM_INT_REMAP_SHIFT       (43)
+#define AMDVI_MMIO_CONTROL_NUM_INT_REMAP_2K          (0x1)
 
  /* MMIO status register bits */
  #define AMDVI_MMIO_STATUS_CMDBUF_RUN  (1 << 4)
@@ -160,6 +163,7 @@
  #define AMDVI_PERM_READ             (1 << 0)
  #define AMDVI_PERM_WRITE            (1 << 1)
 
+/* EFR */
  #define AMDVI_FEATURE_PREFETCH            (1ULL << 0) /* page prefetch       */
  #define AMDVI_FEATURE_PPR                 (1ULL << 1) /* PPR Support         */
  #define AMDVI_FEATURE_XT                  (1ULL << 2) /* x2APIC Support      */
@@ -169,6 +173,9 @@
  #define AMDVI_FEATURE_HE                  (1ULL << 8) /* hardware error regs */
  #define AMDVI_FEATURE_PC                  (1ULL << 9) /* Perf counters       */
 
+/* EFR2 */
+#define AMDVI_FEATURE_NUM_INT_REMAP_SUP   (1ULL << 8) /* 2K int support      */
+
  /* reserved DTE bits */
  #define AMDVI_DTE_QUAD0_RESERVED        (GENMASK64(6, 2) | GENMASK64(63, 63))
  #define AMDVI_DTE_QUAD1_RESERVED        0
@@ -380,6 +387,8 @@ struct AMDVIState {
      bool evtlog_enabled;         /* event log enabled            */
      bool excl_enabled;
 
+    uint8_t num_int_enabled;
+
      hwaddr devtab;               /* base address device table    */
      uint64_t devtab_len;         /* device table length          */
 
@@ -433,6 +442,9 @@ struct AMDVIState {
 
      /* DMA address translation */
      bool dma_remap;
+
+    /* upto 2048 interrupt support */
+    bool num_int_sup_2k;
  };
 
  uint64_t amdvi_extended_feature_register(AMDVIState *s);
#11 lore raw
In reply to Sairaj Kodilkar

Hello all,

Gentle ping,

Quoted 110 lines · Sairaj Kodilkar

Resending this series with KVM and IOMMU maintainers in CC.

AMD IOMMU can route upto 2048 MSI vectors through a single Interrupt Remapping Table (IRT) entry. This series brings the same capability to the emulated AMD IOMMU in QEMU.

Highlights
----------
* Sets bits [9:8] in Extended-Feature-Register-2 to advertise 2K MSI support to the guest.
* Uses bits [10:0] of the MSI data to select the IRTE when the guest programs MSIs in logical-destination mode.
* Introduces a new IOMMU device property:

         -device amd-iommu,...,numint2k=on

The feature is **opt-in**; guests keep the 512-MSI behaviour unless numint2k=on is supplied.

Passthrough devices
-------------------
When a PCI function is passed through via iommufd the code checks the host’s vendor capabilities. If the host IOMMU has not enabled 2K-MSI support (bits [44:43] set in the control register) the guest feature is disabled even if numint2k=on was requested.

The detection logic relies on the iommufd interface; with the legacy VFIO container the guest always falls back to 512 MSIs.

Example
-------
qemu-system-x86_64 \

(quoted diff) +0 -14
-enable-kvm -m 10G -smp cpus=8 \
-kernel /boot/vmlinuz \
-initrd /boot/initrd.img \
-append "console=ttyS0 earlyprintk=serial root=<DEVICE>"
-device amd-iommu,dma-remap=on,numint2k=on \
-object iommufd,id=iommufd0 \
-device vfio-pci,host=<DEVID>,iommufd=iommufd0 \
-global kvm-pit.lost_tick_policy=discard \
-cpu host \
-machine q35,kernel_irqchip=split \
-nographic \
-smbios type=0,version=2.8 \
-blockdev node-name=drive0,driver=qcow2,file.driver=file,file.filename=<IMAGE> \
-device virtio-blk-pci,drive=drive0

Limitations
-----------
This approach works well for features queried after IOMMUFD initialization but cannot handle features needed during early QEMU setup, before IOMMUFD is available.

A key example is EFR2[HTRangeIgnore]. When this bit is set, the physical IOMMU treats HyperTransport (HT) address ranges as regular memory accesses rather than reserved regions. This has important implications for memory layout:

* Without HTRangeIgnore: QEMU must relocate RAM above 4G to above 1T on AMD platforms to avoid HT conflicts
* With HTRangeIgnore: QEMU can safely place RAM immediately above 4G, improving memory utilization

Since RAM layout must be determined before IOMMUFD initialization, QEMU cannot use hwinfo to query EFR2[HTRangeIgnore] feature bit.

Another limitation with using the control register is that, if BIOS enables particular feature (e.g. ControlRegister[GCR3TRPMode) without kernel support QEMU incorrectly assumes that host kernel supports that feature potentially causing guest failure.

Alternative considered
----------------------
We also explored alternate approach which uses KVM capability "KVM_CAP_AMD_NUM_INT_2K_SUP", which user can query to know if host kernel supports 2K MSIs. Similarly, this enables qemu to detect the presence of EFR2[HTRangeIgnore] during RAM initialization.

Although current implementation allows 2K MSI support only with iommufd, it keeps the logic inside the vfio/iommufd and avoids modifying KVM ABI. I am happy to discuss advantages and drawbacks of both approaches.

------------------------------------------------------------------------

The patches are based on top of bc831f37398b (qemu master). Additionally it requires linux kernel with patches[1] which expose control register via IOMMU_GET_HW_INFO ioctl.

[1] https://lore.kernel.org/linux-iommu/[email protected]/

------------------------------------------------------------------------

Sairaj Kodilkar (3):
vfio/iommufd: Add amd specific hardware info struct to vendor
capability
amd_iommu: Add support for extended feature register 2
amd_iommu: Add support for upto 2048 interrupts per IRT

Suravee Suthikulpanit (2):
[DO NOT MERGE] linux-headers: Introduce struct iommu_hw_info_amd
amd-iommu: Add support for set/unset IOMMU for VFIO PCI devices

  hw/i386/acpi-build.c               |   4 +-
  hw/i386/amd_iommu-stub.c           |   5 +
  hw/i386/amd_iommu.c                | 163 +++++++++++++++++++++++++++--
  hw/i386/amd_iommu.h                |  24 +++++
  include/system/host_iommu_device.h |   1 +
  linux-headers/linux/iommufd.h      |  20 ++++
  6 files changed, 207 insertions(+), 10 deletions(-)
#12 lore raw

Hi Sairaj,

Quoted 4 lines · Sairaj Kodilkar

Hello all,

Gentle ping,

I mentioned privately that I am investigating the main limitations/alternatives that are listed below, but I don't yet have any suggestions as to the best way to move forward. So while I am still hoping from feedback from others in the list, I'll reply to this series as if it will be merged using the current approach.

Thank you,
Alejandro

Quoted 114 lines · Sairaj Kodilkar
Quoted 112 lines

Resending this series with KVM and IOMMU maintainers in CC.

AMD IOMMU can route upto 2048 MSI vectors through a single Interrupt Remapping Table (IRT) entry. This series brings the same capability to the emulated AMD IOMMU in QEMU.

Highlights
----------
* Sets bits [9:8] in Extended-Feature-Register-2 to advertise 2K MSI support to the guest.
* Uses bits [10:0] of the MSI data to select the IRTE when the guest programs MSIs in logical-destination mode.
* Introduces a new IOMMU device property:
         -device amd-iommu,...,numint2k=on

   The feature is **opt-in**; guests keep the 512-MSI behaviour unless numint2k=on is supplied.

Passthrough devices
-------------------
When a PCI function is passed through via iommufd the code checks the host’s vendor capabilities.  If the host IOMMU has not enabled 2K-MSI support (bits [44:43] set in the control register) the guest feature is disabled even if numint2k=on was requested.

The detection logic relies on the iommufd interface; with the legacy VFIO container the guest always falls back to 512 MSIs.

Example
-------
qemu-system-x86_64 \

(quoted diff) +0 -13
-enable-kvm -m 10G -smp cpus=8 \
-kernel /boot/vmlinuz \
-initrd /boot/initrd.img \
-append "console=ttyS0 earlyprintk=serial root=<DEVICE>"
-device amd-iommu,dma-remap=on,numint2k=on \
-object iommufd,id=iommufd0 \
-device vfio-pci,host=<DEVID>,iommufd=iommufd0 \
-global kvm-pit.lost_tick_policy=discard \
-cpu host \
-machine q35,kernel_irqchip=split \
-nographic \
-smbios type=0,version=2.8 \
-blockdev node-

name=drive0,driver=qcow2,file.driver=file,file.filename=<IMAGE> \
-device virtio-blk-pci,drive=drive0

Limitations
-----------
This approach works well for features queried after IOMMUFD initialization but cannot handle features needed during early QEMU setup, before IOMMUFD is available.

A key example is EFR2[HTRangeIgnore]. When this bit is set, the physical IOMMU treats HyperTransport (HT) address ranges as regular memory accesses rather than reserved regions. This has important implications for memory layout:

* Without HTRangeIgnore: QEMU must relocate RAM above 4G to above 1T on AMD platforms to avoid HT conflicts
* With HTRangeIgnore: QEMU can safely place RAM immediately above 4G, improving memory utilization

Since RAM layout must be determined before IOMMUFD initialization, QEMU cannot use hwinfo to query EFR2[HTRangeIgnore] feature bit.

Another limitation with using the control register is that, if BIOS enables particular feature (e.g. ControlRegister[GCR3TRPMode) without kernel support QEMU incorrectly assumes that host kernel supports that feature potentially causing guest failure.

Alternative considered
----------------------
We also explored alternate approach which uses KVM capability "KVM_CAP_AMD_NUM_INT_2K_SUP", which user can query to know if host kernel supports 2K MSIs. Similarly, this enables qemu to detect the presence of EFR2[HTRangeIgnore] during RAM initialization.

Although current implementation allows 2K MSI support only with iommufd, it keeps the logic inside the vfio/iommufd and avoids modifying KVM ABI. I am happy to discuss advantages and drawbacks of both approaches.

------------------------------------------------------------------------

The patches are based on top of bc831f37398b (qemu master). Additionally it requires linux kernel with patches[1] which expose control register via IOMMU_GET_HW_INFO ioctl.

[1] https://lore.kernel.org/linux-iommu/20251029095846.4486-1-
[email protected]/

------------------------------------------------------------------------

Sairaj Kodilkar (3):
   vfio/iommufd: Add amd specific hardware info struct to vendor capability
   amd_iommu: Add support for extended feature register 2
   amd_iommu: Add support for upto 2048 interrupts per IRT

Suravee Suthikulpanit (2):
   [DO NOT MERGE] linux-headers: Introduce struct iommu_hw_info_amd
   amd-iommu: Add support for set/unset IOMMU for VFIO PCI devices

  hw/i386/acpi-build.c               |   4 +- hw/i386/amd_iommu-stub.c           |   5 + hw/i386/amd_iommu.c                | 163 +++++++++++++++++++++++++++-- hw/i386/amd_iommu.h                |  24 +++++ include/system/host_iommu_device.h |   1 + linux-headers/linux/iommufd.h      |  20 ++++

  6 files changed, 207 insertions(+), 10 deletions(-)