Skip to contentSkip to conversation

tests/qtest: add Intel IOMMU IOTLB invalidation test

PATCH New Junjie Cao 1 participant 4 messages Jul 3 Arm/Amlogic Meson SoC support

lore
Share & tools
lore

Junjie Cao

author
#0 lore raw

cover tests/qtest: add Intel IOMMU IOTLB invalidation test

The iommu-testdev framework lets a qtest drive an emulated vIOMMU without a guest, and Intel translation tests already build on it [1]. The next behaviour worth covering is invalidation: no in-tree qtest checks that any vIOMMU actually flushes a stale IOTLB entry. This series adds that for VT-d.

Each case populates the IOTLB (map IOVA->PA_A, DMA), rewrites the leaf PTE to PA_B without invalidating and confirms the stale entry still serves PA_A, then invalidates and confirms a fresh walk reaches PA_B. Without the stale check, a device that never cached would also pass. Cases cover {legacy, scalable-slt, scalable-flt} x {global, domain, page}, driving vtd_process_inv_desc -> vtd_process_iotlb_desc -> vtd_iotlb_{global,domain,page}_invalidate. Three more cases cache a second page to check page-selective scope: it survives for second-level and, as QEMU flushes first-stage entries domain-wide, is evicted for scalable-flt.

Patch 1 moves the VT-d setup helpers out of iommu-intel-test.c into the shared qos-intel-iommu module with no functional change; patch 2 adds the invalidation helpers; patch 3 adds the test.

scalable-flt uses the legacy VTD_INV_DESC_IOTLB descriptor, the path QEMU runs for its unified IOTLB; PASID-selective PIOTLB and domain-level selectivity are left as follow-ups.

[1] https://lore.kernel.org/qemu-devel/[email protected]/

Junjie Cao (3):
tests/qtest/libqos: share Intel IOMMU test setup helpers tests/qtest/libqos: add Intel IOMMU invalidation helpers tests/qtest: add IOTLB invalidation test for Intel IOMMU

 MAINTAINERS                          |   1 +
 tests/qtest/iommu-intel-inv-test.c   | 346 +++++++++++++++++++++++++++
 tests/qtest/iommu-intel-test.c       |  78 +-----
 tests/qtest/libqos/qos-intel-iommu.c | 185 ++++++++++++++
 tests/qtest/libqos/qos-intel-iommu.h |  74 ++++++
 tests/qtest/meson.build              |   2 +-
 6 files changed, 608 insertions(+), 78 deletions(-)
 create mode 100644 tests/qtest/iommu-intel-inv-test.c

base-commit: b4bdad7dcef623d2874df4e79e0be01075096c3b

Signature · 1 line
2.43.0
author
#1 lore raw

PATCH 1/3 tests/qtest/libqos: share Intel IOMMU test setup helpers

iommu-intel-test.c keeps the iommu-testdev PCI setup (save_fn(), setup_qtest_pci_device()) and the VT-d command-line / capability helpers (qvtd_iommu_args(), qvtd_check_caps()) as file-local statics. A second Intel IOMMU test would have to copy them, which defeats the purpose of the shared qos-intel-iommu module.

Move them into qos-intel-iommu so sibling tests can reuse them:
save_fn() becomes qvtd_save_pci_dev() and setup_qtest_pci_device() becomes qvtd_setup_qtest_pci_device(); qvtd_iommu_args() and qvtd_check_caps() keep their names.

No functional change: iommu-intel-test now calls the public qvtd_setup_qtest_pci_device() instead of its file-local copy.

---
 tests/qtest/iommu-intel-test.c       | 78 +---------------------------
 tests/qtest/libqos/qos-intel-iommu.c | 77 +++++++++++++++++++++++++++
 tests/qtest/libqos/qos-intel-iommu.h | 23 ++++++++
 3 files changed, 101 insertions(+), 77 deletions(-)
tests/qtest/iommu-intel-test.c +1 -77
index 703bbfef73..c2744def6f 100644
@@ -24,82 +24,6 @@ static uint64_t intel_iommu_expected_gpa(uint64_t iova)
     return (QVTD_PT_VAL & VTD_PAGE_MASK_4K) + (iova & 0xfff);
 }
 
-static void save_fn(QPCIDevice *dev, int devfn, void *data)
-{
-    QPCIDevice **pdev = (QPCIDevice **) data;
-
-    *pdev = dev;
-}
-
-static QPCIDevice *setup_qtest_pci_device(QTestState *qts, QPCIBus **pcibus,
-                                          QPCIBar *bar)
-{
-    QPCIDevice *dev = NULL;
-
-    *pcibus = qpci_new_pc(qts, NULL);
-    g_assert(*pcibus != NULL);
-
-    qpci_device_foreach(*pcibus, IOMMU_TESTDEV_VENDOR_ID,
-                        IOMMU_TESTDEV_DEVICE_ID, save_fn, &dev);
-
-    g_assert(dev);
-    qpci_device_enable(dev);
-    *bar = qpci_iomap(dev, 0, NULL);
-    g_assert_false(bar->is_io);
-
-    return dev;
-}
-
-static const char *qvtd_iommu_args(QVTDTransMode mode)
-{
-    switch (mode) {
-    case QVTD_TM_SCALABLE_FLT:
-        return "-device intel-iommu,scalable-mode=on,fsts=on ";
-    case QVTD_TM_SCALABLE_PT:
-    case QVTD_TM_SCALABLE_SLT:
-        return "-device intel-iommu,scalable-mode=on ";
-    default:
-        return "-device intel-iommu ";
-    }
-}
-
-static bool qvtd_check_caps(QTestState *qts, QVTDTransMode mode)
-{
-    uint64_t ecap = qtest_readq(qts,
-                                Q35_HOST_BRIDGE_IOMMU_ADDR + DMAR_ECAP_REG);
-
-    /* All scalable modes require SMTS */
-    if (qvtd_is_scalable(mode) && !(ecap & VTD_ECAP_SMTS)) {
-        g_test_skip("ECAP.SMTS not supported");
-        return false;
-    }
-
-    switch (mode) {
-    case QVTD_TM_SCALABLE_PT:
-        if (!(ecap & VTD_ECAP_PT)) {
-            g_test_skip("ECAP.PT not supported");
-            return false;
-        }
-        break;
-    case QVTD_TM_SCALABLE_SLT:
-        if (!(ecap & VTD_ECAP_SSTS)) {
-            g_test_skip("ECAP.SSTS not supported");
-            return false;
-        }
-        break;
-    case QVTD_TM_SCALABLE_FLT:
-        if (!(ecap & VTD_ECAP_FSTS)) {
-            g_test_skip("ECAP.FSTS not supported");
-            return false;
-        }
-        break;
-    default:
-        break;
-    }
-
-    return true;
-}
-
 static void run_intel_iommu_translation(const QVTDTestConfig *cfg)
 {
     QTestState *qts;
@@ -124,7 +48,7 @@ static void run_intel_iommu_translation(const QVTDTestConfig *cfg)
     }
 
     /* Setup and configure IOMMU-testdev PCI device */
-    dev = setup_qtest_pci_device(qts, &pcibus, &bar);
+    dev = qvtd_setup_qtest_pci_device(qts, &pcibus, &bar);
     g_assert(dev);
 
     g_test_message("### Intel IOMMU translation mode=%d ###", cfg->trans_mode);
tests/qtest/libqos/qos-intel-iommu.c +77 -0

86 lines not on this page · raw message

tests/qtest/libqos/qos-intel-iommu.h +23 -0
index c6cacc5c3f..dd6f920464 100644
@@ -182,4 +182,27 @@ void qvtd_run_translation_case(QTestState *qts, QPCIDevice *dev,
                                QPCIBar bar, uint64_t iommu_base,
                                const QVTDTestConfig *cfg);
 
+/*
+ * qvtd_iommu_args - Build the -device intel-iommu command-line fragment
+ *                   for the requested translation mode.
+ */
+const char *qvtd_iommu_args(QVTDTransMode mode);
+
+/*
+ * qvtd_check_caps - Check whether the running QEMU exposes the ECAP bits
+ *                   required by @mode.  Calls g_test_skip() and returns
+ *                   false when a required capability is missing.
+ */
+bool qvtd_check_caps(QTestState *qts, QVTDTransMode mode);
+
+/*
+ * qvtd_setup_qtest_pci_device - Create a QPCIBus, locate the iommu-testdev
+ *                               PCI function, enable it and map BAR0.
+ *
+ * On success, *pcibus and *bar are populated and the returned QPCIDevice
+ * must be released by the caller via g_free().
+ */
+QPCIDevice *qvtd_setup_qtest_pci_device(QTestState *qts, QPCIBus **pcibus,
+                                        QPCIBar *bar);
+
 #endif /* QTEST_LIBQOS_INTEL_IOMMU_H */
Signature · 1 line
2.43.0
author
#2 lore raw

PATCH 2/3 tests/qtest/libqos: add Intel IOMMU invalidation helpers

In reply to Junjie Cao

Add the building blocks a queued-invalidation test needs on top of the existing translation helpers:

- qvtd_leaf_pte_addr() / qvtd_make_leaf_pte() let a test locate and rewrite the leaf PTE built by qvtd_setup_translation_tables() without re-deriving the page-table index or leaf attributes by hand. The attributes reuse qvtd_get_fl_pte_attrs()/qvtd_get_pte_attrs() so the first- and second-level leaf formats stay defined in one place.

- qvtd_submit_iotlb_global_inv() / _domain_inv() / _page_inv() write an IOTLB Invalidation Descriptor (global / domain-selective / page-selective) into the Invalidation Queue and advance IQT_REG.

- qvtd_submit_inv_wait_and_poll() submits an Invalidation Wait Descriptor with Status Write and polls the status word with a bounded retry loop, asserting on timeout.

No caller yet; used by the IOTLB invalidation test that follows.

---
 tests/qtest/libqos/qos-intel-iommu.c | 108 +++++++++++++++++++++++++++
 tests/qtest/libqos/qos-intel-iommu.h |  51 +++++++++++++
 2 files changed, 159 insertions(+)
tests/qtest/libqos/qos-intel-iommu.c +108 -0
index 4095ac0917..c793ec899c 100644
@@ -15,6 +15,10 @@
 #include "qos-iommu-testdev.h"
 #include "qos-intel-iommu.h"
 
+/* Bounded poll for the invalidation-wait Status Write. */
+#define QVTD_INV_WAIT_POLL_MAX_ITERS    1000
+#define QVTD_INV_WAIT_POLL_INTERVAL_US  1000
+
 #define QVTD_AW_48BIT_ENCODING    2
 
 uint32_t qvtd_expected_dma_result(QVTDTestContext *ctx)
@@ -232,6 +236,30 @@ static uint64_t qvtd_get_fl_pte_attrs(bool is_leaf)
     return attrs;
 }
 
+uint64_t qvtd_leaf_pte_addr(uint64_t iova)
+{
+    return qvtd_get_table_addr(QVTD_PT_L1_BASE, 1, iova);
+}
+
+uint64_t qvtd_make_leaf_pte(uint64_t pa, QVTDTransMode mode)
+{
+    uint64_t attrs;
+
+    /*
+     * Reuse the same leaf attributes qvtd_setup_translation_tables() writes,
+     * so a rewritten PTE stays consistent with the initial mapping.  US=1 in
+     * the first-level format is required because the PASID entry runs with
+     * SRE=0, which makes every DMA appear as a user-mode access
+     * (VT-d 3.6.2 / 9.9).
+     */
+    if (mode == QVTD_TM_SCALABLE_FLT) {
+        attrs = qvtd_get_fl_pte_attrs(true);
+    } else {
+        attrs = qvtd_get_pte_attrs();
+    }
+    return (pa & VTD_PAGE_MASK_4K) | attrs;
+}
+
 void qvtd_setup_translation_tables(QTestState *qts, uint64_t iova,
                                    QVTDTransMode mode)
 {
@@ -529,3 +557,83 @@ QPCIDevice *qvtd_setup_qtest_pci_device(QTestState *qts, QPCIBus **pcibus,
 
     return dev;
 }
+
+/*
+ * Write a 128-bit invalidation descriptor at the current tail and advance
+ * IQT_REG.  Internal helper for the IOTLB / wait variants below.
+ */
+static uint32_t qvtd_submit_inv_desc(QTestState *qts, uint64_t iommu_base,
+                                     uint64_t desc_lo, uint64_t desc_hi,
+                                     uint32_t tail)
+{
+    uint64_t desc_addr = QVTD_IQ_BASE + (uint64_t)tail * QVTD_IQ_DESC_SIZE;
+
+    qtest_writeq(qts, desc_addr, desc_lo);
+    qtest_writeq(qts, desc_addr + 8, desc_hi);
+    tail++;
+
+    qtest_writeq(qts, iommu_base + DMAR_IQT_REG,
+                 (uint64_t)tail << QVTD_IQT_SHIFT);
+    return tail;
+}
+
+uint32_t qvtd_submit_inv_wait_and_poll(QTestState *qts, uint64_t iommu_base,
+                                       uint32_t tail)
+{
+    uint64_t lo, hi;
+    uint32_t status = 0;
+    int i;
+
+    qtest_writel(qts, QVTD_INV_WAIT_ADDR, 0);
+
+    lo = VTD_INV_DESC_WAIT | VTD_INV_DESC_WAIT_SW |
+         ((uint64_t)QVTD_INV_WAIT_DATA << VTD_INV_DESC_WAIT_DATA_SHIFT);
+    hi = QVTD_INV_WAIT_ADDR;
+
+    tail = qvtd_submit_inv_desc(qts, iommu_base, lo, hi, tail);
+
+    for (i = 0; i < QVTD_INV_WAIT_POLL_MAX_ITERS; i++) {
+        status = qtest_readl(qts, QVTD_INV_WAIT_ADDR);
+        if (status == QVTD_INV_WAIT_DATA) {
+            return tail;
+        }
+        g_usleep(QVTD_INV_WAIT_POLL_INTERVAL_US);
+    }
+
+    g_assert_cmphex(status, ==, QVTD_INV_WAIT_DATA);
+    return tail;
+}
+
+uint32_t qvtd_submit_iotlb_global_inv(QTestState *qts, uint64_t iommu_base,
+                                      uint32_t tail)
+{
+    uint64_t lo = VTD_INV_DESC_IOTLB | VTD_INV_DESC_IOTLB_GLOBAL;
+
+    return qvtd_submit_inv_desc(qts, iommu_base, lo, 0, tail);
+}
+
+uint32_t qvtd_submit_iotlb_domain_inv(QTestState *qts, uint64_t iommu_base,
+                                      uint16_t domain_id, uint32_t tail)
+{
+    uint64_t lo = VTD_INV_DESC_IOTLB | VTD_INV_DESC_IOTLB_DOMAIN |
+                  ((uint64_t)domain_id << 16);
+
+    return qvtd_submit_inv_desc(qts, iommu_base, lo, 0, tail);
+}
+
+uint32_t qvtd_submit_iotlb_page_inv(QTestState *qts, uint64_t iommu_base,
+                                    uint16_t domain_id, uint64_t addr,
+                                    uint8_t am, uint32_t tail)
+{
+    uint64_t lo = VTD_INV_DESC_IOTLB | VTD_INV_DESC_IOTLB_PAGE |
+                  ((uint64_t)domain_id << 16);
+    /*
+     * AM selects the invalidation range per VT-d 6.5.2.4:
+     *   am=0  → 4 KB,  am=9 → 2 MB,  am=18 → 1 GB.
+     * IH (hi[6]) is left clear, requesting full invalidation
+     * including non-leaf paging-structure caches.
+     */
+    uint64_t hi = (addr & ~0xfffULL) | (am & 0x3fULL);
+
+    return qvtd_submit_inv_desc(qts, iommu_base, lo, hi, tail);
+}
tests/qtest/libqos/qos-intel-iommu.h +51 -0

61 lines not on this page · raw message

Signature · 1 line
2.43.0
author
#3 lore raw

PATCH 3/3 tests/qtest: add IOTLB invalidation test for Intel IOMMU

In reply to Junjie Cao

Nothing in tree exercises IOTLB invalidation for any emulated vIOMMU:
the existing iommu-testdev tests only check one-shot translation, so a regression that failed to flush a stale IOTLB entry would go unnoticed.

Add a test that drives the queued-invalidation path end to end (vtd_process_inv_desc -> vtd_process_iotlb_desc -> vtd_iotlb_{global,domain,page}_invalidate). For each {legacy, scalable-slt, scalable-flt} x {global, domain, page} combination it:

1. maps IOVA -> PA_A and DMAs, populating the IOTLB;
2. rewrites the leaf PTE to PA_B *without* invalidating and DMAs

     again, asserting the stale entry is still served (MISMATCH);

3. submits the IOTLB invalidation plus a wait descriptor, then DMAs and asserts the fresh page walk now reaches PA_B.

Step 2 makes the flush observable: it fails loudly if the IOTLB is not actually caching the first translation.

For scalable first-level (flt), QEMU keeps first- and second-level mappings in a single IOTLB that the legacy VTD_INV_DESC_IOTLB descriptor flushes for every level, so the test uses that descriptor across all modes. PASID-selective invalidation (VTD_INV_DESC_PIOTLB, vtd_piotlb_*) is a separate path and is left for a follow-up.

It also adds three page-selective cases that cache a second page and check its fate after invalidating the first: for second-level (legacy, scalable-slt) the second page survives, while for first-level (scalable-flt) QEMU flushes all first-stage entries of the domain, so it does not. This distinguishes a page-selective flush from a domain-wide or global one.

---
 MAINTAINERS                        |   1 +
 tests/qtest/iommu-intel-inv-test.c | 346 +++++++++++++++++++++++++++++
 tests/qtest/meson.build            |   2 +-
 3 files changed, 348 insertions(+), 1 deletion(-)
 create mode 100644 tests/qtest/iommu-intel-inv-test.c
MAINTAINERS +1 -0
index 97dcc78ded..906de6e39f 100644
@@ -4103,6 +4103,7 @@ F: include/hw/i386/intel_iommu.h
 F: tests/functional/x86_64/test_intel_iommu.py
 F: tests/qtest/intel-iommu-test.c
 F: tests/qtest/iommu-intel-test.c
+F: tests/qtest/iommu-intel-inv-test.c
 
 AMD-Vi Emulation
 M: Alejandro Jimenez <[email protected]>
tests/qtest/iommu-intel-inv-test.c new file +346 -0

346 lines not on this page · raw message

tests/qtest/meson.build +1 -1
index 5e18b947c7..7da66802e1 100644
@@ -99,7 +99,7 @@ qtests_i386 = \
   (config_all_devices.has_key('CONFIG_AMD_IOMMU') ? ['amd-iommu-test'] : []) +             \
   (config_all_devices.has_key('CONFIG_VTD') ? ['intel-iommu-test'] : []) +                 \
   (config_all_devices.has_key('CONFIG_VTD') and
-   config_all_devices.has_key('CONFIG_IOMMU_TESTDEV') ? ['iommu-intel-test'] : []) +      \
+   config_all_devices.has_key('CONFIG_IOMMU_TESTDEV') ? ['iommu-intel-test', 'iommu-intel-inv-test'] : []) +      \
   (host_os != 'windows' and                                                                \
    config_all_devices.has_key('CONFIG_ACPI_ERST') ? ['erst-test'] : []) +                   \
   (config_all_devices.has_key('CONFIG_PCIE_PORT') and                                       \
Signature · 1 line
2.43.0