Skip to contentSkip to conversation

hw/arm/virt: Introduce Tegra241 CMDQV support for accelerated SMMUv3

PATCH Superseded Shameer Kolothum 4 participants 84 messages May 19 – May 26

Subsystems (2)

v5 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

v5 cover hw/arm/virt: Introduce Tegra241 CMDQV support for accelerated SMMUv3

Hi,

Changes from v4:
https://lore.kernel.org/qemu-devel/[email protected]/

- Addressed v4 feedback and picked up Reviewed-by tags. Thanks!
- Improved comments to specify optional/mandatory for CMDQ ops(patch#6)
- Introduced writel_mmio for 4-byte and writell_mmio for 8-byte(patch #14)
- Separated vcmdq Page0 and Page1 access helpers(patches #15, #16)
- Ignore VCMDQ BASE writes if the VCMDQ is already enabled(patch #16)
- Modified address_space_is_ram helper to address_space_range_is_ram() to cover the entire range(patch #18)
- Allocate HW QUEUE once all pre-conditions are met(BASE write, CMDQ_ALLOC_MAP, and CMDQV / VINTF enabled) - patch #19
- Sync the VCMDQ Page0 register cache when mmap'd VINTF page0 is available(patch #20)
- Introduced memory_region_skip_iommu_map()/_set_skip_iommu_map() accessors(patch 21).
- Moved the renamed helper qemu_ram_backend_pagesize_min() to system/physmem.c (patch #26)
- Updated commit log with DSDT/IORT AML descriptions(patch #30).
- Based on top of Nathan's "Resolve AUTO properties" v3 series [0].

Please find the complete branch here:
https://github.com/shamiali2008/qemu-master/tree/master-vcmdq-v5-ext

Sanity tested on NVIDIA Grace. Further testing in progress.

Feedback and testing are very welcome.

Thanks,
Shameer
[0] https://lore.kernel.org/qemu-devel/[email protected]/

---
Background(from RFCv1):
https://lore.kernel.org/qemu-devel/[email protected]/

Thanks to Nicolin for the initial patches and testing on which this is based.

Tegra241 CMDQV extends SMMUv3 by allocating per-VM "virtual interfaces" (VINTFs), each hosting up to 128 VCMDQs.

Each VINTF exposes two 64KB MMIO pages:
- Page0 – guest owned control and status registers (directly mapped into the VM)
- Page1 – queue configuration registers (trapped/emulated by QEMU)

Unlike the standard SMMU CMDQ, a guest owned Tegra241 VCMDQ does not support the full command set. Only a subset, primarily invalidation related commands, is accepted by the CMDQV hardware. For this reason, a distinct CMDQV device must be exposed to the guest, and the guest OS must include a Tegra241 CMDQV aware driver to take advantage of the hardware acceleration.

VCMDQ support is integrated via the IOMMU_HW_QUEUE_ALLOC mechanism, allowing QEMU to attach guest configured VCMDQ buffers to the underlying CMDQV hardware through IOMMUFD. The Linux kernel already supports the full CMDQV virtualisation model via IOMMUFD[0].
---

Nicolin Chen (15):
backends/iommufd: Update iommufd_backend_get_device_info backends/iommufd: Update iommufd_backend_alloc_viommu to allow user
ptr
backends/iommufd: Introduce iommufd_backend_alloc_hw_queue backends/iommufd: Introduce iommufd_backend_viommu_mmap hw/arm/tegra241-cmdqv: Implement CMDQV init hw/arm/tegra241-cmdqv: Implement CMDQV vIOMMU alloc/free hw/arm/tegra241-cmdqv: Emulate CMDQ-V Config region hw/arm/tegra241-cmdqv: Emulate VCMDQ register reads hw/arm/tegra241-cmdqv: Emulate VCMDQ register writes hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV hw/arm/tegra241-cmdqv: Allocate HW VCMDQs once configured hw/arm/tegra241-cmdqv: Map VINTF page0 into guest MMIO space hw/arm/tegra241-cmdqv: Add reset handler hw/arm/tegra241-cmdqv: Limit queue size based on backend page size hw/arm/virt-acpi: Advertise Tegra241 CMDQV nodes in DSDT

Shameer Kolothum (17):
system/iommufd: Remove unused viommu pointer from IOMMUFDVeventq hw/arm/smmuv3-accel: Introduce CMDQV ops interface hw/arm/tegra241-cmdqv: Add Tegra241 CMDQV ops backend stub hw/arm/smmuv3-accel: Wire CMDQV ops into accel lifecycle hw/arm/virt: Use stored SMMUv3 device list for IORT build hw/arm/tegra241-cmdqv: Probe host Tegra241 CMDQV support hw/arm/virt: Link SMMUv3 CMDQV resources to platform bus system/physmem: Add address_space_range_is_ram() helper hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backing
memory: Allow RAM device regions to skip IOMMU mapping hw/arm/smmuv3-accel: Introduce common helper for veventq read hw/arm/tegra241-cmdqv: Read and propagate Tegra241 CMDQV errors system/physmem: Add qemu_ram_backend_pagesize_min() helper hw/arm/smmuv3: Add per-device identifier property hw/arm/smmuv3-accel: Introduce helper to query CMDQV type hw/arm/smmuv3-accel: Enforce viommu association when CMDQV is active hw/arm/smmuv3: Add cmdqv property for SMMUv3 device

 hw/arm/smmuv3-accel.h         |   49 ++
 hw/arm/tegra241-cmdqv.h       |  381 +++++++++++++
 include/hw/arm/smmuv3.h       |    3 +
 include/hw/arm/virt.h         |    1 +
 include/system/iommufd.h      |   17 +-
 include/system/memory.h       |   34 ++
 include/system/ramblock.h     |    1 +
 backends/iommufd.c            |   64 +++
 hw/arm/smmuv3-accel-stubs.c   |   12 +
 hw/arm/smmuv3-accel.c         |  196 +++++--
 hw/arm/smmuv3.c               |   10 +
 hw/arm/tegra241-cmdqv-stubs.c |   16 +
 hw/arm/tegra241-cmdqv.c       | 1007 +++++++++++++++++++++++++++++++++
 hw/arm/virt-acpi-build.c      |  127 +++--
 hw/arm/virt.c                 |   37 ++
 hw/vfio/iommufd.c             |    4 +-
 hw/vfio/listener.c            |    4 +
 system/memory.c               |   10 +
 system/physmem.c              |   48 ++
 backends/trace-events         |    4 +-
 hw/arm/Kconfig                |    5 +
 hw/arm/meson.build            |    2 +
 hw/arm/trace-events           |   11 +
 qemu-options.hx               |    8 +
 24 files changed, 1971 insertions(+), 80 deletions(-)
 create mode 100644 hw/arm/tegra241-cmdqv.h
 create mode 100644 hw/arm/tegra241-cmdqv-stubs.c
 create mode 100644 hw/arm/tegra241-cmdqv.c
Signature · 1 line
2.43.0
#1 lore raw

PATCH v5 1/32 backends/iommufd: Update iommufd_backend_get_device_info

From: Nicolin Chen <[email protected]>

The updated IOMMUFD uAPI introduces the ability for userspace to request a specific hardware info data type via IOMMU_GET_HW_INFO. Update iommufd_backend_get_device_info() to set IOMMU_HW_INFO_FLAG_INPUT_TYPE when a non-zero type is supplied, and adjust all callers to pass a type value explicitly initialised to zero (IOMMU_HW_INFO_TYPE_DEFAULT) when no specific type is requested.

---
 backends/iommufd.c    | 7 +++++++
 hw/arm/smmuv3-accel.c | 2 +-
 hw/vfio/iommufd.c     | 4 ++--
 3 files changed, 10 insertions(+), 3 deletions(-)
backends/iommufd.c +7 -0
index 410b044370..8c3a981392 100644
@@ -390,16 +390,23 @@ bool iommufd_backend_get_dirty_bitmap(IOMMUFDBackend *be,
     return true;
 }
 
+/*
+ * @type can carry a desired HW info type defined in the uapi headers. If caller
+ * doesn't have one, indicating it wants the default type, then @type should be
+ * zeroed (i.e. IOMMU_HW_INFO_TYPE_DEFAULT).
+ */
 bool iommufd_backend_get_device_info(IOMMUFDBackend *be, uint32_t devid,
                                      uint32_t *type, void *data, uint32_t len,
                                      uint64_t *caps, uint8_t *max_pasid_log2,
                                      Error **errp)
 {
     struct iommu_hw_info info = {
+        .flags = (*type) ? IOMMU_HW_INFO_FLAG_INPUT_TYPE : 0,
         .size = sizeof(info),
         .dev_id = devid,
         .data_len = len,
         .data_uptr = (uintptr_t)data,
+        .in_data_type = *type,
     };
 
     if (ioctl(be->fd, IOMMU_GET_HW_INFO, &info)) {
hw/arm/smmuv3-accel.c +1 -1
index 05257f74ed..eb13c0b91c 100644
@@ -176,7 +176,7 @@ smmuv3_accel_hw_compatible(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *hiodi,
                            Error **errp)
 {
     struct iommu_hw_info_arm_smmuv3 info;
-    uint32_t data_type;
+    uint32_t data_type = IOMMU_HW_INFO_TYPE_DEFAULT;
     uint64_t caps;
 
     if (!iommufd_backend_get_device_info(hiodi->iommufd, hiodi->devid,
hw/vfio/iommufd.c +2 -2
index df148a49a7..495a59cc2e 100644
@@ -353,7 +353,7 @@ static bool iommufd_cdev_autodomains_get(VFIODevice *vbasedev,
     IOMMUFDBackend *iommufd = vbasedev->iommufd;
     VFIOContainer *bcontainer = VFIO_IOMMU(container);
     bool viommu_nesting, viommu_nesting_dirty;
-    uint32_t type, flags = 0;
+    uint32_t type = IOMMU_HW_INFO_TYPE_DEFAULT, flags = 0;
     uint64_t hw_caps;
     VendorCaps caps;
     VFIOIOASHwpt *hwpt;
@@ -948,7 +948,7 @@ static bool hiod_iommufd_vfio_realize(HostIOMMUDevice *hiod, void *opaque,
     HostIOMMUDeviceIOMMUFD *hiodi;
     HostIOMMUDeviceCaps *caps = &hiod->caps;
     VendorCaps *vendor_caps = &caps->vendor_caps;
-    enum iommu_hw_info_type type;
+    uint32_t type = IOMMU_HW_INFO_TYPE_DEFAULT;
     uint8_t max_pasid_log2;
     uint64_t hw_caps;
 
Signature · 1 line
2.43.0
#2 lore raw

PATCH v5 2/32 backends/iommufd: Update iommufd_backend_alloc_viommu to allow user ptr

In reply to Shameer Kolothum

From: Nicolin Chen <[email protected]>

The updated IOMMUFD VIOMMU_ALLOC uAPI allows userspace to provide a data buffer when creating a vIOMMU (e.g. for Tegra241 CMDQV). Extend iommufd_backend_alloc_viommu() to pass a user pointer and size to the kernel.

Update the caller accordingly.

---
 include/system/iommufd.h | 1 +
 backends/iommufd.c       | 4 ++++
 hw/arm/smmuv3-accel.c    | 4 ++--
 backends/trace-events    | 2 +-
 4 files changed, 8 insertions(+), 3 deletions(-)
include/system/iommufd.h +1 -0
index 2925d116ac..1b0266f660 100644
@@ -89,6 +89,7 @@ bool iommufd_backend_alloc_hwpt(IOMMUFDBackend *be, uint32_t dev_id,
                                 Error **errp);
 bool iommufd_backend_alloc_viommu(IOMMUFDBackend *be, uint32_t dev_id,
                                   uint32_t viommu_type, uint32_t hwpt_id,
+                                  void *data_ptr, uint32_t data_len,
                                   uint32_t *out_hwpt, Error **errp);
 
 bool iommufd_backend_alloc_vdev(IOMMUFDBackend *be, uint32_t dev_id,
backends/iommufd.c +4 -0
index 8c3a981392..c745dbd2b1 100644
@@ -463,6 +463,7 @@ bool iommufd_backend_invalidate_cache(IOMMUFDBackend *be, uint32_t id,
 
 bool iommufd_backend_alloc_viommu(IOMMUFDBackend *be, uint32_t dev_id,
                                   uint32_t viommu_type, uint32_t hwpt_id,
+                                  void *data_ptr, uint32_t data_len,
                                   uint32_t *out_viommu_id, Error **errp)
 {
     int ret;
@@ -471,11 +472,14 @@ bool iommufd_backend_alloc_viommu(IOMMUFDBackend *be, uint32_t dev_id,
         .type = viommu_type,
         .dev_id = dev_id,
         .hwpt_id = hwpt_id,
+        .data_len = data_len,
+        .data_uptr = (uintptr_t)data_ptr,
     };
 
     ret = ioctl(be->fd, IOMMU_VIOMMU_ALLOC, &alloc_viommu);
 
     trace_iommufd_backend_alloc_viommu(be->fd, dev_id, viommu_type, hwpt_id,
+                                       (uintptr_t)data_ptr, data_len,
                                        alloc_viommu.out_viommu_id, ret);
     if (ret) {
         error_setg_errno(errp, errno, "IOMMU_VIOMMU_ALLOC failed");
hw/arm/smmuv3-accel.c +2 -2
index eb13c0b91c..bfa6a03f9e 100644
@@ -582,8 +582,8 @@ smmuv3_accel_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *hiodi,
     IOMMUFDViommu *viommu;
 
     if (!iommufd_backend_alloc_viommu(hiodi->iommufd, hiodi->devid,
-                                      IOMMU_VIOMMU_TYPE_ARM_SMMUV3,
-                                      s2_hwpt_id, &viommu_id, errp)) {
+                                      IOMMU_VIOMMU_TYPE_ARM_SMMUV3, s2_hwpt_id,
+                                      NULL, 0, &viommu_id, errp)) {
         return false;
     }
 
backends/trace-events +1 -1
index b9365113e7..3ba0c3503c 100644
@@ -21,7 +21,7 @@ iommufd_backend_free_id(int iommufd, uint32_t id, int ret) " iommufd=%d id=%d (%
 iommufd_backend_set_dirty(int iommufd, uint32_t hwpt_id, bool start, int ret) " iommufd=%d hwpt=%u enable=%d (%d)"
 iommufd_backend_get_dirty_bitmap(int iommufd, uint32_t hwpt_id, uint64_t iova, uint64_t size, uint64_t flags, uint64_t page_size, int ret) " iommufd=%d hwpt=%u iova=0x%"PRIx64" size=0x%"PRIx64" flags=0x%"PRIx64" page_size=0x%"PRIx64" (%d)"
 iommufd_backend_invalidate_cache(int iommufd, uint32_t id, uint32_t data_type, uint32_t entry_len, uint32_t entry_num, uint32_t done_num, uint64_t data_ptr, int ret) " iommufd=%d id=%u data_type=%u entry_len=%u entry_num=%u done_num=%u data_ptr=0x%"PRIx64" (%d)"
-iommufd_backend_alloc_viommu(int iommufd, uint32_t dev_id, uint32_t type, uint32_t hwpt_id, uint32_t viommu_id, int ret) " iommufd=%d type=%u dev_id=%u hwpt_id=%u viommu_id=%u (%d)"
+iommufd_backend_alloc_viommu(int iommufd, uint32_t dev_id, uint32_t type, uint32_t hwpt_id, uint64_t data_ptr, uint32_t data_len, uint32_t viommu_id, int ret) " iommufd=%d type=%u dev_id=%u hwpt_id=%u data_ptr=0x%"PRIx64" data_len=0x%x viommu_id=%u (%d)"
 iommufd_backend_alloc_vdev(int iommufd, uint32_t dev_id, uint32_t viommu_id, uint64_t virt_id, uint32_t vdev_id, int ret) " iommufd=%d dev_id=%u viommu_id=%u virt_id=0x%"PRIx64" vdev_id=%u (%d)"
 iommufd_viommu_alloc_eventq(int iommufd, uint32_t viommu_id, uint32_t type, uint32_t veventq_id, uint32_t veventq_fd, int ret) " iommufd=%d viommu_id=%u type=%u veventq_id=%u veventq_fd=%u (%d)"
 
Signature · 1 line
2.43.0
#3 lore raw

PATCH v5 3/32 backends/iommufd: Introduce iommufd_backend_alloc_hw_queue

In reply to Shameer Kolothum

From: Nicolin Chen <[email protected]>

Add a helper to allocate an iommufd backed HW queue for a vIOMMU.

While at it, define a struct IOMMUFDHWqueue for use by vendor implementations.

---
 include/system/iommufd.h | 11 +++++++++++
 backends/iommufd.c       | 31 +++++++++++++++++++++++++++++++
 backends/trace-events    |  1 +
 3 files changed, 43 insertions(+)
include/system/iommufd.h +11 -0
index 1b0266f660..c6f2e87a7e 100644
@@ -65,6 +65,12 @@ typedef struct IOMMUFDVeventq {
     bool event_start; /* True after first valid event; cleared on overflow */
 } IOMMUFDVeventq;
 
+/* HW queue object for a vIOMMU-specific HW-accelerated queue */
+typedef struct IOMMUFDHWqueue {
+    IOMMUFDViommu *viommu;
+    uint32_t hw_queue_id;
+} IOMMUFDHWqueue;
+
 bool iommufd_backend_connect(IOMMUFDBackend *be, Error **errp);
 void iommufd_backend_disconnect(IOMMUFDBackend *be);
 
@@ -101,6 +107,11 @@ bool iommufd_backend_alloc_veventq(IOMMUFDBackend *be, uint32_t viommu_id,
                                    uint32_t *out_veventq_id,
                                    uint32_t *out_veventq_fd, Error **errp);
 
+bool iommufd_backend_alloc_hw_queue(IOMMUFDBackend *be, uint32_t viommu_id,
+                                    uint32_t queue_type, uint32_t index,
+                                    uint64_t addr, uint64_t length,
+                                    uint32_t *out_hw_queue_id, Error **errp);
+
 bool iommufd_backend_set_dirty_tracking(IOMMUFDBackend *be, uint32_t hwpt_id,
                                         bool start, Error **errp);
 bool iommufd_backend_get_dirty_bitmap(IOMMUFDBackend *be, uint32_t hwpt_id,
backends/iommufd.c +31 -0
index c745dbd2b1..8eaaf456e8 100644
@@ -549,6 +549,37 @@ bool iommufd_backend_alloc_veventq(IOMMUFDBackend *be, uint32_t viommu_id,
     return true;
 }
 
+bool iommufd_backend_alloc_hw_queue(IOMMUFDBackend *be, uint32_t viommu_id,
+                                    uint32_t queue_type, uint32_t index,
+                                    uint64_t addr, uint64_t length,
+                                    uint32_t *out_hw_queue_id, Error **errp)
+{
+    int ret;
+    struct iommu_hw_queue_alloc alloc_hw_queue = {
+        .size = sizeof(alloc_hw_queue),
+        .flags = 0,
+        .viommu_id = viommu_id,
+        .type = queue_type,
+        .index = index,
+        .nesting_parent_iova = addr,
+        .length = length,
+    };
+
+    ret = ioctl(be->fd, IOMMU_HW_QUEUE_ALLOC, &alloc_hw_queue);
+
+    trace_iommufd_backend_alloc_hw_queue(be->fd, viommu_id, queue_type,
+                                         index, addr, length,
+                                         alloc_hw_queue.out_hw_queue_id, ret);
+    if (ret) {
+        error_setg_errno(errp, errno, "IOMMU_HW_QUEUE_ALLOC failed");
+        return false;
+    }
+
+    g_assert(out_hw_queue_id);
+    *out_hw_queue_id = alloc_hw_queue.out_hw_queue_id;
+    return true;
+}
+
 bool host_iommu_device_iommufd_attach_hwpt(HostIOMMUDeviceIOMMUFD *hiodi,
                                            uint32_t hwpt_id, Error **errp)
 {
backends/trace-events +1 -0
index 3ba0c3503c..c5c1d95aad 100644
@@ -24,6 +24,7 @@ iommufd_backend_invalidate_cache(int iommufd, uint32_t id, uint32_t data_type, u
 iommufd_backend_alloc_viommu(int iommufd, uint32_t dev_id, uint32_t type, uint32_t hwpt_id, uint64_t data_ptr, uint32_t data_len, uint32_t viommu_id, int ret) " iommufd=%d type=%u dev_id=%u hwpt_id=%u data_ptr=0x%"PRIx64" data_len=0x%x viommu_id=%u (%d)"
 iommufd_backend_alloc_vdev(int iommufd, uint32_t dev_id, uint32_t viommu_id, uint64_t virt_id, uint32_t vdev_id, int ret) " iommufd=%d dev_id=%u viommu_id=%u virt_id=0x%"PRIx64" vdev_id=%u (%d)"
 iommufd_viommu_alloc_eventq(int iommufd, uint32_t viommu_id, uint32_t type, uint32_t veventq_id, uint32_t veventq_fd, int ret) " iommufd=%d viommu_id=%u type=%u veventq_id=%u veventq_fd=%u (%d)"
+iommufd_backend_alloc_hw_queue(int iommufd, uint32_t viommu_id, uint32_t queue_type, uint32_t index, uint64_t addr, uint64_t size, uint32_t queue_id, int ret) " iommufd=%d viommu_id=%u queue_type=%u index=%u addr=0x%"PRIx64" size=0x%"PRIx64" queue_id=%u (%d)"
 
 # igvm-cfg.c
 igvm_reset_enter(int type) "type=%u"
Signature · 1 line
2.43.0
#4 lore raw

PATCH v5 4/32 backends/iommufd: Introduce iommufd_backend_viommu_mmap

In reply to Shameer Kolothum

From: Nicolin Chen <[email protected]>

Add a backend helper to mmap hardware MMIO regions exposed via iommufd for a vIOMMU instance. This allows user space to access HW-accelerated MMIO pages provided by the vIOMMU.

The caller is responsible for unmapping the returned region.

---
 include/system/iommufd.h |  4 ++++
 backends/iommufd.c       | 22 ++++++++++++++++++++++
 backends/trace-events    |  1 +
 3 files changed, 27 insertions(+)
include/system/iommufd.h +4 -0
index c6f2e87a7e..bb8d5081d7 100644
@@ -112,6 +112,10 @@ bool iommufd_backend_alloc_hw_queue(IOMMUFDBackend *be, uint32_t viommu_id,
                                     uint64_t addr, uint64_t length,
                                     uint32_t *out_hw_queue_id, Error **errp);
 
+bool iommufd_backend_viommu_mmap(IOMMUFDBackend *be, uint32_t viommu_id,
+                                 uint64_t size, off_t offset, void **out_ptr,
+                                 Error **errp);
+
 bool iommufd_backend_set_dirty_tracking(IOMMUFDBackend *be, uint32_t hwpt_id,
                                         bool start, Error **errp);
 bool iommufd_backend_get_dirty_bitmap(IOMMUFDBackend *be, uint32_t hwpt_id,
backends/iommufd.c +22 -0
index 8eaaf456e8..440c1b82bc 100644
@@ -580,6 +580,28 @@ bool iommufd_backend_alloc_hw_queue(IOMMUFDBackend *be, uint32_t viommu_id,
     return true;
 }
 
+/*
+ * Helper to mmap HW MMIO regions exposed via iommufd for a vIOMMU instance.
+ * The caller is responsible for unmapping the mapped region.
+ */
+bool iommufd_backend_viommu_mmap(IOMMUFDBackend *be, uint32_t viommu_id,
+                                 uint64_t size, off_t offset, void **out_ptr,
+                                 Error **errp)
+{
+    g_assert(viommu_id);
+    g_assert(out_ptr);
+
+    *out_ptr = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, be->fd,
+                   offset);
+    trace_iommufd_backend_viommu_mmap(be->fd, viommu_id, size, offset);
+    if (*out_ptr == MAP_FAILED) {
+        error_setg_errno(errp, errno, "IOMMUFD vIOMMU mmap failed");
+        return false;
+    }
+
+    return true;
+}
+
 bool host_iommu_device_iommufd_attach_hwpt(HostIOMMUDeviceIOMMUFD *hiodi,
                                            uint32_t hwpt_id, Error **errp)
 {
backends/trace-events +1 -0
index c5c1d95aad..b63420b73e 100644
@@ -25,6 +25,7 @@ iommufd_backend_alloc_viommu(int iommufd, uint32_t dev_id, uint32_t type, uint32
 iommufd_backend_alloc_vdev(int iommufd, uint32_t dev_id, uint32_t viommu_id, uint64_t virt_id, uint32_t vdev_id, int ret) " iommufd=%d dev_id=%u viommu_id=%u virt_id=0x%"PRIx64" vdev_id=%u (%d)"
 iommufd_viommu_alloc_eventq(int iommufd, uint32_t viommu_id, uint32_t type, uint32_t veventq_id, uint32_t veventq_fd, int ret) " iommufd=%d viommu_id=%u type=%u veventq_id=%u veventq_fd=%u (%d)"
 iommufd_backend_alloc_hw_queue(int iommufd, uint32_t viommu_id, uint32_t queue_type, uint32_t index, uint64_t addr, uint64_t size, uint32_t queue_id, int ret) " iommufd=%d viommu_id=%u queue_type=%u index=%u addr=0x%"PRIx64" size=0x%"PRIx64" queue_id=%u (%d)"
+iommufd_backend_viommu_mmap(int iommufd, uint32_t viommu_id, uint64_t size, uint64_t offset) " iommufd=%d viommu_id=%u size=0x%"PRIx64" offset=0x%"PRIx64
 
 # igvm-cfg.c
 igvm_reset_enter(int type) "type=%u"
Signature · 1 line
2.43.0
#5 lore raw

PATCH v5 5/32 system/iommufd: Remove unused viommu pointer from IOMMUFDVeventq

In reply to Shameer Kolothum

The viommu field is assigned but never used. Callers freeing the veventq already have access to the IOMMUFDViommu object through other references, so this field is redundant.

Removing it also simplifies upcoming changes where veventq is allocated based on the viommu id before the IOMMUFDViommu object is created (e.g. vendor CMDQV-based veventq allocation).

No functional change.

---
 include/system/iommufd.h | 1 -
 hw/arm/smmuv3-accel.c    | 1 -
 2 files changed, 2 deletions(-)
include/system/iommufd.h +0 -1
index bb8d5081d7..da68ba0037 100644
@@ -58,7 +58,6 @@ typedef struct IOMMUFDVdev {
 
 /* Virtual event queue interface for a vIOMMU */
 typedef struct IOMMUFDVeventq {
-    IOMMUFDViommu *viommu;
     uint32_t veventq_id;
     uint32_t veventq_fd;
     uint32_t last_event_seq; /* Sequence number of last processed event */
hw/arm/smmuv3-accel.c +0 -1
index bfa6a03f9e..30c498ffcf 100644
@@ -553,7 +553,6 @@ bool smmuv3_accel_alloc_veventq(SMMUv3State *s, Error **errp)
     veventq = g_new0(IOMMUFDVeventq, 1);
     veventq->veventq_id = veventq_id;
     veventq->veventq_fd = veventq_fd;
-    veventq->viommu = accel->viommu;
     accel->veventq = veventq;
 
     /* Set up event handler for veventq fd */
Signature · 1 line
2.43.0
#6 lore raw

PATCH v5 6/32 hw/arm/smmuv3-accel: Introduce CMDQV ops interface

In reply to Shameer Kolothum

Command Queue Virtualization (CMDQV) is a hardware extension available on certain platforms that allows the SMMUv3 command queue to be virtualized and passed through to a VM, improving performance.

For example, NVIDIA Tegra241 implements CMDQV to support virtualization of multiple command queues (VCMDQs).

The term CMDQV is used here generically to refer to any platform that provides hardware support to virtualize the SMMUv3 command queue.

CMDQV support is a specialization of the IOMMUFD-backed accelerated SMMUv3 path. Introduce an ops interface to factor out CMDQV-specific probe, initialization, and vIOMMU allocation logic from the base implementation. The ops pointer and associated state are stored in the accelerated SMMUv3 state.

This provides an extensible design to support future vendor-specific CMDQV implementations.

No functional change.

---
 hw/arm/smmuv3-accel.h | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)
hw/arm/smmuv3-accel.h +35 -0
index 87fecb5c68..f82a7112d8 100644
@@ -10,11 +10,45 @@
 #define HW_ARM_SMMUV3_ACCEL_H
 
 #include "hw/arm/smmu-common.h"
+#include "hw/arm/smmuv3.h"
 #include "system/iommufd.h"
 #ifdef CONFIG_LINUX
 #include <linux/iommufd.h>
 #endif
 
+/*
+ * CMDQ-Virtualization (CMDQV) hardware support, extends the SMMUv3 to
+ * support multiple VCMDQs with virtualization capabilities.
+ * CMDQV specific behavior is factored behind this ops interface.
+ */
+typedef struct SMMUv3AccelCmdqvOps {
+    /**
+     * @probe: Optional callback. Vendor-specific device probing.
+     */
+    bool (*probe)(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev, Error **errp);
+    /**
+     * @init: Optional callback. Initialize CMDQV hardware.
+     */
+    bool (*init)(SMMUv3State *s, Error **errp);
+    /**
+     * @alloc_viommu: Mandatory when probe is implemented.
+     * Allocate CMDQV viommu resources.
+     */
+    bool (*alloc_viommu)(SMMUv3State *s,
+                         HostIOMMUDeviceIOMMUFD *idev,
+                         uint32_t *out_viommu_id,
+                         Error **errp);
+    /**
+     * @free_viommu: Optional callback. Free CMDQV viommu resources.
+     * If NULL, the viommu_id is freed directly via iommufd_backend_free_id().
+     */
+    void (*free_viommu)(SMMUv3State *s);
+    /**
+     * @reset: Optional callback. Reset CMDQV state.
+     */
+    void (*reset)(SMMUv3State *s);
+} SMMUv3AccelCmdqvOps;
+
 /*
  * Represents an accelerated SMMU instance backed by an iommufd vIOMMU object.
  * Holds bypass and abort proxy HWPT IDs used for device attachment.
@@ -27,6 +61,7 @@ typedef struct SMMUv3AccelState {
     QLIST_HEAD(, SMMUv3AccelDevice) device_list;
     bool auto_mode;
     bool auto_finalised;
+    const SMMUv3AccelCmdqvOps *cmdqv_ops;
 } SMMUv3AccelState;
 
 typedef struct SMMUS1Hwpt {
Signature · 1 line
2.43.0
#7 lore raw

Re: [PATCH v5 06/32] hw/arm/smmuv3-accel: Introduce CMDQV ops interface

Quoted 47 lines · Shameer Kolothum

Command Queue Virtualization (CMDQV) is a hardware extension available on certain platforms that allows the SMMUv3 command queue to be virtualized and passed through to a VM, improving performance.

For example, NVIDIA Tegra241 implements CMDQV to support virtualization of multiple command queues (VCMDQs).

The term CMDQV is used here generically to refer to any platform that provides hardware support to virtualize the SMMUv3 command queue.

CMDQV support is a specialization of the IOMMUFD-backed accelerated SMMUv3 path. Introduce an ops interface to factor out CMDQV-specific probe, initialization, and vIOMMU allocation logic from the base implementation. The ops pointer and associated state are stored in the accelerated SMMUv3 state.

This provides an extensible design to support future vendor-specific CMDQV implementations.

No functional change.

---
 hw/arm/smmuv3-accel.h | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)
hw/arm/smmuv3-accel.h +7 -0
index 87fecb5c68..f82a7112d8 100644
@@ -10,11 +10,45 @@
 #define HW_ARM_SMMUV3_ACCEL_H
 
 #include "hw/arm/smmu-common.h"
+#include "hw/arm/smmuv3.h"
 #include "system/iommufd.h"
 #ifdef CONFIG_LINUX
 #include <linux/iommufd.h>
 #endif
 
+/*
+ * CMDQ-Virtualization (CMDQV) hardware support, extends the SMMUv3 to
+ * support multiple VCMDQs with virtualization capabilities.
+ * CMDQV specific behavior is factored behind this ops interface.
+ */
+typedef struct SMMUv3AccelCmdqvOps {

hw/arm/smmuv3-accel.h line 34

Quoted 10 lines
hw/arm/smmuv3-accel.h +10 -0
+    /**
+     * @probe: Optional callback. Vendor-specific device probing.
+     */
+    bool (*probe)(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev, Error **errp);
+    /**
+     * @init: Optional callback. Initialize CMDQV hardware.
+     */
+    bool (*init)(SMMUv3State *s, Error **errp);
+    /**
+     * @alloc_viommu: Mandatory when probe is implemented.

see my comment on 8/32. Aren't probe and alloc_viommu somehow mandatory?

Eric

Quoted 28 lines
hw/arm/smmuv3-accel.h +18 -0
+     * Allocate CMDQV viommu resources.
+     */
+    bool (*alloc_viommu)(SMMUv3State *s,
+                         HostIOMMUDeviceIOMMUFD *idev,
+                         uint32_t *out_viommu_id,
+                         Error **errp);
+    /**
+     * @free_viommu: Optional callback. Free CMDQV viommu resources.
+     * If NULL, the viommu_id is freed directly via iommufd_backend_free_id().
+     */
+    void (*free_viommu)(SMMUv3State *s);
+    /**
+     * @reset: Optional callback. Reset CMDQV state.
+     */
+    void (*reset)(SMMUv3State *s);
+} SMMUv3AccelCmdqvOps;
+
 /*
  * Represents an accelerated SMMU instance backed by an iommufd vIOMMU object.
  * Holds bypass and abort proxy HWPT IDs used for device attachment.
@@ -27,6 +61,7 @@ typedef struct SMMUv3AccelState {
     QLIST_HEAD(, SMMUv3AccelDevice) device_list;
     bool auto_mode;
     bool auto_finalised;
+    const SMMUv3AccelCmdqvOps *cmdqv_ops;
 } SMMUv3AccelState;
 
 typedef struct SMMUS1Hwpt {
#8 lore raw

PATCH v5 7/32 hw/arm/tegra241-cmdqv: Add Tegra241 CMDQV ops backend stub

In reply to Shameer Kolothum

Introduce a Tegra241 CMDQV backend that plugs into the SMMUv3 accelerated CMDQV ops interface.

This patch wires up the Tegra241 CMDQV backend and provides a stub implementation for CMDQV probe, initialization, vIOMMU allocation and reset handling.

Functional CMDQV support is added in follow-up patches.

---
 hw/arm/tegra241-cmdqv.h       | 15 ++++++++++
 hw/arm/tegra241-cmdqv-stubs.c | 16 ++++++++++
 hw/arm/tegra241-cmdqv.c       | 56 +++++++++++++++++++++++++++++++++++
 hw/arm/Kconfig                |  5 ++++
 hw/arm/meson.build            |  2 ++
 5 files changed, 94 insertions(+)
 create mode 100644 hw/arm/tegra241-cmdqv.h
 create mode 100644 hw/arm/tegra241-cmdqv-stubs.c
 create mode 100644 hw/arm/tegra241-cmdqv.c
hw/arm/tegra241-cmdqv.h new file +15 -0
new file mode 100644
index 0000000000..74a6954017
@@ -0,0 +1,15 @@
+/*
+ * Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved
+ * NVIDIA Tegra241 CMDQ-Virtualization extension for SMMUv3
+ *
+ * Written by Nicolin Chen, Shameer Kolothum
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#ifndef HW_ARM_TEGRA241_CMDQV_H
+#define HW_ARM_TEGRA241_CMDQV_H
+
+const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops(void);
+
+#endif /* HW_ARM_TEGRA241_CMDQV_H */
hw/arm/tegra241-cmdqv-stubs.c new file +16 -0
new file mode 100644
index 0000000000..4669f5c5f5
@@ -0,0 +1,16 @@
+/*
+ * Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved
+ *
+ * Stubs for Tegra241 CMDQ-Virtualization extension for SMMUv3
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#include "qemu/osdep.h"
+#include "smmuv3-accel.h"
+#include "hw/arm/tegra241-cmdqv.h"
+
+const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops(void)
+{
+    return NULL;
+}
hw/arm/tegra241-cmdqv.c new file +56 -0

56 lines not on this page · raw message

hw/arm/Kconfig +5 -0
index 5b198402d5..f73ad9727a 100644
@@ -642,6 +642,10 @@ config FSL_IMX8MM_EVK
     depends on TCG
     select FSL_IMX8MM
 
+config TEGRA241_CMDQV
+    bool
+    depends on ARM_SMMUV3_ACCEL
+
 config ARM_SMMUV3_ACCEL
     bool
     depends on ARM_SMMUV3
@@ -649,6 +653,7 @@ config ARM_SMMUV3_ACCEL
 config ARM_SMMUV3
     bool
     select ARM_SMMUV3_ACCEL if IOMMUFD
+    imply TEGRA241_CMDQV
 
 config FSL_IMX6UL
     bool
hw/arm/meson.build +2 -0
index 80068f70bb..eb16ffcc34 100644
@@ -84,6 +84,8 @@ arm_common_ss.add(when: 'CONFIG_FSL_IMX8MM_EVK', if_true: files('imx8mm-evk.c'))
 arm_common_ss.add(when: 'CONFIG_ARM_SMMUV3', if_true: files('smmuv3.c'))
 arm_common_ss.add(when: 'CONFIG_ARM_SMMUV3_ACCEL', if_true: files('smmuv3-accel.c'))
 stub_ss.add(files('smmuv3-accel-stubs.c'))
+arm_common_ss.add(when: 'CONFIG_TEGRA241_CMDQV', if_true: files('tegra241-cmdqv.c'))
+stub_ss.add(files('tegra241-cmdqv-stubs.c'))
 arm_common_ss.add(when: 'CONFIG_FSL_IMX6UL', if_true: files('fsl-imx6ul.c', 'mcimx6ul-evk.c'))
 arm_common_ss.add(when: 'CONFIG_NRF51_SOC', if_true: files('nrf51_soc.c'))
 arm_common_ss.add(when: 'CONFIG_XEN', if_true: files(
Signature · 1 line
2.43.0
#9 lore raw

PATCH v5 8/32 hw/arm/smmuv3-accel: Wire CMDQV ops into accel lifecycle

In reply to Shameer Kolothum

Add support for selecting and initializing a CMDQV backend based on the cmdqv OnOffAuto property.

If set to OFF, CMDQV is not used and the default IOMMUFD-backed allocation path is taken.

If set to AUTO, QEMU attempts to probe a CMDQV backend during device setup. If probing succeeds, the selected ops are stored in the accelerated SMMUv3 state and used. If probing fails, QEMU silently falls back to the default path.

If set to ON, QEMU requires CMDQV support. Probing is performed during setup and failure results in an error.

When a CMDQV backend is active, its callbacks are used for vIOMMU allocation, free, and reset handling. Otherwise, the base implementation is used.

The current implementation wires up the Tegra241 CMDQV backend through the generic ops interface. Functional CMDQV behaviour is added in subsequent patches.

No functional change.

---
 include/hw/arm/smmuv3.h |  2 +
 hw/arm/smmuv3-accel.c   | 96 ++++++++++++++++++++++++++++++++++++++---
 2 files changed, 91 insertions(+), 7 deletions(-)
include/hw/arm/smmuv3.h +2 -0
index fe0493c1aa..aa6a79237a 100644
@@ -74,6 +74,8 @@ struct SMMUv3State {
     OnOffAuto ats;
     OasMode oas;
     SsidSizeMode ssidsize;
+    /* SMMU CMDQV extension */
+    OnOffAuto cmdqv;
 
     Notifier machine_done;
 };
hw/arm/smmuv3-accel.c +89 -7

137 lines not on this page · raw message

Signature · 1 line
2.43.0
#10 lore raw

Re: [PATCH v5 08/32] hw/arm/smmuv3-accel: Wire CMDQV ops into accel lifecycle

Hi SHameer,

Quoted 66 lines · Shameer Kolothum

Add support for selecting and initializing a CMDQV backend based on the cmdqv OnOffAuto property.

If set to OFF, CMDQV is not used and the default IOMMUFD-backed allocation path is taken.

If set to AUTO, QEMU attempts to probe a CMDQV backend during device setup. If probing succeeds, the selected ops are stored in the accelerated SMMUv3 state and used. If probing fails, QEMU silently falls back to the default path.

If set to ON, QEMU requires CMDQV support. Probing is performed during setup and failure results in an error.

When a CMDQV backend is active, its callbacks are used for vIOMMU allocation, free, and reset handling. Otherwise, the base implementation is used.

The current implementation wires up the Tegra241 CMDQV backend through the generic ops interface. Functional CMDQV behaviour is added in subsequent patches.

No functional change.

---
 include/hw/arm/smmuv3.h |  2 +
 hw/arm/smmuv3-accel.c   | 96 ++++++++++++++++++++++++++++++++++++++---
 2 files changed, 91 insertions(+), 7 deletions(-)
include/hw/arm/smmuv3.h +2 -0
index fe0493c1aa..aa6a79237a 100644
@@ -74,6 +74,8 @@ struct SMMUv3State {
     OnOffAuto ats;
     OasMode oas;
     SsidSizeMode ssidsize;
+    /* SMMU CMDQV extension */
+    OnOffAuto cmdqv;
 
     Notifier machine_done;
 };
hw/arm/smmuv3-accel.c +2 -0
index 30c498ffcf..202b1aedd9 100644
@@ -19,6 +19,7 @@
 #include "smmuv3-internal.h"
 #include "smmuv3-accel.h"
 #include "system/system.h"
+#include "tegra241-cmdqv.h"
 
 /*
  * The root region aliases the global system memory, and shared_as_sysmem
@@ -570,6 +571,7 @@ smmuv3_accel_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *hiodi,
                           Error **errp)
 {
     SMMUv3AccelState *accel = s->s_accel;
+    const SMMUv3AccelCmdqvOps *cmdqv_ops = accel->cmdqv_ops;
     struct iommu_hwpt_arm_smmuv3 bypass_data = {
         .ste = { SMMU_STE_CFG_BYPASS | SMMU_STE_VALID, 0x0ULL },
     };

hw/arm/smmuv3-accel.c smmuv3_accel_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *hiodi, · line 586

Quoted 9 lines
hw/arm/smmuv3-accel.c +2 -4
@@ -580,10 +582,17 @@ smmuv3_accel_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *hiodi,
     uint32_t viommu_id, hwpt_id;
     IOMMUFDViommu *viommu;
 
-    if (!iommufd_backend_alloc_viommu(hiodi->iommufd, hiodi->devid,
-                                      IOMMU_VIOMMU_TYPE_ARM_SMMUV3, s2_hwpt_id,
-                                      NULL, 0, &viommu_id, errp)) {
-        return false;
+    if (cmdqv_ops) {
+        if (!cmdqv_ops->alloc_viommu(s, hiodi, &viommu_id, errp)) {

the doc comment in 6/32 says that alloc_iommu is mandatory if probe is implemented. So it looks there are situations where alloc_iommu may be null.

But actually given the implementation of smmuv3_accel_probe_cmdqv below, the ops is only non NULL if probe has succeeded. So shouldn't probe be also mandatory?

Quoted 64 lines
hw/arm/smmuv3-accel.c +51 -1
+            return false;
+        }
+    } else {
+        if (!iommufd_backend_alloc_viommu(hiodi->iommufd, hiodi->devid,
+                                          IOMMU_VIOMMU_TYPE_ARM_SMMUV3,
+                                          s2_hwpt_id, NULL, 0, &viommu_id,
+                                          errp)) {
+            return false;
+        }
     }
 
     viommu = g_new0(IOMMUFDViommu, 1);
@@ -629,12 +638,72 @@ free_bypass_hwpt:
 free_abort_hwpt:
     iommufd_backend_free_id(hiodi->iommufd, accel->abort_hwpt_id);
 free_viommu:
-    iommufd_backend_free_id(hiodi->iommufd, viommu->viommu_id);
+    if (cmdqv_ops && cmdqv_ops->free_viommu) {
+        cmdqv_ops->free_viommu(s);
+    } else {
+        iommufd_backend_free_id(hiodi->iommufd, viommu->viommu_id);
+    }
     g_free(viommu);
     accel->viommu = NULL;
     return false;
 }
 
+static const SMMUv3AccelCmdqvOps *
+smmuv3_accel_probe_cmdqv(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
+                         Error **errp)
+{
+    const SMMUv3AccelCmdqvOps *ops = tegra241_cmdqv_get_ops();
+
+    if (!ops || !ops->probe) {
+        error_setg(errp, "No CMDQV ops found");
+        return NULL;
+    }
+
+    if (!ops->probe(s, idev, errp)) {
+        return NULL;
+    }
+    return ops;
+}
+
+static bool
+smmuv3_accel_select_cmdqv(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
+                          Error **errp)
+{
+    const SMMUv3AccelCmdqvOps *ops = NULL;
+
+    if (s->s_accel->cmdqv_ops) {
+        return true;
+    }
+
+    switch (s->cmdqv) {
+    case ON_OFF_AUTO_OFF:
+        s->s_accel->cmdqv_ops = NULL;
+        return true;
+    case ON_OFF_AUTO_AUTO:
+        ops = smmuv3_accel_probe_cmdqv(s, idev, NULL);
+        break;
+    case ON_OFF_AUTO_ON:
+        ops = smmuv3_accel_probe_cmdqv(s, idev, errp);
+        if (!ops) {

hw/arm/smmuv3-accel.c free_bypass_hwpt: · line 697

Quoted 10 lines
hw/arm/smmuv3-accel.c +10 -0
+            error_append_hint(errp, "CMDQV requested but not supported");
+            return false;
+        }
+        break;
+    default:
+        g_assert_not_reached();
+    }
+
+    if (ops) {
+        g_assert(ops->alloc_viommu);

I think I would move the assert at the end of

smmuv3_accel_probe_cmdqv()

Quoted 33 lines
hw/arm/smmuv3-accel.c +17 -2
+    }
+
+    if (ops && ops->init && !ops->init(s, errp)) {
+        return false;
+    }
+    s->s_accel->cmdqv_ops = ops;
+    return true;
+}
+
 static bool smmuv3_accel_set_iommu_device(PCIBus *bus, void *opaque, int devfn,
                                           HostIOMMUDevice *hiod, Error **errp)
 {
@@ -669,6 +738,10 @@ static bool smmuv3_accel_set_iommu_device(PCIBus *bus, void *opaque, int devfn,
         goto done;
     }
 
+    if (!smmuv3_accel_select_cmdqv(s, hiodi, errp)) {
+        return false;
+    }
+
     if (!smmuv3_accel_alloc_viommu(s, hiodi, errp)) {
         error_append_hint(errp, "Unable to alloc vIOMMU: hiodi devid 0x%x: ",
                           hiodi->devid);
@@ -946,8 +1019,17 @@ bool smmuv3_accel_attach_gbpa_hwpt(SMMUv3State *s, Error **errp)
 
 void smmuv3_accel_reset(SMMUv3State *s)
 {
-     /* Attach a HWPT based on GBPA reset value */
-     smmuv3_accel_attach_gbpa_hwpt(s, NULL);
+    SMMUv3AccelState *accel = s->s_accel;
+
+    if (!accel) {
+        return;

hw/arm/smmuv3-accel.c bool smmuv3_accel_attach_gbpa_hwpt(SMMUv3State *s, Error **errp) · line 1035

Quoted 10 lines
hw/arm/smmuv3-accel.c +7 -0
+    }
+    /* Attach a HWPT based on GBPA reset value */
+    smmuv3_accel_attach_gbpa_hwpt(s, NULL);
+
+    if (accel->cmdqv_ops && accel->cmdqv_ops->reset) {
+        accel->cmdqv_ops->reset(s);
+    }
 }
 
 static void smmuv3_accel_as_init(SMMUv3State *s)

Eric

#11 lore raw

RE: [PATCH v5 08/32] hw/arm/smmuv3-accel: Wire CMDQV ops into accel lifecycle

Hi Eric,

Quoted 102 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 20 May 2026 06:27
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 08/32] hw/arm/smmuv3-accel: Wire CMDQV ops into accel lifecycle

External email: Use caution opening links or attachments

Hi SHameer,

Quoted 4 lines · Shameer Kolothum

Add support for selecting and initializing a CMDQV backend based on the cmdqv OnOffAuto property.

If set to OFF, CMDQV is not used and the default IOMMUFD-backed

allocation

Quoted 3 lines

path is taken.

If set to AUTO, QEMU attempts to probe a CMDQV backend during device

setup.

Quoted 4 lines

If probing succeeds, the selected ops are stored in the accelerated SMMUv3 state and used. If probing fails, QEMU silently falls back to the default path.

Quoted 8 lines

If set to ON, QEMU requires CMDQV support. Probing is performed during setup and failure results in an error.

When a CMDQV backend is active, its callbacks are used for vIOMMU allocation, free, and reset handling. Otherwise, the base implementation is used.

The current implementation wires up the Tegra241 CMDQV backend

through the

Quoted 11 lines

generic ops interface. Functional CMDQV behaviour is added in subsequent patches.

No functional change.

---
 include/hw/arm/smmuv3.h |  2 +
 hw/arm/smmuv3-accel.c   | 96

++++++++++++++++++++++++++++++++++++++---

Quoted 20 lines
 2 files changed, 91 insertions(+), 7 deletions(-)
include/hw/arm/smmuv3.h +2 -0
index fe0493c1aa..aa6a79237a 100644
@@ -74,6 +74,8 @@ struct SMMUv3State {
     OnOffAuto ats;
     OasMode oas;
     SsidSizeMode ssidsize;
+    /* SMMU CMDQV extension */
+    OnOffAuto cmdqv;
 
     Notifier machine_done;
 };
hw/arm/smmuv3-accel.c +0 -0
index 30c498ffcf..202b1aedd9 100644

hw/arm/smmuv3-accel.c line 25

Quoted 7 lines
hw/arm/smmuv3-accel.c +1 -0
@@ -19,6 +19,7 @@
 #include "smmuv3-internal.h"
 #include "smmuv3-accel.h"
 #include "system/system.h"
+#include "tegra241-cmdqv.h"
 
 /*
  * The root region aliases the global system memory, and

shared_as_sysmem

hw/arm/smmuv3-accel.c smmuv3_accel_alloc_viommu(SMMUv3State *s,

Quoted 1 line
hw/arm/smmuv3-accel.c +0 -0
@@ -570,6 +571,7 @@ smmuv3_accel_alloc_viommu(SMMUv3State *s,

HostIOMMUDeviceIOMMUFD *hiodi,

hw/arm/smmuv3-accel.c smmuv3_accel_alloc_viommu(SMMUv3State *s,

Quoted 8 lines
hw/arm/smmuv3-accel.c +1 -0
                           Error **errp)
 {
     SMMUv3AccelState *accel = s->s_accel;
+    const SMMUv3AccelCmdqvOps *cmdqv_ops = accel->cmdqv_ops;
     struct iommu_hwpt_arm_smmuv3 bypass_data = {
         .ste = { SMMU_STE_CFG_BYPASS | SMMU_STE_VALID, 0x0ULL },
     };
@@ -580,10 +582,17 @@ smmuv3_accel_alloc_viommu(SMMUv3State *s,

HostIOMMUDeviceIOMMUFD *hiodi,

hw/arm/smmuv3-accel.c smmuv3_accel_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *hiodi,

Quoted 5 lines
hw/arm/smmuv3-accel.c +0 -2
     uint32_t viommu_id, hwpt_id;
     IOMMUFDViommu *viommu;
 
-    if (!iommufd_backend_alloc_viommu(hiodi->iommufd, hiodi->devid,
-                                      IOMMU_VIOMMU_TYPE_ARM_SMMUV3,

s2_hwpt_id,

hw/arm/smmuv3-accel.c smmuv3_accel_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *hiodi, · line 586

Quoted 4 lines
hw/arm/smmuv3-accel.c +2 -2
-                                      NULL, 0, &viommu_id, errp)) {
-        return false;
+    if (cmdqv_ops) {
+        if (!cmdqv_ops->alloc_viommu(s, hiodi, &viommu_id, errp)) {
Quoted 7 lines

the doc comment in 6/32 says that alloc_iommu is mandatory if probe is implemented. So it looks there are situations where alloc_iommu may be null.

But actually given the implementation of smmuv3_accel_probe_cmdqv below,
the ops is only non NULL if probe has succeeded. So shouldn't probe be also mandatory?

Yeah. You are right. Probe is mandatory if ops is non NULL. I will revise the comments.

Quoted 78 lines
Quoted 19 lines
hw/arm/smmuv3-accel.c +11 -1
+            return false;
+        }
+    } else {
+        if (!iommufd_backend_alloc_viommu(hiodi->iommufd, hiodi->devid,
+                                          IOMMU_VIOMMU_TYPE_ARM_SMMUV3,
+                                          s2_hwpt_id, NULL, 0, &viommu_id,
+                                          errp)) {
+            return false;
+        }
     }
 
     viommu = g_new0(IOMMUFDViommu, 1);
@@ -629,12 +638,72 @@ free_bypass_hwpt:
 free_abort_hwpt:
     iommufd_backend_free_id(hiodi->iommufd, accel->abort_hwpt_id);
 free_viommu:
-    iommufd_backend_free_id(hiodi->iommufd, viommu->viommu_id);
+    if (cmdqv_ops && cmdqv_ops->free_viommu) {
+        cmdqv_ops->free_viommu(s);

hw/arm/smmuv3-accel.c free_bypass_hwpt: · line 652

Quoted 10 lines
hw/arm/smmuv3-accel.c +5 -0
+    } else {
+        iommufd_backend_free_id(hiodi->iommufd, viommu->viommu_id);
+    }
     g_free(viommu);
     accel->viommu = NULL;
     return false;
 }
 
+static const SMMUv3AccelCmdqvOps *
+smmuv3_accel_probe_cmdqv(SMMUv3State *s,

HostIOMMUDeviceIOMMUFD *idev,

Quoted 7 lines
hw/arm/smmuv3-accel.c +7 -0
+                         Error **errp)
+{
+    const SMMUv3AccelCmdqvOps *ops = tegra241_cmdqv_get_ops();
+
+    if (!ops || !ops->probe) {
+        error_setg(errp, "No CMDQV ops found");
+        return NULL;

hw/arm/smmuv3-accel.c free_bypass_hwpt:

Quoted 10 lines
hw/arm/smmuv3-accel.c +10 -0
+    }
+
+    if (!ops->probe(s, idev, errp)) {
+        return NULL;
+    }
+    return ops;
+}
+
+static bool
+smmuv3_accel_select_cmdqv(SMMUv3State *s,

HostIOMMUDeviceIOMMUFD *idev,

Quoted 18 lines
hw/arm/smmuv3-accel.c +18 -0
+                          Error **errp)
+{
+    const SMMUv3AccelCmdqvOps *ops = NULL;
+
+    if (s->s_accel->cmdqv_ops) {
+        return true;
+    }
+
+    switch (s->cmdqv) {
+    case ON_OFF_AUTO_OFF:
+        s->s_accel->cmdqv_ops = NULL;
+        return true;
+    case ON_OFF_AUTO_AUTO:
+        ops = smmuv3_accel_probe_cmdqv(s, idev, NULL);
+        break;
+    case ON_OFF_AUTO_ON:
+        ops = smmuv3_accel_probe_cmdqv(s, idev, errp);
+        if (!ops) {

hw/arm/smmuv3-accel.c free_bypass_hwpt: · line 697

Quoted 10 lines
hw/arm/smmuv3-accel.c +10 -0
+            error_append_hint(errp, "CMDQV requested but not supported");
+            return false;
+        }
+        break;
+    default:
+        g_assert_not_reached();
+    }
+
+    if (ops) {
+        g_assert(ops->alloc_viommu);

I think I would move the assert at the end of

Quoted 1 line

smmuv3_accel_probe_cmdqv()

Agree. I will change it into:

smmuv3_accel_probe_cmdqv()

  {
     ....
      if (!ops) {
          error_setg(errp, "No CMDQV ops found");
          return NULL;
      }
      g_assert(ops->probe);
      g_assert(ops->alloc_viommu);

      if (!ops->probe(s, idev, errp)) {
          return NULL;
      }
      return ops;

}

Thanks,
Shameer

#12 lore raw

PATCH v5 9/32 hw/arm/virt: Use stored SMMUv3 device list for IORT build

In reply to Shameer Kolothum

Introduce a GPtrArray in VirtMachineState to track all SMMUv3 devices created on the virt machine, and use it when building the IORT table instead of relying on object_child_foreach_recursive() walks of the object tree.

This avoids recursive object traversal and provides a foundation for subsequent patches that need direct access to SMMUv3 instances for CMDQV-related handling.

No functional change. No bios-tables qtest failures observed.

---
 include/hw/arm/virt.h    |  1 +
 hw/arm/virt-acpi-build.c | 70 ++++++++++++++++++----------------------
 hw/arm/virt.c            |  3 ++
 3 files changed, 35 insertions(+), 39 deletions(-)
include/hw/arm/virt.h +1 -0
index 3ba33b4bd2..171d44c644 100644
@@ -207,6 +207,7 @@ struct VirtMachineState {
     MemoryRegion *sysmem;
     MemoryRegion *secure_sysmem;
     bool pci_preserve_config;
+    GPtrArray *smmuv3_devices;
 };
 
 #define VIRT_ECAM_ID(high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
hw/arm/virt-acpi-build.c +31 -39

87 lines not on this page · raw message

hw/arm/virt.c +3 -0
index b090233893..ac0606fe87 100644
@@ -3865,6 +3865,7 @@ static void virt_machine_device_plug_cb(HotplugHandler *hotplug_dev,
             }
 
             create_smmuv3_dev_dtb(vms, dev, bus, errp);
+            g_ptr_array_add(vms->smmuv3_devices, dev);
         }
     }
 
@@ -4319,6 +4320,8 @@ static void virt_instance_init(Object *obj)
     vms->oem_id = g_strndup(ACPI_BUILD_APPNAME6, 6);
     vms->oem_table_id = g_strndup(ACPI_BUILD_APPNAME8, 8);
     cxl_machine_init(obj, &vms->cxl_devices_state);
+
+    vms->smmuv3_devices = g_ptr_array_new_with_free_func(NULL);
 }
 
 static void virt_instance_finalize(Object *obj)
Signature · 1 line
2.43.0
#13 lore raw

PATCH v5 10/32 hw/arm/tegra241-cmdqv: Probe host Tegra241 CMDQV support

In reply to Shameer Kolothum

Use IOMMU_GET_HW_INFO to query host support for Tegra241 CMDQV.

Validate the returned data type, version, and minimum number of vCMDQs and SIDs per Tegra241 CMDQ Virtual Interface(VI). Fail the probe if the host does not meet these requirements.

The QEMU model supports one Virtual Interface(VI) per VM with 2 vCMDQs and 16 SIDs per VI, so the probe ensures the host implementation is compatible with these limits.

---
 hw/arm/tegra241-cmdqv.h |  4 ++++
 hw/arm/tegra241-cmdqv.c | 32 ++++++++++++++++++++++++++++++--
 2 files changed, 34 insertions(+), 2 deletions(-)
hw/arm/tegra241-cmdqv.h +4 -0
index 74a6954017..38c8b27b4d 100644
@@ -10,6 +10,10 @@
 #ifndef HW_ARM_TEGRA241_CMDQV_H
 #define HW_ARM_TEGRA241_CMDQV_H
 
+#define CMDQV_VER                 1
+#define CMDQV_NUM_CMDQ_LOG2       1
+#define CMDQV_NUM_SID_PER_VI_LOG2 4
+
 const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops(void);
 
 #endif /* HW_ARM_TEGRA241_CMDQV_H */
hw/arm/tegra241-cmdqv.c +30 -2
index ad5a0d4611..3a19a1af56 100644
@@ -38,8 +38,36 @@ static bool tegra241_cmdqv_init(SMMUv3State *s, Error **errp)
 static bool tegra241_cmdqv_probe(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
                                  Error **errp)
 {
-    error_setg(errp, "NVIDIA Tegra241 CMDQV is unsupported");
-    return false;
+    uint32_t data_type = IOMMU_HW_INFO_TYPE_TEGRA241_CMDQV;
+    struct iommu_hw_info_tegra241_cmdqv cmdqv_info;
+    uint64_t caps;
+
+    if (!iommufd_backend_get_device_info(idev->iommufd, idev->devid, &data_type,
+                                         &cmdqv_info, sizeof(cmdqv_info), &caps,
+                                         NULL, errp)) {
+        return false;
+    }
+    if (data_type != IOMMU_HW_INFO_TYPE_TEGRA241_CMDQV) {
+        error_setg(errp, "Host CMDQV: unexpected data type %u (expected %u)",
+                   data_type, IOMMU_HW_INFO_TYPE_TEGRA241_CMDQV);
+        return false;
+    }
+    if (cmdqv_info.version != CMDQV_VER) {
+        error_setg(errp, "Host CMDQV: unsupported version %u (expected %u)",
+                   cmdqv_info.version, CMDQV_VER);
+        return false;
+    }
+    if (cmdqv_info.log2vcmdqs < CMDQV_NUM_CMDQ_LOG2) {
+        error_setg(errp, "Host CMDQV: insufficient vCMDQs log2=%u (need >= %u)",
+                   cmdqv_info.log2vcmdqs, CMDQV_NUM_CMDQ_LOG2);
+        return false;
+    }
+    if (cmdqv_info.log2vsids < CMDQV_NUM_SID_PER_VI_LOG2) {
+        error_setg(errp, "Host CMDQV: insufficient SIDs log2=%u (need >= %u)",
+                   cmdqv_info.log2vsids, CMDQV_NUM_SID_PER_VI_LOG2);
+        return false;
+    }
+    return true;
 }
 
 static const SMMUv3AccelCmdqvOps tegra241_cmdqv_ops = {
Signature · 1 line
2.43.0
#14 lore raw

PATCH v5 11/32 hw/arm/tegra241-cmdqv: Implement CMDQV init

In reply to Shameer Kolothum

From: Nicolin Chen <[email protected]>

Tegra241 CMDQV extends SMMUv3 with support for virtual command queues (VCMDQs) exposed via a CMDQV MMIO region. The CMDQV MMIO space is split into 64KB pages:

0x00000  (CMDQ-V Config page)
0x10000  (CMDQ-V CMDQ Page0)
0x20000  (CMDQ-V CMDQ Page1)
0x30000  (Virtual Interface Page0)
0x40000  (Virtual Interface Page1)

This patch wires up the Tegra241 CMDQV init callback and allocates vendor-specific CMDQV state. The state pointer is stored in SMMUv3AccelState for use by subsequent CMDQV operations.

The CMDQV MMIO region and a dedicated IRQ line are registered with the SMMUv3 device. The MMIO read/write handlers are currently stubs and will be implemented in later patches.

The CMDQV interrupt is edge-triggered and indicates VCMDQ or VINTF error conditions. This patch only registers the IRQ line. Interrupt generation and propagation to the guest will be added in a subsequent patch.

---
 hw/arm/smmuv3-accel.h   |  1 +
 hw/arm/tegra241-cmdqv.h | 18 ++++++++++++++++++
 hw/arm/tegra241-cmdqv.c | 38 ++++++++++++++++++++++++++++++++++++--
 3 files changed, 55 insertions(+), 2 deletions(-)
hw/arm/smmuv3-accel.h +1 -0
index f82a7112d8..49c10535cf 100644
@@ -62,6 +62,7 @@ typedef struct SMMUv3AccelState {
     bool auto_mode;
     bool auto_finalised;
     const SMMUv3AccelCmdqvOps *cmdqv_ops;
+    void *cmdqv;  /* vendor specific CMDQV state */
 } SMMUv3AccelState;
 
 typedef struct SMMUS1Hwpt {
hw/arm/tegra241-cmdqv.h +18 -0
index 38c8b27b4d..030f5758d5 100644
@@ -14,6 +14,24 @@
 #define CMDQV_NUM_CMDQ_LOG2       1
 #define CMDQV_NUM_SID_PER_VI_LOG2 4
 
+/*
+ * Tegra241 CMDQV MMIO layout (64KB pages)
+ *
+ * 0x00000  (CMDQ-V Config page)
+ * 0x10000  (CMDQ-V CMDQ Page0)
+ * 0x20000  (CMDQ-V CMDQ Page1)
+ * 0x30000  (Virtual Interface Page0)
+ * 0x40000  (Virtual Interface Page1)
+ */
+#define TEGRA241_CMDQV_IO_LEN 0x50000
+
+typedef struct Tegra241CMDQV {
+    struct iommu_viommu_tegra241_cmdqv cmdqv_data;
+    SMMUv3AccelState *s_accel;
+    MemoryRegion mmio_cmdqv;
+    qemu_irq irq;
+} Tegra241CMDQV;
+
 const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops(void);
 
 #endif /* HW_ARM_TEGRA241_CMDQV_H */
hw/arm/tegra241-cmdqv.c +36 -2
index 3a19a1af56..298671e0ce 100644
@@ -13,6 +13,16 @@
 #include "smmuv3-accel.h"
 #include "tegra241-cmdqv.h"
 
+static uint64_t tegra241_cmdqv_read_mmio(void *opaque, hwaddr offset, unsigned size)
+{
+    return 0;
+}
+
+static void tegra241_cmdqv_write_mmio(void *opaque, hwaddr offset, uint64_t value,
+                                 unsigned size)
+{
+}
+
 static void tegra241_cmdqv_free_viommu(SMMUv3State *s)
 {
 }
@@ -29,10 +39,34 @@ static void tegra241_cmdqv_reset(SMMUv3State *s)
 {
 }
 
+static const MemoryRegionOps mmio_cmdqv_ops = {
+    .read = tegra241_cmdqv_read_mmio,
+    .write = tegra241_cmdqv_write_mmio,
+    .endianness = DEVICE_LITTLE_ENDIAN,
+    .valid = {
+        .min_access_size = 4,
+        .max_access_size = 8,
+    },
+    .impl = {
+        .min_access_size = 4,
+        .max_access_size = 8,
+    },
+};
+
 static bool tegra241_cmdqv_init(SMMUv3State *s, Error **errp)
 {
-    error_setg(errp, "NVIDIA Tegra241 CMDQV is unsupported");
-    return false;
+    SysBusDevice *sbd = SYS_BUS_DEVICE(OBJECT(s));
+    SMMUv3AccelState *accel = s->s_accel;
+    Tegra241CMDQV *cmdqv;
+
+    cmdqv = g_new0(Tegra241CMDQV, 1);
+    memory_region_init_io(&cmdqv->mmio_cmdqv, OBJECT(s), &mmio_cmdqv_ops, cmdqv,
+                          "tegra241-cmdqv", TEGRA241_CMDQV_IO_LEN);
+    sysbus_init_mmio(sbd, &cmdqv->mmio_cmdqv);
+    sysbus_init_irq(sbd, &cmdqv->irq);
+    cmdqv->s_accel = accel;
+    accel->cmdqv = cmdqv;
+    return true;
 }
 
 static bool tegra241_cmdqv_probe(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
Signature · 1 line
2.43.0
#15 lore raw

Re: [PATCH v5 11/32] hw/arm/tegra241-cmdqv: Implement CMDQV init

Hello Shameer,

Quoted 58 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

Tegra241 CMDQV extends SMMUv3 with support for virtual command queues (VCMDQs) exposed via a CMDQV MMIO region. The CMDQV MMIO space is split into 64KB pages:

0x00000  (CMDQ-V Config page)
0x10000  (CMDQ-V CMDQ Page0)
0x20000  (CMDQ-V CMDQ Page1)
0x30000  (Virtual Interface Page0)
0x40000  (Virtual Interface Page1)

This patch wires up the Tegra241 CMDQV init callback and allocates vendor-specific CMDQV state. The state pointer is stored in SMMUv3AccelState for use by subsequent CMDQV operations.

The CMDQV MMIO region and a dedicated IRQ line are registered with the SMMUv3 device. The MMIO read/write handlers are currently stubs and will be implemented in later patches.

The CMDQV interrupt is edge-triggered and indicates VCMDQ or VINTF error conditions. This patch only registers the IRQ line. Interrupt generation and propagation to the guest will be added in a subsequent patch.

---
  hw/arm/smmuv3-accel.h   |  1 +
  hw/arm/tegra241-cmdqv.h | 18 ++++++++++++++++++
  hw/arm/tegra241-cmdqv.c | 38 ++++++++++++++++++++++++++++++++++++--
  3 files changed, 55 insertions(+), 2 deletions(-)
hw/arm/smmuv3-accel.h +1 -0
index f82a7112d8..49c10535cf 100644
@@ -62,6 +62,7 @@ typedef struct SMMUv3AccelState {
      bool auto_mode;
      bool auto_finalised;
      const SMMUv3AccelCmdqvOps *cmdqv_ops;
+    void *cmdqv;  /* vendor specific CMDQV state */
  } SMMUv3AccelState;
 
  typedef struct SMMUS1Hwpt {
hw/arm/tegra241-cmdqv.h +3 -0
index 38c8b27b4d..030f5758d5 100644
@@ -14,6 +14,24 @@
  #define CMDQV_NUM_CMDQ_LOG2       1
  #define CMDQV_NUM_SID_PER_VI_LOG2 4
 
+/*
+ * Tegra241 CMDQV MMIO layout (64KB pages)
+ *

hw/arm/tegra241-cmdqv.h line 29

Quoted 10 lines
hw/arm/tegra241-cmdqv.h +10 -0
+ * 0x00000  (CMDQ-V Config page)
+ * 0x10000  (CMDQ-V CMDQ Page0)
+ * 0x20000  (CMDQ-V CMDQ Page1)
+ * 0x30000  (Virtual Interface Page0)
+ * 0x40000  (Virtual Interface Page1)
+ */
+#define TEGRA241_CMDQV_IO_LEN 0x50000
+
+typedef struct Tegra241CMDQV {
+    struct iommu_viommu_tegra241_cmdqv cmdqv_data;

This won't build on Windows.

C.

Quoted 66 lines
hw/arm/tegra241-cmdqv.h +5 -0
+    SMMUv3AccelState *s_accel;
+    MemoryRegion mmio_cmdqv;
+    qemu_irq irq;
+} Tegra241CMDQV;
+
  const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops(void);
 
  #endif /* HW_ARM_TEGRA241_CMDQV_H */
hw/arm/tegra241-cmdqv.c +36 -2
index 3a19a1af56..298671e0ce 100644
@@ -13,6 +13,16 @@
  #include "smmuv3-accel.h"
  #include "tegra241-cmdqv.h"
 
+static uint64_t tegra241_cmdqv_read_mmio(void *opaque, hwaddr offset, unsigned size)
+{
+    return 0;
+}
+
+static void tegra241_cmdqv_write_mmio(void *opaque, hwaddr offset, uint64_t value,
+                                 unsigned size)
+{
+}
+
  static void tegra241_cmdqv_free_viommu(SMMUv3State *s)
  {
  }
@@ -29,10 +39,34 @@ static void tegra241_cmdqv_reset(SMMUv3State *s)
  {
  }
 
+static const MemoryRegionOps mmio_cmdqv_ops = {
+    .read = tegra241_cmdqv_read_mmio,
+    .write = tegra241_cmdqv_write_mmio,
+    .endianness = DEVICE_LITTLE_ENDIAN,
+    .valid = {
+        .min_access_size = 4,
+        .max_access_size = 8,
+    },
+    .impl = {
+        .min_access_size = 4,
+        .max_access_size = 8,
+    },
+};
+
  static bool tegra241_cmdqv_init(SMMUv3State *s, Error **errp)
  {
-    error_setg(errp, "NVIDIA Tegra241 CMDQV is unsupported");
-    return false;
+    SysBusDevice *sbd = SYS_BUS_DEVICE(OBJECT(s));
+    SMMUv3AccelState *accel = s->s_accel;
+    Tegra241CMDQV *cmdqv;
+
+    cmdqv = g_new0(Tegra241CMDQV, 1);
+    memory_region_init_io(&cmdqv->mmio_cmdqv, OBJECT(s), &mmio_cmdqv_ops, cmdqv,
+                          "tegra241-cmdqv", TEGRA241_CMDQV_IO_LEN);
+    sysbus_init_mmio(sbd, &cmdqv->mmio_cmdqv);
+    sysbus_init_irq(sbd, &cmdqv->irq);
+    cmdqv->s_accel = accel;
+    accel->cmdqv = cmdqv;
+    return true;
  }
 
  static bool tegra241_cmdqv_probe(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
#16 lore raw

RE: [PATCH v5 11/32] hw/arm/tegra241-cmdqv: Implement CMDQV init

Hi Cédric,

Quoted 76 lines

-----Original Message-----
From: Cédric Le Goater <[email protected]>
Sent: 21 May 2026 09:19
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 11/32] hw/arm/tegra241-cmdqv: Implement CMDQV init

External email: Use caution opening links or attachments

Hello Shameer,

Quoted 3 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

Tegra241 CMDQV extends SMMUv3 with support for virtual command

queues

Quoted 22 lines

(VCMDQs) exposed via a CMDQV MMIO region. The CMDQV MMIO space is split into 64KB pages:

0x00000  (CMDQ-V Config page)
0x10000  (CMDQ-V CMDQ Page0)
0x20000  (CMDQ-V CMDQ Page1)
0x30000  (Virtual Interface Page0)
0x40000  (Virtual Interface Page1)

This patch wires up the Tegra241 CMDQV init callback and allocates vendor-specific CMDQV state. The state pointer is stored in SMMUv3AccelState for use by subsequent CMDQV operations.

The CMDQV MMIO region and a dedicated IRQ line are registered with the SMMUv3 device. The MMIO read/write handlers are currently stubs and will be implemented in later patches.

The CMDQV interrupt is edge-triggered and indicates VCMDQ or VINTF error conditions. This patch only registers the IRQ line. Interrupt generation and propagation to the guest will be added in a subsequent patch.

Quoted 8 lines
---
  hw/arm/smmuv3-accel.h   |  1 +
  hw/arm/tegra241-cmdqv.h | 18 ++++++++++++++++++
  hw/arm/tegra241-cmdqv.c | 38

++++++++++++++++++++++++++++++++++++--

Quoted 22 lines
  3 files changed, 55 insertions(+), 2 deletions(-)
hw/arm/smmuv3-accel.h index renamed from hw/arm/smmuv3-accel.h +0 -0

f82a7112d8..49c10535cf 100644

hw/arm/smmuv3-accel.h +1 -0
@@ -62,6 +62,7 @@ typedef struct SMMUv3AccelState {
      bool auto_mode;
      bool auto_finalised;
      const SMMUv3AccelCmdqvOps *cmdqv_ops;
+    void *cmdqv;  /* vendor specific CMDQV state */
  } SMMUv3AccelState;
 
  typedef struct SMMUS1Hwpt {
hw/arm/tegra241-cmdqv.h index renamed from hw/arm/tegra241-cmdqv.h +0 -0

38c8b27b4d..030f5758d5 100644

hw/arm/tegra241-cmdqv.h +0 -0
@@ -14,6 +14,24 @@
  #define CMDQV_NUM_CMDQ_LOG2       1
  #define CMDQV_NUM_SID_PER_VI_LOG2 4
 
Quoted 14 lines

hw/arm/tegra241-cmdqv.h line 29

Quoted 12 lines
hw/arm/tegra241-cmdqv.h +12 -0
+/*
+ * Tegra241 CMDQV MMIO layout (64KB pages)
+ *
+ * 0x00000  (CMDQ-V Config page)
+ * 0x10000  (CMDQ-V CMDQ Page0)
+ * 0x20000  (CMDQ-V CMDQ Page1)
+ * 0x30000  (Virtual Interface Page0)
+ * 0x40000  (Virtual Interface Page1)  */ #define
+TEGRA241_CMDQV_IO_LEN 0x50000
+
+typedef struct Tegra241CMDQV {
+    struct iommu_viommu_tegra241_cmdqv cmdqv_data;

This won't build on Windows.

Oops..I did that again!. Will fix it in v6.

Thanks,
Shameer

#17 lore raw

PATCH v5 12/32 hw/arm/virt: Link SMMUv3 CMDQV resources to platform bus

In reply to Shameer Kolothum

SMMUv3 devices with acceleration may enable CMDQV extensions after device realize. In that case, additional MMIO regions and IRQ lines may be registered but not yet mapped to the platform bus.

Ensure SMMUv3 device resources are linked to the platform bus during machine_done().

This is safe to do unconditionally since the platform bus helpers skip resources that are already mapped.

---
 hw/arm/virt.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
hw/arm/virt.c +22 -0
index ac0606fe87..2add7401a1 100644
@@ -2361,6 +2361,25 @@ static void virt_build_smbios(VirtMachineState *vms)
     }
 }
 
+/*
+ * SMMUv3 devices with acceleration may enable CMDQV extensions
+ * after device realize. In that case, additional MMIO regions and
+ * IRQ lines may be registered but not yet mapped to the platform bus.
+ *
+ * Ensure all resources are linked to the platform bus before final
+ * machine setup.
+ */
+
+static void virt_smmuv3_dev_link_cmdqv(VirtMachineState *vms)
+{
+    for (int i = 0; i < vms->smmuv3_devices->len; i++) {
+        DeviceState *dev = g_ptr_array_index(vms->smmuv3_devices, i);
+
+        platform_bus_link_device(PLATFORM_BUS_DEVICE(vms->platform_bus_dev),
+                                 SYS_BUS_DEVICE(dev));
+    }
+}
+
 static
 void virt_machine_done(Notifier *notifier, void *data)
 {
@@ -2377,6 +2396,9 @@ void virt_machine_done(Notifier *notifier, void *data)
     if (vms->cxl_devices_state.is_enabled) {
         cxl_fmws_link_targets(&error_fatal);
     }
+
+    virt_smmuv3_dev_link_cmdqv(vms);
+
     /*
      * If the user provided a dtb, we assume the dynamic sysbus nodes
      * already are integrated there. This corresponds to a use case where
Signature · 1 line
2.43.0
#18 lore raw

PATCH v5 13/32 hw/arm/tegra241-cmdqv: Implement CMDQV vIOMMU alloc/free

In reply to Shameer Kolothum

From: Nicolin Chen <[email protected]>

Replace the stub implementation with real vIOMMU allocation for Tegra241 CMDQV.

Allocate a matching vEVENTQ together with the vIOMMU, since it is specific to the Tegra241 CMDQV vIOMMU and used to receive CMDQV events.

Free both objects on teardown.

---
 hw/arm/tegra241-cmdqv.h |  1 +
 hw/arm/tegra241-cmdqv.c | 48 ++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 48 insertions(+), 1 deletion(-)
hw/arm/tegra241-cmdqv.h +1 -0
index 030f5758d5..54854421f2 100644
@@ -30,6 +30,7 @@ typedef struct Tegra241CMDQV {
     SMMUv3AccelState *s_accel;
     MemoryRegion mmio_cmdqv;
     qemu_irq irq;
+    IOMMUFDVeventq *veventq;
 } Tegra241CMDQV;
 
 const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops(void);
hw/arm/tegra241-cmdqv.c +47 -1

66 lines not on this page · raw message

Signature · 1 line
2.43.0
#19 lore raw

PATCH v5 14/32 hw/arm/tegra241-cmdqv: Emulate CMDQ-V Config region

In reply to Shameer Kolothum

From: Nicolin Chen <[email protected]>

Tegra241 CMDQV exposes control and status registers in the CMDQ-V Config page (offset [0x0, 0x10000)) used to configure virtual command queue allocation and interrupt behavior.

Add read/write emulation for the CMDQ-V Config region ([CMDQV_BASE, CMDQV_CMDQ_BASE]), backed by a simple register cache. This includes CONFIG, PARAM, STATUS, VI error and interrupt maps, CMDQ allocation map and the VINTF0 related registers defined in the CMDQ-V Config space. Only VINTF0 is supported; VINTF1-63 are not.

Dispatch writes on access size: Introduced writel_mmio for 4-byte and writell_mmio for 8-byte. Reads need no split as the MMIO framework masks the returned value to the access size.

---
 hw/arm/tegra241-cmdqv.h | 110 +++++++++++++++++++++++
 hw/arm/tegra241-cmdqv.c | 188 +++++++++++++++++++++++++++++++++++++++-
 hw/arm/trace-events     |   4 +
 3 files changed, 298 insertions(+), 4 deletions(-)
hw/arm/tegra241-cmdqv.h +110 -0
index 54854421f2..ace327443a 100644
@@ -10,10 +10,15 @@
 #ifndef HW_ARM_TEGRA241_CMDQV_H
 #define HW_ARM_TEGRA241_CMDQV_H
 
+#include "hw/core/registerfields.h"
+
 #define CMDQV_VER                 1
 #define CMDQV_NUM_CMDQ_LOG2       1
 #define CMDQV_NUM_SID_PER_VI_LOG2 4
 
+#define TEGRA241_CMDQV_MAX_CMDQ      (1U << CMDQV_NUM_CMDQ_LOG2)
+#define TEGRA241_CMDQV_MAX_NUM_SID   (1U << CMDQV_NUM_SID_PER_VI_LOG2)
+
 /*
  * Tegra241 CMDQV MMIO layout (64KB pages)
  *
@@ -31,8 +36,113 @@ typedef struct Tegra241CMDQV {
     MemoryRegion mmio_cmdqv;
     qemu_irq irq;
     IOMMUFDVeventq *veventq;
+
+    /* CMDQ-V Config page register cache */
+    uint32_t config;
+    uint32_t param;
+    uint32_t status;
+    uint32_t vi_err_map[2];
+    uint32_t vi_int_mask[2];
+    uint32_t cmdq_err_map[4];
+    uint32_t cmdq_alloc_map[TEGRA241_CMDQV_MAX_CMDQ];
+
+    /* VINTF0 register cache (within CMDQ-V Config page) */
+    uint32_t vintf_config;
+    uint32_t vintf_status;
+    uint32_t vintf_sid_match[TEGRA241_CMDQV_MAX_NUM_SID];
+    uint32_t vintf_sid_replace[TEGRA241_CMDQV_MAX_NUM_SID];
+    uint32_t vintf_cmdq_err_map[4];
 } Tegra241CMDQV;
 
+/* CMDQ-V Config page registers (offset 0x00000) */
+REG32(CONFIG, 0x0)
+FIELD(CONFIG, CMDQV_EN, 0, 1)
+FIELD(CONFIG, CMDQV_PER_CMD_OFFSET, 1, 3)
+FIELD(CONFIG, CMDQ_MAX_CLK_BATCH, 4, 8)
+FIELD(CONFIG, CMDQ_MAX_CMD_BATCH, 12, 8)
+FIELD(CONFIG, CONS_DRAM_EN, 20, 1)
+
+REG32(PARAM, 0x4)
+FIELD(PARAM, CMDQV_VER, 0, 4)
+FIELD(PARAM, CMDQV_NUM_CMDQ_LOG2, 4, 4)
+FIELD(PARAM, CMDQV_NUM_VI_LOG2, 8, 4)
+FIELD(PARAM, CMDQV_NUM_SID_PER_VI_LOG2, 12, 4)
+
+REG32(STATUS, 0x8)
+FIELD(STATUS, CMDQV_ENABLED, 0, 1)
+
+/* SMMU_CMDQV_VI_ERR_MAP_0/1 definitions */
+#define A_VI_ERR_MAP_0 0x14
+#define A_VI_ERR_MAP_1 0x18
+#define V_VI_ERR_MAP_NO_ERROR (0)
+#define V_VI_ERR_MAP_ERROR (1)
+
+/* SMMU_CMDQV_VI_INT_MASK_0/1 definitions */
+#define A_VI_INT_MASK_0 0x1c
+#define A_VI_INT_MASK_1 0x20
+#define V_VI_INT_MASK_NOT_MASKED (0)
+#define V_VI_INT_MASK_MASKED (1)
+
+/* SMMU_CMDQV_CMDQ_ERR_MAP_0-3 definitions */
+#define A_CMDQ_ERR_MAP_0 0x24
+#define A_CMDQ_ERR_MAP_1 0x28
+#define A_CMDQ_ERR_MAP_2 0x2c
+#define A_CMDQ_ERR_MAP_3 0x30
+
+/*
+ * CMDQ_ALLOC_MAP: one entry per physical VCMDQ. Hardware supports up to 128
+ * entries (CMDQV_NUM_CMDQ_LOG2=7), but QEMU only exposes
+ * TEGRA241_CMDQV_MAX_CMDQ (=2) VCMDQs per VM so only entries 0 and 1 are
+ * defined here.
+ */
+/* 2 identical register entries */
+#define SMMU_CMDQV_CMDQ_ALLOC_MAP_(i)                       \
+    REG32(CMDQ_ALLOC_MAP_##i, 0x200 + i * 4)                \
+    FIELD(CMDQ_ALLOC_MAP_##i, ALLOC, 0, 1)                  \
+    FIELD(CMDQ_ALLOC_MAP_##i, LVCMDQ, 1, 7)                 \
+    FIELD(CMDQ_ALLOC_MAP_##i, VIRT_INTF_INDX, 15, 6)
+
+SMMU_CMDQV_CMDQ_ALLOC_MAP_(0)
+SMMU_CMDQV_CMDQ_ALLOC_MAP_(1)
+
+/* SMMU_CMDQV_VINTF0 registers (only VINTF0 is exposed to the guest) */
+REG32(VINTF0_CONFIG, 0x1000)
+FIELD(VINTF0_CONFIG, ENABLE, 0, 1)
+FIELD(VINTF0_CONFIG, VMID, 1, 16)
+FIELD(VINTF0_CONFIG, HYP_OWN, 17, 1)
+
+REG32(VINTF0_STATUS, 0x1004)
+FIELD(VINTF0_STATUS, ENABLE_OK, 0, 1)
+FIELD(VINTF0_STATUS, STATUS, 1, 3)
+FIELD(VINTF0_STATUS, VI_NUM_LVCMDQ, 16, 8)
+
+#define V_VINTF_STATUS_NO_ERROR    (0 << 1)
+#define V_VINTF_STATUS_VCMDQ_ERROR (1 << 1)
+
+/*
+ * SMMU_CMDQV_VINTF0_SID_MATCH/_REPLACE: 16 entries per VINTF
+ * (CMDQV_NUM_SID_PER_VI_LOG2=4). Only _0 and _15 are defined,
+ * used as switch case range bounds.
+ */
+REG32(VINTF0_SID_MATCH_0, 0x1040)
+FIELD(VINTF0_SID_MATCH_0, ENABLE, 0, 1)
+FIELD(VINTF0_SID_MATCH_0, VIRT_SID, 1, 20)
+#define A_VINTF0_SID_MATCH_15  (A_VINTF0_SID_MATCH_0 + 15 * 4)
+
+REG32(VINTF0_SID_REPLACE_0, 0x1080)
+FIELD(VINTF0_SID_REPLACE_0, PHYS_SID, 0, 20)
+#define A_VINTF0_SID_REPLACE_15 (A_VINTF0_SID_REPLACE_0 + 15 * 4)
+
+/*
+ * SMMU_CMDQV_VINTF0_LVCMDQ_ERR_MAP: 4 registers per VINTF covering 32 logical
+ * VCMDQs each. With TEGRA241_CMDQV_MAX_CMDQ=2, only MAP_0 bits [1:0] carry
+ * error state. MAP_1..MAP_3 always read as 0. Only _0 and _3 are defined,
+ * used as switch case range bounds.
+ */
+REG32(VINTF0_LVCMDQ_ERR_MAP_0, 0x10c0)
+FIELD(VINTF0_LVCMDQ_ERR_MAP_0, LVCMDQ_ERR_MAP, 0, 32)
+#define A_VINTF0_LVCMDQ_ERR_MAP_3 (A_VINTF0_LVCMDQ_ERR_MAP_0 + 3 * 4)
+
 const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops(void);
 
 #endif /* HW_ARM_TEGRA241_CMDQV_H */
hw/arm/tegra241-cmdqv.c +184 -4

204 lines not on this page · raw message

hw/arm/trace-events +4 -0
index 3457536fb0..8c61d66a26 100644
@@ -72,6 +72,10 @@ smmuv3_accel_unset_iommu_device(int devfn, uint32_t devid) "devfn=0x%x (idev dev
 smmuv3_accel_translate_ste(uint32_t vsid, uint32_t hwpt_id, uint64_t ste_1, uint64_t ste_0) "vSID=0x%x hwpt_id=0x%x ste=%"PRIx64":%"PRIx64
 smmuv3_accel_install_ste(uint32_t vsid, const char * type, uint32_t hwpt_id) "vSID=0x%x ste type=%s hwpt_id=0x%x"
 
+# tegra241-cmdqv
+tegra241_cmdqv_read_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x"
+tegra241_cmdqv_write_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x"
+
 # strongarm.c
 strongarm_uart_update_parameters(const char *label, int speed, char parity, int data_bits, int stop_bits) "%s speed=%d parity=%c data=%d stop=%d"
 strongarm_ssp_read_underrun(void) "SSP rx underrun"
Signature · 1 line
2.43.0
Reviewed-by on v5 14/32
#20 lore raw

Re: [PATCH v5 14/32] hw/arm/tegra241-cmdqv: Emulate CMDQ-V Config region

Quoted 373 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

Tegra241 CMDQV exposes control and status registers in the CMDQ-V Config page (offset [0x0, 0x10000)) used to configure virtual command queue allocation and interrupt behavior.

Add read/write emulation for the CMDQ-V Config region ([CMDQV_BASE, CMDQV_CMDQ_BASE]), backed by a simple register cache. This includes CONFIG, PARAM, STATUS, VI error and interrupt maps, CMDQ allocation map and the VINTF0 related registers defined in the CMDQ-V Config space. Only VINTF0 is supported; VINTF1-63 are not.

Dispatch writes on access size: Introduced writel_mmio for 4-byte and writell_mmio for 8-byte. Reads need no split as the MMIO framework masks the returned value to the access size.

---
 hw/arm/tegra241-cmdqv.h | 110 +++++++++++++++++++++++
 hw/arm/tegra241-cmdqv.c | 188 +++++++++++++++++++++++++++++++++++++++-
 hw/arm/trace-events     |   4 +
 3 files changed, 298 insertions(+), 4 deletions(-)
hw/arm/tegra241-cmdqv.h +110 -0
index 54854421f2..ace327443a 100644
@@ -10,10 +10,15 @@
 #ifndef HW_ARM_TEGRA241_CMDQV_H
 #define HW_ARM_TEGRA241_CMDQV_H
 
+#include "hw/core/registerfields.h"
+
 #define CMDQV_VER                 1
 #define CMDQV_NUM_CMDQ_LOG2       1
 #define CMDQV_NUM_SID_PER_VI_LOG2 4
 
+#define TEGRA241_CMDQV_MAX_CMDQ      (1U << CMDQV_NUM_CMDQ_LOG2)
+#define TEGRA241_CMDQV_MAX_NUM_SID   (1U << CMDQV_NUM_SID_PER_VI_LOG2)
+
 /*
  * Tegra241 CMDQV MMIO layout (64KB pages)
  *
@@ -31,8 +36,113 @@ typedef struct Tegra241CMDQV {
     MemoryRegion mmio_cmdqv;
     qemu_irq irq;
     IOMMUFDVeventq *veventq;
+
+    /* CMDQ-V Config page register cache */
+    uint32_t config;
+    uint32_t param;
+    uint32_t status;
+    uint32_t vi_err_map[2];
+    uint32_t vi_int_mask[2];
+    uint32_t cmdq_err_map[4];
+    uint32_t cmdq_alloc_map[TEGRA241_CMDQV_MAX_CMDQ];
+
+    /* VINTF0 register cache (within CMDQ-V Config page) */
+    uint32_t vintf_config;
+    uint32_t vintf_status;
+    uint32_t vintf_sid_match[TEGRA241_CMDQV_MAX_NUM_SID];
+    uint32_t vintf_sid_replace[TEGRA241_CMDQV_MAX_NUM_SID];
+    uint32_t vintf_cmdq_err_map[4];
 } Tegra241CMDQV;
 
+/* CMDQ-V Config page registers (offset 0x00000) */
+REG32(CONFIG, 0x0)
+FIELD(CONFIG, CMDQV_EN, 0, 1)
+FIELD(CONFIG, CMDQV_PER_CMD_OFFSET, 1, 3)
+FIELD(CONFIG, CMDQ_MAX_CLK_BATCH, 4, 8)
+FIELD(CONFIG, CMDQ_MAX_CMD_BATCH, 12, 8)
+FIELD(CONFIG, CONS_DRAM_EN, 20, 1)
+
+REG32(PARAM, 0x4)
+FIELD(PARAM, CMDQV_VER, 0, 4)
+FIELD(PARAM, CMDQV_NUM_CMDQ_LOG2, 4, 4)
+FIELD(PARAM, CMDQV_NUM_VI_LOG2, 8, 4)
+FIELD(PARAM, CMDQV_NUM_SID_PER_VI_LOG2, 12, 4)
+
+REG32(STATUS, 0x8)
+FIELD(STATUS, CMDQV_ENABLED, 0, 1)
+
+/* SMMU_CMDQV_VI_ERR_MAP_0/1 definitions */
+#define A_VI_ERR_MAP_0 0x14
+#define A_VI_ERR_MAP_1 0x18
+#define V_VI_ERR_MAP_NO_ERROR (0)
+#define V_VI_ERR_MAP_ERROR (1)
+
+/* SMMU_CMDQV_VI_INT_MASK_0/1 definitions */
+#define A_VI_INT_MASK_0 0x1c
+#define A_VI_INT_MASK_1 0x20
+#define V_VI_INT_MASK_NOT_MASKED (0)
+#define V_VI_INT_MASK_MASKED (1)
+
+/* SMMU_CMDQV_CMDQ_ERR_MAP_0-3 definitions */
+#define A_CMDQ_ERR_MAP_0 0x24
+#define A_CMDQ_ERR_MAP_1 0x28
+#define A_CMDQ_ERR_MAP_2 0x2c
+#define A_CMDQ_ERR_MAP_3 0x30
+
+/*
+ * CMDQ_ALLOC_MAP: one entry per physical VCMDQ. Hardware supports up to 128
+ * entries (CMDQV_NUM_CMDQ_LOG2=7), but QEMU only exposes
+ * TEGRA241_CMDQV_MAX_CMDQ (=2) VCMDQs per VM so only entries 0 and 1 are
+ * defined here.
+ */
+/* 2 identical register entries */
+#define SMMU_CMDQV_CMDQ_ALLOC_MAP_(i)                       \
+    REG32(CMDQ_ALLOC_MAP_##i, 0x200 + i * 4)                \
+    FIELD(CMDQ_ALLOC_MAP_##i, ALLOC, 0, 1)                  \
+    FIELD(CMDQ_ALLOC_MAP_##i, LVCMDQ, 1, 7)                 \
+    FIELD(CMDQ_ALLOC_MAP_##i, VIRT_INTF_INDX, 15, 6)
+
+SMMU_CMDQV_CMDQ_ALLOC_MAP_(0)
+SMMU_CMDQV_CMDQ_ALLOC_MAP_(1)
+
+/* SMMU_CMDQV_VINTF0 registers (only VINTF0 is exposed to the guest) */
+REG32(VINTF0_CONFIG, 0x1000)
+FIELD(VINTF0_CONFIG, ENABLE, 0, 1)
+FIELD(VINTF0_CONFIG, VMID, 1, 16)
+FIELD(VINTF0_CONFIG, HYP_OWN, 17, 1)
+
+REG32(VINTF0_STATUS, 0x1004)
+FIELD(VINTF0_STATUS, ENABLE_OK, 0, 1)
+FIELD(VINTF0_STATUS, STATUS, 1, 3)
+FIELD(VINTF0_STATUS, VI_NUM_LVCMDQ, 16, 8)
+
+#define V_VINTF_STATUS_NO_ERROR    (0 << 1)
+#define V_VINTF_STATUS_VCMDQ_ERROR (1 << 1)
+
+/*
+ * SMMU_CMDQV_VINTF0_SID_MATCH/_REPLACE: 16 entries per VINTF
+ * (CMDQV_NUM_SID_PER_VI_LOG2=4). Only _0 and _15 are defined,
+ * used as switch case range bounds.
+ */
+REG32(VINTF0_SID_MATCH_0, 0x1040)
+FIELD(VINTF0_SID_MATCH_0, ENABLE, 0, 1)
+FIELD(VINTF0_SID_MATCH_0, VIRT_SID, 1, 20)
+#define A_VINTF0_SID_MATCH_15  (A_VINTF0_SID_MATCH_0 + 15 * 4)
+
+REG32(VINTF0_SID_REPLACE_0, 0x1080)
+FIELD(VINTF0_SID_REPLACE_0, PHYS_SID, 0, 20)
+#define A_VINTF0_SID_REPLACE_15 (A_VINTF0_SID_REPLACE_0 + 15 * 4)
+
+/*
+ * SMMU_CMDQV_VINTF0_LVCMDQ_ERR_MAP: 4 registers per VINTF covering 32 logical
+ * VCMDQs each. With TEGRA241_CMDQV_MAX_CMDQ=2, only MAP_0 bits [1:0] carry
+ * error state. MAP_1..MAP_3 always read as 0. Only _0 and _3 are defined,
+ * used as switch case range bounds.
+ */
+REG32(VINTF0_LVCMDQ_ERR_MAP_0, 0x10c0)
+FIELD(VINTF0_LVCMDQ_ERR_MAP_0, LVCMDQ_ERR_MAP, 0, 32)
+#define A_VINTF0_LVCMDQ_ERR_MAP_3 (A_VINTF0_LVCMDQ_ERR_MAP_0 + 3 * 4)
+
 const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops(void);
 
 #endif /* HW_ARM_TEGRA241_CMDQV_H */
hw/arm/tegra241-cmdqv.c +184 -4
index 6a6cfc2f71..813b9a61ce 100644
@@ -8,20 +8,200 @@
  */
 
 #include "qemu/osdep.h"
+#include "qemu/log.h"
 
 #include "hw/arm/smmuv3.h"
 #include "hw/arm/smmuv3-common.h"
 #include "smmuv3-accel.h"
 #include "tegra241-cmdqv.h"
+#include "trace.h"
 
-static uint64_t tegra241_cmdqv_read_mmio(void *opaque, hwaddr offset, unsigned size)
+static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv,
+                                                 hwaddr offset)
 {
-    return 0;
+    int i;
+
+    switch (offset) {
+    case A_VINTF0_CONFIG:
+        return cmdqv->vintf_config;
+    case A_VINTF0_STATUS:
+        return cmdqv->vintf_status;
+    case A_VINTF0_SID_MATCH_0 ... A_VINTF0_SID_MATCH_15:
+        i = (offset - A_VINTF0_SID_MATCH_0) / 4;
+        return cmdqv->vintf_sid_match[i];
+    case A_VINTF0_SID_REPLACE_0 ... A_VINTF0_SID_REPLACE_15:
+        i = (offset - A_VINTF0_SID_REPLACE_0) / 4;
+        return cmdqv->vintf_sid_replace[i];
+    case A_VINTF0_LVCMDQ_ERR_MAP_0 ... A_VINTF0_LVCMDQ_ERR_MAP_3:
+        i = (offset - A_VINTF0_LVCMDQ_ERR_MAP_0) / 4;
+        return cmdqv->vintf_cmdq_err_map[i];
+    default:
+        /*
+         * GLB_FILT_CFG_0 (offset 0xC) and GLB_FILT_DATA_0 (offset 0x10) are
+         * filter config and filter data registers. They are not required for
+         * normal VINTF operation and are not emulated.
+         */
+        qemu_log_mask(LOG_UNIMP, "%s unhandled read access at 0x%" PRIx64 "\n",
+                      __func__, offset);
+        return 0;
+    }
+}
+
+static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv,
+                                              hwaddr offset, uint64_t value)
+{
+    int i;
+
+    switch (offset) {
+    case A_VINTF0_CONFIG:
+        /*
+         * Mask out HYP_OWN on guest writes. This bit selects Hypervisor (1) vs
+         * Guest (0) ownership of the CMDQ. Force it to 0 so the VINTF always
+         * remains guest-owned.
+         */
+        value &= ~R_VINTF0_CONFIG_HYP_OWN_MASK;
+
+        cmdqv->vintf_config = value;
+        if (value & R_VINTF0_CONFIG_ENABLE_MASK) {
+            cmdqv->vintf_status |= R_VINTF0_STATUS_ENABLE_OK_MASK;
+        } else {
+            cmdqv->vintf_status &= ~R_VINTF0_STATUS_ENABLE_OK_MASK;
+        }
+        break;
+    case A_VINTF0_SID_MATCH_0 ... A_VINTF0_SID_MATCH_15:
+        i = (offset - A_VINTF0_SID_MATCH_0) / 4;
+        cmdqv->vintf_sid_match[i] = value;
+        break;
+    case A_VINTF0_SID_REPLACE_0 ... A_VINTF0_SID_REPLACE_15:
+        i = (offset - A_VINTF0_SID_REPLACE_0) / 4;
+        cmdqv->vintf_sid_replace[i] = value;
+        break;
+    default:
+        /*
+         * GLB_FILT_CFG_0 (offset 0xC) and GLB_FILT_DATA_0 (offset 0x10) are
+         * filter config and filter data registers. They are not required for
+         * normal VINTF operation and are not emulated.
+         */
+        qemu_log_mask(LOG_UNIMP, "%s unhandled write access at 0x%" PRIx64 "\n",
+                      __func__, offset);
+        return;
+    }
 }
 
-static void tegra241_cmdqv_write_mmio(void *opaque, hwaddr offset, uint64_t value,
-                                 unsigned size)
+static uint64_t tegra241_cmdqv_read_mmio(void *opaque, hwaddr offset,
+                                         unsigned size)
 {
+    Tegra241CMDQV *cmdqv = (Tegra241CMDQV *)opaque;
+    uint64_t val = 0;
+
+    if (offset >= TEGRA241_CMDQV_IO_LEN) {
+        qemu_log_mask(LOG_UNIMP,
+                      "%s offset 0x%" PRIx64 " off limit (0x%x)\n", __func__,
+                      offset, TEGRA241_CMDQV_IO_LEN);
+        goto out;
+    }
+
+    switch (offset) {
+    case A_CONFIG:
+        val = cmdqv->config;
+        break;
+    case A_PARAM:
+        val = cmdqv->param;
+        break;
+    case A_STATUS:
+        val = cmdqv->status;
+        break;
+    case A_VI_ERR_MAP_0 ... A_VI_ERR_MAP_1:
+        val = cmdqv->vi_err_map[(offset - A_VI_ERR_MAP_0) / 4];
+        break;
+    case A_VI_INT_MASK_0 ... A_VI_INT_MASK_1:
+        val = cmdqv->vi_int_mask[(offset - A_VI_INT_MASK_0) / 4];
+        break;
+    case A_CMDQ_ERR_MAP_0 ... A_CMDQ_ERR_MAP_3:
+        val = cmdqv->cmdq_err_map[(offset - A_CMDQ_ERR_MAP_0) / 4];
+        break;
+    case A_CMDQ_ALLOC_MAP_0 ... A_CMDQ_ALLOC_MAP_1:
+        val = cmdqv->cmdq_alloc_map[(offset - A_CMDQ_ALLOC_MAP_0) / 4];
+        break;
+    case A_VINTF0_CONFIG ... A_VINTF0_LVCMDQ_ERR_MAP_3:
+        val = tegra241_cmdqv_config_vintf_read(cmdqv, offset);
+        break;
+    default:
+        qemu_log_mask(LOG_UNIMP, "%s unhandled read access at 0x%" PRIx64 "\n",
+                      __func__, offset);
+    }
+
+out:
+    trace_tegra241_cmdqv_read_mmio(offset, val, size);
+    return val;
+}
+
+/* 4-byte MMIO write handler. */
+static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,
+                                       uint32_t value)
+{
+    switch (offset) {
+    case A_CONFIG:
+        cmdqv->config = value;
+        if (value & R_CONFIG_CMDQV_EN_MASK) {
+            cmdqv->status |= R_STATUS_CMDQV_ENABLED_MASK;
+        } else {
+            cmdqv->status &= ~R_STATUS_CMDQV_ENABLED_MASK;
+        }
+        break;
+    case A_VI_INT_MASK_0 ... A_VI_INT_MASK_1:
+        cmdqv->vi_int_mask[(offset - A_VI_INT_MASK_0) / 4] = value;
+        break;
+    case A_CMDQ_ALLOC_MAP_0 ... A_CMDQ_ALLOC_MAP_1:
+        cmdqv->cmdq_alloc_map[(offset - A_CMDQ_ALLOC_MAP_0) / 4] = value;
+        break;
+    case A_VINTF0_CONFIG ... A_VINTF0_LVCMDQ_ERR_MAP_3:
+        tegra241_cmdqv_config_vintf_write(cmdqv, offset, value);
+        break;
+    default:
+        qemu_log_mask(LOG_UNIMP, "%s unhandled write access at 0x%" PRIx64 "\n",
+                      __func__, offset);
+    }
+}
+
+/*
+ * 8-byte MMIO write handler.
+ */
+static void tegra241_cmdqv_writell_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,
+                                        uint64_t value)
+{
+    qemu_log_mask(LOG_UNIMP,
+                      "%s unhandled 64-bit write at 0x%" PRIx64 " (WI)\n",
+                      __func__, offset);
+}
+
+static void tegra241_cmdqv_write_mmio(void *opaque, hwaddr offset,
+                                      uint64_t value, unsigned size)
+{
+    Tegra241CMDQV *cmdqv = (Tegra241CMDQV *)opaque;
+
+    if (offset >= TEGRA241_CMDQV_IO_LEN) {
+        qemu_log_mask(LOG_UNIMP,
+                      "%s offset 0x%" PRIx64 " off limit (0x%x)\n", __func__,
+                      offset, TEGRA241_CMDQV_IO_LEN);
+        goto out;
+    }
+
+    switch (size) {
+    case 4:
+        tegra241_cmdqv_writel_mmio(cmdqv, offset, value);
+        break;
+    case 8:
+        tegra241_cmdqv_writell_mmio(cmdqv, offset, value);
+        break;
+    default:
+        qemu_log_mask(LOG_GUEST_ERROR,
+                      "%s bad write size %u at 0x%" PRIx64 "\n",
+                      __func__, size, offset);
+    }
+
+out:
+    trace_tegra241_cmdqv_write_mmio(offset, value, size);
 }
 
 static void tegra241_cmdqv_free_viommu(SMMUv3State *s)
hw/arm/trace-events +0 -0
index 3457536fb0..8c61d66a26 100644

hw/arm/trace-events smmuv3_accel_unset_iommu_device(int devfn, uint32_t devid) "devfn=0x%x (idev dev · line 81

Quoted 10 lines
hw/arm/trace-events +4 -0
@@ -72,6 +72,10 @@ smmuv3_accel_unset_iommu_device(int devfn, uint32_t devid) "devfn=0x%x (idev dev
 smmuv3_accel_translate_ste(uint32_t vsid, uint32_t hwpt_id, uint64_t ste_1, uint64_t ste_0) "vSID=0x%x hwpt_id=0x%x ste=%"PRIx64":%"PRIx64
 smmuv3_accel_install_ste(uint32_t vsid, const char * type, uint32_t hwpt_id) "vSID=0x%x ste type=%s hwpt_id=0x%x"
 
+# tegra241-cmdqv
+tegra241_cmdqv_read_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x"
+tegra241_cmdqv_write_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x"
+
 # strongarm.c
 strongarm_uart_update_parameters(const char *label, int speed, char parity, int data_bits, int stop_bits) "%s speed=%d parity=%c data=%d stop=%d"
 strongarm_ssp_read_underrun(void) "SSP rx underrun"

Eric

#21 lore raw

PATCH v5 15/32 hw/arm/tegra241-cmdqv: Emulate VCMDQ register reads

In reply to Shameer Kolothum

From: Nicolin Chen <[email protected]>

Tegra241 CMDQV exposes per-VCMDQ register windows through two MMIO apertures:

Direct VCMDQ aperture (0x10000/0x20000): VCMDQ Page0/Page1 VINTF logical aperture (0x30000/0x40000): VINTF0 LVCMDQ Page0/Page1

Both apertures are hardware aliases of the same underlying registers:

Page 0 (control/status): CONS_INDX, PROD_INDX, CONFIG, STATUS,
GERROR, GERRORN
Page 1 (base/DRAM): BASE_L/H, CONS_INDX_BASE_DRAM_L/H

The direct aperture Page 0 is programmable at any time so long as CMDQV_EN is enabled. The VINTF (logical) aperture Page 0 is programmable only once SW has mapped a VCMDQ to a VINTF; the "logical" view is local to that VINTF.

Add read emulation for both apertures, backed by a single per-VCMDQ register cache. VINTF aperture reads are translated to their equivalent direct-aperture offset and served from the same cached state.

Once IOMMU_HW_QUEUE_ALLOC and viommu_mmap are wired up in a subsequent patch, Page 0 reads will be served directly from the hardware-backed mmap'd page instead of the cache. Page 1 is also a hardware alias, but the kernel only exposes mmap for Page 0, so Page 1 reads always trap to QEMU and are served from cache.

---
 hw/arm/tegra241-cmdqv.h | 215 ++++++++++++++++++++++++++++++++++++++++
 hw/arm/tegra241-cmdqv.c |  95 ++++++++++++++++++
 hw/arm/trace-events     |   2 +
 3 files changed, 312 insertions(+)
hw/arm/tegra241-cmdqv.h +215 -0
index ace327443a..6b21a407af 100644
@@ -30,6 +30,13 @@
  */
 #define TEGRA241_CMDQV_IO_LEN 0x50000
 
+/* CMDQV MMIO aperture bases and VCMDQ stride */
+#define CMDQV_VCMDQ_PAGE0_BASE  0x10000  /* CMDQV_CMDQ_BASE */
+#define CMDQV_VCMDQ_PAGE1_BASE  0x20000
+#define CMDQV_VINTF_PAGE0_BASE  0x30000  /* CMDQV_VI_CMDQ_BASE */
+#define CMDQV_VINTF_PAGE1_BASE  0x40000
+#define CMDQV_VCMDQ_STRIDE      0x80
+
 typedef struct Tegra241CMDQV {
     struct iommu_viommu_tegra241_cmdqv cmdqv_data;
     SMMUv3AccelState *s_accel;
@@ -52,6 +59,19 @@ typedef struct Tegra241CMDQV {
     uint32_t vintf_sid_match[TEGRA241_CMDQV_MAX_NUM_SID];
     uint32_t vintf_sid_replace[TEGRA241_CMDQV_MAX_NUM_SID];
     uint32_t vintf_cmdq_err_map[4];
+    /*
+     * VCMDQ register cache. The direct (VCMDQ aperture) and logical
+     * (VINTF aperture) views are hardware aliases; both are served from
+     * this single cached copy.
+     */
+    uint32_t vcmdq_cons_indx[TEGRA241_CMDQV_MAX_CMDQ];
+    uint32_t vcmdq_prod_indx[TEGRA241_CMDQV_MAX_CMDQ];
+    uint32_t vcmdq_config[TEGRA241_CMDQV_MAX_CMDQ];
+    uint32_t vcmdq_status[TEGRA241_CMDQV_MAX_CMDQ];
+    uint32_t vcmdq_gerror[TEGRA241_CMDQV_MAX_CMDQ];
+    uint32_t vcmdq_gerrorn[TEGRA241_CMDQV_MAX_CMDQ];
+    uint64_t vcmdq_base[TEGRA241_CMDQV_MAX_CMDQ];
+    uint64_t vcmdq_cons_indx_base[TEGRA241_CMDQV_MAX_CMDQ];
 } Tegra241CMDQV;
 
 /* CMDQ-V Config page registers (offset 0x00000) */
@@ -143,6 +163,201 @@ REG32(VINTF0_LVCMDQ_ERR_MAP_0, 0x10c0)
 FIELD(VINTF0_LVCMDQ_ERR_MAP_0, LVCMDQ_ERR_MAP, 0, 32)
 #define A_VINTF0_LVCMDQ_ERR_MAP_3 (A_VINTF0_LVCMDQ_ERR_MAP_0 + 3 * 4)
 
+/*
+ * VCMDQ Page 0 register window @ CMDQV_VCMDQ_PAGE0_BASE.
+ * Direct VCMDQ aperture; control and status registers.
+ */
+#define SMMU_CMDQV_VCMDQi_CONS_INDX_(i)                     \
+    REG32(VCMDQ##i##_CONS_INDX,                             \
+          CMDQV_VCMDQ_PAGE0_BASE + i * CMDQV_VCMDQ_STRIDE)  \
+    FIELD(VCMDQ##i##_CONS_INDX, RD, 0, 20)                  \
+    FIELD(VCMDQ##i##_CONS_INDX, ERR, 24, 7)
+
+SMMU_CMDQV_VCMDQi_CONS_INDX_(0)
+SMMU_CMDQV_VCMDQi_CONS_INDX_(1)
+
+#define V_VCMDQ_CONS_INDX_ERR_CERROR_NONE         0
+#define V_VCMDQ_CONS_INDX_ERR_CERROR_ILL_OPCODE   1
+#define V_VCMDQ_CONS_INDX_ERR_CERROR_ABT          2
+#define V_VCMDQ_CONS_INDX_ERR_CERROR_ATC_INV_SYNC 3
+#define V_VCMDQ_CONS_INDX_ERR_CERROR_ILL_ACCESS   4
+
+#define SMMU_CMDQV_VCMDQi_PROD_INDX_(i)                          \
+    REG32(VCMDQ##i##_PROD_INDX,                                  \
+          CMDQV_VCMDQ_PAGE0_BASE + 0x4 + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VCMDQ##i##_PROD_INDX, WR, 0, 20)
+
+SMMU_CMDQV_VCMDQi_PROD_INDX_(0)
+SMMU_CMDQV_VCMDQi_PROD_INDX_(1)
+
+#define SMMU_CMDQV_VCMDQi_CONFIG_(i)                             \
+    REG32(VCMDQ##i##_CONFIG,                                     \
+          CMDQV_VCMDQ_PAGE0_BASE + 0x8 + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VCMDQ##i##_CONFIG, CMDQ_EN, 0, 1)
+
+SMMU_CMDQV_VCMDQi_CONFIG_(0)
+SMMU_CMDQV_VCMDQi_CONFIG_(1)
+
+#define SMMU_CMDQV_VCMDQi_STATUS_(i)                             \
+    REG32(VCMDQ##i##_STATUS,                                     \
+          CMDQV_VCMDQ_PAGE0_BASE + 0xc + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VCMDQ##i##_STATUS, CMDQ_EN_OK, 0, 1)
+
+SMMU_CMDQV_VCMDQi_STATUS_(0)
+SMMU_CMDQV_VCMDQi_STATUS_(1)
+
+#define SMMU_CMDQV_VCMDQi_GERROR_(i)                              \
+    REG32(VCMDQ##i##_GERROR,                                      \
+          CMDQV_VCMDQ_PAGE0_BASE + 0x10 + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VCMDQ##i##_GERROR, CMDQ_ERR, 0, 1)                      \
+    FIELD(VCMDQ##i##_GERROR, CONS_DRAM_WR_ABT_ERR, 1, 1)          \
+    FIELD(VCMDQ##i##_GERROR, CMDQ_INIT_ERR, 2, 1)
+
+SMMU_CMDQV_VCMDQi_GERROR_(0)
+SMMU_CMDQV_VCMDQi_GERROR_(1)
+
+#define SMMU_CMDQV_VCMDQi_GERRORN_(i)                             \
+    REG32(VCMDQ##i##_GERRORN,                                     \
+          CMDQV_VCMDQ_PAGE0_BASE + 0x14 + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VCMDQ##i##_GERRORN, CMDQ_ERR, 0, 1)                     \
+    FIELD(VCMDQ##i##_GERRORN, CONS_DRAM_WR_ABT_ERR, 1, 1)         \
+    FIELD(VCMDQ##i##_GERRORN, CMDQ_INIT_ERR, 2, 1)
+
+SMMU_CMDQV_VCMDQi_GERRORN_(0)
+SMMU_CMDQV_VCMDQi_GERRORN_(1)
+
+/*
+ * VCMDQ Page 1 register window @ CMDQV_VCMDQ_PAGE1_BASE.
+ * Direct VCMDQ aperture; base and DRAM address registers.
+ */
+#define SMMU_CMDQV_VCMDQi_BASE_L_(i)                                          \
+    REG32(VCMDQ##i##_BASE_L, CMDQV_VCMDQ_PAGE1_BASE + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VCMDQ##i##_BASE_L, LOG2SIZE, 0, 5)                                  \
+    FIELD(VCMDQ##i##_BASE_L, ADDR, 5, 27)
+
+SMMU_CMDQV_VCMDQi_BASE_L_(0)
+SMMU_CMDQV_VCMDQi_BASE_L_(1)
+
+#define SMMU_CMDQV_VCMDQi_BASE_H_(i)                             \
+    REG32(VCMDQ##i##_BASE_H,                                     \
+          CMDQV_VCMDQ_PAGE1_BASE + 0x4 + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VCMDQ##i##_BASE_H, ADDR, 0, 16)
+
+SMMU_CMDQV_VCMDQi_BASE_H_(0)
+SMMU_CMDQV_VCMDQi_BASE_H_(1)
+
+#define SMMU_CMDQV_VCMDQi_CONS_INDX_BASE_DRAM_L_(i)              \
+    REG32(VCMDQ##i##_CONS_INDX_BASE_DRAM_L,                      \
+          CMDQV_VCMDQ_PAGE1_BASE + 0x8 + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VCMDQ##i##_CONS_INDX_BASE_DRAM_L, ADDR, 0, 32)
+
+SMMU_CMDQV_VCMDQi_CONS_INDX_BASE_DRAM_L_(0)
+SMMU_CMDQV_VCMDQi_CONS_INDX_BASE_DRAM_L_(1)
+
+#define SMMU_CMDQV_VCMDQi_CONS_INDX_BASE_DRAM_H_(i)              \
+    REG32(VCMDQ##i##_CONS_INDX_BASE_DRAM_H,                      \
+          CMDQV_VCMDQ_PAGE1_BASE + 0xc + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VCMDQ##i##_CONS_INDX_BASE_DRAM_H, ADDR, 0, 16)
+
+SMMU_CMDQV_VCMDQi_CONS_INDX_BASE_DRAM_H_(0)
+SMMU_CMDQV_VCMDQi_CONS_INDX_BASE_DRAM_H_(1)
+
+/*
+ * VINTF0 LVCMDQ Page 0 register window @ CMDQV_VINTF_PAGE0_BASE.
+ * Logical VCMDQ aperture (VCMDQs mapped via VINTF); control and
+ * status registers. Hardware alias of VCMDQ Page 0 above.
+ */
+#define SMMU_CMDQV_VI_VCMDQi_CONS_INDX_(i)                  \
+    REG32(VI_VCMDQ##i##_CONS_INDX,                          \
+          CMDQV_VINTF_PAGE0_BASE + i * CMDQV_VCMDQ_STRIDE)  \
+    FIELD(VI_VCMDQ##i##_CONS_INDX, RD, 0, 20)               \
+    FIELD(VI_VCMDQ##i##_CONS_INDX, ERR, 24, 7)
+
+SMMU_CMDQV_VI_VCMDQi_CONS_INDX_(0)
+SMMU_CMDQV_VI_VCMDQi_CONS_INDX_(1)
+
+#define SMMU_CMDQV_VI_VCMDQi_PROD_INDX_(i)                       \
+    REG32(VI_VCMDQ##i##_PROD_INDX,                               \
+          CMDQV_VINTF_PAGE0_BASE + 0x4 + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VI_VCMDQ##i##_PROD_INDX, WR, 0, 20)
+
+SMMU_CMDQV_VI_VCMDQi_PROD_INDX_(0)
+SMMU_CMDQV_VI_VCMDQi_PROD_INDX_(1)
+
+#define SMMU_CMDQV_VI_VCMDQi_CONFIG_(i)                          \
+    REG32(VI_VCMDQ##i##_CONFIG,                                  \
+          CMDQV_VINTF_PAGE0_BASE + 0x8 + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VI_VCMDQ##i##_CONFIG, CMDQ_EN, 0, 1)
+
+SMMU_CMDQV_VI_VCMDQi_CONFIG_(0)
+SMMU_CMDQV_VI_VCMDQi_CONFIG_(1)
+
+#define SMMU_CMDQV_VI_VCMDQi_STATUS_(i)                          \
+    REG32(VI_VCMDQ##i##_STATUS,                                  \
+          CMDQV_VINTF_PAGE0_BASE + 0xc + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VI_VCMDQ##i##_STATUS, CMDQ_EN_OK, 0, 1)
+
+SMMU_CMDQV_VI_VCMDQi_STATUS_(0)
+SMMU_CMDQV_VI_VCMDQi_STATUS_(1)
+
+#define SMMU_CMDQV_VI_VCMDQi_GERROR_(i)                           \
+    REG32(VI_VCMDQ##i##_GERROR,                                   \
+          CMDQV_VINTF_PAGE0_BASE + 0x10 + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VI_VCMDQ##i##_GERROR, CMDQ_ERR, 0, 1)                   \
+    FIELD(VI_VCMDQ##i##_GERROR, CONS_DRAM_WR_ABT_ERR, 1, 1)       \
+    FIELD(VI_VCMDQ##i##_GERROR, CMDQ_INIT_ERR, 2, 1)
+
+SMMU_CMDQV_VI_VCMDQi_GERROR_(0)
+SMMU_CMDQV_VI_VCMDQi_GERROR_(1)
+
+#define SMMU_CMDQV_VI_VCMDQi_GERRORN_(i)                          \
+    REG32(VI_VCMDQ##i##_GERRORN,                                  \
+          CMDQV_VINTF_PAGE0_BASE + 0x14 + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VI_VCMDQ##i##_GERRORN, CMDQ_ERR, 0, 1)                  \
+    FIELD(VI_VCMDQ##i##_GERRORN, CONS_DRAM_WR_ABT_ERR, 1, 1)      \
+    FIELD(VI_VCMDQ##i##_GERRORN, CMDQ_INIT_ERR, 2, 1)
+
+SMMU_CMDQV_VI_VCMDQi_GERRORN_(0)
+SMMU_CMDQV_VI_VCMDQi_GERRORN_(1)
+
+/*
+ * VINTF0 LVCMDQ Page 1 register window @ CMDQV_VINTF_PAGE1_BASE.
+ * Logical VCMDQ aperture (VCMDQs mapped via VINTF); base and DRAM
+ * address registers. Hardware alias of VCMDQ Page 1 above.
+ */
+#define SMMU_CMDQV_VI_VCMDQi_BASE_L_(i)                     \
+    REG32(VI_VCMDQ##i##_BASE_L,                             \
+          CMDQV_VINTF_PAGE1_BASE + i * CMDQV_VCMDQ_STRIDE)  \
+    FIELD(VI_VCMDQ##i##_BASE_L, LOG2SIZE, 0, 5)             \
+    FIELD(VI_VCMDQ##i##_BASE_L, ADDR, 5, 27)
+
+SMMU_CMDQV_VI_VCMDQi_BASE_L_(0)
+SMMU_CMDQV_VI_VCMDQi_BASE_L_(1)
+
+#define SMMU_CMDQV_VI_VCMDQi_BASE_H_(i)                          \
+    REG32(VI_VCMDQ##i##_BASE_H,                                  \
+          CMDQV_VINTF_PAGE1_BASE + 0x4 + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VI_VCMDQ##i##_BASE_H, ADDR, 0, 16)
+
+SMMU_CMDQV_VI_VCMDQi_BASE_H_(0)
+SMMU_CMDQV_VI_VCMDQi_BASE_H_(1)
+
+#define SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_L_(i)           \
+    REG32(VI_VCMDQ##i##_CONS_INDX_BASE_DRAM_L,                   \
+          CMDQV_VINTF_PAGE1_BASE + 0x8 + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VI_VCMDQ##i##_CONS_INDX_BASE_DRAM_L, ADDR, 0, 32)
+
+SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_L_(0)
+SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_L_(1)
+
+#define SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_(i)           \
+    REG32(VI_VCMDQ##i##_CONS_INDX_BASE_DRAM_H,                   \
+          CMDQV_VINTF_PAGE1_BASE + 0xc + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VI_VCMDQ##i##_CONS_INDX_BASE_DRAM_H, ADDR, 0, 16)
+
+SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_(0)
+SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_(1)
+
 const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops(void);
 
 #endif /* HW_ARM_TEGRA241_CMDQV_H */
hw/arm/tegra241-cmdqv.c +95 -0

113 lines not on this page · raw message

hw/arm/trace-events +2 -0
index 8c61d66a26..8c34a04b24 100644
@@ -75,6 +75,8 @@ smmuv3_accel_install_ste(uint32_t vsid, const char * type, uint32_t hwpt_id) "vS
 # tegra241-cmdqv
 tegra241_cmdqv_read_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x"
 tegra241_cmdqv_write_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x"
+tegra241_cmdqv_read_vcmdq_page0(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64
+tegra241_cmdqv_read_vcmdq_page1(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64
 
 # strongarm.c
 strongarm_uart_update_parameters(const char *label, int speed, char parity, int data_bits, int stop_bits) "%s speed=%d parity=%c data=%d stop=%d"
Signature · 1 line
2.43.0
Reviewed-by on v5 15/32
#22 lore raw

Re: [PATCH v5 15/32] hw/arm/tegra241-cmdqv: Emulate VCMDQ register reads

Quoted 133 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

Tegra241 CMDQV exposes per-VCMDQ register windows through two MMIO apertures:

Direct VCMDQ aperture (0x10000/0x20000): VCMDQ Page0/Page1 VINTF logical aperture (0x30000/0x40000): VINTF0 LVCMDQ Page0/Page1

Both apertures are hardware aliases of the same underlying registers:

Page 0 (control/status): CONS_INDX, PROD_INDX, CONFIG, STATUS,
GERROR, GERRORN
Page 1 (base/DRAM): BASE_L/H, CONS_INDX_BASE_DRAM_L/H

The direct aperture Page 0 is programmable at any time so long as CMDQV_EN is enabled. The VINTF (logical) aperture Page 0 is programmable only once SW has mapped a VCMDQ to a VINTF; the "logical" view is local to that VINTF.

Add read emulation for both apertures, backed by a single per-VCMDQ register cache. VINTF aperture reads are translated to their equivalent direct-aperture offset and served from the same cached state.

Once IOMMU_HW_QUEUE_ALLOC and viommu_mmap are wired up in a subsequent patch, Page 0 reads will be served directly from the hardware-backed mmap'd page instead of the cache. Page 1 is also a hardware alias, but the kernel only exposes mmap for Page 0, so Page 1 reads always trap to QEMU and are served from cache.

---
 hw/arm/tegra241-cmdqv.h | 215 ++++++++++++++++++++++++++++++++++++++++
 hw/arm/tegra241-cmdqv.c |  95 ++++++++++++++++++
 hw/arm/trace-events     |   2 +
 3 files changed, 312 insertions(+)
hw/arm/tegra241-cmdqv.h +72 -0
index ace327443a..6b21a407af 100644
@@ -30,6 +30,13 @@
  */
 #define TEGRA241_CMDQV_IO_LEN 0x50000
 
+/* CMDQV MMIO aperture bases and VCMDQ stride */
+#define CMDQV_VCMDQ_PAGE0_BASE  0x10000  /* CMDQV_CMDQ_BASE */
+#define CMDQV_VCMDQ_PAGE1_BASE  0x20000
+#define CMDQV_VINTF_PAGE0_BASE  0x30000  /* CMDQV_VI_CMDQ_BASE */
+#define CMDQV_VINTF_PAGE1_BASE  0x40000
+#define CMDQV_VCMDQ_STRIDE      0x80
+
 typedef struct Tegra241CMDQV {
     struct iommu_viommu_tegra241_cmdqv cmdqv_data;
     SMMUv3AccelState *s_accel;
@@ -52,6 +59,19 @@ typedef struct Tegra241CMDQV {
     uint32_t vintf_sid_match[TEGRA241_CMDQV_MAX_NUM_SID];
     uint32_t vintf_sid_replace[TEGRA241_CMDQV_MAX_NUM_SID];
     uint32_t vintf_cmdq_err_map[4];
+    /*
+     * VCMDQ register cache. The direct (VCMDQ aperture) and logical
+     * (VINTF aperture) views are hardware aliases; both are served from
+     * this single cached copy.
+     */
+    uint32_t vcmdq_cons_indx[TEGRA241_CMDQV_MAX_CMDQ];
+    uint32_t vcmdq_prod_indx[TEGRA241_CMDQV_MAX_CMDQ];
+    uint32_t vcmdq_config[TEGRA241_CMDQV_MAX_CMDQ];
+    uint32_t vcmdq_status[TEGRA241_CMDQV_MAX_CMDQ];
+    uint32_t vcmdq_gerror[TEGRA241_CMDQV_MAX_CMDQ];
+    uint32_t vcmdq_gerrorn[TEGRA241_CMDQV_MAX_CMDQ];
+    uint64_t vcmdq_base[TEGRA241_CMDQV_MAX_CMDQ];
+    uint64_t vcmdq_cons_indx_base[TEGRA241_CMDQV_MAX_CMDQ];
 } Tegra241CMDQV;
 
 /* CMDQ-V Config page registers (offset 0x00000) */
@@ -143,6 +163,201 @@ REG32(VINTF0_LVCMDQ_ERR_MAP_0, 0x10c0)
 FIELD(VINTF0_LVCMDQ_ERR_MAP_0, LVCMDQ_ERR_MAP, 0, 32)
 #define A_VINTF0_LVCMDQ_ERR_MAP_3 (A_VINTF0_LVCMDQ_ERR_MAP_0 + 3 * 4)
 
+/*
+ * VCMDQ Page 0 register window @ CMDQV_VCMDQ_PAGE0_BASE.
+ * Direct VCMDQ aperture; control and status registers.
+ */
+#define SMMU_CMDQV_VCMDQi_CONS_INDX_(i)                     \
+    REG32(VCMDQ##i##_CONS_INDX,                             \
+          CMDQV_VCMDQ_PAGE0_BASE + i * CMDQV_VCMDQ_STRIDE)  \
+    FIELD(VCMDQ##i##_CONS_INDX, RD, 0, 20)                  \
+    FIELD(VCMDQ##i##_CONS_INDX, ERR, 24, 7)
+
+SMMU_CMDQV_VCMDQi_CONS_INDX_(0)
+SMMU_CMDQV_VCMDQi_CONS_INDX_(1)
+
+#define V_VCMDQ_CONS_INDX_ERR_CERROR_NONE         0
+#define V_VCMDQ_CONS_INDX_ERR_CERROR_ILL_OPCODE   1
+#define V_VCMDQ_CONS_INDX_ERR_CERROR_ABT          2
+#define V_VCMDQ_CONS_INDX_ERR_CERROR_ATC_INV_SYNC 3
+#define V_VCMDQ_CONS_INDX_ERR_CERROR_ILL_ACCESS   4
+
+#define SMMU_CMDQV_VCMDQi_PROD_INDX_(i)                          \
+    REG32(VCMDQ##i##_PROD_INDX,                                  \
+          CMDQV_VCMDQ_PAGE0_BASE + 0x4 + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VCMDQ##i##_PROD_INDX, WR, 0, 20)
+
+SMMU_CMDQV_VCMDQi_PROD_INDX_(0)
+SMMU_CMDQV_VCMDQi_PROD_INDX_(1)
+
+#define SMMU_CMDQV_VCMDQi_CONFIG_(i)                             \
+    REG32(VCMDQ##i##_CONFIG,                                     \
+          CMDQV_VCMDQ_PAGE0_BASE + 0x8 + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VCMDQ##i##_CONFIG, CMDQ_EN, 0, 1)
+
+SMMU_CMDQV_VCMDQi_CONFIG_(0)
+SMMU_CMDQV_VCMDQi_CONFIG_(1)
+
+#define SMMU_CMDQV_VCMDQi_STATUS_(i)                             \
+    REG32(VCMDQ##i##_STATUS,                                     \
+          CMDQV_VCMDQ_PAGE0_BASE + 0xc + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VCMDQ##i##_STATUS, CMDQ_EN_OK, 0, 1)
+
+SMMU_CMDQV_VCMDQi_STATUS_(0)
+SMMU_CMDQV_VCMDQi_STATUS_(1)
+
+#define SMMU_CMDQV_VCMDQi_GERROR_(i)                              \
+    REG32(VCMDQ##i##_GERROR,                                      \
+          CMDQV_VCMDQ_PAGE0_BASE + 0x10 + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VCMDQ##i##_GERROR, CMDQ_ERR, 0, 1)                      \
+    FIELD(VCMDQ##i##_GERROR, CONS_DRAM_WR_ABT_ERR, 1, 1)          \
+    FIELD(VCMDQ##i##_GERROR, CMDQ_INIT_ERR, 2, 1)
+
+SMMU_CMDQV_VCMDQi_GERROR_(0)
+SMMU_CMDQV_VCMDQi_GERROR_(1)

hw/arm/tegra241-cmdqv.h REG32(VINTF0_LVCMDQ_ERR_MAP_0, 0x10c0) · line 227

Quoted 10 lines
hw/arm/tegra241-cmdqv.h +10 -0
+
+#define SMMU_CMDQV_VCMDQi_GERRORN_(i)                             \
+    REG32(VCMDQ##i##_GERRORN,                                     \
+          CMDQV_VCMDQ_PAGE0_BASE + 0x14 + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VCMDQ##i##_GERRORN, CMDQ_ERR, 0, 1)                     \
+    FIELD(VCMDQ##i##_GERRORN, CONS_DRAM_WR_ABT_ERR, 1, 1)         \
+    FIELD(VCMDQ##i##_GERRORN, CMDQ_INIT_ERR, 2, 1)
+
+SMMU_CMDQV_VCMDQi_GERRORN_(0)
+SMMU_CMDQV_VCMDQi_GERRORN_(1)

nit, what I meant in my v4 comments is I would prefer we group the eventual register definition to represent the actual layout of each page, instead of interleaving them with actual macros:
This would look like:

/* page 0 */

/* vcmdq0 */

+SMMU_CMDQV_VCMDQi_CONS_INDX_(0)
+SMMU_CMDQV_VCMDQi_PROD_INDX_(0)
+SMMU_CMDQV_VCMDQi_CONFIG_(0)
+SMMU_CMDQV_VCMDQi_STATUS_(0)
+SMMU_CMDQV_VCMDQi_GERROR_(0)
+SMMU_CMDQV_VCMDQi_GERRORN_(0)

/* vcmdq1 */

+SMMU_CMDQV_VCMDQi_CONS_INDX_(1)
+SMMU_CMDQV_VCMDQi_PROD_INDX_(1)
+SMMU_CMDQV_VCMDQi_CONFIG_(1)
+SMMU_CMDQV_VCMDQi_STATUS_(1)
+SMMU_CMDQV_VCMDQi_GERROR_(1)
+SMMU_CMDQV_VCMDQi_GERRORN_(1)

same for page 1

To me it looks easier to remember the layout without looking at the spec.

Taste & colors, what do you prefer?

Quoted 261 lines
hw/arm/tegra241-cmdqv.h +133 -0
+
+/*
+ * VCMDQ Page 1 register window @ CMDQV_VCMDQ_PAGE1_BASE.
+ * Direct VCMDQ aperture; base and DRAM address registers.
+ */
+#define SMMU_CMDQV_VCMDQi_BASE_L_(i)                                          \
+    REG32(VCMDQ##i##_BASE_L, CMDQV_VCMDQ_PAGE1_BASE + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VCMDQ##i##_BASE_L, LOG2SIZE, 0, 5)                                  \
+    FIELD(VCMDQ##i##_BASE_L, ADDR, 5, 27)
+
+SMMU_CMDQV_VCMDQi_BASE_L_(0)
+SMMU_CMDQV_VCMDQi_BASE_L_(1)
+
+#define SMMU_CMDQV_VCMDQi_BASE_H_(i)                             \
+    REG32(VCMDQ##i##_BASE_H,                                     \
+          CMDQV_VCMDQ_PAGE1_BASE + 0x4 + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VCMDQ##i##_BASE_H, ADDR, 0, 16)
+
+SMMU_CMDQV_VCMDQi_BASE_H_(0)
+SMMU_CMDQV_VCMDQi_BASE_H_(1)
+
+#define SMMU_CMDQV_VCMDQi_CONS_INDX_BASE_DRAM_L_(i)              \
+    REG32(VCMDQ##i##_CONS_INDX_BASE_DRAM_L,                      \
+          CMDQV_VCMDQ_PAGE1_BASE + 0x8 + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VCMDQ##i##_CONS_INDX_BASE_DRAM_L, ADDR, 0, 32)
+
+SMMU_CMDQV_VCMDQi_CONS_INDX_BASE_DRAM_L_(0)
+SMMU_CMDQV_VCMDQi_CONS_INDX_BASE_DRAM_L_(1)
+
+#define SMMU_CMDQV_VCMDQi_CONS_INDX_BASE_DRAM_H_(i)              \
+    REG32(VCMDQ##i##_CONS_INDX_BASE_DRAM_H,                      \
+          CMDQV_VCMDQ_PAGE1_BASE + 0xc + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VCMDQ##i##_CONS_INDX_BASE_DRAM_H, ADDR, 0, 16)
+
+SMMU_CMDQV_VCMDQi_CONS_INDX_BASE_DRAM_H_(0)
+SMMU_CMDQV_VCMDQi_CONS_INDX_BASE_DRAM_H_(1)
+
+/*
+ * VINTF0 LVCMDQ Page 0 register window @ CMDQV_VINTF_PAGE0_BASE.
+ * Logical VCMDQ aperture (VCMDQs mapped via VINTF); control and
+ * status registers. Hardware alias of VCMDQ Page 0 above.
+ */
+#define SMMU_CMDQV_VI_VCMDQi_CONS_INDX_(i)                  \
+    REG32(VI_VCMDQ##i##_CONS_INDX,                          \
+          CMDQV_VINTF_PAGE0_BASE + i * CMDQV_VCMDQ_STRIDE)  \
+    FIELD(VI_VCMDQ##i##_CONS_INDX, RD, 0, 20)               \
+    FIELD(VI_VCMDQ##i##_CONS_INDX, ERR, 24, 7)
+
+SMMU_CMDQV_VI_VCMDQi_CONS_INDX_(0)
+SMMU_CMDQV_VI_VCMDQi_CONS_INDX_(1)
+
+#define SMMU_CMDQV_VI_VCMDQi_PROD_INDX_(i)                       \
+    REG32(VI_VCMDQ##i##_PROD_INDX,                               \
+          CMDQV_VINTF_PAGE0_BASE + 0x4 + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VI_VCMDQ##i##_PROD_INDX, WR, 0, 20)
+
+SMMU_CMDQV_VI_VCMDQi_PROD_INDX_(0)
+SMMU_CMDQV_VI_VCMDQi_PROD_INDX_(1)
+
+#define SMMU_CMDQV_VI_VCMDQi_CONFIG_(i)                          \
+    REG32(VI_VCMDQ##i##_CONFIG,                                  \
+          CMDQV_VINTF_PAGE0_BASE + 0x8 + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VI_VCMDQ##i##_CONFIG, CMDQ_EN, 0, 1)
+
+SMMU_CMDQV_VI_VCMDQi_CONFIG_(0)
+SMMU_CMDQV_VI_VCMDQi_CONFIG_(1)
+
+#define SMMU_CMDQV_VI_VCMDQi_STATUS_(i)                          \
+    REG32(VI_VCMDQ##i##_STATUS,                                  \
+          CMDQV_VINTF_PAGE0_BASE + 0xc + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VI_VCMDQ##i##_STATUS, CMDQ_EN_OK, 0, 1)
+
+SMMU_CMDQV_VI_VCMDQi_STATUS_(0)
+SMMU_CMDQV_VI_VCMDQi_STATUS_(1)
+
+#define SMMU_CMDQV_VI_VCMDQi_GERROR_(i)                           \
+    REG32(VI_VCMDQ##i##_GERROR,                                   \
+          CMDQV_VINTF_PAGE0_BASE + 0x10 + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VI_VCMDQ##i##_GERROR, CMDQ_ERR, 0, 1)                   \
+    FIELD(VI_VCMDQ##i##_GERROR, CONS_DRAM_WR_ABT_ERR, 1, 1)       \
+    FIELD(VI_VCMDQ##i##_GERROR, CMDQ_INIT_ERR, 2, 1)
+
+SMMU_CMDQV_VI_VCMDQi_GERROR_(0)
+SMMU_CMDQV_VI_VCMDQi_GERROR_(1)
+
+#define SMMU_CMDQV_VI_VCMDQi_GERRORN_(i)                          \
+    REG32(VI_VCMDQ##i##_GERRORN,                                  \
+          CMDQV_VINTF_PAGE0_BASE + 0x14 + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VI_VCMDQ##i##_GERRORN, CMDQ_ERR, 0, 1)                  \
+    FIELD(VI_VCMDQ##i##_GERRORN, CONS_DRAM_WR_ABT_ERR, 1, 1)      \
+    FIELD(VI_VCMDQ##i##_GERRORN, CMDQ_INIT_ERR, 2, 1)
+
+SMMU_CMDQV_VI_VCMDQi_GERRORN_(0)
+SMMU_CMDQV_VI_VCMDQi_GERRORN_(1)
+
+/*
+ * VINTF0 LVCMDQ Page 1 register window @ CMDQV_VINTF_PAGE1_BASE.
+ * Logical VCMDQ aperture (VCMDQs mapped via VINTF); base and DRAM
+ * address registers. Hardware alias of VCMDQ Page 1 above.
+ */
+#define SMMU_CMDQV_VI_VCMDQi_BASE_L_(i)                     \
+    REG32(VI_VCMDQ##i##_BASE_L,                             \
+          CMDQV_VINTF_PAGE1_BASE + i * CMDQV_VCMDQ_STRIDE)  \
+    FIELD(VI_VCMDQ##i##_BASE_L, LOG2SIZE, 0, 5)             \
+    FIELD(VI_VCMDQ##i##_BASE_L, ADDR, 5, 27)
+
+SMMU_CMDQV_VI_VCMDQi_BASE_L_(0)
+SMMU_CMDQV_VI_VCMDQi_BASE_L_(1)
+
+#define SMMU_CMDQV_VI_VCMDQi_BASE_H_(i)                          \
+    REG32(VI_VCMDQ##i##_BASE_H,                                  \
+          CMDQV_VINTF_PAGE1_BASE + 0x4 + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VI_VCMDQ##i##_BASE_H, ADDR, 0, 16)
+
+SMMU_CMDQV_VI_VCMDQi_BASE_H_(0)
+SMMU_CMDQV_VI_VCMDQi_BASE_H_(1)
+
+#define SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_L_(i)           \
+    REG32(VI_VCMDQ##i##_CONS_INDX_BASE_DRAM_L,                   \
+          CMDQV_VINTF_PAGE1_BASE + 0x8 + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VI_VCMDQ##i##_CONS_INDX_BASE_DRAM_L, ADDR, 0, 32)
+
+SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_L_(0)
+SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_L_(1)
+
+#define SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_(i)           \
+    REG32(VI_VCMDQ##i##_CONS_INDX_BASE_DRAM_H,                   \
+          CMDQV_VINTF_PAGE1_BASE + 0xc + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VI_VCMDQ##i##_CONS_INDX_BASE_DRAM_H, ADDR, 0, 16)
+
+SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_(0)
+SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_(1)
+
 const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops(void);
 
 #endif /* HW_ARM_TEGRA241_CMDQV_H */
hw/arm/tegra241-cmdqv.c +95 -0
index 813b9a61ce..f859126ad6 100644
@@ -16,6 +16,75 @@
 #include "tegra241-cmdqv.h"
 #include "trace.h"
 
+/*
+ * Read a VCMDQ Page 0 register (control/status) using VCMDQ0_* offsets.
+ *
+ * The caller normalizes the MMIO offset such that @offset0 always refers
+ * to a VCMDQ0_* register, while @index selects the VCMDQ instance.
+ */
+static uint64_t tegra241_cmdqv_read_vcmdq_page0(Tegra241CMDQV *cmdqv,
+                                                hwaddr offset0, int index)
+{
+    uint64_t val = 0;
+
+    switch (offset0) {
+    case A_VCMDQ0_CONS_INDX:
+        val = cmdqv->vcmdq_cons_indx[index];
+        break;
+    case A_VCMDQ0_PROD_INDX:
+        val = cmdqv->vcmdq_prod_indx[index];
+        break;
+    case A_VCMDQ0_CONFIG:
+        val = cmdqv->vcmdq_config[index];
+        break;
+    case A_VCMDQ0_STATUS:
+        val = cmdqv->vcmdq_status[index];
+        break;
+    case A_VCMDQ0_GERROR:
+        val = cmdqv->vcmdq_gerror[index];
+        break;
+    case A_VCMDQ0_GERRORN:
+        val = cmdqv->vcmdq_gerrorn[index];
+        break;
+    default:
+        qemu_log_mask(LOG_UNIMP,
+                      "%s unhandled read access at 0x%" PRIx64 "\n",
+                      __func__, offset0);
+    }
+    trace_tegra241_cmdqv_read_vcmdq_page0(index, offset0, val);
+    return val;
+}
+
+/*
+ * Read a VCMDQ Page 1 register (base / DRAM address) using VCMDQ0_* offsets.
+ */
+static uint64_t tegra241_cmdqv_read_vcmdq_page1(Tegra241CMDQV *cmdqv,
+                                                hwaddr offset0, int index)
+{
+    uint64_t val = 0;
+
+    switch (offset0) {
+    case A_VCMDQ0_BASE_L:
+        val = cmdqv->vcmdq_base[index];
+        break;
+    case A_VCMDQ0_BASE_H:
+        val = cmdqv->vcmdq_base[index] >> 32;
+        break;
+    case A_VCMDQ0_CONS_INDX_BASE_DRAM_L:
+        val = cmdqv->vcmdq_cons_indx_base[index];
+        break;
+    case A_VCMDQ0_CONS_INDX_BASE_DRAM_H:
+        val = cmdqv->vcmdq_cons_indx_base[index] >> 32;
+        break;
+    default:
+        qemu_log_mask(LOG_UNIMP,
+                      "%s unhandled read access at 0x%" PRIx64 "\n",
+                      __func__, offset0);
+    }
+    trace_tegra241_cmdqv_read_vcmdq_page1(index, offset0, val);
+    return val;
+}
+
 static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv,
                                                  hwaddr offset)
 {
@@ -93,6 +162,7 @@ static uint64_t tegra241_cmdqv_read_mmio(void *opaque, hwaddr offset,
 {
     Tegra241CMDQV *cmdqv = (Tegra241CMDQV *)opaque;
     uint64_t val = 0;
+    int index;
 
     if (offset >= TEGRA241_CMDQV_IO_LEN) {
         qemu_log_mask(LOG_UNIMP,
@@ -126,6 +196,31 @@ static uint64_t tegra241_cmdqv_read_mmio(void *opaque, hwaddr offset,
     case A_VINTF0_CONFIG ... A_VINTF0_LVCMDQ_ERR_MAP_3:
         val = tegra241_cmdqv_config_vintf_read(cmdqv, offset);
         break;
+    case A_VI_VCMDQ0_CONS_INDX ... A_VI_VCMDQ1_GERRORN:
+        /*
+         * VINTF Page0 registers are hardware aliases of VCMDQ Page0 registers.
+         * Translate the VINTF aperture offset to its VCMDQ Page0 equivalent
+         * and fall through to the Page0 dispatch below.
+         */
+        offset -= CMDQV_VINTF_PAGE0_BASE - CMDQV_VCMDQ_PAGE0_BASE;
+        QEMU_FALLTHROUGH;
+    case A_VCMDQ0_CONS_INDX ... A_VCMDQ1_GERRORN:
+        /*
+         * Decode a per-VCMDQ Page 0 access. Each VCMDQ occupies a
+         * CMDQV_VCMDQ_STRIDE-byte window; extract the index and normalize
+         * to the VCMDQ0_* offset before calling the Page 0 helper.
+         */
+        index = (offset - CMDQV_VCMDQ_PAGE0_BASE) / CMDQV_VCMDQ_STRIDE;
+        return tegra241_cmdqv_read_vcmdq_page0(cmdqv,
+                offset - index * CMDQV_VCMDQ_STRIDE, index);
+    case A_VI_VCMDQ0_BASE_L ... A_VI_VCMDQ1_CONS_INDX_BASE_DRAM_H:
+        /* Same VINTF-to-VCMDQ translation as VINTF Page0 case above. */
+        offset -= CMDQV_VINTF_PAGE1_BASE - CMDQV_VCMDQ_PAGE1_BASE;
+        QEMU_FALLTHROUGH;
+    case A_VCMDQ0_BASE_L ... A_VCMDQ1_CONS_INDX_BASE_DRAM_H:
+        index = (offset - CMDQV_VCMDQ_PAGE1_BASE) / CMDQV_VCMDQ_STRIDE;
+        return tegra241_cmdqv_read_vcmdq_page1(cmdqv,
+                offset - index * CMDQV_VCMDQ_STRIDE, index);
     default:
         qemu_log_mask(LOG_UNIMP, "%s unhandled read access at 0x%" PRIx64 "\n",
                       __func__, offset);
hw/arm/trace-events +0 -0
index 8c61d66a26..8c34a04b24 100644

hw/arm/trace-events smmuv3_accel_install_ste(uint32_t vsid, const char * type, uint32_t hwpt_id) "vS · line 82

Quoted 8 lines
hw/arm/trace-events +2 -0
@@ -75,6 +75,8 @@ smmuv3_accel_install_ste(uint32_t vsid, const char * type, uint32_t hwpt_id) "vS
 # tegra241-cmdqv
 tegra241_cmdqv_read_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x"
 tegra241_cmdqv_write_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x"
+tegra241_cmdqv_read_vcmdq_page0(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64
+tegra241_cmdqv_read_vcmdq_page1(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64
 
 # strongarm.c
 strongarm_uart_update_parameters(const char *label, int speed, char parity, int data_bits, int stop_bits) "%s speed=%d parity=%c data=%d stop=%d"

Besides looks good to me

Eric

#23 lore raw

RE: [PATCH v5 15/32] hw/arm/tegra241-cmdqv: Emulate VCMDQ register reads

Quoted 15 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 20 May 2026 06:49
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 15/32] hw/arm/tegra241-cmdqv: Emulate VCMDQ register reads

[...]

Quoted 37 lines

hw/arm/tegra241-cmdqv.h REG32(VINTF0_LVCMDQ_ERR_MAP_0, 0x10c0)

Quoted 3 lines
hw/arm/tegra241-cmdqv.h +3 -0
+#define SMMU_CMDQV_VCMDQi_GERRORN_(i)                             \
+    REG32(VCMDQ##i##_GERRORN,                                     \
+          CMDQV_VCMDQ_PAGE0_BASE + 0x14 + i * CMDQV_VCMDQ_STRIDE)

\

hw/arm/tegra241-cmdqv.h REG32(VINTF0_LVCMDQ_ERR_MAP_0, 0x10c0) · line 227

Quoted 6 lines
hw/arm/tegra241-cmdqv.h +6 -0
+    FIELD(VCMDQ##i##_GERRORN, CMDQ_ERR, 0, 1)                     \
+    FIELD(VCMDQ##i##_GERRORN, CONS_DRAM_WR_ABT_ERR, 1, 1)         \
+    FIELD(VCMDQ##i##_GERRORN, CMDQ_INIT_ERR, 2, 1)
+
+SMMU_CMDQV_VCMDQi_GERRORN_(0)
+SMMU_CMDQV_VCMDQi_GERRORN_(1)

nit, what I meant in my v4 comments is I would prefer we group the eventual register definition to represent the actual layout of each page, instead of interleaving them with actual macros:
This would look like:

/* page 0 */

/* vcmdq0 */

hw/arm/tegra241-cmdqv.h +6 -0
+SMMU_CMDQV_VCMDQi_CONS_INDX_(0)
+SMMU_CMDQV_VCMDQi_PROD_INDX_(0)
+SMMU_CMDQV_VCMDQi_CONFIG_(0)
+SMMU_CMDQV_VCMDQi_STATUS_(0)
+SMMU_CMDQV_VCMDQi_GERROR_(0)
+SMMU_CMDQV_VCMDQi_GERRORN_(0)

/* vcmdq1 */

hw/arm/tegra241-cmdqv.h +6 -0
+SMMU_CMDQV_VCMDQi_CONS_INDX_(1)
+SMMU_CMDQV_VCMDQi_PROD_INDX_(1)
+SMMU_CMDQV_VCMDQi_CONFIG_(1)
+SMMU_CMDQV_VCMDQi_STATUS_(1)
+SMMU_CMDQV_VCMDQi_GERROR_(1)
+SMMU_CMDQV_VCMDQi_GERRORN_(1)
Quoted 6 lines

same for page 1

To me it looks easier to remember the layout without looking at the spec.

Taste & colors, what do you prefer?

Looks like I misunderstood your v4 comment. Sure, I will switch to this layout.

Thanks,
Shameer

#24 lore raw

PATCH v5 16/32 hw/arm/tegra241-cmdqv: Emulate VCMDQ register writes

In reply to Shameer Kolothum

From: Nicolin Chen <[email protected]>

This is the write side counterpart of the VCMDQ read emulation. Add write handling for both the direct VCMDQ aperture and the VINTF logical aperture using the same index decoding and VINTF-to-VCMDQ translation logic as the read path.

VINTF aperture writes are translated to their direct-aperture equivalent and update the same cached state. Page 1 registers (BASE, CONS_INDX_BASE) always update the cache. Once IOMMU_HW_QUEUE_ALLOC and viommu_mmap are wired up in a subsequent patch, Page 0 register writes will be forwarded to the hardware- backed mmap'd page.

Ignore VCMDQ BASE writes if the VCMDQ is already enabled.

---
 hw/arm/tegra241-cmdqv.c | 164 +++++++++++++++++++++++++++++++++++++++-
 hw/arm/trace-events     |   2 +
 2 files changed, 164 insertions(+), 2 deletions(-)
hw/arm/tegra241-cmdqv.c +162 -2
index f859126ad6..5d2d9d2e26 100644
@@ -16,6 +16,16 @@
 #include "tegra241-cmdqv.h"
 #include "trace.h"
 
+/*
+ * Returns true if the per-VCMDQ CMDQ_EN_OK bit is set.
+ */
+static bool tegra241_vcmdq_enabled(Tegra241CMDQV *cmdqv, int index)
+{
+    uint32_t status = cmdqv->vcmdq_status[index];
+
+    return status & R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;
+}
+
 /*
  * Read a VCMDQ Page 0 register (control/status) using VCMDQ0_* offsets.
  *
@@ -116,6 +126,108 @@ static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv,
     }
 }
 
+/*
+ * Write a VCMDQ Page 0 register (control/status) using VCMDQ0_* offsets.
+ *
+ * The caller normalizes the MMIO offset such that @offset0 always refers
+ * to a VCMDQ0_* register, while @index selects the VCMDQ instance.
+ *
+ * Page 0 registers are all 32-bit; this helper is only called for 4-byte
+ * writes.
+ */
+static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv,
+                                             hwaddr offset0, int index,
+                                             uint32_t value)
+{
+    switch (offset0) {
+    case A_VCMDQ0_CONS_INDX:
+        cmdqv->vcmdq_cons_indx[index] = value;
+        break;
+    case A_VCMDQ0_PROD_INDX:
+        cmdqv->vcmdq_prod_indx[index] = value;
+        break;
+    case A_VCMDQ0_CONFIG:
+        if (value & R_VCMDQ0_CONFIG_CMDQ_EN_MASK) {
+            cmdqv->vcmdq_status[index] |= R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;
+        } else {
+            cmdqv->vcmdq_status[index] &= ~R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;
+        }
+        cmdqv->vcmdq_config[index] = value;
+        break;
+    case A_VCMDQ0_GERRORN:
+        cmdqv->vcmdq_gerrorn[index] = value;
+        break;
+    default:
+        qemu_log_mask(LOG_UNIMP,
+                      "%s unhandled write access at 0x%" PRIx64 "\n",
+                      __func__, offset0);
+    }
+    trace_tegra241_cmdqv_write_vcmdq_page0(index, offset0, value);
+}
+
+/*
+ * Write a VCMDQ Page 1 register (base / DRAM address) - 4-byte access.
+ */
+static void tegra241_cmdqv_write_vcmdq_page1(Tegra241CMDQV *cmdqv,
+                                             hwaddr offset0, int index,
+                                             uint32_t value)
+{
+    switch (offset0) {
+    case A_VCMDQ0_BASE_L:
+        if (tegra241_vcmdq_enabled(cmdqv, index)) {
+            return;
+        }
+        cmdqv->vcmdq_base[index] =
+            deposit64(cmdqv->vcmdq_base[index], 0, 32, value);
+        break;
+    case A_VCMDQ0_BASE_H:
+        if (tegra241_vcmdq_enabled(cmdqv, index)) {
+            return;
+        }
+        cmdqv->vcmdq_base[index] =
+            deposit64(cmdqv->vcmdq_base[index], 32, 32, value);
+        break;
+    case A_VCMDQ0_CONS_INDX_BASE_DRAM_L:
+        cmdqv->vcmdq_cons_indx_base[index] =
+            deposit64(cmdqv->vcmdq_cons_indx_base[index], 0, 32, value);
+        break;
+    case A_VCMDQ0_CONS_INDX_BASE_DRAM_H:
+        cmdqv->vcmdq_cons_indx_base[index] =
+            deposit64(cmdqv->vcmdq_cons_indx_base[index], 32, 32, value);
+        break;
+    default:
+        qemu_log_mask(LOG_UNIMP,
+                      "%s unhandled write access at 0x%" PRIx64 "\n",
+                      __func__, offset0);
+    }
+    trace_tegra241_cmdqv_write_vcmdq_page1(index, offset0, value);
+}
+
+/*
+ * Write a VCMDQ Page 1 register - 8-byte access at BASE_L or DRAM_L.
+ */
+static void tegra241_cmdqv_write_vcmdq_page1_64(Tegra241CMDQV *cmdqv,
+                                                hwaddr offset0, int index,
+                                                uint64_t value)
+{
+    switch (offset0) {
+    case A_VCMDQ0_BASE_L:
+        if (tegra241_vcmdq_enabled(cmdqv, index)) {
+            return;
+        }
+        cmdqv->vcmdq_base[index] = value;
+        break;
+    case A_VCMDQ0_CONS_INDX_BASE_DRAM_L:
+        cmdqv->vcmdq_cons_indx_base[index] = value;
+        break;
+    default:
+        qemu_log_mask(LOG_UNIMP,
+                      "%s unhandled 64-bit write at 0x%" PRIx64 "\n",
+                      __func__, offset0);
+    }
+    trace_tegra241_cmdqv_write_vcmdq_page1(index, offset0, value);
+}
+
 static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv,
                                               hwaddr offset, uint64_t value)
 {
@@ -235,6 +347,8 @@ out:
 static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,
                                        uint32_t value)
 {
+    int index;
+
     switch (offset) {
     case A_CONFIG:
         cmdqv->config = value;
@@ -253,6 +367,33 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,
     case A_VINTF0_CONFIG ... A_VINTF0_LVCMDQ_ERR_MAP_3:
         tegra241_cmdqv_config_vintf_write(cmdqv, offset, value);
         break;
+    case A_VI_VCMDQ0_CONS_INDX ... A_VI_VCMDQ1_GERRORN:
+        /*
+         * VINTF Page0 registers are hardware aliases of VCMDQ Page0 registers.
+         * Translate the VINTF aperture offset to its VCMDQ Page0 equivalent
+         * and fall through to the Page0 dispatch below.
+         */
+        offset -= CMDQV_VINTF_PAGE0_BASE - CMDQV_VCMDQ_PAGE0_BASE;
+        QEMU_FALLTHROUGH;
+    case A_VCMDQ0_CONS_INDX ... A_VCMDQ1_GERRORN:
+        /*
+         * Decode a per-VCMDQ Page 0 access. Each VCMDQ occupies a
+         * CMDQV_VCMDQ_STRIDE-byte window; extract the index and normalize
+         * to the VCMDQ0_* offset before calling the Page 0 helper.
+         */
+        index = (offset - CMDQV_VCMDQ_PAGE0_BASE) / CMDQV_VCMDQ_STRIDE;
+        tegra241_cmdqv_write_vcmdq_page0(cmdqv,
+                offset - index * CMDQV_VCMDQ_STRIDE, index, value);
+        break;
+    case A_VI_VCMDQ0_BASE_L ... A_VI_VCMDQ1_CONS_INDX_BASE_DRAM_H:
+        /* Same VINTF-to-VCMDQ translation as VINTF Page0 case above. */
+        offset -= CMDQV_VINTF_PAGE1_BASE - CMDQV_VCMDQ_PAGE1_BASE;
+        QEMU_FALLTHROUGH;
+    case A_VCMDQ0_BASE_L ... A_VCMDQ1_CONS_INDX_BASE_DRAM_H:
+        index = (offset - CMDQV_VCMDQ_PAGE1_BASE) / CMDQV_VCMDQ_STRIDE;
+        tegra241_cmdqv_write_vcmdq_page1(cmdqv,
+                offset - index * CMDQV_VCMDQ_STRIDE, index, value);
+        break;
     default:
         qemu_log_mask(LOG_UNIMP, "%s unhandled write access at 0x%" PRIx64 "\n",
                       __func__, offset);
@@ -260,14 +401,33 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,
 }
 
 /*
- * 8-byte MMIO write handler.
+ * 8-byte MMIO write handler. Only Page 1 BASE / CONS_INDX_BASE_DRAM accept
+ * full 64-bit writes; other offsets are write-ignored.
  */
 static void tegra241_cmdqv_writell_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,
                                         uint64_t value)
 {
-    qemu_log_mask(LOG_UNIMP,
+    int index;
+
+    switch (offset) {
+    case A_VI_VCMDQ0_BASE_L ... A_VI_VCMDQ1_CONS_INDX_BASE_DRAM_H:
+        /*
+         * VINTF Page1 registers are hardware aliases of VCMDQ Page1 registers.
+         * Translate the VINTF aperture offset to its VCMDQ Page1 equivalent
+         * and fall through to the Page1 dispatch below.
+         */
+        offset -= CMDQV_VINTF_PAGE1_BASE - CMDQV_VCMDQ_PAGE1_BASE;
+        QEMU_FALLTHROUGH;
+    case A_VCMDQ0_BASE_L ... A_VCMDQ1_CONS_INDX_BASE_DRAM_H:
+        index = (offset - CMDQV_VCMDQ_PAGE1_BASE) / CMDQV_VCMDQ_STRIDE;
+        tegra241_cmdqv_write_vcmdq_page1_64(cmdqv,
+                offset - index * CMDQV_VCMDQ_STRIDE, index, value);
+        break;
+    default:
+        qemu_log_mask(LOG_UNIMP,
                       "%s unhandled 64-bit write at 0x%" PRIx64 " (WI)\n",
                       __func__, offset);
+    }
 }
 
 static void tegra241_cmdqv_write_mmio(void *opaque, hwaddr offset,
hw/arm/trace-events +2 -0
index 8c34a04b24..c4262bb2be 100644
@@ -77,6 +77,8 @@ tegra241_cmdqv_read_mmio(uint64_t offset, uint64_t val, unsigned size) "offset:
 tegra241_cmdqv_write_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x"
 tegra241_cmdqv_read_vcmdq_page0(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64
 tegra241_cmdqv_read_vcmdq_page1(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64
+tegra241_cmdqv_write_vcmdq_page0(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64
+tegra241_cmdqv_write_vcmdq_page1(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64
 
 # strongarm.c
 strongarm_uart_update_parameters(const char *label, int speed, char parity, int data_bits, int stop_bits) "%s speed=%d parity=%c data=%d stop=%d"
Signature · 1 line
2.43.0
#25 lore raw

Re: [PATCH v5 16/32] hw/arm/tegra241-cmdqv: Emulate VCMDQ register writes

Quoted 238 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

This is the write side counterpart of the VCMDQ read emulation. Add write handling for both the direct VCMDQ aperture and the VINTF logical aperture using the same index decoding and VINTF-to-VCMDQ translation logic as the read path.

VINTF aperture writes are translated to their direct-aperture equivalent and update the same cached state. Page 1 registers (BASE, CONS_INDX_BASE) always update the cache. Once IOMMU_HW_QUEUE_ALLOC and viommu_mmap are wired up in a subsequent patch, Page 0 register writes will be forwarded to the hardware- backed mmap'd page.

Ignore VCMDQ BASE writes if the VCMDQ is already enabled.

---
 hw/arm/tegra241-cmdqv.c | 164 +++++++++++++++++++++++++++++++++++++++-
 hw/arm/trace-events     |   2 +
 2 files changed, 164 insertions(+), 2 deletions(-)
hw/arm/tegra241-cmdqv.c +162 -2
index f859126ad6..5d2d9d2e26 100644
@@ -16,6 +16,16 @@
 #include "tegra241-cmdqv.h"
 #include "trace.h"
 
+/*
+ * Returns true if the per-VCMDQ CMDQ_EN_OK bit is set.
+ */
+static bool tegra241_vcmdq_enabled(Tegra241CMDQV *cmdqv, int index)
+{
+    uint32_t status = cmdqv->vcmdq_status[index];
+
+    return status & R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;
+}
+
 /*
  * Read a VCMDQ Page 0 register (control/status) using VCMDQ0_* offsets.
  *
@@ -116,6 +126,108 @@ static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv,
     }
 }
 
+/*
+ * Write a VCMDQ Page 0 register (control/status) using VCMDQ0_* offsets.
+ *
+ * The caller normalizes the MMIO offset such that @offset0 always refers
+ * to a VCMDQ0_* register, while @index selects the VCMDQ instance.
+ *
+ * Page 0 registers are all 32-bit; this helper is only called for 4-byte
+ * writes.
+ */
+static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv,
+                                             hwaddr offset0, int index,
+                                             uint32_t value)
+{
+    switch (offset0) {
+    case A_VCMDQ0_CONS_INDX:
+        cmdqv->vcmdq_cons_indx[index] = value;
+        break;
+    case A_VCMDQ0_PROD_INDX:
+        cmdqv->vcmdq_prod_indx[index] = value;
+        break;
+    case A_VCMDQ0_CONFIG:
+        if (value & R_VCMDQ0_CONFIG_CMDQ_EN_MASK) {
+            cmdqv->vcmdq_status[index] |= R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;
+        } else {
+            cmdqv->vcmdq_status[index] &= ~R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;
+        }
+        cmdqv->vcmdq_config[index] = value;
+        break;
+    case A_VCMDQ0_GERRORN:
+        cmdqv->vcmdq_gerrorn[index] = value;
+        break;
+    default:
+        qemu_log_mask(LOG_UNIMP,
+                      "%s unhandled write access at 0x%" PRIx64 "\n",
+                      __func__, offset0);
+    }
+    trace_tegra241_cmdqv_write_vcmdq_page0(index, offset0, value);
+}
+
+/*
+ * Write a VCMDQ Page 1 register (base / DRAM address) - 4-byte access.
+ */
+static void tegra241_cmdqv_write_vcmdq_page1(Tegra241CMDQV *cmdqv,
+                                             hwaddr offset0, int index,
+                                             uint32_t value)
+{
+    switch (offset0) {
+    case A_VCMDQ0_BASE_L:
+        if (tegra241_vcmdq_enabled(cmdqv, index)) {
+            return;
+        }
+        cmdqv->vcmdq_base[index] =
+            deposit64(cmdqv->vcmdq_base[index], 0, 32, value);
+        break;
+    case A_VCMDQ0_BASE_H:
+        if (tegra241_vcmdq_enabled(cmdqv, index)) {
+            return;
+        }
+        cmdqv->vcmdq_base[index] =
+            deposit64(cmdqv->vcmdq_base[index], 32, 32, value);
+        break;
+    case A_VCMDQ0_CONS_INDX_BASE_DRAM_L:
+        cmdqv->vcmdq_cons_indx_base[index] =
+            deposit64(cmdqv->vcmdq_cons_indx_base[index], 0, 32, value);
+        break;
+    case A_VCMDQ0_CONS_INDX_BASE_DRAM_H:
+        cmdqv->vcmdq_cons_indx_base[index] =
+            deposit64(cmdqv->vcmdq_cons_indx_base[index], 32, 32, value);
+        break;
+    default:
+        qemu_log_mask(LOG_UNIMP,
+                      "%s unhandled write access at 0x%" PRIx64 "\n",
+                      __func__, offset0);
+    }
+    trace_tegra241_cmdqv_write_vcmdq_page1(index, offset0, value);
+}
+
+/*
+ * Write a VCMDQ Page 1 register - 8-byte access at BASE_L or DRAM_L.
+ */
+static void tegra241_cmdqv_write_vcmdq_page1_64(Tegra241CMDQV *cmdqv,
+                                                hwaddr offset0, int index,
+                                                uint64_t value)
+{
+    switch (offset0) {
+    case A_VCMDQ0_BASE_L:
+        if (tegra241_vcmdq_enabled(cmdqv, index)) {
+            return;
+        }
+        cmdqv->vcmdq_base[index] = value;
+        break;
+    case A_VCMDQ0_CONS_INDX_BASE_DRAM_L:
+        cmdqv->vcmdq_cons_indx_base[index] = value;
+        break;
+    default:
+        qemu_log_mask(LOG_UNIMP,
+                      "%s unhandled 64-bit write at 0x%" PRIx64 "\n",
+                      __func__, offset0);
+    }
+    trace_tegra241_cmdqv_write_vcmdq_page1(index, offset0, value);
+}
+
 static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv,
                                               hwaddr offset, uint64_t value)
 {
@@ -235,6 +347,8 @@ out:
 static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,
                                        uint32_t value)
 {
+    int index;
+
     switch (offset) {
     case A_CONFIG:
         cmdqv->config = value;
@@ -253,6 +367,33 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,
     case A_VINTF0_CONFIG ... A_VINTF0_LVCMDQ_ERR_MAP_3:
         tegra241_cmdqv_config_vintf_write(cmdqv, offset, value);
         break;
+    case A_VI_VCMDQ0_CONS_INDX ... A_VI_VCMDQ1_GERRORN:
+        /*
+         * VINTF Page0 registers are hardware aliases of VCMDQ Page0 registers.
+         * Translate the VINTF aperture offset to its VCMDQ Page0 equivalent
+         * and fall through to the Page0 dispatch below.
+         */
+        offset -= CMDQV_VINTF_PAGE0_BASE - CMDQV_VCMDQ_PAGE0_BASE;
+        QEMU_FALLTHROUGH;
+    case A_VCMDQ0_CONS_INDX ... A_VCMDQ1_GERRORN:
+        /*
+         * Decode a per-VCMDQ Page 0 access. Each VCMDQ occupies a
+         * CMDQV_VCMDQ_STRIDE-byte window; extract the index and normalize
+         * to the VCMDQ0_* offset before calling the Page 0 helper.
+         */
+        index = (offset - CMDQV_VCMDQ_PAGE0_BASE) / CMDQV_VCMDQ_STRIDE;
+        tegra241_cmdqv_write_vcmdq_page0(cmdqv,
+                offset - index * CMDQV_VCMDQ_STRIDE, index, value);
+        break;
+    case A_VI_VCMDQ0_BASE_L ... A_VI_VCMDQ1_CONS_INDX_BASE_DRAM_H:
+        /* Same VINTF-to-VCMDQ translation as VINTF Page0 case above. */
+        offset -= CMDQV_VINTF_PAGE1_BASE - CMDQV_VCMDQ_PAGE1_BASE;
+        QEMU_FALLTHROUGH;
+    case A_VCMDQ0_BASE_L ... A_VCMDQ1_CONS_INDX_BASE_DRAM_H:
+        index = (offset - CMDQV_VCMDQ_PAGE1_BASE) / CMDQV_VCMDQ_STRIDE;
+        tegra241_cmdqv_write_vcmdq_page1(cmdqv,
+                offset - index * CMDQV_VCMDQ_STRIDE, index, value);
+        break;
     default:
         qemu_log_mask(LOG_UNIMP, "%s unhandled write access at 0x%" PRIx64 "\n",
                       __func__, offset);
@@ -260,14 +401,33 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,
 }
 
 /*
- * 8-byte MMIO write handler.
+ * 8-byte MMIO write handler. Only Page 1 BASE / CONS_INDX_BASE_DRAM accept
+ * full 64-bit writes; other offsets are write-ignored.
  */
 static void tegra241_cmdqv_writell_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,
                                         uint64_t value)
 {
-    qemu_log_mask(LOG_UNIMP,
+    int index;
+
+    switch (offset) {
+    case A_VI_VCMDQ0_BASE_L ... A_VI_VCMDQ1_CONS_INDX_BASE_DRAM_H:
+        /*
+         * VINTF Page1 registers are hardware aliases of VCMDQ Page1 registers.
+         * Translate the VINTF aperture offset to its VCMDQ Page1 equivalent
+         * and fall through to the Page1 dispatch below.
+         */
+        offset -= CMDQV_VINTF_PAGE1_BASE - CMDQV_VCMDQ_PAGE1_BASE;
+        QEMU_FALLTHROUGH;
+    case A_VCMDQ0_BASE_L ... A_VCMDQ1_CONS_INDX_BASE_DRAM_H:
+        index = (offset - CMDQV_VCMDQ_PAGE1_BASE) / CMDQV_VCMDQ_STRIDE;
+        tegra241_cmdqv_write_vcmdq_page1_64(cmdqv,
+                offset - index * CMDQV_VCMDQ_STRIDE, index, value);
+        break;
+    default:
+        qemu_log_mask(LOG_UNIMP,
                       "%s unhandled 64-bit write at 0x%" PRIx64 " (WI)\n",
                       __func__, offset);
+    }
 }
 
 static void tegra241_cmdqv_write_mmio(void *opaque, hwaddr offset,
hw/arm/trace-events +0 -0
index 8c34a04b24..c4262bb2be 100644

hw/arm/trace-events tegra241_cmdqv_read_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: · line 81

Quoted 5 lines
hw/arm/trace-events +2 -0
@@ -77,6 +77,8 @@ tegra241_cmdqv_read_mmio(uint64_t offset, uint64_t val, unsigned size) "offset:
 tegra241_cmdqv_write_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x"
 tegra241_cmdqv_read_vcmdq_page0(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64
 tegra241_cmdqv_read_vcmdq_page1(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64
+tegra241_cmdqv_write_vcmdq_page0(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64
+tegra241_cmdqv_write_vcmdq_page1(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64

actually would it be possible to improve the trace points by adding the info about whether the VI or the global page is accessed. This would allow to discriminate both. Same for read

Eric

Quoted 3 lines
hw/arm/trace-events +0 -0
 # strongarm.c
 strongarm_uart_update_parameters(const char *label, int speed, char parity, int data_bits, int stop_bits) "%s speed=%d parity=%c data=%d stop=%d"
#26 lore raw

RE: [PATCH v5 16/32] hw/arm/tegra241-cmdqv: Emulate VCMDQ register writes

Quoted 14 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 20 May 2026 06:56
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 16/32] hw/arm/tegra241-cmdqv: Emulate VCMDQ register writes

[...]

Quoted 7 lines
Quoted 1 line

+tegra241_cmdqv_write_vcmdq_page0(int index, uint64_t offset0, uint64_t

val) "vcmdq[%d] page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64

Quoted 1 line

+tegra241_cmdqv_write_vcmdq_page1(int index, uint64_t offset0, uint64_t

val) "vcmdq[%d] page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64 actually would it be possible to improve the trace points by adding the info about whether the VI or the global page is accessed. This would allow to discriminate both. Same for read

Ok. Will add for v6.

Thanks,
Shameer

#27 lore raw

Re: [PATCH v5 16/32] hw/arm/tegra241-cmdqv: Emulate VCMDQ register writes

In reply to Shameer Kolothum

Hi Shameer,

Quoted 58 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

This is the write side counterpart of the VCMDQ read emulation. Add write handling for both the direct VCMDQ aperture and the VINTF logical aperture using the same index decoding and VINTF-to-VCMDQ translation logic as the read path.

VINTF aperture writes are translated to their direct-aperture equivalent and update the same cached state. Page 1 registers (BASE, CONS_INDX_BASE) always update the cache. Once IOMMU_HW_QUEUE_ALLOC and viommu_mmap are wired up in a subsequent patch, Page 0 register writes will be forwarded to the hardware- backed mmap'd page.

Ignore VCMDQ BASE writes if the VCMDQ is already enabled.

---
 hw/arm/tegra241-cmdqv.c | 164 +++++++++++++++++++++++++++++++++++++++-
 hw/arm/trace-events     |   2 +
 2 files changed, 164 insertions(+), 2 deletions(-)
hw/arm/tegra241-cmdqv.c +19 -0
index f859126ad6..5d2d9d2e26 100644
@@ -16,6 +16,16 @@
 #include "tegra241-cmdqv.h"
 #include "trace.h"
 
+/*
+ * Returns true if the per-VCMDQ CMDQ_EN_OK bit is set.
+ */
+static bool tegra241_vcmdq_enabled(Tegra241CMDQV *cmdqv, int index)
+{
+    uint32_t status = cmdqv->vcmdq_status[index];
+
+    return status & R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;
+}
+
 /*
  * Read a VCMDQ Page 0 register (control/status) using VCMDQ0_* offsets.
  *
@@ -116,6 +126,108 @@ static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv,
     }
 }
 
+/*
+ * Write a VCMDQ Page 0 register (control/status) using VCMDQ0_* offsets.
+ *
+ * The caller normalizes the MMIO offset such that @offset0 always refers
+ * to a VCMDQ0_* register, while @index selects the VCMDQ instance.
+ *
+ * Page 0 registers are all 32-bit; this helper is only called for 4-byte
+ * writes.
+ */

hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv, · line 147

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10 -0
+static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv,
+                                             hwaddr offset0, int index,
+                                             uint32_t value)
+{
+    switch (offset0) {
+    case A_VCMDQ0_CONS_INDX:
+        cmdqv->vcmdq_cons_indx[index] = value;
+        break;
+    case A_VCMDQ0_PROD_INDX:
+        cmdqv->vcmdq_prod_indx[index] = value;

so there is no cmd consumption triggered by an increment of the producer index as opposed to std smmu emulation. This shall at least be explained in the commit msg and maybe in the code, + the justification. It is a bit strange to emulate the MMIO accesses while not doing anything functional out of them.

hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv, · line 158

Quoted 11 lines
hw/arm/tegra241-cmdqv.c +11 -0
+        break;
+    case A_VCMDQ0_CONFIG:
+        if (value & R_VCMDQ0_CONFIG_CMDQ_EN_MASK) {
+            cmdqv->vcmdq_status[index] |= R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;
+        } else {
+            cmdqv->vcmdq_status[index] &= ~R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;
+        }
+        cmdqv->vcmdq_config[index] = value;
+        break;
+    case A_VCMDQ0_GERRORN:
+        cmdqv->vcmdq_gerrorn[index] = value;

shouldn't it do something functional besides storing the value on std SMMU we resume consuming cmds. interaction with interrupts, CONS_INDX?

Quoted 16 lines
hw/arm/tegra241-cmdqv.c +16 -0
+        break;
+    default:
+        qemu_log_mask(LOG_UNIMP,
+                      "%s unhandled write access at 0x%" PRIx64 "\n",
+                      __func__, offset0);
+    }
+    trace_tegra241_cmdqv_write_vcmdq_page0(index, offset0, value);
+}
+
+/*
+ * Write a VCMDQ Page 1 register (base / DRAM address) - 4-byte access.
+ */
+static void tegra241_cmdqv_write_vcmdq_page1(Tegra241CMDQV *cmdqv,
+                                             hwaddr offset0, int index,
+                                             uint32_t value)
+{

hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv, · line 184

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10 -0
+    switch (offset0) {
+    case A_VCMDQ0_BASE_L:
+        if (tegra241_vcmdq_enabled(cmdqv, index)) {
+            return;
+        }
+        cmdqv->vcmdq_base[index] =
+            deposit64(cmdqv->vcmdq_base[index], 0, 32, value);
+        break;
+    case A_VCMDQ0_BASE_H:
+        if (tegra241_vcmdq_enabled(cmdqv, index)) {

where is it documented in the spec? I don't see this restriction in the reg doc. Then "Enabling the virtual CMDQ" paragraph describes a std setup but does not say - afaics- that it should be ignored - worth commenting

Quoted 133 lines
hw/arm/tegra241-cmdqv.c +96 -2
+            return;
+        }
+        cmdqv->vcmdq_base[index] =
+            deposit64(cmdqv->vcmdq_base[index], 32, 32, value);
+        break;
+    case A_VCMDQ0_CONS_INDX_BASE_DRAM_L:
+        cmdqv->vcmdq_cons_indx_base[index] =
+            deposit64(cmdqv->vcmdq_cons_indx_base[index], 0, 32, value);
+        break;
+    case A_VCMDQ0_CONS_INDX_BASE_DRAM_H:
+        cmdqv->vcmdq_cons_indx_base[index] =
+            deposit64(cmdqv->vcmdq_cons_indx_base[index], 32, 32, value);
+        break;
+    default:
+        qemu_log_mask(LOG_UNIMP,
+                      "%s unhandled write access at 0x%" PRIx64 "\n",
+                      __func__, offset0);
+    }
+    trace_tegra241_cmdqv_write_vcmdq_page1(index, offset0, value);
+}
+
+/*
+ * Write a VCMDQ Page 1 register - 8-byte access at BASE_L or DRAM_L.
+ */
+static void tegra241_cmdqv_write_vcmdq_page1_64(Tegra241CMDQV *cmdqv,
+                                                hwaddr offset0, int index,
+                                                uint64_t value)
+{
+    switch (offset0) {
+    case A_VCMDQ0_BASE_L:
+        if (tegra241_vcmdq_enabled(cmdqv, index)) {
+            return;
+        }
+        cmdqv->vcmdq_base[index] = value;
+        break;
+    case A_VCMDQ0_CONS_INDX_BASE_DRAM_L:
+        cmdqv->vcmdq_cons_indx_base[index] = value;
+        break;
+    default:
+        qemu_log_mask(LOG_UNIMP,
+                      "%s unhandled 64-bit write at 0x%" PRIx64 "\n",
+                      __func__, offset0);
+    }
+    trace_tegra241_cmdqv_write_vcmdq_page1(index, offset0, value);
+}
+
 static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv,
                                               hwaddr offset, uint64_t value)
 {
@@ -235,6 +347,8 @@ out:
 static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,
                                        uint32_t value)
 {
+    int index;
+
     switch (offset) {
     case A_CONFIG:
         cmdqv->config = value;
@@ -253,6 +367,33 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,
     case A_VINTF0_CONFIG ... A_VINTF0_LVCMDQ_ERR_MAP_3:
         tegra241_cmdqv_config_vintf_write(cmdqv, offset, value);
         break;
+    case A_VI_VCMDQ0_CONS_INDX ... A_VI_VCMDQ1_GERRORN:
+        /*
+         * VINTF Page0 registers are hardware aliases of VCMDQ Page0 registers.
+         * Translate the VINTF aperture offset to its VCMDQ Page0 equivalent
+         * and fall through to the Page0 dispatch below.
+         */
+        offset -= CMDQV_VINTF_PAGE0_BASE - CMDQV_VCMDQ_PAGE0_BASE;
+        QEMU_FALLTHROUGH;
+    case A_VCMDQ0_CONS_INDX ... A_VCMDQ1_GERRORN:
+        /*
+         * Decode a per-VCMDQ Page 0 access. Each VCMDQ occupies a
+         * CMDQV_VCMDQ_STRIDE-byte window; extract the index and normalize
+         * to the VCMDQ0_* offset before calling the Page 0 helper.
+         */
+        index = (offset - CMDQV_VCMDQ_PAGE0_BASE) / CMDQV_VCMDQ_STRIDE;
+        tegra241_cmdqv_write_vcmdq_page0(cmdqv,
+                offset - index * CMDQV_VCMDQ_STRIDE, index, value);
+        break;
+    case A_VI_VCMDQ0_BASE_L ... A_VI_VCMDQ1_CONS_INDX_BASE_DRAM_H:
+        /* Same VINTF-to-VCMDQ translation as VINTF Page0 case above. */
+        offset -= CMDQV_VINTF_PAGE1_BASE - CMDQV_VCMDQ_PAGE1_BASE;
+        QEMU_FALLTHROUGH;
+    case A_VCMDQ0_BASE_L ... A_VCMDQ1_CONS_INDX_BASE_DRAM_H:
+        index = (offset - CMDQV_VCMDQ_PAGE1_BASE) / CMDQV_VCMDQ_STRIDE;
+        tegra241_cmdqv_write_vcmdq_page1(cmdqv,
+                offset - index * CMDQV_VCMDQ_STRIDE, index, value);
+        break;
     default:
         qemu_log_mask(LOG_UNIMP, "%s unhandled write access at 0x%" PRIx64 "\n",
                       __func__, offset);
@@ -260,14 +401,33 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,
 }
 
 /*
- * 8-byte MMIO write handler.
+ * 8-byte MMIO write handler. Only Page 1 BASE / CONS_INDX_BASE_DRAM accept
+ * full 64-bit writes; other offsets are write-ignored.
  */
 static void tegra241_cmdqv_writell_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,
                                         uint64_t value)
 {
-    qemu_log_mask(LOG_UNIMP,
+    int index;
+
+    switch (offset) {
+    case A_VI_VCMDQ0_BASE_L ... A_VI_VCMDQ1_CONS_INDX_BASE_DRAM_H:
+        /*
+         * VINTF Page1 registers are hardware aliases of VCMDQ Page1 registers.
+         * Translate the VINTF aperture offset to its VCMDQ Page1 equivalent
+         * and fall through to the Page1 dispatch below.
+         */
+        offset -= CMDQV_VINTF_PAGE1_BASE - CMDQV_VCMDQ_PAGE1_BASE;
+        QEMU_FALLTHROUGH;
+    case A_VCMDQ0_BASE_L ... A_VCMDQ1_CONS_INDX_BASE_DRAM_H:
+        index = (offset - CMDQV_VCMDQ_PAGE1_BASE) / CMDQV_VCMDQ_STRIDE;
+        tegra241_cmdqv_write_vcmdq_page1_64(cmdqv,
+                offset - index * CMDQV_VCMDQ_STRIDE, index, value);
+        break;
+    default:
+        qemu_log_mask(LOG_UNIMP,
                       "%s unhandled 64-bit write at 0x%" PRIx64 " (WI)\n",
                       __func__, offset);
+    }
 }
 
 static void tegra241_cmdqv_write_mmio(void *opaque, hwaddr offset,
hw/arm/trace-events +0 -0
index 8c34a04b24..c4262bb2be 100644

hw/arm/trace-events tegra241_cmdqv_read_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: · line 84

Quoted 8 lines
hw/arm/trace-events +2 -0
@@ -77,6 +77,8 @@ tegra241_cmdqv_read_mmio(uint64_t offset, uint64_t val, unsigned size) "offset:
 tegra241_cmdqv_write_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x"
 tegra241_cmdqv_read_vcmdq_page0(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64
 tegra241_cmdqv_read_vcmdq_page1(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64
+tegra241_cmdqv_write_vcmdq_page0(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64
+tegra241_cmdqv_write_vcmdq_page1(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64
 
 # strongarm.c
 strongarm_uart_update_parameters(const char *label, int speed, char parity, int data_bits, int stop_bits) "%s speed=%d parity=%c data=%d stop=%d"

Thanks

Eric

#28 lore raw

RE: [PATCH v5 16/32] hw/arm/tegra241-cmdqv: Emulate VCMDQ register writes

Quoted 93 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 20 May 2026 08:54
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 16/32] hw/arm/tegra241-cmdqv: Emulate VCMDQ register writes

External email: Use caution opening links or attachments

Hi Shameer,

Quoted 11 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

This is the write side counterpart of the VCMDQ read emulation. Add write handling for both the direct VCMDQ aperture and the VINTF logical aperture using the same index decoding and VINTF-to-VCMDQ translation logic as the read path.

VINTF aperture writes are translated to their direct-aperture equivalent and update the same cached state. Page 1 registers (BASE, CONS_INDX_BASE) always update the cache. Once IOMMU_HW_QUEUE_ALLOC and viommu_mmap are wired up in a

subsequent

Quoted 10 lines

patch, Page 0 register writes will be forwarded to the hardware- backed mmap'd page.

Ignore VCMDQ BASE writes if the VCMDQ is already enabled.

---
 hw/arm/tegra241-cmdqv.c | 164

+++++++++++++++++++++++++++++++++++++++-

hw/arm/tegra241-cmdqv.c static uint64_t

Quoted 25 lines
 hw/arm/trace-events     |   2 +
 2 files changed, 164 insertions(+), 2 deletions(-)
hw/arm/tegra241-cmdqv.c +10 -0
index f859126ad6..5d2d9d2e26 100644
@@ -16,6 +16,16 @@
 #include "tegra241-cmdqv.h"
 #include "trace.h"
 
+/*
+ * Returns true if the per-VCMDQ CMDQ_EN_OK bit is set.
+ */
+static bool tegra241_vcmdq_enabled(Tegra241CMDQV *cmdqv, int index)
+{
+    uint32_t status = cmdqv->vcmdq_status[index];
+
+    return status & R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;
+}
+
 /*
  * Read a VCMDQ Page 0 register (control/status) using VCMDQ0_* offsets.
  *
@@ -116,6 +126,108 @@ static uint64_t

tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv,

hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv,

Quoted 13 lines
hw/arm/tegra241-cmdqv.c +10 -0
     }
 }
 
+/*
+ * Write a VCMDQ Page 0 register (control/status) using VCMDQ0_* offsets.
+ *
+ * The caller normalizes the MMIO offset such that @offset0 always refers
+ * to a VCMDQ0_* register, while @index selects the VCMDQ instance.
+ *
+ * Page 0 registers are all 32-bit; this helper is only called for 4-byte
+ * writes.
+ */
+static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV

*cmdqv,

hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv, · line 147

Quoted 9 lines
hw/arm/tegra241-cmdqv.c +9 -0
+                                             hwaddr offset0, int index,
+                                             uint32_t value)
+{
+    switch (offset0) {
+    case A_VCMDQ0_CONS_INDX:
+        cmdqv->vcmdq_cons_indx[index] = value;
+        break;
+    case A_VCMDQ0_PROD_INDX:
+        cmdqv->vcmdq_prod_indx[index] = value;
Quoted 5 lines

so there is no cmd consumption triggered by an increment of the producer index as opposed to std smmu emulation. This shall at least be explained in the commit msg and maybe in the code, + the justification. It is a bit strange to emulate the MMIO accesses while not doing anything functional out of them.

There is indeed a difference here w.r.t. std SMMU emulation. CMDQV
is a hardware-accelerated feature, and there isn't much value in emulating
the queue functionality without the HW backing.

From Spec, p.176:

"While the software can program the Virtual CMDQ(s) directly using the direct VCMDQ aperture (and not through the Virtual Interface), it is required that the VCMDQ be allocated to a Virtual Interface before it is used to send commands to the SMMU"

So no actual command processing can happen until the VCMDQ is allocated to a Virtual Interface. Until then, reads/writes only update the register cache.

I will make this clear in the commit log.

Quoted 8 lines

hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv,

Quoted 4 lines
hw/arm/tegra241-cmdqv.c +4 -0
+        break;
+    case A_VCMDQ0_CONFIG:
+        if (value & R_VCMDQ0_CONFIG_CMDQ_EN_MASK) {
+            cmdqv->vcmdq_status[index] |=

R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;

hw/arm/tegra241-cmdqv.c static uint64_t · line 153

Quoted 2 lines
hw/arm/tegra241-cmdqv.c +2 -0
+        } else {
+            cmdqv->vcmdq_status[index] &=

~R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;

Quoted 8 lines

hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv, · line 158

Quoted 5 lines
hw/arm/tegra241-cmdqv.c +5 -0
+        }
+        cmdqv->vcmdq_config[index] = value;
+        break;
+    case A_VCMDQ0_GERRORN:
+        cmdqv->vcmdq_gerrorn[index] = value;

shouldn't it do something functional besides storing the value on std SMMU we resume consuming cmds. interaction with interrupts, CONS_INDX?

Same rationale as above. In the pre-allocation state there is no real queue function to drive, so there is nothing functional for QEMU to do on a GERRORN write.

Quoted 27 lines

hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv,

Quoted 13 lines
hw/arm/tegra241-cmdqv.c +13 -0
+        break;
+    default:
+        qemu_log_mask(LOG_UNIMP,
+                      "%s unhandled write access at 0x%" PRIx64 "\n",
+                      __func__, offset0);
+    }
+    trace_tegra241_cmdqv_write_vcmdq_page0(index, offset0, value);
+}
+
+/*
+ * Write a VCMDQ Page 1 register (base / DRAM address) - 4-byte access.
+ */
+static void tegra241_cmdqv_write_vcmdq_page1(Tegra241CMDQV

*cmdqv,

hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv, · line 184

Quoted 13 lines
hw/arm/tegra241-cmdqv.c +13 -0
+                                             hwaddr offset0, int index,
+                                             uint32_t value)
+{
+    switch (offset0) {
+    case A_VCMDQ0_BASE_L:
+        if (tegra241_vcmdq_enabled(cmdqv, index)) {
+            return;
+        }
+        cmdqv->vcmdq_base[index] =
+            deposit64(cmdqv->vcmdq_base[index], 0, 32, value);
+        break;
+    case A_VCMDQ0_BASE_H:
+        if (tegra241_vcmdq_enabled(cmdqv, index)) {
Quoted 4 lines

where is it documented in the spec? I don't see this restriction in the reg doc. Then "Enabling the virtual CMDQ" paragraph describes a std setup but does not say - afaics- that it should be ignored - worth commenting

Right. I don’t think it is explicitly there in the spec.

However, Nicolin mentioned about such a restriction during v4 discussion:
https://lore.kernel.org/qemu-devel/[email protected]/

AFAICS, it looks like an implied assumption based on the "Enabling the virtual CMDQ" sequence or I missed it.

I will double check.

Thanks,
Shameer

#29 lore raw

Re: [PATCH v5 16/32] hw/arm/tegra241-cmdqv: Emulate VCMDQ register writes

Quoted 108 lines · Shameer Kolothum Thodi
Quoted 92 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 20 May 2026 08:54
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 16/32] hw/arm/tegra241-cmdqv: Emulate VCMDQ register writes

External email: Use caution opening links or attachments

Hi Shameer,

Quoted 11 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

This is the write side counterpart of the VCMDQ read emulation. Add write handling for both the direct VCMDQ aperture and the VINTF logical aperture using the same index decoding and VINTF-to-VCMDQ translation logic as the read path.

VINTF aperture writes are translated to their direct-aperture equivalent and update the same cached state. Page 1 registers (BASE, CONS_INDX_BASE) always update the cache. Once IOMMU_HW_QUEUE_ALLOC and viommu_mmap are wired up in a

subsequent

Quoted 10 lines

patch, Page 0 register writes will be forwarded to the hardware- backed mmap'd page.

Ignore VCMDQ BASE writes if the VCMDQ is already enabled.

---
 hw/arm/tegra241-cmdqv.c | 164

+++++++++++++++++++++++++++++++++++++++-

hw/arm/tegra241-cmdqv.c static uint64_t

Quoted 25 lines
 hw/arm/trace-events     |   2 +
 2 files changed, 164 insertions(+), 2 deletions(-)
hw/arm/tegra241-cmdqv.c +10 -0
index f859126ad6..5d2d9d2e26 100644
@@ -16,6 +16,16 @@
 #include "tegra241-cmdqv.h"
 #include "trace.h"
 
+/*
+ * Returns true if the per-VCMDQ CMDQ_EN_OK bit is set.
+ */
+static bool tegra241_vcmdq_enabled(Tegra241CMDQV *cmdqv, int index)
+{
+    uint32_t status = cmdqv->vcmdq_status[index];
+
+    return status & R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;
+}
+
 /*
  * Read a VCMDQ Page 0 register (control/status) using VCMDQ0_* offsets.
  *
@@ -116,6 +126,108 @@ static uint64_t

tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv,

hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv,

Quoted 13 lines
hw/arm/tegra241-cmdqv.c +10 -0
     }
 }
 
+/*
+ * Write a VCMDQ Page 0 register (control/status) using VCMDQ0_* offsets.
+ *
+ * The caller normalizes the MMIO offset such that @offset0 always refers
+ * to a VCMDQ0_* register, while @index selects the VCMDQ instance.
+ *
+ * Page 0 registers are all 32-bit; this helper is only called for 4-byte
+ * writes.
+ */
+static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV

*cmdqv,

hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv, · line 147

Quoted 9 lines
hw/arm/tegra241-cmdqv.c +9 -0
+                                             hwaddr offset0, int index,
+                                             uint32_t value)
+{
+    switch (offset0) {
+    case A_VCMDQ0_CONS_INDX:
+        cmdqv->vcmdq_cons_indx[index] = value;
+        break;
+    case A_VCMDQ0_PROD_INDX:
+        cmdqv->vcmdq_prod_indx[index] = value;
Quoted 5 lines

so there is no cmd consumption triggered by an increment of the producer index as opposed to std smmu emulation. This shall at least be explained in the commit msg and maybe in the code, + the justification. It is a bit strange to emulate the MMIO accesses while not doing anything functional out of them.

There is indeed a difference here w.r.t. std SMMU emulation. CMDQV
is a hardware-accelerated feature, and there isn't much value in emulating
the queue functionality without the HW backing.

From Spec, p.176:

"While the software can program the Virtual CMDQ(s) directly using the direct VCMDQ aperture (and not through the Virtual Interface), it is required that the VCMDQ be allocated to a Virtual Interface before it is used to send commands to the SMMU"

Quoted 4 lines

So no actual command processing can happen until the VCMDQ is allocated to a Virtual Interface. Until then, reads/writes only update the register cache.

I will make this clear in the commit log.

Yes this helps

Quoted 14 lines
Quoted 14 lines

hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv,

Quoted 5 lines
hw/arm/tegra241-cmdqv.c +4 -0
+        break;
+    case A_VCMDQ0_CONFIG:
+        if (value & R_VCMDQ0_CONFIG_CMDQ_EN_MASK) {
+            cmdqv->vcmdq_status[index] |=

R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;

hw/arm/tegra241-cmdqv.c static uint64_t · line 153

Quoted 2 lines
hw/arm/tegra241-cmdqv.c +2 -0
+        } else {
+            cmdqv->vcmdq_status[index] &=

~R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;

Quoted 5 lines
hw/arm/tegra241-cmdqv.c +5 -0
+        }
+        cmdqv->vcmdq_config[index] = value;
+        break;
+    case A_VCMDQ0_GERRORN:
+        cmdqv->vcmdq_gerrorn[index] = value;
Quoted 6 lines
Quoted 3 lines

shouldn't it do something functional besides storing the value on std SMMU we resume consuming cmds. interaction with interrupts, CONS_INDX?

Same rationale as above. In the pre-allocation state there is no real queue function to drive, so there is nothing functional for QEMU to do on a GERRORN write.

also I would add a comment in the code

Quoted 34 lines
Quoted 28 lines
Quoted 4 lines
hw/arm/tegra241-cmdqv.c +3 -0
+        break;
+    default:
+        qemu_log_mask(LOG_UNIMP,

hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv,

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10 -0
+                      "%s unhandled write access at 0x%" PRIx64 "\n",
+                      __func__, offset0);
+    }
+    trace_tegra241_cmdqv_write_vcmdq_page0(index, offset0, value);
+}
+
+/*
+ * Write a VCMDQ Page 1 register (base / DRAM address) - 4-byte access.
+ */
+static void tegra241_cmdqv_write_vcmdq_page1(Tegra241CMDQV

*cmdqv,

hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv, · line 184

Quoted 13 lines
hw/arm/tegra241-cmdqv.c +13 -0
+                                             hwaddr offset0, int index,
+                                             uint32_t value)
+{
+    switch (offset0) {
+    case A_VCMDQ0_BASE_L:
+        if (tegra241_vcmdq_enabled(cmdqv, index)) {
+            return;
+        }
+        cmdqv->vcmdq_base[index] =
+            deposit64(cmdqv->vcmdq_base[index], 0, 32, value);
+        break;
+    case A_VCMDQ0_BASE_H:
+        if (tegra241_vcmdq_enabled(cmdqv, index)) {
Quoted 4 lines

where is it documented in the spec? I don't see this restriction in the reg doc. Then "Enabling the virtual CMDQ" paragraph describes a std setup but does not say - afaics- that it should be ignored - worth commenting

Right. I don’t think it is explicitly there in the spec.

Quoted 7 lines

However, Nicolin mentioned about such a restriction during v4 discussion:
https://lore.kernel.org/qemu-devel/[email protected]/

AFAICS, it looks like an implied assumption based on the "Enabling the virtual CMDQ" sequence or I missed it.

I will double check.

OK

Thanks

Eric

Quoted 3 lines

Thanks,
Shameer

#30 lore raw

RE: [PATCH v5 16/32] hw/arm/tegra241-cmdqv: Emulate VCMDQ register writes

Quoted 14 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 20 May 2026 13:08
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 16/32] hw/arm/tegra241-cmdqv: Emulate VCMDQ register writes

[...]

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+    case A_VCMDQ0_BASE_H:
+        if (tegra241_vcmdq_enabled(cmdqv, index)) {

where is it documented in the spec? I don't see this restriction in the reg doc. Then "Enabling the virtual CMDQ" paragraph describes a std setup but does not say - afaics- that it should be ignored - worth commenting

Right. I don’t think it is explicitly there in the spec.

Quoted 8 lines
Quoted 7 lines

However, Nicolin mentioned about such a restriction during v4 discussion:
https://lore.kernel.org/qemu-devel/[email protected]/

AFAICS, it looks like an implied assumption based on the "Enabling the virtual CMDQ" sequence or I missed it.

I will double check.

OK

I did a hack to the host kernel driver to see how the HW behaves if we reprogram the BASE after the VCMDQ is enabled:

+ /* HACK: re-write BASE after VCMDQ_EN */
+ {
+ u64 orig = vcmdq->cmdq.q.q_base;
+

+               printk("%s: Shameer: re writing base after vcmdq enable..\n", __func__);
+               writeq_relaxed(0, REG_VCMDQ_PAGE1(vcmdq, BASE));
+               pr_info("Base : wrote 0, readback=0x%llx GERROR=0x%x STATUS=0x%x\n",
+                       readq_relaxed(REG_VCMDQ_PAGE1(vcmdq, BASE)),
+                       readl_relaxed(REG_VCMDQ_PAGE0(vcmdq, GERROR)),
+                       readl_relaxed(REG_VCMDQ_PAGE0(vcmdq, STATUS)));
+
+               /* Put it back before something tries to use it */
+               writeq_relaxed(orig, REG_VCMDQ_PAGE1(vcmdq, BASE));
+               pr_info("base: restored, readback=0x%llx\n",

+ readq_relaxed(REG_VCMDQ_PAGE1(vcmdq, BASE)));
+ }
+

From the boot logs, the HW accepted the BASE rewrite:

  [    1.504735] tegra241_vcmdq_hw_init: Shameer: re writing base after vcmdq enable..
  [    1.504737] Base : wrote 0, readback=0x0 GERROR=0x0 STATUS=0x1
  [    1.504739] base: restored, readback=0xfa000013

So writes to BASE after enable are not ignored at the register level. But I don't think command processing will work if we change BASE on a live queue.

From the QEMU side, the current setup_vcmdq() frees the existing hw_queue before calling IOMMU_HW_QUEUE_ALLOC again, otherwise the kernel returns -EEXIST. So if we allow BASE rewrite while the VCMDQ is enabled, it will trigger a destroy+alloc cycle in the host driver.

I tried removing the tegra241_vcmdq_enabled(cmdqv, index) restriction and rewriting BASE triggering a setup_vcmdq() again, and the guest ended up with:

arm-smmu-v3 arm-smmu-v3.0.auto: CMD_SYNC timeout at 0x00000023 [hwprod 0x00000023, hwcons 0x00000000] arm-smmu-v3 arm-smmu-v3.0.auto: CMD_SYNC timeout at 0x00000025 [hwprod 0x00000025, hwcons 0x00000000]

I think it is probably better to keep this behaviour (let the rewrite go through and surface as a CMD_SYNC timeout) than to silently ignore the BASE reprogram.

@Nicolin, thoughts?

Thanks,
Shameer

#31 lore raw

Re: [PATCH v5 16/32] hw/arm/tegra241-cmdqv: Emulate VCMDQ register writes

Quoted 40 lines · Shameer Kolothum Thodi

I did a hack to the host kernel driver to see how the HW behaves if we reprogram the BASE after the VCMDQ is enabled:

(quoted diff) +17 -0
+       /* HACK: re-write BASE after VCMDQ_EN */
+       {
+               u64 orig = vcmdq->cmdq.q.q_base;
+
+               printk("%s: Shameer: re writing base after vcmdq enable..\n", __func__);
+               writeq_relaxed(0, REG_VCMDQ_PAGE1(vcmdq, BASE));
+               pr_info("Base : wrote 0, readback=0x%llx GERROR=0x%x STATUS=0x%x\n",
+                       readq_relaxed(REG_VCMDQ_PAGE1(vcmdq, BASE)),
+                       readl_relaxed(REG_VCMDQ_PAGE0(vcmdq, GERROR)),
+                       readl_relaxed(REG_VCMDQ_PAGE0(vcmdq, STATUS)));
+
+               /* Put it back before something tries to use it */
+               writeq_relaxed(orig, REG_VCMDQ_PAGE1(vcmdq, BASE));
+               pr_info("base: restored, readback=0x%llx\n",
+                       readq_relaxed(REG_VCMDQ_PAGE1(vcmdq, BASE)));
+       }
+

From the boot logs, the HW accepted the BASE rewrite:

  [    1.504735] tegra241_vcmdq_hw_init: Shameer: re writing base after vcmdq enable..
  [    1.504737] Base : wrote 0, readback=0x0 GERROR=0x0 STATUS=0x1
  [    1.504739] base: restored, readback=0xfa000013

So writes to BASE after enable are not ignored at the register level. But I don't think command processing will work if we change BASE on a live queue.

From the QEMU side, the current setup_vcmdq() frees the existing hw_queue before calling IOMMU_HW_QUEUE_ALLOC again, otherwise the kernel returns -EEXIST. So if we allow BASE rewrite while the VCMDQ is enabled, it will trigger a destroy+alloc cycle in the host driver.

I tried removing the tegra241_vcmdq_enabled(cmdqv, index) restriction and rewriting BASE triggering a setup_vcmdq() again, and the guest ended up with:

Quoted 8 lines

arm-smmu-v3 arm-smmu-v3.0.auto: CMD_SYNC timeout at 0x00000023 [hwprod 0x00000023, hwcons 0x00000000] arm-smmu-v3 arm-smmu-v3.0.auto: CMD_SYNC timeout at 0x00000025 [hwprod 0x00000025, hwcons 0x00000000]

I think it is probably better to keep this behaviour (let the rewrite go through and surface as a CMD_SYNC timeout) than to silently ignore the BASE reprogram.

@Nicolin, thoughts?

You're asking about guest writing the direct MMIO space (trapped):
VCMDQ_EN and then BASE?

If so, yes.

Nicolin

#32 lore raw

PATCH v5 17/32 hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV

In reply to Shameer Kolothum

From: Nicolin Chen <[email protected]>

The CMDQ-V CMDQ pages provide a VM wide view of all VCMDQs, while the VINTF pages expose a logical view local to a given VINTF. Although real hardware may support multiple VINTFs, the kernel currently exposes a single VINTF per VM.

The kernel provides an mmap offset for the VINTF Page0 region during vIOMMU allocation. However, the logical-to-physical association between VCMDQs and a VINTF is only established after HW_QUEUE allocation. Prior to that, the mapped Page0 does not back any real VCMDQ state.

When VINTF is enabled, mmap the kernel provided Page0 region and set ENABLE_OK only if the mmap succeeds. Unmap it when VINTF is disabled. This prepares the VINTF mapping in advance of subsequent patches that add VCMDQ allocation support.

---
 hw/arm/tegra241-cmdqv.h |  3 +++
 hw/arm/tegra241-cmdqv.c | 50 ++++++++++++++++++++++++++++++++++++++---
 2 files changed, 50 insertions(+), 3 deletions(-)
hw/arm/tegra241-cmdqv.h +3 -0
index 6b21a407af..f0b031b4dc 100644
@@ -37,12 +37,15 @@
 #define CMDQV_VINTF_PAGE1_BASE  0x40000
 #define CMDQV_VCMDQ_STRIDE      0x80
 
+#define VINTF_PAGE_SIZE 0x10000
+
 typedef struct Tegra241CMDQV {
     struct iommu_viommu_tegra241_cmdqv cmdqv_data;
     SMMUv3AccelState *s_accel;
     MemoryRegion mmio_cmdqv;
     qemu_irq irq;
     IOMMUFDVeventq *veventq;
+    void *vintf_page0;
 
     /* CMDQ-V Config page register cache */
     uint32_t config;
hw/arm/tegra241-cmdqv.c +47 -3

88 lines not on this page · raw message

Signature · 1 line
2.43.0
#33 lore raw

Re: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV

Hi Shameer,

Quoted 3 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

The CMDQ-V CMDQ pages provide a VM wide view of all VCMDQs, while the

s/CMDQ pages/VCMDQ pages - yeah that's all very confusing ;-) -

Quoted 5 lines

VINTF pages expose a logical view local to a given VINTF. Although real hardware may support multiple VINTFs, the kernel currently exposes a single VINTF per VM.

The kernel provides an mmap offset for the VINTF Page0 region during

for the host VINTF page0 allocated for the VM

Actually is the whole host passthrough principle is not really explained anywhere. At least that's my feeling. It would be nice to have a summary for it, in the coverletter and in individual patch. Or maybe we can link to another doc. Reading the kernel uapi does not really provide the full picture, at least that's my own feeling.

when a VM allocates a VIOMMU for the CMDQV, the kernel allocates this and this resource that is then given for exclusive usage of the VM ... How does it guarantee security between VMs, etc...

Furthermore, what is not straightforward in this description is the sequence that will be executed
- viommu_alloc done on which trap event?
- hw_queue_alloc on which trap event?
- mmap on guest VINT enable

Quoted 4 lines

vIOMMU allocation. However, the logical-to-physical association between VCMDQs and a VINTF is only established after HW_QUEUE allocation. Prior to that, the mapped Page0 does not back any real VCMDQ state.

Quoted 8 lines

When VINTF is enabled, mmap the kernel provided Page0 region and set ENABLE_OK only if the mmap succeeds. Unmap it when VINTF is disabled. This prepares the VINTF mapping in advance of subsequent patches that add VCMDQ allocation support.

Thanks

Eric

Quoted 123 lines
---
 hw/arm/tegra241-cmdqv.h |  3 +++
 hw/arm/tegra241-cmdqv.c | 50 ++++++++++++++++++++++++++++++++++++++---
 2 files changed, 50 insertions(+), 3 deletions(-)
hw/arm/tegra241-cmdqv.h +3 -0
index 6b21a407af..f0b031b4dc 100644
@@ -37,12 +37,15 @@
 #define CMDQV_VINTF_PAGE1_BASE  0x40000
 #define CMDQV_VCMDQ_STRIDE      0x80
 
+#define VINTF_PAGE_SIZE 0x10000
+
 typedef struct Tegra241CMDQV {
     struct iommu_viommu_tegra241_cmdqv cmdqv_data;
     SMMUv3AccelState *s_accel;
     MemoryRegion mmio_cmdqv;
     qemu_irq irq;
     IOMMUFDVeventq *veventq;
+    void *vintf_page0;
 
     /* CMDQ-V Config page register cache */
     uint32_t config;
hw/arm/tegra241-cmdqv.c +47 -3
index 5d2d9d2e26..4d9f094b2a 100644
@@ -9,6 +9,8 @@
 
 #include "qemu/osdep.h"
 #include "qemu/log.h"
+#include "qemu/error-report.h"
+#include "qapi/error.h"
 
 #include "hw/arm/smmuv3.h"
 #include "hw/arm/smmuv3-common.h"
@@ -228,8 +230,42 @@ static void tegra241_cmdqv_write_vcmdq_page1_64(Tegra241CMDQV *cmdqv,
     trace_tegra241_cmdqv_write_vcmdq_page1(index, offset0, value);
 }
 
+static bool
+tegra241_cmdqv_munmap_vintf_page0(Tegra241CMDQV *cmdqv, Error **errp)
+{
+    if (!cmdqv->vintf_page0) {
+        return true;
+    }
+
+    if (munmap(cmdqv->vintf_page0, VINTF_PAGE_SIZE) < 0) {
+        error_setg_errno(errp, errno, "Failed to unmap VINTF page0");
+        return false;
+    }
+    cmdqv->vintf_page0 = NULL;
+    return true;
+}
+
+static bool tegra241_cmdqv_mmap_vintf_page0(Tegra241CMDQV *cmdqv, Error **errp)
+{
+    IOMMUFDViommu *viommu = cmdqv->s_accel->viommu;
+
+    if (cmdqv->vintf_page0) {
+        return true;
+    }
+
+    if (!iommufd_backend_viommu_mmap(viommu->iommufd, viommu->viommu_id,
+                                     VINTF_PAGE_SIZE,
+                                     cmdqv->cmdqv_data.out_vintf_mmap_offset,
+                                     &cmdqv->vintf_page0, errp)) {
+        return false;
+    }
+
+    return true;
+}
+
 static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv,
-                                              hwaddr offset, uint64_t value)
+                                              hwaddr offset, uint64_t value,
+                                              Error **errp)
 {
     int i;
 
@@ -244,8 +280,11 @@ static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv,
 
         cmdqv->vintf_config = value;
         if (value & R_VINTF0_CONFIG_ENABLE_MASK) {
-            cmdqv->vintf_status |= R_VINTF0_STATUS_ENABLE_OK_MASK;
+            if (tegra241_cmdqv_mmap_vintf_page0(cmdqv, errp)) {
+                cmdqv->vintf_status |= R_VINTF0_STATUS_ENABLE_OK_MASK;
+            }
         } else {
+            tegra241_cmdqv_munmap_vintf_page0(cmdqv, errp);
             cmdqv->vintf_status &= ~R_VINTF0_STATUS_ENABLE_OK_MASK;
         }
         break;
@@ -347,6 +386,7 @@ out:
 static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,
                                        uint32_t value)
 {
+    Error *local_err = NULL;
     int index;
 
     switch (offset) {
@@ -365,7 +405,7 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,
         cmdqv->cmdq_alloc_map[(offset - A_CMDQ_ALLOC_MAP_0) / 4] = value;
         break;
     case A_VINTF0_CONFIG ... A_VINTF0_LVCMDQ_ERR_MAP_3:
-        tegra241_cmdqv_config_vintf_write(cmdqv, offset, value);
+        tegra241_cmdqv_config_vintf_write(cmdqv, offset, value, &local_err);
         break;
     case A_VI_VCMDQ0_CONS_INDX ... A_VI_VCMDQ1_GERRORN:
         /*
@@ -398,6 +438,10 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,
         qemu_log_mask(LOG_UNIMP, "%s unhandled write access at 0x%" PRIx64 "\n",
                       __func__, offset);
     }
+
+    if (local_err) {
+        error_report_err(local_err);
+    }
 }
 
 /*
#34 lore raw

RE: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV

Hi Eric,

Quoted 27 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 20 May 2026 13:54
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV

External email: Use caution opening links or attachments

Hi Shameer,

Quoted 3 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

The CMDQ-V CMDQ pages provide a VM wide view of all VCMDQs, while the

s/CMDQ pages/VCMDQ pages - yeah that's all very confusing ;-) -

Quoted 4 lines

VINTF pages expose a logical view local to a given VINTF. Although real hardware may support multiple VINTFs, the kernel currently exposes a single VINTF per VM.

Quoted 8 lines
Quoted 1 line

The kernel provides an mmap offset for the VINTF Page0 region during

for the host VINTF page0 allocated for the VM

Actually is the whole host passthrough principle is not really explained anywhere. At least that's my feeling. It would be nice to have a summary for it, in the coverletter and in individual patch. Or maybe we can link to another doc. Reading the kernel uapi does not really provide the full picture, at least that's my own feeling.

Fair point. A summary of operation is useful. I am thinking of adding it at the top of hw/arm/tegra241-cmdqv.c:

/*
* Tegra241 CMDQV - overview
* ---------------------------------------
*...
*/

I will populate the details and share it for review before v6.

Quoted 4 lines

when a VM allocates a VIOMMU for the CMDQV, the kernel allocates this and this resource that is then given for exclusive usage of the VM ... How does it guarantee security between VMs, etc...

Isolation is enforced by the kernel and the CMDQV hardware. I think the key points here are:

- Each VM has its own iommufd FD. All iommufd objects (VINTF, hw queues, mmap regions) belong to that FD; one VM cannot reach another VM's IDs.

- Each FD gets its own VINTF, allocated by IOMMU_VIOMMU_ALLOC. The kernel writes the VM's VMID into the VINTF and the host CMDQV hardware checks it on every command.

- The kernel allocates the VINTF with HYP_OWN = 0, which restricts the guest to a safe subset of commands.

- Per VINTF, the kernel programs SID_MATCH and SID_REPLACE so only the VM's own vfio-pci devices can issue commands; other SIDs are rejected by the hardware.

- IOMMU_HW_QUEUE_ALLOC binds each VCMDQ to one specific VINTF, so a guest cannot reach a VCMDQ that belongs to another VM.

Quoted 8 lines

Furthermore, what is not straightforward in this description is the sequence that will be executed
- viommu_alloc done on which trap event?
- hw_queue_alloc on which trap event?
- mmap on guest VINT enable

Quoted 3 lines

vIOMMU allocation. However, the logical-to-physical association between VCMDQs and a VINTF is only established after HW_QUEUE allocation. Prior to that, the mapped Page0 does not back any real VCMDQ state.

Ok. Will capture the sequence of operations and the surrounding behaviour in the same top of file comment in hw/arm/tegra241-cmdqv.c.

Thanks,
Shameer

#35 lore raw

Re: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV

Hi Shameer,

Quoted 54 lines · Shameer Kolothum Thodi

Hi Eric,

Quoted 27 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 20 May 2026 13:54
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV

External email: Use caution opening links or attachments

Hi Shameer,

Quoted 3 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

The CMDQ-V CMDQ pages provide a VM wide view of all VCMDQs, while the

s/CMDQ pages/VCMDQ pages - yeah that's all very confusing ;-) -

Quoted 4 lines

VINTF pages expose a logical view local to a given VINTF. Although real hardware may support multiple VINTFs, the kernel currently exposes a single VINTF per VM.

Quoted 8 lines
Quoted 1 line

The kernel provides an mmap offset for the VINTF Page0 region during

for the host VINTF page0 allocated for the VM

Actually is the whole host passthrough principle is not really explained anywhere. At least that's my feeling. It would be nice to have a summary for it, in the coverletter and in individual patch. Or maybe we can link to another doc. Reading the kernel uapi does not really provide the full picture, at least that's my own feeling.

Fair point. A summary of operation is useful. I am thinking of adding it at the top of hw/arm/tegra241-cmdqv.c:

/*
* Tegra241 CMDQV - overview
* ---------------------------------------
*...
*/

I will populate the details and share it for review before v6.

Quoted 3 lines

when a VM allocates a VIOMMU for the CMDQV, the kernel allocates this and this resource that is then given for exclusive usage of the VM ... How does it guarantee security between VMs, etc...

Isolation is enforced by the kernel and the CMDQV hardware. I think the key points here are:

Quoted 6 lines

- Each VM has its own iommufd FD. All iommufd objects (VINTF, hw queues, mmap regions) belong to that FD; one VM cannot reach another VM's IDs.

- Each FD gets its own VINTF, allocated by IOMMU_VIOMMU_ALLOC. The kernel writes the VM's VMID into the VINTF and the host CMDQV

s/writes/configures

Quoted 11 lines

hardware checks it on every command.

- The kernel allocates the VINTF with HYP_OWN = 0, which restricts the guest to a safe subset of commands.

- Per VINTF, the kernel programs SID_MATCH and SID_REPLACE so only the VM's own vfio-pci devices can issue commands; other SIDs are rejected by the hardware.

- IOMMU_HW_QUEUE_ALLOC binds each VCMDQ to one specific VINTF, so a guest cannot reach a VCMDQ that belongs to another VM.

at the moment a single VCMDQ is used by VM, is that a correct understanding?

Yes this general principle explanation sound very useful to me to help the reader understand the passthrough principle.

Quoted 6 lines
Quoted 6 lines

Furthermore, what is not straightforward in this description is the sequence that will be executed
- viommu_alloc done on which trap event?
- hw_queue_alloc on which trap event?
- mmap on guest VINT enable

Quoted 9 lines
Quoted 3 lines
Quoted 3 lines

vIOMMU allocation. However, the logical-to-physical association between VCMDQs and a VINTF is only established after HW_QUEUE allocation. Prior to that, the mapped Page0 does not back any real VCMDQ state.

Ok. Will capture the sequence of operations and the surrounding behaviour in the same top of file comment in hw/arm/tegra241-cmdqv.c.

Thanks,
Shameer

Thanks

Eric

#36 lore raw

RE: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV

Quoted 93 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 21 May 2026 09:00
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV

External email: Use caution opening links or attachments

Hi Shameer,

Quoted 13 lines · Shameer Kolothum Thodi

Hi Eric,

Quoted 11 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 20 May 2026 13:54
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF

Page0

Quoted 42 lines
Quoted 17 lines

for CMDQV

External email: Use caution opening links or attachments

Hi Shameer,

Quoted 4 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

The CMDQ-V CMDQ pages provide a VM wide view of all VCMDQs, while the

s/CMDQ pages/VCMDQ pages - yeah that's all very confusing ;-) -

Quoted 4 lines

VINTF pages expose a logical view local to a given VINTF. Although real hardware may support multiple VINTFs, the kernel currently exposes a single VINTF per VM.

Quoted 8 lines
Quoted 1 line

The kernel provides an mmap offset for the VINTF Page0 region during

for the host VINTF page0 allocated for the VM

Actually is the whole host passthrough principle is not really explained anywhere. At least that's my feeling. It would be nice to have a summary for it, in the coverletter and in individual patch. Or maybe we can link to another doc. Reading the kernel uapi does not really provide the full picture, at least that's my own feeling.

Fair point. A summary of operation is useful. I am thinking of adding it at the top of hw/arm/tegra241-cmdqv.c:

/*
* Tegra241 CMDQV - overview
* ---------------------------------------
*...
*/

I will populate the details and share it for review before v6.

Quoted 3 lines

when a VM allocates a VIOMMU for the CMDQV, the kernel allocates this and this resource that is then given for exclusive usage of the VM ... How does it guarantee security between VMs, etc...

Isolation is enforced by the kernel and the CMDQV hardware. I think the key points here are:

Quoted 6 lines

- Each VM has its own iommufd FD. All iommufd objects (VINTF, hw queues, mmap regions) belong to that FD; one VM cannot reach another VM's IDs.

- Each FD gets its own VINTF, allocated by IOMMU_VIOMMU_ALLOC. The kernel writes the VM's VMID into the VINTF and the host CMDQV

s/writes/configures

Quoted 9 lines

hardware checks it on every command.

- The kernel allocates the VINTF with HYP_OWN = 0, which restricts the guest to a safe subset of commands.

- Per VINTF, the kernel programs SID_MATCH and SID_REPLACE so only the VM's own vfio-pci devices can issue commands; other SIDs are rejected by the hardware.

Quoted 5 lines
Quoted 1 line

- IOMMU_HW_QUEUE_ALLOC binds each VCMDQ to one specific VINTF, so

a

Quoted 1 line

guest cannot reach a VCMDQ that belongs to another VM.

at the moment a single VCMDQ is used by VM, is that a correct understanding?

Not quite. QEMU exposes one VINTF (VINTF0) with two VCMDQs to the guest. The guest decides how many of them to enable. A Linux guest enables both:
...
arm-smmu-v3 arm-smmu-v3.0.auto: allocated 524288 entries for vcmdq0 arm-smmu-v3 arm-smmu-v3.0.auto: allocated 524288 entries for vcmdq1
...

Thanks,
Shameer

#37 lore raw

Re: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV

Quoted 95 lines · Shameer Kolothum Thodi
Quoted 95 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 21 May 2026 09:00
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV

External email: Use caution opening links or attachments

Hi Shameer,

Quoted 13 lines · Shameer Kolothum Thodi

Hi Eric,

Quoted 11 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 20 May 2026 13:54
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF

Page0

Quoted 42 lines
Quoted 17 lines

for CMDQV

External email: Use caution opening links or attachments

Hi Shameer,

Quoted 4 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

The CMDQ-V CMDQ pages provide a VM wide view of all VCMDQs, while the

s/CMDQ pages/VCMDQ pages - yeah that's all very confusing ;-) -

Quoted 4 lines

VINTF pages expose a logical view local to a given VINTF. Although real hardware may support multiple VINTFs, the kernel currently exposes a single VINTF per VM.

Quoted 8 lines
Quoted 1 line

The kernel provides an mmap offset for the VINTF Page0 region during

for the host VINTF page0 allocated for the VM

Actually is the whole host passthrough principle is not really explained anywhere. At least that's my feeling. It would be nice to have a summary for it, in the coverletter and in individual patch. Or maybe we can link to another doc. Reading the kernel uapi does not really provide the full picture, at least that's my own feeling.

Fair point. A summary of operation is useful. I am thinking of adding it at the top of hw/arm/tegra241-cmdqv.c:

/*
* Tegra241 CMDQV - overview
* ---------------------------------------
*...
*/

I will populate the details and share it for review before v6.

Quoted 3 lines

when a VM allocates a VIOMMU for the CMDQV, the kernel allocates this and this resource that is then given for exclusive usage of the VM ... How does it guarantee security between VMs, etc...

Isolation is enforced by the kernel and the CMDQV hardware. I think the key points here are:

Quoted 6 lines

- Each VM has its own iommufd FD. All iommufd objects (VINTF, hw queues, mmap regions) belong to that FD; one VM cannot reach another VM's IDs.

- Each FD gets its own VINTF, allocated by IOMMU_VIOMMU_ALLOC. The kernel writes the VM's VMID into the VINTF and the host CMDQV

s/writes/configures

Quoted 10 lines

hardware checks it on every command.

- The kernel allocates the VINTF with HYP_OWN = 0, which restricts the guest to a safe subset of commands.

- Per VINTF, the kernel programs SID_MATCH and SID_REPLACE so only the VM's own vfio-pci devices can issue commands; other SIDs are rejected by the hardware.

- IOMMU_HW_QUEUE_ALLOC binds each VCMDQ to one specific VINTF, so

a

Quoted 1 line

guest cannot reach a VCMDQ that belongs to another VM.

Quoted 8 lines
Quoted 2 lines

at the moment a single VCMDQ is used by VM, is that a correct understanding?

Not quite. QEMU exposes one VINTF (VINTF0) with two VCMDQs to the guest. The guest decides how many of them to enable. A Linux guest enables both:
...
arm-smmu-v3 arm-smmu-v3.0.auto: allocated 524288 entries for vcmdq0 arm-smmu-v3 arm-smmu-v3.0.auto: allocated 524288 entries for vcmdq1

understood!

Eric

Quoted 4 lines

...

Thanks,
Shameer

#38 lore raw

RE: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV

In reply to Shameer Kolothum Thodi
Quoted 10 lines
Quoted 6 lines · Shameer Kolothum

Actually is the whole host passthrough principle is not really explained anywhere. At least that's my feeling. It would be nice to have a summary for it, in the coverletter and in individual patch. Or maybe we can link to another doc. Reading the kernel uapi does not really provide the full picture, at least that's my own feeling.

Fair point. A summary of operation is useful. I am thinking of adding it at the top of hw/arm/tegra241-cmdqv.c:

Quoted 7 lines

/*
* Tegra241 CMDQV - overview
* ---------------------------------------
*...
*/

I will populate the details and share it for review before v6.

Please find below. Hopefully, I have captured all the important details. Please take a look and let me know.

Thanks,
Shameer

hw/arm/tegra241-cmdqv.c +78 -0
index ebf12d0597..5a103f37b8 100644
@@ -7,6 +7,84 @@
  * SPDX-License-Identifier: GPL-2.0-or-later
  */
 
+/*
+ * Tegra241 CMDQV - overview
+ * =========================
+ *
+ * NVIDIA Tegra241 extends SMMUv3 with a Command Queue Virtualization (CMDQ-V)
+ * block. It lets a guest issue SMMU invalidation commands directly to
+ * dedicated hardware queues (vCMDQs) without trapping into the hypervisor on
+ * the fast path. vCMDQs are grouped into Virtual Interfaces (VINTFs); the
+ * host kernel allocates one VINTF per emulated SMMUv3 instance via iommufd.
+ * QEMU emulates the CMDQV MMIO region and drives the host kernel calls
+ * (VIOMMU_ALLOC, HW_QUEUE_ALLOC, mmap); the actual command processing happens
+ * on real hardware.
+ *
+ * MMIO layout (64KB pages, total TEGRA241_CMDQV_IO_LEN)
+ * -----------------------------------------------------
+ * The direct vCMDQ apertures (0x10000/0x20000) are HW aliases of the VINTF
+ * apertures (0x30000/0x40000); they expose the same per-vCMDQ register slots
+ * under different addressing.
+ *
+ *   0x00000  CMDQV Config page: QEMU-trapped.
+ *   0x10000  Direct vCMDQ Page 0 (control/status): QEMU-trapped and routed
+ *            via vintf_ptr() to either the mmap'd VINTF page (allocated
+ *            slot) or a per-vCMDQ register cache (unallocated slot).
+ *   0x20000  Direct vCMDQ Page 1 (BASE / DRAM addresses): QEMU-trapped.
+ *   0x30000  VINTF Page 0 (per-VINTF control/status): mmap'd from the host
+ *            via iommufd and installed into guest MMIO as a RAM-device
+ *            subregion after the first HW_QUEUE_ALLOC; subsequent accesses
+ *            bypass QEMU.
+ *   0x40000  VINTF Page 1 (per-VINTF BASE): QEMU-trapped.
+ *
+ * The direct vCMDQ aperture stays trapped (rather than aliased to the VINTF
+ * mmap) to preserve the spec's R/W register semantics for unallocated
+ * vCMDQs: the direct aperture allows programming before VINTF allocation,
+ * while aliasing would route through the VINTF drop path instead.
+ *
+ * Lifecycle (driven by guest events)
+ * ----------------------------------
+ * 1. First vfio-pci device attach (.set_iommu_device) triggers:
+ *    - tegra241_cmdqv_probe(): IOMMU_GET_HW_INFO confirms host CMDQV support.
+ *    - IOMMU_VIOMMU_ALLOC: the kernel allocates a VINTF for this VM,
+ *      configures the VM's VMID (from its stage-2 HWPT) in VINTF_CONFIG,
+ *      forces HYP_OWN=0, and returns the mmap offset/length for VINTF Page 0.
+ *
+ * 2. Guest writes VINTF_CONFIG.ENABLE = 1:
+ *    QEMU mmap()s the offset from step 1 into its address space and reports
+ *    STATUS.ENABLE_OK = 1. The host VINTF was already enabled by
+ *    IOMMU_VIOMMU_ALLOC; QEMU only acks back.
+ *
+ * 3. Guest completes vCMDQ setup (BASE, CMDQ_ALLOC_MAP.ALLOC, CMDQV_EN,
+ *    VINTF.ENABLE, in any order; each precondition write retries the
+ *    allocation):
+ *    IOMMU_HW_QUEUE_ALLOC binds the guest BASE GPA (translated through
+ *    stage-2 and pinned by the kernel) to a host vCMDQ in this VM's VINTF.
+ *
+ * 4. After the first successful HW_QUEUE_ALLOC, the mmap'd VINTF Page 0 is
+ *    installed into guest MMIO as a RAM-device subregion. Guest vCMDQ Page 0
+ *    accesses (CMDQ_EN, PROD/CONS_INDX, STATUS, GERROR/GERRORN) thereafter
+ *    go straight to host hardware, bypassing QEMU.
+ *
+ * Per-VM isolation
+ * ----------------
+ * - Each VM has its own iommufd FD; all iommufd objects (VINTF, hw_queues,
+ *   mmap regions) belong to that FD. Cross-FD lookups fail, so one VM
+ *   cannot reach another VM's IDs.
+ * - IOMMU_VIOMMU_ALLOC configures the VM's VMID in VINTF_CONFIG; the CMDQV
+ *   hardware substitutes / checks VMID on every command the guest issues.
+ * - The kernel allocates the VINTF with HYP_OWN = 0, which restricts the
+ *   guest to a safe subset of commands.
+ * - Per VINTF, the kernel programs SID_MATCH and SID_REPLACE to restrict
+ *   invalidations to the StreamIDs assigned to this VM.
+ * - IOMMU_HW_QUEUE_ALLOC binds each vCMDQ to a single VINTF, so a guest
+ *   cannot reach a vCMDQ that belongs to another VM.
+ *
+ * Limits exposed to the guest
+ * ---------------------------
+ * One VINTF per emulated SMMUv3 and two vCMDQs per VINTF.
+ */
+
 #include "qemu/osdep.h"
 #include "qemu/log.h"
 #include "qemu/error-report.h"
#39 lore raw

Re: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV

Hi Shameer,

Quoted 51 lines · Shameer Kolothum Thodi
Quoted 9 lines
Quoted 6 lines · Shameer Kolothum

Actually is the whole host passthrough principle is not really explained anywhere. At least that's my feeling. It would be nice to have a summary for it, in the coverletter and in individual patch. Or maybe we can link to another doc. Reading the kernel uapi does not really provide the full picture, at least that's my own feeling.

Fair point. A summary of operation is useful. I am thinking of adding it at the top of hw/arm/tegra241-cmdqv.c:

Quoted 7 lines

/*
* Tegra241 CMDQV - overview
* ---------------------------------------
*...
*/

I will populate the details and share it for review before v6.

Please find below. Hopefully, I have captured all the important details. Please take a look and let me know.

Thanks,
Shameer

hw/arm/tegra241-cmdqv.c +21 -0
index ebf12d0597..5a103f37b8 100644
@@ -7,6 +7,84 @@
  * SPDX-License-Identifier: GPL-2.0-or-later
  */
 
+/*
+ * Tegra241 CMDQV - overview
+ * =========================
+ *
+ * NVIDIA Tegra241 extends SMMUv3 with a Command Queue Virtualization (CMDQ-V)
+ * block. It lets a guest issue SMMU invalidation commands directly to
+ * dedicated hardware queues (vCMDQs) without trapping into the hypervisor on
+ * the fast path. vCMDQs are grouped into Virtual Interfaces (VINTFs); the
+ * host kernel allocates one VINTF per emulated SMMUv3 instance via iommufd.
+ * QEMU emulates the CMDQV MMIO region and drives the host kernel calls
+ * (VIOMMU_ALLOC, HW_QUEUE_ALLOC, mmap); the actual command processing happens
+ * on real hardware.
+ *
+ * MMIO layout (64KB pages, total TEGRA241_CMDQV_IO_LEN)
+ * -----------------------------------------------------
+ * The direct vCMDQ apertures (0x10000/0x20000) are HW aliases of the VINTF
+ * apertures (0x30000/0x40000); they expose the same per-vCMDQ register slots
+ * under different addressing.
+ *
+ *   0x00000  CMDQV Config page: QEMU-trapped.
+ *   0x10000  Direct vCMDQ Page 0 (control/status): QEMU-trapped and routed

hw/arm/tegra241-cmdqv.c line 40

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10 -0
+ *            via vintf_ptr() to either the mmap'd VINTF page (allocated
+ *            slot) or a per-vCMDQ register cache (unallocated slot).
+ *   0x20000  Direct vCMDQ Page 1 (BASE / DRAM addresses): QEMU-trapped.
+ *   0x30000  VINTF Page 0 (per-VINTF control/status): mmap'd from the host
+ *            via iommufd and installed into guest MMIO as a RAM-device
+ *            subregion after the first HW_QUEUE_ALLOC; subsequent accesses
+ *            bypass QEMU.
+ *   0x40000  VINTF Page 1 (per-VINTF BASE): QEMU-trapped.
+ *
+ * The direct vCMDQ aperture stays trapped (rather than aliased to the VINTF

direct vCMDQ aperture page 0 stays trapped as opposed to the VIINTF page0

hw/arm/tegra241-cmdqv.c line 43

Quoted 3 lines
hw/arm/tegra241-cmdqv.c +3 -0
+ * mmap) to preserve the spec's R/W register semantics for unallocated
+ * vCMDQs: the direct aperture allows programming before VINTF allocation,
+ * while aliasing would route through the VINTF drop path instead.

see last discussion

hw/arm/tegra241-cmdqv.c line 51

Quoted 8 lines
hw/arm/tegra241-cmdqv.c +8 -0
+ *
+ * Lifecycle (driven by guest events)
+ * ----------------------------------
+ * 1. First vfio-pci device attach (.set_iommu_device) triggers:
+ *    - tegra241_cmdqv_probe(): IOMMU_GET_HW_INFO confirms host CMDQV support.
+ *    - IOMMU_VIOMMU_ALLOC: the kernel allocates a VINTF for this VM,
+ *      configures the VM's VMID (from its stage-2 HWPT) in VINTF_CONFIG,
+ *      forces HYP_OWN=0, and returns the mmap offset/length for VINTF Page 0.

what about the v/p SID mapping. How does the kernel know which SIDs are supposed to write into that VINTF? where do we pass this info?

Quoted 29 lines
hw/arm/tegra241-cmdqv.c +29 -0
+ *
+ * 2. Guest writes VINTF_CONFIG.ENABLE = 1:
+ *    QEMU mmap()s the offset from step 1 into its address space and reports
+ *    STATUS.ENABLE_OK = 1. The host VINTF was already enabled by
+ *    IOMMU_VIOMMU_ALLOC; QEMU only acks back.
+ *
+ * 3. Guest completes vCMDQ setup (BASE, CMDQ_ALLOC_MAP.ALLOC, CMDQV_EN,
+ *    VINTF.ENABLE, in any order; each precondition write retries the
+ *    allocation):
+ *    IOMMU_HW_QUEUE_ALLOC binds the guest BASE GPA (translated through
+ *    stage-2 and pinned by the kernel) to a host vCMDQ in this VM's VINTF.
+ *
+ * 4. After the first successful HW_QUEUE_ALLOC, the mmap'd VINTF Page 0 is
+ *    installed into guest MMIO as a RAM-device subregion. Guest vCMDQ Page 0
+ *    accesses (CMDQ_EN, PROD/CONS_INDX, STATUS, GERROR/GERRORN) thereafter
+ *    go straight to host hardware, bypassing QEMU.
+ *
+ * Per-VM isolation
+ * ----------------
+ * - Each VM has its own iommufd FD; all iommufd objects (VINTF, hw_queues,
+ *   mmap regions) belong to that FD. Cross-FD lookups fail, so one VM
+ *   cannot reach another VM's IDs.
+ * - IOMMU_VIOMMU_ALLOC configures the VM's VMID in VINTF_CONFIG; the CMDQV
+ *   hardware substitutes / checks VMID on every command the guest issues.
+ * - The kernel allocates the VINTF with HYP_OWN = 0, which restricts the
+ *   guest to a safe subset of commands.
+ * - Per VINTF, the kernel programs SID_MATCH and SID_REPLACE to restrict
+ *   invalidations to the StreamIDs assigned to this VM.
+ * - IOMMU_HW_QUEUE_ALLOC binds each vCMDQ to a single VINTF, so a guest

hw/arm/tegra241-cmdqv.c line 90

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +7 -0
+ *   cannot reach a vCMDQ that belongs to another VM.
+ *
+ * Limits exposed to the guest
+ * ---------------------------
+ * One VINTF per emulated SMMUv3 and two vCMDQs per VINTF.
+ */
+
 #include "qemu/osdep.h"
 #include "qemu/log.h"
 #include "qemu/error-report.h"

Eric

#40 lore raw

RE: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV

Quoted 92 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 26 May 2026 13:32
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV

External email: Use caution opening links or attachments

Hi Shameer,

Quoted 27 lines · Shameer Kolothum Thodi
Quoted 9 lines
Quoted 6 lines · Shameer Kolothum

Actually is the whole host passthrough principle is not really explained anywhere. At least that's my feeling. It would be nice to have a summary for it, in the coverletter and in individual patch. Or maybe we can link to another doc. Reading the kernel uapi does not really provide the full picture, at least that's my own feeling.

Fair point. A summary of operation is useful. I am thinking of adding it at the top of hw/arm/tegra241-cmdqv.c:

Quoted 7 lines

/*
* Tegra241 CMDQV - overview
* ---------------------------------------
*...
*/

I will populate the details and share it for review before v6.

Please find below. Hopefully, I have captured all the important details. Please take a look and let me know.

Thanks,
Shameer

hw/arm/tegra241-cmdqv.c +0 -0
index ebf12d0597..5a103f37b8 100644

hw/arm/tegra241-cmdqv.c line 14

Quoted 8 lines
hw/arm/tegra241-cmdqv.c +5 -0
@@ -7,6 +7,84 @@
  * SPDX-License-Identifier: GPL-2.0-or-later
  */
 
+/*
+ * Tegra241 CMDQV - overview
+ * =========================
+ *
+ * NVIDIA Tegra241 extends SMMUv3 with a Command Queue

Virtualization (CMDQ-V)

hw/arm/tegra241-cmdqv.c line 16

Quoted 2 lines
hw/arm/tegra241-cmdqv.c +2 -0
+ * block. It lets a guest issue SMMU invalidation commands directly to
+ * dedicated hardware queues (vCMDQs) without trapping into the

hypervisor on

hw/arm/tegra241-cmdqv.c line 18

Quoted 2 lines
hw/arm/tegra241-cmdqv.c +2 -0
+ * the fast path. vCMDQs are grouped into Virtual Interfaces (VINTFs); the
+ * host kernel allocates one VINTF per emulated SMMUv3 instance via

iommufd.

hw/arm/tegra241-cmdqv.c line 19

Quoted 1 line
hw/arm/tegra241-cmdqv.c +1 -0
+ * QEMU emulates the CMDQV MMIO region and drives the host kernel

calls

hw/arm/tegra241-cmdqv.c line 20

Quoted 1 line
hw/arm/tegra241-cmdqv.c +1 -0
+ * (VIOMMU_ALLOC, HW_QUEUE_ALLOC, mmap); the actual command

processing happens

hw/arm/tegra241-cmdqv.c

Quoted 5 lines
hw/arm/tegra241-cmdqv.c +5 -0
+ * on real hardware.
+ *
+ * MMIO layout (64KB pages, total TEGRA241_CMDQV_IO_LEN)
+ * -----------------------------------------------------
+ * The direct vCMDQ apertures (0x10000/0x20000) are HW aliases of the

VINTF

hw/arm/tegra241-cmdqv.c line 26

Quoted 1 line
hw/arm/tegra241-cmdqv.c +1 -0
+ * apertures (0x30000/0x40000); they expose the same per-vCMDQ

register slots

hw/arm/tegra241-cmdqv.c

Quoted 4 lines
hw/arm/tegra241-cmdqv.c +4 -0
+ * under different addressing.
+ *
+ *   0x00000  CMDQV Config page: QEMU-trapped.
+ *   0x10000  Direct vCMDQ Page 0 (control/status): QEMU-trapped and

routed

hw/arm/tegra241-cmdqv.c

Quoted 3 lines
hw/arm/tegra241-cmdqv.c +3 -0
+ *            via vintf_ptr() to either the mmap'd VINTF page (allocated
+ *            slot) or a per-vCMDQ register cache (unallocated slot).
+ *   0x20000  Direct vCMDQ Page 1 (BASE / DRAM addresses): QEMU-

trapped.

hw/arm/tegra241-cmdqv.c line 34

Quoted 1 line
hw/arm/tegra241-cmdqv.c +1 -0
+ *   0x30000  VINTF Page 0 (per-VINTF control/status): mmap'd from the

host

hw/arm/tegra241-cmdqv.c

Quoted 6 lines
hw/arm/tegra241-cmdqv.c +6 -0
+ *            via iommufd and installed into guest MMIO as a RAM-device
+ *            subregion after the first HW_QUEUE_ALLOC; subsequent accesses
+ *            bypass QEMU.
+ *   0x40000  VINTF Page 1 (per-VINTF BASE): QEMU-trapped.
+ *
+ * The direct vCMDQ aperture stays trapped (rather than aliased to the
Quoted 3 lines

VINTF

direct vCMDQ aperture page 0 stays trapped as opposed to the VIINTF page0

Ok.

Quoted 6 lines

hw/arm/tegra241-cmdqv.c line 42

Quoted 3 lines
hw/arm/tegra241-cmdqv.c +2 -0
+ * mmap) to preserve the spec's R/W register semantics for unallocated
+ * vCMDQs: the direct aperture allows programming before VINTF

allocation,

hw/arm/tegra241-cmdqv.c line 43

Quoted 1 line
hw/arm/tegra241-cmdqv.c +1 -0
+ * while aliasing would route through the VINTF drop path instead.

see last discussion

Replied there.

Quoted 5 lines

hw/arm/tegra241-cmdqv.c

Quoted 5 lines
hw/arm/tegra241-cmdqv.c +5 -0
+ *
+ * Lifecycle (driven by guest events)
+ * ----------------------------------
+ * 1. First vfio-pci device attach (.set_iommu_device) triggers:
+ *    - tegra241_cmdqv_probe(): IOMMU_GET_HW_INFO confirms host
Quoted 7 lines

CMDQV support.

hw/arm/tegra241-cmdqv.c

Quoted 3 lines
hw/arm/tegra241-cmdqv.c +3 -0
+ *    - IOMMU_VIOMMU_ALLOC: the kernel allocates a VINTF for this VM,
+ *      configures the VM's VMID (from its stage-2 HWPT) in VINTF_CONFIG,
+ *      forces HYP_OWN=0, and returns the mmap offset/length for VINTF

Page 0.
what about the v/p SID mapping. How does the kernel know which SIDs are supposed to write into that VINTF? where do we pass this info?

That happens later, driven by the guest SMMU driver, not at vfio-pci device attach.

Flow:
- Guest SMMU driver configures the Stream Table Entry (STE) for the device and issues CMD_CFGI_STE on the SMMU command queue.
- QEMU calls smmuv3_accel_install_ste(), which in turn calls iommufd_backend_alloc_vdev() -> IOMMU_VDEVICE_ALLOC ioctl.
- iommufd core (iommufd_vdevice_alloc_ioctl) dispatches to the Tegra CMDQV driver via viommu->ops->vdevice_init, which is tegra241_vintf_init_vsid().
- That writes the host stream id into SID_REPLACE and the guest virtual SID into SID_MATCH.

I will summarise and include this in the overview.

Thanks,
Shameer

#41 lore raw

PATCH v5 18/32 system/physmem: Add address_space_range_is_ram() helper

In reply to Shameer Kolothum

Introduce address_space_range_is_ram(), a helper to determine whether a guest physical address range within an AddressSpace resolves entirely to RAM-backed MemoryRegions. The range is walked using address_space_translate() so contiguous RAM-only ranges return true even when they cross multiple MemoryRegion boundaries; any byte backed by non-RAM (MMIO, ROM device, hole) returns false.

---
 include/system/memory.h | 13 +++++++++++++
 system/physmem.c        | 19 +++++++++++++++++++
 2 files changed, 32 insertions(+)
include/system/memory.h +13 -0
index 1417132f6d..af5f96f6ff 100644
@@ -2841,6 +2841,19 @@ bool address_space_access_valid(AddressSpace *as, hwaddr addr, hwaddr len,
  */
 bool address_space_is_io(AddressSpace *as, hwaddr addr);
 
+/**
+ * address_space_range_is_ram: check whether a guest physical address range
+ *                             within an address space resolves entirely to RAM.
+ *
+ * @as: #AddressSpace to be accessed
+ * @addr: address within that address space
+ * @size: number of bytes in the range starting at @addr
+ *
+ * Returns true if every byte in [@addr, @addr + @size) maps to a RAM
+ * MemoryRegion, false otherwise.
+ */
+bool address_space_range_is_ram(AddressSpace *as, hwaddr addr, hwaddr size);
+
 /* address_space_map: map a physical memory region into a host virtual address
  *
  * May map a subset of the requested range, given by and returned in @plen.
system/physmem.c +19 -0
index 7bcbf87573..46b36c7b10 100644
@@ -3668,6 +3668,25 @@ bool address_space_is_io(AddressSpace *as, hwaddr addr)
     return !(memory_region_is_ram(mr) || memory_region_is_romd(mr));
 }
 
+bool address_space_range_is_ram(AddressSpace *as, hwaddr addr, hwaddr size)
+{
+    MemoryRegion *mr;
+    hwaddr xlat, l;
+
+    RCU_READ_LOCK_GUARD();
+    while (size > 0) {
+        l = size;
+        mr = address_space_translate(as, addr, &xlat, &l, false,
+                                     MEMTXATTRS_UNSPECIFIED);
+        if (!memory_region_is_ram(mr)) {
+            return false;
+        }
+        size -= l;
+        addr += l;
+    }
+    return true;
+}
+
 static hwaddr
 flatview_extend_translation(FlatView *fv, hwaddr addr,
                             hwaddr target_len,
Signature · 1 line
2.43.0
#42 lore raw

PATCH v5 19/32 hw/arm/tegra241-cmdqv: Allocate HW VCMDQs once configured

In reply to Shameer Kolothum

From: Nicolin Chen <[email protected]>

Add support for allocating IOMMUFD hardware queues when the guest programs the VCMDQ BASE registers.

VCMDQ_EN lives in VCMDQ_CONFIG, which is on the VINTF Page0 region that a later patch maps directly into the guest — so QEMU won't trap its writes. Allocate the hardware queue instead once all of these are set: a RAM-backed BASE, CMDQ_ALLOC_MAP.ALLOC, and CMDQV / VINTF enabled. Each precondition write retries the allocation, so the guest may program them in any order.

If a hardware queue was previously allocated for the same VCMDQ, free it before reallocation.

Writes with invalid addresses are ignored.

All allocated VCMDQs are freed when CMDQV or VINTF is disabled, or when the ALLOC bit is cleared.

---
 hw/arm/tegra241-cmdqv.h |  11 ++++
 hw/arm/tegra241-cmdqv.c | 138 ++++++++++++++++++++++++++++++++++++++--
 2 files changed, 143 insertions(+), 6 deletions(-)
hw/arm/tegra241-cmdqv.h +11 -0
index f0b031b4dc..7a8cb2ebc7 100644
@@ -45,6 +45,7 @@ typedef struct Tegra241CMDQV {
     MemoryRegion mmio_cmdqv;
     qemu_irq irq;
     IOMMUFDVeventq *veventq;
+    IOMMUFDHWqueue *vcmdq[TEGRA241_CMDQV_MAX_CMDQ];
     void *vintf_page0;
 
     /* CMDQ-V Config page register cache */
@@ -361,6 +362,16 @@ SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_L_(1)
 SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_(0)
 SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_(1)
 
+static inline bool tegra241_cmdqv_enabled(Tegra241CMDQV *cmdqv)
+{
+    return cmdqv->status & R_STATUS_CMDQV_ENABLED_MASK;
+}
+
+static inline bool tegra241_vintf_enabled(Tegra241CMDQV *cmdqv)
+{
+    return cmdqv->vintf_status & R_VINTF0_STATUS_ENABLE_OK_MASK;
+}
+
 const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops(void);
 
 #endif /* HW_ARM_TEGRA241_CMDQV_H */
hw/arm/tegra241-cmdqv.c +132 -6

220 lines not on this page · raw message

Signature · 1 line
2.43.0
#43 lore raw

Re: [PATCH v5 19/32] hw/arm/tegra241-cmdqv: Allocate HW VCMDQs once configured

Hi,

Quoted 4 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

Add support for allocating IOMMUFD hardware queues when the guest programs the VCMDQ BASE registers.

You should add the fact that

iommufd_backend_alloc_hw_queue() requires the GPA of the VCMDQ. This explains why you need to those operations once the GPA has been populated.

Quoted 106 lines

VCMDQ_EN lives in VCMDQ_CONFIG, which is on the VINTF Page0 region that a later patch maps directly into the guest — so QEMU won't trap its writes. Allocate the hardware queue instead once all of these are set: a RAM-backed BASE, CMDQ_ALLOC_MAP.ALLOC, and CMDQV / VINTF enabled. Each precondition write retries the allocation, so the guest may program them in any order.
If a hardware queue was previously allocated for the same VCMDQ, free it before reallocation.

Writes with invalid addresses are ignored.

All allocated VCMDQs are freed when CMDQV or VINTF is disabled, or when the ALLOC bit is cleared.

---
 hw/arm/tegra241-cmdqv.h |  11 ++++
 hw/arm/tegra241-cmdqv.c | 138 ++++++++++++++++++++++++++++++++++++++--
 2 files changed, 143 insertions(+), 6 deletions(-)
hw/arm/tegra241-cmdqv.h +11 -0
index f0b031b4dc..7a8cb2ebc7 100644
@@ -45,6 +45,7 @@ typedef struct Tegra241CMDQV {
     MemoryRegion mmio_cmdqv;
     qemu_irq irq;
     IOMMUFDVeventq *veventq;
+    IOMMUFDHWqueue *vcmdq[TEGRA241_CMDQV_MAX_CMDQ];
     void *vintf_page0;
 
     /* CMDQ-V Config page register cache */
@@ -361,6 +362,16 @@ SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_L_(1)
 SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_(0)
 SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_(1)
 
+static inline bool tegra241_cmdqv_enabled(Tegra241CMDQV *cmdqv)
+{
+    return cmdqv->status & R_STATUS_CMDQV_ENABLED_MASK;
+}
+
+static inline bool tegra241_vintf_enabled(Tegra241CMDQV *cmdqv)
+{
+    return cmdqv->vintf_status & R_VINTF0_STATUS_ENABLE_OK_MASK;
+}
+
 const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops(void);
 
 #endif /* HW_ARM_TEGRA241_CMDQV_H */
hw/arm/tegra241-cmdqv.c +46 -0
index 4d9f094b2a..f4968520f3 100644
@@ -18,6 +18,95 @@
 #include "tegra241-cmdqv.h"
 #include "trace.h"
 
+static void tegra241_cmdqv_free_vcmdq(Tegra241CMDQV *cmdqv, int index)
+{
+    IOMMUFDViommu *viommu = cmdqv->s_accel->viommu;
+    IOMMUFDHWqueue *vcmdq = cmdqv->vcmdq[index];
+
+    if (!vcmdq) {
+        return;
+    }
+    iommufd_backend_free_id(viommu->iommufd, vcmdq->hw_queue_id);
+    g_free(vcmdq);
+    cmdqv->vcmdq[index] = NULL;
+}
+
+static void tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv)
+{
+    /* uapi/linux/iommufd.h: hw_queue destroy must be in descending @index. */
+    for (int i = (TEGRA241_CMDQV_MAX_CMDQ - 1); i >= 0; i--) {
+        tegra241_cmdqv_free_vcmdq(cmdqv, i);
+    }
+}
+
+/*
+ * Allocate a host HW VCMDQ from the current cached BASE / size for @index.
+ *
+ * Setup is a no-op (returns true) when any of the following preconditions
+ * isn't met yet:
+ *  - BASE not prgrammed yet.
+ *  - VCMDQ is not mapped to a VINTF (CMDQ_ALLOC_MAP.ALLOC == 0)
+ *  - BASE / size don't resolve to a RAM region
+ *  - CMDQV global enable or VINTF enable is not yet asserted
+ */
+static bool tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int index,
+                                       Error **errp)
+{
+    SMMUv3AccelState *accel = cmdqv->s_accel;
+    uint64_t base_mask = (uint64_t)R_VCMDQ0_BASE_L_ADDR_MASK |
+                         (uint64_t)R_VCMDQ0_BASE_H_ADDR_MASK << 32;
+    uint64_t addr = cmdqv->vcmdq_base[index] & base_mask;
+    uint64_t log2 = cmdqv->vcmdq_base[index] & R_VCMDQ0_BASE_L_LOG2SIZE_MASK;
+    uint64_t size = 1ULL << (log2 + 4);
+    IOMMUFDViommu *viommu = accel->viommu;
+    IOMMUFDHWqueue *hw_queue;
+    uint32_t hw_queue_id;
+
+    /* BASE not programmed yet. */
+    if (!cmdqv->vcmdq_base[index]) {

hw/arm/tegra241-cmdqv.c line 76

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10 -0
+        return true;
+    }
+
+    /* VCMDQ not yet mapped to a VINTF. */
+    if (!(cmdqv->cmdq_alloc_map[index] & R_CMDQ_ALLOC_MAP_0_ALLOC_MASK)) {
+        return true;
+    }
+
+    /* Ignore any invalid address. This may come as part of reset etc. */
+    if (!address_space_range_is_ram(&address_space_memory, addr, size)) {

why do you need to test the whole range and not only the start address

Quoted 18 lines
hw/arm/tegra241-cmdqv.c +18 -0
+        return true;
+    }
+
+    if (!tegra241_cmdqv_enabled(cmdqv) || !tegra241_vintf_enabled(cmdqv)) {
+        return true;
+    }
+
+    tegra241_cmdqv_free_vcmdq(cmdqv, index);
+
+    if (!iommufd_backend_alloc_hw_queue(viommu->iommufd, viommu->viommu_id,
+                                        IOMMU_HW_QUEUE_TYPE_TEGRA241_CMDQV,
+                                        index, addr, size, &hw_queue_id,
+                                        errp)) {
+        return false;
+    }
+    hw_queue = g_new(IOMMUFDHWqueue, 1);
+    hw_queue->hw_queue_id = hw_queue_id;
+    hw_queue->viommu = viommu;

hw/arm/tegra241-cmdqv.c line 104

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10 -0
+    cmdqv->vcmdq[index] = hw_queue;
+
+    return true;
+}
+
+static void tegra241_cmdqv_setup_all_vcmdq(Tegra241CMDQV *cmdqv,
+                                           Error **errp)
+{
+    for (int i = 0; i < TEGRA241_CMDQV_MAX_CMDQ; i++) {
+        if (!tegra241_cmdqv_setup_vcmdq(cmdqv, i, errp)) {

if case of failure,what is the functional consequence. Can you comment on this in the commit msg or here.

Quoted 133 lines
hw/arm/tegra241-cmdqv.c +44 -6
+            return;
+        }
+    }
+}
+
 /*
  * Returns true if the per-VCMDQ CMDQ_EN_OK bit is set.
  */
@@ -172,7 +261,7 @@ static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv,
  */
 static void tegra241_cmdqv_write_vcmdq_page1(Tegra241CMDQV *cmdqv,
                                              hwaddr offset0, int index,
-                                             uint32_t value)
+                                             uint32_t value, Error **errp)
 {
     switch (offset0) {
     case A_VCMDQ0_BASE_L:
@@ -181,6 +270,7 @@ static void tegra241_cmdqv_write_vcmdq_page1(Tegra241CMDQV *cmdqv,
         }
         cmdqv->vcmdq_base[index] =
             deposit64(cmdqv->vcmdq_base[index], 0, 32, value);
+        tegra241_cmdqv_setup_vcmdq(cmdqv, index, errp);
         break;
     case A_VCMDQ0_BASE_H:
         if (tegra241_vcmdq_enabled(cmdqv, index)) {
@@ -188,6 +278,7 @@ static void tegra241_cmdqv_write_vcmdq_page1(Tegra241CMDQV *cmdqv,
         }
         cmdqv->vcmdq_base[index] =
             deposit64(cmdqv->vcmdq_base[index], 32, 32, value);
+        tegra241_cmdqv_setup_vcmdq(cmdqv, index, errp);
         break;
     case A_VCMDQ0_CONS_INDX_BASE_DRAM_L:
         cmdqv->vcmdq_cons_indx_base[index] =
@@ -210,7 +301,7 @@ static void tegra241_cmdqv_write_vcmdq_page1(Tegra241CMDQV *cmdqv,
  */
 static void tegra241_cmdqv_write_vcmdq_page1_64(Tegra241CMDQV *cmdqv,
                                                 hwaddr offset0, int index,
-                                                uint64_t value)
+                                                uint64_t value, Error **errp)
 {
     switch (offset0) {
     case A_VCMDQ0_BASE_L:
@@ -218,6 +309,7 @@ static void tegra241_cmdqv_write_vcmdq_page1_64(Tegra241CMDQV *cmdqv,
             return;
         }
         cmdqv->vcmdq_base[index] = value;
+        tegra241_cmdqv_setup_vcmdq(cmdqv, index, errp);
         break;
     case A_VCMDQ0_CONS_INDX_BASE_DRAM_L:
         cmdqv->vcmdq_cons_indx_base[index] = value;
@@ -282,8 +374,14 @@ static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv,
         if (value & R_VINTF0_CONFIG_ENABLE_MASK) {
             if (tegra241_cmdqv_mmap_vintf_page0(cmdqv, errp)) {
                 cmdqv->vintf_status |= R_VINTF0_STATUS_ENABLE_OK_MASK;
+                /*
+                 * VCMDQs whose BASE was programmed before VINTF was
+                 * enabled need their hw_queue allocated now.
+                 */
+                tegra241_cmdqv_setup_all_vcmdq(cmdqv, errp);
             }
         } else {
+            tegra241_cmdqv_free_all_vcmdq(cmdqv);
             tegra241_cmdqv_munmap_vintf_page0(cmdqv, errp);
             cmdqv->vintf_status &= ~R_VINTF0_STATUS_ENABLE_OK_MASK;
         }
@@ -394,16 +492,37 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,
         cmdqv->config = value;
         if (value & R_CONFIG_CMDQV_EN_MASK) {
             cmdqv->status |= R_STATUS_CMDQV_ENABLED_MASK;
+            /*
+             * VCMDQs whose BASE was programmed before CMDQV was enabled
+             * need their hw_queue allocated now.
+             */
+            tegra241_cmdqv_setup_all_vcmdq(cmdqv, &local_err);
         } else {
+            tegra241_cmdqv_free_all_vcmdq(cmdqv);
             cmdqv->status &= ~R_STATUS_CMDQV_ENABLED_MASK;
         }
         break;
     case A_VI_INT_MASK_0 ... A_VI_INT_MASK_1:
         cmdqv->vi_int_mask[(offset - A_VI_INT_MASK_0) / 4] = value;
         break;
-    case A_CMDQ_ALLOC_MAP_0 ... A_CMDQ_ALLOC_MAP_1:
-        cmdqv->cmdq_alloc_map[(offset - A_CMDQ_ALLOC_MAP_0) / 4] = value;
+    case A_CMDQ_ALLOC_MAP_0 ... A_CMDQ_ALLOC_MAP_1: {
+        int idx = (offset - A_CMDQ_ALLOC_MAP_0) / 4;
+        bool was_alloc = cmdqv->cmdq_alloc_map[idx] &
+                         R_CMDQ_ALLOC_MAP_0_ALLOC_MASK;
+        bool now_alloc = value & R_CMDQ_ALLOC_MAP_0_ALLOC_MASK;
+
+        cmdqv->cmdq_alloc_map[idx] = value;
+        /*
+         * If the VCMDQ was already programmed (BASE) before mapping, fire
+         * setup on the ALLOC 0->1 transition; tear down on 1->0.
+         */
+        if (!was_alloc && now_alloc) {
+            tegra241_cmdqv_setup_vcmdq(cmdqv, idx, &local_err);
+        } else if (was_alloc && !now_alloc) {
+            tegra241_cmdqv_free_vcmdq(cmdqv, idx);
+        }
         break;
+    }
     case A_VINTF0_CONFIG ... A_VINTF0_LVCMDQ_ERR_MAP_3:
         tegra241_cmdqv_config_vintf_write(cmdqv, offset, value, &local_err);
         break;
@@ -432,7 +551,8 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,
     case A_VCMDQ0_BASE_L ... A_VCMDQ1_CONS_INDX_BASE_DRAM_H:
         index = (offset - CMDQV_VCMDQ_PAGE1_BASE) / CMDQV_VCMDQ_STRIDE;
         tegra241_cmdqv_write_vcmdq_page1(cmdqv,
-                offset - index * CMDQV_VCMDQ_STRIDE, index, value);
+                offset - index * CMDQV_VCMDQ_STRIDE, index, value,
+                &local_err);
         break;
     default:
         qemu_log_mask(LOG_UNIMP, "%s unhandled write access at 0x%" PRIx64 "\n",
@@ -451,6 +571,7 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,
 static void tegra241_cmdqv_writell_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,
                                         uint64_t value)
 {
+    Error *local_err = NULL;
     int index;
 
     switch (offset) {
@@ -465,13 +586,18 @@ static void tegra241_cmdqv_writell_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,
     case A_VCMDQ0_BASE_L ... A_VCMDQ1_CONS_INDX_BASE_DRAM_H:
         index = (offset - CMDQV_VCMDQ_PAGE1_BASE) / CMDQV_VCMDQ_STRIDE;
         tegra241_cmdqv_write_vcmdq_page1_64(cmdqv,
-                offset - index * CMDQV_VCMDQ_STRIDE, index, value);
+                offset - index * CMDQV_VCMDQ_STRIDE, index, value,
+                &local_err);
         break;
     default:
         qemu_log_mask(LOG_UNIMP,

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_writell_mmio(Tegra241CMDQV *cmdqv, hwaddr offset, · line 603

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +4 -0
                       "%s unhandled 64-bit write at 0x%" PRIx64 " (WI)\n",
                       __func__, offset);
     }
+
+    if (local_err) {
+        error_report_err(local_err);
+    }
 }
 
 static void tegra241_cmdqv_write_mmio(void *opaque, hwaddr offset,

Thanks

Eric

#44 lore raw

RE: [PATCH v5 19/32] hw/arm/tegra241-cmdqv: Allocate HW VCMDQs once configured

Quoted 21 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 21 May 2026 10:30
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 19/32] hw/arm/tegra241-cmdqv: Allocate HW VCMDQs once configured

External email: Use caution opening links or attachments

Hi,

On 5/19/26 12:36 PM, Shameer Kolothum wrote:

Quoted 8 lines
Quoted 4 lines

From: Nicolin Chen <[email protected]>

Add support for allocating IOMMUFD hardware queues when the guest programs the VCMDQ BASE registers.

You should add the fact that

iommufd_backend_alloc_hw_queue() requires the GPA of the VCMDQ. This explains why you need to those operations once the GPA has been populated.

Ok.

Quoted 120 lines
Quoted 5 lines

VCMDQ_EN lives in VCMDQ_CONFIG, which is on the VINTF Page0 region that a later patch maps directly into the guest — so QEMU won't trap its writes. Allocate the hardware queue instead once all of these are set: a RAM-backed BASE, CMDQ_ALLOC_MAP.ALLOC, and CMDQV /

VINTF

Quoted 10 lines

enabled. Each precondition write retries the allocation, so the guest may program them in any order.
If a hardware queue was previously allocated for the same VCMDQ, free it before reallocation.

Writes with invalid addresses are ignored.

All allocated VCMDQs are freed when CMDQV or VINTF is disabled, or when the ALLOC bit is cleared.

Quoted 6 lines
---
 hw/arm/tegra241-cmdqv.h |  11 ++++
 hw/arm/tegra241-cmdqv.c | 138

++++++++++++++++++++++++++++++++++++++--

hw/arm/tegra241-cmdqv.h

Quoted 15 lines
 2 files changed, 143 insertions(+), 6 deletions(-)
hw/arm/tegra241-cmdqv.h +1 -0
index f0b031b4dc..7a8cb2ebc7 100644
@@ -45,6 +45,7 @@ typedef struct Tegra241CMDQV {
     MemoryRegion mmio_cmdqv;
     qemu_irq irq;
     IOMMUFDVeventq *veventq;
+    IOMMUFDHWqueue *vcmdq[TEGRA241_CMDQV_MAX_CMDQ];
     void *vintf_page0;
 
     /* CMDQ-V Config page register cache */
@@ -361,6 +362,16 @@

SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_L_(1)

Quoted 21 lines
hw/arm/tegra241-cmdqv.h +10 -0
 SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_(0)
 SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_(1)
 
+static inline bool tegra241_cmdqv_enabled(Tegra241CMDQV *cmdqv)
+{
+    return cmdqv->status & R_STATUS_CMDQV_ENABLED_MASK;
+}
+
+static inline bool tegra241_vintf_enabled(Tegra241CMDQV *cmdqv)
+{
+    return cmdqv->vintf_status & R_VINTF0_STATUS_ENABLE_OK_MASK;
+}
+
 const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops(void);
 
 #endif /* HW_ARM_TEGRA241_CMDQV_H */
hw/arm/tegra241-cmdqv.c +0 -0
index 4d9f094b2a..f4968520f3 100644

hw/arm/tegra241-cmdqv.c line 21

Quoted 4 lines
hw/arm/tegra241-cmdqv.c +1 -0
@@ -18,6 +18,95 @@
 #include "tegra241-cmdqv.h"
 #include "trace.h"
 
+static void tegra241_cmdqv_free_vcmdq(Tegra241CMDQV *cmdqv, int

index)

Quoted 5 lines
hw/arm/tegra241-cmdqv.c +5 -0
+{
+    IOMMUFDViommu *viommu = cmdqv->s_accel->viommu;
+    IOMMUFDHWqueue *vcmdq = cmdqv->vcmdq[index];
+
+    if (!vcmdq) {

hw/arm/tegra241-cmdqv.c

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10 -0
+        return;
+    }
+    iommufd_backend_free_id(viommu->iommufd, vcmdq->hw_queue_id);
+    g_free(vcmdq);
+    cmdqv->vcmdq[index] = NULL;
+}
+
+static void tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv)
+{
+    /* uapi/linux/iommufd.h: hw_queue destroy must be in descending

@index. */

hw/arm/tegra241-cmdqv.c

Quoted 7 lines
hw/arm/tegra241-cmdqv.c +7 -0
+    for (int i = (TEGRA241_CMDQV_MAX_CMDQ - 1); i >= 0; i--) {
+        tegra241_cmdqv_free_vcmdq(cmdqv, i);
+    }
+}
+
+/*
+ * Allocate a host HW VCMDQ from the current cached BASE / size for

@index.

hw/arm/tegra241-cmdqv.c

Quoted 9 lines
hw/arm/tegra241-cmdqv.c +9 -0
+ *
+ * Setup is a no-op (returns true) when any of the following preconditions
+ * isn't met yet:
+ *  - BASE not prgrammed yet.
+ *  - VCMDQ is not mapped to a VINTF (CMDQ_ALLOC_MAP.ALLOC == 0)
+ *  - BASE / size don't resolve to a RAM region
+ *  - CMDQV global enable or VINTF enable is not yet asserted
+ */
+static bool tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int

index,

hw/arm/tegra241-cmdqv.c

Quoted 7 lines
hw/arm/tegra241-cmdqv.c +7 -0
+                                       Error **errp)
+{
+    SMMUv3AccelState *accel = cmdqv->s_accel;
+    uint64_t base_mask = (uint64_t)R_VCMDQ0_BASE_L_ADDR_MASK |
+                         (uint64_t)R_VCMDQ0_BASE_H_ADDR_MASK << 32;
+    uint64_t addr = cmdqv->vcmdq_base[index] & base_mask;
+    uint64_t log2 = cmdqv->vcmdq_base[index] &

R_VCMDQ0_BASE_L_LOG2SIZE_MASK;

hw/arm/tegra241-cmdqv.c

Quoted 12 lines
hw/arm/tegra241-cmdqv.c +12 -0
+    uint64_t size = 1ULL << (log2 + 4);
+    IOMMUFDViommu *viommu = accel->viommu;
+    IOMMUFDHWqueue *hw_queue;
+    uint32_t hw_queue_id;
+
+    /* BASE not programmed yet. */
+    if (!cmdqv->vcmdq_base[index]) {
+        return true;
+    }
+
+    /* VCMDQ not yet mapped to a VINTF. */
+    if (!(cmdqv->cmdq_alloc_map[index] &

R_CMDQ_ALLOC_MAP_0_ALLOC_MASK)) {

Quoted 6 lines

hw/arm/tegra241-cmdqv.c line 76

Quoted 5 lines
hw/arm/tegra241-cmdqv.c +5 -0
+        return true;
+    }
+
+    /* Ignore any invalid address. This may come as part of reset etc. */
+    if (!address_space_range_is_ram(&address_space_memory, addr, size)) {

why do you need to test the whole range and not only the start address

That was to make it more defensive based on v4 discussion here:
https://lore.kernel.org/qemu-devel/aft%[email protected]/

Please let me know if there is a better way.

Quoted 24 lines

hw/arm/tegra241-cmdqv.c

Quoted 4 lines
hw/arm/tegra241-cmdqv.c +4 -0
+        return true;
+    }
+
+    if (!tegra241_cmdqv_enabled(cmdqv) ||

!tegra241_vintf_enabled(cmdqv)) {

Quoted 19 lines
hw/arm/tegra241-cmdqv.c +18 -0
+        return true;
+    }
+
+    tegra241_cmdqv_free_vcmdq(cmdqv, index);
+
+    if (!iommufd_backend_alloc_hw_queue(viommu->iommufd, viommu-
 viommu_id,
+                                        IOMMU_HW_QUEUE_TYPE_TEGRA241_CMDQV,
+                                        index, addr, size, &hw_queue_id,
+                                        errp)) {
+        return false;
+    }
+    hw_queue = g_new(IOMMUFDHWqueue, 1);
+    hw_queue->hw_queue_id = hw_queue_id;
+    hw_queue->viommu = viommu;
+    cmdqv->vcmdq[index] = hw_queue;
+
+    return true;
+}
Quoted 8 lines

hw/arm/tegra241-cmdqv.c line 104

Quoted 6 lines
hw/arm/tegra241-cmdqv.c +6 -0
+
+static void tegra241_cmdqv_setup_all_vcmdq(Tegra241CMDQV *cmdqv,
+                                           Error **errp)
+{
+    for (int i = 0; i < TEGRA241_CMDQV_MAX_CMDQ; i++) {
+        if (!tegra241_cmdqv_setup_vcmdq(cmdqv, i, errp)) {

if case of failure,what is the functional consequence. Can you comment on this in the commit msg or here.

Today QEMU only logs the error:

qemu-system-aarch64: IOMMU_HW_QUEUE_ALLOC failed: error -22

The guest visible behaviour then depends on whether VINTF0 Page 0 was mapped into guest memory before this failure:

Case 1: setup_vcmdq() succeeds for vcmdq0, fails for vcmdq1:

- VINTF0 Page 0 was mapped to the guest after vcmdq0's success.
- Guest writes VCMDQ1_CONFIG go directly to HW.
- Guest reads VCMDQ1_STATUS directly from HW:

    ...
    tegra241_cmdqv: VINTF0: VCMDQ1/LVCMDQ1: failed to enable, STATUS=0x00000000
    tegra241_cmdqv: VINTF0: VCMDQ1/LVCMDQ1: GERRORN=0x0, GERROR=0x0, CONS=0x0
    ...

Case 2: setup_vcmdq() fails for vcmdq0:

- VINTF0 Page 0 is not mapped to the guest.
- Guest writes VCMDQ0_CONFIG. Trapped by QEMU.
- Guest reads VCMDQ0_STATUS. Trapped by QEMU.
- Both served from the register cache. The cache currently doesn't reflect the failure, so the guest thinks the queue is enabled and proceeds to issue commands, eventually:

    arm-smmu-v3 arm-smmu-v3.0.auto: CMD_SYNC timeout at 0x00000002 [hwprod 0x00000002, hwcons 0x00000000]
    arm-smmu-v3 arm-smmu-v3.0.auto: CMD_SYNC timeout at 0x00000004 [hwprod 0x00000004, hwcons 0x00000000]

Case 2 needs fixing. I'll fix the register cache on the error path for v6 and document it.

Thanks,
Shameer

#45 lore raw

Re: [PATCH v5 19/32] hw/arm/tegra241-cmdqv: Allocate HW VCMDQs once configured

Quoted 154 lines · Shameer Kolothum Thodi
Quoted 20 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 21 May 2026 10:30
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 19/32] hw/arm/tegra241-cmdqv: Allocate HW VCMDQs once configured

External email: Use caution opening links or attachments

Hi,

On 5/19/26 12:36 PM, Shameer Kolothum wrote:

Quoted 8 lines
Quoted 4 lines

From: Nicolin Chen <[email protected]>

Add support for allocating IOMMUFD hardware queues when the guest programs the VCMDQ BASE registers.

You should add the fact that

iommufd_backend_alloc_hw_queue() requires the GPA of the VCMDQ. This explains why you need to those operations once the GPA has been populated.

Ok.

Quoted 124 lines
Quoted 4 lines

VCMDQ_EN lives in VCMDQ_CONFIG, which is on the VINTF Page0 region that a later patch maps directly into the guest — so QEMU won't trap its writes. Allocate the hardware queue instead once all of these are set: a RAM-backed BASE, CMDQ_ALLOC_MAP.ALLOC, and CMDQV /

VINTF

Quoted 10 lines

enabled. Each precondition write retries the allocation, so the guest may program them in any order.
If a hardware queue was previously allocated for the same VCMDQ, free it before reallocation.

Writes with invalid addresses are ignored.

All allocated VCMDQs are freed when CMDQV or VINTF is disabled, or when the ALLOC bit is cleared.

Quoted 6 lines
---
 hw/arm/tegra241-cmdqv.h |  11 ++++
 hw/arm/tegra241-cmdqv.c | 138

++++++++++++++++++++++++++++++++++++++--

hw/arm/tegra241-cmdqv.h

Quoted 15 lines
 2 files changed, 143 insertions(+), 6 deletions(-)
hw/arm/tegra241-cmdqv.h +1 -0
index f0b031b4dc..7a8cb2ebc7 100644
@@ -45,6 +45,7 @@ typedef struct Tegra241CMDQV {
     MemoryRegion mmio_cmdqv;
     qemu_irq irq;
     IOMMUFDVeventq *veventq;
+    IOMMUFDHWqueue *vcmdq[TEGRA241_CMDQV_MAX_CMDQ];
     void *vintf_page0;
 
     /* CMDQ-V Config page register cache */
@@ -361,6 +362,16 @@

SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_L_(1)

Quoted 21 lines
hw/arm/tegra241-cmdqv.h +10 -0
 SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_(0)
 SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_(1)
 
+static inline bool tegra241_cmdqv_enabled(Tegra241CMDQV *cmdqv)
+{
+    return cmdqv->status & R_STATUS_CMDQV_ENABLED_MASK;
+}
+
+static inline bool tegra241_vintf_enabled(Tegra241CMDQV *cmdqv)
+{
+    return cmdqv->vintf_status & R_VINTF0_STATUS_ENABLE_OK_MASK;
+}
+
 const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops(void);
 
 #endif /* HW_ARM_TEGRA241_CMDQV_H */
hw/arm/tegra241-cmdqv.c +0 -0
index 4d9f094b2a..f4968520f3 100644

hw/arm/tegra241-cmdqv.c line 21

Quoted 4 lines
hw/arm/tegra241-cmdqv.c +1 -0
@@ -18,6 +18,95 @@
 #include "tegra241-cmdqv.h"
 #include "trace.h"
 
+static void tegra241_cmdqv_free_vcmdq(Tegra241CMDQV *cmdqv, int

index)

Quoted 5 lines
hw/arm/tegra241-cmdqv.c +5 -0
+{
+    IOMMUFDViommu *viommu = cmdqv->s_accel->viommu;
+    IOMMUFDHWqueue *vcmdq = cmdqv->vcmdq[index];
+
+    if (!vcmdq) {

hw/arm/tegra241-cmdqv.c

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10 -0
+        return;
+    }
+    iommufd_backend_free_id(viommu->iommufd, vcmdq->hw_queue_id);
+    g_free(vcmdq);
+    cmdqv->vcmdq[index] = NULL;
+}
+
+static void tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv)
+{
+    /* uapi/linux/iommufd.h: hw_queue destroy must be in descending

@index. */

hw/arm/tegra241-cmdqv.c

Quoted 7 lines
hw/arm/tegra241-cmdqv.c +7 -0
+    for (int i = (TEGRA241_CMDQV_MAX_CMDQ - 1); i >= 0; i--) {
+        tegra241_cmdqv_free_vcmdq(cmdqv, i);
+    }
+}
+
+/*
+ * Allocate a host HW VCMDQ from the current cached BASE / size for

@index.

hw/arm/tegra241-cmdqv.c

Quoted 9 lines
hw/arm/tegra241-cmdqv.c +9 -0
+ *
+ * Setup is a no-op (returns true) when any of the following preconditions
+ * isn't met yet:
+ *  - BASE not prgrammed yet.
+ *  - VCMDQ is not mapped to a VINTF (CMDQ_ALLOC_MAP.ALLOC == 0)
+ *  - BASE / size don't resolve to a RAM region
+ *  - CMDQV global enable or VINTF enable is not yet asserted
+ */
+static bool tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int

index,

hw/arm/tegra241-cmdqv.c

Quoted 7 lines
hw/arm/tegra241-cmdqv.c +7 -0
+                                       Error **errp)
+{
+    SMMUv3AccelState *accel = cmdqv->s_accel;
+    uint64_t base_mask = (uint64_t)R_VCMDQ0_BASE_L_ADDR_MASK |
+                         (uint64_t)R_VCMDQ0_BASE_H_ADDR_MASK << 32;
+    uint64_t addr = cmdqv->vcmdq_base[index] & base_mask;
+    uint64_t log2 = cmdqv->vcmdq_base[index] &

R_VCMDQ0_BASE_L_LOG2SIZE_MASK;

hw/arm/tegra241-cmdqv.c

Quoted 12 lines
hw/arm/tegra241-cmdqv.c +12 -0
+    uint64_t size = 1ULL << (log2 + 4);
+    IOMMUFDViommu *viommu = accel->viommu;
+    IOMMUFDHWqueue *hw_queue;
+    uint32_t hw_queue_id;
+
+    /* BASE not programmed yet. */
+    if (!cmdqv->vcmdq_base[index]) {
+        return true;
+    }
+
+    /* VCMDQ not yet mapped to a VINTF. */
+    if (!(cmdqv->cmdq_alloc_map[index] &

R_CMDQ_ALLOC_MAP_0_ALLOC_MASK)) {

Quoted 5 lines
hw/arm/tegra241-cmdqv.c +5 -0
+        return true;
+    }
+
+    /* Ignore any invalid address. This may come as part of reset etc. */
+    if (!address_space_range_is_ram(&address_space_memory, addr, size)) {
Quoted 5 lines
Quoted 1 line

why do you need to test the whole range and not only the start address

That was to make it more defensive based on v4 discussion here:
https://lore.kernel.org/qemu-devel/aft%[email protected]/

Please let me know if there is a better way.

Well it is unclear to me. You want to make sure the GPA was fully set through the H and L regs. To me that's enough. As far as I understand Nicolin's reply, the kernel checks the rest. But please sync with Jason or Nicolin

Eric

Quoted 60 lines
Quoted 25 lines

hw/arm/tegra241-cmdqv.c

Quoted 5 lines
hw/arm/tegra241-cmdqv.c +4 -0
+        return true;
+    }
+
+    if (!tegra241_cmdqv_enabled(cmdqv) ||

!tegra241_vintf_enabled(cmdqv)) {

Quoted 19 lines
hw/arm/tegra241-cmdqv.c +18 -0
+        return true;
+    }
+
+    tegra241_cmdqv_free_vcmdq(cmdqv, index);
+
+    if (!iommufd_backend_alloc_hw_queue(viommu->iommufd, viommu-
 viommu_id,
+                                        IOMMU_HW_QUEUE_TYPE_TEGRA241_CMDQV,
+                                        index, addr, size, &hw_queue_id,
+                                        errp)) {
+        return false;
+    }
+    hw_queue = g_new(IOMMUFDHWqueue, 1);
+    hw_queue->hw_queue_id = hw_queue_id;
+    hw_queue->viommu = viommu;
+    cmdqv->vcmdq[index] = hw_queue;
+
+    return true;
+}
Quoted 8 lines

hw/arm/tegra241-cmdqv.c line 104

Quoted 6 lines
hw/arm/tegra241-cmdqv.c +6 -0
+
+static void tegra241_cmdqv_setup_all_vcmdq(Tegra241CMDQV *cmdqv,
+                                           Error **errp)
+{
+    for (int i = 0; i < TEGRA241_CMDQV_MAX_CMDQ; i++) {
+        if (!tegra241_cmdqv_setup_vcmdq(cmdqv, i, errp)) {

if case of failure,what is the functional consequence. Can you comment on this in the commit msg or here.

Today QEMU only logs the error:

qemu-system-aarch64: IOMMU_HW_QUEUE_ALLOC failed: error -22

The guest visible behaviour then depends on whether VINTF0 Page 0 was mapped into guest memory before this failure:

Case 1: setup_vcmdq() succeeds for vcmdq0, fails for vcmdq1:

- VINTF0 Page 0 was mapped to the guest after vcmdq0's success.
- Guest writes VCMDQ1_CONFIG go directly to HW.
- Guest reads VCMDQ1_STATUS directly from HW:

    ...
    tegra241_cmdqv: VINTF0: VCMDQ1/LVCMDQ1: failed to enable, STATUS=0x00000000
    tegra241_cmdqv: VINTF0: VCMDQ1/LVCMDQ1: GERRORN=0x0, GERROR=0x0, CONS=0x0
    ...

Case 2: setup_vcmdq() fails for vcmdq0:

- VINTF0 Page 0 is not mapped to the guest.
- Guest writes VCMDQ0_CONFIG. Trapped by QEMU.
- Guest reads VCMDQ0_STATUS. Trapped by QEMU.
- Both served from the register cache. The cache currently doesn't reflect the failure, so the guest thinks the queue is enabled and proceeds to issue commands, eventually:

Quoted 5 lines
    arm-smmu-v3 arm-smmu-v3.0.auto: CMD_SYNC timeout at 0x00000002 [hwprod 0x00000002, hwcons 0x00000000]
    arm-smmu-v3 arm-smmu-v3.0.auto: CMD_SYNC timeout at 0x00000004 [hwprod 0x00000004, hwcons 0x00000000]

Case 2 needs fixing. I'll fix the register cache on the error path for v6 and document it.

OK ;-)

Thanks

Eric

Quoted 3 lines

Thanks,
Shameer

#46 lore raw

Re: [PATCH v5 19/32] hw/arm/tegra241-cmdqv: Allocate HW VCMDQs once configured

Quoted 9 lines · Eric Auger
Quoted 8 lines · Shameer Kolothum Thodi
Quoted 4 lines

hw/arm/tegra241-cmdqv.c line 76

Quoted 2 lines
hw/arm/tegra241-cmdqv.c +2 -0
+    /* Ignore any invalid address. This may come as part of reset etc. */
+    if (!address_space_range_is_ram(&address_space_memory, addr, size)) {

why do you need to test the whole range and not only the start addressa

That was to make it more defensive based on v4 discussion here:
https://lore.kernel.org/qemu-devel/aft%[email protected]/

Quoted 6 lines
Quoted 1 line

Please let me know if there is a better way.

Well it is unclear to me. You want to make sure the GPA was fully set through the H and L regs. To me that's enough. As far as I understand Nicolin's reply, the kernel checks the rest. But please sync with Jason or Nicolin

Yes. Kernel checks if the range is in the stage-2 page table, i.e., mapped GPAs. One caveat: mapped GPA doesn't guarantee it's is_ram.

So, what if the GPA is in the IO space (PCI BAR) or reserved space (firmware)?

AFAIK, kernel doesn't have a way to know. So, it's VMM's job to do so.

Nicolin

#47 lore raw

Re: [PATCH v5 19/32] hw/arm/tegra241-cmdqv: Allocate HW VCMDQs once configured

Hi Nicolin,

Quoted 13 lines · Nicolin Chen
Quoted 4 lines · Eric Auger
Quoted 3 lines · Shameer Kolothum Thodi
Quoted 3 lines

hw/arm/tegra241-cmdqv.c line 76

Quoted 2 lines
hw/arm/tegra241-cmdqv.c +2 -0
+    /* Ignore any invalid address. This may come as part of reset etc. */
+    if (!address_space_range_is_ram(&address_space_memory, addr, size)) {

why do you need to test the whole range and not only the start addressa

Quoted 8 lines
Quoted 4 lines

That was to make it more defensive based on v4 discussion here:
https://lore.kernel.org/qemu-devel/aft%[email protected]/

Please let me know if there is a better way.

Well it is unclear to me. You want to make sure the GPA was fully set through the H and L regs. To me that's enough. As far as I understand Nicolin's reply, the kernel checks the rest. But please sync with Jason or Nicolin

Quoted 8 lines

Yes. Kernel checks if the range is in the stage-2 page table, i.e., mapped GPAs. One caveat: mapped GPA doesn't guarantee it's is_ram.

So, what if the GPA is in the IO space (PCI BAR) or reserved space (firmware)?

AFAIK, kernel doesn't have a way to know. So, it's VMM's job to do so.

would it jeopardize the host integrity or would it just break the misbehaving guest. If the latest I am not sure this is critical to check. We are not checking either with regular SMMU cmd queues

Thanks

Eric

Quoted 3 lines

Nicolin

#48 lore raw

RE: [PATCH v5 19/32] hw/arm/tegra241-cmdqv: Allocate HW VCMDQs once configured

Quoted 18 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 26 May 2026 09:37
To: Nicolin Chen <[email protected]>
Cc: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 19/32] hw/arm/tegra241-cmdqv: Allocate HW VCMDQs once configured

External email: Use caution opening links or attachments

Quoted 22 lines

Hi Nicolin,

Quoted 4 lines · Nicolin Chen
Quoted 3 lines · Eric Auger
Quoted 2 lines · Shameer Kolothum Thodi
Quoted 2 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+    /* Ignore any invalid address. This may come as part of reset etc. */
+    if (!address_space_range_is_ram(&address_space_memory, addr,

size)) {

Quoted 14 lines
Quoted 9 lines
Quoted 5 lines
Quoted 1 line

why do you need to test the whole range and not only the start addressa

That was to make it more defensive based on v4 discussion here:
https://lore.kernel.org/qemu-devel/aft%[email protected]/

Please let me know if there is a better way.

Well it is unclear to me. You want to make sure the GPA was fully set through the H and L regs. To me that's enough. As far as I understand Nicolin's reply, the kernel checks the rest. But please sync with Jason or Nicolin

Yes. Kernel checks if the range is in the stage-2 page table, i.e., mapped GPAs. One caveat: mapped GPA doesn't guarantee it's is_ram.

So, what if the GPA is in the IO space (PCI BAR) or reserved space (firmware)?

Looking at our helper address_space_range_is_ram(), it isn't foolproof for the above cases. For the PCI BAR case we could add:

    if (!memory_region_is_ram(mr) || memory_region_is_ram_device(mr)) {
        return false;
    }

But I'm not sure how to tackle the reserved spaces. For firmware ROM we could add a memory_region_is_rom() check, but the others will still look like normal RAM, so they're hard to detect at this layer.

Quoted 6 lines
Quoted 3 lines

AFAIK, kernel doesn't have a way to know. So, it's VMM's job to do so.

would it jeopardize the host integrity or would it just break the misbehaving guest. If the latest I am not sure this is critical to check. We are not checking either with regular SMMU cmd queues

I think it would just be the guest shooting itself in the foot. Can't think of a host security/integrity issue here. So we could drop the check if that's preferred.

Thanks,
Shameer

#49 lore raw

Re: [PATCH v5 19/32] hw/arm/tegra241-cmdqv: Allocate HW VCMDQs once configured

Quoted 50 lines · Shameer Kolothum Thodi
Quoted 17 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 26 May 2026 09:37
To: Nicolin Chen <[email protected]>
Cc: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 19/32] hw/arm/tegra241-cmdqv: Allocate HW VCMDQs once configured

External email: Use caution opening links or attachments

Quoted 22 lines

Hi Nicolin,

Quoted 4 lines · Nicolin Chen
Quoted 3 lines · Eric Auger
Quoted 2 lines · Shameer Kolothum Thodi
Quoted 2 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+    /* Ignore any invalid address. This may come as part of reset etc. */
+    if (!address_space_range_is_ram(&address_space_memory, addr,

size)) {

Quoted 14 lines
Quoted 9 lines
Quoted 5 lines
Quoted 1 line

why do you need to test the whole range and not only the start addressa

That was to make it more defensive based on v4 discussion here:
https://lore.kernel.org/qemu-devel/aft%[email protected]/

Please let me know if there is a better way.

Well it is unclear to me. You want to make sure the GPA was fully set through the H and L regs. To me that's enough. As far as I understand Nicolin's reply, the kernel checks the rest. But please sync with Jason or Nicolin

Yes. Kernel checks if the range is in the stage-2 page table, i.e., mapped GPAs. One caveat: mapped GPA doesn't guarantee it's is_ram.

So, what if the GPA is in the IO space (PCI BAR) or reserved space (firmware)?

Looking at our helper address_space_range_is_ram(), it isn't foolproof for the above cases. For the PCI BAR case we could add:

    if (!memory_region_is_ram(mr) || memory_region_is_ram_device(mr)) {
        return false;
    }

But I'm not sure how to tackle the reserved spaces. For firmware ROM we could add a memory_region_is_rom() check, but the others will still look like normal RAM, so they're hard to detect at this layer.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

AFAIK, kernel doesn't have a way to know. So, it's VMM's job to do so.

would it jeopardize the host integrity or would it just break the misbehaving guest. If the latest I am not sure this is critical to check. We are not checking either with regular SMMU cmd queues

I think it would just be the guest shooting itself in the foot. Can't think of a host security/integrity issue here. So we could drop the check if that's preferred.

In that case I would rather simplify and remove the check. But that's my humble opinion ;-)

Eric

Quoted 3 lines

Thanks,
Shameer

#50 lore raw

PATCH v5 20/32 hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backing

In reply to Shameer Kolothum

Introduce tegra241_cmdqv_vintf_ptr() to route VCMDQ Page0 register accesses through the mmap'd VINTF page0 backing once a hardware queue has been allocated.

There are two QEMU trapped MMIO apertures for VCMDQ Page0 registers:

- Direct VCMDQ Page0 aperture (offset 0x10000)
- VINTF Page0 (offset 0x30000)

These are hardware aliases: they address the same underlying registers. A subsequent patch maps the VINTF aperture as a guest-direct RAM region; in this patch both remain QEMU-trapped.

VCMDQ Page0 accesses operate in one of two mutually exclusive modes, depending on whether a hardware queue (IOMMU_HW_QUEUE_ALLOC) has been allocated for the VCMDQ:

Pre-alloc: vintf_ptr is NULL. Both apertures use QEMU's register
cache. Hardware is not yet engaged.

Post-alloc: vintf_ptr is valid. Both QEMU trapped apertures access
registers directly via the mmap'd vintf_page0 pointer, bypassing the cache. Hardware is the single source of truth.

The pre-to-post-alloc transition is triggered by the IOMMUFD hardware queue allocation. The tegra241_cmdqv_sync_vcmdq() copies any pre-alloc cached writes (CONS_INDX, PROD_INDX, CONFIG, GERRORN) into the mmap'd page so the guest's view survives the transition.

CMDQV acceleration only becomes active once the guest enables VINTF and the corresponding HW QUEUE is allocated through IOMMUFD. Until then, all VCMDQ accesses are served from the emulated register cache with no real hardware command processing. This matches the CMDQV hardware specification: if the logical CMDQ index does not map to any allocated Virtual CMDQ, "the access is dropped with no Fault/Interrupt".

---
 hw/arm/tegra241-cmdqv.c | 72 +++++++++++++++++++++++++++++++++++++++--
 1 file changed, 70 insertions(+), 2 deletions(-)
hw/arm/tegra241-cmdqv.c +70 -2
index f4968520f3..a7e89905df 100644
@@ -39,6 +39,39 @@ static void tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv)
     }
 }
 
+/* Pointer into the mmap'd VINTF page0 slot for @index, or NULL. */
+static inline uint32_t *tegra241_cmdqv_vintf_ptr(Tegra241CMDQV *cmdqv,
+                                                 int index, hwaddr offset0)
+{
+    if (!cmdqv->vcmdq[index] || !cmdqv->vintf_page0) {
+        return NULL;
+    }
+    return (uint32_t *)(cmdqv->vintf_page0 +
+                        (index * CMDQV_VCMDQ_STRIDE) +
+                        (offset0 - CMDQV_VCMDQ_PAGE0_BASE));
+}
+
+/* Flush cached pre-alloc writes into the mmap'd VINTF page0 slot. */
+static void tegra241_cmdqv_sync_vcmdq(Tegra241CMDQV *cmdqv, int index)
+{
+    uint32_t *ptr;
+
+    ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index, A_VCMDQ0_CONS_INDX);
+    if (!ptr) {
+        return;
+    }
+    *ptr = cmdqv->vcmdq_cons_indx[index];
+
+    ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index, A_VCMDQ0_PROD_INDX);
+    *ptr = cmdqv->vcmdq_prod_indx[index];
+
+    ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index, A_VCMDQ0_CONFIG);
+    *ptr = cmdqv->vcmdq_config[index];
+
+    ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index, A_VCMDQ0_GERRORN);
+    *ptr = cmdqv->vcmdq_gerrorn[index];
+}
+
 /*
  * Allocate a host HW VCMDQ from the current cached BASE / size for @index.
  *
@@ -94,6 +127,9 @@ static bool tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int index,
     hw_queue->viommu = viommu;
     cmdqv->vcmdq[index] = hw_queue;
 
+    /* Pre-alloc cached writes survive the cache-to-hardware transition. */
+    tegra241_cmdqv_sync_vcmdq(cmdqv, index);
+
     return true;
 }
 
@@ -108,11 +144,14 @@ static void tegra241_cmdqv_setup_all_vcmdq(Tegra241CMDQV *cmdqv,
 }
 
 /*
- * Returns true if the per-VCMDQ CMDQ_EN_OK bit is set.
+ * Returns true if the per-VCMDQ CMDQ_EN_OK bit is set. Reads from the
+ * mmap'd VINTF page0 when the VCMDQ is allocated (HW is the source of
+ * truth post-alloc); otherwise reads from the register cache.
  */
 static bool tegra241_vcmdq_enabled(Tegra241CMDQV *cmdqv, int index)
 {
-    uint32_t status = cmdqv->vcmdq_status[index];
+    uint32_t *ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index, A_VCMDQ0_STATUS);
+    uint32_t status = ptr ? *ptr : cmdqv->vcmdq_status[index];
 
     return status & R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;
 }
@@ -122,12 +161,21 @@ static bool tegra241_vcmdq_enabled(Tegra241CMDQV *cmdqv, int index)
  *
  * The caller normalizes the MMIO offset such that @offset0 always refers
  * to a VCMDQ0_* register, while @index selects the VCMDQ instance.
+ *
+ * If the VCMDQ is allocated and VINTF page0 is mmap'd, read directly
+ * from the VINTF page0 backing. Otherwise, fall back to the cache.
  */
 static uint64_t tegra241_cmdqv_read_vcmdq_page0(Tegra241CMDQV *cmdqv,
                                                 hwaddr offset0, int index)
 {
+    uint32_t *ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index, offset0);
     uint64_t val = 0;
 
+    if (ptr) {
+        val = *ptr;
+        goto out;
+    }
+
     switch (offset0) {
     case A_VCMDQ0_CONS_INDX:
         val = cmdqv->vcmdq_cons_indx[index];
@@ -152,6 +200,7 @@ static uint64_t tegra241_cmdqv_read_vcmdq_page0(Tegra241CMDQV *cmdqv,
                       "%s unhandled read access at 0x%" PRIx64 "\n",
                       __func__, offset0);
     }
+out:
     trace_tegra241_cmdqv_read_vcmdq_page0(index, offset0, val);
     return val;
 }
@@ -225,11 +274,29 @@ static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv,
  *
  * Page 0 registers are all 32-bit; this helper is only called for 4-byte
  * writes.
+ *
+ * If the VCMDQ is allocated and VINTF page0 is mmap'd, write directly
+ * to the VINTF page0 backing. Otherwise, update the cache.
  */
 static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv,
                                              hwaddr offset0, int index,
                                              uint32_t value)
 {
+    uint32_t *ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index, offset0);
+
+    if (ptr) {
+        switch (offset0) {
+        case A_VCMDQ0_CONS_INDX:
+        case A_VCMDQ0_PROD_INDX:
+        case A_VCMDQ0_CONFIG:
+        case A_VCMDQ0_GERRORN:
+            *ptr = value;
+            goto out;
+        default:
+            break;
+        }
+    }
+
     switch (offset0) {
     case A_VCMDQ0_CONS_INDX:
         cmdqv->vcmdq_cons_indx[index] = value;
@@ -253,6 +320,7 @@ static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv,
                       "%s unhandled write access at 0x%" PRIx64 "\n",
                       __func__, offset0);
     }
+out:
     trace_tegra241_cmdqv_write_vcmdq_page0(index, offset0, value);
 }
 
Signature · 1 line
2.43.0
#51 lore raw

Re: [PATCH v5 20/32] hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backing

Hi Shameer,

Quoted 13 lines · Shameer Kolothum

Introduce tegra241_cmdqv_vintf_ptr() to route VCMDQ Page0 register accesses through the mmap'd VINTF page0 backing once a hardware queue has been allocated.

There are two QEMU trapped MMIO apertures for VCMDQ Page0 registers:

- Direct VCMDQ Page0 aperture (offset 0x10000)
- VINTF Page0 (offset 0x30000)

These are hardware aliases: they address the same underlying registers. A subsequent patch maps the VINTF aperture as a guest-direct RAM region; in this patch both remain QEMU-trapped.

Quoted 5 lines

VCMDQ Page0 accesses operate in one of two mutually exclusive modes, depending on whether a hardware queue (IOMMU_HW_QUEUE_ALLOC) has been allocated for the VCMDQ:

Pre-alloc: vintf_ptr is NULL. Both apertures use QEMU's register

do you mean vintf_page0 instead of vintf_ptr.

vintf_page0 is set on mmap, right? and not on hwqueue_alloc

Besides pre/post-alloc sounds a bit confusing to me since it can mean queue_alloc or mmap.
If I understand correctly vintf_page0 can be set while hwqueue_alloc has not been done yet and conversely, hence the double check below:

+ if (!cmdqv->vcmdq[index] || !cmdqv->vintf_page0) {

Quoted 8 lines

cache. Hardware is not yet engaged.

Post-alloc: vintf_ptr is valid. Both QEMU trapped apertures access
registers directly via the mmap'd vintf_page0 pointer, bypassing the cache. Hardware is the single source of truth.

The pre-to-post-alloc transition is triggered by the IOMMUFD hardware

so here you mean alloc is hw queue alloc

Quoted 10 lines

queue allocation. The tegra241_cmdqv_sync_vcmdq() copies any pre-alloc cached writes (CONS_INDX, PROD_INDX, CONFIG, GERRORN) into the mmap'd page so the guest's view survives the transition.

CMDQV acceleration only becomes active once the guest enables VINTF and the corresponding HW QUEUE is allocated through IOMMUFD. Until then, all VCMDQ accesses are served from the emulated register cache with no real hardware command processing. This matches the CMDQV hardware specification: if the logical CMDQ index does not map to any allocated Virtual CMDQ, "the access is dropped with no Fault/Interrupt".

I still does not realy understand what would be the problem of directly mapping the direct cmdqv page0 too - I understand that theoretically the direct view is larger than the VINTF one though -. I still think a proper explanation is missed in this patch of 22/32

Assuming the guest tries to set up some unmapped vcmdqs in the direct cmdqv page and this eventually ends up writing into the VINTF view, what would happen and in which terms does it violate the spec?

Quoted 11 lines
---
 hw/arm/tegra241-cmdqv.c | 72 +++++++++++++++++++++++++++++++++++++++--
 1 file changed, 70 insertions(+), 2 deletions(-)
hw/arm/tegra241-cmdqv.c +0 -0
index f4968520f3..a7e89905df 100644

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv) · line 42

Quoted 4 lines
hw/arm/tegra241-cmdqv.c +1 -0
@@ -39,6 +39,39 @@ static void tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv)
     }
 }
 
+/* Pointer into the mmap'd VINTF page0 slot for @index, or NULL. */

@index: index of the vcmdq
@offset0, offset within vcmdq page0

Quoted 123 lines
hw/arm/tegra241-cmdqv.c +68 -2
+static inline uint32_t *tegra241_cmdqv_vintf_ptr(Tegra241CMDQV *cmdqv,
+                                                 int index, hwaddr offset0)
+{
+    if (!cmdqv->vcmdq[index] || !cmdqv->vintf_page0) {
+        return NULL;
+    }
+    return (uint32_t *)(cmdqv->vintf_page0 +
+                        (index * CMDQV_VCMDQ_STRIDE) +
+                        (offset0 - CMDQV_VCMDQ_PAGE0_BASE));
+}
+
+/* Flush cached pre-alloc writes into the mmap'd VINTF page0 slot. */
+static void tegra241_cmdqv_sync_vcmdq(Tegra241CMDQV *cmdqv, int index)
+{
+    uint32_t *ptr;
+
+    ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index, A_VCMDQ0_CONS_INDX);
+    if (!ptr) {
+        return;
+    }
+    *ptr = cmdqv->vcmdq_cons_indx[index];
+
+    ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index, A_VCMDQ0_PROD_INDX);
+    *ptr = cmdqv->vcmdq_prod_indx[index];
+
+    ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index, A_VCMDQ0_CONFIG);
+    *ptr = cmdqv->vcmdq_config[index];
+
+    ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index, A_VCMDQ0_GERRORN);
+    *ptr = cmdqv->vcmdq_gerrorn[index];
+}
+
 /*
  * Allocate a host HW VCMDQ from the current cached BASE / size for @index.
  *
@@ -94,6 +127,9 @@ static bool tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int index,
     hw_queue->viommu = viommu;
     cmdqv->vcmdq[index] = hw_queue;
 
+    /* Pre-alloc cached writes survive the cache-to-hardware transition. */
+    tegra241_cmdqv_sync_vcmdq(cmdqv, index);
+
     return true;
 }
 
@@ -108,11 +144,14 @@ static void tegra241_cmdqv_setup_all_vcmdq(Tegra241CMDQV *cmdqv,
 }
 
 /*
- * Returns true if the per-VCMDQ CMDQ_EN_OK bit is set.
+ * Returns true if the per-VCMDQ CMDQ_EN_OK bit is set. Reads from the
+ * mmap'd VINTF page0 when the VCMDQ is allocated (HW is the source of
+ * truth post-alloc); otherwise reads from the register cache.
  */
 static bool tegra241_vcmdq_enabled(Tegra241CMDQV *cmdqv, int index)
 {
-    uint32_t status = cmdqv->vcmdq_status[index];
+    uint32_t *ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index, A_VCMDQ0_STATUS);
+    uint32_t status = ptr ? *ptr : cmdqv->vcmdq_status[index];
 
     return status & R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;
 }
@@ -122,12 +161,21 @@ static bool tegra241_vcmdq_enabled(Tegra241CMDQV *cmdqv, int index)
  *
  * The caller normalizes the MMIO offset such that @offset0 always refers
  * to a VCMDQ0_* register, while @index selects the VCMDQ instance.
+ *
+ * If the VCMDQ is allocated and VINTF page0 is mmap'd, read directly
+ * from the VINTF page0 backing. Otherwise, fall back to the cache.
  */
 static uint64_t tegra241_cmdqv_read_vcmdq_page0(Tegra241CMDQV *cmdqv,
                                                 hwaddr offset0, int index)
 {
+    uint32_t *ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index, offset0);
     uint64_t val = 0;
 
+    if (ptr) {
+        val = *ptr;
+        goto out;
+    }
+
     switch (offset0) {
     case A_VCMDQ0_CONS_INDX:
         val = cmdqv->vcmdq_cons_indx[index];
@@ -152,6 +200,7 @@ static uint64_t tegra241_cmdqv_read_vcmdq_page0(Tegra241CMDQV *cmdqv,
                       "%s unhandled read access at 0x%" PRIx64 "\n",
                       __func__, offset0);
     }
+out:
     trace_tegra241_cmdqv_read_vcmdq_page0(index, offset0, val);
     return val;
 }
@@ -225,11 +274,29 @@ static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv,
  *
  * Page 0 registers are all 32-bit; this helper is only called for 4-byte
  * writes.
+ *
+ * If the VCMDQ is allocated and VINTF page0 is mmap'd, write directly
+ * to the VINTF page0 backing. Otherwise, update the cache.
  */
 static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv,
                                              hwaddr offset0, int index,
                                              uint32_t value)
 {
+    uint32_t *ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index, offset0);
+
+    if (ptr) {
+        switch (offset0) {
+        case A_VCMDQ0_CONS_INDX:
+        case A_VCMDQ0_PROD_INDX:
+        case A_VCMDQ0_CONFIG:
+        case A_VCMDQ0_GERRORN:
+            *ptr = value;
+            goto out;
+        default:
+            break;
+        }
+    }
+
     switch (offset0) {
     case A_VCMDQ0_CONS_INDX:
         cmdqv->vcmdq_cons_indx[index] = value;

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv, · line 326

Quoted 7 lines
hw/arm/tegra241-cmdqv.c +1 -0
@@ -253,6 +320,7 @@ static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv,
                       "%s unhandled write access at 0x%" PRIx64 "\n",
                       __func__, offset0);
     }
+out:
     trace_tegra241_cmdqv_write_vcmdq_page0(index, offset0, value);
 }
 

Thanks

Eric

#52 lore raw

RE: [PATCH v5 20/32] hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backing

Hi Eric,

Quoted 34 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 21 May 2026 14:41
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 20/32] hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backing

External email: Use caution opening links or attachments

Hi Shameer,

Quoted 9 lines · Shameer Kolothum

Introduce tegra241_cmdqv_vintf_ptr() to route VCMDQ Page0 register accesses through the mmap'd VINTF page0 backing once a hardware queue has been allocated.

There are two QEMU trapped MMIO apertures for VCMDQ Page0 registers:

- Direct VCMDQ Page0 aperture (offset 0x10000)
- VINTF Page0 (offset 0x30000)

Quoted 6 lines

These are hardware aliases: they address the same underlying registers. A subsequent patch maps the VINTF aperture as a guest-direct RAM region; in this patch both remain QEMU-trapped.

VCMDQ Page0 accesses operate in one of two mutually exclusive modes, depending on whether a hardware queue (IOMMU_HW_QUEUE_ALLOC)

Quoted 7 lines

has

Quoted 3 lines

been allocated for the VCMDQ:

Pre-alloc: vintf_ptr is NULL. Both apertures use QEMU's register

do you mean vintf_page0 instead of vintf_ptr.

vintf_page0 is set on mmap, right? and not on hwqueue_alloc

Yeah. I actually meant the ptr returned by tegra241_cmdqv_vintf_ptr()

Quoted 4 lines

Besides pre/post-alloc sounds a bit confusing to me since it can mean queue_alloc or mmap.
If I understand correctly vintf_page0 can be set while hwqueue_alloc has not been done yet and conversely, hence the double check below:

The "alloc" is in reference to a vcmdq is "allocated" to a vintf. I think spec has that usage.

Anyway, I will make it clear.

Quoted 25 lines

+ if (!cmdqv->vcmdq[index] || !cmdqv->vintf_page0) {

Quoted 8 lines

cache. Hardware is not yet engaged.

Post-alloc: vintf_ptr is valid. Both QEMU trapped apertures access
registers directly via the mmap'd vintf_page0 pointer, bypassing the cache. Hardware is the single source of truth.

The pre-to-post-alloc transition is triggered by the IOMMUFD hardware

so here you mean alloc is hw queue alloc

Quoted 2 lines

queue allocation. The tegra241_cmdqv_sync_vcmdq() copies any pre-alloc cached writes (CONS_INDX, PROD_INDX, CONFIG, GERRORN) into the

mmap'd

Quoted 8 lines

page so the guest's view survives the transition.

CMDQV acceleration only becomes active once the guest enables VINTF and the corresponding HW QUEUE is allocated through IOMMUFD. Until then, all VCMDQ accesses are served from the emulated register cache with no real hardware command processing. This matches the CMDQV hardware specification: if the logical CMDQ index does not map to any allocated Virtual CMDQ, "the access is dropped with no Fault/Interrupt".

Quoted 8 lines

I still does not realy understand what would be the problem of directly mapping the direct cmdqv page0 too - I understand that theoretically the direct view is larger than the VINTF one though -. I still think a proper explanation is missed in this patch of 22/32

Assuming the guest tries to set up some unmapped vcmdqs in the direct cmdqv page and this eventually ends up writing into the VINTF view, what would happen and in which terms does it violate the spec?

Ok. Let me try to explain my understanding.

This is my rationale after reading Nicolin's explanation here on the RFC where we had both direct aperture and vintf page 0 mapped to Guest .
https://lore.kernel.org/qemu-devel/aXkTzRM695oTWFaD@Asurada-Nvidia/

Consider the example:

QEMU advertises 2 VCMDQs (CMDQV_NUM_CMDQ_LOG2=1).
Guest allocates vcmdq0 only, leaves vcmdq1 unallocated, then on the direct aperture for vcmdq1:

write 0x10 -> A_VCMDQ1_PROD_INDX
read A_VCMDQ1_PROD_INDX

If we aliased the direct aperture to the VINTF mmap, the write would physically hit VINTF Page 0 in HW at an unmapped logical slot. And the spec says(p.172):

" If no Virtual CMDQ is mapped to the Guest, or if the logical CMDQ index in the Virtual Interface being accessed by the software does not map to any Virtual CMDQ, the access is dropped with no Fault/Interrupt."

So even though guest is using the direct aperture, in turn it ends up accessing the corresponding vinf page0 and won't see whatever it wrote.

I think this contradicts spec p.176 where it says:

"While the software can program the Virtual CMDQ(s) directly using the direct VCMDQ aperture (and not through the Virtual Interface), it is required that the VCMDQ be allocated to a Virtual Interface before it is used to send commands to the SMMU."

So the spec allows register programming through the direct aperture without requiring the VCMDQ to be allocated first.

@Nicolin: did I understand your earlier reasoning correctly here?

Thanks,
Shameer

#53 lore raw

Re: [PATCH v5 20/32] hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backing

Quoted 25 lines · Shameer Kolothum Thodi

QEMU advertises 2 VCMDQs (CMDQV_NUM_CMDQ_LOG2=1).
Guest allocates vcmdq0 only, leaves vcmdq1 unallocated, then on the direct aperture for vcmdq1:

write 0x10 -> A_VCMDQ1_PROD_INDX
read A_VCMDQ1_PROD_INDX

If we aliased the direct aperture to the VINTF mmap, the write would physically hit VINTF Page 0 in HW at an unmapped logical slot. And the spec says(p.172):

" If no Virtual CMDQ is mapped to the Guest, or if the logical CMDQ index in the Virtual Interface being accessed by the software does not map to any Virtual CMDQ, the access is dropped with no Fault/Interrupt."

So even though guest is using the direct aperture, in turn it ends up accessing the corresponding vinf page0 and won't see whatever it wrote.

I think this contradicts spec p.176 where it says:

"While the software can program the Virtual CMDQ(s) directly using the direct VCMDQ aperture (and not through the Virtual Interface), it is required that the VCMDQ be allocated to a Virtual Interface before it is used to send commands to the SMMU."

Quoted 4 lines

So the spec allows register programming through the direct aperture without requiring the VCMDQ to be allocated first.

@Nicolin: did I understand your earlier reasoning correctly here?

Yes.

Nicolin

#54 lore raw

Re: [PATCH v5 20/32] hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backing

In reply to Shameer Kolothum Thodi

Hi Shameer,

Quoted 100 lines · Shameer Kolothum Thodi

Hi Eric,

Quoted 34 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 21 May 2026 14:41
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 20/32] hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backing

External email: Use caution opening links or attachments

Hi Shameer,

Quoted 9 lines · Shameer Kolothum

Introduce tegra241_cmdqv_vintf_ptr() to route VCMDQ Page0 register accesses through the mmap'd VINTF page0 backing once a hardware queue has been allocated.

There are two QEMU trapped MMIO apertures for VCMDQ Page0 registers:

- Direct VCMDQ Page0 aperture (offset 0x10000)
- VINTF Page0 (offset 0x30000)

Quoted 6 lines

These are hardware aliases: they address the same underlying registers. A subsequent patch maps the VINTF aperture as a guest-direct RAM region; in this patch both remain QEMU-trapped.

VCMDQ Page0 accesses operate in one of two mutually exclusive modes, depending on whether a hardware queue (IOMMU_HW_QUEUE_ALLOC)

Quoted 7 lines

has

Quoted 3 lines

been allocated for the VCMDQ:

Pre-alloc: vintf_ptr is NULL. Both apertures use QEMU's register

do you mean vintf_page0 instead of vintf_ptr.

vintf_page0 is set on mmap, right? and not on hwqueue_alloc

Yeah. I actually meant the ptr returned by tegra241_cmdqv_vintf_ptr()

Quoted 4 lines

Besides pre/post-alloc sounds a bit confusing to me since it can mean queue_alloc or mmap.
If I understand correctly vintf_page0 can be set while hwqueue_alloc has not been done yet and conversely, hence the double check below:

The "alloc" is in reference to a vcmdq is "allocated" to a vintf. I think spec has that usage.

Anyway, I will make it clear.

Quoted 23 lines

+ if (!cmdqv->vcmdq[index] || !cmdqv->vintf_page0) {

Quoted 8 lines

cache. Hardware is not yet engaged.

Post-alloc: vintf_ptr is valid. Both QEMU trapped apertures access
registers directly via the mmap'd vintf_page0 pointer, bypassing the cache. Hardware is the single source of truth.

The pre-to-post-alloc transition is triggered by the IOMMUFD hardware

so here you mean alloc is hw queue alloc

Quoted 2 lines

queue allocation. The tegra241_cmdqv_sync_vcmdq() copies any pre-alloc cached writes (CONS_INDX, PROD_INDX, CONFIG, GERRORN) into the

mmap'd

Quoted 8 lines

page so the guest's view survives the transition.

CMDQV acceleration only becomes active once the guest enables VINTF and the corresponding HW QUEUE is allocated through IOMMUFD. Until then, all VCMDQ accesses are served from the emulated register cache with no real hardware command processing. This matches the CMDQV hardware specification: if the logical CMDQ index does not map to any allocated Virtual CMDQ, "the access is dropped with no Fault/Interrupt".

Quoted 8 lines

I still does not realy understand what would be the problem of directly mapping the direct cmdqv page0 too - I understand that theoretically the direct view is larger than the VINTF one though -. I still think a proper explanation is missed in this patch of 22/32

Assuming the guest tries to set up some unmapped vcmdqs in the direct cmdqv page and this eventually ends up writing into the VINTF view, what would happen and in which terms does it violate the spec?

Ok. Let me try to explain my understanding.

This is my rationale after reading Nicolin's explanation here on the RFC where we had both direct aperture and vintf page 0 mapped to Guest .
https://lore.kernel.org/qemu-devel/aXkTzRM695oTWFaD@Asurada-Nvidia/

Consider the example:

QEMU advertises 2 VCMDQs (CMDQV_NUM_CMDQ_LOG2=1).
Guest allocates vcmdq0 only, leaves vcmdq1 unallocated, then on the direct aperture for vcmdq1:

write 0x10 -> A_VCMDQ1_PROD_INDX
read A_VCMDQ1_PROD_INDX

Quoted 7 lines

If we aliased the direct aperture to the VINTF mmap, the write would physically hit VINTF Page 0 in HW at an unmapped logical slot. And the spec says(p.172):

" If no Virtual CMDQ is mapped to the Guest, or if the logical CMDQ index in the Virtual Interface being accessed by the software does not map to any Virtual CMDQ, the access is dropped with no Fault/Interrupt."

OK

Quoted 6 lines

So even though guest is using the direct aperture, in turn it ends up accessing the corresponding vinf page0 and won't see whatever it wrote.

I think this contradicts spec p.176 where it says:

Quoted 7 lines

"While the software can program the Virtual CMDQ(s) directly using the direct VCMDQ aperture (and not through the Virtual Interface), it is required that the VCMDQ be allocated to a Virtual Interface before it is used to send commands to the SMMU."

So the spec allows register programming through the direct aperture without requiring the VCMDQ to be allocated first.

Hum. So it is said if the cmdqv is not mapped on the VI you cannot send commands to it through the direct aperture. But it does not say what does happen if you do so. Why wouldn't the VI behavior be wrong, ie. "

the access is dropped with no Fault/Interrupt."

Eric

Quoted 5 lines

@Nicolin: did I understand your earlier reasoning correctly here?

Thanks,
Shameer

#55 lore raw

RE: [PATCH v5 20/32] hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backing

Quoted 144 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 26 May 2026 13:20
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 20/32] hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backing

External email: Use caution opening links or attachments

Hi Shameer,

Quoted 13 lines · Shameer Kolothum Thodi

Hi Eric,

Quoted 11 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 21 May 2026 14:41
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 20/32] hw/arm/tegra241-cmdqv: Use mmap'd

VINTF

Quoted 5 lines
Quoted 5 lines

page0 as VCMDQ backing

External email: Use caution opening links or attachments

Quoted 8 lines
Quoted 8 lines

Hi Shameer,

Quoted 5 lines · Shameer Kolothum

Introduce tegra241_cmdqv_vintf_ptr() to route VCMDQ Page0 register accesses through the mmap'd VINTF page0 backing once a hardware queue has been allocated.

There are two QEMU trapped MMIO apertures for VCMDQ Page0

registers:

Quoted 19 lines
Quoted 10 lines
Quoted 10 lines

- Direct VCMDQ Page0 aperture (offset 0x10000)
- VINTF Page0 (offset 0x30000)

These are hardware aliases: they address the same underlying registers. A subsequent patch maps the VINTF aperture as a guest-direct RAM region; in this patch both remain QEMU-trapped.

VCMDQ Page0 accesses operate in one of two mutually exclusive modes, depending on whether a hardware queue (IOMMU_HW_QUEUE_ALLOC)

Quoted 7 lines

has

Quoted 3 lines

been allocated for the VCMDQ:

Pre-alloc: vintf_ptr is NULL. Both apertures use QEMU's register

do you mean vintf_page0 instead of vintf_ptr.

vintf_page0 is set on mmap, right? and not on hwqueue_alloc

Yeah. I actually meant the ptr returned by tegra241_cmdqv_vintf_ptr()

Quoted 22 lines
Quoted 4 lines

Besides pre/post-alloc sounds a bit confusing to me since it can mean queue_alloc or mmap.
If I understand correctly vintf_page0 can be set while hwqueue_alloc has not been done yet and conversely, hence the double check below:

The "alloc" is in reference to a vcmdq is "allocated" to a vintf. I think spec has that usage.

Anyway, I will make it clear.

Quoted 13 lines

+ if (!cmdqv->vcmdq[index] || !cmdqv->vintf_page0) {

Quoted 8 lines

cache. Hardware is not yet engaged.

Post-alloc: vintf_ptr is valid. Both QEMU trapped apertures access
registers directly via the mmap'd vintf_page0 pointer, bypassing the cache. Hardware is the single source of truth.

The pre-to-post-alloc transition is triggered by the IOMMUFD hardware

so here you mean alloc is hw queue alloc

Quoted 1 line

queue allocation. The tegra241_cmdqv_sync_vcmdq() copies any pre-

alloc

Quoted 15 lines
Quoted 15 lines
Quoted 1 line

cached writes (CONS_INDX, PROD_INDX, CONFIG, GERRORN) into the

mmap'd

Quoted 8 lines

page so the guest's view survives the transition.

CMDQV acceleration only becomes active once the guest enables VINTF and the corresponding HW QUEUE is allocated through IOMMUFD. Until then, all VCMDQ accesses are served from the emulated register cache with no real hardware command processing. This matches the CMDQV hardware specification: if the logical CMDQ index does not map to any allocated Virtual CMDQ, "the access is dropped with no Fault/Interrupt".

I still does not realy understand what would be the problem of directly mapping the direct cmdqv page0 too - I understand that theoretically the direct view is larger than the VINTF one though -. I still think a proper explanation is missed in this patch of 22/32

Quoted 8 lines
Quoted 3 lines

Assuming the guest tries to set up some unmapped vcmdqs in the direct cmdqv page and this eventually ends up writing into the VINTF view, what would happen and in which terms does it violate the spec?

Ok. Let me try to explain my understanding.

This is my rationale after reading Nicolin's explanation here on the RFC where we had both direct aperture and vintf page 0 mapped to Guest .
https://lore.kernel.org/qemu-devel/aXkTzRM695oTWFaD@Asurada-

Nvidia/

Quoted 10 lines

Consider the example:

QEMU advertises 2 VCMDQs (CMDQV_NUM_CMDQ_LOG2=1).
Guest allocates vcmdq0 only, leaves vcmdq1 unallocated, then on the direct aperture for vcmdq1:

write 0x10 -> A_VCMDQ1_PROD_INDX
read A_VCMDQ1_PROD_INDX

Quoted 6 lines

If we aliased the direct aperture to the VINTF mmap, the write would physically hit VINTF Page 0 in HW at an unmapped logical slot. And the spec says(p.172):

" If no Virtual CMDQ is mapped to the Guest, or if the logical CMDQ index in the Virtual Interface being accessed by the software does not map to any

Virtual

Quoted 1 line

CMDQ, the access is dropped with no Fault/Interrupt."

OK

Quoted 11 lines

So even though guest is using the direct aperture, in turn it ends up accessing the corresponding vinf page0 and won't see whatever it wrote.

I think this contradicts spec p.176 where it says:

"While the software can program the Virtual CMDQ(s) directly using the direct VCMDQ aperture (and not through the Virtual Interface), it is required that the VCMDQ be allocated to a Virtual Interface before it is used to send commands to the SMMU."

Quoted 7 lines
Quoted 2 lines

So the spec allows register programming through the direct aperture without requiring the VCMDQ to be allocated first.

Hum. So it is said if the cmdqv is not mapped on the VI you cannot send commands to it through the direct aperture. But it does not say what does happen if you do so. Why wouldn't the VI behavior be wrong, ie. "

the access is dropped with no Fault/Interrupt."

"While the software can program the Virtual CMDQ(s) directly using the
direct VCMDQ aperture (and not through the Virtual Interface), it is required that the VCMDQ be allocated to a Virtual Interface before it is used to send commands to the SMMU."

Assuming your question is regarding the above quote:

The quote is about the direct aperture and says the software "can program the Virtual CMDQ(s) directly using the direct VCMDQ aperture", and only the "send commands to the SMMU" step is gated by VINTF allocation. If direct-aperture writes were also dropped for unallocated VCMDQs, the "can program directly" wording wouldn't be meaningful.

"Can program" behaviour:

write 0x10 -> A_VCMDQ1_PROD_INDX
read A_VCMDQ1_PROD_INDX // read returns 0x10

"Drop" behaviour:

write 0x10 -> A_VCMDQ1_PROD_INDX
read A_VCMDQ1_PROD_INDX // read returns 0 or undefined

Hope this answers your question. let me know if not.

Thanks,
Shameer

#56 lore raw

Re: [PATCH v5 20/32] hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backing

Quoted 166 lines · Shameer Kolothum Thodi
Quoted 140 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 26 May 2026 13:20
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 20/32] hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backing

External email: Use caution opening links or attachments

Hi Shameer,

Quoted 13 lines · Shameer Kolothum Thodi

Hi Eric,

Quoted 11 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 21 May 2026 14:41
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 20/32] hw/arm/tegra241-cmdqv: Use mmap'd

VINTF

Quoted 5 lines
Quoted 5 lines

page0 as VCMDQ backing

External email: Use caution opening links or attachments

Quoted 8 lines
Quoted 8 lines

Hi Shameer,

Quoted 5 lines · Shameer Kolothum

Introduce tegra241_cmdqv_vintf_ptr() to route VCMDQ Page0 register accesses through the mmap'd VINTF page0 backing once a hardware queue has been allocated.

There are two QEMU trapped MMIO apertures for VCMDQ Page0

registers:

Quoted 18 lines
Quoted 9 lines
Quoted 9 lines

- Direct VCMDQ Page0 aperture (offset 0x10000)
- VINTF Page0 (offset 0x30000)

These are hardware aliases: they address the same underlying registers. A subsequent patch maps the VINTF aperture as a guest-direct RAM region; in this patch both remain QEMU-trapped.

VCMDQ Page0 accesses operate in one of two mutually exclusive modes, depending on whether a hardware queue (IOMMU_HW_QUEUE_ALLOC)

Quoted 7 lines

has

Quoted 3 lines

been allocated for the VCMDQ:

Pre-alloc: vintf_ptr is NULL. Both apertures use QEMU's register

do you mean vintf_page0 instead of vintf_ptr.

vintf_page0 is set on mmap, right? and not on hwqueue_alloc

Yeah. I actually meant the ptr returned by tegra241_cmdqv_vintf_ptr()

Quoted 22 lines
Quoted 4 lines

Besides pre/post-alloc sounds a bit confusing to me since it can mean queue_alloc or mmap.
If I understand correctly vintf_page0 can be set while hwqueue_alloc has not been done yet and conversely, hence the double check below:

The "alloc" is in reference to a vcmdq is "allocated" to a vintf. I think spec has that usage.

Anyway, I will make it clear.

Quoted 13 lines

+ if (!cmdqv->vcmdq[index] || !cmdqv->vintf_page0) {

Quoted 8 lines

cache. Hardware is not yet engaged.

Post-alloc: vintf_ptr is valid. Both QEMU trapped apertures access
registers directly via the mmap'd vintf_page0 pointer, bypassing the cache. Hardware is the single source of truth.

The pre-to-post-alloc transition is triggered by the IOMMUFD hardware

so here you mean alloc is hw queue alloc

Quoted 1 line

queue allocation. The tegra241_cmdqv_sync_vcmdq() copies any pre-

alloc

Quoted 15 lines
Quoted 15 lines
Quoted 1 line

cached writes (CONS_INDX, PROD_INDX, CONFIG, GERRORN) into the

mmap'd

Quoted 8 lines

page so the guest's view survives the transition.

CMDQV acceleration only becomes active once the guest enables VINTF and the corresponding HW QUEUE is allocated through IOMMUFD. Until then, all VCMDQ accesses are served from the emulated register cache with no real hardware command processing. This matches the CMDQV hardware specification: if the logical CMDQ index does not map to any allocated Virtual CMDQ, "the access is dropped with no Fault/Interrupt".

I still does not realy understand what would be the problem of directly mapping the direct cmdqv page0 too - I understand that theoretically the direct view is larger than the VINTF one though -. I still think a proper explanation is missed in this patch of 22/32

Quoted 8 lines
Quoted 3 lines

Assuming the guest tries to set up some unmapped vcmdqs in the direct cmdqv page and this eventually ends up writing into the VINTF view, what would happen and in which terms does it violate the spec?

Ok. Let me try to explain my understanding.

This is my rationale after reading Nicolin's explanation here on the RFC where we had both direct aperture and vintf page 0 mapped to Guest .
https://lore.kernel.org/qemu-devel/aXkTzRM695oTWFaD@Asurada-

Nvidia/

Quoted 9 lines

Consider the example:

QEMU advertises 2 VCMDQs (CMDQV_NUM_CMDQ_LOG2=1).
Guest allocates vcmdq0 only, leaves vcmdq1 unallocated, then on the direct aperture for vcmdq1:

write 0x10 -> A_VCMDQ1_PROD_INDX
read A_VCMDQ1_PROD_INDX

Quoted 6 lines

If we aliased the direct aperture to the VINTF mmap, the write would physically hit VINTF Page 0 in HW at an unmapped logical slot. And the spec says(p.172):

" If no Virtual CMDQ is mapped to the Guest, or if the logical CMDQ index in the Virtual Interface being accessed by the software does not map to any

Virtual

Quoted 1 line

CMDQ, the access is dropped with no Fault/Interrupt."

OK

Quoted 10 lines

So even though guest is using the direct aperture, in turn it ends up accessing the corresponding vinf page0 and won't see whatever it wrote.

I think this contradicts spec p.176 where it says:

"While the software can program the Virtual CMDQ(s) directly using the direct VCMDQ aperture (and not through the Virtual Interface), it is required that the VCMDQ be allocated to a Virtual Interface before it is used to send commands to the SMMU."

Quoted 7 lines
Quoted 2 lines

So the spec allows register programming through the direct aperture without requiring the VCMDQ to be allocated first.

Hum. So it is said if the cmdqv is not mapped on the VI you cannot send commands to it through the direct aperture. But it does not say what does happen if you do so. Why wouldn't the VI behavior be wrong, ie. "

the access is dropped with no Fault/Interrupt."

"While the software can program the Virtual CMDQ(s) directly using the
direct VCMDQ aperture (and not through the Virtual Interface), it is required that the VCMDQ be allocated to a Virtual Interface before it is used to send commands to the SMMU."

Assuming your question is regarding the above quote:

The quote is about the direct aperture and says the software "can program the Virtual CMDQ(s) directly using the direct VCMDQ aperture", and only the "send commands to the SMMU" step is gated by VINTF allocation. If direct-aperture writes were also dropped for unallocated VCMDQs, the "can program directly" wording wouldn't be meaningful.

"Can program" behaviour:

write 0x10 -> A_VCMDQ1_PROD_INDX
read A_VCMDQ1_PROD_INDX // read returns 0x10

Quoted 6 lines

"Drop" behaviour:

write 0x10 -> A_VCMDQ1_PROD_INDX
read A_VCMDQ1_PROD_INDX // read returns 0 or undefined

Hope this answers your question. let me know if not.

On my end, I read that the software can use either the direct view or the VINTF view as long as the cmdqv is mapped. And if the vcmdq is not mapped we are not really told what happens.
But anyway this use case of using the direct view looks pretty marginal, doesn't it. So feel free to interpret it as you agree with Nicolin and HW experts.

Thanks

Eric

Quoted 8 lines

Thanks,
Shameer

#57 lore raw

RE: [PATCH v5 20/32] hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backing

Quoted 44 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 26 May 2026 14:53
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 20/32] hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backing

External email: Use caution opening links or attachments

Quoted 25 lines · Shameer Kolothum Thodi
Quoted 1 line

the access is dropped with no Fault/Interrupt."

"While the software can program the Virtual CMDQ(s) directly using the
direct VCMDQ aperture (and not through the Virtual Interface), it is required that the VCMDQ be allocated to a Virtual Interface before it is used to send commands to the SMMU."

Assuming your question is regarding the above quote:

The quote is about the direct aperture and says the software "can program the Virtual CMDQ(s) directly using the direct VCMDQ aperture", and only the "send commands to the SMMU" step is gated by VINTF allocation. If direct-aperture writes were also dropped for unallocated VCMDQs, the "can program directly" wording wouldn't be meaningful.

"Can program" behaviour:

write 0x10 -> A_VCMDQ1_PROD_INDX
read A_VCMDQ1_PROD_INDX // read returns 0x10

"Drop" behaviour:

write 0x10 -> A_VCMDQ1_PROD_INDX
read A_VCMDQ1_PROD_INDX // read returns 0 or undefined

Quoted 7 lines
Quoted 1 line

Hope this answers your question. let me know if not.

On my end, I read that the software can use either the direct view or the VINTF view as long as the cmdqv is mapped. And if the vcmdq is not mapped we are not really told what happens.
But anyway this use case of using the direct view looks pretty marginal, doesn't it.

Yes, the direct aperture read/write isn't an expected common use case.

So feel free to interpret it as you agree with Nicolin and

Quoted 1 line

HW experts.

Thanks. I will keep it as is and have another look at the "overview" section to improve the description of this.

Thanks,
Shameer

#58 lore raw

PATCH v5 21/32 memory: Allow RAM device regions to skip IOMMU mapping

In reply to Shameer Kolothum

Some RAM device regions created with memory_region_init_ram_device_ptr() are not intended to be P2P DMA targets.

The VFIO listener currently treats all RAM device regions as DMA capable and attempts to map them into the IOMMU. For regions without dma-buf backing this fails and prints warnings such as:

IOMMU_IOAS_MAP failed: Bad address, PCI BAR?

Introduce a MemoryRegion flag (ram_device_skip_iommu_map) to mark RAM device regions that should not be IOMMU mapped, paired with memory_region_skip_iommu_map() / memory_region_set_skip_iommu_map() accessors. When the flag is set, the VFIO listener skips DMA mapping for that region.

---
 include/system/memory.h | 21 +++++++++++++++++++++
 hw/vfio/listener.c      |  4 ++++
 system/memory.c         | 10 ++++++++++
 3 files changed, 35 insertions(+)
include/system/memory.h +21 -0
index af5f96f6ff..b11d6bf07f 100644
@@ -864,6 +864,8 @@ struct MemoryRegion {
 
     /* For devices designed to perform re-entrant IO into their own IO MRs */
     bool disable_reentrancy_guard;
+    /* RAM device region that does not require IOMMU mapping for P2P */
+    bool ram_device_skip_iommu_map;
 };
 
 struct IOMMUMemoryRegion {
@@ -1743,6 +1745,25 @@ static inline bool memory_region_is_romd(const MemoryRegion *mr)
  */
 bool memory_region_is_protected(const MemoryRegion *mr);
 
+/**
+ * memory_region_skip_iommu_map: check whether a memory region is excluded
+ *                               from IOMMU mapping
+ *
+ * Returns %true if @mr is a RAM device region marked to skip IOMMU mapping.
+ *
+ * @mr: the memory region being queried
+ */
+bool memory_region_skip_iommu_map(const MemoryRegion *mr);
+
+/**
+ * memory_region_set_skip_iommu_map: mark a RAM device region to skip IOMMU
+ *                                   mapping
+ *
+ * @mr: the memory region being modified
+ * @skip: %true to skip IOMMU mapping, %false to allow it
+ */
+void memory_region_set_skip_iommu_map(MemoryRegion *mr, bool skip);
+
 /**
  * memory_region_has_guest_memfd: check whether a memory region has guest_memfd
  *     associated
hw/vfio/listener.c +4 -0
index 0b72a2cf5e..18100e9897 100644
@@ -610,6 +610,10 @@ void vfio_container_region_add(VFIOContainer *bcontainer,
         }
     }
 
+    if (memory_region_skip_iommu_map(section->mr)) {
+        return;
+    }
+
     ret = vfio_container_dma_map(bcontainer, iova, int128_get64(llsize),
                                  vaddr, section->readonly, section->mr);
     if (ret) {
system/memory.c +10 -0
index 739ba11da6..48245fd01b 100644
@@ -1814,6 +1814,16 @@ bool memory_region_is_protected(const MemoryRegion *mr)
     return mr->ram && (mr->ram_block->flags & RAM_PROTECTED);
 }
 
+bool memory_region_skip_iommu_map(const MemoryRegion *mr)
+{
+    return memory_region_is_ram_device(mr) && mr->ram_device_skip_iommu_map;
+}
+
+void memory_region_set_skip_iommu_map(MemoryRegion *mr, bool skip)
+{
+    mr->ram_device_skip_iommu_map = skip;
+}
+
 bool memory_region_has_guest_memfd(const MemoryRegion *mr)
 {
     return mr->ram_block && mr->ram_block->guest_memfd >= 0;
Signature · 1 line
2.43.0
Reviewed-by Tested-by on v5 21/32
#59 lore raw

Re: [PATCH v5 21/32] memory: Allow RAM device regions to skip IOMMU mapping

Hi Shameer,

Quoted 66 lines · Shameer Kolothum

Some RAM device regions created with memory_region_init_ram_device_ptr() are not intended to be P2P DMA targets.

The VFIO listener currently treats all RAM device regions as DMA capable and attempts to map them into the IOMMU. For regions without dma-buf backing this fails and prints warnings such as:

IOMMU_IOAS_MAP failed: Bad address, PCI BAR?

Introduce a MemoryRegion flag (ram_device_skip_iommu_map) to mark RAM device regions that should not be IOMMU mapped, paired with memory_region_skip_iommu_map() / memory_region_set_skip_iommu_map() accessors. When the flag is set, the VFIO listener skips DMA mapping for that region.

---
 include/system/memory.h | 21 +++++++++++++++++++++
 hw/vfio/listener.c      |  4 ++++
 system/memory.c         | 10 ++++++++++
 3 files changed, 35 insertions(+)
include/system/memory.h +21 -0
index af5f96f6ff..b11d6bf07f 100644
@@ -864,6 +864,8 @@ struct MemoryRegion {
 
     /* For devices designed to perform re-entrant IO into their own IO MRs */
     bool disable_reentrancy_guard;
+    /* RAM device region that does not require IOMMU mapping for P2P */
+    bool ram_device_skip_iommu_map;
 };
 
 struct IOMMUMemoryRegion {
@@ -1743,6 +1745,25 @@ static inline bool memory_region_is_romd(const MemoryRegion *mr)
  */
 bool memory_region_is_protected(const MemoryRegion *mr);
 
+/**
+ * memory_region_skip_iommu_map: check whether a memory region is excluded
+ *                               from IOMMU mapping
+ *
+ * Returns %true if @mr is a RAM device region marked to skip IOMMU mapping.
+ *
+ * @mr: the memory region being queried
+ */
+bool memory_region_skip_iommu_map(const MemoryRegion *mr);
+
+/**
+ * memory_region_set_skip_iommu_map: mark a RAM device region to skip IOMMU
+ *                                   mapping
+ *
+ * @mr: the memory region being modified
+ * @skip: %true to skip IOMMU mapping, %false to allow it
+ */
+void memory_region_set_skip_iommu_map(MemoryRegion *mr, bool skip);
+
 /**
  * memory_region_has_guest_memfd: check whether a memory region has guest_memfd
  *     associated
hw/vfio/listener.c +0 -0
index 0b72a2cf5e..18100e9897 100644

hw/vfio/listener.c void vfio_container_region_add(VFIOContainer *bcontainer, · line 613

Quoted 4 lines
hw/vfio/listener.c +1 -0
@@ -610,6 +610,10 @@ void vfio_container_region_add(VFIOContainer *bcontainer,
         }
     }
 
+    if (memory_region_skip_iommu_map(section->mr)) {

Maybe add a trace-point just as above
trace_vfio_listener_region_add_no_dma_map()

Otherwise looks good to me.

Thanks

Eric

Quoted 27 lines
hw/vfio/listener.c +3 -0
+        return;
+    }
+
     ret = vfio_container_dma_map(bcontainer, iova, int128_get64(llsize),
                                  vaddr, section->readonly, section->mr);
     if (ret) {
system/memory.c +10 -0
index 739ba11da6..48245fd01b 100644
@@ -1814,6 +1814,16 @@ bool memory_region_is_protected(const MemoryRegion *mr)
     return mr->ram && (mr->ram_block->flags & RAM_PROTECTED);
 }
 
+bool memory_region_skip_iommu_map(const MemoryRegion *mr)
+{
+    return memory_region_is_ram_device(mr) && mr->ram_device_skip_iommu_map;
+}
+
+void memory_region_set_skip_iommu_map(MemoryRegion *mr, bool skip)
+{
+    mr->ram_device_skip_iommu_map = skip;
+}
+
 bool memory_region_has_guest_memfd(const MemoryRegion *mr)
 {
     return mr->ram_block && mr->ram_block->guest_memfd >= 0;
#60 lore raw

PATCH v5 22/32 hw/arm/tegra241-cmdqv: Map VINTF page0 into guest MMIO space

In reply to Shameer Kolothum

From: Nicolin Chen <[email protected]>

Once a VCMDQ is allocated, map the mmap'd vintf_page0 region directly into the guest-visible MMIO space at offset 0x30000 as a RAM-backed MemoryRegion. This eliminates QEMU trapping for hot-path CONS/PROD index updates.

After this patch, the two VCMDQ apertures use different access paths:
the direct aperture (0x10000) remains QEMU-trapped and writes via vintf_ptr, while the VI aperture (0x30000) is a direct guest RAM mapping. Both paths write to the same underlying vintf_page0 memory, so no synchronisation between the apertures is needed.

The mapping is installed lazily on first successful VCMDQ hardware queue allocation and removed when CMDQV or VINTF is disabled.

---
 hw/arm/tegra241-cmdqv.h |  1 +
 hw/arm/tegra241-cmdqv.c | 39 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 40 insertions(+)
hw/arm/tegra241-cmdqv.h +1 -0
index 7a8cb2ebc7..e9e1933a19 100644
@@ -47,6 +47,7 @@ typedef struct Tegra241CMDQV {
     IOMMUFDVeventq *veventq;
     IOMMUFDHWqueue *vcmdq[TEGRA241_CMDQV_MAX_CMDQ];
     void *vintf_page0;
+    MemoryRegion *mr_vintf_page0;
 
     /* CMDQ-V Config page register cache */
     uint32_t config;
hw/arm/tegra241-cmdqv.c +39 -0
index a7e89905df..853c548e58 100644
@@ -18,6 +18,40 @@
 #include "tegra241-cmdqv.h"
 #include "trace.h"
 
+static void tegra241_cmdqv_guest_unmap_vintf_page0(Tegra241CMDQV *cmdqv)
+{
+    if (!cmdqv->mr_vintf_page0) {
+        return;
+    }
+
+    memory_region_del_subregion(&cmdqv->mmio_cmdqv, cmdqv->mr_vintf_page0);
+    object_unparent(OBJECT(cmdqv->mr_vintf_page0));
+    g_free(cmdqv->mr_vintf_page0);
+    cmdqv->mr_vintf_page0 = NULL;
+}
+
+static void tegra241_cmdqv_guest_map_vintf_page0(Tegra241CMDQV *cmdqv)
+{
+    char *name;
+
+    if (cmdqv->mr_vintf_page0) {
+        return;
+    }
+
+    name = g_strdup_printf("%s vintf-page0",
+                           memory_region_name(&cmdqv->mmio_cmdqv));
+    cmdqv->mr_vintf_page0 = g_malloc0(sizeof(*cmdqv->mr_vintf_page0));
+    memory_region_init_ram_device_ptr(cmdqv->mr_vintf_page0,
+                                      memory_region_owner(&cmdqv->mmio_cmdqv),
+                                      name, VINTF_PAGE_SIZE,
+                                      cmdqv->vintf_page0);
+    memory_region_set_skip_iommu_map(cmdqv->mr_vintf_page0, true);
+    memory_region_add_subregion_overlap(&cmdqv->mmio_cmdqv,
+                                        CMDQV_VINTF_PAGE0_BASE,
+                                        cmdqv->mr_vintf_page0, 1);
+    g_free(name);
+}
+
 static void tegra241_cmdqv_free_vcmdq(Tegra241CMDQV *cmdqv, int index)
 {
     IOMMUFDViommu *viommu = cmdqv->s_accel->viommu;
@@ -130,6 +164,9 @@ static bool tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int index,
     /* Pre-alloc cached writes survive the cache-to-hardware transition. */
     tegra241_cmdqv_sync_vcmdq(cmdqv, index);
 
+    /* Map the mmap'd VINTF page0 into guest MMIO space. */
+    tegra241_cmdqv_guest_map_vintf_page0(cmdqv);
+
     return true;
 }
 
@@ -449,6 +486,7 @@ static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv,
                 tegra241_cmdqv_setup_all_vcmdq(cmdqv, errp);
             }
         } else {
+            tegra241_cmdqv_guest_unmap_vintf_page0(cmdqv);
             tegra241_cmdqv_free_all_vcmdq(cmdqv);
             tegra241_cmdqv_munmap_vintf_page0(cmdqv, errp);
             cmdqv->vintf_status &= ~R_VINTF0_STATUS_ENABLE_OK_MASK;
@@ -566,6 +604,7 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,
              */
             tegra241_cmdqv_setup_all_vcmdq(cmdqv, &local_err);
         } else {
+            tegra241_cmdqv_guest_unmap_vintf_page0(cmdqv);
             tegra241_cmdqv_free_all_vcmdq(cmdqv);
             cmdqv->status &= ~R_STATUS_CMDQV_ENABLED_MASK;
         }
Signature · 1 line
2.43.0
#61 lore raw

Re: [PATCH v5 22/32] hw/arm/tegra241-cmdqv: Map VINTF page0 into guest MMIO space

Quoted 7 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

Once a VCMDQ is allocated, map the mmap'd vintf_page0 region directly into the guest-visible MMIO space at offset 0x30000 as a RAM-backed MemoryRegion. This eliminates QEMU trapping for hot-path CONS/PROD index updates.

Quoted 5 lines

After this patch, the two VCMDQ apertures use different access paths:
the direct aperture (0x10000) remains QEMU-trapped and writes via vintf_ptr, while the VI aperture (0x30000) is a direct guest RAM mapping. Both paths write to the same underlying vintf_page0 memory, so no synchronisation between the apertures is needed.

Please further explain the rationale of having 2 separate paths

Eric

Quoted 80 lines

The mapping is installed lazily on first successful VCMDQ hardware queue allocation and removed when CMDQV or VINTF is disabled.

---
 hw/arm/tegra241-cmdqv.h |  1 +
 hw/arm/tegra241-cmdqv.c | 39 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 40 insertions(+)
hw/arm/tegra241-cmdqv.h +1 -0
index 7a8cb2ebc7..e9e1933a19 100644
@@ -47,6 +47,7 @@ typedef struct Tegra241CMDQV {
     IOMMUFDVeventq *veventq;
     IOMMUFDHWqueue *vcmdq[TEGRA241_CMDQV_MAX_CMDQ];
     void *vintf_page0;
+    MemoryRegion *mr_vintf_page0;
 
     /* CMDQ-V Config page register cache */
     uint32_t config;
hw/arm/tegra241-cmdqv.c +37 -0
index a7e89905df..853c548e58 100644
@@ -18,6 +18,40 @@
 #include "tegra241-cmdqv.h"
 #include "trace.h"
 
+static void tegra241_cmdqv_guest_unmap_vintf_page0(Tegra241CMDQV *cmdqv)
+{
+    if (!cmdqv->mr_vintf_page0) {
+        return;
+    }
+
+    memory_region_del_subregion(&cmdqv->mmio_cmdqv, cmdqv->mr_vintf_page0);
+    object_unparent(OBJECT(cmdqv->mr_vintf_page0));
+    g_free(cmdqv->mr_vintf_page0);
+    cmdqv->mr_vintf_page0 = NULL;
+}
+
+static void tegra241_cmdqv_guest_map_vintf_page0(Tegra241CMDQV *cmdqv)
+{
+    char *name;
+
+    if (cmdqv->mr_vintf_page0) {
+        return;
+    }
+
+    name = g_strdup_printf("%s vintf-page0",
+                           memory_region_name(&cmdqv->mmio_cmdqv));
+    cmdqv->mr_vintf_page0 = g_malloc0(sizeof(*cmdqv->mr_vintf_page0));
+    memory_region_init_ram_device_ptr(cmdqv->mr_vintf_page0,
+                                      memory_region_owner(&cmdqv->mmio_cmdqv),
+                                      name, VINTF_PAGE_SIZE,
+                                      cmdqv->vintf_page0);
+    memory_region_set_skip_iommu_map(cmdqv->mr_vintf_page0, true);
+    memory_region_add_subregion_overlap(&cmdqv->mmio_cmdqv,
+                                        CMDQV_VINTF_PAGE0_BASE,
+                                        cmdqv->mr_vintf_page0, 1);
+    g_free(name);
+}
+
 static void tegra241_cmdqv_free_vcmdq(Tegra241CMDQV *cmdqv, int index)
 {
     IOMMUFDViommu *viommu = cmdqv->s_accel->viommu;
@@ -130,6 +164,9 @@ static bool tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int index,
     /* Pre-alloc cached writes survive the cache-to-hardware transition. */
     tegra241_cmdqv_sync_vcmdq(cmdqv, index);
 
+    /* Map the mmap'd VINTF page0 into guest MMIO space. */
+    tegra241_cmdqv_guest_map_vintf_page0(cmdqv);
+
     return true;
 }
 

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv, · line 486

Quoted 1 line
hw/arm/tegra241-cmdqv.c +0 -0
@@ -449,6 +486,7 @@ static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv,
                 tegra241_cmdqv_setup_all_vcmdq(cmdqv, errp);

I would rather put the     tegra241_cmdqv_guest_map_vintf_page0(cmdqv) here to have symetric map/unmap

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset, · line 605

Quoted 9 lines
hw/arm/tegra241-cmdqv.c +1 -0
             }
         } else {
+            tegra241_cmdqv_guest_unmap_vintf_page0(cmdqv);
             tegra241_cmdqv_free_all_vcmdq(cmdqv);
             tegra241_cmdqv_munmap_vintf_page0(cmdqv, errp);
             cmdqv->vintf_status &= ~R_VINTF0_STATUS_ENABLE_OK_MASK;
@@ -566,6 +604,7 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,
              */
             tegra241_cmdqv_setup_all_vcmdq(cmdqv, &local_err);

same

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset, · line 610

Quoted 5 lines
hw/arm/tegra241-cmdqv.c +1 -0
         } else {
+            tegra241_cmdqv_guest_unmap_vintf_page0(cmdqv);
             tegra241_cmdqv_free_all_vcmdq(cmdqv);
             cmdqv->status &= ~R_STATUS_CMDQV_ENABLED_MASK;
         }

Thanks

Eric

#62 lore raw

PATCH v5 23/32 hw/arm/smmuv3-accel: Introduce common helper for veventq read

In reply to Shameer Kolothum

Move the vEVENTQ read and validation logic into a common helper smmuv3_accel_event_read_validate(). The helper performs the read(), checks for overflow and short reads, validates the sequence number, and updates the sequence state.

This helper can be reused for Tegra241 CMDQV vEVENTQ support in a subsequent patch.

Error handling is slightly adjusted: instead of reporting errors directly in the read handler, the helper now returns errors via Error **. Sequence gaps are reported as warnings.

---
 hw/arm/smmuv3-accel.h       |  2 ++
 hw/arm/smmuv3-accel-stubs.c |  7 ++++
 hw/arm/smmuv3-accel.c       | 67 ++++++++++++++++++++++---------------
 3 files changed, 49 insertions(+), 27 deletions(-)
hw/arm/smmuv3-accel.h +2 -0
index 49c10535cf..241639ec8e 100644
@@ -88,6 +88,8 @@ bool smmuv3_accel_issue_inv_cmd(SMMUv3State *s, void *cmd, SMMUDevice *sdev,
                                 Error **errp);
 void smmuv3_accel_idr_override(SMMUv3State *s);
 bool smmuv3_accel_alloc_veventq(SMMUv3State *s, Error **errp);
+bool smmuv3_accel_event_read_validate(IOMMUFDVeventq *veventq, uint32_t type,
+                                      void *buf, size_t size, Error **errp);
 void smmuv3_accel_reset(SMMUv3State *s);
 
 #endif /* HW_ARM_SMMUV3_ACCEL_H */
hw/arm/smmuv3-accel-stubs.c +7 -0
index 70cef66966..9e6c44a282 100644
@@ -47,6 +47,13 @@ bool smmuv3_accel_alloc_veventq(SMMUv3State *s, Error **errp)
     return true;
 }
 
+bool smmuv3_accel_event_read_validate(IOMMUFDVeventq *veventq, uint32_t type,
+                                      void *buf, size_t size, Error **errp)
+{
+    return true;
+}
+
+
 void smmuv3_accel_reset(SMMUv3State *s)
 {
 }
hw/arm/smmuv3-accel.c +40 -27

87 lines not on this page · raw message

Signature · 1 line
2.43.0
#63 lore raw

PATCH v5 24/32 hw/arm/tegra241-cmdqv: Read and propagate Tegra241 CMDQV errors

In reply to Shameer Kolothum

Install an event handler on the CMDQV vEVENTQ fd to read and propagate host received CMDQV errors to the guest.

The handler runs in QEMU's main loop, using a non-blocking fd registered via qemu_set_fd_handler().

---
 hw/arm/tegra241-cmdqv.c | 60 +++++++++++++++++++++++++++++++++++++++++
 hw/arm/trace-events     |  1 +
 2 files changed, 61 insertions(+)
hw/arm/tegra241-cmdqv.c +60 -0
index 853c548e58..0bba2c1801 100644
@@ -14,6 +14,7 @@
 
 #include "hw/arm/smmuv3.h"
 #include "hw/arm/smmuv3-common.h"
+#include "hw/core/irq.h"
 #include "smmuv3-accel.h"
 #include "tegra241-cmdqv.h"
 #include "trace.h"
@@ -736,6 +737,48 @@ out:
     trace_tegra241_cmdqv_write_mmio(offset, value, size);
 }
 
+static void tegra241_cmdqv_event_read(void *opaque)
+{
+    Tegra241CMDQV *cmdqv = opaque;
+    IOMMUFDVeventq *veventq = cmdqv->veventq;
+    struct {
+        struct iommufd_vevent_header hdr;
+        struct iommu_vevent_tegra241_cmdqv vevent;
+    } buf;
+    Error *local_err = NULL;
+
+    if (!smmuv3_accel_event_read_validate(veventq,
+                                          IOMMU_VEVENTQ_TYPE_TEGRA241_CMDQV,
+                                          &buf, sizeof(buf), &local_err)) {
+        warn_report_err_once(local_err);
+        return;
+    }
+
+    if (buf.vevent.lvcmdq_err_map[0] || buf.vevent.lvcmdq_err_map[1]) {
+        cmdqv->vintf_cmdq_err_map[0] =
+            extract64(buf.vevent.lvcmdq_err_map[0], 0, 32);
+        cmdqv->vintf_cmdq_err_map[1] =
+            extract64(buf.vevent.lvcmdq_err_map[0], 32, 32);
+        cmdqv->vintf_cmdq_err_map[2] =
+            extract64(buf.vevent.lvcmdq_err_map[1], 0, 32);
+        cmdqv->vintf_cmdq_err_map[3] =
+            extract64(buf.vevent.lvcmdq_err_map[1], 32, 32);
+        /*
+         * CMDQV_CMDQ_ERR_MAP and VINTF0_LVCMDQ_ERR_MAP are distinct
+         * registers (different MMIO offsets). With only VINTF0 exposed
+         * they carry the same data, so mirror.
+         */
+        for (int i = 0; i < 4; i++) {
+            cmdqv->cmdq_err_map[i] = cmdqv->vintf_cmdq_err_map[i];
+        }
+        cmdqv->vi_err_map[0] |= 0x1;
+        qemu_irq_pulse(cmdqv->irq);
+        trace_tegra241_cmdqv_err_map(
+            cmdqv->vintf_cmdq_err_map[3], cmdqv->vintf_cmdq_err_map[2],
+            cmdqv->vintf_cmdq_err_map[1], cmdqv->vintf_cmdq_err_map[0]);
+    }
+}
+
 static void tegra241_cmdqv_free_viommu(SMMUv3State *s)
 {
     SMMUv3AccelState *accel = s->s_accel;
@@ -747,6 +790,7 @@ static void tegra241_cmdqv_free_viommu(SMMUv3State *s)
         return;
     }
     if (veventq) {
+        qemu_set_fd_handler(veventq->veventq_fd, NULL, NULL, NULL);
         close(veventq->veventq_fd);
         iommufd_backend_free_id(viommu->iommufd, veventq->veventq_id);
         g_free(veventq);
@@ -762,6 +806,7 @@ tegra241_cmdqv_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
     Tegra241CMDQV *cmdqv = s->s_accel->cmdqv;
     uint32_t viommu_id, veventq_id, veventq_fd;
     IOMMUFDVeventq *veventq;
+    int flags;
 
     if (!iommufd_backend_alloc_viommu(idev->iommufd, idev->devid,
                                       IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV,
@@ -780,14 +825,29 @@ tegra241_cmdqv_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
         goto free_viommu;
     }
 
+    flags = fcntl(veventq_fd, F_GETFL);
+    if (flags < 0) {
+        error_setg(errp, "Failed to get flags for vEVENTQ fd");
+        goto free_veventq;
+    }
+    if (fcntl(veventq_fd, F_SETFL, O_NONBLOCK | flags) < 0) {
+        error_setg(errp, "Failed to set O_NONBLOCK on vEVENTQ fd");
+        goto free_veventq;
+    }
+
     veventq = g_new(IOMMUFDVeventq, 1);
     veventq->veventq_id = veventq_id;
     veventq->veventq_fd = veventq_fd;
     cmdqv->veventq = veventq;
 
+    /* Set up event handler for veventq fd */
+    qemu_set_fd_handler(veventq_fd, tegra241_cmdqv_event_read, NULL, cmdqv);
     *out_viommu_id = viommu_id;
     return true;
 
+free_veventq:
+    close(veventq_fd);
+    iommufd_backend_free_id(idev->iommufd, veventq_id);
 free_viommu:
     iommufd_backend_free_id(idev->iommufd, viommu_id);
     return false;
hw/arm/trace-events +1 -0
index c4262bb2be..5afbceee83 100644
@@ -75,6 +75,7 @@ smmuv3_accel_install_ste(uint32_t vsid, const char * type, uint32_t hwpt_id) "vS
 # tegra241-cmdqv
 tegra241_cmdqv_read_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x"
 tegra241_cmdqv_write_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x"
+tegra241_cmdqv_err_map(uint32_t map3, uint32_t map2, uint32_t map1, uint32_t map0) "hw irq received. error (hex) maps: %04X:%04X:%04X:%04X"
 tegra241_cmdqv_read_vcmdq_page0(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64
 tegra241_cmdqv_read_vcmdq_page1(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64
 tegra241_cmdqv_write_vcmdq_page0(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64
Signature · 1 line
2.43.0
#64 lore raw

Re: [PATCH v5 24/32] hw/arm/tegra241-cmdqv: Read and propagate Tegra241 CMDQV errors

Hi Shameer,

Quoted 55 lines · Shameer Kolothum

Install an event handler on the CMDQV vEVENTQ fd to read and propagate host received CMDQV errors to the guest.

The handler runs in QEMU's main loop, using a non-blocking fd registered via qemu_set_fd_handler().

---
 hw/arm/tegra241-cmdqv.c | 60 +++++++++++++++++++++++++++++++++++++++++
 hw/arm/trace-events     |  1 +
 2 files changed, 61 insertions(+)
hw/arm/tegra241-cmdqv.c +27 -0
index 853c548e58..0bba2c1801 100644
@@ -14,6 +14,7 @@
 
 #include "hw/arm/smmuv3.h"
 #include "hw/arm/smmuv3-common.h"
+#include "hw/core/irq.h"
 #include "smmuv3-accel.h"
 #include "tegra241-cmdqv.h"
 #include "trace.h"
@@ -736,6 +737,48 @@ out:
     trace_tegra241_cmdqv_write_mmio(offset, value, size);
 }
 
+static void tegra241_cmdqv_event_read(void *opaque)
+{
+    Tegra241CMDQV *cmdqv = opaque;
+    IOMMUFDVeventq *veventq = cmdqv->veventq;
+    struct {
+        struct iommufd_vevent_header hdr;
+        struct iommu_vevent_tegra241_cmdqv vevent;
+    } buf;
+    Error *local_err = NULL;
+
+    if (!smmuv3_accel_event_read_validate(veventq,
+                                          IOMMU_VEVENTQ_TYPE_TEGRA241_CMDQV,
+                                          &buf, sizeof(buf), &local_err)) {
+        warn_report_err_once(local_err);
+        return;
+    }
+
+    if (buf.vevent.lvcmdq_err_map[0] || buf.vevent.lvcmdq_err_map[1]) {
+        cmdqv->vintf_cmdq_err_map[0] =
+            extract64(buf.vevent.lvcmdq_err_map[0], 0, 32);
+        cmdqv->vintf_cmdq_err_map[1] =
+            extract64(buf.vevent.lvcmdq_err_map[0], 32, 32);
+        cmdqv->vintf_cmdq_err_map[2] =
+            extract64(buf.vevent.lvcmdq_err_map[1], 0, 32);
+        cmdqv->vintf_cmdq_err_map[3] =
+            extract64(buf.vevent.lvcmdq_err_map[1], 32, 32);

hw/arm/tegra241-cmdqv.c out: · line 775

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10 -0
+        /*
+         * CMDQV_CMDQ_ERR_MAP and VINTF0_LVCMDQ_ERR_MAP are distinct
+         * registers (different MMIO offsets). With only VINTF0 exposed
+         * they carry the same data, so mirror.
+         */
+        for (int i = 0; i < 4; i++) {
+            cmdqv->cmdq_err_map[i] = cmdqv->vintf_cmdq_err_map[i];
+        }
+        cmdqv->vi_err_map[0] |= 0x1;
+        qemu_irq_pulse(cmdqv->irq);

I don't see SMMU_CMDQV_VI_ERR_MAP_0 updated anywhere. If I understand correctly, if you get an error on any VCMDQ you should update it. Also the triggering of the irq should depnd on SMMU_CMDQ_VI_INT_MASK0, no?

Quoted 60 lines
hw/arm/tegra241-cmdqv.c +23 -0
+        trace_tegra241_cmdqv_err_map(
+            cmdqv->vintf_cmdq_err_map[3], cmdqv->vintf_cmdq_err_map[2],
+            cmdqv->vintf_cmdq_err_map[1], cmdqv->vintf_cmdq_err_map[0]);
+    }
+}
+
 static void tegra241_cmdqv_free_viommu(SMMUv3State *s)
 {
     SMMUv3AccelState *accel = s->s_accel;
@@ -747,6 +790,7 @@ static void tegra241_cmdqv_free_viommu(SMMUv3State *s)
         return;
     }
     if (veventq) {
+        qemu_set_fd_handler(veventq->veventq_fd, NULL, NULL, NULL);
         close(veventq->veventq_fd);
         iommufd_backend_free_id(viommu->iommufd, veventq->veventq_id);
         g_free(veventq);
@@ -762,6 +806,7 @@ tegra241_cmdqv_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
     Tegra241CMDQV *cmdqv = s->s_accel->cmdqv;
     uint32_t viommu_id, veventq_id, veventq_fd;
     IOMMUFDVeventq *veventq;
+    int flags;
 
     if (!iommufd_backend_alloc_viommu(idev->iommufd, idev->devid,
                                       IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV,
@@ -780,14 +825,29 @@ tegra241_cmdqv_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
         goto free_viommu;
     }
 
+    flags = fcntl(veventq_fd, F_GETFL);
+    if (flags < 0) {
+        error_setg(errp, "Failed to get flags for vEVENTQ fd");
+        goto free_veventq;
+    }
+    if (fcntl(veventq_fd, F_SETFL, O_NONBLOCK | flags) < 0) {
+        error_setg(errp, "Failed to set O_NONBLOCK on vEVENTQ fd");
+        goto free_veventq;
+    }
+
     veventq = g_new(IOMMUFDVeventq, 1);
     veventq->veventq_id = veventq_id;
     veventq->veventq_fd = veventq_fd;
     cmdqv->veventq = veventq;
 
+    /* Set up event handler for veventq fd */
+    qemu_set_fd_handler(veventq_fd, tegra241_cmdqv_event_read, NULL, cmdqv);
     *out_viommu_id = viommu_id;
     return true;
 
+free_veventq:
+    close(veventq_fd);
+    iommufd_backend_free_id(idev->iommufd, veventq_id);
 free_viommu:
     iommufd_backend_free_id(idev->iommufd, viommu_id);
     return false;
hw/arm/trace-events +0 -0
index c4262bb2be..5afbceee83 100644

hw/arm/trace-events smmuv3_accel_install_ste(uint32_t vsid, const char * type, uint32_t hwpt_id) "vS · line 81

Quoted 7 lines
hw/arm/trace-events +1 -0
@@ -75,6 +75,7 @@ smmuv3_accel_install_ste(uint32_t vsid, const char * type, uint32_t hwpt_id) "vS
 # tegra241-cmdqv
 tegra241_cmdqv_read_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x"
 tegra241_cmdqv_write_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x"
+tegra241_cmdqv_err_map(uint32_t map3, uint32_t map2, uint32_t map1, uint32_t map0) "hw irq received. error (hex) maps: %04X:%04X:%04X:%04X"
 tegra241_cmdqv_read_vcmdq_page0(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64
 tegra241_cmdqv_read_vcmdq_page1(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64
 tegra241_cmdqv_write_vcmdq_page0(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64

Thanks

Eric

#65 lore raw

RE: [PATCH v5 24/32] hw/arm/tegra241-cmdqv: Read and propagate Tegra241 CMDQV errors

Hi Eric,

Quoted 86 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 26 May 2026 09:27
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 24/32] hw/arm/tegra241-cmdqv: Read and propagate Tegra241 CMDQV errors

External email: Use caution opening links or attachments

Hi Shameer,

Quoted 10 lines · Shameer Kolothum

Install an event handler on the CMDQV vEVENTQ fd to read and propagate host received CMDQV errors to the guest.

The handler runs in QEMU's main loop, using a non-blocking fd registered via qemu_set_fd_handler().

---
 hw/arm/tegra241-cmdqv.c | 60

+++++++++++++++++++++++++++++++++++++++++

Quoted 37 lines
 hw/arm/trace-events     |  1 +
 2 files changed, 61 insertions(+)
hw/arm/tegra241-cmdqv.c +19 -0
index 853c548e58..0bba2c1801 100644
@@ -14,6 +14,7 @@
 
 #include "hw/arm/smmuv3.h"
 #include "hw/arm/smmuv3-common.h"
+#include "hw/core/irq.h"
 #include "smmuv3-accel.h"
 #include "tegra241-cmdqv.h"
 #include "trace.h"
@@ -736,6 +737,48 @@ out:
     trace_tegra241_cmdqv_write_mmio(offset, value, size);
 }
 
+static void tegra241_cmdqv_event_read(void *opaque)
+{
+    Tegra241CMDQV *cmdqv = opaque;
+    IOMMUFDVeventq *veventq = cmdqv->veventq;
+    struct {
+        struct iommufd_vevent_header hdr;
+        struct iommu_vevent_tegra241_cmdqv vevent;
+    } buf;
+    Error *local_err = NULL;
+
+    if (!smmuv3_accel_event_read_validate(veventq,
+                                          IOMMU_VEVENTQ_TYPE_TEGRA241_CMDQV,
+                                          &buf, sizeof(buf), &local_err)) {
+        warn_report_err_once(local_err);
+        return;
+    }
+
+    if (buf.vevent.lvcmdq_err_map[0] || buf.vevent.lvcmdq_err_map[1]) {

hw/arm/tegra241-cmdqv.c out: · line 767

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10 -0
+        cmdqv->vintf_cmdq_err_map[0] =
+            extract64(buf.vevent.lvcmdq_err_map[0], 0, 32);
+        cmdqv->vintf_cmdq_err_map[1] =
+            extract64(buf.vevent.lvcmdq_err_map[0], 32, 32);
+        cmdqv->vintf_cmdq_err_map[2] =
+            extract64(buf.vevent.lvcmdq_err_map[1], 0, 32);
+        cmdqv->vintf_cmdq_err_map[3] =
+            extract64(buf.vevent.lvcmdq_err_map[1], 32, 32);
+        /*
+         * CMDQV_CMDQ_ERR_MAP and VINTF0_LVCMDQ_ERR_MAP are

distinct

hw/arm/tegra241-cmdqv.c out: · line 775

Quoted 8 lines
hw/arm/tegra241-cmdqv.c +8 -0
+         * registers (different MMIO offsets). With only VINTF0 exposed
+         * they carry the same data, so mirror.
+         */
+        for (int i = 0; i < 4; i++) {
+            cmdqv->cmdq_err_map[i] = cmdqv->vintf_cmdq_err_map[i];
+        }
+        cmdqv->vi_err_map[0] |= 0x1;
+        qemu_irq_pulse(cmdqv->irq);
Quoted 3 lines

I don't see SMMU_CMDQV_VI_ERR_MAP_0 updated anywhere. If I understand
correctly, if you get an error on any VCMDQ you should update it.

This one above does that:
cmdqv->vi_err_map[0] |= 0x1; it sets bit 0, which is the error bit for VINTF0 (the only VINTF QEMU exposes).

I will add a comment to make it clear.

Also

Quoted 1 line

the triggering of the irq should depnd on SMMU_CMDQ_VI_INT_MASK0, no?

Right. will add:

if (!(cmdqv->vi_int_mask[0] & BIT(0))) {
        qemu_irq_pulse(cmdqv->irq);
    }

Thanks,
Shameer

#66 lore raw

Re: [PATCH v5 24/32] hw/arm/tegra241-cmdqv: Read and propagate Tegra241 CMDQV errors

Quoted 88 lines · Shameer Kolothum Thodi

Hi Eric,

Quoted 86 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 26 May 2026 09:27
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 24/32] hw/arm/tegra241-cmdqv: Read and propagate Tegra241 CMDQV errors

External email: Use caution opening links or attachments

Hi Shameer,

Quoted 10 lines · Shameer Kolothum

Install an event handler on the CMDQV vEVENTQ fd to read and propagate host received CMDQV errors to the guest.

The handler runs in QEMU's main loop, using a non-blocking fd registered via qemu_set_fd_handler().

---
 hw/arm/tegra241-cmdqv.c | 60

+++++++++++++++++++++++++++++++++++++++++

Quoted 37 lines
 hw/arm/trace-events     |  1 +
 2 files changed, 61 insertions(+)
hw/arm/tegra241-cmdqv.c +19 -0
index 853c548e58..0bba2c1801 100644
@@ -14,6 +14,7 @@
 
 #include "hw/arm/smmuv3.h"
 #include "hw/arm/smmuv3-common.h"
+#include "hw/core/irq.h"
 #include "smmuv3-accel.h"
 #include "tegra241-cmdqv.h"
 #include "trace.h"
@@ -736,6 +737,48 @@ out:
     trace_tegra241_cmdqv_write_mmio(offset, value, size);
 }
 
+static void tegra241_cmdqv_event_read(void *opaque)
+{
+    Tegra241CMDQV *cmdqv = opaque;
+    IOMMUFDVeventq *veventq = cmdqv->veventq;
+    struct {
+        struct iommufd_vevent_header hdr;
+        struct iommu_vevent_tegra241_cmdqv vevent;
+    } buf;
+    Error *local_err = NULL;
+
+    if (!smmuv3_accel_event_read_validate(veventq,
+                                          IOMMU_VEVENTQ_TYPE_TEGRA241_CMDQV,
+                                          &buf, sizeof(buf), &local_err)) {
+        warn_report_err_once(local_err);
+        return;
+    }
+
+    if (buf.vevent.lvcmdq_err_map[0] || buf.vevent.lvcmdq_err_map[1]) {

hw/arm/tegra241-cmdqv.c out: · line 767

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10 -0
+        cmdqv->vintf_cmdq_err_map[0] =
+            extract64(buf.vevent.lvcmdq_err_map[0], 0, 32);
+        cmdqv->vintf_cmdq_err_map[1] =
+            extract64(buf.vevent.lvcmdq_err_map[0], 32, 32);
+        cmdqv->vintf_cmdq_err_map[2] =
+            extract64(buf.vevent.lvcmdq_err_map[1], 0, 32);
+        cmdqv->vintf_cmdq_err_map[3] =
+            extract64(buf.vevent.lvcmdq_err_map[1], 32, 32);
+        /*
+         * CMDQV_CMDQ_ERR_MAP and VINTF0_LVCMDQ_ERR_MAP are

distinct

Quoted 8 lines
hw/arm/tegra241-cmdqv.c +8 -0
+         * registers (different MMIO offsets). With only VINTF0 exposed
+         * they carry the same data, so mirror.
+         */
+        for (int i = 0; i < 4; i++) {
+            cmdqv->cmdq_err_map[i] = cmdqv->vintf_cmdq_err_map[i];
+        }
+        cmdqv->vi_err_map[0] |= 0x1;
+        qemu_irq_pulse(cmdqv->irq);
Quoted 8 lines
Quoted 3 lines

I don't see SMMU_CMDQV_VI_ERR_MAP_0 updated anywhere. If I understand
correctly, if you get an error on any VCMDQ you should update it.

This one above does that:
cmdqv->vi_err_map[0] |= 0x1; it sets bit 0, which is the error bit for VINTF0 (the only VINTF QEMU exposes).

I will add a comment to make it clear.

Ah OK. Let's add a comment then.

Eric

Quoted 12 lines

Also

Quoted 1 line

the triggering of the irq should depnd on SMMU_CMDQ_VI_INT_MASK0, no?

Right. will add:

if (!(cmdqv->vi_int_mask[0] & BIT(0))) {
        qemu_irq_pulse(cmdqv->irq);
    }

Thanks,
Shameer

#67 lore raw

PATCH v5 25/32 hw/arm/tegra241-cmdqv: Add reset handler

In reply to Shameer Kolothum

From: Nicolin Chen <[email protected]>

Introduce a reset handler for the Tegra241 CMDQV and initialize its register state.

---
 hw/arm/tegra241-cmdqv.h |  3 +++
 hw/arm/tegra241-cmdqv.c | 48 +++++++++++++++++++++++++++++++++++++++++
 hw/arm/trace-events     |  1 +
 3 files changed, 52 insertions(+)
hw/arm/tegra241-cmdqv.h +3 -0
index e9e1933a19..34cb5efb84 100644
@@ -87,6 +87,9 @@ FIELD(CONFIG, CMDQ_MAX_CLK_BATCH, 4, 8)
 FIELD(CONFIG, CMDQ_MAX_CMD_BATCH, 12, 8)
 FIELD(CONFIG, CONS_DRAM_EN, 20, 1)
 
+/* CMDQV_EN=1, PER_CMD_OFFSET=16B, CLK_BATCH=256, CMD_BATCH=32. */
+#define V_CONFIG_RESET 0x00020083
+
 REG32(PARAM, 0x4)
 FIELD(PARAM, CMDQV_VER, 0, 4)
 FIELD(PARAM, CMDQV_NUM_CMDQ_LOG2, 4, 4)
hw/arm/tegra241-cmdqv.c +48 -0

56 lines not on this page · raw message

hw/arm/trace-events +1 -0
index 5afbceee83..5ebb3dc9ea 100644
@@ -76,6 +76,7 @@ smmuv3_accel_install_ste(uint32_t vsid, const char * type, uint32_t hwpt_id) "vS
 tegra241_cmdqv_read_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x"
 tegra241_cmdqv_write_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x"
 tegra241_cmdqv_err_map(uint32_t map3, uint32_t map2, uint32_t map1, uint32_t map0) "hw irq received. error (hex) maps: %04X:%04X:%04X:%04X"
+tegra241_cmdqv_init_regs(uint32_t param) "hw info received. param: 0x%04X"
 tegra241_cmdqv_read_vcmdq_page0(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64
 tegra241_cmdqv_read_vcmdq_page1(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64
 tegra241_cmdqv_write_vcmdq_page0(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64
Signature · 1 line
2.43.0
Reviewed-by on v5 25/32
#68 lore raw

Re: [PATCH v5 25/32] hw/arm/tegra241-cmdqv: Add reset handler

Hi Shameer,

Quoted 93 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

Introduce a reset handler for the Tegra241 CMDQV and initialize its register state.

---
 hw/arm/tegra241-cmdqv.h |  3 +++
 hw/arm/tegra241-cmdqv.c | 48 +++++++++++++++++++++++++++++++++++++++++
 hw/arm/trace-events     |  1 +
 3 files changed, 52 insertions(+)
hw/arm/tegra241-cmdqv.h +3 -0
index e9e1933a19..34cb5efb84 100644
@@ -87,6 +87,9 @@ FIELD(CONFIG, CMDQ_MAX_CLK_BATCH, 4, 8)
 FIELD(CONFIG, CMDQ_MAX_CMD_BATCH, 12, 8)
 FIELD(CONFIG, CONS_DRAM_EN, 20, 1)
 
+/* CMDQV_EN=1, PER_CMD_OFFSET=16B, CLK_BATCH=256, CMD_BATCH=32. */
+#define V_CONFIG_RESET 0x00020083
+
 REG32(PARAM, 0x4)
 FIELD(PARAM, CMDQV_VER, 0, 4)
 FIELD(PARAM, CMDQV_NUM_CMDQ_LOG2, 4, 4)
hw/arm/tegra241-cmdqv.c +48 -0
index 0bba2c1801..ad64f06260 100644
@@ -853,8 +853,56 @@ free_viommu:
     return false;
 }
 
+static void tegra241_cmdqv_init_regs(SMMUv3State *s, Tegra241CMDQV *cmdqv)
+{
+    int i;
+
+    cmdqv->config = V_CONFIG_RESET;
+    cmdqv->param = FIELD_DP32(0, PARAM, CMDQV_VER, CMDQV_VER);
+    cmdqv->param = FIELD_DP32(cmdqv->param, PARAM, CMDQV_NUM_CMDQ_LOG2,
+                              CMDQV_NUM_CMDQ_LOG2);
+    cmdqv->param = FIELD_DP32(cmdqv->param, PARAM, CMDQV_NUM_SID_PER_VI_LOG2,
+                              CMDQV_NUM_SID_PER_VI_LOG2);
+    trace_tegra241_cmdqv_init_regs(cmdqv->param);
+    cmdqv->status = R_STATUS_CMDQV_ENABLED_MASK;
+
+    for (i = 0; i < 2; i++) {
+        cmdqv->vi_err_map[i] = 0;
+        cmdqv->vi_int_mask[i] = 0;
+    }
+    for (i = 0; i < 4; i++) {
+        cmdqv->cmdq_err_map[i] = 0;
+        cmdqv->vintf_cmdq_err_map[i] = 0;
+    }
+    cmdqv->vintf_config = 0;
+    cmdqv->vintf_status = 0;
+    for (i = 0; i < TEGRA241_CMDQV_MAX_CMDQ; i++) {
+        cmdqv->cmdq_alloc_map[i] = 0;
+        cmdqv->vcmdq_cons_indx[i] = 0;
+        cmdqv->vcmdq_prod_indx[i] = 0;
+        cmdqv->vcmdq_config[i] = 0;
+        cmdqv->vcmdq_status[i] = 0;
+        cmdqv->vcmdq_gerror[i] = 0;
+        cmdqv->vcmdq_gerrorn[i] = 0;
+        cmdqv->vcmdq_base[i] = 0;
+        cmdqv->vcmdq_cons_indx_base[i] = 0;
+    }
+}
+
 static void tegra241_cmdqv_reset(SMMUv3State *s)
 {
+    SMMUv3AccelState *accel = s->s_accel;
+    Tegra241CMDQV *cmdqv = accel->cmdqv;
+
+    if (!cmdqv) {
+        return;
+    }
+
+    tegra241_cmdqv_guest_unmap_vintf_page0(cmdqv);
+    tegra241_cmdqv_munmap_vintf_page0(cmdqv, NULL);
+    tegra241_cmdqv_free_all_vcmdq(cmdqv);
+
+    tegra241_cmdqv_init_regs(s, cmdqv);
 }
 
 static const MemoryRegionOps mmio_cmdqv_ops = {
hw/arm/trace-events +0 -0
index 5afbceee83..5ebb3dc9ea 100644

hw/arm/trace-events smmuv3_accel_install_ste(uint32_t vsid, const char * type, uint32_t hwpt_id) "vS · line 79

Quoted 4 lines
hw/arm/trace-events +1 -0
@@ -76,6 +76,7 @@ smmuv3_accel_install_ste(uint32_t vsid, const char * type, uint32_t hwpt_id) "vS
 tegra241_cmdqv_read_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x"
 tegra241_cmdqv_write_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x"
 tegra241_cmdqv_err_map(uint32_t map3, uint32_t map2, uint32_t map1, uint32_t map0) "hw irq received. error (hex) maps: %04X:%04X:%04X:%04X"
+tegra241_cmdqv_init_regs(uint32_t param) "hw info received. param: 0x%04X"

why 'hw info received'.

Besides

Eric

Quoted 3 lines
hw/arm/trace-events +0 -0
 tegra241_cmdqv_read_vcmdq_page0(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64
 tegra241_cmdqv_read_vcmdq_page1(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64
 tegra241_cmdqv_write_vcmdq_page0(int index, uint64_t offset0, uint64_t val) "vcmdq[%d] page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64
#69 lore raw

PATCH v5 26/32 system/physmem: Add qemu_ram_backend_pagesize_min() helper

In reply to Shameer Kolothum

Some hardware accelerators DMA directly to the host physical address of a guest-resident buffer (e.g. set up via IOMMUFD). The HPA is contiguous only within a single host backing page; a buffer larger than the smallest backing page in use may span non-contiguous host pages and fault.

Introduce qemu_ram_backend_pagesize_min(), which returns the smallest page size among RAM regions backed by a memory-backend object. Falls back to qemu_real_host_page_size() if no memory-backend RAM blocks are found. Callers can use this to bound the size of buffers that must remain host-physically contiguous.

A subsequent Tegra241 CMDQV support patch will use this to bound the command queue size exposed to the guest.

Cc Peter Xu <[email protected]>
Cc "Philippe Mathieu-Daudé" <[email protected]>
---
 include/system/ramblock.h |  1 +
 system/physmem.c          | 29 +++++++++++++++++++++++++++++
 2 files changed, 30 insertions(+)
include/system/ramblock.h +1 -0
index 4435f8d55f..7f7ea53557 100644
@@ -175,6 +175,7 @@ int qemu_ram_get_fd(const RAMBlock *rb);
 
 size_t qemu_ram_pagesize(const RAMBlock *block);
 size_t qemu_ram_pagesize_largest(void);
+size_t qemu_ram_backend_pagesize_min(void);
 #include "exec/target_page.h"
 #include "exec/hwaddr.h"
 
system/physmem.c +29 -0
index 46b36c7b10..7ccec52aa2 100644
@@ -1999,6 +1999,35 @@ size_t qemu_ram_pagesize_largest(void)
     return largest;
 }
 
+/*
+ * Returns the smallest page size among RAM regions backed by a
+ * memory-backend object. Falls back to qemu_real_host_page_size() if no
+ * memory-backend RAM blocks are found.
+ */
+size_t qemu_ram_backend_pagesize_min(void)
+{
+    RAMBlock *rb;
+    size_t pg, min_pg = SIZE_MAX;
+
+    RAMBLOCK_FOREACH(rb) {
+        MemoryRegion *mr = rb->mr;
+
+        if (!mr || !memory_region_is_ram(mr)) {
+            continue;
+        }
+        if (!object_dynamic_cast(mr->owner, TYPE_MEMORY_BACKEND)) {
+            continue;
+        }
+
+        pg = qemu_ram_pagesize(rb);
+        if (pg && pg < min_pg) {
+            min_pg = pg;
+        }
+    }
+
+    return (min_pg == SIZE_MAX) ? qemu_real_host_page_size() : min_pg;
+}
+
 static int memory_try_enable_merging(void *addr, size_t len)
 {
     if (!machine_mem_merge(current_machine)) {
Signature · 1 line
2.43.0
#70 lore raw

Re: [PATCH v5 26/32] system/physmem: Add qemu_ram_backend_pagesize_min() helper

Quoted 1 line · Shameer Kolothum

Some hardware accelerators DMA directly to the host physical address

s/DMA/perform DMA accesses

Quoted 7 lines

of a guest-resident buffer (e.g. set up via IOMMUFD). The HPA is contiguous only within a single host backing page; a buffer larger than the smallest backing page in use may span non-contiguous host pages and fault.

Introduce qemu_ram_backend_pagesize_min(), which returns the smallest page size among RAM regions backed by a memory-backend object. Falls

Don't you have means to be more precise? I mean you know the GPA of the cmdq, can't you make sure it belongs to a specific ramblock and check its min page_size?

By the way there is find_min_backend_pagesize() that exists. Can't you reuse that?

Thanks

Eric

Quoted 67 lines

back to qemu_real_host_page_size() if no memory-backend RAM blocks are found. Callers can use this to bound the size of buffers that must remain host-physically contiguous.

A subsequent Tegra241 CMDQV support patch will use this to bound the command queue size exposed to the guest.

Cc Peter Xu <[email protected]>
Cc "Philippe Mathieu-Daudé" <[email protected]>
---
 include/system/ramblock.h |  1 +
 system/physmem.c          | 29 +++++++++++++++++++++++++++++
 2 files changed, 30 insertions(+)
include/system/ramblock.h +1 -0
index 4435f8d55f..7f7ea53557 100644
@@ -175,6 +175,7 @@ int qemu_ram_get_fd(const RAMBlock *rb);
 
 size_t qemu_ram_pagesize(const RAMBlock *block);
 size_t qemu_ram_pagesize_largest(void);
+size_t qemu_ram_backend_pagesize_min(void);
 #include "exec/target_page.h"
 #include "exec/hwaddr.h"
 
system/physmem.c +29 -0
index 46b36c7b10..7ccec52aa2 100644
@@ -1999,6 +1999,35 @@ size_t qemu_ram_pagesize_largest(void)
     return largest;
 }
 
+/*
+ * Returns the smallest page size among RAM regions backed by a
+ * memory-backend object. Falls back to qemu_real_host_page_size() if no
+ * memory-backend RAM blocks are found.
+ */
+size_t qemu_ram_backend_pagesize_min(void)
+{
+    RAMBlock *rb;
+    size_t pg, min_pg = SIZE_MAX;
+
+    RAMBLOCK_FOREACH(rb) {
+        MemoryRegion *mr = rb->mr;
+
+        if (!mr || !memory_region_is_ram(mr)) {
+            continue;
+        }
+        if (!object_dynamic_cast(mr->owner, TYPE_MEMORY_BACKEND)) {
+            continue;
+        }
+
+        pg = qemu_ram_pagesize(rb);
+        if (pg && pg < min_pg) {
+            min_pg = pg;
+        }
+    }
+
+    return (min_pg == SIZE_MAX) ? qemu_real_host_page_size() : min_pg;
+}
+
 static int memory_try_enable_merging(void *addr, size_t len)
 {
     if (!machine_mem_merge(current_machine)) {
#71 lore raw

RE: [PATCH v5 26/32] system/physmem: Add qemu_ram_backend_pagesize_min() helper

Quoted 27 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 26 May 2026 10:31
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]; Peter Xu <[email protected]>; Philippe Mathieu-Daudé <[email protected]>
Subject: Re: [PATCH v5 26/32] system/physmem: Add qemu_ram_backend_pagesize_min() helper

External email: Use caution opening links or attachments

Quoted 1 line · Shameer Kolothum

Some hardware accelerators DMA directly to the host physical address

s/DMA/perform DMA accesses

Quoted 5 lines

of a guest-resident buffer (e.g. set up via IOMMUFD). The HPA is contiguous only within a single host backing page; a buffer larger than the smallest backing page in use may span non-contiguous host pages and fault.

Quoted 5 lines
Quoted 2 lines

Introduce qemu_ram_backend_pagesize_min(), which returns the smallest page size among RAM regions backed by a memory-backend object. Falls

Don't you have means to be more precise? I mean you know the GPA of the cmdq, can't you make sure it belongs to a specific ramblock and check its min page_size?

As you noted in the next patch we don’t have GPA at the time of init.

Quoted 3 lines

By the way there is find_min_backend_pagesize() that exists. Can't you reuse that?

Ah.. Thanks!. That looks like is doing the same thing. And we also have warpper that uses it, qemu_minrampagesize().

I will get rid of this patch and use that instead.

Thanks,
Shameer

#72 lore raw

PATCH v5 27/32 hw/arm/tegra241-cmdqv: Limit queue size based on backend page size

In reply to Shameer Kolothum

From: Nicolin Chen <[email protected]>

CMDQV HW reads guest queue memory by its host physical address set up via IOMMUFD. This requires the guest queue to be contiguous in both guest PA and host PA space. With Tegra241 CMDQV enabled, we must only advertise a command queue size (CMDQS) that the host can safely back with physically contiguous memory. Allowing a queue size larger than the host page size could cause the hardware to DMA across page boundaries, leading to faults.

Use qemu_ram_backend_pagesize_min() to find the smallest memory- backend page size in use, then cap IDR1.CMDQS so the guest cannot configure a command queue that exceeds that contiguous backing.

---
 hw/arm/tegra241-cmdqv.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
hw/arm/tegra241-cmdqv.c +17 -0
index ad64f06260..7f617bcc97 100644
@@ -16,6 +16,8 @@
 #include "hw/arm/smmuv3-common.h"
 #include "hw/core/irq.h"
 #include "smmuv3-accel.h"
+#include "smmuv3-internal.h"
+#include "system/ramblock.h"
 #include "tegra241-cmdqv.h"
 #include "trace.h"
 
@@ -856,6 +858,8 @@ free_viommu:
 static void tegra241_cmdqv_init_regs(SMMUv3State *s, Tegra241CMDQV *cmdqv)
 {
     int i;
+    size_t pgsize;
+    uint32_t val;
 
     cmdqv->config = V_CONFIG_RESET;
     cmdqv->param = FIELD_DP32(0, PARAM, CMDQV_VER, CMDQV_VER);
@@ -887,6 +891,19 @@ static void tegra241_cmdqv_init_regs(SMMUv3State *s, Tegra241CMDQV *cmdqv)
         cmdqv->vcmdq_base[i] = 0;
         cmdqv->vcmdq_cons_indx_base[i] = 0;
     }
+
+    /*
+     * CMDQ must not cross a physical RAM backend page. Adjust CMDQS so the
+     * queue fits entirely within the smallest backend page size, ensuring
+     * the command queue is physically contiguous in host memory.
+     *
+     *   IDR1.CMDQS = log2(max_qsz) - entry_shift
+     *
+     * where entry_shift = 4 (each CMDQ entry is 16 bytes = 2^4).
+     */
+    pgsize = qemu_ram_backend_pagesize_min();
+    val = FIELD_EX32(s->idr[1], IDR1, CMDQS);
+    s->idr[1] = FIELD_DP32(s->idr[1], IDR1, CMDQS, MIN(ctz64(pgsize) - 4, val));
 }
 
 static void tegra241_cmdqv_reset(SMMUv3State *s)
Signature · 1 line
2.43.0
#73 lore raw

Re: [PATCH v5 27/32] hw/arm/tegra241-cmdqv: Limit queue size based on backend page size

Hi Shameer,

Quoted 47 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

CMDQV HW reads guest queue memory by its host physical address set up via IOMMUFD. This requires the guest queue to be contiguous in both guest PA and host PA space. With Tegra241 CMDQV enabled, we must only advertise a command queue size (CMDQS) that the host can safely back with physically contiguous memory. Allowing a queue size larger than the host page size could cause the hardware to DMA across page boundaries, leading to faults.

Use qemu_ram_backend_pagesize_min() to find the smallest memory- backend page size in use, then cap IDR1.CMDQS so the guest cannot configure a command queue that exceeds that contiguous backing.

---
 hw/arm/tegra241-cmdqv.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
hw/arm/tegra241-cmdqv.c +5 -0
index ad64f06260..7f617bcc97 100644
@@ -16,6 +16,8 @@
 #include "hw/arm/smmuv3-common.h"
 #include "hw/core/irq.h"
 #include "smmuv3-accel.h"
+#include "smmuv3-internal.h"
+#include "system/ramblock.h"
 #include "tegra241-cmdqv.h"
 #include "trace.h"
 
@@ -856,6 +858,8 @@ free_viommu:
 static void tegra241_cmdqv_init_regs(SMMUv3State *s, Tegra241CMDQV *cmdqv)
 {
     int i;
+    size_t pgsize;
+    uint32_t val;
 
     cmdqv->config = V_CONFIG_RESET;
     cmdqv->param = FIELD_DP32(0, PARAM, CMDQV_VER, CMDQV_VER);
@@ -887,6 +891,19 @@ static void tegra241_cmdqv_init_regs(SMMUv3State *s, Tegra241CMDQV *cmdqv)
         cmdqv->vcmdq_base[i] = 0;
         cmdqv->vcmdq_cons_indx_base[i] = 0;
     }
+

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_init_regs(SMMUv3State *s, Tegra241CMDQV *cmdqv) · line 904

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10 -0
+    /*
+     * CMDQ must not cross a physical RAM backend page. Adjust CMDQS so the
+     * queue fits entirely within the smallest backend page size, ensuring
+     * the command queue is physically contiguous in host memory.
+     *
+     *   IDR1.CMDQS = log2(max_qsz) - entry_shift
+     *
+     * where entry_shift = 4 (each CMDQ entry is 16 bytes = 2^4).
+     */
+    pgsize = qemu_ram_backend_pagesize_min();

OK I see this is done at init size before we know the GPA. But I meant you may end up with a page size that is suboptimal, no?

Could you elaborate on the actual requirements with regards to huge page usage in this patch and in the cover letter?

Thanks

Eric

Quoted 5 lines
hw/arm/tegra241-cmdqv.c +2 -0
+    val = FIELD_EX32(s->idr[1], IDR1, CMDQS);
+    s->idr[1] = FIELD_DP32(s->idr[1], IDR1, CMDQS, MIN(ctz64(pgsize) - 4, val));
 }
 
 static void tegra241_cmdqv_reset(SMMUv3State *s)
#74 lore raw

RE: [PATCH v5 27/32] hw/arm/tegra241-cmdqv: Limit queue size based on backend page size

Quoted 81 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 26 May 2026 10:35
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 27/32] hw/arm/tegra241-cmdqv: Limit queue size based on backend page size

External email: Use caution opening links or attachments

Hi Shameer,

Quoted 34 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

CMDQV HW reads guest queue memory by its host physical address set up via IOMMUFD. This requires the guest queue to be contiguous in both guest PA and host PA space. With Tegra241 CMDQV enabled, we must only advertise a command queue size (CMDQS) that the host can safely back with physically contiguous memory. Allowing a queue size larger than the host page size could cause the hardware to DMA across page boundaries, leading to faults.

Use qemu_ram_backend_pagesize_min() to find the smallest memory- backend page size in use, then cap IDR1.CMDQS so the guest cannot configure a command queue that exceeds that contiguous backing.

---
 hw/arm/tegra241-cmdqv.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
hw/arm/tegra241-cmdqv.c +2 -0
index ad64f06260..7f617bcc97 100644
@@ -16,6 +16,8 @@
 #include "hw/arm/smmuv3-common.h"
 #include "hw/core/irq.h"
 #include "smmuv3-accel.h"
+#include "smmuv3-internal.h"
+#include "system/ramblock.h"
 #include "tegra241-cmdqv.h"
 #include "trace.h"
 

hw/arm/tegra241-cmdqv.c free_viommu: · line 858

Quoted 1 line
hw/arm/tegra241-cmdqv.c +0 -0
@@ -856,6 +858,8 @@ free_viommu:
 static void tegra241_cmdqv_init_regs(SMMUv3State *s, Tegra241CMDQV

*cmdqv)

hw/arm/tegra241-cmdqv.c static void

Quoted 8 lines
hw/arm/tegra241-cmdqv.c +2 -0
 {
     int i;
+    size_t pgsize;
+    uint32_t val;
 
     cmdqv->config = V_CONFIG_RESET;
     cmdqv->param = FIELD_DP32(0, PARAM, CMDQV_VER, CMDQV_VER);
@@ -887,6 +891,19 @@ static void

tegra241_cmdqv_init_regs(SMMUv3State *s, Tegra241CMDQV *cmdqv)

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_init_regs(SMMUv3State *s, Tegra241CMDQV *cmdqv)

Quoted 6 lines
hw/arm/tegra241-cmdqv.c +3 -0
         cmdqv->vcmdq_base[i] = 0;
         cmdqv->vcmdq_cons_indx_base[i] = 0;
     }
+
+    /*
+     * CMDQ must not cross a physical RAM backend page. Adjust CMDQS so

the

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_init_regs(SMMUv3State *s, Tegra241CMDQV *cmdqv) · line 904

Quoted 8 lines
hw/arm/tegra241-cmdqv.c +8 -0
+     * queue fits entirely within the smallest backend page size, ensuring
+     * the command queue is physically contiguous in host memory.
+     *
+     *   IDR1.CMDQS = log2(max_qsz) - entry_shift
+     *
+     * where entry_shift = 4 (each CMDQ entry is 16 bytes = 2^4).
+     */
+    pgsize = qemu_ram_backend_pagesize_min();
Quoted 2 lines

OK I see this is done at init size before we know the GPA. But I meant you may end up with a page size that is suboptimal, no?

Yes. The recommended CMDQV queue size is the max 8M and we recommend using hugepages to achieve that.

Quoted 2 lines

Could you elaborate on the actual requirements with regards to huge page usage in this patch and in the cover letter?

Sure. I will add that info.

Thanks,
Shameer

#75 lore raw

PATCH v5 28/32 hw/arm/smmuv3: Add per-device identifier property

In reply to Shameer Kolothum

Add an "identifier" property to the SMMUv3 device and use it when building the ACPI IORT SMMUv3 node Identifier field.

This avoids relying on device enumeration order and provides a stable per-device identifier. A subsequent patch will use the same identifier when generating the DSDT description for Tegra241 CMDQV, ensuring that the IORT and DSDT entries refer to the same SMMUv3 instance.

The identifier is assigned at pre-plug time, accounting for the ITS Group node that build_iort() places before SMMUv3 nodes in the IORT table, so that identifiers are globally unique across all IORT nodes.

No functional change: IORT blob content for bios-tables qtest is identical to before.

---
 include/hw/arm/smmuv3.h  |  1 +
 hw/arm/smmuv3.c          |  2 ++
 hw/arm/virt-acpi-build.c |  5 ++++-
 hw/arm/virt.c            | 12 ++++++++++++
 4 files changed, 19 insertions(+), 1 deletion(-)
include/hw/arm/smmuv3.h +1 -0
index aa6a79237a..0fce564619 100644
@@ -64,6 +64,7 @@ struct SMMUv3State {
     qemu_irq     irq[4];
     QemuMutex mutex;
     char *stage;
+    uint8_t identifier;
 
     /* SMMU has HW accelerator support for nested S1 + s2 */
     bool accel;
hw/arm/smmuv3.c +2 -0
index 665e6a2538..763b86c417 100644
@@ -2126,6 +2126,8 @@ static const Property smmuv3_properties[] = {
      * Defaults to stage 1
      */
     DEFINE_PROP_STRING("stage", SMMUv3State, stage),
+    /* Identifier used for ACPI IORT SMMUv3 (and DSDT for CMDQV) generation */
+    DEFINE_PROP_UINT8("identifier", SMMUv3State, identifier, 0),
     DEFINE_PROP_BOOL("accel", SMMUv3State, accel, false),
     /* GPA of MSI doorbell, for SMMUv3 accel use. */
     DEFINE_PROP_UINT64("msi-gpa", SMMUv3State, msi_gpa, 0),
hw/arm/virt-acpi-build.c +4 -1
index b00f3477ca..9d05982137 100644
@@ -349,6 +349,7 @@ static int iort_idmap_compare(gconstpointer a, gconstpointer b)
 typedef struct AcpiIortSMMUv3Dev {
     int irq;
     hwaddr base;
+    uint8_t id;
     GArray *rc_smmu_idmaps;
     /* Offset of the SMMUv3 IORT Node relative to the start of the IORT */
     size_t offset;
@@ -411,6 +412,7 @@ static int populate_smmuv3_dev(VirtMachineState *vms, GArray *sdev_blob)
                                                &error_abort));
         sdev.accel = object_property_get_bool(obj, "accel", &error_abort);
         sdev.ats = smmuv3_ats_enabled(ARM_SMMUV3(obj));
+        sdev.id = object_property_get_uint(obj, "identifier", &error_abort);
         pbus = PLATFORM_BUS_DEVICE(vms->platform_bus_dev);
         sbdev = SYS_BUS_DEVICE(obj);
         sdev.base = platform_bus_get_mmio_addr(pbus, sbdev, 0);
@@ -637,7 +639,8 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
                      (ID_MAPPING_ENTRY_SIZE * smmu_mapping_count);
         build_append_int_noprefix(table_data, node_size, 2); /* Length */
         build_append_int_noprefix(table_data, 4, 1); /* Revision */
-        build_append_int_noprefix(table_data, id++, 4); /* Identifier */
+        build_append_int_noprefix(table_data, sdev->id, 4); /* Identifier */
+        id++;  /* advance shared counter for RC/RMR node uniqueness */
         /* Number of ID mappings */
         build_append_int_noprefix(table_data, smmu_mapping_count, 4);
         /* Reference to ID Array */
hw/arm/virt.c +12 -0
index 2add7401a1..d8d27f2ef6 100644
@@ -254,6 +254,9 @@ static MemMapEntry extended_memmap[] = {
     /* Any CXL Fixed memory windows come here */
 };
 
+/* Counts SMMUv3 devices plugged; used to assign stable IORT identifiers */
+static uint8_t smmuv3_dev_id;
+
 static const int a15irqmap[] = {
     [VIRT_UART0] = 1,
     [VIRT_RTC] = 2,
@@ -3830,6 +3833,15 @@ static void virt_machine_device_pre_plug_cb(HotplugHandler *hotplug_dev,
                                      OBJECT(vms->sysmem), NULL);
             object_property_set_link(OBJECT(dev), "secure-memory",
                                      OBJECT(vms->secure_sysmem), NULL);
+            /*
+             * In build_iort(), the ITS node(id=0) precedes SMMUv3 nodes
+             * when present. Account for it so this SMMUv3's identifier
+             * is globally unique across all IORT nodes.
+             */
+            uint8_t its_offset = (vms->msi_controller == VIRT_MSI_CTRL_ITS)
+                                  ? 1 : 0;
+            object_property_set_uint(OBJECT(dev), "identifier",
+                                     its_offset + smmuv3_dev_id++, NULL);
         }
         if (object_property_get_bool(OBJECT(dev), "accel", &error_abort)) {
             hwaddr db_start = 0;
Signature · 1 line
2.43.0
#76 lore raw

PATCH v5 29/32 hw/arm/smmuv3-accel: Introduce helper to query CMDQV type

In reply to Shameer Kolothum

Introduce a SMMUv3AccelCmdqvType enum and a helper to query the CMDQV implementation type associated with an accelerated SMMUv3 instance.

A subsequent patch will use this helper when generating the Tegra241 CMDQV DSDT.

---
 hw/arm/smmuv3-accel.h       | 10 ++++++++++
 hw/arm/smmuv3-accel-stubs.c |  5 +++++
 hw/arm/smmuv3-accel.c       | 12 ++++++++++++
 hw/arm/tegra241-cmdqv.c     |  6 ++++++
 4 files changed, 33 insertions(+)
hw/arm/smmuv3-accel.h +10 -0
index 241639ec8e..8cf35c2936 100644
@@ -16,6 +16,11 @@
 #include <linux/iommufd.h>
 #endif
 
+typedef enum SMMUv3AccelCmdqvType {
+    SMMUV3_CMDQV_NONE = 0,
+    SMMUV3_CMDQV_TEGRA241,
+} SMMUv3AccelCmdqvType;
+
 /*
  * CMDQ-Virtualization (CMDQV) hardware support, extends the SMMUv3 to
  * support multiple VCMDQs with virtualization capabilities.
@@ -43,6 +48,10 @@ typedef struct SMMUv3AccelCmdqvOps {
      * If NULL, the viommu_id is freed directly via iommufd_backend_free_id().
      */
     void (*free_viommu)(SMMUv3State *s);
+    /**
+     * @get_type: Optional callback. Return the CMDQV implementation type.
+     */
+    SMMUv3AccelCmdqvType (*get_type)(void);
     /**
      * @reset: Optional callback. Reset CMDQV state.
      */
@@ -91,5 +100,6 @@ bool smmuv3_accel_alloc_veventq(SMMUv3State *s, Error **errp);
 bool smmuv3_accel_event_read_validate(IOMMUFDVeventq *veventq, uint32_t type,
                                       void *buf, size_t size, Error **errp);
 void smmuv3_accel_reset(SMMUv3State *s);
+SMMUv3AccelCmdqvType smmuv3_accel_cmdqv_type(Object *obj);
 
 #endif /* HW_ARM_SMMUV3_ACCEL_H */
hw/arm/smmuv3-accel-stubs.c +5 -0
index 9e6c44a282..147ae06163 100644
@@ -57,3 +57,8 @@ bool smmuv3_accel_event_read_validate(IOMMUFDVeventq *veventq, uint32_t type,
 void smmuv3_accel_reset(SMMUv3State *s)
 {
 }
+
+SMMUv3AccelCmdqvType smmuv3_accel_cmdqv_type(Object *obj)
+{
+    return SMMUV3_CMDQV_NONE;
+}
hw/arm/smmuv3-accel.c +12 -0
index acc0ca5251..dbb50d1795 100644
@@ -1062,6 +1062,18 @@ static void smmuv3_accel_as_init(SMMUv3State *s)
     address_space_init(shared_as_sysmem, &root, "smmuv3-accel-as-sysmem");
 }
 
+SMMUv3AccelCmdqvType smmuv3_accel_cmdqv_type(Object *obj)
+{
+    SMMUv3State *s = ARM_SMMUV3(obj);
+    SMMUv3AccelState *accel = s->s_accel;
+
+    if (!accel || !accel->cmdqv_ops || !accel->cmdqv_ops->get_type) {
+        return SMMUV3_CMDQV_NONE;
+    }
+
+    return accel->cmdqv_ops->get_type();
+}
+
 static void smmuv3_machine_done(Notifier *notifier, void *data)
 {
     SMMUv3State *s = container_of(notifier, SMMUv3State, machine_done);
hw/arm/tegra241-cmdqv.c +6 -0
index 7f617bcc97..fb4301aa7d 100644
@@ -952,6 +952,11 @@ static bool tegra241_cmdqv_init(SMMUv3State *s, Error **errp)
     return true;
 }
 
+static SMMUv3AccelCmdqvType tegra241_cmdqv_get_type(void)
+{
+    return SMMUV3_CMDQV_TEGRA241;
+}
+
 static bool tegra241_cmdqv_probe(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
                                  Error **errp)
 {
@@ -992,6 +997,7 @@ static const SMMUv3AccelCmdqvOps tegra241_cmdqv_ops = {
     .init = tegra241_cmdqv_init,
     .alloc_viommu = tegra241_cmdqv_alloc_viommu,
     .free_viommu = tegra241_cmdqv_free_viommu,
+    .get_type = tegra241_cmdqv_get_type,
     .reset = tegra241_cmdqv_reset,
 };
 
Signature · 1 line
2.43.0
#77 lore raw

PATCH v5 30/32 hw/arm/virt-acpi: Advertise Tegra241 CMDQV nodes in DSDT

In reply to Shameer Kolothum

From: Nicolin Chen <[email protected]>

Add ACPI DSDT support for Tegra241 CMDQV when the SMMUv3 instance is created with tegra241-cmdqv.

For each accelerated SMMUv3 instance, add a Tegra241 CMDQV device object under the DSDT \_SB namespace, with HID "NVDA200C" and a UID that matches the Identifier of the corresponding SMMUv3 IORT node, so the guest OS can associate the DSDT device with the right SMMU. The _CRS covers the CMDQV MMIO aperture plus its interrupt, and _CCA declares I/O cache coherency.

See ACPI Specification 6.5, Section 6 (Device Configuration) for _HID/_UID/_CCA/_CRS.

Generated DSDT entry for a CMDQV instance paired with SMMUv3 Identifier=1:
...
Device (CV01)

  {
      Name (_HID, "NVDA200C")  // _HID: Hardware ID
      Name (_UID, One)  // _UID: Unique ID
      Name (_CCA, One)  // _CCA: Cache Coherency Attribute
      Name (_CRS, ResourceTemplate ()  // _CRS: Current Resource Settings
      {
          QWordMemory (ResourceProducer, PosDecode, MinFixed, MaxFixed, Cacheable, ReadWrite,
              0x0000000000000000, // Granularity
              0x000000000C080000, // Range Minimum
              0x000000000C0CFFFF, // Range Maximum
              0x0000000000000000, // Translation Offset
              0x0000000000050000, // Length
              ,, , AddressRangeMemory, TypeStatic)
          Interrupt (ResourceConsumer, Edge, ActiveHigh, Exclusive, ,, )
          {
              0x00000094,
          }
      })

}
...
Generated IORT SMMUv3 node (Identifier = 1):

...
[048h 0072 001h] Type : 04
[049h 0073 002h] Length : 0058
[04Bh 0075 001h] Revision : 04
[04Ch 0076 004h] Identifier : 00000001
[050h 0080 004h] Mapping Count : 00000001
[054h 0084 004h] Mapping Offset : 00000044
...

---
 hw/arm/virt-acpi-build.c | 52 ++++++++++++++++++++++++++++++++++++++++
 hw/arm/trace-events      |  1 +
 2 files changed, 53 insertions(+)
hw/arm/virt-acpi-build.c +52 -0
index 9d05982137..99490aa7b1 100644
@@ -65,6 +65,9 @@
 #include "target/arm/cpu.h"
 #include "target/arm/multiprocessing.h"
 
+#include "smmuv3-accel.h"
+#include "tegra241-cmdqv.h"
+
 #define ARM_SPI_BASE 32
 
 #define ACPI_BUILD_TABLE_SIZE             0x20000
@@ -1121,6 +1124,51 @@ static void build_fadt_rev6(GArray *table_data, BIOSLinker *linker,
     build_fadt(table_data, linker, &fadt, vms->oem_id, vms->oem_table_id);
 }
 
+static void acpi_dsdt_add_tegra241_cmdqv(Aml *scope, VirtMachineState *vms)
+{
+    for (int i = 0; i < vms->smmuv3_devices->len; i++) {
+        Object *obj = OBJECT(g_ptr_array_index(vms->smmuv3_devices, i));
+        PlatformBusDevice *pbus;
+        Aml *dev, *crs, *addr;
+        SysBusDevice *sbdev;
+        hwaddr base;
+        uint32_t id;
+        int irq;
+
+        if (smmuv3_accel_cmdqv_type(obj) != SMMUV3_CMDQV_TEGRA241) {
+            continue;
+        }
+        id = object_property_get_uint(obj, "identifier", &error_abort);
+        pbus = PLATFORM_BUS_DEVICE(vms->platform_bus_dev);
+        sbdev = SYS_BUS_DEVICE(obj);
+        base = platform_bus_get_mmio_addr(pbus, sbdev, 1);
+        base += vms->memmap[VIRT_PLATFORM_BUS].base;
+        irq = platform_bus_get_irqn(pbus, sbdev, NUM_SMMU_IRQS);
+        irq += vms->irqmap[VIRT_PLATFORM_BUS];
+        irq += ARM_SPI_BASE;
+
+        dev = aml_device("CV%.02u", id);
+        aml_append(dev, aml_name_decl("_HID", aml_string("NVDA200C")));
+        aml_append(dev, aml_name_decl("_UID", aml_int(id)));
+        aml_append(dev, aml_name_decl("_CCA", aml_int(1)));
+
+        crs = aml_resource_template();
+        addr = aml_qword_memory(AML_POS_DECODE, AML_MIN_FIXED, AML_MAX_FIXED,
+                                AML_CACHEABLE, AML_READ_WRITE, 0x0, base,
+                                base + TEGRA241_CMDQV_IO_LEN - 0x1, 0x0,
+                                TEGRA241_CMDQV_IO_LEN);
+        aml_append(crs, addr);
+        aml_append(crs, aml_interrupt(AML_CONSUMER, AML_EDGE,
+                                      AML_ACTIVE_HIGH, AML_EXCLUSIVE,
+                                      (uint32_t *)&irq, 1));
+        aml_append(dev, aml_name_decl("_CRS", crs));
+
+        aml_append(scope, dev);
+
+        trace_virt_acpi_dsdt_tegra241_cmdqv(id, base, irq);
+    }
+}
+
 /* DSDT */
 static void
 build_dsdt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
@@ -1185,6 +1233,10 @@ build_dsdt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
     acpi_dsdt_add_tpm(scope, vms);
 #endif
 
+    if (!vms->legacy_smmuv3_present) {
+        acpi_dsdt_add_tegra241_cmdqv(scope, vms);
+    }
+
     aml_append(dsdt, scope);
 
     pci0_scope = aml_scope("\\_SB.PCI0");
hw/arm/trace-events +1 -0
index 5ebb3dc9ea..e49a2b324d 100644
@@ -9,6 +9,7 @@ omap1_lpg_led(const char *onoff) "omap1 LPG: LED is %s"
 
 # virt-acpi-build.c
 virt_acpi_setup(void) "No fw cfg or ACPI disabled. Bailing out."
+virt_acpi_dsdt_tegra241_cmdqv(int smmu_id, uint64_t base, uint32_t irq) "DSDT: add cmdqv node for (id=%d), base=0x%" PRIx64 ", irq=%d"
 
 # smmu-common.c
 smmu_add_mr(const char *name) "%s"
Signature · 1 line
2.43.0
Reviewed-by on v5 30/32
#78 lore raw

Re: [PATCH v5 30/32] hw/arm/virt-acpi: Advertise Tegra241 CMDQV nodes in DSDT

Quoted 49 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

Add ACPI DSDT support for Tegra241 CMDQV when the SMMUv3 instance is created with tegra241-cmdqv.

For each accelerated SMMUv3 instance, add a Tegra241 CMDQV device object under the DSDT \_SB namespace, with HID "NVDA200C" and a UID that matches the Identifier of the corresponding SMMUv3 IORT node, so the guest OS can associate the DSDT device with the right SMMU. The _CRS covers the CMDQV MMIO aperture plus its interrupt, and _CCA declares I/O cache coherency.

See ACPI Specification 6.5, Section 6 (Device Configuration) for _HID/_UID/_CCA/_CRS.

Generated DSDT entry for a CMDQV instance paired with SMMUv3 Identifier=1:
...
Device (CV01)

  {
      Name (_HID, "NVDA200C")  // _HID: Hardware ID
      Name (_UID, One)  // _UID: Unique ID
      Name (_CCA, One)  // _CCA: Cache Coherency Attribute
      Name (_CRS, ResourceTemplate ()  // _CRS: Current Resource Settings
      {
          QWordMemory (ResourceProducer, PosDecode, MinFixed, MaxFixed, Cacheable, ReadWrite,
              0x0000000000000000, // Granularity
              0x000000000C080000, // Range Minimum
              0x000000000C0CFFFF, // Range Maximum
              0x0000000000000000, // Translation Offset
              0x0000000000050000, // Length
              ,, , AddressRangeMemory, TypeStatic)
          Interrupt (ResourceConsumer, Edge, ActiveHigh, Exclusive, ,, )
          {
              0x00000094,
          }
      })

}
...
Generated IORT SMMUv3 node (Identifier = 1):

...
[048h 0072 001h] Type : 04
[049h 0073 002h] Length : 0058
[04Bh 0075 001h] Revision : 04
[04Ch 0076 004h] Identifier : 00000001
[050h 0080 004h] Mapping Count : 00000001
[054h 0084 004h] Mapping Offset : 00000044
...

Quoted 3 lines

Eric

Quoted 94 lines
---
 hw/arm/virt-acpi-build.c | 52 ++++++++++++++++++++++++++++++++++++++++
 hw/arm/trace-events      |  1 +
 2 files changed, 53 insertions(+)
hw/arm/virt-acpi-build.c +52 -0
index 9d05982137..99490aa7b1 100644
@@ -65,6 +65,9 @@
 #include "target/arm/cpu.h"
 #include "target/arm/multiprocessing.h"
 
+#include "smmuv3-accel.h"
+#include "tegra241-cmdqv.h"
+
 #define ARM_SPI_BASE 32
 
 #define ACPI_BUILD_TABLE_SIZE             0x20000
@@ -1121,6 +1124,51 @@ static void build_fadt_rev6(GArray *table_data, BIOSLinker *linker,
     build_fadt(table_data, linker, &fadt, vms->oem_id, vms->oem_table_id);
 }
 
+static void acpi_dsdt_add_tegra241_cmdqv(Aml *scope, VirtMachineState *vms)
+{
+    for (int i = 0; i < vms->smmuv3_devices->len; i++) {
+        Object *obj = OBJECT(g_ptr_array_index(vms->smmuv3_devices, i));
+        PlatformBusDevice *pbus;
+        Aml *dev, *crs, *addr;
+        SysBusDevice *sbdev;
+        hwaddr base;
+        uint32_t id;
+        int irq;
+
+        if (smmuv3_accel_cmdqv_type(obj) != SMMUV3_CMDQV_TEGRA241) {
+            continue;
+        }
+        id = object_property_get_uint(obj, "identifier", &error_abort);
+        pbus = PLATFORM_BUS_DEVICE(vms->platform_bus_dev);
+        sbdev = SYS_BUS_DEVICE(obj);
+        base = platform_bus_get_mmio_addr(pbus, sbdev, 1);
+        base += vms->memmap[VIRT_PLATFORM_BUS].base;
+        irq = platform_bus_get_irqn(pbus, sbdev, NUM_SMMU_IRQS);
+        irq += vms->irqmap[VIRT_PLATFORM_BUS];
+        irq += ARM_SPI_BASE;
+
+        dev = aml_device("CV%.02u", id);
+        aml_append(dev, aml_name_decl("_HID", aml_string("NVDA200C")));
+        aml_append(dev, aml_name_decl("_UID", aml_int(id)));
+        aml_append(dev, aml_name_decl("_CCA", aml_int(1)));
+
+        crs = aml_resource_template();
+        addr = aml_qword_memory(AML_POS_DECODE, AML_MIN_FIXED, AML_MAX_FIXED,
+                                AML_CACHEABLE, AML_READ_WRITE, 0x0, base,
+                                base + TEGRA241_CMDQV_IO_LEN - 0x1, 0x0,
+                                TEGRA241_CMDQV_IO_LEN);
+        aml_append(crs, addr);
+        aml_append(crs, aml_interrupt(AML_CONSUMER, AML_EDGE,
+                                      AML_ACTIVE_HIGH, AML_EXCLUSIVE,
+                                      (uint32_t *)&irq, 1));
+        aml_append(dev, aml_name_decl("_CRS", crs));
+
+        aml_append(scope, dev);
+
+        trace_virt_acpi_dsdt_tegra241_cmdqv(id, base, irq);
+    }
+}
+
 /* DSDT */
 static void
 build_dsdt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
@@ -1185,6 +1233,10 @@ build_dsdt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
     acpi_dsdt_add_tpm(scope, vms);
 #endif
 
+    if (!vms->legacy_smmuv3_present) {
+        acpi_dsdt_add_tegra241_cmdqv(scope, vms);
+    }
+
     aml_append(dsdt, scope);
 
     pci0_scope = aml_scope("\\_SB.PCI0");
hw/arm/trace-events +1 -0
index 5ebb3dc9ea..e49a2b324d 100644
@@ -9,6 +9,7 @@ omap1_lpg_led(const char *onoff) "omap1 LPG: LED is %s"
 
 # virt-acpi-build.c
 virt_acpi_setup(void) "No fw cfg or ACPI disabled. Bailing out."
+virt_acpi_dsdt_tegra241_cmdqv(int smmu_id, uint64_t base, uint32_t irq) "DSDT: add cmdqv node for (id=%d), base=0x%" PRIx64 ", irq=%d"
 
 # smmu-common.c
 smmu_add_mr(const char *name) "%s"
#79 lore raw

PATCH v5 31/32 hw/arm/smmuv3-accel: Enforce viommu association when CMDQV is active

In reply to Shameer Kolothum

When CMDQV is active, the first cold-plugged VFIO device establishes the viommu to host SMMUv3 association. Block its hot-unplug to preserve this association and the guest's boot time CMDQV configuration.

Also abort at machine_done if cmdqv=on is requested but no cold-plugged VFIO device was present to initialize it.

---
 hw/arm/smmuv3-accel.h |  1 +
 hw/arm/smmuv3-accel.c | 18 ++++++++++++++++++
 2 files changed, 19 insertions(+)
hw/arm/smmuv3-accel.h +1 -0
index 8cf35c2936..4d05eb100a 100644
@@ -85,6 +85,7 @@ typedef struct SMMUv3AccelDevice {
     IOMMUFDVdev *vdev;
     QLIST_ENTRY(SMMUv3AccelDevice) next;
     SMMUv3AccelState *s_accel;
+    Error *unplug_blocker; /* set when CMDQV is active to block hot-unplug */
 } SMMUv3AccelDevice;
 
 bool smmuv3_accel_init(SMMUv3State *s, Error **errp);
hw/arm/smmuv3-accel.c +18 -0
index dbb50d1795..03575808b3 100644
@@ -761,6 +761,18 @@ static bool smmuv3_accel_set_iommu_device(PCIBus *bus, void *opaque, int devfn,
         return false;
     }
 
+    /*
+     * CMDQV is active: block hot-unplug of the device that established the
+     * viommu association. Removing it would cause the vIOMMU to host SMMUv3
+     * association be changed via device hot-plug.
+     */
+    if (s->s_accel->cmdqv_ops) {
+        PCIDevice *pdev = pci_find_device(bus, pci_bus_num(bus), devfn);
+        error_setg(&accel_dev->unplug_blocker,
+                   "CMDQV is active: removing the device that established the "
+                   "viommu association would break the guest CMDQV");
+        qdev_add_unplug_blocker(DEVICE(pdev), accel_dev->unplug_blocker);
+    }
 done:
     accel_dev->hiodi = hiodi;
     accel_dev->s_accel = s->s_accel;
@@ -1088,6 +1100,12 @@ static void smmuv3_machine_done(Notifier *notifier, void *data)
                      "at least one cold-plugged VFIO device");
         exit(1);
     }
+
+    if (s->cmdqv == ON_OFF_AUTO_ON && !accel->cmdqv) {
+        error_report("arm-smmuv3 cmdqv=on requires at least one cold-plugged "
+                     "VFIO device");
+        exit(1);
+    }
 }
 
 bool smmuv3_accel_init(SMMUv3State *s, Error **errp)
Signature · 1 line
2.43.0
#80 lore raw

Re: [PATCH v5 31/32] hw/arm/smmuv3-accel: Enforce viommu association when CMDQV is active

Quoted 6 lines · Shameer Kolothum

When CMDQV is active, the first cold-plugged VFIO device establishes the viommu to host SMMUv3 association. Block its hot-unplug to preserve this association and the guest's boot time CMDQV configuration.

Also abort at machine_done if cmdqv=on is requested but no cold-plugged VFIO device was present to initialize it.

To me this is still missing the "why", something along what you put in
https://lore.kernel.org/all/IA0PR12MB755515924AD70A273B7879A1AB3D2@IA0PR12MB7555.namprd12.prod.outlook.com/

Thanks

Eric

Quoted 56 lines
---
 hw/arm/smmuv3-accel.h |  1 +
 hw/arm/smmuv3-accel.c | 18 ++++++++++++++++++
 2 files changed, 19 insertions(+)
hw/arm/smmuv3-accel.h +1 -0
index 8cf35c2936..4d05eb100a 100644
@@ -85,6 +85,7 @@ typedef struct SMMUv3AccelDevice {
     IOMMUFDVdev *vdev;
     QLIST_ENTRY(SMMUv3AccelDevice) next;
     SMMUv3AccelState *s_accel;
+    Error *unplug_blocker; /* set when CMDQV is active to block hot-unplug */
 } SMMUv3AccelDevice;
 
 bool smmuv3_accel_init(SMMUv3State *s, Error **errp);
hw/arm/smmuv3-accel.c +18 -0
index dbb50d1795..03575808b3 100644
@@ -761,6 +761,18 @@ static bool smmuv3_accel_set_iommu_device(PCIBus *bus, void *opaque, int devfn,
         return false;
     }
 
+    /*
+     * CMDQV is active: block hot-unplug of the device that established the
+     * viommu association. Removing it would cause the vIOMMU to host SMMUv3
+     * association be changed via device hot-plug.
+     */
+    if (s->s_accel->cmdqv_ops) {
+        PCIDevice *pdev = pci_find_device(bus, pci_bus_num(bus), devfn);
+        error_setg(&accel_dev->unplug_blocker,
+                   "CMDQV is active: removing the device that established the "
+                   "viommu association would break the guest CMDQV");
+        qdev_add_unplug_blocker(DEVICE(pdev), accel_dev->unplug_blocker);
+    }
 done:
     accel_dev->hiodi = hiodi;
     accel_dev->s_accel = s->s_accel;
@@ -1088,6 +1100,12 @@ static void smmuv3_machine_done(Notifier *notifier, void *data)
                      "at least one cold-plugged VFIO device");
         exit(1);
     }
+
+    if (s->cmdqv == ON_OFF_AUTO_ON && !accel->cmdqv) {
+        error_report("arm-smmuv3 cmdqv=on requires at least one cold-plugged "
+                     "VFIO device");
+        exit(1);
+    }
 }
 
 bool smmuv3_accel_init(SMMUv3State *s, Error **errp)
#81 lore raw

RE: [PATCH v5 31/32] hw/arm/smmuv3-accel: Enforce viommu association when CMDQV is active

Quoted 23 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 26 May 2026 13:09
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v5 31/32] hw/arm/smmuv3-accel: Enforce viommu association when CMDQV is active

External email: Use caution opening links or attachments

Quoted 4 lines · Shameer Kolothum

When CMDQV is active, the first cold-plugged VFIO device establishes the viommu to host SMMUv3 association. Block its hot-unplug to preserve this association and the guest's boot time CMDQV configuration.

Quoted 6 lines
Quoted 2 lines

Also abort at machine_done if cmdqv=on is requested but no cold-plugged VFIO device was present to initialize it.

To me this is still missing the "why", something along what you put in
https://lore.kernel.org/all/IA0PR12MB755515924AD70A273B7879A1AB3D
[email protected]/

Ok. I will summarise that and add into this commit msg.

Thanks,
Shameer

#82 lore raw

PATCH v5 32/32 hw/arm/smmuv3: Add cmdqv property for SMMUv3 device

In reply to Shameer Kolothum

Introduce a "cmdqv" property to enable Tegra241 CMDQV support. This is only enabled for accelerated SMMUv3 devices.

---
 hw/arm/smmuv3.c | 8 ++++++++
 qemu-options.hx | 8 ++++++++
 2 files changed, 16 insertions(+)
hw/arm/smmuv3.c +8 -0
index 763b86c417..8c13489a5c 100644
@@ -1994,6 +1994,10 @@ static bool smmu_validate_property(SMMUv3State *s, Error **errp)
                        "bits if accel=on");
             return false;
         }
+        if (s->cmdqv == ON_OFF_AUTO_ON) {
+            error_setg(errp, "cmdqv can only be enabled if accel=on");
+            return false;
+        }
         return true;
     }
 
@@ -2144,6 +2148,7 @@ static const Property smmuv3_properties[] = {
     DEFINE_PROP_OAS_MODE("oas", SMMUv3State, oas, OAS_MODE_AUTO),
     DEFINE_PROP_SSIDSIZE_MODE("ssidsize", SMMUv3State, ssidsize,
                               SSID_SIZE_MODE_AUTO),
+    DEFINE_PROP_ON_OFF_AUTO("cmdqv", SMMUv3State, cmdqv, ON_OFF_AUTO_AUTO),
 };
 
 static void smmuv3_instance_init(Object *obj)
@@ -2188,6 +2193,9 @@ static void smmuv3_class_init(ObjectClass *klass, const void *data)
         "A value of N allows SSIDs in the range [0 .. 2^N - 1]. "
         "A value of 0 disables SubstreamID support. A value greater "
         "than 0 is required to enable PASID support.");
+    object_class_property_set_description(klass, "cmdqv",
+        "Enable/disable CMDQV support (for accel=on). "
+        "Valid values are on, off, and auto. Defaults to auto.");
 }
 
 static int smmuv3_notify_flag_changed(IOMMUMemoryRegion *iommu,
qemu-options.hx +8 -0
index 46b02a1bb3..c8b8f4e69a 100644
@@ -1329,6 +1329,14 @@ SRST
         - With accel=on, auto means the value is automatically derived from the host SMMU.
         - With accel=off, auto is resolved to 0.
 
+    ``cmdqv=on|off|auto`` (default: auto)
+        Enable hardware Command Queue Virtualization (CMDQV) for the
+        SMMUv3 command queue. Currently only the NVIDIA Tegra241 CMDQV
+        implementation is supported.
+
+        - With accel=on, auto means the value is automatically derived from the host SMMU.
+        - With accel=off, auto is resolved to 'off'.
+
 ``-device amd-iommu[,option=...]``
     Enables emulation of an AMD-Vi I/O Memory Management Unit (IOMMU).
     Only available with ``-machine q35``, it supports the following options:
Signature · 1 line
2.43.0
Reviewed-by on v5 32/32
#83 lore raw

Re: [PATCH v5 32/32] hw/arm/smmuv3: Add cmdqv property for SMMUv3 device

Quoted 5 lines · Shameer Kolothum

Introduce a "cmdqv" property to enable Tegra241 CMDQV support. This is only enabled for accelerated SMMUv3 devices.

Thanks

Eric

Quoted 57 lines
---
 hw/arm/smmuv3.c | 8 ++++++++
 qemu-options.hx | 8 ++++++++
 2 files changed, 16 insertions(+)
hw/arm/smmuv3.c +8 -0
index 763b86c417..8c13489a5c 100644
@@ -1994,6 +1994,10 @@ static bool smmu_validate_property(SMMUv3State *s, Error **errp)
                        "bits if accel=on");
             return false;
         }
+        if (s->cmdqv == ON_OFF_AUTO_ON) {
+            error_setg(errp, "cmdqv can only be enabled if accel=on");
+            return false;
+        }
         return true;
     }
 
@@ -2144,6 +2148,7 @@ static const Property smmuv3_properties[] = {
     DEFINE_PROP_OAS_MODE("oas", SMMUv3State, oas, OAS_MODE_AUTO),
     DEFINE_PROP_SSIDSIZE_MODE("ssidsize", SMMUv3State, ssidsize,
                               SSID_SIZE_MODE_AUTO),
+    DEFINE_PROP_ON_OFF_AUTO("cmdqv", SMMUv3State, cmdqv, ON_OFF_AUTO_AUTO),
 };
 
 static void smmuv3_instance_init(Object *obj)
@@ -2188,6 +2193,9 @@ static void smmuv3_class_init(ObjectClass *klass, const void *data)
         "A value of N allows SSIDs in the range [0 .. 2^N - 1]. "
         "A value of 0 disables SubstreamID support. A value greater "
         "than 0 is required to enable PASID support.");
+    object_class_property_set_description(klass, "cmdqv",
+        "Enable/disable CMDQV support (for accel=on). "
+        "Valid values are on, off, and auto. Defaults to auto.");
 }
 
 static int smmuv3_notify_flag_changed(IOMMUMemoryRegion *iommu,
qemu-options.hx +8 -0
index 46b02a1bb3..c8b8f4e69a 100644
@@ -1329,6 +1329,14 @@ SRST
         - With accel=on, auto means the value is automatically derived from the host SMMU.
         - With accel=off, auto is resolved to 0.
 
+    ``cmdqv=on|off|auto`` (default: auto)
+        Enable hardware Command Queue Virtualization (CMDQV) for the
+        SMMUv3 command queue. Currently only the NVIDIA Tegra241 CMDQV
+        implementation is supported.
+
+        - With accel=on, auto means the value is automatically derived from the host SMMU.
+        - With accel=off, auto is resolved to 'off'.
+
 ``-device amd-iommu[,option=...]``
     Enables emulation of an AMD-Vi I/O Memory Management Unit (IOMMU).
     Only available with ``-machine q35``, it supports the following options: