Skip to contentSkip to conversation

hw/arm/virt: Introduce Tegra241 CMDQV support for accelerated SMMUv3

PATCH Superseded Shameer Kolothum 3 participants 55 messages Jun 1 – Jun 4

Subsystems (2)

v6 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

v6 cover hw/arm/virt: Introduce Tegra241 CMDQV support for accelerated SMMUv3

Hi,

Changes from v5:
https://lore.kernel.org/qemu-devel/[email protected]/

- Addressed v5 feedback and picked up Reviewed-by tags. Thanks!
- Added documentation for design and lifecycle details. Please see patch #30.
- Changed probe() and alloc_viommu() CMDQV ops both mandatory (patches #6, #8)
- Fixed build on windows failure (patch #11)
- Improved tracing with "Direct" vs "VI" info (patches #15, #16)
- Removed vcmdq_enabled() check on BASE reprogramming (patch #16)
- Dropped address_space_range_is_ram() check; record error on setup_vcmdq() failure (patch #18)
- Dropped qemu_ram_backend_pagesize_min() and instead uses qemu_minrampagesize()- patch #25
- Based on top of Nathan's "Resolve AUTO properties" v4 series [0].

Please find the complete branch here:
https://github.com/shamiali2008/qemu-master/tree/master-vcmdq-v6-ext

Sanity tested on NVIDIA Grace. Further testing in progress.

Feedback and testing are very welcome.

Thanks,
Shameer
[0] https://lore.kernel.org/qemu-devel/[email protected]/

---
Background(from RFCv1):
https://lore.kernel.org/qemu-devel/[email protected]/

Thanks to Nicolin for the initial patches and testing on which this is based.

Tegra241 CMDQV extends SMMUv3 by allocating per-VM "virtual interfaces" (VINTFs), each hosting up to 128 VCMDQs.

Each VINTF exposes two 64KB MMIO pages:
- Page0 – guest owned control and status registers (directly mapped into the VM)
- Page1 – queue configuration registers (trapped/emulated by QEMU)

Unlike the standard SMMU CMDQ, a guest owned Tegra241 VCMDQ does not support the full command set. Only a subset, primarily invalidation related commands, is accepted by the CMDQV hardware. For this reason, a distinct CMDQV device must be exposed to the guest, and the guest OS must include a Tegra241 CMDQV aware driver to take advantage of the hardware acceleration.

VCMDQ support is integrated via the IOMMU_HW_QUEUE_ALLOC mechanism, allowing QEMU to attach guest configured VCMDQ buffers to the underlying CMDQV hardware through IOMMUFD. The Linux kernel already supports the full CMDQV virtualisation model via IOMMUFD[0].
---

Nicolin Chen (15):
backends/iommufd: Update iommufd_backend_get_device_info backends/iommufd: Update iommufd_backend_alloc_viommu to allow user
ptr
backends/iommufd: Introduce iommufd_backend_alloc_hw_queue backends/iommufd: Introduce iommufd_backend_viommu_mmap hw/arm/tegra241-cmdqv: Implement CMDQV init hw/arm/tegra241-cmdqv: Implement CMDQV vIOMMU alloc/free hw/arm/tegra241-cmdqv: Emulate CMDQ-V Config region hw/arm/tegra241-cmdqv: Emulate VCMDQ register reads hw/arm/tegra241-cmdqv: Emulate VCMDQ register writes hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV hw/arm/tegra241-cmdqv: Allocate HW VCMDQs once configured hw/arm/tegra241-cmdqv: Map VINTF page0 into guest MMIO space hw/arm/tegra241-cmdqv: Add reset handler hw/arm/tegra241-cmdqv: Limit queue size based on backend page size hw/arm/virt-acpi: Advertise Tegra241 CMDQV nodes in DSDT

Shameer Kolothum (16):
system/iommufd: Remove unused viommu pointer from IOMMUFDVeventq hw/arm/smmuv3-accel: Introduce CMDQV ops interface hw/arm/tegra241-cmdqv: Add Tegra241 CMDQV ops backend stub hw/arm/smmuv3-accel: Wire CMDQV ops into accel lifecycle hw/arm/virt: Use stored SMMUv3 device list for IORT build hw/arm/tegra241-cmdqv: Probe host Tegra241 CMDQV support hw/arm/virt: Link SMMUv3 CMDQV resources to platform bus hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backing
memory: Allow RAM device regions to skip IOMMU mapping hw/arm/smmuv3-accel: Introduce common helper for veventq read hw/arm/tegra241-cmdqv: Read and propagate Tegra241 CMDQV errors hw/arm/smmuv3: Add per-device identifier property hw/arm/smmuv3-accel: Introduce helper to query CMDQV type hw/arm/smmuv3-accel: Enforce viommu association when CMDQV is active hw/arm/tegra241-cmdqv: Document the CMDQV design and lifecycle hw/arm/smmuv3: Add cmdqv property for SMMUv3 device

 hw/arm/smmuv3-accel.h         |   48 ++
 hw/arm/tegra241-cmdqv.h       |  384 +++++++++++
 include/hw/arm/smmuv3.h       |    3 +
 include/hw/arm/virt.h         |    1 +
 include/system/iommufd.h      |   17 +-
 include/system/memory.h       |   21 +
 backends/iommufd.c            |   64 ++
 hw/arm/smmuv3-accel-stubs.c   |   12 +
 hw/arm/smmuv3-accel.c         |  194 ++++--
 hw/arm/smmuv3.c               |   10 +
 hw/arm/tegra241-cmdqv-stubs.c |   16 +
 hw/arm/tegra241-cmdqv.c       | 1122 +++++++++++++++++++++++++++++++++
 hw/arm/virt-acpi-build.c      |  127 ++--
 hw/arm/virt.c                 |   37 ++
 hw/vfio/iommufd.c             |    4 +-
 hw/vfio/listener.c            |    6 +
 system/memory.c               |   10 +
 backends/trace-events         |    4 +-
 hw/arm/Kconfig                |    5 +
 hw/arm/meson.build            |    2 +
 hw/arm/trace-events           |   11 +
 hw/vfio/trace-events          |    1 +
 qemu-options.hx               |    8 +
 23 files changed, 2027 insertions(+), 80 deletions(-)
 create mode 100644 hw/arm/tegra241-cmdqv.h
 create mode 100644 hw/arm/tegra241-cmdqv-stubs.c
 create mode 100644 hw/arm/tegra241-cmdqv.c
Signature · 1 line
2.43.0
#1 lore raw

PATCH v6 1/31 backends/iommufd: Update iommufd_backend_get_device_info

From: Nicolin Chen <[email protected]>

The updated IOMMUFD uAPI introduces the ability for userspace to request a specific hardware info data type via IOMMU_GET_HW_INFO. Update iommufd_backend_get_device_info() to set IOMMU_HW_INFO_FLAG_INPUT_TYPE when a non-zero type is supplied, and adjust all callers to pass a type value explicitly initialised to zero (IOMMU_HW_INFO_TYPE_DEFAULT) when no specific type is requested.

---
 backends/iommufd.c    | 7 +++++++
 hw/arm/smmuv3-accel.c | 2 +-
 hw/vfio/iommufd.c     | 4 ++--
 3 files changed, 10 insertions(+), 3 deletions(-)
backends/iommufd.c +7 -0
index 410b044370..8c3a981392 100644
@@ -390,16 +390,23 @@ bool iommufd_backend_get_dirty_bitmap(IOMMUFDBackend *be,
     return true;
 }
 
+/*
+ * @type can carry a desired HW info type defined in the uapi headers. If caller
+ * doesn't have one, indicating it wants the default type, then @type should be
+ * zeroed (i.e. IOMMU_HW_INFO_TYPE_DEFAULT).
+ */
 bool iommufd_backend_get_device_info(IOMMUFDBackend *be, uint32_t devid,
                                      uint32_t *type, void *data, uint32_t len,
                                      uint64_t *caps, uint8_t *max_pasid_log2,
                                      Error **errp)
 {
     struct iommu_hw_info info = {
+        .flags = (*type) ? IOMMU_HW_INFO_FLAG_INPUT_TYPE : 0,
         .size = sizeof(info),
         .dev_id = devid,
         .data_len = len,
         .data_uptr = (uintptr_t)data,
+        .in_data_type = *type,
     };
 
     if (ioctl(be->fd, IOMMU_GET_HW_INFO, &info)) {
hw/arm/smmuv3-accel.c +1 -1
index 06dadf9cc7..163d3e7279 100644
@@ -176,7 +176,7 @@ smmuv3_accel_hw_compatible(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *hiodi,
                            Error **errp)
 {
     struct iommu_hw_info_arm_smmuv3 info;
-    uint32_t data_type;
+    uint32_t data_type = IOMMU_HW_INFO_TYPE_DEFAULT;
     uint64_t caps;
 
     if (!iommufd_backend_get_device_info(hiodi->iommufd, hiodi->devid,
hw/vfio/iommufd.c +2 -2
index df148a49a7..495a59cc2e 100644
@@ -353,7 +353,7 @@ static bool iommufd_cdev_autodomains_get(VFIODevice *vbasedev,
     IOMMUFDBackend *iommufd = vbasedev->iommufd;
     VFIOContainer *bcontainer = VFIO_IOMMU(container);
     bool viommu_nesting, viommu_nesting_dirty;
-    uint32_t type, flags = 0;
+    uint32_t type = IOMMU_HW_INFO_TYPE_DEFAULT, flags = 0;
     uint64_t hw_caps;
     VendorCaps caps;
     VFIOIOASHwpt *hwpt;
@@ -948,7 +948,7 @@ static bool hiod_iommufd_vfio_realize(HostIOMMUDevice *hiod, void *opaque,
     HostIOMMUDeviceIOMMUFD *hiodi;
     HostIOMMUDeviceCaps *caps = &hiod->caps;
     VendorCaps *vendor_caps = &caps->vendor_caps;
-    enum iommu_hw_info_type type;
+    uint32_t type = IOMMU_HW_INFO_TYPE_DEFAULT;
     uint8_t max_pasid_log2;
     uint64_t hw_caps;
 
Signature · 1 line
2.43.0
#2 lore raw

PATCH v6 2/31 backends/iommufd: Update iommufd_backend_alloc_viommu to allow user ptr

In reply to Shameer Kolothum

From: Nicolin Chen <[email protected]>

The updated IOMMUFD VIOMMU_ALLOC uAPI allows userspace to provide a data buffer when creating a vIOMMU (e.g. for Tegra241 CMDQV). Extend iommufd_backend_alloc_viommu() to pass a user pointer and size to the kernel.

Update the caller accordingly.

---
 include/system/iommufd.h | 1 +
 backends/iommufd.c       | 4 ++++
 hw/arm/smmuv3-accel.c    | 4 ++--
 backends/trace-events    | 2 +-
 4 files changed, 8 insertions(+), 3 deletions(-)
include/system/iommufd.h +1 -0
index 2925d116ac..1b0266f660 100644
@@ -89,6 +89,7 @@ bool iommufd_backend_alloc_hwpt(IOMMUFDBackend *be, uint32_t dev_id,
                                 Error **errp);
 bool iommufd_backend_alloc_viommu(IOMMUFDBackend *be, uint32_t dev_id,
                                   uint32_t viommu_type, uint32_t hwpt_id,
+                                  void *data_ptr, uint32_t data_len,
                                   uint32_t *out_hwpt, Error **errp);
 
 bool iommufd_backend_alloc_vdev(IOMMUFDBackend *be, uint32_t dev_id,
backends/iommufd.c +4 -0
index 8c3a981392..c745dbd2b1 100644
@@ -463,6 +463,7 @@ bool iommufd_backend_invalidate_cache(IOMMUFDBackend *be, uint32_t id,
 
 bool iommufd_backend_alloc_viommu(IOMMUFDBackend *be, uint32_t dev_id,
                                   uint32_t viommu_type, uint32_t hwpt_id,
+                                  void *data_ptr, uint32_t data_len,
                                   uint32_t *out_viommu_id, Error **errp)
 {
     int ret;
@@ -471,11 +472,14 @@ bool iommufd_backend_alloc_viommu(IOMMUFDBackend *be, uint32_t dev_id,
         .type = viommu_type,
         .dev_id = dev_id,
         .hwpt_id = hwpt_id,
+        .data_len = data_len,
+        .data_uptr = (uintptr_t)data_ptr,
     };
 
     ret = ioctl(be->fd, IOMMU_VIOMMU_ALLOC, &alloc_viommu);
 
     trace_iommufd_backend_alloc_viommu(be->fd, dev_id, viommu_type, hwpt_id,
+                                       (uintptr_t)data_ptr, data_len,
                                        alloc_viommu.out_viommu_id, ret);
     if (ret) {
         error_setg_errno(errp, errno, "IOMMU_VIOMMU_ALLOC failed");
hw/arm/smmuv3-accel.c +2 -2
index 163d3e7279..4cef487679 100644
@@ -582,8 +582,8 @@ smmuv3_accel_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *hiodi,
     IOMMUFDViommu *viommu;
 
     if (!iommufd_backend_alloc_viommu(hiodi->iommufd, hiodi->devid,
-                                      IOMMU_VIOMMU_TYPE_ARM_SMMUV3,
-                                      s2_hwpt_id, &viommu_id, errp)) {
+                                      IOMMU_VIOMMU_TYPE_ARM_SMMUV3, s2_hwpt_id,
+                                      NULL, 0, &viommu_id, errp)) {
         return false;
     }
 
backends/trace-events +1 -1
index b9365113e7..3ba0c3503c 100644
@@ -21,7 +21,7 @@ iommufd_backend_free_id(int iommufd, uint32_t id, int ret) " iommufd=%d id=%d (%
 iommufd_backend_set_dirty(int iommufd, uint32_t hwpt_id, bool start, int ret) " iommufd=%d hwpt=%u enable=%d (%d)"
 iommufd_backend_get_dirty_bitmap(int iommufd, uint32_t hwpt_id, uint64_t iova, uint64_t size, uint64_t flags, uint64_t page_size, int ret) " iommufd=%d hwpt=%u iova=0x%"PRIx64" size=0x%"PRIx64" flags=0x%"PRIx64" page_size=0x%"PRIx64" (%d)"
 iommufd_backend_invalidate_cache(int iommufd, uint32_t id, uint32_t data_type, uint32_t entry_len, uint32_t entry_num, uint32_t done_num, uint64_t data_ptr, int ret) " iommufd=%d id=%u data_type=%u entry_len=%u entry_num=%u done_num=%u data_ptr=0x%"PRIx64" (%d)"
-iommufd_backend_alloc_viommu(int iommufd, uint32_t dev_id, uint32_t type, uint32_t hwpt_id, uint32_t viommu_id, int ret) " iommufd=%d type=%u dev_id=%u hwpt_id=%u viommu_id=%u (%d)"
+iommufd_backend_alloc_viommu(int iommufd, uint32_t dev_id, uint32_t type, uint32_t hwpt_id, uint64_t data_ptr, uint32_t data_len, uint32_t viommu_id, int ret) " iommufd=%d type=%u dev_id=%u hwpt_id=%u data_ptr=0x%"PRIx64" data_len=0x%x viommu_id=%u (%d)"
 iommufd_backend_alloc_vdev(int iommufd, uint32_t dev_id, uint32_t viommu_id, uint64_t virt_id, uint32_t vdev_id, int ret) " iommufd=%d dev_id=%u viommu_id=%u virt_id=0x%"PRIx64" vdev_id=%u (%d)"
 iommufd_viommu_alloc_eventq(int iommufd, uint32_t viommu_id, uint32_t type, uint32_t veventq_id, uint32_t veventq_fd, int ret) " iommufd=%d viommu_id=%u type=%u veventq_id=%u veventq_fd=%u (%d)"
 
Signature · 1 line
2.43.0
#3 lore raw

PATCH v6 3/31 backends/iommufd: Introduce iommufd_backend_alloc_hw_queue

In reply to Shameer Kolothum

From: Nicolin Chen <[email protected]>

Add a helper to allocate an iommufd backed HW queue for a vIOMMU.

While at it, define a struct IOMMUFDHWqueue for use by vendor implementations.

---
 include/system/iommufd.h | 11 +++++++++++
 backends/iommufd.c       | 31 +++++++++++++++++++++++++++++++
 backends/trace-events    |  1 +
 3 files changed, 43 insertions(+)
include/system/iommufd.h +11 -0
index 1b0266f660..c6f2e87a7e 100644
@@ -65,6 +65,12 @@ typedef struct IOMMUFDVeventq {
     bool event_start; /* True after first valid event; cleared on overflow */
 } IOMMUFDVeventq;
 
+/* HW queue object for a vIOMMU-specific HW-accelerated queue */
+typedef struct IOMMUFDHWqueue {
+    IOMMUFDViommu *viommu;
+    uint32_t hw_queue_id;
+} IOMMUFDHWqueue;
+
 bool iommufd_backend_connect(IOMMUFDBackend *be, Error **errp);
 void iommufd_backend_disconnect(IOMMUFDBackend *be);
 
@@ -101,6 +107,11 @@ bool iommufd_backend_alloc_veventq(IOMMUFDBackend *be, uint32_t viommu_id,
                                    uint32_t *out_veventq_id,
                                    uint32_t *out_veventq_fd, Error **errp);
 
+bool iommufd_backend_alloc_hw_queue(IOMMUFDBackend *be, uint32_t viommu_id,
+                                    uint32_t queue_type, uint32_t index,
+                                    uint64_t addr, uint64_t length,
+                                    uint32_t *out_hw_queue_id, Error **errp);
+
 bool iommufd_backend_set_dirty_tracking(IOMMUFDBackend *be, uint32_t hwpt_id,
                                         bool start, Error **errp);
 bool iommufd_backend_get_dirty_bitmap(IOMMUFDBackend *be, uint32_t hwpt_id,
backends/iommufd.c +31 -0
index c745dbd2b1..8eaaf456e8 100644
@@ -549,6 +549,37 @@ bool iommufd_backend_alloc_veventq(IOMMUFDBackend *be, uint32_t viommu_id,
     return true;
 }
 
+bool iommufd_backend_alloc_hw_queue(IOMMUFDBackend *be, uint32_t viommu_id,
+                                    uint32_t queue_type, uint32_t index,
+                                    uint64_t addr, uint64_t length,
+                                    uint32_t *out_hw_queue_id, Error **errp)
+{
+    int ret;
+    struct iommu_hw_queue_alloc alloc_hw_queue = {
+        .size = sizeof(alloc_hw_queue),
+        .flags = 0,
+        .viommu_id = viommu_id,
+        .type = queue_type,
+        .index = index,
+        .nesting_parent_iova = addr,
+        .length = length,
+    };
+
+    ret = ioctl(be->fd, IOMMU_HW_QUEUE_ALLOC, &alloc_hw_queue);
+
+    trace_iommufd_backend_alloc_hw_queue(be->fd, viommu_id, queue_type,
+                                         index, addr, length,
+                                         alloc_hw_queue.out_hw_queue_id, ret);
+    if (ret) {
+        error_setg_errno(errp, errno, "IOMMU_HW_QUEUE_ALLOC failed");
+        return false;
+    }
+
+    g_assert(out_hw_queue_id);
+    *out_hw_queue_id = alloc_hw_queue.out_hw_queue_id;
+    return true;
+}
+
 bool host_iommu_device_iommufd_attach_hwpt(HostIOMMUDeviceIOMMUFD *hiodi,
                                            uint32_t hwpt_id, Error **errp)
 {
backends/trace-events +1 -0
index 3ba0c3503c..c5c1d95aad 100644
@@ -24,6 +24,7 @@ iommufd_backend_invalidate_cache(int iommufd, uint32_t id, uint32_t data_type, u
 iommufd_backend_alloc_viommu(int iommufd, uint32_t dev_id, uint32_t type, uint32_t hwpt_id, uint64_t data_ptr, uint32_t data_len, uint32_t viommu_id, int ret) " iommufd=%d type=%u dev_id=%u hwpt_id=%u data_ptr=0x%"PRIx64" data_len=0x%x viommu_id=%u (%d)"
 iommufd_backend_alloc_vdev(int iommufd, uint32_t dev_id, uint32_t viommu_id, uint64_t virt_id, uint32_t vdev_id, int ret) " iommufd=%d dev_id=%u viommu_id=%u virt_id=0x%"PRIx64" vdev_id=%u (%d)"
 iommufd_viommu_alloc_eventq(int iommufd, uint32_t viommu_id, uint32_t type, uint32_t veventq_id, uint32_t veventq_fd, int ret) " iommufd=%d viommu_id=%u type=%u veventq_id=%u veventq_fd=%u (%d)"
+iommufd_backend_alloc_hw_queue(int iommufd, uint32_t viommu_id, uint32_t queue_type, uint32_t index, uint64_t addr, uint64_t size, uint32_t queue_id, int ret) " iommufd=%d viommu_id=%u queue_type=%u index=%u addr=0x%"PRIx64" size=0x%"PRIx64" queue_id=%u (%d)"
 
 # igvm-cfg.c
 igvm_reset_enter(int type) "type=%u"
Signature · 1 line
2.43.0
#4 lore raw

PATCH v6 4/31 backends/iommufd: Introduce iommufd_backend_viommu_mmap

In reply to Shameer Kolothum

From: Nicolin Chen <[email protected]>

Add a backend helper to mmap hardware MMIO regions exposed via iommufd for a vIOMMU instance. This allows user space to access HW-accelerated MMIO pages provided by the vIOMMU.

The caller is responsible for unmapping the returned region.

---
 include/system/iommufd.h |  4 ++++
 backends/iommufd.c       | 22 ++++++++++++++++++++++
 backends/trace-events    |  1 +
 3 files changed, 27 insertions(+)
include/system/iommufd.h +4 -0
index c6f2e87a7e..bb8d5081d7 100644
@@ -112,6 +112,10 @@ bool iommufd_backend_alloc_hw_queue(IOMMUFDBackend *be, uint32_t viommu_id,
                                     uint64_t addr, uint64_t length,
                                     uint32_t *out_hw_queue_id, Error **errp);
 
+bool iommufd_backend_viommu_mmap(IOMMUFDBackend *be, uint32_t viommu_id,
+                                 uint64_t size, off_t offset, void **out_ptr,
+                                 Error **errp);
+
 bool iommufd_backend_set_dirty_tracking(IOMMUFDBackend *be, uint32_t hwpt_id,
                                         bool start, Error **errp);
 bool iommufd_backend_get_dirty_bitmap(IOMMUFDBackend *be, uint32_t hwpt_id,
backends/iommufd.c +22 -0
index 8eaaf456e8..440c1b82bc 100644
@@ -580,6 +580,28 @@ bool iommufd_backend_alloc_hw_queue(IOMMUFDBackend *be, uint32_t viommu_id,
     return true;
 }
 
+/*
+ * Helper to mmap HW MMIO regions exposed via iommufd for a vIOMMU instance.
+ * The caller is responsible for unmapping the mapped region.
+ */
+bool iommufd_backend_viommu_mmap(IOMMUFDBackend *be, uint32_t viommu_id,
+                                 uint64_t size, off_t offset, void **out_ptr,
+                                 Error **errp)
+{
+    g_assert(viommu_id);
+    g_assert(out_ptr);
+
+    *out_ptr = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, be->fd,
+                   offset);
+    trace_iommufd_backend_viommu_mmap(be->fd, viommu_id, size, offset);
+    if (*out_ptr == MAP_FAILED) {
+        error_setg_errno(errp, errno, "IOMMUFD vIOMMU mmap failed");
+        return false;
+    }
+
+    return true;
+}
+
 bool host_iommu_device_iommufd_attach_hwpt(HostIOMMUDeviceIOMMUFD *hiodi,
                                            uint32_t hwpt_id, Error **errp)
 {
backends/trace-events +1 -0
index c5c1d95aad..b63420b73e 100644
@@ -25,6 +25,7 @@ iommufd_backend_alloc_viommu(int iommufd, uint32_t dev_id, uint32_t type, uint32
 iommufd_backend_alloc_vdev(int iommufd, uint32_t dev_id, uint32_t viommu_id, uint64_t virt_id, uint32_t vdev_id, int ret) " iommufd=%d dev_id=%u viommu_id=%u virt_id=0x%"PRIx64" vdev_id=%u (%d)"
 iommufd_viommu_alloc_eventq(int iommufd, uint32_t viommu_id, uint32_t type, uint32_t veventq_id, uint32_t veventq_fd, int ret) " iommufd=%d viommu_id=%u type=%u veventq_id=%u veventq_fd=%u (%d)"
 iommufd_backend_alloc_hw_queue(int iommufd, uint32_t viommu_id, uint32_t queue_type, uint32_t index, uint64_t addr, uint64_t size, uint32_t queue_id, int ret) " iommufd=%d viommu_id=%u queue_type=%u index=%u addr=0x%"PRIx64" size=0x%"PRIx64" queue_id=%u (%d)"
+iommufd_backend_viommu_mmap(int iommufd, uint32_t viommu_id, uint64_t size, uint64_t offset) " iommufd=%d viommu_id=%u size=0x%"PRIx64" offset=0x%"PRIx64
 
 # igvm-cfg.c
 igvm_reset_enter(int type) "type=%u"
Signature · 1 line
2.43.0
#5 lore raw

PATCH v6 5/31 system/iommufd: Remove unused viommu pointer from IOMMUFDVeventq

In reply to Shameer Kolothum

The viommu field is assigned but never used. Callers freeing the veventq already have access to the IOMMUFDViommu object through other references, so this field is redundant.

Removing it also simplifies upcoming changes where veventq is allocated based on the viommu id before the IOMMUFDViommu object is created (e.g. vendor CMDQV-based veventq allocation).

No functional change.

---
 include/system/iommufd.h | 1 -
 hw/arm/smmuv3-accel.c    | 1 -
 2 files changed, 2 deletions(-)
include/system/iommufd.h +0 -1
index bb8d5081d7..da68ba0037 100644
@@ -58,7 +58,6 @@ typedef struct IOMMUFDVdev {
 
 /* Virtual event queue interface for a vIOMMU */
 typedef struct IOMMUFDVeventq {
-    IOMMUFDViommu *viommu;
     uint32_t veventq_id;
     uint32_t veventq_fd;
     uint32_t last_event_seq; /* Sequence number of last processed event */
hw/arm/smmuv3-accel.c +0 -1
index 4cef487679..70b2581c17 100644
@@ -553,7 +553,6 @@ bool smmuv3_accel_alloc_veventq(SMMUv3State *s, Error **errp)
     veventq = g_new0(IOMMUFDVeventq, 1);
     veventq->veventq_id = veventq_id;
     veventq->veventq_fd = veventq_fd;
-    veventq->viommu = accel->viommu;
     accel->veventq = veventq;
 
     /* Set up event handler for veventq fd */
Signature · 1 line
2.43.0
#6 lore raw

PATCH v6 6/31 hw/arm/smmuv3-accel: Introduce CMDQV ops interface

In reply to Shameer Kolothum

Command Queue Virtualization (CMDQV) is a hardware extension available on certain platforms that allows the SMMUv3 command queue to be virtualized and passed through to a VM, improving performance.

For example, NVIDIA Tegra241 implements CMDQV to support virtualization of multiple command queues (VCMDQs).

The term CMDQV is used here generically to refer to any platform that provides hardware support to virtualize the SMMUv3 command queue.

CMDQV support is a specialization of the IOMMUFD-backed accelerated SMMUv3 path. Introduce an ops interface to factor out CMDQV-specific probe, initialization, and vIOMMU allocation logic from the base implementation. The ops pointer and associated state are stored in the accelerated SMMUv3 state.

This provides an extensible design to support future vendor-specific CMDQV implementations.

No functional change.

---
 hw/arm/smmuv3-accel.h | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)
hw/arm/smmuv3-accel.h +34 -0
index 87fecb5c68..b45f25ad03 100644
@@ -10,11 +10,44 @@
 #define HW_ARM_SMMUV3_ACCEL_H
 
 #include "hw/arm/smmu-common.h"
+#include "hw/arm/smmuv3.h"
 #include "system/iommufd.h"
 #ifdef CONFIG_LINUX
 #include <linux/iommufd.h>
 #endif
 
+/*
+ * CMDQ-Virtualization (CMDQV) hardware support, extends the SMMUv3 to
+ * support multiple VCMDQs with virtualization capabilities.
+ * CMDQV specific behavior is factored behind this ops interface.
+ */
+typedef struct SMMUv3AccelCmdqvOps {
+    /**
+     * @probe: Mandatory. Vendor-specific device probing.
+     */
+    bool (*probe)(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev, Error **errp);
+    /**
+     * @init: Optional callback. Initialize CMDQV hardware.
+     */
+    bool (*init)(SMMUv3State *s, Error **errp);
+    /**
+     * @alloc_viommu: Mandatory. Allocate CMDQV viommu resources.
+     */
+    bool (*alloc_viommu)(SMMUv3State *s,
+                         HostIOMMUDeviceIOMMUFD *idev,
+                         uint32_t *out_viommu_id,
+                         Error **errp);
+    /**
+     * @free_viommu: Optional callback. Free CMDQV viommu resources.
+     * If NULL, the viommu_id is freed directly via iommufd_backend_free_id().
+     */
+    void (*free_viommu)(SMMUv3State *s);
+    /**
+     * @reset: Optional callback. Reset CMDQV state.
+     */
+    void (*reset)(SMMUv3State *s);
+} SMMUv3AccelCmdqvOps;
+
 /*
  * Represents an accelerated SMMU instance backed by an iommufd vIOMMU object.
  * Holds bypass and abort proxy HWPT IDs used for device attachment.
@@ -27,6 +60,7 @@ typedef struct SMMUv3AccelState {
     QLIST_HEAD(, SMMUv3AccelDevice) device_list;
     bool auto_mode;
     bool auto_finalised;
+    const SMMUv3AccelCmdqvOps *cmdqv_ops;
 } SMMUv3AccelState;
 
 typedef struct SMMUS1Hwpt {
Signature · 1 line
2.43.0
Reviewed-by on v6 6/31
#7 lore raw

Re: [PATCH v6 06/31] hw/arm/smmuv3-accel: Introduce CMDQV ops interface

Hi Shameer,

Quoted 16 lines · Shameer Kolothum

Command Queue Virtualization (CMDQV) is a hardware extension available on certain platforms that allows the SMMUv3 command queue to be virtualized and passed through to a VM, improving performance.

For example, NVIDIA Tegra241 implements CMDQV to support virtualization of multiple command queues (VCMDQs).

The term CMDQV is used here generically to refer to any platform that provides hardware support to virtualize the SMMUv3 command queue.

CMDQV support is a specialization of the IOMMUFD-backed accelerated SMMUv3 path. Introduce an ops interface to factor out CMDQV-specific probe, initialization, and vIOMMU allocation logic from the base implementation. The ops pointer and associated state are stored in the accelerated SMMUv3 state.

Quoted 7 lines

This provides an extensible design to support future vendor-specific CMDQV implementations.

No functional change.

Thanks

Eric

Quoted 61 lines
---
 hw/arm/smmuv3-accel.h | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)
hw/arm/smmuv3-accel.h +34 -0
index 87fecb5c68..b45f25ad03 100644
@@ -10,11 +10,44 @@
 #define HW_ARM_SMMUV3_ACCEL_H
 
 #include "hw/arm/smmu-common.h"
+#include "hw/arm/smmuv3.h"
 #include "system/iommufd.h"
 #ifdef CONFIG_LINUX
 #include <linux/iommufd.h>
 #endif
 
+/*
+ * CMDQ-Virtualization (CMDQV) hardware support, extends the SMMUv3 to
+ * support multiple VCMDQs with virtualization capabilities.
+ * CMDQV specific behavior is factored behind this ops interface.
+ */
+typedef struct SMMUv3AccelCmdqvOps {
+    /**
+     * @probe: Mandatory. Vendor-specific device probing.
+     */
+    bool (*probe)(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev, Error **errp);
+    /**
+     * @init: Optional callback. Initialize CMDQV hardware.
+     */
+    bool (*init)(SMMUv3State *s, Error **errp);
+    /**
+     * @alloc_viommu: Mandatory. Allocate CMDQV viommu resources.
+     */
+    bool (*alloc_viommu)(SMMUv3State *s,
+                         HostIOMMUDeviceIOMMUFD *idev,
+                         uint32_t *out_viommu_id,
+                         Error **errp);
+    /**
+     * @free_viommu: Optional callback. Free CMDQV viommu resources.
+     * If NULL, the viommu_id is freed directly via iommufd_backend_free_id().
+     */
+    void (*free_viommu)(SMMUv3State *s);
+    /**
+     * @reset: Optional callback. Reset CMDQV state.
+     */
+    void (*reset)(SMMUv3State *s);
+} SMMUv3AccelCmdqvOps;
+
 /*
  * Represents an accelerated SMMU instance backed by an iommufd vIOMMU object.
  * Holds bypass and abort proxy HWPT IDs used for device attachment.
@@ -27,6 +60,7 @@ typedef struct SMMUv3AccelState {
     QLIST_HEAD(, SMMUv3AccelDevice) device_list;
     bool auto_mode;
     bool auto_finalised;
+    const SMMUv3AccelCmdqvOps *cmdqv_ops;
 } SMMUv3AccelState;
 
 typedef struct SMMUS1Hwpt {
#8 lore raw

PATCH v6 7/31 hw/arm/tegra241-cmdqv: Add Tegra241 CMDQV ops backend stub

In reply to Shameer Kolothum

Introduce a Tegra241 CMDQV backend that plugs into the SMMUv3 accelerated CMDQV ops interface.

This patch wires up the Tegra241 CMDQV backend and provides a stub implementation for CMDQV probe, initialization, vIOMMU allocation and reset handling.

Functional CMDQV support is added in follow-up patches.

---
 hw/arm/tegra241-cmdqv.h       | 15 ++++++++++
 hw/arm/tegra241-cmdqv-stubs.c | 16 ++++++++++
 hw/arm/tegra241-cmdqv.c       | 56 +++++++++++++++++++++++++++++++++++
 hw/arm/Kconfig                |  5 ++++
 hw/arm/meson.build            |  2 ++
 5 files changed, 94 insertions(+)
 create mode 100644 hw/arm/tegra241-cmdqv.h
 create mode 100644 hw/arm/tegra241-cmdqv-stubs.c
 create mode 100644 hw/arm/tegra241-cmdqv.c
hw/arm/tegra241-cmdqv.h new file +15 -0
new file mode 100644
index 0000000000..74a6954017
@@ -0,0 +1,15 @@
+/*
+ * Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved
+ * NVIDIA Tegra241 CMDQ-Virtualization extension for SMMUv3
+ *
+ * Written by Nicolin Chen, Shameer Kolothum
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#ifndef HW_ARM_TEGRA241_CMDQV_H
+#define HW_ARM_TEGRA241_CMDQV_H
+
+const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops(void);
+
+#endif /* HW_ARM_TEGRA241_CMDQV_H */
hw/arm/tegra241-cmdqv-stubs.c new file +16 -0
new file mode 100644
index 0000000000..4669f5c5f5
@@ -0,0 +1,16 @@
+/*
+ * Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved
+ *
+ * Stubs for Tegra241 CMDQ-Virtualization extension for SMMUv3
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#include "qemu/osdep.h"
+#include "smmuv3-accel.h"
+#include "hw/arm/tegra241-cmdqv.h"
+
+const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops(void)
+{
+    return NULL;
+}
hw/arm/tegra241-cmdqv.c new file +56 -0

56 lines not on this page · raw message

hw/arm/Kconfig +5 -0
index 5b198402d5..f73ad9727a 100644
@@ -642,6 +642,10 @@ config FSL_IMX8MM_EVK
     depends on TCG
     select FSL_IMX8MM
 
+config TEGRA241_CMDQV
+    bool
+    depends on ARM_SMMUV3_ACCEL
+
 config ARM_SMMUV3_ACCEL
     bool
     depends on ARM_SMMUV3
@@ -649,6 +653,7 @@ config ARM_SMMUV3_ACCEL
 config ARM_SMMUV3
     bool
     select ARM_SMMUV3_ACCEL if IOMMUFD
+    imply TEGRA241_CMDQV
 
 config FSL_IMX6UL
     bool
hw/arm/meson.build +2 -0
index 8f66a80e10..4233a800be 100644
@@ -89,6 +89,8 @@ arm_common_ss.add(when: 'CONFIG_FSL_IMX8MM_EVK', if_true: files('imx8mm-evk.c'))
 arm_common_ss.add(when: 'CONFIG_ARM_SMMUV3', if_true: files('smmuv3.c'))
 arm_common_ss.add(when: 'CONFIG_ARM_SMMUV3_ACCEL', if_true: files('smmuv3-accel.c'))
 stub_ss.add(files('smmuv3-accel-stubs.c'))
+arm_common_ss.add(when: 'CONFIG_TEGRA241_CMDQV', if_true: files('tegra241-cmdqv.c'))
+stub_ss.add(files('tegra241-cmdqv-stubs.c'))
 arm_common_ss.add(when: 'CONFIG_FSL_IMX6UL', if_true: files('fsl-imx6ul.c', 'mcimx6ul-evk.c'))
 arm_common_ss.add(when: 'CONFIG_NRF51_SOC', if_true: files('nrf51_soc.c'))
 arm_common_ss.add(when: 'CONFIG_XEN', if_true: files(
Signature · 1 line
2.43.0
#9 lore raw

PATCH v6 8/31 hw/arm/smmuv3-accel: Wire CMDQV ops into accel lifecycle

In reply to Shameer Kolothum

Add support for selecting and initializing a CMDQV backend based on the cmdqv OnOffAuto property.

If set to OFF, CMDQV is not used and the default IOMMUFD-backed allocation path is taken.

If set to AUTO, QEMU attempts to probe a CMDQV backend during device setup. If probing succeeds, the selected ops are stored in the accelerated SMMUv3 state and used. If probing fails, QEMU silently falls back to the default path.

If set to ON, QEMU requires CMDQV support. Probing is performed during setup and failure results in an error.

When a CMDQV backend is active, its callbacks are used for vIOMMU allocation, free, and reset handling. Otherwise, the base implementation is used.

The current implementation wires up the Tegra241 CMDQV backend through the generic ops interface. Functional CMDQV behaviour is added in subsequent patches.

No functional change.

---
 include/hw/arm/smmuv3.h |  2 +
 hw/arm/smmuv3-accel.c   | 94 ++++++++++++++++++++++++++++++++++++++---
 2 files changed, 89 insertions(+), 7 deletions(-)
include/hw/arm/smmuv3.h +2 -0
index 85be3d7467..34d0f65eaa 100644
@@ -75,6 +75,8 @@ struct SMMUv3State {
     OnOffAuto ats;
     OasMode oas;
     SsidSizeMode ssidsize;
+    /* SMMU CMDQV extension */
+    OnOffAuto cmdqv;
 
     Notifier machine_done;
 };
hw/arm/smmuv3-accel.c +87 -7

135 lines not on this page · raw message

Signature · 1 line
2.43.0
#10 lore raw

PATCH v6 9/31 hw/arm/virt: Use stored SMMUv3 device list for IORT build

In reply to Shameer Kolothum

Introduce a GPtrArray in VirtMachineState to track all SMMUv3 devices created on the virt machine, and use it when building the IORT table instead of relying on object_child_foreach_recursive() walks of the object tree.

This avoids recursive object traversal and provides a foundation for subsequent patches that need direct access to SMMUv3 instances for CMDQV-related handling.

No functional change. No bios-tables qtest failures observed.

---
 include/hw/arm/virt.h    |  1 +
 hw/arm/virt-acpi-build.c | 70 ++++++++++++++++++----------------------
 hw/arm/virt.c            |  3 ++
 3 files changed, 35 insertions(+), 39 deletions(-)
include/hw/arm/virt.h +1 -0
index 3ba33b4bd2..171d44c644 100644
@@ -207,6 +207,7 @@ struct VirtMachineState {
     MemoryRegion *sysmem;
     MemoryRegion *secure_sysmem;
     bool pci_preserve_config;
+    GPtrArray *smmuv3_devices;
 };
 
 #define VIRT_ECAM_ID(high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
hw/arm/virt-acpi-build.c +31 -39

87 lines not on this page · raw message

hw/arm/virt.c +3 -0
index b090233893..ac0606fe87 100644
@@ -3865,6 +3865,7 @@ static void virt_machine_device_plug_cb(HotplugHandler *hotplug_dev,
             }
 
             create_smmuv3_dev_dtb(vms, dev, bus, errp);
+            g_ptr_array_add(vms->smmuv3_devices, dev);
         }
     }
 
@@ -4319,6 +4320,8 @@ static void virt_instance_init(Object *obj)
     vms->oem_id = g_strndup(ACPI_BUILD_APPNAME6, 6);
     vms->oem_table_id = g_strndup(ACPI_BUILD_APPNAME8, 8);
     cxl_machine_init(obj, &vms->cxl_devices_state);
+
+    vms->smmuv3_devices = g_ptr_array_new_with_free_func(NULL);
 }
 
 static void virt_instance_finalize(Object *obj)
Signature · 1 line
2.43.0
#11 lore raw

PATCH v6 10/31 hw/arm/tegra241-cmdqv: Probe host Tegra241 CMDQV support

In reply to Shameer Kolothum

Use IOMMU_GET_HW_INFO to query host support for Tegra241 CMDQV.

Validate the returned data type, version, and minimum number of vCMDQs and SIDs per Tegra241 CMDQ Virtual Interface(VI). Fail the probe if the host does not meet these requirements.

The QEMU model supports one Virtual Interface(VI) per VM with 2 vCMDQs and 16 SIDs per VI, so the probe ensures the host implementation is compatible with these limits.

---
 hw/arm/tegra241-cmdqv.h |  4 ++++
 hw/arm/tegra241-cmdqv.c | 32 ++++++++++++++++++++++++++++++--
 2 files changed, 34 insertions(+), 2 deletions(-)
hw/arm/tegra241-cmdqv.h +4 -0
index 74a6954017..38c8b27b4d 100644
@@ -10,6 +10,10 @@
 #ifndef HW_ARM_TEGRA241_CMDQV_H
 #define HW_ARM_TEGRA241_CMDQV_H
 
+#define CMDQV_VER                 1
+#define CMDQV_NUM_CMDQ_LOG2       1
+#define CMDQV_NUM_SID_PER_VI_LOG2 4
+
 const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops(void);
 
 #endif /* HW_ARM_TEGRA241_CMDQV_H */
hw/arm/tegra241-cmdqv.c +30 -2
index ad5a0d4611..3a19a1af56 100644
@@ -38,8 +38,36 @@ static bool tegra241_cmdqv_init(SMMUv3State *s, Error **errp)
 static bool tegra241_cmdqv_probe(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
                                  Error **errp)
 {
-    error_setg(errp, "NVIDIA Tegra241 CMDQV is unsupported");
-    return false;
+    uint32_t data_type = IOMMU_HW_INFO_TYPE_TEGRA241_CMDQV;
+    struct iommu_hw_info_tegra241_cmdqv cmdqv_info;
+    uint64_t caps;
+
+    if (!iommufd_backend_get_device_info(idev->iommufd, idev->devid, &data_type,
+                                         &cmdqv_info, sizeof(cmdqv_info), &caps,
+                                         NULL, errp)) {
+        return false;
+    }
+    if (data_type != IOMMU_HW_INFO_TYPE_TEGRA241_CMDQV) {
+        error_setg(errp, "Host CMDQV: unexpected data type %u (expected %u)",
+                   data_type, IOMMU_HW_INFO_TYPE_TEGRA241_CMDQV);
+        return false;
+    }
+    if (cmdqv_info.version != CMDQV_VER) {
+        error_setg(errp, "Host CMDQV: unsupported version %u (expected %u)",
+                   cmdqv_info.version, CMDQV_VER);
+        return false;
+    }
+    if (cmdqv_info.log2vcmdqs < CMDQV_NUM_CMDQ_LOG2) {
+        error_setg(errp, "Host CMDQV: insufficient vCMDQs log2=%u (need >= %u)",
+                   cmdqv_info.log2vcmdqs, CMDQV_NUM_CMDQ_LOG2);
+        return false;
+    }
+    if (cmdqv_info.log2vsids < CMDQV_NUM_SID_PER_VI_LOG2) {
+        error_setg(errp, "Host CMDQV: insufficient SIDs log2=%u (need >= %u)",
+                   cmdqv_info.log2vsids, CMDQV_NUM_SID_PER_VI_LOG2);
+        return false;
+    }
+    return true;
 }
 
 static const SMMUv3AccelCmdqvOps tegra241_cmdqv_ops = {
Signature · 1 line
2.43.0
#12 lore raw

PATCH v6 11/31 hw/arm/tegra241-cmdqv: Implement CMDQV init

In reply to Shameer Kolothum

From: Nicolin Chen <[email protected]>

Tegra241 CMDQV extends SMMUv3 with support for virtual command queues (VCMDQs) exposed via a CMDQV MMIO region. The CMDQV MMIO space is split into 64KB pages:

0x00000  (CMDQ-V Config page)
0x10000  (CMDQ-V CMDQ Page0)
0x20000  (CMDQ-V CMDQ Page1)
0x30000  (Virtual Interface Page0)
0x40000  (Virtual Interface Page1)

This patch wires up the Tegra241 CMDQV init callback and allocates vendor-specific CMDQV state. The state pointer is stored in SMMUv3AccelState for use by subsequent CMDQV operations.

The CMDQV MMIO region and a dedicated IRQ line are registered with the SMMUv3 device. The MMIO read/write handlers are currently stubs and will be implemented in later patches.

The CMDQV interrupt is edge-triggered and indicates VCMDQ or VINTF error conditions. This patch only registers the IRQ line. Interrupt generation and propagation to the guest will be added in a subsequent patch.

---
 hw/arm/smmuv3-accel.h   |  1 +
 hw/arm/tegra241-cmdqv.h | 20 ++++++++++++++++++++
 hw/arm/tegra241-cmdqv.c | 40 ++++++++++++++++++++++++++++++++++++++--
 3 files changed, 59 insertions(+), 2 deletions(-)
hw/arm/smmuv3-accel.h +1 -0
index b45f25ad03..e0bbec8581 100644
@@ -61,6 +61,7 @@ typedef struct SMMUv3AccelState {
     bool auto_mode;
     bool auto_finalised;
     const SMMUv3AccelCmdqvOps *cmdqv_ops;
+    void *cmdqv;  /* vendor specific CMDQV state */
 } SMMUv3AccelState;
 
 typedef struct SMMUS1Hwpt {
hw/arm/tegra241-cmdqv.h +20 -0
index 38c8b27b4d..a5f65a8991 100644
@@ -14,6 +14,26 @@
 #define CMDQV_NUM_CMDQ_LOG2       1
 #define CMDQV_NUM_SID_PER_VI_LOG2 4
 
+/*
+ * Tegra241 CMDQV MMIO layout (64KB pages)
+ *
+ * 0x00000  (CMDQ-V Config page)
+ * 0x10000  (CMDQ-V CMDQ Page0)
+ * 0x20000  (CMDQ-V CMDQ Page1)
+ * 0x30000  (Virtual Interface Page0)
+ * 0x40000  (Virtual Interface Page1)
+ */
+#define TEGRA241_CMDQV_IO_LEN 0x50000
+
+struct iommu_viommu_tegra241_cmdqv;
+
+typedef struct Tegra241CMDQV {
+    struct iommu_viommu_tegra241_cmdqv *cmdqv_data;
+    SMMUv3AccelState *s_accel;
+    MemoryRegion mmio_cmdqv;
+    qemu_irq irq;
+} Tegra241CMDQV;
+
 const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops(void);
 
 #endif /* HW_ARM_TEGRA241_CMDQV_H */
hw/arm/tegra241-cmdqv.c +38 -2
index 3a19a1af56..2875affc8d 100644
@@ -13,6 +13,17 @@
 #include "smmuv3-accel.h"
 #include "tegra241-cmdqv.h"
 
+static uint64_t tegra241_cmdqv_read_mmio(void *opaque, hwaddr offset,
+                                         unsigned size)
+{
+    return 0;
+}
+
+static void tegra241_cmdqv_write_mmio(void *opaque, hwaddr offset,
+                                      uint64_t value, unsigned size)
+{
+}
+
 static void tegra241_cmdqv_free_viommu(SMMUv3State *s)
 {
 }
@@ -29,10 +40,35 @@ static void tegra241_cmdqv_reset(SMMUv3State *s)
 {
 }
 
+static const MemoryRegionOps mmio_cmdqv_ops = {
+    .read = tegra241_cmdqv_read_mmio,
+    .write = tegra241_cmdqv_write_mmio,
+    .endianness = DEVICE_LITTLE_ENDIAN,
+    .valid = {
+        .min_access_size = 4,
+        .max_access_size = 8,
+    },
+    .impl = {
+        .min_access_size = 4,
+        .max_access_size = 8,
+    },
+};
+
 static bool tegra241_cmdqv_init(SMMUv3State *s, Error **errp)
 {
-    error_setg(errp, "NVIDIA Tegra241 CMDQV is unsupported");
-    return false;
+    SysBusDevice *sbd = SYS_BUS_DEVICE(OBJECT(s));
+    SMMUv3AccelState *accel = s->s_accel;
+    Tegra241CMDQV *cmdqv;
+
+    cmdqv = g_new0(Tegra241CMDQV, 1);
+    cmdqv->cmdqv_data = g_new0(struct iommu_viommu_tegra241_cmdqv, 1);
+    memory_region_init_io(&cmdqv->mmio_cmdqv, OBJECT(s), &mmio_cmdqv_ops, cmdqv,
+                          "tegra241-cmdqv", TEGRA241_CMDQV_IO_LEN);
+    sysbus_init_mmio(sbd, &cmdqv->mmio_cmdqv);
+    sysbus_init_irq(sbd, &cmdqv->irq);
+    cmdqv->s_accel = accel;
+    accel->cmdqv = cmdqv;
+    return true;
 }
 
 static bool tegra241_cmdqv_probe(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
Signature · 1 line
2.43.0
#13 lore raw

PATCH v6 12/31 hw/arm/virt: Link SMMUv3 CMDQV resources to platform bus

In reply to Shameer Kolothum

SMMUv3 devices with acceleration may enable CMDQV extensions after device realize. In that case, additional MMIO regions and IRQ lines may be registered but not yet mapped to the platform bus.

Ensure SMMUv3 device resources are linked to the platform bus during machine_done().

This is safe to do unconditionally since the platform bus helpers skip resources that are already mapped.

---
 hw/arm/virt.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
hw/arm/virt.c +22 -0
index ac0606fe87..2add7401a1 100644
@@ -2361,6 +2361,25 @@ static void virt_build_smbios(VirtMachineState *vms)
     }
 }
 
+/*
+ * SMMUv3 devices with acceleration may enable CMDQV extensions
+ * after device realize. In that case, additional MMIO regions and
+ * IRQ lines may be registered but not yet mapped to the platform bus.
+ *
+ * Ensure all resources are linked to the platform bus before final
+ * machine setup.
+ */
+
+static void virt_smmuv3_dev_link_cmdqv(VirtMachineState *vms)
+{
+    for (int i = 0; i < vms->smmuv3_devices->len; i++) {
+        DeviceState *dev = g_ptr_array_index(vms->smmuv3_devices, i);
+
+        platform_bus_link_device(PLATFORM_BUS_DEVICE(vms->platform_bus_dev),
+                                 SYS_BUS_DEVICE(dev));
+    }
+}
+
 static
 void virt_machine_done(Notifier *notifier, void *data)
 {
@@ -2377,6 +2396,9 @@ void virt_machine_done(Notifier *notifier, void *data)
     if (vms->cxl_devices_state.is_enabled) {
         cxl_fmws_link_targets(&error_fatal);
     }
+
+    virt_smmuv3_dev_link_cmdqv(vms);
+
     /*
      * If the user provided a dtb, we assume the dynamic sysbus nodes
      * already are integrated there. This corresponds to a use case where
Signature · 1 line
2.43.0
#14 lore raw

PATCH v6 13/31 hw/arm/tegra241-cmdqv: Implement CMDQV vIOMMU alloc/free

In reply to Shameer Kolothum

From: Nicolin Chen <[email protected]>

Replace the stub implementation with real vIOMMU allocation for Tegra241 CMDQV.

Allocate a matching vEVENTQ together with the vIOMMU, since it is specific to the Tegra241 CMDQV vIOMMU and used to receive CMDQV events.

Free both objects on teardown.

---
 hw/arm/tegra241-cmdqv.h |  1 +
 hw/arm/tegra241-cmdqv.c | 48 ++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 48 insertions(+), 1 deletion(-)
hw/arm/tegra241-cmdqv.h +1 -0
index a5f65a8991..9fc720e96c 100644
@@ -32,6 +32,7 @@ typedef struct Tegra241CMDQV {
     SMMUv3AccelState *s_accel;
     MemoryRegion mmio_cmdqv;
     qemu_irq irq;
+    IOMMUFDVeventq *veventq;
 } Tegra241CMDQV;
 
 const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops(void);
hw/arm/tegra241-cmdqv.c +47 -1

66 lines not on this page · raw message

Signature · 1 line
2.43.0
#15 lore raw

PATCH v6 14/31 hw/arm/tegra241-cmdqv: Emulate CMDQ-V Config region

In reply to Shameer Kolothum

From: Nicolin Chen <[email protected]>

Tegra241 CMDQV exposes control and status registers in the CMDQ-V Config page (offset [0x0, 0x10000)) used to configure virtual command queue allocation and interrupt behavior.

Add read/write emulation for the CMDQ-V Config region ([CMDQV_BASE, CMDQV_CMDQ_BASE]), backed by a simple register cache. This includes CONFIG, PARAM, STATUS, VI error and interrupt maps, CMDQ allocation map and the VINTF0 related registers defined in the CMDQ-V Config space. Only VINTF0 is supported; VINTF1-63 are not.

Dispatch writes on access size: Introduced writel_mmio for 4-byte and writell_mmio for 8-byte. Reads need no split as the MMIO framework masks the returned value to the access size.

---
 hw/arm/tegra241-cmdqv.h | 110 ++++++++++++++++++++++++
 hw/arm/tegra241-cmdqv.c | 181 +++++++++++++++++++++++++++++++++++++++-
 hw/arm/trace-events     |   4 +
 3 files changed, 294 insertions(+), 1 deletion(-)
hw/arm/tegra241-cmdqv.h +110 -0
index 9fc720e96c..8034b3b752 100644
@@ -10,10 +10,15 @@
 #ifndef HW_ARM_TEGRA241_CMDQV_H
 #define HW_ARM_TEGRA241_CMDQV_H
 
+#include "hw/core/registerfields.h"
+
 #define CMDQV_VER                 1
 #define CMDQV_NUM_CMDQ_LOG2       1
 #define CMDQV_NUM_SID_PER_VI_LOG2 4
 
+#define TEGRA241_CMDQV_MAX_CMDQ      (1U << CMDQV_NUM_CMDQ_LOG2)
+#define TEGRA241_CMDQV_MAX_NUM_SID   (1U << CMDQV_NUM_SID_PER_VI_LOG2)
+
 /*
  * Tegra241 CMDQV MMIO layout (64KB pages)
  *
@@ -33,8 +38,113 @@ typedef struct Tegra241CMDQV {
     MemoryRegion mmio_cmdqv;
     qemu_irq irq;
     IOMMUFDVeventq *veventq;
+
+    /* CMDQ-V Config page register cache */
+    uint32_t config;
+    uint32_t param;
+    uint32_t status;
+    uint32_t vi_err_map[2];
+    uint32_t vi_int_mask[2];
+    uint32_t cmdq_err_map[4];
+    uint32_t cmdq_alloc_map[TEGRA241_CMDQV_MAX_CMDQ];
+
+    /* VINTF0 register cache (within CMDQ-V Config page) */
+    uint32_t vintf_config;
+    uint32_t vintf_status;
+    uint32_t vintf_sid_match[TEGRA241_CMDQV_MAX_NUM_SID];
+    uint32_t vintf_sid_replace[TEGRA241_CMDQV_MAX_NUM_SID];
+    uint32_t vintf_cmdq_err_map[4];
 } Tegra241CMDQV;
 
+/* CMDQ-V Config page registers (offset 0x00000) */
+REG32(CONFIG, 0x0)
+FIELD(CONFIG, CMDQV_EN, 0, 1)
+FIELD(CONFIG, CMDQV_PER_CMD_OFFSET, 1, 3)
+FIELD(CONFIG, CMDQ_MAX_CLK_BATCH, 4, 8)
+FIELD(CONFIG, CMDQ_MAX_CMD_BATCH, 12, 8)
+FIELD(CONFIG, CONS_DRAM_EN, 20, 1)
+
+REG32(PARAM, 0x4)
+FIELD(PARAM, CMDQV_VER, 0, 4)
+FIELD(PARAM, CMDQV_NUM_CMDQ_LOG2, 4, 4)
+FIELD(PARAM, CMDQV_NUM_VI_LOG2, 8, 4)
+FIELD(PARAM, CMDQV_NUM_SID_PER_VI_LOG2, 12, 4)
+
+REG32(STATUS, 0x8)
+FIELD(STATUS, CMDQV_ENABLED, 0, 1)
+
+/* SMMU_CMDQV_VI_ERR_MAP_0/1 definitions */
+#define A_VI_ERR_MAP_0 0x14
+#define A_VI_ERR_MAP_1 0x18
+#define V_VI_ERR_MAP_NO_ERROR (0)
+#define V_VI_ERR_MAP_ERROR (1)
+
+/* SMMU_CMDQV_VI_INT_MASK_0/1 definitions */
+#define A_VI_INT_MASK_0 0x1c
+#define A_VI_INT_MASK_1 0x20
+#define V_VI_INT_MASK_NOT_MASKED (0)
+#define V_VI_INT_MASK_MASKED (1)
+
+/* SMMU_CMDQV_CMDQ_ERR_MAP_0-3 definitions */
+#define A_CMDQ_ERR_MAP_0 0x24
+#define A_CMDQ_ERR_MAP_1 0x28
+#define A_CMDQ_ERR_MAP_2 0x2c
+#define A_CMDQ_ERR_MAP_3 0x30
+
+/*
+ * CMDQ_ALLOC_MAP: one entry per physical VCMDQ. Hardware supports up to 128
+ * entries (CMDQV_NUM_CMDQ_LOG2=7), but QEMU only exposes
+ * TEGRA241_CMDQV_MAX_CMDQ (=2) VCMDQs per VM so only entries 0 and 1 are
+ * defined here.
+ */
+/* 2 identical register entries */
+#define SMMU_CMDQV_CMDQ_ALLOC_MAP_(i)                       \
+    REG32(CMDQ_ALLOC_MAP_##i, 0x200 + i * 4)                \
+    FIELD(CMDQ_ALLOC_MAP_##i, ALLOC, 0, 1)                  \
+    FIELD(CMDQ_ALLOC_MAP_##i, LVCMDQ, 1, 7)                 \
+    FIELD(CMDQ_ALLOC_MAP_##i, VIRT_INTF_INDX, 15, 6)
+
+SMMU_CMDQV_CMDQ_ALLOC_MAP_(0)
+SMMU_CMDQV_CMDQ_ALLOC_MAP_(1)
+
+/* SMMU_CMDQV_VINTF0 registers (only VINTF0 is exposed to the guest) */
+REG32(VINTF0_CONFIG, 0x1000)
+FIELD(VINTF0_CONFIG, ENABLE, 0, 1)
+FIELD(VINTF0_CONFIG, VMID, 1, 16)
+FIELD(VINTF0_CONFIG, HYP_OWN, 17, 1)
+
+REG32(VINTF0_STATUS, 0x1004)
+FIELD(VINTF0_STATUS, ENABLE_OK, 0, 1)
+FIELD(VINTF0_STATUS, STATUS, 1, 3)
+FIELD(VINTF0_STATUS, VI_NUM_LVCMDQ, 16, 8)
+
+#define V_VINTF_STATUS_NO_ERROR    (0 << 1)
+#define V_VINTF_STATUS_VCMDQ_ERROR (1 << 1)
+
+/*
+ * SMMU_CMDQV_VINTF0_SID_MATCH/_REPLACE: 16 entries per VINTF
+ * (CMDQV_NUM_SID_PER_VI_LOG2=4). Only _0 and _15 are defined,
+ * used as switch case range bounds.
+ */
+REG32(VINTF0_SID_MATCH_0, 0x1040)
+FIELD(VINTF0_SID_MATCH_0, ENABLE, 0, 1)
+FIELD(VINTF0_SID_MATCH_0, VIRT_SID, 1, 20)
+#define A_VINTF0_SID_MATCH_15  (A_VINTF0_SID_MATCH_0 + 15 * 4)
+
+REG32(VINTF0_SID_REPLACE_0, 0x1080)
+FIELD(VINTF0_SID_REPLACE_0, PHYS_SID, 0, 20)
+#define A_VINTF0_SID_REPLACE_15 (A_VINTF0_SID_REPLACE_0 + 15 * 4)
+
+/*
+ * SMMU_CMDQV_VINTF0_LVCMDQ_ERR_MAP: 4 registers per VINTF covering 32 logical
+ * VCMDQs each. With TEGRA241_CMDQV_MAX_CMDQ=2, only MAP_0 bits [1:0] carry
+ * error state. MAP_1..MAP_3 always read as 0. Only _0 and _3 are defined,
+ * used as switch case range bounds.
+ */
+REG32(VINTF0_LVCMDQ_ERR_MAP_0, 0x10c0)
+FIELD(VINTF0_LVCMDQ_ERR_MAP_0, LVCMDQ_ERR_MAP, 0, 32)
+#define A_VINTF0_LVCMDQ_ERR_MAP_3 (A_VINTF0_LVCMDQ_ERR_MAP_0 + 3 * 4)
+
 const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops(void);
 
 #endif /* HW_ARM_TEGRA241_CMDQV_H */
hw/arm/tegra241-cmdqv.c +180 -1

201 lines not on this page · raw message

hw/arm/trace-events +4 -0
index 3457536fb0..8c61d66a26 100644
@@ -72,6 +72,10 @@ smmuv3_accel_unset_iommu_device(int devfn, uint32_t devid) "devfn=0x%x (idev dev
 smmuv3_accel_translate_ste(uint32_t vsid, uint32_t hwpt_id, uint64_t ste_1, uint64_t ste_0) "vSID=0x%x hwpt_id=0x%x ste=%"PRIx64":%"PRIx64
 smmuv3_accel_install_ste(uint32_t vsid, const char * type, uint32_t hwpt_id) "vSID=0x%x ste type=%s hwpt_id=0x%x"
 
+# tegra241-cmdqv
+tegra241_cmdqv_read_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x"
+tegra241_cmdqv_write_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x"
+
 # strongarm.c
 strongarm_uart_update_parameters(const char *label, int speed, char parity, int data_bits, int stop_bits) "%s speed=%d parity=%c data=%d stop=%d"
 strongarm_ssp_read_underrun(void) "SSP rx underrun"
Signature · 1 line
2.43.0
#16 lore raw

PATCH v6 15/31 hw/arm/tegra241-cmdqv: Emulate VCMDQ register reads

In reply to Shameer Kolothum

From: Nicolin Chen <[email protected]>

Tegra241 CMDQV exposes per-VCMDQ register windows through two MMIO apertures:

Direct VCMDQ aperture (0x10000/0x20000): VCMDQ Page0/Page1 VINTF logical aperture (0x30000/0x40000): VINTF0 LVCMDQ Page0/Page1

Both apertures are hardware aliases of the same underlying registers:

Page 0 (control/status): CONS_INDX, PROD_INDX, CONFIG, STATUS,
GERROR, GERRORN
Page 1 (base/DRAM): BASE_L/H, CONS_INDX_BASE_DRAM_L/H

The direct aperture Page 0 is programmable at any time so long as CMDQV_EN is enabled. The VINTF (logical) aperture Page 0 is programmable only once SW has mapped a VCMDQ to a VINTF; the "logical" view is local to that VINTF.

Add read emulation for both apertures, backed by a single per-VCMDQ register cache. VINTF aperture reads are translated to their equivalent direct-aperture offset and served from the same cached state.

Per the CMDQV architecture, a VCMDQ must be allocated to a Virtual Interface before it is used to send commands to the SMMU. Until that allocation happens, reads return cached register state with no HW interaction. Subsequent patches wire up IOMMU_HW_QUEUE_ALLOC, mmap the host VINTF Page 0, and install it into guest MMIO; after that, Page 0 reads from either aperture are served from the hardware-backed mmap'd page instead of the cache. Page 1 is also a hardware alias, but the kernel only exposes mmap for Page 0, so Page 1 reads always trap to QEMU and are served from cache.

---
 hw/arm/tegra241-cmdqv.h | 216 ++++++++++++++++++++++++++++++++++++++++
 hw/arm/tegra241-cmdqv.c | 103 +++++++++++++++++++
 hw/arm/trace-events     |   2 +
 3 files changed, 321 insertions(+)
hw/arm/tegra241-cmdqv.h +216 -0
index 8034b3b752..ad7fb8725f 100644
@@ -30,6 +30,13 @@
  */
 #define TEGRA241_CMDQV_IO_LEN 0x50000
 
+/* CMDQV MMIO aperture bases and VCMDQ stride */
+#define CMDQV_VCMDQ_PAGE0_BASE  0x10000  /* CMDQV_CMDQ_BASE */
+#define CMDQV_VCMDQ_PAGE1_BASE  0x20000
+#define CMDQV_VINTF_PAGE0_BASE  0x30000  /* CMDQV_VI_CMDQ_BASE */
+#define CMDQV_VINTF_PAGE1_BASE  0x40000
+#define CMDQV_VCMDQ_STRIDE      0x80
+
 struct iommu_viommu_tegra241_cmdqv;
 
 typedef struct Tegra241CMDQV {
@@ -54,6 +61,19 @@ typedef struct Tegra241CMDQV {
     uint32_t vintf_sid_match[TEGRA241_CMDQV_MAX_NUM_SID];
     uint32_t vintf_sid_replace[TEGRA241_CMDQV_MAX_NUM_SID];
     uint32_t vintf_cmdq_err_map[4];
+    /*
+     * VCMDQ register cache. The direct (VCMDQ aperture) and logical
+     * (VINTF aperture) views are hardware aliases; both are served from
+     * this single cached copy.
+     */
+    uint32_t vcmdq_cons_indx[TEGRA241_CMDQV_MAX_CMDQ];
+    uint32_t vcmdq_prod_indx[TEGRA241_CMDQV_MAX_CMDQ];
+    uint32_t vcmdq_config[TEGRA241_CMDQV_MAX_CMDQ];
+    uint32_t vcmdq_status[TEGRA241_CMDQV_MAX_CMDQ];
+    uint32_t vcmdq_gerror[TEGRA241_CMDQV_MAX_CMDQ];
+    uint32_t vcmdq_gerrorn[TEGRA241_CMDQV_MAX_CMDQ];
+    uint64_t vcmdq_base[TEGRA241_CMDQV_MAX_CMDQ];
+    uint64_t vcmdq_cons_indx_base[TEGRA241_CMDQV_MAX_CMDQ];
 } Tegra241CMDQV;
 
 /* CMDQ-V Config page registers (offset 0x00000) */
@@ -145,6 +165,202 @@ REG32(VINTF0_LVCMDQ_ERR_MAP_0, 0x10c0)
 FIELD(VINTF0_LVCMDQ_ERR_MAP_0, LVCMDQ_ERR_MAP, 0, 32)
 #define A_VINTF0_LVCMDQ_ERR_MAP_3 (A_VINTF0_LVCMDQ_ERR_MAP_0 + 3 * 4)
 
+/*
+ * Direct VCMDQ aperture register windows.
+ *
+ * Page 0 @ CMDQV_VCMDQ_PAGE0_BASE: VCMDQ control and status registers.
+ * Page 1 @ CMDQV_VCMDQ_PAGE1_BASE: VCMDQ base and DRAM address registers.
+ *
+ * Each VCMDQ occupies a CMDQV_VCMDQ_STRIDE-byte slot within its page.
+ */
+
+/* --- Page 0 register macros --- */
+#define SMMU_CMDQV_VCMDQi_CONS_INDX_(i)                     \
+    REG32(VCMDQ##i##_CONS_INDX,                             \
+          CMDQV_VCMDQ_PAGE0_BASE + i * CMDQV_VCMDQ_STRIDE)  \
+    FIELD(VCMDQ##i##_CONS_INDX, RD, 0, 20)                  \
+    FIELD(VCMDQ##i##_CONS_INDX, ERR, 24, 7)
+
+#define V_VCMDQ_CONS_INDX_ERR_CERROR_NONE         0
+#define V_VCMDQ_CONS_INDX_ERR_CERROR_ILL_OPCODE   1
+#define V_VCMDQ_CONS_INDX_ERR_CERROR_ABT          2
+#define V_VCMDQ_CONS_INDX_ERR_CERROR_ATC_INV_SYNC 3
+#define V_VCMDQ_CONS_INDX_ERR_CERROR_ILL_ACCESS   4
+
+#define SMMU_CMDQV_VCMDQi_PROD_INDX_(i)                          \
+    REG32(VCMDQ##i##_PROD_INDX,                                  \
+          CMDQV_VCMDQ_PAGE0_BASE + 0x4 + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VCMDQ##i##_PROD_INDX, WR, 0, 20)
+
+#define SMMU_CMDQV_VCMDQi_CONFIG_(i)                             \
+    REG32(VCMDQ##i##_CONFIG,                                     \
+          CMDQV_VCMDQ_PAGE0_BASE + 0x8 + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VCMDQ##i##_CONFIG, CMDQ_EN, 0, 1)
+
+#define SMMU_CMDQV_VCMDQi_STATUS_(i)                             \
+    REG32(VCMDQ##i##_STATUS,                                     \
+          CMDQV_VCMDQ_PAGE0_BASE + 0xc + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VCMDQ##i##_STATUS, CMDQ_EN_OK, 0, 1)
+
+#define SMMU_CMDQV_VCMDQi_GERROR_(i)                              \
+    REG32(VCMDQ##i##_GERROR,                                      \
+          CMDQV_VCMDQ_PAGE0_BASE + 0x10 + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VCMDQ##i##_GERROR, CMDQ_ERR, 0, 1)                      \
+    FIELD(VCMDQ##i##_GERROR, CONS_DRAM_WR_ABT_ERR, 1, 1)          \
+    FIELD(VCMDQ##i##_GERROR, CMDQ_INIT_ERR, 2, 1)
+
+#define SMMU_CMDQV_VCMDQi_GERRORN_(i)                             \
+    REG32(VCMDQ##i##_GERRORN,                                     \
+          CMDQV_VCMDQ_PAGE0_BASE + 0x14 + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VCMDQ##i##_GERRORN, CMDQ_ERR, 0, 1)                     \
+    FIELD(VCMDQ##i##_GERRORN, CONS_DRAM_WR_ABT_ERR, 1, 1)         \
+    FIELD(VCMDQ##i##_GERRORN, CMDQ_INIT_ERR, 2, 1)
+
+/* Page 0 layout: VCMDQ0 */
+SMMU_CMDQV_VCMDQi_CONS_INDX_(0)
+SMMU_CMDQV_VCMDQi_PROD_INDX_(0)
+SMMU_CMDQV_VCMDQi_CONFIG_(0)
+SMMU_CMDQV_VCMDQi_STATUS_(0)
+SMMU_CMDQV_VCMDQi_GERROR_(0)
+SMMU_CMDQV_VCMDQi_GERRORN_(0)
+
+/* Page 0 layout: VCMDQ1 */
+SMMU_CMDQV_VCMDQi_CONS_INDX_(1)
+SMMU_CMDQV_VCMDQi_PROD_INDX_(1)
+SMMU_CMDQV_VCMDQi_CONFIG_(1)
+SMMU_CMDQV_VCMDQi_STATUS_(1)
+SMMU_CMDQV_VCMDQi_GERROR_(1)
+SMMU_CMDQV_VCMDQi_GERRORN_(1)
+
+/* --- Page 1 register macros --- */
+#define SMMU_CMDQV_VCMDQi_BASE_L_(i)                                          \
+    REG32(VCMDQ##i##_BASE_L, CMDQV_VCMDQ_PAGE1_BASE + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VCMDQ##i##_BASE_L, LOG2SIZE, 0, 5)                                  \
+    FIELD(VCMDQ##i##_BASE_L, ADDR, 5, 27)
+
+#define SMMU_CMDQV_VCMDQi_BASE_H_(i)                             \
+    REG32(VCMDQ##i##_BASE_H,                                     \
+          CMDQV_VCMDQ_PAGE1_BASE + 0x4 + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VCMDQ##i##_BASE_H, ADDR, 0, 16)
+
+#define SMMU_CMDQV_VCMDQi_CONS_INDX_BASE_DRAM_L_(i)              \
+    REG32(VCMDQ##i##_CONS_INDX_BASE_DRAM_L,                      \
+          CMDQV_VCMDQ_PAGE1_BASE + 0x8 + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VCMDQ##i##_CONS_INDX_BASE_DRAM_L, ADDR, 0, 32)
+
+#define SMMU_CMDQV_VCMDQi_CONS_INDX_BASE_DRAM_H_(i)              \
+    REG32(VCMDQ##i##_CONS_INDX_BASE_DRAM_H,                      \
+          CMDQV_VCMDQ_PAGE1_BASE + 0xc + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VCMDQ##i##_CONS_INDX_BASE_DRAM_H, ADDR, 0, 16)
+
+/* Page 1 layout: VCMDQ0 */
+SMMU_CMDQV_VCMDQi_BASE_L_(0)
+SMMU_CMDQV_VCMDQi_BASE_H_(0)
+SMMU_CMDQV_VCMDQi_CONS_INDX_BASE_DRAM_L_(0)
+SMMU_CMDQV_VCMDQi_CONS_INDX_BASE_DRAM_H_(0)
+
+/* Page 1 layout: VCMDQ1 */
+SMMU_CMDQV_VCMDQi_BASE_L_(1)
+SMMU_CMDQV_VCMDQi_BASE_H_(1)
+SMMU_CMDQV_VCMDQi_CONS_INDX_BASE_DRAM_L_(1)
+SMMU_CMDQV_VCMDQi_CONS_INDX_BASE_DRAM_H_(1)
+
+/*
+ * VINTF0 logical VCMDQ aperture register windows.
+ *
+ * Page 0 @ CMDQV_VINTF_PAGE0_BASE: VCMDQ control and status registers.
+ * Page 1 @ CMDQV_VINTF_PAGE1_BASE: VCMDQ base and DRAM address registers.
+ *
+ * VCMDQs mapped via VINTF are accessed through this aperture as
+ * hardware aliases of the direct VCMDQ aperture above.
+ */
+
+/* --- Page 0 register macros --- */
+#define SMMU_CMDQV_VI_VCMDQi_CONS_INDX_(i)                  \
+    REG32(VI_VCMDQ##i##_CONS_INDX,                          \
+          CMDQV_VINTF_PAGE0_BASE + i * CMDQV_VCMDQ_STRIDE)  \
+    FIELD(VI_VCMDQ##i##_CONS_INDX, RD, 0, 20)               \
+    FIELD(VI_VCMDQ##i##_CONS_INDX, ERR, 24, 7)
+
+#define SMMU_CMDQV_VI_VCMDQi_PROD_INDX_(i)                       \
+    REG32(VI_VCMDQ##i##_PROD_INDX,                               \
+          CMDQV_VINTF_PAGE0_BASE + 0x4 + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VI_VCMDQ##i##_PROD_INDX, WR, 0, 20)
+
+#define SMMU_CMDQV_VI_VCMDQi_CONFIG_(i)                          \
+    REG32(VI_VCMDQ##i##_CONFIG,                                  \
+          CMDQV_VINTF_PAGE0_BASE + 0x8 + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VI_VCMDQ##i##_CONFIG, CMDQ_EN, 0, 1)
+
+#define SMMU_CMDQV_VI_VCMDQi_STATUS_(i)                          \
+    REG32(VI_VCMDQ##i##_STATUS,                                  \
+          CMDQV_VINTF_PAGE0_BASE + 0xc + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VI_VCMDQ##i##_STATUS, CMDQ_EN_OK, 0, 1)
+
+#define SMMU_CMDQV_VI_VCMDQi_GERROR_(i)                           \
+    REG32(VI_VCMDQ##i##_GERROR,                                   \
+          CMDQV_VINTF_PAGE0_BASE + 0x10 + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VI_VCMDQ##i##_GERROR, CMDQ_ERR, 0, 1)                   \
+    FIELD(VI_VCMDQ##i##_GERROR, CONS_DRAM_WR_ABT_ERR, 1, 1)       \
+    FIELD(VI_VCMDQ##i##_GERROR, CMDQ_INIT_ERR, 2, 1)
+
+#define SMMU_CMDQV_VI_VCMDQi_GERRORN_(i)                          \
+    REG32(VI_VCMDQ##i##_GERRORN,                                  \
+          CMDQV_VINTF_PAGE0_BASE + 0x14 + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VI_VCMDQ##i##_GERRORN, CMDQ_ERR, 0, 1)                  \
+    FIELD(VI_VCMDQ##i##_GERRORN, CONS_DRAM_WR_ABT_ERR, 1, 1)      \
+    FIELD(VI_VCMDQ##i##_GERRORN, CMDQ_INIT_ERR, 2, 1)
+
+/* Page 0 layout: VCMDQ0 */
+SMMU_CMDQV_VI_VCMDQi_CONS_INDX_(0)
+SMMU_CMDQV_VI_VCMDQi_PROD_INDX_(0)
+SMMU_CMDQV_VI_VCMDQi_CONFIG_(0)
+SMMU_CMDQV_VI_VCMDQi_STATUS_(0)
+SMMU_CMDQV_VI_VCMDQi_GERROR_(0)
+SMMU_CMDQV_VI_VCMDQi_GERRORN_(0)
+
+/* Page 0 layout: VCMDQ1 */
+SMMU_CMDQV_VI_VCMDQi_CONS_INDX_(1)
+SMMU_CMDQV_VI_VCMDQi_PROD_INDX_(1)
+SMMU_CMDQV_VI_VCMDQi_CONFIG_(1)
+SMMU_CMDQV_VI_VCMDQi_STATUS_(1)
+SMMU_CMDQV_VI_VCMDQi_GERROR_(1)
+SMMU_CMDQV_VI_VCMDQi_GERRORN_(1)
+
+/* --- Page 1 register macros --- */
+#define SMMU_CMDQV_VI_VCMDQi_BASE_L_(i)                     \
+    REG32(VI_VCMDQ##i##_BASE_L,                             \
+          CMDQV_VINTF_PAGE1_BASE + i * CMDQV_VCMDQ_STRIDE)  \
+    FIELD(VI_VCMDQ##i##_BASE_L, LOG2SIZE, 0, 5)             \
+    FIELD(VI_VCMDQ##i##_BASE_L, ADDR, 5, 27)
+
+#define SMMU_CMDQV_VI_VCMDQi_BASE_H_(i)                          \
+    REG32(VI_VCMDQ##i##_BASE_H,                                  \
+          CMDQV_VINTF_PAGE1_BASE + 0x4 + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VI_VCMDQ##i##_BASE_H, ADDR, 0, 16)
+
+#define SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_L_(i)           \
+    REG32(VI_VCMDQ##i##_CONS_INDX_BASE_DRAM_L,                   \
+          CMDQV_VINTF_PAGE1_BASE + 0x8 + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VI_VCMDQ##i##_CONS_INDX_BASE_DRAM_L, ADDR, 0, 32)
+
+#define SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_(i)           \
+    REG32(VI_VCMDQ##i##_CONS_INDX_BASE_DRAM_H,                   \
+          CMDQV_VINTF_PAGE1_BASE + 0xc + i * CMDQV_VCMDQ_STRIDE) \
+    FIELD(VI_VCMDQ##i##_CONS_INDX_BASE_DRAM_H, ADDR, 0, 16)
+
+/* Page 1 layout: VCMDQ0 */
+SMMU_CMDQV_VI_VCMDQi_BASE_L_(0)
+SMMU_CMDQV_VI_VCMDQi_BASE_H_(0)
+SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_L_(0)
+SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_(0)
+
+/* Page 1 layout: VCMDQ1 */
+SMMU_CMDQV_VI_VCMDQi_BASE_L_(1)
+SMMU_CMDQV_VI_VCMDQi_BASE_H_(1)
+SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_L_(1)
+SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_(1)
+
 const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops(void);
 
 #endif /* HW_ARM_TEGRA241_CMDQV_H */
hw/arm/tegra241-cmdqv.c +103 -0

121 lines not on this page · raw message

hw/arm/trace-events +2 -0
index 8c61d66a26..4392135fa2 100644
@@ -75,6 +75,8 @@ smmuv3_accel_install_ste(uint32_t vsid, const char * type, uint32_t hwpt_id) "vS
 # tegra241-cmdqv
 tegra241_cmdqv_read_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x"
 tegra241_cmdqv_write_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x"
+tegra241_cmdqv_read_vcmdq_page0(int index, const char *aperture, uint64_t offset0, uint64_t val) "vcmdq[%d] %s page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64
+tegra241_cmdqv_read_vcmdq_page1(int index, const char *aperture, uint64_t offset0, uint64_t val) "vcmdq[%d] %s page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64
 
 # strongarm.c
 strongarm_uart_update_parameters(const char *label, int speed, char parity, int data_bits, int stop_bits) "%s speed=%d parity=%c data=%d stop=%d"
Signature · 1 line
2.43.0
#17 lore raw

PATCH v6 16/31 hw/arm/tegra241-cmdqv: Emulate VCMDQ register writes

In reply to Shameer Kolothum

From: Nicolin Chen <[email protected]>

This is the write side counterpart of the VCMDQ read emulation. Add write handling for both the direct VCMDQ aperture and the VINTF logical aperture using the same index decoding and VINTF-to-VCMDQ translation logic as the read path.

VINTF aperture writes are translated to their direct-aperture equivalent and update the same cached state. Page 1 registers (BASE, CONS_INDX_BASE) always update the cache.

Per the CMDQV architecture, a VCMDQ must be allocated to a Virtual Interface before it is used to send commands to the SMMU. Until that allocation happens, MMIO writes only update cached register state - no command consumption, error handling, or interrupt activity is driven from these writes. Subsequent patches wire up IOMMU_HW_QUEUE_ALLOC, mmap the host VINTF Page 0, and install it into guest MMIO; after that, Page 0 writes from either aperture reach the hardware-backed mmap'd page instead of just the cache.

---
 hw/arm/tegra241-cmdqv.c | 159 +++++++++++++++++++++++++++++++++++++++-
 hw/arm/trace-events     |   2 +
 2 files changed, 159 insertions(+), 2 deletions(-)
hw/arm/tegra241-cmdqv.c +157 -2
index ff150fe0f3..3cb1b45575 100644
@@ -120,6 +120,104 @@ static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv,
     }
 }
 
+/*
+ * Write a VCMDQ Page 0 register (control/status) using VCMDQ0_* offsets.
+ *
+ * The caller normalizes the MMIO offset such that @offset0 always refers
+ * to a VCMDQ0_* register, while @index selects the VCMDQ instance.
+ *
+ * Page 0 registers are all 32-bit; this helper is only called for 4-byte
+ * writes.
+ */
+static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv,
+                                             hwaddr offset0, int index,
+                                             uint32_t value, bool direct)
+{
+    switch (offset0) {
+    case A_VCMDQ0_CONS_INDX:
+        cmdqv->vcmdq_cons_indx[index] = value;
+        break;
+    case A_VCMDQ0_PROD_INDX:
+        /* VCMDQ is functional only once allocated to a VINTF; cache only. */
+        cmdqv->vcmdq_prod_indx[index] = value;
+        break;
+    case A_VCMDQ0_CONFIG:
+        if (value & R_VCMDQ0_CONFIG_CMDQ_EN_MASK) {
+            cmdqv->vcmdq_status[index] |= R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;
+        } else {
+            cmdqv->vcmdq_status[index] &= ~R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;
+        }
+        cmdqv->vcmdq_config[index] = value;
+        break;
+    case A_VCMDQ0_GERRORN:
+        /* VCMDQ is functional only once allocated to a VINTF; cache only. */
+        cmdqv->vcmdq_gerrorn[index] = value;
+        break;
+    default:
+        qemu_log_mask(LOG_UNIMP,
+                      "%s unhandled write access at 0x%" PRIx64 "\n",
+                      __func__, offset0);
+    }
+    trace_tegra241_cmdqv_write_vcmdq_page0(index, direct ? "direct" : "vi",
+                                           offset0, value);
+}
+
+/*
+ * Write a VCMDQ Page 1 register (base / DRAM address) - 4-byte access.
+ */
+static void tegra241_cmdqv_write_vcmdq_page1(Tegra241CMDQV *cmdqv,
+                                             hwaddr offset0, int index,
+                                             uint32_t value, bool direct)
+{
+    switch (offset0) {
+    case A_VCMDQ0_BASE_L:
+        cmdqv->vcmdq_base[index] =
+            deposit64(cmdqv->vcmdq_base[index], 0, 32, value);
+        break;
+    case A_VCMDQ0_BASE_H:
+        cmdqv->vcmdq_base[index] =
+            deposit64(cmdqv->vcmdq_base[index], 32, 32, value);
+        break;
+    case A_VCMDQ0_CONS_INDX_BASE_DRAM_L:
+        cmdqv->vcmdq_cons_indx_base[index] =
+            deposit64(cmdqv->vcmdq_cons_indx_base[index], 0, 32, value);
+        break;
+    case A_VCMDQ0_CONS_INDX_BASE_DRAM_H:
+        cmdqv->vcmdq_cons_indx_base[index] =
+            deposit64(cmdqv->vcmdq_cons_indx_base[index], 32, 32, value);
+        break;
+    default:
+        qemu_log_mask(LOG_UNIMP,
+                      "%s unhandled write access at 0x%" PRIx64 "\n",
+                      __func__, offset0);
+    }
+    trace_tegra241_cmdqv_write_vcmdq_page1(index, direct ? "direct" : "vi",
+                                           offset0, value);
+}
+
+/*
+ * Write a VCMDQ Page 1 register - 8-byte access at BASE_L or DRAM_L.
+ */
+static void tegra241_cmdqv_write_vcmdq_page1_64(Tegra241CMDQV *cmdqv,
+                                                hwaddr offset0, int index,
+                                                uint64_t value, bool direct)
+{
+    switch (offset0) {
+    case A_VCMDQ0_BASE_L:
+        cmdqv->vcmdq_base[index] = value;
+        break;
+    case A_VCMDQ0_CONS_INDX_BASE_DRAM_L:
+        cmdqv->vcmdq_cons_indx_base[index] = value;
+        break;
+    default:
+        qemu_log_mask(LOG_UNIMP,
+                      "%s unhandled 64-bit write at 0x%" PRIx64 "\n",
+                      __func__, offset0);
+    }
+    trace_tegra241_cmdqv_write_vcmdq_page1(index, direct ? "direct" : "vi",
+                                           offset0, value);
+}
+
 static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv,
                                               hwaddr offset, uint64_t value)
 {
@@ -243,6 +341,8 @@ out:
 static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,
                                        uint32_t value)
 {
+    int index;
+
     switch (offset) {
     case A_CONFIG:
         cmdqv->config = value;
@@ -261,6 +361,39 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,
     case A_VINTF0_CONFIG ... A_VINTF0_LVCMDQ_ERR_MAP_3:
         tegra241_cmdqv_config_vintf_write(cmdqv, offset, value);
         break;
+    case A_VI_VCMDQ0_CONS_INDX ... A_VI_VCMDQ1_GERRORN:
+        /*
+         * VINTF Page0 registers are hardware aliases of VCMDQ Page0 registers.
+         * Translate the VINTF aperture offset to its VCMDQ Page0 equivalent
+         * before dispatching to the Page 0 helper.
+         */
+        offset -= CMDQV_VINTF_PAGE0_BASE - CMDQV_VCMDQ_PAGE0_BASE;
+        index = (offset - CMDQV_VCMDQ_PAGE0_BASE) / CMDQV_VCMDQ_STRIDE;
+        tegra241_cmdqv_write_vcmdq_page0(cmdqv,
+                offset - index * CMDQV_VCMDQ_STRIDE, index, value, false);
+        break;
+    case A_VCMDQ0_CONS_INDX ... A_VCMDQ1_GERRORN:
+        /*
+         * Decode a per-VCMDQ Page 0 access. Each VCMDQ occupies a
+         * CMDQV_VCMDQ_STRIDE-byte window; extract the index and normalize
+         * to the VCMDQ0_* offset before calling the Page 0 helper.
+         */
+        index = (offset - CMDQV_VCMDQ_PAGE0_BASE) / CMDQV_VCMDQ_STRIDE;
+        tegra241_cmdqv_write_vcmdq_page0(cmdqv,
+                offset - index * CMDQV_VCMDQ_STRIDE, index, value, true);
+        break;
+    case A_VI_VCMDQ0_BASE_L ... A_VI_VCMDQ1_CONS_INDX_BASE_DRAM_H:
+        /* Same VINTF-to-VCMDQ translation as VINTF Page0 case above. */
+        offset -= CMDQV_VINTF_PAGE1_BASE - CMDQV_VCMDQ_PAGE1_BASE;
+        index = (offset - CMDQV_VCMDQ_PAGE1_BASE) / CMDQV_VCMDQ_STRIDE;
+        tegra241_cmdqv_write_vcmdq_page1(cmdqv,
+                offset - index * CMDQV_VCMDQ_STRIDE, index, value, false);
+        break;
+    case A_VCMDQ0_BASE_L ... A_VCMDQ1_CONS_INDX_BASE_DRAM_H:
+        index = (offset - CMDQV_VCMDQ_PAGE1_BASE) / CMDQV_VCMDQ_STRIDE;
+        tegra241_cmdqv_write_vcmdq_page1(cmdqv,
+                offset - index * CMDQV_VCMDQ_STRIDE, index, value, true);
+        break;
     default:
         qemu_log_mask(LOG_UNIMP, "%s unhandled write access at 0x%" PRIx64 "\n",
                       __func__, offset);
@@ -268,14 +401,36 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,
 }
 
 /*
- * 8-byte MMIO write handler.
+ * 8-byte MMIO write handler. Only Page 1 BASE / CONS_INDX_BASE_DRAM accept
+ * full 64-bit writes; other offsets are write-ignored.
  */
 static void tegra241_cmdqv_writell_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,
                                         uint64_t value)
 {
-    qemu_log_mask(LOG_UNIMP,
+    int index;
+
+    switch (offset) {
+    case A_VI_VCMDQ0_BASE_L ... A_VI_VCMDQ1_CONS_INDX_BASE_DRAM_H:
+        /*
+         * VINTF Page1 registers are hardware aliases of VCMDQ Page1 registers.
+         * Translate the VINTF aperture offset to its VCMDQ Page1 equivalent
+         * before dispatching to the Page 1 helper.
+         */
+        offset -= CMDQV_VINTF_PAGE1_BASE - CMDQV_VCMDQ_PAGE1_BASE;
+        index = (offset - CMDQV_VCMDQ_PAGE1_BASE) / CMDQV_VCMDQ_STRIDE;
+        tegra241_cmdqv_write_vcmdq_page1_64(cmdqv,
+                offset - index * CMDQV_VCMDQ_STRIDE, index, value, false);
+        break;
+    case A_VCMDQ0_BASE_L ... A_VCMDQ1_CONS_INDX_BASE_DRAM_H:
+        index = (offset - CMDQV_VCMDQ_PAGE1_BASE) / CMDQV_VCMDQ_STRIDE;
+        tegra241_cmdqv_write_vcmdq_page1_64(cmdqv,
+                offset - index * CMDQV_VCMDQ_STRIDE, index, value, true);
+        break;
+    default:
+        qemu_log_mask(LOG_UNIMP,
                       "%s unhandled 64-bit write at 0x%" PRIx64 " (WI)\n",
                       __func__, offset);
+    }
 }
 
 static void tegra241_cmdqv_write_mmio(void *opaque, hwaddr offset,
hw/arm/trace-events +2 -0
index 4392135fa2..d76f408d8a 100644
@@ -77,6 +77,8 @@ tegra241_cmdqv_read_mmio(uint64_t offset, uint64_t val, unsigned size) "offset:
 tegra241_cmdqv_write_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x"
 tegra241_cmdqv_read_vcmdq_page0(int index, const char *aperture, uint64_t offset0, uint64_t val) "vcmdq[%d] %s page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64
 tegra241_cmdqv_read_vcmdq_page1(int index, const char *aperture, uint64_t offset0, uint64_t val) "vcmdq[%d] %s page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64
+tegra241_cmdqv_write_vcmdq_page0(int index, const char *aperture, uint64_t offset0, uint64_t val) "vcmdq[%d] %s page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64
+tegra241_cmdqv_write_vcmdq_page1(int index, const char *aperture, uint64_t offset0, uint64_t val) "vcmdq[%d] %s page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64
 
 # strongarm.c
 strongarm_uart_update_parameters(const char *label, int speed, char parity, int data_bits, int stop_bits) "%s speed=%d parity=%c data=%d stop=%d"
Signature · 1 line
2.43.0
Reviewed-by on v6 16/31
#18 lore raw

Re: [PATCH v6 16/31] hw/arm/tegra241-cmdqv: Emulate VCMDQ register writes

Quoted 20 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

This is the write side counterpart of the VCMDQ read emulation. Add write handling for both the direct VCMDQ aperture and the VINTF logical aperture using the same index decoding and VINTF-to-VCMDQ translation logic as the read path.

VINTF aperture writes are translated to their direct-aperture equivalent and update the same cached state. Page 1 registers (BASE, CONS_INDX_BASE) always update the cache.

Per the CMDQV architecture, a VCMDQ must be allocated to a Virtual Interface before it is used to send commands to the SMMU. Until that allocation happens, MMIO writes only update cached register state - no command consumption, error handling, or interrupt activity is driven from these writes. Subsequent patches wire up IOMMU_HW_QUEUE_ALLOC, mmap the host VINTF Page 0, and install it into guest MMIO; after that, Page 0 writes from either aperture reach the hardware-backed mmap'd page instead of just the cache.

Quoted 3 lines

Thanks

Eric

Quoted 215 lines
---
 hw/arm/tegra241-cmdqv.c | 159 +++++++++++++++++++++++++++++++++++++++-
 hw/arm/trace-events     |   2 +
 2 files changed, 159 insertions(+), 2 deletions(-)
hw/arm/tegra241-cmdqv.c +157 -2
index ff150fe0f3..3cb1b45575 100644
@@ -120,6 +120,104 @@ static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv,
     }
 }
 
+/*
+ * Write a VCMDQ Page 0 register (control/status) using VCMDQ0_* offsets.
+ *
+ * The caller normalizes the MMIO offset such that @offset0 always refers
+ * to a VCMDQ0_* register, while @index selects the VCMDQ instance.
+ *
+ * Page 0 registers are all 32-bit; this helper is only called for 4-byte
+ * writes.
+ */
+static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv,
+                                             hwaddr offset0, int index,
+                                             uint32_t value, bool direct)
+{
+    switch (offset0) {
+    case A_VCMDQ0_CONS_INDX:
+        cmdqv->vcmdq_cons_indx[index] = value;
+        break;
+    case A_VCMDQ0_PROD_INDX:
+        /* VCMDQ is functional only once allocated to a VINTF; cache only. */
+        cmdqv->vcmdq_prod_indx[index] = value;
+        break;
+    case A_VCMDQ0_CONFIG:
+        if (value & R_VCMDQ0_CONFIG_CMDQ_EN_MASK) {
+            cmdqv->vcmdq_status[index] |= R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;
+        } else {
+            cmdqv->vcmdq_status[index] &= ~R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;
+        }
+        cmdqv->vcmdq_config[index] = value;
+        break;
+    case A_VCMDQ0_GERRORN:
+        /* VCMDQ is functional only once allocated to a VINTF; cache only. */
+        cmdqv->vcmdq_gerrorn[index] = value;
+        break;
+    default:
+        qemu_log_mask(LOG_UNIMP,
+                      "%s unhandled write access at 0x%" PRIx64 "\n",
+                      __func__, offset0);
+    }
+    trace_tegra241_cmdqv_write_vcmdq_page0(index, direct ? "direct" : "vi",
+                                           offset0, value);
+}
+
+/*
+ * Write a VCMDQ Page 1 register (base / DRAM address) - 4-byte access.
+ */
+static void tegra241_cmdqv_write_vcmdq_page1(Tegra241CMDQV *cmdqv,
+                                             hwaddr offset0, int index,
+                                             uint32_t value, bool direct)
+{
+    switch (offset0) {
+    case A_VCMDQ0_BASE_L:
+        cmdqv->vcmdq_base[index] =
+            deposit64(cmdqv->vcmdq_base[index], 0, 32, value);
+        break;
+    case A_VCMDQ0_BASE_H:
+        cmdqv->vcmdq_base[index] =
+            deposit64(cmdqv->vcmdq_base[index], 32, 32, value);
+        break;
+    case A_VCMDQ0_CONS_INDX_BASE_DRAM_L:
+        cmdqv->vcmdq_cons_indx_base[index] =
+            deposit64(cmdqv->vcmdq_cons_indx_base[index], 0, 32, value);
+        break;
+    case A_VCMDQ0_CONS_INDX_BASE_DRAM_H:
+        cmdqv->vcmdq_cons_indx_base[index] =
+            deposit64(cmdqv->vcmdq_cons_indx_base[index], 32, 32, value);
+        break;
+    default:
+        qemu_log_mask(LOG_UNIMP,
+                      "%s unhandled write access at 0x%" PRIx64 "\n",
+                      __func__, offset0);
+    }
+    trace_tegra241_cmdqv_write_vcmdq_page1(index, direct ? "direct" : "vi",
+                                           offset0, value);
+}
+
+/*
+ * Write a VCMDQ Page 1 register - 8-byte access at BASE_L or DRAM_L.
+ */
+static void tegra241_cmdqv_write_vcmdq_page1_64(Tegra241CMDQV *cmdqv,
+                                                hwaddr offset0, int index,
+                                                uint64_t value, bool direct)
+{
+    switch (offset0) {
+    case A_VCMDQ0_BASE_L:
+        cmdqv->vcmdq_base[index] = value;
+        break;
+    case A_VCMDQ0_CONS_INDX_BASE_DRAM_L:
+        cmdqv->vcmdq_cons_indx_base[index] = value;
+        break;
+    default:
+        qemu_log_mask(LOG_UNIMP,
+                      "%s unhandled 64-bit write at 0x%" PRIx64 "\n",
+                      __func__, offset0);
+    }
+    trace_tegra241_cmdqv_write_vcmdq_page1(index, direct ? "direct" : "vi",
+                                           offset0, value);
+}
+
 static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv,
                                               hwaddr offset, uint64_t value)
 {
@@ -243,6 +341,8 @@ out:
 static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,
                                        uint32_t value)
 {
+    int index;
+
     switch (offset) {
     case A_CONFIG:
         cmdqv->config = value;
@@ -261,6 +361,39 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,
     case A_VINTF0_CONFIG ... A_VINTF0_LVCMDQ_ERR_MAP_3:
         tegra241_cmdqv_config_vintf_write(cmdqv, offset, value);
         break;
+    case A_VI_VCMDQ0_CONS_INDX ... A_VI_VCMDQ1_GERRORN:
+        /*
+         * VINTF Page0 registers are hardware aliases of VCMDQ Page0 registers.
+         * Translate the VINTF aperture offset to its VCMDQ Page0 equivalent
+         * before dispatching to the Page 0 helper.
+         */
+        offset -= CMDQV_VINTF_PAGE0_BASE - CMDQV_VCMDQ_PAGE0_BASE;
+        index = (offset - CMDQV_VCMDQ_PAGE0_BASE) / CMDQV_VCMDQ_STRIDE;
+        tegra241_cmdqv_write_vcmdq_page0(cmdqv,
+                offset - index * CMDQV_VCMDQ_STRIDE, index, value, false);
+        break;
+    case A_VCMDQ0_CONS_INDX ... A_VCMDQ1_GERRORN:
+        /*
+         * Decode a per-VCMDQ Page 0 access. Each VCMDQ occupies a
+         * CMDQV_VCMDQ_STRIDE-byte window; extract the index and normalize
+         * to the VCMDQ0_* offset before calling the Page 0 helper.
+         */
+        index = (offset - CMDQV_VCMDQ_PAGE0_BASE) / CMDQV_VCMDQ_STRIDE;
+        tegra241_cmdqv_write_vcmdq_page0(cmdqv,
+                offset - index * CMDQV_VCMDQ_STRIDE, index, value, true);
+        break;
+    case A_VI_VCMDQ0_BASE_L ... A_VI_VCMDQ1_CONS_INDX_BASE_DRAM_H:
+        /* Same VINTF-to-VCMDQ translation as VINTF Page0 case above. */
+        offset -= CMDQV_VINTF_PAGE1_BASE - CMDQV_VCMDQ_PAGE1_BASE;
+        index = (offset - CMDQV_VCMDQ_PAGE1_BASE) / CMDQV_VCMDQ_STRIDE;
+        tegra241_cmdqv_write_vcmdq_page1(cmdqv,
+                offset - index * CMDQV_VCMDQ_STRIDE, index, value, false);
+        break;
+    case A_VCMDQ0_BASE_L ... A_VCMDQ1_CONS_INDX_BASE_DRAM_H:
+        index = (offset - CMDQV_VCMDQ_PAGE1_BASE) / CMDQV_VCMDQ_STRIDE;
+        tegra241_cmdqv_write_vcmdq_page1(cmdqv,
+                offset - index * CMDQV_VCMDQ_STRIDE, index, value, true);
+        break;
     default:
         qemu_log_mask(LOG_UNIMP, "%s unhandled write access at 0x%" PRIx64 "\n",
                       __func__, offset);
@@ -268,14 +401,36 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,
 }
 
 /*
- * 8-byte MMIO write handler.
+ * 8-byte MMIO write handler. Only Page 1 BASE / CONS_INDX_BASE_DRAM accept
+ * full 64-bit writes; other offsets are write-ignored.
  */
 static void tegra241_cmdqv_writell_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,
                                         uint64_t value)
 {
-    qemu_log_mask(LOG_UNIMP,
+    int index;
+
+    switch (offset) {
+    case A_VI_VCMDQ0_BASE_L ... A_VI_VCMDQ1_CONS_INDX_BASE_DRAM_H:
+        /*
+         * VINTF Page1 registers are hardware aliases of VCMDQ Page1 registers.
+         * Translate the VINTF aperture offset to its VCMDQ Page1 equivalent
+         * before dispatching to the Page 1 helper.
+         */
+        offset -= CMDQV_VINTF_PAGE1_BASE - CMDQV_VCMDQ_PAGE1_BASE;
+        index = (offset - CMDQV_VCMDQ_PAGE1_BASE) / CMDQV_VCMDQ_STRIDE;
+        tegra241_cmdqv_write_vcmdq_page1_64(cmdqv,
+                offset - index * CMDQV_VCMDQ_STRIDE, index, value, false);
+        break;
+    case A_VCMDQ0_BASE_L ... A_VCMDQ1_CONS_INDX_BASE_DRAM_H:
+        index = (offset - CMDQV_VCMDQ_PAGE1_BASE) / CMDQV_VCMDQ_STRIDE;
+        tegra241_cmdqv_write_vcmdq_page1_64(cmdqv,
+                offset - index * CMDQV_VCMDQ_STRIDE, index, value, true);
+        break;
+    default:
+        qemu_log_mask(LOG_UNIMP,
                       "%s unhandled 64-bit write at 0x%" PRIx64 " (WI)\n",
                       __func__, offset);
+    }
 }
 
 static void tegra241_cmdqv_write_mmio(void *opaque, hwaddr offset,
hw/arm/trace-events +2 -0
index 4392135fa2..d76f408d8a 100644
@@ -77,6 +77,8 @@ tegra241_cmdqv_read_mmio(uint64_t offset, uint64_t val, unsigned size) "offset:
 tegra241_cmdqv_write_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x"
 tegra241_cmdqv_read_vcmdq_page0(int index, const char *aperture, uint64_t offset0, uint64_t val) "vcmdq[%d] %s page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64
 tegra241_cmdqv_read_vcmdq_page1(int index, const char *aperture, uint64_t offset0, uint64_t val) "vcmdq[%d] %s page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64
+tegra241_cmdqv_write_vcmdq_page0(int index, const char *aperture, uint64_t offset0, uint64_t val) "vcmdq[%d] %s page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64
+tegra241_cmdqv_write_vcmdq_page1(int index, const char *aperture, uint64_t offset0, uint64_t val) "vcmdq[%d] %s page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64
 
 # strongarm.c
 strongarm_uart_update_parameters(const char *label, int speed, char parity, int data_bits, int stop_bits) "%s speed=%d parity=%c data=%d stop=%d"
#19 lore raw

PATCH v6 17/31 hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV

In reply to Shameer Kolothum

From: Nicolin Chen <[email protected]>

The kernel currently exposes a single VINTF per emulated SMMUv3 instance. IOMMU_VIOMMU_ALLOC returns an mmap offset for the host VINTF Page0 allocated for this SMMU. However, VCMDQs only become bound to that VINTF after IOMMU_HW_QUEUE_ALLOC, so until then the mapped Page0 does not back any real VCMDQ state.

When VINTF is enabled, mmap the kernel-provided Page0 region and set STATUS.ENABLE_OK only if the mmap succeeds; unmap it when VINTF is disabled. This prepares the VINTF mapping in advance of subsequent patches that add VCMDQ allocation and the guest MMIO install.

---
 hw/arm/tegra241-cmdqv.h |  3 +++
 hw/arm/tegra241-cmdqv.c | 50 ++++++++++++++++++++++++++++++++++++++---
 2 files changed, 50 insertions(+), 3 deletions(-)
hw/arm/tegra241-cmdqv.h +3 -0
index ad7fb8725f..c4d327a9a5 100644
@@ -37,6 +37,8 @@
 #define CMDQV_VINTF_PAGE1_BASE  0x40000
 #define CMDQV_VCMDQ_STRIDE      0x80
 
+#define VINTF_PAGE_SIZE 0x10000
+
 struct iommu_viommu_tegra241_cmdqv;
 
 typedef struct Tegra241CMDQV {
@@ -45,6 +47,7 @@ typedef struct Tegra241CMDQV {
     MemoryRegion mmio_cmdqv;
     qemu_irq irq;
     IOMMUFDVeventq *veventq;
+    void *vintf_page0;
 
     /* CMDQ-V Config page register cache */
     uint32_t config;
hw/arm/tegra241-cmdqv.c +47 -3

88 lines not on this page · raw message

Signature · 1 line
2.43.0
#20 lore raw

Re: [PATCH v6 17/31] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV

Hi Shameer,

Quoted 100 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

The kernel currently exposes a single VINTF per emulated SMMUv3 instance. IOMMU_VIOMMU_ALLOC returns an mmap offset for the host VINTF Page0 allocated for this SMMU. However, VCMDQs only become bound to that VINTF after IOMMU_HW_QUEUE_ALLOC, so until then the mapped Page0 does not back any real VCMDQ state.

When VINTF is enabled, mmap the kernel-provided Page0 region and set STATUS.ENABLE_OK only if the mmap succeeds; unmap it when VINTF is disabled. This prepares the VINTF mapping in advance of subsequent patches that add VCMDQ allocation and the guest MMIO install.

---
 hw/arm/tegra241-cmdqv.h |  3 +++
 hw/arm/tegra241-cmdqv.c | 50 ++++++++++++++++++++++++++++++++++++++---
 2 files changed, 50 insertions(+), 3 deletions(-)
hw/arm/tegra241-cmdqv.h +3 -0
index ad7fb8725f..c4d327a9a5 100644
@@ -37,6 +37,8 @@
 #define CMDQV_VINTF_PAGE1_BASE  0x40000
 #define CMDQV_VCMDQ_STRIDE      0x80
 
+#define VINTF_PAGE_SIZE 0x10000
+
 struct iommu_viommu_tegra241_cmdqv;
 
 typedef struct Tegra241CMDQV {
@@ -45,6 +47,7 @@ typedef struct Tegra241CMDQV {
     MemoryRegion mmio_cmdqv;
     qemu_irq irq;
     IOMMUFDVeventq *veventq;
+    void *vintf_page0;
 
     /* CMDQ-V Config page register cache */
     uint32_t config;
hw/arm/tegra241-cmdqv.c +37 -1
index 3cb1b45575..a52e153501 100644
@@ -9,6 +9,8 @@
 
 #include "qemu/osdep.h"
 #include "qemu/log.h"
+#include "qemu/error-report.h"
+#include "qapi/error.h"
 
 #include "hw/arm/smmuv3.h"
 #include "hw/arm/smmuv3-common.h"
@@ -218,8 +220,42 @@ static void tegra241_cmdqv_write_vcmdq_page1_64(Tegra241CMDQV *cmdqv,
                                            offset0, value);
 }
 
+static bool
+tegra241_cmdqv_munmap_vintf_page0(Tegra241CMDQV *cmdqv, Error **errp)
+{
+    if (!cmdqv->vintf_page0) {
+        return true;
+    }
+
+    if (munmap(cmdqv->vintf_page0, VINTF_PAGE_SIZE) < 0) {
+        error_setg_errno(errp, errno, "Failed to unmap VINTF page0");
+        return false;
+    }
+    cmdqv->vintf_page0 = NULL;
+    return true;
+}
+
+static bool tegra241_cmdqv_mmap_vintf_page0(Tegra241CMDQV *cmdqv, Error **errp)
+{
+    IOMMUFDViommu *viommu = cmdqv->s_accel->viommu;
+
+    if (cmdqv->vintf_page0) {
+        return true;
+    }
+
+    if (!iommufd_backend_viommu_mmap(viommu->iommufd, viommu->viommu_id,
+                                     VINTF_PAGE_SIZE,
+                                     cmdqv->cmdqv_data->out_vintf_mmap_offset,
+                                     &cmdqv->vintf_page0, errp)) {
+        return false;
+    }
+
+    return true;
+}
+
 static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv,
-                                              hwaddr offset, uint64_t value)
+                                              hwaddr offset, uint64_t value,
+                                              Error **errp)
 {
     int i;
 

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv, · line 277

Quoted 9 lines
hw/arm/tegra241-cmdqv.c +4 -1
@@ -234,8 +270,11 @@ static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv,
 
         cmdqv->vintf_config = value;
         if (value & R_VINTF0_CONFIG_ENABLE_MASK) {
-            cmdqv->vintf_status |= R_VINTF0_STATUS_ENABLE_OK_MASK;
+            if (tegra241_cmdqv_mmap_vintf_page0(cmdqv, errp)) {
+                cmdqv->vintf_status |= R_VINTF0_STATUS_ENABLE_OK_MASK;
+            }
         } else {
+            tegra241_cmdqv_munmap_vintf_page0(cmdqv, errp);

the munmap is only done upon guest action in this patch. Later on, it is also done on reset (in smmuv3_accel_reset called on smmu_reset_exit called in exit phase). So I think we are good overall but it is a bit annoying to leave it done only upon guest action in this patch. What do you think?

Thanks

Eric

Quoted 31 lines
hw/arm/tegra241-cmdqv.c +6 -1
             cmdqv->vintf_status &= ~R_VINTF0_STATUS_ENABLE_OK_MASK;
         }
         break;
@@ -341,6 +380,7 @@ out:
 static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,
                                        uint32_t value)
 {
+    Error *local_err = NULL;
     int index;
 
     switch (offset) {
@@ -359,7 +399,7 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,
         cmdqv->cmdq_alloc_map[(offset - A_CMDQ_ALLOC_MAP_0) / 4] = value;
         break;
     case A_VINTF0_CONFIG ... A_VINTF0_LVCMDQ_ERR_MAP_3:
-        tegra241_cmdqv_config_vintf_write(cmdqv, offset, value);
+        tegra241_cmdqv_config_vintf_write(cmdqv, offset, value, &local_err);
         break;
     case A_VI_VCMDQ0_CONS_INDX ... A_VI_VCMDQ1_GERRORN:
         /*
@@ -398,6 +438,10 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,
         qemu_log_mask(LOG_UNIMP, "%s unhandled write access at 0x%" PRIx64 "\n",
                       __func__, offset);
     }
+
+    if (local_err) {
+        error_report_err(local_err);
+    }
 }
 
 /*
#21 lore raw

RE: [PATCH v6 17/31] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV

Hi Eric,

Quoted 134 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 04 June 2026 10:01
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v6 17/31] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV

External email: Use caution opening links or attachments

Hi Shameer,

Quoted 13 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

The kernel currently exposes a single VINTF per emulated SMMUv3 instance. IOMMU_VIOMMU_ALLOC returns an mmap offset for the host VINTF Page0 allocated for this SMMU. However, VCMDQs only become bound to that VINTF after IOMMU_HW_QUEUE_ALLOC, so until then the mapped Page0 does not back any real VCMDQ state.

When VINTF is enabled, mmap the kernel-provided Page0 region and set STATUS.ENABLE_OK only if the mmap succeeds; unmap it when VINTF is disabled. This prepares the VINTF mapping in advance of subsequent patches that add VCMDQ allocation and the guest MMIO install.

Quoted 6 lines
---
 hw/arm/tegra241-cmdqv.h |  3 +++
 hw/arm/tegra241-cmdqv.c | 50

++++++++++++++++++++++++++++++++++++++---

hw/arm/tegra241-cmdqv.c static void

Quoted 37 lines
 2 files changed, 50 insertions(+), 3 deletions(-)
hw/arm/tegra241-cmdqv.h +3 -0
index ad7fb8725f..c4d327a9a5 100644
@@ -37,6 +37,8 @@
 #define CMDQV_VINTF_PAGE1_BASE  0x40000
 #define CMDQV_VCMDQ_STRIDE      0x80
 
+#define VINTF_PAGE_SIZE 0x10000
+
 struct iommu_viommu_tegra241_cmdqv;
 
 typedef struct Tegra241CMDQV {
@@ -45,6 +47,7 @@ typedef struct Tegra241CMDQV {
     MemoryRegion mmio_cmdqv;
     qemu_irq irq;
     IOMMUFDVeventq *veventq;
+    void *vintf_page0;
 
     /* CMDQ-V Config page register cache */
     uint32_t config;
hw/arm/tegra241-cmdqv.c +2 -0
index 3cb1b45575..a52e153501 100644
@@ -9,6 +9,8 @@
 
 #include "qemu/osdep.h"
 #include "qemu/log.h"
+#include "qemu/error-report.h"
+#include "qapi/error.h"
 
 #include "hw/arm/smmuv3.h"
 #include "hw/arm/smmuv3-common.h"
@@ -218,8 +220,42 @@ static void

tegra241_cmdqv_write_vcmdq_page1_64(Tegra241CMDQV *cmdqv,

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_write_vcmdq_page1_64(Tegra241CMDQV *cmdqv,

Quoted 5 lines
hw/arm/tegra241-cmdqv.c +2 -0
                                            offset0, value);
 }
 
+static bool
+tegra241_cmdqv_munmap_vintf_page0(Tegra241CMDQV *cmdqv, Error

**errp)

Quoted 4 lines
hw/arm/tegra241-cmdqv.c +4 -0
+{
+    if (!cmdqv->vintf_page0) {
+        return true;
+    }

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_write_vcmdq_page1_64(Tegra241CMDQV *cmdqv,

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10 -0
+
+    if (munmap(cmdqv->vintf_page0, VINTF_PAGE_SIZE) < 0) {
+        error_setg_errno(errp, errno, "Failed to unmap VINTF page0");
+        return false;
+    }
+    cmdqv->vintf_page0 = NULL;
+    return true;
+}
+
+static bool tegra241_cmdqv_mmap_vintf_page0(Tegra241CMDQV

*cmdqv, Error **errp)

hw/arm/tegra241-cmdqv.c static void

Quoted 26 lines
hw/arm/tegra241-cmdqv.c +19 -1
+{
+    IOMMUFDViommu *viommu = cmdqv->s_accel->viommu;
+
+    if (cmdqv->vintf_page0) {
+        return true;
+    }
+
+    if (!iommufd_backend_viommu_mmap(viommu->iommufd, viommu-
 viommu_id,
+                                     VINTF_PAGE_SIZE,
+                                     cmdqv->cmdqv_data->out_vintf_mmap_offset,
+                                     &cmdqv->vintf_page0, errp)) {
+        return false;
+    }
+
+    return true;
+}
+
 static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv,
-                                              hwaddr offset, uint64_t value)
+                                              hwaddr offset, uint64_t value,
+                                              Error **errp)
 {
     int i;
@@ -234,8 +270,11 @@ static void

tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv,

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv, · line 277

Quoted 9 lines
hw/arm/tegra241-cmdqv.c +4 -1
         cmdqv->vintf_config = value;
         if (value & R_VINTF0_CONFIG_ENABLE_MASK) {
-            cmdqv->vintf_status |= R_VINTF0_STATUS_ENABLE_OK_MASK;
+            if (tegra241_cmdqv_mmap_vintf_page0(cmdqv, errp)) {
+                cmdqv->vintf_status |= R_VINTF0_STATUS_ENABLE_OK_MASK;
+            }
         } else {
+            tegra241_cmdqv_munmap_vintf_page0(cmdqv, errp);
Quoted 5 lines

the munmap is only done upon guest action in this patch. Later on, it is also done on reset (in smmuv3_accel_reset called on smmu_reset_exit called in exit phase). So I think we are good overall but it is a bit annoying to leave it done only upon guest action in this patch. What do you think?

Though I didn’t really get the concern here, there is a suggestion to move the mmap just after IOMMU_VIOMMU_ALLOC as it doesn't need to be tied to VINTF enable.

https://lore.kernel.org/qemu-devel/[email protected]/T/#m972247a4652b09e3d86def921d9a8551b7620341

If we do that it will keep mapped until VIOMMU is alive. Does that address the concern above?

Thanks,
Shameer

#22 lore raw

Re: [PATCH v6 17/31] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV

Quoted 140 lines · Shameer Kolothum Thodi

Hi Eric,

Quoted 133 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 04 June 2026 10:01
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v6 17/31] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV

External email: Use caution opening links or attachments

Hi Shameer,

Quoted 13 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

The kernel currently exposes a single VINTF per emulated SMMUv3 instance. IOMMU_VIOMMU_ALLOC returns an mmap offset for the host VINTF Page0 allocated for this SMMU. However, VCMDQs only become bound to that VINTF after IOMMU_HW_QUEUE_ALLOC, so until then the mapped Page0 does not back any real VCMDQ state.

When VINTF is enabled, mmap the kernel-provided Page0 region and set STATUS.ENABLE_OK only if the mmap succeeds; unmap it when VINTF is disabled. This prepares the VINTF mapping in advance of subsequent patches that add VCMDQ allocation and the guest MMIO install.

Quoted 6 lines
---
 hw/arm/tegra241-cmdqv.h |  3 +++
 hw/arm/tegra241-cmdqv.c | 50

++++++++++++++++++++++++++++++++++++++---

hw/arm/tegra241-cmdqv.c static void

Quoted 37 lines
 2 files changed, 50 insertions(+), 3 deletions(-)
hw/arm/tegra241-cmdqv.h +3 -0
index ad7fb8725f..c4d327a9a5 100644
@@ -37,6 +37,8 @@
 #define CMDQV_VINTF_PAGE1_BASE  0x40000
 #define CMDQV_VCMDQ_STRIDE      0x80
 
+#define VINTF_PAGE_SIZE 0x10000
+
 struct iommu_viommu_tegra241_cmdqv;
 
 typedef struct Tegra241CMDQV {
@@ -45,6 +47,7 @@ typedef struct Tegra241CMDQV {
     MemoryRegion mmio_cmdqv;
     qemu_irq irq;
     IOMMUFDVeventq *veventq;
+    void *vintf_page0;
 
     /* CMDQ-V Config page register cache */
     uint32_t config;
hw/arm/tegra241-cmdqv.c +2 -0
index 3cb1b45575..a52e153501 100644
@@ -9,6 +9,8 @@
 
 #include "qemu/osdep.h"
 #include "qemu/log.h"
+#include "qemu/error-report.h"
+#include "qapi/error.h"
 
 #include "hw/arm/smmuv3.h"
 #include "hw/arm/smmuv3-common.h"
@@ -218,8 +220,42 @@ static void

tegra241_cmdqv_write_vcmdq_page1_64(Tegra241CMDQV *cmdqv,

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_write_vcmdq_page1_64(Tegra241CMDQV *cmdqv,

Quoted 5 lines
hw/arm/tegra241-cmdqv.c +2 -0
                                            offset0, value);
 }
 
+static bool
+tegra241_cmdqv_munmap_vintf_page0(Tegra241CMDQV *cmdqv, Error

**errp)

Quoted 4 lines
hw/arm/tegra241-cmdqv.c +4 -0
+{
+    if (!cmdqv->vintf_page0) {
+        return true;
+    }

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_write_vcmdq_page1_64(Tegra241CMDQV *cmdqv,

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10 -0
+
+    if (munmap(cmdqv->vintf_page0, VINTF_PAGE_SIZE) < 0) {
+        error_setg_errno(errp, errno, "Failed to unmap VINTF page0");
+        return false;
+    }
+    cmdqv->vintf_page0 = NULL;
+    return true;
+}
+
+static bool tegra241_cmdqv_mmap_vintf_page0(Tegra241CMDQV

*cmdqv, Error **errp)

hw/arm/tegra241-cmdqv.c static void

Quoted 26 lines
hw/arm/tegra241-cmdqv.c +19 -1
+{
+    IOMMUFDViommu *viommu = cmdqv->s_accel->viommu;
+
+    if (cmdqv->vintf_page0) {
+        return true;
+    }
+
+    if (!iommufd_backend_viommu_mmap(viommu->iommufd, viommu-
 viommu_id,
+                                     VINTF_PAGE_SIZE,
+                                     cmdqv->cmdqv_data->out_vintf_mmap_offset,
+                                     &cmdqv->vintf_page0, errp)) {
+        return false;
+    }
+
+    return true;
+}
+
 static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv,
-                                              hwaddr offset, uint64_t value)
+                                              hwaddr offset, uint64_t value,
+                                              Error **errp)
 {
     int i;
@@ -234,8 +270,11 @@ static void

tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv,

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv, · line 277

Quoted 8 lines
hw/arm/tegra241-cmdqv.c +4 -1
         cmdqv->vintf_config = value;
         if (value & R_VINTF0_CONFIG_ENABLE_MASK) {
-            cmdqv->vintf_status |= R_VINTF0_STATUS_ENABLE_OK_MASK;
+            if (tegra241_cmdqv_mmap_vintf_page0(cmdqv, errp)) {
+                cmdqv->vintf_status |= R_VINTF0_STATUS_ENABLE_OK_MASK;
+            }
         } else {
+            tegra241_cmdqv_munmap_vintf_page0(cmdqv, errp);
Quoted 5 lines

the munmap is only done upon guest action in this patch. Later on, it is also done on reset (in smmuv3_accel_reset called on smmu_reset_exit called in exit phase). So I think we are good overall but it is a bit annoying to leave it done only upon guest action in this patch. What do you think?

Quoted 8 lines

Though I didn’t really get the concern here, there is a suggestion to move the mmap just after IOMMU_VIOMMU_ALLOC as it doesn't need to be tied to VINTF enable.

https://lore.kernel.org/qemu-devel/[email protected]/T/#m972247a4652b09e3d86def921d9a8551b7620341

If we do that it will keep mapped until VIOMMU is alive. Does that address the concern above?

my concern is the munmap shall be eventually done on exit even if the guest does not issue the associated command, no? This is done later in the patch that introduces the reset. My concern is about patch sequencing. I think this is unrelated to Nicolin's suggestion.

Thanks

Eric

Quoted 7 lines

Thanks,
Shameer

#23 lore raw

PATCH v6 18/31 hw/arm/tegra241-cmdqv: Allocate HW VCMDQs once configured

In reply to Shameer Kolothum

From: Nicolin Chen <[email protected]>

Add support for allocating IOMMUFD hardware queues when the guest programs the VCMDQ BASE registers.

VCMDQ_EN lives in VCMDQ_CONFIG, which is on the VINTF Page0 region that a later patch installs into guest MMIO — so QEMU won't trap its writes. Allocate the hardware queue instead once all of these are
set: BASE programmed, CMDQ_ALLOC_MAP.ALLOC, and CMDQV / VINTF enabled. Each precondition write retries the allocation, so the guest may program them in any order.

iommufd_backend_alloc_hw_queue() needs the guest physical address of the VCMDQ ring buffer, so allocation is deferred until the guest has populated BASE.

If a hardware queue was previously allocated for the same VCMDQ, free it before reallocation. All allocated VCMDQs are freed when CMDQV or VINTF is disabled, or when the ALLOC bit is cleared.

On allocation failure, set CMDQ_INIT_ERR and clear CMDQ_EN_OK in the cache so trapped guest reads see the failure rather than a queue that looks live. Clear them on a later successful allocation.

---
 hw/arm/tegra241-cmdqv.h |  11 +++
 hw/arm/tegra241-cmdqv.c | 154 +++++++++++++++++++++++++++++++++++++---
 2 files changed, 156 insertions(+), 9 deletions(-)
hw/arm/tegra241-cmdqv.h +11 -0
index c4d327a9a5..84499b840d 100644
@@ -47,6 +47,7 @@ typedef struct Tegra241CMDQV {
     MemoryRegion mmio_cmdqv;
     qemu_irq irq;
     IOMMUFDVeventq *veventq;
+    IOMMUFDHWqueue *vcmdq[TEGRA241_CMDQV_MAX_CMDQ];
     void *vintf_page0;
 
     /* CMDQ-V Config page register cache */
@@ -364,6 +365,16 @@ SMMU_CMDQV_VI_VCMDQi_BASE_H_(1)
 SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_L_(1)
 SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_(1)
 
+static inline bool tegra241_cmdqv_enabled(Tegra241CMDQV *cmdqv)
+{
+    return cmdqv->status & R_STATUS_CMDQV_ENABLED_MASK;
+}
+
+static inline bool tegra241_vintf_enabled(Tegra241CMDQV *cmdqv)
+{
+    return cmdqv->vintf_status & R_VINTF0_STATUS_ENABLE_OK_MASK;
+}
+
 const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops(void);
 
 #endif /* HW_ARM_TEGRA241_CMDQV_H */
hw/arm/tegra241-cmdqv.c +145 -9

245 lines not on this page · raw message

Signature · 1 line
2.43.0
#24 lore raw

Re: [PATCH v6 18/31] hw/arm/tegra241-cmdqv: Allocate HW VCMDQs once configured

Hi Shameer,

Quoted 78 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

Add support for allocating IOMMUFD hardware queues when the guest programs the VCMDQ BASE registers.

VCMDQ_EN lives in VCMDQ_CONFIG, which is on the VINTF Page0 region that a later patch installs into guest MMIO — so QEMU won't trap its writes. Allocate the hardware queue instead once all of these are
set: BASE programmed, CMDQ_ALLOC_MAP.ALLOC, and CMDQV / VINTF enabled. Each precondition write retries the allocation, so the guest may program them in any order.

iommufd_backend_alloc_hw_queue() needs the guest physical address of the VCMDQ ring buffer, so allocation is deferred until the guest has populated BASE.

If a hardware queue was previously allocated for the same VCMDQ, free it before reallocation. All allocated VCMDQs are freed when CMDQV or VINTF is disabled, or when the ALLOC bit is cleared.

On allocation failure, set CMDQ_INIT_ERR and clear CMDQ_EN_OK in the cache so trapped guest reads see the failure rather than a queue that looks live. Clear them on a later successful allocation.

---
 hw/arm/tegra241-cmdqv.h |  11 +++
 hw/arm/tegra241-cmdqv.c | 154 +++++++++++++++++++++++++++++++++++++---
 2 files changed, 156 insertions(+), 9 deletions(-)
hw/arm/tegra241-cmdqv.h +11 -0
index c4d327a9a5..84499b840d 100644
@@ -47,6 +47,7 @@ typedef struct Tegra241CMDQV {
     MemoryRegion mmio_cmdqv;
     qemu_irq irq;
     IOMMUFDVeventq *veventq;
+    IOMMUFDHWqueue *vcmdq[TEGRA241_CMDQV_MAX_CMDQ];
     void *vintf_page0;
 
     /* CMDQ-V Config page register cache */
@@ -364,6 +365,16 @@ SMMU_CMDQV_VI_VCMDQi_BASE_H_(1)
 SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_L_(1)
 SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_(1)
 
+static inline bool tegra241_cmdqv_enabled(Tegra241CMDQV *cmdqv)
+{
+    return cmdqv->status & R_STATUS_CMDQV_ENABLED_MASK;
+}
+
+static inline bool tegra241_vintf_enabled(Tegra241CMDQV *cmdqv)
+{
+    return cmdqv->vintf_status & R_VINTF0_STATUS_ENABLE_OK_MASK;
+}
+
 const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops(void);
 
 #endif /* HW_ARM_TEGRA241_CMDQV_H */
hw/arm/tegra241-cmdqv.c +9 -0
index a52e153501..54d95f4e4e 100644
@@ -18,6 +18,96 @@
 #include "tegra241-cmdqv.h"
 #include "trace.h"
 
+static void tegra241_cmdqv_free_vcmdq(Tegra241CMDQV *cmdqv, int index)
+{
+    IOMMUFDViommu *viommu = cmdqv->s_accel->viommu;
+    IOMMUFDHWqueue *vcmdq = cmdqv->vcmdq[index];
+
+    if (!vcmdq) {
+        return;
+    }
+    iommufd_backend_free_id(viommu->iommufd, vcmdq->hw_queue_id);

hw/arm/tegra241-cmdqv.c line 39

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10 -0
+    g_free(vcmdq);
+    cmdqv->vcmdq[index] = NULL;
+}
+
+static void tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv)
+{
+    /* uapi/linux/iommufd.h: hw_queue destroy must be in descending @index. */
+    for (int i = (TEGRA241_CMDQV_MAX_CMDQ - 1); i >= 0; i--) {
+        tegra241_cmdqv_free_vcmdq(cmdqv, i);
+    }

basically the same comment as previous patch. I would add in the exit phase reset at this stage too.

Quoted 28 lines
hw/arm/tegra241-cmdqv.c +28 -0
+}
+
+/*
+ * Allocate a host HW VCMDQ from the current cached BASE / size for @index.
+ *
+ * Setup is a no-op (returns true) when any of the following preconditions
+ * isn't met yet:
+ *  - BASE not prgrammed yet.
+ *  - VCMDQ is not mapped to a VINTF (CMDQ_ALLOC_MAP.ALLOC == 0)
+ *  - BASE / size don't resolve to a RAM region
+ *  - CMDQV global enable or VINTF enable is not yet asserted
+ */
+static bool tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int index,
+                                       Error **errp)
+{
+    SMMUv3AccelState *accel = cmdqv->s_accel;
+    uint64_t base_mask = (uint64_t)R_VCMDQ0_BASE_L_ADDR_MASK |
+                         (uint64_t)R_VCMDQ0_BASE_H_ADDR_MASK << 32;
+    uint64_t addr = cmdqv->vcmdq_base[index] & base_mask;
+    uint64_t log2 = cmdqv->vcmdq_base[index] & R_VCMDQ0_BASE_L_LOG2SIZE_MASK;
+    uint64_t size = 1ULL << (log2 + 4);
+    IOMMUFDViommu *viommu = accel->viommu;
+    IOMMUFDHWqueue *hw_queue;
+    uint32_t hw_queue_id;
+
+    /* BASE not programmed yet. */
+    if (!cmdqv->vcmdq_base[index]) {
+        return true;

hw/arm/tegra241-cmdqv.c line 77

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10 -0
+    }
+
+    /* VCMDQ not yet mapped to a VINTF. */
+    if (!(cmdqv->cmdq_alloc_map[index] & R_CMDQ_ALLOC_MAP_0_ALLOC_MASK)) {
+        return true;
+    }
+
+    if (!tegra241_cmdqv_enabled(cmdqv) || !tegra241_vintf_enabled(cmdqv)) {
+        return true;
+    }

nit: I think I would introduce an helper vcmdq_hw_queue_ready_to_be_allocated(cmdqv, index) where you could put a proper doc comment.

Not that once flag the base is set it returns true. ie. you do not check that both L & H are set.

But anyway the queue will be reallocated

Quoted 21 lines
hw/arm/tegra241-cmdqv.c +21 -0
+
+    tegra241_cmdqv_free_vcmdq(cmdqv, index);
+
+    if (!iommufd_backend_alloc_hw_queue(viommu->iommufd, viommu->viommu_id,
+                                        IOMMU_HW_QUEUE_TYPE_TEGRA241_CMDQV,
+                                        index, addr, size, &hw_queue_id,
+                                        errp)) {
+        /* Record the failure in the cache. */
+        cmdqv->vcmdq_gerror[index] |= R_VCMDQ0_GERROR_CMDQ_INIT_ERR_MASK;
+        cmdqv->vcmdq_status[index] &= ~R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;
+        return false;
+    }
+    hw_queue = g_new(IOMMUFDHWqueue, 1);
+    hw_queue->hw_queue_id = hw_queue_id;
+    hw_queue->viommu = viommu;
+    cmdqv->vcmdq[index] = hw_queue;
+
+    cmdqv->vcmdq_gerror[index] &= ~R_VCMDQ0_GERROR_CMDQ_INIT_ERR_MASK;
+    cmdqv->vcmdq_status[index] |= R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;
+
+    return true;

hw/arm/tegra241-cmdqv.c line 108

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10 -0
+}
+
+static void tegra241_cmdqv_setup_all_vcmdq(Tegra241CMDQV *cmdqv,
+                                           Error **errp)
+{
+    for (int i = 0; i < TEGRA241_CMDQV_MAX_CMDQ; i++) {
+        if (!tegra241_cmdqv_setup_vcmdq(cmdqv, i, errp)) {
+            return;
+        }
+    }

nit: you you could after the tegra241_cmdqv_free_all_vcmdq helper

Quoted 6 lines
hw/arm/tegra241-cmdqv.c +2 -0
+}
+
 /*
  * Read a VCMDQ Page 0 register (control/status) using VCMDQ0_* offsets.
  *

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv, · line 243

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +6 -1
@@ -145,7 +235,12 @@ static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv,
         break;
     case A_VCMDQ0_CONFIG:
         if (value & R_VCMDQ0_CONFIG_CMDQ_EN_MASK) {
-            cmdqv->vcmdq_status[index] |= R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;
+            /* Report init error if any. */
+            if (!(cmdqv->vcmdq_gerror[index] &
+                  R_VCMDQ0_GERROR_CMDQ_INIT_ERR_MASK)) {
+                cmdqv->vcmdq_status[index] |=
+                    R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;
+            }

does it really belong to this patch?

Quoted 136 lines
hw/arm/tegra241-cmdqv.c +45 -8
         } else {
             cmdqv->vcmdq_status[index] &= ~R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;
         }
@@ -169,16 +264,19 @@ static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv,
  */
 static void tegra241_cmdqv_write_vcmdq_page1(Tegra241CMDQV *cmdqv,
                                              hwaddr offset0, int index,
-                                             uint32_t value, bool direct)
+                                             uint32_t value, bool direct,
+                                             Error **errp)
 {
     switch (offset0) {
     case A_VCMDQ0_BASE_L:
         cmdqv->vcmdq_base[index] =
             deposit64(cmdqv->vcmdq_base[index], 0, 32, value);
+        tegra241_cmdqv_setup_vcmdq(cmdqv, index, errp);
         break;
     case A_VCMDQ0_BASE_H:
         cmdqv->vcmdq_base[index] =
             deposit64(cmdqv->vcmdq_base[index], 32, 32, value);
+        tegra241_cmdqv_setup_vcmdq(cmdqv, index, errp);
         break;
     case A_VCMDQ0_CONS_INDX_BASE_DRAM_L:
         cmdqv->vcmdq_cons_indx_base[index] =
@@ -202,11 +300,13 @@ static void tegra241_cmdqv_write_vcmdq_page1(Tegra241CMDQV *cmdqv,
  */
 static void tegra241_cmdqv_write_vcmdq_page1_64(Tegra241CMDQV *cmdqv,
                                                 hwaddr offset0, int index,
-                                                uint64_t value, bool direct)
+                                                uint64_t value, bool direct,
+                                                Error **errp)
 {
     switch (offset0) {
     case A_VCMDQ0_BASE_L:
         cmdqv->vcmdq_base[index] = value;
+        tegra241_cmdqv_setup_vcmdq(cmdqv, index, errp);
         break;
     case A_VCMDQ0_CONS_INDX_BASE_DRAM_L:
         cmdqv->vcmdq_cons_indx_base[index] = value;
@@ -272,8 +372,14 @@ static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv,
         if (value & R_VINTF0_CONFIG_ENABLE_MASK) {
             if (tegra241_cmdqv_mmap_vintf_page0(cmdqv, errp)) {
                 cmdqv->vintf_status |= R_VINTF0_STATUS_ENABLE_OK_MASK;
+                /*
+                 * VCMDQs whose BASE was programmed before VINTF was
+                 * enabled need their hw_queue allocated now.
+                 */
+                tegra241_cmdqv_setup_all_vcmdq(cmdqv, errp);
             }
         } else {
+            tegra241_cmdqv_free_all_vcmdq(cmdqv);
             tegra241_cmdqv_munmap_vintf_page0(cmdqv, errp);
             cmdqv->vintf_status &= ~R_VINTF0_STATUS_ENABLE_OK_MASK;
         }
@@ -388,16 +494,37 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,
         cmdqv->config = value;
         if (value & R_CONFIG_CMDQV_EN_MASK) {
             cmdqv->status |= R_STATUS_CMDQV_ENABLED_MASK;
+            /*
+             * VCMDQs whose BASE was programmed before CMDQV was enabled
+             * need their hw_queue allocated now.
+             */
+            tegra241_cmdqv_setup_all_vcmdq(cmdqv, &local_err);
         } else {
+            tegra241_cmdqv_free_all_vcmdq(cmdqv);
             cmdqv->status &= ~R_STATUS_CMDQV_ENABLED_MASK;
         }
         break;
     case A_VI_INT_MASK_0 ... A_VI_INT_MASK_1:
         cmdqv->vi_int_mask[(offset - A_VI_INT_MASK_0) / 4] = value;
         break;
-    case A_CMDQ_ALLOC_MAP_0 ... A_CMDQ_ALLOC_MAP_1:
-        cmdqv->cmdq_alloc_map[(offset - A_CMDQ_ALLOC_MAP_0) / 4] = value;
+    case A_CMDQ_ALLOC_MAP_0 ... A_CMDQ_ALLOC_MAP_1: {
+        int idx = (offset - A_CMDQ_ALLOC_MAP_0) / 4;
+        bool was_alloc = cmdqv->cmdq_alloc_map[idx] &
+                         R_CMDQ_ALLOC_MAP_0_ALLOC_MASK;
+        bool now_alloc = value & R_CMDQ_ALLOC_MAP_0_ALLOC_MASK;
+
+        cmdqv->cmdq_alloc_map[idx] = value;
+        /*
+         * If the VCMDQ was already programmed (BASE) before mapping, fire
+         * setup on the ALLOC 0->1 transition; tear down on 1->0.
+         */
+        if (!was_alloc && now_alloc) {
+            tegra241_cmdqv_setup_vcmdq(cmdqv, idx, &local_err);
+        } else if (was_alloc && !now_alloc) {
+            tegra241_cmdqv_free_vcmdq(cmdqv, idx);
+        }
         break;
+    }
     case A_VINTF0_CONFIG ... A_VINTF0_LVCMDQ_ERR_MAP_3:
         tegra241_cmdqv_config_vintf_write(cmdqv, offset, value, &local_err);
         break;
@@ -427,12 +554,14 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,
         offset -= CMDQV_VINTF_PAGE1_BASE - CMDQV_VCMDQ_PAGE1_BASE;
         index = (offset - CMDQV_VCMDQ_PAGE1_BASE) / CMDQV_VCMDQ_STRIDE;
         tegra241_cmdqv_write_vcmdq_page1(cmdqv,
-                offset - index * CMDQV_VCMDQ_STRIDE, index, value, false);
+                offset - index * CMDQV_VCMDQ_STRIDE, index, value, false,
+                &local_err);
         break;
     case A_VCMDQ0_BASE_L ... A_VCMDQ1_CONS_INDX_BASE_DRAM_H:
         index = (offset - CMDQV_VCMDQ_PAGE1_BASE) / CMDQV_VCMDQ_STRIDE;
         tegra241_cmdqv_write_vcmdq_page1(cmdqv,
-                offset - index * CMDQV_VCMDQ_STRIDE, index, value, true);
+                offset - index * CMDQV_VCMDQ_STRIDE, index, value, true,
+                &local_err);
         break;
     default:
         qemu_log_mask(LOG_UNIMP, "%s unhandled write access at 0x%" PRIx64 "\n",
@@ -451,6 +580,7 @@ static void tegra241_cmdqv_writel_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,
 static void tegra241_cmdqv_writell_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,
                                         uint64_t value)
 {
+    Error *local_err = NULL;
     int index;
 
     switch (offset) {
@@ -463,18 +593,24 @@ static void tegra241_cmdqv_writell_mmio(Tegra241CMDQV *cmdqv, hwaddr offset,
         offset -= CMDQV_VINTF_PAGE1_BASE - CMDQV_VCMDQ_PAGE1_BASE;
         index = (offset - CMDQV_VCMDQ_PAGE1_BASE) / CMDQV_VCMDQ_STRIDE;
         tegra241_cmdqv_write_vcmdq_page1_64(cmdqv,
-                offset - index * CMDQV_VCMDQ_STRIDE, index, value, false);
+                offset - index * CMDQV_VCMDQ_STRIDE, index, value, false,
+                &local_err);
         break;
     case A_VCMDQ0_BASE_L ... A_VCMDQ1_CONS_INDX_BASE_DRAM_H:
         index = (offset - CMDQV_VCMDQ_PAGE1_BASE) / CMDQV_VCMDQ_STRIDE;
         tegra241_cmdqv_write_vcmdq_page1_64(cmdqv,
-                offset - index * CMDQV_VCMDQ_STRIDE, index, value, true);
+                offset - index * CMDQV_VCMDQ_STRIDE, index, value, true,
+                &local_err);
         break;
     default:
         qemu_log_mask(LOG_UNIMP,

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_writell_mmio(Tegra241CMDQV *cmdqv, hwaddr offset, · line 616

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +4 -0
                       "%s unhandled 64-bit write at 0x%" PRIx64 " (WI)\n",
                       __func__, offset);
     }
+
+    if (local_err) {
+        error_report_err(local_err);
+    }
 }
 
 static void tegra241_cmdqv_write_mmio(void *opaque, hwaddr offset,

Otherwise looks good to me

Thanks

Eric

#25 lore raw

RE: [PATCH v6 18/31] hw/arm/tegra241-cmdqv: Allocate HW VCMDQs once configured

Quoted 109 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 04 June 2026 10:37
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v6 18/31] hw/arm/tegra241-cmdqv: Allocate HW VCMDQs once configured

External email: Use caution opening links or attachments

Hi Shameer,

Quoted 24 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

Add support for allocating IOMMUFD hardware queues when the guest programs the VCMDQ BASE registers.

VCMDQ_EN lives in VCMDQ_CONFIG, which is on the VINTF Page0 region that a later patch installs into guest MMIO - so QEMU won't trap its writes. Allocate the hardware queue instead once all of these are
set: BASE programmed, CMDQ_ALLOC_MAP.ALLOC, and CMDQV / VINTF enabled. Each precondition write retries the allocation, so the guest may program them in any order.

iommufd_backend_alloc_hw_queue() needs the guest physical address of the VCMDQ ring buffer, so allocation is deferred until the guest has populated BASE.

If a hardware queue was previously allocated for the same VCMDQ, free it before reallocation. All allocated VCMDQs are freed when CMDQV or VINTF is disabled, or when the ALLOC bit is cleared.

On allocation failure, set CMDQ_INIT_ERR and clear CMDQ_EN_OK in the cache so trapped guest reads see the failure rather than a queue that looks live. Clear them on a later successful allocation.

Quoted 6 lines
---
 hw/arm/tegra241-cmdqv.h |  11 +++
 hw/arm/tegra241-cmdqv.c | 154

+++++++++++++++++++++++++++++++++++++---

Quoted 36 lines
 2 files changed, 156 insertions(+), 9 deletions(-)
hw/arm/tegra241-cmdqv.h +11 -0
index c4d327a9a5..84499b840d 100644
@@ -47,6 +47,7 @@ typedef struct Tegra241CMDQV {
     MemoryRegion mmio_cmdqv;
     qemu_irq irq;
     IOMMUFDVeventq *veventq;
+    IOMMUFDHWqueue *vcmdq[TEGRA241_CMDQV_MAX_CMDQ];
     void *vintf_page0;
 
     /* CMDQ-V Config page register cache */
@@ -364,6 +365,16 @@ SMMU_CMDQV_VI_VCMDQi_BASE_H_(1)
 SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_L_(1)
 SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_(1)
 
+static inline bool tegra241_cmdqv_enabled(Tegra241CMDQV *cmdqv)
+{
+    return cmdqv->status & R_STATUS_CMDQV_ENABLED_MASK;
+}
+
+static inline bool tegra241_vintf_enabled(Tegra241CMDQV *cmdqv)
+{
+    return cmdqv->vintf_status & R_VINTF0_STATUS_ENABLE_OK_MASK;
+}
+
 const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops(void);
 
 #endif /* HW_ARM_TEGRA241_CMDQV_H */
hw/arm/tegra241-cmdqv.c +0 -0
index a52e153501..54d95f4e4e 100644

hw/arm/tegra241-cmdqv.c line 21

Quoted 4 lines
hw/arm/tegra241-cmdqv.c +1 -0
@@ -18,6 +18,96 @@
 #include "tegra241-cmdqv.h"
 #include "trace.h"
 
+static void tegra241_cmdqv_free_vcmdq(Tegra241CMDQV *cmdqv, int

index)

Quoted 5 lines
hw/arm/tegra241-cmdqv.c +5 -0
+{
+    IOMMUFDViommu *viommu = cmdqv->s_accel->viommu;
+    IOMMUFDHWqueue *vcmdq = cmdqv->vcmdq[index];
+
+    if (!vcmdq) {

hw/arm/tegra241-cmdqv.c

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10 -0
+        return;
+    }
+    iommufd_backend_free_id(viommu->iommufd, vcmdq->hw_queue_id);
+    g_free(vcmdq);
+    cmdqv->vcmdq[index] = NULL;
+}
+
+static void tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv)
+{
+    /* uapi/linux/iommufd.h: hw_queue destroy must be in descending

@index. */

Quoted 5 lines

hw/arm/tegra241-cmdqv.c line 39

Quoted 3 lines
hw/arm/tegra241-cmdqv.c +3 -0
+    for (int i = (TEGRA241_CMDQV_MAX_CMDQ - 1); i >= 0; i--) {
+        tegra241_cmdqv_free_vcmdq(cmdqv, i);
+    }

basically the same comment as previous patch. I would add in the exit phase reset at this stage too.

Yes. I get that now. Will populate the rest handler.

Quoted 40 lines

hw/arm/tegra241-cmdqv.c

Quoted 4 lines
hw/arm/tegra241-cmdqv.c +4 -0
+}
+
+/*
+ * Allocate a host HW VCMDQ from the current cached BASE / size for

@index.

hw/arm/tegra241-cmdqv.c

Quoted 9 lines
hw/arm/tegra241-cmdqv.c +9 -0
+ *
+ * Setup is a no-op (returns true) when any of the following preconditions
+ * isn't met yet:
+ *  - BASE not prgrammed yet.
+ *  - VCMDQ is not mapped to a VINTF (CMDQ_ALLOC_MAP.ALLOC == 0)
+ *  - BASE / size don't resolve to a RAM region
+ *  - CMDQV global enable or VINTF enable is not yet asserted
+ */
+static bool tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int

index,

hw/arm/tegra241-cmdqv.c

Quoted 7 lines
hw/arm/tegra241-cmdqv.c +7 -0
+                                       Error **errp)
+{
+    SMMUv3AccelState *accel = cmdqv->s_accel;
+    uint64_t base_mask = (uint64_t)R_VCMDQ0_BASE_L_ADDR_MASK |
+                         (uint64_t)R_VCMDQ0_BASE_H_ADDR_MASK << 32;
+    uint64_t addr = cmdqv->vcmdq_base[index] & base_mask;
+    uint64_t log2 = cmdqv->vcmdq_base[index] &

R_VCMDQ0_BASE_L_LOG2SIZE_MASK;

hw/arm/tegra241-cmdqv.c

Quoted 12 lines
hw/arm/tegra241-cmdqv.c +12 -0
+    uint64_t size = 1ULL << (log2 + 4);
+    IOMMUFDViommu *viommu = accel->viommu;
+    IOMMUFDHWqueue *hw_queue;
+    uint32_t hw_queue_id;
+
+    /* BASE not programmed yet. */
+    if (!cmdqv->vcmdq_base[index]) {
+        return true;
+    }
+
+    /* VCMDQ not yet mapped to a VINTF. */
+    if (!(cmdqv->cmdq_alloc_map[index] &

R_CMDQ_ALLOC_MAP_0_ALLOC_MASK)) {

hw/arm/tegra241-cmdqv.c

Quoted 4 lines
hw/arm/tegra241-cmdqv.c +4 -0
+        return true;
+    }
+
+    if (!tegra241_cmdqv_enabled(cmdqv) ||
Quoted 6 lines

!tegra241_vintf_enabled(cmdqv)) {

hw/arm/tegra241-cmdqv.c line 77

Quoted 2 lines
hw/arm/tegra241-cmdqv.c +2 -0
+        return true;
+    }

nit: I think I would introduce an helper vcmdq_hw_queue_ready_to_be_allocated(cmdqv, index) where you could put a proper doc comment.

Sure.

Quoted 5 lines

Not that once flag the base is set it returns true. ie. you do not check that both L & H are set.

But anyway the queue will be reallocated

That's true. I will capture it in comments.

Quoted 48 lines

hw/arm/tegra241-cmdqv.c

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +9 -0
+
+    tegra241_cmdqv_free_vcmdq(cmdqv, index);
+
+    if (!iommufd_backend_alloc_hw_queue(viommu->iommufd, viommu-
 viommu_id,
+                                        IOMMU_HW_QUEUE_TYPE_TEGRA241_CMDQV,
+                                        index, addr, size, &hw_queue_id,
+                                        errp)) {
+        /* Record the failure in the cache. */
+        cmdqv->vcmdq_gerror[index] |=

R_VCMDQ0_GERROR_CMDQ_INIT_ERR_MASK;

hw/arm/tegra241-cmdqv.c line 88

Quoted 1 line
hw/arm/tegra241-cmdqv.c +1 -0
+        cmdqv->vcmdq_status[index] &=

~R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;

hw/arm/tegra241-cmdqv.c

Quoted 8 lines
hw/arm/tegra241-cmdqv.c +8 -0
+        return false;
+    }
+    hw_queue = g_new(IOMMUFDHWqueue, 1);
+    hw_queue->hw_queue_id = hw_queue_id;
+    hw_queue->viommu = viommu;
+    cmdqv->vcmdq[index] = hw_queue;
+
+    cmdqv->vcmdq_gerror[index] &=

~R_VCMDQ0_GERROR_CMDQ_INIT_ERR_MASK;

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv, · line 241

Quoted 1 line
hw/arm/tegra241-cmdqv.c +1 -0
+    cmdqv->vcmdq_status[index] |=

R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;

hw/arm/tegra241-cmdqv.c line 108

Quoted 12 lines
hw/arm/tegra241-cmdqv.c +12 -0
+
+    return true;
+}
+
+static void tegra241_cmdqv_setup_all_vcmdq(Tegra241CMDQV *cmdqv,
+                                           Error **errp)
+{
+    for (int i = 0; i < TEGRA241_CMDQV_MAX_CMDQ; i++) {
+        if (!tegra241_cmdqv_setup_vcmdq(cmdqv, i, errp)) {
+            return;
+        }
+    }

nit: you you could after the tegra241_cmdqv_free_all_vcmdq helper

hw/arm/tegra241-cmdqv.c static void

Quoted 6 lines
hw/arm/tegra241-cmdqv.c +2 -0
+}
+
 /*
  * Read a VCMDQ Page 0 register (control/status) using VCMDQ0_* offsets.

*

hw/arm/tegra241-cmdqv.c +0 -0
@@ -145,7 +235,12 @@ static void

tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv,

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv,

Quoted 4 lines
hw/arm/tegra241-cmdqv.c +0 -1
         break;
     case A_VCMDQ0_CONFIG:
         if (value & R_VCMDQ0_CONFIG_CMDQ_EN_MASK) {
-            cmdqv->vcmdq_status[index] |=
Quoted 8 lines

R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv, · line 243

Quoted 6 lines
hw/arm/tegra241-cmdqv.c +6 -0
+            /* Report init error if any. */
+            if (!(cmdqv->vcmdq_gerror[index] &
+                  R_VCMDQ0_GERROR_CMDQ_INIT_ERR_MASK)) {
+                cmdqv->vcmdq_status[index] |=
+                    R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;
+            }

does it really belong to this patch?

I put it here because this is the patch that sets that error in the first place. Commit log kind of mentions it. I could explain it better in commit log if that's enough. The other option is splitting the error handling into a separate patch, but THB don't see a need for that.

Thanks,
Shameer

#26 lore raw

Re: [PATCH v6 18/31] hw/arm/tegra241-cmdqv: Allocate HW VCMDQs once configured

Quoted 224 lines · Shameer Kolothum Thodi
Quoted 108 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 04 June 2026 10:37
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v6 18/31] hw/arm/tegra241-cmdqv: Allocate HW VCMDQs once configured

External email: Use caution opening links or attachments

Hi Shameer,

Quoted 24 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

Add support for allocating IOMMUFD hardware queues when the guest programs the VCMDQ BASE registers.

VCMDQ_EN lives in VCMDQ_CONFIG, which is on the VINTF Page0 region that a later patch installs into guest MMIO - so QEMU won't trap its writes. Allocate the hardware queue instead once all of these are
set: BASE programmed, CMDQ_ALLOC_MAP.ALLOC, and CMDQV / VINTF enabled. Each precondition write retries the allocation, so the guest may program them in any order.

iommufd_backend_alloc_hw_queue() needs the guest physical address of the VCMDQ ring buffer, so allocation is deferred until the guest has populated BASE.

If a hardware queue was previously allocated for the same VCMDQ, free it before reallocation. All allocated VCMDQs are freed when CMDQV or VINTF is disabled, or when the ALLOC bit is cleared.

On allocation failure, set CMDQ_INIT_ERR and clear CMDQ_EN_OK in the cache so trapped guest reads see the failure rather than a queue that looks live. Clear them on a later successful allocation.

Quoted 6 lines
---
 hw/arm/tegra241-cmdqv.h |  11 +++
 hw/arm/tegra241-cmdqv.c | 154

+++++++++++++++++++++++++++++++++++++---

Quoted 36 lines
 2 files changed, 156 insertions(+), 9 deletions(-)
hw/arm/tegra241-cmdqv.h +11 -0
index c4d327a9a5..84499b840d 100644
@@ -47,6 +47,7 @@ typedef struct Tegra241CMDQV {
     MemoryRegion mmio_cmdqv;
     qemu_irq irq;
     IOMMUFDVeventq *veventq;
+    IOMMUFDHWqueue *vcmdq[TEGRA241_CMDQV_MAX_CMDQ];
     void *vintf_page0;
 
     /* CMDQ-V Config page register cache */
@@ -364,6 +365,16 @@ SMMU_CMDQV_VI_VCMDQi_BASE_H_(1)
 SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_L_(1)
 SMMU_CMDQV_VI_VCMDQi_CONS_INDX_BASE_DRAM_H_(1)
 
+static inline bool tegra241_cmdqv_enabled(Tegra241CMDQV *cmdqv)
+{
+    return cmdqv->status & R_STATUS_CMDQV_ENABLED_MASK;
+}
+
+static inline bool tegra241_vintf_enabled(Tegra241CMDQV *cmdqv)
+{
+    return cmdqv->vintf_status & R_VINTF0_STATUS_ENABLE_OK_MASK;
+}
+
 const SMMUv3AccelCmdqvOps *tegra241_cmdqv_get_ops(void);
 
 #endif /* HW_ARM_TEGRA241_CMDQV_H */
hw/arm/tegra241-cmdqv.c +0 -0
index a52e153501..54d95f4e4e 100644

hw/arm/tegra241-cmdqv.c line 21

Quoted 4 lines
hw/arm/tegra241-cmdqv.c +1 -0
@@ -18,6 +18,96 @@
 #include "tegra241-cmdqv.h"
 #include "trace.h"
 
+static void tegra241_cmdqv_free_vcmdq(Tegra241CMDQV *cmdqv, int

index)

Quoted 5 lines
hw/arm/tegra241-cmdqv.c +5 -0
+{
+    IOMMUFDViommu *viommu = cmdqv->s_accel->viommu;
+    IOMMUFDHWqueue *vcmdq = cmdqv->vcmdq[index];
+
+    if (!vcmdq) {

hw/arm/tegra241-cmdqv.c

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10 -0
+        return;
+    }
+    iommufd_backend_free_id(viommu->iommufd, vcmdq->hw_queue_id);
+    g_free(vcmdq);
+    cmdqv->vcmdq[index] = NULL;
+}
+
+static void tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv)
+{
+    /* uapi/linux/iommufd.h: hw_queue destroy must be in descending

@index. */

Quoted 5 lines

hw/arm/tegra241-cmdqv.c line 39

Quoted 3 lines
hw/arm/tegra241-cmdqv.c +3 -0
+    for (int i = (TEGRA241_CMDQV_MAX_CMDQ - 1); i >= 0; i--) {
+        tegra241_cmdqv_free_vcmdq(cmdqv, i);
+    }

basically the same comment as previous patch. I would add in the exit phase reset at this stage too.

Yes. I get that now. Will populate the rest handler.

Quoted 40 lines

hw/arm/tegra241-cmdqv.c

Quoted 4 lines
hw/arm/tegra241-cmdqv.c +4 -0
+}
+
+/*
+ * Allocate a host HW VCMDQ from the current cached BASE / size for

@index.

hw/arm/tegra241-cmdqv.c

Quoted 9 lines
hw/arm/tegra241-cmdqv.c +9 -0
+ *
+ * Setup is a no-op (returns true) when any of the following preconditions
+ * isn't met yet:
+ *  - BASE not prgrammed yet.
+ *  - VCMDQ is not mapped to a VINTF (CMDQ_ALLOC_MAP.ALLOC == 0)
+ *  - BASE / size don't resolve to a RAM region
+ *  - CMDQV global enable or VINTF enable is not yet asserted
+ */
+static bool tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int

index,

hw/arm/tegra241-cmdqv.c

Quoted 7 lines
hw/arm/tegra241-cmdqv.c +7 -0
+                                       Error **errp)
+{
+    SMMUv3AccelState *accel = cmdqv->s_accel;
+    uint64_t base_mask = (uint64_t)R_VCMDQ0_BASE_L_ADDR_MASK |
+                         (uint64_t)R_VCMDQ0_BASE_H_ADDR_MASK << 32;
+    uint64_t addr = cmdqv->vcmdq_base[index] & base_mask;
+    uint64_t log2 = cmdqv->vcmdq_base[index] &

R_VCMDQ0_BASE_L_LOG2SIZE_MASK;

hw/arm/tegra241-cmdqv.c

Quoted 12 lines
hw/arm/tegra241-cmdqv.c +12 -0
+    uint64_t size = 1ULL << (log2 + 4);
+    IOMMUFDViommu *viommu = accel->viommu;
+    IOMMUFDHWqueue *hw_queue;
+    uint32_t hw_queue_id;
+
+    /* BASE not programmed yet. */
+    if (!cmdqv->vcmdq_base[index]) {
+        return true;
+    }
+
+    /* VCMDQ not yet mapped to a VINTF. */
+    if (!(cmdqv->cmdq_alloc_map[index] &

R_CMDQ_ALLOC_MAP_0_ALLOC_MASK)) {

hw/arm/tegra241-cmdqv.c

Quoted 4 lines
hw/arm/tegra241-cmdqv.c +4 -0
+        return true;
+    }
+
+    if (!tegra241_cmdqv_enabled(cmdqv) ||
Quoted 6 lines

!tegra241_vintf_enabled(cmdqv)) {

hw/arm/tegra241-cmdqv.c line 77

Quoted 2 lines
hw/arm/tegra241-cmdqv.c +2 -0
+        return true;
+    }

nit: I think I would introduce an helper vcmdq_hw_queue_ready_to_be_allocated(cmdqv, index) where you could put a proper doc comment.

Sure.

Quoted 4 lines

Not that once flag the base is set it returns true. ie. you do not check that both L & H are set.

But anyway the queue will be reallocated

That's true. I will capture it in comments.

Quoted 55 lines

hw/arm/tegra241-cmdqv.c

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +9 -0
+
+    tegra241_cmdqv_free_vcmdq(cmdqv, index);
+
+    if (!iommufd_backend_alloc_hw_queue(viommu->iommufd, viommu-
 viommu_id,
+                                        IOMMU_HW_QUEUE_TYPE_TEGRA241_CMDQV,
+                                        index, addr, size, &hw_queue_id,
+                                        errp)) {
+        /* Record the failure in the cache. */
+        cmdqv->vcmdq_gerror[index] |=

R_VCMDQ0_GERROR_CMDQ_INIT_ERR_MASK;

hw/arm/tegra241-cmdqv.c line 88

Quoted 1 line
hw/arm/tegra241-cmdqv.c +1 -0
+        cmdqv->vcmdq_status[index] &=

~R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;

hw/arm/tegra241-cmdqv.c

Quoted 8 lines
hw/arm/tegra241-cmdqv.c +8 -0
+        return false;
+    }
+    hw_queue = g_new(IOMMUFDHWqueue, 1);
+    hw_queue->hw_queue_id = hw_queue_id;
+    hw_queue->viommu = viommu;
+    cmdqv->vcmdq[index] = hw_queue;
+
+    cmdqv->vcmdq_gerror[index] &=

~R_VCMDQ0_GERROR_CMDQ_INIT_ERR_MASK;

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv, · line 241

Quoted 1 line
hw/arm/tegra241-cmdqv.c +1 -0
+    cmdqv->vcmdq_status[index] |=

R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;

hw/arm/tegra241-cmdqv.c line 108

Quoted 12 lines
hw/arm/tegra241-cmdqv.c +12 -0
+
+    return true;
+}
+
+static void tegra241_cmdqv_setup_all_vcmdq(Tegra241CMDQV *cmdqv,
+                                           Error **errp)
+{
+    for (int i = 0; i < TEGRA241_CMDQV_MAX_CMDQ; i++) {
+        if (!tegra241_cmdqv_setup_vcmdq(cmdqv, i, errp)) {
+            return;
+        }
+    }

nit: you you could after the tegra241_cmdqv_free_all_vcmdq helper

hw/arm/tegra241-cmdqv.c static void

Quoted 6 lines
hw/arm/tegra241-cmdqv.c +2 -0
+}
+
 /*
  * Read a VCMDQ Page 0 register (control/status) using VCMDQ0_* offsets.

*

hw/arm/tegra241-cmdqv.c +0 -0
@@ -145,7 +235,12 @@ static void

tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv,

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv,

Quoted 4 lines
hw/arm/tegra241-cmdqv.c +0 -1
         break;
     case A_VCMDQ0_CONFIG:
         if (value & R_VCMDQ0_CONFIG_CMDQ_EN_MASK) {
-            cmdqv->vcmdq_status[index] |=

R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;

Quoted 6 lines
hw/arm/tegra241-cmdqv.c +6 -0
+            /* Report init error if any. */
+            if (!(cmdqv->vcmdq_gerror[index] &
+                  R_VCMDQ0_GERROR_CMDQ_INIT_ERR_MASK)) {
+                cmdqv->vcmdq_status[index] |=
+                    R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;
+            }
Quoted 5 lines
Quoted 1 line

does it really belong to this patch?

I put it here because this is the patch that sets that error in the first place. Commit log kind of mentions it. I could explain it better in commit log if that's enough. The other option is splitting the error handling into a separate patch, but THB don't see a need for that.

OK Thanks

Eric

Quoted 4 lines

Thanks,
Shameer

#27 lore raw

PATCH v6 19/31 hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backing

In reply to Shameer Kolothum

hw/arm/tegra241-cmdqv: Use mmap'd VINTF Page 0 as VCMDQ backing

Introduce tegra241_cmdqv_vintf_ptr() to route VCMDQ Page 0 register accesses through the mmap'd VINTF Page 0 backing once a hardware queue has been allocated for the VCMDQ.

The two QEMU-trapped Page 0 apertures (direct at 0x10000, VINTF at 0x30000) are hardware aliases of the same underlying registers. A subsequent patch installs the VINTF aperture as a RAM-device into guest MMIO; in this patch both remain QEMU-trapped.

The direct VCMDQ aperture stays QEMU-trapped (rather than aliased to the VINTF mmap) so that writes to an unallocated VCMDQ remain well-defined. The CMDQV architecture allows software to program a VCMDQ through the direct aperture without first allocating it to a VINTF; aliasing to the VINTF mmap would route those writes into unallocated logical slots where the hardware silently drops them.

A VCMDQ Page 0 access is served from one of two sources:

- Cache-backed: no hw_queue is allocated for the VCMDQ (HW_QUEUE_ALLOC has not yet succeeded). Both apertures use QEMU's register cache.

- HW-backed: HW_QUEUE_ALLOC has succeeded. Both apertures access the registers directly through the mmap'd VINTF Page 0.

tegra241_cmdqv_sync_vcmdq() copies any cached writes (CONS_INDX, PROD_INDX, CONFIG, GERRORN) into the mmap'd page on the cache-to-HW transition so the guest's earlier register state survives.

---
 hw/arm/tegra241-cmdqv.c | 73 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 73 insertions(+)
hw/arm/tegra241-cmdqv.c +73 -0
index 54d95f4e4e..dce7089697 100644
@@ -39,6 +39,47 @@ static void tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv)
     }
 }
 
+/*
+ * Return a pointer into the mmap'd VINTF page0 for the VCMDQ Page 0
+ * register at @offset0 in VCMDQ slot @index, or NULL when the VCMDQ
+ * has no hw_queue allocated or the VINTF page0 is not mmap'd.
+ */
+static inline uint32_t *tegra241_cmdqv_vintf_ptr(Tegra241CMDQV *cmdqv,
+                                                 int index, hwaddr offset0)
+{
+    if (!cmdqv->vcmdq[index] || !cmdqv->vintf_page0) {
+        return NULL;
+    }
+    return (uint32_t *)(cmdqv->vintf_page0 +
+                        (index * CMDQV_VCMDQ_STRIDE) +
+                        (offset0 - CMDQV_VCMDQ_PAGE0_BASE));
+}
+
+/*
+ * Flush cached register writes into the mmap'd VINTF page0 after a
+ * successful HW_QUEUE_ALLOC, so the guest's earlier writes survive
+ * the cache-to-hardware transition.
+ */
+static void tegra241_cmdqv_sync_vcmdq(Tegra241CMDQV *cmdqv, int index)
+{
+    uint32_t *ptr;
+
+    ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index, A_VCMDQ0_CONS_INDX);
+    if (!ptr) {
+        return;
+    }
+    *ptr = cmdqv->vcmdq_cons_indx[index];
+
+    ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index, A_VCMDQ0_PROD_INDX);
+    *ptr = cmdqv->vcmdq_prod_indx[index];
+
+    ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index, A_VCMDQ0_CONFIG);
+    *ptr = cmdqv->vcmdq_config[index];
+
+    ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index, A_VCMDQ0_GERRORN);
+    *ptr = cmdqv->vcmdq_gerrorn[index];
+}
+
 /*
  * Allocate a host HW VCMDQ from the current cached BASE / size for @index.
  *
@@ -95,6 +136,9 @@ static bool tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int index,
     cmdqv->vcmdq_gerror[index] &= ~R_VCMDQ0_GERROR_CMDQ_INIT_ERR_MASK;
     cmdqv->vcmdq_status[index] |= R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;
 
+    /* Pre-alloc cached writes survive the cache-to-hardware transition. */
+    tegra241_cmdqv_sync_vcmdq(cmdqv, index);
+
     return true;
 }
 
@@ -113,13 +157,22 @@ static void tegra241_cmdqv_setup_all_vcmdq(Tegra241CMDQV *cmdqv,
  *
  * The caller normalizes the MMIO offset such that @offset0 always refers
  * to a VCMDQ0_* register, while @index selects the VCMDQ instance.
+ *
+ * If the VCMDQ is allocated and VINTF page0 is mmap'd, read directly
+ * from the VINTF page0 backing. Otherwise, fall back to the cache.
  */
 static uint64_t tegra241_cmdqv_read_vcmdq_page0(Tegra241CMDQV *cmdqv,
                                                 hwaddr offset0, int index,
                                                 bool direct)
 {
+    uint32_t *ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index, offset0);
     uint64_t val = 0;
 
+    if (ptr) {
+        val = *ptr;
+        goto out;
+    }
+
     switch (offset0) {
     case A_VCMDQ0_CONS_INDX:
         val = cmdqv->vcmdq_cons_indx[index];
@@ -144,6 +197,7 @@ static uint64_t tegra241_cmdqv_read_vcmdq_page0(Tegra241CMDQV *cmdqv,
                       "%s unhandled read access at 0x%" PRIx64 "\n",
                       __func__, offset0);
     }
+out:
     trace_tegra241_cmdqv_read_vcmdq_page0(index, direct ? "direct" : "vi",
                                           offset0, val);
     return val;
@@ -220,11 +274,29 @@ static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv,
  *
  * Page 0 registers are all 32-bit; this helper is only called for 4-byte
  * writes.
+ *
+ * If the VCMDQ is allocated and VINTF page0 is mmap'd, write directly
+ * to the VINTF page0 backing. Otherwise, update the cache.
  */
 static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv,
                                              hwaddr offset0, int index,
                                              uint32_t value, bool direct)
 {
+    uint32_t *ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index, offset0);
+
+    if (ptr) {
+        switch (offset0) {
+        case A_VCMDQ0_CONS_INDX:
+        case A_VCMDQ0_PROD_INDX:
+        case A_VCMDQ0_CONFIG:
+        case A_VCMDQ0_GERRORN:
+            *ptr = value;
+            goto out;
+        default:
+            break;
+        }
+    }
+
     switch (offset0) {
     case A_VCMDQ0_CONS_INDX:
         cmdqv->vcmdq_cons_indx[index] = value;
@@ -255,6 +327,7 @@ static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv,
                       "%s unhandled write access at 0x%" PRIx64 "\n",
                       __func__, offset0);
     }
+out:
     trace_tegra241_cmdqv_write_vcmdq_page0(index, direct ? "direct" : "vi",
                                            offset0, value);
 }
Signature · 1 line
2.43.0
Reviewed-by on v6 19/31
#28 lore raw

Re: [PATCH v6 19/31] hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backing

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv) · line 48

Quoted 7 lines · Shameer Kolothum
hw/arm/tegra241-cmdqv.c +7 -0
+/*
+ * Return a pointer into the mmap'd VINTF page0 for the VCMDQ Page 0
+ * register at @offset0 in VCMDQ slot @index, or NULL when the VCMDQ
+ * has no hw_queue allocated or the VINTF page0 is not mmap'd.
+ */
+static inline uint32_t *tegra241_cmdqv_vintf_ptr(Tegra241CMDQV *cmdqv,
+                                                 int index, hwaddr offset0)

I'd do:
tegra241_cmdqv_vintf_lvcmdq_ptr()
or just:
tegra241_vintf_lvcmdq_ptr()

Quoted 8 lines

static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv,
hwaddr offset0, int index, uint32_t value, bool direct)
{

hw/arm/tegra241-cmdqv.c +4 -0
+    uint32_t *ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index, offset0);
+
+    if (ptr) {
+        switch (offset0) {

hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv, · line 298

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10 -0
+        case A_VCMDQ0_CONS_INDX:
+        case A_VCMDQ0_PROD_INDX:
+        case A_VCMDQ0_CONFIG:
+        case A_VCMDQ0_GERRORN:
+            *ptr = value;
+            goto out;
+        default:
+            break;
+        }
+    }

Could stuff this into the helper? Return a valid ptr only when the offset0 is one of these four registers; otherwise, return NULL.

Then, here if would be:

    if (ptr) {
        *ptr = value;
        goto out;
    }

Doing so inside the helper could also sanitize the other callers.

Otherwise, looks good to me.

#29 lore raw

RE: [PATCH v6 19/31] hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backing

Quoted 23 lines

-----Original Message-----
From: Nicolin Chen <[email protected]>
Sent: 01 June 2026 23:06
To: Shameer Kolothum Thodi <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v6 19/31] hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backing

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv)

Quoted 6 lines · Shameer Kolothum
hw/arm/tegra241-cmdqv.c +6 -0
+/*
+ * Return a pointer into the mmap'd VINTF page0 for the VCMDQ Page 0
+ * register at @offset0 in VCMDQ slot @index, or NULL when the VCMDQ
+ * has no hw_queue allocated or the VINTF page0 is not mmap'd.
+ */
+static inline uint32_t *tegra241_cmdqv_vintf_ptr(Tegra241CMDQV

*cmdqv,

Quoted 7 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+                                                 int index, hwaddr
+offset0)

I'd do:
tegra241_cmdqv_vintf_lvcmdq_ptr()
or just:
tegra241_vintf_lvcmdq_ptr()

Ok. I will go with tegra241_cmdqv_vintf_lvcmdq_ptr()

Quoted 20 lines
Quoted 1 line

static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV

*cmdqv,

Quoted 7 lines

hwaddr offset0, int index, uint32_t value, bool
direct) {

hw/arm/tegra241-cmdqv.c +4 -0
+    uint32_t *ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index, offset0);
+
+    if (ptr) {
+        switch (offset0) {

hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv, · line 298

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10 -0
+        case A_VCMDQ0_CONS_INDX:
+        case A_VCMDQ0_PROD_INDX:
+        case A_VCMDQ0_CONFIG:
+        case A_VCMDQ0_GERRORN:
+            *ptr = value;
+            goto out;
+        default:
+            break;
+        }
+    }
Quoted 8 lines

Could stuff this into the helper? Return a valid ptr only when the offset0 is one of these four registers; otherwise, return NULL.

Then, here if would be:

    if (ptr) {
        *ptr = value;
        goto out;
    }

The helper is shared with _read_vcmdq_page0(), where STATUS/ GERROR reads also need to route to the mmap when allocated. Folding the above would require a separate handling for "write" inside the helper. Could do. But I think this is more readable...

Thanks,
Shameer

#30 lore raw

Re: [PATCH v6 19/31] hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backing

In reply to Shameer Kolothum

Hi Shameer,

Quoted 1 line · Shameer Kolothum

hw/arm/tegra241-cmdqv: Use mmap'd VINTF Page 0 as VCMDQ backing

the title is double here.
By the way may I suggest
hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 for mapped vcmdqs direct view

to emphasize this relates the direct view and only for mapped vcmdqs

Quoted 88 lines

Introduce tegra241_cmdqv_vintf_ptr() to route VCMDQ Page 0 register accesses through the mmap'd VINTF Page 0 backing once a hardware queue has been allocated for the VCMDQ.

The two QEMU-trapped Page 0 apertures (direct at 0x10000, VINTF at 0x30000) are hardware aliases of the same underlying registers. A subsequent patch installs the VINTF aperture as a RAM-device into guest MMIO; in this patch both remain QEMU-trapped.

The direct VCMDQ aperture stays QEMU-trapped (rather than aliased to the VINTF mmap) so that writes to an unallocated VCMDQ remain well-defined. The CMDQV architecture allows software to program a VCMDQ through the direct aperture without first allocating it to a VINTF; aliasing to the VINTF mmap would route those writes into unallocated logical slots where the hardware silently drops them.

A VCMDQ Page 0 access is served from one of two sources:

- Cache-backed: no hw_queue is allocated for the VCMDQ (HW_QUEUE_ALLOC has not yet succeeded). Both apertures use QEMU's register cache.

- HW-backed: HW_QUEUE_ALLOC has succeeded. Both apertures access the registers directly through the mmap'd VINTF Page 0.

tegra241_cmdqv_sync_vcmdq() copies any cached writes (CONS_INDX, PROD_INDX, CONFIG, GERRORN) into the mmap'd page on the cache-to-HW transition so the guest's earlier register state survives.

---
 hw/arm/tegra241-cmdqv.c | 73 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 73 insertions(+)
hw/arm/tegra241-cmdqv.c +41 -0
index 54d95f4e4e..dce7089697 100644
@@ -39,6 +39,47 @@ static void tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv)
     }
 }
 
+/*
+ * Return a pointer into the mmap'd VINTF page0 for the VCMDQ Page 0
+ * register at @offset0 in VCMDQ slot @index, or NULL when the VCMDQ
+ * has no hw_queue allocated or the VINTF page0 is not mmap'd.
+ */
+static inline uint32_t *tegra241_cmdqv_vintf_ptr(Tegra241CMDQV *cmdqv,
+                                                 int index, hwaddr offset0)
+{
+    if (!cmdqv->vcmdq[index] || !cmdqv->vintf_page0) {
+        return NULL;
+    }
+    return (uint32_t *)(cmdqv->vintf_page0 +
+                        (index * CMDQV_VCMDQ_STRIDE) +
+                        (offset0 - CMDQV_VCMDQ_PAGE0_BASE));
+}
+
+/*
+ * Flush cached register writes into the mmap'd VINTF page0 after a
+ * successful HW_QUEUE_ALLOC, so the guest's earlier writes survive
+ * the cache-to-hardware transition.
+ */
+static void tegra241_cmdqv_sync_vcmdq(Tegra241CMDQV *cmdqv, int index)
+{
+    uint32_t *ptr;
+
+    ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index, A_VCMDQ0_CONS_INDX);
+    if (!ptr) {
+        return;
+    }
+    *ptr = cmdqv->vcmdq_cons_indx[index];
+
+    ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index, A_VCMDQ0_PROD_INDX);
+    *ptr = cmdqv->vcmdq_prod_indx[index];
+
+    ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index, A_VCMDQ0_CONFIG);
+    *ptr = cmdqv->vcmdq_config[index];
+
+    ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index, A_VCMDQ0_GERRORN);
+    *ptr = cmdqv->vcmdq_gerrorn[index];
+}
+
 /*
  * Allocate a host HW VCMDQ from the current cached BASE / size for @index.
  *

hw/arm/tegra241-cmdqv.c static bool tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int index, · line 139

Quoted 4 lines
hw/arm/tegra241-cmdqv.c +1 -0
@@ -95,6 +136,9 @@ static bool tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int index,
     cmdqv->vcmdq_gerror[index] &= ~R_VCMDQ0_GERROR_CMDQ_INIT_ERR_MASK;
     cmdqv->vcmdq_status[index] |= R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;
 
+    /* Pre-alloc cached writes survive the cache-to-hardware transition. */

what about the opposite transition?

Quoted 11 lines
hw/arm/tegra241-cmdqv.c +4 -0
+    tegra241_cmdqv_sync_vcmdq(cmdqv, index);
+
     return true;
 }
 
@@ -113,13 +157,22 @@ static void tegra241_cmdqv_setup_all_vcmdq(Tegra241CMDQV *cmdqv,
  *
  * The caller normalizes the MMIO offset such that @offset0 always refers
  * to a VCMDQ0_* register, while @index selects the VCMDQ instance.
+ *
+ * If the VCMDQ is allocated and VINTF page0 is mmap'd, read directly

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_setup_all_vcmdq(Tegra241CMDQV *cmdqv, · line 171

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +3 -0
+ * from the VINTF page0 backing. Otherwise, fall back to the cache.
  */
 static uint64_t tegra241_cmdqv_read_vcmdq_page0(Tegra241CMDQV *cmdqv,
                                                 hwaddr offset0, int index,
                                                 bool direct)
 {
+    uint32_t *ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index, offset0);
     uint64_t val = 0;
 
+    if (ptr) {

shouldn't you test "direct" as well? I know that later on this is will be backed by another mr but I think this would be clearer.

Quoted 18 lines
hw/arm/tegra241-cmdqv.c +5 -0
+        val = *ptr;
+        goto out;
+    }
+
     switch (offset0) {
     case A_VCMDQ0_CONS_INDX:
         val = cmdqv->vcmdq_cons_indx[index];
@@ -144,6 +197,7 @@ static uint64_t tegra241_cmdqv_read_vcmdq_page0(Tegra241CMDQV *cmdqv,
                       "%s unhandled read access at 0x%" PRIx64 "\n",
                       __func__, offset0);
     }
+out:
     trace_tegra241_cmdqv_read_vcmdq_page0(index, direct ? "direct" : "vi",
                                           offset0, val);
     return val;
@@ -220,11 +274,29 @@ static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv,
  *
  * Page 0 registers are all 32-bit; this helper is only called for 4-byte

hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv, · line 285

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +4 -0
  * writes.
+ *
+ * If the VCMDQ is allocated and VINTF page0 is mmap'd, write directly
+ * to the VINTF page0 backing. Otherwise, update the cache.
  */
 static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv,
                                              hwaddr offset0, int index,
                                              uint32_t value, bool direct)
 {
+    uint32_t *ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index, offset0);

same here?

Quoted 18 lines
hw/arm/tegra241-cmdqv.c +14 -0
+
+    if (ptr) {
+        switch (offset0) {
+        case A_VCMDQ0_CONS_INDX:
+        case A_VCMDQ0_PROD_INDX:
+        case A_VCMDQ0_CONFIG:
+        case A_VCMDQ0_GERRORN:
+            *ptr = value;
+            goto out;
+        default:
+            break;
+        }
+    }
+
     switch (offset0) {
     case A_VCMDQ0_CONS_INDX:
         cmdqv->vcmdq_cons_indx[index] = value;

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv, · line 331

Quoted 5 lines
hw/arm/tegra241-cmdqv.c +1 -0
@@ -255,6 +327,7 @@ static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv,
                       "%s unhandled write access at 0x%" PRIx64 "\n",
                       __func__, offset0);
     }
+out:
     trace_tegra241_cmdqv_write_vcmdq_page0(index, direct ? "direct" : "vi",

by the way could be interesting to log in the trace event that mmapped was used instead of cache regs.

besides this looks good to me

Thanks

Eric

Quoted 2 lines
hw/arm/tegra241-cmdqv.c +0 -0
                                            offset0, value);
 }
#31 lore raw

RE: [PATCH v6 19/31] hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backing

Quoted 15 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 04 June 2026 10:56
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v6 19/31] hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backing

Quoted 8 lines

External email: Use caution opening links or attachments

Hi Shameer,

Quoted 1 line · Shameer Kolothum

hw/arm/tegra241-cmdqv: Use mmap'd VINTF Page 0 as VCMDQ backing

the title is double here.

Ah..Ok.

Quoted 5 lines

By the way may I suggest
hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 for mapped vcmdqs direct view

to emphasize this relates the direct view and only for mapped vcmdqs

It's true this eventually relates only to the direct aperture, but not yet. VINTF page0 only gets mapped into the guest in patch 21. Until then, for allocated vCMDQs we use the mmap'd VINTF page0 for both direct and VINTF aperture accesses.

May be we can rename to:
Route allocated VCMDQ Page0 accesses to the mmap'd VINTF page0

Quoted 96 lines
Quoted 26 lines

Introduce tegra241_cmdqv_vintf_ptr() to route VCMDQ Page 0 register accesses through the mmap'd VINTF Page 0 backing once a hardware queue has been allocated for the VCMDQ.

The two QEMU-trapped Page 0 apertures (direct at 0x10000, VINTF at 0x30000) are hardware aliases of the same underlying registers. A subsequent patch installs the VINTF aperture as a RAM-device into guest MMIO; in this patch both remain QEMU-trapped.

The direct VCMDQ aperture stays QEMU-trapped (rather than aliased to the VINTF mmap) so that writes to an unallocated VCMDQ remain well-defined. The CMDQV architecture allows software to program a VCMDQ through the direct aperture without first allocating it to a VINTF; aliasing to the VINTF mmap would route those writes into unallocated logical slots where the hardware silently drops them.

A VCMDQ Page 0 access is served from one of two sources:

- Cache-backed: no hw_queue is allocated for the VCMDQ (HW_QUEUE_ALLOC has not yet succeeded). Both apertures use QEMU's register cache.

- HW-backed: HW_QUEUE_ALLOC has succeeded. Both apertures access the registers directly through the mmap'd VINTF Page 0.

Quoted 7 lines

tegra241_cmdqv_sync_vcmdq() copies any cached writes (CONS_INDX, PROD_INDX, CONFIG, GERRORN) into the mmap'd page on the cache-to-HW transition so the guest's earlier register state survives.

---
 hw/arm/tegra241-cmdqv.c | 73

+++++++++++++++++++++++++++++++++++++++++

hw/arm/tegra241-cmdqv.c static void

Quoted 7 lines
 1 file changed, 73 insertions(+)
hw/arm/tegra241-cmdqv.c +0 -0
index 54d95f4e4e..dce7089697 100644
@@ -39,6 +39,47 @@ static void

tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv)

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv)

Quoted 9 lines
hw/arm/tegra241-cmdqv.c +6 -0
     }
 }
 
+/*
+ * Return a pointer into the mmap'd VINTF page0 for the VCMDQ Page 0
+ * register at @offset0 in VCMDQ slot @index, or NULL when the VCMDQ
+ * has no hw_queue allocated or the VINTF page0 is not mmap'd.
+ */
+static inline uint32_t *tegra241_cmdqv_vintf_ptr(Tegra241CMDQV

*cmdqv,

Quoted 6 lines
hw/arm/tegra241-cmdqv.c +6 -0
+                                                 int index, hwaddr offset0)
+{
+    if (!cmdqv->vcmdq[index] || !cmdqv->vintf_page0) {
+        return NULL;
+    }
+    return (uint32_t *)(cmdqv->vintf_page0 +

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv)

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10 -0
+                        (index * CMDQV_VCMDQ_STRIDE) +
+                        (offset0 - CMDQV_VCMDQ_PAGE0_BASE));
+}
+
+/*
+ * Flush cached register writes into the mmap'd VINTF page0 after a
+ * successful HW_QUEUE_ALLOC, so the guest's earlier writes survive
+ * the cache-to-hardware transition.
+ */
+static void tegra241_cmdqv_sync_vcmdq(Tegra241CMDQV *cmdqv, int

index)

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv)

Quoted 4 lines
hw/arm/tegra241-cmdqv.c +4 -0
+{
+    uint32_t *ptr;
+
+    ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index,

A_VCMDQ0_CONS_INDX);

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv)

Quoted 6 lines
hw/arm/tegra241-cmdqv.c +6 -0
+    if (!ptr) {
+        return;
+    }
+    *ptr = cmdqv->vcmdq_cons_indx[index];
+
+    ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index,

A_VCMDQ0_PROD_INDX);

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv)

Quoted 11 lines
hw/arm/tegra241-cmdqv.c +9 -0
+    *ptr = cmdqv->vcmdq_prod_indx[index];
+
+    ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index, A_VCMDQ0_CONFIG);
+    *ptr = cmdqv->vcmdq_config[index];
+
+    ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index, A_VCMDQ0_GERRORN);
+    *ptr = cmdqv->vcmdq_gerrorn[index];
+}
+
 /*
  * Allocate a host HW VCMDQ from the current cached BASE / size for

@index.

hw/arm/tegra241-cmdqv.c static bool

Quoted 2 lines
hw/arm/tegra241-cmdqv.c +0 -0
  *
@@ -95,6 +136,9 @@ static bool

tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int index,

Quoted 7 lines

hw/arm/tegra241-cmdqv.c static bool · line 136

Quoted 1 line
hw/arm/tegra241-cmdqv.c +0 -0
     cmdqv->vcmdq_gerror[index] &=

~R_VCMDQ0_GERROR_CMDQ_INIT_ERR_MASK;

hw/arm/tegra241-cmdqv.c static bool · line 137

Quoted 1 line
hw/arm/tegra241-cmdqv.c +0 -0
     cmdqv->vcmdq_status[index] |=

R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;

hw/arm/tegra241-cmdqv.c static bool tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int index, · line 139

Quoted 2 lines
hw/arm/tegra241-cmdqv.c +1 -0
+    /* Pre-alloc cached writes survive the cache-to-hardware transition. */

what about the opposite transition?

Yeah. We don’t do anything now. I think better to reset the cache on opposite transition.

Quoted 22 lines

hw/arm/tegra241-cmdqv.c static void

Quoted 6 lines
hw/arm/tegra241-cmdqv.c +2 -0
+    tegra241_cmdqv_sync_vcmdq(cmdqv, index);
+
     return true;
 }
 
@@ -113,13 +157,22 @@ static void

tegra241_cmdqv_setup_all_vcmdq(Tegra241CMDQV *cmdqv,

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_setup_all_vcmdq(Tegra241CMDQV *cmdqv,

Quoted 8 lines
hw/arm/tegra241-cmdqv.c +3 -0
  *
  * The caller normalizes the MMIO offset such that @offset0 always refers
  * to a VCMDQ0_* register, while @index selects the VCMDQ instance.
+ *
+ * If the VCMDQ is allocated and VINTF page0 is mmap'd, read directly
+ * from the VINTF page0 backing. Otherwise, fall back to the cache.
  */
 static uint64_t tegra241_cmdqv_read_vcmdq_page0(Tegra241CMDQV

*cmdqv,

Quoted 6 lines
hw/arm/tegra241-cmdqv.c +1 -0
                                                 hwaddr offset0, int index,
                                                 bool direct)
 {
+    uint32_t *ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index, offset0);
     uint64_t val = 0;
 
Quoted 3 lines

hw/arm/tegra241-cmdqv.c static void · line 171

Quoted 1 line
hw/arm/tegra241-cmdqv.c +1 -0
+    if (ptr) {

shouldn't you test "direct" as well? I know that later on this is will be backed by another mr but I think this would be clearer.

Is the suggestion here to just limit this to "direct"? We can't do that. Both apertures alias the same registers, and for an allocated and mmapped VINTF page0 we access the hw directly.

The vintf page0 gets mapped to the guest only in patch 21, so after that only the direct aperture is trapped here anyway.

Quoted 51 lines

hw/arm/tegra241-cmdqv.c static uint64_t

Quoted 8 lines
hw/arm/tegra241-cmdqv.c +4 -0
+        val = *ptr;
+        goto out;
+    }
+
     switch (offset0) {
     case A_VCMDQ0_CONS_INDX:
         val = cmdqv->vcmdq_cons_indx[index];
@@ -144,6 +197,7 @@ static uint64_t

tegra241_cmdqv_read_vcmdq_page0(Tegra241CMDQV *cmdqv,

hw/arm/tegra241-cmdqv.c static uint64_t

Quoted 8 lines
hw/arm/tegra241-cmdqv.c +1 -0
                       "%s unhandled read access at 0x%" PRIx64 "\n",
                       __func__, offset0);
     }
+out:
     trace_tegra241_cmdqv_read_vcmdq_page0(index, direct ? "direct" : "vi",
                                           offset0, val);
     return val;
@@ -220,11 +274,29 @@ static uint64_t

tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv,

hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv,

Quoted 8 lines
hw/arm/tegra241-cmdqv.c +3 -0
  *
  * Page 0 registers are all 32-bit; this helper is only called for 4-byte
  * writes.
+ *
+ * If the VCMDQ is allocated and VINTF page0 is mmap'd, write directly
+ * to the VINTF page0 backing. Otherwise, update the cache.
  */
 static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV

*cmdqv,

hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv, · line 285

Quoted 4 lines
hw/arm/tegra241-cmdqv.c +1 -0
                                              hwaddr offset0, int index,
                                              uint32_t value, bool direct)
 {
+    uint32_t *ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index, offset0);

same here?

hw/arm/tegra241-cmdqv.c static void

Quoted 18 lines
hw/arm/tegra241-cmdqv.c +14 -0
+
+    if (ptr) {
+        switch (offset0) {
+        case A_VCMDQ0_CONS_INDX:
+        case A_VCMDQ0_PROD_INDX:
+        case A_VCMDQ0_CONFIG:
+        case A_VCMDQ0_GERRORN:
+            *ptr = value;
+            goto out;
+        default:
+            break;
+        }
+    }
+
     switch (offset0) {
     case A_VCMDQ0_CONS_INDX:
         cmdqv->vcmdq_cons_indx[index] = value;
@@ -255,6 +327,7 @@ static void

tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv,

Quoted 7 lines

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv, · line 331

Quoted 5 lines
hw/arm/tegra241-cmdqv.c +1 -0
                       "%s unhandled write access at 0x%" PRIx64 "\n",
                       __func__, offset0);
     }
+out:
     trace_tegra241_cmdqv_write_vcmdq_page0(index, direct ? "direct" : "vi",

by the way could be interesting to log in the trace event that mmapped was used instead of cache regs.

Could do.

Thanks,
Shameer

#32 lore raw

Re: [PATCH v6 19/31] hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backing

Quoted 28 lines · Shameer Kolothum Thodi
Quoted 14 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 04 June 2026 10:56
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v6 19/31] hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 as VCMDQ backing

Quoted 8 lines

External email: Use caution opening links or attachments

Hi Shameer,

Quoted 1 line · Shameer Kolothum

hw/arm/tegra241-cmdqv: Use mmap'd VINTF Page 0 as VCMDQ backing

the title is double here.

Ah..Ok.

Quoted 4 lines

By the way may I suggest
hw/arm/tegra241-cmdqv: Use mmap'd VINTF page0 for mapped vcmdqs direct view

Quoted 8 lines
Quoted 1 line

to emphasize this relates the direct view and only for mapped vcmdqs

It's true this eventually relates only to the direct aperture, but not yet. VINTF page0 only gets mapped into the guest in patch 21. Until then, for allocated vCMDQs we use the mmap'd VINTF page0 for both direct and VINTF aperture accesses.

May be we can rename to:
Route allocated VCMDQ Page0 accesses to the mmap'd VINTF page0

Indeed. Yes above makes sense. It is about allocated VCMDQs only

Quoted 96 lines
Quoted 96 lines
Quoted 26 lines

Introduce tegra241_cmdqv_vintf_ptr() to route VCMDQ Page 0 register accesses through the mmap'd VINTF Page 0 backing once a hardware queue has been allocated for the VCMDQ.

The two QEMU-trapped Page 0 apertures (direct at 0x10000, VINTF at 0x30000) are hardware aliases of the same underlying registers. A subsequent patch installs the VINTF aperture as a RAM-device into guest MMIO; in this patch both remain QEMU-trapped.

The direct VCMDQ aperture stays QEMU-trapped (rather than aliased to the VINTF mmap) so that writes to an unallocated VCMDQ remain well-defined. The CMDQV architecture allows software to program a VCMDQ through the direct aperture without first allocating it to a VINTF; aliasing to the VINTF mmap would route those writes into unallocated logical slots where the hardware silently drops them.

A VCMDQ Page 0 access is served from one of two sources:

- Cache-backed: no hw_queue is allocated for the VCMDQ (HW_QUEUE_ALLOC has not yet succeeded). Both apertures use QEMU's register cache.

- HW-backed: HW_QUEUE_ALLOC has succeeded. Both apertures access the registers directly through the mmap'd VINTF Page 0.

Quoted 7 lines

tegra241_cmdqv_sync_vcmdq() copies any cached writes (CONS_INDX, PROD_INDX, CONFIG, GERRORN) into the mmap'd page on the cache-to-HW transition so the guest's earlier register state survives.

---
 hw/arm/tegra241-cmdqv.c | 73

+++++++++++++++++++++++++++++++++++++++++

hw/arm/tegra241-cmdqv.c static void

Quoted 7 lines
 1 file changed, 73 insertions(+)
hw/arm/tegra241-cmdqv.c +0 -0
index 54d95f4e4e..dce7089697 100644
@@ -39,6 +39,47 @@ static void

tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv)

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv)

Quoted 9 lines
hw/arm/tegra241-cmdqv.c +6 -0
     }
 }
 
+/*
+ * Return a pointer into the mmap'd VINTF page0 for the VCMDQ Page 0
+ * register at @offset0 in VCMDQ slot @index, or NULL when the VCMDQ
+ * has no hw_queue allocated or the VINTF page0 is not mmap'd.
+ */
+static inline uint32_t *tegra241_cmdqv_vintf_ptr(Tegra241CMDQV

*cmdqv,

Quoted 6 lines
hw/arm/tegra241-cmdqv.c +6 -0
+                                                 int index, hwaddr offset0)
+{
+    if (!cmdqv->vcmdq[index] || !cmdqv->vintf_page0) {
+        return NULL;
+    }
+    return (uint32_t *)(cmdqv->vintf_page0 +

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv)

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10 -0
+                        (index * CMDQV_VCMDQ_STRIDE) +
+                        (offset0 - CMDQV_VCMDQ_PAGE0_BASE));
+}
+
+/*
+ * Flush cached register writes into the mmap'd VINTF page0 after a
+ * successful HW_QUEUE_ALLOC, so the guest's earlier writes survive
+ * the cache-to-hardware transition.
+ */
+static void tegra241_cmdqv_sync_vcmdq(Tegra241CMDQV *cmdqv, int

index)

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv)

Quoted 4 lines
hw/arm/tegra241-cmdqv.c +4 -0
+{
+    uint32_t *ptr;
+
+    ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index,

A_VCMDQ0_CONS_INDX);

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv)

Quoted 6 lines
hw/arm/tegra241-cmdqv.c +6 -0
+    if (!ptr) {
+        return;
+    }
+    *ptr = cmdqv->vcmdq_cons_indx[index];
+
+    ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index,

A_VCMDQ0_PROD_INDX);

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_free_all_vcmdq(Tegra241CMDQV *cmdqv)

Quoted 11 lines
hw/arm/tegra241-cmdqv.c +9 -0
+    *ptr = cmdqv->vcmdq_prod_indx[index];
+
+    ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index, A_VCMDQ0_CONFIG);
+    *ptr = cmdqv->vcmdq_config[index];
+
+    ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index, A_VCMDQ0_GERRORN);
+    *ptr = cmdqv->vcmdq_gerrorn[index];
+}
+
 /*
  * Allocate a host HW VCMDQ from the current cached BASE / size for

@index.

hw/arm/tegra241-cmdqv.c static bool

Quoted 2 lines
hw/arm/tegra241-cmdqv.c +0 -0
  *
@@ -95,6 +136,9 @@ static bool

tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int index,

Quoted 8 lines
Quoted 6 lines

hw/arm/tegra241-cmdqv.c static bool · line 136

Quoted 1 line
hw/arm/tegra241-cmdqv.c +0 -0
     cmdqv->vcmdq_gerror[index] &=

~R_VCMDQ0_GERROR_CMDQ_INIT_ERR_MASK;

hw/arm/tegra241-cmdqv.c static bool · line 137

Quoted 1 line
hw/arm/tegra241-cmdqv.c +0 -0
     cmdqv->vcmdq_status[index] |=

R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;

hw/arm/tegra241-cmdqv.c static bool tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int index, · line 139

Quoted 1 line
hw/arm/tegra241-cmdqv.c +1 -0
+    /* Pre-alloc cached writes survive the cache-to-hardware transition. */

what about the opposite transition?

Yeah. We don’t do anything now. I think better to reset the cache on opposite transition.

OK. I would add this explanation.

Quoted 24 lines
Quoted 24 lines

hw/arm/tegra241-cmdqv.c static void

Quoted 7 lines
hw/arm/tegra241-cmdqv.c +2 -0
+    tegra241_cmdqv_sync_vcmdq(cmdqv, index);
+
     return true;
 }
 
@@ -113,13 +157,22 @@ static void

tegra241_cmdqv_setup_all_vcmdq(Tegra241CMDQV *cmdqv,

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_setup_all_vcmdq(Tegra241CMDQV *cmdqv,

Quoted 8 lines
hw/arm/tegra241-cmdqv.c +3 -0
  *
  * The caller normalizes the MMIO offset such that @offset0 always refers
  * to a VCMDQ0_* register, while @index selects the VCMDQ instance.
+ *
+ * If the VCMDQ is allocated and VINTF page0 is mmap'd, read directly
+ * from the VINTF page0 backing. Otherwise, fall back to the cache.
  */
 static uint64_t tegra241_cmdqv_read_vcmdq_page0(Tegra241CMDQV

*cmdqv,

Quoted 7 lines
hw/arm/tegra241-cmdqv.c +2 -0
                                                 hwaddr offset0, int index,
                                                 bool direct)
 {
+    uint32_t *ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index, offset0);
     uint64_t val = 0;
 
+    if (ptr) {
Quoted 8 lines
Quoted 2 lines

shouldn't you test "direct" as well? I know that later on this is will be backed by another mr but I think this would be clearer.

Is the suggestion here to just limit this to "direct"? We can't do that. Both apertures alias the same registers, and for an allocated and mmapped VINTF page0 we access the hw directly.

The vintf page0 gets mapped to the guest only in patch 21, so after that only the direct aperture is trapped here anyway.

Yes I agree this would be true only once patch 21 is applied. Sorry for the noise.

Quoted 57 lines
Quoted 57 lines

hw/arm/tegra241-cmdqv.c static uint64_t

Quoted 9 lines
hw/arm/tegra241-cmdqv.c +4 -0
+        val = *ptr;
+        goto out;
+    }
+
     switch (offset0) {
     case A_VCMDQ0_CONS_INDX:
         val = cmdqv->vcmdq_cons_indx[index];
@@ -144,6 +197,7 @@ static uint64_t

tegra241_cmdqv_read_vcmdq_page0(Tegra241CMDQV *cmdqv,

hw/arm/tegra241-cmdqv.c static uint64_t

Quoted 8 lines
hw/arm/tegra241-cmdqv.c +1 -0
                       "%s unhandled read access at 0x%" PRIx64 "\n",
                       __func__, offset0);
     }
+out:
     trace_tegra241_cmdqv_read_vcmdq_page0(index, direct ? "direct" : "vi",
                                           offset0, val);
     return val;
@@ -220,11 +274,29 @@ static uint64_t

tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv,

hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv,

Quoted 8 lines
hw/arm/tegra241-cmdqv.c +3 -0
  *
  * Page 0 registers are all 32-bit; this helper is only called for 4-byte
  * writes.
+ *
+ * If the VCMDQ is allocated and VINTF page0 is mmap'd, write directly
+ * to the VINTF page0 backing. Otherwise, update the cache.
  */
 static void tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV

*cmdqv,

hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_config_vintf_read(Tegra241CMDQV *cmdqv, · line 285

Quoted 4 lines
hw/arm/tegra241-cmdqv.c +1 -0
                                              hwaddr offset0, int index,
                                              uint32_t value, bool direct)
 {
+    uint32_t *ptr = tegra241_cmdqv_vintf_ptr(cmdqv, index, offset0);

same here?

hw/arm/tegra241-cmdqv.c static void

Quoted 18 lines
hw/arm/tegra241-cmdqv.c +14 -0
+
+    if (ptr) {
+        switch (offset0) {
+        case A_VCMDQ0_CONS_INDX:
+        case A_VCMDQ0_PROD_INDX:
+        case A_VCMDQ0_CONFIG:
+        case A_VCMDQ0_GERRORN:
+            *ptr = value;
+            goto out;
+        default:
+            break;
+        }
+    }
+
     switch (offset0) {
     case A_VCMDQ0_CONS_INDX:
         cmdqv->vcmdq_cons_indx[index] = value;
@@ -255,6 +327,7 @@ static void

tegra241_cmdqv_write_vcmdq_page0(Tegra241CMDQV *cmdqv,

Quoted 5 lines
hw/arm/tegra241-cmdqv.c +1 -0
                       "%s unhandled write access at 0x%" PRIx64 "\n",
                       __func__, offset0);
     }
+out:
     trace_tegra241_cmdqv_write_vcmdq_page0(index, direct ? "direct" : "vi",
Quoted 6 lines
Quoted 2 lines

by the way could be interesting to log in the trace event that mmapped was used instead of cache regs.

Could do.

Thanks,
Shameer

Thanks

Eric

#33 lore raw

PATCH v6 20/31 memory: Allow RAM device regions to skip IOMMU mapping

In reply to Shameer Kolothum

Some RAM device regions created with memory_region_init_ram_device_ptr() are not intended to be P2P DMA targets.

The VFIO listener currently treats all RAM device regions as DMA capable and attempts to map them into the IOMMU. For regions without dma-buf backing this fails and prints warnings such as:

IOMMU_IOAS_MAP failed: Bad address, PCI BAR?

Introduce a MemoryRegion flag (ram_device_skip_iommu_map) to mark RAM device regions that should not be IOMMU mapped, paired with memory_region_skip_iommu_map() / memory_region_set_skip_iommu_map() accessors. When the flag is set, the VFIO listener skips DMA mapping for that region.

---
 include/system/memory.h | 21 +++++++++++++++++++++
 hw/vfio/listener.c      |  6 ++++++
 system/memory.c         | 10 ++++++++++
 hw/vfio/trace-events    |  1 +
 4 files changed, 38 insertions(+)
include/system/memory.h +21 -0
index 1417132f6d..4560809013 100644
@@ -864,6 +864,8 @@ struct MemoryRegion {
 
     /* For devices designed to perform re-entrant IO into their own IO MRs */
     bool disable_reentrancy_guard;
+    /* RAM device region that does not require IOMMU mapping for P2P */
+    bool ram_device_skip_iommu_map;
 };
 
 struct IOMMUMemoryRegion {
@@ -1743,6 +1745,25 @@ static inline bool memory_region_is_romd(const MemoryRegion *mr)
  */
 bool memory_region_is_protected(const MemoryRegion *mr);
 
+/**
+ * memory_region_skip_iommu_map: check whether a memory region is excluded
+ *                               from IOMMU mapping
+ *
+ * Returns %true if @mr is a RAM device region marked to skip IOMMU mapping.
+ *
+ * @mr: the memory region being queried
+ */
+bool memory_region_skip_iommu_map(const MemoryRegion *mr);
+
+/**
+ * memory_region_set_skip_iommu_map: mark a RAM device region to skip IOMMU
+ *                                   mapping
+ *
+ * @mr: the memory region being modified
+ * @skip: %true to skip IOMMU mapping, %false to allow it
+ */
+void memory_region_set_skip_iommu_map(MemoryRegion *mr, bool skip);
+
 /**
  * memory_region_has_guest_memfd: check whether a memory region has guest_memfd
  *     associated
hw/vfio/listener.c +6 -0
index 0b72a2cf5e..14cca678ae 100644
@@ -610,6 +610,12 @@ void vfio_container_region_add(VFIOContainer *bcontainer,
         }
     }
 
+    if (memory_region_skip_iommu_map(section->mr)) {
+        trace_vfio_listener_region_skip_dma_map(memory_region_name(section->mr),
+                                                iova, int128_get64(llsize));
+        return;
+    }
+
     ret = vfio_container_dma_map(bcontainer, iova, int128_get64(llsize),
                                  vaddr, section->readonly, section->mr);
     if (ret) {
system/memory.c +10 -0
index 739ba11da6..48245fd01b 100644
@@ -1814,6 +1814,16 @@ bool memory_region_is_protected(const MemoryRegion *mr)
     return mr->ram && (mr->ram_block->flags & RAM_PROTECTED);
 }
 
+bool memory_region_skip_iommu_map(const MemoryRegion *mr)
+{
+    return memory_region_is_ram_device(mr) && mr->ram_device_skip_iommu_map;
+}
+
+void memory_region_set_skip_iommu_map(MemoryRegion *mr, bool skip)
+{
+    mr->ram_device_skip_iommu_map = skip;
+}
+
 bool memory_region_has_guest_memfd(const MemoryRegion *mr)
 {
     return mr->ram_block && mr->ram_block->guest_memfd >= 0;
hw/vfio/trace-events +1 -0
index 2049159015..70c5aa1bcc 100644
@@ -100,6 +100,7 @@ vfio_listener_region_del_iommu(const char *name) "region_del [iommu] %s"
 vfio_listener_region_add_ram(uint64_t iova_start, uint64_t iova_end, void *vaddr) "region_add [ram] 0x%"PRIx64" - 0x%"PRIx64" [%p]"
 vfio_known_safe_misalignment(const char *name, uint64_t iova, uint64_t offset_within_region, uintptr_t page_size) "Region \"%s\" iova=0x%"PRIx64" offset_within_region=0x%"PRIx64" qemu_real_host_page_size=0x%"PRIxPTR
 vfio_listener_region_add_no_dma_map(const char *name, uint64_t iova, uint64_t size, uint64_t page_size) "Region \"%s\" 0x%"PRIx64" size=0x%"PRIx64" is not aligned to 0x%"PRIx64" and cannot be mapped for DMA"
+vfio_listener_region_skip_dma_map(const char *name, uint64_t iova, uint64_t size) "Region \"%s\" 0x%"PRIx64" size=0x%"PRIx64" marked to skip IOMMU mapping"
 vfio_listener_region_del(uint64_t start, uint64_t end) "region_del 0x%"PRIx64" - 0x%"PRIx64
 vfio_device_dirty_tracking_update(uint64_t start, uint64_t end, uint64_t min, uint64_t max) "section 0x%"PRIx64" - 0x%"PRIx64" -> update [0x%"PRIx64" - 0x%"PRIx64"]"
 vfio_device_dirty_tracking_start(int nr_ranges, uint64_t min32, uint64_t max32, uint64_t min64, uint64_t max64, uint64_t minpci, uint64_t maxpci) "nr_ranges %d 32:[0x%"PRIx64" - 0x%"PRIx64"], 64:[0x%"PRIx64" - 0x%"PRIx64"], pci64:[0x%"PRIx64" - 0x%"PRIx64"]"
Signature · 1 line
2.43.0
#34 lore raw

PATCH v6 21/31 hw/arm/tegra241-cmdqv: Map VINTF page0 into guest MMIO space

In reply to Shameer Kolothum

From: Nicolin Chen <[email protected]>

Install the mmap'd vintf_page0 region as a RAM-device MemoryRegion at guest MMIO offset 0x30000 (the VINTF Page 0 aperture) when VINTF is enabled, and remove it on VINTF disable. This eliminates QEMU trapping for hot-path CONS/PROD index updates via the VINTF aperture.

After this patch, the two VCMDQ Page 0 apertures use different access paths: the direct aperture (0x10000) remains QEMU-trapped, while the VINTF aperture (0x30000) is a guest-direct RAM mapping.

The direct aperture is intentionally kept trapped (not aliased to the VINTF mmap) so that writes to an unallocated VCMDQ remain well-defined. The CMDQV architecture allows software to program a VCMDQ through the direct aperture without first allocating it to a VINTF; aliasing would route those writes to unallocated logical slots in the VINTF page, where the hardware silently drops them.

---
 hw/arm/tegra241-cmdqv.h |  1 +
 hw/arm/tegra241-cmdqv.c | 36 ++++++++++++++++++++++++++++++++++++
 2 files changed, 37 insertions(+)
hw/arm/tegra241-cmdqv.h +1 -0
index 84499b840d..01cd6af97d 100644
@@ -49,6 +49,7 @@ typedef struct Tegra241CMDQV {
     IOMMUFDVeventq *veventq;
     IOMMUFDHWqueue *vcmdq[TEGRA241_CMDQV_MAX_CMDQ];
     void *vintf_page0;
+    MemoryRegion *mr_vintf_page0;
 
     /* CMDQ-V Config page register cache */
     uint32_t config;
hw/arm/tegra241-cmdqv.c +36 -0
index dce7089697..cdd4fa4f1a 100644
@@ -18,6 +18,40 @@
 #include "tegra241-cmdqv.h"
 #include "trace.h"
 
+static void tegra241_cmdqv_guest_unmap_vintf_page0(Tegra241CMDQV *cmdqv)
+{
+    if (!cmdqv->mr_vintf_page0) {
+        return;
+    }
+
+    memory_region_del_subregion(&cmdqv->mmio_cmdqv, cmdqv->mr_vintf_page0);
+    object_unparent(OBJECT(cmdqv->mr_vintf_page0));
+    g_free(cmdqv->mr_vintf_page0);
+    cmdqv->mr_vintf_page0 = NULL;
+}
+
+static void tegra241_cmdqv_guest_map_vintf_page0(Tegra241CMDQV *cmdqv)
+{
+    char *name;
+
+    if (cmdqv->mr_vintf_page0) {
+        return;
+    }
+
+    name = g_strdup_printf("%s vintf-page0",
+                           memory_region_name(&cmdqv->mmio_cmdqv));
+    cmdqv->mr_vintf_page0 = g_malloc0(sizeof(*cmdqv->mr_vintf_page0));
+    memory_region_init_ram_device_ptr(cmdqv->mr_vintf_page0,
+                                      memory_region_owner(&cmdqv->mmio_cmdqv),
+                                      name, VINTF_PAGE_SIZE,
+                                      cmdqv->vintf_page0);
+    memory_region_set_skip_iommu_map(cmdqv->mr_vintf_page0, true);
+    memory_region_add_subregion_overlap(&cmdqv->mmio_cmdqv,
+                                        CMDQV_VINTF_PAGE0_BASE,
+                                        cmdqv->mr_vintf_page0, 1);
+    g_free(name);
+}
+
 static void tegra241_cmdqv_free_vcmdq(Tegra241CMDQV *cmdqv, int index)
 {
     IOMMUFDViommu *viommu = cmdqv->s_accel->viommu;
@@ -445,6 +479,7 @@ static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv,
         if (value & R_VINTF0_CONFIG_ENABLE_MASK) {
             if (tegra241_cmdqv_mmap_vintf_page0(cmdqv, errp)) {
                 cmdqv->vintf_status |= R_VINTF0_STATUS_ENABLE_OK_MASK;
+                tegra241_cmdqv_guest_map_vintf_page0(cmdqv);
                 /*
                  * VCMDQs whose BASE was programmed before VINTF was
                  * enabled need their hw_queue allocated now.
@@ -452,6 +487,7 @@ static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv,
                 tegra241_cmdqv_setup_all_vcmdq(cmdqv, errp);
             }
         } else {
+            tegra241_cmdqv_guest_unmap_vintf_page0(cmdqv);
             tegra241_cmdqv_free_all_vcmdq(cmdqv);
             tegra241_cmdqv_munmap_vintf_page0(cmdqv, errp);
             cmdqv->vintf_status &= ~R_VINTF0_STATUS_ENABLE_OK_MASK;
Signature · 1 line
2.43.0
#35 lore raw

Re: [PATCH v6 21/31] hw/arm/tegra241-cmdqv: Map VINTF page0 into guest MMIO space

Hi Shammer,

Quoted 1 line · Shameer Kolothum

From: Nicolin Chen <[email protected]>

I would disambiguate the title a little bit:
Use mmap'd host VINTF page0 for virtual VINTF page0

Also in previous patch titles you could emphase the mmap'd page is the host one in a similar way.

Quoted 83 lines

Install the mmap'd vintf_page0 region as a RAM-device MemoryRegion at guest MMIO offset 0x30000 (the VINTF Page 0 aperture) when VINTF is enabled, and remove it on VINTF disable. This eliminates QEMU trapping for hot-path CONS/PROD index updates via the VINTF aperture.

After this patch, the two VCMDQ Page 0 apertures use different access paths: the direct aperture (0x10000) remains QEMU-trapped, while the VINTF aperture (0x30000) is a guest-direct RAM mapping.

The direct aperture is intentionally kept trapped (not aliased to the VINTF mmap) so that writes to an unallocated VCMDQ remain well-defined. The CMDQV architecture allows software to program a VCMDQ through the direct aperture without first allocating it to a VINTF; aliasing would route those writes to unallocated logical slots in the VINTF page, where the hardware silently drops them.

---
 hw/arm/tegra241-cmdqv.h |  1 +
 hw/arm/tegra241-cmdqv.c | 36 ++++++++++++++++++++++++++++++++++++
 2 files changed, 37 insertions(+)
hw/arm/tegra241-cmdqv.h +1 -0
index 84499b840d..01cd6af97d 100644
@@ -49,6 +49,7 @@ typedef struct Tegra241CMDQV {
     IOMMUFDVeventq *veventq;
     IOMMUFDHWqueue *vcmdq[TEGRA241_CMDQV_MAX_CMDQ];
     void *vintf_page0;
+    MemoryRegion *mr_vintf_page0;
 
     /* CMDQ-V Config page register cache */
     uint32_t config;
hw/arm/tegra241-cmdqv.c +34 -0
index dce7089697..cdd4fa4f1a 100644
@@ -18,6 +18,40 @@
 #include "tegra241-cmdqv.h"
 #include "trace.h"
 
+static void tegra241_cmdqv_guest_unmap_vintf_page0(Tegra241CMDQV *cmdqv)
+{
+    if (!cmdqv->mr_vintf_page0) {
+        return;
+    }
+
+    memory_region_del_subregion(&cmdqv->mmio_cmdqv, cmdqv->mr_vintf_page0);
+    object_unparent(OBJECT(cmdqv->mr_vintf_page0));
+    g_free(cmdqv->mr_vintf_page0);
+    cmdqv->mr_vintf_page0 = NULL;
+}
+
+static void tegra241_cmdqv_guest_map_vintf_page0(Tegra241CMDQV *cmdqv)
+{
+    char *name;
+
+    if (cmdqv->mr_vintf_page0) {
+        return;
+    }
+
+    name = g_strdup_printf("%s vintf-page0",
+                           memory_region_name(&cmdqv->mmio_cmdqv));
+    cmdqv->mr_vintf_page0 = g_malloc0(sizeof(*cmdqv->mr_vintf_page0));
+    memory_region_init_ram_device_ptr(cmdqv->mr_vintf_page0,
+                                      memory_region_owner(&cmdqv->mmio_cmdqv),
+                                      name, VINTF_PAGE_SIZE,
+                                      cmdqv->vintf_page0);
+    memory_region_set_skip_iommu_map(cmdqv->mr_vintf_page0, true);
+    memory_region_add_subregion_overlap(&cmdqv->mmio_cmdqv,
+                                        CMDQV_VINTF_PAGE0_BASE,
+                                        cmdqv->mr_vintf_page0, 1);
+    g_free(name);
+}
+
 static void tegra241_cmdqv_free_vcmdq(Tegra241CMDQV *cmdqv, int index)
 {
     IOMMUFDViommu *viommu = cmdqv->s_accel->viommu;

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv, · line 482

Quoted 4 lines
hw/arm/tegra241-cmdqv.c +1 -0
@@ -445,6 +479,7 @@ static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv,
         if (value & R_VINTF0_CONFIG_ENABLE_MASK) {
             if (tegra241_cmdqv_mmap_vintf_page0(cmdqv, errp)) {
                 cmdqv->vintf_status |= R_VINTF0_STATUS_ENABLE_OK_MASK;
+                tegra241_cmdqv_guest_map_vintf_page0(cmdqv);

don't you want to do that only after the tegra241_cmdqv_setup_all_vcmdq()?

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv, · line 493

Quoted 11 lines
hw/arm/tegra241-cmdqv.c +1 -0
                 /*
                  * VCMDQs whose BASE was programmed before VINTF was
                  * enabled need their hw_queue allocated now.
@@ -452,6 +487,7 @@ static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv,
                 tegra241_cmdqv_setup_all_vcmdq(cmdqv, errp);
             }
         } else {
+            tegra241_cmdqv_guest_unmap_vintf_page0(cmdqv);
             tegra241_cmdqv_free_all_vcmdq(cmdqv);
             tegra241_cmdqv_munmap_vintf_page0(cmdqv, errp);
             cmdqv->vintf_status &= ~R_VINTF0_STATUS_ENABLE_OK_MASK;

Thanks

Eric

#36 lore raw

RE: [PATCH v6 21/31] hw/arm/tegra241-cmdqv: Map VINTF page0 into guest MMIO space

Quoted 20 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 04 June 2026 11:07
To: Shameer Kolothum Thodi <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v6 21/31] hw/arm/tegra241-cmdqv: Map VINTF page0 into guest MMIO space

External email: Use caution opening links or attachments

Hi Shammer,

Quoted 7 lines
Quoted 1 line · Shameer Kolothum

From: Nicolin Chen <[email protected]>

I would disambiguate the title a little bit:
Use mmap'd host VINTF page0 for virtual VINTF page0

Also in previous patch titles you could emphase the mmap'd page is the host one in a similar way.

Ok.

Quoted 84 lines
Quoted 16 lines

Install the mmap'd vintf_page0 region as a RAM-device MemoryRegion at guest MMIO offset 0x30000 (the VINTF Page 0 aperture) when VINTF is enabled, and remove it on VINTF disable. This eliminates QEMU trapping for hot-path CONS/PROD index updates via the VINTF aperture.

After this patch, the two VCMDQ Page 0 apertures use different access
paths: the direct aperture (0x10000) remains QEMU-trapped, while the VINTF aperture (0x30000) is a guest-direct RAM mapping.

The direct aperture is intentionally kept trapped (not aliased to the VINTF mmap) so that writes to an unallocated VCMDQ remain well-defined. The CMDQV architecture allows software to program a VCMDQ through the direct aperture without first allocating it to a VINTF; aliasing would route those writes to unallocated logical slots in the VINTF page, where the hardware silently drops them.

Quoted 6 lines
---
 hw/arm/tegra241-cmdqv.h |  1 +
 hw/arm/tegra241-cmdqv.c | 36

++++++++++++++++++++++++++++++++++++

Quoted 59 lines
 2 files changed, 37 insertions(+)
hw/arm/tegra241-cmdqv.h index renamed from hw/arm/tegra241-cmdqv.h +0 -0

84499b840d..01cd6af97d 100644

hw/arm/tegra241-cmdqv.h +1 -0
@@ -49,6 +49,7 @@ typedef struct Tegra241CMDQV {
     IOMMUFDVeventq *veventq;
     IOMMUFDHWqueue *vcmdq[TEGRA241_CMDQV_MAX_CMDQ];
     void *vintf_page0;
+    MemoryRegion *mr_vintf_page0;
 
     /* CMDQ-V Config page register cache */
     uint32_t config;
hw/arm/tegra241-cmdqv.c index renamed from hw/arm/tegra241-cmdqv.c +0 -0

dce7089697..cdd4fa4f1a 100644

hw/arm/tegra241-cmdqv.c +34 -0
@@ -18,6 +18,40 @@
 #include "tegra241-cmdqv.h"
 #include "trace.h"
 
+static void tegra241_cmdqv_guest_unmap_vintf_page0(Tegra241CMDQV
+*cmdqv) {
+    if (!cmdqv->mr_vintf_page0) {
+        return;
+    }
+
+    memory_region_del_subregion(&cmdqv->mmio_cmdqv, cmdqv-
 mr_vintf_page0);
+    object_unparent(OBJECT(cmdqv->mr_vintf_page0));
+    g_free(cmdqv->mr_vintf_page0);
+    cmdqv->mr_vintf_page0 = NULL;
+}
+
+static void tegra241_cmdqv_guest_map_vintf_page0(Tegra241CMDQV
+*cmdqv) {
+    char *name;
+
+    if (cmdqv->mr_vintf_page0) {
+        return;
+    }
+
+    name = g_strdup_printf("%s vintf-page0",
+                           memory_region_name(&cmdqv->mmio_cmdqv));
+    cmdqv->mr_vintf_page0 = g_malloc0(sizeof(*cmdqv->mr_vintf_page0));
+    memory_region_init_ram_device_ptr(cmdqv->mr_vintf_page0,
+                                      memory_region_owner(&cmdqv->mmio_cmdqv),
+                                      name, VINTF_PAGE_SIZE,
+                                      cmdqv->vintf_page0);
+    memory_region_set_skip_iommu_map(cmdqv->mr_vintf_page0, true);
+    memory_region_add_subregion_overlap(&cmdqv->mmio_cmdqv,
+                                        CMDQV_VINTF_PAGE0_BASE,
+                                        cmdqv->mr_vintf_page0, 1);
+    g_free(name);
+}
+
 static void tegra241_cmdqv_free_vcmdq(Tegra241CMDQV *cmdqv, int
 index)  {
Quoted 1 line
     IOMMUFDViommu *viommu = cmdqv->s_accel->viommu; @@ -445,6

+479,7

Quoted 8 lines
Quoted 1 line

@@ static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV

*cmdqv,

Quoted 5 lines
         if (value & R_VINTF0_CONFIG_ENABLE_MASK) {
             if (tegra241_cmdqv_mmap_vintf_page0(cmdqv, errp)) {
                 cmdqv->vintf_status |=

R_VINTF0_STATUS_ENABLE_OK_MASK;
+ tegra241_cmdqv_guest_map_vintf_page0(cmdqv);

don't you want to do that only after the tegra241_cmdqv_setup_all_vcmdq()?

Functionally same. Could do, reads better, I think.

Thanks,
Shameer

#37 lore raw

PATCH v6 22/31 hw/arm/smmuv3-accel: Introduce common helper for veventq read

In reply to Shameer Kolothum

Move the vEVENTQ read and validation logic into a common helper smmuv3_accel_event_read_validate(). The helper performs the read(), checks for overflow and short reads, validates the sequence number, and updates the sequence state.

This helper can be reused for Tegra241 CMDQV vEVENTQ support in a subsequent patch.

Error handling is slightly adjusted: instead of reporting errors directly in the read handler, the helper now returns errors via Error **. Sequence gaps are reported as warnings.

---
 hw/arm/smmuv3-accel.h       |  2 ++
 hw/arm/smmuv3-accel-stubs.c |  7 ++++
 hw/arm/smmuv3-accel.c       | 67 ++++++++++++++++++++++---------------
 3 files changed, 49 insertions(+), 27 deletions(-)
hw/arm/smmuv3-accel.h +2 -0
index e0bbec8581..8d0f636338 100644
@@ -87,6 +87,8 @@ bool smmuv3_accel_issue_inv_cmd(SMMUv3State *s, void *cmd, SMMUDevice *sdev,
                                 Error **errp);
 void smmuv3_accel_idr_override(SMMUv3State *s);
 bool smmuv3_accel_alloc_veventq(SMMUv3State *s, Error **errp);
+bool smmuv3_accel_event_read_validate(IOMMUFDVeventq *veventq, uint32_t type,
+                                      void *buf, size_t size, Error **errp);
 void smmuv3_accel_reset(SMMUv3State *s);
 
 #endif /* HW_ARM_SMMUV3_ACCEL_H */
hw/arm/smmuv3-accel-stubs.c +7 -0
index 70cef66966..9e6c44a282 100644
@@ -47,6 +47,13 @@ bool smmuv3_accel_alloc_veventq(SMMUv3State *s, Error **errp)
     return true;
 }
 
+bool smmuv3_accel_event_read_validate(IOMMUFDVeventq *veventq, uint32_t type,
+                                      void *buf, size_t size, Error **errp)
+{
+    return true;
+}
+
+
 void smmuv3_accel_reset(SMMUv3State *s)
 {
 }
hw/arm/smmuv3-accel.c +40 -27

87 lines not on this page · raw message

Signature · 1 line
2.43.0
#38 lore raw

PATCH v6 23/31 hw/arm/tegra241-cmdqv: Read and propagate Tegra241 CMDQV errors

In reply to Shameer Kolothum

Install an event handler on the CMDQV vEVENTQ fd to read and propagate host received CMDQV errors to the guest.

The handler runs in QEMU's main loop, using a non-blocking fd registered via qemu_set_fd_handler().

---
 hw/arm/tegra241-cmdqv.c | 63 +++++++++++++++++++++++++++++++++++++++++
 hw/arm/trace-events     |  1 +
 2 files changed, 64 insertions(+)
hw/arm/tegra241-cmdqv.c +63 -0
index cdd4fa4f1a..d8b36164a1 100644
@@ -14,6 +14,7 @@
 
 #include "hw/arm/smmuv3.h"
 #include "hw/arm/smmuv3-common.h"
+#include "hw/core/irq.h"
 #include "smmuv3-accel.h"
 #include "tegra241-cmdqv.h"
 #include "trace.h"
@@ -751,6 +752,51 @@ out:
     trace_tegra241_cmdqv_write_mmio(offset, value, size);
 }
 
+static void tegra241_cmdqv_event_read(void *opaque)
+{
+    Tegra241CMDQV *cmdqv = opaque;
+    IOMMUFDVeventq *veventq = cmdqv->veventq;
+    struct {
+        struct iommufd_vevent_header hdr;
+        struct iommu_vevent_tegra241_cmdqv vevent;
+    } buf;
+    Error *local_err = NULL;
+
+    if (!smmuv3_accel_event_read_validate(veventq,
+                                          IOMMU_VEVENTQ_TYPE_TEGRA241_CMDQV,
+                                          &buf, sizeof(buf), &local_err)) {
+        warn_report_err_once(local_err);
+        return;
+    }
+
+    if (buf.vevent.lvcmdq_err_map[0] || buf.vevent.lvcmdq_err_map[1]) {
+        cmdqv->vintf_cmdq_err_map[0] =
+            extract64(buf.vevent.lvcmdq_err_map[0], 0, 32);
+        cmdqv->vintf_cmdq_err_map[1] =
+            extract64(buf.vevent.lvcmdq_err_map[0], 32, 32);
+        cmdqv->vintf_cmdq_err_map[2] =
+            extract64(buf.vevent.lvcmdq_err_map[1], 0, 32);
+        cmdqv->vintf_cmdq_err_map[3] =
+            extract64(buf.vevent.lvcmdq_err_map[1], 32, 32);
+        /*
+         * CMDQV_CMDQ_ERR_MAP and VINTF0_LVCMDQ_ERR_MAP are distinct
+         * registers (different MMIO offsets). With only VINTF0 exposed
+         * they carry the same data, so mirror.
+         */
+        for (int i = 0; i < 4; i++) {
+            cmdqv->cmdq_err_map[i] = cmdqv->vintf_cmdq_err_map[i];
+        }
+        /* Set the VINTF0 bit in VI_ERR_MAP_0 (only VINTF0 is exposed). */
+        cmdqv->vi_err_map[0] |= BIT(0);
+        if (!(cmdqv->vi_int_mask[0] & BIT(0))) {
+            qemu_irq_pulse(cmdqv->irq);
+        }
+        trace_tegra241_cmdqv_err_map(
+            cmdqv->vintf_cmdq_err_map[3], cmdqv->vintf_cmdq_err_map[2],
+            cmdqv->vintf_cmdq_err_map[1], cmdqv->vintf_cmdq_err_map[0]);
+    }
+}
+
 static void tegra241_cmdqv_free_viommu(SMMUv3State *s)
 {
     SMMUv3AccelState *accel = s->s_accel;
@@ -762,6 +808,7 @@ static void tegra241_cmdqv_free_viommu(SMMUv3State *s)
         return;
     }
     if (veventq) {
+        qemu_set_fd_handler(veventq->veventq_fd, NULL, NULL, NULL);
         close(veventq->veventq_fd);
         iommufd_backend_free_id(viommu->iommufd, veventq->veventq_id);
         g_free(veventq);
@@ -777,6 +824,7 @@ tegra241_cmdqv_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
     Tegra241CMDQV *cmdqv = s->s_accel->cmdqv;
     uint32_t viommu_id, veventq_id, veventq_fd;
     IOMMUFDVeventq *veventq;
+    int flags;
 
     if (!iommufd_backend_alloc_viommu(idev->iommufd, idev->devid,
                                       IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV,
@@ -795,14 +843,29 @@ tegra241_cmdqv_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
         goto free_viommu;
     }
 
+    flags = fcntl(veventq_fd, F_GETFL);
+    if (flags < 0) {
+        error_setg(errp, "Failed to get flags for vEVENTQ fd");
+        goto free_veventq;
+    }
+    if (fcntl(veventq_fd, F_SETFL, O_NONBLOCK | flags) < 0) {
+        error_setg(errp, "Failed to set O_NONBLOCK on vEVENTQ fd");
+        goto free_veventq;
+    }
+
     veventq = g_new(IOMMUFDVeventq, 1);
     veventq->veventq_id = veventq_id;
     veventq->veventq_fd = veventq_fd;
     cmdqv->veventq = veventq;
 
+    /* Set up event handler for veventq fd */
+    qemu_set_fd_handler(veventq_fd, tegra241_cmdqv_event_read, NULL, cmdqv);
     *out_viommu_id = viommu_id;
     return true;
 
+free_veventq:
+    close(veventq_fd);
+    iommufd_backend_free_id(idev->iommufd, veventq_id);
 free_viommu:
     iommufd_backend_free_id(idev->iommufd, viommu_id);
     return false;
hw/arm/trace-events +1 -0
index d76f408d8a..115f1add9d 100644
@@ -75,6 +75,7 @@ smmuv3_accel_install_ste(uint32_t vsid, const char * type, uint32_t hwpt_id) "vS
 # tegra241-cmdqv
 tegra241_cmdqv_read_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x"
 tegra241_cmdqv_write_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x"
+tegra241_cmdqv_err_map(uint32_t map3, uint32_t map2, uint32_t map1, uint32_t map0) "hw irq received. error (hex) maps: %04X:%04X:%04X:%04X"
 tegra241_cmdqv_read_vcmdq_page0(int index, const char *aperture, uint64_t offset0, uint64_t val) "vcmdq[%d] %s page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64
 tegra241_cmdqv_read_vcmdq_page1(int index, const char *aperture, uint64_t offset0, uint64_t val) "vcmdq[%d] %s page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64
 tegra241_cmdqv_write_vcmdq_page0(int index, const char *aperture, uint64_t offset0, uint64_t val) "vcmdq[%d] %s page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64
Signature · 1 line
2.43.0
Reviewed-by on v6 23/31
#39 lore raw

Re: [PATCH v6 23/31] hw/arm/tegra241-cmdqv: Read and propagate Tegra241 CMDQV errors

Quoted 8 lines · Shameer Kolothum

Install an event handler on the CMDQV vEVENTQ fd to read and propagate host received CMDQV errors to the guest.

The handler runs in QEMU's main loop, using a non-blocking fd registered via qemu_set_fd_handler().

Eric

Quoted 127 lines
---
 hw/arm/tegra241-cmdqv.c | 63 +++++++++++++++++++++++++++++++++++++++++
 hw/arm/trace-events     |  1 +
 2 files changed, 64 insertions(+)
hw/arm/tegra241-cmdqv.c +63 -0
index cdd4fa4f1a..d8b36164a1 100644
@@ -14,6 +14,7 @@
 
 #include "hw/arm/smmuv3.h"
 #include "hw/arm/smmuv3-common.h"
+#include "hw/core/irq.h"
 #include "smmuv3-accel.h"
 #include "tegra241-cmdqv.h"
 #include "trace.h"
@@ -751,6 +752,51 @@ out:
     trace_tegra241_cmdqv_write_mmio(offset, value, size);
 }
 
+static void tegra241_cmdqv_event_read(void *opaque)
+{
+    Tegra241CMDQV *cmdqv = opaque;
+    IOMMUFDVeventq *veventq = cmdqv->veventq;
+    struct {
+        struct iommufd_vevent_header hdr;
+        struct iommu_vevent_tegra241_cmdqv vevent;
+    } buf;
+    Error *local_err = NULL;
+
+    if (!smmuv3_accel_event_read_validate(veventq,
+                                          IOMMU_VEVENTQ_TYPE_TEGRA241_CMDQV,
+                                          &buf, sizeof(buf), &local_err)) {
+        warn_report_err_once(local_err);
+        return;
+    }
+
+    if (buf.vevent.lvcmdq_err_map[0] || buf.vevent.lvcmdq_err_map[1]) {
+        cmdqv->vintf_cmdq_err_map[0] =
+            extract64(buf.vevent.lvcmdq_err_map[0], 0, 32);
+        cmdqv->vintf_cmdq_err_map[1] =
+            extract64(buf.vevent.lvcmdq_err_map[0], 32, 32);
+        cmdqv->vintf_cmdq_err_map[2] =
+            extract64(buf.vevent.lvcmdq_err_map[1], 0, 32);
+        cmdqv->vintf_cmdq_err_map[3] =
+            extract64(buf.vevent.lvcmdq_err_map[1], 32, 32);
+        /*
+         * CMDQV_CMDQ_ERR_MAP and VINTF0_LVCMDQ_ERR_MAP are distinct
+         * registers (different MMIO offsets). With only VINTF0 exposed
+         * they carry the same data, so mirror.
+         */
+        for (int i = 0; i < 4; i++) {
+            cmdqv->cmdq_err_map[i] = cmdqv->vintf_cmdq_err_map[i];
+        }
+        /* Set the VINTF0 bit in VI_ERR_MAP_0 (only VINTF0 is exposed). */
+        cmdqv->vi_err_map[0] |= BIT(0);
+        if (!(cmdqv->vi_int_mask[0] & BIT(0))) {
+            qemu_irq_pulse(cmdqv->irq);
+        }
+        trace_tegra241_cmdqv_err_map(
+            cmdqv->vintf_cmdq_err_map[3], cmdqv->vintf_cmdq_err_map[2],
+            cmdqv->vintf_cmdq_err_map[1], cmdqv->vintf_cmdq_err_map[0]);
+    }
+}
+
 static void tegra241_cmdqv_free_viommu(SMMUv3State *s)
 {
     SMMUv3AccelState *accel = s->s_accel;
@@ -762,6 +808,7 @@ static void tegra241_cmdqv_free_viommu(SMMUv3State *s)
         return;
     }
     if (veventq) {
+        qemu_set_fd_handler(veventq->veventq_fd, NULL, NULL, NULL);
         close(veventq->veventq_fd);
         iommufd_backend_free_id(viommu->iommufd, veventq->veventq_id);
         g_free(veventq);
@@ -777,6 +824,7 @@ tegra241_cmdqv_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
     Tegra241CMDQV *cmdqv = s->s_accel->cmdqv;
     uint32_t viommu_id, veventq_id, veventq_fd;
     IOMMUFDVeventq *veventq;
+    int flags;
 
     if (!iommufd_backend_alloc_viommu(idev->iommufd, idev->devid,
                                       IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV,
@@ -795,14 +843,29 @@ tegra241_cmdqv_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
         goto free_viommu;
     }
 
+    flags = fcntl(veventq_fd, F_GETFL);
+    if (flags < 0) {
+        error_setg(errp, "Failed to get flags for vEVENTQ fd");
+        goto free_veventq;
+    }
+    if (fcntl(veventq_fd, F_SETFL, O_NONBLOCK | flags) < 0) {
+        error_setg(errp, "Failed to set O_NONBLOCK on vEVENTQ fd");
+        goto free_veventq;
+    }
+
     veventq = g_new(IOMMUFDVeventq, 1);
     veventq->veventq_id = veventq_id;
     veventq->veventq_fd = veventq_fd;
     cmdqv->veventq = veventq;
 
+    /* Set up event handler for veventq fd */
+    qemu_set_fd_handler(veventq_fd, tegra241_cmdqv_event_read, NULL, cmdqv);
     *out_viommu_id = viommu_id;
     return true;
 
+free_veventq:
+    close(veventq_fd);
+    iommufd_backend_free_id(idev->iommufd, veventq_id);
 free_viommu:
     iommufd_backend_free_id(idev->iommufd, viommu_id);
     return false;
hw/arm/trace-events +1 -0
index d76f408d8a..115f1add9d 100644
@@ -75,6 +75,7 @@ smmuv3_accel_install_ste(uint32_t vsid, const char * type, uint32_t hwpt_id) "vS
 # tegra241-cmdqv
 tegra241_cmdqv_read_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x"
 tegra241_cmdqv_write_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x"
+tegra241_cmdqv_err_map(uint32_t map3, uint32_t map2, uint32_t map1, uint32_t map0) "hw irq received. error (hex) maps: %04X:%04X:%04X:%04X"
 tegra241_cmdqv_read_vcmdq_page0(int index, const char *aperture, uint64_t offset0, uint64_t val) "vcmdq[%d] %s page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64
 tegra241_cmdqv_read_vcmdq_page1(int index, const char *aperture, uint64_t offset0, uint64_t val) "vcmdq[%d] %s page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64
 tegra241_cmdqv_write_vcmdq_page0(int index, const char *aperture, uint64_t offset0, uint64_t val) "vcmdq[%d] %s page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64
#40 lore raw

PATCH v6 24/31 hw/arm/tegra241-cmdqv: Add reset handler

In reply to Shameer Kolothum

From: Nicolin Chen <[email protected]>

Introduce a reset handler for the Tegra241 CMDQV and initialize its register state.

---
 hw/arm/tegra241-cmdqv.h |  3 +++
 hw/arm/tegra241-cmdqv.c | 48 +++++++++++++++++++++++++++++++++++++++++
 hw/arm/trace-events     |  1 +
 3 files changed, 52 insertions(+)
hw/arm/tegra241-cmdqv.h +3 -0
index 01cd6af97d..de4c1e5335 100644
@@ -89,6 +89,9 @@ FIELD(CONFIG, CMDQ_MAX_CLK_BATCH, 4, 8)
 FIELD(CONFIG, CMDQ_MAX_CMD_BATCH, 12, 8)
 FIELD(CONFIG, CONS_DRAM_EN, 20, 1)
 
+/* CMDQV_EN=1, PER_CMD_OFFSET=16B, CLK_BATCH=256, CMD_BATCH=32. */
+#define V_CONFIG_RESET 0x00020083
+
 REG32(PARAM, 0x4)
 FIELD(PARAM, CMDQV_VER, 0, 4)
 FIELD(PARAM, CMDQV_NUM_CMDQ_LOG2, 4, 4)
hw/arm/tegra241-cmdqv.c +48 -0

56 lines not on this page · raw message

hw/arm/trace-events +1 -0
index 115f1add9d..e655b2e26e 100644
@@ -76,6 +76,7 @@ smmuv3_accel_install_ste(uint32_t vsid, const char * type, uint32_t hwpt_id) "vS
 tegra241_cmdqv_read_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x"
 tegra241_cmdqv_write_mmio(uint64_t offset, uint64_t val, unsigned size) "offset: 0x%"PRIx64" val: 0x%"PRIx64" size: 0x%x"
 tegra241_cmdqv_err_map(uint32_t map3, uint32_t map2, uint32_t map1, uint32_t map0) "hw irq received. error (hex) maps: %04X:%04X:%04X:%04X"
+tegra241_cmdqv_init_regs(uint32_t param) "register init, param=0x%08X"
 tegra241_cmdqv_read_vcmdq_page0(int index, const char *aperture, uint64_t offset0, uint64_t val) "vcmdq[%d] %s page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64
 tegra241_cmdqv_read_vcmdq_page1(int index, const char *aperture, uint64_t offset0, uint64_t val) "vcmdq[%d] %s page1 offset0: 0x%"PRIx64" val: 0x%"PRIx64
 tegra241_cmdqv_write_vcmdq_page0(int index, const char *aperture, uint64_t offset0, uint64_t val) "vcmdq[%d] %s page0 offset0: 0x%"PRIx64" val: 0x%"PRIx64
Signature · 1 line
2.43.0
#41 lore raw

PATCH v6 25/31 hw/arm/tegra241-cmdqv: Limit queue size based on backend page size

In reply to Shameer Kolothum

From: Nicolin Chen <[email protected]>

CMDQV HW performs DMA accesses to guest queue memory by its host physical address set up via IOMMUFD. This requires the guest queue to be contiguous in both guest PA and host PA space. With Tegra241 CMDQV enabled, we must only advertise a command queue size (CMDQS) that the host can safely back with physically contiguous memory. Allowing a queue size larger than the host page size could cause the hardware to DMA across page boundaries, leading to faults.

Use qemu_minrampagesize() to find the smallest memory-backend page size in use, then cap IDR1.CMDQS so the guest cannot configure a command queue that exceeds that contiguous backing.

Note this is done at SMMUv3 init, before any guest queue GPA is known, so the cap is conservative. Maximum queue size is 8MiB; it is recommended to back the VM with hugepage sizes large enough so CMDQS stays at the HW maximum. Smaller backing pages reduce CMDQS accordingly.

---
 hw/arm/tegra241-cmdqv.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
hw/arm/tegra241-cmdqv.c +20 -0
index ac17b56c1c..661c46e3d9 100644
@@ -16,6 +16,8 @@
 #include "hw/arm/smmuv3-common.h"
 #include "hw/core/irq.h"
 #include "smmuv3-accel.h"
+#include "smmuv3-internal.h"
+#include "system/hostmem.h"
 #include "tegra241-cmdqv.h"
 #include "trace.h"
 
@@ -874,6 +876,8 @@ free_viommu:
 static void tegra241_cmdqv_init_regs(SMMUv3State *s, Tegra241CMDQV *cmdqv)
 {
     int i;
+    long pgsize;
+    uint32_t val;
 
     cmdqv->config = V_CONFIG_RESET;
     cmdqv->param = FIELD_DP32(0, PARAM, CMDQV_VER, CMDQV_VER);
@@ -905,6 +909,22 @@ static void tegra241_cmdqv_init_regs(SMMUv3State *s, Tegra241CMDQV *cmdqv)
         cmdqv->vcmdq_base[i] = 0;
         cmdqv->vcmdq_cons_indx_base[i] = 0;
     }
+
+    /*
+     * CMDQ must not cross a physical RAM backend page. Adjust CMDQS so the
+     * queue fits entirely within the smallest backend page size, ensuring
+     * the command queue is physically contiguous in host memory.
+     *
+     *   IDR1.CMDQS = log2(max_qsz) - entry_shift
+     *
+     * where entry_shift = 4 (each CMDQ entry is 16 bytes = 2^4).
+     */
+    pgsize = qemu_minrampagesize();
+    if (pgsize == LONG_MAX) {
+        pgsize = qemu_real_host_page_size();
+    }
+    val = FIELD_EX32(s->idr[1], IDR1, CMDQS);
+    s->idr[1] = FIELD_DP32(s->idr[1], IDR1, CMDQS, MIN(ctz64(pgsize) - 4, val));
 }
 
 static void tegra241_cmdqv_reset(SMMUv3State *s)
Signature · 1 line
2.43.0
Reviewed-by on v6 25/31
#42 lore raw

Re: [PATCH v6 25/31] hw/arm/tegra241-cmdqv: Limit queue size based on backend page size

Quoted 14 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

CMDQV HW performs DMA accesses to guest queue memory by its host physical address set up via IOMMUFD. This requires the guest queue to be contiguous in both guest PA and host PA space. With Tegra241 CMDQV enabled, we must only advertise a command queue size (CMDQS) that the host can safely back with physically contiguous memory. Allowing a queue size larger than the host page size could cause the hardware to DMA across page boundaries, leading to faults.

Use qemu_minrampagesize() to find the smallest memory-backend page size in use, then cap IDR1.CMDQS so the guest cannot configure a command queue that exceeds that contiguous backing.

Quoted 8 lines

Note this is done at SMMUv3 init, before any guest queue GPA is known, so the cap is conservative. Maximum queue size is 8MiB; it is recommended to back the VM with hugepage sizes large enough so CMDQS stays at the HW maximum. Smaller backing pages reduce CMDQS accordingly.

Eric

Quoted 49 lines
---
 hw/arm/tegra241-cmdqv.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
hw/arm/tegra241-cmdqv.c +20 -0
index ac17b56c1c..661c46e3d9 100644
@@ -16,6 +16,8 @@
 #include "hw/arm/smmuv3-common.h"
 #include "hw/core/irq.h"
 #include "smmuv3-accel.h"
+#include "smmuv3-internal.h"
+#include "system/hostmem.h"
 #include "tegra241-cmdqv.h"
 #include "trace.h"
 
@@ -874,6 +876,8 @@ free_viommu:
 static void tegra241_cmdqv_init_regs(SMMUv3State *s, Tegra241CMDQV *cmdqv)
 {
     int i;
+    long pgsize;
+    uint32_t val;
 
     cmdqv->config = V_CONFIG_RESET;
     cmdqv->param = FIELD_DP32(0, PARAM, CMDQV_VER, CMDQV_VER);
@@ -905,6 +909,22 @@ static void tegra241_cmdqv_init_regs(SMMUv3State *s, Tegra241CMDQV *cmdqv)
         cmdqv->vcmdq_base[i] = 0;
         cmdqv->vcmdq_cons_indx_base[i] = 0;
     }
+
+    /*
+     * CMDQ must not cross a physical RAM backend page. Adjust CMDQS so the
+     * queue fits entirely within the smallest backend page size, ensuring
+     * the command queue is physically contiguous in host memory.
+     *
+     *   IDR1.CMDQS = log2(max_qsz) - entry_shift
+     *
+     * where entry_shift = 4 (each CMDQ entry is 16 bytes = 2^4).
+     */
+    pgsize = qemu_minrampagesize();
+    if (pgsize == LONG_MAX) {
+        pgsize = qemu_real_host_page_size();
+    }
+    val = FIELD_EX32(s->idr[1], IDR1, CMDQS);
+    s->idr[1] = FIELD_DP32(s->idr[1], IDR1, CMDQS, MIN(ctz64(pgsize) - 4, val));
 }
 
 static void tegra241_cmdqv_reset(SMMUv3State *s)
#43 lore raw

PATCH v6 26/31 hw/arm/smmuv3: Add per-device identifier property

In reply to Shameer Kolothum

Add an "identifier" property to the SMMUv3 device and use it when building the ACPI IORT SMMUv3 node Identifier field.

This avoids relying on device enumeration order and provides a stable per-device identifier. A subsequent patch will use the same identifier when generating the DSDT description for Tegra241 CMDQV, ensuring that the IORT and DSDT entries refer to the same SMMUv3 instance.

The identifier is assigned at pre-plug time, accounting for the ITS Group node that build_iort() places before SMMUv3 nodes in the IORT table, so that identifiers are globally unique across all IORT nodes.

No functional change: IORT blob content for bios-tables qtest is identical to before.

---
 include/hw/arm/smmuv3.h  |  1 +
 hw/arm/smmuv3.c          |  2 ++
 hw/arm/virt-acpi-build.c |  5 ++++-
 hw/arm/virt.c            | 12 ++++++++++++
 4 files changed, 19 insertions(+), 1 deletion(-)
include/hw/arm/smmuv3.h +1 -0
index 34d0f65eaa..d39fe8850b 100644
@@ -65,6 +65,7 @@ struct SMMUv3State {
     qemu_irq     irq[4];
     QemuMutex mutex;
     char *stage;
+    uint8_t identifier;
 
     /* SMMU has HW accelerator support for nested S1 + s2 */
     bool accel;
hw/arm/smmuv3.c +2 -0
index 4e630b6403..0d111d638b 100644
@@ -2126,6 +2126,8 @@ static const Property smmuv3_properties[] = {
      * Defaults to stage 1
      */
     DEFINE_PROP_STRING("stage", SMMUv3State, stage),
+    /* Identifier used for ACPI IORT SMMUv3 (and DSDT for CMDQV) generation */
+    DEFINE_PROP_UINT8("identifier", SMMUv3State, identifier, 0),
     DEFINE_PROP_BOOL("accel", SMMUv3State, accel, false),
     /* GPA of MSI doorbell, for SMMUv3 accel use. */
     DEFINE_PROP_UINT64("msi-gpa", SMMUv3State, msi_gpa, 0),
hw/arm/virt-acpi-build.c +4 -1
index b00f3477ca..9d05982137 100644
@@ -349,6 +349,7 @@ static int iort_idmap_compare(gconstpointer a, gconstpointer b)
 typedef struct AcpiIortSMMUv3Dev {
     int irq;
     hwaddr base;
+    uint8_t id;
     GArray *rc_smmu_idmaps;
     /* Offset of the SMMUv3 IORT Node relative to the start of the IORT */
     size_t offset;
@@ -411,6 +412,7 @@ static int populate_smmuv3_dev(VirtMachineState *vms, GArray *sdev_blob)
                                                &error_abort));
         sdev.accel = object_property_get_bool(obj, "accel", &error_abort);
         sdev.ats = smmuv3_ats_enabled(ARM_SMMUV3(obj));
+        sdev.id = object_property_get_uint(obj, "identifier", &error_abort);
         pbus = PLATFORM_BUS_DEVICE(vms->platform_bus_dev);
         sbdev = SYS_BUS_DEVICE(obj);
         sdev.base = platform_bus_get_mmio_addr(pbus, sbdev, 0);
@@ -637,7 +639,8 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
                      (ID_MAPPING_ENTRY_SIZE * smmu_mapping_count);
         build_append_int_noprefix(table_data, node_size, 2); /* Length */
         build_append_int_noprefix(table_data, 4, 1); /* Revision */
-        build_append_int_noprefix(table_data, id++, 4); /* Identifier */
+        build_append_int_noprefix(table_data, sdev->id, 4); /* Identifier */
+        id++;  /* advance shared counter for RC/RMR node uniqueness */
         /* Number of ID mappings */
         build_append_int_noprefix(table_data, smmu_mapping_count, 4);
         /* Reference to ID Array */
hw/arm/virt.c +12 -0
index 2add7401a1..d8d27f2ef6 100644
@@ -254,6 +254,9 @@ static MemMapEntry extended_memmap[] = {
     /* Any CXL Fixed memory windows come here */
 };
 
+/* Counts SMMUv3 devices plugged; used to assign stable IORT identifiers */
+static uint8_t smmuv3_dev_id;
+
 static const int a15irqmap[] = {
     [VIRT_UART0] = 1,
     [VIRT_RTC] = 2,
@@ -3830,6 +3833,15 @@ static void virt_machine_device_pre_plug_cb(HotplugHandler *hotplug_dev,
                                      OBJECT(vms->sysmem), NULL);
             object_property_set_link(OBJECT(dev), "secure-memory",
                                      OBJECT(vms->secure_sysmem), NULL);
+            /*
+             * In build_iort(), the ITS node(id=0) precedes SMMUv3 nodes
+             * when present. Account for it so this SMMUv3's identifier
+             * is globally unique across all IORT nodes.
+             */
+            uint8_t its_offset = (vms->msi_controller == VIRT_MSI_CTRL_ITS)
+                                  ? 1 : 0;
+            object_property_set_uint(OBJECT(dev), "identifier",
+                                     its_offset + smmuv3_dev_id++, NULL);
         }
         if (object_property_get_bool(OBJECT(dev), "accel", &error_abort)) {
             hwaddr db_start = 0;
Signature · 1 line
2.43.0
#44 lore raw

PATCH v6 27/31 hw/arm/smmuv3-accel: Introduce helper to query CMDQV type

In reply to Shameer Kolothum

Introduce a SMMUv3AccelCmdqvType enum and a helper to query the CMDQV implementation type associated with an accelerated SMMUv3 instance.

A subsequent patch will use this helper when generating the Tegra241 CMDQV DSDT.

---
 hw/arm/smmuv3-accel.h       | 10 ++++++++++
 hw/arm/smmuv3-accel-stubs.c |  5 +++++
 hw/arm/smmuv3-accel.c       | 12 ++++++++++++
 hw/arm/tegra241-cmdqv.c     |  6 ++++++
 4 files changed, 33 insertions(+)
hw/arm/smmuv3-accel.h +10 -0
index 8d0f636338..5fc85fb89d 100644
@@ -16,6 +16,11 @@
 #include <linux/iommufd.h>
 #endif
 
+typedef enum SMMUv3AccelCmdqvType {
+    SMMUV3_CMDQV_NONE = 0,
+    SMMUV3_CMDQV_TEGRA241,
+} SMMUv3AccelCmdqvType;
+
 /*
  * CMDQ-Virtualization (CMDQV) hardware support, extends the SMMUv3 to
  * support multiple VCMDQs with virtualization capabilities.
@@ -42,6 +47,10 @@ typedef struct SMMUv3AccelCmdqvOps {
      * If NULL, the viommu_id is freed directly via iommufd_backend_free_id().
      */
     void (*free_viommu)(SMMUv3State *s);
+    /**
+     * @get_type: Optional callback. Return the CMDQV implementation type.
+     */
+    SMMUv3AccelCmdqvType (*get_type)(void);
     /**
      * @reset: Optional callback. Reset CMDQV state.
      */
@@ -90,5 +99,6 @@ bool smmuv3_accel_alloc_veventq(SMMUv3State *s, Error **errp);
 bool smmuv3_accel_event_read_validate(IOMMUFDVeventq *veventq, uint32_t type,
                                       void *buf, size_t size, Error **errp);
 void smmuv3_accel_reset(SMMUv3State *s);
+SMMUv3AccelCmdqvType smmuv3_accel_cmdqv_type(Object *obj);
 
 #endif /* HW_ARM_SMMUV3_ACCEL_H */
hw/arm/smmuv3-accel-stubs.c +5 -0
index 9e6c44a282..147ae06163 100644
@@ -57,3 +57,8 @@ bool smmuv3_accel_event_read_validate(IOMMUFDVeventq *veventq, uint32_t type,
 void smmuv3_accel_reset(SMMUv3State *s)
 {
 }
+
+SMMUv3AccelCmdqvType smmuv3_accel_cmdqv_type(Object *obj)
+{
+    return SMMUV3_CMDQV_NONE;
+}
hw/arm/smmuv3-accel.c +12 -0
index c1c2a67c97..9c3bd4413d 100644
@@ -1060,6 +1060,18 @@ static void smmuv3_accel_as_init(SMMUv3State *s)
     address_space_init(shared_as_sysmem, &root, "smmuv3-accel-as-sysmem");
 }
 
+SMMUv3AccelCmdqvType smmuv3_accel_cmdqv_type(Object *obj)
+{
+    SMMUv3State *s = ARM_SMMUV3(obj);
+    SMMUv3AccelState *accel = s->s_accel;
+
+    if (!accel || !accel->cmdqv_ops || !accel->cmdqv_ops->get_type) {
+        return SMMUV3_CMDQV_NONE;
+    }
+
+    return accel->cmdqv_ops->get_type();
+}
+
 static void smmuv3_accel_machine_done(Notifier *notifier, void *data)
 {
     SMMUv3State *s = container_of(notifier, SMMUv3State, machine_done);
hw/arm/tegra241-cmdqv.c +6 -0
index 661c46e3d9..4dba1783fa 100644
@@ -974,6 +974,11 @@ static bool tegra241_cmdqv_init(SMMUv3State *s, Error **errp)
     return true;
 }
 
+static SMMUv3AccelCmdqvType tegra241_cmdqv_get_type(void)
+{
+    return SMMUV3_CMDQV_TEGRA241;
+}
+
 static bool tegra241_cmdqv_probe(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
                                  Error **errp)
 {
@@ -1014,6 +1019,7 @@ static const SMMUv3AccelCmdqvOps tegra241_cmdqv_ops = {
     .init = tegra241_cmdqv_init,
     .alloc_viommu = tegra241_cmdqv_alloc_viommu,
     .free_viommu = tegra241_cmdqv_free_viommu,
+    .get_type = tegra241_cmdqv_get_type,
     .reset = tegra241_cmdqv_reset,
 };
 
Signature · 1 line
2.43.0
#45 lore raw

PATCH v6 28/31 hw/arm/virt-acpi: Advertise Tegra241 CMDQV nodes in DSDT

In reply to Shameer Kolothum

From: Nicolin Chen <[email protected]>

Add ACPI DSDT support for Tegra241 CMDQV when the SMMUv3 instance is created with tegra241-cmdqv.

For each accelerated SMMUv3 instance, add a Tegra241 CMDQV device object under the DSDT \_SB namespace, with HID "NVDA200C" and a UID that matches the Identifier of the corresponding SMMUv3 IORT node, so the guest OS can associate the DSDT device with the right SMMU. The _CRS covers the CMDQV MMIO aperture plus its interrupt, and _CCA declares I/O cache coherency.

See ACPI Specification 6.5, Section 6 (Device Configuration) for _HID/_UID/_CCA/_CRS.

Generated DSDT entry for a CMDQV instance paired with SMMUv3 Identifier=1:
...
Device (CV01)

  {
      Name (_HID, "NVDA200C")  // _HID: Hardware ID
      Name (_UID, One)  // _UID: Unique ID
      Name (_CCA, One)  // _CCA: Cache Coherency Attribute
      Name (_CRS, ResourceTemplate ()  // _CRS: Current Resource Settings
      {
          QWordMemory (ResourceProducer, PosDecode, MinFixed, MaxFixed, Cacheable, ReadWrite,
              0x0000000000000000, // Granularity
              0x000000000C080000, // Range Minimum
              0x000000000C0CFFFF, // Range Maximum
              0x0000000000000000, // Translation Offset
              0x0000000000050000, // Length
              ,, , AddressRangeMemory, TypeStatic)
          Interrupt (ResourceConsumer, Edge, ActiveHigh, Exclusive, ,, )
          {
              0x00000094,
          }
      })

}
...
Generated IORT SMMUv3 node (Identifier = 1):

...
[048h 0072 001h] Type : 04
[049h 0073 002h] Length : 0058
[04Bh 0075 001h] Revision : 04
[04Ch 0076 004h] Identifier : 00000001
[050h 0080 004h] Mapping Count : 00000001
[054h 0084 004h] Mapping Offset : 00000044
...

---
 hw/arm/virt-acpi-build.c | 52 ++++++++++++++++++++++++++++++++++++++++
 hw/arm/trace-events      |  1 +
 2 files changed, 53 insertions(+)
hw/arm/virt-acpi-build.c +52 -0
index 9d05982137..99490aa7b1 100644
@@ -65,6 +65,9 @@
 #include "target/arm/cpu.h"
 #include "target/arm/multiprocessing.h"
 
+#include "smmuv3-accel.h"
+#include "tegra241-cmdqv.h"
+
 #define ARM_SPI_BASE 32
 
 #define ACPI_BUILD_TABLE_SIZE             0x20000
@@ -1121,6 +1124,51 @@ static void build_fadt_rev6(GArray *table_data, BIOSLinker *linker,
     build_fadt(table_data, linker, &fadt, vms->oem_id, vms->oem_table_id);
 }
 
+static void acpi_dsdt_add_tegra241_cmdqv(Aml *scope, VirtMachineState *vms)
+{
+    for (int i = 0; i < vms->smmuv3_devices->len; i++) {
+        Object *obj = OBJECT(g_ptr_array_index(vms->smmuv3_devices, i));
+        PlatformBusDevice *pbus;
+        Aml *dev, *crs, *addr;
+        SysBusDevice *sbdev;
+        hwaddr base;
+        uint32_t id;
+        int irq;
+
+        if (smmuv3_accel_cmdqv_type(obj) != SMMUV3_CMDQV_TEGRA241) {
+            continue;
+        }
+        id = object_property_get_uint(obj, "identifier", &error_abort);
+        pbus = PLATFORM_BUS_DEVICE(vms->platform_bus_dev);
+        sbdev = SYS_BUS_DEVICE(obj);
+        base = platform_bus_get_mmio_addr(pbus, sbdev, 1);
+        base += vms->memmap[VIRT_PLATFORM_BUS].base;
+        irq = platform_bus_get_irqn(pbus, sbdev, NUM_SMMU_IRQS);
+        irq += vms->irqmap[VIRT_PLATFORM_BUS];
+        irq += ARM_SPI_BASE;
+
+        dev = aml_device("CV%.02u", id);
+        aml_append(dev, aml_name_decl("_HID", aml_string("NVDA200C")));
+        aml_append(dev, aml_name_decl("_UID", aml_int(id)));
+        aml_append(dev, aml_name_decl("_CCA", aml_int(1)));
+
+        crs = aml_resource_template();
+        addr = aml_qword_memory(AML_POS_DECODE, AML_MIN_FIXED, AML_MAX_FIXED,
+                                AML_CACHEABLE, AML_READ_WRITE, 0x0, base,
+                                base + TEGRA241_CMDQV_IO_LEN - 0x1, 0x0,
+                                TEGRA241_CMDQV_IO_LEN);
+        aml_append(crs, addr);
+        aml_append(crs, aml_interrupt(AML_CONSUMER, AML_EDGE,
+                                      AML_ACTIVE_HIGH, AML_EXCLUSIVE,
+                                      (uint32_t *)&irq, 1));
+        aml_append(dev, aml_name_decl("_CRS", crs));
+
+        aml_append(scope, dev);
+
+        trace_virt_acpi_dsdt_tegra241_cmdqv(id, base, irq);
+    }
+}
+
 /* DSDT */
 static void
 build_dsdt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
@@ -1185,6 +1233,10 @@ build_dsdt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
     acpi_dsdt_add_tpm(scope, vms);
 #endif
 
+    if (!vms->legacy_smmuv3_present) {
+        acpi_dsdt_add_tegra241_cmdqv(scope, vms);
+    }
+
     aml_append(dsdt, scope);
 
     pci0_scope = aml_scope("\\_SB.PCI0");
hw/arm/trace-events +1 -0
index e655b2e26e..1d5d8d1f6c 100644
@@ -9,6 +9,7 @@ omap1_lpg_led(const char *onoff) "omap1 LPG: LED is %s"
 
 # virt-acpi-build.c
 virt_acpi_setup(void) "No fw cfg or ACPI disabled. Bailing out."
+virt_acpi_dsdt_tegra241_cmdqv(int smmu_id, uint64_t base, uint32_t irq) "DSDT: add cmdqv node for (id=%d), base=0x%" PRIx64 ", irq=%d"
 
 # smmu-common.c
 smmu_add_mr(const char *name) "%s"
Signature · 1 line
2.43.0
#46 lore raw

PATCH v6 29/31 hw/arm/smmuv3-accel: Enforce viommu association when CMDQV is active

In reply to Shameer Kolothum

When CMDQV is active, the first cold-plugged VFIO device establishes the viommu to host SMMUv3 association, and the guest's boot-time CMDQV configuration (VINTFs, VCMDQs) is built on top of that association.

Hot-unplugging that device would release the viommu and tear down all CMDQV state. Hot-plugging another device behind a different host SMMUv3+CMDQV would then re-bind the same vSMMUv3 to new host hardware, while the guest keeps using its boot-time configuration and ends up issuing commands to the wrong host. Block hot-unplug of the establishing device to avoid this; retaining the binding across unplug is non-trivial and not required by any current use case.

Also abort at machine_done if cmdqv=on is requested but no cold-plugged VFIO device was present to initialize it.

---
 hw/arm/smmuv3-accel.h |  1 +
 hw/arm/smmuv3-accel.c | 18 ++++++++++++++++++
 2 files changed, 19 insertions(+)
hw/arm/smmuv3-accel.h +1 -0
index 5fc85fb89d..dd755c394d 100644
@@ -84,6 +84,7 @@ typedef struct SMMUv3AccelDevice {
     IOMMUFDVdev *vdev;
     QLIST_ENTRY(SMMUv3AccelDevice) next;
     SMMUv3AccelState *s_accel;
+    Error *unplug_blocker; /* set when CMDQV is active to block hot-unplug */
 } SMMUv3AccelDevice;
 
 bool smmuv3_accel_init(SMMUv3State *s, Error **errp);
hw/arm/smmuv3-accel.c +18 -0
index 9c3bd4413d..80900c2521 100644
@@ -759,6 +759,18 @@ static bool smmuv3_accel_set_iommu_device(PCIBus *bus, void *opaque, int devfn,
         return false;
     }
 
+    /*
+     * CMDQV is active: block hot-unplug of the device that established the
+     * viommu association. Removing it would cause the vIOMMU to host SMMUv3
+     * association be changed via device hot-plug.
+     */
+    if (s->s_accel->cmdqv_ops) {
+        PCIDevice *pdev = pci_find_device(bus, pci_bus_num(bus), devfn);
+        error_setg(&accel_dev->unplug_blocker,
+                   "CMDQV is active: removing the device that established the "
+                   "viommu association would break the guest CMDQV");
+        qdev_add_unplug_blocker(DEVICE(pdev), accel_dev->unplug_blocker);
+    }
 done:
     accel_dev->hiodi = hiodi;
     accel_dev->s_accel = s->s_accel;
@@ -1082,6 +1094,12 @@ static void smmuv3_accel_machine_done(Notifier *notifier, void *data)
                      "at least one cold-plugged VFIO device");
         exit(1);
     }
+
+    if (s->cmdqv == ON_OFF_AUTO_ON && !accel->cmdqv) {
+        error_report("arm-smmuv3 cmdqv=on requires at least one cold-plugged "
+                     "VFIO device");
+        exit(1);
+    }
 }
 
 bool smmuv3_accel_init(SMMUv3State *s, Error **errp)
Signature · 1 line
2.43.0
Reviewed-by on v6 29/31
#47 lore raw

Re: [PATCH v6 29/31] hw/arm/smmuv3-accel: Enforce viommu association when CMDQV is active

Quoted 12 lines · Shameer Kolothum

When CMDQV is active, the first cold-plugged VFIO device establishes the viommu to host SMMUv3 association, and the guest's boot-time CMDQV configuration (VINTFs, VCMDQs) is built on top of that association.

Hot-unplugging that device would release the viommu and tear down all CMDQV state. Hot-plugging another device behind a different host SMMUv3+CMDQV would then re-bind the same vSMMUv3 to new host hardware, while the guest keeps using its boot-time configuration and ends up issuing commands to the wrong host. Block hot-unplug of the establishing device to avoid this; retaining the binding across unplug is non-trivial and not required by any current use case.

Quoted 5 lines

Also abort at machine_done if cmdqv=on is requested but no cold-plugged VFIO device was present to initialize it.

Eric

Quoted 53 lines
---
 hw/arm/smmuv3-accel.h |  1 +
 hw/arm/smmuv3-accel.c | 18 ++++++++++++++++++
 2 files changed, 19 insertions(+)
hw/arm/smmuv3-accel.h +1 -0
index 5fc85fb89d..dd755c394d 100644
@@ -84,6 +84,7 @@ typedef struct SMMUv3AccelDevice {
     IOMMUFDVdev *vdev;
     QLIST_ENTRY(SMMUv3AccelDevice) next;
     SMMUv3AccelState *s_accel;
+    Error *unplug_blocker; /* set when CMDQV is active to block hot-unplug */
 } SMMUv3AccelDevice;
 
 bool smmuv3_accel_init(SMMUv3State *s, Error **errp);
hw/arm/smmuv3-accel.c +18 -0
index 9c3bd4413d..80900c2521 100644
@@ -759,6 +759,18 @@ static bool smmuv3_accel_set_iommu_device(PCIBus *bus, void *opaque, int devfn,
         return false;
     }
 
+    /*
+     * CMDQV is active: block hot-unplug of the device that established the
+     * viommu association. Removing it would cause the vIOMMU to host SMMUv3
+     * association be changed via device hot-plug.
+     */
+    if (s->s_accel->cmdqv_ops) {
+        PCIDevice *pdev = pci_find_device(bus, pci_bus_num(bus), devfn);
+        error_setg(&accel_dev->unplug_blocker,
+                   "CMDQV is active: removing the device that established the "
+                   "viommu association would break the guest CMDQV");
+        qdev_add_unplug_blocker(DEVICE(pdev), accel_dev->unplug_blocker);
+    }
 done:
     accel_dev->hiodi = hiodi;
     accel_dev->s_accel = s->s_accel;
@@ -1082,6 +1094,12 @@ static void smmuv3_accel_machine_done(Notifier *notifier, void *data)
                      "at least one cold-plugged VFIO device");
         exit(1);
     }
+
+    if (s->cmdqv == ON_OFF_AUTO_ON && !accel->cmdqv) {
+        error_report("arm-smmuv3 cmdqv=on requires at least one cold-plugged "
+                     "VFIO device");
+        exit(1);
+    }
 }
 
 bool smmuv3_accel_init(SMMUv3State *s, Error **errp)
#48 lore raw

PATCH v6 30/31 hw/arm/tegra241-cmdqv: Document the CMDQV design and lifecycle

In reply to Shameer Kolothum

Add an overview describing the Tegra241 CMDQV passthrough model, MMIO layout, guest-driven lifecycle, and per-VM isolation.

---
 hw/arm/tegra241-cmdqv.c | 93 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 93 insertions(+)
hw/arm/tegra241-cmdqv.c +93 -0
index 4dba1783fa..5dbf68b421 100644
@@ -7,6 +7,99 @@
  * SPDX-License-Identifier: GPL-2.0-or-later
  */
 
+/*
+ * Tegra241 CMDQV - overview
+ * =========================
+ *
+ * NVIDIA Tegra241 extends SMMUv3 with a Command Queue Virtualization (CMDQ-V)
+ * block. It lets a guest issue SMMU invalidation commands directly to
+ * dedicated hardware queues (vCMDQs) without trapping into the hypervisor on
+ * the fast path. vCMDQs are grouped into Virtual Interfaces (VINTFs); the
+ * host kernel allocates one VINTF per emulated SMMUv3 instance via iommufd.
+ * QEMU emulates the CMDQV MMIO region and drives the host kernel calls
+ * (VIOMMU_ALLOC, HW_QUEUE_ALLOC, mmap); the actual command processing
+ * happens on real hardware.
+ *
+ * A vCMDQ becomes functional only once allocated to a VINTF; until then,
+ * register accesses only touch the QEMU-side cache and no command
+ * processing happens. After allocation, command processing runs on the
+ * host hardware, and guest accesses to the live control/status registers
+ * bypass QEMU and reach the host directly.
+ *
+ * MMIO layout (64KB pages, total TEGRA241_CMDQV_IO_LEN)
+ * -----------------------------------------------------
+ * The direct vCMDQ apertures (0x10000/0x20000) are HW aliases of the VINTF
+ * apertures (0x30000/0x40000); they expose the same per-vCMDQ register slots
+ * under different addressing.
+ *
+ *   0x00000  CMDQV Config page: QEMU-trapped.
+ *   0x10000  Direct vCMDQ Page 0 (control/status): QEMU-trapped and routed
+ *            to either the mmap'd VINTF Page 0 (if the vCMDQ has been
+ *            allocated to a VINTF) or a per-vCMDQ register cache (otherwise).
+ *   0x20000  Direct vCMDQ Page 1 (BASE / DRAM addresses): QEMU-trapped.
+ *   0x30000  VINTF Page 0 (per-VINTF control/status): mmap'd from the host
+ *            via iommufd and installed into guest MMIO as a RAM-device
+ *            subregion after the first HW_QUEUE_ALLOC; subsequent accesses
+ *            bypass QEMU.
+ *   0x40000  VINTF Page 1 (per-VINTF BASE): QEMU-trapped. Although this is
+ *            a HW alias of the direct Page 1, the kernel only exposes mmap
+ *            for VINTF Page 0; VINTF Page 1 is not mmap'd and stays trapped.
+ *
+ * The direct vCMDQ Page 0 stays trapped rather than aliased to the VINTF
+ * Page 0 mmap. Direct aperture read/write is not an expected common usecase,
+ * but trapping keeps accesses well-defined for an unallocated vCMDQ.
+ *
+ * Lifecycle (driven by guest events)
+ * ----------------------------------
+ * 1. First vfio-pci device attach (.set_iommu_device) triggers:
+ *    - tegra241_cmdqv_probe(): IOMMU_GET_HW_INFO confirms host CMDQV support.
+ *    - IOMMU_VIOMMU_ALLOC: the kernel allocates a VINTF for this VM,
+ *      configures the VM's VMID (from its stage-2 HWPT) in VINTF_CONFIG,
+ *      forces HYP_OWN=0, and returns the mmap offset/length for VINTF Page 0.
+ *
+ * 2. Guest writes VINTF_CONFIG.ENABLE = 1:
+ *    QEMU mmap()s the offset from step 1 into its address space and reports
+ *    STATUS.ENABLE_OK = 1. The host VINTF was already enabled by
+ *    IOMMU_VIOMMU_ALLOC; QEMU only acks back.
+ *
+ * 3. Guest completes vCMDQ setup (BASE, CMDQ_ALLOC_MAP.ALLOC, CMDQV_EN,
+ *    VINTF.ENABLE, in any order; each precondition write retries the
+ *    allocation):
+ *    IOMMU_HW_QUEUE_ALLOC binds the guest BASE GPA (translated through
+ *    stage-2 and pinned by the kernel) to a host vCMDQ in this VM's VINTF.
+ *
+ * 4. After the first successful HW_QUEUE_ALLOC, the mmap'd VINTF Page 0 is
+ *    installed into guest MMIO as a RAM-device subregion. Guest VINTF Page 0
+ *    accesses (CMDQ_EN, PROD/CONS_INDX, STATUS, GERROR/GERRORN) thereafter
+ *    go straight to host hardware, bypassing QEMU.
+ *
+ * 5. Guest SMMU driver programs a Stream Table Entry for a passthrough
+ *    device: IOMMU_VDEVICE_ALLOC programs SID_MATCH/SID_REPLACE in this
+ *    VM's VINTF so the device's guest vSID translates to its host pSID.
+ *    Commands referencing unmapped SIDs are rejected by HW.
+ *
+ * Per-VM isolation
+ * ----------------
+ * - Each VM has its own iommufd FD; all iommufd objects (VINTF, vdevices,
+ *   hw_queues, mmap regions) belong to that FD. Cross-FD lookups fail, so
+ *   one VM cannot reach another VM's IDs.
+ * - IOMMU_VIOMMU_ALLOC configures the VM's VMID in VINTF_CONFIG; the CMDQV
+ *   hardware substitutes / checks VMID on every command the guest issues.
+ * - The kernel allocates the VINTF with HYP_OWN = 0, which restricts the
+ *   guest to a safe subset of commands.
+ * - IOMMU_VDEVICE_ALLOC populates SID_MATCH/SID_REPLACE so invalidations
+ *   only reach the host StreamIDs assigned to this VM (see step 5).
+ * - IOMMU_HW_QUEUE_ALLOC binds each vCMDQ to a single VINTF, so a guest
+ *   cannot reach a vCMDQ that belongs to another VM.
+ *
+ * Limits exposed to the guest
+ * ---------------------------
+ * One VINTF per emulated SMMUv3 and two vCMDQs per VINTF. Maximum vCMDQ
+ * size is 8MiB. The queue must be physically contiguous (the HW reads it
+ * via host PA), so QEMU caps it to the host memory-backend page size. Use
+ * hugepage backing large enough to keep CMDQS at the HW maximum.
+ */
+
 #include "qemu/osdep.h"
 #include "qemu/log.h"
 #include "qemu/error-report.h"
Signature · 1 line
2.43.0
#49 lore raw

Re: [PATCH v6 30/31] hw/arm/tegra241-cmdqv: Document the CMDQV design and lifecycle

Quoted 9 lines · Shameer Kolothum

+ * Lifecycle (driven by guest events)
+ * ----------------------------------
+ * 1. First vfio-pci device attach (.set_iommu_device) triggers:
+ * - tegra241_cmdqv_probe(): IOMMU_GET_HW_INFO confirms host CMDQV support.
+ * - IOMMU_VIOMMU_ALLOC: the kernel allocates a VINTF for this VM,
+ * configures the VM's VMID (from its stage-2 HWPT) in VINTF_CONFIG,
+ * forces HYP_OWN=0, and returns the mmap offset/length for VINTF Page 0.
+ *
+ * 2. Guest writes VINTF_CONFIG.ENABLE = 1:

Quoted 3 lines
+ *    QEMU mmap()s the offset from step 1 into its address space and reports
+ *    STATUS.ENABLE_OK = 1. The host VINTF was already enabled by
+ *    IOMMU_VIOMMU_ALLOC; QEMU only acks back.

I wonder this mmap should be done after the CMDQV_CONFIG.CMDQV_EN=1 rather than VINTF_CONFIG.ENABLE=1.

Spec says that:
"Program the CMDQV_CMDQ_ALLOC_MAP_X register(s) to map the Virtual
CMDQ(s) to the logical CMDQs on Virtual Interface following the CMDQ allocation rules. This again is an optional step and the CMDQ allocation can be done after the Virtual Interface is initialized."

This means the LVCMDQ mapping can happen before VINTF_CONFIG.ENABLE=1.

And the mmap info is returned with IOMMU_VIOMMU_ALLOC. So, this is doable?

FWIW, kernel finalizes a LVCMDQ mapping when handling HW_QUEUE_ALLOC.

Quoted 10 lines

+ * 3. Guest completes vCMDQ setup (BASE, CMDQ_ALLOC_MAP.ALLOC, CMDQV_EN,

+ *    VINTF.ENABLE, in any order; each precondition write retries the
+ *    allocation):
+ *    IOMMU_HW_QUEUE_ALLOC binds the guest BASE GPA (translated through
+ *    stage-2 and pinned by the kernel) to a host vCMDQ in this VM's VINTF.
+ *
+ * 4. After the first successful HW_QUEUE_ALLOC, the mmap'd VINTF Page 0 is
+ *    installed into guest MMIO as a RAM-device subregion. Guest VINTF Page 0
+ *    accesses (CMDQ_EN, PROD/CONS_INDX, STATUS, GERROR/GERRORN) thereafter
+ *    go straight to host hardware, bypassing QEMU.
Quoted 3 lines

+ *
+ * 5. Guest SMMU driver programs a Stream Table Entry for a passthrough
+ * device: IOMMU_VDEVICE_ALLOC programs SID_MATCH/SID_REPLACE in this

Ideally, the VDEVICE_ALLOC should happen right after the device gets plugged *and* its vSID is available; similarly the vdevice should be only freed when the device gets unplugged. So, its lifecycle should stay with the plugged state.

We do VDEVICE_ALLOC when guest programming STE is because that's the only place we found in the existing functions. We could (or arguably maybe we should) find a better place to allocate VDEVICE. E.g., vfio code can call qemu_add_vm_change_state_handler_prio() to register a callback function, where vSID (vBDF) should be available once the VM start to be "running".

I think your narrative fits the current design. But we might need to be more accurate here, since this is about "lifecycle".

Quoted 1 line

+ * VM's VINTF so the device's guest vSID translates to its host pSID.

Nit: so that the HW translates the guest vSID into host pSID.

Quoted 6 lines

+ * Limits exposed to the guest
+ * ---------------------------
+ * One VINTF per emulated SMMUv3 and two vCMDQs per VINTF. Maximum vCMDQ
+ * size is 8MiB. The queue must be physically contiguous (the HW reads it
+ * via host PA), so QEMU caps it to the host memory-backend page size. Use
+ * hugepage backing large enough to keep CMDQS at the HW maximum.

The "physically contiguous" limit is handled by QEMU, not exposed to the guest at all; guest is unaware of this.

Nicolin

#50 lore raw

RE: [PATCH v6 30/31] hw/arm/tegra241-cmdqv: Document the CMDQV design and lifecycle

Quoted 42 lines

-----Original Message-----
From: Nicolin Chen <[email protected]>
Sent: 02 June 2026 00:47
To: Shameer Kolothum Thodi <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jiandi An <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; Krishnakant Jaju <[email protected]>; [email protected]
Subject: Re: [PATCH v6 30/31] hw/arm/tegra241-cmdqv: Document the CMDQV design and lifecycle

Quoted 4 lines · Shameer Kolothum

+ * Lifecycle (driven by guest events)
+ * ----------------------------------
+ * 1. First vfio-pci device attach (.set_iommu_device) triggers:
+ * - tegra241_cmdqv_probe(): IOMMU_GET_HW_INFO confirms host

CMDQV support.

Quoted 3 lines

+ * - IOMMU_VIOMMU_ALLOC: the kernel allocates a VINTF for this VM,
+ * configures the VM's VMID (from its stage-2 HWPT) in VINTF_CONFIG,
+ * forces HYP_OWN=0, and returns the mmap offset/length for VINTF

Page 0.

Quoted 3 lines

+ *
+ * 2. Guest writes VINTF_CONFIG.ENABLE = 1:
+ * QEMU mmap()s the offset from step 1 into its address space and

reports

Quoted 2 lines

+ * STATUS.ENABLE_OK = 1. The host VINTF was already enabled by
+ * IOMMU_VIOMMU_ALLOC; QEMU only acks back.

I wonder this mmap should be done after the
CMDQV_CONFIG.CMDQV_EN=1
rather than VINTF_CONFIG.ENABLE=1.

Spec says that:
"Program the CMDQV_CMDQ_ALLOC_MAP_X register(s) to map the Virtual
CMDQ(s) to the logical CMDQs on Virtual Interface following the CMDQ allocation rules. This again is an optional step and the CMDQ allocation can be done after the Virtual Interface is initialized."

Quoted 5 lines

This means the LVCMDQ mapping can happen before
VINTF_CONFIG.ENABLE=1.

And the mmap info is returned with IOMMU_VIOMMU_ALLOC. So, this is doable?

I think it is. We could in theory as well do it just after IOMMU_VIOMMU_ALLOC. What you think?

Quoted 36 lines

FWIW, kernel finalizes a LVCMDQ mapping when handling HW_QUEUE_ALLOC.

Quoted 1 line

+ * 3. Guest completes vCMDQ setup (BASE, CMDQ_ALLOC_MAP.ALLOC,

CMDQV_EN,

Quoted 3 lines
+ *    VINTF.ENABLE, in any order; each precondition write retries the
+ *    allocation):
+ *    IOMMU_HW_QUEUE_ALLOC binds the guest BASE GPA (translated

through

Quoted 3 lines

+ * stage-2 and pinned by the kernel) to a host vCMDQ in this VM's VINTF.
+ *
+ * 4. After the first successful HW_QUEUE_ALLOC, the mmap'd VINTF Page

0 is

Quoted 1 line

+ * installed into guest MMIO as a RAM-device subregion. Guest VINTF

Page 0

Quoted 1 line

+ * accesses (CMDQ_EN, PROD/CONS_INDX, STATUS, GERROR/GERRORN)

thereafter

Quoted 4 lines

+ * go straight to host hardware, bypassing QEMU.
+ *
+ * 5. Guest SMMU driver programs a Stream Table Entry for a passthrough
+ * device: IOMMU_VDEVICE_ALLOC programs SID_MATCH/SID_REPLACE

in this

Ideally, the VDEVICE_ALLOC should happen right after the device gets plugged *and* its vSID is available; similarly the vdevice should be only freed when the device gets unplugged. So, its lifecycle should stay with the plugged state.

We do VDEVICE_ALLOC when guest programming STE is because that's the only place we found in the existing functions. We could (or arguably maybe we should) find a better place to allocate VDEVICE. E.g., vfio code can call qemu_add_vm_change_state_handler_prio() to register a callback function, where vSID (vBDF) should be available once the VM start to be "running".

Quoted 2 lines

I think your narrative fits the current design. But we might need to be more accurate here, since this is about "lifecycle".

Yes, the above is based on the current design. I will add a caveat to cover that.

I remember discussions about use cases(CCA realm ?) requiring an early mandatory VDEVICE_ALLOC before guest boot. Regarding the suggestion to use qemu_add_vm_change_state_handler_prio(), it might help, but I'm not sure it covers all the cases like, hotplug, guest PCI re-enumeration during boot, etc. (Having _DSM #5 might help here, but per the PCI Firmware Spec it only "indicates to an operating system that it should preserve assignments of PCI bus numbers", so, guests are not strictly required to honour it.) Anyway, I'll take another look as a follow up.

Quoted 7 lines
Quoted 1 line

+ * VM's VINTF so the device's guest vSID translates to its host pSID.

Nit: so that the HW translates the guest vSID into host pSID.

Quoted 3 lines

+ * Limits exposed to the guest
+ * ---------------------------
+ * One VINTF per emulated SMMUv3 and two vCMDQs per VINTF.

Quoted 8 lines

Maximum vCMDQ

Quoted 2 lines

+ * size is 8MiB. The queue must be physically contiguous (the HW reads it
+ * via host PA), so QEMU caps it to the host memory-backend page size.

Use

Quoted 1 line

+ * hugepage backing large enough to keep CMDQS at the HW maximum.

The "physically contiguous" limit is handled by QEMU, not exposed to the guest at all; guest is unaware of this.

Ok. The intend was the queue size exposed may be limited because of the "physically contiguous" requirement. I will make it clear.

Thanks,
Shameer

#51 lore raw

Re: [PATCH v6 30/31] hw/arm/tegra241-cmdqv: Document the CMDQV design and lifecycle

Quoted 17 lines · Shameer Kolothum Thodi
Quoted 17 lines

From: Nicolin Chen <[email protected]>

Quoted 2 lines

+ * 2. Guest writes VINTF_CONFIG.ENABLE = 1:
+ * QEMU mmap()s the offset from step 1 into its address space and

reports

Quoted 2 lines

+ * STATUS.ENABLE_OK = 1. The host VINTF was already enabled by
+ * IOMMU_VIOMMU_ALLOC; QEMU only acks back.

I wonder this mmap should be done after the
CMDQV_CONFIG.CMDQV_EN=1
rather than VINTF_CONFIG.ENABLE=1.

Spec says that:
"Program the CMDQV_CMDQ_ALLOC_MAP_X register(s) to map the Virtual
CMDQ(s) to the logical CMDQs on Virtual Interface following the CMDQ allocation rules. This again is an optional step and the CMDQ allocation can be done after the Virtual Interface is initialized."

Quoted 8 lines
Quoted 5 lines

This means the LVCMDQ mapping can happen before
VINTF_CONFIG.ENABLE=1.

And the mmap info is returned with IOMMU_VIOMMU_ALLOC. So, this is doable?

I think it is. We could in theory as well do it just after IOMMU_VIOMMU_ALLOC. What you think?

Yea. That sounds right to me.

Quoted 22 lines
Quoted 16 lines
Quoted 2 lines

+ * 5. Guest SMMU driver programs a Stream Table Entry for a passthrough
+ * device: IOMMU_VDEVICE_ALLOC programs SID_MATCH/SID_REPLACE

in this

Ideally, the VDEVICE_ALLOC should happen right after the device gets plugged *and* its vSID is available; similarly the vdevice should be only freed when the device gets unplugged. So, its lifecycle should stay with the plugged state.

We do VDEVICE_ALLOC when guest programming STE is because that's the only place we found in the existing functions. We could (or arguably maybe we should) find a better place to allocate VDEVICE. E.g., vfio code can call qemu_add_vm_change_state_handler_prio() to register a callback function, where vSID (vBDF) should be available once the VM start to be "running".

Quoted 2 lines

I think your narrative fits the current design. But we might need to be more accurate here, since this is about "lifecycle".

Yes, the above is based on the current design. I will add a caveat to cover that.

Quoted 9 lines

I remember discussions about use cases(CCA realm ?) requiring an early mandatory VDEVICE_ALLOC before guest boot. Regarding the suggestion to use qemu_add_vm_change_state_handler_prio(), it might help, but I'm not sure it covers all the cases like, hotplug, guest PCI re-enumeration during boot, etc. (Having _DSM #5 might help here, but per the PCI Firmware Spec it only "indicates to an operating system that it should preserve assignments of PCI bus numbers", so, guests are not strictly required to honour it.) Anyway, I'll take another look as a follow up.

Yea. I am not giving a suggestion that would work perfectly here. I think we are good with what we implemented for now. REALM stuff will need to take another look.

Nicolin

#52 lore raw

Re: [PATCH v6 30/31] hw/arm/tegra241-cmdqv: Document the CMDQV design and lifecycle

In reply to Shameer Kolothum

Hi Shammer,

Quoted 2 lines · Shameer Kolothum

Add an overview describing the Tegra241 CMDQV passthrough model, MMIO layout, guest-driven lifecycle, and per-VM isolation.

some suggestions inline. Thank you very much for the documentation efforts! That's really helpful.

Quoted 12 lines
---
 hw/arm/tegra241-cmdqv.c | 93 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 93 insertions(+)
hw/arm/tegra241-cmdqv.c +0 -0
index 4dba1783fa..5dbf68b421 100644
@@ -7,6 +7,99 @@
  * SPDX-License-Identifier: GPL-2.0-or-later

hw/arm/tegra241-cmdqv.c line 17

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +8 -0
  */
 
+/*
+ * Tegra241 CMDQV - overview
+ * =========================
+ *
+ * NVIDIA Tegra241 extends SMMUv3 with a Command Queue Virtualization (CMDQ-V)
+ * block. It lets a guest issue SMMU invalidation commands directly to
+ * dedicated hardware queues (vCMDQs) without trapping into the hypervisor on
+ * the fast path. vCMDQs are grouped into Virtual Interfaces (VINTFs); the

s/are grouped/are exclusively allocated to VINTFS?

hw/arm/tegra241-cmdqv.c line 25

Quoted 8 lines
hw/arm/tegra241-cmdqv.c +8 -0
+ * host kernel allocates one VINTF per emulated SMMUv3 instance via iommufd.
+ * QEMU emulates the CMDQV MMIO region and drives the host kernel calls
+ * (VIOMMU_ALLOC, HW_QUEUE_ALLOC, mmap); the actual command processing
+ * happens on real hardware.
+ *
+ * A vCMDQ becomes functional only once allocated to a VINTF; until then,
+ * register accesses only touch the QEMU-side cache and no command
+ * processing happens. After allocation, command processing runs on the

state is migrated from cache to HW and command processing ...

hw/arm/tegra241-cmdqv.c line 33

Quoted 8 lines
hw/arm/tegra241-cmdqv.c +8 -0
+ * host hardware, and guest accesses to the live control/status registers
+ * bypass QEMU and reach the host directly.
+ *
+ * MMIO layout (64KB pages, total TEGRA241_CMDQV_IO_LEN)
+ * -----------------------------------------------------
+ * The direct vCMDQ apertures (0x10000/0x20000) are HW aliases of the VINTF
+ * apertures (0x30000/0x40000); they expose the same per-vCMDQ register slots
+ * under different addressing.

I would put above sentence after the layout description.

Quoted 6 lines
hw/arm/tegra241-cmdqv.c +6 -0
+ *
+ *   0x00000  CMDQV Config page: QEMU-trapped.
+ *   0x10000  Direct vCMDQ Page 0 (control/status): QEMU-trapped and routed
+ *            to either the mmap'd VINTF Page 0 (if the vCMDQ has been
+ *            allocated to a VINTF) or a per-vCMDQ register cache (otherwise).
+ *   0x20000  Direct vCMDQ Page 1 (BASE / DRAM addresses): QEMU-trapped.

hw/arm/tegra241-cmdqv.c line 49

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10 -0
+ *   0x30000  VINTF Page 0 (per-VINTF control/status): mmap'd from the host
+ *            via iommufd and installed into guest MMIO as a RAM-device
+ *            subregion after the first HW_QUEUE_ALLOC; subsequent accesses
+ *            bypass QEMU.
+ *   0x40000  VINTF Page 1 (per-VINTF BASE): QEMU-trapped. Although this is
+ *            a HW alias of the direct Page 1, the kernel only exposes mmap
+ *            for VINTF Page 0; VINTF Page 1 is not mmap'd and stays trapped.
+ *
+ * The direct vCMDQ Page 0 stays trapped rather than aliased to the VINTF
+ * Page 0 mmap. Direct aperture read/write is not an expected common usecase,

You can justify it as you did in the individual patch (to keep management of unmapped vcmds in direct page compliant with the spec)

Quoted 4 lines
hw/arm/tegra241-cmdqv.c +4 -0
+ * but trapping keeps accesses well-defined for an unallocated vCMDQ.
+ *
+ * Lifecycle (driven by guest events)
+ * ----------------------------------

hw/arm/tegra241-cmdqv.c line 63

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10 -0
+ * 1. First vfio-pci device attach (.set_iommu_device) triggers:
+ *    - tegra241_cmdqv_probe(): IOMMU_GET_HW_INFO confirms host CMDQV support.
+ *    - IOMMU_VIOMMU_ALLOC: the kernel allocates a VINTF for this VM,
+ *      configures the VM's VMID (from its stage-2 HWPT) in VINTF_CONFIG,
+ *      forces HYP_OWN=0, and returns the mmap offset/length for VINTF Page 0.
+ *
+ * 2. Guest writes VINTF_CONFIG.ENABLE = 1:
+ *    QEMU mmap()s the offset from step 1 into its address space and reports
+ *    STATUS.ENABLE_OK = 1. The host VINTF was already enabled by
+ *    IOMMU_VIOMMU_ALLOC; QEMU only acks back.

the host VINTF ... : I would add this in 1. I would remove "QEMU only acks back."

hw/arm/tegra241-cmdqv.c line 66

Quoted 3 lines
hw/arm/tegra241-cmdqv.c +3 -0
+ *
+ * 3. Guest completes vCMDQ setup (BASE, CMDQ_ALLOC_MAP.ALLOC, CMDQV_EN,
+ *    VINTF.ENABLE, in any order; each precondition write retries the

the HW queue allocation:

hw/arm/tegra241-cmdqv.c line 69

Quoted 3 lines
hw/arm/tegra241-cmdqv.c +3 -0
+ *    allocation):
+ *    IOMMU_HW_QUEUE_ALLOC binds the guest BASE GPA (translated through
+ *    stage-2 and pinned by the kernel) to a host vCMDQ in this VM's VINTF.

the guest is granted a new host vcmdq in its vintf?

Quoted 26 lines
hw/arm/tegra241-cmdqv.c +26 -0
+ *
+ * 4. After the first successful HW_QUEUE_ALLOC, the mmap'd VINTF Page 0 is
+ *    installed into guest MMIO as a RAM-device subregion. Guest VINTF Page 0
+ *    accesses (CMDQ_EN, PROD/CONS_INDX, STATUS, GERROR/GERRORN) thereafter
+ *    go straight to host hardware, bypassing QEMU.
+ *
+ * 5. Guest SMMU driver programs a Stream Table Entry for a passthrough
+ *    device: IOMMU_VDEVICE_ALLOC programs SID_MATCH/SID_REPLACE in this
+ *    VM's VINTF so the device's guest vSID translates to its host pSID.
+ *    Commands referencing unmapped SIDs are rejected by HW.
+ *
+ * Per-VM isolation
+ * ----------------
+ * - Each VM has its own iommufd FD; all iommufd objects (VINTF, vdevices,
+ *   hw_queues, mmap regions) belong to that FD. Cross-FD lookups fail, so
+ *   one VM cannot reach another VM's IDs.
+ * - IOMMU_VIOMMU_ALLOC configures the VM's VMID in VINTF_CONFIG; the CMDQV
+ *   hardware substitutes / checks VMID on every command the guest issues.
+ * - The kernel allocates the VINTF with HYP_OWN = 0, which restricts the
+ *   guest to a safe subset of commands.
+ * - IOMMU_VDEVICE_ALLOC populates SID_MATCH/SID_REPLACE so invalidations
+ *   only reach the host StreamIDs assigned to this VM (see step 5).
+ * - IOMMU_HW_QUEUE_ALLOC binds each vCMDQ to a single VINTF, so a guest
+ *   cannot reach a vCMDQ that belongs to another VM.
+ *
+ * Limits exposed to the guest

hw/arm/tegra241-cmdqv.c line 105

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +7 -0
+ * ---------------------------
+ * One VINTF per emulated SMMUv3 and two vCMDQs per VINTF. Maximum vCMDQ
+ * size is 8MiB. The queue must be physically contiguous (the HW reads it
+ * via host PA), so QEMU caps it to the host memory-backend page size. Use
+ * hugepage backing large enough to keep CMDQS at the HW maximum.
+ */
+
 #include "qemu/osdep.h"
 #include "qemu/log.h"
 #include "qemu/error-report.h"

Thanks

Eric

#53 lore raw

PATCH v6 31/31 hw/arm/smmuv3: Add cmdqv property for SMMUv3 device

In reply to Shameer Kolothum

Introduce a "cmdqv" property to enable Tegra241 CMDQV support. This is only enabled for accelerated SMMUv3 devices.

---
 hw/arm/smmuv3.c | 8 ++++++++
 qemu-options.hx | 8 ++++++++
 2 files changed, 16 insertions(+)
hw/arm/smmuv3.c +8 -0
index 0d111d638b..23592354ee 100644
@@ -1994,6 +1994,10 @@ static bool smmu_validate_property(SMMUv3State *s, Error **errp)
                        "bits if accel=on");
             return false;
         }
+        if (s->cmdqv == ON_OFF_AUTO_ON) {
+            error_setg(errp, "cmdqv can only be enabled if accel=on");
+            return false;
+        }
         return true;
     }
 
@@ -2144,6 +2148,7 @@ static const Property smmuv3_properties[] = {
     DEFINE_PROP_OAS_MODE("oas", SMMUv3State, oas, OAS_MODE_AUTO),
     DEFINE_PROP_SSIDSIZE_MODE("ssidsize", SMMUv3State, ssidsize,
                               SSID_SIZE_MODE_AUTO),
+    DEFINE_PROP_ON_OFF_AUTO("cmdqv", SMMUv3State, cmdqv, ON_OFF_AUTO_AUTO),
 };
 
 static void smmuv3_instance_init(Object *obj)
@@ -2194,6 +2199,9 @@ static void smmuv3_class_init(ObjectClass *klass, const void *data)
         "than 0 is required to enable PASID support."
         "Please ensure the value does not exceed the maximum "
         "SubtreamID size supported by the host platform.");
+    object_class_property_set_description(klass, "cmdqv",
+        "Enable/disable CMDQV support (for accel=on). "
+        "Valid values are on, off, and auto. Defaults to auto.");
 }
 
 static int smmuv3_notify_flag_changed(IOMMUMemoryRegion *iommu,
qemu-options.hx +8 -0
index d7e48688b5..2eee466be7 100644
@@ -1329,6 +1329,14 @@ SRST
 
         - With accel=off, auto is resolved to 0.
 
+    ``cmdqv=on|off|auto`` (default: auto)
+        Enable hardware Command Queue Virtualization (CMDQV) for the
+        SMMUv3 command queue. Currently only the NVIDIA Tegra241 CMDQV
+        implementation is supported.
+
+        - With accel=on, auto means the value is automatically derived from the host SMMU.
+        - With accel=off, auto is resolved to 'off'.
+
 ``-device amd-iommu[,option=...]``
     Enables emulation of an AMD-Vi I/O Memory Management Unit (IOMMU).
     Only available with ``-machine q35``, it supports the following options:
Signature · 1 line
2.43.0