Skip to contentSkip to conversation

hw/arm: Introduce Tegra241 CMDQV support for accelerated SMMUv3

RFC PATCH Superseded Shameer Kolothum 4 participants 68 messages Dec 10, 2025 – Jan 27

Subsystems (2)

v1 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

cover hw/arm: Introduce Tegra241 CMDQV support for accelerated SMMUv3

Hi,

This RFC series adds initial support for NVIDIA Tegra241 CMDQV (Command Queue Virtualisation), an extension to ARM SMMUv3 that provides hardware accelerated virtual command queues (VCMDQs) for guests. CMDQV allows guests to issue SMMU invalidation commands directly to hardware without VM exits, significantly reducing TLBI overhead.

Thanks to Nicolin for the initial patches and testing on which this RFC is based.

This is based on v6[0] of the SMMUv3 accel series, which is still under review, though nearing convergence. This is sent as an RFC, with the goal of gathering early feedback on the CMDQV design and its integration with the SMMUv3 acceleration path.

Background:

Tegra241 CMDQV extends SMMUv3 by allocating per-VM "virtual interfaces" (VINTFs), each hosting up to 128 VCMDQs.

Each VINTF exposes two 64KB MMIO pages:
- Page0 – guest owned control and status registers (directly mapped into the VM)
- Page1 – queue configuration registers (trapped/emulated by QEMU)

Unlike the standard SMMU CMDQ, a guest owned Tegra241 VCMDQ does not support the full command set. Only a subset, primarily invalidation related commands, is accepted by the CMDQV hardware. For this reason, a distinct CMDQV device must be exposed to the guest, and the guest OS must include a Tegra241 CMDQV aware driver to take advantage of the hardware acceleration.

VCMDQ support is integrated via the IOMMU_HW_QUEUE_ALLOC mechanism, allowing QEMU to attach guest configured VCMDQ buffers to the underlying CMDQV hardware through IOMMUFD. The Linux kernel already supports the full CMDQV virtualisation model via IOMMUFD[0].

Summary of QEMU changes:

- Integrated into the existing SMMUv3 accel path via a "tegra241-cmdqv" property.
- Support for allocating vIOMMU objects of type IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV.
- Mapping and emulation of the CMDQV MMIO register layout.
- VCMDQ/VINTF read/write handling and queue allocation using IOMMUFD APIs.
- Reset and initialisation hooks, including checks for at least one cold-plugged device.
- CMDQV hardware reads guest queue memory using host physical addresses provided through IOMMUFD, which requires that the VCMDQ buffer be physically contiguous not only in guest PA space but also in host PA space. When Tegra241 CMDQV is enabled, QEMU must therefore only expose a CMDQV size that the host can reliably back with contiguous physical memory. Because of this constraint, it is suggested to use huge pages to back the guest RAM.
- ACPI DSDT node generation for CMDQV devices on the virt machine.

These patches have been sanity tested on NVIDIA Grace platforms.

ToDo / revisit:
- Prevent hot-unplug of the last device associated with vIOMMU as this might allow associating a different host SMMU/CMDQV.
- Locking requirements around error event propagation.

Feedback and testing are very welcome.

Thanks,
Shameer
[0] https://lore.kernel.org/qemu-devel/[email protected]/
[1] https://lore.kernel.org/all/[email protected]/

Nicolin Chen (12):
backends/iommufd: Update iommufd_backend_get_device_info backends/iommufd: Update iommufd_backend_alloc_viommu to allow user
ptr
backends/iommufd: Introduce iommufd_backend_alloc_hw_queue backends/iommufd: Introduce iommufd_backend_viommu_mmap hw/arm/tegra241-cmdqv: Add initial Tegra241 CMDQ-Virtualisation
support
hw/arm/tegra241-cmdqv: Map VINTF Page0 into guest hw/arm/tegra241-cmdqv: Add read emulation support for registers system/physmem: Add helper to check whether a guest PA maps to RAM hw/arm/tegra241-cmdqv:: Add write emulation for registers hw/arm/tegra241-cmdqv: Add reset handler hw/arm/tegra241-cmdqv: Limit queue size based on backend page size hw/arm/virt-acpi: Advertise Tegra241 CMDQV nodes in DSDT

Shameer Kolothum (4):
hw/arm/tegra241-cmdqv: Allocate vEVENTQ object hw/arm/tegra241-cmdqv: Read and propagate Tegra241 CMDQV errors
virt-acpi-build: Rename AcpiIortSMMUv3Dev to AcpiSMMUv3Dev hw/arm/smmuv3: Add tegra241-cmdqv property for SMMUv3 device

 backends/iommufd.c        |  65 ++++
 backends/trace-events     |   2 +
 hw/arm/Kconfig            |   5 +
 hw/arm/meson.build        |   1 +
 hw/arm/smmuv3-accel.c     |  16 +-
 hw/arm/smmuv3.c           |  18 +
 hw/arm/tegra241-cmdqv.c   | 759 ++++++++++++++++++++++++++++++++++++++
 hw/arm/tegra241-cmdqv.h   | 337 +++++++++++++++++
 hw/arm/trace-events       |   5 +
 hw/arm/virt-acpi-build.c  | 110 +++++-
 hw/vfio/iommufd.c         |   6 +-
 include/exec/cpu-common.h |   2 +
 include/hw/arm/smmuv3.h   |   3 +
 include/hw/arm/virt.h     |   2 +
 include/system/iommufd.h  |  16 +
 system/physmem.c          |  12 +
 16 files changed, 1332 insertions(+), 27 deletions(-)
 create mode 100644 hw/arm/tegra241-cmdqv.c
 create mode 100644 hw/arm/tegra241-cmdqv.h
Signature · 1 line
2.43.0
#1 lore raw

RFC PATCH 1/16 backends/iommufd: Update iommufd_backend_get_device_info

From: Nicolin Chen <[email protected]>

The updated IOMMUFD uAPI introduces the ability for userspace to request a specific hardware info data type via IOMMU_GET_HW_INFO. Update iommufd_backend_get_device_info() to set IOMMU_HW_INFO_FLAG_INPUT_TYPE when a non-zero type is supplied, and adjust all callers to pass an explicitly initialised type value.

---
 backends/iommufd.c    | 7 +++++++
 hw/arm/smmuv3-accel.c | 2 +-
 hw/vfio/iommufd.c     | 6 ++----
 3 files changed, 10 insertions(+), 5 deletions(-)
backends/iommufd.c +7 -0
index 633aecd525..938c8fe669 100644
@@ -386,16 +386,23 @@ bool iommufd_backend_get_dirty_bitmap(IOMMUFDBackend *be,
     return true;
 }
 
+/*
+ * @type can carry a desired HW info type defined in the uapi headers. If caller
+ * doesn't have one, indicating it wants the default type, then @type should be
+ * zeroed (i.e. IOMMU_HW_INFO_TYPE_DEFAULT).
+ */
 bool iommufd_backend_get_device_info(IOMMUFDBackend *be, uint32_t devid,
                                      uint32_t *type, void *data, uint32_t len,
                                      uint64_t *caps, uint8_t *max_pasid_log2,
                                      Error **errp)
 {
     struct iommu_hw_info info = {
+        .flags = (*type) ? IOMMU_HW_INFO_FLAG_INPUT_TYPE : 0,
         .size = sizeof(info),
         .dev_id = devid,
         .data_len = len,
         .data_uptr = (uintptr_t)data,
+        .in_data_type = *type,
     };
 
     if (ioctl(be->fd, IOMMU_GET_HW_INFO, &info)) {
hw/arm/smmuv3-accel.c +1 -1
index d320c62b04..300c35ccb5 100644
@@ -115,7 +115,7 @@ smmuv3_accel_hw_compatible(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
                            Error **errp)
 {
     struct iommu_hw_info_arm_smmuv3 info;
-    uint32_t data_type;
+    uint32_t data_type = 0;
     uint64_t caps;
 
     if (!iommufd_backend_get_device_info(idev->iommufd, idev->devid, &data_type,
hw/vfio/iommufd.c +2 -4
index bbe944d7cc..670bdfc53b 100644
@@ -306,7 +306,7 @@ static bool iommufd_cdev_autodomains_get(VFIODevice *vbasedev,
     ERRP_GUARD();
     IOMMUFDBackend *iommufd = vbasedev->iommufd;
     VFIOContainer *bcontainer = VFIO_IOMMU(container);
-    uint32_t type, flags = 0;
+    uint32_t type = 0, flags = 0;
     uint64_t hw_caps;
     VFIOIOASHwpt *hwpt;
     uint32_t hwpt_id;
@@ -631,8 +631,6 @@ skip_ioas_alloc:
     bcontainer->initialized = true;
 
 found_container:
-    vbasedev->cpr.ioas_id = container->ioas_id;
-
     ret = ioctl(devfd, VFIO_DEVICE_GET_INFO, &dev_info);
     if (ret) {
         error_setg_errno(errp, errno, "error getting device info");
@@ -889,7 +887,7 @@ static bool hiod_iommufd_vfio_realize(HostIOMMUDevice *hiod, void *opaque,
     HostIOMMUDeviceIOMMUFD *idev;
     HostIOMMUDeviceCaps *caps = &hiod->caps;
     VendorCaps *vendor_caps = &caps->vendor_caps;
-    enum iommu_hw_info_type type;
+    enum iommu_hw_info_type type = 0;
     uint8_t max_pasid_log2;
     uint64_t hw_caps;
 
Signature · 1 line
2.43.0
#2 lore raw

Re: [RFC PATCH 01/16] backends/iommufd: Update iommufd_backend_get_device_info

Quoted 5 lines · Shameer Kolothum

The updated IOMMUFD uAPI introduces the ability for userspace to request a specific hardware info data type via IOMMU_GET_HW_INFO. Update iommufd_backend_get_device_info() to set IOMMU_HW_INFO_FLAG_INPUT_TYPE when a non-zero type is supplied, and adjust all callers to pass an explicitly initialised type value.

Let's mention IOMMU_HW_INFO_TYPE_DEFAULT=0 v.s. just "initialized".

Quoted 2 lines

+++ b/hw/vfio/iommufd.c

(quoted diff) +0 -0
@@ -306,7 +306,7 @@ static bool iommufd_cdev_autodomains_get(VFIODevice *vbasedev,

[...]

hw/vfio/iommufd.c skip_ioas_alloc: · line 636

Quoted 8 lines
     bcontainer->initialized = true;

found_container:

hw/vfio/iommufd.c +0 -2
-    vbasedev->cpr.ioas_id = container->ioas_id;
-
     ret = ioctl(devfd, VFIO_DEVICE_GET_INFO, &dev_info);
     if (ret) {
         error_setg_errno(errp, errno, "error getting device info");

I might have missed some details but this seems unrelated?

Nicolin

#3 lore raw

RE: [RFC PATCH 01/16] backends/iommufd: Update iommufd_backend_get_device_info

Quoted 20 lines

-----Original Message-----
From: Nicolin Chen <[email protected]>
Sent: 29 December 2025 18:40
To: Shameer Kolothum <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; [email protected]; [email protected]; Krishnakant Jaju <[email protected]>
Subject: Re: [RFC PATCH 01/16] backends/iommufd: Update iommufd_backend_get_device_info

Quoted 4 lines · Shameer Kolothum

The updated IOMMUFD uAPI introduces the ability for userspace to request a specific hardware info data type via IOMMU_GET_HW_INFO. Update
iommufd_backend_get_device_info() to set

Quoted 5 lines

IOMMU_HW_INFO_FLAG_INPUT_TYPE

Quoted 2 lines

when a non-zero type is supplied, and adjust all callers to pass an explicitly initialised type value.

Let's mention IOMMU_HW_INFO_TYPE_DEFAULT=0 v.s. just "initialized".

Ok.

Quoted 7 lines
Quoted 4 lines

+++ b/hw/vfio/iommufd.c

(quoted diff) +0 -0
@@ -306,7 +306,7 @@ static bool
 iommufd_cdev_autodomains_get(VFIODevice *vbasedev,

[...]

Quoted 2 lines
     bcontainer->initialized = true;
Quoted 8 lines

hw/vfio/iommufd.c skip_ioas_alloc: · line 636

Quoted 6 lines

found_container:

hw/vfio/iommufd.c +0 -2
-    vbasedev->cpr.ioas_id = container->ioas_id;
-
     ret = ioctl(devfd, VFIO_DEVICE_GET_INFO, &dev_info);
     if (ret) {
         error_setg_errno(errp, errno, "error getting device info");

I might have missed some details but this seems unrelated?

Yes, that is likely an artifact from a careless rebase on my side. Thanks for catching it.

Shameer

Reviewed-by on v1 1/16
#4 lore raw

Re: [RFC PATCH 01/16] backends/iommufd: Update iommufd_backend_get_device_info

In reply to Shameer Kolothum

Hi Shameer,

Quoted 1 line · Shameer Kolothum

From: Nicolin Chen <[email protected]>

I would suggest to be more explicit in the commit title like:
Convey input type in iommufd_backend_get_device_info()

Quoted 20 lines

The updated IOMMUFD uAPI introduces the ability for userspace to request a specific hardware info data type via IOMMU_GET_HW_INFO. Update iommufd_backend_get_device_info() to set IOMMU_HW_INFO_FLAG_INPUT_TYPE when a non-zero type is supplied, and adjust all callers to pass an explicitly initialised type value.

---
 backends/iommufd.c    | 7 +++++++
 hw/arm/smmuv3-accel.c | 2 +-
 hw/vfio/iommufd.c     | 6 ++----
 3 files changed, 10 insertions(+), 5 deletions(-)
backends/iommufd.c +0 -0
index 633aecd525..938c8fe669 100644

backends/iommufd.c bool iommufd_backend_get_dirty_bitmap(IOMMUFDBackend *be, · line 392

Quoted 7 lines
backends/iommufd.c +4 -0
@@ -386,16 +386,23 @@ bool iommufd_backend_get_dirty_bitmap(IOMMUFDBackend *be,
     return true;
 }
 
+/*
+ * @type can carry a desired HW info type defined in the uapi headers. If caller
+ * doesn't have one, indicating it wants the default type, then @type should be
+ * zeroed (i.e. IOMMU_HW_INFO_TYPE_DEFAULT).

the kernel uapi doc is a bit ambiguous as it is said:
 * @flags: Must be 0
while a flag value does exist.

Quoted 43 lines
backends/iommufd.c +3 -0
+ */
 bool iommufd_backend_get_device_info(IOMMUFDBackend *be, uint32_t devid,
                                      uint32_t *type, void *data, uint32_t len,
                                      uint64_t *caps, uint8_t *max_pasid_log2,
                                      Error **errp)
 {
     struct iommu_hw_info info = {
+        .flags = (*type) ? IOMMU_HW_INFO_FLAG_INPUT_TYPE : 0,
         .size = sizeof(info),
         .dev_id = devid,
         .data_len = len,
         .data_uptr = (uintptr_t)data,
+        .in_data_type = *type,
     };
 
     if (ioctl(be->fd, IOMMU_GET_HW_INFO, &info)) {
hw/arm/smmuv3-accel.c +1 -1
index d320c62b04..300c35ccb5 100644
@@ -115,7 +115,7 @@ smmuv3_accel_hw_compatible(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
                            Error **errp)
 {
     struct iommu_hw_info_arm_smmuv3 info;
-    uint32_t data_type;
+    uint32_t data_type = 0;
     uint64_t caps;
 
     if (!iommufd_backend_get_device_info(idev->iommufd, idev->devid, &data_type,
hw/vfio/iommufd.c +1 -1
index bbe944d7cc..670bdfc53b 100644
@@ -306,7 +306,7 @@ static bool iommufd_cdev_autodomains_get(VFIODevice *vbasedev,
     ERRP_GUARD();
     IOMMUFDBackend *iommufd = vbasedev->iommufd;
     VFIOContainer *bcontainer = VFIO_IOMMU(container);
-    uint32_t type, flags = 0;
+    uint32_t type = 0, flags = 0;
     uint64_t hw_caps;
     VFIOIOASHwpt *hwpt;
     uint32_t hwpt_id;

hw/vfio/iommufd.c skip_ioas_alloc: · line 635

Quoted 5 lines
hw/vfio/iommufd.c +0 -2
@@ -631,8 +631,6 @@ skip_ioas_alloc:
     bcontainer->initialized = true;
 
 found_container:
-    vbasedev->cpr.ioas_id = container->ioas_id;
-

indeed looks unrelated to that patch

hw/vfio/iommufd.c static bool hiod_iommufd_vfio_realize(HostIOMMUDevice *hiod, void *opaque, · line 893

Quoted 12 lines
hw/vfio/iommufd.c +1 -1
     ret = ioctl(devfd, VFIO_DEVICE_GET_INFO, &dev_info);
     if (ret) {
         error_setg_errno(errp, errno, "error getting device info");
@@ -889,7 +887,7 @@ static bool hiod_iommufd_vfio_realize(HostIOMMUDevice *hiod, void *opaque,
     HostIOMMUDeviceIOMMUFD *idev;
     HostIOMMUDeviceCaps *caps = &hiod->caps;
     VendorCaps *vendor_caps = &caps->vendor_caps;
-    enum iommu_hw_info_type type;
+    enum iommu_hw_info_type type = 0;
     uint8_t max_pasid_log2;
     uint64_t hw_caps;
 

Besides

Thanks

Eric

#5 lore raw

RFC PATCH 2/16 backends/iommufd: Update iommufd_backend_alloc_viommu to allow user ptr

In reply to Shameer Kolothum

From: Nicolin Chen <[email protected]>

The updated IOMMUFD VIOMMU_ALLOC uAPI allows userspace to provide a data buffer when creating a vIOMMU (e.g. for Tegra241 CMDQV). Extend iommufd_backend_alloc_viommu() to pass a user pointer and size to the kernel.

Update the caller accordingly.

---
 backends/iommufd.c       | 3 +++
 hw/arm/smmuv3-accel.c    | 4 ++--
 include/system/iommufd.h | 1 +
 3 files changed, 6 insertions(+), 2 deletions(-)
backends/iommufd.c +3 -0
index 938c8fe669..2f6fa832a7 100644
@@ -459,6 +459,7 @@ bool iommufd_backend_invalidate_cache(IOMMUFDBackend *be, uint32_t id,
 
 bool iommufd_backend_alloc_viommu(IOMMUFDBackend *be, uint32_t dev_id,
                                   uint32_t viommu_type, uint32_t hwpt_id,
+                                  void *data_ptr, uint32_t len,
                                   uint32_t *out_viommu_id, Error **errp)
 {
     int ret;
@@ -467,6 +468,8 @@ bool iommufd_backend_alloc_viommu(IOMMUFDBackend *be, uint32_t dev_id,
         .type = viommu_type,
         .dev_id = dev_id,
         .hwpt_id = hwpt_id,
+        .data_len = len,
+        .data_uptr = (uintptr_t)data_ptr,
     };
 
     ret = ioctl(be->fd, IOMMU_VIOMMU_ALLOC, &alloc_viommu);
hw/arm/smmuv3-accel.c +2 -2
index 300c35ccb5..939898c9b0 100644
@@ -503,8 +503,8 @@ smmuv3_accel_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
     SMMUv3AccelState *accel;
 
     if (!iommufd_backend_alloc_viommu(idev->iommufd, idev->devid,
-                                      IOMMU_VIOMMU_TYPE_ARM_SMMUV3,
-                                      s2_hwpt_id, &viommu_id, errp)) {
+                                      IOMMU_VIOMMU_TYPE_ARM_SMMUV3, s2_hwpt_id,
+                                      NULL, 0, &viommu_id, errp)) {
         return false;
     }
 
include/system/iommufd.h +1 -0
index 9770ff1484..a3e8087b3a 100644
@@ -87,6 +87,7 @@ bool iommufd_backend_alloc_hwpt(IOMMUFDBackend *be, uint32_t dev_id,
                                 Error **errp);
 bool iommufd_backend_alloc_viommu(IOMMUFDBackend *be, uint32_t dev_id,
                                   uint32_t viommu_type, uint32_t hwpt_id,
+                                  void *data_ptr, uint32_t len,
                                   uint32_t *out_hwpt, Error **errp);
 
 bool iommufd_backend_alloc_vdev(IOMMUFDBackend *be, uint32_t dev_id,
Signature · 1 line
2.43.0
#6 lore raw

Re: [RFC PATCH 02/16] backends/iommufd: Update iommufd_backend_alloc_viommu to allow user ptr

backends/iommufd.c bool iommufd_backend_alloc_viommu(IOMMUFDBackend *be, uint32_t dev_id, · line 473

Quoted 13 lines · Shameer Kolothum

bool iommufd_backend_alloc_viommu(IOMMUFDBackend *be, uint32_t dev_id,

                                   uint32_t viommu_type, uint32_t hwpt_id,
backends/iommufd.c +3 -0
+                                  void *data_ptr, uint32_t len,
                                   uint32_t *out_viommu_id, Error **errp)
 {
     int ret;
@@ -467,6 +468,8 @@ bool iommufd_backend_alloc_viommu(IOMMUFDBackend *be, uint32_t dev_id,
         .type = viommu_type,
         .dev_id = dev_id,
         .hwpt_id = hwpt_id,
+        .data_len = len,
+        .data_uptr = (uintptr_t)data_ptr,
     };

Perhaps we should update the trace as well, given that we do print data_len/ptr for iommufd_backend_alloc_hwpt().

Nicolin

Reviewed-by on v1 2/16
#7 lore raw

Re: [RFC PATCH 02/16] backends/iommufd: Update iommufd_backend_alloc_viommu to allow user ptr

In reply to Shameer Kolothum
Quoted 10 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

The updated IOMMUFD VIOMMU_ALLOC uAPI allows userspace to provide a data buffer when creating a vIOMMU (e.g. for Tegra241 CMDQV). Extend iommufd_backend_alloc_viommu() to pass a user pointer and size to the kernel.

Update the caller accordingly.

with or without trace update:

Eric

Quoted 55 lines
---
 backends/iommufd.c       | 3 +++
 hw/arm/smmuv3-accel.c    | 4 ++--
 include/system/iommufd.h | 1 +
 3 files changed, 6 insertions(+), 2 deletions(-)
backends/iommufd.c +3 -0
index 938c8fe669..2f6fa832a7 100644
@@ -459,6 +459,7 @@ bool iommufd_backend_invalidate_cache(IOMMUFDBackend *be, uint32_t id,
 
 bool iommufd_backend_alloc_viommu(IOMMUFDBackend *be, uint32_t dev_id,
                                   uint32_t viommu_type, uint32_t hwpt_id,
+                                  void *data_ptr, uint32_t len,
                                   uint32_t *out_viommu_id, Error **errp)
 {
     int ret;
@@ -467,6 +468,8 @@ bool iommufd_backend_alloc_viommu(IOMMUFDBackend *be, uint32_t dev_id,
         .type = viommu_type,
         .dev_id = dev_id,
         .hwpt_id = hwpt_id,
+        .data_len = len,
+        .data_uptr = (uintptr_t)data_ptr,
     };
 
     ret = ioctl(be->fd, IOMMU_VIOMMU_ALLOC, &alloc_viommu);
hw/arm/smmuv3-accel.c +2 -2
index 300c35ccb5..939898c9b0 100644
@@ -503,8 +503,8 @@ smmuv3_accel_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
     SMMUv3AccelState *accel;
 
     if (!iommufd_backend_alloc_viommu(idev->iommufd, idev->devid,
-                                      IOMMU_VIOMMU_TYPE_ARM_SMMUV3,
-                                      s2_hwpt_id, &viommu_id, errp)) {
+                                      IOMMU_VIOMMU_TYPE_ARM_SMMUV3, s2_hwpt_id,
+                                      NULL, 0, &viommu_id, errp)) {
         return false;
     }
 
include/system/iommufd.h +1 -0
index 9770ff1484..a3e8087b3a 100644
@@ -87,6 +87,7 @@ bool iommufd_backend_alloc_hwpt(IOMMUFDBackend *be, uint32_t dev_id,
                                 Error **errp);
 bool iommufd_backend_alloc_viommu(IOMMUFDBackend *be, uint32_t dev_id,
                                   uint32_t viommu_type, uint32_t hwpt_id,
+                                  void *data_ptr, uint32_t len,
                                   uint32_t *out_hwpt, Error **errp);
 
 bool iommufd_backend_alloc_vdev(IOMMUFDBackend *be, uint32_t dev_id,
#8 lore raw

RFC PATCH 3/16 backends/iommufd: Introduce iommufd_backend_alloc_hw_queue

In reply to Shameer Kolothum

From: Nicolin Chen <[email protected]>

Add a helper to allocate an iommufd backed HW queue for a vIOMMU.

While at it, define a struct IOMMUFDHWqueue for use by vendor implementations.

---
 backends/iommufd.c       | 31 +++++++++++++++++++++++++++++++
 backends/trace-events    |  1 +
 include/system/iommufd.h | 11 +++++++++++
 3 files changed, 43 insertions(+)
backends/iommufd.c +31 -0
index 2f6fa832a7..a644763239 100644
@@ -544,6 +544,37 @@ bool iommufd_backend_alloc_veventq(IOMMUFDBackend *be, uint32_t viommu_id,
     return true;
 }
 
+bool iommufd_backend_alloc_hw_queue(IOMMUFDBackend *be, uint32_t viommu_id,
+                                    uint32_t data_type, uint32_t index,
+                                    uint64_t addr, uint64_t length,
+                                    uint32_t *out_hw_queue_id, Error **errp)
+{
+    int ret;
+    struct iommu_hw_queue_alloc alloc_hw_queue = {
+        .size = sizeof(alloc_hw_queue),
+        .flags = 0,
+        .viommu_id = viommu_id,
+        .type = data_type,
+        .index = index,
+        .nesting_parent_iova = addr,
+        .length = length,
+    };
+
+    ret = ioctl(be->fd, IOMMU_HW_QUEUE_ALLOC, &alloc_hw_queue);
+
+    trace_iommufd_backend_alloc_hw_queue(be->fd, viommu_id, data_type,
+                                         index, addr, length,
+                                         alloc_hw_queue.out_hw_queue_id, ret);
+    if (ret) {
+        error_setg_errno(errp, errno, "IOMMU_HW_QUEUE_ALLOC failed");
+        return false;
+    }
+
+    g_assert(out_hw_queue_id);
+    *out_hw_queue_id = alloc_hw_queue.out_hw_queue_id;
+    return true;
+}
+
 bool host_iommu_device_iommufd_attach_hwpt(HostIOMMUDeviceIOMMUFD *idev,
                                            uint32_t hwpt_id, Error **errp)
 {
backends/trace-events +1 -0
index 5afa7a40be..a22ad30e55 100644
@@ -24,3 +24,4 @@ iommufd_backend_invalidate_cache(int iommufd, uint32_t id, uint32_t data_type, u
 iommufd_backend_alloc_viommu(int iommufd, uint32_t dev_id, uint32_t type, uint32_t hwpt_id, uint32_t viommu_id, int ret) " iommufd=%d type=%u dev_id=%u hwpt_id=%u viommu_id=%u (%d)"
 iommufd_backend_alloc_vdev(int iommufd, uint32_t dev_id, uint32_t viommu_id, uint64_t virt_id, uint32_t vdev_id, int ret) " iommufd=%d dev_id=%u viommu_id=%u virt_id=0x%"PRIx64" vdev_id=%u (%d)"
 iommufd_viommu_alloc_eventq(int iommufd, uint32_t viommu_id, uint32_t type, uint32_t veventq_id, uint32_t veventq_fd, int ret) " iommufd=%d viommu_id=%u type=%u veventq_id=%u veventq_fd=%u (%d)"
+iommufd_backend_alloc_hw_queue(int iommufd, uint32_t viommu_id, uint32_t vqueue_type, uint32_t index, uint64_t addr, uint64_t size, uint32_t vqueue_id, int ret) " iommufd=%d viommu_id=%u vqueue_type=%u index=%u addr=0x%"PRIx64" size=0x%"PRIx64" vqueue_id=%u (%d)"
include/system/iommufd.h +11 -0
index a3e8087b3a..9b8602a558 100644
@@ -63,6 +63,12 @@ typedef struct IOMMUFDVeventq {
     uint32_t veventq_fd;
 } IOMMUFDVeventq;
 
+/* HW queue object for a vIOMMU-specific HW-accelerated queue */
+typedef struct IOMMUFDHWqueue {
+    IOMMUFDViommu *viommu;
+    uint32_t hw_queue_id;
+} IOMMUFDHWqueue;
+
 bool iommufd_backend_connect(IOMMUFDBackend *be, Error **errp);
 void iommufd_backend_disconnect(IOMMUFDBackend *be);
 
@@ -99,6 +105,11 @@ bool iommufd_backend_alloc_veventq(IOMMUFDBackend *be, uint32_t viommu_id,
                                    uint32_t *out_veventq_id,
                                    uint32_t *out_veventq_fd, Error **errp);
 
+bool iommufd_backend_alloc_hw_queue(IOMMUFDBackend *be, uint32_t viommu_id,
+                                    uint32_t data_type, uint32_t index,
+                                    uint64_t addr, uint64_t length,
+                                    uint32_t *out_hw_queue_id, Error **errp);
+
 bool iommufd_backend_set_dirty_tracking(IOMMUFDBackend *be, uint32_t hwpt_id,
                                         bool start, Error **errp);
 bool iommufd_backend_get_dirty_bitmap(IOMMUFDBackend *be, uint32_t hwpt_id,
Signature · 1 line
2.43.0
Reviewed-by on v1 3/16
#9 lore raw

Re: [RFC PATCH 03/16] backends/iommufd: Introduce iommufd_backend_alloc_hw_queue

Hi Shameer,

Quoted 20 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

Add a helper to allocate an iommufd backed HW queue for a vIOMMU.

While at it, define a struct IOMMUFDHWqueue for use by vendor implementations.

---
 backends/iommufd.c       | 31 +++++++++++++++++++++++++++++++
 backends/trace-events    |  1 +
 include/system/iommufd.h | 11 +++++++++++
 3 files changed, 43 insertions(+)
backends/iommufd.c +0 -0
index 2f6fa832a7..a644763239 100644

backends/iommufd.c bool iommufd_backend_alloc_veventq(IOMMUFDBackend *be, uint32_t viommu_id, · line 548

Quoted 5 lines
backends/iommufd.c +2 -0
@@ -544,6 +544,37 @@ bool iommufd_backend_alloc_veventq(IOMMUFDBackend *be, uint32_t viommu_id,
     return true;
 }
 
+bool iommufd_backend_alloc_hw_queue(IOMMUFDBackend *be, uint32_t viommu_id,
+                                    uint32_t data_type, uint32_t index,

this is named type in the uapi

Quoted 37 lines
backends/iommufd.c +29 -0
+                                    uint64_t addr, uint64_t length,
+                                    uint32_t *out_hw_queue_id, Error **errp)
+{
+    int ret;
+    struct iommu_hw_queue_alloc alloc_hw_queue = {
+        .size = sizeof(alloc_hw_queue),
+        .flags = 0,
+        .viommu_id = viommu_id,
+        .type = data_type,
+        .index = index,
+        .nesting_parent_iova = addr,
+        .length = length,
+    };
+
+    ret = ioctl(be->fd, IOMMU_HW_QUEUE_ALLOC, &alloc_hw_queue);
+
+    trace_iommufd_backend_alloc_hw_queue(be->fd, viommu_id, data_type,
+                                         index, addr, length,
+                                         alloc_hw_queue.out_hw_queue_id, ret);
+    if (ret) {
+        error_setg_errno(errp, errno, "IOMMU_HW_QUEUE_ALLOC failed");
+        return false;
+    }
+
+    g_assert(out_hw_queue_id);
+    *out_hw_queue_id = alloc_hw_queue.out_hw_queue_id;
+    return true;
+}
+
 bool host_iommu_device_iommufd_attach_hwpt(HostIOMMUDeviceIOMMUFD *idev,
                                            uint32_t hwpt_id, Error **errp)
 {
backends/trace-events +0 -0
index 5afa7a40be..a22ad30e55 100644

backends/trace-events iommufd_backend_invalidate_cache(int iommufd, uint32_t id, uint32_t data_type, u · line 27

Quoted 4 lines
backends/trace-events +1 -0
@@ -24,3 +24,4 @@ iommufd_backend_invalidate_cache(int iommufd, uint32_t id, uint32_t data_type, u
 iommufd_backend_alloc_viommu(int iommufd, uint32_t dev_id, uint32_t type, uint32_t hwpt_id, uint32_t viommu_id, int ret) " iommufd=%d type=%u dev_id=%u hwpt_id=%u viommu_id=%u (%d)"
 iommufd_backend_alloc_vdev(int iommufd, uint32_t dev_id, uint32_t viommu_id, uint64_t virt_id, uint32_t vdev_id, int ret) " iommufd=%d dev_id=%u viommu_id=%u virt_id=0x%"PRIx64" vdev_id=%u (%d)"
 iommufd_viommu_alloc_eventq(int iommufd, uint32_t viommu_id, uint32_t type, uint32_t veventq_id, uint32_t veventq_fd, int ret) " iommufd=%d viommu_id=%u type=%u veventq_id=%u veventq_fd=%u (%d)"
+iommufd_backend_alloc_hw_queue(int iommufd, uint32_t viommu_id, uint32_t vqueue_type, uint32_t index, uint64_t addr, uint64_t size, uint32_t vqueue_id, int ret) " iommufd=%d viommu_id=%u vqueue_type=%u index=%u addr=0x%"PRIx64" size=0x%"PRIx64" vqueue_id=%u (%d)"

prefer using length instead of size to stick to the uapi terms

Quoted 19 lines
include/system/iommufd.h +6 -0
index a3e8087b3a..9b8602a558 100644
@@ -63,6 +63,12 @@ typedef struct IOMMUFDVeventq {
     uint32_t veventq_fd;
 } IOMMUFDVeventq;
 
+/* HW queue object for a vIOMMU-specific HW-accelerated queue */
+typedef struct IOMMUFDHWqueue {
+    IOMMUFDViommu *viommu;
+    uint32_t hw_queue_id;
+} IOMMUFDHWqueue;
+
 bool iommufd_backend_connect(IOMMUFDBackend *be, Error **errp);
 void iommufd_backend_disconnect(IOMMUFDBackend *be);
 
@@ -99,6 +105,11 @@ bool iommufd_backend_alloc_veventq(IOMMUFDBackend *be, uint32_t viommu_id,
                                    uint32_t *out_veventq_id,

include/system/iommufd.h bool iommufd_backend_alloc_veventq(IOMMUFDBackend *be, uint32_t viommu_id, · line 115

Quoted 10 lines
include/system/iommufd.h +5 -0
                                    uint32_t *out_veventq_fd, Error **errp);
 
+bool iommufd_backend_alloc_hw_queue(IOMMUFDBackend *be, uint32_t viommu_id,
+                                    uint32_t data_type, uint32_t index,
+                                    uint64_t addr, uint64_t length,
+                                    uint32_t *out_hw_queue_id, Error **errp);
+
 bool iommufd_backend_set_dirty_tracking(IOMMUFDBackend *be, uint32_t hwpt_id,
                                         bool start, Error **errp);
 bool iommufd_backend_get_dirty_bitmap(IOMMUFDBackend *be, uint32_t hwpt_id,

Besides

Eric

#10 lore raw

RFC PATCH 4/16 backends/iommufd: Introduce iommufd_backend_viommu_mmap

In reply to Shameer Kolothum

From: Nicolin Chen <[email protected]>

Add a backend helper to mmap hardware MMIO regions exposed via iommufd for a vIOMMU instance. This allows user space to access HW-accelerated MMIO pages provided by the vIOMMU.

The caller is responsible for unmapping the returned region.

---
 backends/iommufd.c       | 24 ++++++++++++++++++++++++
 backends/trace-events    |  1 +
 include/system/iommufd.h |  4 ++++
 3 files changed, 29 insertions(+)
backends/iommufd.c +24 -0
index a644763239..015e5249d6 100644
@@ -575,6 +575,30 @@ bool iommufd_backend_alloc_hw_queue(IOMMUFDBackend *be, uint32_t viommu_id,
     return true;
 }
 
+/*
+ * Helper to mmap HW MMIO regions exposed via iommufd for a vIOMMU instance.
+ * The caller is responsible for unmapping the mapped region.
+ */
+bool iommufd_backend_viommu_mmap(IOMMUFDBackend *be, uint32_t viommu_id,
+                                 uint64_t size, off_t offset, void **out_ptr,
+                                 Error **errp)
+{
+    g_assert(viommu_id);
+    g_assert(out_ptr);
+
+    *out_ptr = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, be->fd,
+                   offset);
+    if (*out_ptr == MAP_FAILED) {
+        error_setg_errno(errp, errno, "failed to mmap (size=0x%" PRIx64
+                         " offset=0x%" PRIx64 ") for viommu (id=%d)",
+                         size, offset, viommu_id);
+        return false;
+    }
+
+    trace_iommufd_backend_viommu_mmap(be->fd, viommu_id, size, offset);
+    return true;
+}
+
 bool host_iommu_device_iommufd_attach_hwpt(HostIOMMUDeviceIOMMUFD *idev,
                                            uint32_t hwpt_id, Error **errp)
 {
backends/trace-events +1 -0
index a22ad30e55..046f453caa 100644
@@ -25,3 +25,4 @@ iommufd_backend_alloc_viommu(int iommufd, uint32_t dev_id, uint32_t type, uint32
 iommufd_backend_alloc_vdev(int iommufd, uint32_t dev_id, uint32_t viommu_id, uint64_t virt_id, uint32_t vdev_id, int ret) " iommufd=%d dev_id=%u viommu_id=%u virt_id=0x%"PRIx64" vdev_id=%u (%d)"
 iommufd_viommu_alloc_eventq(int iommufd, uint32_t viommu_id, uint32_t type, uint32_t veventq_id, uint32_t veventq_fd, int ret) " iommufd=%d viommu_id=%u type=%u veventq_id=%u veventq_fd=%u (%d)"
 iommufd_backend_alloc_hw_queue(int iommufd, uint32_t viommu_id, uint32_t vqueue_type, uint32_t index, uint64_t addr, uint64_t size, uint32_t vqueue_id, int ret) " iommufd=%d viommu_id=%u vqueue_type=%u index=%u addr=0x%"PRIx64" size=0x%"PRIx64" vqueue_id=%u (%d)"
+iommufd_backend_viommu_mmap(int iommufd, uint32_t viommu_id, uint64_t size, uint64_t offset) " iommufd=%d viommu_id=%u size=0x%"PRIx64" offset=0x%"PRIx64
include/system/iommufd.h +4 -0
index 9b8602a558..e3905c9a40 100644
@@ -110,6 +110,10 @@ bool iommufd_backend_alloc_hw_queue(IOMMUFDBackend *be, uint32_t viommu_id,
                                     uint64_t addr, uint64_t length,
                                     uint32_t *out_hw_queue_id, Error **errp);
 
+bool iommufd_backend_viommu_mmap(IOMMUFDBackend *be, uint32_t viommu_id,
+                                 uint64_t size, off_t offset, void **out_ptr,
+                                 Error **errp);
+
 bool iommufd_backend_set_dirty_tracking(IOMMUFDBackend *be, uint32_t hwpt_id,
                                         bool start, Error **errp);
 bool iommufd_backend_get_dirty_bitmap(IOMMUFDBackend *be, uint32_t hwpt_id,
Signature · 1 line
2.43.0
#11 lore raw

Re: [RFC PATCH 04/16] backends/iommufd: Introduce iommufd_backend_viommu_mmap

Quoted 7 lines · Shameer Kolothum
backends/iommufd.c +7 -0
+bool iommufd_backend_viommu_mmap(IOMMUFDBackend *be, uint32_t viommu_id,
+                                 uint64_t size, off_t offset, void **out_ptr,
+                                 Error **errp)
+{
+    g_assert(viommu_id);
+    g_assert(out_ptr);
+

backends/iommufd.c bool iommufd_backend_alloc_hw_queue(IOMMUFDBackend *be, uint32_t viommu_id, · line 598

Quoted 10 lines
backends/iommufd.c +10 -0
+    *out_ptr = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, be->fd,
+                   offset);
+    if (*out_ptr == MAP_FAILED) {
+        error_setg_errno(errp, errno, "failed to mmap (size=0x%" PRIx64
+                         " offset=0x%" PRIx64 ") for viommu (id=%d)",
+                         size, offset, viommu_id);
+        return false;
+    }
+
+    trace_iommufd_backend_viommu_mmap(be->fd, viommu_id, size, offset);

Like the other two APIs, maybe move the trace next to mmap() and simplify the error_setg_errno()?

Nicolin

Reviewed-by on v1 4/16
#12 lore raw

Re: [RFC PATCH 04/16] backends/iommufd: Introduce iommufd_backend_viommu_mmap

In reply to Shameer Kolothum
Quoted 10 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

Add a backend helper to mmap hardware MMIO regions exposed via iommufd for a vIOMMU instance. This allows user space to access HW-accelerated MMIO pages provided by the vIOMMU.

The caller is responsible for unmapping the returned region.

Eric

Quoted 65 lines
---
 backends/iommufd.c       | 24 ++++++++++++++++++++++++
 backends/trace-events    |  1 +
 include/system/iommufd.h |  4 ++++
 3 files changed, 29 insertions(+)
backends/iommufd.c +24 -0
index a644763239..015e5249d6 100644
@@ -575,6 +575,30 @@ bool iommufd_backend_alloc_hw_queue(IOMMUFDBackend *be, uint32_t viommu_id,
     return true;
 }
 
+/*
+ * Helper to mmap HW MMIO regions exposed via iommufd for a vIOMMU instance.
+ * The caller is responsible for unmapping the mapped region.
+ */
+bool iommufd_backend_viommu_mmap(IOMMUFDBackend *be, uint32_t viommu_id,
+                                 uint64_t size, off_t offset, void **out_ptr,
+                                 Error **errp)
+{
+    g_assert(viommu_id);
+    g_assert(out_ptr);
+
+    *out_ptr = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, be->fd,
+                   offset);
+    if (*out_ptr == MAP_FAILED) {
+        error_setg_errno(errp, errno, "failed to mmap (size=0x%" PRIx64
+                         " offset=0x%" PRIx64 ") for viommu (id=%d)",
+                         size, offset, viommu_id);
+        return false;
+    }
+
+    trace_iommufd_backend_viommu_mmap(be->fd, viommu_id, size, offset);
+    return true;
+}
+
 bool host_iommu_device_iommufd_attach_hwpt(HostIOMMUDeviceIOMMUFD *idev,
                                            uint32_t hwpt_id, Error **errp)
 {
backends/trace-events +1 -0
index a22ad30e55..046f453caa 100644
@@ -25,3 +25,4 @@ iommufd_backend_alloc_viommu(int iommufd, uint32_t dev_id, uint32_t type, uint32
 iommufd_backend_alloc_vdev(int iommufd, uint32_t dev_id, uint32_t viommu_id, uint64_t virt_id, uint32_t vdev_id, int ret) " iommufd=%d dev_id=%u viommu_id=%u virt_id=0x%"PRIx64" vdev_id=%u (%d)"
 iommufd_viommu_alloc_eventq(int iommufd, uint32_t viommu_id, uint32_t type, uint32_t veventq_id, uint32_t veventq_fd, int ret) " iommufd=%d viommu_id=%u type=%u veventq_id=%u veventq_fd=%u (%d)"
 iommufd_backend_alloc_hw_queue(int iommufd, uint32_t viommu_id, uint32_t vqueue_type, uint32_t index, uint64_t addr, uint64_t size, uint32_t vqueue_id, int ret) " iommufd=%d viommu_id=%u vqueue_type=%u index=%u addr=0x%"PRIx64" size=0x%"PRIx64" vqueue_id=%u (%d)"
+iommufd_backend_viommu_mmap(int iommufd, uint32_t viommu_id, uint64_t size, uint64_t offset) " iommufd=%d viommu_id=%u size=0x%"PRIx64" offset=0x%"PRIx64
include/system/iommufd.h +4 -0
index 9b8602a558..e3905c9a40 100644
@@ -110,6 +110,10 @@ bool iommufd_backend_alloc_hw_queue(IOMMUFDBackend *be, uint32_t viommu_id,
                                     uint64_t addr, uint64_t length,
                                     uint32_t *out_hw_queue_id, Error **errp);
 
+bool iommufd_backend_viommu_mmap(IOMMUFDBackend *be, uint32_t viommu_id,
+                                 uint64_t size, off_t offset, void **out_ptr,
+                                 Error **errp);
+
 bool iommufd_backend_set_dirty_tracking(IOMMUFDBackend *be, uint32_t hwpt_id,
                                         bool start, Error **errp);
 bool iommufd_backend_get_dirty_bitmap(IOMMUFDBackend *be, uint32_t hwpt_id,
#13 lore raw

RFC PATCH 5/16 hw/arm/tegra241-cmdqv: Add initial Tegra241 CMDQ-Virtualisation support

In reply to Shameer Kolothum

From: Nicolin Chen <[email protected]>

Introduce initial support for NVIDIA Tegra241 CMDQ-Virtualisation (CMDQV), an extension to SMMUv3 providing virtualizable hardware command queues. This adds the basic MMIO handling, and integration hooks in the SMMUv3 accelerated path. When enabled, the SMMUv3 backend allocates a Tegra241 specific vIOMMU object via IOMMUFD and exposes a CMDQV MMIO region and IRQ to the guest.

The "tegra241-cmdqv" property isn't user visible yet and it will be introduced in a later patch once all the supporting pieces are ready.

---
 hw/arm/Kconfig          |  5 ++++
 hw/arm/meson.build      |  1 +
 hw/arm/smmuv3-accel.c   | 10 +++++--
 hw/arm/smmuv3.c         |  4 +++
 hw/arm/tegra241-cmdqv.c | 65 +++++++++++++++++++++++++++++++++++++++++
 hw/arm/tegra241-cmdqv.h | 40 +++++++++++++++++++++++++
 include/hw/arm/smmuv3.h |  3 ++
 7 files changed, 126 insertions(+), 2 deletions(-)
 create mode 100644 hw/arm/tegra241-cmdqv.c
 create mode 100644 hw/arm/tegra241-cmdqv.h
hw/arm/Kconfig +5 -0
index 702b79a02b..42b6b95285 100644
@@ -37,6 +37,7 @@ config ARM_VIRT
     select VIRTIO_MEM_SUPPORTED
     select ACPI_CXL
     select ACPI_HMAT
+    select TEGRA241_CMDQV
 
 config CUBIEBOARD
     bool
@@ -634,6 +635,10 @@ config ARM_SMMUV3_ACCEL
     bool
     depends on ARM_SMMUV3 && IOMMUFD
 
+config TEGRA241_CMDQV
+    bool
+    depends on ARM_SMMUV3_ACCEL
+
 config FSL_IMX6UL
     bool
     default y
hw/arm/meson.build +1 -0
index c250487e64..4ec91db50a 100644
@@ -86,6 +86,7 @@ arm_common_ss.add(when: 'CONFIG_FSL_IMX8MP', if_true: files('fsl-imx8mp.c'))
 arm_common_ss.add(when: 'CONFIG_FSL_IMX8MP_EVK', if_true: files('imx8mp-evk.c'))
 arm_ss.add(when: 'CONFIG_ARM_SMMUV3', if_true: files('smmuv3.c'))
 arm_ss.add(when: 'CONFIG_ARM_SMMUV3_ACCEL', if_true: files('smmuv3-accel.c'))
+arm_ss.add(when: 'CONFIG_TEGRA241_CMDQV', if_true: files('tegra241-cmdqv.c'))
 arm_common_ss.add(when: 'CONFIG_FSL_IMX6UL', if_true: files('fsl-imx6ul.c', 'mcimx6ul-evk.c'))
 arm_common_ss.add(when: 'CONFIG_NRF51_SOC', if_true: files('nrf51_soc.c'))
 arm_common_ss.add(when: 'CONFIG_XEN', if_true: files(
hw/arm/smmuv3-accel.c +8 -2
index 939898c9b0..e50c4b3bb7 100644
@@ -18,6 +18,7 @@
 
 #include "smmuv3-internal.h"
 #include "smmuv3-accel.h"
+#include "tegra241-cmdqv.h"
 
 /*
  * The root region aliases the global system memory, and shared_as_sysmem
@@ -499,10 +500,15 @@ smmuv3_accel_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
         .ste = { SMMU_STE_VALID, 0x0ULL },
     };
     uint32_t s2_hwpt_id = idev->hwpt_id;
-    uint32_t viommu_id, hwpt_id;
+    uint32_t viommu_id = 0, hwpt_id;
     SMMUv3AccelState *accel;
 
-    if (!iommufd_backend_alloc_viommu(idev->iommufd, idev->devid,
+    if (s->tegra241_cmdqv && !tegra241_cmdqv_alloc_viommu(s, idev, &viommu_id,
+                                                          errp)) {
+        return false;
+    }
+
+    if (!viommu_id && !iommufd_backend_alloc_viommu(idev->iommufd, idev->devid,
                                       IOMMU_VIOMMU_TYPE_ARM_SMMUV3, s2_hwpt_id,
                                       NULL, 0, &viommu_id, errp)) {
         return false;
hw/arm/smmuv3.c +4 -0
index 9b7b85fb49..02e1a925a4 100644
@@ -36,6 +36,7 @@
 #include "smmuv3-accel.h"
 #include "smmuv3-internal.h"
 #include "smmu-internal.h"
+#include "tegra241-cmdqv.h"
 
 #define PTW_RECORD_FAULT(ptw_info, cfg) (((ptw_info).stage == SMMU_STAGE_1 && \
                                         (cfg)->record_faults) || \
@@ -2017,6 +2018,9 @@ static void smmu_realize(DeviceState *d, Error **errp)
 
     smmu_init_irq(s, dev);
     smmuv3_init_id_regs(s);
+    if (s->tegra241_cmdqv) {
+        tegra241_cmdqv_init(s);
+    }
 }
 
 static const VMStateDescription vmstate_smmuv3_queue = {
hw/arm/tegra241-cmdqv.c new file +65 -0

65 lines not on this page · raw message

hw/arm/tegra241-cmdqv.h new file +40 -0
new file mode 100644
index 0000000000..9bc72b24d9
@@ -0,0 +1,40 @@
+/*
+ * Copyright (C) 2025, NVIDIA CORPORATION
+ * NVIDIA Tegra241 CMDQ-Virtualiisation extension for SMMUv3
+ *
+ * Written by Nicolin Chen, Shameer Kolothum
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#ifndef HW_TEGRA241_CMDQV_H
+#define HW_TEGRA241_CMDQV_H
+
+#include CONFIG_DEVICES
+
+#define TEGRA241_CMDQV_IO_LEN 0x50000
+
+typedef struct Tegra241CMDQV {
+    struct iommu_viommu_tegra241_cmdqv cmdqv_data;
+    SMMUv3State *smmu;
+    MemoryRegion mmio_cmdqv;
+    qemu_irq irq;
+} Tegra241CMDQV;
+
+#ifdef CONFIG_TEGRA241_CMDQV
+bool tegra241_cmdqv_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
+                                 uint32_t *out_viommu_id, Error **errp);
+void tegra241_cmdqv_init(SMMUv3State *s);
+#else
+static inline void tegra241_cmdqv_init(SMMUv3State *s)
+{
+}
+static inline bool
+tegra241_cmdqv_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
+                            uint32_t *out_viommu_id, Error **errp)
+{
+    return true;
+}
+#endif
+
+#endif /* HW_TEGRA241_CMDQV_H */
include/hw/arm/smmuv3.h +3 -0
index 2d4970fe19..8e56e480a0 100644
@@ -73,6 +73,9 @@ struct SMMUv3State {
     bool ats;
     uint8_t oas;
     bool pasid;
+    /* Support for NVIDIA Tegra241 SMMU CMDQV extension */
+    struct Tegra241CMDQV *cmdqv;
+    bool tegra241_cmdqv;
 };
 
 typedef enum {
Signature · 1 line
2.43.0
#14 lore raw

Re: [RFC PATCH 05/16] hw/arm/tegra241-cmdqv: Add initial Tegra241 CMDQ-Virtualisation support

hw/arm/tegra241-cmdqv.c line 56

Quoted 8 lines · Shameer Kolothum
hw/arm/tegra241-cmdqv.c new file +8 -0
+void tegra241_cmdqv_init(SMMUv3State *s)
+{
+    SysBusDevice *sbd = SYS_BUS_DEVICE(OBJECT(s));
+    Tegra241CMDQV *cmdqv;
+
+    if (!s->tegra241_cmdqv) {
+        return;
+    }

Maybe g_assert?

hw/arm/tegra241-cmdqv.h line 19

Quoted 3 lines
hw/arm/tegra241-cmdqv.h new file +3 -0
+typedef struct Tegra241CMDQV {
+    struct iommu_viommu_tegra241_cmdqv cmdqv_data;
+    SMMUv3State *smmu;

I see all the cmdqv functions want "smmu->s_accel", so maybe store "s_accel" instead?

hw/arm/tegra241-cmdqv.h line 36

Quoted 13 lines
hw/arm/tegra241-cmdqv.h new file +13 -0
+#ifdef CONFIG_TEGRA241_CMDQV
+bool tegra241_cmdqv_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
+                                 uint32_t *out_viommu_id, Error **errp);
+void tegra241_cmdqv_init(SMMUv3State *s);
+#else
+static inline void tegra241_cmdqv_init(SMMUv3State *s)
+{
+}
+static inline bool
+tegra241_cmdqv_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
+                            uint32_t *out_viommu_id, Error **errp)
+{
+    return true;

Should it return false?

include/hw/arm/smmuv3.h struct SMMUv3State { · line 78

Quoted 10 lines

index 2d4970fe19..8e56e480a0 100644

include/hw/arm/smmuv3.h +3 -0
@@ -73,6 +73,9 @@ struct SMMUv3State {
     bool ats;
     uint8_t oas;
     bool pasid;
+    /* Support for NVIDIA Tegra241 SMMU CMDQV extension */
+    struct Tegra241CMDQV *cmdqv;
+    bool tegra241_cmdqv;

tegra241_cmdqv is a Property, so it has to stay with SMMUv3State.

But "struct Tegra241CMDQV *cmdqv" might be in the SMMUv3AccelState?

Nicolin

#15 lore raw

RE: [RFC PATCH 05/16] hw/arm/tegra241-cmdqv: Add initial Tegra241 CMDQ-Virtualisation support

Quoted 16 lines

-----Original Message-----
From: Nicolin Chen <[email protected]>
Sent: 29 December 2025 19:07
To: Shameer Kolothum <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; [email protected]; [email protected]; Krishnakant Jaju <[email protected]>
Subject: Re: [RFC PATCH 05/16] hw/arm/tegra241-cmdqv: Add initial Tegra241 CMDQ-Virtualisation support

On Wed, Dec 10, 2025 at 01:37:26PM +0000, Shameer Kolothum wrote:

Quoted 10 lines

hw/arm/tegra241-cmdqv.c line 56

Quoted 8 lines
hw/arm/tegra241-cmdqv.c new file +8 -0
+void tegra241_cmdqv_init(SMMUv3State *s)
+{
+    SysBusDevice *sbd = SYS_BUS_DEVICE(OBJECT(s));
+    Tegra241CMDQV *cmdqv;
+
+    if (!s->tegra241_cmdqv) {
+        return;
+    }

Maybe g_assert?

Sure.

Quoted 7 lines

hw/arm/tegra241-cmdqv.h line 19

Quoted 4 lines
hw/arm/tegra241-cmdqv.h new file +3 -0
+typedef struct Tegra241CMDQV {
+    struct iommu_viommu_tegra241_cmdqv cmdqv_data;
+    SMMUv3State *smmu;

I see all the cmdqv functions want "smmu->s_accel", so maybe store "s_accel" instead?

Yes. But see below.

Quoted 12 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+#ifdef CONFIG_TEGRA241_CMDQV
+bool tegra241_cmdqv_alloc_viommu(SMMUv3State *s,

HostIOMMUDeviceIOMMUFD *idev,

hw/arm/tegra241-cmdqv.h

Quoted 8 lines
hw/arm/tegra241-cmdqv.h new file +8 -0
+                                 uint32_t *out_viommu_id, Error **errp);
+void tegra241_cmdqv_init(SMMUv3State *s);
+#else
+static inline void tegra241_cmdqv_init(SMMUv3State *s)
+{
+}
+static inline bool
+tegra241_cmdqv_alloc_viommu(SMMUv3State *s,

HostIOMMUDeviceIOMMUFD *idev,

Quoted 5 lines

hw/arm/tegra241-cmdqv.c line 46

Quoted 3 lines
hw/arm/tegra241-cmdqv.c new file +3 -0
+                            uint32_t *out_viommu_id, Error **errp)
+{
+    return true;

Should it return false?

Hmm..it should be, given how it is invoked here.

Quoted 14 lines

include/hw/arm/smmuv3.h struct SMMUv3State { · line 78

Quoted 11 lines

index 2d4970fe19..8e56e480a0 100644

include/hw/arm/smmuv3.h +3 -0
@@ -73,6 +73,9 @@ struct SMMUv3State {
     bool ats;
     uint8_t oas;
     bool pasid;
+    /* Support for NVIDIA Tegra241 SMMU CMDQV extension */
+    struct Tegra241CMDQV *cmdqv;
+    bool tegra241_cmdqv;

tegra241_cmdqv is a Property, so it has to stay with SMMUv3State.

Quoted 1 line

But "struct Tegra241CMDQV *cmdqv" might be in the SMMUv3AccelState?

I have considered this. Currently, SMMUv3AccelState is allocated in smmuv3_accel_alloc_viommu(), which happens later than tegra241_cmdqv_init().

Changing this would require some rework. I will look into whether we can address that as part of the next respin of the SMMUv3 accel series to make this cleaner.

Thanks,
Shameer

#16 lore raw

Re: [RFC PATCH 05/16] hw/arm/tegra241-cmdqv: Add initial Tegra241 CMDQ-Virtualisation support

In reply to Shameer Kolothum
Quoted 75 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

Introduce initial support for NVIDIA Tegra241 CMDQ-Virtualisation (CMDQV), an extension to SMMUv3 providing virtualizable hardware command queues. This adds the basic MMIO handling, and integration hooks in the SMMUv3 accelerated path. When enabled, the SMMUv3 backend allocates a Tegra241 specific vIOMMU object via IOMMUFD and exposes a CMDQV MMIO region and IRQ to the guest.

The "tegra241-cmdqv" property isn't user visible yet and it will be introduced in a later patch once all the supporting pieces are ready.

---
 hw/arm/Kconfig          |  5 ++++
 hw/arm/meson.build      |  1 +
 hw/arm/smmuv3-accel.c   | 10 +++++--
 hw/arm/smmuv3.c         |  4 +++
 hw/arm/tegra241-cmdqv.c | 65 +++++++++++++++++++++++++++++++++++++++++
 hw/arm/tegra241-cmdqv.h | 40 +++++++++++++++++++++++++
 include/hw/arm/smmuv3.h |  3 ++
 7 files changed, 126 insertions(+), 2 deletions(-)
 create mode 100644 hw/arm/tegra241-cmdqv.c
 create mode 100644 hw/arm/tegra241-cmdqv.h
hw/arm/Kconfig +5 -0
index 702b79a02b..42b6b95285 100644
@@ -37,6 +37,7 @@ config ARM_VIRT
     select VIRTIO_MEM_SUPPORTED
     select ACPI_CXL
     select ACPI_HMAT
+    select TEGRA241_CMDQV
 
 config CUBIEBOARD
     bool
@@ -634,6 +635,10 @@ config ARM_SMMUV3_ACCEL
     bool
     depends on ARM_SMMUV3 && IOMMUFD
 
+config TEGRA241_CMDQV
+    bool
+    depends on ARM_SMMUV3_ACCEL
+
 config FSL_IMX6UL
     bool
     default y
hw/arm/meson.build +1 -0
index c250487e64..4ec91db50a 100644
@@ -86,6 +86,7 @@ arm_common_ss.add(when: 'CONFIG_FSL_IMX8MP', if_true: files('fsl-imx8mp.c'))
 arm_common_ss.add(when: 'CONFIG_FSL_IMX8MP_EVK', if_true: files('imx8mp-evk.c'))
 arm_ss.add(when: 'CONFIG_ARM_SMMUV3', if_true: files('smmuv3.c'))
 arm_ss.add(when: 'CONFIG_ARM_SMMUV3_ACCEL', if_true: files('smmuv3-accel.c'))
+arm_ss.add(when: 'CONFIG_TEGRA241_CMDQV', if_true: files('tegra241-cmdqv.c'))
 arm_common_ss.add(when: 'CONFIG_FSL_IMX6UL', if_true: files('fsl-imx6ul.c', 'mcimx6ul-evk.c'))
 arm_common_ss.add(when: 'CONFIG_NRF51_SOC', if_true: files('nrf51_soc.c'))
 arm_common_ss.add(when: 'CONFIG_XEN', if_true: files(
hw/arm/smmuv3-accel.c +1 -0
index 939898c9b0..e50c4b3bb7 100644
@@ -18,6 +18,7 @@
 
 #include "smmuv3-internal.h"
 #include "smmuv3-accel.h"
+#include "tegra241-cmdqv.h"
 
 /*
  * The root region aliases the global system memory, and shared_as_sysmem
@@ -499,10 +500,15 @@ smmuv3_accel_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
         .ste = { SMMU_STE_VALID, 0x0ULL },

hw/arm/smmuv3-accel.c smmuv3_accel_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev, · line 508

Quoted 10 lines
hw/arm/smmuv3-accel.c +4 -2
     };
     uint32_t s2_hwpt_id = idev->hwpt_id;
-    uint32_t viommu_id, hwpt_id;
+    uint32_t viommu_id = 0, hwpt_id;
     SMMUv3AccelState *accel;
 
-    if (!iommufd_backend_alloc_viommu(idev->iommufd, idev->devid,
+    if (s->tegra241_cmdqv && !tegra241_cmdqv_alloc_viommu(s, idev, &viommu_id,
+                                                          errp)) {
+        return false;

I am confused. In tegra241_cmdqv_alloc_viommu() it returns false if alloc_viommu fails. but you seem to reset s->tegra241_cmdqv as if you would fall back to non cmdqv setup. What do you try do, fallback or execute either tegra241 code or default code. Or maybe I misunderstand the uapi call sequence?

I would not reset a property value in general under the hood. If the end user has set up this option, I guess he expects it to be enabled when he queries it back, no?

hw/arm/smmuv3-accel.c smmuv3_accel_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev, · line 513

Quoted 5 lines
hw/arm/smmuv3-accel.c +3 -0
+    }
+
+    if (!viommu_id && !iommufd_backend_alloc_viommu(idev->iommufd, idev->devid,
                                       IOMMU_VIOMMU_TYPE_ARM_SMMUV3, s2_hwpt_id,
                                       NULL, 0, &viommu_id, errp)) {

If this is a specialization of alloc_viommu code, it generally points to a class or ops interface. You may face such kind of comments later on so better to justify that choice or adopt a new one ;-)
same for init which is overloaded compared to original implementation.

Quoted 145 lines
hw/arm/smmuv3-accel.c +0 -0
         return false;
hw/arm/smmuv3.c +4 -0
index 9b7b85fb49..02e1a925a4 100644
@@ -36,6 +36,7 @@
 #include "smmuv3-accel.h"
 #include "smmuv3-internal.h"
 #include "smmu-internal.h"
+#include "tegra241-cmdqv.h"
 
 #define PTW_RECORD_FAULT(ptw_info, cfg) (((ptw_info).stage == SMMU_STAGE_1 && \
                                         (cfg)->record_faults) || \
@@ -2017,6 +2018,9 @@ static void smmu_realize(DeviceState *d, Error **errp)
 
     smmu_init_irq(s, dev);
     smmuv3_init_id_regs(s);
+    if (s->tegra241_cmdqv) {
+        tegra241_cmdqv_init(s);
+    }
 }
 
 static const VMStateDescription vmstate_smmuv3_queue = {
hw/arm/tegra241-cmdqv.c new file +65 -0
new file mode 100644
index 0000000000..899325877e
@@ -0,0 +1,65 @@
+/*
+ * Copyright (C) 2025, NVIDIA CORPORATION
+ * NVIDIA Tegra241 CMDQ-Virtualization extension for SMMUv3
+ *
+ * Written by Nicolin Chen, Shameer Kolothum
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#include "qemu/osdep.h"
+
+#include "hw/arm/smmuv3.h"
+#include "smmuv3-accel.h"
+#include "tegra241-cmdqv.h"
+
+static uint64_t tegra241_cmdqv_read(void *opaque, hwaddr offset, unsigned size)
+{
+    return 0;
+}
+
+static void tegra241_cmdqv_write(void *opaque, hwaddr offset, uint64_t value,
+                                 unsigned size)
+{
+}
+
+static const MemoryRegionOps mmio_cmdqv_ops = {
+    .read = tegra241_cmdqv_read,
+    .write = tegra241_cmdqv_write,
+    .endianness = DEVICE_LITTLE_ENDIAN,
+};
+
+bool tegra241_cmdqv_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
+                                 uint32_t *out_viommu_id, Error **errp)
+{
+    Tegra241CMDQV *cmdqv = s->cmdqv;
+
+    if (!iommufd_backend_alloc_viommu(idev->iommufd, idev->devid,
+                                      IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV,
+                                      idev->hwpt_id, &cmdqv->cmdqv_data,
+                                      sizeof(cmdqv->cmdqv_data), out_viommu_id,
+                                      errp)) {
+        error_append_hint(errp, "NVIDIA Tegra241 CMDQV is unsupported");
+        s->tegra241_cmdqv = false;
+        return false;
+    }
+    return true;
+}
+
+void tegra241_cmdqv_init(SMMUv3State *s)
+{
+    SysBusDevice *sbd = SYS_BUS_DEVICE(OBJECT(s));
+    Tegra241CMDQV *cmdqv;
+
+    if (!s->tegra241_cmdqv) {
+        return;
+    }
+
+    cmdqv = g_new0(Tegra241CMDQV, 1);
+    memory_region_init_io(&cmdqv->mmio_cmdqv, OBJECT(s), &mmio_cmdqv_ops, cmdqv,
+                          "tegra241-cmdqv", TEGRA241_CMDQV_IO_LEN);
+    sysbus_init_mmio(sbd, &cmdqv->mmio_cmdqv);
+    sysbus_init_irq(sbd, &cmdqv->irq);
+    cmdqv->smmu = s;
+    s->cmdqv = cmdqv;
+}
hw/arm/tegra241-cmdqv.h new file +40 -0
new file mode 100644
index 0000000000..9bc72b24d9
@@ -0,0 +1,40 @@
+/*
+ * Copyright (C) 2025, NVIDIA CORPORATION
+ * NVIDIA Tegra241 CMDQ-Virtualiisation extension for SMMUv3
+ *
+ * Written by Nicolin Chen, Shameer Kolothum
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#ifndef HW_TEGRA241_CMDQV_H
+#define HW_TEGRA241_CMDQV_H
+
+#include CONFIG_DEVICES
+
+#define TEGRA241_CMDQV_IO_LEN 0x50000
+
+typedef struct Tegra241CMDQV {
+    struct iommu_viommu_tegra241_cmdqv cmdqv_data;
+    SMMUv3State *smmu;
+    MemoryRegion mmio_cmdqv;
+    qemu_irq irq;
+} Tegra241CMDQV;
+
+#ifdef CONFIG_TEGRA241_CMDQV
+bool tegra241_cmdqv_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
+                                 uint32_t *out_viommu_id, Error **errp);
+void tegra241_cmdqv_init(SMMUv3State *s);
+#else
+static inline void tegra241_cmdqv_init(SMMUv3State *s)
+{
+}
+static inline bool
+tegra241_cmdqv_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
+                            uint32_t *out_viommu_id, Error **errp)
+{
+    return true;
+}
+#endif
+
+#endif /* HW_TEGRA241_CMDQV_H */
include/hw/arm/smmuv3.h +0 -0
index 2d4970fe19..8e56e480a0 100644

include/hw/arm/smmuv3.h struct SMMUv3State { · line 81

Quoted 9 lines
include/hw/arm/smmuv3.h +3 -0
@@ -73,6 +73,9 @@ struct SMMUv3State {
     bool ats;
     uint8_t oas;
     bool pasid;
+    /* Support for NVIDIA Tegra241 SMMU CMDQV extension */
+    struct Tegra241CMDQV *cmdqv;
+    bool tegra241_cmdqv;
 };
 
 typedef enum {

Thanks

Eric

#17 lore raw

RE: [RFC PATCH 05/16] hw/arm/tegra241-cmdqv: Add initial Tegra241 CMDQ-Virtualisation support

Hi Eric,

Quoted 112 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 12 January 2026 15:29
To: Shameer Kolothum <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; [email protected]; [email protected]; Krishnakant Jaju <[email protected]>
Subject: Re: [RFC PATCH 05/16] hw/arm/tegra241-cmdqv: Add initial Tegra241 CMDQ-Virtualisation support

External email: Use caution opening links or attachments

Quoted 3 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

Introduce initial support for NVIDIA Tegra241 CMDQ-Virtualisation

(CMDQV),

Quoted 1 line

an extension to SMMUv3 providing virtualizable hardware command

queues.

Quoted 3 lines

This adds the basic MMIO handling, and integration hooks in the SMMUv3 accelerated path. When enabled, the SMMUv3 backend allocates a Tegra241 specific vIOMMU object via IOMMUFD and exposes a CMDQV MMIO region

and

Quoted 5 lines

IRQ to the guest.

The "tegra241-cmdqv" property isn't user visible yet and it will be introduced in a later patch once all the supporting pieces are ready.

Quoted 8 lines
---
 hw/arm/Kconfig          |  5 ++++
 hw/arm/meson.build      |  1 +
 hw/arm/smmuv3-accel.c   | 10 +++++--
 hw/arm/smmuv3.c         |  4 +++
 hw/arm/tegra241-cmdqv.c | 65

+++++++++++++++++++++++++++++++++++++++++

hw/arm/meson.build arm_common_ss.add(when: 'CONFIG_FSL_IMX8MP',

Quoted 34 lines
 hw/arm/tegra241-cmdqv.h | 40 +++++++++++++++++++++++++
 include/hw/arm/smmuv3.h |  3 ++
 7 files changed, 126 insertions(+), 2 deletions(-)
 create mode 100644 hw/arm/tegra241-cmdqv.c
 create mode 100644 hw/arm/tegra241-cmdqv.h
hw/arm/Kconfig +5 -0
index 702b79a02b..42b6b95285 100644
@@ -37,6 +37,7 @@ config ARM_VIRT
     select VIRTIO_MEM_SUPPORTED
     select ACPI_CXL
     select ACPI_HMAT
+    select TEGRA241_CMDQV
 
 config CUBIEBOARD
     bool
@@ -634,6 +635,10 @@ config ARM_SMMUV3_ACCEL
     bool
     depends on ARM_SMMUV3 && IOMMUFD
 
+config TEGRA241_CMDQV
+    bool
+    depends on ARM_SMMUV3_ACCEL
+
 config FSL_IMX6UL
     bool
     default y
hw/arm/meson.build +0 -0
index c250487e64..4ec91db50a 100644
@@ -86,6 +86,7 @@ arm_common_ss.add(when: 'CONFIG_FSL_IMX8MP',

if_true: files('fsl-imx8mp.c'))

hw/arm/meson.build arm_common_ss.add(when: 'CONFIG_FSL_IMX8MP', · line 86

Quoted 1 line
hw/arm/meson.build +0 -0
 arm_common_ss.add(when: 'CONFIG_FSL_IMX8MP_EVK', if_true:

files('imx8mp-evk.c'))

hw/arm/meson.build arm_common_ss.add(when: 'CONFIG_FSL_IMX8MP', · line 88

Quoted 2 lines
hw/arm/meson.build +0 -0
 arm_ss.add(when: 'CONFIG_ARM_SMMUV3', if_true: files('smmuv3.c'))
 arm_ss.add(when: 'CONFIG_ARM_SMMUV3_ACCEL', if_true:

files('smmuv3-accel.c'))

hw/arm/meson.build arm_common_ss.add(when: 'CONFIG_FSL_IMX8MP', · line 89

Quoted 1 line
hw/arm/meson.build +1 -0
+arm_ss.add(when: 'CONFIG_TEGRA241_CMDQV', if_true: files('tegra241-

cmdqv.c'))

hw/arm/meson.build arm_common_ss.add(when: 'CONFIG_FSL_IMX8MP', · line 90

Quoted 1 line
hw/arm/meson.build +0 -0
 arm_common_ss.add(when: 'CONFIG_FSL_IMX6UL', if_true: files('fsl-

imx6ul.c', 'mcimx6ul-evk.c'))

hw/arm/meson.build arm_common_ss.add(when: 'CONFIG_FSL_IMX8MP', · line 91

Quoted 1 line
hw/arm/meson.build +0 -0
 arm_common_ss.add(when: 'CONFIG_NRF51_SOC', if_true:

files('nrf51_soc.c'))

hw/arm/smmuv3-accel.c line 24

Quoted 13 lines
hw/arm/meson.build +0 -0
 arm_common_ss.add(when: 'CONFIG_XEN', if_true: files(
hw/arm/smmuv3-accel.c +1 -0
index 939898c9b0..e50c4b3bb7 100644
@@ -18,6 +18,7 @@
 
 #include "smmuv3-internal.h"
 #include "smmuv3-accel.h"
+#include "tegra241-cmdqv.h"
 
 /*
  * The root region aliases the global system memory, and

shared_as_sysmem

hw/arm/smmuv3-accel.c smmuv3_accel_alloc_viommu(SMMUv3State *s,

Quoted 1 line
hw/arm/smmuv3-accel.c +0 -0
@@ -499,10 +500,15 @@ smmuv3_accel_alloc_viommu(SMMUv3State *s,

HostIOMMUDeviceIOMMUFD *idev,

hw/arm/smmuv3-accel.c smmuv3_accel_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,

Quoted 9 lines
hw/arm/smmuv3-accel.c +2 -2
         .ste = { SMMU_STE_VALID, 0x0ULL },
     };
     uint32_t s2_hwpt_id = idev->hwpt_id;
-    uint32_t viommu_id, hwpt_id;
+    uint32_t viommu_id = 0, hwpt_id;
     SMMUv3AccelState *accel;
 
-    if (!iommufd_backend_alloc_viommu(idev->iommufd, idev->devid,
+    if (s->tegra241_cmdqv && !tegra241_cmdqv_alloc_viommu(s, idev,
Quoted 8 lines

&viommu_id,

hw/arm/smmuv3-accel.c smmuv3_accel_alloc_viommu(SMMUv3State *s, · line 508

Quoted 2 lines
hw/arm/smmuv3-accel.c +2 -0
+                                                          errp)) {
+        return false;

I am confused. In tegra241_cmdqv_alloc_viommu() it returns false if alloc_viommu fails. but you seem to reset s->tegra241_cmdqv as if you would fall back to non cmdqv setup. What do you try do, fallback or execute either tegra241 code or default code. Or maybe I misunderstand the uapi call sequence?

No fallback intended. Currently, if the user has enabled tegra241_cmdqv and tegra241_cmdqv_alloc_viommu() fails, we fail the device init. Sorry about that to reset s->tegra241_cmdqv and !viommu_id logic, , that was a left over logic from previous internal branch I had.

@Nicolin, is there any such requirement for a fallback in this case?

Quoted 3 lines

I would not reset a property value in general under the hood. If the end user has set up this option, I guess he expects it to be enabled when he queries it back, no?

Yes, that’s the idea.

Quoted 11 lines

hw/arm/smmuv3-accel.c smmuv3_accel_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,

Quoted 3 lines
hw/arm/smmuv3-accel.c +3 -0
+    }
+
+    if (!viommu_id && !iommufd_backend_alloc_viommu(idev->iommufd,

idev->devid,

hw/arm/smmuv3-accel.c smmuv3_accel_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev, · line 513

Quoted 2 lines
hw/arm/smmuv3-accel.c +0 -0
                                       IOMMU_VIOMMU_TYPE_ARM_SMMUV3, s2_hwpt_id,
                                       NULL, 0, &viommu_id, errp)) {

If this is a specialization of alloc_viommu code, it generally points to a class or ops interface. You may face such kind of comments later on so better to justify that choice or adopt a new one ;-)
same for init which is overloaded compared to original implementation.

I agree. I will take another look at this. Also, this is where any help would be really appreciated 😊. Please let me know if you have some ideas on how we can improve this.

Thanks,
Shameer

#18 lore raw

Re: [RFC PATCH 05/16] hw/arm/tegra241-cmdqv: Add initial Tegra241 CMDQ-Virtualisation support

Quoted 12 lines · Shameer Kolothum
Quoted 11 lines

Review

Quoted 2 lines
(quoted diff) +1 -1
-    if (!iommufd_backend_alloc_viommu(idev->iommufd, idev->devid,
+    if (s->tegra241_cmdqv && !tegra241_cmdqv_alloc_viommu(s, idev,

&viommu_id,

Review

Quoted 2 lines
(quoted diff) +2 -0
+                                                          errp)) {
+        return false;

I am confused. In tegra241_cmdqv_alloc_viommu() it returns false if alloc_viommu fails. but you seem to reset s->tegra241_cmdqv as if you would fall back to non cmdqv setup. What do you try do, fallback or execute either tegra241 code or default code. Or maybe I misunderstand the uapi call sequence?

Quoted 6 lines

No fallback intended. Currently, if the user has enabled tegra241_cmdqv and tegra241_cmdqv_alloc_viommu() fails, we fail the device init. Sorry about that to reset s->tegra241_cmdqv and !viommu_id logic, , that was a left over logic from previous internal branch I had.

@Nicolin, is there any such requirement for a fallback in this case?

Likely no. I agree that we should do in the cleaner way.

Nicolin

#19 lore raw

Re: [RFC PATCH 05/16] hw/arm/tegra241-cmdqv: Add initial Tegra241 CMDQ-Virtualisation support

In reply to Shameer Kolothum

Hi Shameer,

Quoted 200 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

Introduce initial support for NVIDIA Tegra241 CMDQ-Virtualisation (CMDQV), an extension to SMMUv3 providing virtualizable hardware command queues. This adds the basic MMIO handling, and integration hooks in the SMMUv3 accelerated path. When enabled, the SMMUv3 backend allocates a Tegra241 specific vIOMMU object via IOMMUFD and exposes a CMDQV MMIO region and IRQ to the guest.

The "tegra241-cmdqv" property isn't user visible yet and it will be introduced in a later patch once all the supporting pieces are ready.

---
 hw/arm/Kconfig          |  5 ++++
 hw/arm/meson.build      |  1 +
 hw/arm/smmuv3-accel.c   | 10 +++++--
 hw/arm/smmuv3.c         |  4 +++
 hw/arm/tegra241-cmdqv.c | 65 +++++++++++++++++++++++++++++++++++++++++
 hw/arm/tegra241-cmdqv.h | 40 +++++++++++++++++++++++++
 include/hw/arm/smmuv3.h |  3 ++
 7 files changed, 126 insertions(+), 2 deletions(-)
 create mode 100644 hw/arm/tegra241-cmdqv.c
 create mode 100644 hw/arm/tegra241-cmdqv.h
hw/arm/Kconfig +5 -0
index 702b79a02b..42b6b95285 100644
@@ -37,6 +37,7 @@ config ARM_VIRT
     select VIRTIO_MEM_SUPPORTED
     select ACPI_CXL
     select ACPI_HMAT
+    select TEGRA241_CMDQV
 
 config CUBIEBOARD
     bool
@@ -634,6 +635,10 @@ config ARM_SMMUV3_ACCEL
     bool
     depends on ARM_SMMUV3 && IOMMUFD
 
+config TEGRA241_CMDQV
+    bool
+    depends on ARM_SMMUV3_ACCEL
+
 config FSL_IMX6UL
     bool
     default y
hw/arm/meson.build +1 -0
index c250487e64..4ec91db50a 100644
@@ -86,6 +86,7 @@ arm_common_ss.add(when: 'CONFIG_FSL_IMX8MP', if_true: files('fsl-imx8mp.c'))
 arm_common_ss.add(when: 'CONFIG_FSL_IMX8MP_EVK', if_true: files('imx8mp-evk.c'))
 arm_ss.add(when: 'CONFIG_ARM_SMMUV3', if_true: files('smmuv3.c'))
 arm_ss.add(when: 'CONFIG_ARM_SMMUV3_ACCEL', if_true: files('smmuv3-accel.c'))
+arm_ss.add(when: 'CONFIG_TEGRA241_CMDQV', if_true: files('tegra241-cmdqv.c'))
 arm_common_ss.add(when: 'CONFIG_FSL_IMX6UL', if_true: files('fsl-imx6ul.c', 'mcimx6ul-evk.c'))
 arm_common_ss.add(when: 'CONFIG_NRF51_SOC', if_true: files('nrf51_soc.c'))
 arm_common_ss.add(when: 'CONFIG_XEN', if_true: files(
hw/arm/smmuv3-accel.c +8 -2
index 939898c9b0..e50c4b3bb7 100644
@@ -18,6 +18,7 @@
 
 #include "smmuv3-internal.h"
 #include "smmuv3-accel.h"
+#include "tegra241-cmdqv.h"
 
 /*
  * The root region aliases the global system memory, and shared_as_sysmem
@@ -499,10 +500,15 @@ smmuv3_accel_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
         .ste = { SMMU_STE_VALID, 0x0ULL },
     };
     uint32_t s2_hwpt_id = idev->hwpt_id;
-    uint32_t viommu_id, hwpt_id;
+    uint32_t viommu_id = 0, hwpt_id;
     SMMUv3AccelState *accel;
 
-    if (!iommufd_backend_alloc_viommu(idev->iommufd, idev->devid,
+    if (s->tegra241_cmdqv && !tegra241_cmdqv_alloc_viommu(s, idev, &viommu_id,
+                                                          errp)) {
+        return false;
+    }
+
+    if (!viommu_id && !iommufd_backend_alloc_viommu(idev->iommufd, idev->devid,
                                       IOMMU_VIOMMU_TYPE_ARM_SMMUV3, s2_hwpt_id,
                                       NULL, 0, &viommu_id, errp)) {
         return false;
hw/arm/smmuv3.c +4 -0
index 9b7b85fb49..02e1a925a4 100644
@@ -36,6 +36,7 @@
 #include "smmuv3-accel.h"
 #include "smmuv3-internal.h"
 #include "smmu-internal.h"
+#include "tegra241-cmdqv.h"
 
 #define PTW_RECORD_FAULT(ptw_info, cfg) (((ptw_info).stage == SMMU_STAGE_1 && \
                                         (cfg)->record_faults) || \
@@ -2017,6 +2018,9 @@ static void smmu_realize(DeviceState *d, Error **errp)
 
     smmu_init_irq(s, dev);
     smmuv3_init_id_regs(s);
+    if (s->tegra241_cmdqv) {
+        tegra241_cmdqv_init(s);
+    }
 }
 
 static const VMStateDescription vmstate_smmuv3_queue = {
hw/arm/tegra241-cmdqv.c new file +65 -0
new file mode 100644
index 0000000000..899325877e
@@ -0,0 +1,65 @@
+/*
+ * Copyright (C) 2025, NVIDIA CORPORATION
+ * NVIDIA Tegra241 CMDQ-Virtualization extension for SMMUv3
+ *
+ * Written by Nicolin Chen, Shameer Kolothum
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#include "qemu/osdep.h"
+
+#include "hw/arm/smmuv3.h"
+#include "smmuv3-accel.h"
+#include "tegra241-cmdqv.h"
+
+static uint64_t tegra241_cmdqv_read(void *opaque, hwaddr offset, unsigned size)
+{
+    return 0;
+}
+
+static void tegra241_cmdqv_write(void *opaque, hwaddr offset, uint64_t value,
+                                 unsigned size)
+{
+}
+
+static const MemoryRegionOps mmio_cmdqv_ops = {
+    .read = tegra241_cmdqv_read,
+    .write = tegra241_cmdqv_write,
+    .endianness = DEVICE_LITTLE_ENDIAN,
+};
+
+bool tegra241_cmdqv_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
+                                 uint32_t *out_viommu_id, Error **errp)
+{
+    Tegra241CMDQV *cmdqv = s->cmdqv;
+
+    if (!iommufd_backend_alloc_viommu(idev->iommufd, idev->devid,
+                                      IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV,
+                                      idev->hwpt_id, &cmdqv->cmdqv_data,
+                                      sizeof(cmdqv->cmdqv_data), out_viommu_id,
+                                      errp)) {
+        error_append_hint(errp, "NVIDIA Tegra241 CMDQV is unsupported");
+        s->tegra241_cmdqv = false;
+        return false;
+    }
+    return true;
+}
+
+void tegra241_cmdqv_init(SMMUv3State *s)
+{
+    SysBusDevice *sbd = SYS_BUS_DEVICE(OBJECT(s));
+    Tegra241CMDQV *cmdqv;
+
+    if (!s->tegra241_cmdqv) {
+        return;
+    }
+
+    cmdqv = g_new0(Tegra241CMDQV, 1);
+    memory_region_init_io(&cmdqv->mmio_cmdqv, OBJECT(s), &mmio_cmdqv_ops, cmdqv,
+                          "tegra241-cmdqv", TEGRA241_CMDQV_IO_LEN);
+    sysbus_init_mmio(sbd, &cmdqv->mmio_cmdqv);
+    sysbus_init_irq(sbd, &cmdqv->irq);
+    cmdqv->smmu = s;
+    s->cmdqv = cmdqv;
+}
hw/arm/tegra241-cmdqv.h new file +10 -0
new file mode 100644
index 0000000000..9bc72b24d9
@@ -0,0 +1,40 @@
+/*
+ * Copyright (C) 2025, NVIDIA CORPORATION
+ * NVIDIA Tegra241 CMDQ-Virtualiisation extension for SMMUv3
+ *
+ * Written by Nicolin Chen, Shameer Kolothum
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#ifndef HW_TEGRA241_CMDQV_H

hw/arm/tegra241-cmdqv.h line 20

Quoted 10 lines
hw/arm/tegra241-cmdqv.h new file +10 -0
+#define HW_TEGRA241_CMDQV_H
+
+#include CONFIG_DEVICES
+
+#define TEGRA241_CMDQV_IO_LEN 0x50000
+
+typedef struct Tegra241CMDQV {
+    struct iommu_viommu_tegra241_cmdqv cmdqv_data;
+    SMMUv3State *smmu;
+    MemoryRegion mmio_cmdqv;

please use a _mr suffix. It eases the review of subsequent patches

Eric

Quoted 34 lines
hw/arm/tegra241-cmdqv.h new file +20 -0
+    qemu_irq irq;
+} Tegra241CMDQV;
+
+#ifdef CONFIG_TEGRA241_CMDQV
+bool tegra241_cmdqv_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
+                                 uint32_t *out_viommu_id, Error **errp);
+void tegra241_cmdqv_init(SMMUv3State *s);
+#else
+static inline void tegra241_cmdqv_init(SMMUv3State *s)
+{
+}
+static inline bool
+tegra241_cmdqv_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
+                            uint32_t *out_viommu_id, Error **errp)
+{
+    return true;
+}
+#endif
+
+#endif /* HW_TEGRA241_CMDQV_H */
include/hw/arm/smmuv3.h +3 -0
index 2d4970fe19..8e56e480a0 100644
@@ -73,6 +73,9 @@ struct SMMUv3State {
     bool ats;
     uint8_t oas;
     bool pasid;
+    /* Support for NVIDIA Tegra241 SMMU CMDQV extension */
+    struct Tegra241CMDQV *cmdqv;
+    bool tegra241_cmdqv;
 };
 
 typedef enum {
#20 lore raw

Re: [RFC PATCH 05/16] hw/arm/tegra241-cmdqv: Add initial Tegra241 CMDQ-Virtualisation support

In reply to Shameer Kolothum
Quoted 3 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

Introduce initial support for NVIDIA Tegra241 CMDQ-Virtualisation (CMDQV),

well this is rather a skeleton and at this stage it does not introduce any support

Quoted 5 lines

an extension to SMMUv3 providing virtualizable hardware command queues. This adds the basic MMIO handling, and integration hooks in the SMMUv3 accelerated path. When enabled, the SMMUv3 backend allocates a Tegra241 specific vIOMMU object via IOMMUFD and exposes a CMDQV MMIO region and IRQ to the guest.

I would elaborate on the nature of region and interrupt here.

Quoted 3 lines

The "tegra241-cmdqv" property isn't user visible yet and it will be introduced in a later patch once all the supporting pieces are ready.

I would suggest "tegra241_cmdqv will be set by an SMMU property that does not exist yet. As such it is currently unset."

Quoted 108 lines
---
 hw/arm/Kconfig          |  5 ++++
 hw/arm/meson.build      |  1 +
 hw/arm/smmuv3-accel.c   | 10 +++++--
 hw/arm/smmuv3.c         |  4 +++
 hw/arm/tegra241-cmdqv.c | 65 +++++++++++++++++++++++++++++++++++++++++
 hw/arm/tegra241-cmdqv.h | 40 +++++++++++++++++++++++++
 include/hw/arm/smmuv3.h |  3 ++
 7 files changed, 126 insertions(+), 2 deletions(-)
 create mode 100644 hw/arm/tegra241-cmdqv.c
 create mode 100644 hw/arm/tegra241-cmdqv.h
hw/arm/Kconfig +5 -0
index 702b79a02b..42b6b95285 100644
@@ -37,6 +37,7 @@ config ARM_VIRT
     select VIRTIO_MEM_SUPPORTED
     select ACPI_CXL
     select ACPI_HMAT
+    select TEGRA241_CMDQV
 
 config CUBIEBOARD
     bool
@@ -634,6 +635,10 @@ config ARM_SMMUV3_ACCEL
     bool
     depends on ARM_SMMUV3 && IOMMUFD
 
+config TEGRA241_CMDQV
+    bool
+    depends on ARM_SMMUV3_ACCEL
+
 config FSL_IMX6UL
     bool
     default y
hw/arm/meson.build +1 -0
index c250487e64..4ec91db50a 100644
@@ -86,6 +86,7 @@ arm_common_ss.add(when: 'CONFIG_FSL_IMX8MP', if_true: files('fsl-imx8mp.c'))
 arm_common_ss.add(when: 'CONFIG_FSL_IMX8MP_EVK', if_true: files('imx8mp-evk.c'))
 arm_ss.add(when: 'CONFIG_ARM_SMMUV3', if_true: files('smmuv3.c'))
 arm_ss.add(when: 'CONFIG_ARM_SMMUV3_ACCEL', if_true: files('smmuv3-accel.c'))
+arm_ss.add(when: 'CONFIG_TEGRA241_CMDQV', if_true: files('tegra241-cmdqv.c'))
 arm_common_ss.add(when: 'CONFIG_FSL_IMX6UL', if_true: files('fsl-imx6ul.c', 'mcimx6ul-evk.c'))
 arm_common_ss.add(when: 'CONFIG_NRF51_SOC', if_true: files('nrf51_soc.c'))
 arm_common_ss.add(when: 'CONFIG_XEN', if_true: files(
hw/arm/smmuv3-accel.c +8 -2
index 939898c9b0..e50c4b3bb7 100644
@@ -18,6 +18,7 @@
 
 #include "smmuv3-internal.h"
 #include "smmuv3-accel.h"
+#include "tegra241-cmdqv.h"
 
 /*
  * The root region aliases the global system memory, and shared_as_sysmem
@@ -499,10 +500,15 @@ smmuv3_accel_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
         .ste = { SMMU_STE_VALID, 0x0ULL },
     };
     uint32_t s2_hwpt_id = idev->hwpt_id;
-    uint32_t viommu_id, hwpt_id;
+    uint32_t viommu_id = 0, hwpt_id;
     SMMUv3AccelState *accel;
 
-    if (!iommufd_backend_alloc_viommu(idev->iommufd, idev->devid,
+    if (s->tegra241_cmdqv && !tegra241_cmdqv_alloc_viommu(s, idev, &viommu_id,
+                                                          errp)) {
+        return false;
+    }
+
+    if (!viommu_id && !iommufd_backend_alloc_viommu(idev->iommufd, idev->devid,
                                       IOMMU_VIOMMU_TYPE_ARM_SMMUV3, s2_hwpt_id,
                                       NULL, 0, &viommu_id, errp)) {
         return false;
hw/arm/smmuv3.c +4 -0
index 9b7b85fb49..02e1a925a4 100644
@@ -36,6 +36,7 @@
 #include "smmuv3-accel.h"
 #include "smmuv3-internal.h"
 #include "smmu-internal.h"
+#include "tegra241-cmdqv.h"
 
 #define PTW_RECORD_FAULT(ptw_info, cfg) (((ptw_info).stage == SMMU_STAGE_1 && \
                                         (cfg)->record_faults) || \
@@ -2017,6 +2018,9 @@ static void smmu_realize(DeviceState *d, Error **errp)
 
     smmu_init_irq(s, dev);
     smmuv3_init_id_regs(s);
+    if (s->tegra241_cmdqv) {
+        tegra241_cmdqv_init(s);
+    }
 }
 
 static const VMStateDescription vmstate_smmuv3_queue = {
hw/arm/tegra241-cmdqv.c new file +0 -0
new file mode 100644
index 0000000000..899325877e

hw/arm/tegra241-cmdqv.c line 2

Quoted 2 lines
hw/arm/tegra241-cmdqv.c new file +2 -0
@@ -0,0 +1,65 @@
+/*
+ * Copyright (C) 2025, NVIDIA CORPORATION

nit: s/2025/2026

Quoted 30 lines
hw/arm/tegra241-cmdqv.c new file +30 -0
+ * NVIDIA Tegra241 CMDQ-Virtualization extension for SMMUv3
+ *
+ * Written by Nicolin Chen, Shameer Kolothum
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#include "qemu/osdep.h"
+
+#include "hw/arm/smmuv3.h"
+#include "smmuv3-accel.h"
+#include "tegra241-cmdqv.h"
+
+static uint64_t tegra241_cmdqv_read(void *opaque, hwaddr offset, unsigned size)
+{
+    return 0;
+}
+
+static void tegra241_cmdqv_write(void *opaque, hwaddr offset, uint64_t value,
+                                 unsigned size)
+{
+}
+
+static const MemoryRegionOps mmio_cmdqv_ops = {
+    .read = tegra241_cmdqv_read,
+    .write = tegra241_cmdqv_write,
+    .endianness = DEVICE_LITTLE_ENDIAN,
+};
+
+bool tegra241_cmdqv_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,

hw/arm/tegra241-cmdqv.c line 42

Quoted 10 lines
hw/arm/tegra241-cmdqv.c new file +10 -0
+                                 uint32_t *out_viommu_id, Error **errp)
+{
+    Tegra241CMDQV *cmdqv = s->cmdqv;
+
+    if (!iommufd_backend_alloc_viommu(idev->iommufd, idev->devid,
+                                      IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV,
+                                      idev->hwpt_id, &cmdqv->cmdqv_data,
+                                      sizeof(cmdqv->cmdqv_data), out_viommu_id,
+                                      errp)) {
+        error_append_hint(errp, "NVIDIA Tegra241 CMDQV is unsupported");

are we sure this is the case here and not a radom failure of other source?

Quoted 29 lines
hw/arm/tegra241-cmdqv.c new file +23 -0
+        s->tegra241_cmdqv = false;
+        return false;
+    }
+    return true;
+}
+
+void tegra241_cmdqv_init(SMMUv3State *s)
+{
+    SysBusDevice *sbd = SYS_BUS_DEVICE(OBJECT(s));
+    Tegra241CMDQV *cmdqv;
+
+    if (!s->tegra241_cmdqv) {
+        return;
+    }
+
+    cmdqv = g_new0(Tegra241CMDQV, 1);
+    memory_region_init_io(&cmdqv->mmio_cmdqv, OBJECT(s), &mmio_cmdqv_ops, cmdqv,
+                          "tegra241-cmdqv", TEGRA241_CMDQV_IO_LEN);
+    sysbus_init_mmio(sbd, &cmdqv->mmio_cmdqv);
+    sysbus_init_irq(sbd, &cmdqv->irq);
+    cmdqv->smmu = s;
+    s->cmdqv = cmdqv;
+}
hw/arm/tegra241-cmdqv.h new file +0 -0
new file mode 100644
index 0000000000..9bc72b24d9

hw/arm/tegra241-cmdqv.h line 3

Quoted 3 lines
hw/arm/tegra241-cmdqv.h new file +3 -0
@@ -0,0 +1,40 @@
+/*
+ * Copyright (C) 2025, NVIDIA CORPORATION
+ * NVIDIA Tegra241 CMDQ-Virtualiisation extension for SMMUv3

virtualization

hw/arm/tegra241-cmdqv.h line 15

Quoted 12 lines
hw/arm/tegra241-cmdqv.h new file +12 -0
+ *
+ * Written by Nicolin Chen, Shameer Kolothum
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#ifndef HW_TEGRA241_CMDQV_H
+#define HW_TEGRA241_CMDQV_H
+
+#include CONFIG_DEVICES
+
+#define TEGRA241_CMDQV_IO_LEN 0x50000

can you explain the size

Quoted 39 lines
hw/arm/tegra241-cmdqv.h new file +25 -0
+
+typedef struct Tegra241CMDQV {
+    struct iommu_viommu_tegra241_cmdqv cmdqv_data;
+    SMMUv3State *smmu;
+    MemoryRegion mmio_cmdqv;
+    qemu_irq irq;
+} Tegra241CMDQV;
+
+#ifdef CONFIG_TEGRA241_CMDQV
+bool tegra241_cmdqv_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
+                                 uint32_t *out_viommu_id, Error **errp);
+void tegra241_cmdqv_init(SMMUv3State *s);
+#else
+static inline void tegra241_cmdqv_init(SMMUv3State *s)
+{
+}
+static inline bool
+tegra241_cmdqv_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
+                            uint32_t *out_viommu_id, Error **errp)
+{
+    return true;
+}
+#endif
+
+#endif /* HW_TEGRA241_CMDQV_H */
include/hw/arm/smmuv3.h +3 -0
index 2d4970fe19..8e56e480a0 100644
@@ -73,6 +73,9 @@ struct SMMUv3State {
     bool ats;
     uint8_t oas;
     bool pasid;
+    /* Support for NVIDIA Tegra241 SMMU CMDQV extension */
+    struct Tegra241CMDQV *cmdqv;
+    bool tegra241_cmdqv;
 };
 
 typedef enum {
#21 lore raw

RFC PATCH 6/16 hw/arm/tegra241-cmdqv: Map VINTF Page0 into guest

In reply to Shameer Kolothum

From: Nicolin Chen <[email protected]>

Tegra241 CMDQV assigns each VINTF a 128KB MMIO region split into two 64 KB pages:
- Page0: guest accessible control/status registers for all VCMDQs
- Page1: configuration registers (queue GPA/size) that must be trapped by the VMM and translated before programming the HW queue.

This patch implements the Page0 handling in QEMU. Using the vintf offset returned by IOMMUFD during VIOMMU allocation, QEMU maps Page0 into guest physical address space and exposes it via two guest MMIO windows:
- 0x10000 :VCMDQ register
- 0x30000 :VINTF register

The mapping is lazily initialized on first read/write.

---
 hw/arm/tegra241-cmdqv.c | 60 +++++++++++++++++++++++++++++++++++++++++
 hw/arm/tegra241-cmdqv.h |  5 ++++
 2 files changed, 65 insertions(+)
hw/arm/tegra241-cmdqv.c +60 -0
index 899325877e..d8858322dc 100644
@@ -13,14 +13,74 @@
 #include "smmuv3-accel.h"
 #include "tegra241-cmdqv.h"
 
+static bool tegra241_cmdqv_init_vcmdq_page0(Tegra241CMDQV *cmdqv, Error **errp)
+{
+    SMMUv3State *smmu = cmdqv->smmu;
+    SMMUv3AccelState *s_accel = smmu->s_accel;
+    IOMMUFDViommu *viommu;
+    char *name;
+
+    if (!s_accel) {
+        return true;
+    }
+
+    viommu = &s_accel->viommu;
+    if (!iommufd_backend_viommu_mmap(viommu->iommufd, viommu->viommu_id,
+                                     VCMDQ_REG_PAGE_SIZE,
+                                     cmdqv->cmdqv_data.out_vintf_mmap_offset,
+                                     &cmdqv->vcmdq_page0, errp)) {
+        cmdqv->vcmdq_page0 = NULL;
+        return false;
+    }
+
+    name = g_strdup_printf("%s vcmdq", memory_region_name(&cmdqv->mmio_cmdqv));
+    memory_region_init_ram_device_ptr(&cmdqv->mmio_vcmdq_page,
+                                      memory_region_owner(&cmdqv->mmio_cmdqv),
+                                      name, 0x10000, cmdqv->vcmdq_page0);
+    memory_region_add_subregion_overlap(&cmdqv->mmio_cmdqv, 0x10000,
+                                        &cmdqv->mmio_vcmdq_page, 1);
+    g_free(name);
+
+    name = g_strdup_printf("%s vintf", memory_region_name(&cmdqv->mmio_cmdqv));
+    memory_region_init_ram_device_ptr(&cmdqv->mmio_vintf_page,
+                                      memory_region_owner(&cmdqv->mmio_cmdqv),
+                                      name, 0x10000, cmdqv->vcmdq_page0);
+    memory_region_add_subregion_overlap(&cmdqv->mmio_cmdqv, 0x30000,
+                                        &cmdqv->mmio_vintf_page, 1);
+    g_free(name);
+
+    return true;
+}
+
 static uint64_t tegra241_cmdqv_read(void *opaque, hwaddr offset, unsigned size)
 {
+    Tegra241CMDQV *cmdqv = (Tegra241CMDQV *)opaque;
+    Error *local_err = NULL;
+
+    if (!cmdqv->vcmdq_page0) {
+        tegra241_cmdqv_init_vcmdq_page0(cmdqv, &local_err);
+        if (local_err) {
+            error_report_err(local_err);
+            local_err = NULL;
+        }
+    }
+
     return 0;
 }
 
 static void tegra241_cmdqv_write(void *opaque, hwaddr offset, uint64_t value,
                                  unsigned size)
 {
+    Tegra241CMDQV *cmdqv = (Tegra241CMDQV *)opaque;
+    Error *local_err = NULL;
+
+    if (!cmdqv->vcmdq_page0) {
+        tegra241_cmdqv_init_vcmdq_page0(cmdqv, &local_err);
+        if (local_err) {
+            error_report_err(local_err);
+            local_err = NULL;
+        }
+    }
 }
 
 static const MemoryRegionOps mmio_cmdqv_ops = {
hw/arm/tegra241-cmdqv.h +5 -0
index 9bc72b24d9..ccdf0651be 100644
@@ -19,8 +19,13 @@ typedef struct Tegra241CMDQV {
     SMMUv3State *smmu;
     MemoryRegion mmio_cmdqv;
     qemu_irq irq;
+    MemoryRegion mmio_vcmdq_page;
+    MemoryRegion mmio_vintf_page;
+    void *vcmdq_page0;
 } Tegra241CMDQV;
 
+#define VCMDQ_REG_PAGE_SIZE 0x10000
+
 #ifdef CONFIG_TEGRA241_CMDQV
 bool tegra241_cmdqv_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
                                  uint32_t *out_viommu_id, Error **errp);
Signature · 1 line
2.43.0
#22 lore raw

Re: [RFC PATCH 06/16] hw/arm/tegra241-cmdqv: Map VINTF Page0 into guest

Quoted 8 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

Tegra241 CMDQV assigns each VINTF a 128KB MMIO region split into two 64 KB pages:
- Page0: guest accessible control/status registers for all VCMDQs
- Page1: configuration registers (queue GPA/size) that must be trapped by the VMM and translated before programming the HW queue.

Quoted 4 lines

This patch implements the Page0 handling in QEMU. Using the vintf offset returned by IOMMUFD during VIOMMU allocation, QEMU maps Page0 into guest physical address space and exposes it via two guest MMIO windows:
- 0x10000 :VCMDQ register

global VCMDQ MMIO pages.

Quoted 1 line

- 0x30000 :VINTF register

private VINTF MMIO pages

hw/arm/tegra241-cmdqv.c line 25

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10 -0
+static bool tegra241_cmdqv_init_vcmdq_page0(Tegra241CMDQV *cmdqv, Error **errp)
+{
+    SMMUv3State *smmu = cmdqv->smmu;
+    SMMUv3AccelState *s_accel = smmu->s_accel;
+    IOMMUFDViommu *viommu;
+    char *name;
+
+    if (!s_accel) {
+        return true;
+    }

g_assert?

The entire thing can't work without s_accel, so returning true doesn't seem to make sense.

hw/arm/tegra241-cmdqv.c line 32

Quoted 6 lines
hw/arm/tegra241-cmdqv.c +6 -0
+    viommu = &s_accel->viommu;
+    if (!iommufd_backend_viommu_mmap(viommu->iommufd, viommu->viommu_id,
+                                     VCMDQ_REG_PAGE_SIZE,
+                                     cmdqv->cmdqv_data.out_vintf_mmap_offset,
+                                     &cmdqv->vcmdq_page0, errp)) {
+        cmdqv->vcmdq_page0 = NULL;

We probably shouldn't nuke the vcmdq_page0.

And I think we should add g_assert(!cmdqv->vcmdq_page0) too. It would be a bug if we pass in a valid page0 pointer.

Quoted 4 lines
hw/arm/tegra241-cmdqv.c +4 -0
+    name = g_strdup_printf("%s vcmdq", memory_region_name(&cmdqv->mmio_cmdqv));
+    memory_region_init_ram_device_ptr(&cmdqv->mmio_vcmdq_page,
+                                      memory_region_owner(&cmdqv->mmio_cmdqv),
+                                      name, 0x10000, cmdqv->vcmdq_page0);

hw/arm/tegra241-cmdqv.c line 49

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10 -0
+    memory_region_add_subregion_overlap(&cmdqv->mmio_cmdqv, 0x10000,
+                                        &cmdqv->mmio_vcmdq_page, 1);
+    g_free(name);
+
+    name = g_strdup_printf("%s vintf", memory_region_name(&cmdqv->mmio_cmdqv));
+    memory_region_init_ram_device_ptr(&cmdqv->mmio_vintf_page,
+                                      memory_region_owner(&cmdqv->mmio_cmdqv),
+                                      name, 0x10000, cmdqv->vcmdq_page0);
+    memory_region_add_subregion_overlap(&cmdqv->mmio_cmdqv, 0x30000,
+                                        &cmdqv->mmio_vintf_page, 1);

Let's add some comments here (maybe something similar in commit log also):

/*
* Each VM can only own one VINTF exposed by the kernel via a VIOMMU object.
* And all available VCMDQs are already preallocated in the VINTF. Thus, the
* global VCMDQ MMIO page0 and the private VINTF MMIO page0 are effectively
* the same, i.e. cmdqv->vcmdq_page0.
*/

Nicolin

#23 lore raw

Re: [RFC PATCH 06/16] hw/arm/tegra241-cmdqv: Map VINTF Page0 into guest

In reply to Shameer Kolothum

Hi Shameer,

Quoted 53 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

Tegra241 CMDQV assigns each VINTF a 128KB MMIO region split into two 64 KB pages:
- Page0: guest accessible control/status registers for all VCMDQs
- Page1: configuration registers (queue GPA/size) that must be trapped by the VMM and translated before programming the HW queue.

This patch implements the Page0 handling in QEMU. Using the vintf offset returned by IOMMUFD during VIOMMU allocation, QEMU maps Page0 into guest physical address space and exposes it via two guest MMIO windows:
- 0x10000 :VCMDQ register
- 0x30000 :VINTF register

The mapping is lazily initialized on first read/write.

---
 hw/arm/tegra241-cmdqv.c | 60 +++++++++++++++++++++++++++++++++++++++++
 hw/arm/tegra241-cmdqv.h |  5 ++++
 2 files changed, 65 insertions(+)
hw/arm/tegra241-cmdqv.c +22 -0
index 899325877e..d8858322dc 100644
@@ -13,14 +13,74 @@
 #include "smmuv3-accel.h"
 #include "tegra241-cmdqv.h"
 
+static bool tegra241_cmdqv_init_vcmdq_page0(Tegra241CMDQV *cmdqv, Error **errp)
+{
+    SMMUv3State *smmu = cmdqv->smmu;
+    SMMUv3AccelState *s_accel = smmu->s_accel;
+    IOMMUFDViommu *viommu;
+    char *name;
+
+    if (!s_accel) {
+        return true;
+    }
+
+    viommu = &s_accel->viommu;
+    if (!iommufd_backend_viommu_mmap(viommu->iommufd, viommu->viommu_id,
+                                     VCMDQ_REG_PAGE_SIZE,
+                                     cmdqv->cmdqv_data.out_vintf_mmap_offset,
+                                     &cmdqv->vcmdq_page0, errp)) {
+        cmdqv->vcmdq_page0 = NULL;
+        return false;
+    }
+
+    name = g_strdup_printf("%s vcmdq", memory_region_name(&cmdqv->mmio_cmdqv));
+    memory_region_init_ram_device_ptr(&cmdqv->mmio_vcmdq_page,

hw/arm/tegra241-cmdqv.c line 47

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10 -0
+                                      memory_region_owner(&cmdqv->mmio_cmdqv),
+                                      name, 0x10000, cmdqv->vcmdq_page0);
+    memory_region_add_subregion_overlap(&cmdqv->mmio_cmdqv, 0x10000,
+                                        &cmdqv->mmio_vcmdq_page, 1);
+    g_free(name);
+
+    name = g_strdup_printf("%s vintf", memory_region_name(&cmdqv->mmio_cmdqv));
+    memory_region_init_ram_device_ptr(&cmdqv->mmio_vintf_page,
+                                      memory_region_owner(&cmdqv->mmio_cmdqv),
+                                      name, 0x10000, cmdqv->vcmdq_page0);

I don't get why we need/have 2 RAM devices pointing to the same @ptr = cmdqv->vcmdq_page0. Is 0x10000 ~ VCMDQ_REG_PAGE_SIZE?

The names of the MRs are quite confusing: If my understanding is correct we have;

cmdqv->mmio_cmdqv (0x50000 sized) which as the container for the 2 subregions. Then within that one we have 2 subregions, one at offset 0x10000 (mmio_vcmdq_page ), one at offset 0x30000 (cmdqv->mmio_vintf_page).

I have difficulties to link that with the commit message

"Tegra241 CMDQV assigns each VINTF a 128KB MMIO region split into two 64 KB pages:
- Page0: guest accessible control/status registers for all VCMDQs
- Page1:configuration registers ../.."
Those 2 pages, are they part of mmio_vcmdq_page?

Then you talk about 0x30000 :VINTF register I guess this is the second cmdqv->mmio_vintf_page

Well I am confused at this stage of the reading.

Also without any spec, this is difficult to understand. Is there any public doc?

Quoted 44 lines
hw/arm/tegra241-cmdqv.c +28 -0
+    memory_region_add_subregion_overlap(&cmdqv->mmio_cmdqv, 0x30000,
+                                        &cmdqv->mmio_vintf_page, 1);
+    g_free(name);
+
+    return true;
+}
+
 static uint64_t tegra241_cmdqv_read(void *opaque, hwaddr offset, unsigned size)
 {
+    Tegra241CMDQV *cmdqv = (Tegra241CMDQV *)opaque;
+    Error *local_err = NULL;
+
+    if (!cmdqv->vcmdq_page0) {
+        tegra241_cmdqv_init_vcmdq_page0(cmdqv, &local_err);
+        if (local_err) {
+            error_report_err(local_err);
+            local_err = NULL;
+        }
+    }
+
     return 0;
 }
 
 static void tegra241_cmdqv_write(void *opaque, hwaddr offset, uint64_t value,
                                  unsigned size)
 {
+    Tegra241CMDQV *cmdqv = (Tegra241CMDQV *)opaque;
+    Error *local_err = NULL;
+
+    if (!cmdqv->vcmdq_page0) {
+        tegra241_cmdqv_init_vcmdq_page0(cmdqv, &local_err);
+        if (local_err) {
+            error_report_err(local_err);
+            local_err = NULL;
+        }
+    }
 }
 
 static const MemoryRegionOps mmio_cmdqv_ops = {
hw/arm/tegra241-cmdqv.h +0 -0
index 9bc72b24d9..ccdf0651be 100644

hw/arm/tegra241-cmdqv.h typedef struct Tegra241CMDQV { · line 22

Quoted 4 lines
hw/arm/tegra241-cmdqv.h +1 -0
@@ -19,8 +19,13 @@ typedef struct Tegra241CMDQV {
     SMMUv3State *smmu;
     MemoryRegion mmio_cmdqv;
     qemu_irq irq;
+    MemoryRegion mmio_vcmdq_page;

please also use _mr suffix as in the rest of the code

hw/arm/tegra241-cmdqv.h typedef struct Tegra241CMDQV { · line 31

Quoted 9 lines
hw/arm/tegra241-cmdqv.h +4 -0
+    MemoryRegion mmio_vintf_page;
+    void *vcmdq_page0;
 } Tegra241CMDQV;
 
+#define VCMDQ_REG_PAGE_SIZE 0x10000
+
 #ifdef CONFIG_TEGRA241_CMDQV
 bool tegra241_cmdqv_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
                                  uint32_t *out_viommu_id, Error **errp);

Thanks

Eric

#24 lore raw

RE: [RFC PATCH 06/16] hw/arm/tegra241-cmdqv: Map VINTF Page0 into guest

Hi Eric,

Quoted 88 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 26 January 2026 13:49
To: Shameer Kolothum <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; [email protected]; [email protected]; Krishnakant Jaju <[email protected]>
Subject: Re: [RFC PATCH 06/16] hw/arm/tegra241-cmdqv: Map VINTF Page0 into guest

External email: Use caution opening links or attachments

Hi Shameer,

Quoted 14 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

Tegra241 CMDQV assigns each VINTF a 128KB MMIO region split into two 64 KB pages:
- Page0: guest accessible control/status registers for all VCMDQs
- Page1: configuration registers (queue GPA/size) that must be trapped by the VMM and translated before programming the HW queue.

This patch implements the Page0 handling in QEMU. Using the vintf offset returned by IOMMUFD during VIOMMU allocation, QEMU maps Page0 into guest physical address space and exposes it via two guest MMIO windows:
- 0x10000 :VCMDQ register
- 0x30000 :VINTF register

Quoted 6 lines

The mapping is lazily initialized on first read/write.

---
 hw/arm/tegra241-cmdqv.c | 60

+++++++++++++++++++++++++++++++++++++++++

hw/arm/tegra241-cmdqv.c line 16

Quoted 12 lines
 hw/arm/tegra241-cmdqv.h |  5 ++++
 2 files changed, 65 insertions(+)
hw/arm/tegra241-cmdqv.c +1 -0
index 899325877e..d8858322dc 100644
@@ -13,14 +13,74 @@
 #include "smmuv3-accel.h"
 #include "tegra241-cmdqv.h"
 
+static bool tegra241_cmdqv_init_vcmdq_page0(Tegra241CMDQV *cmdqv,

Error **errp)

Quoted 16 lines
hw/arm/tegra241-cmdqv.c +15 -0
+{
+    SMMUv3State *smmu = cmdqv->smmu;
+    SMMUv3AccelState *s_accel = smmu->s_accel;
+    IOMMUFDViommu *viommu;
+    char *name;
+
+    if (!s_accel) {
+        return true;
+    }
+
+    viommu = &s_accel->viommu;
+    if (!iommufd_backend_viommu_mmap(viommu->iommufd, viommu-
 viommu_id,
+                                     VCMDQ_REG_PAGE_SIZE,
+                                     cmdqv->cmdqv_data.out_vintf_mmap_offset,
+                                     &cmdqv->vcmdq_page0, errp)) {

hw/arm/tegra241-cmdqv.c

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +9 -0
+        cmdqv->vcmdq_page0 = NULL;
+        return false;
+    }
+
+    name = g_strdup_printf("%s vcmdq", memory_region_name(&cmdqv-
 mmio_cmdqv));
+    memory_region_init_ram_device_ptr(&cmdqv->mmio_vcmdq_page,
+                                      memory_region_owner(&cmdqv->mmio_cmdqv),
+                                      name, 0x10000, cmdqv->vcmdq_page0);
+    memory_region_add_subregion_overlap(&cmdqv->mmio_cmdqv,

0x10000,

hw/arm/tegra241-cmdqv.c line 47

Quoted 8 lines
hw/arm/tegra241-cmdqv.c +7 -0
+                                        &cmdqv->mmio_vcmdq_page, 1);
+    g_free(name);
+
+    name = g_strdup_printf("%s vintf", memory_region_name(&cmdqv-
 mmio_cmdqv));
+    memory_region_init_ram_device_ptr(&cmdqv->mmio_vintf_page,
+                                      memory_region_owner(&cmdqv->mmio_cmdqv),
+                                      name, 0x10000, cmdqv->vcmdq_page0);
Quoted 2 lines

I don't get why we need/have 2 RAM devices pointing to the same @ptr = cmdqv->vcmdq_page0. Is 0x10000 ~ VCMDQ_REG_PAGE_SIZE?

Looking at this again, I am now a bit unsure as well why we need these memory_region_init_ram_device_ptr() calls here.

I will go back and discuss with Nicolin and come back. I think we need to document the memory layout better here. That will avoid the confusion.

(Also, will check if we have a public link for the spec or not).

Thanks,
Shameer

#25 lore raw

Re: [RFC PATCH 06/16] hw/arm/tegra241-cmdqv: Map VINTF Page0 into guest

In reply to Eric Auger
Quoted 13 lines · Eric Auger

hw/arm/tegra241-cmdqv.c line 47

Quoted 12 lines
hw/arm/tegra241-cmdqv.c +12 -0
+    name = g_strdup_printf("%s vcmdq", memory_region_name(&cmdqv->mmio_cmdqv));
+    memory_region_init_ram_device_ptr(&cmdqv->mmio_vcmdq_page,
+                                      memory_region_owner(&cmdqv->mmio_cmdqv),
+                                      name, 0x10000, cmdqv->vcmdq_page0);
+    memory_region_add_subregion_overlap(&cmdqv->mmio_cmdqv, 0x10000,
+                                        &cmdqv->mmio_vcmdq_page, 1);
+    g_free(name);
+
+    name = g_strdup_printf("%s vintf", memory_region_name(&cmdqv->mmio_cmdqv));
+    memory_region_init_ram_device_ptr(&cmdqv->mmio_vintf_page,
+                                      memory_region_owner(&cmdqv->mmio_cmdqv),
+                                      name, 0x10000, cmdqv->vcmdq_page0);
Quoted 2 lines

I don't get why we need/have 2 RAM devices pointing to the same @ptr = cmdqv->vcmdq_page0. Is 0x10000 ~ VCMDQ_REG_PAGE_SIZE?

The first one is for "vcmdq" and the second one is for "vintf". Explaining below...

Quoted 6 lines

The names of the MRs are quite confusing: If my understanding is correct we have;

cmdqv->mmio_cmdqv (0x50000 sized) which as the container for the 2 subregions. Then within that one we have 2 subregions, one at offset 0x10000 (mmio_vcmdq_page ), one at offset 0x30000 (cmdqv->mmio_vintf_page).

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/iommu/arm/arm-smmu-v3/tegra241-cmdqv.c?h=v6.19-rc7#n19
It's defined in the kernel driver (yea, we should clarify in the QEMU code as well).

So, the MMIO regions look like this:
1st 64 KB page -- Global CMDQV registers
2nd 64 KB page -- Global VCMDQ registers Page0 (mmap) 3rd 64 KB page -- Global VCMDQ registers Page1 (trap) 4th 64 KB page -- VINTF0 Logical VCMDQ registers Page0 (mmap) 5th 64 KB page -- VINTF0 Logical VCMDQ registers Page1 (trap)

In real hardware, there will be 6th 64KB and beyond, for VINTF1 and others. But, we're omitting here in QEMU as only VINTF0 will be supported -- kernel only exposes one VINTF per VM as well. (Yes, we should clarify this too.)

Quoted 7 lines

I have difficulties to link that with the commit message

"Tegra241 CMDQV assigns each VINTF a 128KB MMIO region split into two 64 KB pages:
- Page0: guest accessible control/status registers for all VCMDQs
- Page1:configuration registers ../.."
Those 2 pages, are they part of mmio_vcmdq_page?

Not exactly. Both the global VCMDQ region and VINTF region have their own pages (at 0x10000 and 0x30000).

Here, it duplicates the mapping to 0x10000 (Global VCMDQ page0) and 0x30000 (VINTF0 page0) for simplification, because we only support VINTF0 in this case.

There is a little catch in this implementation. The real physical mapping between a global VCMDQ and a logical VCMDQ happens when QEMU calls HW_QUEUE ioctl. So, the mmap'd page0 doesn't have any real VCMDQ backing up the emulated VCMDQ. So, perhaps QEMU should trap the page0 and delay the memory_region_init_ram_device_ptr() until the HW_QUEUE ioctl is done?

There might be also a corner case: when the kernel exposes two physical VCMDQs, but the guest OS only uses one, i.e. QEMU only allocates one HW_QUEUE for VCMDQ0 but doesn't allocate VCMDQ1. In such a case, the VTINF0 page0 should be able to control the logical VCMDQ0 only, while the global page0 should control both.

We are getting away this corner case with any guest OS running Linux kernel because it only accesses VTINF pages. But likely we should do something about it..

Quoted 5 lines

Then you talk about 0x30000 :VINTF register I guess this is the second cmdqv->mmio_vintf_page

Well I am confused at this stage of the reading.

Also without any spec, this is difficult to understand. Is there any public doc?

It seems that Red Hat can get the doc under NDA..

Thanks
Nicolin

#26 lore raw

Re: [RFC PATCH 06/16] hw/arm/tegra241-cmdqv: Map VINTF Page0 into guest

Hi Nicolin,

Quoted 29 lines · Nicolin Chen
Quoted 12 lines · Eric Auger

hw/arm/tegra241-cmdqv.c line 47

Quoted 12 lines
hw/arm/tegra241-cmdqv.c +12 -0
+    name = g_strdup_printf("%s vcmdq", memory_region_name(&cmdqv->mmio_cmdqv));
+    memory_region_init_ram_device_ptr(&cmdqv->mmio_vcmdq_page,
+                                      memory_region_owner(&cmdqv->mmio_cmdqv),
+                                      name, 0x10000, cmdqv->vcmdq_page0);
+    memory_region_add_subregion_overlap(&cmdqv->mmio_cmdqv, 0x10000,
+                                        &cmdqv->mmio_vcmdq_page, 1);
+    g_free(name);
+
+    name = g_strdup_printf("%s vintf", memory_region_name(&cmdqv->mmio_cmdqv));
+    memory_region_init_ram_device_ptr(&cmdqv->mmio_vintf_page,
+                                      memory_region_owner(&cmdqv->mmio_cmdqv),
+                                      name, 0x10000, cmdqv->vcmdq_page0);
Quoted 2 lines

I don't get why we need/have 2 RAM devices pointing to the same @ptr = cmdqv->vcmdq_page0. Is 0x10000 ~ VCMDQ_REG_PAGE_SIZE?

The first one is for "vcmdq" and the second one is for "vintf". Explaining below...

Quoted 6 lines

The names of the MRs are quite confusing: If my understanding is correct we have;

cmdqv->mmio_cmdqv (0x50000 sized) which as the container for the 2 subregions. Then within that one we have 2 subregions, one at offset 0x10000 (mmio_vcmdq_page ), one at offset 0x30000 (cmdqv->mmio_vintf_page).

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/iommu/arm/arm-smmu-v3/tegra241-cmdqv.c?h=v6.19-rc7#n19
It's defined in the kernel driver (yea, we should clarify in the QEMU code as well).

Quoted 6 lines

So, the MMIO regions look like this:
1st 64 KB page -- Global CMDQV registers
2nd 64 KB page -- Global VCMDQ registers Page0 (mmap) 3rd 64 KB page -- Global VCMDQ registers Page1 (trap) 4th 64 KB page -- VINTF0 Logical VCMDQ registers Page0 (mmap) 5th 64 KB page -- VINTF0 Logical VCMDQ registers Page1 (trap)

This kind of layout representation is really helpful

Quoted 5 lines

In real hardware, there will be 6th 64KB and beyond, for VINTF1 and others. But, we're omitting here in QEMU as only VINTF0 will be supported -- kernel only exposes one VINTF per VM as well. (Yes, we should clarify this too.)

OK

Quoted 8 lines
Quoted 8 lines

I have difficulties to link that with the commit message

"Tegra241 CMDQV assigns each VINTF a 128KB MMIO region split into two 64 KB pages:
- Page0: guest accessible control/status registers for all VCMDQs
- Page1:configuration registers ../.."
Those 2 pages, are they part of mmio_vcmdq_page?

Quoted 6 lines

Not exactly. Both the global VCMDQ region and VINTF region have their own pages (at 0x10000 and 0x30000).

Here, it duplicates the mapping to 0x10000 (Global VCMDQ page0) and 0x30000 (VINTF0 page0) for simplification, because we only support VINTF0 in this case.

so Global VCMDQ registers Page0 and VINTF0 Logical VCMDQ registers Page0 are basically the same?

I would recommend to use cmdqv->mmio_vcmdq_page0 and cmdqv->mmio_vintf_page0 to avoid any misunderstanding

Quoted 7 lines

There is a little catch in this implementation. The real physical mapping between a global VCMDQ and a logical VCMDQ happens when QEMU calls HW_QUEUE ioctl. So, the mmap'd page0 doesn't have any real VCMDQ backing up the emulated VCMDQ. So, perhaps QEMU should trap the page0 and delay the memory_region_init_ram_device_ptr() until the HW_QUEUE ioctl is done?

might be safer indeed.

Quoted 6 lines

There might be also a corner case: when the kernel exposes two physical VCMDQs, but the guest OS only uses one, i.e. QEMU only allocates one HW_QUEUE for VCMDQ0 but doesn't allocate VCMDQ1. In such a case, the VTINF0 page0 should be able to control the logical VCMDQ0 only, while the global page0 should control both.

you lost me. Need to look at the kernel or spec ;-)

Quoted 11 lines

We are getting away this corner case with any guest OS running Linux kernel because it only accesses VTINF pages. But likely we should do something about it..

Quoted 5 lines

Then you talk about 0x30000 :VINTF register I guess this is the second cmdqv->mmio_vintf_page

Well I am confused at this stage of the reading.

Also without any spec, this is difficult to understand. Is there any public doc?

It seems that Red Hat can get the doc under NDA..

OK thanks

Eric

Quoted 4 lines

Thanks
Nicolin

#27 lore raw

Re: [RFC PATCH 06/16] hw/arm/tegra241-cmdqv: Map VINTF Page0 into guest

Quoted 8 lines · Eric Auger
Quoted 7 lines · Nicolin Chen

On Mon, Jan 26, 2026 at 02:48:55PM +0100, Eric Auger wrote:
So, the MMIO regions look like this:
1st 64 KB page -- Global CMDQV registers
2nd 64 KB page -- Global VCMDQ registers Page0 (mmap) 3rd 64 KB page -- Global VCMDQ registers Page1 (trap) 4th 64 KB page -- VINTF0 Logical VCMDQ registers Page0 (mmap) 5th 64 KB page -- VINTF0 Logical VCMDQ registers Page1 (trap)

[...]

Quoted 10 lines
Quoted 10 lines
Quoted 7 lines

I have difficulties to link that with the commit message

"Tegra241 CMDQV assigns each VINTF a 128KB MMIO region split into two 64 KB pages:
- Page0: guest accessible control/status registers for all VCMDQs
- Page1:configuration registers ../.."
Those 2 pages, are they part of mmio_vcmdq_page?

Not exactly. Both the global VCMDQ region and VINTF region have their own pages (at 0x10000 and 0x30000).

Quoted 6 lines
Quoted 3 lines

Here, it duplicates the mapping to 0x10000 (Global VCMDQ page0) and 0x30000 (VINTF0 page0) for simplification, because we only support VINTF0 in this case.

so Global VCMDQ registers Page0 and VINTF0 Logical VCMDQ registers Page0 are basically the same?

Not exactly the same.

The global page0 is programmable at any time so long as CMDQV_EN is enabled.

The logical page0 is programmable only when SW allocates and maps global vcmdq(s) to a VINTF. "logical" also means "local" to that VINTF.

Quoted 2 lines

I would recommend to use cmdqv->mmio_vcmdq_page0 and cmdqv->mmio_vintf_page0 to avoid any misunderstanding

Yea, that makes sense to me.

Quoted 8 lines
Quoted 6 lines

There is a little catch in this implementation. The real physical mapping between a global VCMDQ and a logical VCMDQ happens when QEMU calls HW_QUEUE ioctl. So, the mmap'd page0 doesn't have any real VCMDQ backing up the emulated VCMDQ. So, perhaps QEMU should trap the page0 and delay the memory_region_init_ram_device_ptr() until the HW_QUEUE ioctl is done?

might be safer indeed.

Quoted 1 line
Quoted 7 lines
Quoted 5 lines

There might be also a corner case: when the kernel exposes two physical VCMDQs, but the guest OS only uses one, i.e. QEMU only allocates one HW_QUEUE for VCMDQ0 but doesn't allocate VCMDQ1. In such a case, the VTINF0 page0 should be able to control the logical VCMDQ0 only, while the global page0 should control both.

you lost me. Need to look at the kernel or spec ;-)

That was for supporting a partial local vcmdq mapping with QEMU, as guest OS is allowed to do so from HW simulation perspective.

E.g.

A VTINF exposed by the kernel supports maximum 2 VCMDQs. IOW, VM owns 2 global vcmdqs. And mmio_vcmdq_page0 is supposed to have the access to both vcmdqs.

Kernel only exposes physical VINTF's page0 via mmap, for security reason, which starts with 0 logical vcmdq, i.e. no access to any global vcmdq via mmio_vcmdq_page0 or mmio_vintf_page0.

A guest SW only allocates and maps one global vcmdq to the VINTF, which means only one HW_QUEUE ioctl is invoked so that the kernel only maps one global vcmdq accordingly. Then, both page0 only has the access to one global vcmdq via its logical mapping.

In such a case, memory_region_init_ram_device_ptr() to 0x10000 for mmio_vcmdq_page0 is basically wrong, since it's supposed to have access to both global vcmdqs.

So, this makes things uneasy to support.

What we could likely do:
- only mmap mmio_vintf_page0 to 0x30000 (logical page0)
- drop mmio_vcmdq_page0 and trap 0x10000 (global page0)
a) if vcmdq is indexed to a mapped global vcmdq, read/write mmio_vintf_page0.
b) if vcmdq is indexed to an unmapped global vcmdq, read/write the value in the register array cached by QEMU.

This would likely slow down the perf if guest OS is using a LVCMDQ via the global page0. But functional wise it should be okay.

Nicolin

#28 lore raw

Re: [RFC PATCH 06/16] hw/arm/tegra241-cmdqv: Map VINTF Page0 into guest

In reply to Shameer Kolothum
Quoted 8 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

Tegra241 CMDQV assigns each VINTF a 128KB MMIO region split into two 64 KB pages:
- Page0: guest accessible control/status registers for all VCMDQs
- Page1: configuration registers (queue GPA/size) that must be trapped by the VMM and translated before programming the HW queue.

Quoted 5 lines

This patch implements the Page0 handling in QEMU. Using the vintf offset returned by IOMMUFD during VIOMMU allocation, QEMU maps Page0 into guest physical address space and exposes it via two guest MMIO windows:
- 0x10000 :VCMDQ register
- 0x30000 :VINTF register

I would recommend to have a drawing showing the mr container, the subregions, their offsets, and content/nature of each region

Eric

Quoted 107 lines

The mapping is lazily initialized on first read/write.

---
 hw/arm/tegra241-cmdqv.c | 60 +++++++++++++++++++++++++++++++++++++++++
 hw/arm/tegra241-cmdqv.h |  5 ++++
 2 files changed, 65 insertions(+)
hw/arm/tegra241-cmdqv.c +60 -0
index 899325877e..d8858322dc 100644
@@ -13,14 +13,74 @@
 #include "smmuv3-accel.h"
 #include "tegra241-cmdqv.h"
 
+static bool tegra241_cmdqv_init_vcmdq_page0(Tegra241CMDQV *cmdqv, Error **errp)
+{
+    SMMUv3State *smmu = cmdqv->smmu;
+    SMMUv3AccelState *s_accel = smmu->s_accel;
+    IOMMUFDViommu *viommu;
+    char *name;
+
+    if (!s_accel) {
+        return true;
+    }
+
+    viommu = &s_accel->viommu;
+    if (!iommufd_backend_viommu_mmap(viommu->iommufd, viommu->viommu_id,
+                                     VCMDQ_REG_PAGE_SIZE,
+                                     cmdqv->cmdqv_data.out_vintf_mmap_offset,
+                                     &cmdqv->vcmdq_page0, errp)) {
+        cmdqv->vcmdq_page0 = NULL;
+        return false;
+    }
+
+    name = g_strdup_printf("%s vcmdq", memory_region_name(&cmdqv->mmio_cmdqv));
+    memory_region_init_ram_device_ptr(&cmdqv->mmio_vcmdq_page,
+                                      memory_region_owner(&cmdqv->mmio_cmdqv),
+                                      name, 0x10000, cmdqv->vcmdq_page0);
+    memory_region_add_subregion_overlap(&cmdqv->mmio_cmdqv, 0x10000,
+                                        &cmdqv->mmio_vcmdq_page, 1);
+    g_free(name);
+
+    name = g_strdup_printf("%s vintf", memory_region_name(&cmdqv->mmio_cmdqv));
+    memory_region_init_ram_device_ptr(&cmdqv->mmio_vintf_page,
+                                      memory_region_owner(&cmdqv->mmio_cmdqv),
+                                      name, 0x10000, cmdqv->vcmdq_page0);
+    memory_region_add_subregion_overlap(&cmdqv->mmio_cmdqv, 0x30000,
+                                        &cmdqv->mmio_vintf_page, 1);
+    g_free(name);
+
+    return true;
+}
+
 static uint64_t tegra241_cmdqv_read(void *opaque, hwaddr offset, unsigned size)
 {
+    Tegra241CMDQV *cmdqv = (Tegra241CMDQV *)opaque;
+    Error *local_err = NULL;
+
+    if (!cmdqv->vcmdq_page0) {
+        tegra241_cmdqv_init_vcmdq_page0(cmdqv, &local_err);
+        if (local_err) {
+            error_report_err(local_err);
+            local_err = NULL;
+        }
+    }
+
     return 0;
 }
 
 static void tegra241_cmdqv_write(void *opaque, hwaddr offset, uint64_t value,
                                  unsigned size)
 {
+    Tegra241CMDQV *cmdqv = (Tegra241CMDQV *)opaque;
+    Error *local_err = NULL;
+
+    if (!cmdqv->vcmdq_page0) {
+        tegra241_cmdqv_init_vcmdq_page0(cmdqv, &local_err);
+        if (local_err) {
+            error_report_err(local_err);
+            local_err = NULL;
+        }
+    }
 }
 
 static const MemoryRegionOps mmio_cmdqv_ops = {
hw/arm/tegra241-cmdqv.h +5 -0
index 9bc72b24d9..ccdf0651be 100644
@@ -19,8 +19,13 @@ typedef struct Tegra241CMDQV {
     SMMUv3State *smmu;
     MemoryRegion mmio_cmdqv;
     qemu_irq irq;
+    MemoryRegion mmio_vcmdq_page;
+    MemoryRegion mmio_vintf_page;
+    void *vcmdq_page0;
 } Tegra241CMDQV;
 
+#define VCMDQ_REG_PAGE_SIZE 0x10000
+
 #ifdef CONFIG_TEGRA241_CMDQV
 bool tegra241_cmdqv_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
                                  uint32_t *out_viommu_id, Error **errp);
#29 lore raw

RFC PATCH 7/16 hw/arm/tegra241-cmdqv: Add read emulation support for registers

In reply to Shameer Kolothum

From: Nicolin Chen <[email protected]>

Implement read support for Tegra241 CMDQV register blocks, including VINTF and per VCMDQ register regions. The patch decodes offsets, extracts queue indices, and returns the corresponding cached register state.

Subsequent patch will add write support.

---
 hw/arm/tegra241-cmdqv.c | 144 +++++++++++++++++++-
 hw/arm/tegra241-cmdqv.h | 282 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 425 insertions(+), 1 deletion(-)
hw/arm/tegra241-cmdqv.c +143 -1
index d8858322dc..185ef957bc 100644
@@ -8,6 +8,7 @@
  */
 
 #include "qemu/osdep.h"
+#include "qemu/log.h"
 
 #include "hw/arm/smmuv3.h"
 #include "smmuv3-accel.h"
@@ -52,10 +53,94 @@ static bool tegra241_cmdqv_init_vcmdq_page0(Tegra241CMDQV *cmdqv, Error **errp)
     return true;
 }
 
+/* Note that offset aligns down to 0x1000 */
+static uint64_t tegra241_cmdqv_read_vintf(Tegra241CMDQV *cmdqv, hwaddr offset)
+{
+    int i;
+
+    switch (offset) {
+    case A_VINTF0_CONFIG:
+        return cmdqv->vintf_config;
+    case A_VINTF0_STATUS:
+        return cmdqv->vintf_status;
+    case A_VINTF0_LVCMDQ_ERR_MAP_0 ... A_VINTF0_LVCMDQ_ERR_MAP_3:
+        i = (offset - A_VINTF0_LVCMDQ_ERR_MAP_0) / 4;
+        return cmdqv->vintf_cmdq_err_map[i];
+    default:
+        qemu_log_mask(LOG_UNIMP, "%s unhandled read access at 0x%" PRIx64 "\n",
+                      __func__, offset);
+        return 0;
+    }
+}
+
+/* Note that offset aligns down to 0x10000 */
+static uint64_t tegra241_cmdqv_read_vcmdq(Tegra241CMDQV *cmdqv, hwaddr offset,
+                                          int index)
+{
+    uint32_t *ptr = NULL;
+    uint64_t off;
+
+    /*
+     * Each VCMDQ instance occupies a 128 byte region (0x80).
+     * The hardware layout is:
+     *   vcmdq_page0 + (index * 0x80) + (offset - 0x10000)
+     */
+    if (cmdqv->vcmdq_page0) {
+        off = (0x80 * index) + (offset - 0x10000);
+        ptr = (uint32_t *)(cmdqv->vcmdq_page0 + off);
+    }
+
+    switch (offset) {
+    case A_VCMDQ0_CONS_INDX:
+        if (ptr) {
+            cmdqv->vcmdq_cons_indx[index] = *ptr;
+        }
+        return cmdqv->vcmdq_cons_indx[index];
+    case A_VCMDQ0_PROD_INDX:
+        if (ptr) {
+            cmdqv->vcmdq_prod_indx[index] = *ptr;
+        }
+        return cmdqv->vcmdq_prod_indx[index];
+    case A_VCMDQ0_CONFIG:
+        if (ptr) {
+            cmdqv->vcmdq_config[index] = *ptr;
+        }
+        return cmdqv->vcmdq_config[index];
+    case A_VCMDQ0_STATUS:
+        if (ptr) {
+            cmdqv->vcmdq_status[index] = *ptr;
+        }
+        return cmdqv->vcmdq_status[index];
+    case A_VCMDQ0_GERROR:
+        if (ptr) {
+            cmdqv->vcmdq_gerror[index] = *ptr;
+        }
+        return cmdqv->vcmdq_gerror[index];
+    case A_VCMDQ0_GERRORN:
+        if (ptr) {
+            cmdqv->vcmdq_gerrorn[index] = *ptr;
+        }
+        return cmdqv->vcmdq_gerrorn[index];
+    case A_VCMDQ0_BASE_L:
+        return cmdqv->vcmdq_base[index];
+    case A_VCMDQ0_BASE_H:
+        return cmdqv->vcmdq_base[index] >> 32;
+    case A_VCMDQ0_CONS_INDX_BASE_DRAM_L:
+        return cmdqv->vcmdq_cons_indx_base[index];
+    case A_VCMDQ0_CONS_INDX_BASE_DRAM_H:
+        return cmdqv->vcmdq_cons_indx_base[index] >> 32;
+    default:
+        qemu_log_mask(LOG_UNIMP,
+                      "%s unhandled read access at 0x%" PRIx64 "\n",
+                      __func__, offset);
+        return 0;
+    }
+}
 static uint64_t tegra241_cmdqv_read(void *opaque, hwaddr offset, unsigned size)
 {
     Tegra241CMDQV *cmdqv = (Tegra241CMDQV *)opaque;
     Error *local_err = NULL;
+    int index;
 
     if (!cmdqv->vcmdq_page0) {
         tegra241_cmdqv_init_vcmdq_page0(cmdqv, &local_err);
@@ -65,7 +150,64 @@ static uint64_t tegra241_cmdqv_read(void *opaque, hwaddr offset, unsigned size)
         }
     }
 
-    return 0;
+    if (offset > TEGRA241_CMDQV_IO_LEN) {
+        qemu_log_mask(LOG_UNIMP,
+                      "%s offset 0x%" PRIx64 " off limit (0x50000)\n", __func__,
+                      offset);
+        return 0;
+    }
+
+    /* Fallback to cached register values */
+    switch (offset) {
+    case A_CONFIG:
+        return cmdqv->config;
+    case A_PARAM:
+        return cmdqv->param;
+    case A_STATUS:
+        return cmdqv->status;
+    case A_VI_ERR_MAP ... A_VI_ERR_MAP_1:
+        return cmdqv->vi_err_map[(offset - A_VI_ERR_MAP) / 4];
+    case A_VI_INT_MASK ... A_VI_INT_MASK_1:
+        return cmdqv->vi_int_mask[(offset - A_VI_INT_MASK) / 4];
+    case A_CMDQ_ERR_MAP ... A_CMDQ_ERR_MAP_3:
+        return cmdqv->cmdq_err_map[(offset - A_CMDQ_ERR_MAP) / 4];
+    case A_CMDQ_ALLOC_MAP_0 ... A_CMDQ_ALLOC_MAP_127:
+        return cmdqv->cmdq_alloc_map[(offset - A_CMDQ_ALLOC_MAP_0) / 4];
+    case A_VINTF0_CONFIG ... A_VINTF0_LVCMDQ_ERR_MAP_3:
+        return tegra241_cmdqv_read_vintf(cmdqv, offset);
+    case A_VI_VCMDQ0_CONS_INDX ... A_VI_VCMDQ127_GERRORN:
+        offset -= 0x20000;
+        QEMU_FALLTHROUGH;
+    case A_VCMDQ0_CONS_INDX ... A_VCMDQ127_GERRORN:
+        /*
+         * Align offset down to 0x10000 while extracting the index:
+         *   VCMDQ0_CONS_INDX  (0x10000) => 0x10000, 0
+         *   VCMDQ1_CONS_INDX  (0x10080) => 0x10000, 1
+         *   VCMDQ2_CONS_INDX  (0x10100) => 0x10000, 2
+         *   ...
+         *   VCMDQ127_CONS_INDX (0x13f80) => 0x10000, 127
+         */
+        index = (offset - 0x10000) / 0x80;
+        return tegra241_cmdqv_read_vcmdq(cmdqv, offset - 0x80 * index, index);
+    case A_VI_VCMDQ0_BASE_L ... A_VI_VCMDQ127_CONS_INDX_BASE_DRAM_H:
+        offset -= 0x20000;
+        QEMU_FALLTHROUGH;
+    case A_VCMDQ0_BASE_L ... A_VCMDQ127_CONS_INDX_BASE_DRAM_H:
+        /*
+         * Align offset down to 0x20000 while extracting the index:
+         *   VCMDQ0_BASE_L  (0x20000) => 0x20000, 0
+         *   VCMDQ1_BASE_L  (0x20080) => 0x20000, 1
+         *   VCMDQ2_BASE_L  (0x20100) => 0x20000, 2
+         *   ...
+         *   VCMDQ127_BASE_L (0x23f80) => 0x20000, 127
+         */
+        index = (offset - 0x20000) / 0x80;
+        return tegra241_cmdqv_read_vcmdq(cmdqv, offset - 0x80 * index, index);
+    default:
+        qemu_log_mask(LOG_UNIMP, "%s unhandled read access at 0x%" PRIx64 "\n",
+                      __func__, offset);
+        return 0;
+    }
 }
 
 static void tegra241_cmdqv_write(void *opaque, hwaddr offset, uint64_t value,
hw/arm/tegra241-cmdqv.h +282 -0

298 lines not on this page · raw message

Signature · 1 line
2.43.0
#30 lore raw

Re: [RFC PATCH 07/16] hw/arm/tegra241-cmdqv: Add read emulation support for registers

Quoted 13 lines · Shameer Kolothum
hw/arm/tegra241-cmdqv.c +13 -0
+/* Note that offset aligns down to 0x10000 */
+static uint64_t tegra241_cmdqv_read_vcmdq(Tegra241CMDQV *cmdqv, hwaddr offset,
+                                          int index)
+{
+    uint32_t *ptr = NULL;
+    uint64_t off;
+
+    /*
+     * Each VCMDQ instance occupies a 128 byte region (0x80).
+     * The hardware layout is:
+     *   vcmdq_page0 + (index * 0x80) + (offset - 0x10000)
+     */
+    if (cmdqv->vcmdq_page0) {

hw/arm/tegra241-cmdqv.c static bool tegra241_cmdqv_init_vcmdq_page0(Tegra241CMDQV *cmdqv, Error **errp) · line 98

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10 -0
+        off = (0x80 * index) + (offset - 0x10000);
+        ptr = (uint32_t *)(cmdqv->vcmdq_page0 + off);
+    }
+
+    switch (offset) {
+    case A_VCMDQ0_CONS_INDX:
+        if (ptr) {
+            cmdqv->vcmdq_cons_indx[index] = *ptr;
+        }
+        return cmdqv->vcmdq_cons_indx[index];

I forgot the details. But this seems a deadcode:
+ if (ptr) {
+ cmdqv->vcmdq_cons_indx[index] = *ptr;
+ }
?

hw/arm/tegra241-cmdqv.h typedef struct Tegra241CMDQV { · line 123

Quoted 8 lines
hw/arm/tegra241-cmdqv.h +8 -0
+/* i = [0, 0], j = [0, 3] */
+#define A_VINTFi_LVCMDQ_ERR_MAP_(i, j)                               \
+    REG32(VINTF##i##_LVCMDQ_ERR_MAP_##j, 0x10c0 + j * 4 + i * 0x100) \
+    FIELD(VINTF##i##_LVCMDQ_ERR_MAP_##j, LVCMDQ_ERR_MAP, 0, 32)
+
+        A_VINTFi_LVCMDQ_ERR_MAP_(0, 0)
+    /* Omitting [0][1~2] as not being directly called */
+    A_VINTFi_LVCMDQ_ERR_MAP_(0, 3)

The indentations at 0 is odd related to 3. And should fix all the following macros too.

Nicolin

#31 lore raw

Re: [RFC PATCH 07/16] hw/arm/tegra241-cmdqv: Add read emulation support for registers

In reply to Shameer Kolothum

Hi Shameer,

Quoted 44 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

Implement read support for Tegra241 CMDQV register blocks, including VINTF and per VCMDQ register regions. The patch decodes offsets, extracts queue indices, and returns the corresponding cached register state.

Subsequent patch will add write support.

---
 hw/arm/tegra241-cmdqv.c | 144 +++++++++++++++++++-
 hw/arm/tegra241-cmdqv.h | 282 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 425 insertions(+), 1 deletion(-)
hw/arm/tegra241-cmdqv.c +14 -0
index d8858322dc..185ef957bc 100644
@@ -8,6 +8,7 @@
  */
 
 #include "qemu/osdep.h"
+#include "qemu/log.h"
 
 #include "hw/arm/smmuv3.h"
 #include "smmuv3-accel.h"
@@ -52,10 +53,94 @@ static bool tegra241_cmdqv_init_vcmdq_page0(Tegra241CMDQV *cmdqv, Error **errp)
     return true;
 }
 
+/* Note that offset aligns down to 0x1000 */
+static uint64_t tegra241_cmdqv_read_vintf(Tegra241CMDQV *cmdqv, hwaddr offset)
+{
+    int i;
+
+    switch (offset) {
+    case A_VINTF0_CONFIG:
+        return cmdqv->vintf_config;
+    case A_VINTF0_STATUS:
+        return cmdqv->vintf_status;
+    case A_VINTF0_LVCMDQ_ERR_MAP_0 ... A_VINTF0_LVCMDQ_ERR_MAP_3:
+        i = (offset - A_VINTF0_LVCMDQ_ERR_MAP_0) / 4;
+        return cmdqv->vintf_cmdq_err_map[i];

hw/arm/tegra241-cmdqv.c static bool tegra241_cmdqv_init_vcmdq_page0(Tegra241CMDQV *cmdqv, Error **errp) · line 78

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10 -0
+    default:
+        qemu_log_mask(LOG_UNIMP, "%s unhandled read access at 0x%" PRIx64 "\n",
+                      __func__, offset);
+        return 0;
+    }
+}
+
+/* Note that offset aligns down to 0x10000 */
+static uint64_t tegra241_cmdqv_read_vcmdq(Tegra241CMDQV *cmdqv, hwaddr offset,
+                                          int index)

the offset is within the instance, right? Better to rename it differently from the main offset

hw/arm/tegra241-cmdqv.c static bool tegra241_cmdqv_init_vcmdq_page0(Tegra241CMDQV *cmdqv, Error **errp) · line 81

Quoted 3 lines
hw/arm/tegra241-cmdqv.c +3 -0
+{
+    uint32_t *ptr = NULL;
+    uint64_t off;

better initialize it?

hw/arm/tegra241-cmdqv.c static bool tegra241_cmdqv_init_vcmdq_page0(Tegra241CMDQV *cmdqv, Error **errp) · line 90

Quoted 9 lines
hw/arm/tegra241-cmdqv.c +9 -0
+
+    /*
+     * Each VCMDQ instance occupies a 128 byte region (0x80).
+     * The hardware layout is:
+     *   vcmdq_page0 + (index * 0x80) + (offset - 0x10000)
+     */
+    if (cmdqv->vcmdq_page0) {
+        off = (0x80 * index) + (offset - 0x10000);
+        ptr = (uint32_t *)(cmdqv->vcmdq_page0 + off);

I am confused because I understood page0 was mmapped so that the guest could read it without VMM mediation?

Quoted 82 lines
hw/arm/tegra241-cmdqv.c +70 -1
+    }
+
+    switch (offset) {
+    case A_VCMDQ0_CONS_INDX:
+        if (ptr) {
+            cmdqv->vcmdq_cons_indx[index] = *ptr;
+        }
+        return cmdqv->vcmdq_cons_indx[index];
+    case A_VCMDQ0_PROD_INDX:
+        if (ptr) {
+            cmdqv->vcmdq_prod_indx[index] = *ptr;
+        }
+        return cmdqv->vcmdq_prod_indx[index];
+    case A_VCMDQ0_CONFIG:
+        if (ptr) {
+            cmdqv->vcmdq_config[index] = *ptr;
+        }
+        return cmdqv->vcmdq_config[index];
+    case A_VCMDQ0_STATUS:
+        if (ptr) {
+            cmdqv->vcmdq_status[index] = *ptr;
+        }
+        return cmdqv->vcmdq_status[index];
+    case A_VCMDQ0_GERROR:
+        if (ptr) {
+            cmdqv->vcmdq_gerror[index] = *ptr;
+        }
+        return cmdqv->vcmdq_gerror[index];
+    case A_VCMDQ0_GERRORN:
+        if (ptr) {
+            cmdqv->vcmdq_gerrorn[index] = *ptr;
+        }
+        return cmdqv->vcmdq_gerrorn[index];
+    case A_VCMDQ0_BASE_L:
+        return cmdqv->vcmdq_base[index];
+    case A_VCMDQ0_BASE_H:
+        return cmdqv->vcmdq_base[index] >> 32;
+    case A_VCMDQ0_CONS_INDX_BASE_DRAM_L:
+        return cmdqv->vcmdq_cons_indx_base[index];
+    case A_VCMDQ0_CONS_INDX_BASE_DRAM_H:
+        return cmdqv->vcmdq_cons_indx_base[index] >> 32;
+    default:
+        qemu_log_mask(LOG_UNIMP,
+                      "%s unhandled read access at 0x%" PRIx64 "\n",
+                      __func__, offset);
+        return 0;
+    }
+}
 static uint64_t tegra241_cmdqv_read(void *opaque, hwaddr offset, unsigned size)
 {
     Tegra241CMDQV *cmdqv = (Tegra241CMDQV *)opaque;
     Error *local_err = NULL;
+    int index;
 
     if (!cmdqv->vcmdq_page0) {
         tegra241_cmdqv_init_vcmdq_page0(cmdqv, &local_err);
@@ -65,7 +150,64 @@ static uint64_t tegra241_cmdqv_read(void *opaque, hwaddr offset, unsigned size)
         }
     }
 
-    return 0;
+    if (offset > TEGRA241_CMDQV_IO_LEN) {
+        qemu_log_mask(LOG_UNIMP,
+                      "%s offset 0x%" PRIx64 " off limit (0x50000)\n", __func__,
+                      offset);
+        return 0;
+    }
+
+    /* Fallback to cached register values */
+    switch (offset) {
+    case A_CONFIG:
+        return cmdqv->config;
+    case A_PARAM:
+        return cmdqv->param;
+    case A_STATUS:
+        return cmdqv->status;
+    case A_VI_ERR_MAP ... A_VI_ERR_MAP_1:
+        return cmdqv->vi_err_map[(offset - A_VI_ERR_MAP) / 4];
+    case A_VI_INT_MASK ... A_VI_INT_MASK_1:
+        return cmdqv->vi_int_mask[(offset - A_VI_INT_MASK) / 4];
+    case A_CMDQ_ERR_MAP ... A_CMDQ_ERR_MAP_3:
+        return cmdqv->cmdq_err_map[(offset - A_CMDQ_ERR_MAP) / 4];

hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_read(void *opaque, hwaddr offset, unsigned size) · line 183

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10 -0
+    case A_CMDQ_ALLOC_MAP_0 ... A_CMDQ_ALLOC_MAP_127:
+        return cmdqv->cmdq_alloc_map[(offset - A_CMDQ_ALLOC_MAP_0) / 4];
+    case A_VINTF0_CONFIG ... A_VINTF0_LVCMDQ_ERR_MAP_3:
+        return tegra241_cmdqv_read_vintf(cmdqv, offset);
+    case A_VI_VCMDQ0_CONS_INDX ... A_VI_VCMDQ127_GERRORN:
+        offset -= 0x20000;
+        QEMU_FALLTHROUGH;
+    case A_VCMDQ0_CONS_INDX ... A_VCMDQ127_GERRORN:
+        /*
+         * Align offset down to 0x10000 while extracting the index:

this is a bit cryptic to me. Could you explain this in the commit msg

Quoted 334 lines
hw/arm/tegra241-cmdqv.c +27 -0
+         *   VCMDQ0_CONS_INDX  (0x10000) => 0x10000, 0
+         *   VCMDQ1_CONS_INDX  (0x10080) => 0x10000, 1
+         *   VCMDQ2_CONS_INDX  (0x10100) => 0x10000, 2
+         *   ...
+         *   VCMDQ127_CONS_INDX (0x13f80) => 0x10000, 127
+         */
+        index = (offset - 0x10000) / 0x80;
+        return tegra241_cmdqv_read_vcmdq(cmdqv, offset - 0x80 * index, index);
+    case A_VI_VCMDQ0_BASE_L ... A_VI_VCMDQ127_CONS_INDX_BASE_DRAM_H:
+        offset -= 0x20000;
+        QEMU_FALLTHROUGH;
+    case A_VCMDQ0_BASE_L ... A_VCMDQ127_CONS_INDX_BASE_DRAM_H:
+        /*
+         * Align offset down to 0x20000 while extracting the index:
+         *   VCMDQ0_BASE_L  (0x20000) => 0x20000, 0
+         *   VCMDQ1_BASE_L  (0x20080) => 0x20000, 1
+         *   VCMDQ2_BASE_L  (0x20100) => 0x20000, 2
+         *   ...
+         *   VCMDQ127_BASE_L (0x23f80) => 0x20000, 127
+         */
+        index = (offset - 0x20000) / 0x80;
+        return tegra241_cmdqv_read_vcmdq(cmdqv, offset - 0x80 * index, index);
+    default:
+        qemu_log_mask(LOG_UNIMP, "%s unhandled read access at 0x%" PRIx64 "\n",
+                      __func__, offset);
+        return 0;
+    }
 }
 
 static void tegra241_cmdqv_write(void *opaque, hwaddr offset, uint64_t value,
hw/arm/tegra241-cmdqv.h +282 -0
index ccdf0651be..4972e367f6 100644
@@ -10,6 +10,7 @@
 #ifndef HW_TEGRA241_CMDQV_H
 #define HW_TEGRA241_CMDQV_H
 
+#include "hw/registerfields.h"
 #include CONFIG_DEVICES
 
 #define TEGRA241_CMDQV_IO_LEN 0x50000
@@ -22,10 +23,291 @@ typedef struct Tegra241CMDQV {
     MemoryRegion mmio_vcmdq_page;
     MemoryRegion mmio_vintf_page;
     void *vcmdq_page0;
+    IOMMUFDHWqueue *vcmdq[128];
+
+    /* Register Cache */
+    uint32_t config;
+    uint32_t param;
+    uint32_t status;
+    uint32_t vi_err_map[2];
+    uint32_t vi_int_mask[2];
+    uint32_t cmdq_err_map[4];
+    uint32_t cmdq_alloc_map[128];
+    uint32_t vintf_config;
+    uint32_t vintf_status;
+    uint32_t vintf_cmdq_err_map[4];
+    uint32_t vcmdq_cons_indx[128];
+    uint32_t vcmdq_prod_indx[128];
+    uint32_t vcmdq_config[128];
+    uint32_t vcmdq_status[128];
+    uint32_t vcmdq_gerror[128];
+    uint32_t vcmdq_gerrorn[128];
+    uint64_t vcmdq_base[128];
+    uint64_t vcmdq_cons_indx_base[128];
 } Tegra241CMDQV;
 
+/* MMIO Registers */
+REG32(CONFIG, 0x0)
+FIELD(CONFIG, CMDQV_EN, 0, 1)
+FIELD(CONFIG, CMDQV_PER_CMD_OFFSET, 1, 3)
+FIELD(CONFIG, CMDQ_MAX_CLK_BATCH, 4, 8)
+FIELD(CONFIG, CMDQ_MAX_CMD_BATCH, 12, 8)
+FIELD(CONFIG, CONS_DRAM_EN, 20, 1)
+
+#define V_CONFIG_RESET 0x00020403
+
+REG32(PARAM, 0x4)
+FIELD(PARAM, CMDQV_VER, 0, 4)
+FIELD(PARAM, CMDQV_NUM_CMDQ_LOG2, 4, 4)
+FIELD(PARAM, CMDQV_NUM_VM_LOG2, 8, 4)
+FIELD(PARAM, CMDQV_NUM_SID_PER_VM_LOG2, 12, 4)
+
+#define V_PARAM_RESET 0x00004011
+
+REG32(STATUS, 0x8)
+FIELD(STATUS, CMDQV_ENABLED, 0, 1)
+
+#define A_VI_ERR_MAP 0x14
+#define A_VI_ERR_MAP_1 0x18
+#define V_VI_ERR_MAP_NO_ERROR (0)
+#define V_VI_ERR_MAP_ERROR (1)
+
+#define A_VI_INT_MASK 0x1c
+#define A_VI_INT_MASK_1 0x20
+#define V_VI_INT_MASK_NOT_MASKED (0)
+#define V_VI_INT_MASK_MASKED (1)
+
+#define A_CMDQ_ERR_MAP 0x24
+#define A_CMDQ_ERR_MAP_1 0x28
+#define A_CMDQ_ERR_MAP_2 0x2c
+#define A_CMDQ_ERR_MAP_3 0x30
+
+/* i = [0, 127] */
+#define A_CMDQ_ALLOC_MAP_(i)                 \
+    REG32(CMDQ_ALLOC_MAP_##i, 0x200 + i * 4) \
+    FIELD(CMDQ_ALLOC_MAP_##i, ALLOC, 0, 1)   \
+    FIELD(CMDQ_ALLOC_MAP_##i, LVCMDQ, 1, 7)  \
+    FIELD(CMDQ_ALLOC_MAP_##i, VIRT_INTF_INDX, 15, 6)
+
+A_CMDQ_ALLOC_MAP_(0)cmdq_alloc_map
 
+/* Omitting 1~126 as not being directly called */
+A_CMDQ_ALLOC_MAP_(127)
+
+/* i = [0, 0] */
+#define A_VINTFi_CONFIG(i)                       \
+    REG32(VINTF##i##_CONFIG, 0x1000 + i * 0x100) \
+    FIELD(VINTF##i##_CONFIG, ENABLE, 0, 1)       \
+    FIELD(VINTF##i##_CONFIG, VMID, 1, 16)        \
+    FIELD(VINTF##i##_CONFIG, HYP_OWN, 17, 1)
+
+A_VINTFi_CONFIG(0)
+
+#define A_VINTFi_STATUS(i)                       \
+    REG32(VINTF##i##_STATUS, 0x1004 + i * 0x100) \
+    FIELD(VINTF##i##_STATUS, ENABLE_OK, 0, 1)    \
+    FIELD(VINTF##i##_STATUS, STATUS, 1, 3)       \
+    FIELD(VINTF##i##_STATUS, VI_NUM_LVCMDQ, 16, 8)
+
+    A_VINTFi_STATUS(0)
+
+#define V_VINTF_STATUS_NO_ERROR (0 << 1)
+#define V_VINTF_STATUS_VCMDQ_EROR (1 << 1)
+
+/* i = [0, 0], j = [0, 3] */
+#define A_VINTFi_LVCMDQ_ERR_MAP_(i, j)                               \
+    REG32(VINTF##i##_LVCMDQ_ERR_MAP_##j, 0x10c0 + j * 4 + i * 0x100) \
+    FIELD(VINTF##i##_LVCMDQ_ERR_MAP_##j, LVCMDQ_ERR_MAP, 0, 32)
+
+        A_VINTFi_LVCMDQ_ERR_MAP_(0, 0)
+    /* Omitting [0][1~2] as not being directly called */
+    A_VINTFi_LVCMDQ_ERR_MAP_(0, 3)
+
+/* VCMDQ registers -- starting from 0x10000 with size 64KB * 2 (0x20000) */
+#define VCMDQ_REG_OFFSET 0x10000
 #define VCMDQ_REG_PAGE_SIZE 0x10000
 
+#define A_VCMDQi_CONS_INDX(i)                       \
+    REG32(VCMDQ##i##_CONS_INDX, 0x10000 + i * 0x80) \
+    FIELD(VCMDQ##i##_CONS_INDX, RD, 0, 20)          \
+    FIELD(VCMDQ##i##_CONS_INDX, ERR, 24, 7)
+
+        A_VCMDQi_CONS_INDX(0)
+    /* Omitting [1~126] as not being directly called */
+    A_VCMDQi_CONS_INDX(127)
+
+#define V_VCMDQ_CONS_INDX_ERR_CERROR_NONE 0
+#define V_VCMDQ_CONS_INDX_ERR_CERROR_ILL_OPCODE 1
+#define V_VCMDQ_CONS_INDX_ERR_CERROR_ABT 2
+#define V_VCMDQ_CONS_INDX_ERR_CERROR_ATC_INV_SYNC 3
+#define V_VCMDQ_CONS_INDX_ERR_CERROR_ILL_ACCESS 4
+
+#define A_VCMDQi_PROD_INDX(i)                             \
+    REG32(VCMDQ##i##_PROD_INDX, 0x10000 + 0x4 + i * 0x80) \
+    FIELD(VCMDQ##i##_PROD_INDX, WR, 0, 20)
+
+        A_VCMDQi_PROD_INDX(0)
+    /* Omitting [1~126] as not being directly called */
+    A_VCMDQi_PROD_INDX(127)
+
+#define A_VCMDQi_CONFIG(i)                             \
+    REG32(VCMDQ##i##_CONFIG, 0x10000 + 0x8 + i * 0x80) \
+    FIELD(VCMDQ##i##_CONFIG, CMDQ_EN, 0, 1)
+
+        A_VCMDQi_CONFIG(0)
+    /* Omitting [1~126] as not being directly called */
+    A_VCMDQi_CONFIG(127)
+
+#define A_VCMDQi_STATUS(i)                             \
+    REG32(VCMDQ##i##_STATUS, 0x10000 + 0xc + i * 0x80) \
+    FIELD(VCMDQ##i##_STATUS, CMDQ_EN_OK, 0, 1)
+
+        A_VCMDQi_STATUS(0)
+    /* Omitting [1~126] as not being directly called */
+    A_VCMDQi_STATUS(127)
+
+#define A_VCMDQi_GERROR(i)                               \
+    REG32(VCMDQ##i##_GERROR, 0x10000 + 0x10 + i * 0x80)  \
+    FIELD(VCMDQ##i##_GERROR, CMDQ_ERR, 0, 1)             \
+    FIELD(VCMDQ##i##_GERROR, CONS_DRAM_WR_ABT_ERR, 1, 1) \
+    FIELD(VCMDQ##i##_GERROR, CMDQ_INIT_ERR, 2, 1)
+
+        A_VCMDQi_GERROR(0)
+    /* Omitting [1~126] as not being directly called */
+    A_VCMDQi_GERROR(127)
+
+#define A_VCMDQi_GERRORN(i)                               \
+    REG32(VCMDQ##i##_GERRORN, 0x10000 + 0x14 + i * 0x80)  \
+    FIELD(VCMDQ##i##_GERRORN, CMDQ_ERR, 0, 1)             \
+    FIELD(VCMDQ##i##_GERRORN, CONS_DRAM_WR_ABT_ERR, 1, 1) \
+    FIELD(VCMDQ##i##_GERRORN, CMDQ_INIT_ERR, 2, 1)
+
+        A_VCMDQi_GERRORN(0)
+    /* Omitting [1~126] as not being directly called */
+    A_VCMDQi_GERRORN(127)
+
+#define A_VCMDQi_BASE_L(i)                       \
+    REG32(VCMDQ##i##_BASE_L, 0x20000 + i * 0x80) \
+    FIELD(VCMDQ##i##_BASE_L, LOG2SIZE, 0, 5)     \
+    FIELD(VCMDQ##i##_BASE_L, ADDR, 5, 27)
+
+        A_VCMDQi_BASE_L(0)
+    /* Omitting [1~126] as not being directly called */
+    A_VCMDQi_BASE_L(127)
+
+#define A_VCMDQi_BASE_H(i)                             \
+    REG32(VCMDQ##i##_BASE_H, 0x20000 + 0x4 + i * 0x80) \
+    FIELD(VCMDQ##i##_BASE_H, ADDR, 0, 16)
+
+        A_VCMDQi_BASE_H(0)
+    /* Omitting [1~126] as not being directly called */
+    A_VCMDQi_BASE_H(127)
+
+#define A_VCMDQi_CONS_INDX_BASE_DRAM_L(i)                             \
+    REG32(VCMDQ##i##_CONS_INDX_BASE_DRAM_L, 0x20000 + 0x8 + i * 0x80) \
+    FIELD(VCMDQ##i##_CONS_INDX_BASE_DRAM_L, ADDR, 0, 32)
+
+        A_VCMDQi_CONS_INDX_BASE_DRAM_L(0)
+    /* Omitting [1~126] as not being directly called */
+    A_VCMDQi_CONS_INDX_BASE_DRAM_L(127)
+
+#define A_VCMDQi_CONS_INDX_BASE_DRAM_H(i)                             \
+    REG32(VCMDQ##i##_CONS_INDX_BASE_DRAM_H, 0x20000 + 0xc + i * 0x80) \
+    FIELD(VCMDQ##i##_CONS_INDX_BASE_DRAM_H, ADDR, 0, 16)
+
+        A_VCMDQi_CONS_INDX_BASE_DRAM_H(0)
+    /* Omitting [1~126] as not being directly called */
+    A_VCMDQi_CONS_INDX_BASE_DRAM_H(127)
+
+/*
+ * VINTF VI_VCMDQ registers -- starting from 0x30000 with size 64KB * 2
+ * (0x20000)
+ */
+#define A_VI_VCMDQi_CONS_INDX(i)                       \
+    REG32(VI_VCMDQ##i##_CONS_INDX, 0x30000 + i * 0x80) \
+    FIELD(VI_VCMDQ##i##_CONS_INDX, RD, 0, 20)          \
+    FIELD(VI_VCMDQ##i##_CONS_INDX, ERR, 24, 7)
+
+        A_VI_VCMDQi_CONS_INDX(0)
+    /* Omitting [1~126] as not being directly called */
+    A_VI_VCMDQi_CONS_INDX(127)
+
+#define A_VI_VCMDQi_PROD_INDX(i)                             \
+    REG32(VI_VCMDQ##i##_PROD_INDX, 0x30000 + 0x4 + i * 0x80) \
+    FIELD(VI_VCMDQ##i##_PROD_INDX, WR, 0, 20)
+
+        A_VI_VCMDQi_PROD_INDX(0)
+    /* Omitting [1~126] as not being directly called */
+    A_VI_VCMDQi_PROD_INDX(127)
+
+#define A_VI_VCMDQi_CONFIG(i)                             \
+    REG32(VI_VCMDQ##i##_CONFIG, 0x30000 + 0x8 + i * 0x80) \
+    FIELD(VI_VCMDQ##i##_CONFIG, CMDQ_EN, 0, 1)
+
+        A_VI_VCMDQi_CONFIG(0)
+    /* Omitting [1~126] as not being directly called */
+    A_VI_VCMDQi_CONFIG(127)
+
+#define A_VI_VCMDQi_STATUS(i)                             \
+    REG32(VI_VCMDQ##i##_STATUS, 0x30000 + 0xc + i * 0x80) \
+    FIELD(VI_VCMDQ##i##_STATUS, CMDQ_EN_OK, 0, 1)
+
+        A_VI_VCMDQi_STATUS(0)
+    /* Omitting [1~126] as not being directly called */
+    A_VI_VCMDQi_STATUS(127)
+
+#define A_VI_VCMDQi_GERROR(i)                               \
+    REG32(VI_VCMDQ##i##_GERROR, 0x30000 + 0x10 + i * 0x80)  \
+    FIELD(VI_VCMDQ##i##_GERROR, CMDQ_ERR, 0, 1)             \
+    FIELD(VI_VCMDQ##i##_GERROR, CONS_DRAM_WR_ABT_ERR, 1, 1) \
+    FIELD(VI_VCMDQ##i##_GERROR, CMDQ_INIT_ERR, 2, 1)
+
+        A_VI_VCMDQi_GERROR(0)
+    /* Omitting [1~126] as not being directly called */
+    A_VI_VCMDQi_GERROR(127)
+
+#define A_VI_VCMDQi_GERRORN(i)                               \
+    REG32(VI_VCMDQ##i##_GERRORN, 0x30000 + 0x14 + i * 0x80)  \
+    FIELD(VI_VCMDQ##i##_GERRORN, CMDQ_ERR, 0, 1)             \
+    FIELD(VI_VCMDQ##i##_GERRORN, CONS_DRAM_WR_ABT_ERR, 1, 1) \
+    FIELD(VI_VCMDQ##i##_GERRORN, CMDQ_INIT_ERR, 2, 1)
+
+        A_VI_VCMDQi_GERRORN(0)
+    /* Omitting [1~126] as not being directly called */
+    A_VI_VCMDQi_GERRORN(127)
+
+#define A_VI_VCMDQi_BASE_L(i)                       \
+    REG32(VI_VCMDQ##i##_BASE_L, 0x40000 + i * 0x80) \
+    FIELD(VI_VCMDQ##i##_BASE_L, LOG2SIZE, 0, 5)     \
+    FIELD(VI_VCMDQ##i##_BASE_L, ADDR, 5, 27)
+
+        A_VI_VCMDQi_BASE_L(0)
+    /* Omitting [1~126] as not being directly called */
+    A_VI_VCMDQi_BASE_L(127)
+
+#define A_VI_VCMDQi_BASE_H(i)                             \
+    REG32(VI_VCMDQ##i##_BASE_H, 0x40000 + 0x4 + i * 0x80) \
+    FIELD(VI_VCMDQ##i##_BASE_H, ADDR, 0, 16)
+
+        A_VI_VCMDQi_BASE_H(0)
+    /* Omitting [1~126] as not being directly called */
+    A_VI_VCMDQi_BASE_H(127)
+
+#define A_VI_VCMDQi_CONS_INDX_BASE_DRAM_L(i)                             \
+    REG32(VI_VCMDQ##i##_CONS_INDX_BASE_DRAM_L, 0x40000 + 0x8 + i * 0x80) \
+    FIELD(VI_VCMDQ##i##_CONS_INDX_BASE_DRAM_L, ADDR, 0, 32)
+
+        A_VI_VCMDQi_CONS_INDX_BASE_DRAM_L(0)
+    /* Omitting [1~126] as not being directly called */
+    A_VI_VCMDQi_CONS_INDX_BASE_DRAM_L(127)
+
+#define A_VI_VCMDQi_CONS_INDX_BASE_DRAM_H(i)                             \
+    REG32(VI_VCMDQ##i##_CONS_INDX_BASE_DRAM_H, 0x40000 + 0xc + i * 0x80) \
+    FIELD(VI_VCMDQ##i##_CONS_INDX_BASE_DRAM_H, ADDR, 0, 16)
+
+        A_VI_VCMDQi_CONS_INDX_BASE_DRAM_H(0)
+    /* Omitting [1~126] as not being directly called */
+    A_VI_VCMDQi_CONS_INDX_BASE_DRAM_H(127)
+
 #ifdef CONFIG_TEGRA241_CMDQV
 bool tegra241_cmdqv_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
Quoted 1 line
                                  uint32_t *out_viommu_id, Error **errp);

Thanks

Eric

#32 lore raw

Re: [RFC PATCH 07/16] hw/arm/tegra241-cmdqv: Add read emulation support for registers

Hi Eric,

Quoted 6 lines · Eric Auger

hw/arm/tegra241-cmdqv.c static bool tegra241_cmdqv_init_vcmdq_page0(Tegra241CMDQV *cmdqv, Error **errp) · line 78

Quoted 3 lines
hw/arm/tegra241-cmdqv.c +3 -0
+/* Note that offset aligns down to 0x10000 */
+static uint64_t tegra241_cmdqv_read_vcmdq(Tegra241CMDQV *cmdqv, hwaddr offset,
+                                          int index)

the offset is within the instance, right? Better to rename it differently from the main offset

In tegra241_cmdqv_read(), its offset is the register offset to the entire CMDQV. That can be A_VCMDQ0_CONS_INDX, A_VCMDQ1_CONS_INDX, until A_VCMDQ127_CONS_INDX.

Then, we align it down to 0x10000 (A_VCMDQ0_CONS_INDX) to pass it into this function, so this function can handle any VCMDQ via the A_VCMDQ0_* registers only, as "index" tells which VCMDQ instead.

Perhaps we could rename this "offset" to "offset0", and elaborate this with a better function comments, e.g.:
/*
* The caller striped away the VCMDQ index from its offset and passed that in
* via @offset0 so any VCMDQ can fit into A_VCMDQ0_CONS_INDX for simplicity. *
* On the other hand, @index will tell which VCMDQ is being accessed here. */

Quoted 4 lines

hw/arm/tegra241-cmdqv.c static bool tegra241_cmdqv_init_vcmdq_page0(Tegra241CMDQV *cmdqv, Error **errp) · line 81

Quoted 3 lines
hw/arm/tegra241-cmdqv.c +3 -0
+{
+    uint32_t *ptr = NULL;
+    uint64_t off;

better initialize it?

It's not used unless cmdqv->vcmdq_page0, so we should probably move it to the if statement instead.

Quoted 10 lines

hw/arm/tegra241-cmdqv.c static bool tegra241_cmdqv_init_vcmdq_page0(Tegra241CMDQV *cmdqv, Error **errp) · line 90

Quoted 9 lines
hw/arm/tegra241-cmdqv.c +9 -0
+
+    /*
+     * Each VCMDQ instance occupies a 128 byte region (0x80).
+     * The hardware layout is:
+     *   vcmdq_page0 + (index * 0x80) + (offset - 0x10000)
+     */
+    if (cmdqv->vcmdq_page0) {
+        off = (0x80 * index) + (offset - 0x10000);
+        ptr = (uint32_t *)(cmdqv->vcmdq_page0 + off);
Quoted 2 lines

I am confused because I understood page0 was mmapped so that the guest could read it without VMM mediation?

Yes. I pointed it out too. This should be cleaned away:
https://lore.kernel.org/qemu-devel/aVLZv+nwUB7+ohuw@Asurada-Nvidia/

So, only return the cached regval unless !cmdqv_page0:

    if (!cmdqv->vcmdq_page0) {
        return cmdqv->vcmdq_cons_indx[index];
    }
Quoted 12 lines

hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_read(void *opaque, hwaddr offset, unsigned size) · line 183

Quoted 3 lines
hw/arm/tegra241-cmdqv.c +3 -0
+    case A_VCMDQ0_CONS_INDX ... A_VCMDQ127_GERRORN:
+        /*
+         * Align offset down to 0x10000 while extracting the index:

this is a bit cryptic to me. Could you explain this in the commit msg

Quoted 8 lines
hw/arm/tegra241-cmdqv.c +8 -0
+         *   VCMDQ0_CONS_INDX  (0x10000) => 0x10000, 0
+         *   VCMDQ1_CONS_INDX  (0x10080) => 0x10000, 1
+         *   VCMDQ2_CONS_INDX  (0x10100) => 0x10000, 2
+         *   ...
+         *   VCMDQ127_CONS_INDX (0x13f80) => 0x10000, 127
+         */
+        index = (offset - 0x10000) / 0x80;
+        return tegra241_cmdqv_read_vcmdq(cmdqv, offset - 0x80 * index, index);

As I mentioned above, it is to ease the tegra241_cmdqv_read_vcmdq() that only needs to handle A_VCMDQ0_* cases in the switch statement.

Yes, let's add in the commit message.

Thanks
Nicolin

#33 lore raw

RFC PATCH 8/16 system/physmem: Add helper to check whether a guest PA maps to RAM

In reply to Shameer Kolothum

From: Nicolin Chen <[email protected]>

Introduce cpu_physical_memory_is_ram(), a helper that performs an address_space translation and returns whether the resolved MemoryRegion is backed by RAM.

This will be used by the upcoming Tegra241 CMDQV support to validate guest provided VCMDQ buffer addresses.

---
 include/exec/cpu-common.h |  2 ++
 system/physmem.c          | 12 ++++++++++++
 2 files changed, 14 insertions(+)
include/exec/cpu-common.h +2 -0
index e0be4ee2b8..76b91d1b9b 100644
@@ -148,6 +148,8 @@ void qemu_flush_coalesced_mmio_buffer(void);
 
 typedef int (RAMBlockIterFunc)(RAMBlock *rb, void *opaque);
 
+bool cpu_physical_memory_is_ram(hwaddr phys_addr);
+
 int qemu_ram_foreach_block(RAMBlockIterFunc func, void *opaque);
 
 /* vl.c */
system/physmem.c +12 -0
index c9869e4049..1f6c821a0e 100644
@@ -4068,6 +4068,18 @@ int cpu_memory_rw_debug(CPUState *cpu, vaddr addr,
     return 0;
 }
 
+bool cpu_physical_memory_is_ram(hwaddr phys_addr)
+{
+    MemoryRegion *mr;
+    hwaddr l = 1;
+
+    RCU_READ_LOCK_GUARD();
+    mr = address_space_translate(&address_space_memory, phys_addr, &phys_addr,
+                                 &l, false, MEMTXATTRS_UNSPECIFIED);
+
+    return memory_region_is_ram(mr);
+}
+
 int qemu_ram_foreach_block(RAMBlockIterFunc func, void *opaque)
 {
     RAMBlock *block;
Signature · 1 line
2.43.0
#34 lore raw

Re: [RFC PATCH 08/16] system/physmem: Add helper to check whether a guest PA maps to RAM

Quoted 34 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

Introduce cpu_physical_memory_is_ram(), a helper that performs an address_space translation and returns whether the resolved MemoryRegion is backed by RAM.

This will be used by the upcoming Tegra241 CMDQV support to validate guest provided VCMDQ buffer addresses.

---
 include/exec/cpu-common.h |  2 ++
 system/physmem.c          | 12 ++++++++++++
 2 files changed, 14 insertions(+)
include/exec/cpu-common.h +2 -0
index e0be4ee2b8..76b91d1b9b 100644
@@ -148,6 +148,8 @@ void qemu_flush_coalesced_mmio_buffer(void);
 
 typedef int (RAMBlockIterFunc)(RAMBlock *rb, void *opaque);
 
+bool cpu_physical_memory_is_ram(hwaddr phys_addr);
+
 int qemu_ram_foreach_block(RAMBlockIterFunc func, void *opaque);
 
 /* vl.c */
system/physmem.c +0 -0
index c9869e4049..1f6c821a0e 100644

system/physmem.c int cpu_memory_rw_debug(CPUState *cpu, vaddr addr, · line 4071

Quoted 4 lines
system/physmem.c +1 -0
@@ -4068,6 +4068,18 @@ int cpu_memory_rw_debug(CPUState *cpu, vaddr addr,
     return 0;
 }
 
+bool cpu_physical_memory_is_ram(hwaddr phys_addr)

I guess this would need to match the new address_space_is_io()

Eric

Quoted 14 lines
system/physmem.c +11 -0
+{
+    MemoryRegion *mr;
+    hwaddr l = 1;
+
+    RCU_READ_LOCK_GUARD();
+    mr = address_space_translate(&address_space_memory, phys_addr, &phys_addr,
+                                 &l, false, MEMTXATTRS_UNSPECIFIED);
+
+    return memory_region_is_ram(mr);
+}
+
 int qemu_ram_foreach_block(RAMBlockIterFunc func, void *opaque)
 {
     RAMBlock *block;
#35 lore raw

RFC PATCH 9/16 hw/arm/tegra241-cmdqv:: Add write emulation for registers

In reply to Shameer Kolothum

From: Nicolin Chen <[email protected]>

Introduces write handling for VINTF and VCMDQ MMIO regions, including status/config updates, queue index tracking, and BASE_L/BASE_H processing. Writes to VCMDQ BASE_L/BASE_H trigger allocation or teardown of an IOMMUFD HW queue.

---
 hw/arm/tegra241-cmdqv.c | 213 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 213 insertions(+)
hw/arm/tegra241-cmdqv.c +213 -0
index 185ef957bc..5e9a980d27 100644
@@ -210,11 +210,158 @@ static uint64_t tegra241_cmdqv_read(void *opaque, hwaddr offset, unsigned size)
     }
 }
 
+/* Note that offset aligns down to 0x1000 */
+static void tegra241_cmdqv_write_vintf(Tegra241CMDQV *cmdqv, hwaddr offset,
+                                       uint64_t value, unsigned size)
+{
+    switch (offset) {
+    case A_VINTF0_CONFIG:
+        /* Strip off HYP_OWN setting from guest kernel */
+        value &= ~R_VINTF0_CONFIG_HYP_OWN_MASK;
+
+        cmdqv->vintf_config = value;
+        if (value & R_VINTF0_CONFIG_ENABLE_MASK) {
+            cmdqv->vintf_status |= R_VINTF0_STATUS_ENABLE_OK_MASK;
+        } else {
+            cmdqv->vintf_status &= ~R_VINTF0_STATUS_ENABLE_OK_MASK;
+        }
+        break;
+    default:
+        qemu_log_mask(LOG_UNIMP, "%s unhandled write access at 0x%" PRIx64 "\n",
+                      __func__, offset);
+        return;
+    }
+}
+
+static bool tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int index,
+                                       Error **errp)
+{
+    SMMUv3State *smmu = cmdqv->smmu;
+    SMMUv3AccelState *s_accel = smmu->s_accel;
+    uint64_t base_mask = (uint64_t)R_VCMDQ0_BASE_L_ADDR_MASK |
+                         (uint64_t)R_VCMDQ0_BASE_H_ADDR_MASK << 32;
+    uint64_t addr = cmdqv->vcmdq_base[index] & base_mask;
+    uint64_t log2 = cmdqv->vcmdq_base[index] & R_VCMDQ0_BASE_L_LOG2SIZE_MASK;
+    uint64_t size = 1ULL << (log2 + 4);
+    IOMMUFDHWqueue *vcmdq = cmdqv->vcmdq[index];
+    IOMMUFDViommu *viommu;
+    IOMMUFDHWqueue *hw_queue;
+    uint32_t hw_queue_id;
+
+    /* Ignore any invalid address. This may come as part of reset etc */
+    if (!cpu_physical_memory_is_ram(addr)) {
+        return true;
+    }
+
+    if (vcmdq) {
+        iommufd_backend_free_id(s_accel->viommu.iommufd, vcmdq->hw_queue_id);
+        cmdqv->vcmdq[index] = NULL;
+        g_free(vcmdq);
+    }
+
+    viommu = &s_accel->viommu;
+    if (!iommufd_backend_alloc_hw_queue(viommu->iommufd, viommu->viommu_id,
+                                        IOMMU_HW_QUEUE_TYPE_TEGRA241_CMDQV,
+                                        index, addr, size, &hw_queue_id,
+                                        errp)) {
+        return false;
+    }
+    hw_queue = g_new(IOMMUFDHWqueue, 1);
+    hw_queue->hw_queue_id = hw_queue_id;
+    hw_queue->viommu = viommu;
+
+    cmdqv->vcmdq[index] = hw_queue;
+    return true;
+}
+
+/* Note that offset aligns down to 0x10000 */
+static void
+tegra241_cmdqv_write_vcmdq(Tegra241CMDQV *cmdqv, hwaddr offset, int index,
+                           uint64_t value, unsigned size, Error **errp)
+{
+    uint32_t *ptr = NULL;
+    uint64_t off;
+
+    if (cmdqv->vcmdq_page0) {
+        off = (0x80 * index) + (offset - 0x10000);
+        ptr = (uint32_t *)(cmdqv->vcmdq_page0 + off);
+    }
+
+    switch (offset) {
+    case A_VCMDQ0_CONS_INDX:
+        if (ptr) {
+            *ptr = value;
+        }
+        cmdqv->vcmdq_cons_indx[index] = value;
+        return;
+    case A_VCMDQ0_PROD_INDX:
+        if (ptr) {
+            *ptr = value;
+        }
+        cmdqv->vcmdq_prod_indx[index] = (uint32_t)value;
+        return;
+    case A_VCMDQ0_CONFIG:
+        if (ptr) {
+            *ptr = (uint32_t)value;
+        } else {
+            if (value & R_VCMDQ0_CONFIG_CMDQ_EN_MASK) {
+                cmdqv->vcmdq_status[index] |= R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;
+            } else {
+                cmdqv->vcmdq_status[index] &= ~R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;
+            }
+        }
+        cmdqv->vcmdq_config[index] = (uint32_t)value;
+        return;
+    case A_VCMDQ0_GERRORN:
+        if (ptr) {
+            *ptr = (uint32_t)value;
+        }
+        cmdqv->vcmdq_gerrorn[index] = (uint32_t)value;
+        return;
+    case A_VCMDQ0_BASE_L:
+        if (size == 8) {
+            cmdqv->vcmdq_base[index] = value;
+        } else if (size == 4) {
+            cmdqv->vcmdq_base[index] =
+                (cmdqv->vcmdq_base[index] & 0xffffffff00000000ULL) |
+                (value & 0xffffffffULL);
+        }
+        tegra241_cmdqv_setup_vcmdq(cmdqv, index, errp);
+        return;
+    case A_VCMDQ0_BASE_H:
+        cmdqv->vcmdq_base[index] =
+            (cmdqv->vcmdq_base[index] & 0xffffffffULL) |
+            ((uint64_t)value << 32);
+        tegra241_cmdqv_setup_vcmdq(cmdqv, index, errp);
+        return;
+    case A_VCMDQ0_CONS_INDX_BASE_DRAM_L:
+        if (size == 8) {
+            cmdqv->vcmdq_cons_indx_base[index] = value;
+        } else if (size == 4) {
+            cmdqv->vcmdq_cons_indx_base[index] =
+                (cmdqv->vcmdq_cons_indx_base[index] & 0xffffffff00000000ULL) |
+                (value & 0xffffffffULL);
+        }
+        return;
+    case A_VCMDQ0_CONS_INDX_BASE_DRAM_H:
+        cmdqv->vcmdq_cons_indx_base[index] =
+            (cmdqv->vcmdq_cons_indx_base[index] & 0xffffffffULL) |
+            ((uint64_t)value << 32);
+        return;
+    default:
+        qemu_log_mask(LOG_UNIMP,
+                      "%s unhandled write access at 0x%" PRIx64 "\n",
+                      __func__, offset);
+        return;
+    }
+}
+
 static void tegra241_cmdqv_write(void *opaque, hwaddr offset, uint64_t value,
                                  unsigned size)
 {
     Tegra241CMDQV *cmdqv = (Tegra241CMDQV *)opaque;
     Error *local_err = NULL;
+    int index;
 
     if (!cmdqv->vcmdq_page0) {
         tegra241_cmdqv_init_vcmdq_page0(cmdqv, &local_err);
@@ -223,6 +370,72 @@ static void tegra241_cmdqv_write(void *opaque, hwaddr offset, uint64_t value,
             local_err = NULL;
         }
     }
+
+    if (offset > TEGRA241_CMDQV_IO_LEN) {
+        qemu_log_mask(LOG_UNIMP,
+                      "%s offset 0x%" PRIx64 " off limit (0x50000)\n", __func__,
+                      offset);
+        return;
+    }
+
+    switch (offset) {
+    case A_CONFIG:
+        cmdqv->config = value;
+        if (value & R_CONFIG_CMDQV_EN_MASK) {
+            cmdqv->status |= R_STATUS_CMDQV_ENABLED_MASK;
+        } else {
+            cmdqv->status &= ~R_STATUS_CMDQV_ENABLED_MASK;
+        }
+        break;
+    case A_VI_INT_MASK ... A_VI_INT_MASK_1:
+        cmdqv->vi_int_mask[(offset - A_VI_INT_MASK) / 4] = value;
+        break;
+    case A_CMDQ_ALLOC_MAP_0 ... A_CMDQ_ALLOC_MAP_127:
+        cmdqv->cmdq_alloc_map[(offset - A_CMDQ_ALLOC_MAP_0) / 4] = value;
+        break;
+    case A_VINTF0_CONFIG ... A_VINTF0_LVCMDQ_ERR_MAP_3:
+        tegra241_cmdqv_write_vintf(cmdqv, offset, value, size);
+        break;
+    case A_VI_VCMDQ0_CONS_INDX ... A_VI_VCMDQ127_GERRORN:
+        offset -= 0x20000;
+        QEMU_FALLTHROUGH;
+    case A_VCMDQ0_CONS_INDX ... A_VCMDQ127_GERRORN:
+        /*
+         * Align offset down to 0x10000 while extracting the index:
+         *   VCMDQ0_CONS_INDX  (0x10000) => 0x10000, 0
+         *   VCMDQ1_CONS_INDX  (0x10080) => 0x10000, 1
+         *   VCMDQ2_CONS_INDX  (0x10100) => 0x10000, 2
+         *   ...
+         *   VCMDQ127_CONS_INDX (0x13f80) => 0x10000, 127
+         */
+        index = (offset - 0x10000) / 0x80;
+        tegra241_cmdqv_write_vcmdq(cmdqv, offset - 0x80 * index, index, value,
+                                   size, &local_err);
+        break;
+    case A_VI_VCMDQ0_BASE_L ... A_VI_VCMDQ127_CONS_INDX_BASE_DRAM_H:
+        offset -= 0x20000;
+        QEMU_FALLTHROUGH;
+    case A_VCMDQ0_BASE_L ... A_VCMDQ127_CONS_INDX_BASE_DRAM_H:
+        /*
+         * Align offset down to 0x20000 while extracting the index:
+         *   VCMDQ0_BASE_L  (0x20000) => 0x20000, 0
+         *   VCMDQ1_BASE_L  (0x20080) => 0x20000, 1
+         *   VCMDQ2_BASE_L  (0x20100) => 0x20000, 2
+         *   ...
+         *   VCMDQ127_BASE_L (0x23f80) => 0x20000, 127
+         */
+        index = (offset - 0x20000) / 0x80;
+        tegra241_cmdqv_write_vcmdq(cmdqv, offset - 0x80 * index, index, value,
+                                   size, &local_err);
+        break;
+    default:
+        qemu_log_mask(LOG_UNIMP, "%s unhandled write access at 0x%" PRIx64 "\n",
+                      __func__, offset);
+    }
+
+    if (local_err) {
+        error_report_err(local_err);
+    }
 }
 
 static const MemoryRegionOps mmio_cmdqv_ops = {
Signature · 1 line
2.43.0
#36 lore raw

Re: [RFC PATCH 09/16] hw/arm/tegra241-cmdqv:: Add write emulation for registers

Hi Shameer,

Quoted 6 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

Introduces write handling for VINTF and VCMDQ MMIO regions, including status/config updates, queue index tracking, and BASE_L/BASE_H processing. Writes to VCMDQ BASE_L/BASE_H trigger allocation or teardown of an IOMMUFD HW queue.

I would strongly recommend to split read/write emulation patches into several patches. The code diff is huge and I am pretty sure you can split that into several functional changes. Then comes the question about to review that with a proper spec.

Eric

Quoted 243 lines
---
 hw/arm/tegra241-cmdqv.c | 213 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 213 insertions(+)
hw/arm/tegra241-cmdqv.c +213 -0
index 185ef957bc..5e9a980d27 100644
@@ -210,11 +210,158 @@ static uint64_t tegra241_cmdqv_read(void *opaque, hwaddr offset, unsigned size)
     }
 }
 
+/* Note that offset aligns down to 0x1000 */
+static void tegra241_cmdqv_write_vintf(Tegra241CMDQV *cmdqv, hwaddr offset,
+                                       uint64_t value, unsigned size)
+{
+    switch (offset) {
+    case A_VINTF0_CONFIG:
+        /* Strip off HYP_OWN setting from guest kernel */
+        value &= ~R_VINTF0_CONFIG_HYP_OWN_MASK;
+
+        cmdqv->vintf_config = value;
+        if (value & R_VINTF0_CONFIG_ENABLE_MASK) {
+            cmdqv->vintf_status |= R_VINTF0_STATUS_ENABLE_OK_MASK;
+        } else {
+            cmdqv->vintf_status &= ~R_VINTF0_STATUS_ENABLE_OK_MASK;
+        }
+        break;
+    default:
+        qemu_log_mask(LOG_UNIMP, "%s unhandled write access at 0x%" PRIx64 "\n",
+                      __func__, offset);
+        return;
+    }
+}
+
+static bool tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int index,
+                                       Error **errp)
+{
+    SMMUv3State *smmu = cmdqv->smmu;
+    SMMUv3AccelState *s_accel = smmu->s_accel;
+    uint64_t base_mask = (uint64_t)R_VCMDQ0_BASE_L_ADDR_MASK |
+                         (uint64_t)R_VCMDQ0_BASE_H_ADDR_MASK << 32;
+    uint64_t addr = cmdqv->vcmdq_base[index] & base_mask;
+    uint64_t log2 = cmdqv->vcmdq_base[index] & R_VCMDQ0_BASE_L_LOG2SIZE_MASK;
+    uint64_t size = 1ULL << (log2 + 4);
+    IOMMUFDHWqueue *vcmdq = cmdqv->vcmdq[index];
+    IOMMUFDViommu *viommu;
+    IOMMUFDHWqueue *hw_queue;
+    uint32_t hw_queue_id;
+
+    /* Ignore any invalid address. This may come as part of reset etc */
+    if (!cpu_physical_memory_is_ram(addr)) {
+        return true;
+    }
+
+    if (vcmdq) {
+        iommufd_backend_free_id(s_accel->viommu.iommufd, vcmdq->hw_queue_id);
+        cmdqv->vcmdq[index] = NULL;
+        g_free(vcmdq);
+    }
+
+    viommu = &s_accel->viommu;
+    if (!iommufd_backend_alloc_hw_queue(viommu->iommufd, viommu->viommu_id,
+                                        IOMMU_HW_QUEUE_TYPE_TEGRA241_CMDQV,
+                                        index, addr, size, &hw_queue_id,
+                                        errp)) {
+        return false;
+    }
+    hw_queue = g_new(IOMMUFDHWqueue, 1);
+    hw_queue->hw_queue_id = hw_queue_id;
+    hw_queue->viommu = viommu;
+
+    cmdqv->vcmdq[index] = hw_queue;
+    return true;
+}
+
+/* Note that offset aligns down to 0x10000 */
+static void
+tegra241_cmdqv_write_vcmdq(Tegra241CMDQV *cmdqv, hwaddr offset, int index,
+                           uint64_t value, unsigned size, Error **errp)
+{
+    uint32_t *ptr = NULL;
+    uint64_t off;
+
+    if (cmdqv->vcmdq_page0) {
+        off = (0x80 * index) + (offset - 0x10000);
+        ptr = (uint32_t *)(cmdqv->vcmdq_page0 + off);
+    }
+
+    switch (offset) {
+    case A_VCMDQ0_CONS_INDX:
+        if (ptr) {
+            *ptr = value;
+        }
+        cmdqv->vcmdq_cons_indx[index] = value;
+        return;
+    case A_VCMDQ0_PROD_INDX:
+        if (ptr) {
+            *ptr = value;
+        }
+        cmdqv->vcmdq_prod_indx[index] = (uint32_t)value;
+        return;
+    case A_VCMDQ0_CONFIG:
+        if (ptr) {
+            *ptr = (uint32_t)value;
+        } else {
+            if (value & R_VCMDQ0_CONFIG_CMDQ_EN_MASK) {
+                cmdqv->vcmdq_status[index] |= R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;
+            } else {
+                cmdqv->vcmdq_status[index] &= ~R_VCMDQ0_STATUS_CMDQ_EN_OK_MASK;
+            }
+        }
+        cmdqv->vcmdq_config[index] = (uint32_t)value;
+        return;
+    case A_VCMDQ0_GERRORN:
+        if (ptr) {
+            *ptr = (uint32_t)value;
+        }
+        cmdqv->vcmdq_gerrorn[index] = (uint32_t)value;
+        return;
+    case A_VCMDQ0_BASE_L:
+        if (size == 8) {
+            cmdqv->vcmdq_base[index] = value;
+        } else if (size == 4) {
+            cmdqv->vcmdq_base[index] =
+                (cmdqv->vcmdq_base[index] & 0xffffffff00000000ULL) |
+                (value & 0xffffffffULL);
+        }
+        tegra241_cmdqv_setup_vcmdq(cmdqv, index, errp);
+        return;
+    case A_VCMDQ0_BASE_H:
+        cmdqv->vcmdq_base[index] =
+            (cmdqv->vcmdq_base[index] & 0xffffffffULL) |
+            ((uint64_t)value << 32);
+        tegra241_cmdqv_setup_vcmdq(cmdqv, index, errp);
+        return;
+    case A_VCMDQ0_CONS_INDX_BASE_DRAM_L:
+        if (size == 8) {
+            cmdqv->vcmdq_cons_indx_base[index] = value;
+        } else if (size == 4) {
+            cmdqv->vcmdq_cons_indx_base[index] =
+                (cmdqv->vcmdq_cons_indx_base[index] & 0xffffffff00000000ULL) |
+                (value & 0xffffffffULL);
+        }
+        return;
+    case A_VCMDQ0_CONS_INDX_BASE_DRAM_H:
+        cmdqv->vcmdq_cons_indx_base[index] =
+            (cmdqv->vcmdq_cons_indx_base[index] & 0xffffffffULL) |
+            ((uint64_t)value << 32);
+        return;
+    default:
+        qemu_log_mask(LOG_UNIMP,
+                      "%s unhandled write access at 0x%" PRIx64 "\n",
+                      __func__, offset);
+        return;
+    }
+}
+
 static void tegra241_cmdqv_write(void *opaque, hwaddr offset, uint64_t value,
                                  unsigned size)
 {
     Tegra241CMDQV *cmdqv = (Tegra241CMDQV *)opaque;
     Error *local_err = NULL;
+    int index;
 
     if (!cmdqv->vcmdq_page0) {
         tegra241_cmdqv_init_vcmdq_page0(cmdqv, &local_err);
@@ -223,6 +370,72 @@ static void tegra241_cmdqv_write(void *opaque, hwaddr offset, uint64_t value,
             local_err = NULL;
         }
     }
+
+    if (offset > TEGRA241_CMDQV_IO_LEN) {
+        qemu_log_mask(LOG_UNIMP,
+                      "%s offset 0x%" PRIx64 " off limit (0x50000)\n", __func__,
+                      offset);
+        return;
+    }
+
+    switch (offset) {
+    case A_CONFIG:
+        cmdqv->config = value;
+        if (value & R_CONFIG_CMDQV_EN_MASK) {
+            cmdqv->status |= R_STATUS_CMDQV_ENABLED_MASK;
+        } else {
+            cmdqv->status &= ~R_STATUS_CMDQV_ENABLED_MASK;
+        }
+        break;
+    case A_VI_INT_MASK ... A_VI_INT_MASK_1:
+        cmdqv->vi_int_mask[(offset - A_VI_INT_MASK) / 4] = value;
+        break;
+    case A_CMDQ_ALLOC_MAP_0 ... A_CMDQ_ALLOC_MAP_127:
+        cmdqv->cmdq_alloc_map[(offset - A_CMDQ_ALLOC_MAP_0) / 4] = value;
+        break;
+    case A_VINTF0_CONFIG ... A_VINTF0_LVCMDQ_ERR_MAP_3:
+        tegra241_cmdqv_write_vintf(cmdqv, offset, value, size);
+        break;
+    case A_VI_VCMDQ0_CONS_INDX ... A_VI_VCMDQ127_GERRORN:
+        offset -= 0x20000;
+        QEMU_FALLTHROUGH;
+    case A_VCMDQ0_CONS_INDX ... A_VCMDQ127_GERRORN:
+        /*
+         * Align offset down to 0x10000 while extracting the index:
+         *   VCMDQ0_CONS_INDX  (0x10000) => 0x10000, 0
+         *   VCMDQ1_CONS_INDX  (0x10080) => 0x10000, 1
+         *   VCMDQ2_CONS_INDX  (0x10100) => 0x10000, 2
+         *   ...
+         *   VCMDQ127_CONS_INDX (0x13f80) => 0x10000, 127
+         */
+        index = (offset - 0x10000) / 0x80;
+        tegra241_cmdqv_write_vcmdq(cmdqv, offset - 0x80 * index, index, value,
+                                   size, &local_err);
+        break;
+    case A_VI_VCMDQ0_BASE_L ... A_VI_VCMDQ127_CONS_INDX_BASE_DRAM_H:
+        offset -= 0x20000;
+        QEMU_FALLTHROUGH;
+    case A_VCMDQ0_BASE_L ... A_VCMDQ127_CONS_INDX_BASE_DRAM_H:
+        /*
+         * Align offset down to 0x20000 while extracting the index:
+         *   VCMDQ0_BASE_L  (0x20000) => 0x20000, 0
+         *   VCMDQ1_BASE_L  (0x20080) => 0x20000, 1
+         *   VCMDQ2_BASE_L  (0x20100) => 0x20000, 2
+         *   ...
+         *   VCMDQ127_BASE_L (0x23f80) => 0x20000, 127
+         */
+        index = (offset - 0x20000) / 0x80;
+        tegra241_cmdqv_write_vcmdq(cmdqv, offset - 0x80 * index, index, value,
+                                   size, &local_err);
+        break;
+    default:
+        qemu_log_mask(LOG_UNIMP, "%s unhandled write access at 0x%" PRIx64 "\n",
+                      __func__, offset);
+    }
+
+    if (local_err) {
+        error_report_err(local_err);
+    }
 }
 
 static const MemoryRegionOps mmio_cmdqv_ops = {
#37 lore raw

RFC PATCH 10/16 hw/arm/tegra241-cmdqv: Allocate vEVENTQ object

In reply to Shameer Kolothum

Allocate a Tegra241 CMDQV type vEVENTQ object so that any host side errors related to the CMDQV can be received and propagated back to the guest.

Event read and propagation will be added in a later patch.

---
 hw/arm/tegra241-cmdqv.c | 51 +++++++++++++++++++++++++++++++++++++++++
 hw/arm/tegra241-cmdqv.h |  1 +
 2 files changed, 52 insertions(+)
hw/arm/tegra241-cmdqv.c +51 -0
index 5e9a980d27..812b027923 100644
@@ -136,6 +136,52 @@ static uint64_t tegra241_cmdqv_read_vcmdq(Tegra241CMDQV *cmdqv, hwaddr offset,
         return 0;
     }
 }
+
+static void tegra241_cmdqv_free_veventq(Tegra241CMDQV *cmdqv)
+{
+    SMMUv3State *smmu = cmdqv->smmu;
+    SMMUv3AccelState *s_accel = smmu->s_accel;
+    IOMMUFDViommu *viommu = &s_accel->viommu;
+    IOMMUFDVeventq *veventq = cmdqv->veventq;
+
+    if (!veventq) {
+        return;
+    }
+
+    iommufd_backend_free_id(viommu->iommufd, veventq->veventq_id);
+    g_free(veventq);
+    cmdqv->veventq = NULL;
+}
+
+static bool tegra241_cmdqv_alloc_veventq(Tegra241CMDQV *cmdqv, Error **errp)
+{
+    SMMUv3State *smmu = cmdqv->smmu;
+    SMMUv3AccelState *s_accel = smmu->s_accel;
+    IOMMUFDViommu *viommu = &s_accel->viommu;
+    IOMMUFDVeventq *veventq;
+    uint32_t veventq_id;
+    uint32_t veventq_fd;
+
+    if (cmdqv->veventq) {
+        return true;
+    }
+
+    if (!iommufd_backend_alloc_veventq(viommu->iommufd, viommu->viommu_id,
+                                       IOMMU_VEVENTQ_TYPE_TEGRA241_CMDQV,
+                                       1 << 16, &veventq_id, &veventq_fd,
+                                       errp)) {
+        error_append_hint(errp, "Tegra241 CMDQV: failed to alloc veventq");
+        return false;
+    }
+
+    veventq = g_new(IOMMUFDVeventq, 1);
+    veventq->veventq_id = veventq_id;
+    veventq->veventq_fd = veventq_fd;
+    veventq->viommu = viommu;
+    cmdqv->veventq = veventq;
+    return true;
+}
+
 static uint64_t tegra241_cmdqv_read(void *opaque, hwaddr offset, unsigned size)
 {
     Tegra241CMDQV *cmdqv = (Tegra241CMDQV *)opaque;
@@ -259,11 +305,16 @@ static bool tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int index,
         g_free(vcmdq);
     }
 
+    if (!tegra241_cmdqv_alloc_veventq(cmdqv, errp)) {
+        return false;
+    }
+
     viommu = &s_accel->viommu;
     if (!iommufd_backend_alloc_hw_queue(viommu->iommufd, viommu->viommu_id,
                                         IOMMU_HW_QUEUE_TYPE_TEGRA241_CMDQV,
                                         index, addr, size, &hw_queue_id,
                                         errp)) {
+        tegra241_cmdqv_free_veventq(cmdqv);
         return false;
     }
     hw_queue = g_new(IOMMUFDHWqueue, 1);
hw/arm/tegra241-cmdqv.h +1 -0
index 4972e367f6..ba7f2a0b1b 100644
@@ -24,6 +24,7 @@ typedef struct Tegra241CMDQV {
     MemoryRegion mmio_vintf_page;
     void *vcmdq_page0;
     IOMMUFDHWqueue *vcmdq[128];
+    IOMMUFDVeventq *veventq;
 
     /* Register Cache */
     uint32_t config;
Signature · 1 line
2.43.0
#38 lore raw

Re: [RFC PATCH 10/16] hw/arm/tegra241-cmdqv: Allocate vEVENTQ object

hw/arm/tegra241-cmdqv.c static bool tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int index, · line 311

Quoted 8 lines · Shameer Kolothum
hw/arm/tegra241-cmdqv.c +4 -0
@@ -259,11 +305,16 @@ static bool tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int index,
         g_free(vcmdq);
     }
 
+    if (!tegra241_cmdqv_alloc_veventq(cmdqv, errp)) {
+        return false;
+    }
+

I think this should be called in tegra241_cmdqv_alloc_viommu().

FWIW, vEVENTQ is crucial for CMDQV, because VINTF/VCMDQ will not work if guest OS doesn't recover the HW error correctly.

Nicolin

#39 lore raw

RE: [RFC PATCH 10/16] hw/arm/tegra241-cmdqv: Allocate vEVENTQ object

Quoted 21 lines

-----Original Message-----
From: Nicolin Chen <[email protected]>
Sent: 29 December 2025 19:50
To: Shameer Kolothum <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; [email protected]; [email protected]; Krishnakant Jaju <[email protected]>
Subject: Re: [RFC PATCH 10/16] hw/arm/tegra241-cmdqv: Allocate vEVENTQ object

Review static bool

Quoted 1 line · Shameer Kolothum
(quoted diff) +0 -0
@@ -259,11 +305,16 @@ static bool

tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int index,

Quoted 3 lines
hw/arm/tegra241-cmdqv.c +0 -0
         g_free(vcmdq);
     }
 
Quoted 6 lines

hw/arm/tegra241-cmdqv.c static bool tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int index, · line 311

Quoted 4 lines
hw/arm/tegra241-cmdqv.c +4 -0
+    if (!tegra241_cmdqv_alloc_veventq(cmdqv, errp)) {
+        return false;
+    }
+

I think this should be called in tegra241_cmdqv_alloc_viommu().

Should/Can we? _alloc_viommu() will happen before the Guest boot. Unlike the SMMUv3 vEVENTQ for accel case, we restrict the hotplug use case when the tegra214-cmdqv is set. And as soon as Guest setup vCMDQ the vEVENTQ is enabled here. Do you see any issues with this approach?

Thanks,
Shameer

Thanks,
Shameer

#40 lore raw

Re: [RFC PATCH 10/16] hw/arm/tegra241-cmdqv: Allocate vEVENTQ object

Quoted 18 lines · Shameer Kolothum
Quoted 18 lines

From: Nicolin Chen <[email protected]>
Sent: 29 December 2025 19:50
To: Shameer Kolothum <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; [email protected]; [email protected]; Krishnakant Jaju <[email protected]>
Subject: Re: [RFC PATCH 10/16] hw/arm/tegra241-cmdqv: Allocate vEVENTQ object

Review static bool

Quoted 1 line · Shameer Kolothum
(quoted diff) +0 -0
@@ -259,11 +305,16 @@ static bool

tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int index,

Quoted 3 lines
hw/arm/tegra241-cmdqv.c +0 -0
         g_free(vcmdq);
     }
 
Quoted 8 lines
Quoted 6 lines

hw/arm/tegra241-cmdqv.c static bool tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int index, · line 311

Quoted 4 lines
hw/arm/tegra241-cmdqv.c +4 -0
+    if (!tegra241_cmdqv_alloc_veventq(cmdqv, errp)) {
+        return false;
+    }
+

I think this should be called in tegra241_cmdqv_alloc_viommu().

Should/Can we? _alloc_viommu() will happen before the Guest boot.

I think we can. vEVENTQ only needs vIOMMU ID. It's per VINTF v.s. per VCMDQ (HW_QUEUE). So, it should follow vIOMMU.

Basically, we can allocate vIOMMU (VINTF) and its vEVENTQ even if the VINTF doesn't contain any LVCMDQs, which are allocated/mapped to the VINTF later via HW_QUEUE allocations.

Quoted 3 lines

Unlike the SMMUv3 vEVENTQ for accel case, we restrict the hotplug use case when the tegra214-cmdqv is set. And as soon as Guest setup vCMDQ the vEVENTQ is enabled here. Do you see any issues with this approach?

Once guest OS is booting, !tegra241_cmdqv_alloc_veventq will fail tegra241_cmdqv_setup_vcmdq while it cannot break the guest OS any more. Then, the whole thing wouldn't work although guest OS runs?

Nicolin

#41 lore raw

RE: [RFC PATCH 10/16] hw/arm/tegra241-cmdqv: Allocate vEVENTQ object

Quoted 43 lines

-----Original Message-----
From: Nicolin Chen <[email protected]>
Sent: 30 December 2025 17:54
To: Shameer Kolothum <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; [email protected]; [email protected]; Krishnakant Jaju <[email protected]>
Subject: Re: [RFC PATCH 10/16] hw/arm/tegra241-cmdqv: Allocate vEVENTQ object

Quoted 6 lines · Shameer Kolothum
Quoted 6 lines

From: Nicolin Chen <[email protected]>
Sent: 29 December 2025 19:50
To: Shameer Kolothum <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jason

Gunthorpe

Quoted 4 lines
Quoted 4 lines

<[email protected]>; [email protected]; [email protected]; [email protected]; Krishnakant Jaju <[email protected]>
Subject: Re: [RFC PATCH 10/16] hw/arm/tegra241-cmdqv: Allocate

vEVENTQ

Quoted 15 lines
Quoted 14 lines

object

Review static bool

Quoted 1 line · Shameer Kolothum
(quoted diff) +0 -0
@@ -259,11 +305,16 @@ static bool

tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int index,

hw/arm/tegra241-cmdqv.c static bool tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int index, · line 311

Quoted 7 lines
hw/arm/tegra241-cmdqv.c +4 -0
         g_free(vcmdq);
     }
 
+    if (!tegra241_cmdqv_alloc_veventq(cmdqv, errp)) {
+        return false;
+    }
+

I think this should be called in tegra241_cmdqv_alloc_viommu().

Quoted 8 lines
Quoted 1 line

Should/Can we? _alloc_viommu() will happen before the Guest boot.

I think we can. vEVENTQ only needs vIOMMU ID. It's per VINTF v.s. per VCMDQ (HW_QUEUE). So, it should follow vIOMMU.

Basically, we can allocate vIOMMU (VINTF) and its vEVENTQ even if the VINTF doesn't contain any LVCMDQs, which are allocated/mapped to the VINTF later via HW_QUEUE allocations.

But what happens when you receive an event from host during that time? Should we just ignore it? Or is that not possible at all from a hoist driver perspective? To me, it looks like the natural flow is setup vEVENTQ when Guest is ready to receive the events.

Quoted 8 lines
Quoted 2 lines

Unlike the SMMUv3 vEVENTQ for accel case, we restrict the hotplug use case when the tegra214-cmdqv is set. And as soon as Guest setup

vCMDQ

Quoted 1 line

the vEVENTQ is enabled here. Do you see any issues with this approach?

Once guest OS is booting, !tegra241_cmdqv_alloc_veventq will fail tegra241_cmdqv_setup_vcmdq while it cannot break the guest OS any more. Then, the whole thing wouldn't work although guest OS runs?

If vEVENTQ is mandatory we could exit the Guest with a proper msg in case tegra241_cmdqv_alloc_veventq() fails, right?

Thanks,
Shameer

Quoted 1 line
#42 lore raw

Re: [RFC PATCH 10/16] hw/arm/tegra241-cmdqv: Allocate vEVENTQ object

Quoted 18 lines · Shameer Kolothum
Quoted 18 lines
Quoted 6 lines
Quoted 6 lines · Shameer Kolothum

Review static bool

Quoted 1 line
(quoted diff) +0 -0
@@ -259,11 +305,16 @@ static bool

tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int index,

Quoted 3 lines
hw/arm/tegra241-cmdqv.c +0 -0
         g_free(vcmdq);
     }
 
Quoted 8 lines
Quoted 6 lines

hw/arm/tegra241-cmdqv.c static bool tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int index, · line 311

Quoted 4 lines
hw/arm/tegra241-cmdqv.c +4 -0
+    if (!tegra241_cmdqv_alloc_veventq(cmdqv, errp)) {
+        return false;
+    }
+

I think this should be called in tegra241_cmdqv_alloc_viommu().

Should/Can we? _alloc_viommu() will happen before the Guest boot.

I think we can. vEVENTQ only needs vIOMMU ID. It's per VINTF v.s. per VCMDQ (HW_QUEUE). So, it should follow vIOMMU.

Quoted 8 lines
Quoted 3 lines

Basically, we can allocate vIOMMU (VINTF) and its vEVENTQ even if the VINTF doesn't contain any LVCMDQs, which are allocated/mapped to the VINTF later via HW_QUEUE allocations.

But what happens when you receive an event from host during that time? Should we just ignore it? Or is that not possible at all from a hoist driver perspective? To me, it looks like the natural flow is setup vEVENTQ when Guest is ready to receive the events.

I don't think that will happen. The IRQ register for VINTF/vIOMMU is LVCMDQ_ERR_MAP. If there is no HW_QUEUE allocated to the VINTF, there shouldn't be a vEVENT. Otherwise, it would be a kernel bug, IMHO.

Quoted 11 lines
Quoted 8 lines
Quoted 2 lines

Unlike the SMMUv3 vEVENTQ for accel case, we restrict the hotplug use case when the tegra214-cmdqv is set. And as soon as Guest setup

vCMDQ

Quoted 1 line

the vEVENTQ is enabled here. Do you see any issues with this approach?

Once guest OS is booting, !tegra241_cmdqv_alloc_veventq will fail tegra241_cmdqv_setup_vcmdq while it cannot break the guest OS any more. Then, the whole thing wouldn't work although guest OS runs?

If vEVENTQ is mandatory we could exit the Guest with a proper msg in case tegra241_cmdqv_alloc_veventq() fails, right?

I don't know. But that doesn't sound right to me. In my mind, once the guest OS is booting, it could only stop with something like a blue screen..

Maybe QEMU experts can shed some light here.

Nicolin

#43 lore raw

RE: [RFC PATCH 10/16] hw/arm/tegra241-cmdqv: Allocate vEVENTQ object

Quoted 44 lines

-----Original Message-----
From: Nicolin Chen <[email protected]>
Sent: 30 December 2025 18:22
To: Shameer Kolothum <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; [email protected]; [email protected]; Krishnakant Jaju <[email protected]>
Subject: Re: [RFC PATCH 10/16] hw/arm/tegra241-cmdqv: Allocate vEVENTQ object

Quoted 1 line · Shameer Kolothum
Quoted 1 line
Quoted 1 line
Quoted 1 line

On Wed, Dec 10, 2025 at 01:37:31PM +0000, Shameer Kolothum

Quoted 17 lines · wrote:
Quoted 17 lines
Quoted 5 lines
Quoted 5 lines

Review static bool

Quoted 1 line
(quoted diff) +0 -0
@@ -259,11 +305,16 @@ static bool

tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int index,

Quoted 3 lines
hw/arm/tegra241-cmdqv.c +0 -0
         g_free(vcmdq);
     }
 
Quoted 8 lines
Quoted 6 lines

hw/arm/tegra241-cmdqv.c static bool tegra241_cmdqv_setup_vcmdq(Tegra241CMDQV *cmdqv, int index, · line 311

Quoted 4 lines
hw/arm/tegra241-cmdqv.c +4 -0
+    if (!tegra241_cmdqv_alloc_veventq(cmdqv, errp)) {
+        return false;
+    }
+

I think this should be called in tegra241_cmdqv_alloc_viommu().

Should/Can we? _alloc_viommu() will happen before the Guest boot.

I think we can. vEVENTQ only needs vIOMMU ID. It's per VINTF v.s. per VCMDQ (HW_QUEUE). So, it should follow vIOMMU.

Quoted 8 lines
Quoted 3 lines

Basically, we can allocate vIOMMU (VINTF) and its vEVENTQ even if the VINTF doesn't contain any LVCMDQs, which are allocated/mapped to the VINTF later via HW_QUEUE allocations.

But what happens when you receive an event from host during that time? Should we just ignore it? Or is that not possible at all from a hoist driver perspective? To me, it looks like the natural flow is setup vEVENTQ when Guest is ready to receive the events.

Quoted 4 lines

I don't think that will happen. The IRQ register for VINTF/vIOMMU is LVCMDQ_ERR_MAP. If there is no HW_QUEUE allocated to the VINTF, there shouldn't be a vEVENT. Otherwise, it would be a kernel bug, IMHO.

Ok. That is reassuring then. I will add that as a comment and move it.

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 2 lines

Unlike the SMMUv3 vEVENTQ for accel case, we restrict the hotplug use case when the tegra214-cmdqv is set. And as soon as Guest setup

vCMDQ

Quoted 1 line

the vEVENTQ is enabled here. Do you see any issues with this approach?

Once guest OS is booting, !tegra241_cmdqv_alloc_veventq will fail tegra241_cmdqv_setup_vcmdq while it cannot break the guest OS any more. Then, the whole thing wouldn't work although guest OS runs?

Quoted 8 lines
Quoted 2 lines

If vEVENTQ is mandatory we could exit the Guest with a proper msg in case tegra241_cmdqv_alloc_veventq() fails, right?

I don't know. But that doesn't sound right to me. In my mind, once the guest OS is booting, it could only stop with something like a blue screen..

Maybe QEMU experts can shed some light here.

I think, generally it is a last resort option and should be avoided.

Thanks,
Shameer

#44 lore raw

RFC PATCH 11/16 hw/arm/tegra241-cmdqv: Read and propagate Tegra241 CMDQV errors

In reply to Shameer Kolothum

Install an event handler on the CMDQV vEVENTQ fd to read and propagate host received CMDQV errors to the guest.

The handler runs in QEMU’s main loop, using a non-blocking fd registered via qemu_set_fd_handler().

---
 hw/arm/tegra241-cmdqv.c | 80 +++++++++++++++++++++++++++++++++++++++++
 hw/arm/tegra241-cmdqv.h |  2 ++
 hw/arm/trace-events     |  3 ++
 3 files changed, 85 insertions(+)
hw/arm/tegra241-cmdqv.c +80 -0
index 812b027923..5b8a7bdff2 100644
@@ -8,9 +8,12 @@
  */
 
 #include "qemu/osdep.h"
+#include "qemu/error-report.h"
 #include "qemu/log.h"
+#include "trace.h"
 
 #include "hw/arm/smmuv3.h"
+#include "hw/irq.h"
 #include "smmuv3-accel.h"
 #include "tegra241-cmdqv.h"
 
@@ -137,6 +140,79 @@ static uint64_t tegra241_cmdqv_read_vcmdq(Tegra241CMDQV *cmdqv, hwaddr offset,
     }
 }
 
+static void tegra241_cmdqv_event_read(void *opaque)
+{
+    Tegra241CMDQV *cmdqv = opaque;
+    struct {
+        struct iommufd_vevent_header hdr;
+        struct iommu_vevent_tegra241_cmdqv vevent;
+    } buf;
+    ssize_t readsz = sizeof(buf);
+    uint32_t last_seq = cmdqv->last_event_seq;
+    ssize_t bytes;
+
+    bytes = read(cmdqv->veventq->veventq_fd, &buf, readsz);
+    if (bytes <= 0) {
+        if (errno == EAGAIN || errno == EINTR) {
+            return;
+        }
+        error_report("Tegra241 CMDQV: vEVENTQ: read failed (%s)",
+                     strerror(errno));
+        return;
+    }
+
+    if (bytes < readsz) {
+        error_report("Tegra241 CMDQV: vEVENTQ: incomplete read (%zd/%zd bytes)",
+                     bytes, readsz);
+        return;
+    }
+
+    if (buf.hdr.flags & IOMMU_VEVENTQ_FLAG_LOST_EVENTS) {
+        error_report("Tegra241 CMDQV: vEVENTQ has lost events");
+        return;
+    }
+
+    /* Check sequence in hdr for lost events if any */
+    if (cmdqv->event_start) {
+        uint32_t expected = (last_seq == INT_MAX) ? 0 : last_seq + 1;
+
+        if (buf.hdr.sequence != expected) {
+            uint32_t delta;
+
+            if (buf.hdr.sequence >= last_seq) {
+                delta = buf.hdr.sequence - last_seq;
+            } else {
+                /* Handle wraparound from INT_MAX */
+                delta = (INT_MAX - last_seq) + buf.hdr.sequence + 1;
+            }
+            error_report("Tegra241 CMDQV: vEVENTQ: detected lost %u event(s)",
+                         delta - 1);
+        }
+    }
+
+    if (buf.vevent.lvcmdq_err_map[0] || buf.vevent.lvcmdq_err_map[1]) {
+        cmdqv->vintf_cmdq_err_map[0] =
+            buf.vevent.lvcmdq_err_map[0] & 0xffffffff;
+        cmdqv->vintf_cmdq_err_map[1] =
+            (buf.vevent.lvcmdq_err_map[0] >> 32) & 0xffffffff;
+        cmdqv->vintf_cmdq_err_map[2] =
+            buf.vevent.lvcmdq_err_map[1] & 0xffffffff;
+        cmdqv->vintf_cmdq_err_map[3] =
+            (buf.vevent.lvcmdq_err_map[1] >> 32) & 0xffffffff;
+        for (int i = 0; i < 4; i++) {
+            cmdqv->cmdq_err_map[i] = cmdqv->vintf_cmdq_err_map[i];
+        }
+        cmdqv->vi_err_map[0] |= 0x1;
+        qemu_irq_pulse(cmdqv->irq);
+        trace_tegra241_cmdqv_err_map(
+        cmdqv->vintf_cmdq_err_map[3], cmdqv->vintf_cmdq_err_map[2],
+        cmdqv->vintf_cmdq_err_map[1], cmdqv->vintf_cmdq_err_map[0]);
+    }
+
+    cmdqv->last_event_seq = buf.hdr.sequence;
+    cmdqv->event_start = true;
+}
+
 static void tegra241_cmdqv_free_veventq(Tegra241CMDQV *cmdqv)
 {
     SMMUv3State *smmu = cmdqv->smmu;
@@ -179,6 +255,10 @@ static bool tegra241_cmdqv_alloc_veventq(Tegra241CMDQV *cmdqv, Error **errp)
     veventq->veventq_fd = veventq_fd;
     veventq->viommu = viommu;
     cmdqv->veventq = veventq;
+
+    /* Set up event handler for veventq fd */
+    fcntl(veventq_fd, F_SETFL, O_NONBLOCK);
+    qemu_set_fd_handler(veventq_fd, tegra241_cmdqv_event_read, NULL, cmdqv);
     return true;
 }
 
hw/arm/tegra241-cmdqv.h +2 -0
index ba7f2a0b1b..97eaef8a72 100644
@@ -25,6 +25,8 @@ typedef struct Tegra241CMDQV {
     void *vcmdq_page0;
     IOMMUFDHWqueue *vcmdq[128];
     IOMMUFDVeventq *veventq;
+    uint32_t last_event_seq;
+    bool event_start;
 
     /* Register Cache */
     uint32_t config;
hw/arm/trace-events +3 -0
index 3457536fb0..76bda0efef 100644
@@ -72,6 +72,9 @@ smmuv3_accel_unset_iommu_device(int devfn, uint32_t devid) "devfn=0x%x (idev dev
 smmuv3_accel_translate_ste(uint32_t vsid, uint32_t hwpt_id, uint64_t ste_1, uint64_t ste_0) "vSID=0x%x hwpt_id=0x%x ste=%"PRIx64":%"PRIx64
 smmuv3_accel_install_ste(uint32_t vsid, const char * type, uint32_t hwpt_id) "vSID=0x%x ste type=%s hwpt_id=0x%x"
 
+# tegra241-cmdqv
+tegra241_cmdqv_err_map(uint32_t map3, uint32_t map2, uint32_t map1, uint32_t map0) "hw irq received. error (hex) maps: %04X:%04X:%04X:%04X"
+
 # strongarm.c
 strongarm_uart_update_parameters(const char *label, int speed, char parity, int data_bits, int stop_bits) "%s speed=%d parity=%c data=%d stop=%d"
 strongarm_ssp_read_underrun(void) "SSP rx underrun"
Signature · 1 line
2.43.0
#45 lore raw

Re: [RFC PATCH 11/16] hw/arm/tegra241-cmdqv: Read and propagate Tegra241 CMDQV errors

Quoted 39 lines · Shameer Kolothum
hw/arm/tegra241-cmdqv.c +39 -0
+static void tegra241_cmdqv_event_read(void *opaque)
+{
+    Tegra241CMDQV *cmdqv = opaque;
+    struct {
+        struct iommufd_vevent_header hdr;
+        struct iommu_vevent_tegra241_cmdqv vevent;
+    } buf;
+    ssize_t readsz = sizeof(buf);
+    uint32_t last_seq = cmdqv->last_event_seq;
+    ssize_t bytes;
+
+    bytes = read(cmdqv->veventq->veventq_fd, &buf, readsz);
+    if (bytes <= 0) {
+        if (errno == EAGAIN || errno == EINTR) {
+            return;
+        }
+        error_report("Tegra241 CMDQV: vEVENTQ: read failed (%s)",
+                     strerror(errno));
+        return;
+    }
+
+    if (bytes < readsz) {
+        error_report("Tegra241 CMDQV: vEVENTQ: incomplete read (%zd/%zd bytes)",
+                     bytes, readsz);
+        return;
+    }
+
+    if (buf.hdr.flags & IOMMU_VEVENTQ_FLAG_LOST_EVENTS) {
+        error_report("Tegra241 CMDQV: vEVENTQ has lost events");
+        return;
+    }
+
+    /* Check sequence in hdr for lost events if any */
+    if (cmdqv->event_start) {
+        uint32_t expected = (last_seq == INT_MAX) ? 0 : last_seq + 1;
+
+        if (buf.hdr.sequence != expected) {
+            uint32_t delta;
+

hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_read_vcmdq(Tegra241CMDQV *cmdqv, hwaddr offset, · line 191

Quoted 10 lines
hw/arm/tegra241-cmdqv.c +10 -0
+            if (buf.hdr.sequence >= last_seq) {
+                delta = buf.hdr.sequence - last_seq;
+            } else {
+                /* Handle wraparound from INT_MAX */
+                delta = (INT_MAX - last_seq) + buf.hdr.sequence + 1;
+            }
+            error_report("Tegra241 CMDQV: vEVENTQ: detected lost %u event(s)",
+                         delta - 1);
+        }
+    }

This part looks quite similar to SMMUv3's vEVENTQ function. Maybe add a vEVENTQ helper in the iommufd backend?

Nicolin

#46 lore raw

Re: [RFC PATCH 11/16] hw/arm/tegra241-cmdqv: Read and propagate Tegra241 CMDQV errors

Quoted 44 lines · Nicolin Chen
Quoted 43 lines · Shameer Kolothum
hw/arm/tegra241-cmdqv.c +43 -0
+static void tegra241_cmdqv_event_read(void *opaque)
+{
+    Tegra241CMDQV *cmdqv = opaque;
+    struct {
+        struct iommufd_vevent_header hdr;
+        struct iommu_vevent_tegra241_cmdqv vevent;
+    } buf;
+    ssize_t readsz = sizeof(buf);
+    uint32_t last_seq = cmdqv->last_event_seq;
+    ssize_t bytes;
+
+    bytes = read(cmdqv->veventq->veventq_fd, &buf, readsz);
+    if (bytes <= 0) {
+        if (errno == EAGAIN || errno == EINTR) {
+            return;
+        }
+        error_report("Tegra241 CMDQV: vEVENTQ: read failed (%s)",
+                     strerror(errno));
+        return;
+    }
+
+    if (bytes < readsz) {
+        error_report("Tegra241 CMDQV: vEVENTQ: incomplete read (%zd/%zd bytes)",
+                     bytes, readsz);
+        return;
+    }
+
+    if (buf.hdr.flags & IOMMU_VEVENTQ_FLAG_LOST_EVENTS) {
+        error_report("Tegra241 CMDQV: vEVENTQ has lost events");
+        return;
+    }
+
+    /* Check sequence in hdr for lost events if any */
+    if (cmdqv->event_start) {
+        uint32_t expected = (last_seq == INT_MAX) ? 0 : last_seq + 1;
+
+        if (buf.hdr.sequence != expected) {
+            uint32_t delta;
+
+            if (buf.hdr.sequence >= last_seq) {
+                delta = buf.hdr.sequence - last_seq;
+            } else {
+                /* Handle wraparound from INT_MAX */
Quoted 8 lines

hw/arm/tegra241-cmdqv.c static uint64_t tegra241_cmdqv_read_vcmdq(Tegra241CMDQV *cmdqv, hwaddr offset, · line 191

Quoted 6 lines
hw/arm/tegra241-cmdqv.c +6 -0
+                delta = (INT_MAX - last_seq) + buf.hdr.sequence + 1;
+            }
+            error_report("Tegra241 CMDQV: vEVENTQ: detected lost %u event(s)",
+                         delta - 1);
+        }
+    }

This part looks quite similar to SMMUv3's vEVENTQ function. Maybe add a vEVENTQ helper in the iommufd backend?

+1

Eric

Quoted 3 lines

Nicolin

#47 lore raw

RFC PATCH 12/16 hw/arm/tegra241-cmdqv: Add reset handler

In reply to Shameer Kolothum

From: Nicolin Chen <[email protected]>

Introduce a reset handler for the Tegra241 CMDQV and initialize its register state.

CMDQV gets initialized early during guest boot, hence the handler verifies that at least one cold-plugged device is attached to the associated vIOMMU before proceeding. This is required to retrieve host CMDQV info and to validate it against the QEMU implementation support.

---
 hw/arm/smmuv3.c         |   1 +
 hw/arm/tegra241-cmdqv.c | 105 ++++++++++++++++++++++++++++++++++++++++
 hw/arm/tegra241-cmdqv.h |   7 +++
 hw/arm/trace-events     |   1 +
 4 files changed, 114 insertions(+)
hw/arm/smmuv3.c +1 -0
index 02e1a925a4..ec8687d39a 100644
@@ -1943,6 +1943,7 @@ static void smmu_reset_exit(Object *obj, ResetType type)
 
     smmuv3_reset(s);
     smmuv3_accel_reset(s);
+    tegra241_cmdqv_reset(s);
 }
 
 static bool smmu_validate_property(SMMUv3State *s, Error **errp)
hw/arm/tegra241-cmdqv.c +105 -0

111 lines not on this page · raw message

hw/arm/tegra241-cmdqv.h +7 -0
index 97eaef8a72..0e8729c0b0 100644
@@ -13,6 +13,9 @@
 #include "hw/registerfields.h"
 #include CONFIG_DEVICES
 
+#define TEGRA241_CMDQV_VERSION 0x1
+#define TEGRA241_CMDQV_NUM_CMDQ_LOG2 0x1
+#define TEGRA241_CMDQV_NUM_SID_PER_VM_LOG2 0x4
 #define TEGRA241_CMDQV_IO_LEN 0x50000
 
 typedef struct Tegra241CMDQV {
@@ -314,11 +317,15 @@ A_VINTFi_CONFIG(0)
 #ifdef CONFIG_TEGRA241_CMDQV
 bool tegra241_cmdqv_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
                                  uint32_t *out_viommu_id, Error **errp);
+void tegra241_cmdqv_reset(SMMUv3State *s);
 void tegra241_cmdqv_init(SMMUv3State *s);
 #else
 static inline void tegra241_cmdqv_init(SMMUv3State *s)
 {
 }
+static inline void tegra241_cmdqv_reset(SMMUv3State *s)
+{
+}
 static inline bool
 tegra241_cmdqv_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
                             uint32_t *out_viommu_id, Error **errp)
hw/arm/trace-events +1 -0
index 76bda0efef..ef495c040c 100644
@@ -74,6 +74,7 @@ smmuv3_accel_install_ste(uint32_t vsid, const char * type, uint32_t hwpt_id) "vS
 
 # tegra241-cmdqv
 tegra241_cmdqv_err_map(uint32_t map3, uint32_t map2, uint32_t map1, uint32_t map0) "hw irq received. error (hex) maps: %04X:%04X:%04X:%04X"
+tegra241_cmdqv_init_regs(uint32_t param) "hw info received. param: 0x%04X"
 
 # strongarm.c
 strongarm_uart_update_parameters(const char *label, int speed, char parity, int data_bits, int stop_bits) "%s speed=%d parity=%c data=%d stop=%d"
Signature · 1 line
2.43.0
#48 lore raw

RFC PATCH 13/16 hw/arm/tegra241-cmdqv: Limit queue size based on backend page size

In reply to Shameer Kolothum

From: Nicolin Chen <[email protected]>

CMDQV HW reads guest queue memory in its host physical address setup via IOMUUFD. This requires the guest queue memory isn't only contiguous in guest PA space but also in host PA space. With Tegra241 CMDQV enabled, we must only advertise a CMDQV size that the host can safely back with physically contiguous memory. Allowing a CMDQV larger than the host page size could cause the hardware to DMA across page boundaries leading to faults.

Limit IDR1.CMDQS so the guest cannot configure a CMDQV that exceeds the host’s contiguous backing.

---
 hw/arm/tegra241-cmdqv.c | 43 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 43 insertions(+)
hw/arm/tegra241-cmdqv.c +43 -0
index 1f62b7627a..1996d899a1 100644
@@ -11,10 +11,14 @@
 #include "qemu/error-report.h"
 #include "qemu/log.h"
 #include "trace.h"
+#include <math.h>
 
 #include "hw/arm/smmuv3.h"
 #include "hw/irq.h"
 #include "smmuv3-accel.h"
+#include "smmuv3-internal.h"
+#include "system/ramblock.h"
+#include "exec/ramlist.h"
 #include "tegra241-cmdqv.h"
 
 static bool tegra241_cmdqv_init_vcmdq_page0(Tegra241CMDQV *cmdqv, Error **errp)
@@ -592,6 +596,33 @@ bool tegra241_cmdqv_alloc_viommu(SMMUv3State *s, HostIOMMUDeviceIOMMUFD *idev,
     return true;
 }
 
+static size_t tegra241_cmdqv_min_ram_pagesize(void)
+{
+    RAMBlock *rb;
+    size_t pg, min_pg = SIZE_MAX;
+
+    RAMBLOCK_FOREACH(rb) {
+        MemoryRegion *mr = rb->mr;
+
+        /* Only consider real RAM regions */
+        if (!mr || !memory_region_is_ram(mr)) {
+            continue;
+        }
+
+        /* Skip RAM regions that are not backed by a memory-backend */
+        if (!object_dynamic_cast(mr->owner, TYPE_MEMORY_BACKEND)) {
+            continue;
+        }
+
+        pg = qemu_ram_pagesize(rb);
+        if (pg && pg < min_pg) {
+            min_pg = pg;
+        }
+    }
+
+    return (min_pg == SIZE_MAX) ? qemu_real_host_page_size() : min_pg;
+}
+
 static void tegra241_cmdqv_init_regs(SMMUv3State *s, Tegra241CMDQV *cmdqv)
 {
     SMMUv3AccelState *s_accel = s->s_accel;
@@ -599,7 +630,9 @@ static void tegra241_cmdqv_init_regs(SMMUv3State *s, Tegra241CMDQV *cmdqv)
     struct iommu_hw_info_tegra241_cmdqv cmdqv_info;
     SMMUv3AccelDevice *accel_dev;
     Error *local_err = NULL;
+    size_t pgsize;
     uint64_t caps;
+    uint32_t val;
     int i;
 
     if (QLIST_EMPTY(&s_accel->device_list)) {
@@ -670,6 +703,16 @@ static void tegra241_cmdqv_init_regs(SMMUv3State *s, Tegra241CMDQV *cmdqv)
         cmdqv->vcmdq_base[i] = 0;
         cmdqv->vcmdq_cons_indx_base[i] = 0;
     }
+
+    /*
+     * CMDQ must not cross a physical RAM backend page. Adjust CMDQS so the
+     * queue fits entirely within the smallest backend page size.
+     * FIXME: Migration support requires this to be taken care.
+     */
+    pgsize = tegra241_cmdqv_min_ram_pagesize();
+    val = FIELD_EX32(s->idr[1], IDR1, CMDQS);
+    s->idr[1] = FIELD_DP32(s->idr[1], IDR1, CMDQS, MIN(log2(pgsize) - 4, val));
+
     return;
 
 out_err:
Signature · 1 line
2.43.0
#49 lore raw

Re: [RFC PATCH 13/16] hw/arm/tegra241-cmdqv: Limit queue size based on backend page size

hw/arm/tegra241-cmdqv.c static void tegra241_cmdqv_init_regs(SMMUv3State *s, Tegra241CMDQV *cmdqv) · line 714

Quoted 8 lines · Shameer Kolothum
hw/arm/tegra241-cmdqv.c +8 -0
+    /*
+     * CMDQ must not cross a physical RAM backend page. Adjust CMDQS so the
+     * queue fits entirely within the smallest backend page size.
+     * FIXME: Migration support requires this to be taken care.
+     */
+    pgsize = tegra241_cmdqv_min_ram_pagesize();
+    val = FIELD_EX32(s->idr[1], IDR1, CMDQS);
+    s->idr[1] = FIELD_DP32(s->idr[1], IDR1, CMDQS, MIN(log2(pgsize) - 4, val));

Let's mention it is for the physical contiguity of the Q memory.

Nicolin

#50 lore raw

RFC PATCH 14/16 virt-acpi-build: Rename AcpiIortSMMUv3Dev to AcpiSMMUv3Dev

In reply to Shameer Kolothum

Rename struct AcpiIortSMMUv3Dev to AcpiSMMUv3Dev so that it is not specific to IORT. Subsequent Tegra241 CMDQV support patch will use the same struct to build CMDQV DSDT support as well.

No functional changes intended.

---
 hw/arm/virt-acpi-build.c | 36 ++++++++++++++++++------------------
 1 file changed, 18 insertions(+), 18 deletions(-)
hw/arm/virt-acpi-build.c +18 -18
index 1e3779991e..4f8d36dae0 100644
@@ -339,7 +339,7 @@ static int iort_idmap_compare(gconstpointer a, gconstpointer b)
     return idmap_a->input_base - idmap_b->input_base;
 }
 
-typedef struct AcpiIortSMMUv3Dev {
+typedef struct AcpiSMMUv3Dev {
     int irq;
     hwaddr base;
     GArray *rc_smmu_idmaps;
@@ -347,16 +347,16 @@ typedef struct AcpiIortSMMUv3Dev {
     size_t offset;
     bool accel;
     bool ats;
-} AcpiIortSMMUv3Dev;
+} AcpiSMMUv3Dev;
 
 /*
- * Populate the struct AcpiIortSMMUv3Dev for the legacy SMMUv3 and
+ * Populate the struct AcpiSMMUv3Dev for the legacy SMMUv3 and
  * return the total number of associated idmaps.
  */
 static int populate_smmuv3_legacy_dev(GArray *sdev_blob)
 {
     VirtMachineState *vms = VIRT_MACHINE(qdev_get_machine());
-    AcpiIortSMMUv3Dev sdev;
+    AcpiSMMUv3Dev sdev;
 
     sdev.rc_smmu_idmaps = g_array_new(false, true, sizeof(AcpiIortIdMapping));
     object_child_foreach_recursive(object_get_root(), iort_host_bridges,
@@ -376,8 +376,8 @@ static int populate_smmuv3_legacy_dev(GArray *sdev_blob)
 
 static int smmuv3_dev_idmap_compare(gconstpointer a, gconstpointer b)
 {
-    AcpiIortSMMUv3Dev *sdev_a = (AcpiIortSMMUv3Dev *)a;
-    AcpiIortSMMUv3Dev *sdev_b = (AcpiIortSMMUv3Dev *)b;
+    AcpiSMMUv3Dev *sdev_a = (AcpiSMMUv3Dev *)a;
+    AcpiSMMUv3Dev *sdev_b = (AcpiSMMUv3Dev *)b;
     AcpiIortIdMapping *map_a = &g_array_index(sdev_a->rc_smmu_idmaps,
                                               AcpiIortIdMapping, 0);
     AcpiIortIdMapping *map_b = &g_array_index(sdev_b->rc_smmu_idmaps,
@@ -391,7 +391,7 @@ static int iort_smmuv3_devices(Object *obj, void *opaque)
     GArray *sdev_blob = opaque;
     AcpiIortIdMapping idmap;
     PlatformBusDevice *pbus;
-    AcpiIortSMMUv3Dev sdev;
+    AcpiSMMUv3Dev sdev;
     int min_bus, max_bus;
     SysBusDevice *sbdev;
     PCIBus *bus;
@@ -421,7 +421,7 @@ static int iort_smmuv3_devices(Object *obj, void *opaque)
 }
 
 /*
- * Populate the struct AcpiIortSMMUv3Dev for all SMMUv3 devices and
+ * Populate the struct AcpiSMMUv3Dev for all SMMUv3 devices and
  * return the total number of idmaps.
  */
 static int populate_smmuv3_dev(GArray *sdev_blob)
@@ -442,10 +442,10 @@ static void create_rc_its_idmaps(GArray *its_idmaps, GArray *smmuv3_devs)
 {
     AcpiIortIdMapping *idmap;
     AcpiIortIdMapping next_range = {0};
-    AcpiIortSMMUv3Dev *sdev;
+    AcpiSMMUv3Dev *sdev;
 
     for (int i = 0; i < smmuv3_devs->len; i++) {
-        sdev = &g_array_index(smmuv3_devs, AcpiIortSMMUv3Dev, i);
+        sdev = &g_array_index(smmuv3_devs, AcpiSMMUv3Dev, i);
         /*
          * Based on the RID ranges that are directed to the SMMU, determine the
          * bypassed RID ranges, i.e., the ones that are directed to the ITS
@@ -479,7 +479,7 @@ static void create_rc_its_idmaps(GArray *its_idmaps, GArray *smmuv3_devs)
 static void
 build_iort_rmr_nodes(GArray *table_data, GArray *smmuv3_devices, uint32_t *id)
 {
-    AcpiIortSMMUv3Dev *sdev;
+    AcpiSMMUv3Dev *sdev;
     AcpiIortIdMapping *idmap;
     int i;
 
@@ -487,7 +487,7 @@ build_iort_rmr_nodes(GArray *table_data, GArray *smmuv3_devices, uint32_t *id)
         uint16_t rmr_len;
         int bdf;
 
-        sdev = &g_array_index(smmuv3_devices, AcpiIortSMMUv3Dev, i);
+        sdev = &g_array_index(smmuv3_devices, AcpiSMMUv3Dev, i);
         if (!sdev->accel) {
             continue;
         }
@@ -544,13 +544,13 @@ static void
 build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
 {
     int i, nb_nodes, rc_mapping_count;
-    AcpiIortSMMUv3Dev *sdev;
+    AcpiSMMUv3Dev *sdev;
     size_t node_size;
     bool ats_needed = false;
     int num_smmus = 0;
     uint32_t id = 0;
     int rc_smmu_idmaps_len = 0;
-    GArray *smmuv3_devs = g_array_new(false, true, sizeof(AcpiIortSMMUv3Dev));
+    GArray *smmuv3_devs = g_array_new(false, true, sizeof(AcpiSMMUv3Dev));
     GArray *rc_its_idmaps = g_array_new(false, true, sizeof(AcpiIortIdMapping));
 
     AcpiTable table = { .sig = "IORT", .rev = 5, .oem_id = vms->oem_id,
@@ -581,7 +581,7 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
         }
         /* Calculate RMR nodes required. One per SMMUv3 with accelerated mode */
         for (i = 0; i < num_smmus; i++) {
-            sdev = &g_array_index(smmuv3_devs, AcpiIortSMMUv3Dev, i);
+            sdev = &g_array_index(smmuv3_devs, AcpiSMMUv3Dev, i);
             if (sdev->ats) {
                 ats_needed = true;
             }
@@ -620,7 +620,7 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
     }
 
     for (i = 0; i < num_smmus; i++) {
-        sdev = &g_array_index(smmuv3_devs, AcpiIortSMMUv3Dev, i);
+        sdev = &g_array_index(smmuv3_devs, AcpiSMMUv3Dev, i);
         int smmu_mapping_count, offset_to_id_array;
         int irq = sdev->irq;
 
@@ -699,7 +699,7 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
         AcpiIortIdMapping *range;
 
         for (i = 0; i < num_smmus; i++) {
-            sdev = &g_array_index(smmuv3_devs, AcpiIortSMMUv3Dev, i);
+            sdev = &g_array_index(smmuv3_devs, AcpiSMMUv3Dev, i);
 
             /*
              * Map RIDs (input) from RC to SMMUv3 nodes: RC -> SMMUv3.
@@ -742,7 +742,7 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
     acpi_table_end(linker, &table);
     g_array_free(rc_its_idmaps, true);
     for (i = 0; i < num_smmus; i++) {
-        sdev = &g_array_index(smmuv3_devs, AcpiIortSMMUv3Dev, i);
+        sdev = &g_array_index(smmuv3_devs, AcpiSMMUv3Dev, i);
         g_array_free(sdev->rc_smmu_idmaps, true);
     }
     g_array_free(smmuv3_devs, true);
Signature · 1 line
2.43.0
Reviewed-by on v1 14/16
#51 lore raw

Re: [RFC PATCH 14/16] virt-acpi-build: Rename AcpiIortSMMUv3Dev to AcpiSMMUv3Dev

Quoted 7 lines · Shameer Kolothum

Rename struct AcpiIortSMMUv3Dev to AcpiSMMUv3Dev so that it is not specific to IORT. Subsequent Tegra241 CMDQV support patch will use the same struct to build CMDQV DSDT support as well.

No functional changes intended.

#52 lore raw

Re: [RFC PATCH 14/16] virt-acpi-build: Rename AcpiIortSMMUv3Dev to AcpiSMMUv3Dev

In reply to Shameer Kolothum
Quoted 16 lines · Shameer Kolothum

Rename struct AcpiIortSMMUv3Dev to AcpiSMMUv3Dev so that it is not specific to IORT. Subsequent Tegra241 CMDQV support patch will use the same struct to build CMDQV DSDT support as well.

No functional changes intended.

---
 hw/arm/virt-acpi-build.c | 36 ++++++++++++++++++------------------
 1 file changed, 18 insertions(+), 18 deletions(-)
hw/arm/virt-acpi-build.c +0 -0
index 1e3779991e..4f8d36dae0 100644

hw/arm/virt-acpi-build.c static int iort_idmap_compare(gconstpointer a, gconstpointer b) · line 345

Quoted 8 lines
hw/arm/virt-acpi-build.c +1 -1
@@ -339,7 +339,7 @@ static int iort_idmap_compare(gconstpointer a, gconstpointer b)
     return idmap_a->input_base - idmap_b->input_base;
 }
 
-typedef struct AcpiIortSMMUv3Dev {
+typedef struct AcpiSMMUv3Dev {
     int irq;
     hwaddr base;
     GArray *rc_smmu_idmaps;

is the concept of idmap also used in DSDT?

hw/arm/virt-acpi-build.c typedef struct AcpiIortSMMUv3Dev { · line 347

Quoted 2 lines
hw/arm/virt-acpi-build.c +0 -0
@@ -347,16 +347,16 @@ typedef struct AcpiIortSMMUv3Dev {
     size_t offset;

also above offset there is
/* Offset of the SMMUv3 IORT Node relative to the start of the IORT */

Eric

Quoted 130 lines
hw/arm/virt-acpi-build.c +17 -17
     bool accel;
     bool ats;
-} AcpiIortSMMUv3Dev;
+} AcpiSMMUv3Dev;
 
 /*
- * Populate the struct AcpiIortSMMUv3Dev for the legacy SMMUv3 and
+ * Populate the struct AcpiSMMUv3Dev for the legacy SMMUv3 and
  * return the total number of associated idmaps.
  */
 static int populate_smmuv3_legacy_dev(GArray *sdev_blob)
 {
     VirtMachineState *vms = VIRT_MACHINE(qdev_get_machine());
-    AcpiIortSMMUv3Dev sdev;
+    AcpiSMMUv3Dev sdev;
 
     sdev.rc_smmu_idmaps = g_array_new(false, true, sizeof(AcpiIortIdMapping));
     object_child_foreach_recursive(object_get_root(), iort_host_bridges,
@@ -376,8 +376,8 @@ static int populate_smmuv3_legacy_dev(GArray *sdev_blob)
 
 static int smmuv3_dev_idmap_compare(gconstpointer a, gconstpointer b)
 {
-    AcpiIortSMMUv3Dev *sdev_a = (AcpiIortSMMUv3Dev *)a;
-    AcpiIortSMMUv3Dev *sdev_b = (AcpiIortSMMUv3Dev *)b;
+    AcpiSMMUv3Dev *sdev_a = (AcpiSMMUv3Dev *)a;
+    AcpiSMMUv3Dev *sdev_b = (AcpiSMMUv3Dev *)b;
     AcpiIortIdMapping *map_a = &g_array_index(sdev_a->rc_smmu_idmaps,
                                               AcpiIortIdMapping, 0);
     AcpiIortIdMapping *map_b = &g_array_index(sdev_b->rc_smmu_idmaps,
@@ -391,7 +391,7 @@ static int iort_smmuv3_devices(Object *obj, void *opaque)
     GArray *sdev_blob = opaque;
     AcpiIortIdMapping idmap;
     PlatformBusDevice *pbus;
-    AcpiIortSMMUv3Dev sdev;
+    AcpiSMMUv3Dev sdev;
     int min_bus, max_bus;
     SysBusDevice *sbdev;
     PCIBus *bus;
@@ -421,7 +421,7 @@ static int iort_smmuv3_devices(Object *obj, void *opaque)
 }
 
 /*
- * Populate the struct AcpiIortSMMUv3Dev for all SMMUv3 devices and
+ * Populate the struct AcpiSMMUv3Dev for all SMMUv3 devices and
  * return the total number of idmaps.
  */
 static int populate_smmuv3_dev(GArray *sdev_blob)
@@ -442,10 +442,10 @@ static void create_rc_its_idmaps(GArray *its_idmaps, GArray *smmuv3_devs)
 {
     AcpiIortIdMapping *idmap;
     AcpiIortIdMapping next_range = {0};
-    AcpiIortSMMUv3Dev *sdev;
+    AcpiSMMUv3Dev *sdev;
 
     for (int i = 0; i < smmuv3_devs->len; i++) {
-        sdev = &g_array_index(smmuv3_devs, AcpiIortSMMUv3Dev, i);
+        sdev = &g_array_index(smmuv3_devs, AcpiSMMUv3Dev, i);
         /*
          * Based on the RID ranges that are directed to the SMMU, determine the
          * bypassed RID ranges, i.e., the ones that are directed to the ITS
@@ -479,7 +479,7 @@ static void create_rc_its_idmaps(GArray *its_idmaps, GArray *smmuv3_devs)
 static void
 build_iort_rmr_nodes(GArray *table_data, GArray *smmuv3_devices, uint32_t *id)
 {
-    AcpiIortSMMUv3Dev *sdev;
+    AcpiSMMUv3Dev *sdev;
     AcpiIortIdMapping *idmap;
     int i;
 
@@ -487,7 +487,7 @@ build_iort_rmr_nodes(GArray *table_data, GArray *smmuv3_devices, uint32_t *id)
         uint16_t rmr_len;
         int bdf;
 
-        sdev = &g_array_index(smmuv3_devices, AcpiIortSMMUv3Dev, i);
+        sdev = &g_array_index(smmuv3_devices, AcpiSMMUv3Dev, i);
         if (!sdev->accel) {
             continue;
         }
@@ -544,13 +544,13 @@ static void
 build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
 {
     int i, nb_nodes, rc_mapping_count;
-    AcpiIortSMMUv3Dev *sdev;
+    AcpiSMMUv3Dev *sdev;
     size_t node_size;
     bool ats_needed = false;
     int num_smmus = 0;
     uint32_t id = 0;
     int rc_smmu_idmaps_len = 0;
-    GArray *smmuv3_devs = g_array_new(false, true, sizeof(AcpiIortSMMUv3Dev));
+    GArray *smmuv3_devs = g_array_new(false, true, sizeof(AcpiSMMUv3Dev));
     GArray *rc_its_idmaps = g_array_new(false, true, sizeof(AcpiIortIdMapping));
 
     AcpiTable table = { .sig = "IORT", .rev = 5, .oem_id = vms->oem_id,
@@ -581,7 +581,7 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
         }
         /* Calculate RMR nodes required. One per SMMUv3 with accelerated mode */
         for (i = 0; i < num_smmus; i++) {
-            sdev = &g_array_index(smmuv3_devs, AcpiIortSMMUv3Dev, i);
+            sdev = &g_array_index(smmuv3_devs, AcpiSMMUv3Dev, i);
             if (sdev->ats) {
                 ats_needed = true;
             }
@@ -620,7 +620,7 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
     }
 
     for (i = 0; i < num_smmus; i++) {
-        sdev = &g_array_index(smmuv3_devs, AcpiIortSMMUv3Dev, i);
+        sdev = &g_array_index(smmuv3_devs, AcpiSMMUv3Dev, i);
         int smmu_mapping_count, offset_to_id_array;
         int irq = sdev->irq;
 
@@ -699,7 +699,7 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
         AcpiIortIdMapping *range;
 
         for (i = 0; i < num_smmus; i++) {
-            sdev = &g_array_index(smmuv3_devs, AcpiIortSMMUv3Dev, i);
+            sdev = &g_array_index(smmuv3_devs, AcpiSMMUv3Dev, i);
 
             /*
              * Map RIDs (input) from RC to SMMUv3 nodes: RC -> SMMUv3.
@@ -742,7 +742,7 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
     acpi_table_end(linker, &table);
     g_array_free(rc_its_idmaps, true);
     for (i = 0; i < num_smmus; i++) {
-        sdev = &g_array_index(smmuv3_devs, AcpiIortSMMUv3Dev, i);
+        sdev = &g_array_index(smmuv3_devs, AcpiSMMUv3Dev, i);
         g_array_free(sdev->rc_smmu_idmaps, true);
     }
     g_array_free(smmuv3_devs, true);
#53 lore raw

RFC PATCH 15/16 hw/arm/virt-acpi: Advertise Tegra241 CMDQV nodes in DSDT

In reply to Shameer Kolothum

From: Nicolin Chen <[email protected]>

Add ACPI DSDT support for Tegra241 CMDQV when the SMMUv3 instance is created with tegra241-cmdqv=on.

---
 hw/arm/trace-events      |  1 +
 hw/arm/virt-acpi-build.c | 74 ++++++++++++++++++++++++++++++++++++++++
 include/hw/arm/virt.h    |  2 ++
 3 files changed, 77 insertions(+)
hw/arm/trace-events +1 -0
index ef495c040c..e7e3ccfe9f 100644
@@ -9,6 +9,7 @@ omap1_lpg_led(const char *onoff) "omap1 LPG: LED is %s"
 
 # virt-acpi-build.c
 virt_acpi_setup(void) "No fw cfg or ACPI disabled. Bailing out."
+virt_acpi_dsdt_tegra241_cmdqv(int smmu_id, uint64_t base, uint32_t irq) "DSDT: add cmdqv node for (id=%d), base=0x%" PRIx64 ", irq=%d"
 
 # smmu-common.c
 smmu_add_mr(const char *name) "%s"
hw/arm/virt-acpi-build.c +74 -0

86 lines not on this page · raw message

include/hw/arm/virt.h +2 -0
index efbc1758c5..842143cc85 100644
@@ -46,6 +46,8 @@
 #define NUM_VIRTIO_TRANSPORTS 32
 #define NUM_SMMU_IRQS          4
 
+#define TEGRA241_CMDQV_IO_LEN 0x50000
+
 /* See Linux kernel arch/arm64/include/asm/pvclock-abi.h */
 #define PVTIME_SIZE_PER_CPU 64
 
Signature · 1 line
2.43.0
#54 lore raw

Re: [RFC PATCH 15/16] hw/arm/virt-acpi: Advertise Tegra241 CMDQV nodes in DSDT

Quoted 15 lines · Shameer Kolothum
hw/arm/virt-acpi-build.c +15 -0
+static int smmuv3_cmdqv_devices(Object *obj, void *opaque)
+{
+    VirtMachineState *vms = VIRT_MACHINE(qdev_get_machine());
+    GArray *sdev_blob = opaque;
+    PlatformBusDevice *pbus;
+    AcpiSMMUv3Dev sdev;
+    SysBusDevice *sbdev;
+
+    if (!object_dynamic_cast(obj, TYPE_ARM_SMMUV3)) {
+        return 0;
+    }
+
+    if (!object_property_get_bool(obj, "tegra241-cmdqv", NULL)) {
+        return 0;
+    }

hw/arm/virt-acpi-build.c static void build_fadt_rev6(GArray *table_data, BIOSLinker *linker, · line 1142

Quoted 10 lines
hw/arm/virt-acpi-build.c +10 -0
+
+    pbus = PLATFORM_BUS_DEVICE(vms->platform_bus_dev);
+    sbdev = SYS_BUS_DEVICE(obj);
+    sdev.base = platform_bus_get_mmio_addr(pbus, sbdev, 1);
+    sdev.base += vms->memmap[VIRT_PLATFORM_BUS].base;
+    sdev.irq = platform_bus_get_irqn(pbus, sbdev, NUM_SMMU_IRQS);
+    sdev.irq += vms->irqmap[VIRT_PLATFORM_BUS];
+    sdev.irq += ARM_SPI_BASE;
+    g_array_append_val(sdev_blob, sdev);
+    return 0;

This is pre-building SMMU's IORT nodes right? Maybe a different naming? And can be shared with the existing iort_smmuv3_devices?

We do so, because we need to link CMDQV's DSDT node to the SMMU's IORT node but it is created in build_iort() that is called after build_dsdt().

Let's explain why we have this pre-building.

Nicolin

#55 lore raw

RE: [RFC PATCH 15/16] hw/arm/virt-acpi: Advertise Tegra241 CMDQV nodes in DSDT

Quoted 36 lines

-----Original Message-----
From: Nicolin Chen <[email protected]>
Sent: 29 December 2025 20:11
To: Shameer Kolothum <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; [email protected]; [email protected]; Krishnakant Jaju <[email protected]>
Subject: Re: [RFC PATCH 15/16] hw/arm/virt-acpi: Advertise Tegra241 CMDQV nodes in DSDT

Quoted 20 lines · Shameer Kolothum
hw/arm/virt-acpi-build.c +20 -0
+static int smmuv3_cmdqv_devices(Object *obj, void *opaque)
+{
+    VirtMachineState *vms = VIRT_MACHINE(qdev_get_machine());
+    GArray *sdev_blob = opaque;
+    PlatformBusDevice *pbus;
+    AcpiSMMUv3Dev sdev;
+    SysBusDevice *sbdev;
+
+    if (!object_dynamic_cast(obj, TYPE_ARM_SMMUV3)) {
+        return 0;
+    }
+
+    if (!object_property_get_bool(obj, "tegra241-cmdqv", NULL)) {
+        return 0;
+    }
+
+    pbus = PLATFORM_BUS_DEVICE(vms->platform_bus_dev);
+    sbdev = SYS_BUS_DEVICE(obj);
+    sdev.base = platform_bus_get_mmio_addr(pbus, sbdev, 1);
+    sdev.base += vms->memmap[VIRT_PLATFORM_BUS].base;
Quoted 8 lines

hw/arm/virt-acpi-build.c static void build_fadt_rev6(GArray *table_data, BIOSLinker *linker, · line 1142

Quoted 5 lines
hw/arm/virt-acpi-build.c +5 -0
+    sdev.irq = platform_bus_get_irqn(pbus, sbdev, NUM_SMMU_IRQS);
+    sdev.irq += vms->irqmap[VIRT_PLATFORM_BUS];
+    sdev.irq += ARM_SPI_BASE;
+    g_array_append_val(sdev_blob, sdev);
+    return 0;

This is pre-building SMMU's IORT nodes right? Maybe a different naming? And can be shared with the existing iort_smmuv3_devices?

Not really, if you are referring about this patch here,
https://github.com/NVIDIA/QEMU/commit/cc3b65e6a49a9b7addf44b377d4ef1de99bfee3f

I didn't find a clean way to store the pre-built smmuv3_devs other than placing them in struct AcpiBuildTables. It's not clear we gain much by restructuring things to populate and store them separately. At best, it might slightly improve boot time, and if that becomes important we can always add it as an optimization later.

This patch enumerates SMMUv3 accel instances with CMDQV separately, uses that information to build the DSDT, and then frees the device array.

Quoted 4 lines

We do so, because we need to link CMDQV's DSDT node to the SMMU's IORT node but it is created in build_iort() that is called after build_dsdt().

Hmm..not sure I get that. Does this mean IORT has a link to DSDT? I can't find one.. Maybe I missed. Please let me know.

Thanks,
Shameer

#56 lore raw

Re: [RFC PATCH 15/16] hw/arm/virt-acpi: Advertise Tegra241 CMDQV nodes in DSDT

Quoted 45 lines · Shameer Kolothum
Quoted 36 lines

-----Original Message-----
From: Nicolin Chen <[email protected]>
Sent: 29 December 2025 20:11
To: Shameer Kolothum <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; [email protected]; [email protected]; Krishnakant Jaju <[email protected]>
Subject: Re: [RFC PATCH 15/16] hw/arm/virt-acpi: Advertise Tegra241 CMDQV nodes in DSDT

Quoted 20 lines · Shameer Kolothum
hw/arm/virt-acpi-build.c +20 -0
+static int smmuv3_cmdqv_devices(Object *obj, void *opaque)
+{
+    VirtMachineState *vms = VIRT_MACHINE(qdev_get_machine());
+    GArray *sdev_blob = opaque;
+    PlatformBusDevice *pbus;
+    AcpiSMMUv3Dev sdev;
+    SysBusDevice *sbdev;
+
+    if (!object_dynamic_cast(obj, TYPE_ARM_SMMUV3)) {
+        return 0;
+    }
+
+    if (!object_property_get_bool(obj, "tegra241-cmdqv", NULL)) {
+        return 0;
+    }
+
+    pbus = PLATFORM_BUS_DEVICE(vms->platform_bus_dev);
+    sbdev = SYS_BUS_DEVICE(obj);
+    sdev.base = platform_bus_get_mmio_addr(pbus, sbdev, 1);
+    sdev.base += vms->memmap[VIRT_PLATFORM_BUS].base;
Quoted 8 lines

hw/arm/virt-acpi-build.c static void build_fadt_rev6(GArray *table_data, BIOSLinker *linker, · line 1142

Quoted 5 lines
hw/arm/virt-acpi-build.c +5 -0
+    sdev.irq = platform_bus_get_irqn(pbus, sbdev, NUM_SMMU_IRQS);
+    sdev.irq += vms->irqmap[VIRT_PLATFORM_BUS];
+    sdev.irq += ARM_SPI_BASE;
+    g_array_append_val(sdev_blob, sdev);
+    return 0;

This is pre-building SMMU's IORT nodes right? Maybe a different naming? And can be shared with the existing iort_smmuv3_devices?

Quoted 2 lines

Not really, if you are referring about this patch here,
https://github.com/NVIDIA/QEMU/commit/cc3b65e6a49a9b7addf44b377d4ef1de99bfee3f

Yes. I am talking about this. But you are actually pre-building an ACPI SMMU device array here. So, my question wasn't accurate..

Quoted 8 lines

I didn’t find a clean way to store the pre-built smmuv3_devs other than placing them in struct AcpiBuildTables. It’s not clear we gain much by restructuring things to populate and store them separately. At best, it might slightly improve boot time, and if that becomes important we can always add it as an optimization later.

This patch enumerates SMMUv3 accel instances with CMDQV separately, uses that information to build the DSDT, and then frees the device array.

Oh, I missed the g_array_free() in acpi_dsdt_add_tegra241_cmdqv().

Let's have a note inline and copy to the commit message too.

Quoted 6 lines
Quoted 3 lines

We do so, because we need to link CMDQV's DSDT node to the SMMU's IORT node but it is created in build_iort() that is called after build_dsdt().

Hmm..not sure I get that. Does this mean IORT has a link to DSDT? I can't find one.. Maybe I missed. Please let me know.

The _UID field of the DSDT node links to the Identifier field of an SMMU's IORT node. E.g.

[DSDT] // Qemu builds DSDT first

CMDQV0 {
    _UID = 0;
}

[IORT] // Then builds IORT

SMMU0 {
    Identifier = 0;
}

If we are sure that the SMMU sequence is fixed every time we rebuild the sdev array so that IORT.Identifier matches with DSDT._UID, we'd be fine. That being said, we should probably make a node inline too.

Otherwise, it would be safer to pre-build the sdev array and use the same array in both DSDT and IORT.

Thanks
Nicolin

#57 lore raw

Re: [RFC PATCH 15/16] hw/arm/virt-acpi: Advertise Tegra241 CMDQV nodes in DSDT

Quoted 80 lines · Nicolin Chen
Quoted 45 lines · Shameer Kolothum
Quoted 36 lines

-----Original Message-----
From: Nicolin Chen <[email protected]>
Sent: 29 December 2025 20:11
To: Shameer Kolothum <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; [email protected]; [email protected]; Krishnakant Jaju <[email protected]>
Subject: Re: [RFC PATCH 15/16] hw/arm/virt-acpi: Advertise Tegra241 CMDQV nodes in DSDT

Quoted 20 lines · Shameer Kolothum
hw/arm/virt-acpi-build.c +20 -0
+static int smmuv3_cmdqv_devices(Object *obj, void *opaque)
+{
+    VirtMachineState *vms = VIRT_MACHINE(qdev_get_machine());
+    GArray *sdev_blob = opaque;
+    PlatformBusDevice *pbus;
+    AcpiSMMUv3Dev sdev;
+    SysBusDevice *sbdev;
+
+    if (!object_dynamic_cast(obj, TYPE_ARM_SMMUV3)) {
+        return 0;
+    }
+
+    if (!object_property_get_bool(obj, "tegra241-cmdqv", NULL)) {
+        return 0;
+    }
+
+    pbus = PLATFORM_BUS_DEVICE(vms->platform_bus_dev);
+    sbdev = SYS_BUS_DEVICE(obj);
+    sdev.base = platform_bus_get_mmio_addr(pbus, sbdev, 1);
+    sdev.base += vms->memmap[VIRT_PLATFORM_BUS].base;
Quoted 8 lines

hw/arm/virt-acpi-build.c static void build_fadt_rev6(GArray *table_data, BIOSLinker *linker, · line 1142

Quoted 5 lines
hw/arm/virt-acpi-build.c +5 -0
+    sdev.irq = platform_bus_get_irqn(pbus, sbdev, NUM_SMMU_IRQS);
+    sdev.irq += vms->irqmap[VIRT_PLATFORM_BUS];
+    sdev.irq += ARM_SPI_BASE;
+    g_array_append_val(sdev_blob, sdev);
+    return 0;

This is pre-building SMMU's IORT nodes right? Maybe a different naming? And can be shared with the existing iort_smmuv3_devices?

Quoted 2 lines

Not really, if you are referring about this patch here,
https://github.com/NVIDIA/QEMU/commit/cc3b65e6a49a9b7addf44b377d4ef1de99bfee3f

Yes. I am talking about this. But you are actually pre-building an ACPI SMMU device array here. So, my question wasn't accurate..

Quoted 8 lines

I didn’t find a clean way to store the pre-built smmuv3_devs other than placing them in struct AcpiBuildTables. It’s not clear we gain much by restructuring things to populate and store them separately. At best, it might slightly improve boot time, and if that becomes important we can always add it as an optimization later.

This patch enumerates SMMUv3 accel instances with CMDQV separately, uses that information to build the DSDT, and then frees the device array.

Oh, I missed the g_array_free() in acpi_dsdt_add_tegra241_cmdqv().

Let's have a note inline and copy to the commit message too.

Quoted 6 lines
Quoted 3 lines

We do so, because we need to link CMDQV's DSDT node to the SMMU's IORT node but it is created in build_iort() that is called after build_dsdt().

Hmm..not sure I get that. Does this mean IORT has a link to DSDT? I can't find one.. Maybe I missed. Please let me know.

The _UID field of the DSDT node links to the Identifier field of an SMMU's IORT node. E.g.

[DSDT] // Qemu builds DSDT first

CMDQV0 {
    _UID = 0;
}
Quoted 8 lines

[IORT] // Then builds IORT

SMMU0 {
    Identifier = 0;
}

If we are sure that the SMMU sequence is fixed every time we rebuild the sdev array so that IORT.Identifier matches with DSDT._UID, we'd be fine. That being said, we should probably make a node inline too.

Typo: node->note

Quoted 5 lines

Otherwise, it would be safer to pre-build the sdev array and use the same array in both DSDT and IORT.

Thanks
Nicolin

#58 lore raw

RE: [RFC PATCH 15/16] hw/arm/virt-acpi: Advertise Tegra241 CMDQV nodes in DSDT

Quoted 81 lines

-----Original Message-----
From: Nicolin Chen <[email protected]>
Sent: 30 December 2025 18:24
To: Shameer Kolothum <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; [email protected]; [email protected]; Krishnakant Jaju <[email protected]>
Subject: Re: [RFC PATCH 15/16] hw/arm/virt-acpi: Advertise Tegra241 CMDQV nodes in DSDT

Quoted 45 lines · Nicolin Chen
Quoted 44 lines · Shameer Kolothum
Quoted 43 lines

-----Original Message-----
From: Nicolin Chen <[email protected]>
Sent: 29 December 2025 20:11
To: Shameer Kolothum <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; [email protected]; [email protected]; Krishnakant Jaju <[email protected]>
Subject: Re: [RFC PATCH 15/16] hw/arm/virt-acpi: Advertise Tegra241 CMDQV nodes in DSDT

Quoted 14 lines · Shameer Kolothum
hw/arm/virt-acpi-build.c +14 -0
+static int smmuv3_cmdqv_devices(Object *obj, void *opaque) {
+    VirtMachineState *vms = VIRT_MACHINE(qdev_get_machine());
+    GArray *sdev_blob = opaque;
+    PlatformBusDevice *pbus;
+    AcpiSMMUv3Dev sdev;
+    SysBusDevice *sbdev;
+
+    if (!object_dynamic_cast(obj, TYPE_ARM_SMMUV3)) {
+        return 0;
+    }
+
+    if (!object_property_get_bool(obj, "tegra241-cmdqv", NULL)) {
+        return 0;
+    }

hw/arm/virt-acpi-build.c static void build_fadt_rev6(GArray *table_data, BIOSLinker *linker, · line 1142

Quoted 10 lines
hw/arm/virt-acpi-build.c +10 -0
+
+    pbus = PLATFORM_BUS_DEVICE(vms->platform_bus_dev);
+    sbdev = SYS_BUS_DEVICE(obj);
+    sdev.base = platform_bus_get_mmio_addr(pbus, sbdev, 1);
+    sdev.base += vms->memmap[VIRT_PLATFORM_BUS].base;
+    sdev.irq = platform_bus_get_irqn(pbus, sbdev, NUM_SMMU_IRQS);
+    sdev.irq += vms->irqmap[VIRT_PLATFORM_BUS];
+    sdev.irq += ARM_SPI_BASE;
+    g_array_append_val(sdev_blob, sdev);
+    return 0;

This is pre-building SMMU's IORT nodes right? Maybe a different naming? And can be shared with the existing iort_smmuv3_devices?

Quoted 2 lines
Quoted 2 lines

Not really, if you are referring about this patch here,

https://github.com/NVIDIA/QEMU/commit/cc3b65e6a49a9b7addf44b377d
4ef1

Quoted 16 lines
Quoted 1 line

de99bfee3f

Yes. I am talking about this. But you are actually pre-building an ACPI SMMU device array here. So, my question wasn't accurate..

Quoted 8 lines

I didn't find a clean way to store the pre-built smmuv3_devs other than placing them in struct AcpiBuildTables. It's not clear we gain much by restructuring things to populate and store them separately. At best, it might slightly improve boot time, and if that becomes important we can always add it as an optimization later.

This patch enumerates SMMUv3 accel instances with CMDQV separately, uses that information to build the DSDT, and then frees the device array.

Oh, I missed the g_array_free() in acpi_dsdt_add_tegra241_cmdqv().

Quoted 26 lines
Quoted 7 lines

Let's have a note inline and copy to the commit message too.

Quoted 5 lines
Quoted 3 lines

We do so, because we need to link CMDQV's DSDT node to the SMMU's IORT node but it is created in build_iort() that is called after build_dsdt().

Hmm..not sure I get that. Does this mean IORT has a link to DSDT? I can't

find one..

Quoted 18 lines
Quoted 1 line

Maybe I missed. Please let me know.

The _UID field of the DSDT node links to the Identifier field of an SMMU's IORT node. E.g.

[DSDT] // Qemu builds DSDT first

CMDQV0 {
    _UID = 0;
}

[IORT] // Then builds IORT

SMMU0 {
    Identifier = 0;
}

If we are sure that the SMMU sequence is fixed every time we rebuild the sdev array so that IORT.Identifier matches with DSDT._UID, we'd be fine. That being said, we should probably make a node inline too.

I see that the IORT node Identifier is used in the kernel SMMUv3 driver acpi_smmu_dsdt_probe_tegra241_cmdqv() to retrieve the corresponding ACPI device. This appears to be the relationship you were referring to.

However, I don't think this holds for the current series as is. It will likely fail if we have SMMUv3 accel devices both with and without tegra241-cmdqv enabled, since the "identifier" will not match in that case. I will fix that.

Regarding QEMU keeping the same sequence across both iterations, I couldn't find any guarantee that object_child_foreach_recursive() does that. Need to check.

Thanks,
Shameer

#59 lore raw

Re: [RFC PATCH 15/16] hw/arm/virt-acpi: Advertise Tegra241 CMDQV nodes in DSDT

In reply to Shameer Kolothum
Quoted 4 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

Add ACPI DSDT support for Tegra241 CMDQV when the SMMUv3 instance is created with tegra241-cmdqv=on.

Pls can you give a point to the DSDT spec to understand why this is needed and what needs to be added.

Quoted 118 lines
---
 hw/arm/trace-events      |  1 +
 hw/arm/virt-acpi-build.c | 74 ++++++++++++++++++++++++++++++++++++++++
 include/hw/arm/virt.h    |  2 ++
 3 files changed, 77 insertions(+)
hw/arm/trace-events +1 -0
index ef495c040c..e7e3ccfe9f 100644
@@ -9,6 +9,7 @@ omap1_lpg_led(const char *onoff) "omap1 LPG: LED is %s"
 
 # virt-acpi-build.c
 virt_acpi_setup(void) "No fw cfg or ACPI disabled. Bailing out."
+virt_acpi_dsdt_tegra241_cmdqv(int smmu_id, uint64_t base, uint32_t irq) "DSDT: add cmdqv node for (id=%d), base=0x%" PRIx64 ", irq=%d"
 
 # smmu-common.c
 smmu_add_mr(const char *name) "%s"
hw/arm/virt-acpi-build.c +74 -0
index 4f8d36dae0..11494b29ad 100644
@@ -1115,6 +1115,78 @@ static void build_fadt_rev6(GArray *table_data, BIOSLinker *linker,
     build_fadt(table_data, linker, &fadt, vms->oem_id, vms->oem_table_id);
 }
 
+static int smmuv3_cmdqv_devices(Object *obj, void *opaque)
+{
+    VirtMachineState *vms = VIRT_MACHINE(qdev_get_machine());
+    GArray *sdev_blob = opaque;
+    PlatformBusDevice *pbus;
+    AcpiSMMUv3Dev sdev;
+    SysBusDevice *sbdev;
+
+    if (!object_dynamic_cast(obj, TYPE_ARM_SMMUV3)) {
+        return 0;
+    }
+
+    if (!object_property_get_bool(obj, "tegra241-cmdqv", NULL)) {
+        return 0;
+    }
+
+    pbus = PLATFORM_BUS_DEVICE(vms->platform_bus_dev);
+    sbdev = SYS_BUS_DEVICE(obj);
+    sdev.base = platform_bus_get_mmio_addr(pbus, sbdev, 1);
+    sdev.base += vms->memmap[VIRT_PLATFORM_BUS].base;
+    sdev.irq = platform_bus_get_irqn(pbus, sbdev, NUM_SMMU_IRQS);
+    sdev.irq += vms->irqmap[VIRT_PLATFORM_BUS];
+    sdev.irq += ARM_SPI_BASE;
+    g_array_append_val(sdev_blob, sdev);
+    return 0;
+}
+
+static void acpi_dsdt_add_tegra241_cmdqv(Aml *scope, VirtMachineState *vms)
+{
+    GArray *smmuv3_devs = g_array_new(false, true, sizeof(AcpiSMMUv3Dev));
+    int i;
+
+    if (vms->legacy_smmuv3_present) {
+        return;
+    }
+
+    object_child_foreach_recursive(object_get_root(), smmuv3_cmdqv_devices,
+                                   smmuv3_devs);
+
+    for (i = 0; i < smmuv3_devs->len; i++) {
+        uint32_t identifier = i;
+        AcpiSMMUv3Dev *sdev;
+        Aml *dev, *crs, *addr;
+
+        sdev = &g_array_index(smmuv3_devs, AcpiSMMUv3Dev, i);
+
+        dev = aml_device("CV%.02u", identifier);
+        aml_append(dev, aml_name_decl("_HID", aml_string("NVDA200C")));
+        if (vms->its) {
+            identifier++;
+        }
+        aml_append(dev, aml_name_decl("_UID", aml_int(identifier)));
+        aml_append(dev, aml_name_decl("_CCA", aml_int(1)));
+
+        crs = aml_resource_template();
+        addr = aml_qword_memory(AML_POS_DECODE, AML_MIN_FIXED, AML_MAX_FIXED,
+                                AML_CACHEABLE, AML_READ_WRITE, 0x0, sdev->base,
+                                sdev->base + TEGRA241_CMDQV_IO_LEN - 0x1, 0x0,
+                                TEGRA241_CMDQV_IO_LEN);
+        aml_append(crs, addr);
+        aml_append(crs, aml_interrupt(AML_CONSUMER, AML_EDGE,
+                                      AML_ACTIVE_HIGH, AML_EXCLUSIVE,
+                                      (uint32_t *)&sdev->irq, 1));
+        aml_append(dev, aml_name_decl("_CRS", crs));
+
+        aml_append(scope, dev);
+
+        trace_virt_acpi_dsdt_tegra241_cmdqv(identifier, sdev->base, sdev->irq);
+    }
+    g_array_free(smmuv3_devs, true);
+}
+
 /* DSDT */
 static void
 build_dsdt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
@@ -1179,6 +1251,8 @@ build_dsdt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
     acpi_dsdt_add_tpm(scope, vms);
 #endif
 
+    acpi_dsdt_add_tegra241_cmdqv(scope, vms);
+
     aml_append(dsdt, scope);
 
     pci0_scope = aml_scope("\\_SB.PCI0");
include/hw/arm/virt.h +0 -0
index efbc1758c5..842143cc85 100644

include/hw/arm/virt.h line 49

Quoted 4 lines
include/hw/arm/virt.h +1 -0
@@ -46,6 +46,8 @@
 #define NUM_VIRTIO_TRANSPORTS 32
 #define NUM_SMMU_IRQS          4
 
+#define TEGRA241_CMDQV_IO_LEN 0x50000

isn't it already defined elsewhere, use the corresponding include?

Eric

Quoted 4 lines
include/hw/arm/virt.h +1 -0
+
 /* See Linux kernel arch/arm64/include/asm/pvclock-abi.h */
 #define PVTIME_SIZE_PER_CPU 64
 
#60 lore raw

Re: [RFC PATCH 15/16] hw/arm/virt-acpi: Advertise Tegra241 CMDQV nodes in DSDT

Quoted 8 lines · Eric Auger
Quoted 4 lines · Shameer Kolothum

From: Nicolin Chen <[email protected]>

Add ACPI DSDT support for Tegra241 CMDQV when the SMMUv3 instance is created with tegra241-cmdqv=on.

Pls can you give a point to the DSDT spec to understand why this is needed and what needs to be added.

This was an implementation decision by ARM that CMDQV node resides in DSDT v.s. IORT.

Kernel driver scans DSDT unconditionally:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c#n4572

NVIDIA also has a public DSDT:
https://github.com/NVIDIA/edk2-nvidia/blob/main/Silicon/NVIDIA/Drivers/ConfigurationManagerData/AcpiTableList/PlatformASLTablesLib/Dsdt_TH500.asl#L1005

I don't think there is a DSDT spec though...

Nicolin

#61 lore raw

RFC PATCH 16/16 hw/arm/smmuv3: Add tegra241-cmdqv property for SMMUv3 device

In reply to Shameer Kolothum

Introduce a “tegra241-cmdqv” property to enable Tegra241 CMDQV support. This is only enabled for accelerated SMMUv3 devices.

---
 hw/arm/smmuv3.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)
hw/arm/smmuv3.c +13 -0
index ec8687d39a..58c35c2af3 100644
@@ -1953,6 +1953,12 @@ static bool smmu_validate_property(SMMUv3State *s, Error **errp)
         error_setg(errp, "accel=on support not compiled in");
         return false;
     }
+#endif
+#ifndef CONFIG_TEGRA241_CMDQ
+    if (s->tegra241_cmdqv) {
+        error_setg(errp, "tegra241_cmdqv=on support not compiled in");
+        return false;
+    }
 #endif
     if (!s->accel) {
         if (!s->ril) {
@@ -1971,6 +1977,10 @@ static bool smmu_validate_property(SMMUv3State *s, Error **errp)
             error_setg(errp, "pasid can only be enabled if accel=on");
             return false;
         }
+        if (s->tegra241_cmdqv) {
+            error_setg(errp, "tegra241_cmdqv can only be enabled if accel=on");
+            return false;
+        }
         return true;
     }
 
@@ -2109,6 +2119,7 @@ static const Property smmuv3_properties[] = {
     DEFINE_PROP_BOOL("ats", SMMUv3State, ats, false),
     DEFINE_PROP_UINT8("oas", SMMUv3State, oas, 44),
     DEFINE_PROP_BOOL("pasid", SMMUv3State, pasid, false),
+    DEFINE_PROP_BOOL("tegra241-cmdqv", SMMUv3State, tegra241_cmdqv, false),
 };
 
 static void smmuv3_instance_init(Object *obj)
@@ -2144,6 +2155,8 @@ static void smmuv3_class_init(ObjectClass *klass, const void *data)
         "are 44 or 48 bits. Defaults to 44 bits");
     object_class_property_set_description(klass, "pasid",
         "Enable/disable PASID support (for accel=on)");
+    object_class_property_set_description(klass, "tegra241-cmdqv",
+        "Enable/disable Tegra241 CMDQ-Virtualisation support (for accel=on)");
 }
 
 static int smmuv3_notify_flag_changed(IOMMUMemoryRegion *iommu,
Signature · 1 line
2.43.0
Reviewed-by on v1 16/16
#62 lore raw

Re: [RFC PATCH 16/16] hw/arm/smmuv3: Add tegra241-cmdqv property for SMMUv3 device

Quoted 13 lines · Shameer Kolothum

Introduce a “tegra241-cmdqv” property to enable Tegra241 CMDQV support. This is only enabled for accelerated SMMUv3 devices.

---
 hw/arm/smmuv3.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)
hw/arm/smmuv3.c +0 -0
index ec8687d39a..58c35c2af3 100644

hw/arm/smmuv3.c static bool smmu_validate_property(SMMUv3State *s, Error **errp) · line 1962

Quoted 10 lines
hw/arm/smmuv3.c +6 -0
@@ -1953,6 +1953,12 @@ static bool smmu_validate_property(SMMUv3State *s, Error **errp)
         error_setg(errp, "accel=on support not compiled in");
         return false;
     }
+#endif
+#ifndef CONFIG_TEGRA241_CMDQ
+    if (s->tegra241_cmdqv) {
+        error_setg(errp, "tegra241_cmdqv=on support not compiled in");
+        return false;
+    }
 #endif

Instead of adding this..

hw/arm/smmuv3.c static const Property smmuv3_properties[] = { · line 2122

Quoted 5 lines
hw/arm/smmuv3.c +1 -0
@@ -2109,6 +2119,7 @@ static const Property smmuv3_properties[] = {
     DEFINE_PROP_BOOL("ats", SMMUv3State, ats, false),
     DEFINE_PROP_UINT8("oas", SMMUv3State, oas, 44),
     DEFINE_PROP_BOOL("pasid", SMMUv3State, pasid, false),
+    DEFINE_PROP_BOOL("tegra241-cmdqv", SMMUv3State, tegra241_cmdqv, false),

Could this be

+#if CONFIG_TEGRA241_CMDQ
+ DEFINE_PROP_BOOL("tegra241-cmdqv", SMMUv3State, tegra241_cmdqv, false),
+endif

?

hw/arm/smmuv3.c static void smmuv3_class_init(ObjectClass *klass, const void *data) · line 2159

Quoted 6 lines
hw/arm/smmuv3.c +2 -0
@@ -2144,6 +2155,8 @@ static void smmuv3_class_init(ObjectClass *klass, const void *data)
         "are 44 or 48 bits. Defaults to 44 bits");
     object_class_property_set_description(klass, "pasid",
         "Enable/disable PASID support (for accel=on)");
+    object_class_property_set_description(klass, "tegra241-cmdqv",
+        "Enable/disable Tegra241 CMDQ-Virtualisation support (for accel=on)");

And here too.

The current approach looks okay to me though. So either way:

#63 lore raw
In reply to Shameer Kolothum

Hi Shameer,

Quoted 66 lines · Shameer Kolothum

Hi,

This RFC series adds initial support for NVIDIA Tegra241 CMDQV (Command Queue Virtualisation), an extension to ARM SMMUv3 that provides hardware accelerated virtual command queues (VCMDQs) for guests. CMDQV allows guests to issue SMMU invalidation commands directly to hardware without VM exits, significantly reducing TLBI overhead.

Thanks to Nicolin for the initial patches and testing on which this RFC is based.

This is based on v6[0] of the SMMUv3 accel series, which is still under review, though nearing convergence. This is sent as an RFC, with the goal of gathering early feedback on the CMDQV design and its integration with the SMMUv3 acceleration path.

Background:

Tegra241 CMDQV extends SMMUv3 by allocating per-VM "virtual interfaces" (VINTFs), each hosting up to 128 VCMDQs.

Each VINTF exposes two 64KB MMIO pages:
- Page0 – guest owned control and status registers (directly mapped into the VM)
- Page1 – queue configuration registers (trapped/emulated by QEMU)

Unlike the standard SMMU CMDQ, a guest owned Tegra241 VCMDQ does not support the full command set. Only a subset, primarily invalidation related commands, is accepted by the CMDQV hardware. For this reason, a distinct CMDQV device must be exposed to the guest, and the guest OS must include a Tegra241 CMDQV aware driver to take advantage of the hardware acceleration.

VCMDQ support is integrated via the IOMMU_HW_QUEUE_ALLOC mechanism, allowing QEMU to attach guest configured VCMDQ buffers to the underlying CMDQV hardware through IOMMUFD. The Linux kernel already supports the full CMDQV virtualisation model via IOMMUFD[0].

Summary of QEMU changes:

- Integrated into the existing SMMUv3 accel path via a "tegra241-cmdqv" property.
- Support for allocating vIOMMU objects of type IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV.
- Mapping and emulation of the CMDQV MMIO register layout.
- VCMDQ/VINTF read/write handling and queue allocation using IOMMUFD APIs.
- Reset and initialisation hooks, including checks for at least one cold-plugged device.
- CMDQV hardware reads guest queue memory using host physical addresses provided through IOMMUFD, which requires that the VCMDQ buffer be physically contiguous not only in guest PA space but also in host PA space. When Tegra241 CMDQV is enabled, QEMU must therefore only expose a CMDQV size that the host can reliably back with contiguous physical memory. Because of this constraint, it is suggested to use huge pages to back the guest RAM.
- ACPI DSDT node generation for CMDQV devices on the virt machine.

These patches have been sanity tested on NVIDIA Grace platforms.

ToDo / revisit:
- Prevent hot-unplug of the last device associated with vIOMMU as this might allow associating a different host SMMU/CMDQV.
- Locking requirements around error event propagation.

Quoted 6 lines

Feedback and testing are very welcome.

Thanks,
Shameer
[0] https://lore.kernel.org/qemu-devel/[email protected]/
[1] https://lore.kernel.org/all/[email protected]/

do you have a branch to share with all the bits?

Thanks

Eric

Quoted 43 lines

Nicolin Chen (12):
backends/iommufd: Update iommufd_backend_get_device_info backends/iommufd: Update iommufd_backend_alloc_viommu to allow user
ptr
backends/iommufd: Introduce iommufd_backend_alloc_hw_queue backends/iommufd: Introduce iommufd_backend_viommu_mmap hw/arm/tegra241-cmdqv: Add initial Tegra241 CMDQ-Virtualisation
support
hw/arm/tegra241-cmdqv: Map VINTF Page0 into guest hw/arm/tegra241-cmdqv: Add read emulation support for registers system/physmem: Add helper to check whether a guest PA maps to RAM hw/arm/tegra241-cmdqv:: Add write emulation for registers hw/arm/tegra241-cmdqv: Add reset handler hw/arm/tegra241-cmdqv: Limit queue size based on backend page size hw/arm/virt-acpi: Advertise Tegra241 CMDQV nodes in DSDT

Shameer Kolothum (4):
hw/arm/tegra241-cmdqv: Allocate vEVENTQ object hw/arm/tegra241-cmdqv: Read and propagate Tegra241 CMDQV errors
virt-acpi-build: Rename AcpiIortSMMUv3Dev to AcpiSMMUv3Dev hw/arm/smmuv3: Add tegra241-cmdqv property for SMMUv3 device

 backends/iommufd.c        |  65 ++++
 backends/trace-events     |   2 +
 hw/arm/Kconfig            |   5 +
 hw/arm/meson.build        |   1 +
 hw/arm/smmuv3-accel.c     |  16 +-
 hw/arm/smmuv3.c           |  18 +
 hw/arm/tegra241-cmdqv.c   | 759 ++++++++++++++++++++++++++++++++++++++
 hw/arm/tegra241-cmdqv.h   | 337 +++++++++++++++++
 hw/arm/trace-events       |   5 +
 hw/arm/virt-acpi-build.c  | 110 +++++-
 hw/vfio/iommufd.c         |   6 +-
 include/exec/cpu-common.h |   2 +
 include/hw/arm/smmuv3.h   |   3 +
 include/hw/arm/virt.h     |   2 +
 include/system/iommufd.h  |  16 +
 system/physmem.c          |  12 +
 16 files changed, 1332 insertions(+), 27 deletions(-)
 create mode 100644 hw/arm/tegra241-cmdqv.c
 create mode 100644 hw/arm/tegra241-cmdqv.h
#64 lore raw

Hi Eric,

Quoted 14 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 11 December 2025 17:55
To: Shameer Kolothum <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; [email protected]; [email protected]; Krishnakant Jaju <[email protected]>
Subject: Re: [RFC PATCH 00/16] hw/arm: Introduce Tegra241 CMDQV support for accelerated SMMUv3

External email: Use caution opening links or attachments

[...]

Quoted 6 lines
Quoted 2 lines

[0] https://lore.kernel.org/qemu-devel/20251120132213.56581-1-

[email protected]/

Quoted 1 line

[1] https://lore.kernel.org/all/[email protected]/

do you have a branch to share with all the bits?

Here:
https://github.com/shamiali2008/qemu-master.git master-smmuv3-accel-v6-veventq-v2-vcmdq-rfcv1

Thanks,
Shameer

author Applied
#65 lore raw
Quoted 43 lines

-----Original Message-----
From: Shameer Kolothum <[email protected]>
Sent: 12 December 2025 00:23
To: [email protected]; [email protected]; qemu- [email protected]
Cc: [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; [email protected]; [email protected]; Krishnakant Jaju <[email protected]>
Subject: RE: [RFC PATCH 00/16] hw/arm: Introduce Tegra241 CMDQV support for accelerated SMMUv3

Hi Eric,

Quoted 4 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 11 December 2025 17:55
To: Shameer Kolothum <[email protected]>; qemu-

[email protected];

Quoted 3 lines

[email protected]
Cc: [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>;

Jason

Quoted 7 lines

Gunthorpe <[email protected]>; [email protected]; [email protected]; [email protected]; Krishnakant Jaju <[email protected]>
Subject: Re: [RFC PATCH 00/16] hw/arm: Introduce Tegra241 CMDQV support for accelerated SMMUv3

External email: Use caution opening links or attachments

[...]

Quoted 7 lines
Quoted 2 lines

[0] https://lore.kernel.org/qemu-devel/20251120132213.56581-1-

[email protected]/

Quoted 3 lines

[1]
https://lore.kernel.org/all/[email protected]
/

Quoted 5 lines
Quoted 1 line

do you have a branch to share with all the bits?

Here:
https://github.com/shamiali2008/qemu-master.git master-smmuv3-accel- v6-veventq-v2-vcmdq-rfcv1

I just realised this needs a fix in patch #16,

hw/arm/smmuv3.c +1 -1
index 58c35c2af3..c32b35a9a7 100644
@@ -1954,7 +1954,7 @@ static bool smmu_validate_property(SMMUv3State *s, Error **errp)
         return false;
     }
 #endif
-#ifndef CONFIG_TEGRA241_CMDQ
+#ifndef CONFIG_TEGRA241_CMDQV
     if (s->tegra241_cmdqv) {
         error_setg(errp, "tegra241_cmdqv=on support not compiled in");
         return false;

--

Pushed to above branch.

Thanks,
Shameer

#66 lore raw
In reply to Shameer Kolothum
Quoted 17 lines · Shameer Kolothum

Hi,

This RFC series adds initial support for NVIDIA Tegra241 CMDQV (Command Queue Virtualisation), an extension to ARM SMMUv3 that provides hardware accelerated virtual command queues (VCMDQs) for guests. CMDQV allows guests to issue SMMU invalidation commands directly to hardware without VM exits, significantly reducing TLBI overhead.

Thanks to Nicolin for the initial patches and testing on which this RFC is based.

This is based on v6[0] of the SMMUv3 accel series, which is still under review, though nearing convergence. This is sent as an RFC, with the goal of gathering early feedback on the CMDQV design and its integration with the SMMUv3 acceleration path.

Quoted 4 lines

Background:

Tegra241 CMDQV extends SMMUv3 by allocating per-VM "virtual interfaces" (VINTFs), each hosting up to 128 VCMDQs.

Can you add a reference to some specification documet please?

Quoted 6 lines

Each VINTF exposes two 64KB MMIO pages:
- Page0 – guest owned control and status registers (directly mapped into the VM)
- Page1 – queue configuration registers (trapped/emulated by QEMU)

Quoted 6 lines

Unlike the standard SMMU CMDQ, a guest owned Tegra241 VCMDQ does not support the full command set. Only a subset, primarily invalidation related commands, is accepted by the CMDQV hardware. For this reason, a distinct CMDQV device must be exposed to the guest, and the guest OS must include a Tegra241 CMDQV aware driver to take advantage of the hardware acceleration.

Do I understand correctly that this Tegra241 CMDQV aware driveris enabled by the CONFIG_TEGRA241_CMDQV on guest? Is it fully supported upstream?

Eric

Quoted 82 lines

VCMDQ support is integrated via the IOMMU_HW_QUEUE_ALLOC mechanism, allowing QEMU to attach guest configured VCMDQ buffers to the underlying CMDQV hardware through IOMMUFD. The Linux kernel already supports the full CMDQV virtualisation model via IOMMUFD[0].

Summary of QEMU changes:

- Integrated into the existing SMMUv3 accel path via a "tegra241-cmdqv" property.
- Support for allocating vIOMMU objects of type IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV.
- Mapping and emulation of the CMDQV MMIO register layout.
- VCMDQ/VINTF read/write handling and queue allocation using IOMMUFD APIs.
- Reset and initialisation hooks, including checks for at least one cold-plugged device.
- CMDQV hardware reads guest queue memory using host physical addresses provided through IOMMUFD, which requires that the VCMDQ buffer be physically contiguous not only in guest PA space but also in host PA space. When Tegra241 CMDQV is enabled, QEMU must therefore only expose a CMDQV size that the host can reliably back with contiguous physical memory. Because of this constraint, it is suggested to use huge pages to back the guest RAM.
- ACPI DSDT node generation for CMDQV devices on the virt machine.

These patches have been sanity tested on NVIDIA Grace platforms.

ToDo / revisit:
- Prevent hot-unplug of the last device associated with vIOMMU as this might allow associating a different host SMMU/CMDQV.
- Locking requirements around error event propagation.

Feedback and testing are very welcome.

Thanks,
Shameer
[0] https://lore.kernel.org/qemu-devel/[email protected]/
[1] https://lore.kernel.org/all/[email protected]/

Nicolin Chen (12):
backends/iommufd: Update iommufd_backend_get_device_info backends/iommufd: Update iommufd_backend_alloc_viommu to allow user
ptr
backends/iommufd: Introduce iommufd_backend_alloc_hw_queue backends/iommufd: Introduce iommufd_backend_viommu_mmap hw/arm/tegra241-cmdqv: Add initial Tegra241 CMDQ-Virtualisation
support
hw/arm/tegra241-cmdqv: Map VINTF Page0 into guest hw/arm/tegra241-cmdqv: Add read emulation support for registers system/physmem: Add helper to check whether a guest PA maps to RAM hw/arm/tegra241-cmdqv:: Add write emulation for registers hw/arm/tegra241-cmdqv: Add reset handler hw/arm/tegra241-cmdqv: Limit queue size based on backend page size hw/arm/virt-acpi: Advertise Tegra241 CMDQV nodes in DSDT

Shameer Kolothum (4):
hw/arm/tegra241-cmdqv: Allocate vEVENTQ object hw/arm/tegra241-cmdqv: Read and propagate Tegra241 CMDQV errors
virt-acpi-build: Rename AcpiIortSMMUv3Dev to AcpiSMMUv3Dev hw/arm/smmuv3: Add tegra241-cmdqv property for SMMUv3 device

 backends/iommufd.c        |  65 ++++
 backends/trace-events     |   2 +
 hw/arm/Kconfig            |   5 +
 hw/arm/meson.build        |   1 +
 hw/arm/smmuv3-accel.c     |  16 +-
 hw/arm/smmuv3.c           |  18 +
 hw/arm/tegra241-cmdqv.c   | 759 ++++++++++++++++++++++++++++++++++++++
 hw/arm/tegra241-cmdqv.h   | 337 +++++++++++++++++
 hw/arm/trace-events       |   5 +
 hw/arm/virt-acpi-build.c  | 110 +++++-
 hw/vfio/iommufd.c         |   6 +-
 include/exec/cpu-common.h |   2 +
 include/hw/arm/smmuv3.h   |   3 +
 include/hw/arm/virt.h     |   2 +
 include/system/iommufd.h  |  16 +
 system/physmem.c          |  12 +
 16 files changed, 1332 insertions(+), 27 deletions(-)
 create mode 100644 hw/arm/tegra241-cmdqv.c
 create mode 100644 hw/arm/tegra241-cmdqv.h
#67 lore raw
Quoted 50 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Sent: 12 January 2026 15:44
To: Shameer Kolothum <[email protected]>; qemu- [email protected]; [email protected]
Cc: [email protected]; Nicolin Chen <[email protected]>; Nathan Chen <[email protected]>; Matt Ochs <[email protected]>; Jason Gunthorpe <[email protected]>; [email protected]; [email protected]; [email protected]; Krishnakant Jaju <[email protected]>
Subject: Re: [RFC PATCH 00/16] hw/arm: Introduce Tegra241 CMDQV support for accelerated SMMUv3

External email: Use caution opening links or attachments

Quoted 12 lines · Shameer Kolothum

Hi,

This RFC series adds initial support for NVIDIA Tegra241 CMDQV (Command Queue Virtualisation), an extension to ARM SMMUv3 that provides hardware accelerated virtual command queues (VCMDQs) for guests. CMDQV allows guests to issue SMMU invalidation commands directly to hardware without VM exits, significantly reducing TLBI overhead.

Thanks to Nicolin for the initial patches and testing on which this RFC is based.

Quoted 8 lines

This is based on v6[0] of the SMMUv3 accel series, which is still under review, though nearing convergence. This is sent as an RFC, with the goal of gathering early feedback on the CMDQV design and its integration with the SMMUv3 acceleration path.

Background:

Tegra241 CMDQV extends SMMUv3 by allocating per-VM "virtual

interfaces"

Quoted 1 line

(VINTFs), each hosting up to 128 VCMDQs.

Can you add a reference to some specification documet please?

Quoted 7 lines

Each VINTF exposes two 64KB MMIO pages:
- Page0 – guest owned control and status registers (directly mapped into the VM)
- Page1 – queue configuration registers (trapped/emulated by QEMU)

Unlike the standard SMMU CMDQ, a guest owned Tegra241 VCMDQ does

not

Quoted 8 lines
Quoted 5 lines

support the full command set. Only a subset, primarily invalidation related commands, is accepted by the CMDQV hardware. For this reason, a distinct CMDQV device must be exposed to the guest, and the guest OS must include a Tegra241 CMDQV aware driver to take advantage of the hardware acceleration.

Do I understand correctly that this Tegra241 CMDQV aware driveris enabled by the CONFIG_TEGRA241_CMDQV on guest? Is it fully supported upstream?

Yes. With CONFIG_TEGRA241_CMDQV enabled, it should work.

Thanks,
Shameer